2024-01-15T17:44:18.043 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-15T17:44:18.043 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_rishabh@teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --archive /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115 --name rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/orig.config.yaml 2024-01-15T17:44:18.083 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T17:44:18.161 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7517115' kernel: flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8754 sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVN2HzCjZzUS/rDqxDK4fI5PQ6Hqfv8NUsoE7qkLEBkoNhkeIfrx8AYn6LBKQs/L1bP/Xx3vpgnfhF63n6FjdE= smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbj9Ii0wnblDymIKws/pcdeuhJ0D9SbBrDDD61dzeofHosaESMxAJlvn1GsMltQv695/PYjgdKxaC35iKRBjj8= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-15_17:39:27 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-15T17:44:18.161 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa; will attempt to use it 2024-01-15T17:44:18.161 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks 2024-01-15T17:44:18.162 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-15T17:44:18.163 INFO:teuthology.task.internal:Checking packages... 2024-01-15T17:44:18.183 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '95c90f413c4bb65e3e56909855f2308fd9bda05b' 2024-01-15T17:44:18.183 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-15T17:44:18.183 INFO:teuthology.packaging:ref: None 2024-01-15T17:44:18.183 INFO:teuthology.packaging:tag: None 2024-01-15T17:44:18.183 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-15T17:44:18.183 INFO:teuthology.packaging:sha1: 95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T17:44:18.183 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-clone-stats 2024-01-15T17:44:18.341 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g95c90f41 2024-01-15T17:44:18.342 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-15T17:44:18.348 INFO:teuthology.task.internal:no buildpackages task found 2024-01-15T17:44:18.348 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-15T17:44:18.354 INFO:teuthology.task.internal:Saving configuration 2024-01-15T17:44:18.369 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-15T17:44:18.374 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T17:44:18.398 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:39:34.525941', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVN2HzCjZzUS/rDqxDK4fI5PQ6Hqfv8NUsoE7qkLEBkoNhkeIfrx8AYn6LBKQs/L1bP/Xx3vpgnfhF63n6FjdE='} 2024-01-15T17:44:18.424 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:39:34.527854', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbj9Ii0wnblDymIKws/pcdeuhJ0D9SbBrDDD61dzeofHosaESMxAJlvn1GsMltQv695/PYjgdKxaC35iKRBjj8='} 2024-01-15T17:44:18.424 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-15T17:44:18.430 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-15T17:44:18.430 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-15T17:44:18.430 INFO:teuthology.run_tasks:Running task console_log... 2024-01-15T17:44:18.495 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5ce5af5160>, signals=[15]) 2024-01-15T17:44:18.496 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-15T17:44:18.501 INFO:teuthology.task.internal:Opening connections... 2024-01-15T17:44:18.501 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:44:18.503 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:44:18.605 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:44:18.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:44:18.688 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-15T17:44:18.694 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-01-15T17:44:18.711 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-01-15T17:44:18.711 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-01-15T17:44:18.768 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-15T17:44:18.769 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-15T17:44:18.769 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-01-15T17:44:18.796 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-01-15T17:44:18.813 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-01-15T17:44:18.814 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-01-15T17:44:18.871 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-01-15T17:44:18.871 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-01-15T17:44:18.871 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-01-15T17:44:18.871 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-01-15T17:44:18.871 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-01-15T17:44:18.872 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-01-15T17:44:18.872 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-15T17:44:18.872 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-01-15T17:44:18.872 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-01-15T17:44:18.872 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-15T17:44:18.872 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-01-15T17:44:18.873 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-15T17:44:18.873 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-15T17:44:18.873 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-15T17:44:18.873 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-01-15T17:44:18.895 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-15T17:44:18.904 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-15T17:44:18.909 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-15T17:44:18.910 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-01-15T17:44:18.912 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-01-15T17:44:18.928 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-15T17:44:18.934 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-15T17:44:18.934 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-01-15T17:44:18.970 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-01-15T17:44:18.994 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-15T17:44:19.046 INFO:teuthology.run_tasks:Running task kernel... 2024-01-15T17:44:19.066 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-15T17:44:19.066 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, timeout 300 2024-01-15T17:44:19.066 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T17:44:19.067 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-15T17:44:19.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:44:19.083 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:44:19.083 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-15T17:44:19.083 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-01-15T17:44:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:44:19.084 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:44:19.084 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-15T17:44:19.084 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-01-15T17:44:19.139 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-361.el9.x86_64 2024-01-15T17:44:19.139 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:44:19.139 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-15T17:44:19.139 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:44:19.142 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-361.el9.x86_64 2024-01-15T17:44:19.143 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:44:19.143 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-15T17:44:19.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:44:19.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum upgrade -y linux-firmware 2024-01-15T17:44:19.284 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum upgrade -y linux-firmware 2024-01-15T17:44:21.541 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 10 MB/s | 6.4 MB 00:00 2024-01-15T17:44:21.736 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 7.8 MB/s | 6.4 MB 00:00 2024-01-15T17:44:23.211 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2024-01-15T17:44:23.283 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-01-15T17:44:25.015 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 7.9 MB 00:01 2024-01-15T17:44:25.185 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 7.9 MB 00:01 2024-01-15T17:44:27.619 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 14 MB/s | 18 MB 00:01 2024-01-15T17:44:31.465 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 18 MB 00:05 2024-01-15T17:44:32.170 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 6.6 MB/s | 5.9 MB 00:00 2024-01-15T17:44:34.096 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 15 kB 00:00 2024-01-15T17:44:35.564 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-01-15T17:44:35.757 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 5.9 MB 00:00 2024-01-15T17:44:37.108 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 15 kB 00:00 2024-01-15T17:44:39.595 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 20 MB 00:02 2024-01-15T17:44:40.497 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-01-15T17:44:42.853 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-01-15T17:44:42.868 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-01-15T17:44:42.868 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-01-15T17:44:42.868 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-01-15T17:44:42.868 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2024-01-15T17:44:42.868 INFO:teuthology.orchestra.run.smithi002.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 1 Package 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 375 M 2024-01-15T17:44:42.869 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-01-15T17:44:44.343 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-01-15T17:44:46.652 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-15T17:44:46.663 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 1 Package 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:44:46.664 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 375 M 2024-01-15T17:44:46.665 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-15T17:44:55.385 INFO:teuthology.orchestra.run.smithi137.stdout:linux-firmware-20240109-142.el9.noarch.rpm 43 MB/s | 375 MB 00:08 2024-01-15T17:44:55.386 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-15T17:44:55.386 INFO:teuthology.orchestra.run.smithi137.stdout:Total 43 MB/s | 375 MB 00:08 2024-01-15T17:44:55.386 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-15T17:44:55.470 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-15T17:44:55.470 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-15T17:44:57.282 INFO:teuthology.orchestra.run.smithi002.stdout:linux-firmware-20240109-142.el9.noarch.rpm 26 MB/s | 375 MB 00:14 2024-01-15T17:44:57.283 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-01-15T17:44:57.284 INFO:teuthology.orchestra.run.smithi002.stdout:Total 26 MB/s | 375 MB 00:14 2024-01-15T17:44:57.284 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-01-15T17:44:57.389 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-01-15T17:44:57.389 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-01-15T17:44:57.848 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-15T17:44:57.848 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-15T17:44:59.882 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-01-15T17:44:59.882 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-01-15T17:45:04.144 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-15T17:45:04.402 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-15T17:45:04.483 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-15T17:45:05.904 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-15T17:45:05.904 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-15T17:45:06.296 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-15T17:45:06.296 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:45:06.296 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-01-15T17:45:06.297 INFO:teuthology.orchestra.run.smithi137.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-15T17:45:06.297 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:45:06.297 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-15T17:45:06.388 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-01-15T17:45:06.603 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-15T17:45:06.603 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:45:06.676 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-15T17:45:06.747 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-15T17:45:06.904 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-15T17:45:06.904 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-15T17:45:06.907 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi137.front.sepia.ceph.com to complete... 2024-01-15T17:45:07.891 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-15T17:45:07.892 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-15T17:45:08.287 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-15T17:45:08.287 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:45:08.287 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2024-01-15T17:45:08.287 INFO:teuthology.orchestra.run.smithi002.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-15T17:45:08.287 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:45:08.288 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-01-15T17:45:08.555 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-15T17:45:08.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-15T17:45:08.810 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-15T17:45:08.811 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-15T17:45:08.814 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi002.front.sepia.ceph.com to complete... 2024-01-15T17:45:20.752 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15 17:45:20 URL:https://2.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330211636/330211636] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-15T17:45:20.753 INFO:teuthology.orchestra.run.smithi137.stderr:FINISHED --2024-01-15 17:45:20-- 2024-01-15T17:45:20.753 INFO:teuthology.orchestra.run.smithi137.stderr:Total wall clock time: 14s 2024-01-15T17:45:20.753 INFO:teuthology.orchestra.run.smithi137.stderr:Downloaded: 1 files, 315M in 14s (23.2 MB/s) 2024-01-15T17:45:20.756 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-15T17:45:20.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-15T17:45:22.649 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying... ######################################## 2024-01-15T17:45:22.779 INFO:teuthology.orchestra.run.smithi137.stdout:Preparing... ######################################## 2024-01-15T17:45:22.857 INFO:teuthology.orchestra.run.smithi137.stdout:Updating / installing... 2024-01-15T17:45:29.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15 17:45:29 URL:https://2.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330211636/330211636] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-15T17:45:29.681 INFO:teuthology.orchestra.run.smithi002.stderr:FINISHED --2024-01-15 17:45:29-- 2024-01-15T17:45:29.681 INFO:teuthology.orchestra.run.smithi002.stderr:Total wall clock time: 21s 2024-01-15T17:45:29.681 INFO:teuthology.orchestra.run.smithi002.stderr:Downloaded: 1 files, 315M in 20s (15.5 MB/s) 2024-01-15T17:45:29.685 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-15T17:45:29.685 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-15T17:45:30.185 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-15T17:45:31.682 INFO:teuthology.orchestra.run.smithi002.stdout:Verifying... ######################################## 2024-01-15T17:45:31.748 INFO:teuthology.orchestra.run.smithi002.stdout:Preparing... ######################################## 2024-01-15T17:45:31.832 INFO:teuthology.orchestra.run.smithi002.stdout:Updating / installing... 2024-01-15T17:45:39.526 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-15T17:45:45.741 INFO:teuthology.orchestra.run.smithi137.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-15T17:45:55.170 INFO:teuthology.orchestra.run.smithi002.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-15T17:46:07.683 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-15T17:46:07.683 DEBUG:teuthology.orchestra.run.smithi137:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-15T17:46:07.760 INFO:teuthology.orchestra.run.smithi137.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-15T17:46:07.760 INFO:teuthology.orchestra.run.smithi137.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-15T17:46:07.760 INFO:teuthology.orchestra.run.smithi137.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:46:07.760 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-15T17:46:07.761 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-15T17:46:07.762 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-15T17:46:07.763 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-15T17:46:07.764 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-15T17:46:07.765 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-15T17:46:07.766 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-15T17:46:07.767 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-15T17:46:07.768 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-15T17:46:07.769 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-15T17:46:07.770 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-15T17:46:07.771 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-15T17:46:07.772 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-15T17:46:07.773 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-15T17:46:07.774 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-15T17:46:07.775 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-15T17:46:07.776 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-15T17:46:07.777 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-15T17:46:07.778 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-15T17:46:07.779 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-15T17:46:07.780 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-15T17:46:07.781 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-15T17:46:07.782 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-15T17:46:07.783 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-15T17:46:07.784 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-15T17:46:07.785 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-15T17:46:07.786 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-15T17:46:07.787 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-15T17:46:07.788 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-15T17:46:07.789 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-15T17:46:07.790 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-15T17:46:07.791 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-15T17:46:07.792 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-15T17:46:07.793 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-15T17:46:07.794 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-15T17:46:07.795 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-15T17:46:07.796 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-15T17:46:07.797 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-15T17:46:07.798 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-15T17:46:07.799 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-15T17:46:07.800 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-15T17:46:07.801 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-15T17:46:07.802 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-15T17:46:07.803 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-15T17:46:07.804 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-15T17:46:07.805 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-15T17:46:07.806 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-15T17:46:07.807 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-15T17:46:07.808 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-15T17:46:07.809 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-15T17:46:07.810 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-15T17:46:07.811 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-15T17:46:07.812 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-15T17:46:07.813 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-15T17:46:07.814 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-15T17:46:07.815 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-15T17:46:07.816 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-15T17:46:07.817 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-15T17:46:07.818 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-15T17:46:07.819 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-15T17:46:07.820 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-15T17:46:07.821 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-15T17:46:07.822 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-15T17:46:07.823 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-15T17:46:07.824 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-15T17:46:07.825 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-15T17:46:07.826 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-15T17:46:07.827 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-15T17:46:07.828 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-15T17:46:07.829 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-15T17:46:07.830 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-15T17:46:07.831 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-15T17:46:07.832 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-15T17:46:07.833 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-15T17:46:07.834 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-15T17:46:07.835 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-15T17:46:07.836 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-15T17:46:07.837 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-15T17:46:07.838 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-15T17:46:07.839 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-15T17:46:07.840 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-15T17:46:07.841 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-15T17:46:07.842 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-15T17:46:07.843 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-15T17:46:07.844 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-15T17:46:07.845 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-15T17:46:07.846 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-15T17:46:07.847 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-15T17:46:07.848 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-15T17:46:07.849 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-15T17:46:07.850 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-15T17:46:07.851 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-15T17:46:07.852 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-15T17:46:07.853 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-15T17:46:07.854 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-15T17:46:07.855 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-15T17:46:07.856 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-15T17:46:07.857 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-15T17:46:07.858 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-15T17:46:07.859 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-15T17:46:07.860 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-15T17:46:07.861 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-15T17:46:07.862 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-15T17:46:07.863 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-15T17:46:07.864 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-15T17:46:07.865 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-15T17:46:07.866 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-15T17:46:07.867 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-15T17:46:07.868 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-15T17:46:07.869 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-15T17:46:07.870 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-15T17:46:07.871 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-15T17:46:07.872 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-15T17:46:07.873 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-15T17:46:07.874 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-15T17:46:07.875 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-15T17:46:07.876 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-15T17:46:07.877 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-15T17:46:07.878 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-15T17:46:07.879 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-15T17:46:07.880 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-15T17:46:07.881 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-15T17:46:07.882 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-15T17:46:07.883 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-15T17:46:07.884 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-15T17:46:07.885 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-15T17:46:07.886 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-15T17:46:07.887 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-15T17:46:07.888 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-15T17:46:07.889 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-15T17:46:07.890 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-15T17:46:07.891 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-15T17:46:07.892 INFO:teuthology.orchestra.run.smithi137.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-15T17:46:07.893 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-15T17:46:07.893 DEBUG:teuthology.orchestra.run.smithi137:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:postinstall scriptlet (using /bin/sh): 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:fi 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:fi 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-15T17:46:07.915 INFO:teuthology.orchestra.run.smithi137.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-15T17:46:07.916 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-15T17:46:07.916 INFO:teuthology.orchestra.run.smithi137.stdout:fi 2024-01-15T17:46:07.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 68.el9 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8264254 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-15T17:46:07.991 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:46:07.992 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-01-15T17:46:07.993 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-15T17:46:07.993 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 6.7.0-g2668949e0971 2024-01-15T17:46:07.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-15T17:46:08.612 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-01-15T17:46:10.030 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-15T17:46:10.050 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-01-15T17:46:10.052 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-15T17:46:10.067 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.6VxeYfacH0 2024-01-15T17:46:10.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6VxeYfacH0 2024-01-15T17:46:10.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.6VxeYfacH0 2024-01-15T17:46:10.279 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.6VxeYfacH0 is 6KB 2024-01-15T17:46:10.329 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.6VxeYfacH0 2024-01-15T17:46:10.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-01-15T17:46:10.422 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-15T17:46:10.422 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-15T17:46:10.422 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-15T17:46:10.422 INFO:teuthology.orchestra.run.smithi137.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-15T17:46:10.423 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-15T17:46:10.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-01-15T17:46:18.149 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-15T17:46:18.149 DEBUG:teuthology.orchestra.run.smithi002:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-15T17:46:18.240 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-15T17:46:18.241 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-15T17:46:18.242 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-15T17:46:18.243 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-15T17:46:18.244 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-15T17:46:18.245 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-15T17:46:18.246 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-15T17:46:18.247 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-15T17:46:18.248 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-15T17:46:18.249 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-15T17:46:18.250 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-15T17:46:18.251 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-15T17:46:18.252 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-15T17:46:18.253 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-15T17:46:18.254 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-15T17:46:18.255 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-15T17:46:18.256 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-15T17:46:18.257 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-15T17:46:18.258 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-15T17:46:18.259 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-15T17:46:18.260 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-15T17:46:18.261 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-15T17:46:18.262 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-15T17:46:18.263 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-15T17:46:18.264 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-15T17:46:18.265 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-15T17:46:18.266 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-15T17:46:18.267 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-15T17:46:18.268 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-15T17:46:18.269 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-15T17:46:18.270 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-15T17:46:18.271 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-15T17:46:18.272 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-15T17:46:18.273 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-15T17:46:18.274 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-15T17:46:18.275 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-15T17:46:18.276 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-15T17:46:18.277 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-15T17:46:18.278 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-15T17:46:18.279 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-15T17:46:18.280 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-15T17:46:18.281 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-15T17:46:18.282 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-15T17:46:18.283 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-15T17:46:18.284 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-15T17:46:18.285 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-15T17:46:18.286 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-15T17:46:18.287 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-15T17:46:18.288 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-15T17:46:18.289 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-15T17:46:18.290 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-15T17:46:18.291 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-15T17:46:18.292 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-15T17:46:18.293 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-15T17:46:18.294 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-15T17:46:18.295 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-15T17:46:18.296 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-15T17:46:18.297 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-15T17:46:18.298 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-15T17:46:18.299 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-15T17:46:18.300 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-15T17:46:18.301 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-15T17:46:18.302 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-15T17:46:18.303 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-15T17:46:18.304 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-15T17:46:18.305 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-15T17:46:18.306 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-15T17:46:18.307 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-15T17:46:18.308 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-15T17:46:18.309 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-15T17:46:18.310 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-15T17:46:18.311 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-15T17:46:18.312 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-15T17:46:18.313 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-15T17:46:18.314 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-15T17:46:18.315 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-15T17:46:18.316 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-15T17:46:18.317 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-15T17:46:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-15T17:46:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-15T17:46:18.320 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-15T17:46:18.321 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-15T17:46:18.322 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-15T17:46:18.323 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-15T17:46:18.324 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-15T17:46:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-15T17:46:18.326 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-15T17:46:18.327 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-15T17:46:18.328 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-15T17:46:18.329 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-15T17:46:18.330 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-15T17:46:18.331 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-15T17:46:18.332 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-15T17:46:18.333 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-15T17:46:18.334 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-15T17:46:18.335 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-15T17:46:18.336 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-15T17:46:18.337 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-15T17:46:18.338 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-15T17:46:18.339 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-15T17:46:18.340 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-15T17:46:18.341 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-15T17:46:18.342 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-15T17:46:18.343 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-15T17:46:18.344 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-15T17:46:18.345 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-15T17:46:18.346 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-15T17:46:18.347 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-15T17:46:18.348 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-15T17:46:18.349 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-15T17:46:18.350 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-15T17:46:18.351 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-15T17:46:18.352 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-15T17:46:18.353 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-15T17:46:18.354 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-15T17:46:18.355 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-15T17:46:18.356 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-15T17:46:18.357 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-15T17:46:18.358 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-15T17:46:18.359 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-15T17:46:18.360 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-15T17:46:18.361 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-15T17:46:18.362 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-15T17:46:18.363 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-15T17:46:18.364 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-15T17:46:18.365 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-15T17:46:18.366 INFO:teuthology.orchestra.run.smithi002.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-15T17:46:18.366 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-15T17:46:18.366 DEBUG:teuthology.orchestra.run.smithi002:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-15T17:46:18.389 INFO:teuthology.orchestra.run.smithi002.stdout:postinstall scriptlet (using /bin/sh): 2024-01-15T17:46:18.389 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-15T17:46:18.390 INFO:teuthology.orchestra.run.smithi002.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-15T17:46:18.391 INFO:teuthology.orchestra.run.smithi002.stdout:fi 2024-01-15T17:46:18.391 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 68.el9 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-01-15T17:46:18.481 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8264254 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:46:18.482 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-01-15T17:46:18.483 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-15T17:46:18.483 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 6.7.0-g2668949e0971 2024-01-15T17:46:18.483 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-15T17:46:19.053 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-01-15T17:46:20.536 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-15T17:46:20.556 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-01-15T17:46:20.558 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-01-15T17:46:20.574 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.W6CPXTyJYE 2024-01-15T17:46:20.574 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.W6CPXTyJYE 2024-01-15T17:46:20.646 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.W6CPXTyJYE 2024-01-15T17:46:20.787 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.W6CPXTyJYE is 6KB 2024-01-15T17:46:20.809 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.W6CPXTyJYE 2024-01-15T17:46:20.824 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-01-15T17:46:20.892 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-15T17:46:20.892 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-15T17:46:20.892 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-15T17:46:20.892 INFO:teuthology.orchestra.run.smithi002.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-15T17:46:20.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-15T17:46:21.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-01-15T17:46:40.600 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-15T17:46:40.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-15T17:46:40.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:46:51.052 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-15T17:46:51.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-01-15T17:46:51.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:46:59.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-15T17:47:08.014 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-15T17:47:08.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:47:09.501 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2024-01-15T17:47:11.068 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-01-15T17:47:18.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-01-15T17:47:18.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:47:23.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-15T17:47:23.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:47:54.930 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-01-15T17:47:55.534 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-15T17:47:55.534 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-g2668949e0971"... 2024-01-15T17:47:55.534 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-01-15T17:47:55.575 INFO:teuthology.orchestra.run.smithi137.stdout:6.7.0-g2668949e0971 2024-01-15T17:47:55.575 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-15T17:47:55.575 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-15T17:47:55.575 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-15T17:47:56.576 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-15T17:47:56.577 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-15T17:47:56.702 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-01-15T17:47:56.755 DEBUG:teuthology.parallel:result is None 2024-01-15T17:48:18.511 DEBUG:teuthology.orchestra.remote:timed out 2024-01-15T17:48:30.512 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-01-15T17:48:30.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:48:30.897 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-01-15T17:48:31.586 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-01-15T17:48:31.586 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-g2668949e0971"... 2024-01-15T17:48:31.586 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-01-15T17:48:31.642 INFO:teuthology.orchestra.run.smithi002.stdout:6.7.0-g2668949e0971 2024-01-15T17:48:31.643 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-15T17:48:31.643 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-15T17:48:31.643 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-15T17:48:32.643 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-15T17:48:32.644 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-15T17:48:32.789 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-01-15T17:48:32.837 DEBUG:teuthology.parallel:result is None 2024-01-15T17:48:32.837 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-15T17:48:32.846 INFO:teuthology.task.internal:Creating test directory... 2024-01-15T17:48:32.846 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T17:48:32.853 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-15T17:48:32.911 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-15T17:48:32.922 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-15T17:48:32.943 INFO:teuthology.task.internal:Creating archive directory... 2024-01-15T17:48:32.944 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T17:48:32.951 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-15T17:48:33.021 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-15T17:48:33.033 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-15T17:48:33.033 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T17:48:33.040 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-15T17:48:33.164 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:48:33.167 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:48:33.209 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:48:33.211 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-15T17:48:33.216 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-15T17:48:33.224 INFO:teuthology.task.internal:Configuring sudo... 2024-01-15T17:48:33.225 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T17:48:33.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-15T17:48:33.325 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-15T17:48:33.335 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-15T17:48:33.336 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T17:48:33.344 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-15T17:48:33.397 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T17:48:33.486 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T17:48:33.609 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T17:48:33.698 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T17:48:33.800 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:48:33.800 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T17:48:33.898 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T17:48:33.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-15T17:48:34.094 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T17:48:34.178 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-15T17:48:34.273 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:48:34.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-15T17:48:34.367 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-01-15T17:48:34.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-01-15T17:48:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T17:48:34.537 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T17:48:34.883 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-15T17:48:34.891 INFO:teuthology.task.internal:Starting timer... 2024-01-15T17:48:34.891 INFO:teuthology.run_tasks:Running task pcp... 2024-01-15T17:48:34.913 INFO:teuthology.run_tasks:Running task selinux... 2024-01-15T17:48:34.922 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-01-15T17:48:35.051 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-01-15T17:48:35.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-01-15T17:48:35.207 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-01-15T17:48:35.211 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-15T17:48:35.211 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-01-15T17:48:35.278 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-01-15T17:48:35.279 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-01-15T17:48:35.342 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-01-15T17:48:35.343 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi137.front.sepia.ceph.com': 'permissive'} 2024-01-15T17:48:35.343 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T17:48:35.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:48:35.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T17:48:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:48:35.532 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-15T17:48:35.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-01-15T17:48:35.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-01-15T17:48:35.748 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-15T17:48:35.757 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-01-15T17:48:35.763 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-01-15T17:48:35.868 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-15T17:48:35.885 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-15T17:48:35.886 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi137.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-15T17:54:21.685 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi137.front.sepia.ceph.com')] 2024-01-15T17:54:21.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-01-15T17:54:21.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:54:21.834 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-01-15T17:54:21.993 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-01-15T17:54:21.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-15T17:54:21.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-15T17:54:22.134 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-01-15T17:54:22.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-01-15T17:54:22.283 INFO:teuthology.run_tasks:Running task clock... 2024-01-15T17:54:22.293 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-15T17:54:22.293 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T17:54:22.294 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T17:54:22.302 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-15T17:54:22.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T17:54:22.401 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T17:54:22.401 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-15T17:54:22.454 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T17:54:22.458 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-15T17:54:22.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-01-15T17:54:22.557 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-01-15T17:54:22.586 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-01-15T17:54:22.604 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-01-15T17:54:22.637 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T17:54:22.660 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-15T17:54:22.693 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T17:54:22.714 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-15T17:54:22.820 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-01-15T17:54:22.830 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T17:54:22.830 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-01-15T17:54:22.830 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.830 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.830 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.830 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.844 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-01-15T17:54:22.851 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T17:54:22.851 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-01-15T17:54:22.851 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.851 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.852 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.852 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T17:54:22.853 INFO:teuthology.run_tasks:Running task install... 2024-01-15T17:54:22.866 DEBUG:teuthology.task.install:project ceph 2024-01-15T17:54:22.866 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}} 2024-01-15T17:54:22.866 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'} 2024-01-15T17:54:22.867 INFO:teuthology.task.install:Using flavor: default 2024-01-15T17:54:22.876 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-15T17:54:22.876 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-15T17:54:22.877 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'tag': None, 'wait_for_package': False} 2024-01-15T17:54:22.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T17:54:22.879 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'tag': None, 'wait_for_package': False} 2024-01-15T17:54:22.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T17:54:23.028 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/ 2024-01-15T17:54:23.028 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g95c90f41 2024-01-15T17:54:23.200 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/ 2024-01-15T17:54:23.200 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g95c90f41 2024-01-15T17:54:23.307 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-15T17:54:23.307 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:54:23.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-15T17:54:23.315 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/rishabh-clone-stats/95c90f413c4bb65e3e56909855f2308fd9bda05b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-15T17:54:23.315 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:54:23.315 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-15T17:54:23.405 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-15T17:54:23.405 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/95c90f413c4bb65e3e56909855f2308fd9bda05b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-15T17:54:23.424 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-15T17:54:23.425 DEBUG:teuthology.orchestra.run.smithi002:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/95c90f413c4bb65e3e56909855f2308fd9bda05b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-15T17:54:23.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-15T17:54:23.565 DEBUG:teuthology.orchestra.run.smithi002:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-15T17:54:23.675 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-15T17:54:23.699 DEBUG:teuthology.orchestra.run.smithi002:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-15T17:54:23.760 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-01-15T17:54:23.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-01-15T17:54:23.796 INFO:teuthology.orchestra.run.smithi002.stdout:check_obsoletes = 1 2024-01-15T17:54:23.801 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum clean all 2024-01-15T17:54:24.123 INFO:teuthology.orchestra.run.smithi137.stdout:73 files removed 2024-01-15T17:54:24.163 INFO:teuthology.orchestra.run.smithi002.stdout:73 files removed 2024-01-15T17:54:24.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-15T17:54:24.203 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-15T17:54:25.226 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 139 kB/s | 78 kB 00:00 2024-01-15T17:54:25.294 INFO:teuthology.orchestra.run.smithi002.stdout:ceph packages for x86_64 137 kB/s | 78 kB 00:00 2024-01-15T17:54:25.801 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 20 kB/s | 9.5 kB 00:00 2024-01-15T17:54:25.811 INFO:teuthology.orchestra.run.smithi002.stdout:ceph noarch packages 23 kB/s | 9.5 kB 00:00 2024-01-15T17:54:26.293 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-01-15T17:54:26.396 INFO:teuthology.orchestra.run.smithi002.stdout:ceph source packages 3.7 kB/s | 1.9 kB 00:00 2024-01-15T17:54:26.930 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2024-01-15T17:54:26.965 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2024-01-15T17:54:28.558 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-01-15T17:54:28.675 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-01-15T17:54:30.344 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 7.9 MB 00:01 2024-01-15T17:54:30.912 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 7.9 MB 00:01 2024-01-15T17:54:38.994 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 18 MB 00:07 2024-01-15T17:54:43.694 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 5.9 MB 00:00 2024-01-15T17:54:45.120 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 18 MB 00:12 2024-01-15T17:54:46.061 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 15 kB 00:01 2024-01-15T17:54:47.113 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2024-01-15T17:54:49.629 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 10 MB/s | 5.9 MB 00:00 2024-01-15T17:54:51.455 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-01-15T17:54:52.821 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-01-15T17:54:53.264 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 12 MB/s | 20 MB 00:01 2024-01-15T17:54:55.586 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-15T17:54:55.588 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-15T17:54:55.609 INFO:teuthology.orchestra.run.smithi137.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-15T17:54:55.610 INFO:teuthology.orchestra.run.smithi137.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-15T17:54:55.612 INFO:teuthology.orchestra.run.smithi137.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-15T17:54:55.613 INFO:teuthology.orchestra.run.smithi137.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-15T17:54:55.616 INFO:teuthology.orchestra.run.smithi137.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-15T17:54:55.619 INFO:teuthology.orchestra.run.smithi137.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-15T17:54:55.621 INFO:teuthology.orchestra.run.smithi137.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-15T17:54:55.701 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-01-15T17:54:55.716 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-01-15T17:54:55.716 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-01-15T17:54:55.716 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-01-15T17:54:55.716 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-01-15T17:54:55.716 INFO:teuthology.orchestra.run.smithi137.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 6.6 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 5.2 M 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 860 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 141 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 1.5 M 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 133 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 1.8 M 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 7.4 M 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 50 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 8.6 M 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 41 M 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 262 k 2024-01-15T17:54:55.717 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 774 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 217 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-top noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 18 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 31 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 718 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 127 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-15T17:54:55.718 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 162 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 320 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 298 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 99 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 85 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 3.0 M 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 170 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-15T17:54:55.719 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 3.3 M 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 3.0 M 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-15T17:54:55.720 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 19 M 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 28 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 2.1 M 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 246 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 4.4 M 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 18 M 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 15 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 25 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-15T17:54:55.721 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 163 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 476 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 4.8 M 2024-01-15T17:54:55.722 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-15T17:54:55.723 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 45 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 129 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-15T17:54:55.724 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-15T17:54:55.725 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-15T17:54:55.726 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-01-15T17:54:55.727 INFO:teuthology.orchestra.run.smithi137.stdout:============================================================================================ 2024-01-15T17:54:55.728 INFO:teuthology.orchestra.run.smithi137.stdout:Install 135 Packages 2024-01-15T17:54:55.728 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 16 Packages 2024-01-15T17:54:55.728 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:54:55.728 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 204 M 2024-01-15T17:54:55.728 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-01-15T17:54:56.323 INFO:teuthology.orchestra.run.smithi137.stdout:(1/151): ceph-18.0.0-7629.g95c90f41.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-01-15T17:54:56.556 INFO:teuthology.orchestra.run.smithi137.stdout:(2/151): ceph-fuse-18.0.0-7629.g95c90f41.el9.x8 3.6 MB/s | 860 kB 00:00 2024-01-15T17:54:56.614 INFO:teuthology.orchestra.run.smithi137.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 2.4 MB/s | 141 kB 00:00 2024-01-15T17:54:56.982 INFO:teuthology.orchestra.run.smithi137.stdout:(4/151): ceph-base-18.0.0-7629.g95c90f41.el9.x8 5.7 MB/s | 5.2 MB 00:00 2024-01-15T17:54:57.040 INFO:teuthology.orchestra.run.smithi137.stdout:(5/151): ceph-mds-18.0.0-7629.g95c90f41.el9.x86 4.9 MB/s | 2.1 MB 00:00 2024-01-15T17:54:57.199 INFO:teuthology.orchestra.run.smithi137.stdout:(6/151): ceph-mgr-18.0.0-7629.g95c90f41.el9.x86 6.8 MB/s | 1.5 MB 00:00 2024-01-15T17:54:57.724 INFO:teuthology.orchestra.run.smithi137.stdout:(7/151): ceph-mon-18.0.0-7629.g95c90f41.el9.x86 6.4 MB/s | 4.4 MB 00:00 2024-01-15T17:54:58.416 INFO:teuthology.orchestra.run.smithi137.stdout:(8/151): ceph-common-18.0.0-7629.g95c90f41.el9. 8.1 MB/s | 19 MB 00:02 2024-01-15T17:54:58.482 INFO:teuthology.orchestra.run.smithi137.stdout:(9/151): ceph-selinux-18.0.0-7629.g95c90f41.el9 384 kB/s | 25 kB 00:00 2024-01-15T17:54:58.908 INFO:teuthology.orchestra.run.smithi137.stdout:(10/151): ceph-radosgw-18.0.0-7629.g95c90f41.el 7.3 MB/s | 8.6 MB 00:01 2024-01-15T17:54:59.106 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-01-15T17:54:59.133 INFO:teuthology.orchestra.run.smithi137.stdout:(11/151): ceph-osd-18.0.0-7629.g95c90f41.el9.x8 9.2 MB/s | 18 MB 00:01 2024-01-15T17:54:59.167 INFO:teuthology.orchestra.run.smithi137.stdout:(12/151): cephfs-mirror-18.0.0-7629.g95c90f41.e 842 kB/s | 217 kB 00:00 2024-01-15T17:54:59.273 INFO:teuthology.orchestra.run.smithi137.stdout:(13/151): libcephfs-devel-18.0.0-7629.g95c90f41 225 kB/s | 31 kB 00:00 2024-01-15T17:54:59.351 INFO:teuthology.orchestra.run.smithi137.stdout:(14/151): libcephfs2-18.0.0-7629.g95c90f41.el9. 3.8 MB/s | 718 kB 00:00 2024-01-15T17:54:59.376 INFO:teuthology.orchestra.run.smithi137.stdout:(15/151): libcephsqlite-18.0.0-7629.g95c90f41.e 1.6 MB/s | 163 kB 00:00 2024-01-15T17:54:59.410 INFO:teuthology.orchestra.run.smithi137.stdout:(16/151): librados-devel-18.0.0-7629.g95c90f41. 2.1 MB/s | 127 kB 00:00 2024-01-15T17:54:59.468 INFO:teuthology.orchestra.run.smithi137.stdout:(17/151): libradosstriper1-18.0.0-7629.g95c90f4 5.1 MB/s | 476 kB 00:00 2024-01-15T17:54:59.535 INFO:teuthology.orchestra.run.smithi137.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g95 690 kB/s | 45 kB 00:00 2024-01-15T17:54:59.602 INFO:teuthology.orchestra.run.smithi137.stdout:(19/151): python3-ceph-common-18.0.0-7629.g95c9 1.9 MB/s | 129 kB 00:00 2024-01-15T17:54:59.703 INFO:teuthology.orchestra.run.smithi137.stdout:(20/151): python3-cephfs-18.0.0-7629.g95c90f41. 1.6 MB/s | 162 kB 00:00 2024-01-15T17:54:59.778 INFO:teuthology.orchestra.run.smithi137.stdout:(21/151): python3-rados-18.0.0-7629.g95c90f41.e 4.2 MB/s | 320 kB 00:00 2024-01-15T17:54:59.911 INFO:teuthology.orchestra.run.smithi137.stdout:(22/151): librgw2-18.0.0-7629.g95c90f41.el9.x86 9.5 MB/s | 4.8 MB 00:00 2024-01-15T17:54:59.945 INFO:teuthology.orchestra.run.smithi137.stdout:(23/151): python3-rbd-18.0.0-7629.g95c90f41.el9 1.8 MB/s | 298 kB 00:00 2024-01-15T17:54:59.970 INFO:teuthology.orchestra.run.smithi137.stdout:(24/151): python3-rgw-18.0.0-7629.g95c90f41.el9 1.7 MB/s | 99 kB 00:00 2024-01-15T17:55:00.004 INFO:teuthology.orchestra.run.smithi137.stdout:(25/151): rbd-fuse-18.0.0-7629.g95c90f41.el9.x8 1.4 MB/s | 85 kB 00:00 2024-01-15T17:55:00.071 INFO:teuthology.orchestra.run.smithi137.stdout:(26/151): rbd-nbd-18.0.0-7629.g95c90f41.el9.x86 2.5 MB/s | 170 kB 00:00 2024-01-15T17:55:00.138 INFO:teuthology.orchestra.run.smithi137.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 426 kB/s | 28 kB 00:00 2024-01-15T17:55:00.204 INFO:teuthology.orchestra.run.smithi137.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g95c90f4 2.0 MB/s | 133 kB 00:00 2024-01-15T17:55:00.347 INFO:teuthology.orchestra.run.smithi137.stdout:(29/151): rbd-mirror-18.0.0-7629.g95c90f41.el9. 8.0 MB/s | 3.0 MB 00:00 2024-01-15T17:55:00.430 INFO:teuthology.orchestra.run.smithi137.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g95c90 7.8 MB/s | 1.8 MB 00:00 2024-01-15T17:55:00.505 INFO:teuthology.orchestra.run.smithi137.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g95 3.2 MB/s | 246 kB 00:00 2024-01-15T17:55:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g95c90f41.e 766 kB/s | 50 kB 00:00 2024-01-15T17:55:00.631 INFO:teuthology.orchestra.run.smithi137.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g9 263 kB/s | 15 kB 00:00 2024-01-15T17:55:00.706 INFO:teuthology.orchestra.run.smithi137.stdout:(34/151): ceph-volume-18.0.0-7629.g95c90f41.el9 3.5 MB/s | 262 kB 00:00 2024-01-15T17:55:00.848 INFO:teuthology.orchestra.run.smithi137.stdout:(35/151): cephadm-18.0.0-7629.g95c90f41.el9.noa 5.4 MB/s | 774 kB 00:00 2024-01-15T17:55:00.915 INFO:teuthology.orchestra.run.smithi137.stdout:(36/151): cephfs-top-18.0.0-7629.g95c90f41.el9. 278 kB/s | 18 kB 00:00 2024-01-15T17:55:01.074 INFO:teuthology.orchestra.run.smithi137.stdout:(37/151): automake-1.16.2-8.el9.noarch.rpm 4.1 MB/s | 662 kB 00:00 2024-01-15T17:55:01.182 INFO:teuthology.orchestra.run.smithi137.stdout:(38/151): ceph-mgr-diskprediction-local-18.0.0- 8.9 MB/s | 7.4 MB 00:00 2024-01-15T17:55:01.224 INFO:teuthology.orchestra.run.smithi137.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 6.0 MB/s | 920 kB 00:00 2024-01-15T17:55:01.266 INFO:teuthology.orchestra.run.smithi137.stdout:(40/151): boost-program-options-1.75.0-8.el9.x8 1.4 MB/s | 106 kB 00:00 2024-01-15T17:55:01.292 INFO:teuthology.orchestra.run.smithi137.stdout:(41/151): elfutils-libelf-devel-0.190-2.el9.x86 312 kB/s | 20 kB 00:00 2024-01-15T17:55:01.325 INFO:teuthology.orchestra.run.smithi137.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-01-15T17:55:01.351 INFO:teuthology.orchestra.run.smithi137.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 573 kB/s | 32 kB 00:00 2024-01-15T17:55:01.451 INFO:teuthology.orchestra.run.smithi137.stdout:(44/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-01-15T17:55:01.476 INFO:teuthology.orchestra.run.smithi137.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 154 kB/s | 17 kB 00:00 2024-01-15T17:55:01.510 INFO:teuthology.orchestra.run.smithi137.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 312 kB/s | 10 kB 00:00 2024-01-15T17:55:01.526 INFO:teuthology.orchestra.run.smithi137.stdout:(47/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-01-15T17:55:01.560 INFO:teuthology.orchestra.run.smithi137.stdout:(48/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-01-15T17:55:01.602 INFO:teuthology.orchestra.run.smithi137.stdout:(49/151): librdkafka-1.6.1-102.el9.x86_64.rpm 8.8 MB/s | 661 kB 00:00 2024-01-15T17:55:01.644 INFO:teuthology.orchestra.run.smithi137.stdout:(50/151): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2024-01-15T17:55:01.669 INFO:teuthology.orchestra.run.smithi137.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 398 kB/s | 24 kB 00:00 2024-01-15T17:55:01.694 INFO:teuthology.orchestra.run.smithi137.stdout:(52/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 942 kB/s | 46 kB 00:00 2024-01-15T17:55:01.728 INFO:teuthology.orchestra.run.smithi137.stdout:(53/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-01-15T17:55:01.770 INFO:teuthology.orchestra.run.smithi137.stdout:(54/151): lua-5.4.4-4.el9.x86_64.rpm 2.5 MB/s | 187 kB 00:00 2024-01-15T17:55:01.796 INFO:teuthology.orchestra.run.smithi137.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 529 kB/s | 34 kB 00:00 2024-01-15T17:55:01.929 INFO:teuthology.orchestra.run.smithi137.stdout:(56/151): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-01-15T17:55:02.013 INFO:teuthology.orchestra.run.smithi137.stdout:(57/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 15 MB/s | 3.0 MB 00:00 2024-01-15T17:55:02.016 INFO:teuthology.orchestra.run.smithi002.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-15T17:55:02.018 INFO:teuthology.orchestra.run.smithi002.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-15T17:55:02.039 INFO:teuthology.orchestra.run.smithi002.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-15T17:55:02.041 INFO:teuthology.orchestra.run.smithi002.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-15T17:55:02.042 INFO:teuthology.orchestra.run.smithi002.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-15T17:55:02.043 INFO:teuthology.orchestra.run.smithi002.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-15T17:55:02.046 INFO:teuthology.orchestra.run.smithi002.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-15T17:55:02.050 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-15T17:55:02.051 INFO:teuthology.orchestra.run.smithi002.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-15T17:55:02.071 INFO:teuthology.orchestra.run.smithi137.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-01-15T17:55:02.113 INFO:teuthology.orchestra.run.smithi137.stdout:(59/151): postgresql-contrib-13.11-1.el9.x86_64 9.1 MB/s | 811 kB 00:00 2024-01-15T17:55:02.138 INFO:teuthology.orchestra.run.smithi137.stdout:(60/151): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-01-15T17:55:02.141 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================ 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: Package Arch Version Repository Size 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================ 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: ceph x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 6.6 k 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 5.2 M 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 860 k 2024-01-15T17:55:02.168 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 141 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 1.5 M 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 133 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 1.8 M 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 7.4 M 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 50 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 8.6 M 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 41 M 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 262 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 774 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 217 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 18 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-15T17:55:02.169 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 31 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 718 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 127 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 162 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 320 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 298 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 99 k 2024-01-15T17:55:02.170 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 85 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 3.0 M 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 170 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrading: 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-15T17:55:02.171 INFO:teuthology.orchestra.run.smithi002.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: librados2 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 3.3 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 3.0 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 19 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 28 k 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 2.1 M 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 246 k 2024-01-15T17:55:02.172 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 4.4 M 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 18 M 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g95c90f41.el9 ceph-noarch 15 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 25 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 163 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-15T17:55:02.173 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 476 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2 x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 4.8 M 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-15T17:55:02.174 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 45 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g95c90f41.el9 ceph 129 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-15T17:55:02.175 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-15T17:55:02.176 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-15T17:55:02.177 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-15T17:55:02.178 INFO:teuthology.orchestra.run.smithi002.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout:Installing weak dependencies: 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout:============================================================================================ 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout:Install 135 Packages 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout:Upgrade 16 Packages 2024-01-15T17:55:02.179 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:55:02.180 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 204 M 2024-01-15T17:55:02.180 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-01-15T17:55:02.289 INFO:teuthology.orchestra.run.smithi137.stdout:(61/151): postgresql-server-13.11-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-01-15T17:55:02.414 INFO:teuthology.orchestra.run.smithi137.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-01-15T17:55:02.439 INFO:teuthology.orchestra.run.smithi137.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 1.6 MB/s | 228 kB 00:00 2024-01-15T17:55:02.472 INFO:teuthology.orchestra.run.smithi137.stdout:(64/151): python3-jsonpatch-1.21-16.el9.noarch. 761 kB/s | 24 kB 00:00 2024-01-15T17:55:02.498 INFO:teuthology.orchestra.run.smithi137.stdout:(65/151): python3-jsonpointer-2.0-4.el9.noarch. 316 kB/s | 18 kB 00:00 2024-01-15T17:55:02.523 INFO:teuthology.orchestra.run.smithi137.stdout:(66/151): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-01-15T17:55:02.540 INFO:teuthology.orchestra.run.smithi137.stdout:(67/151): python3-markupsafe-1.1.1-12.el9.x86_6 792 kB/s | 32 kB 00:00 2024-01-15T17:55:02.707 INFO:teuthology.orchestra.run.smithi137.stdout:(68/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-01-15T17:55:02.732 INFO:teuthology.orchestra.run.smithi137.stdout:(69/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2024-01-15T17:55:02.757 INFO:teuthology.orchestra.run.smithi137.stdout:(70/151): python3-prettytable-0.7.2-27.el9.noar 1.7 MB/s | 41 kB 00:00 2024-01-15T17:55:02.783 INFO:teuthology.orchestra.run.smithi137.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-01-15T17:55:02.808 INFO:teuthology.orchestra.run.smithi137.stdout:(72/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 132 kB 00:00 2024-01-15T17:55:02.842 INFO:teuthology.orchestra.run.smithi137.stdout:(73/151): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-01-15T17:55:02.867 INFO:teuthology.orchestra.run.smithi137.stdout:(74/151): python3-pytz-2021.1-5.el9.noarch.rpm 844 kB/s | 47 kB 00:00 2024-01-15T17:55:02.893 INFO:teuthology.orchestra.run.smithi137.stdout:(75/151): python3-requests-oauthlib-1.3.0-12.el 872 kB/s | 43 kB 00:00 2024-01-15T17:55:03.018 INFO:teuthology.orchestra.run.smithi137.stdout:(76/151): python3-toml-0.10.2-6.el9.noarch.rpm 347 kB/s | 39 kB 00:00 2024-01-15T17:55:03.227 INFO:teuthology.orchestra.run.smithi137.stdout:(77/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-01-15T17:55:03.249 INFO:teuthology.orchestra.run.smithi002.stdout:(1/151): ceph-18.0.0-7629.g95c90f41.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-01-15T17:55:03.260 INFO:teuthology.orchestra.run.smithi137.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-01-15T17:55:03.285 INFO:teuthology.orchestra.run.smithi137.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-01-15T17:55:03.310 INFO:teuthology.orchestra.run.smithi137.stdout:(80/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-01-15T17:55:03.336 INFO:teuthology.orchestra.run.smithi137.stdout:(81/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 63 kB 00:00 2024-01-15T17:55:03.361 INFO:teuthology.orchestra.run.smithi137.stdout:(82/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 894 kB/s | 44 kB 00:00 2024-01-15T17:55:03.378 INFO:teuthology.orchestra.run.smithi137.stdout:(83/151): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-01-15T17:55:03.412 INFO:teuthology.orchestra.run.smithi137.stdout:(84/151): jq-1.6-16.el9.x86_64.rpm 3.7 MB/s | 186 kB 00:00 2024-01-15T17:55:03.462 INFO:teuthology.orchestra.run.smithi137.stdout:(85/151): libgfortran-11.4.1-3.el9.x86_64.rpm 9.6 MB/s | 802 kB 00:00 2024-01-15T17:55:03.487 INFO:teuthology.orchestra.run.smithi137.stdout:(86/151): libquadmath-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 194 kB 00:00 2024-01-15T17:55:03.521 INFO:teuthology.orchestra.run.smithi137.stdout:(87/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.8 MB/s | 217 kB 00:00 2024-01-15T17:55:03.555 INFO:teuthology.orchestra.run.smithi137.stdout:(88/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 241 kB 00:00 2024-01-15T17:55:03.605 INFO:teuthology.orchestra.run.smithi137.stdout:(89/151): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-01-15T17:55:03.630 INFO:teuthology.orchestra.run.smithi137.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-01-15T17:55:03.655 INFO:teuthology.orchestra.run.smithi137.stdout:(91/151): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-01-15T17:55:03.658 INFO:teuthology.orchestra.run.smithi002.stdout:(2/151): ceph-fuse-18.0.0-7629.g95c90f41.el9.x8 2.1 MB/s | 860 kB 00:00 2024-01-15T17:55:03.681 INFO:teuthology.orchestra.run.smithi137.stdout:(92/151): python3-requests-2.25.1-7.el9.noarch. 2.3 MB/s | 113 kB 00:00 2024-01-15T17:55:03.714 INFO:teuthology.orchestra.run.smithi137.stdout:(93/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-01-15T17:55:03.733 INFO:teuthology.orchestra.run.smithi002.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 1.9 MB/s | 141 kB 00:00 2024-01-15T17:55:04.267 INFO:teuthology.orchestra.run.smithi002.stdout:(4/151): ceph-mds-18.0.0-7629.g95c90f41.el9.x86 3.9 MB/s | 2.1 MB 00:00 2024-01-15T17:55:04.290 INFO:teuthology.orchestra.run.smithi137.stdout:(94/151): ceph-test-18.0.0-7629.g95c90f41.el9.x 7.1 MB/s | 41 MB 00:05 2024-01-15T17:55:04.364 INFO:teuthology.orchestra.run.smithi137.stdout:(95/151): lua-devel-5.4.4-4.el9.x86_64.rpm 34 kB/s | 22 kB 00:00 2024-01-15T17:55:04.390 INFO:teuthology.orchestra.run.smithi137.stdout:(96/151): python3-devel-3.9.17-2.el9.x86_64.rpm 347 kB/s | 245 kB 00:00 2024-01-15T17:55:04.575 INFO:teuthology.orchestra.run.smithi002.stdout:(5/151): ceph-common-18.0.0-7629.g95c90f41.el9. 11 MB/s | 19 MB 00:01 2024-01-15T17:55:04.615 INFO:teuthology.orchestra.run.smithi137.stdout:(97/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 116 kB/s | 26 kB 00:00 2024-01-15T17:55:04.766 INFO:teuthology.orchestra.run.smithi137.stdout:(98/151): liboath-2.6.7-2.el9.x86_64.rpm 325 kB/s | 49 kB 00:00 2024-01-15T17:55:04.792 INFO:teuthology.orchestra.run.smithi002.stdout:(6/151): ceph-mgr-18.0.0-7629.g95c90f41.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-01-15T17:55:04.799 INFO:teuthology.orchestra.run.smithi137.stdout:(99/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 608 kB/s | 309 kB 00:00 2024-01-15T17:55:04.850 INFO:teuthology.orchestra.run.smithi137.stdout:(100/151): libunwind-1.6.2-1.el9.x86_64.rpm 811 kB/s | 67 kB 00:00 2024-01-15T17:55:04.910 INFO:teuthology.orchestra.run.smithi002.stdout:(7/151): ceph-mon-18.0.0-7629.g95c90f41.el9.x86 13 MB/s | 4.4 MB 00:00 2024-01-15T17:55:04.975 INFO:teuthology.orchestra.run.smithi137.stdout:(101/151): luarocks-3.9.2-1.el9.noarch.rpm 864 kB/s | 151 kB 00:00 2024-01-15T17:55:05.159 INFO:teuthology.orchestra.run.smithi137.stdout:(102/151): libarrow-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-01-15T17:55:05.201 INFO:teuthology.orchestra.run.smithi137.stdout:(103/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-01-15T17:55:05.226 INFO:teuthology.orchestra.run.smithi137.stdout:(104/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 658 kB/s | 43 kB 00:00 2024-01-15T17:55:05.251 INFO:teuthology.orchestra.run.smithi137.stdout:(105/151): python3-cachetools-4.2.4-1.el9.noarc 651 kB/s | 32 kB 00:00 2024-01-15T17:55:05.302 INFO:teuthology.orchestra.run.smithi137.stdout:(106/151): python3-asyncssh-2.13.2-3.el9.noarch 1.6 MB/s | 548 kB 00:00 2024-01-15T17:55:05.327 INFO:teuthology.orchestra.run.smithi137.stdout:(107/151): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-01-15T17:55:05.360 INFO:teuthology.orchestra.run.smithi137.stdout:(108/151): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-01-15T17:55:05.394 INFO:teuthology.orchestra.run.smithi137.stdout:(109/151): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-01-15T17:55:05.401 INFO:teuthology.orchestra.run.smithi002.stdout:(8/151): ceph-radosgw-18.0.0-7629.g95c90f41.el9 18 MB/s | 8.6 MB 00:00 2024-01-15T17:55:05.411 INFO:teuthology.orchestra.run.smithi137.stdout:(110/151): python3-jaraco-8.2.1-3.el9.noarch.rp 216 kB/s | 11 kB 00:00 2024-01-15T17:55:05.443 INFO:teuthology.orchestra.run.smithi002.stdout:(9/151): ceph-selinux-18.0.0-7629.g95c90f41.el9 620 kB/s | 25 kB 00:00 2024-01-15T17:55:05.444 INFO:teuthology.orchestra.run.smithi137.stdout:(111/151): python3-google-auth-2.26.0-1.el9.noa 1.7 MB/s | 202 kB 00:00 2024-01-15T17:55:05.461 INFO:teuthology.orchestra.run.smithi137.stdout:(112/151): python3-jaraco-classes-3.2.1-5.el9.n 266 kB/s | 18 kB 00:00 2024-01-15T17:55:05.478 INFO:teuthology.orchestra.run.smithi137.stdout:(113/151): python3-jaraco-collections-3.0.0-8.e 349 kB/s | 23 kB 00:00 2024-01-15T17:55:05.503 INFO:teuthology.orchestra.run.smithi137.stdout:(114/151): python3-jaraco-functools-3.5.0-2.el9 336 kB/s | 19 kB 00:00 2024-01-15T17:55:05.528 INFO:teuthology.orchestra.run.smithi137.stdout:(115/151): python3-jwt+crypto-2.4.0-1.el9.noarc 184 kB/s | 9.0 kB 00:00 2024-01-15T17:55:05.554 INFO:teuthology.orchestra.run.smithi137.stdout:(116/151): python3-jaraco-text-3.2.0-6.el9.noar 264 kB/s | 20 kB 00:00 2024-01-15T17:55:05.579 INFO:teuthology.orchestra.run.smithi137.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 547 kB/s | 41 kB 00:00 2024-01-15T17:55:05.621 INFO:teuthology.orchestra.run.smithi137.stdout:(118/151): python3-logutils-0.3.5-21.el9.noarch 698 kB/s | 46 kB 00:00 2024-01-15T17:55:05.638 INFO:teuthology.orchestra.run.smithi137.stdout:(119/151): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-01-15T17:55:05.680 INFO:teuthology.orchestra.run.smithi137.stdout:(120/151): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-01-15T17:55:05.705 INFO:teuthology.orchestra.run.smithi137.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 695 kB/s | 58 kB 00:00 2024-01-15T17:55:05.739 INFO:teuthology.orchestra.run.smithi137.stdout:(122/151): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-01-15T17:55:05.764 INFO:teuthology.orchestra.run.smithi137.stdout:(123/151): python3-portend-3.1.0-2.el9.noarch.r 198 kB/s | 16 kB 00:00 2024-01-15T17:55:05.789 INFO:teuthology.orchestra.run.smithi137.stdout:(124/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-01-15T17:55:05.815 INFO:teuthology.orchestra.run.smithi137.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-01-15T17:55:05.848 INFO:teuthology.orchestra.run.smithi137.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-01-15T17:55:05.873 INFO:teuthology.orchestra.run.smithi137.stdout:(127/151): python3-rsa-4.9-2.el9.noarch.rpm 710 kB/s | 59 kB 00:00 2024-01-15T17:55:05.899 INFO:teuthology.orchestra.run.smithi137.stdout:(128/151): python3-tempora-5.0.0-2.el9.noarch.r 430 kB/s | 36 kB 00:00 2024-01-15T17:55:05.924 INFO:teuthology.orchestra.run.smithi137.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 682 kB/s | 51 kB 00:00 2024-01-15T17:55:05.949 INFO:teuthology.orchestra.run.smithi137.stdout:(130/151): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-01-15T17:55:05.991 INFO:teuthology.orchestra.run.smithi137.stdout:(131/151): python3-werkzeug-2.0.3-3.el9.1.noarc 6.4 MB/s | 427 kB 00:00 2024-01-15T17:55:06.025 INFO:teuthology.orchestra.run.smithi137.stdout:(132/151): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-01-15T17:55:06.051 INFO:teuthology.orchestra.run.smithi137.stdout:(133/151): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-01-15T17:55:06.075 INFO:teuthology.orchestra.run.smithi137.stdout:(134/151): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-01-15T17:55:06.159 INFO:teuthology.orchestra.run.smithi137.stdout:(135/151): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-01-15T17:55:06.193 INFO:teuthology.orchestra.run.smithi137.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-01-15T17:55:06.287 INFO:teuthology.orchestra.run.smithi137.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 25 MB/s | 2.2 MB 00:00 2024-01-15T17:55:06.360 INFO:teuthology.orchestra.run.smithi137.stdout:(138/151): NetworkManager-libnm-1.45.9-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-01-15T17:55:06.394 INFO:teuthology.orchestra.run.smithi137.stdout:(139/151): NetworkManager-team-1.45.9-1.el9.x86 1.3 MB/s | 42 kB 00:00 2024-01-15T17:55:06.436 INFO:teuthology.orchestra.run.smithi137.stdout:(140/151): NetworkManager-tui-1.45.9-1.el9.x86_ 5.9 MB/s | 246 kB 00:00 2024-01-15T17:55:06.469 INFO:teuthology.orchestra.run.smithi137.stdout:(141/151): acl-2.3.1-4.el9.x86_64.rpm 2.1 MB/s | 69 kB 00:00 2024-01-15T17:55:06.503 INFO:teuthology.orchestra.run.smithi137.stdout:(142/151): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-01-15T17:55:06.536 INFO:teuthology.orchestra.run.smithi137.stdout:(143/151): elfutils-libelf-0.190-2.el9.x86_64.r 5.8 MB/s | 191 kB 00:00 2024-01-15T17:55:06.578 INFO:teuthology.orchestra.run.smithi137.stdout:(144/151): elfutils-libs-0.190-2.el9.x86_64.rpm 6.1 MB/s | 254 kB 00:00 2024-01-15T17:55:06.612 INFO:teuthology.orchestra.run.smithi137.stdout:(145/151): libacl-2.3.1-4.el9.x86_64.rpm 676 kB/s | 22 kB 00:00 2024-01-15T17:55:06.671 INFO:teuthology.orchestra.run.smithi137.stdout:(146/151): openssl-3.0.7-25.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-01-15T17:55:06.754 INFO:teuthology.orchestra.run.smithi137.stdout:(147/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 26 MB/s | 2.1 MB 00:00 2024-01-15T17:55:06.796 INFO:teuthology.orchestra.run.smithi137.stdout:(148/151): xfsdump-3.1.12-4.el9.x86_64.rpm 7.8 MB/s | 321 kB 00:00 2024-01-15T17:55:06.855 INFO:teuthology.orchestra.run.smithi137.stdout:(149/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-01-15T17:55:06.938 INFO:teuthology.orchestra.run.smithi137.stdout:(150/151): librbd1-18.0.0-7629.g95c90f41.el9.x8 3.5 MB/s | 3.0 MB 00:00 2024-01-15T17:55:07.013 INFO:teuthology.orchestra.run.smithi137.stdout:(151/151): librados2-18.0.0-7629.g95c90f41.el9. 3.4 MB/s | 3.3 MB 00:00 2024-01-15T17:55:07.021 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-01-15T17:55:07.022 INFO:teuthology.orchestra.run.smithi137.stdout:Total 18 MB/s | 204 MB 00:11 2024-01-15T17:55:07.107 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-01-15T17:55:07.446 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-01-15T17:55:07.446 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-01-15T17:55:07.826 INFO:teuthology.orchestra.run.smithi002.stdout:(10/151): ceph-test-18.0.0-7629.g95c90f41.el9.x 17 MB/s | 41 MB 00:02 2024-01-15T17:55:07.901 INFO:teuthology.orchestra.run.smithi002.stdout:(11/151): cephfs-mirror-18.0.0-7629.g95c90f41.e 2.9 MB/s | 217 kB 00:00 2024-01-15T17:55:07.952 INFO:teuthology.orchestra.run.smithi002.stdout:(12/151): libcephfs-devel-18.0.0-7629.g95c90f41 637 kB/s | 31 kB 00:00 2024-01-15T17:55:08.036 INFO:teuthology.orchestra.run.smithi002.stdout:(13/151): libcephfs2-18.0.0-7629.g95c90f41.el9. 8.5 MB/s | 718 kB 00:00 2024-01-15T17:55:08.086 INFO:teuthology.orchestra.run.smithi002.stdout:(14/151): libcephsqlite-18.0.0-7629.g95c90f41.e 3.3 MB/s | 163 kB 00:00 2024-01-15T17:55:08.136 INFO:teuthology.orchestra.run.smithi002.stdout:(15/151): librados-devel-18.0.0-7629.g95c90f41. 2.5 MB/s | 127 kB 00:00 2024-01-15T17:55:08.203 INFO:teuthology.orchestra.run.smithi002.stdout:(16/151): libradosstriper1-18.0.0-7629.g95c90f4 7.1 MB/s | 476 kB 00:00 2024-01-15T17:55:08.537 INFO:teuthology.orchestra.run.smithi002.stdout:(17/151): librgw2-18.0.0-7629.g95c90f41.el9.x86 14 MB/s | 4.8 MB 00:00 2024-01-15T17:55:08.612 INFO:teuthology.orchestra.run.smithi002.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g95 611 kB/s | 45 kB 00:00 2024-01-15T17:55:08.662 INFO:teuthology.orchestra.run.smithi002.stdout:(19/151): python3-ceph-common-18.0.0-7629.g95c9 2.6 MB/s | 129 kB 00:00 2024-01-15T17:55:08.712 INFO:teuthology.orchestra.run.smithi002.stdout:(20/151): python3-cephfs-18.0.0-7629.g95c90f41. 3.2 MB/s | 162 kB 00:00 2024-01-15T17:55:08.771 INFO:teuthology.orchestra.run.smithi002.stdout:(21/151): python3-rados-18.0.0-7629.g95c90f41.e 5.4 MB/s | 320 kB 00:00 2024-01-15T17:55:08.830 INFO:teuthology.orchestra.run.smithi002.stdout:(22/151): python3-rbd-18.0.0-7629.g95c90f41.el9 5.1 MB/s | 298 kB 00:00 2024-01-15T17:55:08.880 INFO:teuthology.orchestra.run.smithi002.stdout:(23/151): python3-rgw-18.0.0-7629.g95c90f41.el9 2.0 MB/s | 99 kB 00:00 2024-01-15T17:55:08.930 INFO:teuthology.orchestra.run.smithi002.stdout:(24/151): rbd-fuse-18.0.0-7629.g95c90f41.el9.x8 1.7 MB/s | 85 kB 00:00 2024-01-15T17:55:09.156 INFO:teuthology.orchestra.run.smithi002.stdout:(25/151): rbd-mirror-18.0.0-7629.g95c90f41.el9. 13 MB/s | 3.0 MB 00:00 2024-01-15T17:55:09.206 INFO:teuthology.orchestra.run.smithi002.stdout:(26/151): rbd-nbd-18.0.0-7629.g95c90f41.el9.x86 3.4 MB/s | 170 kB 00:00 2024-01-15T17:55:09.256 INFO:teuthology.orchestra.run.smithi002.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 571 kB/s | 28 kB 00:00 2024-01-15T17:55:09.306 INFO:teuthology.orchestra.run.smithi002.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g95c90f4 2.6 MB/s | 133 kB 00:00 2024-01-15T17:55:09.576 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-01-15T17:55:09.576 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-01-15T17:55:09.607 INFO:teuthology.orchestra.run.smithi002.stdout:(29/151): ceph-osd-18.0.0-7629.g95c90f41.el9.x8 3.7 MB/s | 18 MB 00:04 2024-01-15T17:55:09.674 INFO:teuthology.orchestra.run.smithi002.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g95c90 4.8 MB/s | 1.8 MB 00:00 2024-01-15T17:55:09.724 INFO:teuthology.orchestra.run.smithi002.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g95 4.9 MB/s | 246 kB 00:00 2024-01-15T17:55:09.774 INFO:teuthology.orchestra.run.smithi002.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g95c90f41.e 1.0 MB/s | 50 kB 00:00 2024-01-15T17:55:09.824 INFO:teuthology.orchestra.run.smithi002.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g9 308 kB/s | 15 kB 00:00 2024-01-15T17:55:09.883 INFO:teuthology.orchestra.run.smithi002.stdout:(34/151): ceph-volume-18.0.0-7629.g95c90f41.el9 4.5 MB/s | 262 kB 00:00 2024-01-15T17:55:09.991 INFO:teuthology.orchestra.run.smithi002.stdout:(35/151): ceph-base-18.0.0-7629.g95c90f41.el9.x 753 kB/s | 5.2 MB 00:07 2024-01-15T17:55:10.050 INFO:teuthology.orchestra.run.smithi002.stdout:(36/151): cephadm-18.0.0-7629.g95c90f41.el9.noa 4.6 MB/s | 774 kB 00:00 2024-01-15T17:55:10.092 INFO:teuthology.orchestra.run.smithi002.stdout:(37/151): cephfs-top-18.0.0-7629.g95c90f41.el9. 183 kB/s | 18 kB 00:00 2024-01-15T17:55:10.301 INFO:teuthology.orchestra.run.smithi002.stdout:(38/151): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 662 kB 00:00 2024-01-15T17:55:10.343 INFO:teuthology.orchestra.run.smithi002.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 3.6 MB/s | 920 kB 00:00 2024-01-15T17:55:10.376 INFO:teuthology.orchestra.run.smithi002.stdout:(40/151): elfutils-libelf-devel-0.190-2.el9.x86 627 kB/s | 20 kB 00:00 2024-01-15T17:55:10.401 INFO:teuthology.orchestra.run.smithi002.stdout:(41/151): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-01-15T17:55:10.435 INFO:teuthology.orchestra.run.smithi002.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-01-15T17:55:10.452 INFO:teuthology.orchestra.run.smithi002.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 673 kB/s | 32 kB 00:00 2024-01-15T17:55:10.485 INFO:teuthology.orchestra.run.smithi002.stdout:(44/151): flexiblas-openblas-openmp-3.0.4-8.el9 568 kB/s | 17 kB 00:00 2024-01-15T17:55:10.594 INFO:teuthology.orchestra.run.smithi002.stdout:(45/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-01-15T17:55:10.611 INFO:teuthology.orchestra.run.smithi002.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 90 kB/s | 10 kB 00:00 2024-01-15T17:55:10.645 INFO:teuthology.orchestra.run.smithi002.stdout:(47/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-01-15T17:55:10.670 INFO:teuthology.orchestra.run.smithi002.stdout:(48/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 793 kB/s | 44 kB 00:00 2024-01-15T17:55:10.712 INFO:teuthology.orchestra.run.smithi002.stdout:(49/151): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-01-15T17:55:10.762 INFO:teuthology.orchestra.run.smithi002.stdout:(50/151): libtool-2.4.6-45.el9.x86_64.rpm 6.7 MB/s | 578 kB 00:00 2024-01-15T17:55:10.788 INFO:teuthology.orchestra.run.smithi002.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 352 kB/s | 24 kB 00:00 2024-01-15T17:55:10.813 INFO:teuthology.orchestra.run.smithi002.stdout:(52/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 938 kB/s | 46 kB 00:00 2024-01-15T17:55:10.847 INFO:teuthology.orchestra.run.smithi002.stdout:(53/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 282 kB 00:00 2024-01-15T17:55:10.880 INFO:teuthology.orchestra.run.smithi002.stdout:(54/151): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-01-15T17:55:10.905 INFO:teuthology.orchestra.run.smithi002.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 608 kB/s | 34 kB 00:00 2024-01-15T17:55:11.048 INFO:teuthology.orchestra.run.smithi002.stdout:(56/151): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-01-15T17:55:11.140 INFO:teuthology.orchestra.run.smithi002.stdout:(57/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-01-15T17:55:11.199 INFO:teuthology.orchestra.run.smithi002.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-01-15T17:55:11.249 INFO:teuthology.orchestra.run.smithi002.stdout:(59/151): postgresql-contrib-13.11-1.el9.x86_64 8.4 MB/s | 811 kB 00:00 2024-01-15T17:55:11.274 INFO:teuthology.orchestra.run.smithi002.stdout:(60/151): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-01-15T17:55:11.433 INFO:teuthology.orchestra.run.smithi002.stdout:(61/151): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-01-15T17:55:11.566 INFO:teuthology.orchestra.run.smithi002.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-01-15T17:55:11.591 INFO:teuthology.orchestra.run.smithi002.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 1.6 MB/s | 228 kB 00:00 2024-01-15T17:55:11.625 INFO:teuthology.orchestra.run.smithi002.stdout:(64/151): python3-jsonpatch-1.21-16.el9.noarch. 755 kB/s | 24 kB 00:00 2024-01-15T17:55:11.650 INFO:teuthology.orchestra.run.smithi002.stdout:(65/151): python3-jsonpointer-2.0-4.el9.noarch. 315 kB/s | 18 kB 00:00 2024-01-15T17:55:11.675 INFO:teuthology.orchestra.run.smithi002.stdout:(66/151): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-01-15T17:55:11.701 INFO:teuthology.orchestra.run.smithi002.stdout:(67/151): python3-markupsafe-1.1.1-12.el9.x86_6 657 kB/s | 32 kB 00:00 2024-01-15T17:55:11.835 INFO:teuthology.orchestra.run.smithi002.stdout:(68/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-01-15T17:55:11.860 INFO:teuthology.orchestra.run.smithi002.stdout:(69/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-01-15T17:55:11.901 INFO:teuthology.orchestra.run.smithi002.stdout:(70/151): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-01-15T17:55:11.927 INFO:teuthology.orchestra.run.smithi002.stdout:(71/151): python3-prettytable-0.7.2-27.el9.noar 630 kB/s | 41 kB 00:00 2024-01-15T17:55:11.969 INFO:teuthology.orchestra.run.smithi002.stdout:(72/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2024-01-15T17:55:11.994 INFO:teuthology.orchestra.run.smithi002.stdout:(73/151): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-01-15T17:55:12.028 INFO:teuthology.orchestra.run.smithi002.stdout:(74/151): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-15T17:55:12.052 INFO:teuthology.orchestra.run.smithi002.stdout:(75/151): python3-requests-oauthlib-1.3.0-12.el 749 kB/s | 43 kB 00:00 2024-01-15T17:55:12.086 INFO:teuthology.orchestra.run.smithi002.stdout:(76/151): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 39 kB 00:00 2024-01-15T17:55:12.353 INFO:teuthology.orchestra.run.smithi002.stdout:(77/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 52 MB/s | 16 MB 00:00 2024-01-15T17:55:12.378 INFO:teuthology.orchestra.run.smithi002.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-01-15T17:55:12.412 INFO:teuthology.orchestra.run.smithi002.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 1.7 MB/s | 56 kB 00:00 2024-01-15T17:55:12.462 INFO:teuthology.orchestra.run.smithi002.stdout:(80/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 87 kB 00:00 2024-01-15T17:55:12.604 INFO:teuthology.orchestra.run.smithi002.stdout:(81/151): ceph-mgr-diskprediction-local-18.0.0- 2.5 MB/s | 7.4 MB 00:02 2024-01-15T17:55:12.629 INFO:teuthology.orchestra.run.smithi002.stdout:(82/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 266 kB/s | 44 kB 00:00 2024-01-15T17:55:12.654 INFO:teuthology.orchestra.run.smithi002.stdout:(83/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 327 kB/s | 63 kB 00:00 2024-01-15T17:55:12.680 INFO:teuthology.orchestra.run.smithi002.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-01-15T17:55:12.713 INFO:teuthology.orchestra.run.smithi002.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 186 kB 00:00 2024-01-15T17:55:12.755 INFO:teuthology.orchestra.run.smithi002.stdout:(86/151): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 194 kB 00:00 2024-01-15T17:55:12.806 INFO:teuthology.orchestra.run.smithi002.stdout:(87/151): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 802 kB 00:00 2024-01-15T17:55:12.831 INFO:teuthology.orchestra.run.smithi002.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.8 MB/s | 217 kB 00:00 2024-01-15T17:55:12.873 INFO:teuthology.orchestra.run.smithi002.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 241 kB 00:00 2024-01-15T17:55:12.898 INFO:teuthology.orchestra.run.smithi002.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-01-15T17:55:12.957 INFO:teuthology.orchestra.run.smithi002.stdout:(91/151): python3-cryptography-36.0.1-4.el9.x86 9.4 MB/s | 1.1 MB 00:00 2024-01-15T17:55:12.982 INFO:teuthology.orchestra.run.smithi002.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-01-15T17:55:13.007 INFO:teuthology.orchestra.run.smithi002.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 1.1 MB/s | 113 kB 00:00 2024-01-15T17:55:13.041 INFO:teuthology.orchestra.run.smithi002.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-01-15T17:55:13.250 INFO:teuthology.orchestra.run.smithi002.stdout:(95/151): python3-devel-3.9.17-2.el9.x86_64.rpm 927 kB/s | 245 kB 00:00 2024-01-15T17:55:13.517 INFO:teuthology.orchestra.run.smithi002.stdout:(96/151): lua-devel-5.4.4-4.el9.x86_64.rpm 44 kB/s | 22 kB 00:00 2024-01-15T17:55:13.575 INFO:teuthology.orchestra.run.smithi002.stdout:(97/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 579 kB/s | 309 kB 00:00 2024-01-15T17:55:13.684 INFO:teuthology.orchestra.run.smithi002.stdout:(98/151): liboath-2.6.7-2.el9.x86_64.rpm 451 kB/s | 49 kB 00:00 2024-01-15T17:55:13.734 INFO:teuthology.orchestra.run.smithi002.stdout:(99/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 120 kB/s | 26 kB 00:00 2024-01-15T17:55:13.801 INFO:teuthology.orchestra.run.smithi002.stdout:(100/151): libunwind-1.6.2-1.el9.x86_64.rpm 579 kB/s | 67 kB 00:00 2024-01-15T17:55:13.968 INFO:teuthology.orchestra.run.smithi002.stdout:(101/151): luarocks-3.9.2-1.el9.noarch.rpm 649 kB/s | 151 kB 00:00 2024-01-15T17:55:14.118 INFO:teuthology.orchestra.run.smithi002.stdout:(102/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-01-15T17:55:14.243 INFO:teuthology.orchestra.run.smithi002.stdout:(103/151): python3-asyncssh-2.13.2-3.el9.noarch 1.9 MB/s | 548 kB 00:00 2024-01-15T17:55:14.269 INFO:teuthology.orchestra.run.smithi002.stdout:(104/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 291 kB/s | 43 kB 00:00 2024-01-15T17:55:14.386 INFO:teuthology.orchestra.run.smithi002.stdout:(105/151): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-01-15T17:55:14.411 INFO:teuthology.orchestra.run.smithi002.stdout:(106/151): python3-cachetools-4.2.4-1.el9.noarc 193 kB/s | 32 kB 00:00 2024-01-15T17:55:14.445 INFO:teuthology.orchestra.run.smithi002.stdout:(107/151): python3-certifi-2021.10.8-2.el9.noar 86 kB/s | 15 kB 00:00 2024-01-15T17:55:14.570 INFO:teuthology.orchestra.run.smithi002.stdout:(108/151): python3-cheroot-8.6.0-4.el9.noarch.r 938 kB/s | 172 kB 00:00 2024-01-15T17:55:14.612 INFO:teuthology.orchestra.run.smithi002.stdout:(109/151): python3-google-auth-2.26.0-1.el9.noa 1.2 MB/s | 202 kB 00:00 2024-01-15T17:55:14.645 INFO:teuthology.orchestra.run.smithi002.stdout:(110/151): python3-cherrypy-18.6.1-2.el9.noarch 1.5 MB/s | 358 kB 00:00 2024-01-15T17:55:14.704 INFO:teuthology.orchestra.run.smithi002.stdout:(111/151): python3-jaraco-8.2.1-3.el9.noarch.rp 80 kB/s | 11 kB 00:00 2024-01-15T17:55:14.779 INFO:teuthology.orchestra.run.smithi002.stdout:(112/151): python3-jaraco-classes-3.2.1-5.el9.n 106 kB/s | 18 kB 00:00 2024-01-15T17:55:14.829 INFO:teuthology.orchestra.run.smithi002.stdout:(113/151): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2024-01-15T17:55:14.863 INFO:teuthology.orchestra.run.smithi002.stdout:(114/151): python3-jaraco-collections-3.0.0-8.e 107 kB/s | 23 kB 00:00 2024-01-15T17:55:14.938 INFO:teuthology.orchestra.run.smithi002.stdout:(115/151): python3-jaraco-text-3.2.0-6.el9.noar 124 kB/s | 20 kB 00:00 2024-01-15T17:55:14.964 INFO:teuthology.orchestra.run.smithi002.stdout:(116/151): python3-jwt+crypto-2.4.0-1.el9.noarc 68 kB/s | 9.0 kB 00:00 2024-01-15T17:55:15.047 INFO:teuthology.orchestra.run.smithi002.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 223 kB/s | 41 kB 00:00 2024-01-15T17:55:15.139 INFO:teuthology.orchestra.run.smithi002.stdout:(118/151): python3-logutils-0.3.5-21.el9.noarch 265 kB/s | 46 kB 00:00 2024-01-15T17:55:15.215 INFO:teuthology.orchestra.run.smithi002.stdout:(119/151): python3-more-itertools-8.12.0-2.el9. 473 kB/s | 79 kB 00:00 2024-01-15T17:55:15.348 INFO:teuthology.orchestra.run.smithi002.stdout:(120/151): python3-kubernetes-26.1.0-2.el9.noar 2.5 MB/s | 1.0 MB 00:00 2024-01-15T17:55:15.374 INFO:teuthology.orchestra.run.smithi002.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 247 kB/s | 58 kB 00:00 2024-01-15T17:55:15.449 INFO:teuthology.orchestra.run.smithi002.stdout:(122/151): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2024-01-15T17:55:15.499 INFO:teuthology.orchestra.run.smithi002.stdout:(123/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 722 kB/s | 90 kB 00:00 2024-01-15T17:55:15.524 INFO:teuthology.orchestra.run.smithi002.stdout:(124/151): python3-portend-3.1.0-2.el9.noarch.r 94 kB/s | 16 kB 00:00 2024-01-15T17:55:15.583 INFO:teuthology.orchestra.run.smithi002.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 230 kB/s | 31 kB 00:00 2024-01-15T17:55:15.616 INFO:teuthology.orchestra.run.smithi002.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-01-15T17:55:15.658 INFO:teuthology.orchestra.run.smithi002.stdout:(127/151): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-01-15T17:55:15.734 INFO:teuthology.orchestra.run.smithi002.stdout:(128/151): python3-tempora-5.0.0-2.el9.noarch.r 238 kB/s | 36 kB 00:00 2024-01-15T17:55:15.766 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-01-15T17:55:15.775 INFO:teuthology.orchestra.run.smithi002.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 321 kB/s | 51 kB 00:00 2024-01-15T17:55:15.842 INFO:teuthology.orchestra.run.smithi002.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-01-15T17:55:15.868 INFO:teuthology.orchestra.run.smithi002.stdout:(131/151): python3-websocket-client-1.2.3-2.el9 673 kB/s | 90 kB 00:00 2024-01-15T17:55:15.910 INFO:teuthology.orchestra.run.smithi002.stdout:(132/151): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-01-15T17:55:15.934 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-15T17:55:15.971 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-15T17:55:16.055 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-15T17:55:16.060 INFO:teuthology.orchestra.run.smithi002.stdout:(133/151): python3-zc-lockfile-2.0-10.el9.noarc 92 kB/s | 20 kB 00:00 2024-01-15T17:55:16.109 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-15T17:55:16.146 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-15T17:55:16.219 INFO:teuthology.orchestra.run.smithi002.stdout:(134/151): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-01-15T17:55:16.244 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-15T17:55:16.310 INFO:teuthology.orchestra.run.smithi002.stdout:(135/151): re2-20211101-3.el9.x86_64.rpm 434 kB/s | 192 kB 00:00 2024-01-15T17:55:16.344 INFO:teuthology.orchestra.run.smithi002.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-01-15T17:55:16.436 INFO:teuthology.orchestra.run.smithi002.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 25 MB/s | 2.2 MB 00:00 2024-01-15T17:55:16.511 INFO:teuthology.orchestra.run.smithi002.stdout:(138/151): NetworkManager-libnm-1.45.9-1.el9.x8 24 MB/s | 1.8 MB 00:00 2024-01-15T17:55:16.545 INFO:teuthology.orchestra.run.smithi002.stdout:(139/151): NetworkManager-team-1.45.9-1.el9.x86 1.3 MB/s | 42 kB 00:00 2024-01-15T17:55:16.587 INFO:teuthology.orchestra.run.smithi002.stdout:(140/151): NetworkManager-tui-1.45.9-1.el9.x86_ 5.9 MB/s | 246 kB 00:00 2024-01-15T17:55:16.620 INFO:teuthology.orchestra.run.smithi002.stdout:(141/151): acl-2.3.1-4.el9.x86_64.rpm 2.1 MB/s | 69 kB 00:00 2024-01-15T17:55:16.655 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-15T17:55:16.662 INFO:teuthology.orchestra.run.smithi002.stdout:(142/151): elfutils-debuginfod-client-0.190-2.e 864 kB/s | 35 kB 00:00 2024-01-15T17:55:16.704 INFO:teuthology.orchestra.run.smithi002.stdout:(143/151): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-01-15T17:55:16.746 INFO:teuthology.orchestra.run.smithi002.stdout:(144/151): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-01-15T17:55:16.780 INFO:teuthology.orchestra.run.smithi002.stdout:(145/151): libacl-2.3.1-4.el9.x86_64.rpm 677 kB/s | 22 kB 00:00 2024-01-15T17:55:16.847 INFO:teuthology.orchestra.run.smithi002.stdout:(146/151): openssl-3.0.7-25.el9.x86_64.rpm 18 MB/s | 1.2 MB 00:00 2024-01-15T17:55:16.870 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 8/167 2024-01-15T17:55:16.933 INFO:teuthology.orchestra.run.smithi002.stdout:(147/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 25 MB/s | 2.1 MB 00:00 2024-01-15T17:55:16.975 INFO:teuthology.orchestra.run.smithi002.stdout:(148/151): xfsdump-3.1.12-4.el9.x86_64.rpm 7.7 MB/s | 321 kB 00:00 2024-01-15T17:55:17.033 INFO:teuthology.orchestra.run.smithi002.stdout:(149/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-01-15T17:55:17.043 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 8/167 2024-01-15T17:55:17.070 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 9/167 2024-01-15T17:55:17.192 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 9/167 2024-01-15T17:55:17.233 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:18.0.0-7629.g95c90f41.el9.x86_64 10/167 2024-01-15T17:55:17.284 INFO:teuthology.orchestra.run.smithi002.stdout:(150/151): librados2-18.0.0-7629.g95c90f41.el9. 2.7 MB/s | 3.3 MB 00:01 2024-01-15T17:55:17.375 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-15T17:55:17.480 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-15T17:55:17.797 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-15T17:55:17.854 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-15T17:55:17.904 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 15/167 2024-01-15T17:55:18.049 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 15/167 2024-01-15T17:55:18.071 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86 16/167 2024-01-15T17:55:18.164 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86 16/167 2024-01-15T17:55:18.497 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-15T17:55:18.558 INFO:teuthology.orchestra.run.smithi002.stdout:(151/151): librbd1-18.0.0-7629.g95c90f41.el9.x8 1.3 MB/s | 3.0 MB 00:02 2024-01-15T17:55:18.568 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-01-15T17:55:18.569 INFO:teuthology.orchestra.run.smithi002.stdout:Total 12 MB/s | 204 MB 00:16 2024-01-15T17:55:18.651 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-15T17:55:18.657 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-01-15T17:55:18.748 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-15T17:55:18.938 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-15T17:55:19.012 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-15T17:55:19.018 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-01-15T17:55:19.018 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-01-15T17:55:19.672 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-15T17:55:19.751 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-15T17:55:19.865 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-15T17:55:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-15T17:55:19.991 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-15T17:55:20.052 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-15T17:55:20.109 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-15T17:55:20.210 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-15T17:55:20.263 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-15T17:55:20.310 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-15T17:55:20.351 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-15T17:55:20.465 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-15T17:55:20.504 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g95c90f41.el9. 31/167 2024-01-15T17:55:20.556 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 32/167 2024-01-15T17:55:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:18.0.0-7629.g95c90f41.el9.x86_6 33/167 2024-01-15T17:55:20.698 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-15T17:55:20.796 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-15T17:55:20.994 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-15T17:55:21.064 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-15T17:55:21.219 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-01-15T17:55:21.219 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-01-15T17:55:21.720 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-15T17:55:21.820 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-15T17:55:21.892 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-15T17:55:21.950 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-15T17:55:22.005 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:18.0.0-7629.g95c90f41.el9.x86_6 42/167 2024-01-15T17:55:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-15T17:55:22.266 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-15T17:55:22.420 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-15T17:55:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-15T17:55:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-15T17:55:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-15T17:55:22.823 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-15T17:55:22.861 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-15T17:55:22.924 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-15T17:55:22.996 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-15T17:55:23.059 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-15T17:55:23.119 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-15T17:55:23.306 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-15T17:55:23.350 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-15T17:55:23.396 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-15T17:55:23.524 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-15T17:55:23.558 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-15T17:55:23.604 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-15T17:55:23.640 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-15T17:55:23.787 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-15T17:55:23.836 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-15T17:55:23.912 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-15T17:55:23.988 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-15T17:55:24.269 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-15T17:55:24.332 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-15T17:55:24.498 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-15T17:55:24.551 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-15T17:55:24.577 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-15T17:55:24.617 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-15T17:55:24.669 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-15T17:55:24.712 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-15T17:55:24.766 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-15T17:55:24.835 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-15T17:55:24.888 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-15T17:55:24.945 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-15T17:55:25.010 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-15T17:55:25.052 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-15T17:55:25.181 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-15T17:55:25.222 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-15T17:55:25.266 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-15T17:55:25.332 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-15T17:55:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-15T17:55:25.523 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-15T17:55:25.565 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-15T17:55:25.694 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-15T17:55:25.763 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-15T17:55:26.659 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-15T17:55:26.729 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-15T17:55:26.771 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-15T17:55:26.815 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-15T17:55:27.071 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-15T17:55:27.101 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-15T17:55:27.192 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-15T17:55:27.622 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-01-15T17:55:27.784 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-15T17:55:27.825 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-15T17:55:27.914 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-15T17:55:27.971 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-15T17:55:27.993 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-15T17:55:28.008 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-15T17:55:28.113 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-15T17:55:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-15T17:55:28.541 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-15T17:55:28.589 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 8/167 2024-01-15T17:55:28.748 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 8/167 2024-01-15T17:55:28.774 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 9/167 2024-01-15T17:55:28.897 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 9/167 2024-01-15T17:55:28.938 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rados-2:18.0.0-7629.g95c90f41.el9.x86_64 10/167 2024-01-15T17:55:29.095 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-15T17:55:29.202 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-15T17:55:29.502 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-15T17:55:29.555 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-15T17:55:29.580 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 15/167 2024-01-15T17:55:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-15T17:55:29.715 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-15T17:55:29.743 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 15/167 2024-01-15T17:55:29.751 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-15T17:55:29.773 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86 16/167 2024-01-15T17:55:29.787 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-15T17:55:29.865 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86 16/167 2024-01-15T17:55:30.147 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-15T17:55:30.185 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-15T17:55:30.195 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 103/167 2024-01-15T17:55:30.306 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 103/167 2024-01-15T17:55:30.334 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:18.0.0-7629.g95c90f41.el9.x86_64 104/167 2024-01-15T17:55:30.344 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-15T17:55:30.389 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-15T17:55:30.433 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-15T17:55:30.644 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-15T17:55:30.715 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-15T17:55:30.882 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-15T17:55:30.915 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 107/167 2024-01-15T17:55:31.005 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 107/167 2024-01-15T17:55:31.033 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:18.0.0-7629.g95c90f41.el9.x86_64 108/167 2024-01-15T17:55:31.162 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-15T17:55:31.228 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-15T17:55:31.341 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-15T17:55:31.415 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-15T17:55:31.449 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-15T17:55:31.511 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-15T17:55:31.571 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-15T17:55:31.690 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-15T17:55:31.735 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-15T17:55:31.781 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-15T17:55:31.820 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-15T17:55:31.941 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-15T17:55:31.983 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g95c90f41.el9. 31/167 2024-01-15T17:55:32.037 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 32/167 2024-01-15T17:55:32.072 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cephfs-2:18.0.0-7629.g95c90f41.el9.x86_6 33/167 2024-01-15T17:55:32.196 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-15T17:55:32.314 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-15T17:55:32.491 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-15T17:55:32.552 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-15T17:55:33.280 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-15T17:55:33.377 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-15T17:55:33.428 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-15T17:55:33.475 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-15T17:55:33.497 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 109/167 2024-01-15T17:55:33.523 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librados-devel-2:18.0.0-7629.g95c90f41.el9.x86_6 42/167 2024-01-15T17:55:33.538 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 109/167 2024-01-15T17:55:33.619 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-15T17:55:33.841 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-15T17:55:33.934 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-15T17:55:34.112 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-15T17:55:34.154 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 109/167 2024-01-15T17:55:34.154 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-15T17:55:34.208 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 110/167 2024-01-15T17:55:34.230 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-15T17:55:34.331 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-15T17:55:34.371 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-15T17:55:34.375 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 110/167 2024-01-15T17:55:34.375 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-15T17:55:34.375 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-15T17:55:34.375 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:34.411 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-15T17:55:34.443 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 111/167 2024-01-15T17:55:34.481 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-15T17:55:34.564 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-15T17:55:34.623 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-15T17:55:34.809 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-15T17:55:34.852 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-15T17:55:34.900 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-15T17:55:35.031 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-15T17:55:35.068 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-15T17:55:35.115 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-15T17:55:35.142 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-15T17:55:35.287 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-15T17:55:35.339 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-15T17:55:35.408 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-15T17:55:35.489 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-15T17:55:35.754 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-15T17:55:35.818 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-15T17:55:35.992 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-15T17:55:36.045 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-15T17:55:36.070 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-15T17:55:36.109 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-15T17:55:36.154 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-15T17:55:36.195 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-15T17:55:36.236 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-15T17:55:36.276 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-15T17:55:36.321 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-15T17:55:36.378 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-15T17:55:36.476 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-15T17:55:36.529 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-15T17:55:36.673 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-15T17:55:36.712 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-15T17:55:36.747 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-15T17:55:36.803 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-15T17:55:36.842 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-15T17:55:36.988 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-15T17:55:37.040 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-15T17:55:37.187 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-15T17:55:37.244 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-15T17:55:38.217 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-15T17:55:38.261 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-15T17:55:38.303 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-15T17:55:38.336 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-15T17:55:38.603 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-15T17:55:38.632 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-15T17:55:38.726 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-15T17:55:39.563 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-15T17:55:39.697 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-15T17:55:41.298 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-15T17:55:41.395 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-15T17:55:41.423 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-15T17:55:41.446 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-15T17:55:41.818 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-15T17:55:41.849 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 103/167 2024-01-15T17:55:41.970 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 103/167 2024-01-15T17:55:42.240 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rbd-2:18.0.0-7629.g95c90f41.el9.x86_64 104/167 2024-01-15T17:55:42.563 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-15T17:55:43.111 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-15T17:55:43.144 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 107/167 2024-01-15T17:55:43.261 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 107/167 2024-01-15T17:55:43.297 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : python3-rgw-2:18.0.0-7629.g95c90f41.el9.x86_64 108/167 2024-01-15T17:55:45.845 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 109/167 2024-01-15T17:55:45.883 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 109/167 2024-01-15T17:55:46.537 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 109/167 2024-01-15T17:55:46.596 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 110/167 2024-01-15T17:55:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 110/167 2024-01-15T17:55:46.760 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-15T17:55:46.760 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-15T17:55:46.760 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:55:46.804 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 111/167 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 111/167 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-01-15T17:55:46.894 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:47.091 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 112/167 2024-01-15T17:55:47.135 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 112/167 2024-01-15T17:55:47.135 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:47.135 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-15T17:55:47.135 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:48.759 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 113/167 2024-01-15T17:55:48.838 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 113/167 2024-01-15T17:55:48.838 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:48.838 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-15T17:55:48.838 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-15T17:55:48.838 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-15T17:55:48.839 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:49.060 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 114/167 2024-01-15T17:55:49.141 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 114/167 2024-01-15T17:55:49.141 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:49.141 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-15T17:55:49.141 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-15T17:55:49.141 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-15T17:55:49.141 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:49.619 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 115/167 2024-01-15T17:55:49.703 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 115/167 2024-01-15T17:55:49.703 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:49.703 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-15T17:55:49.703 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-15T17:55:49.703 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-15T17:55:49.703 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:49.821 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-15T17:55:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 117/167 2024-01-15T17:55:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 117/167 2024-01-15T17:55:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 118/167 2024-01-15T17:55:50.553 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 119/167 2024-01-15T17:55:50.584 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noa 120/167 2024-01-15T17:55:50.903 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noa 120/167 2024-01-15T17:55:50.935 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.n 121/167 2024-01-15T17:55:52.049 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.n 121/167 2024-01-15T17:55:52.077 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 122/167 2024-01-15T17:55:52.217 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 122/167 2024-01-15T17:55:52.384 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 123/167 2024-01-15T17:55:52.412 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 124/167 2024-01-15T17:55:52.480 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 124/167 2024-01-15T17:55:52.480 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:52.480 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-15T17:55:52.480 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-15T17:55:52.480 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-15T17:55:52.480 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:52.538 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 125/167 2024-01-15T17:55:52.573 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 125/167 2024-01-15T17:55:52.656 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:18.0.0-7629.g95c90f41.el9.x86_64 126/167 2024-01-15T17:55:53.509 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-15T17:55:53.540 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 128/167 2024-01-15T17:55:53.606 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 128/167 2024-01-15T17:55:53.606 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:53.606 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-15T17:55:53.606 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-15T17:55:53.606 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-15T17:55:53.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:53.648 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 129/167 2024-01-15T17:55:53.726 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g95c90 129/167 2024-01-15T17:55:53.726 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:53.726 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-15T17:55:53.726 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:53.791 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 130/167 2024-01-15T17:55:53.875 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 130/167 2024-01-15T17:55:53.875 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:53.876 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-15T17:55:53.876 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-15T17:55:53.876 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-15T17:55:53.876 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:54.250 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 131/167 2024-01-15T17:55:54.323 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 131/167 2024-01-15T17:55:54.323 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:55:54.323 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-15T17:55:54.324 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-15T17:55:54.324 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-15T17:55:54.324 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:55:58.456 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:18.0.0-7629.g95c90f41.el9.x86_64 132/167 2024-01-15T17:55:58.512 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 133/167 2024-01-15T17:55:58.568 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:18.0.0-7629.g95c90f41.el9.x86_64 134/167 2024-01-15T17:55:58.605 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-15T17:55:58.879 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-15T17:55:59.003 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-15T17:55:59.042 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 138/167 2024-01-15T17:55:59.075 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g95c90f41.el9.x86_ 139/167 2024-01-15T17:55:59.106 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-15T17:55:59.142 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-15T17:55:59.237 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-15T17:55:59.296 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-15T17:55:59.335 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 111/167 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /sys 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /proc 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /mnt 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /var/tmp 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /home 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /root 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout:skipping the directory /tmp 2024-01-15T17:55:59.772 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:00.019 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 112/167 2024-01-15T17:56:00.070 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 112/167 2024-01-15T17:56:00.070 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:00.070 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-15T17:56:00.070 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:00.239 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-15T17:56:00.283 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-15T17:56:00.367 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-15T17:56:00.428 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-15T17:56:01.740 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephfs-top-2:18.0.0-7629.g95c90f41.el9.noarch 147/167 2024-01-15T17:56:01.847 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-15T17:56:01.855 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 113/167 2024-01-15T17:56:01.897 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-15T17:56:01.939 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 113/167 2024-01-15T17:56:01.939 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:01.939 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-15T17:56:01.939 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-15T17:56:01.940 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-15T17:56:01.940 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-15T17:56:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-15T17:56:02.100 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-15T17:56:02.133 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-15T17:56:02.179 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-15T17:56:02.179 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-15T17:56:02.188 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 114/167 2024-01-15T17:56:02.286 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 114/167 2024-01-15T17:56:02.286 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:02.286 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-15T17:56:02.286 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-15T17:56:02.286 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-15T17:56:02.287 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:02.308 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-15T17:56:02.359 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-15T17:56:02.406 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-15T17:56:02.447 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-15T17:56:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-15T17:56:02.480 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-15T17:56:02.593 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-15T17:56:02.623 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-15T17:56:02.660 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-15T17:56:02.661 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-15T17:56:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 115/167 2024-01-15T17:56:02.904 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 115/167 2024-01-15T17:56:02.905 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:02.905 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-15T17:56:02.905 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-15T17:56:02.905 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-15T17:56:02.905 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:02.989 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-15T17:56:02.989 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-15T17:56:03.024 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-15T17:56:03.131 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-15T17:56:03.167 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-15T17:56:03.206 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-15T17:56:03.252 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-15T17:56:03.371 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-15T17:56:03.626 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-15T17:56:03.627 INFO:teuthology.orchestra.run.smithi137.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-15T17:56:03.627 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:03.660 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 117/167 2024-01-15T17:56:03.699 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 117/167 2024-01-15T17:56:03.745 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 118/167 2024-01-15T17:56:03.843 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 119/167 2024-01-15T17:56:03.899 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noa 120/167 2024-01-15T17:56:04.234 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noa 120/167 2024-01-15T17:56:04.262 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.n 121/167 2024-01-15T17:56:05.433 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.n 121/167 2024-01-15T17:56:05.463 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 122/167 2024-01-15T17:56:05.611 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 122/167 2024-01-15T17:56:05.795 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 123/167 2024-01-15T17:56:05.840 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 124/167 2024-01-15T17:56:05.940 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 124/167 2024-01-15T17:56:05.940 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:05.940 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-15T17:56:05.940 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-15T17:56:05.940 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-15T17:56:05.940 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:06.007 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 125/167 2024-01-15T17:56:06.049 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 125/167 2024-01-15T17:56:06.128 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-2:18.0.0-7629.g95c90f41.el9.x86_64 126/167 2024-01-15T17:56:07.054 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-15T17:56:07.084 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 128/167 2024-01-15T17:56:07.181 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 128/167 2024-01-15T17:56:07.181 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:07.181 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-15T17:56:07.182 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-15T17:56:07.182 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-15T17:56:07.182 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:07.235 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 129/167 2024-01-15T17:56:07.306 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g95c90 129/167 2024-01-15T17:56:07.306 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:07.306 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-15T17:56:07.306 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:07.377 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 130/167 2024-01-15T17:56:07.459 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 130/167 2024-01-15T17:56:07.459 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:07.460 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-15T17:56:07.460 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-15T17:56:07.460 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-15T17:56:07.460 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:07.852 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 131/167 2024-01-15T17:56:07.926 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 131/167 2024-01-15T17:56:07.926 INFO:teuthology.orchestra.run.smithi002.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-15T17:56:07.926 INFO:teuthology.orchestra.run.smithi002.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-15T17:56:07.926 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-15T17:56:07.926 INFO:teuthology.orchestra.run.smithi002.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-15T17:56:07.927 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:08.298 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:18.0.0-7629.g95c90f41.el9.x86_64 1/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 2/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 3/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 4/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 5/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 6/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 7/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 8/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 9/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 10/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 11/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:18.0.0-7629.g95c90f41.el9.x86_64 12/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 13/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g95c90f41.el9.x86_ 14/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 15/167 2024-01-15T17:56:08.299 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 16/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:18.0.0-7629.g95c90f41.el9.x86_6 17/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86 18/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 19/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 20/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g95c90f41.el9. 21/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g95c90f41.el9.x86_6 22/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:18.0.0-7629.g95c90f41.el9.x86_64 23/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:18.0.0-7629.g95c90f41.el9.x86_64 24/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:18.0.0-7629.g95c90f41.el9.x86_64 25/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 26/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 27/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g95c90f41.el9.x86_64 28/167 2024-01-15T17:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 29/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noa 30/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.n 31/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 32/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 33/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 34/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 35/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 36/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 37/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephfs-top-2:18.0.0-7629.g95c90f41.el9.noarch 38/167 2024-01-15T17:56:08.312 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-15T17:56:08.313 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-15T17:56:08.314 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-15T17:56:08.315 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-15T17:56:08.316 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-15T17:56:08.317 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-15T17:56:08.318 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 136/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 138/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-15T17:56:08.319 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-15T17:56:08.320 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-15T17:56:08.321 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-15T17:56:10.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-15T17:56:10.481 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:10.481 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-01-15T17:56:10.481 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:10.481 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: acl-2.3.1-4.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-15T17:56:10.482 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: automake-1.16.2-8.el9.noarch 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: bison-3.7.4-5.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.483 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: cephfs-top-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: flex-2.6.4-9.el9.x86_64 2024-01-15T17:56:10.484 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: jq-1.6-16.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-15T17:56:10.485 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-15T17:56:10.486 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-13.11-1.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-15T17:56:10.487 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-15T17:56:10.488 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.489 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-15T17:56:10.490 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-3.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:10.491 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-01-15T17:56:11.086 DEBUG:teuthology.parallel:result is None 2024-01-15T17:56:12.301 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-test-2:18.0.0-7629.g95c90f41.el9.x86_64 132/167 2024-01-15T17:56:12.358 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 133/167 2024-01-15T17:56:12.415 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : rbd-nbd-2:18.0.0-7629.g95c90f41.el9.x86_64 134/167 2024-01-15T17:56:12.449 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-15T17:56:12.777 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-15T17:56:12.908 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-15T17:56:12.937 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : ceph-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 138/167 2024-01-15T17:56:12.971 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g95c90f41.el9.x86_ 139/167 2024-01-15T17:56:13.009 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-15T17:56:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-15T17:56:13.186 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-15T17:56:13.243 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-15T17:56:13.279 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-15T17:56:14.138 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-15T17:56:14.185 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-15T17:56:14.280 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-15T17:56:14.364 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-15T17:56:15.821 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : cephfs-top-2:18.0.0-7629.g95c90f41.el9.noarch 147/167 2024-01-15T17:56:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-15T17:56:15.970 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-15T17:56:16.113 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-15T17:56:16.152 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-15T17:56:16.186 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-15T17:56:16.220 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-15T17:56:16.262 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-15T17:56:16.263 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-15T17:56:16.386 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-15T17:56:16.468 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-15T17:56:16.525 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-15T17:56:16.566 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-15T17:56:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-15T17:56:16.599 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-15T17:56:16.729 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-15T17:56:16.767 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-15T17:56:16.805 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-15T17:56:16.805 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-15T17:56:17.133 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-15T17:56:17.134 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-15T17:56:17.268 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-15T17:56:17.312 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-15T17:56:17.352 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-15T17:56:17.421 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-15T17:56:17.545 INFO:teuthology.orchestra.run.smithi002.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-15T17:56:17.805 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-15T17:56:17.806 INFO:teuthology.orchestra.run.smithi002.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-15T17:56:17.806 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:22.196 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-2:18.0.0-7629.g95c90f41.el9.x86_64 1/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 2/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 3/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 4/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g95c90 5/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 6/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 7/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 8/167 2024-01-15T17:56:22.197 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 9/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 10/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 11/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-test-2:18.0.0-7629.g95c90f41.el9.x86_64 12/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 13/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g95c90f41.el9.x86_ 14/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 15/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 16/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados-devel-2:18.0.0-7629.g95c90f41.el9.x86_6 17/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86 18/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 19/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el 20/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g95c90f41.el9. 21/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g95c90f41.el9.x86_6 22/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rados-2:18.0.0-7629.g95c90f41.el9.x86_64 23/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rbd-2:18.0.0-7629.g95c90f41.el9.x86_64 24/167 2024-01-15T17:56:22.198 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rgw-2:18.0.0-7629.g95c90f41.el9.x86_64 25/167 2024-01-15T17:56:22.199 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 26/167 2024-01-15T17:56:22.199 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 27/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g95c90f41.el9.x86_64 28/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41. 29/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noa 30/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.n 31/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c 32/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el 33/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 34/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.e 35/167 2024-01-15T17:56:22.211 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 36/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 37/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : cephfs-top-2:18.0.0-7629.g95c90f41.el9.noarch 38/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-15T17:56:22.212 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-15T17:56:22.213 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-15T17:56:22.214 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-15T17:56:22.215 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-15T17:56:22.216 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-15T17:56:22.217 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 136/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 138/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-15T17:56:22.218 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-15T17:56:22.219 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-15T17:56:22.220 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-15T17:56:24.456 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout:Upgraded: 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: acl-2.3.1-4.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-15T17:56:24.457 INFO:teuthology.orchestra.run.smithi002.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: librados2-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: librbd1-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: automake-1.16.2-8.el9.noarch 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: bison-3.7.4-5.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-base-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.458 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mds-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mgr-rook-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-mon-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-osd-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-radosgw-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-selinux-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-test-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: ceph-volume-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.459 INFO:teuthology.orchestra.run.smithi002.stdout: cephadm-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: cephfs-top-2:18.0.0-7629.g95c90f41.el9.noarch 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: flex-2.6.4-9.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: jq-1.6-16.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs-devel-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.460 INFO:teuthology.orchestra.run.smithi002.stdout: libcephfs2-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libcephsqlite-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: librados-devel-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libradosstriper1-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: librgw2-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-15T17:56:24.461 INFO:teuthology.orchestra.run.smithi002.stdout: lua-5.4.4-4.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-13.11-1.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-15T17:56:24.462 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-argparse-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ceph-common-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cephfs-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-15T17:56:24.463 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-15T17:56:24.464 INFO:teuthology.orchestra.run.smithi002.stdout: python3-ply-3.11-14.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rados-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rbd-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rgw-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-15T17:56:24.465 INFO:teuthology.orchestra.run.smithi002.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-fuse-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-mirror-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: rbd-nbd-2:18.0.0-7629.g95c90f41.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: re2-1:20211101-3.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-15T17:56:24.466 INFO:teuthology.orchestra.run.smithi002.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-15T17:56:24.467 INFO:teuthology.orchestra.run.smithi002.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-15T17:56:24.467 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:24.467 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-01-15T17:56:25.193 DEBUG:teuthology.parallel:result is None 2024-01-15T17:56:25.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T17:56:25.474 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-15T17:56:25.546 INFO:teuthology.orchestra.run.smithi002.stdout:18.0.0-7629.g95c90f41.el9 2024-01-15T17:56:25.546 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g95c90f41.el9 2024-01-15T17:56:25.547 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g95c90f41 is installed. 2024-01-15T17:56:25.548 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=95c90f413c4bb65e3e56909855f2308fd9bda05b 2024-01-15T17:56:25.712 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-15T17:56:25.780 INFO:teuthology.orchestra.run.smithi137.stdout:18.0.0-7629.g95c90f41.el9 2024-01-15T17:56:25.780 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g95c90f41.el9 2024-01-15T17:56:25.780 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g95c90f41 is installed. 2024-01-15T17:56:25.782 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-15T17:56:25.782 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:25.782 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T17:56:25.861 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:25.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-15T17:56:25.964 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-15T17:56:25.965 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:25.965 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T17:56:26.041 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T17:56:26.154 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:26.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-01-15T17:56:26.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-15T17:56:26.342 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-15T17:56:26.343 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:26.343 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T17:56:26.425 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T17:56:26.545 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:26.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-15T17:56:26.644 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-15T17:56:26.781 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-15T17:56:26.781 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:26.781 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T17:56:26.884 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T17:56:26.984 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:26.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-01-15T17:56:27.086 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-15T17:56:27.180 INFO:teuthology.run_tasks:Running task ceph... 2024-01-15T17:56:27.259 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-15T17:56:27.259 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /var/log/ceph 2024-01-15T17:56:27.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-01-15T17:56:27.362 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-15T17:56:27.363 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T17:56:27.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-15T17:56:27.455 INFO:tasks.ceph:Creating extra log directories... 2024-01-15T17:56:27.456 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T17:56:27.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-15T17:56:27.560 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-15T17:56:27.560 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-15T17:56:27.560 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-15T17:56:27.560 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:27.560 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T17:56:27.603 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T17:56:27.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-15T17:56:27.898 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:27.898 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-15T17:56:27.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-15T17:56:28.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-15T17:56:28.322 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-15T17:56:28.323 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-15T17:56:28.323 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7517115', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8754, 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi002.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVN2HzCjZzUS/rDqxDK4fI5PQ6Hqfv8NUsoE7qkLEBkoNhkeIfrx8AYn6LBKQs/L1bP/Xx3vpgnfhF63n6FjdE=', 'smithi137.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbj9Ii0wnblDymIKws/pcdeuhJ0D9SbBrDDD61dzeofHosaESMxAJlvn1GsMltQv695/PYjgdKxaC35iKRBjj8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '95c90f413c4bb65e3e56909855f2308fd9bda05b', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-15_17:39:27', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-15T17:56:28.323 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T17:56:28.329 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-15T17:56:28.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T17:56:28.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m0777 -- /var/run/ceph 2024-01-15T17:56:28.485 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:28.485 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T17:56:28.528 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T17:56:28.528 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-01-15 17:56:18.268879514 +0000 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-01-15 17:53:50.501699532 +0000 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-01-15 17:53:50.501699532 +0000 2024-01-15T17:56:28.613 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-01-15 17:53:50.501699532 +0000 2024-01-15T17:56:28.614 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T17:56:28.733 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-01-15T17:56:28.733 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-01-15T17:56:28.733 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000262343 s, 2.0 MB/s 2024-01-15T17:56:28.736 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T17:56:28.780 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-01-15 17:56:18.269879492 +0000 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-01-15 17:53:51.067686716 +0000 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-01-15 17:53:51.067686716 +0000 2024-01-15T17:56:28.863 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-01-15 17:53:51.067686716 +0000 2024-01-15T17:56:28.863 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T17:56:28.982 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-01-15T17:56:28.982 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-01-15T17:56:28.982 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000199036 s, 2.6 MB/s 2024-01-15T17:56:28.986 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T17:56:29.026 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-01-15 17:56:18.269879492 +0000 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-01-15 17:53:51.650673515 +0000 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-01-15 17:53:51.650673515 +0000 2024-01-15T17:56:29.107 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-01-15 17:53:51.650673515 +0000 2024-01-15T17:56:29.107 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T17:56:29.221 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-01-15T17:56:29.221 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-01-15T17:56:29.221 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000242403 s, 2.1 MB/s 2024-01-15T17:56:29.224 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T17:56:29.268 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-01-15 17:56:18.270879469 +0000 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-01-15 17:53:52.209660858 +0000 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-01-15 17:53:52.209660858 +0000 2024-01-15T17:56:29.357 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-01-15 17:53:52.209660858 +0000 2024-01-15T17:56:29.358 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T17:56:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-01-15T17:56:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-01-15T17:56:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000186653 s, 2.7 MB/s 2024-01-15T17:56:29.469 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T17:56:29.510 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:29.510 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:29.510 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-01-15T17:56:29.567 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-15T17:56:29.567 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-01-15T17:56:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-15T17:56:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-15 17:56:04.062504384 +0000 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-15 17:53:40.728215670 +0000 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-15 17:53:40.728215670 +0000 2024-01-15T17:56:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-15 17:53:40.728215670 +0000 2024-01-15T17:56:29.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-15T17:56:29.704 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-15T17:56:29.704 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-15T17:56:29.705 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000350433 s, 1.5 MB/s 2024-01-15T17:56:29.707 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-15T17:56:29.758 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-01-15T17:56:29.808 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-15 17:56:04.062504384 +0000 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-15 17:53:41.297205412 +0000 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-15 17:53:41.297205412 +0000 2024-01-15T17:56:29.809 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-15 17:53:41.297205412 +0000 2024-01-15T17:56:29.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-15T17:56:29.891 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-15T17:56:29.891 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-15T17:56:29.891 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000325254 s, 1.6 MB/s 2024-01-15T17:56:29.895 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-15T17:56:29.950 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-01-15T17:56:29.998 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-15T17:56:29.998 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-01-15T17:56:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:29.998 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-15 17:56:04.063504367 +0000 2024-01-15T17:56:29.999 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-15 17:53:42.015192469 +0000 2024-01-15T17:56:29.999 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-15 17:53:42.015192469 +0000 2024-01-15T17:56:29.999 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-15 17:53:42.015192469 +0000 2024-01-15T17:56:29.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-15T17:56:30.083 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-15T17:56:30.084 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-15T17:56:30.084 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000285127 s, 1.8 MB/s 2024-01-15T17:56:30.087 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-15T17:56:30.142 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-01-15T17:56:30.189 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-15T17:56:30.189 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-15T17:56:30.189 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-01-15T17:56:30.189 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-15T17:56:30.189 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-01-15T17:56:30.189 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-01-15 17:56:04.064504349 +0000 2024-01-15T17:56:30.190 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-01-15 17:53:42.580182284 +0000 2024-01-15T17:56:30.190 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-01-15 17:53:42.580182284 +0000 2024-01-15T17:56:30.190 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-01-15 17:53:42.580182284 +0000 2024-01-15T17:56:30.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-15T17:56:30.318 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-01-15T17:56:30.318 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-01-15T17:56:30.318 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000323563 s, 1.6 MB/s 2024-01-15T17:56:30.322 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-15T17:56:30.375 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:30.376 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T17:56:30.376 INFO:tasks.ceph:Generating config... 2024-01-15T17:56:30.377 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-15T17:56:30.377 INFO:tasks.ceph:[client] debug client = 20 2024-01-15T17:56:30.377 INFO:tasks.ceph:[client] debug ms = 1 2024-01-15T17:56:30.377 INFO:tasks.ceph:[client] pid file = 2024-01-15T17:56:30.377 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-15T17:56:30.377 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-15T17:56:30.377 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-15T17:56:30.378 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-15T17:56:30.379 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-15T17:56:30.379 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-15T17:56:30.379 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-15T17:56:30.380 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-15T17:56:30.380 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-15T17:56:30.380 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-15T17:56:30.380 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-15T17:56:30.380 INFO:tasks.ceph:Setting up mon.a... 2024-01-15T17:56:30.380 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-15T17:56:30.536 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.keyring 2024-01-15T17:56:30.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-15T17:56:30.665 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T17:56:30.750 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.2'), ('mon.b', '172.21.15.137'), ('mon.c', '[v2:172.21.15.137:3301,v1:172.21.15.137:6790]')] 2024-01-15T17:56:30.750 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.2,172.21.15.137,[v2:172.21.15.137:3301,v1:172.21.15.137:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-15T17:56:30.751 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:30.751 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T17:56:30.801 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.2 --add b 172.21.15.137 --addv c '[v2:172.21.15.137:3301,v1:172.21.15.137:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:30.944 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:30.944 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: generated fsid c0e5773f-513d-405e-88b5-a3133821721f 2024-01-15T17:56:30.944 INFO:teuthology.orchestra.run.smithi002.stdout:setting min_mon_release = pacific 2024-01-15T17:56:30.944 INFO:teuthology.orchestra.run.smithi002.stdout:epoch 0 2024-01-15T17:56:30.944 INFO:teuthology.orchestra.run.smithi002.stdout:fsid c0e5773f-513d-405e-88b5-a3133821721f 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:last_changed 2024-01-15T17:56:30.942371+0000 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:created 2024-01-15T17:56:30.942371+0000 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:min_mon_release 16 (pacific) 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:election_strategy: 1 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:1: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.b 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:2: [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] mon.c 2024-01-15T17:56:30.945 INFO:teuthology.orchestra.run.smithi002.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-15T17:56:30.948 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-15T17:56:30.987 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c0e5773f-513d-405e-88b5-a3133821721f... 2024-01-15T17:56:30.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T17:56:31.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-15T17:56:31.198 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-01-15T17:56:31.198 INFO:teuthology.orchestra.run.smithi002.stdout: chdir = "" 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: auth supported = cephx 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: filestore xattr use omap = true 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: mon clock drift allowed = 1.000 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: osd crush chooseleaf type = 0 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: auth debug = true 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on old message = true 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: ms die on bug = true 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: mon pg warn max object skew = 0 2024-01-15T17:56:31.199 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default size = 2 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow primary affinity = true 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd allow pg remap = true 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on legacy crush tunables = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on crush straw calc version zero = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on no sortbitwise = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on osd down out interval zero = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon warn on too few osds = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T17:56:31.200 INFO:teuthology.orchestra.run.smithi002.stdout: mon_allow_pool_size_one = true 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: osd default data pool replay window = 5 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: mon allow pool delete = true 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: mon cluster log file level = debug 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: debug asserts on shutdown = true 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: mon health detail to clog = false 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: mon host = "172.21.15.2,172.21.15.137,[v2:172.21.15.137:3301,v1:172.21.15.137:6790]" 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = c0e5773f-513d-405e-88b5-a3133821721f 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout:[osd] 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: osd journal size = 100 2024-01-15T17:56:31.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub load threshold = 5.0 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd scrub max interval = 600 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd mclock profile = high_recovery_ops 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd recover clone overlap = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd recovery max chunk = 1048576 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug shutdown = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug op order = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug verify stray on activate = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug trim objects = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd open classes on start = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug pg log writeout = true 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.202 INFO:teuthology.orchestra.run.smithi002.stdout: osd deep scrub update digest min age = 30 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: osd map max advance = 10 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: journal zero on create = true 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: filestore ondisk finisher threads = 3 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: filestore apply finisher threads = 3 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: bdev debug aio = true 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: osd debug misdirected ops = true 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: bdev async discard = True 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: bdev enable discard = True 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore allocator = bitmap 2024-01-15T17:56:31.203 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore block size = 96636764160 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: bluestore fsck on mount = True 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluefs = 1/20 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: debug bluestore = 1/20 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: debug osd = 20 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: debug rocksdb = 4/10 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd full ratio = 0.9 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd nearfull ratio = 0.8 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: osd failsafe full ratio = 0.95 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: osd objectstore = bluestore 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: osd op complaint time = 180 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr] 2024-01-15T17:56:31.204 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug mgr = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: mgr/telemetry/nag = false 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug client = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug finisher = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout:[mon] 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug mon = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug paxos = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: debug auth = 20 2024-01-15T17:56:31.205 INFO:teuthology.orchestra.run.smithi002.stdout: mon data avail warn = 5 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon mgr mkfs grace = 240 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min pgs per osd = 4 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd reporter subtree level = osd 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon osd prime pg temp = true 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon reweight min bytes per osd = 10 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: # 1m isn't quite enough 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: mon_down_mkfs_grace = 2m 2024-01-15T17:56:31.206 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: mon_warn_on_filestore_osds = false 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: mon op complaint time = 120 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout:[client] 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: rgw cache enabled = true 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable ops log = true 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: rgw enable usage log = true 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: client mount timeout = 600 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: debug client = 20 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: pid file = "" 2024-01-15T17:56:31.207 INFO:teuthology.orchestra.run.smithi002.stdout: rados mon op timeout = 900 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: rados osd op timeout = 900 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.a] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.b] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mon.c] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout:[mds] 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: debug mds = 20 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: debug mds balancer = 20 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: debug ms = 1 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: mds debug frag = True 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: mds debug scatterstat = True 2024-01-15T17:56:31.208 INFO:teuthology.orchestra.run.smithi002.stdout: mds op complaint time = 180 2024-01-15T17:56:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: mds verify scatter = True 2024-01-15T17:56:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: osd op complaint time = 180 2024-01-15T17:56:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: rados mon op timeout = 900 2024-01-15T17:56:31.209 INFO:teuthology.orchestra.run.smithi002.stdout: rados osd op timeout = 900 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: chdir = "" 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: auth supported = cephx 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: filestore xattr use omap = true 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: mon clock drift allowed = 1.000 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: osd crush chooseleaf type = 0 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: auth debug = true 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on old message = true 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: ms die on bug = true 2024-01-15T17:56:31.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon pg warn max object skew = 0 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: # disable pg_autoscaler by default for new pools 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default size = 2 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow primary affinity = true 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd allow pg remap = true 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on legacy crush tunables = false 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on crush straw calc version zero = false 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on no sortbitwise = false 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on osd down out interval zero = false 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon warn on too few osds = false 2024-01-15T17:56:31.211 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: mon_allow_pool_size_one = true 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: osd default data pool replay window = 5 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: mon allow pool delete = true 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: mon cluster log file level = debug 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: debug asserts on shutdown = true 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: mon health detail to clog = false 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: mon host = "172.21.15.2,172.21.15.137,[v2:172.21.15.137:3301,v1:172.21.15.137:6790]" 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = c0e5773f-513d-405e-88b5-a3133821721f 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.212 INFO:teuthology.orchestra.run.smithi137.stdout:[osd] 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd journal size = 100 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub load threshold = 5.0 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd scrub max interval = 600 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd mclock profile = high_recovery_ops 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd recover clone overlap = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd recovery max chunk = 1048576 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug shutdown = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug op order = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug verify stray on activate = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug trim objects = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd open classes on start = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug pg log writeout = true 2024-01-15T17:56:31.213 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: osd deep scrub update digest min age = 30 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: osd map max advance = 10 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: journal zero on create = true 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: filestore ondisk finisher threads = 3 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: filestore apply finisher threads = 3 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: bdev debug aio = true 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: osd debug misdirected ops = true 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: bdev async discard = True 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: bdev enable discard = True 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore allocator = bitmap 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore block size = 96636764160 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: bluestore fsck on mount = True 2024-01-15T17:56:31.214 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluefs = 1/20 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug bluestore = 1/20 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug osd = 20 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug rocksdb = 4/10 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd backfillfull_ratio = 0.85 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd full ratio = 0.9 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd nearfull ratio = 0.8 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: osd failsafe full ratio = 0.95 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: osd objectstore = bluestore 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr] 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug mgr = 20 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-01-15T17:56:31.215 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mgr/telemetry/nag = false 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: debug finisher = 20 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout:[mon] 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: debug mon = 20 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: debug paxos = 20 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: debug auth = 20 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mon data avail warn = 5 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mon mgr mkfs grace = 240 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min pgs per osd = 4 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd reporter subtree level = osd 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mon osd prime pg temp = true 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: mon reweight min bytes per osd = 10 2024-01-15T17:56:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: auth mon ticket ttl = 660 # 11m 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: auth service ticket ttl = 240 # 4m 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: # don't complain about insecure global_id in the test suite 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: # 1m isn't quite enough 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: mon_down_mkfs_grace = 2m 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: mon_warn_on_filestore_osds = false 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: mon op complaint time = 120 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout:[client] 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: rgw cache enabled = true 2024-01-15T17:56:31.217 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable ops log = true 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: rgw enable usage log = true 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: client mount timeout = 600 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: debug client = 20 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: pid file = "" 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 900 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 900 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.a] 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.b] 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mon.c] 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.c] 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2024-01-15T17:56:31.218 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.d] 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout:[mds] 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds = 20 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: debug mds balancer = 20 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: debug ms = 1 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug frag = True 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: mds debug scatterstat = True 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: mds op complaint time = 180 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: mds verify scatter = True 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: osd op complaint time = 180 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: rados mon op timeout = 900 2024-01-15T17:56:31.219 INFO:teuthology.orchestra.run.smithi137.stdout: rados osd op timeout = 900 2024-01-15T17:56:31.251 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-15T17:56:31.252 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-15T17:56:31.402 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-15T17:56:31.402 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:31.402 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-15T17:56:31.452 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:31.452 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-15T17:56:31.549 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:56:31.550 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:31.550 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T17:56:31.550 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T17:56:31.698 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:56:31.699 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:31.797 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:56:31.797 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:31.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-15T17:56:31.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-15T17:56:31.952 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:56:31.953 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:32.050 INFO:tasks.ceph:Setting up mon nodes... 2024-01-15T17:56:32.050 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-15T17:56:32.050 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T17:56:32.245 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-15T17:56:32.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T17:56:32.443 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-15T17:56:32.454 INFO:tasks.ceph:Setting up mds nodes... 2024-01-15T17:56:32.454 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T17:56:32.631 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-15T17:56:32.639 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-15T17:56:32.744 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T17:56:32.935 INFO:teuthology.orchestra.run.smithi002.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-15T17:56:32.944 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-15T17:56:33.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T17:56:33.233 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-15T17:56:33.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-15T17:56:33.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T17:56:33.525 INFO:teuthology.orchestra.run.smithi137.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-15T17:56:33.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-15T17:56:33.626 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-15T17:56:33.627 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-15T17:56:33.762 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-15T17:56:33.820 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-15T17:56:33.956 INFO:teuthology.orchestra.run.smithi002.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-15T17:56:34.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-15T17:56:34.146 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-15T17:56:34.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-15T17:56:34.337 INFO:teuthology.orchestra.run.smithi137.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-15T17:56:34.392 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-15T17:56:34.392 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi002.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi137.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-15T17:56:34.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-15T17:56:34.499 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:34.499 INFO:tasks.ceph:role: osd.0 2024-01-15T17:56:34.499 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:56:34.499 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T17:56:34.599 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:34.599 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:34.599 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:34.600 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:34.633 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-01-15T17:56:34.639 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-01-15T17:56:34.639 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-15T17:56:34.752 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-15T17:56:34.834 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-15T17:56:34.924 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:34.925 INFO:tasks.ceph:role: osd.1 2024-01-15T17:56:34.925 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:56:34.925 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T17:56:35.023 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:35.023 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:35.023 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:35.023 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:35.024 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:35.024 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:35.024 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:35.057 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-01-15T17:56:35.064 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-01-15T17:56:35.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-15T17:56:35.179 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-15T17:56:35.266 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-15T17:56:35.356 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:35.356 INFO:tasks.ceph:role: osd.2 2024-01-15T17:56:35.356 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:56:35.356 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T17:56:35.452 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:35.452 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:35.452 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:35.452 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:35.452 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:35.453 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:35.453 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:35.453 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:35.453 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:35.453 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:35.486 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-01-15T17:56:35.492 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-01-15T17:56:35.493 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-15T17:56:35.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-15T17:56:35.691 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-15T17:56:35.776 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:35.776 INFO:tasks.ceph:role: osd.3 2024-01-15T17:56:35.777 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:56:35.777 DEBUG:teuthology.orchestra.run.smithi002:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T17:56:35.867 INFO:teuthology.orchestra.run.smithi002.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:35.868 INFO:teuthology.orchestra.run.smithi002.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:35.903 INFO:teuthology.orchestra.run.smithi002.stdout:Discarding blocks...Done. 2024-01-15T17:56:35.909 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi002.front.sepia.ceph.com -o noatime 2024-01-15T17:56:35.909 DEBUG:teuthology.orchestra.run.smithi002:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-15T17:56:36.026 DEBUG:teuthology.orchestra.run.smithi002:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-15T17:56:36.125 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:36.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:36.283+0000 7eff1fa05500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-15T17:56:36.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:36.283+0000 7eff1fa05500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-15T17:56:36.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:36.283+0000 7eff1fa05500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T17:56:36.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:36.283+0000 7eff1fa05500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T17:56:36.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:36.283+0000 7eff1fa05500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-15T17:56:36.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:36.287+0000 7eff1fa05500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-15T17:56:39.461 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-15T17:56:39.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:39.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:39.705+0000 7fc39d816500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-15T17:56:39.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:39.705+0000 7fc39d816500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-15T17:56:39.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:39.706+0000 7fc39d816500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T17:56:39.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:39.706+0000 7fc39d816500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T17:56:39.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:39.706+0000 7fc39d816500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-15T17:56:39.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:39.708+0000 7fc39d816500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-15T17:56:42.627 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-15T17:56:42.729 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:42.872+0000 7f8db3e1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-15T17:56:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:42.872+0000 7f8db3e1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-15T17:56:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:42.873+0000 7f8db3e1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T17:56:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:42.873+0000 7f8db3e1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T17:56:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:42.873+0000 7f8db3e1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-15T17:56:42.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:42.875+0000 7f8db3e1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-15T17:56:46.053 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-15T17:56:46.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:46.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:46.303+0000 7f59fbb29500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-15T17:56:46.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:46.304+0000 7f59fbb29500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-15T17:56:46.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:46.304+0000 7f59fbb29500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T17:56:46.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:46.304+0000 7f59fbb29500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T17:56:46.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:46.304+0000 7f59fbb29500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-15T17:56:46.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:56:46.306+0000 7f59fbb29500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-15T17:56:49.485 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-15T17:56:49.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-15T17:56:49.685 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:49.685 INFO:tasks.ceph:role: osd.4 2024-01-15T17:56:49.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:56:49.685 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-15T17:56:49.785 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:49.785 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:49.785 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:49.785 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:49.785 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:49.786 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:49.786 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:49.786 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:49.786 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:49.786 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:50.299 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-15T17:56:50.307 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-15T17:56:50.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-15T17:56:50.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-15T17:56:50.575 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-15T17:56:50.648 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:50.648 INFO:tasks.ceph:role: osd.5 2024-01-15T17:56:50.648 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:56:50.648 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-15T17:56:50.731 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:50.731 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:50.732 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:51.189 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-15T17:56:51.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-15T17:56:51.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-15T17:56:51.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-15T17:56:51.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-15T17:56:51.492 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:51.492 INFO:tasks.ceph:role: osd.6 2024-01-15T17:56:51.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:56:51.493 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:51.575 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:52.111 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-15T17:56:52.116 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-15T17:56:52.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-15T17:56:52.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-15T17:56:52.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-15T17:56:52.520 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-15T17:56:52.521 INFO:tasks.ceph:role: osd.7 2024-01-15T17:56:52.521 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:56:52.521 DEBUG:teuthology.orchestra.run.smithi137:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-15T17:56:52.616 INFO:teuthology.orchestra.run.smithi137.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-15T17:56:52.616 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-15T17:56:52.616 INFO:teuthology.orchestra.run.smithi137.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-15T17:56:52.616 INFO:teuthology.orchestra.run.smithi137.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-15T17:56:52.616 INFO:teuthology.orchestra.run.smithi137.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-15T17:56:52.616 INFO:teuthology.orchestra.run.smithi137.stdout: = sunit=0 swidth=0 blks 2024-01-15T17:56:52.617 INFO:teuthology.orchestra.run.smithi137.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-15T17:56:52.617 INFO:teuthology.orchestra.run.smithi137.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-15T17:56:52.617 INFO:teuthology.orchestra.run.smithi137.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-15T17:56:52.617 INFO:teuthology.orchestra.run.smithi137.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-15T17:56:53.081 INFO:teuthology.orchestra.run.smithi137.stdout:Discarding blocks...Done. 2024-01-15T17:56:53.086 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi137.front.sepia.ceph.com -o noatime 2024-01-15T17:56:53.087 DEBUG:teuthology.orchestra.run.smithi137:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-15T17:56:53.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-15T17:56:53.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:53.475 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:53.471+0000 7ff73d422500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-15T17:56:53.477 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:53.474+0000 7ff73d422500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-15T17:56:53.478 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:53.474+0000 7ff73d422500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T17:56:53.478 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:53.474+0000 7ff73d422500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T17:56:53.478 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:53.474+0000 7ff73d422500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-15T17:56:53.480 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:53.477+0000 7ff73d422500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-15T17:56:56.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-15T17:56:56.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:56:56.889 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:56.885+0000 7f0509f46500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-15T17:56:56.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:56.886+0000 7f0509f46500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-15T17:56:56.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:56.886+0000 7f0509f46500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T17:56:56.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:56.886+0000 7f0509f46500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T17:56:56.890 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:56.886+0000 7f0509f46500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-15T17:56:56.892 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:56:56.888+0000 7f0509f46500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-15T17:56:58.325 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:56:58.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:57:00.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-15T17:57:00.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:57:00.296 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:00.293+0000 7f4bcc60c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-15T17:57:00.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:00.293+0000 7f4bcc60c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-15T17:57:00.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:00.293+0000 7f4bcc60c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T17:57:00.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:00.293+0000 7f4bcc60c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T17:57:00.297 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:00.293+0000 7f4bcc60c500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-15T17:57:00.299 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:00.296+0000 7f4bcc60c500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-15T17:57:03.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-15T17:57:03.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:57:03.467 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:03.464+0000 7f46ffe16500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-15T17:57:03.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:03.465+0000 7f46ffe16500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-15T17:57:03.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:03.465+0000 7f46ffe16500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T17:57:03.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:03.465+0000 7f46ffe16500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T17:57:03.468 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:03.465+0000 7f46ffe16500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-15T17:57:03.470 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:03.468+0000 7f46ffe16500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-15T17:57:06.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-15T17:57:06.487 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-15T17:57:06.487 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:06.487 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-15T17:57:06.593 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:06.594 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-15T17:57:06.697 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:06.697 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-15T17:57:06.796 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:06.797 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-15T17:57:06.905 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:06.905 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-15T17:57:06.992 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:06.992 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-15T17:57:07.128 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:07.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-15T17:57:07.220 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-15T17:57:07.316 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-15T17:57:07.413 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-15T17:57:07.501 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-15T17:57:07.635 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-15T17:57:07.728 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.728 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-15T17:57:07.822 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:07.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-15T17:57:07.955 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:07.955 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-15T17:57:08.007 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:08.007 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-15T17:57:08.051 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:08.051 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-15T17:57:08.107 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:08.107 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-15T17:57:08.160 INFO:tasks.ceph:Adding keys to all mons... 2024-01-15T17:57:08.161 DEBUG:teuthology.orchestra.run.smithi002:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T17:57:08.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-15T17:57:08.251 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.y] 2024-01-15T17:57:08.251 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw== 2024-01-15T17:57:08.251 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.a] 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA== 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.c] 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ== 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.0] 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A== 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.1] 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA== 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.2] 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig== 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.3] 2024-01-15T17:57:08.252 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw== 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout:[mgr.x] 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw== 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.b] 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA== 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout:[mds.d] 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ== 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.4] 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q== 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.5] 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw== 2024-01-15T17:57:08.253 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.6] 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ== 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout:[osd.7] 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ== 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout:[client.2] 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw== 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout:[client.1] 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw== 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout:[client.3] 2024-01-15T17:57:08.254 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw== 2024-01-15T17:57:08.258 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.y] 2024-01-15T17:57:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw== 2024-01-15T17:57:08.258 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.a] 2024-01-15T17:57:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA== 2024-01-15T17:57:08.258 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.c] 2024-01-15T17:57:08.258 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ== 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.0] 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A== 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.1] 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA== 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.2] 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig== 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.3] 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw== 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout:[mgr.x] 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw== 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.b] 2024-01-15T17:57:08.259 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA== 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout:[mds.d] 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ== 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.4] 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q== 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.5] 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw== 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.6] 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ== 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout:[osd.7] 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ== 2024-01-15T17:57:08.260 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout:[client.2] 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw== 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout:[client.1] 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw== 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout:[client.3] 2024-01-15T17:57:08.261 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw== 2024-01-15T17:57:08.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T17:57:08.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T17:57:08.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:08.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:08.598 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:08.603 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:08.749 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:08.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:08.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:08.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.050 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.202 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.352 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T17:57:09.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-15T17:57:09.505 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:09.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:09.658 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:09.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-15T17:57:09.818 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.969 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:09.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:10.122 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:10.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:10.274 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:10.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-15T17:57:10.428 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.580 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.736 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.890 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:10.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-15T17:57:11.043 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-15T17:57:11.044 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-15T17:57:11.138 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T17:57:11.299 DEBUG:teuthology.orchestra.run.smithi002:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-15T17:57:11.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-15T17:57:11.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T17:57:11.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-15T17:57:11.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-15T17:57:11.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-15T17:57:11.961 DEBUG:teuthology.orchestra.run.smithi137:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-15T17:57:12.045 DEBUG:teuthology.orchestra.run.smithi002:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:57:12.053 DEBUG:teuthology.orchestra.run.smithi137:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-15T17:57:12.106 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-15T17:57:12.106 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-15T17:57:12.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-15T17:57:12.111 INFO:tasks.ceph.mon.a:Started 2024-01-15T17:57:12.111 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-15T17:57:12.111 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-15T17:57:12.116 INFO:tasks.ceph.mon.b:Started 2024-01-15T17:57:12.116 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-15T17:57:12.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-15T17:57:12.121 INFO:tasks.ceph.mon.c:Started 2024-01-15T17:57:12.121 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-15T17:57:12.121 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-15T17:57:12.121 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-15T17:57:12.126 INFO:tasks.ceph.mgr.y:Started 2024-01-15T17:57:12.126 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-15T17:57:12.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-15T17:57:12.131 INFO:tasks.ceph.mgr.x:Started 2024-01-15T17:57:12.131 INFO:tasks.ceph:Setting crush tunables to default 2024-01-15T17:57:12.131 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph osd crush tunables default 2024-01-15T17:57:12.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.310+0000 7efbf7e00640 1 Processor -- start 2024-01-15T17:57:12.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.311+0000 7efbf7e00640 1 -- start start 2024-01-15T17:57:12.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 0x7efbf80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf7e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80d1250 con 0x7efbf80d1870 2024-01-15T17:57:12.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf7e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80ce8e0 con 0x7efbf80d1c50 2024-01-15T17:57:12.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf7e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf80ce6d0 con 0x7efbf80b2210 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 0x7efbf80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 0x7efbf80d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53840/0 (socket says 172.21.15.2:53840) 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6000640 1 -- 172.21.15.2:0/1406134303 learned_addr learned my addr 172.21.15.2:0/1406134303 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6a00640 1 --1- 172.21.15.2:0/1406134303 >> v1:172.21.15.2:6789/0 conn(0x7efbf80d1870 0x7efbf815d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35024/0 (socket says 172.21.15.2:35024) 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6000640 1 -- 172.21.15.2:0/1406134303 >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 msgr2=0x7efbf80d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6000640 1 -- 172.21.15.2:0/1406134303 >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 msgr2=0x7efbf80d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.312+0000 7efbf6000640 1 --2- 172.21.15.2:0/1406134303 >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 0x7efbf80d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-15T17:57:12.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.313+0000 7efbf6000640 1 --2- 172.21.15.2:0/1406134303 >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 0x7efbf80d3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-15T17:57:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403108437 0 0) 0x7efbf80d1250 con 0x7efbf80d1870 2024-01-15T17:57:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbd8003520 con 0x7efbf80d1870 2024-01-15T17:57:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884031410 0 0) 0x7efbd8003520 con 0x7efbf80d1870 2024-01-15T17:57:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 msgr2=0x7efbf80d3bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T17:57:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 --2- 172.21.15.2:0/1406134303 >> v2:172.21.15.2:3300/0 conn(0x7efbf80b2210 0x7efbf80d3bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 >> v1:172.21.15.137:6789/0 conn(0x7efbf80d1c50 legacy=0x7efbf80b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80cebc0 con 0x7efbf80d1870 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7efbe8001670 con 0x7efbf80d1870 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 >> v1:172.21.15.2:6789/0 conn(0x7efbf80d1870 legacy=0x7efbf815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbd80039c0 0x7efbd8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbd80047c0 0x7efbd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80ce6d0 con 0x7efbd80051d0 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbd800c120 con 0x7efbd80039c0 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.327+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbd800c2e0 con 0x7efbd80047c0 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf6a00640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbd80039c0 0x7efbd8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf6000640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbd80047c0 0x7efbd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf7400640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf7400640 1 -- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbd80047c0 msgr2=0x7efbd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf7400640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbd80047c0 0x7efbd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf7400640 1 -- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbd80039c0 msgr2=0x7efbd8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:12.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf7400640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbd80039c0 0x7efbd8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.328+0000 7efbf7400640 1 -- 172.21.15.2:0/1406134303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbd8003520 con 0x7efbd80051d0 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.329+0000 7efbf7400640 1 -- 172.21.15.2:0/1406134303 reap_dead start 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.329+0000 7efbf7400640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbd800b980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efbf0001e50 tx=0x7efbd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133b202a508d612a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.330+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0003b70 con 0x7efbd80051d0 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.330+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf0003cf0 con 0x7efbd80051d0 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.330+0000 7efbf5600640 1 -- 172.21.15.2:0/1406134303 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0005310 con 0x7efbd80051d0 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.330+0000 7efbf7e00640 1 -- 172.21.15.2:0/1406134303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 msgr2=0x7efbd800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.330+0000 7efbf7e00640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbd800b980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7efbf0001e50 tx=0x7efbd80057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.331+0000 7efbf7e00640 1 -- 172.21.15.2:0/1406134303 shutdown_connections 2024-01-15T17:57:12.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.331+0000 7efbf7e00640 1 --2- 172.21.15.2:0/1406134303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbd800b980 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.331+0000 7efbf7e00640 1 -- 172.21.15.2:0/1406134303 >> 172.21.15.2:0/1406134303 conn(0x7efbf80d30b0 msgr2=0x7efbf80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.331+0000 7efbf7e00640 1 -- 172.21.15.2:0/1406134303 shutdown_connections 2024-01-15T17:57:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.331+0000 7efbf7e00640 1 -- 172.21.15.2:0/1406134303 wait complete. 2024-01-15T17:57:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.333+0000 7efbf7e00640 1 Processor -- start 2024-01-15T17:57:12.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.333+0000 7efbf7e00640 1 -- start start 2024-01-15T17:57:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbf815f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbf815ceb0 0x7efbf81f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbf815d830 0x7efbf815dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf815e1f0 con 0x7efbd80051d0 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf81609f0 con 0x7efbf815d830 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf815fde0 con 0x7efbf815ceb0 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbf815f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbf815f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53858/0 (socket says 172.21.15.2:53858) 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf6a00640 1 -- 172.21.15.2:0/1720107590 learned_addr learned my addr 172.21.15.2:0/1720107590 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf7400640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbf815d830 0x7efbf815dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.334+0000 7efbf6000640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbf815ceb0 0x7efbf81f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf6a00640 1 -- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbf815ceb0 msgr2=0x7efbf81f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf6a00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbf815ceb0 0x7efbf81f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf6a00640 1 -- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbf815d830 msgr2=0x7efbf815dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf6a00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbf815d830 0x7efbf815dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf6a00640 1 -- 172.21.15.2:0/1720107590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf81600c0 con 0x7efbd80051d0 2024-01-15T17:57:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf6a00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbf815f790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efbe8007740 tx=0x7efbe8004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe8004e00 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe800f940 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe800fcd0 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.335+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf81603a0 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.336+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf815d310 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.336+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efbe801d760 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.336+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7efbe80136e0 con 0x7efbd80051d0 2024-01-15T17:57:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.337+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf815d4d0 con 0x7efbd80051d0 2024-01-15T17:57:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.340+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7efbf815d4d0 con 0x7efbd80051d0 2024-01-15T17:57:12.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.412+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7efbf80b5ac0 con 0x7efbd80051d0 2024-01-15T17:57:12.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.415+0000 7efbeea00640 1 -- 172.21.15.2:0/1720107590 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7efbe801dbf0 con 0x7efbd80051d0 2024-01-15T17:57:12.419 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 msgr2=0x7efbf815f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbf815f790 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7efbe8007740 tx=0x7efbe8004bb0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 shutdown_connections 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efbf815d830 0x7efbf815dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efbf815ceb0 0x7efbf81f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 --2- 172.21.15.2:0/1720107590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efbd80051d0 0x7efbf815f790 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 >> 172.21.15.2:0/1720107590 conn(0x7efbf80d30b0 msgr2=0x7efbf80b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 shutdown_connections 2024-01-15T17:57:12.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:12.417+0000 7efbf7e00640 1 -- 172.21.15.2:0/1720107590 wait complete. 2024-01-15T17:57:12.438 INFO:tasks.ceph:check_enable_crimson: False 2024-01-15T17:57:12.438 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-15T17:57:12.439 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:12.439 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-15T17:57:12.516 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:12.516 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-15T17:57:12.642 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:12.642 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-15T17:57:12.775 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:57:12.775 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-15T17:57:12.864 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:12.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-15T17:57:12.947 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:12.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-15T17:57:13.033 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.030+0000 7fe29584a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T17:57:13.044 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.040+0000 7fe4168b0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-15T17:57:13.072 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:13.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-15T17:57:13.098 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.096+0000 7fe29584a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T17:57:13.111 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.107+0000 7fe4168b0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-15T17:57:13.180 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:57:13.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-15T17:57:13.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 23efca61-d1a7-4ae6-8b77-93e0343256c5 0 2024-01-15T17:57:13.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.537+0000 7fceeb000640 1 Processor -- start 2024-01-15T17:57:13.540 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.537+0000 7fceeb000640 1 -- start start 2024-01-15T17:57:13.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fceeb000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcee40b4c70 0x7fcee40d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fceeb000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcee40b8c30 0x7fcee40d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fceeb000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40d1900 con 0x7fcee415d730 2024-01-15T17:57:13.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fceeb000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40d1be0 con 0x7fcee40b4c70 2024-01-15T17:57:13.542 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fceeb000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee40d1ec0 con 0x7fcee40b8c30 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fcedbe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fcee415d730 0x7fcee40b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:49952/0 (socket says 172.21.15.137:49952) 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fcedbe00640 1 -- 172.21.15.137:0/2897936086 learned_addr learned my addr 172.21.15.137:0/2897936086 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fcee9000640 1 --2- 172.21.15.137:0/2897936086 >> v2:172.21.15.137:3300/0 conn(0x7fcee40b8c30 0x7fcee40d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.539+0000 7fcee3e00640 1 --2- 172.21.15.137:0/2897936086 >> v2:172.21.15.2:3300/0 conn(0x7fcee40b4c70 0x7fcee40d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.540+0000 7fcee9000640 1 -- 172.21.15.137:0/2897936086 >> v2:172.21.15.2:3300/0 conn(0x7fcee40b4c70 msgr2=0x7fcee40d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.540+0000 7fcee9000640 1 --2- 172.21.15.137:0/2897936086 >> v2:172.21.15.2:3300/0 conn(0x7fcee40b4c70 0x7fcee40d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.540+0000 7fcee9000640 1 -- 172.21.15.137:0/2897936086 >> v1:172.21.15.137:6789/0 conn(0x7fcee415d730 legacy=0x7fcee40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.540+0000 7fcee9000640 1 -- 172.21.15.137:0/2897936086 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40d21a0 con 0x7fcee40b8c30 2024-01-15T17:57:13.543 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.540+0000 7fcee3400640 1 -- 172.21.15.137:0/2897936086 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325742969 0 0) 0x7fcee40d1900 con 0x7fcee415d730 2024-01-15T17:57:13.544 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.541+0000 7fcee9000640 1 --2- 172.21.15.137:0/2897936086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40d29a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcee40d0d50 tx=0x7fcedc02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41bb1b783f25a2b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:13.544 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.541+0000 7fcee3400640 1 -- 172.21.15.137:0/2897936086 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc02cc80 con 0x7fcee40b8c30 2024-01-15T17:57:13.544 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.541+0000 7fcee3400640 1 -- 172.21.15.137:0/2897936086 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc02cde0 con 0x7fcee40b8c30 2024-01-15T17:57:13.544 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.541+0000 7fcee3400640 1 -- 172.21.15.137:0/2897936086 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc034660 con 0x7fcee40b8c30 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.541+0000 7fceeb000640 1 -- 172.21.15.137:0/2897936086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 msgr2=0x7fcee40d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.541+0000 7fceeb000640 1 --2- 172.21.15.137:0/2897936086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40d29a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcee40d0d50 tx=0x7fcedc02bed0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.542+0000 7fceeb000640 1 -- 172.21.15.137:0/2897936086 shutdown_connections 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.542+0000 7fceeb000640 1 --2- 172.21.15.137:0/2897936086 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40d29a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.542+0000 7fceeb000640 1 --2- 172.21.15.137:0/2897936086 >> v2:172.21.15.2:3300/0 conn(0x7fcee40b4c70 0x7fcee40d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.542+0000 7fceeb000640 1 -- 172.21.15.137:0/2897936086 >> 172.21.15.137:0/2897936086 conn(0x7fcee4072210 msgr2=0x7fcee40cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.542+0000 7fceeb000640 1 -- 172.21.15.137:0/2897936086 shutdown_connections 2024-01-15T17:57:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.542+0000 7fceeb000640 1 -- 172.21.15.137:0/2897936086 wait complete. 2024-01-15T17:57:13.546 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.543+0000 7fceeb000640 1 Processor -- start 2024-01-15T17:57:13.546 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.543+0000 7fceeb000640 1 -- start start 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.544+0000 7fceeb000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b4c70 0x7fcee415a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.544+0000 7fceeb000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.544+0000 7fceeb000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcee415d730 0x7fcee40b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.544+0000 7fceeb000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40c8e70 con 0x7fcee415d730 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.544+0000 7fceeb000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40c9150 con 0x7fcee40b8c30 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.544+0000 7fceeb000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee40c9430 con 0x7fcee40b4c70 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b4c70 0x7fcee415a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:34962/0 (socket says 172.21.15.137:34962) 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 -- 172.21.15.137:0/3641761953 learned_addr learned my addr 172.21.15.137:0/3641761953 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b4c70 0x7fcee415a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58286/0 (socket says 172.21.15.137:58286) 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee9000640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcee415d730 0x7fcee40b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.549 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 -- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b4c70 msgr2=0x7fcee415a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b4c70 0x7fcee415a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 -- 172.21.15.137:0/3641761953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcee415d730 msgr2=0x7fcee40b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcee415d730 0x7fcee40b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 -- 172.21.15.137:0/3641761953 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc009a70 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee9000640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcee415d730 0x7fcee40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcedbe00640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40b3ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00c410 tx=0x7fcecc00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc0074e0 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcecc007660 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcecc004a20 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.545+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee40d3060 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.546+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee40c99f0 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.546+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcecc01d3f0 con 0x7fcee40b8c30 2024-01-15T17:57:13.550 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.546+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcecc0137d0 con 0x7fcee40b8c30 2024-01-15T17:57:13.551 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.547+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee40c9c10 con 0x7fcee40b8c30 2024-01-15T17:57:13.552 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.550+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fcecc013990 con 0x7fcee40b8c30 2024-01-15T17:57:13.631 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.628+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "23efca61-d1a7-4ae6-8b77-93e0343256c5", "id": 0} v 0) v1 -- 0x7fcee40b7510 con 0x7fcee40b8c30 2024-01-15T17:57:13.638 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.635+0000 7fcee0c00640 1 -- 172.21.15.137:0/3641761953 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "23efca61-d1a7-4ae6-8b77-93e0343256c5", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcecc013c00 con 0x7fcee40b8c30 2024-01-15T17:57:13.638 INFO:teuthology.orchestra.run.smithi137.stdout:0 2024-01-15T17:57:13.639 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.636+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 msgr2=0x7fcee40b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.639 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.636+0000 7fceeb000640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40b3ca0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcecc00c410 tx=0x7fcecc00c8d0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.639 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 shutdown_connections 2024-01-15T17:57:13.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcee415d730 0x7fcee40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcee40b8c30 0x7fcee40b3ca0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 --2- 172.21.15.137:0/3641761953 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcee40b4c70 0x7fcee415a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 >> 172.21.15.137:0/3641761953 conn(0x7fcee4072210 msgr2=0x7fcee40b6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:13.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 shutdown_connections 2024-01-15T17:57:13.640 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.637+0000 7fceeb000640 1 -- 172.21.15.137:0/3641761953 wait complete. 2024-01-15T17:57:13.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 97935595-7baf-4b8d-8f4b-09d72b71b15f 1 2024-01-15T17:57:13.706 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.704+0000 7fe29584a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T17:57:13.749 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.746+0000 7fe4168b0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-15T17:57:13.771 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.768+0000 7fe29584a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T17:57:13.813 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.809+0000 7fe4168b0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-15T17:57:13.824 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.821+0000 7fe29584a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T17:57:13.840 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.837+0000 7f432de00640 1 Processor -- start 2024-01-15T17:57:13.840 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.837+0000 7f432de00640 1 -- start start 2024-01-15T17:57:13.842 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f432de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43280d1870 0x7f43280d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.842 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f432de00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280d1250 con 0x7f43280d1c50 2024-01-15T17:57:13.842 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f432de00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43280ce8e0 con 0x7f43280b2210 2024-01-15T17:57:13.842 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f432de00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43280ce6d0 con 0x7f43280d1870 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f432ca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43280d1870 0x7f43280d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f431fe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f43280b2210 0x7f432815d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:49956/0 (socket says 172.21.15.137:49956) 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.839+0000 7f431fe00640 1 -- 172.21.15.137:0/1039311686 learned_addr learned my addr 172.21.15.137:0/1039311686 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.840+0000 7f432ca00640 1 -- 172.21.15.137:0/1039311686 >> v1:172.21.15.137:6789/0 conn(0x7f43280b2210 legacy=0x7f432815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.840+0000 7f432ca00640 1 -- 172.21.15.137:0/1039311686 >> v1:172.21.15.2:6789/0 conn(0x7f43280d1c50 legacy=0x7f43280b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.840+0000 7f432ca00640 1 -- 172.21.15.137:0/1039311686 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43280cebc0 con 0x7f43280d1870 2024-01-15T17:57:13.843 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.841+0000 7f432ca00640 1 --2- 172.21.15.137:0/1039311686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1870 0x7f43280d3bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f43180097a0 tx=0x7f431802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80dd652f0ac89c50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:13.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.841+0000 7f4327e00640 1 -- 172.21.15.137:0/1039311686 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318030610 con 0x7f43280d1870 2024-01-15T17:57:13.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.841+0000 7f4327e00640 1 -- 172.21.15.137:0/1039311686 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318030770 con 0x7f43280d1870 2024-01-15T17:57:13.844 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.841+0000 7f4327e00640 1 -- 172.21.15.137:0/1039311686 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318030b00 con 0x7f43280d1870 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 -- 172.21.15.137:0/1039311686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1870 msgr2=0x7f43280d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 --2- 172.21.15.137:0/1039311686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1870 0x7f43280d3bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f43180097a0 tx=0x7f431802b960 comp rx=0 tx=0).stop 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 -- 172.21.15.137:0/1039311686 shutdown_connections 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 --2- 172.21.15.137:0/1039311686 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1870 0x7f43280d3bd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 -- 172.21.15.137:0/1039311686 >> 172.21.15.137:0/1039311686 conn(0x7f43280d30b0 msgr2=0x7f43280d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 -- 172.21.15.137:0/1039311686 shutdown_connections 2024-01-15T17:57:13.845 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.842+0000 7f432de00640 1 -- 172.21.15.137:0/1039311686 wait complete. 2024-01-15T17:57:13.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.844+0000 7f432de00640 1 Processor -- start 2024-01-15T17:57:13.846 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.844+0000 7f432de00640 1 -- start start 2024-01-15T17:57:13.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 0x7f43281f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.848 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43280d1870 0x7f43280b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1c50 0x7f432815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432de00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280b7f50 con 0x7f43280d1870 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43280c9c40 con 0x7f43280d1c50 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43281f34d0 con 0x7f43280b2210 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 0x7f43281f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432d400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1c50 0x7f432815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 0x7f43281f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58296/0 (socket says 172.21.15.137:58296) 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f432ca00640 1 -- 172.21.15.137:0/1825108901 learned_addr learned my addr 172.21.15.137:0/1825108901 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.845+0000 7f431fe00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43280d1870 0x7f43280b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432ca00640 1 -- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1c50 msgr2=0x7f432815a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432ca00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1c50 0x7f432815a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432ca00640 1 -- 172.21.15.137:0/1825108901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43280d1870 msgr2=0x7f43280b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432ca00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43280d1870 0x7f43280b8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432ca00640 1 -- 172.21.15.137:0/1825108901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4318009400 con 0x7f43280b2210 2024-01-15T17:57:13.849 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432d400640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1c50 0x7f432815a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f432ca00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 0x7f43281f2f90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f431802bf70 tx=0x7f4318004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.846+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431803e040 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.847+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4318039cb0 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.847+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4318044630 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.847+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f432814d070 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.847+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43281f7ae0 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.847+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4318044a90 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.847+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f431804dcd0 con 0x7f43280b2210 2024-01-15T17:57:13.851 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.848+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43280d0f00 con 0x7f43280b2210 2024-01-15T17:57:13.853 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.851+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f43280d0f00 con 0x7f43280b2210 2024-01-15T17:57:13.866 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.862+0000 7fe4168b0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-15T17:57:13.891 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.889+0000 7fe29584a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T17:57:13.931 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.927+0000 7fe4168b0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-15T17:57:13.931 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.929+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "97935595-7baf-4b8d-8f4b-09d72b71b15f", "id": 1} v 0) v1 -- 0x7f43280b81c0 con 0x7f43280b2210 2024-01-15T17:57:13.937 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.934+0000 7f4325600640 1 -- 172.21.15.137:0/1825108901 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "97935595-7baf-4b8d-8f4b-09d72b71b15f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4318049070 con 0x7f43280b2210 2024-01-15T17:57:13.937 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T17:57:13.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 msgr2=0x7f43281f2f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:13.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 0x7f43281f2f90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f431802bf70 tx=0x7f4318004730 comp rx=0 tx=0).stop 2024-01-15T17:57:13.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 shutdown_connections 2024-01-15T17:57:13.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43280d1c50 0x7f432815a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43280d1870 0x7f43280b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 --2- 172.21.15.137:0/1825108901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43280b2210 0x7f43281f2f90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:13.940 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 >> 172.21.15.137:0/1825108901 conn(0x7f43280d30b0 msgr2=0x7f432806c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:13.940 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 shutdown_connections 2024-01-15T17:57:13.940 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:13.936+0000 7f432de00640 1 -- 172.21.15.137:0/1825108901 wait complete. 2024-01-15T17:57:13.944 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:13.941+0000 7fe29584a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T17:57:13.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new c591f415-915a-45d6-9a06-70ca6673ddb6 2 2024-01-15T17:57:13.982 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:13.979+0000 7fe4168b0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-15T17:57:14.064 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.061+0000 7fe29584a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-15T17:57:14.101 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.098+0000 7fe4168b0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-15T17:57:14.133 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.130+0000 7fd31b400640 1 Processor -- start 2024-01-15T17:57:14.133 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.131+0000 7fd31b400640 1 -- start start 2024-01-15T17:57:14.135 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.132+0000 7fd31b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd31415c050 0x7fd31406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.135 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.132+0000 7fd31b400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31406dea0 con 0x7fd3140ced00 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.132+0000 7fd31b400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31406e160 con 0x7fd314158cb0 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.132+0000 7fd31b400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3140668c0 con 0x7fd31415c050 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd31a000640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fd314158cb0 0x7fd3140ce5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:49972/0 (socket says 172.21.15.137:49972) 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd31a000640 1 -- 172.21.15.137:0/2300273119 learned_addr learned my addr 172.21.15.137:0/2300273119 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd31aa00640 1 --2- 172.21.15.137:0/2300273119 >> v2:172.21.15.2:3300/0 conn(0x7fd31415c050 0x7fd31406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953737062 0 0) 0x7fd31406e160 con 0x7fd314158cb0 2024-01-15T17:57:14.136 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f4003520 con 0x7fd314158cb0 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3602079434 0 0) 0x7fd2f4003520 con 0x7fd314158cb0 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 >> v2:172.21.15.2:3300/0 conn(0x7fd31415c050 msgr2=0x7fd31406a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 --2- 172.21.15.137:0/2300273119 >> v2:172.21.15.2:3300/0 conn(0x7fd31415c050 0x7fd31406a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 >> v1:172.21.15.2:6789/0 conn(0x7fd3140ced00 legacy=0x7fd31406e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31415c5a0 con 0x7fd314158cb0 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd304001670 con 0x7fd314158cb0 2024-01-15T17:57:14.138 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.133+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 >> v1:172.21.15.137:6789/0 conn(0x7fd314158cb0 legacy=0x7fd3140ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd318c00640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd318c00640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd318c00640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3140668c0 con 0x7fd2f40047c0 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f400c080 con 0x7fd2f4005180 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f400c240 con 0x7fd2f40039c0 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31a000640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31aa00640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd319600640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31a000640 1 -- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2f4005180 msgr2=0x7fd2f400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31a000640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31a000640 1 -- 172.21.15.137:0/2300273119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40047c0 msgr2=0x7fd2f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31a000640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31a000640 1 -- 172.21.15.137:0/2300273119 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4003520 con 0x7fd2f40039c0 2024-01-15T17:57:14.139 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd319600640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:14.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.134+0000 7fd31aa00640 1 -- 172.21.15.137:0/2300273119 reap_dead start 2024-01-15T17:57:14.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.135+0000 7fd31a000640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd304007fc0 tx=0x7fd2f40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffa7970a28bfe0e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:14.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd304005ad0 con 0x7fd2f40039c0 2024-01-15T17:57:14.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd304005c30 con 0x7fd2f40039c0 2024-01-15T17:57:14.140 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd318c00640 1 -- 172.21.15.137:0/2300273119 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3040072b0 con 0x7fd2f40039c0 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 -- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 msgr2=0x7fd2f4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd304007fc0 tx=0x7fd2f40057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 -- 172.21.15.137:0/2300273119 shutdown_connections 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 --2- 172.21.15.137:0/2300273119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 -- 172.21.15.137:0/2300273119 >> 172.21.15.137:0/2300273119 conn(0x7fd3140cbf70 msgr2=0x7fd3140cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 -- 172.21.15.137:0/2300273119 shutdown_connections 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.136+0000 7fd31b400640 1 -- 172.21.15.137:0/2300273119 wait complete. 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.138+0000 7fd31b400640 1 Processor -- start 2024-01-15T17:57:14.141 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.138+0000 7fd31b400640 1 -- start start 2024-01-15T17:57:14.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd31414e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 0x7fd314157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd314157a80 0x7fd3140ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.143 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31b400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd314157f10 con 0x7fd314157a80 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31b400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31414e590 con 0x7fd314150e10 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd314150050 con 0x7fd2f40039c0 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd31414e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd319600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 0x7fd314157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd319600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 0x7fd314157540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:34990/0 (socket says 172.21.15.137:34990) 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd31a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd31414e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58306/0 (socket says 172.21.15.137:58306) 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.139+0000 7fd319600640 1 -- 172.21.15.137:0/2802041544 learned_addr learned my addr 172.21.15.137:0/2802041544 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd31aa00640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd314157a80 0x7fd3140ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd319600640 1 -- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 msgr2=0x7fd31414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd319600640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd31414e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd319600640 1 -- 172.21.15.137:0/2802041544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd314157a80 msgr2=0x7fd3140ae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd319600640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd314157a80 0x7fd3140ae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd319600640 1 -- 172.21.15.137:0/2802041544 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd314150330 con 0x7fd314150e10 2024-01-15T17:57:14.144 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd31aa00640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd314157a80 0x7fd3140ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd319600640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 0x7fd314157540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00e480 tx=0x7fd2fc00e940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc0052e0 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2fc005440 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc003810 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.140+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd314150610 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.141+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31414fc90 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.141+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd2fc003c70 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.141+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fd2fc01d880 con 0x7fd314150e10 2024-01-15T17:57:14.145 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.142+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31406c260 con 0x7fd314150e10 2024-01-15T17:57:14.147 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.144+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd2fc01dcd0 con 0x7fd314150e10 2024-01-15T17:57:14.149 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.146+0000 7fe29584a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T17:57:14.185 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.182+0000 7fe4168b0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-15T17:57:14.204 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.201+0000 7fe29584a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T17:57:14.225 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.222+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c591f415-915a-45d6-9a06-70ca6673ddb6", "id": 2} v 0) v1 -- 0x7fd314074770 con 0x7fd314150e10 2024-01-15T17:57:14.231 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.228+0000 7fd312000640 1 -- 172.21.15.137:0/2802041544 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c591f415-915a-45d6-9a06-70ca6673ddb6", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd2fc01d460 con 0x7fd314150e10 2024-01-15T17:57:14.231 INFO:teuthology.orchestra.run.smithi137.stdout:2 2024-01-15T17:57:14.232 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 msgr2=0x7fd314157540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 0x7fd314157540 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc00e480 tx=0x7fd2fc00e940 comp rx=0 tx=0).stop 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 shutdown_connections 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd314157a80 0x7fd3140ae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd314150e10 0x7fd314157540 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 --2- 172.21.15.137:0/2802041544 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2f40039c0 0x7fd31414e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.229+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 >> 172.21.15.137:0/2802041544 conn(0x7fd3140cbf70 msgr2=0x7fd31406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.230+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 shutdown_connections 2024-01-15T17:57:14.233 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.230+0000 7fd31b400640 1 -- 172.21.15.137:0/2802041544 wait complete. 2024-01-15T17:57:14.241 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.237+0000 7fe4168b0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-15T17:57:14.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 3699a393-59fb-49ae-9d69-a211393cd800 3 2024-01-15T17:57:14.268 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.265+0000 7fe29584a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T17:57:14.304 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.300+0000 7fe4168b0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-15T17:57:14.326 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.323+0000 7fe29584a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T17:57:14.361 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.357+0000 7fe4168b0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-15T17:57:14.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.431+0000 7f0c8f600640 1 Processor -- start 2024-01-15T17:57:14.434 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.431+0000 7f0c8f600640 1 -- start start 2024-01-15T17:57:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c8f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c8815d730 0x7f0c880d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c8f600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0c880b8c30 0x7f0c880d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c8f600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c880d1900 con 0x7f0c880b4c70 2024-01-15T17:57:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c8f600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c880d1be0 con 0x7f0c880b8c30 2024-01-15T17:57:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c8f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c880d1ec0 con 0x7f0c8815d730 2024-01-15T17:57:14.437 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c8815d730 0x7f0c880d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.433+0000 7f0c8cc00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f0c880b4c70 0x7f0c880b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:49976/0 (socket says 172.21.15.137:49976) 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c8cc00640 1 -- 172.21.15.137:0/3606286539 learned_addr learned my addr 172.21.15.137:0/3606286539 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c8d600640 1 --2- 172.21.15.137:0/3606286539 >> v2:172.21.15.2:3300/0 conn(0x7f0c880b8c30 0x7f0c880d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c87400640 1 -- 172.21.15.137:0/3606286539 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341886843 0 0) 0x7f0c880d1900 con 0x7f0c880b4c70 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c87400640 1 -- 172.21.15.137:0/3606286539 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c68003520 con 0x7f0c880b4c70 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c87e00640 1 -- 172.21.15.137:0/3606286539 >> v2:172.21.15.2:3300/0 conn(0x7f0c880b8c30 msgr2=0x7f0c880d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c87e00640 1 --2- 172.21.15.137:0/3606286539 >> v2:172.21.15.2:3300/0 conn(0x7f0c880b8c30 0x7f0c880d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c87e00640 1 -- 172.21.15.137:0/3606286539 >> v1:172.21.15.137:6789/0 conn(0x7f0c880b4c70 legacy=0x7f0c880b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.438 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.434+0000 7f0c87e00640 1 -- 172.21.15.137:0/3606286539 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c880d21a0 con 0x7f0c8815d730 2024-01-15T17:57:14.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.435+0000 7f0c87e00640 1 --2- 172.21.15.137:0/3606286539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c8815d730 0x7f0c880d1270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0c700097a0 tx=0x7f0c7002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c63e5154b1a51bc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:14.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.435+0000 7f0c87400640 1 -- 172.21.15.137:0/3606286539 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7002d040 con 0x7f0c8815d730 2024-01-15T17:57:14.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.435+0000 7f0c87400640 1 -- 172.21.15.137:0/3606286539 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c70031810 con 0x7f0c8815d730 2024-01-15T17:57:14.439 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.435+0000 7f0c87400640 1 -- 172.21.15.137:0/3606286539 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70031ba0 con 0x7f0c8815d730 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 -- 172.21.15.137:0/3606286539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c8815d730 msgr2=0x7f0c880d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 --2- 172.21.15.137:0/3606286539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c8815d730 0x7f0c880d1270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0c700097a0 tx=0x7f0c7002b8b0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 -- 172.21.15.137:0/3606286539 shutdown_connections 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 --2- 172.21.15.137:0/3606286539 >> v2:172.21.15.2:3300/0 conn(0x7f0c880b8c30 0x7f0c880d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 --2- 172.21.15.137:0/3606286539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c8815d730 0x7f0c880d1270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 -- 172.21.15.137:0/3606286539 >> 172.21.15.137:0/3606286539 conn(0x7f0c88072210 msgr2=0x7f0c880cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 -- 172.21.15.137:0/3606286539 shutdown_connections 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.436+0000 7f0c8f600640 1 -- 172.21.15.137:0/3606286539 wait complete. 2024-01-15T17:57:14.440 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.438+0000 7f0c8f600640 1 Processor -- start 2024-01-15T17:57:14.441 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.438+0000 7f0c8f600640 1 -- start start 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8f600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c880b4c70 0x7f0c880b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 0x7f0c880b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8f600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 0x7f0c881f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8f600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c88156f70 con 0x7f0c880b4c70 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8f600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8815fb60 con 0x7f0c8815d730 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8815fe40 con 0x7f0c880b8c30 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 0x7f0c881f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 0x7f0c880b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.443 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c87e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 0x7f0c880b3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58334/0 (socket says 172.21.15.137:58334) 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c87e00640 1 -- 172.21.15.137:0/2446029212 learned_addr learned my addr 172.21.15.137:0/2446029212 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 0x7f0c881f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:34996/0 (socket says 172.21.15.137:34996) 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.439+0000 7f0c8cc00640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c880b4c70 0x7f0c880b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c87e00640 1 -- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 msgr2=0x7f0c881f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c87e00640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 0x7f0c881f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c87e00640 1 -- 172.21.15.137:0/2446029212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c880b4c70 msgr2=0x7f0c880b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c87e00640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c880b4c70 0x7f0c880b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c87e00640 1 -- 172.21.15.137:0/2446029212 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c70009400 con 0x7f0c880b8c30 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c8d600640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 0x7f0c881f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c87e00640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 0x7f0c880b3a90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c70000c00 tx=0x7f0c700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70006e10 con 0x7f0c880b8c30 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7003ad60 con 0x7f0c880b8c30 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c880d3060 con 0x7f0c880b8c30 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70044630 con 0x7f0c880b8c30 2024-01-15T17:57:14.444 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.440+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c88160400 con 0x7f0c880b8c30 2024-01-15T17:57:14.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.441+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0c70044a90 con 0x7f0c880b8c30 2024-01-15T17:57:14.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.441+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f0c70044d40 con 0x7f0c880b8c30 2024-01-15T17:57:14.445 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.441+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8815fca0 con 0x7f0c880b8c30 2024-01-15T17:57:14.446 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.444+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0c8815fca0 con 0x7f0c880b8c30 2024-01-15T17:57:14.524 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.521+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3699a393-59fb-49ae-9d69-a211393cd800", "id": 3} v 0) v1 -- 0x7f0c880735d0 con 0x7f0c880b8c30 2024-01-15T17:57:14.531 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.528+0000 7f0c84c00640 1 -- 172.21.15.137:0/2446029212 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3699a393-59fb-49ae-9d69-a211393cd800", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0c70052070 con 0x7f0c880b8c30 2024-01-15T17:57:14.531 INFO:teuthology.orchestra.run.smithi137.stdout:3 2024-01-15T17:57:14.532 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.529+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 msgr2=0x7f0c880b3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.532 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.529+0000 7f0c8f600640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 0x7f0c880b3a90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c70000c00 tx=0x7f0c700046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 shutdown_connections 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c8815d730 0x7f0c881f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c880b8c30 0x7f0c880b3a90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 --2- 172.21.15.137:0/2446029212 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c880b4c70 0x7f0c880b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 >> 172.21.15.137:0/2446029212 conn(0x7f0c88072210 msgr2=0x7f0c880b62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 shutdown_connections 2024-01-15T17:57:14.533 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.530+0000 7f0c8f600640 1 -- 172.21.15.137:0/2446029212 wait complete. 2024-01-15T17:57:14.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 764ae6bd-491a-47ba-b7c7-7530e7c38ad7 4 2024-01-15T17:57:14.786 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.783+0000 7fa977e00640 1 Processor -- start 2024-01-15T17:57:14.786 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.783+0000 7fa977e00640 1 -- start start 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.784+0000 7fa977e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa97015d730 0x7fa9700d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.784+0000 7fa977e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa9700b8c30 0x7fa9700d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.784+0000 7fa977e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9700d1900 con 0x7fa9700b4c70 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.784+0000 7fa977e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9700d1be0 con 0x7fa9700b8c30 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.784+0000 7fa977e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9700d1ec0 con 0x7fa97015d730 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa97015d730 0x7fa9700d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa97015d730 0x7fa9700d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:34998/0 (socket says 172.21.15.137:34998) 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 -- 172.21.15.137:0/341729648 learned_addr learned my addr 172.21.15.137:0/341729648 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa975e00640 1 --2- 172.21.15.137:0/341729648 >> v2:172.21.15.2:3300/0 conn(0x7fa9700b8c30 0x7fa9700d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 -- 172.21.15.137:0/341729648 >> v2:172.21.15.2:3300/0 conn(0x7fa9700b8c30 msgr2=0x7fa9700d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:14.789 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 --2- 172.21.15.137:0/341729648 >> v2:172.21.15.2:3300/0 conn(0x7fa9700b8c30 0x7fa9700d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.789 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 -- 172.21.15.137:0/341729648 >> v1:172.21.15.2:6789/0 conn(0x7fa9700b4c70 legacy=0x7fa9700b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:14.789 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.785+0000 7fa974a00640 1 -- 172.21.15.137:0/341729648 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9700d21a0 con 0x7fa97015d730 2024-01-15T17:57:14.789 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.787+0000 7fa974a00640 1 --2- 172.21.15.137:0/341729648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa97015d730 0x7fa9700d1270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa97008c6b0 tx=0x7fa95802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e965048e288bbc96 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:14.790 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.787+0000 7fa96fe00640 1 -- 172.21.15.137:0/341729648 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958031870 con 0x7fa97015d730 2024-01-15T17:57:14.790 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.787+0000 7fa96fe00640 1 -- 172.21.15.137:0/341729648 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9580319d0 con 0x7fa97015d730 2024-01-15T17:57:14.790 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.787+0000 7fa96fe00640 1 -- 172.21.15.137:0/341729648 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958031d60 con 0x7fa97015d730 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 -- 172.21.15.137:0/341729648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa97015d730 msgr2=0x7fa9700d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 --2- 172.21.15.137:0/341729648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa97015d730 0x7fa9700d1270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa97008c6b0 tx=0x7fa95802b910 comp rx=0 tx=0).stop 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 -- 172.21.15.137:0/341729648 shutdown_connections 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 --2- 172.21.15.137:0/341729648 >> v2:172.21.15.2:3300/0 conn(0x7fa9700b8c30 0x7fa9700d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 --2- 172.21.15.137:0/341729648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa97015d730 0x7fa9700d1270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 -- 172.21.15.137:0/341729648 >> 172.21.15.137:0/341729648 conn(0x7fa970072210 msgr2=0x7fa9700cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 -- 172.21.15.137:0/341729648 shutdown_connections 2024-01-15T17:57:14.791 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.788+0000 7fa977e00640 1 -- 172.21.15.137:0/341729648 wait complete. 2024-01-15T17:57:14.792 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.790+0000 7fa977e00640 1 Processor -- start 2024-01-15T17:57:14.792 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.790+0000 7fa977e00640 1 -- start start 2024-01-15T17:57:14.794 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa977e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 0x7fa9701f0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.794 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa977e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9700b8c30 0x7fa9700c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa977e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa97015d730 0x7fa9700c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa977e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9701f3f10 con 0x7fa9700b8c30 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa977e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9701f3000 con 0x7fa9700b4c70 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa977e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9701f3290 con 0x7fa97015d730 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa975400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 0x7fa9701f0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa975400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 0x7fa9701f0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:35012/0 (socket says 172.21.15.137:35012) 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa975400640 1 -- 172.21.15.137:0/1055269257 learned_addr learned my addr 172.21.15.137:0/1055269257 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa975e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa97015d730 0x7fa9700c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.791+0000 7fa974a00640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9700b8c30 0x7fa9700c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa975400640 1 -- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa97015d730 msgr2=0x7fa9700c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa975400640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa97015d730 0x7fa9700c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa975400640 1 -- 172.21.15.137:0/1055269257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9700b8c30 msgr2=0x7fa9700c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa975400640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9700b8c30 0x7fa9700c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa975400640 1 -- 172.21.15.137:0/1055269257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9580094d0 con 0x7fa9700b4c70 2024-01-15T17:57:14.795 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa975400640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 0x7fa9701f0600 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa96000c430 tx=0x7fa96000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:14.796 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960011790 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9600118f0 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa960011ce0 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9700d3060 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.792+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9701f3850 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.793+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa96001c760 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.793+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fa960011e40 con 0x7fa9700b4c70 2024-01-15T17:57:14.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.794+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9700686f0 con 0x7fa9700b4c70 2024-01-15T17:57:14.798 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.796+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa960012740 con 0x7fa9700b4c70 2024-01-15T17:57:14.804 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.801+0000 7fe29584a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-15T17:57:14.833 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.829+0000 7fe4168b0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-15T17:57:14.877 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.874+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "764ae6bd-491a-47ba-b7c7-7530e7c38ad7", "id": 4} v 0) v1 -- 0x7fa97006c260 con 0x7fa9700b4c70 2024-01-15T17:57:14.884 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.881+0000 7fa96d600640 1 -- 172.21.15.137:0/1055269257 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "764ae6bd-491a-47ba-b7c7-7530e7c38ad7", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa96001ca10 con 0x7fa9700b4c70 2024-01-15T17:57:14.884 INFO:teuthology.orchestra.run.smithi137.stdout:4 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 msgr2=0x7fa9701f0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 0x7fa9701f0600 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa96000c430 tx=0x7fa96000c8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 shutdown_connections 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa97015d730 0x7fa9700c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa9700b8c30 0x7fa9700c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 --2- 172.21.15.137:0/1055269257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa9700b4c70 0x7fa9701f0600 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 >> 172.21.15.137:0/1055269257 conn(0x7fa970072210 msgr2=0x7fa9700b3f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 shutdown_connections 2024-01-15T17:57:14.886 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:14.883+0000 7fa977e00640 1 -- 172.21.15.137:0/1055269257 wait complete. 2024-01-15T17:57:14.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 495645fe-5096-4216-bc23-705f01242354 5 2024-01-15T17:57:14.970 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:14.967+0000 7fe29584a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T17:57:15.001 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:14.997+0000 7fe4168b0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-15T17:57:15.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.085+0000 7fda37200640 1 Processor -- start 2024-01-15T17:57:15.087 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.085+0000 7fda37200640 1 -- start start 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda37200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda38160830 0x7fda380ad750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda37200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 0x7fda38160d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda37200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda380af3a0 con 0x7fda380b4ae0 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda37200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3806d1f0 con 0x7fda38160830 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda37200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda380b31c0 con 0x7fda3806abc0 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda35e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda38160830 0x7fda380ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda36800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 0x7fda38160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda35e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda38160830 0x7fda380ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:35020/0 (socket says 172.21.15.137:35020) 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda35e00640 1 -- 172.21.15.137:0/2171236436 learned_addr learned my addr 172.21.15.137:0/2171236436 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda36800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 0x7fda38160d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58360/0 (socket says 172.21.15.137:58360) 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.086+0000 7fda2d200640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fda380b4ae0 0x7fda3806a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:49982/0 (socket says 172.21.15.137:49982) 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.087+0000 7fda35400640 1 -- 172.21.15.137:0/2171236436 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091726658 0 0) 0x7fda380af3a0 con 0x7fda380b4ae0 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.087+0000 7fda36800640 1 -- 172.21.15.137:0/2171236436 >> v2:172.21.15.137:3300/0 conn(0x7fda38160830 msgr2=0x7fda380ad750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.087+0000 7fda36800640 1 --2- 172.21.15.137:0/2171236436 >> v2:172.21.15.137:3300/0 conn(0x7fda38160830 0x7fda380ad750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.087+0000 7fda36800640 1 -- 172.21.15.137:0/2171236436 >> v1:172.21.15.137:6789/0 conn(0x7fda380b4ae0 legacy=0x7fda3806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.090 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.087+0000 7fda36800640 1 -- 172.21.15.137:0/2171236436 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda380b34a0 con 0x7fda3806abc0 2024-01-15T17:57:15.091 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.088+0000 7fda36800640 1 --2- 172.21.15.137:0/2171236436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 0x7fda38160d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda300097a0 tx=0x7fda3002b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11588f4e54eeeba3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.088+0000 7fda35400640 1 -- 172.21.15.137:0/2171236436 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30031780 con 0x7fda3806abc0 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.088+0000 7fda35400640 1 -- 172.21.15.137:0/2171236436 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda300318e0 con 0x7fda3806abc0 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.088+0000 7fda35400640 1 -- 172.21.15.137:0/2171236436 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30031c70 con 0x7fda3806abc0 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 -- 172.21.15.137:0/2171236436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 msgr2=0x7fda38160d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 --2- 172.21.15.137:0/2171236436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 0x7fda38160d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda300097a0 tx=0x7fda3002b820 comp rx=0 tx=0).stop 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 -- 172.21.15.137:0/2171236436 shutdown_connections 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 --2- 172.21.15.137:0/2171236436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda3806abc0 0x7fda38160d40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 --2- 172.21.15.137:0/2171236436 >> v2:172.21.15.137:3300/0 conn(0x7fda38160830 0x7fda380ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 -- 172.21.15.137:0/2171236436 >> 172.21.15.137:0/2171236436 conn(0x7fda3806b5c0 msgr2=0x7fda3806b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 -- 172.21.15.137:0/2171236436 shutdown_connections 2024-01-15T17:57:15.092 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.089+0000 7fda37200640 1 -- 172.21.15.137:0/2171236436 wait complete. 2024-01-15T17:57:15.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.091+0000 7fda37200640 1 Processor -- start 2024-01-15T17:57:15.093 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.091+0000 7fda37200640 1 -- start start 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda37200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda3806abc0 0x7fda3815c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda37200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 0x7fda381e6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda37200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda38160830 0x7fda38146360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda37200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3815a100 con 0x7fda3806abc0 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda37200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda381e59c0 con 0x7fda38160830 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda37200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda381e5ca0 con 0x7fda380b4ae0 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 0x7fda381e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda36800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda38160830 0x7fda38146360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.096 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 0x7fda381e6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58372/0 (socket says 172.21.15.137:58372) 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 -- 172.21.15.137:0/2736248506 learned_addr learned my addr 172.21.15.137:0/2736248506 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda35e00640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda3806abc0 0x7fda3815c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 -- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda38160830 msgr2=0x7fda38146360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda38160830 0x7fda38146360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 -- 172.21.15.137:0/2736248506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda3806abc0 msgr2=0x7fda3815c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda3806abc0 0x7fda3815c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.092+0000 7fda2d200640 1 -- 172.21.15.137:0/2736248506 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda30009400 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda2d200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 0x7fda381e6780 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fda2000c550 tx=0x7fda2000ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20007500 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda200057a0 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda20005b50 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda381e5f80 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3815c660 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fda2001c440 con 0x7fda380b4ae0 2024-01-15T17:57:15.097 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.093+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fda20012980 con 0x7fda380b4ae0 2024-01-15T17:57:15.098 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.094+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda381e64a0 con 0x7fda380b4ae0 2024-01-15T17:57:15.100 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.098+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fda381e64a0 con 0x7fda380b4ae0 2024-01-15T17:57:15.106 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.103+0000 7fe29584a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T17:57:15.141 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.137+0000 7fe4168b0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-15T17:57:15.161 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.158+0000 7fe29584a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T17:57:15.178 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.175+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "495645fe-5096-4216-bc23-705f01242354", "id": 5} v 0) v1 -- 0x7fda38074770 con 0x7fda380b4ae0 2024-01-15T17:57:15.184 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.181+0000 7fda2ea00640 1 -- 172.21.15.137:0/2736248506 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "495645fe-5096-4216-bc23-705f01242354", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fda20017070 con 0x7fda380b4ae0 2024-01-15T17:57:15.184 INFO:teuthology.orchestra.run.smithi137.stdout:5 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 msgr2=0x7fda381e6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 0x7fda381e6780 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fda2000c550 tx=0x7fda2000ca10 comp rx=0 tx=0).stop 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 shutdown_connections 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda38160830 0x7fda38146360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda380b4ae0 0x7fda381e6780 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 --2- 172.21.15.137:0/2736248506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda3806abc0 0x7fda3815c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 >> 172.21.15.137:0/2736248506 conn(0x7fda3806b5c0 msgr2=0x7fda380b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 shutdown_connections 2024-01-15T17:57:15.186 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.183+0000 7fda37200640 1 -- 172.21.15.137:0/2736248506 wait complete. 2024-01-15T17:57:15.196 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.193+0000 7fe4168b0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-15T17:57:15.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new 8d8a340d-f414-41b6-8003-0a7ec08d73a8 6 2024-01-15T17:57:15.280 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.277+0000 7fe29584a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T17:57:15.311 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.308+0000 7fe4168b0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-15T17:57:15.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.390+0000 7fe0fe600640 1 Processor -- start 2024-01-15T17:57:15.393 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.390+0000 7fe0fe600640 1 -- start start 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0fe600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f815d730 0x7fe0f80d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0fe600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe0f80b8c30 0x7fe0f80d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0fe600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f80d1900 con 0x7fe0f80b4c70 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0fe600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0f80d1be0 con 0x7fe0f80b8c30 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0fe600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80d1ec0 con 0x7fe0f815d730 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0efe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f815d730 0x7fe0f80d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0f7400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fe0f80b4c70 0x7fe0f80b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.137:49994/0 (socket says 172.21.15.137:49994) 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0f7400640 1 -- 172.21.15.137:0/4278641187 learned_addr learned my addr 172.21.15.137:0/4278641187 (peer_addr_for_me v1:172.21.15.137:0/0) 2024-01-15T17:57:15.395 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0f7e00640 1 --2- 172.21.15.137:0/4278641187 >> v2:172.21.15.2:3300/0 conn(0x7fe0f80b8c30 0x7fe0f80d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.396 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0efe00640 1 -- 172.21.15.137:0/4278641187 >> v2:172.21.15.2:3300/0 conn(0x7fe0f80b8c30 msgr2=0x7fe0f80d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:15.396 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0efe00640 1 --2- 172.21.15.137:0/4278641187 >> v2:172.21.15.2:3300/0 conn(0x7fe0f80b8c30 0x7fe0f80d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.396 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0efe00640 1 -- 172.21.15.137:0/4278641187 >> v1:172.21.15.137:6789/0 conn(0x7fe0f80b4c70 legacy=0x7fe0f80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.396 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0efe00640 1 -- 172.21.15.137:0/4278641187 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f80d21a0 con 0x7fe0f815d730 2024-01-15T17:57:15.396 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.392+0000 7fe0f6a00640 1 -- 172.21.15.137:0/4278641187 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273386258 0 0) 0x7fe0f80d1900 con 0x7fe0f80b4c70 2024-01-15T17:57:15.397 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.393+0000 7fe0efe00640 1 --2- 172.21.15.137:0/4278641187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f815d730 0x7fe0f80d1270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00097a0 tx=0x7fe0e002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eafe258bfab141a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:15.397 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0f6a00640 1 -- 172.21.15.137:0/4278641187 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0031830 con 0x7fe0f815d730 2024-01-15T17:57:15.397 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0f6a00640 1 -- 172.21.15.137:0/4278641187 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e0031990 con 0x7fe0f815d730 2024-01-15T17:57:15.397 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0f6a00640 1 -- 172.21.15.137:0/4278641187 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0031d20 con 0x7fe0f815d730 2024-01-15T17:57:15.397 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe29584a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 -- 172.21.15.137:0/4278641187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f815d730 msgr2=0x7fe0f80d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 --2- 172.21.15.137:0/4278641187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f815d730 0x7fe0f80d1270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00097a0 tx=0x7fe0e002b8d0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 -- 172.21.15.137:0/4278641187 shutdown_connections 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 --2- 172.21.15.137:0/4278641187 >> v2:172.21.15.2:3300/0 conn(0x7fe0f80b8c30 0x7fe0f80d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 --2- 172.21.15.137:0/4278641187 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f815d730 0x7fe0f80d1270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 -- 172.21.15.137:0/4278641187 >> 172.21.15.137:0/4278641187 conn(0x7fe0f8072210 msgr2=0x7fe0f80cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 -- 172.21.15.137:0/4278641187 shutdown_connections 2024-01-15T17:57:15.398 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.394+0000 7fe0fe600640 1 -- 172.21.15.137:0/4278641187 wait complete. 2024-01-15T17:57:15.399 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.396+0000 7fe0fe600640 1 Processor -- start 2024-01-15T17:57:15.399 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.396+0000 7fe0fe600640 1 -- start start 2024-01-15T17:57:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0fe600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 0x7fe0f8148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0fe600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f80b8c30 0x7fe0f80c7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0fe600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0f815d730 0x7fe0f80c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0fe600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f8160670 con 0x7fe0f815d730 2024-01-15T17:57:15.401 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0fe600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80cc4a0 con 0x7fe0f80b8c30 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0fe600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f80cc780 con 0x7fe0f80b4c70 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0f7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 0x7fe0f8148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0efe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f80b8c30 0x7fe0f80c7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0f7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 0x7fe0f8148f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58386/0 (socket says 172.21.15.137:58386) 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.397+0000 7fe0f7400640 1 -- 172.21.15.137:0/3372385257 learned_addr learned my addr 172.21.15.137:0/3372385257 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7e00640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0f815d730 0x7fe0f80c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7400640 1 -- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f80b8c30 msgr2=0x7fe0f80c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7400640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f80b8c30 0x7fe0f80c7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7400640 1 -- 172.21.15.137:0/3372385257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0f815d730 msgr2=0x7fe0f80c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7400640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0f815d730 0x7fe0f80c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7400640 1 -- 172.21.15.137:0/3372385257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e0009400 con 0x7fe0f80b4c70 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7e00640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0f815d730 0x7fe0f80c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:15.402 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0f7400640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 0x7fe0f8148f70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800c4d0 tx=0x7fe0e800c990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e80074e0 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e8007660 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f80d3060 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8004a20 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.398+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f8160950 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.399+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe0e801d440 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.399+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fe0e80129f0 con 0x7fe0f80b4c70 2024-01-15T17:57:15.403 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.400+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f80ccca0 con 0x7fe0f80b4c70 2024-01-15T17:57:15.404 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.402+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe0f80ccca0 con 0x7fe0f80b4c70 2024-01-15T17:57:15.427 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.423+0000 7fe4168b0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-15T17:57:15.482 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.479+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8d8a340d-f414-41b6-8003-0a7ec08d73a8", "id": 6} v 0) v1 -- 0x7fe0f80b2990 con 0x7fe0f80b4c70 2024-01-15T17:57:15.489 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.486+0000 7fe0ef400640 1 -- 172.21.15.137:0/3372385257 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8d8a340d-f414-41b6-8003-0a7ec08d73a8", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe0e8018070 con 0x7fe0f80b4c70 2024-01-15T17:57:15.489 INFO:teuthology.orchestra.run.smithi137.stdout:6 2024-01-15T17:57:15.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.487+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 msgr2=0x7fe0f8148f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.490 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.487+0000 7fe0fe600640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 0x7fe0f8148f70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800c4d0 tx=0x7fe0e800c990 comp rx=0 tx=0).stop 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 shutdown_connections 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0f815d730 0x7fe0f80c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0f80b8c30 0x7fe0f80c7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 --2- 172.21.15.137:0/3372385257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0f80b4c70 0x7fe0f8148f70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 >> 172.21.15.137:0/3372385257 conn(0x7fe0f8072210 msgr2=0x7fe0f8073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 shutdown_connections 2024-01-15T17:57:15.491 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.488+0000 7fe0fe600640 1 -- 172.21.15.137:0/3372385257 wait complete. 2024-01-15T17:57:15.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --cluster ceph osd new a1687635-8b85-4352-940f-5f607a359beb 7 2024-01-15T17:57:15.564 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.562+0000 7fe29584a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T17:57:15.594 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.589+0000 7fe4168b0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-15T17:57:15.616 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.613+0000 7fe29584a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T17:57:15.644 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.641+0000 7fe4168b0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-15T17:57:15.698 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.695+0000 7efdb7e00640 1 Processor -- start 2024-01-15T17:57:15.698 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.695+0000 7efdb7e00640 1 -- start start 2024-01-15T17:57:15.700 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efdb8160980 0x7efdb80cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.700 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb80cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efdb80d01a0 0x7efdb80d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80cf010 con 0x7efdb8160980 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb80ce7b0 con 0x7efdb80d01a0 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80cea50 con 0x7efdb80b4040 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efdb80d01a0 0x7efdb80d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efdb80d01a0 0x7efdb80d0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:35054/0 (socket says 172.21.15.137:35054) 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdb7400640 1 -- 172.21.15.137:0/2077990666 learned_addr learned my addr 172.21.15.137:0/2077990666 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.697+0000 7efdafe00640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb80cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdb6a00640 1 --2- 172.21.15.137:0/2077990666 >> v2:172.21.15.2:3300/0 conn(0x7efdb8160980 0x7efdb80cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdb7400640 1 -- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 msgr2=0x7efdb80cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdb7400640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb80cfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdb7400640 1 -- 172.21.15.137:0/2077990666 >> v2:172.21.15.2:3300/0 conn(0x7efdb8160980 msgr2=0x7efdb80cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdb7400640 1 --2- 172.21.15.137:0/2077990666 >> v2:172.21.15.2:3300/0 conn(0x7efdb8160980 0x7efdb80cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdb7400640 1 -- 172.21.15.137:0/2077990666 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb80ced30 con 0x7efdb80d01a0 2024-01-15T17:57:15.701 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.698+0000 7efdafe00640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb80cfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:15.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.699+0000 7efdb7400640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80d0dd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efdb808c360 tx=0x7efdb002bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91c60f5efd206f88 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:15.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.699+0000 7efdb6000640 1 -- 172.21.15.137:0/2077990666 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb002ca70 con 0x7efdb80d01a0 2024-01-15T17:57:15.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.699+0000 7efdb6000640 1 -- 172.21.15.137:0/2077990666 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb002cbd0 con 0x7efdb80d01a0 2024-01-15T17:57:15.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.699+0000 7efdb6000640 1 -- 172.21.15.137:0/2077990666 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb00343f0 con 0x7efdb80d01a0 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 -- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 msgr2=0x7efdb80d0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80d0dd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efdb808c360 tx=0x7efdb002bed0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 -- 172.21.15.137:0/2077990666 shutdown_connections 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80d0dd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2077990666 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb80cfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2077990666 >> v2:172.21.15.2:3300/0 conn(0x7efdb8160980 0x7efdb80cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 -- 172.21.15.137:0/2077990666 >> 172.21.15.137:0/2077990666 conn(0x7efdb80d2ff0 msgr2=0x7efdb80d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 -- 172.21.15.137:0/2077990666 shutdown_connections 2024-01-15T17:57:15.703 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.700+0000 7efdb7e00640 1 -- 172.21.15.137:0/2077990666 wait complete. 2024-01-15T17:57:15.704 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.702+0000 7efdb7e00640 1 Processor -- start 2024-01-15T17:57:15.704 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.702+0000 7efdb7e00640 1 -- start start 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb8159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efdb8160980 0x7efdb80d0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81f2c60 con 0x7efdb8160980 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb80cad20 con 0x7efdb80d01a0 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb81f2f00 con 0x7efdb80b4040 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdafe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb6a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb8159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdafe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.137:35062/0 (socket says 172.21.15.137:35062) 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdafe00640 1 -- 172.21.15.137:0/2507336943 learned_addr learned my addr 172.21.15.137:0/2507336943 (peer_addr_for_me v2:172.21.15.137:0/0) 2024-01-15T17:57:15.706 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.703+0000 7efdb6a00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb8159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.137:58400/0 (socket says 172.21.15.137:58400) 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdb7400640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efdb8160980 0x7efdb80d0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdafe00640 1 -- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 msgr2=0x7efdb8159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdafe00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb8159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdafe00640 1 -- 172.21.15.137:0/2507336943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efdb8160980 msgr2=0x7efdb80d0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdafe00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efdb8160980 0x7efdb80d0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdafe00640 1 -- 172.21.15.137:0/2507336943 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda0009400 con 0x7efdb80d01a0 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdafe00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80ca360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efda000e980 tx=0x7efda000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda00096e0 con 0x7efdb80d01a0 2024-01-15T17:57:15.708 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda000ca00 con 0x7efdb80d01a0 2024-01-15T17:57:15.709 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb0009b30 con 0x7efdb80d01a0 2024-01-15T17:57:15.709 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda000cd90 con 0x7efdb80d01a0 2024-01-15T17:57:15.709 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.704+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb80cb000 con 0x7efdb80d01a0 2024-01-15T17:57:15.709 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.705+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7efda0005c90 con 0x7efdb80d01a0 2024-01-15T17:57:15.709 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.705+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7efda0013470 con 0x7efdb80d01a0 2024-01-15T17:57:15.709 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.705+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb81f3420 con 0x7efdb80d01a0 2024-01-15T17:57:15.710 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.708+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7efda0021060 con 0x7efdb80d01a0 2024-01-15T17:57:15.783 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:15.781+0000 7fe29584a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T17:57:15.788 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.785+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a1687635-8b85-4352-940f-5f607a359beb", "id": 7} v 0) v1 -- 0x7efdb8074770 con 0x7efdb80d01a0 2024-01-15T17:57:15.797 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.795+0000 7efdaea00640 1 -- 172.21.15.137:0/2507336943 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a1687635-8b85-4352-940f-5f607a359beb", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7efda0029020 con 0x7efdb80d01a0 2024-01-15T17:57:15.797 INFO:teuthology.orchestra.run.smithi137.stdout:7 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 msgr2=0x7efdb80ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80ca360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7efda000e980 tx=0x7efda000ee40 comp rx=0 tx=0).stop 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 shutdown_connections 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efdb8160980 0x7efdb80d0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efdb80d01a0 0x7efdb80ca360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 --2- 172.21.15.137:0/2507336943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efdb80b4040 0x7efdb8159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:15.799 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 >> 172.21.15.137:0/2507336943 conn(0x7efdb80d2ff0 msgr2=0x7efdb815fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:15.800 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 shutdown_connections 2024-01-15T17:57:15.800 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:57:15.796+0000 7efdb7e00640 1 -- 172.21.15.137:0/2507336943 wait complete. 2024-01-15T17:57:15.812 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:15.808+0000 7fe4168b0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-15T17:57:15.814 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-15T17:57:15.815 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-15T17:57:15.822 INFO:tasks.ceph.osd.0:Started 2024-01-15T17:57:15.822 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-15T17:57:15.822 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-15T17:57:15.829 INFO:tasks.ceph.osd.1:Started 2024-01-15T17:57:15.829 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-15T17:57:15.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-15T17:57:15.834 INFO:tasks.ceph.osd.2:Started 2024-01-15T17:57:15.834 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-15T17:57:15.834 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-15T17:57:15.839 INFO:tasks.ceph.osd.3:Started 2024-01-15T17:57:15.839 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-15T17:57:15.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-15T17:57:15.846 INFO:tasks.ceph.osd.4:Started 2024-01-15T17:57:15.846 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-15T17:57:15.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-15T17:57:15.853 INFO:tasks.ceph.osd.5:Started 2024-01-15T17:57:15.853 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-15T17:57:15.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-15T17:57:15.860 INFO:tasks.ceph.osd.6:Started 2024-01-15T17:57:15.860 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-15T17:57:15.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-15T17:57:15.866 INFO:tasks.ceph.osd.7:Started 2024-01-15T17:57:15.867 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:16.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.078+0000 7f5700e00640 1 Processor -- start 2024-01-15T17:57:16.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.078+0000 7f5700e00640 1 -- start start 2024-01-15T17:57:16.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f5700e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f5700e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f56fc0d1d90 0x7f56fc0d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f5700e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56fc0d3a30 con 0x7f56fc0d3090 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f5700e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56fc0b3de0 con 0x7f56fc0d1d90 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f5700e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc0b40c0 con 0x7f56fc15d340 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f56f3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f56fc0d3090 0x7f56fc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35692/0 (socket says 172.21.15.2:35692) 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f56f3e00640 1 -- 172.21.15.2:0/2691286932 learned_addr learned my addr 172.21.15.2:0/2691286932 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f56fbe00640 1 --2- 172.21.15.2:0/2691286932 >> v2:172.21.15.2:3300/0 conn(0x7f56fc0d1d90 0x7f56fc0d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.079+0000 7f56fb400640 1 --2- 172.21.15.2:0/2691286932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.080+0000 7f56fbe00640 1 -- 172.21.15.2:0/2691286932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 msgr2=0x7f56fc0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.080+0000 7f56fbe00640 1 --2- 172.21.15.2:0/2691286932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc0cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.080+0000 7f56fbe00640 1 -- 172.21.15.2:0/2691286932 >> v1:172.21.15.2:6789/0 conn(0x7f56fc0d3090 legacy=0x7f56fc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.080+0000 7f56fbe00640 1 -- 172.21.15.2:0/2691286932 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc0d0e80 con 0x7f56fc0d1d90 2024-01-15T17:57:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.081+0000 7f56fbe00640 1 --2- 172.21.15.2:0/2691286932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d1d90 0x7f56fc0d34f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009d90 tx=0x7f56f402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb9f0b3e7ef1492 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.081+0000 7f56faa00640 1 -- 172.21.15.2:0/2691286932 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f402cc50 con 0x7f56fc0d1d90 2024-01-15T17:57:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.081+0000 7f56faa00640 1 -- 172.21.15.2:0/2691286932 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f402cdb0 con 0x7f56fc0d1d90 2024-01-15T17:57:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.081+0000 7f56faa00640 1 -- 172.21.15.2:0/2691286932 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4034660 con 0x7f56fc0d1d90 2024-01-15T17:57:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.081+0000 7f5700e00640 1 -- 172.21.15.2:0/2691286932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d1d90 msgr2=0x7f56fc0d34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.081+0000 7f5700e00640 1 --2- 172.21.15.2:0/2691286932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d1d90 0x7f56fc0d34f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f56f4009d90 tx=0x7f56f402bea0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.082+0000 7f5700e00640 1 -- 172.21.15.2:0/2691286932 shutdown_connections 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.082+0000 7f5700e00640 1 --2- 172.21.15.2:0/2691286932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d1d90 0x7f56fc0d34f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.082+0000 7f5700e00640 1 --2- 172.21.15.2:0/2691286932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc0cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.082+0000 7f5700e00640 1 -- 172.21.15.2:0/2691286932 >> 172.21.15.2:0/2691286932 conn(0x7f56fc08caf0 msgr2=0x7f56fc09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.082+0000 7f5700e00640 1 -- 172.21.15.2:0/2691286932 shutdown_connections 2024-01-15T17:57:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.082+0000 7f5700e00640 1 -- 172.21.15.2:0/2691286932 wait complete. 2024-01-15T17:57:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.083+0000 7f5700e00640 1 Processor -- start 2024-01-15T17:57:16.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.083+0000 7f5700e00640 1 -- start start 2024-01-15T17:57:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.084+0000 7f5700e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 0x7f56fc14ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.084+0000 7f5700e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d3090 0x7f56fc0c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.084+0000 7f5700e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc15b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.084+0000 7f5700e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc0b85b0 con 0x7f56fc0d3090 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.084+0000 7f5700e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc0b7690 con 0x7f56fc0d1d90 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.084+0000 7f5700e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56fc0b7930 con 0x7f56fc15d340 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56f3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d3090 0x7f56fc0c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 0x7f56fc14ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 0x7f56fc14ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:44058/0 (socket says 172.21.15.2:44058) 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 -- 172.21.15.2:0/180185099 learned_addr learned my addr 172.21.15.2:0/180185099 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:16.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fbe00640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc15b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 -- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 msgr2=0x7f56fc15b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc15b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 -- 172.21.15.2:0/180185099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d3090 msgr2=0x7f56fc0c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d3090 0x7f56fc0c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 -- 172.21.15.2:0/180185099 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f4009a70 con 0x7f56fc0d1d90 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.085+0000 7f56fb400640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 0x7f56fc14ce30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00e9b0 tx=0x7f56ec00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec009670 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56ec00c9a0 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56ec00cd50 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56fc0b7c10 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56fc0b81d0 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f56ec005c90 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.086+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f56ec013470 con 0x7f56fc0d1d90 2024-01-15T17:57:16.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.087+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56fc074770 con 0x7f56fc0d1d90 2024-01-15T17:57:16.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.090+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f56ec021060 con 0x7f56fc0d1d90 2024-01-15T17:57:16.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.156+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56fc0678e0 con 0x7f56fc0d1d90 2024-01-15T17:57:16.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.157+0000 7f56f3400640 1 -- 172.21.15.2:0/180185099 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f56ec029020 con 0x7f56fc0d1d90 2024-01-15T17:57:16.161 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:16.161 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":10,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:15.787748+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.158+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 msgr2=0x7f56fc14ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.158+0000 7f5700e00640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 0x7f56fc14ce30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f56ec00e9b0 tx=0x7f56ec00ee70 comp rx=0 tx=0).stop 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 shutdown_connections 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56fc15d340 0x7f56fc15b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56fc0d3090 0x7f56fc0c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 --2- 172.21.15.2:0/180185099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56fc0d1d90 0x7f56fc14ce30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 >> 172.21.15.2:0/180185099 conn(0x7f56fc08caf0 msgr2=0x7f56fc073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 shutdown_connections 2024-01-15T17:57:16.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:16.159+0000 7f5700e00640 1 -- 172.21.15.2:0/180185099 wait complete. 2024-01-15T17:57:16.178 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-15T17:57:16.178 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-15T17:57:16.528 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-15T17:57:16.526+0000 7f384380d500 -1 Falling back to public interface 2024-01-15T17:57:16.529 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-01-15T17:57:16.525+0000 7ff1b0cf6500 -1 Falling back to public interface 2024-01-15T17:57:16.536 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-15T17:57:16.533+0000 7f3814025500 -1 Falling back to public interface 2024-01-15T17:57:16.559 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-01-15T17:57:16.555+0000 7f0d5a41c500 -1 Falling back to public interface 2024-01-15T17:57:16.565 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-01-15T17:57:16.561+0000 7fa9a720c500 -1 Falling back to public interface 2024-01-15T17:57:16.565 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-01-15T17:57:16.562+0000 7f6838610500 -1 Falling back to public interface 2024-01-15T17:57:16.570 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-15T17:57:16.568+0000 7fd29b8a3500 -1 Falling back to public interface 2024-01-15T17:57:16.590 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-15T17:57:16.588+0000 7f6b9ef03500 -1 Falling back to public interface 2024-01-15T17:57:16.867 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:16.864+0000 7fe29584a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T17:57:16.939 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:16.935+0000 7fe4168b0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-15T17:57:16.985 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:17.147 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.143+0000 7fdf3dc00640 1 Processor -- start 2024-01-15T17:57:17.147 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.143+0000 7fdf3dc00640 1 -- start start 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf3dc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf3806c7f0 0x7fdf3806cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf3dc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf38070950 con 0x7fdf38143be0 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf3dc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf380b3ec0 con 0x7fdf380b3270 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf3dc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf380ce070 con 0x7fdf3806c7f0 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf36a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdf38143be0 0x7fdf3815a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35694/0 (socket says 172.21.15.2:35694) 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf36a00640 1 -- 172.21.15.2:0/1788393819 learned_addr learned my addr 172.21.15.2:0/1788393819 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.145+0000 7fdf2fe00640 1 --2- 172.21.15.2:0/1788393819 >> v2:172.21.15.137:3300/0 conn(0x7fdf3806c7f0 0x7fdf3806cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.146+0000 7fdf36000640 1 -- 172.21.15.2:0/1788393819 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313002801 0 0) 0x7fdf38070950 con 0x7fdf38143be0 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.146+0000 7fdf36000640 1 -- 172.21.15.2:0/1788393819 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf18003520 con 0x7fdf38143be0 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.146+0000 7fdf2fe00640 1 -- 172.21.15.2:0/1788393819 >> v1:172.21.15.137:6789/0 conn(0x7fdf380b3270 legacy=0x7fdf380b36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.146+0000 7fdf2fe00640 1 -- 172.21.15.2:0/1788393819 >> v1:172.21.15.2:6789/0 conn(0x7fdf38143be0 legacy=0x7fdf3815a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:17.150 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.146+0000 7fdf2fe00640 1 -- 172.21.15.2:0/1788393819 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380b41a0 con 0x7fdf3806c7f0 2024-01-15T17:57:17.151 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.147+0000 7fdf2fe00640 1 --2- 172.21.15.2:0/1788393819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3806c7f0 0x7fdf3806cc80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdf200097d0 tx=0x7fdf2002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7404260628a9cefb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:17.151 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf36000640 1 -- 172.21.15.2:0/1788393819 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20030610 con 0x7fdf3806c7f0 2024-01-15T17:57:17.151 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf36000640 1 -- 172.21.15.2:0/1788393819 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf20030770 con 0x7fdf3806c7f0 2024-01-15T17:57:17.151 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf36000640 1 -- 172.21.15.2:0/1788393819 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf20030b00 con 0x7fdf3806c7f0 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1788393819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3806c7f0 msgr2=0x7fdf3806cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 --2- 172.21.15.2:0/1788393819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3806c7f0 0x7fdf3806cc80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdf200097d0 tx=0x7fdf2002b960 comp rx=0 tx=0).stop 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1788393819 shutdown_connections 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 --2- 172.21.15.2:0/1788393819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3806c7f0 0x7fdf3806cc80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1788393819 >> 172.21.15.2:0/1788393819 conn(0x7fdf3806a6e0 msgr2=0x7fdf3806aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1788393819 shutdown_connections 2024-01-15T17:57:17.152 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.148+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1788393819 wait complete. 2024-01-15T17:57:17.154 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.150+0000 7fdf3dc00640 1 Processor -- start 2024-01-15T17:57:17.154 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.150+0000 7fdf3dc00640 1 -- start start 2024-01-15T17:57:17.155 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.151+0000 7fdf3dc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 0x7fdf38140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:17.155 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.151+0000 7fdf3dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf380b3270 0x7fdf381e3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:17.155 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.151+0000 7fdf3dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf38143be0 0x7fdf381e3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:17.155 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.151+0000 7fdf3dc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf38159640 con 0x7fdf3806c7f0 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.151+0000 7fdf3dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf381585a0 con 0x7fdf38143be0 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.151+0000 7fdf3dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf38158880 con 0x7fdf380b3270 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 0x7fdf38140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf2fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf380b3270 0x7fdf381e3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 0x7fdf38140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53286/0 (socket says 172.21.15.2:53286) 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 -- 172.21.15.2:0/1037279927 learned_addr learned my addr 172.21.15.2:0/1037279927 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:17.156 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf37400640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf38143be0 0x7fdf381e3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:17.157 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 -- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf380b3270 msgr2=0x7fdf381e3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:17.157 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf380b3270 0x7fdf381e3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.157 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 -- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf38143be0 msgr2=0x7fdf381e3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:17.157 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf38143be0 0x7fdf381e3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.157 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.152+0000 7fdf36a00640 1 -- 172.21.15.2:0/1037279927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf20009400 con 0x7fdf3806c7f0 2024-01-15T17:57:17.157 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf36a00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 0x7fdf38140300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800c410 tx=0x7fdf2800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28005ce0 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf28005e40 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf280072f0 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf38158b60 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf38159120 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.153+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdf2801c440 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.154+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fdf28012bc0 con 0x7fdf3806c7f0 2024-01-15T17:57:17.158 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.154+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38068620 con 0x7fdf3806c7f0 2024-01-15T17:57:17.161 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.157+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fdf28017070 con 0x7fdf3806c7f0 2024-01-15T17:57:17.172 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:17.169+0000 7fe29584a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T17:57:17.229 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.225+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf3806e650 con 0x7fdf3806c7f0 2024-01-15T17:57:17.230 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.226+0000 7fdf2ea00640 1 -- 172.21.15.2:0/1037279927 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fdf280124f0 con 0x7fdf3806c7f0 2024-01-15T17:57:17.231 INFO:teuthology.misc.health.smithi002.stdout: 2024-01-15T17:57:17.231 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":10,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:15.787748+0000","last_up_change":"0.000000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 msgr2=0x7fdf38140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 0x7fdf38140300 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800c410 tx=0x7fdf2800c8d0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 shutdown_connections 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf38143be0 0x7fdf381e3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf380b3270 0x7fdf381e3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 --2- 172.21.15.2:0/1037279927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3806c7f0 0x7fdf38140300 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 >> 172.21.15.2:0/1037279927 conn(0x7fdf3806a6e0 msgr2=0x7fdf3806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 shutdown_connections 2024-01-15T17:57:17.232 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:17.228+0000 7fdf3dc00640 1 -- 172.21.15.2:0/1037279927 wait complete. 2024-01-15T17:57:17.233 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:17.229+0000 7fe29584a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T17:57:17.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-15T17:57:17.284 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:17.280+0000 7fe4168b0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-15T17:57:17.346 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:17.344+0000 7fe29584a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T17:57:17.347 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:17.344+0000 7fe4168b0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-15T17:57:17.468 INFO:tasks.ceph.mgr.y.smithi002.stderr:2024-01-15T17:57:17.465+0000 7fe4168b0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-15T17:57:18.393 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:18.390+0000 7fe236800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-15T17:57:18.396 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:57:18.393+0000 7fe236800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-15T17:57:20.095 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-01-15T17:57:20.092+0000 7f0d5a41c500 -1 osd.0 0 log_to_monitors true 2024-01-15T17:57:20.100 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-15T17:57:20.098+0000 7fd29b8a3500 -1 osd.7 0 log_to_monitors true 2024-01-15T17:57:20.135 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-01-15T17:57:20.133+0000 7fa9a720c500 -1 osd.3 0 log_to_monitors true 2024-01-15T17:57:20.278 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-01-15T17:57:20.275+0000 7ff1b0cf6500 -1 osd.1 0 log_to_monitors true 2024-01-15T17:57:20.304 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-15T17:57:20.302+0000 7f3814025500 -1 osd.6 0 log_to_monitors true 2024-01-15T17:57:20.339 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-15T17:57:20.336+0000 7f384380d500 -1 osd.5 0 log_to_monitors true 2024-01-15T17:57:20.377 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-15T17:57:20.374+0000 7f6b9ef03500 -1 osd.4 0 log_to_monitors true 2024-01-15T17:57:20.401 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-01-15T17:57:20.399+0000 7f6838610500 -1 osd.2 0 log_to_monitors true 2024-01-15T17:57:22.937 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-01-15T17:57:22.933+0000 7fa9a1000640 -1 osd.3 0 waiting for initial osdmap 2024-01-15T17:57:22.942 INFO:tasks.ceph.osd.3.smithi002.stderr:2024-01-15T17:57:22.939+0000 7fa997a00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:22.947 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-01-15T17:57:22.945+0000 7f0d54400640 -1 osd.0 0 waiting for initial osdmap 2024-01-15T17:57:22.957 INFO:tasks.ceph.osd.0.smithi002.stderr:2024-01-15T17:57:22.955+0000 7f0d4ae00640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:23.003 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-01-15T17:57:23.000+0000 7ff1ad000640 -1 osd.1 0 waiting for initial osdmap 2024-01-15T17:57:23.009 INFO:tasks.ceph.osd.1.smithi002.stderr:2024-01-15T17:57:23.007+0000 7ff1a1c00640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:23.455 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-15T17:57:23.453+0000 7fd295e00640 -1 osd.7 0 waiting for initial osdmap 2024-01-15T17:57:23.474 INFO:tasks.ceph.osd.7.smithi137.stderr:2024-01-15T17:57:23.471+0000 7fd28c800640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:23.487 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-15T17:57:23.485+0000 7f383f400640 -1 osd.5 0 waiting for initial osdmap 2024-01-15T17:57:23.494 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-15T17:57:23.492+0000 7f380de00640 -1 osd.6 0 waiting for initial osdmap 2024-01-15T17:57:23.497 INFO:tasks.ceph.osd.5.smithi137.stderr:2024-01-15T17:57:23.494+0000 7f3834000640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:23.501 INFO:tasks.ceph.osd.6.smithi137.stderr:2024-01-15T17:57:23.499+0000 7f3804800640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:23.652 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-01-15T17:57:23.649+0000 7f6832400640 -1 osd.2 0 waiting for initial osdmap 2024-01-15T17:57:23.660 INFO:tasks.ceph.osd.2.smithi002.stderr:2024-01-15T17:57:23.657+0000 7f6828e00640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:23.845 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-15T17:57:23.842+0000 7f6b99400640 -1 osd.4 0 waiting for initial osdmap 2024-01-15T17:57:23.853 INFO:tasks.ceph.osd.4.smithi137.stderr:2024-01-15T17:57:23.850+0000 7f6b8fe00640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-15T17:57:24.051 DEBUG:teuthology.orchestra.run.smithi002:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:24.240 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.237+0000 7fe25fe00640 1 Processor -- start 2024-01-15T17:57:24.240 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.237+0000 7fe25fe00640 1 -- start start 2024-01-15T17:57:24.242 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe26006d710 0x7fe2600719a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.242 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe26014b1e0 0x7fe2600cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2600b2a90 con 0x7fe260070fa0 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25fe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2600b2f20 con 0x7fe26014b1e0 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2600b1e80 con 0x7fe26006d710 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe26014b1e0 0x7fe2600cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe26014b1e0 0x7fe2600cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39668/0 (socket says 172.21.15.2:39668) 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25f400640 1 -- 172.21.15.2:0/886493613 learned_addr learned my addr 172.21.15.2:0/886493613 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.239+0000 7fe25ea00640 1 --2- 172.21.15.2:0/886493613 >> v2:172.21.15.137:3300/0 conn(0x7fe26006d710 0x7fe2600719a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.240+0000 7fe25f400640 1 -- 172.21.15.2:0/886493613 >> v2:172.21.15.137:3300/0 conn(0x7fe26006d710 msgr2=0x7fe2600719a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.240+0000 7fe25f400640 1 --2- 172.21.15.2:0/886493613 >> v2:172.21.15.137:3300/0 conn(0x7fe26006d710 0x7fe2600719a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.240+0000 7fe25f400640 1 -- 172.21.15.2:0/886493613 >> v1:172.21.15.137:6789/0 conn(0x7fe260070fa0 legacy=0x7fe26014aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:24.243 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.240+0000 7fe25f400640 1 -- 172.21.15.2:0/886493613 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2600b2160 con 0x7fe26014b1e0 2024-01-15T17:57:24.244 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.241+0000 7fe25f400640 1 --2- 172.21.15.2:0/886493613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26014b1e0 0x7fe2600cc590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe26006fe90 tx=0x7fe25802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df8624f241d3501 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:24.244 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.241+0000 7fe25d600640 1 -- 172.21.15.2:0/886493613 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258031870 con 0x7fe26014b1e0 2024-01-15T17:57:24.244 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.241+0000 7fe25d600640 1 -- 172.21.15.2:0/886493613 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2580319d0 con 0x7fe26014b1e0 2024-01-15T17:57:24.244 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.241+0000 7fe25d600640 1 -- 172.21.15.2:0/886493613 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe258031d60 con 0x7fe26014b1e0 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 -- 172.21.15.2:0/886493613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26014b1e0 msgr2=0x7fe2600cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 --2- 172.21.15.2:0/886493613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26014b1e0 0x7fe2600cc590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe26006fe90 tx=0x7fe25802b910 comp rx=0 tx=0).stop 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 -- 172.21.15.2:0/886493613 shutdown_connections 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 --2- 172.21.15.2:0/886493613 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26014b1e0 0x7fe2600cc590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 --2- 172.21.15.2:0/886493613 >> v2:172.21.15.137:3300/0 conn(0x7fe26006d710 0x7fe2600719a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 -- 172.21.15.2:0/886493613 >> 172.21.15.2:0/886493613 conn(0x7fe2600b0b80 msgr2=0x7fe2600b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 -- 172.21.15.2:0/886493613 shutdown_connections 2024-01-15T17:57:24.245 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.242+0000 7fe25fe00640 1 -- 172.21.15.2:0/886493613 wait complete. 2024-01-15T17:57:24.247 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.244+0000 7fe25fe00640 1 Processor -- start 2024-01-15T17:57:24.247 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.244+0000 7fe25fe00640 1 -- start start 2024-01-15T17:57:24.249 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 0x7fe26013efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.249 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe260070fa0 0x7fe260141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26014b1e0 0x7fe260140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe260141580 con 0x7fe26006d710 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe260140a80 con 0x7fe260070fa0 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe260161e10 con 0x7fe26014b1e0 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 0x7fe26013efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 0x7fe26013efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39672/0 (socket says 172.21.15.2:39672) 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.245+0000 7fe25ea00640 1 -- 172.21.15.2:0/3259249852 learned_addr learned my addr 172.21.15.2:0/3259249852 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25e000640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe260070fa0 0x7fe260141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25ea00640 1 -- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26014b1e0 msgr2=0x7fe260140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25ea00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26014b1e0 0x7fe260140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25ea00640 1 -- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe260070fa0 msgr2=0x7fe260141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25ea00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe260070fa0 0x7fe260141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25ea00640 1 -- 172.21.15.2:0/3259249852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2580094d0 con 0x7fe26006d710 2024-01-15T17:57:24.250 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25e000640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe260070fa0 0x7fe260141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25ea00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 0x7fe26013efd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe250007e70 tx=0x7fe25000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250014620 con 0x7fe26006d710 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe250014780 con 0x7fe26006d710 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe250014b30 con 0x7fe26006d710 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2601620f0 con 0x7fe26006d710 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.246+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2601626b0 con 0x7fe26006d710 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.247+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe260162870 con 0x7fe26006d710 2024-01-15T17:57:24.251 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.248+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fe250014c90 con 0x7fe26006d710 2024-01-15T17:57:24.252 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.249+0000 7fe256a00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23806b260 0x7fe23806d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.252 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.249+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fe250089fb0 con 0x7fe26006d710 2024-01-15T17:57:24.252 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.250+0000 7fe25e000640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23806b260 0x7fe23806d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.254 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.251+0000 7fe25e000640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23806b260 0x7fe23806d720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe2601faad0 tx=0x7fe248009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:24.257 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.254+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe250055330 con 0x7fe26006d710 2024-01-15T17:57:24.443 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.440+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe26004e840 con 0x7fe26006d710 2024-01-15T17:57:24.445 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.442+0000 7fe256a00640 1 -- 172.21.15.2:0/3259249852 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10848 (secure 0 0 0) 0x7fe250058950 con 0x7fe26006d710 2024-01-15T17:57:24.446 INFO:teuthology.misc.health.smithi002.stdout: 2024-01-15T17:57:24.446 INFO:teuthology.misc.health.smithi002.stdout:{"epoch":14,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:23.931822+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.445+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23806b260 msgr2=0x7fe23806d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.445+0000 7fe25fe00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23806b260 0x7fe23806d720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe2601faad0 tx=0x7fe248009040 comp rx=0 tx=0).stop 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.445+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 msgr2=0x7fe26013efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.445+0000 7fe25fe00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 0x7fe26013efd0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe250007e70 tx=0x7fe25000e4b0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 shutdown_connections 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23806b260 0x7fe23806d720 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26014b1e0 0x7fe260140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe260070fa0 0x7fe260141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 --2- 172.21.15.2:0/3259249852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26006d710 0x7fe26013efd0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 >> 172.21.15.2:0/3259249852 conn(0x7fe2600b0b80 msgr2=0x7fe26014aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 shutdown_connections 2024-01-15T17:57:24.449 INFO:teuthology.misc.health.smithi002.stderr:2024-01-15T17:57:24.446+0000 7fe25fe00640 1 -- 172.21.15.2:0/3259249852 wait complete. 2024-01-15T17:57:24.462 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-15T17:57:24.462 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-15T17:57:24.462 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T17:57:24.462 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T17:57:24.467 INFO:tasks.ceph.mds.a:Started 2024-01-15T17:57:24.468 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T17:57:24.468 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T17:57:24.474 INFO:tasks.ceph.mds.c:Started 2024-01-15T17:57:24.474 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T17:57:24.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T17:57:24.482 INFO:tasks.ceph.mds.b:Started 2024-01-15T17:57:24.483 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T17:57:24.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T17:57:24.490 INFO:tasks.ceph.mds.d:Started 2024-01-15T17:57:24.490 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-15T17:57:24.492 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:24.618 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T17:57:24.622 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T17:57:24.638 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T17:57:24.642 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T17:57:24.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.675+0000 7f286de00640 1 Processor -- start 2024-01-15T17:57:24.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.675+0000 7f286de00640 1 -- start start 2024-01-15T17:57:24.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f286de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f28680b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f286de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f286806cb60 0x7f28680b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f286de00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f286806bac0 con 0x7f2868145f30 2024-01-15T17:57:24.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f286de00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286806bda0 con 0x7f286806cb60 2024-01-15T17:57:24.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f286de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286806c080 con 0x7f286806b5c0 2024-01-15T17:57:24.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f2866a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2868145f30 0x7f2868146830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53596/0 (socket says 172.21.15.2:53596) 2024-01-15T17:57:24.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f2866a00640 1 -- 172.21.15.2:0/1220928272 learned_addr learned my addr 172.21.15.2:0/1220928272 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f2867400640 1 --2- 172.21.15.2:0/1220928272 >> v2:172.21.15.137:3300/0 conn(0x7f286806cb60 0x7f28680b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f285de00640 1 --2- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f28680b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f2866000640 1 -- 172.21.15.2:0/1220928272 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290048280 0 0) 0x7f286806bac0 con 0x7f2868145f30 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.677+0000 7f2866000640 1 -- 172.21.15.2:0/1220928272 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2848003520 con 0x7f2868145f30 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.678+0000 7f2867400640 1 -- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 msgr2=0x7f28680b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.678+0000 7f2867400640 1 --2- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f28680b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.678+0000 7f2867400640 1 -- 172.21.15.2:0/1220928272 >> v1:172.21.15.2:6789/0 conn(0x7f2868145f30 legacy=0x7f2868146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.678+0000 7f2867400640 1 -- 172.21.15.2:0/1220928272 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286806c360 con 0x7f286806cb60 2024-01-15T17:57:24.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.679+0000 7f2867400640 1 --2- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f286806cb60 0x7f28680b3490 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f28600097a0 tx=0x7f286002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4262389b735cf87e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.679+0000 7f2866000640 1 -- 172.21.15.2:0/1220928272 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286002d040 con 0x7f286806cb60 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.679+0000 7f2866000640 1 -- 172.21.15.2:0/1220928272 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2860031810 con 0x7f286806cb60 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.679+0000 7f2866000640 1 -- 172.21.15.2:0/1220928272 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2860031ba0 con 0x7f286806cb60 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.679+0000 7f286de00640 1 -- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f286806cb60 msgr2=0x7f28680b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 --2- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f286806cb60 0x7f28680b3490 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f28600097a0 tx=0x7f286002b8b0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 -- 172.21.15.2:0/1220928272 shutdown_connections 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 --2- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f286806cb60 0x7f28680b3490 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 --2- 172.21.15.2:0/1220928272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f28680b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 -- 172.21.15.2:0/1220928272 >> 172.21.15.2:0/1220928272 conn(0x7f28680663d0 msgr2=0x7f28680667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 -- 172.21.15.2:0/1220928272 shutdown_connections 2024-01-15T17:57:24.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.680+0000 7f286de00640 1 -- 172.21.15.2:0/1220928272 wait complete. 2024-01-15T17:57:24.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.682+0000 7f286de00640 1 Processor -- start 2024-01-15T17:57:24.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.682+0000 7f286de00640 1 -- start start 2024-01-15T17:57:24.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f286de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f286815c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f286de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 0x7f2868143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f286de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2868145f30 0x7f28681443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f286de00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28680ce670 con 0x7f286806cb60 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f286de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28680ce910 con 0x7f2868145f30 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f286de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28680cebf0 con 0x7f286806b5c0 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f285de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 0x7f2868143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f285de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 0x7f2868143eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39710/0 (socket says 172.21.15.2:39710) 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f285de00640 1 -- 172.21.15.2:0/4053518191 learned_addr learned my addr 172.21.15.2:0/4053518191 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f2866a00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f286815c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.683+0000 7f2867400640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2868145f30 0x7f28681443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285de00640 1 -- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 msgr2=0x7f286815c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f286815c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285de00640 1 -- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2868145f30 msgr2=0x7f28681443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2868145f30 0x7f28681443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285de00640 1 -- 172.21.15.2:0/4053518191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2860009400 con 0x7f286806cb60 2024-01-15T17:57:24.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 0x7f2868143eb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f285000e9b0 tx=0x7f285000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850009670 con 0x7f286806cb60 2024-01-15T17:57:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f285000c9a0 con 0x7f286806cb60 2024-01-15T17:57:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f285000cd50 con 0x7f286806cb60 2024-01-15T17:57:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28680ceed0 con 0x7f286806cb60 2024-01-15T17:57:24.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.684+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286815ce20 con 0x7f286806cb60 2024-01-15T17:57:24.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.685+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28680686f0 con 0x7f286806cb60 2024-01-15T17:57:24.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.686+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f2850013470 con 0x7f286806cb60 2024-01-15T17:57:24.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.687+0000 7f285fe00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f284006b230 0x7f284006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:24.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.687+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f2850089820 con 0x7f286806cb60 2024-01-15T17:57:24.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.688+0000 7f2866a00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f284006b230 0x7f284006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:24.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.688+0000 7f2866a00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f284006b230 0x7f284006d6f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2858005c00 tx=0x7f2858009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:24.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.691+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2850054c50 con 0x7f286806cb60 2024-01-15T17:57:24.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.893+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28681602a0 con 0x7f286806cb60 2024-01-15T17:57:24.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.895+0000 7f285fe00640 1 -- 172.21.15.2:0/4053518191 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10848 (secure 0 0 0) 0x7f2850017090 con 0x7f286806cb60 2024-01-15T17:57:24.897 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:24.898 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":14,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:23.931822+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:24.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f284006b230 msgr2=0x7f284006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f284006b230 0x7f284006d6f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2858005c00 tx=0x7f2858009040 comp rx=0 tx=0).stop 2024-01-15T17:57:24.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 msgr2=0x7f2868143eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:24.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 0x7f2868143eb0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f285000e9b0 tx=0x7f285000ee70 comp rx=0 tx=0).stop 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 shutdown_connections 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f284006b230 0x7f284006d6f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2868145f30 0x7f28681443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f286806cb60 0x7f2868143eb0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 --2- 172.21.15.2:0/4053518191 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f286806b5c0 0x7f286815c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 >> 172.21.15.2:0/4053518191 conn(0x7f28680663d0 msgr2=0x7f28680667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.897+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 shutdown_connections 2024-01-15T17:57:24.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:24.898+0000 7f286de00640 1 -- 172.21.15.2:0/4053518191 wait complete. 2024-01-15T17:57:24.918 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-01-15T17:57:24.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:57:25.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.135+0000 7f58da800640 1 Processor -- start 2024-01-15T17:57:25.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.136+0000 7f58da800640 1 -- start start 2024-01-15T17:57:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.137+0000 7f58da800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 0x7f58d4166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.137+0000 7f58da800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d415cb00 con 0x7f58d415b7d0 2024-01-15T17:57:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.137+0000 7f58da800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d40ccec0 con 0x7f58d415c0e0 2024-01-15T17:57:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.137+0000 7f58da800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d40cccd0 con 0x7f58d406cb80 2024-01-15T17:57:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.137+0000 7f58d2a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f58d415b7d0 0x7f58d406c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53604/0 (socket says 172.21.15.2:53604) 2024-01-15T17:57:25.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.137+0000 7f58d2a00640 1 -- 172.21.15.2:0/3365510042 learned_addr learned my addr 172.21.15.2:0/3365510042 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d3e00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 0x7f58d4166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292407860 0 0) 0x7f58d415cb00 con 0x7f58d415b7d0 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b4003520 con 0x7f58d415b7d0 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857802312 0 0) 0x7f58b4003520 con 0x7f58d415b7d0 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 msgr2=0x7f58d4166070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 0x7f58d4166070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 >> v1:172.21.15.137:6789/0 conn(0x7f58d415c0e0 legacy=0x7f58d40cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d40cd1a0 con 0x7f58d415b7d0 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f58bc001670 con 0x7f58d415b7d0 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 >> v1:172.21.15.2:6789/0 conn(0x7f58d415b7d0 legacy=0x7f58d406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 0x7f58b40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58b4003f70 0x7f58b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b400ba90 con 0x7f58b4004d00 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b400bc00 con 0x7f58b4003f70 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.138+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b400bdc0 con 0x7f58d406cb80 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3e00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3400640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 0x7f58b40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d2a00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58b4003f70 0x7f58b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3e00640 1 -- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 msgr2=0x7f58b40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3e00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d406cb80 0x7f58b40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3e00640 1 -- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58b4003f70 msgr2=0x7f58b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3e00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58b4003f70 0x7f58b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d3e00640 1 -- 172.21.15.2:0/3365510042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4003520 con 0x7f58b4004d00 2024-01-15T17:57:25.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.139+0000 7f58d2a00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58b4003f70 0x7f58b40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.140+0000 7f58d3e00640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58b4005180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009fc0 tx=0x7f58b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc15d42249e06bb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.140+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc02cc20 con 0x7f58b4004d00 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.140+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58cc00f040 con 0x7f58b4004d00 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.140+0000 7f58d2000640 1 -- 172.21.15.2:0/3365510042 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58cc034440 con 0x7f58b4004d00 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58da800640 1 -- 172.21.15.2:0/3365510042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 msgr2=0x7f58b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58da800640 1 --2- 172.21.15.2:0/3365510042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58b4005180 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009fc0 tx=0x7f58b40057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58d3e00640 1 -- 172.21.15.2:0/3365510042 reap_dead start 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58da800640 1 -- 172.21.15.2:0/3365510042 shutdown_connections 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58da800640 1 -- 172.21.15.2:0/3365510042 >> 172.21.15.2:0/3365510042 conn(0x7f58d406da20 msgr2=0x7f58d406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:25.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58da800640 1 -- 172.21.15.2:0/3365510042 shutdown_connections 2024-01-15T17:57:25.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.141+0000 7f58da800640 1 -- 172.21.15.2:0/3365510042 wait complete. 2024-01-15T17:57:25.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.143+0000 7f58da800640 1 Processor -- start 2024-01-15T17:57:25.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.143+0000 7f58da800640 1 -- start start 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58da800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58d413c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58da800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58d4138690 0x7f58d4138b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58da800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d4136600 0x7f58d4136a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58da800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d4137ff0 con 0x7f58b4004d00 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58da800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d4139050 con 0x7f58d4138690 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58da800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d413cbc0 con 0x7f58d4136600 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58d3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58d413c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58d3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58d413c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39724/0 (socket says 172.21.15.2:39724) 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.144+0000 7f58d3400640 1 -- 172.21.15.2:0/4211112125 learned_addr learned my addr 172.21.15.2:0/4211112125 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3e00640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d4136600 0x7f58d4136a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3400640 1 -- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d4136600 msgr2=0x7f58d4136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3400640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d4136600 0x7f58d4136a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d2a00640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58d4138690 0x7f58d4138b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3400640 1 -- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58d4138690 msgr2=0x7f58d4138b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3400640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58d4138690 0x7f58d4138b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3400640 1 -- 172.21.15.2:0/4211112125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4138410 con 0x7f58b4004d00 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d2a00640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58d4138690 0x7f58d4138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58d3400640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58d413c680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e580 tx=0x7f58c400ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c40052e0 con 0x7f58b4004d00 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c4005440 con 0x7f58b4004d00 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4003810 con 0x7f58b4004d00 2024-01-15T17:57:25.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.145+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d4137370 con 0x7f58b4004d00 2024-01-15T17:57:25.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.146+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d4137930 con 0x7f58b4004d00 2024-01-15T17:57:25.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.147+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d40686f0 con 0x7f58b4004d00 2024-01-15T17:57:25.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.147+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f58c4003970 con 0x7f58b4004d00 2024-01-15T17:57:25.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.148+0000 7f58cb400640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f58ac06b230 0x7f58ac06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:25.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.148+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f58c40893f0 con 0x7f58b4004d00 2024-01-15T17:57:25.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.149+0000 7f58d2a00640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f58ac06b230 0x7f58ac06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:25.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.150+0000 7f58d2a00640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f58ac06b230 0x7f58ac06d6f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f58bc001eb0 tx=0x7f58bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:25.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.153+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58c4054780 con 0x7f58b4004d00 2024-01-15T17:57:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.385+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58d406b260 con 0x7f58b4004d00 2024-01-15T17:57:25.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.386+0000 7f58cb400640 1 -- 172.21.15.2:0/4211112125 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1321 (secure 0 0 0) 0x7f58c4089030 con 0x7f58b4004d00 2024-01-15T17:57:25.389 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:25.389 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-15T17:57:25.389 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 2 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f58ac06b230 msgr2=0x7f58ac06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f58ac06b230 0x7f58ac06d6f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f58bc001eb0 tx=0x7f58bc009040 comp rx=0 tx=0).stop 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 msgr2=0x7f58d413c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58d413c680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f58c400e580 tx=0x7f58c400ea40 comp rx=0 tx=0).stop 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 shutdown_connections 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f58ac06b230 0x7f58ac06d6f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58d4136600 0x7f58d4136a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58d4138690 0x7f58d4138b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 --2- 172.21.15.2:0/4211112125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58b4004d00 0x7f58d413c680 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 >> 172.21.15.2:0/4211112125 conn(0x7f58d406da20 msgr2=0x7f58d415ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 shutdown_connections 2024-01-15T17:57:25.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:25.389+0000 7f58da800640 1 -- 172.21.15.2:0/4211112125 wait complete. 2024-01-15T17:57:27.412 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:57:27.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.620+0000 7f5a89a00640 1 Processor -- start 2024-01-15T17:57:27.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.620+0000 7f5a89a00640 1 -- start start 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a89a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a840d15e0 0x7f5a840b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a89a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5a840d1a70 0x7f5a840b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a89a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a840d06f0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a89a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a840d3e50 con 0x7f5a840d15e0 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a89a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a840d3b70 con 0x7f5a840d1a70 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a83400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a840d15e0 0x7f5a840b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a83400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a840d15e0 0x7f5a840b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39726/0 (socket says 172.21.15.2:39726) 2024-01-15T17:57:27.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a83400640 1 -- 172.21.15.2:0/3875410529 learned_addr learned my addr 172.21.15.2:0/3875410529 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a89000640 1 --2- 172.21.15.2:0/3875410529 >> v2:172.21.15.137:3300/0 conn(0x7f5a840d1a70 0x7f5a840b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a83400640 1 -- 172.21.15.2:0/3875410529 >> v2:172.21.15.137:3300/0 conn(0x7f5a840d1a70 msgr2=0x7f5a840b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.622+0000 7f5a83400640 1 --2- 172.21.15.2:0/3875410529 >> v2:172.21.15.137:3300/0 conn(0x7f5a840d1a70 0x7f5a840b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.623+0000 7f5a83400640 1 -- 172.21.15.2:0/3875410529 >> v1:172.21.15.137:6789/0 conn(0x7f5a8415d3e0 legacy=0x7f5a840d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.623+0000 7f5a83400640 1 -- 172.21.15.2:0/3875410529 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a840ce800 con 0x7f5a840d15e0 2024-01-15T17:57:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.624+0000 7f5a83400640 1 --2- 172.21.15.2:0/3875410529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a840d15e0 0x7f5a840b2bc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5a8408ca40 tx=0x7f5a6c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23621ac56e103a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:27.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.624+0000 7f5a82a00640 1 -- 172.21.15.2:0/3875410529 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c0317e0 con 0x7f5a840d15e0 2024-01-15T17:57:27.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.624+0000 7f5a82a00640 1 -- 172.21.15.2:0/3875410529 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a6c031940 con 0x7f5a840d15e0 2024-01-15T17:57:27.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.624+0000 7f5a82a00640 1 -- 172.21.15.2:0/3875410529 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c031cd0 con 0x7f5a840d15e0 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 -- 172.21.15.2:0/3875410529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a840d15e0 msgr2=0x7f5a840b2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 --2- 172.21.15.2:0/3875410529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a840d15e0 0x7f5a840b2bc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5a8408ca40 tx=0x7f5a6c02b880 comp rx=0 tx=0).stop 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 -- 172.21.15.2:0/3875410529 shutdown_connections 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 --2- 172.21.15.2:0/3875410529 >> v2:172.21.15.137:3300/0 conn(0x7f5a840d1a70 0x7f5a840b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 --2- 172.21.15.2:0/3875410529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a840d15e0 0x7f5a840b2bc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 -- 172.21.15.2:0/3875410529 >> 172.21.15.2:0/3875410529 conn(0x7f5a8409c3d0 msgr2=0x7f5a840d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 -- 172.21.15.2:0/3875410529 shutdown_connections 2024-01-15T17:57:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.625+0000 7f5a89a00640 1 -- 172.21.15.2:0/3875410529 wait complete. 2024-01-15T17:57:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.627+0000 7f5a89a00640 1 Processor -- start 2024-01-15T17:57:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.627+0000 7f5a89a00640 1 -- start start 2024-01-15T17:57:27.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.628+0000 7f5a89a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a840d15e0 0x7f5a841576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:27.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.628+0000 7f5a89a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a840d1a70 0x7f5a841e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:27.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.628+0000 7f5a89a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 0x7f5a8415da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:27.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.628+0000 7f5a89a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840b60f0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.628+0000 7f5a89a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840b63d0 con 0x7f5a840d1a70 2024-01-15T17:57:27.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.628+0000 7f5a89a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a840b66b0 con 0x7f5a840d15e0 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 0x7f5a8415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 0x7f5a8415da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39732/0 (socket says 172.21.15.2:39732) 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 -- 172.21.15.2:0/2868620035 learned_addr learned my addr 172.21.15.2:0/2868620035 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a83400640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a840d1a70 0x7f5a841e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a83e00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a840d15e0 0x7f5a841576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 -- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a840d15e0 msgr2=0x7f5a841576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a840d15e0 0x7f5a841576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 -- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a840d1a70 msgr2=0x7f5a841e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a840d1a70 0x7f5a841e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a89000640 1 -- 172.21.15.2:0/2868620035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0094d0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.629+0000 7f5a83e00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a840d15e0 0x7f5a841576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:27.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.630+0000 7f5a89000640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 0x7f5a8415da90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c007e60 tx=0x7f5a7c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.630+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c014420 con 0x7f5a8415d3e0 2024-01-15T17:57:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.630+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7c0145a0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.630+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c014950 con 0x7f5a8415d3e0 2024-01-15T17:57:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.630+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a840b6990 con 0x7f5a8415d3e0 2024-01-15T17:57:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.630+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a840ca740 con 0x7f5a8415d3e0 2024-01-15T17:57:27.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.631+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a840b6eb0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.632+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f5a7c014ab0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.633+0000 7f5a7be00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a5c06b230 0x7f5a5c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.634+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5a7c08a150 con 0x7f5a8415d3e0 2024-01-15T17:57:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.634+0000 7f5a83e00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a5c06b230 0x7f5a5c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:27.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.635+0000 7f5a83e00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a5c06b230 0x7f5a5c06d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5a74007fd0 tx=0x7f5a74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:27.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.639+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a840b6eb0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.880+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a840b6eb0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.882+0000 7f5a7be00640 1 -- 172.21.15.2:0/2868620035 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3694 (secure 0 0 0) 0x7f5a840b6eb0 con 0x7f5a8415d3e0 2024-01-15T17:57:27.885 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:27.885 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4233,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-01-15T17:57:27.885 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 3 2024-01-15T17:57:27.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a5c06b230 msgr2=0x7f5a5c06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:27.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a5c06b230 0x7f5a5c06d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5a74007fd0 tx=0x7f5a74009040 comp rx=0 tx=0).stop 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 msgr2=0x7f5a8415da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 0x7f5a8415da90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c007e60 tx=0x7f5a7c00e4b0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 shutdown_connections 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a5c06b230 0x7f5a5c06d6f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a8415d3e0 0x7f5a8415da90 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a840d1a70 0x7f5a841e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 --2- 172.21.15.2:0/2868620035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a840d15e0 0x7f5a841576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.885+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 >> 172.21.15.2:0/2868620035 conn(0x7f5a8409c3d0 msgr2=0x7f5a840d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.886+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 shutdown_connections 2024-01-15T17:57:27.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:27.886+0000 7f5a89a00640 1 -- 172.21.15.2:0/2868620035 wait complete. 2024-01-15T17:57:27.913 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:28.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.106+0000 7fbc29e00640 1 Processor -- start 2024-01-15T17:57:28.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.106+0000 7fbc29e00640 1 -- start start 2024-01-15T17:57:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.107+0000 7fbc29e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbc240ce480 0x7fbc24167fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.107+0000 7fbc29e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc240ce930 con 0x7fbc2415d670 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc29e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc240cec10 con 0x7fbc240d2f10 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc29e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc240ceef0 con 0x7fbc240ce480 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc23400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbc240ce480 0x7fbc24167fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc23400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbc240ce480 0x7fbc24167fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39740/0 (socket says 172.21.15.2:39740) 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc23400640 1 -- 172.21.15.2:0/3279107158 learned_addr learned my addr 172.21.15.2:0/3279107158 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc1be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbc2415d670 0x7fbc240cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53610/0 (socket says 172.21.15.2:53610) 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc23400640 1 -- 172.21.15.2:0/3279107158 >> v1:172.21.15.137:6789/0 conn(0x7fbc240d2f10 legacy=0x7fbc240d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc23400640 1 -- 172.21.15.2:0/3279107158 >> v1:172.21.15.2:6789/0 conn(0x7fbc2415d670 legacy=0x7fbc240cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.108+0000 7fbc23400640 1 -- 172.21.15.2:0/3279107158 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc240cf1d0 con 0x7fbc240ce480 2024-01-15T17:57:28.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.110+0000 7fbc23400640 1 --2- 172.21.15.2:0/3279107158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240ce480 0x7fbc24167fa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009e50 tx=0x7fbc1c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb85bf538594cc5c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.110+0000 7fbc22000640 1 -- 172.21.15.2:0/3279107158 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c02cb10 con 0x7fbc240ce480 2024-01-15T17:57:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.110+0000 7fbc22000640 1 -- 172.21.15.2:0/3279107158 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc1c02cc70 con 0x7fbc240ce480 2024-01-15T17:57:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.110+0000 7fbc22000640 1 -- 172.21.15.2:0/3279107158 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0343f0 con 0x7fbc240ce480 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.110+0000 7fbc29e00640 1 -- 172.21.15.2:0/3279107158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240ce480 msgr2=0x7fbc24167fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.110+0000 7fbc29e00640 1 --2- 172.21.15.2:0/3279107158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240ce480 0x7fbc24167fa0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c009e50 tx=0x7fbc1c02bf70 comp rx=0 tx=0).stop 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.111+0000 7fbc29e00640 1 -- 172.21.15.2:0/3279107158 shutdown_connections 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.111+0000 7fbc29e00640 1 --2- 172.21.15.2:0/3279107158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240ce480 0x7fbc24167fa0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.111+0000 7fbc29e00640 1 -- 172.21.15.2:0/3279107158 >> 172.21.15.2:0/3279107158 conn(0x7fbc2408cc30 msgr2=0x7fbc2409c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.111+0000 7fbc29e00640 1 -- 172.21.15.2:0/3279107158 shutdown_connections 2024-01-15T17:57:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.111+0000 7fbc29e00640 1 -- 172.21.15.2:0/3279107158 wait complete. 2024-01-15T17:57:28.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.113+0000 7fbc29e00640 1 Processor -- start 2024-01-15T17:57:28.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.113+0000 7fbc29e00640 1 -- start start 2024-01-15T17:57:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc29e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc240ce480 0x7fbc240b8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc29e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 0x7fbc240cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc29e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc241f2f20 0x7fbc241f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc29e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc240cc2d0 con 0x7fbc240d2f10 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc29e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc240cc530 con 0x7fbc240ce480 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc29e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc240cc810 con 0x7fbc241f2f20 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 0x7fbc240cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc1be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 0x7fbc240cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39742/0 (socket says 172.21.15.2:39742) 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc1be00640 1 -- 172.21.15.2:0/2836764022 learned_addr learned my addr 172.21.15.2:0/2836764022 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc22a00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc240ce480 0x7fbc240b8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.114+0000 7fbc23400640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc241f2f20 0x7fbc241f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1be00640 1 -- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc241f2f20 msgr2=0x7fbc241f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1be00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc241f2f20 0x7fbc241f33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1be00640 1 -- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc240ce480 msgr2=0x7fbc240b8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1be00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc240ce480 0x7fbc240b8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1be00640 1 -- 172.21.15.2:0/2836764022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c009b30 con 0x7fbc240d2f10 2024-01-15T17:57:28.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1be00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 0x7fbc240cbd90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c3e0 tx=0x7fbc0c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:28.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c0074e0 con 0x7fbc240d2f10 2024-01-15T17:57:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc0c007640 con 0x7fbc240d2f10 2024-01-15T17:57:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.115+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0c0049e0 con 0x7fbc240d2f10 2024-01-15T17:57:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.116+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc240ccaf0 con 0x7fbc240d2f10 2024-01-15T17:57:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.116+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc240b1570 con 0x7fbc240d2f10 2024-01-15T17:57:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.117+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc240b1730 con 0x7fbc240d2f10 2024-01-15T17:57:28.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.119+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7fbc0c004b40 con 0x7fbc240d2f10 2024-01-15T17:57:28.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.120+0000 7fbc1b400640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbbfc06b230 0x7fbbfc06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.120+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbc0c089140 con 0x7fbc240d2f10 2024-01-15T17:57:28.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.121+0000 7fbc22a00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbbfc06b230 0x7fbbfc06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.122+0000 7fbc22a00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbbfc06b230 0x7fbbfc06d6f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbc14004ea0 tx=0x7fbc14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:28.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.125+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc0c0543d0 con 0x7fbc240d2f10 2024-01-15T17:57:28.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.313+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbc0c0547a0 con 0x7fbc240d2f10 2024-01-15T17:57:28.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.333+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc240b1b00 con 0x7fbc240d2f10 2024-01-15T17:57:28.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.334+0000 7fbc1b400640 1 -- 172.21.15.2:0/2836764022 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12790 (secure 0 0 0) 0x7fbc240b1b00 con 0x7fbc240d2f10 2024-01-15T17:57:28.337 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:28.337 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":18,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:27.952585+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:28.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbbfc06b230 msgr2=0x7fbbfc06d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbbfc06b230 0x7fbbfc06d6f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbc14004ea0 tx=0x7fbc14009040 comp rx=0 tx=0).stop 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 msgr2=0x7fbc240cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 0x7fbc240cbd90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbc0c00c3e0 tx=0x7fbc0c00c8a0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 shutdown_connections 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbbfc06b230 0x7fbbfc06d6f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc241f2f20 0x7fbc241f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc240d2f10 0x7fbc240cbd90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 --2- 172.21.15.2:0/2836764022 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc240ce480 0x7fbc240b8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.336+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 >> 172.21.15.2:0/2836764022 conn(0x7fbc2408cc30 msgr2=0x7fbc240b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.337+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 shutdown_connections 2024-01-15T17:57:28.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.337+0000 7fbc29e00640 1 -- 172.21.15.2:0/2836764022 wait complete. 2024-01-15T17:57:28.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:57:28.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:57:28.471 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:57:28.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:57:28.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.556+0000 7f5b8f000640 1 Processor -- start 2024-01-15T17:57:28.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.556+0000 7f5b8f000640 1 -- start start 2024-01-15T17:57:28.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.557+0000 7f5b8f000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5b8806d000 0x7f5b88165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b8f000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880ce230 con 0x7f5b8815c450 2024-01-15T17:57:28.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b8f000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b880ce510 con 0x7f5b88144c00 2024-01-15T17:57:28.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b8f000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b880ce7f0 con 0x7f5b8806d000 2024-01-15T17:57:28.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b7fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5b8815c450 0x7f5b8806c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53616/0 (socket says 172.21.15.2:53616) 2024-01-15T17:57:28.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b7fe00640 1 -- 172.21.15.2:0/3960192622 learned_addr learned my addr 172.21.15.2:0/3960192622 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:28.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b8d000640 1 --2- 172.21.15.2:0/3960192622 >> v2:172.21.15.137:3300/0 conn(0x7f5b8806d000 0x7f5b88165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824529238 0 0) 0x7f5b880ce230 con 0x7f5b8815c450 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b68003520 con 0x7f5b8815c450 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223417676 0 0) 0x7f5b880ce510 con 0x7f5b88144c00 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.558+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b880ce230 con 0x7f5b88144c00 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.559+0000 7f5b8d000640 1 -- 172.21.15.2:0/3960192622 >> v1:172.21.15.137:6789/0 conn(0x7f5b88144c00 legacy=0x7f5b880cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.559+0000 7f5b8d000640 1 -- 172.21.15.2:0/3960192622 >> v1:172.21.15.2:6789/0 conn(0x7f5b8815c450 legacy=0x7f5b8806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.559+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3360600990 0 0) 0x7f5b68003520 con 0x7f5b8815c450 2024-01-15T17:57:28.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.559+0000 7f5b8d000640 1 -- 172.21.15.2:0/3960192622 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b880705f0 con 0x7f5b8806d000 2024-01-15T17:57:28.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.560+0000 7f5b8d000640 1 --2- 172.21.15.2:0/3960192622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8806d000 0x7f5b88165d60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009d90 tx=0x7f5b8002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=842e5530eafec4a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:28.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b70001670 con 0x7f5b8806d000 2024-01-15T17:57:28.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8002c940 con 0x7f5b8806d000 2024-01-15T17:57:28.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b87400640 1 -- 172.21.15.2:0/3960192622 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8002ccd0 con 0x7f5b8806d000 2024-01-15T17:57:28.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b8f000640 1 -- 172.21.15.2:0/3960192622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8806d000 msgr2=0x7f5b88165d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b8f000640 1 --2- 172.21.15.2:0/3960192622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8806d000 0x7f5b88165d60 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5b80009d90 tx=0x7f5b8002be10 comp rx=0 tx=0).stop 2024-01-15T17:57:28.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b8f000640 1 -- 172.21.15.2:0/3960192622 shutdown_connections 2024-01-15T17:57:28.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b8f000640 1 --2- 172.21.15.2:0/3960192622 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b8806d000 0x7f5b88165d60 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.561+0000 7f5b8f000640 1 -- 172.21.15.2:0/3960192622 >> 172.21.15.2:0/3960192622 conn(0x7f5b880b45f0 msgr2=0x7f5b880b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:28.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.562+0000 7f5b8f000640 1 -- 172.21.15.2:0/3960192622 shutdown_connections 2024-01-15T17:57:28.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.562+0000 7f5b8f000640 1 -- 172.21.15.2:0/3960192622 wait complete. 2024-01-15T17:57:28.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.563+0000 7f5b8f000640 1 Processor -- start 2024-01-15T17:57:28.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.563+0000 7f5b8f000640 1 -- start start 2024-01-15T17:57:28.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.564+0000 7f5b8f000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 0x7f5b881389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.564+0000 7f5b8f000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b88144c00 0x7f5b881ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.564+0000 7f5b8f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b8815c450 0x7f5b8813a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.564+0000 7f5b8f000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880afc10 con 0x7f5b8806d000 2024-01-15T17:57:28.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.564+0000 7f5b8f000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880b0130 con 0x7f5b88144c00 2024-01-15T17:57:28.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.564+0000 7f5b8f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b880b0410 con 0x7f5b8815c450 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 0x7f5b881389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 0x7f5b881389d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39744/0 (socket says 172.21.15.2:39744) 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 -- 172.21.15.2:0/1247380965 learned_addr learned my addr 172.21.15.2:0/1247380965 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b7fe00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b88144c00 0x7f5b881ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b8d000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b8815c450 0x7f5b8813a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 -- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b8815c450 msgr2=0x7f5b8813a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b8815c450 0x7f5b8813a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 -- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b88144c00 msgr2=0x7f5b881ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b88144c00 0x7f5b881ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.565+0000 7f5b87e00640 1 -- 172.21.15.2:0/1247380965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b80009a70 con 0x7f5b8806d000 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.566+0000 7f5b87e00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 0x7f5b881389d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c490 tx=0x7f5b7800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.566+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780052d0 con 0x7f5b8806d000 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.566+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b78005430 con 0x7f5b8806d000 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.566+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b78012870 con 0x7f5b8806d000 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.566+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b880b06f0 con 0x7f5b8806d000 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.566+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881eecd0 con 0x7f5b8806d000 2024-01-15T17:57:28.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.567+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b881eee60 con 0x7f5b8806d000 2024-01-15T17:57:28.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.568+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5b780129d0 con 0x7f5b8806d000 2024-01-15T17:57:28.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.569+0000 7f5b84c00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b6006b2c0 0x7f5b6006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:28.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.569+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5b780893d0 con 0x7f5b8806d000 2024-01-15T17:57:28.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.570+0000 7f5b7fe00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b6006b2c0 0x7f5b6006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:28.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.570+0000 7f5b7fe00640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b6006b2c0 0x7f5b6006d780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5b88139660 tx=0x7f5b700016f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:28.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.574+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b780545a0 con 0x7f5b8806d000 2024-01-15T17:57:28.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.778+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5b880686f0 con 0x7f5b8806d000 2024-01-15T17:57:28.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.779+0000 7f5b84c00640 1 -- 172.21.15.2:0/1247380965 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5b78057bc0 con 0x7f5b8806d000 2024-01-15T17:57:28.781 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.781+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b6006b2c0 msgr2=0x7f5b6006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.781+0000 7f5b8f000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b6006b2c0 0x7f5b6006d780 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5b88139660 tx=0x7f5b700016f0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.781+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 msgr2=0x7f5b881389d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.781+0000 7f5b8f000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 0x7f5b881389d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5b7800c490 tx=0x7f5b7800c950 comp rx=0 tx=0).stop 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 shutdown_connections 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b6006b2c0 0x7f5b6006d780 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b8815c450 0x7f5b8813a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b88144c00 0x7f5b881ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 --2- 172.21.15.2:0/1247380965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b8806d000 0x7f5b881389d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 >> 172.21.15.2:0/1247380965 conn(0x7f5b880b45f0 msgr2=0x7f5b8813fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:28.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 shutdown_connections 2024-01-15T17:57:28.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:28.782+0000 7f5b8f000640 1 -- 172.21.15.2:0/1247380965 wait complete. 2024-01-15T17:57:28.803 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T17:57:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.006+0000 7f0d61a00640 1 Processor -- start 2024-01-15T17:57:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.006+0000 7f0d61a00640 1 -- start start 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d61a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0d5c141880 0x7f0d5c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d61a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0d5c15f0e0 0x7f0d5c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d61a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d5c06dae0 con 0x7f0d5c15ec60 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d61a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c071f20 con 0x7f0d5c141880 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d61a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d5c06c6b0 con 0x7f0d5c15f0e0 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d53200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0d5c15ec60 0x7f0d5c0cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53628/0 (socket says 172.21.15.2:53628) 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.008+0000 7f0d53200640 1 -- 172.21.15.2:0/3054133879 learned_addr learned my addr 172.21.15.2:0/3054133879 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:29.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5be00640 1 --2- 172.21.15.2:0/3054133879 >> v2:172.21.15.2:3300/0 conn(0x7f0d5c141880 0x7f0d5c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d61000640 1 --2- 172.21.15.2:0/3054133879 >> v2:172.21.15.137:3300/0 conn(0x7f0d5c15f0e0 0x7f0d5c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5b400640 1 -- 172.21.15.2:0/3054133879 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 889880879 0 0) 0x7f0d5c06dae0 con 0x7f0d5c15ec60 2024-01-15T17:57:29.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5b400640 1 -- 172.21.15.2:0/3054133879 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d3c003520 con 0x7f0d5c15ec60 2024-01-15T17:57:29.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5be00640 1 -- 172.21.15.2:0/3054133879 >> v2:172.21.15.137:3300/0 conn(0x7f0d5c15f0e0 msgr2=0x7f0d5c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:29.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5be00640 1 --2- 172.21.15.2:0/3054133879 >> v2:172.21.15.137:3300/0 conn(0x7f0d5c15f0e0 0x7f0d5c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5be00640 1 -- 172.21.15.2:0/3054133879 >> v1:172.21.15.2:6789/0 conn(0x7f0d5c15ec60 legacy=0x7f0d5c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.009+0000 7f0d5be00640 1 -- 172.21.15.2:0/3054133879 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c071960 con 0x7f0d5c141880 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.010+0000 7f0d5be00640 1 --2- 172.21.15.2:0/3054133879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c141880 0x7f0d5c070fc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c070060 tx=0x7f0d4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b29c7317b02a8cb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d5b400640 1 -- 172.21.15.2:0/3054133879 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c02e070 con 0x7f0d5c141880 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d5b400640 1 -- 172.21.15.2:0/3054133879 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d4c0315a0 con 0x7f0d5c141880 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d5b400640 1 -- 172.21.15.2:0/3054133879 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c031930 con 0x7f0d5c141880 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 -- 172.21.15.2:0/3054133879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c141880 msgr2=0x7f0d5c070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 --2- 172.21.15.2:0/3054133879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c141880 0x7f0d5c070fc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0d5c070060 tx=0x7f0d4c02b840 comp rx=0 tx=0).stop 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 -- 172.21.15.2:0/3054133879 shutdown_connections 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 --2- 172.21.15.2:0/3054133879 >> v2:172.21.15.137:3300/0 conn(0x7f0d5c15f0e0 0x7f0d5c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 --2- 172.21.15.2:0/3054133879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c141880 0x7f0d5c070fc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 -- 172.21.15.2:0/3054133879 >> 172.21.15.2:0/3054133879 conn(0x7f0d5c0b0840 msgr2=0x7f0d5c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:29.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 -- 172.21.15.2:0/3054133879 shutdown_connections 2024-01-15T17:57:29.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.011+0000 7f0d61a00640 1 -- 172.21.15.2:0/3054133879 wait complete. 2024-01-15T17:57:29.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.014+0000 7f0d61a00640 1 Processor -- start 2024-01-15T17:57:29.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.014+0000 7f0d61a00640 1 -- start start 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5c141880 0x7f0d5c140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 0x7f0d5c143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d5c15f0e0 0x7f0d5c1436b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c140c90 con 0x7f0d5c15ec60 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1547c0 con 0x7f0d5c15f0e0 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c153bb0 con 0x7f0d5c141880 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d53200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 0x7f0d5c143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d53200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 0x7f0d5c143170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39756/0 (socket says 172.21.15.2:39756) 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d53200640 1 -- 172.21.15.2:0/2306547611 learned_addr learned my addr 172.21.15.2:0/2306547611 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.015+0000 7f0d61000640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d5c15f0e0 0x7f0d5c1436b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d5be00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5c141880 0x7f0d5c140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d53200640 1 -- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5c141880 msgr2=0x7f0d5c140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d53200640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5c141880 0x7f0d5c140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d53200640 1 -- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d5c15f0e0 msgr2=0x7f0d5c1436b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d53200640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d5c15f0e0 0x7f0d5c1436b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d53200640 1 -- 172.21.15.2:0/2306547611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c009400 con 0x7f0d5c15ec60 2024-01-15T17:57:29.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d53200640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 0x7f0d5c143170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400cbc0 tx=0x7f0d4400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4400ad40 con 0x7f0d5c15ec60 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d44012400 con 0x7f0d5c15ec60 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4400aea0 con 0x7f0d5c15ec60 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.016+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c153e90 con 0x7f0d5c15ec60 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.017+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c154450 con 0x7f0d5c15ec60 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.018+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c074770 con 0x7f0d5c15ec60 2024-01-15T17:57:29.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.019+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0d44007070 con 0x7f0d5c15ec60 2024-01-15T17:57:29.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.020+0000 7f0d58c00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d3406b2c0 0x7f0d3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.020+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d44089870 con 0x7f0d5c15ec60 2024-01-15T17:57:29.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.020+0000 7f0d5be00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d3406b2c0 0x7f0d3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.021+0000 7f0d5be00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d3406b2c0 0x7f0d3406d780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c036000 tx=0x7f0d4c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:29.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.024+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d44054a40 con 0x7f0d5c15ec60 2024-01-15T17:57:29.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.221+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0d5c06e1c0 con 0x7f0d5c15ec60 2024-01-15T17:57:29.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.222+0000 7f0d58c00640 1 -- 172.21.15.2:0/2306547611 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0d44017090 con 0x7f0d5c15ec60 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:57:29.225 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d3406b2c0 msgr2=0x7f0d3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d3406b2c0 0x7f0d3406d780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c036000 tx=0x7f0d4c03d040 comp rx=0 tx=0).stop 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 msgr2=0x7f0d5c143170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 0x7f0d5c143170 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0d4400cbc0 tx=0x7f0d4400a430 comp rx=0 tx=0).stop 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 shutdown_connections 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d3406b2c0 0x7f0d3406d780 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d5c15f0e0 0x7f0d5c1436b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d5c15ec60 0x7f0d5c143170 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 --2- 172.21.15.2:0/2306547611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d5c141880 0x7f0d5c140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 >> 172.21.15.2:0/2306547611 conn(0x7f0d5c0b0840 msgr2=0x7f0d5c0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 shutdown_connections 2024-01-15T17:57:29.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.225+0000 7f0d61a00640 1 -- 172.21.15.2:0/2306547611 wait complete. 2024-01-15T17:57:29.247 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T17:57:29.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T17:57:29.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.462+0000 7f2f93e00640 1 Processor -- start 2024-01-15T17:57:29.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.463+0000 7f2f93e00640 1 -- start start 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f9415d450 0x7f2f940d1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f93e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2f940d3ad0 0x7f2f940cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f93e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9415d830 con 0x7f2f940cffc0 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f93e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f940d0450 con 0x7f2f940d3ad0 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f93e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f940d2050 con 0x7f2f9415d450 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f92000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2f940d3ad0 0x7f2f940cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f92000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2f940d3ad0 0x7f2f940cfa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39768/0 (socket says 172.21.15.2:39768) 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f92000640 1 -- 172.21.15.2:0/1008436223 learned_addr learned my addr 172.21.15.2:0/1008436223 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.464+0000 7f2f92a00640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f9415d450 0x7f2f940d1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.465+0000 7f2f92000640 1 -- 172.21.15.2:0/1008436223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f9415d450 msgr2=0x7f2f940d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.465+0000 7f2f92000640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f9415d450 0x7f2f940d1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.465+0000 7f2f92000640 1 -- 172.21.15.2:0/1008436223 >> v1:172.21.15.137:6789/0 conn(0x7f2f940cffc0 legacy=0x7f2f940cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.465+0000 7f2f92000640 1 -- 172.21.15.2:0/1008436223 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f940d3f30 con 0x7f2f940d3ad0 2024-01-15T17:57:29.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.465+0000 7f2f92a00640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f9415d450 0x7f2f940d1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:29.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.466+0000 7f2f92000640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940d3ad0 0x7f2f940cfa80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009870 tx=0x7f2f7c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b8d947d29fb555 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:29.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.466+0000 7f2f91600640 1 -- 172.21.15.2:0/1008436223 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c030840 con 0x7f2f940d3ad0 2024-01-15T17:57:29.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.466+0000 7f2f91600640 1 -- 172.21.15.2:0/1008436223 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c0309a0 con 0x7f2f940d3ad0 2024-01-15T17:57:29.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.466+0000 7f2f91600640 1 -- 172.21.15.2:0/1008436223 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c030d30 con 0x7f2f940d3ad0 2024-01-15T17:57:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 -- 172.21.15.2:0/1008436223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940d3ad0 msgr2=0x7f2f940cfa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940d3ad0 0x7f2f940cfa80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c009870 tx=0x7f2f7c02b980 comp rx=0 tx=0).stop 2024-01-15T17:57:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 -- 172.21.15.2:0/1008436223 shutdown_connections 2024-01-15T17:57:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940d3ad0 0x7f2f940cfa80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 --2- 172.21.15.2:0/1008436223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f9415d450 0x7f2f940d1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 -- 172.21.15.2:0/1008436223 >> 172.21.15.2:0/1008436223 conn(0x7f2f9409c390 msgr2=0x7f2f940d30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:29.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 -- 172.21.15.2:0/1008436223 shutdown_connections 2024-01-15T17:57:29.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.467+0000 7f2f93e00640 1 -- 172.21.15.2:0/1008436223 wait complete. 2024-01-15T17:57:29.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.469+0000 7f2f93e00640 1 Processor -- start 2024-01-15T17:57:29.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.469+0000 7f2f93e00640 1 -- start start 2024-01-15T17:57:29.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 0x7f2f941537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f940d3ad0 0x7f2f9415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f93e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f9415d450 0x7f2f9415f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f93e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94153d20 con 0x7f2f940cffc0 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f93e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f94158ec0 con 0x7f2f9415d450 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f93e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f941591a0 con 0x7f2f940d3ad0 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.470+0000 7f2f92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 0x7f2f941537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 0x7f2f941537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39772/0 (socket says 172.21.15.2:39772) 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 -- 172.21.15.2:0/3467290318 learned_addr learned my addr 172.21.15.2:0/3467290318 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92000640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f940d3ad0 0x7f2f9415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f93400640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f9415d450 0x7f2f9415f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 -- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f940d3ad0 msgr2=0x7f2f9415ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f940d3ad0 0x7f2f9415ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 -- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f9415d450 msgr2=0x7f2f9415f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f9415d450 0x7f2f9415f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 -- 172.21.15.2:0/3467290318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c0094d0 con 0x7f2f940cffc0 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.471+0000 7f2f92a00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 0x7f2f941537e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400d7d0 tx=0x7f2f8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.472+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84004030 con 0x7f2f940cffc0 2024-01-15T17:57:29.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.472+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f84004190 con 0x7f2f940cffc0 2024-01-15T17:57:29.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.472+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84004520 con 0x7f2f940cffc0 2024-01-15T17:57:29.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.472+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f94159480 con 0x7f2f940cffc0 2024-01-15T17:57:29.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.472+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f94159a40 con 0x7f2f940cffc0 2024-01-15T17:57:29.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.473+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f940d0ad0 con 0x7f2f940cffc0 2024-01-15T17:57:29.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.474+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2f8400b6c0 con 0x7f2f940cffc0 2024-01-15T17:57:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.475+0000 7f2f8aa00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f6c06b2c0 0x7f2f6c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.475+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2f840898b0 con 0x7f2f940cffc0 2024-01-15T17:57:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.476+0000 7f2f92000640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f6c06b2c0 0x7f2f6c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:29.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.476+0000 7f2f92000640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f6c06b2c0 0x7f2f6c06d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c036000 tx=0x7f2f7c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:29.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.479+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f84054b10 con 0x7f2f940cffc0 2024-01-15T17:57:29.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:29.678+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f2f94074770 con 0x7f2f940cffc0 2024-01-15T17:57:30.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.092+0000 7f2f8aa00640 1 -- 172.21.15.2:0/3467290318 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2f84058130 con 0x7f2f940cffc0 2024-01-15T17:57:30.095 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f6c06b2c0 msgr2=0x7f2f6c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f6c06b2c0 0x7f2f6c06d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c036000 tx=0x7f2f7c008030 comp rx=0 tx=0).stop 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 msgr2=0x7f2f941537e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 0x7f2f941537e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400d7d0 tx=0x7f2f8400dc90 comp rx=0 tx=0).stop 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 shutdown_connections 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f6c06b2c0 0x7f2f6c06d780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f9415d450 0x7f2f9415f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f940d3ad0 0x7f2f9415ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 --2- 172.21.15.2:0/3467290318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f940cffc0 0x7f2f941537e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.096+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 >> 172.21.15.2:0/3467290318 conn(0x7f2f9409c390 msgr2=0x7f2f940afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:30.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.097+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 shutdown_connections 2024-01-15T17:57:30.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.097+0000 7f2f93e00640 1 -- 172.21.15.2:0/3467290318 wait complete. 2024-01-15T17:57:30.126 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T17:57:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.331+0000 7f9b17200640 1 Processor -- start 2024-01-15T17:57:30.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.331+0000 7f9b17200640 1 -- start start 2024-01-15T17:57:30.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.333+0000 7f9b17200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9b1806abc0 0x7f9b18167ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.333+0000 7f9b17200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b180d0ba0 con 0x7f9b18160800 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.333+0000 7f9b17200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1806d250 con 0x7f9b180ce010 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.333+0000 7f9b17200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b18160be0 con 0x7f9b1806abc0 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b15e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9b18160800 0x7f9b180cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53630/0 (socket says 172.21.15.2:53630) 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b15e00640 1 -- 172.21.15.2:0/389122462 learned_addr learned my addr 172.21.15.2:0/389122462 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b16800640 1 --2- 172.21.15.2:0/389122462 >> v2:172.21.15.137:3300/0 conn(0x7f9b1806abc0 0x7f9b18167ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782852713 0 0) 0x7f9b180d0ba0 con 0x7f9b18160800 2024-01-15T17:57:30.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af8003520 con 0x7f9b18160800 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2685378708 0 0) 0x7f9af8003520 con 0x7f9b18160800 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 >> v2:172.21.15.137:3300/0 conn(0x7f9b1806abc0 msgr2=0x7f9b18167ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 --2- 172.21.15.2:0/389122462 >> v2:172.21.15.137:3300/0 conn(0x7f9b1806abc0 0x7f9b18167ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 >> v1:172.21.15.137:6789/0 conn(0x7f9b180ce010 legacy=0x7f9b1806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.334+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b180712b0 con 0x7f9b18160800 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9b08001670 con 0x7f9b18160800 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 >> v1:172.21.15.2:6789/0 conn(0x7f9b18160800 legacy=0x7f9b180cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9af8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af80047c0 0x7f9af8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af80052a0 0x7f9af800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800c140 con 0x7f9af8003970 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800c2b0 con 0x7f9af80052a0 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af800c470 con 0x7f9af80047c0 2024-01-15T17:57:30.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b15e00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9af8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b16800640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af80052a0 0x7f9af800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b15e00640 1 -- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af80047c0 msgr2=0x7f9af8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b15e00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af80047c0 0x7f9af8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b15e00640 1 -- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af80052a0 msgr2=0x7f9af800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b15e00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af80052a0 0x7f9af800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b15e00640 1 -- 172.21.15.2:0/389122462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af8003520 con 0x7f9af8003970 2024-01-15T17:57:30.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.335+0000 7f9b16800640 1 -- 172.21.15.2:0/389122462 reap_dead start 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.336+0000 7f9b15e00640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9af8003d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c210 tx=0x7f9af80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26d201cb2fa010cc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08005ad0 con 0x7f9af8003970 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b08005c30 con 0x7f9af8003970 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b14a00640 1 -- 172.21.15.2:0/389122462 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0802ca90 con 0x7f9af8003970 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b17200640 1 -- 172.21.15.2:0/389122462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 msgr2=0x7f9af8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b17200640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9af8003d90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c210 tx=0x7f9af80057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b17200640 1 -- 172.21.15.2:0/389122462 shutdown_connections 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b17200640 1 --2- 172.21.15.2:0/389122462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9af8003d90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.337+0000 7f9b17200640 1 -- 172.21.15.2:0/389122462 >> 172.21.15.2:0/389122462 conn(0x7f9b180cfd30 msgr2=0x7f9b180d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:30.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.338+0000 7f9b17200640 1 -- 172.21.15.2:0/389122462 shutdown_connections 2024-01-15T17:57:30.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.338+0000 7f9b17200640 1 -- 172.21.15.2:0/389122462 wait complete. 2024-01-15T17:57:30.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.339+0000 7f9b17200640 1 Processor -- start 2024-01-15T17:57:30.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.339+0000 7f9b17200640 1 -- start start 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.340+0000 7f9b17200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9b181468b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.340+0000 7f9b17200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b18146fd0 0x7f9b181f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.340+0000 7f9b17200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181f6c50 0x7f9b180acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.340+0000 7f9b17200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180c8580 con 0x7f9af8003970 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.340+0000 7f9b17200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180c79b0 con 0x7f9b181f6c50 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.340+0000 7f9b17200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180c7c50 con 0x7f9b18146fd0 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9b181468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9b181468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39790/0 (socket says 172.21.15.2:39790) 2024-01-15T17:57:30.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 -- 172.21.15.2:0/3638610542 learned_addr learned my addr 172.21.15.2:0/3638610542 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b16800640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181f6c50 0x7f9b180acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15400640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b18146fd0 0x7f9b181f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 -- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b18146fd0 msgr2=0x7f9b181f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b18146fd0 0x7f9b181f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 -- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181f6c50 msgr2=0x7f9b180acd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181f6c50 0x7f9b180acd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 -- 172.21.15.2:0/3638610542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b180c7f30 con 0x7f9af8003970 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15400640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b18146fd0 0x7f9b181f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:30.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.341+0000 7f9b15e00640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9b181468b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c210 tx=0x7f9b08001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:30.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.342+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b080049e0 con 0x7f9af8003970 2024-01-15T17:57:30.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.342+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b0803a070 con 0x7f9af8003970 2024-01-15T17:57:30.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.342+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08004d80 con 0x7f9af8003970 2024-01-15T17:57:30.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.342+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b180c8210 con 0x7f9af8003970 2024-01-15T17:57:30.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.342+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b181f63a0 con 0x7f9af8003970 2024-01-15T17:57:30.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.343+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b18074770 con 0x7f9af8003970 2024-01-15T17:57:30.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.345+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9b08032070 con 0x7f9af8003970 2024-01-15T17:57:30.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.345+0000 7f9b0e000640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006b2c0 0x7f9af006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:30.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.346+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f9b080ab080 con 0x7f9af8003970 2024-01-15T17:57:30.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.346+0000 7f9b15400640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006b2c0 0x7f9af006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:30.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.347+0000 7f9b15400640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006b2c0 0x7f9af006d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b180b1040 tx=0x7f9b00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:30.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.349+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b08076310 con 0x7f9af8003970 2024-01-15T17:57:30.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:30.551+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9b180b1070 con 0x7f9af8003970 2024-01-15T17:57:31.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.098+0000 7f9b0e000640 1 -- 172.21.15.2:0/3638610542 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9b080450d0 con 0x7f9af8003970 2024-01-15T17:57:31.102 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T17:57:31.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006b2c0 msgr2=0x7f9af006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:31.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006b2c0 0x7f9af006d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b180b1040 tx=0x7f9b00009040 comp rx=0 tx=0).stop 2024-01-15T17:57:31.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 msgr2=0x7f9b181468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:31.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9b181468b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c210 tx=0x7f9b08001a00 comp rx=0 tx=0).stop 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 shutdown_connections 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006b2c0 0x7f9af006d780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181f6c50 0x7f9b180acd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b18146fd0 0x7f9b181f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 --2- 172.21.15.2:0/3638610542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af8003970 0x7f9b181468b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.102+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 >> 172.21.15.2:0/3638610542 conn(0x7f9b180cfd30 msgr2=0x7f9b180ca360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.103+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 shutdown_connections 2024-01-15T17:57:31.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.103+0000 7f9b17200640 1 -- 172.21.15.2:0/3638610542 wait complete. 2024-01-15T17:57:31.142 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T17:57:31.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.327+0000 7fec97600640 1 Processor -- start 2024-01-15T17:57:31.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.327+0000 7fec97600640 1 -- start start 2024-01-15T17:57:31.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.329+0000 7fec97600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec90144c00 0x7fec900cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:31.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.329+0000 7fec97600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec900cdd70 0x7fec9006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:31.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.329+0000 7fec97600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec900ce230 con 0x7fec9015c450 2024-01-15T17:57:31.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.329+0000 7fec97600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec900ce510 con 0x7fec90144c00 2024-01-15T17:57:31.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.329+0000 7fec97600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec900ce7f0 con 0x7fec900cdd70 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec94c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec90144c00 0x7fec900cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec94c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec90144c00 0x7fec900cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:49128/0 (socket says 172.21.15.2:49128) 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec94c00640 1 -- 172.21.15.2:0/2023018846 learned_addr learned my addr 172.21.15.2:0/2023018846 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec87c00640 1 --2- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec900cdd70 0x7fec9006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec94c00640 1 -- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec900cdd70 msgr2=0x7fec9006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec94c00640 1 --2- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec900cdd70 0x7fec9006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.330+0000 7fec94c00640 1 -- 172.21.15.2:0/2023018846 >> v1:172.21.15.137:6789/0 conn(0x7fec9015c450 legacy=0x7fec900b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:31.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.331+0000 7fec94c00640 1 -- 172.21.15.2:0/2023018846 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec900705f0 con 0x7fec90144c00 2024-01-15T17:57:31.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.332+0000 7fec94c00640 1 --2- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec90144c00 0x7fec900cd830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec80009870 tx=0x7fec8002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8fe8623b839fd5b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:31.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.332+0000 7fec8fe00640 1 -- 172.21.15.2:0/2023018846 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80030840 con 0x7fec90144c00 2024-01-15T17:57:31.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.332+0000 7fec8fe00640 1 -- 172.21.15.2:0/2023018846 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec800309a0 con 0x7fec90144c00 2024-01-15T17:57:31.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.332+0000 7fec8fe00640 1 -- 172.21.15.2:0/2023018846 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80030d30 con 0x7fec90144c00 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 -- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec90144c00 msgr2=0x7fec900cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 --2- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec90144c00 0x7fec900cd830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec80009870 tx=0x7fec8002b980 comp rx=0 tx=0).stop 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 -- 172.21.15.2:0/2023018846 shutdown_connections 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 --2- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec900cdd70 0x7fec9006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 --2- 172.21.15.2:0/2023018846 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec90144c00 0x7fec900cd830 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 -- 172.21.15.2:0/2023018846 >> 172.21.15.2:0/2023018846 conn(0x7fec900b45f0 msgr2=0x7fec900b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 -- 172.21.15.2:0/2023018846 shutdown_connections 2024-01-15T17:57:31.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.333+0000 7fec97600640 1 -- 172.21.15.2:0/2023018846 wait complete. 2024-01-15T17:57:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.336+0000 7fec97600640 1 Processor -- start 2024-01-15T17:57:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.336+0000 7fec97600640 1 -- start start 2024-01-15T17:57:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec97600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 0x7fec901f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec97600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec90144c00 0x7fec901f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec97600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec9015c450 0x7fec901568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec97600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec901f33f0 con 0x7fec900cdd70 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec97600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90157e30 con 0x7fec9015c450 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec97600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec90157220 con 0x7fec90144c00 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec94c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 0x7fec901f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec94c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 0x7fec901f2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39792/0 (socket says 172.21.15.2:39792) 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.337+0000 7fec94c00640 1 -- 172.21.15.2:0/3541666651 learned_addr learned my addr 172.21.15.2:0/3541666651 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec87c00640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec90144c00 0x7fec901f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec94c00640 1 -- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec90144c00 msgr2=0x7fec901f5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec94c00640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec90144c00 0x7fec901f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec94c00640 1 -- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec9015c450 msgr2=0x7fec901568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec95600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec9015c450 0x7fec901568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec94c00640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec9015c450 0x7fec901568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec94c00640 1 -- 172.21.15.2:0/3541666651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec800094d0 con 0x7fec900cdd70 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec95600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec9015c450 0x7fec901568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.338+0000 7fec94c00640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 0x7fec901f2eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fec80000c00 tx=0x7fec80004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.339+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80039c90 con 0x7fec900cdd70 2024-01-15T17:57:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.339+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec80039e10 con 0x7fec900cdd70 2024-01-15T17:57:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.339+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec80043640 con 0x7fec900cdd70 2024-01-15T17:57:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.339+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90157500 con 0x7fec900cdd70 2024-01-15T17:57:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.339+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90157ac0 con 0x7fec900cdd70 2024-01-15T17:57:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.340+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec90157c50 con 0x7fec900cdd70 2024-01-15T17:57:31.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.341+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fec800437a0 con 0x7fec900cdd70 2024-01-15T17:57:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.342+0000 7fec8d600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6806b310 0x7fec6806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.342+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fec8003e070 con 0x7fec900cdd70 2024-01-15T17:57:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.343+0000 7fec87c00640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6806b310 0x7fec6806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:31.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.344+0000 7fec87c00640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6806b310 0x7fec6806d7d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec9013b600 tx=0x7fec78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:31.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.347+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec80077a20 con 0x7fec900cdd70 2024-01-15T17:57:31.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:31.581+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec900686f0 con 0x7fec900cdd70 2024-01-15T17:57:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.131+0000 7fec8d600640 1 -- 172.21.15.2:0/3541666651 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fec800340c0 con 0x7fec900cdd70 2024-01-15T17:57:32.137 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 2 and data pool 3 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6806b310 msgr2=0x7fec6806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6806b310 0x7fec6806d7d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fec9013b600 tx=0x7fec78008040 comp rx=0 tx=0).stop 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 msgr2=0x7fec901f2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 0x7fec901f2eb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fec80000c00 tx=0x7fec80004730 comp rx=0 tx=0).stop 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 shutdown_connections 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6806b310 0x7fec6806d7d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec9015c450 0x7fec901568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec90144c00 0x7fec901f5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 --2- 172.21.15.2:0/3541666651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec900cdd70 0x7fec901f2eb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.140+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 >> 172.21.15.2:0/3541666651 conn(0x7fec900b45f0 msgr2=0x7fec90143e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.144+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 shutdown_connections 2024-01-15T17:57:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.144+0000 7fec97600640 1 -- 172.21.15.2:0/3541666651 wait complete. 2024-01-15T17:57:32.174 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:32.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.354+0000 7f53abc00640 1 Processor -- start 2024-01-15T17:57:32.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.354+0000 7f53abc00640 1 -- start start 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53abc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4070f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53abc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f53a406c470 0x7f53a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53abc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f53a4161d30 0x7f53a41621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53abc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a406ce30 con 0x7f53a4161d30 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53abc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a4071cb0 con 0x7f53a406c470 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53abc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4070a70 con 0x7f53a40b3c70 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53a9c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f53a4161d30 0x7f53a41621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53a9c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f53a4161d30 0x7f53a41621c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39794/0 (socket says 172.21.15.2:39794) 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.356+0000 7f53a9c00640 1 -- 172.21.15.2:0/923330594 learned_addr learned my addr 172.21.15.2:0/923330594 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:32.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a9200640 1 --2- 172.21.15.2:0/923330594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a3e00640 1 --2- 172.21.15.2:0/923330594 >> v2:172.21.15.137:3300/0 conn(0x7f53a406c470 0x7f53a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a9c00640 1 -- 172.21.15.2:0/923330594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 msgr2=0x7f53a4070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a9c00640 1 --2- 172.21.15.2:0/923330594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4070f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a9c00640 1 -- 172.21.15.2:0/923330594 >> v2:172.21.15.137:3300/0 conn(0x7f53a406c470 msgr2=0x7f53a406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a9c00640 1 --2- 172.21.15.2:0/923330594 >> v2:172.21.15.137:3300/0 conn(0x7f53a406c470 0x7f53a406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.357+0000 7f53a9c00640 1 -- 172.21.15.2:0/923330594 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4162700 con 0x7f53a4161d30 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.358+0000 7f53a9c00640 1 --2- 172.21.15.2:0/923330594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a4161d30 0x7f53a41621c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53a406e590 tx=0x7f539c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b17f16215749f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.359+0000 7f53a3400640 1 -- 172.21.15.2:0/923330594 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c031870 con 0x7f53a4161d30 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.359+0000 7f53a3400640 1 -- 172.21.15.2:0/923330594 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539c0319d0 con 0x7f53a4161d30 2024-01-15T17:57:32.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.359+0000 7f53a3400640 1 -- 172.21.15.2:0/923330594 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539c031d60 con 0x7f53a4161d30 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.359+0000 7f53abc00640 1 -- 172.21.15.2:0/923330594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a4161d30 msgr2=0x7f53a41621c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.359+0000 7f53abc00640 1 --2- 172.21.15.2:0/923330594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a4161d30 0x7f53a41621c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53a406e590 tx=0x7f539c02b910 comp rx=0 tx=0).stop 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 -- 172.21.15.2:0/923330594 shutdown_connections 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 --2- 172.21.15.2:0/923330594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a4161d30 0x7f53a41621c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 --2- 172.21.15.2:0/923330594 >> v2:172.21.15.137:3300/0 conn(0x7f53a406c470 0x7f53a406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 --2- 172.21.15.2:0/923330594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4070f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 -- 172.21.15.2:0/923330594 >> 172.21.15.2:0/923330594 conn(0x7f53a406da20 msgr2=0x7f53a406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 -- 172.21.15.2:0/923330594 shutdown_connections 2024-01-15T17:57:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.360+0000 7f53abc00640 1 -- 172.21.15.2:0/923330594 wait complete. 2024-01-15T17:57:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.362+0000 7f53abc00640 1 Processor -- start 2024-01-15T17:57:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.362+0000 7f53abc00640 1 -- start start 2024-01-15T17:57:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53abc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 0x7f53a414c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53abc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4163270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53abc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53a4161d30 0x7f53a41637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53abc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a414d0b0 con 0x7f53a406c470 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53abc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a414d950 con 0x7f53a4161d30 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53abc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a41f8220 con 0x7f53a40b3c70 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 0x7f53a414c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 0x7f53a414c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39802/0 (socket says 172.21.15.2:39802) 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 -- 172.21.15.2:0/2992361502 learned_addr learned my addr 172.21.15.2:0/2992361502 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 -- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 msgr2=0x7f53a4163270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4163270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9c00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53a4161d30 0x7f53a41637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 -- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53a4161d30 msgr2=0x7f53a41637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53a4161d30 0x7f53a41637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.363+0000 7f53a9200640 1 -- 172.21.15.2:0/2992361502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c0094d0 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.364+0000 7f53a9200640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 0x7f53a414c7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5394007f20 tx=0x7f539400e570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.364+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394014620 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.364+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394014780 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.364+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5394014b30 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.364+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a41f7e20 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.364+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a414d570 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.365+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a40686f0 con 0x7f53a406c470 2024-01-15T17:57:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.366+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5394014c90 con 0x7f53a406c470 2024-01-15T17:57:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.368+0000 7f53a0c00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f537c06b2c0 0x7f537c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.368+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f539408a580 con 0x7f53a406c470 2024-01-15T17:57:32.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.369+0000 7f53a3e00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f537c06b2c0 0x7f537c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.370+0000 7f53a3e00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f537c06b2c0 0x7f537c06d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f53a4165070 tx=0x7f538c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:32.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.373+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53940553e0 con 0x7f53a406c470 2024-01-15T17:57:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.572+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a414dbc0 con 0x7f53a406c470 2024-01-15T17:57:32.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.576+0000 7f53a0c00640 1 -- 172.21.15.2:0/2992361502 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16458 (secure 0 0 0) 0x7f5394058a00 con 0x7f53a406c470 2024-01-15T17:57:32.579 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:32.580 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":21,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:31.582997+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:32.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f537c06b2c0 msgr2=0x7f537c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f537c06b2c0 0x7f537c06d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f53a4165070 tx=0x7f538c008040 comp rx=0 tx=0).stop 2024-01-15T17:57:32.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 msgr2=0x7f53a414c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 0x7f53a414c7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f5394007f20 tx=0x7f539400e570 comp rx=0 tx=0).stop 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 shutdown_connections 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f537c06b2c0 0x7f537c06d780 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53a4161d30 0x7f53a41637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53a40b3c70 0x7f53a4163270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 --2- 172.21.15.2:0/2992361502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53a406c470 0x7f53a414c7b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 >> 172.21.15.2:0/2992361502 conn(0x7f53a406da20 msgr2=0x7f53a415eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 shutdown_connections 2024-01-15T17:57:32.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.579+0000 7f53abc00640 1 -- 172.21.15.2:0/2992361502 wait complete. 2024-01-15T17:57:32.601 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:32.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.805+0000 7fb722a00640 1 Processor -- start 2024-01-15T17:57:32.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.805+0000 7fb722a00640 1 -- start start 2024-01-15T17:57:32.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.806+0000 7fb722a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb71c161770 0x7fb71c0d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb722a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb71c06b5c0 0x7fb71c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb722a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb71c06bd30 con 0x7fb71c0c3df0 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb722a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71c0b16a0 con 0x7fb71c161770 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb722a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71c0b03a0 con 0x7fb71c06b5c0 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb71c161770 0x7fb71c0d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb71c161770 0x7fb71c0d1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39808/0 (socket says 172.21.15.2:39808) 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 -- 172.21.15.2:0/165530341 learned_addr learned my addr 172.21.15.2:0/165530341 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb722000640 1 --1- 172.21.15.2:0/165530341 >> v1:172.21.15.2:6789/0 conn(0x7fb71c0c3df0 0x7fb71c0ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53632/0 (socket says 172.21.15.2:53632) 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb720c00640 1 --2- 172.21.15.2:0/165530341 >> v2:172.21.15.137:3300/0 conn(0x7fb71c06b5c0 0x7fb71c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 -- 172.21.15.2:0/165530341 >> v2:172.21.15.137:3300/0 conn(0x7fb71c06b5c0 msgr2=0x7fb71c0b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:32.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 --2- 172.21.15.2:0/165530341 >> v2:172.21.15.137:3300/0 conn(0x7fb71c06b5c0 0x7fb71c0b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 -- 172.21.15.2:0/165530341 >> v1:172.21.15.2:6789/0 conn(0x7fb71c0c3df0 legacy=0x7fb71c0ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb721600640 1 -- 172.21.15.2:0/165530341 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71c0b0680 con 0x7fb71c161770 2024-01-15T17:57:32.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.807+0000 7fb720c00640 1 --2- 172.21.15.2:0/165530341 >> v2:172.21.15.137:3300/0 conn(0x7fb71c06b5c0 0x7fb71c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:32.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.808+0000 7fb721600640 1 --2- 172.21.15.2:0/165530341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c161770 0x7fb71c0d1940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb71c071720 tx=0x7fb70c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1760a827b7fd6d32 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:32.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb71be00640 1 -- 172.21.15.2:0/165530341 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c031870 con 0x7fb71c161770 2024-01-15T17:57:32.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb71be00640 1 -- 172.21.15.2:0/165530341 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70c0319d0 con 0x7fb71c161770 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb71be00640 1 -- 172.21.15.2:0/165530341 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70c031d60 con 0x7fb71c161770 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb722a00640 1 -- 172.21.15.2:0/165530341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c161770 msgr2=0x7fb71c0d1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb722a00640 1 --2- 172.21.15.2:0/165530341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c161770 0x7fb71c0d1940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb71c071720 tx=0x7fb70c02b910 comp rx=0 tx=0).stop 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb722a00640 1 -- 172.21.15.2:0/165530341 shutdown_connections 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb722a00640 1 --2- 172.21.15.2:0/165530341 >> v2:172.21.15.137:3300/0 conn(0x7fb71c06b5c0 0x7fb71c0b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.809+0000 7fb722a00640 1 --2- 172.21.15.2:0/165530341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c161770 0x7fb71c0d1940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.810+0000 7fb722a00640 1 -- 172.21.15.2:0/165530341 >> 172.21.15.2:0/165530341 conn(0x7fb71c0d0e20 msgr2=0x7fb71c0d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.810+0000 7fb722a00640 1 -- 172.21.15.2:0/165530341 shutdown_connections 2024-01-15T17:57:32.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.810+0000 7fb722a00640 1 -- 172.21.15.2:0/165530341 wait complete. 2024-01-15T17:57:32.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.811+0000 7fb722a00640 1 Processor -- start 2024-01-15T17:57:32.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.812+0000 7fb722a00640 1 -- start start 2024-01-15T17:57:32.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.812+0000 7fb722a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb71c06b5c0 0x7fb71c1604d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb722a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 0x7fb71c0c5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb722a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb71c161770 0x7fb71c0c5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb722a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c160040 con 0x7fb71c0c3df0 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb722a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c1f7c00 con 0x7fb71c161770 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb722a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71c15f710 con 0x7fb71c06b5c0 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 0x7fb71c0c5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 0x7fb71c0c5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39810/0 (socket says 172.21.15.2:39810) 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 -- 172.21.15.2:0/1908583396 learned_addr learned my addr 172.21.15.2:0/1908583396 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb721600640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb71c06b5c0 0x7fb71c1604d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb722000640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb71c161770 0x7fb71c0c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 -- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb71c06b5c0 msgr2=0x7fb71c1604d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb71c06b5c0 0x7fb71c1604d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 -- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb71c161770 msgr2=0x7fb71c0c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb71c161770 0x7fb71c0c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:32.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.813+0000 7fb720c00640 1 -- 172.21.15.2:0/1908583396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0094d0 con 0x7fb71c0c3df0 2024-01-15T17:57:32.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.814+0000 7fb720c00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 0x7fb71c0c5420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb70400c7f0 tx=0x7fb70400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:32.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.814+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704007820 con 0x7fb71c0c3df0 2024-01-15T17:57:32.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.814+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb704007980 con 0x7fb71c0c3df0 2024-01-15T17:57:32.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.814+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704007cf0 con 0x7fb71c0c3df0 2024-01-15T17:57:32.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.814+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb71c15f9f0 con 0x7fb71c0c3df0 2024-01-15T17:57:32.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.814+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb71c160a10 con 0x7fb71c0c3df0 2024-01-15T17:57:32.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.816+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71c074770 con 0x7fb71c0c3df0 2024-01-15T17:57:32.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.816+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb7040113f0 con 0x7fb71c0c3df0 2024-01-15T17:57:32.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.817+0000 7fb719600640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6f406b2c0 0x7fb6f406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:32.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.817+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb704089e50 con 0x7fb71c0c3df0 2024-01-15T17:57:32.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.818+0000 7fb721600640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6f406b2c0 0x7fb6f406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:32.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.819+0000 7fb721600640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6f406b2c0 0x7fb6f406d780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb70c036000 tx=0x7fb70c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:32.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:32.822+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb704054d50 con 0x7fb71c0c3df0 2024-01-15T17:57:33.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.014+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb71c0c7950 con 0x7fb71c0c3df0 2024-01-15T17:57:33.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.017+0000 7fb719600640 1 -- 172.21.15.2:0/1908583396 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16458 (secure 0 0 0) 0x7fb70401f090 con 0x7fb71c0c3df0 2024-01-15T17:57:33.021 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:33.021 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":21,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:31.582997+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6f406b2c0 msgr2=0x7fb6f406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6f406b2c0 0x7fb6f406d780 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb70c036000 tx=0x7fb70c03c040 comp rx=0 tx=0).stop 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 msgr2=0x7fb71c0c5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 0x7fb71c0c5420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb70400c7f0 tx=0x7fb70400ccb0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 shutdown_connections 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6f406b2c0 0x7fb6f406d780 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb71c161770 0x7fb71c0c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb71c0c3df0 0x7fb71c0c5420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 --2- 172.21.15.2:0/1908583396 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb71c06b5c0 0x7fb71c1604d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 >> 172.21.15.2:0/1908583396 conn(0x7fb71c0d0e20 msgr2=0x7fb71c15c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:33.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 shutdown_connections 2024-01-15T17:57:33.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.020+0000 7fb722a00640 1 -- 172.21.15.2:0/1908583396 wait complete. 2024-01-15T17:57:33.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T17:57:33.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.265+0000 7f23aa400640 1 Processor -- start 2024-01-15T17:57:33.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.265+0000 7f23aa400640 1 -- start start 2024-01-15T17:57:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23aa400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f23a40d1870 0x7f23a40d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23aa400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f23a40d1c50 0x7f23a40b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23aa400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23a40d1250 con 0x7f23a40b2210 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23aa400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a40ce6d0 con 0x7f23a40d1870 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23aa400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23a40ce8e0 con 0x7f23a40d1c50 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23a9000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f23a40d1870 0x7f23a40d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23a9000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f23a40d1870 0x7f23a40d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49484/0 (socket says 172.21.15.2:49484) 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.266+0000 7f23a9000640 1 -- 172.21.15.2:0/2462015218 learned_addr learned my addr 172.21.15.2:0/2462015218 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.267+0000 7f23a9000640 1 -- 172.21.15.2:0/2462015218 >> v2:172.21.15.137:3300/0 conn(0x7f23a40d1c50 msgr2=0x7f23a40b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.267+0000 7f23a9000640 1 --2- 172.21.15.2:0/2462015218 >> v2:172.21.15.137:3300/0 conn(0x7f23a40d1c50 0x7f23a40b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.267+0000 7f23a9000640 1 -- 172.21.15.2:0/2462015218 >> v1:172.21.15.2:6789/0 conn(0x7f23a40b2210 legacy=0x7f23a415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:33.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.267+0000 7f23a9000640 1 -- 172.21.15.2:0/2462015218 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23a40cebc0 con 0x7f23a40d1870 2024-01-15T17:57:33.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.268+0000 7f23a9000640 1 --2- 172.21.15.2:0/2462015218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a40d3bd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f23a408d7e0 tx=0x7f239402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dd8cc5eb14127bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:33.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23a3400640 1 -- 172.21.15.2:0/2462015218 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394030870 con 0x7f23a40d1870 2024-01-15T17:57:33.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23a3400640 1 -- 172.21.15.2:0/2462015218 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23940309d0 con 0x7f23a40d1870 2024-01-15T17:57:33.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23a3400640 1 -- 172.21.15.2:0/2462015218 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2394030d60 con 0x7f23a40d1870 2024-01-15T17:57:33.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 -- 172.21.15.2:0/2462015218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 msgr2=0x7f23a40d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 --2- 172.21.15.2:0/2462015218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a40d3bd0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f23a408d7e0 tx=0x7f239402b9b0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 -- 172.21.15.2:0/2462015218 shutdown_connections 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 --2- 172.21.15.2:0/2462015218 >> v2:172.21.15.137:3300/0 conn(0x7f23a40d1c50 0x7f23a40b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 --2- 172.21.15.2:0/2462015218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a40d3bd0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 -- 172.21.15.2:0/2462015218 >> 172.21.15.2:0/2462015218 conn(0x7f23a40d30b0 msgr2=0x7f23a40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 -- 172.21.15.2:0/2462015218 shutdown_connections 2024-01-15T17:57:33.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.269+0000 7f23aa400640 1 -- 172.21.15.2:0/2462015218 wait complete. 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.271+0000 7f23aa400640 1 Processor -- start 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.271+0000 7f23aa400640 1 -- start start 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.272+0000 7f23aa400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f23a40b2210 0x7f23a4150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.272+0000 7f23aa400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a415e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.272+0000 7f23aa400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23a40d1c50 0x7f23a40cc190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.272+0000 7f23aa400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a41f3e00 con 0x7f23a40d1870 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.272+0000 7f23aa400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a41f4290 con 0x7f23a40b2210 2024-01-15T17:57:33.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.272+0000 7f23aa400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23a41f31f0 con 0x7f23a40d1c50 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a415e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a415e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49490/0 (socket says 172.21.15.2:49490) 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 -- 172.21.15.2:0/3724394699 learned_addr learned my addr 172.21.15.2:0/3724394699 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a9000640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f23a40b2210 0x7f23a4150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a9a00640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23a40d1c50 0x7f23a40cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 -- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23a40d1c50 msgr2=0x7f23a40cc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23a40d1c50 0x7f23a40cc190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 -- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f23a40b2210 msgr2=0x7f23a4150640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f23a40b2210 0x7f23a4150640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:33.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 -- 172.21.15.2:0/3724394699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940094d0 con 0x7f23a40d1870 2024-01-15T17:57:33.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.273+0000 7f23a3e00640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a415e300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c440 tx=0x7f238c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:33.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.274+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c0052d0 con 0x7f23a40d1870 2024-01-15T17:57:33.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.274+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c005430 con 0x7f23a40d1870 2024-01-15T17:57:33.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.274+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a41f34d0 con 0x7f23a40d1870 2024-01-15T17:57:33.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.274+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c007260 con 0x7f23a40d1870 2024-01-15T17:57:33.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.274+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a41f3a90 con 0x7f23a40d1870 2024-01-15T17:57:33.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.275+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a41f3f70 con 0x7f23a40d1870 2024-01-15T17:57:33.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.276+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f238c0073c0 con 0x7f23a40d1870 2024-01-15T17:57:33.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.277+0000 7f23a0c00640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f237c06b2c0 0x7f237c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:33.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.277+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f238c089070 con 0x7f23a40d1870 2024-01-15T17:57:33.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.277+0000 7f23a9000640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f237c06b2c0 0x7f237c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:33.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.279+0000 7f23a9000640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f237c06b2c0 0x7f237c06d780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2394036000 tx=0x7f2394008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:33.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.281+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f238c0bbe90 con 0x7f23a40d1870 2024-01-15T17:57:33.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:33.520+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f23a4074770 con 0x7f23a40d1870 2024-01-15T17:57:34.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.144+0000 7f23a0c00640 1 -- 172.21.15.2:0/3724394699 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f238c053e40 con 0x7f23a40d1870 2024-01-15T17:57:34.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f237c06b2c0 msgr2=0x7f237c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f237c06b2c0 0x7f237c06d780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2394036000 tx=0x7f2394008030 comp rx=0 tx=0).stop 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 msgr2=0x7f23a415e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a415e300 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f238c00c440 tx=0x7f238c001e70 comp rx=0 tx=0).stop 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 shutdown_connections 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f237c06b2c0 0x7f237c06d780 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23a40d1c50 0x7f23a40cc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f23a40d1870 0x7f23a415e300 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 --2- 172.21.15.2:0/3724394699 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f23a40b2210 0x7f23a4150640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 >> 172.21.15.2:0/3724394699 conn(0x7f23a40d30b0 msgr2=0x7f23a40d2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.148+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 shutdown_connections 2024-01-15T17:57:34.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.149+0000 7f23aa400640 1 -- 172.21.15.2:0/3724394699 wait complete. 2024-01-15T17:57:34.173 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T17:57:34.173 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T17:57:34.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.394+0000 7fcaa9800640 1 Processor -- start 2024-01-15T17:57:34.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.394+0000 7fcaa9800640 1 -- start start 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.395+0000 7fcaa9800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcaa4144c00 0x7fcaa40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.395+0000 7fcaa9800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40cdd70 0x7fcaa406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa9800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40ce230 con 0x7fcaa415c450 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa9800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa40ce510 con 0x7fcaa4144c00 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa9800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa40ce7f0 con 0x7fcaa40cdd70 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcaa4144c00 0x7fcaa40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:34.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcaa4144c00 0x7fcaa40cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49502/0 (socket says 172.21.15.2:49502) 2024-01-15T17:57:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 -- 172.21.15.2:0/2612611253 learned_addr learned my addr 172.21.15.2:0/2612611253 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fca99e00640 1 --2- 172.21.15.2:0/2612611253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40cdd70 0x7fcaa406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 -- 172.21.15.2:0/2612611253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40cdd70 msgr2=0x7fcaa406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/2612611253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40cdd70 0x7fcaa406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 -- 172.21.15.2:0/2612611253 >> v1:172.21.15.137:6789/0 conn(0x7fcaa415c450 legacy=0x7fcaa40b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:34.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.396+0000 7fcaa2a00640 1 -- 172.21.15.2:0/2612611253 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa40705f0 con 0x7fcaa4144c00 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.398+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/2612611253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa4144c00 0x7fcaa40cd830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca94009870 tx=0x7fca9402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a3a4441a5c6401b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.398+0000 7fcaa2000640 1 -- 172.21.15.2:0/2612611253 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94031640 con 0x7fcaa4144c00 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.398+0000 7fcaa2000640 1 -- 172.21.15.2:0/2612611253 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca940317a0 con 0x7fcaa4144c00 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.398+0000 7fcaa2000640 1 -- 172.21.15.2:0/2612611253 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94031b30 con 0x7fcaa4144c00 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 -- 172.21.15.2:0/2612611253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa4144c00 msgr2=0x7fcaa40cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 --2- 172.21.15.2:0/2612611253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa4144c00 0x7fcaa40cd830 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fca94009870 tx=0x7fca9402b8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 -- 172.21.15.2:0/2612611253 shutdown_connections 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 --2- 172.21.15.2:0/2612611253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40cdd70 0x7fcaa406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 --2- 172.21.15.2:0/2612611253 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa4144c00 0x7fcaa40cd830 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 -- 172.21.15.2:0/2612611253 >> 172.21.15.2:0/2612611253 conn(0x7fcaa40b45f0 msgr2=0x7fcaa40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:34.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 -- 172.21.15.2:0/2612611253 shutdown_connections 2024-01-15T17:57:34.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.399+0000 7fcaa9800640 1 -- 172.21.15.2:0/2612611253 wait complete. 2024-01-15T17:57:34.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.401+0000 7fcaa9800640 1 Processor -- start 2024-01-15T17:57:34.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.401+0000 7fcaa9800640 1 -- start start 2024-01-15T17:57:34.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 0x7fcaa41f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:34.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa9800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa4144c00 0x7fcaa4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:34.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa9800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa415c450 0x7fcaa4156130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:34.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa9800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa4141b60 con 0x7fcaa40cdd70 2024-01-15T17:57:34.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa9800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa4154e80 con 0x7fcaa415c450 2024-01-15T17:57:34.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa9800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa4155160 con 0x7fcaa4144c00 2024-01-15T17:57:34.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 0x7fcaa41f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:34.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 0x7fcaa41f5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49506/0 (socket says 172.21.15.2:49506) 2024-01-15T17:57:34.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.402+0000 7fcaa2a00640 1 -- 172.21.15.2:0/605109199 learned_addr learned my addr 172.21.15.2:0/605109199 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:34.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa2a00640 1 -- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa4144c00 msgr2=0x7fcaa4155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:34.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa4144c00 0x7fcaa4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa2a00640 1 -- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa415c450 msgr2=0x7fcaa4156130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:34.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa415c450 0x7fcaa4156130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:34.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa2a00640 1 -- 172.21.15.2:0/605109199 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca940094d0 con 0x7fcaa40cdd70 2024-01-15T17:57:34.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 0x7fcaa41f5aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca94009870 tx=0x7fca940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:34.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9403ac60 con 0x7fcaa40cdd70 2024-01-15T17:57:34.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9403adc0 con 0x7fcaa40cdd70 2024-01-15T17:57:34.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa4141e00 con 0x7fcaa40cdd70 2024-01-15T17:57:34.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94044610 con 0x7fcaa40cdd70 2024-01-15T17:57:34.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.403+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa4158010 con 0x7fcaa40cdd70 2024-01-15T17:57:34.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.404+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa4142320 con 0x7fcaa40cdd70 2024-01-15T17:57:34.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.405+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fca9402f050 con 0x7fcaa40cdd70 2024-01-15T17:57:34.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.406+0000 7fca9b400640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b2c0 0x7fca7c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:34.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.406+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca940aba70 con 0x7fcaa40cdd70 2024-01-15T17:57:34.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.407+0000 7fca99e00640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b2c0 0x7fca7c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:34.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.408+0000 7fca99e00640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b2c0 0x7fca7c06d780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcaa413c400 tx=0x7fca8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:34.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.411+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcaa4142320 con 0x7fcaa40cdd70 2024-01-15T17:57:34.637 INFO:tasks.ceph.mds.a.smithi002.stderr:2024-01-15T17:57:34.634+0000 7f407e400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T17:57:34.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:34.646+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fcaa4142320 con 0x7fcaa40cdd70 2024-01-15T17:57:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.155+0000 7fca9b400640 1 -- 172.21.15.2:0/605109199 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fca940350c0 con 0x7fcaa40cdd70 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.158+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b2c0 msgr2=0x7fca7c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.158+0000 7fcaa9800640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b2c0 0x7fca7c06d780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcaa413c400 tx=0x7fca8c008040 comp rx=0 tx=0).stop 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.158+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 msgr2=0x7fcaa41f5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.158+0000 7fcaa9800640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 0x7fcaa41f5aa0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fca94009870 tx=0x7fca940046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 shutdown_connections 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b2c0 0x7fca7c06d780 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa415c450 0x7fcaa4156130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa4144c00 0x7fcaa4155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 --2- 172.21.15.2:0/605109199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaa40cdd70 0x7fcaa41f5aa0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 >> 172.21.15.2:0/605109199 conn(0x7fcaa40b45f0 msgr2=0x7fcaa40b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 shutdown_connections 2024-01-15T17:57:35.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.159+0000 7fcaa9800640 1 -- 172.21.15.2:0/605109199 wait complete. 2024-01-15T17:57:35.182 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T17:57:35.182 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:57:35.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.396+0000 7f1bf8400640 1 Processor -- start 2024-01-15T17:57:35.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.396+0000 7f1bf8400640 1 -- start start 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf8400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bf006cb80 0x7f1bf0166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf8400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf015cb00 con 0x7f1bf015c0e0 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf8400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf00ccec0 con 0x7f1bf015b7d0 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf8400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf00cccd0 con 0x7f1bf006cb80 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf5a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1bf015c0e0 0x7f1bf00cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35370/0 (socket says 172.21.15.2:35370) 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf5a00640 1 -- 172.21.15.2:0/2614679021 learned_addr learned my addr 172.21.15.2:0/2614679021 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:35.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.398+0000 7f1bf6400640 1 --2- 172.21.15.2:0/2614679021 >> v2:172.21.15.2:3300/0 conn(0x7f1bf006cb80 0x7f1bf0166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.399+0000 7f1befe00640 1 -- 172.21.15.2:0/2614679021 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467608711 0 0) 0x7f1bf015cb00 con 0x7f1bf015c0e0 2024-01-15T17:57:35.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.399+0000 7f1befe00640 1 -- 172.21.15.2:0/2614679021 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd0003520 con 0x7f1bf015c0e0 2024-01-15T17:57:35.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.399+0000 7f1bf6400640 1 -- 172.21.15.2:0/2614679021 >> v1:172.21.15.137:6789/0 conn(0x7f1bf015b7d0 legacy=0x7f1bf006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.399+0000 7f1bf6400640 1 -- 172.21.15.2:0/2614679021 >> v1:172.21.15.2:6789/0 conn(0x7f1bf015c0e0 legacy=0x7f1bf00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.399+0000 7f1bf6400640 1 -- 172.21.15.2:0/2614679021 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf00cd1a0 con 0x7f1bf006cb80 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.400+0000 7f1bf6400640 1 --2- 172.21.15.2:0/2614679021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf0166070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009d90 tx=0x7f1be802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122b460e655bfffb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.400+0000 7f1befe00640 1 -- 172.21.15.2:0/2614679021 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be802ccf0 con 0x7f1bf006cb80 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.400+0000 7f1befe00640 1 -- 172.21.15.2:0/2614679021 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be802ce50 con 0x7f1bf006cb80 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.400+0000 7f1befe00640 1 -- 172.21.15.2:0/2614679021 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be8034660 con 0x7f1bf006cb80 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 -- 172.21.15.2:0/2614679021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 msgr2=0x7f1bf0166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 --2- 172.21.15.2:0/2614679021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf0166070 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1be8009d90 tx=0x7f1be802bf40 comp rx=0 tx=0).stop 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 -- 172.21.15.2:0/2614679021 shutdown_connections 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 --2- 172.21.15.2:0/2614679021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf0166070 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 -- 172.21.15.2:0/2614679021 >> 172.21.15.2:0/2614679021 conn(0x7f1bf006da20 msgr2=0x7f1bf006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:35.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 -- 172.21.15.2:0/2614679021 shutdown_connections 2024-01-15T17:57:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.401+0000 7f1bf8400640 1 -- 172.21.15.2:0/2614679021 wait complete. 2024-01-15T17:57:35.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.403+0000 7f1bf8400640 1 Processor -- start 2024-01-15T17:57:35.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.403+0000 7f1bf8400640 1 -- start start 2024-01-15T17:57:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf8400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf014d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf8400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf015b7d0 0x7f1bf0153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf8400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf015c0e0 0x7f1bf0158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf8400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf014dae0 con 0x7f1bf006cb80 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf8400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0156e40 con 0x7f1bf015c0e0 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf8400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf0157120 con 0x7f1bf015b7d0 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf014d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf014d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49526/0 (socket says 172.21.15.2:49526) 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.404+0000 7f1bf5a00640 1 -- 172.21.15.2:0/3935227458 learned_addr learned my addr 172.21.15.2:0/3935227458 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf6400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf015c0e0 0x7f1bf0158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5a00640 1 -- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf015b7d0 msgr2=0x7f1bf0153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5000640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf015b7d0 0x7f1bf0153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5a00640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf015b7d0 0x7f1bf0153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5a00640 1 -- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf015c0e0 msgr2=0x7f1bf0158950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5a00640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf015c0e0 0x7f1bf0158950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5a00640 1 -- 172.21.15.2:0/3935227458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be8009a70 con 0x7f1bf006cb80 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf6400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf015c0e0 0x7f1bf0158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5000640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf015b7d0 0x7f1bf0153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf5a00640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf014d5a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1be000c410 tx=0x7f1be000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be0005ce0 con 0x7f1bf006cb80 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be0005e40 con 0x7f1bf006cb80 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be00072f0 con 0x7f1bf006cb80 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.405+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf0154310 con 0x7f1bf006cb80 2024-01-15T17:57:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.406+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf0153f90 con 0x7f1bf006cb80 2024-01-15T17:57:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.407+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf00686f0 con 0x7f1bf006cb80 2024-01-15T17:57:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.408+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1be0007510 con 0x7f1bf006cb80 2024-01-15T17:57:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.408+0000 7f1bed600640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bc806b2c0 0x7f1bc806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.409+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1be00892a0 con 0x7f1bf006cb80 2024-01-15T17:57:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.409+0000 7f1bf5000640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bc806b2c0 0x7f1bc806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.410+0000 7f1bf5000640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bc806b2c0 0x7f1bc806d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1bf01f3d50 tx=0x7f1bd8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.413+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1be00541d0 con 0x7f1bf006cb80 2024-01-15T17:57:35.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.644+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bf006b260 con 0x7f1bf006cb80 2024-01-15T17:57:35.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.646+0000 7f1bed600640 1 -- 172.21.15.2:0/3935227458 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f1be00577f0 con 0x7f1bf006cb80 2024-01-15T17:57:35.649 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:35.649 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:57:35.649 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:57:35.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bc806b2c0 msgr2=0x7f1bc806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bc806b2c0 0x7f1bc806d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1bf01f3d50 tx=0x7f1bd8009040 comp rx=0 tx=0).stop 2024-01-15T17:57:35.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 msgr2=0x7f1bf014d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf014d5a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1be000c410 tx=0x7f1be000c8d0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 shutdown_connections 2024-01-15T17:57:35.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bc806b2c0 0x7f1bc806d780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf015c0e0 0x7f1bf0158950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf015b7d0 0x7f1bf0153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 --2- 172.21.15.2:0/3935227458 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf006cb80 0x7f1bf014d5a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 >> 172.21.15.2:0/3935227458 conn(0x7f1bf006da20 msgr2=0x7f1bf00aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:35.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 shutdown_connections 2024-01-15T17:57:35.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.648+0000 7f1bf8400640 1 -- 172.21.15.2:0/3935227458 wait complete. 2024-01-15T17:57:35.669 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:35.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.887+0000 7f88ef200640 1 Processor -- start 2024-01-15T17:57:35.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.887+0000 7f88ef200640 1 -- start start 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.889+0000 7f88ef200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f88f015d340 0x7f88f00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.889+0000 7f88ef200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f88f00d1d90 0x7f88f00d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.889+0000 7f88ef200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88f00d3a60 con 0x7f88f00d3090 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.889+0000 7f88ef200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00b3de0 con 0x7f88f00d1d90 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.889+0000 7f88ef200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88f00b40c0 con 0x7f88f015d340 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f88f00d1d90 0x7f88f00d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f88f00d1d90 0x7f88f00d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49538/0 (socket says 172.21.15.2:49538) 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 -- 172.21.15.2:0/597563959 learned_addr learned my addr 172.21.15.2:0/597563959 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ede00640 1 --2- 172.21.15.2:0/597563959 >> v2:172.21.15.137:3300/0 conn(0x7f88f015d340 0x7f88f00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 -- 172.21.15.2:0/597563959 >> v2:172.21.15.137:3300/0 conn(0x7f88f015d340 msgr2=0x7f88f00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:35.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 --2- 172.21.15.2:0/597563959 >> v2:172.21.15.137:3300/0 conn(0x7f88f015d340 0x7f88f00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 -- 172.21.15.2:0/597563959 >> v1:172.21.15.137:6789/0 conn(0x7f88f00d3090 legacy=0x7f88f00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.890+0000 7f88ee800640 1 -- 172.21.15.2:0/597563959 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f00d0e80 con 0x7f88f00d1d90 2024-01-15T17:57:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.891+0000 7f88ee800640 1 --2- 172.21.15.2:0/597563959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d1d90 0x7f88f00d3520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88f008cf70 tx=0x7f88e802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed4c1d230aed1aa9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.891+0000 7f88ed400640 1 -- 172.21.15.2:0/597563959 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e802ca80 con 0x7f88f00d1d90 2024-01-15T17:57:35.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ed400640 1 -- 172.21.15.2:0/597563959 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e802cbe0 con 0x7f88f00d1d90 2024-01-15T17:57:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ed400640 1 -- 172.21.15.2:0/597563959 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e80343f0 con 0x7f88f00d1d90 2024-01-15T17:57:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 -- 172.21.15.2:0/597563959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d1d90 msgr2=0x7f88f00d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 --2- 172.21.15.2:0/597563959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d1d90 0x7f88f00d3520 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f88f008cf70 tx=0x7f88e802bee0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 -- 172.21.15.2:0/597563959 shutdown_connections 2024-01-15T17:57:35.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 --2- 172.21.15.2:0/597563959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d1d90 0x7f88f00d3520 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 --2- 172.21.15.2:0/597563959 >> v2:172.21.15.137:3300/0 conn(0x7f88f015d340 0x7f88f00cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 -- 172.21.15.2:0/597563959 >> 172.21.15.2:0/597563959 conn(0x7f88f008caf0 msgr2=0x7f88f009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:35.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 -- 172.21.15.2:0/597563959 shutdown_connections 2024-01-15T17:57:35.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.892+0000 7f88ef200640 1 -- 172.21.15.2:0/597563959 wait complete. 2024-01-15T17:57:35.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.894+0000 7f88ef200640 1 Processor -- start 2024-01-15T17:57:35.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.894+0000 7f88ef200640 1 -- start start 2024-01-15T17:57:35.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88ef200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88f00d1d90 0x7f88f015da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88ef200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 0x7f88f0160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88ef200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88f015d340 0x7f88f015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88ef200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f015b860 con 0x7f88f00d3090 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88ef200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00c86e0 con 0x7f88f00d1d90 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88ef200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88f00cb5b0 con 0x7f88f015d340 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.895+0000 7f88e7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 0x7f88f0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 0x7f88f0160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49546/0 (socket says 172.21.15.2:49546) 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 -- 172.21.15.2:0/4282743852 learned_addr learned my addr 172.21.15.2:0/4282743852 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88ede00640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88f00d1d90 0x7f88f015da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88ee800640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88f015d340 0x7f88f015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 -- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88f015d340 msgr2=0x7f88f015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88f015d340 0x7f88f015b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 -- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88f00d1d90 msgr2=0x7f88f015da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88f00d1d90 0x7f88f015da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 -- 172.21.15.2:0/4282743852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e8009b30 con 0x7f88f00d3090 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88ee800640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88f015d340 0x7f88f015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.896+0000 7f88e7e00640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 0x7f88f0160420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88d800c430 tx=0x7f88d800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.897+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8011790 con 0x7f88f00d3090 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.897+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d80118f0 con 0x7f88f00d3090 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.897+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f00c9a70 con 0x7f88f00d3090 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.897+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d8011ce0 con 0x7f88f00d3090 2024-01-15T17:57:35.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.897+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88f00c8c10 con 0x7f88f00d3090 2024-01-15T17:57:35.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.898+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88f0074770 con 0x7f88f00d3090 2024-01-15T17:57:35.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.899+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f88d801c460 con 0x7f88f00d3090 2024-01-15T17:57:35.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.900+0000 7f88e6000640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88c806b2c0 0x7f88c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:35.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.900+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88d8089b80 con 0x7f88f00d3090 2024-01-15T17:57:35.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.900+0000 7f88ede00640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88c806b2c0 0x7f88c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:35.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.901+0000 7f88ede00640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88c806b2c0 0x7f88c806d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f88e0006fd0 tx=0x7f88e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:35.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:35.904+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88d80549e0 con 0x7f88f00d3090 2024-01-15T17:57:36.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.099+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88f00678e0 con 0x7f88f00d3090 2024-01-15T17:57:36.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.102+0000 7f88e6000640 1 -- 172.21.15.2:0/4282743852 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f88d8089e60 con 0x7f88f00d3090 2024-01-15T17:57:36.106 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:36.106 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:36.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88c806b2c0 msgr2=0x7f88c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88c806b2c0 0x7f88c806d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f88e0006fd0 tx=0x7f88e0008040 comp rx=0 tx=0).stop 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 msgr2=0x7f88f0160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 0x7f88f0160420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f88d800c430 tx=0x7f88d800c8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 shutdown_connections 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88c806b2c0 0x7f88c806d780 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88f015d340 0x7f88f015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88f00d3090 0x7f88f0160420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 --2- 172.21.15.2:0/4282743852 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88f00d1d90 0x7f88f015da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 >> 172.21.15.2:0/4282743852 conn(0x7f88f008caf0 msgr2=0x7f88f00d14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 shutdown_connections 2024-01-15T17:57:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.105+0000 7f88ef200640 1 -- 172.21.15.2:0/4282743852 wait complete. 2024-01-15T17:57:36.128 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T17:57:36.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:57:36.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.340+0000 7f7892000640 1 Processor -- start 2024-01-15T17:57:36.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.340+0000 7f7892000640 1 -- start start 2024-01-15T17:57:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f7892000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f788c15c0e0 0x7f788c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f7892000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f788c06cb80 0x7f788c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f7892000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788c15cb00 con 0x7f788c15b7d0 2024-01-15T17:57:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f7892000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0ccec0 con 0x7f788c06cb80 2024-01-15T17:57:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f7892000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788c0cd1a0 con 0x7f788c15c0e0 2024-01-15T17:57:36.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f788aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f788c15b7d0 0x7f788c06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35384/0 (socket says 172.21.15.2:35384) 2024-01-15T17:57:36.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f788aa00640 1 -- 172.21.15.2:0/3342841065 learned_addr learned my addr 172.21.15.2:0/3342841065 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:36.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.342+0000 7f788be00640 1 --2- 172.21.15.2:0/3342841065 >> v2:172.21.15.2:3300/0 conn(0x7f788c06cb80 0x7f788c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788b400640 1 --2- 172.21.15.2:0/3342841065 >> v2:172.21.15.137:3300/0 conn(0x7f788c15c0e0 0x7f788c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788a000640 1 -- 172.21.15.2:0/3342841065 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3841551351 0 0) 0x7f788c15cb00 con 0x7f788c15b7d0 2024-01-15T17:57:36.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788a000640 1 -- 172.21.15.2:0/3342841065 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f786c003520 con 0x7f788c15b7d0 2024-01-15T17:57:36.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788be00640 1 -- 172.21.15.2:0/3342841065 >> v2:172.21.15.137:3300/0 conn(0x7f788c15c0e0 msgr2=0x7f788c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788be00640 1 --2- 172.21.15.2:0/3342841065 >> v2:172.21.15.137:3300/0 conn(0x7f788c15c0e0 0x7f788c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788be00640 1 -- 172.21.15.2:0/3342841065 >> v1:172.21.15.2:6789/0 conn(0x7f788c15b7d0 legacy=0x7f788c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.343+0000 7f788be00640 1 -- 172.21.15.2:0/3342841065 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c0cd480 con 0x7f788c06cb80 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.344+0000 7f788be00640 1 --2- 172.21.15.2:0/3342841065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c06cb80 0x7f788c15cf90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f788c0725f0 tx=0x7f788402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a731547892a2723 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.344+0000 7f788a000640 1 -- 172.21.15.2:0/3342841065 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884031830 con 0x7f788c06cb80 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.344+0000 7f788a000640 1 -- 172.21.15.2:0/3342841065 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7884031990 con 0x7f788c06cb80 2024-01-15T17:57:36.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.344+0000 7f788a000640 1 -- 172.21.15.2:0/3342841065 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7884031d20 con 0x7f788c06cb80 2024-01-15T17:57:36.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 -- 172.21.15.2:0/3342841065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c06cb80 msgr2=0x7f788c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 --2- 172.21.15.2:0/3342841065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c06cb80 0x7f788c15cf90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f788c0725f0 tx=0x7f788402b8d0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 -- 172.21.15.2:0/3342841065 shutdown_connections 2024-01-15T17:57:36.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 --2- 172.21.15.2:0/3342841065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c06cb80 0x7f788c15cf90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 --2- 172.21.15.2:0/3342841065 >> v2:172.21.15.137:3300/0 conn(0x7f788c15c0e0 0x7f788c0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 -- 172.21.15.2:0/3342841065 >> 172.21.15.2:0/3342841065 conn(0x7f788c06da20 msgr2=0x7f788c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:36.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 -- 172.21.15.2:0/3342841065 shutdown_connections 2024-01-15T17:57:36.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.345+0000 7f7892000640 1 -- 172.21.15.2:0/3342841065 wait complete. 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.347+0000 7f7892000640 1 Processor -- start 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.347+0000 7f7892000640 1 -- start start 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.348+0000 7f7892000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f788c06cb80 0x7f788c13e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.348+0000 7f7892000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f788c15b7d0 0x7f788c0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.348+0000 7f7892000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 0x7f788c071270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.348+0000 7f7892000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c0b3010 con 0x7f788c15c0e0 2024-01-15T17:57:36.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.348+0000 7f7892000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788c13eed0 con 0x7f788c06cb80 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.348+0000 7f7892000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788c153480 con 0x7f788c15b7d0 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 0x7f788c071270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 0x7f788c071270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49560/0 (socket says 172.21.15.2:49560) 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 -- 172.21.15.2:0/1453579627 learned_addr learned my addr 172.21.15.2:0/1453579627 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788aa00640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f788c15b7d0 0x7f788c0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788b400640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f788c06cb80 0x7f788c13e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 -- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f788c15b7d0 msgr2=0x7f788c0708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f788c15b7d0 0x7f788c0708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 -- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f788c06cb80 msgr2=0x7f788c13e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f788c06cb80 0x7f788c13e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 -- 172.21.15.2:0/1453579627 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884009400 con 0x7f788c15c0e0 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.349+0000 7f788be00640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 0x7f788c071270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7884034040 tx=0x7f78840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.350+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788403a830 con 0x7f788c15c0e0 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.350+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788403a9b0 con 0x7f788c15c0e0 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.350+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c153fa0 con 0x7f788c15c0e0 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.350+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788403ad60 con 0x7f788c15c0e0 2024-01-15T17:57:36.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.350+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f788c1539e0 con 0x7f788c15c0e0 2024-01-15T17:57:36.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.351+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f788c0b3150 con 0x7f788c15c0e0 2024-01-15T17:57:36.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.352+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f78840443f0 con 0x7f788c15c0e0 2024-01-15T17:57:36.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.353+0000 7f7883400640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786406b2c0 0x7f786406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.354+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78840abc20 con 0x7f788c15c0e0 2024-01-15T17:57:36.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.354+0000 7f788b400640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786406b2c0 0x7f786406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.355+0000 7f788b400640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786406b2c0 0x7f786406d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f788c1f63c0 tx=0x7f787c0039b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:36.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.358+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f788c0b3150 con 0x7f788c15c0e0 2024-01-15T17:57:36.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.547+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f788c0b3150 con 0x7f786406b2c0 2024-01-15T17:57:36.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.572+0000 7f7883400640 1 -- 172.21.15.2:0/1453579627 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264446 (secure 0 0 0) 0x7f788c0b3150 con 0x7f786406b2c0 2024-01-15T17:57:36.580 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:36.582 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.577+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786406b2c0 msgr2=0x7f786406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.577+0000 7f7892000640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786406b2c0 0x7f786406d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f788c1f63c0 tx=0x7f787c0039b0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.577+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 msgr2=0x7f788c071270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.577+0000 7f7892000640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 0x7f788c071270 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7884034040 tx=0x7f78840046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 shutdown_connections 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786406b2c0 0x7f786406d780 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f788c15c0e0 0x7f788c071270 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f788c15b7d0 0x7f788c0708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 --2- 172.21.15.2:0/1453579627 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f788c06cb80 0x7f788c13e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 >> 172.21.15.2:0/1453579627 conn(0x7f788c06da20 msgr2=0x7f788c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:36.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 shutdown_connections 2024-01-15T17:57:36.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.578+0000 7f7892000640 1 -- 172.21.15.2:0/1453579627 wait complete. 2024-01-15T17:57:36.602 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-15T17:57:35.368991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236116,"kb_used_data":2972,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738604,"statfs":{"total":773094113280,"available":772852330496,"internally_reserved":0,"allocated":3043328,"data_stored":1809672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025600"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139938+0000","last_change":"2024-01-15T17:57:32.145196+0000","last_active":"2024-01-15T17:57:33.139938+0000","last_peered":"2024-01-15T17:57:33.139938+0000","last_clean":"2024-01-15T17:57:33.139938+0000","last_became_active":"2024-01-15T17:57:32.144373+0000","last_became_peered":"2024-01-15T17:57:32.144373+0000","last_unstale":"2024-01-15T17:57:33.139938+0000","last_undegraded":"2024-01-15T17:57:33.139938+0000","last_fullsized":"2024-01-15T17:57:33.139938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136557+0000","last_change":"2024-01-15T17:57:32.143595+0000","last_active":"2024-01-15T17:57:33.136557+0000","last_peered":"2024-01-15T17:57:33.136557+0000","last_clean":"2024-01-15T17:57:33.136557+0000","last_became_active":"2024-01-15T17:57:32.143249+0000","last_became_peered":"2024-01-15T17:57:32.143249+0000","last_unstale":"2024-01-15T17:57:33.136557+0000","last_undegraded":"2024-01-15T17:57:33.136557+0000","last_fullsized":"2024-01-15T17:57:33.136557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419213+0000","last_change":"2024-01-15T17:57:32.143090+0000","last_active":"2024-01-15T17:57:33.419213+0000","last_peered":"2024-01-15T17:57:33.419213+0000","last_clean":"2024-01-15T17:57:33.419213+0000","last_became_active":"2024-01-15T17:57:32.142828+0000","last_became_peered":"2024-01-15T17:57:32.142828+0000","last_unstale":"2024-01-15T17:57:33.419213+0000","last_undegraded":"2024-01-15T17:57:33.419213+0000","last_fullsized":"2024-01-15T17:57:33.419213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139101+0000","last_change":"2024-01-15T17:57:32.146690+0000","last_active":"2024-01-15T17:57:33.139101+0000","last_peered":"2024-01-15T17:57:33.139101+0000","last_clean":"2024-01-15T17:57:33.139101+0000","last_became_active":"2024-01-15T17:57:32.142878+0000","last_became_peered":"2024-01-15T17:57:32.142878+0000","last_unstale":"2024-01-15T17:57:33.139101+0000","last_undegraded":"2024-01-15T17:57:33.139101+0000","last_fullsized":"2024-01-15T17:57:33.139101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141233+0000","last_change":"2024-01-15T17:57:32.145000+0000","last_active":"2024-01-15T17:57:33.141233+0000","last_peered":"2024-01-15T17:57:33.141233+0000","last_clean":"2024-01-15T17:57:33.141233+0000","last_became_active":"2024-01-15T17:57:32.144715+0000","last_became_peered":"2024-01-15T17:57:32.144715+0000","last_unstale":"2024-01-15T17:57:33.141233+0000","last_undegraded":"2024-01-15T17:57:33.141233+0000","last_fullsized":"2024-01-15T17:57:33.141233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:03.467001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138047+0000","last_change":"2024-01-15T17:57:32.154833+0000","last_active":"2024-01-15T17:57:33.138047+0000","last_peered":"2024-01-15T17:57:33.138047+0000","last_clean":"2024-01-15T17:57:33.138047+0000","last_became_active":"2024-01-15T17:57:32.153996+0000","last_became_peered":"2024-01-15T17:57:32.153996+0000","last_unstale":"2024-01-15T17:57:33.138047+0000","last_undegraded":"2024-01-15T17:57:33.138047+0000","last_fullsized":"2024-01-15T17:57:33.138047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138900+0000","last_change":"2024-01-15T17:57:32.147665+0000","last_active":"2024-01-15T17:57:33.138900+0000","last_peered":"2024-01-15T17:57:33.138900+0000","last_clean":"2024-01-15T17:57:33.138900+0000","last_became_active":"2024-01-15T17:57:32.147333+0000","last_became_peered":"2024-01-15T17:57:32.147333+0000","last_unstale":"2024-01-15T17:57:33.138900+0000","last_undegraded":"2024-01-15T17:57:33.138900+0000","last_fullsized":"2024-01-15T17:57:33.138900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415231+0000","last_change":"2024-01-15T17:57:32.144764+0000","last_active":"2024-01-15T17:57:33.415231+0000","last_peered":"2024-01-15T17:57:33.415231+0000","last_clean":"2024-01-15T17:57:33.415231+0000","last_became_active":"2024-01-15T17:57:32.144315+0000","last_became_peered":"2024-01-15T17:57:32.144315+0000","last_unstale":"2024-01-15T17:57:33.415231+0000","last_undegraded":"2024-01-15T17:57:33.415231+0000","last_fullsized":"2024-01-15T17:57:33.415231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140177+0000","last_change":"2024-01-15T17:57:32.157864+0000","last_active":"2024-01-15T17:57:33.140177+0000","last_peered":"2024-01-15T17:57:33.140177+0000","last_clean":"2024-01-15T17:57:33.140177+0000","last_became_active":"2024-01-15T17:57:32.157556+0000","last_became_peered":"2024-01-15T17:57:32.157556+0000","last_unstale":"2024-01-15T17:57:33.140177+0000","last_undegraded":"2024-01-15T17:57:33.140177+0000","last_fullsized":"2024-01-15T17:57:33.140177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415132+0000","last_change":"2024-01-15T17:57:32.161393+0000","last_active":"2024-01-15T17:57:33.415132+0000","last_peered":"2024-01-15T17:57:33.415132+0000","last_clean":"2024-01-15T17:57:33.415132+0000","last_became_active":"2024-01-15T17:57:32.161079+0000","last_became_peered":"2024-01-15T17:57:32.161079+0000","last_unstale":"2024-01-15T17:57:33.415132+0000","last_undegraded":"2024-01-15T17:57:33.415132+0000","last_fullsized":"2024-01-15T17:57:33.415132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414932+0000","last_change":"2024-01-15T17:57:32.145780+0000","last_active":"2024-01-15T17:57:33.414932+0000","last_peered":"2024-01-15T17:57:33.414932+0000","last_clean":"2024-01-15T17:57:33.414932+0000","last_became_active":"2024-01-15T17:57:32.145409+0000","last_became_peered":"2024-01-15T17:57:32.145409+0000","last_unstale":"2024-01-15T17:57:33.414932+0000","last_undegraded":"2024-01-15T17:57:33.414932+0000","last_fullsized":"2024-01-15T17:57:33.414932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186642+0000","last_change":"2024-01-15T17:57:32.144056+0000","last_active":"2024-01-15T17:57:33.186642+0000","last_peered":"2024-01-15T17:57:33.186642+0000","last_clean":"2024-01-15T17:57:33.186642+0000","last_became_active":"2024-01-15T17:57:32.143687+0000","last_became_peered":"2024-01-15T17:57:32.143687+0000","last_unstale":"2024-01-15T17:57:33.186642+0000","last_undegraded":"2024-01-15T17:57:33.186642+0000","last_fullsized":"2024-01-15T17:57:33.186642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417307+0000","last_change":"2024-01-15T17:57:32.141036+0000","last_active":"2024-01-15T17:57:33.417307+0000","last_peered":"2024-01-15T17:57:33.417307+0000","last_clean":"2024-01-15T17:57:33.417307+0000","last_became_active":"2024-01-15T17:57:32.139707+0000","last_became_peered":"2024-01-15T17:57:32.139707+0000","last_unstale":"2024-01-15T17:57:33.417307+0000","last_undegraded":"2024-01-15T17:57:33.417307+0000","last_fullsized":"2024-01-15T17:57:33.417307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415834+0000","last_change":"2024-01-15T17:57:32.148492+0000","last_active":"2024-01-15T17:57:33.415834+0000","last_peered":"2024-01-15T17:57:33.415834+0000","last_clean":"2024-01-15T17:57:33.415834+0000","last_became_active":"2024-01-15T17:57:32.148169+0000","last_became_peered":"2024-01-15T17:57:32.148169+0000","last_unstale":"2024-01-15T17:57:33.415834+0000","last_undegraded":"2024-01-15T17:57:33.415834+0000","last_fullsized":"2024-01-15T17:57:33.415834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:00.974439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.150453+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.150115+0000","last_became_peered":"2024-01-15T17:57:32.150115+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.137808+0000","last_change":"2024-01-15T17:57:32.148256+0000","last_active":"2024-01-15T17:57:33.137808+0000","last_peered":"2024-01-15T17:57:33.137808+0000","last_clean":"2024-01-15T17:57:33.137808+0000","last_became_active":"2024-01-15T17:57:32.147620+0000","last_became_peered":"2024-01-15T17:57:32.147620+0000","last_unstale":"2024-01-15T17:57:33.137808+0000","last_undegraded":"2024-01-15T17:57:33.137808+0000","last_fullsized":"2024-01-15T17:57:33.137808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140840+0000","last_change":"2024-01-15T17:57:32.139828+0000","last_active":"2024-01-15T17:57:33.140840+0000","last_peered":"2024-01-15T17:57:33.140840+0000","last_clean":"2024-01-15T17:57:33.140840+0000","last_became_active":"2024-01-15T17:57:32.139522+0000","last_became_peered":"2024-01-15T17:57:32.139522+0000","last_unstale":"2024-01-15T17:57:33.140840+0000","last_undegraded":"2024-01-15T17:57:33.140840+0000","last_fullsized":"2024-01-15T17:57:33.140840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416706+0000","last_change":"2024-01-15T17:57:32.140652+0000","last_active":"2024-01-15T17:57:33.416706+0000","last_peered":"2024-01-15T17:57:33.416706+0000","last_clean":"2024-01-15T17:57:33.416706+0000","last_became_active":"2024-01-15T17:57:32.140366+0000","last_became_peered":"2024-01-15T17:57:32.140366+0000","last_unstale":"2024-01-15T17:57:33.416706+0000","last_undegraded":"2024-01-15T17:57:33.416706+0000","last_fullsized":"2024-01-15T17:57:33.416706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141414+0000","last_change":"2024-01-15T17:57:32.149137+0000","last_active":"2024-01-15T17:57:33.141414+0000","last_peered":"2024-01-15T17:57:33.141414+0000","last_clean":"2024-01-15T17:57:33.141414+0000","last_became_active":"2024-01-15T17:57:32.148658+0000","last_became_peered":"2024-01-15T17:57:32.148658+0000","last_unstale":"2024-01-15T17:57:33.141414+0000","last_undegraded":"2024-01-15T17:57:33.141414+0000","last_fullsized":"2024-01-15T17:57:33.141414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:18.642472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414742+0000","last_change":"2024-01-15T17:57:32.156212+0000","last_active":"2024-01-15T17:57:33.414742+0000","last_peered":"2024-01-15T17:57:33.414742+0000","last_clean":"2024-01-15T17:57:33.414742+0000","last_became_active":"2024-01-15T17:57:32.153278+0000","last_became_peered":"2024-01-15T17:57:32.153278+0000","last_unstale":"2024-01-15T17:57:33.414742+0000","last_undegraded":"2024-01-15T17:57:33.414742+0000","last_fullsized":"2024-01-15T17:57:33.414742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417083+0000","last_change":"2024-01-15T17:57:32.147107+0000","last_active":"2024-01-15T17:57:33.417083+0000","last_peered":"2024-01-15T17:57:33.417083+0000","last_clean":"2024-01-15T17:57:33.417083+0000","last_became_active":"2024-01-15T17:57:32.146730+0000","last_became_peered":"2024-01-15T17:57:32.146730+0000","last_unstale":"2024-01-15T17:57:33.417083+0000","last_undegraded":"2024-01-15T17:57:33.417083+0000","last_fullsized":"2024-01-15T17:57:33.417083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138213+0000","last_change":"2024-01-15T17:57:32.139188+0000","last_active":"2024-01-15T17:57:33.138213+0000","last_peered":"2024-01-15T17:57:33.138213+0000","last_clean":"2024-01-15T17:57:33.138213+0000","last_became_active":"2024-01-15T17:57:32.138898+0000","last_became_peered":"2024-01-15T17:57:32.138898+0000","last_unstale":"2024-01-15T17:57:33.138213+0000","last_undegraded":"2024-01-15T17:57:33.138213+0000","last_fullsized":"2024-01-15T17:57:33.138213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416604+0000","last_change":"2024-01-15T17:57:32.148598+0000","last_active":"2024-01-15T17:57:33.416604+0000","last_peered":"2024-01-15T17:57:33.416604+0000","last_clean":"2024-01-15T17:57:33.416604+0000","last_became_active":"2024-01-15T17:57:32.148266+0000","last_became_peered":"2024-01-15T17:57:32.148266+0000","last_unstale":"2024-01-15T17:57:33.416604+0000","last_undegraded":"2024-01-15T17:57:33.416604+0000","last_fullsized":"2024-01-15T17:57:33.416604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138790+0000","last_change":"2024-01-15T17:57:32.135029+0000","last_active":"2024-01-15T17:57:33.138790+0000","last_peered":"2024-01-15T17:57:33.138790+0000","last_clean":"2024-01-15T17:57:33.138790+0000","last_became_active":"2024-01-15T17:57:32.134762+0000","last_became_peered":"2024-01-15T17:57:32.134762+0000","last_unstale":"2024-01-15T17:57:33.138790+0000","last_undegraded":"2024-01-15T17:57:33.138790+0000","last_fullsized":"2024-01-15T17:57:33.138790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418117+0000","last_change":"2024-01-15T17:57:32.153167+0000","last_active":"2024-01-15T17:57:33.418117+0000","last_peered":"2024-01-15T17:57:33.418117+0000","last_clean":"2024-01-15T17:57:33.418117+0000","last_became_active":"2024-01-15T17:57:32.151003+0000","last_became_peered":"2024-01-15T17:57:32.151003+0000","last_unstale":"2024-01-15T17:57:33.418117+0000","last_undegraded":"2024-01-15T17:57:33.418117+0000","last_fullsized":"2024-01-15T17:57:33.418117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:32.142988+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:32.142613+0000","last_became_peered":"2024-01-15T17:57:32.142613+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136341+0000","last_change":"2024-01-15T17:57:32.134987+0000","last_active":"2024-01-15T17:57:33.136341+0000","last_peered":"2024-01-15T17:57:33.136341+0000","last_clean":"2024-01-15T17:57:33.136341+0000","last_became_active":"2024-01-15T17:57:32.134719+0000","last_became_peered":"2024-01-15T17:57:32.134719+0000","last_unstale":"2024-01-15T17:57:33.136341+0000","last_undegraded":"2024-01-15T17:57:33.136341+0000","last_fullsized":"2024-01-15T17:57:33.136341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418595+0000","last_change":"2024-01-15T17:57:32.153961+0000","last_active":"2024-01-15T17:57:33.418595+0000","last_peered":"2024-01-15T17:57:33.418595+0000","last_clean":"2024-01-15T17:57:33.418595+0000","last_became_active":"2024-01-15T17:57:32.151097+0000","last_became_peered":"2024-01-15T17:57:32.151097+0000","last_unstale":"2024-01-15T17:57:33.418595+0000","last_undegraded":"2024-01-15T17:57:33.418595+0000","last_fullsized":"2024-01-15T17:57:33.418595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418195+0000","last_change":"2024-01-15T17:57:32.144509+0000","last_active":"2024-01-15T17:57:33.418195+0000","last_peered":"2024-01-15T17:57:33.418195+0000","last_clean":"2024-01-15T17:57:33.418195+0000","last_became_active":"2024-01-15T17:57:32.143921+0000","last_became_peered":"2024-01-15T17:57:32.143921+0000","last_unstale":"2024-01-15T17:57:33.418195+0000","last_undegraded":"2024-01-15T17:57:33.418195+0000","last_fullsized":"2024-01-15T17:57:33.418195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138415+0000","last_change":"2024-01-15T17:57:32.137686+0000","last_active":"2024-01-15T17:57:33.138415+0000","last_peered":"2024-01-15T17:57:33.138415+0000","last_clean":"2024-01-15T17:57:33.138415+0000","last_became_active":"2024-01-15T17:57:32.137409+0000","last_became_peered":"2024-01-15T17:57:32.137409+0000","last_unstale":"2024-01-15T17:57:33.138415+0000","last_undegraded":"2024-01-15T17:57:33.138415+0000","last_fullsized":"2024-01-15T17:57:33.138415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414292+0000","last_change":"2024-01-15T17:57:32.129661+0000","last_active":"2024-01-15T17:57:33.414292+0000","last_peered":"2024-01-15T17:57:33.414292+0000","last_clean":"2024-01-15T17:57:33.414292+0000","last_became_active":"2024-01-15T17:57:32.129283+0000","last_became_peered":"2024-01-15T17:57:32.129283+0000","last_unstale":"2024-01-15T17:57:33.414292+0000","last_undegraded":"2024-01-15T17:57:33.414292+0000","last_fullsized":"2024-01-15T17:57:33.414292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414350+0000","last_change":"2024-01-15T17:57:32.147901+0000","last_active":"2024-01-15T17:57:33.414350+0000","last_peered":"2024-01-15T17:57:33.414350+0000","last_clean":"2024-01-15T17:57:33.414350+0000","last_became_active":"2024-01-15T17:57:32.147185+0000","last_became_peered":"2024-01-15T17:57:32.147185+0000","last_unstale":"2024-01-15T17:57:33.414350+0000","last_undegraded":"2024-01-15T17:57:33.414350+0000","last_fullsized":"2024-01-15T17:57:33.414350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186262+0000","last_change":"2024-01-15T17:57:32.138254+0000","last_active":"2024-01-15T17:57:33.186262+0000","last_peered":"2024-01-15T17:57:33.186262+0000","last_clean":"2024-01-15T17:57:33.186262+0000","last_became_active":"2024-01-15T17:57:32.137849+0000","last_became_peered":"2024-01-15T17:57:32.137849+0000","last_unstale":"2024-01-15T17:57:33.186262+0000","last_undegraded":"2024-01-15T17:57:33.186262+0000","last_fullsized":"2024-01-15T17:57:33.186262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:12.340083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417690+0000","last_change":"2024-01-15T17:57:31.132496+0000","last_active":"2024-01-15T17:57:33.417690+0000","last_peered":"2024-01-15T17:57:33.417690+0000","last_clean":"2024-01-15T17:57:33.417690+0000","last_became_active":"2024-01-15T17:57:31.132095+0000","last_became_peered":"2024-01-15T17:57:31.132095+0000","last_unstale":"2024-01-15T17:57:33.417690+0000","last_undegraded":"2024-01-15T17:57:33.417690+0000","last_fullsized":"2024-01-15T17:57:33.417690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418981+0000","last_change":"2024-01-15T17:57:32.154160+0000","last_active":"2024-01-15T17:57:33.418981+0000","last_peered":"2024-01-15T17:57:33.418981+0000","last_clean":"2024-01-15T17:57:33.418981+0000","last_became_active":"2024-01-15T17:57:32.153881+0000","last_became_peered":"2024-01-15T17:57:32.153881+0000","last_unstale":"2024-01-15T17:57:33.418981+0000","last_undegraded":"2024-01-15T17:57:33.418981+0000","last_fullsized":"2024-01-15T17:57:33.418981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414244+0000","last_change":"2024-01-15T17:57:31.143303+0000","last_active":"2024-01-15T17:57:33.414244+0000","last_peered":"2024-01-15T17:57:33.414244+0000","last_clean":"2024-01-15T17:57:33.414244+0000","last_became_active":"2024-01-15T17:57:31.142917+0000","last_became_peered":"2024-01-15T17:57:31.142917+0000","last_unstale":"2024-01-15T17:57:33.414244+0000","last_undegraded":"2024-01-15T17:57:33.414244+0000","last_fullsized":"2024-01-15T17:57:33.414244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416836+0000","last_change":"2024-01-15T17:57:31.137389+0000","last_active":"2024-01-15T17:57:33.416836+0000","last_peered":"2024-01-15T17:57:33.416836+0000","last_clean":"2024-01-15T17:57:33.416836+0000","last_became_active":"2024-01-15T17:57:31.137115+0000","last_became_peered":"2024-01-15T17:57:31.137115+0000","last_unstale":"2024-01-15T17:57:33.416836+0000","last_undegraded":"2024-01-15T17:57:33.416836+0000","last_fullsized":"2024-01-15T17:57:33.416836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415574+0000","last_change":"2024-01-15T17:57:32.156574+0000","last_active":"2024-01-15T17:57:33.415574+0000","last_peered":"2024-01-15T17:57:33.415574+0000","last_clean":"2024-01-15T17:57:33.415574+0000","last_became_active":"2024-01-15T17:57:32.156179+0000","last_became_peered":"2024-01-15T17:57:32.156179+0000","last_unstale":"2024-01-15T17:57:33.415574+0000","last_undegraded":"2024-01-15T17:57:33.415574+0000","last_fullsized":"2024-01-15T17:57:33.415574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139052+0000","last_change":"2024-01-15T17:57:31.132492+0000","last_active":"2024-01-15T17:57:33.139052+0000","last_peered":"2024-01-15T17:57:33.139052+0000","last_clean":"2024-01-15T17:57:33.139052+0000","last_became_active":"2024-01-15T17:57:31.132141+0000","last_became_peered":"2024-01-15T17:57:31.132141+0000","last_unstale":"2024-01-15T17:57:33.139052+0000","last_undegraded":"2024-01-15T17:57:33.139052+0000","last_fullsized":"2024-01-15T17:57:33.139052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.134222+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.133920+0000","last_became_peered":"2024-01-15T17:57:32.133920+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:31.144178+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:31.143915+0000","last_became_peered":"2024-01-15T17:57:31.143915+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140200+0000","last_change":"2024-01-15T17:57:32.141022+0000","last_active":"2024-01-15T17:57:33.140200+0000","last_peered":"2024-01-15T17:57:33.140200+0000","last_clean":"2024-01-15T17:57:33.140200+0000","last_became_active":"2024-01-15T17:57:32.140122+0000","last_became_peered":"2024-01-15T17:57:32.140122+0000","last_unstale":"2024-01-15T17:57:33.140200+0000","last_undegraded":"2024-01-15T17:57:33.140200+0000","last_fullsized":"2024-01-15T17:57:33.140200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140234+0000","last_change":"2024-01-15T17:57:31.146237+0000","last_active":"2024-01-15T17:57:33.140234+0000","last_peered":"2024-01-15T17:57:33.140234+0000","last_clean":"2024-01-15T17:57:33.140234+0000","last_became_active":"2024-01-15T17:57:31.145808+0000","last_became_peered":"2024-01-15T17:57:31.145808+0000","last_unstale":"2024-01-15T17:57:33.140234+0000","last_undegraded":"2024-01-15T17:57:33.140234+0000","last_fullsized":"2024-01-15T17:57:33.140234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:05.346229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419713+0000","last_change":"2024-01-15T17:57:32.142125+0000","last_active":"2024-01-15T17:57:33.419713+0000","last_peered":"2024-01-15T17:57:33.419713+0000","last_clean":"2024-01-15T17:57:33.419713+0000","last_became_active":"2024-01-15T17:57:32.141806+0000","last_became_peered":"2024-01-15T17:57:32.141806+0000","last_unstale":"2024-01-15T17:57:33.419713+0000","last_undegraded":"2024-01-15T17:57:33.419713+0000","last_fullsized":"2024-01-15T17:57:33.419713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136484+0000","last_change":"2024-01-15T17:57:31.131511+0000","last_active":"2024-01-15T17:57:33.136484+0000","last_peered":"2024-01-15T17:57:33.136484+0000","last_clean":"2024-01-15T17:57:33.136484+0000","last_became_active":"2024-01-15T17:57:31.131207+0000","last_became_peered":"2024-01-15T17:57:31.131207+0000","last_unstale":"2024-01-15T17:57:33.136484+0000","last_undegraded":"2024-01-15T17:57:33.136484+0000","last_fullsized":"2024-01-15T17:57:33.136484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186851+0000","last_change":"2024-01-15T17:57:32.141464+0000","last_active":"2024-01-15T17:57:33.186851+0000","last_peered":"2024-01-15T17:57:33.186851+0000","last_clean":"2024-01-15T17:57:33.186851+0000","last_became_active":"2024-01-15T17:57:32.140204+0000","last_became_peered":"2024-01-15T17:57:32.140204+0000","last_unstale":"2024-01-15T17:57:33.186851+0000","last_undegraded":"2024-01-15T17:57:33.186851+0000","last_fullsized":"2024-01-15T17:57:33.186851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:23.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415055+0000","last_change":"2024-01-15T17:57:31.140465+0000","last_active":"2024-01-15T17:57:33.415055+0000","last_peered":"2024-01-15T17:57:33.415055+0000","last_clean":"2024-01-15T17:57:33.415055+0000","last_became_active":"2024-01-15T17:57:31.140132+0000","last_became_peered":"2024-01-15T17:57:31.140132+0000","last_unstale":"2024-01-15T17:57:33.415055+0000","last_undegraded":"2024-01-15T17:57:33.415055+0000","last_fullsized":"2024-01-15T17:57:33.415055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141003+0000","last_change":"2024-01-15T17:57:32.140539+0000","last_active":"2024-01-15T17:57:33.141003+0000","last_peered":"2024-01-15T17:57:33.141003+0000","last_clean":"2024-01-15T17:57:33.141003+0000","last_became_active":"2024-01-15T17:57:32.140117+0000","last_became_peered":"2024-01-15T17:57:32.140117+0000","last_unstale":"2024-01-15T17:57:33.141003+0000","last_undegraded":"2024-01-15T17:57:33.141003+0000","last_fullsized":"2024-01-15T17:57:33.141003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418517+0000","last_change":"2024-01-15T17:57:31.134124+0000","last_active":"2024-01-15T17:57:33.418517+0000","last_peered":"2024-01-15T17:57:33.418517+0000","last_clean":"2024-01-15T17:57:33.418517+0000","last_became_active":"2024-01-15T17:57:31.133698+0000","last_became_peered":"2024-01-15T17:57:31.133698+0000","last_unstale":"2024-01-15T17:57:33.418517+0000","last_undegraded":"2024-01-15T17:57:33.418517+0000","last_fullsized":"2024-01-15T17:57:33.418517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416094+0000","last_change":"2024-01-15T17:57:32.148065+0000","last_active":"2024-01-15T17:57:33.416094+0000","last_peered":"2024-01-15T17:57:33.416094+0000","last_clean":"2024-01-15T17:57:33.416094+0000","last_became_active":"2024-01-15T17:57:32.145370+0000","last_became_peered":"2024-01-15T17:57:32.145370+0000","last_unstale":"2024-01-15T17:57:33.416094+0000","last_undegraded":"2024-01-15T17:57:33.416094+0000","last_fullsized":"2024-01-15T17:57:33.416094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:46.538517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138441+0000","last_change":"2024-01-15T17:57:31.128237+0000","last_active":"2024-01-15T17:57:33.138441+0000","last_peered":"2024-01-15T17:57:33.138441+0000","last_clean":"2024-01-15T17:57:33.138441+0000","last_became_active":"2024-01-15T17:57:31.127956+0000","last_became_peered":"2024-01-15T17:57:31.127956+0000","last_unstale":"2024-01-15T17:57:33.138441+0000","last_undegraded":"2024-01-15T17:57:33.138441+0000","last_fullsized":"2024-01-15T17:57:33.138441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141534+0000","last_change":"2024-01-15T17:57:32.150884+0000","last_active":"2024-01-15T17:57:33.141534+0000","last_peered":"2024-01-15T17:57:33.141534+0000","last_clean":"2024-01-15T17:57:33.141534+0000","last_became_active":"2024-01-15T17:57:32.150381+0000","last_became_peered":"2024-01-15T17:57:32.150381+0000","last_unstale":"2024-01-15T17:57:33.141534+0000","last_undegraded":"2024-01-15T17:57:33.141534+0000","last_fullsized":"2024-01-15T17:57:33.141534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:23.429091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414222+0000","last_change":"2024-01-15T17:57:31.138127+0000","last_active":"2024-01-15T17:57:33.414222+0000","last_peered":"2024-01-15T17:57:33.414222+0000","last_clean":"2024-01-15T17:57:33.414222+0000","last_became_active":"2024-01-15T17:57:31.137579+0000","last_became_peered":"2024-01-15T17:57:31.137579+0000","last_unstale":"2024-01-15T17:57:33.414222+0000","last_undegraded":"2024-01-15T17:57:33.414222+0000","last_fullsized":"2024-01-15T17:57:33.414222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414356+0000","last_change":"2024-01-15T17:57:32.150432+0000","last_active":"2024-01-15T17:57:33.414356+0000","last_peered":"2024-01-15T17:57:33.414356+0000","last_clean":"2024-01-15T17:57:33.414356+0000","last_became_active":"2024-01-15T17:57:32.147572+0000","last_became_peered":"2024-01-15T17:57:32.147572+0000","last_unstale":"2024-01-15T17:57:33.414356+0000","last_undegraded":"2024-01-15T17:57:33.414356+0000","last_fullsized":"2024-01-15T17:57:33.414356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414449+0000","last_change":"2024-01-15T17:57:31.134499+0000","last_active":"2024-01-15T17:57:33.414449+0000","last_peered":"2024-01-15T17:57:33.414449+0000","last_clean":"2024-01-15T17:57:33.414449+0000","last_became_active":"2024-01-15T17:57:31.131997+0000","last_became_peered":"2024-01-15T17:57:31.131997+0000","last_unstale":"2024-01-15T17:57:33.414449+0000","last_undegraded":"2024-01-15T17:57:33.414449+0000","last_fullsized":"2024-01-15T17:57:33.414449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:11.214052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139065+0000","last_change":"2024-01-15T17:57:32.151412+0000","last_active":"2024-01-15T17:57:33.139065+0000","last_peered":"2024-01-15T17:57:33.139065+0000","last_clean":"2024-01-15T17:57:33.139065+0000","last_became_active":"2024-01-15T17:57:32.151061+0000","last_became_peered":"2024-01-15T17:57:32.151061+0000","last_unstale":"2024-01-15T17:57:33.139065+0000","last_undegraded":"2024-01-15T17:57:33.139065+0000","last_fullsized":"2024-01-15T17:57:33.139065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186594+0000","last_change":"2024-01-15T17:57:31.133132+0000","last_active":"2024-01-15T17:57:33.186594+0000","last_peered":"2024-01-15T17:57:33.186594+0000","last_clean":"2024-01-15T17:57:33.186594+0000","last_became_active":"2024-01-15T17:57:31.131606+0000","last_became_peered":"2024-01-15T17:57:31.131606+0000","last_unstale":"2024-01-15T17:57:33.186594+0000","last_undegraded":"2024-01-15T17:57:33.186594+0000","last_fullsized":"2024-01-15T17:57:33.186594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:58:03.663628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139362+0000","last_change":"2024-01-15T17:57:32.152397+0000","last_active":"2024-01-15T17:57:33.139362+0000","last_peered":"2024-01-15T17:57:33.139362+0000","last_clean":"2024-01-15T17:57:33.139362+0000","last_became_active":"2024-01-15T17:57:32.152096+0000","last_became_peered":"2024-01-15T17:57:32.152096+0000","last_unstale":"2024-01-15T17:57:33.139362+0000","last_undegraded":"2024-01-15T17:57:33.139362+0000","last_fullsized":"2024-01-15T17:57:33.139362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417449+0000","last_change":"2024-01-15T17:57:31.127107+0000","last_active":"2024-01-15T17:57:33.417449+0000","last_peered":"2024-01-15T17:57:33.417449+0000","last_clean":"2024-01-15T17:57:33.417449+0000","last_became_active":"2024-01-15T17:57:31.126871+0000","last_became_peered":"2024-01-15T17:57:31.126871+0000","last_unstale":"2024-01-15T17:57:33.417449+0000","last_undegraded":"2024-01-15T17:57:33.417449+0000","last_fullsized":"2024-01-15T17:57:33.417449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:20.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140382+0000","last_change":"2024-01-15T17:57:32.150858+0000","last_active":"2024-01-15T17:57:33.140382+0000","last_peered":"2024-01-15T17:57:33.140382+0000","last_clean":"2024-01-15T17:57:33.140382+0000","last_became_active":"2024-01-15T17:57:32.150312+0000","last_became_peered":"2024-01-15T17:57:32.150312+0000","last_unstale":"2024-01-15T17:57:33.140382+0000","last_undegraded":"2024-01-15T17:57:33.140382+0000","last_fullsized":"2024-01-15T17:57:33.140382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136681+0000","last_change":"2024-01-15T17:57:31.136226+0000","last_active":"2024-01-15T17:57:33.136681+0000","last_peered":"2024-01-15T17:57:33.136681+0000","last_clean":"2024-01-15T17:57:33.136681+0000","last_became_active":"2024-01-15T17:57:31.135879+0000","last_became_peered":"2024-01-15T17:57:31.135879+0000","last_unstale":"2024-01-15T17:57:33.136681+0000","last_undegraded":"2024-01-15T17:57:33.136681+0000","last_fullsized":"2024-01-15T17:57:33.136681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.149505+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.144115+0000","last_became_peered":"2024-01-15T17:57:32.144115+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415379+0000","last_change":"2024-01-15T17:57:31.130353+0000","last_active":"2024-01-15T17:57:33.415379+0000","last_peered":"2024-01-15T17:57:33.415379+0000","last_clean":"2024-01-15T17:57:33.415379+0000","last_became_active":"2024-01-15T17:57:31.130063+0000","last_became_peered":"2024-01-15T17:57:31.130063+0000","last_unstale":"2024-01-15T17:57:33.415379+0000","last_undegraded":"2024-01-15T17:57:33.415379+0000","last_fullsized":"2024-01-15T17:57:33.415379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:42.825666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141047+0000","last_change":"2024-01-15T17:57:32.148976+0000","last_active":"2024-01-15T17:57:33.141047+0000","last_peered":"2024-01-15T17:57:33.141047+0000","last_clean":"2024-01-15T17:57:33.141047+0000","last_became_active":"2024-01-15T17:57:32.148285+0000","last_became_peered":"2024-01-15T17:57:32.148285+0000","last_unstale":"2024-01-15T17:57:33.141047+0000","last_undegraded":"2024-01-15T17:57:33.141047+0000","last_fullsized":"2024-01-15T17:57:33.141047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T17:57:32.126659+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:57:32.126659+0000","last_peered":"2024-01-15T17:57:32.126659+0000","last_clean":"2024-01-15T17:57:32.126659+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:57:32.126659+0000","last_undegraded":"2024-01-15T17:57:32.126659+0000","last_fullsized":"2024-01-15T17:57:32.126659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138248+0000","last_change":"2024-01-15T17:57:31.127921+0000","last_active":"2024-01-15T17:57:33.138248+0000","last_peered":"2024-01-15T17:57:33.138248+0000","last_clean":"2024-01-15T17:57:33.138248+0000","last_became_active":"2024-01-15T17:57:31.126423+0000","last_became_peered":"2024-01-15T17:57:31.126423+0000","last_unstale":"2024-01-15T17:57:33.138248+0000","last_undegraded":"2024-01-15T17:57:33.138248+0000","last_fullsized":"2024-01-15T17:57:33.138248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:43:31.770732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418628+0000","last_change":"2024-01-15T17:57:32.151647+0000","last_active":"2024-01-15T17:57:33.418628+0000","last_peered":"2024-01-15T17:57:33.418628+0000","last_clean":"2024-01-15T17:57:33.418628+0000","last_became_active":"2024-01-15T17:57:32.149470+0000","last_became_peered":"2024-01-15T17:57:32.149470+0000","last_unstale":"2024-01-15T17:57:33.418628+0000","last_undegraded":"2024-01-15T17:57:33.418628+0000","last_fullsized":"2024-01-15T17:57:33.418628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139414+0000","last_change":"2024-01-15T17:57:31.141208+0000","last_active":"2024-01-15T17:57:33.139414+0000","last_peered":"2024-01-15T17:57:33.139414+0000","last_clean":"2024-01-15T17:57:33.139414+0000","last_became_active":"2024-01-15T17:57:31.140756+0000","last_became_peered":"2024-01-15T17:57:31.140756+0000","last_unstale":"2024-01-15T17:57:33.139414+0000","last_undegraded":"2024-01-15T17:57:33.139414+0000","last_fullsized":"2024-01-15T17:57:33.139414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:17:40.803403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415471+0000","last_change":"2024-01-15T17:57:32.150522+0000","last_active":"2024-01-15T17:57:33.415471+0000","last_peered":"2024-01-15T17:57:33.415471+0000","last_clean":"2024-01-15T17:57:33.415471+0000","last_became_active":"2024-01-15T17:57:32.150224+0000","last_became_peered":"2024-01-15T17:57:32.150224+0000","last_unstale":"2024-01-15T17:57:33.415471+0000","last_undegraded":"2024-01-15T17:57:33.415471+0000","last_fullsized":"2024-01-15T17:57:33.415471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:03.799210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414399+0000","last_change":"2024-01-15T17:57:31.127558+0000","last_active":"2024-01-15T17:57:33.414399+0000","last_peered":"2024-01-15T17:57:33.414399+0000","last_clean":"2024-01-15T17:57:33.414399+0000","last_became_active":"2024-01-15T17:57:31.125325+0000","last_became_peered":"2024-01-15T17:57:31.125325+0000","last_unstale":"2024-01-15T17:57:33.414399+0000","last_undegraded":"2024-01-15T17:57:33.414399+0000","last_fullsized":"2024-01-15T17:57:33.414399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:56.231915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136344+0000","last_change":"2024-01-15T17:57:32.134693+0000","last_active":"2024-01-15T17:57:33.136344+0000","last_peered":"2024-01-15T17:57:33.136344+0000","last_clean":"2024-01-15T17:57:33.136344+0000","last_became_active":"2024-01-15T17:57:32.133773+0000","last_became_peered":"2024-01-15T17:57:32.133773+0000","last_unstale":"2024-01-15T17:57:33.136344+0000","last_undegraded":"2024-01-15T17:57:33.136344+0000","last_fullsized":"2024-01-15T17:57:33.136344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138654+0000","last_change":"2024-01-15T17:57:31.133660+0000","last_active":"2024-01-15T17:57:33.138654+0000","last_peered":"2024-01-15T17:57:33.138654+0000","last_clean":"2024-01-15T17:57:33.138654+0000","last_became_active":"2024-01-15T17:57:31.132627+0000","last_became_peered":"2024-01-15T17:57:31.132627+0000","last_unstale":"2024-01-15T17:57:33.138654+0000","last_undegraded":"2024-01-15T17:57:33.138654+0000","last_fullsized":"2024-01-15T17:57:33.138654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186541+0000","last_change":"2024-01-15T17:57:32.147182+0000","last_active":"2024-01-15T17:57:33.186541+0000","last_peered":"2024-01-15T17:57:33.186541+0000","last_clean":"2024-01-15T17:57:33.186541+0000","last_became_active":"2024-01-15T17:57:32.146838+0000","last_became_peered":"2024-01-15T17:57:32.146838+0000","last_unstale":"2024-01-15T17:57:33.186541+0000","last_undegraded":"2024-01-15T17:57:33.186541+0000","last_fullsized":"2024-01-15T17:57:33.186541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136810+0000","last_change":"2024-01-15T17:57:31.126189+0000","last_active":"2024-01-15T17:57:33.136810+0000","last_peered":"2024-01-15T17:57:33.136810+0000","last_clean":"2024-01-15T17:57:33.136810+0000","last_became_active":"2024-01-15T17:57:31.125519+0000","last_became_peered":"2024-01-15T17:57:31.125519+0000","last_unstale":"2024-01-15T17:57:33.136810+0000","last_undegraded":"2024-01-15T17:57:33.136810+0000","last_fullsized":"2024-01-15T17:57:33.136810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.137059+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.136012+0000","last_became_peered":"2024-01-15T17:57:32.136012+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141434+0000","last_change":"2024-01-15T17:57:31.139980+0000","last_active":"2024-01-15T17:57:33.141434+0000","last_peered":"2024-01-15T17:57:33.141434+0000","last_clean":"2024-01-15T17:57:33.141434+0000","last_became_active":"2024-01-15T17:57:31.139489+0000","last_became_peered":"2024-01-15T17:57:31.139489+0000","last_unstale":"2024-01-15T17:57:33.141434+0000","last_undegraded":"2024-01-15T17:57:33.141434+0000","last_fullsized":"2024-01-15T17:57:33.141434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:54.272945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136716+0000","last_change":"2024-01-15T17:57:32.130379+0000","last_active":"2024-01-15T17:57:33.136716+0000","last_peered":"2024-01-15T17:57:33.136716+0000","last_clean":"2024-01-15T17:57:33.136716+0000","last_became_active":"2024-01-15T17:57:32.129645+0000","last_became_peered":"2024-01-15T17:57:32.129645+0000","last_unstale":"2024-01-15T17:57:33.136716+0000","last_undegraded":"2024-01-15T17:57:33.136716+0000","last_fullsized":"2024-01-15T17:57:33.136716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136624+0000","last_change":"2024-01-15T17:57:31.146382+0000","last_active":"2024-01-15T17:57:33.136624+0000","last_peered":"2024-01-15T17:57:33.136624+0000","last_clean":"2024-01-15T17:57:33.136624+0000","last_became_active":"2024-01-15T17:57:31.146119+0000","last_became_peered":"2024-01-15T17:57:31.146119+0000","last_unstale":"2024-01-15T17:57:33.136624+0000","last_undegraded":"2024-01-15T17:57:33.136624+0000","last_fullsized":"2024-01-15T17:57:33.136624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:30.690538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418481+0000","last_change":"2024-01-15T17:57:32.144183+0000","last_active":"2024-01-15T17:57:33.418481+0000","last_peered":"2024-01-15T17:57:33.418481+0000","last_clean":"2024-01-15T17:57:33.418481+0000","last_became_active":"2024-01-15T17:57:32.143880+0000","last_became_peered":"2024-01-15T17:57:32.143880+0000","last_unstale":"2024-01-15T17:57:33.418481+0000","last_undegraded":"2024-01-15T17:57:33.418481+0000","last_fullsized":"2024-01-15T17:57:33.418481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415224+0000","last_change":"2024-01-15T17:57:31.144154+0000","last_active":"2024-01-15T17:57:33.415224+0000","last_peered":"2024-01-15T17:57:33.415224+0000","last_clean":"2024-01-15T17:57:33.415224+0000","last_became_active":"2024-01-15T17:57:31.143857+0000","last_became_peered":"2024-01-15T17:57:31.143857+0000","last_unstale":"2024-01-15T17:57:33.415224+0000","last_undegraded":"2024-01-15T17:57:33.415224+0000","last_fullsized":"2024-01-15T17:57:33.415224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419365+0000","last_change":"2024-01-15T17:57:32.142460+0000","last_active":"2024-01-15T17:57:33.419365+0000","last_peered":"2024-01-15T17:57:33.419365+0000","last_clean":"2024-01-15T17:57:33.419365+0000","last_became_active":"2024-01-15T17:57:32.141816+0000","last_became_peered":"2024-01-15T17:57:32.141816+0000","last_unstale":"2024-01-15T17:57:33.419365+0000","last_undegraded":"2024-01-15T17:57:33.419365+0000","last_fullsized":"2024-01-15T17:57:33.419365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139202+0000","last_change":"2024-01-15T17:57:31.139784+0000","last_active":"2024-01-15T17:57:33.139202+0000","last_peered":"2024-01-15T17:57:33.139202+0000","last_clean":"2024-01-15T17:57:33.139202+0000","last_became_active":"2024-01-15T17:57:31.138181+0000","last_became_peered":"2024-01-15T17:57:31.138181+0000","last_unstale":"2024-01-15T17:57:33.139202+0000","last_undegraded":"2024-01-15T17:57:33.139202+0000","last_fullsized":"2024-01-15T17:57:33.139202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:50.662857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186323+0000","last_change":"2024-01-15T17:57:32.130289+0000","last_active":"2024-01-15T17:57:33.186323+0000","last_peered":"2024-01-15T17:57:33.186323+0000","last_clean":"2024-01-15T17:57:33.186323+0000","last_became_active":"2024-01-15T17:57:32.129807+0000","last_became_peered":"2024-01-15T17:57:32.129807+0000","last_unstale":"2024-01-15T17:57:33.186323+0000","last_undegraded":"2024-01-15T17:57:33.186323+0000","last_fullsized":"2024-01-15T17:57:33.186323+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:43.951697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135874+0000","last_change":"2024-01-15T17:57:31.132917+0000","last_active":"2024-01-15T17:57:33.135874+0000","last_peered":"2024-01-15T17:57:33.135874+0000","last_clean":"2024-01-15T17:57:33.135874+0000","last_became_active":"2024-01-15T17:57:31.132650+0000","last_became_peered":"2024-01-15T17:57:31.132650+0000","last_unstale":"2024-01-15T17:57:33.135874+0000","last_undegraded":"2024-01-15T17:57:33.135874+0000","last_fullsized":"2024-01-15T17:57:33.135874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415920+0000","last_change":"2024-01-15T17:57:32.136009+0000","last_active":"2024-01-15T17:57:33.415920+0000","last_peered":"2024-01-15T17:57:33.415920+0000","last_clean":"2024-01-15T17:57:33.415920+0000","last_became_active":"2024-01-15T17:57:32.135200+0000","last_became_peered":"2024-01-15T17:57:32.135200+0000","last_unstale":"2024-01-15T17:57:33.415920+0000","last_undegraded":"2024-01-15T17:57:33.415920+0000","last_fullsized":"2024-01-15T17:57:33.415920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136090+0000","last_change":"2024-01-15T17:57:31.130465+0000","last_active":"2024-01-15T17:57:33.136090+0000","last_peered":"2024-01-15T17:57:33.136090+0000","last_clean":"2024-01-15T17:57:33.136090+0000","last_became_active":"2024-01-15T17:57:31.130032+0000","last_became_peered":"2024-01-15T17:57:31.130032+0000","last_unstale":"2024-01-15T17:57:33.136090+0000","last_undegraded":"2024-01-15T17:57:33.136090+0000","last_fullsized":"2024-01-15T17:57:33.136090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417702+0000","last_change":"2024-01-15T17:57:32.154915+0000","last_active":"2024-01-15T17:57:33.417702+0000","last_peered":"2024-01-15T17:57:33.417702+0000","last_clean":"2024-01-15T17:57:33.417702+0000","last_became_active":"2024-01-15T17:57:32.154599+0000","last_became_peered":"2024-01-15T17:57:32.154599+0000","last_unstale":"2024-01-15T17:57:33.417702+0000","last_undegraded":"2024-01-15T17:57:33.417702+0000","last_fullsized":"2024-01-15T17:57:33.417702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141769+0000","last_change":"2024-01-15T17:57:31.143380+0000","last_active":"2024-01-15T17:57:33.141769+0000","last_peered":"2024-01-15T17:57:33.141769+0000","last_clean":"2024-01-15T17:57:33.141769+0000","last_became_active":"2024-01-15T17:57:31.143046+0000","last_became_peered":"2024-01-15T17:57:31.143046+0000","last_unstale":"2024-01-15T17:57:33.141769+0000","last_undegraded":"2024-01-15T17:57:33.141769+0000","last_fullsized":"2024-01-15T17:57:33.141769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414379+0000","last_change":"2024-01-15T17:57:32.149247+0000","last_active":"2024-01-15T17:57:33.414379+0000","last_peered":"2024-01-15T17:57:33.414379+0000","last_clean":"2024-01-15T17:57:33.414379+0000","last_became_active":"2024-01-15T17:57:32.148620+0000","last_became_peered":"2024-01-15T17:57:32.148620+0000","last_unstale":"2024-01-15T17:57:33.414379+0000","last_undegraded":"2024-01-15T17:57:33.414379+0000","last_fullsized":"2024-01-15T17:57:33.414379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139434+0000","last_change":"2024-01-15T17:57:32.159609+0000","last_active":"2024-01-15T17:57:33.139434+0000","last_peered":"2024-01-15T17:57:33.139434+0000","last_clean":"2024-01-15T17:57:33.139434+0000","last_became_active":"2024-01-15T17:57:32.155889+0000","last_became_peered":"2024-01-15T17:57:32.155889+0000","last_unstale":"2024-01-15T17:57:33.139434+0000","last_undegraded":"2024-01-15T17:57:33.139434+0000","last_fullsized":"2024-01-15T17:57:33.139434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138536+0000","last_change":"2024-01-15T17:57:31.144524+0000","last_active":"2024-01-15T17:57:33.138536+0000","last_peered":"2024-01-15T17:57:33.138536+0000","last_clean":"2024-01-15T17:57:33.138536+0000","last_became_active":"2024-01-15T17:57:31.143299+0000","last_became_peered":"2024-01-15T17:57:31.143299+0000","last_unstale":"2024-01-15T17:57:33.138536+0000","last_undegraded":"2024-01-15T17:57:33.138536+0000","last_fullsized":"2024-01-15T17:57:33.138536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139433+0000","last_change":"2024-01-15T17:57:32.151244+0000","last_active":"2024-01-15T17:57:33.139433+0000","last_peered":"2024-01-15T17:57:33.139433+0000","last_clean":"2024-01-15T17:57:33.139433+0000","last_became_active":"2024-01-15T17:57:32.150834+0000","last_became_peered":"2024-01-15T17:57:32.150834+0000","last_unstale":"2024-01-15T17:57:33.139433+0000","last_undegraded":"2024-01-15T17:57:33.139433+0000","last_fullsized":"2024-01-15T17:57:33.139433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141240+0000","last_change":"2024-01-15T17:57:31.144722+0000","last_active":"2024-01-15T17:57:33.141240+0000","last_peered":"2024-01-15T17:57:33.141240+0000","last_clean":"2024-01-15T17:57:33.141240+0000","last_became_active":"2024-01-15T17:57:31.144378+0000","last_became_peered":"2024-01-15T17:57:31.144378+0000","last_unstale":"2024-01-15T17:57:33.141240+0000","last_undegraded":"2024-01-15T17:57:33.141240+0000","last_fullsized":"2024-01-15T17:57:33.141240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:48.548981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419383+0000","last_change":"2024-01-15T17:57:32.142742+0000","last_active":"2024-01-15T17:57:33.419383+0000","last_peered":"2024-01-15T17:57:33.419383+0000","last_clean":"2024-01-15T17:57:33.419383+0000","last_became_active":"2024-01-15T17:57:32.142372+0000","last_became_peered":"2024-01-15T17:57:32.142372+0000","last_unstale":"2024-01-15T17:57:33.419383+0000","last_undegraded":"2024-01-15T17:57:33.419383+0000","last_fullsized":"2024-01-15T17:57:33.419383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417735+0000","last_change":"2024-01-15T17:57:31.128188+0000","last_active":"2024-01-15T17:57:33.417735+0000","last_peered":"2024-01-15T17:57:33.417735+0000","last_clean":"2024-01-15T17:57:33.417735+0000","last_became_active":"2024-01-15T17:57:31.127089+0000","last_became_peered":"2024-01-15T17:57:31.127089+0000","last_unstale":"2024-01-15T17:57:33.417735+0000","last_undegraded":"2024-01-15T17:57:33.417735+0000","last_fullsized":"2024-01-15T17:57:33.417735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417498+0000","last_change":"2024-01-15T17:57:32.142205+0000","last_active":"2024-01-15T17:57:33.417498+0000","last_peered":"2024-01-15T17:57:33.417498+0000","last_clean":"2024-01-15T17:57:33.417498+0000","last_became_active":"2024-01-15T17:57:32.139881+0000","last_became_peered":"2024-01-15T17:57:32.139881+0000","last_unstale":"2024-01-15T17:57:33.417498+0000","last_undegraded":"2024-01-15T17:57:33.417498+0000","last_fullsized":"2024-01-15T17:57:33.417498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419783+0000","last_change":"2024-01-15T17:57:31.140404+0000","last_active":"2024-01-15T17:57:33.419783+0000","last_peered":"2024-01-15T17:57:33.419783+0000","last_clean":"2024-01-15T17:57:33.419783+0000","last_became_active":"2024-01-15T17:57:31.140098+0000","last_became_peered":"2024-01-15T17:57:31.140098+0000","last_unstale":"2024-01-15T17:57:33.419783+0000","last_undegraded":"2024-01-15T17:57:33.419783+0000","last_fullsized":"2024-01-15T17:57:33.419783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.407}]}]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8450000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6449999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6579999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.161}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1789999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9830000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]}]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":831488,"data_stored":669808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.74}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7789999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:57:36.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:57:36.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.864+0000 7f53b7c00640 1 Processor -- start 2024-01-15T17:57:36.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.864+0000 7f53b7c00640 1 -- start start 2024-01-15T17:57:36.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.866+0000 7f53b7c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b0144b70 0x7f53b006eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.866+0000 7f53b7c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f53b00ce140 0x7f53b00b5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.866+0000 7f53b7c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b006f480 con 0x7f53b00b59d0 2024-01-15T17:57:36.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.866+0000 7f53b7c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b00b5e60 con 0x7f53b00ce140 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.866+0000 7f53b7c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b006f760 con 0x7f53b0144b70 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f53b00ce140 0x7f53b00b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f53b00ce140 0x7f53b00b5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49574/0 (socket says 172.21.15.2:49574) 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 -- 172.21.15.2:0/3674471309 learned_addr learned my addr 172.21.15.2:0/3674471309 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53b5200640 1 --2- 172.21.15.2:0/3674471309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b0144b70 0x7f53b006eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 -- 172.21.15.2:0/3674471309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b0144b70 msgr2=0x7f53b006eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 --2- 172.21.15.2:0/3674471309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b0144b70 0x7f53b006eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 -- 172.21.15.2:0/3674471309 >> v1:172.21.15.137:6789/0 conn(0x7f53b00b59d0 legacy=0x7f53b006a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:36.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.867+0000 7f53afe00640 1 -- 172.21.15.2:0/3674471309 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b006fa40 con 0x7f53b00ce140 2024-01-15T17:57:36.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.868+0000 7f53afe00640 1 --2- 172.21.15.2:0/3674471309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b00b5490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5398009870 tx=0x7f539802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ab06ae1bdec87a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:36.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53af400640 1 -- 172.21.15.2:0/3674471309 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398031640 con 0x7f53b00ce140 2024-01-15T17:57:36.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53af400640 1 -- 172.21.15.2:0/3674471309 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53980317a0 con 0x7f53b00ce140 2024-01-15T17:57:36.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53af400640 1 -- 172.21.15.2:0/3674471309 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398031b30 con 0x7f53b00ce140 2024-01-15T17:57:36.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 -- 172.21.15.2:0/3674471309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 msgr2=0x7f53b00b5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 --2- 172.21.15.2:0/3674471309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b00b5490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5398009870 tx=0x7f539802b8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 -- 172.21.15.2:0/3674471309 shutdown_connections 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 --2- 172.21.15.2:0/3674471309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b00b5490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 --2- 172.21.15.2:0/3674471309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b0144b70 0x7f53b006eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 -- 172.21.15.2:0/3674471309 >> 172.21.15.2:0/3674471309 conn(0x7f53b006d490 msgr2=0x7f53b006d870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 -- 172.21.15.2:0/3674471309 shutdown_connections 2024-01-15T17:57:36.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.869+0000 7f53b7c00640 1 -- 172.21.15.2:0/3674471309 wait complete. 2024-01-15T17:57:36.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.871+0000 7f53b7c00640 1 Processor -- start 2024-01-15T17:57:36.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.871+0000 7f53b7c00640 1 -- start start 2024-01-15T17:57:36.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.872+0000 7f53b7c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b00b59d0 0x7f53b015aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.872+0000 7f53b7c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b01e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.872+0000 7f53b7c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53b0144b70 0x7f53b01e6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.872+0000 7f53b7c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01e5530 con 0x7f53b00ce140 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.872+0000 7f53b7c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01e4920 con 0x7f53b0144b70 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.872+0000 7f53b7c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01e4c00 con 0x7f53b00b59d0 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b01e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b01e3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49586/0 (socket says 172.21.15.2:49586) 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 -- 172.21.15.2:0/2815199041 learned_addr learned my addr 172.21.15.2:0/2815199041 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53b5c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53b0144b70 0x7f53b01e6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53b5200640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b00b59d0 0x7f53b015aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 -- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b00b59d0 msgr2=0x7f53b015aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b00b59d0 0x7f53b015aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 -- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53b0144b70 msgr2=0x7f53b01e6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53b0144b70 0x7f53b01e6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.873+0000 7f53afe00640 1 -- 172.21.15.2:0/2815199041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53980094d0 con 0x7f53b00ce140 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.874+0000 7f53afe00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b01e3fb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5398009870 tx=0x7f53980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.874+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539803ac60 con 0x7f53b00ce140 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.874+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539803adc0 con 0x7f53b00ce140 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.874+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398044630 con 0x7f53b00ce140 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.874+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b01e4ee0 con 0x7f53b00ce140 2024-01-15T17:57:36.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.874+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b015afc0 con 0x7f53b00ce140 2024-01-15T17:57:36.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.875+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b01e56a0 con 0x7f53b00ce140 2024-01-15T17:57:36.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.876+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f539802f050 con 0x7f53b00ce140 2024-01-15T17:57:36.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.877+0000 7f53acc00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f538806b2f0 0x7f538806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:36.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.877+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53980aba50 con 0x7f53b00ce140 2024-01-15T17:57:36.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.878+0000 7f53b5200640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f538806b2f0 0x7f538806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:36.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.879+0000 7f53b5200640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f538806b2f0 0x7f538806d7b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009a60 tx=0x7f53a0003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:36.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:36.882+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53980768c0 con 0x7f53b00ce140 2024-01-15T17:57:37.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.074+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53b015b740 con 0x7f538806b2f0 2024-01-15T17:57:37.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.094+0000 7f53acc00640 1 -- 172.21.15.2:0/2815199041 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264446 (secure 0 0 0) 0x7f53a0004780 con 0x7f538806b2f0 2024-01-15T17:57:37.102 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:37.103 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:57:37.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f538806b2f0 msgr2=0x7f538806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f538806b2f0 0x7f538806d7b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009a60 tx=0x7f53a0003990 comp rx=0 tx=0).stop 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 msgr2=0x7f53b01e3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b01e3fb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5398009870 tx=0x7f53980046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 shutdown_connections 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f538806b2f0 0x7f538806d7b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f53b0144b70 0x7f53b01e6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f53b00ce140 0x7f53b01e3fb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 --2- 172.21.15.2:0/2815199041 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f53b00b59d0 0x7f53b015aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 >> 172.21.15.2:0/2815199041 conn(0x7f53b006d490 msgr2=0x7f53b013e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.101+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 shutdown_connections 2024-01-15T17:57:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.102+0000 7f53b7c00640 1 -- 172.21.15.2:0/2815199041 wait complete. 2024-01-15T17:57:37.124 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-01-15T17:57:35.368991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236116,"kb_used_data":2972,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738604,"statfs":{"total":773094113280,"available":772852330496,"internally_reserved":0,"allocated":3043328,"data_stored":1809672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":58,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.025600"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139938+0000","last_change":"2024-01-15T17:57:32.145196+0000","last_active":"2024-01-15T17:57:33.139938+0000","last_peered":"2024-01-15T17:57:33.139938+0000","last_clean":"2024-01-15T17:57:33.139938+0000","last_became_active":"2024-01-15T17:57:32.144373+0000","last_became_peered":"2024-01-15T17:57:32.144373+0000","last_unstale":"2024-01-15T17:57:33.139938+0000","last_undegraded":"2024-01-15T17:57:33.139938+0000","last_fullsized":"2024-01-15T17:57:33.139938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136557+0000","last_change":"2024-01-15T17:57:32.143595+0000","last_active":"2024-01-15T17:57:33.136557+0000","last_peered":"2024-01-15T17:57:33.136557+0000","last_clean":"2024-01-15T17:57:33.136557+0000","last_became_active":"2024-01-15T17:57:32.143249+0000","last_became_peered":"2024-01-15T17:57:32.143249+0000","last_unstale":"2024-01-15T17:57:33.136557+0000","last_undegraded":"2024-01-15T17:57:33.136557+0000","last_fullsized":"2024-01-15T17:57:33.136557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419213+0000","last_change":"2024-01-15T17:57:32.143090+0000","last_active":"2024-01-15T17:57:33.419213+0000","last_peered":"2024-01-15T17:57:33.419213+0000","last_clean":"2024-01-15T17:57:33.419213+0000","last_became_active":"2024-01-15T17:57:32.142828+0000","last_became_peered":"2024-01-15T17:57:32.142828+0000","last_unstale":"2024-01-15T17:57:33.419213+0000","last_undegraded":"2024-01-15T17:57:33.419213+0000","last_fullsized":"2024-01-15T17:57:33.419213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139101+0000","last_change":"2024-01-15T17:57:32.146690+0000","last_active":"2024-01-15T17:57:33.139101+0000","last_peered":"2024-01-15T17:57:33.139101+0000","last_clean":"2024-01-15T17:57:33.139101+0000","last_became_active":"2024-01-15T17:57:32.142878+0000","last_became_peered":"2024-01-15T17:57:32.142878+0000","last_unstale":"2024-01-15T17:57:33.139101+0000","last_undegraded":"2024-01-15T17:57:33.139101+0000","last_fullsized":"2024-01-15T17:57:33.139101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141233+0000","last_change":"2024-01-15T17:57:32.145000+0000","last_active":"2024-01-15T17:57:33.141233+0000","last_peered":"2024-01-15T17:57:33.141233+0000","last_clean":"2024-01-15T17:57:33.141233+0000","last_became_active":"2024-01-15T17:57:32.144715+0000","last_became_peered":"2024-01-15T17:57:32.144715+0000","last_unstale":"2024-01-15T17:57:33.141233+0000","last_undegraded":"2024-01-15T17:57:33.141233+0000","last_fullsized":"2024-01-15T17:57:33.141233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:03.467001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138047+0000","last_change":"2024-01-15T17:57:32.154833+0000","last_active":"2024-01-15T17:57:33.138047+0000","last_peered":"2024-01-15T17:57:33.138047+0000","last_clean":"2024-01-15T17:57:33.138047+0000","last_became_active":"2024-01-15T17:57:32.153996+0000","last_became_peered":"2024-01-15T17:57:32.153996+0000","last_unstale":"2024-01-15T17:57:33.138047+0000","last_undegraded":"2024-01-15T17:57:33.138047+0000","last_fullsized":"2024-01-15T17:57:33.138047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138900+0000","last_change":"2024-01-15T17:57:32.147665+0000","last_active":"2024-01-15T17:57:33.138900+0000","last_peered":"2024-01-15T17:57:33.138900+0000","last_clean":"2024-01-15T17:57:33.138900+0000","last_became_active":"2024-01-15T17:57:32.147333+0000","last_became_peered":"2024-01-15T17:57:32.147333+0000","last_unstale":"2024-01-15T17:57:33.138900+0000","last_undegraded":"2024-01-15T17:57:33.138900+0000","last_fullsized":"2024-01-15T17:57:33.138900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415231+0000","last_change":"2024-01-15T17:57:32.144764+0000","last_active":"2024-01-15T17:57:33.415231+0000","last_peered":"2024-01-15T17:57:33.415231+0000","last_clean":"2024-01-15T17:57:33.415231+0000","last_became_active":"2024-01-15T17:57:32.144315+0000","last_became_peered":"2024-01-15T17:57:32.144315+0000","last_unstale":"2024-01-15T17:57:33.415231+0000","last_undegraded":"2024-01-15T17:57:33.415231+0000","last_fullsized":"2024-01-15T17:57:33.415231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140177+0000","last_change":"2024-01-15T17:57:32.157864+0000","last_active":"2024-01-15T17:57:33.140177+0000","last_peered":"2024-01-15T17:57:33.140177+0000","last_clean":"2024-01-15T17:57:33.140177+0000","last_became_active":"2024-01-15T17:57:32.157556+0000","last_became_peered":"2024-01-15T17:57:32.157556+0000","last_unstale":"2024-01-15T17:57:33.140177+0000","last_undegraded":"2024-01-15T17:57:33.140177+0000","last_fullsized":"2024-01-15T17:57:33.140177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415132+0000","last_change":"2024-01-15T17:57:32.161393+0000","last_active":"2024-01-15T17:57:33.415132+0000","last_peered":"2024-01-15T17:57:33.415132+0000","last_clean":"2024-01-15T17:57:33.415132+0000","last_became_active":"2024-01-15T17:57:32.161079+0000","last_became_peered":"2024-01-15T17:57:32.161079+0000","last_unstale":"2024-01-15T17:57:33.415132+0000","last_undegraded":"2024-01-15T17:57:33.415132+0000","last_fullsized":"2024-01-15T17:57:33.415132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414932+0000","last_change":"2024-01-15T17:57:32.145780+0000","last_active":"2024-01-15T17:57:33.414932+0000","last_peered":"2024-01-15T17:57:33.414932+0000","last_clean":"2024-01-15T17:57:33.414932+0000","last_became_active":"2024-01-15T17:57:32.145409+0000","last_became_peered":"2024-01-15T17:57:32.145409+0000","last_unstale":"2024-01-15T17:57:33.414932+0000","last_undegraded":"2024-01-15T17:57:33.414932+0000","last_fullsized":"2024-01-15T17:57:33.414932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186642+0000","last_change":"2024-01-15T17:57:32.144056+0000","last_active":"2024-01-15T17:57:33.186642+0000","last_peered":"2024-01-15T17:57:33.186642+0000","last_clean":"2024-01-15T17:57:33.186642+0000","last_became_active":"2024-01-15T17:57:32.143687+0000","last_became_peered":"2024-01-15T17:57:32.143687+0000","last_unstale":"2024-01-15T17:57:33.186642+0000","last_undegraded":"2024-01-15T17:57:33.186642+0000","last_fullsized":"2024-01-15T17:57:33.186642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417307+0000","last_change":"2024-01-15T17:57:32.141036+0000","last_active":"2024-01-15T17:57:33.417307+0000","last_peered":"2024-01-15T17:57:33.417307+0000","last_clean":"2024-01-15T17:57:33.417307+0000","last_became_active":"2024-01-15T17:57:32.139707+0000","last_became_peered":"2024-01-15T17:57:32.139707+0000","last_unstale":"2024-01-15T17:57:33.417307+0000","last_undegraded":"2024-01-15T17:57:33.417307+0000","last_fullsized":"2024-01-15T17:57:33.417307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415834+0000","last_change":"2024-01-15T17:57:32.148492+0000","last_active":"2024-01-15T17:57:33.415834+0000","last_peered":"2024-01-15T17:57:33.415834+0000","last_clean":"2024-01-15T17:57:33.415834+0000","last_became_active":"2024-01-15T17:57:32.148169+0000","last_became_peered":"2024-01-15T17:57:32.148169+0000","last_unstale":"2024-01-15T17:57:33.415834+0000","last_undegraded":"2024-01-15T17:57:33.415834+0000","last_fullsized":"2024-01-15T17:57:33.415834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:00.974439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.150453+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.150115+0000","last_became_peered":"2024-01-15T17:57:32.150115+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.137808+0000","last_change":"2024-01-15T17:57:32.148256+0000","last_active":"2024-01-15T17:57:33.137808+0000","last_peered":"2024-01-15T17:57:33.137808+0000","last_clean":"2024-01-15T17:57:33.137808+0000","last_became_active":"2024-01-15T17:57:32.147620+0000","last_became_peered":"2024-01-15T17:57:32.147620+0000","last_unstale":"2024-01-15T17:57:33.137808+0000","last_undegraded":"2024-01-15T17:57:33.137808+0000","last_fullsized":"2024-01-15T17:57:33.137808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140840+0000","last_change":"2024-01-15T17:57:32.139828+0000","last_active":"2024-01-15T17:57:33.140840+0000","last_peered":"2024-01-15T17:57:33.140840+0000","last_clean":"2024-01-15T17:57:33.140840+0000","last_became_active":"2024-01-15T17:57:32.139522+0000","last_became_peered":"2024-01-15T17:57:32.139522+0000","last_unstale":"2024-01-15T17:57:33.140840+0000","last_undegraded":"2024-01-15T17:57:33.140840+0000","last_fullsized":"2024-01-15T17:57:33.140840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416706+0000","last_change":"2024-01-15T17:57:32.140652+0000","last_active":"2024-01-15T17:57:33.416706+0000","last_peered":"2024-01-15T17:57:33.416706+0000","last_clean":"2024-01-15T17:57:33.416706+0000","last_became_active":"2024-01-15T17:57:32.140366+0000","last_became_peered":"2024-01-15T17:57:32.140366+0000","last_unstale":"2024-01-15T17:57:33.416706+0000","last_undegraded":"2024-01-15T17:57:33.416706+0000","last_fullsized":"2024-01-15T17:57:33.416706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141414+0000","last_change":"2024-01-15T17:57:32.149137+0000","last_active":"2024-01-15T17:57:33.141414+0000","last_peered":"2024-01-15T17:57:33.141414+0000","last_clean":"2024-01-15T17:57:33.141414+0000","last_became_active":"2024-01-15T17:57:32.148658+0000","last_became_peered":"2024-01-15T17:57:32.148658+0000","last_unstale":"2024-01-15T17:57:33.141414+0000","last_undegraded":"2024-01-15T17:57:33.141414+0000","last_fullsized":"2024-01-15T17:57:33.141414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:18.642472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414742+0000","last_change":"2024-01-15T17:57:32.156212+0000","last_active":"2024-01-15T17:57:33.414742+0000","last_peered":"2024-01-15T17:57:33.414742+0000","last_clean":"2024-01-15T17:57:33.414742+0000","last_became_active":"2024-01-15T17:57:32.153278+0000","last_became_peered":"2024-01-15T17:57:32.153278+0000","last_unstale":"2024-01-15T17:57:33.414742+0000","last_undegraded":"2024-01-15T17:57:33.414742+0000","last_fullsized":"2024-01-15T17:57:33.414742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417083+0000","last_change":"2024-01-15T17:57:32.147107+0000","last_active":"2024-01-15T17:57:33.417083+0000","last_peered":"2024-01-15T17:57:33.417083+0000","last_clean":"2024-01-15T17:57:33.417083+0000","last_became_active":"2024-01-15T17:57:32.146730+0000","last_became_peered":"2024-01-15T17:57:32.146730+0000","last_unstale":"2024-01-15T17:57:33.417083+0000","last_undegraded":"2024-01-15T17:57:33.417083+0000","last_fullsized":"2024-01-15T17:57:33.417083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138213+0000","last_change":"2024-01-15T17:57:32.139188+0000","last_active":"2024-01-15T17:57:33.138213+0000","last_peered":"2024-01-15T17:57:33.138213+0000","last_clean":"2024-01-15T17:57:33.138213+0000","last_became_active":"2024-01-15T17:57:32.138898+0000","last_became_peered":"2024-01-15T17:57:32.138898+0000","last_unstale":"2024-01-15T17:57:33.138213+0000","last_undegraded":"2024-01-15T17:57:33.138213+0000","last_fullsized":"2024-01-15T17:57:33.138213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416604+0000","last_change":"2024-01-15T17:57:32.148598+0000","last_active":"2024-01-15T17:57:33.416604+0000","last_peered":"2024-01-15T17:57:33.416604+0000","last_clean":"2024-01-15T17:57:33.416604+0000","last_became_active":"2024-01-15T17:57:32.148266+0000","last_became_peered":"2024-01-15T17:57:32.148266+0000","last_unstale":"2024-01-15T17:57:33.416604+0000","last_undegraded":"2024-01-15T17:57:33.416604+0000","last_fullsized":"2024-01-15T17:57:33.416604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138790+0000","last_change":"2024-01-15T17:57:32.135029+0000","last_active":"2024-01-15T17:57:33.138790+0000","last_peered":"2024-01-15T17:57:33.138790+0000","last_clean":"2024-01-15T17:57:33.138790+0000","last_became_active":"2024-01-15T17:57:32.134762+0000","last_became_peered":"2024-01-15T17:57:32.134762+0000","last_unstale":"2024-01-15T17:57:33.138790+0000","last_undegraded":"2024-01-15T17:57:33.138790+0000","last_fullsized":"2024-01-15T17:57:33.138790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418117+0000","last_change":"2024-01-15T17:57:32.153167+0000","last_active":"2024-01-15T17:57:33.418117+0000","last_peered":"2024-01-15T17:57:33.418117+0000","last_clean":"2024-01-15T17:57:33.418117+0000","last_became_active":"2024-01-15T17:57:32.151003+0000","last_became_peered":"2024-01-15T17:57:32.151003+0000","last_unstale":"2024-01-15T17:57:33.418117+0000","last_undegraded":"2024-01-15T17:57:33.418117+0000","last_fullsized":"2024-01-15T17:57:33.418117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:32.142988+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:32.142613+0000","last_became_peered":"2024-01-15T17:57:32.142613+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136341+0000","last_change":"2024-01-15T17:57:32.134987+0000","last_active":"2024-01-15T17:57:33.136341+0000","last_peered":"2024-01-15T17:57:33.136341+0000","last_clean":"2024-01-15T17:57:33.136341+0000","last_became_active":"2024-01-15T17:57:32.134719+0000","last_became_peered":"2024-01-15T17:57:32.134719+0000","last_unstale":"2024-01-15T17:57:33.136341+0000","last_undegraded":"2024-01-15T17:57:33.136341+0000","last_fullsized":"2024-01-15T17:57:33.136341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418595+0000","last_change":"2024-01-15T17:57:32.153961+0000","last_active":"2024-01-15T17:57:33.418595+0000","last_peered":"2024-01-15T17:57:33.418595+0000","last_clean":"2024-01-15T17:57:33.418595+0000","last_became_active":"2024-01-15T17:57:32.151097+0000","last_became_peered":"2024-01-15T17:57:32.151097+0000","last_unstale":"2024-01-15T17:57:33.418595+0000","last_undegraded":"2024-01-15T17:57:33.418595+0000","last_fullsized":"2024-01-15T17:57:33.418595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418195+0000","last_change":"2024-01-15T17:57:32.144509+0000","last_active":"2024-01-15T17:57:33.418195+0000","last_peered":"2024-01-15T17:57:33.418195+0000","last_clean":"2024-01-15T17:57:33.418195+0000","last_became_active":"2024-01-15T17:57:32.143921+0000","last_became_peered":"2024-01-15T17:57:32.143921+0000","last_unstale":"2024-01-15T17:57:33.418195+0000","last_undegraded":"2024-01-15T17:57:33.418195+0000","last_fullsized":"2024-01-15T17:57:33.418195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138415+0000","last_change":"2024-01-15T17:57:32.137686+0000","last_active":"2024-01-15T17:57:33.138415+0000","last_peered":"2024-01-15T17:57:33.138415+0000","last_clean":"2024-01-15T17:57:33.138415+0000","last_became_active":"2024-01-15T17:57:32.137409+0000","last_became_peered":"2024-01-15T17:57:32.137409+0000","last_unstale":"2024-01-15T17:57:33.138415+0000","last_undegraded":"2024-01-15T17:57:33.138415+0000","last_fullsized":"2024-01-15T17:57:33.138415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414292+0000","last_change":"2024-01-15T17:57:32.129661+0000","last_active":"2024-01-15T17:57:33.414292+0000","last_peered":"2024-01-15T17:57:33.414292+0000","last_clean":"2024-01-15T17:57:33.414292+0000","last_became_active":"2024-01-15T17:57:32.129283+0000","last_became_peered":"2024-01-15T17:57:32.129283+0000","last_unstale":"2024-01-15T17:57:33.414292+0000","last_undegraded":"2024-01-15T17:57:33.414292+0000","last_fullsized":"2024-01-15T17:57:33.414292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414350+0000","last_change":"2024-01-15T17:57:32.147901+0000","last_active":"2024-01-15T17:57:33.414350+0000","last_peered":"2024-01-15T17:57:33.414350+0000","last_clean":"2024-01-15T17:57:33.414350+0000","last_became_active":"2024-01-15T17:57:32.147185+0000","last_became_peered":"2024-01-15T17:57:32.147185+0000","last_unstale":"2024-01-15T17:57:33.414350+0000","last_undegraded":"2024-01-15T17:57:33.414350+0000","last_fullsized":"2024-01-15T17:57:33.414350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186262+0000","last_change":"2024-01-15T17:57:32.138254+0000","last_active":"2024-01-15T17:57:33.186262+0000","last_peered":"2024-01-15T17:57:33.186262+0000","last_clean":"2024-01-15T17:57:33.186262+0000","last_became_active":"2024-01-15T17:57:32.137849+0000","last_became_peered":"2024-01-15T17:57:32.137849+0000","last_unstale":"2024-01-15T17:57:33.186262+0000","last_undegraded":"2024-01-15T17:57:33.186262+0000","last_fullsized":"2024-01-15T17:57:33.186262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:12.340083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417690+0000","last_change":"2024-01-15T17:57:31.132496+0000","last_active":"2024-01-15T17:57:33.417690+0000","last_peered":"2024-01-15T17:57:33.417690+0000","last_clean":"2024-01-15T17:57:33.417690+0000","last_became_active":"2024-01-15T17:57:31.132095+0000","last_became_peered":"2024-01-15T17:57:31.132095+0000","last_unstale":"2024-01-15T17:57:33.417690+0000","last_undegraded":"2024-01-15T17:57:33.417690+0000","last_fullsized":"2024-01-15T17:57:33.417690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418981+0000","last_change":"2024-01-15T17:57:32.154160+0000","last_active":"2024-01-15T17:57:33.418981+0000","last_peered":"2024-01-15T17:57:33.418981+0000","last_clean":"2024-01-15T17:57:33.418981+0000","last_became_active":"2024-01-15T17:57:32.153881+0000","last_became_peered":"2024-01-15T17:57:32.153881+0000","last_unstale":"2024-01-15T17:57:33.418981+0000","last_undegraded":"2024-01-15T17:57:33.418981+0000","last_fullsized":"2024-01-15T17:57:33.418981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414244+0000","last_change":"2024-01-15T17:57:31.143303+0000","last_active":"2024-01-15T17:57:33.414244+0000","last_peered":"2024-01-15T17:57:33.414244+0000","last_clean":"2024-01-15T17:57:33.414244+0000","last_became_active":"2024-01-15T17:57:31.142917+0000","last_became_peered":"2024-01-15T17:57:31.142917+0000","last_unstale":"2024-01-15T17:57:33.414244+0000","last_undegraded":"2024-01-15T17:57:33.414244+0000","last_fullsized":"2024-01-15T17:57:33.414244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416836+0000","last_change":"2024-01-15T17:57:31.137389+0000","last_active":"2024-01-15T17:57:33.416836+0000","last_peered":"2024-01-15T17:57:33.416836+0000","last_clean":"2024-01-15T17:57:33.416836+0000","last_became_active":"2024-01-15T17:57:31.137115+0000","last_became_peered":"2024-01-15T17:57:31.137115+0000","last_unstale":"2024-01-15T17:57:33.416836+0000","last_undegraded":"2024-01-15T17:57:33.416836+0000","last_fullsized":"2024-01-15T17:57:33.416836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415574+0000","last_change":"2024-01-15T17:57:32.156574+0000","last_active":"2024-01-15T17:57:33.415574+0000","last_peered":"2024-01-15T17:57:33.415574+0000","last_clean":"2024-01-15T17:57:33.415574+0000","last_became_active":"2024-01-15T17:57:32.156179+0000","last_became_peered":"2024-01-15T17:57:32.156179+0000","last_unstale":"2024-01-15T17:57:33.415574+0000","last_undegraded":"2024-01-15T17:57:33.415574+0000","last_fullsized":"2024-01-15T17:57:33.415574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139052+0000","last_change":"2024-01-15T17:57:31.132492+0000","last_active":"2024-01-15T17:57:33.139052+0000","last_peered":"2024-01-15T17:57:33.139052+0000","last_clean":"2024-01-15T17:57:33.139052+0000","last_became_active":"2024-01-15T17:57:31.132141+0000","last_became_peered":"2024-01-15T17:57:31.132141+0000","last_unstale":"2024-01-15T17:57:33.139052+0000","last_undegraded":"2024-01-15T17:57:33.139052+0000","last_fullsized":"2024-01-15T17:57:33.139052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.134222+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.133920+0000","last_became_peered":"2024-01-15T17:57:32.133920+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:31.144178+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:31.143915+0000","last_became_peered":"2024-01-15T17:57:31.143915+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140200+0000","last_change":"2024-01-15T17:57:32.141022+0000","last_active":"2024-01-15T17:57:33.140200+0000","last_peered":"2024-01-15T17:57:33.140200+0000","last_clean":"2024-01-15T17:57:33.140200+0000","last_became_active":"2024-01-15T17:57:32.140122+0000","last_became_peered":"2024-01-15T17:57:32.140122+0000","last_unstale":"2024-01-15T17:57:33.140200+0000","last_undegraded":"2024-01-15T17:57:33.140200+0000","last_fullsized":"2024-01-15T17:57:33.140200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140234+0000","last_change":"2024-01-15T17:57:31.146237+0000","last_active":"2024-01-15T17:57:33.140234+0000","last_peered":"2024-01-15T17:57:33.140234+0000","last_clean":"2024-01-15T17:57:33.140234+0000","last_became_active":"2024-01-15T17:57:31.145808+0000","last_became_peered":"2024-01-15T17:57:31.145808+0000","last_unstale":"2024-01-15T17:57:33.140234+0000","last_undegraded":"2024-01-15T17:57:33.140234+0000","last_fullsized":"2024-01-15T17:57:33.140234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:05.346229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419713+0000","last_change":"2024-01-15T17:57:32.142125+0000","last_active":"2024-01-15T17:57:33.419713+0000","last_peered":"2024-01-15T17:57:33.419713+0000","last_clean":"2024-01-15T17:57:33.419713+0000","last_became_active":"2024-01-15T17:57:32.141806+0000","last_became_peered":"2024-01-15T17:57:32.141806+0000","last_unstale":"2024-01-15T17:57:33.419713+0000","last_undegraded":"2024-01-15T17:57:33.419713+0000","last_fullsized":"2024-01-15T17:57:33.419713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136484+0000","last_change":"2024-01-15T17:57:31.131511+0000","last_active":"2024-01-15T17:57:33.136484+0000","last_peered":"2024-01-15T17:57:33.136484+0000","last_clean":"2024-01-15T17:57:33.136484+0000","last_became_active":"2024-01-15T17:57:31.131207+0000","last_became_peered":"2024-01-15T17:57:31.131207+0000","last_unstale":"2024-01-15T17:57:33.136484+0000","last_undegraded":"2024-01-15T17:57:33.136484+0000","last_fullsized":"2024-01-15T17:57:33.136484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186851+0000","last_change":"2024-01-15T17:57:32.141464+0000","last_active":"2024-01-15T17:57:33.186851+0000","last_peered":"2024-01-15T17:57:33.186851+0000","last_clean":"2024-01-15T17:57:33.186851+0000","last_became_active":"2024-01-15T17:57:32.140204+0000","last_became_peered":"2024-01-15T17:57:32.140204+0000","last_unstale":"2024-01-15T17:57:33.186851+0000","last_undegraded":"2024-01-15T17:57:33.186851+0000","last_fullsized":"2024-01-15T17:57:33.186851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:23.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415055+0000","last_change":"2024-01-15T17:57:31.140465+0000","last_active":"2024-01-15T17:57:33.415055+0000","last_peered":"2024-01-15T17:57:33.415055+0000","last_clean":"2024-01-15T17:57:33.415055+0000","last_became_active":"2024-01-15T17:57:31.140132+0000","last_became_peered":"2024-01-15T17:57:31.140132+0000","last_unstale":"2024-01-15T17:57:33.415055+0000","last_undegraded":"2024-01-15T17:57:33.415055+0000","last_fullsized":"2024-01-15T17:57:33.415055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141003+0000","last_change":"2024-01-15T17:57:32.140539+0000","last_active":"2024-01-15T17:57:33.141003+0000","last_peered":"2024-01-15T17:57:33.141003+0000","last_clean":"2024-01-15T17:57:33.141003+0000","last_became_active":"2024-01-15T17:57:32.140117+0000","last_became_peered":"2024-01-15T17:57:32.140117+0000","last_unstale":"2024-01-15T17:57:33.141003+0000","last_undegraded":"2024-01-15T17:57:33.141003+0000","last_fullsized":"2024-01-15T17:57:33.141003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418517+0000","last_change":"2024-01-15T17:57:31.134124+0000","last_active":"2024-01-15T17:57:33.418517+0000","last_peered":"2024-01-15T17:57:33.418517+0000","last_clean":"2024-01-15T17:57:33.418517+0000","last_became_active":"2024-01-15T17:57:31.133698+0000","last_became_peered":"2024-01-15T17:57:31.133698+0000","last_unstale":"2024-01-15T17:57:33.418517+0000","last_undegraded":"2024-01-15T17:57:33.418517+0000","last_fullsized":"2024-01-15T17:57:33.418517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416094+0000","last_change":"2024-01-15T17:57:32.148065+0000","last_active":"2024-01-15T17:57:33.416094+0000","last_peered":"2024-01-15T17:57:33.416094+0000","last_clean":"2024-01-15T17:57:33.416094+0000","last_became_active":"2024-01-15T17:57:32.145370+0000","last_became_peered":"2024-01-15T17:57:32.145370+0000","last_unstale":"2024-01-15T17:57:33.416094+0000","last_undegraded":"2024-01-15T17:57:33.416094+0000","last_fullsized":"2024-01-15T17:57:33.416094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:46.538517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138441+0000","last_change":"2024-01-15T17:57:31.128237+0000","last_active":"2024-01-15T17:57:33.138441+0000","last_peered":"2024-01-15T17:57:33.138441+0000","last_clean":"2024-01-15T17:57:33.138441+0000","last_became_active":"2024-01-15T17:57:31.127956+0000","last_became_peered":"2024-01-15T17:57:31.127956+0000","last_unstale":"2024-01-15T17:57:33.138441+0000","last_undegraded":"2024-01-15T17:57:33.138441+0000","last_fullsized":"2024-01-15T17:57:33.138441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141534+0000","last_change":"2024-01-15T17:57:32.150884+0000","last_active":"2024-01-15T17:57:33.141534+0000","last_peered":"2024-01-15T17:57:33.141534+0000","last_clean":"2024-01-15T17:57:33.141534+0000","last_became_active":"2024-01-15T17:57:32.150381+0000","last_became_peered":"2024-01-15T17:57:32.150381+0000","last_unstale":"2024-01-15T17:57:33.141534+0000","last_undegraded":"2024-01-15T17:57:33.141534+0000","last_fullsized":"2024-01-15T17:57:33.141534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:23.429091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414222+0000","last_change":"2024-01-15T17:57:31.138127+0000","last_active":"2024-01-15T17:57:33.414222+0000","last_peered":"2024-01-15T17:57:33.414222+0000","last_clean":"2024-01-15T17:57:33.414222+0000","last_became_active":"2024-01-15T17:57:31.137579+0000","last_became_peered":"2024-01-15T17:57:31.137579+0000","last_unstale":"2024-01-15T17:57:33.414222+0000","last_undegraded":"2024-01-15T17:57:33.414222+0000","last_fullsized":"2024-01-15T17:57:33.414222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414356+0000","last_change":"2024-01-15T17:57:32.150432+0000","last_active":"2024-01-15T17:57:33.414356+0000","last_peered":"2024-01-15T17:57:33.414356+0000","last_clean":"2024-01-15T17:57:33.414356+0000","last_became_active":"2024-01-15T17:57:32.147572+0000","last_became_peered":"2024-01-15T17:57:32.147572+0000","last_unstale":"2024-01-15T17:57:33.414356+0000","last_undegraded":"2024-01-15T17:57:33.414356+0000","last_fullsized":"2024-01-15T17:57:33.414356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414449+0000","last_change":"2024-01-15T17:57:31.134499+0000","last_active":"2024-01-15T17:57:33.414449+0000","last_peered":"2024-01-15T17:57:33.414449+0000","last_clean":"2024-01-15T17:57:33.414449+0000","last_became_active":"2024-01-15T17:57:31.131997+0000","last_became_peered":"2024-01-15T17:57:31.131997+0000","last_unstale":"2024-01-15T17:57:33.414449+0000","last_undegraded":"2024-01-15T17:57:33.414449+0000","last_fullsized":"2024-01-15T17:57:33.414449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:11.214052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139065+0000","last_change":"2024-01-15T17:57:32.151412+0000","last_active":"2024-01-15T17:57:33.139065+0000","last_peered":"2024-01-15T17:57:33.139065+0000","last_clean":"2024-01-15T17:57:33.139065+0000","last_became_active":"2024-01-15T17:57:32.151061+0000","last_became_peered":"2024-01-15T17:57:32.151061+0000","last_unstale":"2024-01-15T17:57:33.139065+0000","last_undegraded":"2024-01-15T17:57:33.139065+0000","last_fullsized":"2024-01-15T17:57:33.139065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186594+0000","last_change":"2024-01-15T17:57:31.133132+0000","last_active":"2024-01-15T17:57:33.186594+0000","last_peered":"2024-01-15T17:57:33.186594+0000","last_clean":"2024-01-15T17:57:33.186594+0000","last_became_active":"2024-01-15T17:57:31.131606+0000","last_became_peered":"2024-01-15T17:57:31.131606+0000","last_unstale":"2024-01-15T17:57:33.186594+0000","last_undegraded":"2024-01-15T17:57:33.186594+0000","last_fullsized":"2024-01-15T17:57:33.186594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:58:03.663628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139362+0000","last_change":"2024-01-15T17:57:32.152397+0000","last_active":"2024-01-15T17:57:33.139362+0000","last_peered":"2024-01-15T17:57:33.139362+0000","last_clean":"2024-01-15T17:57:33.139362+0000","last_became_active":"2024-01-15T17:57:32.152096+0000","last_became_peered":"2024-01-15T17:57:32.152096+0000","last_unstale":"2024-01-15T17:57:33.139362+0000","last_undegraded":"2024-01-15T17:57:33.139362+0000","last_fullsized":"2024-01-15T17:57:33.139362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417449+0000","last_change":"2024-01-15T17:57:31.127107+0000","last_active":"2024-01-15T17:57:33.417449+0000","last_peered":"2024-01-15T17:57:33.417449+0000","last_clean":"2024-01-15T17:57:33.417449+0000","last_became_active":"2024-01-15T17:57:31.126871+0000","last_became_peered":"2024-01-15T17:57:31.126871+0000","last_unstale":"2024-01-15T17:57:33.417449+0000","last_undegraded":"2024-01-15T17:57:33.417449+0000","last_fullsized":"2024-01-15T17:57:33.417449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:20.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140382+0000","last_change":"2024-01-15T17:57:32.150858+0000","last_active":"2024-01-15T17:57:33.140382+0000","last_peered":"2024-01-15T17:57:33.140382+0000","last_clean":"2024-01-15T17:57:33.140382+0000","last_became_active":"2024-01-15T17:57:32.150312+0000","last_became_peered":"2024-01-15T17:57:32.150312+0000","last_unstale":"2024-01-15T17:57:33.140382+0000","last_undegraded":"2024-01-15T17:57:33.140382+0000","last_fullsized":"2024-01-15T17:57:33.140382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136681+0000","last_change":"2024-01-15T17:57:31.136226+0000","last_active":"2024-01-15T17:57:33.136681+0000","last_peered":"2024-01-15T17:57:33.136681+0000","last_clean":"2024-01-15T17:57:33.136681+0000","last_became_active":"2024-01-15T17:57:31.135879+0000","last_became_peered":"2024-01-15T17:57:31.135879+0000","last_unstale":"2024-01-15T17:57:33.136681+0000","last_undegraded":"2024-01-15T17:57:33.136681+0000","last_fullsized":"2024-01-15T17:57:33.136681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.149505+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.144115+0000","last_became_peered":"2024-01-15T17:57:32.144115+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415379+0000","last_change":"2024-01-15T17:57:31.130353+0000","last_active":"2024-01-15T17:57:33.415379+0000","last_peered":"2024-01-15T17:57:33.415379+0000","last_clean":"2024-01-15T17:57:33.415379+0000","last_became_active":"2024-01-15T17:57:31.130063+0000","last_became_peered":"2024-01-15T17:57:31.130063+0000","last_unstale":"2024-01-15T17:57:33.415379+0000","last_undegraded":"2024-01-15T17:57:33.415379+0000","last_fullsized":"2024-01-15T17:57:33.415379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:42.825666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141047+0000","last_change":"2024-01-15T17:57:32.148976+0000","last_active":"2024-01-15T17:57:33.141047+0000","last_peered":"2024-01-15T17:57:33.141047+0000","last_clean":"2024-01-15T17:57:33.141047+0000","last_became_active":"2024-01-15T17:57:32.148285+0000","last_became_peered":"2024-01-15T17:57:32.148285+0000","last_unstale":"2024-01-15T17:57:33.141047+0000","last_undegraded":"2024-01-15T17:57:33.141047+0000","last_fullsized":"2024-01-15T17:57:33.141047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T17:57:32.126659+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:57:32.126659+0000","last_peered":"2024-01-15T17:57:32.126659+0000","last_clean":"2024-01-15T17:57:32.126659+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:57:32.126659+0000","last_undegraded":"2024-01-15T17:57:32.126659+0000","last_fullsized":"2024-01-15T17:57:32.126659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138248+0000","last_change":"2024-01-15T17:57:31.127921+0000","last_active":"2024-01-15T17:57:33.138248+0000","last_peered":"2024-01-15T17:57:33.138248+0000","last_clean":"2024-01-15T17:57:33.138248+0000","last_became_active":"2024-01-15T17:57:31.126423+0000","last_became_peered":"2024-01-15T17:57:31.126423+0000","last_unstale":"2024-01-15T17:57:33.138248+0000","last_undegraded":"2024-01-15T17:57:33.138248+0000","last_fullsized":"2024-01-15T17:57:33.138248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:43:31.770732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418628+0000","last_change":"2024-01-15T17:57:32.151647+0000","last_active":"2024-01-15T17:57:33.418628+0000","last_peered":"2024-01-15T17:57:33.418628+0000","last_clean":"2024-01-15T17:57:33.418628+0000","last_became_active":"2024-01-15T17:57:32.149470+0000","last_became_peered":"2024-01-15T17:57:32.149470+0000","last_unstale":"2024-01-15T17:57:33.418628+0000","last_undegraded":"2024-01-15T17:57:33.418628+0000","last_fullsized":"2024-01-15T17:57:33.418628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139414+0000","last_change":"2024-01-15T17:57:31.141208+0000","last_active":"2024-01-15T17:57:33.139414+0000","last_peered":"2024-01-15T17:57:33.139414+0000","last_clean":"2024-01-15T17:57:33.139414+0000","last_became_active":"2024-01-15T17:57:31.140756+0000","last_became_peered":"2024-01-15T17:57:31.140756+0000","last_unstale":"2024-01-15T17:57:33.139414+0000","last_undegraded":"2024-01-15T17:57:33.139414+0000","last_fullsized":"2024-01-15T17:57:33.139414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:17:40.803403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415471+0000","last_change":"2024-01-15T17:57:32.150522+0000","last_active":"2024-01-15T17:57:33.415471+0000","last_peered":"2024-01-15T17:57:33.415471+0000","last_clean":"2024-01-15T17:57:33.415471+0000","last_became_active":"2024-01-15T17:57:32.150224+0000","last_became_peered":"2024-01-15T17:57:32.150224+0000","last_unstale":"2024-01-15T17:57:33.415471+0000","last_undegraded":"2024-01-15T17:57:33.415471+0000","last_fullsized":"2024-01-15T17:57:33.415471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:03.799210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414399+0000","last_change":"2024-01-15T17:57:31.127558+0000","last_active":"2024-01-15T17:57:33.414399+0000","last_peered":"2024-01-15T17:57:33.414399+0000","last_clean":"2024-01-15T17:57:33.414399+0000","last_became_active":"2024-01-15T17:57:31.125325+0000","last_became_peered":"2024-01-15T17:57:31.125325+0000","last_unstale":"2024-01-15T17:57:33.414399+0000","last_undegraded":"2024-01-15T17:57:33.414399+0000","last_fullsized":"2024-01-15T17:57:33.414399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:56.231915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136344+0000","last_change":"2024-01-15T17:57:32.134693+0000","last_active":"2024-01-15T17:57:33.136344+0000","last_peered":"2024-01-15T17:57:33.136344+0000","last_clean":"2024-01-15T17:57:33.136344+0000","last_became_active":"2024-01-15T17:57:32.133773+0000","last_became_peered":"2024-01-15T17:57:32.133773+0000","last_unstale":"2024-01-15T17:57:33.136344+0000","last_undegraded":"2024-01-15T17:57:33.136344+0000","last_fullsized":"2024-01-15T17:57:33.136344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138654+0000","last_change":"2024-01-15T17:57:31.133660+0000","last_active":"2024-01-15T17:57:33.138654+0000","last_peered":"2024-01-15T17:57:33.138654+0000","last_clean":"2024-01-15T17:57:33.138654+0000","last_became_active":"2024-01-15T17:57:31.132627+0000","last_became_peered":"2024-01-15T17:57:31.132627+0000","last_unstale":"2024-01-15T17:57:33.138654+0000","last_undegraded":"2024-01-15T17:57:33.138654+0000","last_fullsized":"2024-01-15T17:57:33.138654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186541+0000","last_change":"2024-01-15T17:57:32.147182+0000","last_active":"2024-01-15T17:57:33.186541+0000","last_peered":"2024-01-15T17:57:33.186541+0000","last_clean":"2024-01-15T17:57:33.186541+0000","last_became_active":"2024-01-15T17:57:32.146838+0000","last_became_peered":"2024-01-15T17:57:32.146838+0000","last_unstale":"2024-01-15T17:57:33.186541+0000","last_undegraded":"2024-01-15T17:57:33.186541+0000","last_fullsized":"2024-01-15T17:57:33.186541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136810+0000","last_change":"2024-01-15T17:57:31.126189+0000","last_active":"2024-01-15T17:57:33.136810+0000","last_peered":"2024-01-15T17:57:33.136810+0000","last_clean":"2024-01-15T17:57:33.136810+0000","last_became_active":"2024-01-15T17:57:31.125519+0000","last_became_peered":"2024-01-15T17:57:31.125519+0000","last_unstale":"2024-01-15T17:57:33.136810+0000","last_undegraded":"2024-01-15T17:57:33.136810+0000","last_fullsized":"2024-01-15T17:57:33.136810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.137059+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.136012+0000","last_became_peered":"2024-01-15T17:57:32.136012+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141434+0000","last_change":"2024-01-15T17:57:31.139980+0000","last_active":"2024-01-15T17:57:33.141434+0000","last_peered":"2024-01-15T17:57:33.141434+0000","last_clean":"2024-01-15T17:57:33.141434+0000","last_became_active":"2024-01-15T17:57:31.139489+0000","last_became_peered":"2024-01-15T17:57:31.139489+0000","last_unstale":"2024-01-15T17:57:33.141434+0000","last_undegraded":"2024-01-15T17:57:33.141434+0000","last_fullsized":"2024-01-15T17:57:33.141434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:54.272945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136716+0000","last_change":"2024-01-15T17:57:32.130379+0000","last_active":"2024-01-15T17:57:33.136716+0000","last_peered":"2024-01-15T17:57:33.136716+0000","last_clean":"2024-01-15T17:57:33.136716+0000","last_became_active":"2024-01-15T17:57:32.129645+0000","last_became_peered":"2024-01-15T17:57:32.129645+0000","last_unstale":"2024-01-15T17:57:33.136716+0000","last_undegraded":"2024-01-15T17:57:33.136716+0000","last_fullsized":"2024-01-15T17:57:33.136716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136624+0000","last_change":"2024-01-15T17:57:31.146382+0000","last_active":"2024-01-15T17:57:33.136624+0000","last_peered":"2024-01-15T17:57:33.136624+0000","last_clean":"2024-01-15T17:57:33.136624+0000","last_became_active":"2024-01-15T17:57:31.146119+0000","last_became_peered":"2024-01-15T17:57:31.146119+0000","last_unstale":"2024-01-15T17:57:33.136624+0000","last_undegraded":"2024-01-15T17:57:33.136624+0000","last_fullsized":"2024-01-15T17:57:33.136624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:30.690538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418481+0000","last_change":"2024-01-15T17:57:32.144183+0000","last_active":"2024-01-15T17:57:33.418481+0000","last_peered":"2024-01-15T17:57:33.418481+0000","last_clean":"2024-01-15T17:57:33.418481+0000","last_became_active":"2024-01-15T17:57:32.143880+0000","last_became_peered":"2024-01-15T17:57:32.143880+0000","last_unstale":"2024-01-15T17:57:33.418481+0000","last_undegraded":"2024-01-15T17:57:33.418481+0000","last_fullsized":"2024-01-15T17:57:33.418481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415224+0000","last_change":"2024-01-15T17:57:31.144154+0000","last_active":"2024-01-15T17:57:33.415224+0000","last_peered":"2024-01-15T17:57:33.415224+0000","last_clean":"2024-01-15T17:57:33.415224+0000","last_became_active":"2024-01-15T17:57:31.143857+0000","last_became_peered":"2024-01-15T17:57:31.143857+0000","last_unstale":"2024-01-15T17:57:33.415224+0000","last_undegraded":"2024-01-15T17:57:33.415224+0000","last_fullsized":"2024-01-15T17:57:33.415224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419365+0000","last_change":"2024-01-15T17:57:32.142460+0000","last_active":"2024-01-15T17:57:33.419365+0000","last_peered":"2024-01-15T17:57:33.419365+0000","last_clean":"2024-01-15T17:57:33.419365+0000","last_became_active":"2024-01-15T17:57:32.141816+0000","last_became_peered":"2024-01-15T17:57:32.141816+0000","last_unstale":"2024-01-15T17:57:33.419365+0000","last_undegraded":"2024-01-15T17:57:33.419365+0000","last_fullsized":"2024-01-15T17:57:33.419365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139202+0000","last_change":"2024-01-15T17:57:31.139784+0000","last_active":"2024-01-15T17:57:33.139202+0000","last_peered":"2024-01-15T17:57:33.139202+0000","last_clean":"2024-01-15T17:57:33.139202+0000","last_became_active":"2024-01-15T17:57:31.138181+0000","last_became_peered":"2024-01-15T17:57:31.138181+0000","last_unstale":"2024-01-15T17:57:33.139202+0000","last_undegraded":"2024-01-15T17:57:33.139202+0000","last_fullsized":"2024-01-15T17:57:33.139202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:50.662857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186323+0000","last_change":"2024-01-15T17:57:32.130289+0000","last_active":"2024-01-15T17:57:33.186323+0000","last_peered":"2024-01-15T17:57:33.186323+0000","last_clean":"2024-01-15T17:57:33.186323+0000","last_became_active":"2024-01-15T17:57:32.129807+0000","last_became_peered":"2024-01-15T17:57:32.129807+0000","last_unstale":"2024-01-15T17:57:33.186323+0000","last_undegraded":"2024-01-15T17:57:33.186323+0000","last_fullsized":"2024-01-15T17:57:33.186323+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:43.951697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135874+0000","last_change":"2024-01-15T17:57:31.132917+0000","last_active":"2024-01-15T17:57:33.135874+0000","last_peered":"2024-01-15T17:57:33.135874+0000","last_clean":"2024-01-15T17:57:33.135874+0000","last_became_active":"2024-01-15T17:57:31.132650+0000","last_became_peered":"2024-01-15T17:57:31.132650+0000","last_unstale":"2024-01-15T17:57:33.135874+0000","last_undegraded":"2024-01-15T17:57:33.135874+0000","last_fullsized":"2024-01-15T17:57:33.135874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415920+0000","last_change":"2024-01-15T17:57:32.136009+0000","last_active":"2024-01-15T17:57:33.415920+0000","last_peered":"2024-01-15T17:57:33.415920+0000","last_clean":"2024-01-15T17:57:33.415920+0000","last_became_active":"2024-01-15T17:57:32.135200+0000","last_became_peered":"2024-01-15T17:57:32.135200+0000","last_unstale":"2024-01-15T17:57:33.415920+0000","last_undegraded":"2024-01-15T17:57:33.415920+0000","last_fullsized":"2024-01-15T17:57:33.415920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136090+0000","last_change":"2024-01-15T17:57:31.130465+0000","last_active":"2024-01-15T17:57:33.136090+0000","last_peered":"2024-01-15T17:57:33.136090+0000","last_clean":"2024-01-15T17:57:33.136090+0000","last_became_active":"2024-01-15T17:57:31.130032+0000","last_became_peered":"2024-01-15T17:57:31.130032+0000","last_unstale":"2024-01-15T17:57:33.136090+0000","last_undegraded":"2024-01-15T17:57:33.136090+0000","last_fullsized":"2024-01-15T17:57:33.136090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417702+0000","last_change":"2024-01-15T17:57:32.154915+0000","last_active":"2024-01-15T17:57:33.417702+0000","last_peered":"2024-01-15T17:57:33.417702+0000","last_clean":"2024-01-15T17:57:33.417702+0000","last_became_active":"2024-01-15T17:57:32.154599+0000","last_became_peered":"2024-01-15T17:57:32.154599+0000","last_unstale":"2024-01-15T17:57:33.417702+0000","last_undegraded":"2024-01-15T17:57:33.417702+0000","last_fullsized":"2024-01-15T17:57:33.417702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141769+0000","last_change":"2024-01-15T17:57:31.143380+0000","last_active":"2024-01-15T17:57:33.141769+0000","last_peered":"2024-01-15T17:57:33.141769+0000","last_clean":"2024-01-15T17:57:33.141769+0000","last_became_active":"2024-01-15T17:57:31.143046+0000","last_became_peered":"2024-01-15T17:57:31.143046+0000","last_unstale":"2024-01-15T17:57:33.141769+0000","last_undegraded":"2024-01-15T17:57:33.141769+0000","last_fullsized":"2024-01-15T17:57:33.141769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414379+0000","last_change":"2024-01-15T17:57:32.149247+0000","last_active":"2024-01-15T17:57:33.414379+0000","last_peered":"2024-01-15T17:57:33.414379+0000","last_clean":"2024-01-15T17:57:33.414379+0000","last_became_active":"2024-01-15T17:57:32.148620+0000","last_became_peered":"2024-01-15T17:57:32.148620+0000","last_unstale":"2024-01-15T17:57:33.414379+0000","last_undegraded":"2024-01-15T17:57:33.414379+0000","last_fullsized":"2024-01-15T17:57:33.414379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139434+0000","last_change":"2024-01-15T17:57:32.159609+0000","last_active":"2024-01-15T17:57:33.139434+0000","last_peered":"2024-01-15T17:57:33.139434+0000","last_clean":"2024-01-15T17:57:33.139434+0000","last_became_active":"2024-01-15T17:57:32.155889+0000","last_became_peered":"2024-01-15T17:57:32.155889+0000","last_unstale":"2024-01-15T17:57:33.139434+0000","last_undegraded":"2024-01-15T17:57:33.139434+0000","last_fullsized":"2024-01-15T17:57:33.139434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138536+0000","last_change":"2024-01-15T17:57:31.144524+0000","last_active":"2024-01-15T17:57:33.138536+0000","last_peered":"2024-01-15T17:57:33.138536+0000","last_clean":"2024-01-15T17:57:33.138536+0000","last_became_active":"2024-01-15T17:57:31.143299+0000","last_became_peered":"2024-01-15T17:57:31.143299+0000","last_unstale":"2024-01-15T17:57:33.138536+0000","last_undegraded":"2024-01-15T17:57:33.138536+0000","last_fullsized":"2024-01-15T17:57:33.138536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139433+0000","last_change":"2024-01-15T17:57:32.151244+0000","last_active":"2024-01-15T17:57:33.139433+0000","last_peered":"2024-01-15T17:57:33.139433+0000","last_clean":"2024-01-15T17:57:33.139433+0000","last_became_active":"2024-01-15T17:57:32.150834+0000","last_became_peered":"2024-01-15T17:57:32.150834+0000","last_unstale":"2024-01-15T17:57:33.139433+0000","last_undegraded":"2024-01-15T17:57:33.139433+0000","last_fullsized":"2024-01-15T17:57:33.139433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141240+0000","last_change":"2024-01-15T17:57:31.144722+0000","last_active":"2024-01-15T17:57:33.141240+0000","last_peered":"2024-01-15T17:57:33.141240+0000","last_clean":"2024-01-15T17:57:33.141240+0000","last_became_active":"2024-01-15T17:57:31.144378+0000","last_became_peered":"2024-01-15T17:57:31.144378+0000","last_unstale":"2024-01-15T17:57:33.141240+0000","last_undegraded":"2024-01-15T17:57:33.141240+0000","last_fullsized":"2024-01-15T17:57:33.141240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:48.548981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419383+0000","last_change":"2024-01-15T17:57:32.142742+0000","last_active":"2024-01-15T17:57:33.419383+0000","last_peered":"2024-01-15T17:57:33.419383+0000","last_clean":"2024-01-15T17:57:33.419383+0000","last_became_active":"2024-01-15T17:57:32.142372+0000","last_became_peered":"2024-01-15T17:57:32.142372+0000","last_unstale":"2024-01-15T17:57:33.419383+0000","last_undegraded":"2024-01-15T17:57:33.419383+0000","last_fullsized":"2024-01-15T17:57:33.419383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417735+0000","last_change":"2024-01-15T17:57:31.128188+0000","last_active":"2024-01-15T17:57:33.417735+0000","last_peered":"2024-01-15T17:57:33.417735+0000","last_clean":"2024-01-15T17:57:33.417735+0000","last_became_active":"2024-01-15T17:57:31.127089+0000","last_became_peered":"2024-01-15T17:57:31.127089+0000","last_unstale":"2024-01-15T17:57:33.417735+0000","last_undegraded":"2024-01-15T17:57:33.417735+0000","last_fullsized":"2024-01-15T17:57:33.417735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417498+0000","last_change":"2024-01-15T17:57:32.142205+0000","last_active":"2024-01-15T17:57:33.417498+0000","last_peered":"2024-01-15T17:57:33.417498+0000","last_clean":"2024-01-15T17:57:33.417498+0000","last_became_active":"2024-01-15T17:57:32.139881+0000","last_became_peered":"2024-01-15T17:57:32.139881+0000","last_unstale":"2024-01-15T17:57:33.417498+0000","last_undegraded":"2024-01-15T17:57:33.417498+0000","last_fullsized":"2024-01-15T17:57:33.417498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419783+0000","last_change":"2024-01-15T17:57:31.140404+0000","last_active":"2024-01-15T17:57:33.419783+0000","last_peered":"2024-01-15T17:57:33.419783+0000","last_clean":"2024-01-15T17:57:33.419783+0000","last_became_active":"2024-01-15T17:57:31.140098+0000","last_became_peered":"2024-01-15T17:57:31.140098+0000","last_unstale":"2024-01-15T17:57:33.419783+0000","last_undegraded":"2024-01-15T17:57:33.419783+0000","last_fullsized":"2024-01-15T17:57:33.419783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542147,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.798}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.665}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.419}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8380000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.407}]}]},{"osd":6,"up_from":14,"seq":60129542147,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":74114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.333}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4970000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8450000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6449999999999996}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3460000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.53}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6579999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9079999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.2389999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0230000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542148,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96599999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6140000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.889}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5249999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.161}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.278}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1789999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.536}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7230000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.679}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.145}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9830000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542148,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5669999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1990000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.286}]}]},{"osd":0,"up_from":14,"seq":60129542147,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":831488,"data_stored":669808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96299999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1479999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.74}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7789999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:57:37.128 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T17:57:37.128 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-15T17:57:37.128 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-15T17:57:37.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-15T17:57:37.129 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-15T17:57:37.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.389+0000 7f1434800640 1 Processor -- start 2024-01-15T17:57:37.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.389+0000 7f1434800640 1 -- start start 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1434800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f142c15c0e0 0x7f142c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1434800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1434800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f142c15c4c0 0x7f142c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1434800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c15cb00 con 0x7f142c15c4c0 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1434800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c0ccec0 con 0x7f142c15c0e0 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1434800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0cd1a0 con 0x7f142c15b7d0 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1432800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f142c15c4c0 0x7f142c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1432800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f142c15c4c0 0x7f142c06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49590/0 (socket says 172.21.15.2:49590) 2024-01-15T17:57:37.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1432800640 1 -- 172.21.15.2:0/622574524 learned_addr learned my addr 172.21.15.2:0/622574524 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1431e00640 1 --2- 172.21.15.2:0/622574524 >> v2:172.21.15.137:3300/0 conn(0x7f142c15c0e0 0x7f142c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.391+0000 7f1431400640 1 --2- 172.21.15.2:0/622574524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.392+0000 7f1432800640 1 -- 172.21.15.2:0/622574524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 msgr2=0x7f142c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.392+0000 7f1432800640 1 --2- 172.21.15.2:0/622574524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.392+0000 7f1432800640 1 -- 172.21.15.2:0/622574524 >> v2:172.21.15.137:3300/0 conn(0x7f142c15c0e0 msgr2=0x7f142c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.392+0000 7f1432800640 1 --2- 172.21.15.2:0/622574524 >> v2:172.21.15.137:3300/0 conn(0x7f142c15c0e0 0x7f142c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.392+0000 7f1432800640 1 -- 172.21.15.2:0/622574524 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c0cd480 con 0x7f142c15c4c0 2024-01-15T17:57:37.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.393+0000 7f1432800640 1 --2- 172.21.15.2:0/622574524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c06c9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f142c0725f0 tx=0x7f142402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca16cb3c14a9b96 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:37.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.393+0000 7f1430a00640 1 -- 172.21.15.2:0/622574524 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142402ca70 con 0x7f142c15c4c0 2024-01-15T17:57:37.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.393+0000 7f1430a00640 1 -- 172.21.15.2:0/622574524 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142402cbd0 con 0x7f142c15c4c0 2024-01-15T17:57:37.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.393+0000 7f1430a00640 1 -- 172.21.15.2:0/622574524 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14240343f0 con 0x7f142c15c4c0 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 -- 172.21.15.2:0/622574524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 msgr2=0x7f142c06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 --2- 172.21.15.2:0/622574524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c06c9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f142c0725f0 tx=0x7f142402bed0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 -- 172.21.15.2:0/622574524 shutdown_connections 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 --2- 172.21.15.2:0/622574524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c06c9b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 --2- 172.21.15.2:0/622574524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 --2- 172.21.15.2:0/622574524 >> v2:172.21.15.137:3300/0 conn(0x7f142c15c0e0 0x7f142c0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 -- 172.21.15.2:0/622574524 >> 172.21.15.2:0/622574524 conn(0x7f142c06da20 msgr2=0x7f142c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 -- 172.21.15.2:0/622574524 shutdown_connections 2024-01-15T17:57:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.394+0000 7f1434800640 1 -- 172.21.15.2:0/622574524 wait complete. 2024-01-15T17:57:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.396+0000 7f1434800640 1 Processor -- start 2024-01-15T17:57:37.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.396+0000 7f1434800640 1 -- start start 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1434800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c14ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1434800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c15c0e0 0x7f142c0ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1434800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1434800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c14a700 con 0x7f142c15c4c0 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1434800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c14b240 con 0x7f142c15c0e0 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1434800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c0ae350 con 0x7f142c15b7d0 2024-01-15T17:57:37.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1432800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1432800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49596/0 (socket says 172.21.15.2:49596) 2024-01-15T17:57:37.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.397+0000 7f1432800640 1 -- 172.21.15.2:0/4240357342 learned_addr learned my addr 172.21.15.2:0/4240357342 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1431400640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c15c0e0 0x7f142c0ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1431e00640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c14ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1432800640 1 -- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 msgr2=0x7f142c14ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1432800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c14ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1432800640 1 -- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c15c0e0 msgr2=0x7f142c0ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1432800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c15c0e0 0x7f142c0ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1432800640 1 -- 172.21.15.2:0/4240357342 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1424009b30 con 0x7f142c15c4c0 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1431e00640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c14ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1431400640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c15c0e0 0x7f142c0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:37.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1432800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c15cf90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1424036040 tx=0x7f1424004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424009510 con 0x7f142c15c4c0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142403e450 con 0x7f142c15c4c0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142403e800 con 0x7f142c15c4c0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.398+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c14b450 con 0x7f142c15c4c0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.399+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c14ba10 con 0x7f142c15c4c0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.400+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c0686f0 con 0x7f142c15c4c0 2024-01-15T17:57:37.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.401+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f142403e960 con 0x7f142c15c4c0 2024-01-15T17:57:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.402+0000 7f142a000640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406b2c0 0x7f140406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.402+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14240ac970 con 0x7f142c15c4c0 2024-01-15T17:57:37.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.403+0000 7f1431e00640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406b2c0 0x7f140406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.404+0000 7f1431e00640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406b2c0 0x7f140406d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f141c006fd0 tx=0x7f141c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:37.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.407+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f14240777d0 con 0x7f142c15c4c0 2024-01-15T17:57:37.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.648+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f142c143cf0 con 0x7f142c15c4c0 2024-01-15T17:57:37.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.654+0000 7f142a000640 1 -- 172.21.15.2:0/4240357342 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171565 (secure 0 0 0) 0x7f142407adf0 con 0x7f142c15c4c0 2024-01-15T17:57:37.660 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:37.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406b2c0 msgr2=0x7f140406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406b2c0 0x7f140406d780 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f141c006fd0 tx=0x7f141c008040 comp rx=0 tx=0).stop 2024-01-15T17:57:37.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 msgr2=0x7f142c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c15cf90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1424036040 tx=0x7f1424004a40 comp rx=0 tx=0).stop 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 shutdown_connections 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406b2c0 0x7f140406d780 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15c4c0 0x7f142c15cf90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c15c0e0 0x7f142c0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 --2- 172.21.15.2:0/4240357342 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15b7d0 0x7f142c14ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.658+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 >> 172.21.15.2:0/4240357342 conn(0x7f142c06da20 msgr2=0x7f142c0aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.659+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 shutdown_connections 2024-01-15T17:57:37.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.659+0000 7f1434800640 1 -- 172.21.15.2:0/4240357342 wait complete. 2024-01-15T17:57:37.682 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":5,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":2238446093},{"type":"v1","addr":"172.21.15.137:6833","nonce":2238446093}]},"active_addr":"172.21.15.137:6833/2238446093","active_change":"2024-01-15T17:57:17.348509+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1063284537}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2311232397}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3284635469}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2189891883}]}]} 2024-01-15T17:57:37.686 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-15T17:57:37.686 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-15T17:57:37.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.904+0000 7fb681800640 1 Processor -- start 2024-01-15T17:57:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.904+0000 7fb681800640 1 -- start start 2024-01-15T17:57:37.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.906+0000 7fb681800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.906+0000 7fb681800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb67c15ec60 0x7fb67c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.906+0000 7fb681800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67c06dae0 con 0x7fb67c0cd490 2024-01-15T17:57:37.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.906+0000 7fb681800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67c071f20 con 0x7fb67c15ec60 2024-01-15T17:57:37.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.906+0000 7fb681800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c06c6b0 con 0x7fb67c141880 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.907+0000 7fb673200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb67c15ec60 0x7fb67c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.907+0000 7fb67be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.907+0000 7fb673200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb67c15ec60 0x7fb67c15f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43262/0 (socket says 172.21.15.2:43262) 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.907+0000 7fb673200640 1 -- 172.21.15.2:0/376734530 learned_addr learned my addr 172.21.15.2:0/376734530 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.908+0000 7fb673200640 1 -- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 msgr2=0x7fb67c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.908+0000 7fb673200640 1 --2- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.908+0000 7fb673200640 1 -- 172.21.15.2:0/376734530 >> v1:172.21.15.137:6789/0 conn(0x7fb67c0cd490 legacy=0x7fb67c0cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.908+0000 7fb673200640 1 -- 172.21.15.2:0/376734530 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67c071960 con 0x7fb67c15ec60 2024-01-15T17:57:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.909+0000 7fb673200640 1 --2- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c15ec60 0x7fb67c15f0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb664009870 tx=0x7fb66402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77599391d66a20df server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb67b400640 1 -- 172.21.15.2:0/376734530 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb664031640 con 0x7fb67c15ec60 2024-01-15T17:57:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb67b400640 1 -- 172.21.15.2:0/376734530 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6640317a0 con 0x7fb67c15ec60 2024-01-15T17:57:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb67b400640 1 -- 172.21.15.2:0/376734530 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb664031b30 con 0x7fb67c15ec60 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb681800640 1 -- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c15ec60 msgr2=0x7fb67c15f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb681800640 1 --2- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c15ec60 0x7fb67c15f0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb664009870 tx=0x7fb66402b8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb681800640 1 -- 172.21.15.2:0/376734530 shutdown_connections 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb681800640 1 --2- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c15ec60 0x7fb67c15f0e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb681800640 1 --2- 172.21.15.2:0/376734530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.910+0000 7fb681800640 1 -- 172.21.15.2:0/376734530 >> 172.21.15.2:0/376734530 conn(0x7fb67c0b0840 msgr2=0x7fb67c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.911+0000 7fb681800640 1 -- 172.21.15.2:0/376734530 shutdown_connections 2024-01-15T17:57:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.911+0000 7fb681800640 1 -- 172.21.15.2:0/376734530 wait complete. 2024-01-15T17:57:37.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.912+0000 7fb681800640 1 Processor -- start 2024-01-15T17:57:37.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.913+0000 7fb681800640 1 -- start start 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb681800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c0cd490 0x7fb67c13e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb681800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c0b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb681800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 0x7fb67c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb681800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c159680 con 0x7fb67c15ec60 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb681800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0b2800 con 0x7fb67c0cd490 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb681800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c0b2ae0 con 0x7fb67c141880 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb680e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 0x7fb67c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb680e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 0x7fb67c0b2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49604/0 (socket says 172.21.15.2:49604) 2024-01-15T17:57:37.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb680e00640 1 -- 172.21.15.2:0/2934732234 learned_addr learned my addr 172.21.15.2:0/2934732234 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:37.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb67be00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c0cd490 0x7fb67c13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb673200640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c0b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.914+0000 7fb680e00640 1 -- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 msgr2=0x7fb67c0b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb680e00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c0b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb680e00640 1 -- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c0cd490 msgr2=0x7fb67c13e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb680e00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c0cd490 0x7fb67c13e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb680e00640 1 -- 172.21.15.2:0/2934732234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6640094d0 con 0x7fb67c15ec60 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb680e00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 0x7fb67c0b2180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb67400c7a0 tx=0x7fb67400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674003b70 con 0x7fb67c15ec60 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb674003cf0 con 0x7fb67c15ec60 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674003ea0 con 0x7fb67c15ec60 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c0b2dc0 con 0x7fb67c15ec60 2024-01-15T17:57:37.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.915+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c0b3380 con 0x7fb67c15ec60 2024-01-15T17:57:37.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.916+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c074770 con 0x7fb67c15ec60 2024-01-15T17:57:37.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.917+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb67400aa90 con 0x7fb67c15ec60 2024-01-15T17:57:37.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.918+0000 7fb678c00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65406b2f0 0x7fb65406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:37.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.918+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb674089a00 con 0x7fb67c15ec60 2024-01-15T17:57:37.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.919+0000 7fb67be00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65406b2f0 0x7fb65406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:37.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.920+0000 7fb67be00640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65406b2f0 0x7fb65406d7b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009ad0 tx=0x7fb66c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:37.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:37.923+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb674054860 con 0x7fb67c15ec60 2024-01-15T17:57:38.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.117+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb67c067670 con 0x7fb67c15ec60 2024-01-15T17:57:38.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.120+0000 7fb678c00640 1 -- 172.21.15.2:0/2934732234 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fb674057e80 con 0x7fb67c15ec60 2024-01-15T17:57:38.124 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:38.124 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:38.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65406b2f0 msgr2=0x7fb65406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65406b2f0 0x7fb65406d7b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009ad0 tx=0x7fb66c008040 comp rx=0 tx=0).stop 2024-01-15T17:57:38.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 msgr2=0x7fb67c0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 0x7fb67c0b2180 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb67400c7a0 tx=0x7fb67400cc60 comp rx=0 tx=0).stop 2024-01-15T17:57:38.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 shutdown_connections 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65406b2f0 0x7fb65406d7b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb67c15ec60 0x7fb67c0b2180 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb67c141880 0x7fb67c0b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 --2- 172.21.15.2:0/2934732234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb67c0cd490 0x7fb67c13e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 >> 172.21.15.2:0/2934732234 conn(0x7fb67c0b0840 msgr2=0x7fb67c0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.123+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 shutdown_connections 2024-01-15T17:57:38.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.124+0000 7fb681800640 1 -- 172.21.15.2:0/2934732234 wait complete. 2024-01-15T17:57:38.146 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-15T17:57:38.146 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:38.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.368+0000 7f50dc600640 1 Processor -- start 2024-01-15T17:57:38.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.368+0000 7f50dc600640 1 -- start start 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50dc600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50d4162440 0x7f50d406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50dc600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d4071cb0 con 0x7f50d406c470 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50dc600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50d4070a70 con 0x7f50d40b3c70 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50dc600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50d4071610 con 0x7f50d4162440 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50d9200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f50d406c470 0x7f50d4161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35400/0 (socket says 172.21.15.2:35400) 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50d9200640 1 -- 172.21.15.2:0/3806459626 learned_addr learned my addr 172.21.15.2:0/3806459626 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:38.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.370+0000 7f50da600640 1 --2- 172.21.15.2:0/3806459626 >> v2:172.21.15.2:3300/0 conn(0x7f50d4162440 0x7f50d406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.371+0000 7f50d3e00640 1 -- 172.21.15.2:0/3806459626 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734231032 0 0) 0x7f50d4071cb0 con 0x7f50d406c470 2024-01-15T17:57:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.371+0000 7f50d3e00640 1 -- 172.21.15.2:0/3806459626 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b4003520 con 0x7f50d406c470 2024-01-15T17:57:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.371+0000 7f50da600640 1 -- 172.21.15.2:0/3806459626 >> v1:172.21.15.137:6789/0 conn(0x7f50d40b3c70 legacy=0x7f50d4070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.371+0000 7f50da600640 1 -- 172.21.15.2:0/3806459626 >> v1:172.21.15.2:6789/0 conn(0x7f50d406c470 legacy=0x7f50d4161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.371+0000 7f50da600640 1 -- 172.21.15.2:0/3806459626 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d4162960 con 0x7f50d4162440 2024-01-15T17:57:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.372+0000 7f50da600640 1 --2- 172.21.15.2:0/3806459626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d406c8f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0097a0 tx=0x7f50cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26fd2324ce664ea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50d3e00640 1 -- 172.21.15.2:0/3806459626 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc030610 con 0x7f50d4162440 2024-01-15T17:57:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50d3e00640 1 -- 172.21.15.2:0/3806459626 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc030770 con 0x7f50d4162440 2024-01-15T17:57:38.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50d3e00640 1 -- 172.21.15.2:0/3806459626 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc030b00 con 0x7f50d4162440 2024-01-15T17:57:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 -- 172.21.15.2:0/3806459626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 msgr2=0x7f50d406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 --2- 172.21.15.2:0/3806459626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d406c8f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0097a0 tx=0x7f50cc02b960 comp rx=0 tx=0).stop 2024-01-15T17:57:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 -- 172.21.15.2:0/3806459626 shutdown_connections 2024-01-15T17:57:38.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 --2- 172.21.15.2:0/3806459626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d406c8f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 -- 172.21.15.2:0/3806459626 >> 172.21.15.2:0/3806459626 conn(0x7f50d406da20 msgr2=0x7f50d406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 -- 172.21.15.2:0/3806459626 shutdown_connections 2024-01-15T17:57:38.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.373+0000 7f50dc600640 1 -- 172.21.15.2:0/3806459626 wait complete. 2024-01-15T17:57:38.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.375+0000 7f50dc600640 1 Processor -- start 2024-01-15T17:57:38.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.375+0000 7f50dc600640 1 -- start start 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.376+0000 7f50dc600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50d406c470 0x7f50d414ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.376+0000 7f50dc600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50d40b3c70 0x7f50d414caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.376+0000 7f50dc600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d40ac0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.376+0000 7f50dc600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d414d340 con 0x7f50d4162440 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.376+0000 7f50dc600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d414dbe0 con 0x7f50d406c470 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.376+0000 7f50dc600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50d40a96d0 con 0x7f50d40b3c70 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d40ac0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d40ac0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49618/0 (socket says 172.21.15.2:49618) 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 -- 172.21.15.2:0/1845332078 learned_addr learned my addr 172.21.15.2:0/1845332078 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 -- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50d40b3c70 msgr2=0x7f50d414caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50d40b3c70 0x7f50d414caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 -- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50d406c470 msgr2=0x7f50d414ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50d9c00640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50d406c470 0x7f50d414ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50d406c470 0x7f50d414ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 -- 172.21.15.2:0/1845332078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc009400 con 0x7f50d4162440 2024-01-15T17:57:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.377+0000 7f50da600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d40ac0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50cc02bf70 tx=0x7f50cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.378+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc03e070 con 0x7f50d4162440 2024-01-15T17:57:38.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.378+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50cc030e10 con 0x7f50d4162440 2024-01-15T17:57:38.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.378+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50cc039bd0 con 0x7f50d4162440 2024-01-15T17:57:38.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.378+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d414d030 con 0x7f50d4162440 2024-01-15T17:57:38.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.378+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d414d590 con 0x7f50d4162440 2024-01-15T17:57:38.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.379+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d40686f0 con 0x7f50d4162440 2024-01-15T17:57:38.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.380+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f50cc039d30 con 0x7f50d4162440 2024-01-15T17:57:38.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.381+0000 7f50d1600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f50ac06b2c0 0x7f50ac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.381+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50cc0abfe0 con 0x7f50d4162440 2024-01-15T17:57:38.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.381+0000 7f50d9c00640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f50ac06b2c0 0x7f50ac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.383+0000 7f50d9c00640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f50ac06b2c0 0x7f50ac06d780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50c4007fd0 tx=0x7f50c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.386+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50cc076f10 con 0x7f50d4162440 2024-01-15T17:57:38.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.579+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50d414dd20 con 0x7f50d4162440 2024-01-15T17:57:38.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.582+0000 7f50d1600640 1 -- 172.21.15.2:0/1845332078 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f50cc034090 con 0x7f50d4162440 2024-01-15T17:57:38.586 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:38.586 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:38.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f50ac06b2c0 msgr2=0x7f50ac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f50ac06b2c0 0x7f50ac06d780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f50c4007fd0 tx=0x7f50c4009040 comp rx=0 tx=0).stop 2024-01-15T17:57:38.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 msgr2=0x7f50d40ac0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d40ac0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f50cc02bf70 tx=0x7f50cc0046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 shutdown_connections 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f50ac06b2c0 0x7f50ac06d780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50d4162440 0x7f50d40ac0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50d40b3c70 0x7f50d414caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 --2- 172.21.15.2:0/1845332078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50d406c470 0x7f50d414ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 >> 172.21.15.2:0/1845332078 conn(0x7f50d406da20 msgr2=0x7f50d40cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 shutdown_connections 2024-01-15T17:57:38.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.585+0000 7f50dc600640 1 -- 172.21.15.2:0/1845332078 wait complete. 2024-01-15T17:57:38.607 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-15T17:57:38.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-15T17:57:38.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-15T17:57:38.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-15T17:57:38.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-15T17:57:38.609 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-15T17:57:38.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-15T17:57:38.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-15T17:57:38.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.901+0000 7f7638200640 1 Processor -- start 2024-01-15T17:57:38.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.902+0000 7f7638200640 1 -- start start 2024-01-15T17:57:38.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7638200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76300b1360 0x7f76300cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7638200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7630071b60 0x7f76300ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7638200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f76301593d0 0x7f76300cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7638200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76300b3990 con 0x7f76301593d0 2024-01-15T17:57:38.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7638200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76300b3e20 con 0x7f7630071b60 2024-01-15T17:57:38.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7638200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76300ce810 con 0x7f76300b1360 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7636200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f76301593d0 0x7f76300cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7636200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f76301593d0 0x7f76300cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49634/0 (socket says 172.21.15.2:49634) 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.904+0000 7f7636200640 1 -- 172.21.15.2:0/1071907899 learned_addr learned my addr 172.21.15.2:0/1071907899 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.905+0000 7f7636200640 1 -- 172.21.15.2:0/1071907899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76300b1360 msgr2=0x7f76300cdfa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.905+0000 7f7636200640 1 --2- 172.21.15.2:0/1071907899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76300b1360 0x7f76300cdfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.905+0000 7f7636200640 1 -- 172.21.15.2:0/1071907899 >> v2:172.21.15.137:3300/0 conn(0x7f7630071b60 msgr2=0x7f76300ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.905+0000 7f7636200640 1 --2- 172.21.15.2:0/1071907899 >> v2:172.21.15.137:3300/0 conn(0x7f7630071b60 0x7f76300ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.905+0000 7f7636200640 1 -- 172.21.15.2:0/1071907899 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76300b3340 con 0x7f76301593d0 2024-01-15T17:57:38.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.907+0000 7f7636200640 1 --2- 172.21.15.2:0/1071907899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f76300cd090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7630072aa0 tx=0x7f761802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a4e90a64ed7d74 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.908+0000 7f762fe00640 1 -- 172.21.15.2:0/1071907899 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618030610 con 0x7f76301593d0 2024-01-15T17:57:38.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.908+0000 7f762fe00640 1 -- 172.21.15.2:0/1071907899 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618030770 con 0x7f76301593d0 2024-01-15T17:57:38.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.908+0000 7f762fe00640 1 -- 172.21.15.2:0/1071907899 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618030ae0 con 0x7f76301593d0 2024-01-15T17:57:38.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.909+0000 7f7638200640 1 -- 172.21.15.2:0/1071907899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 msgr2=0x7f76300cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.909+0000 7f7638200640 1 --2- 172.21.15.2:0/1071907899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f76300cd090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7630072aa0 tx=0x7f761802b960 comp rx=0 tx=0).stop 2024-01-15T17:57:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.909+0000 7f7638200640 1 -- 172.21.15.2:0/1071907899 shutdown_connections 2024-01-15T17:57:38.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.910+0000 7f7638200640 1 --2- 172.21.15.2:0/1071907899 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f76300cd090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.910+0000 7f7638200640 1 --2- 172.21.15.2:0/1071907899 >> v2:172.21.15.137:3300/0 conn(0x7f7630071b60 0x7f76300ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.910+0000 7f7638200640 1 --2- 172.21.15.2:0/1071907899 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76300b1360 0x7f76300cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.910+0000 7f7638200640 1 -- 172.21.15.2:0/1071907899 >> 172.21.15.2:0/1071907899 conn(0x7f76300b0840 msgr2=0x7f76300b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.910+0000 7f7638200640 1 -- 172.21.15.2:0/1071907899 shutdown_connections 2024-01-15T17:57:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.911+0000 7f1e21c00640 1 Processor -- start 2024-01-15T17:57:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.911+0000 7f7638200640 1 -- 172.21.15.2:0/1071907899 wait complete. 2024-01-15T17:57:38.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.912+0000 7f1e21c00640 1 -- start start 2024-01-15T17:57:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.914+0000 7f4b9c200640 1 Processor -- start 2024-01-15T17:57:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f4b9c200640 1 -- start start 2024-01-15T17:57:38.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f1e21c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c1593d0 0x7f1e1c0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f7638200640 1 Processor -- start 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f1e21c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1e1c06dd80 0x7f1e1c06c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f1e21c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e1c0b3990 con 0x7f1e1c071b60 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f1e21c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e1c0b3e20 con 0x7f1e1c06dd80 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.915+0000 7f1e21c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0ce810 con 0x7f1e1c1593d0 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1e1c06dd80 0x7f1e1c06c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1e1c06dd80 0x7f1e1c06c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49646/0 (socket says 172.21.15.2:49646) 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 -- 172.21.15.2:0/182372441 learned_addr learned my addr 172.21.15.2:0/182372441 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f7638200640 1 -- start start 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 -- 172.21.15.2:0/182372441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c1593d0 msgr2=0x7f1e1c0cdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 --2- 172.21.15.2:0/182372441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c1593d0 0x7f1e1c0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 -- 172.21.15.2:0/182372441 >> v1:172.21.15.137:6789/0 conn(0x7f1e1c071b60 legacy=0x7f1e1c06d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f1e1b400640 1 -- 172.21.15.2:0/182372441 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c0b3340 con 0x7f1e1c06dd80 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f4b9c200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b9415d540 0x7f4b940d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f4b9c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b940b5350 0x7f4b940b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f4b9c200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b940d1ba0 con 0x7f4b940d11e0 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f4b9c200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b940d1e80 con 0x7f4b9415d540 2024-01-15T17:57:38.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.916+0000 7f4b9c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940d2160 con 0x7f4b940b5350 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.917+0000 7f4b9a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b940b5350 0x7f4b940b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.917+0000 7f4b98e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4b940d11e0 0x7f4b940b4c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35402/0 (socket says 172.21.15.2:35402) 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.917+0000 7f4b98e00640 1 -- 172.21.15.2:0/1982770890 learned_addr learned my addr 172.21.15.2:0/1982770890 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f4b9a200640 1 -- 172.21.15.2:0/1982770890 >> v2:172.21.15.137:3300/0 conn(0x7f4b9415d540 msgr2=0x7f4b940d2c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f4b9a200640 1 --2- 172.21.15.2:0/1982770890 >> v2:172.21.15.137:3300/0 conn(0x7f4b9415d540 0x7f4b940d2c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f4b9a200640 1 -- 172.21.15.2:0/1982770890 >> v1:172.21.15.2:6789/0 conn(0x7f4b940d11e0 legacy=0x7f4b940b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f4b9a200640 1 -- 172.21.15.2:0/1982770890 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b940d2440 con 0x7f4b940b5350 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f7638200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7630071b60 0x7f763015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f7638200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76300b1360 0x7f76300cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f7638200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f763014b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7638200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763015b050 con 0x7f76301593d0 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7638200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76300cbb50 con 0x7f76300b1360 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7638200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76300cbe30 con 0x7f7630071b60 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7636200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f763014b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7636200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f763014b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49648/0 (socket says 172.21.15.2:49648) 2024-01-15T17:57:38.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7636200640 1 -- 172.21.15.2:0/4028862914 learned_addr learned my addr 172.21.15.2:0/4028862914 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.918+0000 7f1e1b400640 1 --2- 172.21.15.2:0/182372441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c06c170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e040090e0 tx=0x7f1e0402db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13aaeb8225ebef67 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f1e19600640 1 -- 172.21.15.2:0/182372441 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0402f8c0 con 0x7f1e1c06dd80 2024-01-15T17:57:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f1e19600640 1 -- 172.21.15.2:0/182372441 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0402fa20 con 0x7f1e1c06dd80 2024-01-15T17:57:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f1e19600640 1 -- 172.21.15.2:0/182372441 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0402fdb0 con 0x7f1e1c06dd80 2024-01-15T17:57:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.919+0000 7f7635800640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7630071b60 0x7f763015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7634e00640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76300b1360 0x7f76300cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 -- 172.21.15.2:0/182372441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c06dd80 msgr2=0x7f1e1c06c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7636200640 1 -- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7630071b60 msgr2=0x7f763015b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7636200640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7630071b60 0x7f763015b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7636200640 1 -- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76300b1360 msgr2=0x7f76300cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7636200640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76300b1360 0x7f76300cb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7636200640 1 -- 172.21.15.2:0/4028862914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76180094d0 con 0x7f76301593d0 2024-01-15T17:57:38.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7634e00640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76300b1360 0x7f76300cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:38.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 --2- 172.21.15.2:0/182372441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c06c170 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1e040090e0 tx=0x7f1e0402db00 comp rx=0 tx=0).stop 2024-01-15T17:57:38.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 -- 172.21.15.2:0/182372441 shutdown_connections 2024-01-15T17:57:38.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 --2- 172.21.15.2:0/182372441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c06c170 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 --2- 172.21.15.2:0/182372441 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c1593d0 0x7f1e1c0cdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 -- 172.21.15.2:0/182372441 >> 172.21.15.2:0/182372441 conn(0x7f1e1c0b0840 msgr2=0x7f1e1c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f7636200640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f763014b5f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76180090b0 tx=0x7f76180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.920+0000 7f1e21c00640 1 -- 172.21.15.2:0/182372441 shutdown_connections 2024-01-15T17:57:38.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.921+0000 7f1e21c00640 1 -- 172.21.15.2:0/182372441 wait complete. 2024-01-15T17:57:38.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.921+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618003610 con 0x7f76301593d0 2024-01-15T17:57:38.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.921+0000 7f4b9a200640 1 --2- 172.21.15.2:0/1982770890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b940b5350 0x7f4b940b6180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009d90 tx=0x7f4b8c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee8222c3896bddcc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.921+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618003770 con 0x7f76301593d0 2024-01-15T17:57:38.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.921+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618003b00 con 0x7f76301593d0 2024-01-15T17:57:38.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.921+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763015aa00 con 0x7f76301593d0 2024-01-15T17:57:38.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.922+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7630142680 con 0x7f76301593d0 2024-01-15T17:57:38.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.922+0000 7f4b93e00640 1 -- 172.21.15.2:0/1982770890 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c02cd10 con 0x7f4b940b5350 2024-01-15T17:57:38.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.922+0000 7f4b93e00640 1 -- 172.21.15.2:0/1982770890 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8c02ce70 con 0x7f4b940b5350 2024-01-15T17:57:38.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.922+0000 7f4b93e00640 1 -- 172.21.15.2:0/1982770890 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8c034640 con 0x7f4b940b5350 2024-01-15T17:57:38.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76300b2ff0 con 0x7f76301593d0 2024-01-15T17:57:38.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f39bf400640 1 Processor -- start 2024-01-15T17:57:38.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f1e21c00640 1 Processor -- start 2024-01-15T17:57:38.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f1e21c00640 1 -- start start 2024-01-15T17:57:38.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f1e21c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c1f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f1e21c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c071b60 0x7f1e1c1531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f1e21c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c1593d0 0x7f1e1c1536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f39bf400640 1 -- start start 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f39bf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f39b815d540 0x7f39b80d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f39bf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f39bf400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b80d1ba0 con 0x7f39b80d1640 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f39bf400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39b80d1e80 con 0x7f39b815d540 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f39bf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80d2160 con 0x7f39b80d11e0 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f39bca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f39b815d540 0x7f39b80d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f39bca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f39b815d540 0x7f39b80d2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49672/0 (socket says 172.21.15.2:49672) 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f39bca00640 1 -- 172.21.15.2:0/4255587429 learned_addr learned my addr 172.21.15.2:0/4255587429 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f1e21c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0ae2e0 con 0x7f1e1c1593d0 2024-01-15T17:57:38.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f39b7e00640 1 --2- 172.21.15.2:0/4255587429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f1e1aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c1f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f1e1aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c1f6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43324/0 (socket says 172.21.15.2:43324) 2024-01-15T17:57:38.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f1e1aa00640 1 -- 172.21.15.2:0/1138357554 learned_addr learned my addr 172.21.15.2:0/1138357554 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7618003c60 con 0x7f76301593d0 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f1e1b400640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c1593d0 0x7f1e1c1536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f4b9c200640 1 -- 172.21.15.2:0/1982770890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b940b5350 msgr2=0x7f4b940b6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f4b9c200640 1 --2- 172.21.15.2:0/1982770890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b940b5350 0x7f4b940b6180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c009d90 tx=0x7f4b8c02bf60 comp rx=0 tx=0).stop 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f4b9c200640 1 -- 172.21.15.2:0/1982770890 shutdown_connections 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f4b9c200640 1 --2- 172.21.15.2:0/1982770890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b940b5350 0x7f4b940b6180 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f4b9c200640 1 --2- 172.21.15.2:0/1982770890 >> v2:172.21.15.137:3300/0 conn(0x7f4b9415d540 0x7f4b940d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f4b9c200640 1 -- 172.21.15.2:0/1982770890 >> 172.21.15.2:0/1982770890 conn(0x7f4b94072210 msgr2=0x7f4b940cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f4b9c200640 1 -- 172.21.15.2:0/1982770890 shutdown_connections 2024-01-15T17:57:38.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f4b9c200640 1 -- 172.21.15.2:0/1982770890 wait complete. 2024-01-15T17:57:38.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f39b7e00640 1 -- 172.21.15.2:0/4255587429 >> v2:172.21.15.2:3300/0 conn(0x7f39b815d540 msgr2=0x7f39b80d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f4b9c200640 1 Processor -- start 2024-01-15T17:57:38.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f4b9c200640 1 -- start start 2024-01-15T17:57:38.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.923+0000 7f9e87e00640 1 Processor -- start 2024-01-15T17:57:38.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.924+0000 7f9e87e00640 1 -- start start 2024-01-15T17:57:38.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.925+0000 7f9e87e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9e800e1270 0x7f9e8003a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.925+0000 7f9e87e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9e8003a680 0x7f9e80055cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f9e87e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e80055840 con 0x7f9e80051620 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f9e87e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e800527d0 con 0x7f9e8003a680 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.926+0000 7f9e87e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e80054c30 con 0x7f9e800e1270 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f9e87400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9e80051620 0x7f9e80056210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35420/0 (socket says 172.21.15.2:35420) 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.927+0000 7f9e87400640 1 -- 172.21.15.2:0/654306756 learned_addr learned my addr 172.21.15.2:0/654306756 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f9e86a00640 1 --2- 172.21.15.2:0/654306756 >> v2:172.21.15.137:3300/0 conn(0x7f9e800e1270 0x7f9e8003a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c1f6d60 con 0x7f1e1c06dd80 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c0ad6d0 con 0x7f1e1c071b60 2024-01-15T17:57:38.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f1e1a000640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c071b60 0x7f1e1c1531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f9e86000640 1 --2- 172.21.15.2:0/654306756 >> v2:172.21.15.2:3300/0 conn(0x7f9e8003a680 0x7f9e80055cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f4b9c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b940b5350 0x7f4b94156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f4b9c200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b940d11e0 0x7f4b940c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f4b9c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9415d540 0x7f4b940c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f1e1b400640 1 -- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c071b60 msgr2=0x7f1e1c1531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f1e1b400640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e1c071b60 0x7f1e1c1531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f1e1b400640 1 -- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e1c06dd80 msgr2=0x7f1e1c1f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f39b7e00640 1 --2- 172.21.15.2:0/4255587429 >> v2:172.21.15.2:3300/0 conn(0x7f39b815d540 0x7f39b80d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f1e1b400640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e1c06dd80 0x7f1e1c1f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f1e1b400640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04009d60 con 0x7f1e1c1593d0 2024-01-15T17:57:38.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f4b9c200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940c84b0 con 0x7f4b940d11e0 2024-01-15T17:57:38.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f4b9c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940c7830 con 0x7f4b940b5350 2024-01-15T17:57:38.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.929+0000 7f4b9c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b940c7b10 con 0x7f4b9415d540 2024-01-15T17:57:38.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f39b7e00640 1 -- 172.21.15.2:0/4255587429 >> v1:172.21.15.2:6789/0 conn(0x7f39b80d1640 legacy=0x7f39b806b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f762d600640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760806b390 0x7f760806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b99800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b940b5350 0x7f4b94156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.928+0000 7f39b7e00640 1 -- 172.21.15.2:0/4255587429 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b80d2440 con 0x7f39b80d11e0 2024-01-15T17:57:38.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b99800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b940b5350 0x7f4b94156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43326/0 (socket says 172.21.15.2:43326) 2024-01-15T17:57:38.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b99800640 1 -- 172.21.15.2:0/904162754 learned_addr learned my addr 172.21.15.2:0/904162754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f39b7e00640 1 --2- 172.21.15.2:0/4255587429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b4c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f39a80097a0 tx=0x7f39a802b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=694e4ad180ee969 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76180abba0 con 0x7f76301593d0 2024-01-15T17:57:38.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f762d600640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f76080735b0 0x7f76080759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 --> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7608076080 con 0x7f76080735b0 2024-01-15T17:57:38.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76180b0240 con 0x7f76301593d0 2024-01-15T17:57:38.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f1e1b400640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c1593d0 0x7f1e1c1536e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c06ecc0 tx=0x7f1e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f7634e00640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f76080735b0 0x7f76080759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f7635800640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760806b390 0x7f760806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f39b7400640 1 -- 172.21.15.2:0/4255587429 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8030610 con 0x7f39b80d11e0 2024-01-15T17:57:38.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f39b7400640 1 -- 172.21.15.2:0/4255587429 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8030770 con 0x7f39b80d11e0 2024-01-15T17:57:38.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f39b7400640 1 -- 172.21.15.2:0/4255587429 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a8030b40 con 0x7f39b80d11e0 2024-01-15T17:57:38.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04037be0 con 0x7f1e1c1593d0 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f9e85600640 1 -- 172.21.15.2:0/654306756 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267078804 0 0) 0x7f9e80055840 con 0x7f9e80051620 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f9e85600640 1 -- 172.21.15.2:0/654306756 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e68007930 con 0x7f9e80051620 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f9e85600640 1 -- 172.21.15.2:0/654306756 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2270078543 0 0) 0x7f9e68007930 con 0x7f9e80051620 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f9e86000640 1 -- 172.21.15.2:0/654306756 >> v2:172.21.15.137:3300/0 conn(0x7f9e800e1270 msgr2=0x7f9e8003a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f9e86000640 1 --2- 172.21.15.2:0/654306756 >> v2:172.21.15.137:3300/0 conn(0x7f9e800e1270 0x7f9e8003a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f9e86000640 1 -- 172.21.15.2:0/654306756 >> v1:172.21.15.2:6789/0 conn(0x7f9e80051620 legacy=0x7f9e80056210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f9e86000640 1 -- 172.21.15.2:0/654306756 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e80054f10 con 0x7f9e8003a680 2024-01-15T17:57:38.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0402fe90 con 0x7f1e1c1593d0 2024-01-15T17:57:38.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0402f870 con 0x7f1e1c1593d0 2024-01-15T17:57:38.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f39bf400640 1 -- 172.21.15.2:0/4255587429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 msgr2=0x7f39b80b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f39bf400640 1 --2- 172.21.15.2:0/4255587429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b4c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f39a80097a0 tx=0x7f39a802b960 comp rx=0 tx=0).stop 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b98e00640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b940d11e0 0x7f4b940c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b9a200640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9415d540 0x7f4b940c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b98e00640 1 -- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9415d540 msgr2=0x7f4b940c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b98e00640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9415d540 0x7f4b940c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b98e00640 1 -- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b940b5350 msgr2=0x7f4b94156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b98e00640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b940b5350 0x7f4b94156ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.930+0000 7f4b98e00640 1 -- 172.21.15.2:0/904162754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c009a70 con 0x7f4b940d11e0 2024-01-15T17:57:38.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.931+0000 7f4b98e00640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b940d11e0 0x7f4b940c88d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00c410 tx=0x7f4b7c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f39bf400640 1 -- 172.21.15.2:0/4255587429 shutdown_connections 2024-01-15T17:57:38.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f39bf400640 1 --2- 172.21.15.2:0/4255587429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b4c40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f39bf400640 1 --2- 172.21.15.2:0/4255587429 >> v2:172.21.15.2:3300/0 conn(0x7f39b815d540 0x7f39b80d2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f39bf400640 1 -- 172.21.15.2:0/4255587429 >> 172.21.15.2:0/4255587429 conn(0x7f39b8072210 msgr2=0x7f39b80cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c005ce0 con 0x7f4b940d11e0 2024-01-15T17:57:38.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c005e40 con 0x7f4b940d11e0 2024-01-15T17:57:38.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0072d0 con 0x7f4b940d11e0 2024-01-15T17:57:38.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f7634e00640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f76080735b0 0x7f76080759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.932+0000 7f7635800640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760806b390 0x7f760806d850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7628009f60 tx=0x7f76280094f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f9e86a00640 1 --2- 172.21.15.2:0/654306756 >> v2:172.21.15.137:3300/0 conn(0x7f9e800e1270 0x7f9e8003a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:38.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c0ad9b0 con 0x7f1e1c1593d0 2024-01-15T17:57:38.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b940c7df0 con 0x7f4b940d11e0 2024-01-15T17:57:38.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b94157240 con 0x7f4b940d11e0 2024-01-15T17:57:38.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.933+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== osd.0 v2:172.21.15.2:6808/2449281684 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7608076080 con 0x7f76080735b0 2024-01-15T17:57:38.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7ff20e400640 1 Processor -- start 2024-01-15T17:57:38.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7f39bf400640 1 -- 172.21.15.2:0/4255587429 shutdown_connections 2024-01-15T17:57:38.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7f9e86000640 1 --2- 172.21.15.2:0/654306756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e8003a680 0x7f9e80055cd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9e80003ea0 tx=0x7f9e70033fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d060dd3acd388a4a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c0adf70 con 0x7f1e1c1593d0 2024-01-15T17:57:38.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7f7db4c00640 1 Processor -- start 2024-01-15T17:57:38.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7f9e85600640 1 -- 172.21.15.2:0/654306756 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8804ed60 con 0x7f9e8003a680 2024-01-15T17:57:38.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.935+0000 7f9e85600640 1 -- 172.21.15.2:0/654306756 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e70034b00 con 0x7f9e8003a680 2024-01-15T17:57:38.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.935+0000 7ff20e400640 1 -- start start 2024-01-15T17:57:38.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.935+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1e1c0b2ff0 con 0x7f1e1c1593d0 2024-01-15T17:57:38.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.935+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b9415c9b0 con 0x7f4b940d11e0 2024-01-15T17:57:38.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.934+0000 7f39bf400640 1 -- 172.21.15.2:0/4255587429 wait complete. 2024-01-15T17:57:38.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.936+0000 7f7db4c00640 1 -- start start 2024-01-15T17:57:38.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.936+0000 7f9e87e00640 1 -- 172.21.15.2:0/654306756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e8003a680 msgr2=0x7f9e80055cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.936+0000 7f9e85600640 1 -- 172.21.15.2:0/654306756 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e8804ed60 con 0x7f9e8003a680 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.936+0000 7f9e87e00640 1 --2- 172.21.15.2:0/654306756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e8003a680 0x7f9e80055cd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9e80003ea0 tx=0x7f9e70033fd0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.936+0000 7f9e87e00640 1 -- 172.21.15.2:0/654306756 shutdown_connections 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7f9e87e00640 1 --2- 172.21.15.2:0/654306756 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e8003a680 0x7f9e80055cd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7f9e87e00640 1 --2- 172.21.15.2:0/654306756 >> v2:172.21.15.137:3300/0 conn(0x7f9e800e1270 0x7f9e8003a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7f9e87e00640 1 -- 172.21.15.2:0/654306756 >> 172.21.15.2:0/654306756 conn(0x7f9e80056ca0 msgr2=0x7f9e800570b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.936+0000 7ff20e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815d540 0x7ff2080d2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7f39bf400640 1 Processor -- start 2024-01-15T17:57:38.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7ff20e400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080d1ba0 con 0x7ff2080d11e0 2024-01-15T17:57:38.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7ff20e400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2080d1e80 con 0x7ff2080b5350 2024-01-15T17:57:38.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7f39bf400640 1 -- start start 2024-01-15T17:57:38.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7ff20e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080d2160 con 0x7ff20815d540 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7ff206a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7ff2080d11e0 0x7ff2080b4c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35430/0 (socket says 172.21.15.2:35430) 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7ff206a00640 1 -- 172.21.15.2:0/3835283397 learned_addr learned my addr 172.21.15.2:0/3835283397 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7ff207400640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815d540 0x7ff2080d2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7fcb19e00640 1 Processor -- start 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7db4c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7db01615d0 0x7f7db00ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7db4c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db0165350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7db4c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db00b3010 con 0x7f7db00b18c0 2024-01-15T17:57:38.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7db4c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db00b0de0 con 0x7f7db01615d0 2024-01-15T17:57:38.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7db4c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00b10c0 con 0x7f7db006cfc0 2024-01-15T17:57:38.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745678656 0 0) 0x7ff2080d1ba0 con 0x7ff2080d11e0 2024-01-15T17:57:38.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e8003520 con 0x7ff2080d11e0 2024-01-15T17:57:38.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7dad400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7db00b18c0 0x7f7db006c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:39844/0 (socket says 172.21.15.2:39844) 2024-01-15T17:57:38.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7dad400640 1 -- 172.21.15.2:0/3587239074 learned_addr learned my addr 172.21.15.2:0/3587239074 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:38.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3252427752 0 0) 0x7ff1e8003520 con 0x7ff2080d11e0 2024-01-15T17:57:38.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815d540 msgr2=0x7ff2080d2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815d540 0x7ff2080d2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 >> v1:172.21.15.137:6789/0 conn(0x7ff2080b5350 legacy=0x7ff2080b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080d2440 con 0x7ff2080d11e0 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7ff1f0001670 con 0x7ff2080d11e0 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 >> v1:172.21.15.2:6789/0 conn(0x7ff2080d11e0 legacy=0x7ff2080b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 0x7ff1e8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1e80047c0 0x7ff1e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1e8005180 0x7ff1e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080d2160 con 0x7ff1e8005180 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e800bf80 con 0x7ff1e80039a0 2024-01-15T17:57:38.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e800c140 con 0x7ff1e80047c0 2024-01-15T17:57:38.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.937+0000 7f9e87e00640 1 -- 172.21.15.2:0/654306756 shutdown_connections 2024-01-15T17:57:38.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7f9e87e00640 1 -- 172.21.15.2:0/654306756 wait complete. 2024-01-15T17:57:38.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7fcb19e00640 1 -- start start 2024-01-15T17:57:38.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7f7dae800640 1 --2- 172.21.15.2:0/3587239074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db0165350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.938+0000 7f7dade00640 1 --2- 172.21.15.2:0/3587239074 >> v2:172.21.15.2:3300/0 conn(0x7f7db01615d0 0x7f7db00ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7f7daca00640 1 -- 172.21.15.2:0/3587239074 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758216748 0 0) 0x7f7db00b3010 con 0x7f7db00b18c0 2024-01-15T17:57:38.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7f7daca00640 1 -- 172.21.15.2:0/3587239074 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d90003520 con 0x7f7db00b18c0 2024-01-15T17:57:38.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1e04041400 con 0x7f1e1c1593d0 2024-01-15T17:57:38.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7ff207400640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 0x7ff1e8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7ff206a00640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1e80047c0 0x7ff1e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f39bf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f7dade00640 1 -- 172.21.15.2:0/3587239074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 msgr2=0x7f7db0165350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f7dade00640 1 --2- 172.21.15.2:0/3587239074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db0165350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f39bf400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39b80d1640 0x7f39b815b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7fcb19e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcb140d0cc0 0x7fcb140d1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f39bf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b815d540 0x7f39b81e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7ff207400640 1 -- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1e80047c0 msgr2=0x7ff1e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.939+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4b7c0074f0 con 0x7f4b940d11e0 2024-01-15T17:57:38.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f39bf400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80ca3c0 con 0x7f39b815d540 2024-01-15T17:57:38.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7fcb19e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb140b3e60 0x7fcb140b5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7fcb19e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb140d05b0 con 0x7fcb14161920 2024-01-15T17:57:38.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7fcb19e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb140d1230 con 0x7fcb140b3e60 2024-01-15T17:57:38.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7fcb19e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb140cfad0 con 0x7fcb140d0cc0 2024-01-15T17:57:38.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f7dade00640 1 -- 172.21.15.2:0/3587239074 >> v1:172.21.15.137:6789/0 conn(0x7f7db00b18c0 legacy=0x7f7db006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.940+0000 7f7dade00640 1 -- 172.21.15.2:0/3587239074 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00b13a0 con 0x7f7db01615d0 2024-01-15T17:57:38.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f4b91600640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6c06b310 0x7f4b6c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f39bf400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80ca6a0 con 0x7f39b80d1640 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7fcb0be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcb140d0cc0 0x7fcb140d1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7fcb0be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcb140d0cc0 0x7fcb140d1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43358/0 (socket says 172.21.15.2:43358) 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f39bf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b80ca980 con 0x7f39b80d11e0 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b7c089a30 con 0x7f4b940d11e0 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f4b91600640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f4b6c073530 0x7f4b6c075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 --> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b6c076000 con 0x7f4b6c073530 2024-01-15T17:57:38.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b7c089ed0 con 0x7f4b940d11e0 2024-01-15T17:57:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7fcb0be00640 1 -- 172.21.15.2:0/826971365 learned_addr learned my addr 172.21.15.2:0/826971365 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7ff207400640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1e80047c0 0x7ff1e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7ff207400640 1 -- 172.21.15.2:0/3835283397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1e8005180 msgr2=0x7ff1e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7ff207400640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff1e8005180 0x7ff1e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7ff207400640 1 -- 172.21.15.2:0/3835283397 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080d2440 con 0x7ff1e80039a0 2024-01-15T17:57:38.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f1e12a00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1df406b2c0 0x7f1df406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e04033070 con 0x7f1e1c1593d0 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f1e12a00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7f1df40734e0 0x7f1df4075900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1df4075fb0 con 0x7f1df40734e0 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1e040acda0 con 0x7f1e1c1593d0 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.941+0000 7f1e1aa00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1df406b2c0 0x7f1df406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f4b9a200640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f4b6c073530 0x7f4b6c075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f4b99800640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6c06b310 0x7f4b6c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f1e1a000640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7f1df40734e0 0x7f1df4075900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f39bca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f39b7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39b80d1640 0x7f39b815b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f7dade00640 1 --2- 172.21.15.2:0/3587239074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00ce110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea8fb81959010ef server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f39bca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58594/0 (socket says 172.21.15.2:58594) 2024-01-15T17:57:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7f39bca00640 1 -- 172.21.15.2:0/2950289308 learned_addr learned my addr 172.21.15.2:0/2950289308 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7f39bd400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b815d540 0x7f39b81e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.942+0000 7ff207400640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 0x7ff1e8003da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff208072b60 tx=0x7ff1e80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cd5e2bdc8dd3157 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7f4b9a200640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f4b6c073530 0x7f4b6c075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7fcb19400640 1 --2- 172.21.15.2:0/826971365 >> v2:172.21.15.2:3300/0 conn(0x7fcb140b3e60 0x7fcb140b5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20002cba0 con 0x7ff1e80039a0 2024-01-15T17:57:38.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7f7daca00640 1 -- 172.21.15.2:0/3587239074 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98031830 con 0x7f7db01615d0 2024-01-15T17:57:38.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f9e87e00640 1 Processor -- start 2024-01-15T17:57:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7f7daca00640 1 -- 172.21.15.2:0/3587239074 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d98031990 con 0x7f7db01615d0 2024-01-15T17:57:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7fcb13e00640 1 -- 172.21.15.2:0/826971365 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563179505 0 0) 0x7fcb140d05b0 con 0x7fcb14161920 2024-01-15T17:57:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7fcb13e00640 1 -- 172.21.15.2:0/826971365 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf4003520 con 0x7fcb14161920 2024-01-15T17:57:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f7daca00640 1 -- 172.21.15.2:0/3587239074 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d98031d00 con 0x7f7db01615d0 2024-01-15T17:57:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f39b7e00640 1 -- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 msgr2=0x7f39b80b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f39b7e00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f39b80d11e0 0x7f39b80b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f39b7e00640 1 -- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b815d540 msgr2=0x7f39b81e5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f39b7e00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b815d540 0x7f39b81e5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f39b7e00640 1 -- 172.21.15.2:0/2950289308 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8009400 con 0x7f39b80d1640 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f39bd400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f39b815d540 0x7f39b81e5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.943+0000 7f1e1a000640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7f1df40734e0 0x7f1df4075900 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 -- 172.21.15.2:0/3587239074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 msgr2=0x7f7db00ce110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 --2- 172.21.15.2:0/3587239074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00ce110 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d980097a0 tx=0x7f7d9802b8d0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000f040 con 0x7ff1e80039a0 2024-01-15T17:57:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7ff206000640 1 -- 172.21.15.2:0/3835283397 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff200034440 con 0x7ff1e80039a0 2024-01-15T17:57:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== osd.1 v2:172.21.15.2:6800/3287240406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4b6c076000 con 0x7f4b6c073530 2024-01-15T17:57:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f4b99800640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6c06b310 0x7f4b6c06d7d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b840099a0 tx=0x7f4b840090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== osd.2 v2:172.21.15.2:6816/3553325871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1df4075fb0 con 0x7f1df40734e0 2024-01-15T17:57:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f39b7e00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39b80d1640 0x7f39b815b3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39a8033000 tx=0x7f39a8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f9e87e00640 1 -- start start 2024-01-15T17:57:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7fcb0be00640 1 -- 172.21.15.2:0/826971365 >> v2:172.21.15.2:3300/0 conn(0x7fcb140b3e60 msgr2=0x7fcb140b5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7fcb0be00640 1 --2- 172.21.15.2:0/826971365 >> v2:172.21.15.2:3300/0 conn(0x7fcb140b3e60 0x7fcb140b5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7fcb0be00640 1 -- 172.21.15.2:0/826971365 >> v1:172.21.15.2:6789/0 conn(0x7fcb14161920 legacy=0x7fcb140d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.944+0000 7fcb0be00640 1 -- 172.21.15.2:0/826971365 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb140cfdb0 con 0x7fcb140d0cc0 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 -- 172.21.15.2:0/3587239074 shutdown_connections 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 --2- 172.21.15.2:0/3587239074 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db0165350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 --2- 172.21.15.2:0/3587239074 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00ce110 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 -- 172.21.15.2:0/3587239074 >> 172.21.15.2:0/3587239074 conn(0x7f7db00b46d0 msgr2=0x7f7db00b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 -- 172.21.15.2:0/3587239074 shutdown_connections 2024-01-15T17:57:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7f7db4c00640 1 -- 172.21.15.2:0/3587239074 wait complete. 2024-01-15T17:57:38.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7ff20e400640 1 -- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 msgr2=0x7ff1e8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.945+0000 7ff20e400640 1 --2- 172.21.15.2:0/3835283397 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 0x7ff1e8003da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff208072b60 tx=0x7ff1e80057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7f1e1aa00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1df406b2c0 0x7f1df406d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c15c720 tx=0x7f1e1400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7ff207e00640 1 -- 172.21.15.2:0/3835283397 reap_dead start 2024-01-15T17:57:38.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7ff20e400640 1 -- 172.21.15.2:0/3835283397 shutdown_connections 2024-01-15T17:57:38.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7ff20e400640 1 -- 172.21.15.2:0/3835283397 >> 172.21.15.2:0/3835283397 conn(0x7ff208072210 msgr2=0x7ff2080cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7f9e87e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e8003a680 0x7f9e8018a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7f9e87e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e80051620 0x7f9e80008290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7ff20e400640 1 -- 172.21.15.2:0/3835283397 shutdown_connections 2024-01-15T17:57:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a803e040 con 0x7f39b80d1640 2024-01-15T17:57:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a8039bb0 con 0x7f39b80d1640 2024-01-15T17:57:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7fcb0be00640 1 --2- 172.21.15.2:0/826971365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140d0cc0 0x7fcb140d1c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcb140b48e0 tx=0x7fcb0402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e54e2e654a63cf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e86a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e8003a680 0x7f9e8018a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e86a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e8003a680 0x7f9e8018a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43372/0 (socket says 172.21.15.2:43372) 2024-01-15T17:57:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.946+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a80443f0 con 0x7f39b80d1640 2024-01-15T17:57:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b80cac60 con 0x7f39b80d1640 2024-01-15T17:57:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b814ddb0 con 0x7f39b80d1640 2024-01-15T17:57:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7fcb13e00640 1 -- 172.21.15.2:0/826971365 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04031870 con 0x7fcb140d0cc0 2024-01-15T17:57:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7fcb13e00640 1 -- 172.21.15.2:0/826971365 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb040319d0 con 0x7fcb140d0cc0 2024-01-15T17:57:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7fcb13e00640 1 -- 172.21.15.2:0/826971365 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04031d60 con 0x7fcb140d0cc0 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e86a00640 1 -- 172.21.15.2:0/1766798626 learned_addr learned my addr 172.21.15.2:0/1766798626 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e87e00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e800e1270 0x7f9e800d53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800d7390 con 0x7f9e80051620 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800d75a0 con 0x7f9e8003a680 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.947+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e800d7840 con 0x7f9e800e1270 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86000640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e80051620 0x7f9e80008290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f7db4c00640 1 Processor -- start 2024-01-15T17:57:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f7db4c00640 1 -- start start 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86000640 1 -- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e800e1270 msgr2=0x7f9e800d53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e87400640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e800e1270 0x7f9e800d53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86000640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9e800e1270 0x7f9e800d53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86000640 1 -- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e8003a680 msgr2=0x7f9e8018a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86000640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e8003a680 0x7f9e8018a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86000640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e78009400 con 0x7f9e80051620 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7f9e86a00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9e8003a680 0x7f9e8018a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.949+0000 7f9e86000640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e80051620 0x7f9e80008290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e7003b000 tx=0x7f9e700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.949+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e70003ea0 con 0x7f9e80051620 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.949+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e70014680 con 0x7f9e80051620 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.949+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7003c640 con 0x7f9e80051620 2024-01-15T17:57:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.949+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e70009400 con 0x7f9e80051620 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.949+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e800d7e00 con 0x7f9e80051620 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7db4c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db01f6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7db4c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7db00b18c0 0x7f7db015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7db4c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00c4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7db4c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db01f7290 con 0x7f7db01615d0 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7db4c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db015eb90 con 0x7f7db00b18c0 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7db4c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00c51d0 con 0x7f7db006cfc0 2024-01-15T17:57:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7fcb19e00640 1 -- 172.21.15.2:0/826971365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140d0cc0 msgr2=0x7fcb140d1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7fcb19e00640 1 --2- 172.21.15.2:0/826971365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140d0cc0 0x7fcb140d1c20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcb140b48e0 tx=0x7fcb0402b910 comp rx=0 tx=0).stop 2024-01-15T17:57:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.948+0000 7ff20e400640 1 -- 172.21.15.2:0/3835283397 wait complete. 2024-01-15T17:57:38.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7dade00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db01f6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7dade00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db01f6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58614/0 (socket says 172.21.15.2:58614) 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.950+0000 7f7dade00640 1 -- 172.21.15.2:0/4270752484 learned_addr learned my addr 172.21.15.2:0/4270752484 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.951+0000 7f7dad400640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7db00b18c0 0x7f7db015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.951+0000 7f7dae800640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00c4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.951+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f39b80686f0 con 0x7f39b80d1640 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f7dae800640 1 -- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 msgr2=0x7f7db01f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f7dae800640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7db006cfc0 0x7f7db01f6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f7dae800640 1 -- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7db00b18c0 msgr2=0x7f7db015bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f7dae800640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7db00b18c0 0x7f7db015bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f7dae800640 1 -- 172.21.15.2:0/4270752484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98009400 con 0x7f7db01615d0 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7fcb19e00640 1 -- 172.21.15.2:0/826971365 shutdown_connections 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7fcb19e00640 1 --2- 172.21.15.2:0/826971365 >> v2:172.21.15.2:3300/0 conn(0x7fcb140b3e60 0x7fcb140b5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7fcb19e00640 1 --2- 172.21.15.2:0/826971365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140d0cc0 0x7fcb140d1c20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7fcb19e00640 1 -- 172.21.15.2:0/826971365 >> 172.21.15.2:0/826971365 conn(0x7fcb1409c3c0 msgr2=0x7fcb140d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9e7003c7a0 con 0x7f9e80051620 2024-01-15T17:57:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e800512d0 con 0x7f9e80051620 2024-01-15T17:57:38.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7f7dae800640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00c4580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7db00cba60 tx=0x7f7da000d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0017070 con 0x7f7db01615d0 2024-01-15T17:57:38.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da0004020 con 0x7f7db01615d0 2024-01-15T17:57:38.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da00041b0 con 0x7f7db01615d0 2024-01-15T17:57:38.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f7db4c00640 1 -- 172.21.15.2:0/4270752484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db00c54b0 con 0x7f7db01615d0 2024-01-15T17:57:38.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f9e7ea00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9e6006b310 0x7f9e6006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f7db4c00640 1 -- 172.21.15.2:0/4270752484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db015f0c0 con 0x7f7db01615d0 2024-01-15T17:57:38.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9e700ac540 con 0x7f9e80051620 2024-01-15T17:57:38.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7f9e7ea00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f9e60073530 0x7f9e60075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7f9e86a00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9e6006b310 0x7f9e6006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7ff20e400640 1 Processor -- start 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.953+0000 7ff20e400640 1 -- start start 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7ff20e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 0x7ff2080cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7ff20e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff208160270 0x7ff2081528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7ff20e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815e1a0 0x7ff20815e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7ff20e400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20815eb60 con 0x7ff208160270 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7ff20e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2081531c0 con 0x7ff1e80039a0 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7ff20e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2080ccb30 con 0x7ff20815e1a0 2024-01-15T17:57:38.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e60076000 con 0x7f9e60073530 2024-01-15T17:57:38.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e700aeb20 con 0x7f9e80051620 2024-01-15T17:57:38.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7ff206a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff208160270 0x7ff2081528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7fcb19e00640 1 -- 172.21.15.2:0/826971365 shutdown_connections 2024-01-15T17:57:38.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7ff206a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff208160270 0x7ff2081528e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49756/0 (socket says 172.21.15.2:49756) 2024-01-15T17:57:38.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7ff206a00640 1 -- 172.21.15.2:0/1104124749 learned_addr learned my addr 172.21.15.2:0/1104124749 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7f9e87400640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f9e60073530 0x7f9e60075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.955+0000 7fcb19e00640 1 -- 172.21.15.2:0/826971365 wait complete. 2024-01-15T17:57:38.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.956+0000 7ff206a00640 1 -- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815e1a0 msgr2=0x7ff20815e620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T17:57:38.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.952+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f39a804d3b0 con 0x7f39b80d1640 2024-01-15T17:57:38.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f39b4c00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f399006b310 0x7f399006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f39a80abfb0 con 0x7f39b80d1640 2024-01-15T17:57:38.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.956+0000 7ff206a00640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff20815e1a0 0x7ff20815e620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.956+0000 7ff206a00640 1 -- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 msgr2=0x7ff2080cc4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T17:57:38.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.956+0000 7ff206a00640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1e80039a0 0x7ff2080cc4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.956+0000 7ff206a00640 1 -- 172.21.15.2:0/1104124749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2080cce10 con 0x7ff208160270 2024-01-15T17:57:38.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.956+0000 7f9e86a00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9e6006b310 0x7f9e6006d7d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009d40 tx=0x7f9e78008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.954+0000 7f39b4c00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7f39900734b0 0x7f39900758d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.957+0000 7f9e87400640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f9e60073530 0x7f9e60075950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.957+0000 7f39bca00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f399006b310 0x7f399006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.957+0000 7ff206a00640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff208160270 0x7ff2081528e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00072e0 tx=0x7ff1f00077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.957+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7db01619b0 con 0x7f7db01615d0 2024-01-15T17:57:38.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.957+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 --> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3990075f80 con 0x7f39900734b0 2024-01-15T17:57:38.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.957+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39a8076f40 con 0x7f39b80d1640 2024-01-15T17:57:38.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7f39bd400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7f39900734b0 0x7f39900758d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7fcb19e00640 1 Processor -- start 2024-01-15T17:57:38.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f000f9c0 con 0x7ff208160270 2024-01-15T17:57:38.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f000fb20 con 0x7ff208160270 2024-01-15T17:57:38.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2080cd0f0 con 0x7ff208160270 2024-01-15T17:57:38.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f000feb0 con 0x7ff208160270 2024-01-15T17:57:38.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7fcb19e00640 1 -- start start 2024-01-15T17:57:38.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7f39bca00640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f399006b310 0x7f399006d7d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39b0011420 tx=0x7f39b000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.959+0000 7f39bd400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7f39900734b0 0x7f39900758d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:38.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.958+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2080c8320 con 0x7ff208160270 2024-01-15T17:57:38.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140b3e60 0x7fcb140cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb140d0cc0 0x7fcb140b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7fcb19e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb14161920 0x7fcb14154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== osd.4 v2:172.21.15.137:6824/3407951829 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9e60076000 con 0x7f9e60073530 2024-01-15T17:57:38.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7fcb19e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140c7830 con 0x7fcb140d0cc0 2024-01-15T17:57:38.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7fcb19e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140c7b10 con 0x7fcb140b3e60 2024-01-15T17:57:38.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7fcb19e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb140c7df0 con 0x7fcb14161920 2024-01-15T17:57:38.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.959+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2080d1640 con 0x7ff208160270 2024-01-15T17:57:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.961+0000 7fcb19400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb14161920 0x7fcb14154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.961+0000 7fcb19400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb14161920 0x7fcb14154170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58620/0 (socket says 172.21.15.2:58620) 2024-01-15T17:57:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.961+0000 7fcb19400640 1 -- 172.21.15.2:0/2687729645 learned_addr learned my addr 172.21.15.2:0/2687729645 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.961+0000 7fcb18a00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140b3e60 0x7fcb140cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.961+0000 7fcb0be00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb140d0cc0 0x7fcb140b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.960+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== osd.3 v2:172.21.15.2:6817/1263638529 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f3990075f80 con 0x7f39900734b0 2024-01-15T17:57:38.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7fcb0be00640 1 -- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb14161920 msgr2=0x7fcb14154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7fcb0be00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb14161920 0x7fcb14154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7fcb0be00640 1 -- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140b3e60 msgr2=0x7fcb140cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:38.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7fcb0be00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb140b3e60 0x7fcb140cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:38.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7fcb0be00640 1 -- 172.21.15.2:0/2687729645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb040094d0 con 0x7fcb140d0cc0 2024-01-15T17:57:38.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff1f001d460 con 0x7ff208160270 2024-01-15T17:57:38.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7fcb19400640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb14161920 0x7fcb14154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:38.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.961+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7da0004310 con 0x7f7db01615d0 2024-01-15T17:57:38.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.962+0000 7f7da6000640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d8806b310 0x7f7d8806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:38.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.963+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7da0058120 con 0x7f7db01615d0 2024-01-15T17:57:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.963+0000 7fcb0be00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb140d0cc0 0x7fcb140b7b70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb04033000 tx=0x7fcb040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.963+0000 7ff1ff400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff1e006b310 0x7ff1e006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.963+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1f008a840 con 0x7ff208160270 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb0403ac60 con 0x7fcb140d0cc0 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb0403ade0 con 0x7fcb140d0cc0 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb04044630 con 0x7fcb140d0cc0 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7ff207400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff1e006b310 0x7ff1e006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7ff1ff400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7ff1e0073530 0x7ff1e0075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 --> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff1e0076000 con 0x7ff1e0073530 2024-01-15T17:57:39.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1f008abb0 con 0x7ff208160270 2024-01-15T17:57:39.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7f7da6000640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f7d88074210 0x7f7d880766b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 --> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d88076d80 con 0x7f7d88074210 2024-01-15T17:57:39.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.964+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7da0057ef0 con 0x7f7db01615d0 2024-01-15T17:57:39.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.965+0000 7ff207e00640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7ff1e0073530 0x7ff1e0075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.965+0000 7f7dad400640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f7d88074210 0x7f7d880766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.965+0000 7ff207400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff1e006b310 0x7ff1e006d7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff200036000 tx=0x7ff20002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.966+0000 7fcb19e00640 1 -- 172.21.15.2:0/2687729645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb140c80d0 con 0x7fcb140d0cc0 2024-01-15T17:57:39.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.966+0000 7fcb19e00640 1 -- 172.21.15.2:0/2687729645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb14158820 con 0x7fcb140d0cc0 2024-01-15T17:57:39.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.966+0000 7ff207e00640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7ff1e0073530 0x7ff1e0075950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.966+0000 7f7dade00640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d8806b310 0x7f7d8806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.966+0000 7f7dad400640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f7d88074210 0x7f7d880766b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.967+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcadc000fc0 con 0x7fcb140d0cc0 2024-01-15T17:57:39.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.967+0000 7f7dade00640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d8806b310 0x7f7d8806d7d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d9802b910 tx=0x7f7d9803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.969+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== osd.7 v2:172.21.15.137:6816/463419779 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff1e0076000 con 0x7ff1e0073530 2024-01-15T17:57:39.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.969+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== osd.6 v2:172.21.15.137:6808/1848728157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7d88076d80 con 0x7f7d88074210 2024-01-15T17:57:39.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.969+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcb0404d450 con 0x7fcb140d0cc0 2024-01-15T17:57:39.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.971+0000 7fcb11600640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaec06b390 0x7fcaec06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.972+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb0402d040 con 0x7fcb140d0cc0 2024-01-15T17:57:39.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.972+0000 7fcb11600640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7fcaec0735b0 0x7fcaec0759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.972+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 --> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcaec076080 con 0x7fcaec0735b0 2024-01-15T17:57:39.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.972+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb040ac5c0 con 0x7fcb140d0cc0 2024-01-15T17:57:39.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.973+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 --> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76300ce4e0 con 0x7f76080735b0 2024-01-15T17:57:39.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.973+0000 7fcb18a00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaec06b390 0x7fcaec06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.973+0000 7fcb19400640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7fcaec0735b0 0x7fcaec0759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.973+0000 7f762d600640 1 -- 172.21.15.2:0/4028862914 <== osd.0 v2:172.21.15.2:6808/2449281684 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f762000a560 con 0x7f76080735b0 2024-01-15T17:57:39.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.974+0000 7fcb19400640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7fcaec0735b0 0x7fcaec0759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f76080735b0 msgr2=0x7f76080759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7638200640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f76080735b0 0x7f76080759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760806b390 msgr2=0x7f760806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7638200640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760806b390 0x7f760806d850 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7628009f60 tx=0x7f76280094f0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 msgr2=0x7f763014b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7fcb18a00640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaec06b390 0x7fcaec06d850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcafc000f60 tx=0x7fcafc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7638200640 1 --2- 172.21.15.2:0/4028862914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76301593d0 0x7f763014b5f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f76180090b0 tx=0x7f76180046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.975+0000 7f7636200640 1 -- 172.21.15.2:0/4028862914 reap_dead start 2024-01-15T17:57:39.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.976+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 shutdown_connections 2024-01-15T17:57:39.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.976+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 >> 172.21.15.2:0/4028862914 conn(0x7f76300b0840 msgr2=0x7f76300b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.977+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 --> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b940d1640 con 0x7f4b6c073530 2024-01-15T17:57:39.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.977+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 shutdown_connections 2024-01-15T17:57:39.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.977+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== osd.5 v2:172.21.15.137:6800/915165616 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fcaec076080 con 0x7fcaec0735b0 2024-01-15T17:57:39.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.977+0000 7f7638200640 1 -- 172.21.15.2:0/4028862914 wait complete. 2024-01-15T17:57:39.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.978+0000 7f4b91600640 1 -- 172.21.15.2:0/904162754 <== osd.1 v2:172.21.15.2:6800/3287240406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4b6c076000 con 0x7f4b6c073530 2024-01-15T17:57:39.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.978+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f4b6c073530 msgr2=0x7f4b6c075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.978+0000 7f4b9c200640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f4b6c073530 0x7f4b6c075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6c06b310 msgr2=0x7f4b6c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9c200640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6c06b310 0x7f4b6c06d7d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4b840099a0 tx=0x7f4b840090e0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b940d11e0 msgr2=0x7f4b940c88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9c200640 1 --2- 172.21.15.2:0/904162754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b940d11e0 0x7f4b940c88d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00c410 tx=0x7f4b7c001e70 comp rx=0 tx=0).stop 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9a200640 1 -- 172.21.15.2:0/904162754 reap_dead start 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 shutdown_connections 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.981+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 >> 172.21.15.2:0/904162754 conn(0x7f4b94072210 msgr2=0x7f4b940cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.982+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 shutdown_connections 2024-01-15T17:57:39.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.982+0000 7f4b9c200640 1 -- 172.21.15.2:0/904162754 wait complete. 2024-01-15T17:57:39.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.987+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 --> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1e1c0ce4e0 con 0x7f1df40734e0 2024-01-15T17:57:39.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.988+0000 7f1e12a00640 1 -- 172.21.15.2:0/1138357554 <== osd.2 v2:172.21.15.2:6816/3553325871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1df4075fb0 con 0x7f1df40734e0 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.989+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7f1df40734e0 msgr2=0x7f1df4075900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.989+0000 7f1e21c00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7f1df40734e0 0x7f1df4075900 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1df406b2c0 msgr2=0x7f1df406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e21c00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1df406b2c0 0x7f1df406d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c15c720 tx=0x7f1e1400b040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c1593d0 msgr2=0x7f1e1c1536e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e21c00640 1 --2- 172.21.15.2:0/1138357554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e1c1593d0 0x7f1e1c1536e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1e1c06ecc0 tx=0x7f1e040046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e1b400640 1 -- 172.21.15.2:0/1138357554 reap_dead start 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 shutdown_connections 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.992+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 >> 172.21.15.2:0/1138357554 conn(0x7f1e1c0b0840 msgr2=0x7f1e1c158560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.993+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 shutdown_connections 2024-01-15T17:57:39.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.993+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 --> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e800e1df0 con 0x7f9e60073530 2024-01-15T17:57:39.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.993+0000 7f1e21c00640 1 -- 172.21.15.2:0/1138357554 wait complete. 2024-01-15T17:57:39.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.993+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 --> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2080686f0 con 0x7ff1e0073530 2024-01-15T17:57:39.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.995+0000 7f9e7ea00640 1 -- 172.21.15.2:0/1766798626 <== osd.4 v2:172.21.15.137:6824/3407951829 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e800e1df0 con 0x7f9e60073530 2024-01-15T17:57:39.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.996+0000 7ff1ff400640 1 -- 172.21.15.2:0/1104124749 <== osd.7 v2:172.21.15.137:6816/463419779 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2080686f0 con 0x7ff1e0073530 2024-01-15T17:57:39.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.996+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f9e60073530 msgr2=0x7f9e60075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.996+0000 7f9e87e00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f9e60073530 0x7f9e60075950 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.995+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 --> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f39b80cb180 con 0x7f39900734b0 2024-01-15T17:57:39.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.996+0000 7f39b4c00640 1 -- 172.21.15.2:0/2950289308 <== osd.3 v2:172.21.15.2:6817/1263638529 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f39b80cb180 con 0x7f39900734b0 2024-01-15T17:57:39.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.997+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7f39900734b0 msgr2=0x7f39900758d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.997+0000 7f39bf400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7f39900734b0 0x7f39900758d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9e6006b310 msgr2=0x7f9e6006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87e00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9e6006b310 0x7f9e6006d7d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9e78009d40 tx=0x7f9e78008040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e80051620 msgr2=0x7f9e80008290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87e00640 1 --2- 172.21.15.2:0/1766798626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9e80051620 0x7f9e80008290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9e7003b000 tx=0x7f9e700046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87400640 1 -- 172.21.15.2:0/1766798626 reap_dead start 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 shutdown_connections 2024-01-15T17:57:39.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 >> 172.21.15.2:0/1766798626 conn(0x7f9e80056ca0 msgr2=0x7f9e800c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7ff1e0073530 msgr2=0x7ff1e0075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.998+0000 7ff20e400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7ff1e0073530 0x7ff1e0075950 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff1e006b310 msgr2=0x7ff1e006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff20e400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff1e006b310 0x7ff1e006d7d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff200036000 tx=0x7ff20002e000 comp rx=0 tx=0).stop 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff208160270 msgr2=0x7ff2081528e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff20e400640 1 --2- 172.21.15.2:0/1104124749 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff208160270 0x7ff2081528e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00072e0 tx=0x7ff1f00077a0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff207e00640 1 -- 172.21.15.2:0/1104124749 reap_dead start 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 shutdown_connections 2024-01-15T17:57:39.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 >> 172.21.15.2:0/1104124749 conn(0x7ff208072210 msgr2=0x7ff2080b3100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f399006b310 msgr2=0x7f399006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7f39bf400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f399006b310 0x7f399006d7d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f39b0011420 tx=0x7f39b000b040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39b80d1640 msgr2=0x7f39b815b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7f39bf400640 1 --2- 172.21.15.2:0/2950289308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f39b80d1640 0x7f39b815b3a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f39a8033000 tx=0x7f39a8004730 comp rx=0 tx=0).stop 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7f39bd400640 1 -- 172.21.15.2:0/2950289308 reap_dead start 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.000+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 shutdown_connections 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.000+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 >> 172.21.15.2:0/2950289308 conn(0x7f39b8072210 msgr2=0x7f39b80d31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.000+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 shutdown_connections 2024-01-15T17:57:39.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.000+0000 7f9e87e00640 1 -- 172.21.15.2:0/1766798626 wait complete. 2024-01-15T17:57:39.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.000+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 shutdown_connections 2024-01-15T17:57:39.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.000+0000 7f39bf400640 1 -- 172.21.15.2:0/2950289308 wait complete. 2024-01-15T17:57:39.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:38.999+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 --> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7db00686f0 con 0x7f7d88074210 2024-01-15T17:57:39.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.001+0000 7f7da6000640 1 -- 172.21.15.2:0/4270752484 <== osd.6 v2:172.21.15.137:6808/1848728157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7db00686f0 con 0x7f7d88074210 2024-01-15T17:57:39.021 INFO:teuthology.orchestra.run.smithi002.stdout:60129542148 2024-01-15T17:57:39.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.001+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 shutdown_connections 2024-01-15T17:57:39.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.001+0000 7ff20e400640 1 -- 172.21.15.2:0/1104124749 wait complete. 2024-01-15T17:57:39.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f7d88074210 msgr2=0x7f7d880766b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7d9f400640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f7d88074210 0x7f7d880766b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d8806b310 msgr2=0x7f7d8806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7d9f400640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d8806b310 0x7f7d8806d7d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7d9802b910 tx=0x7f7d9803d040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 msgr2=0x7f7db00c4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7d9f400640 1 --2- 172.21.15.2:0/4270752484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7db01615d0 0x7f7db00c4580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7db00cba60 tx=0x7f7da000d8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.002+0000 7f7dae800640 1 -- 172.21.15.2:0/4270752484 reap_dead start 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.004+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 shutdown_connections 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.004+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 >> 172.21.15.2:0/4270752484 conn(0x7f7db00b46d0 msgr2=0x7f7db006c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.004+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 shutdown_connections 2024-01-15T17:57:39.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.004+0000 7f7d9f400640 1 -- 172.21.15.2:0/4270752484 wait complete. 2024-01-15T17:57:39.025 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:39.025 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T17:57:39.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.010+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 --> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcadc002cb0 con 0x7fcaec0735b0 2024-01-15T17:57:39.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.012+0000 7fcb11600640 1 -- 172.21.15.2:0/2687729645 <== osd.5 v2:172.21.15.137:6800/915165616 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcadc002cb0 con 0x7fcaec0735b0 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.012+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7fcaec0735b0 msgr2=0x7fcaec0759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.012+0000 7fcb0a000640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7fcaec0735b0 0x7fcaec0759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaec06b390 msgr2=0x7fcaec06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb0a000640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaec06b390 0x7fcaec06d850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcafc000f60 tx=0x7fcafc009040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb140d0cc0 msgr2=0x7fcb140b7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb0a000640 1 --2- 172.21.15.2:0/2687729645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb140d0cc0 0x7fcb140b7b70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb04033000 tx=0x7fcb040046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb19400640 1 -- 172.21.15.2:0/2687729645 reap_dead start 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 shutdown_connections 2024-01-15T17:57:39.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.013+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 >> 172.21.15.2:0/2687729645 conn(0x7fcb1409c3c0 msgr2=0x7fcb1415e5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.014+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 shutdown_connections 2024-01-15T17:57:39.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.014+0000 7fcb0a000640 1 -- 172.21.15.2:0/2687729645 wait complete. 2024-01-15T17:57:39.028 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:39.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T17:57:39.029 INFO:teuthology.orchestra.run.smithi002.stdout:60129542148 2024-01-15T17:57:39.030 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T17:57:39.030 INFO:teuthology.orchestra.run.smithi002.stdout:60129542150 2024-01-15T17:57:39.031 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T17:57:39.031 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:39.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T17:57:39.032 INFO:teuthology.orchestra.run.smithi002.stdout:60129542148 2024-01-15T17:57:39.033 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T17:57:39.038 INFO:teuthology.orchestra.run.smithi002.stdout:60129542150 2024-01-15T17:57:39.038 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T17:57:39.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.310+0000 7fc88fa00640 1 Processor -- start 2024-01-15T17:57:39.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.312+0000 7fc88fa00640 1 -- start start 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.314+0000 7fc88fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc88815ad20 0x7fc88806d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.314+0000 7fc88fa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc88806dd80 0x7fc888160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.314+0000 7fc88fa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc888161470 0x7fc888070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.314+0000 7fc88fa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88806e1c0 con 0x7fc88806dd80 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.314+0000 7fc88fa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88815d940 con 0x7fc888161470 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.314+0000 7fc88fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc888161c60 con 0x7fc88815ad20 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.316+0000 7fc88d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc88815ad20 0x7fc88806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc88806dd80 0x7fc888160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc88d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc88815ad20 0x7fc88806d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58632/0 (socket says 172.21.15.2:58632) 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc88806dd80 0x7fc888160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49776/0 (socket says 172.21.15.2:49776) 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc88d000640 1 -- 172.21.15.2:0/1851598320 learned_addr learned my addr 172.21.15.2:0/1851598320 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc88da00640 1 --2- 172.21.15.2:0/1851598320 >> v2:172.21.15.137:3300/0 conn(0x7fc888161470 0x7fc888070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 -- 172.21.15.2:0/1851598320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc88815ad20 msgr2=0x7fc88806d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 --2- 172.21.15.2:0/1851598320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc88815ad20 0x7fc88806d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 -- 172.21.15.2:0/1851598320 >> v2:172.21.15.137:3300/0 conn(0x7fc888161470 msgr2=0x7fc888070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 --2- 172.21.15.2:0/1851598320 >> v2:172.21.15.137:3300/0 conn(0x7fc888161470 0x7fc888070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.317+0000 7fc887e00640 1 -- 172.21.15.2:0/1851598320 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88815dc00 con 0x7fc88806dd80 2024-01-15T17:57:39.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.318+0000 7f4504a00640 1 Processor -- start 2024-01-15T17:57:39.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.319+0000 7f4504a00640 1 -- start start 2024-01-15T17:57:39.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.319+0000 7fc887e00640 1 --2- 172.21.15.2:0/1851598320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88806dd80 0x7fc888160f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc8880b19b0 tx=0x7fc87002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8471029c1d3466b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.319+0000 7fc887400640 1 -- 172.21.15.2:0/1851598320 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870031710 con 0x7fc88806dd80 2024-01-15T17:57:39.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.318+0000 7f32a2c00640 1 Processor -- start 2024-01-15T17:57:39.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.319+0000 7f32a2c00640 1 -- start start 2024-01-15T17:57:39.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f32a2c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f329c0b2a50 0x7f329c06d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f32a2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c0b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.320+0000 7fc887400640 1 -- 172.21.15.2:0/1851598320 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87002f050 con 0x7fc88806dd80 2024-01-15T17:57:39.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.320+0000 7fc887400640 1 -- 172.21.15.2:0/1851598320 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc870031a90 con 0x7fc88806dd80 2024-01-15T17:57:39.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 -- 172.21.15.2:0/1851598320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88806dd80 msgr2=0x7fc888160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 --2- 172.21.15.2:0/1851598320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88806dd80 0x7fc888160f30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc8880b19b0 tx=0x7fc87002b7b0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 -- 172.21.15.2:0/1851598320 shutdown_connections 2024-01-15T17:57:39.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f4504a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc0b3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f4504a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44fc161500 0x7f44fc161970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f4504a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc161070 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f4504a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc0cc1e0 con 0x7f44fc161500 2024-01-15T17:57:39.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7f4504a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0cb940 con 0x7f44fc15d1e0 2024-01-15T17:57:39.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 --2- 172.21.15.2:0/1851598320 >> v2:172.21.15.137:3300/0 conn(0x7fc888161470 0x7fc888070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f329be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f329c0b2a50 0x7f329c06d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f329be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f329c0b2a50 0x7f329c06d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43412/0 (socket says 172.21.15.2:43412) 2024-01-15T17:57:39.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f329be00640 1 -- 172.21.15.2:0/1170559211 learned_addr learned my addr 172.21.15.2:0/1170559211 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 --2- 172.21.15.2:0/1851598320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88806dd80 0x7fc888160f30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f329b400640 1 --2- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c0b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 --2- 172.21.15.2:0/1851598320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc88815ad20 0x7fc88806d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f329c06e1e0 con 0x7f329c0b3930 2024-01-15T17:57:39.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.321+0000 7fc88fa00640 1 -- 172.21.15.2:0/1851598320 >> 172.21.15.2:0/1851598320 conn(0x7fc8880b0dd0 msgr2=0x7fc8880b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f329c1662d0 con 0x7f329c0b2a50 2024-01-15T17:57:39.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c0b3da0 con 0x7f329c06dd80 2024-01-15T17:57:39.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7fc88fa00640 1 -- 172.21.15.2:0/1851598320 shutdown_connections 2024-01-15T17:57:39.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7fc88fa00640 1 -- 172.21.15.2:0/1851598320 wait complete. 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f329aa00640 1 -- 172.21.15.2:0/1170559211 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477064425 0 0) 0x7f329c06e1e0 con 0x7f329c0b3930 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.324+0000 7f329aa00640 1 -- 172.21.15.2:0/1170559211 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327c003520 con 0x7f329c0b3930 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f4502a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44fc161500 0x7f44fc161970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f4502a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44fc161500 0x7f44fc161970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43424/0 (socket says 172.21.15.2:43424) 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f4502a00640 1 -- 172.21.15.2:0/1002366981 learned_addr learned my addr 172.21.15.2:0/1002366981 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.322+0000 7f4502000640 1 --2- 172.21.15.2:0/1002366981 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc0b3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f4502000640 1 -- 172.21.15.2:0/1002366981 >> v2:172.21.15.137:3300/0 conn(0x7f44fc161500 msgr2=0x7f44fc161970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f4502000640 1 --2- 172.21.15.2:0/1002366981 >> v2:172.21.15.137:3300/0 conn(0x7f44fc161500 0x7f44fc161970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f4502000640 1 -- 172.21.15.2:0/1002366981 >> v1:172.21.15.137:6789/0 conn(0x7f44fc0cb3a0 legacy=0x7f44fc0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f4502000640 1 -- 172.21.15.2:0/1002366981 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0cbc20 con 0x7f44fc15d1e0 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.324+0000 7f329be00640 1 -- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 msgr2=0x7f329c0b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.324+0000 7f329be00640 1 --2- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c0b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.324+0000 7f329be00640 1 -- 172.21.15.2:0/1170559211 >> v1:172.21.15.137:6789/0 conn(0x7f329c0b3930 legacy=0x7f329c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.324+0000 7f329be00640 1 -- 172.21.15.2:0/1170559211 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f329c0b4040 con 0x7f329c0b2a50 2024-01-15T17:57:39.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.325+0000 7fc88fa00640 1 Processor -- start 2024-01-15T17:57:39.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f329be00640 1 --2- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b2a50 0x7f329c06d840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3294009e50 tx=0x7f329402bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faa1ecc2cefa0c97 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7fc88fa00640 1 -- start start 2024-01-15T17:57:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f4502000640 1 --2- 172.21.15.2:0/1002366981 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc0b3870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0097a0 tx=0x7f44ec034050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56920fa3c43ad0ce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f329aa00640 1 -- 172.21.15.2:0/1170559211 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329402caa0 con 0x7f329c0b2a50 2024-01-15T17:57:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f329aa00640 1 -- 172.21.15.2:0/1170559211 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329402cc00 con 0x7f329c0b2a50 2024-01-15T17:57:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f4500c00640 1 -- 172.21.15.2:0/1002366981 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec034e00 con 0x7f44fc15d1e0 2024-01-15T17:57:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f4500c00640 1 -- 172.21.15.2:0/1002366981 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec03c440 con 0x7f44fc15d1e0 2024-01-15T17:57:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f4500c00640 1 -- 172.21.15.2:0/1002366981 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec03c810 con 0x7f44fc15d1e0 2024-01-15T17:57:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f329aa00640 1 -- 172.21.15.2:0/1170559211 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32940343f0 con 0x7f329c0b2a50 2024-01-15T17:57:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f7672e00640 1 Processor -- start 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.323+0000 7f7672e00640 1 -- start start 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.325+0000 7f7672e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c071270 0x7f766c0b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.325+0000 7f7672e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f766c06df50 0x7f766c0b3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.325+0000 7f7672e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f766c0b3e70 con 0x7f766c1647b0 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.325+0000 7f7672e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f766c0704b0 con 0x7f766c06df50 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.325+0000 7f7672e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c070790 con 0x7f766c071270 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f766be00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f766c1647b0 0x7f766c06d840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:39880/0 (socket says 172.21.15.2:39880) 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f766be00640 1 -- 172.21.15.2:0/3088360340 learned_addr learned my addr 172.21.15.2:0/3088360340 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f766b400640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c071270 0x7f766c0b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.326+0000 7f7670e00640 1 --2- 172.21.15.2:0/3088360340 >> v2:172.21.15.2:3300/0 conn(0x7f766c06df50 0x7f766c0b3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7fc88fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc88806dd80 0x7fc8880c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761303585 0 0) 0x7f766c0b3e70 con 0x7f766c1647b0 2024-01-15T17:57:39.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c003500 con 0x7f766c1647b0 2024-01-15T17:57:39.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7fc88fa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 0x7fc8880ace90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.327+0000 7fc88fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc888161470 0x7fc8880ab650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f4504a00640 1 -- 172.21.15.2:0/1002366981 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 msgr2=0x7f44fc0b3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f4504a00640 1 --2- 172.21.15.2:0/1002366981 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc0b3870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0097a0 tx=0x7f44ec034050 comp rx=0 tx=0).stop 2024-01-15T17:57:39.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc88fa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8880ad3d0 con 0x7fc88815ad20 2024-01-15T17:57:39.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc88fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8880aca00 con 0x7fc88806dd80 2024-01-15T17:57:39.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc88fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8880abb90 con 0x7fc888161470 2024-01-15T17:57:39.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f4504a00640 1 -- 172.21.15.2:0/1002366981 shutdown_connections 2024-01-15T17:57:39.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f4504a00640 1 --2- 172.21.15.2:0/1002366981 >> v2:172.21.15.137:3300/0 conn(0x7f44fc161500 0x7f44fc161970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f4504a00640 1 --2- 172.21.15.2:0/1002366981 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc0b3870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f4504a00640 1 -- 172.21.15.2:0/1002366981 >> 172.21.15.2:0/1002366981 conn(0x7f44fc0b0840 msgr2=0x7f44fc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc887e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 0x7fc8880ace90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc887e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 0x7fc8880ace90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49798/0 (socket says 172.21.15.2:49798) 2024-01-15T17:57:39.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc887e00640 1 -- 172.21.15.2:0/1820472776 learned_addr learned my addr 172.21.15.2:0/1820472776 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc88d000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc88806dd80 0x7fc8880c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7fc88da00640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc888161470 0x7fc8880ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b2a50 msgr2=0x7f329c06d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f32a2c00640 1 --2- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b2a50 0x7f329c06d840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3294009e50 tx=0x7f329402bf00 comp rx=0 tx=0).stop 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 shutdown_connections 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f32a2c00640 1 --2- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c0b33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f32a2c00640 1 --2- 172.21.15.2:0/1170559211 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b2a50 0x7f329c06d840 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 >> 172.21.15.2:0/1170559211 conn(0x7f329c0b0dd0 msgr2=0x7f329c0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f4504a00640 1 -- 172.21.15.2:0/1002366981 shutdown_connections 2024-01-15T17:57:39.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2890895600 0 0) 0x7f764c003500 con 0x7f766c1647b0 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7fc887e00640 1 -- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc888161470 msgr2=0x7fc8880ab650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7fc887e00640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc888161470 0x7fc8880ab650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7fc887e00640 1 -- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc88806dd80 msgr2=0x7fc8880c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7fc887e00640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc88806dd80 0x7fc8880c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7fc887e00640 1 -- 172.21.15.2:0/1820472776 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870009400 con 0x7fc88815ad20 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 shutdown_connections 2024-01-15T17:57:39.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c071270 msgr2=0x7f766c0b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c071270 0x7f766c0b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 >> v2:172.21.15.2:3300/0 conn(0x7f766c06df50 msgr2=0x7f766c0b3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 --2- 172.21.15.2:0/3088360340 >> v2:172.21.15.2:3300/0 conn(0x7f766c06df50 0x7f766c0b3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c070a70 con 0x7f766c1647b0 2024-01-15T17:57:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f4504a00640 1 -- 172.21.15.2:0/1002366981 wait complete. 2024-01-15T17:57:39.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f765c001670 con 0x7f766c1647b0 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f7670e00640 1 --2- 172.21.15.2:0/3088360340 >> v2:172.21.15.2:3300/0 conn(0x7f766c06df50 0x7f766c0b3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 >> v1:172.21.15.137:6789/0 conn(0x7f766c1647b0 legacy=0x7f766c06d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c003aa0 0x7f764c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f764c004500 0x7f764c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f764c005090 0x7f764c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0704b0 con 0x7f764c003aa0 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c070790 con 0x7f764c004500 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.328+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00bf00 con 0x7f764c005090 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f766be00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c003aa0 0x7f764c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.329+0000 7f766b400640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f764c004500 0x7f764c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7feddb200640 1 Processor -- start 2024-01-15T17:57:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7feddb200640 1 -- start start 2024-01-15T17:57:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7fc887e00640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 0x7fc8880ace90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc87002bdc0 tx=0x7fc8700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f766b400640 1 -- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f764c005090 msgr2=0x7f764c00b990 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T17:57:39.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f766b400640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f764c005090 0x7f764c00b990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f766b400640 1 -- 172.21.15.2:0/3088360340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c003aa0 msgr2=0x7f764c003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f766b400640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c003aa0 0x7f764c003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f766b400640 1 -- 172.21.15.2:0/3088360340 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c070a70 con 0x7f764c004500 2024-01-15T17:57:39.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f32a2c00640 1 -- 172.21.15.2:0/1170559211 wait complete. 2024-01-15T17:57:39.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.330+0000 7f766be00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c003aa0 0x7f764c003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.331+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87003c8d0 con 0x7fc88815ad20 2024-01-15T17:57:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.331+0000 7fc88fa00640 1 -- 172.21.15.2:0/1820472776 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8880abdf0 con 0x7fc88815ad20 2024-01-15T17:57:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.331+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87003ca30 con 0x7fc88815ad20 2024-01-15T17:57:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.332+0000 7feddb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fedd404f4c0 0x7fedd404f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.332+0000 7feddb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd40487b0 0x7fedd4048c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.332+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87003cdc0 con 0x7fc88815ad20 2024-01-15T17:57:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.332+0000 7f766b400640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f764c004500 0x7f764c004980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f766c0ceff0 tx=0x7f764c0037b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3085c5ea7af2ea5b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.333+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765400b040 con 0x7f764c004500 2024-01-15T17:57:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.333+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654038070 con 0x7f764c004500 2024-01-15T17:57:39.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.333+0000 7f766aa00640 1 -- 172.21.15.2:0/3088360340 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765403e600 con 0x7f764c004500 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7672e00640 1 -- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f764c004500 msgr2=0x7f764c004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7672e00640 1 --2- 172.21.15.2:0/3088360340 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f764c004500 0x7f764c004980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f766c0ceff0 tx=0x7f764c0037b0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7670e00640 1 -- 172.21.15.2:0/3088360340 reap_dead start 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.333+0000 7fedd8a00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fedd4053240 0x7fedd4007330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:39896/0 (socket says 172.21.15.2:39896) 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7fedd8a00640 1 -- 172.21.15.2:0/3460741537 learned_addr learned my addr 172.21.15.2:0/3460741537 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd404fde0 con 0x7fedd4053240 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd4007a40 con 0x7fedd404f4c0 2024-01-15T17:57:39.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40537b0 con 0x7fedd40487b0 2024-01-15T17:57:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f32a2c00640 1 Processor -- start 2024-01-15T17:57:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7672e00640 1 -- 172.21.15.2:0/3088360340 shutdown_connections 2024-01-15T17:57:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7672e00640 1 -- 172.21.15.2:0/3088360340 >> 172.21.15.2:0/3088360340 conn(0x7f766c0b0dd0 msgr2=0x7f766c0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7672e00640 1 -- 172.21.15.2:0/3088360340 shutdown_connections 2024-01-15T17:57:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.332+0000 7fc88fa00640 1 -- 172.21.15.2:0/1820472776 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8880ac3b0 con 0x7fc88815ad20 2024-01-15T17:57:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f32a2c00640 1 -- start start 2024-01-15T17:57:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.335+0000 7fedd9400640 1 --2- 172.21.15.2:0/3460741537 >> v2:172.21.15.137:3300/0 conn(0x7fedd404f4c0 0x7fedd404f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.333+0000 7f4504a00640 1 Processor -- start 2024-01-15T17:57:39.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.333+0000 7f4504a00640 1 -- start start 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.335+0000 7fedd9e00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd40487b0 0x7fedd4048c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.335+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377527693 0 0) 0x7fedd404fde0 con 0x7fedd4053240 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.335+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedbc003520 con 0x7fedd4053240 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395218792 0 0) 0x7fedbc003520 con 0x7fedd4053240 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd40487b0 msgr2=0x7fedd4048c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd40487b0 0x7fedd4048c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 >> v2:172.21.15.137:3300/0 conn(0x7fedd404f4c0 msgr2=0x7fedd404f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 --2- 172.21.15.2:0/3460741537 >> v2:172.21.15.137:3300/0 conn(0x7fedd404f4c0 0x7fedd404f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd4053a90 con 0x7fedd4053240 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fedc4001670 con 0x7fedd4053240 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.334+0000 7f7672e00640 1 -- 172.21.15.2:0/3088360340 wait complete. 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 >> v1:172.21.15.137:6789/0 conn(0x7fedd4053240 legacy=0x7fedd4007330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedbc003970 0x7fedbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedbc005180 0x7fedbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd4007a40 con 0x7fedbc0047c0 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40537b0 con 0x7fedbc005180 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc00c070 con 0x7fedbc003970 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fedd9e00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd40487b0 0x7fedd4048c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7f4504a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 0x7f44fc1ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.336+0000 7fedd9e00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedbc005180 0x7fedbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f4504a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc1600a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7fedd9400640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedbc003970 0x7fedbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f4504a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc161500 0x7f44fc15a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f8856000640 1 Processor -- start 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f4504a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc15e490 con 0x7f44fc161500 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f4504a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc15d5c0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f4504a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc1ffc00 con 0x7f44fc15d1e0 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7f4502000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 0x7f44fc1ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f4502000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 0x7f44fc1ff6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43476/0 (socket says 172.21.15.2:43476) 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f4502000640 1 -- 172.21.15.2:0/1842844068 learned_addr learned my addr 172.21.15.2:0/1842844068 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f32a2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c153230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f4501600640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc1600a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f8856000640 1 -- start start 2024-01-15T17:57:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8880acb90 con 0x7fc88815ad20 2024-01-15T17:57:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc87003a040 con 0x7fc88815ad20 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7fc884c00640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc86006b2c0 0x7fc86006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f32a2c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f329c0b2a50 0x7f329c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f32a2c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b3930 0x7f329c0cce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f32a2c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c1fbcf0 con 0x7f329c0b2a50 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f32a2c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f329c153770 con 0x7f329c0b3930 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f32a2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f329c1fb070 con 0x7f329c06dd80 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f329be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c153230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f329be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c153230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58720/0 (socket says 172.21.15.2:58720) 2024-01-15T17:57:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f329be00640 1 -- 172.21.15.2:0/2143172094 learned_addr learned my addr 172.21.15.2:0/2143172094 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f32a0c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b3930 0x7f329c0cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f329be00640 1 -- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b3930 msgr2=0x7f329c0cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f329be00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b3930 0x7f329c0cce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f329be00640 1 -- 172.21.15.2:0/2143172094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f329c0b2a50 msgr2=0x7f329c0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f329be00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f329c0b2a50 0x7f329c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f329be00640 1 -- 172.21.15.2:0/2143172094 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3294009b30 con 0x7f329c06dd80 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.338+0000 7f4502a00640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc161500 0x7f44fc15a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f4502000640 1 -- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 msgr2=0x7f44fc1600a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f4502000640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc1600a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f4502000640 1 -- 172.21.15.2:0/1842844068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc161500 msgr2=0x7f44fc15a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f4502000640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc161500 0x7f44fc15a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f4502000640 1 -- 172.21.15.2:0/1842844068 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ec009400 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f4502a00640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc161500 0x7f44fc15a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f4502000640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 0x7f44fc1ff6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0098c0 tx=0x7f44ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.339+0000 7f8856000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500d1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f8856000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500d2150 con 0x7f88501608b0 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f8856000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88500b34d0 con 0x7f88500d39b0 2024-01-15T17:57:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f8856000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88500b2cd0 con 0x7f88500b28c0 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7fedd8a00640 1 -- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedbc003970 msgr2=0x7fedbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedbc003970 0x7fedbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7fedd8a00640 1 -- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedbc005180 msgr2=0x7fedbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedbc005180 0x7fedbc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.337+0000 7fedd8a00640 1 -- 172.21.15.2:0/3460741537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc003520 con 0x7fedbc0047c0 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7f884fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500d1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.340+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8700ab6f0 con 0x7fc88815ad20 2024-01-15T17:57:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f884fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500d1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58736/0 (socket says 172.21.15.2:58736) 2024-01-15T17:57:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f884fe00640 1 -- 172.21.15.2:0/1166784676 learned_addr learned my addr 172.21.15.2:0/1166784676 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f329be00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c153230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3294036040 tx=0x7f3294004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedbc004c40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedc400c240 tx=0x7fedbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cccd50b843aadfc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7fc88d000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc86006b2c0 0x7fc86006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f884fe00640 1 -- 172.21.15.2:0/1166784676 >> v1:172.21.15.137:6789/0 conn(0x7f88500d39b0 legacy=0x7f88500b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f884fe00640 1 -- 172.21.15.2:0/1166784676 >> v1:172.21.15.2:6789/0 conn(0x7f88501608b0 legacy=0x7f88500d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f884fe00640 1 -- 172.21.15.2:0/1166784676 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8850160c90 con 0x7f88500b28c0 2024-01-15T17:57:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f7672e00640 1 Processor -- start 2024-01-15T17:57:39.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f2017400640 1 Processor -- start 2024-01-15T17:57:39.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329402c8a0 con 0x7f329c06dd80 2024-01-15T17:57:39.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec048920 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec048a80 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.341+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec048e10 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f329c1fb350 con 0x7f329c06dd80 2024-01-15T17:57:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329402ca40 con 0x7f329c06dd80 2024-01-15T17:57:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329404f6a0 con 0x7f329c06dd80 2024-01-15T17:57:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f7672e00640 1 -- start start 2024-01-15T17:57:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f2017400640 1 -- start start 2024-01-15T17:57:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f4504a00640 1 -- 172.21.15.2:0/1842844068 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc0b23d0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.342+0000 7f4504a00640 1 -- 172.21.15.2:0/1842844068 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc15daf0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f329c1fb910 con 0x7f329c06dd80 2024-01-15T17:57:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f7672e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c004500 0x7f766c1fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f7672e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 0x7f766c0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f2017400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 0x7f201006e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f2017400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2010141510 0x7f201006ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f7672e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c16c810 0x7f766c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f7672e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c1ff510 con 0x7f764c004500 2024-01-15T17:57:39.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f7672e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f766c16c380 con 0x7f766c159f00 2024-01-15T17:57:39.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.343+0000 7f7672e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766c0c4630 con 0x7f766c16c810 2024-01-15T17:57:39.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7fc88d000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc86006b2c0 0x7fc86006d780 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc87800c900 tx=0x7fc878008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f766b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 0x7f766c0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f766b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 0x7f766c0c40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43482/0 (socket says 172.21.15.2:43482) 2024-01-15T17:57:39.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f766b400640 1 -- 172.21.15.2:0/3062274173 learned_addr learned my addr 172.21.15.2:0/3062274173 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f884fe00640 1 --2- 172.21.15.2:0/1166784676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500d1a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8848009e80 tx=0x7f884802bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=465ed87ce0339844 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f7670e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c16c810 0x7f766c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f2017400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f201015c570 con 0x7f201015b640 2024-01-15T17:57:39.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f2014a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 0x7f201006e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f2014a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 0x7f201006e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58744/0 (socket says 172.21.15.2:58744) 2024-01-15T17:57:39.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f2014a00640 1 -- 172.21.15.2:0/768406587 learned_addr learned my addr 172.21.15.2:0/768406587 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20100d0df0 con 0x7f2010141510 2024-01-15T17:57:39.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.344+0000 7f884ea00640 1 -- 172.21.15.2:0/1166784676 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884802c930 con 0x7f88500b28c0 2024-01-15T17:57:39.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201015c290 con 0x7f201013e390 2024-01-15T17:57:39.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f884ea00640 1 -- 172.21.15.2:0/1166784676 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884802ca90 con 0x7f88500b28c0 2024-01-15T17:57:39.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f884ea00640 1 -- 172.21.15.2:0/1166784676 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884802ce20 con 0x7f88500b28c0 2024-01-15T17:57:39.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f766b400640 1 -- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c16c810 msgr2=0x7f766c0c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f766b400640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c16c810 0x7f766c0c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f766b400640 1 -- 172.21.15.2:0/3062274173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c004500 msgr2=0x7f766c1fefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f766b400640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c004500 0x7f766c1fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f766b400640 1 -- 172.21.15.2:0/3062274173 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766c16b4d0 con 0x7f766c159f00 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c005250 con 0x7f329c06dd80 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.345+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc15e5d0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 -- 172.21.15.2:0/1166784676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 msgr2=0x7f88500d1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 --2- 172.21.15.2:0/1166784676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500d1a50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8848009e80 tx=0x7f884802bd90 comp rx=0 tx=0).stop 2024-01-15T17:57:39.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f766b400640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 0x7f766c0c40f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7654040040 tx=0x7f765400d5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 -- 172.21.15.2:0/1166784676 shutdown_connections 2024-01-15T17:57:39.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 --2- 172.21.15.2:0/1166784676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500d1a50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 -- 172.21.15.2:0/1166784676 >> 172.21.15.2:0/1166784676 conn(0x7f8850072660 msgr2=0x7f885009c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 -- 172.21.15.2:0/1166784676 shutdown_connections 2024-01-15T17:57:39.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f2014a00640 1 -- 172.21.15.2:0/768406587 >> v2:172.21.15.137:3300/0 conn(0x7f2010141510 msgr2=0x7f201006ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f2014a00640 1 --2- 172.21.15.2:0/768406587 >> v2:172.21.15.137:3300/0 conn(0x7f2010141510 0x7f201006ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f2014a00640 1 -- 172.21.15.2:0/768406587 >> v1:172.21.15.137:6789/0 conn(0x7f201015b640 legacy=0x7f201015ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:39.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f2014a00640 1 -- 172.21.15.2:0/768406587 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2010140a30 con 0x7f201013e390 2024-01-15T17:57:39.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f8856000640 1 -- 172.21.15.2:0/1166784676 wait complete. 2024-01-15T17:57:39.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654005b90 con 0x7f766c159f00 2024-01-15T17:57:39.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7654005d10 con 0x7f766c159f00 2024-01-15T17:57:39.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f766c16b770 con 0x7f766c159f00 2024-01-15T17:57:39.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.346+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654006290 con 0x7f766c159f00 2024-01-15T17:57:39.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.347+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc402cb20 con 0x7fedbc0047c0 2024-01-15T17:57:39.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.347+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc402cc80 con 0x7fedbc0047c0 2024-01-15T17:57:39.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.347+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f766c16bd30 con 0x7f766c159f00 2024-01-15T17:57:39.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.348+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc870076430 con 0x7fc88815ad20 2024-01-15T17:57:39.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.348+0000 7f8856000640 1 Processor -- start 2024-01-15T17:57:39.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.347+0000 7fede0c00640 1 -- 172.21.15.2:0/3460741537 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc4005b10 con 0x7fedbc0047c0 2024-01-15T17:57:39.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.348+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 msgr2=0x7fedbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.348+0000 7feddb200640 1 --2- 172.21.15.2:0/3460741537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedbc004c40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fedc400c240 tx=0x7fedbc0057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.348+0000 7f2014a00640 1 --2- 172.21.15.2:0/768406587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 0x7f201006e7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009870 tx=0x7f1ff802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f20ccc9c046ac383 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7fedd9e00640 1 -- 172.21.15.2:0/3460741537 reap_dead start 2024-01-15T17:57:39.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7f8856000640 1 -- start start 2024-01-15T17:57:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 shutdown_connections 2024-01-15T17:57:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 >> 172.21.15.2:0/3460741537 conn(0x7fedd4054c60 msgr2=0x7fedd4055070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f329404f800 con 0x7f329c06dd80 2024-01-15T17:57:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766c0b40e0 con 0x7f766c159f00 2024-01-15T17:57:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.349+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f44ec03a440 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.350+0000 7f44fa000640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406b310 0x7f44d406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.350+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 shutdown_connections 2024-01-15T17:57:39.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.350+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f44ec0b4cb0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f3293e00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f327406b2f0 0x7f327406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32940bdd00 con 0x7f329c06dd80 2024-01-15T17:57:39.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.352+0000 7f329b400640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f327406b2f0 0x7f327406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f4501600640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406b310 0x7f44d406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f8856000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f8856000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88500d39b0 0x7f88501fdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f8856000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88501608b0 0x7f88500d2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f8856000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88500caca0 con 0x7f88500d39b0 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f8856000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88500cc130 con 0x7f88501608b0 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f8856000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88502010e0 con 0x7f88500b28c0 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f884f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f884f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500cb0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58748/0 (socket says 172.21.15.2:58748) 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f884f400640 1 -- 172.21.15.2:0/1138159493 learned_addr learned my addr 172.21.15.2:0/1138159493 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.352+0000 7f884fe00640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88501608b0 0x7f88500d2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f76540063f0 con 0x7f766c159f00 2024-01-15T17:57:39.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.351+0000 7feddb200640 1 -- 172.21.15.2:0/3460741537 wait complete. 2024-01-15T17:57:39.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.352+0000 7f7663e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f764406b2c0 0x7f764406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.352+0000 7f8847e00640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88500d39b0 0x7f88501fdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f884fe00640 1 -- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 msgr2=0x7f88500cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f884fe00640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f884fe00640 1 -- 172.21.15.2:0/1138159493 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88500d39b0 msgr2=0x7f88501fdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f884fe00640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88500d39b0 0x7f88501fdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f884fe00640 1 -- 172.21.15.2:0/1138159493 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8848009b30 con 0x7f88501608b0 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f8847e00640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88500d39b0 0x7f88501fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:39.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f4501600640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406b310 0x7f44d406d7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f44fc203280 tx=0x7f44f4003d90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f766be00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f764406b2c0 0x7f764406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7feddb200640 1 Processor -- start 2024-01-15T17:57:39.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.353+0000 7f329b400640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f327406b2f0 0x7f327406d7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f328c006a90 tx=0x7f328c006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.354+0000 7feddb200640 1 -- start start 2024-01-15T17:57:39.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.354+0000 7f766be00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f764406b2c0 0x7f764406d780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f765c007ac0 tx=0x7f765c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7feddb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedd40e90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7feddb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedd40e5cc0 0x7fedd41859f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7feddb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd4185f30 0x7fedd41863b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7feddb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40e6120 con 0x7fedbc0047c0 2024-01-15T17:57:39.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7fedd9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedd40e90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7f884fe00640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88501608b0 0x7f88500d2300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8848032000 tx=0x7f8848004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7fedd9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedd40e90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49868/0 (socket says 172.21.15.2:49868) 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7fedd9400640 1 -- 172.21.15.2:0/3885618799 learned_addr learned my addr 172.21.15.2:0/3885618799 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.355+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40e8610 con 0x7fedd40e5cc0 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd402e270 con 0x7fedd4185f30 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd9e00640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd4185f30 0x7fedd41863b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedd40e5cc0 0x7fedd41859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd9400640 1 -- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd4185f30 msgr2=0x7fedd41863b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd9400640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd4185f30 0x7fedd41863b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd9400640 1 -- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedd40e5cc0 msgr2=0x7fedd41859f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd9400640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedd40e5cc0 0x7fedd41859f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.356+0000 7fedd9400640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40e88b0 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.357+0000 7fedd9400640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedd40e90b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feddc04fab0 tx=0x7feddc067f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.357+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc08f070 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.357+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feddc06bc00 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.357+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feddc0721c0 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.357+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd40e8b90 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.357+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd40e64e0 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.358+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd4008b70 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.359+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feddc072320 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.359+0000 7fedd2000640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedb406b2f0 0x7fedb406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.360+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feddc1038c0 con 0x7fedbc0047c0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.360+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedb406b2f0 0x7fedb406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.361+0000 7fedd8a00640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedb406b2f0 0x7fedb406d7b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fedc4034000 tx=0x7fedc4007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.364+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feddc0ce7e0 con 0x7fedbc0047c0 2024-01-15T17:57:39.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.365+0000 7f200f400640 1 -- 172.21.15.2:0/768406587 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff80317c0 con 0x7f201013e390 2024-01-15T17:57:39.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.365+0000 7f200f400640 1 -- 172.21.15.2:0/768406587 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8031920 con 0x7f201013e390 2024-01-15T17:57:39.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.365+0000 7f200f400640 1 -- 172.21.15.2:0/768406587 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8031cb0 con 0x7f201013e390 2024-01-15T17:57:39.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3294088c70 con 0x7f329c06dd80 2024-01-15T17:57:39.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 msgr2=0x7f201006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f2017400640 1 --2- 172.21.15.2:0/768406587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 0x7f201006e7b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8009870 tx=0x7f1ff802b860 comp rx=0 tx=0).stop 2024-01-15T17:57:39.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884803e450 con 0x7f88501608b0 2024-01-15T17:57:39.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 shutdown_connections 2024-01-15T17:57:39.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f2017400640 1 --2- 172.21.15.2:0/768406587 >> v2:172.21.15.137:3300/0 conn(0x7f2010141510 0x7f201006ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f2017400640 1 --2- 172.21.15.2:0/768406587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f201013e390 0x7f201006e7b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 >> 172.21.15.2:0/768406587 conn(0x7f20100d1010 msgr2=0x7f20100d13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.366+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88502013c0 con 0x7f88501608b0 2024-01-15T17:57:39.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.367+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 shutdown_connections 2024-01-15T17:57:39.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.367+0000 7f2017400640 1 -- 172.21.15.2:0/768406587 wait complete. 2024-01-15T17:57:39.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.367+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88500ca8c0 con 0x7f88501608b0 2024-01-15T17:57:39.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.367+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44ec07fc40 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.367+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76540044c0 con 0x7f766c159f00 2024-01-15T17:57:39.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.368+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88500cade0 con 0x7f88501608b0 2024-01-15T17:57:39.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.369+0000 7f2017400640 1 Processor -- start 2024-01-15T17:57:39.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.370+0000 7f2017400640 1 -- start start 2024-01-15T17:57:39.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.370+0000 7f2017400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f201013e390 0x7f201013d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.370+0000 7f2017400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2010141510 0x7f20100c80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f2017400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f201015b640 0x7f201006ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f2017400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100caf60 con 0x7f201013e390 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f2017400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010159760 con 0x7f201015b640 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f2017400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2010159430 con 0x7f2010141510 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f200fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2010141510 0x7f20100c80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f2015400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f201015b640 0x7f201006ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f200fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2010141510 0x7f20100c80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58762/0 (socket says 172.21.15.2:58762) 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.371+0000 7f200fe00640 1 -- 172.21.15.2:0/458034420 learned_addr learned my addr 172.21.15.2:0/458034420 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2014a00640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f201013e390 0x7f201013d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2015400640 1 -- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2010141510 msgr2=0x7f20100c80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2015400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2010141510 0x7f20100c80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2015400640 1 -- 172.21.15.2:0/458034420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f201013e390 msgr2=0x7f201013d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2015400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f201013e390 0x7f201013d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2015400640 1 -- 172.21.15.2:0/458034420 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff80094d0 con 0x7f201015b640 2024-01-15T17:57:39.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.372+0000 7f2014a00640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f201013e390 0x7f201013d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:39.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.373+0000 7f2015400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f201015b640 0x7f201006ca50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f200800be20 tx=0x7f200800bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.381+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f884803e5b0 con 0x7f88501608b0 2024-01-15T17:57:39.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.381+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f884803e920 con 0x7f88501608b0 2024-01-15T17:57:39.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.381+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200800ce70 con 0x7f201015b640 2024-01-15T17:57:39.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.381+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20100c8910 con 0x7f201015b640 2024-01-15T17:57:39.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.381+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f766c0b40e0 con 0x7f766c159f00 2024-01-15T17:57:39.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.381+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100c8620 con 0x7f201015b640 2024-01-15T17:57:39.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.382+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2008014440 con 0x7f201015b640 2024-01-15T17:57:39.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.382+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008014810 con 0x7f201015b640 2024-01-15T17:57:39.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.384+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0005250 con 0x7f201015b640 2024-01-15T17:57:39.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.384+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f884803ea80 con 0x7f88501608b0 2024-01-15T17:57:39.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.386+0000 7f8847400640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f882806b2f0 0x7f882806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.386+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88480acd00 con 0x7f88501608b0 2024-01-15T17:57:39.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.386+0000 7f884f400640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f882806b2f0 0x7f882806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.387+0000 7f884f400640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f882806b2f0 0x7f882806d7b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88400099a0 tx=0x7f88400090e0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.390+0000 7f3ec2800640 1 Processor -- start 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.390+0000 7f3ec2800640 1 -- start start 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.391+0000 7f3ec2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3ebc1469f0 0x7f3ebc071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.391+0000 7f3ec2800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc0ac870 con 0x7f3ebc06cb80 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.391+0000 7f3ec2800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ebc15d970 con 0x7f3ebc146dd0 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.391+0000 7f3ec2800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ebc15dc30 con 0x7f3ebc1469f0 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.392+0000 7f3ebb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3ebc1469f0 0x7f3ebc071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.392+0000 7f3ebbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3ebc06cb80 0x7f3ebc0b34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35466/0 (socket says 172.21.15.2:35466) 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.392+0000 7f3ebbe00640 1 -- 172.21.15.2:0/3272682350 learned_addr learned my addr 172.21.15.2:0/3272682350 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.392+0000 7f3ebb400640 1 -- 172.21.15.2:0/3272682350 >> v1:172.21.15.137:6789/0 conn(0x7f3ebc146dd0 legacy=0x7f3ebc06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.392+0000 7f3ebb400640 1 -- 172.21.15.2:0/3272682350 >> v1:172.21.15.2:6789/0 conn(0x7f3ebc06cb80 legacy=0x7f3ebc0b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.393+0000 7f3ebb400640 1 -- 172.21.15.2:0/3272682350 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc15df10 con 0x7f3ebc1469f0 2024-01-15T17:57:39.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.395+0000 7f3ebb400640 1 --2- 172.21.15.2:0/3272682350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc1469f0 0x7f3ebc071df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097a0 tx=0x7f3eac02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d915f1fefceafa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.395+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8848077dc0 con 0x7f88501608b0 2024-01-15T17:57:39.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.395+0000 7f3eba000640 1 -- 172.21.15.2:0/3272682350 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac030840 con 0x7f3ebc1469f0 2024-01-15T17:57:39.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.395+0000 7f3eba000640 1 -- 172.21.15.2:0/3272682350 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac0309a0 con 0x7f3ebc1469f0 2024-01-15T17:57:39.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.395+0000 7f3eba000640 1 -- 172.21.15.2:0/3272682350 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac030d70 con 0x7f3ebc1469f0 2024-01-15T17:57:39.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 -- 172.21.15.2:0/3272682350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc1469f0 msgr2=0x7f3ebc071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 --2- 172.21.15.2:0/3272682350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc1469f0 0x7f3ebc071df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097a0 tx=0x7f3eac02b980 comp rx=0 tx=0).stop 2024-01-15T17:57:39.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 -- 172.21.15.2:0/3272682350 shutdown_connections 2024-01-15T17:57:39.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 --2- 172.21.15.2:0/3272682350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc1469f0 0x7f3ebc071df0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097a0 tx=0x7f3eac02b980 comp rx=0 tx=0).stop 2024-01-15T17:57:39.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 -- 172.21.15.2:0/3272682350 >> 172.21.15.2:0/3272682350 conn(0x7f3ebc06da20 msgr2=0x7f3ebc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 -- 172.21.15.2:0/3272682350 shutdown_connections 2024-01-15T17:57:39.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f200801d3b0 con 0x7f201015b640 2024-01-15T17:57:39.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.397+0000 7f3ec2800640 1 -- 172.21.15.2:0/3272682350 wait complete. 2024-01-15T17:57:39.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.400+0000 7f3ec2800640 1 Processor -- start 2024-01-15T17:57:39.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.400+0000 7f3ec2800640 1 -- start start 2024-01-15T17:57:39.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ec2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc06cb80 0x7f3ebc14f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ec2800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ebc15c730 0x7f3ebc138390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ec2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 0x7f3ebc0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ec2800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc138d50 con 0x7f3ebc06cb80 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ec2800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc13b770 con 0x7f3ebc15c730 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ec2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc13a6d0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f200cc00640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1fe806b2c0 0x7f1fe806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ebb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc06cb80 0x7f3ebc14f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ebbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 0x7f3ebc0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ebbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 0x7f3ebc0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58774/0 (socket says 172.21.15.2:58774) 2024-01-15T17:57:39.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f3ebbe00640 1 -- 172.21.15.2:0/4155969257 learned_addr learned my addr 172.21.15.2:0/4155969257 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:39.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.401+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f200808a0e0 con 0x7f201015b640 2024-01-15T17:57:39.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f2014a00640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1fe806b2c0 0x7f1fe806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f3ebaa00640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ebc15c730 0x7f3ebc138390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f3ebbe00640 1 -- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ebc15c730 msgr2=0x7f3ebc138390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f3ebbe00640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ebc15c730 0x7f3ebc138390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f3ebbe00640 1 -- 172.21.15.2:0/4155969257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc06cb80 msgr2=0x7f3ebc14f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f3ebbe00640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc06cb80 0x7f3ebc14f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.402+0000 7f3ebbe00640 1 -- 172.21.15.2:0/4155969257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eac009400 con 0x7f3ebc1388d0 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.403+0000 7f3ebaa00640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ebc15c730 0x7f3ebc138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.403+0000 7f3ebbe00640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 0x7f3ebc0afc90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400cbf0 tx=0x7f3eb400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.404+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400ad40 con 0x7f3ebc1388d0 2024-01-15T17:57:39.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.405+0000 7f2014a00640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1fe806b2c0 0x7f1fe806d780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8036000 tx=0x7f1ff803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.404+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb4012400 con 0x7f3ebc1388d0 2024-01-15T17:57:39.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.404+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb400aea0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.404+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ebc13a9b0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.404+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ebc13af70 con 0x7f3ebc1388d0 2024-01-15T17:57:39.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.405+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ebc0686f0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.406+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2008055070 con 0x7f201015b640 2024-01-15T17:57:39.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.407+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3eb4007070 con 0x7f3ebc1388d0 2024-01-15T17:57:39.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.408+0000 7f3eb3400640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3e9406b310 0x7f3e9406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:39.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.408+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3eb4089df0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.410+0000 7f3ebb400640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3e9406b310 0x7f3e9406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:39.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.411+0000 7f3ebb400640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3e9406b310 0x7f3e9406d7d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3eac036000 tx=0x7f3eac008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:39.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.422+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3eb4054df0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.778+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fedd4005460 con 0x7fedbc0047c0 2024-01-15T17:57:39.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.779+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc8880686f0 con 0x7fc88815ad20 2024-01-15T17:57:39.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.779+0000 7fedd2000640 1 -- 172.21.15.2:0/3885618799 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7feddc0d1e00 con 0x7fedbc0047c0 2024-01-15T17:57:39.782 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:39.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.780+0000 7fc884c00640 1 -- 172.21.15.2:0/1820472776 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc870079a50 con 0x7fc88815ad20 2024-01-15T17:57:39.784 INFO:teuthology.orchestra.run.smithi002.stdout:60129542147 2024-01-15T17:57:39.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.782+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedb406b2f0 msgr2=0x7fedb406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7feddb200640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedb406b2f0 0x7fedb406d7b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fedc4034000 tx=0x7fedc4007000 comp rx=0 tx=0).stop 2024-01-15T17:57:39.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 msgr2=0x7fedd40e90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7feddb200640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedd40e90b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feddc04fab0 tx=0x7feddc067f30 comp rx=0 tx=0).stop 2024-01-15T17:57:39.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc86006b2c0 msgr2=0x7fc86006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7fc87e000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc86006b2c0 0x7fc86006d780 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc87800c900 tx=0x7fc878008040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 msgr2=0x7fc8880ace90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7fc87e000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 0x7fc8880ace90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc87002bdc0 tx=0x7fc8700046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 shutdown_connections 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.783+0000 7feddb200640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedb406b2f0 0x7fedb406d7b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7feddb200640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedd4185f30 0x7fedd41863b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7feddb200640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedd40e5cc0 0x7fedd41859f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7feddb200640 1 --2- 172.21.15.2:0/3885618799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fedbc0047c0 0x7fedd40e90b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 >> 172.21.15.2:0/3885618799 conn(0x7fedd4054c60 msgr2=0x7fedd404a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 shutdown_connections 2024-01-15T17:57:39.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7fc87e000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc86006b2c0 0x7fc86006d780 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 shutdown_connections 2024-01-15T17:57:39.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7feddb200640 1 -- 172.21.15.2:0/3885618799 wait complete. 2024-01-15T17:57:39.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7fc87e000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc888161470 0x7fc8880ab650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7fc87e000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc88815ad20 0x7fc8880ace90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7fc87e000640 1 --2- 172.21.15.2:0/1820472776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc88806dd80 0x7fc8880c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.784+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 >> 172.21.15.2:0/1820472776 conn(0x7fc8880b0dd0 msgr2=0x7fc88815d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.785+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 shutdown_connections 2024-01-15T17:57:39.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.785+0000 7fc87e000640 1 -- 172.21.15.2:0/1820472776 wait complete. 2024-01-15T17:57:39.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.790+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f326c0057d0 con 0x7f329c06dd80 2024-01-15T17:57:39.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.791+0000 7f3293e00640 1 -- 172.21.15.2:0/2143172094 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f329408c290 con 0x7f329c06dd80 2024-01-15T17:57:39.797 INFO:teuthology.orchestra.run.smithi002.stdout:60129542148 2024-01-15T17:57:39.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.795+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f327406b2f0 msgr2=0x7f327406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.795+0000 7f32a2c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f327406b2f0 0x7f327406d7b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f328c006a90 tx=0x7f328c006a20 comp rx=0 tx=0).stop 2024-01-15T17:57:39.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.795+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 msgr2=0x7f329c153230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.795+0000 7f32a2c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c153230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3294036040 tx=0x7f3294004a40 comp rx=0 tx=0).stop 2024-01-15T17:57:39.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.795+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f44fc15e5d0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 shutdown_connections 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f327406b2f0 0x7f327406d7b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f329c0b3930 0x7f329c0cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f329c0b2a50 0x7f329c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 --2- 172.21.15.2:0/2143172094 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f329c06dd80 0x7f329c153230 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 >> 172.21.15.2:0/2143172094 conn(0x7f329c0b0dd0 msgr2=0x7f329c158240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 shutdown_connections 2024-01-15T17:57:39.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f32a2c00640 1 -- 172.21.15.2:0/2143172094 wait complete. 2024-01-15T17:57:39.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.796+0000 7f44fa000640 1 -- 172.21.15.2:0/1842844068 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f44fc15e5d0 con 0x7f44fc0cb3a0 2024-01-15T17:57:39.800 INFO:teuthology.orchestra.run.smithi002.stdout:60129542148 2024-01-15T17:57:39.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.803+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406b310 msgr2=0x7f44d406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.803+0000 7f44f3400640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406b310 0x7f44d406d7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f44fc203280 tx=0x7f44f4003d90 comp rx=0 tx=0).stop 2024-01-15T17:57:39.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.803+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 msgr2=0x7f44fc1ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.803+0000 7f44f3400640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 0x7f44fc1ff6c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f44ec0098c0 tx=0x7f44ec0046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 shutdown_connections 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406b310 0x7f44d406d7d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc161500 0x7f44fc15a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc15d1e0 0x7f44fc1600a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 --2- 172.21.15.2:0/1842844068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0cb3a0 0x7f44fc1ff6c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 >> 172.21.15.2:0/1842844068 conn(0x7f44fc0b0840 msgr2=0x7f44fc0afb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 shutdown_connections 2024-01-15T17:57:39.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.804+0000 7f44f3400640 1 -- 172.21.15.2:0/1842844068 wait complete. 2024-01-15T17:57:39.812 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.0 2024-01-15T17:57:39.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.810+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f766c0b40e0 con 0x7f766c159f00 2024-01-15T17:57:39.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.811+0000 7f7663e00640 1 -- 172.21.15.2:0/3062274173 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f765403a020 con 0x7f766c159f00 2024-01-15T17:57:39.816 INFO:teuthology.orchestra.run.smithi002.stdout:60129542147 2024-01-15T17:57:39.817 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.3 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.814+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f764406b2c0 msgr2=0x7f764406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f764406b2c0 0x7f764406d780 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f765c007ac0 tx=0x7f765c009040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 msgr2=0x7f766c0c40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 0x7f766c0c40f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7654040040 tx=0x7f765400d5a0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 shutdown_connections 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f764406b2c0 0x7f764406d780 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f766c16c810 0x7f766c0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f766c159f00 0x7f766c0c40f0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7654040040 tx=0x7f765400d5a0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 --2- 172.21.15.2:0/3062274173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f764c004500 0x7f766c1fefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 >> 172.21.15.2:0/3062274173 conn(0x7f766c0b0dd0 msgr2=0x7f766c166bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 shutdown_connections 2024-01-15T17:57:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.815+0000 7f7672e00640 1 -- 172.21.15.2:0/3062274173 wait complete. 2024-01-15T17:57:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.818+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f88500cade0 con 0x7f88501608b0 2024-01-15T17:57:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.819+0000 7f8847400640 1 -- 172.21.15.2:0/1138159493 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88500cade0 con 0x7f88501608b0 2024-01-15T17:57:39.822 INFO:teuthology.orchestra.run.smithi002.stdout:60129542148 2024-01-15T17:57:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.822+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f1fe00057d0 con 0x7f201015b640 2024-01-15T17:57:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f882806b2f0 msgr2=0x7f882806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f882806b2f0 0x7f882806d7b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f88400099a0 tx=0x7f88400090e0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88501608b0 msgr2=0x7f88500d2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88501608b0 0x7f88500d2300 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8848032000 tx=0x7f8848004a40 comp rx=0 tx=0).stop 2024-01-15T17:57:39.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.1 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f200cc00640 1 -- 172.21.15.2:0/458034420 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2008023090 con 0x7f201015b640 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 shutdown_connections 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f882806b2f0 0x7f882806d7b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88501608b0 0x7f88500d2300 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88500d39b0 0x7f88501fdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 --2- 172.21.15.2:0/1138159493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88500b28c0 0x7f88500cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 >> 172.21.15.2:0/1138159493 conn(0x7f8850072660 msgr2=0x7f88500b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 shutdown_connections 2024-01-15T17:57:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.823+0000 7f8856000640 1 -- 172.21.15.2:0/1138159493 wait complete. 2024-01-15T17:57:39.828 INFO:teuthology.orchestra.run.smithi002.stdout:60129542147 2024-01-15T17:57:39.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.2 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1fe806b2c0 msgr2=0x7f1fe806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1fe806b2c0 0x7f1fe806d780 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1ff8036000 tx=0x7f1ff803d040 comp rx=0 tx=0).stop 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f201015b640 msgr2=0x7f201006ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f201015b640 0x7f201006ca50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f200800be20 tx=0x7f200800bf10 comp rx=0 tx=0).stop 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 shutdown_connections 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1fe806b2c0 0x7f1fe806d780 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f201015b640 0x7f201006ca50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2010141510 0x7f20100c80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 --2- 172.21.15.2:0/458034420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f201013e390 0x7f201013d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 >> 172.21.15.2:0/458034420 conn(0x7f20100d1010 msgr2=0x7f2010158330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 shutdown_connections 2024-01-15T17:57:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.827+0000 7f2017400640 1 -- 172.21.15.2:0/458034420 wait complete. 2024-01-15T17:57:39.837 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.7 2024-01-15T17:57:39.842 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.4 2024-01-15T17:57:39.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.842+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3ebc0716b0 con 0x7f3ebc1388d0 2024-01-15T17:57:39.845 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:39.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.843+0000 7f3eb3400640 1 -- 172.21.15.2:0/4155969257 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3eb4017090 con 0x7f3ebc1388d0 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3e9406b310 msgr2=0x7f3e9406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3e9406b310 0x7f3e9406d7d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3eac036000 tx=0x7f3eac008030 comp rx=0 tx=0).stop 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 msgr2=0x7f3ebc0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 0x7f3ebc0afc90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400cbf0 tx=0x7f3eb400a430 comp rx=0 tx=0).stop 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 shutdown_connections 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3e9406b310 0x7f3e9406d7d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ebc1388d0 0x7f3ebc0afc90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ebc15c730 0x7f3ebc138390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 --2- 172.21.15.2:0/4155969257 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ebc06cb80 0x7f3ebc14f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 >> 172.21.15.2:0/4155969257 conn(0x7f3ebc06da20 msgr2=0x7f3ebc06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 shutdown_connections 2024-01-15T17:57:39.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:39.846+0000 7f3ec2800640 1 -- 172.21.15.2:0/4155969257 wait complete. 2024-01-15T17:57:39.852 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.6 2024-01-15T17:57:39.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2024-01-15T17:57:40.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-15T17:57:40.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-15T17:57:40.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-15T17:57:40.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-15T17:57:40.838 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-15T17:57:40.842 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-15T17:57:40.853 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-15T17:57:40.868 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-15T17:57:41.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.101+0000 7f5270600640 1 Processor -- start 2024-01-15T17:57:41.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.101+0000 7f5270600640 1 -- start start 2024-01-15T17:57:41.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.104+0000 7f5270600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f52600530a0 0x7f52600e0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.105+0000 7f5270600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52600df4e0 con 0x7f52600df100 2024-01-15T17:57:41.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.106+0000 7f5270600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52600535f0 con 0x7f5260050180 2024-01-15T17:57:41.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.105+0000 7f526dc00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f52600df100 0x7f526004fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35470/0 (socket says 172.21.15.2:35470) 2024-01-15T17:57:41.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.106+0000 7f526dc00640 1 -- 172.21.15.2:0/2412039209 learned_addr learned my addr 172.21.15.2:0/2412039209 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.107+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833906798 0 0) 0x7f52600df4e0 con 0x7f52600df100 2024-01-15T17:57:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.107+0000 7f5270600640 1 -- 172.21.15.2:0/2412039209 --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f526004f840 con 0x7f52600530a0 2024-01-15T17:57:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.107+0000 7f526e600640 1 --2- 172.21.15.2:0/2412039209 >> v2:172.21.15.2:3300/0 conn(0x7f52600530a0 0x7f52600e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.107+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5248003520 con 0x7f52600df100 2024-01-15T17:57:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.108+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899806532 0 0) 0x7f52600535f0 con 0x7f5260050180 2024-01-15T17:57:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.108+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52600df4e0 con 0x7f5260050180 2024-01-15T17:57:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.108+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 961623307 0 0) 0x7f5248003520 con 0x7f52600df100 2024-01-15T17:57:41.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.108+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 >> v2:172.21.15.2:3300/0 conn(0x7f52600530a0 msgr2=0x7f52600e0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.109+0000 7f5267e00640 1 --2- 172.21.15.2:0/2412039209 >> v2:172.21.15.2:3300/0 conn(0x7f52600530a0 0x7f52600e0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.109+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 >> v1:172.21.15.137:6789/0 conn(0x7f5260050180 legacy=0x7f5260005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.109+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52600538d0 con 0x7f52600df100 2024-01-15T17:57:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.109+0000 7f526e600640 1 --2- 172.21.15.2:0/2412039209 >> v2:172.21.15.2:3300/0 conn(0x7f52600530a0 0x7f52600e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.109+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f526804ed60 con 0x7f52600df100 2024-01-15T17:57:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.109+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 >> v1:172.21.15.2:6789/0 conn(0x7f52600df100 legacy=0x7f526004fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.110+0000 7f5267e00640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5248003ac0 0x7f5248003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.110+0000 7f5267e00640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5248004420 0x7f5248006cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.110+0000 7f5267e00640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f52480076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.110+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526004f840 con 0x7f5248007230 2024-01-15T17:57:41.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.110+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5248010430 con 0x7f5248004420 2024-01-15T17:57:41.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.110+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248010570 con 0x7f5248003ac0 2024-01-15T17:57:41.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.111+0000 7f526dc00640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5248003ac0 0x7f5248003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.111+0000 7f526d200640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5248004420 0x7f5248006cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.111+0000 7f526e600640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f52480076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.112+0000 7f526e600640 1 -- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5248003ac0 msgr2=0x7f5248003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.113+0000 7f526e600640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5248003ac0 0x7f5248003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.113+0000 7f526e600640 1 -- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5248004420 msgr2=0x7f5248006cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.113+0000 7f526e600640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5248004420 0x7f5248006cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.113+0000 7f526e600640 1 -- 172.21.15.2:0/2412039209 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248003520 con 0x7f5248007230 2024-01-15T17:57:41.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.114+0000 7f526d200640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5248004420 0x7f5248006cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:41.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.115+0000 7f526e600640 1 -- 172.21.15.2:0/2412039209 reap_dead start 2024-01-15T17:57:41.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.115+0000 7f526e600640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f52480076e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5258009b00 tx=0x7f52480037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e47ce7cff6e1508 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.116+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258004220 con 0x7f5248007230 2024-01-15T17:57:41.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.116+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52580043a0 con 0x7f5248007230 2024-01-15T17:57:41.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.116+0000 7f5267e00640 1 -- 172.21.15.2:0/2412039209 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258003850 con 0x7f5248007230 2024-01-15T17:57:41.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.117+0000 7f5270600640 1 -- 172.21.15.2:0/2412039209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 msgr2=0x7f52480076e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.118+0000 7f5270600640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f52480076e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5258009b00 tx=0x7f52480037d0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.119+0000 7f5270600640 1 -- 172.21.15.2:0/2412039209 shutdown_connections 2024-01-15T17:57:41.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.119+0000 7f5270600640 1 --2- 172.21.15.2:0/2412039209 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f52480076e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.119+0000 7f5270600640 1 -- 172.21.15.2:0/2412039209 >> 172.21.15.2:0/2412039209 conn(0x7f5260009fb0 msgr2=0x7f526000a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.119+0000 7f5270600640 1 -- 172.21.15.2:0/2412039209 shutdown_connections 2024-01-15T17:57:41.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.119+0000 7f5270600640 1 -- 172.21.15.2:0/2412039209 wait complete. 2024-01-15T17:57:41.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.122+0000 7f5270600640 1 Processor -- start 2024-01-15T17:57:41.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.123+0000 7f5270600640 1 -- start start 2024-01-15T17:57:41.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.125+0000 7f5270600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f5260174910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.126+0000 7f5270600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52600da420 0x7f5260177720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.126+0000 7f5270600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 0x7f52600dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.126+0000 7f5270600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5260174e50 con 0x7f5248007230 2024-01-15T17:57:41.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.126+0000 7f5270600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52600dc510 con 0x7f52600dbb50 2024-01-15T17:57:41.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.126+0000 7f5270600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5260177c60 con 0x7f52600da420 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 0x7f52600dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 0x7f52600dbfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43526/0 (socket says 172.21.15.2:43526) 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526e600640 1 -- 172.21.15.2:0/4117274392 learned_addr learned my addr 172.21.15.2:0/4117274392 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526d200640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52600da420 0x7f5260177720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526e600640 1 -- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52600da420 msgr2=0x7f5260177720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526e600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52600da420 0x7f5260177720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.127+0000 7f526e600640 1 -- 172.21.15.2:0/4117274392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 msgr2=0x7f5260174910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.128+0000 7f526dc00640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f5260174910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.128+0000 7f526e600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f5260174910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.128+0000 7f526e600640 1 -- 172.21.15.2:0/4117274392 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52600d7940 con 0x7f52600dbb50 2024-01-15T17:57:41.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.128+0000 7f526dc00640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f5260174910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:41.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.129+0000 7f526e600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 0x7f52600dbfd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5258009c20 tx=0x7f5258004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.130+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258006dd0 con 0x7f52600dbb50 2024-01-15T17:57:41.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.130+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525802cd50 con 0x7f52600dbb50 2024-01-15T17:57:41.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.130+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258034680 con 0x7f52600dbb50 2024-01-15T17:57:41.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.130+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526004d230 con 0x7f52600dbb50 2024-01-15T17:57:41.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.130+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f526004d7f0 con 0x7f52600dbb50 2024-01-15T17:57:41.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.134+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526004da10 con 0x7f52600dbb50 2024-01-15T17:57:41.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.136+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f52580347e0 con 0x7f52600dbb50 2024-01-15T17:57:41.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.138+0000 7f5265600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006b2c0 0x7f524006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.138+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52580ab0a0 con 0x7f52600dbb50 2024-01-15T17:57:41.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.139+0000 7f526dc00640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006b2c0 0x7f524006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.140+0000 7f526dc00640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006b2c0 0x7f524006d780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5268069de0 tx=0x7f526808af90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.150+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5258075f00 con 0x7f52600dbb50 2024-01-15T17:57:41.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.152+0000 7f5b2b200640 1 Processor -- start 2024-01-15T17:57:41.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.152+0000 7f5b2b200640 1 -- start start 2024-01-15T17:57:41.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.154+0000 7f5b2b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 0x7f5b24053b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.154+0000 7f5b2b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5b2403d0d0 0x7f5b24006640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.154+0000 7f5b2b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b24007870 con 0x7f5b240061e0 2024-01-15T17:57:41.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.154+0000 7f5b2b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240540a0 con 0x7f5b2403d0d0 2024-01-15T17:57:41.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.154+0000 7f5b2b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24006c60 con 0x7f5b24053780 2024-01-15T17:57:41.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.155+0000 7f5b2a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5b2403d0d0 0x7f5b24006640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.155+0000 7f5b29e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 0x7f5b24053b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.155+0000 7f5b29e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 0x7f5b24053b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58800/0 (socket says 172.21.15.2:58800) 2024-01-15T17:57:41.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.155+0000 7f5b29e00640 1 -- 172.21.15.2:0/1688768177 learned_addr learned my addr 172.21.15.2:0/1688768177 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f4378600640 1 Processor -- start 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.156+0000 7f5b28a00640 1 -- 172.21.15.2:0/1688768177 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855493593 0 0) 0x7f5b24007870 con 0x7f5b240061e0 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f5b28a00640 1 -- 172.21.15.2:0/1688768177 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b0c003520 con 0x7f5b240061e0 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f5b29e00640 1 -- 172.21.15.2:0/1688768177 >> v2:172.21.15.2:3300/0 conn(0x7f5b2403d0d0 msgr2=0x7f5b24006640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f5b29e00640 1 --2- 172.21.15.2:0/1688768177 >> v2:172.21.15.2:3300/0 conn(0x7f5b2403d0d0 0x7f5b24006640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f5b29e00640 1 -- 172.21.15.2:0/1688768177 >> v1:172.21.15.2:6789/0 conn(0x7f5b240061e0 legacy=0x7f5b24057360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f5b29e00640 1 -- 172.21.15.2:0/1688768177 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24006f40 con 0x7f5b24053780 2024-01-15T17:57:41.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f5b2a800640 1 --2- 172.21.15.2:0/1688768177 >> v2:172.21.15.2:3300/0 conn(0x7f5b2403d0d0 0x7f5b24006640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:41.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.157+0000 7f4378600640 1 -- start start 2024-01-15T17:57:41.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.159+0000 7f4378600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43700d2f10 0x7f43700d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.159+0000 7f4378600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700ce930 con 0x7f437015d670 2024-01-15T17:57:41.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.159+0000 7f4378600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700cec10 con 0x7f43700ce480 2024-01-15T17:57:41.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.159+0000 7f4378600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43700ceef0 con 0x7f43700d2f10 2024-01-15T17:57:41.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.160+0000 7f4375200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f437015d670 0x7f43700cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35484/0 (socket says 172.21.15.2:35484) 2024-01-15T17:57:41.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.159+0000 7f5b29e00640 1 --2- 172.21.15.2:0/1688768177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 0x7f5b24053b60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009870 tx=0x7f5b1c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=111e042b4ecfca49 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.160+0000 7f4375200640 1 -- 172.21.15.2:0/196995504 learned_addr learned my addr 172.21.15.2:0/196995504 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:41.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.160+0000 7f4375c00640 1 --2- 172.21.15.2:0/196995504 >> v2:172.21.15.137:3300/0 conn(0x7f43700d2f10 0x7f43700d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.160+0000 7f5b28a00640 1 -- 172.21.15.2:0/1688768177 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c030840 con 0x7f5b24053780 2024-01-15T17:57:41.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.160+0000 7f5b28a00640 1 -- 172.21.15.2:0/1688768177 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c0309a0 con 0x7f5b24053780 2024-01-15T17:57:41.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.160+0000 7f5b28a00640 1 -- 172.21.15.2:0/1688768177 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c030d30 con 0x7f5b24053780 2024-01-15T17:57:41.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.161+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614123117 0 0) 0x7f43700ce930 con 0x7f437015d670 2024-01-15T17:57:41.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.161+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4350003520 con 0x7f437015d670 2024-01-15T17:57:41.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.161+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3158545151 0 0) 0x7f4350003520 con 0x7f437015d670 2024-01-15T17:57:41.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.161+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 >> v2:172.21.15.137:3300/0 conn(0x7f43700d2f10 msgr2=0x7f43700d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f5b2b200640 1 -- 172.21.15.2:0/1688768177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 msgr2=0x7f5b24053b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7fa7f1c00640 1 Processor -- start 2024-01-15T17:57:41.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f5b2b200640 1 --2- 172.21.15.2:0/1688768177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 0x7f5b24053b60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009870 tx=0x7f5b1c02b980 comp rx=0 tx=0).stop 2024-01-15T17:57:41.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 --2- 172.21.15.2:0/196995504 >> v2:172.21.15.137:3300/0 conn(0x7f43700d2f10 0x7f43700d32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 >> v1:172.21.15.137:6789/0 conn(0x7f43700ce480 legacy=0x7f43700b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700cf1d0 con 0x7f437015d670 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4360001670 con 0x7f437015d670 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 >> v1:172.21.15.2:6789/0 conn(0x7f437015d670 legacy=0x7f43700cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43500039c0 0x7f4350003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43500047c0 0x7f4350004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4350005180 0x7f4350014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43700ceef0 con 0x7f4350005180 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4350014890 con 0x7f43500047c0 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4350014a50 con 0x7f43500039c0 2024-01-15T17:57:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f5b2b200640 1 -- 172.21.15.2:0/1688768177 shutdown_connections 2024-01-15T17:57:41.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f5b2b200640 1 --2- 172.21.15.2:0/1688768177 >> v2:172.21.15.2:3300/0 conn(0x7f5b2403d0d0 0x7f5b24006640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f5b2b200640 1 --2- 172.21.15.2:0/1688768177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b24053780 0x7f5b24053b60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f5b2b200640 1 -- 172.21.15.2:0/1688768177 >> 172.21.15.2:0/1688768177 conn(0x7f5b2403aa10 msgr2=0x7f5b2403ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7fa7f1c00640 1 -- start start 2024-01-15T17:57:41.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.162+0000 7f4375c00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43500039c0 0x7f4350003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f5b2b200640 1 -- 172.21.15.2:0/1688768177 shutdown_connections 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f5b2b200640 1 -- 172.21.15.2:0/1688768177 wait complete. 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4376600640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4350005180 0x7f4350014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4375200640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43500047c0 0x7f4350004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4375c00640 1 -- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43500047c0 msgr2=0x7f4350004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4375c00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43500047c0 0x7f4350004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4375c00640 1 -- 172.21.15.2:0/196995504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4350005180 msgr2=0x7f4350014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4375c00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4350005180 0x7f4350014190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.163+0000 7f4375c00640 1 -- 172.21.15.2:0/196995504 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700cf1d0 con 0x7f43500039c0 2024-01-15T17:57:41.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa136800640 1 Processor -- start 2024-01-15T17:57:41.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7f4376600640 1 -- 172.21.15.2:0/196995504 reap_dead start 2024-01-15T17:57:41.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7f1c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa7ec06cb80 0x7fa7ec166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa136800640 1 -- start start 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7eaa00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fa7ec15c0e0 0x7fa7ec0cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:39962/0 (socket says 172.21.15.2:39962) 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7f1c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec15cb00 con 0x7fa7ec15b7d0 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7ea000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa7ec15b7d0 0x7fa7ec06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35494/0 (socket says 172.21.15.2:35494) 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7ea000640 1 -- 172.21.15.2:0/3242586436 learned_addr learned my addr 172.21.15.2:0/3242586436 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec0ccec0 con 0x7fa7ec15c0e0 2024-01-15T17:57:41.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.165+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec0cccd0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7f5b2b200640 1 Processor -- start 2024-01-15T17:57:41.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7fa7eb400640 1 --2- 172.21.15.2:0/3242586436 >> v2:172.21.15.137:3300/0 conn(0x7fa7ec06cb80 0x7fa7ec166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7f5b2b200640 1 -- start start 2024-01-15T17:57:41.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7fd67b000640 1 Processor -- start 2024-01-15T17:57:41.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7fd67b000640 1 -- start start 2024-01-15T17:57:41.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7f4375c00640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43500039c0 0x7f4350003de0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4358009ac0 tx=0x7f43500057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60325a3e12bd7ad7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315315630 0 0) 0x7fa7ec0ccec0 con 0x7fa7ec15c0e0 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.166+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc003520 con 0x7fa7ec15c0e0 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462315017 0 0) 0x7fa7ec15cb00 con 0x7fa7ec15b7d0 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc003840 con 0x7fa7ec15b7d0 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7fa7eb400640 1 -- 172.21.15.2:0/3242586436 >> v1:172.21.15.137:6789/0 conn(0x7fa7ec15c0e0 legacy=0x7fa7ec0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7fa7eb400640 1 -- 172.21.15.2:0/3242586436 >> v1:172.21.15.2:6789/0 conn(0x7fa7ec15b7d0 legacy=0x7fa7ec06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7fa7eb400640 1 -- 172.21.15.2:0/3242586436 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec0cd1a0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f5b2b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 0x7f5b2404ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f5b2b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2403d0d0 0x7f5b240e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f5b2b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b24053780 0x7f5b240e4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fa136800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa1301469f0 0x7fa130071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fa136800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa1300b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fa136800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300ac870 con 0x7fa130146dd0 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fa136800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa13015d970 con 0x7fa1301469f0 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fa136800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa13015dc30 con 0x7fa13006cb80 2024-01-15T17:57:41.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f5b2b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24057670 con 0x7f5b2403d0d0 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f5b2b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24057e80 con 0x7f5b24053780 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f5b2b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240578c0 con 0x7f5b240061e0 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7f5b29e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 0x7f5b2404ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7f5b2a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b24053780 0x7f5b240e4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7f5b29e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 0x7f5b2404ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58816/0 (socket says 172.21.15.2:58816) 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7f5b29e00640 1 -- 172.21.15.2:0/2112250445 learned_addr learned my addr 172.21.15.2:0/2112250445 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fd67b000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd6741618e0 0x7fd6740d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fd67b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740d1e80 0x7fd6740cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fd67b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd674162280 con 0x7fd6740c5ff0 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fd67b000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6740d1580 con 0x7fd6741618e0 2024-01-15T17:57:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.168+0000 7fd67b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd674162560 con 0x7fd6740d1e80 2024-01-15T17:57:41.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435802c710 con 0x7f43500039c0 2024-01-15T17:57:41.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435802c870 con 0x7f43500039c0 2024-01-15T17:57:41.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.167+0000 7f436fe00640 1 -- 172.21.15.2:0/196995504 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435802cc40 con 0x7f43500039c0 2024-01-15T17:57:41.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f4378600640 1 -- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43500039c0 msgr2=0x7f4350003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f4378600640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43500039c0 0x7f4350003de0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4358009ac0 tx=0x7f43500057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7fd673e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd6741618e0 0x7fd6740d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7fd673e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd6741618e0 0x7fd6740d1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49962/0 (socket says 172.21.15.2:49962) 2024-01-15T17:57:41.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7fd673e00640 1 -- 172.21.15.2:0/1485190721 learned_addr learned my addr 172.21.15.2:0/1485190721 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f4378600640 1 -- 172.21.15.2:0/196995504 shutdown_connections 2024-01-15T17:57:41.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7fd679000640 1 --2- 172.21.15.2:0/1485190721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740d1e80 0x7fd6740cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f4378600640 1 --2- 172.21.15.2:0/196995504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43500039c0 0x7f4350003de0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4358009ac0 tx=0x7f43500057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f4378600640 1 -- 172.21.15.2:0/196995504 >> 172.21.15.2:0/196995504 conn(0x7f437008cc30 msgr2=0x7f437009c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f5b29400640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2403d0d0 0x7f5b240e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7fa7eb400640 1 --2- 172.21.15.2:0/3242586436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec166070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009d90 tx=0x7fa7e402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9692753878877b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f5b29e00640 1 -- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b24053780 msgr2=0x7f5b240e4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f5b29e00640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b24053780 0x7f5b240e4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7fa12ea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa130146dd0 0x7fa13006c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35500/0 (socket says 172.21.15.2:35500) 2024-01-15T17:57:41.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f5b29e00640 1 -- 172.21.15.2:0/2112250445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2403d0d0 msgr2=0x7f5b240e4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f5b29e00640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2403d0d0 0x7f5b240e4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.169+0000 7f5b29e00640 1 -- 172.21.15.2:0/2112250445 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14009400 con 0x7f5b240061e0 2024-01-15T17:57:41.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa12ea00640 1 -- 172.21.15.2:0/2751494226 learned_addr learned my addr 172.21.15.2:0/2751494226 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:41.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fd679000640 1 -- 172.21.15.2:0/1485190721 >> v2:172.21.15.2:3300/0 conn(0x7fd6741618e0 msgr2=0x7fd6740d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa12fe00640 1 --2- 172.21.15.2:0/2751494226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa1300b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa12f400640 1 --2- 172.21.15.2:0/2751494226 >> v2:172.21.15.2:3300/0 conn(0x7fa1301469f0 0x7fa130071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fd679000640 1 --2- 172.21.15.2:0/1485190721 >> v2:172.21.15.2:3300/0 conn(0x7fd6741618e0 0x7fd6740d1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fd679000640 1 -- 172.21.15.2:0/1485190721 >> v1:172.21.15.137:6789/0 conn(0x7fd6740c5ff0 legacy=0x7fd6740d1770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T17:57:41.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fd679000640 1 -- 172.21.15.2:0/1485190721 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd674162840 con 0x7fd6740d1e80 2024-01-15T17:57:41.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa12fe00640 1 -- 172.21.15.2:0/2751494226 >> v2:172.21.15.2:3300/0 conn(0x7fa1301469f0 msgr2=0x7fa130071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa12e000640 1 -- 172.21.15.2:0/2751494226 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978729523 0 0) 0x7fa1300ac870 con 0x7fa130146dd0 2024-01-15T17:57:41.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7f4378600640 1 -- 172.21.15.2:0/196995504 shutdown_connections 2024-01-15T17:57:41.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa12fe00640 1 --2- 172.21.15.2:0/2751494226 >> v2:172.21.15.2:3300/0 conn(0x7fa1301469f0 0x7fa130071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7f5b29e00640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 0x7f5b2404ee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c000c00 tx=0x7f5b1c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7f4378600640 1 -- 172.21.15.2:0/196995504 wait complete. 2024-01-15T17:57:41.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7fa12fe00640 1 -- 172.21.15.2:0/2751494226 >> v1:172.21.15.2:6789/0 conn(0x7fa130146dd0 legacy=0x7fa13006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7fa12fe00640 1 -- 172.21.15.2:0/2751494226 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa13015df10 con 0x7fa13006cb80 2024-01-15T17:57:41.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e402ccf0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e402ce50 con 0x7fa7ec06cb80 2024-01-15T17:57:41.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.170+0000 7fa7e9600640 1 -- 172.21.15.2:0/3242586436 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e4034660 con 0x7fa7ec06cb80 2024-01-15T17:57:41.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c03e040 con 0x7f5b240061e0 2024-01-15T17:57:41.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c039c90 con 0x7f5b240061e0 2024-01-15T17:57:41.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.171+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0443f0 con 0x7f5b240061e0 2024-01-15T17:57:41.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c0094d0 con 0x7f5b240061e0 2024-01-15T17:57:41.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7fd679000640 1 --2- 172.21.15.2:0/1485190721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740d1e80 0x7fd6740cdd70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6740d4030 tx=0x7fd65c02b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d13c2126405bafbe server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240e3930 con 0x7f5b240061e0 2024-01-15T17:57:41.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7fd672a00640 1 -- 172.21.15.2:0/1485190721 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031780 con 0x7fd6740d1e80 2024-01-15T17:57:41.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7fd672a00640 1 -- 172.21.15.2:0/1485190721 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c0318e0 con 0x7fd6740d1e80 2024-01-15T17:57:41.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7fd672a00640 1 -- 172.21.15.2:0/1485190721 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c031c70 con 0x7fd6740d1e80 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.172+0000 7fa12fe00640 1 --2- 172.21.15.2:0/2751494226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa1300b34a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1180097a0 tx=0x7fa11802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a6cc2f9cb38b436 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 msgr2=0x7fa7ec166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/3242586436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec166070 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa7e4009d90 tx=0x7fa7e402bf40 comp rx=0 tx=0).stop 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 shutdown_connections 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/3242586436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec166070 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 >> 172.21.15.2:0/3242586436 conn(0x7fa7ec06da20 msgr2=0x7fa7ec06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 shutdown_connections 2024-01-15T17:57:41.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7fa7f1c00640 1 -- 172.21.15.2:0/3242586436 wait complete. 2024-01-15T17:57:41.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7fd67b000640 1 -- 172.21.15.2:0/1485190721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740d1e80 msgr2=0x7fd6740cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7fd67b000640 1 --2- 172.21.15.2:0/1485190721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740d1e80 0x7fd6740cdd70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd6740d4030 tx=0x7fd65c02b820 comp rx=0 tx=0).stop 2024-01-15T17:57:41.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7facbea00640 1 Processor -- start 2024-01-15T17:57:41.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7facbea00640 1 -- start start 2024-01-15T17:57:41.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7f4378600640 1 Processor -- start 2024-01-15T17:57:41.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa12e000640 1 -- 172.21.15.2:0/2751494226 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11802d040 con 0x7fa13006cb80 2024-01-15T17:57:41.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.173+0000 7fa12e000640 1 -- 172.21.15.2:0/2751494226 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa118031810 con 0x7fa13006cb80 2024-01-15T17:57:41.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.174+0000 7fd67b000640 1 -- 172.21.15.2:0/1485190721 shutdown_connections 2024-01-15T17:57:41.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.175+0000 7fa12e000640 1 -- 172.21.15.2:0/2751494226 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118031ba0 con 0x7fa13006cb80 2024-01-15T17:57:41.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.175+0000 7f4378600640 1 -- start start 2024-01-15T17:57:41.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.175+0000 7fd67b000640 1 --2- 172.21.15.2:0/1485190721 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740d1e80 0x7fd6740cdd70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.175+0000 7fd67b000640 1 --2- 172.21.15.2:0/1485190721 >> v2:172.21.15.2:3300/0 conn(0x7fd6741618e0 0x7fd6740d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.175+0000 7fd67b000640 1 -- 172.21.15.2:0/1485190721 >> 172.21.15.2:0/1485190721 conn(0x7fd67408cc30 msgr2=0x7fd67409c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7fa136800640 1 -- 172.21.15.2:0/2751494226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 msgr2=0x7fa1300b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7fa136800640 1 --2- 172.21.15.2:0/2751494226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa1300b34a0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa1180097a0 tx=0x7fa11802b8b0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.175+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5afc005250 con 0x7f5b240061e0 2024-01-15T17:57:41.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7fd67b000640 1 -- 172.21.15.2:0/1485190721 shutdown_connections 2024-01-15T17:57:41.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7facbea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb806cb80 0x7facb8166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7facbea00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb815cb00 con 0x7facb815c0e0 2024-01-15T17:57:41.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7facbea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb80ccec0 con 0x7facb815b7d0 2024-01-15T17:57:41.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7facbea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb80cccd0 con 0x7facb806cb80 2024-01-15T17:57:41.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7fd67b000640 1 -- 172.21.15.2:0/1485190721 wait complete. 2024-01-15T17:57:41.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7facb7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7facb815c0e0 0x7facb80cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35504/0 (socket says 172.21.15.2:35504) 2024-01-15T17:57:41.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.176+0000 7facb7e00640 1 -- 172.21.15.2:0/1241377795 learned_addr learned my addr 172.21.15.2:0/1241377795 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:41.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849492236 0 0) 0x7facb815cb00 con 0x7facb815c0e0 2024-01-15T17:57:41.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac98003520 con 0x7facb815c0e0 2024-01-15T17:57:41.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7fa7f1c00640 1 Processor -- start 2024-01-15T17:57:41.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 13411922 0 0) 0x7fac98003520 con 0x7facb815c0e0 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb806cb80 msgr2=0x7facb8166070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb806cb80 0x7facb8166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 >> v1:172.21.15.137:6789/0 conn(0x7facb815b7d0 legacy=0x7facb806c470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb80cd1a0 con 0x7facb815c0e0 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7fa136800640 1 -- 172.21.15.2:0/2751494226 shutdown_connections 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7fa136800640 1 --2- 172.21.15.2:0/2751494226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa1300b34a0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7fa136800640 1 --2- 172.21.15.2:0/2751494226 >> v2:172.21.15.2:3300/0 conn(0x7fa1301469f0 0x7fa130071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7fa136800640 1 -- 172.21.15.2:0/2751494226 >> 172.21.15.2:0/2751494226 conn(0x7fa13006da20 msgr2=0x7fa13006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.177+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7faca8001670 con 0x7facb815c0e0 2024-01-15T17:57:41.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7fa136800640 1 -- 172.21.15.2:0/2751494226 shutdown_connections 2024-01-15T17:57:41.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7fa136800640 1 -- 172.21.15.2:0/2751494226 wait complete. 2024-01-15T17:57:41.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 >> v1:172.21.15.2:6789/0 conn(0x7facb815c0e0 legacy=0x7facb80cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7fac98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac980047c0 0x7fac98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac980052a0 0x7fac9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9800c0d0 con 0x7fac980039a0 2024-01-15T17:57:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac9800c240 con 0x7fac980047c0 2024-01-15T17:57:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.178+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9800c400 con 0x7fac980052a0 2024-01-15T17:57:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.179+0000 7fd67b000640 1 Processor -- start 2024-01-15T17:57:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.179+0000 7facb7e00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7fac98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.179+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5b1c044550 con 0x7f5b240061e0 2024-01-15T17:57:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7f4378600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f437014e6b0 0x7f4370154b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7f4378600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4370155060 0x7f43700b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7facb7e00640 1 -- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac980052a0 msgr2=0x7fac9800b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T17:57:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7facb7e00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac980052a0 0x7fac9800b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7facb7e00640 1 -- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac980047c0 msgr2=0x7fac98004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T17:57:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7facb7e00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac980047c0 0x7fac98004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7facb7e00640 1 -- 172.21.15.2:0/1241377795 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac98003520 con 0x7fac980039a0 2024-01-15T17:57:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7f5b22000640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b0406b310 0x7f5b0406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7f4378600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43700b4640 0x7f437015a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4375c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f437014e6b0 0x7f4370154b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4375c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f437014e6b0 0x7f4370154b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43578/0 (socket says 172.21.15.2:43578) 2024-01-15T17:57:41.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b1c0776a0 con 0x7f5b240061e0 2024-01-15T17:57:41.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4375c00640 1 -- 172.21.15.2:0/1106571775 learned_addr learned my addr 172.21.15.2:0/1106571775 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f5b29400640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b0406b310 0x7f5b0406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4375200640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4370155060 0x7f43700b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43701555c0 con 0x7f43700b4640 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4376600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43700b4640 0x7f437015a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43700b4ac0 con 0x7f437014e6b0 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43700c6880 con 0x7f4370155060 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7facb7e00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7fac98003da0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faca8001eb0 tx=0x7fac980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cd484e315703a15 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.180+0000 7fa136800640 1 Processor -- start 2024-01-15T17:57:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fd67b000640 1 -- start start 2024-01-15T17:57:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7f5b29400640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b0406b310 0x7f5b0406d7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c068da0 tx=0x7f5b2c08af90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca8005ad0 con 0x7fac980039a0 2024-01-15T17:57:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca8005c30 con 0x7fac980039a0 2024-01-15T17:57:41.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7facb6a00640 1 -- 172.21.15.2:0/1241377795 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca802cac0 con 0x7fac980039a0 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.181+0000 7fa7f1c00640 1 -- start start 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fa7f1c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec14f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fa7f1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ec15b7d0 0x7fa7ec14eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fa7f1c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ec15c0e0 0x7fa7ec0afc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fa7f1c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1f7630 con 0x7fa7ec15c0e0 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fa7f1c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec14f970 con 0x7fa7ec06cb80 2024-01-15T17:57:41.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.182+0000 7fa7f1c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec14fc50 con 0x7fa7ec15b7d0 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7fa7eaa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec14f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7fa7eaa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec14f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43590/0 (socket says 172.21.15.2:43590) 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7fa7eaa00640 1 -- 172.21.15.2:0/2497461793 learned_addr learned my addr 172.21.15.2:0/2497461793 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7facbea00640 1 -- 172.21.15.2:0/1241377795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 msgr2=0x7fac98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7facbea00640 1 --2- 172.21.15.2:0/1241377795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7fac98003da0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faca8001eb0 tx=0x7fac980057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7facbca00640 1 -- 172.21.15.2:0/1241377795 reap_dead start 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7facbea00640 1 -- 172.21.15.2:0/1241377795 shutdown_connections 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7facbea00640 1 -- 172.21.15.2:0/1241377795 >> 172.21.15.2:0/1241377795 conn(0x7facb806da20 msgr2=0x7facb806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7facbea00640 1 -- 172.21.15.2:0/1241377795 shutdown_connections 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.183+0000 7fa136800640 1 -- start start 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fa7eaa00640 1 -- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ec15b7d0 msgr2=0x7fa7ec14eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7f4376600640 1 -- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4370155060 msgr2=0x7f43700b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7f4376600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4370155060 0x7f43700b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7f4376600640 1 -- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f437014e6b0 msgr2=0x7f4370154b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7f4376600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f437014e6b0 0x7f4370154b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7f4376600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700c6b60 con 0x7f43700b4640 2024-01-15T17:57:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7facbea00640 1 -- 172.21.15.2:0/1241377795 wait complete. 2024-01-15T17:57:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fa7ea000640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ec15b7d0 0x7fa7ec14eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fa7eaa00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ec15b7d0 0x7fa7ec14eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fa7eaa00640 1 -- 172.21.15.2:0/2497461793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ec15c0e0 msgr2=0x7fa7ec0afc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fd67b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740c5ff0 0x7fd674160380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fa7eb400640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ec15c0e0 0x7fa7ec0afc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fd67b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 0x7fd6741f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fd67b000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6741618e0 0x7fd67409cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fd67b000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6740c5b10 con 0x7fd6741618e0 2024-01-15T17:57:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fd67b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6741f83a0 con 0x7fd6740d1e80 2024-01-15T17:57:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.184+0000 7fd67b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6740b6320 con 0x7fd6740c5ff0 2024-01-15T17:57:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7f4376600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43700b4640 0x7f437015a260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f43700d0d80 tx=0x7f436800e890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa7eaa00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ec15c0e0 0x7fa7ec0afc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa7eaa00640 1 -- 172.21.15.2:0/2497461793 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e4009a70 con 0x7fa7ec06cb80 2024-01-15T17:57:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa7ea000640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ec15b7d0 0x7fa7ec14eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa7eb400640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ec15c0e0 0x7fa7ec0afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7facbea00640 1 Processor -- start 2024-01-15T17:57:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa7eaa00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec14f390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400c3c0 tx=0x7fa7d400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa136800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa13013d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d40075a0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa136800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1301469f0 0x7fa130141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa136800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa130141f70 0x7fa1300aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa136800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa13013dec0 con 0x7fa1301469f0 2024-01-15T17:57:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa136800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1301ffab0 con 0x7fa130141f70 2024-01-15T17:57:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fa136800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1301ffd90 con 0x7fa13006cb80 2024-01-15T17:57:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7d40047e0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7d4004bd0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec14ff30 con 0x7fa7ec06cb80 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec14f0c0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa12f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa13013d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fa12f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa13013d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58878/0 (socket says 172.21.15.2:58878) 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12f400640 1 -- 172.21.15.2:0/2334829499 learned_addr learned my addr 172.21.15.2:0/2334829499 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12fe00640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa130141f70 0x7fa1300aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fd673400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 0x7fd6741f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fd673e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740c5ff0 0x7fd674160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4368003b80 con 0x7f43700b4640 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4368003d20 con 0x7f43700b4640 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4368005310 con 0x7f43700b4640 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fd673400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 0x7fd6741f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:43594/0 (socket says 172.21.15.2:43594) 2024-01-15T17:57:41.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fd673400640 1 -- 172.21.15.2:0/4237074256 learned_addr learned my addr 172.21.15.2:0/4237074256 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.185+0000 7fd679000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6741618e0 0x7fd67409cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fd673400640 1 -- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740c5ff0 msgr2=0x7fd674160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fd673400640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740c5ff0 0x7fd674160380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7fd673400640 1 -- 172.21.15.2:0/4237074256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6741618e0 msgr2=0x7fd67409cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12f400640 1 -- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa130141f70 msgr2=0x7fa1300aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12f400640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa130141f70 0x7fa1300aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12f400640 1 -- 172.21.15.2:0/2334829499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1301469f0 msgr2=0x7fa130141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12f400640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1301469f0 0x7fa130141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7fa12f400640 1 -- 172.21.15.2:0/2334829499 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa118009400 con 0x7fa13006cb80 2024-01-15T17:57:41.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec0686f0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7fd673400640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6741618e0 0x7fd67409cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7fd673400640 1 -- 172.21.15.2:0/4237074256 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c009400 con 0x7fd6740d1e80 2024-01-15T17:57:41.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7fd679000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6741618e0 0x7fd67409cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:41.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.186+0000 7facbea00640 1 -- start start 2024-01-15T17:57:41.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7fa12f400640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa13013d980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa120009db0 tx=0x7fa12000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43700c6e40 con 0x7f43700b4640 2024-01-15T17:57:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4370153e30 con 0x7f43700b4640 2024-01-15T17:57:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7facbea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7facb81464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7facbea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb8154850 0x7facb81489f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7facbea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb8148f30 0x7facb815ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7facbea00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8147950 con 0x7fac980039a0 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7facbea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb8147c30 con 0x7facb8148f30 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.187+0000 7facbea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb8147f10 con 0x7facb8154850 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7facb81464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7facb81464c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50028/0 (socket says 172.21.15.2:50028) 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 -- 172.21.15.2:0/3162199462 learned_addr learned my addr 172.21.15.2:0/3162199462 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 -- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb8154850 msgr2=0x7facb81489f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T17:57:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.189+0000 7fd673400640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 0x7fd6741f7470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b450 tx=0x7fd66400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb8154850 0x7facb81489f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 -- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb8148f30 msgr2=0x7facb815ec70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb8148f30 0x7facb815ec70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.188+0000 7facb7e00640 1 -- 172.21.15.2:0/3162199462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb81481f0 con 0x7fac980039a0 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.189+0000 7facb7e00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7facb81464c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faca8000c00 tx=0x7facb80ce740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.189+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca802cbd0 con 0x7fac980039a0 2024-01-15T17:57:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.190+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb81484d0 con 0x7fac980039a0 2024-01-15T17:57:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.190+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43700c69c0 con 0x7f43700b4640 2024-01-15T17:57:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.190+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb815f1b0 con 0x7fac980039a0 2024-01-15T17:57:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.190+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca802cd30 con 0x7fac980039a0 2024-01-15T17:57:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.190+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca802cea0 con 0x7fac980039a0 2024-01-15T17:57:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.190+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa7d4004d30 con 0x7fa7ec06cb80 2024-01-15T17:57:41.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.191+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4368005470 con 0x7f43700b4640 2024-01-15T17:57:41.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.192+0000 7fa7e2a00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7c406b2c0 0x7fa7c406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.192+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7d4089ff0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.192+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac88005250 con 0x7fac980039a0 2024-01-15T17:57:41.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.192+0000 7f436d600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f434806b2f0 0x7f434806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.193+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4368088f50 con 0x7f43700b4640 2024-01-15T17:57:41.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.193+0000 7fa7ea000640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7c406b2c0 0x7fa7c406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.193+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faca8032070 con 0x7fac980039a0 2024-01-15T17:57:41.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.193+0000 7f4375c00640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f434806b2f0 0x7f434806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.194+0000 7facafe00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac9006b2f0 0x7fac9006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.194+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faca80ab680 con 0x7fac980039a0 2024-01-15T17:57:41.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.194+0000 7f4375c00640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f434806b2f0 0x7f434806d7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f435802c840 tx=0x7f4358030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.194+0000 7fa7ea000640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7c406b2c0 0x7fa7c406d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec1530c0 tx=0x7fa7dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.195+0000 7facb7400640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac9006b2f0 0x7fac9006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.196+0000 7facb7400640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac9006b2f0 0x7fac9006d7b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7facb8153800 tx=0x7faca0006bd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43700c69c0 con 0x7f43700b4640 2024-01-15T17:57:41.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120014480 con 0x7fa13006cb80 2024-01-15T17:57:41.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12000f040 con 0x7fa13006cb80 2024-01-15T17:57:41.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120014800 con 0x7fa13006cb80 2024-01-15T17:57:41.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa130200070 con 0x7fa13006cb80 2024-01-15T17:57:41.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b1c0772d0 con 0x7f5b240061e0 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.198+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1301ff7a0 con 0x7fa13006cb80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.200+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1300686f0 con 0x7fa13006cb80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.203+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa120014960 con 0x7fa13006cb80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.204+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664011890 con 0x7fd6740d1e80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.204+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664011a10 con 0x7fd6740d1e80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.204+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664011dc0 con 0x7fd6740d1e80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.204+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6741f7de0 con 0x7fd6740d1e80 2024-01-15T17:57:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.205+0000 7fa127400640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa10806b390 0x7fa10806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.205+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa12008af50 con 0x7fa13006cb80 2024-01-15T17:57:41.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.205+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7d4054ef0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.206+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6740b5fb0 con 0x7fd6740d1e80 2024-01-15T17:57:41.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.206+0000 7fa12ea00640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa10806b390 0x7fa10806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.206+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faca80763c0 con 0x7fac980039a0 2024-01-15T17:57:41.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.209+0000 7fa12ea00640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa10806b390 0x7fa10806d850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa128007fd0 tx=0x7fa128007960 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.207+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6740b5cf0 con 0x7fd6740d1e80 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.210+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd66401a400 con 0x7fd6740d1e80 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.211+0000 7fd66be00640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd64c06b2c0 0x7fd64c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.212+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd66408a9a0 con 0x7fd6740d1e80 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.212+0000 7fd673e00640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd64c06b2c0 0x7fd64c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.213+0000 7fd673e00640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd64c06b2c0 0x7fd64c06d780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd66c011340 tx=0x7fd66c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.216+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa120055da0 con 0x7fa13006cb80 2024-01-15T17:57:41.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.216+0000 7effb6400640 1 Processor -- start 2024-01-15T17:57:41.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.217+0000 7effb6400640 1 -- start start 2024-01-15T17:57:41.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.219+0000 7effb6400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7effb015b7d0 0x7effb006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.219+0000 7effb6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7effb015c4c0 0x7effb006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.219+0000 7effb6400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb015cb00 con 0x7effb015c0e0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.219+0000 7effb6400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00cccd0 con 0x7effb015b7d0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.219+0000 7effb6400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb00ccec0 con 0x7effb015c4c0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.220+0000 7effafe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7effb015c4c0 0x7effb006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.220+0000 7effaf400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7effb015c0e0 0x7effb00cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:40000/0 (socket says 172.21.15.2:40000) 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.220+0000 7effaf400640 1 -- 172.21.15.2:0/1119322573 learned_addr learned my addr 172.21.15.2:0/1119322573 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.220+0000 7effaea00640 1 --2- 172.21.15.2:0/1119322573 >> v2:172.21.15.2:3300/0 conn(0x7effb015b7d0 0x7effb006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.221+0000 7effaea00640 1 -- 172.21.15.2:0/1119322573 >> v2:172.21.15.137:3300/0 conn(0x7effb015c4c0 msgr2=0x7effb006c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.221+0000 7effaea00640 1 --2- 172.21.15.2:0/1119322573 >> v2:172.21.15.137:3300/0 conn(0x7effb015c4c0 0x7effb006c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.221+0000 7effaea00640 1 -- 172.21.15.2:0/1119322573 >> v1:172.21.15.137:6789/0 conn(0x7effb015c0e0 legacy=0x7effb00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.221+0000 7effaea00640 1 -- 172.21.15.2:0/1119322573 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb00cd1a0 con 0x7effb015b7d0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.223+0000 7effaea00640 1 --2- 172.21.15.2:0/1119322573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb006c470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7effb0152750 tx=0x7eff9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1acc688e58c941b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.223+0000 7effae000640 1 -- 172.21.15.2:0/1119322573 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98031870 con 0x7effb015b7d0 2024-01-15T17:57:41.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.224+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd664055880 con 0x7fd6740d1e80 2024-01-15T17:57:41.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.223+0000 7effae000640 1 -- 172.21.15.2:0/1119322573 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff980319d0 con 0x7effb015b7d0 2024-01-15T17:57:41.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.223+0000 7effae000640 1 -- 172.21.15.2:0/1119322573 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff98031d80 con 0x7effb015b7d0 2024-01-15T17:57:41.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.225+0000 7effb6400640 1 -- 172.21.15.2:0/1119322573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 msgr2=0x7effb006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.225+0000 7effb6400640 1 --2- 172.21.15.2:0/1119322573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb006c470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7effb0152750 tx=0x7eff9802b910 comp rx=0 tx=0).stop 2024-01-15T17:57:41.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.225+0000 7effb6400640 1 -- 172.21.15.2:0/1119322573 shutdown_connections 2024-01-15T17:57:41.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.225+0000 7effb6400640 1 --2- 172.21.15.2:0/1119322573 >> v2:172.21.15.137:3300/0 conn(0x7effb015c4c0 0x7effb006c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.225+0000 7effb6400640 1 --2- 172.21.15.2:0/1119322573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb006c470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.225+0000 7effb6400640 1 -- 172.21.15.2:0/1119322573 >> 172.21.15.2:0/1119322573 conn(0x7effb006da20 msgr2=0x7effb006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.226+0000 7effb6400640 1 -- 172.21.15.2:0/1119322573 shutdown_connections 2024-01-15T17:57:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.226+0000 7effb6400640 1 -- 172.21.15.2:0/1119322573 wait complete. 2024-01-15T17:57:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.229+0000 7effb6400640 1 Processor -- start 2024-01-15T17:57:41.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.229+0000 7effb6400640 1 -- start start 2024-01-15T17:57:41.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.231+0000 7effb6400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb0152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.231+0000 7effb6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effb015c0e0 0x7effb0152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.231+0000 7effb6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb015c4c0 0x7effb0153b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.232+0000 7effb6400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b2cc0 con 0x7effb015b7d0 2024-01-15T17:57:41.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.232+0000 7effb6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b2fa0 con 0x7effb015c4c0 2024-01-15T17:57:41.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.232+0000 7effb6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb00b3280 con 0x7effb015c0e0 2024-01-15T17:57:41.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effaf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb0152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effaea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effb015c0e0 0x7effb0152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effaea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effb015c0e0 0x7effb0152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58886/0 (socket says 172.21.15.2:58886) 2024-01-15T17:57:41.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effafe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb015c4c0 0x7effb0153b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effaea00640 1 -- 172.21.15.2:0/1141494495 learned_addr learned my addr 172.21.15.2:0/1141494495 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effafe00640 1 -- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effb015c0e0 msgr2=0x7effb0152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.233+0000 7effafe00640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effb015c0e0 0x7effb0152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.234+0000 7effafe00640 1 -- 172.21.15.2:0/1141494495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 msgr2=0x7effb0152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.234+0000 7effafe00640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb0152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.234+0000 7effafe00640 1 -- 172.21.15.2:0/1141494495 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff98009400 con 0x7effb015c4c0 2024-01-15T17:57:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.234+0000 7effaf400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb0152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:41.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.235+0000 7effafe00640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb015c4c0 0x7effb0153b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7effa8007cf0 tx=0x7effa800e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.236+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8018070 con 0x7effb015c4c0 2024-01-15T17:57:41.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.237+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa800edd0 con 0x7effb015c4c0 2024-01-15T17:57:41.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.237+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effa8003bb0 con 0x7effb015c4c0 2024-01-15T17:57:41.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.237+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb00b3560 con 0x7effb015c4c0 2024-01-15T17:57:41.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.237+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb0153280 con 0x7effb015c4c0 2024-01-15T17:57:41.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.239+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb00686f0 con 0x7effb015c4c0 2024-01-15T17:57:41.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.241+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7effa8014420 con 0x7effb015c4c0 2024-01-15T17:57:41.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.243+0000 7effa7400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff8806b390 0x7eff8806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.243+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7effa808ac80 con 0x7effb015c4c0 2024-01-15T17:57:41.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.244+0000 7effaf400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff8806b390 0x7eff8806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.245+0000 7effaf400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff8806b390 0x7eff8806d850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7effa0007fd0 tx=0x7effa0007e80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.254+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effa8055ad0 con 0x7effb015c4c0 2024-01-15T17:57:41.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.575+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f526004da10 con 0x7f52600dbb50 2024-01-15T17:57:41.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.576+0000 7f5265600640 1 -- 172.21.15.2:0/4117274392 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f525800f040 con 0x7f52600dbb50 2024-01-15T17:57:41.580 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:41.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.579+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006b2c0 msgr2=0x7f524006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.579+0000 7f5270600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006b2c0 0x7f524006d780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5268069de0 tx=0x7f526808af90 comp rx=0 tx=0).stop 2024-01-15T17:57:41.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.579+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 msgr2=0x7f52600dbfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.579+0000 7f5270600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 0x7f52600dbfd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5258009c20 tx=0x7f5258004730 comp rx=0 tx=0).stop 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 shutdown_connections 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006b2c0 0x7f524006d780 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52600dbb50 0x7f52600dbfd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52600da420 0x7f5260177720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 --2- 172.21.15.2:0/4117274392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5248007230 0x7f5260174910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 >> 172.21.15.2:0/4117274392 conn(0x7f5260009fb0 msgr2=0x7f5260038cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 shutdown_connections 2024-01-15T17:57:41.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.580+0000 7f5270600640 1 -- 172.21.15.2:0/4117274392 wait complete. 2024-01-15T17:57:41.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.594+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5afc005500 con 0x7f5b240061e0 2024-01-15T17:57:41.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.598+0000 7f5b22000640 1 -- 172.21.15.2:0/2112250445 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5b1c083430 con 0x7f5b240061e0 2024-01-15T17:57:41.604 INFO:teuthology.orchestra.run.smithi002.stdout:60129542150 2024-01-15T17:57:41.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.601+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b0406b310 msgr2=0x7f5b0406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.601+0000 7f5b2b200640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b0406b310 0x7f5b0406d7d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c068da0 tx=0x7f5b2c08af90 comp rx=0 tx=0).stop 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.601+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 msgr2=0x7f5b2404ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.601+0000 7f5b2b200640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 0x7f5b2404ee70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c000c00 tx=0x7f5b1c004730 comp rx=0 tx=0).stop 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.601+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 shutdown_connections 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b0406b310 0x7f5b0406d7d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b24053780 0x7f5b240e4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2403d0d0 0x7f5b240e4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 --2- 172.21.15.2:0/2112250445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b240061e0 0x7f5b2404ee70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 >> 172.21.15.2:0/2112250445 conn(0x7f5b2403aa10 msgr2=0x7f5b24039fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 shutdown_connections 2024-01-15T17:57:41.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.602+0000 7f5b2b200640 1 -- 172.21.15.2:0/2112250445 wait complete. 2024-01-15T17:57:41.613 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.0 2024-01-15T17:57:41.613 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.618+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa7ec0b40d0 con 0x7fa7ec06cb80 2024-01-15T17:57:41.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.619+0000 7fa7e2a00640 1 -- 172.21.15.2:0/2497461793 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7d4058510 con 0x7fa7ec06cb80 2024-01-15T17:57:41.623 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:41.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.621+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f43700c7360 con 0x7f43700b4640 2024-01-15T17:57:41.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.621+0000 7f436d600640 1 -- 172.21.15.2:0/1106571775 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f43700c7360 con 0x7f43700b4640 2024-01-15T17:57:41.625 INFO:teuthology.orchestra.run.smithi002.stdout:60129542151 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7c406b2c0 msgr2=0x7fa7c406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7c406b2c0 0x7fa7c406d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa7ec1530c0 tx=0x7fa7dc009040 comp rx=0 tx=0).stop 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 msgr2=0x7fa7ec14f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec14f390 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7d400c3c0 tx=0x7fa7d400c880 comp rx=0 tx=0).stop 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 shutdown_connections 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7c406b2c0 0x7fa7c406d780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ec15c0e0 0x7fa7ec0afc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ec15b7d0 0x7fa7ec14eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 --2- 172.21.15.2:0/2497461793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ec06cb80 0x7fa7ec14f390 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 >> 172.21.15.2:0/2497461793 conn(0x7fa7ec06da20 msgr2=0x7fa7ec06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 shutdown_connections 2024-01-15T17:57:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.623+0000 7fa7f1c00640 1 -- 172.21.15.2:0/2497461793 wait complete. 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f434806b2f0 msgr2=0x7f434806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f434806b2f0 0x7f434806d7b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f435802c840 tx=0x7f4358030000 comp rx=0 tx=0).stop 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43700b4640 msgr2=0x7f437015a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43700b4640 0x7f437015a260 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f43700d0d80 tx=0x7f436800e890 comp rx=0 tx=0).stop 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 shutdown_connections 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f434806b2f0 0x7f434806d7b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43700b4640 0x7f437015a260 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4370155060 0x7f43700b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 --2- 172.21.15.2:0/1106571775 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f437014e6b0 0x7f4370154b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 >> 172.21.15.2:0/1106571775 conn(0x7f437008cc30 msgr2=0x7f43700b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 shutdown_connections 2024-01-15T17:57:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.625+0000 7f4378600640 1 -- 172.21.15.2:0/1106571775 wait complete. 2024-01-15T17:57:41.631 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.1 2024-01-15T17:57:41.631 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.634+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa1301ffed0 con 0x7fa13006cb80 2024-01-15T17:57:41.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.635+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fac88005500 con 0x7fac980039a0 2024-01-15T17:57:41.639 INFO:teuthology.orchestra.run.smithi002.stdout:60129542150 2024-01-15T17:57:41.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.635+0000 7fa127400640 1 -- 172.21.15.2:0/2334829499 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa1301ffed0 con 0x7fa13006cb80 2024-01-15T17:57:41.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.636+0000 7facafe00640 1 -- 172.21.15.2:0/3162199462 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faca80799e0 con 0x7fac980039a0 2024-01-15T17:57:41.640 INFO:teuthology.orchestra.run.smithi002.stdout:60129542149 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.637+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa10806b390 msgr2=0x7fa10806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.637+0000 7fa136800640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa10806b390 0x7fa10806d850 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa128007fd0 tx=0x7fa128007960 comp rx=0 tx=0).stop 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 msgr2=0x7fa13013d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa13013d980 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa120009db0 tx=0x7fa12000c490 comp rx=0 tx=0).stop 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 shutdown_connections 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa10806b390 0x7fa10806d850 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa130141f70 0x7fa1300aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa1301469f0 0x7fa130141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 --2- 172.21.15.2:0/2334829499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa13006cb80 0x7fa13013d980 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 >> 172.21.15.2:0/2334829499 conn(0x7fa13006da20 msgr2=0x7fa130145a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 shutdown_connections 2024-01-15T17:57:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.638+0000 7fa136800640 1 -- 172.21.15.2:0/2334829499 wait complete. 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac9006b2f0 msgr2=0x7fac9006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac9006b2f0 0x7fac9006d7b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7facb8153800 tx=0x7faca0006bd0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 msgr2=0x7facb81464c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7facb81464c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7faca8000c00 tx=0x7facb80ce740 comp rx=0 tx=0).stop 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 shutdown_connections 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac9006b2f0 0x7fac9006d7b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facb8148f30 0x7facb815ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facb8154850 0x7facb81489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 --2- 172.21.15.2:0/3162199462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac980039a0 0x7facb81464c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.640+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 >> 172.21.15.2:0/3162199462 conn(0x7facb806da20 msgr2=0x7facb80b3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.641+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 shutdown_connections 2024-01-15T17:57:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.641+0000 7facbea00640 1 -- 172.21.15.2:0/3162199462 wait complete. 2024-01-15T17:57:41.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.7 2024-01-15T17:57:41.651 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.649+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd674067970 con 0x7fd6740d1e80 2024-01-15T17:57:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.650+0000 7fd66be00640 1 -- 172.21.15.2:0/4237074256 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd664015090 con 0x7fd6740d1e80 2024-01-15T17:57:41.654 INFO:teuthology.orchestra.run.smithi002.stdout:60129542150 2024-01-15T17:57:41.656 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.3 2024-01-15T17:57:41.656 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd64c06b2c0 msgr2=0x7fd64c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd64c06b2c0 0x7fd64c06d780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd66c011340 tx=0x7fd66c00b040 comp rx=0 tx=0).stop 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 msgr2=0x7fd6741f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 0x7fd6741f7470 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd66400b450 tx=0x7fd66400b910 comp rx=0 tx=0).stop 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 shutdown_connections 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd64c06b2c0 0x7fd64c06d780 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6741618e0 0x7fd67409cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6740d1e80 0x7fd6741f7470 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 --2- 172.21.15.2:0/4237074256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6740c5ff0 0x7fd674160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 >> 172.21.15.2:0/4237074256 conn(0x7fd67408cc30 msgr2=0x7fd6740c8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.653+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 shutdown_connections 2024-01-15T17:57:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.654+0000 7fd67b000640 1 -- 172.21.15.2:0/4237074256 wait complete. 2024-01-15T17:57:41.659 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.2 2024-01-15T17:57:41.660 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.6 2024-01-15T17:57:41.663 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.660+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7effb00b30e0 con 0x7effb015c4c0 2024-01-15T17:57:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.661+0000 7effa7400640 1 -- 172.21.15.2:0/1141494495 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7effb00b30e0 con 0x7effb015c4c0 2024-01-15T17:57:41.664 INFO:teuthology.orchestra.run.smithi002.stdout:60129542151 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff8806b390 msgr2=0x7eff8806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff8806b390 0x7eff8806d850 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7effa0007fd0 tx=0x7effa0007e80 comp rx=0 tx=0).stop 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb015c4c0 msgr2=0x7effb0153b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb015c4c0 0x7effb0153b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7effa8007cf0 tx=0x7effa800e4b0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 shutdown_connections 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff8806b390 0x7eff8806d850 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effb015c4c0 0x7effb0153b20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effb015c0e0 0x7effb0152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 --2- 172.21.15.2:0/1141494495 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effb015b7d0 0x7effb0152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 >> 172.21.15.2:0/1141494495 conn(0x7effb006da20 msgr2=0x7effb00aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 shutdown_connections 2024-01-15T17:57:41.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.663+0000 7effb6400640 1 -- 172.21.15.2:0/1141494495 wait complete. 2024-01-15T17:57:41.674 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.4 2024-01-15T17:57:41.674 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.682 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.5 2024-01-15T17:57:41.682 DEBUG:teuthology.parallel:result is None 2024-01-15T17:57:41.682 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T17:57:41.682 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:57:41.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.936+0000 7f7d5be00640 1 Processor -- start 2024-01-15T17:57:41.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.936+0000 7f7d5be00640 1 -- start start 2024-01-15T17:57:41.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d5be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 0x7f7d5c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d5be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7d5c06cad0 0x7f7d5c06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d5be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d5c0716d0 con 0x7f7d5c141880 2024-01-15T17:57:41.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d5be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d5c071f20 con 0x7f7d5c06cad0 2024-01-15T17:57:41.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d5be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c06c6b0 con 0x7f7d5c15ec60 2024-01-15T17:57:41.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d53e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 0x7f7d5c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d53e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 0x7f7d5c15f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58890/0 (socket says 172.21.15.2:58890) 2024-01-15T17:57:41.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d53e00640 1 -- 172.21.15.2:0/4134159519 learned_addr learned my addr 172.21.15.2:0/4134159519 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.938+0000 7f7d5b400640 1 --2- 172.21.15.2:0/4134159519 >> v2:172.21.15.137:3300/0 conn(0x7f7d5c06cad0 0x7f7d5c06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.939+0000 7f7d53e00640 1 -- 172.21.15.2:0/4134159519 >> v2:172.21.15.137:3300/0 conn(0x7f7d5c06cad0 msgr2=0x7f7d5c06cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:41.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.939+0000 7f7d53e00640 1 --2- 172.21.15.2:0/4134159519 >> v2:172.21.15.137:3300/0 conn(0x7f7d5c06cad0 0x7f7d5c06cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.939+0000 7f7d53e00640 1 -- 172.21.15.2:0/4134159519 >> v1:172.21.15.137:6789/0 conn(0x7f7d5c141880 legacy=0x7f7d5c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.939+0000 7f7d53e00640 1 -- 172.21.15.2:0/4134159519 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c071960 con 0x7f7d5c15ec60 2024-01-15T17:57:41.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.940+0000 7f7d53e00640 1 --2- 172.21.15.2:0/4134159519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 0x7f7d5c15f0e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009870 tx=0x7f7d4402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a2a96827b785b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.941+0000 7f7d5a000640 1 -- 172.21.15.2:0/4134159519 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44031640 con 0x7f7d5c15ec60 2024-01-15T17:57:41.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.941+0000 7f7d5a000640 1 -- 172.21.15.2:0/4134159519 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d440317a0 con 0x7f7d5c15ec60 2024-01-15T17:57:41.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.941+0000 7f7d5a000640 1 -- 172.21.15.2:0/4134159519 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d44031b30 con 0x7f7d5c15ec60 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 -- 172.21.15.2:0/4134159519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 msgr2=0x7f7d5c15f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 --2- 172.21.15.2:0/4134159519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 0x7f7d5c15f0e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009870 tx=0x7f7d4402b8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 -- 172.21.15.2:0/4134159519 shutdown_connections 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 --2- 172.21.15.2:0/4134159519 >> v2:172.21.15.137:3300/0 conn(0x7f7d5c06cad0 0x7f7d5c06cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 --2- 172.21.15.2:0/4134159519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c15ec60 0x7f7d5c15f0e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 -- 172.21.15.2:0/4134159519 >> 172.21.15.2:0/4134159519 conn(0x7f7d5c0b0840 msgr2=0x7f7d5c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:41.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 -- 172.21.15.2:0/4134159519 shutdown_connections 2024-01-15T17:57:41.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.942+0000 7f7d5be00640 1 -- 172.21.15.2:0/4134159519 wait complete. 2024-01-15T17:57:41.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.944+0000 7f7d5be00640 1 Processor -- start 2024-01-15T17:57:41.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.944+0000 7f7d5be00640 1 -- start start 2024-01-15T17:57:41.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.945+0000 7f7d5be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 0x7f7d5c1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.945+0000 7f7d5be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c141880 0x7f7d5c1f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.945+0000 7f7d5be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d5c15ec60 0x7f7d5c1542b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.945+0000 7f7d5be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c155ea0 con 0x7f7d5c06cad0 2024-01-15T17:57:41.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.945+0000 7f7d5be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c1f5b30 con 0x7f7d5c15ec60 2024-01-15T17:57:41.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.945+0000 7f7d5be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c156440 con 0x7f7d5c141880 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 0x7f7d5c1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 0x7f7d5c1568d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50050/0 (socket says 172.21.15.2:50050) 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 -- 172.21.15.2:0/103079329 learned_addr learned my addr 172.21.15.2:0/103079329 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d53e00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c141880 0x7f7d5c1f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5b400640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d5c15ec60 0x7f7d5c1542b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 -- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c141880 msgr2=0x7f7d5c1f4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c141880 0x7f7d5c1f4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 -- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d5c15ec60 msgr2=0x7f7d5c1542b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d5c15ec60 0x7f7d5c1542b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5aa00640 1 -- 172.21.15.2:0/103079329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d440094d0 con 0x7f7d5c06cad0 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d53e00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c141880 0x7f7d5c1f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.946+0000 7f7d5b400640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d5c15ec60 0x7f7d5c1542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.947+0000 7f7d5aa00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 0x7f7d5c1568d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c00c430 tx=0x7f7d4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.947+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c0052d0 con 0x7f7d5c06cad0 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.947+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4c005430 con 0x7f7d5c06cad0 2024-01-15T17:57:41.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.947+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4c003830 con 0x7f7d5c06cad0 2024-01-15T17:57:41.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.947+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c1f5570 con 0x7f7d5c06cad0 2024-01-15T17:57:41.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.947+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c1560d0 con 0x7f7d5c06cad0 2024-01-15T17:57:41.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.949+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d5c074770 con 0x7f7d5c06cad0 2024-01-15T17:57:41.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.950+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d4c003990 con 0x7f7d5c06cad0 2024-01-15T17:57:41.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.950+0000 7f7d52a00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d3406b2c0 0x7f7d3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:41.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.951+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d4c089140 con 0x7f7d5c06cad0 2024-01-15T17:57:41.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.951+0000 7f7d53e00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d3406b2c0 0x7f7d3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:41.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.952+0000 7f7d53e00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d3406b2c0 0x7f7d3406d780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7d44036000 tx=0x7f7d4403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:41.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:41.955+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d4c053fa0 con 0x7f7d5c06cad0 2024-01-15T17:57:42.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.143+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d5c141e80 con 0x7f7d3406b2c0 2024-01-15T17:57:42.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.164+0000 7f7d52a00640 1 -- 172.21.15.2:0/103079329 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264464 (secure 0 0 0) 0x7f7d44044400 con 0x7f7d3406b2c0 2024-01-15T17:57:42.172 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:42.173 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d3406b2c0 msgr2=0x7f7d3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d3406b2c0 0x7f7d3406d780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7d44036000 tx=0x7f7d4403c040 comp rx=0 tx=0).stop 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 msgr2=0x7f7d5c1568d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 0x7f7d5c1568d0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7d4c00c430 tx=0x7f7d4c00c8f0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 shutdown_connections 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d3406b2c0 0x7f7d3406d780 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d5c15ec60 0x7f7d5c1542b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d5c141880 0x7f7d5c1f4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 --2- 172.21.15.2:0/103079329 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d5c06cad0 0x7f7d5c1568d0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 >> 172.21.15.2:0/103079329 conn(0x7f7d5c0b0840 msgr2=0x7f7d5c0af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 shutdown_connections 2024-01-15T17:57:42.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.169+0000 7f7d5be00640 1 -- 172.21.15.2:0/103079329 wait complete. 2024-01-15T17:57:42.190 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-15T17:57:41.371769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236132,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738588,"statfs":{"total":773094113280,"available":772852314112,"internally_reserved":0,"allocated":3055616,"data_stored":1815972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.264117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139938+0000","last_change":"2024-01-15T17:57:32.145196+0000","last_active":"2024-01-15T17:57:33.139938+0000","last_peered":"2024-01-15T17:57:33.139938+0000","last_clean":"2024-01-15T17:57:33.139938+0000","last_became_active":"2024-01-15T17:57:32.144373+0000","last_became_peered":"2024-01-15T17:57:32.144373+0000","last_unstale":"2024-01-15T17:57:33.139938+0000","last_undegraded":"2024-01-15T17:57:33.139938+0000","last_fullsized":"2024-01-15T17:57:33.139938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136557+0000","last_change":"2024-01-15T17:57:32.143595+0000","last_active":"2024-01-15T17:57:33.136557+0000","last_peered":"2024-01-15T17:57:33.136557+0000","last_clean":"2024-01-15T17:57:33.136557+0000","last_became_active":"2024-01-15T17:57:32.143249+0000","last_became_peered":"2024-01-15T17:57:32.143249+0000","last_unstale":"2024-01-15T17:57:33.136557+0000","last_undegraded":"2024-01-15T17:57:33.136557+0000","last_fullsized":"2024-01-15T17:57:33.136557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419213+0000","last_change":"2024-01-15T17:57:32.143090+0000","last_active":"2024-01-15T17:57:33.419213+0000","last_peered":"2024-01-15T17:57:33.419213+0000","last_clean":"2024-01-15T17:57:33.419213+0000","last_became_active":"2024-01-15T17:57:32.142828+0000","last_became_peered":"2024-01-15T17:57:32.142828+0000","last_unstale":"2024-01-15T17:57:33.419213+0000","last_undegraded":"2024-01-15T17:57:33.419213+0000","last_fullsized":"2024-01-15T17:57:33.419213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139101+0000","last_change":"2024-01-15T17:57:32.146690+0000","last_active":"2024-01-15T17:57:33.139101+0000","last_peered":"2024-01-15T17:57:33.139101+0000","last_clean":"2024-01-15T17:57:33.139101+0000","last_became_active":"2024-01-15T17:57:32.142878+0000","last_became_peered":"2024-01-15T17:57:32.142878+0000","last_unstale":"2024-01-15T17:57:33.139101+0000","last_undegraded":"2024-01-15T17:57:33.139101+0000","last_fullsized":"2024-01-15T17:57:33.139101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141233+0000","last_change":"2024-01-15T17:57:32.145000+0000","last_active":"2024-01-15T17:57:33.141233+0000","last_peered":"2024-01-15T17:57:33.141233+0000","last_clean":"2024-01-15T17:57:33.141233+0000","last_became_active":"2024-01-15T17:57:32.144715+0000","last_became_peered":"2024-01-15T17:57:32.144715+0000","last_unstale":"2024-01-15T17:57:33.141233+0000","last_undegraded":"2024-01-15T17:57:33.141233+0000","last_fullsized":"2024-01-15T17:57:33.141233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:03.467001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138047+0000","last_change":"2024-01-15T17:57:32.154833+0000","last_active":"2024-01-15T17:57:33.138047+0000","last_peered":"2024-01-15T17:57:33.138047+0000","last_clean":"2024-01-15T17:57:33.138047+0000","last_became_active":"2024-01-15T17:57:32.153996+0000","last_became_peered":"2024-01-15T17:57:32.153996+0000","last_unstale":"2024-01-15T17:57:33.138047+0000","last_undegraded":"2024-01-15T17:57:33.138047+0000","last_fullsized":"2024-01-15T17:57:33.138047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138900+0000","last_change":"2024-01-15T17:57:32.147665+0000","last_active":"2024-01-15T17:57:33.138900+0000","last_peered":"2024-01-15T17:57:33.138900+0000","last_clean":"2024-01-15T17:57:33.138900+0000","last_became_active":"2024-01-15T17:57:32.147333+0000","last_became_peered":"2024-01-15T17:57:32.147333+0000","last_unstale":"2024-01-15T17:57:33.138900+0000","last_undegraded":"2024-01-15T17:57:33.138900+0000","last_fullsized":"2024-01-15T17:57:33.138900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415231+0000","last_change":"2024-01-15T17:57:32.144764+0000","last_active":"2024-01-15T17:57:33.415231+0000","last_peered":"2024-01-15T17:57:33.415231+0000","last_clean":"2024-01-15T17:57:33.415231+0000","last_became_active":"2024-01-15T17:57:32.144315+0000","last_became_peered":"2024-01-15T17:57:32.144315+0000","last_unstale":"2024-01-15T17:57:33.415231+0000","last_undegraded":"2024-01-15T17:57:33.415231+0000","last_fullsized":"2024-01-15T17:57:33.415231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140177+0000","last_change":"2024-01-15T17:57:32.157864+0000","last_active":"2024-01-15T17:57:33.140177+0000","last_peered":"2024-01-15T17:57:33.140177+0000","last_clean":"2024-01-15T17:57:33.140177+0000","last_became_active":"2024-01-15T17:57:32.157556+0000","last_became_peered":"2024-01-15T17:57:32.157556+0000","last_unstale":"2024-01-15T17:57:33.140177+0000","last_undegraded":"2024-01-15T17:57:33.140177+0000","last_fullsized":"2024-01-15T17:57:33.140177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415132+0000","last_change":"2024-01-15T17:57:32.161393+0000","last_active":"2024-01-15T17:57:33.415132+0000","last_peered":"2024-01-15T17:57:33.415132+0000","last_clean":"2024-01-15T17:57:33.415132+0000","last_became_active":"2024-01-15T17:57:32.161079+0000","last_became_peered":"2024-01-15T17:57:32.161079+0000","last_unstale":"2024-01-15T17:57:33.415132+0000","last_undegraded":"2024-01-15T17:57:33.415132+0000","last_fullsized":"2024-01-15T17:57:33.415132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414932+0000","last_change":"2024-01-15T17:57:32.145780+0000","last_active":"2024-01-15T17:57:33.414932+0000","last_peered":"2024-01-15T17:57:33.414932+0000","last_clean":"2024-01-15T17:57:33.414932+0000","last_became_active":"2024-01-15T17:57:32.145409+0000","last_became_peered":"2024-01-15T17:57:32.145409+0000","last_unstale":"2024-01-15T17:57:33.414932+0000","last_undegraded":"2024-01-15T17:57:33.414932+0000","last_fullsized":"2024-01-15T17:57:33.414932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186642+0000","last_change":"2024-01-15T17:57:32.144056+0000","last_active":"2024-01-15T17:57:33.186642+0000","last_peered":"2024-01-15T17:57:33.186642+0000","last_clean":"2024-01-15T17:57:33.186642+0000","last_became_active":"2024-01-15T17:57:32.143687+0000","last_became_peered":"2024-01-15T17:57:32.143687+0000","last_unstale":"2024-01-15T17:57:33.186642+0000","last_undegraded":"2024-01-15T17:57:33.186642+0000","last_fullsized":"2024-01-15T17:57:33.186642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417307+0000","last_change":"2024-01-15T17:57:32.141036+0000","last_active":"2024-01-15T17:57:33.417307+0000","last_peered":"2024-01-15T17:57:33.417307+0000","last_clean":"2024-01-15T17:57:33.417307+0000","last_became_active":"2024-01-15T17:57:32.139707+0000","last_became_peered":"2024-01-15T17:57:32.139707+0000","last_unstale":"2024-01-15T17:57:33.417307+0000","last_undegraded":"2024-01-15T17:57:33.417307+0000","last_fullsized":"2024-01-15T17:57:33.417307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415834+0000","last_change":"2024-01-15T17:57:32.148492+0000","last_active":"2024-01-15T17:57:33.415834+0000","last_peered":"2024-01-15T17:57:33.415834+0000","last_clean":"2024-01-15T17:57:33.415834+0000","last_became_active":"2024-01-15T17:57:32.148169+0000","last_became_peered":"2024-01-15T17:57:32.148169+0000","last_unstale":"2024-01-15T17:57:33.415834+0000","last_undegraded":"2024-01-15T17:57:33.415834+0000","last_fullsized":"2024-01-15T17:57:33.415834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:00.974439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.150453+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.150115+0000","last_became_peered":"2024-01-15T17:57:32.150115+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.137808+0000","last_change":"2024-01-15T17:57:32.148256+0000","last_active":"2024-01-15T17:57:33.137808+0000","last_peered":"2024-01-15T17:57:33.137808+0000","last_clean":"2024-01-15T17:57:33.137808+0000","last_became_active":"2024-01-15T17:57:32.147620+0000","last_became_peered":"2024-01-15T17:57:32.147620+0000","last_unstale":"2024-01-15T17:57:33.137808+0000","last_undegraded":"2024-01-15T17:57:33.137808+0000","last_fullsized":"2024-01-15T17:57:33.137808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140840+0000","last_change":"2024-01-15T17:57:32.139828+0000","last_active":"2024-01-15T17:57:33.140840+0000","last_peered":"2024-01-15T17:57:33.140840+0000","last_clean":"2024-01-15T17:57:33.140840+0000","last_became_active":"2024-01-15T17:57:32.139522+0000","last_became_peered":"2024-01-15T17:57:32.139522+0000","last_unstale":"2024-01-15T17:57:33.140840+0000","last_undegraded":"2024-01-15T17:57:33.140840+0000","last_fullsized":"2024-01-15T17:57:33.140840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416706+0000","last_change":"2024-01-15T17:57:32.140652+0000","last_active":"2024-01-15T17:57:33.416706+0000","last_peered":"2024-01-15T17:57:33.416706+0000","last_clean":"2024-01-15T17:57:33.416706+0000","last_became_active":"2024-01-15T17:57:32.140366+0000","last_became_peered":"2024-01-15T17:57:32.140366+0000","last_unstale":"2024-01-15T17:57:33.416706+0000","last_undegraded":"2024-01-15T17:57:33.416706+0000","last_fullsized":"2024-01-15T17:57:33.416706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141414+0000","last_change":"2024-01-15T17:57:32.149137+0000","last_active":"2024-01-15T17:57:33.141414+0000","last_peered":"2024-01-15T17:57:33.141414+0000","last_clean":"2024-01-15T17:57:33.141414+0000","last_became_active":"2024-01-15T17:57:32.148658+0000","last_became_peered":"2024-01-15T17:57:32.148658+0000","last_unstale":"2024-01-15T17:57:33.141414+0000","last_undegraded":"2024-01-15T17:57:33.141414+0000","last_fullsized":"2024-01-15T17:57:33.141414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:18.642472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414742+0000","last_change":"2024-01-15T17:57:32.156212+0000","last_active":"2024-01-15T17:57:33.414742+0000","last_peered":"2024-01-15T17:57:33.414742+0000","last_clean":"2024-01-15T17:57:33.414742+0000","last_became_active":"2024-01-15T17:57:32.153278+0000","last_became_peered":"2024-01-15T17:57:32.153278+0000","last_unstale":"2024-01-15T17:57:33.414742+0000","last_undegraded":"2024-01-15T17:57:33.414742+0000","last_fullsized":"2024-01-15T17:57:33.414742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417083+0000","last_change":"2024-01-15T17:57:32.147107+0000","last_active":"2024-01-15T17:57:33.417083+0000","last_peered":"2024-01-15T17:57:33.417083+0000","last_clean":"2024-01-15T17:57:33.417083+0000","last_became_active":"2024-01-15T17:57:32.146730+0000","last_became_peered":"2024-01-15T17:57:32.146730+0000","last_unstale":"2024-01-15T17:57:33.417083+0000","last_undegraded":"2024-01-15T17:57:33.417083+0000","last_fullsized":"2024-01-15T17:57:33.417083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138213+0000","last_change":"2024-01-15T17:57:32.139188+0000","last_active":"2024-01-15T17:57:33.138213+0000","last_peered":"2024-01-15T17:57:33.138213+0000","last_clean":"2024-01-15T17:57:33.138213+0000","last_became_active":"2024-01-15T17:57:32.138898+0000","last_became_peered":"2024-01-15T17:57:32.138898+0000","last_unstale":"2024-01-15T17:57:33.138213+0000","last_undegraded":"2024-01-15T17:57:33.138213+0000","last_fullsized":"2024-01-15T17:57:33.138213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416604+0000","last_change":"2024-01-15T17:57:32.148598+0000","last_active":"2024-01-15T17:57:33.416604+0000","last_peered":"2024-01-15T17:57:33.416604+0000","last_clean":"2024-01-15T17:57:33.416604+0000","last_became_active":"2024-01-15T17:57:32.148266+0000","last_became_peered":"2024-01-15T17:57:32.148266+0000","last_unstale":"2024-01-15T17:57:33.416604+0000","last_undegraded":"2024-01-15T17:57:33.416604+0000","last_fullsized":"2024-01-15T17:57:33.416604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138790+0000","last_change":"2024-01-15T17:57:32.135029+0000","last_active":"2024-01-15T17:57:33.138790+0000","last_peered":"2024-01-15T17:57:33.138790+0000","last_clean":"2024-01-15T17:57:33.138790+0000","last_became_active":"2024-01-15T17:57:32.134762+0000","last_became_peered":"2024-01-15T17:57:32.134762+0000","last_unstale":"2024-01-15T17:57:33.138790+0000","last_undegraded":"2024-01-15T17:57:33.138790+0000","last_fullsized":"2024-01-15T17:57:33.138790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418117+0000","last_change":"2024-01-15T17:57:32.153167+0000","last_active":"2024-01-15T17:57:33.418117+0000","last_peered":"2024-01-15T17:57:33.418117+0000","last_clean":"2024-01-15T17:57:33.418117+0000","last_became_active":"2024-01-15T17:57:32.151003+0000","last_became_peered":"2024-01-15T17:57:32.151003+0000","last_unstale":"2024-01-15T17:57:33.418117+0000","last_undegraded":"2024-01-15T17:57:33.418117+0000","last_fullsized":"2024-01-15T17:57:33.418117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:32.142988+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:32.142613+0000","last_became_peered":"2024-01-15T17:57:32.142613+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136341+0000","last_change":"2024-01-15T17:57:32.134987+0000","last_active":"2024-01-15T17:57:33.136341+0000","last_peered":"2024-01-15T17:57:33.136341+0000","last_clean":"2024-01-15T17:57:33.136341+0000","last_became_active":"2024-01-15T17:57:32.134719+0000","last_became_peered":"2024-01-15T17:57:32.134719+0000","last_unstale":"2024-01-15T17:57:33.136341+0000","last_undegraded":"2024-01-15T17:57:33.136341+0000","last_fullsized":"2024-01-15T17:57:33.136341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418595+0000","last_change":"2024-01-15T17:57:32.153961+0000","last_active":"2024-01-15T17:57:33.418595+0000","last_peered":"2024-01-15T17:57:33.418595+0000","last_clean":"2024-01-15T17:57:33.418595+0000","last_became_active":"2024-01-15T17:57:32.151097+0000","last_became_peered":"2024-01-15T17:57:32.151097+0000","last_unstale":"2024-01-15T17:57:33.418595+0000","last_undegraded":"2024-01-15T17:57:33.418595+0000","last_fullsized":"2024-01-15T17:57:33.418595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418195+0000","last_change":"2024-01-15T17:57:32.144509+0000","last_active":"2024-01-15T17:57:33.418195+0000","last_peered":"2024-01-15T17:57:33.418195+0000","last_clean":"2024-01-15T17:57:33.418195+0000","last_became_active":"2024-01-15T17:57:32.143921+0000","last_became_peered":"2024-01-15T17:57:32.143921+0000","last_unstale":"2024-01-15T17:57:33.418195+0000","last_undegraded":"2024-01-15T17:57:33.418195+0000","last_fullsized":"2024-01-15T17:57:33.418195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138415+0000","last_change":"2024-01-15T17:57:32.137686+0000","last_active":"2024-01-15T17:57:33.138415+0000","last_peered":"2024-01-15T17:57:33.138415+0000","last_clean":"2024-01-15T17:57:33.138415+0000","last_became_active":"2024-01-15T17:57:32.137409+0000","last_became_peered":"2024-01-15T17:57:32.137409+0000","last_unstale":"2024-01-15T17:57:33.138415+0000","last_undegraded":"2024-01-15T17:57:33.138415+0000","last_fullsized":"2024-01-15T17:57:33.138415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414292+0000","last_change":"2024-01-15T17:57:32.129661+0000","last_active":"2024-01-15T17:57:33.414292+0000","last_peered":"2024-01-15T17:57:33.414292+0000","last_clean":"2024-01-15T17:57:33.414292+0000","last_became_active":"2024-01-15T17:57:32.129283+0000","last_became_peered":"2024-01-15T17:57:32.129283+0000","last_unstale":"2024-01-15T17:57:33.414292+0000","last_undegraded":"2024-01-15T17:57:33.414292+0000","last_fullsized":"2024-01-15T17:57:33.414292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414350+0000","last_change":"2024-01-15T17:57:32.147901+0000","last_active":"2024-01-15T17:57:33.414350+0000","last_peered":"2024-01-15T17:57:33.414350+0000","last_clean":"2024-01-15T17:57:33.414350+0000","last_became_active":"2024-01-15T17:57:32.147185+0000","last_became_peered":"2024-01-15T17:57:32.147185+0000","last_unstale":"2024-01-15T17:57:33.414350+0000","last_undegraded":"2024-01-15T17:57:33.414350+0000","last_fullsized":"2024-01-15T17:57:33.414350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186262+0000","last_change":"2024-01-15T17:57:32.138254+0000","last_active":"2024-01-15T17:57:33.186262+0000","last_peered":"2024-01-15T17:57:33.186262+0000","last_clean":"2024-01-15T17:57:33.186262+0000","last_became_active":"2024-01-15T17:57:32.137849+0000","last_became_peered":"2024-01-15T17:57:32.137849+0000","last_unstale":"2024-01-15T17:57:33.186262+0000","last_undegraded":"2024-01-15T17:57:33.186262+0000","last_fullsized":"2024-01-15T17:57:33.186262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:12.340083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417690+0000","last_change":"2024-01-15T17:57:31.132496+0000","last_active":"2024-01-15T17:57:33.417690+0000","last_peered":"2024-01-15T17:57:33.417690+0000","last_clean":"2024-01-15T17:57:33.417690+0000","last_became_active":"2024-01-15T17:57:31.132095+0000","last_became_peered":"2024-01-15T17:57:31.132095+0000","last_unstale":"2024-01-15T17:57:33.417690+0000","last_undegraded":"2024-01-15T17:57:33.417690+0000","last_fullsized":"2024-01-15T17:57:33.417690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418981+0000","last_change":"2024-01-15T17:57:32.154160+0000","last_active":"2024-01-15T17:57:33.418981+0000","last_peered":"2024-01-15T17:57:33.418981+0000","last_clean":"2024-01-15T17:57:33.418981+0000","last_became_active":"2024-01-15T17:57:32.153881+0000","last_became_peered":"2024-01-15T17:57:32.153881+0000","last_unstale":"2024-01-15T17:57:33.418981+0000","last_undegraded":"2024-01-15T17:57:33.418981+0000","last_fullsized":"2024-01-15T17:57:33.418981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414244+0000","last_change":"2024-01-15T17:57:31.143303+0000","last_active":"2024-01-15T17:57:33.414244+0000","last_peered":"2024-01-15T17:57:33.414244+0000","last_clean":"2024-01-15T17:57:33.414244+0000","last_became_active":"2024-01-15T17:57:31.142917+0000","last_became_peered":"2024-01-15T17:57:31.142917+0000","last_unstale":"2024-01-15T17:57:33.414244+0000","last_undegraded":"2024-01-15T17:57:33.414244+0000","last_fullsized":"2024-01-15T17:57:33.414244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416836+0000","last_change":"2024-01-15T17:57:31.137389+0000","last_active":"2024-01-15T17:57:33.416836+0000","last_peered":"2024-01-15T17:57:33.416836+0000","last_clean":"2024-01-15T17:57:33.416836+0000","last_became_active":"2024-01-15T17:57:31.137115+0000","last_became_peered":"2024-01-15T17:57:31.137115+0000","last_unstale":"2024-01-15T17:57:33.416836+0000","last_undegraded":"2024-01-15T17:57:33.416836+0000","last_fullsized":"2024-01-15T17:57:33.416836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415574+0000","last_change":"2024-01-15T17:57:32.156574+0000","last_active":"2024-01-15T17:57:33.415574+0000","last_peered":"2024-01-15T17:57:33.415574+0000","last_clean":"2024-01-15T17:57:33.415574+0000","last_became_active":"2024-01-15T17:57:32.156179+0000","last_became_peered":"2024-01-15T17:57:32.156179+0000","last_unstale":"2024-01-15T17:57:33.415574+0000","last_undegraded":"2024-01-15T17:57:33.415574+0000","last_fullsized":"2024-01-15T17:57:33.415574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139052+0000","last_change":"2024-01-15T17:57:31.132492+0000","last_active":"2024-01-15T17:57:33.139052+0000","last_peered":"2024-01-15T17:57:33.139052+0000","last_clean":"2024-01-15T17:57:33.139052+0000","last_became_active":"2024-01-15T17:57:31.132141+0000","last_became_peered":"2024-01-15T17:57:31.132141+0000","last_unstale":"2024-01-15T17:57:33.139052+0000","last_undegraded":"2024-01-15T17:57:33.139052+0000","last_fullsized":"2024-01-15T17:57:33.139052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.134222+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.133920+0000","last_became_peered":"2024-01-15T17:57:32.133920+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:31.144178+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:31.143915+0000","last_became_peered":"2024-01-15T17:57:31.143915+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140200+0000","last_change":"2024-01-15T17:57:32.141022+0000","last_active":"2024-01-15T17:57:33.140200+0000","last_peered":"2024-01-15T17:57:33.140200+0000","last_clean":"2024-01-15T17:57:33.140200+0000","last_became_active":"2024-01-15T17:57:32.140122+0000","last_became_peered":"2024-01-15T17:57:32.140122+0000","last_unstale":"2024-01-15T17:57:33.140200+0000","last_undegraded":"2024-01-15T17:57:33.140200+0000","last_fullsized":"2024-01-15T17:57:33.140200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140234+0000","last_change":"2024-01-15T17:57:31.146237+0000","last_active":"2024-01-15T17:57:33.140234+0000","last_peered":"2024-01-15T17:57:33.140234+0000","last_clean":"2024-01-15T17:57:33.140234+0000","last_became_active":"2024-01-15T17:57:31.145808+0000","last_became_peered":"2024-01-15T17:57:31.145808+0000","last_unstale":"2024-01-15T17:57:33.140234+0000","last_undegraded":"2024-01-15T17:57:33.140234+0000","last_fullsized":"2024-01-15T17:57:33.140234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:05.346229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419713+0000","last_change":"2024-01-15T17:57:32.142125+0000","last_active":"2024-01-15T17:57:33.419713+0000","last_peered":"2024-01-15T17:57:33.419713+0000","last_clean":"2024-01-15T17:57:33.419713+0000","last_became_active":"2024-01-15T17:57:32.141806+0000","last_became_peered":"2024-01-15T17:57:32.141806+0000","last_unstale":"2024-01-15T17:57:33.419713+0000","last_undegraded":"2024-01-15T17:57:33.419713+0000","last_fullsized":"2024-01-15T17:57:33.419713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136484+0000","last_change":"2024-01-15T17:57:31.131511+0000","last_active":"2024-01-15T17:57:33.136484+0000","last_peered":"2024-01-15T17:57:33.136484+0000","last_clean":"2024-01-15T17:57:33.136484+0000","last_became_active":"2024-01-15T17:57:31.131207+0000","last_became_peered":"2024-01-15T17:57:31.131207+0000","last_unstale":"2024-01-15T17:57:33.136484+0000","last_undegraded":"2024-01-15T17:57:33.136484+0000","last_fullsized":"2024-01-15T17:57:33.136484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186851+0000","last_change":"2024-01-15T17:57:32.141464+0000","last_active":"2024-01-15T17:57:33.186851+0000","last_peered":"2024-01-15T17:57:33.186851+0000","last_clean":"2024-01-15T17:57:33.186851+0000","last_became_active":"2024-01-15T17:57:32.140204+0000","last_became_peered":"2024-01-15T17:57:32.140204+0000","last_unstale":"2024-01-15T17:57:33.186851+0000","last_undegraded":"2024-01-15T17:57:33.186851+0000","last_fullsized":"2024-01-15T17:57:33.186851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:23.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415055+0000","last_change":"2024-01-15T17:57:31.140465+0000","last_active":"2024-01-15T17:57:33.415055+0000","last_peered":"2024-01-15T17:57:33.415055+0000","last_clean":"2024-01-15T17:57:33.415055+0000","last_became_active":"2024-01-15T17:57:31.140132+0000","last_became_peered":"2024-01-15T17:57:31.140132+0000","last_unstale":"2024-01-15T17:57:33.415055+0000","last_undegraded":"2024-01-15T17:57:33.415055+0000","last_fullsized":"2024-01-15T17:57:33.415055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141003+0000","last_change":"2024-01-15T17:57:32.140539+0000","last_active":"2024-01-15T17:57:33.141003+0000","last_peered":"2024-01-15T17:57:33.141003+0000","last_clean":"2024-01-15T17:57:33.141003+0000","last_became_active":"2024-01-15T17:57:32.140117+0000","last_became_peered":"2024-01-15T17:57:32.140117+0000","last_unstale":"2024-01-15T17:57:33.141003+0000","last_undegraded":"2024-01-15T17:57:33.141003+0000","last_fullsized":"2024-01-15T17:57:33.141003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418517+0000","last_change":"2024-01-15T17:57:31.134124+0000","last_active":"2024-01-15T17:57:33.418517+0000","last_peered":"2024-01-15T17:57:33.418517+0000","last_clean":"2024-01-15T17:57:33.418517+0000","last_became_active":"2024-01-15T17:57:31.133698+0000","last_became_peered":"2024-01-15T17:57:31.133698+0000","last_unstale":"2024-01-15T17:57:33.418517+0000","last_undegraded":"2024-01-15T17:57:33.418517+0000","last_fullsized":"2024-01-15T17:57:33.418517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416094+0000","last_change":"2024-01-15T17:57:32.148065+0000","last_active":"2024-01-15T17:57:33.416094+0000","last_peered":"2024-01-15T17:57:33.416094+0000","last_clean":"2024-01-15T17:57:33.416094+0000","last_became_active":"2024-01-15T17:57:32.145370+0000","last_became_peered":"2024-01-15T17:57:32.145370+0000","last_unstale":"2024-01-15T17:57:33.416094+0000","last_undegraded":"2024-01-15T17:57:33.416094+0000","last_fullsized":"2024-01-15T17:57:33.416094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:46.538517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138441+0000","last_change":"2024-01-15T17:57:31.128237+0000","last_active":"2024-01-15T17:57:33.138441+0000","last_peered":"2024-01-15T17:57:33.138441+0000","last_clean":"2024-01-15T17:57:33.138441+0000","last_became_active":"2024-01-15T17:57:31.127956+0000","last_became_peered":"2024-01-15T17:57:31.127956+0000","last_unstale":"2024-01-15T17:57:33.138441+0000","last_undegraded":"2024-01-15T17:57:33.138441+0000","last_fullsized":"2024-01-15T17:57:33.138441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141534+0000","last_change":"2024-01-15T17:57:32.150884+0000","last_active":"2024-01-15T17:57:33.141534+0000","last_peered":"2024-01-15T17:57:33.141534+0000","last_clean":"2024-01-15T17:57:33.141534+0000","last_became_active":"2024-01-15T17:57:32.150381+0000","last_became_peered":"2024-01-15T17:57:32.150381+0000","last_unstale":"2024-01-15T17:57:33.141534+0000","last_undegraded":"2024-01-15T17:57:33.141534+0000","last_fullsized":"2024-01-15T17:57:33.141534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:23.429091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414222+0000","last_change":"2024-01-15T17:57:31.138127+0000","last_active":"2024-01-15T17:57:33.414222+0000","last_peered":"2024-01-15T17:57:33.414222+0000","last_clean":"2024-01-15T17:57:33.414222+0000","last_became_active":"2024-01-15T17:57:31.137579+0000","last_became_peered":"2024-01-15T17:57:31.137579+0000","last_unstale":"2024-01-15T17:57:33.414222+0000","last_undegraded":"2024-01-15T17:57:33.414222+0000","last_fullsized":"2024-01-15T17:57:33.414222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414356+0000","last_change":"2024-01-15T17:57:32.150432+0000","last_active":"2024-01-15T17:57:33.414356+0000","last_peered":"2024-01-15T17:57:33.414356+0000","last_clean":"2024-01-15T17:57:33.414356+0000","last_became_active":"2024-01-15T17:57:32.147572+0000","last_became_peered":"2024-01-15T17:57:32.147572+0000","last_unstale":"2024-01-15T17:57:33.414356+0000","last_undegraded":"2024-01-15T17:57:33.414356+0000","last_fullsized":"2024-01-15T17:57:33.414356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414449+0000","last_change":"2024-01-15T17:57:31.134499+0000","last_active":"2024-01-15T17:57:33.414449+0000","last_peered":"2024-01-15T17:57:33.414449+0000","last_clean":"2024-01-15T17:57:33.414449+0000","last_became_active":"2024-01-15T17:57:31.131997+0000","last_became_peered":"2024-01-15T17:57:31.131997+0000","last_unstale":"2024-01-15T17:57:33.414449+0000","last_undegraded":"2024-01-15T17:57:33.414449+0000","last_fullsized":"2024-01-15T17:57:33.414449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:11.214052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139065+0000","last_change":"2024-01-15T17:57:32.151412+0000","last_active":"2024-01-15T17:57:33.139065+0000","last_peered":"2024-01-15T17:57:33.139065+0000","last_clean":"2024-01-15T17:57:33.139065+0000","last_became_active":"2024-01-15T17:57:32.151061+0000","last_became_peered":"2024-01-15T17:57:32.151061+0000","last_unstale":"2024-01-15T17:57:33.139065+0000","last_undegraded":"2024-01-15T17:57:33.139065+0000","last_fullsized":"2024-01-15T17:57:33.139065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186594+0000","last_change":"2024-01-15T17:57:31.133132+0000","last_active":"2024-01-15T17:57:33.186594+0000","last_peered":"2024-01-15T17:57:33.186594+0000","last_clean":"2024-01-15T17:57:33.186594+0000","last_became_active":"2024-01-15T17:57:31.131606+0000","last_became_peered":"2024-01-15T17:57:31.131606+0000","last_unstale":"2024-01-15T17:57:33.186594+0000","last_undegraded":"2024-01-15T17:57:33.186594+0000","last_fullsized":"2024-01-15T17:57:33.186594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:58:03.663628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139362+0000","last_change":"2024-01-15T17:57:32.152397+0000","last_active":"2024-01-15T17:57:33.139362+0000","last_peered":"2024-01-15T17:57:33.139362+0000","last_clean":"2024-01-15T17:57:33.139362+0000","last_became_active":"2024-01-15T17:57:32.152096+0000","last_became_peered":"2024-01-15T17:57:32.152096+0000","last_unstale":"2024-01-15T17:57:33.139362+0000","last_undegraded":"2024-01-15T17:57:33.139362+0000","last_fullsized":"2024-01-15T17:57:33.139362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417449+0000","last_change":"2024-01-15T17:57:31.127107+0000","last_active":"2024-01-15T17:57:33.417449+0000","last_peered":"2024-01-15T17:57:33.417449+0000","last_clean":"2024-01-15T17:57:33.417449+0000","last_became_active":"2024-01-15T17:57:31.126871+0000","last_became_peered":"2024-01-15T17:57:31.126871+0000","last_unstale":"2024-01-15T17:57:33.417449+0000","last_undegraded":"2024-01-15T17:57:33.417449+0000","last_fullsized":"2024-01-15T17:57:33.417449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:20.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140382+0000","last_change":"2024-01-15T17:57:32.150858+0000","last_active":"2024-01-15T17:57:33.140382+0000","last_peered":"2024-01-15T17:57:33.140382+0000","last_clean":"2024-01-15T17:57:33.140382+0000","last_became_active":"2024-01-15T17:57:32.150312+0000","last_became_peered":"2024-01-15T17:57:32.150312+0000","last_unstale":"2024-01-15T17:57:33.140382+0000","last_undegraded":"2024-01-15T17:57:33.140382+0000","last_fullsized":"2024-01-15T17:57:33.140382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136681+0000","last_change":"2024-01-15T17:57:31.136226+0000","last_active":"2024-01-15T17:57:33.136681+0000","last_peered":"2024-01-15T17:57:33.136681+0000","last_clean":"2024-01-15T17:57:33.136681+0000","last_became_active":"2024-01-15T17:57:31.135879+0000","last_became_peered":"2024-01-15T17:57:31.135879+0000","last_unstale":"2024-01-15T17:57:33.136681+0000","last_undegraded":"2024-01-15T17:57:33.136681+0000","last_fullsized":"2024-01-15T17:57:33.136681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.149505+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.144115+0000","last_became_peered":"2024-01-15T17:57:32.144115+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415379+0000","last_change":"2024-01-15T17:57:31.130353+0000","last_active":"2024-01-15T17:57:33.415379+0000","last_peered":"2024-01-15T17:57:33.415379+0000","last_clean":"2024-01-15T17:57:33.415379+0000","last_became_active":"2024-01-15T17:57:31.130063+0000","last_became_peered":"2024-01-15T17:57:31.130063+0000","last_unstale":"2024-01-15T17:57:33.415379+0000","last_undegraded":"2024-01-15T17:57:33.415379+0000","last_fullsized":"2024-01-15T17:57:33.415379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:42.825666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141047+0000","last_change":"2024-01-15T17:57:32.148976+0000","last_active":"2024-01-15T17:57:33.141047+0000","last_peered":"2024-01-15T17:57:33.141047+0000","last_clean":"2024-01-15T17:57:33.141047+0000","last_became_active":"2024-01-15T17:57:32.148285+0000","last_became_peered":"2024-01-15T17:57:32.148285+0000","last_unstale":"2024-01-15T17:57:33.141047+0000","last_undegraded":"2024-01-15T17:57:33.141047+0000","last_fullsized":"2024-01-15T17:57:33.141047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T17:57:32.126659+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:57:32.126659+0000","last_peered":"2024-01-15T17:57:32.126659+0000","last_clean":"2024-01-15T17:57:32.126659+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:57:32.126659+0000","last_undegraded":"2024-01-15T17:57:32.126659+0000","last_fullsized":"2024-01-15T17:57:32.126659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138248+0000","last_change":"2024-01-15T17:57:31.127921+0000","last_active":"2024-01-15T17:57:33.138248+0000","last_peered":"2024-01-15T17:57:33.138248+0000","last_clean":"2024-01-15T17:57:33.138248+0000","last_became_active":"2024-01-15T17:57:31.126423+0000","last_became_peered":"2024-01-15T17:57:31.126423+0000","last_unstale":"2024-01-15T17:57:33.138248+0000","last_undegraded":"2024-01-15T17:57:33.138248+0000","last_fullsized":"2024-01-15T17:57:33.138248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:43:31.770732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418628+0000","last_change":"2024-01-15T17:57:32.151647+0000","last_active":"2024-01-15T17:57:33.418628+0000","last_peered":"2024-01-15T17:57:33.418628+0000","last_clean":"2024-01-15T17:57:33.418628+0000","last_became_active":"2024-01-15T17:57:32.149470+0000","last_became_peered":"2024-01-15T17:57:32.149470+0000","last_unstale":"2024-01-15T17:57:33.418628+0000","last_undegraded":"2024-01-15T17:57:33.418628+0000","last_fullsized":"2024-01-15T17:57:33.418628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139414+0000","last_change":"2024-01-15T17:57:31.141208+0000","last_active":"2024-01-15T17:57:33.139414+0000","last_peered":"2024-01-15T17:57:33.139414+0000","last_clean":"2024-01-15T17:57:33.139414+0000","last_became_active":"2024-01-15T17:57:31.140756+0000","last_became_peered":"2024-01-15T17:57:31.140756+0000","last_unstale":"2024-01-15T17:57:33.139414+0000","last_undegraded":"2024-01-15T17:57:33.139414+0000","last_fullsized":"2024-01-15T17:57:33.139414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:17:40.803403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415471+0000","last_change":"2024-01-15T17:57:32.150522+0000","last_active":"2024-01-15T17:57:33.415471+0000","last_peered":"2024-01-15T17:57:33.415471+0000","last_clean":"2024-01-15T17:57:33.415471+0000","last_became_active":"2024-01-15T17:57:32.150224+0000","last_became_peered":"2024-01-15T17:57:32.150224+0000","last_unstale":"2024-01-15T17:57:33.415471+0000","last_undegraded":"2024-01-15T17:57:33.415471+0000","last_fullsized":"2024-01-15T17:57:33.415471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:03.799210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414399+0000","last_change":"2024-01-15T17:57:31.127558+0000","last_active":"2024-01-15T17:57:33.414399+0000","last_peered":"2024-01-15T17:57:33.414399+0000","last_clean":"2024-01-15T17:57:33.414399+0000","last_became_active":"2024-01-15T17:57:31.125325+0000","last_became_peered":"2024-01-15T17:57:31.125325+0000","last_unstale":"2024-01-15T17:57:33.414399+0000","last_undegraded":"2024-01-15T17:57:33.414399+0000","last_fullsized":"2024-01-15T17:57:33.414399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:56.231915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136344+0000","last_change":"2024-01-15T17:57:32.134693+0000","last_active":"2024-01-15T17:57:33.136344+0000","last_peered":"2024-01-15T17:57:33.136344+0000","last_clean":"2024-01-15T17:57:33.136344+0000","last_became_active":"2024-01-15T17:57:32.133773+0000","last_became_peered":"2024-01-15T17:57:32.133773+0000","last_unstale":"2024-01-15T17:57:33.136344+0000","last_undegraded":"2024-01-15T17:57:33.136344+0000","last_fullsized":"2024-01-15T17:57:33.136344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138654+0000","last_change":"2024-01-15T17:57:31.133660+0000","last_active":"2024-01-15T17:57:33.138654+0000","last_peered":"2024-01-15T17:57:33.138654+0000","last_clean":"2024-01-15T17:57:33.138654+0000","last_became_active":"2024-01-15T17:57:31.132627+0000","last_became_peered":"2024-01-15T17:57:31.132627+0000","last_unstale":"2024-01-15T17:57:33.138654+0000","last_undegraded":"2024-01-15T17:57:33.138654+0000","last_fullsized":"2024-01-15T17:57:33.138654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186541+0000","last_change":"2024-01-15T17:57:32.147182+0000","last_active":"2024-01-15T17:57:33.186541+0000","last_peered":"2024-01-15T17:57:33.186541+0000","last_clean":"2024-01-15T17:57:33.186541+0000","last_became_active":"2024-01-15T17:57:32.146838+0000","last_became_peered":"2024-01-15T17:57:32.146838+0000","last_unstale":"2024-01-15T17:57:33.186541+0000","last_undegraded":"2024-01-15T17:57:33.186541+0000","last_fullsized":"2024-01-15T17:57:33.186541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136810+0000","last_change":"2024-01-15T17:57:31.126189+0000","last_active":"2024-01-15T17:57:33.136810+0000","last_peered":"2024-01-15T17:57:33.136810+0000","last_clean":"2024-01-15T17:57:33.136810+0000","last_became_active":"2024-01-15T17:57:31.125519+0000","last_became_peered":"2024-01-15T17:57:31.125519+0000","last_unstale":"2024-01-15T17:57:33.136810+0000","last_undegraded":"2024-01-15T17:57:33.136810+0000","last_fullsized":"2024-01-15T17:57:33.136810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.137059+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.136012+0000","last_became_peered":"2024-01-15T17:57:32.136012+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141434+0000","last_change":"2024-01-15T17:57:31.139980+0000","last_active":"2024-01-15T17:57:33.141434+0000","last_peered":"2024-01-15T17:57:33.141434+0000","last_clean":"2024-01-15T17:57:33.141434+0000","last_became_active":"2024-01-15T17:57:31.139489+0000","last_became_peered":"2024-01-15T17:57:31.139489+0000","last_unstale":"2024-01-15T17:57:33.141434+0000","last_undegraded":"2024-01-15T17:57:33.141434+0000","last_fullsized":"2024-01-15T17:57:33.141434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:54.272945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136716+0000","last_change":"2024-01-15T17:57:32.130379+0000","last_active":"2024-01-15T17:57:33.136716+0000","last_peered":"2024-01-15T17:57:33.136716+0000","last_clean":"2024-01-15T17:57:33.136716+0000","last_became_active":"2024-01-15T17:57:32.129645+0000","last_became_peered":"2024-01-15T17:57:32.129645+0000","last_unstale":"2024-01-15T17:57:33.136716+0000","last_undegraded":"2024-01-15T17:57:33.136716+0000","last_fullsized":"2024-01-15T17:57:33.136716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:34.417066+0000","last_change":"2024-01-15T17:57:31.146382+0000","last_active":"2024-01-15T17:57:34.417066+0000","last_peered":"2024-01-15T17:57:34.417066+0000","last_clean":"2024-01-15T17:57:34.417066+0000","last_became_active":"2024-01-15T17:57:31.146119+0000","last_became_peered":"2024-01-15T17:57:31.146119+0000","last_unstale":"2024-01-15T17:57:34.417066+0000","last_undegraded":"2024-01-15T17:57:34.417066+0000","last_fullsized":"2024-01-15T17:57:34.417066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:30.690538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418481+0000","last_change":"2024-01-15T17:57:32.144183+0000","last_active":"2024-01-15T17:57:33.418481+0000","last_peered":"2024-01-15T17:57:33.418481+0000","last_clean":"2024-01-15T17:57:33.418481+0000","last_became_active":"2024-01-15T17:57:32.143880+0000","last_became_peered":"2024-01-15T17:57:32.143880+0000","last_unstale":"2024-01-15T17:57:33.418481+0000","last_undegraded":"2024-01-15T17:57:33.418481+0000","last_fullsized":"2024-01-15T17:57:33.418481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415224+0000","last_change":"2024-01-15T17:57:31.144154+0000","last_active":"2024-01-15T17:57:33.415224+0000","last_peered":"2024-01-15T17:57:33.415224+0000","last_clean":"2024-01-15T17:57:33.415224+0000","last_became_active":"2024-01-15T17:57:31.143857+0000","last_became_peered":"2024-01-15T17:57:31.143857+0000","last_unstale":"2024-01-15T17:57:33.415224+0000","last_undegraded":"2024-01-15T17:57:33.415224+0000","last_fullsized":"2024-01-15T17:57:33.415224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419365+0000","last_change":"2024-01-15T17:57:32.142460+0000","last_active":"2024-01-15T17:57:33.419365+0000","last_peered":"2024-01-15T17:57:33.419365+0000","last_clean":"2024-01-15T17:57:33.419365+0000","last_became_active":"2024-01-15T17:57:32.141816+0000","last_became_peered":"2024-01-15T17:57:32.141816+0000","last_unstale":"2024-01-15T17:57:33.419365+0000","last_undegraded":"2024-01-15T17:57:33.419365+0000","last_fullsized":"2024-01-15T17:57:33.419365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139202+0000","last_change":"2024-01-15T17:57:31.139784+0000","last_active":"2024-01-15T17:57:33.139202+0000","last_peered":"2024-01-15T17:57:33.139202+0000","last_clean":"2024-01-15T17:57:33.139202+0000","last_became_active":"2024-01-15T17:57:31.138181+0000","last_became_peered":"2024-01-15T17:57:31.138181+0000","last_unstale":"2024-01-15T17:57:33.139202+0000","last_undegraded":"2024-01-15T17:57:33.139202+0000","last_fullsized":"2024-01-15T17:57:33.139202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:50.662857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186323+0000","last_change":"2024-01-15T17:57:32.130289+0000","last_active":"2024-01-15T17:57:33.186323+0000","last_peered":"2024-01-15T17:57:33.186323+0000","last_clean":"2024-01-15T17:57:33.186323+0000","last_became_active":"2024-01-15T17:57:32.129807+0000","last_became_peered":"2024-01-15T17:57:32.129807+0000","last_unstale":"2024-01-15T17:57:33.186323+0000","last_undegraded":"2024-01-15T17:57:33.186323+0000","last_fullsized":"2024-01-15T17:57:33.186323+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:43.951697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135874+0000","last_change":"2024-01-15T17:57:31.132917+0000","last_active":"2024-01-15T17:57:33.135874+0000","last_peered":"2024-01-15T17:57:33.135874+0000","last_clean":"2024-01-15T17:57:33.135874+0000","last_became_active":"2024-01-15T17:57:31.132650+0000","last_became_peered":"2024-01-15T17:57:31.132650+0000","last_unstale":"2024-01-15T17:57:33.135874+0000","last_undegraded":"2024-01-15T17:57:33.135874+0000","last_fullsized":"2024-01-15T17:57:33.135874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415920+0000","last_change":"2024-01-15T17:57:32.136009+0000","last_active":"2024-01-15T17:57:33.415920+0000","last_peered":"2024-01-15T17:57:33.415920+0000","last_clean":"2024-01-15T17:57:33.415920+0000","last_became_active":"2024-01-15T17:57:32.135200+0000","last_became_peered":"2024-01-15T17:57:32.135200+0000","last_unstale":"2024-01-15T17:57:33.415920+0000","last_undegraded":"2024-01-15T17:57:33.415920+0000","last_fullsized":"2024-01-15T17:57:33.415920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136090+0000","last_change":"2024-01-15T17:57:31.130465+0000","last_active":"2024-01-15T17:57:33.136090+0000","last_peered":"2024-01-15T17:57:33.136090+0000","last_clean":"2024-01-15T17:57:33.136090+0000","last_became_active":"2024-01-15T17:57:31.130032+0000","last_became_peered":"2024-01-15T17:57:31.130032+0000","last_unstale":"2024-01-15T17:57:33.136090+0000","last_undegraded":"2024-01-15T17:57:33.136090+0000","last_fullsized":"2024-01-15T17:57:33.136090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417702+0000","last_change":"2024-01-15T17:57:32.154915+0000","last_active":"2024-01-15T17:57:33.417702+0000","last_peered":"2024-01-15T17:57:33.417702+0000","last_clean":"2024-01-15T17:57:33.417702+0000","last_became_active":"2024-01-15T17:57:32.154599+0000","last_became_peered":"2024-01-15T17:57:32.154599+0000","last_unstale":"2024-01-15T17:57:33.417702+0000","last_undegraded":"2024-01-15T17:57:33.417702+0000","last_fullsized":"2024-01-15T17:57:33.417702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141769+0000","last_change":"2024-01-15T17:57:31.143380+0000","last_active":"2024-01-15T17:57:33.141769+0000","last_peered":"2024-01-15T17:57:33.141769+0000","last_clean":"2024-01-15T17:57:33.141769+0000","last_became_active":"2024-01-15T17:57:31.143046+0000","last_became_peered":"2024-01-15T17:57:31.143046+0000","last_unstale":"2024-01-15T17:57:33.141769+0000","last_undegraded":"2024-01-15T17:57:33.141769+0000","last_fullsized":"2024-01-15T17:57:33.141769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414379+0000","last_change":"2024-01-15T17:57:32.149247+0000","last_active":"2024-01-15T17:57:33.414379+0000","last_peered":"2024-01-15T17:57:33.414379+0000","last_clean":"2024-01-15T17:57:33.414379+0000","last_became_active":"2024-01-15T17:57:32.148620+0000","last_became_peered":"2024-01-15T17:57:32.148620+0000","last_unstale":"2024-01-15T17:57:33.414379+0000","last_undegraded":"2024-01-15T17:57:33.414379+0000","last_fullsized":"2024-01-15T17:57:33.414379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139434+0000","last_change":"2024-01-15T17:57:32.159609+0000","last_active":"2024-01-15T17:57:33.139434+0000","last_peered":"2024-01-15T17:57:33.139434+0000","last_clean":"2024-01-15T17:57:33.139434+0000","last_became_active":"2024-01-15T17:57:32.155889+0000","last_became_peered":"2024-01-15T17:57:32.155889+0000","last_unstale":"2024-01-15T17:57:33.139434+0000","last_undegraded":"2024-01-15T17:57:33.139434+0000","last_fullsized":"2024-01-15T17:57:33.139434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138536+0000","last_change":"2024-01-15T17:57:31.144524+0000","last_active":"2024-01-15T17:57:33.138536+0000","last_peered":"2024-01-15T17:57:33.138536+0000","last_clean":"2024-01-15T17:57:33.138536+0000","last_became_active":"2024-01-15T17:57:31.143299+0000","last_became_peered":"2024-01-15T17:57:31.143299+0000","last_unstale":"2024-01-15T17:57:33.138536+0000","last_undegraded":"2024-01-15T17:57:33.138536+0000","last_fullsized":"2024-01-15T17:57:33.138536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139433+0000","last_change":"2024-01-15T17:57:32.151244+0000","last_active":"2024-01-15T17:57:33.139433+0000","last_peered":"2024-01-15T17:57:33.139433+0000","last_clean":"2024-01-15T17:57:33.139433+0000","last_became_active":"2024-01-15T17:57:32.150834+0000","last_became_peered":"2024-01-15T17:57:32.150834+0000","last_unstale":"2024-01-15T17:57:33.139433+0000","last_undegraded":"2024-01-15T17:57:33.139433+0000","last_fullsized":"2024-01-15T17:57:33.139433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141240+0000","last_change":"2024-01-15T17:57:31.144722+0000","last_active":"2024-01-15T17:57:33.141240+0000","last_peered":"2024-01-15T17:57:33.141240+0000","last_clean":"2024-01-15T17:57:33.141240+0000","last_became_active":"2024-01-15T17:57:31.144378+0000","last_became_peered":"2024-01-15T17:57:31.144378+0000","last_unstale":"2024-01-15T17:57:33.141240+0000","last_undegraded":"2024-01-15T17:57:33.141240+0000","last_fullsized":"2024-01-15T17:57:33.141240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:48.548981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419383+0000","last_change":"2024-01-15T17:57:32.142742+0000","last_active":"2024-01-15T17:57:33.419383+0000","last_peered":"2024-01-15T17:57:33.419383+0000","last_clean":"2024-01-15T17:57:33.419383+0000","last_became_active":"2024-01-15T17:57:32.142372+0000","last_became_peered":"2024-01-15T17:57:32.142372+0000","last_unstale":"2024-01-15T17:57:33.419383+0000","last_undegraded":"2024-01-15T17:57:33.419383+0000","last_fullsized":"2024-01-15T17:57:33.419383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417735+0000","last_change":"2024-01-15T17:57:31.128188+0000","last_active":"2024-01-15T17:57:33.417735+0000","last_peered":"2024-01-15T17:57:33.417735+0000","last_clean":"2024-01-15T17:57:33.417735+0000","last_became_active":"2024-01-15T17:57:31.127089+0000","last_became_peered":"2024-01-15T17:57:31.127089+0000","last_unstale":"2024-01-15T17:57:33.417735+0000","last_undegraded":"2024-01-15T17:57:33.417735+0000","last_fullsized":"2024-01-15T17:57:33.417735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417498+0000","last_change":"2024-01-15T17:57:32.142205+0000","last_active":"2024-01-15T17:57:33.417498+0000","last_peered":"2024-01-15T17:57:33.417498+0000","last_clean":"2024-01-15T17:57:33.417498+0000","last_became_active":"2024-01-15T17:57:32.139881+0000","last_became_peered":"2024-01-15T17:57:32.139881+0000","last_unstale":"2024-01-15T17:57:33.417498+0000","last_undegraded":"2024-01-15T17:57:33.417498+0000","last_fullsized":"2024-01-15T17:57:33.417498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419783+0000","last_change":"2024-01-15T17:57:31.140404+0000","last_active":"2024-01-15T17:57:33.419783+0000","last_peered":"2024-01-15T17:57:33.419783+0000","last_clean":"2024-01-15T17:57:33.419783+0000","last_became_active":"2024-01-15T17:57:31.140098+0000","last_became_peered":"2024-01-15T17:57:31.140098+0000","last_unstale":"2024-01-15T17:57:33.419783+0000","last_undegraded":"2024-01-15T17:57:33.419783+0000","last_fullsized":"2024-01-15T17:57:33.419783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8739999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.355}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1429999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.611}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9339999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0680000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5150000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5950000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":831488,"data_stored":669808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:57:42.195 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:57:42.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.439+0000 7f379ba00640 1 Processor -- start 2024-01-15T17:57:42.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.439+0000 7f379ba00640 1 -- start start 2024-01-15T17:57:42.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.441+0000 7f379ba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f379406a4b0 0x7f379406a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.441+0000 7f379ba00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379406add0 con 0x7f3794160f20 2024-01-15T17:57:42.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.442+0000 7f379ba00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3794161cc0 con 0x7f37940b3070 2024-01-15T17:57:42.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.442+0000 7f379ba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3794161ab0 con 0x7f379406a4b0 2024-01-15T17:57:42.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.442+0000 7f378be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3794160f20 0x7f37941613a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35514/0 (socket says 172.21.15.2:35514) 2024-01-15T17:57:42.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.442+0000 7f378be00640 1 -- 172.21.15.2:0/2011882998 learned_addr learned my addr 172.21.15.2:0/2011882998 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:42.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.442+0000 7f3799000640 1 --2- 172.21.15.2:0/2011882998 >> v2:172.21.15.137:3300/0 conn(0x7f379406a4b0 0x7f379406a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.443+0000 7f3799000640 1 -- 172.21.15.2:0/2011882998 >> v1:172.21.15.137:6789/0 conn(0x7f37940b3070 legacy=0x7f37940b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.443+0000 7f3799000640 1 -- 172.21.15.2:0/2011882998 >> v1:172.21.15.2:6789/0 conn(0x7f3794160f20 legacy=0x7f37941613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.443+0000 7f3799000640 1 -- 172.21.15.2:0/2011882998 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379415d920 con 0x7f379406a4b0 2024-01-15T17:57:42.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.444+0000 7f3799000640 1 --2- 172.21.15.2:0/2011882998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f379406a4b0 0x7f379406a890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3794161bf0 tx=0x7f378402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7eb80a254e65e991 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:42.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.448+0000 7f3793e00640 1 -- 172.21.15.2:0/2011882998 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784030800 con 0x7f379406a4b0 2024-01-15T17:57:42.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.448+0000 7f3793e00640 1 -- 172.21.15.2:0/2011882998 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3784030960 con 0x7f379406a4b0 2024-01-15T17:57:42.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.448+0000 7f3793e00640 1 -- 172.21.15.2:0/2011882998 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3784030cf0 con 0x7f379406a4b0 2024-01-15T17:57:42.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.448+0000 7f379ba00640 1 -- 172.21.15.2:0/2011882998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f379406a4b0 msgr2=0x7f379406a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.448+0000 7f379ba00640 1 --2- 172.21.15.2:0/2011882998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f379406a4b0 0x7f379406a890 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3794161bf0 tx=0x7f378402b940 comp rx=0 tx=0).stop 2024-01-15T17:57:42.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.449+0000 7f379ba00640 1 -- 172.21.15.2:0/2011882998 shutdown_connections 2024-01-15T17:57:42.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.449+0000 7f379ba00640 1 --2- 172.21.15.2:0/2011882998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f379406a4b0 0x7f379406a890 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.449+0000 7f379ba00640 1 -- 172.21.15.2:0/2011882998 >> 172.21.15.2:0/2011882998 conn(0x7f379406d410 msgr2=0x7f379406d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:42.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.449+0000 7f379ba00640 1 -- 172.21.15.2:0/2011882998 shutdown_connections 2024-01-15T17:57:42.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.449+0000 7f379ba00640 1 -- 172.21.15.2:0/2011882998 wait complete. 2024-01-15T17:57:42.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.450+0000 7f379ba00640 1 Processor -- start 2024-01-15T17:57:42.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.451+0000 7f379ba00640 1 -- start start 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.451+0000 7f379ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f379406a4b0 0x7f379415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.451+0000 7f379ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37940b3070 0x7f37940adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.451+0000 7f379ba00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 0x7f37941f7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f379ba00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37940afb30 con 0x7f3794160f20 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f379ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37941e6db0 con 0x7f37940b3070 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f379ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37940af290 con 0x7f379406a4b0 2024-01-15T17:57:42.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 0x7f37941f7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 0x7f37941f7120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50058/0 (socket says 172.21.15.2:50058) 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 -- 172.21.15.2:0/1383772652 learned_addr learned my addr 172.21.15.2:0/1383772652 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f378be00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37940b3070 0x7f37940adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799000640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f379406a4b0 0x7f379415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 -- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f379406a4b0 msgr2=0x7f379415a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f379406a4b0 0x7f379415a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 -- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37940b3070 msgr2=0x7f37940adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37940b3070 0x7f37940adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.452+0000 7f3799a00640 1 -- 172.21.15.2:0/1383772652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3784009400 con 0x7f3794160f20 2024-01-15T17:57:42.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.453+0000 7f3799a00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 0x7f37941f7120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c7a0 tx=0x7f378c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:42.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.453+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c003b70 con 0x7f3794160f20 2024-01-15T17:57:42.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.453+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c003cf0 con 0x7f3794160f20 2024-01-15T17:57:42.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.453+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00aa90 con 0x7f3794160f20 2024-01-15T17:57:42.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.453+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37940af570 con 0x7f3794160f20 2024-01-15T17:57:42.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.453+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37941e7050 con 0x7f3794160f20 2024-01-15T17:57:42.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.454+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3794067970 con 0x7f3794160f20 2024-01-15T17:57:42.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.455+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f378c00abf0 con 0x7f3794160f20 2024-01-15T17:57:42.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.456+0000 7f3791600640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f376c06b2f0 0x7f376c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.456+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f378c089a30 con 0x7f3794160f20 2024-01-15T17:57:42.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.456+0000 7f3799000640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f376c06b2f0 0x7f376c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.457+0000 7f3799000640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f376c06b2f0 0x7f376c06d7b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3784036000 tx=0x7f3784008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:42.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.460+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f378c054890 con 0x7f3794160f20 2024-01-15T17:57:42.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.642+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f379415e1c0 con 0x7f376c06b2f0 2024-01-15T17:57:42.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.663+0000 7f3791600640 1 -- 172.21.15.2:0/1383772652 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264464 (secure 0 0 0) 0x7f3784044470 con 0x7f376c06b2f0 2024-01-15T17:57:42.670 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:42.672 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f376c06b2f0 msgr2=0x7f376c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f376c06b2f0 0x7f376c06d7b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3784036000 tx=0x7f3784008030 comp rx=0 tx=0).stop 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 msgr2=0x7f37941f7120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 0x7f37941f7120 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c7a0 tx=0x7f378c00cc60 comp rx=0 tx=0).stop 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 shutdown_connections 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f376c06b2f0 0x7f376c06d7b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3794160f20 0x7f37941f7120 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f37940b3070 0x7f37940adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 --2- 172.21.15.2:0/1383772652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f379406a4b0 0x7f379415a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 >> 172.21.15.2:0/1383772652 conn(0x7f379406d410 msgr2=0x7f379406e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:42.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 shutdown_connections 2024-01-15T17:57:42.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.669+0000 7f379ba00640 1 -- 172.21.15.2:0/1383772652 wait complete. 2024-01-15T17:57:42.691 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-15T17:57:41.371769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236132,"kb_used_data":2984,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738588,"statfs":{"total":773094113280,"available":772852314112,"internally_reserved":0,"allocated":3055616,"data_stored":1815972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":59,"num_write_kb":276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.264117"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139938+0000","last_change":"2024-01-15T17:57:32.145196+0000","last_active":"2024-01-15T17:57:33.139938+0000","last_peered":"2024-01-15T17:57:33.139938+0000","last_clean":"2024-01-15T17:57:33.139938+0000","last_became_active":"2024-01-15T17:57:32.144373+0000","last_became_peered":"2024-01-15T17:57:32.144373+0000","last_unstale":"2024-01-15T17:57:33.139938+0000","last_undegraded":"2024-01-15T17:57:33.139938+0000","last_fullsized":"2024-01-15T17:57:33.139938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136557+0000","last_change":"2024-01-15T17:57:32.143595+0000","last_active":"2024-01-15T17:57:33.136557+0000","last_peered":"2024-01-15T17:57:33.136557+0000","last_clean":"2024-01-15T17:57:33.136557+0000","last_became_active":"2024-01-15T17:57:32.143249+0000","last_became_peered":"2024-01-15T17:57:32.143249+0000","last_unstale":"2024-01-15T17:57:33.136557+0000","last_undegraded":"2024-01-15T17:57:33.136557+0000","last_fullsized":"2024-01-15T17:57:33.136557+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419213+0000","last_change":"2024-01-15T17:57:32.143090+0000","last_active":"2024-01-15T17:57:33.419213+0000","last_peered":"2024-01-15T17:57:33.419213+0000","last_clean":"2024-01-15T17:57:33.419213+0000","last_became_active":"2024-01-15T17:57:32.142828+0000","last_became_peered":"2024-01-15T17:57:32.142828+0000","last_unstale":"2024-01-15T17:57:33.419213+0000","last_undegraded":"2024-01-15T17:57:33.419213+0000","last_fullsized":"2024-01-15T17:57:33.419213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139101+0000","last_change":"2024-01-15T17:57:32.146690+0000","last_active":"2024-01-15T17:57:33.139101+0000","last_peered":"2024-01-15T17:57:33.139101+0000","last_clean":"2024-01-15T17:57:33.139101+0000","last_became_active":"2024-01-15T17:57:32.142878+0000","last_became_peered":"2024-01-15T17:57:32.142878+0000","last_unstale":"2024-01-15T17:57:33.139101+0000","last_undegraded":"2024-01-15T17:57:33.139101+0000","last_fullsized":"2024-01-15T17:57:33.139101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141233+0000","last_change":"2024-01-15T17:57:32.145000+0000","last_active":"2024-01-15T17:57:33.141233+0000","last_peered":"2024-01-15T17:57:33.141233+0000","last_clean":"2024-01-15T17:57:33.141233+0000","last_became_active":"2024-01-15T17:57:32.144715+0000","last_became_peered":"2024-01-15T17:57:32.144715+0000","last_unstale":"2024-01-15T17:57:33.141233+0000","last_undegraded":"2024-01-15T17:57:33.141233+0000","last_fullsized":"2024-01-15T17:57:33.141233+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:03.467001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138047+0000","last_change":"2024-01-15T17:57:32.154833+0000","last_active":"2024-01-15T17:57:33.138047+0000","last_peered":"2024-01-15T17:57:33.138047+0000","last_clean":"2024-01-15T17:57:33.138047+0000","last_became_active":"2024-01-15T17:57:32.153996+0000","last_became_peered":"2024-01-15T17:57:32.153996+0000","last_unstale":"2024-01-15T17:57:33.138047+0000","last_undegraded":"2024-01-15T17:57:33.138047+0000","last_fullsized":"2024-01-15T17:57:33.138047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138900+0000","last_change":"2024-01-15T17:57:32.147665+0000","last_active":"2024-01-15T17:57:33.138900+0000","last_peered":"2024-01-15T17:57:33.138900+0000","last_clean":"2024-01-15T17:57:33.138900+0000","last_became_active":"2024-01-15T17:57:32.147333+0000","last_became_peered":"2024-01-15T17:57:32.147333+0000","last_unstale":"2024-01-15T17:57:33.138900+0000","last_undegraded":"2024-01-15T17:57:33.138900+0000","last_fullsized":"2024-01-15T17:57:33.138900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415231+0000","last_change":"2024-01-15T17:57:32.144764+0000","last_active":"2024-01-15T17:57:33.415231+0000","last_peered":"2024-01-15T17:57:33.415231+0000","last_clean":"2024-01-15T17:57:33.415231+0000","last_became_active":"2024-01-15T17:57:32.144315+0000","last_became_peered":"2024-01-15T17:57:32.144315+0000","last_unstale":"2024-01-15T17:57:33.415231+0000","last_undegraded":"2024-01-15T17:57:33.415231+0000","last_fullsized":"2024-01-15T17:57:33.415231+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140177+0000","last_change":"2024-01-15T17:57:32.157864+0000","last_active":"2024-01-15T17:57:33.140177+0000","last_peered":"2024-01-15T17:57:33.140177+0000","last_clean":"2024-01-15T17:57:33.140177+0000","last_became_active":"2024-01-15T17:57:32.157556+0000","last_became_peered":"2024-01-15T17:57:32.157556+0000","last_unstale":"2024-01-15T17:57:33.140177+0000","last_undegraded":"2024-01-15T17:57:33.140177+0000","last_fullsized":"2024-01-15T17:57:33.140177+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415132+0000","last_change":"2024-01-15T17:57:32.161393+0000","last_active":"2024-01-15T17:57:33.415132+0000","last_peered":"2024-01-15T17:57:33.415132+0000","last_clean":"2024-01-15T17:57:33.415132+0000","last_became_active":"2024-01-15T17:57:32.161079+0000","last_became_peered":"2024-01-15T17:57:32.161079+0000","last_unstale":"2024-01-15T17:57:33.415132+0000","last_undegraded":"2024-01-15T17:57:33.415132+0000","last_fullsized":"2024-01-15T17:57:33.415132+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414932+0000","last_change":"2024-01-15T17:57:32.145780+0000","last_active":"2024-01-15T17:57:33.414932+0000","last_peered":"2024-01-15T17:57:33.414932+0000","last_clean":"2024-01-15T17:57:33.414932+0000","last_became_active":"2024-01-15T17:57:32.145409+0000","last_became_peered":"2024-01-15T17:57:32.145409+0000","last_unstale":"2024-01-15T17:57:33.414932+0000","last_undegraded":"2024-01-15T17:57:33.414932+0000","last_fullsized":"2024-01-15T17:57:33.414932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186642+0000","last_change":"2024-01-15T17:57:32.144056+0000","last_active":"2024-01-15T17:57:33.186642+0000","last_peered":"2024-01-15T17:57:33.186642+0000","last_clean":"2024-01-15T17:57:33.186642+0000","last_became_active":"2024-01-15T17:57:32.143687+0000","last_became_peered":"2024-01-15T17:57:32.143687+0000","last_unstale":"2024-01-15T17:57:33.186642+0000","last_undegraded":"2024-01-15T17:57:33.186642+0000","last_fullsized":"2024-01-15T17:57:33.186642+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417307+0000","last_change":"2024-01-15T17:57:32.141036+0000","last_active":"2024-01-15T17:57:33.417307+0000","last_peered":"2024-01-15T17:57:33.417307+0000","last_clean":"2024-01-15T17:57:33.417307+0000","last_became_active":"2024-01-15T17:57:32.139707+0000","last_became_peered":"2024-01-15T17:57:32.139707+0000","last_unstale":"2024-01-15T17:57:33.417307+0000","last_undegraded":"2024-01-15T17:57:33.417307+0000","last_fullsized":"2024-01-15T17:57:33.417307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415834+0000","last_change":"2024-01-15T17:57:32.148492+0000","last_active":"2024-01-15T17:57:33.415834+0000","last_peered":"2024-01-15T17:57:33.415834+0000","last_clean":"2024-01-15T17:57:33.415834+0000","last_became_active":"2024-01-15T17:57:32.148169+0000","last_became_peered":"2024-01-15T17:57:32.148169+0000","last_unstale":"2024-01-15T17:57:33.415834+0000","last_undegraded":"2024-01-15T17:57:33.415834+0000","last_fullsized":"2024-01-15T17:57:33.415834+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:00.974439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.150453+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.150115+0000","last_became_peered":"2024-01-15T17:57:32.150115+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.137808+0000","last_change":"2024-01-15T17:57:32.148256+0000","last_active":"2024-01-15T17:57:33.137808+0000","last_peered":"2024-01-15T17:57:33.137808+0000","last_clean":"2024-01-15T17:57:33.137808+0000","last_became_active":"2024-01-15T17:57:32.147620+0000","last_became_peered":"2024-01-15T17:57:32.147620+0000","last_unstale":"2024-01-15T17:57:33.137808+0000","last_undegraded":"2024-01-15T17:57:33.137808+0000","last_fullsized":"2024-01-15T17:57:33.137808+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140840+0000","last_change":"2024-01-15T17:57:32.139828+0000","last_active":"2024-01-15T17:57:33.140840+0000","last_peered":"2024-01-15T17:57:33.140840+0000","last_clean":"2024-01-15T17:57:33.140840+0000","last_became_active":"2024-01-15T17:57:32.139522+0000","last_became_peered":"2024-01-15T17:57:32.139522+0000","last_unstale":"2024-01-15T17:57:33.140840+0000","last_undegraded":"2024-01-15T17:57:33.140840+0000","last_fullsized":"2024-01-15T17:57:33.140840+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416706+0000","last_change":"2024-01-15T17:57:32.140652+0000","last_active":"2024-01-15T17:57:33.416706+0000","last_peered":"2024-01-15T17:57:33.416706+0000","last_clean":"2024-01-15T17:57:33.416706+0000","last_became_active":"2024-01-15T17:57:32.140366+0000","last_became_peered":"2024-01-15T17:57:32.140366+0000","last_unstale":"2024-01-15T17:57:33.416706+0000","last_undegraded":"2024-01-15T17:57:33.416706+0000","last_fullsized":"2024-01-15T17:57:33.416706+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:47:13.862491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141414+0000","last_change":"2024-01-15T17:57:32.149137+0000","last_active":"2024-01-15T17:57:33.141414+0000","last_peered":"2024-01-15T17:57:33.141414+0000","last_clean":"2024-01-15T17:57:33.141414+0000","last_became_active":"2024-01-15T17:57:32.148658+0000","last_became_peered":"2024-01-15T17:57:32.148658+0000","last_unstale":"2024-01-15T17:57:33.141414+0000","last_undegraded":"2024-01-15T17:57:33.141414+0000","last_fullsized":"2024-01-15T17:57:33.141414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:33:18.642472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414742+0000","last_change":"2024-01-15T17:57:32.156212+0000","last_active":"2024-01-15T17:57:33.414742+0000","last_peered":"2024-01-15T17:57:33.414742+0000","last_clean":"2024-01-15T17:57:33.414742+0000","last_became_active":"2024-01-15T17:57:32.153278+0000","last_became_peered":"2024-01-15T17:57:32.153278+0000","last_unstale":"2024-01-15T17:57:33.414742+0000","last_undegraded":"2024-01-15T17:57:33.414742+0000","last_fullsized":"2024-01-15T17:57:33.414742+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417083+0000","last_change":"2024-01-15T17:57:32.147107+0000","last_active":"2024-01-15T17:57:33.417083+0000","last_peered":"2024-01-15T17:57:33.417083+0000","last_clean":"2024-01-15T17:57:33.417083+0000","last_became_active":"2024-01-15T17:57:32.146730+0000","last_became_peered":"2024-01-15T17:57:32.146730+0000","last_unstale":"2024-01-15T17:57:33.417083+0000","last_undegraded":"2024-01-15T17:57:33.417083+0000","last_fullsized":"2024-01-15T17:57:33.417083+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:12.778997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138213+0000","last_change":"2024-01-15T17:57:32.139188+0000","last_active":"2024-01-15T17:57:33.138213+0000","last_peered":"2024-01-15T17:57:33.138213+0000","last_clean":"2024-01-15T17:57:33.138213+0000","last_became_active":"2024-01-15T17:57:32.138898+0000","last_became_peered":"2024-01-15T17:57:32.138898+0000","last_unstale":"2024-01-15T17:57:33.138213+0000","last_undegraded":"2024-01-15T17:57:33.138213+0000","last_fullsized":"2024-01-15T17:57:33.138213+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416604+0000","last_change":"2024-01-15T17:57:32.148598+0000","last_active":"2024-01-15T17:57:33.416604+0000","last_peered":"2024-01-15T17:57:33.416604+0000","last_clean":"2024-01-15T17:57:33.416604+0000","last_became_active":"2024-01-15T17:57:32.148266+0000","last_became_peered":"2024-01-15T17:57:32.148266+0000","last_unstale":"2024-01-15T17:57:33.416604+0000","last_undegraded":"2024-01-15T17:57:33.416604+0000","last_fullsized":"2024-01-15T17:57:33.416604+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138790+0000","last_change":"2024-01-15T17:57:32.135029+0000","last_active":"2024-01-15T17:57:33.138790+0000","last_peered":"2024-01-15T17:57:33.138790+0000","last_clean":"2024-01-15T17:57:33.138790+0000","last_became_active":"2024-01-15T17:57:32.134762+0000","last_became_peered":"2024-01-15T17:57:32.134762+0000","last_unstale":"2024-01-15T17:57:33.138790+0000","last_undegraded":"2024-01-15T17:57:33.138790+0000","last_fullsized":"2024-01-15T17:57:33.138790+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418117+0000","last_change":"2024-01-15T17:57:32.153167+0000","last_active":"2024-01-15T17:57:33.418117+0000","last_peered":"2024-01-15T17:57:33.418117+0000","last_clean":"2024-01-15T17:57:33.418117+0000","last_became_active":"2024-01-15T17:57:32.151003+0000","last_became_peered":"2024-01-15T17:57:32.151003+0000","last_unstale":"2024-01-15T17:57:33.418117+0000","last_undegraded":"2024-01-15T17:57:33.418117+0000","last_fullsized":"2024-01-15T17:57:33.418117+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:15.239117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:32.142988+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:32.142613+0000","last_became_peered":"2024-01-15T17:57:32.142613+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136341+0000","last_change":"2024-01-15T17:57:32.134987+0000","last_active":"2024-01-15T17:57:33.136341+0000","last_peered":"2024-01-15T17:57:33.136341+0000","last_clean":"2024-01-15T17:57:33.136341+0000","last_became_active":"2024-01-15T17:57:32.134719+0000","last_became_peered":"2024-01-15T17:57:32.134719+0000","last_unstale":"2024-01-15T17:57:33.136341+0000","last_undegraded":"2024-01-15T17:57:33.136341+0000","last_fullsized":"2024-01-15T17:57:33.136341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418595+0000","last_change":"2024-01-15T17:57:32.153961+0000","last_active":"2024-01-15T17:57:33.418595+0000","last_peered":"2024-01-15T17:57:33.418595+0000","last_clean":"2024-01-15T17:57:33.418595+0000","last_became_active":"2024-01-15T17:57:32.151097+0000","last_became_peered":"2024-01-15T17:57:32.151097+0000","last_unstale":"2024-01-15T17:57:33.418595+0000","last_undegraded":"2024-01-15T17:57:33.418595+0000","last_fullsized":"2024-01-15T17:57:33.418595+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:44.586316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418195+0000","last_change":"2024-01-15T17:57:32.144509+0000","last_active":"2024-01-15T17:57:33.418195+0000","last_peered":"2024-01-15T17:57:33.418195+0000","last_clean":"2024-01-15T17:57:33.418195+0000","last_became_active":"2024-01-15T17:57:32.143921+0000","last_became_peered":"2024-01-15T17:57:32.143921+0000","last_unstale":"2024-01-15T17:57:33.418195+0000","last_undegraded":"2024-01-15T17:57:33.418195+0000","last_fullsized":"2024-01-15T17:57:33.418195+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138415+0000","last_change":"2024-01-15T17:57:32.137686+0000","last_active":"2024-01-15T17:57:33.138415+0000","last_peered":"2024-01-15T17:57:33.138415+0000","last_clean":"2024-01-15T17:57:33.138415+0000","last_became_active":"2024-01-15T17:57:32.137409+0000","last_became_peered":"2024-01-15T17:57:32.137409+0000","last_unstale":"2024-01-15T17:57:33.138415+0000","last_undegraded":"2024-01-15T17:57:33.138415+0000","last_fullsized":"2024-01-15T17:57:33.138415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414292+0000","last_change":"2024-01-15T17:57:32.129661+0000","last_active":"2024-01-15T17:57:33.414292+0000","last_peered":"2024-01-15T17:57:33.414292+0000","last_clean":"2024-01-15T17:57:33.414292+0000","last_became_active":"2024-01-15T17:57:32.129283+0000","last_became_peered":"2024-01-15T17:57:32.129283+0000","last_unstale":"2024-01-15T17:57:33.414292+0000","last_undegraded":"2024-01-15T17:57:33.414292+0000","last_fullsized":"2024-01-15T17:57:33.414292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414350+0000","last_change":"2024-01-15T17:57:32.147901+0000","last_active":"2024-01-15T17:57:33.414350+0000","last_peered":"2024-01-15T17:57:33.414350+0000","last_clean":"2024-01-15T17:57:33.414350+0000","last_became_active":"2024-01-15T17:57:32.147185+0000","last_became_peered":"2024-01-15T17:57:32.147185+0000","last_unstale":"2024-01-15T17:57:33.414350+0000","last_undegraded":"2024-01-15T17:57:33.414350+0000","last_fullsized":"2024-01-15T17:57:33.414350+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186262+0000","last_change":"2024-01-15T17:57:32.138254+0000","last_active":"2024-01-15T17:57:33.186262+0000","last_peered":"2024-01-15T17:57:33.186262+0000","last_clean":"2024-01-15T17:57:33.186262+0000","last_became_active":"2024-01-15T17:57:32.137849+0000","last_became_peered":"2024-01-15T17:57:32.137849+0000","last_unstale":"2024-01-15T17:57:33.186262+0000","last_undegraded":"2024-01-15T17:57:33.186262+0000","last_fullsized":"2024-01-15T17:57:33.186262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:12.340083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417690+0000","last_change":"2024-01-15T17:57:31.132496+0000","last_active":"2024-01-15T17:57:33.417690+0000","last_peered":"2024-01-15T17:57:33.417690+0000","last_clean":"2024-01-15T17:57:33.417690+0000","last_became_active":"2024-01-15T17:57:31.132095+0000","last_became_peered":"2024-01-15T17:57:31.132095+0000","last_unstale":"2024-01-15T17:57:33.417690+0000","last_undegraded":"2024-01-15T17:57:33.417690+0000","last_fullsized":"2024-01-15T17:57:33.417690+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418981+0000","last_change":"2024-01-15T17:57:32.154160+0000","last_active":"2024-01-15T17:57:33.418981+0000","last_peered":"2024-01-15T17:57:33.418981+0000","last_clean":"2024-01-15T17:57:33.418981+0000","last_became_active":"2024-01-15T17:57:32.153881+0000","last_became_peered":"2024-01-15T17:57:32.153881+0000","last_unstale":"2024-01-15T17:57:33.418981+0000","last_undegraded":"2024-01-15T17:57:33.418981+0000","last_fullsized":"2024-01-15T17:57:33.418981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414244+0000","last_change":"2024-01-15T17:57:31.143303+0000","last_active":"2024-01-15T17:57:33.414244+0000","last_peered":"2024-01-15T17:57:33.414244+0000","last_clean":"2024-01-15T17:57:33.414244+0000","last_became_active":"2024-01-15T17:57:31.142917+0000","last_became_peered":"2024-01-15T17:57:31.142917+0000","last_unstale":"2024-01-15T17:57:33.414244+0000","last_undegraded":"2024-01-15T17:57:33.414244+0000","last_fullsized":"2024-01-15T17:57:33.414244+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416836+0000","last_change":"2024-01-15T17:57:31.137389+0000","last_active":"2024-01-15T17:57:33.416836+0000","last_peered":"2024-01-15T17:57:33.416836+0000","last_clean":"2024-01-15T17:57:33.416836+0000","last_became_active":"2024-01-15T17:57:31.137115+0000","last_became_peered":"2024-01-15T17:57:31.137115+0000","last_unstale":"2024-01-15T17:57:33.416836+0000","last_undegraded":"2024-01-15T17:57:33.416836+0000","last_fullsized":"2024-01-15T17:57:33.416836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415574+0000","last_change":"2024-01-15T17:57:32.156574+0000","last_active":"2024-01-15T17:57:33.415574+0000","last_peered":"2024-01-15T17:57:33.415574+0000","last_clean":"2024-01-15T17:57:33.415574+0000","last_became_active":"2024-01-15T17:57:32.156179+0000","last_became_peered":"2024-01-15T17:57:32.156179+0000","last_unstale":"2024-01-15T17:57:33.415574+0000","last_undegraded":"2024-01-15T17:57:33.415574+0000","last_fullsized":"2024-01-15T17:57:33.415574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139052+0000","last_change":"2024-01-15T17:57:31.132492+0000","last_active":"2024-01-15T17:57:33.139052+0000","last_peered":"2024-01-15T17:57:33.139052+0000","last_clean":"2024-01-15T17:57:33.139052+0000","last_became_active":"2024-01-15T17:57:31.132141+0000","last_became_peered":"2024-01-15T17:57:31.132141+0000","last_unstale":"2024-01-15T17:57:33.139052+0000","last_undegraded":"2024-01-15T17:57:33.139052+0000","last_fullsized":"2024-01-15T17:57:33.139052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135758+0000","last_change":"2024-01-15T17:57:32.134222+0000","last_active":"2024-01-15T17:57:33.135758+0000","last_peered":"2024-01-15T17:57:33.135758+0000","last_clean":"2024-01-15T17:57:33.135758+0000","last_became_active":"2024-01-15T17:57:32.133920+0000","last_became_peered":"2024-01-15T17:57:32.133920+0000","last_unstale":"2024-01-15T17:57:33.135758+0000","last_undegraded":"2024-01-15T17:57:33.135758+0000","last_fullsized":"2024-01-15T17:57:33.135758+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:02.264148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415073+0000","last_change":"2024-01-15T17:57:31.144178+0000","last_active":"2024-01-15T17:57:33.415073+0000","last_peered":"2024-01-15T17:57:33.415073+0000","last_clean":"2024-01-15T17:57:33.415073+0000","last_became_active":"2024-01-15T17:57:31.143915+0000","last_became_peered":"2024-01-15T17:57:31.143915+0000","last_unstale":"2024-01-15T17:57:33.415073+0000","last_undegraded":"2024-01-15T17:57:33.415073+0000","last_fullsized":"2024-01-15T17:57:33.415073+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140200+0000","last_change":"2024-01-15T17:57:32.141022+0000","last_active":"2024-01-15T17:57:33.140200+0000","last_peered":"2024-01-15T17:57:33.140200+0000","last_clean":"2024-01-15T17:57:33.140200+0000","last_became_active":"2024-01-15T17:57:32.140122+0000","last_became_peered":"2024-01-15T17:57:32.140122+0000","last_unstale":"2024-01-15T17:57:33.140200+0000","last_undegraded":"2024-01-15T17:57:33.140200+0000","last_fullsized":"2024-01-15T17:57:33.140200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140234+0000","last_change":"2024-01-15T17:57:31.146237+0000","last_active":"2024-01-15T17:57:33.140234+0000","last_peered":"2024-01-15T17:57:33.140234+0000","last_clean":"2024-01-15T17:57:33.140234+0000","last_became_active":"2024-01-15T17:57:31.145808+0000","last_became_peered":"2024-01-15T17:57:31.145808+0000","last_unstale":"2024-01-15T17:57:33.140234+0000","last_undegraded":"2024-01-15T17:57:33.140234+0000","last_fullsized":"2024-01-15T17:57:33.140234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:05.346229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419713+0000","last_change":"2024-01-15T17:57:32.142125+0000","last_active":"2024-01-15T17:57:33.419713+0000","last_peered":"2024-01-15T17:57:33.419713+0000","last_clean":"2024-01-15T17:57:33.419713+0000","last_became_active":"2024-01-15T17:57:32.141806+0000","last_became_peered":"2024-01-15T17:57:32.141806+0000","last_unstale":"2024-01-15T17:57:33.419713+0000","last_undegraded":"2024-01-15T17:57:33.419713+0000","last_fullsized":"2024-01-15T17:57:33.419713+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136484+0000","last_change":"2024-01-15T17:57:31.131511+0000","last_active":"2024-01-15T17:57:33.136484+0000","last_peered":"2024-01-15T17:57:33.136484+0000","last_clean":"2024-01-15T17:57:33.136484+0000","last_became_active":"2024-01-15T17:57:31.131207+0000","last_became_peered":"2024-01-15T17:57:31.131207+0000","last_unstale":"2024-01-15T17:57:33.136484+0000","last_undegraded":"2024-01-15T17:57:33.136484+0000","last_fullsized":"2024-01-15T17:57:33.136484+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186851+0000","last_change":"2024-01-15T17:57:32.141464+0000","last_active":"2024-01-15T17:57:33.186851+0000","last_peered":"2024-01-15T17:57:33.186851+0000","last_clean":"2024-01-15T17:57:33.186851+0000","last_became_active":"2024-01-15T17:57:32.140204+0000","last_became_peered":"2024-01-15T17:57:32.140204+0000","last_unstale":"2024-01-15T17:57:33.186851+0000","last_undegraded":"2024-01-15T17:57:33.186851+0000","last_fullsized":"2024-01-15T17:57:33.186851+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:23.056372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415055+0000","last_change":"2024-01-15T17:57:31.140465+0000","last_active":"2024-01-15T17:57:33.415055+0000","last_peered":"2024-01-15T17:57:33.415055+0000","last_clean":"2024-01-15T17:57:33.415055+0000","last_became_active":"2024-01-15T17:57:31.140132+0000","last_became_peered":"2024-01-15T17:57:31.140132+0000","last_unstale":"2024-01-15T17:57:33.415055+0000","last_undegraded":"2024-01-15T17:57:33.415055+0000","last_fullsized":"2024-01-15T17:57:33.415055+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141003+0000","last_change":"2024-01-15T17:57:32.140539+0000","last_active":"2024-01-15T17:57:33.141003+0000","last_peered":"2024-01-15T17:57:33.141003+0000","last_clean":"2024-01-15T17:57:33.141003+0000","last_became_active":"2024-01-15T17:57:32.140117+0000","last_became_peered":"2024-01-15T17:57:32.140117+0000","last_unstale":"2024-01-15T17:57:33.141003+0000","last_undegraded":"2024-01-15T17:57:33.141003+0000","last_fullsized":"2024-01-15T17:57:33.141003+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:15.161573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418517+0000","last_change":"2024-01-15T17:57:31.134124+0000","last_active":"2024-01-15T17:57:33.418517+0000","last_peered":"2024-01-15T17:57:33.418517+0000","last_clean":"2024-01-15T17:57:33.418517+0000","last_became_active":"2024-01-15T17:57:31.133698+0000","last_became_peered":"2024-01-15T17:57:31.133698+0000","last_unstale":"2024-01-15T17:57:33.418517+0000","last_undegraded":"2024-01-15T17:57:33.418517+0000","last_fullsized":"2024-01-15T17:57:33.418517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.416094+0000","last_change":"2024-01-15T17:57:32.148065+0000","last_active":"2024-01-15T17:57:33.416094+0000","last_peered":"2024-01-15T17:57:33.416094+0000","last_clean":"2024-01-15T17:57:33.416094+0000","last_became_active":"2024-01-15T17:57:32.145370+0000","last_became_peered":"2024-01-15T17:57:32.145370+0000","last_unstale":"2024-01-15T17:57:33.416094+0000","last_undegraded":"2024-01-15T17:57:33.416094+0000","last_fullsized":"2024-01-15T17:57:33.416094+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:46.538517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138441+0000","last_change":"2024-01-15T17:57:31.128237+0000","last_active":"2024-01-15T17:57:33.138441+0000","last_peered":"2024-01-15T17:57:33.138441+0000","last_clean":"2024-01-15T17:57:33.138441+0000","last_became_active":"2024-01-15T17:57:31.127956+0000","last_became_peered":"2024-01-15T17:57:31.127956+0000","last_unstale":"2024-01-15T17:57:33.138441+0000","last_undegraded":"2024-01-15T17:57:33.138441+0000","last_fullsized":"2024-01-15T17:57:33.138441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141534+0000","last_change":"2024-01-15T17:57:32.150884+0000","last_active":"2024-01-15T17:57:33.141534+0000","last_peered":"2024-01-15T17:57:33.141534+0000","last_clean":"2024-01-15T17:57:33.141534+0000","last_became_active":"2024-01-15T17:57:32.150381+0000","last_became_peered":"2024-01-15T17:57:32.150381+0000","last_unstale":"2024-01-15T17:57:33.141534+0000","last_undegraded":"2024-01-15T17:57:33.141534+0000","last_fullsized":"2024-01-15T17:57:33.141534+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:23.429091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414222+0000","last_change":"2024-01-15T17:57:31.138127+0000","last_active":"2024-01-15T17:57:33.414222+0000","last_peered":"2024-01-15T17:57:33.414222+0000","last_clean":"2024-01-15T17:57:33.414222+0000","last_became_active":"2024-01-15T17:57:31.137579+0000","last_became_peered":"2024-01-15T17:57:31.137579+0000","last_unstale":"2024-01-15T17:57:33.414222+0000","last_undegraded":"2024-01-15T17:57:33.414222+0000","last_fullsized":"2024-01-15T17:57:33.414222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414356+0000","last_change":"2024-01-15T17:57:32.150432+0000","last_active":"2024-01-15T17:57:33.414356+0000","last_peered":"2024-01-15T17:57:33.414356+0000","last_clean":"2024-01-15T17:57:33.414356+0000","last_became_active":"2024-01-15T17:57:32.147572+0000","last_became_peered":"2024-01-15T17:57:32.147572+0000","last_unstale":"2024-01-15T17:57:33.414356+0000","last_undegraded":"2024-01-15T17:57:33.414356+0000","last_fullsized":"2024-01-15T17:57:33.414356+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414449+0000","last_change":"2024-01-15T17:57:31.134499+0000","last_active":"2024-01-15T17:57:33.414449+0000","last_peered":"2024-01-15T17:57:33.414449+0000","last_clean":"2024-01-15T17:57:33.414449+0000","last_became_active":"2024-01-15T17:57:31.131997+0000","last_became_peered":"2024-01-15T17:57:31.131997+0000","last_unstale":"2024-01-15T17:57:33.414449+0000","last_undegraded":"2024-01-15T17:57:33.414449+0000","last_fullsized":"2024-01-15T17:57:33.414449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:11.214052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139065+0000","last_change":"2024-01-15T17:57:32.151412+0000","last_active":"2024-01-15T17:57:33.139065+0000","last_peered":"2024-01-15T17:57:33.139065+0000","last_clean":"2024-01-15T17:57:33.139065+0000","last_became_active":"2024-01-15T17:57:32.151061+0000","last_became_peered":"2024-01-15T17:57:32.151061+0000","last_unstale":"2024-01-15T17:57:33.139065+0000","last_undegraded":"2024-01-15T17:57:33.139065+0000","last_fullsized":"2024-01-15T17:57:33.139065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186594+0000","last_change":"2024-01-15T17:57:31.133132+0000","last_active":"2024-01-15T17:57:33.186594+0000","last_peered":"2024-01-15T17:57:33.186594+0000","last_clean":"2024-01-15T17:57:33.186594+0000","last_became_active":"2024-01-15T17:57:31.131606+0000","last_became_peered":"2024-01-15T17:57:31.131606+0000","last_unstale":"2024-01-15T17:57:33.186594+0000","last_undegraded":"2024-01-15T17:57:33.186594+0000","last_fullsized":"2024-01-15T17:57:33.186594+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:58:03.663628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139362+0000","last_change":"2024-01-15T17:57:32.152397+0000","last_active":"2024-01-15T17:57:33.139362+0000","last_peered":"2024-01-15T17:57:33.139362+0000","last_clean":"2024-01-15T17:57:33.139362+0000","last_became_active":"2024-01-15T17:57:32.152096+0000","last_became_peered":"2024-01-15T17:57:32.152096+0000","last_unstale":"2024-01-15T17:57:33.139362+0000","last_undegraded":"2024-01-15T17:57:33.139362+0000","last_fullsized":"2024-01-15T17:57:33.139362+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417449+0000","last_change":"2024-01-15T17:57:31.127107+0000","last_active":"2024-01-15T17:57:33.417449+0000","last_peered":"2024-01-15T17:57:33.417449+0000","last_clean":"2024-01-15T17:57:33.417449+0000","last_became_active":"2024-01-15T17:57:31.126871+0000","last_became_peered":"2024-01-15T17:57:31.126871+0000","last_unstale":"2024-01-15T17:57:33.417449+0000","last_undegraded":"2024-01-15T17:57:33.417449+0000","last_fullsized":"2024-01-15T17:57:33.417449+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:20.856850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.140382+0000","last_change":"2024-01-15T17:57:32.150858+0000","last_active":"2024-01-15T17:57:33.140382+0000","last_peered":"2024-01-15T17:57:33.140382+0000","last_clean":"2024-01-15T17:57:33.140382+0000","last_became_active":"2024-01-15T17:57:32.150312+0000","last_became_peered":"2024-01-15T17:57:32.150312+0000","last_unstale":"2024-01-15T17:57:33.140382+0000","last_undegraded":"2024-01-15T17:57:33.140382+0000","last_fullsized":"2024-01-15T17:57:33.140382+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136681+0000","last_change":"2024-01-15T17:57:31.136226+0000","last_active":"2024-01-15T17:57:33.136681+0000","last_peered":"2024-01-15T17:57:33.136681+0000","last_clean":"2024-01-15T17:57:33.136681+0000","last_became_active":"2024-01-15T17:57:31.135879+0000","last_became_peered":"2024-01-15T17:57:31.135879+0000","last_unstale":"2024-01-15T17:57:33.136681+0000","last_undegraded":"2024-01-15T17:57:33.136681+0000","last_fullsized":"2024-01-15T17:57:33.136681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:59.848408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.149505+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.144115+0000","last_became_peered":"2024-01-15T17:57:32.144115+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:58.154470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415379+0000","last_change":"2024-01-15T17:57:31.130353+0000","last_active":"2024-01-15T17:57:33.415379+0000","last_peered":"2024-01-15T17:57:33.415379+0000","last_clean":"2024-01-15T17:57:33.415379+0000","last_became_active":"2024-01-15T17:57:31.130063+0000","last_became_peered":"2024-01-15T17:57:31.130063+0000","last_unstale":"2024-01-15T17:57:33.415379+0000","last_undegraded":"2024-01-15T17:57:33.415379+0000","last_fullsized":"2024-01-15T17:57:33.415379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:42.825666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141047+0000","last_change":"2024-01-15T17:57:32.148976+0000","last_active":"2024-01-15T17:57:33.141047+0000","last_peered":"2024-01-15T17:57:33.141047+0000","last_clean":"2024-01-15T17:57:33.141047+0000","last_became_active":"2024-01-15T17:57:32.148285+0000","last_became_peered":"2024-01-15T17:57:32.148285+0000","last_unstale":"2024-01-15T17:57:33.141047+0000","last_undegraded":"2024-01-15T17:57:33.141047+0000","last_fullsized":"2024-01-15T17:57:33.141047+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:09.548950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-15T17:57:32.126659+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:57:32.126659+0000","last_peered":"2024-01-15T17:57:32.126659+0000","last_clean":"2024-01-15T17:57:32.126659+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:57:32.126659+0000","last_undegraded":"2024-01-15T17:57:32.126659+0000","last_fullsized":"2024-01-15T17:57:32.126659+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138248+0000","last_change":"2024-01-15T17:57:31.127921+0000","last_active":"2024-01-15T17:57:33.138248+0000","last_peered":"2024-01-15T17:57:33.138248+0000","last_clean":"2024-01-15T17:57:33.138248+0000","last_became_active":"2024-01-15T17:57:31.126423+0000","last_became_peered":"2024-01-15T17:57:31.126423+0000","last_unstale":"2024-01-15T17:57:33.138248+0000","last_undegraded":"2024-01-15T17:57:33.138248+0000","last_fullsized":"2024-01-15T17:57:33.138248+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:43:31.770732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418628+0000","last_change":"2024-01-15T17:57:32.151647+0000","last_active":"2024-01-15T17:57:33.418628+0000","last_peered":"2024-01-15T17:57:33.418628+0000","last_clean":"2024-01-15T17:57:33.418628+0000","last_became_active":"2024-01-15T17:57:32.149470+0000","last_became_peered":"2024-01-15T17:57:32.149470+0000","last_unstale":"2024-01-15T17:57:33.418628+0000","last_undegraded":"2024-01-15T17:57:33.418628+0000","last_fullsized":"2024-01-15T17:57:33.418628+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:17.187347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139414+0000","last_change":"2024-01-15T17:57:31.141208+0000","last_active":"2024-01-15T17:57:33.139414+0000","last_peered":"2024-01-15T17:57:33.139414+0000","last_clean":"2024-01-15T17:57:33.139414+0000","last_became_active":"2024-01-15T17:57:31.140756+0000","last_became_peered":"2024-01-15T17:57:31.140756+0000","last_unstale":"2024-01-15T17:57:33.139414+0000","last_undegraded":"2024-01-15T17:57:33.139414+0000","last_fullsized":"2024-01-15T17:57:33.139414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:17:40.803403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415471+0000","last_change":"2024-01-15T17:57:32.150522+0000","last_active":"2024-01-15T17:57:33.415471+0000","last_peered":"2024-01-15T17:57:33.415471+0000","last_clean":"2024-01-15T17:57:33.415471+0000","last_became_active":"2024-01-15T17:57:32.150224+0000","last_became_peered":"2024-01-15T17:57:32.150224+0000","last_unstale":"2024-01-15T17:57:33.415471+0000","last_undegraded":"2024-01-15T17:57:33.415471+0000","last_fullsized":"2024-01-15T17:57:33.415471+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:03.799210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414399+0000","last_change":"2024-01-15T17:57:31.127558+0000","last_active":"2024-01-15T17:57:33.414399+0000","last_peered":"2024-01-15T17:57:33.414399+0000","last_clean":"2024-01-15T17:57:33.414399+0000","last_became_active":"2024-01-15T17:57:31.125325+0000","last_became_peered":"2024-01-15T17:57:31.125325+0000","last_unstale":"2024-01-15T17:57:33.414399+0000","last_undegraded":"2024-01-15T17:57:33.414399+0000","last_fullsized":"2024-01-15T17:57:33.414399+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:33:56.231915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136344+0000","last_change":"2024-01-15T17:57:32.134693+0000","last_active":"2024-01-15T17:57:33.136344+0000","last_peered":"2024-01-15T17:57:33.136344+0000","last_clean":"2024-01-15T17:57:33.136344+0000","last_became_active":"2024-01-15T17:57:32.133773+0000","last_became_peered":"2024-01-15T17:57:32.133773+0000","last_unstale":"2024-01-15T17:57:33.136344+0000","last_undegraded":"2024-01-15T17:57:33.136344+0000","last_fullsized":"2024-01-15T17:57:33.136344+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138654+0000","last_change":"2024-01-15T17:57:31.133660+0000","last_active":"2024-01-15T17:57:33.138654+0000","last_peered":"2024-01-15T17:57:33.138654+0000","last_clean":"2024-01-15T17:57:33.138654+0000","last_became_active":"2024-01-15T17:57:31.132627+0000","last_became_peered":"2024-01-15T17:57:31.132627+0000","last_unstale":"2024-01-15T17:57:33.138654+0000","last_undegraded":"2024-01-15T17:57:33.138654+0000","last_fullsized":"2024-01-15T17:57:33.138654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186541+0000","last_change":"2024-01-15T17:57:32.147182+0000","last_active":"2024-01-15T17:57:33.186541+0000","last_peered":"2024-01-15T17:57:33.186541+0000","last_clean":"2024-01-15T17:57:33.186541+0000","last_became_active":"2024-01-15T17:57:32.146838+0000","last_became_peered":"2024-01-15T17:57:32.146838+0000","last_unstale":"2024-01-15T17:57:33.186541+0000","last_undegraded":"2024-01-15T17:57:33.186541+0000","last_fullsized":"2024-01-15T17:57:33.186541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:53.156782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136810+0000","last_change":"2024-01-15T17:57:31.126189+0000","last_active":"2024-01-15T17:57:33.136810+0000","last_peered":"2024-01-15T17:57:33.136810+0000","last_clean":"2024-01-15T17:57:33.136810+0000","last_became_active":"2024-01-15T17:57:31.125519+0000","last_became_peered":"2024-01-15T17:57:31.125519+0000","last_unstale":"2024-01-15T17:57:33.136810+0000","last_undegraded":"2024-01-15T17:57:33.136810+0000","last_fullsized":"2024-01-15T17:57:33.136810+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139796+0000","last_change":"2024-01-15T17:57:32.137059+0000","last_active":"2024-01-15T17:57:33.139796+0000","last_peered":"2024-01-15T17:57:33.139796+0000","last_clean":"2024-01-15T17:57:33.139796+0000","last_became_active":"2024-01-15T17:57:32.136012+0000","last_became_peered":"2024-01-15T17:57:32.136012+0000","last_unstale":"2024-01-15T17:57:33.139796+0000","last_undegraded":"2024-01-15T17:57:33.139796+0000","last_fullsized":"2024-01-15T17:57:33.139796+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:51:14.134719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141434+0000","last_change":"2024-01-15T17:57:31.139980+0000","last_active":"2024-01-15T17:57:33.141434+0000","last_peered":"2024-01-15T17:57:33.141434+0000","last_clean":"2024-01-15T17:57:33.141434+0000","last_became_active":"2024-01-15T17:57:31.139489+0000","last_became_peered":"2024-01-15T17:57:31.139489+0000","last_unstale":"2024-01-15T17:57:33.141434+0000","last_undegraded":"2024-01-15T17:57:33.141434+0000","last_fullsized":"2024-01-15T17:57:33.141434+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:54.272945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136716+0000","last_change":"2024-01-15T17:57:32.130379+0000","last_active":"2024-01-15T17:57:33.136716+0000","last_peered":"2024-01-15T17:57:33.136716+0000","last_clean":"2024-01-15T17:57:33.136716+0000","last_became_active":"2024-01-15T17:57:32.129645+0000","last_became_peered":"2024-01-15T17:57:32.129645+0000","last_unstale":"2024-01-15T17:57:33.136716+0000","last_undegraded":"2024-01-15T17:57:33.136716+0000","last_fullsized":"2024-01-15T17:57:33.136716+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:34.417066+0000","last_change":"2024-01-15T17:57:31.146382+0000","last_active":"2024-01-15T17:57:34.417066+0000","last_peered":"2024-01-15T17:57:34.417066+0000","last_clean":"2024-01-15T17:57:34.417066+0000","last_became_active":"2024-01-15T17:57:31.146119+0000","last_became_peered":"2024-01-15T17:57:31.146119+0000","last_unstale":"2024-01-15T17:57:34.417066+0000","last_undegraded":"2024-01-15T17:57:34.417066+0000","last_fullsized":"2024-01-15T17:57:34.417066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:30.690538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.418481+0000","last_change":"2024-01-15T17:57:32.144183+0000","last_active":"2024-01-15T17:57:33.418481+0000","last_peered":"2024-01-15T17:57:33.418481+0000","last_clean":"2024-01-15T17:57:33.418481+0000","last_became_active":"2024-01-15T17:57:32.143880+0000","last_became_peered":"2024-01-15T17:57:32.143880+0000","last_unstale":"2024-01-15T17:57:33.418481+0000","last_undegraded":"2024-01-15T17:57:33.418481+0000","last_fullsized":"2024-01-15T17:57:33.418481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:09.173713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415224+0000","last_change":"2024-01-15T17:57:31.144154+0000","last_active":"2024-01-15T17:57:33.415224+0000","last_peered":"2024-01-15T17:57:33.415224+0000","last_clean":"2024-01-15T17:57:33.415224+0000","last_became_active":"2024-01-15T17:57:31.143857+0000","last_became_peered":"2024-01-15T17:57:31.143857+0000","last_unstale":"2024-01-15T17:57:33.415224+0000","last_undegraded":"2024-01-15T17:57:33.415224+0000","last_fullsized":"2024-01-15T17:57:33.415224+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:41:45.412486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419365+0000","last_change":"2024-01-15T17:57:32.142460+0000","last_active":"2024-01-15T17:57:33.419365+0000","last_peered":"2024-01-15T17:57:33.419365+0000","last_clean":"2024-01-15T17:57:33.419365+0000","last_became_active":"2024-01-15T17:57:32.141816+0000","last_became_peered":"2024-01-15T17:57:32.141816+0000","last_unstale":"2024-01-15T17:57:33.419365+0000","last_undegraded":"2024-01-15T17:57:33.419365+0000","last_fullsized":"2024-01-15T17:57:33.419365+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:25:06.472260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139202+0000","last_change":"2024-01-15T17:57:31.139784+0000","last_active":"2024-01-15T17:57:33.139202+0000","last_peered":"2024-01-15T17:57:33.139202+0000","last_clean":"2024-01-15T17:57:33.139202+0000","last_became_active":"2024-01-15T17:57:31.138181+0000","last_became_peered":"2024-01-15T17:57:31.138181+0000","last_unstale":"2024-01-15T17:57:33.139202+0000","last_undegraded":"2024-01-15T17:57:33.139202+0000","last_fullsized":"2024-01-15T17:57:33.139202+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:05:50.662857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.186323+0000","last_change":"2024-01-15T17:57:32.130289+0000","last_active":"2024-01-15T17:57:33.186323+0000","last_peered":"2024-01-15T17:57:33.186323+0000","last_clean":"2024-01-15T17:57:33.186323+0000","last_became_active":"2024-01-15T17:57:32.129807+0000","last_became_peered":"2024-01-15T17:57:32.129807+0000","last_unstale":"2024-01-15T17:57:33.186323+0000","last_undegraded":"2024-01-15T17:57:33.186323+0000","last_fullsized":"2024-01-15T17:57:33.186323+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:43.951697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.135874+0000","last_change":"2024-01-15T17:57:31.132917+0000","last_active":"2024-01-15T17:57:33.135874+0000","last_peered":"2024-01-15T17:57:33.135874+0000","last_clean":"2024-01-15T17:57:33.135874+0000","last_became_active":"2024-01-15T17:57:31.132650+0000","last_became_peered":"2024-01-15T17:57:31.132650+0000","last_unstale":"2024-01-15T17:57:33.135874+0000","last_undegraded":"2024-01-15T17:57:33.135874+0000","last_fullsized":"2024-01-15T17:57:33.135874+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.415920+0000","last_change":"2024-01-15T17:57:32.136009+0000","last_active":"2024-01-15T17:57:33.415920+0000","last_peered":"2024-01-15T17:57:33.415920+0000","last_clean":"2024-01-15T17:57:33.415920+0000","last_became_active":"2024-01-15T17:57:32.135200+0000","last_became_peered":"2024-01-15T17:57:32.135200+0000","last_unstale":"2024-01-15T17:57:33.415920+0000","last_undegraded":"2024-01-15T17:57:33.415920+0000","last_fullsized":"2024-01-15T17:57:33.415920+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:32:58.832507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.136090+0000","last_change":"2024-01-15T17:57:31.130465+0000","last_active":"2024-01-15T17:57:33.136090+0000","last_peered":"2024-01-15T17:57:33.136090+0000","last_clean":"2024-01-15T17:57:33.136090+0000","last_became_active":"2024-01-15T17:57:31.130032+0000","last_became_peered":"2024-01-15T17:57:31.130032+0000","last_unstale":"2024-01-15T17:57:33.136090+0000","last_undegraded":"2024-01-15T17:57:33.136090+0000","last_fullsized":"2024-01-15T17:57:33.136090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417702+0000","last_change":"2024-01-15T17:57:32.154915+0000","last_active":"2024-01-15T17:57:33.417702+0000","last_peered":"2024-01-15T17:57:33.417702+0000","last_clean":"2024-01-15T17:57:33.417702+0000","last_became_active":"2024-01-15T17:57:32.154599+0000","last_became_peered":"2024-01-15T17:57:32.154599+0000","last_unstale":"2024-01-15T17:57:33.417702+0000","last_undegraded":"2024-01-15T17:57:33.417702+0000","last_fullsized":"2024-01-15T17:57:33.417702+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:22.724742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141769+0000","last_change":"2024-01-15T17:57:31.143380+0000","last_active":"2024-01-15T17:57:33.141769+0000","last_peered":"2024-01-15T17:57:33.141769+0000","last_clean":"2024-01-15T17:57:33.141769+0000","last_became_active":"2024-01-15T17:57:31.143046+0000","last_became_peered":"2024-01-15T17:57:31.143046+0000","last_unstale":"2024-01-15T17:57:33.141769+0000","last_undegraded":"2024-01-15T17:57:33.141769+0000","last_fullsized":"2024-01-15T17:57:33.141769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:11.652966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.414379+0000","last_change":"2024-01-15T17:57:32.149247+0000","last_active":"2024-01-15T17:57:33.414379+0000","last_peered":"2024-01-15T17:57:33.414379+0000","last_clean":"2024-01-15T17:57:33.414379+0000","last_became_active":"2024-01-15T17:57:32.148620+0000","last_became_peered":"2024-01-15T17:57:32.148620+0000","last_unstale":"2024-01-15T17:57:33.414379+0000","last_undegraded":"2024-01-15T17:57:33.414379+0000","last_fullsized":"2024-01-15T17:57:33.414379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:49.675012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139434+0000","last_change":"2024-01-15T17:57:32.159609+0000","last_active":"2024-01-15T17:57:33.139434+0000","last_peered":"2024-01-15T17:57:33.139434+0000","last_clean":"2024-01-15T17:57:33.139434+0000","last_became_active":"2024-01-15T17:57:32.155889+0000","last_became_peered":"2024-01-15T17:57:32.155889+0000","last_unstale":"2024-01-15T17:57:33.139434+0000","last_undegraded":"2024-01-15T17:57:33.139434+0000","last_fullsized":"2024-01-15T17:57:33.139434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:07:37.295710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.138536+0000","last_change":"2024-01-15T17:57:31.144524+0000","last_active":"2024-01-15T17:57:33.138536+0000","last_peered":"2024-01-15T17:57:33.138536+0000","last_clean":"2024-01-15T17:57:33.138536+0000","last_became_active":"2024-01-15T17:57:31.143299+0000","last_became_peered":"2024-01-15T17:57:31.143299+0000","last_unstale":"2024-01-15T17:57:33.138536+0000","last_undegraded":"2024-01-15T17:57:33.138536+0000","last_fullsized":"2024-01-15T17:57:33.138536+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:01:02.673180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.139433+0000","last_change":"2024-01-15T17:57:32.151244+0000","last_active":"2024-01-15T17:57:33.139433+0000","last_peered":"2024-01-15T17:57:33.139433+0000","last_clean":"2024-01-15T17:57:33.139433+0000","last_became_active":"2024-01-15T17:57:32.150834+0000","last_became_peered":"2024-01-15T17:57:32.150834+0000","last_unstale":"2024-01-15T17:57:33.139433+0000","last_undegraded":"2024-01-15T17:57:33.139433+0000","last_fullsized":"2024-01-15T17:57:33.139433+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:01.571223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.141240+0000","last_change":"2024-01-15T17:57:31.144722+0000","last_active":"2024-01-15T17:57:33.141240+0000","last_peered":"2024-01-15T17:57:33.141240+0000","last_clean":"2024-01-15T17:57:33.141240+0000","last_became_active":"2024-01-15T17:57:31.144378+0000","last_became_peered":"2024-01-15T17:57:31.144378+0000","last_unstale":"2024-01-15T17:57:33.141240+0000","last_undegraded":"2024-01-15T17:57:33.141240+0000","last_fullsized":"2024-01-15T17:57:33.141240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:47:48.548981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419383+0000","last_change":"2024-01-15T17:57:32.142742+0000","last_active":"2024-01-15T17:57:33.419383+0000","last_peered":"2024-01-15T17:57:33.419383+0000","last_clean":"2024-01-15T17:57:33.419383+0000","last_became_active":"2024-01-15T17:57:32.142372+0000","last_became_peered":"2024-01-15T17:57:32.142372+0000","last_unstale":"2024-01-15T17:57:33.419383+0000","last_undegraded":"2024-01-15T17:57:33.419383+0000","last_fullsized":"2024-01-15T17:57:33.419383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417735+0000","last_change":"2024-01-15T17:57:31.128188+0000","last_active":"2024-01-15T17:57:33.417735+0000","last_peered":"2024-01-15T17:57:33.417735+0000","last_clean":"2024-01-15T17:57:33.417735+0000","last_became_active":"2024-01-15T17:57:31.127089+0000","last_became_peered":"2024-01-15T17:57:31.127089+0000","last_unstale":"2024-01-15T17:57:33.417735+0000","last_undegraded":"2024-01-15T17:57:33.417735+0000","last_fullsized":"2024-01-15T17:57:33.417735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:55.830216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.417498+0000","last_change":"2024-01-15T17:57:32.142205+0000","last_active":"2024-01-15T17:57:33.417498+0000","last_peered":"2024-01-15T17:57:33.417498+0000","last_clean":"2024-01-15T17:57:33.417498+0000","last_became_active":"2024-01-15T17:57:32.139881+0000","last_became_peered":"2024-01-15T17:57:32.139881+0000","last_unstale":"2024-01-15T17:57:33.417498+0000","last_undegraded":"2024-01-15T17:57:33.417498+0000","last_fullsized":"2024-01-15T17:57:33.417498+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:31.093206+0000","last_clean_scrub_stamp":"2024-01-15T17:57:31.093206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:55.398975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-15T17:57:33.419783+0000","last_change":"2024-01-15T17:57:31.140404+0000","last_active":"2024-01-15T17:57:33.419783+0000","last_peered":"2024-01-15T17:57:33.419783+0000","last_clean":"2024-01-15T17:57:33.419783+0000","last_became_active":"2024-01-15T17:57:31.140098+0000","last_became_peered":"2024-01-15T17:57:31.140098+0000","last_unstale":"2024-01-15T17:57:33.419783+0000","last_undegraded":"2024-01-15T17:57:33.419783+0000","last_fullsized":"2024-01-15T17:57:33.419783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:29.967175+0000","last_clean_scrub_stamp":"2024-01-15T17:57:29.967175+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:54.493975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28020,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343820,"statfs":{"total":96636764160,"available":96608071680,"internally_reserved":0,"allocated":823296,"data_stored":669752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.8739999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.355}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85999999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.198}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5509999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.835}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":78892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6539999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.556}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1070000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1429999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.266}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":237568,"data_stored":78928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95099999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.611}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9339999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":237568,"data_stored":80414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3160000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2450000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90000000000000002}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":225280,"data_stored":78802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0680000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5150000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5950000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.22}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6619999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":233472,"data_stored":78962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1930000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.139}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32128,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339712,"statfs":{"total":96636764160,"available":96603865088,"internally_reserved":0,"allocated":831488,"data_stored":669808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0509999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92700000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8260000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3009999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:57:42.695 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-15T17:57:42.695 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-15T17:57:42.695 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-15T17:57:42.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-15T17:57:42.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.951+0000 7f57dbe00640 1 Processor -- start 2024-01-15T17:57:42.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.951+0000 7f57dbe00640 1 -- start start 2024-01-15T17:57:42.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57dbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57dc15ec60 0x7f57dc15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57dbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f57dc06cad0 0x7f57dc06cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57dbe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57dc0716d0 con 0x7f57dc141880 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57dbe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57dc071f20 con 0x7f57dc15ec60 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57dbe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57dc06c6b0 con 0x7f57dc06cad0 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57da000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57dc15ec60 0x7f57dc15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57da000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57dc15ec60 0x7f57dc15f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46012/0 (socket says 172.21.15.2:46012) 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.952+0000 7f57da000640 1 -- 172.21.15.2:0/2517034956 learned_addr learned my addr 172.21.15.2:0/2517034956 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.953+0000 7f57db400640 1 --2- 172.21.15.2:0/2517034956 >> v2:172.21.15.137:3300/0 conn(0x7f57dc06cad0 0x7f57dc06cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.953+0000 7f57da000640 1 -- 172.21.15.2:0/2517034956 >> v2:172.21.15.137:3300/0 conn(0x7f57dc06cad0 msgr2=0x7f57dc06cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.953+0000 7f57da000640 1 --2- 172.21.15.2:0/2517034956 >> v2:172.21.15.137:3300/0 conn(0x7f57dc06cad0 0x7f57dc06cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.953+0000 7f57da000640 1 -- 172.21.15.2:0/2517034956 >> v1:172.21.15.137:6789/0 conn(0x7f57dc141880 legacy=0x7f57dc070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:42.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.953+0000 7f57da000640 1 -- 172.21.15.2:0/2517034956 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc071960 con 0x7f57dc15ec60 2024-01-15T17:57:42.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.954+0000 7f57da000640 1 --2- 172.21.15.2:0/2517034956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc15ec60 0x7f57dc15f0e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f57dc070060 tx=0x7f57c402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50b8b81fdb9ef2a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:42.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.954+0000 7f57d9600640 1 -- 172.21.15.2:0/2517034956 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c40317e0 con 0x7f57dc15ec60 2024-01-15T17:57:42.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.954+0000 7f57d9600640 1 -- 172.21.15.2:0/2517034956 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c4031940 con 0x7f57dc15ec60 2024-01-15T17:57:42.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.954+0000 7f57d9600640 1 -- 172.21.15.2:0/2517034956 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c4031cd0 con 0x7f57dc15ec60 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 -- 172.21.15.2:0/2517034956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc15ec60 msgr2=0x7f57dc15f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 --2- 172.21.15.2:0/2517034956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc15ec60 0x7f57dc15f0e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f57dc070060 tx=0x7f57c402b880 comp rx=0 tx=0).stop 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 -- 172.21.15.2:0/2517034956 shutdown_connections 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 --2- 172.21.15.2:0/2517034956 >> v2:172.21.15.137:3300/0 conn(0x7f57dc06cad0 0x7f57dc06cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 --2- 172.21.15.2:0/2517034956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc15ec60 0x7f57dc15f0e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 -- 172.21.15.2:0/2517034956 >> 172.21.15.2:0/2517034956 conn(0x7f57dc0b0840 msgr2=0x7f57dc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 -- 172.21.15.2:0/2517034956 shutdown_connections 2024-01-15T17:57:42.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.955+0000 7f57dbe00640 1 -- 172.21.15.2:0/2517034956 wait complete. 2024-01-15T17:57:42.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.957+0000 7f57dbe00640 1 Processor -- start 2024-01-15T17:57:42.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.957+0000 7f57dbe00640 1 -- start start 2024-01-15T17:57:42.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 0x7f57dc15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57dbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57dc141880 0x7f57dc143c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57dbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57dc15ec60 0x7f57dc13df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57dbe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc159900 con 0x7f57dc06cad0 2024-01-15T17:57:42.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57dbe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc159be0 con 0x7f57dc141880 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57dbe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57dc159ec0 con 0x7f57dc15ec60 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 0x7f57dc15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 0x7f57dc15a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46014/0 (socket says 172.21.15.2:46014) 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 -- 172.21.15.2:0/1273205987 learned_addr learned my addr 172.21.15.2:0/1273205987 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57da000640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57dc141880 0x7f57dc143c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57db400640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57dc15ec60 0x7f57dc13df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 -- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57dc15ec60 msgr2=0x7f57dc13df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57dc15ec60 0x7f57dc13df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 -- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57dc141880 msgr2=0x7f57dc143c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57dc141880 0x7f57dc143c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57daa00640 1 -- 172.21.15.2:0/1273205987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c40094d0 con 0x7f57dc06cad0 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.958+0000 7f57da000640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57dc141880 0x7f57dc143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:42.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.959+0000 7f57daa00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 0x7f57dc15a9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00c7a0 tx=0x7f57cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.959+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc003d80 con 0x7f57dc06cad0 2024-01-15T17:57:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.959+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57cc005070 con 0x7f57dc06cad0 2024-01-15T17:57:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.959+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57cc005460 con 0x7f57dc06cad0 2024-01-15T17:57:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.959+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57dc15a1a0 con 0x7f57dc06cad0 2024-01-15T17:57:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.959+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57dc15aee0 con 0x7f57dc06cad0 2024-01-15T17:57:42.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.960+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57dc074770 con 0x7f57dc06cad0 2024-01-15T17:57:42.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.961+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f57cc0055c0 con 0x7f57dc06cad0 2024-01-15T17:57:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.962+0000 7f57d2a00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57b406b2c0 0x7f57b406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.962+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57cc0890d0 con 0x7f57dc06cad0 2024-01-15T17:57:42.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.963+0000 7f57da000640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57b406b2c0 0x7f57b406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:42.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.964+0000 7f57da000640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57b406b2c0 0x7f57b406d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57c4036000 tx=0x7f57c403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:42.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:42.966+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57cc053f30 con 0x7f57dc06cad0 2024-01-15T17:57:43.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.202+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f57dc06e1c0 con 0x7f57dc06cad0 2024-01-15T17:57:43.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.202+0000 7f57d2a00640 1 -- 172.21.15.2:0/1273205987 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f57cc057550 con 0x7f57dc06cad0 2024-01-15T17:57:43.206 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:43.206 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-15T17:57:43.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57b406b2c0 msgr2=0x7f57b406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57b406b2c0 0x7f57b406d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57c4036000 tx=0x7f57c403d040 comp rx=0 tx=0).stop 2024-01-15T17:57:43.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 msgr2=0x7f57dc15a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 0x7f57dc15a9a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57cc00c7a0 tx=0x7f57cc00cc60 comp rx=0 tx=0).stop 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 shutdown_connections 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57b406b2c0 0x7f57b406d780 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57dc15ec60 0x7f57dc13df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57dc141880 0x7f57dc143c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 --2- 172.21.15.2:0/1273205987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57dc06cad0 0x7f57dc15a9a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.205+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 >> 172.21.15.2:0/1273205987 conn(0x7f57dc0b0840 msgr2=0x7f57dc0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.206+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 shutdown_connections 2024-01-15T17:57:43.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.206+0000 7f57dbe00640 1 -- 172.21.15.2:0/1273205987 wait complete. 2024-01-15T17:57:43.228 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-15T17:57:43.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:43.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.443+0000 7f0752800640 1 Processor -- start 2024-01-15T17:57:43.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.444+0000 7f0752800640 1 -- start start 2024-01-15T17:57:43.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0752800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f074c141880 0x7f074c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0752800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c15f0e0 0x7f074c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0752800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f074c06dae0 con 0x7f074c15ec60 2024-01-15T17:57:43.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0752800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f074c071f20 con 0x7f074c141880 2024-01-15T17:57:43.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0752800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c06c6b0 con 0x7f074c15f0e0 2024-01-15T17:57:43.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0751400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f074c141880 0x7f074c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0751400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f074c141880 0x7f074c070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46028/0 (socket says 172.21.15.2:46028) 2024-01-15T17:57:43.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.445+0000 7f0751400640 1 -- 172.21.15.2:0/1289972561 learned_addr learned my addr 172.21.15.2:0/1289972561 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:43.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.446+0000 7f0751e00640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c15f0e0 0x7f074c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.446+0000 7f0751400640 1 -- 172.21.15.2:0/1289972561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c15f0e0 msgr2=0x7f074c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.446+0000 7f0751400640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c15f0e0 0x7f074c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.446+0000 7f0751400640 1 -- 172.21.15.2:0/1289972561 >> v1:172.21.15.137:6789/0 conn(0x7f074c15ec60 legacy=0x7f074c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.446+0000 7f0751400640 1 -- 172.21.15.2:0/1289972561 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c15f570 con 0x7f074c141880 2024-01-15T17:57:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.446+0000 7f0751e00640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c15f0e0 0x7f074c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.447+0000 7f0751400640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c141880 0x7f074c070fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f073c0098a0 tx=0x7f073c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12df8843d796d54 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:43.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.447+0000 7f074be00640 1 -- 172.21.15.2:0/1289972561 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c0317c0 con 0x7f074c141880 2024-01-15T17:57:43.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.447+0000 7f074be00640 1 -- 172.21.15.2:0/1289972561 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f073c031920 con 0x7f074c141880 2024-01-15T17:57:43.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.447+0000 7f074be00640 1 -- 172.21.15.2:0/1289972561 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073c031cb0 con 0x7f074c141880 2024-01-15T17:57:43.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 -- 172.21.15.2:0/1289972561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c141880 msgr2=0x7f074c070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c141880 0x7f074c070fc0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f073c0098a0 tx=0x7f073c02b860 comp rx=0 tx=0).stop 2024-01-15T17:57:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 -- 172.21.15.2:0/1289972561 shutdown_connections 2024-01-15T17:57:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c15f0e0 0x7f074c0cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 --2- 172.21.15.2:0/1289972561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c141880 0x7f074c070fc0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 -- 172.21.15.2:0/1289972561 >> 172.21.15.2:0/1289972561 conn(0x7f074c0b0840 msgr2=0x7f074c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:43.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 -- 172.21.15.2:0/1289972561 shutdown_connections 2024-01-15T17:57:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.448+0000 7f0752800640 1 -- 172.21.15.2:0/1289972561 wait complete. 2024-01-15T17:57:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.450+0000 7f0752800640 1 Processor -- start 2024-01-15T17:57:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.450+0000 7f0752800640 1 -- start start 2024-01-15T17:57:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0752800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c141880 0x7f074c0b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0752800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f074c15ec60 0x7f074c1433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0752800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 0x7f074c0b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0752800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b2180 con 0x7f074c15f0e0 2024-01-15T17:57:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0752800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b2420 con 0x7f074c15ec60 2024-01-15T17:57:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0752800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074c0b2700 con 0x7f074c141880 2024-01-15T17:57:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.451+0000 7f0751e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 0x7f074c0b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 0x7f074c0b34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46034/0 (socket says 172.21.15.2:46034) 2024-01-15T17:57:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 -- 172.21.15.2:0/2669488221 learned_addr learned my addr 172.21.15.2:0/2669488221 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0750a00640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f074c15ec60 0x7f074c1433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751400640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c141880 0x7f074c0b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 -- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c141880 msgr2=0x7f074c0b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c141880 0x7f074c0b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 -- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f074c15ec60 msgr2=0x7f074c1433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f074c15ec60 0x7f074c1433c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 -- 172.21.15.2:0/2669488221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c0094d0 con 0x7f074c15f0e0 2024-01-15T17:57:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.452+0000 7f0751e00640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 0x7f074c0b34c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f074400cef0 tx=0x7f074400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.453+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074400d840 con 0x7f074c15f0e0 2024-01-15T17:57:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.453+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f074400d9c0 con 0x7f074c15f0e0 2024-01-15T17:57:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.453+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f074400dd70 con 0x7f074c15f0e0 2024-01-15T17:57:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.453+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c0b29e0 con 0x7f074c15f0e0 2024-01-15T17:57:43.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.453+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c0b2fa0 con 0x7f074c15f0e0 2024-01-15T17:57:43.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.454+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c0b3190 con 0x7f074c15f0e0 2024-01-15T17:57:43.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.455+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f074401e3b0 con 0x7f074c15f0e0 2024-01-15T17:57:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.456+0000 7f0749600640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f072406b2c0 0x7f072406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.456+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f074408a7e0 con 0x7f074c15f0e0 2024-01-15T17:57:43.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.456+0000 7f0751400640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f072406b2c0 0x7f072406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.457+0000 7f0751400640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f072406b2c0 0x7f072406d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f073c036000 tx=0x7f073c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:43.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.460+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07440555e0 con 0x7f074c15f0e0 2024-01-15T17:57:43.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.655+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f074c074770 con 0x7f074c15f0e0 2024-01-15T17:57:43.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.658+0000 7f0749600640 1 -- 172.21.15.2:0/2669488221 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f074401a020 con 0x7f074c15f0e0 2024-01-15T17:57:43.663 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:43.663 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:43.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f072406b2c0 msgr2=0x7f072406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f072406b2c0 0x7f072406d780 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f073c036000 tx=0x7f073c03d040 comp rx=0 tx=0).stop 2024-01-15T17:57:43.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 msgr2=0x7f074c0b34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 0x7f074c0b34c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f074400cef0 tx=0x7f074400a710 comp rx=0 tx=0).stop 2024-01-15T17:57:43.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 shutdown_connections 2024-01-15T17:57:43.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f072406b2c0 0x7f072406d780 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f074c15f0e0 0x7f074c0b34c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f074c15ec60 0x7f074c1433c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 --2- 172.21.15.2:0/2669488221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f074c141880 0x7f074c0b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 >> 172.21.15.2:0/2669488221 conn(0x7f074c0b0840 msgr2=0x7f074c0afa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:43.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.661+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 shutdown_connections 2024-01-15T17:57:43.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.662+0000 7f0752800640 1 -- 172.21.15.2:0/2669488221 wait complete. 2024-01-15T17:57:43.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:57:43.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:57:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.908+0000 7f101d600640 1 Processor -- start 2024-01-15T17:57:43.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.908+0000 7f101d600640 1 -- start start 2024-01-15T17:57:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.909+0000 7f101d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1018143d20 0x7f101815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f101d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10180720d0 0x7f101815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f101d600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f10181447d0 0x7f10180cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f101d600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1018071430 con 0x7f1018143d20 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f101d600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1018144100 con 0x7f10181447d0 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f101d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1018071090 con 0x7f10180720d0 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1018143d20 0x7f101815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1018143d20 0x7f101815eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46048/0 (socket says 172.21.15.2:46048) 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 -- 172.21.15.2:0/512025971 learned_addr learned my addr 172.21.15.2:0/512025971 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017400640 1 --2- 172.21.15.2:0/512025971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10180720d0 0x7f101815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f101cc00640 1 --2- 172.21.15.2:0/512025971 >> v2:172.21.15.137:3300/0 conn(0x7f10181447d0 0x7f10180cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 -- 172.21.15.2:0/512025971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10180720d0 msgr2=0x7f101815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:43.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 --2- 172.21.15.2:0/512025971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10180720d0 0x7f101815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 -- 172.21.15.2:0/512025971 >> v2:172.21.15.137:3300/0 conn(0x7f10181447d0 msgr2=0x7f10180cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:43.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.910+0000 7f1017e00640 1 --2- 172.21.15.2:0/512025971 >> v2:172.21.15.137:3300/0 conn(0x7f10181447d0 0x7f10180cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.911+0000 7f1017e00640 1 -- 172.21.15.2:0/512025971 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1018144560 con 0x7f1018143d20 2024-01-15T17:57:43.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.912+0000 7f1017e00640 1 --2- 172.21.15.2:0/512025971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1018143d20 0x7f101815eb50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f101806e0a0 tx=0x7f100802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bbf268d55fdf8b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:43.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.912+0000 7f1016a00640 1 -- 172.21.15.2:0/512025971 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f100802c9e0 con 0x7f1018143d20 2024-01-15T17:57:43.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.912+0000 7f1016a00640 1 -- 172.21.15.2:0/512025971 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f100802cb40 con 0x7f1018143d20 2024-01-15T17:57:43.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.912+0000 7f1016a00640 1 -- 172.21.15.2:0/512025971 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080343f0 con 0x7f1018143d20 2024-01-15T17:57:43.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 -- 172.21.15.2:0/512025971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1018143d20 msgr2=0x7f101815eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 --2- 172.21.15.2:0/512025971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1018143d20 0x7f101815eb50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f101806e0a0 tx=0x7f100802be40 comp rx=0 tx=0).stop 2024-01-15T17:57:43.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 -- 172.21.15.2:0/512025971 shutdown_connections 2024-01-15T17:57:43.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 --2- 172.21.15.2:0/512025971 >> v2:172.21.15.137:3300/0 conn(0x7f10181447d0 0x7f10180cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 --2- 172.21.15.2:0/512025971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10180720d0 0x7f101815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 --2- 172.21.15.2:0/512025971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1018143d20 0x7f101815eb50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 -- 172.21.15.2:0/512025971 >> 172.21.15.2:0/512025971 conn(0x7f10180b0840 msgr2=0x7f10180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:43.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 -- 172.21.15.2:0/512025971 shutdown_connections 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.913+0000 7f101d600640 1 -- 172.21.15.2:0/512025971 wait complete. 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.915+0000 7f101d600640 1 Processor -- start 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.915+0000 7f101d600640 1 -- start start 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10180720d0 0x7f101813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1018143d20 0x7f1018158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 0x7f10181591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101d600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181f6ea0 con 0x7f10181447d0 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181598e0 con 0x7f10180720d0 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1018159bc0 con 0x7f1018143d20 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 0x7f10181591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 0x7f10181591f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46056/0 (socket says 172.21.15.2:46056) 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f101cc00640 1 -- 172.21.15.2:0/2697342126 learned_addr learned my addr 172.21.15.2:0/2697342126 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.916+0000 7f1017e00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10180720d0 0x7f101813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f1017400640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1018143d20 0x7f1018158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101cc00640 1 -- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1018143d20 msgr2=0x7f1018158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101cc00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1018143d20 0x7f1018158cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101cc00640 1 -- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10180720d0 msgr2=0x7f101813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:43.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101cc00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10180720d0 0x7f101813fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:43.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101cc00640 1 -- 172.21.15.2:0/2697342126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1008009b30 con 0x7f10181447d0 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f1017e00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10180720d0 0x7f101813fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101cc00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 0x7f10181591f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f101000d5d0 tx=0x7f101000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10100125f0 con 0x7f10181447d0 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1010012770 con 0x7f10181447d0 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010012b20 con 0x7f10181447d0 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1018159ea0 con 0x7f10181447d0 2024-01-15T17:57:43.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.917+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101815a460 con 0x7f10181447d0 2024-01-15T17:57:43.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.919+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101815a680 con 0x7f10181447d0 2024-01-15T17:57:43.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.919+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1010012c80 con 0x7f10181447d0 2024-01-15T17:57:43.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.921+0000 7f100fe00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ff006b2c0 0x7f0ff006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:43.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.921+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f101008a140 con 0x7f10181447d0 2024-01-15T17:57:43.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.921+0000 7f1017e00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ff006b2c0 0x7f0ff006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:43.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.922+0000 7f1017e00640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ff006b2c0 0x7f0ff006d780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1008035000 tx=0x7f1008030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:43.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:43.925+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1010054f20 con 0x7f10181447d0 2024-01-15T17:57:44.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.121+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1018074770 con 0x7f10181447d0 2024-01-15T17:57:44.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.121+0000 7f100fe00640 1 -- 172.21.15.2:0/2697342126 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1010058540 con 0x7f10181447d0 2024-01-15T17:57:44.125 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ff006b2c0 msgr2=0x7f0ff006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ff006b2c0 0x7f0ff006d780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1008035000 tx=0x7f1008030000 comp rx=0 tx=0).stop 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 msgr2=0x7f10181591f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 0x7f10181591f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f101000d5d0 tx=0x7f101000da90 comp rx=0 tx=0).stop 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 shutdown_connections 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ff006b2c0 0x7f0ff006d780 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10181447d0 0x7f10181591f0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1018143d20 0x7f1018158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 --2- 172.21.15.2:0/2697342126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10180720d0 0x7f101813fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 >> 172.21.15.2:0/2697342126 conn(0x7f10180b0840 msgr2=0x7f10180ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:44.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 shutdown_connections 2024-01-15T17:57:44.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.125+0000 7f101d600640 1 -- 172.21.15.2:0/2697342126 wait complete. 2024-01-15T17:57:44.148 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:57:44.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.363+0000 7f979f200640 1 Processor -- start 2024-01-15T17:57:44.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.364+0000 7f979f200640 1 -- start start 2024-01-15T17:57:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.365+0000 7f979f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9798070820 0x7f9798161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.365+0000 7f979f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9798072100 0x7f979806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.365+0000 7f979f200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97981620d0 con 0x7f979814b160 2024-01-15T17:57:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.365+0000 7f979f200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97980b35b0 con 0x7f9798070820 2024-01-15T17:57:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.365+0000 7f979f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97980b1510 con 0x7f9798072100 2024-01-15T17:57:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9798070820 0x7f9798161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9798070820 0x7f9798161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46058/0 (socket says 172.21.15.2:46058) 2024-01-15T17:57:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 -- 172.21.15.2:0/2321957015 learned_addr learned my addr 172.21.15.2:0/2321957015 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:44.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979e800640 1 --2- 172.21.15.2:0/2321957015 >> v2:172.21.15.137:3300/0 conn(0x7f9798072100 0x7f979806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 -- 172.21.15.2:0/2321957015 >> v2:172.21.15.137:3300/0 conn(0x7f9798072100 msgr2=0x7f979806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 --2- 172.21.15.2:0/2321957015 >> v2:172.21.15.137:3300/0 conn(0x7f9798072100 0x7f979806cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 -- 172.21.15.2:0/2321957015 >> v1:172.21.15.2:6789/0 conn(0x7f979814b160 legacy=0x7f9798070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979d400640 1 -- 172.21.15.2:0/2321957015 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97980b32d0 con 0x7f9798070820 2024-01-15T17:57:44.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.366+0000 7f979ca00640 1 -- 172.21.15.2:0/2321957015 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034438745 0 0) 0x7f97981620d0 con 0x7f979814b160 2024-01-15T17:57:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.367+0000 7f979d400640 1 --2- 172.21.15.2:0/2321957015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798070820 0x7f9798161b90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f979806fc80 tx=0x7f978002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf2191e28d5e77ed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979ca00640 1 -- 172.21.15.2:0/2321957015 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780030860 con 0x7f9798070820 2024-01-15T17:57:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979ca00640 1 -- 172.21.15.2:0/2321957015 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97800309c0 con 0x7f9798070820 2024-01-15T17:57:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979ca00640 1 -- 172.21.15.2:0/2321957015 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780030d50 con 0x7f9798070820 2024-01-15T17:57:44.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979f200640 1 -- 172.21.15.2:0/2321957015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798070820 msgr2=0x7f9798161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979f200640 1 --2- 172.21.15.2:0/2321957015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798070820 0x7f9798161b90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f979806fc80 tx=0x7f978002b9a0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979f200640 1 -- 172.21.15.2:0/2321957015 shutdown_connections 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979f200640 1 --2- 172.21.15.2:0/2321957015 >> v2:172.21.15.137:3300/0 conn(0x7f9798072100 0x7f979806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979f200640 1 --2- 172.21.15.2:0/2321957015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798070820 0x7f9798161b90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.368+0000 7f979f200640 1 -- 172.21.15.2:0/2321957015 >> 172.21.15.2:0/2321957015 conn(0x7f97980b0840 msgr2=0x7f97980b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.369+0000 7f979f200640 1 -- 172.21.15.2:0/2321957015 shutdown_connections 2024-01-15T17:57:44.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.369+0000 7f979f200640 1 -- 172.21.15.2:0/2321957015 wait complete. 2024-01-15T17:57:44.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.370+0000 7f979f200640 1 Processor -- start 2024-01-15T17:57:44.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.370+0000 7f979f200640 1 -- start start 2024-01-15T17:57:44.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.371+0000 7f979f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9798070820 0x7f97981faf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.371+0000 7f979f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 0x7f97980b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.371+0000 7f979f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f979814b160 0x7f97980b25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.371+0000 7f979f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97981612e0 con 0x7f9798072100 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.371+0000 7f979f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9798143a50 con 0x7f9798070820 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.371+0000 7f979f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9798143cf0 con 0x7f979814b160 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 0x7f97980b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 0x7f97980b2060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46072/0 (socket says 172.21.15.2:46072) 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 -- 172.21.15.2:0/2379439313 learned_addr learned my addr 172.21.15.2:0/2379439313 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979de00640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9798070820 0x7f97981faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979e800640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f979814b160 0x7f97980b25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 -- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f979814b160 msgr2=0x7f97980b25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f979814b160 0x7f97980b25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 -- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9798070820 msgr2=0x7f97981faf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9798070820 0x7f97981faf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 -- 172.21.15.2:0/2379439313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97800094d0 con 0x7f9798072100 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.372+0000 7f979d400640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 0x7f97980b2060 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9780000c00 tx=0x7f9780004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:44.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.373+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f978003e040 con 0x7f9798072100 2024-01-15T17:57:44.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.373+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9780039cb0 con 0x7f9798072100 2024-01-15T17:57:44.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.373+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9798143fd0 con 0x7f9798072100 2024-01-15T17:57:44.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.373+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9780004270 con 0x7f9798072100 2024-01-15T17:57:44.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.373+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979813e070 con 0x7f9798072100 2024-01-15T17:57:44.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.374+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979813e260 con 0x7f9798072100 2024-01-15T17:57:44.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.375+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f97800043d0 con 0x7f9798072100 2024-01-15T17:57:44.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.376+0000 7f9796000640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f977006b340 0x7f977006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.376+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97800ac5b0 con 0x7f9798072100 2024-01-15T17:57:44.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.376+0000 7f979de00640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f977006b340 0x7f977006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.377+0000 7f979de00640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f977006b340 0x7f977006d800 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9788007fd0 tx=0x7f9788009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:44.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.380+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9780077520 con 0x7f9798072100 2024-01-15T17:57:44.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.573+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9798074770 con 0x7f9798072100 2024-01-15T17:57:44.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.574+0000 7f9796000640 1 -- 172.21.15.2:0/2379439313 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f978007ab40 con 0x7f9798072100 2024-01-15T17:57:44.578 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.576+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f977006b340 msgr2=0x7f977006d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.576+0000 7f979f200640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f977006b340 0x7f977006d800 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9788007fd0 tx=0x7f9788009040 comp rx=0 tx=0).stop 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.576+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 msgr2=0x7f97980b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.576+0000 7f979f200640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 0x7f97980b2060 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9780000c00 tx=0x7f9780004730 comp rx=0 tx=0).stop 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 shutdown_connections 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f977006b340 0x7f977006d800 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f979814b160 0x7f97980b25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9798072100 0x7f97980b2060 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 --2- 172.21.15.2:0/2379439313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9798070820 0x7f97981faf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 >> 172.21.15.2:0/2379439313 conn(0x7f97980b0840 msgr2=0x7f9798149b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 shutdown_connections 2024-01-15T17:57:44.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.577+0000 7f979f200640 1 -- 172.21.15.2:0/2379439313 wait complete. 2024-01-15T17:57:44.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:57:44.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.806+0000 7f2827e00640 1 Processor -- start 2024-01-15T17:57:44.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.807+0000 7f2827e00640 1 -- start start 2024-01-15T17:57:44.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.808+0000 7f2827e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f282815d120 0x7f28280b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.808+0000 7f2827e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.808+0000 7f2827e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28280d0280 con 0x7f28280cddd0 2024-01-15T17:57:44.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.808+0000 7f2827e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28280cf100 con 0x7f282815d120 2024-01-15T17:57:44.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.808+0000 7f2827e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28280ceb40 con 0x7f28280cf920 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.808+0000 7f2826a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f282815d120 0x7f28280b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f2826a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f282815d120 0x7f28280b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46088/0 (socket says 172.21.15.2:46088) 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f2826a00640 1 -- 172.21.15.2:0/3771395498 learned_addr learned my addr 172.21.15.2:0/3771395498 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f281fe00640 1 --2- 172.21.15.2:0/3771395498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f2826a00640 1 -- 172.21.15.2:0/3771395498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 msgr2=0x7f28280cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f2826a00640 1 --2- 172.21.15.2:0/3771395498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280cfd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f2826a00640 1 -- 172.21.15.2:0/3771395498 >> v1:172.21.15.2:6789/0 conn(0x7f28280cddd0 legacy=0x7f28280ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.809+0000 7f2826a00640 1 -- 172.21.15.2:0/3771395498 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28280cee20 con 0x7f282815d120 2024-01-15T17:57:44.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.810+0000 7f2826a00640 1 --2- 172.21.15.2:0/3771395498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f28280b2800 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2818009870 tx=0x7f281802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34eb3002258da028 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:44.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2826000640 1 -- 172.21.15.2:0/3771395498 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818030840 con 0x7f282815d120 2024-01-15T17:57:44.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2826000640 1 -- 172.21.15.2:0/3771395498 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28180309a0 con 0x7f282815d120 2024-01-15T17:57:44.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2826000640 1 -- 172.21.15.2:0/3771395498 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818030d30 con 0x7f282815d120 2024-01-15T17:57:44.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 -- 172.21.15.2:0/3771395498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 msgr2=0x7f28280b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 --2- 172.21.15.2:0/3771395498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f28280b2800 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2818009870 tx=0x7f281802b980 comp rx=0 tx=0).stop 2024-01-15T17:57:44.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 -- 172.21.15.2:0/3771395498 shutdown_connections 2024-01-15T17:57:44.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 --2- 172.21.15.2:0/3771395498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280cfd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 --2- 172.21.15.2:0/3771395498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f28280b2800 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 -- 172.21.15.2:0/3771395498 >> 172.21.15.2:0/3771395498 conn(0x7f282809c3e0 msgr2=0x7f28280d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:44.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.811+0000 7f2827e00640 1 -- 172.21.15.2:0/3771395498 shutdown_connections 2024-01-15T17:57:44.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.812+0000 7f2827e00640 1 -- 172.21.15.2:0/3771395498 wait complete. 2024-01-15T17:57:44.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.813+0000 7f2827e00640 1 Processor -- start 2024-01-15T17:57:44.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.813+0000 7f2827e00640 1 -- start start 2024-01-15T17:57:44.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28280cddd0 0x7f282815e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f2828156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282815ffe0 con 0x7f282815d120 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282815fb50 con 0x7f28280cddd0 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282815ef40 con 0x7f28280cf920 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.814+0000 7f2827400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f2828156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f2828156240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46102/0 (socket says 172.21.15.2:46102) 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 -- 172.21.15.2:0/3940921481 learned_addr learned my addr 172.21.15.2:0/3940921481 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f281fe00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2826a00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28280cddd0 0x7f282815e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 -- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 msgr2=0x7f28280b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280b0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 -- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28280cddd0 msgr2=0x7f282815e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28280cddd0 0x7f282815e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 -- 172.21.15.2:0/3940921481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28180094d0 con 0x7f282815d120 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f281fe00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f2827400640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f2828156240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f282000c6b0 tx=0x7f282000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820003b70 con 0x7f282815d120 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820003cf0 con 0x7f282815d120 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.815+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820003ea0 con 0x7f282815d120 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.816+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282815f220 con 0x7f282815d120 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.816+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282815f7e0 con 0x7f282815d120 2024-01-15T17:57:44.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.817+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2828074770 con 0x7f282815d120 2024-01-15T17:57:44.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.818+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2820005310 con 0x7f282815d120 2024-01-15T17:57:44.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.819+0000 7f281ea00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f280006b2c0 0x7f280006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:44.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.819+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2820089aa0 con 0x7f282815d120 2024-01-15T17:57:44.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.819+0000 7f2826a00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f280006b2c0 0x7f280006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:44.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.820+0000 7f2826a00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f280006b2c0 0x7f280006d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2818036000 tx=0x7f281803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:44.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:44.825+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28200547e0 con 0x7f282815d120 2024-01-15T17:57:45.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.017+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f282815fc90 con 0x7f282815d120 2024-01-15T17:57:45.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.018+0000 7f281ea00640 1 -- 172.21.15.2:0/3940921481 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2820057e00 con 0x7f282815d120 2024-01-15T17:57:45.021 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:57:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f280006b2c0 msgr2=0x7f280006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f280006b2c0 0x7f280006d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2818036000 tx=0x7f281803b040 comp rx=0 tx=0).stop 2024-01-15T17:57:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 msgr2=0x7f2828156240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f2828156240 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f282000c6b0 tx=0x7f282000cb70 comp rx=0 tx=0).stop 2024-01-15T17:57:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 shutdown_connections 2024-01-15T17:57:45.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f280006b2c0 0x7f280006d780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f282815d120 0x7f2828156240 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f28280cf920 0x7f28280b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 --2- 172.21.15.2:0/3940921481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f28280cddd0 0x7f282815e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 >> 172.21.15.2:0/3940921481 conn(0x7f282809c3e0 msgr2=0x7f28280b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:45.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 shutdown_connections 2024-01-15T17:57:45.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.020+0000 7f2827e00640 1 -- 172.21.15.2:0/3940921481 wait complete. 2024-01-15T17:57:45.042 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:57:45.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.249+0000 7fbe2d000640 1 Processor -- start 2024-01-15T17:57:45.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.249+0000 7fbe2d000640 1 -- start start 2024-01-15T17:57:45.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe2d000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbe28143810 0x7fbe2806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe2d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe280cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe2d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2806e470 con 0x7fbe2806e010 2024-01-15T17:57:45.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe2d000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe280b3520 con 0x7fbe28143810 2024-01-15T17:57:45.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe2d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280ccbe0 con 0x7fbe280b2250 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe27400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbe28143810 0x7fbe2806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.251+0000 7fbe27e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe280cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.252+0000 7fbe27e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe280cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:38092/0 (socket says 172.21.15.2:38092) 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.252+0000 7fbe27e00640 1 -- 172.21.15.2:0/3931840896 learned_addr learned my addr 172.21.15.2:0/3931840896 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.252+0000 7fbe27e00640 1 -- 172.21.15.2:0/3931840896 >> v2:172.21.15.137:3300/0 conn(0x7fbe28143810 msgr2=0x7fbe2806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.252+0000 7fbe27e00640 1 --2- 172.21.15.2:0/3931840896 >> v2:172.21.15.137:3300/0 conn(0x7fbe28143810 0x7fbe2806dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.252+0000 7fbe27e00640 1 -- 172.21.15.2:0/3931840896 >> v1:172.21.15.137:6789/0 conn(0x7fbe2806e010 legacy=0x7fbe280b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.252+0000 7fbe27e00640 1 -- 172.21.15.2:0/3931840896 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe28143bf0 con 0x7fbe280b2250 2024-01-15T17:57:45.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.254+0000 7fbe27e00640 1 --2- 172.21.15.2:0/3931840896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe280cd070 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009e50 tx=0x7fbe2002bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b8c504a43a7c022 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:45.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.254+0000 7fbe26a00640 1 -- 172.21.15.2:0/3931840896 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2002ca80 con 0x7fbe280b2250 2024-01-15T17:57:45.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.254+0000 7fbe26a00640 1 -- 172.21.15.2:0/3931840896 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2002cbe0 con 0x7fbe280b2250 2024-01-15T17:57:45.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.254+0000 7fbe26a00640 1 -- 172.21.15.2:0/3931840896 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe200343f0 con 0x7fbe280b2250 2024-01-15T17:57:45.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 -- 172.21.15.2:0/3931840896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 msgr2=0x7fbe280cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 --2- 172.21.15.2:0/3931840896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe280cd070 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009e50 tx=0x7fbe2002bee0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 -- 172.21.15.2:0/3931840896 shutdown_connections 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 --2- 172.21.15.2:0/3931840896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe280cd070 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 --2- 172.21.15.2:0/3931840896 >> v2:172.21.15.137:3300/0 conn(0x7fbe28143810 0x7fbe2806dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 -- 172.21.15.2:0/3931840896 >> 172.21.15.2:0/3931840896 conn(0x7fbe280b0840 msgr2=0x7fbe280b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 -- 172.21.15.2:0/3931840896 shutdown_connections 2024-01-15T17:57:45.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.255+0000 7fbe2d000640 1 -- 172.21.15.2:0/3931840896 wait complete. 2024-01-15T17:57:45.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.257+0000 7fbe2d000640 1 Processor -- start 2024-01-15T17:57:45.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.258+0000 7fbe2d000640 1 -- start start 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe2d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe2806e010 0x7fbe281e06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe2d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe28133870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe2d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 0x7fbe28133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe2d000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe281df600 con 0x7fbe28143810 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe2d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe281df8e0 con 0x7fbe2806e010 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe2d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe281dfbc0 con 0x7fbe280b2250 2024-01-15T17:57:45.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe27e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 0x7fbe28133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe27e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 0x7fbe28133db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46114/0 (socket says 172.21.15.2:46114) 2024-01-15T17:57:45.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe27e00640 1 -- 172.21.15.2:0/2044410837 learned_addr learned my addr 172.21.15.2:0/2044410837 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:45.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.259+0000 7fbe27400640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe2806e010 0x7fbe281e06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe1fe00640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe28133870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe27e00640 1 -- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 msgr2=0x7fbe28133870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe27e00640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe28133870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe27e00640 1 -- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe2806e010 msgr2=0x7fbe281e06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe27e00640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe2806e010 0x7fbe281e06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe27e00640 1 -- 172.21.15.2:0/2044410837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe20009b30 con 0x7fbe28143810 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe1fe00640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe28133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.260+0000 7fbe27e00640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 0x7fbe28133db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009e50 tx=0x7fbe20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.261+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2003e450 con 0x7fbe28143810 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.261+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe2003e5b0 con 0x7fbe28143810 2024-01-15T17:57:45.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.261+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe281dfea0 con 0x7fbe28143810 2024-01-15T17:57:45.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.261+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe2003e960 con 0x7fbe28143810 2024-01-15T17:57:45.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.261+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe281e0be0 con 0x7fbe28143810 2024-01-15T17:57:45.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.262+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe281e03c0 con 0x7fbe28143810 2024-01-15T17:57:45.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.263+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbe2002cd50 con 0x7fbe28143810 2024-01-15T17:57:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.264+0000 7fbe1f400640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbe0006b2c0 0x7fbe0006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.265+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe200ac660 con 0x7fbe28143810 2024-01-15T17:57:45.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.265+0000 7fbe27400640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbe0006b2c0 0x7fbe0006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.266+0000 7fbe27400640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbe0006b2c0 0x7fbe0006d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009b40 tx=0x7fbe18009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:45.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.269+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe281e03c0 con 0x7fbe28143810 2024-01-15T17:57:45.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.507+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe281e03c0 con 0x7fbe28143810 2024-01-15T17:57:45.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.508+0000 7fbe1f400640 1 -- 172.21.15.2:0/2044410837 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fbe281e03c0 con 0x7fbe28143810 2024-01-15T17:57:45.512 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:45.512 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:57:45.513 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.511+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbe0006b2c0 msgr2=0x7fbe0006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.511+0000 7fbe2d000640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbe0006b2c0 0x7fbe0006d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009b40 tx=0x7fbe18009190 comp rx=0 tx=0).stop 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.511+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 msgr2=0x7fbe28133db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.511+0000 7fbe2d000640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 0x7fbe28133db0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbe20009e50 tx=0x7fbe20004a40 comp rx=0 tx=0).stop 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 shutdown_connections 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbe0006b2c0 0x7fbe0006d780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbe28143810 0x7fbe28133db0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbe280b2250 0x7fbe28133870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 --2- 172.21.15.2:0/2044410837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbe2806e010 0x7fbe281e06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 >> 172.21.15.2:0/2044410837 conn(0x7fbe280b0840 msgr2=0x7fbe280cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:45.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 shutdown_connections 2024-01-15T17:57:45.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.512+0000 7fbe2d000640 1 -- 172.21.15.2:0/2044410837 wait complete. 2024-01-15T17:57:45.535 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:45.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.742+0000 7f7c83e00640 1 Processor -- start 2024-01-15T17:57:45.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.742+0000 7f7c83e00640 1 -- start start 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c83e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7c7c15b7d0 0x7f7c7c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 0x7f7c7c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c83e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c15cb00 con 0x7f7c7c15c0e0 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c83e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7c0cccd0 con 0x7f7c7c15b7d0 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c83e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0ccec0 con 0x7f7c7c15c4c0 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c81400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7c7c15c0e0 0x7f7c7c0cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45900/0 (socket says 172.21.15.2:45900) 2024-01-15T17:57:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.744+0000 7f7c81400640 1 -- 172.21.15.2:0/2840869330 learned_addr learned my addr 172.21.15.2:0/2840869330 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:45.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c81e00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 0x7f7c7c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c80a00640 1 --2- 172.21.15.2:0/2840869330 >> v2:172.21.15.137:3300/0 conn(0x7f7c7c15b7d0 0x7f7c7c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468816670 0 0) 0x7f7c7c15cb00 con 0x7f7c7c15c0e0 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c003520 con 0x7f7c7c15c0e0 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2102109042 0 0) 0x7f7c5c003520 con 0x7f7c7c15c0e0 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 msgr2=0x7f7c7c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 0x7f7c7c06c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 >> v2:172.21.15.137:3300/0 conn(0x7f7c7c15b7d0 msgr2=0x7f7c7c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 --2- 172.21.15.2:0/2840869330 >> v2:172.21.15.137:3300/0 conn(0x7f7c7c15b7d0 0x7f7c7c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0cd1a0 con 0x7f7c7c15c0e0 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7c6c001670 con 0x7f7c7c15c0e0 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 >> v1:172.21.15.2:6789/0 conn(0x7f7c7c15c0e0 legacy=0x7f7c7c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 0x7f7c5c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c5c003f40 0x7f7c5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0ccec0 con 0x7f7c5c004d00 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c00b980 con 0x7f7c5c003f40 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.745+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5c00bac0 con 0x7f7c7c15c4c0 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81e00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81400640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 0x7f7c5c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c80a00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c5c003f40 0x7f7c5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81e00640 1 -- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 msgr2=0x7f7c5c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81e00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c7c15c4c0 0x7f7c5c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81e00640 1 -- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c5c003f40 msgr2=0x7f7c5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81e00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c5c003f40 0x7f7c5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c81e00640 1 -- 172.21.15.2:0/2840869330 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5c003520 con 0x7f7c5c004d00 2024-01-15T17:57:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.746+0000 7f7c80a00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c5c003f40 0x7f7c5c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.747+0000 7f7c81e00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400e170 tx=0x7f7c5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2b876776e0cf91 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.747+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7402cbf0 con 0x7f7c5c004d00 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.747+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7400f040 con 0x7f7c5c004d00 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c7be00640 1 -- 172.21.15.2:0/2840869330 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74034440 con 0x7f7c5c004d00 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c83e00640 1 -- 172.21.15.2:0/2840869330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c5c004d00 msgr2=0x7f7c5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c83e00640 1 --2- 172.21.15.2:0/2840869330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c5c004d00 0x7f7c5c005180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f7c7400e170 tx=0x7f7c5c0057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c81e00640 1 -- 172.21.15.2:0/2840869330 reap_dead start 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c83e00640 1 -- 172.21.15.2:0/2840869330 shutdown_connections 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c83e00640 1 -- 172.21.15.2:0/2840869330 >> 172.21.15.2:0/2840869330 conn(0x7f7c7c06da20 msgr2=0x7f7c7c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c83e00640 1 -- 172.21.15.2:0/2840869330 shutdown_connections 2024-01-15T17:57:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.748+0000 7f7c83e00640 1 -- 172.21.15.2:0/2840869330 wait complete. 2024-01-15T17:57:45.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.750+0000 7f7c83e00640 1 Processor -- start 2024-01-15T17:57:45.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.750+0000 7f7c83e00640 1 -- start start 2024-01-15T17:57:45.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c5c004d00 0x7f7c7c1f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 0x7f7c7c1f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c83e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c7c146980 0x7f7c7c0b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c83e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f4eb0 con 0x7f7c7c155e00 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c83e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c156730 con 0x7f7c7c146980 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c83e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1f0970 con 0x7f7c5c004d00 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.751+0000 7f7c80a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 0x7f7c7c1f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 0x7f7c7c1f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46140/0 (socket says 172.21.15.2:46140) 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 -- 172.21.15.2:0/256411528 learned_addr learned my addr 172.21.15.2:0/256411528 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c81e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c7c146980 0x7f7c7c0b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c81400640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c5c004d00 0x7f7c7c1f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 -- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c5c004d00 msgr2=0x7f7c7c1f4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c5c004d00 0x7f7c7c1f4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 -- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c7c146980 msgr2=0x7f7c7c0b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c7c146980 0x7f7c7c0b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 -- 172.21.15.2:0/256411528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c1f0f30 con 0x7f7c7c155e00 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c81e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c7c146980 0x7f7c7c0b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.752+0000 7f7c80a00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 0x7f7c7c1f13c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c640109d0 tx=0x7f7c64010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.753+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400ec20 con 0x7f7c7c155e00 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.753+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6400ed80 con 0x7f7c7c155e00 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.753+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6401c620 con 0x7f7c7c155e00 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.753+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c156f40 con 0x7f7c7c155e00 2024-01-15T17:57:45.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.753+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c156980 con 0x7f7c7c155e00 2024-01-15T17:57:45.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.754+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c1f10c0 con 0x7f7c7c155e00 2024-01-15T17:57:45.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.755+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7c6401c780 con 0x7f7c7c155e00 2024-01-15T17:57:45.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.756+0000 7f7c79600640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c5406b2f0 0x7f7c5406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:45.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.756+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c6408a310 con 0x7f7c7c155e00 2024-01-15T17:57:45.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.757+0000 7f7c81400640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c5406b2f0 0x7f7c5406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:45.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.758+0000 7f7c81400640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c5406b2f0 0x7f7c5406d7b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c005af0 tx=0x7f7c6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:45.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.760+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c64055190 con 0x7f7c7c155e00 2024-01-15T17:57:45.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.954+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c7c068720 con 0x7f7c7c155e00 2024-01-15T17:57:45.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.956+0000 7f7c79600640 1 -- 172.21.15.2:0/256411528 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f7c640587b0 con 0x7f7c7c155e00 2024-01-15T17:57:45.961 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:45.961 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:45.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c5406b2f0 msgr2=0x7f7c5406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c5406b2f0 0x7f7c5406d7b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c005af0 tx=0x7f7c6c009040 comp rx=0 tx=0).stop 2024-01-15T17:57:45.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 msgr2=0x7f7c7c1f13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:45.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 0x7f7c7c1f13c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7c640109d0 tx=0x7f7c64010e90 comp rx=0 tx=0).stop 2024-01-15T17:57:45.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 shutdown_connections 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c5406b2f0 0x7f7c5406d7b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c7c146980 0x7f7c7c0b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7c155e00 0x7f7c7c1f13c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 --2- 172.21.15.2:0/256411528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c5c004d00 0x7f7c7c1f4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 >> 172.21.15.2:0/256411528 conn(0x7f7c7c06da20 msgr2=0x7f7c7c0aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 shutdown_connections 2024-01-15T17:57:45.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:45.959+0000 7f7c83e00640 1 -- 172.21.15.2:0/256411528 wait complete. 2024-01-15T17:57:45.982 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:57:45.983 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:57:46.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.203+0000 7fe26ac00640 1 Processor -- start 2024-01-15T17:57:46.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.203+0000 7fe26ac00640 1 -- start start 2024-01-15T17:57:46.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.204+0000 7fe26ac00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe26406d000 0x7fe2640acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.204+0000 7fe26ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe2640b2430 0x7fe2640b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.204+0000 7fe26ac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26415dbc0 con 0x7fe26415ae80 2024-01-15T17:57:46.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.204+0000 7fe26ac00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26415e840 con 0x7fe26406d000 2024-01-15T17:57:46.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.204+0000 7fe26ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26415dea0 con 0x7fe2640b2430 2024-01-15T17:57:46.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe26406d000 0x7fe2640acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe26406d000 0x7fe2640acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46156/0 (socket says 172.21.15.2:46156) 2024-01-15T17:57:46.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 -- 172.21.15.2:0/3054643398 learned_addr learned my addr 172.21.15.2:0/3054643398 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:46.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe268c00640 1 --2- 172.21.15.2:0/3054643398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe2640b2430 0x7fe2640b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 -- 172.21.15.2:0/3054643398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe2640b2430 msgr2=0x7fe2640b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:46.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 --2- 172.21.15.2:0/3054643398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe2640b2430 0x7fe2640b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 -- 172.21.15.2:0/3054643398 >> v1:172.21.15.137:6789/0 conn(0x7fe26415ae80 legacy=0x7fe26406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:46.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.205+0000 7fe263400640 1 -- 172.21.15.2:0/3054643398 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26415e180 con 0x7fe26406d000 2024-01-15T17:57:46.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.206+0000 7fe263400640 1 --2- 172.21.15.2:0/3054643398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26406d000 0x7fe2640acd00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009870 tx=0x7fe24c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244a5403067a7a64 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:46.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.207+0000 7fe262a00640 1 -- 172.21.15.2:0/3054643398 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c030840 con 0x7fe26406d000 2024-01-15T17:57:46.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.207+0000 7fe262a00640 1 -- 172.21.15.2:0/3054643398 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c0309a0 con 0x7fe26406d000 2024-01-15T17:57:46.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.207+0000 7fe262a00640 1 -- 172.21.15.2:0/3054643398 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c030d30 con 0x7fe26406d000 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.207+0000 7fe26ac00640 1 -- 172.21.15.2:0/3054643398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26406d000 msgr2=0x7fe2640acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.207+0000 7fe26ac00640 1 --2- 172.21.15.2:0/3054643398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26406d000 0x7fe2640acd00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009870 tx=0x7fe24c02b980 comp rx=0 tx=0).stop 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.208+0000 7fe26ac00640 1 -- 172.21.15.2:0/3054643398 shutdown_connections 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.208+0000 7fe26ac00640 1 --2- 172.21.15.2:0/3054643398 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe2640b2430 0x7fe2640b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.208+0000 7fe26ac00640 1 --2- 172.21.15.2:0/3054643398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe26406d000 0x7fe2640acd00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.208+0000 7fe26ac00640 1 -- 172.21.15.2:0/3054643398 >> 172.21.15.2:0/3054643398 conn(0x7fe2640b45f0 msgr2=0x7fe2640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.208+0000 7fe26ac00640 1 -- 172.21.15.2:0/3054643398 shutdown_connections 2024-01-15T17:57:46.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.208+0000 7fe26ac00640 1 -- 172.21.15.2:0/3054643398 wait complete. 2024-01-15T17:57:46.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.209+0000 7fe26ac00640 1 Processor -- start 2024-01-15T17:57:46.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.210+0000 7fe26ac00640 1 -- start start 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe26ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26406d000 0x7fe264159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe26ac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 0x7fe2640aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe26ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe26415ae80 0x7fe26414a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe26ac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640afcc0 con 0x7fe2640b2430 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe26ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe264154d10 con 0x7fe26415ae80 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe26ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2640af0b0 con 0x7fe26406d000 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 0x7fe2640aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 0x7fe2640aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46166/0 (socket says 172.21.15.2:46166) 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 -- 172.21.15.2:0/1915294586 learned_addr learned my addr 172.21.15.2:0/1915294586 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263e00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26406d000 0x7fe264159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe268c00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe26415ae80 0x7fe26414a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 -- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26406d000 msgr2=0x7fe264159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26406d000 0x7fe264159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 -- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe26415ae80 msgr2=0x7fe26414a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe26415ae80 0x7fe26414a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe263400640 1 -- 172.21.15.2:0/1915294586 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c0094d0 con 0x7fe2640b2430 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.211+0000 7fe268c00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe26415ae80 0x7fe26414a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.212+0000 7fe263400640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 0x7fe2640aea60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.212+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c039c90 con 0x7fe2640b2430 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.212+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c039df0 con 0x7fe2640b2430 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.212+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c043640 con 0x7fe2640b2430 2024-01-15T17:57:46.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.212+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2640af390 con 0x7fe2640b2430 2024-01-15T17:57:46.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.212+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2640af950 con 0x7fe2640b2430 2024-01-15T17:57:46.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.213+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2640686f0 con 0x7fe2640b2430 2024-01-15T17:57:46.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.214+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe24c0437a0 con 0x7fe2640b2430 2024-01-15T17:57:46.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.216+0000 7fe25be00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23c06b2c0 0x7fe23c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.216+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe24c03e070 con 0x7fe2640b2430 2024-01-15T17:57:46.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.216+0000 7fe263e00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23c06b2c0 0x7fe23c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.217+0000 7fe263e00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23c06b2c0 0x7fe23c06d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe254000f60 tx=0x7fe254009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:46.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.220+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe24c077b50 con 0x7fe2640b2430 2024-01-15T17:57:46.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.413+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe2640b4be0 con 0x7fe2640b2430 2024-01-15T17:57:46.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.414+0000 7fe25be00640 1 -- 172.21.15.2:0/1915294586 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe24c034020 con 0x7fe2640b2430 2024-01-15T17:57:46.418 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23c06b2c0 msgr2=0x7fe23c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23c06b2c0 0x7fe23c06d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe254000f60 tx=0x7fe254009040 comp rx=0 tx=0).stop 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 msgr2=0x7fe2640aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 0x7fe2640aea60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c004730 comp rx=0 tx=0).stop 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 shutdown_connections 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe23c06b2c0 0x7fe23c06d780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe26415ae80 0x7fe26414a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2640b2430 0x7fe2640aea60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 --2- 172.21.15.2:0/1915294586 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe26406d000 0x7fe264159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 >> 172.21.15.2:0/1915294586 conn(0x7fe2640b45f0 msgr2=0x7fe264070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 shutdown_connections 2024-01-15T17:57:46.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.417+0000 7fe26ac00640 1 -- 172.21.15.2:0/1915294586 wait complete. 2024-01-15T17:57:46.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:57:46.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.660+0000 7fe4d6e00640 1 Processor -- start 2024-01-15T17:57:46.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.661+0000 7fe4d6e00640 1 -- start start 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d0144c00 0x7fe4d00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d6e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe4d00cdd70 0x7fe4d006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d6e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4d00ce230 con 0x7fe4d015c450 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d6e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4d00ce510 con 0x7fe4d00cdd70 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00ce7f0 con 0x7fe4d0144c00 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4cf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe4d00cdd70 0x7fe4d006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d4e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe4d015c450 0x7fe4d00b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45904/0 (socket says 172.21.15.2:45904) 2024-01-15T17:57:46.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.662+0000 7fe4d4e00640 1 -- 172.21.15.2:0/240106999 learned_addr learned my addr 172.21.15.2:0/240106999 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:46.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.663+0000 7fe4cfe00640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d0144c00 0x7fe4d00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.663+0000 7fe4cf400640 1 -- 172.21.15.2:0/240106999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d0144c00 msgr2=0x7fe4d00cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:46.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.663+0000 7fe4cf400640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d0144c00 0x7fe4d00cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.663+0000 7fe4cf400640 1 -- 172.21.15.2:0/240106999 >> v1:172.21.15.2:6789/0 conn(0x7fe4d015c450 legacy=0x7fe4d00b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.663+0000 7fe4cf400640 1 -- 172.21.15.2:0/240106999 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4d00705f0 con 0x7fe4d00cdd70 2024-01-15T17:57:46.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.663+0000 7fe4cfe00640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d0144c00 0x7fe4d00cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:46.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.664+0000 7fe4cf400640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d006c8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80097a0 tx=0x7fe4b802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167c6e5dac7259e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.664+0000 7fe4cea00640 1 -- 172.21.15.2:0/240106999 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b802d040 con 0x7fe4d00cdd70 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.664+0000 7fe4cea00640 1 -- 172.21.15.2:0/240106999 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b8031810 con 0x7fe4d00cdd70 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.664+0000 7fe4cea00640 1 -- 172.21.15.2:0/240106999 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8031ba0 con 0x7fe4d00cdd70 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 -- 172.21.15.2:0/240106999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 msgr2=0x7fe4d006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d006c8f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe4b80097a0 tx=0x7fe4b802b8b0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 -- 172.21.15.2:0/240106999 shutdown_connections 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d006c8f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/240106999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d0144c00 0x7fe4d00cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 -- 172.21.15.2:0/240106999 >> 172.21.15.2:0/240106999 conn(0x7fe4d00b45f0 msgr2=0x7fe4d00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 -- 172.21.15.2:0/240106999 shutdown_connections 2024-01-15T17:57:46.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.665+0000 7fe4d6e00640 1 -- 172.21.15.2:0/240106999 wait complete. 2024-01-15T17:57:46.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.667+0000 7fe4d6e00640 1 Processor -- start 2024-01-15T17:57:46.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.667+0000 7fe4d6e00640 1 -- start start 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4d6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d0157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4d6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4d0144c00 0x7fe4d0141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4d6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d015c450 0x7fe4d0141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4d6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00b1a80 con 0x7fe4d00cdd70 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4d6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00aed00 con 0x7fe4d0144c00 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4d6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d00b11e0 con 0x7fe4d015c450 2024-01-15T17:57:46.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d0157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d0157e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46184/0 (socket says 172.21.15.2:46184) 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.668+0000 7fe4cfe00640 1 -- 172.21.15.2:0/3880600851 learned_addr learned my addr 172.21.15.2:0/3880600851 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cf400640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4d0144c00 0x7fe4d0141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cfe00640 1 -- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d015c450 msgr2=0x7fe4d0141650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4d4e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d015c450 0x7fe4d0141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cfe00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d015c450 0x7fe4d0141650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cfe00640 1 -- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4d0144c00 msgr2=0x7fe4d0141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cfe00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4d0144c00 0x7fe4d0141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cfe00640 1 -- 172.21.15.2:0/3880600851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b8009400 con 0x7fe4d00cdd70 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cf400640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4d0144c00 0x7fe4d0141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4cfe00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d0157e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000cf60 tx=0x7fe4c000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000d840 con 0x7fe4d00cdd70 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4c000d9a0 con 0x7fe4d00cdd70 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d00b14c0 con 0x7fe4d00cdd70 2024-01-15T17:57:46.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4c000dd30 con 0x7fe4d00cdd70 2024-01-15T17:57:46.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.669+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d0158370 con 0x7fe4d00cdd70 2024-01-15T17:57:46.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.671+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d00aeb60 con 0x7fe4d00cdd70 2024-01-15T17:57:46.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.672+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe4c0015440 con 0x7fe4d00cdd70 2024-01-15T17:57:46.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.672+0000 7fe4c7e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe4a806b2c0 0x7fe4a806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:46.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.673+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4c008a5c0 con 0x7fe4d00cdd70 2024-01-15T17:57:46.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.673+0000 7fe4cf400640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe4a806b2c0 0x7fe4a806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:46.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.674+0000 7fe4cf400640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe4a806b2c0 0x7fe4a806d780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8036000 tx=0x7fe4b803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:46.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.677+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4d00aeb60 con 0x7fe4d00cdd70 2024-01-15T17:57:46.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.869+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4d00aeb60 con 0x7fe4d00cdd70 2024-01-15T17:57:46.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.870+0000 7fe4c7e00640 1 -- 172.21.15.2:0/3880600851 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe4d00aeb60 con 0x7fe4d00cdd70 2024-01-15T17:57:46.874 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.872+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe4a806b2c0 msgr2=0x7fe4a806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.872+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe4a806b2c0 0x7fe4a806d780 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe4b8036000 tx=0x7fe4b803c040 comp rx=0 tx=0).stop 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.872+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 msgr2=0x7fe4d0157e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.872+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d0157e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe4c000cf60 tx=0x7fe4c000cf90 comp rx=0 tx=0).stop 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 shutdown_connections 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe4a806b2c0 0x7fe4a806d780 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe4d015c450 0x7fe4d0141650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe4d0144c00 0x7fe4d0141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 --2- 172.21.15.2:0/3880600851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe4d00cdd70 0x7fe4d0157e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 >> 172.21.15.2:0/3880600851 conn(0x7fe4d00b45f0 msgr2=0x7fe4d00ad2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 shutdown_connections 2024-01-15T17:57:46.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:46.873+0000 7fe4d6e00640 1 -- 172.21.15.2:0/3880600851 wait complete. 2024-01-15T17:57:46.896 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:57:47.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.108+0000 7f7e5a000640 1 Processor -- start 2024-01-15T17:57:47.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.109+0000 7f7e5a000640 1 -- start start 2024-01-15T17:57:47.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.110+0000 7f7e5a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e540d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.110+0000 7f7e5a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7e540d1c50 0x7f7e540b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.110+0000 7f7e5a000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e540d1250 con 0x7f7e540d1870 2024-01-15T17:57:47.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.110+0000 7f7e5a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e540ce6d0 con 0x7f7e540d1c50 2024-01-15T17:57:47.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.110+0000 7f7e5a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540ce8e0 con 0x7f7e540b2210 2024-01-15T17:57:47.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.111+0000 7f7e58c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7e540d1870 0x7f7e5415d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45914/0 (socket says 172.21.15.2:45914) 2024-01-15T17:57:47.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.111+0000 7f7e58c00640 1 -- 172.21.15.2:0/3748487611 learned_addr learned my addr 172.21.15.2:0/3748487611 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.111+0000 7f7e53e00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e540d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.111+0000 7f7e59600640 1 --2- 172.21.15.2:0/3748487611 >> v2:172.21.15.137:3300/0 conn(0x7f7e540d1c50 0x7f7e540b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.111+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412513121 0 0) 0x7f7e540d1250 con 0x7f7e540d1870 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.111+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e34003520 con 0x7f7e540d1870 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570294006 0 0) 0x7f7e34003520 con 0x7f7e540d1870 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 msgr2=0x7f7e540d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e540d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 >> v2:172.21.15.137:3300/0 conn(0x7f7e540d1c50 msgr2=0x7f7e540b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 --2- 172.21.15.2:0/3748487611 >> v2:172.21.15.137:3300/0 conn(0x7f7e540d1c50 0x7f7e540b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540cebc0 con 0x7f7e540d1870 2024-01-15T17:57:47.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7e44001670 con 0x7f7e540d1870 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 >> v1:172.21.15.2:6789/0 conn(0x7f7e540d1870 legacy=0x7f7e5415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e34003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e34003f40 0x7f7e340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540ce6d0 con 0x7f7e34003f40 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540ce8e0 con 0x7f7e34004d00 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3400b980 con 0x7f7e540b2210 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53e00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e34003f40 0x7f7e340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e59600640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e53e00640 1 -- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 msgr2=0x7f7e34003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.112+0000 7f7e58c00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e34003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.113+0000 7f7e53e00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e34003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.113+0000 7f7e53e00640 1 -- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e34004d00 msgr2=0x7f7e34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.113+0000 7f7e53e00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e34004d00 0x7f7e34005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.113+0000 7f7e53e00640 1 -- 172.21.15.2:0/3748487611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34003520 con 0x7f7e34003f40 2024-01-15T17:57:47.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.113+0000 7f7e58c00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e540b2210 0x7f7e34003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.114+0000 7f7e53e00640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e34003f40 0x7f7e340047c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009ac0 tx=0x7f7e340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b459fdee305f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.114+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c02c680 con 0x7f7e34003f40 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.114+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c02c7e0 con 0x7f7e34003f40 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.114+0000 7f7e53400640 1 -- 172.21.15.2:0/3748487611 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c02cb90 con 0x7f7e34003f40 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e5a000640 1 -- 172.21.15.2:0/3748487611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e34003f40 msgr2=0x7f7e340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e5a000640 1 --2- 172.21.15.2:0/3748487611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e34003f40 0x7f7e340047c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c009ac0 tx=0x7f7e340057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e59600640 1 -- 172.21.15.2:0/3748487611 reap_dead start 2024-01-15T17:57:47.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e5a000640 1 -- 172.21.15.2:0/3748487611 shutdown_connections 2024-01-15T17:57:47.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e5a000640 1 -- 172.21.15.2:0/3748487611 >> 172.21.15.2:0/3748487611 conn(0x7f7e540d30b0 msgr2=0x7f7e540d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:47.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e5a000640 1 -- 172.21.15.2:0/3748487611 shutdown_connections 2024-01-15T17:57:47.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.115+0000 7f7e5a000640 1 -- 172.21.15.2:0/3748487611 wait complete. 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.117+0000 7f7e5a000640 1 Processor -- start 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.117+0000 7f7e5a000640 1 -- start start 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e5a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e34003f40 0x7f7e54152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e5a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5415f290 0x7f7e5415f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e5a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 0x7f7e540b7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e5a000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540cbbc0 con 0x7f7e540b6c80 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e5a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5415fc50 con 0x7f7e5415f290 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e5a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e54152b80 con 0x7f7e34003f40 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 0x7f7e540b7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 0x7f7e540b7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46210/0 (socket says 172.21.15.2:46210) 2024-01-15T17:57:47.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e59600640 1 -- 172.21.15.2:0/727098710 learned_addr learned my addr 172.21.15.2:0/727098710 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e58c00640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e34003f40 0x7f7e54152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.118+0000 7f7e53e00640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5415f290 0x7f7e5415f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e59600640 1 -- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e34003f40 msgr2=0x7f7e54152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e59600640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e34003f40 0x7f7e54152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e59600640 1 -- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5415f290 msgr2=0x7f7e5415f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e59600640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5415f290 0x7f7e5415f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e59600640 1 -- 172.21.15.2:0/727098710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540b5ec0 con 0x7f7e540b6c80 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e59600640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 0x7f7e540b7100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7e540b9160 tx=0x7f7e4c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c00e070 con 0x7f7e540b6c80 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4c011e10 con 0x7f7e540b6c80 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e540b61a0 con 0x7f7e540b6c80 2024-01-15T17:57:47.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.119+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c003990 con 0x7f7e540b6c80 2024-01-15T17:57:47.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.120+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e540b6760 con 0x7f7e540b6c80 2024-01-15T17:57:47.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.121+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e540b8db0 con 0x7f7e540b6c80 2024-01-15T17:57:47.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.121+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7e4c003af0 con 0x7f7e540b6c80 2024-01-15T17:57:47.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.122+0000 7f7e50c00640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7e2c06b2c0 0x7f7e2c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.122+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e4c08a250 con 0x7f7e540b6c80 2024-01-15T17:57:47.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.123+0000 7f7e58c00640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7e2c06b2c0 0x7f7e2c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.124+0000 7f7e58c00640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7e2c06b2c0 0x7f7e2c06d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e44001eb0 tx=0x7f7e44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:47.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.127+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e4c0550b0 con 0x7f7e540b6c80 2024-01-15T17:57:47.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.321+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7e54074770 con 0x7f7e540b6c80 2024-01-15T17:57:47.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.322+0000 7f7e50c00640 1 -- 172.21.15.2:0/727098710 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7e4c0586d0 con 0x7f7e540b6c80 2024-01-15T17:57:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:57:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7e2c06b2c0 msgr2=0x7f7e2c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7e2c06b2c0 0x7f7e2c06d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e44001eb0 tx=0x7f7e44009040 comp rx=0 tx=0).stop 2024-01-15T17:57:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 msgr2=0x7f7e540b7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 0x7f7e540b7100 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7e540b9160 tx=0x7f7e4c0115a0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 shutdown_connections 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7e2c06b2c0 0x7f7e2c06d780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e540b6c80 0x7f7e540b7100 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e5415f290 0x7f7e5415f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 --2- 172.21.15.2:0/727098710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e34003f40 0x7f7e54152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 >> 172.21.15.2:0/727098710 conn(0x7f7e540d30b0 msgr2=0x7f7e5415d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.325+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 shutdown_connections 2024-01-15T17:57:47.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.326+0000 7f7e5a000640 1 -- 172.21.15.2:0/727098710 wait complete. 2024-01-15T17:57:47.347 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:57:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.562+0000 7fac08600640 1 Processor -- start 2024-01-15T17:57:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.562+0000 7fac08600640 1 -- start start 2024-01-15T17:57:47.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.563+0000 7fac08600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 0x7fac0015dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fac08600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac000aeab0 con 0x7fac0015e170 2024-01-15T17:57:47.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fac08600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac00070380 con 0x7fac000ce170 2024-01-15T17:57:47.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fac08600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0015e5f0 con 0x7fac0013ce30 2024-01-15T17:57:47.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fac05200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fac0015e170 0x7fac000cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45922/0 (socket says 172.21.15.2:45922) 2024-01-15T17:57:47.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fac05200640 1 -- 172.21.15.2:0/3239957066 learned_addr learned my addr 172.21.15.2:0/3239957066 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fac05c00640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 0x7fac0015dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000298363 0 0) 0x7fac000aeab0 con 0x7fac0015e170 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.564+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0003520 con 0x7fac0015e170 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4139105972 0 0) 0x7fabe0003520 con 0x7fac0015e170 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 msgr2=0x7fac0015dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 0x7fac0015dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 >> v1:172.21.15.137:6789/0 conn(0x7fac000ce170 legacy=0x7fac00070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac000ce6d0 con 0x7fac0015e170 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fabe8001670 con 0x7fac0015e170 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 >> v1:172.21.15.2:6789/0 conn(0x7fac0015e170 legacy=0x7fac000cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe00039a0 0x7fabe0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 0x7fabe00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabe0004d00 0x7fabe0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0015e5f0 con 0x7fabe0004d00 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe000dc40 con 0x7fabe00039a0 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe000de00 con 0x7fac0013ce30 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.565+0000 7fac06600640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabe0004d00 0x7fabe0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac05c00640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe00039a0 0x7fabe0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac05200640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 0x7fabe00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac06600640 1 -- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 msgr2=0x7fabe00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac06600640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac0013ce30 0x7fabe00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac06600640 1 -- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe00039a0 msgr2=0x7fabe0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac06600640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe00039a0 0x7fabe0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.566+0000 7fac06600640 1 -- 172.21.15.2:0/3239957066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe0003520 con 0x7fabe0004d00 2024-01-15T17:57:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.567+0000 7fac06600640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabe0004d00 0x7fabe0005180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fabf8001e20 tx=0x7fabe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3563003317b8e1cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.567+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8003b70 con 0x7fabe0004d00 2024-01-15T17:57:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.567+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8003cd0 con 0x7fabe0004d00 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.567+0000 7fabffe00640 1 -- 172.21.15.2:0/3239957066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8005310 con 0x7fabe0004d00 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac08600640 1 -- 172.21.15.2:0/3239957066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabe0004d00 msgr2=0x7fabe0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac08600640 1 --2- 172.21.15.2:0/3239957066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fabe0004d00 0x7fabe0005180 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fabf8001e20 tx=0x7fabe00057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac06600640 1 -- 172.21.15.2:0/3239957066 reap_dead start 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac08600640 1 -- 172.21.15.2:0/3239957066 shutdown_connections 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac08600640 1 -- 172.21.15.2:0/3239957066 >> 172.21.15.2:0/3239957066 conn(0x7fac0006da20 msgr2=0x7fac0006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac08600640 1 -- 172.21.15.2:0/3239957066 shutdown_connections 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.568+0000 7fac08600640 1 -- 172.21.15.2:0/3239957066 wait complete. 2024-01-15T17:57:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.570+0000 7fac08600640 1 Processor -- start 2024-01-15T17:57:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.570+0000 7fac08600640 1 -- start start 2024-01-15T17:57:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac08600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe0004d00 0x7fac001502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac08600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac00132160 0x7fac001325c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac08600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 0x7fac0013e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac08600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac00150800 con 0x7fac0013e560 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac08600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0015a6b0 con 0x7fabe0004d00 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac08600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0013f000 con 0x7fac00132160 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 0x7fac0013e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 0x7fac0013e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46222/0 (socket says 172.21.15.2:46222) 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 -- 172.21.15.2:0/947783303 learned_addr learned my addr 172.21.15.2:0/947783303 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac05c00640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe0004d00 0x7fac001502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac05200640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac00132160 0x7fac001325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 -- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac00132160 msgr2=0x7fac001325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac00132160 0x7fac001325c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 -- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe0004d00 msgr2=0x7fac001502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe0004d00 0x7fac001502c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.571+0000 7fac06600640 1 -- 172.21.15.2:0/947783303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0013f2e0 con 0x7fac0013e560 2024-01-15T17:57:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.572+0000 7fac06600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 0x7fac0013e9e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fabf8005040 tx=0x7fabf80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.572+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf803e070 con 0x7fac0013e560 2024-01-15T17:57:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.572+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf8003e40 con 0x7fac0013e560 2024-01-15T17:57:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.572+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf8006d50 con 0x7fac0013e560 2024-01-15T17:57:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.572+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac00150800 con 0x7fac0013e560 2024-01-15T17:57:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.572+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0015ac50 con 0x7fac0013e560 2024-01-15T17:57:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.573+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac000686f0 con 0x7fac0013e560 2024-01-15T17:57:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.574+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fabf8007030 con 0x7fac0013e560 2024-01-15T17:57:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.575+0000 7fabfd600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabd806b2c0 0x7fabd806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.575+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fabf80aae40 con 0x7fac0013e560 2024-01-15T17:57:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.576+0000 7fac05c00640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabd806b2c0 0x7fabd806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:47.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.577+0000 7fac05c00640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabd806b2c0 0x7fabd806d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009a10 tx=0x7fabf0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:47.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.580+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fabf8075d40 con 0x7fac0013e560 2024-01-15T17:57:47.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.811+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac00159f60 con 0x7fac0013e560 2024-01-15T17:57:47.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.813+0000 7fabfd600640 1 -- 172.21.15.2:0/947783303 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fabf8048070 con 0x7fac0013e560 2024-01-15T17:57:47.817 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:47.817 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:57:47.817 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabd806b2c0 msgr2=0x7fabd806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabd806b2c0 0x7fabd806d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009a10 tx=0x7fabf0008040 comp rx=0 tx=0).stop 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 msgr2=0x7fac0013e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 0x7fac0013e9e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fabf8005040 tx=0x7fabf80014a0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 shutdown_connections 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabd806b2c0 0x7fabd806d780 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac0013e560 0x7fac0013e9e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac00132160 0x7fac001325c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 --2- 172.21.15.2:0/947783303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fabe0004d00 0x7fac001502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:47.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 >> 172.21.15.2:0/947783303 conn(0x7fac0006da20 msgr2=0x7fac000b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:47.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 shutdown_connections 2024-01-15T17:57:47.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:47.816+0000 7fac08600640 1 -- 172.21.15.2:0/947783303 wait complete. 2024-01-15T17:57:47.838 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:57:48.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.050+0000 7f86b2a00640 1 Processor -- start 2024-01-15T17:57:48.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.051+0000 7f86b2a00640 1 -- start start 2024-01-15T17:57:48.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.052+0000 7f86b2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac15db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.052+0000 7f86b2a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f86ac15e070 0x7f86ac166d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.052+0000 7f86b2a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac0b2e70 con 0x7f86ac15d750 2024-01-15T17:57:48.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.052+0000 7f86b2a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac06c430 con 0x7f86ac15e070 2024-01-15T17:57:48.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.052+0000 7f86b2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0b30c0 con 0x7f86ac0ad960 2024-01-15T17:57:48.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.053+0000 7f86b2000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f86ac15e070 0x7f86ac166d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.053+0000 7f86b0c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac15db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.053+0000 7f86b2000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f86ac15e070 0x7f86ac166d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:41264/0 (socket says 172.21.15.2:41264) 2024-01-15T17:57:48.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.053+0000 7f86b1600640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f86ac15d750 0x7f86ac06cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:58720/0 (socket says 172.21.15.2:58720) 2024-01-15T17:57:48.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.053+0000 7f86b2000640 1 -- 172.21.15.2:0/2130322500 learned_addr learned my addr 172.21.15.2:0/2130322500 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:48.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.054+0000 7f86b2000640 1 -- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 msgr2=0x7f86ac15db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:48.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.054+0000 7f86b2000640 1 --2- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac15db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.054+0000 7f86b2000640 1 -- 172.21.15.2:0/2130322500 >> v1:172.21.15.137:6789/0 conn(0x7f86ac15d750 legacy=0x7f86ac06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.054+0000 7f86b2000640 1 -- 172.21.15.2:0/2130322500 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac0b33a0 con 0x7f86ac15e070 2024-01-15T17:57:48.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.055+0000 7f86b2000640 1 --2- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15e070 0x7f86ac166d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86a4009e50 tx=0x7f86a402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ee3bdc1d7f07799 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:48.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.056+0000 7f86abe00640 1 -- 172.21.15.2:0/2130322500 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a402cb00 con 0x7f86ac15e070 2024-01-15T17:57:48.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.056+0000 7f86abe00640 1 -- 172.21.15.2:0/2130322500 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a402cc60 con 0x7f86ac15e070 2024-01-15T17:57:48.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.056+0000 7f86abe00640 1 -- 172.21.15.2:0/2130322500 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a40343f0 con 0x7f86ac15e070 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 -- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15e070 msgr2=0x7f86ac166d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 --2- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15e070 0x7f86ac166d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f86a4009e50 tx=0x7f86a402bf60 comp rx=0 tx=0).stop 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 -- 172.21.15.2:0/2130322500 shutdown_connections 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 --2- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15e070 0x7f86ac166d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 --2- 172.21.15.2:0/2130322500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac15db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 -- 172.21.15.2:0/2130322500 >> 172.21.15.2:0/2130322500 conn(0x7f86ac0b0840 msgr2=0x7f86ac0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 -- 172.21.15.2:0/2130322500 shutdown_connections 2024-01-15T17:57:48.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.057+0000 7f86b2a00640 1 -- 172.21.15.2:0/2130322500 wait complete. 2024-01-15T17:57:48.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.059+0000 7f86b2a00640 1 Processor -- start 2024-01-15T17:57:48.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.059+0000 7f86b2a00640 1 -- start start 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.060+0000 7f86b2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15d750 0x7f86ac1403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 0x7f86ac140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac145240 con 0x7f86ac15e070 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac140e60 con 0x7f86ac15d750 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac155ab0 con 0x7f86ac0ad960 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 0x7f86ac140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 0x7f86ac140920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46228/0 (socket says 172.21.15.2:46228) 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b2000640 1 -- 172.21.15.2:0/887671315 learned_addr learned my addr 172.21.15.2:0/887671315 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b0c00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15d750 0x7f86ac1403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.061+0000 7f86b1600640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2000640 1 -- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 msgr2=0x7f86ac155570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2000640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac155570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2000640 1 -- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15d750 msgr2=0x7f86ac1403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2000640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15d750 0x7f86ac1403e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2000640 1 -- 172.21.15.2:0/887671315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a4009b30 con 0x7f86ac15e070 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2000640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 0x7f86ac140920 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86a4036040 tx=0x7f86a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4009510 con 0x7f86ac15e070 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a403e450 con 0x7f86ac15e070 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac1445c0 con 0x7f86ac15e070 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a403e800 con 0x7f86ac15e070 2024-01-15T17:57:48.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.062+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac144b80 con 0x7f86ac15e070 2024-01-15T17:57:48.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.064+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac074770 con 0x7f86ac15e070 2024-01-15T17:57:48.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.065+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f86a403e960 con 0x7f86ac15e070 2024-01-15T17:57:48.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.065+0000 7f86a9600640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f868406b2c0 0x7f868406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.066+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86a40ac8e0 con 0x7f86ac15e070 2024-01-15T17:57:48.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.066+0000 7f86b1600640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f868406b2c0 0x7f868406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.067+0000 7f86b1600640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f868406b2c0 0x7f868406d780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f869c000f60 tx=0x7f869c007c80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:48.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.071+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86a4077740 con 0x7f86ac15e070 2024-01-15T17:57:48.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.273+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86ac0b2460 con 0x7f86ac15e070 2024-01-15T17:57:48.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.275+0000 7f86a9600640 1 -- 172.21.15.2:0/887671315 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f86a407ad60 con 0x7f86ac15e070 2024-01-15T17:57:48.279 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:48.280 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f868406b2c0 msgr2=0x7f868406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f868406b2c0 0x7f868406d780 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f869c000f60 tx=0x7f869c007c80 comp rx=0 tx=0).stop 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 msgr2=0x7f86ac140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 0x7f86ac140920 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f86a4036040 tx=0x7f86a4004a40 comp rx=0 tx=0).stop 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 shutdown_connections 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f868406b2c0 0x7f868406d780 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f86ac15e070 0x7f86ac140920 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f86ac15d750 0x7f86ac1403e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 --2- 172.21.15.2:0/887671315 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f86ac0ad960 0x7f86ac155570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 >> 172.21.15.2:0/887671315 conn(0x7f86ac0b0840 msgr2=0x7f86ac070440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:48.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 shutdown_connections 2024-01-15T17:57:48.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.278+0000 7f86b2a00640 1 -- 172.21.15.2:0/887671315 wait complete. 2024-01-15T17:57:48.303 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:57:48.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.523+0000 7f5645a00640 1 Processor -- start 2024-01-15T17:57:48.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.524+0000 7f5645a00640 1 -- start start 2024-01-15T17:57:48.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.525+0000 7f5645a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f564015d340 0x7f56400cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.525+0000 7f5645a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 0x7f56400d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.525+0000 7f5645a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56400d3a30 con 0x7f56400d1680 2024-01-15T17:57:48.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.525+0000 7f5645a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56400b3de0 con 0x7f564015d340 2024-01-15T17:57:48.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.525+0000 7f5645a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400b40c0 con 0x7f56400d3090 2024-01-15T17:57:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.526+0000 7f563f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 0x7f56400d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.526+0000 7f563fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f564015d340 0x7f56400cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.526+0000 7f5645000640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f56400d1680 0x7f56400d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:58732/0 (socket says 172.21.15.2:58732) 2024-01-15T17:57:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.526+0000 7f563f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 0x7f56400d34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:38294/0 (socket says 172.21.15.2:38294) 2024-01-15T17:57:48.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.526+0000 7f5645000640 1 -- 172.21.15.2:0/1947249210 learned_addr learned my addr 172.21.15.2:0/1947249210 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:48.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.527+0000 7f563fe00640 1 -- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 msgr2=0x7f56400d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:48.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.527+0000 7f563fe00640 1 --2- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 0x7f56400d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.527+0000 7f563fe00640 1 -- 172.21.15.2:0/1947249210 >> v1:172.21.15.137:6789/0 conn(0x7f56400d1680 legacy=0x7f56400d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.527+0000 7f563fe00640 1 -- 172.21.15.2:0/1947249210 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56400d0e80 con 0x7f564015d340 2024-01-15T17:57:48.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.527+0000 7f563f400640 1 --2- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 0x7f56400d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.528+0000 7f563fe00640 1 --2- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564015d340 0x7f56400cdd20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5630009d90 tx=0x7f563002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c16bd16f9559ef9f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.528+0000 7f563ea00640 1 -- 172.21.15.2:0/1947249210 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563002cc50 con 0x7f564015d340 2024-01-15T17:57:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f563ea00640 1 -- 172.21.15.2:0/1947249210 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563002cdb0 con 0x7f564015d340 2024-01-15T17:57:48.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f563ea00640 1 -- 172.21.15.2:0/1947249210 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5630034660 con 0x7f564015d340 2024-01-15T17:57:48.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 -- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564015d340 msgr2=0x7f56400cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 --2- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564015d340 0x7f56400cdd20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5630009d90 tx=0x7f563002bea0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 -- 172.21.15.2:0/1947249210 shutdown_connections 2024-01-15T17:57:48.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 --2- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56400d3090 0x7f56400d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 --2- 172.21.15.2:0/1947249210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f564015d340 0x7f56400cdd20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 -- 172.21.15.2:0/1947249210 >> 172.21.15.2:0/1947249210 conn(0x7f564008caf0 msgr2=0x7f564009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 -- 172.21.15.2:0/1947249210 shutdown_connections 2024-01-15T17:57:48.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.529+0000 7f5645a00640 1 -- 172.21.15.2:0/1947249210 wait complete. 2024-01-15T17:57:48.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.531+0000 7f5645a00640 1 Processor -- start 2024-01-15T17:57:48.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.531+0000 7f5645a00640 1 -- start start 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f5645a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56400d1680 0x7f564014e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f5645a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 0x7f56400b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f5645a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564015d340 0x7f56400b62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f5645a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564014eac0 con 0x7f56400d3090 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f5645a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56401f7220 con 0x7f56400d1680 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f5645a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56400c7dc0 con 0x7f564015d340 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.532+0000 7f563f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 0x7f56400b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 0x7f56400b0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46230/0 (socket says 172.21.15.2:46230) 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 -- 172.21.15.2:0/2587424182 learned_addr learned my addr 172.21.15.2:0/2587424182 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f5645000640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564015d340 0x7f56400b62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563fe00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56400d1680 0x7f564014e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 -- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564015d340 msgr2=0x7f56400b62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564015d340 0x7f56400b62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 -- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56400d1680 msgr2=0x7f564014e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56400d1680 0x7f564014e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 -- 172.21.15.2:0/2587424182 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5628009400 con 0x7f56400d3090 2024-01-15T17:57:48.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.533+0000 7f563f400640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 0x7f56400b0e80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f562800e960 tx=0x7f562800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.534+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800ca00 con 0x7f56400d3090 2024-01-15T17:57:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.534+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562800cb60 con 0x7f56400d3090 2024-01-15T17:57:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.534+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5628013470 con 0x7f56400d3090 2024-01-15T17:57:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.534+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5630009a70 con 0x7f56400d3090 2024-01-15T17:57:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.534+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56400c8380 con 0x7f56400d3090 2024-01-15T17:57:48.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.535+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640074770 con 0x7f56400d3090 2024-01-15T17:57:48.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.536+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f56280135d0 con 0x7f56400d3090 2024-01-15T17:57:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.537+0000 7f5637e00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f561806b2c0 0x7f561806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.537+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5628089e80 con 0x7f56400d3090 2024-01-15T17:57:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.538+0000 7f563fe00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f561806b2c0 0x7f561806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:48.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.539+0000 7f563fe00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f561806b2c0 0x7f561806d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5630036000 tx=0x7f5630030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.541+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5628054ce0 con 0x7f56400d3090 2024-01-15T17:57:48.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.773+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56400678e0 con 0x7f56400d3090 2024-01-15T17:57:48.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.774+0000 7f5637e00640 1 -- 172.21.15.2:0/2587424182 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f5628058300 con 0x7f56400d3090 2024-01-15T17:57:48.778 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:57:48.779 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:57:48.779 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f561806b2c0 msgr2=0x7f561806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f561806b2c0 0x7f561806d780 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5630036000 tx=0x7f5630030000 comp rx=0 tx=0).stop 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 msgr2=0x7f56400b0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 0x7f56400b0e80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f562800e960 tx=0x7f562800ee20 comp rx=0 tx=0).stop 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 shutdown_connections 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f561806b2c0 0x7f561806d780 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f564015d340 0x7f56400b62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56400d3090 0x7f56400b0e80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 --2- 172.21.15.2:0/2587424182 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56400d1680 0x7f564014e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 >> 172.21.15.2:0/2587424182 conn(0x7f564008caf0 msgr2=0x7f56400d0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 shutdown_connections 2024-01-15T17:57:48.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:48.777+0000 7f5645a00640 1 -- 172.21.15.2:0/2587424182 wait complete. 2024-01-15T17:57:49.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T17:57:31.583413+0000', 'modified': '2024-01-15T17:57:34.648185+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4233}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4233': {'gid': 4233, 'name': 'a', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.2:6833/3854928362', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6832', 'nonce': 3854928362}, {'type': 'v1', 'addr': '172.21.15.2:6833', 'nonce': 3854928362}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T17:57:49.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T17:57:49.202 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T17:57:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.431+0000 7f0c53e00640 1 Processor -- start 2024-01-15T17:57:49.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.431+0000 7f0c53e00640 1 -- start start 2024-01-15T17:57:49.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c53e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0c5406d350 0x7f0c5406d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c540ce580 0x7f0c5415d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c53e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5406dc70 con 0x7f0c54160e50 2024-01-15T17:57:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c53e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c540cb3a0 con 0x7f0c5406d350 2024-01-15T17:57:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c53e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540cb680 con 0x7f0c540ce580 2024-01-15T17:57:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c52a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0c5406d350 0x7f0c5406d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c53400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c540ce580 0x7f0c5415d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c52a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0c5406d350 0x7f0c5406d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:41306/0 (socket says 172.21.15.2:41306) 2024-01-15T17:57:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.433+0000 7f0c52a00640 1 -- 172.21.15.2:0/4047645803 learned_addr learned my addr 172.21.15.2:0/4047645803 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.434+0000 7f0c52a00640 1 -- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c540ce580 msgr2=0x7f0c5415d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.434+0000 7f0c52a00640 1 --2- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c540ce580 0x7f0c5415d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.434+0000 7f0c52a00640 1 -- 172.21.15.2:0/4047645803 >> v1:172.21.15.137:6789/0 conn(0x7f0c54160e50 legacy=0x7f0c540cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.434+0000 7f0c52a00640 1 -- 172.21.15.2:0/4047645803 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540cb960 con 0x7f0c5406d350 2024-01-15T17:57:49.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.436+0000 7f0c52a00640 1 --2- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c5406d350 0x7f0c5406d730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009870 tx=0x7f0c4402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=865c4ab890ec5832 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.436+0000 7f0c52000640 1 -- 172.21.15.2:0/4047645803 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44030840 con 0x7f0c5406d350 2024-01-15T17:57:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.436+0000 7f0c52000640 1 -- 172.21.15.2:0/4047645803 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c440309a0 con 0x7f0c5406d350 2024-01-15T17:57:49.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.436+0000 7f0c52000640 1 -- 172.21.15.2:0/4047645803 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44030d30 con 0x7f0c5406d350 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 -- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c5406d350 msgr2=0x7f0c5406d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 --2- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c5406d350 0x7f0c5406d730 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009870 tx=0x7f0c4402b980 comp rx=0 tx=0).stop 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 -- 172.21.15.2:0/4047645803 shutdown_connections 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 --2- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c540ce580 0x7f0c5415d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 --2- 172.21.15.2:0/4047645803 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c5406d350 0x7f0c5406d730 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 -- 172.21.15.2:0/4047645803 >> 172.21.15.2:0/4047645803 conn(0x7f0c540b0840 msgr2=0x7f0c540b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 -- 172.21.15.2:0/4047645803 shutdown_connections 2024-01-15T17:57:49.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.437+0000 7f0c53e00640 1 -- 172.21.15.2:0/4047645803 wait complete. 2024-01-15T17:57:49.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.439+0000 7f0c53e00640 1 Processor -- start 2024-01-15T17:57:49.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.439+0000 7f0c53e00640 1 -- start start 2024-01-15T17:57:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c5406d350 0x7f0c540b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c540ce580 0x7f0c54140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 0x7f0c5415e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415efa0 con 0x7f0c54160e50 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415f280 con 0x7f0c540ce580 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415f560 con 0x7f0c5406d350 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 0x7f0c5415e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 0x7f0c5415e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46240/0 (socket says 172.21.15.2:46240) 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53400640 1 -- 172.21.15.2:0/483470367 learned_addr learned my addr 172.21.15.2:0/483470367 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c52a00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c5406d350 0x7f0c540b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c4be00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c540ce580 0x7f0c54140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53400640 1 -- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c5406d350 msgr2=0x7f0c540b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.440+0000 7f0c53400640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c5406d350 0x7f0c540b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c53400640 1 -- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c540ce580 msgr2=0x7f0c54140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c53400640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c540ce580 0x7f0c54140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c53400640 1 -- 172.21.15.2:0/483470367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c440094d0 con 0x7f0c54160e50 2024-01-15T17:57:49.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c53400640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 0x7f0c5415e550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00efc0 tx=0x7f0c4c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c4aa00640 1 -- 172.21.15.2:0/483470367 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c014480 con 0x7f0c54160e50 2024-01-15T17:57:49.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c4aa00640 1 -- 172.21.15.2:0/483470367 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c00f040 con 0x7f0c54160e50 2024-01-15T17:57:49.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c4aa00640 1 -- 172.21.15.2:0/483470367 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c014820 con 0x7f0c54160e50 2024-01-15T17:57:49.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c5415f840 con 0x7f0c54160e50 2024-01-15T17:57:49.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.441+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c540b3730 con 0x7f0c54160e50 2024-01-15T17:57:49.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.444+0000 7f0c4aa00640 1 -- 172.21.15.2:0/483470367 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0c4c014980 con 0x7f0c54160e50 2024-01-15T17:57:49.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.444+0000 7f0c4aa00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c2c06b2c0 0x7f0c2c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.445+0000 7f0c4aa00640 1 -- 172.21.15.2:0/483470367 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c4c08aa80 con 0x7f0c54160e50 2024-01-15T17:57:49.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.445+0000 7f0c52a00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c2c06b2c0 0x7f0c2c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.445+0000 7f0c5c98e740 1 Processor -- start 2024-01-15T17:57:49.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.445+0000 7f0c5c98e740 1 -- start start 2024-01-15T17:57:49.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.446+0000 7f0c52a00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c2c06b2c0 0x7f0c2c06d780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c44036000 tx=0x7f0c44008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.446+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 0x55ed1806b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.446+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed18065d70 0x55ed1806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.446+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180ae730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed17fc9880 con 0x55ed1806b070 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed17fd8ae0 con 0x55ed18065d70 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed17fd61d0 con 0x55ed1806a7b0 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 0x55ed1806b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 0x55ed1806b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46254/0 (socket says 172.21.15.2:46254) 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 -- 172.21.15.2:0/71631093 learned_addr learned my addr 172.21.15.2:0/71631093 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c4be00640 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed18065d70 0x55ed1806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c53400640 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 -- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 msgr2=0x55ed180ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 -- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed18065d70 msgr2=0x55ed1806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed18065d70 0x55ed1806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c52a00640 1 -- 172.21.15.2:0/71631093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed17f99860 con 0x55ed1806b070 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.447+0000 7f0c53400640 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.448+0000 7f0c52a00640 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 0x55ed1806b450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009960 tx=0x7f0c44030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf362c869e88d66 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.448+0000 7f0c43400640 1 -- 172.21.15.2:0/71631093 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44045e40 con 0x55ed1806b070 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.448+0000 7f0c43400640 1 -- 172.21.15.2:0/71631093 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c44039450 con 0x55ed1806b070 2024-01-15T17:57:49.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.448+0000 7f0c43400640 1 -- 172.21.15.2:0/71631093 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c440397c0 con 0x55ed1806b070 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.448+0000 7f0c5c98e740 1 -- 172.21.15.2:0/71631093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 msgr2=0x55ed1806b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.448+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 0x55ed1806b450 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009960 tx=0x7f0c44030000 comp rx=0 tx=0).stop 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 -- 172.21.15.2:0/71631093 shutdown_connections 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed18065d70 0x55ed1806a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/71631093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed1806b070 0x55ed1806b450 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 -- 172.21.15.2:0/71631093 >> 172.21.15.2:0/71631093 conn(0x55ed1802cc70 msgr2=0x55ed180b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:49.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 -- 172.21.15.2:0/71631093 shutdown_connections 2024-01-15T17:57:49.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.449+0000 7f0c5c98e740 1 -- 172.21.15.2:0/71631093 wait complete. 2024-01-15T17:57:49.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.450+0000 7f0c5c98e740 1 -- start start 2024-01-15T17:57:49.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.450+0000 7f0c5c98e740 1 Processor -- start 2024-01-15T17:57:49.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed1808a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed17fd1ac0 con 0x55ed18065d70 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed17f6e2f0 con 0x55ed1806b070 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed17f8bc70 con 0x55ed1806a7b0 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c5c98e740 10 client.0 _ll_register_callbacks cb 0x55ed17f903b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed1808a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed1808a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46260/0 (socket says 172.21.15.2:46260) 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 -- 172.21.15.2:0/3279149300 learned_addr learned my addr 172.21.15.2:0/3279149300 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 -- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 msgr2=0x55ed180b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c53400640 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 -- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 msgr2=0x55ed180b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.451+0000 7f0c52a00640 1 -- 172.21.15.2:0/3279149300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c540cb960 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c52a00640 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed1808a2d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009990 tx=0x7f0c44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b5ed715917be318 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c3be00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c3be00640 1 -- 172.21.15.2:0/3279149300 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44039c40 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c3be00640 1 -- 172.21.15.2:0/3279149300 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c4403e040 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed17fcaaa0 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c3be00640 1 -- 172.21.15.2:0/3279149300 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c44039da0 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c5c98e740 10 client.4614 fetch_fsmap learned FSMap version 7 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.452+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55ed17fd0140 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c3be00640 1 -- 172.21.15.2:0/3279149300 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ed17fcaaa0 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c3be00640 1 -- 172.21.15.2:0/3279149300 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0c44042050 con 0x55ed18065d70 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c5c98e740 10 client.4614 fetch_fsmap finished waiting for FSMap version 7 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c5c98e740 10 client.4614 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c5c98e740 20 client.4614 populate_metadata read hostname 'smithi002' 2024-01-15T17:57:49.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed1808b9b0 0x55ed180ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c5c98e740 4 client.4614 mds_command: new command op to 4233 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T17:57:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 --> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed17fca7c0 con 0x55ed1808b9b0 2024-01-15T17:57:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.453+0000 7f0c4be00640 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed1808b9b0 0x55ed180ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.454+0000 7f0c4be00640 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed1808b9b0 0x55ed180ae4f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75a593b6bc7518d2 server_cookie=479dce9e745a2560 in_seq=0 out_seq=0 2024-01-15T17:57:49.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.454+0000 7f0c3be00640 10 client.4614 ms_handle_connect on v2:172.21.15.2:6832/3854928362 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.455+0000 7f0c3be00640 1 -- 172.21.15.2:0/3279149300 <== mds.0 v2:172.21.15.2:6832/3854928362 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ed17fca7c0 con 0x55ed1808b9b0 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.455+0000 7f0c3be00640 10 client.4614 handle_command_reply: tid=0 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.455+0000 7f0c5c98e740 1 client.4614 shutdown 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.455+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 msgr2=0x55ed1808a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.455+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed1808a2d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009990 tx=0x7f0c44004730 comp rx=0 tx=0).stop 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 shutdown_connections 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed1808b9b0 0x55ed180ae4f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3279149300 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed1808a2d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c3be00640 0 client.4614 ms_handle_reset on v2:172.21.15.2:6832/3854928362 2024-01-15T17:57:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 >> 172.21.15.2:0/3279149300 conn(0x55ed1802cc70 msgr2=0x55ed180b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 shutdown_connections 2024-01-15T17:57:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3279149300 wait complete. 2024-01-15T17:57:49.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.456+0000 7f0c5c98e740 20 client.4614 trim_cache size 0 max 16384 2024-01-15T17:57:49.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.473+0000 7f0c5c98e740 1 Processor -- start 2024-01-15T17:57:49.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.473+0000 7f0c5c98e740 1 -- start start 2024-01-15T17:57:49.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed180b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed18064dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed17d34f20 con 0x55ed18065d70 2024-01-15T17:57:49.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed180b2cf0 con 0x55ed1806b070 2024-01-15T17:57:49.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed180b4f40 con 0x55ed1806a7b0 2024-01-15T17:57:49.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.474+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed180b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed180b37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46270/0 (socket says 172.21.15.2:46270) 2024-01-15T17:57:49.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 -- 172.21.15.2:0/3808869692 learned_addr learned my addr 172.21.15.2:0/3808869692 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:49.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c4be00640 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c53400640 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed18064dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 -- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 msgr2=0x55ed180b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 -- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 msgr2=0x55ed18064dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed18064dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 -- 172.21.15.2:0/3808869692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed17f6db80 con 0x55ed18065d70 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c52a00640 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed180b37d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c440090b0 tx=0x7f0c44053000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6754f9c1da5fe8ab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c43e00640 1 -- 172.21.15.2:0/3808869692 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44045910 con 0x55ed18065d70 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c43e00640 1 -- 172.21.15.2:0/3808869692 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4404e550 con 0x55ed18065d70 2024-01-15T17:57:49.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.475+0000 7f0c43e00640 1 -- 172.21.15.2:0/3808869692 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4404ea60 con 0x55ed18065d70 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3808869692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 msgr2=0x55ed180b37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed180b37d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0c440090b0 tx=0x7f0c44053000 comp rx=0 tx=0).stop 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3808869692 shutdown_connections 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed18064dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed180b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/3808869692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed180b37d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3808869692 >> 172.21.15.2:0/3808869692 conn(0x55ed1802cc70 msgr2=0x55ed180994a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:49.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3808869692 shutdown_connections 2024-01-15T17:57:49.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.476+0000 7f0c5c98e740 1 -- 172.21.15.2:0/3808869692 wait complete. 2024-01-15T17:57:49.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.477+0000 7f0c5c98e740 1 -- start start 2024-01-15T17:57:49.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.477+0000 7f0c5c98e740 1 Processor -- start 2024-01-15T17:57:49.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed18119ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed1808f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed180919a0 con 0x55ed18065d70 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed18065850 con 0x55ed1806b070 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ed180b1110 con 0x55ed1806a7b0 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.478+0000 7f0c5c98e740 10 client.0 _ll_register_callbacks cb 0x7f0c440096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed18119ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed18119ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46286/0 (socket says 172.21.15.2:46286) 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 -- 172.21.15.2:0/1308548224 learned_addr learned my addr 172.21.15.2:0/1308548224 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c4be00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed1808f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 -- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 msgr2=0x55ed1808f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed1808f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 -- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 msgr2=0x55ed180b7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 -- 172.21.15.2:0/1308548224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed17fd0140 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c4be00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed1808f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c52a00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed18119ec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009990 tx=0x7f0c44009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784c28593eab3a03 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.479+0000 7f0c3be00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c3be00640 1 -- 172.21.15.2:0/1308548224 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44045910 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c3be00640 1 -- 172.21.15.2:0/1308548224 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c4404eb80 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ed180c1150 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c3be00640 1 -- 172.21.15.2:0/1308548224 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c4404ed40 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 10 client.4620 fetch_fsmap learned FSMap version 7 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x55ed17f93170 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c3be00640 1 -- 172.21.15.2:0/1308548224 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ed180c1150 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c3be00640 1 -- 172.21.15.2:0/1308548224 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0c44058410 con 0x55ed18065d70 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 10 client.4620 fetch_fsmap finished waiting for FSMap version 7 2024-01-15T17:57:49.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 10 client.4620 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 20 client.4620 populate_metadata read hostname 'smithi002' 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed180b81e0 0x55ed180b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 4 client.4620 mds_command: new command op to 4233 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.480+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 --> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f0c44039450 con 0x55ed180b81e0 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c4be00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed180b81e0 0x55ed180b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c3be00640 10 client.4620 ms_handle_connect on v2:172.21.15.2:6832/3854928362 2024-01-15T17:57:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c4be00640 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed180b81e0 0x55ed180b37d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb1a5a9911fe4eee server_cookie=544603102fd2838 in_seq=0 out_seq=0 2024-01-15T17:57:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c3be00640 1 -- 172.21.15.2:0/1308548224 <== mds.0 v2:172.21.15.2:6832/3854928362 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x7f0c44039450 con 0x55ed180b81e0 2024-01-15T17:57:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c3be00640 10 client.4620 handle_command_reply: tid=0 2024-01-15T17:57:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c5c98e740 1 client.4620 shutdown 2024-01-15T17:57:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 msgr2=0x55ed18119ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.481+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed18119ec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009990 tx=0x7f0c44009240 comp rx=0 tx=0).stop 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 shutdown_connections 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55ed1806b070 0x55ed180b7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:6832/3854928362,v1:172.21.15.2:6833/3854928362] conn(0x55ed180b81e0 0x55ed180b37d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55ed1806a7b0 0x55ed1808f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 --2- 172.21.15.2:0/1308548224 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55ed18065d70 0x55ed18119ec0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c3be00640 0 client.4620 ms_handle_reset on v2:172.21.15.2:6832/3854928362 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 >> 172.21.15.2:0/1308548224 conn(0x55ed1802cc70 msgr2=0x55ed180b4900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 shutdown_connections 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 1 -- 172.21.15.2:0/1308548224 wait complete. 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.482+0000 7f0c5c98e740 20 client.4620 trim_cache size 0 max 16384 2024-01-15T17:57:49.489 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4233, 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 17.354443402000001, 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 7, 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 22, 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 22, 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 24.862385884999998 2024-01-15T17:57:49.490 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c2c06b2c0 msgr2=0x7f0c2c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c2c06b2c0 0x7f0c2c06d780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c44036000 tx=0x7f0c44008030 comp rx=0 tx=0).stop 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 msgr2=0x7f0c5415e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 0x7f0c5415e550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00efc0 tx=0x7f0c4c00c490 comp rx=0 tx=0).stop 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 shutdown_connections 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c2c06b2c0 0x7f0c2c06d780 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0c54160e50 0x7f0c5415e550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0c540ce580 0x7f0c54140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 --2- 172.21.15.2:0/483470367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0c5406d350 0x7f0c540b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:49.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 >> 172.21.15.2:0/483470367 conn(0x7f0c540b0840 msgr2=0x7f0c540ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:49.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 shutdown_connections 2024-01-15T17:57:49.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:49.483+0000 7f0c53e00640 1 -- 172.21.15.2:0/483470367 wait complete. 2024-01-15T17:57:49.508 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-15T17:57:49.533 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-15T17:57:49.533 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-15T17:57:49.533 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-15T17:57:49.533 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T17:57:49.533 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:49.533 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-15T17:57:49.533 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T17:57:49.534 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T17:57:49.534 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-15T17:57:49.534 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T17:57:49.534 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T17:57:49.534 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-15T17:57:49.534 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-15T17:57:49.534 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T17:57:49.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:49.534 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T17:57:49.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:49.591 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2024-01-15T17:57:49.696 INFO:teuthology.orchestra.run.smithi002.stderr:Cannot find device "ceph-brx" 2024-01-15T17:57:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:57:49.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:49.701 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T17:57:49.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:49.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link delete ceph-brx 2024-01-15T17:57:49.846 INFO:teuthology.orchestra.run.smithi137.stderr:Cannot find device "ceph-brx" 2024-01-15T17:57:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T17:57:49.850 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-15T17:57:49.850 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T17:57:50.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.075+0000 7f4721800640 1 Processor -- start 2024-01-15T17:57:50.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.075+0000 7f4721800640 1 -- start start 2024-01-15T17:57:50.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.077+0000 7f4721800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 0x7f471c160e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:50.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.077+0000 7f4721800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 0x7f471c06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:50.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.077+0000 7f4721800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c146ed0 con 0x7f471c15d890 2024-01-15T17:57:50.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.077+0000 7f4721800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c070f00 con 0x7f471c070a70 2024-01-15T17:57:50.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.077+0000 7f4721800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c161430 con 0x7f471c0b2f90 2024-01-15T17:57:50.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.078+0000 7f471aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 0x7f471c160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:50.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.078+0000 7f471b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 0x7f471c06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:50.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.078+0000 7f471b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 0x7f471c06bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:41352/0 (socket says 172.21.15.2:41352) 2024-01-15T17:57:50.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.078+0000 7f471b400640 1 -- 172.21.15.2:0/4292416137 learned_addr learned my addr 172.21.15.2:0/4292416137 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:50.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.078+0000 7f471aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 0x7f471c160e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:38374/0 (socket says 172.21.15.2:38374) 2024-01-15T17:57:50.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.079+0000 7f471aa00640 1 -- 172.21.15.2:0/4292416137 >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 msgr2=0x7f471c06bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:50.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.079+0000 7f471aa00640 1 --2- 172.21.15.2:0/4292416137 >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 0x7f471c06bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.079+0000 7f471aa00640 1 -- 172.21.15.2:0/4292416137 >> v1:172.21.15.137:6789/0 conn(0x7f471c15d890 legacy=0x7f471c0b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:50.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.079+0000 7f471aa00640 1 -- 172.21.15.2:0/4292416137 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c161710 con 0x7f471c0b2f90 2024-01-15T17:57:50.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.079+0000 7f471b400640 1 --2- 172.21.15.2:0/4292416137 >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 0x7f471c06bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:57:50.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.079+0000 7f471a000640 1 -- 172.21.15.2:0/4292416137 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074186904 0 0) 0x7f471c146ed0 con 0x7f471c15d890 2024-01-15T17:57:50.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.081+0000 7f471aa00640 1 --2- 172.21.15.2:0/4292416137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 0x7f471c160e50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f470c0097a0 tx=0x7f470c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b76cbc42a9c77db5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:50.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.081+0000 7f471a000640 1 -- 172.21.15.2:0/4292416137 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c02d040 con 0x7f471c0b2f90 2024-01-15T17:57:50.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.081+0000 7f471a000640 1 -- 172.21.15.2:0/4292416137 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c031810 con 0x7f471c0b2f90 2024-01-15T17:57:50.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.081+0000 7f471a000640 1 -- 172.21.15.2:0/4292416137 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c031ba0 con 0x7f471c0b2f90 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 -- 172.21.15.2:0/4292416137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 msgr2=0x7f471c160e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 --2- 172.21.15.2:0/4292416137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 0x7f471c160e50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f470c0097a0 tx=0x7f470c02b8b0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 -- 172.21.15.2:0/4292416137 shutdown_connections 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 --2- 172.21.15.2:0/4292416137 >> v2:172.21.15.137:3300/0 conn(0x7f471c070a70 0x7f471c06bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 --2- 172.21.15.2:0/4292416137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c0b2f90 0x7f471c160e50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 -- 172.21.15.2:0/4292416137 >> 172.21.15.2:0/4292416137 conn(0x7f471c0663d0 msgr2=0x7f471c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 -- 172.21.15.2:0/4292416137 shutdown_connections 2024-01-15T17:57:50.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.082+0000 7f4721800640 1 -- 172.21.15.2:0/4292416137 wait complete. 2024-01-15T17:57:50.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.085+0000 7f4721800640 1 Processor -- start 2024-01-15T17:57:50.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.085+0000 7f4721800640 1 -- start start 2024-01-15T17:57:50.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4721800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 0x7f471c15c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:50.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4721800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f471c0b2f90 0x7f471c0ab300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:50.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4721800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c15d890 0x7f471c0cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:50.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4721800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0a9e60 con 0x7f471c070a70 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4721800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c15c580 con 0x7f471c0b2f90 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4721800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c0cb9f0 con 0x7f471c15d890 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f471aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 0x7f471c15c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f471aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 0x7f471c15c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46290/0 (socket says 172.21.15.2:46290) 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f471aa00640 1 -- 172.21.15.2:0/366411385 learned_addr learned my addr 172.21.15.2:0/366411385 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f4713e00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f471c0b2f90 0x7f471c0ab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f471b400640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c15d890 0x7f471c0cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.087+0000 7f471aa00640 1 -- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c15d890 msgr2=0x7f471c0cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f471aa00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c15d890 0x7f471c0cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f471aa00640 1 -- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f471c0b2f90 msgr2=0x7f471c0ab300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f471aa00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f471c0b2f90 0x7f471c0ab300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f471aa00640 1 -- 172.21.15.2:0/366411385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714009b30 con 0x7f471c070a70 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f471b400640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c15d890 0x7f471c0cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:50.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f471aa00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 0x7f471c15c040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f470c0098c0 tx=0x7f470c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:50.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c006e10 con 0x7f471c070a70 2024-01-15T17:57:50.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470c03ad60 con 0x7f471c070a70 2024-01-15T17:57:50.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470c009400 con 0x7f471c070a70 2024-01-15T17:57:50.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470c044620 con 0x7f471c070a70 2024-01-15T17:57:50.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.088+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c0cbfb0 con 0x7f471c070a70 2024-01-15T17:57:50.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.090+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c0686f0 con 0x7f471c070a70 2024-01-15T17:57:50.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.090+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f470c04d440 con 0x7f471c070a70 2024-01-15T17:57:50.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.091+0000 7f4712a00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46f406b2c0 0x7f46f406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:50.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.091+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f470c0abf50 con 0x7f471c070a70 2024-01-15T17:57:50.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.092+0000 7f4713e00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46f406b2c0 0x7f46f406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:50.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.093+0000 7f4713e00640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46f406b2c0 0x7f46f406d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4704007fd0 tx=0x7f4704009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:50.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.097+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f470c076db0 con 0x7f471c070a70 2024-01-15T17:57:50.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.334+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f471c0b5270 con 0x7f471c070a70 2024-01-15T17:57:50.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.335+0000 7f4712a00640 1 -- 172.21.15.2:0/366411385 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f470c035070 con 0x7f471c070a70 2024-01-15T17:57:50.339 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46f406b2c0 msgr2=0x7f46f406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46f406b2c0 0x7f46f406d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4704007fd0 tx=0x7f4704009040 comp rx=0 tx=0).stop 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 msgr2=0x7f471c15c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 0x7f471c15c040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f470c0098c0 tx=0x7f470c0046c0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 shutdown_connections 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46f406b2c0 0x7f46f406d780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f471c15d890 0x7f471c0cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f471c0b2f90 0x7f471c0ab300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 --2- 172.21.15.2:0/366411385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f471c070a70 0x7f471c15c040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 >> 172.21.15.2:0/366411385 conn(0x7f471c0663d0 msgr2=0x7f471c15fe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:50.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 shutdown_connections 2024-01-15T17:57:50.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:50.338+0000 7f4721800640 1 -- 172.21.15.2:0/366411385 wait complete. 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T17:57:50.362 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T17:57:50.363 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T17:57:50.363 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:50.363 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:50.413 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42612sec preferred_lft 42612sec 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:50.414 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T17:57:50.415 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T17:57:50.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:50.415 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:50.415 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:50.416 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add name ceph-brx type bridge 2024-01-15T17:57:50.416 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr flush dev ceph-brx 2024-01-15T17:57:50.416 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set ceph-brx up 2024-01-15T17:57:50.416 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T17:57:50.416 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:50.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:50.859 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T17:57:50.949 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T17:57:50.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:50.954 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T17:57:51.018 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T17:57:51.018 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T17:57:51.018 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T17:57:51.018 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T17:57:51.018 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T17:57:51.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:51.019 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:51.019 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:51.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T17:57:51.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T17:57:51.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T17:57:51.019 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:51.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:51.732 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T17:57:51.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:51.792 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2024-01-15T17:57:51.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:51.848 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:51.848 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:51.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:51.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-15T17:57:51.849 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:52.004 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-15T17:57:52.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-15T17:57:52.004 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:52.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:52.300 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:52.300 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:52.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T17:57:52.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T17:57:52.300 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:52.476 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T17:57:52.476 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:57:52.476 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:52.523 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T17:57:52.524 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:57:52.524 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:52.621 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:57:52.719 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:57:52.772 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:57:52.772 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:57:52.772 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:57:52.772 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:57:52.772 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:57:52.773 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:52.774 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:52.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:52.775 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:57:52.824 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T17:57:52.824 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T17:57:52.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:57:52.902 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:57:52.941 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:57:52.942 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:52.943 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:52.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:52.944 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:57:53.025 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T17:57:53.022+0000 7f05fc4d4480 -1 init, newargv = 0x5649f00309b0 newargc=15 2024-01-15T17:57:53.026 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[43330]: starting ceph client 2024-01-15T17:57:53.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[43330]: starting fuse 2024-01-15T17:57:54.132 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:57:54.236 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:57:54.295 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:57:54.296 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:57:54.297 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:57:54.297 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:57:54.297 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:54.297 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:54.297 INFO:teuthology.orchestra.run.smithi002.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:54.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:54.298 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:57:54.353 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T17:57:54.353 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T17:57:54.353 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T17:57:54.353 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T17:57:54.354 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T17:57:54.355 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T17:57:54.541 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.43330.asok 2024-01-15T17:57:54.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T17:57:54.559 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43330.asok 2024-01-15T17:57:54.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:54.560 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43330.asok status 2024-01-15T17:57:54.806 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "43330", 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4629, 2024-01-15T17:57:54.807 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 4629 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3915850578 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:57:54.808 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3915850578 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.4629 192.168.0.1:0/3915850578", 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/3915850578", 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 7, 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 22, 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T17:57:54.809 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T17:57:54.823 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T17:57:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.041+0000 7fcaa8e00640 1 Processor -- start 2024-01-15T17:57:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.041+0000 7fcaa8e00640 1 -- start start 2024-01-15T17:57:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.043+0000 7fcaa8e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcaa415ec60 0x7fcaa415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.043+0000 7fcaa8e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa40716d0 con 0x7fcaa4141880 2024-01-15T17:57:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.043+0000 7fcaa8e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaa4071960 con 0x7fcaa406cad0 2024-01-15T17:57:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.043+0000 7fcaa8e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaa4071f20 con 0x7fcaa415ec60 2024-01-15T17:57:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.043+0000 7fcaa3400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcaa4141880 0x7fcaa4070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50040/0 (socket says 172.21.15.2:50040) 2024-01-15T17:57:55.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.043+0000 7fcaa3400640 1 -- 172.21.15.2:0/4235375892 learned_addr learned my addr 172.21.15.2:0/4235375892 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fca9be00640 1 --2- 172.21.15.2:0/4235375892 >> v2:172.21.15.137:3300/0 conn(0x7fcaa415ec60 0x7fcaa415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1187422189 0 0) 0x7fcaa40716d0 con 0x7fcaa4141880 2024-01-15T17:57:55.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca84003520 con 0x7fcaa4141880 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804457866 0 0) 0x7fca84003520 con 0x7fcaa4141880 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 >> v2:172.21.15.137:3300/0 conn(0x7fcaa415ec60 msgr2=0x7fcaa415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/4235375892 >> v2:172.21.15.137:3300/0 conn(0x7fcaa415ec60 0x7fcaa415f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 >> v1:172.21.15.137:6789/0 conn(0x7fcaa406cad0 legacy=0x7fcaa406cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa4071c40 con 0x7fcaa4141880 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.044+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fca94001670 con 0x7fcaa4141880 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 >> v1:172.21.15.2:6789/0 conn(0x7fcaa4141880 legacy=0x7fcaa4070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca84003970 0x7fca84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fca8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa4071f20 con 0x7fca84005180 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca8400c120 con 0x7fca840047c0 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8400c2e0 con 0x7fca84003970 2024-01-15T17:57:55.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa3e00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fca8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fca9be00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa3400640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca84003970 0x7fca84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa3e00640 1 -- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca84003970 msgr2=0x7fca84003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.045+0000 7fcaa3e00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca84003970 0x7fca84003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.046+0000 7fcaa3e00640 1 -- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca840047c0 msgr2=0x7fca84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.046+0000 7fcaa3e00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca840047c0 0x7fca84004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.046+0000 7fcaa3e00640 1 -- 172.21.15.2:0/4235375892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca84003520 con 0x7fca84005180 2024-01-15T17:57:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.046+0000 7fcaa3e00640 1 -- 172.21.15.2:0/4235375892 reap_dead start 2024-01-15T17:57:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.047+0000 7fcaa3e00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fca8400b980 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fca9c001e50 tx=0x7fca840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93538522d347c7c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.047+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c003b70 con 0x7fca84005180 2024-01-15T17:57:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.047+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9c003d10 con 0x7fca84005180 2024-01-15T17:57:55.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.047+0000 7fcaa2a00640 1 -- 172.21.15.2:0/4235375892 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9c005310 con 0x7fca84005180 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.047+0000 7fcaa8e00640 1 -- 172.21.15.2:0/4235375892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 msgr2=0x7fca8400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.047+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fca8400b980 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fca9c001e50 tx=0x7fca840057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.048+0000 7fcaa8e00640 1 -- 172.21.15.2:0/4235375892 shutdown_connections 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.048+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/4235375892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fca8400b980 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.048+0000 7fcaa8e00640 1 -- 172.21.15.2:0/4235375892 >> 172.21.15.2:0/4235375892 conn(0x7fcaa40b0840 msgr2=0x7fcaa40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.048+0000 7fcaa8e00640 1 -- 172.21.15.2:0/4235375892 shutdown_connections 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.048+0000 7fcaa8e00640 1 -- 172.21.15.2:0/4235375892 wait complete. 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.050+0000 7fcaa8e00640 1 Processor -- start 2024-01-15T17:57:55.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.050+0000 7fcaa8e00640 1 -- start start 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa8e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fcaa41432c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40caf40 0x7fcaa40cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa40acb10 0x7fcaa40acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa8e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa406c6b0 con 0x7fca84005180 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa41507d0 con 0x7fcaa40acb10 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaa40cb900 con 0x7fcaa40caf40 2024-01-15T17:57:55.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fcaa41432c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fcaa41432c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38890/0 (socket says 172.21.15.2:38890) 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa3400640 1 -- 172.21.15.2:0/540893236 learned_addr learned my addr 172.21.15.2:0/540893236 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa3e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa40acb10 0x7fcaa40acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fca9be00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40caf40 0x7fcaa40cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa3400640 1 -- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40caf40 msgr2=0x7fcaa40cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.051+0000 7fcaa3400640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40caf40 0x7fcaa40cb3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fcaa3400640 1 -- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa40acb10 msgr2=0x7fcaa40acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fcaa3400640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa40acb10 0x7fcaa40acfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fcaa3400640 1 -- 172.21.15.2:0/540893236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaa414fa30 con 0x7fca84005180 2024-01-15T17:57:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fcaa3400640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fcaa41432c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fca9400e7b0 tx=0x7fca9400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9400c870 con 0x7fca84005180 2024-01-15T17:57:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9400c9d0 con 0x7fca84005180 2024-01-15T17:57:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9400cd80 con 0x7fca84005180 2024-01-15T17:57:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaa414fd10 con 0x7fca84005180 2024-01-15T17:57:55.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.052+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaa41502d0 con 0x7fca84005180 2024-01-15T17:57:55.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.054+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaa4074770 con 0x7fca84005180 2024-01-15T17:57:55.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.054+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fca9401d460 con 0x7fca84005180 2024-01-15T17:57:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.056+0000 7fca9b400640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b340 0x7fca7c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.056+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca94089bc0 con 0x7fca84005180 2024-01-15T17:57:55.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.056+0000 7fca9be00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b340 0x7fca7c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:55.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.057+0000 7fca9be00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b340 0x7fca7c06d800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40721f0 tx=0x7fca8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:55.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.061+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca94054a10 con 0x7fca84005180 2024-01-15T17:57:55.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.296+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcaa406e1c0 con 0x7fca84005180 2024-01-15T17:57:55.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.296+0000 7fca9b400640 1 -- 172.21.15.2:0/540893236 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fca94058030 con 0x7fca84005180 2024-01-15T17:57:55.300 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T17:57:55.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b340 msgr2=0x7fca7c06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b340 0x7fca7c06d800 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcaa40721f0 tx=0x7fca8c008040 comp rx=0 tx=0).stop 2024-01-15T17:57:55.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 msgr2=0x7fcaa41432c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:55.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fcaa41432c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fca9400e7b0 tx=0x7fca9400ec70 comp rx=0 tx=0).stop 2024-01-15T17:57:55.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 shutdown_connections 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca7c06b340 0x7fca7c06d800 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaa40acb10 0x7fcaa40acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaa40caf40 0x7fcaa40cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 --2- 172.21.15.2:0/540893236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca84005180 0x7fcaa41432c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 >> 172.21.15.2:0/540893236 conn(0x7fcaa40b0840 msgr2=0x7fcaa40cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 shutdown_connections 2024-01-15T17:57:55.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:55.299+0000 7fcaa8e00640 1 -- 172.21.15.2:0/540893236 wait complete. 2024-01-15T17:57:55.321 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T17:57:55.321 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T17:57:55.321 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T17:57:55.321 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-15T17:57:55.321 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-15T17:57:55.321 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T17:57:55.322 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T17:57:55.322 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T17:57:55.322 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-15T17:57:55.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.322 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T17:57:55.371 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:57:55.371 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:57:55.371 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:57:55.371 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:55.371 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42607sec preferred_lft 42607sec 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:55.372 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T17:57:55.373 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:55.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.374 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T17:57:55.510 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T17:57:55.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.515 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T17:57:55.570 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T17:57:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T17:57:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T17:57:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T17:57:55.571 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T17:57:55.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.572 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:55.572 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:55.572 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T17:57:55.572 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T17:57:55.572 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T17:57:55.572 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:55.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.769 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T17:57:55.821 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T17:57:55.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.822 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list-id 2024-01-15T17:57:55.874 INFO:teuthology.orchestra.run.smithi002.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-15T17:57:55.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:55.875 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:55.875 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:55.875 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:57:55.875 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-15T17:57:55.875 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:56.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:56.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T17:57:56.117 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:57:56.117 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:57:56.117 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:57:56.117 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.117 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 7e:43:b0:22:9d:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::7c43:b0ff:fe22:9d32/64 scope link 2024-01-15T17:57:56.118 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:56.122 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-15T17:57:56.216 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:57:56.216 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 7e:43:b0:22:9d:32 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::7c43:b0ff:fe22:9d32/64 scope link 2024-01-15T17:57:56.217 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:56.223 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-15T17:57:56.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-15T17:57:56.223 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-15T17:57:56.224 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:56.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:56.486 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:57:56.486 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:57:56.486 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.1 up 2024-01-15T17:57:56.486 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T17:57:56.486 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:57:56.651 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-15T17:57:56.651 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:57:56.652 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-15T17:57:56.701 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-15T17:57:56.701 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:57:56.701 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-15T17:57:56.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:57:56.892 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:57:56.951 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:57:56.952 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:57:56.953 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:57:56.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:56.954 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:57:57.009 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T17:57:57.009 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-01-15T17:57:57.009 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-15T17:57:57.017 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:57:57.102 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:57:57.145 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:57.146 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:57:57.147 INFO:teuthology.orchestra.run.smithi002.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-15T17:57:57.147 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:57:57.147 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:57:57.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:57.148 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:57:57.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi002.stderr:2024-01-15T17:57:57.209+0000 7f922ae41480 -1 init, newargv = 0x56108d97ec80 newargc=15 2024-01-15T17:57:57.213 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi002.stderr:ceph-fuse[43892]: starting ceph client 2024-01-15T17:57:57.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi002.stderr:ceph-fuse[43892]: starting fuse 2024-01-15T17:57:57.234 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T17:57:57.234 INFO:teuthology.orchestra.run.smithi002.stdout:42 2024-01-15T17:57:57.234 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 42] 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:57.235 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.2")) 2024-01-15T17:57:57.236 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T17:57:57.446 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.2.43892.asok 2024-01-15T17:57:57.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T17:57:57.464 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.43892.asok 2024-01-15T17:57:57.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:57.464 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.43892.asok status 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "2", 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "43892", 2024-01-15T17:57:57.709 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4641, 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 4641 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.2:0", 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2171502093 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:57:57.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.2:0", 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2171502093 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.4641 192.168.0.2:0/2171502093", 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.2:0/2171502093", 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 7, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 22, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T17:57:57.711 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T17:57:57.724 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T17:57:57.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.944+0000 7fd5d7e00640 1 Processor -- start 2024-01-15T17:57:57.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.944+0000 7fd5d7e00640 1 -- start start 2024-01-15T17:57:57.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5d806abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d7e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d80ce000 con 0x7fd5d815d1d0 2024-01-15T17:57:57.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d7e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d806b530 con 0x7fd5d80c9da0 2024-01-15T17:57:57.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80ca6b0 con 0x7fd5d80ca220 2024-01-15T17:57:57.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d6a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd5d815d1d0 0x7fd5d80ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50050/0 (socket says 172.21.15.2:50050) 2024-01-15T17:57:57.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d6a00640 1 -- 172.21.15.2:0/2026213436 learned_addr learned my addr 172.21.15.2:0/2026213436 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:57:57.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.946+0000 7fd5d7400640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5d806abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570662854 0 0) 0x7fd5d80ce000 con 0x7fd5d815d1d0 2024-01-15T17:57:57.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b8003520 con 0x7fd5d815d1d0 2024-01-15T17:57:57.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953351550 0 0) 0x7fd5b8003520 con 0x7fd5d815d1d0 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 msgr2=0x7fd5d806abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5d806abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 >> v1:172.21.15.137:6789/0 conn(0x7fd5d80c9da0 legacy=0x7fd5d806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d80cd3f0 con 0x7fd5d815d1d0 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd5c8001670 con 0x7fd5d815d1d0 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 >> v1:172.21.15.2:6789/0 conn(0x7fd5d815d1d0 legacy=0x7fd5d80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b80039a0 0x7fd5b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80ca6b0 con 0x7fd5b8004d00 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b800ba60 con 0x7fd5b80039a0 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.947+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b800bc20 con 0x7fd5d80ca220 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d7400640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d6000640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d6a00640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b80039a0 0x7fd5b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d7400640 1 -- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 msgr2=0x7fd5b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d7400640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d7400640 1 -- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b80039a0 msgr2=0x7fd5b8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d7400640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5b80039a0 0x7fd5b8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d7400640 1 -- 172.21.15.2:0/2026213436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b8003520 con 0x7fd5b8004d00 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.948+0000 7fd5d6000640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d80ca220 0x7fd5b80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.949+0000 7fd5d7400640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5b8005180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009a90 tx=0x7fd5b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d4e017f236c1b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d002c680 con 0x7fd5b8004d00 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d002c7e0 con 0x7fd5b8004d00 2024-01-15T17:57:57.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d5600640 1 -- 172.21.15.2:0/2026213436 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d002cb70 con 0x7fd5b8004d00 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d7e00640 1 -- 172.21.15.2:0/2026213436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 msgr2=0x7fd5b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/2026213436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5b8005180 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd5d0009a90 tx=0x7fd5b80057f0 comp rx=0 tx=0).stop 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d7400640 1 -- 172.21.15.2:0/2026213436 reap_dead start 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d7e00640 1 -- 172.21.15.2:0/2026213436 shutdown_connections 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d7e00640 1 -- 172.21.15.2:0/2026213436 >> 172.21.15.2:0/2026213436 conn(0x7fd5d80cbfc0 msgr2=0x7fd5d80cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.950+0000 7fd5d7e00640 1 -- 172.21.15.2:0/2026213436 shutdown_connections 2024-01-15T17:57:57.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.951+0000 7fd5d7e00640 1 -- 172.21.15.2:0/2026213436 wait complete. 2024-01-15T17:57:57.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.952+0000 7fd5d7e00640 1 Processor -- start 2024-01-15T17:57:57.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.952+0000 7fd5d7e00640 1 -- start start 2024-01-15T17:57:57.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.953+0000 7fd5d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5d814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.953+0000 7fd5d7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d8160db0 0x7fd5d8161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.953+0000 7fd5d7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5d81f3180 0x7fd5d81f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.953+0000 7fd5d7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80b2200 con 0x7fd5b8004d00 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.953+0000 7fd5d7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d8161770 con 0x7fd5d81f3180 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.953+0000 7fd5d7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d80c5280 con 0x7fd5d8160db0 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5d814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5d814f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38906/0 (socket says 172.21.15.2:38906) 2024-01-15T17:57:57.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 -- 172.21.15.2:0/1885210120 learned_addr learned my addr 172.21.15.2:0/1885210120 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:57:57.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d7400640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5d81f3180 0x7fd5d81f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 -- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d8160db0 msgr2=0x7fd5d8161230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6000640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d8160db0 0x7fd5d8161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d8160db0 0x7fd5d8161230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 -- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5d81f3180 msgr2=0x7fd5d81f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5d81f3180 0x7fd5d81f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6a00640 1 -- 172.21.15.2:0/1885210120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d81f63d0 con 0x7fd5b8004d00 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d7400640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5d81f3180 0x7fd5d81f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.954+0000 7fd5d6000640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d8160db0 0x7fd5d8161230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.955+0000 7fd5d6a00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5d814f360 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800e780 tx=0x7fd5c800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.955+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800ee30 con 0x7fd5b8004d00 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.955+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5c800c8d0 con 0x7fd5b8004d00 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.955+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c800cc80 con 0x7fd5b8004d00 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.955+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d814f8a0 con 0x7fd5b8004d00 2024-01-15T17:57:57.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.955+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d81f6900 con 0x7fd5b8004d00 2024-01-15T17:57:57.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.956+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d8074770 con 0x7fd5b8004d00 2024-01-15T17:57:57.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.957+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd5c801d460 con 0x7fd5b8004d00 2024-01-15T17:57:57.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.959+0000 7fd5cf400640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd5b006b2f0 0x7fd5b006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:57:57.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.959+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5c8089c00 con 0x7fd5b8004d00 2024-01-15T17:57:57.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.959+0000 7fd5d6000640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd5b006b2f0 0x7fd5b006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:57:57.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.960+0000 7fd5d6000640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd5b006b2f0 0x7fd5b006d7b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0007fd0 tx=0x7fd5c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:57:57.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:57.964+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5c8058460 con 0x7fd5b8004d00 2024-01-15T17:57:58.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.203+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd5d80735f0 con 0x7fd5b8004d00 2024-01-15T17:57:58.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.204+0000 7fd5cf400640 1 -- 172.21.15.2:0/1885210120 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd5c8057ff0 con 0x7fd5b8004d00 2024-01-15T17:57:58.208 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T17:57:58.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd5b006b2f0 msgr2=0x7fd5b006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:58.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd5b006b2f0 0x7fd5b006d7b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0007fd0 tx=0x7fd5c0009040 comp rx=0 tx=0).stop 2024-01-15T17:57:58.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 msgr2=0x7fd5d814f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:57:58.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5d814f360 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800e780 tx=0x7fd5c800ec40 comp rx=0 tx=0).stop 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 shutdown_connections 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd5b006b2f0 0x7fd5b006d7b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd5d81f3180 0x7fd5d81f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd5d8160db0 0x7fd5d8161230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 --2- 172.21.15.2:0/1885210120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd5b8004d00 0x7fd5d814f360 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.207+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 >> 172.21.15.2:0/1885210120 conn(0x7fd5d80cbfc0 msgr2=0x7fd5d806b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.208+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 shutdown_connections 2024-01-15T17:57:58.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:57:58.208+0000 7fd5d7e00640 1 -- 172.21.15.2:0/1885210120 wait complete. 2024-01-15T17:57:58.231 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T17:57:58.231 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T17:57:58.231 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T17:57:58.231 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T17:57:58.232 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T17:57:58.232 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T17:57:58.232 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T17:57:58.232 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T17:57:58.232 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T17:57:58.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:58.232 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T17:57:58.287 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:57:58.287 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:57:58.287 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:57:58.287 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:58.287 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42592sec preferred_lft 42592sec 2024-01-15T17:57:58.288 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T17:57:58.289 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:57:58.289 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-15T17:57:58.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add name ceph-brx type bridge 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr flush dev ceph-brx 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set ceph-brx up 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-15T17:57:58.290 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:57:58.598 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:57:58.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:57:58.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:58.715 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T17:57:58.828 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T17:57:58.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:58.832 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T17:57:58.884 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T17:57:58.884 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T17:57:58.884 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T17:57:58.884 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T17:57:58.884 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T17:57:58.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:58.884 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:57:58.885 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:57:58.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T17:57:58.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T17:57:58.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T17:57:58.885 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:57:59.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:59.608 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T17:57:59.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:59.665 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list-id 2024-01-15T17:57:59.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:59.717 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:57:59.717 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:57:59.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:57:59.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-15T17:57:59.717 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:57:59.863 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-15T17:57:59.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-15T17:57:59.864 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:58:00.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:00.133 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:58:00.133 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:58:00.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T17:58:00.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set dev brx.0 master ceph-brx 2024-01-15T17:58:00.134 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:58:00.291 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T17:58:00.291 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:58:00.292 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:00.336 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T17:58:00.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:58:00.336 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:00.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:58:00.523 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:58:00.571 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:58:00.571 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:58:00.571 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:58:00.572 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:00.573 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:00.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:00.574 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:58:00.618 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T17:58:00.619 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T17:58:00.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:58:00.744 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:58:00.785 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:58:00.786 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:00.787 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:00.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:00.788 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:58:00.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T17:58:00.861+0000 7f3d72443480 -1 init, newargv = 0x55e2cd7a1c90 newargc=15 2024-01-15T17:58:00.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[41408]: starting ceph client 2024-01-15T17:58:00.876 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[41408]: starting fuse 2024-01-15T17:58:01.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:58:02.079 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:58:02.133 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:58:02.134 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:02.135 INFO:teuthology.orchestra.run.smithi137.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:02.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:02.136 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:58:02.185 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T17:58:02.185 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:02.185 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T17:58:02.186 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T17:58:02.187 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:02.187 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T17:58:02.187 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T17:58:02.365 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.41408.asok 2024-01-15T17:58:02.369 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:58:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T17:58:02.383 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41408.asok 2024-01-15T17:58:02.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:02.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41408.asok status 2024-01-15T17:58:02.619 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T17:58:02.619 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "41408", 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4609, 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T17:58:02.620 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4609 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3928418775 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3928418775 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:02.621 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4609 192.168.0.1:0/3928418775", 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/3928418775", 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 7, 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 22, 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T17:58:02.622 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T17:58:02.635 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T17:58:02.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.861+0000 7faea2200640 1 Processor -- start 2024-01-15T17:58:02.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.861+0000 7faea2200640 1 -- start start 2024-01-15T17:58:02.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7faea2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c15c0e0 0x7fae9c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:02.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7faea2200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fae9c06cb80 0x7fae9c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:02.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7faea2200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c15cb00 con 0x7fae9c15b7d0 2024-01-15T17:58:02.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7faea2200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c0ccec0 con 0x7fae9c06cb80 2024-01-15T17:58:02.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7faea2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0cd1a0 con 0x7fae9c15c0e0 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7fae9aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fae9c15b7d0 0x7fae9c06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50052/0 (socket says 172.21.15.2:50052) 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7fae9be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fae9c06cb80 0x7fae9c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.863+0000 7fae9aa00640 1 -- 172.21.15.2:0/1188382070 learned_addr learned my addr 172.21.15.2:0/1188382070 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9b400640 1 --2- 172.21.15.2:0/1188382070 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c15c0e0 0x7fae9c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9a000640 1 -- 172.21.15.2:0/1188382070 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626216533 0 0) 0x7fae9c15cb00 con 0x7fae9c15b7d0 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9a000640 1 -- 172.21.15.2:0/1188382070 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7c003520 con 0x7fae9c15b7d0 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9be00640 1 -- 172.21.15.2:0/1188382070 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c15c0e0 msgr2=0x7fae9c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:02.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9be00640 1 --2- 172.21.15.2:0/1188382070 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c15c0e0 0x7fae9c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:02.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9be00640 1 -- 172.21.15.2:0/1188382070 >> v1:172.21.15.2:6789/0 conn(0x7fae9c15b7d0 legacy=0x7fae9c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:02.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.864+0000 7fae9be00640 1 -- 172.21.15.2:0/1188382070 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0cd480 con 0x7fae9c06cb80 2024-01-15T17:58:02.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.865+0000 7fae9be00640 1 --2- 172.21.15.2:0/1188382070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c06cb80 0x7fae9c15cf90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae94009d90 tx=0x7fae9402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5718921cbfe07352 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:02.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7fae9a000640 1 -- 172.21.15.2:0/1188382070 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9402cc50 con 0x7fae9c06cb80 2024-01-15T17:58:02.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7fae9a000640 1 -- 172.21.15.2:0/1188382070 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9402cdb0 con 0x7fae9c06cb80 2024-01-15T17:58:02.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7fae9a000640 1 -- 172.21.15.2:0/1188382070 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94034660 con 0x7fae9c06cb80 2024-01-15T17:58:02.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7faea2200640 1 -- 172.21.15.2:0/1188382070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c06cb80 msgr2=0x7fae9c15cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:02.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7faea2200640 1 --2- 172.21.15.2:0/1188382070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c06cb80 0x7fae9c15cf90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fae94009d90 tx=0x7fae9402bea0 comp rx=0 tx=0).stop 2024-01-15T17:58:02.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7faea2200640 1 -- 172.21.15.2:0/1188382070 shutdown_connections 2024-01-15T17:58:02.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7faea2200640 1 --2- 172.21.15.2:0/1188382070 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c06cb80 0x7fae9c15cf90 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:02.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7faea2200640 1 --2- 172.21.15.2:0/1188382070 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c15c0e0 0x7fae9c0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:02.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.866+0000 7faea2200640 1 -- 172.21.15.2:0/1188382070 >> 172.21.15.2:0/1188382070 conn(0x7fae9c06da20 msgr2=0x7fae9c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:02.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.867+0000 7faea2200640 1 -- 172.21.15.2:0/1188382070 shutdown_connections 2024-01-15T17:58:02.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.867+0000 7faea2200640 1 -- 172.21.15.2:0/1188382070 wait complete. 2024-01-15T17:58:02.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.868+0000 7faea2200640 1 Processor -- start 2024-01-15T17:58:02.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 -- start start 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c06cb80 0x7fae9c145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9c15b7d0 0x7fae9c0a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 0x7fae9c0aa350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0a92e0 con 0x7fae9c15c0e0 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0a95c0 con 0x7fae9c15b7d0 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.869+0000 7faea2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0a98a0 con 0x7fae9c06cb80 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 0x7fae9c0aa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:02.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 0x7fae9c0aa350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38916/0 (socket says 172.21.15.2:38916) 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 -- 172.21.15.2:0/199077385 learned_addr learned my addr 172.21.15.2:0/199077385 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9b400640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c06cb80 0x7fae9c145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9aa00640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9c15b7d0 0x7fae9c0a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 -- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c06cb80 msgr2=0x7fae9c145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c06cb80 0x7fae9c145b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 -- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9c15b7d0 msgr2=0x7fae9c0a9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9c15b7d0 0x7fae9c0a9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:02.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.870+0000 7fae9be00640 1 -- 172.21.15.2:0/199077385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94009a70 con 0x7fae9c15c0e0 2024-01-15T17:58:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.871+0000 7fae9be00640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 0x7fae9c0aa350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae94009d90 tx=0x7fae94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.871+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94040650 con 0x7fae9c15c0e0 2024-01-15T17:58:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.871+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae940407b0 con 0x7fae9c15c0e0 2024-01-15T17:58:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.871+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c152900 con 0x7fae9c15c0e0 2024-01-15T17:58:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.871+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94040b60 con 0x7fae9c15c0e0 2024-01-15T17:58:02.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.871+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c0a8760 con 0x7fae9c15c0e0 2024-01-15T17:58:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.872+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c152760 con 0x7fae9c15c0e0 2024-01-15T17:58:02.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.873+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fae94040cc0 con 0x7fae9c15c0e0 2024-01-15T17:58:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.874+0000 7fae93400640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7406b2f0 0x7fae7406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.874+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae940ac670 con 0x7fae9c15c0e0 2024-01-15T17:58:02.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.875+0000 7fae9b400640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7406b2f0 0x7fae7406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.876+0000 7fae9b400640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7406b2f0 0x7fae7406d7b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009a30 tx=0x7fae8c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:02.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:02.879+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae9c152760 con 0x7fae9c15c0e0 2024-01-15T17:58:03.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.111+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fae9c152760 con 0x7fae9c15c0e0 2024-01-15T17:58:03.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.112+0000 7fae93400640 1 -- 172.21.15.2:0/199077385 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fae9c152760 con 0x7fae9c15c0e0 2024-01-15T17:58:03.115 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T17:58:03.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7406b2f0 msgr2=0x7fae7406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:03.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7406b2f0 0x7fae7406d7b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fae8c009a30 tx=0x7fae8c008040 comp rx=0 tx=0).stop 2024-01-15T17:58:03.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 msgr2=0x7fae9c0aa350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:03.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 0x7fae9c0aa350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae94009d90 tx=0x7fae94004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:03.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 shutdown_connections 2024-01-15T17:58:03.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7406b2f0 0x7fae7406d7b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9c15c0e0 0x7fae9c0aa350 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9c15b7d0 0x7fae9c0a9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 --2- 172.21.15.2:0/199077385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9c06cb80 0x7fae9c145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 >> 172.21.15.2:0/199077385 conn(0x7fae9c06da20 msgr2=0x7fae9c15eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 shutdown_connections 2024-01-15T17:58:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:03.115+0000 7faea2200640 1 -- 172.21.15.2:0/199077385 wait complete. 2024-01-15T17:58:03.138 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T17:58:03.138 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T17:58:03.138 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T17:58:03.139 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-15T17:58:03.139 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:03.139 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T17:58:03.139 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T17:58:03.139 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T17:58:03.139 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-15T17:58:03.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.139 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T17:58:03.195 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42587sec preferred_lft 42587sec 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T17:58:03.196 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.197 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T17:58:03.197 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:03.197 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T17:58:03.197 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.198 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T17:58:03.285 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T17:58:03.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.289 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T17:58:03.340 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T17:58:03.341 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T17:58:03.341 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T17:58:03.341 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T17:58:03.341 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T17:58:03.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.342 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:58:03.342 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:58:03.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T17:58:03.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T17:58:03.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T17:58:03.342 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:58:03.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.517 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T17:58:03.563 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T17:58:03.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.564 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list-id 2024-01-15T17:58:03.657 INFO:teuthology.orchestra.run.smithi137.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-15T17:58:03.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.658 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:58:03.658 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:58:03.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:58:03.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-15T17:58:03.659 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:58:03.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T17:58:03.884 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 82:da:a7:87:a7:df brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::80da:a7ff:fe87:a7df/64 scope link 2024-01-15T17:58:03.885 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-15T17:58:03.979 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 82:da:a7:87:a7:df brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::80da:a7ff:fe87:a7df/64 scope link 2024-01-15T17:58:03.980 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:58:03.983 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-15T17:58:03.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:03.984 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:58:03.984 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:58:03.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-15T17:58:03.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-15T17:58:03.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-15T17:58:03.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-15T17:58:03.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-15T17:58:03.987 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:58:04.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:04.246 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:58:04.246 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:58:04.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.1 up 2024-01-15T17:58:04.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set dev brx.1 master ceph-brx 2024-01-15T17:58:04.246 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:58:04.414 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-15T17:58:04.414 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:58:04.414 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:04.461 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-15T17:58:04.462 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:58:04.462 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:04.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:58:04.643 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:58:04.695 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:58:04.696 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:58:04.697 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:58:04.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:04.698 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:58:04.746 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T17:58:04.746 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [40] 2024-01-15T17:58:04.746 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-15T17:58:04.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:58:04.824 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:58:04.864 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:58:04.864 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:58:04.864 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:58:04.865 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:58:04.866 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:58:04.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:04.867 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:58:04.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi137.stderr:2024-01-15T17:58:04.928+0000 7fb5f960b480 -1 init, newargv = 0x555bf590bc20 newargc=15 2024-01-15T17:58:04.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi137.stderr:ceph-fuse[41928]: starting ceph client 2024-01-15T17:58:04.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi137.stderr:ceph-fuse[41928]: starting fuse 2024-01-15T17:58:04.952 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T17:58:04.952 INFO:teuthology.orchestra.run.smithi137.stdout:42 2024-01-15T17:58:04.952 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40, 42] 2024-01-15T17:58:04.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T17:58:04.953 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.3")) 2024-01-15T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T17:58:05.147 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.3.41928.asok 2024-01-15T17:58:05.152 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:58:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T17:58:05.165 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.41928.asok 2024-01-15T17:58:05.165 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.41928.asok status 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "3", 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "41928", 2024-01-15T17:58:05.400 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4490, 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4490 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.2:0", 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 909096354 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T17:58:05.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.2:0", 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 909096354 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4490 192.168.0.2:0/909096354", 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.2:0/909096354", 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 7, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 22, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T17:58:05.402 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T17:58:05.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.415 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T17:58:05.415 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T17:58:05.476 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T17:58:05.476 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T17:58:05.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T17:58:05.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.581 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T17:58:05.581 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-15T17:58:05.641 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T17:58:05.641 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-15T17:58:05.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-15T17:58:05.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.749 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T17:58:05.749 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:05.807 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T17:58:05.807 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:05.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:05.895 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.895 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T17:58:05.895 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:05.949 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T17:58:05.949 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:05.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:05.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:06.045 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-15T17:58:06.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:06.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.277+0000 7fd6a9600640 1 Processor -- start 2024-01-15T17:58:06.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.278+0000 7fd6a9600640 1 -- start start 2024-01-15T17:58:06.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.279+0000 7fd6a9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd6a40d1d90 0x7fd6a40d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.279+0000 7fd6a9600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40d3a60 con 0x7fd6a415d340 2024-01-15T17:58:06.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.279+0000 7fd6a9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40ce430 con 0x7fd6a40d3090 2024-01-15T17:58:06.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.279+0000 7fd6a9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a40b3de0 con 0x7fd6a40d1d90 2024-01-15T17:58:06.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.279+0000 7fd6a3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd6a415d340 0x7fd6a40cdd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48936/0 (socket says 172.21.15.2:48936) 2024-01-15T17:58:06.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.279+0000 7fd6a3e00640 1 -- 172.21.15.2:0/1005656393 learned_addr learned my addr 172.21.15.2:0/1005656393 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a8c00640 1 --2- 172.21.15.2:0/1005656393 >> v2:172.21.15.137:3300/0 conn(0x7fd6a40d1d90 0x7fd6a40d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736062961 0 0) 0x7fd6a40d3a60 con 0x7fd6a415d340 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd684003520 con 0x7fd6a415d340 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 284169690 0 0) 0x7fd684003520 con 0x7fd6a415d340 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 >> v2:172.21.15.137:3300/0 conn(0x7fd6a40d1d90 msgr2=0x7fd6a40d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 --2- 172.21.15.2:0/1005656393 >> v2:172.21.15.137:3300/0 conn(0x7fd6a40d1d90 0x7fd6a40d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 >> v1:172.21.15.137:6789/0 conn(0x7fd6a40d3090 legacy=0x7fd6a40d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a40b40c0 con 0x7fd6a415d340 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd6940019f0 con 0x7fd6a415d340 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 >> v1:172.21.15.2:6789/0 conn(0x7fd6a415d340 legacy=0x7fd6a40cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd684003970 0x7fd684003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6840047c0 0x7fd684004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd68400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40b3de0 con 0x7fd684005180 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68400c020 con 0x7fd6840047c0 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.280+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68400c1e0 con 0x7fd684003970 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a8c00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd68400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a3e00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd684003970 0x7fd684003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a3400640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6840047c0 0x7fd684004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a8c00640 1 -- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd684003970 msgr2=0x7fd684003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a8c00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd684003970 0x7fd684003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a8c00640 1 -- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6840047c0 msgr2=0x7fd684004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:06.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a8c00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6840047c0 0x7fd684004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.281+0000 7fd6a8c00640 1 -- 172.21.15.2:0/1005656393 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684003520 con 0x7fd684005180 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.282+0000 7fd6a8c00640 1 -- 172.21.15.2:0/1005656393 reap_dead start 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.282+0000 7fd6a8c00640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd68400b980 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009ac0 tx=0x7fd6840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434780ef89a71f57 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c004020 con 0x7fd684005180 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c004180 con 0x7fd684005180 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a2a00640 1 -- 172.21.15.2:0/1005656393 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c02c710 con 0x7fd684005180 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a9600640 1 -- 172.21.15.2:0/1005656393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 msgr2=0x7fd68400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a9600640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd68400b980 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009ac0 tx=0x7fd6840057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a9600640 1 -- 172.21.15.2:0/1005656393 shutdown_connections 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a9600640 1 --2- 172.21.15.2:0/1005656393 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd68400b980 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.283+0000 7fd6a9600640 1 -- 172.21.15.2:0/1005656393 >> 172.21.15.2:0/1005656393 conn(0x7fd6a408caf0 msgr2=0x7fd6a409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.284+0000 7fd6a9600640 1 -- 172.21.15.2:0/1005656393 shutdown_connections 2024-01-15T17:58:06.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.284+0000 7fd6a9600640 1 -- 172.21.15.2:0/1005656393 wait complete. 2024-01-15T17:58:06.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.285+0000 7fd6a9600640 1 Processor -- start 2024-01-15T17:58:06.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.285+0000 7fd6a9600640 1 -- start start 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.286+0000 7fd6a9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd6a414d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.286+0000 7fd6a9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6a40b7170 0x7fd6a40b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.286+0000 7fd6a9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6a40ce5c0 0x7fd6a40cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.286+0000 7fd6a9600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40cef80 con 0x7fd684005180 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.286+0000 7fd6a9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40b8840 con 0x7fd6a40ce5c0 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.286+0000 7fd6a9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40b7c30 con 0x7fd6a40b7170 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd6a414d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd6a414d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44518/0 (socket says 172.21.15.2:44518) 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 -- 172.21.15.2:0/3400420087 learned_addr learned my addr 172.21.15.2:0/3400420087 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:06.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3400640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6a40b7170 0x7fd6a40b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a8c00640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6a40ce5c0 0x7fd6a40cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 -- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6a40b7170 msgr2=0x7fd6a40b75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6a40b7170 0x7fd6a40b75d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 -- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6a40ce5c0 msgr2=0x7fd6a40cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6a40ce5c0 0x7fd6a40cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.287+0000 7fd6a3e00640 1 -- 172.21.15.2:0/3400420087 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a40b7f10 con 0x7fd684005180 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.288+0000 7fd6a3e00640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd6a414d440 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd694001580 tx=0x7fd69400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.288+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694004d60 con 0x7fd684005180 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.288+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69400be80 con 0x7fd684005180 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.288+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6940072d0 con 0x7fd684005180 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.288+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a40b81f0 con 0x7fd684005180 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.288+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a414d980 con 0x7fd684005180 2024-01-15T17:58:06.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.289+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a4074770 con 0x7fd684005180 2024-01-15T17:58:06.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.290+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd694007430 con 0x7fd684005180 2024-01-15T17:58:06.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.291+0000 7fd69be00640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67c06b2c0 0x7fd67c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.291+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd694089cb0 con 0x7fd684005180 2024-01-15T17:58:06.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.292+0000 7fd6a3400640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67c06b2c0 0x7fd67c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.293+0000 7fd6a3400640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67c06b2c0 0x7fd67c06d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4157b40 tx=0x7fd68c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:06.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.296+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6940549f0 con 0x7fd684005180 2024-01-15T17:58:06.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.492+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd6a406cf30 con 0x7fd684005180 2024-01-15T17:58:06.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.494+0000 7fd69be00640 1 -- 172.21.15.2:0/3400420087 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fd694019070 con 0x7fd684005180 2024-01-15T17:58:06.498 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:06.498 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:06.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67c06b2c0 msgr2=0x7fd67c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67c06b2c0 0x7fd67c06d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4157b40 tx=0x7fd68c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:06.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 msgr2=0x7fd6a414d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd6a414d440 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd694001580 tx=0x7fd69400b4a0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 shutdown_connections 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67c06b2c0 0x7fd67c06d780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6a40ce5c0 0x7fd6a40cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6a40b7170 0x7fd6a40b75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 --2- 172.21.15.2:0/3400420087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd684005180 0x7fd6a414d440 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.497+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 >> 172.21.15.2:0/3400420087 conn(0x7fd6a408caf0 msgr2=0x7fd6a415c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.498+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 shutdown_connections 2024-01-15T17:58:06.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.498+0000 7fd6a9600640 1 -- 172.21.15.2:0/3400420087 wait complete. 2024-01-15T17:58:06.519 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:06.519 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:06.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.715+0000 7f3597400640 1 Processor -- start 2024-01-15T17:58:06.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.716+0000 7f3597400640 1 -- start start 2024-01-15T17:58:06.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.717+0000 7f3597400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35900b4c70 0x7f35900d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.717+0000 7f3597400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f359015d730 0x7f35900b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.717+0000 7f3597400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35900b5590 con 0x7f35900b8c30 2024-01-15T17:58:06.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.717+0000 7f3597400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35900d30b0 con 0x7f359015d730 2024-01-15T17:58:06.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.717+0000 7f3597400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35900d1900 con 0x7f35900b4c70 2024-01-15T17:58:06.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3587c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f359015d730 0x7f35900b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3595400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f35900b8c30 0x7f35900d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48940/0 (socket says 172.21.15.2:48940) 2024-01-15T17:58:06.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3595400640 1 -- 172.21.15.2:0/2359858155 learned_addr learned my addr 172.21.15.2:0/2359858155 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:06.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3594a00640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35900b4c70 0x7f35900d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3587c00640 1 -- 172.21.15.2:0/2359858155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35900b4c70 msgr2=0x7f35900d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:06.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3587c00640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35900b4c70 0x7f35900d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3587c00640 1 -- 172.21.15.2:0/2359858155 >> v1:172.21.15.2:6789/0 conn(0x7f35900b8c30 legacy=0x7f35900d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f358fe00640 1 -- 172.21.15.2:0/2359858155 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155688636 0 0) 0x7f35900b5590 con 0x7f35900b8c30 2024-01-15T17:58:06.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3587c00640 1 -- 172.21.15.2:0/2359858155 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35900d1be0 con 0x7f359015d730 2024-01-15T17:58:06.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.718+0000 7f3594a00640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35900b4c70 0x7f35900d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:06.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.720+0000 7f3587c00640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359015d730 0x7f35900b5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f35780097a0 tx=0x7f357802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d20391e4897bc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:06.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.721+0000 7f358fe00640 1 -- 172.21.15.2:0/2359858155 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578031830 con 0x7f359015d730 2024-01-15T17:58:06.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.721+0000 7f358fe00640 1 -- 172.21.15.2:0/2359858155 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3578031990 con 0x7f359015d730 2024-01-15T17:58:06.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.721+0000 7f358fe00640 1 -- 172.21.15.2:0/2359858155 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578031d20 con 0x7f359015d730 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 -- 172.21.15.2:0/2359858155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359015d730 msgr2=0x7f35900b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359015d730 0x7f35900b5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f35780097a0 tx=0x7f357802b8d0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 -- 172.21.15.2:0/2359858155 shutdown_connections 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f359015d730 0x7f35900b5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 --2- 172.21.15.2:0/2359858155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35900b4c70 0x7f35900d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 -- 172.21.15.2:0/2359858155 >> 172.21.15.2:0/2359858155 conn(0x7f3590072210 msgr2=0x7f35900cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 -- 172.21.15.2:0/2359858155 shutdown_connections 2024-01-15T17:58:06.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.722+0000 7f3597400640 1 -- 172.21.15.2:0/2359858155 wait complete. 2024-01-15T17:58:06.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.724+0000 7f3597400640 1 Processor -- start 2024-01-15T17:58:06.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.724+0000 7f3597400640 1 -- start start 2024-01-15T17:58:06.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3597400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 0x7f359014db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3597400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35900b8c30 0x7f359015f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3597400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f359015d730 0x7f35900c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3597400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900c8db0 con 0x7f35900b4c70 2024-01-15T17:58:06.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3597400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35900c9090 con 0x7f35900b8c30 2024-01-15T17:58:06.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3597400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35900c9370 con 0x7f359015d730 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3594a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 0x7f359014db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3594a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 0x7f359014db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44540/0 (socket says 172.21.15.2:44540) 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.725+0000 7f3594a00640 1 -- 172.21.15.2:0/593855893 learned_addr learned my addr 172.21.15.2:0/593855893 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3595400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f359015d730 0x7f35900c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3587c00640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35900b8c30 0x7f359015f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3594a00640 1 -- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f359015d730 msgr2=0x7f35900c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3594a00640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f359015d730 0x7f35900c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3594a00640 1 -- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35900b8c30 msgr2=0x7f359015f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3594a00640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35900b8c30 0x7f359015f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3594a00640 1 -- 172.21.15.2:0/593855893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3578009400 con 0x7f35900b4c70 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3595400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f359015d730 0x7f35900c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3594a00640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 0x7f359014db10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f358000e7e0 tx=0x7f358000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580004030 con 0x7f35900b4c70 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3580004190 con 0x7f35900b4c70 2024-01-15T17:58:06.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3580004520 con 0x7f35900b4c70 2024-01-15T17:58:06.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.726+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35900c9650 con 0x7f35900b4c70 2024-01-15T17:58:06.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.727+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35900c8a90 con 0x7f35900b4c70 2024-01-15T17:58:06.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.728+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35900686f0 con 0x7f35900b4c70 2024-01-15T17:58:06.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.729+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f358000c6d0 con 0x7f35900b4c70 2024-01-15T17:58:06.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.730+0000 7f358d600640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f356806b2c0 0x7f356806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:06.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.730+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35800898d0 con 0x7f35900b4c70 2024-01-15T17:58:06.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.730+0000 7f3587c00640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f356806b2c0 0x7f356806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:06.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.732+0000 7f3587c00640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f356806b2c0 0x7f356806d780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3578036000 tx=0x7f357803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:06.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.735+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3580054730 con 0x7f35900b4c70 2024-01-15T17:58:06.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.933+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f35900c8ef0 con 0x7f35900b4c70 2024-01-15T17:58:06.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.934+0000 7f358d600640 1 -- 172.21.15.2:0/593855893 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f35900c8ef0 con 0x7f35900b4c70 2024-01-15T17:58:06.937 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:06.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.936+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f356806b2c0 msgr2=0x7f356806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.936+0000 7f3597400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f356806b2c0 0x7f356806d780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3578036000 tx=0x7f357803c040 comp rx=0 tx=0).stop 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.936+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 msgr2=0x7f359014db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.936+0000 7f3597400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 0x7f359014db10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f358000e7e0 tx=0x7f358000eca0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 shutdown_connections 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f356806b2c0 0x7f356806d780 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f359015d730 0x7f35900c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35900b8c30 0x7f359015f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 --2- 172.21.15.2:0/593855893 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35900b4c70 0x7f359014db10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 >> 172.21.15.2:0/593855893 conn(0x7f3590072210 msgr2=0x7f35900b62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 shutdown_connections 2024-01-15T17:58:06.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:06.937+0000 7f3597400640 1 -- 172.21.15.2:0/593855893 wait complete. 2024-01-15T17:58:06.965 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:07.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.173+0000 7f1faa400640 1 Processor -- start 2024-01-15T17:58:07.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.173+0000 7f1faa400640 1 -- start start 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1faa400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa40b2210 0x7f1fa40d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1faa400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1fa40d1c50 0x7f1fa40b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1faa400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fa40d1250 con 0x7f1fa40d1870 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1faa400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fa40ce6d0 con 0x7f1fa40d1c50 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1faa400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40ce8e0 con 0x7f1fa40b2210 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1fa9000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1fa40d1870 0x7f1fa415d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48954/0 (socket says 172.21.15.2:48954) 2024-01-15T17:58:07.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1fa9000640 1 -- 172.21.15.2:0/1321176848 learned_addr learned my addr 172.21.15.2:0/1321176848 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:07.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.175+0000 7f1fa9a00640 1 --2- 172.21.15.2:0/1321176848 >> v2:172.21.15.137:3300/0 conn(0x7f1fa40d1c50 0x7f1fa40b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033025197 0 0) 0x7f1fa40d1250 con 0x7f1fa40d1870 2024-01-15T17:58:07.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1f9bc00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa40b2210 0x7f1fa40d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f84003520 con 0x7f1fa40d1870 2024-01-15T17:58:07.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1307235931 0 0) 0x7f1f84003520 con 0x7f1fa40d1870 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa40b2210 msgr2=0x7f1fa40d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa40b2210 0x7f1fa40d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 >> v2:172.21.15.137:3300/0 conn(0x7f1fa40d1c50 msgr2=0x7f1fa40b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 --2- 172.21.15.2:0/1321176848 >> v2:172.21.15.137:3300/0 conn(0x7f1fa40d1c50 0x7f1fa40b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa40cebc0 con 0x7f1fa40d1870 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1f94001670 con 0x7f1fa40d1870 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 >> v1:172.21.15.2:6789/0 conn(0x7f1fa40d1870 legacy=0x7f1fa415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f84003970 0x7f1f84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f840047c0 0x7f1f84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f84005180 0x7f1f8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40ce6d0 con 0x7f1f84003970 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa40ce8e0 con 0x7f1f840047c0 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8400bef0 con 0x7f1f84005180 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1f9bc00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa40b2210 0x7f1fa40d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:07.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.176+0000 7f1fa9000640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f84003970 0x7f1f84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9a00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f84005180 0x7f1f8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9000640 1 -- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f84005180 msgr2=0x7f1f8400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9000640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f84005180 0x7f1f8400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9000640 1 -- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f840047c0 msgr2=0x7f1f84004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9000640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f840047c0 0x7f1f84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9000640 1 -- 172.21.15.2:0/1321176848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84003520 con 0x7f1f84003970 2024-01-15T17:58:07.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.177+0000 7f1fa9a00640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f84005180 0x7f1f8400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.178+0000 7f1fa9000640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f84003970 0x7f1f84003d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400c210 tx=0x7f1f840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9b4d0bc35189e43 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.178+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f94005ad0 con 0x7f1f84003970 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.178+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f94005c30 con 0x7f1f84003970 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.178+0000 7f1fa3e00640 1 -- 172.21.15.2:0/1321176848 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9402c8d0 con 0x7f1f84003970 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1faa400640 1 -- 172.21.15.2:0/1321176848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f84003970 msgr2=0x7f1f84003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1faa400640 1 --2- 172.21.15.2:0/1321176848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f84003970 0x7f1f84003d90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1f9400c210 tx=0x7f1f840057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1fa9a00640 1 -- 172.21.15.2:0/1321176848 reap_dead start 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1faa400640 1 -- 172.21.15.2:0/1321176848 shutdown_connections 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1faa400640 1 -- 172.21.15.2:0/1321176848 >> 172.21.15.2:0/1321176848 conn(0x7f1fa40d30b0 msgr2=0x7f1fa40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:07.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1faa400640 1 -- 172.21.15.2:0/1321176848 shutdown_connections 2024-01-15T17:58:07.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.179+0000 7f1faa400640 1 -- 172.21.15.2:0/1321176848 wait complete. 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.181+0000 7f1faa400640 1 Processor -- start 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.181+0000 7f1faa400640 1 -- start start 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1faa400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f84003970 0x7f1fa40b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1faa400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa415b2c0 0x7f1fa40cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1faa400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 0x7f1fa40cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1faa400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa41f6380 con 0x7f1fa415b740 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1faa400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa41f3270 con 0x7f1f84003970 2024-01-15T17:58:07.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1faa400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fa41f3550 con 0x7f1fa415b2c0 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 0x7f1fa40cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 0x7f1fa40cc3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44558/0 (socket says 172.21.15.2:44558) 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 -- 172.21.15.2:0/808190299 learned_addr learned my addr 172.21.15.2:0/808190299 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1f9bc00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa415b2c0 0x7f1fa40cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9000640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f84003970 0x7f1fa40b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 -- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa415b2c0 msgr2=0x7f1fa40cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa415b2c0 0x7f1fa40cbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 -- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f84003970 msgr2=0x7f1fa40b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f84003970 0x7f1fa40b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1fa9a00640 1 -- 172.21.15.2:0/808190299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa41f3830 con 0x7f1fa415b740 2024-01-15T17:58:07.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.182+0000 7f1f9bc00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa415b2c0 0x7f1fa40cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:07.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.183+0000 7f1fa9a00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 0x7f1fa40cc3d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40b9160 tx=0x7f1f9c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:07.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.183+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c016620 con 0x7f1fa415b740 2024-01-15T17:58:07.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.183+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f9c0167a0 con 0x7f1fa415b740 2024-01-15T17:58:07.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.183+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f9c016b50 con 0x7f1fa415b740 2024-01-15T17:58:07.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.183+0000 7f1faa400640 1 -- 172.21.15.2:0/808190299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fa41f3b10 con 0x7f1fa415b740 2024-01-15T17:58:07.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.183+0000 7f1faa400640 1 -- 172.21.15.2:0/808190299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fa41f67d0 con 0x7f1fa415b740 2024-01-15T17:58:07.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.185+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c005250 con 0x7f1fa415b740 2024-01-15T17:58:07.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.185+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1f9c016cb0 con 0x7f1fa415b740 2024-01-15T17:58:07.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.186+0000 7f1fa1600640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f7c06b2f0 0x7f1f7c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.186+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f9c08a910 con 0x7f1fa415b740 2024-01-15T17:58:07.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.187+0000 7f1fa9000640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f7c06b2f0 0x7f1f7c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.188+0000 7f1fa9000640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f7c06b2f0 0x7f1f7c06d7b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f94034000 tx=0x7f1f94007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:07.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.191+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f9c0556e0 con 0x7f1fa415b740 2024-01-15T17:58:07.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.383+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1f6c005a80 con 0x7f1fa415b740 2024-01-15T17:58:07.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.384+0000 7f1fa1600640 1 -- 172.21.15.2:0/808190299 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1f9c058d00 con 0x7f1fa415b740 2024-01-15T17:58:07.387 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:07.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f7c06b2f0 msgr2=0x7f1f7c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f7c06b2f0 0x7f1f7c06d7b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f94034000 tx=0x7f1f94007000 comp rx=0 tx=0).stop 2024-01-15T17:58:07.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 msgr2=0x7f1fa40cc3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 0x7f1fa40cc3d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1fa40b9160 tx=0x7f1f9c00f520 comp rx=0 tx=0).stop 2024-01-15T17:58:07.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 shutdown_connections 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f7c06b2f0 0x7f1f7c06d7b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1fa415b740 0x7f1fa40cc3d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1fa415b2c0 0x7f1fa40cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 --2- 172.21.15.2:0/808190299 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f84003970 0x7f1fa40b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 >> 172.21.15.2:0/808190299 conn(0x7f1fa40d30b0 msgr2=0x7f1fa40b2c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.386+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 shutdown_connections 2024-01-15T17:58:07.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.387+0000 7f1f99e00640 1 -- 172.21.15.2:0/808190299 wait complete. 2024-01-15T17:58:07.405 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:07.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.596+0000 7fcbed400640 1 Processor -- start 2024-01-15T17:58:07.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.597+0000 7fcbed400640 1 -- start start 2024-01-15T17:58:07.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.598+0000 7fcbed400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbe80ad6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.598+0000 7fcbed400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe806cb10 con 0x7fcbe81607c0 2024-01-15T17:58:07.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.598+0000 7fcbed400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80adbe0 con 0x7fcbe80b3980 2024-01-15T17:58:07.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.598+0000 7fcbed400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80b3560 con 0x7fcbe80ad280 2024-01-15T17:58:07.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcbe81607c0 0x7fcbe806ae80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48970/0 (socket says 172.21.15.2:48970) 2024-01-15T17:58:07.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe6800640 1 -- 172.21.15.2:0/2526772681 learned_addr learned my addr 172.21.15.2:0/2526772681 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587907295 0 0) 0x7fcbe806cb10 con 0x7fcbe81607c0 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc8003520 con 0x7fcbe81607c0 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbe80ad6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146077427 0 0) 0x7fcbc8003520 con 0x7fcbe81607c0 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 msgr2=0x7fcbe80ad6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbe80ad6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 >> v1:172.21.15.137:6789/0 conn(0x7fcbe80b3980 legacy=0x7fcbe80b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe806b960 con 0x7fcbe81607c0 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fcbd8001670 con 0x7fcbe81607c0 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.599+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 >> v1:172.21.15.2:6789/0 conn(0x7fcbe81607c0 legacy=0x7fcbe806ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe5e00640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc80039a0 0x7fcbc8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe5e00640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80047c0 0x7fcbc8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe5e00640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbc800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80b3560 con 0x7fcbc80039a0 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc80054e0 con 0x7fcbc80047c0 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc800bec0 con 0x7fcbe80ad280 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc80039a0 0x7fcbc8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80047c0 0x7fcbc8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe7200640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbc800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe6800640 1 -- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 msgr2=0x7fcbc800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbc800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe6800640 1 -- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80047c0 msgr2=0x7fcbc8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80047c0 0x7fcbc8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe6800640 1 -- 172.21.15.2:0/2526772681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8003520 con 0x7fcbc80039a0 2024-01-15T17:58:07.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.600+0000 7fcbe7200640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80ad280 0x7fcbc800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.601+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc80039a0 0x7fcbc8003dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800c190 tx=0x7fcbc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f6c0404db4920ec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8005ad0 con 0x7fcbc80039a0 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8005c30 con 0x7fcbc80039a0 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbe5e00640 1 -- 172.21.15.2:0/2526772681 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd802c8a0 con 0x7fcbc80039a0 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbed400640 1 -- 172.21.15.2:0/2526772681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc80039a0 msgr2=0x7fcbc8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbed400640 1 --2- 172.21.15.2:0/2526772681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc80039a0 0x7fcbc8003dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcbd800c190 tx=0x7fcbc80057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbe7200640 1 -- 172.21.15.2:0/2526772681 reap_dead start 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbed400640 1 -- 172.21.15.2:0/2526772681 shutdown_connections 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbed400640 1 -- 172.21.15.2:0/2526772681 >> 172.21.15.2:0/2526772681 conn(0x7fcbe80cf840 msgr2=0x7fcbe80cfc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbed400640 1 -- 172.21.15.2:0/2526772681 shutdown_connections 2024-01-15T17:58:07.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.602+0000 7fcbed400640 1 -- 172.21.15.2:0/2526772681 wait complete. 2024-01-15T17:58:07.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.604+0000 7fcbed400640 1 Processor -- start 2024-01-15T17:58:07.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.604+0000 7fcbed400640 1 -- start start 2024-01-15T17:58:07.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbed400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80039a0 0x7fcbe8150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbed400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe81542e0 0x7fcbe8158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbed400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 0x7fcbe80cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbed400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe8158d40 con 0x7fcbe81587c0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbed400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe815a0c0 con 0x7fcbc80039a0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbed400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe815a3a0 con 0x7fcbe81542e0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.605+0000 7fcbe7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 0x7fcbe80cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 0x7fcbe80cd3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44564/0 (socket says 172.21.15.2:44564) 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 -- 172.21.15.2:0/2634121949 learned_addr learned my addr 172.21.15.2:0/2634121949 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe81542e0 0x7fcbe8158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80039a0 0x7fcbe8150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 -- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe81542e0 msgr2=0x7fcbe8158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe81542e0 0x7fcbe8158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 -- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80039a0 msgr2=0x7fcbe8150750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80039a0 0x7fcbe8150750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 -- 172.21.15.2:0/2634121949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe815a680 con 0x7fcbe81587c0 2024-01-15T17:58:07.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.606+0000 7fcbe7200640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 0x7fcbe80cd3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8071c60 tx=0x7fcbe000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.607+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0003b70 con 0x7fcbe81587c0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.607+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0003cf0 con 0x7fcbe81587c0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.607+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00052b0 con 0x7fcbe81587c0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.607+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe815a960 con 0x7fcbe81587c0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.607+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe80cd8f0 con 0x7fcbe81587c0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.608+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe80686f0 con 0x7fcbe81587c0 2024-01-15T17:58:07.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.609+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcbe0005410 con 0x7fcbe81587c0 2024-01-15T17:58:07.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.610+0000 7fcbdea00640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:07.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.610+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcbe00895d0 con 0x7fcbe81587c0 2024-01-15T17:58:07.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.610+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:07.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.611+0000 7fcbe6800640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8034000 tx=0x7fcbd8037040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:07.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.615+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbe0054430 con 0x7fcbe81587c0 2024-01-15T17:58:07.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.810+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcbe80b2d50 con 0x7fcbe81587c0 2024-01-15T17:58:07.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.811+0000 7fcbdea00640 1 -- 172.21.15.2:0/2634121949 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcbe0003e60 con 0x7fcbe81587c0 2024-01-15T17:58:07.814 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.813+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 msgr2=0x7fcbc006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.813+0000 7fcbed400640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8034000 tx=0x7fcbd8037040 comp rx=0 tx=0).stop 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.813+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 msgr2=0x7fcbe80cd3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.813+0000 7fcbed400640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 0x7fcbe80cd3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8071c60 tx=0x7fcbe000dc00 comp rx=0 tx=0).stop 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 shutdown_connections 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbe81587c0 0x7fcbe80cd3b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe81542e0 0x7fcbe8158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 --2- 172.21.15.2:0/2634121949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc80039a0 0x7fcbe8150750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 >> 172.21.15.2:0/2634121949 conn(0x7fcbe80cf840 msgr2=0x7fcbe80af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 shutdown_connections 2024-01-15T17:58:07.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:07.814+0000 7fcbed400640 1 -- 172.21.15.2:0/2634121949 wait complete. 2024-01-15T17:58:07.834 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:08.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.040+0000 7fe76f400640 1 Processor -- start 2024-01-15T17:58:08.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.040+0000 7fe76f400640 1 -- start start 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 0x7fe76815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe76806cad0 0x7fe76806cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76f400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680716d0 con 0x7fe768141880 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76f400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe768071f20 con 0x7fe76806cad0 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76806c6b0 con 0x7fe76815ec60 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76e000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe768141880 0x7fe768070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48972/0 (socket says 172.21.15.2:48972) 2024-01-15T17:58:08.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.042+0000 7fe76e000640 1 -- 172.21.15.2:0/4197612813 learned_addr learned my addr 172.21.15.2:0/4197612813 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:08.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504242781 0 0) 0x7fe7680716d0 con 0x7fe768141880 2024-01-15T17:58:08.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe748003520 con 0x7fe768141880 2024-01-15T17:58:08.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76d600640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 0x7fe76815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76ea00640 1 --2- 172.21.15.2:0/4197612813 >> v2:172.21.15.137:3300/0 conn(0x7fe76806cad0 0x7fe76806cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703440447 0 0) 0x7fe748003520 con 0x7fe768141880 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 msgr2=0x7fe76815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 0x7fe76815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 >> v2:172.21.15.137:3300/0 conn(0x7fe76806cad0 msgr2=0x7fe76806cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 --2- 172.21.15.2:0/4197612813 >> v2:172.21.15.137:3300/0 conn(0x7fe76806cad0 0x7fe76806cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768071960 con 0x7fe768141880 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe760001670 con 0x7fe768141880 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76ea00640 1 --2- 172.21.15.2:0/4197612813 >> v2:172.21.15.137:3300/0 conn(0x7fe76806cad0 0x7fe76806cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 >> v1:172.21.15.2:6789/0 conn(0x7fe768141880 legacy=0x7fe768070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe748003970 0x7fe748003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 0x7fe7480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe748004d00 0x7fe748005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe768071f20 con 0x7fe748003970 2024-01-15T17:58:08.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76806c6b0 con 0x7fe748004d00 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.043+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74800ba30 con 0x7fe76815ec60 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76e000640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe748003970 0x7fe748003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76d600640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 0x7fe7480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76ea00640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe748004d00 0x7fe748005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76e000640 1 -- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 msgr2=0x7fe7480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76e000640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe76815ec60 0x7fe7480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76e000640 1 -- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe748004d00 msgr2=0x7fe748005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76e000640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe748004d00 0x7fe748005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76e000640 1 -- 172.21.15.2:0/4197612813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748003520 con 0x7fe748003970 2024-01-15T17:58:08.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.044+0000 7fe76ea00640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe748004d00 0x7fe748005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:08.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.045+0000 7fe76e000640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe748003970 0x7fe748003d90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe76000c290 tx=0x7fe7480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efaf6d1b0d8e22cc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.045+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760005ad0 con 0x7fe748003970 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.045+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe760005c30 con 0x7fe748003970 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.045+0000 7fe76cc00640 1 -- 172.21.15.2:0/4197612813 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76002c8d0 con 0x7fe748003970 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76f400640 1 -- 172.21.15.2:0/4197612813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe748003970 msgr2=0x7fe748003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76f400640 1 --2- 172.21.15.2:0/4197612813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe748003970 0x7fe748003d90 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe76000c290 tx=0x7fe7480057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76ea00640 1 -- 172.21.15.2:0/4197612813 reap_dead start 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76f400640 1 -- 172.21.15.2:0/4197612813 shutdown_connections 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76f400640 1 -- 172.21.15.2:0/4197612813 >> 172.21.15.2:0/4197612813 conn(0x7fe7680b0840 msgr2=0x7fe7680b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76f400640 1 -- 172.21.15.2:0/4197612813 shutdown_connections 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.046+0000 7fe76f400640 1 -- 172.21.15.2:0/4197612813 wait complete. 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.048+0000 7fe76f400640 1 Processor -- start 2024-01-15T17:58:08.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.048+0000 7fe76f400640 1 -- start start 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe748003970 0x7fe76813e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 0x7fe7681f1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7680b1320 0x7fe7681edce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76f400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681f2d60 con 0x7fe7680b0ec0 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76f400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680a92a0 con 0x7fe7680b1320 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681f27a0 con 0x7fe748003970 2024-01-15T17:58:08.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 0x7fe7681f1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 0x7fe7681f1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44578/0 (socket says 172.21.15.2:44578) 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.049+0000 7fe76d600640 1 -- 172.21.15.2:0/4234885390 learned_addr learned my addr 172.21.15.2:0/4234885390 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76ea00640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7680b1320 0x7fe7681edce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76e000640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe748003970 0x7fe76813e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76d600640 1 -- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe748003970 msgr2=0x7fe76813e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76d600640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe748003970 0x7fe76813e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76d600640 1 -- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7680b1320 msgr2=0x7fe7681edce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76d600640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7680b1320 0x7fe7681edce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76d600640 1 -- 172.21.15.2:0/4234885390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7681f2a80 con 0x7fe7680b0ec0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76ea00640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7680b1320 0x7fe7681edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76d600640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 0x7fe7681f1e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe758007f30 tx=0x7fe758011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75800e030 con 0x7fe7680b0ec0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe758016470 con 0x7fe7680b0ec0 2024-01-15T17:58:08.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758016820 con 0x7fe7680b0ec0 2024-01-15T17:58:08.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76815baa0 con 0x7fe7680b0ec0 2024-01-15T17:58:08.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.050+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681f2360 con 0x7fe7680b0ec0 2024-01-15T17:58:08.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.051+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe768074770 con 0x7fe7680b0ec0 2024-01-15T17:58:08.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.052+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe758016980 con 0x7fe7680b0ec0 2024-01-15T17:58:08.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.053+0000 7fe766000640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe74006b2f0 0x7fe74006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.053+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe75800f070 con 0x7fe7680b0ec0 2024-01-15T17:58:08.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.054+0000 7fe76e000640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe74006b2f0 0x7fe74006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.055+0000 7fe76e000640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe74006b2f0 0x7fe74006d7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe760034000 tx=0x7fe760007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.057+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe758055980 con 0x7fe7680b0ec0 2024-01-15T17:58:08.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.244+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe76806e1c0 con 0x7fe7680b0ec0 2024-01-15T17:58:08.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.246+0000 7fe766000640 1 -- 172.21.15.2:0/4234885390 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7fe758058fa0 con 0x7fe7680b0ec0 2024-01-15T17:58:08.249 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:08.249 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.248+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe74006b2f0 msgr2=0x7fe74006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.248+0000 7fe76f400640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe74006b2f0 0x7fe74006d7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe760034000 tx=0x7fe760007000 comp rx=0 tx=0).stop 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.248+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 msgr2=0x7fe7681f1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.248+0000 7fe76f400640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 0x7fe7681f1e20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe758007f30 tx=0x7fe758011660 comp rx=0 tx=0).stop 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 shutdown_connections 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe74006b2f0 0x7fe74006d7b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7680b1320 0x7fe7681edce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7680b0ec0 0x7fe7681f1e20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 --2- 172.21.15.2:0/4234885390 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe748003970 0x7fe76813e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 >> 172.21.15.2:0/4234885390 conn(0x7fe7680b0840 msgr2=0x7fe76815f0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 shutdown_connections 2024-01-15T17:58:08.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.249+0000 7fe76f400640 1 -- 172.21.15.2:0/4234885390 wait complete. 2024-01-15T17:58:08.270 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:08.270 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:08.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.485+0000 7f2b54200640 1 Processor -- start 2024-01-15T17:58:08.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.485+0000 7f2b54200640 1 -- start start 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b54200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2b4c0d1e80 0x7f2b4c0d02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b54200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0cec30 con 0x7f2b4c0b3310 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b54200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0cef10 con 0x7f2b4c15d4b0 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b54200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b4c0b2ef0 con 0x7f2b4c0d1e80 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b50e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2b4c0d1e80 0x7f2b4c0d02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b52200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2b4c0b3310 0x7f2b4c0b3760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48976/0 (socket says 172.21.15.2:48976) 2024-01-15T17:58:08.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.487+0000 7f2b52200640 1 -- 172.21.15.2:0/2676523500 learned_addr learned my addr 172.21.15.2:0/2676523500 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:08.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.488+0000 7f2b50e00640 1 -- 172.21.15.2:0/2676523500 >> v1:172.21.15.137:6789/0 conn(0x7f2b4c15d4b0 legacy=0x7f2b4c0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.488+0000 7f2b50e00640 1 -- 172.21.15.2:0/2676523500 >> v1:172.21.15.2:6789/0 conn(0x7f2b4c0b3310 legacy=0x7f2b4c0b3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.488+0000 7f2b50e00640 1 -- 172.21.15.2:0/2676523500 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c0cf1f0 con 0x7f2b4c0d1e80 2024-01-15T17:58:08.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.489+0000 7f2b50e00640 1 --2- 172.21.15.2:0/2676523500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0d02a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097a0 tx=0x7f2b3402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83523b170a3f68b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.489+0000 7f2b4be00640 1 -- 172.21.15.2:0/2676523500 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3402e070 con 0x7f2b4c0d1e80 2024-01-15T17:58:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.489+0000 7f2b4be00640 1 -- 172.21.15.2:0/2676523500 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b340315a0 con 0x7f2b4c0d1e80 2024-01-15T17:58:08.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.489+0000 7f2b4be00640 1 -- 172.21.15.2:0/2676523500 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34031930 con 0x7f2b4c0d1e80 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 -- 172.21.15.2:0/2676523500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0d1e80 msgr2=0x7f2b4c0d02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 --2- 172.21.15.2:0/2676523500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0d02a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2b340097a0 tx=0x7f2b3402b840 comp rx=0 tx=0).stop 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 -- 172.21.15.2:0/2676523500 shutdown_connections 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 --2- 172.21.15.2:0/2676523500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0d02a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 -- 172.21.15.2:0/2676523500 >> 172.21.15.2:0/2676523500 conn(0x7f2b4c09c510 msgr2=0x7f2b4c0d3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 -- 172.21.15.2:0/2676523500 shutdown_connections 2024-01-15T17:58:08.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.490+0000 7f2b54200640 1 -- 172.21.15.2:0/2676523500 wait complete. 2024-01-15T17:58:08.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.492+0000 7f2b54200640 1 Processor -- start 2024-01-15T17:58:08.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.492+0000 7f2b54200640 1 -- start start 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b54200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 0x7f2b4c152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b54200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0cad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b54200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b4c15d4b0 0x7f2b4c0b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b54200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0b6b00 con 0x7f2b4c0b3310 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b54200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0b5ef0 con 0x7f2b4c0d1e80 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b54200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b4c0b61d0 con 0x7f2b4c15d4b0 2024-01-15T17:58:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b51800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 0x7f2b4c152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b51800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 0x7f2b4c152590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44600/0 (socket says 172.21.15.2:44600) 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b51800640 1 -- 172.21.15.2:0/2295375208 learned_addr learned my addr 172.21.15.2:0/2295375208 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b50e00640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.493+0000 7f2b52200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b4c15d4b0 0x7f2b4c0b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b51800640 1 -- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b4c15d4b0 msgr2=0x7f2b4c0b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b51800640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b4c15d4b0 0x7f2b4c0b6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b51800640 1 -- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b4c0d1e80 msgr2=0x7f2b4c0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b51800640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0cad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b51800640 1 -- 172.21.15.2:0/2295375208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b34009400 con 0x7f2b4c0b3310 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b50e00640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b51800640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 0x7f2b4c152590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00c430 tx=0x7f2b3c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c011790 con 0x7f2b4c0b3310 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3c0118f0 con 0x7f2b4c0b3310 2024-01-15T17:58:08.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3c011ce0 con 0x7f2b4c0b3310 2024-01-15T17:58:08.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c0b64b0 con 0x7f2b4c0b3310 2024-01-15T17:58:08.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.494+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c152ad0 con 0x7f2b4c0b3310 2024-01-15T17:58:08.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.496+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c074850 con 0x7f2b4c0b3310 2024-01-15T17:58:08.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.497+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2b3c01c460 con 0x7f2b4c0b3310 2024-01-15T17:58:08.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.497+0000 7f2b49600640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b2406b310 0x7f2b2406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.498+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b3c089b50 con 0x7f2b4c0b3310 2024-01-15T17:58:08.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.498+0000 7f2b50e00640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b2406b310 0x7f2b2406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.499+0000 7f2b50e00640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b2406b310 0x7f2b2406d7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b34036000 tx=0x7f2b3403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.502+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b3c0549a0 con 0x7f2b4c0b3310 2024-01-15T17:58:08.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.696+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2b4c0b90b0 con 0x7f2b4c0b3310 2024-01-15T17:58:08.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.697+0000 7f2b49600640 1 -- 172.21.15.2:0/2295375208 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2b3c089030 con 0x7f2b4c0b3310 2024-01-15T17:58:08.700 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:08.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.699+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b2406b310 msgr2=0x7f2b2406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.699+0000 7f2b54200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b2406b310 0x7f2b2406d7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2b34036000 tx=0x7f2b3403d040 comp rx=0 tx=0).stop 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.699+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 msgr2=0x7f2b4c152590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.699+0000 7f2b54200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 0x7f2b4c152590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c00c430 tx=0x7f2b3c00c8f0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 shutdown_connections 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b2406b310 0x7f2b2406d7d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b4c15d4b0 0x7f2b4c0b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b4c0d1e80 0x7f2b4c0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 --2- 172.21.15.2:0/2295375208 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b4c0b3310 0x7f2b4c152590 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 >> 172.21.15.2:0/2295375208 conn(0x7f2b4c09c510 msgr2=0x7f2b4c0b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 shutdown_connections 2024-01-15T17:58:08.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.700+0000 7f2b54200640 1 -- 172.21.15.2:0/2295375208 wait complete. 2024-01-15T17:58:08.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.927+0000 7f4a76e00640 1 Processor -- start 2024-01-15T17:58:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.927+0000 7f4a76e00640 1 -- start start 2024-01-15T17:58:08.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.929+0000 7f4a76e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4a7006dbb0 0x7f4a70070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.929+0000 7f4a76e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4a70161210 0x7f4a70163600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.929+0000 7f4a76e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a70070650 con 0x7f4a70070b20 2024-01-15T17:58:08.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.929+0000 7f4a76e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a7006e030 con 0x7f4a70161210 2024-01-15T17:58:08.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.929+0000 7f4a76e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a70070880 con 0x7f4a7006dbb0 2024-01-15T17:58:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.929+0000 7f4a76400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4a70161210 0x7f4a70163600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a76400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4a70161210 0x7f4a70163600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44608/0 (socket says 172.21.15.2:44608) 2024-01-15T17:58:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a76400640 1 -- 172.21.15.2:0/1317195940 learned_addr learned my addr 172.21.15.2:0/1317195940 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a76400640 1 -- 172.21.15.2:0/1317195940 >> v2:172.21.15.137:3300/0 conn(0x7f4a7006dbb0 msgr2=0x7f4a70070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a75000640 1 --2- 172.21.15.2:0/1317195940 >> v2:172.21.15.137:3300/0 conn(0x7f4a7006dbb0 0x7f4a70070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a76400640 1 --2- 172.21.15.2:0/1317195940 >> v2:172.21.15.137:3300/0 conn(0x7f4a7006dbb0 0x7f4a70070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a76400640 1 -- 172.21.15.2:0/1317195940 >> v1:172.21.15.137:6789/0 conn(0x7f4a70070b20 legacy=0x7f4a70070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a76400640 1 -- 172.21.15.2:0/1317195940 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700b2b50 con 0x7f4a70161210 2024-01-15T17:58:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.930+0000 7f4a75000640 1 --2- 172.21.15.2:0/1317195940 >> v2:172.21.15.137:3300/0 conn(0x7f4a7006dbb0 0x7f4a70070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.931+0000 7f4a76400640 1 --2- 172.21.15.2:0/1317195940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a70163600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a7004f900 tx=0x7f4a6802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eda43bb1bc0e800 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.932+0000 7f4a6fe00640 1 -- 172.21.15.2:0/1317195940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6802ca80 con 0x7f4a70161210 2024-01-15T17:58:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.932+0000 7f4a6fe00640 1 -- 172.21.15.2:0/1317195940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6802cbe0 con 0x7f4a70161210 2024-01-15T17:58:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.932+0000 7f4a6fe00640 1 -- 172.21.15.2:0/1317195940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a680343f0 con 0x7f4a70161210 2024-01-15T17:58:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.932+0000 7f4a76e00640 1 -- 172.21.15.2:0/1317195940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 msgr2=0x7f4a70163600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.932+0000 7f4a76e00640 1 --2- 172.21.15.2:0/1317195940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a70163600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4a7004f900 tx=0x7f4a6802bee0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.933+0000 7f4a76e00640 1 -- 172.21.15.2:0/1317195940 shutdown_connections 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.933+0000 7f4a76e00640 1 --2- 172.21.15.2:0/1317195940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a70163600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.933+0000 7f4a76e00640 1 --2- 172.21.15.2:0/1317195940 >> v2:172.21.15.137:3300/0 conn(0x7f4a7006dbb0 0x7f4a70070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.933+0000 7f4a76e00640 1 -- 172.21.15.2:0/1317195940 >> 172.21.15.2:0/1317195940 conn(0x7f4a700b0840 msgr2=0x7f4a700b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.933+0000 7f4a76e00640 1 -- 172.21.15.2:0/1317195940 shutdown_connections 2024-01-15T17:58:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.933+0000 7f4a76e00640 1 -- 172.21.15.2:0/1317195940 wait complete. 2024-01-15T17:58:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.934+0000 7f4a76e00640 1 Processor -- start 2024-01-15T17:58:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.935+0000 7f4a76e00640 1 -- start start 2024-01-15T17:58:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.935+0000 7f4a76e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a7006dbb0 0x7f4a70150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a70070b20 0x7f4a701541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a700c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a701558a0 con 0x7f4a70161210 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70154c90 con 0x7f4a7006dbb0 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70154f70 con 0x7f4a70070b20 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a700c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a700c94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44620/0 (socket says 172.21.15.2:44620) 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 -- 172.21.15.2:0/748295835 learned_addr learned my addr 172.21.15.2:0/748295835 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a75a00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a7006dbb0 0x7f4a70150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a75000640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a70070b20 0x7f4a701541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 -- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a70070b20 msgr2=0x7f4a701541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a70070b20 0x7f4a701541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 -- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a7006dbb0 msgr2=0x7f4a70150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a7006dbb0 0x7f4a70150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.936+0000 7f4a76400640 1 -- 172.21.15.2:0/748295835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68009b30 con 0x7f4a70161210 2024-01-15T17:58:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.937+0000 7f4a76400640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a700c94a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a68036040 tx=0x7f4a68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.937+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a68009510 con 0x7f4a70161210 2024-01-15T17:58:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.937+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a6803e450 con 0x7f4a70161210 2024-01-15T17:58:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.937+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6803e800 con 0x7f4a70161210 2024-01-15T17:58:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.937+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a70155250 con 0x7f4a70161210 2024-01-15T17:58:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.937+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a70151580 con 0x7f4a70161210 2024-01-15T17:58:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.938+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a7006e430 con 0x7f4a70161210 2024-01-15T17:58:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.939+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4a6802cd50 con 0x7f4a70161210 2024-01-15T17:58:08.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.940+0000 7f4a6d600640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4a4806b2c0 0x7f4a4806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:08.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.940+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a680ac810 con 0x7f4a70161210 2024-01-15T17:58:08.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.941+0000 7f4a75a00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4a4806b2c0 0x7f4a4806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.942+0000 7f4a75a00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4a4806b2c0 0x7f4a4806d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a60004ea0 tx=0x7f4a60009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:08.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:08.945+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a68077690 con 0x7f4a70161210 2024-01-15T17:58:09.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.137+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a70074770 con 0x7f4a70161210 2024-01-15T17:58:09.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.138+0000 7f4a6d600640 1 -- 172.21.15.2:0/748295835 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a6807acb0 con 0x7f4a70161210 2024-01-15T17:58:09.141 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4a4806b2c0 msgr2=0x7f4a4806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4a4806b2c0 0x7f4a4806d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4a60004ea0 tx=0x7f4a60009040 comp rx=0 tx=0).stop 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 msgr2=0x7f4a700c94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a700c94a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a68036040 tx=0x7f4a68004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 shutdown_connections 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4a4806b2c0 0x7f4a4806d780 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4a70161210 0x7f4a700c94a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4a70070b20 0x7f4a701541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 --2- 172.21.15.2:0/748295835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4a7006dbb0 0x7f4a70150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 >> 172.21.15.2:0/748295835 conn(0x7f4a700b0840 msgr2=0x7f4a70158ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:09.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 shutdown_connections 2024-01-15T17:58:09.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.141+0000 7f4a76e00640 1 -- 172.21.15.2:0/748295835 wait complete. 2024-01-15T17:58:09.163 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:09.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.387+0000 7fae9fe00640 1 Processor -- start 2024-01-15T17:58:09.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.387+0000 7fae9fe00640 1 -- start start 2024-01-15T17:58:09.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.389+0000 7fae9fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fae9806abc0 0x7fae980b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.389+0000 7fae9fe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9806f060 con 0x7fae9815d4d0 2024-01-15T17:58:09.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.389+0000 7fae9fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae980aef50 con 0x7fae980af6a0 2024-01-15T17:58:09.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.389+0000 7fae9fe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9815d8b0 con 0x7fae9806abc0 2024-01-15T17:58:09.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.389+0000 7fae9d400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fae9815d4d0 0x7fae9806f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48984/0 (socket says 172.21.15.2:48984) 2024-01-15T17:58:09.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.389+0000 7fae9d400640 1 -- 172.21.15.2:0/1311747171 learned_addr learned my addr 172.21.15.2:0/1311747171 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:09.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae9de00640 1 --2- 172.21.15.2:0/1311747171 >> v2:172.21.15.2:3300/0 conn(0x7fae9806abc0 0x7fae980b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae97e00640 1 -- 172.21.15.2:0/1311747171 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734988757 0 0) 0x7fae9806f060 con 0x7fae9815d4d0 2024-01-15T17:58:09.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae97e00640 1 -- 172.21.15.2:0/1311747171 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae78003520 con 0x7fae9815d4d0 2024-01-15T17:58:09.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae9de00640 1 -- 172.21.15.2:0/1311747171 >> v1:172.21.15.137:6789/0 conn(0x7fae980af6a0 legacy=0x7fae9806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae9de00640 1 -- 172.21.15.2:0/1311747171 >> v1:172.21.15.2:6789/0 conn(0x7fae9815d4d0 legacy=0x7fae9806f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae97e00640 1 -- 172.21.15.2:0/1311747171 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273472297 0 0) 0x7fae78003520 con 0x7fae9815d4d0 2024-01-15T17:58:09.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.390+0000 7fae9de00640 1 -- 172.21.15.2:0/1311747171 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae980af230 con 0x7fae9806abc0 2024-01-15T17:58:09.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.391+0000 7fae9de00640 1 --2- 172.21.15.2:0/1311747171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9806abc0 0x7fae980b2860 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae90009d90 tx=0x7fae9002bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cc5ceb18a27435 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:09.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae97e00640 1 -- 172.21.15.2:0/1311747171 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88001670 con 0x7fae9806abc0 2024-01-15T17:58:09.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae97e00640 1 -- 172.21.15.2:0/1311747171 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9002c820 con 0x7fae9806abc0 2024-01-15T17:58:09.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae97e00640 1 -- 172.21.15.2:0/1311747171 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9002cbb0 con 0x7fae9806abc0 2024-01-15T17:58:09.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae9fe00640 1 -- 172.21.15.2:0/1311747171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9806abc0 msgr2=0x7fae980b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae9fe00640 1 --2- 172.21.15.2:0/1311747171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9806abc0 0x7fae980b2860 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fae90009d90 tx=0x7fae9002bcf0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae9fe00640 1 -- 172.21.15.2:0/1311747171 shutdown_connections 2024-01-15T17:58:09.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae9fe00640 1 --2- 172.21.15.2:0/1311747171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae9806abc0 0x7fae980b2860 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae9fe00640 1 -- 172.21.15.2:0/1311747171 >> 172.21.15.2:0/1311747171 conn(0x7fae980b1d70 msgr2=0x7fae980b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:09.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.392+0000 7fae9fe00640 1 -- 172.21.15.2:0/1311747171 shutdown_connections 2024-01-15T17:58:09.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.393+0000 7fae9fe00640 1 -- 172.21.15.2:0/1311747171 wait complete. 2024-01-15T17:58:09.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.394+0000 7fae9fe00640 1 Processor -- start 2024-01-15T17:58:09.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.394+0000 7fae9fe00640 1 -- start start 2024-01-15T17:58:09.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.395+0000 7fae9fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9806abc0 0x7fae9815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.395+0000 7fae9fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 0x7fae98147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.395+0000 7fae9fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9815d4d0 0x7fae9815fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.395+0000 7fae9fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae980cf280 con 0x7fae980af6a0 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.395+0000 7fae9fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9815df30 con 0x7fae9806abc0 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.395+0000 7fae9fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9815e1b0 con 0x7fae9815d4d0 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 0x7fae98147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 0x7fae98147850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44648/0 (socket says 172.21.15.2:44648) 2024-01-15T17:58:09.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 -- 172.21.15.2:0/2059456485 learned_addr learned my addr 172.21.15.2:0/2059456485 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:09.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9d400640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9806abc0 0x7fae9815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 -- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9815d4d0 msgr2=0x7fae9815fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9de00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9815d4d0 0x7fae9815fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9815d4d0 0x7fae9815fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 -- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9806abc0 msgr2=0x7fae9815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9806abc0 0x7fae9815d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 -- 172.21.15.2:0/2059456485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae90009a70 con 0x7fae980af6a0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9d400640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9806abc0 0x7fae9815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.396+0000 7fae9ca00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 0x7fae98147850 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae8000c400 tx=0x7fae8000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.397+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae800052d0 con 0x7fae980af6a0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.397+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae80005430 con 0x7fae980af6a0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.397+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9815e490 con 0x7fae980af6a0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.397+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80003830 con 0x7fae980af6a0 2024-01-15T17:58:09.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.397+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9815ea50 con 0x7fae980af6a0 2024-01-15T17:58:09.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.398+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae98067f30 con 0x7fae980af6a0 2024-01-15T17:58:09.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.399+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fae80003990 con 0x7fae980af6a0 2024-01-15T17:58:09.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.400+0000 7fae95600640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7006b2c0 0x7fae7006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.401+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae80089240 con 0x7fae980af6a0 2024-01-15T17:58:09.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.401+0000 7fae9d400640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7006b2c0 0x7fae7006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.402+0000 7fae9d400640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7006b2c0 0x7fae7006d780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae880011e0 tx=0x7fae88009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:09.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.405+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae80054010 con 0x7fae980af6a0 2024-01-15T17:58:09.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.600+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae980686f0 con 0x7fae980af6a0 2024-01-15T17:58:09.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.601+0000 7fae95600640 1 -- 172.21.15.2:0/2059456485 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fae80057630 con 0x7fae980af6a0 2024-01-15T17:58:09.604 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7006b2c0 msgr2=0x7fae7006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7006b2c0 0x7fae7006d780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fae880011e0 tx=0x7fae88009040 comp rx=0 tx=0).stop 2024-01-15T17:58:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 msgr2=0x7fae98147850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 0x7fae98147850 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fae8000c400 tx=0x7fae8000c8c0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 shutdown_connections 2024-01-15T17:58:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae7006b2c0 0x7fae7006d780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae9815d4d0 0x7fae9815fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae980af6a0 0x7fae98147850 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 --2- 172.21.15.2:0/2059456485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae9806abc0 0x7fae9815d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 >> 172.21.15.2:0/2059456485 conn(0x7fae980b1d70 msgr2=0x7fae980b1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 shutdown_connections 2024-01-15T17:58:09.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.604+0000 7fae9fe00640 1 -- 172.21.15.2:0/2059456485 wait complete. 2024-01-15T17:58:09.625 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:09.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.840+0000 7f1ae5e00640 1 Processor -- start 2024-01-15T17:58:09.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.840+0000 7f1ae5e00640 1 -- start start 2024-01-15T17:58:09.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.841+0000 7f1ae5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1ae015d140 0x7f1ae006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.841+0000 7f1ae5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1ae015d520 0x7f1ae0161330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.841+0000 7f1ae5e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae015e600 con 0x7f1ae0161870 2024-01-15T17:58:09.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.841+0000 7f1ae5e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae006fbc0 con 0x7f1ae015d140 2024-01-15T17:58:09.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.841+0000 7f1ae5e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae015dd60 con 0x7f1ae015d520 2024-01-15T17:58:09.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1ae015d140 0x7f1ae006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1ae015d140 0x7f1ae006f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44660/0 (socket says 172.21.15.2:44660) 2024-01-15T17:58:09.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 -- 172.21.15.2:0/3695200783 learned_addr learned my addr 172.21.15.2:0/3695200783 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:09.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 -- 172.21.15.2:0/3695200783 >> v2:172.21.15.137:3300/0 conn(0x7f1ae015d520 msgr2=0x7f1ae0161330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:09.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 --2- 172.21.15.2:0/3695200783 >> v2:172.21.15.137:3300/0 conn(0x7f1ae015d520 0x7f1ae0161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 -- 172.21.15.2:0/3695200783 >> v1:172.21.15.137:6789/0 conn(0x7f1ae0161870 legacy=0x7f1ae00cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:09.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.842+0000 7f1ae4a00640 1 -- 172.21.15.2:0/3695200783 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae015e040 con 0x7f1ae015d140 2024-01-15T17:58:09.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.843+0000 7f1ae4a00640 1 --2- 172.21.15.2:0/3695200783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae006f680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1ae006b1f0 tx=0x7f1ad002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc8a84d882089261 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:09.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.844+0000 7f1adfe00640 1 -- 172.21.15.2:0/3695200783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0031870 con 0x7f1ae015d140 2024-01-15T17:58:09.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.844+0000 7f1adfe00640 1 -- 172.21.15.2:0/3695200783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad00319d0 con 0x7f1ae015d140 2024-01-15T17:58:09.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.844+0000 7f1adfe00640 1 -- 172.21.15.2:0/3695200783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0031d60 con 0x7f1ae015d140 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 -- 172.21.15.2:0/3695200783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 msgr2=0x7f1ae006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/3695200783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae006f680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1ae006b1f0 tx=0x7f1ad002b910 comp rx=0 tx=0).stop 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 -- 172.21.15.2:0/3695200783 shutdown_connections 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/3695200783 >> v2:172.21.15.137:3300/0 conn(0x7f1ae015d520 0x7f1ae0161330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/3695200783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae006f680 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 -- 172.21.15.2:0/3695200783 >> 172.21.15.2:0/3695200783 conn(0x7f1ae00cfd00 msgr2=0x7f1ae00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 -- 172.21.15.2:0/3695200783 shutdown_connections 2024-01-15T17:58:09.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.845+0000 7f1ae5e00640 1 -- 172.21.15.2:0/3695200783 wait complete. 2024-01-15T17:58:09.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.847+0000 7f1ae5e00640 1 Processor -- start 2024-01-15T17:58:09.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.847+0000 7f1ae5e00640 1 -- start start 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae00b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1ae015d520 0x7f1ae015b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae5e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ae00b4c50 0x7f1ae00b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae5e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00b2dc0 con 0x7f1ae015d140 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae5e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00b57b0 con 0x7f1ae00b4c50 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae00b51f0 con 0x7f1ae015d520 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae00b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae00b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44672/0 (socket says 172.21.15.2:44672) 2024-01-15T17:58:09.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.848+0000 7f1ae4a00640 1 -- 172.21.15.2:0/315229541 learned_addr learned my addr 172.21.15.2:0/315229541 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae4a00640 1 -- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1ae015d520 msgr2=0x7f1ae015b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae4a00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1ae015d520 0x7f1ae015b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae5400640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ae00b4c50 0x7f1ae00b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae4a00640 1 -- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ae00b4c50 msgr2=0x7f1ae00b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae4a00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ae00b4c50 0x7f1ae00b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae4a00640 1 -- 172.21.15.2:0/315229541 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad00094d0 con 0x7f1ae015d140 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae5400640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ae00b4c50 0x7f1ae00b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae4a00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae00b4220 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad003adf0 con 0x7f1ae015d140 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad00443f0 con 0x7f1ae015d140 2024-01-15T17:58:09.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae00b54d0 con 0x7f1ae015d140 2024-01-15T17:58:09.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.849+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad00447a0 con 0x7f1ae015d140 2024-01-15T17:58:09.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.850+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae00b4760 con 0x7f1ae015d140 2024-01-15T17:58:09.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.851+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae00b2f00 con 0x7f1ae015d140 2024-01-15T17:58:09.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.852+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1ad004d450 con 0x7f1ae015d140 2024-01-15T17:58:09.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.853+0000 7f1add600640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ab806b2c0 0x7f1ab806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:09.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.853+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ad002d040 con 0x7f1ae015d140 2024-01-15T17:58:09.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.853+0000 7f1ad7c00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ab806b2c0 0x7f1ab806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:09.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.854+0000 7f1ad7c00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ab806b2c0 0x7f1ab806d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00b04c0 tx=0x7f1ac8006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:09.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:09.857+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad0077180 con 0x7f1ae015d140 2024-01-15T17:58:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.055+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ae0074770 con 0x7f1ae015d140 2024-01-15T17:58:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.058+0000 7f1add600640 1 -- 172.21.15.2:0/315229541 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f1ad007a7a0 con 0x7f1ae015d140 2024-01-15T17:58:10.061 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:10.061 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ab806b2c0 msgr2=0x7f1ab806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ab806b2c0 0x7f1ab806d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ae00b04c0 tx=0x7f1ac8006c90 comp rx=0 tx=0).stop 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 msgr2=0x7f1ae00b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae00b4220 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0000c00 tx=0x7f1ad00046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 shutdown_connections 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ab806b2c0 0x7f1ab806d780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ae00b4c50 0x7f1ae00b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1ae015d520 0x7f1ae015b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 --2- 172.21.15.2:0/315229541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ae015d140 0x7f1ae00b4220 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 >> 172.21.15.2:0/315229541 conn(0x7f1ae00cfd00 msgr2=0x7f1ae015ff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:10.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 shutdown_connections 2024-01-15T17:58:10.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.061+0000 7f1ae5e00640 1 -- 172.21.15.2:0/315229541 wait complete. 2024-01-15T17:58:10.081 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:10.081 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:10.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.302+0000 7fb414c00640 1 Processor -- start 2024-01-15T17:58:10.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.302+0000 7fb414c00640 1 -- start start 2024-01-15T17:58:10.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.304+0000 7fb414c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb4100b4c70 0x7fb4100d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.304+0000 7fb414c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 0x7fb4100b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.304+0000 7fb414c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4100b5590 con 0x7fb4100b8c30 2024-01-15T17:58:10.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.304+0000 7fb414c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4100d30b0 con 0x7fb4100b4c70 2024-01-15T17:58:10.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.304+0000 7fb414c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100d1900 con 0x7fb41015d730 2024-01-15T17:58:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.305+0000 7fb40e800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb4100b8c30 0x7fb4100d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48998/0 (socket says 172.21.15.2:48998) 2024-01-15T17:58:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.305+0000 7fb40e800640 1 -- 172.21.15.2:0/3757404254 learned_addr learned my addr 172.21.15.2:0/3757404254 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:10.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.305+0000 7fb40de00640 1 --2- 172.21.15.2:0/3757404254 >> v2:172.21.15.137:3300/0 conn(0x7fb4100b4c70 0x7fb4100d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.305+0000 7fb40d400640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 0x7fb4100b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.305+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184268013 0 0) 0x7fb4100b5590 con 0x7fb4100b8c30 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.305+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3f0003520 con 0x7fb4100b8c30 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2480301728 0 0) 0x7fb3f0003520 con 0x7fb4100b8c30 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 msgr2=0x7fb4100b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 0x7fb4100b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 >> v2:172.21.15.137:3300/0 conn(0x7fb4100b4c70 msgr2=0x7fb4100d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 --2- 172.21.15.2:0/3757404254 >> v2:172.21.15.137:3300/0 conn(0x7fb4100b4c70 0x7fb4100d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4100d1be0 con 0x7fb4100b8c30 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb4080019f0 con 0x7fb4100b8c30 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 >> v1:172.21.15.2:6789/0 conn(0x7fb4100b8c30 legacy=0x7fb4100d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 0x7fb3f0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3f0003f40 0x7fb3f00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0004d00 0x7fb3f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100d30b0 con 0x7fb3f0003f40 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100d1900 con 0x7fb3f0004d00 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f000b980 con 0x7fb41015d730 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40d400640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3f0003f40 0x7fb3f00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40de00640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 0x7fb3f0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.306+0000 7fb40e800640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0004d00 0x7fb3f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.307+0000 7fb40d400640 1 -- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 msgr2=0x7fb3f0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.307+0000 7fb40d400640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb41015d730 0x7fb3f0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.307+0000 7fb40d400640 1 -- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0004d00 msgr2=0x7fb3f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.307+0000 7fb40d400640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0004d00 0x7fb3f0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.307+0000 7fb40d400640 1 -- 172.21.15.2:0/3757404254 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0003520 con 0x7fb3f0003f40 2024-01-15T17:58:10.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.308+0000 7fb40d400640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3f0003f40 0x7fb3f00047c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009ac0 tx=0x7fb3f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ff4a5fe0de6b24 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:10.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.308+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f802c680 con 0x7fb3f0003f40 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.308+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f802c7e0 con 0x7fb3f0003f40 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.308+0000 7fb40ca00640 1 -- 172.21.15.2:0/3757404254 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f802cb70 con 0x7fb3f0003f40 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb414c00640 1 -- 172.21.15.2:0/3757404254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3f0003f40 msgr2=0x7fb3f00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb414c00640 1 --2- 172.21.15.2:0/3757404254 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3f0003f40 0x7fb3f00047c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009ac0 tx=0x7fb3f00057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb40e800640 1 -- 172.21.15.2:0/3757404254 reap_dead start 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb414c00640 1 -- 172.21.15.2:0/3757404254 shutdown_connections 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb414c00640 1 -- 172.21.15.2:0/3757404254 >> 172.21.15.2:0/3757404254 conn(0x7fb410072210 msgr2=0x7fb4100cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb414c00640 1 -- 172.21.15.2:0/3757404254 shutdown_connections 2024-01-15T17:58:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.309+0000 7fb414c00640 1 -- 172.21.15.2:0/3757404254 wait complete. 2024-01-15T17:58:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.311+0000 7fb414c00640 1 Processor -- start 2024-01-15T17:58:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.311+0000 7fb414c00640 1 -- start start 2024-01-15T17:58:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb414c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0003f40 0x7fb4100ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb414c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 0x7fb4100d3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb414c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb410143060 0x7fb4101434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb414c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100d3bb0 con 0x7fb4100d31f0 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb414c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100cc920 con 0x7fb3f0003f40 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb414c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4100cc050 con 0x7fb410143060 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb40d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 0x7fb4100d3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb40d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 0x7fb4100d3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44686/0 (socket says 172.21.15.2:44686) 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.312+0000 7fb40d400640 1 -- 172.21.15.2:0/3064516157 learned_addr learned my addr 172.21.15.2:0/3064516157 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40de00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0003f40 0x7fb4100ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40e800640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb410143060 0x7fb4101434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40d400640 1 -- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb410143060 msgr2=0x7fb4101434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40d400640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb410143060 0x7fb4101434e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40d400640 1 -- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0003f40 msgr2=0x7fb4100ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40d400640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0003f40 0x7fb4100ccdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40d400640 1 -- 172.21.15.2:0/3064516157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4101f37e0 con 0x7fb4100d31f0 2024-01-15T17:58:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb40d400640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 0x7fb4100d3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009be0 tx=0x7fb3f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8003810 con 0x7fb4100d31f0 2024-01-15T17:58:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f8003970 con 0x7fb4100d31f0 2024-01-15T17:58:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8003d20 con 0x7fb4100d31f0 2024-01-15T17:58:10.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.313+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4100cc2d0 con 0x7fb4100d31f0 2024-01-15T17:58:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.314+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb410143a20 con 0x7fb4100d31f0 2024-01-15T17:58:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.315+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4100d2440 con 0x7fb4100d31f0 2024-01-15T17:58:10.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.316+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb3f8034440 con 0x7fb4100d31f0 2024-01-15T17:58:10.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.317+0000 7fb406000640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb3e806b2c0 0x7fb3e806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.317+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3f80ab9c0 con 0x7fb4100d31f0 2024-01-15T17:58:10.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.317+0000 7fb40de00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb3e806b2c0 0x7fb3e806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.318+0000 7fb40de00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb3e806b2c0 0x7fb3e806d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb400009a90 tx=0x7fb400008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:10.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.322+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3f8076a90 con 0x7fb4100d31f0 2024-01-15T17:58:10.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.507+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb4100ccab0 con 0x7fb4100d31f0 2024-01-15T17:58:10.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.508+0000 7fb406000640 1 -- 172.21.15.2:0/3064516157 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3f80044e0 con 0x7fb4100d31f0 2024-01-15T17:58:10.511 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.510+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb3e806b2c0 msgr2=0x7fb3e806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.510+0000 7fb414c00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb3e806b2c0 0x7fb3e806d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb400009a90 tx=0x7fb400008040 comp rx=0 tx=0).stop 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.510+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 msgr2=0x7fb4100d3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.510+0000 7fb414c00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 0x7fb4100d3670 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009be0 tx=0x7fb3f8004730 comp rx=0 tx=0).stop 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.510+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 shutdown_connections 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.510+0000 7fb414c00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb3e806b2c0 0x7fb3e806d780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.511+0000 7fb414c00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb410143060 0x7fb4101434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.511+0000 7fb414c00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb4100d31f0 0x7fb4100d3670 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.511+0000 7fb414c00640 1 --2- 172.21.15.2:0/3064516157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3f0003f40 0x7fb4100ccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.511+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 >> 172.21.15.2:0/3064516157 conn(0x7fb410072210 msgr2=0x7fb4100b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.511+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 shutdown_connections 2024-01-15T17:58:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.511+0000 7fb414c00640 1 -- 172.21.15.2:0/3064516157 wait complete. 2024-01-15T17:58:10.531 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:10.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.747+0000 7f0ed5200640 1 Processor -- start 2024-01-15T17:58:10.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.748+0000 7f0ed5200640 1 -- start start 2024-01-15T17:58:10.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.749+0000 7f0ed5200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ed0144c00 0x7f0ed00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.749+0000 7f0ed5200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00ce230 con 0x7f0ed006d000 2024-01-15T17:58:10.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.749+0000 7f0ed5200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed00ce510 con 0x7f0ed00cdd70 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.749+0000 7f0ed5200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed00ce7f0 con 0x7f0ed0144c00 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ecde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ed0144c00 0x7f0ed00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ecde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ed0144c00 0x7f0ed00cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44692/0 (socket says 172.21.15.2:44692) 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ecde00640 1 -- 172.21.15.2:0/465773088 learned_addr learned my addr 172.21.15.2:0/465773088 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ece800640 1 --1- 172.21.15.2:0/465773088 >> v1:172.21.15.2:6789/0 conn(0x7f0ed006d000 0x7f0ed0070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49006/0 (socket says 172.21.15.2:49006) 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ecde00640 1 -- 172.21.15.2:0/465773088 >> v1:172.21.15.137:6789/0 conn(0x7f0ed00cdd70 legacy=0x7f0ed006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ecde00640 1 -- 172.21.15.2:0/465773088 >> v1:172.21.15.2:6789/0 conn(0x7f0ed006d000 legacy=0x7f0ed0070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.750+0000 7f0ecde00640 1 -- 172.21.15.2:0/465773088 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed00705f0 con 0x7f0ed0144c00 2024-01-15T17:58:10.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.751+0000 7f0ecde00640 1 --2- 172.21.15.2:0/465773088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed0144c00 0x7f0ed00cd830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009870 tx=0x7f0ec002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b2a4297da565bf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.751+0000 7f0ecd400640 1 -- 172.21.15.2:0/465773088 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0030860 con 0x7f0ed0144c00 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ecd400640 1 -- 172.21.15.2:0/465773088 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec00309c0 con 0x7f0ed0144c00 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ecd400640 1 -- 172.21.15.2:0/465773088 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0030d50 con 0x7f0ed0144c00 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 -- 172.21.15.2:0/465773088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed0144c00 msgr2=0x7f0ed00cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 --2- 172.21.15.2:0/465773088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed0144c00 0x7f0ed00cd830 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0009870 tx=0x7f0ec002b9a0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 -- 172.21.15.2:0/465773088 shutdown_connections 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 --2- 172.21.15.2:0/465773088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed0144c00 0x7f0ed00cd830 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 -- 172.21.15.2:0/465773088 >> 172.21.15.2:0/465773088 conn(0x7f0ed00b45f0 msgr2=0x7f0ed00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:10.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 -- 172.21.15.2:0/465773088 shutdown_connections 2024-01-15T17:58:10.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.752+0000 7f0ed5200640 1 -- 172.21.15.2:0/465773088 wait complete. 2024-01-15T17:58:10.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.754+0000 7f0ed5200640 1 Processor -- start 2024-01-15T17:58:10.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.754+0000 7f0ed5200640 1 -- start start 2024-01-15T17:58:10.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006d000 0x7f0ed01f3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 0x7f0ed013bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed0144c00 0x7f0ed0157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ed5200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed013c050 con 0x7f0ed00cdd70 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ed5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0158980 con 0x7f0ed006d000 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ed5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed01f2c40 con 0x7f0ed0144c00 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.755+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 0x7f0ed013bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 0x7f0ed013bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44694/0 (socket says 172.21.15.2:44694) 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 -- 172.21.15.2:0/3638661147 learned_addr learned my addr 172.21.15.2:0/3638661147 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ece800640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed0144c00 0x7f0ed0157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ecde00640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006d000 0x7f0ed01f3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 -- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed0144c00 msgr2=0x7f0ed0157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed0144c00 0x7f0ed0157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 -- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006d000 msgr2=0x7f0ed01f3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006d000 0x7f0ed01f3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 -- 172.21.15.2:0/3638661147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00094d0 con 0x7f0ed00cdd70 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.756+0000 7f0ec7e00640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 0x7f0ed013bb10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c440 tx=0x7f0eb800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.757+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb80035f0 con 0x7f0ed00cdd70 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.757+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb8003770 con 0x7f0ed00cdd70 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.757+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8003b20 con 0x7f0ed00cdd70 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.757+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed0157660 con 0x7f0ed00cdd70 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.757+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed0157bc0 con 0x7f0ed00cdd70 2024-01-15T17:58:10.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.758+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed00686f0 con 0x7f0ed00cdd70 2024-01-15T17:58:10.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.759+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0eb8003c80 con 0x7f0ed00cdd70 2024-01-15T17:58:10.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.760+0000 7f0ec6000640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806b2c0 0x7f0ea806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:10.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.760+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0eb8088db0 con 0x7f0ed00cdd70 2024-01-15T17:58:10.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.761+0000 7f0ecde00640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806b2c0 0x7f0ea806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:10.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.762+0000 7f0ecde00640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806b2c0 0x7f0ea806d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0036000 tx=0x7f0ec003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:10.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.764+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0eb8053cb0 con 0x7f0ed00cdd70 2024-01-15T17:58:10.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.961+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ed00b2950 con 0x7f0ed00cdd70 2024-01-15T17:58:10.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.962+0000 7f0ec6000640 1 -- 172.21.15.2:0/3638661147 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0eb8017020 con 0x7f0ed00cdd70 2024-01-15T17:58:10.965 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:10.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806b2c0 msgr2=0x7f0ea806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806b2c0 0x7f0ea806d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0036000 tx=0x7f0ec003b040 comp rx=0 tx=0).stop 2024-01-15T17:58:10.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 msgr2=0x7f0ed013bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:10.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 0x7f0ed013bb10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800c440 tx=0x7f0eb800c900 comp rx=0 tx=0).stop 2024-01-15T17:58:10.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 shutdown_connections 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806b2c0 0x7f0ea806d780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed0144c00 0x7f0ed0157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed00cdd70 0x7f0ed013bb10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 --2- 172.21.15.2:0/3638661147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006d000 0x7f0ed01f3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 >> 172.21.15.2:0/3638661147 conn(0x7f0ed00b45f0 msgr2=0x7f0ed00b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 shutdown_connections 2024-01-15T17:58:10.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:10.965+0000 7f0ed5200640 1 -- 172.21.15.2:0/3638661147 wait complete. 2024-01-15T17:58:10.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:11.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.206+0000 7f56cb200640 1 Processor -- start 2024-01-15T17:58:11.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.206+0000 7f56cb200640 1 -- start start 2024-01-15T17:58:11.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56cb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f56cc070820 0x7f56cc161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56cb200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f56cc072100 0x7f56cc06cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56cb200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56cc1620d0 con 0x7f56cc14b160 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56cb200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0b35b0 con 0x7f56cc072100 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56cb200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56cc0b1510 con 0x7f56cc070820 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56c9e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f56cc14b160 0x7f56cc070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49008/0 (socket says 172.21.15.2:49008) 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56c9e00640 1 -- 172.21.15.2:0/3906346624 learned_addr learned my addr 172.21.15.2:0/3906346624 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56ca800640 1 --2- 172.21.15.2:0/3906346624 >> v2:172.21.15.2:3300/0 conn(0x7f56cc072100 0x7f56cc06cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56c9400640 1 --2- 172.21.15.2:0/3906346624 >> v2:172.21.15.137:3300/0 conn(0x7f56cc070820 0x7f56cc161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56ca800640 1 -- 172.21.15.2:0/3906346624 >> v2:172.21.15.137:3300/0 conn(0x7f56cc070820 msgr2=0x7f56cc161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56ca800640 1 --2- 172.21.15.2:0/3906346624 >> v2:172.21.15.137:3300/0 conn(0x7f56cc070820 0x7f56cc161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56ca800640 1 -- 172.21.15.2:0/3906346624 >> v1:172.21.15.2:6789/0 conn(0x7f56cc14b160 legacy=0x7f56cc070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.208+0000 7f56ca800640 1 -- 172.21.15.2:0/3906346624 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56cc0b32d0 con 0x7f56cc072100 2024-01-15T17:58:11.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.210+0000 7f56ca800640 1 --2- 172.21.15.2:0/3906346624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc072100 0x7f56cc06cdd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06e040 tx=0x7f56c402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac7eec12cf875f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:11.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.210+0000 7f56c8a00640 1 -- 172.21.15.2:0/3906346624 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c402cc70 con 0x7f56cc072100 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.210+0000 7f56c8a00640 1 -- 172.21.15.2:0/3906346624 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c402cdd0 con 0x7f56cc072100 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.210+0000 7f56c8a00640 1 -- 172.21.15.2:0/3906346624 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4034660 con 0x7f56cc072100 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.210+0000 7f56cb200640 1 -- 172.21.15.2:0/3906346624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc072100 msgr2=0x7f56cc06cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.210+0000 7f56cb200640 1 --2- 172.21.15.2:0/3906346624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc072100 0x7f56cc06cdd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f56cc06e040 tx=0x7f56c402bec0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.211+0000 7f56cb200640 1 -- 172.21.15.2:0/3906346624 shutdown_connections 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.211+0000 7f56cb200640 1 --2- 172.21.15.2:0/3906346624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc072100 0x7f56cc06cdd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.211+0000 7f56cb200640 1 --2- 172.21.15.2:0/3906346624 >> v2:172.21.15.137:3300/0 conn(0x7f56cc070820 0x7f56cc161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.211+0000 7f56cb200640 1 -- 172.21.15.2:0/3906346624 >> 172.21.15.2:0/3906346624 conn(0x7f56cc0b0840 msgr2=0x7f56cc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.211+0000 7f56cb200640 1 -- 172.21.15.2:0/3906346624 shutdown_connections 2024-01-15T17:58:11.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.211+0000 7f56cb200640 1 -- 172.21.15.2:0/3906346624 wait complete. 2024-01-15T17:58:11.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.212+0000 7f56cb200640 1 Processor -- start 2024-01-15T17:58:11.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.212+0000 7f56cb200640 1 -- start start 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.213+0000 7f56cb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56cc070820 0x7f56cc160650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.213+0000 7f56cb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56cc072100 0x7f56cc1632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.213+0000 7f56cb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 0x7f56cc1637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.213+0000 7f56cb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc160b90 con 0x7f56cc14b160 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.213+0000 7f56cb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc1ebe30 con 0x7f56cc070820 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.213+0000 7f56cb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56cc1ec110 con 0x7f56cc072100 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 0x7f56cc1637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 0x7f56cc1637f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44706/0 (socket says 172.21.15.2:44706) 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 -- 172.21.15.2:0/2504361824 learned_addr learned my addr 172.21.15.2:0/2504361824 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56c9400640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56cc072100 0x7f56cc1632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56c9e00640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56cc070820 0x7f56cc160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 -- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56cc072100 msgr2=0x7f56cc1632b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56cc072100 0x7f56cc1632b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 -- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56cc070820 msgr2=0x7f56cc160650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56cc070820 0x7f56cc160650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 -- 172.21.15.2:0/2504361824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009a70 con 0x7f56cc14b160 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.214+0000 7f56ca800640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 0x7f56cc1637f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009e80 tx=0x7f56c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.215+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4040650 con 0x7f56cc14b160 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.215+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40407d0 con 0x7f56cc14b160 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.215+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56cc1ec3f0 con 0x7f56cc14b160 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.215+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4040b80 con 0x7f56cc14b160 2024-01-15T17:58:11.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.215+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56cc1ec9b0 con 0x7f56cc14b160 2024-01-15T17:58:11.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.216+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56cc074770 con 0x7f56cc14b160 2024-01-15T17:58:11.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.217+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f56c4040ce0 con 0x7f56cc14b160 2024-01-15T17:58:11.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.218+0000 7f56c2000640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f56a406b2c0 0x7f56a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.218+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56c40ac990 con 0x7f56cc14b160 2024-01-15T17:58:11.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.219+0000 7f56c9e00640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f56a406b2c0 0x7f56a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.220+0000 7f56c9e00640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f56a406b2c0 0x7f56a406d780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f56bc005880 tx=0x7f56bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:11.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.222+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f56c4077760 con 0x7f56cc14b160 2024-01-15T17:58:11.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.417+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56cc0b16e0 con 0x7f56cc14b160 2024-01-15T17:58:11.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.418+0000 7f56c2000640 1 -- 172.21.15.2:0/2504361824 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56c4039020 con 0x7f56cc14b160 2024-01-15T17:58:11.421 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.420+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f56a406b2c0 msgr2=0x7f56a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.420+0000 7f56cb200640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f56a406b2c0 0x7f56a406d780 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f56bc005880 tx=0x7f56bc009040 comp rx=0 tx=0).stop 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.420+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 msgr2=0x7f56cc1637f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.420+0000 7f56cb200640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 0x7f56cc1637f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f56c4009e80 tx=0x7f56c4004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 shutdown_connections 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f56a406b2c0 0x7f56a406d780 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f56cc14b160 0x7f56cc1637f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f56cc072100 0x7f56cc1632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 --2- 172.21.15.2:0/2504361824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f56cc070820 0x7f56cc160650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 >> 172.21.15.2:0/2504361824 conn(0x7f56cc0b0840 msgr2=0x7f56cc0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 shutdown_connections 2024-01-15T17:58:11.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.421+0000 7f56cb200640 1 -- 172.21.15.2:0/2504361824 wait complete. 2024-01-15T17:58:11.444 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:11.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.661+0000 7f1669000640 1 Processor -- start 2024-01-15T17:58:11.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.661+0000 7f1669000640 1 -- start start 2024-01-15T17:58:11.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1669000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 0x7f166406c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1669000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f166406c9b0 0x7f16640b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1669000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166406cdf0 con 0x7f16641469f0 2024-01-15T17:58:11.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1669000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16640ac870 con 0x7f166406c9b0 2024-01-15T17:58:11.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1669000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166415d970 con 0x7f1664146dd0 2024-01-15T17:58:11.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1659200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 0x7f166406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1659200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 0x7f166406c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:35662/0 (socket says 172.21.15.2:35662) 2024-01-15T17:58:11.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1659200640 1 -- 172.21.15.2:0/434746002 learned_addr learned my addr 172.21.15.2:0/434746002 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:11.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.663+0000 7f1661e00640 1 --1- 172.21.15.2:0/434746002 >> v1:172.21.15.137:6789/0 conn(0x7f16641469f0 0x7f1664071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:45318/0 (socket says 172.21.15.2:45318) 2024-01-15T17:58:11.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.664+0000 7f1662800640 1 --2- 172.21.15.2:0/434746002 >> v2:172.21.15.137:3300/0 conn(0x7f166406c9b0 0x7f16640b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.664+0000 7f1659200640 1 -- 172.21.15.2:0/434746002 >> v2:172.21.15.137:3300/0 conn(0x7f166406c9b0 msgr2=0x7f16640b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:11.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.664+0000 7f1659200640 1 --2- 172.21.15.2:0/434746002 >> v2:172.21.15.137:3300/0 conn(0x7f166406c9b0 0x7f16640b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.664+0000 7f1659200640 1 -- 172.21.15.2:0/434746002 >> v1:172.21.15.137:6789/0 conn(0x7f16641469f0 legacy=0x7f1664071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.664+0000 7f1659200640 1 -- 172.21.15.2:0/434746002 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166415dc30 con 0x7f1664146dd0 2024-01-15T17:58:11.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.666+0000 7f1659200640 1 --2- 172.21.15.2:0/434746002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 0x7f166406c470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f164c009e50 tx=0x7f164c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ff67544db4d69d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:11.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.666+0000 7f1661400640 1 -- 172.21.15.2:0/434746002 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c02cb00 con 0x7f1664146dd0 2024-01-15T17:58:11.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.666+0000 7f1661400640 1 -- 172.21.15.2:0/434746002 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c02cc60 con 0x7f1664146dd0 2024-01-15T17:58:11.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.666+0000 7f1661400640 1 -- 172.21.15.2:0/434746002 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c0343f0 con 0x7f1664146dd0 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 -- 172.21.15.2:0/434746002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 msgr2=0x7f166406c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 --2- 172.21.15.2:0/434746002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 0x7f166406c470 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f164c009e50 tx=0x7f164c02bf60 comp rx=0 tx=0).stop 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 -- 172.21.15.2:0/434746002 shutdown_connections 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 --2- 172.21.15.2:0/434746002 >> v2:172.21.15.137:3300/0 conn(0x7f166406c9b0 0x7f16640b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 --2- 172.21.15.2:0/434746002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1664146dd0 0x7f166406c470 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 -- 172.21.15.2:0/434746002 >> 172.21.15.2:0/434746002 conn(0x7f166406da20 msgr2=0x7f166406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 -- 172.21.15.2:0/434746002 shutdown_connections 2024-01-15T17:58:11.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.667+0000 7f1669000640 1 -- 172.21.15.2:0/434746002 wait complete. 2024-01-15T17:58:11.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.669+0000 7f1669000640 1 Processor -- start 2024-01-15T17:58:11.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.669+0000 7f1669000640 1 -- start start 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1669000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 0x7f16640af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1669000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16641469f0 0x7f166414c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1669000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16641381d0 0x7f16640afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1669000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16640af6c0 con 0x7f166406c9b0 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1669000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1664138040 con 0x7f16641381d0 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1669000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1664138690 con 0x7f16641469f0 2024-01-15T17:58:11.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1661e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 0x7f16640af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1661e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 0x7f16640af0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44708/0 (socket says 172.21.15.2:44708) 2024-01-15T17:58:11.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.670+0000 7f1661e00640 1 -- 172.21.15.2:0/3012923676 learned_addr learned my addr 172.21.15.2:0/3012923676 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:11.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1662800640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16641381d0 0x7f16640afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1659200640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16641469f0 0x7f166414c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1661e00640 1 -- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16641469f0 msgr2=0x7f166414c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1661e00640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16641469f0 0x7f166414c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1661e00640 1 -- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16641381d0 msgr2=0x7f16640afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1661e00640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16641381d0 0x7f16640afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1661e00640 1 -- 172.21.15.2:0/3012923676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c009b30 con 0x7f166406c9b0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f1661e00640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 0x7f16640af0e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f165400c4d0 tx=0x7f165400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1654007480 con 0x7f166406c9b0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16540075e0 con 0x7f166406c9b0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.671+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16540049e0 con 0x7f166406c9b0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.672+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166413b090 con 0x7f166406c9b0 2024-01-15T17:58:11.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.672+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166415cbf0 con 0x7f166406c9b0 2024-01-15T17:58:11.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.673+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16640686f0 con 0x7f166406c9b0 2024-01-15T17:58:11.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.674+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1654004b40 con 0x7f166406c9b0 2024-01-15T17:58:11.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.675+0000 7f165aa00640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f163c06b2c0 0x7f163c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:11.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.675+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1654088d50 con 0x7f166406c9b0 2024-01-15T17:58:11.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.676+0000 7f1659200640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f163c06b2c0 0x7f163c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:11.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.677+0000 7f1659200640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f163c06b2c0 0x7f163c06d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f164c035000 tx=0x7f164c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:11.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.679+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1654053b40 con 0x7f166406c9b0 2024-01-15T17:58:11.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.873+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f166406c500 con 0x7f166406c9b0 2024-01-15T17:58:11.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.875+0000 7f165aa00640 1 -- 172.21.15.2:0/3012923676 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f1654018070 con 0x7f166406c9b0 2024-01-15T17:58:11.878 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:11.878 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f163c06b2c0 msgr2=0x7f163c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f163c06b2c0 0x7f163c06d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f164c035000 tx=0x7f164c030000 comp rx=0 tx=0).stop 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 msgr2=0x7f16640af0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 0x7f16640af0e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f165400c4d0 tx=0x7f165400c990 comp rx=0 tx=0).stop 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 shutdown_connections 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f163c06b2c0 0x7f163c06d780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16641381d0 0x7f16640afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16641469f0 0x7f166414c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 --2- 172.21.15.2:0/3012923676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f166406c9b0 0x7f16640af0e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 >> 172.21.15.2:0/3012923676 conn(0x7f166406da20 msgr2=0x7f16640b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 shutdown_connections 2024-01-15T17:58:11.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:11.878+0000 7f1669000640 1 -- 172.21.15.2:0/3012923676 wait complete. 2024-01-15T17:58:11.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:11.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:12.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.124+0000 7f26b8a00640 1 Processor -- start 2024-01-15T17:58:12.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.125+0000 7f26b8a00640 1 -- start start 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.126+0000 7f26b8a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f26b40d1870 0x7f26b40d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.126+0000 7f26b8a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f26b40b2210 0x7f26b415d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.126+0000 7f26b8a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40d1250 con 0x7f26b40d1c50 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.126+0000 7f26b8a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b40ce6d0 con 0x7f26b40d1870 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.126+0000 7f26b8a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26b40ce8e0 con 0x7f26b40b2210 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f26b40d1870 0x7f26b40d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b3200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f26b40d1c50 0x7f26b40cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49024/0 (socket says 172.21.15.2:49024) 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f26b40d1870 0x7f26b40d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44724/0 (socket says 172.21.15.2:44724) 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b3200640 1 -- 172.21.15.2:0/3742445490 learned_addr learned my addr 172.21.15.2:0/3742445490 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26abe00640 1 --2- 172.21.15.2:0/3742445490 >> v2:172.21.15.137:3300/0 conn(0x7f26b40b2210 0x7f26b415d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b2800640 1 -- 172.21.15.2:0/3742445490 >> v2:172.21.15.137:3300/0 conn(0x7f26b40b2210 msgr2=0x7f26b415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:12.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b2800640 1 --2- 172.21.15.2:0/3742445490 >> v2:172.21.15.137:3300/0 conn(0x7f26b40b2210 0x7f26b415d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b2800640 1 -- 172.21.15.2:0/3742445490 >> v1:172.21.15.2:6789/0 conn(0x7f26b40d1c50 legacy=0x7f26b40cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b2800640 1 -- 172.21.15.2:0/3742445490 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40cebc0 con 0x7f26b40d1870 2024-01-15T17:58:12.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.127+0000 7f26b1e00640 1 -- 172.21.15.2:0/3742445490 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813821116 0 0) 0x7f26b40d1250 con 0x7f26b40d1c50 2024-01-15T17:58:12.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.129+0000 7f26b2800640 1 --2- 172.21.15.2:0/3742445490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40d3bd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f26b4067800 tx=0x7f26a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e472f3badbfbc0e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:12.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.129+0000 7f26b1e00640 1 -- 172.21.15.2:0/3742445490 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4031830 con 0x7f26b40d1870 2024-01-15T17:58:12.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.129+0000 7f26b1e00640 1 -- 172.21.15.2:0/3742445490 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26a4031990 con 0x7f26b40d1870 2024-01-15T17:58:12.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.129+0000 7f26b1e00640 1 -- 172.21.15.2:0/3742445490 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4031d20 con 0x7f26b40d1870 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.129+0000 7f26b8a00640 1 -- 172.21.15.2:0/3742445490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 msgr2=0x7f26b40d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.129+0000 7f26b8a00640 1 --2- 172.21.15.2:0/3742445490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40d3bd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f26b4067800 tx=0x7f26a402b8d0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.130+0000 7f26b8a00640 1 -- 172.21.15.2:0/3742445490 shutdown_connections 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.130+0000 7f26b8a00640 1 --2- 172.21.15.2:0/3742445490 >> v2:172.21.15.137:3300/0 conn(0x7f26b40b2210 0x7f26b415d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.130+0000 7f26b8a00640 1 --2- 172.21.15.2:0/3742445490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40d3bd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.130+0000 7f26b8a00640 1 -- 172.21.15.2:0/3742445490 >> 172.21.15.2:0/3742445490 conn(0x7f26b40d30b0 msgr2=0x7f26b40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.130+0000 7f26b8a00640 1 -- 172.21.15.2:0/3742445490 shutdown_connections 2024-01-15T17:58:12.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.130+0000 7f26b8a00640 1 -- 172.21.15.2:0/3742445490 wait complete. 2024-01-15T17:58:12.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.131+0000 7f26b8a00640 1 Processor -- start 2024-01-15T17:58:12.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.132+0000 7f26b8a00640 1 -- start start 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b8a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b40b2210 0x7f26b4152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b40d1c50 0x7f26b41f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b8a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b40cd030 con 0x7f26b40d1870 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b8a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b4145180 con 0x7f26b40b2210 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b415a3c0 con 0x7f26b40d1c50 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44736/0 (socket says 172.21.15.2:44736) 2024-01-15T17:58:12.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 -- 172.21.15.2:0/2055443483 learned_addr learned my addr 172.21.15.2:0/2055443483 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b2800640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b40b2210 0x7f26b4152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26b3200640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b40d1c50 0x7f26b41f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 -- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b40d1c50 msgr2=0x7f26b41f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b40d1c50 0x7f26b41f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 -- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b40b2210 msgr2=0x7f26b4152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b40b2210 0x7f26b4152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.133+0000 7f26abe00640 1 -- 172.21.15.2:0/2055443483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a4009400 con 0x7f26b40d1870 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.134+0000 7f26abe00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40ccaf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f269c00d7d0 tx=0x7f269c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.134+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c018070 con 0x7f26b40d1870 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.134+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269c004020 con 0x7f26b40d1870 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.134+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c0041b0 con 0x7f26b40d1870 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.134+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b41532b0 con 0x7f26b40d1870 2024-01-15T17:58:12.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.134+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b415a960 con 0x7f26b40d1870 2024-01-15T17:58:12.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.136+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b4074770 con 0x7f26b40d1870 2024-01-15T17:58:12.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.137+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f269c004310 con 0x7f26b40d1870 2024-01-15T17:58:12.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.138+0000 7f26aaa00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f268c06b2c0 0x7f268c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.138+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f269c089ae0 con 0x7f26b40d1870 2024-01-15T17:58:12.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.138+0000 7f26b2800640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f268c06b2c0 0x7f268c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.140+0000 7f26b2800640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f268c06b2c0 0x7f268c06d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f26a4036000 tx=0x7f26a403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:12.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.143+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f269c054950 con 0x7f26b40d1870 2024-01-15T17:58:12.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.346+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26b40b86a0 con 0x7f26b40d1870 2024-01-15T17:58:12.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.347+0000 7f26aaa00640 1 -- 172.21.15.2:0/2055443483 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f269c057f70 con 0x7f26b40d1870 2024-01-15T17:58:12.349 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f268c06b2c0 msgr2=0x7f268c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f268c06b2c0 0x7f268c06d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f26a4036000 tx=0x7f26a403c040 comp rx=0 tx=0).stop 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 msgr2=0x7f26b40ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40ccaf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f269c00d7d0 tx=0x7f269c00dc90 comp rx=0 tx=0).stop 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 shutdown_connections 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f268c06b2c0 0x7f268c06d780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b40d1c50 0x7f26b41f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b40d1870 0x7f26b40ccaf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 --2- 172.21.15.2:0/2055443483 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b40b2210 0x7f26b4152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 >> 172.21.15.2:0/2055443483 conn(0x7f26b40d30b0 msgr2=0x7f26b4152430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:12.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 shutdown_connections 2024-01-15T17:58:12.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.350+0000 7f26b8a00640 1 -- 172.21.15.2:0/2055443483 wait complete. 2024-01-15T17:58:12.372 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:12.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.597+0000 7f3367400640 1 Processor -- start 2024-01-15T17:58:12.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.597+0000 7f3367400640 1 -- start start 2024-01-15T17:58:12.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3367400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 0x7f33600cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3367400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33600ce230 con 0x7f336006d000 2024-01-15T17:58:12.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3367400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33600ce510 con 0x7f33600cdd70 2024-01-15T17:58:12.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3367400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600ce7f0 con 0x7f3360144c00 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3365400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f336006d000 0x7f3360070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49038/0 (socket says 172.21.15.2:49038) 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3365400640 1 -- 172.21.15.2:0/3660994656 learned_addr learned my addr 172.21.15.2:0/3660994656 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.599+0000 7f3364a00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 0x7f33600cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934283979 0 0) 0x7f33600ce230 con 0x7f336006d000 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3340003520 con 0x7f336006d000 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4085297727 0 0) 0x7f3340003520 con 0x7f336006d000 2024-01-15T17:58:12.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 msgr2=0x7f33600cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 0x7f33600cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 >> v1:172.21.15.137:6789/0 conn(0x7f33600cdd70 legacy=0x7f336006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33600705f0 con 0x7f336006d000 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f33580019f0 con 0x7f336006d000 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 >> v1:172.21.15.2:6789/0 conn(0x7f336006d000 legacy=0x7f3360070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33400039a0 0x7f3340003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3340004350 0x7f3340004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 0x7f3340005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600ce7f0 con 0x7f3340004350 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334000b980 con 0x7f33400039a0 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.600+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334000bb40 con 0x7f3360144c00 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3357c00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3340004350 0x7f3340004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3364a00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33400039a0 0x7f3340003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3365400640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 0x7f3340005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3357c00640 1 -- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 msgr2=0x7f3340005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:12.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3357c00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3360144c00 0x7f3340005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3357c00640 1 -- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33400039a0 msgr2=0x7f3340003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:12.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3357c00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33400039a0 0x7f3340003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.601+0000 7f3357c00640 1 -- 172.21.15.2:0/3660994656 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340003520 con 0x7f3340004350 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.602+0000 7f3357c00640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3340004350 0x7f3340004820 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3348001860 tx=0x7f33400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c946dea97501ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348033410 con 0x7f3340004350 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3348033590 con 0x7f3340004350 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f335fe00640 1 -- 172.21.15.2:0/3660994656 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348033940 con 0x7f3340004350 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3367400640 1 -- 172.21.15.2:0/3660994656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3340004350 msgr2=0x7f3340004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3367400640 1 --2- 172.21.15.2:0/3660994656 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3340004350 0x7f3340004820 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3348001860 tx=0x7f33400057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3365400640 1 -- 172.21.15.2:0/3660994656 reap_dead start 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3367400640 1 -- 172.21.15.2:0/3660994656 shutdown_connections 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3367400640 1 -- 172.21.15.2:0/3660994656 >> 172.21.15.2:0/3660994656 conn(0x7f33600b45f0 msgr2=0x7f33600b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3367400640 1 -- 172.21.15.2:0/3660994656 shutdown_connections 2024-01-15T17:58:12.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.603+0000 7f3367400640 1 -- 172.21.15.2:0/3660994656 wait complete. 2024-01-15T17:58:12.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.605+0000 7f3367400640 1 Processor -- start 2024-01-15T17:58:12.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.605+0000 7f3367400640 1 -- start start 2024-01-15T17:58:12.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.606+0000 7f3367400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3340004350 0x7f33601f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.606+0000 7f3367400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33600b1150 0x7f33600b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.606+0000 7f3367400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 0x7f33600c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.606+0000 7f3367400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33600c5fb0 con 0x7f33600c55f0 2024-01-15T17:58:12.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.606+0000 7f3367400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336015b2a0 con 0x7f33600b1150 2024-01-15T17:58:12.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.606+0000 7f3367400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33600c4b10 con 0x7f3340004350 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 0x7f33600c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 0x7f33600c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44752/0 (socket says 172.21.15.2:44752) 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 -- 172.21.15.2:0/1098415729 learned_addr learned my addr 172.21.15.2:0/1098415729 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3357c00640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33600b1150 0x7f33600b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3364a00640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3340004350 0x7f33601f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 -- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3340004350 msgr2=0x7f33601f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3340004350 0x7f33601f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 -- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33600b1150 msgr2=0x7f33600b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33600b1150 0x7f33600b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3365400640 1 -- 172.21.15.2:0/1098415729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33600c3f00 con 0x7f33600c55f0 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.607+0000 7f3364a00640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3340004350 0x7f33601f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.608+0000 7f3365400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 0x7f33600c5a70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3358007960 tx=0x7f335800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.608+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335800ec40 con 0x7f33600c55f0 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.608+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335800edc0 con 0x7f33600c55f0 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.608+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33600c41e0 con 0x7f33600c55f0 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.608+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3358014670 con 0x7f33600c55f0 2024-01-15T17:58:12.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.608+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33600c47a0 con 0x7f33600c55f0 2024-01-15T17:58:12.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.609+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33600686f0 con 0x7f33600c55f0 2024-01-15T17:58:12.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.610+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f33580149e0 con 0x7f33600c55f0 2024-01-15T17:58:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.611+0000 7f335d600640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f333806b2c0 0x7f333806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:12.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.612+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3358089be0 con 0x7f33600c55f0 2024-01-15T17:58:12.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.612+0000 7f3364a00640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f333806b2c0 0x7f333806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:12.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.613+0000 7f3364a00640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f333806b2c0 0x7f333806d780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3350009ab0 tx=0x7f3350008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:12.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.616+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3358054a50 con 0x7f33600c55f0 2024-01-15T17:58:12.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.812+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f336006b4c0 con 0x7f33600c55f0 2024-01-15T17:58:12.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.812+0000 7f335d600640 1 -- 172.21.15.2:0/1098415729 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3358058070 con 0x7f33600c55f0 2024-01-15T17:58:12.815 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:12.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f333806b2c0 msgr2=0x7f333806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f333806b2c0 0x7f333806d780 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3350009ab0 tx=0x7f3350008040 comp rx=0 tx=0).stop 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 msgr2=0x7f33600c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 0x7f33600c5a70 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3358007960 tx=0x7f335800e530 comp rx=0 tx=0).stop 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 shutdown_connections 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f333806b2c0 0x7f333806d780 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33600c55f0 0x7f33600c5a70 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33600b1150 0x7f33600b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 --2- 172.21.15.2:0/1098415729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3340004350 0x7f33601f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 >> 172.21.15.2:0/1098415729 conn(0x7f33600b45f0 msgr2=0x7f33600ad3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 shutdown_connections 2024-01-15T17:58:12.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:12.815+0000 7f3367400640 1 -- 172.21.15.2:0/1098415729 wait complete. 2024-01-15T17:58:12.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.059+0000 7f380ce00640 1 Processor -- start 2024-01-15T17:58:13.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.059+0000 7f380ce00640 1 -- start start 2024-01-15T17:58:13.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.060+0000 7f380ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f380806dbb0 0x7f3808070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f380ce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3808161210 0x7f3808163600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f380ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3808070650 con 0x7f3808070b20 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f380ce00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380806e030 con 0x7f3808161210 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f380ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3808070880 con 0x7f380806dbb0 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3808161210 0x7f3808163600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3808161210 0x7f3808163600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49428/0 (socket says 172.21.15.2:49428) 2024-01-15T17:58:13.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 -- 172.21.15.2:0/1233527719 learned_addr learned my addr 172.21.15.2:0/1233527719 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 -- 172.21.15.2:0/1233527719 >> v2:172.21.15.137:3300/0 conn(0x7f380806dbb0 msgr2=0x7f3808070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 --2- 172.21.15.2:0/1233527719 >> v2:172.21.15.137:3300/0 conn(0x7f380806dbb0 0x7f3808070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 -- 172.21.15.2:0/1233527719 >> v1:172.21.15.137:6789/0 conn(0x7f3808070b20 legacy=0x7f3808070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.061+0000 7f3807e00640 1 -- 172.21.15.2:0/1233527719 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38080b2b50 con 0x7f3808161210 2024-01-15T17:58:13.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.063+0000 7f3807e00640 1 --2- 172.21.15.2:0/1233527719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3808161210 0x7f3808163600 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f380804f900 tx=0x7f380002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b46b396705740d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.063+0000 7f3806000640 1 -- 172.21.15.2:0/1233527719 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380002ca80 con 0x7f3808161210 2024-01-15T17:58:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.063+0000 7f3806000640 1 -- 172.21.15.2:0/1233527719 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380002cbe0 con 0x7f3808161210 2024-01-15T17:58:13.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.063+0000 7f3806000640 1 -- 172.21.15.2:0/1233527719 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000343f0 con 0x7f3808161210 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 -- 172.21.15.2:0/1233527719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3808161210 msgr2=0x7f3808163600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 --2- 172.21.15.2:0/1233527719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3808161210 0x7f3808163600 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f380804f900 tx=0x7f380002bee0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 -- 172.21.15.2:0/1233527719 shutdown_connections 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 --2- 172.21.15.2:0/1233527719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3808161210 0x7f3808163600 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 --2- 172.21.15.2:0/1233527719 >> v2:172.21.15.137:3300/0 conn(0x7f380806dbb0 0x7f3808070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 -- 172.21.15.2:0/1233527719 >> 172.21.15.2:0/1233527719 conn(0x7f38080b0840 msgr2=0x7f38080b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 -- 172.21.15.2:0/1233527719 shutdown_connections 2024-01-15T17:58:13.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.064+0000 7f380ce00640 1 -- 172.21.15.2:0/1233527719 wait complete. 2024-01-15T17:58:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.066+0000 7f380ce00640 1 Processor -- start 2024-01-15T17:58:13.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.067+0000 7f380ce00640 1 -- start start 2024-01-15T17:58:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f380ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 0x7f3808156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f380ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3808070b20 0x7f3808154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f380ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3808161210 0x7f38081414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f380ce00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808141a30 con 0x7f380806dbb0 2024-01-15T17:58:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f380ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808155680 con 0x7f3808070b20 2024-01-15T17:58:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f380ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3808155920 con 0x7f3808161210 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f3807400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 0x7f3808156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f3807400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 0x7f3808156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49442/0 (socket says 172.21.15.2:49442) 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.068+0000 7f3807400640 1 -- 172.21.15.2:0/1661020197 learned_addr learned my addr 172.21.15.2:0/1661020197 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3806a00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3808070b20 0x7f3808154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807400640 1 -- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3808161210 msgr2=0x7f38081414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807e00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3808161210 0x7f38081414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807400640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3808161210 0x7f38081414f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807400640 1 -- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3808070b20 msgr2=0x7f3808154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807400640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3808070b20 0x7f3808154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807400640 1 -- 172.21.15.2:0/1661020197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800009b30 con 0x7f380806dbb0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3806a00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3808070b20 0x7f3808154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f3807400640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 0x7f3808156bb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f37f800c410 tx=0x7f37f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f80035f0 con 0x7f380806dbb0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f8003750 con 0x7f380806dbb0 2024-01-15T17:58:13.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f8003b40 con 0x7f380806dbb0 2024-01-15T17:58:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.069+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3808155c00 con 0x7f380806dbb0 2024-01-15T17:58:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.070+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3808154940 con 0x7f380806dbb0 2024-01-15T17:58:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.071+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380806e430 con 0x7f380806dbb0 2024-01-15T17:58:13.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.072+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f37f8003ca0 con 0x7f380806dbb0 2024-01-15T17:58:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.072+0000 7f37ff400640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f37e006b2c0 0x7f37e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.073+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37f8088e10 con 0x7f380806dbb0 2024-01-15T17:58:13.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.073+0000 7f3806a00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f37e006b2c0 0x7f37e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.074+0000 7f3806a00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f37e006b2c0 0x7f37e006d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f38081555b0 tx=0x7f37f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:13.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.078+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37f8053c80 con 0x7f380806dbb0 2024-01-15T17:58:13.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.271+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3808156120 con 0x7f380806dbb0 2024-01-15T17:58:13.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.272+0000 7f37ff400640 1 -- 172.21.15.2:0/1661020197 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f37f8017020 con 0x7f380806dbb0 2024-01-15T17:58:13.275 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.274+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f37e006b2c0 msgr2=0x7f37e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.274+0000 7f380ce00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f37e006b2c0 0x7f37e006d780 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f38081555b0 tx=0x7f37f0008040 comp rx=0 tx=0).stop 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.274+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 msgr2=0x7f3808156bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.274+0000 7f380ce00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 0x7f3808156bb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f37f800c410 tx=0x7f37f800c8d0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 shutdown_connections 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f37e006b2c0 0x7f37e006d780 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3808161210 0x7f38081414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3808070b20 0x7f3808154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 --2- 172.21.15.2:0/1661020197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f380806dbb0 0x7f3808156bb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 >> 172.21.15.2:0/1661020197 conn(0x7f38080b0840 msgr2=0x7f380815ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 shutdown_connections 2024-01-15T17:58:13.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.275+0000 7f380ce00640 1 -- 172.21.15.2:0/1661020197 wait complete. 2024-01-15T17:58:13.367 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-15T17:58:13.368 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T17:58:13.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.594+0000 7f6ae5a00640 1 Processor -- start 2024-01-15T17:58:13.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.594+0000 7f6ae5a00640 1 -- start start 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6ae014af80 0x7f6ae006d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae006da60 0x7f6ae0161c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ae01621d0 0x7f6ae00700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae00b28f0 con 0x7f6ae01621d0 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae00b1c70 con 0x7f6ae014af80 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00b1f50 con 0x7f6ae006da60 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ae01621d0 0x7f6ae00700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ae01621d0 0x7f6ae00700d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49454/0 (socket says 172.21.15.2:49454) 2024-01-15T17:58:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.596+0000 7f6ae5000640 1 -- 172.21.15.2:0/2689969430 learned_addr learned my addr 172.21.15.2:0/2689969430 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6ad7e00640 1 --2- 172.21.15.2:0/2689969430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae006da60 0x7f6ae0161c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6adfe00640 1 --2- 172.21.15.2:0/2689969430 >> v2:172.21.15.137:3300/0 conn(0x7f6ae014af80 0x7f6ae006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6ae5000640 1 -- 172.21.15.2:0/2689969430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae006da60 msgr2=0x7f6ae0161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6ae5000640 1 --2- 172.21.15.2:0/2689969430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae006da60 0x7f6ae0161c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6ae5000640 1 -- 172.21.15.2:0/2689969430 >> v2:172.21.15.137:3300/0 conn(0x7f6ae014af80 msgr2=0x7f6ae006d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6ae5000640 1 --2- 172.21.15.2:0/2689969430 >> v2:172.21.15.137:3300/0 conn(0x7f6ae014af80 0x7f6ae006d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.597+0000 7f6ae5000640 1 -- 172.21.15.2:0/2689969430 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae00b2230 con 0x7f6ae01621d0 2024-01-15T17:58:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.598+0000 7f6ae5000640 1 --2- 172.21.15.2:0/2689969430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae01621d0 0x7f6ae00700d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00cfc80 tx=0x7f6ad802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=784578b5d0dbca39 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.598+0000 7f6adf400640 1 -- 172.21.15.2:0/2689969430 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802c9e0 con 0x7f6ae01621d0 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.598+0000 7f6adf400640 1 -- 172.21.15.2:0/2689969430 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad802cb40 con 0x7f6ae01621d0 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.598+0000 7f6adf400640 1 -- 172.21.15.2:0/2689969430 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80343f0 con 0x7f6ae01621d0 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2689969430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae01621d0 msgr2=0x7f6ae00700d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2689969430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae01621d0 0x7f6ae00700d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00cfc80 tx=0x7f6ad802be40 comp rx=0 tx=0).stop 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2689969430 shutdown_connections 2024-01-15T17:58:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2689969430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae01621d0 0x7f6ae00700d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2689969430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae006da60 0x7f6ae0161c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2689969430 >> v2:172.21.15.137:3300/0 conn(0x7f6ae014af80 0x7f6ae006d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2689969430 >> 172.21.15.2:0/2689969430 conn(0x7f6ae00b0840 msgr2=0x7f6ae00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2689969430 shutdown_connections 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.599+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2689969430 wait complete. 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.601+0000 7f6ae5a00640 1 Processor -- start 2024-01-15T17:58:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.601+0000 7f6ae5a00640 1 -- start start 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ae5a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ae006da60 0x7f6ae01f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ae5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 0x7f6ae01448d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ae5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae01621d0 0x7f6ae0144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ae5a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae01f6ee0 con 0x7f6ae014af80 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ae5a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0140b20 con 0x7f6ae006da60 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ae5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae0140d30 con 0x7f6ae01621d0 2024-01-15T17:58:13.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ad7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 0x7f6ae01448d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ad7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 0x7f6ae01448d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49456/0 (socket says 172.21.15.2:49456) 2024-01-15T17:58:13.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.602+0000 7f6ad7e00640 1 -- 172.21.15.2:0/2730451093 learned_addr learned my addr 172.21.15.2:0/2730451093 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ad7e00640 1 -- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae01621d0 msgr2=0x7f6ae0144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ad7e00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae01621d0 0x7f6ae0144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ad7e00640 1 -- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ae006da60 msgr2=0x7f6ae01f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ad7e00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ae006da60 0x7f6ae01f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ad7e00640 1 -- 172.21.15.2:0/2730451093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad8009b30 con 0x7f6ae014af80 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ad7e00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 0x7f6ae01448d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c7f0 tx=0x7f6ac800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8004220 con 0x7f6ae014af80 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac8004380 con 0x7f6ae014af80 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8007860 con 0x7f6ae014af80 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae0141010 con 0x7f6ae014af80 2024-01-15T17:58:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.603+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae01415d0 con 0x7f6ae014af80 2024-01-15T17:58:13.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.605+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae0074770 con 0x7f6ae014af80 2024-01-15T17:58:13.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.606+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6ac8007bd0 con 0x7f6ae014af80 2024-01-15T17:58:13.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.607+0000 7f6adcc00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ab806b2c0 0x7f6ab806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:13.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.607+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ac8089bb0 con 0x7f6ae014af80 2024-01-15T17:58:13.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.607+0000 7f6adfe00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ab806b2c0 0x7f6ab806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:13.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.608+0000 7f6adfe00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ab806b2c0 0x7f6ab806d780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0006fd0 tx=0x7f6ad0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:13.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.611+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ac8054990 con 0x7f6ae014af80 2024-01-15T17:58:13.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.806+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ae01446b0 con 0x7f6ae014af80 2024-01-15T17:58:13.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.809+0000 7f6adcc00640 1 -- 172.21.15.2:0/2730451093 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f6ac801e070 con 0x7f6ae014af80 2024-01-15T17:58:13.812 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:13.813 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:13.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ab806b2c0 msgr2=0x7f6ab806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ab806b2c0 0x7f6ab806d780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0006fd0 tx=0x7f6ad0008040 comp rx=0 tx=0).stop 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 msgr2=0x7f6ae01448d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 0x7f6ae01448d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ac800c7f0 tx=0x7f6ac800ccb0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 shutdown_connections 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ab806b2c0 0x7f6ab806d780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ae01621d0 0x7f6ae0144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ae014af80 0x7f6ae01448d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 --2- 172.21.15.2:0/2730451093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ae006da60 0x7f6ae01f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.812+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 >> 172.21.15.2:0/2730451093 conn(0x7f6ae00b0840 msgr2=0x7f6ae01611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.813+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 shutdown_connections 2024-01-15T17:58:13.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:13.813+0000 7f6ae5a00640 1 -- 172.21.15.2:0/2730451093 wait complete. 2024-01-15T17:58:13.835 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:13.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:14.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.057+0000 7fb8aa600640 1 Processor -- start 2024-01-15T17:58:14.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.057+0000 7fb8aa600640 1 -- start start 2024-01-15T17:58:14.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8aa600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a4143d20 0x7fb8a415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8aa600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb8a41447d0 0x7fb8a415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8aa600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a4071090 con 0x7fb8a40720d0 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8aa600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a4071430 con 0x7fb8a41447d0 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8aa600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a4144100 con 0x7fb8a4143d20 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8a9c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb8a41447d0 0x7fb8a415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8a9c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb8a41447d0 0x7fb8a415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49466/0 (socket says 172.21.15.2:49466) 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8a9c00640 1 -- 172.21.15.2:0/4230507028 learned_addr learned my addr 172.21.15.2:0/4230507028 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.059+0000 7fb8a3e00640 1 --1- 172.21.15.2:0/4230507028 >> v1:172.21.15.2:6789/0 conn(0x7fb8a40720d0 0x7fb8a40cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35818/0 (socket says 172.21.15.2:35818) 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.060+0000 7fb8a9200640 1 --2- 172.21.15.2:0/4230507028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a4143d20 0x7fb8a415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.060+0000 7fb8a9c00640 1 -- 172.21.15.2:0/4230507028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a4143d20 msgr2=0x7fb8a415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.060+0000 7fb8a9c00640 1 --2- 172.21.15.2:0/4230507028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a4143d20 0x7fb8a415eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.060+0000 7fb8a9c00640 1 -- 172.21.15.2:0/4230507028 >> v1:172.21.15.2:6789/0 conn(0x7fb8a40720d0 legacy=0x7fb8a40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.060+0000 7fb8a9c00640 1 -- 172.21.15.2:0/4230507028 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a406d160 con 0x7fb8a41447d0 2024-01-15T17:58:14.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.061+0000 7fb8a9c00640 1 --2- 172.21.15.2:0/4230507028 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a41447d0 0x7fb8a415f090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009e50 tx=0x7fb89c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc2db0e9e6c04391 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.061+0000 7fb8a3400640 1 -- 172.21.15.2:0/4230507028 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c02ca70 con 0x7fb8a41447d0 2024-01-15T17:58:14.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.061+0000 7fb8a3400640 1 -- 172.21.15.2:0/4230507028 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89c02cbd0 con 0x7fb8a41447d0 2024-01-15T17:58:14.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.061+0000 7fb8a3400640 1 -- 172.21.15.2:0/4230507028 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89c0343f0 con 0x7fb8a41447d0 2024-01-15T17:58:14.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 -- 172.21.15.2:0/4230507028 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a41447d0 msgr2=0x7fb8a415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 --2- 172.21.15.2:0/4230507028 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a41447d0 0x7fb8a415f090 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009e50 tx=0x7fb89c02bed0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 -- 172.21.15.2:0/4230507028 shutdown_connections 2024-01-15T17:58:14.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 --2- 172.21.15.2:0/4230507028 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a41447d0 0x7fb8a415f090 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 --2- 172.21.15.2:0/4230507028 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a4143d20 0x7fb8a415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 -- 172.21.15.2:0/4230507028 >> 172.21.15.2:0/4230507028 conn(0x7fb8a40b0840 msgr2=0x7fb8a40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:14.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 -- 172.21.15.2:0/4230507028 shutdown_connections 2024-01-15T17:58:14.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.062+0000 7fb8aa600640 1 -- 172.21.15.2:0/4230507028 wait complete. 2024-01-15T17:58:14.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.064+0000 7fb8aa600640 1 Processor -- start 2024-01-15T17:58:14.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.064+0000 7fb8aa600640 1 -- start start 2024-01-15T17:58:14.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8aa600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8a40720d0 0x7fb8a415a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8aa600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 0x7fb8a4142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8aa600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a41447d0 0x7fb8a4142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8aa600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a415acf0 con 0x7fb8a4143d20 2024-01-15T17:58:14.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8aa600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a413f2c0 con 0x7fb8a40720d0 2024-01-15T17:58:14.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8aa600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a413f5a0 con 0x7fb8a41447d0 2024-01-15T17:58:14.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 0x7fb8a4142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.065+0000 7fb8a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 0x7fb8a4142950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49474/0 (socket says 172.21.15.2:49474) 2024-01-15T17:58:14.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 -- 172.21.15.2:0/3545614050 learned_addr learned my addr 172.21.15.2:0/3545614050 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a9c00640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a41447d0 0x7fb8a4142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 -- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a41447d0 msgr2=0x7fb8a4142e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a41447d0 0x7fb8a4142e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a9200640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8a40720d0 0x7fb8a415a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 -- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8a40720d0 msgr2=0x7fb8a415a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8a40720d0 0x7fb8a415a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 -- 172.21.15.2:0/3545614050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c009b30 con 0x7fb8a4143d20 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.066+0000 7fb8a3e00640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 0x7fb8a4142950 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00c3e0 tx=0x7fb88c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.067+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0074e0 con 0x7fb8a4143d20 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.067+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c007640 con 0x7fb8a4143d20 2024-01-15T17:58:14.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.067+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a413f880 con 0x7fb8a4143d20 2024-01-15T17:58:14.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.067+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb88c0049e0 con 0x7fb8a4143d20 2024-01-15T17:58:14.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.067+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a4140a30 con 0x7fb8a4143d20 2024-01-15T17:58:14.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.068+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a4140c20 con 0x7fb8a4143d20 2024-01-15T17:58:14.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.069+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb88c004b40 con 0x7fb8a4143d20 2024-01-15T17:58:14.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.070+0000 7fb8a0c00640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb87c06b2f0 0x7fb87c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.070+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb88c089500 con 0x7fb8a4143d20 2024-01-15T17:58:14.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.071+0000 7fb8a9200640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb87c06b2f0 0x7fb87c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.072+0000 7fb8a9200640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb87c06b2f0 0x7fb87c06d7b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb894006fd0 tx=0x7fb894008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.075+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8a4140c20 con 0x7fb8a4143d20 2024-01-15T17:58:14.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.269+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8a4140c20 con 0x7fb8a4143d20 2024-01-15T17:58:14.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.270+0000 7fb8a0c00640 1 -- 172.21.15.2:0/3545614050 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8a4140c20 con 0x7fb8a4143d20 2024-01-15T17:58:14.273 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:14.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb87c06b2f0 msgr2=0x7fb87c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb87c06b2f0 0x7fb87c06d7b0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb894006fd0 tx=0x7fb894008040 comp rx=0 tx=0).stop 2024-01-15T17:58:14.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 msgr2=0x7fb8a4142950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 0x7fb8a4142950 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb88c00c3e0 tx=0x7fb88c00c8a0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 shutdown_connections 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb87c06b2f0 0x7fb87c06d7b0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8a41447d0 0x7fb8a4142e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8a4143d20 0x7fb8a4142950 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 --2- 172.21.15.2:0/3545614050 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8a40720d0 0x7fb8a415a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 >> 172.21.15.2:0/3545614050 conn(0x7fb8a40b0840 msgr2=0x7fb8a4142170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.273+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 shutdown_connections 2024-01-15T17:58:14.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.274+0000 7fb8aa600640 1 -- 172.21.15.2:0/3545614050 wait complete. 2024-01-15T17:58:14.296 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:14.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.511+0000 7f95d3400640 1 Processor -- start 2024-01-15T17:58:14.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.512+0000 7f95d3400640 1 -- start start 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d3400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f95cc15c450 0x7f95cc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f95cc06ce30 0x7f95cc070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d3400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95cc06d290 con 0x7f95cc144c00 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d3400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95cc0ce230 con 0x7f95cc06ce30 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d3400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95cc0ce510 con 0x7f95cc15c450 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d0a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f95cc144c00 0x7f95cc0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35820/0 (socket says 172.21.15.2:35820) 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d0a00640 1 -- 172.21.15.2:0/4091632622 learned_addr learned my addr 172.21.15.2:0/4091632622 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:14.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.514+0000 7f95d1400640 1 --2- 172.21.15.2:0/4091632622 >> v2:172.21.15.2:3300/0 conn(0x7f95cc06ce30 0x7f95cc070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95cbe00640 1 --2- 172.21.15.2:0/4091632622 >> v2:172.21.15.137:3300/0 conn(0x7f95cc15c450 0x7f95cc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95cb400640 1 -- 172.21.15.2:0/4091632622 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145922555 0 0) 0x7f95cc06d290 con 0x7f95cc144c00 2024-01-15T17:58:14.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95cb400640 1 -- 172.21.15.2:0/4091632622 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95ac003520 con 0x7f95cc144c00 2024-01-15T17:58:14.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95d1400640 1 -- 172.21.15.2:0/4091632622 >> v2:172.21.15.137:3300/0 conn(0x7f95cc15c450 msgr2=0x7f95cc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:14.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95d1400640 1 --2- 172.21.15.2:0/4091632622 >> v2:172.21.15.137:3300/0 conn(0x7f95cc15c450 0x7f95cc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95d1400640 1 -- 172.21.15.2:0/4091632622 >> v1:172.21.15.2:6789/0 conn(0x7f95cc144c00 legacy=0x7f95cc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.515+0000 7f95d1400640 1 -- 172.21.15.2:0/4091632622 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc0ce7f0 con 0x7f95cc06ce30 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.516+0000 7f95d1400640 1 --2- 172.21.15.2:0/4091632622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc06ce30 0x7f95cc070b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95cc06f9f0 tx=0x7f95c402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e515b6b754ca3304 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.516+0000 7f95cb400640 1 -- 172.21.15.2:0/4091632622 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c402cc70 con 0x7f95cc06ce30 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.516+0000 7f95cb400640 1 -- 172.21.15.2:0/4091632622 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c402cdd0 con 0x7f95cc06ce30 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.516+0000 7f95cb400640 1 -- 172.21.15.2:0/4091632622 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4034660 con 0x7f95cc06ce30 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 -- 172.21.15.2:0/4091632622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc06ce30 msgr2=0x7f95cc070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 --2- 172.21.15.2:0/4091632622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc06ce30 0x7f95cc070b10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f95cc06f9f0 tx=0x7f95c402bec0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 -- 172.21.15.2:0/4091632622 shutdown_connections 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 --2- 172.21.15.2:0/4091632622 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc06ce30 0x7f95cc070b10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 --2- 172.21.15.2:0/4091632622 >> v2:172.21.15.137:3300/0 conn(0x7f95cc15c450 0x7f95cc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 -- 172.21.15.2:0/4091632622 >> 172.21.15.2:0/4091632622 conn(0x7f95cc0b45f0 msgr2=0x7f95cc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 -- 172.21.15.2:0/4091632622 shutdown_connections 2024-01-15T17:58:14.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.517+0000 7f95d3400640 1 -- 172.21.15.2:0/4091632622 wait complete. 2024-01-15T17:58:14.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.519+0000 7f95d3400640 1 Processor -- start 2024-01-15T17:58:14.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.519+0000 7f95d3400640 1 -- start start 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95cc06ce30 0x7f95cc1f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95cc144c00 0x7f95cc0ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 0x7f95cc0acde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d3400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc13d360 con 0x7f95cc15c450 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d3400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc13c700 con 0x7f95cc144c00 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95cc13c9c0 con 0x7f95cc06ce30 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 0x7f95cc0acde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 0x7f95cc0acde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49500/0 (socket says 172.21.15.2:49500) 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.520+0000 7f95d1400640 1 -- 172.21.15.2:0/3582025990 learned_addr learned my addr 172.21.15.2:0/3582025990 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d0a00640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95cc06ce30 0x7f95cc1f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95cbe00640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95cc144c00 0x7f95cc0ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d1400640 1 -- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95cc06ce30 msgr2=0x7f95cc1f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d1400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95cc06ce30 0x7f95cc1f2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d1400640 1 -- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95cc144c00 msgr2=0x7f95cc0ac8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d1400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95cc144c00 0x7f95cc0ac8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d1400640 1 -- 172.21.15.2:0/3582025990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c4009a70 con 0x7f95cc15c450 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d1400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 0x7f95cc0acde0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009e80 tx=0x7f95c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4040650 con 0x7f95cc15c450 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95c40407b0 con 0x7f95cc15c450 2024-01-15T17:58:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95c4040b60 con 0x7f95cc15c450 2024-01-15T17:58:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.521+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95cc13cca0 con 0x7f95cc15c450 2024-01-15T17:58:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.522+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95cc1f28d0 con 0x7f95cc15c450 2024-01-15T17:58:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.523+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95cc13d4a0 con 0x7f95cc15c450 2024-01-15T17:58:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.524+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f95c4040cc0 con 0x7f95cc15c450 2024-01-15T17:58:14.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.525+0000 7f95c8c00640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f95a406b2f0 0x7f95a406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.525+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f95c40ac6f0 con 0x7f95cc15c450 2024-01-15T17:58:14.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.525+0000 7f95d0a00640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f95a406b2f0 0x7f95a406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.526+0000 7f95d0a00640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f95a406b2f0 0x7f95a406d7b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0011e0 tx=0x7f95bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.529+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95cc13d4a0 con 0x7f95cc15c450 2024-01-15T17:58:14.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.721+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95cc13d4a0 con 0x7f95cc15c450 2024-01-15T17:58:14.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.721+0000 7f95c8c00640 1 -- 172.21.15.2:0/3582025990 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95cc13d4a0 con 0x7f95cc15c450 2024-01-15T17:58:14.725 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:14.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f95a406b2f0 msgr2=0x7f95a406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f95a406b2f0 0x7f95a406d7b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f95bc0011e0 tx=0x7f95bc009040 comp rx=0 tx=0).stop 2024-01-15T17:58:14.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 msgr2=0x7f95cc0acde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 0x7f95cc0acde0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f95c4009e80 tx=0x7f95c4004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 shutdown_connections 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f95a406b2f0 0x7f95a406d7b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f95cc15c450 0x7f95cc0acde0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f95cc144c00 0x7f95cc0ac8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 --2- 172.21.15.2:0/3582025990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f95cc06ce30 0x7f95cc1f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.724+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 >> 172.21.15.2:0/3582025990 conn(0x7f95cc0b45f0 msgr2=0x7f95cc15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.725+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 shutdown_connections 2024-01-15T17:58:14.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.725+0000 7f95d3400640 1 -- 172.21.15.2:0/3582025990 wait complete. 2024-01-15T17:58:14.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:14.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.965+0000 7fdd8ca00640 1 Processor -- start 2024-01-15T17:58:14.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.965+0000 7fdd8ca00640 1 -- start start 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8ca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdd840b4c70 0x7fdd840d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8ca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdd840b8c30 0x7fdd840d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8ca00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd840d1900 con 0x7fdd8415d730 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8ca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd840d1be0 con 0x7fdd840b8c30 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8ca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd840d1ec0 con 0x7fdd840b4c70 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdd840b8c30 0x7fdd840d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdd840b8c30 0x7fdd840d29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49508/0 (socket says 172.21.15.2:49508) 2024-01-15T17:58:14.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8aa00640 1 -- 172.21.15.2:0/203481090 learned_addr learned my addr 172.21.15.2:0/203481090 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:14.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.967+0000 7fdd8a000640 1 --2- 172.21.15.2:0/203481090 >> v2:172.21.15.137:3300/0 conn(0x7fdd840b4c70 0x7fdd840d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.968+0000 7fdd8aa00640 1 -- 172.21.15.2:0/203481090 >> v2:172.21.15.137:3300/0 conn(0x7fdd840b4c70 msgr2=0x7fdd840d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:14.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.968+0000 7fdd8aa00640 1 --2- 172.21.15.2:0/203481090 >> v2:172.21.15.137:3300/0 conn(0x7fdd840b4c70 0x7fdd840d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.968+0000 7fdd8aa00640 1 -- 172.21.15.2:0/203481090 >> v1:172.21.15.2:6789/0 conn(0x7fdd8415d730 legacy=0x7fdd840b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.968+0000 7fdd8aa00640 1 -- 172.21.15.2:0/203481090 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd840d21a0 con 0x7fdd840b8c30 2024-01-15T17:58:14.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.968+0000 7fdd88c00640 1 -- 172.21.15.2:0/203481090 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283626138 0 0) 0x7fdd840d1900 con 0x7fdd8415d730 2024-01-15T17:58:14.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.969+0000 7fdd8aa00640 1 --2- 172.21.15.2:0/203481090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b8c30 0x7fdd840d29a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdd840b5c40 tx=0x7fdd7c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26aee582ae2a13a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.969+0000 7fdd88c00640 1 -- 172.21.15.2:0/203481090 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c02cb20 con 0x7fdd840b8c30 2024-01-15T17:58:14.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.969+0000 7fdd88c00640 1 -- 172.21.15.2:0/203481090 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c02cc80 con 0x7fdd840b8c30 2024-01-15T17:58:14.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.969+0000 7fdd88c00640 1 -- 172.21.15.2:0/203481090 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c0343f0 con 0x7fdd840b8c30 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 -- 172.21.15.2:0/203481090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b8c30 msgr2=0x7fdd840d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/203481090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b8c30 0x7fdd840d29a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdd840b5c40 tx=0x7fdd7c02bf80 comp rx=0 tx=0).stop 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 -- 172.21.15.2:0/203481090 shutdown_connections 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/203481090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b8c30 0x7fdd840d29a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/203481090 >> v2:172.21.15.137:3300/0 conn(0x7fdd840b4c70 0x7fdd840d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 -- 172.21.15.2:0/203481090 >> 172.21.15.2:0/203481090 conn(0x7fdd84072210 msgr2=0x7fdd840cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 -- 172.21.15.2:0/203481090 shutdown_connections 2024-01-15T17:58:14.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.970+0000 7fdd8ca00640 1 -- 172.21.15.2:0/203481090 wait complete. 2024-01-15T17:58:14.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.972+0000 7fdd8ca00640 1 Processor -- start 2024-01-15T17:58:14.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.972+0000 7fdd8ca00640 1 -- start start 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 0x7fdd841f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd840b8c30 0x7fdd8415b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd8415d730 0x7fdd841f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8ca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd841f60b0 con 0x7fdd840b4c70 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8415b5d0 con 0x7fdd840b8c30 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd841f3530 con 0x7fdd8415d730 2024-01-15T17:58:14.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 0x7fdd841f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 0x7fdd841f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49510/0 (socket says 172.21.15.2:49510) 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.973+0000 7fdd8a000640 1 -- 172.21.15.2:0/2403855614 learned_addr learned my addr 172.21.15.2:0/2403855614 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8aa00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd8415d730 0x7fdd841f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd89600640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd840b8c30 0x7fdd8415b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8a000640 1 -- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd8415d730 msgr2=0x7fdd841f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8a000640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd8415d730 0x7fdd841f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8a000640 1 -- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd840b8c30 msgr2=0x7fdd8415b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8a000640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd840b8c30 0x7fdd8415b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8a000640 1 -- 172.21.15.2:0/2403855614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c009b30 con 0x7fdd840b4c70 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd89600640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd840b8c30 0x7fdd8415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8a000640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 0x7fdd841f6b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400d800 tx=0x7fdd7400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400b6c0 con 0x7fdd840b4c70 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7400b820 con 0x7fdd840b4c70 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd840d3060 con 0x7fdd840b4c70 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.974+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7400bbb0 con 0x7fdd840b4c70 2024-01-15T17:58:14.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.975+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd841f6300 con 0x7fdd840b4c70 2024-01-15T17:58:14.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.976+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd840686f0 con 0x7fdd840b4c70 2024-01-15T17:58:14.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.977+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdd7400bd10 con 0x7fdd840b4c70 2024-01-15T17:58:14.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.978+0000 7fdd82000640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd5c06b2c0 0x7fdd5c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:14.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.978+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd7408a180 con 0x7fdd840b4c70 2024-01-15T17:58:14.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.979+0000 7fdd89600640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd5c06b2c0 0x7fdd5c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:14.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.980+0000 7fdd89600640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd5c06b2c0 0x7fdd5c06d780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd840b1990 tx=0x7fdd6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:14.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:14.983+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd74054fe0 con 0x7fdd840b4c70 2024-01-15T17:58:15.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.175+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd84067760 con 0x7fdd840b4c70 2024-01-15T17:58:15.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.176+0000 7fdd82000640 1 -- 172.21.15.2:0/2403855614 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd74058600 con 0x7fdd840b4c70 2024-01-15T17:58:15.179 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:15.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd5c06b2c0 msgr2=0x7fdd5c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd5c06b2c0 0x7fdd5c06d780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdd840b1990 tx=0x7fdd6c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 msgr2=0x7fdd841f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 0x7fdd841f6b00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400d800 tx=0x7fdd7400dcc0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 shutdown_connections 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd5c06b2c0 0x7fdd5c06d780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd8415d730 0x7fdd841f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd840b8c30 0x7fdd8415b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 --2- 172.21.15.2:0/2403855614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd840b4c70 0x7fdd841f6b00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 >> 172.21.15.2:0/2403855614 conn(0x7fdd84072210 msgr2=0x7fdd840b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:15.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 shutdown_connections 2024-01-15T17:58:15.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.179+0000 7fdd8ca00640 1 -- 172.21.15.2:0/2403855614 wait complete. 2024-01-15T17:58:15.200 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-15T17:58:15.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.410+0000 7fb38be00640 1 Processor -- start 2024-01-15T17:58:15.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.410+0000 7fb38be00640 1 -- start start 2024-01-15T17:58:15.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.412+0000 7fb38be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb38c0d3090 0x7fb38c0d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:15.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.412+0000 7fb38be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:15.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.412+0000 7fb38be00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38c0d3a60 con 0x7fb38c15d340 2024-01-15T17:58:15.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.412+0000 7fb38be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38c0ce430 con 0x7fb38c0d3090 2024-01-15T17:58:15.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.412+0000 7fb38be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0b3de0 con 0x7fb38c0d1680 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.412+0000 7fb38aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb38c15d340 0x7fb38c0cdd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35832/0 (socket says 172.21.15.2:35832) 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38aa00640 1 -- 172.21.15.2:0/1637935446 learned_addr learned my addr 172.21.15.2:0/1637935446 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38b400640 1 --2- 172.21.15.2:0/1637935446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb383e00640 1 --2- 172.21.15.2:0/1637935446 >> v2:172.21.15.137:3300/0 conn(0x7fb38c0d3090 0x7fb38c0d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38a000640 1 -- 172.21.15.2:0/1637935446 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060372949 0 0) 0x7fb38c0d3a60 con 0x7fb38c15d340 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38a000640 1 -- 172.21.15.2:0/1637935446 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb36c003520 con 0x7fb38c15d340 2024-01-15T17:58:15.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38b400640 1 -- 172.21.15.2:0/1637935446 >> v2:172.21.15.137:3300/0 conn(0x7fb38c0d3090 msgr2=0x7fb38c0d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38b400640 1 --2- 172.21.15.2:0/1637935446 >> v2:172.21.15.137:3300/0 conn(0x7fb38c0d3090 0x7fb38c0d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38b400640 1 -- 172.21.15.2:0/1637935446 >> v1:172.21.15.2:6789/0 conn(0x7fb38c15d340 legacy=0x7fb38c0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.413+0000 7fb38b400640 1 -- 172.21.15.2:0/1637935446 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0b40c0 con 0x7fb38c0d1680 2024-01-15T17:58:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.415+0000 7fb38b400640 1 --2- 172.21.15.2:0/1637935446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c0d1b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb384009d90 tx=0x7fb38402beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=247444f588e63828 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:15.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.415+0000 7fb38a000640 1 -- 172.21.15.2:0/1637935446 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38402cc60 con 0x7fb38c0d1680 2024-01-15T17:58:15.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.415+0000 7fb38a000640 1 -- 172.21.15.2:0/1637935446 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38402cdc0 con 0x7fb38c0d1680 2024-01-15T17:58:15.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.415+0000 7fb38a000640 1 -- 172.21.15.2:0/1637935446 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384034660 con 0x7fb38c0d1680 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 -- 172.21.15.2:0/1637935446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 msgr2=0x7fb38c0d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 --2- 172.21.15.2:0/1637935446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c0d1b10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb384009d90 tx=0x7fb38402beb0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 -- 172.21.15.2:0/1637935446 shutdown_connections 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 --2- 172.21.15.2:0/1637935446 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c0d1b10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 --2- 172.21.15.2:0/1637935446 >> v2:172.21.15.137:3300/0 conn(0x7fb38c0d3090 0x7fb38c0d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 -- 172.21.15.2:0/1637935446 >> 172.21.15.2:0/1637935446 conn(0x7fb38c08caf0 msgr2=0x7fb38c09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 -- 172.21.15.2:0/1637935446 shutdown_connections 2024-01-15T17:58:15.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.416+0000 7fb38be00640 1 -- 172.21.15.2:0/1637935446 wait complete. 2024-01-15T17:58:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.419+0000 7fb38be00640 1 Processor -- start 2024-01-15T17:58:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.419+0000 7fb38be00640 1 -- start start 2024-01-15T17:58:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.420+0000 7fb38be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.420+0000 7fb38be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb38c0d3090 0x7fb38c15ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.420+0000 7fb38be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 0x7fb38c0c9120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.420+0000 7fb38be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c0c6cd0 con 0x7fb38c15d340 2024-01-15T17:58:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.420+0000 7fb38be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c15f930 con 0x7fb38c0d3090 2024-01-15T17:58:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.420+0000 7fb38be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38c15fc10 con 0x7fb38c0d1680 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 0x7fb38c0c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 0x7fb38c0c9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49514/0 (socket says 172.21.15.2:49514) 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 -- 172.21.15.2:0/1404218367 learned_addr learned my addr 172.21.15.2:0/1404218367 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb383e00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb38c0d3090 0x7fb38c15ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38aa00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 -- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 msgr2=0x7fb38c156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 -- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb38c0d3090 msgr2=0x7fb38c15ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb38c0d3090 0x7fb38c15ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.421+0000 7fb38b400640 1 -- 172.21.15.2:0/1404218367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb384009a70 con 0x7fb38c15d340 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.422+0000 7fb38b400640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 0x7fb38c0c9120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb384009d90 tx=0x7fb384004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.422+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384040650 con 0x7fb38c15d340 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.422+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3840407b0 con 0x7fb38c15d340 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.422+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384040b60 con 0x7fb38c15d340 2024-01-15T17:58:15.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.422+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c15fef0 con 0x7fb38c15d340 2024-01-15T17:58:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.422+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb38c156d40 con 0x7fb38c15d340 2024-01-15T17:58:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.423+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb38c0c6e10 con 0x7fb38c15d340 2024-01-15T17:58:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.424+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb384040cc0 con 0x7fb38c15d340 2024-01-15T17:58:15.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.425+0000 7fb383400640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb36406b2c0 0x7fb36406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:15.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.425+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3840ac760 con 0x7fb38c15d340 2024-01-15T17:58:15.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.426+0000 7fb38aa00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb36406b2c0 0x7fb36406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:15.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.427+0000 7fb38aa00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb36406b2c0 0x7fb36406d780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0011e0 tx=0x7fb37c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.429+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb38c0c6e10 con 0x7fb38c15d340 2024-01-15T17:58:15.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:15.686+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7fb38c0c6e10 con 0x7fb38c15d340 2024-01-15T17:58:16.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.502+0000 7fb383400640 1 -- 172.21.15.2:0/1404218367 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v57) v1 ==== 155+0+0 (secure 0 0 0) 0x7fb38c0c6e10 con 0x7fb38c15d340 2024-01-15T17:58:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb36406b2c0 msgr2=0x7fb36406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb36406b2c0 0x7fb36406d780 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0011e0 tx=0x7fb37c007d30 comp rx=0 tx=0).stop 2024-01-15T17:58:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 msgr2=0x7fb38c0c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 0x7fb38c0c9120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb384009d90 tx=0x7fb384004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 shutdown_connections 2024-01-15T17:58:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb36406b2c0 0x7fb36406d780 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb38c15d340 0x7fb38c0c9120 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb38c0d3090 0x7fb38c15ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 --2- 172.21.15.2:0/1404218367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb38c0d1680 0x7fb38c156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 >> 172.21.15.2:0/1404218367 conn(0x7fb38c08caf0 msgr2=0x7fb38c0b4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 shutdown_connections 2024-01-15T17:58:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.506+0000 7fb38be00640 1 -- 172.21.15.2:0/1404218367 wait complete. 2024-01-15T17:58:16.531 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T17:58:16.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.754+0000 7f5a5ca00640 1 Processor -- start 2024-01-15T17:58:16.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.754+0000 7f5a5ca00640 1 -- start start 2024-01-15T17:58:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.755+0000 7f5a5ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580b1520 0x7f5a58168110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.755+0000 7f5a5ca00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a580b3780 con 0x7f5a581608c0 2024-01-15T17:58:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.755+0000 7f5a5ca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a580b1ef0 con 0x7f5a580d3ca0 2024-01-15T17:58:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.755+0000 7f5a5ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580b21b0 con 0x7f5a580b1520 2024-01-15T17:58:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.755+0000 7f5a57400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5a581608c0 0x7f5a580b2960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35842/0 (socket says 172.21.15.2:35842) 2024-01-15T17:58:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.755+0000 7f5a57400640 1 -- 172.21.15.2:0/887341286 learned_addr learned my addr 172.21.15.2:0/887341286 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a57e00640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580b1520 0x7f5a58168110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790632854 0 0) 0x7f5a580b3780 con 0x7f5a581608c0 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a38003520 con 0x7f5a581608c0 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856503529 0 0) 0x7f5a38003520 con 0x7f5a581608c0 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580b1520 msgr2=0x7f5a58168110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580b1520 0x7f5a58168110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 >> v1:172.21.15.137:6789/0 conn(0x7f5a580d3ca0 legacy=0x7f5a580b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580b2490 con 0x7f5a581608c0 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5a480019f0 con 0x7f5a581608c0 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 >> v1:172.21.15.2:6789/0 conn(0x7f5a581608c0 legacy=0x7f5a580b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a380039a0 0x7f5a38003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a380047c0 0x7f5a38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a38005180 0x7f5a3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580b21b0 con 0x7f5a380039a0 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3800c050 con 0x7f5a38005180 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3800c210 con 0x7f5a380047c0 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.756+0000 7f5a57e00640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580b1520 0x7f5a58168110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.757+0000 7f5a57400640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a380039a0 0x7f5a38003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.757+0000 7f5a57400640 1 -- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a380047c0 msgr2=0x7f5a38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.757+0000 7f5a57400640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a380047c0 0x7f5a38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.757+0000 7f5a57400640 1 -- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a38005180 msgr2=0x7f5a3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.757+0000 7f5a57400640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a38005180 0x7f5a3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.757+0000 7f5a57400640 1 -- 172.21.15.2:0/887341286 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a38003520 con 0x7f5a380039a0 2024-01-15T17:58:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.758+0000 7f5a57400640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a380039a0 0x7f5a38003dc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800a830 tx=0x7f5a380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4843bdfb8dcdedec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a480072d0 con 0x7f5a380039a0 2024-01-15T17:58:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4802f070 con 0x7f5a380039a0 2024-01-15T17:58:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a56000640 1 -- 172.21.15.2:0/887341286 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a480076b0 con 0x7f5a380039a0 2024-01-15T17:58:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a5ca00640 1 -- 172.21.15.2:0/887341286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a380039a0 msgr2=0x7f5a38003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/887341286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a380039a0 0x7f5a38003dc0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800a830 tx=0x7f5a380057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a57e00640 1 -- 172.21.15.2:0/887341286 reap_dead start 2024-01-15T17:58:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a5ca00640 1 -- 172.21.15.2:0/887341286 shutdown_connections 2024-01-15T17:58:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a5ca00640 1 -- 172.21.15.2:0/887341286 >> 172.21.15.2:0/887341286 conn(0x7f5a5809c3b0 msgr2=0x7f5a580d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a5ca00640 1 -- 172.21.15.2:0/887341286 shutdown_connections 2024-01-15T17:58:16.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.759+0000 7f5a5ca00640 1 -- 172.21.15.2:0/887341286 wait complete. 2024-01-15T17:58:16.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.761+0000 7f5a5ca00640 1 Processor -- start 2024-01-15T17:58:16.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.761+0000 7f5a5ca00640 1 -- start start 2024-01-15T17:58:16.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a5ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a380039a0 0x7f5a58149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a5ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580ca190 0x7f5a5814c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a5ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 0x7f5a580cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a5ca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5814ccd0 con 0x7f5a5814c820 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a5ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580cdc70 con 0x7f5a380039a0 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a5ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a580c6da0 con 0x7f5a580ca190 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a57e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 0x7f5a580cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.762+0000 7f5a57e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 0x7f5a580cbaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49534/0 (socket says 172.21.15.2:49534) 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 -- 172.21.15.2:0/1608319029 learned_addr learned my addr 172.21.15.2:0/1608319029 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57400640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a380039a0 0x7f5a58149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a56a00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580ca190 0x7f5a5814c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 -- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580ca190 msgr2=0x7f5a5814c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580ca190 0x7f5a5814c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 -- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a380039a0 msgr2=0x7f5a58149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a380039a0 0x7f5a58149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 -- 172.21.15.2:0/1608319029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a580c9480 con 0x7f5a5814c820 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57400640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a380039a0 0x7f5a58149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a57e00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 0x7f5a580cbaf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5a50006fd0 tx=0x7f5a5000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:16.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.763+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a5000c070 con 0x7f5a5814c820 2024-01-15T17:58:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.764+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a50004230 con 0x7f5a5814c820 2024-01-15T17:58:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.764+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a580c67e0 con 0x7f5a5814c820 2024-01-15T17:58:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.764+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a50003610 con 0x7f5a5814c820 2024-01-15T17:58:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.764+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a580ca610 con 0x7f5a5814c820 2024-01-15T17:58:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.765+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a58074770 con 0x7f5a5814c820 2024-01-15T17:58:16.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.766+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a50003770 con 0x7f5a5814c820 2024-01-15T17:58:16.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.766+0000 7f5a4f400640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a3006b2f0 0x7f5a3006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:16.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.766+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5a50088960 con 0x7f5a5814c820 2024-01-15T17:58:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.767+0000 7f5a57400640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a3006b2f0 0x7f5a3006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:16.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.768+0000 7f5a57400640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a3006b2f0 0x7f5a3006d7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800a830 tx=0x7f5a48007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:16.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.770+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a500537c0 con 0x7f5a5814c820 2024-01-15T17:58:16.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.955+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5a580d1ed0 con 0x7f5a5814c820 2024-01-15T17:58:16.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.960+0000 7f5a4f400640 1 -- 172.21.15.2:0/1608319029 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5a50056de0 con 0x7f5a5814c820 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.962+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a3006b2f0 msgr2=0x7f5a3006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.962+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a3006b2f0 0x7f5a3006d7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800a830 tx=0x7f5a48007000 comp rx=0 tx=0).stop 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 msgr2=0x7f5a580cbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 0x7f5a580cbaf0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5a50006fd0 tx=0x7f5a5000e830 comp rx=0 tx=0).stop 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 shutdown_connections 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a3006b2f0 0x7f5a3006d7b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a5814c820 0x7f5a580cbaf0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a580ca190 0x7f5a5814c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 --2- 172.21.15.2:0/1608319029 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a380039a0 0x7f5a58149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:16.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 >> 172.21.15.2:0/1608319029 conn(0x7f5a5809c3b0 msgr2=0x7f5a5815d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:16.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 shutdown_connections 2024-01-15T17:58:16.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:16.963+0000 7f5a5ca00640 1 -- 172.21.15.2:0/1608319029 wait complete. 2024-01-15T17:58:16.984 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T17:58:16.985 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T17:58:17.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T17:58:17.230 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T17:58:17.230 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:58:17.230 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:17.290 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:58:17.290 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:17.346 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:17.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.556+0000 7f939be00640 1 Processor -- start 2024-01-15T17:58:17.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.556+0000 7f939be00640 1 -- start start 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c15ec60 0x7f939c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f939c06cad0 0x7f939c06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f939c0716d0 con 0x7f939c141880 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939be00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f939c071f20 con 0x7f939c06cad0 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939c06c6b0 con 0x7f939c15ec60 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f939c06cad0 0x7f939c06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f939c06cad0 0x7f939c06cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49550/0 (socket says 172.21.15.2:49550) 2024-01-15T17:58:17.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.558+0000 7f939b400640 1 -- 172.21.15.2:0/2487225046 learned_addr learned my addr 172.21.15.2:0/2487225046 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:17.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.559+0000 7f939b400640 1 -- 172.21.15.2:0/2487225046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c15ec60 msgr2=0x7f939c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:17.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.559+0000 7f939b400640 1 --2- 172.21.15.2:0/2487225046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c15ec60 0x7f939c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.559+0000 7f939b400640 1 -- 172.21.15.2:0/2487225046 >> v1:172.21.15.137:6789/0 conn(0x7f939c141880 legacy=0x7f939c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:17.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.559+0000 7f939b400640 1 -- 172.21.15.2:0/2487225046 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c071960 con 0x7f939c06cad0 2024-01-15T17:58:17.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.560+0000 7f939b400640 1 --2- 172.21.15.2:0/2487225046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c06cad0 0x7f939c06cf60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9394009e50 tx=0x7f939402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7274dab0e1b187d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:17.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.560+0000 7f939a000640 1 -- 172.21.15.2:0/2487225046 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939402c9f0 con 0x7f939c06cad0 2024-01-15T17:58:17.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.560+0000 7f939a000640 1 -- 172.21.15.2:0/2487225046 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939402cb50 con 0x7f939c06cad0 2024-01-15T17:58:17.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.560+0000 7f939a000640 1 -- 172.21.15.2:0/2487225046 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93940343f0 con 0x7f939c06cad0 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 -- 172.21.15.2:0/2487225046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c06cad0 msgr2=0x7f939c06cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 --2- 172.21.15.2:0/2487225046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c06cad0 0x7f939c06cf60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9394009e50 tx=0x7f939402be50 comp rx=0 tx=0).stop 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 -- 172.21.15.2:0/2487225046 shutdown_connections 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 --2- 172.21.15.2:0/2487225046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c06cad0 0x7f939c06cf60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 --2- 172.21.15.2:0/2487225046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c15ec60 0x7f939c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 -- 172.21.15.2:0/2487225046 >> 172.21.15.2:0/2487225046 conn(0x7f939c0b0840 msgr2=0x7f939c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 -- 172.21.15.2:0/2487225046 shutdown_connections 2024-01-15T17:58:17.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.561+0000 7f939be00640 1 -- 172.21.15.2:0/2487225046 wait complete. 2024-01-15T17:58:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.563+0000 7f939be00640 1 Processor -- start 2024-01-15T17:58:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.563+0000 7f939be00640 1 -- start start 2024-01-15T17:58:17.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c06cad0 0x7f939c1f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:17.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c141880 0x7f939c13e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:17.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 0x7f939c0b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:17.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c1f71b0 con 0x7f939c15ec60 2024-01-15T17:58:17.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0b1d50 con 0x7f939c141880 2024-01-15T17:58:17.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939c0b2030 con 0x7f939c06cad0 2024-01-15T17:58:17.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 0x7f939c0b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:17.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 0x7f939c0b2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49558/0 (socket says 172.21.15.2:49558) 2024-01-15T17:58:17.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.564+0000 7f939b400640 1 -- 172.21.15.2:0/3647142612 learned_addr learned my addr 172.21.15.2:0/3647142612 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:17.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f9393e00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c141880 0x7f939c13e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939b400640 1 -- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c06cad0 msgr2=0x7f939c1f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939aa00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c06cad0 0x7f939c1f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939b400640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c06cad0 0x7f939c1f6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939b400640 1 -- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c141880 msgr2=0x7f939c13e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939b400640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c141880 0x7f939c13e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939b400640 1 -- 172.21.15.2:0/3647142612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394009b30 con 0x7f939c15ec60 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f9393e00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c141880 0x7f939c13e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939b400640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 0x7f939c0b2df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9394036040 tx=0x7f9394004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939403e450 con 0x7f939c15ec60 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939403e5d0 con 0x7f939c15ec60 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939403e980 con 0x7f939c15ec60 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.565+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939c0b2310 con 0x7f939c15ec60 2024-01-15T17:58:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.566+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f939c0b28d0 con 0x7f939c15ec60 2024-01-15T17:58:17.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.567+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939c0b2ac0 con 0x7f939c15ec60 2024-01-15T17:58:17.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.568+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f939403eae0 con 0x7f939c15ec60 2024-01-15T17:58:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.568+0000 7f9392a00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f937406b2c0 0x7f937406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.569+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93940acb30 con 0x7f939c15ec60 2024-01-15T17:58:17.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.569+0000 7f939aa00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f937406b2c0 0x7f937406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.570+0000 7f939aa00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f937406b2c0 0x7f937406d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f938c0015d0 tx=0x7f938c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:17.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.573+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9394077900 con 0x7f939c15ec60 2024-01-15T17:58:17.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.770+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f939c074770 con 0x7f939c15ec60 2024-01-15T17:58:17.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.773+0000 7f9392a00640 1 -- 172.21.15.2:0/3647142612 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16445 (secure 0 0 0) 0x7f939407af20 con 0x7f939c15ec60 2024-01-15T17:58:17.776 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:17.776 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":22,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:57:33.127110+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f937406b2c0 msgr2=0x7f937406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f937406b2c0 0x7f937406d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f938c0015d0 tx=0x7f938c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 msgr2=0x7f939c0b2df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 0x7f939c0b2df0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9394036040 tx=0x7f9394004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 shutdown_connections 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f937406b2c0 0x7f937406d780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f939c15ec60 0x7f939c0b2df0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f939c141880 0x7f939c13e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 --2- 172.21.15.2:0/3647142612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f939c06cad0 0x7f939c1f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:17.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 >> 172.21.15.2:0/3647142612 conn(0x7f939c0b0840 msgr2=0x7f939c1408c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:17.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 shutdown_connections 2024-01-15T17:58:17.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:17.776+0000 7f939be00640 1 -- 172.21.15.2:0/3647142612 wait complete. 2024-01-15T17:58:17.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:17.799 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:18.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.021+0000 7f1174c00640 1 Processor -- start 2024-01-15T17:58:18.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.021+0000 7f1174c00640 1 -- start start 2024-01-15T17:58:18.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f1174c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1170144c00 0x7f11700cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f1174c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11700ce230 con 0x7f11700cdd70 2024-01-15T17:58:18.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f1174c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11700ce510 con 0x7f117006d000 2024-01-15T17:58:18.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f1174c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11700ce7f0 con 0x7f1170144c00 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f1165200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f11700cdd70 0x7f117006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35852/0 (socket says 172.21.15.2:35852) 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f1165200640 1 -- 172.21.15.2:0/3707968040 learned_addr learned my addr 172.21.15.2:0/3707968040 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.023+0000 7f116de00640 1 --2- 172.21.15.2:0/3707968040 >> v2:172.21.15.137:3300/0 conn(0x7f1170144c00 0x7f11700cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509210376 0 0) 0x7f11700ce230 con 0x7f11700cdd70 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150003520 con 0x7f11700cdd70 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202236306 0 0) 0x7f1150003520 con 0x7f11700cdd70 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 >> v2:172.21.15.137:3300/0 conn(0x7f1170144c00 msgr2=0x7f11700cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 --2- 172.21.15.2:0/3707968040 >> v2:172.21.15.137:3300/0 conn(0x7f1170144c00 0x7f11700cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 >> v1:172.21.15.137:6789/0 conn(0x7f117006d000 legacy=0x7f1170070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11700705f0 con 0x7f11700cdd70 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1158001670 con 0x7f11700cdd70 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 >> v1:172.21.15.2:6789/0 conn(0x7f11700cdd70 legacy=0x7f117006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11500039c0 0x7f1150003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11500047c0 0x7f1150004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1150005180 0x7f115000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11700ce7f0 con 0x7f1150005180 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115000c080 con 0x7f11500047c0 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115000c240 con 0x7f11500039c0 2024-01-15T17:58:18.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.024+0000 7f116e800640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1150005180 0x7f115000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f116de00640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11500039c0 0x7f1150003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f1165200640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11500047c0 0x7f1150004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f116e800640 1 -- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11500039c0 msgr2=0x7f1150003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f116e800640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11500039c0 0x7f1150003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f116e800640 1 -- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11500047c0 msgr2=0x7f1150004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f116e800640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11500047c0 0x7f1150004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f116e800640 1 -- 172.21.15.2:0/3707968040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1150003520 con 0x7f1150005180 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.025+0000 7f1165200640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f11500047c0 0x7f1150004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.026+0000 7f116e800640 1 -- 172.21.15.2:0/3707968040 reap_dead start 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.026+0000 7f116e800640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1150005180 0x7f115000b980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1168001720 tx=0x7f11500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad17c4d4329a2005 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.026+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680052d0 con 0x7f1150005180 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.026+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116802e050 con 0x7f1150005180 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.026+0000 7f116d400640 1 -- 172.21.15.2:0/3707968040 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680035f0 con 0x7f1150005180 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 -- 172.21.15.2:0/3707968040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1150005180 msgr2=0x7f115000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1150005180 0x7f115000b980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f1168001720 tx=0x7f11500057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 -- 172.21.15.2:0/3707968040 shutdown_connections 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 --2- 172.21.15.2:0/3707968040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1150005180 0x7f115000b980 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 -- 172.21.15.2:0/3707968040 >> 172.21.15.2:0/3707968040 conn(0x7f11700b45f0 msgr2=0x7f11700b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 -- 172.21.15.2:0/3707968040 shutdown_connections 2024-01-15T17:58:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.027+0000 7f1174c00640 1 -- 172.21.15.2:0/3707968040 wait complete. 2024-01-15T17:58:18.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.029+0000 7f1174c00640 1 Processor -- start 2024-01-15T17:58:18.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.029+0000 7f1174c00640 1 -- start start 2024-01-15T17:58:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f1174c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1150005180 0x7f11701e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f1174c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11701e5f30 0x7f117013b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f1174c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 0x7f117013bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f1174c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701f3e60 con 0x7f1170139be0 2024-01-15T17:58:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f1174c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117013b3a0 con 0x7f1150005180 2024-01-15T17:58:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f1174c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11700b28b0 con 0x7f11701e5f30 2024-01-15T17:58:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f116e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 0x7f117013bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f116e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 0x7f117013bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49576/0 (socket says 172.21.15.2:49576) 2024-01-15T17:58:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.030+0000 7f116e800640 1 -- 172.21.15.2:0/566610554 learned_addr learned my addr 172.21.15.2:0/566610554 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f116e800640 1 -- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11701e5f30 msgr2=0x7f117013b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f116e800640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11701e5f30 0x7f117013b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f116e800640 1 -- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1150005180 msgr2=0x7f11701e59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f116e800640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1150005180 0x7f11701e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f116e800640 1 -- 172.21.15.2:0/566610554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11700b2b90 con 0x7f1170139be0 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f116e800640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 0x7f117013bb40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1168001840 tx=0x7f1168001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.031+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168007a30 con 0x7f1170139be0 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.032+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1168038040 con 0x7f1170139be0 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.032+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1168007250 con 0x7f1170139be0 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.032+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11700b2e70 con 0x7f1170139be0 2024-01-15T17:58:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.032+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11700b3670 con 0x7f1170139be0 2024-01-15T17:58:18.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.033+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11700b3390 con 0x7f1170139be0 2024-01-15T17:58:18.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.034+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f11680073b0 con 0x7f1170139be0 2024-01-15T17:58:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.035+0000 7f1166a00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f114806b2f0 0x7f114806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.035+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11680aa820 con 0x7f1170139be0 2024-01-15T17:58:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.036+0000 7f116de00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f114806b2f0 0x7f114806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.037+0000 7f116de00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f114806b2f0 0x7f114806d7b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1160007440 tx=0x7f1160009750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.039+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f11700b3390 con 0x7f1170139be0 2024-01-15T17:58:18.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.234+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11700b3390 con 0x7f1170139be0 2024-01-15T17:58:18.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.235+0000 7f1166a00640 1 -- 172.21.15.2:0/566610554 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11700b3390 con 0x7f1170139be0 2024-01-15T17:58:18.238 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f114806b2f0 msgr2=0x7f114806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f114806b2f0 0x7f114806d7b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1160007440 tx=0x7f1160009750 comp rx=0 tx=0).stop 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 msgr2=0x7f117013bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 0x7f117013bb40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1168001840 tx=0x7f1168001120 comp rx=0 tx=0).stop 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 shutdown_connections 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f114806b2f0 0x7f114806d7b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1170139be0 0x7f117013bb40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f11701e5f30 0x7f117013b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 --2- 172.21.15.2:0/566610554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1150005180 0x7f11701e59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 >> 172.21.15.2:0/566610554 conn(0x7f11700b45f0 msgr2=0x7f11701438f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:18.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.238+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 shutdown_connections 2024-01-15T17:58:18.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.239+0000 7f1174c00640 1 -- 172.21.15.2:0/566610554 wait complete. 2024-01-15T17:58:18.260 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:18.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.482+0000 7f8e5f600640 1 Processor -- start 2024-01-15T17:58:18.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.482+0000 7f8e5f600640 1 -- start start 2024-01-15T17:58:18.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.483+0000 7f8e5f600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8e580b4c70 0x7f8e580d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5f600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e580b8c30 0x7f8e580d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5f600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e580d1900 con 0x7f8e5815d730 2024-01-15T17:58:18.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5f600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e580d1be0 con 0x7f8e580b8c30 2024-01-15T17:58:18.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5f600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e580d1ec0 con 0x7f8e580b4c70 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e580b8c30 0x7f8e580d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e580b8c30 0x7f8e580d29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49578/0 (socket says 172.21.15.2:49578) 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 -- 172.21.15.2:0/4173047661 learned_addr learned my addr 172.21.15.2:0/4173047661 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 -- 172.21.15.2:0/4173047661 >> v2:172.21.15.137:3300/0 conn(0x7f8e580b4c70 msgr2=0x7f8e580d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 --2- 172.21.15.2:0/4173047661 >> v2:172.21.15.137:3300/0 conn(0x7f8e580b4c70 0x7f8e580d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 -- 172.21.15.2:0/4173047661 >> v1:172.21.15.137:6789/0 conn(0x7f8e5815d730 legacy=0x7f8e580b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.484+0000 7f8e5d600640 1 -- 172.21.15.2:0/4173047661 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e580d21a0 con 0x7f8e580b8c30 2024-01-15T17:58:18.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.486+0000 7f8e5d600640 1 --2- 172.21.15.2:0/4173047661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e580d29a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e580b5c40 tx=0x7f8e5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a51241fc0f13e4f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.486+0000 7f8e57e00640 1 -- 172.21.15.2:0/4173047661 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5002ca80 con 0x7f8e580b8c30 2024-01-15T17:58:18.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.486+0000 7f8e57e00640 1 -- 172.21.15.2:0/4173047661 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5002cbe0 con 0x7f8e580b8c30 2024-01-15T17:58:18.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.486+0000 7f8e57e00640 1 -- 172.21.15.2:0/4173047661 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e500343f0 con 0x7f8e580b8c30 2024-01-15T17:58:18.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.486+0000 7f8e5f600640 1 -- 172.21.15.2:0/4173047661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 msgr2=0x7f8e580d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.486+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4173047661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e580d29a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8e580b5c40 tx=0x7f8e5002bee0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.487+0000 7f8e5f600640 1 -- 172.21.15.2:0/4173047661 shutdown_connections 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.487+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4173047661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e580d29a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.487+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4173047661 >> v2:172.21.15.137:3300/0 conn(0x7f8e580b4c70 0x7f8e580d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.487+0000 7f8e5f600640 1 -- 172.21.15.2:0/4173047661 >> 172.21.15.2:0/4173047661 conn(0x7f8e58072210 msgr2=0x7f8e580cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.487+0000 7f8e5f600640 1 -- 172.21.15.2:0/4173047661 shutdown_connections 2024-01-15T17:58:18.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.487+0000 7f8e5f600640 1 -- 172.21.15.2:0/4173047661 wait complete. 2024-01-15T17:58:18.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.488+0000 7f8e5f600640 1 Processor -- start 2024-01-15T17:58:18.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.489+0000 7f8e5f600640 1 -- start start 2024-01-15T17:58:18.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.489+0000 7f8e5f600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e580b4c70 0x7f8e580c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.489+0000 7f8e5f600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e581605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e5f600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e5815d730 0x7f8e580cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e5f600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5815db60 con 0x7f8e580b8c30 2024-01-15T17:58:18.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e5f600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e580c7ec0 con 0x7f8e580b4c70 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e5f600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5815ddb0 con 0x7f8e5815d730 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e4fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e581605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e4fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e581605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49588/0 (socket says 172.21.15.2:49588) 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e4fe00640 1 -- 172.21.15.2:0/4110120829 learned_addr learned my addr 172.21.15.2:0/4110120829 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e5d600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e5815d730 0x7f8e580cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e5cc00640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e580b4c70 0x7f8e580c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.490+0000 7f8e4fe00640 1 -- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e5815d730 msgr2=0x7f8e580cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e4fe00640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e5815d730 0x7f8e580cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e4fe00640 1 -- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e580b4c70 msgr2=0x7f8e580c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e4fe00640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e580b4c70 0x7f8e580c77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e4fe00640 1 -- 172.21.15.2:0/4110120829 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50009b30 con 0x7f8e580b8c30 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e4fe00640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e581605c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000b3c0 tx=0x7f8e4000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40011830 con 0x7f8e580b8c30 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e40011990 con 0x7f8e580b8c30 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e580d3060 con 0x7f8e580b8c30 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e40011d80 con 0x7f8e580b8c30 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.491+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5815e370 con 0x7f8e580b8c30 2024-01-15T17:58:18.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.492+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5815e590 con 0x7f8e580b8c30 2024-01-15T17:58:18.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.493+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8e4001b460 con 0x7f8e580b8c30 2024-01-15T17:58:18.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.494+0000 7f8e55600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e3006b2c0 0x7f8e3006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.494+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8e400897e0 con 0x7f8e580b8c30 2024-01-15T17:58:18.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.495+0000 7f8e5cc00640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e3006b2c0 0x7f8e3006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.496+0000 7f8e5cc00640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e3006b2c0 0x7f8e3006d780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009890 tx=0x7f8e48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.498+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e40054530 con 0x7f8e580b8c30 2024-01-15T17:58:18.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.693+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8e580686f0 con 0x7f8e580b8c30 2024-01-15T17:58:18.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.694+0000 7f8e55600640 1 -- 172.21.15.2:0/4110120829 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8e40057b50 con 0x7f8e580b8c30 2024-01-15T17:58:18.697 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e3006b2c0 msgr2=0x7f8e3006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e3006b2c0 0x7f8e3006d780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8e48009890 tx=0x7f8e48008040 comp rx=0 tx=0).stop 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 msgr2=0x7f8e581605c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e581605c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000b3c0 tx=0x7f8e4000b880 comp rx=0 tx=0).stop 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 shutdown_connections 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e3006b2c0 0x7f8e3006d780 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e5815d730 0x7f8e580cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e580b8c30 0x7f8e581605c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 --2- 172.21.15.2:0/4110120829 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e580b4c70 0x7f8e580c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 >> 172.21.15.2:0/4110120829 conn(0x7f8e58072210 msgr2=0x7f8e580b5050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:18.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 shutdown_connections 2024-01-15T17:58:18.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.697+0000 7f8e5f600640 1 -- 172.21.15.2:0/4110120829 wait complete. 2024-01-15T17:58:18.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:18.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.945+0000 7f9d99e00640 1 Processor -- start 2024-01-15T17:58:18.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.946+0000 7f9d99e00640 1 -- start start 2024-01-15T17:58:18.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.947+0000 7f9d99e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9d940d1870 0x7f9d940d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.947+0000 7f9d99e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d940b2210 0x7f9d9415d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.947+0000 7f9d99e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d940d1250 con 0x7f9d940d1c50 2024-01-15T17:58:18.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.947+0000 7f9d99e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940ce6d0 con 0x7f9d940b2210 2024-01-15T17:58:18.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.947+0000 7f9d99e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d940ce8e0 con 0x7f9d940d1870 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d940b2210 0x7f9d9415d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d940b2210 0x7f9d9415d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49600/0 (socket says 172.21.15.2:49600) 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 -- 172.21.15.2:0/1405801034 learned_addr learned my addr 172.21.15.2:0/1405801034 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d98a00640 1 --2- 172.21.15.2:0/1405801034 >> v2:172.21.15.137:3300/0 conn(0x7f9d940d1870 0x7f9d940d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 -- 172.21.15.2:0/1405801034 >> v2:172.21.15.137:3300/0 conn(0x7f9d940d1870 msgr2=0x7f9d940d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 --2- 172.21.15.2:0/1405801034 >> v2:172.21.15.137:3300/0 conn(0x7f9d940d1870 0x7f9d940d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 -- 172.21.15.2:0/1405801034 >> v1:172.21.15.137:6789/0 conn(0x7f9d940d1c50 legacy=0x7f9d940cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:18.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.948+0000 7f9d93e00640 1 -- 172.21.15.2:0/1405801034 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940cebc0 con 0x7f9d940b2210 2024-01-15T17:58:18.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.949+0000 7f9d93e00640 1 --2- 172.21.15.2:0/1405801034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940b2210 0x7f9d9415d0e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9d94067890 tx=0x7f9d7c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd12dc4fce3ceb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.950+0000 7f9d93400640 1 -- 172.21.15.2:0/1405801034 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c031870 con 0x7f9d940b2210 2024-01-15T17:58:18.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.950+0000 7f9d93400640 1 -- 172.21.15.2:0/1405801034 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c0319d0 con 0x7f9d940b2210 2024-01-15T17:58:18.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.950+0000 7f9d93400640 1 -- 172.21.15.2:0/1405801034 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c031d60 con 0x7f9d940b2210 2024-01-15T17:58:18.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 -- 172.21.15.2:0/1405801034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940b2210 msgr2=0x7f9d9415d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1405801034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940b2210 0x7f9d9415d0e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9d94067890 tx=0x7f9d7c02b910 comp rx=0 tx=0).stop 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 -- 172.21.15.2:0/1405801034 shutdown_connections 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1405801034 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940b2210 0x7f9d9415d0e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1405801034 >> v2:172.21.15.137:3300/0 conn(0x7f9d940d1870 0x7f9d940d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 -- 172.21.15.2:0/1405801034 >> 172.21.15.2:0/1405801034 conn(0x7f9d940d30b0 msgr2=0x7f9d940d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 -- 172.21.15.2:0/1405801034 shutdown_connections 2024-01-15T17:58:18.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.951+0000 7f9d99e00640 1 -- 172.21.15.2:0/1405801034 wait complete. 2024-01-15T17:58:18.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.953+0000 7f9d99e00640 1 Processor -- start 2024-01-15T17:58:18.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.953+0000 7f9d99e00640 1 -- start start 2024-01-15T17:58:18.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d940b2210 0x7f9d94150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 0x7f9d940b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d940d1c50 0x7f9d941f3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d941508a0 con 0x7f9d940d1870 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940b4bd0 con 0x7f9d940d1c50 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d940b4eb0 con 0x7f9d940b2210 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 0x7f9d940b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 0x7f9d940b7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49606/0 (socket says 172.21.15.2:49606) 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 -- 172.21.15.2:0/1511933568 learned_addr learned my addr 172.21.15.2:0/1511933568 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d98a00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d940b2210 0x7f9d94150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d99400640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d940d1c50 0x7f9d941f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 -- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d940b2210 msgr2=0x7f9d94150360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d940b2210 0x7f9d94150360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 -- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d940d1c50 msgr2=0x7f9d941f3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:18.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.954+0000 7f9d93e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d940d1c50 0x7f9d941f3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:18.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d93e00640 1 -- 172.21.15.2:0/1511933568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d7c0094d0 con 0x7f9d940d1870 2024-01-15T17:58:18.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d99400640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d940d1c50 0x7f9d941f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:18.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d93e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 0x7f9d940b7ea0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c000c00 tx=0x7f9d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c03aca0 con 0x7f9d940d1870 2024-01-15T17:58:18.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d7c03ae00 con 0x7f9d940d1870 2024-01-15T17:58:18.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7c044630 con 0x7f9d940d1870 2024-01-15T17:58:18.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.955+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d940b5190 con 0x7f9d940d1870 2024-01-15T17:58:18.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.956+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d940b5750 con 0x7f9d940d1870 2024-01-15T17:58:18.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.957+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d94074770 con 0x7f9d940d1870 2024-01-15T17:58:18.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.958+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9d7c04d450 con 0x7f9d940d1870 2024-01-15T17:58:18.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.959+0000 7f9d90c00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9d6c06b2c0 0x7f9d6c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:18.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.959+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9d7c02d040 con 0x7f9d940d1870 2024-01-15T17:58:18.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.959+0000 7f9d98a00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9d6c06b2c0 0x7f9d6c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:18.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.960+0000 7f9d98a00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9d6c06b2c0 0x7f9d6c06d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d84006fd0 tx=0x7f9d84008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:18.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:18.963+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d7c0770d0 con 0x7f9d940d1870 2024-01-15T17:58:19.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.162+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d940d2e90 con 0x7f9d940d1870 2024-01-15T17:58:19.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.163+0000 7f9d90c00640 1 -- 172.21.15.2:0/1511933568 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9d7c07a6f0 con 0x7f9d940d1870 2024-01-15T17:58:19.166 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:19.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9d6c06b2c0 msgr2=0x7f9d6c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9d6c06b2c0 0x7f9d6c06d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9d84006fd0 tx=0x7f9d84008040 comp rx=0 tx=0).stop 2024-01-15T17:58:19.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 msgr2=0x7f9d940b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 0x7f9d940b7ea0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9d7c000c00 tx=0x7f9d7c0046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 shutdown_connections 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9d6c06b2c0 0x7f9d6c06d780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d940d1c50 0x7f9d941f3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d940d1870 0x7f9d940b7ea0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 --2- 172.21.15.2:0/1511933568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d940b2210 0x7f9d94150360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 >> 172.21.15.2:0/1511933568 conn(0x7f9d940d30b0 msgr2=0x7f9d940b8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 shutdown_connections 2024-01-15T17:58:19.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.167+0000 7f9d99e00640 1 -- 172.21.15.2:0/1511933568 wait complete. 2024-01-15T17:58:19.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:19.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.415+0000 7fc82ce00640 1 Processor -- start 2024-01-15T17:58:19.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.415+0000 7fc82ce00640 1 -- start start 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc82ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 0x7fc82806c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc82ce00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc8280cb620 0x7fc8280cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc82ce00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc828071340 con 0x7fc8280ccb80 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc82ce00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8280cbff0 con 0x7fc8280cb620 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc82ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82806c670 con 0x7fc8280ce1d0 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc81e800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc8280ccb80 0x7fc82815b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35866/0 (socket says 172.21.15.2:35866) 2024-01-15T17:58:19.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc81e800640 1 -- 172.21.15.2:0/3670552321 learned_addr learned my addr 172.21.15.2:0/3670552321 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc827400640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 0x7fc82806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.417+0000 7fc827e00640 1 --2- 172.21.15.2:0/3670552321 >> v2:172.21.15.137:3300/0 conn(0x7fc8280cb620 0x7fc8280cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318718655 0 0) 0x7fc828071340 con 0x7fc8280ccb80 2024-01-15T17:58:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc808003520 con 0x7fc8280ccb80 2024-01-15T17:58:19.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3439617737 0 0) 0x7fc808003520 con 0x7fc8280ccb80 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 msgr2=0x7fc82806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 0x7fc82806c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 >> v2:172.21.15.137:3300/0 conn(0x7fc8280cb620 msgr2=0x7fc8280cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 --2- 172.21.15.2:0/3670552321 >> v2:172.21.15.137:3300/0 conn(0x7fc8280cb620 0x7fc8280cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82815bd50 con 0x7fc8280ccb80 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc8100019f0 con 0x7fc8280ccb80 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 >> v1:172.21.15.2:6789/0 conn(0x7fc8280ccb80 legacy=0x7fc82815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc808003970 0x7fc808003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 0x7fc8080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc808005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8280cbff0 con 0x7fc808004d00 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.418+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82806c670 con 0x7fc808003970 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc80800b980 con 0x7fc8280ce1d0 2024-01-15T17:58:19.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc827e00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc808005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc81e800640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 0x7fc8080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc827400640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc808003970 0x7fc808003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc827e00640 1 -- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 msgr2=0x7fc8080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc827e00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8280ce1d0 0x7fc8080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc827e00640 1 -- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc808003970 msgr2=0x7fc808003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.419+0000 7fc827e00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc808003970 0x7fc808003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.420+0000 7fc827e00640 1 -- 172.21.15.2:0/3670552321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc808003520 con 0x7fc808004d00 2024-01-15T17:58:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.421+0000 7fc827e00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc808005180 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc820009ac0 tx=0x7fc8080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eaff80b6acbf6f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:19.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.421+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82002c710 con 0x7fc808004d00 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.421+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82002c870 con 0x7fc808004d00 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.421+0000 7fc826a00640 1 -- 172.21.15.2:0/3670552321 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82002cc00 con 0x7fc808004d00 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc82ce00640 1 -- 172.21.15.2:0/3670552321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 msgr2=0x7fc808005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc82ce00640 1 --2- 172.21.15.2:0/3670552321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc808005180 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc820009ac0 tx=0x7fc8080057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc827e00640 1 -- 172.21.15.2:0/3670552321 reap_dead start 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc82ce00640 1 -- 172.21.15.2:0/3670552321 shutdown_connections 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc82ce00640 1 -- 172.21.15.2:0/3670552321 >> 172.21.15.2:0/3670552321 conn(0x7fc8280b0840 msgr2=0x7fc8280b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc82ce00640 1 -- 172.21.15.2:0/3670552321 shutdown_connections 2024-01-15T17:58:19.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.422+0000 7fc82ce00640 1 -- 172.21.15.2:0/3670552321 wait complete. 2024-01-15T17:58:19.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.424+0000 7fc82ce00640 1 Processor -- start 2024-01-15T17:58:19.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.424+0000 7fc82ce00640 1 -- start start 2024-01-15T17:58:19.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc82ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc828146720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc82ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc82814e820 0x7fc82814ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc82ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc82814cdb0 0x7fc82814d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc82ce00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82814c500 con 0x7fc808004d00 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc82ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82814d770 con 0x7fc82814cdb0 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc82ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc82814fd10 con 0x7fc82814e820 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.425+0000 7fc827400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc828146720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc828146720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49622/0 (socket says 172.21.15.2:49622) 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 -- 172.21.15.2:0/1736996862 learned_addr learned my addr 172.21.15.2:0/1736996862 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827e00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc82814cdb0 0x7fc82814d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc81e800640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc82814e820 0x7fc82814ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 -- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc82814e820 msgr2=0x7fc82814ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc82814e820 0x7fc82814ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 -- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc82814cdb0 msgr2=0x7fc82814d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc82814cdb0 0x7fc82814d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 -- 172.21.15.2:0/1736996862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82814b270 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.426+0000 7fc827400640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc828146720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc818010860 tx=0x7fc818010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.427+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc818009610 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.427+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81800eb00 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.427+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81800ee90 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.427+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc82814a660 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.427+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc82814ac20 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.428+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82806bfa0 con 0x7fc808004d00 2024-01-15T17:58:19.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.429+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc81801c400 con 0x7fc808004d00 2024-01-15T17:58:19.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.430+0000 7fc81fe00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc80006b2c0 0x7fc80006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.430+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc81808a240 con 0x7fc808004d00 2024-01-15T17:58:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.431+0000 7fc81e800640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc80006b2c0 0x7fc80006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.432+0000 7fc81e800640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc80006b2c0 0x7fc80006d780 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc810001330 tx=0x7fc810009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:19.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.435+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc818054f00 con 0x7fc808004d00 2024-01-15T17:58:19.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.641+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc828074770 con 0x7fc808004d00 2024-01-15T17:58:19.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.642+0000 7fc81fe00640 1 -- 172.21.15.2:0/1736996862 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc818058520 con 0x7fc808004d00 2024-01-15T17:58:19.645 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.645+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc80006b2c0 msgr2=0x7fc80006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.645+0000 7fc82ce00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc80006b2c0 0x7fc80006d780 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc810001330 tx=0x7fc810009040 comp rx=0 tx=0).stop 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 msgr2=0x7fc828146720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc828146720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc818010860 tx=0x7fc818010d20 comp rx=0 tx=0).stop 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 shutdown_connections 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc80006b2c0 0x7fc80006d780 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc82814cdb0 0x7fc82814d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc82814e820 0x7fc82814ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 --2- 172.21.15.2:0/1736996862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc808004d00 0x7fc828146720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 >> 172.21.15.2:0/1736996862 conn(0x7fc8280b0840 msgr2=0x7fc82806ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:19.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 shutdown_connections 2024-01-15T17:58:19.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.646+0000 7fc82ce00640 1 -- 172.21.15.2:0/1736996862 wait complete. 2024-01-15T17:58:19.668 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:58:19.668 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:19.722 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:19.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.947+0000 7f8e90c00640 1 Processor -- start 2024-01-15T17:58:19.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.947+0000 7f8e90c00640 1 -- start start 2024-01-15T17:58:19.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e90c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c143810 0x7f8e8c06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e90c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e8c0b2250 0x7f8e8c0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e90c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c06e470 con 0x7f8e8c06e010 2024-01-15T17:58:19.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e90c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c0b3520 con 0x7f8e8c0b2250 2024-01-15T17:58:19.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e90c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c0ccbe0 con 0x7f8e8c143810 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e8be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e8c0b2250 0x7f8e8c0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.949+0000 7f8e8be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e8c0b2250 0x7f8e8c0cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49636/0 (socket says 172.21.15.2:49636) 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.950+0000 7f8e8be00640 1 -- 172.21.15.2:0/4104614992 learned_addr learned my addr 172.21.15.2:0/4104614992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.950+0000 7f8e8b400640 1 --2- 172.21.15.2:0/4104614992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c143810 0x7f8e8c06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.950+0000 7f8e8be00640 1 -- 172.21.15.2:0/4104614992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c143810 msgr2=0x7f8e8c06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.950+0000 7f8e8be00640 1 --2- 172.21.15.2:0/4104614992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c143810 0x7f8e8c06dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.950+0000 7f8e8be00640 1 -- 172.21.15.2:0/4104614992 >> v1:172.21.15.2:6789/0 conn(0x7f8e8c06e010 legacy=0x7f8e8c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.950+0000 7f8e8be00640 1 -- 172.21.15.2:0/4104614992 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c143bf0 con 0x7f8e8c0b2250 2024-01-15T17:58:19.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.951+0000 7f8e8be00640 1 --2- 172.21.15.2:0/4104614992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c0cd070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009e50 tx=0x7f8e8402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c78b4416f21cd0d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:19.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.951+0000 7f8e8a000640 1 -- 172.21.15.2:0/4104614992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8402ca80 con 0x7f8e8c0b2250 2024-01-15T17:58:19.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e8a000640 1 -- 172.21.15.2:0/4104614992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8402cbe0 con 0x7f8e8c0b2250 2024-01-15T17:58:19.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e8a000640 1 -- 172.21.15.2:0/4104614992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e840343f0 con 0x7f8e8c0b2250 2024-01-15T17:58:19.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 -- 172.21.15.2:0/4104614992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c0b2250 msgr2=0x7f8e8c0cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 --2- 172.21.15.2:0/4104614992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c0cd070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009e50 tx=0x7f8e8402bee0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 -- 172.21.15.2:0/4104614992 shutdown_connections 2024-01-15T17:58:19.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 --2- 172.21.15.2:0/4104614992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c0cd070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 --2- 172.21.15.2:0/4104614992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c143810 0x7f8e8c06dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 -- 172.21.15.2:0/4104614992 >> 172.21.15.2:0/4104614992 conn(0x7f8e8c0b0840 msgr2=0x7f8e8c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:19.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 -- 172.21.15.2:0/4104614992 shutdown_connections 2024-01-15T17:58:19.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.952+0000 7f8e90c00640 1 -- 172.21.15.2:0/4104614992 wait complete. 2024-01-15T17:58:19.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.954+0000 7f8e90c00640 1 Processor -- start 2024-01-15T17:58:19.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.954+0000 7f8e90c00640 1 -- start start 2024-01-15T17:58:19.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.955+0000 7f8e90c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c06e010 0x7f8e8c0a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.955+0000 7f8e90c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c135770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.955+0000 7f8e90c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 0x7f8e8c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.955+0000 7f8e90c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c135cb0 con 0x7f8e8c143810 2024-01-15T17:58:19.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.955+0000 7f8e90c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c136af0 con 0x7f8e8c0b2250 2024-01-15T17:58:19.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.955+0000 7f8e90c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c15d950 con 0x7f8e8c06e010 2024-01-15T17:58:19.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 0x7f8e8c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 0x7f8e8c15d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49640/0 (socket says 172.21.15.2:49640) 2024-01-15T17:58:19.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 -- 172.21.15.2:0/965207715 learned_addr learned my addr 172.21.15.2:0/965207715 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:19.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8b400640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c06e010 0x7f8e8c0a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8aa00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 -- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c06e010 msgr2=0x7f8e8c0a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c06e010 0x7f8e8c0a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 -- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e8c0b2250 msgr2=0x7f8e8c135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:19.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c135770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 -- 172.21.15.2:0/965207715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84009b30 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8b400640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c06e010 0x7f8e8c0a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.956+0000 7f8e8be00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 0x7f8e8c15d410 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009e50 tx=0x7f8e84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.957+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8403e450 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.957+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e8403e5b0 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.957+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e8403e960 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.957+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c135ee0 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.957+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c1364a0 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.958+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c136ca0 con 0x7f8e8c143810 2024-01-15T17:58:19.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.959+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8e8402cd50 con 0x7f8e8c143810 2024-01-15T17:58:19.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.960+0000 7f8e83400640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e6406b2c0 0x7f8e6406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:19.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.960+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8e840ac7d0 con 0x7f8e8c143810 2024-01-15T17:58:19.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.960+0000 7f8e8b400640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e6406b2c0 0x7f8e6406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:19.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.962+0000 7f8e8b400640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e6406b2c0 0x7f8e6406d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c006fd0 tx=0x7f8e7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:19.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:19.964+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e8c136ca0 con 0x7f8e8c143810 2024-01-15T17:58:20.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.166+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e8c136c60 con 0x7f8e8c143810 2024-01-15T17:58:20.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.168+0000 7f8e83400640 1 -- 172.21.15.2:0/965207715 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f8e84077520 con 0x7f8e8c143810 2024-01-15T17:58:20.171 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:20.171 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e6406b2c0 msgr2=0x7f8e6406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e6406b2c0 0x7f8e6406d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c006fd0 tx=0x7f8e7c008040 comp rx=0 tx=0).stop 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 msgr2=0x7f8e8c15d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 0x7f8e8c15d410 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8e84009e50 tx=0x7f8e84004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 shutdown_connections 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8e6406b2c0 0x7f8e6406d780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e8c143810 0x7f8e8c15d410 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e8c0b2250 0x7f8e8c135770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 --2- 172.21.15.2:0/965207715 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e8c06e010 0x7f8e8c0a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 >> 172.21.15.2:0/965207715 conn(0x7f8e8c0b0840 msgr2=0x7f8e8c06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:20.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 shutdown_connections 2024-01-15T17:58:20.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.171+0000 7f8e90c00640 1 -- 172.21.15.2:0/965207715 wait complete. 2024-01-15T17:58:20.193 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:20.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:20.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.415+0000 7fe1c9000640 1 Processor -- start 2024-01-15T17:58:20.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.415+0000 7fe1c9000640 1 -- start start 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 0x7fe1c415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c9000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe1c40720d0 0x7fe1c415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c9000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe1c41447d0 0x7fe1c40cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c9000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c4071430 con 0x7fe1c40720d0 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c9000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1c4144100 con 0x7fe1c41447d0 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c4071090 con 0x7fe1c4143d20 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe1c40720d0 0x7fe1c415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe1c40720d0 0x7fe1c415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49644/0 (socket says 172.21.15.2:49644) 2024-01-15T17:58:20.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.417+0000 7fe1c2a00640 1 -- 172.21.15.2:0/3312044087 learned_addr learned my addr 172.21.15.2:0/3312044087 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:20.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.418+0000 7fe1c2a00640 1 -- 172.21.15.2:0/3312044087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 msgr2=0x7fe1c415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:20.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.418+0000 7fe1c2a00640 1 --2- 172.21.15.2:0/3312044087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 0x7fe1c415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.418+0000 7fe1c2a00640 1 -- 172.21.15.2:0/3312044087 >> v2:172.21.15.137:3300/0 conn(0x7fe1c41447d0 msgr2=0x7fe1c40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:20.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.418+0000 7fe1c2a00640 1 --2- 172.21.15.2:0/3312044087 >> v2:172.21.15.137:3300/0 conn(0x7fe1c41447d0 0x7fe1c40cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.418+0000 7fe1c2a00640 1 -- 172.21.15.2:0/3312044087 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c4144560 con 0x7fe1c40720d0 2024-01-15T17:58:20.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.419+0000 7fe1c2a00640 1 --2- 172.21.15.2:0/3312044087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c40720d0 0x7fe1c415f090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe1c406e0a0 tx=0x7fe1ac02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1009234a4fb3a523 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.419+0000 7fe1c2000640 1 -- 172.21.15.2:0/3312044087 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac0317d0 con 0x7fe1c40720d0 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.419+0000 7fe1c2000640 1 -- 172.21.15.2:0/3312044087 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1ac031930 con 0x7fe1c40720d0 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.419+0000 7fe1c2000640 1 -- 172.21.15.2:0/3312044087 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac031cc0 con 0x7fe1c40720d0 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 -- 172.21.15.2:0/3312044087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c40720d0 msgr2=0x7fe1c415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 --2- 172.21.15.2:0/3312044087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c40720d0 0x7fe1c415f090 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe1c406e0a0 tx=0x7fe1ac02b870 comp rx=0 tx=0).stop 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 -- 172.21.15.2:0/3312044087 shutdown_connections 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 --2- 172.21.15.2:0/3312044087 >> v2:172.21.15.137:3300/0 conn(0x7fe1c41447d0 0x7fe1c40cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 --2- 172.21.15.2:0/3312044087 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c40720d0 0x7fe1c415f090 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 --2- 172.21.15.2:0/3312044087 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 0x7fe1c415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 -- 172.21.15.2:0/3312044087 >> 172.21.15.2:0/3312044087 conn(0x7fe1c40b0840 msgr2=0x7fe1c40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 -- 172.21.15.2:0/3312044087 shutdown_connections 2024-01-15T17:58:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.420+0000 7fe1c9000640 1 -- 172.21.15.2:0/3312044087 wait complete. 2024-01-15T17:58:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.422+0000 7fe1c9000640 1 Processor -- start 2024-01-15T17:58:20.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.422+0000 7fe1c9000640 1 -- start start 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1c40720d0 0x7fe1c40b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 0x7fe1c413d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c9000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 0x7fe1c415d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c9000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40acaa0 con 0x7fe1c41447d0 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40b3580 con 0x7fe1c40720d0 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c40b2ce0 con 0x7fe1c4143d20 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 0x7fe1c415d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 0x7fe1c415d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49654/0 (socket says 172.21.15.2:49654) 2024-01-15T17:58:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.423+0000 7fe1c3e00640 1 -- 172.21.15.2:0/461652507 learned_addr learned my addr 172.21.15.2:0/461652507 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:20.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3e00640 1 -- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 msgr2=0x7fe1c413d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3e00640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 0x7fe1c413d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3e00640 1 -- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1c40720d0 msgr2=0x7fe1c40b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3400640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1c40720d0 0x7fe1c40b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3e00640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1c40720d0 0x7fe1c40b2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3e00640 1 -- 172.21.15.2:0/461652507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ac0094d0 con 0x7fe1c41447d0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c3e00640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 0x7fe1c415d4d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00b970 tx=0x7fe1bc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc00cb40 con 0x7fe1c41447d0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc00ccc0 con 0x7fe1c41447d0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c40b2fc0 con 0x7fe1c41447d0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.424+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc014680 con 0x7fe1c41447d0 2024-01-15T17:58:20.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.425+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c40cb8a0 con 0x7fe1c41447d0 2024-01-15T17:58:20.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.426+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c40acbe0 con 0x7fe1c41447d0 2024-01-15T17:58:20.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.426+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe1bc01d3b0 con 0x7fe1c41447d0 2024-01-15T17:58:20.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.427+0000 7fe1bb400640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe19c06b2c0 0x7fe19c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.427+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe1bc089c10 con 0x7fe1c41447d0 2024-01-15T17:58:20.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.428+0000 7fe1c3400640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe19c06b2c0 0x7fe19c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.429+0000 7fe1c3400640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe19c06b2c0 0x7fe19c06d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400a770 tx=0x7fe1b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:20.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.432+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1c40acbe0 con 0x7fe1c41447d0 2024-01-15T17:58:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.626+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe1c40acbe0 con 0x7fe1c41447d0 2024-01-15T17:58:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.627+0000 7fe1bb400640 1 -- 172.21.15.2:0/461652507 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe1c40acbe0 con 0x7fe1c41447d0 2024-01-15T17:58:20.630 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.629+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe19c06b2c0 msgr2=0x7fe19c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.629+0000 7fe1c9000640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe19c06b2c0 0x7fe19c06d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe1b400a770 tx=0x7fe1b4008040 comp rx=0 tx=0).stop 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.629+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 msgr2=0x7fe1c415d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.629+0000 7fe1c9000640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 0x7fe1c415d4d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00b970 tx=0x7fe1bc00be30 comp rx=0 tx=0).stop 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 shutdown_connections 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe19c06b2c0 0x7fe19c06d780 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1c41447d0 0x7fe1c415d4d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1c4143d20 0x7fe1c413d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 --2- 172.21.15.2:0/461652507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1c40720d0 0x7fe1c40b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 >> 172.21.15.2:0/461652507 conn(0x7fe1c40b0840 msgr2=0x7fe1c4142fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 shutdown_connections 2024-01-15T17:58:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.630+0000 7fe1c9000640 1 -- 172.21.15.2:0/461652507 wait complete. 2024-01-15T17:58:20.652 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.876+0000 7fcbed600640 1 Processor -- start 2024-01-15T17:58:20.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.876+0000 7fcbed600640 1 -- start start 2024-01-15T17:58:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbed600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 0x7fcbe80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbed600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcbe80cdd70 0x7fcbe806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbed600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe80ce230 con 0x7fcbe815c450 2024-01-15T17:58:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbed600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe80ce510 con 0x7fcbe80cdd70 2024-01-15T17:58:20.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbed600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80ce7f0 con 0x7fcbe8144c00 2024-01-15T17:58:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbe7400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcbe815c450 0x7fcbe80b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35892/0 (socket says 172.21.15.2:35892) 2024-01-15T17:58:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.878+0000 7fcbe7400640 1 -- 172.21.15.2:0/240849760 learned_addr learned my addr 172.21.15.2:0/240849760 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/240849760 >> v2:172.21.15.137:3300/0 conn(0x7fcbe80cdd70 0x7fcbe806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 0x7fcbe80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720939388 0 0) 0x7fcbe80ce230 con 0x7fcbe815c450 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc8003520 con 0x7fcbe815c450 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2200285827 0 0) 0x7fcbc8003520 con 0x7fcbe815c450 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 msgr2=0x7fcbe80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 0x7fcbe80cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 >> v2:172.21.15.137:3300/0 conn(0x7fcbe80cdd70 msgr2=0x7fcbe806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 --2- 172.21.15.2:0/240849760 >> v2:172.21.15.137:3300/0 conn(0x7fcbe80cdd70 0x7fcbe806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80705f0 con 0x7fcbe815c450 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fcbe00019f0 con 0x7fcbe815c450 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 >> v1:172.21.15.2:6789/0 conn(0x7fcbe815c450 legacy=0x7fcbe80b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbc8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 0x7fcbc80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.879+0000 7fcbe6000640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc8004d00 0x7fcbc8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80ce510 con 0x7fcbc8003970 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80ce7f0 con 0x7fcbc8004d00 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc800b980 con 0x7fcbe8144c00 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbc8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 0x7fcbc80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6a00640 1 -- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 msgr2=0x7fcbc80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe8144c00 0x7fcbc80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe7400640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc8004d00 0x7fcbc8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6a00640 1 -- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc8004d00 msgr2=0x7fcbc8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc8004d00 0x7fcbc8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe6a00640 1 -- 172.21.15.2:0/240849760 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc8003520 con 0x7fcbc8003970 2024-01-15T17:58:20.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.880+0000 7fcbe7400640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbc8004d00 0x7fcbc8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:20.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbc8003d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009ac0 tx=0x7fcbc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9ed4ef6ae18469 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:20.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd802c6b0 con 0x7fcbc8003970 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd802c810 con 0x7fcbc8003970 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbe6000640 1 -- 172.21.15.2:0/240849760 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd802cba0 con 0x7fcbc8003970 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbed600640 1 -- 172.21.15.2:0/240849760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 msgr2=0x7fcbc8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbed600640 1 --2- 172.21.15.2:0/240849760 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbc8003d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8009ac0 tx=0x7fcbc80057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.882+0000 7fcbe7400640 1 -- 172.21.15.2:0/240849760 reap_dead start 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.883+0000 7fcbed600640 1 -- 172.21.15.2:0/240849760 shutdown_connections 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.883+0000 7fcbed600640 1 -- 172.21.15.2:0/240849760 >> 172.21.15.2:0/240849760 conn(0x7fcbe80b45f0 msgr2=0x7fcbe80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.883+0000 7fcbed600640 1 -- 172.21.15.2:0/240849760 shutdown_connections 2024-01-15T17:58:20.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.883+0000 7fcbed600640 1 -- 172.21.15.2:0/240849760 wait complete. 2024-01-15T17:58:20.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.885+0000 7fcbed600640 1 Processor -- start 2024-01-15T17:58:20.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.885+0000 7fcbed600640 1 -- start start 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbed600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbe81f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbed600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80b1ff0 0x7fcbe81405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbed600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbe815a5d0 0x7fcbe8140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbed600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80b3100 con 0x7fcbc8003970 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbed600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80ae270 con 0x7fcbe815a5d0 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbed600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe80b2480 con 0x7fcbe80b1ff0 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbe81f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbe81f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49680/0 (socket says 172.21.15.2:49680) 2024-01-15T17:58:20.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 -- 172.21.15.2:0/1990549669 learned_addr learned my addr 172.21.15.2:0/1990549669 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe7400640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbe815a5d0 0x7fcbe8140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80b1ff0 0x7fcbe81405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 -- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80b1ff0 msgr2=0x7fcbe81405c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80b1ff0 0x7fcbe81405c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 -- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbe815a5d0 msgr2=0x7fcbe8140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbe815a5d0 0x7fcbe8140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.886+0000 7fcbe6a00640 1 -- 172.21.15.2:0/1990549669 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe80b2760 con 0x7fcbc8003970 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.887+0000 7fcbe6a00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbe81f2db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8000c00 tx=0x7fcbd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.887+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8003810 con 0x7fcbc8003970 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.887+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd8003990 con 0x7fcbc8003970 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.887+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe80b2a40 con 0x7fcbc8003970 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.887+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8003d40 con 0x7fcbc8003970 2024-01-15T17:58:20.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.887+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe80b3e70 con 0x7fcbc8003970 2024-01-15T17:58:20.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.888+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe80b3370 con 0x7fcbc8003970 2024-01-15T17:58:20.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.889+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcbd803b070 con 0x7fcbc8003970 2024-01-15T17:58:20.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.890+0000 7fcbdea00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:20.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.890+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcbd80abba0 con 0x7fcbc8003970 2024-01-15T17:58:20.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.891+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:20.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.892+0000 7fcbdfe00640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8143e30 tx=0x7fcbd0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:20.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:20.895+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbe80b3370 con 0x7fcbc8003970 2024-01-15T17:58:21.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.088+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcbe80b3370 con 0x7fcbc8003970 2024-01-15T17:58:21.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.089+0000 7fcbdea00640 1 -- 172.21.15.2:0/1990549669 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcbd804c020 con 0x7fcbc8003970 2024-01-15T17:58:21.092 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 msgr2=0x7fcbc006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8143e30 tx=0x7fcbd0008040 comp rx=0 tx=0).stop 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 msgr2=0x7fcbe81f2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbe81f2db0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcbd8000c00 tx=0x7fcbd8004730 comp rx=0 tx=0).stop 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 shutdown_connections 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbc006b2c0 0x7fcbc006d780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcbe815a5d0 0x7fcbe8140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcbe80b1ff0 0x7fcbe81405c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 --2- 172.21.15.2:0/1990549669 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcbc8003970 0x7fcbe81f2db0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 >> 172.21.15.2:0/1990549669 conn(0x7fcbe80b45f0 msgr2=0x7fcbe815b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:21.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 shutdown_connections 2024-01-15T17:58:21.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.092+0000 7fcbed600640 1 -- 172.21.15.2:0/1990549669 wait complete. 2024-01-15T17:58:21.112 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:21.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.327+0000 7fea89c00640 1 Processor -- start 2024-01-15T17:58:21.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.327+0000 7fea89c00640 1 -- start start 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fea84143d20 0x7fea8415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fea841447d0 0x7fea840cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea84071430 con 0x7fea84143d20 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea84144100 con 0x7fea841447d0 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea84071090 con 0x7fea840720d0 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea83e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fea84143d20 0x7fea8415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea83e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fea84143d20 0x7fea8415eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49694/0 (socket says 172.21.15.2:49694) 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea83e00640 1 -- 172.21.15.2:0/1703476754 learned_addr learned my addr 172.21.15.2:0/1703476754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.329+0000 7fea89200640 1 --2- 172.21.15.2:0/1703476754 >> v2:172.21.15.137:3300/0 conn(0x7fea841447d0 0x7fea840cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea7b200640 1 --2- 172.21.15.2:0/1703476754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea83e00640 1 -- 172.21.15.2:0/1703476754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 msgr2=0x7fea8415f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea83e00640 1 --2- 172.21.15.2:0/1703476754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea83e00640 1 -- 172.21.15.2:0/1703476754 >> v2:172.21.15.137:3300/0 conn(0x7fea841447d0 msgr2=0x7fea840cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea83e00640 1 --2- 172.21.15.2:0/1703476754 >> v2:172.21.15.137:3300/0 conn(0x7fea841447d0 0x7fea840cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea83e00640 1 -- 172.21.15.2:0/1703476754 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea84144560 con 0x7fea84143d20 2024-01-15T17:58:21.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.330+0000 7fea89200640 1 --2- 172.21.15.2:0/1703476754 >> v2:172.21.15.137:3300/0 conn(0x7fea841447d0 0x7fea840cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.331+0000 7fea83e00640 1 --2- 172.21.15.2:0/1703476754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea84143d20 0x7fea8415eb50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fea8406e0a0 tx=0x7fea7402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2430368d21da942d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.331+0000 7fea83400640 1 -- 172.21.15.2:0/1703476754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7402c9e0 con 0x7fea84143d20 2024-01-15T17:58:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.331+0000 7fea83400640 1 -- 172.21.15.2:0/1703476754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7402cb40 con 0x7fea84143d20 2024-01-15T17:58:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.331+0000 7fea83400640 1 -- 172.21.15.2:0/1703476754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea740343f0 con 0x7fea84143d20 2024-01-15T17:58:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 -- 172.21.15.2:0/1703476754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea84143d20 msgr2=0x7fea8415eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 --2- 172.21.15.2:0/1703476754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea84143d20 0x7fea8415eb50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fea8406e0a0 tx=0x7fea7402be40 comp rx=0 tx=0).stop 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 -- 172.21.15.2:0/1703476754 shutdown_connections 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 --2- 172.21.15.2:0/1703476754 >> v2:172.21.15.137:3300/0 conn(0x7fea841447d0 0x7fea840cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 --2- 172.21.15.2:0/1703476754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 --2- 172.21.15.2:0/1703476754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea84143d20 0x7fea8415eb50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 -- 172.21.15.2:0/1703476754 >> 172.21.15.2:0/1703476754 conn(0x7fea840b0840 msgr2=0x7fea840b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 -- 172.21.15.2:0/1703476754 shutdown_connections 2024-01-15T17:58:21.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.332+0000 7fea89c00640 1 -- 172.21.15.2:0/1703476754 wait complete. 2024-01-15T17:58:21.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.334+0000 7fea89c00640 1 Processor -- start 2024-01-15T17:58:21.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.334+0000 7fea89c00640 1 -- start start 2024-01-15T17:58:21.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.335+0000 7fea89c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.335+0000 7fea89c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea84143d20 0x7fea84140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.335+0000 7fea89c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 0x7fea8415e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.335+0000 7fea89c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8415af10 con 0x7fea841447d0 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.335+0000 7fea89c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea84141270 con 0x7fea84143d20 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.335+0000 7fea89c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea84141550 con 0x7fea840720d0 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 0x7fea8415e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 0x7fea8415e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49702/0 (socket says 172.21.15.2:49702) 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 -- 172.21.15.2:0/2151491152 learned_addr learned my addr 172.21.15.2:0/2151491152 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea7b200640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea84143d20 0x7fea84140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea83e00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 -- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 msgr2=0x7fea8415a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 -- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea84143d20 msgr2=0x7fea84140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea84143d20 0x7fea84140900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 -- 172.21.15.2:0/2151491152 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74009b30 con 0x7fea841447d0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.336+0000 7fea89200640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 0x7fea8415e120 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00c7c0 tx=0x7fea7c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.337+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c004230 con 0x7fea841447d0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.337+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c004390 con 0x7fea841447d0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.337+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c007860 con 0x7fea841447d0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.337+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea84141830 con 0x7fea841447d0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.337+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea84140e40 con 0x7fea841447d0 2024-01-15T17:58:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.338+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea84130100 con 0x7fea841447d0 2024-01-15T17:58:21.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.339+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fea7c0079c0 con 0x7fea841447d0 2024-01-15T17:58:21.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.340+0000 7fea80c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fea5c06b2c0 0x7fea5c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.340+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fea7c089fa0 con 0x7fea841447d0 2024-01-15T17:58:21.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.341+0000 7fea83e00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fea5c06b2c0 0x7fea5c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.342+0000 7fea83e00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fea5c06b2c0 0x7fea5c06d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea74035000 tx=0x7fea74030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:21.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.344+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea84130100 con 0x7fea841447d0 2024-01-15T17:58:21.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.542+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea841304e0 con 0x7fea841447d0 2024-01-15T17:58:21.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.543+0000 7fea80c00640 1 -- 172.21.15.2:0/2151491152 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fea841304e0 con 0x7fea841447d0 2024-01-15T17:58:21.547 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fea5c06b2c0 msgr2=0x7fea5c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fea5c06b2c0 0x7fea5c06d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea74035000 tx=0x7fea74030000 comp rx=0 tx=0).stop 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 msgr2=0x7fea8415e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 0x7fea8415e120 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00c7c0 tx=0x7fea7c00cc80 comp rx=0 tx=0).stop 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 shutdown_connections 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fea5c06b2c0 0x7fea5c06d780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea841447d0 0x7fea8415e120 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea84143d20 0x7fea84140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 --2- 172.21.15.2:0/2151491152 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea840720d0 0x7fea8415a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 >> 172.21.15.2:0/2151491152 conn(0x7fea840b0840 msgr2=0x7fea840ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:21.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 shutdown_connections 2024-01-15T17:58:21.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.547+0000 7fea89c00640 1 -- 172.21.15.2:0/2151491152 wait complete. 2024-01-15T17:58:21.569 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:21.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.777+0000 7f0a7aa00640 1 Processor -- start 2024-01-15T17:58:21.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.778+0000 7f0a7aa00640 1 -- start start 2024-01-15T17:58:21.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.779+0000 7f0a7aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740ce1d0 0x7f0a7406c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.779+0000 7f0a7aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0a740ccb80 0x7f0a7415b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a7aa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0a740cb620 0x7f0a740cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a7aa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a740cbff0 con 0x7f0a740ccb80 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a7aa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a74071340 con 0x7f0a740cb620 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a7aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7406c670 con 0x7f0a740ce1d0 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a78c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0a740ccb80 0x7f0a7415b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a78c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0a740ccb80 0x7f0a7415b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49708/0 (socket says 172.21.15.2:49708) 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a78c00640 1 -- 172.21.15.2:0/3907391466 learned_addr learned my addr 172.21.15.2:0/3907391466 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a79600640 1 --2- 172.21.15.2:0/3907391466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740ce1d0 0x7f0a7406c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.780+0000 7f0a7a000640 1 --2- 172.21.15.2:0/3907391466 >> v2:172.21.15.137:3300/0 conn(0x7f0a740cb620 0x7f0a740cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.781+0000 7f0a78c00640 1 -- 172.21.15.2:0/3907391466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740ce1d0 msgr2=0x7f0a7406c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.781+0000 7f0a78c00640 1 --2- 172.21.15.2:0/3907391466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740ce1d0 0x7f0a7406c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.781+0000 7f0a78c00640 1 -- 172.21.15.2:0/3907391466 >> v2:172.21.15.137:3300/0 conn(0x7f0a740cb620 msgr2=0x7f0a740cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.781+0000 7f0a78c00640 1 --2- 172.21.15.2:0/3907391466 >> v2:172.21.15.137:3300/0 conn(0x7f0a740cb620 0x7f0a740cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.781+0000 7f0a78c00640 1 -- 172.21.15.2:0/3907391466 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7415bfb0 con 0x7f0a740ccb80 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.782+0000 7f0a78c00640 1 --2- 172.21.15.2:0/3907391466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ccb80 0x7f0a7415b640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a740b16a0 tx=0x7f0a5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97233ab4b77219c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.782+0000 7f0a73e00640 1 -- 172.21.15.2:0/3907391466 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c031870 con 0x7f0a740ccb80 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.782+0000 7f0a73e00640 1 -- 172.21.15.2:0/3907391466 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c0319d0 con 0x7f0a740ccb80 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.782+0000 7f0a73e00640 1 -- 172.21.15.2:0/3907391466 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c031d60 con 0x7f0a740ccb80 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 -- 172.21.15.2:0/3907391466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ccb80 msgr2=0x7f0a7415b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/3907391466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ccb80 0x7f0a7415b640 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0a740b16a0 tx=0x7f0a5c02b910 comp rx=0 tx=0).stop 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 -- 172.21.15.2:0/3907391466 shutdown_connections 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/3907391466 >> v2:172.21.15.137:3300/0 conn(0x7f0a740cb620 0x7f0a740cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/3907391466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ccb80 0x7f0a7415b640 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/3907391466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740ce1d0 0x7f0a7406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 -- 172.21.15.2:0/3907391466 >> 172.21.15.2:0/3907391466 conn(0x7f0a740b0840 msgr2=0x7f0a740b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:21.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 -- 172.21.15.2:0/3907391466 shutdown_connections 2024-01-15T17:58:21.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.783+0000 7f0a7aa00640 1 -- 172.21.15.2:0/3907391466 wait complete. 2024-01-15T17:58:21.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.785+0000 7f0a7aa00640 1 Processor -- start 2024-01-15T17:58:21.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.785+0000 7f0a7aa00640 1 -- start start 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740cb620 0x7f0a7415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a740ccb80 0x7f0a741514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 0x7f0a741519f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7aa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740a8600 con 0x7f0a740ce1d0 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740aa690 con 0x7f0a740ccb80 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a740a9b10 con 0x7f0a740cb620 2024-01-15T17:58:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 0x7f0a741519f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 0x7f0a741519f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49720/0 (socket says 172.21.15.2:49720) 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a7a000640 1 -- 172.21.15.2:0/638650207 learned_addr learned my addr 172.21.15.2:0/638650207 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a78c00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a740ccb80 0x7f0a741514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.786+0000 7f0a79600640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740cb620 0x7f0a7415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7a000640 1 -- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740cb620 msgr2=0x7f0a7415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7a000640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740cb620 0x7f0a7415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7a000640 1 -- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a740ccb80 msgr2=0x7f0a741514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7a000640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a740ccb80 0x7f0a741514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7a000640 1 -- 172.21.15.2:0/638650207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c0094d0 con 0x7f0a740ce1d0 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a78c00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a740ccb80 0x7f0a741514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a79600640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740cb620 0x7f0a7415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7a000640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 0x7f0a741519f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c007e60 tx=0x7f0a6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c014620 con 0x7f0a740ce1d0 2024-01-15T17:58:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c00ee90 con 0x7f0a740ce1d0 2024-01-15T17:58:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c0149e0 con 0x7f0a740ce1d0 2024-01-15T17:58:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.787+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a740a9df0 con 0x7f0a740ce1d0 2024-01-15T17:58:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.788+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a740aa3b0 con 0x7f0a740ce1d0 2024-01-15T17:58:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.789+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7406bfa0 con 0x7f0a740ce1d0 2024-01-15T17:58:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.789+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0a6c014b40 con 0x7f0a740ce1d0 2024-01-15T17:58:21.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.790+0000 7f0a71600640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a4c06b2c0 0x7f0a4c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:21.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.790+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0a6c08a650 con 0x7f0a740ce1d0 2024-01-15T17:58:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.791+0000 7f0a79600640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a4c06b2c0 0x7f0a4c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.792+0000 7f0a79600640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a4c06b2c0 0x7f0a4c06d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a64006fd0 tx=0x7f0a64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:21.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:21.795+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a6c0553a0 con 0x7f0a740ce1d0 2024-01-15T17:58:22.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.002+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a740a8460 con 0x7f0a740ce1d0 2024-01-15T17:58:22.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.003+0000 7f0a71600640 1 -- 172.21.15.2:0/638650207 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0a740a8460 con 0x7f0a740ce1d0 2024-01-15T17:58:22.006 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.005+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a4c06b2c0 msgr2=0x7f0a4c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.005+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a4c06b2c0 0x7f0a4c06d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0a64006fd0 tx=0x7f0a64008040 comp rx=0 tx=0).stop 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 msgr2=0x7f0a741519f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 0x7f0a741519f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c007e60 tx=0x7f0a6c00e4b0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 shutdown_connections 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a4c06b2c0 0x7f0a4c06d780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a740ce1d0 0x7f0a741519f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a740ccb80 0x7f0a741514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 --2- 172.21.15.2:0/638650207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a740cb620 0x7f0a7415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 >> 172.21.15.2:0/638650207 conn(0x7f0a740b0840 msgr2=0x7f0a740714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:22.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 shutdown_connections 2024-01-15T17:58:22.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.006+0000 7f0a7aa00640 1 -- 172.21.15.2:0/638650207 wait complete. 2024-01-15T17:58:22.026 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T17:58:22.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:22.026 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T17:58:22.124 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:22.125 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:58:22.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:22.125 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T17:58:22.191 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:58:22.191 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:22.249 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:58:22.249 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:22.298 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:22.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.522+0000 7f6937e00640 1 Processor -- start 2024-01-15T17:58:22.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.522+0000 7f6937e00640 1 -- start start 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6937e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693815ec60 0x7f693815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6937e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f693806cad0 0x7f693806cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6937e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69380716d0 con 0x7f6938141880 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6937e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6938071f20 con 0x7f693806cad0 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6937e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693806c6b0 con 0x7f693815ec60 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6936a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6938141880 0x7f6938070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35904/0 (socket says 172.21.15.2:35904) 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6936a00640 1 -- 172.21.15.2:0/1013948092 learned_addr learned my addr 172.21.15.2:0/1013948092 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:22.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.524+0000 7f6937400640 1 --2- 172.21.15.2:0/1013948092 >> v2:172.21.15.2:3300/0 conn(0x7f693806cad0 0x7f693806cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.525+0000 7f6936000640 1 --2- 172.21.15.2:0/1013948092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693815ec60 0x7f693815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.525+0000 7f6937400640 1 -- 172.21.15.2:0/1013948092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693815ec60 msgr2=0x7f693815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.525+0000 7f6937400640 1 --2- 172.21.15.2:0/1013948092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693815ec60 0x7f693815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.525+0000 7f6937400640 1 -- 172.21.15.2:0/1013948092 >> v1:172.21.15.2:6789/0 conn(0x7f6938141880 legacy=0x7f6938070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.525+0000 7f6937400640 1 -- 172.21.15.2:0/1013948092 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6938071960 con 0x7f693806cad0 2024-01-15T17:58:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.525+0000 7f6935600640 1 -- 172.21.15.2:0/1013948092 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2839538289 0 0) 0x7f69380716d0 con 0x7f6938141880 2024-01-15T17:58:22.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.527+0000 7f6937400640 1 --2- 172.21.15.2:0/1013948092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f693806cad0 0x7f693806cf60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6930009d90 tx=0x7f693002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82609f1d218d219 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.529+0000 7f6935600640 1 -- 172.21.15.2:0/1013948092 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693002cbc0 con 0x7f693806cad0 2024-01-15T17:58:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.529+0000 7f6935600640 1 -- 172.21.15.2:0/1013948092 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693002cd20 con 0x7f693806cad0 2024-01-15T17:58:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.529+0000 7f6935600640 1 -- 172.21.15.2:0/1013948092 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930034660 con 0x7f693806cad0 2024-01-15T17:58:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 -- 172.21.15.2:0/1013948092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f693806cad0 msgr2=0x7f693806cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 --2- 172.21.15.2:0/1013948092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f693806cad0 0x7f693806cf60 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6930009d90 tx=0x7f693002be10 comp rx=0 tx=0).stop 2024-01-15T17:58:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 -- 172.21.15.2:0/1013948092 shutdown_connections 2024-01-15T17:58:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 --2- 172.21.15.2:0/1013948092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f693806cad0 0x7f693806cf60 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 --2- 172.21.15.2:0/1013948092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693815ec60 0x7f693815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 -- 172.21.15.2:0/1013948092 >> 172.21.15.2:0/1013948092 conn(0x7f69380b0840 msgr2=0x7f69380b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 -- 172.21.15.2:0/1013948092 shutdown_connections 2024-01-15T17:58:22.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.530+0000 7f6937e00640 1 -- 172.21.15.2:0/1013948092 wait complete. 2024-01-15T17:58:22.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.532+0000 7f6937e00640 1 Processor -- start 2024-01-15T17:58:22.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.532+0000 7f6937e00640 1 -- start start 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6937e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693806cad0 0x7f69381f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6937e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 0x7f69380b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6937e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f693815ec60 0x7f69380cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6937e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69381f39c0 con 0x7f6938141880 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6937e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693815a430 con 0x7f693815ec60 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6937e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69380b2180 con 0x7f693806cad0 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6936000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 0x7f69380b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6936000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 0x7f69380b1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49734/0 (socket says 172.21.15.2:49734) 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.533+0000 7f6936000640 1 -- 172.21.15.2:0/2027764217 learned_addr learned my addr 172.21.15.2:0/2027764217 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936a00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693806cad0 0x7f69381f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6937400640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f693815ec60 0x7f69380cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936000640 1 -- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693806cad0 msgr2=0x7f69381f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936000640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693806cad0 0x7f69381f3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936000640 1 -- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f693815ec60 msgr2=0x7f69380cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936000640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f693815ec60 0x7f69380cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936000640 1 -- 172.21.15.2:0/2027764217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6930009a70 con 0x7f6938141880 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6937400640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f693815ec60 0x7f69380cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:22.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.534+0000 7f6936000640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 0x7f69380b1c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f692000e870 tx=0x7f692000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.535+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69200094f0 con 0x7f6938141880 2024-01-15T17:58:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.535+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692000c940 con 0x7f6938141880 2024-01-15T17:58:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.535+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692000ccf0 con 0x7f6938141880 2024-01-15T17:58:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.535+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69380b2390 con 0x7f6938141880 2024-01-15T17:58:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.535+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69380b2950 con 0x7f6938141880 2024-01-15T17:58:22.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.536+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6938074770 con 0x7f6938141880 2024-01-15T17:58:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.537+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6920013470 con 0x7f6938141880 2024-01-15T17:58:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.538+0000 7f692ea00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f691006b2c0 0x7f691006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.538+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6920089e10 con 0x7f6938141880 2024-01-15T17:58:22.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.538+0000 7f6936a00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f691006b2c0 0x7f691006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.539+0000 7f6936a00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f691006b2c0 0x7f691006d780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6928000f60 tx=0x7f6928007b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:22.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.542+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f692008d030 con 0x7f6938141880 2024-01-15T17:58:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.738+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f693806e1c0 con 0x7f6938141880 2024-01-15T17:58:22.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.741+0000 7f692ea00640 1 -- 172.21.15.2:0/2027764217 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f6920017090 con 0x7f6938141880 2024-01-15T17:58:22.744 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:22.744 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.743+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f691006b2c0 msgr2=0x7f691006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.743+0000 7f6937e00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f691006b2c0 0x7f691006d780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f6928000f60 tx=0x7f6928007b70 comp rx=0 tx=0).stop 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.743+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 msgr2=0x7f69380b1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.743+0000 7f6937e00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 0x7f69380b1c40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f692000e870 tx=0x7f692000ed30 comp rx=0 tx=0).stop 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 shutdown_connections 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f691006b2c0 0x7f691006d780 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f693815ec60 0x7f69380cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6938141880 0x7f69380b1c40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 --2- 172.21.15.2:0/2027764217 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f693806cad0 0x7f69381f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 >> 172.21.15.2:0/2027764217 conn(0x7f69380b0840 msgr2=0x7f69381406a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 shutdown_connections 2024-01-15T17:58:22.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.744+0000 7f6937e00640 1 -- 172.21.15.2:0/2027764217 wait complete. 2024-01-15T17:58:22.768 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:22.768 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:22.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.987+0000 7f3a41e00640 1 Processor -- start 2024-01-15T17:58:22.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.988+0000 7f3a41e00640 1 -- start start 2024-01-15T17:58:22.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.989+0000 7f3a41e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3a3c0d1870 0x7f3a3c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.989+0000 7f3a41e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3a3c0b2210 0x7f3a3c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.989+0000 7f3a41e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a3c0d1250 con 0x7f3a3c0d1c50 2024-01-15T17:58:22.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.989+0000 7f3a41e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c0ce6d0 con 0x7f3a3c0d1870 2024-01-15T17:58:22.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.989+0000 7f3a41e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a3c0ce8e0 con 0x7f3a3c0b2210 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3a3c0d1870 0x7f3a3c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3a3c0d1870 0x7f3a3c0d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49174/0 (socket says 172.21.15.2:49174) 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 -- 172.21.15.2:0/2075976994 learned_addr learned my addr 172.21.15.2:0/2075976994 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a3be00640 1 --2- 172.21.15.2:0/2075976994 >> v2:172.21.15.137:3300/0 conn(0x7f3a3c0b2210 0x7f3a3c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 -- 172.21.15.2:0/2075976994 >> v2:172.21.15.137:3300/0 conn(0x7f3a3c0b2210 msgr2=0x7f3a3c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 --2- 172.21.15.2:0/2075976994 >> v2:172.21.15.137:3300/0 conn(0x7f3a3c0b2210 0x7f3a3c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 -- 172.21.15.2:0/2075976994 >> v1:172.21.15.137:6789/0 conn(0x7f3a3c0d1c50 legacy=0x7f3a3c0cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:22.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.990+0000 7f3a40a00640 1 -- 172.21.15.2:0/2075976994 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0cebc0 con 0x7f3a3c0d1870 2024-01-15T17:58:22.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.991+0000 7f3a40a00640 1 --2- 172.21.15.2:0/2075976994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1870 0x7f3a3c0d3bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c067800 tx=0x7f3a2c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecaa74eaa31d902e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:22.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a3b400640 1 -- 172.21.15.2:0/2075976994 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c031870 con 0x7f3a3c0d1870 2024-01-15T17:58:22.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a3b400640 1 -- 172.21.15.2:0/2075976994 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a2c0319d0 con 0x7f3a3c0d1870 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a3b400640 1 -- 172.21.15.2:0/2075976994 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a2c031d60 con 0x7f3a3c0d1870 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a41e00640 1 -- 172.21.15.2:0/2075976994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1870 msgr2=0x7f3a3c0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a41e00640 1 --2- 172.21.15.2:0/2075976994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1870 0x7f3a3c0d3bd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c067800 tx=0x7f3a2c02b910 comp rx=0 tx=0).stop 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a41e00640 1 -- 172.21.15.2:0/2075976994 shutdown_connections 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a41e00640 1 --2- 172.21.15.2:0/2075976994 >> v2:172.21.15.137:3300/0 conn(0x7f3a3c0b2210 0x7f3a3c15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a41e00640 1 --2- 172.21.15.2:0/2075976994 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1870 0x7f3a3c0d3bd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.992+0000 7f3a41e00640 1 -- 172.21.15.2:0/2075976994 >> 172.21.15.2:0/2075976994 conn(0x7f3a3c0d30b0 msgr2=0x7f3a3c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.993+0000 7f3a41e00640 1 -- 172.21.15.2:0/2075976994 shutdown_connections 2024-01-15T17:58:22.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.993+0000 7f3a41e00640 1 -- 172.21.15.2:0/2075976994 wait complete. 2024-01-15T17:58:22.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.994+0000 7f3a41e00640 1 Processor -- start 2024-01-15T17:58:22.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.994+0000 7f3a41e00640 1 -- start start 2024-01-15T17:58:22.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.995+0000 7f3a41e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3c0b2210 0x7f3a3c151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.995+0000 7f3a41e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3c0d1870 0x7f3a3c1553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.995+0000 7f3a41e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 0x7f3a3c155910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:22.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.995+0000 7f3a41e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c1f40e0 con 0x7f3a3c0d1c50 2024-01-15T17:58:22.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.995+0000 7f3a41e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0cb1d0 con 0x7f3a3c0b2210 2024-01-15T17:58:22.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.995+0000 7f3a41e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a3c0cb4b0 con 0x7f3a3c0d1870 2024-01-15T17:58:22.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 0x7f3a3c155910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:22.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 0x7f3a3c155910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49176/0 (socket says 172.21.15.2:49176) 2024-01-15T17:58:22.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 -- 172.21.15.2:0/234479490 learned_addr learned my addr 172.21.15.2:0/234479490 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:23.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a3be00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3c0d1870 0x7f3a3c1553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a40a00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3c0b2210 0x7f3a3c151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 -- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3c0d1870 msgr2=0x7f3a3c1553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3c0d1870 0x7f3a3c1553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 -- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3c0b2210 msgr2=0x7f3a3c151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3c0b2210 0x7f3a3c151810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 -- 172.21.15.2:0/234479490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a2c0094d0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a3be00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3c0d1870 0x7f3a3c1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.996+0000 7f3a41400640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 0x7f3a3c155910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b490 tx=0x7f3a3400b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.997+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34011830 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.997+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a340119b0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.997+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34011d60 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.997+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a3c0cb790 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.997+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a3c151d50 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.998+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a3c0cbcb0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:22.999+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3a3401a400 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.000+0000 7f3a38c00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a1406b2f0 0x7f3a1406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.000+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3a34089db0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.001+0000 7f3a40a00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a1406b2f0 0x7f3a1406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.002+0000 7f3a40a00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a1406b2f0 0x7f3a1406d7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c036000 tx=0x7f3a2c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.004+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a3c0cbcb0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.199+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a3c0cbcb0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.200+0000 7f3a38c00640 1 -- 172.21.15.2:0/234479490 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3a3c0cbcb0 con 0x7f3a3c0d1c50 2024-01-15T17:58:23.203 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a1406b2f0 msgr2=0x7f3a1406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a1406b2f0 0x7f3a1406d7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3a2c036000 tx=0x7f3a2c03c040 comp rx=0 tx=0).stop 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 msgr2=0x7f3a3c155910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 0x7f3a3c155910 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a3400b490 tx=0x7f3a3400b950 comp rx=0 tx=0).stop 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 shutdown_connections 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a1406b2f0 0x7f3a1406d7b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a3c0d1c50 0x7f3a3c155910 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a3c0d1870 0x7f3a3c1553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 --2- 172.21.15.2:0/234479490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a3c0b2210 0x7f3a3c151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.202+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 >> 172.21.15.2:0/234479490 conn(0x7f3a3c0d30b0 msgr2=0x7f3a3c0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.203+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 shutdown_connections 2024-01-15T17:58:23.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.203+0000 7f3a41e00640 1 -- 172.21.15.2:0/234479490 wait complete. 2024-01-15T17:58:23.224 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:23.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.442+0000 7f5a35800640 1 Processor -- start 2024-01-15T17:58:23.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.443+0000 7f5a35800640 1 -- start start 2024-01-15T17:58:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.444+0000 7f5a35800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5a30072100 0x7f5a30161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.444+0000 7f5a35800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a301620d0 con 0x7f5a3014b160 2024-01-15T17:58:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.444+0000 7f5a35800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300b32d0 con 0x7f5a30070820 2024-01-15T17:58:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.444+0000 7f5a35800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a300b35b0 con 0x7f5a30072100 2024-01-15T17:58:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.445+0000 7f5a2fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5a3014b160 0x7f5a30070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35176/0 (socket says 172.21.15.2:35176) 2024-01-15T17:58:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.445+0000 7f5a2fe00640 1 -- 172.21.15.2:0/575554026 learned_addr learned my addr 172.21.15.2:0/575554026 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.445+0000 7f5a34e00640 1 --2- 172.21.15.2:0/575554026 >> v2:172.21.15.137:3300/0 conn(0x7f5a30072100 0x7f5a30161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.445+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710836954 0 0) 0x7f5a301620d0 con 0x7f5a3014b160 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.445+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a10003520 con 0x7f5a3014b160 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1185142449 0 0) 0x7f5a10003520 con 0x7f5a3014b160 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 >> v2:172.21.15.137:3300/0 conn(0x7f5a30072100 msgr2=0x7f5a30161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 --2- 172.21.15.2:0/575554026 >> v2:172.21.15.137:3300/0 conn(0x7f5a30072100 0x7f5a30161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 >> v1:172.21.15.137:6789/0 conn(0x7f5a30070820 legacy=0x7f5a3006cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3006c2f0 con 0x7f5a3014b160 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5a200019f0 con 0x7f5a3014b160 2024-01-15T17:58:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 >> v1:172.21.15.2:6789/0 conn(0x7f5a3014b160 legacy=0x7f5a30070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a10003970 0x7f5a10003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a100047c0 0x7f5a10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a10005180 0x7f5a1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300b35b0 con 0x7f5a100047c0 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1000c020 con 0x7f5a10005180 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1000c1e0 con 0x7f5a10003970 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.446+0000 7f5a2f400640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a100047c0 0x7f5a10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a2fe00640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a10003970 0x7f5a10003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a34e00640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a10005180 0x7f5a1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a2f400640 1 -- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a10003970 msgr2=0x7f5a10003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a2f400640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a10003970 0x7f5a10003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a2f400640 1 -- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a10005180 msgr2=0x7f5a1000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a2f400640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a10005180 0x7f5a1000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a2f400640 1 -- 172.21.15.2:0/575554026 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10003520 con 0x7f5a100047c0 2024-01-15T17:58:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.447+0000 7f5a34e00640 1 -- 172.21.15.2:0/575554026 reap_dead start 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.448+0000 7f5a2f400640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a100047c0 0x7f5a10004c40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5a18001750 tx=0x7f5a100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d828593a215628a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.448+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a180052d0 con 0x7f5a100047c0 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.448+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a1802e050 con 0x7f5a100047c0 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.448+0000 7f5a2ea00640 1 -- 172.21.15.2:0/575554026 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a18003830 con 0x7f5a100047c0 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 -- 172.21.15.2:0/575554026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a100047c0 msgr2=0x7f5a10004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a100047c0 0x7f5a10004c40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5a18001750 tx=0x7f5a100057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 -- 172.21.15.2:0/575554026 shutdown_connections 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 --2- 172.21.15.2:0/575554026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a100047c0 0x7f5a10004c40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 -- 172.21.15.2:0/575554026 >> 172.21.15.2:0/575554026 conn(0x7f5a300b0840 msgr2=0x7f5a300b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 -- 172.21.15.2:0/575554026 shutdown_connections 2024-01-15T17:58:23.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.449+0000 7f5a35800640 1 -- 172.21.15.2:0/575554026 wait complete. 2024-01-15T17:58:23.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.451+0000 7f5a35800640 1 Processor -- start 2024-01-15T17:58:23.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.451+0000 7f5a35800640 1 -- start start 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a35800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a100047c0 0x7f5a30149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a35800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a300adad0 0x7f5a300adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a35800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 0x7f5a300a9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a35800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301fb340 con 0x7f5a300a94b0 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a35800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a300ae490 con 0x7f5a300adad0 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a35800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a301fb760 con 0x7f5a100047c0 2024-01-15T17:58:23.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a34e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 0x7f5a300a9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a34e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 0x7f5a300a9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49180/0 (socket says 172.21.15.2:49180) 2024-01-15T17:58:23.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.452+0000 7f5a34e00640 1 -- 172.21.15.2:0/1027252939 learned_addr learned my addr 172.21.15.2:0/1027252939 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a2f400640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a300adad0 0x7f5a300adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a2fe00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a100047c0 0x7f5a30149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a34e00640 1 -- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a100047c0 msgr2=0x7f5a30149ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a34e00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a100047c0 0x7f5a30149ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a34e00640 1 -- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a300adad0 msgr2=0x7f5a300adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a34e00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a300adad0 0x7f5a300adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.453+0000 7f5a34e00640 1 -- 172.21.15.2:0/1027252939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300ae670 con 0x7f5a300a94b0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.454+0000 7f5a34e00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 0x7f5a300a9940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5a30072590 tx=0x7f5a28011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.454+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28004500 con 0x7f5a300a94b0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.454+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28003610 con 0x7f5a300a94b0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.454+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a280039c0 con 0x7f5a300a94b0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.454+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a301fa9a0 con 0x7f5a300a94b0 2024-01-15T17:58:23.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.454+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a301faf60 con 0x7f5a300a94b0 2024-01-15T17:58:23.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.455+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a301fb480 con 0x7f5a300a94b0 2024-01-15T17:58:23.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.456+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a28003b20 con 0x7f5a300a94b0 2024-01-15T17:58:23.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.457+0000 7f5a27e00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a0806b2c0 0x7f5a0806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.457+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5a2808a0f0 con 0x7f5a300a94b0 2024-01-15T17:58:23.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.458+0000 7f5a2fe00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a0806b2c0 0x7f5a0806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.459+0000 7f5a2fe00640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a0806b2c0 0x7f5a0806d780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a20002010 tx=0x7f5a20009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.461+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a301fb480 con 0x7f5a300a94b0 2024-01-15T17:58:23.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.654+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5a301fb480 con 0x7f5a300a94b0 2024-01-15T17:58:23.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.655+0000 7f5a27e00640 1 -- 172.21.15.2:0/1027252939 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5a301fb480 con 0x7f5a300a94b0 2024-01-15T17:58:23.658 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a0806b2c0 msgr2=0x7f5a0806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a0806b2c0 0x7f5a0806d780 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a20002010 tx=0x7f5a20009040 comp rx=0 tx=0).stop 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 msgr2=0x7f5a300a9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 0x7f5a300a9940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5a30072590 tx=0x7f5a28011920 comp rx=0 tx=0).stop 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 shutdown_connections 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a0806b2c0 0x7f5a0806d780 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a300a94b0 0x7f5a300a9940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a300adad0 0x7f5a300adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 --2- 172.21.15.2:0/1027252939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a100047c0 0x7f5a30149ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.657+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 >> 172.21.15.2:0/1027252939 conn(0x7f5a300b0840 msgr2=0x7f5a30145fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.658+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 shutdown_connections 2024-01-15T17:58:23.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.658+0000 7f5a35800640 1 -- 172.21.15.2:0/1027252939 wait complete. 2024-01-15T17:58:23.677 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:23.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.883+0000 7fb7b1600640 1 Processor -- start 2024-01-15T17:58:23.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.883+0000 7fb7b1600640 1 -- start start 2024-01-15T17:58:23.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.884+0000 7fb7b1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 0x7fb7ac0aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.884+0000 7fb7b1600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb7ac06b7a0 0x7fb7ac0661f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.885+0000 7fb7b1600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7ac15a070 con 0x7fb7ac0b2840 2024-01-15T17:58:23.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.885+0000 7fb7b1600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ac06a4b0 con 0x7fb7ac06b7a0 2024-01-15T17:58:23.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.885+0000 7fb7b1600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac06a700 con 0x7fb7ac159c90 2024-01-15T17:58:23.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.885+0000 7fb7abe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 0x7fb7ac0aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.885+0000 7fb7ab400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb7ac06b7a0 0x7fb7ac0661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.886+0000 7fb7abe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 0x7fb7ac0aeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:53110/0 (socket says 172.21.15.2:53110) 2024-01-15T17:58:23.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.886+0000 7fb7abe00640 1 -- 172.21.15.2:0/1560523253 learned_addr learned my addr 172.21.15.2:0/1560523253 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:23.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.886+0000 7fb7abe00640 1 -- 172.21.15.2:0/1560523253 >> v2:172.21.15.137:3300/0 conn(0x7fb7ac06b7a0 msgr2=0x7fb7ac0661f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:23.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.886+0000 7fb7abe00640 1 --2- 172.21.15.2:0/1560523253 >> v2:172.21.15.137:3300/0 conn(0x7fb7ac06b7a0 0x7fb7ac0661f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.886+0000 7fb7abe00640 1 -- 172.21.15.2:0/1560523253 >> v1:172.21.15.137:6789/0 conn(0x7fb7ac0b2840 legacy=0x7fb7ac0b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.886+0000 7fb7abe00640 1 -- 172.21.15.2:0/1560523253 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ac06a9e0 con 0x7fb7ac159c90 2024-01-15T17:58:23.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.888+0000 7fb7abe00640 1 --2- 172.21.15.2:0/1560523253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 0x7fb7ac0aeb60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00bb10 tx=0x7fb79c02e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=355b6765b255cf84 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.889+0000 7fb7aaa00640 1 -- 172.21.15.2:0/1560523253 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c036440 con 0x7fb7ac159c90 2024-01-15T17:58:23.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.889+0000 7fb7aaa00640 1 -- 172.21.15.2:0/1560523253 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79c0365a0 con 0x7fb7ac159c90 2024-01-15T17:58:23.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.889+0000 7fb7aaa00640 1 -- 172.21.15.2:0/1560523253 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c036930 con 0x7fb7ac159c90 2024-01-15T17:58:23.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 -- 172.21.15.2:0/1560523253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 msgr2=0x7fb7ac0aeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 --2- 172.21.15.2:0/1560523253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 0x7fb7ac0aeb60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb79c00bb10 tx=0x7fb79c02e150 comp rx=0 tx=0).stop 2024-01-15T17:58:23.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 -- 172.21.15.2:0/1560523253 shutdown_connections 2024-01-15T17:58:23.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 --2- 172.21.15.2:0/1560523253 >> v2:172.21.15.137:3300/0 conn(0x7fb7ac06b7a0 0x7fb7ac0661f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 --2- 172.21.15.2:0/1560523253 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac159c90 0x7fb7ac0aeb60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 -- 172.21.15.2:0/1560523253 >> 172.21.15.2:0/1560523253 conn(0x7fb7ac0cbf70 msgr2=0x7fb7ac0cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:23.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 -- 172.21.15.2:0/1560523253 shutdown_connections 2024-01-15T17:58:23.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.890+0000 7fb7b1600640 1 -- 172.21.15.2:0/1560523253 wait complete. 2024-01-15T17:58:23.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.892+0000 7fb7b1600640 1 Processor -- start 2024-01-15T17:58:23.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.893+0000 7fb7b1600640 1 -- start start 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac06b7a0 0x7fb7ac1e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b1600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7ac0b2840 0x7fb7ac155ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 0x7fb7ac1e77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b1600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1e4090 con 0x7fb7ac159c90 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b1600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1e4370 con 0x7fb7ac0b2840 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b1600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ac1e4650 con 0x7fb7ac06b7a0 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 0x7fb7ac1e77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 0x7fb7ac1e77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49192/0 (socket says 172.21.15.2:49192) 2024-01-15T17:58:23.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7b0c00640 1 -- 172.21.15.2:0/3908104844 learned_addr learned my addr 172.21.15.2:0/3908104844 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:23.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7abe00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac06b7a0 0x7fb7ac1e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.894+0000 7fb7ab400640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7ac0b2840 0x7fb7ac155ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b0c00640 1 -- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac06b7a0 msgr2=0x7fb7ac1e3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b0c00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac06b7a0 0x7fb7ac1e3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b0c00640 1 -- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7ac0b2840 msgr2=0x7fb7ac155ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b0c00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7ac0b2840 0x7fb7ac155ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b0c00640 1 -- 172.21.15.2:0/3908104844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79c00b770 con 0x7fb7ac159c90 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b0c00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 0x7fb7ac1e77a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb7a400c710 tx=0x7fb7a400cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4006d50 con 0x7fb7ac159c90 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a400ce20 con 0x7fb7ac159c90 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7a4003bb0 con 0x7fb7ac159c90 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.895+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ac1e4930 con 0x7fb7ac159c90 2024-01-15T17:58:23.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.896+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ac1e7ce0 con 0x7fb7ac159c90 2024-01-15T17:58:23.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.897+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac1e4f10 con 0x7fb7ac159c90 2024-01-15T17:58:23.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.898+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb7a4003d10 con 0x7fb7ac159c90 2024-01-15T17:58:23.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.898+0000 7fb7a3e00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb78406b2c0 0x7fb78406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:23.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.899+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb7a40896a0 con 0x7fb7ac159c90 2024-01-15T17:58:23.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.899+0000 7fb7abe00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb78406b2c0 0x7fb78406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:23.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.900+0000 7fb7abe00640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb78406b2c0 0x7fb78406d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb79c038000 tx=0x7fb79c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:23.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:23.905+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7ac1e4f10 con 0x7fb7ac159c90 2024-01-15T17:58:24.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.097+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb7ac15a070 con 0x7fb7ac159c90 2024-01-15T17:58:24.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.098+0000 7fb7a3e00640 1 -- 172.21.15.2:0/3908104844 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb7ac15a070 con 0x7fb7ac159c90 2024-01-15T17:58:24.101 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb78406b2c0 msgr2=0x7fb78406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb78406b2c0 0x7fb78406d780 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb79c038000 tx=0x7fb79c008040 comp rx=0 tx=0).stop 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 msgr2=0x7fb7ac1e77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 0x7fb7ac1e77a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb7a400c710 tx=0x7fb7a400cbd0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 shutdown_connections 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb78406b2c0 0x7fb78406d780 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7ac159c90 0x7fb7ac1e77a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7ac0b2840 0x7fb7ac155ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 --2- 172.21.15.2:0/3908104844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7ac06b7a0 0x7fb7ac1e3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 >> 172.21.15.2:0/3908104844 conn(0x7fb7ac0cbf70 msgr2=0x7fb7ac157890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:24.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 shutdown_connections 2024-01-15T17:58:24.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.101+0000 7fb7b1600640 1 -- 172.21.15.2:0/3908104844 wait complete. 2024-01-15T17:58:24.124 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:24.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.339+0000 7f25bde00640 1 Processor -- start 2024-01-15T17:58:24.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.340+0000 7f25bde00640 1 -- start start 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bde00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f25b806d350 0x7f25b806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f25b8160e50 0x7f25b80cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bde00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25b806dc70 con 0x7f25b80ce3b0 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bde00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b80cb3a0 con 0x7f25b8160e50 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bde00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25b80cb680 con 0x7f25b806d350 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bd400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f25b80ce3b0 0x7f25b806c130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35184/0 (socket says 172.21.15.2:35184) 2024-01-15T17:58:24.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.342+0000 7f25bd400640 1 -- 172.21.15.2:0/1833146162 learned_addr learned my addr 172.21.15.2:0/1833146162 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:24.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25afe00640 1 --2- 172.21.15.2:0/1833146162 >> v2:172.21.15.2:3300/0 conn(0x7f25b8160e50 0x7f25b80cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25bca00640 1 --2- 172.21.15.2:0/1833146162 >> v2:172.21.15.137:3300/0 conn(0x7f25b806d350 0x7f25b806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25b7e00640 1 -- 172.21.15.2:0/1833146162 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250688899 0 0) 0x7f25b806dc70 con 0x7f25b80ce3b0 2024-01-15T17:58:24.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25b7e00640 1 -- 172.21.15.2:0/1833146162 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2598003520 con 0x7f25b80ce3b0 2024-01-15T17:58:24.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25afe00640 1 -- 172.21.15.2:0/1833146162 >> v2:172.21.15.137:3300/0 conn(0x7f25b806d350 msgr2=0x7f25b806d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:24.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25afe00640 1 --2- 172.21.15.2:0/1833146162 >> v2:172.21.15.137:3300/0 conn(0x7f25b806d350 0x7f25b806d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25afe00640 1 -- 172.21.15.2:0/1833146162 >> v1:172.21.15.2:6789/0 conn(0x7f25b80ce3b0 legacy=0x7f25b806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.343+0000 7f25afe00640 1 -- 172.21.15.2:0/1833146162 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80cb960 con 0x7f25b8160e50 2024-01-15T17:58:24.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.344+0000 7f25afe00640 1 --2- 172.21.15.2:0/1833146162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b8160e50 0x7f25b80cde70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f25b806bb60 tx=0x7f25a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e705b3f4b2c3fbb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:24.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25b7e00640 1 -- 172.21.15.2:0/1833146162 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a002e070 con 0x7f25b8160e50 2024-01-15T17:58:24.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25b7e00640 1 -- 172.21.15.2:0/1833146162 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a00315a0 con 0x7f25b8160e50 2024-01-15T17:58:24.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25b7e00640 1 -- 172.21.15.2:0/1833146162 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0031930 con 0x7f25b8160e50 2024-01-15T17:58:24.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 -- 172.21.15.2:0/1833146162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b8160e50 msgr2=0x7f25b80cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 --2- 172.21.15.2:0/1833146162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b8160e50 0x7f25b80cde70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f25b806bb60 tx=0x7f25a002b840 comp rx=0 tx=0).stop 2024-01-15T17:58:24.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 -- 172.21.15.2:0/1833146162 shutdown_connections 2024-01-15T17:58:24.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 --2- 172.21.15.2:0/1833146162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b8160e50 0x7f25b80cde70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 --2- 172.21.15.2:0/1833146162 >> v2:172.21.15.137:3300/0 conn(0x7f25b806d350 0x7f25b806d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 -- 172.21.15.2:0/1833146162 >> 172.21.15.2:0/1833146162 conn(0x7f25b80b0840 msgr2=0x7f25b80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:24.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 -- 172.21.15.2:0/1833146162 shutdown_connections 2024-01-15T17:58:24.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.345+0000 7f25bde00640 1 -- 172.21.15.2:0/1833146162 wait complete. 2024-01-15T17:58:24.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.347+0000 7f25bde00640 1 Processor -- start 2024-01-15T17:58:24.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.347+0000 7f25bde00640 1 -- start start 2024-01-15T17:58:24.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.348+0000 7f25bde00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25b806d350 0x7f25b813f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.348+0000 7f25bde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 0x7f25b8154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.348+0000 7f25bde00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25b8160e50 0x7f25b806c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.348+0000 7f25bde00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8070440 con 0x7f25b80ce3b0 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.348+0000 7f25bde00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8155360 con 0x7f25b8160e50 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.348+0000 7f25bde00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25b8155640 con 0x7f25b806d350 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 0x7f25b8154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 0x7f25b8154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49220/0 (socket says 172.21.15.2:49220) 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 -- 172.21.15.2:0/218576502 learned_addr learned my addr 172.21.15.2:0/218576502 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25bca00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25b806d350 0x7f25b813f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25bd400640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25b8160e50 0x7f25b806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 -- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25b806d350 msgr2=0x7f25b813f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25b806d350 0x7f25b813f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 -- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25b8160e50 msgr2=0x7f25b806c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25b8160e50 0x7f25b806c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25afe00640 1 -- 172.21.15.2:0/218576502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0009400 con 0x7f25b80ce3b0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.349+0000 7f25bca00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25b806d350 0x7f25b813f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.350+0000 7f25afe00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 0x7f25b8154980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f25a0000c00 tx=0x7f25a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.350+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0032840 con 0x7f25b80ce3b0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.350+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a0031f00 con 0x7f25b80ce3b0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.350+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25b8155920 con 0x7f25b80ce3b0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.350+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a0032c00 con 0x7f25b80ce3b0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.350+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25b813fb10 con 0x7f25b80ce3b0 2024-01-15T17:58:24.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.351+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25b806bfa0 con 0x7f25b80ce3b0 2024-01-15T17:58:24.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.352+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25a004d3f0 con 0x7f25b80ce3b0 2024-01-15T17:58:24.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.353+0000 7f25b5600640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f259006b2c0 0x7f259006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.353+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f25a00ab7e0 con 0x7f25b80ce3b0 2024-01-15T17:58:24.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.354+0000 7f25bca00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f259006b2c0 0x7f259006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.355+0000 7f25bca00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f259006b2c0 0x7f259006d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009af0 tx=0x7f25a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:24.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.357+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25a00764a0 con 0x7f25b80ce3b0 2024-01-15T17:58:24.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.568+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f25b8074770 con 0x7f25b80ce3b0 2024-01-15T17:58:24.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.568+0000 7f25b5600640 1 -- 172.21.15.2:0/218576502 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f25a0079ac0 con 0x7f25b80ce3b0 2024-01-15T17:58:24.571 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:24.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f259006b2c0 msgr2=0x7f259006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f259006b2c0 0x7f259006d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f25a8009af0 tx=0x7f25a8008040 comp rx=0 tx=0).stop 2024-01-15T17:58:24.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 msgr2=0x7f25b8154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 0x7f25b8154980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f25a0000c00 tx=0x7f25a00046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 shutdown_connections 2024-01-15T17:58:24.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f259006b2c0 0x7f259006d780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25b8160e50 0x7f25b806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25b80ce3b0 0x7f25b8154980 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 --2- 172.21.15.2:0/218576502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25b806d350 0x7f25b813f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 >> 172.21.15.2:0/218576502 conn(0x7f25b80b0840 msgr2=0x7f25b80ad1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:24.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.571+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 shutdown_connections 2024-01-15T17:58:24.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.572+0000 7f25bde00640 1 -- 172.21.15.2:0/218576502 wait complete. 2024-01-15T17:58:24.592 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:58:24.592 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:24.651 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:24.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.871+0000 7f536b200640 1 Processor -- start 2024-01-15T17:58:24.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.872+0000 7f536b200640 1 -- start start 2024-01-15T17:58:24.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.873+0000 7f536b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f536c141880 0x7f536c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.873+0000 7f536b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f536c15f0e0 0x7f536c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.873+0000 7f536b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c06dae0 con 0x7f536c15ec60 2024-01-15T17:58:24.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.874+0000 7f536b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536c071f20 con 0x7f536c15f0e0 2024-01-15T17:58:24.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.874+0000 7f536b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536c06c6b0 con 0x7f536c141880 2024-01-15T17:58:24.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.874+0000 7f536a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f536c15f0e0 0x7f536c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.874+0000 7f536a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f536c15f0e0 0x7f536c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49232/0 (socket says 172.21.15.2:49232) 2024-01-15T17:58:24.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.874+0000 7f536a800640 1 -- 172.21.15.2:0/3176232131 learned_addr learned my addr 172.21.15.2:0/3176232131 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:24.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.874+0000 7f5369e00640 1 --2- 172.21.15.2:0/3176232131 >> v2:172.21.15.137:3300/0 conn(0x7f536c141880 0x7f536c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.875+0000 7f536a800640 1 -- 172.21.15.2:0/3176232131 >> v2:172.21.15.137:3300/0 conn(0x7f536c141880 msgr2=0x7f536c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:24.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.875+0000 7f536a800640 1 --2- 172.21.15.2:0/3176232131 >> v2:172.21.15.137:3300/0 conn(0x7f536c141880 0x7f536c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.875+0000 7f536a800640 1 -- 172.21.15.2:0/3176232131 >> v1:172.21.15.137:6789/0 conn(0x7f536c15ec60 legacy=0x7f536c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:24.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.875+0000 7f536a800640 1 -- 172.21.15.2:0/3176232131 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c071960 con 0x7f536c15f0e0 2024-01-15T17:58:24.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.876+0000 7f536a800640 1 --2- 172.21.15.2:0/3176232131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0cdba0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f536c068540 tx=0x7f536402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a299466a6f594c9d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:24.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.876+0000 7f5369400640 1 -- 172.21.15.2:0/3176232131 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536402c9f0 con 0x7f536c15f0e0 2024-01-15T17:58:24.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.876+0000 7f5369400640 1 -- 172.21.15.2:0/3176232131 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536402cb50 con 0x7f536c15f0e0 2024-01-15T17:58:24.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.876+0000 7f5369400640 1 -- 172.21.15.2:0/3176232131 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53640343f0 con 0x7f536c15f0e0 2024-01-15T17:58:24.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 -- 172.21.15.2:0/3176232131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 msgr2=0x7f536c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 --2- 172.21.15.2:0/3176232131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0cdba0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f536c068540 tx=0x7f536402be50 comp rx=0 tx=0).stop 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 -- 172.21.15.2:0/3176232131 shutdown_connections 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 --2- 172.21.15.2:0/3176232131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0cdba0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 --2- 172.21.15.2:0/3176232131 >> v2:172.21.15.137:3300/0 conn(0x7f536c141880 0x7f536c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 -- 172.21.15.2:0/3176232131 >> 172.21.15.2:0/3176232131 conn(0x7f536c0b0840 msgr2=0x7f536c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 -- 172.21.15.2:0/3176232131 shutdown_connections 2024-01-15T17:58:24.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.877+0000 7f536b200640 1 -- 172.21.15.2:0/3176232131 wait complete. 2024-01-15T17:58:24.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.879+0000 7f536b200640 1 Processor -- start 2024-01-15T17:58:24.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.879+0000 7f536b200640 1 -- start start 2024-01-15T17:58:24.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f536c141880 0x7f536c131660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536c15ec60 0x7f536c1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1f34d0 con 0x7f536c15f0e0 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c142710 con 0x7f536c141880 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536c1429f0 con 0x7f536c15ec60 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49248/0 (socket says 172.21.15.2:49248) 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f536a800640 1 -- 172.21.15.2:0/2575394123 learned_addr learned my addr 172.21.15.2:0/2575394123 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f5369e00640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f536c141880 0x7f536c131660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.880+0000 7f5363e00640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536c15ec60 0x7f536c1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536a800640 1 -- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536c15ec60 msgr2=0x7f536c1f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536a800640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536c15ec60 0x7f536c1f2f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536a800640 1 -- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f536c141880 msgr2=0x7f536c131660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:24.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536a800640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f536c141880 0x7f536c131660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:24.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536a800640 1 -- 172.21.15.2:0/2575394123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009b30 con 0x7f536c15f0e0 2024-01-15T17:58:24.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536a800640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0afa90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5364036040 tx=0x7f5364004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:24.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364009510 con 0x7f536c15f0e0 2024-01-15T17:58:24.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536403e450 con 0x7f536c15f0e0 2024-01-15T17:58:24.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c142cd0 con 0x7f536c15f0e0 2024-01-15T17:58:24.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.881+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536403e7e0 con 0x7f536c15f0e0 2024-01-15T17:58:24.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.882+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c143290 con 0x7f536c15f0e0 2024-01-15T17:58:24.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.883+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c143480 con 0x7f536c15f0e0 2024-01-15T17:58:24.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.884+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f536403e940 con 0x7f536c15f0e0 2024-01-15T17:58:24.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.885+0000 7f5362000640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f534406b340 0x7f534406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:24.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.885+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f53640ac680 con 0x7f536c15f0e0 2024-01-15T17:58:24.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.886+0000 7f5369e00640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f534406b340 0x7f534406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:24.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.887+0000 7f5369e00640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f534406b340 0x7f534406d800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f535c006fd0 tx=0x7f535c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:24.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:24.890+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5364077550 con 0x7f536c15f0e0 2024-01-15T17:58:25.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.086+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f536c074770 con 0x7f536c15f0e0 2024-01-15T17:58:25.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.090+0000 7f5362000640 1 -- 172.21.15.2:0/2575394123 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f5364033e80 con 0x7f536c15f0e0 2024-01-15T17:58:25.093 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:25.093 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.092+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f534406b340 msgr2=0x7f534406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.092+0000 7f536b200640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f534406b340 0x7f534406d800 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f535c006fd0 tx=0x7f535c008040 comp rx=0 tx=0).stop 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 msgr2=0x7f536c0afa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0afa90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5364036040 tx=0x7f5364004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 shutdown_connections 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f534406b340 0x7f534406d800 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f536c15f0e0 0x7f536c0afa90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f536c15ec60 0x7f536c1f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 --2- 172.21.15.2:0/2575394123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f536c141880 0x7f536c131660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 >> 172.21.15.2:0/2575394123 conn(0x7f536c0b0840 msgr2=0x7f536c0ad9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:25.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 shutdown_connections 2024-01-15T17:58:25.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.093+0000 7f536b200640 1 -- 172.21.15.2:0/2575394123 wait complete. 2024-01-15T17:58:25.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:25.117 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:25.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.337+0000 7f2c02400640 1 Processor -- start 2024-01-15T17:58:25.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.337+0000 7f2c02400640 1 -- start start 2024-01-15T17:58:25.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.338+0000 7f2c02400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2bfc0b40b0 0x7f2bfc168110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2c02400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc0d0970 con 0x7f2bfc160c50 2024-01-15T17:58:25.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2c02400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc0b2c10 con 0x7f2bfc160870 2024-01-15T17:58:25.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2c02400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bfc0d0430 con 0x7f2bfc0b40b0 2024-01-15T17:58:25.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2bfbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2bfc160c50 0x7f2bfc0cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35188/0 (socket says 172.21.15.2:35188) 2024-01-15T17:58:25.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2bfbe00640 1 -- 172.21.15.2:0/2912049588 learned_addr learned my addr 172.21.15.2:0/2912049588 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2c01a00640 1 --2- 172.21.15.2:0/2912049588 >> v2:172.21.15.137:3300/0 conn(0x7f2bfc0b40b0 0x7f2bfc168110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29707573 0 0) 0x7f2be4001110 con 0x7f2bfc160c50 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.339+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bdc003520 con 0x7f2bfc160c50 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2492062942 0 0) 0x7f2bdc003520 con 0x7f2bfc160c50 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 >> v2:172.21.15.137:3300/0 conn(0x7f2bfc0b40b0 msgr2=0x7f2bfc168110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 --2- 172.21.15.2:0/2912049588 >> v2:172.21.15.137:3300/0 conn(0x7f2bfc0b40b0 0x7f2bfc168110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 >> v1:172.21.15.137:6789/0 conn(0x7f2bfc160870 legacy=0x7f2bfc0cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0ceb00 con 0x7f2bfc160c50 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2be4001800 con 0x7f2bfc160c50 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 >> v1:172.21.15.2:6789/0 conn(0x7f2bfc160c50 legacy=0x7f2bfc0cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bdc0039c0 0x7f2bdc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0047c0 0x7f2bdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bdc005180 0x7f2bdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0d0430 con 0x7f2bdc0039c0 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00c080 con 0x7f2bdc005180 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bdc00c240 con 0x7f2bdc0047c0 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.340+0000 7f2c01000640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bdc0039c0 0x7f2bdc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01a00640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bdc005180 0x7f2bdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2bfbe00640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0047c0 0x7f2bdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01000640 1 -- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0047c0 msgr2=0x7f2bdc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01000640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0047c0 0x7f2bdc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01000640 1 -- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bdc005180 msgr2=0x7f2bdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01000640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bdc005180 0x7f2bdc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01a00640 1 -- 172.21.15.2:0/2912049588 reap_dead start 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2c01000640 1 -- 172.21.15.2:0/2912049588 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc003520 con 0x7f2bdc0039c0 2024-01-15T17:58:25.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.341+0000 7f2bfbe00640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0047c0 0x7f2bdc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:25.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.342+0000 7f2c01000640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bdc0039c0 0x7f2bdc003de0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2bec001750 tx=0x7f2bdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e10dddbd2963ed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.342+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0052d0 con 0x7f2bdc0039c0 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.342+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec02e050 con 0x7f2bdc0039c0 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.342+0000 7f2bfb400640 1 -- 172.21.15.2:0/2912049588 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec003830 con 0x7f2bdc0039c0 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 -- 172.21.15.2:0/2912049588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bdc0039c0 msgr2=0x7f2bdc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bdc0039c0 0x7f2bdc003de0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2bec001750 tx=0x7f2bdc0057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 -- 172.21.15.2:0/2912049588 shutdown_connections 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 --2- 172.21.15.2:0/2912049588 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bdc0039c0 0x7f2bdc003de0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 -- 172.21.15.2:0/2912049588 >> 172.21.15.2:0/2912049588 conn(0x7f2bfc0d2fe0 msgr2=0x7f2bfc0d33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 -- 172.21.15.2:0/2912049588 shutdown_connections 2024-01-15T17:58:25.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.343+0000 7f2c02400640 1 -- 172.21.15.2:0/2912049588 wait complete. 2024-01-15T17:58:25.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.345+0000 7f2c02400640 1 Processor -- start 2024-01-15T17:58:25.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.345+0000 7f2c02400640 1 -- start start 2024-01-15T17:58:25.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c02400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0039c0 0x7f2bfc0c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c02400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bfc0b6710 0x7f2bfc14cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c02400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 0x7f2bfc0c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c02400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0c9630 con 0x7f2bfc150240 2024-01-15T17:58:25.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c02400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0ccba0 con 0x7f2bfc0b6710 2024-01-15T17:58:25.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c02400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc14d510 con 0x7f2bdc0039c0 2024-01-15T17:58:25.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.346+0000 7f2c01a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 0x7f2bfc0c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 0x7f2bfc0c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49276/0 (socket says 172.21.15.2:49276) 2024-01-15T17:58:25.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 -- 172.21.15.2:0/1883332902 learned_addr learned my addr 172.21.15.2:0/1883332902 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:25.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01000640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0039c0 0x7f2bfc0c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2bfbe00640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bfc0b6710 0x7f2bfc14cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 -- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0039c0 msgr2=0x7f2bfc0c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0039c0 0x7f2bfc0c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 -- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bfc0b6710 msgr2=0x7f2bfc14cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bfc0b6710 0x7f2bfc14cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 -- 172.21.15.2:0/1883332902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0c89b0 con 0x7f2bfc150240 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.347+0000 7f2c01a00640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 0x7f2bfc0c66b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc08cbe0 tx=0x7f2bf400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.348+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4016420 con 0x7f2bfc150240 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.348+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf4016580 con 0x7f2bfc150240 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.348+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc0c8c90 con 0x7f2bfc150240 2024-01-15T17:58:25.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.348+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4016930 con 0x7f2bfc150240 2024-01-15T17:58:25.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.348+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc0c9250 con 0x7f2bfc150240 2024-01-15T17:58:25.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.349+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc0c9770 con 0x7f2bfc150240 2024-01-15T17:58:25.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.350+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2bf4016a90 con 0x7f2bfc150240 2024-01-15T17:58:25.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.351+0000 7f2bf8c00640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bd406b2c0 0x7f2bd406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.351+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2bf408ab20 con 0x7f2bfc150240 2024-01-15T17:58:25.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.352+0000 7f2c01000640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bd406b2c0 0x7f2bd406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.353+0000 7f2c01000640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bd406b2c0 0x7f2bd406d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2bec033000 tx=0x7f2bec009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:25.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.356+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bfc0c9770 con 0x7f2bfc150240 2024-01-15T17:58:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.547+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bfc0c9770 con 0x7f2bfc150240 2024-01-15T17:58:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.548+0000 7f2bf8c00640 1 -- 172.21.15.2:0/1883332902 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bfc0c9770 con 0x7f2bfc150240 2024-01-15T17:58:25.551 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bd406b2c0 msgr2=0x7f2bd406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bd406b2c0 0x7f2bd406d780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2bec033000 tx=0x7f2bec009030 comp rx=0 tx=0).stop 2024-01-15T17:58:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 msgr2=0x7f2bfc0c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 0x7f2bfc0c66b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc08cbe0 tx=0x7f2bf400f520 comp rx=0 tx=0).stop 2024-01-15T17:58:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 shutdown_connections 2024-01-15T17:58:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bd406b2c0 0x7f2bd406d780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2bfc150240 0x7f2bfc0c66b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2bfc0b6710 0x7f2bfc14cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 --2- 172.21.15.2:0/1883332902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2bdc0039c0 0x7f2bfc0c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 >> 172.21.15.2:0/1883332902 conn(0x7f2bfc0d2fe0 msgr2=0x7f2bfc15de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.551+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 shutdown_connections 2024-01-15T17:58:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.552+0000 7f2c02400640 1 -- 172.21.15.2:0/1883332902 wait complete. 2024-01-15T17:58:25.575 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.789+0000 7f877d600640 1 Processor -- start 2024-01-15T17:58:25.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.789+0000 7f877d600640 1 -- start start 2024-01-15T17:58:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f877d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b4c70 0x7f87780d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f877d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f877815d730 0x7f87780b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f877d600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87780b5590 con 0x7f87780b8c30 2024-01-15T17:58:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f877d600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87780d30b0 con 0x7f877815d730 2024-01-15T17:58:25.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f877d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87780d1900 con 0x7f87780b4c70 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f876fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f877815d730 0x7f87780b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f876fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f877815d730 0x7f87780b5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49278/0 (socket says 172.21.15.2:49278) 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.791+0000 7f876fe00640 1 -- 172.21.15.2:0/2377321768 learned_addr learned my addr 172.21.15.2:0/2377321768 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.792+0000 7f8776a00640 1 --2- 172.21.15.2:0/2377321768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b4c70 0x7f87780d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.792+0000 7f876fe00640 1 -- 172.21.15.2:0/2377321768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b4c70 msgr2=0x7f87780d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.792+0000 7f876fe00640 1 --2- 172.21.15.2:0/2377321768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b4c70 0x7f87780d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.792+0000 7f876fe00640 1 -- 172.21.15.2:0/2377321768 >> v1:172.21.15.2:6789/0 conn(0x7f87780b8c30 legacy=0x7f87780d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.792+0000 7f876fe00640 1 -- 172.21.15.2:0/2377321768 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87780d1be0 con 0x7f877815d730 2024-01-15T17:58:25.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.793+0000 7f876fe00640 1 --2- 172.21.15.2:0/2377321768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87780b5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8760009870 tx=0x7f876002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51938c1e155b350 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.794+0000 7f8776000640 1 -- 172.21.15.2:0/2377321768 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8760030840 con 0x7f877815d730 2024-01-15T17:58:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.794+0000 7f8776000640 1 -- 172.21.15.2:0/2377321768 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87600309a0 con 0x7f877815d730 2024-01-15T17:58:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.794+0000 7f8776000640 1 -- 172.21.15.2:0/2377321768 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8760030d30 con 0x7f877815d730 2024-01-15T17:58:25.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.794+0000 7f877d600640 1 -- 172.21.15.2:0/2377321768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 msgr2=0x7f87780b5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.794+0000 7f877d600640 1 --2- 172.21.15.2:0/2377321768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87780b5050 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8760009870 tx=0x7f876002b980 comp rx=0 tx=0).stop 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.795+0000 7f877d600640 1 -- 172.21.15.2:0/2377321768 shutdown_connections 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.795+0000 7f877d600640 1 --2- 172.21.15.2:0/2377321768 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87780b5050 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.795+0000 7f877d600640 1 --2- 172.21.15.2:0/2377321768 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b4c70 0x7f87780d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.795+0000 7f877d600640 1 -- 172.21.15.2:0/2377321768 >> 172.21.15.2:0/2377321768 conn(0x7f8778072210 msgr2=0x7f87780cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.795+0000 7f877d600640 1 -- 172.21.15.2:0/2377321768 shutdown_connections 2024-01-15T17:58:25.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.795+0000 7f877d600640 1 -- 172.21.15.2:0/2377321768 wait complete. 2024-01-15T17:58:25.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.796+0000 7f877d600640 1 Processor -- start 2024-01-15T17:58:25.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.797+0000 7f877d600640 1 -- start start 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f877d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87780b4c70 0x7f87780cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f877d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b8c30 0x7f877814c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f877d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87781f5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f877d600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877814c710 con 0x7f877815d730 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f877d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87780cc2c0 con 0x7f87780b4c70 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f877d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87780cc5a0 con 0x7f87780b8c30 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f8777400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87781f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f8777400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87781f5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49286/0 (socket says 172.21.15.2:49286) 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f8777400640 1 -- 172.21.15.2:0/390130729 learned_addr learned my addr 172.21.15.2:0/390130729 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f876fe00640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b8c30 0x7f877814c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.798+0000 7f8776a00640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87780b4c70 0x7f87780cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f8777400640 1 -- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b8c30 msgr2=0x7f877814c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f8777400640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b8c30 0x7f877814c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f8777400640 1 -- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87780b4c70 msgr2=0x7f87780cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f8777400640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87780b4c70 0x7f87780cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f8777400640 1 -- 172.21.15.2:0/390130729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600094d0 con 0x7f877815d730 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f8777400640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87781f5750 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f877000c710 tx=0x7f877000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770006d50 con 0x7f877815d730 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877000ce20 con 0x7f877815d730 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87780cc880 con 0x7f877815d730 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770003bb0 con 0x7f877815d730 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.799+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87780cce40 con 0x7f877815d730 2024-01-15T17:58:25.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.801+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87780cd030 con 0x7f877815d730 2024-01-15T17:58:25.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.802+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8770003d10 con 0x7f877815d730 2024-01-15T17:58:25.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.803+0000 7f876ea00640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f875006b2c0 0x7f875006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:25.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.803+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8770089820 con 0x7f877815d730 2024-01-15T17:58:25.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.804+0000 7f8776a00640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f875006b2c0 0x7f875006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:25.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.805+0000 7f8776a00640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f875006b2c0 0x7f875006d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8768006fd0 tx=0x7f8768008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:25.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:25.808+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87700580a0 con 0x7f877815d730 2024-01-15T17:58:26.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.001+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f87780686f0 con 0x7f877815d730 2024-01-15T17:58:26.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.002+0000 7f876ea00640 1 -- 172.21.15.2:0/390130729 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8770057c30 con 0x7f877815d730 2024-01-15T17:58:26.006 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:26.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f875006b2c0 msgr2=0x7f875006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f875006b2c0 0x7f875006d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8768006fd0 tx=0x7f8768008040 comp rx=0 tx=0).stop 2024-01-15T17:58:26.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 msgr2=0x7f87781f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87781f5750 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f877000c710 tx=0x7f877000cbd0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 shutdown_connections 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f875006b2c0 0x7f875006d780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f877815d730 0x7f87781f5750 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87780b8c30 0x7f877814c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 --2- 172.21.15.2:0/390130729 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87780b4c70 0x7f87780cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.005+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 >> 172.21.15.2:0/390130729 conn(0x7f8778072210 msgr2=0x7f87780d29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.006+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 shutdown_connections 2024-01-15T17:58:26.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.006+0000 7f877d600640 1 -- 172.21.15.2:0/390130729 wait complete. 2024-01-15T17:58:26.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:26.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.247+0000 7f5b21200640 1 Processor -- start 2024-01-15T17:58:26.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.247+0000 7f5b21200640 1 -- start start 2024-01-15T17:58:26.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.249+0000 7f5b21200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5b1c06a4b0 0x7f5b1c06a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.249+0000 7f5b21200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c06ae40 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.249+0000 7f5b21200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b1c071a20 con 0x7f5b1c15d330 2024-01-15T17:58:26.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.249+0000 7f5b21200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1c0af820 con 0x7f5b1c06a4b0 2024-01-15T17:58:26.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.249+0000 7f5b1aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5b1c0b2f60 0x7f5b1c06cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35218/0 (socket says 172.21.15.2:35218) 2024-01-15T17:58:26.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.249+0000 7f5b1aa00640 1 -- 172.21.15.2:0/2225845481 learned_addr learned my addr 172.21.15.2:0/2225845481 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1be00640 1 --2- 172.21.15.2:0/2225845481 >> v2:172.21.15.137:3300/0 conn(0x7f5b1c06a4b0 0x7f5b1c06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1a000640 1 -- 172.21.15.2:0/2225845481 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719646735 0 0) 0x7f5b04001110 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1a000640 1 -- 172.21.15.2:0/2225845481 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5afc003520 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1be00640 1 -- 172.21.15.2:0/2225845481 >> v1:172.21.15.137:6789/0 conn(0x7f5b1c15d330 legacy=0x7f5b1c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1be00640 1 -- 172.21.15.2:0/2225845481 >> v1:172.21.15.2:6789/0 conn(0x7f5b1c0b2f60 legacy=0x7f5b1c06cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1a000640 1 -- 172.21.15.2:0/2225845481 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 764193603 0 0) 0x7f5afc003520 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.250+0000 7f5b1be00640 1 -- 172.21.15.2:0/2225845481 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c071d00 con 0x7f5b1c06a4b0 2024-01-15T17:58:26.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.252+0000 7f5b1be00640 1 --2- 172.21.15.2:0/2225845481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c06a4b0 0x7f5b1c06a900 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009d90 tx=0x7f5b1402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e62d492a358b6261 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:26.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.252+0000 7f5b1a000640 1 -- 172.21.15.2:0/2225845481 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04001800 con 0x7f5b1c06a4b0 2024-01-15T17:58:26.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.252+0000 7f5b1a000640 1 -- 172.21.15.2:0/2225845481 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1402c940 con 0x7f5b1c06a4b0 2024-01-15T17:58:26.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.252+0000 7f5b1a000640 1 -- 172.21.15.2:0/2225845481 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1402ccd0 con 0x7f5b1c06a4b0 2024-01-15T17:58:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.253+0000 7f5b21200640 1 -- 172.21.15.2:0/2225845481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c06a4b0 msgr2=0x7f5b1c06a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.253+0000 7f5b21200640 1 --2- 172.21.15.2:0/2225845481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c06a4b0 0x7f5b1c06a900 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5b14009d90 tx=0x7f5b1402be10 comp rx=0 tx=0).stop 2024-01-15T17:58:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.253+0000 7f5b21200640 1 -- 172.21.15.2:0/2225845481 shutdown_connections 2024-01-15T17:58:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.253+0000 7f5b21200640 1 --2- 172.21.15.2:0/2225845481 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c06a4b0 0x7f5b1c06a900 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.253+0000 7f5b21200640 1 -- 172.21.15.2:0/2225845481 >> 172.21.15.2:0/2225845481 conn(0x7f5b1c06b5c0 msgr2=0x7f5b1c06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.253+0000 7f5b21200640 1 -- 172.21.15.2:0/2225845481 shutdown_connections 2024-01-15T17:58:26.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.254+0000 7f5b21200640 1 -- 172.21.15.2:0/2225845481 wait complete. 2024-01-15T17:58:26.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.255+0000 7f5b21200640 1 Processor -- start 2024-01-15T17:58:26.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.255+0000 7f5b21200640 1 -- start start 2024-01-15T17:58:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.256+0000 7f5b21200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1c06a4b0 0x7f5b1c14d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.256+0000 7f5b21200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 0x7f5b1c157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b21200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c0c4d60 0x7f5b1c0c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b21200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c157c40 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b21200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c157f20 con 0x7f5b1c0c4d60 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b21200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1c158200 con 0x7f5b1c06a4b0 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 0x7f5b1c157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 0x7f5b1c157260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49292/0 (socket says 172.21.15.2:49292) 2024-01-15T17:58:26.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 -- 172.21.15.2:0/1054540357 learned_addr learned my addr 172.21.15.2:0/1054540357 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1be00640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c0c4d60 0x7f5b1c0c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1b400640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1c06a4b0 0x7f5b1c14d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 -- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1c06a4b0 msgr2=0x7f5b1c14d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1c06a4b0 0x7f5b1c14d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 -- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c0c4d60 msgr2=0x7f5b1c0c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c0c4d60 0x7f5b1c0c6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.257+0000 7f5b1aa00640 1 -- 172.21.15.2:0/1054540357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14009a70 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.258+0000 7f5b1aa00640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 0x7f5b1c157260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5b04001370 tx=0x7f5b0400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.258+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04005ad0 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.258+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b04005c30 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.258+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b04007070 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.258+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1c1584e0 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.258+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1c14dd40 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.259+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1c067970 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.260+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5b040071d0 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.262+0000 7f5b13400640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5af406b2c0 0x7f5af406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.262+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5b04089890 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.262+0000 7f5b1b400640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5af406b2c0 0x7f5af406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.263+0000 7f5b1b400640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5af406b2c0 0x7f5af406d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c000f60 tx=0x7f5b0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.266+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b04054670 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.463+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b1c158a00 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.464+0000 7f5b13400640 1 -- 172.21.15.2:0/1054540357 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b1c158a00 con 0x7f5b1c0b2f60 2024-01-15T17:58:26.468 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5af406b2c0 msgr2=0x7f5af406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5af406b2c0 0x7f5af406d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c000f60 tx=0x7f5b0c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 msgr2=0x7f5b1c157260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 0x7f5b1c157260 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5b04001370 tx=0x7f5b0400cc00 comp rx=0 tx=0).stop 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 shutdown_connections 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5af406b2c0 0x7f5af406d780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b1c0c4d60 0x7f5b1c0c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1c0b2f60 0x7f5b1c157260 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 --2- 172.21.15.2:0/1054540357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1c06a4b0 0x7f5b1c14d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 >> 172.21.15.2:0/1054540357 conn(0x7f5b1c06b5c0 msgr2=0x7f5b1c06cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 shutdown_connections 2024-01-15T17:58:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.467+0000 7f5b21200640 1 -- 172.21.15.2:0/1054540357 wait complete. 2024-01-15T17:58:26.490 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.709+0000 7f1688200640 1 Processor -- start 2024-01-15T17:58:26.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.710+0000 7f1688200640 1 -- start start 2024-01-15T17:58:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.711+0000 7f1688200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1680143850 0x7f168015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.711+0000 7f1688200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168006cfc0 0x7f1680071450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1688200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168015e970 con 0x7f1680160f60 2024-01-15T17:58:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1688200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16801619b0 con 0x7f1680143850 2024-01-15T17:58:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1688200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1680161c90 con 0x7f168006cfc0 2024-01-15T17:58:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1680143850 0x7f168015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1680143850 0x7f168015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49298/0 (socket says 172.21.15.2:49298) 2024-01-15T17:58:26.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 -- 172.21.15.2:0/1612268227 learned_addr learned my addr 172.21.15.2:0/1612268227 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1686200640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168006cfc0 0x7f1680071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 -- 172.21.15.2:0/1612268227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168006cfc0 msgr2=0x7f1680071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168006cfc0 0x7f1680071450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 -- 172.21.15.2:0/1612268227 >> v1:172.21.15.137:6789/0 conn(0x7f1680160f60 legacy=0x7f168006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1685800640 1 -- 172.21.15.2:0/1612268227 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168015dc30 con 0x7f1680143850 2024-01-15T17:58:26.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.712+0000 7f1686200640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168006cfc0 0x7f1680071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:26.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.713+0000 7f1685800640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1680143850 0x7f168015e430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16800acfb0 tx=0x7f167002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3efd9d55fd0500a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:26.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.714+0000 7f167fe00640 1 -- 172.21.15.2:0/1612268227 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16700317c0 con 0x7f1680143850 2024-01-15T17:58:26.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.714+0000 7f167fe00640 1 -- 172.21.15.2:0/1612268227 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1670031920 con 0x7f1680143850 2024-01-15T17:58:26.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.714+0000 7f167fe00640 1 -- 172.21.15.2:0/1612268227 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1670031cb0 con 0x7f1680143850 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 -- 172.21.15.2:0/1612268227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1680143850 msgr2=0x7f168015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1680143850 0x7f168015e430 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f16800acfb0 tx=0x7f167002b860 comp rx=0 tx=0).stop 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 -- 172.21.15.2:0/1612268227 shutdown_connections 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168006cfc0 0x7f1680071450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 --2- 172.21.15.2:0/1612268227 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1680143850 0x7f168015e430 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 -- 172.21.15.2:0/1612268227 >> 172.21.15.2:0/1612268227 conn(0x7f16800b46d0 msgr2=0x7f16800b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 -- 172.21.15.2:0/1612268227 shutdown_connections 2024-01-15T17:58:26.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.715+0000 7f1688200640 1 -- 172.21.15.2:0/1612268227 wait complete. 2024-01-15T17:58:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.717+0000 7f1688200640 1 Processor -- start 2024-01-15T17:58:26.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.717+0000 7f1688200640 1 -- start start 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1688200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 0x7f16800ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1688200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1680143850 0x7f16800c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1688200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1680160f60 0x7f16800ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1688200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800c71e0 con 0x7f168006cfc0 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1688200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1680130f10 con 0x7f1680160f60 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1688200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16800ce910 con 0x7f1680143850 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1685800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 0x7f16800ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1685800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 0x7f16800ce3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49310/0 (socket says 172.21.15.2:49310) 2024-01-15T17:58:26.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1685800640 1 -- 172.21.15.2:0/1799403854 learned_addr learned my addr 172.21.15.2:0/1799403854 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:26.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.718+0000 7f1684e00640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1680143850 0x7f16800c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1685800640 1 -- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1680143850 msgr2=0x7f16800c9c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1685800640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1680143850 0x7f16800c9c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1685800640 1 -- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1680160f60 msgr2=0x7f16800ca140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1685800640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1680160f60 0x7f16800ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1685800640 1 -- 172.21.15.2:0/1799403854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16700094d0 con 0x7f168006cfc0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1685800640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 0x7f16800ce3d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f167002b8a0 tx=0x7f16700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16700329a0 con 0x7f168006cfc0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1670032b20 con 0x7f168006cfc0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16700453f0 con 0x7f168006cfc0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.719+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16800c6690 con 0x7f168006cfc0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.720+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1680130ba0 con 0x7f168006cfc0 2024-01-15T17:58:26.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.721+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680130720 con 0x7f168006cfc0 2024-01-15T17:58:26.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.722+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1670045550 con 0x7f168006cfc0 2024-01-15T17:58:26.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.723+0000 7f167d600640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165806b2c0 0x7f165806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:26.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.723+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f16700ac2e0 con 0x7f168006cfc0 2024-01-15T17:58:26.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.724+0000 7f1684e00640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165806b2c0 0x7f165806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:26.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.725+0000 7f1684e00640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165806b2c0 0x7f165806d780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1680142a90 tx=0x7f1668008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:26.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.728+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1680130720 con 0x7f168006cfc0 2024-01-15T17:58:26.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.933+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1680130720 con 0x7f168006cfc0 2024-01-15T17:58:26.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.934+0000 7f167d600640 1 -- 172.21.15.2:0/1799403854 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1680130720 con 0x7f168006cfc0 2024-01-15T17:58:26.938 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165806b2c0 msgr2=0x7f165806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165806b2c0 0x7f165806d780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1680142a90 tx=0x7f1668008040 comp rx=0 tx=0).stop 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 msgr2=0x7f16800ce3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 0x7f16800ce3d0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f167002b8a0 tx=0x7f16700046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 shutdown_connections 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165806b2c0 0x7f165806d780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1680160f60 0x7f16800ca140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1680143850 0x7f16800c9c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 --2- 172.21.15.2:0/1799403854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168006cfc0 0x7f16800ce3d0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:26.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 >> 172.21.15.2:0/1799403854 conn(0x7f16800b46d0 msgr2=0x7f1680071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:26.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 shutdown_connections 2024-01-15T17:58:26.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:26.937+0000 7f1688200640 1 -- 172.21.15.2:0/1799403854 wait complete. 2024-01-15T17:58:26.959 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T17:58:26.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:26.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T17:58:27.063 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:27.063 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:58:27.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:27.064 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T17:58:27.126 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:58:27.126 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:27.184 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:58:27.184 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:27.233 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:27.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.449+0000 7f59de000640 1 Processor -- start 2024-01-15T17:58:27.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.450+0000 7f59de000640 1 -- start start 2024-01-15T17:58:27.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.451+0000 7f59de000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f59d815d340 0x7f59d80cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.451+0000 7f59de000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f59d80d1d90 0x7f59d80d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.451+0000 7f59de000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80d3a60 con 0x7f59d80d3090 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.451+0000 7f59de000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80b3de0 con 0x7f59d80d1d90 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.451+0000 7f59de000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d80b40c0 con 0x7f59d815d340 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f59d80d1d90 0x7f59d80d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f59d80d1d90 0x7f59d80d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49318/0 (socket says 172.21.15.2:49318) 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 -- 172.21.15.2:0/3053917373 learned_addr learned my addr 172.21.15.2:0/3053917373 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59cfe00640 1 --1- 172.21.15.2:0/3053917373 >> v1:172.21.15.2:6789/0 conn(0x7f59d80d3090 0x7f59d80d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35222/0 (socket says 172.21.15.2:35222) 2024-01-15T17:58:27.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dcc00640 1 --2- 172.21.15.2:0/3053917373 >> v2:172.21.15.137:3300/0 conn(0x7f59d815d340 0x7f59d80cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 -- 172.21.15.2:0/3053917373 >> v2:172.21.15.137:3300/0 conn(0x7f59d815d340 msgr2=0x7f59d80cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:27.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 --2- 172.21.15.2:0/3053917373 >> v2:172.21.15.137:3300/0 conn(0x7f59d815d340 0x7f59d80cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 -- 172.21.15.2:0/3053917373 >> v1:172.21.15.2:6789/0 conn(0x7f59d80d3090 legacy=0x7f59d80d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dd600640 1 -- 172.21.15.2:0/3053917373 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80d0e80 con 0x7f59d80d1d90 2024-01-15T17:58:27.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.452+0000 7f59dcc00640 1 --2- 172.21.15.2:0/3053917373 >> v2:172.21.15.137:3300/0 conn(0x7f59d815d340 0x7f59d80cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:27.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.454+0000 7f59dd600640 1 --2- 172.21.15.2:0/3053917373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d80d1d90 0x7f59d80d3520 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f59d808cf70 tx=0x7f59d002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47599c73faf7e551 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:27.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.454+0000 7f59d7e00640 1 -- 172.21.15.2:0/3053917373 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d002cb10 con 0x7f59d80d1d90 2024-01-15T17:58:27.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.454+0000 7f59d7e00640 1 -- 172.21.15.2:0/3053917373 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d002cc70 con 0x7f59d80d1d90 2024-01-15T17:58:27.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.454+0000 7f59d7e00640 1 -- 172.21.15.2:0/3053917373 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d00343f0 con 0x7f59d80d1d90 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 -- 172.21.15.2:0/3053917373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d80d1d90 msgr2=0x7f59d80d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 --2- 172.21.15.2:0/3053917373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d80d1d90 0x7f59d80d3520 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f59d808cf70 tx=0x7f59d002bf70 comp rx=0 tx=0).stop 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 -- 172.21.15.2:0/3053917373 shutdown_connections 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 --2- 172.21.15.2:0/3053917373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d80d1d90 0x7f59d80d3520 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 --2- 172.21.15.2:0/3053917373 >> v2:172.21.15.137:3300/0 conn(0x7f59d815d340 0x7f59d80cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 -- 172.21.15.2:0/3053917373 >> 172.21.15.2:0/3053917373 conn(0x7f59d808caf0 msgr2=0x7f59d809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 -- 172.21.15.2:0/3053917373 shutdown_connections 2024-01-15T17:58:27.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.455+0000 7f59de000640 1 -- 172.21.15.2:0/3053917373 wait complete. 2024-01-15T17:58:27.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.457+0000 7f59de000640 1 Processor -- start 2024-01-15T17:58:27.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.457+0000 7f59de000640 1 -- start start 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.458+0000 7f59de000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d80d1d90 0x7f59d8160420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.458+0000 7f59de000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80d3090 0x7f59d80b7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.458+0000 7f59de000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 0x7f59d81f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59de000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80b81a0 con 0x7f59d815d340 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59de000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80b8420 con 0x7f59d80d1d90 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59de000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80b8700 con 0x7f59d80d3090 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 0x7f59d81f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 0x7f59d81f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49320/0 (socket says 172.21.15.2:49320) 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 -- 172.21.15.2:0/630151623 learned_addr learned my addr 172.21.15.2:0/630151623 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:27.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dcc00640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d80d1d90 0x7f59d8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59cfe00640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80d3090 0x7f59d80b7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 -- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80d3090 msgr2=0x7f59d80b7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80d3090 0x7f59d80b7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 -- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d80d1d90 msgr2=0x7f59d8160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d80d1d90 0x7f59d8160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.459+0000 7f59dd600640 1 -- 172.21.15.2:0/630151623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d0009b30 con 0x7f59d815d340 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.460+0000 7f59dd600640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 0x7f59d81f2580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f59d0036040 tx=0x7f59d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.460+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0009510 con 0x7f59d815d340 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.460+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d003e450 con 0x7f59d815d340 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.460+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d815dac0 con 0x7f59d815d340 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.460+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d003e800 con 0x7f59d815d340 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.460+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d815e080 con 0x7f59d815d340 2024-01-15T17:58:27.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.461+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d815e270 con 0x7f59d815d340 2024-01-15T17:58:27.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.463+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f59d003e960 con 0x7f59d815d340 2024-01-15T17:58:27.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.464+0000 7f59d5600640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59b006b2f0 0x7f59b006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.464+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f59d00ac920 con 0x7f59d815d340 2024-01-15T17:58:27.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.464+0000 7f59dcc00640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59b006b2f0 0x7f59b006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.465+0000 7f59dcc00640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59b006b2f0 0x7f59b006d7b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59c8006fd0 tx=0x7f59c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:27.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.469+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59d00777a0 con 0x7f59d815d340 2024-01-15T17:58:27.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.667+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59d8074770 con 0x7f59d815d340 2024-01-15T17:58:27.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.670+0000 7f59d5600640 1 -- 172.21.15.2:0/630151623 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f59d007adc0 con 0x7f59d815d340 2024-01-15T17:58:27.674 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:27.674 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.672+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59b006b2f0 msgr2=0x7f59b006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.672+0000 7f59de000640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59b006b2f0 0x7f59b006d7b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59c8006fd0 tx=0x7f59c8008040 comp rx=0 tx=0).stop 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.672+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 msgr2=0x7f59d81f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.672+0000 7f59de000640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 0x7f59d81f2580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f59d0036040 tx=0x7f59d0004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 shutdown_connections 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59b006b2f0 0x7f59b006d7b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f59d815d340 0x7f59d81f2580 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f59d80d3090 0x7f59d80b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 --2- 172.21.15.2:0/630151623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f59d80d1d90 0x7f59d8160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 >> 172.21.15.2:0/630151623 conn(0x7f59d808caf0 msgr2=0x7f59d815c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 shutdown_connections 2024-01-15T17:58:27.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.673+0000 7f59de000640 1 -- 172.21.15.2:0/630151623 wait complete. 2024-01-15T17:58:27.697 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:27.698 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:27.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.915+0000 7f664d600640 1 Processor -- start 2024-01-15T17:58:27.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.915+0000 7f664d600640 1 -- start start 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f664d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66480b2210 0x7f66480d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f664d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480d1c50 0x7f66480b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f664d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66480d1250 con 0x7f66480d1870 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f664d600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66480ce6d0 con 0x7f66480b2210 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f664d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66480ce8e0 con 0x7f66480d1c50 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f663f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66480b2210 0x7f66480d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f663f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66480b2210 0x7f66480d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49326/0 (socket says 172.21.15.2:49326) 2024-01-15T17:58:27.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.917+0000 7f663f200640 1 -- 172.21.15.2:0/3263602933 learned_addr learned my addr 172.21.15.2:0/3263602933 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.918+0000 7f664cc00640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480d1c50 0x7f66480b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.918+0000 7f663f200640 1 -- 172.21.15.2:0/3263602933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480d1c50 msgr2=0x7f66480b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.918+0000 7f663f200640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480d1c50 0x7f66480b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.918+0000 7f663f200640 1 -- 172.21.15.2:0/3263602933 >> v1:172.21.15.137:6789/0 conn(0x7f66480d1870 legacy=0x7f664815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.918+0000 7f663f200640 1 -- 172.21.15.2:0/3263602933 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66480cebc0 con 0x7f66480b2210 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.918+0000 7f664cc00640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480d1c50 0x7f66480b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:27.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.919+0000 7f663f200640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480b2210 0x7f66480d3bd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6630009870 tx=0x7f663002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692aa07cb673cd9b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:27.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.919+0000 7f6647400640 1 -- 172.21.15.2:0/3263602933 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630030840 con 0x7f66480b2210 2024-01-15T17:58:27.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.919+0000 7f6647400640 1 -- 172.21.15.2:0/3263602933 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66300309a0 con 0x7f66480b2210 2024-01-15T17:58:27.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.919+0000 7f6647400640 1 -- 172.21.15.2:0/3263602933 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630030d30 con 0x7f66480b2210 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 -- 172.21.15.2:0/3263602933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480b2210 msgr2=0x7f66480d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480b2210 0x7f66480d3bd0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6630009870 tx=0x7f663002b980 comp rx=0 tx=0).stop 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 -- 172.21.15.2:0/3263602933 shutdown_connections 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480d1c50 0x7f66480b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 --2- 172.21.15.2:0/3263602933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480b2210 0x7f66480d3bd0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 -- 172.21.15.2:0/3263602933 >> 172.21.15.2:0/3263602933 conn(0x7f66480d30b0 msgr2=0x7f66480d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 -- 172.21.15.2:0/3263602933 shutdown_connections 2024-01-15T17:58:27.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.920+0000 7f664d600640 1 -- 172.21.15.2:0/3263602933 wait complete. 2024-01-15T17:58:27.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.922+0000 7f664d600640 1 Processor -- start 2024-01-15T17:58:27.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.922+0000 7f664d600640 1 -- start start 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f664d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480b2210 0x7f664814e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f664d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 0x7f664814efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f664d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66480d1c50 0x7f664815efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f664d600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664814eb40 con 0x7f66480d1870 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f664d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664815af00 con 0x7f66480d1c50 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f664d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664815fb60 con 0x7f66480b2210 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.923+0000 7f663f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 0x7f664814efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 0x7f664814efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49334/0 (socket says 172.21.15.2:49334) 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 -- 172.21.15.2:0/1359688371 learned_addr learned my addr 172.21.15.2:0/1359688371 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f6647e00640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480b2210 0x7f664814e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f664cc00640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66480d1c50 0x7f664815efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 -- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480b2210 msgr2=0x7f664814e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480b2210 0x7f664814e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 -- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66480d1c50 msgr2=0x7f664815efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:27.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66480d1c50 0x7f664815efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:27.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 -- 172.21.15.2:0/1359688371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66300094d0 con 0x7f66480d1870 2024-01-15T17:58:27.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.924+0000 7f663f200640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 0x7f664814efe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6630009990 tx=0x7f6630004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:27.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.925+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630039e20 con 0x7f66480d1870 2024-01-15T17:58:27.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.925+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6630044480 con 0x7f66480d1870 2024-01-15T17:58:27.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.925+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6630044830 con 0x7f66480d1870 2024-01-15T17:58:27.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.925+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66480b6b80 con 0x7f66480d1870 2024-01-15T17:58:27.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.925+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66480b7140 con 0x7f66480d1870 2024-01-15T17:58:27.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.926+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6648074770 con 0x7f66480d1870 2024-01-15T17:58:27.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.927+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f663002f390 con 0x7f66480d1870 2024-01-15T17:58:27.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.928+0000 7f6644c00640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f662006b2c0 0x7f662006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:27.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.928+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f663003e070 con 0x7f66480d1870 2024-01-15T17:58:27.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.929+0000 7f6647e00640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f662006b2c0 0x7f662006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:27.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.930+0000 7f6647e00640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f662006b2c0 0x7f662006d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6638001d40 tx=0x7f6638009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:27.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:27.932+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6630077260 con 0x7f66480d1870 2024-01-15T17:58:28.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.124+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66480b7300 con 0x7f66480d1870 2024-01-15T17:58:28.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.125+0000 7f6644c00640 1 -- 172.21.15.2:0/1359688371 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f663007a880 con 0x7f66480d1870 2024-01-15T17:58:28.129 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f662006b2c0 msgr2=0x7f662006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f662006b2c0 0x7f662006d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6638001d40 tx=0x7f6638009040 comp rx=0 tx=0).stop 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 msgr2=0x7f664814efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 0x7f664814efe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f6630009990 tx=0x7f6630004730 comp rx=0 tx=0).stop 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 shutdown_connections 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f662006b2c0 0x7f662006d780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66480d1c50 0x7f664815efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66480d1870 0x7f664814efe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 --2- 172.21.15.2:0/1359688371 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66480b2210 0x7f664814e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 >> 172.21.15.2:0/1359688371 conn(0x7f66480d30b0 msgr2=0x7f66480d35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 shutdown_connections 2024-01-15T17:58:28.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.128+0000 7f664d600640 1 -- 172.21.15.2:0/1359688371 wait complete. 2024-01-15T17:58:28.152 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:28.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.372+0000 7f7ccdc00640 1 Processor -- start 2024-01-15T17:58:28.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.373+0000 7f7ccdc00640 1 -- start start 2024-01-15T17:58:28.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.374+0000 7f7ccdc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7cc80d1870 0x7f7cc80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.374+0000 7f7ccdc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7cc80b2210 0x7f7cc815d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.374+0000 7f7ccdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1c50 0x7f7cc815d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.374+0000 7f7ccdc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc80ce6d0 con 0x7f7cc80b2210 2024-01-15T17:58:28.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.374+0000 7f7ccdc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc815db60 con 0x7f7cc80d1870 2024-01-15T17:58:28.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.374+0000 7f7ccdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80d1250 con 0x7f7cc80d1c50 2024-01-15T17:58:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7cc80b2210 0x7f7cc815d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7cc80b2210 0x7f7cc815d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49348/0 (socket says 172.21.15.2:49348) 2024-01-15T17:58:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 -- 172.21.15.2:0/1308059821 learned_addr learned my addr 172.21.15.2:0/1308059821 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:28.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7e00640 1 --2- 172.21.15.2:0/1308059821 >> v2:172.21.15.137:3300/0 conn(0x7f7cc80d1870 0x7f7cc80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7ccd200640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1c50 0x7f7cc815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 -- 172.21.15.2:0/1308059821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1c50 msgr2=0x7f7cc815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1c50 0x7f7cc815d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 -- 172.21.15.2:0/1308059821 >> v2:172.21.15.137:3300/0 conn(0x7f7cc80d1870 msgr2=0x7f7cc80d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 --2- 172.21.15.2:0/1308059821 >> v2:172.21.15.137:3300/0 conn(0x7f7cc80d1870 0x7f7cc80d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7cc7400640 1 -- 172.21.15.2:0/1308059821 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc80ce8e0 con 0x7f7cc80b2210 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.375+0000 7f7ccd200640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1c50 0x7f7cc815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:28.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.376+0000 7f7cc7400640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80b2210 0x7f7cc815d0e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8067890 tx=0x7f7cb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389f8b76027c6f7d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:28.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.377+0000 7f7cc6a00640 1 -- 172.21.15.2:0/1308059821 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0031870 con 0x7f7cc80b2210 2024-01-15T17:58:28.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.377+0000 7f7cc6a00640 1 -- 172.21.15.2:0/1308059821 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb00319d0 con 0x7f7cc80b2210 2024-01-15T17:58:28.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.377+0000 7f7cc6a00640 1 -- 172.21.15.2:0/1308059821 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb0031d60 con 0x7f7cc80b2210 2024-01-15T17:58:28.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.377+0000 7f7ccdc00640 1 -- 172.21.15.2:0/1308059821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80b2210 msgr2=0x7f7cc815d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.377+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80b2210 0x7f7cc815d0e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8067890 tx=0x7f7cb002b910 comp rx=0 tx=0).stop 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 -- 172.21.15.2:0/1308059821 shutdown_connections 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1c50 0x7f7cc815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/1308059821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80b2210 0x7f7cc815d0e0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/1308059821 >> v2:172.21.15.137:3300/0 conn(0x7f7cc80d1870 0x7f7cc80d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 -- 172.21.15.2:0/1308059821 >> 172.21.15.2:0/1308059821 conn(0x7f7cc80d30b0 msgr2=0x7f7cc80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 -- 172.21.15.2:0/1308059821 shutdown_connections 2024-01-15T17:58:28.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.378+0000 7f7ccdc00640 1 -- 172.21.15.2:0/1308059821 wait complete. 2024-01-15T17:58:28.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.380+0000 7f7ccdc00640 1 Processor -- start 2024-01-15T17:58:28.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.380+0000 7f7ccdc00640 1 -- start start 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccdc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cc80b2210 0x7f7cc81f0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccdc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1870 0x7f7cc80cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccdc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 0x7f7cc81f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccdc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc81f0d50 con 0x7f7cc80d1c50 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccdc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80cbcd0 con 0x7f7cc80b2210 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccdc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc80cac30 con 0x7f7cc80d1870 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 0x7f7cc81f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 0x7f7cc81f68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49354/0 (socket says 172.21.15.2:49354) 2024-01-15T17:58:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7ccd200640 1 -- 172.21.15.2:0/4168328628 learned_addr learned my addr 172.21.15.2:0/4168328628 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.381+0000 7f7cc7400640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1870 0x7f7cc80cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7cc7e00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cc80b2210 0x7f7cc81f0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccd200640 1 -- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1870 msgr2=0x7f7cc80cc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccd200640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1870 0x7f7cc80cc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccd200640 1 -- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cc80b2210 msgr2=0x7f7cc81f0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccd200640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cc80b2210 0x7f7cc81f0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccd200640 1 -- 172.21.15.2:0/4168328628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb00094d0 con 0x7f7cc80d1c50 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7cc7e00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cc80b2210 0x7f7cc81f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccd200640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 0x7f7cc81f68f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000cd80 tx=0x7f7cc000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0003b90 con 0x7f7cc80d1c50 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc0003d10 con 0x7f7cc80d1c50 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0004260 con 0x7f7cc80d1c50 2024-01-15T17:58:28.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.382+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc80caf10 con 0x7f7cc80d1c50 2024-01-15T17:58:28.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.383+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc80cb4d0 con 0x7f7cc80d1c50 2024-01-15T17:58:28.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.384+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc8074770 con 0x7f7cc80d1c50 2024-01-15T17:58:28.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.385+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7cc000b040 con 0x7f7cc80d1c50 2024-01-15T17:58:28.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.385+0000 7f7cbfe00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ca006b2c0 0x7f7ca006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.386+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7cc008a430 con 0x7f7cc80d1c50 2024-01-15T17:58:28.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.386+0000 7f7cc7e00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ca006b2c0 0x7f7ca006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.387+0000 7f7cc7e00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ca006b2c0 0x7f7ca006d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006fd0 tx=0x7f7cb8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.391+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7cc0055180 con 0x7f7cc80d1c50 2024-01-15T17:58:28.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.585+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7cc80b1f10 con 0x7f7cc80d1c50 2024-01-15T17:58:28.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.585+0000 7f7cbfe00640 1 -- 172.21.15.2:0/4168328628 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7cc0003e80 con 0x7f7cc80d1c50 2024-01-15T17:58:28.589 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ca006b2c0 msgr2=0x7f7ca006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ca006b2c0 0x7f7ca006d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8006fd0 tx=0x7f7cb8008040 comp rx=0 tx=0).stop 2024-01-15T17:58:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 msgr2=0x7f7cc81f68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 0x7f7cc81f68f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000cd80 tx=0x7f7cc000a650 comp rx=0 tx=0).stop 2024-01-15T17:58:28.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 shutdown_connections 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ca006b2c0 0x7f7ca006d780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cc80d1c50 0x7f7cc81f68f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cc80d1870 0x7f7cc80cc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 --2- 172.21.15.2:0/4168328628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cc80b2210 0x7f7cc81f0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 >> 172.21.15.2:0/4168328628 conn(0x7f7cc80d30b0 msgr2=0x7f7cc806c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 shutdown_connections 2024-01-15T17:58:28.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.588+0000 7f7ccdc00640 1 -- 172.21.15.2:0/4168328628 wait complete. 2024-01-15T17:58:28.612 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:28.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.837+0000 7fdf83200640 1 Processor -- start 2024-01-15T17:58:28.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.837+0000 7fdf83200640 1 -- start start 2024-01-15T17:58:28.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf83200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf840b2210 0x7fdf840d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf83200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf840d1c50 0x7fdf840b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf83200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840d1250 con 0x7fdf840d1870 2024-01-15T17:58:28.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf83200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840ce6d0 con 0x7fdf840b2210 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf83200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840ce8e0 con 0x7fdf840d1c50 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf81400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf840b2210 0x7fdf840d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf81400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf840b2210 0x7fdf840d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49368/0 (socket says 172.21.15.2:49368) 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf81400640 1 -- 172.21.15.2:0/223504686 learned_addr learned my addr 172.21.15.2:0/223504686 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.839+0000 7fdf82800640 1 --2- 172.21.15.2:0/223504686 >> v2:172.21.15.137:3300/0 conn(0x7fdf840d1c50 0x7fdf840b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.840+0000 7fdf81400640 1 -- 172.21.15.2:0/223504686 >> v2:172.21.15.137:3300/0 conn(0x7fdf840d1c50 msgr2=0x7fdf840b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.840+0000 7fdf81400640 1 --2- 172.21.15.2:0/223504686 >> v2:172.21.15.137:3300/0 conn(0x7fdf840d1c50 0x7fdf840b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.840+0000 7fdf81400640 1 -- 172.21.15.2:0/223504686 >> v1:172.21.15.137:6789/0 conn(0x7fdf840d1870 legacy=0x7fdf8415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:28.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.840+0000 7fdf81400640 1 -- 172.21.15.2:0/223504686 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840cebc0 con 0x7fdf840b2210 2024-01-15T17:58:28.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.841+0000 7fdf81400640 1 --2- 172.21.15.2:0/223504686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf840d3bd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdf8408d7e0 tx=0x7fdf6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666f651a9cf17a7a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:28.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.841+0000 7fdf80a00640 1 -- 172.21.15.2:0/223504686 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c031870 con 0x7fdf840b2210 2024-01-15T17:58:28.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.841+0000 7fdf80a00640 1 -- 172.21.15.2:0/223504686 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c0319d0 con 0x7fdf840b2210 2024-01-15T17:58:28.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.841+0000 7fdf80a00640 1 -- 172.21.15.2:0/223504686 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c031d60 con 0x7fdf840b2210 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 -- 172.21.15.2:0/223504686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 msgr2=0x7fdf840d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 --2- 172.21.15.2:0/223504686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf840d3bd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdf8408d7e0 tx=0x7fdf6c02b910 comp rx=0 tx=0).stop 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 -- 172.21.15.2:0/223504686 shutdown_connections 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 --2- 172.21.15.2:0/223504686 >> v2:172.21.15.137:3300/0 conn(0x7fdf840d1c50 0x7fdf840b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 --2- 172.21.15.2:0/223504686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf840d3bd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 -- 172.21.15.2:0/223504686 >> 172.21.15.2:0/223504686 conn(0x7fdf840d30b0 msgr2=0x7fdf840d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 -- 172.21.15.2:0/223504686 shutdown_connections 2024-01-15T17:58:28.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.842+0000 7fdf83200640 1 -- 172.21.15.2:0/223504686 wait complete. 2024-01-15T17:58:28.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.844+0000 7fdf83200640 1 Processor -- start 2024-01-15T17:58:28.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.844+0000 7fdf83200640 1 -- start start 2024-01-15T17:58:28.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf83200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf8415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf83200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf840d1870 0x7fdf840c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf83200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf840d1c50 0x7fdf840c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf83200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840c7f10 con 0x7fdf840b2210 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf83200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8415ac90 con 0x7fdf840d1870 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf83200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840c75b0 con 0x7fdf840d1c50 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf8415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf8415a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49380/0 (socket says 172.21.15.2:49380) 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf81e00640 1 -- 172.21.15.2:0/2025184248 learned_addr learned my addr 172.21.15.2:0/2025184248 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf82800640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf840d1c50 0x7fdf840c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.845+0000 7fdf81e00640 1 -- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf840d1c50 msgr2=0x7fdf840c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf81e00640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf840d1c50 0x7fdf840c8e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf81e00640 1 -- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf840d1870 msgr2=0x7fdf840c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf81e00640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf840d1870 0x7fdf840c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf81e00640 1 -- 172.21.15.2:0/2025184248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0094d0 con 0x7fdf840b2210 2024-01-15T17:58:28.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf81e00640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf8415a750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c410 tx=0x7fdf7400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:28.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf740035f0 con 0x7fdf840b2210 2024-01-15T17:58:28.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74003750 con 0x7fdf840b2210 2024-01-15T17:58:28.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74003b40 con 0x7fdf840b2210 2024-01-15T17:58:28.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf84160ce0 con 0x7fdf840b2210 2024-01-15T17:58:28.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.846+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf840c7b30 con 0x7fdf840b2210 2024-01-15T17:58:28.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.847+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf840c7cc0 con 0x7fdf840b2210 2024-01-15T17:58:28.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.848+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdf74003ca0 con 0x7fdf840b2210 2024-01-15T17:58:28.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.849+0000 7fdf7a000640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5c06b2f0 0x7fdf5c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:28.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.849+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdf74088eb0 con 0x7fdf840b2210 2024-01-15T17:58:28.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.850+0000 7fdf81400640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5c06b2f0 0x7fdf5c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:28.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.851+0000 7fdf81400640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5c06b2f0 0x7fdf5c06d7b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c036000 tx=0x7fdf6c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:28.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:28.854+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf74053c00 con 0x7fdf840b2210 2024-01-15T17:58:28.860 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:58:28.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:58:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.055+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdf84074770 con 0x7fdf840b2210 2024-01-15T17:58:29.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.056+0000 7fdf7a000640 1 -- 172.21.15.2:0/2025184248 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdf74017020 con 0x7fdf840b2210 2024-01-15T17:58:29.060 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.058+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5c06b2f0 msgr2=0x7fdf5c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.058+0000 7fdf83200640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5c06b2f0 0x7fdf5c06d7b0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c036000 tx=0x7fdf6c030000 comp rx=0 tx=0).stop 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.058+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 msgr2=0x7fdf8415a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf8415a750 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c410 tx=0x7fdf7400c8d0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 shutdown_connections 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5c06b2f0 0x7fdf5c06d7b0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf840d1c50 0x7fdf840c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf840d1870 0x7fdf840c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 --2- 172.21.15.2:0/2025184248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf840b2210 0x7fdf8415a750 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 >> 172.21.15.2:0/2025184248 conn(0x7fdf840d30b0 msgr2=0x7fdf8415ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 shutdown_connections 2024-01-15T17:58:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.059+0000 7fdf83200640 1 -- 172.21.15.2:0/2025184248 wait complete. 2024-01-15T17:58:29.081 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:29.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.295+0000 7f003a000640 1 Processor -- start 2024-01-15T17:58:29.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.295+0000 7f003a000640 1 -- start start 2024-01-15T17:58:29.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.296+0000 7f003a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f003415d200 0x7f00340cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.296+0000 7f003a000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340b30e0 con 0x7f00340d2de0 2024-01-15T17:58:29.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.296+0000 7f003a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340ce180 con 0x7f00340d0ed0 2024-01-15T17:58:29.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.296+0000 7f003a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340d1a90 con 0x7f003415d200 2024-01-15T17:58:29.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.297+0000 7f0033e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f00340d2de0 0x7f00340d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35228/0 (socket says 172.21.15.2:35228) 2024-01-15T17:58:29.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.297+0000 7f0033e00640 1 -- 172.21.15.2:0/891587914 learned_addr learned my addr 172.21.15.2:0/891587914 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:29.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.297+0000 7f0038c00640 1 --2- 172.21.15.2:0/891587914 >> v2:172.21.15.137:3300/0 conn(0x7f003415d200 0x7f00340cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.297+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668527057 0 0) 0x7f00340b30e0 con 0x7f00340d2de0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.297+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0014003520 con 0x7f00340d2de0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3742183565 0 0) 0x7f0014003520 con 0x7f00340d2de0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 >> v2:172.21.15.137:3300/0 conn(0x7f003415d200 msgr2=0x7f00340cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 --2- 172.21.15.2:0/891587914 >> v2:172.21.15.137:3300/0 conn(0x7f003415d200 0x7f00340cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 >> v1:172.21.15.137:6789/0 conn(0x7f00340d0ed0 legacy=0x7f00340d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340d1ca0 con 0x7f00340d2de0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f001c001670 con 0x7f00340d2de0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 >> v1:172.21.15.2:6789/0 conn(0x7f00340d2de0 legacy=0x7f00340d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00140039c0 0x7f0014003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140047c0 0x7f0014004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00140051d0 0x7f001400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340d1a90 con 0x7f00140039c0 2024-01-15T17:58:29.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001400c120 con 0x7f00140051d0 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001400c2e0 con 0x7f00140047c0 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0038c00640 1 --2- 172.21.15.2:0/891587914 >> v2:172.21.15.137:3300/0 conn(0x7f003415d200 0x7f00340cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0038c00640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00140039c0 0x7f0014003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0033e00640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140047c0 0x7f0014004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.298+0000 7f0039600640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00140051d0 0x7f001400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.299+0000 7f0038c00640 1 -- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140047c0 msgr2=0x7f0014004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.299+0000 7f0038c00640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140047c0 0x7f0014004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.299+0000 7f0038c00640 1 -- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00140051d0 msgr2=0x7f001400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.299+0000 7f0038c00640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00140051d0 0x7f001400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.299+0000 7f0038c00640 1 -- 172.21.15.2:0/891587914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014003520 con 0x7f00140039c0 2024-01-15T17:58:29.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.299+0000 7f0039600640 1 -- 172.21.15.2:0/891587914 reap_dead start 2024-01-15T17:58:29.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.300+0000 7f0038c00640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00140039c0 0x7f0014003de0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f00240098a0 tx=0x7f00140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1de5a80b298ae48 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.300+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024030630 con 0x7f00140039c0 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.300+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0024030790 con 0x7f00140039c0 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.300+0000 7f0033400640 1 -- 172.21.15.2:0/891587914 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0024030b00 con 0x7f00140039c0 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 -- 172.21.15.2:0/891587914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00140039c0 msgr2=0x7f0014003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00140039c0 0x7f0014003de0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f00240098a0 tx=0x7f00140057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 -- 172.21.15.2:0/891587914 shutdown_connections 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 --2- 172.21.15.2:0/891587914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00140039c0 0x7f0014003de0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 -- 172.21.15.2:0/891587914 >> 172.21.15.2:0/891587914 conn(0x7f003408caf0 msgr2=0x7f003409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 -- 172.21.15.2:0/891587914 shutdown_connections 2024-01-15T17:58:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.301+0000 7f003a000640 1 -- 172.21.15.2:0/891587914 wait complete. 2024-01-15T17:58:29.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.303+0000 7f003a000640 1 Processor -- start 2024-01-15T17:58:29.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.303+0000 7f003a000640 1 -- start start 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f003a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140039c0 0x7f003414e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f003a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00340cb0f0 0x7f0034155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f003a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 0x7f0034156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f003a000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003415e840 con 0x7f00340cb570 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f003a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003414ebc0 con 0x7f00340cb0f0 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f003a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003415d970 con 0x7f00140039c0 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f0039600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 0x7f0034156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f0039600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 0x7f0034156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49398/0 (socket says 172.21.15.2:49398) 2024-01-15T17:58:29.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.304+0000 7f0039600640 1 -- 172.21.15.2:0/3771736364 learned_addr learned my addr 172.21.15.2:0/3771736364 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0038c00640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140039c0 0x7f003414e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0033e00640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00340cb0f0 0x7f0034155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0039600640 1 -- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140039c0 msgr2=0x7f003414e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0039600640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140039c0 0x7f003414e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0039600640 1 -- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00340cb0f0 msgr2=0x7f0034155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0039600640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00340cb0f0 0x7f0034155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0039600640 1 -- 172.21.15.2:0/3771736364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f003415dbc0 con 0x7f00340cb570 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0033e00640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00340cb0f0 0x7f0034155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:29.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.305+0000 7f0039600640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 0x7f0034156520 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f003408ba60 tx=0x7f002c00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.306+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c006e20 con 0x7f00340cb570 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.306+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c003970 con 0x7f00340cb570 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.306+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003415dea0 con 0x7f00340cb570 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.306+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f002c003d20 con 0x7f00340cb570 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.306+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003415e460 con 0x7f00340cb570 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.307+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003415e980 con 0x7f00340cb570 2024-01-15T17:58:29.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.308+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f002c005110 con 0x7f00340cb570 2024-01-15T17:58:29.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.309+0000 7f0030c00640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f000c06b2f0 0x7f000c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.309+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f002c0890c0 con 0x7f00340cb570 2024-01-15T17:58:29.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.310+0000 7f0038c00640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f000c06b2f0 0x7f000c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.311+0000 7f0038c00640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f000c06b2f0 0x7f000c06d7b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0024036000 tx=0x7f0024031000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:29.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.313+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f003415e980 con 0x7f00340cb570 2024-01-15T17:58:29.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.521+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f003415e980 con 0x7f00340cb570 2024-01-15T17:58:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.522+0000 7f0030c00640 1 -- 172.21.15.2:0/3771736364 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f003415e980 con 0x7f00340cb570 2024-01-15T17:58:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f000c06b2f0 msgr2=0x7f000c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f000c06b2f0 0x7f000c06d7b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f0024036000 tx=0x7f0024031000 comp rx=0 tx=0).stop 2024-01-15T17:58:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 msgr2=0x7f0034156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 0x7f0034156520 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f003408ba60 tx=0x7f002c00e960 comp rx=0 tx=0).stop 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 shutdown_connections 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f000c06b2f0 0x7f000c06d7b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00340cb570 0x7f0034156520 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00340cb0f0 0x7f0034155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 --2- 172.21.15.2:0/3771736364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00140039c0 0x7f003414e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 >> 172.21.15.2:0/3771736364 conn(0x7f003408caf0 msgr2=0x7f00340d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 shutdown_connections 2024-01-15T17:58:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.525+0000 7f003a000640 1 -- 172.21.15.2:0/3771736364 wait complete. 2024-01-15T17:58:29.549 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:58:29.549 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:29.605 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:29.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.821+0000 7fbd60400640 1 Processor -- start 2024-01-15T17:58:29.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.821+0000 7fbd60400640 1 -- start start 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.823+0000 7fbd60400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806a4b0 0x7fbd5806a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.823+0000 7fbd60400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbd58160f20 0x7fbd581613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.823+0000 7fbd60400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbd5806e8c0 0x7fbd5806ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.823+0000 7fbd60400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5806add0 con 0x7fbd5806e8c0 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.823+0000 7fbd60400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd58161cc0 con 0x7fbd58160f20 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.823+0000 7fbd60400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5815d920 con 0x7fbd5806a4b0 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.824+0000 7fbd5e400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbd5806e8c0 0x7fbd5806ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.824+0000 7fbd5e400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbd5806e8c0 0x7fbd5806ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49414/0 (socket says 172.21.15.2:49414) 2024-01-15T17:58:29.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.824+0000 7fbd5e400640 1 -- 172.21.15.2:0/2008161860 learned_addr learned my addr 172.21.15.2:0/2008161860 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:29.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.824+0000 7fbd5d000640 1 --2- 172.21.15.2:0/2008161860 >> v2:172.21.15.137:3300/0 conn(0x7fbd58160f20 0x7fbd581613a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.824+0000 7fbd5da00640 1 --2- 172.21.15.2:0/2008161860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806a4b0 0x7fbd5806a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.825+0000 7fbd5e400640 1 -- 172.21.15.2:0/2008161860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806a4b0 msgr2=0x7fbd5806a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:29.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.825+0000 7fbd5e400640 1 --2- 172.21.15.2:0/2008161860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806a4b0 0x7fbd5806a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.825+0000 7fbd5e400640 1 -- 172.21.15.2:0/2008161860 >> v2:172.21.15.137:3300/0 conn(0x7fbd58160f20 msgr2=0x7fbd581613a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:29.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.825+0000 7fbd5e400640 1 --2- 172.21.15.2:0/2008161860 >> v2:172.21.15.137:3300/0 conn(0x7fbd58160f20 0x7fbd581613a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.825+0000 7fbd5e400640 1 -- 172.21.15.2:0/2008161860 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5815dc00 con 0x7fbd5806e8c0 2024-01-15T17:58:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.826+0000 7fbd5e400640 1 --2- 172.21.15.2:0/2008161860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd5806e8c0 0x7fbd5806ed50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806ffd0 tx=0x7fbd5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a876ed190db70e38 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.827+0000 7fbd57e00640 1 -- 172.21.15.2:0/2008161860 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50031870 con 0x7fbd5806e8c0 2024-01-15T17:58:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.827+0000 7fbd57e00640 1 -- 172.21.15.2:0/2008161860 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd500319d0 con 0x7fbd5806e8c0 2024-01-15T17:58:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.827+0000 7fbd57e00640 1 -- 172.21.15.2:0/2008161860 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50031d60 con 0x7fbd5806e8c0 2024-01-15T17:58:29.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.827+0000 7fbd60400640 1 -- 172.21.15.2:0/2008161860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd5806e8c0 msgr2=0x7fbd5806ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.827+0000 7fbd60400640 1 --2- 172.21.15.2:0/2008161860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd5806e8c0 0x7fbd5806ed50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fbd5806ffd0 tx=0x7fbd5002b910 comp rx=0 tx=0).stop 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 -- 172.21.15.2:0/2008161860 shutdown_connections 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 --2- 172.21.15.2:0/2008161860 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd5806e8c0 0x7fbd5806ed50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 --2- 172.21.15.2:0/2008161860 >> v2:172.21.15.137:3300/0 conn(0x7fbd58160f20 0x7fbd581613a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 --2- 172.21.15.2:0/2008161860 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806a4b0 0x7fbd5806a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 -- 172.21.15.2:0/2008161860 >> 172.21.15.2:0/2008161860 conn(0x7fbd5806d410 msgr2=0x7fbd5806d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 -- 172.21.15.2:0/2008161860 shutdown_connections 2024-01-15T17:58:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.828+0000 7fbd60400640 1 -- 172.21.15.2:0/2008161860 wait complete. 2024-01-15T17:58:29.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.830+0000 7fbd60400640 1 Processor -- start 2024-01-15T17:58:29.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.830+0000 7fbd60400640 1 -- start start 2024-01-15T17:58:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd60400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5806a4b0 0x7fbd5815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd60400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806e8c0 0x7fbd5815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd60400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 0x7fbd5815a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd60400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580ae060 con 0x7fbd58160f20 2024-01-15T17:58:29.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd60400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580ad6c0 con 0x7fbd5806a4b0 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd60400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd580ad9a0 con 0x7fbd5806e8c0 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd5e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 0x7fbd5815a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd5e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 0x7fbd5815a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49430/0 (socket says 172.21.15.2:49430) 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd5e400640 1 -- 172.21.15.2:0/779856239 learned_addr learned my addr 172.21.15.2:0/779856239 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd5da00640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5806a4b0 0x7fbd5815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.831+0000 7fbd5d000640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806e8c0 0x7fbd5815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd5e400640 1 -- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806e8c0 msgr2=0x7fbd5815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd5e400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806e8c0 0x7fbd5815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd5e400640 1 -- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5806a4b0 msgr2=0x7fbd5815fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd5e400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5806a4b0 0x7fbd5815fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd5e400640 1 -- 172.21.15.2:0/779856239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd500094d0 con 0x7fbd58160f20 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd5e400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 0x7fbd5815a8f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009960 tx=0x7fbd500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.832+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5003ab60 con 0x7fbd58160f20 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.833+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5003acc0 con 0x7fbd58160f20 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.833+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd580adc80 con 0x7fbd58160f20 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.833+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50044610 con 0x7fbd58160f20 2024-01-15T17:58:29.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.833+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd58160060 con 0x7fbd58160f20 2024-01-15T17:58:29.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.834+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd580ae1a0 con 0x7fbd58160f20 2024-01-15T17:58:29.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.835+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbd50044770 con 0x7fbd58160f20 2024-01-15T17:58:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.836+0000 7fbd55600640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd3006b2c0 0x7fbd3006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.836+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbd5002d040 con 0x7fbd58160f20 2024-01-15T17:58:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.837+0000 7fbd5da00640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd3006b2c0 0x7fbd3006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:29.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.838+0000 7fbd5da00640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd3006b2c0 0x7fbd3006d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006fd0 tx=0x7fbd40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:29.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:29.841+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd580ae1a0 con 0x7fbd58160f20 2024-01-15T17:58:30.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.037+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd580ae1a0 con 0x7fbd58160f20 2024-01-15T17:58:30.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.040+0000 7fbd55600640 1 -- 172.21.15.2:0/779856239 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fbd580ae1a0 con 0x7fbd58160f20 2024-01-15T17:58:30.045 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:30.045 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd3006b2c0 msgr2=0x7fbd3006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd3006b2c0 0x7fbd3006d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbd40006fd0 tx=0x7fbd40008040 comp rx=0 tx=0).stop 2024-01-15T17:58:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 msgr2=0x7fbd5815a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 0x7fbd5815a8f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009960 tx=0x7fbd500046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 shutdown_connections 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd3006b2c0 0x7fbd3006d780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd58160f20 0x7fbd5815a8f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd5806e8c0 0x7fbd5815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 --2- 172.21.15.2:0/779856239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd5806a4b0 0x7fbd5815fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 >> 172.21.15.2:0/779856239 conn(0x7fbd5806d410 msgr2=0x7fbd580b13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 shutdown_connections 2024-01-15T17:58:30.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.043+0000 7fbd60400640 1 -- 172.21.15.2:0/779856239 wait complete. 2024-01-15T17:58:30.066 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:30.066 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:30.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.280+0000 7f3af9600640 1 Processor -- start 2024-01-15T17:58:30.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.281+0000 7f3af9600640 1 -- start start 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.282+0000 7f3af9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3af40b2210 0x7f3af40d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.282+0000 7f3af9600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40d1250 con 0x7f3af40d1870 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.282+0000 7f3af9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40ce8e0 con 0x7f3af40d1c50 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.282+0000 7f3af9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40ce6d0 con 0x7f3af40b2210 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.282+0000 7f3af3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3af40d1870 0x7f3af415d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35244/0 (socket says 172.21.15.2:35244) 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.282+0000 7f3af3e00640 1 -- 172.21.15.2:0/3667986651 learned_addr learned my addr 172.21.15.2:0/3667986651 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af3400640 1 --2- 172.21.15.2:0/3667986651 >> v2:172.21.15.137:3300/0 conn(0x7f3af40b2210 0x7f3af40d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888040495 0 0) 0x7f3af40d1250 con 0x7f3af40d1870 2024-01-15T17:58:30.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad4003520 con 0x7f3af40d1870 2024-01-15T17:58:30.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831590977 0 0) 0x7f3ad4003520 con 0x7f3af40d1870 2024-01-15T17:58:30.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 >> v2:172.21.15.137:3300/0 conn(0x7f3af40b2210 msgr2=0x7f3af40d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:30.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 --2- 172.21.15.2:0/3667986651 >> v2:172.21.15.137:3300/0 conn(0x7f3af40b2210 0x7f3af40d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 >> v1:172.21.15.137:6789/0 conn(0x7f3af40d1c50 legacy=0x7f3af40b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40cebc0 con 0x7f3af40d1870 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.283+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3ae4001670 con 0x7f3af40d1870 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 >> v1:172.21.15.2:6789/0 conn(0x7f3af40d1870 legacy=0x7f3af415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3ad4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ad40047c0 0x7f3ad4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ad4005180 0x7f3ad400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40ce6d0 con 0x7f3ad40039c0 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad400c080 con 0x7f3ad40047c0 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad400c240 con 0x7f3ad4005180 2024-01-15T17:58:30.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3e00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3ad4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3400640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ad40047c0 0x7f3ad4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af8c00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ad4005180 0x7f3ad400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3e00640 1 -- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ad4005180 msgr2=0x7f3ad400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3e00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ad4005180 0x7f3ad400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3e00640 1 -- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ad40047c0 msgr2=0x7f3ad4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3e00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ad40047c0 0x7f3ad4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af3e00640 1 -- 172.21.15.2:0/3667986651 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4003520 con 0x7f3ad40039c0 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.284+0000 7f3af8c00640 1 -- 172.21.15.2:0/3667986651 reap_dead start 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.285+0000 7f3af3e00640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3ad4003de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c0e0 tx=0x7f3ad40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728c6c93f828dc86 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4005ad0 con 0x7f3ad40039c0 2024-01-15T17:58:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4005c30 con 0x7f3ad40039c0 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af2a00640 1 -- 172.21.15.2:0/3667986651 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae402c7d0 con 0x7f3ad40039c0 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 -- 172.21.15.2:0/3667986651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 msgr2=0x7f3ad4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3ad4003de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c0e0 tx=0x7f3ad40057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 -- 172.21.15.2:0/3667986651 shutdown_connections 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 --2- 172.21.15.2:0/3667986651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3ad4003de0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 -- 172.21.15.2:0/3667986651 >> 172.21.15.2:0/3667986651 conn(0x7f3af40d30b0 msgr2=0x7f3af40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 -- 172.21.15.2:0/3667986651 shutdown_connections 2024-01-15T17:58:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.286+0000 7f3af9600640 1 -- 172.21.15.2:0/3667986651 wait complete. 2024-01-15T17:58:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.288+0000 7f3af9600640 1 Processor -- start 2024-01-15T17:58:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.288+0000 7f3af9600640 1 -- start start 2024-01-15T17:58:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.289+0000 7f3af9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3af41f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.289+0000 7f3af9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3af41510e0 0x7f3af40c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.289+0000 7f3af9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3af40b85b0 0x7f3af40cfa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.289+0000 7f3af9600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af41f6410 con 0x7f3ad40039c0 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.289+0000 7f3af9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af41f3370 con 0x7f3af41510e0 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.289+0000 7f3af9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af41f3650 con 0x7f3af40b85b0 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3af41f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3af41f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49440/0 (socket says 172.21.15.2:49440) 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 -- 172.21.15.2:0/1984870501 learned_addr learned my addr 172.21.15.2:0/1984870501 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3400640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3af41510e0 0x7f3af40c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af8c00640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3af40b85b0 0x7f3af40cfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 -- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3af40b85b0 msgr2=0x7f3af40cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3af40b85b0 0x7f3af40cfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 -- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3af41510e0 msgr2=0x7f3af40c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3af41510e0 0x7f3af40c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3e00640 1 -- 172.21.15.2:0/1984870501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af41f3930 con 0x7f3ad40039c0 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.290+0000 7f3af3400640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3af41510e0 0x7f3af40c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.291+0000 7f3af3e00640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3af41f2c70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c370 tx=0x7f3af406ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.291+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae403c070 con 0x7f3ad40039c0 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.291+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae402c9e0 con 0x7f3ad40039c0 2024-01-15T17:58:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.291+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af41f3c10 con 0x7f3ad40039c0 2024-01-15T17:58:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.291+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae402cd70 con 0x7f3ad40039c0 2024-01-15T17:58:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.291+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af40c6d90 con 0x7f3ad40039c0 2024-01-15T17:58:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.292+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af4074770 con 0x7f3ad40039c0 2024-01-15T17:58:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.293+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3ae4032070 con 0x7f3ad40039c0 2024-01-15T17:58:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.294+0000 7f3aebe00640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3acc06b2c0 0x7f3acc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.295+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3ae40aad60 con 0x7f3ad40039c0 2024-01-15T17:58:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.295+0000 7f3af3400640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3acc06b2c0 0x7f3acc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.296+0000 7f3af3400640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3acc06b2c0 0x7f3acc06d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3af41f6370 tx=0x7f3adc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:30.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.299+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ae4075b40 con 0x7f3ad40039c0 2024-01-15T17:58:30.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.488+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3af40ce4b0 con 0x7f3ad40039c0 2024-01-15T17:58:30.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.489+0000 7f3aebe00640 1 -- 172.21.15.2:0/1984870501 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ae4007070 con 0x7f3ad40039c0 2024-01-15T17:58:30.493 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:30.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.491+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3acc06b2c0 msgr2=0x7f3acc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.491+0000 7f3af9600640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3acc06b2c0 0x7f3acc06d780 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3af41f6370 tx=0x7f3adc008040 comp rx=0 tx=0).stop 2024-01-15T17:58:30.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.491+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 msgr2=0x7f3af41f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.491+0000 7f3af9600640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3af41f2c70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402c370 tx=0x7f3af406ae70 comp rx=0 tx=0).stop 2024-01-15T17:58:30.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 shutdown_connections 2024-01-15T17:58:30.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3acc06b2c0 0x7f3acc06d780 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3af40b85b0 0x7f3af40cfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3af41510e0 0x7f3af40c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 --2- 172.21.15.2:0/1984870501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ad40039c0 0x7f3af41f2c70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 >> 172.21.15.2:0/1984870501 conn(0x7f3af40d30b0 msgr2=0x7f3af40b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:30.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 shutdown_connections 2024-01-15T17:58:30.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.492+0000 7f3af9600640 1 -- 172.21.15.2:0/1984870501 wait complete. 2024-01-15T17:58:30.513 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:30.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.718+0000 7fe329400640 1 Processor -- start 2024-01-15T17:58:30.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.719+0000 7fe329400640 1 -- start start 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.720+0000 7fe329400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe324143d20 0x7fe32415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.720+0000 7fe329400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe3240720d0 0x7fe32415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.720+0000 7fe329400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3241447d0 0x7fe3240cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.720+0000 7fe329400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe324071430 con 0x7fe324143d20 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.720+0000 7fe329400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe324144100 con 0x7fe3240720d0 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.720+0000 7fe329400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe324071090 con 0x7fe3241447d0 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe324143d20 0x7fe32415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe324143d20 0x7fe32415eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49450/0 (socket says 172.21.15.2:49450) 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 -- 172.21.15.2:0/358310127 learned_addr learned my addr 172.21.15.2:0/358310127 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:30.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323400640 1 --2- 172.21.15.2:0/358310127 >> v2:172.21.15.137:3300/0 conn(0x7fe3240720d0 0x7fe32415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe328a00640 1 --2- 172.21.15.2:0/358310127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3241447d0 0x7fe3240cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 -- 172.21.15.2:0/358310127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3241447d0 msgr2=0x7fe3240cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 --2- 172.21.15.2:0/358310127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3241447d0 0x7fe3240cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 -- 172.21.15.2:0/358310127 >> v2:172.21.15.137:3300/0 conn(0x7fe3240720d0 msgr2=0x7fe32415f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 --2- 172.21.15.2:0/358310127 >> v2:172.21.15.137:3300/0 conn(0x7fe3240720d0 0x7fe32415f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.721+0000 7fe323e00640 1 -- 172.21.15.2:0/358310127 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324144560 con 0x7fe324143d20 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.722+0000 7fe323e00640 1 --2- 172.21.15.2:0/358310127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe324143d20 0x7fe32415eb50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe32404f900 tx=0x7fe31c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab401f46508c9915 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe322a00640 1 -- 172.21.15.2:0/358310127 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c02c9e0 con 0x7fe324143d20 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe322a00640 1 -- 172.21.15.2:0/358310127 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c02cb40 con 0x7fe324143d20 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe322a00640 1 -- 172.21.15.2:0/358310127 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0343f0 con 0x7fe324143d20 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 -- 172.21.15.2:0/358310127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe324143d20 msgr2=0x7fe32415eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 --2- 172.21.15.2:0/358310127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe324143d20 0x7fe32415eb50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe32404f900 tx=0x7fe31c02be40 comp rx=0 tx=0).stop 2024-01-15T17:58:30.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 -- 172.21.15.2:0/358310127 shutdown_connections 2024-01-15T17:58:30.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 --2- 172.21.15.2:0/358310127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3241447d0 0x7fe3240cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 --2- 172.21.15.2:0/358310127 >> v2:172.21.15.137:3300/0 conn(0x7fe3240720d0 0x7fe32415f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 --2- 172.21.15.2:0/358310127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe324143d20 0x7fe32415eb50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 -- 172.21.15.2:0/358310127 >> 172.21.15.2:0/358310127 conn(0x7fe3240b0840 msgr2=0x7fe3240b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:30.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 -- 172.21.15.2:0/358310127 shutdown_connections 2024-01-15T17:58:30.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.723+0000 7fe329400640 1 -- 172.21.15.2:0/358310127 wait complete. 2024-01-15T17:58:30.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.725+0000 7fe329400640 1 Processor -- start 2024-01-15T17:58:30.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.725+0000 7fe329400640 1 -- start start 2024-01-15T17:58:30.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe329400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3240720d0 0x7fe3241360b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe329400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe324143d20 0x7fe3240c7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe329400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 0x7fe3240ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe329400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe324136870 con 0x7fe3241447d0 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe329400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3241365f0 con 0x7fe324143d20 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe329400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe324136ae0 con 0x7fe3240720d0 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe328a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 0x7fe3240ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe328a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 0x7fe3240ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49468/0 (socket says 172.21.15.2:49468) 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.726+0000 7fe328a00640 1 -- 172.21.15.2:0/4221579766 learned_addr learned my addr 172.21.15.2:0/4221579766 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe323e00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3240720d0 0x7fe3241360b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe323400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe324143d20 0x7fe3240c7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe328a00640 1 -- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3240720d0 msgr2=0x7fe3241360b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe328a00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3240720d0 0x7fe3241360b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe328a00640 1 -- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe324143d20 msgr2=0x7fe3240c7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe328a00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe324143d20 0x7fe3240c7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe328a00640 1 -- 172.21.15.2:0/4221579766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe31c009b30 con 0x7fe3241447d0 2024-01-15T17:58:30.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe328a00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 0x7fe3240ce650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c550 tx=0x7fe31400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:30.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314004220 con 0x7fe3241447d0 2024-01-15T17:58:30.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe314004380 con 0x7fe3241447d0 2024-01-15T17:58:30.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314003830 con 0x7fe3241447d0 2024-01-15T17:58:30.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe324136dc0 con 0x7fe3241447d0 2024-01-15T17:58:30.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.727+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324137380 con 0x7fe3241447d0 2024-01-15T17:58:30.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.728+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe324137570 con 0x7fe3241447d0 2024-01-15T17:58:30.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.730+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe314003990 con 0x7fe3241447d0 2024-01-15T17:58:30.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.730+0000 7fe31be00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2fc06b2c0 0x7fe2fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:30.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.731+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe314089a10 con 0x7fe3241447d0 2024-01-15T17:58:30.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.731+0000 7fe323e00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2fc06b2c0 0x7fe2fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.732+0000 7fe323e00640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2fc06b2c0 0x7fe2fc06d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe31c035000 tx=0x7fe31c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:30.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.735+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe314054760 con 0x7fe3241447d0 2024-01-15T17:58:30.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.927+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe324074770 con 0x7fe3241447d0 2024-01-15T17:58:30.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.928+0000 7fe31be00640 1 -- 172.21.15.2:0/4221579766 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe31401e0a0 con 0x7fe3241447d0 2024-01-15T17:58:30.932 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.930+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2fc06b2c0 msgr2=0x7fe2fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.930+0000 7fe329400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2fc06b2c0 0x7fe2fc06d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe31c035000 tx=0x7fe31c030000 comp rx=0 tx=0).stop 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.930+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 msgr2=0x7fe3240ce650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.930+0000 7fe329400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 0x7fe3240ce650 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fe31400c550 tx=0x7fe31400ca10 comp rx=0 tx=0).stop 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 shutdown_connections 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2fc06b2c0 0x7fe2fc06d780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3241447d0 0x7fe3240ce650 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe324143d20 0x7fe3240c7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 --2- 172.21.15.2:0/4221579766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3240720d0 0x7fe3241360b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 >> 172.21.15.2:0/4221579766 conn(0x7fe3240b0840 msgr2=0x7fe32415d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 shutdown_connections 2024-01-15T17:58:30.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:30.931+0000 7fe329400640 1 -- 172.21.15.2:0/4221579766 wait complete. 2024-01-15T17:58:30.953 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:31.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.176+0000 7f9b18200640 1 Processor -- start 2024-01-15T17:58:31.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.177+0000 7f9b18200640 1 -- start start 2024-01-15T17:58:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.178+0000 7f9b18200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9b1006d000 0x7f9b10165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.178+0000 7f9b18200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b100ce230 con 0x7f9b10144c00 2024-01-15T17:58:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.178+0000 7f9b18200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b100ce510 con 0x7f9b1015c450 2024-01-15T17:58:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.178+0000 7f9b18200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b100ce7f0 con 0x7f9b1006d000 2024-01-15T17:58:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b15800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9b10144c00 0x7f9b100cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35246/0 (socket says 172.21.15.2:35246) 2024-01-15T17:58:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b15800640 1 -- 172.21.15.2:0/2965570124 learned_addr learned my addr 172.21.15.2:0/2965570124 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b16200640 1 --2- 172.21.15.2:0/2965570124 >> v2:172.21.15.137:3300/0 conn(0x7f9b1006d000 0x7f9b10165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413969039 0 0) 0x7f9b100ce230 con 0x7f9b10144c00 2024-01-15T17:58:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af0003520 con 0x7f9b10144c00 2024-01-15T17:58:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634750691 0 0) 0x7f9af0003520 con 0x7f9b10144c00 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 >> v2:172.21.15.137:3300/0 conn(0x7f9b1006d000 msgr2=0x7f9b10165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 --2- 172.21.15.2:0/2965570124 >> v2:172.21.15.137:3300/0 conn(0x7f9b1006d000 0x7f9b10165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 >> v1:172.21.15.137:6789/0 conn(0x7f9b1015c450 legacy=0x7f9b1006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b100705f0 con 0x7f9b10144c00 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.179+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9b00001670 con 0x7f9b10144c00 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 >> v1:172.21.15.2:6789/0 conn(0x7f9b10144c00 legacy=0x7f9b100cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af00039c0 0x7f9af0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00047c0 0x7f9af0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af0005180 0x7f9af000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b100ce7f0 con 0x7f9af00039c0 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af000c080 con 0x7f9af0005180 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9af000c240 con 0x7f9af00047c0 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b15800640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af00039c0 0x7f9af0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b14e00640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00047c0 0x7f9af0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b16200640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af0005180 0x7f9af000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b15800640 1 -- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00047c0 msgr2=0x7f9af0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b15800640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00047c0 0x7f9af0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b15800640 1 -- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af0005180 msgr2=0x7f9af000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b15800640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af0005180 0x7f9af000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b15800640 1 -- 172.21.15.2:0/2965570124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af0003520 con 0x7f9af00039c0 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b14e00640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00047c0 0x7f9af0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.180+0000 7f9b16200640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9af0005180 0x7f9af000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.181+0000 7f9b16200640 1 -- 172.21.15.2:0/2965570124 reap_dead start 2024-01-15T17:58:31.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.182+0000 7f9b15800640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af00039c0 0x7f9af0003de0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000c0e0 tx=0x7f9af00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7623dc9a806b1a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:31.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.182+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00005ad0 con 0x7f9af00039c0 2024-01-15T17:58:31.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.182+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b00005c30 con 0x7f9af00039c0 2024-01-15T17:58:31.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.182+0000 7f9b0fe00640 1 -- 172.21.15.2:0/2965570124 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0002ca50 con 0x7f9af00039c0 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.182+0000 7f9b18200640 1 -- 172.21.15.2:0/2965570124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af00039c0 msgr2=0x7f9af0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.182+0000 7f9b18200640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af00039c0 0x7f9af0003de0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f9b0000c0e0 tx=0x7f9af00057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.183+0000 7f9b18200640 1 -- 172.21.15.2:0/2965570124 shutdown_connections 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.183+0000 7f9b18200640 1 --2- 172.21.15.2:0/2965570124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9af00039c0 0x7f9af0003de0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.183+0000 7f9b18200640 1 -- 172.21.15.2:0/2965570124 >> 172.21.15.2:0/2965570124 conn(0x7f9b100b45f0 msgr2=0x7f9b100b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.183+0000 7f9b18200640 1 -- 172.21.15.2:0/2965570124 shutdown_connections 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.183+0000 7f9b18200640 1 -- 172.21.15.2:0/2965570124 wait complete. 2024-01-15T17:58:31.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.184+0000 7f9b18200640 1 Processor -- start 2024-01-15T17:58:31.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.185+0000 7f9b18200640 1 -- start start 2024-01-15T17:58:31.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b18200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00039c0 0x7f9b101ee1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b18200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 0x7f9b10143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b18200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b100af970 0x7f9b10143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b18200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b101f1ef0 con 0x7f9b100af4f0 2024-01-15T17:58:31.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b18200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b101f2d10 con 0x7f9b100af970 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b18200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b10137fe0 con 0x7f9af00039c0 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 0x7f9b10143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 0x7f9b10143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49488/0 (socket says 172.21.15.2:49488) 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 -- 172.21.15.2:0/2402906840 learned_addr learned my addr 172.21.15.2:0/2402906840 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b15800640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00039c0 0x7f9b101ee1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b16200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b100af970 0x7f9b10143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 -- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00039c0 msgr2=0x7f9b101ee1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00039c0 0x7f9b101ee1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 -- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b100af970 msgr2=0x7f9b10143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b100af970 0x7f9b10143ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b14e00640 1 -- 172.21.15.2:0/2402906840 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b101f2100 con 0x7f9b100af4f0 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.186+0000 7f9b16200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b100af970 0x7f9b10143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:31.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.187+0000 7f9b14e00640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 0x7f9b10143660 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9af800d4a0 tx=0x7f9af800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:31.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.187+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8005480 con 0x7f9b100af4f0 2024-01-15T17:58:31.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.187+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af8007030 con 0x7f9b100af4f0 2024-01-15T17:58:31.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.187+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af8007400 con 0x7f9b100af4f0 2024-01-15T17:58:31.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.187+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b101f23e0 con 0x7f9b100af4f0 2024-01-15T17:58:31.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.187+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b101f29a0 con 0x7f9b100af4f0 2024-01-15T17:58:31.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.188+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b101f2b30 con 0x7f9b100af4f0 2024-01-15T17:58:31.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.189+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9af8007560 con 0x7f9b100af4f0 2024-01-15T17:58:31.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.190+0000 7f9b0d600640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ae806b2f0 0x7f9ae806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.190+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9af8089130 con 0x7f9b100af4f0 2024-01-15T17:58:31.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.191+0000 7f9b15800640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ae806b2f0 0x7f9ae806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.192+0000 7f9b15800640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ae806b2f0 0x7f9ae806d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9b00034000 tx=0x7f9b00009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:31.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.195+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9af8053f30 con 0x7f9b100af4f0 2024-01-15T17:58:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.390+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9b100686f0 con 0x7f9b100af4f0 2024-01-15T17:58:31.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.392+0000 7f9b0d600640 1 -- 172.21.15.2:0/2402906840 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9af8057550 con 0x7f9b100af4f0 2024-01-15T17:58:31.396 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ae806b2f0 msgr2=0x7f9ae806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ae806b2f0 0x7f9ae806d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9b00034000 tx=0x7f9b00009030 comp rx=0 tx=0).stop 2024-01-15T17:58:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 msgr2=0x7f9b10143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 0x7f9b10143660 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f9af800d4a0 tx=0x7f9af800d960 comp rx=0 tx=0).stop 2024-01-15T17:58:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 shutdown_connections 2024-01-15T17:58:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ae806b2f0 0x7f9ae806d7b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b100af970 0x7f9b10143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b100af4f0 0x7f9b10143660 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 --2- 172.21.15.2:0/2402906840 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9af00039c0 0x7f9b101ee1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 >> 172.21.15.2:0/2402906840 conn(0x7f9b100b45f0 msgr2=0x7f9b1013a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 shutdown_connections 2024-01-15T17:58:31.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.394+0000 7f9b18200640 1 -- 172.21.15.2:0/2402906840 wait complete. 2024-01-15T17:58:31.419 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.641+0000 7fafb1e00640 1 Processor -- start 2024-01-15T17:58:31.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.641+0000 7fafb1e00640 1 -- start start 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafb1e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fafac1439e0 0x7fafac06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 0x7fafac070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafb1e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafac143dc0 con 0x7fafac06cdf0 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafb1e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafac06d280 con 0x7fafac1439e0 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafb1e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac15b620 con 0x7fafac0cc0a0 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafa3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fafac06cdf0 0x7fafac0cb990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35256/0 (socket says 172.21.15.2:35256) 2024-01-15T17:58:31.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.643+0000 7fafa3e00640 1 -- 172.21.15.2:0/1360824060 learned_addr learned my addr 172.21.15.2:0/1360824060 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:31.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.644+0000 7fafab400640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 0x7fafac070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.644+0000 7fafaaa00640 1 --2- 172.21.15.2:0/1360824060 >> v2:172.21.15.137:3300/0 conn(0x7fafac1439e0 0x7fafac06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.644+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027204206 0 0) 0x7fafac143dc0 con 0x7fafac06cdf0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.644+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c003520 con 0x7fafac06cdf0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.644+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 156952431 0 0) 0x7faf8c003520 con 0x7fafac06cdf0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 msgr2=0x7fafac070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 0x7fafac070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 >> v2:172.21.15.137:3300/0 conn(0x7fafac1439e0 msgr2=0x7fafac06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 --2- 172.21.15.2:0/1360824060 >> v2:172.21.15.137:3300/0 conn(0x7fafac1439e0 0x7fafac06c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0cd9f0 con 0x7fafac06cdf0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7faf94001670 con 0x7fafac06cdf0 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 >> v1:172.21.15.2:6789/0 conn(0x7fafac06cdf0 legacy=0x7fafac0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c003970 0x7faf8c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 0x7faf8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf8c004d00 0x7faf8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac06d280 con 0x7faf8c004d00 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac15b620 con 0x7faf8c003970 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8c00b980 con 0x7fafac0cc0a0 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafab400640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf8c004d00 0x7faf8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafaaa00640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c003970 0x7faf8c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.645+0000 7fafa3e00640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 0x7faf8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.646+0000 7fafab400640 1 -- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 msgr2=0x7faf8c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.646+0000 7fafab400640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0cc0a0 0x7faf8c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.646+0000 7fafab400640 1 -- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c003970 msgr2=0x7faf8c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.646+0000 7fafab400640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c003970 0x7faf8c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.646+0000 7fafab400640 1 -- 172.21.15.2:0/1360824060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8c003520 con 0x7faf8c004d00 2024-01-15T17:58:31.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.647+0000 7fafab400640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf8c004d00 0x7faf8c005180 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009bd0 tx=0x7faf8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d40bdfb55ef5f54 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.647+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa402ede0 con 0x7faf8c004d00 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.647+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa400f040 con 0x7faf8c004d00 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.647+0000 7fafaa000640 1 -- 172.21.15.2:0/1360824060 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4036640 con 0x7faf8c004d00 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafb1e00640 1 -- 172.21.15.2:0/1360824060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf8c004d00 msgr2=0x7faf8c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafb1e00640 1 --2- 172.21.15.2:0/1360824060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf8c004d00 0x7faf8c005180 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fafa4009bd0 tx=0x7faf8c0057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafab400640 1 -- 172.21.15.2:0/1360824060 reap_dead start 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafb1e00640 1 -- 172.21.15.2:0/1360824060 shutdown_connections 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafb1e00640 1 -- 172.21.15.2:0/1360824060 >> 172.21.15.2:0/1360824060 conn(0x7fafac0b46d0 msgr2=0x7fafac0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafb1e00640 1 -- 172.21.15.2:0/1360824060 shutdown_connections 2024-01-15T17:58:31.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.648+0000 7fafb1e00640 1 -- 172.21.15.2:0/1360824060 wait complete. 2024-01-15T17:58:31.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.650+0000 7fafb1e00640 1 Processor -- start 2024-01-15T17:58:31.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.650+0000 7fafb1e00640 1 -- start start 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c004d00 0x7fafac1355e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 0x7fafac159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafb1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0b26d0 0x7fafac159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafb1e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac0ad860 con 0x7fafac0b2250 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafb1e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac159f90 con 0x7faf8c004d00 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafb1e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac135b20 con 0x7fafac0b26d0 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafa3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 0x7fafac159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafa3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 0x7fafac159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49516/0 (socket says 172.21.15.2:49516) 2024-01-15T17:58:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.651+0000 7fafa3e00640 1 -- 172.21.15.2:0/4236928114 learned_addr learned my addr 172.21.15.2:0/4236928114 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafaaa00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c004d00 0x7fafac1355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafab400640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0b26d0 0x7fafac159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3e00640 1 -- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0b26d0 msgr2=0x7fafac159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0b26d0 0x7fafac159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3e00640 1 -- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c004d00 msgr2=0x7fafac1355e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c004d00 0x7fafac1355e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3e00640 1 -- 172.21.15.2:0/4236928114 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0acbe0 con 0x7fafac0b2250 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 0x7fafac159510 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faf9400e870 tx=0x7faf9400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9400c8d0 con 0x7fafac0b2250 2024-01-15T17:58:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9400ca30 con 0x7fafac0b2250 2024-01-15T17:58:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafac0acec0 con 0x7fafac0b2250 2024-01-15T17:58:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.652+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9400cde0 con 0x7fafac0b2250 2024-01-15T17:58:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.653+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafac0ad480 con 0x7fafac0b2250 2024-01-15T17:58:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.654+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafac0686f0 con 0x7fafac0b2250 2024-01-15T17:58:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.655+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faf9401d460 con 0x7fafac0b2250 2024-01-15T17:58:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.656+0000 7fafa3400640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf8406b2f0 0x7faf8406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.656+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faf94089bc0 con 0x7fafac0b2250 2024-01-15T17:58:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.656+0000 7fafaaa00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf8406b2f0 0x7faf8406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.657+0000 7fafaaa00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf8406b2f0 0x7faf8406d7b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009a10 tx=0x7faf9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.660+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf94054a40 con 0x7fafac0b2250 2024-01-15T17:58:31.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.867+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fafac070520 con 0x7fafac0b2250 2024-01-15T17:58:31.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.868+0000 7fafa3400640 1 -- 172.21.15.2:0/4236928114 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7faf94058060 con 0x7fafac0b2250 2024-01-15T17:58:31.872 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:31.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.870+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf8406b2f0 msgr2=0x7faf8406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.870+0000 7fafb1e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf8406b2f0 0x7faf8406d7b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7faf9c009a10 tx=0x7faf9c008040 comp rx=0 tx=0).stop 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.870+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 msgr2=0x7fafac159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.870+0000 7fafb1e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 0x7fafac159510 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faf9400e870 tx=0x7faf9400ed30 comp rx=0 tx=0).stop 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 shutdown_connections 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf8406b2f0 0x7faf8406d7b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fafac0b26d0 0x7fafac159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fafac0b2250 0x7fafac159510 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 --2- 172.21.15.2:0/4236928114 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf8c004d00 0x7fafac1355e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 >> 172.21.15.2:0/4236928114 conn(0x7fafac0b46d0 msgr2=0x7fafac15b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 shutdown_connections 2024-01-15T17:58:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:31.871+0000 7fafb1e00640 1 -- 172.21.15.2:0/4236928114 wait complete. 2024-01-15T17:58:31.893 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T17:58:31.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:31.893 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-15T17:58:31.990 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:31.991 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T17:58:31.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:31.991 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T17:58:32.042 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:58:32.043 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:32.100 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:58:32.100 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:32.150 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:32.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.377+0000 7f7cf1400640 1 Processor -- start 2024-01-15T17:58:32.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.377+0000 7f7cf1400640 1 -- start start 2024-01-15T17:58:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.379+0000 7f7cf1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 0x7f7cec071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.379+0000 7f7cf1400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7cec146dd0 0x7f7cec06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.379+0000 7f7cf1400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cec06cdf0 con 0x7f7cec06c9b0 2024-01-15T17:58:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.379+0000 7f7cf1400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cec0ac870 con 0x7f7cec146dd0 2024-01-15T17:58:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.379+0000 7f7cf1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec15d970 con 0x7f7cec1469f0 2024-01-15T17:58:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.380+0000 7f7ce3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7cec146dd0 0x7f7cec06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.380+0000 7f7cea800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 0x7f7cec071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.380+0000 7f7cea800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 0x7f7cec071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:53448/0 (socket says 172.21.15.2:53448) 2024-01-15T17:58:32.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.380+0000 7f7cea800640 1 -- 172.21.15.2:0/835988333 learned_addr learned my addr 172.21.15.2:0/835988333 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:32.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.381+0000 7f7cea800640 1 -- 172.21.15.2:0/835988333 >> v2:172.21.15.137:3300/0 conn(0x7f7cec146dd0 msgr2=0x7f7cec06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:32.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.381+0000 7f7cea800640 1 --2- 172.21.15.2:0/835988333 >> v2:172.21.15.137:3300/0 conn(0x7f7cec146dd0 0x7f7cec06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.381+0000 7f7cea800640 1 -- 172.21.15.2:0/835988333 >> v1:172.21.15.137:6789/0 conn(0x7f7cec06c9b0 legacy=0x7f7cec070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.381+0000 7f7cea800640 1 -- 172.21.15.2:0/835988333 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cec15dc30 con 0x7f7cec1469f0 2024-01-15T17:58:32.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.382+0000 7f7cea800640 1 --2- 172.21.15.2:0/835988333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 0x7f7cec071df0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67b8034124df355 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:32.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.382+0000 7f7ce9e00640 1 -- 172.21.15.2:0/835988333 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc030840 con 0x7f7cec1469f0 2024-01-15T17:58:32.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.382+0000 7f7ce9e00640 1 -- 172.21.15.2:0/835988333 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc0309a0 con 0x7f7cec1469f0 2024-01-15T17:58:32.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.382+0000 7f7ce9e00640 1 -- 172.21.15.2:0/835988333 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc030d30 con 0x7f7cec1469f0 2024-01-15T17:58:32.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.383+0000 7f7cf1400640 1 -- 172.21.15.2:0/835988333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 msgr2=0x7f7cec071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.383+0000 7f7cf1400640 1 --2- 172.21.15.2:0/835988333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 0x7f7cec071df0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc009870 tx=0x7f7cdc02b980 comp rx=0 tx=0).stop 2024-01-15T17:58:32.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.383+0000 7f7cf1400640 1 -- 172.21.15.2:0/835988333 shutdown_connections 2024-01-15T17:58:32.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.383+0000 7f7cf1400640 1 --2- 172.21.15.2:0/835988333 >> v2:172.21.15.137:3300/0 conn(0x7f7cec146dd0 0x7f7cec06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.383+0000 7f7cf1400640 1 --2- 172.21.15.2:0/835988333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec1469f0 0x7f7cec071df0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.383+0000 7f7cf1400640 1 -- 172.21.15.2:0/835988333 >> 172.21.15.2:0/835988333 conn(0x7f7cec06da20 msgr2=0x7f7cec06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:32.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.384+0000 7f7cf1400640 1 -- 172.21.15.2:0/835988333 shutdown_connections 2024-01-15T17:58:32.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.384+0000 7f7cf1400640 1 -- 172.21.15.2:0/835988333 wait complete. 2024-01-15T17:58:32.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.386+0000 7f7cf1400640 1 Processor -- start 2024-01-15T17:58:32.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.386+0000 7f7cf1400640 1 -- start start 2024-01-15T17:58:32.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.387+0000 7f7cf1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec06c9b0 0x7f7cec070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.387+0000 7f7cf1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cec1469f0 0x7f7cec138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.387+0000 7f7cf1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 0x7f7cec13a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.387+0000 7f7cf1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec138c30 con 0x7f7cec138780 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.387+0000 7f7cf1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec0708c0 con 0x7f7cec1469f0 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.387+0000 7f7cf1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cec071700 con 0x7f7cec06c9b0 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 0x7f7cec13a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 0x7f7cec13a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49518/0 (socket says 172.21.15.2:49518) 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 -- 172.21.15.2:0/2473258006 learned_addr learned my addr 172.21.15.2:0/2473258006 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ce3e00640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cec1469f0 0x7f7cec138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7cea800640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec06c9b0 0x7f7cec070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 -- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec06c9b0 msgr2=0x7f7cec070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec06c9b0 0x7f7cec070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 -- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cec1469f0 msgr2=0x7f7cec138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cec1469f0 0x7f7cec138240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.388+0000 7f7ceb200640 1 -- 172.21.15.2:0/2473258006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cdc0094d0 con 0x7f7cec138780 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.389+0000 7f7ceb200640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 0x7f7cec13a980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400c7d0 tx=0x7f7ce400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.389+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce4003b70 con 0x7f7cec138780 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.389+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce4003cf0 con 0x7f7cec138780 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.389+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce400aa90 con 0x7f7cec138780 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.389+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cec070af0 con 0x7f7cec138780 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.389+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec0710b0 con 0x7f7cec138780 2024-01-15T17:58:32.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.390+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cec0686f0 con 0x7f7cec138780 2024-01-15T17:58:32.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.391+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7ce400abf0 con 0x7f7cec138780 2024-01-15T17:58:32.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.392+0000 7f7ce2a00640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7cc406b2c0 0x7f7cc406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.392+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7ce4089920 con 0x7f7cec138780 2024-01-15T17:58:32.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.393+0000 7f7cea800640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7cc406b2c0 0x7f7cc406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.394+0000 7f7cea800640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7cc406b2c0 0x7f7cc406d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc036000 tx=0x7f7cdc008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.397+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ce40547b0 con 0x7f7cec138780 2024-01-15T17:58:32.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.597+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cec071840 con 0x7f7cec138780 2024-01-15T17:58:32.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.599+0000 7f7ce2a00640 1 -- 172.21.15.2:0/2473258006 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f7ce4003e60 con 0x7f7cec138780 2024-01-15T17:58:32.603 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:32.604 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7cc406b2c0 msgr2=0x7f7cc406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7cc406b2c0 0x7f7cc406d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc036000 tx=0x7f7cdc008030 comp rx=0 tx=0).stop 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 msgr2=0x7f7cec13a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 0x7f7cec13a980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f7ce400c7d0 tx=0x7f7ce400cc90 comp rx=0 tx=0).stop 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 shutdown_connections 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7cc406b2c0 0x7f7cc406d780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7cec138780 0x7f7cec13a980 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7cec1469f0 0x7f7cec138240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 --2- 172.21.15.2:0/2473258006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7cec06c9b0 0x7f7cec070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 >> 172.21.15.2:0/2473258006 conn(0x7f7cec06da20 msgr2=0x7f7cec0ce2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 shutdown_connections 2024-01-15T17:58:32.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.602+0000 7f7cf1400640 1 -- 172.21.15.2:0/2473258006 wait complete. 2024-01-15T17:58:32.626 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:32.627 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:32.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.847+0000 7fa933e00640 1 Processor -- start 2024-01-15T17:58:32.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.847+0000 7fa933e00640 1 -- start start 2024-01-15T17:58:32.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.848+0000 7fa933e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa93406d6d0 0x7fa9340ce770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.848+0000 7fa933e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9340ce2e0 con 0x7fa93415d410 2024-01-15T17:58:32.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.848+0000 7fa933e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93406cd70 con 0x7fa93415e470 2024-01-15T17:58:32.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.848+0000 7fa933e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9340cd9b0 con 0x7fa93406d6d0 2024-01-15T17:58:32.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.848+0000 7fa932a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa93415d410 0x7fa93406c520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35262/0 (socket says 172.21.15.2:35262) 2024-01-15T17:58:32.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.848+0000 7fa932a00640 1 -- 172.21.15.2:0/2426052689 learned_addr learned my addr 172.21.15.2:0/2426052689 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:32.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.849+0000 7fa933400640 1 --2- 172.21.15.2:0/2426052689 >> v2:172.21.15.2:3300/0 conn(0x7fa93406d6d0 0x7fa9340ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.849+0000 7fa931600640 1 -- 172.21.15.2:0/2426052689 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2960839689 0 0) 0x7fa9340ce2e0 con 0x7fa93415d410 2024-01-15T17:58:32.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.849+0000 7fa931600640 1 -- 172.21.15.2:0/2426052689 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa914003520 con 0x7fa93415d410 2024-01-15T17:58:32.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.849+0000 7fa933400640 1 -- 172.21.15.2:0/2426052689 >> v1:172.21.15.137:6789/0 conn(0x7fa93415e470 legacy=0x7fa93406cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.849+0000 7fa933400640 1 -- 172.21.15.2:0/2426052689 >> v1:172.21.15.2:6789/0 conn(0x7fa93415d410 legacy=0x7fa93406c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.849+0000 7fa933400640 1 -- 172.21.15.2:0/2426052689 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9340cdc90 con 0x7fa93406d6d0 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.850+0000 7fa933400640 1 --2- 172.21.15.2:0/2426052689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340ce770 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009d90 tx=0x7fa92c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01a709bcf47129 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa931600640 1 -- 172.21.15.2:0/2426052689 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c02cd00 con 0x7fa93406d6d0 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa931600640 1 -- 172.21.15.2:0/2426052689 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92c02ce60 con 0x7fa93406d6d0 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa931600640 1 -- 172.21.15.2:0/2426052689 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c034640 con 0x7fa93406d6d0 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 -- 172.21.15.2:0/2426052689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 msgr2=0x7fa9340ce770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 --2- 172.21.15.2:0/2426052689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340ce770 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009d90 tx=0x7fa92c02bf50 comp rx=0 tx=0).stop 2024-01-15T17:58:32.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 -- 172.21.15.2:0/2426052689 shutdown_connections 2024-01-15T17:58:32.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 --2- 172.21.15.2:0/2426052689 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340ce770 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 -- 172.21.15.2:0/2426052689 >> 172.21.15.2:0/2426052689 conn(0x7fa9340b0c10 msgr2=0x7fa9340b1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:32.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 -- 172.21.15.2:0/2426052689 shutdown_connections 2024-01-15T17:58:32.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.851+0000 7fa933e00640 1 -- 172.21.15.2:0/2426052689 wait complete. 2024-01-15T17:58:32.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.853+0000 7fa933e00640 1 Processor -- start 2024-01-15T17:58:32.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.853+0000 7fa933e00640 1 -- start start 2024-01-15T17:58:32.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa933e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa933e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa93415d410 0x7fa934160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa933e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa93415e470 0x7fa93415a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa933e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9340b3020 con 0x7fa93406d6d0 2024-01-15T17:58:32.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa933e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93415a890 con 0x7fa93415d410 2024-01-15T17:58:32.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa933e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9340b3890 con 0x7fa93415e470 2024-01-15T17:58:32.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.854+0000 7fa932a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49524/0 (socket says 172.21.15.2:49524) 2024-01-15T17:58:32.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 -- 172.21.15.2:0/1585468109 learned_addr learned my addr 172.21.15.2:0/1585468109 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa933400640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa93415e470 0x7fa93415a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932000640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa93415d410 0x7fa934160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 -- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa93415e470 msgr2=0x7fa93415a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa93415e470 0x7fa93415a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 -- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa93415d410 msgr2=0x7fa934160390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa93415d410 0x7fa934160390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 -- 172.21.15.2:0/1585468109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c009a70 con 0x7fa93406d6d0 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.855+0000 7fa932a00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340c5690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c500 tx=0x7fa92400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:32.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.856+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924005ce0 con 0x7fa93406d6d0 2024-01-15T17:58:32.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.856+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa924005e40 con 0x7fa93406d6d0 2024-01-15T17:58:32.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.856+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924007330 con 0x7fa93406d6d0 2024-01-15T17:58:32.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.856+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9340b3b70 con 0x7fa93406d6d0 2024-01-15T17:58:32.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.856+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa934070500 con 0x7fa93406d6d0 2024-01-15T17:58:32.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.857+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9340ceeb0 con 0x7fa93406d6d0 2024-01-15T17:58:32.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.858+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa924007550 con 0x7fa93406d6d0 2024-01-15T17:58:32.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.859+0000 7fa92aa00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa90c06b2c0 0x7fa90c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:32.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.859+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa9240892f0 con 0x7fa93406d6d0 2024-01-15T17:58:32.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.860+0000 7fa932000640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa90c06b2c0 0x7fa90c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:32.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.861+0000 7fa932000640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa90c06b2c0 0x7fa90c06d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa9340b0790 tx=0x7fa91c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:32.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:32.864+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9240541a0 con 0x7fa93406d6d0 2024-01-15T17:58:33.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.057+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa934074770 con 0x7fa93406d6d0 2024-01-15T17:58:33.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.058+0000 7fa92aa00640 1 -- 172.21.15.2:0/1585468109 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9240577c0 con 0x7fa93406d6d0 2024-01-15T17:58:33.062 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:33.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa90c06b2c0 msgr2=0x7fa90c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa90c06b2c0 0x7fa90c06d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa9340b0790 tx=0x7fa91c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 msgr2=0x7fa9340c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340c5690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c500 tx=0x7fa92400c9c0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 shutdown_connections 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa90c06b2c0 0x7fa90c06d780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa93415e470 0x7fa93415a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa93415d410 0x7fa934160390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 --2- 172.21.15.2:0/1585468109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa93406d6d0 0x7fa9340c5690 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.060+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 >> 172.21.15.2:0/1585468109 conn(0x7fa9340b0c10 msgr2=0x7fa9340cafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.061+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 shutdown_connections 2024-01-15T17:58:33.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.061+0000 7fa933e00640 1 -- 172.21.15.2:0/1585468109 wait complete. 2024-01-15T17:58:33.085 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:33.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.307+0000 7f7402a00640 1 Processor -- start 2024-01-15T17:58:33.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.307+0000 7f7402a00640 1 -- start start 2024-01-15T17:58:33.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f7402a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 0x7f73fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f7402a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc0ce230 con 0x7f73fc144c00 2024-01-15T17:58:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f7402a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc0ce510 con 0x7f73fc06ce30 2024-01-15T17:58:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f7402a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc06d2c0 con 0x7f73fc15c450 2024-01-15T17:58:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f73fbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f73fc144c00 0x7f73fc0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49086/0 (socket says 172.21.15.2:49086) 2024-01-15T17:58:33.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f73fbe00640 1 -- 172.21.15.2:0/2712559246 learned_addr learned my addr 172.21.15.2:0/2712559246 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f73f3e00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 0x7f73fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.309+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649737709 0 0) 0x7f73fc0ce230 con 0x7f73fc144c00 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73dc003520 con 0x7f73fc144c00 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 519422769 0 0) 0x7f73dc003520 con 0x7f73fc144c00 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 msgr2=0x7f73fc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 0x7f73fc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 >> v1:172.21.15.137:6789/0 conn(0x7f73fc06ce30 legacy=0x7f73fc070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc0ce7f0 con 0x7f73fc144c00 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f73ec001670 con 0x7f73fc144c00 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 >> v1:172.21.15.2:6789/0 conn(0x7f73fc144c00 legacy=0x7f73fc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 0x7f73dc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc003f20 0x7f73dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc06d2c0 con 0x7f73dc004d00 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc00ba60 con 0x7f73dc003f20 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.310+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73dc00bc20 con 0x7f73fc15c450 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f7400a00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f73fbe00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 0x7f73dc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f73f3e00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc003f20 0x7f73dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f7400a00640 1 -- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 msgr2=0x7f73dc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f7400a00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc15c450 0x7f73dc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f7400a00640 1 -- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc003f20 msgr2=0x7f73dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f7400a00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73dc003f20 0x7f73dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.311+0000 7f7400a00640 1 -- 172.21.15.2:0/2712559246 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc003520 con 0x7f73dc004d00 2024-01-15T17:58:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.313+0000 7f7400a00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73dc005180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f73f4001e50 tx=0x7f73dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c97ba13c81412b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.313+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4003b70 con 0x7f73dc004d00 2024-01-15T17:58:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.313+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f4003cd0 con 0x7f73dc004d00 2024-01-15T17:58:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.313+0000 7f73fb400640 1 -- 172.21.15.2:0/2712559246 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4005310 con 0x7f73dc004d00 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7402a00640 1 -- 172.21.15.2:0/2712559246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 msgr2=0x7f73dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7402a00640 1 --2- 172.21.15.2:0/2712559246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73dc005180 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f73f4001e50 tx=0x7f73dc0057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7400a00640 1 -- 172.21.15.2:0/2712559246 reap_dead start 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7402a00640 1 -- 172.21.15.2:0/2712559246 shutdown_connections 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7402a00640 1 -- 172.21.15.2:0/2712559246 >> 172.21.15.2:0/2712559246 conn(0x7f73fc0b45f0 msgr2=0x7f73fc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7402a00640 1 -- 172.21.15.2:0/2712559246 shutdown_connections 2024-01-15T17:58:33.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.314+0000 7f7402a00640 1 -- 172.21.15.2:0/2712559246 wait complete. 2024-01-15T17:58:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.316+0000 7f7402a00640 1 Processor -- start 2024-01-15T17:58:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.316+0000 7f7402a00640 1 -- start start 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f7402a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73fc0b0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f7402a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc0a8460 0x7f73fc13b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f7402a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc140b10 0x7f73fc140f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f7402a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc1414d0 con 0x7f73dc004d00 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f7402a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0b1410 con 0x7f73fc140b10 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f7402a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0afe30 con 0x7f73fc0a8460 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73fc0b0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f73fbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73fc0b0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60952/0 (socket says 172.21.15.2:60952) 2024-01-15T17:58:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.317+0000 7f73fbe00640 1 -- 172.21.15.2:0/4251335873 learned_addr learned my addr 172.21.15.2:0/4251335873 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f7400a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc140b10 0x7f73fc140f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73f3e00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc0a8460 0x7f73fc13b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73fbe00640 1 -- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc0a8460 msgr2=0x7f73fc13b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73fbe00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc0a8460 0x7f73fc13b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73fbe00640 1 -- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc140b10 msgr2=0x7f73fc140f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73fbe00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc140b10 0x7f73fc140f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73fbe00640 1 -- 172.21.15.2:0/4251335873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc0b0110 con 0x7f73dc004d00 2024-01-15T17:58:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73fbe00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73fc0b0ed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0072c0 tx=0x7f73ec007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec00f960 con 0x7f73dc004d00 2024-01-15T17:58:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec00fac0 con 0x7f73dc004d00 2024-01-15T17:58:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec00fe70 con 0x7f73dc004d00 2024-01-15T17:58:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.318+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc0b03f0 con 0x7f73dc004d00 2024-01-15T17:58:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.319+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc0b09b0 con 0x7f73dc004d00 2024-01-15T17:58:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.320+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc0686f0 con 0x7f73dc004d00 2024-01-15T17:58:33.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.321+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f73ec01d460 con 0x7f73dc004d00 2024-01-15T17:58:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.322+0000 7f73f8c00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73d406b310 0x7f73d406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.322+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f73ec089e60 con 0x7f73dc004d00 2024-01-15T17:58:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.323+0000 7f73f3e00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73d406b310 0x7f73d406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.324+0000 7f73f3e00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73d406b310 0x7f73d406d7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0ae490 tx=0x7f73e4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:33.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.327+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73ec054c30 con 0x7f73dc004d00 2024-01-15T17:58:33.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.531+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f73fc0cc5b0 con 0x7f73dc004d00 2024-01-15T17:58:33.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.532+0000 7f73f8c00640 1 -- 172.21.15.2:0/4251335873 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f73ec058250 con 0x7f73dc004d00 2024-01-15T17:58:33.536 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73d406b310 msgr2=0x7f73d406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73d406b310 0x7f73d406d7d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f73fc0ae490 tx=0x7f73e4008040 comp rx=0 tx=0).stop 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 msgr2=0x7f73fc0b0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73fc0b0ed0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f73ec0072c0 tx=0x7f73ec007780 comp rx=0 tx=0).stop 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 shutdown_connections 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73d406b310 0x7f73d406d7d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73fc140b10 0x7f73fc140f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73fc0a8460 0x7f73fc13b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 --2- 172.21.15.2:0/4251335873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73dc004d00 0x7f73fc0b0ed0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 >> 172.21.15.2:0/4251335873 conn(0x7f73fc0b45f0 msgr2=0x7f73fc070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:33.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 shutdown_connections 2024-01-15T17:58:33.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.535+0000 7f7402a00640 1 -- 172.21.15.2:0/4251335873 wait complete. 2024-01-15T17:58:33.559 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:33.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.775+0000 7ffaa4000640 1 Processor -- start 2024-01-15T17:58:33.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.775+0000 7ffaa4000640 1 -- start start 2024-01-15T17:58:33.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.777+0000 7ffaa4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c15c450 0x7ffa9c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.777+0000 7ffaa4000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ffa9c06ce30 0x7ffa9c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.777+0000 7ffaa4000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0ce230 con 0x7ffa9c144c00 2024-01-15T17:58:33.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.777+0000 7ffaa4000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa9c06d2c0 con 0x7ffa9c06ce30 2024-01-15T17:58:33.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.777+0000 7ffaa4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0ce510 con 0x7ffa9c15c450 2024-01-15T17:58:33.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.778+0000 7ffaa1600640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7ffa9c144c00 0x7ffa9c0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:36408/0 (socket says 172.21.15.2:36408) 2024-01-15T17:58:33.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.778+0000 7ffaa1600640 1 -- 172.21.15.2:0/3540071758 learned_addr learned my addr 172.21.15.2:0/3540071758 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:33.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.778+0000 7ffaa2000640 1 --2- 172.21.15.2:0/3540071758 >> v2:172.21.15.137:3300/0 conn(0x7ffa9c06ce30 0x7ffa9c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.778+0000 7ffaa0c00640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c15c450 0x7ffa9c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffa9be00640 1 -- 172.21.15.2:0/3540071758 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308824533 0 0) 0x7ffa9c0ce230 con 0x7ffa9c144c00 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffa9be00640 1 -- 172.21.15.2:0/3540071758 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa7c003520 con 0x7ffa9c144c00 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffaa2000640 1 -- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c15c450 msgr2=0x7ffa9c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffaa2000640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c15c450 0x7ffa9c06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffaa2000640 1 -- 172.21.15.2:0/3540071758 >> v1:172.21.15.137:6789/0 conn(0x7ffa9c144c00 legacy=0x7ffa9c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffaa2000640 1 -- 172.21.15.2:0/3540071758 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0ce7f0 con 0x7ffa9c06ce30 2024-01-15T17:58:33.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.779+0000 7ffaa0c00640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c15c450 0x7ffa9c06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:58:33.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.780+0000 7ffaa2000640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c070b10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009d90 tx=0x7ffa9402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=651fa39a6bdf0a22 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:33.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.781+0000 7ffa9be00640 1 -- 172.21.15.2:0/3540071758 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9402cc60 con 0x7ffa9c06ce30 2024-01-15T17:58:33.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.781+0000 7ffa9be00640 1 -- 172.21.15.2:0/3540071758 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9402cdc0 con 0x7ffa9c06ce30 2024-01-15T17:58:33.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.781+0000 7ffa9be00640 1 -- 172.21.15.2:0/3540071758 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94034660 con 0x7ffa9c06ce30 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 -- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 msgr2=0x7ffa9c070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c070b10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009d90 tx=0x7ffa9402beb0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 -- 172.21.15.2:0/3540071758 shutdown_connections 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c070b10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 --2- 172.21.15.2:0/3540071758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c15c450 0x7ffa9c06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 -- 172.21.15.2:0/3540071758 >> 172.21.15.2:0/3540071758 conn(0x7ffa9c0b45f0 msgr2=0x7ffa9c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 -- 172.21.15.2:0/3540071758 shutdown_connections 2024-01-15T17:58:33.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.782+0000 7ffaa4000640 1 -- 172.21.15.2:0/3540071758 wait complete. 2024-01-15T17:58:33.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.784+0000 7ffaa4000640 1 Processor -- start 2024-01-15T17:58:33.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.784+0000 7ffaa4000640 1 -- start start 2024-01-15T17:58:33.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.785+0000 7ffaa4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c0b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.785+0000 7ffaa4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c144c00 0x7ffa9c155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.785+0000 7ffaa4000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 0x7ffa9c13e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.785+0000 7ffaa4000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0b2f00 con 0x7ffa9c15c450 2024-01-15T17:58:33.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.785+0000 7ffaa4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0b08b0 con 0x7ffa9c06ce30 2024-01-15T17:58:33.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.785+0000 7ffaa4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa9c0b0b90 con 0x7ffa9c144c00 2024-01-15T17:58:33.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 0x7ffa9c13e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 0x7ffa9c13e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60956/0 (socket says 172.21.15.2:60956) 2024-01-15T17:58:33.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 -- 172.21.15.2:0/2164428245 learned_addr learned my addr 172.21.15.2:0/2164428245 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa1600640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c0b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa0c00640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c144c00 0x7ffa9c155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 -- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c144c00 msgr2=0x7ffa9c155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c144c00 0x7ffa9c155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 -- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 msgr2=0x7ffa9c0b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c0b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 -- 172.21.15.2:0/2164428245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa84009400 con 0x7ffa9c15c450 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.786+0000 7ffaa2000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 0x7ffa9c13e8d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009eb0 tx=0x7ffa94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.787+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9403e610 con 0x7ffa9c15c450 2024-01-15T17:58:33.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.787+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9403e770 con 0x7ffa9c15c450 2024-01-15T17:58:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.787+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9403eb20 con 0x7ffa9c15c450 2024-01-15T17:58:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.787+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa94009a70 con 0x7ffa9c15c450 2024-01-15T17:58:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.787+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c0b18d0 con 0x7ffa9c15c450 2024-01-15T17:58:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.789+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c0686f0 con 0x7ffa9c15c450 2024-01-15T17:58:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.789+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ffa9403b050 con 0x7ffa9c15c450 2024-01-15T17:58:33.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.790+0000 7ffa99600640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa7406b2c0 0x7ffa7406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:33.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.790+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ffa940ac440 con 0x7ffa9c15c450 2024-01-15T17:58:33.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.791+0000 7ffaa1600640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa7406b2c0 0x7ffa7406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:33.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.792+0000 7ffaa1600640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa7406b2c0 0x7ffa7406d780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c005880 tx=0x7ffa8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:33.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.795+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffa94077230 con 0x7ffa9c15c450 2024-01-15T17:58:33.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.990+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffa9c15af20 con 0x7ffa9c15c450 2024-01-15T17:58:33.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.991+0000 7ffa99600640 1 -- 172.21.15.2:0/2164428245 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffa9407a850 con 0x7ffa9c15c450 2024-01-15T17:58:33.995 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.993+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa7406b2c0 msgr2=0x7ffa7406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.993+0000 7ffaa4000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa7406b2c0 0x7ffa7406d780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c005880 tx=0x7ffa8c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.993+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 msgr2=0x7ffa9c13e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.993+0000 7ffaa4000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 0x7ffa9c13e8d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ffa94009eb0 tx=0x7ffa94004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 shutdown_connections 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa7406b2c0 0x7ffa7406d780 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffa9c15c450 0x7ffa9c13e8d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffa9c144c00 0x7ffa9c155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 --2- 172.21.15.2:0/2164428245 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffa9c06ce30 0x7ffa9c0b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 >> 172.21.15.2:0/2164428245 conn(0x7ffa9c0b45f0 msgr2=0x7ffa9c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 shutdown_connections 2024-01-15T17:58:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:33.994+0000 7ffaa4000640 1 -- 172.21.15.2:0/2164428245 wait complete. 2024-01-15T17:58:34.016 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:34.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.237+0000 7f25cd800640 1 Processor -- start 2024-01-15T17:58:34.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.238+0000 7f25cd800640 1 -- start start 2024-01-15T17:58:34.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.239+0000 7f25cd800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f25c80d2de0 0x7f25c80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.239+0000 7f25cd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c80d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.239+0000 7f25cd800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c80b30e0 con 0x7f25c815d200 2024-01-15T17:58:34.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.239+0000 7f25cd800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c80d3780 con 0x7f25c80d2de0 2024-01-15T17:58:34.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.239+0000 7f25cd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80d1ca0 con 0x7f25c80d0ed0 2024-01-15T17:58:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.240+0000 7f25cce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c80d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.240+0000 7f25c7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f25c80d2de0 0x7f25c80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.240+0000 7f25c7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f25c80d2de0 0x7f25c80d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:34540/0 (socket says 172.21.15.2:34540) 2024-01-15T17:58:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.240+0000 7f25c7400640 1 -- 172.21.15.2:0/1055845119 learned_addr learned my addr 172.21.15.2:0/1055845119 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.240+0000 7f25c7e00640 1 --1- 172.21.15.2:0/1055845119 >> v1:172.21.15.137:6789/0 conn(0x7f25c815d200 0x7f25c80cdc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:36416/0 (socket says 172.21.15.2:36416) 2024-01-15T17:58:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.240+0000 7f25cce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c80d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:40152/0 (socket says 172.21.15.2:40152) 2024-01-15T17:58:34.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.241+0000 7f25c7400640 1 -- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 msgr2=0x7f25c80d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:34.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.241+0000 7f25c7400640 1 --2- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c80d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.241+0000 7f25c7400640 1 -- 172.21.15.2:0/1055845119 >> v1:172.21.15.137:6789/0 conn(0x7f25c815d200 legacy=0x7f25c80cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.241+0000 7f25c7400640 1 -- 172.21.15.2:0/1055845119 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c80d1f80 con 0x7f25c80d2de0 2024-01-15T17:58:34.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.242+0000 7f25c7400640 1 --2- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c80d2de0 0x7f25c80d3240 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009870 tx=0x7f25b002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2778264fe456aee9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.243+0000 7f25c6a00640 1 -- 172.21.15.2:0/1055845119 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0030840 con 0x7f25c80d2de0 2024-01-15T17:58:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.243+0000 7f25c6a00640 1 -- 172.21.15.2:0/1055845119 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b00309a0 con 0x7f25c80d2de0 2024-01-15T17:58:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.243+0000 7f25c6a00640 1 -- 172.21.15.2:0/1055845119 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0030d30 con 0x7f25c80d2de0 2024-01-15T17:58:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.243+0000 7f25cd800640 1 -- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c80d2de0 msgr2=0x7f25c80d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.243+0000 7f25cd800640 1 --2- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c80d2de0 0x7f25c80d3240 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009870 tx=0x7f25b002b980 comp rx=0 tx=0).stop 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.244+0000 7f25cd800640 1 -- 172.21.15.2:0/1055845119 shutdown_connections 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.244+0000 7f25cd800640 1 --2- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c80d1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.244+0000 7f25cd800640 1 --2- 172.21.15.2:0/1055845119 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c80d2de0 0x7f25c80d3240 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.244+0000 7f25cd800640 1 -- 172.21.15.2:0/1055845119 >> 172.21.15.2:0/1055845119 conn(0x7f25c808caf0 msgr2=0x7f25c809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.244+0000 7f25cd800640 1 -- 172.21.15.2:0/1055845119 shutdown_connections 2024-01-15T17:58:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.244+0000 7f25cd800640 1 -- 172.21.15.2:0/1055845119 wait complete. 2024-01-15T17:58:34.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.245+0000 7f25cd800640 1 Processor -- start 2024-01-15T17:58:34.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.246+0000 7f25cd800640 1 -- start start 2024-01-15T17:58:34.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c8157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cd800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 0x7f25c81f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c815d200 0x7f25c80b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cd800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8159830 con 0x7f25c80d2de0 2024-01-15T17:58:34.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80b5d50 con 0x7f25c815d200 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c81f7380 con 0x7f25c80d0ed0 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 0x7f25c81f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 0x7f25c81f68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60972/0 (socket says 172.21.15.2:60972) 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25c7400640 1 -- 172.21.15.2:0/2186975132 learned_addr learned my addr 172.21.15.2:0/2186975132 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25c7e00640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c8157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.247+0000 7f25cce00640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c815d200 0x7f25c80b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25c7400640 1 -- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 msgr2=0x7f25c8157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25c7400640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c8157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25c7400640 1 -- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c815d200 msgr2=0x7f25c80b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25c7400640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c815d200 0x7f25c80b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25c7400640 1 -- 172.21.15.2:0/2186975132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00094d0 con 0x7f25c80d2de0 2024-01-15T17:58:34.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25cce00640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c815d200 0x7f25c80b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:34.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25c7400640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 0x7f25c81f68a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009990 tx=0x7f25b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:34.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0039c90 con 0x7f25c80d2de0 2024-01-15T17:58:34.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0039e10 con 0x7f25c80d2de0 2024-01-15T17:58:34.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b00446c0 con 0x7f25c80d2de0 2024-01-15T17:58:34.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c81f7660 con 0x7f25c80d2de0 2024-01-15T17:58:34.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.248+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c8159450 con 0x7f25c80d2de0 2024-01-15T17:58:34.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.250+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c8074770 con 0x7f25c80d2de0 2024-01-15T17:58:34.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.250+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25b002f390 con 0x7f25c80d2de0 2024-01-15T17:58:34.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.251+0000 7f25bfe00640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f25a006b2c0 0x7f25a006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.251+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f25b003e070 con 0x7f25c80d2de0 2024-01-15T17:58:34.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.252+0000 7f25c7e00640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f25a006b2c0 0x7f25a006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.253+0000 7f25c7e00640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f25a006b2c0 0x7f25a006d780 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f25b80012e0 tx=0x7f25b8005b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:34.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.256+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25b0077140 con 0x7f25c80d2de0 2024-01-15T17:58:34.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.465+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f25c80678e0 con 0x7f25c80d2de0 2024-01-15T17:58:34.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.465+0000 7f25bfe00640 1 -- 172.21.15.2:0/2186975132 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f25b007a760 con 0x7f25c80d2de0 2024-01-15T17:58:34.469 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:34.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f25a006b2c0 msgr2=0x7f25a006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f25a006b2c0 0x7f25a006d780 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f25b80012e0 tx=0x7f25b8005b70 comp rx=0 tx=0).stop 2024-01-15T17:58:34.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 msgr2=0x7f25c81f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 0x7f25c81f68a0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f25b0009990 tx=0x7f25b0004730 comp rx=0 tx=0).stop 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 shutdown_connections 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f25a006b2c0 0x7f25a006d780 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f25c815d200 0x7f25c80b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25c80d2de0 0x7f25c81f68a0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 --2- 172.21.15.2:0/2186975132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25c80d0ed0 0x7f25c8157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 >> 172.21.15.2:0/2186975132 conn(0x7f25c808caf0 msgr2=0x7f25c80afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 shutdown_connections 2024-01-15T17:58:34.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.468+0000 7f25cd800640 1 -- 172.21.15.2:0/2186975132 wait complete. 2024-01-15T17:58:34.493 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T17:58:34.493 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:58:34.548 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:34.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.770+0000 7f2373e00640 1 Processor -- start 2024-01-15T17:58:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.771+0000 7f2373e00640 1 -- start start 2024-01-15T17:58:34.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.772+0000 7f2373e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f236c15c2f0 0x7f236c15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.772+0000 7f2373e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 0x7f236c0b3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2373e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236c15f4f0 con 0x7f236c1445f0 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2373e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236c1449d0 con 0x7f236c15c2f0 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2373e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0b39d0 con 0x7f236c0b3000 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2371e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 0x7f236c0b3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2370a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f236c15c2f0 0x7f236c15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2371e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 0x7f236c0b3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:40168/0 (socket says 172.21.15.2:40168) 2024-01-15T17:58:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.773+0000 7f2371e00640 1 -- 172.21.15.2:0/3644510810 learned_addr learned my addr 172.21.15.2:0/3644510810 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.774+0000 7f2371e00640 1 -- 172.21.15.2:0/3644510810 >> v2:172.21.15.137:3300/0 conn(0x7f236c15c2f0 msgr2=0x7f236c15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.774+0000 7f2371e00640 1 --2- 172.21.15.2:0/3644510810 >> v2:172.21.15.137:3300/0 conn(0x7f236c15c2f0 0x7f236c15ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.774+0000 7f2371e00640 1 -- 172.21.15.2:0/3644510810 >> v1:172.21.15.137:6789/0 conn(0x7f236c1445f0 legacy=0x7f236c0b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.774+0000 7f2371e00640 1 -- 172.21.15.2:0/3644510810 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c15f7d0 con 0x7f236c0b3000 2024-01-15T17:58:34.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.776+0000 7f2371e00640 1 --2- 172.21.15.2:0/3644510810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 0x7f236c0b3490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2364009e50 tx=0x7f236402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b595cffe2a175d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:34.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.776+0000 7f236be00640 1 -- 172.21.15.2:0/3644510810 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236402cb00 con 0x7f236c0b3000 2024-01-15T17:58:34.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.776+0000 7f236be00640 1 -- 172.21.15.2:0/3644510810 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236402cc60 con 0x7f236c0b3000 2024-01-15T17:58:34.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f236be00640 1 -- 172.21.15.2:0/3644510810 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640343f0 con 0x7f236c0b3000 2024-01-15T17:58:34.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 -- 172.21.15.2:0/3644510810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 msgr2=0x7f236c0b3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 --2- 172.21.15.2:0/3644510810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 0x7f236c0b3490 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2364009e50 tx=0x7f236402bf60 comp rx=0 tx=0).stop 2024-01-15T17:58:34.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 -- 172.21.15.2:0/3644510810 shutdown_connections 2024-01-15T17:58:34.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 --2- 172.21.15.2:0/3644510810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c0b3000 0x7f236c0b3490 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 --2- 172.21.15.2:0/3644510810 >> v2:172.21.15.137:3300/0 conn(0x7f236c15c2f0 0x7f236c15ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 -- 172.21.15.2:0/3644510810 >> 172.21.15.2:0/3644510810 conn(0x7f236c0663d0 msgr2=0x7f236c0667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:34.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.777+0000 7f2373e00640 1 -- 172.21.15.2:0/3644510810 shutdown_connections 2024-01-15T17:58:34.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.778+0000 7f2373e00640 1 -- 172.21.15.2:0/3644510810 wait complete. 2024-01-15T17:58:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.780+0000 7f2373e00640 1 Processor -- start 2024-01-15T17:58:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.780+0000 7f2373e00640 1 -- start start 2024-01-15T17:58:34.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.781+0000 7f2373e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f236c0b3000 0x7f236c1f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.781+0000 7f2373e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c1445f0 0x7f236c141e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.781+0000 7f2373e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 0x7f236c13ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.781+0000 7f2373e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1426e0 con 0x7f236c15c2f0 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.781+0000 7f2373e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c1f3530 con 0x7f236c0b3000 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.781+0000 7f2373e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c158e90 con 0x7f236c1445f0 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 0x7f236c13ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 0x7f236c13ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60976/0 (socket says 172.21.15.2:60976) 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 -- 172.21.15.2:0/1596968429 learned_addr learned my addr 172.21.15.2:0/1596968429 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2370a00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c1445f0 0x7f236c141e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371400640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f236c0b3000 0x7f236c1f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 -- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c1445f0 msgr2=0x7f236c141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c1445f0 0x7f236c141e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 -- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f236c0b3000 msgr2=0x7f236c1f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f236c0b3000 0x7f236c1f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371e00640 1 -- 172.21.15.2:0/1596968429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2364009b30 con 0x7f236c15c2f0 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.782+0000 7f2371400640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f236c0b3000 0x7f236c1f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.783+0000 7f2371e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 0x7f236c13ddf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2364009e50 tx=0x7f2364004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.783+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236403e450 con 0x7f236c15c2f0 2024-01-15T17:58:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.783+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236403e5b0 con 0x7f236c15c2f0 2024-01-15T17:58:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.783+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236403e960 con 0x7f236c15c2f0 2024-01-15T17:58:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.783+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c159170 con 0x7f236c15c2f0 2024-01-15T17:58:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.783+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c159730 con 0x7f236c15c2f0 2024-01-15T17:58:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.785+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c0686f0 con 0x7f236c15c2f0 2024-01-15T17:58:34.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.786+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f236403eac0 con 0x7f236c15c2f0 2024-01-15T17:58:34.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.787+0000 7f2369600640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f234406b2c0 0x7f234406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:34.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.787+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f23640ac910 con 0x7f236c15c2f0 2024-01-15T17:58:34.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.788+0000 7f2371400640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f234406b2c0 0x7f234406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:34.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.789+0000 7f2371400640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f234406b2c0 0x7f234406d780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f235c007fd0 tx=0x7f235c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:34.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.792+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2364077660 con 0x7f236c15c2f0 2024-01-15T17:58:34.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.985+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f236c070a80 con 0x7f236c15c2f0 2024-01-15T17:58:34.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.988+0000 7f2369600640 1 -- 172.21.15.2:0/1596968429 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f236407ac80 con 0x7f236c15c2f0 2024-01-15T17:58:34.993 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:34.993 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f234406b2c0 msgr2=0x7f234406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f234406b2c0 0x7f234406d780 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f235c007fd0 tx=0x7f235c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 msgr2=0x7f236c13ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 0x7f236c13ddf0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2364009e50 tx=0x7f2364004a40 comp rx=0 tx=0).stop 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 shutdown_connections 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f234406b2c0 0x7f234406d780 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f236c15c2f0 0x7f236c13ddf0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f236c1445f0 0x7f236c141e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 --2- 172.21.15.2:0/1596968429 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f236c0b3000 0x7f236c1f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.991+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 >> 172.21.15.2:0/1596968429 conn(0x7f236c0663d0 msgr2=0x7f236c0ce460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.992+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 shutdown_connections 2024-01-15T17:58:34.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:34.992+0000 7f2373e00640 1 -- 172.21.15.2:0/1596968429 wait complete. 2024-01-15T17:58:35.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:35.014 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:35.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.235+0000 7f3f18a00640 1 Processor -- start 2024-01-15T17:58:35.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.235+0000 7f3f18a00640 1 -- start start 2024-01-15T17:58:35.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.237+0000 7f3f18a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f140b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.237+0000 7f3f18a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3f140ad8b0 0x7f3f1406ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.237+0000 7f3f18a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140b40f0 con 0x7f3f1406c930 2024-01-15T17:58:35.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.237+0000 7f3f18a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f140b46b0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.237+0000 7f3f18a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1406aa30 con 0x7f3f1415d320 2024-01-15T17:58:35.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.238+0000 7f3f13400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f140b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.238+0000 7f3f13e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3f140ad8b0 0x7f3f1406ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.238+0000 7f3f13400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f140b37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:40180/0 (socket says 172.21.15.2:40180) 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.238+0000 7f3f13400640 1 -- 172.21.15.2:0/2812532435 learned_addr learned my addr 172.21.15.2:0/2812532435 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.239+0000 7f3f13e00640 1 -- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 msgr2=0x7f3f140b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.239+0000 7f3f13e00640 1 --2- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f140b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.239+0000 7f3f13e00640 1 -- 172.21.15.2:0/2812532435 >> v1:172.21.15.137:6789/0 conn(0x7f3f1406c930 legacy=0x7f3f140ad1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.239+0000 7f3f13e00640 1 -- 172.21.15.2:0/2812532435 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140b43d0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.239+0000 7f3f13400640 1 --2- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f140b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:58:35.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.240+0000 7f3f13e00640 1 --2- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f140ad8b0 0x7f3f1406ae50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009d90 tx=0x7f3f0c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cab64099dbd7ef5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:35.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.240+0000 7f3f12a00640 1 -- 172.21.15.2:0/2812532435 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c02cc60 con 0x7f3f140ad8b0 2024-01-15T17:58:35.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.240+0000 7f3f12a00640 1 -- 172.21.15.2:0/2812532435 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c02cdc0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.240+0000 7f3f12a00640 1 -- 172.21.15.2:0/2812532435 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c034660 con 0x7f3f140ad8b0 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 -- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f140ad8b0 msgr2=0x7f3f1406ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f140ad8b0 0x7f3f1406ae50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c009d90 tx=0x7f3f0c02beb0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 -- 172.21.15.2:0/2812532435 shutdown_connections 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f140ad8b0 0x7f3f1406ae50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2812532435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f140b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 -- 172.21.15.2:0/2812532435 >> 172.21.15.2:0/2812532435 conn(0x7f3f140cf700 msgr2=0x7f3f140cfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 -- 172.21.15.2:0/2812532435 shutdown_connections 2024-01-15T17:58:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.241+0000 7f3f18a00640 1 -- 172.21.15.2:0/2812532435 wait complete. 2024-01-15T17:58:35.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.243+0000 7f3f18a00640 1 Processor -- start 2024-01-15T17:58:35.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.243+0000 7f3f18a00640 1 -- start start 2024-01-15T17:58:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.244+0000 7f3f18a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f1406c930 0x7f3f141f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.244+0000 7f3f18a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 0x7f3f1415f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.244+0000 7f3f18a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f1415fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.244+0000 7f3f18a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f1414d5f0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.244+0000 7f3f18a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f141f6f00 con 0x7f3f1406c930 2024-01-15T17:58:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.244+0000 7f3f18a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140c4c00 con 0x7f3f1415d320 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 0x7f3f1415f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 0x7f3f1415f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60990/0 (socket says 172.21.15.2:60990) 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 -- 172.21.15.2:0/2573811582 learned_addr learned my addr 172.21.15.2:0/2573811582 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f13e00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f1415fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f13400640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f1406c930 0x7f3f141f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 -- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 msgr2=0x7f3f1415fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f1415fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 -- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f1406c930 msgr2=0x7f3f141f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f1406c930 0x7f3f141f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 -- 172.21.15.2:0/2573811582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040094d0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.245+0000 7f3f0be00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 0x7f3f1415f690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c3f0 tx=0x7f3efc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.246+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc0052d0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.246+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3efc005450 con 0x7f3f140ad8b0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.246+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c009a70 con 0x7f3f140ad8b0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.246+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3efc003830 con 0x7f3f140ad8b0 2024-01-15T17:58:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.246+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f140c45b0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.247+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f14074770 con 0x7f3f140ad8b0 2024-01-15T17:58:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.248+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3efc003990 con 0x7f3f140ad8b0 2024-01-15T17:58:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.249+0000 7f3f0b400640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3eec06b2c0 0x7f3eec06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.249+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3efc089140 con 0x7f3f140ad8b0 2024-01-15T17:58:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.250+0000 7f3f13400640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3eec06b2c0 0x7f3eec06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.251+0000 7f3f13400640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3eec06b2c0 0x7f3eec06d780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f1414d930 tx=0x7f3f04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:35.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.254+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3efc053f20 con 0x7f3f140ad8b0 2024-01-15T17:58:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.446+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f140cffa0 con 0x7f3f140ad8b0 2024-01-15T17:58:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.447+0000 7f3f0b400640 1 -- 172.21.15.2:0/2573811582 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3efc057540 con 0x7f3f140ad8b0 2024-01-15T17:58:35.450 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3eec06b2c0 msgr2=0x7f3eec06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3eec06b2c0 0x7f3eec06d780 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f1414d930 tx=0x7f3f04008040 comp rx=0 tx=0).stop 2024-01-15T17:58:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 msgr2=0x7f3f1415f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 0x7f3f1415f690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f3efc00c3f0 tx=0x7f3efc00c8b0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 shutdown_connections 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3eec06b2c0 0x7f3eec06d780 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f1415d320 0x7f3f1415fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f140ad8b0 0x7f3f1415f690 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 --2- 172.21.15.2:0/2573811582 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f1406c930 0x7f3f141f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 >> 172.21.15.2:0/2573811582 conn(0x7f3f140cf700 msgr2=0x7f3f140b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.449+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 shutdown_connections 2024-01-15T17:58:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.450+0000 7f3f18a00640 1 -- 172.21.15.2:0/2573811582 wait complete. 2024-01-15T17:58:35.472 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:35.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.691+0000 7f8762600640 1 Processor -- start 2024-01-15T17:58:35.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.691+0000 7f8762600640 1 -- start start 2024-01-15T17:58:35.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.693+0000 7f8762600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f875c06d000 0x7f875c165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.693+0000 7f8762600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0ce230 con 0x7f875c144c00 2024-01-15T17:58:35.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.693+0000 7f8762600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f875c0ce510 con 0x7f875c15c450 2024-01-15T17:58:35.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.693+0000 7f8762600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f875c0ce7f0 con 0x7f875c06d000 2024-01-15T17:58:35.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.693+0000 7f875b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f875c144c00 0x7f875c0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49088/0 (socket says 172.21.15.2:49088) 2024-01-15T17:58:35.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.693+0000 7f875b400640 1 -- 172.21.15.2:0/1402963466 learned_addr learned my addr 172.21.15.2:0/1402963466 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:35.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875be00640 1 --2- 172.21.15.2:0/1402963466 >> v2:172.21.15.137:3300/0 conn(0x7f875c06d000 0x7f875c165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705208478 0 0) 0x7f875c0ce230 con 0x7f875c144c00 2024-01-15T17:58:35.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f873c003520 con 0x7f875c144c00 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2499208055 0 0) 0x7f873c003520 con 0x7f875c144c00 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 >> v2:172.21.15.137:3300/0 conn(0x7f875c06d000 msgr2=0x7f875c165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 --2- 172.21.15.2:0/1402963466 >> v2:172.21.15.137:3300/0 conn(0x7f875c06d000 0x7f875c165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 >> v1:172.21.15.137:6789/0 conn(0x7f875c15c450 legacy=0x7f875c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c0705f0 con 0x7f875c144c00 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f874c001670 con 0x7f875c144c00 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 >> v1:172.21.15.2:6789/0 conn(0x7f875c144c00 legacy=0x7f875c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.694+0000 7f875a000640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f873c003970 0x7f873c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875a000640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f873c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875a000640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f873c005180 0x7f873c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c0ce7f0 con 0x7f873c0047c0 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c00c020 con 0x7f873c003970 2024-01-15T17:58:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f873c00c1e0 con 0x7f873c005180 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875aa00640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f873c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875b400640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f873c003970 0x7f873c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875be00640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f873c005180 0x7f873c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875aa00640 1 -- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f873c005180 msgr2=0x7f873c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875aa00640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f873c005180 0x7f873c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875aa00640 1 -- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f873c003970 msgr2=0x7f873c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875aa00640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f873c003970 0x7f873c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875aa00640 1 -- 172.21.15.2:0/1402963466 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c003520 con 0x7f873c0047c0 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.695+0000 7f875be00640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f873c005180 0x7f873c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:35.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.696+0000 7f875be00640 1 -- 172.21.15.2:0/1402963466 reap_dead start 2024-01-15T17:58:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.697+0000 7f875aa00640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f873c004c40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8744001750 tx=0x7f873c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b676f3e59df8ed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.697+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87440052d0 con 0x7f873c0047c0 2024-01-15T17:58:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.697+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874402e050 con 0x7f873c0047c0 2024-01-15T17:58:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.697+0000 7f875a000640 1 -- 172.21.15.2:0/1402963466 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8744003830 con 0x7f873c0047c0 2024-01-15T17:58:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.697+0000 7f8762600640 1 -- 172.21.15.2:0/1402963466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 msgr2=0x7f873c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.697+0000 7f8762600640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f873c004c40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f8744001750 tx=0x7f873c0057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.698+0000 7f8762600640 1 -- 172.21.15.2:0/1402963466 shutdown_connections 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.698+0000 7f8762600640 1 --2- 172.21.15.2:0/1402963466 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f873c004c40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.698+0000 7f8762600640 1 -- 172.21.15.2:0/1402963466 >> 172.21.15.2:0/1402963466 conn(0x7f875c0b45f0 msgr2=0x7f875c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.698+0000 7f8762600640 1 -- 172.21.15.2:0/1402963466 shutdown_connections 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.698+0000 7f8762600640 1 -- 172.21.15.2:0/1402963466 wait complete. 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.700+0000 7f8762600640 1 Processor -- start 2024-01-15T17:58:35.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.700+0000 7f8762600640 1 -- start start 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f8762600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f875c1f5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f8762600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f875c153830 0x7f875c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f8762600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f875c153cb0 0x7f875c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f8762600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c12d6d0 con 0x7f873c0047c0 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f8762600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875c12d9b0 con 0x7f875c153830 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f8762600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875c12dc90 con 0x7f875c153cb0 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f875c1f5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f875c1f5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32782/0 (socket says 172.21.15.2:32782) 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875b400640 1 -- 172.21.15.2:0/951618726 learned_addr learned my addr 172.21.15.2:0/951618726 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875b400640 1 -- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f875c153cb0 msgr2=0x7f875c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875b400640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f875c153cb0 0x7f875c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875b400640 1 -- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f875c153830 msgr2=0x7f875c143bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.701+0000 7f875aa00640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f875c153830 0x7f875c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f875b400640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f875c153830 0x7f875c143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f875b400640 1 -- 172.21.15.2:0/951618726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875c12df70 con 0x7f873c0047c0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f875b400640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f875c1f5ae0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f874c00e700 tx=0x7f874c00ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c00c640 con 0x7f873c0047c0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874c00c7a0 con 0x7f873c0047c0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875c12e250 con 0x7f873c0047c0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c00cb50 con 0x7f873c0047c0 2024-01-15T17:58:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.702+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875c1f6020 con 0x7f873c0047c0 2024-01-15T17:58:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.704+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875c0686f0 con 0x7f873c0047c0 2024-01-15T17:58:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.705+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f874c00ccb0 con 0x7f873c0047c0 2024-01-15T17:58:35.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.706+0000 7f8753400640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f873406b2c0 0x7f873406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:35.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.706+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f874c089aa0 con 0x7f873c0047c0 2024-01-15T17:58:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.706+0000 7f875aa00640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f873406b2c0 0x7f873406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.708+0000 7f875aa00640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f873406b2c0 0x7f873406d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8744033000 tx=0x7f8744009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:35.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.710+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f874c054890 con 0x7f873c0047c0 2024-01-15T17:58:35.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.904+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f875c0cfbb0 con 0x7f873c0047c0 2024-01-15T17:58:35.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.906+0000 7f8753400640 1 -- 172.21.15.2:0/951618726 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f874c057eb0 con 0x7f873c0047c0 2024-01-15T17:58:35.910 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.908+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f873406b2c0 msgr2=0x7f873406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.908+0000 7f8762600640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f873406b2c0 0x7f873406d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8744033000 tx=0x7f8744009030 comp rx=0 tx=0).stop 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 msgr2=0x7f875c1f5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f875c1f5ae0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f874c00e700 tx=0x7f874c00ebc0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 shutdown_connections 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f873406b2c0 0x7f873406d780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f875c153cb0 0x7f875c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f875c153830 0x7f875c143bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 --2- 172.21.15.2:0/951618726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f873c0047c0 0x7f875c1f5ae0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 >> 172.21.15.2:0/951618726 conn(0x7f875c0b45f0 msgr2=0x7f875c15b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 shutdown_connections 2024-01-15T17:58:35.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:35.909+0000 7f8762600640 1 -- 172.21.15.2:0/951618726 wait complete. 2024-01-15T17:58:35.931 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.148+0000 7f9d05400640 1 Processor -- start 2024-01-15T17:58:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.148+0000 7f9d05400640 1 -- start start 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9d05400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d0006d000 0x7f9d000b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9d05400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d000702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9d05400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d000b25f0 con 0x7f9d0014afd0 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9d05400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d00070ff0 con 0x7f9d0006d000 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9d05400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d000b28d0 con 0x7f9d0014b3b0 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9cfde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d0006d000 0x7f9d000b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9cfe800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9d0014afd0 0x7f9d0006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49096/0 (socket says 172.21.15.2:49096) 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9cfde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9d0006d000 0x7f9d000b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32792/0 (socket says 172.21.15.2:32792) 2024-01-15T17:58:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9cfe800640 1 -- 172.21.15.2:0/2079151996 learned_addr learned my addr 172.21.15.2:0/2079151996 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.150+0000 7f9cff200640 1 --2- 172.21.15.2:0/2079151996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d000702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.151+0000 7f9cfde00640 1 -- 172.21.15.2:0/2079151996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 msgr2=0x7f9d000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.151+0000 7f9cfde00640 1 --2- 172.21.15.2:0/2079151996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d000702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.151+0000 7f9cfde00640 1 -- 172.21.15.2:0/2079151996 >> v1:172.21.15.2:6789/0 conn(0x7f9d0014afd0 legacy=0x7f9d0006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.151+0000 7f9cfde00640 1 -- 172.21.15.2:0/2079151996 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d000b2bb0 con 0x7f9d0006d000 2024-01-15T17:58:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.152+0000 7f9cfde00640 1 --2- 172.21.15.2:0/2079151996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d000b33b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80097a0 tx=0x7f9ce802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4e366534720887 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.152+0000 7f9cfd400640 1 -- 172.21.15.2:0/2079151996 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8031780 con 0x7f9d0006d000 2024-01-15T17:58:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.152+0000 7f9cfd400640 1 -- 172.21.15.2:0/2079151996 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce80318e0 con 0x7f9d0006d000 2024-01-15T17:58:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.152+0000 7f9cfd400640 1 -- 172.21.15.2:0/2079151996 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8031c70 con 0x7f9d0006d000 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 -- 172.21.15.2:0/2079151996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 msgr2=0x7f9d000b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 --2- 172.21.15.2:0/2079151996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d000b33b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9ce80097a0 tx=0x7f9ce802b820 comp rx=0 tx=0).stop 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 -- 172.21.15.2:0/2079151996 shutdown_connections 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 --2- 172.21.15.2:0/2079151996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d000702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 --2- 172.21.15.2:0/2079151996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d000b33b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 -- 172.21.15.2:0/2079151996 >> 172.21.15.2:0/2079151996 conn(0x7f9d000b45f0 msgr2=0x7f9d000b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 -- 172.21.15.2:0/2079151996 shutdown_connections 2024-01-15T17:58:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.153+0000 7f9d05400640 1 -- 172.21.15.2:0/2079151996 wait complete. 2024-01-15T17:58:36.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.155+0000 7f9d05400640 1 Processor -- start 2024-01-15T17:58:36.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.155+0000 7f9d05400640 1 -- start start 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9d05400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d00160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9d05400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d0014afd0 0x7f9d00133b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9d05400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d00134070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9d05400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00159020 con 0x7f9d0006d000 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9d05400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d001616d0 con 0x7f9d0014afd0 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9d05400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d001619b0 con 0x7f9d0014b3b0 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.156+0000 7f9cfe800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d00160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d00160c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32796/0 (socket says 172.21.15.2:32796) 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 -- 172.21.15.2:0/3260670232 learned_addr learned my addr 172.21.15.2:0/3260670232 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cff200640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d00134070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfde00640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d0014afd0 0x7f9d00133b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 -- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 msgr2=0x7f9d00134070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d00134070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 -- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d0014afd0 msgr2=0x7f9d00133b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d0014afd0 0x7f9d00133b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 -- 172.21.15.2:0/3260670232 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce8009400 con 0x7f9d0006d000 2024-01-15T17:58:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfde00640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d0014afd0 0x7f9d00133b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.157+0000 7f9cfe800640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d00160c10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000c4a0 tx=0x7f9cf000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.158+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0005ce0 con 0x7f9d0006d000 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.158+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf0005e40 con 0x7f9d0006d000 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.158+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0007270 con 0x7f9d0006d000 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.158+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d00161c90 con 0x7f9d0006d000 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.158+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d001410c0 con 0x7f9d0006d000 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.159+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d000686f0 con 0x7f9d0006d000 2024-01-15T17:58:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.160+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9cf0007490 con 0x7f9d0006d000 2024-01-15T17:58:36.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.161+0000 7f9cf6a00640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd806b2c0 0x7f9cd806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.161+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9cf00891b0 con 0x7f9d0006d000 2024-01-15T17:58:36.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.162+0000 7f9cfde00640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd806b2c0 0x7f9cd806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.163+0000 7f9cfde00640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd806b2c0 0x7f9cd806d780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8036000 tx=0x7f9ce803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:36.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.166+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cf0053fb0 con 0x7f9d0006d000 2024-01-15T17:58:36.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.367+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d00144e00 con 0x7f9d0006d000 2024-01-15T17:58:36.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.368+0000 7f9cf6a00640 1 -- 172.21.15.2:0/3260670232 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9cf00575d0 con 0x7f9d0006d000 2024-01-15T17:58:36.372 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd806b2c0 msgr2=0x7f9cd806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd806b2c0 0x7f9cd806d780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8036000 tx=0x7f9ce803d040 comp rx=0 tx=0).stop 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 msgr2=0x7f9d00160c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d00160c10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000c4a0 tx=0x7f9cf000c960 comp rx=0 tx=0).stop 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 shutdown_connections 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd806b2c0 0x7f9cd806d780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d0014b3b0 0x7f9d00134070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d0014afd0 0x7f9d00133b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 --2- 172.21.15.2:0/3260670232 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d0006d000 0x7f9d00160c10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 >> 172.21.15.2:0/3260670232 conn(0x7f9d000b45f0 msgr2=0x7f9d0006c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:36.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 shutdown_connections 2024-01-15T17:58:36.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.371+0000 7f9d05400640 1 -- 172.21.15.2:0/3260670232 wait complete. 2024-01-15T17:58:36.393 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T17:58:36.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.604+0000 7f409ba00640 1 Processor -- start 2024-01-15T17:58:36.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.605+0000 7f409ba00640 1 -- start start 2024-01-15T17:58:36.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.606+0000 7f409ba00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f409415e430 0x7f409406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.606+0000 7f409ba00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40941619b0 con 0x7f409406cdf0 2024-01-15T17:58:36.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f409ba00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4094161c90 con 0x7f4094143850 2024-01-15T17:58:36.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f409ba00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f409406d260 con 0x7f409415e430 2024-01-15T17:58:36.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f4099a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f409406cdf0 0x7f4094071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49100/0 (socket says 172.21.15.2:49100) 2024-01-15T17:58:36.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f4099a00640 1 -- 172.21.15.2:0/2301751147 learned_addr learned my addr 172.21.15.2:0/2301751147 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:36.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f408bc00640 1 --2- 172.21.15.2:0/2301751147 >> v2:172.21.15.2:3300/0 conn(0x7f409415e430 0x7f409406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f408bc00640 1 -- 172.21.15.2:0/2301751147 >> v1:172.21.15.137:6789/0 conn(0x7f4094143850 legacy=0x7f4094160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f408bc00640 1 -- 172.21.15.2:0/2301751147 >> v1:172.21.15.2:6789/0 conn(0x7f409406cdf0 legacy=0x7f4094071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f408bc00640 1 -- 172.21.15.2:0/2301751147 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409415dc30 con 0x7f409415e430 2024-01-15T17:58:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.607+0000 7f4093e00640 1 -- 172.21.15.2:0/2301751147 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395222151 0 0) 0x7f40941619b0 con 0x7f409406cdf0 2024-01-15T17:58:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.609+0000 7f408bc00640 1 --2- 172.21.15.2:0/2301751147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f409406c8b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f407c0097a0 tx=0x7f407c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62b1e1718e035e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:36.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.609+0000 7f4093e00640 1 -- 172.21.15.2:0/2301751147 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c02e070 con 0x7f409415e430 2024-01-15T17:58:36.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.609+0000 7f4093e00640 1 -- 172.21.15.2:0/2301751147 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c0315a0 con 0x7f409415e430 2024-01-15T17:58:36.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.609+0000 7f4093e00640 1 -- 172.21.15.2:0/2301751147 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c031930 con 0x7f409415e430 2024-01-15T17:58:36.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.609+0000 7f409ba00640 1 -- 172.21.15.2:0/2301751147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 msgr2=0x7f409406c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.609+0000 7f409ba00640 1 --2- 172.21.15.2:0/2301751147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f409406c8b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f407c0097a0 tx=0x7f407c02b840 comp rx=0 tx=0).stop 2024-01-15T17:58:36.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.610+0000 7f409ba00640 1 -- 172.21.15.2:0/2301751147 shutdown_connections 2024-01-15T17:58:36.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.610+0000 7f409ba00640 1 --2- 172.21.15.2:0/2301751147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f409406c8b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.610+0000 7f409ba00640 1 -- 172.21.15.2:0/2301751147 >> 172.21.15.2:0/2301751147 conn(0x7f40940b46d0 msgr2=0x7f40940b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:36.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.610+0000 7f409ba00640 1 -- 172.21.15.2:0/2301751147 shutdown_connections 2024-01-15T17:58:36.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.610+0000 7f409ba00640 1 -- 172.21.15.2:0/2301751147 wait complete. 2024-01-15T17:58:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.611+0000 7f409ba00640 1 Processor -- start 2024-01-15T17:58:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.612+0000 7f409ba00640 1 -- start start 2024-01-15T17:58:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.612+0000 7f409ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f409406cdf0 0x7f409415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f409ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4094143850 0x7f40940b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f409ba00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f40940b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f409ba00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40941321a0 con 0x7f409415e430 2024-01-15T17:58:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f409ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40940cb670 con 0x7f4094143850 2024-01-15T17:58:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f409ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40940cb910 con 0x7f409406cdf0 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f40940b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f40940b2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32808/0 (socket says 172.21.15.2:32808) 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 -- 172.21.15.2:0/1567799962 learned_addr learned my addr 172.21.15.2:0/1567799962 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099000640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f409406cdf0 0x7f409415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f408bc00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4094143850 0x7f40940b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 -- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f409406cdf0 msgr2=0x7f409415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f409406cdf0 0x7f409415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 -- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4094143850 msgr2=0x7f40940b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4094143850 0x7f40940b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.613+0000 7f4099a00640 1 -- 172.21.15.2:0/1567799962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c009400 con 0x7f409415e430 2024-01-15T17:58:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.614+0000 7f4099a00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f40940b2280 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c780 tx=0x7f408c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.614+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00ce90 con 0x7f409415e430 2024-01-15T17:58:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.614+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f408c00a640 con 0x7f409415e430 2024-01-15T17:58:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.614+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f408c00a9d0 con 0x7f409415e430 2024-01-15T17:58:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.614+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40940cbbf0 con 0x7f409415e430 2024-01-15T17:58:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.614+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40940cc1b0 con 0x7f409415e430 2024-01-15T17:58:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.616+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40940686f0 con 0x7f409415e430 2024-01-15T17:58:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.617+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f408c00ab30 con 0x7f409415e430 2024-01-15T17:58:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.617+0000 7f4091600640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f406c06b2c0 0x7f406c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.618+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f408c089170 con 0x7f409415e430 2024-01-15T17:58:36.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.618+0000 7f4099000640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f406c06b2c0 0x7f406c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.620+0000 7f4099000640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f406c06b2c0 0x7f406c06d780 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4084004ea0 tx=0x7f4084009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.623+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f408c053f60 con 0x7f409415e430 2024-01-15T17:58:36.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.830+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4094142c40 con 0x7f409415e430 2024-01-15T17:58:36.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.831+0000 7f4091600640 1 -- 172.21.15.2:0/1567799962 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f408c018080 con 0x7f409415e430 2024-01-15T17:58:36.835 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f406c06b2c0 msgr2=0x7f406c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f406c06b2c0 0x7f406c06d780 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4084004ea0 tx=0x7f4084009040 comp rx=0 tx=0).stop 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 msgr2=0x7f40940b2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f40940b2280 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f408c00c780 tx=0x7f408c00cc40 comp rx=0 tx=0).stop 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 shutdown_connections 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f406c06b2c0 0x7f406c06d780 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f409415e430 0x7f40940b2280 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4094143850 0x7f40940b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 --2- 172.21.15.2:0/1567799962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f409406cdf0 0x7f409415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:36.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 >> 172.21.15.2:0/1567799962 conn(0x7f40940b46d0 msgr2=0x7f40940b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:36.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 shutdown_connections 2024-01-15T17:58:36.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:36.834+0000 7f409ba00640 1 -- 172.21.15.2:0/1567799962 wait complete. 2024-01-15T17:58:36.857 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T17:58:36.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:36.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-15T17:58:36.956 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:36.956 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T17:58:36.956 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:58:36.956 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T17:58:37.007 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T17:58:37.007 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:58:37.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.228+0000 7f9a96c00640 1 Processor -- start 2024-01-15T17:58:37.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.229+0000 7f9a96c00640 1 -- start start 2024-01-15T17:58:37.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.230+0000 7f9a96c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9a9015c0e0 0x7f9a900cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.230+0000 7f9a96c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9a9015b7d0 0x7f9a9006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.230+0000 7f9a96c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.230+0000 7f9a96c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a9015cb00 con 0x7f9a9015b7d0 2024-01-15T17:58:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.230+0000 7f9a96c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a900ccec0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.230+0000 7f9a96c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a900cd1a0 con 0x7f9a9015c4c0 2024-01-15T17:58:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9a9015b7d0 0x7f9a9006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9a9015b7d0 0x7f9a9006c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32816/0 (socket says 172.21.15.2:32816) 2024-01-15T17:58:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 -- 172.21.15.2:0/2575525711 learned_addr learned my addr 172.21.15.2:0/2575525711 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:37.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8fe00640 1 --2- 172.21.15.2:0/2575525711 >> v2:172.21.15.137:3300/0 conn(0x7f9a9015c0e0 0x7f9a900cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a94c00640 1 --2- 172.21.15.2:0/2575525711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 -- 172.21.15.2:0/2575525711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 msgr2=0x7f9a9006c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:37.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 --2- 172.21.15.2:0/2575525711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9006c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 -- 172.21.15.2:0/2575525711 >> v2:172.21.15.137:3300/0 conn(0x7f9a9015c0e0 msgr2=0x7f9a900cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:37.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 --2- 172.21.15.2:0/2575525711 >> v2:172.21.15.137:3300/0 conn(0x7f9a9015c0e0 0x7f9a900cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.231+0000 7f9a8f400640 1 -- 172.21.15.2:0/2575525711 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a900cd480 con 0x7f9a9015b7d0 2024-01-15T17:58:37.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.233+0000 7f9a8f400640 1 --2- 172.21.15.2:0/2575525711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015b7d0 0x7f9a9006c470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9a900725f0 tx=0x7f9a7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56e221a2d4ce6fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.233+0000 7f9a8ea00640 1 -- 172.21.15.2:0/2575525711 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78031870 con 0x7f9a9015b7d0 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.233+0000 7f9a8ea00640 1 -- 172.21.15.2:0/2575525711 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a780319d0 con 0x7f9a9015b7d0 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.233+0000 7f9a8ea00640 1 -- 172.21.15.2:0/2575525711 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a78031d60 con 0x7f9a9015b7d0 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.233+0000 7f9a96c00640 1 -- 172.21.15.2:0/2575525711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015b7d0 msgr2=0x7f9a9006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.233+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2575525711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015b7d0 0x7f9a9006c470 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f9a900725f0 tx=0x7f9a7802b910 comp rx=0 tx=0).stop 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 -- 172.21.15.2:0/2575525711 shutdown_connections 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2575525711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9006c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2575525711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015b7d0 0x7f9a9006c470 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2575525711 >> v2:172.21.15.137:3300/0 conn(0x7f9a9015c0e0 0x7f9a900cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 -- 172.21.15.2:0/2575525711 >> 172.21.15.2:0/2575525711 conn(0x7f9a9006da20 msgr2=0x7f9a9006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 -- 172.21.15.2:0/2575525711 shutdown_connections 2024-01-15T17:58:37.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.234+0000 7f9a96c00640 1 -- 172.21.15.2:0/2575525711 wait complete. 2024-01-15T17:58:37.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.236+0000 7f9a96c00640 1 Processor -- start 2024-01-15T17:58:37.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.236+0000 7f9a96c00640 1 -- start start 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a96c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a9015b7d0 0x7f9a9013c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a96c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 0x7f9a90147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a96c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9014c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a96c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90148ec0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a96c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90148240 con 0x7f9a9015b7d0 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a96c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a90148520 con 0x7f9a9015c4c0 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 0x7f9a90147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 0x7f9a90147910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32832/0 (socket says 172.21.15.2:32832) 2024-01-15T17:58:37.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.237+0000 7f9a8f400640 1 -- 172.21.15.2:0/2369259711 learned_addr learned my addr 172.21.15.2:0/2369259711 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a94c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9014c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8fe00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a9015b7d0 0x7f9a9013c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8f400640 1 -- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 msgr2=0x7f9a9014c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8f400640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9014c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8f400640 1 -- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a9015b7d0 msgr2=0x7f9a9013c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8f400640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a9015b7d0 0x7f9a9013c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8f400640 1 -- 172.21.15.2:0/2369259711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a780094d0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a94c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9014c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.238+0000 7f9a8f400640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 0x7f9a90147910 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a7802b950 tx=0x7f9a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.239+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7803ac60 con 0x7f9a9015c0e0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.239+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7803adc0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.239+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a780445f0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.239+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a90148800 con 0x7f9a9015c0e0 2024-01-15T17:58:37.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.239+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a9013ce20 con 0x7f9a9015c0e0 2024-01-15T17:58:37.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.240+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a900686f0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.241+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9a7804d450 con 0x7f9a9015c0e0 2024-01-15T17:58:37.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.242+0000 7f9a87e00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a6806b2c0 0x7f9a6806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.242+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9a7802d040 con 0x7f9a9015c0e0 2024-01-15T17:58:37.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.242+0000 7f9a8fe00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a6806b2c0 0x7f9a6806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.244+0000 7f9a8fe00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a6806b2c0 0x7f9a6806d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9a80006fd0 tx=0x7f9a80008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:37.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.246+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a780770b0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.484+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a9006b260 con 0x7f9a9015c0e0 2024-01-15T17:58:37.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.485+0000 7f9a87e00640 1 -- 172.21.15.2:0/2369259711 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7f9a7807a6d0 con 0x7f9a9015c0e0 2024-01-15T17:58:37.489 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:37.490 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:58:37.490 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.488+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a6806b2c0 msgr2=0x7f9a6806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.488+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a6806b2c0 0x7f9a6806d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f9a80006fd0 tx=0x7f9a80008040 comp rx=0 tx=0).stop 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.488+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 msgr2=0x7f9a90147910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.488+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 0x7f9a90147910 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9a7802b950 tx=0x7f9a780046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 shutdown_connections 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a6806b2c0 0x7f9a6806d780 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a9015c4c0 0x7f9a9014c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a9015c0e0 0x7f9a90147910 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 --2- 172.21.15.2:0/2369259711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a9015b7d0 0x7f9a9013c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 >> 172.21.15.2:0/2369259711 conn(0x7f9a9006da20 msgr2=0x7f9a900b0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 shutdown_connections 2024-01-15T17:58:37.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.489+0000 7f9a96c00640 1 -- 172.21.15.2:0/2369259711 wait complete. 2024-01-15T17:58:37.514 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:37.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.732+0000 7fe6e5600640 1 Processor -- start 2024-01-15T17:58:37.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.732+0000 7fe6e5600640 1 -- start start 2024-01-15T17:58:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.734+0000 7fe6e5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 0x7fe6e015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.734+0000 7fe6e5600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e00716d0 con 0x7fe6e006cad0 2024-01-15T17:58:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.734+0000 7fe6e5600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e0071960 con 0x7fe6e0141880 2024-01-15T17:58:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.734+0000 7fe6e5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0071f20 con 0x7fe6e015ec60 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.734+0000 7fe6e4c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe6e006cad0 0x7fe6e006cf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49104/0 (socket says 172.21.15.2:49104) 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.734+0000 7fe6e4c00640 1 -- 172.21.15.2:0/2618929519 learned_addr learned my addr 172.21.15.2:0/2618929519 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6df400640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 0x7fe6e015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144089807 0 0) 0x7fe6e00716d0 con 0x7fe6e006cad0 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c0003520 con 0x7fe6e006cad0 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261663468 0 0) 0x7fe6c0003520 con 0x7fe6e006cad0 2024-01-15T17:58:37.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 msgr2=0x7fe6e015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 0x7fe6e015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 >> v1:172.21.15.137:6789/0 conn(0x7fe6e0141880 legacy=0x7fe6e0070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0071c40 con 0x7fe6e006cad0 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe6d80019f0 con 0x7fe6e006cad0 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 >> v1:172.21.15.2:6789/0 conn(0x7fe6e006cad0 legacy=0x7fe6e006cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 0x7fe6c00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6c0003f70 0x7fe6c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6c0004d00 0x7fe6c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e0071f20 con 0x7fe6c0004d00 2024-01-15T17:58:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c000bb70 con 0x7fe6c0003f70 2024-01-15T17:58:37.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.735+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c000bd30 con 0x7fe6e015ec60 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6c0004d00 0x7fe6c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 0x7fe6c00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6df400640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6c0003f70 0x7fe6c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6e4c00640 1 -- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 msgr2=0x7fe6c00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e015ec60 0x7fe6c00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6e4c00640 1 -- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6c0003f70 msgr2=0x7fe6c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6c0003f70 0x7fe6c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.736+0000 7fe6e4c00640 1 -- 172.21.15.2:0/2618929519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c0003520 con 0x7fe6c0004d00 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.737+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6c0004d00 0x7fe6c0005180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800a760 tx=0x7fe6c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256af3488b6c6ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8034630 con 0x7fe6c0004d00 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d802f070 con 0x7fe6c0004d00 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6dea00640 1 -- 172.21.15.2:0/2618929519 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8034a50 con 0x7fe6c0004d00 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e5600640 1 -- 172.21.15.2:0/2618929519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6c0004d00 msgr2=0x7fe6c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e5600640 1 --2- 172.21.15.2:0/2618929519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6c0004d00 0x7fe6c0005180 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800a760 tx=0x7fe6c00057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e4c00640 1 -- 172.21.15.2:0/2618929519 reap_dead start 2024-01-15T17:58:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e5600640 1 -- 172.21.15.2:0/2618929519 shutdown_connections 2024-01-15T17:58:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e5600640 1 -- 172.21.15.2:0/2618929519 >> 172.21.15.2:0/2618929519 conn(0x7fe6e00b0840 msgr2=0x7fe6e00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e5600640 1 -- 172.21.15.2:0/2618929519 shutdown_connections 2024-01-15T17:58:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.738+0000 7fe6e5600640 1 -- 172.21.15.2:0/2618929519 wait complete. 2024-01-15T17:58:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.740+0000 7fe6e5600640 1 Processor -- start 2024-01-15T17:58:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.740+0000 7fe6e5600640 1 -- start start 2024-01-15T17:58:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.741+0000 7fe6e5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6c0004d00 0x7fe6e01384a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.741+0000 7fe6e5600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e0140430 0x7fe6e0159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.741+0000 7fe6e5600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 0x7fe6e01f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.741+0000 7fe6e5600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e006c6b0 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.741+0000 7fe6e5600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e01560f0 con 0x7fe6e0140430 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.741+0000 7fe6e5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e01389e0 con 0x7fe6c0004d00 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 0x7fe6e01f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 0x7fe6e01f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32848/0 (socket says 172.21.15.2:32848) 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 -- 172.21.15.2:0/1499845220 learned_addr learned my addr 172.21.15.2:0/1499845220 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6c0004d00 0x7fe6e01384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6df400640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e0140430 0x7fe6e0159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 -- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6c0004d00 msgr2=0x7fe6e01384a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6c0004d00 0x7fe6e01384a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 -- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e0140430 msgr2=0x7fe6e0159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e0140430 0x7fe6e0159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.742+0000 7fe6e4c00640 1 -- 172.21.15.2:0/1499845220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e01554e0 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.743+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 0x7fe6e01f3440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8000f80 tx=0x7fe6d8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:37.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.743+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8034430 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.743+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8034590 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.743+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8034920 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.743+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e01557c0 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.743+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e0155d80 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.744+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e0074770 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.745+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe6d8031030 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.746+0000 7fe6d7e00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6b806b2c0 0x7fe6b806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.746+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe6d80abf60 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.747+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6b806b2c0 0x7fe6b806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:37.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.748+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6b806b2c0 0x7fe6b806d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8007fd0 tx=0x7fe6c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:37.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.751+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6d8076dd0 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.947+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6e0074770 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.949+0000 7fe6d7e00640 1 -- 172.21.15.2:0/1499845220 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fe6d807a3f0 con 0x7fe6e01f2fc0 2024-01-15T17:58:37.953 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:37.953 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:37.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.951+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6b806b2c0 msgr2=0x7fe6b806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.951+0000 7fe6e5600640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6b806b2c0 0x7fe6b806d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8007fd0 tx=0x7fe6c8009040 comp rx=0 tx=0).stop 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.951+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 msgr2=0x7fe6e01f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.951+0000 7fe6e5600640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 0x7fe6e01f3440 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8000f80 tx=0x7fe6d8001cb0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 shutdown_connections 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6b806b2c0 0x7fe6b806d780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e01f2fc0 0x7fe6e01f3440 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e0140430 0x7fe6e0159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 --2- 172.21.15.2:0/1499845220 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6c0004d00 0x7fe6e01384a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 >> 172.21.15.2:0/1499845220 conn(0x7fe6e00b0840 msgr2=0x7fe6e00cd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 shutdown_connections 2024-01-15T17:58:37.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:37.952+0000 7fe6e5600640 1 -- 172.21.15.2:0/1499845220 wait complete. 2024-01-15T17:58:37.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T17:57:29.871174+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T17:57:30.603017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:37.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:38.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.204+0000 7fcf32800640 1 Processor -- start 2024-01-15T17:58:38.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.204+0000 7fcf32800640 1 -- start start 2024-01-15T17:58:38.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.206+0000 7fcf32800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcf2c0702c0 0x7fcf2c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.206+0000 7fcf32800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c15dd70 con 0x7fcf2c06c8f0 2024-01-15T17:58:38.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.206+0000 7fcf32800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c0711b0 con 0x7fcf2c1439f0 2024-01-15T17:58:38.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.206+0000 7fcf32800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c143dd0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.206+0000 7fcf2be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcf2c06c8f0 0x7fcf2c06ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49120/0 (socket says 172.21.15.2:49120) 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf2be00640 1 -- 172.21.15.2:0/70127823 learned_addr learned my addr 172.21.15.2:0/70127823 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf22800640 1 --2- 172.21.15.2:0/70127823 >> v2:172.21.15.137:3300/0 conn(0x7fcf2c0702c0 0x7fcf2c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf2aa00640 1 -- 172.21.15.2:0/70127823 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973265687 0 0) 0x7fcf2c15dd70 con 0x7fcf2c06c8f0 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf2aa00640 1 -- 172.21.15.2:0/70127823 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf0c003520 con 0x7fcf2c06c8f0 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf22800640 1 -- 172.21.15.2:0/70127823 >> v1:172.21.15.137:6789/0 conn(0x7fcf2c1439f0 legacy=0x7fcf2c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf22800640 1 -- 172.21.15.2:0/70127823 >> v1:172.21.15.2:6789/0 conn(0x7fcf2c06c8f0 legacy=0x7fcf2c06ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.207+0000 7fcf22800640 1 -- 172.21.15.2:0/70127823 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c070910 con 0x7fcf2c0702c0 2024-01-15T17:58:38.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.209+0000 7fcf22800640 1 --2- 172.21.15.2:0/70127823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c0b3420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcf140097a0 tx=0x7fcf1402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36255ae3c44facbf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:38.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.209+0000 7fcf2aa00640 1 -- 172.21.15.2:0/70127823 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14030610 con 0x7fcf2c0702c0 2024-01-15T17:58:38.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.209+0000 7fcf2aa00640 1 -- 172.21.15.2:0/70127823 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf14030770 con 0x7fcf2c0702c0 2024-01-15T17:58:38.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.209+0000 7fcf2aa00640 1 -- 172.21.15.2:0/70127823 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14030b00 con 0x7fcf2c0702c0 2024-01-15T17:58:38.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 -- 172.21.15.2:0/70127823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c0702c0 msgr2=0x7fcf2c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 --2- 172.21.15.2:0/70127823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c0b3420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcf140097a0 tx=0x7fcf1402b960 comp rx=0 tx=0).stop 2024-01-15T17:58:38.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 -- 172.21.15.2:0/70127823 shutdown_connections 2024-01-15T17:58:38.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 --2- 172.21.15.2:0/70127823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c0b3420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 -- 172.21.15.2:0/70127823 >> 172.21.15.2:0/70127823 conn(0x7fcf2c0b45f0 msgr2=0x7fcf2c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:38.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 -- 172.21.15.2:0/70127823 shutdown_connections 2024-01-15T17:58:38.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.210+0000 7fcf32800640 1 -- 172.21.15.2:0/70127823 wait complete. 2024-01-15T17:58:38.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.212+0000 7fcf32800640 1 Processor -- start 2024-01-15T17:58:38.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.212+0000 7fcf32800640 1 -- start start 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf32800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c06c8f0 0x7fcf2c159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf32800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf32800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf2c1439f0 0x7fcf2c137ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf32800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c16a0a0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf32800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c1688f0 con 0x7fcf2c06c8f0 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf32800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0b1300 con 0x7fcf2c1439f0 2024-01-15T17:58:38.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf22800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf22800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c137580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32862/0 (socket says 172.21.15.2:32862) 2024-01-15T17:58:38.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.213+0000 7fcf22800640 1 -- 172.21.15.2:0/1630196355 learned_addr learned my addr 172.21.15.2:0/1630196355 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf2be00640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf2c1439f0 0x7fcf2c137ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf2b400640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c06c8f0 0x7fcf2c159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf22800640 1 -- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf2c1439f0 msgr2=0x7fcf2c137ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf22800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf2c1439f0 0x7fcf2c137ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf22800640 1 -- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c06c8f0 msgr2=0x7fcf2c159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf22800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c06c8f0 0x7fcf2c159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf22800640 1 -- 172.21.15.2:0/1630196355 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf14009400 con 0x7fcf2c0702c0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.214+0000 7fcf22800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c137580 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcf1402bf70 tx=0x7fcf14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.215+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1403e040 con 0x7fcf2c0702c0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.215+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf14039cb0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.215+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14044630 con 0x7fcf2c0702c0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.215+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c169dc0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.215+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c159000 con 0x7fcf2c0702c0 2024-01-15T17:58:38.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.216+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c158b60 con 0x7fcf2c0702c0 2024-01-15T17:58:38.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.217+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcf14043390 con 0x7fcf2c0702c0 2024-01-15T17:58:38.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.218+0000 7fcf23e00640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf0406b2f0 0x7fcf0406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.218+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcf140abe90 con 0x7fcf2c0702c0 2024-01-15T17:58:38.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.219+0000 7fcf2b400640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf0406b2f0 0x7fcf0406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.220+0000 7fcf2b400640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf0406b2f0 0x7fcf0406d7b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0012e0 tx=0x7fcf1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:38.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.223+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf14076be0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.417+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf2c0686f0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.418+0000 7fcf23e00640 1 -- 172.21.15.2:0/1630196355 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf140340a0 con 0x7fcf2c0702c0 2024-01-15T17:58:38.422 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:38.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf0406b2f0 msgr2=0x7fcf0406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf0406b2f0 0x7fcf0406d7b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c0012e0 tx=0x7fcf1c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 msgr2=0x7fcf2c137580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c137580 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcf1402bf70 tx=0x7fcf14004730 comp rx=0 tx=0).stop 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 shutdown_connections 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf0406b2f0 0x7fcf0406d7b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf2c1439f0 0x7fcf2c137ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf2c0702c0 0x7fcf2c137580 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 --2- 172.21.15.2:0/1630196355 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf2c06c8f0 0x7fcf2c159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 >> 172.21.15.2:0/1630196355 conn(0x7fcf2c0b45f0 msgr2=0x7fcf2c15b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.421+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 shutdown_connections 2024-01-15T17:58:38.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.422+0000 7fcf32800640 1 -- 172.21.15.2:0/1630196355 wait complete. 2024-01-15T17:58:38.446 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T17:58:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.653+0000 7faef1e00640 1 Processor -- start 2024-01-15T17:58:38.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.653+0000 7faef1e00640 1 -- start start 2024-01-15T17:58:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.654+0000 7faef1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 0x7faeec15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.654+0000 7faef1e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faeec0720d0 0x7faeec15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faef1e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec071090 con 0x7faeec1447d0 2024-01-15T17:58:38.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faef1e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeec071430 con 0x7faeec0720d0 2024-01-15T17:58:38.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faef1e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec144100 con 0x7faeec143d20 2024-01-15T17:58:38.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faef1400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7faeec1447d0 0x7faeec0cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49126/0 (socket says 172.21.15.2:49126) 2024-01-15T17:58:38.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faef1400640 1 -- 172.21.15.2:0/1023998183 learned_addr learned my addr 172.21.15.2:0/1023998183 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faeebe00640 1 --2- 172.21.15.2:0/1023998183 >> v2:172.21.15.137:3300/0 conn(0x7faeec0720d0 0x7faeec15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faef0a00640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 0x7faeec15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439452812 0 0) 0x7faeec071090 con 0x7faeec1447d0 2024-01-15T17:58:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.655+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc003520 con 0x7faeec1447d0 2024-01-15T17:58:38.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 556611278 0 0) 0x7faecc003520 con 0x7faeec1447d0 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 msgr2=0x7faeec15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 0x7faeec15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 >> v2:172.21.15.137:3300/0 conn(0x7faeec0720d0 msgr2=0x7faeec15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 --2- 172.21.15.2:0/1023998183 >> v2:172.21.15.137:3300/0 conn(0x7faeec0720d0 0x7faeec15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec06d160 con 0x7faeec1447d0 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7faee40019f0 con 0x7faeec1447d0 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 >> v1:172.21.15.2:6789/0 conn(0x7faeec1447d0 legacy=0x7faeec0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc003970 0x7faecc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 0x7faecc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faecc004d00 0x7faecc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec071430 con 0x7faecc004d00 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec144100 con 0x7faecc003970 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.656+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faecc00b980 con 0x7faeec143d20 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef1400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faecc004d00 0x7faecc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faeebe00640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 0x7faecc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef0a00640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc003970 0x7faecc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef1400640 1 -- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 msgr2=0x7faecc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:38.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef1400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec143d20 0x7faecc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef1400640 1 -- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc003970 msgr2=0x7faecc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:38.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef1400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc003970 0x7faecc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.657+0000 7faef1400640 1 -- 172.21.15.2:0/1023998183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc003520 con 0x7faecc004d00 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.658+0000 7faef1400640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faecc004d00 0x7faecc005180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7faee400a860 tx=0x7faecc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c0802c333c90e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.658+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee4034630 con 0x7faecc004d00 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.658+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee402f070 con 0x7faecc004d00 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.658+0000 7faeeb400640 1 -- 172.21.15.2:0/1023998183 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee40349f0 con 0x7faecc004d00 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1e00640 1 -- 172.21.15.2:0/1023998183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faecc004d00 msgr2=0x7faecc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1e00640 1 --2- 172.21.15.2:0/1023998183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faecc004d00 0x7faecc005180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7faee400a860 tx=0x7faecc0057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1400640 1 -- 172.21.15.2:0/1023998183 reap_dead start 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1e00640 1 -- 172.21.15.2:0/1023998183 shutdown_connections 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1e00640 1 -- 172.21.15.2:0/1023998183 >> 172.21.15.2:0/1023998183 conn(0x7faeec0b0840 msgr2=0x7faeec0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:38.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1e00640 1 -- 172.21.15.2:0/1023998183 shutdown_connections 2024-01-15T17:58:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.659+0000 7faef1e00640 1 -- 172.21.15.2:0/1023998183 wait complete. 2024-01-15T17:58:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.661+0000 7faef1e00640 1 Processor -- start 2024-01-15T17:58:38.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.661+0000 7faef1e00640 1 -- start start 2024-01-15T17:58:38.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faef1e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc004d00 0x7faeec0cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faef1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 0x7faeec139710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faef1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec0c3c70 0x7faeec0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faef1e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0c5620 con 0x7faeec139290 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faef1e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeec139c50 con 0x7faecc004d00 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faef1e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeec0c49a0 con 0x7faeec0c3c70 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faeebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 0x7faeec139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faeebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 0x7faeec139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32890/0 (socket says 172.21.15.2:32890) 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.662+0000 7faeebe00640 1 -- 172.21.15.2:0/3181782549 learned_addr learned my addr 172.21.15.2:0/3181782549 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faef0a00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc004d00 0x7faeec0cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faef1400640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec0c3c70 0x7faeec0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faeebe00640 1 -- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec0c3c70 msgr2=0x7faeec0c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faeebe00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec0c3c70 0x7faeec0c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faeebe00640 1 -- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc004d00 msgr2=0x7faeec0cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faeebe00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc004d00 0x7faeec0cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faeebe00640 1 -- 172.21.15.2:0/3181782549 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0c4c80 con 0x7faeec139290 2024-01-15T17:58:38.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faef0a00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc004d00 0x7faeec0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faef1400640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec0c3c70 0x7faeec0c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.663+0000 7faeebe00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 0x7faeec139710 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faed40116d0 tx=0x7faed4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.664+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400f9c0 con 0x7faeec139290 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.664+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400fb20 con 0x7faeec139290 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.664+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400fe90 con 0x7faeec139290 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.664+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeec0c4f60 con 0x7faeec139290 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.664+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeec0c4630 con 0x7faeec139290 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.665+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeec074770 con 0x7faeec139290 2024-01-15T17:58:38.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.666+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faed4015460 con 0x7faeec139290 2024-01-15T17:58:38.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.667+0000 7faee8c00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faec406b2f0 0x7faec406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:38.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.667+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faed408a1d0 con 0x7faeec139290 2024-01-15T17:58:38.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.667+0000 7faef0a00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faec406b2f0 0x7faec406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:38.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.668+0000 7faef0a00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faec406b2f0 0x7faec406d7b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faedc009980 tx=0x7faedc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:38.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.671+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faed4055060 con 0x7faeec139290 2024-01-15T17:58:38.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.869+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faeec0ad460 con 0x7faeec139290 2024-01-15T17:58:38.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.870+0000 7faee8c00640 1 -- 172.21.15.2:0/3181782549 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7faed4058680 con 0x7faeec139290 2024-01-15T17:58:38.874 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faec406b2f0 msgr2=0x7faec406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faec406b2f0 0x7faec406d7b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faedc009980 tx=0x7faedc008040 comp rx=0 tx=0).stop 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 msgr2=0x7faeec139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 0x7faeec139710 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7faed40116d0 tx=0x7faed4011b90 comp rx=0 tx=0).stop 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 shutdown_connections 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faec406b2f0 0x7faec406d7b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faeec0c3c70 0x7faeec0c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faeec139290 0x7faeec139710 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 --2- 172.21.15.2:0/3181782549 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faecc004d00 0x7faeec0cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 >> 172.21.15.2:0/3181782549 conn(0x7faeec0b0840 msgr2=0x7faeec0adc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 shutdown_connections 2024-01-15T17:58:38.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:38.873+0000 7faef1e00640 1 -- 172.21.15.2:0/3181782549 wait complete. 2024-01-15T17:58:38.894 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T17:58:39.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.113+0000 7f3533e00640 1 Processor -- start 2024-01-15T17:58:39.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.113+0000 7f3533e00640 1 -- start start 2024-01-15T17:58:39.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.115+0000 7f3533e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f353414b160 0x7f3534161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.115+0000 7f3533e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.115+0000 7f3533e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35341620d0 con 0x7f35340b2060 2024-01-15T17:58:39.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.115+0000 7f3533e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35340b35b0 con 0x7f353414b160 2024-01-15T17:58:39.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.115+0000 7f3533e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35340b1510 con 0x7f3534072100 2024-01-15T17:58:39.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.116+0000 7f3533400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f35340b2060 0x7f353406cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49130/0 (socket says 172.21.15.2:49130) 2024-01-15T17:58:39.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.116+0000 7f3533400640 1 -- 172.21.15.2:0/2225327797 learned_addr learned my addr 172.21.15.2:0/2225327797 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.116+0000 7f3529e00640 1 --2- 172.21.15.2:0/2225327797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.116+0000 7f3532a00640 1 --2- 172.21.15.2:0/2225327797 >> v2:172.21.15.137:3300/0 conn(0x7f353414b160 0x7f3534161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.116+0000 7f3532000640 1 -- 172.21.15.2:0/2225327797 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445513690 0 0) 0x7f35341620d0 con 0x7f35340b2060 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.116+0000 7f3532000640 1 -- 172.21.15.2:0/2225327797 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514003520 con 0x7f35340b2060 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.117+0000 7f3529e00640 1 -- 172.21.15.2:0/2225327797 >> v2:172.21.15.137:3300/0 conn(0x7f353414b160 msgr2=0x7f3534161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.117+0000 7f3532000640 1 -- 172.21.15.2:0/2225327797 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1245086739 0 0) 0x7f3514003520 con 0x7f35340b2060 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.117+0000 7f3529e00640 1 --2- 172.21.15.2:0/2225327797 >> v2:172.21.15.137:3300/0 conn(0x7f353414b160 0x7f3534161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.117+0000 7f3529e00640 1 -- 172.21.15.2:0/2225327797 >> v1:172.21.15.2:6789/0 conn(0x7f35340b2060 legacy=0x7f353406cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.117+0000 7f3529e00640 1 -- 172.21.15.2:0/2225327797 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340b32d0 con 0x7f3534072100 2024-01-15T17:58:39.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.117+0000 7f3532a00640 1 --2- 172.21.15.2:0/2225327797 >> v2:172.21.15.137:3300/0 conn(0x7f353414b160 0x7f3534161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:58:39.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.118+0000 7f3529e00640 1 --2- 172.21.15.2:0/2225327797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534070110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f351c009d90 tx=0x7f351c02bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baea4994eb6350fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3532000640 1 -- 172.21.15.2:0/2225327797 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c001670 con 0x7f3534072100 2024-01-15T17:58:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3532000640 1 -- 172.21.15.2:0/2225327797 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351c02c8a0 con 0x7f3534072100 2024-01-15T17:58:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3532000640 1 -- 172.21.15.2:0/2225327797 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351c02cc30 con 0x7f3534072100 2024-01-15T17:58:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3533e00640 1 -- 172.21.15.2:0/2225327797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 msgr2=0x7f3534070110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3533e00640 1 --2- 172.21.15.2:0/2225327797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534070110 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f351c009d90 tx=0x7f351c02bd70 comp rx=0 tx=0).stop 2024-01-15T17:58:39.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3533e00640 1 -- 172.21.15.2:0/2225327797 shutdown_connections 2024-01-15T17:58:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3533e00640 1 --2- 172.21.15.2:0/2225327797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534070110 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.119+0000 7f3533e00640 1 --2- 172.21.15.2:0/2225327797 >> v2:172.21.15.137:3300/0 conn(0x7f353414b160 0x7f3534161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.120+0000 7f3533e00640 1 -- 172.21.15.2:0/2225327797 >> 172.21.15.2:0/2225327797 conn(0x7f35340b0840 msgr2=0x7f35340b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.120+0000 7f3533e00640 1 -- 172.21.15.2:0/2225327797 shutdown_connections 2024-01-15T17:58:39.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.120+0000 7f3533e00640 1 -- 172.21.15.2:0/2225327797 wait complete. 2024-01-15T17:58:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.121+0000 7f3533e00640 1 Processor -- start 2024-01-15T17:58:39.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.122+0000 7f3533e00640 1 -- start start 2024-01-15T17:58:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534163920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35340b2060 0x7f3534143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 0x7f35341416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534140e20 con 0x7f353414b160 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3534140b40 con 0x7f35340b2060 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3534162e40 con 0x7f3534072100 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 0x7f35341416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 0x7f35341416b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32892/0 (socket says 172.21.15.2:32892) 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 -- 172.21.15.2:0/3715339886 learned_addr learned my addr 172.21.15.2:0/3715339886 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3532a00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534163920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3529e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35340b2060 0x7f3534143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 -- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 msgr2=0x7f3534163920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534163920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 -- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35340b2060 msgr2=0x7f3534143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.123+0000 7f3533400640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35340b2060 0x7f3534143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f3533400640 1 -- 172.21.15.2:0/3715339886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3524009400 con 0x7f353414b160 2024-01-15T17:58:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f3533400640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 0x7f35341416b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f352c00c750 tx=0x7f352c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c005ad0 con 0x7f353414b160 2024-01-15T17:58:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f352c005c30 con 0x7f353414b160 2024-01-15T17:58:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c009a70 con 0x7f353414b160 2024-01-15T17:58:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352c007070 con 0x7f353414b160 2024-01-15T17:58:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.124+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3534163400 con 0x7f353414b160 2024-01-15T17:58:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.125+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3534162ca0 con 0x7f353414b160 2024-01-15T17:58:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.126+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f352c0071d0 con 0x7f353414b160 2024-01-15T17:58:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.128+0000 7f352be00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350c06b2f0 0x7f350c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.128+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f352c089900 con 0x7f353414b160 2024-01-15T17:58:39.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.128+0000 7f3532a00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350c06b2f0 0x7f350c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.130+0000 7f3532a00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350c06b2f0 0x7f350c06d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3534143290 tx=0x7f3524008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.132+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3534162ca0 con 0x7f353414b160 2024-01-15T17:58:39.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.325+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3534162ca0 con 0x7f353414b160 2024-01-15T17:58:39.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.326+0000 7f352be00640 1 -- 172.21.15.2:0/3715339886 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3534162ca0 con 0x7f353414b160 2024-01-15T17:58:39.331 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350c06b2f0 msgr2=0x7f350c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350c06b2f0 0x7f350c06d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3534143290 tx=0x7f3524008040 comp rx=0 tx=0).stop 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 msgr2=0x7f35341416b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 0x7f35341416b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f352c00c750 tx=0x7f352c00cc10 comp rx=0 tx=0).stop 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 shutdown_connections 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350c06b2f0 0x7f350c06d7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f353414b160 0x7f35341416b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35340b2060 0x7f3534143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 --2- 172.21.15.2:0/3715339886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3534072100 0x7f3534163920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 >> 172.21.15.2:0/3715339886 conn(0x7f35340b0840 msgr2=0x7f35340af7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 shutdown_connections 2024-01-15T17:58:39.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.329+0000 7f3533e00640 1 -- 172.21.15.2:0/3715339886 wait complete. 2024-01-15T17:58:39.353 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:58:39.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.570+0000 7fa63fa00640 1 Processor -- start 2024-01-15T17:58:39.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.571+0000 7fa63fa00640 1 -- start start 2024-01-15T17:58:39.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.572+0000 7fa63fa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa638144bf0 0x7fa63815f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.572+0000 7fa63fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6380cf9c0 0x7fa6380b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.572+0000 7fa63fa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa6380b5460 0x7fa63806e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.572+0000 7fa63fa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6380d0a90 con 0x7fa6380b5460 2024-01-15T17:58:39.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.572+0000 7fa63fa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa638144fd0 con 0x7fa638144bf0 2024-01-15T17:58:39.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.572+0000 7fa63fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380d01f0 con 0x7fa6380cf9c0 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa6380b5460 0x7fa63806e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa6380b5460 0x7fa63806e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32900/0 (socket says 172.21.15.2:32900) 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 -- 172.21.15.2:0/3356813270 learned_addr learned my addr 172.21.15.2:0/3356813270 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63d000640 1 --2- 172.21.15.2:0/3356813270 >> v2:172.21.15.137:3300/0 conn(0x7fa638144bf0 0x7fa63815f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa637e00640 1 --2- 172.21.15.2:0/3356813270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6380cf9c0 0x7fa6380b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 -- 172.21.15.2:0/3356813270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6380cf9c0 msgr2=0x7fa6380b4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 --2- 172.21.15.2:0/3356813270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6380cf9c0 0x7fa6380b4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 -- 172.21.15.2:0/3356813270 >> v2:172.21.15.137:3300/0 conn(0x7fa638144bf0 msgr2=0x7fa63815f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 --2- 172.21.15.2:0/3356813270 >> v2:172.21.15.137:3300/0 conn(0x7fa638144bf0 0x7fa63815f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.573+0000 7fa63da00640 1 -- 172.21.15.2:0/3356813270 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380d04d0 con 0x7fa6380b5460 2024-01-15T17:58:39.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.575+0000 7fa63da00640 1 --2- 172.21.15.2:0/3356813270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63806e850 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa638067170 tx=0x7fa63002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1555e2709db35d8e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:39.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.575+0000 7fa637400640 1 -- 172.21.15.2:0/3356813270 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63002ca70 con 0x7fa6380b5460 2024-01-15T17:58:39.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.575+0000 7fa637400640 1 -- 172.21.15.2:0/3356813270 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63002cbd0 con 0x7fa6380b5460 2024-01-15T17:58:39.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.575+0000 7fa637400640 1 -- 172.21.15.2:0/3356813270 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6300343f0 con 0x7fa6380b5460 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 -- 172.21.15.2:0/3356813270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 msgr2=0x7fa63806e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 --2- 172.21.15.2:0/3356813270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63806e850 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa638067170 tx=0x7fa63002bed0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 -- 172.21.15.2:0/3356813270 shutdown_connections 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 --2- 172.21.15.2:0/3356813270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63806e850 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 --2- 172.21.15.2:0/3356813270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6380cf9c0 0x7fa6380b4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 --2- 172.21.15.2:0/3356813270 >> v2:172.21.15.137:3300/0 conn(0x7fa638144bf0 0x7fa63815f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 -- 172.21.15.2:0/3356813270 >> 172.21.15.2:0/3356813270 conn(0x7fa6380b2c10 msgr2=0x7fa6380b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 -- 172.21.15.2:0/3356813270 shutdown_connections 2024-01-15T17:58:39.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.576+0000 7fa63fa00640 1 -- 172.21.15.2:0/3356813270 wait complete. 2024-01-15T17:58:39.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.577+0000 7fa63fa00640 1 Processor -- start 2024-01-15T17:58:39.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.578+0000 7fa63fa00640 1 -- start start 2024-01-15T17:58:39.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63fa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63815aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6380cf9c0 0x7fa6380cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa638144bf0 0x7fa6380cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63fa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380cb8f0 con 0x7fa6380b5460 2024-01-15T17:58:39.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6380cbbd0 con 0x7fa6380cf9c0 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa63815b820 con 0x7fa638144bf0 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63815aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63815aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32906/0 (socket says 172.21.15.2:32906) 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 -- 172.21.15.2:0/1368031780 learned_addr learned my addr 172.21.15.2:0/1368031780 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa637e00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6380cf9c0 0x7fa6380cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63da00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa638144bf0 0x7fa6380cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 -- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa638144bf0 msgr2=0x7fa6380cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa638144bf0 0x7fa6380cc620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 -- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6380cf9c0 msgr2=0x7fa6380cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6380cf9c0 0x7fa6380cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa63d000640 1 -- 172.21.15.2:0/1368031780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630009b30 con 0x7fa6380b5460 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.579+0000 7fa637e00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6380cf9c0 0x7fa6380cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:39.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.580+0000 7fa63d000640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63815aa90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa62800d5f0 tx=0x7fa62800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:39.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.580+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6280125f0 con 0x7fa6380b5460 2024-01-15T17:58:39.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.580+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa628012750 con 0x7fa6380b5460 2024-01-15T17:58:39.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.580+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa63815bb00 con 0x7fa6380b5460 2024-01-15T17:58:39.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.580+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628012b00 con 0x7fa6380b5460 2024-01-15T17:58:39.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.580+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63815afd0 con 0x7fa6380b5460 2024-01-15T17:58:39.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.581+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6380686f0 con 0x7fa6380b5460 2024-01-15T17:58:39.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.582+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa628012c60 con 0x7fa6380b5460 2024-01-15T17:58:39.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.583+0000 7fa634c00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa61006b2c0 0x7fa61006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:39.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.583+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa62808a170 con 0x7fa6380b5460 2024-01-15T17:58:39.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.584+0000 7fa637e00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa61006b2c0 0x7fa61006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:39.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.585+0000 7fa637e00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa61006b2c0 0x7fa61006d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa620006fd0 tx=0x7fa620008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:39.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.587+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa628054ed0 con 0x7fa6380b5460 2024-01-15T17:58:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.819+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6380cd3c0 con 0x7fa6380b5460 2024-01-15T17:58:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.820+0000 7fa634c00640 1 -- 172.21.15.2:0/1368031780 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fa6280584f0 con 0x7fa6380b5460 2024-01-15T17:58:39.824 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:39.824 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:58:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:58:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.822+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa61006b2c0 msgr2=0x7fa61006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.822+0000 7fa63fa00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa61006b2c0 0x7fa61006d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa620006fd0 tx=0x7fa620008040 comp rx=0 tx=0).stop 2024-01-15T17:58:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.822+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 msgr2=0x7fa63815aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.822+0000 7fa63fa00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63815aa90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa62800d5f0 tx=0x7fa62800dab0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.822+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 shutdown_connections 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa61006b2c0 0x7fa61006d780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa638144bf0 0x7fa6380cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa6380cf9c0 0x7fa6380cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 --2- 172.21.15.2:0/1368031780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6380b5460 0x7fa63815aa90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 >> 172.21.15.2:0/1368031780 conn(0x7fa6380b2c10 msgr2=0x7fa638141790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 shutdown_connections 2024-01-15T17:58:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:39.823+0000 7fa63fa00640 1 -- 172.21.15.2:0/1368031780 wait complete. 2024-01-15T17:58:39.846 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:40.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.065+0000 7f2c26000640 1 Processor -- start 2024-01-15T17:58:40.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.065+0000 7f2c26000640 1 -- start start 2024-01-15T17:58:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.067+0000 7f2c26000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2c2015ec60 0x7f2c2015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.067+0000 7f2c26000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c200716d0 con 0x7f2c20141880 2024-01-15T17:58:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.067+0000 7f2c26000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c20071960 con 0x7f2c2006cad0 2024-01-15T17:58:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.067+0000 7f2c26000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c20071f20 con 0x7f2c2015ec60 2024-01-15T17:58:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.067+0000 7f2c24c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2c20141880 0x7f2c20070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49138/0 (socket says 172.21.15.2:49138) 2024-01-15T17:58:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.067+0000 7f2c24c00640 1 -- 172.21.15.2:0/3904001017 learned_addr learned my addr 172.21.15.2:0/3904001017 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:40.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c17e00640 1 --2- 172.21.15.2:0/3904001017 >> v2:172.21.15.137:3300/0 conn(0x7f2c2015ec60 0x7f2c2015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444622736 0 0) 0x7f2c200716d0 con 0x7f2c20141880 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c00003520 con 0x7f2c20141880 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1821682525 0 0) 0x7f2c00003520 con 0x7f2c20141880 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 >> v2:172.21.15.137:3300/0 conn(0x7f2c2015ec60 msgr2=0x7f2c2015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 --2- 172.21.15.2:0/3904001017 >> v2:172.21.15.137:3300/0 conn(0x7f2c2015ec60 0x7f2c2015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 >> v1:172.21.15.137:6789/0 conn(0x7f2c2006cad0 legacy=0x7f2c2006cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c20071c40 con 0x7f2c20141880 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2c10001670 con 0x7f2c20141880 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 >> v1:172.21.15.2:6789/0 conn(0x7f2c20141880 legacy=0x7f2c20070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c000047c0 0x7f2c00004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.068+0000 7f2c1fe00640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c00005180 0x7f2c0000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20071f20 con 0x7f2c00005180 2024-01-15T17:58:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0000c120 con 0x7f2c00003970 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0000c2e0 con 0x7f2c000047c0 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c25600640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c00005180 0x7f2c0000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c17e00640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c000047c0 0x7f2c00004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c24c00640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c25600640 1 -- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c000047c0 msgr2=0x7f2c00004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c25600640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c000047c0 0x7f2c00004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c25600640 1 -- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c00003970 msgr2=0x7f2c00003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c25600640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c25600640 1 -- 172.21.15.2:0/3904001017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c00003520 con 0x7f2c00005180 2024-01-15T17:58:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.069+0000 7f2c24c00640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c00003970 0x7f2c00003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:40.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.070+0000 7f2c25600640 1 -- 172.21.15.2:0/3904001017 reap_dead start 2024-01-15T17:58:40.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.071+0000 7f2c25600640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c00005180 0x7f2c0000b980 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2c18001e50 tx=0x7f2c000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9bd24311a34e896 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.071+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18003b70 con 0x7f2c00005180 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.071+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c18003d10 con 0x7f2c00005180 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.071+0000 7f2c1fe00640 1 -- 172.21.15.2:0/3904001017 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18005310 con 0x7f2c00005180 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.071+0000 7f2c26000640 1 -- 172.21.15.2:0/3904001017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c00005180 msgr2=0x7f2c0000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.071+0000 7f2c26000640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c00005180 0x7f2c0000b980 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f2c18001e50 tx=0x7f2c000057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.072+0000 7f2c26000640 1 -- 172.21.15.2:0/3904001017 shutdown_connections 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.072+0000 7f2c26000640 1 --2- 172.21.15.2:0/3904001017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c00005180 0x7f2c0000b980 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.072+0000 7f2c26000640 1 -- 172.21.15.2:0/3904001017 >> 172.21.15.2:0/3904001017 conn(0x7f2c200b0840 msgr2=0x7f2c200b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.072+0000 7f2c26000640 1 -- 172.21.15.2:0/3904001017 shutdown_connections 2024-01-15T17:58:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.072+0000 7f2c26000640 1 -- 172.21.15.2:0/3904001017 wait complete. 2024-01-15T17:58:40.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.074+0000 7f2c26000640 1 Processor -- start 2024-01-15T17:58:40.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.074+0000 7f2c26000640 1 -- start start 2024-01-15T17:58:40.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c26000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c00005180 0x7f2c20136520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c26000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c200b26f0 0x7f2c2013fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c26000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 0x7f2c2015a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c26000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2006c6b0 con 0x7f2c20140400 2024-01-15T17:58:40.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c26000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20140890 con 0x7f2c200b26f0 2024-01-15T17:58:40.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c26000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c20136a60 con 0x7f2c00005180 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c25600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 0x7f2c2015a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c25600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 0x7f2c2015a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32934/0 (socket says 172.21.15.2:32934) 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c25600640 1 -- 172.21.15.2:0/3696995936 learned_addr learned my addr 172.21.15.2:0/3696995936 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.075+0000 7f2c24c00640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c00005180 0x7f2c20136520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c17e00640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c200b26f0 0x7f2c2013fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c25600640 1 -- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c00005180 msgr2=0x7f2c20136520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c25600640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c00005180 0x7f2c20136520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c25600640 1 -- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c200b26f0 msgr2=0x7f2c2013fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c25600640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c200b26f0 0x7f2c2013fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c25600640 1 -- 172.21.15.2:0/3696995936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2015b0d0 con 0x7f2c20140400 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c17e00640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c200b26f0 0x7f2c2013fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c25600640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 0x7f2c2015a3a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2c18001f40 tx=0x7f2c180014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c18006d50 con 0x7f2c20140400 2024-01-15T17:58:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c18007030 con 0x7f2c20140400 2024-01-15T17:58:40.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c180073a0 con 0x7f2c20140400 2024-01-15T17:58:40.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.076+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c200b1930 con 0x7f2c20140400 2024-01-15T17:58:40.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.077+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c200b1ef0 con 0x7f2c20140400 2024-01-15T17:58:40.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.078+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c20074770 con 0x7f2c20140400 2024-01-15T17:58:40.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.079+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2c18007500 con 0x7f2c20140400 2024-01-15T17:58:40.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.079+0000 7f2c1d600640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bf806b2c0 0x7f2bf806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.080+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2c180ab840 con 0x7f2c20140400 2024-01-15T17:58:40.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.080+0000 7f2c24c00640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bf806b2c0 0x7f2bf806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.081+0000 7f2c24c00640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bf806b2c0 0x7f2bf806d780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c10001eb0 tx=0x7f2c10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:40.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.084+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c180766a0 con 0x7f2c20140400 2024-01-15T17:58:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.278+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2c2006e1c0 con 0x7f2c20140400 2024-01-15T17:58:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.280+0000 7f2c1d600640 1 -- 172.21.15.2:0/3696995936 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f2c18079cc0 con 0x7f2c20140400 2024-01-15T17:58:40.285 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:40.285 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bf806b2c0 msgr2=0x7f2bf806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bf806b2c0 0x7f2bf806d780 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2c10001eb0 tx=0x7f2c10009040 comp rx=0 tx=0).stop 2024-01-15T17:58:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 msgr2=0x7f2c2015a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 0x7f2c2015a3a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f2c18001f40 tx=0x7f2c180014a0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 shutdown_connections 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2bf806b2c0 0x7f2bf806d780 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c20140400 0x7f2c2015a3a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c200b26f0 0x7f2c2013fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 --2- 172.21.15.2:0/3696995936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c00005180 0x7f2c20136520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 >> 172.21.15.2:0/3696995936 conn(0x7f2c200b0840 msgr2=0x7f2c200cc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.283+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 shutdown_connections 2024-01-15T17:58:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.284+0000 7f2c26000640 1 -- 172.21.15.2:0/3696995936 wait complete. 2024-01-15T17:58:40.305 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-15T17:58:40.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:58:40.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.527+0000 7fbf07200640 1 Processor -- start 2024-01-15T17:58:40.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.527+0000 7fbf07200640 1 -- start start 2024-01-15T17:58:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.528+0000 7fbf07200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbf0015d5d0 0x7fbf000d2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.528+0000 7fbf07200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbf000b5120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.528+0000 7fbf07200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000b5660 con 0x7fbf000b8ed0 2024-01-15T17:58:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.528+0000 7fbf07200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf000d1980 con 0x7fbf0015d5d0 2024-01-15T17:58:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.528+0000 7fbf07200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000d2780 con 0x7fbf000b4cd0 2024-01-15T17:58:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbef7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbf000b8ed0 0x7fbf000d1270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49150/0 (socket says 172.21.15.2:49150) 2024-01-15T17:58:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbef7e00640 1 -- 172.21.15.2:0/4083100909 learned_addr learned my addr 172.21.15.2:0/4083100909 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbeffe00640 1 --2- 172.21.15.2:0/4083100909 >> v2:172.21.15.137:3300/0 conn(0x7fbf0015d5d0 0x7fbf000d2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbf05200640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbf000b5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783783598 0 0) 0x7fbee8001110 con 0x7fbf000b8ed0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee0003520 con 0x7fbf000b8ed0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289936884 0 0) 0x7fbee0003520 con 0x7fbf000b8ed0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.529+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 msgr2=0x7fbf000b5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbf000b5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 >> v2:172.21.15.137:3300/0 conn(0x7fbf0015d5d0 msgr2=0x7fbf000d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 --2- 172.21.15.2:0/4083100909 >> v2:172.21.15.137:3300/0 conn(0x7fbf0015d5d0 0x7fbf000d2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000d1b70 con 0x7fbf000b8ed0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fbee8001800 con 0x7fbf000b8ed0 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 >> v1:172.21.15.2:6789/0 conn(0x7fbf000b8ed0 legacy=0x7fbf000d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbee0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbee0003ef0 0x7fbee00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0004d00 0x7fbee0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000d2780 con 0x7fbee0003ef0 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee000b980 con 0x7fbee0004d00 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee000baf0 con 0x7fbf000b4cd0 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbef7e00640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbee0003ef0 0x7fbee00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbeffe00640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbee0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.530+0000 7fbef7e00640 1 -- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 msgr2=0x7fbee0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.531+0000 7fbf05200640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0004d00 0x7fbee0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.531+0000 7fbef7e00640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbee0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.531+0000 7fbef7e00640 1 -- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0004d00 msgr2=0x7fbee0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.531+0000 7fbef7e00640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0004d00 0x7fbee0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.531+0000 7fbef7e00640 1 -- 172.21.15.2:0/4083100909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee0003520 con 0x7fbee0003ef0 2024-01-15T17:58:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.531+0000 7fbeffe00640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000b4cd0 0x7fbee0003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:40.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.532+0000 7fbef7e00640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbee0003ef0 0x7fbee00047c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c210 tx=0x7fbee00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5144ff9dccc117a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:40.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.532+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee802cae0 con 0x7fbee0003ef0 2024-01-15T17:58:40.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.532+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee802cc60 con 0x7fbee0003ef0 2024-01-15T17:58:40.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.532+0000 7fbeff400640 1 -- 172.21.15.2:0/4083100909 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8005b10 con 0x7fbee0003ef0 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf07200640 1 -- 172.21.15.2:0/4083100909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbee0003ef0 msgr2=0x7fbee00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf07200640 1 --2- 172.21.15.2:0/4083100909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbee0003ef0 0x7fbee00047c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbee800c210 tx=0x7fbee00057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf05200640 1 -- 172.21.15.2:0/4083100909 reap_dead start 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf07200640 1 -- 172.21.15.2:0/4083100909 shutdown_connections 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf07200640 1 -- 172.21.15.2:0/4083100909 >> 172.21.15.2:0/4083100909 conn(0x7fbf00072210 msgr2=0x7fbf000cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf07200640 1 -- 172.21.15.2:0/4083100909 shutdown_connections 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.533+0000 7fbf07200640 1 -- 172.21.15.2:0/4083100909 wait complete. 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.535+0000 7fbf07200640 1 Processor -- start 2024-01-15T17:58:40.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.535+0000 7fbf07200640 1 -- start start 2024-01-15T17:58:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf07200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0003ef0 0x7fbf00150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf07200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000c8ed0 0x7fbf00151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf07200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 0x7fbf00153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf07200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000c9350 con 0x7fbf00153250 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf07200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf00151d40 con 0x7fbee0003ef0 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf07200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf001514e0 con 0x7fbf000c8ed0 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 0x7fbf00153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 0x7fbf00153700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32962/0 (socket says 172.21.15.2:32962) 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 -- 172.21.15.2:0/3983443594 learned_addr learned my addr 172.21.15.2:0/3983443594 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbeffe00640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0003ef0 0x7fbf00150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbef7e00640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000c8ed0 0x7fbf00151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 -- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000c8ed0 msgr2=0x7fbf00151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000c8ed0 0x7fbf00151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 -- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0003ef0 msgr2=0x7fbf00150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.536+0000 7fbf05200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0003ef0 0x7fbf00150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbf05200640 1 -- 172.21.15.2:0/3983443594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf001600d0 con 0x7fbf00153250 2024-01-15T17:58:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbf05200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 0x7fbf00153700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbf00160eb0 tx=0x7fbef800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:40.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8017400 con 0x7fbf00153250 2024-01-15T17:58:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8011040 con 0x7fbf00153250 2024-01-15T17:58:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80177c0 con 0x7fbf00153250 2024-01-15T17:58:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0015f7a0 con 0x7fbf00153250 2024-01-15T17:58:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.537+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0015fd60 con 0x7fbf00153250 2024-01-15T17:58:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.539+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf00160260 con 0x7fbf00153250 2024-01-15T17:58:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.539+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbef8017920 con 0x7fbf00153250 2024-01-15T17:58:40.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.541+0000 7fbefcc00640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbed806b2f0 0x7fbed806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:40.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.541+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbef8089fe0 con 0x7fbf00153250 2024-01-15T17:58:40.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.541+0000 7fbeffe00640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbed806b2f0 0x7fbed806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:40.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.543+0000 7fbeffe00640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbed806b2f0 0x7fbed806d7b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009b20 tx=0x7fbef0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:40.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.545+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbef8054e00 con 0x7fbf00153250 2024-01-15T17:58:40.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.778+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf000686f0 con 0x7fbf00153250 2024-01-15T17:58:40.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.780+0000 7fbefcc00640 1 -- 172.21.15.2:0/3983443594 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fbef801c090 con 0x7fbf00153250 2024-01-15T17:58:40.784 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:40.784 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:58:40.784 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:58:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.782+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbed806b2f0 msgr2=0x7fbed806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.782+0000 7fbf07200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbed806b2f0 0x7fbed806d7b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009b20 tx=0x7fbef0008040 comp rx=0 tx=0).stop 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.782+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 msgr2=0x7fbf00153700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.782+0000 7fbf07200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 0x7fbf00153700 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbf00160eb0 tx=0x7fbef800f4a0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 shutdown_connections 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbed806b2f0 0x7fbed806d7b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbf00153250 0x7fbf00153700 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbf000c8ed0 0x7fbf00151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 --2- 172.21.15.2:0/3983443594 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbee0003ef0 0x7fbf00150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 >> 172.21.15.2:0/3983443594 conn(0x7fbf00072210 msgr2=0x7fbf0015a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 shutdown_connections 2024-01-15T17:58:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:40.783+0000 7fbf07200640 1 -- 172.21.15.2:0/3983443594 wait complete. 2024-01-15T17:58:40.804 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:58:41.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.016+0000 7fc90b000640 1 Processor -- start 2024-01-15T17:58:41.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.017+0000 7fc90b000640 1 -- start start 2024-01-15T17:58:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.018+0000 7fc90b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc9040d1d90 0x7fc9040d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.018+0000 7fc90b000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9040d3a60 con 0x7fc9040d3090 2024-01-15T17:58:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.018+0000 7fc90b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9040ce430 con 0x7fc90415d340 2024-01-15T17:58:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.018+0000 7fc90b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9040b3de0 con 0x7fc9040d1d90 2024-01-15T17:58:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.019+0000 7fc909200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc9040d3090 0x7fc9040d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49154/0 (socket says 172.21.15.2:49154) 2024-01-15T17:58:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.019+0000 7fc909200640 1 -- 172.21.15.2:0/3870023521 learned_addr learned my addr 172.21.15.2:0/3870023521 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.019+0000 7fc90a600640 1 --2- 172.21.15.2:0/3870023521 >> v2:172.21.15.137:3300/0 conn(0x7fc9040d1d90 0x7fc9040d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.019+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973736378 0 0) 0x7fc9040d3a60 con 0x7fc9040d3090 2024-01-15T17:58:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.019+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4003520 con 0x7fc9040d3090 2024-01-15T17:58:41.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3870441629 0 0) 0x7fc8e4003520 con 0x7fc9040d3090 2024-01-15T17:58:41.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 >> v2:172.21.15.137:3300/0 conn(0x7fc9040d1d90 msgr2=0x7fc9040d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 --2- 172.21.15.2:0/3870023521 >> v2:172.21.15.137:3300/0 conn(0x7fc9040d1d90 0x7fc9040d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 >> v1:172.21.15.137:6789/0 conn(0x7fc90415d340 legacy=0x7fc9040cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9040b40c0 con 0x7fc9040d3090 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc8ec001670 con 0x7fc9040d3090 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 >> v1:172.21.15.2:6789/0 conn(0x7fc9040d3090 legacy=0x7fc9040d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc8e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e40047c0 0x7fc8e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e4005180 0x7fc8e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040b3de0 con 0x7fc8e40039c0 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e400c080 con 0x7fc8e4005180 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e400c240 con 0x7fc8e40047c0 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.020+0000 7fc909c00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc8e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc90a600640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e4005180 0x7fc8e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909200640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e40047c0 0x7fc8e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909c00640 1 -- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e40047c0 msgr2=0x7fc8e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909c00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e40047c0 0x7fc8e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909c00640 1 -- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e4005180 msgr2=0x7fc8e400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909c00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e4005180 0x7fc8e400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909c00640 1 -- 172.21.15.2:0/3870023521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4003520 con 0x7fc8e40039c0 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc90a600640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8e4005180 0x7fc8e400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc909200640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8e40047c0 0x7fc8e4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.021+0000 7fc90a600640 1 -- 172.21.15.2:0/3870023521 reap_dead start 2024-01-15T17:58:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.022+0000 7fc909c00640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc8e4003de0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4001750 tx=0x7fc8e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398b5cc972c4ad12 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.022+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40052d0 con 0x7fc8e40039c0 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.022+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f402e050 con 0x7fc8e40039c0 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.022+0000 7fc903e00640 1 -- 172.21.15.2:0/3870023521 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003830 con 0x7fc8e40039c0 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 -- 172.21.15.2:0/3870023521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 msgr2=0x7fc8e4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc8e4003de0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4001750 tx=0x7fc8e40057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 -- 172.21.15.2:0/3870023521 shutdown_connections 2024-01-15T17:58:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 --2- 172.21.15.2:0/3870023521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc8e4003de0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 -- 172.21.15.2:0/3870023521 >> 172.21.15.2:0/3870023521 conn(0x7fc90408caf0 msgr2=0x7fc90409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 -- 172.21.15.2:0/3870023521 shutdown_connections 2024-01-15T17:58:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.023+0000 7fc90b000640 1 -- 172.21.15.2:0/3870023521 wait complete. 2024-01-15T17:58:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.025+0000 7fc90b000640 1 Processor -- start 2024-01-15T17:58:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.025+0000 7fc90b000640 1 -- start start 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc90b000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc9040b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc90b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9040c6c60 0x7fc9040c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc90b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9040c70e0 0x7fc9040c95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc90b000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90415ea40 con 0x7fc8e40039c0 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc90b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040c9ae0 con 0x7fc9040c6c60 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc90b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9040c7c00 con 0x7fc9040c70e0 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc909c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc9040b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc909c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc9040b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:32992/0 (socket says 172.21.15.2:32992) 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.026+0000 7fc909c00640 1 -- 172.21.15.2:0/3001917540 learned_addr learned my addr 172.21.15.2:0/3001917540 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc90a600640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9040c70e0 0x7fc9040c95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909c00640 1 -- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9040c70e0 msgr2=0x7fc9040c95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909200640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9040c6c60 0x7fc9040c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909c00640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9040c70e0 0x7fc9040c95a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909c00640 1 -- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9040c6c60 msgr2=0x7fc9040c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909c00640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9040c6c60 0x7fc9040c9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909c00640 1 -- 172.21.15.2:0/3001917540 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9040c7640 con 0x7fc8e40039c0 2024-01-15T17:58:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc909c00640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc9040b17a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4005040 tx=0x7fc8f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.027+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f40076c0 con 0x7fc8e40039c0 2024-01-15T17:58:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.028+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f4007a30 con 0x7fc8e40039c0 2024-01-15T17:58:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.028+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4006be0 con 0x7fc8e40039c0 2024-01-15T17:58:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.028+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9040c7920 con 0x7fc8e40039c0 2024-01-15T17:58:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.028+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9040affa0 con 0x7fc8e40039c0 2024-01-15T17:58:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.029+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90415e530 con 0x7fc8e40039c0 2024-01-15T17:58:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.030+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc8f4006d40 con 0x7fc8e40039c0 2024-01-15T17:58:41.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.031+0000 7fc901600640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc8dc06b2f0 0x7fc8dc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.031+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc8f40ab880 con 0x7fc8e40039c0 2024-01-15T17:58:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.031+0000 7fc909200640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc8dc06b2f0 0x7fc8dc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.033+0000 7fc909200640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc8dc06b2f0 0x7fc8dc06d7b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0017a0 tx=0x7fc8ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.035+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc90415e530 con 0x7fc8e40039c0 2024-01-15T17:58:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.277+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc90415e530 con 0x7fc8e40039c0 2024-01-15T17:58:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.278+0000 7fc901600640 1 -- 172.21.15.2:0/3001917540 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4907 (secure 0 0 0) 0x7fc90415e530 con 0x7fc8e40039c0 2024-01-15T17:58:41.283 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:41.283 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4211,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3766281910","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3766281910},{"type":"v1","addr":"172.21.15.2:6835","nonce":3766281910}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4220,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/4132369992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":4132369992},{"type":"v1","addr":"172.21.15.137:6835","nonce":4132369992}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4223,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1791958129","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":1791958129},{"type":"v1","addr":"172.21.15.137:6837","nonce":1791958129}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:57:31.583413+0000","modified":"2024-01-15T17:57:34.648185+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4233},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4233":{"gid":4233,"name":"a","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/3854928362","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3854928362},{"type":"v1","addr":"172.21.15.2:6833","nonce":3854928362}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-15T17:58:41.283 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 7 2024-01-15T17:58:41.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc8dc06b2f0 msgr2=0x7fc8dc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc8dc06b2f0 0x7fc8dc06d7b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0017a0 tx=0x7fc8ec009040 comp rx=0 tx=0).stop 2024-01-15T17:58:41.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 msgr2=0x7fc9040b17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc9040b17a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4005040 tx=0x7fc8f4001120 comp rx=0 tx=0).stop 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 shutdown_connections 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc8dc06b2f0 0x7fc8dc06d7b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc9040c70e0 0x7fc9040c95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc9040c6c60 0x7fc9040c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 --2- 172.21.15.2:0/3001917540 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc8e40039c0 0x7fc9040b17a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 >> 172.21.15.2:0/3001917540 conn(0x7fc90408caf0 msgr2=0x7fc9040d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.281+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 shutdown_connections 2024-01-15T17:58:41.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.282+0000 7fc90b000640 1 -- 172.21.15.2:0/3001917540 wait complete. 2024-01-15T17:58:41.305 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:41.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.520+0000 7f0a4c200640 1 Processor -- start 2024-01-15T17:58:41.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.520+0000 7f0a4c200640 1 -- start start 2024-01-15T17:58:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a441469f0 0x7f0a44071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4c200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0a4406cb80 0x7f0a440b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4c200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a440ac870 con 0x7f0a44146dd0 2024-01-15T17:58:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4c200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4415d970 con 0x7f0a4406cb80 2024-01-15T17:58:41.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4415dc30 con 0x7f0a441469f0 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0a4406cb80 0x7f0a440b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0a4406cb80 0x7f0a440b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33006/0 (socket says 172.21.15.2:33006) 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.522+0000 7f0a4a200640 1 -- 172.21.15.2:0/2843413419 learned_addr learned my addr 172.21.15.2:0/2843413419 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.523+0000 7f0a49800640 1 --2- 172.21.15.2:0/2843413419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a441469f0 0x7f0a44071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.523+0000 7f0a4a200640 1 -- 172.21.15.2:0/2843413419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a441469f0 msgr2=0x7f0a44071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.523+0000 7f0a4a200640 1 --2- 172.21.15.2:0/2843413419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a441469f0 0x7f0a44071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.523+0000 7f0a4a200640 1 -- 172.21.15.2:0/2843413419 >> v1:172.21.15.137:6789/0 conn(0x7f0a44146dd0 legacy=0x7f0a4406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.523+0000 7f0a4a200640 1 -- 172.21.15.2:0/2843413419 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4415df10 con 0x7f0a4406cb80 2024-01-15T17:58:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.524+0000 7f0a4a200640 1 --2- 172.21.15.2:0/2843413419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4406cb80 0x7f0a440b34a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009870 tx=0x7f0a3c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d310315c4c0ef05 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a43e00640 1 -- 172.21.15.2:0/2843413419 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c030840 con 0x7f0a4406cb80 2024-01-15T17:58:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a43e00640 1 -- 172.21.15.2:0/2843413419 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c0309a0 con 0x7f0a4406cb80 2024-01-15T17:58:41.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a43e00640 1 -- 172.21.15.2:0/2843413419 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c030d30 con 0x7f0a4406cb80 2024-01-15T17:58:41.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a4c200640 1 -- 172.21.15.2:0/2843413419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4406cb80 msgr2=0x7f0a440b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a4c200640 1 --2- 172.21.15.2:0/2843413419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4406cb80 0x7f0a440b34a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009870 tx=0x7f0a3c02b980 comp rx=0 tx=0).stop 2024-01-15T17:58:41.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a4c200640 1 -- 172.21.15.2:0/2843413419 shutdown_connections 2024-01-15T17:58:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a4c200640 1 --2- 172.21.15.2:0/2843413419 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4406cb80 0x7f0a440b34a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a4c200640 1 --2- 172.21.15.2:0/2843413419 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a441469f0 0x7f0a44071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.525+0000 7f0a4c200640 1 -- 172.21.15.2:0/2843413419 >> 172.21.15.2:0/2843413419 conn(0x7f0a4406da20 msgr2=0x7f0a4406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.526+0000 7f0a4c200640 1 -- 172.21.15.2:0/2843413419 shutdown_connections 2024-01-15T17:58:41.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.526+0000 7f0a4c200640 1 -- 172.21.15.2:0/2843413419 wait complete. 2024-01-15T17:58:41.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.527+0000 7f0a4c200640 1 Processor -- start 2024-01-15T17:58:41.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.528+0000 7f0a4c200640 1 -- start start 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.528+0000 7f0a4c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a4406cb80 0x7f0a44159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a441469f0 0x7f0a4413a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4c200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 0x7f0a44138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4c200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4413a550 con 0x7f0a4413b730 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44159a90 con 0x7f0a441469f0 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a44139910 con 0x7f0a4406cb80 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 0x7f0a44138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 0x7f0a44138fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33018/0 (socket says 172.21.15.2:33018) 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 -- 172.21.15.2:0/613144111 learned_addr learned my addr 172.21.15.2:0/613144111 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a49800640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a4406cb80 0x7f0a44159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a48e00640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a441469f0 0x7f0a4413a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 -- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a4406cb80 msgr2=0x7f0a44159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a4406cb80 0x7f0a44159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 -- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a441469f0 msgr2=0x7f0a4413a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a441469f0 0x7f0a4413a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.529+0000 7f0a4a200640 1 -- 172.21.15.2:0/613144111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3c0094d0 con 0x7f0a4413b730 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.530+0000 7f0a4a200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 0x7f0a44138fb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009870 tx=0x7f0a3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.530+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c039ce0 con 0x7f0a4413b730 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.530+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a3c039e40 con 0x7f0a4413b730 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.530+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c043620 con 0x7f0a4413b730 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.530+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a44139bb0 con 0x7f0a4413b730 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.530+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4413a170 con 0x7f0a4413b730 2024-01-15T17:58:41.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.531+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a44141a30 con 0x7f0a4413b730 2024-01-15T17:58:41.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.532+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0a3c043780 con 0x7f0a4413b730 2024-01-15T17:58:41.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.533+0000 7f0a41600640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a1c06b2c0 0x7f0a1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.533+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0a3c03e070 con 0x7f0a4413b730 2024-01-15T17:58:41.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.534+0000 7f0a49800640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a1c06b2c0 0x7f0a1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.535+0000 7f0a49800640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a1c06b2c0 0x7f0a1c06d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0a34007330 tx=0x7f0a3400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.537+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a3c077a70 con 0x7f0a4413b730 2024-01-15T17:58:41.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.734+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a4413a690 con 0x7f0a4413b730 2024-01-15T17:58:41.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.736+0000 7f0a41600640 1 -- 172.21.15.2:0/613144111 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f0a4413a690 con 0x7f0a4413b730 2024-01-15T17:58:41.740 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:41.740 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":24,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:19.324873+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:57:29.871174+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-15T17:57:30.603017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.27","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a1c06b2c0 msgr2=0x7f0a1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a1c06b2c0 0x7f0a1c06d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0a34007330 tx=0x7f0a3400b040 comp rx=0 tx=0).stop 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 msgr2=0x7f0a44138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 0x7f0a44138fb0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009870 tx=0x7f0a3c004730 comp rx=0 tx=0).stop 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 shutdown_connections 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a1c06b2c0 0x7f0a1c06d780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0a4413b730 0x7f0a44138fb0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0a441469f0 0x7f0a4413a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 --2- 172.21.15.2:0/613144111 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0a4406cb80 0x7f0a44159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 >> 172.21.15.2:0/613144111 conn(0x7f0a4406da20 msgr2=0x7f0a4406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 shutdown_connections 2024-01-15T17:58:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.739+0000 7f0a4c200640 1 -- 172.21.15.2:0/613144111 wait complete. 2024-01-15T17:58:41.762 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-15T17:58:41.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.982+0000 7f5b18200640 1 Processor -- start 2024-01-15T17:58:41.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.982+0000 7f5b18200640 1 -- start start 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b18200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 0x7f5b100cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b18200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5b100b30a0 0x7f5b1006c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b18200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b100b35a0 con 0x7f5b1006cdf0 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b18200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b100b4220 con 0x7f5b100b30a0 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b18200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006d280 con 0x7f5b10144b80 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b16200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5b1006cdf0 0x7f5b100b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49162/0 (socket says 172.21.15.2:49162) 2024-01-15T17:58:41.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.984+0000 7f5b16200640 1 -- 172.21.15.2:0/996404977 learned_addr learned my addr 172.21.15.2:0/996404977 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b14e00640 1 --2- 172.21.15.2:0/996404977 >> v2:172.21.15.137:3300/0 conn(0x7f5b100b30a0 0x7f5b1006c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b15800640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 0x7f5b100cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951107534 0 0) 0x7f5b100b35a0 con 0x7f5b1006cdf0 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af0003520 con 0x7f5b1006cdf0 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 809052718 0 0) 0x7f5af0003520 con 0x7f5b1006cdf0 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 msgr2=0x7f5b100cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 0x7f5b100cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 >> v2:172.21.15.137:3300/0 conn(0x7f5b100b30a0 msgr2=0x7f5b1006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.985+0000 7f5b0fe00640 1 --2- 172.21.15.2:0/996404977 >> v2:172.21.15.137:3300/0 conn(0x7f5b100b30a0 0x7f5b1006c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100b3880 con 0x7f5b1006cdf0 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5b080019f0 con 0x7f5b1006cdf0 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 >> v1:172.21.15.2:6789/0 conn(0x7f5b1006cdf0 legacy=0x7f5b100b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 0x7f5af0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0003ef0 0x7f5af00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0004d00 0x7f5af0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b100b4220 con 0x7f5af0004d00 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1006d280 con 0x7f5af0003ef0 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af000b9d0 con 0x7f5b10144b80 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b16200640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0004d00 0x7f5af0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b14e00640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0003ef0 0x7f5af00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b15800640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 0x7f5af0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b16200640 1 -- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 msgr2=0x7f5af0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b16200640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b10144b80 0x7f5af0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b16200640 1 -- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0003ef0 msgr2=0x7f5af00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:41.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b16200640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0003ef0 0x7f5af00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.986+0000 7f5b16200640 1 -- 172.21.15.2:0/996404977 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af0003520 con 0x7f5af0004d00 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b16200640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0004d00 0x7f5af0005180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800a890 tx=0x7f5af00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66966d18a150c40e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08034630 con 0x7f5af0004d00 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0802f070 con 0x7f5af0004d00 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b0fe00640 1 -- 172.21.15.2:0/996404977 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b08034a30 con 0x7f5af0004d00 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b18200640 1 -- 172.21.15.2:0/996404977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0004d00 msgr2=0x7f5af0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b18200640 1 --2- 172.21.15.2:0/996404977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5af0004d00 0x7f5af0005180 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800a890 tx=0x7f5af00057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.988+0000 7f5b16200640 1 -- 172.21.15.2:0/996404977 reap_dead start 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.989+0000 7f5b18200640 1 -- 172.21.15.2:0/996404977 shutdown_connections 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.989+0000 7f5b18200640 1 -- 172.21.15.2:0/996404977 >> 172.21.15.2:0/996404977 conn(0x7f5b100b46d0 msgr2=0x7f5b100b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.989+0000 7f5b18200640 1 -- 172.21.15.2:0/996404977 shutdown_connections 2024-01-15T17:58:41.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.989+0000 7f5b18200640 1 -- 172.21.15.2:0/996404977 wait complete. 2024-01-15T17:58:41.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.990+0000 7f5b18200640 1 Processor -- start 2024-01-15T17:58:41.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 -- start start 2024-01-15T17:58:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0004d00 0x7f5b10071890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 0x7f5b1014f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1014fc50 0x7f5b1014d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:41.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10071dd0 con 0x7f5b1013af60 2024-01-15T17:58:41.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10071400 con 0x7f5af0004d00 2024-01-15T17:58:41.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.991+0000 7f5b18200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101500d0 con 0x7f5b1014fc50 2024-01-15T17:58:41.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 0x7f5b1014f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 0x7f5b1014f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33048/0 (socket says 172.21.15.2:33048) 2024-01-15T17:58:41.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 -- 172.21.15.2:0/272228881 learned_addr learned my addr 172.21.15.2:0/272228881 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b15800640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0004d00 0x7f5b10071890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b16200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1014fc50 0x7f5b1014d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 -- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1014fc50 msgr2=0x7f5b1014d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1014fc50 0x7f5b1014d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 -- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0004d00 msgr2=0x7f5b10071890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0004d00 0x7f5b10071890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:41.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b14e00640 1 -- 172.21.15.2:0/272228881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b100707f0 con 0x7f5b1013af60 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.992+0000 7f5b16200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1014fc50 0x7f5b1014d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.993+0000 7f5b14e00640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 0x7f5b1014f710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5af8011630 tx=0x7f5af8011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.993+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af800f920 con 0x7f5b1013af60 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.993+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af800fa80 con 0x7f5b1013af60 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.993+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af800fdf0 con 0x7f5b1013af60 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.993+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b10070ad0 con 0x7f5b1013af60 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.993+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b10071090 con 0x7f5b1013af60 2024-01-15T17:58:41.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.994+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b100715b0 con 0x7f5b1013af60 2024-01-15T17:58:41.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.995+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5af8015460 con 0x7f5b1013af60 2024-01-15T17:58:42.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.996+0000 7f5b0d600640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ae806b2c0 0x7f5ae806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.996+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5af808a200 con 0x7f5b1013af60 2024-01-15T17:58:42.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.997+0000 7f5b15800640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ae806b2c0 0x7f5ae806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:41.998+0000 7f5b15800640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ae806b2c0 0x7f5ae806d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009950 tx=0x7f5b00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:42.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.000+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b100715b0 con 0x7f5b1013af60 2024-01-15T17:58:42.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.233+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5b10071570 con 0x7f5b1013af60 2024-01-15T17:58:42.239 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T17:58:42.235+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T17:58:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.238+0000 7f5b0d600640 1 -- 172.21.15.2:0/272228881 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5af8054f50 con 0x7f5b1013af60 2024-01-15T17:58:42.242 INFO:teuthology.orchestra.run.smithi002.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.241+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ae806b2c0 msgr2=0x7f5ae806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.241+0000 7f5b18200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ae806b2c0 0x7f5ae806d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009950 tx=0x7f5b00008040 comp rx=0 tx=0).stop 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.241+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 msgr2=0x7f5b1014f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.241+0000 7f5b18200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 0x7f5b1014f710 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5af8011630 tx=0x7f5af8011af0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 shutdown_connections 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ae806b2c0 0x7f5ae806d780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b1014fc50 0x7f5b1014d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b1013af60 0x7f5b1014f710 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 --2- 172.21.15.2:0/272228881 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5af0004d00 0x7f5b10071890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 >> 172.21.15.2:0/272228881 conn(0x7f5b100b46d0 msgr2=0x7f5b101422c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 shutdown_connections 2024-01-15T17:58:42.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.242+0000 7f5b18200640 1 -- 172.21.15.2:0/272228881 wait complete. 2024-01-15T17:58:42.257 INFO:tasks.ceph.mds.a.smithi002.stderr: -3985> 2024-01-15T17:57:34.634+0000 7f407e400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T17:58:42.271 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-15T17:58:42.335 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T17:58:42.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.462+0000 7f83a4000640 1 Processor -- start 2024-01-15T17:58:42.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.463+0000 7f83a4000640 1 -- start start 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c144c00 0x7f839c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a4000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f839c0cdd70 0x7f839c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a4000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f839c15c450 0x7f839c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a4000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0ce230 con 0x7f839c0cdd70 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a4000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0ce510 con 0x7f839c15c450 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0ce7f0 con 0x7f839c144c00 2024-01-15T17:58:42.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a0c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f839c0cdd70 0x7f839c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a0c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f839c0cdd70 0x7f839c06c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33082/0 (socket says 172.21.15.2:33082) 2024-01-15T17:58:42.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.464+0000 7f83a0c00640 1 -- 172.21.15.2:0/1680488473 learned_addr learned my addr 172.21.15.2:0/1680488473 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a1600640 1 --2- 172.21.15.2:0/1680488473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c144c00 0x7f839c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a2000640 1 --2- 172.21.15.2:0/1680488473 >> v2:172.21.15.137:3300/0 conn(0x7f839c15c450 0x7f839c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a0c00640 1 -- 172.21.15.2:0/1680488473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c144c00 msgr2=0x7f839c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a0c00640 1 --2- 172.21.15.2:0/1680488473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c144c00 0x7f839c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a0c00640 1 -- 172.21.15.2:0/1680488473 >> v2:172.21.15.137:3300/0 conn(0x7f839c15c450 msgr2=0x7f839c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a0c00640 1 --2- 172.21.15.2:0/1680488473 >> v2:172.21.15.137:3300/0 conn(0x7f839c15c450 0x7f839c06ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.465+0000 7f83a0c00640 1 -- 172.21.15.2:0/1680488473 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0705f0 con 0x7f839c0cdd70 2024-01-15T17:58:42.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.466+0000 7f83a0c00640 1 --2- 172.21.15.2:0/1680488473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c0cdd70 0x7f839c06c8f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f839c0cf090 tx=0x7f838402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6712b7b85f6df528 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.466+0000 7f839be00640 1 -- 172.21.15.2:0/1680488473 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384031870 con 0x7f839c0cdd70 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f839be00640 1 -- 172.21.15.2:0/1680488473 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83840319d0 con 0x7f839c0cdd70 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f839be00640 1 -- 172.21.15.2:0/1680488473 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384031d60 con 0x7f839c0cdd70 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 -- 172.21.15.2:0/1680488473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c0cdd70 msgr2=0x7f839c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 --2- 172.21.15.2:0/1680488473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c0cdd70 0x7f839c06c8f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f839c0cf090 tx=0x7f838402b910 comp rx=0 tx=0).stop 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 -- 172.21.15.2:0/1680488473 shutdown_connections 2024-01-15T17:58:42.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 --2- 172.21.15.2:0/1680488473 >> v2:172.21.15.137:3300/0 conn(0x7f839c15c450 0x7f839c06ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 --2- 172.21.15.2:0/1680488473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c0cdd70 0x7f839c06c8f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 --2- 172.21.15.2:0/1680488473 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c144c00 0x7f839c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 -- 172.21.15.2:0/1680488473 >> 172.21.15.2:0/1680488473 conn(0x7f839c0b45f0 msgr2=0x7f839c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 -- 172.21.15.2:0/1680488473 shutdown_connections 2024-01-15T17:58:42.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.467+0000 7f83a4000640 1 -- 172.21.15.2:0/1680488473 wait complete. 2024-01-15T17:58:42.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.469+0000 7f83a4000640 1 Processor -- start 2024-01-15T17:58:42.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.469+0000 7f83a4000640 1 -- start start 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c0cdd70 0x7f839c0cf900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a4000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 0x7f839c1f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f839c15c450 0x7f839c13eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a4000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c1f6f70 con 0x7f839c144c00 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c1f37c0 con 0x7f839c15c450 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c13f090 con 0x7f839c0cdd70 2024-01-15T17:58:42.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 0x7f839c1f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 0x7f839c1f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33090/0 (socket says 172.21.15.2:33090) 2024-01-15T17:58:42.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.470+0000 7f83a0c00640 1 -- 172.21.15.2:0/1711473905 learned_addr learned my addr 172.21.15.2:0/1711473905 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a2000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f839c15c450 0x7f839c13eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a1600640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c0cdd70 0x7f839c0cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a0c00640 1 -- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c0cdd70 msgr2=0x7f839c0cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a0c00640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c0cdd70 0x7f839c0cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a0c00640 1 -- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f839c15c450 msgr2=0x7f839c13eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a0c00640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f839c15c450 0x7f839c13eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a0c00640 1 -- 172.21.15.2:0/1711473905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83840094d0 con 0x7f839c144c00 2024-01-15T17:58:42.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a1600640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c0cdd70 0x7f839c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.471+0000 7f83a0c00640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 0x7f839c1f6a30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8384000c00 tx=0x7f83840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.472+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838403ad30 con 0x7f839c144c00 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.472+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83840443f0 con 0x7f839c144c00 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.472+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840447a0 con 0x7f839c144c00 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.472+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c1f2bb0 con 0x7f839c144c00 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.472+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c1f3170 con 0x7f839c144c00 2024-01-15T17:58:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.473+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c1f3970 con 0x7f839c144c00 2024-01-15T17:58:42.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.474+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8384044900 con 0x7f839c144c00 2024-01-15T17:58:42.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.475+0000 7f8399600640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f837406b2c0 0x7f837406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:42.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.475+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f838402d040 con 0x7f839c144c00 2024-01-15T17:58:42.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.476+0000 7f83a1600640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f837406b2c0 0x7f837406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:42.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.477+0000 7f83a1600640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f837406b2c0 0x7f837406d780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f838c006fd0 tx=0x7f838c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:42.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.480+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f839c1f3970 con 0x7f839c144c00 2024-01-15T17:58:42.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:42.718+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f839c1f3970 con 0x7f839c144c00 2024-01-15T17:58:43.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.326+0000 7f8399600640 1 -- 172.21.15.2:0/1711473905 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f839c1f3970 con 0x7f839c144c00 2024-01-15T17:58:43.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f837406b2c0 msgr2=0x7f837406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:43.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f837406b2c0 0x7f837406d780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f838c006fd0 tx=0x7f838c008040 comp rx=0 tx=0).stop 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 msgr2=0x7f839c1f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 0x7f839c1f6a30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f8384000c00 tx=0x7f83840046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 shutdown_connections 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f837406b2c0 0x7f837406d780 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f839c15c450 0x7f839c13eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f839c144c00 0x7f839c1f6a30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.330+0000 7f83a4000640 1 --2- 172.21.15.2:0/1711473905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f839c0cdd70 0x7f839c0cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.331+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 >> 172.21.15.2:0/1711473905 conn(0x7f839c0b45f0 msgr2=0x7f839c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.331+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 shutdown_connections 2024-01-15T17:58:43.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.331+0000 7f83a4000640 1 -- 172.21.15.2:0/1711473905 wait complete. 2024-01-15T17:58:43.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-15T17:58:43.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.571+0000 7f8c23e00640 1 Processor -- start 2024-01-15T17:58:43.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.571+0000 7f8c23e00640 1 -- start start 2024-01-15T17:58:43.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c23e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8c240c7ef0 0x7f8c2415e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c23e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c23e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c240c99b0 con 0x7f8c240c9210 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c23e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c240caa50 con 0x7f8c240c7ef0 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c23e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c240c9c90 con 0x7f8c240709d0 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c22a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8c240c7ef0 0x7f8c2415e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c22000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f8c240c9210 0x7f8c240c82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51942/0 (socket says 172.21.15.2:51942) 2024-01-15T17:58:43.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c22000640 1 -- 172.21.15.2:0/2003470159 learned_addr learned my addr 172.21.15.2:0/2003470159 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.573+0000 7f8c23400640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c21600640 1 -- 172.21.15.2:0/2003470159 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951371593 0 0) 0x7f8c240c99b0 con 0x7f8c240c9210 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c21600640 1 -- 172.21.15.2:0/2003470159 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04003520 con 0x7f8c240c9210 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c22a00640 1 -- 172.21.15.2:0/2003470159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 msgr2=0x7f8c24070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c22a00640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c22a00640 1 -- 172.21.15.2:0/2003470159 >> v1:172.21.15.2:6789/0 conn(0x7f8c240c9210 legacy=0x7f8c240c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c22a00640 1 -- 172.21.15.2:0/2003470159 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c240c9f70 con 0x7f8c240c7ef0 2024-01-15T17:58:43.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.574+0000 7f8c23400640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:43.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.576+0000 7f8c22a00640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c7ef0 0x7f8c2415e1d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009d90 tx=0x7f8c1402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f98ebc91adc50a53 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:43.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.576+0000 7f8c21600640 1 -- 172.21.15.2:0/2003470159 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1402cc50 con 0x7f8c240c7ef0 2024-01-15T17:58:43.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.576+0000 7f8c21600640 1 -- 172.21.15.2:0/2003470159 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1402cdb0 con 0x7f8c240c7ef0 2024-01-15T17:58:43.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.576+0000 7f8c21600640 1 -- 172.21.15.2:0/2003470159 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c14034660 con 0x7f8c240c7ef0 2024-01-15T17:58:43.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.576+0000 7f8c23e00640 1 -- 172.21.15.2:0/2003470159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c7ef0 msgr2=0x7f8c2415e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.576+0000 7f8c23e00640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c7ef0 0x7f8c2415e1d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f8c14009d90 tx=0x7f8c1402bea0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.577+0000 7f8c23e00640 1 -- 172.21.15.2:0/2003470159 shutdown_connections 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.577+0000 7f8c23e00640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.577+0000 7f8c23e00640 1 --2- 172.21.15.2:0/2003470159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c7ef0 0x7f8c2415e1d0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.577+0000 7f8c23e00640 1 -- 172.21.15.2:0/2003470159 >> 172.21.15.2:0/2003470159 conn(0x7f8c240cebc0 msgr2=0x7f8c240cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.577+0000 7f8c23e00640 1 -- 172.21.15.2:0/2003470159 shutdown_connections 2024-01-15T17:58:43.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.577+0000 7f8c23e00640 1 -- 172.21.15.2:0/2003470159 wait complete. 2024-01-15T17:58:43.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.578+0000 7f8c23e00640 1 Processor -- start 2024-01-15T17:58:43.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.579+0000 7f8c23e00640 1 -- start start 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c240c7ef0 0x7f8c240b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 0x7f8c24152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24154180 con 0x7f8c240c9210 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24154460 con 0x7f8c240c7ef0 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c24154740 con 0x7f8c240709d0 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 0x7f8c24152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:43.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 0x7f8c24152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36482/0 (socket says 172.21.15.2:36482) 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 -- 172.21.15.2:0/1476762035 learned_addr learned my addr 172.21.15.2:0/1476762035 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c22a00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c22000640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c240c7ef0 0x7f8c240b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 -- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 msgr2=0x7f8c24155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24155220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 -- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c240c7ef0 msgr2=0x7f8c240b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c240c7ef0 0x7f8c240b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.580+0000 7f8c23400640 1 -- 172.21.15.2:0/1476762035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c14009a70 con 0x7f8c240c9210 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.581+0000 7f8c23400640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 0x7f8c24152aa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00e7e0 tx=0x7f8c1c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.581+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c004030 con 0x7f8c240c9210 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.581+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1c004190 con 0x7f8c240c9210 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.581+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c004500 con 0x7f8c240c9210 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.581+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c24154a20 con 0x7f8c240c9210 2024-01-15T17:58:43.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.581+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c24155ae0 con 0x7f8c240c9210 2024-01-15T17:58:43.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.582+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c24154f40 con 0x7f8c240c9210 2024-01-15T17:58:43.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.583+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8c1c00c6d0 con 0x7f8c240c9210 2024-01-15T17:58:43.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.585+0000 7f8c1aa00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bfc06b2c0 0x7f8bfc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:43.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.585+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f8c1c089980 con 0x7f8c240c9210 2024-01-15T17:58:43.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.586+0000 7f8c22a00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bfc06b2c0 0x7f8bfc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:43.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.587+0000 7f8c22a00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bfc06b2c0 0x7f8bfc06d780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8c14036000 tx=0x7f8c14030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:43.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.589+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c24154f40 con 0x7f8c240c9210 2024-01-15T17:58:43.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:43.784+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8c24154f40 con 0x7f8c240c9210 2024-01-15T17:58:44.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.663+0000 7f8c1aa00640 1 -- 172.21.15.2:0/1476762035 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8c24154f40 con 0x7f8c240c9210 2024-01-15T17:58:44.668 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' removed 2024-01-15T17:58:44.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.668+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bfc06b2c0 msgr2=0x7f8bfc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:44.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.668+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bfc06b2c0 0x7f8bfc06d780 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8c14036000 tx=0x7f8c14030000 comp rx=0 tx=0).stop 2024-01-15T17:58:44.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.668+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 msgr2=0x7f8c24152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:44.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.668+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 0x7f8c24152aa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c00e7e0 tx=0x7f8c1c00eca0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.669+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 shutdown_connections 2024-01-15T17:58:44.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.669+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bfc06b2c0 0x7f8bfc06d780 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.669+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8c240c9210 0x7f8c24152aa0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.669+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8c240c7ef0 0x7f8c240b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.669+0000 7f8c23e00640 1 --2- 172.21.15.2:0/1476762035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8c240709d0 0x7f8c24155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.669+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 >> 172.21.15.2:0/1476762035 conn(0x7f8c240cebc0 msgr2=0x7f8c240af230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:44.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.670+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 shutdown_connections 2024-01-15T17:58:44.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.671+0000 7f8c23e00640 1 -- 172.21.15.2:0/1476762035 wait complete. 2024-01-15T17:58:44.698 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-15T17:58:44.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.912+0000 7fa8a5e00640 1 Processor -- start 2024-01-15T17:58:44.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.912+0000 7fa8a5e00640 1 -- start start 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa8a5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa8a0144c00 0x7fa8a00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa8a5e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a00ce230 con 0x7fa8a015c450 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa8a5e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a00ce510 con 0x7fa8a0144c00 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa8a5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a00ce7f0 con 0x7fa8a00cdd70 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa89ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa8a0144c00 0x7fa8a00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa89ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa8a0144c00 0x7fa8a00cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36486/0 (socket says 172.21.15.2:36486) 2024-01-15T17:58:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa89ea00640 1 -- 172.21.15.2:0/2907628766 learned_addr learned my addr 172.21.15.2:0/2907628766 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.914+0000 7fa897e00640 1 --2- 172.21.15.2:0/2907628766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.915+0000 7fa89ea00640 1 -- 172.21.15.2:0/2907628766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 msgr2=0x7fa8a006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.915+0000 7fa89ea00640 1 --2- 172.21.15.2:0/2907628766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.915+0000 7fa89ea00640 1 -- 172.21.15.2:0/2907628766 >> v1:172.21.15.2:6789/0 conn(0x7fa8a015c450 legacy=0x7fa8a00b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:44.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.915+0000 7fa89ea00640 1 -- 172.21.15.2:0/2907628766 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a00705f0 con 0x7fa8a0144c00 2024-01-15T17:58:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.916+0000 7fa89ea00640 1 --2- 172.21.15.2:0/2907628766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a00cd830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa890009870 tx=0x7fa89002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31560b234ad1694 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.916+0000 7fa89e000640 1 -- 172.21.15.2:0/2907628766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890030840 con 0x7fa8a0144c00 2024-01-15T17:58:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.916+0000 7fa89e000640 1 -- 172.21.15.2:0/2907628766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8900309a0 con 0x7fa8a0144c00 2024-01-15T17:58:44.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.916+0000 7fa89e000640 1 -- 172.21.15.2:0/2907628766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa890030d30 con 0x7fa8a0144c00 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2907628766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 msgr2=0x7fa8a00cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2907628766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a00cd830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa890009870 tx=0x7fa89002b980 comp rx=0 tx=0).stop 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2907628766 shutdown_connections 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2907628766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2907628766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a00cd830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2907628766 >> 172.21.15.2:0/2907628766 conn(0x7fa8a00b45f0 msgr2=0x7fa8a00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:44.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2907628766 shutdown_connections 2024-01-15T17:58:44.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.917+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2907628766 wait complete. 2024-01-15T17:58:44.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.919+0000 7fa8a5e00640 1 Processor -- start 2024-01-15T17:58:44.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.919+0000 7fa8a5e00640 1 -- start start 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a0134890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a0140610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa8a5e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8a015c450 0x7fa8a013d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa8a5e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01f0db0 con 0x7fa8a0144c00 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa8a5e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a01415f0 con 0x7fa8a015c450 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa8a5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a0140b50 con 0x7fa8a00cdd70 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.920+0000 7fa897e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a0140610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a0140610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36490/0 (socket says 172.21.15.2:36490) 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 -- 172.21.15.2:0/2722450113 learned_addr learned my addr 172.21.15.2:0/2722450113 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:44.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa89ea00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a0134890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 -- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 msgr2=0x7fa8a0134890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a0134890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 -- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8a015c450 msgr2=0x7fa8a013d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa89f400640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8a015c450 0x7fa8a013d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8a015c450 0x7fa8a013d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 -- 172.21.15.2:0/2722450113 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900094d0 con 0x7fa8a0144c00 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa89ea00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a0134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.921+0000 7fa897e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a0140610 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa88800c7f0 tx=0x7fa88800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.922+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888004220 con 0x7fa8a0144c00 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.922+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8880043a0 con 0x7fa8a0144c00 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.922+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa888007860 con 0x7fa8a0144c00 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.922+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a013d6f0 con 0x7fa8a0144c00 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.922+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a0141030 con 0x7fa8a0144c00 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.923+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a00686f0 con 0x7fa8a0144c00 2024-01-15T17:58:44.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.924+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa888007bd0 con 0x7fa8a0144c00 2024-01-15T17:58:44.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.925+0000 7fa897400640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa87806b2c0 0x7fa87806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:44.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.925+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fa888089b70 con 0x7fa8a0144c00 2024-01-15T17:58:44.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.926+0000 7fa89ea00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa87806b2c0 0x7fa87806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:44.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.927+0000 7fa89ea00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa87806b2c0 0x7fa87806d780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa890036000 tx=0x7fa89003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:44.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:44.930+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa888054a20 con 0x7fa8a0144c00 2024-01-15T17:58:45.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.123+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8a006d200 con 0x7fa8a0144c00 2024-01-15T17:58:45.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.679+0000 7fa897400640 1 -- 172.21.15.2:0/2722450113 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa888058040 con 0x7fa8a0144c00 2024-01-15T17:58:45.683 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' removed 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa87806b2c0 msgr2=0x7fa87806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa87806b2c0 0x7fa87806d780 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa890036000 tx=0x7fa89003b040 comp rx=0 tx=0).stop 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 msgr2=0x7fa8a0140610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a0140610 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa88800c7f0 tx=0x7fa88800ccb0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 shutdown_connections 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa87806b2c0 0x7fa87806d780 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8a015c450 0x7fa8a013d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8a0144c00 0x7fa8a0140610 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 --2- 172.21.15.2:0/2722450113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8a00cdd70 0x7fa8a0134890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 >> 172.21.15.2:0/2722450113 conn(0x7fa8a00b45f0 msgr2=0x7fa8a0070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:45.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 shutdown_connections 2024-01-15T17:58:45.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.682+0000 7fa8a5e00640 1 -- 172.21.15.2:0/2722450113 wait complete. 2024-01-15T17:58:45.711 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T17:58:45.711 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T17:58:45.711 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T17:58:45.711 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:45.712 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T17:58:45.712 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T17:58:45.712 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T17:58:45.712 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:45.712 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T17:58:45.712 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T17:58:45.712 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T17:58:45.713 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:45.713 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T17:58:45.713 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T17:58:45.713 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T17:58:45.713 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T17:58:45.731 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T17:58:45.731 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T17:58:45.733 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T17:58:45.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T17:58:45.735 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T17:58:45.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T17:58:45.742 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T17:58:45.742 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T17:58:45.782 INFO:tasks.ceph.mds.d:Started 2024-01-15T17:58:45.783 DEBUG:teuthology.parallel:result is None 2024-01-15T17:58:45.788 INFO:tasks.ceph.mds.a:Started 2024-01-15T17:58:45.789 INFO:tasks.ceph.mds.b:Started 2024-01-15T17:58:45.789 DEBUG:teuthology.parallel:result is None 2024-01-15T17:58:45.789 DEBUG:teuthology.parallel:result is None 2024-01-15T17:58:45.794 INFO:tasks.ceph.mds.c:Started 2024-01-15T17:58:45.794 DEBUG:teuthology.parallel:result is None 2024-01-15T17:58:45.794 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T17:58:45.934 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-15T17:58:45.936 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T17:58:45.940 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T17:58:45.945 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T17:58:45.945 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T17:58:45.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.981+0000 7f7c46600640 1 Processor -- start 2024-01-15T17:58:45.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.981+0000 7f7c46600640 1 -- start start 2024-01-15T17:58:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c46600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7c400d0ed0 0x7f7c400d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c46600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c400b30e0 con 0x7f7c4015d200 2024-01-15T17:58:45.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c46600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c400d1ca0 con 0x7f7c400d2de0 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c46600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c400d1f80 con 0x7f7c400d0ed0 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c45200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7c4015d200 0x7f7c400cdc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51960/0 (socket says 172.21.15.2:51960) 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c45200640 1 -- 172.21.15.2:0/1321198066 learned_addr learned my addr 172.21.15.2:0/1321198066 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c45c00640 1 --2- 172.21.15.2:0/1321198066 >> v2:172.21.15.2:3300/0 conn(0x7f7c400d0ed0 0x7f7c400d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c45c00640 1 -- 172.21.15.2:0/1321198066 >> v1:172.21.15.137:6789/0 conn(0x7f7c400d2de0 legacy=0x7f7c400d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c45c00640 1 -- 172.21.15.2:0/1321198066 >> v1:172.21.15.2:6789/0 conn(0x7f7c4015d200 legacy=0x7f7c400cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c45c00640 1 -- 172.21.15.2:0/1321198066 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c400b2a90 con 0x7f7c400d0ed0 2024-01-15T17:58:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.982+0000 7f7c3fe00640 1 -- 172.21.15.2:0/1321198066 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363996977 0 0) 0x7f7c400b30e0 con 0x7f7c4015d200 2024-01-15T17:58:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.983+0000 7f7c45c00640 1 --2- 172.21.15.2:0/1321198066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400d1340 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009d90 tx=0x7f7c3802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c0c086183cf11e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c3fe00640 1 -- 172.21.15.2:0/1321198066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3802cd00 con 0x7f7c400d0ed0 2024-01-15T17:58:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c3fe00640 1 -- 172.21.15.2:0/1321198066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3802ce60 con 0x7f7c400d0ed0 2024-01-15T17:58:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c3fe00640 1 -- 172.21.15.2:0/1321198066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38034660 con 0x7f7c400d0ed0 2024-01-15T17:58:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 -- 172.21.15.2:0/1321198066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 msgr2=0x7f7c400d1340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 --2- 172.21.15.2:0/1321198066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400d1340 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7c38009d90 tx=0x7f7c3802bf50 comp rx=0 tx=0).stop 2024-01-15T17:58:45.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 -- 172.21.15.2:0/1321198066 shutdown_connections 2024-01-15T17:58:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 --2- 172.21.15.2:0/1321198066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400d1340 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 -- 172.21.15.2:0/1321198066 >> 172.21.15.2:0/1321198066 conn(0x7f7c4008caf0 msgr2=0x7f7c4009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 -- 172.21.15.2:0/1321198066 shutdown_connections 2024-01-15T17:58:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.984+0000 7f7c46600640 1 -- 172.21.15.2:0/1321198066 wait complete. 2024-01-15T17:58:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.986+0000 7f7c46600640 1 Processor -- start 2024-01-15T17:58:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.986+0000 7f7c46600640 1 -- start start 2024-01-15T17:58:45.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c46600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400b1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:45.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c46600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c400d2de0 0x7f7c4015ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:45.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c46600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c4015d200 0x7f7c40152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:45.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c46600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400cb160 con 0x7f7c400d0ed0 2024-01-15T17:58:45.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c46600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400cb580 con 0x7f7c4015d200 2024-01-15T17:58:45.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c46600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400ca200 con 0x7f7c400d2de0 2024-01-15T17:58:45.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.987+0000 7f7c45200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400b1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:45.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400b1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36550/0 (socket says 172.21.15.2:36550) 2024-01-15T17:58:45.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 -- 172.21.15.2:0/3265282367 learned_addr learned my addr 172.21.15.2:0/3265282367 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:45.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45c00640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c4015d200 0x7f7c40152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:45.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c37e00640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c400d2de0 0x7f7c4015ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 -- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c400d2de0 msgr2=0x7f7c4015ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c400d2de0 0x7f7c4015ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 -- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c4015d200 msgr2=0x7f7c40152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c4015d200 0x7f7c40152570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 -- 172.21.15.2:0/3265282367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c38009a70 con 0x7f7c400d0ed0 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.988+0000 7f7c45200640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400b1320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000c3f0 tx=0x7f7c3000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.989+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c300074e0 con 0x7f7c400d0ed0 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.989+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c30007640 con 0x7f7c400d0ed0 2024-01-15T17:58:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.989+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c400ca4e0 con 0x7f7c400d0ed0 2024-01-15T17:58:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.989+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c30004a00 con 0x7f7c400d0ed0 2024-01-15T17:58:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.989+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c400caaa0 con 0x7f7c400d0ed0 2024-01-15T17:58:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.990+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c40074770 con 0x7f7c400d0ed0 2024-01-15T17:58:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.991+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7c30004b60 con 0x7f7c400d0ed0 2024-01-15T17:58:45.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.992+0000 7f7c3d600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c1806b2c0 0x7f7c1806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:45.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.992+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7c30089560 con 0x7f7c400d0ed0 2024-01-15T17:58:45.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.993+0000 7f7c37e00640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c1806b2c0 0x7f7c1806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:45.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.993+0000 7f7c37e00640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c1806b2c0 0x7f7c1806d780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7c400cf500 tx=0x7f7c28009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:46.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:45.996+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c30054770 con 0x7f7c400d0ed0 2024-01-15T17:58:46.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.222+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7c400678e0 con 0x7f7c400d0ed0 2024-01-15T17:58:46.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.703+0000 7f7c3d600640 1 -- 172.21.15.2:0/3265282367 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f7c30057d90 con 0x7f7c400d0ed0 2024-01-15T17:58:46.706 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T17:58:46.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c1806b2c0 msgr2=0x7f7c1806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:46.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c1806b2c0 0x7f7c1806d780 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7c400cf500 tx=0x7f7c28009040 comp rx=0 tx=0).stop 2024-01-15T17:58:46.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 msgr2=0x7f7c400b1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:46.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400b1320 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7c3000c3f0 tx=0x7f7c3000c8b0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 shutdown_connections 2024-01-15T17:58:46.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c1806b2c0 0x7f7c1806d780 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c4015d200 0x7f7c40152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c400d2de0 0x7f7c4015ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 --2- 172.21.15.2:0/3265282367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c400d0ed0 0x7f7c400b1320 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 >> 172.21.15.2:0/3265282367 conn(0x7f7c4008caf0 msgr2=0x7f7c400b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:46.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 shutdown_connections 2024-01-15T17:58:46.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.708+0000 7f7c46600640 1 -- 172.21.15.2:0/3265282367 wait complete. 2024-01-15T17:58:46.734 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T17:58:46.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.948+0000 7f5a24600640 1 Processor -- start 2024-01-15T17:58:46.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.949+0000 7f5a24600640 1 -- start start 2024-01-15T17:58:46.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.950+0000 7f5a24600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a1c15c0e0 0x7f5a1c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.950+0000 7f5a24600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.950+0000 7f5a24600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5a1c15c4c0 0x7f5a1c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.950+0000 7f5a24600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1c15cb00 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.950+0000 7f5a24600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1c0ccec0 con 0x7f5a1c15c4c0 2024-01-15T17:58:46.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.950+0000 7f5a24600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0cd1a0 con 0x7f5a1c15b7d0 2024-01-15T17:58:46.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a1c15c0e0 0x7f5a1c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a1c15c0e0 0x7f5a1c0cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36554/0 (socket says 172.21.15.2:36554) 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 -- 172.21.15.2:0/4261018084 learned_addr learned my addr 172.21.15.2:0/4261018084 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21200640 1 --2- 172.21.15.2:0/4261018084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a22600640 1 --2- 172.21.15.2:0/4261018084 >> v2:172.21.15.137:3300/0 conn(0x7f5a1c15c4c0 0x7f5a1c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 -- 172.21.15.2:0/4261018084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 msgr2=0x7f5a1c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 --2- 172.21.15.2:0/4261018084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 -- 172.21.15.2:0/4261018084 >> v2:172.21.15.137:3300/0 conn(0x7f5a1c15c4c0 msgr2=0x7f5a1c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 --2- 172.21.15.2:0/4261018084 >> v2:172.21.15.137:3300/0 conn(0x7f5a1c15c4c0 0x7f5a1c06c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a21c00640 1 -- 172.21.15.2:0/4261018084 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a1c0cd480 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.951+0000 7f5a22600640 1 --2- 172.21.15.2:0/4261018084 >> v2:172.21.15.137:3300/0 conn(0x7f5a1c15c4c0 0x7f5a1c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:46.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.952+0000 7f5a21c00640 1 --2- 172.21.15.2:0/4261018084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0cc5c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0cc230 tx=0x7f5a0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9519a07b36e6d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:46.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a1be00640 1 -- 172.21.15.2:0/4261018084 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c031870 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a1be00640 1 -- 172.21.15.2:0/4261018084 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c0319d0 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a1be00640 1 -- 172.21.15.2:0/4261018084 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c031d60 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a24600640 1 -- 172.21.15.2:0/4261018084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 msgr2=0x7f5a1c0cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a24600640 1 --2- 172.21.15.2:0/4261018084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0cc5c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5a1c0cc230 tx=0x7f5a0c02b910 comp rx=0 tx=0).stop 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a24600640 1 -- 172.21.15.2:0/4261018084 shutdown_connections 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a24600640 1 --2- 172.21.15.2:0/4261018084 >> v2:172.21.15.137:3300/0 conn(0x7f5a1c15c4c0 0x7f5a1c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a24600640 1 --2- 172.21.15.2:0/4261018084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.953+0000 7f5a24600640 1 --2- 172.21.15.2:0/4261018084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0cc5c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.954+0000 7f5a24600640 1 -- 172.21.15.2:0/4261018084 >> 172.21.15.2:0/4261018084 conn(0x7f5a1c06da20 msgr2=0x7f5a1c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.954+0000 7f5a24600640 1 -- 172.21.15.2:0/4261018084 shutdown_connections 2024-01-15T17:58:46.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.954+0000 7f5a24600640 1 -- 172.21.15.2:0/4261018084 wait complete. 2024-01-15T17:58:46.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.956+0000 7f5a24600640 1 Processor -- start 2024-01-15T17:58:46.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.956+0000 7f5a24600640 1 -- start start 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.957+0000 7f5a24600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c12f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.957+0000 7f5a24600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0b2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.957+0000 7f5a24600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a1c15c4c0 0x7f5a1c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a24600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0b1e60 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a24600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0b20b0 con 0x7f5a1c15c4c0 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a24600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1c0b2390 con 0x7f5a1c15b7d0 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0b2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0b2e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36560/0 (socket says 172.21.15.2:36560) 2024-01-15T17:58:46.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 -- 172.21.15.2:0/1992917681 learned_addr learned my addr 172.21.15.2:0/1992917681 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21c00640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c12f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a22600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a1c15c4c0 0x7f5a1c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 -- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 msgr2=0x7f5a1c12f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c12f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 -- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a1c15c4c0 msgr2=0x7f5a1c0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a1c15c4c0 0x7f5a1c0b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21200640 1 -- 172.21.15.2:0/1992917681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c0094d0 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.958+0000 7f5a21c00640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c12f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.959+0000 7f5a21200640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0b2e70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400d7d0 tx=0x7f5a0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.959+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040096a0 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.959+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a04009800 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.959+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04009b90 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.959+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1c0b2670 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.959+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a1c151b40 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.960+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1c0686f0 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.961+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5a04009cf0 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.962+0000 7f5a19600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59f406b2f0 0x7f59f406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:46.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.962+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5a04089dc0 con 0x7f5a1c15c0e0 2024-01-15T17:58:46.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.963+0000 7f5a21c00640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59f406b2f0 0x7f59f406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:46.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.964+0000 7f5a21c00640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59f406b2f0 0x7f59f406d7b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c036000 tx=0x7f5a0c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:46.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:46.967+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a04054ee0 con 0x7f5a1c15c0e0 2024-01-15T17:58:47.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.231+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5a1c165e50 con 0x7f5a1c15c0e0 2024-01-15T17:58:47.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.233+0000 7f5a19600640 1 -- 172.21.15.2:0/1992917681 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f5a04058500 con 0x7f5a1c15c0e0 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T17:58:47.236 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T17:58:47.237 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.238 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.239 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.241 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.242 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T17:58:47.243 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.244 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T17:58:47.245 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.bootstrap-mds", 2024-01-15T17:58:47.246 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQD4caVlAHaLEhAA90J+gjPRPXBjO5O1N8tdyA==", 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile bootstrap-mds" 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.bootstrap-mgr", 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQD4caVlBoGLEhAAELqvlJBvmQWC6Ewl1p8jtA==", 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.bootstrap-osd", 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQD4caVlFIqLEhAADZcaWTTWgeb4KVC7jzcZGg==", 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile bootstrap-osd" 2024-01-15T17:58:47.247 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.bootstrap-rbd", 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQD4caVlYJSLEhAADOo/h5p+6QoHDaySqL/MCg==", 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQD4caVlaJ2LEhAApDw5SuTZe3sn7dpP4PHfYg==", 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.bootstrap-rgw", 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQD4caVlI6mLEhAAKYgnkhqPtLTmWjH9Yf9UbA==", 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T17:58:47.250 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T17:58:47.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59f406b2f0 msgr2=0x7f59f406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59f406b2f0 0x7f59f406d7b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c036000 tx=0x7f5a0c03c040 comp rx=0 tx=0).stop 2024-01-15T17:58:47.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 msgr2=0x7f5a1c0b2e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0b2e70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5a0400d7d0 tx=0x7f5a0400dc90 comp rx=0 tx=0).stop 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 shutdown_connections 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59f406b2f0 0x7f59f406d7b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a1c15c4c0 0x7f5a1c0b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a1c15c0e0 0x7f5a1c0b2e70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 --2- 172.21.15.2:0/1992917681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a1c15b7d0 0x7f5a1c12f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.235+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 >> 172.21.15.2:0/1992917681 conn(0x7f5a1c06da20 msgr2=0x7f5a1c14b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.236+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 shutdown_connections 2024-01-15T17:58:47.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.236+0000 7f5a24600640 1 -- 172.21.15.2:0/1992917681 wait complete. 2024-01-15T17:58:47.257 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-15T17:58:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.472+0000 7faa71600640 1 Processor -- start 2024-01-15T17:58:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.472+0000 7faa71600640 1 -- start start 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa71600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faa6c15e340 0x7faa6c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa71600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa71600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6c0aeab0 con 0x7faa6c13ce30 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa71600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6c0ce6d0 con 0x7faa6c15e340 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa71600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c0ce9b0 con 0x7faa6c0cdfa0 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa61e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faa6c15e340 0x7faa6c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa61e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faa6c15e340 0x7faa6c0cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36568/0 (socket says 172.21.15.2:36568) 2024-01-15T17:58:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.474+0000 7faa61e00640 1 -- 172.21.15.2:0/3617415256 learned_addr learned my addr 172.21.15.2:0/3617415256 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:47.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.475+0000 7faa61e00640 1 -- 172.21.15.2:0/3617415256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 msgr2=0x7faa6c070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:47.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.475+0000 7faa61e00640 1 --2- 172.21.15.2:0/3617415256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.475+0000 7faa61e00640 1 -- 172.21.15.2:0/3617415256 >> v1:172.21.15.137:6789/0 conn(0x7faa6c13ce30 legacy=0x7faa6c15dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:47.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.475+0000 7faa61e00640 1 -- 172.21.15.2:0/3617415256 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6c070380 con 0x7faa6c15e340 2024-01-15T17:58:47.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.476+0000 7faa61e00640 1 --2- 172.21.15.2:0/3617415256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c0cda60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faa54009870 tx=0x7faa5402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d989573a2b36d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.476+0000 7faa6a000640 1 -- 172.21.15.2:0/3617415256 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54031640 con 0x7faa6c15e340 2024-01-15T17:58:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.476+0000 7faa6a000640 1 -- 172.21.15.2:0/3617415256 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa540317a0 con 0x7faa6c15e340 2024-01-15T17:58:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.476+0000 7faa6a000640 1 -- 172.21.15.2:0/3617415256 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa54031b30 con 0x7faa6c15e340 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 -- 172.21.15.2:0/3617415256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 msgr2=0x7faa6c0cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 --2- 172.21.15.2:0/3617415256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c0cda60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faa54009870 tx=0x7faa5402b8f0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 -- 172.21.15.2:0/3617415256 shutdown_connections 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 --2- 172.21.15.2:0/3617415256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 --2- 172.21.15.2:0/3617415256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c0cda60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 -- 172.21.15.2:0/3617415256 >> 172.21.15.2:0/3617415256 conn(0x7faa6c06da20 msgr2=0x7faa6c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 -- 172.21.15.2:0/3617415256 shutdown_connections 2024-01-15T17:58:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.477+0000 7faa71600640 1 -- 172.21.15.2:0/3617415256 wait complete. 2024-01-15T17:58:47.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.479+0000 7faa71600640 1 Processor -- start 2024-01-15T17:58:47.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.479+0000 7faa71600640 1 -- start start 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa71600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c0aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa71600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa6c13ce30 0x7faa6c137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa71600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa71600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c152d00 con 0x7faa6c15e340 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa71600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c14e550 con 0x7faa6c13ce30 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa71600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6c0af280 con 0x7faa6c0cdfa0 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa6b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa6b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36584/0 (socket says 172.21.15.2:36584) 2024-01-15T17:58:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa6b400640 1 -- 172.21.15.2:0/4248293944 learned_addr learned my addr 172.21.15.2:0/4248293944 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa61e00640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa6c13ce30 0x7faa6c137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.480+0000 7faa6aa00640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c0aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6b400640 1 -- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 msgr2=0x7faa6c0aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6b400640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c0aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6b400640 1 -- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa6c13ce30 msgr2=0x7faa6c137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6b400640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa6c13ce30 0x7faa6c137800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6b400640 1 -- 172.21.15.2:0/4248293944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540094d0 con 0x7faa6c15e340 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6aa00640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa6b400640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c153830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faa6400b970 tx=0x7faa6400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6400cae0 con 0x7faa6c15e340 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6400cc60 con 0x7faa6c15e340 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa64014440 con 0x7faa6c15e340 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.481+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c14e760 con 0x7faa6c15e340 2024-01-15T17:58:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.482+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa6c14ed20 con 0x7faa6c15e340 2024-01-15T17:58:47.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.483+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa6c0686f0 con 0x7faa6c15e340 2024-01-15T17:58:47.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.484+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faa6401d3b0 con 0x7faa6c15e340 2024-01-15T17:58:47.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.485+0000 7faa63400640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa4406b2c0 0x7faa4406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:47.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.485+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7faa64089790 con 0x7faa6c15e340 2024-01-15T17:58:47.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.486+0000 7faa6aa00640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa4406b2c0 0x7faa4406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:47.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.487+0000 7faa6aa00640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa4406b2c0 0x7faa4406d780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa5c007fd0 tx=0x7faa5c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:47.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.492+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa640583e0 con 0x7faa6c15e340 2024-01-15T17:58:47.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.772+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7faa6c072450 con 0x7faa6c15e340 2024-01-15T17:58:47.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.776+0000 7faa63400640 1 -- 172.21.15.2:0/4248293944 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7faa640190a0 con 0x7faa6c15e340 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa4406b2c0 msgr2=0x7faa4406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa4406b2c0 0x7faa4406d780 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7faa5c007fd0 tx=0x7faa5c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 msgr2=0x7faa6c153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c153830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faa6400b970 tx=0x7faa6400be30 comp rx=0 tx=0).stop 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 shutdown_connections 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa4406b2c0 0x7faa4406d780 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa6c15e340 0x7faa6c153830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa6c13ce30 0x7faa6c137800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 --2- 172.21.15.2:0/4248293944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa6c0cdfa0 0x7faa6c0aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.779+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 >> 172.21.15.2:0/4248293944 conn(0x7faa6c06da20 msgr2=0x7faa6c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.780+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 shutdown_connections 2024-01-15T17:58:47.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:47.780+0000 7faa71600640 1 -- 172.21.15.2:0/4248293944 wait complete. 2024-01-15T17:58:47.802 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-15T17:58:48.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.022+0000 7f5082000640 1 Processor -- start 2024-01-15T17:58:48.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.022+0000 7f5082000640 1 -- start start 2024-01-15T17:58:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.024+0000 7f5082000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f507c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.024+0000 7f5082000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c06c170 con 0x7f507c06da40 2024-01-15T17:58:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.024+0000 7f5082000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c0cd1b0 con 0x7f507c15eb30 2024-01-15T17:58:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.024+0000 7f5082000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c072420 con 0x7f507c15efc0 2024-01-15T17:58:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.024+0000 7f5080c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f507c06da40 0x7f507c071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51970/0 (socket says 172.21.15.2:51970) 2024-01-15T17:58:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.024+0000 7f5080c00640 1 -- 172.21.15.2:0/311046425 learned_addr learned my addr 172.21.15.2:0/311046425 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453759432 0 0) 0x7f507c06c170 con 0x7f507c06da40 2024-01-15T17:58:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f5081600640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f507c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f505c003520 con 0x7f507c06da40 2024-01-15T17:58:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681311042 0 0) 0x7f505c003520 con 0x7f507c06da40 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 msgr2=0x7f507c0cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f507c0cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 >> v1:172.21.15.137:6789/0 conn(0x7f507c15eb30 legacy=0x7f507c0cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c071560 con 0x7f507c06da40 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f506c001670 con 0x7f507c06da40 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 >> v1:172.21.15.2:6789/0 conn(0x7f507c06da40 legacy=0x7f507c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f505c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c003f20 0x7f505c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f505c004d00 0x7f505c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c072420 con 0x7f505c004d00 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505c00ba60 con 0x7f505c003f20 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.025+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f505c00bc20 con 0x7f507c15efc0 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5081600640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f505c004d00 0x7f505c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5080c00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f505c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5073e00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c003f20 0x7f505c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5081600640 1 -- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 msgr2=0x7f505c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5081600640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f505c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5081600640 1 -- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c003f20 msgr2=0x7f505c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5081600640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c003f20 0x7f505c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5081600640 1 -- 172.21.15.2:0/311046425 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c003520 con 0x7f505c004d00 2024-01-15T17:58:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.026+0000 7f5080c00640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c15efc0 0x7f505c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.028+0000 7f5081600640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f505c004d00 0x7f505c005180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5074009fc0 tx=0x7f505c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c520b34eb741d7e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.028+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507402cc20 con 0x7f505c004d00 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f507400f040 con 0x7f505c004d00 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f507be00640 1 -- 172.21.15.2:0/311046425 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074034440 con 0x7f505c004d00 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5082000640 1 -- 172.21.15.2:0/311046425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f505c004d00 msgr2=0x7f505c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5082000640 1 --2- 172.21.15.2:0/311046425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f505c004d00 0x7f505c005180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5074009fc0 tx=0x7f505c0057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5081600640 1 -- 172.21.15.2:0/311046425 reap_dead start 2024-01-15T17:58:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5082000640 1 -- 172.21.15.2:0/311046425 shutdown_connections 2024-01-15T17:58:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5082000640 1 -- 172.21.15.2:0/311046425 >> 172.21.15.2:0/311046425 conn(0x7f507c0b0840 msgr2=0x7f507c0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5082000640 1 -- 172.21.15.2:0/311046425 shutdown_connections 2024-01-15T17:58:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.029+0000 7f5082000640 1 -- 172.21.15.2:0/311046425 wait complete. 2024-01-15T17:58:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.031+0000 7f5082000640 1 Processor -- start 2024-01-15T17:58:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.031+0000 7f5082000640 1 -- start start 2024-01-15T17:58:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.032+0000 7f5082000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c004d00 0x7f507c0ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.032+0000 7f5082000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c14d2b0 0x7f507c0b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.032+0000 7f5082000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 0x7f507c1f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.032+0000 7f5082000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0b3650 con 0x7f507c0b31d0 2024-01-15T17:58:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.032+0000 7f5082000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c14de30 con 0x7f505c004d00 2024-01-15T17:58:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.032+0000 7f5082000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0cdad0 con 0x7f507c14d2b0 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 0x7f507c1f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 0x7f507c1f6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36610/0 (socket says 172.21.15.2:36610) 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 -- 172.21.15.2:0/3438687684 learned_addr learned my addr 172.21.15.2:0/3438687684 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5073e00640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c14d2b0 0x7f507c0b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5080c00640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c004d00 0x7f507c0ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 -- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c14d2b0 msgr2=0x7f507c0b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c14d2b0 0x7f507c0b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 -- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c004d00 msgr2=0x7f507c0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c004d00 0x7f507c0ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5081600640 1 -- 172.21.15.2:0/3438687684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0cddb0 con 0x7f507c0b31d0 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.033+0000 7f5073e00640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c14d2b0 0x7f507c0b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.034+0000 7f5081600640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 0x7f507c1f6440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5074037040 tx=0x7f5074009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.034+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074043070 con 0x7f507c0b31d0 2024-01-15T17:58:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.034+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5074034ad0 con 0x7f507c0b31d0 2024-01-15T17:58:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.034+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074034e40 con 0x7f507c0b31d0 2024-01-15T17:58:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.034+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0ce090 con 0x7f507c0b31d0 2024-01-15T17:58:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.034+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c0ce650 con 0x7f507c0b31d0 2024-01-15T17:58:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.035+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c074770 con 0x7f507c0b31d0 2024-01-15T17:58:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.036+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f507403e440 con 0x7f507c0b31d0 2024-01-15T17:58:48.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.037+0000 7f5079600640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406b310 0x7f505406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.038+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f50740ab2e0 con 0x7f507c0b31d0 2024-01-15T17:58:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.038+0000 7f5080c00640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406b310 0x7f505406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.039+0000 7f5080c00640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406b310 0x7f505406d7d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f506c001030 tx=0x7f506c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:48.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.042+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5074076530 con 0x7f507c0b31d0 2024-01-15T17:58:48.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.297+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f507c155c00 con 0x7f507c0b31d0 2024-01-15T17:58:48.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.301+0000 7f5079600640 1 -- 172.21.15.2:0/3438687684 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5074079b50 con 0x7f507c0b31d0 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406b310 msgr2=0x7f505406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406b310 0x7f505406d7d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f506c001030 tx=0x7f506c009040 comp rx=0 tx=0).stop 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 msgr2=0x7f507c1f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 0x7f507c1f6440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5074037040 tx=0x7f5074009750 comp rx=0 tx=0).stop 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 shutdown_connections 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406b310 0x7f505406d7d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b31d0 0x7f507c1f6440 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c14d2b0 0x7f507c0b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 --2- 172.21.15.2:0/3438687684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f505c004d00 0x7f507c0ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 >> 172.21.15.2:0/3438687684 conn(0x7f507c0b0840 msgr2=0x7f507c0b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:48.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 shutdown_connections 2024-01-15T17:58:48.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.304+0000 7f5082000640 1 -- 172.21.15.2:0/3438687684 wait complete. 2024-01-15T17:58:48.325 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-15T17:58:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.538+0000 7f4636e00640 1 Processor -- start 2024-01-15T17:58:48.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.538+0000 7f4636e00640 1 -- start start 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.539+0000 7f4636e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4630158ae0 0x7f46300ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4636e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 0x7f46300b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4636e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f463015ea90 con 0x7f463015b4a0 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4636e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4630158ec0 con 0x7f4630158ae0 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4636e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463015b9a0 con 0x7f46300abe10 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4636400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f463015b4a0 0x7f46300cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51986/0 (socket says 172.21.15.2:51986) 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4636400640 1 -- 172.21.15.2:0/3527499789 learned_addr learned my addr 172.21.15.2:0/3527499789 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4635a00640 1 --2- 172.21.15.2:0/3527499789 >> v2:172.21.15.137:3300/0 conn(0x7f4630158ae0 0x7f46300ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.540+0000 7f4635000640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 0x7f46300b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764466064 0 0) 0x7f463015ea90 con 0x7f463015b4a0 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4610003520 con 0x7f463015b4a0 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2952582999 0 0) 0x7f4610003520 con 0x7f463015b4a0 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 msgr2=0x7f46300b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 0x7f46300b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 >> v2:172.21.15.137:3300/0 conn(0x7f4630158ae0 msgr2=0x7f46300ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 --2- 172.21.15.2:0/3527499789 >> v2:172.21.15.137:3300/0 conn(0x7f4630158ae0 0x7f46300ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463015bc80 con 0x7f463015b4a0 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f46280019f0 con 0x7f463015b4a0 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 >> v1:172.21.15.2:6789/0 conn(0x7f463015b4a0 legacy=0x7f46300cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4610003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46100047c0 0x7f4610004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 0x7f4610005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4630158ec0 con 0x7f4610003970 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463015b9a0 con 0x7f46100047c0 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461000b980 con 0x7f46300abe10 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.541+0000 7f4635a00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4610003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635000640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46100047c0 0x7f4610004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4636400640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 0x7f4610005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635a00640 1 -- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 msgr2=0x7f4610005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635a00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46300abe10 0x7f4610005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635a00640 1 -- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46100047c0 msgr2=0x7f4610004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635a00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46100047c0 0x7f4610004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635a00640 1 -- 172.21.15.2:0/3527499789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610003520 con 0x7f4610003970 2024-01-15T17:58:48.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.542+0000 7f4635000640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46100047c0 0x7f4610004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.543+0000 7f4635a00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4610003db0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4620009af0 tx=0x7f46100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91db31d954430d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.543+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462002c710 con 0x7f4610003970 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.543+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462002c870 con 0x7f4610003970 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.543+0000 7f462fe00640 1 -- 172.21.15.2:0/3527499789 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462002cc00 con 0x7f4610003970 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636e00640 1 -- 172.21.15.2:0/3527499789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 msgr2=0x7f4610003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636e00640 1 --2- 172.21.15.2:0/3527499789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4610003db0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4620009af0 tx=0x7f46100057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636400640 1 -- 172.21.15.2:0/3527499789 reap_dead start 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636e00640 1 -- 172.21.15.2:0/3527499789 shutdown_connections 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636e00640 1 -- 172.21.15.2:0/3527499789 >> 172.21.15.2:0/3527499789 conn(0x7f463006a690 msgr2=0x7f463006aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636e00640 1 -- 172.21.15.2:0/3527499789 shutdown_connections 2024-01-15T17:58:48.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.544+0000 7f4636e00640 1 -- 172.21.15.2:0/3527499789 wait complete. 2024-01-15T17:58:48.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.546+0000 7f4636e00640 1 Processor -- start 2024-01-15T17:58:48.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.546+0000 7f4636e00640 1 -- start start 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4636e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4630146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4636e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300b3590 0x7f463014fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4636e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4630150190 0x7f46300b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4636e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4630150610 con 0x7f4610003970 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4636e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46301479b0 con 0x7f46300b3590 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4636e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46301524d0 con 0x7f4630150190 2024-01-15T17:58:48.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4635a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4630146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4635a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4630146f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36628/0 (socket says 172.21.15.2:36628) 2024-01-15T17:58:48.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.547+0000 7f4635a00640 1 -- 172.21.15.2:0/2312034138 learned_addr learned my addr 172.21.15.2:0/2312034138 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635a00640 1 -- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4630150190 msgr2=0x7f46300b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635a00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4630150190 0x7f46300b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635000640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300b3590 0x7f463014fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635a00640 1 -- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300b3590 msgr2=0x7f463014fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635a00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300b3590 0x7f463014fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635a00640 1 -- 172.21.15.2:0/2312034138 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46301527b0 con 0x7f4610003970 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4635a00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4630146f10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4620037040 tx=0x7f4620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620003810 con 0x7f4610003970 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f462002de90 con 0x7f4610003970 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4620003b70 con 0x7f4610003970 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.548+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4630152a90 con 0x7f4610003970 2024-01-15T17:58:48.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.549+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4630153050 con 0x7f4610003970 2024-01-15T17:58:48.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.550+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4630074770 con 0x7f4610003970 2024-01-15T17:58:48.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.550+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4620003cd0 con 0x7f4610003970 2024-01-15T17:58:48.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.551+0000 7f462d600640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f460806b2c0 0x7f460806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:48.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.551+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f46200ab750 con 0x7f4610003970 2024-01-15T17:58:48.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.552+0000 7f4635000640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f460806b2c0 0x7f460806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:48.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.553+0000 7f4635000640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f460806b2c0 0x7f460806d780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4630150850 tx=0x7f4618008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:48.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.556+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4620076910 con 0x7f4610003970 2024-01-15T17:58:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.824+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f463006d530 con 0x7f4610003970 2024-01-15T17:58:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.830+0000 7f462d600640 1 -- 172.21.15.2:0/2312034138 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4620079f30 con 0x7f4610003970 2024-01-15T17:58:48.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f460806b2c0 msgr2=0x7f460806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f460806b2c0 0x7f460806d780 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4630150850 tx=0x7f4618008040 comp rx=0 tx=0).stop 2024-01-15T17:58:48.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 msgr2=0x7f4630146f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:48.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4630146f10 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4620037040 tx=0x7f4620004730 comp rx=0 tx=0).stop 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 shutdown_connections 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f460806b2c0 0x7f460806d780 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4630150190 0x7f46300b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46300b3590 0x7f463014fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 --2- 172.21.15.2:0/2312034138 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4610003970 0x7f4630146f10 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 >> 172.21.15.2:0/2312034138 conn(0x7f463006a690 msgr2=0x7f4630157b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 shutdown_connections 2024-01-15T17:58:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:48.832+0000 7f4636e00640 1 -- 172.21.15.2:0/2312034138 wait complete. 2024-01-15T17:58:48.853 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-15T17:58:49.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.076+0000 7f6e95e00640 1 Processor -- start 2024-01-15T17:58:49.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.076+0000 7f6e95e00640 1 -- start start 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e95e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6e9015efc0 0x7f6e900cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e95e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9006c170 con 0x7f6e9006da40 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e95e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e900cd1b0 con 0x7f6e9015eb30 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e95e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e90072420 con 0x7f6e9015efc0 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e94a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6e9006da40 0x7f6e90071d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51998/0 (socket says 172.21.15.2:51998) 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e94a00640 1 -- 172.21.15.2:0/2501248305 learned_addr learned my addr 172.21.15.2:0/2501248305 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:49.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.078+0000 7f6e95400640 1 --2- 172.21.15.2:0/2501248305 >> v2:172.21.15.2:3300/0 conn(0x7f6e9015efc0 0x7f6e900cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.079+0000 7f6e8f400640 1 -- 172.21.15.2:0/2501248305 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 619301198 0 0) 0x7f6e9006c170 con 0x7f6e9006da40 2024-01-15T17:58:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.079+0000 7f6e8f400640 1 -- 172.21.15.2:0/2501248305 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70003520 con 0x7f6e9006da40 2024-01-15T17:58:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.079+0000 7f6e95400640 1 -- 172.21.15.2:0/2501248305 >> v1:172.21.15.137:6789/0 conn(0x7f6e9015eb30 legacy=0x7f6e900cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.079+0000 7f6e95400640 1 -- 172.21.15.2:0/2501248305 >> v1:172.21.15.2:6789/0 conn(0x7f6e9006da40 legacy=0x7f6e90071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.079+0000 7f6e95400640 1 -- 172.21.15.2:0/2501248305 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90071560 con 0x7f6e9015efc0 2024-01-15T17:58:49.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.080+0000 7f6e95400640 1 --2- 172.21.15.2:0/2501248305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015efc0 0x7f6e900cdfd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009d90 tx=0x7f6e8802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9c56c2f9989712 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:49.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.080+0000 7f6e8f400640 1 -- 172.21.15.2:0/2501248305 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8802ccf0 con 0x7f6e9015efc0 2024-01-15T17:58:49.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.080+0000 7f6e8f400640 1 -- 172.21.15.2:0/2501248305 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8802ce50 con 0x7f6e9015efc0 2024-01-15T17:58:49.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.080+0000 7f6e8f400640 1 -- 172.21.15.2:0/2501248305 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88034660 con 0x7f6e9015efc0 2024-01-15T17:58:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 -- 172.21.15.2:0/2501248305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015efc0 msgr2=0x7f6e900cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2501248305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015efc0 0x7f6e900cdfd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009d90 tx=0x7f6e8802bf40 comp rx=0 tx=0).stop 2024-01-15T17:58:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 -- 172.21.15.2:0/2501248305 shutdown_connections 2024-01-15T17:58:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2501248305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015efc0 0x7f6e900cdfd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 -- 172.21.15.2:0/2501248305 >> 172.21.15.2:0/2501248305 conn(0x7f6e900b0840 msgr2=0x7f6e900b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:49.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 -- 172.21.15.2:0/2501248305 shutdown_connections 2024-01-15T17:58:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.081+0000 7f6e95e00640 1 -- 172.21.15.2:0/2501248305 wait complete. 2024-01-15T17:58:49.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.083+0000 7f6e95e00640 1 Processor -- start 2024-01-15T17:58:49.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.083+0000 7f6e95e00640 1 -- start start 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e95e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e9006da40 0x7f6e900cec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e95e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 0x7f6e90156680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e95e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9015efc0 0x7f6e901583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e95e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90156bc0 con 0x7f6e9015eb30 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e95e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9015c5a0 con 0x7f6e9006da40 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e95e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9015c880 con 0x7f6e9015efc0 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e8fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 0x7f6e90156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e8fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 0x7f6e90156680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36654/0 (socket says 172.21.15.2:36654) 2024-01-15T17:58:49.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.084+0000 7f6e8fe00640 1 -- 172.21.15.2:0/2958546293 learned_addr learned my addr 172.21.15.2:0/2958546293 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8fe00640 1 -- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9015efc0 msgr2=0x7f6e901583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8fe00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9015efc0 0x7f6e901583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e94a00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e9006da40 0x7f6e900cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8fe00640 1 -- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e9006da40 msgr2=0x7f6e900cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8fe00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e9006da40 0x7f6e900cec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8fe00640 1 -- 172.21.15.2:0/2958546293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88009a70 con 0x7f6e9015eb30 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8fe00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 0x7f6e90156680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800c450 tx=0x7f6e7800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780052d0 con 0x7f6e9015eb30 2024-01-15T17:58:49.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78005430 con 0x7f6e9015eb30 2024-01-15T17:58:49.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78012870 con 0x7f6e9015eb30 2024-01-15T17:58:49.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9015cb60 con 0x7f6e9015eb30 2024-01-15T17:58:49.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.085+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9015d120 con 0x7f6e9015eb30 2024-01-15T17:58:49.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.086+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e900cfa50 con 0x7f6e9015eb30 2024-01-15T17:58:49.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.087+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6e780129d0 con 0x7f6e9015eb30 2024-01-15T17:58:49.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.088+0000 7f6e8cc00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806b2c0 0x7f6e6806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.088+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6e78089460 con 0x7f6e9015eb30 2024-01-15T17:58:49.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.089+0000 7f6e94a00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806b2c0 0x7f6e6806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.090+0000 7f6e94a00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806b2c0 0x7f6e6806d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e800011e0 tx=0x7f6e80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:49.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.093+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e78054580 con 0x7f6e9015eb30 2024-01-15T17:58:49.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.347+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f6e90073af0 con 0x7f6e9015eb30 2024-01-15T17:58:49.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.352+0000 7f6e8cc00640 1 -- 172.21.15.2:0/2958546293 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6e78057ba0 con 0x7f6e9015eb30 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.355+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806b2c0 msgr2=0x7f6e6806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.355+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806b2c0 0x7f6e6806d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e800011e0 tx=0x7f6e80009040 comp rx=0 tx=0).stop 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.355+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 msgr2=0x7f6e90156680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.355+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 0x7f6e90156680 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800c450 tx=0x7f6e7800c910 comp rx=0 tx=0).stop 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 shutdown_connections 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806b2c0 0x7f6e6806d780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9015efc0 0x7f6e901583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015eb30 0x7f6e90156680 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 --2- 172.21.15.2:0/2958546293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e9006da40 0x7f6e900cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 >> 172.21.15.2:0/2958546293 conn(0x7f6e900b0840 msgr2=0x7f6e900add40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 shutdown_connections 2024-01-15T17:58:49.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.356+0000 7f6e95e00640 1 -- 172.21.15.2:0/2958546293 wait complete. 2024-01-15T17:58:49.377 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-15T17:58:49.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.581+0000 7f885ea00640 1 Processor -- start 2024-01-15T17:58:49.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.581+0000 7f885ea00640 1 -- start start 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f885ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8858143850 0x7f885815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f885ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8858160f60 0x7f885806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f885ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f885806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f885ea00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f885815e970 con 0x7f8858160f60 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f885ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88581619b0 con 0x7f8858143850 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f885ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8858161c90 con 0x7f8858161410 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f8857400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8858160f60 0x7f885806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f8857400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8858160f60 0x7f885806c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36670/0 (socket says 172.21.15.2:36670) 2024-01-15T17:58:49.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.583+0000 7f8857400640 1 -- 172.21.15.2:0/3061059978 learned_addr learned my addr 172.21.15.2:0/3061059978 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:49.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857e00640 1 --2- 172.21.15.2:0/3061059978 >> v2:172.21.15.137:3300/0 conn(0x7f8858143850 0x7f885815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857400640 1 -- 172.21.15.2:0/3061059978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 msgr2=0x7f885806cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:49.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f885ca00640 1 --2- 172.21.15.2:0/3061059978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f885806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857400640 1 --2- 172.21.15.2:0/3061059978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f885806cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857400640 1 -- 172.21.15.2:0/3061059978 >> v2:172.21.15.137:3300/0 conn(0x7f8858143850 msgr2=0x7f885815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:49.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857400640 1 --2- 172.21.15.2:0/3061059978 >> v2:172.21.15.137:3300/0 conn(0x7f8858143850 0x7f885815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857400640 1 -- 172.21.15.2:0/3061059978 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f885815dc30 con 0x7f8858160f60 2024-01-15T17:58:49.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.584+0000 7f8857e00640 1 --2- 172.21.15.2:0/3061059978 >> v2:172.21.15.137:3300/0 conn(0x7f8858143850 0x7f885815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:49.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.585+0000 7f8857400640 1 --2- 172.21.15.2:0/3061059978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858160f60 0x7f885806c8b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f88580acfb0 tx=0x7f884002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c52c7830c0820c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:49.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.585+0000 7f8856a00640 1 -- 172.21.15.2:0/3061059978 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840031740 con 0x7f8858160f60 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.585+0000 7f8856a00640 1 -- 172.21.15.2:0/3061059978 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88400318a0 con 0x7f8858160f60 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.585+0000 7f8856a00640 1 -- 172.21.15.2:0/3061059978 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8840031c30 con 0x7f8858160f60 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 -- 172.21.15.2:0/3061059978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858160f60 msgr2=0x7f885806c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 --2- 172.21.15.2:0/3061059978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858160f60 0x7f885806c8b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f88580acfb0 tx=0x7f884002b7e0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 -- 172.21.15.2:0/3061059978 shutdown_connections 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 --2- 172.21.15.2:0/3061059978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f885806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 --2- 172.21.15.2:0/3061059978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858160f60 0x7f885806c8b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 --2- 172.21.15.2:0/3061059978 >> v2:172.21.15.137:3300/0 conn(0x7f8858143850 0x7f885815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 -- 172.21.15.2:0/3061059978 >> 172.21.15.2:0/3061059978 conn(0x7f88580b46d0 msgr2=0x7f88580b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 -- 172.21.15.2:0/3061059978 shutdown_connections 2024-01-15T17:58:49.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.586+0000 7f885ea00640 1 -- 172.21.15.2:0/3061059978 wait complete. 2024-01-15T17:58:49.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.588+0000 7f885ea00640 1 Processor -- start 2024-01-15T17:58:49.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.588+0000 7f885ea00640 1 -- start start 2024-01-15T17:58:49.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f885ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 0x7f88580b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f885ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8858160f60 0x7f88580c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f885ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f8858131ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f885ea00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88580c5c40 con 0x7f8858143850 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f885ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88581f2920 con 0x7f8858160f60 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f885ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88580c5030 con 0x7f8858161410 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f8857e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 0x7f88580b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f8857e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 0x7f88580b1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36676/0 (socket says 172.21.15.2:36676) 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.589+0000 7f8857e00640 1 -- 172.21.15.2:0/3962807774 learned_addr learned my addr 172.21.15.2:0/3962807774 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f8857e00640 1 -- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 msgr2=0x7f8858131ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f8857e00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f8858131ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f8857e00640 1 -- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8858160f60 msgr2=0x7f88580c90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f8857e00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8858160f60 0x7f88580c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f8857e00640 1 -- 172.21.15.2:0/3962807774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88400094d0 con 0x7f8858143850 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f8857e00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 0x7f88580b1fb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f884800c7f0 tx=0x7f884800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848007820 con 0x7f8858143850 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8848007980 con 0x7f8858143850 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8848007cf0 con 0x7f8858143850 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.590+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88580c5310 con 0x7f8858143850 2024-01-15T17:58:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.591+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88580c58d0 con 0x7f8858143850 2024-01-15T17:58:49.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.592+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88580686f0 con 0x7f8858143850 2024-01-15T17:58:49.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.593+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f88480113f0 con 0x7f8858143850 2024-01-15T17:58:49.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.593+0000 7f884fe00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f883006b2c0 0x7f883006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:49.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.594+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8848089ba0 con 0x7f8858143850 2024-01-15T17:58:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.594+0000 7f8857400640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f883006b2c0 0x7f883006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:49.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.596+0000 7f8857400640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f883006b2c0 0x7f883006d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8840035000 tx=0x7f8840030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:49.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.598+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8848054d50 con 0x7f8858143850 2024-01-15T17:58:49.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.870+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f88580b4d00 con 0x7f8858143850 2024-01-15T17:58:49.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.874+0000 7f884fe00640 1 -- 172.21.15.2:0/3962807774 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f884801f0a0 con 0x7f8858143850 2024-01-15T17:58:49.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f883006b2c0 msgr2=0x7f883006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f883006b2c0 0x7f883006d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f8840035000 tx=0x7f8840030000 comp rx=0 tx=0).stop 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 msgr2=0x7f88580b1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 0x7f88580b1fb0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f884800c7f0 tx=0x7f884800ccb0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 shutdown_connections 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f883006b2c0 0x7f883006d780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8858161410 0x7f8858131ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8858160f60 0x7f88580c90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.877+0000 7f885ea00640 1 --2- 172.21.15.2:0/3962807774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8858143850 0x7f88580b1fb0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:49.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.878+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 >> 172.21.15.2:0/3962807774 conn(0x7f88580b46d0 msgr2=0x7f8858142b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:49.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.878+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 shutdown_connections 2024-01-15T17:58:49.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:49.878+0000 7f885ea00640 1 -- 172.21.15.2:0/3962807774 wait complete. 2024-01-15T17:58:49.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-15T17:58:50.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.120+0000 7f9fd3600640 1 Processor -- start 2024-01-15T17:58:50.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.121+0000 7f9fd3600640 1 -- start start 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.122+0000 7f9fd3600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9fcc15d670 0x7f9fcc0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.122+0000 7f9fd3600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9fcc0ce2b0 0x7f9fcc0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.122+0000 7f9fd3600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc0ce930 con 0x7f9fcc0d2f10 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.122+0000 7f9fd3600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fcc0ce740 con 0x7f9fcc15d670 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.122+0000 7f9fd3600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fcc0cec10 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9fcc15d670 0x7f9fcc0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9fcc15d670 0x7f9fcc0cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36692/0 (socket says 172.21.15.2:36692) 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 -- 172.21.15.2:0/1741493148 learned_addr learned my addr 172.21.15.2:0/1741493148 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fd1600640 1 --2- 172.21.15.2:0/1741493148 >> v2:172.21.15.137:3300/0 conn(0x7f9fcc0ce2b0 0x7f9fcc0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 -- 172.21.15.2:0/1741493148 >> v2:172.21.15.137:3300/0 conn(0x7f9fcc0ce2b0 msgr2=0x7f9fcc0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 --2- 172.21.15.2:0/1741493148 >> v2:172.21.15.137:3300/0 conn(0x7f9fcc0ce2b0 0x7f9fcc0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 -- 172.21.15.2:0/1741493148 >> v1:172.21.15.137:6789/0 conn(0x7f9fcc0d2f10 legacy=0x7f9fcc0d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.123+0000 7f9fc3e00640 1 -- 172.21.15.2:0/1741493148 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc0ceef0 con 0x7f9fcc15d670 2024-01-15T17:58:50.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.124+0000 7f9fc3e00640 1 --2- 172.21.15.2:0/1741493148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc15d670 0x7f9fcc0cdd70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc160ec0 tx=0x7f9fb402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15b23df67b910c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.125+0000 7f9fcbe00640 1 -- 172.21.15.2:0/1741493148 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4031640 con 0x7f9fcc15d670 2024-01-15T17:58:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.125+0000 7f9fcbe00640 1 -- 172.21.15.2:0/1741493148 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb40317a0 con 0x7f9fcc15d670 2024-01-15T17:58:50.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.125+0000 7f9fcbe00640 1 -- 172.21.15.2:0/1741493148 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4031b30 con 0x7f9fcc15d670 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.125+0000 7f9fd3600640 1 -- 172.21.15.2:0/1741493148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc15d670 msgr2=0x7f9fcc0cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.125+0000 7f9fd3600640 1 --2- 172.21.15.2:0/1741493148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc15d670 0x7f9fcc0cdd70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc160ec0 tx=0x7f9fb402b8f0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.126+0000 7f9fd3600640 1 -- 172.21.15.2:0/1741493148 shutdown_connections 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.126+0000 7f9fd3600640 1 --2- 172.21.15.2:0/1741493148 >> v2:172.21.15.137:3300/0 conn(0x7f9fcc0ce2b0 0x7f9fcc0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.126+0000 7f9fd3600640 1 --2- 172.21.15.2:0/1741493148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc15d670 0x7f9fcc0cdd70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.126+0000 7f9fd3600640 1 -- 172.21.15.2:0/1741493148 >> 172.21.15.2:0/1741493148 conn(0x7f9fcc08cc30 msgr2=0x7f9fcc09c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.126+0000 7f9fd3600640 1 -- 172.21.15.2:0/1741493148 shutdown_connections 2024-01-15T17:58:50.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.126+0000 7f9fd3600640 1 -- 172.21.15.2:0/1741493148 wait complete. 2024-01-15T17:58:50.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.127+0000 7f9fd3600640 1 Processor -- start 2024-01-15T17:58:50.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.128+0000 7f9fd3600640 1 -- start start 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.128+0000 7f9fd3600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 0x7f9fcc15da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.128+0000 7f9fd3600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0d2f10 0x7f9fcc0cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd3600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9fcc1549a0 0x7f9fcc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd3600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0cbd90 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd3600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc0caa90 con 0x7f9fcc0d2f10 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd3600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc154e20 con 0x7f9fcc1549a0 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 0x7f9fcc15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 0x7f9fcc15da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36694/0 (socket says 172.21.15.2:36694) 2024-01-15T17:58:50.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 -- 172.21.15.2:0/3471424879 learned_addr learned my addr 172.21.15.2:0/3471424879 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:50.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fc3e00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0d2f10 0x7f9fcc0cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 -- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9fcc1549a0 msgr2=0x7f9fcc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd1600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9fcc1549a0 0x7f9fcc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9fcc1549a0 0x7f9fcc15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 -- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0d2f10 msgr2=0x7f9fcc0cb850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0d2f10 0x7f9fcc0cb850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd0c00640 1 -- 172.21.15.2:0/3471424879 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb40094d0 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fc3e00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0d2f10 0x7f9fcc0cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.129+0000 7f9fd1600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9fcc1549a0 0x7f9fcc15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.130+0000 7f9fd0c00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 0x7f9fcc15da60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c430 tx=0x7f9fbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.130+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc011790 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.130+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc0118f0 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.130+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc011ce0 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.130+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc15e040 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.130+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc15e600 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.131+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc0686f0 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.132+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9fbc01c460 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.133+0000 7f9fc9600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9fa406b2c0 0x7f9fa406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.133+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9fbc0898f0 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.134+0000 7f9fc3e00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9fa406b2c0 0x7f9fa406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.135+0000 7f9fc3e00640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9fa406b2c0 0x7f9fa406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4036000 tx=0x7f9fb403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:50.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.138+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fbc054a10 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.402+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f9fcc06c4d0 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.406+0000 7f9fc9600640 1 -- 172.21.15.2:0/3471424879 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f9fbc058030 con 0x7f9fcc0ce2b0 2024-01-15T17:58:50.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.409+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9fa406b2c0 msgr2=0x7f9fa406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.409+0000 7f9fd3600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9fa406b2c0 0x7f9fa406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9fb4036000 tx=0x7f9fb403c040 comp rx=0 tx=0).stop 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.409+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 msgr2=0x7f9fcc15da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.409+0000 7f9fd3600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 0x7f9fcc15da60 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc00c430 tx=0x7f9fbc00c8f0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 shutdown_connections 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9fa406b2c0 0x7f9fa406d780 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9fcc1549a0 0x7f9fcc15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9fcc0d2f10 0x7f9fcc0cb850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 --2- 172.21.15.2:0/3471424879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9fcc0ce2b0 0x7f9fcc15da60 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 >> 172.21.15.2:0/3471424879 conn(0x7f9fcc08cc30 msgr2=0x7f9fcc15a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 shutdown_connections 2024-01-15T17:58:50.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.410+0000 7f9fd3600640 1 -- 172.21.15.2:0/3471424879 wait complete. 2024-01-15T17:58:50.432 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:50.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.650+0000 7feb85e00640 1 Processor -- start 2024-01-15T17:58:50.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.650+0000 7feb85e00640 1 -- start start 2024-01-15T17:58:50.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.652+0000 7feb85e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 0x7feb800b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.652+0000 7feb85e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb800b2950 con 0x7feb800aeaf0 2024-01-15T17:58:50.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.652+0000 7feb85e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8015f970 con 0x7feb8006f270 2024-01-15T17:58:50.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.652+0000 7feb85e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb8015fc50 con 0x7feb8015f390 2024-01-15T17:58:50.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.652+0000 7feb7ea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7feb800aeaf0 0x7feb8006eb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52004/0 (socket says 172.21.15.2:52004) 2024-01-15T17:58:50.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.652+0000 7feb7ea00640 1 -- 172.21.15.2:0/1586781886 learned_addr learned my addr 172.21.15.2:0/1586781886 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.653+0000 7feb7f400640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 0x7feb800b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.653+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076908952 0 0) 0x7feb800b2950 con 0x7feb800aeaf0 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.653+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb60003520 con 0x7feb800aeaf0 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4114411045 0 0) 0x7feb60003520 con 0x7feb800aeaf0 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 msgr2=0x7feb800b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 0x7feb800b33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 >> v1:172.21.15.137:6789/0 conn(0x7feb8006f270 legacy=0x7feb8015ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb800b2bf0 con 0x7feb800aeaf0 2024-01-15T17:58:50.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7feb70001670 con 0x7feb800aeaf0 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 >> v1:172.21.15.2:6789/0 conn(0x7feb800aeaf0 legacy=0x7feb8006eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 0x7feb600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb60004d00 0x7feb60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb8015fc50 con 0x7feb600039a0 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6000ba30 con 0x7feb60004d00 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6000bbf0 con 0x7feb8015f390 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7ea00640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7e000640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 0x7feb600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.654+0000 7feb7f400640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb60004d00 0x7feb60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.655+0000 7feb7ea00640 1 -- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 msgr2=0x7feb600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.655+0000 7feb7ea00640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb8015f390 0x7feb600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.655+0000 7feb7ea00640 1 -- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb60004d00 msgr2=0x7feb60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:50.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.655+0000 7feb7ea00640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb60004d00 0x7feb60005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.655+0000 7feb7ea00640 1 -- 172.21.15.2:0/1586781886 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb60003520 con 0x7feb600039a0 2024-01-15T17:58:50.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.655+0000 7feb7f400640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb60004d00 0x7feb60005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:50.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.656+0000 7feb7ea00640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb60003dc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c210 tx=0x7feb600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f0d35f71cbf11 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:50.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.656+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70005ad0 con 0x7feb600039a0 2024-01-15T17:58:50.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.656+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb70005c30 con 0x7feb600039a0 2024-01-15T17:58:50.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.656+0000 7feb7d600640 1 -- 172.21.15.2:0/1586781886 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7002c890 con 0x7feb600039a0 2024-01-15T17:58:50.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb85e00640 1 -- 172.21.15.2:0/1586781886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 msgr2=0x7feb60003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb85e00640 1 --2- 172.21.15.2:0/1586781886 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb60003dc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7feb7000c210 tx=0x7feb600057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb7f400640 1 -- 172.21.15.2:0/1586781886 reap_dead start 2024-01-15T17:58:50.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb85e00640 1 -- 172.21.15.2:0/1586781886 shutdown_connections 2024-01-15T17:58:50.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb85e00640 1 -- 172.21.15.2:0/1586781886 >> 172.21.15.2:0/1586781886 conn(0x7feb80070fc0 msgr2=0x7feb800713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:50.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb85e00640 1 -- 172.21.15.2:0/1586781886 shutdown_connections 2024-01-15T17:58:50.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.657+0000 7feb85e00640 1 -- 172.21.15.2:0/1586781886 wait complete. 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.659+0000 7feb85e00640 1 Processor -- start 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.659+0000 7feb85e00640 1 -- start start 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb85e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb8006a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb85e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb80158e50 0x7feb801592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb85e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb80143c90 0x7feb80144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb85e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb80144650 con 0x7feb600039a0 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb85e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb80159910 con 0x7feb80158e50 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb85e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb80159bf0 con 0x7feb80143c90 2024-01-15T17:58:50.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb7ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb8006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb7ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb8006a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36710/0 (socket says 172.21.15.2:36710) 2024-01-15T17:58:50.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.660+0000 7feb7ea00640 1 -- 172.21.15.2:0/3118617126 learned_addr learned my addr 172.21.15.2:0/3118617126 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7ea00640 1 -- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb80143c90 msgr2=0x7feb80144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7f400640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb80143c90 0x7feb80144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7e000640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb80158e50 0x7feb801592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7ea00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb80143c90 0x7feb80144110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7ea00640 1 -- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb80158e50 msgr2=0x7feb801592d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7ea00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb80158e50 0x7feb801592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7ea00640 1 -- 172.21.15.2:0/3118617126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb80159ed0 con 0x7feb600039a0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb7ea00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb8006a720 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7feb70000c00 tx=0x7feb70001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb70039070 con 0x7feb600039a0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.661+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb7002ccc0 con 0x7feb600039a0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.662+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb7003c040 con 0x7feb600039a0 2024-01-15T17:58:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.662+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb8015a1b0 con 0x7feb600039a0 2024-01-15T17:58:50.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.662+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8006ac60 con 0x7feb600039a0 2024-01-15T17:58:50.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.663+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb8006adf0 con 0x7feb600039a0 2024-01-15T17:58:50.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.664+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feb70032070 con 0x7feb600039a0 2024-01-15T17:58:50.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.664+0000 7feb77400640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feb5806b2c0 0x7feb5806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:50.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.665+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7feb700aab30 con 0x7feb600039a0 2024-01-15T17:58:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.665+0000 7feb7e000640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feb5806b2c0 0x7feb5806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:50.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.666+0000 7feb7e000640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feb5806b2c0 0x7feb5806d780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb68007fd0 tx=0x7feb68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:50.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.669+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb70075d80 con 0x7feb600039a0 2024-01-15T17:58:50.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.867+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb800686f0 con 0x7feb600039a0 2024-01-15T17:58:50.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.869+0000 7feb77400640 1 -- 172.21.15.2:0/3118617126 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12797 (secure 0 0 0) 0x7feb700793a0 con 0x7feb600039a0 2024-01-15T17:58:50.872 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:50.872 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":31,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:46.714668+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.871+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feb5806b2c0 msgr2=0x7feb5806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feb5806b2c0 0x7feb5806d780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feb68007fd0 tx=0x7feb68009040 comp rx=0 tx=0).stop 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 msgr2=0x7feb8006a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb8006a720 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7feb70000c00 tx=0x7feb70001a00 comp rx=0 tx=0).stop 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 shutdown_connections 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feb5806b2c0 0x7feb5806d780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb80143c90 0x7feb80144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb80158e50 0x7feb801592d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 --2- 172.21.15.2:0/3118617126 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb600039a0 0x7feb8006a720 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:50.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 >> 172.21.15.2:0/3118617126 conn(0x7feb80070fc0 msgr2=0x7feb800713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:50.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 shutdown_connections 2024-01-15T17:58:50.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:50.872+0000 7feb85e00640 1 -- 172.21.15.2:0/3118617126 wait complete. 2024-01-15T17:58:50.896 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T17:58:50.896 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T17:58:51.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.120+0000 7f04c5a00640 1 Processor -- start 2024-01-15T17:58:51.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.120+0000 7f04c5a00640 1 -- start start 2024-01-15T17:58:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.122+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 0x7f04c0164750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.122+0000 7f04c5a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c0071250 con 0x7f04c0150890 2024-01-15T17:58:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.122+0000 7f04c5a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c0071530 con 0x7f04c00b1290 2024-01-15T17:58:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.122+0000 7f04c5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00b2ae0 con 0x7f04c0070c70 2024-01-15T17:58:51.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f04c0150890 0x7f04c00b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52020/0 (socket says 172.21.15.2:52020) 2024-01-15T17:58:51.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bea00640 1 -- 172.21.15.2:0/482142923 learned_addr learned my addr 172.21.15.2:0/482142923 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967811575 0 0) 0x7f04c0071250 con 0x7f04c0150890 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04a0003440 con 0x7f04c0150890 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bf400640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 0x7f04c0164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182825059 0 0) 0x7f04a0003440 con 0x7f04c0150890 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 msgr2=0x7f04c0164750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bd600640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 0x7f04c0164750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.123+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 >> v1:172.21.15.137:6789/0 conn(0x7f04c00b1290 legacy=0x7f04c0070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00b2dc0 con 0x7f04c0150890 2024-01-15T17:58:51.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f04b0001670 con 0x7f04c0150890 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 >> v1:172.21.15.2:6789/0 conn(0x7f04c0150890 legacy=0x7f04c00b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04a0003a90 0x7f04a0003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04a0008850 0x7f04a000ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 0x7f04a000d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00b2ae0 con 0x7f04a0008850 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a000dce0 con 0x7f04a0003a90 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a000dea0 con 0x7f04c0070c70 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04be000640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04a0008850 0x7f04a000ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.124+0000 7f04bf400640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 0x7f04a000d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.125+0000 7f04be000640 1 -- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 msgr2=0x7f04a000d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.125+0000 7f04be000640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04c0070c70 0x7f04a000d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.125+0000 7f04be000640 1 -- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04a0003a90 msgr2=0x7f04a0003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.125+0000 7f04be000640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04a0003a90 0x7f04a0003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.125+0000 7f04be000640 1 -- 172.21.15.2:0/482142923 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a0003440 con 0x7f04a0008850 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.126+0000 7f04be000640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04a0008850 0x7f04a000ac60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f04a8001e50 tx=0x7f04a00036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e23478223ff970 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.126+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8003b70 con 0x7f04a0008850 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.126+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8003cd0 con 0x7f04a0008850 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.126+0000 7f04bd600640 1 -- 172.21.15.2:0/482142923 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8005310 con 0x7f04a0008850 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04c5a00640 1 -- 172.21.15.2:0/482142923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04a0008850 msgr2=0x7f04a000ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04c5a00640 1 --2- 172.21.15.2:0/482142923 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04a0008850 0x7f04a000ac60 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f04a8001e50 tx=0x7f04a00036f0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04bf400640 1 -- 172.21.15.2:0/482142923 reap_dead start 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04c5a00640 1 -- 172.21.15.2:0/482142923 shutdown_connections 2024-01-15T17:58:51.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04c5a00640 1 -- 172.21.15.2:0/482142923 >> 172.21.15.2:0/482142923 conn(0x7f04c00b45f0 msgr2=0x7f04c00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04c5a00640 1 -- 172.21.15.2:0/482142923 shutdown_connections 2024-01-15T17:58:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.127+0000 7f04c5a00640 1 -- 172.21.15.2:0/482142923 wait complete. 2024-01-15T17:58:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.129+0000 7f04c5a00640 1 Processor -- start 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.129+0000 7f04c5a00640 1 -- start start 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04a0008850 0x7f04c01f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04c0150df0 0x7f04c0151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04c5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 0x7f04c00ce210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04c5a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00ce750 con 0x7f04c00cdd90 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04c5a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c0152570 con 0x7f04c0150df0 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04c5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01f1f20 con 0x7f04a0008850 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04bf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 0x7f04c00ce210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04bf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 0x7f04c00ce210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36736/0 (socket says 172.21.15.2:36736) 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04bf400640 1 -- 172.21.15.2:0/785574335 learned_addr learned my addr 172.21.15.2:0/785574335 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04bea00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04a0008850 0x7f04c01f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04be000640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04c0150df0 0x7f04c0151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04bf400640 1 -- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04a0008850 msgr2=0x7f04c01f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.130+0000 7f04bf400640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04a0008850 0x7f04c01f19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04bf400640 1 -- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04c0150df0 msgr2=0x7f04c0151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04bf400640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04c0150df0 0x7f04c0151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04bf400640 1 -- 172.21.15.2:0/785574335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00cd140 con 0x7f04c00cdd90 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04be000640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04c0150df0 0x7f04c0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04bf400640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 0x7f04c00ce210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f04c006f9f0 tx=0x7f04b8010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800e840 con 0x7f04c00cdd90 2024-01-15T17:58:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b800e9c0 con 0x7f04c00cdd90 2024-01-15T17:58:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c0151960 con 0x7f04c00cdd90 2024-01-15T17:58:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b800ed70 con 0x7f04c00cdd90 2024-01-15T17:58:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.131+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c0151f20 con 0x7f04c00cdd90 2024-01-15T17:58:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.132+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c00686f0 con 0x7f04c00cdd90 2024-01-15T17:58:51.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.133+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f04b8020410 con 0x7f04c00cdd90 2024-01-15T17:58:51.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.134+0000 7f04b6a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f049806b310 0x7f049806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.134+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f04b8089210 con 0x7f04c00cdd90 2024-01-15T17:58:51.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.135+0000 7f04bea00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f049806b310 0x7f049806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.136+0000 7f04bea00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f049806b310 0x7f049806d7d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04b00017a0 tx=0x7f04b0001590 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.139+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04b80543d0 con 0x7f04c00cdd90 2024-01-15T17:58:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.336+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04c00b2560 con 0x7f04c00cdd90 2024-01-15T17:58:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.336+0000 7f04b6a00640 1 -- 172.21.15.2:0/785574335 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f04b80579f0 con 0x7f04c00cdd90 2024-01-15T17:58:51.340 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f049806b310 msgr2=0x7f049806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f049806b310 0x7f049806d7d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04b00017a0 tx=0x7f04b0001590 comp rx=0 tx=0).stop 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 msgr2=0x7f04c00ce210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 0x7f04c00ce210 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f04c006f9f0 tx=0x7f04b8010c20 comp rx=0 tx=0).stop 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 shutdown_connections 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f049806b310 0x7f049806d7d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04c00cdd90 0x7f04c00ce210 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04c0150df0 0x7f04c0151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 --2- 172.21.15.2:0/785574335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04a0008850 0x7f04c01f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.339+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 >> 172.21.15.2:0/785574335 conn(0x7f04c00b45f0 msgr2=0x7f04c014fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.340+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 shutdown_connections 2024-01-15T17:58:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.340+0000 7f04c5a00640 1 -- 172.21.15.2:0/785574335 wait complete. 2024-01-15T17:58:51.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T17:58:51.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.580+0000 7f99eac00640 1 Processor -- start 2024-01-15T17:58:51.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.581+0000 7f99eac00640 1 -- start start 2024-01-15T17:58:51.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.582+0000 7f99eac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e414af80 0x7f99e406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.582+0000 7f99eac00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f99e406da60 0x7f99e4161c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.582+0000 7f99eac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f99e41621d0 0x7f99e40700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.582+0000 7f99eac00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e40b28f0 con 0x7f99e406da60 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.582+0000 7f99eac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e40b1c70 con 0x7f99e41621d0 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.582+0000 7f99eac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40b1f50 con 0x7f99e414af80 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f99e406da60 0x7f99e4161c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f99e406da60 0x7f99e4161c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36744/0 (socket says 172.21.15.2:36744) 2024-01-15T17:58:51.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 -- 172.21.15.2:0/3132455753 learned_addr learned my addr 172.21.15.2:0/3132455753 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e9800640 1 --2- 172.21.15.2:0/3132455753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e414af80 0x7f99e406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99ea200640 1 --2- 172.21.15.2:0/3132455753 >> v2:172.21.15.137:3300/0 conn(0x7f99e41621d0 0x7f99e40700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 -- 172.21.15.2:0/3132455753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e414af80 msgr2=0x7f99e406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 --2- 172.21.15.2:0/3132455753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e414af80 0x7f99e406d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 -- 172.21.15.2:0/3132455753 >> v2:172.21.15.137:3300/0 conn(0x7f99e41621d0 msgr2=0x7f99e40700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 --2- 172.21.15.2:0/3132455753 >> v2:172.21.15.137:3300/0 conn(0x7f99e41621d0 0x7f99e40700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99e8e00640 1 -- 172.21.15.2:0/3132455753 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e40b2230 con 0x7f99e406da60 2024-01-15T17:58:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.583+0000 7f99ea200640 1 --2- 172.21.15.2:0/3132455753 >> v2:172.21.15.137:3300/0 conn(0x7f99e41621d0 0x7f99e40700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:51.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.585+0000 7f99e8e00640 1 --2- 172.21.15.2:0/3132455753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e406da60 0x7f99e4161c90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f99e406e560 tx=0x7f99cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec303ae9741a4867 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.585+0000 7f99e3e00640 1 -- 172.21.15.2:0/3132455753 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0317e0 con 0x7f99e406da60 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.585+0000 7f99e3e00640 1 -- 172.21.15.2:0/3132455753 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc031940 con 0x7f99e406da60 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.585+0000 7f99e3e00640 1 -- 172.21.15.2:0/3132455753 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc031cd0 con 0x7f99e406da60 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.585+0000 7f99eac00640 1 -- 172.21.15.2:0/3132455753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e406da60 msgr2=0x7f99e4161c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.585+0000 7f99eac00640 1 --2- 172.21.15.2:0/3132455753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e406da60 0x7f99e4161c90 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f99e406e560 tx=0x7f99cc02b880 comp rx=0 tx=0).stop 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 -- 172.21.15.2:0/3132455753 shutdown_connections 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 --2- 172.21.15.2:0/3132455753 >> v2:172.21.15.137:3300/0 conn(0x7f99e41621d0 0x7f99e40700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 --2- 172.21.15.2:0/3132455753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e406da60 0x7f99e4161c90 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 --2- 172.21.15.2:0/3132455753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e414af80 0x7f99e406d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 -- 172.21.15.2:0/3132455753 >> 172.21.15.2:0/3132455753 conn(0x7f99e40b0840 msgr2=0x7f99e40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 -- 172.21.15.2:0/3132455753 shutdown_connections 2024-01-15T17:58:51.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.586+0000 7f99eac00640 1 -- 172.21.15.2:0/3132455753 wait complete. 2024-01-15T17:58:51.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.587+0000 7f99eac00640 1 Processor -- start 2024-01-15T17:58:51.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.588+0000 7f99eac00640 1 -- start start 2024-01-15T17:58:51.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.588+0000 7f99eac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e406da60 0x7f99e41f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99eac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 0x7f99e40a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99eac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e41621d0 0x7f99e41459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99eac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e40ae300 con 0x7f99e414af80 2024-01-15T17:58:51.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99eac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e41fb590 con 0x7f99e41621d0 2024-01-15T17:58:51.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99eac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e4146860 con 0x7f99e406da60 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 0x7f99e40a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 0x7f99e40a98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36760/0 (socket says 172.21.15.2:36760) 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 -- 172.21.15.2:0/516329059 learned_addr learned my addr 172.21.15.2:0/516329059 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e9800640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e406da60 0x7f99e41f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99ea200640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e41621d0 0x7f99e41459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 -- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e406da60 msgr2=0x7f99e41f6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e406da60 0x7f99e41f6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 -- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e41621d0 msgr2=0x7f99e41459f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e41621d0 0x7f99e41459f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.589+0000 7f99e8e00640 1 -- 172.21.15.2:0/516329059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99cc0094d0 con 0x7f99e414af80 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.590+0000 7f99e8e00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 0x7f99e40a98f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f99cc000c00 tx=0x7f99cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.590+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc032880 con 0x7f99e414af80 2024-01-15T17:58:51.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.590+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc032a20 con 0x7f99e414af80 2024-01-15T17:58:51.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.590+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e40a9e30 con 0x7f99e414af80 2024-01-15T17:58:51.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.590+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc032dd0 con 0x7f99e414af80 2024-01-15T17:58:51.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.590+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e41facf0 con 0x7f99e414af80 2024-01-15T17:58:51.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.592+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e4074770 con 0x7f99e414af80 2024-01-15T17:58:51.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.592+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f99cc04e450 con 0x7f99e414af80 2024-01-15T17:58:51.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.593+0000 7f99e1600640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f99bc06b2c0 0x7f99bc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:51.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.593+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f99cc0ab930 con 0x7f99e414af80 2024-01-15T17:58:51.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.594+0000 7f99e9800640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f99bc06b2c0 0x7f99bc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:51.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.595+0000 7f99e9800640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f99bc06b2c0 0x7f99bc06d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99d4004410 tx=0x7f99d40039f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:51.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.598+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99cc076c40 con 0x7f99e414af80 2024-01-15T17:58:51.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.793+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f99e40b0f00 con 0x7f99e414af80 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.794+0000 7f99e1600640 1 -- 172.21.15.2:0/516329059 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7f99cc07a260 con 0x7f99e414af80 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T17:58:51.797 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:58:51.798 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.796+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f99bc06b2c0 msgr2=0x7f99bc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.796+0000 7f99eac00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f99bc06b2c0 0x7f99bc06d780 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f99d4004410 tx=0x7f99d40039f0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.796+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 msgr2=0x7f99e40a98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.796+0000 7f99eac00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 0x7f99e40a98f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f99cc000c00 tx=0x7f99cc0046c0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 shutdown_connections 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f99bc06b2c0 0x7f99bc06d780 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99e41621d0 0x7f99e41459f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99e414af80 0x7f99e40a98f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 --2- 172.21.15.2:0/516329059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99e406da60 0x7f99e41f6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 >> 172.21.15.2:0/516329059 conn(0x7f99e40b0840 msgr2=0x7f99e4070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:51.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 shutdown_connections 2024-01-15T17:58:51.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:51.797+0000 7f99eac00640 1 -- 172.21.15.2:0/516329059 wait complete. 2024-01-15T17:58:51.820 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T17:58:51.820 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T17:58:52.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.041+0000 7f6312c00640 1 Processor -- start 2024-01-15T17:58:52.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.041+0000 7f6312c00640 1 -- start start 2024-01-15T17:58:52.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6312c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06f2b0 0x7f630c071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6312c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f630c06d250 0x7f630c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6312c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630c0ce880 con 0x7f630c06a9f0 2024-01-15T17:58:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6312c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630c06ae80 con 0x7f630c06d250 2024-01-15T17:58:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6312c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c0ca610 con 0x7f630c06f2b0 2024-01-15T17:58:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6310e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f630c06d250 0x7f630c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.043+0000 7f6311800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06f2b0 0x7f630c071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.044+0000 7f6310e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f630c06d250 0x7f630c06a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:57722/0 (socket says 172.21.15.2:57722) 2024-01-15T17:58:52.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.044+0000 7f6310e00640 1 -- 172.21.15.2:0/207188352 learned_addr learned my addr 172.21.15.2:0/207188352 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:52.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.044+0000 7f6310e00640 1 -- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06f2b0 msgr2=0x7f630c071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:52.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.044+0000 7f6310e00640 1 --2- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06f2b0 0x7f630c071c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.044+0000 7f6310e00640 1 -- 172.21.15.2:0/207188352 >> v1:172.21.15.137:6789/0 conn(0x7f630c06a9f0 legacy=0x7f630c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.045+0000 7f6310e00640 1 -- 172.21.15.2:0/207188352 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c072160 con 0x7f630c06d250 2024-01-15T17:58:52.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.044+0000 7f630be00640 1 -- 172.21.15.2:0/207188352 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111158987 0 0) 0x7f630c0ce880 con 0x7f630c06a9f0 2024-01-15T17:58:52.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.046+0000 7f6310e00640 1 --2- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06d250 0x7f630c06a4b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009870 tx=0x7f62f402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8c968f47aeffd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:52.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.046+0000 7f630be00640 1 -- 172.21.15.2:0/207188352 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4030840 con 0x7f630c06d250 2024-01-15T17:58:52.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.046+0000 7f630be00640 1 -- 172.21.15.2:0/207188352 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f40309a0 con 0x7f630c06d250 2024-01-15T17:58:52.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.046+0000 7f630be00640 1 -- 172.21.15.2:0/207188352 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4030d30 con 0x7f630c06d250 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 -- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06d250 msgr2=0x7f630c06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 --2- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06d250 0x7f630c06a4b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009870 tx=0x7f62f402b980 comp rx=0 tx=0).stop 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 -- 172.21.15.2:0/207188352 shutdown_connections 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 --2- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06d250 0x7f630c06a4b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 --2- 172.21.15.2:0/207188352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06f2b0 0x7f630c071c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 -- 172.21.15.2:0/207188352 >> 172.21.15.2:0/207188352 conn(0x7f630c0cfd30 msgr2=0x7f630c0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 -- 172.21.15.2:0/207188352 shutdown_connections 2024-01-15T17:58:52.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.047+0000 7f6312c00640 1 -- 172.21.15.2:0/207188352 wait complete. 2024-01-15T17:58:52.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.049+0000 7f6312c00640 1 Processor -- start 2024-01-15T17:58:52.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.049+0000 7f6312c00640 1 -- start start 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6312c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06a9f0 0x7f630c1f70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6312c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 0x7f630c141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6312c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06f2b0 0x7f630c1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6312c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1425f0 con 0x7f630c06d250 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6312c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c1f7d40 con 0x7f630c06f2b0 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6312c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c142ab0 con 0x7f630c06a9f0 2024-01-15T17:58:52.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6310e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 0x7f630c141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6310e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 0x7f630c141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36770/0 (socket says 172.21.15.2:36770) 2024-01-15T17:58:52.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.050+0000 7f6310e00640 1 -- 172.21.15.2:0/1958429357 learned_addr learned my addr 172.21.15.2:0/1958429357 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6311800640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06a9f0 0x7f630c1f70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6312200640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06f2b0 0x7f630c1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6310e00640 1 -- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06a9f0 msgr2=0x7f630c1f70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6310e00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06a9f0 0x7f630c1f70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6310e00640 1 -- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06f2b0 msgr2=0x7f630c1420b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6310e00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06f2b0 0x7f630c1420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6310e00640 1 -- 172.21.15.2:0/1958429357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f40094d0 con 0x7f630c06d250 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6310e00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 0x7f630c141b70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009990 tx=0x7f62f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4039a90 con 0x7f630c06d250 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f4039bf0 con 0x7f630c06d250 2024-01-15T17:58:52.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4043410 con 0x7f630c06d250 2024-01-15T17:58:52.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.051+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c1427d0 con 0x7f630c06d250 2024-01-15T17:58:52.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.052+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f630c0c4f50 con 0x7f630c06d250 2024-01-15T17:58:52.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.053+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630c0cc2d0 con 0x7f630c06d250 2024-01-15T17:58:52.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.054+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f62f402f390 con 0x7f630c06d250 2024-01-15T17:58:52.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.055+0000 7f6309600640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62e406b2c0 0x7f62e406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.055+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f62f40aba90 con 0x7f630c06d250 2024-01-15T17:58:52.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.055+0000 7f6311800640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62e406b2c0 0x7f62e406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.057+0000 7f6311800640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62e406b2c0 0x7f62e406d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009a10 tx=0x7f62fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:52.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.059+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62f4076c50 con 0x7f630c06d250 2024-01-15T17:58:52.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.256+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f630c0cd900 con 0x7f630c06d250 2024-01-15T17:58:52.646 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-01-15T17:58:52.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.739+0000 7f6309600640 1 -- 172.21.15.2:0/1958429357 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7f62f407a270 con 0x7f630c06d250 2024-01-15T17:58:52.742 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T17:58:52.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.744+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62e406b2c0 msgr2=0x7f62e406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.744+0000 7f6312c00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62e406b2c0 0x7f62e406d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f62fc009a10 tx=0x7f62fc008040 comp rx=0 tx=0).stop 2024-01-15T17:58:52.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.745+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 msgr2=0x7f630c141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.745+0000 7f6312c00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 0x7f630c141b70 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f62f4009990 tx=0x7f62f4004730 comp rx=0 tx=0).stop 2024-01-15T17:58:52.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 shutdown_connections 2024-01-15T17:58:52.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62e406b2c0 0x7f62e406d780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c06f2b0 0x7f630c1420b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c06d250 0x7f630c141b70 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 --2- 172.21.15.2:0/1958429357 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c06a9f0 0x7f630c1f70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 >> 172.21.15.2:0/1958429357 conn(0x7f630c0cfd30 msgr2=0x7f630c0cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:52.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 shutdown_connections 2024-01-15T17:58:52.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.746+0000 7f6312c00640 1 -- 172.21.15.2:0/1958429357 wait complete. 2024-01-15T17:58:52.775 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T17:58:52.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.980+0000 7f762cc00640 1 Processor -- start 2024-01-15T17:58:52.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.980+0000 7f762cc00640 1 -- start start 2024-01-15T17:58:52.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.982+0000 7f762cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f76280b32b0 0x7f76280d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.982+0000 7f762cc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762806f440 con 0x7f762815d9a0 2024-01-15T17:58:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.982+0000 7f762cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762815e5f0 con 0x7f7628146ba0 2024-01-15T17:58:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.982+0000 7f762cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7628146f80 con 0x7f76280b32b0 2024-01-15T17:58:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.982+0000 7f7626800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f762815d9a0 0x7f762806b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32890/0 (socket says 172.21.15.2:32890) 2024-01-15T17:58:52.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.983+0000 7f7626800640 1 -- 172.21.15.2:0/816655837 learned_addr learned my addr 172.21.15.2:0/816655837 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.983+0000 7f7625400640 1 --2- 172.21.15.2:0/816655837 >> v2:172.21.15.137:3300/0 conn(0x7f76280b32b0 0x7f76280d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.983+0000 7f7624a00640 1 -- 172.21.15.2:0/816655837 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319040318 0 0) 0x7f762806f440 con 0x7f762815d9a0 2024-01-15T17:58:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.983+0000 7f7624a00640 1 -- 172.21.15.2:0/816655837 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7608003520 con 0x7f762815d9a0 2024-01-15T17:58:52.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.983+0000 7f7625400640 1 -- 172.21.15.2:0/816655837 >> v1:172.21.15.137:6789/0 conn(0x7f7628146ba0 legacy=0x7f76280b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.983+0000 7f7625400640 1 -- 172.21.15.2:0/816655837 >> v1:172.21.15.2:6789/0 conn(0x7f762815d9a0 legacy=0x7f762806b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.984+0000 7f7625400640 1 -- 172.21.15.2:0/816655837 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762815e8d0 con 0x7f76280b32b0 2024-01-15T17:58:52.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.985+0000 7f7625400640 1 --2- 172.21.15.2:0/816655837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76280b32b0 0x7f76280d1a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f76100097a0 tx=0x7f761002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91a4ebc47e54153d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:52.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.985+0000 7f7624a00640 1 -- 172.21.15.2:0/816655837 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610031830 con 0x7f76280b32b0 2024-01-15T17:58:52.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.985+0000 7f7624a00640 1 -- 172.21.15.2:0/816655837 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7610031990 con 0x7f76280b32b0 2024-01-15T17:58:52.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.985+0000 7f7624a00640 1 -- 172.21.15.2:0/816655837 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610031d20 con 0x7f76280b32b0 2024-01-15T17:58:52.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 -- 172.21.15.2:0/816655837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76280b32b0 msgr2=0x7f76280d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 --2- 172.21.15.2:0/816655837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76280b32b0 0x7f76280d1a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f76100097a0 tx=0x7f761002b8d0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 -- 172.21.15.2:0/816655837 shutdown_connections 2024-01-15T17:58:52.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 --2- 172.21.15.2:0/816655837 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76280b32b0 0x7f76280d1a60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 -- 172.21.15.2:0/816655837 >> 172.21.15.2:0/816655837 conn(0x7f76280b6430 msgr2=0x7f76280b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:52.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 -- 172.21.15.2:0/816655837 shutdown_connections 2024-01-15T17:58:52.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.986+0000 7f762cc00640 1 -- 172.21.15.2:0/816655837 wait complete. 2024-01-15T17:58:52.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.988+0000 7f762cc00640 1 Processor -- start 2024-01-15T17:58:52.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.988+0000 7f762cc00640 1 -- start start 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f762cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 0x7f762815cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f762cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7628146ba0 0x7f762813ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f762cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f762815d9a0 0x7f76280cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f762cc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280ce260 con 0x7f76280b32b0 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f762cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280ce540 con 0x7f762815d9a0 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f762cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280ce820 con 0x7f7628146ba0 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f7625e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 0x7f762815cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f7625e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 0x7f762815cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46706/0 (socket says 172.21.15.2:46706) 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.989+0000 7f7625e00640 1 -- 172.21.15.2:0/3288176677 learned_addr learned my addr 172.21.15.2:0/3288176677 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7626800640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f762815d9a0 0x7f76280cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7625e00640 1 -- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7628146ba0 msgr2=0x7f762813ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7625e00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7628146ba0 0x7f762813ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7625e00640 1 -- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f762815d9a0 msgr2=0x7f76280cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7625e00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f762815d9a0 0x7f76280cda10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7625e00640 1 -- 172.21.15.2:0/3288176677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610009400 con 0x7f76280b32b0 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7626800640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f762815d9a0 0x7f76280cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.990+0000 7f7625e00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 0x7f762815cba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f761800c400 tx=0x7f761800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:52.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.991+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180052d0 con 0x7f76280b32b0 2024-01-15T17:58:52.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.991+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618005450 con 0x7f76280b32b0 2024-01-15T17:58:52.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.991+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618007260 con 0x7f76280b32b0 2024-01-15T17:58:52.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.991+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76280ceb00 con 0x7f76280b32b0 2024-01-15T17:58:52.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.991+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76280cdf50 con 0x7f76280b32b0 2024-01-15T17:58:52.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.992+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762815e210 con 0x7f76280b32b0 2024-01-15T17:58:52.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.993+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f76180073c0 con 0x7f76280b32b0 2024-01-15T17:58:52.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.994+0000 7f761e000640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760006b2c0 0x7f760006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:52.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.994+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f7618088d60 con 0x7f76280b32b0 2024-01-15T17:58:52.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.995+0000 7f7625400640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760006b2c0 0x7f760006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:52.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.996+0000 7f7625400640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760006b2c0 0x7f760006d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7610036000 tx=0x7f7610030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:53.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:52.999+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7618053e40 con 0x7f76280b32b0 2024-01-15T17:58:53.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.241+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f76280686f0 con 0x7f76280b32b0 2024-01-15T17:58:53.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.745+0000 7f761e000640 1 -- 172.21.15.2:0/3288176677 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7618057460 con 0x7f76280b32b0 2024-01-15T17:58:53.749 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.749+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760006b2c0 msgr2=0x7f760006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.749+0000 7f762cc00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760006b2c0 0x7f760006d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7610036000 tx=0x7f7610030000 comp rx=0 tx=0).stop 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.749+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 msgr2=0x7f762815cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.749+0000 7f762cc00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 0x7f762815cba0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f761800c400 tx=0x7f761800c8c0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.750+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 shutdown_connections 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.750+0000 7f762cc00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f760006b2c0 0x7f760006d780 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.750+0000 7f762cc00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f762815d9a0 0x7f76280cda10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.750+0000 7f762cc00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7628146ba0 0x7f762813ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.750+0000 7f762cc00640 1 --2- 172.21.15.2:0/3288176677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76280b32b0 0x7f762815cba0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.750+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 >> 172.21.15.2:0/3288176677 conn(0x7f76280b6430 msgr2=0x7f76280b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:53.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.751+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 shutdown_connections 2024-01-15T17:58:53.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.751+0000 7f762cc00640 1 -- 172.21.15.2:0/3288176677 wait complete. 2024-01-15T17:58:53.789 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T17:58:53.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.972+0000 7f3114400640 1 Processor -- start 2024-01-15T17:58:53.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.972+0000 7f3114400640 1 -- start start 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3114400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f310c144c00 0x7f310c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3114400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f310c06d000 0x7f310c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3114400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310c0ce230 con 0x7f310c0cdd70 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3114400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c0ce510 con 0x7f310c144c00 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3114400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f310c0ce7f0 con 0x7f310c06d000 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f310c144c00 0x7f310c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f310c144c00 0x7f310c0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46716/0 (socket says 172.21.15.2:46716) 2024-01-15T17:58:53.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 -- 172.21.15.2:0/1980357605 learned_addr learned my addr 172.21.15.2:0/1980357605 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111000640 1 --1- 172.21.15.2:0/1980357605 >> v1:172.21.15.2:6789/0 conn(0x7f310c0cdd70 0x7f310c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32902/0 (socket says 172.21.15.2:32902) 2024-01-15T17:58:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 -- 172.21.15.2:0/1980357605 >> v2:172.21.15.137:3300/0 conn(0x7f310c06d000 msgr2=0x7f310c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 --2- 172.21.15.2:0/1980357605 >> v2:172.21.15.137:3300/0 conn(0x7f310c06d000 0x7f310c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 -- 172.21.15.2:0/1980357605 >> v1:172.21.15.2:6789/0 conn(0x7f310c0cdd70 legacy=0x7f310c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:53.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.974+0000 7f3111a00640 1 -- 172.21.15.2:0/1980357605 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c0705f0 con 0x7f310c144c00 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.976+0000 7f3111a00640 1 --2- 172.21.15.2:0/1980357605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c0cd830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f310c15ec80 tx=0x7f30fc02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9745ece58272518d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.976+0000 7f310be00640 1 -- 172.21.15.2:0/1980357605 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc030870 con 0x7f310c144c00 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.976+0000 7f310be00640 1 -- 172.21.15.2:0/1980357605 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30fc0309d0 con 0x7f310c144c00 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.976+0000 7f310be00640 1 -- 172.21.15.2:0/1980357605 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30fc030d60 con 0x7f310c144c00 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 -- 172.21.15.2:0/1980357605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 msgr2=0x7f310c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 --2- 172.21.15.2:0/1980357605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c0cd830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f310c15ec80 tx=0x7f30fc02b9b0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 -- 172.21.15.2:0/1980357605 shutdown_connections 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 --2- 172.21.15.2:0/1980357605 >> v2:172.21.15.137:3300/0 conn(0x7f310c06d000 0x7f310c070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 --2- 172.21.15.2:0/1980357605 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c0cd830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 -- 172.21.15.2:0/1980357605 >> 172.21.15.2:0/1980357605 conn(0x7f310c0b45f0 msgr2=0x7f310c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:53.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 -- 172.21.15.2:0/1980357605 shutdown_connections 2024-01-15T17:58:53.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.977+0000 7f3114400640 1 -- 172.21.15.2:0/1980357605 wait complete. 2024-01-15T17:58:53.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.979+0000 7f3114400640 1 Processor -- start 2024-01-15T17:58:53.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.979+0000 7f3114400640 1 -- start start 2024-01-15T17:58:53.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3114400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f310c06d000 0x7f310c13b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:53.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3114400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f310c0cdd70 0x7f310c140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:53.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3114400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c1413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3114400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c1400e0 con 0x7f310c144c00 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3114400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c13f460 con 0x7f310c0cdd70 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3114400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c13f740 con 0x7f310c06d000 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c1413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c1413a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46726/0 (socket says 172.21.15.2:46726) 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 -- 172.21.15.2:0/1452125654 learned_addr learned my addr 172.21.15.2:0/1452125654 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3111000640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f310c0cdd70 0x7f310c140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3111a00640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f310c06d000 0x7f310c13b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 -- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f310c06d000 msgr2=0x7f310c13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:53.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f310c06d000 0x7f310c13b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 -- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f310c0cdd70 msgr2=0x7f310c140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.980+0000 7f3112400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f310c0cdd70 0x7f310c140e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3112400640 1 -- 172.21.15.2:0/1452125654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0094d0 con 0x7f310c144c00 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3111a00640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f310c06d000 0x7f310c13b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3112400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c1413a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f310400b9e0 tx=0x7f310400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310400cb20 con 0x7f310c144c00 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310400cc80 con 0x7f310c144c00 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3104014680 con 0x7f310c144c00 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f310c13fa20 con 0x7f310c144c00 2024-01-15T17:58:53.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.981+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f310c13bd20 con 0x7f310c144c00 2024-01-15T17:58:53.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.982+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f310c140220 con 0x7f310c144c00 2024-01-15T17:58:53.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.984+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f310401d3b0 con 0x7f310c144c00 2024-01-15T17:58:53.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.984+0000 7f3109600640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30e406b2c0 0x7f30e406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:53.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.985+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f3104089b90 con 0x7f310c144c00 2024-01-15T17:58:53.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.985+0000 7f3111a00640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30e406b2c0 0x7f30e406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:53.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.986+0000 7f3111a00640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30e406b2c0 0x7f30e406d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f30fc036000 tx=0x7f30fc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:53.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:53.989+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f310c140220 con 0x7f310c144c00 2024-01-15T17:58:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.220+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f310c140220 con 0x7f310c144c00 2024-01-15T17:58:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.767+0000 7f3109600640 1 -- 172.21.15.2:0/1452125654 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f310c140220 con 0x7f310c144c00 2024-01-15T17:58:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 4 and data pool 5 2024-01-15T17:58:54.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30e406b2c0 msgr2=0x7f30e406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30e406b2c0 0x7f30e406d780 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f30fc036000 tx=0x7f30fc03b040 comp rx=0 tx=0).stop 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 msgr2=0x7f310c1413a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c1413a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f310400b9e0 tx=0x7f310400bea0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 shutdown_connections 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30e406b2c0 0x7f30e406d780 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f310c144c00 0x7f310c1413a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f310c0cdd70 0x7f310c140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 --2- 172.21.15.2:0/1452125654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f310c06d000 0x7f310c13b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.770+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 >> 172.21.15.2:0/1452125654 conn(0x7f310c0b45f0 msgr2=0x7f310c0ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.771+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 shutdown_connections 2024-01-15T17:58:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.771+0000 7f3114400640 1 -- 172.21.15.2:0/1452125654 wait complete. 2024-01-15T17:58:54.809 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:54.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.986+0000 7fe7eac00640 1 Processor -- start 2024-01-15T17:58:54.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.986+0000 7fe7eac00640 1 -- start start 2024-01-15T17:58:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7eac00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe7e406cb80 0x7fe7e4166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:54.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7eac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e415cb00 con 0x7fe7e415b7d0 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7eac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e40ccec0 con 0x7fe7e415c0e0 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7eac00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e40cccd0 con 0x7fe7e406cb80 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7e3400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe7e415b7d0 0x7fe7e406c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32908/0 (socket says 172.21.15.2:32908) 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7e3400640 1 -- 172.21.15.2:0/3931618910 learned_addr learned my addr 172.21.15.2:0/3931618910 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.988+0000 7fe7e8c00640 1 --2- 172.21.15.2:0/3931618910 >> v2:172.21.15.2:3300/0 conn(0x7fe7e406cb80 0x7fe7e4166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.989+0000 7fe7e2a00640 1 -- 172.21.15.2:0/3931618910 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4026027767 0 0) 0x7fe7e415cb00 con 0x7fe7e415b7d0 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.989+0000 7fe7e2a00640 1 -- 172.21.15.2:0/3931618910 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4003520 con 0x7fe7e415b7d0 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.989+0000 7fe7e8c00640 1 -- 172.21.15.2:0/3931618910 >> v1:172.21.15.137:6789/0 conn(0x7fe7e415c0e0 legacy=0x7fe7e40cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.989+0000 7fe7e8c00640 1 -- 172.21.15.2:0/3931618910 >> v1:172.21.15.2:6789/0 conn(0x7fe7e415b7d0 legacy=0x7fe7e406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:54.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.989+0000 7fe7e8c00640 1 -- 172.21.15.2:0/3931618910 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e40cd1a0 con 0x7fe7e406cb80 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.990+0000 7fe7e8c00640 1 --2- 172.21.15.2:0/3931618910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e406cb80 0x7fe7e4166070 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009d90 tx=0x7fe7dc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1000425ec20a1e87 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.990+0000 7fe7e2a00640 1 -- 172.21.15.2:0/3931618910 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc02cc60 con 0x7fe7e406cb80 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7e2a00640 1 -- 172.21.15.2:0/3931618910 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc02cdc0 con 0x7fe7e406cb80 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7e2a00640 1 -- 172.21.15.2:0/3931618910 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc034660 con 0x7fe7e406cb80 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 -- 172.21.15.2:0/3931618910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e406cb80 msgr2=0x7fe7e4166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 --2- 172.21.15.2:0/3931618910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e406cb80 0x7fe7e4166070 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc009d90 tx=0x7fe7dc02beb0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 -- 172.21.15.2:0/3931618910 shutdown_connections 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 --2- 172.21.15.2:0/3931618910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e406cb80 0x7fe7e4166070 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 -- 172.21.15.2:0/3931618910 >> 172.21.15.2:0/3931618910 conn(0x7fe7e406da20 msgr2=0x7fe7e406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:54.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 -- 172.21.15.2:0/3931618910 shutdown_connections 2024-01-15T17:58:54.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.991+0000 7fe7eac00640 1 -- 172.21.15.2:0/3931618910 wait complete. 2024-01-15T17:58:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.993+0000 7fe7eac00640 1 Processor -- start 2024-01-15T17:58:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.993+0000 7fe7eac00640 1 -- start start 2024-01-15T17:58:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7eac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7e406cb80 0x7fe7e414deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7eac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 0x7fe7e4154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7eac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7e415c0e0 0x7fe7e41549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7eac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41e5510 con 0x7fe7e415b7d0 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7eac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41538b0 con 0x7fe7e406cb80 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7eac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e414e3f0 con 0x7fe7e415c0e0 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.994+0000 7fe7e3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 0x7fe7e4154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 0x7fe7e4154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46746/0 (socket says 172.21.15.2:46746) 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 -- 172.21.15.2:0/2515501099 learned_addr learned my addr 172.21.15.2:0/2515501099 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3e00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7e406cb80 0x7fe7e414deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e8c00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7e415c0e0 0x7fe7e41549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 -- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7e415c0e0 msgr2=0x7fe7e41549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7e415c0e0 0x7fe7e41549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 -- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7e406cb80 msgr2=0x7fe7e414deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7e406cb80 0x7fe7e414deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 -- 172.21.15.2:0/2515501099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7dc009a70 con 0x7fe7e415b7d0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.995+0000 7fe7e3400640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 0x7fe7e4154470 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c500 tx=0x7fe7cc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.996+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc005ce0 con 0x7fe7e415b7d0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.996+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc005e40 con 0x7fe7e415b7d0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.996+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc007330 con 0x7fe7e415b7d0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.996+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e41e5760 con 0x7fe7e415b7d0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.996+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e41e5d20 con 0x7fe7e415b7d0 2024-01-15T17:58:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.997+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e40686f0 con 0x7fe7e415b7d0 2024-01-15T17:58:55.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.998+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe7cc007490 con 0x7fe7e415b7d0 2024-01-15T17:58:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.999+0000 7fe7dbe00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe7bc06b2f0 0x7fe7bc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:54.999+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7cc089220 con 0x7fe7e415b7d0 2024-01-15T17:58:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.000+0000 7fe7e3e00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe7bc06b2f0 0x7fe7bc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.001+0000 7fe7e3e00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe7bc06b2f0 0x7fe7bc06d7b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4006fd0 tx=0x7fe7d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.003+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7cc054140 con 0x7fe7e415b7d0 2024-01-15T17:58:55.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.190+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7e406b260 con 0x7fe7e415b7d0 2024-01-15T17:58:55.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.192+0000 7fe7dbe00640 1 -- 172.21.15.2:0/2515501099 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16522 (secure 0 0 0) 0x7fe7cc057760 con 0x7fe7e415b7d0 2024-01-15T17:58:55.196 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:55.199 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":34,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:54.221858+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:58:52.311750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T17:58:53.297261+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.194+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe7bc06b2f0 msgr2=0x7fe7bc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.194+0000 7fe7eac00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe7bc06b2f0 0x7fe7bc06d7b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4006fd0 tx=0x7fe7d4008040 comp rx=0 tx=0).stop 2024-01-15T17:58:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.194+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 msgr2=0x7fe7e4154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.194+0000 7fe7eac00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 0x7fe7e4154470 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc00c500 tx=0x7fe7cc00c9c0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 shutdown_connections 2024-01-15T17:58:55.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe7bc06b2f0 0x7fe7bc06d7b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe7e415c0e0 0x7fe7e41549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe7e415b7d0 0x7fe7e4154470 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 --2- 172.21.15.2:0/2515501099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe7e406cb80 0x7fe7e414deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 >> 172.21.15.2:0/2515501099 conn(0x7fe7e406da20 msgr2=0x7fe7e415af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 shutdown_connections 2024-01-15T17:58:55.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.195+0000 7fe7eac00640 1 -- 172.21.15.2:0/2515501099 wait complete. 2024-01-15T17:58:55.216 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:55.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.429+0000 7f2f34c00640 1 Processor -- start 2024-01-15T17:58:55.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.429+0000 7f2f34c00640 1 -- start start 2024-01-15T17:58:55.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f34c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 0x7f2f300cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f34c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300b3de0 con 0x7f2f300d1d90 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f34c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300b40c0 con 0x7f2f300d3090 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f34c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300d0e80 con 0x7f2f3015d340 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f2fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2f300d1d90 0x7f2f300b30b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32922/0 (socket says 172.21.15.2:32922) 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f2fe00640 1 -- 172.21.15.2:0/738143523 learned_addr learned my addr 172.21.15.2:0/738143523 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.431+0000 7f2f2f400640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 0x7f2f300cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564018564 0 0) 0x7f2f300b3de0 con 0x7f2f300d1d90 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f10003520 con 0x7f2f300d1d90 2024-01-15T17:58:55.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 802674523 0 0) 0x7f2f10003520 con 0x7f2f300d1d90 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 msgr2=0x7f2f300cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 0x7f2f300cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 >> v1:172.21.15.137:6789/0 conn(0x7f2f300d3090 legacy=0x7f2f300d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300d1160 con 0x7f2f300d1d90 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2f280019f0 con 0x7f2f300d1d90 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 >> v1:172.21.15.2:6789/0 conn(0x7f2f300d1d90 legacy=0x7f2f300b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f100039a0 0x7f2f10003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100047c0 0x7f2f10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 0x7f2f10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300d0e80 con 0x7f2f100039a0 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1000ba60 con 0x7f2f100047c0 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1000bc20 con 0x7f2f3015d340 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.432+0000 7f2f2f400640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f100039a0 0x7f2f10003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2ea00640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100047c0 0x7f2f10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2fe00640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 0x7f2f10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2f400640 1 -- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 msgr2=0x7f2f10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2f400640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3015d340 0x7f2f10005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2f400640 1 -- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100047c0 msgr2=0x7f2f10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:55.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2f400640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100047c0 0x7f2f10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.433+0000 7f2f2f400640 1 -- 172.21.15.2:0/738143523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f10003520 con 0x7f2f100039a0 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.434+0000 7f2f2f400640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f100039a0 0x7f2f10003dc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009a90 tx=0x7f2f100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7339dc6f6a88852b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.434+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2002c680 con 0x7f2f100039a0 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2002c7e0 con 0x7f2f100039a0 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f2e000640 1 -- 172.21.15.2:0/738143523 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2002cb70 con 0x7f2f100039a0 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f34c00640 1 -- 172.21.15.2:0/738143523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f100039a0 msgr2=0x7f2f10003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f34c00640 1 --2- 172.21.15.2:0/738143523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f100039a0 0x7f2f10003dc0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2f20009a90 tx=0x7f2f100057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f2fe00640 1 -- 172.21.15.2:0/738143523 reap_dead start 2024-01-15T17:58:55.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f34c00640 1 -- 172.21.15.2:0/738143523 shutdown_connections 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f34c00640 1 -- 172.21.15.2:0/738143523 >> 172.21.15.2:0/738143523 conn(0x7f2f3008caf0 msgr2=0x7f2f3009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f34c00640 1 -- 172.21.15.2:0/738143523 shutdown_connections 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.435+0000 7f2f34c00640 1 -- 172.21.15.2:0/738143523 wait complete. 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.437+0000 7f2f34c00640 1 Processor -- start 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.437+0000 7f2f34c00640 1 -- start start 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.438+0000 7f2f34c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100039a0 0x7f2f3015a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.438+0000 7f2f34c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 0x7f2f3014c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.438+0000 7f2f34c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3014ca30 0x7f2f300b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.438+0000 7f2f34c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f301503a0 con 0x7f2f30149ec0 2024-01-15T17:58:55.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.438+0000 7f2f34c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3014d470 con 0x7f2f100039a0 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.438+0000 7f2f34c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300c6c30 con 0x7f2f3014ca30 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 0x7f2f3014c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 0x7f2f3014c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46768/0 (socket says 172.21.15.2:46768) 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 -- 172.21.15.2:0/3795657722 learned_addr learned my addr 172.21.15.2:0/3795657722 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2f400640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100039a0 0x7f2f3015a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2fe00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3014ca30 0x7f2f300b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 -- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3014ca30 msgr2=0x7f2f300b72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3014ca30 0x7f2f300b72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 -- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100039a0 msgr2=0x7f2f3015a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100039a0 0x7f2f3015a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 -- 172.21.15.2:0/3795657722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300b84e0 con 0x7f2f30149ec0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2f400640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100039a0 0x7f2f3015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2fe00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3014ca30 0x7f2f300b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.439+0000 7f2f2ea00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 0x7f2f3014c4f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800e580 tx=0x7f2f1800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.440+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f180052e0 con 0x7f2f30149ec0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.440+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f18005440 con 0x7f2f30149ec0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.440+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f18003810 con 0x7f2f30149ec0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.440+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f300b87c0 con 0x7f2f30149ec0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.440+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f30150650 con 0x7f2f30149ec0 2024-01-15T17:58:55.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.441+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f30074770 con 0x7f2f30149ec0 2024-01-15T17:58:55.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.442+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2f18003970 con 0x7f2f30149ec0 2024-01-15T17:58:55.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.443+0000 7f2f27400640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f0806b2c0 0x7f2f0806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.443+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f18089a70 con 0x7f2f30149ec0 2024-01-15T17:58:55.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.443+0000 7f2f2f400640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f0806b2c0 0x7f2f0806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.445+0000 7f2f2f400640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f0806b2c0 0x7f2f0806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f20036000 tx=0x7f2f20030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.447+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f180547c0 con 0x7f2f30149ec0 2024-01-15T17:58:55.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.640+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f300678e0 con 0x7f2f30149ec0 2024-01-15T17:58:55.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.643+0000 7f2f27400640 1 -- 172.21.15.2:0/3795657722 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16522 (secure 0 0 0) 0x7f2f18057de0 con 0x7f2f30149ec0 2024-01-15T17:58:55.647 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:55.647 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":34,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:54.221858+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:58:52.311750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T17:58:53.297261+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f0806b2c0 msgr2=0x7f2f0806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f0806b2c0 0x7f2f0806d780 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f20036000 tx=0x7f2f20030000 comp rx=0 tx=0).stop 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 msgr2=0x7f2f3014c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 0x7f2f3014c4f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2f1800e580 tx=0x7f2f1800ea40 comp rx=0 tx=0).stop 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 shutdown_connections 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f0806b2c0 0x7f2f0806d780 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f3014ca30 0x7f2f300b72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f30149ec0 0x7f2f3014c4f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 --2- 172.21.15.2:0/3795657722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f100039a0 0x7f2f3015a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.646+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 >> 172.21.15.2:0/3795657722 conn(0x7f2f3008caf0 msgr2=0x7f2f3006b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.647+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 shutdown_connections 2024-01-15T17:58:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.647+0000 7f2f34c00640 1 -- 172.21.15.2:0/3795657722 wait complete. 2024-01-15T17:58:55.668 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T17:58:55.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.889+0000 7f662a400640 1 Processor -- start 2024-01-15T17:58:55.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.890+0000 7f662a400640 1 -- start start 2024-01-15T17:58:55.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.891+0000 7f662a400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f662415ae80 0x7f66240acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.891+0000 7f662a400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f662406c8f0 0x7f662406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.891+0000 7f662a400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662406d290 con 0x7f66240b2430 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.891+0000 7f662a400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662415e840 con 0x7f662406c8f0 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.891+0000 7f662a400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662415dbc0 con 0x7f662415ae80 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f662406c8f0 0x7f662406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f662406c8f0 0x7f662406cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46782/0 (socket says 172.21.15.2:46782) 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 -- 172.21.15.2:0/344773627 learned_addr learned my addr 172.21.15.2:0/344773627 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f6623e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f66240b2430 0x7f66240b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32928/0 (socket says 172.21.15.2:32928) 2024-01-15T17:58:55.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f6623400640 1 --2- 172.21.15.2:0/344773627 >> v2:172.21.15.137:3300/0 conn(0x7f662415ae80 0x7f66240acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 -- 172.21.15.2:0/344773627 >> v2:172.21.15.137:3300/0 conn(0x7f662415ae80 msgr2=0x7f66240acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 --2- 172.21.15.2:0/344773627 >> v2:172.21.15.137:3300/0 conn(0x7f662415ae80 0x7f66240acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 -- 172.21.15.2:0/344773627 >> v1:172.21.15.2:6789/0 conn(0x7f66240b2430 legacy=0x7f66240b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f661be00640 1 -- 172.21.15.2:0/344773627 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662415dea0 con 0x7f662406c8f0 2024-01-15T17:58:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f6622a00640 1 -- 172.21.15.2:0/344773627 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594997609 0 0) 0x7f662406d290 con 0x7f66240b2430 2024-01-15T17:58:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.892+0000 7f6623400640 1 --2- 172.21.15.2:0/344773627 >> v2:172.21.15.137:3300/0 conn(0x7f662415ae80 0x7f66240acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.893+0000 7f661be00640 1 --2- 172.21.15.2:0/344773627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662406c8f0 0x7f662406cd50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f662406e660 tx=0x7f660c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f98d449b6cf4b65 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f6622a00640 1 -- 172.21.15.2:0/344773627 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c030860 con 0x7f662406c8f0 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f6622a00640 1 -- 172.21.15.2:0/344773627 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660c0309c0 con 0x7f662406c8f0 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f6622a00640 1 -- 172.21.15.2:0/344773627 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660c030d50 con 0x7f662406c8f0 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f662a400640 1 -- 172.21.15.2:0/344773627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662406c8f0 msgr2=0x7f662406cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f662a400640 1 --2- 172.21.15.2:0/344773627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662406c8f0 0x7f662406cd50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f662406e660 tx=0x7f660c02b9a0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f662a400640 1 -- 172.21.15.2:0/344773627 shutdown_connections 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f662a400640 1 --2- 172.21.15.2:0/344773627 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662406c8f0 0x7f662406cd50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f662a400640 1 --2- 172.21.15.2:0/344773627 >> v2:172.21.15.137:3300/0 conn(0x7f662415ae80 0x7f66240acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.894+0000 7f662a400640 1 -- 172.21.15.2:0/344773627 >> 172.21.15.2:0/344773627 conn(0x7f66240b45f0 msgr2=0x7f66240b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.895+0000 7f662a400640 1 -- 172.21.15.2:0/344773627 shutdown_connections 2024-01-15T17:58:55.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.895+0000 7f662a400640 1 -- 172.21.15.2:0/344773627 wait complete. 2024-01-15T17:58:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.896+0000 7f662a400640 1 Processor -- start 2024-01-15T17:58:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.896+0000 7f662a400640 1 -- start start 2024-01-15T17:58:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.897+0000 7f662a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f662406c8f0 0x7f6624159f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.897+0000 7f662a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66240b2430 0x7f66241527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.897+0000 7f662a400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 0x7f66241f2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.897+0000 7f662a400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662414f670 con 0x7f662415ae80 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.897+0000 7f662a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662414ecd0 con 0x7f662406c8f0 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.897+0000 7f662a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662414efb0 con 0x7f66240b2430 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 0x7f66241f2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 0x7f66241f2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46792/0 (socket says 172.21.15.2:46792) 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 -- 172.21.15.2:0/2056499030 learned_addr learned my addr 172.21.15.2:0/2056499030 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f661be00640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66240b2430 0x7f66241527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f662406c8f0 0x7f6624159f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 -- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66240b2430 msgr2=0x7f66241527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66240b2430 0x7f66241527b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 -- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f662406c8f0 msgr2=0x7f6624159f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f662406c8f0 0x7f6624159f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 -- 172.21.15.2:0/2056499030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c0094d0 con 0x7f662415ae80 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.898+0000 7f6623e00640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 0x7f66241f2410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f661c00c7e0 tx=0x7f661c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.899+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00a640 con 0x7f662415ae80 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.899+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f661c00a7c0 con 0x7f662415ae80 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.899+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f661c00ab70 con 0x7f662415ae80 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.899+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f662414f290 con 0x7f662415ae80 2024-01-15T17:58:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.899+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66241f2950 con 0x7f662415ae80 2024-01-15T17:58:55.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.900+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66240686f0 con 0x7f662415ae80 2024-01-15T17:58:55.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.901+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f661c00acd0 con 0x7f662415ae80 2024-01-15T17:58:55.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.902+0000 7f661b400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f65fc06b2c0 0x7f65fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.902+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f661c089200 con 0x7f662415ae80 2024-01-15T17:58:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.902+0000 7f6623400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f65fc06b2c0 0x7f65fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.904+0000 7f6623400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f65fc06b2c0 0x7f65fc06d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f661400b840 tx=0x7f6614008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:55.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:55.906+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f661c054060 con 0x7f662415ae80 2024-01-15T17:58:56.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.142+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6624071cf0 con 0x7f662415ae80 2024-01-15T17:58:56.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.797+0000 7f661b400640 1 -- 172.21.15.2:0/2056499030 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f661c057680 con 0x7f662415ae80 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f65fc06b2c0 msgr2=0x7f65fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f65fc06b2c0 0x7f65fc06d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f661400b840 tx=0x7f6614008040 comp rx=0 tx=0).stop 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 msgr2=0x7f66241f2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 0x7f66241f2410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f661c00c7e0 tx=0x7f661c00cca0 comp rx=0 tx=0).stop 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 shutdown_connections 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f65fc06b2c0 0x7f65fc06d780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f662415ae80 0x7f66241f2410 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66240b2430 0x7f66241527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:56.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 --2- 172.21.15.2:0/2056499030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f662406c8f0 0x7f6624159f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:56.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 >> 172.21.15.2:0/2056499030 conn(0x7f66240b45f0 msgr2=0x7f66240b28a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:56.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 shutdown_connections 2024-01-15T17:58:56.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:56.801+0000 7f662a400640 1 -- 172.21.15.2:0/2056499030 wait complete. 2024-01-15T17:58:56.828 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T17:58:56.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T17:58:57.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.050+0000 7fec87a00640 1 Processor -- start 2024-01-15T17:58:57.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.050+0000 7fec87a00640 1 -- start start 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec87a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fec801439f0 0x7fec800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec87a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8015dd70 con 0x7fec800b3420 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec87a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800b38b0 con 0x7fec8006c8f0 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec87a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec800711b0 con 0x7fec801439f0 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec85000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fec801439f0 0x7fec800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec85000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fec801439f0 0x7fec800702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46800/0 (socket says 172.21.15.2:46800) 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.052+0000 7fec85000640 1 -- 172.21.15.2:0/1817546441 learned_addr learned my addr 172.21.15.2:0/1817546441 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:57.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.053+0000 7fec85000640 1 -- 172.21.15.2:0/1817546441 >> v1:172.21.15.137:6789/0 conn(0x7fec8006c8f0 legacy=0x7fec8006ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:57.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.053+0000 7fec85000640 1 -- 172.21.15.2:0/1817546441 >> v1:172.21.15.2:6789/0 conn(0x7fec800b3420 legacy=0x7fec800cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:57.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.053+0000 7fec85000640 1 -- 172.21.15.2:0/1817546441 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80143dd0 con 0x7fec801439f0 2024-01-15T17:58:57.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.053+0000 7fec7fe00640 1 -- 172.21.15.2:0/1817546441 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217242664 0 0) 0x7fec8015dd70 con 0x7fec800b3420 2024-01-15T17:58:57.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.054+0000 7fec85000640 1 --2- 172.21.15.2:0/1817546441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec801439f0 0x7fec800702c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fec70009870 tx=0x7fec7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f0c6d1c556f0b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:57.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.054+0000 7fec7fe00640 1 -- 172.21.15.2:0/1817546441 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70030860 con 0x7fec801439f0 2024-01-15T17:58:57.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.054+0000 7fec7fe00640 1 -- 172.21.15.2:0/1817546441 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec700309c0 con 0x7fec801439f0 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.054+0000 7fec7fe00640 1 -- 172.21.15.2:0/1817546441 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70030d50 con 0x7fec801439f0 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 -- 172.21.15.2:0/1817546441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec801439f0 msgr2=0x7fec800702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 --2- 172.21.15.2:0/1817546441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec801439f0 0x7fec800702c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fec70009870 tx=0x7fec7002b9a0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 -- 172.21.15.2:0/1817546441 shutdown_connections 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 --2- 172.21.15.2:0/1817546441 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec801439f0 0x7fec800702c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 -- 172.21.15.2:0/1817546441 >> 172.21.15.2:0/1817546441 conn(0x7fec800b45f0 msgr2=0x7fec800b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 -- 172.21.15.2:0/1817546441 shutdown_connections 2024-01-15T17:58:57.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.055+0000 7fec87a00640 1 -- 172.21.15.2:0/1817546441 wait complete. 2024-01-15T17:58:57.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.057+0000 7fec87a00640 1 Processor -- start 2024-01-15T17:58:57.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.057+0000 7fec87a00640 1 -- start start 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec87a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 0x7fec80140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec87a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec800b3420 0x7fec801416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec87a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec801439f0 0x7fec80159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec87a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8015d840 con 0x7fec8006c8f0 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec87a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec801408d0 con 0x7fec801439f0 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec87a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8013f2f0 con 0x7fec800b3420 2024-01-15T17:58:57.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec85000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 0x7fec80140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec85000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 0x7fec80140390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46814/0 (socket says 172.21.15.2:46814) 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.058+0000 7fec85000640 1 -- 172.21.15.2:0/3484526336 learned_addr learned my addr 172.21.15.2:0/3484526336 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec85000640 1 -- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec800b3420 msgr2=0x7fec801416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec85000640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec800b3420 0x7fec801416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec85000640 1 -- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec801439f0 msgr2=0x7fec80159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec85000640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec801439f0 0x7fec80159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec85000640 1 -- 172.21.15.2:0/3484526336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700094d0 con 0x7fec8006c8f0 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec85000640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 0x7fec80140390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fec70009870 tx=0x7fec70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:57.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.059+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70039b60 con 0x7fec8006c8f0 2024-01-15T17:58:57.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.060+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70039cc0 con 0x7fec8006c8f0 2024-01-15T17:58:57.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.060+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70043630 con 0x7fec8006c8f0 2024-01-15T17:58:57.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.060+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8013f5d0 con 0x7fec8006c8f0 2024-01-15T17:58:57.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.060+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8013fb90 con 0x7fec8006c8f0 2024-01-15T17:58:57.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.061+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec800686f0 con 0x7fec8006c8f0 2024-01-15T17:58:57.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.062+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fec7002f3b0 con 0x7fec8006c8f0 2024-01-15T17:58:57.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.063+0000 7fec7d600640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec5806b2c0 0x7fec5806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:57.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.063+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec700abf70 con 0x7fec8006c8f0 2024-01-15T17:58:57.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.063+0000 7fec77e00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec5806b2c0 0x7fec5806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:57.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.064+0000 7fec77e00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec5806b2c0 0x7fec5806d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec801f64a0 tx=0x7fec68007910 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.067+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec70076dd0 con 0x7fec8006c8f0 2024-01-15T17:58:57.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.296+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fec800b2db0 con 0x7fec8006c8f0 2024-01-15T17:58:57.730 INFO:tasks.ceph.mds.c.smithi002.stderr:2024-01-15T17:58:57.727+0000 7fb9e9600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T17:58:57.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.820+0000 7fec7d600640 1 -- 172.21.15.2:0/3484526336 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fec700340c0 con 0x7fec8006c8f0 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.823+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec5806b2c0 msgr2=0x7fec5806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.823+0000 7fec87a00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec5806b2c0 0x7fec5806d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fec801f64a0 tx=0x7fec68007910 comp rx=0 tx=0).stop 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.823+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 msgr2=0x7fec80140390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.823+0000 7fec87a00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 0x7fec80140390 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fec70009870 tx=0x7fec70004730 comp rx=0 tx=0).stop 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 shutdown_connections 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec5806b2c0 0x7fec5806d780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec801439f0 0x7fec80159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec800b3420 0x7fec801416d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 --2- 172.21.15.2:0/3484526336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8006c8f0 0x7fec80140390 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:57.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 >> 172.21.15.2:0/3484526336 conn(0x7fec800b45f0 msgr2=0x7fec800b11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:57.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 shutdown_connections 2024-01-15T17:58:57.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:57.824+0000 7fec87a00640 1 -- 172.21.15.2:0/3484526336 wait complete. 2024-01-15T17:58:57.850 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T17:58:57.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:58:58.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.075+0000 7f19e7e00640 1 Processor -- start 2024-01-15T17:58:58.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.075+0000 7f19e7e00640 1 -- start start 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19e7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f19e815ec60 0x7f19e815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19e7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f19e806cad0 0x7f19e806cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19e7e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80716d0 con 0x7f19e8141880 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19e7e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e8071f20 con 0x7f19e815ec60 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19e7e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e806c6b0 con 0x7f19e806cad0 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19dde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f19e815ec60 0x7f19e815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19dde00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f19e815ec60 0x7f19e815f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46828/0 (socket says 172.21.15.2:46828) 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.077+0000 7f19dde00640 1 -- 172.21.15.2:0/1176143096 learned_addr learned my addr 172.21.15.2:0/1176143096 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.078+0000 7f19e7400640 1 --2- 172.21.15.2:0/1176143096 >> v2:172.21.15.137:3300/0 conn(0x7f19e806cad0 0x7f19e806cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.078+0000 7f19dde00640 1 -- 172.21.15.2:0/1176143096 >> v2:172.21.15.137:3300/0 conn(0x7f19e806cad0 msgr2=0x7f19e806cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.078+0000 7f19dde00640 1 --2- 172.21.15.2:0/1176143096 >> v2:172.21.15.137:3300/0 conn(0x7f19e806cad0 0x7f19e806cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.078+0000 7f19dde00640 1 -- 172.21.15.2:0/1176143096 >> v1:172.21.15.137:6789/0 conn(0x7f19e8141880 legacy=0x7f19e8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:58.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.078+0000 7f19dde00640 1 -- 172.21.15.2:0/1176143096 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e8071960 con 0x7f19e815ec60 2024-01-15T17:58:58.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.078+0000 7f19e7400640 1 --2- 172.21.15.2:0/1176143096 >> v2:172.21.15.137:3300/0 conn(0x7f19e806cad0 0x7f19e806cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:58.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.079+0000 7f19dde00640 1 --2- 172.21.15.2:0/1176143096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e815ec60 0x7f19e815f0e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f19e8070060 tx=0x7f19d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c4887ad0a658024 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:58.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.080+0000 7f19e6000640 1 -- 172.21.15.2:0/1176143096 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00317e0 con 0x7f19e815ec60 2024-01-15T17:58:58.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.080+0000 7f19e6000640 1 -- 172.21.15.2:0/1176143096 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d0031940 con 0x7f19e815ec60 2024-01-15T17:58:58.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.080+0000 7f19e6000640 1 -- 172.21.15.2:0/1176143096 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0031cd0 con 0x7f19e815ec60 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 -- 172.21.15.2:0/1176143096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e815ec60 msgr2=0x7f19e815f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 --2- 172.21.15.2:0/1176143096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e815ec60 0x7f19e815f0e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f19e8070060 tx=0x7f19d002b880 comp rx=0 tx=0).stop 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 -- 172.21.15.2:0/1176143096 shutdown_connections 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 --2- 172.21.15.2:0/1176143096 >> v2:172.21.15.137:3300/0 conn(0x7f19e806cad0 0x7f19e806cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 --2- 172.21.15.2:0/1176143096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e815ec60 0x7f19e815f0e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 -- 172.21.15.2:0/1176143096 >> 172.21.15.2:0/1176143096 conn(0x7f19e80b0840 msgr2=0x7f19e80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 -- 172.21.15.2:0/1176143096 shutdown_connections 2024-01-15T17:58:58.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.081+0000 7f19e7e00640 1 -- 172.21.15.2:0/1176143096 wait complete. 2024-01-15T17:58:58.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.083+0000 7f19e7e00640 1 Processor -- start 2024-01-15T17:58:58.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.083+0000 7f19e7e00640 1 -- start start 2024-01-15T17:58:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 0x7f19e8138b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8141880 0x7f19e8139200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e815ec60 0x7f19e8139740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e8139060 con 0x7f19e806cad0 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e813b160 con 0x7f19e8141880 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e813b440 con 0x7f19e815ec60 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 0x7f19e8138b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 0x7f19e8138b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46830/0 (socket says 172.21.15.2:46830) 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 -- 172.21.15.2:0/202274385 learned_addr learned my addr 172.21.15.2:0/202274385 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19dde00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8141880 0x7f19e8139200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 -- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e815ec60 msgr2=0x7f19e8139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e815ec60 0x7f19e8139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 -- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8141880 msgr2=0x7f19e8139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.084+0000 7f19e6a00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8141880 0x7f19e8139200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19e6a00640 1 -- 172.21.15.2:0/202274385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d00094d0 con 0x7f19e806cad0 2024-01-15T17:58:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19e6a00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 0x7f19e8138b20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c7c0 tx=0x7f19d800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800aa50 con 0x7f19e806cad0 2024-01-15T17:58:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d800abb0 con 0x7f19e806cad0 2024-01-15T17:58:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e813b720 con 0x7f19e806cad0 2024-01-15T17:58:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8012420 con 0x7f19e806cad0 2024-01-15T17:58:58.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.085+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e813bce0 con 0x7f19e806cad0 2024-01-15T17:58:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.086+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e813bed0 con 0x7f19e806cad0 2024-01-15T17:58:58.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.087+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f19d8012580 con 0x7f19e806cad0 2024-01-15T17:58:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.088+0000 7f19dfe00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f19c006b2f0 0x7f19c006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.088+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19d808a1c0 con 0x7f19e806cad0 2024-01-15T17:58:58.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.089+0000 7f19dde00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f19c006b2f0 0x7f19c006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.090+0000 7f19dde00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f19c006b2f0 0x7f19c006d7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f19d0036000 tx=0x7f19d003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:58.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.093+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19d8054f90 con 0x7f19e806cad0 2024-01-15T17:58:58.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.323+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19e8074770 con 0x7f19e806cad0 2024-01-15T17:58:58.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.325+0000 7f19dfe00640 1 -- 172.21.15.2:0/202274385 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4904 (secure 0 0 0) 0x7f19d80585b0 con 0x7f19e806cad0 2024-01-15T17:58:58.328 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:58.328 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/122233456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":122233456},{"type":"v1","addr":"172.21.15.137:6837","nonce":122233456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1272974394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":1272974394},{"type":"v1","addr":"172.21.15.2:6835","nonce":1272974394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/84168043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":84168043},{"type":"v1","addr":"172.21.15.137:6835","nonce":84168043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:58:54.222210+0000","modified":"2024-01-15T17:58:57.297868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4954":{"gid":4954,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.2:6834/2859107199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2859107199},{"type":"v1","addr":"172.21.15.2:6834","nonce":2859107199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T17:58:58.328 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-01-15T17:58:58.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f19c006b2f0 msgr2=0x7f19c006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f19c006b2f0 0x7f19c006d7b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f19d0036000 tx=0x7f19d003d040 comp rx=0 tx=0).stop 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 msgr2=0x7f19e8138b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 0x7f19e8138b20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c7c0 tx=0x7f19d800cc80 comp rx=0 tx=0).stop 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 shutdown_connections 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f19c006b2f0 0x7f19c006d7b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f19e815ec60 0x7f19e8139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f19e8141880 0x7f19e8139200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 --2- 172.21.15.2:0/202274385 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f19e806cad0 0x7f19e8138b20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.327+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 >> 172.21.15.2:0/202274385 conn(0x7f19e80b0840 msgr2=0x7f19e80afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.328+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 shutdown_connections 2024-01-15T17:58:58.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.328+0000 7f19e7e00640 1 -- 172.21.15.2:0/202274385 wait complete. 2024-01-15T17:58:58.349 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T17:58:58.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.567+0000 7fc264e00640 1 Processor -- start 2024-01-15T17:58:58.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.567+0000 7fc264e00640 1 -- start start 2024-01-15T17:58:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.569+0000 7fc264e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2600ad960 0x7fc26015db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.569+0000 7fc264e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2600b2e70 con 0x7fc26015e070 2024-01-15T17:58:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.569+0000 7fc264e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26006c430 con 0x7fc26015d750 2024-01-15T17:58:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.569+0000 7fc264e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2600b30c0 con 0x7fc2600ad960 2024-01-15T17:58:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc26015e070 0x7fc260166d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32944/0 (socket says 172.21.15.2:32944) 2024-01-15T17:58:58.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25fe00640 1 -- 172.21.15.2:0/1631370568 learned_addr learned my addr 172.21.15.2:0/1631370568 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc256800640 1 --2- 172.21.15.2:0/1631370568 >> v2:172.21.15.137:3300/0 conn(0x7fc2600ad960 0x7fc26015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389847374 0 0) 0x7fc2600b2e70 con 0x7fc26015e070 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc240003520 con 0x7fc26015e070 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879131076 0 0) 0x7fc240003520 con 0x7fc26015e070 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 >> v2:172.21.15.137:3300/0 conn(0x7fc2600ad960 msgr2=0x7fc26015db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 --2- 172.21.15.2:0/1631370568 >> v2:172.21.15.137:3300/0 conn(0x7fc2600ad960 0x7fc26015db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 >> v1:172.21.15.137:6789/0 conn(0x7fc26015d750 legacy=0x7fc26006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600b33a0 con 0x7fc26015e070 2024-01-15T17:58:58.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc2580019f0 con 0x7fc26015e070 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.570+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 >> v1:172.21.15.2:6789/0 conn(0x7fc26015e070 legacy=0x7fc260166d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25ea00640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc240003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25ea00640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2400047c0 0x7fc240004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25ea00640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc240005180 0x7fc24000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600b30c0 con 0x7fc240003970 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24000c120 con 0x7fc240005180 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24000c2e0 con 0x7fc2400047c0 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25f400640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc240003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25fe00640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc240005180 0x7fc24000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc256800640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2400047c0 0x7fc240004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25f400640 1 -- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2400047c0 msgr2=0x7fc240004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25f400640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2400047c0 0x7fc240004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25f400640 1 -- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc240005180 msgr2=0x7fc24000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25f400640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc240005180 0x7fc24000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25f400640 1 -- 172.21.15.2:0/1631370568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc240003520 con 0x7fc240003970 2024-01-15T17:58:58.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.571+0000 7fc25fe00640 1 -- 172.21.15.2:0/1631370568 reap_dead start 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.573+0000 7fc25f400640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc240003d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc250001750 tx=0x7fc2400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc522607f938eb44 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.573+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2500052d0 con 0x7fc240003970 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.573+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25002e050 con 0x7fc240003970 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.573+0000 7fc25ea00640 1 -- 172.21.15.2:0/1631370568 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250003830 con 0x7fc240003970 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 -- 172.21.15.2:0/1631370568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 msgr2=0x7fc240003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc240003d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc250001750 tx=0x7fc2400057f0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 -- 172.21.15.2:0/1631370568 shutdown_connections 2024-01-15T17:58:58.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 --2- 172.21.15.2:0/1631370568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc240003d90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 -- 172.21.15.2:0/1631370568 >> 172.21.15.2:0/1631370568 conn(0x7fc2600b0840 msgr2=0x7fc2600b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 -- 172.21.15.2:0/1631370568 shutdown_connections 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.574+0000 7fc264e00640 1 -- 172.21.15.2:0/1631370568 wait complete. 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.576+0000 7fc264e00640 1 Processor -- start 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.576+0000 7fc264e00640 1 -- start start 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc264e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc2600cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc264e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26015c8e0 0x7fc26015fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc264e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26015ff50 0x7fc260153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc264e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc260160400 con 0x7fc240003970 2024-01-15T17:58:58.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc264e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600cd050 con 0x7fc26015c8e0 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc264e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc260143f30 con 0x7fc26015ff50 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc25f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc2600cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.577+0000 7fc25f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc2600cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46852/0 (socket says 172.21.15.2:46852) 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 -- 172.21.15.2:0/2384951628 learned_addr learned my addr 172.21.15.2:0/2384951628 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 -- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26015ff50 msgr2=0x7fc260153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26015ff50 0x7fc260153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 -- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26015c8e0 msgr2=0x7fc26015fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26015c8e0 0x7fc26015fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 -- 172.21.15.2:0/2384951628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600cc720 con 0x7fc240003970 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.578+0000 7fc25f400640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc2600cd4e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc25002bc00 tx=0x7fc250001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:58.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.579+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250007a30 con 0x7fc240003970 2024-01-15T17:58:58.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.579+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc250007640 con 0x7fc240003970 2024-01-15T17:58:58.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.579+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc250006c00 con 0x7fc240003970 2024-01-15T17:58:58.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.579+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2600cca00 con 0x7fc240003970 2024-01-15T17:58:58.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.579+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2600cda20 con 0x7fc240003970 2024-01-15T17:58:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.580+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc260074770 con 0x7fc240003970 2024-01-15T17:58:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.581+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc2500035f0 con 0x7fc240003970 2024-01-15T17:58:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.582+0000 7fc257e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23806b2c0 0x7fc23806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.582+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2500ab820 con 0x7fc240003970 2024-01-15T17:58:58.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.582+0000 7fc256800640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23806b2c0 0x7fc23806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:58.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.584+0000 7fc256800640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23806b2c0 0x7fc23806d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2600cdd50 tx=0x7fc248008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:58.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.586+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc25007a080 con 0x7fc240003970 2024-01-15T17:58:58.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.783+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc26006cca0 con 0x7fc240003970 2024-01-15T17:58:58.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.786+0000 7fc257e00640 1 -- 172.21.15.2:0/2384951628 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16509 (secure 0 0 0) 0x7fc250079c10 con 0x7fc240003970 2024-01-15T17:58:58.790 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:58.790 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":35,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T17:58:55.782775+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-15T17:58:52.311750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-15T17:58:53.297261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T17:58:58.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.788+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23806b2c0 msgr2=0x7fc23806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.788+0000 7fc264e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23806b2c0 0x7fc23806d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc2600cdd50 tx=0x7fc248008040 comp rx=0 tx=0).stop 2024-01-15T17:58:58.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.788+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 msgr2=0x7fc2600cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:58.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.788+0000 7fc264e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc2600cd4e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc25002bc00 tx=0x7fc250001120 comp rx=0 tx=0).stop 2024-01-15T17:58:58.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.788+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 shutdown_connections 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23806b2c0 0x7fc23806d780 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26015ff50 0x7fc260153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26015c8e0 0x7fc26015fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 --2- 172.21.15.2:0/2384951628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc240003970 0x7fc2600cd4e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 >> 172.21.15.2:0/2384951628 conn(0x7fc2600b0840 msgr2=0x7fc2600cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 shutdown_connections 2024-01-15T17:58:58.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:58.789+0000 7fc264e00640 1 -- 172.21.15.2:0/2384951628 wait complete. 2024-01-15T17:58:58.811 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T17:58:58.812 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:58:58.997 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:58:59.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:58:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.028+0000 7f66b2a00640 1 Processor -- start 2024-01-15T17:58:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.028+0000 7f66b2a00640 1 -- start start 2024-01-15T17:58:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.029+0000 7f66b2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66ac15b3d0 0x7f66ac0d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.029+0000 7f66b2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac0cfb10 0x7f66ac0b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.029+0000 7f66b2a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66ac15b7b0 con 0x7f66ac0cc3f0 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.029+0000 7f66b2a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66ac0cef10 con 0x7f66ac15b3d0 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.029+0000 7f66b2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac0cc9b0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66ac15b3d0 0x7f66ac0d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66ac15b3d0 0x7f66ac0d1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46860/0 (socket says 172.21.15.2:46860) 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 -- 172.21.15.2:0/3330855558 learned_addr learned my addr 172.21.15.2:0/3330855558 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b2000640 1 --2- 172.21.15.2:0/3330855558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac0cfb10 0x7f66ac0b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 -- 172.21.15.2:0/3330855558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac0cfb10 msgr2=0x7f66ac0b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 --2- 172.21.15.2:0/3330855558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac0cfb10 0x7f66ac0b0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 -- 172.21.15.2:0/3330855558 >> v1:172.21.15.137:6789/0 conn(0x7f66ac0cc3f0 legacy=0x7f66ac0cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.030+0000 7f66b1600640 1 -- 172.21.15.2:0/3330855558 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ac0ccc90 con 0x7f66ac15b3d0 2024-01-15T17:58:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.031+0000 7f66b1600640 1 --2- 172.21.15.2:0/3330855558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac15b3d0 0x7f66ac0d1940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f669c009870 tx=0x7f669c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6741626a4e7e990 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66abe00640 1 -- 172.21.15.2:0/3330855558 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c031640 con 0x7f66ac15b3d0 2024-01-15T17:58:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66abe00640 1 -- 172.21.15.2:0/3330855558 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c0317a0 con 0x7f66ac15b3d0 2024-01-15T17:58:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66abe00640 1 -- 172.21.15.2:0/3330855558 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c031b30 con 0x7f66ac15b3d0 2024-01-15T17:58:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 -- 172.21.15.2:0/3330855558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac15b3d0 msgr2=0x7f66ac0d1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 --2- 172.21.15.2:0/3330855558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac15b3d0 0x7f66ac0d1940 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f669c009870 tx=0x7f669c02b8f0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 -- 172.21.15.2:0/3330855558 shutdown_connections 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 --2- 172.21.15.2:0/3330855558 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac0cfb10 0x7f66ac0b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 --2- 172.21.15.2:0/3330855558 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac15b3d0 0x7f66ac0d1940 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 -- 172.21.15.2:0/3330855558 >> 172.21.15.2:0/3330855558 conn(0x7f66ac0d0e20 msgr2=0x7f66ac0d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.032+0000 7f66b2a00640 1 -- 172.21.15.2:0/3330855558 shutdown_connections 2024-01-15T17:58:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.033+0000 7f66b2a00640 1 -- 172.21.15.2:0/3330855558 wait complete. 2024-01-15T17:58:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.034+0000 7f66b2a00640 1 Processor -- start 2024-01-15T17:58:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.034+0000 7f66b2a00640 1 -- start start 2024-01-15T17:58:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.035+0000 7f66b2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66ac0cc3f0 0x7f66ac0ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.035+0000 7f66b2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 0x7f66ac1f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.035+0000 7f66b2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac15b3d0 0x7f66ac146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.035+0000 7f66b2a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac0ca170 con 0x7f66ac0cfb10 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.035+0000 7f66b2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac0b6980 con 0x7f66ac0cc3f0 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.035+0000 7f66b2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac0c9af0 con 0x7f66ac15b3d0 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 0x7f66ac1f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 0x7f66ac1f25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46872/0 (socket says 172.21.15.2:46872) 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 -- 172.21.15.2:0/92051056 learned_addr learned my addr 172.21.15.2:0/92051056 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b2000640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac15b3d0 0x7f66ac146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b1600640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66ac0cc3f0 0x7f66ac0ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 -- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac15b3d0 msgr2=0x7f66ac146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac15b3d0 0x7f66ac146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 -- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66ac0cc3f0 msgr2=0x7f66ac0ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66ac0cc3f0 0x7f66ac0ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 -- 172.21.15.2:0/92051056 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c0094d0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.036+0000 7f66b0c00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 0x7f66ac1f25b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f669400c400 tx=0x7f669400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.037+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66940052d0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.037+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6694005450 con 0x7f66ac0cfb10 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.037+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694007260 con 0x7f66ac0cfb10 2024-01-15T17:58:59.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.037+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac0b66a0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.037+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac1f2af0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.038+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac0c9950 con 0x7f66ac0cfb10 2024-01-15T17:58:59.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.039+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f66940073c0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.040+0000 7f66a9600640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668406b2c0 0x7f668406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.040+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6694088ec0 con 0x7f66ac0cfb10 2024-01-15T17:58:59.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.041+0000 7f66b1600640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668406b2c0 0x7f668406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.041+0000 7f66b1600640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668406b2c0 0x7f668406d780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f669c036000 tx=0x7f669c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:59.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.044+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66ac0c9950 con 0x7f66ac0cfb10 2024-01-15T17:58:59.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.240+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66ac0b09f0 con 0x7f668406b2c0 2024-01-15T17:58:59.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.267+0000 7f66a9600640 1 -- 172.21.15.2:0/92051056 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281387 (secure 0 0 0) 0x7f669c044400 con 0x7f668406b2c0 2024-01-15T17:58:59.276 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:59.279 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668406b2c0 msgr2=0x7f668406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668406b2c0 0x7f668406d780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f669c036000 tx=0x7f669c03c040 comp rx=0 tx=0).stop 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 msgr2=0x7f66ac1f25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 0x7f66ac1f25b0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f669400c400 tx=0x7f669400c8c0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 shutdown_connections 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668406b2c0 0x7f668406d780 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66ac15b3d0 0x7f66ac146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66ac0cfb10 0x7f66ac1f25b0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 --2- 172.21.15.2:0/92051056 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66ac0cc3f0 0x7f66ac0ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.273+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 >> 172.21.15.2:0/92051056 conn(0x7f66ac0d0e20 msgr2=0x7f66ac0d1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.274+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 shutdown_connections 2024-01-15T17:58:59.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.274+0000 7f66b2a00640 1 -- 172.21.15.2:0/92051056 wait complete. 2024-01-15T17:58:59.300 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-01-15T17:58:57.399478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236820,"kb_used_data":4004,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737900,"statfs":{"total":773094113280,"available":772851609600,"internally_reserved":0,"allocated":4100096,"data_stored":2293207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.655393"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768649+0000","last_change":"2024-01-15T17:58:53.768649+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768649+0000","last_undegraded":"2024-01-15T17:58:53.768649+0000","last_fullsized":"2024-01-15T17:58:53.768649+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765915+0000","last_change":"2024-01-15T17:58:53.765915+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765915+0000","last_undegraded":"2024-01-15T17:58:53.765915+0000","last_fullsized":"2024-01-15T17:58:53.765915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756460+0000","last_change":"2024-01-15T17:58:53.756460+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756460+0000","last_undegraded":"2024-01-15T17:58:53.756460+0000","last_fullsized":"2024-01-15T17:58:53.756460+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767200+0000","last_change":"2024-01-15T17:58:53.767200+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767200+0000","last_undegraded":"2024-01-15T17:58:53.767200+0000","last_fullsized":"2024-01-15T17:58:53.767200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762688+0000","last_change":"2024-01-15T17:58:53.762688+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762688+0000","last_undegraded":"2024-01-15T17:58:53.762688+0000","last_fullsized":"2024-01-15T17:58:53.762688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.764285+0000","last_change":"2024-01-15T17:58:53.764285+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.764285+0000","last_undegraded":"2024-01-15T17:58:53.764285+0000","last_fullsized":"2024-01-15T17:58:53.764285+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770853+0000","last_change":"2024-01-15T17:58:53.770853+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770853+0000","last_undegraded":"2024-01-15T17:58:53.770853+0000","last_fullsized":"2024-01-15T17:58:53.770853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.779368+0000","last_change":"2024-01-15T17:58:53.779368+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.779368+0000","last_undegraded":"2024-01-15T17:58:53.779368+0000","last_fullsized":"2024-01-15T17:58:53.779368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760076+0000","last_change":"2024-01-15T17:58:53.760076+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760076+0000","last_undegraded":"2024-01-15T17:58:53.760076+0000","last_fullsized":"2024-01-15T17:58:53.760076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768131+0000","last_change":"2024-01-15T17:58:53.768131+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768131+0000","last_undegraded":"2024-01-15T17:58:53.768131+0000","last_fullsized":"2024-01-15T17:58:53.768131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777933+0000","last_change":"2024-01-15T17:58:53.777933+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777933+0000","last_undegraded":"2024-01-15T17:58:53.777933+0000","last_fullsized":"2024-01-15T17:58:53.777933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760794+0000","last_change":"2024-01-15T17:58:53.760794+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760794+0000","last_undegraded":"2024-01-15T17:58:53.760794+0000","last_fullsized":"2024-01-15T17:58:53.760794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761983+0000","last_change":"2024-01-15T17:58:53.761983+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761983+0000","last_undegraded":"2024-01-15T17:58:53.761983+0000","last_fullsized":"2024-01-15T17:58:53.761983+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763819+0000","last_change":"2024-01-15T17:58:53.763819+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763819+0000","last_undegraded":"2024-01-15T17:58:53.763819+0000","last_fullsized":"2024-01-15T17:58:53.763819+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756158+0000","last_change":"2024-01-15T17:58:53.756158+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756158+0000","last_undegraded":"2024-01-15T17:58:53.756158+0000","last_fullsized":"2024-01-15T17:58:53.756158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769648+0000","last_change":"2024-01-15T17:58:53.769648+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769648+0000","last_undegraded":"2024-01-15T17:58:53.769648+0000","last_fullsized":"2024-01-15T17:58:53.769648+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776565+0000","last_change":"2024-01-15T17:58:53.776565+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776565+0000","last_undegraded":"2024-01-15T17:58:53.776565+0000","last_fullsized":"2024-01-15T17:58:53.776565+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763960+0000","last_change":"2024-01-15T17:58:53.763960+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763960+0000","last_undegraded":"2024-01-15T17:58:53.763960+0000","last_fullsized":"2024-01-15T17:58:53.763960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758899+0000","last_change":"2024-01-15T17:58:53.758899+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758899+0000","last_undegraded":"2024-01-15T17:58:53.758899+0000","last_fullsized":"2024-01-15T17:58:53.758899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760041+0000","last_change":"2024-01-15T17:58:53.760041+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760041+0000","last_undegraded":"2024-01-15T17:58:53.760041+0000","last_fullsized":"2024-01-15T17:58:53.760041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759520+0000","last_change":"2024-01-15T17:58:53.759520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759520+0000","last_undegraded":"2024-01-15T17:58:53.759520+0000","last_fullsized":"2024-01-15T17:58:53.759520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756476+0000","last_change":"2024-01-15T17:58:53.756476+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756476+0000","last_undegraded":"2024-01-15T17:58:53.756476+0000","last_fullsized":"2024-01-15T17:58:53.756476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763859+0000","last_change":"2024-01-15T17:58:53.763859+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763859+0000","last_undegraded":"2024-01-15T17:58:53.763859+0000","last_fullsized":"2024-01-15T17:58:53.763859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757006+0000","last_change":"2024-01-15T17:58:53.757006+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757006+0000","last_undegraded":"2024-01-15T17:58:53.757006+0000","last_fullsized":"2024-01-15T17:58:53.757006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769359+0000","last_change":"2024-01-15T17:58:53.769359+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769359+0000","last_undegraded":"2024-01-15T17:58:53.769359+0000","last_fullsized":"2024-01-15T17:58:53.769359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758923+0000","last_change":"2024-01-15T17:58:53.758923+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758923+0000","last_undegraded":"2024-01-15T17:58:53.758923+0000","last_fullsized":"2024-01-15T17:58:53.758923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768774+0000","last_change":"2024-01-15T17:58:53.768774+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768774+0000","last_undegraded":"2024-01-15T17:58:53.768774+0000","last_fullsized":"2024-01-15T17:58:53.768774+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759857+0000","last_change":"2024-01-15T17:58:53.759857+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759857+0000","last_undegraded":"2024-01-15T17:58:53.759857+0000","last_fullsized":"2024-01-15T17:58:53.759857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758420+0000","last_change":"2024-01-15T17:58:53.758420+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758420+0000","last_undegraded":"2024-01-15T17:58:53.758420+0000","last_fullsized":"2024-01-15T17:58:53.758420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770607+0000","last_change":"2024-01-15T17:58:53.770607+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770607+0000","last_undegraded":"2024-01-15T17:58:53.770607+0000","last_fullsized":"2024-01-15T17:58:53.770607+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769816+0000","last_change":"2024-01-15T17:58:53.769816+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769816+0000","last_undegraded":"2024-01-15T17:58:53.769816+0000","last_fullsized":"2024-01-15T17:58:53.769816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777957+0000","last_change":"2024-01-15T17:58:53.777957+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777957+0000","last_undegraded":"2024-01-15T17:58:53.777957+0000","last_fullsized":"2024-01-15T17:58:53.777957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760467+0000","last_change":"2024-01-15T17:58:53.760467+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760467+0000","last_undegraded":"2024-01-15T17:58:53.760467+0000","last_fullsized":"2024-01-15T17:58:53.760467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781733+0000","last_change":"2024-01-15T17:58:53.779462+0000","last_active":"2024-01-15T17:58:53.781733+0000","last_peered":"2024-01-15T17:58:53.781733+0000","last_clean":"2024-01-15T17:58:53.781733+0000","last_became_active":"2024-01-15T17:58:53.779161+0000","last_became_peered":"2024-01-15T17:58:53.779161+0000","last_unstale":"2024-01-15T17:58:53.781733+0000","last_undegraded":"2024-01-15T17:58:53.781733+0000","last_fullsized":"2024-01-15T17:58:53.781733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767896+0000","last_change":"2024-01-15T17:58:53.767896+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767896+0000","last_undegraded":"2024-01-15T17:58:53.767896+0000","last_fullsized":"2024-01-15T17:58:53.767896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784085+0000","last_change":"2024-01-15T17:58:53.782887+0000","last_active":"2024-01-15T17:58:53.784085+0000","last_peered":"2024-01-15T17:58:53.784085+0000","last_clean":"2024-01-15T17:58:53.784085+0000","last_became_active":"2024-01-15T17:58:53.782599+0000","last_became_peered":"2024-01-15T17:58:53.782599+0000","last_unstale":"2024-01-15T17:58:53.784085+0000","last_undegraded":"2024-01-15T17:58:53.784085+0000","last_fullsized":"2024-01-15T17:58:53.784085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761593+0000","last_change":"2024-01-15T17:58:53.761593+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761593+0000","last_undegraded":"2024-01-15T17:58:53.761593+0000","last_fullsized":"2024-01-15T17:58:53.761593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782129+0000","last_change":"2024-01-15T17:58:53.779558+0000","last_active":"2024-01-15T17:58:53.782129+0000","last_peered":"2024-01-15T17:58:53.782129+0000","last_clean":"2024-01-15T17:58:53.782129+0000","last_became_active":"2024-01-15T17:58:53.779269+0000","last_became_peered":"2024-01-15T17:58:53.779269+0000","last_unstale":"2024-01-15T17:58:53.782129+0000","last_undegraded":"2024-01-15T17:58:53.782129+0000","last_fullsized":"2024-01-15T17:58:53.782129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:19.366878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768274+0000","last_change":"2024-01-15T17:58:53.768274+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768274+0000","last_undegraded":"2024-01-15T17:58:53.768274+0000","last_fullsized":"2024-01-15T17:58:53.768274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.778130+0000","last_change":"2024-01-15T17:58:53.776027+0000","last_active":"2024-01-15T17:58:53.778130+0000","last_peered":"2024-01-15T17:58:53.778130+0000","last_clean":"2024-01-15T17:58:53.778130+0000","last_became_active":"2024-01-15T17:58:53.775717+0000","last_became_peered":"2024-01-15T17:58:53.775717+0000","last_unstale":"2024-01-15T17:58:53.778130+0000","last_undegraded":"2024-01-15T17:58:53.778130+0000","last_fullsized":"2024-01-15T17:58:53.778130+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770108+0000","last_change":"2024-01-15T17:58:53.770108+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770108+0000","last_undegraded":"2024-01-15T17:58:53.770108+0000","last_fullsized":"2024-01-15T17:58:53.770108+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782834+0000","last_change":"2024-01-15T17:58:53.780792+0000","last_active":"2024-01-15T17:58:53.782834+0000","last_peered":"2024-01-15T17:58:53.782834+0000","last_clean":"2024-01-15T17:58:53.782834+0000","last_became_active":"2024-01-15T17:58:53.780523+0000","last_became_peered":"2024-01-15T17:58:53.780523+0000","last_unstale":"2024-01-15T17:58:53.782834+0000","last_undegraded":"2024-01-15T17:58:53.782834+0000","last_fullsized":"2024-01-15T17:58:53.782834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:18.235321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756009+0000","last_change":"2024-01-15T17:58:53.756009+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756009+0000","last_undegraded":"2024-01-15T17:58:53.756009+0000","last_fullsized":"2024-01-15T17:58:53.756009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782764+0000","last_change":"2024-01-15T17:58:53.778730+0000","last_active":"2024-01-15T17:58:53.782764+0000","last_peered":"2024-01-15T17:58:53.782764+0000","last_clean":"2024-01-15T17:58:53.782764+0000","last_became_active":"2024-01-15T17:58:53.777103+0000","last_became_peered":"2024-01-15T17:58:53.777103+0000","last_unstale":"2024-01-15T17:58:53.782764+0000","last_undegraded":"2024-01-15T17:58:53.782764+0000","last_fullsized":"2024-01-15T17:58:53.782764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:24:27.345667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760047+0000","last_change":"2024-01-15T17:58:53.760047+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760047+0000","last_undegraded":"2024-01-15T17:58:53.760047+0000","last_fullsized":"2024-01-15T17:58:53.760047+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.775615+0000","last_change":"2024-01-15T17:58:53.769315+0000","last_active":"2024-01-15T17:58:53.775615+0000","last_peered":"2024-01-15T17:58:53.775615+0000","last_clean":"2024-01-15T17:58:53.775615+0000","last_became_active":"2024-01-15T17:58:53.769028+0000","last_became_peered":"2024-01-15T17:58:53.769028+0000","last_unstale":"2024-01-15T17:58:53.775615+0000","last_undegraded":"2024-01-15T17:58:53.775615+0000","last_fullsized":"2024-01-15T17:58:53.775615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765380+0000","last_change":"2024-01-15T17:58:53.765380+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765380+0000","last_undegraded":"2024-01-15T17:58:53.765380+0000","last_fullsized":"2024-01-15T17:58:53.765380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781943+0000","last_change":"2024-01-15T17:58:53.778012+0000","last_active":"2024-01-15T17:58:53.781943+0000","last_peered":"2024-01-15T17:58:53.781943+0000","last_clean":"2024-01-15T17:58:53.781943+0000","last_became_active":"2024-01-15T17:58:53.776887+0000","last_became_peered":"2024-01-15T17:58:53.776887+0000","last_unstale":"2024-01-15T17:58:53.781943+0000","last_undegraded":"2024-01-15T17:58:53.781943+0000","last_fullsized":"2024-01-15T17:58:53.781943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:24.725739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757492+0000","last_change":"2024-01-15T17:58:53.757492+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757492+0000","last_undegraded":"2024-01-15T17:58:53.757492+0000","last_fullsized":"2024-01-15T17:58:53.757492+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787587+0000","last_change":"2024-01-15T17:58:53.786507+0000","last_active":"2024-01-15T17:58:53.787587+0000","last_peered":"2024-01-15T17:58:53.787587+0000","last_clean":"2024-01-15T17:58:53.787587+0000","last_became_active":"2024-01-15T17:58:53.786154+0000","last_became_peered":"2024-01-15T17:58:53.786154+0000","last_unstale":"2024-01-15T17:58:53.787587+0000","last_undegraded":"2024-01-15T17:58:53.787587+0000","last_fullsized":"2024-01-15T17:58:53.787587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:33.284238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760012+0000","last_change":"2024-01-15T17:58:53.760012+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760012+0000","last_undegraded":"2024-01-15T17:58:53.760012+0000","last_fullsized":"2024-01-15T17:58:53.760012+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781046+0000","last_change":"2024-01-15T17:58:53.779129+0000","last_active":"2024-01-15T17:58:53.781046+0000","last_peered":"2024-01-15T17:58:53.781046+0000","last_clean":"2024-01-15T17:58:53.781046+0000","last_became_active":"2024-01-15T17:58:53.778863+0000","last_became_peered":"2024-01-15T17:58:53.778863+0000","last_unstale":"2024-01-15T17:58:53.781046+0000","last_undegraded":"2024-01-15T17:58:53.781046+0000","last_fullsized":"2024-01-15T17:58:53.781046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758478+0000","last_change":"2024-01-15T17:58:53.758478+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758478+0000","last_undegraded":"2024-01-15T17:58:53.758478+0000","last_fullsized":"2024-01-15T17:58:53.758478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784603+0000","last_change":"2024-01-15T17:58:53.782469+0000","last_active":"2024-01-15T17:58:53.784603+0000","last_peered":"2024-01-15T17:58:53.784603+0000","last_clean":"2024-01-15T17:58:53.784603+0000","last_became_active":"2024-01-15T17:58:53.782219+0000","last_became_peered":"2024-01-15T17:58:53.782219+0000","last_unstale":"2024-01-15T17:58:53.784603+0000","last_undegraded":"2024-01-15T17:58:53.784603+0000","last_fullsized":"2024-01-15T17:58:53.784603+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758109+0000","last_change":"2024-01-15T17:58:53.758109+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758109+0000","last_undegraded":"2024-01-15T17:58:53.758109+0000","last_fullsized":"2024-01-15T17:58:53.758109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781322+0000","last_change":"2024-01-15T17:58:53.778698+0000","last_active":"2024-01-15T17:58:53.781322+0000","last_peered":"2024-01-15T17:58:53.781322+0000","last_clean":"2024-01-15T17:58:53.781322+0000","last_became_active":"2024-01-15T17:58:53.778416+0000","last_became_peered":"2024-01-15T17:58:53.778416+0000","last_unstale":"2024-01-15T17:58:53.781322+0000","last_undegraded":"2024-01-15T17:58:53.781322+0000","last_fullsized":"2024-01-15T17:58:53.781322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:40:50.242725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762084+0000","last_change":"2024-01-15T17:58:53.762084+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762084+0000","last_undegraded":"2024-01-15T17:58:53.762084+0000","last_fullsized":"2024-01-15T17:58:53.762084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782813+0000","last_change":"2024-01-15T17:58:53.780478+0000","last_active":"2024-01-15T17:58:53.782813+0000","last_peered":"2024-01-15T17:58:53.782813+0000","last_clean":"2024-01-15T17:58:53.782813+0000","last_became_active":"2024-01-15T17:58:53.777564+0000","last_became_peered":"2024-01-15T17:58:53.777564+0000","last_unstale":"2024-01-15T17:58:53.782813+0000","last_undegraded":"2024-01-15T17:58:53.782813+0000","last_fullsized":"2024-01-15T17:58:53.782813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:43.478841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757520+0000","last_change":"2024-01-15T17:58:53.757520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757520+0000","last_undegraded":"2024-01-15T17:58:53.757520+0000","last_fullsized":"2024-01-15T17:58:53.757520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783720+0000","last_change":"2024-01-15T17:58:53.777976+0000","last_active":"2024-01-15T17:58:53.783720+0000","last_peered":"2024-01-15T17:58:53.783720+0000","last_clean":"2024-01-15T17:58:53.783720+0000","last_became_active":"2024-01-15T17:58:53.777658+0000","last_became_peered":"2024-01-15T17:58:53.777658+0000","last_unstale":"2024-01-15T17:58:53.783720+0000","last_undegraded":"2024-01-15T17:58:53.783720+0000","last_fullsized":"2024-01-15T17:58:53.783720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776953+0000","last_change":"2024-01-15T17:58:53.776953+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776953+0000","last_undegraded":"2024-01-15T17:58:53.776953+0000","last_fullsized":"2024-01-15T17:58:53.776953+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.785944+0000","last_change":"2024-01-15T17:58:53.782063+0000","last_active":"2024-01-15T17:58:53.785944+0000","last_peered":"2024-01-15T17:58:53.785944+0000","last_clean":"2024-01-15T17:58:53.785944+0000","last_became_active":"2024-01-15T17:58:53.781761+0000","last_became_peered":"2024-01-15T17:58:53.781761+0000","last_unstale":"2024-01-15T17:58:53.785944+0000","last_undegraded":"2024-01-15T17:58:53.785944+0000","last_fullsized":"2024-01-15T17:58:53.785944+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:05:52.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768343+0000","last_change":"2024-01-15T17:58:53.768343+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768343+0000","last_undegraded":"2024-01-15T17:58:53.768343+0000","last_fullsized":"2024-01-15T17:58:53.768343+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779694+0000","last_change":"2024-01-15T17:58:53.775656+0000","last_active":"2024-01-15T17:58:53.779694+0000","last_peered":"2024-01-15T17:58:53.779694+0000","last_clean":"2024-01-15T17:58:53.779694+0000","last_became_active":"2024-01-15T17:58:53.775310+0000","last_became_peered":"2024-01-15T17:58:53.775310+0000","last_unstale":"2024-01-15T17:58:53.779694+0000","last_undegraded":"2024-01-15T17:58:53.779694+0000","last_fullsized":"2024-01-15T17:58:53.779694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:16:51.863117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763189+0000","last_change":"2024-01-15T17:58:53.763189+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763189+0000","last_undegraded":"2024-01-15T17:58:53.763189+0000","last_fullsized":"2024-01-15T17:58:53.763189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783884+0000","last_change":"2024-01-15T17:58:53.779965+0000","last_active":"2024-01-15T17:58:53.783884+0000","last_peered":"2024-01-15T17:58:53.783884+0000","last_clean":"2024-01-15T17:58:53.783884+0000","last_became_active":"2024-01-15T17:58:53.779597+0000","last_became_peered":"2024-01-15T17:58:53.779597+0000","last_unstale":"2024-01-15T17:58:53.783884+0000","last_undegraded":"2024-01-15T17:58:53.783884+0000","last_fullsized":"2024-01-15T17:58:53.783884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762156+0000","last_change":"2024-01-15T17:58:53.762156+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762156+0000","last_undegraded":"2024-01-15T17:58:53.762156+0000","last_fullsized":"2024-01-15T17:58:53.762156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779888+0000","last_change":"2024-01-15T17:58:53.774956+0000","last_active":"2024-01-15T17:58:53.779888+0000","last_peered":"2024-01-15T17:58:53.779888+0000","last_clean":"2024-01-15T17:58:53.779888+0000","last_became_active":"2024-01-15T17:58:53.774680+0000","last_became_peered":"2024-01-15T17:58:53.774680+0000","last_unstale":"2024-01-15T17:58:53.779888+0000","last_undegraded":"2024-01-15T17:58:53.779888+0000","last_fullsized":"2024-01-15T17:58:53.779888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.766670+0000","last_change":"2024-01-15T17:58:53.766670+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.766670+0000","last_undegraded":"2024-01-15T17:58:53.766670+0000","last_fullsized":"2024-01-15T17:58:53.766670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779743+0000","last_change":"2024-01-15T17:58:53.772674+0000","last_active":"2024-01-15T17:58:53.779743+0000","last_peered":"2024-01-15T17:58:53.779743+0000","last_clean":"2024-01-15T17:58:53.779743+0000","last_became_active":"2024-01-15T17:58:53.769008+0000","last_became_peered":"2024-01-15T17:58:53.769008+0000","last_unstale":"2024-01-15T17:58:53.779743+0000","last_undegraded":"2024-01-15T17:58:53.779743+0000","last_fullsized":"2024-01-15T17:58:53.779743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:09.191112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761263+0000","last_change":"2024-01-15T17:58:53.761263+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761263+0000","last_undegraded":"2024-01-15T17:58:53.761263+0000","last_fullsized":"2024-01-15T17:58:53.761263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782726+0000","last_change":"2024-01-15T17:58:53.779480+0000","last_active":"2024-01-15T17:58:53.782726+0000","last_peered":"2024-01-15T17:58:53.782726+0000","last_clean":"2024-01-15T17:58:53.782726+0000","last_became_active":"2024-01-15T17:58:53.778673+0000","last_became_peered":"2024-01-15T17:58:53.778673+0000","last_unstale":"2024-01-15T17:58:53.782726+0000","last_undegraded":"2024-01-15T17:58:53.782726+0000","last_fullsized":"2024-01-15T17:58:53.782726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:40.298878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758081+0000","last_change":"2024-01-15T17:58:53.758081+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758081+0000","last_undegraded":"2024-01-15T17:58:53.758081+0000","last_fullsized":"2024-01-15T17:58:53.758081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781876+0000","last_change":"2024-01-15T17:58:53.780996+0000","last_active":"2024-01-15T17:58:53.781876+0000","last_peered":"2024-01-15T17:58:53.781876+0000","last_clean":"2024-01-15T17:58:53.781876+0000","last_became_active":"2024-01-15T17:58:53.778816+0000","last_became_peered":"2024-01-15T17:58:53.778816+0000","last_unstale":"2024-01-15T17:58:53.781876+0000","last_undegraded":"2024-01-15T17:58:53.781876+0000","last_fullsized":"2024-01-15T17:58:53.781876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758792+0000","last_change":"2024-01-15T17:58:53.758792+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758792+0000","last_undegraded":"2024-01-15T17:58:53.758792+0000","last_fullsized":"2024-01-15T17:58:53.758792+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786081+0000","last_change":"2024-01-15T17:58:53.784613+0000","last_active":"2024-01-15T17:58:53.786081+0000","last_peered":"2024-01-15T17:58:53.786081+0000","last_clean":"2024-01-15T17:58:53.786081+0000","last_became_active":"2024-01-15T17:58:53.784295+0000","last_became_peered":"2024-01-15T17:58:53.784295+0000","last_unstale":"2024-01-15T17:58:53.786081+0000","last_undegraded":"2024-01-15T17:58:53.786081+0000","last_fullsized":"2024-01-15T17:58:53.786081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757313+0000","last_change":"2024-01-15T17:58:53.757313+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757313+0000","last_undegraded":"2024-01-15T17:58:53.757313+0000","last_fullsized":"2024-01-15T17:58:53.757313+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786057+0000","last_change":"2024-01-15T17:58:53.785047+0000","last_active":"2024-01-15T17:58:53.786057+0000","last_peered":"2024-01-15T17:58:53.786057+0000","last_clean":"2024-01-15T17:58:53.786057+0000","last_became_active":"2024-01-15T17:58:53.784573+0000","last_became_peered":"2024-01-15T17:58:53.784573+0000","last_unstale":"2024-01-15T17:58:53.786057+0000","last_undegraded":"2024-01-15T17:58:53.786057+0000","last_fullsized":"2024-01-15T17:58:53.786057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:07.495003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771821+0000","last_change":"2024-01-15T17:58:53.771821+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771821+0000","last_undegraded":"2024-01-15T17:58:53.771821+0000","last_fullsized":"2024-01-15T17:58:53.771821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784703+0000","last_change":"2024-01-15T17:58:53.782707+0000","last_active":"2024-01-15T17:58:53.784703+0000","last_peered":"2024-01-15T17:58:53.784703+0000","last_clean":"2024-01-15T17:58:53.784703+0000","last_became_active":"2024-01-15T17:58:53.782425+0000","last_became_peered":"2024-01-15T17:58:53.782425+0000","last_unstale":"2024-01-15T17:58:53.784703+0000","last_undegraded":"2024-01-15T17:58:53.784703+0000","last_fullsized":"2024-01-15T17:58:53.784703+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:57.822622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762554+0000","last_change":"2024-01-15T17:58:53.762554+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762554+0000","last_undegraded":"2024-01-15T17:58:53.762554+0000","last_fullsized":"2024-01-15T17:58:53.762554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783352+0000","last_change":"2024-01-15T17:58:53.776979+0000","last_active":"2024-01-15T17:58:53.783352+0000","last_peered":"2024-01-15T17:58:53.783352+0000","last_clean":"2024-01-15T17:58:53.783352+0000","last_became_active":"2024-01-15T17:58:53.776609+0000","last_became_peered":"2024-01-15T17:58:53.776609+0000","last_unstale":"2024-01-15T17:58:53.783352+0000","last_undegraded":"2024-01-15T17:58:53.783352+0000","last_fullsized":"2024-01-15T17:58:53.783352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761363+0000","last_change":"2024-01-15T17:58:53.761363+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761363+0000","last_undegraded":"2024-01-15T17:58:53.761363+0000","last_fullsized":"2024-01-15T17:58:53.761363+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779213+0000","last_change":"2024-01-15T17:58:53.777358+0000","last_active":"2024-01-15T17:58:53.779213+0000","last_peered":"2024-01-15T17:58:53.779213+0000","last_clean":"2024-01-15T17:58:53.779213+0000","last_became_active":"2024-01-15T17:58:53.777077+0000","last_became_peered":"2024-01-15T17:58:53.777077+0000","last_unstale":"2024-01-15T17:58:53.779213+0000","last_undegraded":"2024-01-15T17:58:53.779213+0000","last_fullsized":"2024-01-15T17:58:53.779213+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760665+0000","last_change":"2024-01-15T17:58:53.760665+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760665+0000","last_undegraded":"2024-01-15T17:58:53.760665+0000","last_fullsized":"2024-01-15T17:58:53.760665+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.777700+0000","last_change":"2024-01-15T17:58:53.775372+0000","last_active":"2024-01-15T17:58:53.777700+0000","last_peered":"2024-01-15T17:58:53.777700+0000","last_clean":"2024-01-15T17:58:53.777700+0000","last_became_active":"2024-01-15T17:58:53.775056+0000","last_became_peered":"2024-01-15T17:58:53.775056+0000","last_unstale":"2024-01-15T17:58:53.777700+0000","last_undegraded":"2024-01-15T17:58:53.777700+0000","last_fullsized":"2024-01-15T17:58:53.777700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770135+0000","last_change":"2024-01-15T17:58:53.770135+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770135+0000","last_undegraded":"2024-01-15T17:58:53.770135+0000","last_fullsized":"2024-01-15T17:58:53.770135+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.772530+0000","last_change":"2024-01-15T17:58:53.770121+0000","last_active":"2024-01-15T17:58:53.772530+0000","last_peered":"2024-01-15T17:58:53.772530+0000","last_clean":"2024-01-15T17:58:53.772530+0000","last_became_active":"2024-01-15T17:58:53.769805+0000","last_became_peered":"2024-01-15T17:58:53.769805+0000","last_unstale":"2024-01-15T17:58:53.772530+0000","last_undegraded":"2024-01-15T17:58:53.772530+0000","last_fullsized":"2024-01-15T17:58:53.772530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758192+0000","last_change":"2024-01-15T17:58:53.758192+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758192+0000","last_undegraded":"2024-01-15T17:58:53.758192+0000","last_fullsized":"2024-01-15T17:58:53.758192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782268+0000","last_change":"2024-01-15T17:58:53.776755+0000","last_active":"2024-01-15T17:58:53.782268+0000","last_peered":"2024-01-15T17:58:53.782268+0000","last_clean":"2024-01-15T17:58:53.782268+0000","last_became_active":"2024-01-15T17:58:53.776423+0000","last_became_peered":"2024-01-15T17:58:53.776423+0000","last_unstale":"2024-01-15T17:58:53.782268+0000","last_undegraded":"2024-01-15T17:58:53.782268+0000","last_fullsized":"2024-01-15T17:58:53.782268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756946+0000","last_change":"2024-01-15T17:58:53.756946+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756946+0000","last_undegraded":"2024-01-15T17:58:53.756946+0000","last_fullsized":"2024-01-15T17:58:53.756946+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779312+0000","last_change":"2024-01-15T17:58:53.770125+0000","last_active":"2024-01-15T17:58:53.779312+0000","last_peered":"2024-01-15T17:58:53.779312+0000","last_clean":"2024-01-15T17:58:53.779312+0000","last_became_active":"2024-01-15T17:58:53.769709+0000","last_became_peered":"2024-01-15T17:58:53.769709+0000","last_unstale":"2024-01-15T17:58:53.779312+0000","last_undegraded":"2024-01-15T17:58:53.779312+0000","last_fullsized":"2024-01-15T17:58:53.779312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:17.227945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761458+0000","last_change":"2024-01-15T17:58:53.761458+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761458+0000","last_undegraded":"2024-01-15T17:58:53.761458+0000","last_fullsized":"2024-01-15T17:58:53.761458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787274+0000","last_change":"2024-01-15T17:58:53.786548+0000","last_active":"2024-01-15T17:58:53.787274+0000","last_peered":"2024-01-15T17:58:53.787274+0000","last_clean":"2024-01-15T17:58:53.787274+0000","last_became_active":"2024-01-15T17:58:53.786220+0000","last_became_peered":"2024-01-15T17:58:53.786220+0000","last_unstale":"2024-01-15T17:58:53.787274+0000","last_undegraded":"2024-01-15T17:58:53.787274+0000","last_fullsized":"2024-01-15T17:58:53.787274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:06:45.660613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771119+0000","last_change":"2024-01-15T17:58:53.771119+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771119+0000","last_undegraded":"2024-01-15T17:58:53.771119+0000","last_fullsized":"2024-01-15T17:58:53.771119+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784763+0000","last_change":"2024-01-15T17:58:53.783339+0000","last_active":"2024-01-15T17:58:53.784763+0000","last_peered":"2024-01-15T17:58:53.784763+0000","last_clean":"2024-01-15T17:58:53.784763+0000","last_became_active":"2024-01-15T17:58:53.783028+0000","last_became_peered":"2024-01-15T17:58:53.783028+0000","last_unstale":"2024-01-15T17:58:53.784763+0000","last_undegraded":"2024-01-15T17:58:53.784763+0000","last_fullsized":"2024-01-15T17:58:53.784763+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":173,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T17:58:52.760774+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:58:52.760774+0000","last_peered":"2024-01-15T17:58:52.760774+0000","last_clean":"2024-01-15T17:58:52.760774+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:58:52.760774+0000","last_undegraded":"2024-01-15T17:58:52.760774+0000","last_fullsized":"2024-01-15T17:58:52.760774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.6970000000000001,"5min":7.6970000000000001,"15min":7.6970000000000001},"last":1.339}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":1.196},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":1.5760000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.53}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.694}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":7.4809999999999999,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.2619999999999996,"5min":4.2619999999999996,"15min":4.2619999999999996},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.462},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":7.798,"5min":7.798,"15min":7.798},"last":1.4299999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":7.968,"5min":7.968,"15min":7.968},"last":1.611},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.6439999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.5009999999999999}]},{"osd":1,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.321},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.4019999999999999}]},{"osd":2,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.997}]},{"osd":3,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.198},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.4359999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.073,"5min":1.073,"15min":1.073},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.4710000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.653}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.323},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.1639999999999999}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.48},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.431}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.0669999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.5329999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":1.272},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.5840000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.752}]},{"osd":1,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.613},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.1100000000000001}]},{"osd":3,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.4259999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.526}]},{"osd":7,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.7230000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.008}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.161}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.419},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.5369999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.7}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.49},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.669}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5780000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.105}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5150000000000001,"5min":2.5150000000000001,"15min":2.5150000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.002}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.7150000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.675,"15min":1.675},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":1.758},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.7559999999999998,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.5649999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":1.149,"5min":1.149,"15min":1.149},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":1.847},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.6279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.78800000000000003}]},{"osd":3,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.306}]},{"osd":4,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.379},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.5189999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.4430000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.6699999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"last":1.54},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.339}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":962560,"data_stored":731777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.069},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.196},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.337},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.2450000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.389}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.4570000000000001}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.569},{"interface":"front","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.679}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:58:59.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:58:59.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.565+0000 7f6302800640 1 Processor -- start 2024-01-15T17:58:59.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.566+0000 7f6302800640 1 -- start start 2024-01-15T17:58:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.567+0000 7f6302800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc141880 0x7f62fc070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.567+0000 7f6302800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f62fc15ec60 0x7f62fc15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.567+0000 7f6302800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f62fc0cd490 0x7f62fc0cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.567+0000 7f6302800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc071f20 con 0x7f62fc0cd490 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.567+0000 7f6302800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62fc06c6b0 con 0x7f62fc15ec60 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.567+0000 7f6302800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc15f620 con 0x7f62fc141880 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f62fc0cd490 0x7f62fc0cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f62fc0cd490 0x7f62fc0cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46874/0 (socket says 172.21.15.2:46874) 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 -- 172.21.15.2:0/3978929243 learned_addr learned my addr 172.21.15.2:0/3978929243 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6300a00640 1 --2- 172.21.15.2:0/3978929243 >> v2:172.21.15.137:3300/0 conn(0x7f62fc15ec60 0x7f62fc15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301400640 1 --2- 172.21.15.2:0/3978929243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc141880 0x7f62fc070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 -- 172.21.15.2:0/3978929243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc141880 msgr2=0x7f62fc070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 --2- 172.21.15.2:0/3978929243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc141880 0x7f62fc070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 -- 172.21.15.2:0/3978929243 >> v2:172.21.15.137:3300/0 conn(0x7f62fc15ec60 msgr2=0x7f62fc15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:58:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 --2- 172.21.15.2:0/3978929243 >> v2:172.21.15.137:3300/0 conn(0x7f62fc15ec60 0x7f62fc15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.568+0000 7f6301e00640 1 -- 172.21.15.2:0/3978929243 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62fc071960 con 0x7f62fc0cd490 2024-01-15T17:58:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.570+0000 7f6301e00640 1 --2- 172.21.15.2:0/3978929243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc0cd490 0x7f62fc0cd920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f62fc06e5e0 tx=0x7f62f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb313383c9983d5c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.570+0000 7f62fbe00640 1 -- 172.21.15.2:0/3978929243 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f402c9e0 con 0x7f62fc0cd490 2024-01-15T17:58:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.570+0000 7f62fbe00640 1 -- 172.21.15.2:0/3978929243 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f402cb40 con 0x7f62fc0cd490 2024-01-15T17:58:59.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.570+0000 7f62fbe00640 1 -- 172.21.15.2:0/3978929243 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f40343f0 con 0x7f62fc0cd490 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 -- 172.21.15.2:0/3978929243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc0cd490 msgr2=0x7f62fc0cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 --2- 172.21.15.2:0/3978929243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc0cd490 0x7f62fc0cd920 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f62fc06e5e0 tx=0x7f62f402be40 comp rx=0 tx=0).stop 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 -- 172.21.15.2:0/3978929243 shutdown_connections 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 --2- 172.21.15.2:0/3978929243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc0cd490 0x7f62fc0cd920 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 --2- 172.21.15.2:0/3978929243 >> v2:172.21.15.137:3300/0 conn(0x7f62fc15ec60 0x7f62fc15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 --2- 172.21.15.2:0/3978929243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc141880 0x7f62fc070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 -- 172.21.15.2:0/3978929243 >> 172.21.15.2:0/3978929243 conn(0x7f62fc0b0840 msgr2=0x7f62fc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 -- 172.21.15.2:0/3978929243 shutdown_connections 2024-01-15T17:58:59.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.571+0000 7f6302800640 1 -- 172.21.15.2:0/3978929243 wait complete. 2024-01-15T17:58:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.573+0000 7f6302800640 1 Processor -- start 2024-01-15T17:58:59.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.573+0000 7f6302800640 1 -- start start 2024-01-15T17:58:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6302800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc0cd490 0x7f62fc156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6302800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 0x7f62fc1f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6302800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc15ec60 0x7f62fc155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6302800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc15abe0 con 0x7f62fc141880 2024-01-15T17:58:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6302800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc0b3270 con 0x7f62fc15ec60 2024-01-15T17:58:59.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6302800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc156760 con 0x7f62fc0cd490 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 0x7f62fc1f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 0x7f62fc1f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46880/0 (socket says 172.21.15.2:46880) 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 -- 172.21.15.2:0/3265323642 learned_addr learned my addr 172.21.15.2:0/3265323642 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6301400640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc0cd490 0x7f62fc156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6301e00640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc15ec60 0x7f62fc155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 -- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc0cd490 msgr2=0x7f62fc156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc0cd490 0x7f62fc156220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 -- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc15ec60 msgr2=0x7f62fc155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc15ec60 0x7f62fc155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.574+0000 7f6300a00640 1 -- 172.21.15.2:0/3265323642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f4009b30 con 0x7f62fc141880 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f6301400640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc0cd490 0x7f62fc156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:58:59.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f6300a00640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 0x7f62fc1f6880 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d800 tx=0x7f62e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400b6c0 con 0x7f62fc141880 2024-01-15T17:58:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e400b820 con 0x7f62fc141880 2024-01-15T17:58:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62e400bbb0 con 0x7f62fc141880 2024-01-15T17:58:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc15ae10 con 0x7f62fc141880 2024-01-15T17:58:59.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.575+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc15b3d0 con 0x7f62fc141880 2024-01-15T17:58:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.577+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc074770 con 0x7f62fc141880 2024-01-15T17:58:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.577+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f62e400bd10 con 0x7f62fc141880 2024-01-15T17:58:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.578+0000 7f62f9600640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62d406b2c0 0x7f62d406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:58:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.578+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62e408a210 con 0x7f62fc141880 2024-01-15T17:58:59.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.579+0000 7f6301400640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62d406b2c0 0x7f62d406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:58:59.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.580+0000 7f6301400640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62d406b2c0 0x7f62d406d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006fd0 tx=0x7f62ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:58:59.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.583+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62e4054fe0 con 0x7f62fc141880 2024-01-15T17:58:59.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.775+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62fc141e80 con 0x7f62d406b2c0 2024-01-15T17:58:59.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.797+0000 7f62f9600640 1 -- 172.21.15.2:0/3265323642 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281387 (secure 0 0 0) 0x7f62ec004030 con 0x7f62d406b2c0 2024-01-15T17:58:59.804 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:58:59.806 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.801+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62d406b2c0 msgr2=0x7f62d406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.801+0000 7f6302800640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62d406b2c0 0x7f62d406d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f62ec006fd0 tx=0x7f62ec008040 comp rx=0 tx=0).stop 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.801+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 msgr2=0x7f62fc1f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.801+0000 7f6302800640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 0x7f62fc1f6880 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f62e400d800 tx=0x7f62e400dcc0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 shutdown_connections 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62d406b2c0 0x7f62d406d780 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62fc15ec60 0x7f62fc155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62fc141880 0x7f62fc1f6880 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 --2- 172.21.15.2:0/3265323642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62fc0cd490 0x7f62fc156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:58:59.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 >> 172.21.15.2:0/3265323642 conn(0x7f62fc0b0840 msgr2=0x7f62fc0aedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:58:59.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 shutdown_connections 2024-01-15T17:58:59.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:58:59.802+0000 7f6302800640 1 -- 172.21.15.2:0/3265323642 wait complete. 2024-01-15T17:58:59.822 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-01-15T17:58:59.400090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236820,"kb_used_data":4004,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737900,"statfs":{"total":773094113280,"available":772851609600,"internally_reserved":0,"allocated":4100096,"data_stored":2293207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001942"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768649+0000","last_change":"2024-01-15T17:58:53.768649+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768649+0000","last_undegraded":"2024-01-15T17:58:53.768649+0000","last_fullsized":"2024-01-15T17:58:53.768649+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765915+0000","last_change":"2024-01-15T17:58:53.765915+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765915+0000","last_undegraded":"2024-01-15T17:58:53.765915+0000","last_fullsized":"2024-01-15T17:58:53.765915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756460+0000","last_change":"2024-01-15T17:58:53.756460+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756460+0000","last_undegraded":"2024-01-15T17:58:53.756460+0000","last_fullsized":"2024-01-15T17:58:53.756460+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767200+0000","last_change":"2024-01-15T17:58:53.767200+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767200+0000","last_undegraded":"2024-01-15T17:58:53.767200+0000","last_fullsized":"2024-01-15T17:58:53.767200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762688+0000","last_change":"2024-01-15T17:58:53.762688+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762688+0000","last_undegraded":"2024-01-15T17:58:53.762688+0000","last_fullsized":"2024-01-15T17:58:53.762688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.764285+0000","last_change":"2024-01-15T17:58:53.764285+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.764285+0000","last_undegraded":"2024-01-15T17:58:53.764285+0000","last_fullsized":"2024-01-15T17:58:53.764285+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770853+0000","last_change":"2024-01-15T17:58:53.770853+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770853+0000","last_undegraded":"2024-01-15T17:58:53.770853+0000","last_fullsized":"2024-01-15T17:58:53.770853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.779368+0000","last_change":"2024-01-15T17:58:53.779368+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.779368+0000","last_undegraded":"2024-01-15T17:58:53.779368+0000","last_fullsized":"2024-01-15T17:58:53.779368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760076+0000","last_change":"2024-01-15T17:58:53.760076+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760076+0000","last_undegraded":"2024-01-15T17:58:53.760076+0000","last_fullsized":"2024-01-15T17:58:53.760076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768131+0000","last_change":"2024-01-15T17:58:53.768131+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768131+0000","last_undegraded":"2024-01-15T17:58:53.768131+0000","last_fullsized":"2024-01-15T17:58:53.768131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777933+0000","last_change":"2024-01-15T17:58:53.777933+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777933+0000","last_undegraded":"2024-01-15T17:58:53.777933+0000","last_fullsized":"2024-01-15T17:58:53.777933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760794+0000","last_change":"2024-01-15T17:58:53.760794+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760794+0000","last_undegraded":"2024-01-15T17:58:53.760794+0000","last_fullsized":"2024-01-15T17:58:53.760794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761983+0000","last_change":"2024-01-15T17:58:53.761983+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761983+0000","last_undegraded":"2024-01-15T17:58:53.761983+0000","last_fullsized":"2024-01-15T17:58:53.761983+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763819+0000","last_change":"2024-01-15T17:58:53.763819+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763819+0000","last_undegraded":"2024-01-15T17:58:53.763819+0000","last_fullsized":"2024-01-15T17:58:53.763819+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756158+0000","last_change":"2024-01-15T17:58:53.756158+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756158+0000","last_undegraded":"2024-01-15T17:58:53.756158+0000","last_fullsized":"2024-01-15T17:58:53.756158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769648+0000","last_change":"2024-01-15T17:58:53.769648+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769648+0000","last_undegraded":"2024-01-15T17:58:53.769648+0000","last_fullsized":"2024-01-15T17:58:53.769648+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776565+0000","last_change":"2024-01-15T17:58:53.776565+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776565+0000","last_undegraded":"2024-01-15T17:58:53.776565+0000","last_fullsized":"2024-01-15T17:58:53.776565+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763960+0000","last_change":"2024-01-15T17:58:53.763960+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763960+0000","last_undegraded":"2024-01-15T17:58:53.763960+0000","last_fullsized":"2024-01-15T17:58:53.763960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758899+0000","last_change":"2024-01-15T17:58:53.758899+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758899+0000","last_undegraded":"2024-01-15T17:58:53.758899+0000","last_fullsized":"2024-01-15T17:58:53.758899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760041+0000","last_change":"2024-01-15T17:58:53.760041+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760041+0000","last_undegraded":"2024-01-15T17:58:53.760041+0000","last_fullsized":"2024-01-15T17:58:53.760041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759520+0000","last_change":"2024-01-15T17:58:53.759520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759520+0000","last_undegraded":"2024-01-15T17:58:53.759520+0000","last_fullsized":"2024-01-15T17:58:53.759520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756476+0000","last_change":"2024-01-15T17:58:53.756476+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756476+0000","last_undegraded":"2024-01-15T17:58:53.756476+0000","last_fullsized":"2024-01-15T17:58:53.756476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763859+0000","last_change":"2024-01-15T17:58:53.763859+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763859+0000","last_undegraded":"2024-01-15T17:58:53.763859+0000","last_fullsized":"2024-01-15T17:58:53.763859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757006+0000","last_change":"2024-01-15T17:58:53.757006+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757006+0000","last_undegraded":"2024-01-15T17:58:53.757006+0000","last_fullsized":"2024-01-15T17:58:53.757006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769359+0000","last_change":"2024-01-15T17:58:53.769359+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769359+0000","last_undegraded":"2024-01-15T17:58:53.769359+0000","last_fullsized":"2024-01-15T17:58:53.769359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758923+0000","last_change":"2024-01-15T17:58:53.758923+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758923+0000","last_undegraded":"2024-01-15T17:58:53.758923+0000","last_fullsized":"2024-01-15T17:58:53.758923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768774+0000","last_change":"2024-01-15T17:58:53.768774+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768774+0000","last_undegraded":"2024-01-15T17:58:53.768774+0000","last_fullsized":"2024-01-15T17:58:53.768774+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759857+0000","last_change":"2024-01-15T17:58:53.759857+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759857+0000","last_undegraded":"2024-01-15T17:58:53.759857+0000","last_fullsized":"2024-01-15T17:58:53.759857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758420+0000","last_change":"2024-01-15T17:58:53.758420+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758420+0000","last_undegraded":"2024-01-15T17:58:53.758420+0000","last_fullsized":"2024-01-15T17:58:53.758420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770607+0000","last_change":"2024-01-15T17:58:53.770607+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770607+0000","last_undegraded":"2024-01-15T17:58:53.770607+0000","last_fullsized":"2024-01-15T17:58:53.770607+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769816+0000","last_change":"2024-01-15T17:58:53.769816+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769816+0000","last_undegraded":"2024-01-15T17:58:53.769816+0000","last_fullsized":"2024-01-15T17:58:53.769816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777957+0000","last_change":"2024-01-15T17:58:53.777957+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777957+0000","last_undegraded":"2024-01-15T17:58:53.777957+0000","last_fullsized":"2024-01-15T17:58:53.777957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760467+0000","last_change":"2024-01-15T17:58:53.760467+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760467+0000","last_undegraded":"2024-01-15T17:58:53.760467+0000","last_fullsized":"2024-01-15T17:58:53.760467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781733+0000","last_change":"2024-01-15T17:58:53.779462+0000","last_active":"2024-01-15T17:58:53.781733+0000","last_peered":"2024-01-15T17:58:53.781733+0000","last_clean":"2024-01-15T17:58:53.781733+0000","last_became_active":"2024-01-15T17:58:53.779161+0000","last_became_peered":"2024-01-15T17:58:53.779161+0000","last_unstale":"2024-01-15T17:58:53.781733+0000","last_undegraded":"2024-01-15T17:58:53.781733+0000","last_fullsized":"2024-01-15T17:58:53.781733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767896+0000","last_change":"2024-01-15T17:58:53.767896+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767896+0000","last_undegraded":"2024-01-15T17:58:53.767896+0000","last_fullsized":"2024-01-15T17:58:53.767896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784085+0000","last_change":"2024-01-15T17:58:53.782887+0000","last_active":"2024-01-15T17:58:53.784085+0000","last_peered":"2024-01-15T17:58:53.784085+0000","last_clean":"2024-01-15T17:58:53.784085+0000","last_became_active":"2024-01-15T17:58:53.782599+0000","last_became_peered":"2024-01-15T17:58:53.782599+0000","last_unstale":"2024-01-15T17:58:53.784085+0000","last_undegraded":"2024-01-15T17:58:53.784085+0000","last_fullsized":"2024-01-15T17:58:53.784085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761593+0000","last_change":"2024-01-15T17:58:53.761593+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761593+0000","last_undegraded":"2024-01-15T17:58:53.761593+0000","last_fullsized":"2024-01-15T17:58:53.761593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782129+0000","last_change":"2024-01-15T17:58:53.779558+0000","last_active":"2024-01-15T17:58:53.782129+0000","last_peered":"2024-01-15T17:58:53.782129+0000","last_clean":"2024-01-15T17:58:53.782129+0000","last_became_active":"2024-01-15T17:58:53.779269+0000","last_became_peered":"2024-01-15T17:58:53.779269+0000","last_unstale":"2024-01-15T17:58:53.782129+0000","last_undegraded":"2024-01-15T17:58:53.782129+0000","last_fullsized":"2024-01-15T17:58:53.782129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:19.366878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768274+0000","last_change":"2024-01-15T17:58:53.768274+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768274+0000","last_undegraded":"2024-01-15T17:58:53.768274+0000","last_fullsized":"2024-01-15T17:58:53.768274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.778130+0000","last_change":"2024-01-15T17:58:53.776027+0000","last_active":"2024-01-15T17:58:53.778130+0000","last_peered":"2024-01-15T17:58:53.778130+0000","last_clean":"2024-01-15T17:58:53.778130+0000","last_became_active":"2024-01-15T17:58:53.775717+0000","last_became_peered":"2024-01-15T17:58:53.775717+0000","last_unstale":"2024-01-15T17:58:53.778130+0000","last_undegraded":"2024-01-15T17:58:53.778130+0000","last_fullsized":"2024-01-15T17:58:53.778130+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770108+0000","last_change":"2024-01-15T17:58:53.770108+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770108+0000","last_undegraded":"2024-01-15T17:58:53.770108+0000","last_fullsized":"2024-01-15T17:58:53.770108+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782834+0000","last_change":"2024-01-15T17:58:53.780792+0000","last_active":"2024-01-15T17:58:53.782834+0000","last_peered":"2024-01-15T17:58:53.782834+0000","last_clean":"2024-01-15T17:58:53.782834+0000","last_became_active":"2024-01-15T17:58:53.780523+0000","last_became_peered":"2024-01-15T17:58:53.780523+0000","last_unstale":"2024-01-15T17:58:53.782834+0000","last_undegraded":"2024-01-15T17:58:53.782834+0000","last_fullsized":"2024-01-15T17:58:53.782834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:18.235321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756009+0000","last_change":"2024-01-15T17:58:53.756009+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756009+0000","last_undegraded":"2024-01-15T17:58:53.756009+0000","last_fullsized":"2024-01-15T17:58:53.756009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782764+0000","last_change":"2024-01-15T17:58:53.778730+0000","last_active":"2024-01-15T17:58:53.782764+0000","last_peered":"2024-01-15T17:58:53.782764+0000","last_clean":"2024-01-15T17:58:53.782764+0000","last_became_active":"2024-01-15T17:58:53.777103+0000","last_became_peered":"2024-01-15T17:58:53.777103+0000","last_unstale":"2024-01-15T17:58:53.782764+0000","last_undegraded":"2024-01-15T17:58:53.782764+0000","last_fullsized":"2024-01-15T17:58:53.782764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:24:27.345667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760047+0000","last_change":"2024-01-15T17:58:53.760047+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760047+0000","last_undegraded":"2024-01-15T17:58:53.760047+0000","last_fullsized":"2024-01-15T17:58:53.760047+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.775615+0000","last_change":"2024-01-15T17:58:53.769315+0000","last_active":"2024-01-15T17:58:53.775615+0000","last_peered":"2024-01-15T17:58:53.775615+0000","last_clean":"2024-01-15T17:58:53.775615+0000","last_became_active":"2024-01-15T17:58:53.769028+0000","last_became_peered":"2024-01-15T17:58:53.769028+0000","last_unstale":"2024-01-15T17:58:53.775615+0000","last_undegraded":"2024-01-15T17:58:53.775615+0000","last_fullsized":"2024-01-15T17:58:53.775615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765380+0000","last_change":"2024-01-15T17:58:53.765380+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765380+0000","last_undegraded":"2024-01-15T17:58:53.765380+0000","last_fullsized":"2024-01-15T17:58:53.765380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781943+0000","last_change":"2024-01-15T17:58:53.778012+0000","last_active":"2024-01-15T17:58:53.781943+0000","last_peered":"2024-01-15T17:58:53.781943+0000","last_clean":"2024-01-15T17:58:53.781943+0000","last_became_active":"2024-01-15T17:58:53.776887+0000","last_became_peered":"2024-01-15T17:58:53.776887+0000","last_unstale":"2024-01-15T17:58:53.781943+0000","last_undegraded":"2024-01-15T17:58:53.781943+0000","last_fullsized":"2024-01-15T17:58:53.781943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:24.725739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757492+0000","last_change":"2024-01-15T17:58:53.757492+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757492+0000","last_undegraded":"2024-01-15T17:58:53.757492+0000","last_fullsized":"2024-01-15T17:58:53.757492+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787587+0000","last_change":"2024-01-15T17:58:53.786507+0000","last_active":"2024-01-15T17:58:53.787587+0000","last_peered":"2024-01-15T17:58:53.787587+0000","last_clean":"2024-01-15T17:58:53.787587+0000","last_became_active":"2024-01-15T17:58:53.786154+0000","last_became_peered":"2024-01-15T17:58:53.786154+0000","last_unstale":"2024-01-15T17:58:53.787587+0000","last_undegraded":"2024-01-15T17:58:53.787587+0000","last_fullsized":"2024-01-15T17:58:53.787587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:33.284238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760012+0000","last_change":"2024-01-15T17:58:53.760012+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760012+0000","last_undegraded":"2024-01-15T17:58:53.760012+0000","last_fullsized":"2024-01-15T17:58:53.760012+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781046+0000","last_change":"2024-01-15T17:58:53.779129+0000","last_active":"2024-01-15T17:58:53.781046+0000","last_peered":"2024-01-15T17:58:53.781046+0000","last_clean":"2024-01-15T17:58:53.781046+0000","last_became_active":"2024-01-15T17:58:53.778863+0000","last_became_peered":"2024-01-15T17:58:53.778863+0000","last_unstale":"2024-01-15T17:58:53.781046+0000","last_undegraded":"2024-01-15T17:58:53.781046+0000","last_fullsized":"2024-01-15T17:58:53.781046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758478+0000","last_change":"2024-01-15T17:58:53.758478+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758478+0000","last_undegraded":"2024-01-15T17:58:53.758478+0000","last_fullsized":"2024-01-15T17:58:53.758478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784603+0000","last_change":"2024-01-15T17:58:53.782469+0000","last_active":"2024-01-15T17:58:53.784603+0000","last_peered":"2024-01-15T17:58:53.784603+0000","last_clean":"2024-01-15T17:58:53.784603+0000","last_became_active":"2024-01-15T17:58:53.782219+0000","last_became_peered":"2024-01-15T17:58:53.782219+0000","last_unstale":"2024-01-15T17:58:53.784603+0000","last_undegraded":"2024-01-15T17:58:53.784603+0000","last_fullsized":"2024-01-15T17:58:53.784603+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758109+0000","last_change":"2024-01-15T17:58:53.758109+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758109+0000","last_undegraded":"2024-01-15T17:58:53.758109+0000","last_fullsized":"2024-01-15T17:58:53.758109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781322+0000","last_change":"2024-01-15T17:58:53.778698+0000","last_active":"2024-01-15T17:58:53.781322+0000","last_peered":"2024-01-15T17:58:53.781322+0000","last_clean":"2024-01-15T17:58:53.781322+0000","last_became_active":"2024-01-15T17:58:53.778416+0000","last_became_peered":"2024-01-15T17:58:53.778416+0000","last_unstale":"2024-01-15T17:58:53.781322+0000","last_undegraded":"2024-01-15T17:58:53.781322+0000","last_fullsized":"2024-01-15T17:58:53.781322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:40:50.242725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762084+0000","last_change":"2024-01-15T17:58:53.762084+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762084+0000","last_undegraded":"2024-01-15T17:58:53.762084+0000","last_fullsized":"2024-01-15T17:58:53.762084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782813+0000","last_change":"2024-01-15T17:58:53.780478+0000","last_active":"2024-01-15T17:58:53.782813+0000","last_peered":"2024-01-15T17:58:53.782813+0000","last_clean":"2024-01-15T17:58:53.782813+0000","last_became_active":"2024-01-15T17:58:53.777564+0000","last_became_peered":"2024-01-15T17:58:53.777564+0000","last_unstale":"2024-01-15T17:58:53.782813+0000","last_undegraded":"2024-01-15T17:58:53.782813+0000","last_fullsized":"2024-01-15T17:58:53.782813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:43.478841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757520+0000","last_change":"2024-01-15T17:58:53.757520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757520+0000","last_undegraded":"2024-01-15T17:58:53.757520+0000","last_fullsized":"2024-01-15T17:58:53.757520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783720+0000","last_change":"2024-01-15T17:58:53.777976+0000","last_active":"2024-01-15T17:58:53.783720+0000","last_peered":"2024-01-15T17:58:53.783720+0000","last_clean":"2024-01-15T17:58:53.783720+0000","last_became_active":"2024-01-15T17:58:53.777658+0000","last_became_peered":"2024-01-15T17:58:53.777658+0000","last_unstale":"2024-01-15T17:58:53.783720+0000","last_undegraded":"2024-01-15T17:58:53.783720+0000","last_fullsized":"2024-01-15T17:58:53.783720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776953+0000","last_change":"2024-01-15T17:58:53.776953+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776953+0000","last_undegraded":"2024-01-15T17:58:53.776953+0000","last_fullsized":"2024-01-15T17:58:53.776953+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.785944+0000","last_change":"2024-01-15T17:58:53.782063+0000","last_active":"2024-01-15T17:58:53.785944+0000","last_peered":"2024-01-15T17:58:53.785944+0000","last_clean":"2024-01-15T17:58:53.785944+0000","last_became_active":"2024-01-15T17:58:53.781761+0000","last_became_peered":"2024-01-15T17:58:53.781761+0000","last_unstale":"2024-01-15T17:58:53.785944+0000","last_undegraded":"2024-01-15T17:58:53.785944+0000","last_fullsized":"2024-01-15T17:58:53.785944+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:05:52.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768343+0000","last_change":"2024-01-15T17:58:53.768343+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768343+0000","last_undegraded":"2024-01-15T17:58:53.768343+0000","last_fullsized":"2024-01-15T17:58:53.768343+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779694+0000","last_change":"2024-01-15T17:58:53.775656+0000","last_active":"2024-01-15T17:58:53.779694+0000","last_peered":"2024-01-15T17:58:53.779694+0000","last_clean":"2024-01-15T17:58:53.779694+0000","last_became_active":"2024-01-15T17:58:53.775310+0000","last_became_peered":"2024-01-15T17:58:53.775310+0000","last_unstale":"2024-01-15T17:58:53.779694+0000","last_undegraded":"2024-01-15T17:58:53.779694+0000","last_fullsized":"2024-01-15T17:58:53.779694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:16:51.863117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763189+0000","last_change":"2024-01-15T17:58:53.763189+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763189+0000","last_undegraded":"2024-01-15T17:58:53.763189+0000","last_fullsized":"2024-01-15T17:58:53.763189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783884+0000","last_change":"2024-01-15T17:58:53.779965+0000","last_active":"2024-01-15T17:58:53.783884+0000","last_peered":"2024-01-15T17:58:53.783884+0000","last_clean":"2024-01-15T17:58:53.783884+0000","last_became_active":"2024-01-15T17:58:53.779597+0000","last_became_peered":"2024-01-15T17:58:53.779597+0000","last_unstale":"2024-01-15T17:58:53.783884+0000","last_undegraded":"2024-01-15T17:58:53.783884+0000","last_fullsized":"2024-01-15T17:58:53.783884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762156+0000","last_change":"2024-01-15T17:58:53.762156+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762156+0000","last_undegraded":"2024-01-15T17:58:53.762156+0000","last_fullsized":"2024-01-15T17:58:53.762156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779888+0000","last_change":"2024-01-15T17:58:53.774956+0000","last_active":"2024-01-15T17:58:53.779888+0000","last_peered":"2024-01-15T17:58:53.779888+0000","last_clean":"2024-01-15T17:58:53.779888+0000","last_became_active":"2024-01-15T17:58:53.774680+0000","last_became_peered":"2024-01-15T17:58:53.774680+0000","last_unstale":"2024-01-15T17:58:53.779888+0000","last_undegraded":"2024-01-15T17:58:53.779888+0000","last_fullsized":"2024-01-15T17:58:53.779888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.766670+0000","last_change":"2024-01-15T17:58:53.766670+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.766670+0000","last_undegraded":"2024-01-15T17:58:53.766670+0000","last_fullsized":"2024-01-15T17:58:53.766670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779743+0000","last_change":"2024-01-15T17:58:53.772674+0000","last_active":"2024-01-15T17:58:53.779743+0000","last_peered":"2024-01-15T17:58:53.779743+0000","last_clean":"2024-01-15T17:58:53.779743+0000","last_became_active":"2024-01-15T17:58:53.769008+0000","last_became_peered":"2024-01-15T17:58:53.769008+0000","last_unstale":"2024-01-15T17:58:53.779743+0000","last_undegraded":"2024-01-15T17:58:53.779743+0000","last_fullsized":"2024-01-15T17:58:53.779743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:09.191112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761263+0000","last_change":"2024-01-15T17:58:53.761263+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761263+0000","last_undegraded":"2024-01-15T17:58:53.761263+0000","last_fullsized":"2024-01-15T17:58:53.761263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782726+0000","last_change":"2024-01-15T17:58:53.779480+0000","last_active":"2024-01-15T17:58:53.782726+0000","last_peered":"2024-01-15T17:58:53.782726+0000","last_clean":"2024-01-15T17:58:53.782726+0000","last_became_active":"2024-01-15T17:58:53.778673+0000","last_became_peered":"2024-01-15T17:58:53.778673+0000","last_unstale":"2024-01-15T17:58:53.782726+0000","last_undegraded":"2024-01-15T17:58:53.782726+0000","last_fullsized":"2024-01-15T17:58:53.782726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:40.298878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758081+0000","last_change":"2024-01-15T17:58:53.758081+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758081+0000","last_undegraded":"2024-01-15T17:58:53.758081+0000","last_fullsized":"2024-01-15T17:58:53.758081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781876+0000","last_change":"2024-01-15T17:58:53.780996+0000","last_active":"2024-01-15T17:58:53.781876+0000","last_peered":"2024-01-15T17:58:53.781876+0000","last_clean":"2024-01-15T17:58:53.781876+0000","last_became_active":"2024-01-15T17:58:53.778816+0000","last_became_peered":"2024-01-15T17:58:53.778816+0000","last_unstale":"2024-01-15T17:58:53.781876+0000","last_undegraded":"2024-01-15T17:58:53.781876+0000","last_fullsized":"2024-01-15T17:58:53.781876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758792+0000","last_change":"2024-01-15T17:58:53.758792+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758792+0000","last_undegraded":"2024-01-15T17:58:53.758792+0000","last_fullsized":"2024-01-15T17:58:53.758792+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786081+0000","last_change":"2024-01-15T17:58:53.784613+0000","last_active":"2024-01-15T17:58:53.786081+0000","last_peered":"2024-01-15T17:58:53.786081+0000","last_clean":"2024-01-15T17:58:53.786081+0000","last_became_active":"2024-01-15T17:58:53.784295+0000","last_became_peered":"2024-01-15T17:58:53.784295+0000","last_unstale":"2024-01-15T17:58:53.786081+0000","last_undegraded":"2024-01-15T17:58:53.786081+0000","last_fullsized":"2024-01-15T17:58:53.786081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757313+0000","last_change":"2024-01-15T17:58:53.757313+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757313+0000","last_undegraded":"2024-01-15T17:58:53.757313+0000","last_fullsized":"2024-01-15T17:58:53.757313+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786057+0000","last_change":"2024-01-15T17:58:53.785047+0000","last_active":"2024-01-15T17:58:53.786057+0000","last_peered":"2024-01-15T17:58:53.786057+0000","last_clean":"2024-01-15T17:58:53.786057+0000","last_became_active":"2024-01-15T17:58:53.784573+0000","last_became_peered":"2024-01-15T17:58:53.784573+0000","last_unstale":"2024-01-15T17:58:53.786057+0000","last_undegraded":"2024-01-15T17:58:53.786057+0000","last_fullsized":"2024-01-15T17:58:53.786057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:07.495003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771821+0000","last_change":"2024-01-15T17:58:53.771821+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771821+0000","last_undegraded":"2024-01-15T17:58:53.771821+0000","last_fullsized":"2024-01-15T17:58:53.771821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784703+0000","last_change":"2024-01-15T17:58:53.782707+0000","last_active":"2024-01-15T17:58:53.784703+0000","last_peered":"2024-01-15T17:58:53.784703+0000","last_clean":"2024-01-15T17:58:53.784703+0000","last_became_active":"2024-01-15T17:58:53.782425+0000","last_became_peered":"2024-01-15T17:58:53.782425+0000","last_unstale":"2024-01-15T17:58:53.784703+0000","last_undegraded":"2024-01-15T17:58:53.784703+0000","last_fullsized":"2024-01-15T17:58:53.784703+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:57.822622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762554+0000","last_change":"2024-01-15T17:58:53.762554+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762554+0000","last_undegraded":"2024-01-15T17:58:53.762554+0000","last_fullsized":"2024-01-15T17:58:53.762554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783352+0000","last_change":"2024-01-15T17:58:53.776979+0000","last_active":"2024-01-15T17:58:53.783352+0000","last_peered":"2024-01-15T17:58:53.783352+0000","last_clean":"2024-01-15T17:58:53.783352+0000","last_became_active":"2024-01-15T17:58:53.776609+0000","last_became_peered":"2024-01-15T17:58:53.776609+0000","last_unstale":"2024-01-15T17:58:53.783352+0000","last_undegraded":"2024-01-15T17:58:53.783352+0000","last_fullsized":"2024-01-15T17:58:53.783352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761363+0000","last_change":"2024-01-15T17:58:53.761363+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761363+0000","last_undegraded":"2024-01-15T17:58:53.761363+0000","last_fullsized":"2024-01-15T17:58:53.761363+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779213+0000","last_change":"2024-01-15T17:58:53.777358+0000","last_active":"2024-01-15T17:58:53.779213+0000","last_peered":"2024-01-15T17:58:53.779213+0000","last_clean":"2024-01-15T17:58:53.779213+0000","last_became_active":"2024-01-15T17:58:53.777077+0000","last_became_peered":"2024-01-15T17:58:53.777077+0000","last_unstale":"2024-01-15T17:58:53.779213+0000","last_undegraded":"2024-01-15T17:58:53.779213+0000","last_fullsized":"2024-01-15T17:58:53.779213+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760665+0000","last_change":"2024-01-15T17:58:53.760665+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760665+0000","last_undegraded":"2024-01-15T17:58:53.760665+0000","last_fullsized":"2024-01-15T17:58:53.760665+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.777700+0000","last_change":"2024-01-15T17:58:53.775372+0000","last_active":"2024-01-15T17:58:53.777700+0000","last_peered":"2024-01-15T17:58:53.777700+0000","last_clean":"2024-01-15T17:58:53.777700+0000","last_became_active":"2024-01-15T17:58:53.775056+0000","last_became_peered":"2024-01-15T17:58:53.775056+0000","last_unstale":"2024-01-15T17:58:53.777700+0000","last_undegraded":"2024-01-15T17:58:53.777700+0000","last_fullsized":"2024-01-15T17:58:53.777700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770135+0000","last_change":"2024-01-15T17:58:53.770135+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770135+0000","last_undegraded":"2024-01-15T17:58:53.770135+0000","last_fullsized":"2024-01-15T17:58:53.770135+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.772530+0000","last_change":"2024-01-15T17:58:53.770121+0000","last_active":"2024-01-15T17:58:53.772530+0000","last_peered":"2024-01-15T17:58:53.772530+0000","last_clean":"2024-01-15T17:58:53.772530+0000","last_became_active":"2024-01-15T17:58:53.769805+0000","last_became_peered":"2024-01-15T17:58:53.769805+0000","last_unstale":"2024-01-15T17:58:53.772530+0000","last_undegraded":"2024-01-15T17:58:53.772530+0000","last_fullsized":"2024-01-15T17:58:53.772530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758192+0000","last_change":"2024-01-15T17:58:53.758192+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758192+0000","last_undegraded":"2024-01-15T17:58:53.758192+0000","last_fullsized":"2024-01-15T17:58:53.758192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782268+0000","last_change":"2024-01-15T17:58:53.776755+0000","last_active":"2024-01-15T17:58:53.782268+0000","last_peered":"2024-01-15T17:58:53.782268+0000","last_clean":"2024-01-15T17:58:53.782268+0000","last_became_active":"2024-01-15T17:58:53.776423+0000","last_became_peered":"2024-01-15T17:58:53.776423+0000","last_unstale":"2024-01-15T17:58:53.782268+0000","last_undegraded":"2024-01-15T17:58:53.782268+0000","last_fullsized":"2024-01-15T17:58:53.782268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756946+0000","last_change":"2024-01-15T17:58:53.756946+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756946+0000","last_undegraded":"2024-01-15T17:58:53.756946+0000","last_fullsized":"2024-01-15T17:58:53.756946+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779312+0000","last_change":"2024-01-15T17:58:53.770125+0000","last_active":"2024-01-15T17:58:53.779312+0000","last_peered":"2024-01-15T17:58:53.779312+0000","last_clean":"2024-01-15T17:58:53.779312+0000","last_became_active":"2024-01-15T17:58:53.769709+0000","last_became_peered":"2024-01-15T17:58:53.769709+0000","last_unstale":"2024-01-15T17:58:53.779312+0000","last_undegraded":"2024-01-15T17:58:53.779312+0000","last_fullsized":"2024-01-15T17:58:53.779312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:17.227945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761458+0000","last_change":"2024-01-15T17:58:53.761458+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761458+0000","last_undegraded":"2024-01-15T17:58:53.761458+0000","last_fullsized":"2024-01-15T17:58:53.761458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787274+0000","last_change":"2024-01-15T17:58:53.786548+0000","last_active":"2024-01-15T17:58:53.787274+0000","last_peered":"2024-01-15T17:58:53.787274+0000","last_clean":"2024-01-15T17:58:53.787274+0000","last_became_active":"2024-01-15T17:58:53.786220+0000","last_became_peered":"2024-01-15T17:58:53.786220+0000","last_unstale":"2024-01-15T17:58:53.787274+0000","last_undegraded":"2024-01-15T17:58:53.787274+0000","last_fullsized":"2024-01-15T17:58:53.787274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:06:45.660613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771119+0000","last_change":"2024-01-15T17:58:53.771119+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771119+0000","last_undegraded":"2024-01-15T17:58:53.771119+0000","last_fullsized":"2024-01-15T17:58:53.771119+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784763+0000","last_change":"2024-01-15T17:58:53.783339+0000","last_active":"2024-01-15T17:58:53.784763+0000","last_peered":"2024-01-15T17:58:53.784763+0000","last_clean":"2024-01-15T17:58:53.784763+0000","last_became_active":"2024-01-15T17:58:53.783028+0000","last_became_peered":"2024-01-15T17:58:53.783028+0000","last_unstale":"2024-01-15T17:58:53.784763+0000","last_undegraded":"2024-01-15T17:58:53.784763+0000","last_fullsized":"2024-01-15T17:58:53.784763+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":173,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T17:58:52.760774+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:58:52.760774+0000","last_peered":"2024-01-15T17:58:52.760774+0000","last_clean":"2024-01-15T17:58:52.760774+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:58:52.760774+0000","last_undegraded":"2024-01-15T17:58:52.760774+0000","last_fullsized":"2024-01-15T17:58:52.760774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.6970000000000001,"5min":7.6970000000000001,"15min":7.6970000000000001},"last":1.339}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":1.196},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":1.5760000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.53}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.694}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":7.4809999999999999,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.2619999999999996,"5min":4.2619999999999996,"15min":4.2619999999999996},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.462},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":7.798,"5min":7.798,"15min":7.798},"last":1.4299999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":7.968,"5min":7.968,"15min":7.968},"last":1.611},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.6439999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.5009999999999999}]},{"osd":1,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.321},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.4019999999999999}]},{"osd":2,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.997}]},{"osd":3,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.198},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.4359999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.073,"5min":1.073,"15min":1.073},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.4710000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.653}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.323},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.1639999999999999}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.48},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.431}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.0669999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.5329999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":1.272},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.5840000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.752}]},{"osd":1,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.613},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.1100000000000001}]},{"osd":3,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.4259999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.526}]},{"osd":7,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.7230000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.008}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.161}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.419},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.5369999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.7}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.49},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.669}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5780000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.105}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5150000000000001,"5min":2.5150000000000001,"15min":2.5150000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.002}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.7150000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.675,"15min":1.675},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":1.758},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.7559999999999998,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.5649999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":1.149,"5min":1.149,"15min":1.149},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":1.847},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.6279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.78800000000000003}]},{"osd":3,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.306}]},{"osd":4,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.379},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.5189999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.4430000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.6699999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"last":1.54},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.339}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":962560,"data_stored":731777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.069},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.196},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.337},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.2450000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.389}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.4570000000000001}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.569},{"interface":"front","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.679}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:58:59.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:59:00.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.085+0000 7f393cc00640 1 Processor -- start 2024-01-15T17:59:00.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.085+0000 7f393cc00640 1 -- start start 2024-01-15T17:59:00.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.087+0000 7f393cc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f393815f0e0 0x7f393806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.087+0000 7f393cc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39380716d0 con 0x7f3938141880 2024-01-15T17:59:00.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.087+0000 7f393cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3938071960 con 0x7f393815ec60 2024-01-15T17:59:00.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.087+0000 7f393cc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3938071f20 con 0x7f393815f0e0 2024-01-15T17:59:00.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.087+0000 7f3937e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f393815f0e0 0x7f393806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.087+0000 7f3937400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3938141880 0x7f3938070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32946/0 (socket says 172.21.15.2:32946) 2024-01-15T17:59:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.088+0000 7f3937400640 1 -- 172.21.15.2:0/9157736 learned_addr learned my addr 172.21.15.2:0/9157736 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.088+0000 7f3937e00640 1 -- 172.21.15.2:0/9157736 >> v1:172.21.15.137:6789/0 conn(0x7f393815ec60 legacy=0x7f393806cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.088+0000 7f3937e00640 1 -- 172.21.15.2:0/9157736 >> v1:172.21.15.2:6789/0 conn(0x7f3938141880 legacy=0x7f3938070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.088+0000 7f3937e00640 1 -- 172.21.15.2:0/9157736 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3938071c40 con 0x7f393815f0e0 2024-01-15T17:59:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.088+0000 7f3936a00640 1 -- 172.21.15.2:0/9157736 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485684397 0 0) 0x7f39380716d0 con 0x7f3938141880 2024-01-15T17:59:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.089+0000 7f3937e00640 1 --2- 172.21.15.2:0/9157736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815f0e0 0x7f393806d1e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f39300097a0 tx=0x7f393002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af83ec4b3e205cc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f3936a00640 1 -- 172.21.15.2:0/9157736 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393002e070 con 0x7f393815f0e0 2024-01-15T17:59:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f3936a00640 1 -- 172.21.15.2:0/9157736 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39300315a0 con 0x7f393815f0e0 2024-01-15T17:59:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f3936a00640 1 -- 172.21.15.2:0/9157736 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930031930 con 0x7f393815f0e0 2024-01-15T17:59:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 -- 172.21.15.2:0/9157736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815f0e0 msgr2=0x7f393806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 --2- 172.21.15.2:0/9157736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815f0e0 0x7f393806d1e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f39300097a0 tx=0x7f393002b840 comp rx=0 tx=0).stop 2024-01-15T17:59:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 -- 172.21.15.2:0/9157736 shutdown_connections 2024-01-15T17:59:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 --2- 172.21.15.2:0/9157736 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815f0e0 0x7f393806d1e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 -- 172.21.15.2:0/9157736 >> 172.21.15.2:0/9157736 conn(0x7f39380b0840 msgr2=0x7f39380b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 -- 172.21.15.2:0/9157736 shutdown_connections 2024-01-15T17:59:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.090+0000 7f393cc00640 1 -- 172.21.15.2:0/9157736 wait complete. 2024-01-15T17:59:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.092+0000 7f393cc00640 1 Processor -- start 2024-01-15T17:59:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.092+0000 7f393cc00640 1 -- start start 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.093+0000 7f393cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3938141880 0x7f393815d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.093+0000 7f393cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 0x7f39381f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.093+0000 7f393cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f393815f0e0 0x7f393813ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.093+0000 7f393cc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393806c6b0 con 0x7f393815ec60 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.093+0000 7f393cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39381f3650 con 0x7f3938141880 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.093+0000 7f393cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393813f190 con 0x7f393815f0e0 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 0x7f39381f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 0x7f39381f3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46896/0 (socket says 172.21.15.2:46896) 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 -- 172.21.15.2:0/607373758 learned_addr learned my addr 172.21.15.2:0/607373758 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f3937400640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3938141880 0x7f393815d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f3937e00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f393815f0e0 0x7f393813ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 -- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f393815f0e0 msgr2=0x7f393813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f393815f0e0 0x7f393813ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 -- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3938141880 msgr2=0x7f393815d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3938141880 0x7f393815d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 -- 172.21.15.2:0/607373758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3930009400 con 0x7f393815ec60 2024-01-15T17:59:00.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.094+0000 7f392e800640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 0x7f39381f3110 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f392000c410 tx=0x7f392000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.095+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39200035f0 con 0x7f393815ec60 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.095+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920003750 con 0x7f393815ec60 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.095+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393813f430 con 0x7f393815ec60 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.095+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920003b40 con 0x7f393815ec60 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.095+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393813f9f0 con 0x7f393815ec60 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.096+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3938066e50 con 0x7f393815ec60 2024-01-15T17:59:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.097+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3920003ca0 con 0x7f393815ec60 2024-01-15T17:59:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.098+0000 7f392fe00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f391006b2f0 0x7f391006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.098+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3920089060 con 0x7f393815ec60 2024-01-15T17:59:00.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.098+0000 7f3937400640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f391006b2f0 0x7f391006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.099+0000 7f3937400640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f391006b2f0 0x7f391006d7b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f39280012e0 tx=0x7f3928001700 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:00.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.102+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3920054010 con 0x7f393815ec60 2024-01-15T17:59:00.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.294+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f393815f530 con 0x7f391006b2f0 2024-01-15T17:59:00.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.318+0000 7f392fe00640 1 -- 172.21.15.2:0/607373758 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281387 (secure 0 0 0) 0x7f3928004de0 con 0x7f391006b2f0 2024-01-15T17:59:00.325 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:59:00.327 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:59:00.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f391006b2f0 msgr2=0x7f391006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f391006b2f0 0x7f391006d7b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f39280012e0 tx=0x7f3928001700 comp rx=0 tx=0).stop 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 msgr2=0x7f39381f3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 0x7f39381f3110 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f392000c410 tx=0x7f392000c8d0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 shutdown_connections 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f391006b2f0 0x7f391006d7b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f393815f0e0 0x7f393813ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f393815ec60 0x7f39381f3110 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 --2- 172.21.15.2:0/607373758 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3938141880 0x7f393815d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 >> 172.21.15.2:0/607373758 conn(0x7f39380b0840 msgr2=0x7f3938144520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 shutdown_connections 2024-01-15T17:59:00.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.324+0000 7f393cc00640 1 -- 172.21.15.2:0/607373758 wait complete. 2024-01-15T17:59:00.345 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-01-15T17:58:59.400090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236820,"kb_used_data":4004,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737900,"statfs":{"total":773094113280,"available":772851609600,"internally_reserved":0,"allocated":4100096,"data_stored":2293207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001942"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768649+0000","last_change":"2024-01-15T17:58:53.768649+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768649+0000","last_undegraded":"2024-01-15T17:58:53.768649+0000","last_fullsized":"2024-01-15T17:58:53.768649+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765915+0000","last_change":"2024-01-15T17:58:53.765915+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765915+0000","last_undegraded":"2024-01-15T17:58:53.765915+0000","last_fullsized":"2024-01-15T17:58:53.765915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756460+0000","last_change":"2024-01-15T17:58:53.756460+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756460+0000","last_undegraded":"2024-01-15T17:58:53.756460+0000","last_fullsized":"2024-01-15T17:58:53.756460+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767200+0000","last_change":"2024-01-15T17:58:53.767200+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767200+0000","last_undegraded":"2024-01-15T17:58:53.767200+0000","last_fullsized":"2024-01-15T17:58:53.767200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762688+0000","last_change":"2024-01-15T17:58:53.762688+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762688+0000","last_undegraded":"2024-01-15T17:58:53.762688+0000","last_fullsized":"2024-01-15T17:58:53.762688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.764285+0000","last_change":"2024-01-15T17:58:53.764285+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.764285+0000","last_undegraded":"2024-01-15T17:58:53.764285+0000","last_fullsized":"2024-01-15T17:58:53.764285+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770853+0000","last_change":"2024-01-15T17:58:53.770853+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770853+0000","last_undegraded":"2024-01-15T17:58:53.770853+0000","last_fullsized":"2024-01-15T17:58:53.770853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.779368+0000","last_change":"2024-01-15T17:58:53.779368+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.779368+0000","last_undegraded":"2024-01-15T17:58:53.779368+0000","last_fullsized":"2024-01-15T17:58:53.779368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760076+0000","last_change":"2024-01-15T17:58:53.760076+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760076+0000","last_undegraded":"2024-01-15T17:58:53.760076+0000","last_fullsized":"2024-01-15T17:58:53.760076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768131+0000","last_change":"2024-01-15T17:58:53.768131+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768131+0000","last_undegraded":"2024-01-15T17:58:53.768131+0000","last_fullsized":"2024-01-15T17:58:53.768131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777933+0000","last_change":"2024-01-15T17:58:53.777933+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777933+0000","last_undegraded":"2024-01-15T17:58:53.777933+0000","last_fullsized":"2024-01-15T17:58:53.777933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760794+0000","last_change":"2024-01-15T17:58:53.760794+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760794+0000","last_undegraded":"2024-01-15T17:58:53.760794+0000","last_fullsized":"2024-01-15T17:58:53.760794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761983+0000","last_change":"2024-01-15T17:58:53.761983+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761983+0000","last_undegraded":"2024-01-15T17:58:53.761983+0000","last_fullsized":"2024-01-15T17:58:53.761983+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763819+0000","last_change":"2024-01-15T17:58:53.763819+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763819+0000","last_undegraded":"2024-01-15T17:58:53.763819+0000","last_fullsized":"2024-01-15T17:58:53.763819+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756158+0000","last_change":"2024-01-15T17:58:53.756158+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756158+0000","last_undegraded":"2024-01-15T17:58:53.756158+0000","last_fullsized":"2024-01-15T17:58:53.756158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769648+0000","last_change":"2024-01-15T17:58:53.769648+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769648+0000","last_undegraded":"2024-01-15T17:58:53.769648+0000","last_fullsized":"2024-01-15T17:58:53.769648+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776565+0000","last_change":"2024-01-15T17:58:53.776565+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776565+0000","last_undegraded":"2024-01-15T17:58:53.776565+0000","last_fullsized":"2024-01-15T17:58:53.776565+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763960+0000","last_change":"2024-01-15T17:58:53.763960+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763960+0000","last_undegraded":"2024-01-15T17:58:53.763960+0000","last_fullsized":"2024-01-15T17:58:53.763960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758899+0000","last_change":"2024-01-15T17:58:53.758899+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758899+0000","last_undegraded":"2024-01-15T17:58:53.758899+0000","last_fullsized":"2024-01-15T17:58:53.758899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760041+0000","last_change":"2024-01-15T17:58:53.760041+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760041+0000","last_undegraded":"2024-01-15T17:58:53.760041+0000","last_fullsized":"2024-01-15T17:58:53.760041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759520+0000","last_change":"2024-01-15T17:58:53.759520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759520+0000","last_undegraded":"2024-01-15T17:58:53.759520+0000","last_fullsized":"2024-01-15T17:58:53.759520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756476+0000","last_change":"2024-01-15T17:58:53.756476+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756476+0000","last_undegraded":"2024-01-15T17:58:53.756476+0000","last_fullsized":"2024-01-15T17:58:53.756476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763859+0000","last_change":"2024-01-15T17:58:53.763859+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763859+0000","last_undegraded":"2024-01-15T17:58:53.763859+0000","last_fullsized":"2024-01-15T17:58:53.763859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757006+0000","last_change":"2024-01-15T17:58:53.757006+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757006+0000","last_undegraded":"2024-01-15T17:58:53.757006+0000","last_fullsized":"2024-01-15T17:58:53.757006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769359+0000","last_change":"2024-01-15T17:58:53.769359+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769359+0000","last_undegraded":"2024-01-15T17:58:53.769359+0000","last_fullsized":"2024-01-15T17:58:53.769359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758923+0000","last_change":"2024-01-15T17:58:53.758923+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758923+0000","last_undegraded":"2024-01-15T17:58:53.758923+0000","last_fullsized":"2024-01-15T17:58:53.758923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768774+0000","last_change":"2024-01-15T17:58:53.768774+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768774+0000","last_undegraded":"2024-01-15T17:58:53.768774+0000","last_fullsized":"2024-01-15T17:58:53.768774+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759857+0000","last_change":"2024-01-15T17:58:53.759857+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759857+0000","last_undegraded":"2024-01-15T17:58:53.759857+0000","last_fullsized":"2024-01-15T17:58:53.759857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758420+0000","last_change":"2024-01-15T17:58:53.758420+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758420+0000","last_undegraded":"2024-01-15T17:58:53.758420+0000","last_fullsized":"2024-01-15T17:58:53.758420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770607+0000","last_change":"2024-01-15T17:58:53.770607+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770607+0000","last_undegraded":"2024-01-15T17:58:53.770607+0000","last_fullsized":"2024-01-15T17:58:53.770607+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769816+0000","last_change":"2024-01-15T17:58:53.769816+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769816+0000","last_undegraded":"2024-01-15T17:58:53.769816+0000","last_fullsized":"2024-01-15T17:58:53.769816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777957+0000","last_change":"2024-01-15T17:58:53.777957+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777957+0000","last_undegraded":"2024-01-15T17:58:53.777957+0000","last_fullsized":"2024-01-15T17:58:53.777957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760467+0000","last_change":"2024-01-15T17:58:53.760467+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760467+0000","last_undegraded":"2024-01-15T17:58:53.760467+0000","last_fullsized":"2024-01-15T17:58:53.760467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781733+0000","last_change":"2024-01-15T17:58:53.779462+0000","last_active":"2024-01-15T17:58:53.781733+0000","last_peered":"2024-01-15T17:58:53.781733+0000","last_clean":"2024-01-15T17:58:53.781733+0000","last_became_active":"2024-01-15T17:58:53.779161+0000","last_became_peered":"2024-01-15T17:58:53.779161+0000","last_unstale":"2024-01-15T17:58:53.781733+0000","last_undegraded":"2024-01-15T17:58:53.781733+0000","last_fullsized":"2024-01-15T17:58:53.781733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767896+0000","last_change":"2024-01-15T17:58:53.767896+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767896+0000","last_undegraded":"2024-01-15T17:58:53.767896+0000","last_fullsized":"2024-01-15T17:58:53.767896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784085+0000","last_change":"2024-01-15T17:58:53.782887+0000","last_active":"2024-01-15T17:58:53.784085+0000","last_peered":"2024-01-15T17:58:53.784085+0000","last_clean":"2024-01-15T17:58:53.784085+0000","last_became_active":"2024-01-15T17:58:53.782599+0000","last_became_peered":"2024-01-15T17:58:53.782599+0000","last_unstale":"2024-01-15T17:58:53.784085+0000","last_undegraded":"2024-01-15T17:58:53.784085+0000","last_fullsized":"2024-01-15T17:58:53.784085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761593+0000","last_change":"2024-01-15T17:58:53.761593+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761593+0000","last_undegraded":"2024-01-15T17:58:53.761593+0000","last_fullsized":"2024-01-15T17:58:53.761593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782129+0000","last_change":"2024-01-15T17:58:53.779558+0000","last_active":"2024-01-15T17:58:53.782129+0000","last_peered":"2024-01-15T17:58:53.782129+0000","last_clean":"2024-01-15T17:58:53.782129+0000","last_became_active":"2024-01-15T17:58:53.779269+0000","last_became_peered":"2024-01-15T17:58:53.779269+0000","last_unstale":"2024-01-15T17:58:53.782129+0000","last_undegraded":"2024-01-15T17:58:53.782129+0000","last_fullsized":"2024-01-15T17:58:53.782129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:19.366878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768274+0000","last_change":"2024-01-15T17:58:53.768274+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768274+0000","last_undegraded":"2024-01-15T17:58:53.768274+0000","last_fullsized":"2024-01-15T17:58:53.768274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.778130+0000","last_change":"2024-01-15T17:58:53.776027+0000","last_active":"2024-01-15T17:58:53.778130+0000","last_peered":"2024-01-15T17:58:53.778130+0000","last_clean":"2024-01-15T17:58:53.778130+0000","last_became_active":"2024-01-15T17:58:53.775717+0000","last_became_peered":"2024-01-15T17:58:53.775717+0000","last_unstale":"2024-01-15T17:58:53.778130+0000","last_undegraded":"2024-01-15T17:58:53.778130+0000","last_fullsized":"2024-01-15T17:58:53.778130+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770108+0000","last_change":"2024-01-15T17:58:53.770108+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770108+0000","last_undegraded":"2024-01-15T17:58:53.770108+0000","last_fullsized":"2024-01-15T17:58:53.770108+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782834+0000","last_change":"2024-01-15T17:58:53.780792+0000","last_active":"2024-01-15T17:58:53.782834+0000","last_peered":"2024-01-15T17:58:53.782834+0000","last_clean":"2024-01-15T17:58:53.782834+0000","last_became_active":"2024-01-15T17:58:53.780523+0000","last_became_peered":"2024-01-15T17:58:53.780523+0000","last_unstale":"2024-01-15T17:58:53.782834+0000","last_undegraded":"2024-01-15T17:58:53.782834+0000","last_fullsized":"2024-01-15T17:58:53.782834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:18.235321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756009+0000","last_change":"2024-01-15T17:58:53.756009+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756009+0000","last_undegraded":"2024-01-15T17:58:53.756009+0000","last_fullsized":"2024-01-15T17:58:53.756009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782764+0000","last_change":"2024-01-15T17:58:53.778730+0000","last_active":"2024-01-15T17:58:53.782764+0000","last_peered":"2024-01-15T17:58:53.782764+0000","last_clean":"2024-01-15T17:58:53.782764+0000","last_became_active":"2024-01-15T17:58:53.777103+0000","last_became_peered":"2024-01-15T17:58:53.777103+0000","last_unstale":"2024-01-15T17:58:53.782764+0000","last_undegraded":"2024-01-15T17:58:53.782764+0000","last_fullsized":"2024-01-15T17:58:53.782764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:24:27.345667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760047+0000","last_change":"2024-01-15T17:58:53.760047+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760047+0000","last_undegraded":"2024-01-15T17:58:53.760047+0000","last_fullsized":"2024-01-15T17:58:53.760047+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.775615+0000","last_change":"2024-01-15T17:58:53.769315+0000","last_active":"2024-01-15T17:58:53.775615+0000","last_peered":"2024-01-15T17:58:53.775615+0000","last_clean":"2024-01-15T17:58:53.775615+0000","last_became_active":"2024-01-15T17:58:53.769028+0000","last_became_peered":"2024-01-15T17:58:53.769028+0000","last_unstale":"2024-01-15T17:58:53.775615+0000","last_undegraded":"2024-01-15T17:58:53.775615+0000","last_fullsized":"2024-01-15T17:58:53.775615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765380+0000","last_change":"2024-01-15T17:58:53.765380+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765380+0000","last_undegraded":"2024-01-15T17:58:53.765380+0000","last_fullsized":"2024-01-15T17:58:53.765380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781943+0000","last_change":"2024-01-15T17:58:53.778012+0000","last_active":"2024-01-15T17:58:53.781943+0000","last_peered":"2024-01-15T17:58:53.781943+0000","last_clean":"2024-01-15T17:58:53.781943+0000","last_became_active":"2024-01-15T17:58:53.776887+0000","last_became_peered":"2024-01-15T17:58:53.776887+0000","last_unstale":"2024-01-15T17:58:53.781943+0000","last_undegraded":"2024-01-15T17:58:53.781943+0000","last_fullsized":"2024-01-15T17:58:53.781943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:24.725739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757492+0000","last_change":"2024-01-15T17:58:53.757492+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757492+0000","last_undegraded":"2024-01-15T17:58:53.757492+0000","last_fullsized":"2024-01-15T17:58:53.757492+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787587+0000","last_change":"2024-01-15T17:58:53.786507+0000","last_active":"2024-01-15T17:58:53.787587+0000","last_peered":"2024-01-15T17:58:53.787587+0000","last_clean":"2024-01-15T17:58:53.787587+0000","last_became_active":"2024-01-15T17:58:53.786154+0000","last_became_peered":"2024-01-15T17:58:53.786154+0000","last_unstale":"2024-01-15T17:58:53.787587+0000","last_undegraded":"2024-01-15T17:58:53.787587+0000","last_fullsized":"2024-01-15T17:58:53.787587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:33.284238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760012+0000","last_change":"2024-01-15T17:58:53.760012+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760012+0000","last_undegraded":"2024-01-15T17:58:53.760012+0000","last_fullsized":"2024-01-15T17:58:53.760012+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781046+0000","last_change":"2024-01-15T17:58:53.779129+0000","last_active":"2024-01-15T17:58:53.781046+0000","last_peered":"2024-01-15T17:58:53.781046+0000","last_clean":"2024-01-15T17:58:53.781046+0000","last_became_active":"2024-01-15T17:58:53.778863+0000","last_became_peered":"2024-01-15T17:58:53.778863+0000","last_unstale":"2024-01-15T17:58:53.781046+0000","last_undegraded":"2024-01-15T17:58:53.781046+0000","last_fullsized":"2024-01-15T17:58:53.781046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758478+0000","last_change":"2024-01-15T17:58:53.758478+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758478+0000","last_undegraded":"2024-01-15T17:58:53.758478+0000","last_fullsized":"2024-01-15T17:58:53.758478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784603+0000","last_change":"2024-01-15T17:58:53.782469+0000","last_active":"2024-01-15T17:58:53.784603+0000","last_peered":"2024-01-15T17:58:53.784603+0000","last_clean":"2024-01-15T17:58:53.784603+0000","last_became_active":"2024-01-15T17:58:53.782219+0000","last_became_peered":"2024-01-15T17:58:53.782219+0000","last_unstale":"2024-01-15T17:58:53.784603+0000","last_undegraded":"2024-01-15T17:58:53.784603+0000","last_fullsized":"2024-01-15T17:58:53.784603+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758109+0000","last_change":"2024-01-15T17:58:53.758109+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758109+0000","last_undegraded":"2024-01-15T17:58:53.758109+0000","last_fullsized":"2024-01-15T17:58:53.758109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781322+0000","last_change":"2024-01-15T17:58:53.778698+0000","last_active":"2024-01-15T17:58:53.781322+0000","last_peered":"2024-01-15T17:58:53.781322+0000","last_clean":"2024-01-15T17:58:53.781322+0000","last_became_active":"2024-01-15T17:58:53.778416+0000","last_became_peered":"2024-01-15T17:58:53.778416+0000","last_unstale":"2024-01-15T17:58:53.781322+0000","last_undegraded":"2024-01-15T17:58:53.781322+0000","last_fullsized":"2024-01-15T17:58:53.781322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:40:50.242725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762084+0000","last_change":"2024-01-15T17:58:53.762084+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762084+0000","last_undegraded":"2024-01-15T17:58:53.762084+0000","last_fullsized":"2024-01-15T17:58:53.762084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782813+0000","last_change":"2024-01-15T17:58:53.780478+0000","last_active":"2024-01-15T17:58:53.782813+0000","last_peered":"2024-01-15T17:58:53.782813+0000","last_clean":"2024-01-15T17:58:53.782813+0000","last_became_active":"2024-01-15T17:58:53.777564+0000","last_became_peered":"2024-01-15T17:58:53.777564+0000","last_unstale":"2024-01-15T17:58:53.782813+0000","last_undegraded":"2024-01-15T17:58:53.782813+0000","last_fullsized":"2024-01-15T17:58:53.782813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:43.478841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757520+0000","last_change":"2024-01-15T17:58:53.757520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757520+0000","last_undegraded":"2024-01-15T17:58:53.757520+0000","last_fullsized":"2024-01-15T17:58:53.757520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783720+0000","last_change":"2024-01-15T17:58:53.777976+0000","last_active":"2024-01-15T17:58:53.783720+0000","last_peered":"2024-01-15T17:58:53.783720+0000","last_clean":"2024-01-15T17:58:53.783720+0000","last_became_active":"2024-01-15T17:58:53.777658+0000","last_became_peered":"2024-01-15T17:58:53.777658+0000","last_unstale":"2024-01-15T17:58:53.783720+0000","last_undegraded":"2024-01-15T17:58:53.783720+0000","last_fullsized":"2024-01-15T17:58:53.783720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776953+0000","last_change":"2024-01-15T17:58:53.776953+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776953+0000","last_undegraded":"2024-01-15T17:58:53.776953+0000","last_fullsized":"2024-01-15T17:58:53.776953+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.785944+0000","last_change":"2024-01-15T17:58:53.782063+0000","last_active":"2024-01-15T17:58:53.785944+0000","last_peered":"2024-01-15T17:58:53.785944+0000","last_clean":"2024-01-15T17:58:53.785944+0000","last_became_active":"2024-01-15T17:58:53.781761+0000","last_became_peered":"2024-01-15T17:58:53.781761+0000","last_unstale":"2024-01-15T17:58:53.785944+0000","last_undegraded":"2024-01-15T17:58:53.785944+0000","last_fullsized":"2024-01-15T17:58:53.785944+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:05:52.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768343+0000","last_change":"2024-01-15T17:58:53.768343+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768343+0000","last_undegraded":"2024-01-15T17:58:53.768343+0000","last_fullsized":"2024-01-15T17:58:53.768343+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779694+0000","last_change":"2024-01-15T17:58:53.775656+0000","last_active":"2024-01-15T17:58:53.779694+0000","last_peered":"2024-01-15T17:58:53.779694+0000","last_clean":"2024-01-15T17:58:53.779694+0000","last_became_active":"2024-01-15T17:58:53.775310+0000","last_became_peered":"2024-01-15T17:58:53.775310+0000","last_unstale":"2024-01-15T17:58:53.779694+0000","last_undegraded":"2024-01-15T17:58:53.779694+0000","last_fullsized":"2024-01-15T17:58:53.779694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:16:51.863117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763189+0000","last_change":"2024-01-15T17:58:53.763189+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763189+0000","last_undegraded":"2024-01-15T17:58:53.763189+0000","last_fullsized":"2024-01-15T17:58:53.763189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783884+0000","last_change":"2024-01-15T17:58:53.779965+0000","last_active":"2024-01-15T17:58:53.783884+0000","last_peered":"2024-01-15T17:58:53.783884+0000","last_clean":"2024-01-15T17:58:53.783884+0000","last_became_active":"2024-01-15T17:58:53.779597+0000","last_became_peered":"2024-01-15T17:58:53.779597+0000","last_unstale":"2024-01-15T17:58:53.783884+0000","last_undegraded":"2024-01-15T17:58:53.783884+0000","last_fullsized":"2024-01-15T17:58:53.783884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762156+0000","last_change":"2024-01-15T17:58:53.762156+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762156+0000","last_undegraded":"2024-01-15T17:58:53.762156+0000","last_fullsized":"2024-01-15T17:58:53.762156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779888+0000","last_change":"2024-01-15T17:58:53.774956+0000","last_active":"2024-01-15T17:58:53.779888+0000","last_peered":"2024-01-15T17:58:53.779888+0000","last_clean":"2024-01-15T17:58:53.779888+0000","last_became_active":"2024-01-15T17:58:53.774680+0000","last_became_peered":"2024-01-15T17:58:53.774680+0000","last_unstale":"2024-01-15T17:58:53.779888+0000","last_undegraded":"2024-01-15T17:58:53.779888+0000","last_fullsized":"2024-01-15T17:58:53.779888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.766670+0000","last_change":"2024-01-15T17:58:53.766670+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.766670+0000","last_undegraded":"2024-01-15T17:58:53.766670+0000","last_fullsized":"2024-01-15T17:58:53.766670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779743+0000","last_change":"2024-01-15T17:58:53.772674+0000","last_active":"2024-01-15T17:58:53.779743+0000","last_peered":"2024-01-15T17:58:53.779743+0000","last_clean":"2024-01-15T17:58:53.779743+0000","last_became_active":"2024-01-15T17:58:53.769008+0000","last_became_peered":"2024-01-15T17:58:53.769008+0000","last_unstale":"2024-01-15T17:58:53.779743+0000","last_undegraded":"2024-01-15T17:58:53.779743+0000","last_fullsized":"2024-01-15T17:58:53.779743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:09.191112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761263+0000","last_change":"2024-01-15T17:58:53.761263+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761263+0000","last_undegraded":"2024-01-15T17:58:53.761263+0000","last_fullsized":"2024-01-15T17:58:53.761263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782726+0000","last_change":"2024-01-15T17:58:53.779480+0000","last_active":"2024-01-15T17:58:53.782726+0000","last_peered":"2024-01-15T17:58:53.782726+0000","last_clean":"2024-01-15T17:58:53.782726+0000","last_became_active":"2024-01-15T17:58:53.778673+0000","last_became_peered":"2024-01-15T17:58:53.778673+0000","last_unstale":"2024-01-15T17:58:53.782726+0000","last_undegraded":"2024-01-15T17:58:53.782726+0000","last_fullsized":"2024-01-15T17:58:53.782726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:40.298878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758081+0000","last_change":"2024-01-15T17:58:53.758081+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758081+0000","last_undegraded":"2024-01-15T17:58:53.758081+0000","last_fullsized":"2024-01-15T17:58:53.758081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781876+0000","last_change":"2024-01-15T17:58:53.780996+0000","last_active":"2024-01-15T17:58:53.781876+0000","last_peered":"2024-01-15T17:58:53.781876+0000","last_clean":"2024-01-15T17:58:53.781876+0000","last_became_active":"2024-01-15T17:58:53.778816+0000","last_became_peered":"2024-01-15T17:58:53.778816+0000","last_unstale":"2024-01-15T17:58:53.781876+0000","last_undegraded":"2024-01-15T17:58:53.781876+0000","last_fullsized":"2024-01-15T17:58:53.781876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758792+0000","last_change":"2024-01-15T17:58:53.758792+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758792+0000","last_undegraded":"2024-01-15T17:58:53.758792+0000","last_fullsized":"2024-01-15T17:58:53.758792+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786081+0000","last_change":"2024-01-15T17:58:53.784613+0000","last_active":"2024-01-15T17:58:53.786081+0000","last_peered":"2024-01-15T17:58:53.786081+0000","last_clean":"2024-01-15T17:58:53.786081+0000","last_became_active":"2024-01-15T17:58:53.784295+0000","last_became_peered":"2024-01-15T17:58:53.784295+0000","last_unstale":"2024-01-15T17:58:53.786081+0000","last_undegraded":"2024-01-15T17:58:53.786081+0000","last_fullsized":"2024-01-15T17:58:53.786081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757313+0000","last_change":"2024-01-15T17:58:53.757313+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757313+0000","last_undegraded":"2024-01-15T17:58:53.757313+0000","last_fullsized":"2024-01-15T17:58:53.757313+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786057+0000","last_change":"2024-01-15T17:58:53.785047+0000","last_active":"2024-01-15T17:58:53.786057+0000","last_peered":"2024-01-15T17:58:53.786057+0000","last_clean":"2024-01-15T17:58:53.786057+0000","last_became_active":"2024-01-15T17:58:53.784573+0000","last_became_peered":"2024-01-15T17:58:53.784573+0000","last_unstale":"2024-01-15T17:58:53.786057+0000","last_undegraded":"2024-01-15T17:58:53.786057+0000","last_fullsized":"2024-01-15T17:58:53.786057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:07.495003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771821+0000","last_change":"2024-01-15T17:58:53.771821+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771821+0000","last_undegraded":"2024-01-15T17:58:53.771821+0000","last_fullsized":"2024-01-15T17:58:53.771821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784703+0000","last_change":"2024-01-15T17:58:53.782707+0000","last_active":"2024-01-15T17:58:53.784703+0000","last_peered":"2024-01-15T17:58:53.784703+0000","last_clean":"2024-01-15T17:58:53.784703+0000","last_became_active":"2024-01-15T17:58:53.782425+0000","last_became_peered":"2024-01-15T17:58:53.782425+0000","last_unstale":"2024-01-15T17:58:53.784703+0000","last_undegraded":"2024-01-15T17:58:53.784703+0000","last_fullsized":"2024-01-15T17:58:53.784703+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:57.822622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762554+0000","last_change":"2024-01-15T17:58:53.762554+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762554+0000","last_undegraded":"2024-01-15T17:58:53.762554+0000","last_fullsized":"2024-01-15T17:58:53.762554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783352+0000","last_change":"2024-01-15T17:58:53.776979+0000","last_active":"2024-01-15T17:58:53.783352+0000","last_peered":"2024-01-15T17:58:53.783352+0000","last_clean":"2024-01-15T17:58:53.783352+0000","last_became_active":"2024-01-15T17:58:53.776609+0000","last_became_peered":"2024-01-15T17:58:53.776609+0000","last_unstale":"2024-01-15T17:58:53.783352+0000","last_undegraded":"2024-01-15T17:58:53.783352+0000","last_fullsized":"2024-01-15T17:58:53.783352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761363+0000","last_change":"2024-01-15T17:58:53.761363+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761363+0000","last_undegraded":"2024-01-15T17:58:53.761363+0000","last_fullsized":"2024-01-15T17:58:53.761363+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779213+0000","last_change":"2024-01-15T17:58:53.777358+0000","last_active":"2024-01-15T17:58:53.779213+0000","last_peered":"2024-01-15T17:58:53.779213+0000","last_clean":"2024-01-15T17:58:53.779213+0000","last_became_active":"2024-01-15T17:58:53.777077+0000","last_became_peered":"2024-01-15T17:58:53.777077+0000","last_unstale":"2024-01-15T17:58:53.779213+0000","last_undegraded":"2024-01-15T17:58:53.779213+0000","last_fullsized":"2024-01-15T17:58:53.779213+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760665+0000","last_change":"2024-01-15T17:58:53.760665+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760665+0000","last_undegraded":"2024-01-15T17:58:53.760665+0000","last_fullsized":"2024-01-15T17:58:53.760665+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.777700+0000","last_change":"2024-01-15T17:58:53.775372+0000","last_active":"2024-01-15T17:58:53.777700+0000","last_peered":"2024-01-15T17:58:53.777700+0000","last_clean":"2024-01-15T17:58:53.777700+0000","last_became_active":"2024-01-15T17:58:53.775056+0000","last_became_peered":"2024-01-15T17:58:53.775056+0000","last_unstale":"2024-01-15T17:58:53.777700+0000","last_undegraded":"2024-01-15T17:58:53.777700+0000","last_fullsized":"2024-01-15T17:58:53.777700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770135+0000","last_change":"2024-01-15T17:58:53.770135+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770135+0000","last_undegraded":"2024-01-15T17:58:53.770135+0000","last_fullsized":"2024-01-15T17:58:53.770135+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.772530+0000","last_change":"2024-01-15T17:58:53.770121+0000","last_active":"2024-01-15T17:58:53.772530+0000","last_peered":"2024-01-15T17:58:53.772530+0000","last_clean":"2024-01-15T17:58:53.772530+0000","last_became_active":"2024-01-15T17:58:53.769805+0000","last_became_peered":"2024-01-15T17:58:53.769805+0000","last_unstale":"2024-01-15T17:58:53.772530+0000","last_undegraded":"2024-01-15T17:58:53.772530+0000","last_fullsized":"2024-01-15T17:58:53.772530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758192+0000","last_change":"2024-01-15T17:58:53.758192+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758192+0000","last_undegraded":"2024-01-15T17:58:53.758192+0000","last_fullsized":"2024-01-15T17:58:53.758192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782268+0000","last_change":"2024-01-15T17:58:53.776755+0000","last_active":"2024-01-15T17:58:53.782268+0000","last_peered":"2024-01-15T17:58:53.782268+0000","last_clean":"2024-01-15T17:58:53.782268+0000","last_became_active":"2024-01-15T17:58:53.776423+0000","last_became_peered":"2024-01-15T17:58:53.776423+0000","last_unstale":"2024-01-15T17:58:53.782268+0000","last_undegraded":"2024-01-15T17:58:53.782268+0000","last_fullsized":"2024-01-15T17:58:53.782268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756946+0000","last_change":"2024-01-15T17:58:53.756946+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756946+0000","last_undegraded":"2024-01-15T17:58:53.756946+0000","last_fullsized":"2024-01-15T17:58:53.756946+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779312+0000","last_change":"2024-01-15T17:58:53.770125+0000","last_active":"2024-01-15T17:58:53.779312+0000","last_peered":"2024-01-15T17:58:53.779312+0000","last_clean":"2024-01-15T17:58:53.779312+0000","last_became_active":"2024-01-15T17:58:53.769709+0000","last_became_peered":"2024-01-15T17:58:53.769709+0000","last_unstale":"2024-01-15T17:58:53.779312+0000","last_undegraded":"2024-01-15T17:58:53.779312+0000","last_fullsized":"2024-01-15T17:58:53.779312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:17.227945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761458+0000","last_change":"2024-01-15T17:58:53.761458+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761458+0000","last_undegraded":"2024-01-15T17:58:53.761458+0000","last_fullsized":"2024-01-15T17:58:53.761458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787274+0000","last_change":"2024-01-15T17:58:53.786548+0000","last_active":"2024-01-15T17:58:53.787274+0000","last_peered":"2024-01-15T17:58:53.787274+0000","last_clean":"2024-01-15T17:58:53.787274+0000","last_became_active":"2024-01-15T17:58:53.786220+0000","last_became_peered":"2024-01-15T17:58:53.786220+0000","last_unstale":"2024-01-15T17:58:53.787274+0000","last_undegraded":"2024-01-15T17:58:53.787274+0000","last_fullsized":"2024-01-15T17:58:53.787274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:06:45.660613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771119+0000","last_change":"2024-01-15T17:58:53.771119+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771119+0000","last_undegraded":"2024-01-15T17:58:53.771119+0000","last_fullsized":"2024-01-15T17:58:53.771119+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784763+0000","last_change":"2024-01-15T17:58:53.783339+0000","last_active":"2024-01-15T17:58:53.784763+0000","last_peered":"2024-01-15T17:58:53.784763+0000","last_clean":"2024-01-15T17:58:53.784763+0000","last_became_active":"2024-01-15T17:58:53.783028+0000","last_became_peered":"2024-01-15T17:58:53.783028+0000","last_unstale":"2024-01-15T17:58:53.784763+0000","last_undegraded":"2024-01-15T17:58:53.784763+0000","last_fullsized":"2024-01-15T17:58:53.784763+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":173,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T17:58:52.760774+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:58:52.760774+0000","last_peered":"2024-01-15T17:58:52.760774+0000","last_clean":"2024-01-15T17:58:52.760774+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:58:52.760774+0000","last_undegraded":"2024-01-15T17:58:52.760774+0000","last_fullsized":"2024-01-15T17:58:52.760774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.6970000000000001,"5min":7.6970000000000001,"15min":7.6970000000000001},"last":1.339}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":1.196},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":1.5760000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.53}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.694}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":7.4809999999999999,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.2619999999999996,"5min":4.2619999999999996,"15min":4.2619999999999996},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.462},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":7.798,"5min":7.798,"15min":7.798},"last":1.4299999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":7.968,"5min":7.968,"15min":7.968},"last":1.611},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.6439999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.5009999999999999}]},{"osd":1,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.321},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.4019999999999999}]},{"osd":2,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.997}]},{"osd":3,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.198},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.4359999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.073,"5min":1.073,"15min":1.073},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.4710000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.653}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.323},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.1639999999999999}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.48},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.431}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.0669999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.5329999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":1.272},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.5840000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.752}]},{"osd":1,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.613},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.1100000000000001}]},{"osd":3,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.4259999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.526}]},{"osd":7,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.7230000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.008}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.161}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.419},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.5369999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.7}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.49},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.669}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5780000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.105}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5150000000000001,"5min":2.5150000000000001,"15min":2.5150000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.002}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.7150000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.675,"15min":1.675},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":1.758},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.7559999999999998,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.5649999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":1.149,"5min":1.149,"15min":1.149},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":1.847},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.6279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.78800000000000003}]},{"osd":3,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.306}]},{"osd":4,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.379},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.5189999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.4430000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.6699999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"last":1.54},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.339}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":962560,"data_stored":731777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.069},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.196},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.337},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.2450000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.389}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.4570000000000001}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.569},{"interface":"front","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.679}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.768649+0000', 'last_change': '2024-01-15T17:58:53.768649+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.768649+0000', 'last_undegraded': '2024-01-15T17:58:53.768649+0000', 'last_fullsized': '2024-01-15T17:58:53.768649+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.765915+0000', 'last_change': '2024-01-15T17:58:53.765915+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.765915+0000', 'last_undegraded': '2024-01-15T17:58:53.765915+0000', 'last_fullsized': '2024-01-15T17:58:53.765915+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.756460+0000', 'last_change': '2024-01-15T17:58:53.756460+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.756460+0000', 'last_undegraded': '2024-01-15T17:58:53.756460+0000', 'last_fullsized': '2024-01-15T17:58:53.756460+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-01-15T17:59:00.350 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.767200+0000', 'last_change': '2024-01-15T17:58:53.767200+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.767200+0000', 'last_undegraded': '2024-01-15T17:58:53.767200+0000', 'last_fullsized': '2024-01-15T17:58:53.767200+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.762688+0000', 'last_change': '2024-01-15T17:58:53.762688+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.762688+0000', 'last_undegraded': '2024-01-15T17:58:53.762688+0000', 'last_fullsized': '2024-01-15T17:58:53.762688+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.764285+0000', 'last_change': '2024-01-15T17:58:53.764285+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.764285+0000', 'last_undegraded': '2024-01-15T17:58:53.764285+0000', 'last_fullsized': '2024-01-15T17:58:53.764285+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.770853+0000', 'last_change': '2024-01-15T17:58:53.770853+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.770853+0000', 'last_undegraded': '2024-01-15T17:58:53.770853+0000', 'last_fullsized': '2024-01-15T17:58:53.770853+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.779368+0000', 'last_change': '2024-01-15T17:58:53.779368+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.779368+0000', 'last_undegraded': '2024-01-15T17:58:53.779368+0000', 'last_fullsized': '2024-01-15T17:58:53.779368+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-01-15T17:59:00.351 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760076+0000', 'last_change': '2024-01-15T17:58:53.760076+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760076+0000', 'last_undegraded': '2024-01-15T17:58:53.760076+0000', 'last_fullsized': '2024-01-15T17:58:53.760076+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.768131+0000', 'last_change': '2024-01-15T17:58:53.768131+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.768131+0000', 'last_undegraded': '2024-01-15T17:58:53.768131+0000', 'last_fullsized': '2024-01-15T17:58:53.768131+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.777933+0000', 'last_change': '2024-01-15T17:58:53.777933+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.777933+0000', 'last_undegraded': '2024-01-15T17:58:53.777933+0000', 'last_fullsized': '2024-01-15T17:58:53.777933+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760794+0000', 'last_change': '2024-01-15T17:58:53.760794+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760794+0000', 'last_undegraded': '2024-01-15T17:58:53.760794+0000', 'last_fullsized': '2024-01-15T17:58:53.760794+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.761983+0000', 'last_change': '2024-01-15T17:58:53.761983+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.761983+0000', 'last_undegraded': '2024-01-15T17:58:53.761983+0000', 'last_fullsized': '2024-01-15T17:58:53.761983+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.763819+0000', 'last_change': '2024-01-15T17:58:53.763819+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.763819+0000', 'last_undegraded': '2024-01-15T17:58:53.763819+0000', 'last_fullsized': '2024-01-15T17:58:53.763819+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.352 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.756158+0000', 'last_change': '2024-01-15T17:58:53.756158+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.756158+0000', 'last_undegraded': '2024-01-15T17:58:53.756158+0000', 'last_fullsized': '2024-01-15T17:58:53.756158+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.769648+0000', 'last_change': '2024-01-15T17:58:53.769648+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.769648+0000', 'last_undegraded': '2024-01-15T17:58:53.769648+0000', 'last_fullsized': '2024-01-15T17:58:53.769648+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.776565+0000', 'last_change': '2024-01-15T17:58:53.776565+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.776565+0000', 'last_undegraded': '2024-01-15T17:58:53.776565+0000', 'last_fullsized': '2024-01-15T17:58:53.776565+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.763960+0000', 'last_change': '2024-01-15T17:58:53.763960+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.763960+0000', 'last_undegraded': '2024-01-15T17:58:53.763960+0000', 'last_fullsized': '2024-01-15T17:58:53.763960+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758899+0000', 'last_change': '2024-01-15T17:58:53.758899+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758899+0000', 'last_undegraded': '2024-01-15T17:58:53.758899+0000', 'last_fullsized': '2024-01-15T17:58:53.758899+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-01-15T17:59:00.353 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760041+0000', 'last_change': '2024-01-15T17:58:53.760041+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760041+0000', 'last_undegraded': '2024-01-15T17:58:53.760041+0000', 'last_fullsized': '2024-01-15T17:58:53.760041+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.759520+0000', 'last_change': '2024-01-15T17:58:53.759520+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.759520+0000', 'last_undegraded': '2024-01-15T17:58:53.759520+0000', 'last_fullsized': '2024-01-15T17:58:53.759520+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.756476+0000', 'last_change': '2024-01-15T17:58:53.756476+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.756476+0000', 'last_undegraded': '2024-01-15T17:58:53.756476+0000', 'last_fullsized': '2024-01-15T17:58:53.756476+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.763859+0000', 'last_change': '2024-01-15T17:58:53.763859+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.763859+0000', 'last_undegraded': '2024-01-15T17:58:53.763859+0000', 'last_fullsized': '2024-01-15T17:58:53.763859+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.757006+0000', 'last_change': '2024-01-15T17:58:53.757006+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.757006+0000', 'last_undegraded': '2024-01-15T17:58:53.757006+0000', 'last_fullsized': '2024-01-15T17:58:53.757006+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-01-15T17:59:00.354 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.769359+0000', 'last_change': '2024-01-15T17:58:53.769359+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.769359+0000', 'last_undegraded': '2024-01-15T17:58:53.769359+0000', 'last_fullsized': '2024-01-15T17:58:53.769359+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758923+0000', 'last_change': '2024-01-15T17:58:53.758923+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758923+0000', 'last_undegraded': '2024-01-15T17:58:53.758923+0000', 'last_fullsized': '2024-01-15T17:58:53.758923+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.768774+0000', 'last_change': '2024-01-15T17:58:53.768774+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.768774+0000', 'last_undegraded': '2024-01-15T17:58:53.768774+0000', 'last_fullsized': '2024-01-15T17:58:53.768774+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.759857+0000', 'last_change': '2024-01-15T17:58:53.759857+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.759857+0000', 'last_undegraded': '2024-01-15T17:58:53.759857+0000', 'last_fullsized': '2024-01-15T17:58:53.759857+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758420+0000', 'last_change': '2024-01-15T17:58:53.758420+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758420+0000', 'last_undegraded': '2024-01-15T17:58:53.758420+0000', 'last_fullsized': '2024-01-15T17:58:53.758420+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.770607+0000', 'last_change': '2024-01-15T17:58:53.770607+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.770607+0000', 'last_undegraded': '2024-01-15T17:58:53.770607+0000', 'last_fullsized': '2024-01-15T17:58:53.770607+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-01-15T17:59:00.355 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.769816+0000', 'last_change': '2024-01-15T17:58:53.769816+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.769816+0000', 'last_undegraded': '2024-01-15T17:58:53.769816+0000', 'last_fullsized': '2024-01-15T17:58:53.769816+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.777957+0000', 'last_change': '2024-01-15T17:58:53.777957+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.777957+0000', 'last_undegraded': '2024-01-15T17:58:53.777957+0000', 'last_fullsized': '2024-01-15T17:58:53.777957+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760467+0000', 'last_change': '2024-01-15T17:58:53.760467+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760467+0000', 'last_undegraded': '2024-01-15T17:58:53.760467+0000', 'last_fullsized': '2024-01-15T17:58:53.760467+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.767896+0000', 'last_change': '2024-01-15T17:58:53.767896+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.767896+0000', 'last_undegraded': '2024-01-15T17:58:53.767896+0000', 'last_fullsized': '2024-01-15T17:58:53.767896+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.761593+0000', 'last_change': '2024-01-15T17:58:53.761593+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.761593+0000', 'last_undegraded': '2024-01-15T17:58:53.761593+0000', 'last_fullsized': '2024-01-15T17:58:53.761593+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-01-15T17:59:00.356 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.768274+0000', 'last_change': '2024-01-15T17:58:53.768274+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.768274+0000', 'last_undegraded': '2024-01-15T17:58:53.768274+0000', 'last_fullsized': '2024-01-15T17:58:53.768274+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.770108+0000', 'last_change': '2024-01-15T17:58:53.770108+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.770108+0000', 'last_undegraded': '2024-01-15T17:58:53.770108+0000', 'last_fullsized': '2024-01-15T17:58:53.770108+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.756009+0000', 'last_change': '2024-01-15T17:58:53.756009+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.756009+0000', 'last_undegraded': '2024-01-15T17:58:53.756009+0000', 'last_fullsized': '2024-01-15T17:58:53.756009+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760047+0000', 'last_change': '2024-01-15T17:58:53.760047+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760047+0000', 'last_undegraded': '2024-01-15T17:58:53.760047+0000', 'last_fullsized': '2024-01-15T17:58:53.760047+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.765380+0000', 'last_change': '2024-01-15T17:58:53.765380+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.765380+0000', 'last_undegraded': '2024-01-15T17:58:53.765380+0000', 'last_fullsized': '2024-01-15T17:58:53.765380+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.757492+0000', 'last_change': '2024-01-15T17:58:53.757492+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.757492+0000', 'last_undegraded': '2024-01-15T17:58:53.757492+0000', 'last_fullsized': '2024-01-15T17:58:53.757492+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760012+0000', 'last_change': '2024-01-15T17:58:53.760012+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760012+0000', 'last_undegraded': '2024-01-15T17:58:53.760012+0000', 'last_fullsized': '2024-01-15T17:58:53.760012+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.357 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758478+0000', 'last_change': '2024-01-15T17:58:53.758478+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758478+0000', 'last_undegraded': '2024-01-15T17:58:53.758478+0000', 'last_fullsized': '2024-01-15T17:58:53.758478+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758109+0000', 'last_change': '2024-01-15T17:58:53.758109+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758109+0000', 'last_undegraded': '2024-01-15T17:58:53.758109+0000', 'last_fullsized': '2024-01-15T17:58:53.758109+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.762084+0000', 'last_change': '2024-01-15T17:58:53.762084+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.762084+0000', 'last_undegraded': '2024-01-15T17:58:53.762084+0000', 'last_fullsized': '2024-01-15T17:58:53.762084+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.757520+0000', 'last_change': '2024-01-15T17:58:53.757520+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.757520+0000', 'last_undegraded': '2024-01-15T17:58:53.757520+0000', 'last_fullsized': '2024-01-15T17:58:53.757520+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.776953+0000', 'last_change': '2024-01-15T17:58:53.776953+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.776953+0000', 'last_undegraded': '2024-01-15T17:58:53.776953+0000', 'last_fullsized': '2024-01-15T17:58:53.776953+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.768343+0000', 'last_change': '2024-01-15T17:58:53.768343+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.768343+0000', 'last_undegraded': '2024-01-15T17:58:53.768343+0000', 'last_fullsized': '2024-01-15T17:58:53.768343+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T17:59:00.358 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.763189+0000', 'last_change': '2024-01-15T17:58:53.763189+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.763189+0000', 'last_undegraded': '2024-01-15T17:58:53.763189+0000', 'last_fullsized': '2024-01-15T17:58:53.763189+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.762156+0000', 'last_change': '2024-01-15T17:58:53.762156+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.762156+0000', 'last_undegraded': '2024-01-15T17:58:53.762156+0000', 'last_fullsized': '2024-01-15T17:58:53.762156+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.766670+0000', 'last_change': '2024-01-15T17:58:53.766670+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.766670+0000', 'last_undegraded': '2024-01-15T17:58:53.766670+0000', 'last_fullsized': '2024-01-15T17:58:53.766670+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.761263+0000', 'last_change': '2024-01-15T17:58:53.761263+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.761263+0000', 'last_undegraded': '2024-01-15T17:58:53.761263+0000', 'last_fullsized': '2024-01-15T17:58:53.761263+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758081+0000', 'last_change': '2024-01-15T17:58:53.758081+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758081+0000', 'last_undegraded': '2024-01-15T17:58:53.758081+0000', 'last_fullsized': '2024-01-15T17:58:53.758081+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758792+0000', 'last_change': '2024-01-15T17:58:53.758792+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758792+0000', 'last_undegraded': '2024-01-15T17:58:53.758792+0000', 'last_fullsized': '2024-01-15T17:58:53.758792+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.359 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.757313+0000', 'last_change': '2024-01-15T17:58:53.757313+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.757313+0000', 'last_undegraded': '2024-01-15T17:58:53.757313+0000', 'last_fullsized': '2024-01-15T17:58:53.757313+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.771821+0000', 'last_change': '2024-01-15T17:58:53.771821+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.771821+0000', 'last_undegraded': '2024-01-15T17:58:53.771821+0000', 'last_fullsized': '2024-01-15T17:58:53.771821+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.762554+0000', 'last_change': '2024-01-15T17:58:53.762554+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.762554+0000', 'last_undegraded': '2024-01-15T17:58:53.762554+0000', 'last_fullsized': '2024-01-15T17:58:53.762554+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.761363+0000', 'last_change': '2024-01-15T17:58:53.761363+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.761363+0000', 'last_undegraded': '2024-01-15T17:58:53.761363+0000', 'last_fullsized': '2024-01-15T17:58:53.761363+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.760665+0000', 'last_change': '2024-01-15T17:58:53.760665+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.760665+0000', 'last_undegraded': '2024-01-15T17:58:53.760665+0000', 'last_fullsized': '2024-01-15T17:58:53.760665+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-01-15T17:59:00.360 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.770135+0000', 'last_change': '2024-01-15T17:58:53.770135+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.770135+0000', 'last_undegraded': '2024-01-15T17:58:53.770135+0000', 'last_fullsized': '2024-01-15T17:58:53.770135+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T17:59:00.361 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-01-15T17:59:00.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.758192+0000', 'last_change': '2024-01-15T17:58:53.758192+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.758192+0000', 'last_undegraded': '2024-01-15T17:58:53.758192+0000', 'last_fullsized': '2024-01-15T17:58:53.758192+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T17:59:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-01-15T17:59:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.756946+0000', 'last_change': '2024-01-15T17:58:53.756946+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.756946+0000', 'last_undegraded': '2024-01-15T17:58:53.756946+0000', 'last_fullsized': '2024-01-15T17:58:53.756946+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T17:59:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-01-15T17:59:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.761458+0000', 'last_change': '2024-01-15T17:58:53.761458+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.761458+0000', 'last_undegraded': '2024-01-15T17:58:53.761458+0000', 'last_fullsized': '2024-01-15T17:58:53.761458+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T17:59:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-01-15T17:59:00.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 33, 'state': 'creating+peering', 'last_fresh': '2024-01-15T17:58:53.771119+0000', 'last_change': '2024-01-15T17:58:53.771119+0000', 'last_active': '2024-01-15T17:58:53.739527+0000', 'last_peered': '2024-01-15T17:58:53.739527+0000', 'last_clean': '2024-01-15T17:58:53.739527+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T17:58:53.771119+0000', 'last_undegraded': '2024-01-15T17:58:53.771119+0000', 'last_fullsized': '2024-01-15T17:58:53.771119+0000', 'mapping_epoch': 33, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 33, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'last_clean_scrub_stamp': '2024-01-15T17:58:53.739527+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-01-15T17:59:00.365 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T17:59:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.612+0000 7f5e5b200640 1 Processor -- start 2024-01-15T17:59:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.612+0000 7f5e5b200640 1 -- start start 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 0x7f5e5c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5e5c15ec60 0x7f5e5c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c06dae0 con 0x7f5e5c0cd490 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c071f20 con 0x7f5e5c15ec60 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06c6b0 con 0x7f5e5c141880 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5a800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5e5c0cd490 0x7f5e5c0cd920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32952/0 (socket says 172.21.15.2:32952) 2024-01-15T17:59:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e5a800640 1 -- 172.21.15.2:0/2210627894 learned_addr learned my addr 172.21.15.2:0/2210627894 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:00.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.614+0000 7f5e53e00640 1 --2- 172.21.15.2:0/2210627894 >> v2:172.21.15.137:3300/0 conn(0x7f5e5c15ec60 0x7f5e5c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 0x7f5e5c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798805113 0 0) 0x7f5e5c06dae0 con 0x7f5e5c0cd490 2024-01-15T17:59:00.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c003520 con 0x7f5e5c0cd490 2024-01-15T17:59:00.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 952182892 0 0) 0x7f5e3c003520 con 0x7f5e5c0cd490 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 msgr2=0x7f5e5c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 0x7f5e5c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 >> v2:172.21.15.137:3300/0 conn(0x7f5e5c15ec60 msgr2=0x7f5e5c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 --2- 172.21.15.2:0/2210627894 >> v2:172.21.15.137:3300/0 conn(0x7f5e5c15ec60 0x7f5e5c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c071960 con 0x7f5e5c0cd490 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5e540019f0 con 0x7f5e5c0cd490 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 >> v1:172.21.15.2:6789/0 conn(0x7f5e5c0cd490 legacy=0x7f5e5c0cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 0x7f5e3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3c004d00 0x7f5e3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c071f20 con 0x7f5e3c003970 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c06c6b0 con 0x7f5e3c004d00 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.615+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c00b9b0 con 0x7f5e5c141880 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e5a800640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3c004d00 0x7f5e3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e53e00640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 0x7f5e3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e59e00640 1 -- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 msgr2=0x7f5e3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c141880 0x7f5e3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e59e00640 1 -- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3c004d00 msgr2=0x7f5e3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e3c004d00 0x7f5e3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.616+0000 7f5e59e00640 1 -- 172.21.15.2:0/2210627894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c003520 con 0x7f5e3c003970 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.618+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e3c003d90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009ac0 tx=0x7f5e3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b01a31cc4f271be server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.618+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02c6b0 con 0x7f5e3c003970 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.618+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c02c810 con 0x7f5e3c003970 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.618+0000 7f5e59400640 1 -- 172.21.15.2:0/2210627894 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c02cba0 con 0x7f5e3c003970 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.618+0000 7f5e5b200640 1 -- 172.21.15.2:0/2210627894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 msgr2=0x7f5e3c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.618+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2210627894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e3c003d90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009ac0 tx=0x7f5e3c0057f0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.619+0000 7f5e5a800640 1 -- 172.21.15.2:0/2210627894 reap_dead start 2024-01-15T17:59:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.619+0000 7f5e5b200640 1 -- 172.21.15.2:0/2210627894 shutdown_connections 2024-01-15T17:59:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.619+0000 7f5e5b200640 1 -- 172.21.15.2:0/2210627894 >> 172.21.15.2:0/2210627894 conn(0x7f5e5c0b0840 msgr2=0x7f5e5c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.619+0000 7f5e5b200640 1 -- 172.21.15.2:0/2210627894 shutdown_connections 2024-01-15T17:59:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.619+0000 7f5e5b200640 1 -- 172.21.15.2:0/2210627894 wait complete. 2024-01-15T17:59:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.621+0000 7f5e5b200640 1 Processor -- start 2024-01-15T17:59:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.621+0000 7f5e5b200640 1 -- start start 2024-01-15T17:59:00.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e5c1334f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c0b2b00 0x7f5e5c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5c15b6e0 0x7f5e5c15bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c133a30 con 0x7f5e3c003970 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c15c080 con 0x7f5e5c15b6e0 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0b2670 con 0x7f5e5c0b2b00 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e59e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e5c1334f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e59e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e5c1334f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46904/0 (socket says 172.21.15.2:46904) 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e59e00640 1 -- 172.21.15.2:0/2076889804 learned_addr learned my addr 172.21.15.2:0/2076889804 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e53e00640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c0b2b00 0x7f5e5c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.622+0000 7f5e5a800640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5c15b6e0 0x7f5e5c15bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e59e00640 1 -- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c0b2b00 msgr2=0x7f5e5c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c0b2b00 0x7f5e5c0b2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e59e00640 1 -- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5c15b6e0 msgr2=0x7f5e5c15bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5c15b6e0 0x7f5e5c15bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e59e00640 1 -- 172.21.15.2:0/2076889804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0b1a60 con 0x7f5e3c003970 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e59e00640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e5c1334f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c02c250 tx=0x7f5e4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c003810 con 0x7f5e3c003970 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c003970 con 0x7f5e3c003970 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c003ce0 con 0x7f5e3c003970 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c0b1d40 con 0x7f5e3c003970 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.623+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c0b2300 con 0x7f5e3c003970 2024-01-15T17:59:00.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.624+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c074770 con 0x7f5e3c003970 2024-01-15T17:59:00.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.625+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5e4c047450 con 0x7f5e3c003970 2024-01-15T17:59:00.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.626+0000 7f5e52000640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e3406b2c0 0x7f5e3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:00.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.626+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e4c0abe80 con 0x7f5e3c003970 2024-01-15T17:59:00.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.627+0000 7f5e53e00640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e3406b2c0 0x7f5e3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:00.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.628+0000 7f5e53e00640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e3406b2c0 0x7f5e3406d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0aed70 tx=0x7f5e44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:00.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.631+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e4c076eb0 con 0x7f5e3c003970 2024-01-15T17:59:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.897+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5e5c1594a0 con 0x7f5e3c003970 2024-01-15T17:59:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.897+0000 7f5e52000640 1 -- 172.21.15.2:0/2076889804 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1033 (secure 0 0 0) 0x7f5e4c0044e0 con 0x7f5e3c003970 2024-01-15T17:59:00.901 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:59:00.901 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":108,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":35,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341443,"num_in_osds":8,"osd_in_since":1705341435,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":242503680,"bytes_avail":772851609600,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":4954}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-01-15T17:58:35.392461+0000","services":{}},"progress_events":{}} 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e3406b2c0 msgr2=0x7f5e3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e3406b2c0 0x7f5e3406d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c0aed70 tx=0x7f5e44008040 comp rx=0 tx=0).stop 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 msgr2=0x7f5e5c1334f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e5c1334f0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c02c250 tx=0x7f5e4c004730 comp rx=0 tx=0).stop 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 shutdown_connections 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e3406b2c0 0x7f5e3406d780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e5c15b6e0 0x7f5e5c15bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e5c0b2b00 0x7f5e5c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 --2- 172.21.15.2:0/2076889804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e3c003970 0x7f5e5c1334f0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 >> 172.21.15.2:0/2076889804 conn(0x7f5e5c0b0840 msgr2=0x7f5e5c15f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.900+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 shutdown_connections 2024-01-15T17:59:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:00.901+0000 7f5e5b200640 1 -- 172.21.15.2:0/2076889804 wait complete. 2024-01-15T17:59:00.925 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T17:59:00.926 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:59:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.139+0000 7fae43400640 1 Processor -- start 2024-01-15T17:59:01.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.140+0000 7fae43400640 1 -- start start 2024-01-15T17:59:01.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.141+0000 7fae43400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fae3c15b7d0 0x7fae3c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:01.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.141+0000 7fae43400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c15cb00 con 0x7fae3c15c0e0 2024-01-15T17:59:01.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.141+0000 7fae43400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0ccec0 con 0x7fae3c15c4c0 2024-01-15T17:59:01.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.142+0000 7fae43400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0cccd0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.142+0000 7fae40a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fae3c15c0e0 0x7fae3c0cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:32958/0 (socket says 172.21.15.2:32958) 2024-01-15T17:59:01.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.142+0000 7fae40a00640 1 -- 172.21.15.2:0/2949869130 learned_addr learned my addr 172.21.15.2:0/2949869130 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:01.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.142+0000 7fae3be00640 1 --2- 172.21.15.2:0/2949869130 >> v2:172.21.15.137:3300/0 conn(0x7fae3c15b7d0 0x7fae3c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:01.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.143+0000 7fae3b400640 1 -- 172.21.15.2:0/2949869130 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4162319584 0 0) 0x7fae3c15cb00 con 0x7fae3c15c0e0 2024-01-15T17:59:01.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.143+0000 7fae3b400640 1 -- 172.21.15.2:0/2949869130 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae1c003520 con 0x7fae3c15c0e0 2024-01-15T17:59:01.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.143+0000 7fae3be00640 1 -- 172.21.15.2:0/2949869130 >> v1:172.21.15.137:6789/0 conn(0x7fae3c15c4c0 legacy=0x7fae3c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.143+0000 7fae3be00640 1 -- 172.21.15.2:0/2949869130 >> v1:172.21.15.2:6789/0 conn(0x7fae3c15c0e0 legacy=0x7fae3c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.143+0000 7fae3be00640 1 -- 172.21.15.2:0/2949869130 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0cd1a0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.144+0000 7fae3be00640 1 --2- 172.21.15.2:0/2949869130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15b7d0 0x7fae3c06c470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fae24009d90 tx=0x7fae2402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca355a5ac0abcd90 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:01.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae3b400640 1 -- 172.21.15.2:0/2949869130 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2402ccf0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae3b400640 1 -- 172.21.15.2:0/2949869130 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2402ce50 con 0x7fae3c15b7d0 2024-01-15T17:59:01.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae3b400640 1 -- 172.21.15.2:0/2949869130 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae24034660 con 0x7fae3c15b7d0 2024-01-15T17:59:01.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae43400640 1 -- 172.21.15.2:0/2949869130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15b7d0 msgr2=0x7fae3c06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae43400640 1 --2- 172.21.15.2:0/2949869130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15b7d0 0x7fae3c06c470 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fae24009d90 tx=0x7fae2402bf40 comp rx=0 tx=0).stop 2024-01-15T17:59:01.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae43400640 1 -- 172.21.15.2:0/2949869130 shutdown_connections 2024-01-15T17:59:01.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae43400640 1 --2- 172.21.15.2:0/2949869130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15b7d0 0x7fae3c06c470 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.145+0000 7fae43400640 1 -- 172.21.15.2:0/2949869130 >> 172.21.15.2:0/2949869130 conn(0x7fae3c06da20 msgr2=0x7fae3c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:01.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.146+0000 7fae43400640 1 -- 172.21.15.2:0/2949869130 shutdown_connections 2024-01-15T17:59:01.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.146+0000 7fae43400640 1 -- 172.21.15.2:0/2949869130 wait complete. 2024-01-15T17:59:01.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.147+0000 7fae43400640 1 Processor -- start 2024-01-15T17:59:01.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.147+0000 7fae43400640 1 -- start start 2024-01-15T17:59:01.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.148+0000 7fae43400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 0x7fae3c1f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:01.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.148+0000 7fae43400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae3c15c0e0 0x7fae3c153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.148+0000 7fae43400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15c4c0 0x7fae3c1f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.148+0000 7fae43400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1f3220 con 0x7fae3c15b7d0 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.148+0000 7fae43400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1f2c30 con 0x7fae3c15c4c0 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.148+0000 7fae43400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c1f3bf0 con 0x7fae3c15c0e0 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 0x7fae3c1f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 0x7fae3c1f36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46918/0 (socket says 172.21.15.2:46918) 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 -- 172.21.15.2:0/2973000130 learned_addr learned my addr 172.21.15.2:0/2973000130 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 -- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae3c15c0e0 msgr2=0x7fae3c153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae3c15c0e0 0x7fae3c153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 -- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15c4c0 msgr2=0x7fae3c1f6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15c4c0 0x7fae3c1f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 -- 172.21.15.2:0/2973000130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae24009a70 con 0x7fae3c15b7d0 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.149+0000 7fae40a00640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 0x7fae3c1f36b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c410 tx=0x7fae2c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.150+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c005ce0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.150+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c005e40 con 0x7fae3c15b7d0 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.150+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0072f0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.150+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c1f6cc0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.150+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c1f2eb0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.151+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c0686f0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.152+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fae2c007510 con 0x7fae3c15b7d0 2024-01-15T17:59:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.153+0000 7fae38c00640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae1406b2c0 0x7fae1406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.153+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae2c0891f0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.154+0000 7fae3be00640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae1406b2c0 0x7fae1406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:01.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.155+0000 7fae3be00640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae1406b2c0 0x7fae1406d780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fae24036000 tx=0x7fae2402e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:01.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.158+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae2c0541b0 con 0x7fae3c15b7d0 2024-01-15T17:59:01.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.351+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae3c06b0a0 con 0x7fae1406b2c0 2024-01-15T17:59:01.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.375+0000 7fae38c00640 1 -- 172.21.15.2:0/2973000130 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281387 (secure 0 0 0) 0x7fae3c06b0a0 con 0x7fae1406b2c0 2024-01-15T17:59:01.383 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:59:01.385 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:59:01.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae1406b2c0 msgr2=0x7fae1406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae1406b2c0 0x7fae1406d780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fae24036000 tx=0x7fae2402e000 comp rx=0 tx=0).stop 2024-01-15T17:59:01.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 msgr2=0x7fae3c1f36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 0x7fae3c1f36b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fae2c00c410 tx=0x7fae2c00c8d0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 shutdown_connections 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae1406b2c0 0x7fae1406d780 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae3c15c4c0 0x7fae3c1f6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae3c15c0e0 0x7fae3c153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 --2- 172.21.15.2:0/2973000130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae3c15b7d0 0x7fae3c1f36b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.380+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 >> 172.21.15.2:0/2973000130 conn(0x7fae3c06da20 msgr2=0x7fae3c0cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.381+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 shutdown_connections 2024-01-15T17:59:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:01.381+0000 7fae43400640 1 -- 172.21.15.2:0/2973000130 wait complete. 2024-01-15T17:59:01.403 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-01-15T17:58:59.400090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236820,"kb_used_data":4004,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737900,"statfs":{"total":773094113280,"available":772851609600,"internally_reserved":0,"allocated":4100096,"data_stored":2293207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001942"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768649+0000","last_change":"2024-01-15T17:58:53.768649+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768649+0000","last_undegraded":"2024-01-15T17:58:53.768649+0000","last_fullsized":"2024-01-15T17:58:53.768649+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765915+0000","last_change":"2024-01-15T17:58:53.765915+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765915+0000","last_undegraded":"2024-01-15T17:58:53.765915+0000","last_fullsized":"2024-01-15T17:58:53.765915+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756460+0000","last_change":"2024-01-15T17:58:53.756460+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756460+0000","last_undegraded":"2024-01-15T17:58:53.756460+0000","last_fullsized":"2024-01-15T17:58:53.756460+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767200+0000","last_change":"2024-01-15T17:58:53.767200+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767200+0000","last_undegraded":"2024-01-15T17:58:53.767200+0000","last_fullsized":"2024-01-15T17:58:53.767200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762688+0000","last_change":"2024-01-15T17:58:53.762688+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762688+0000","last_undegraded":"2024-01-15T17:58:53.762688+0000","last_fullsized":"2024-01-15T17:58:53.762688+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.764285+0000","last_change":"2024-01-15T17:58:53.764285+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.764285+0000","last_undegraded":"2024-01-15T17:58:53.764285+0000","last_fullsized":"2024-01-15T17:58:53.764285+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770853+0000","last_change":"2024-01-15T17:58:53.770853+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770853+0000","last_undegraded":"2024-01-15T17:58:53.770853+0000","last_fullsized":"2024-01-15T17:58:53.770853+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.779368+0000","last_change":"2024-01-15T17:58:53.779368+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.779368+0000","last_undegraded":"2024-01-15T17:58:53.779368+0000","last_fullsized":"2024-01-15T17:58:53.779368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760076+0000","last_change":"2024-01-15T17:58:53.760076+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760076+0000","last_undegraded":"2024-01-15T17:58:53.760076+0000","last_fullsized":"2024-01-15T17:58:53.760076+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768131+0000","last_change":"2024-01-15T17:58:53.768131+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768131+0000","last_undegraded":"2024-01-15T17:58:53.768131+0000","last_fullsized":"2024-01-15T17:58:53.768131+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777933+0000","last_change":"2024-01-15T17:58:53.777933+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777933+0000","last_undegraded":"2024-01-15T17:58:53.777933+0000","last_fullsized":"2024-01-15T17:58:53.777933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760794+0000","last_change":"2024-01-15T17:58:53.760794+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760794+0000","last_undegraded":"2024-01-15T17:58:53.760794+0000","last_fullsized":"2024-01-15T17:58:53.760794+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761983+0000","last_change":"2024-01-15T17:58:53.761983+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761983+0000","last_undegraded":"2024-01-15T17:58:53.761983+0000","last_fullsized":"2024-01-15T17:58:53.761983+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763819+0000","last_change":"2024-01-15T17:58:53.763819+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763819+0000","last_undegraded":"2024-01-15T17:58:53.763819+0000","last_fullsized":"2024-01-15T17:58:53.763819+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756158+0000","last_change":"2024-01-15T17:58:53.756158+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756158+0000","last_undegraded":"2024-01-15T17:58:53.756158+0000","last_fullsized":"2024-01-15T17:58:53.756158+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769648+0000","last_change":"2024-01-15T17:58:53.769648+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769648+0000","last_undegraded":"2024-01-15T17:58:53.769648+0000","last_fullsized":"2024-01-15T17:58:53.769648+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776565+0000","last_change":"2024-01-15T17:58:53.776565+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776565+0000","last_undegraded":"2024-01-15T17:58:53.776565+0000","last_fullsized":"2024-01-15T17:58:53.776565+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763960+0000","last_change":"2024-01-15T17:58:53.763960+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763960+0000","last_undegraded":"2024-01-15T17:58:53.763960+0000","last_fullsized":"2024-01-15T17:58:53.763960+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758899+0000","last_change":"2024-01-15T17:58:53.758899+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758899+0000","last_undegraded":"2024-01-15T17:58:53.758899+0000","last_fullsized":"2024-01-15T17:58:53.758899+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760041+0000","last_change":"2024-01-15T17:58:53.760041+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760041+0000","last_undegraded":"2024-01-15T17:58:53.760041+0000","last_fullsized":"2024-01-15T17:58:53.760041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759520+0000","last_change":"2024-01-15T17:58:53.759520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759520+0000","last_undegraded":"2024-01-15T17:58:53.759520+0000","last_fullsized":"2024-01-15T17:58:53.759520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756476+0000","last_change":"2024-01-15T17:58:53.756476+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756476+0000","last_undegraded":"2024-01-15T17:58:53.756476+0000","last_fullsized":"2024-01-15T17:58:53.756476+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763859+0000","last_change":"2024-01-15T17:58:53.763859+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763859+0000","last_undegraded":"2024-01-15T17:58:53.763859+0000","last_fullsized":"2024-01-15T17:58:53.763859+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757006+0000","last_change":"2024-01-15T17:58:53.757006+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757006+0000","last_undegraded":"2024-01-15T17:58:53.757006+0000","last_fullsized":"2024-01-15T17:58:53.757006+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769359+0000","last_change":"2024-01-15T17:58:53.769359+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769359+0000","last_undegraded":"2024-01-15T17:58:53.769359+0000","last_fullsized":"2024-01-15T17:58:53.769359+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758923+0000","last_change":"2024-01-15T17:58:53.758923+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758923+0000","last_undegraded":"2024-01-15T17:58:53.758923+0000","last_fullsized":"2024-01-15T17:58:53.758923+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768774+0000","last_change":"2024-01-15T17:58:53.768774+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768774+0000","last_undegraded":"2024-01-15T17:58:53.768774+0000","last_fullsized":"2024-01-15T17:58:53.768774+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.759857+0000","last_change":"2024-01-15T17:58:53.759857+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.759857+0000","last_undegraded":"2024-01-15T17:58:53.759857+0000","last_fullsized":"2024-01-15T17:58:53.759857+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758420+0000","last_change":"2024-01-15T17:58:53.758420+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758420+0000","last_undegraded":"2024-01-15T17:58:53.758420+0000","last_fullsized":"2024-01-15T17:58:53.758420+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770607+0000","last_change":"2024-01-15T17:58:53.770607+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770607+0000","last_undegraded":"2024-01-15T17:58:53.770607+0000","last_fullsized":"2024-01-15T17:58:53.770607+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.769816+0000","last_change":"2024-01-15T17:58:53.769816+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.769816+0000","last_undegraded":"2024-01-15T17:58:53.769816+0000","last_fullsized":"2024-01-15T17:58:53.769816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.777957+0000","last_change":"2024-01-15T17:58:53.777957+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.777957+0000","last_undegraded":"2024-01-15T17:58:53.777957+0000","last_fullsized":"2024-01-15T17:58:53.777957+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760467+0000","last_change":"2024-01-15T17:58:53.760467+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760467+0000","last_undegraded":"2024-01-15T17:58:53.760467+0000","last_fullsized":"2024-01-15T17:58:53.760467+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781733+0000","last_change":"2024-01-15T17:58:53.779462+0000","last_active":"2024-01-15T17:58:53.781733+0000","last_peered":"2024-01-15T17:58:53.781733+0000","last_clean":"2024-01-15T17:58:53.781733+0000","last_became_active":"2024-01-15T17:58:53.779161+0000","last_became_peered":"2024-01-15T17:58:53.779161+0000","last_unstale":"2024-01-15T17:58:53.781733+0000","last_undegraded":"2024-01-15T17:58:53.781733+0000","last_fullsized":"2024-01-15T17:58:53.781733+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.767896+0000","last_change":"2024-01-15T17:58:53.767896+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.767896+0000","last_undegraded":"2024-01-15T17:58:53.767896+0000","last_fullsized":"2024-01-15T17:58:53.767896+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784085+0000","last_change":"2024-01-15T17:58:53.782887+0000","last_active":"2024-01-15T17:58:53.784085+0000","last_peered":"2024-01-15T17:58:53.784085+0000","last_clean":"2024-01-15T17:58:53.784085+0000","last_became_active":"2024-01-15T17:58:53.782599+0000","last_became_peered":"2024-01-15T17:58:53.782599+0000","last_unstale":"2024-01-15T17:58:53.784085+0000","last_undegraded":"2024-01-15T17:58:53.784085+0000","last_fullsized":"2024-01-15T17:58:53.784085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761593+0000","last_change":"2024-01-15T17:58:53.761593+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761593+0000","last_undegraded":"2024-01-15T17:58:53.761593+0000","last_fullsized":"2024-01-15T17:58:53.761593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782129+0000","last_change":"2024-01-15T17:58:53.779558+0000","last_active":"2024-01-15T17:58:53.782129+0000","last_peered":"2024-01-15T17:58:53.782129+0000","last_clean":"2024-01-15T17:58:53.782129+0000","last_became_active":"2024-01-15T17:58:53.779269+0000","last_became_peered":"2024-01-15T17:58:53.779269+0000","last_unstale":"2024-01-15T17:58:53.782129+0000","last_undegraded":"2024-01-15T17:58:53.782129+0000","last_fullsized":"2024-01-15T17:58:53.782129+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:19.366878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768274+0000","last_change":"2024-01-15T17:58:53.768274+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768274+0000","last_undegraded":"2024-01-15T17:58:53.768274+0000","last_fullsized":"2024-01-15T17:58:53.768274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.778130+0000","last_change":"2024-01-15T17:58:53.776027+0000","last_active":"2024-01-15T17:58:53.778130+0000","last_peered":"2024-01-15T17:58:53.778130+0000","last_clean":"2024-01-15T17:58:53.778130+0000","last_became_active":"2024-01-15T17:58:53.775717+0000","last_became_peered":"2024-01-15T17:58:53.775717+0000","last_unstale":"2024-01-15T17:58:53.778130+0000","last_undegraded":"2024-01-15T17:58:53.778130+0000","last_fullsized":"2024-01-15T17:58:53.778130+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770108+0000","last_change":"2024-01-15T17:58:53.770108+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770108+0000","last_undegraded":"2024-01-15T17:58:53.770108+0000","last_fullsized":"2024-01-15T17:58:53.770108+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782834+0000","last_change":"2024-01-15T17:58:53.780792+0000","last_active":"2024-01-15T17:58:53.782834+0000","last_peered":"2024-01-15T17:58:53.782834+0000","last_clean":"2024-01-15T17:58:53.782834+0000","last_became_active":"2024-01-15T17:58:53.780523+0000","last_became_peered":"2024-01-15T17:58:53.780523+0000","last_unstale":"2024-01-15T17:58:53.782834+0000","last_undegraded":"2024-01-15T17:58:53.782834+0000","last_fullsized":"2024-01-15T17:58:53.782834+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:18.235321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756009+0000","last_change":"2024-01-15T17:58:53.756009+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756009+0000","last_undegraded":"2024-01-15T17:58:53.756009+0000","last_fullsized":"2024-01-15T17:58:53.756009+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782764+0000","last_change":"2024-01-15T17:58:53.778730+0000","last_active":"2024-01-15T17:58:53.782764+0000","last_peered":"2024-01-15T17:58:53.782764+0000","last_clean":"2024-01-15T17:58:53.782764+0000","last_became_active":"2024-01-15T17:58:53.777103+0000","last_became_peered":"2024-01-15T17:58:53.777103+0000","last_unstale":"2024-01-15T17:58:53.782764+0000","last_undegraded":"2024-01-15T17:58:53.782764+0000","last_fullsized":"2024-01-15T17:58:53.782764+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:24:27.345667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760047+0000","last_change":"2024-01-15T17:58:53.760047+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760047+0000","last_undegraded":"2024-01-15T17:58:53.760047+0000","last_fullsized":"2024-01-15T17:58:53.760047+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.775615+0000","last_change":"2024-01-15T17:58:53.769315+0000","last_active":"2024-01-15T17:58:53.775615+0000","last_peered":"2024-01-15T17:58:53.775615+0000","last_clean":"2024-01-15T17:58:53.775615+0000","last_became_active":"2024-01-15T17:58:53.769028+0000","last_became_peered":"2024-01-15T17:58:53.769028+0000","last_unstale":"2024-01-15T17:58:53.775615+0000","last_undegraded":"2024-01-15T17:58:53.775615+0000","last_fullsized":"2024-01-15T17:58:53.775615+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.765380+0000","last_change":"2024-01-15T17:58:53.765380+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.765380+0000","last_undegraded":"2024-01-15T17:58:53.765380+0000","last_fullsized":"2024-01-15T17:58:53.765380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781943+0000","last_change":"2024-01-15T17:58:53.778012+0000","last_active":"2024-01-15T17:58:53.781943+0000","last_peered":"2024-01-15T17:58:53.781943+0000","last_clean":"2024-01-15T17:58:53.781943+0000","last_became_active":"2024-01-15T17:58:53.776887+0000","last_became_peered":"2024-01-15T17:58:53.776887+0000","last_unstale":"2024-01-15T17:58:53.781943+0000","last_undegraded":"2024-01-15T17:58:53.781943+0000","last_fullsized":"2024-01-15T17:58:53.781943+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:24.725739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757492+0000","last_change":"2024-01-15T17:58:53.757492+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757492+0000","last_undegraded":"2024-01-15T17:58:53.757492+0000","last_fullsized":"2024-01-15T17:58:53.757492+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787587+0000","last_change":"2024-01-15T17:58:53.786507+0000","last_active":"2024-01-15T17:58:53.787587+0000","last_peered":"2024-01-15T17:58:53.787587+0000","last_clean":"2024-01-15T17:58:53.787587+0000","last_became_active":"2024-01-15T17:58:53.786154+0000","last_became_peered":"2024-01-15T17:58:53.786154+0000","last_unstale":"2024-01-15T17:58:53.787587+0000","last_undegraded":"2024-01-15T17:58:53.787587+0000","last_fullsized":"2024-01-15T17:58:53.787587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:33.284238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760012+0000","last_change":"2024-01-15T17:58:53.760012+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760012+0000","last_undegraded":"2024-01-15T17:58:53.760012+0000","last_fullsized":"2024-01-15T17:58:53.760012+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781046+0000","last_change":"2024-01-15T17:58:53.779129+0000","last_active":"2024-01-15T17:58:53.781046+0000","last_peered":"2024-01-15T17:58:53.781046+0000","last_clean":"2024-01-15T17:58:53.781046+0000","last_became_active":"2024-01-15T17:58:53.778863+0000","last_became_peered":"2024-01-15T17:58:53.778863+0000","last_unstale":"2024-01-15T17:58:53.781046+0000","last_undegraded":"2024-01-15T17:58:53.781046+0000","last_fullsized":"2024-01-15T17:58:53.781046+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758478+0000","last_change":"2024-01-15T17:58:53.758478+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758478+0000","last_undegraded":"2024-01-15T17:58:53.758478+0000","last_fullsized":"2024-01-15T17:58:53.758478+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784603+0000","last_change":"2024-01-15T17:58:53.782469+0000","last_active":"2024-01-15T17:58:53.784603+0000","last_peered":"2024-01-15T17:58:53.784603+0000","last_clean":"2024-01-15T17:58:53.784603+0000","last_became_active":"2024-01-15T17:58:53.782219+0000","last_became_peered":"2024-01-15T17:58:53.782219+0000","last_unstale":"2024-01-15T17:58:53.784603+0000","last_undegraded":"2024-01-15T17:58:53.784603+0000","last_fullsized":"2024-01-15T17:58:53.784603+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758109+0000","last_change":"2024-01-15T17:58:53.758109+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758109+0000","last_undegraded":"2024-01-15T17:58:53.758109+0000","last_fullsized":"2024-01-15T17:58:53.758109+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781322+0000","last_change":"2024-01-15T17:58:53.778698+0000","last_active":"2024-01-15T17:58:53.781322+0000","last_peered":"2024-01-15T17:58:53.781322+0000","last_clean":"2024-01-15T17:58:53.781322+0000","last_became_active":"2024-01-15T17:58:53.778416+0000","last_became_peered":"2024-01-15T17:58:53.778416+0000","last_unstale":"2024-01-15T17:58:53.781322+0000","last_undegraded":"2024-01-15T17:58:53.781322+0000","last_fullsized":"2024-01-15T17:58:53.781322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:40:50.242725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762084+0000","last_change":"2024-01-15T17:58:53.762084+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762084+0000","last_undegraded":"2024-01-15T17:58:53.762084+0000","last_fullsized":"2024-01-15T17:58:53.762084+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782813+0000","last_change":"2024-01-15T17:58:53.780478+0000","last_active":"2024-01-15T17:58:53.782813+0000","last_peered":"2024-01-15T17:58:53.782813+0000","last_clean":"2024-01-15T17:58:53.782813+0000","last_became_active":"2024-01-15T17:58:53.777564+0000","last_became_peered":"2024-01-15T17:58:53.777564+0000","last_unstale":"2024-01-15T17:58:53.782813+0000","last_undegraded":"2024-01-15T17:58:53.782813+0000","last_fullsized":"2024-01-15T17:58:53.782813+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:43.478841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757520+0000","last_change":"2024-01-15T17:58:53.757520+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757520+0000","last_undegraded":"2024-01-15T17:58:53.757520+0000","last_fullsized":"2024-01-15T17:58:53.757520+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783720+0000","last_change":"2024-01-15T17:58:53.777976+0000","last_active":"2024-01-15T17:58:53.783720+0000","last_peered":"2024-01-15T17:58:53.783720+0000","last_clean":"2024-01-15T17:58:53.783720+0000","last_became_active":"2024-01-15T17:58:53.777658+0000","last_became_peered":"2024-01-15T17:58:53.777658+0000","last_unstale":"2024-01-15T17:58:53.783720+0000","last_undegraded":"2024-01-15T17:58:53.783720+0000","last_fullsized":"2024-01-15T17:58:53.783720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.776953+0000","last_change":"2024-01-15T17:58:53.776953+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.776953+0000","last_undegraded":"2024-01-15T17:58:53.776953+0000","last_fullsized":"2024-01-15T17:58:53.776953+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.785944+0000","last_change":"2024-01-15T17:58:53.782063+0000","last_active":"2024-01-15T17:58:53.785944+0000","last_peered":"2024-01-15T17:58:53.785944+0000","last_clean":"2024-01-15T17:58:53.785944+0000","last_became_active":"2024-01-15T17:58:53.781761+0000","last_became_peered":"2024-01-15T17:58:53.781761+0000","last_unstale":"2024-01-15T17:58:53.785944+0000","last_undegraded":"2024-01-15T17:58:53.785944+0000","last_fullsized":"2024-01-15T17:58:53.785944+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:05:52.305613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.768343+0000","last_change":"2024-01-15T17:58:53.768343+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.768343+0000","last_undegraded":"2024-01-15T17:58:53.768343+0000","last_fullsized":"2024-01-15T17:58:53.768343+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779694+0000","last_change":"2024-01-15T17:58:53.775656+0000","last_active":"2024-01-15T17:58:53.779694+0000","last_peered":"2024-01-15T17:58:53.779694+0000","last_clean":"2024-01-15T17:58:53.779694+0000","last_became_active":"2024-01-15T17:58:53.775310+0000","last_became_peered":"2024-01-15T17:58:53.775310+0000","last_unstale":"2024-01-15T17:58:53.779694+0000","last_undegraded":"2024-01-15T17:58:53.779694+0000","last_fullsized":"2024-01-15T17:58:53.779694+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:16:51.863117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.763189+0000","last_change":"2024-01-15T17:58:53.763189+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.763189+0000","last_undegraded":"2024-01-15T17:58:53.763189+0000","last_fullsized":"2024-01-15T17:58:53.763189+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783884+0000","last_change":"2024-01-15T17:58:53.779965+0000","last_active":"2024-01-15T17:58:53.783884+0000","last_peered":"2024-01-15T17:58:53.783884+0000","last_clean":"2024-01-15T17:58:53.783884+0000","last_became_active":"2024-01-15T17:58:53.779597+0000","last_became_peered":"2024-01-15T17:58:53.779597+0000","last_unstale":"2024-01-15T17:58:53.783884+0000","last_undegraded":"2024-01-15T17:58:53.783884+0000","last_fullsized":"2024-01-15T17:58:53.783884+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:22:32.167523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762156+0000","last_change":"2024-01-15T17:58:53.762156+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762156+0000","last_undegraded":"2024-01-15T17:58:53.762156+0000","last_fullsized":"2024-01-15T17:58:53.762156+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779888+0000","last_change":"2024-01-15T17:58:53.774956+0000","last_active":"2024-01-15T17:58:53.779888+0000","last_peered":"2024-01-15T17:58:53.779888+0000","last_clean":"2024-01-15T17:58:53.779888+0000","last_became_active":"2024-01-15T17:58:53.774680+0000","last_became_peered":"2024-01-15T17:58:53.774680+0000","last_unstale":"2024-01-15T17:58:53.779888+0000","last_undegraded":"2024-01-15T17:58:53.779888+0000","last_fullsized":"2024-01-15T17:58:53.779888+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:26:04.369455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.766670+0000","last_change":"2024-01-15T17:58:53.766670+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.766670+0000","last_undegraded":"2024-01-15T17:58:53.766670+0000","last_fullsized":"2024-01-15T17:58:53.766670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779743+0000","last_change":"2024-01-15T17:58:53.772674+0000","last_active":"2024-01-15T17:58:53.779743+0000","last_peered":"2024-01-15T17:58:53.779743+0000","last_clean":"2024-01-15T17:58:53.779743+0000","last_became_active":"2024-01-15T17:58:53.769008+0000","last_became_peered":"2024-01-15T17:58:53.769008+0000","last_unstale":"2024-01-15T17:58:53.779743+0000","last_undegraded":"2024-01-15T17:58:53.779743+0000","last_fullsized":"2024-01-15T17:58:53.779743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:09.191112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761263+0000","last_change":"2024-01-15T17:58:53.761263+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761263+0000","last_undegraded":"2024-01-15T17:58:53.761263+0000","last_fullsized":"2024-01-15T17:58:53.761263+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782726+0000","last_change":"2024-01-15T17:58:53.779480+0000","last_active":"2024-01-15T17:58:53.782726+0000","last_peered":"2024-01-15T17:58:53.782726+0000","last_clean":"2024-01-15T17:58:53.782726+0000","last_became_active":"2024-01-15T17:58:53.778673+0000","last_became_peered":"2024-01-15T17:58:53.778673+0000","last_unstale":"2024-01-15T17:58:53.782726+0000","last_undegraded":"2024-01-15T17:58:53.782726+0000","last_fullsized":"2024-01-15T17:58:53.782726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:28:40.298878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758081+0000","last_change":"2024-01-15T17:58:53.758081+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758081+0000","last_undegraded":"2024-01-15T17:58:53.758081+0000","last_fullsized":"2024-01-15T17:58:53.758081+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.781876+0000","last_change":"2024-01-15T17:58:53.780996+0000","last_active":"2024-01-15T17:58:53.781876+0000","last_peered":"2024-01-15T17:58:53.781876+0000","last_clean":"2024-01-15T17:58:53.781876+0000","last_became_active":"2024-01-15T17:58:53.778816+0000","last_became_peered":"2024-01-15T17:58:53.778816+0000","last_unstale":"2024-01-15T17:58:53.781876+0000","last_undegraded":"2024-01-15T17:58:53.781876+0000","last_fullsized":"2024-01-15T17:58:53.781876+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758792+0000","last_change":"2024-01-15T17:58:53.758792+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758792+0000","last_undegraded":"2024-01-15T17:58:53.758792+0000","last_fullsized":"2024-01-15T17:58:53.758792+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786081+0000","last_change":"2024-01-15T17:58:53.784613+0000","last_active":"2024-01-15T17:58:53.786081+0000","last_peered":"2024-01-15T17:58:53.786081+0000","last_clean":"2024-01-15T17:58:53.786081+0000","last_became_active":"2024-01-15T17:58:53.784295+0000","last_became_peered":"2024-01-15T17:58:53.784295+0000","last_unstale":"2024-01-15T17:58:53.786081+0000","last_undegraded":"2024-01-15T17:58:53.786081+0000","last_fullsized":"2024-01-15T17:58:53.786081+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.757313+0000","last_change":"2024-01-15T17:58:53.757313+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.757313+0000","last_undegraded":"2024-01-15T17:58:53.757313+0000","last_fullsized":"2024-01-15T17:58:53.757313+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.786057+0000","last_change":"2024-01-15T17:58:53.785047+0000","last_active":"2024-01-15T17:58:53.786057+0000","last_peered":"2024-01-15T17:58:53.786057+0000","last_clean":"2024-01-15T17:58:53.786057+0000","last_became_active":"2024-01-15T17:58:53.784573+0000","last_became_peered":"2024-01-15T17:58:53.784573+0000","last_unstale":"2024-01-15T17:58:53.786057+0000","last_undegraded":"2024-01-15T17:58:53.786057+0000","last_fullsized":"2024-01-15T17:58:53.786057+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:07.495003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771821+0000","last_change":"2024-01-15T17:58:53.771821+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771821+0000","last_undegraded":"2024-01-15T17:58:53.771821+0000","last_fullsized":"2024-01-15T17:58:53.771821+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784703+0000","last_change":"2024-01-15T17:58:53.782707+0000","last_active":"2024-01-15T17:58:53.784703+0000","last_peered":"2024-01-15T17:58:53.784703+0000","last_clean":"2024-01-15T17:58:53.784703+0000","last_became_active":"2024-01-15T17:58:53.782425+0000","last_became_peered":"2024-01-15T17:58:53.782425+0000","last_unstale":"2024-01-15T17:58:53.784703+0000","last_undegraded":"2024-01-15T17:58:53.784703+0000","last_fullsized":"2024-01-15T17:58:53.784703+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:57.822622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.762554+0000","last_change":"2024-01-15T17:58:53.762554+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.762554+0000","last_undegraded":"2024-01-15T17:58:53.762554+0000","last_fullsized":"2024-01-15T17:58:53.762554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.783352+0000","last_change":"2024-01-15T17:58:53.776979+0000","last_active":"2024-01-15T17:58:53.783352+0000","last_peered":"2024-01-15T17:58:53.783352+0000","last_clean":"2024-01-15T17:58:53.783352+0000","last_became_active":"2024-01-15T17:58:53.776609+0000","last_became_peered":"2024-01-15T17:58:53.776609+0000","last_unstale":"2024-01-15T17:58:53.783352+0000","last_undegraded":"2024-01-15T17:58:53.783352+0000","last_fullsized":"2024-01-15T17:58:53.783352+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761363+0000","last_change":"2024-01-15T17:58:53.761363+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761363+0000","last_undegraded":"2024-01-15T17:58:53.761363+0000","last_fullsized":"2024-01-15T17:58:53.761363+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779213+0000","last_change":"2024-01-15T17:58:53.777358+0000","last_active":"2024-01-15T17:58:53.779213+0000","last_peered":"2024-01-15T17:58:53.779213+0000","last_clean":"2024-01-15T17:58:53.779213+0000","last_became_active":"2024-01-15T17:58:53.777077+0000","last_became_peered":"2024-01-15T17:58:53.777077+0000","last_unstale":"2024-01-15T17:58:53.779213+0000","last_undegraded":"2024-01-15T17:58:53.779213+0000","last_fullsized":"2024-01-15T17:58:53.779213+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.760665+0000","last_change":"2024-01-15T17:58:53.760665+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.760665+0000","last_undegraded":"2024-01-15T17:58:53.760665+0000","last_fullsized":"2024-01-15T17:58:53.760665+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.777700+0000","last_change":"2024-01-15T17:58:53.775372+0000","last_active":"2024-01-15T17:58:53.777700+0000","last_peered":"2024-01-15T17:58:53.777700+0000","last_clean":"2024-01-15T17:58:53.777700+0000","last_became_active":"2024-01-15T17:58:53.775056+0000","last_became_peered":"2024-01-15T17:58:53.775056+0000","last_unstale":"2024-01-15T17:58:53.777700+0000","last_undegraded":"2024-01-15T17:58:53.777700+0000","last_fullsized":"2024-01-15T17:58:53.777700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:13:06.211493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.770135+0000","last_change":"2024-01-15T17:58:53.770135+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.770135+0000","last_undegraded":"2024-01-15T17:58:53.770135+0000","last_fullsized":"2024-01-15T17:58:53.770135+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.772530+0000","last_change":"2024-01-15T17:58:53.770121+0000","last_active":"2024-01-15T17:58:53.772530+0000","last_peered":"2024-01-15T17:58:53.772530+0000","last_clean":"2024-01-15T17:58:53.772530+0000","last_became_active":"2024-01-15T17:58:53.769805+0000","last_became_peered":"2024-01-15T17:58:53.769805+0000","last_unstale":"2024-01-15T17:58:53.772530+0000","last_undegraded":"2024-01-15T17:58:53.772530+0000","last_fullsized":"2024-01-15T17:58:53.772530+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:40.536007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.758192+0000","last_change":"2024-01-15T17:58:53.758192+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.758192+0000","last_undegraded":"2024-01-15T17:58:53.758192+0000","last_fullsized":"2024-01-15T17:58:53.758192+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.782268+0000","last_change":"2024-01-15T17:58:53.776755+0000","last_active":"2024-01-15T17:58:53.782268+0000","last_peered":"2024-01-15T17:58:53.782268+0000","last_clean":"2024-01-15T17:58:53.782268+0000","last_became_active":"2024-01-15T17:58:53.776423+0000","last_became_peered":"2024-01-15T17:58:53.776423+0000","last_unstale":"2024-01-15T17:58:53.782268+0000","last_undegraded":"2024-01-15T17:58:53.782268+0000","last_fullsized":"2024-01-15T17:58:53.782268+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:11:22.065581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.756946+0000","last_change":"2024-01-15T17:58:53.756946+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.756946+0000","last_undegraded":"2024-01-15T17:58:53.756946+0000","last_fullsized":"2024-01-15T17:58:53.756946+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.779312+0000","last_change":"2024-01-15T17:58:53.770125+0000","last_active":"2024-01-15T17:58:53.779312+0000","last_peered":"2024-01-15T17:58:53.779312+0000","last_clean":"2024-01-15T17:58:53.779312+0000","last_became_active":"2024-01-15T17:58:53.769709+0000","last_became_peered":"2024-01-15T17:58:53.769709+0000","last_unstale":"2024-01-15T17:58:53.779312+0000","last_undegraded":"2024-01-15T17:58:53.779312+0000","last_fullsized":"2024-01-15T17:58:53.779312+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:17.227945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.761458+0000","last_change":"2024-01-15T17:58:53.761458+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.761458+0000","last_undegraded":"2024-01-15T17:58:53.761458+0000","last_fullsized":"2024-01-15T17:58:53.761458+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.787274+0000","last_change":"2024-01-15T17:58:53.786548+0000","last_active":"2024-01-15T17:58:53.787274+0000","last_peered":"2024-01-15T17:58:53.787274+0000","last_clean":"2024-01-15T17:58:53.787274+0000","last_became_active":"2024-01-15T17:58:53.786220+0000","last_became_peered":"2024-01-15T17:58:53.786220+0000","last_unstale":"2024-01-15T17:58:53.787274+0000","last_undegraded":"2024-01-15T17:58:53.787274+0000","last_fullsized":"2024-01-15T17:58:53.787274+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:06:45.660613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":33,"state":"creating+peering","last_fresh":"2024-01-15T17:58:53.771119+0000","last_change":"2024-01-15T17:58:53.771119+0000","last_active":"2024-01-15T17:58:53.739527+0000","last_peered":"2024-01-15T17:58:53.739527+0000","last_clean":"2024-01-15T17:58:53.739527+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T17:58:53.771119+0000","last_undegraded":"2024-01-15T17:58:53.771119+0000","last_fullsized":"2024-01-15T17:58:53.771119+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-01-15T17:58:53.784763+0000","last_change":"2024-01-15T17:58:53.783339+0000","last_active":"2024-01-15T17:58:53.784763+0000","last_peered":"2024-01-15T17:58:53.784763+0000","last_clean":"2024-01-15T17:58:53.784763+0000","last_became_active":"2024-01-15T17:58:53.783028+0000","last_became_peered":"2024-01-15T17:58:53.783028+0000","last_unstale":"2024-01-15T17:58:53.784763+0000","last_undegraded":"2024-01-15T17:58:53.784763+0000","last_fullsized":"2024-01-15T17:58:53.784763+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:31:41.684200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":173,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-01-15T17:58:52.760774+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:58:52.760774+0000","last_peered":"2024-01-15T17:58:52.760774+0000","last_clean":"2024-01-15T17:58:52.760774+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:58:52.760774+0000","last_undegraded":"2024-01-15T17:58:52.760774+0000","last_fullsized":"2024-01-15T17:58:52.760774+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343892,"statfs":{"total":96636764160,"available":96608145408,"internally_reserved":0,"allocated":950272,"data_stored":725510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.6970000000000001,"5min":7.6970000000000001,"15min":7.6970000000000001},"last":1.339}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":1.196},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":1.5760000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.53}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.694}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":7.4809999999999999,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.2619999999999996,"5min":4.2619999999999996,"15min":4.2619999999999996},"last":1.3799999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.462},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":7.798,"5min":7.798,"15min":7.798},"last":1.4299999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":7.968,"5min":7.968,"15min":7.968},"last":1.611},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.6439999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27384,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344456,"statfs":{"total":96636764160,"available":96608722944,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.5009999999999999}]},{"osd":1,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.321},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.4019999999999999}]},{"osd":2,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":0.72999999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":0.997}]},{"osd":3,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.84999999999999998}]},{"osd":4,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.198},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.4359999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.073,"5min":1.073,"15min":1.073},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.4710000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27368,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344472,"statfs":{"total":96636764160,"available":96608739328,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.84299999999999997}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.653}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":1.323},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":1.1639999999999999}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":1.48},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.431}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.0669999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":1.5329999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":1.272},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":1.5840000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":0.752}]},{"osd":1,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.3720000000000001}]},{"osd":2,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.613},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.1100000000000001}]},{"osd":3,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.4259999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.4830000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.8129999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.526}]},{"osd":7,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.7230000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.008}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.161}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.419},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.5369999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.7}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.49},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.669}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.7889999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.5780000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":356352,"data_stored":135142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.105}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5150000000000001,"5min":2.5150000000000001,"15min":2.5150000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":1.002}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.2}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.3400000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":2.0089999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.7150000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.675,"15min":1.675},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":1.758},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.7559999999999998,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.5649999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":1.149,"5min":1.149,"15min":1.149},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":1.847},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.6279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31544,"kb_used_data":360,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340296,"statfs":{"total":96636764160,"available":96604463104,"internally_reserved":0,"allocated":368640,"data_stored":141409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.72799999999999998}]},{"osd":2,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.78800000000000003}]},{"osd":3,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.306}]},{"osd":4,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.379},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.5189999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.4430000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.6699999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"last":1.54},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.339}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":962560,"data_stored":731777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":1.069},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.81499999999999995}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.196},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.68100000000000005}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":1.337},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.3100000000000001}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.2450000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.389}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.4570000000000001}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.569},{"interface":"front","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.679}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:59:04.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T17:59:04.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.633+0000 7fe576600640 1 Processor -- start 2024-01-15T17:59:04.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.634+0000 7fe576600640 1 -- start start 2024-01-15T17:59:04.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.635+0000 7fe576600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5700d3090 0x7fe5700d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:04.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.635+0000 7fe576600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe5700d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:04.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.635+0000 7fe576600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5700d3a60 con 0x7fe57015d340 2024-01-15T17:59:04.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.635+0000 7fe576600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5700ce430 con 0x7fe5700d3090 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.635+0000 7fe576600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700b3de0 con 0x7fe5700d1680 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5700d3090 0x7fe5700d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5700d3090 0x7fe5700d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52548/0 (socket says 172.21.15.2:52548) 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 -- 172.21.15.2:0/3403877071 learned_addr learned my addr 172.21.15.2:0/3403877071 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe575c00640 1 --2- 172.21.15.2:0/3403877071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe5700d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 -- 172.21.15.2:0/3403877071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 msgr2=0x7fe5700d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 --2- 172.21.15.2:0/3403877071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe5700d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 -- 172.21.15.2:0/3403877071 >> v1:172.21.15.137:6789/0 conn(0x7fe57015d340 legacy=0x7fe5700cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:04.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.636+0000 7fe56fe00640 1 -- 172.21.15.2:0/3403877071 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700b40c0 con 0x7fe5700d3090 2024-01-15T17:59:04.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.637+0000 7fe56fe00640 1 --2- 172.21.15.2:0/3403877071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5700d3090 0x7fe5700d3520 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe560009e50 tx=0x7fe56002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7693372d9bcbfd86 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:04.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.638+0000 7fe56f400640 1 -- 172.21.15.2:0/3403877071 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56002ca80 con 0x7fe5700d3090 2024-01-15T17:59:04.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.638+0000 7fe56f400640 1 -- 172.21.15.2:0/3403877071 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56002cbe0 con 0x7fe5700d3090 2024-01-15T17:59:04.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.638+0000 7fe56f400640 1 -- 172.21.15.2:0/3403877071 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5600343f0 con 0x7fe5700d3090 2024-01-15T17:59:04.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.638+0000 7fe576600640 1 -- 172.21.15.2:0/3403877071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5700d3090 msgr2=0x7fe5700d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:04.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.638+0000 7fe576600640 1 --2- 172.21.15.2:0/3403877071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5700d3090 0x7fe5700d3520 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fe560009e50 tx=0x7fe56002bee0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.639+0000 7fe576600640 1 -- 172.21.15.2:0/3403877071 shutdown_connections 2024-01-15T17:59:04.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.639+0000 7fe576600640 1 --2- 172.21.15.2:0/3403877071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe5700d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.639+0000 7fe576600640 1 --2- 172.21.15.2:0/3403877071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5700d3090 0x7fe5700d3520 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.639+0000 7fe576600640 1 -- 172.21.15.2:0/3403877071 >> 172.21.15.2:0/3403877071 conn(0x7fe57008caf0 msgr2=0x7fe57009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:04.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.639+0000 7fe576600640 1 -- 172.21.15.2:0/3403877071 shutdown_connections 2024-01-15T17:59:04.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.639+0000 7fe576600640 1 -- 172.21.15.2:0/3403877071 wait complete. 2024-01-15T17:59:04.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.641+0000 7fe576600640 1 Processor -- start 2024-01-15T17:59:04.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.641+0000 7fe576600640 1 -- start start 2024-01-15T17:59:04.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe576600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe57015efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:04.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe576600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015d340 0x7fe570158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:04.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe576600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 0x7fe5700c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:04.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe576600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57015f4f0 con 0x7fe57015da50 2024-01-15T17:59:04.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe576600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe57015df10 con 0x7fe57015d340 2024-01-15T17:59:04.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe576600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe57015e1f0 con 0x7fe5700d1680 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 0x7fe5700c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 0x7fe5700c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52554/0 (socket says 172.21.15.2:52554) 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 -- 172.21.15.2:0/932073084 learned_addr learned my addr 172.21.15.2:0/932073084 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575200640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe57015efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe56fe00640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015d340 0x7fe570158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 -- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 msgr2=0x7fe57015efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe57015efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 -- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015d340 msgr2=0x7fe570158620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015d340 0x7fe570158620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.642+0000 7fe575c00640 1 -- 172.21.15.2:0/932073084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe560009b30 con 0x7fe57015da50 2024-01-15T17:59:04.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.643+0000 7fe575c00640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 0x7fe5700c93e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe56800c550 tx=0x7fe56800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:04.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.643+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568004220 con 0x7fe57015da50 2024-01-15T17:59:04.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.643+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe568004380 con 0x7fe57015da50 2024-01-15T17:59:04.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.643+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568003830 con 0x7fe57015da50 2024-01-15T17:59:04.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.643+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe57015e4d0 con 0x7fe57015da50 2024-01-15T17:59:04.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.643+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe57015ea90 con 0x7fe57015da50 2024-01-15T17:59:04.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.644+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe57015ec80 con 0x7fe57015da50 2024-01-15T17:59:04.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.645+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe568003990 con 0x7fe57015da50 2024-01-15T17:59:04.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.646+0000 7fe56cc00640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b2f0 0x7fe54806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:04.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.646+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5680898d0 con 0x7fe57015da50 2024-01-15T17:59:04.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.647+0000 7fe575200640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b2f0 0x7fe54806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:04.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.648+0000 7fe575200640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b2f0 0x7fe54806d7b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe558006fd0 tx=0x7fe558008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:04.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.651+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5680546a0 con 0x7fe57015da50 2024-01-15T17:59:04.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.843+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe5700d1160 con 0x7fe54806b2f0 2024-01-15T17:59:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.865+0000 7fe56cc00640 1 -- 172.21.15.2:0/932073084 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286641 (secure 0 0 0) 0x7fe55800be30 con 0x7fe54806b2f0 2024-01-15T17:59:04.874 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:59:04.876 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.871+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b2f0 msgr2=0x7fe54806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.871+0000 7fe576600640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b2f0 0x7fe54806d7b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe558006fd0 tx=0x7fe558008040 comp rx=0 tx=0).stop 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.871+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 msgr2=0x7fe5700c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.871+0000 7fe576600640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 0x7fe5700c93e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fe56800c550 tx=0x7fe56800ca10 comp rx=0 tx=0).stop 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 shutdown_connections 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b2f0 0x7fe54806d7b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015da50 0x7fe5700c93e0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015d340 0x7fe570158620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 --2- 172.21.15.2:0/932073084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5700d1680 0x7fe57015efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:04.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 >> 172.21.15.2:0/932073084 conn(0x7fe57008caf0 msgr2=0x7fe5700b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:04.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 shutdown_connections 2024-01-15T17:59:04.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:04.872+0000 7fe576600640 1 -- 172.21.15.2:0/932073084 wait complete. 2024-01-15T17:59:04.896 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-01-15T17:59:03.401535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237112,"kb_used_data":4296,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737608,"statfs":{"total":773094113280,"available":772851310592,"internally_reserved":0,"allocated":4399104,"data_stored":2421428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.630275"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.028811+0000","last_change":"2024-01-15T17:58:54.820825+0000","last_active":"2024-01-15T17:58:56.028811+0000","last_peered":"2024-01-15T17:58:56.028811+0000","last_clean":"2024-01-15T17:58:56.028811+0000","last_became_active":"2024-01-15T17:58:54.820553+0000","last_became_peered":"2024-01-15T17:58:54.820553+0000","last_unstale":"2024-01-15T17:58:56.028811+0000","last_undegraded":"2024-01-15T17:58:56.028811+0000","last_fullsized":"2024-01-15T17:58:56.028811+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:24.492996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.799936+0000","last_change":"2024-01-15T17:58:54.798187+0000","last_active":"2024-01-15T17:58:55.799936+0000","last_peered":"2024-01-15T17:58:55.799936+0000","last_clean":"2024-01-15T17:58:55.799936+0000","last_became_active":"2024-01-15T17:58:54.797929+0000","last_became_peered":"2024-01-15T17:58:54.797929+0000","last_unstale":"2024-01-15T17:58:55.799936+0000","last_undegraded":"2024-01-15T17:58:55.799936+0000","last_fullsized":"2024-01-15T17:58:55.799936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:37.095623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.805381+0000","last_change":"2024-01-15T17:58:54.812610+0000","last_active":"2024-01-15T17:58:55.805381+0000","last_peered":"2024-01-15T17:58:55.805381+0000","last_clean":"2024-01-15T17:58:55.805381+0000","last_became_active":"2024-01-15T17:58:54.812323+0000","last_became_peered":"2024-01-15T17:58:54.812323+0000","last_unstale":"2024-01-15T17:58:55.805381+0000","last_undegraded":"2024-01-15T17:58:55.805381+0000","last_fullsized":"2024-01-15T17:58:55.805381+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:45:46.963255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.961839+0000","last_change":"2024-01-15T17:58:54.804623+0000","last_active":"2024-01-15T17:58:55.961839+0000","last_peered":"2024-01-15T17:58:55.961839+0000","last_clean":"2024-01-15T17:58:55.961839+0000","last_became_active":"2024-01-15T17:58:54.804240+0000","last_became_peered":"2024-01-15T17:58:54.804240+0000","last_unstale":"2024-01-15T17:58:55.961839+0000","last_undegraded":"2024-01-15T17:58:55.961839+0000","last_fullsized":"2024-01-15T17:58:55.961839+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:24:54.976704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.026783+0000","last_change":"2024-01-15T17:58:54.791530+0000","last_active":"2024-01-15T17:58:56.026783+0000","last_peered":"2024-01-15T17:58:56.026783+0000","last_clean":"2024-01-15T17:58:56.026783+0000","last_became_active":"2024-01-15T17:58:54.791231+0000","last_became_peered":"2024-01-15T17:58:54.791231+0000","last_unstale":"2024-01-15T17:58:56.026783+0000","last_undegraded":"2024-01-15T17:58:56.026783+0000","last_fullsized":"2024-01-15T17:58:56.026783+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:12.703892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.798842+0000","last_change":"2024-01-15T17:58:54.805714+0000","last_active":"2024-01-15T17:58:55.798842+0000","last_peered":"2024-01-15T17:58:55.798842+0000","last_clean":"2024-01-15T17:58:55.798842+0000","last_became_active":"2024-01-15T17:58:54.805428+0000","last_became_peered":"2024-01-15T17:58:54.805428+0000","last_unstale":"2024-01-15T17:58:55.798842+0000","last_undegraded":"2024-01-15T17:58:55.798842+0000","last_fullsized":"2024-01-15T17:58:55.798842+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:04.722030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.802215+0000","last_change":"2024-01-15T17:58:54.807305+0000","last_active":"2024-01-15T17:58:55.802215+0000","last_peered":"2024-01-15T17:58:55.802215+0000","last_clean":"2024-01-15T17:58:55.802215+0000","last_became_active":"2024-01-15T17:58:54.806791+0000","last_became_peered":"2024-01-15T17:58:54.806791+0000","last_unstale":"2024-01-15T17:58:55.802215+0000","last_undegraded":"2024-01-15T17:58:55.802215+0000","last_fullsized":"2024-01-15T17:58:55.802215+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:08:02.249232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.801967+0000","last_change":"2024-01-15T17:58:54.814025+0000","last_active":"2024-01-15T17:58:55.801967+0000","last_peered":"2024-01-15T17:58:55.801967+0000","last_clean":"2024-01-15T17:58:55.801967+0000","last_became_active":"2024-01-15T17:58:54.813757+0000","last_became_peered":"2024-01-15T17:58:54.813757+0000","last_unstale":"2024-01-15T17:58:55.801967+0000","last_undegraded":"2024-01-15T17:58:55.801967+0000","last_fullsized":"2024-01-15T17:58:55.801967+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:33.355168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.029267+0000","last_change":"2024-01-15T17:58:54.800097+0000","last_active":"2024-01-15T17:58:56.029267+0000","last_peered":"2024-01-15T17:58:56.029267+0000","last_clean":"2024-01-15T17:58:56.029267+0000","last_became_active":"2024-01-15T17:58:54.799812+0000","last_became_peered":"2024-01-15T17:58:54.799812+0000","last_unstale":"2024-01-15T17:58:56.029267+0000","last_undegraded":"2024-01-15T17:58:56.029267+0000","last_fullsized":"2024-01-15T17:58:56.029267+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:10:51.353263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.962235+0000","last_change":"2024-01-15T17:58:54.794257+0000","last_active":"2024-01-15T17:58:55.962235+0000","last_peered":"2024-01-15T17:58:55.962235+0000","last_clean":"2024-01-15T17:58:55.962235+0000","last_became_active":"2024-01-15T17:58:54.793957+0000","last_became_peered":"2024-01-15T17:58:54.793957+0000","last_unstale":"2024-01-15T17:58:55.962235+0000","last_undegraded":"2024-01-15T17:58:55.962235+0000","last_fullsized":"2024-01-15T17:58:55.962235+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:45.501702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.804489+0000","last_change":"2024-01-15T17:58:54.817659+0000","last_active":"2024-01-15T17:58:55.804489+0000","last_peered":"2024-01-15T17:58:55.804489+0000","last_clean":"2024-01-15T17:58:55.804489+0000","last_became_active":"2024-01-15T17:58:54.816179+0000","last_became_peered":"2024-01-15T17:58:54.816179+0000","last_unstale":"2024-01-15T17:58:55.804489+0000","last_undegraded":"2024-01-15T17:58:55.804489+0000","last_fullsized":"2024-01-15T17:58:55.804489+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:31:14.000701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.804270+0000","last_change":"2024-01-15T17:58:54.803074+0000","last_active":"2024-01-15T17:58:55.804270+0000","last_peered":"2024-01-15T17:58:55.804270+0000","last_clean":"2024-01-15T17:58:55.804270+0000","last_became_active":"2024-01-15T17:58:54.802008+0000","last_became_peered":"2024-01-15T17:58:54.802008+0000","last_unstale":"2024-01-15T17:58:55.804270+0000","last_undegraded":"2024-01-15T17:58:55.804270+0000","last_fullsized":"2024-01-15T17:58:55.804270+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:58.006337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.026930+0000","last_change":"2024-01-15T17:58:54.792636+0000","last_active":"2024-01-15T17:58:56.026930+0000","last_peered":"2024-01-15T17:58:56.026930+0000","last_clean":"2024-01-15T17:58:56.026930+0000","last_became_active":"2024-01-15T17:58:54.791917+0000","last_became_peered":"2024-01-15T17:58:54.791917+0000","last_unstale":"2024-01-15T17:58:56.026930+0000","last_undegraded":"2024-01-15T17:58:56.026930+0000","last_fullsized":"2024-01-15T17:58:56.026930+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:51.879538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.798911+0000","last_change":"2024-01-15T17:58:54.788912+0000","last_active":"2024-01-15T17:58:55.798911+0000","last_peered":"2024-01-15T17:58:55.798911+0000","last_clean":"2024-01-15T17:58:55.798911+0000","last_became_active":"2024-01-15T17:58:54.788585+0000","last_became_peered":"2024-01-15T17:58:54.788585+0000","last_unstale":"2024-01-15T17:58:55.798911+0000","last_undegraded":"2024-01-15T17:58:55.798911+0000","last_fullsized":"2024-01-15T17:58:55.798911+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:31.587532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.803093+0000","last_change":"2024-01-15T17:58:54.811344+0000","last_active":"2024-01-15T17:58:55.803093+0000","last_peered":"2024-01-15T17:58:55.803093+0000","last_clean":"2024-01-15T17:58:55.803093+0000","last_became_active":"2024-01-15T17:58:54.810983+0000","last_became_peered":"2024-01-15T17:58:54.810983+0000","last_unstale":"2024-01-15T17:58:55.803093+0000","last_undegraded":"2024-01-15T17:58:55.803093+0000","last_fullsized":"2024-01-15T17:58:55.803093+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.913324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.803322+0000","last_change":"2024-01-15T17:58:54.812377+0000","last_active":"2024-01-15T17:58:55.803322+0000","last_peered":"2024-01-15T17:58:55.803322+0000","last_clean":"2024-01-15T17:58:55.803322+0000","last_became_active":"2024-01-15T17:58:54.812090+0000","last_became_peered":"2024-01-15T17:58:54.812090+0000","last_unstale":"2024-01-15T17:58:55.803322+0000","last_undegraded":"2024-01-15T17:58:55.803322+0000","last_fullsized":"2024-01-15T17:58:55.803322+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:44.162291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.805285+0000","last_change":"2024-01-15T17:58:54.818846+0000","last_active":"2024-01-15T17:58:55.805285+0000","last_peered":"2024-01-15T17:58:55.805285+0000","last_clean":"2024-01-15T17:58:55.805285+0000","last_became_active":"2024-01-15T17:58:54.818268+0000","last_became_peered":"2024-01-15T17:58:54.818268+0000","last_unstale":"2024-01-15T17:58:55.805285+0000","last_undegraded":"2024-01-15T17:58:55.805285+0000","last_fullsized":"2024-01-15T17:58:55.805285+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:45:37.095623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.800091+0000","last_change":"2024-01-15T17:58:54.796613+0000","last_active":"2024-01-15T17:58:55.800091+0000","last_peered":"2024-01-15T17:58:55.800091+0000","last_clean":"2024-01-15T17:58:55.800091+0000","last_became_active":"2024-01-15T17:58:54.795089+0000","last_became_peered":"2024-01-15T17:58:54.795089+0000","last_unstale":"2024-01-15T17:58:55.800091+0000","last_undegraded":"2024-01-15T17:58:55.800091+0000","last_fullsized":"2024-01-15T17:58:55.800091+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:41:44.650075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.799368+0000","last_change":"2024-01-15T17:58:54.795531+0000","last_active":"2024-01-15T17:58:55.799368+0000","last_peered":"2024-01-15T17:58:55.799368+0000","last_clean":"2024-01-15T17:58:55.799368+0000","last_became_active":"2024-01-15T17:58:54.795251+0000","last_became_peered":"2024-01-15T17:58:54.795251+0000","last_unstale":"2024-01-15T17:58:55.799368+0000","last_undegraded":"2024-01-15T17:58:55.799368+0000","last_fullsized":"2024-01-15T17:58:55.799368+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:45:46.963255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027295+0000","last_change":"2024-01-15T17:58:54.816129+0000","last_active":"2024-01-15T17:58:56.027295+0000","last_peered":"2024-01-15T17:58:56.027295+0000","last_clean":"2024-01-15T17:58:56.027295+0000","last_became_active":"2024-01-15T17:58:54.813178+0000","last_became_peered":"2024-01-15T17:58:54.813178+0000","last_unstale":"2024-01-15T17:58:56.027295+0000","last_undegraded":"2024-01-15T17:58:56.027295+0000","last_fullsized":"2024-01-15T17:58:56.027295+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:32:15.922039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027614+0000","last_change":"2024-01-15T17:58:54.799354+0000","last_active":"2024-01-15T17:58:56.027614+0000","last_peered":"2024-01-15T17:58:56.027614+0000","last_clean":"2024-01-15T17:58:56.027614+0000","last_became_active":"2024-01-15T17:58:54.799078+0000","last_became_peered":"2024-01-15T17:58:54.799078+0000","last_unstale":"2024-01-15T17:58:56.027614+0000","last_undegraded":"2024-01-15T17:58:56.027614+0000","last_fullsized":"2024-01-15T17:58:56.027614+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:07:53.049260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027585+0000","last_change":"2024-01-15T17:58:54.792625+0000","last_active":"2024-01-15T17:58:56.027585+0000","last_peered":"2024-01-15T17:58:56.027585+0000","last_clean":"2024-01-15T17:58:56.027585+0000","last_became_active":"2024-01-15T17:58:54.792371+0000","last_became_peered":"2024-01-15T17:58:54.792371+0000","last_unstale":"2024-01-15T17:58:56.027585+0000","last_undegraded":"2024-01-15T17:58:56.027585+0000","last_fullsized":"2024-01-15T17:58:56.027585+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:51.226838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.955459+0000","last_change":"2024-01-15T17:58:54.806950+0000","last_active":"2024-01-15T17:58:55.955459+0000","last_peered":"2024-01-15T17:58:55.955459+0000","last_clean":"2024-01-15T17:58:55.955459+0000","last_became_active":"2024-01-15T17:58:54.806650+0000","last_became_peered":"2024-01-15T17:58:54.806650+0000","last_unstale":"2024-01-15T17:58:55.955459+0000","last_undegraded":"2024-01-15T17:58:55.955459+0000","last_fullsized":"2024-01-15T17:58:55.955459+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:06.670278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.802290+0000","last_change":"2024-01-15T17:58:54.798085+0000","last_active":"2024-01-15T17:58:55.802290+0000","last_peered":"2024-01-15T17:58:55.802290+0000","last_clean":"2024-01-15T17:58:55.802290+0000","last_became_active":"2024-01-15T17:58:54.797704+0000","last_became_peered":"2024-01-15T17:58:54.797704+0000","last_unstale":"2024-01-15T17:58:55.802290+0000","last_undegraded":"2024-01-15T17:58:55.802290+0000","last_fullsized":"2024-01-15T17:58:55.802290+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:33.355168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.962706+0000","last_change":"2024-01-15T17:58:54.810592+0000","last_active":"2024-01-15T17:58:55.962706+0000","last_peered":"2024-01-15T17:58:55.962706+0000","last_clean":"2024-01-15T17:58:55.962706+0000","last_became_active":"2024-01-15T17:58:54.810221+0000","last_became_peered":"2024-01-15T17:58:54.810221+0000","last_unstale":"2024-01-15T17:58:55.962706+0000","last_undegraded":"2024-01-15T17:58:55.962706+0000","last_fullsized":"2024-01-15T17:58:55.962706+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:47.686425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.800163+0000","last_change":"2024-01-15T17:58:54.810823+0000","last_active":"2024-01-15T17:58:55.800163+0000","last_peered":"2024-01-15T17:58:55.800163+0000","last_clean":"2024-01-15T17:58:55.800163+0000","last_became_active":"2024-01-15T17:58:54.809955+0000","last_became_peered":"2024-01-15T17:58:54.809955+0000","last_unstale":"2024-01-15T17:58:55.800163+0000","last_undegraded":"2024-01-15T17:58:55.800163+0000","last_fullsized":"2024-01-15T17:58:55.800163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:13:11.516144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.962630+0000","last_change":"2024-01-15T17:58:54.811708+0000","last_active":"2024-01-15T17:58:55.962630+0000","last_peered":"2024-01-15T17:58:55.962630+0000","last_clean":"2024-01-15T17:58:55.962630+0000","last_became_active":"2024-01-15T17:58:54.811215+0000","last_became_peered":"2024-01-15T17:58:54.811215+0000","last_unstale":"2024-01-15T17:58:55.962630+0000","last_undegraded":"2024-01-15T17:58:55.962630+0000","last_fullsized":"2024-01-15T17:58:55.962630+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:36.443300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.804544+0000","last_change":"2024-01-15T17:58:54.800703+0000","last_active":"2024-01-15T17:58:55.804544+0000","last_peered":"2024-01-15T17:58:55.804544+0000","last_clean":"2024-01-15T17:58:55.804544+0000","last_became_active":"2024-01-15T17:58:54.798076+0000","last_became_peered":"2024-01-15T17:58:54.798076+0000","last_unstale":"2024-01-15T17:58:55.804544+0000","last_undegraded":"2024-01-15T17:58:55.804544+0000","last_fullsized":"2024-01-15T17:58:55.804544+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:04.722030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027678+0000","last_change":"2024-01-15T17:58:54.804933+0000","last_active":"2024-01-15T17:58:56.027678+0000","last_peered":"2024-01-15T17:58:56.027678+0000","last_clean":"2024-01-15T17:58:56.027678+0000","last_became_active":"2024-01-15T17:58:54.804647+0000","last_became_peered":"2024-01-15T17:58:54.804647+0000","last_unstale":"2024-01-15T17:58:56.027678+0000","last_undegraded":"2024-01-15T17:58:56.027678+0000","last_fullsized":"2024-01-15T17:58:56.027678+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:03:39.262291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.802249+0000","last_change":"2024-01-15T17:58:54.811929+0000","last_active":"2024-01-15T17:58:55.802249+0000","last_peered":"2024-01-15T17:58:55.802249+0000","last_clean":"2024-01-15T17:58:55.802249+0000","last_became_active":"2024-01-15T17:58:54.811661+0000","last_became_peered":"2024-01-15T17:58:54.811661+0000","last_unstale":"2024-01-15T17:58:55.802249+0000","last_undegraded":"2024-01-15T17:58:55.802249+0000","last_fullsized":"2024-01-15T17:58:55.802249+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:16.913324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027856+0000","last_change":"2024-01-15T17:58:54.818526+0000","last_active":"2024-01-15T17:58:56.027856+0000","last_peered":"2024-01-15T17:58:56.027856+0000","last_clean":"2024-01-15T17:58:56.027856+0000","last_became_active":"2024-01-15T17:58:54.817373+0000","last_became_peered":"2024-01-15T17:58:54.817373+0000","last_unstale":"2024-01-15T17:58:56.027856+0000","last_undegraded":"2024-01-15T17:58:56.027856+0000","last_fullsized":"2024-01-15T17:58:56.027856+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:58.834867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.802339+0000","last_change":"2024-01-15T17:58:54.815538+0000","last_active":"2024-01-15T17:58:55.802339+0000","last_peered":"2024-01-15T17:58:55.802339+0000","last_clean":"2024-01-15T17:58:55.802339+0000","last_became_active":"2024-01-15T17:58:54.814713+0000","last_became_peered":"2024-01-15T17:58:54.814713+0000","last_unstale":"2024-01-15T17:58:55.802339+0000","last_undegraded":"2024-01-15T17:58:55.802339+0000","last_fullsized":"2024-01-15T17:58:55.802339+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:58.006337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.956046+0000","last_change":"2024-01-15T17:58:54.810201+0000","last_active":"2024-01-15T17:58:55.956046+0000","last_peered":"2024-01-15T17:58:55.956046+0000","last_clean":"2024-01-15T17:58:55.956046+0000","last_became_active":"2024-01-15T17:58:54.809239+0000","last_became_peered":"2024-01-15T17:58:54.809239+0000","last_unstale":"2024-01-15T17:58:55.956046+0000","last_undegraded":"2024-01-15T17:58:55.956046+0000","last_fullsized":"2024-01-15T17:58:55.956046+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:24.214341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.822484+0000","last_change":"2024-01-15T17:58:53.779462+0000","last_active":"2024-01-15T17:58:54.822484+0000","last_peered":"2024-01-15T17:58:54.822484+0000","last_clean":"2024-01-15T17:58:54.822484+0000","last_became_active":"2024-01-15T17:58:53.779161+0000","last_became_peered":"2024-01-15T17:58:53.779161+0000","last_unstale":"2024-01-15T17:58:54.822484+0000","last_undegraded":"2024-01-15T17:58:54.822484+0000","last_fullsized":"2024-01-15T17:58:54.822484+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:33.284238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.028993+0000","last_change":"2024-01-15T17:58:54.814653+0000","last_active":"2024-01-15T17:58:56.028993+0000","last_peered":"2024-01-15T17:58:56.028993+0000","last_clean":"2024-01-15T17:58:56.028993+0000","last_became_active":"2024-01-15T17:58:54.814354+0000","last_became_peered":"2024-01-15T17:58:54.814354+0000","last_unstale":"2024-01-15T17:58:56.028993+0000","last_undegraded":"2024-01-15T17:58:56.028993+0000","last_fullsized":"2024-01-15T17:58:56.028993+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:06:51.170752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.813702+0000","last_change":"2024-01-15T17:58:53.782887+0000","last_active":"2024-01-15T17:58:54.813702+0000","last_peered":"2024-01-15T17:58:54.813702+0000","last_clean":"2024-01-15T17:58:54.813702+0000","last_became_active":"2024-01-15T17:58:53.782599+0000","last_became_peered":"2024-01-15T17:58:53.782599+0000","last_unstale":"2024-01-15T17:58:54.813702+0000","last_undegraded":"2024-01-15T17:58:54.813702+0000","last_fullsized":"2024-01-15T17:58:54.813702+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:19.366878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.028410+0000","last_change":"2024-01-15T17:58:54.791336+0000","last_active":"2024-01-15T17:58:56.028410+0000","last_peered":"2024-01-15T17:58:56.028410+0000","last_clean":"2024-01-15T17:58:56.028410+0000","last_became_active":"2024-01-15T17:58:54.790860+0000","last_became_peered":"2024-01-15T17:58:54.790860+0000","last_unstale":"2024-01-15T17:58:56.028410+0000","last_undegraded":"2024-01-15T17:58:56.028410+0000","last_fullsized":"2024-01-15T17:58:56.028410+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:20:43.800820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.818362+0000","last_change":"2024-01-15T17:58:53.779558+0000","last_active":"2024-01-15T17:58:54.818362+0000","last_peered":"2024-01-15T17:58:54.818362+0000","last_clean":"2024-01-15T17:58:54.818362+0000","last_became_active":"2024-01-15T17:58:53.779269+0000","last_became_peered":"2024-01-15T17:58:53.779269+0000","last_unstale":"2024-01-15T17:58:54.818362+0000","last_undegraded":"2024-01-15T17:58:54.818362+0000","last_fullsized":"2024-01-15T17:58:54.818362+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:08:01.236987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.962777+0000","last_change":"2024-01-15T17:58:54.812668+0000","last_active":"2024-01-15T17:58:55.962777+0000","last_peered":"2024-01-15T17:58:55.962777+0000","last_clean":"2024-01-15T17:58:55.962777+0000","last_became_active":"2024-01-15T17:58:54.812321+0000","last_became_peered":"2024-01-15T17:58:54.812321+0000","last_unstale":"2024-01-15T17:58:55.962777+0000","last_undegraded":"2024-01-15T17:58:55.962777+0000","last_fullsized":"2024-01-15T17:58:55.962777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:23:08.289986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.795829+0000","last_change":"2024-01-15T17:58:53.776027+0000","last_active":"2024-01-15T17:58:54.795829+0000","last_peered":"2024-01-15T17:58:54.795829+0000","last_clean":"2024-01-15T17:58:54.795829+0000","last_became_active":"2024-01-15T17:58:53.775717+0000","last_became_peered":"2024-01-15T17:58:53.775717+0000","last_unstale":"2024-01-15T17:58:54.795829+0000","last_undegraded":"2024-01-15T17:58:54.795829+0000","last_fullsized":"2024-01-15T17:58:54.795829+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:05.363189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.802627+0000","last_change":"2024-01-15T17:58:54.816308+0000","last_active":"2024-01-15T17:58:55.802627+0000","last_peered":"2024-01-15T17:58:55.802627+0000","last_clean":"2024-01-15T17:58:55.802627+0000","last_became_active":"2024-01-15T17:58:54.815985+0000","last_became_peered":"2024-01-15T17:58:54.815985+0000","last_unstale":"2024-01-15T17:58:55.802627+0000","last_undegraded":"2024-01-15T17:58:55.802627+0000","last_fullsized":"2024-01-15T17:58:55.802627+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:14:04.722030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.811853+0000","last_change":"2024-01-15T17:58:53.780792+0000","last_active":"2024-01-15T17:58:54.811853+0000","last_peered":"2024-01-15T17:58:54.811853+0000","last_clean":"2024-01-15T17:58:54.811853+0000","last_became_active":"2024-01-15T17:58:53.780523+0000","last_became_peered":"2024-01-15T17:58:53.780523+0000","last_unstale":"2024-01-15T17:58:54.811853+0000","last_undegraded":"2024-01-15T17:58:54.811853+0000","last_fullsized":"2024-01-15T17:58:54.811853+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:21.869370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.804258+0000","last_change":"2024-01-15T17:58:54.798424+0000","last_active":"2024-01-15T17:58:55.804258+0000","last_peered":"2024-01-15T17:58:55.804258+0000","last_clean":"2024-01-15T17:58:55.804258+0000","last_became_active":"2024-01-15T17:58:54.798046+0000","last_became_peered":"2024-01-15T17:58:54.798046+0000","last_unstale":"2024-01-15T17:58:55.804258+0000","last_undegraded":"2024-01-15T17:58:55.804258+0000","last_fullsized":"2024-01-15T17:58:55.804258+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:08:02.249232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.812807+0000","last_change":"2024-01-15T17:58:53.778730+0000","last_active":"2024-01-15T17:58:54.812807+0000","last_peered":"2024-01-15T17:58:54.812807+0000","last_clean":"2024-01-15T17:58:54.812807+0000","last_became_active":"2024-01-15T17:58:53.777103+0000","last_became_peered":"2024-01-15T17:58:53.777103+0000","last_unstale":"2024-01-15T17:58:54.812807+0000","last_undegraded":"2024-01-15T17:58:54.812807+0000","last_fullsized":"2024-01-15T17:58:54.812807+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:32:03.913240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027235+0000","last_change":"2024-01-15T17:58:54.809361+0000","last_active":"2024-01-15T17:58:56.027235+0000","last_peered":"2024-01-15T17:58:56.027235+0000","last_clean":"2024-01-15T17:58:56.027235+0000","last_became_active":"2024-01-15T17:58:54.808794+0000","last_became_peered":"2024-01-15T17:58:54.808794+0000","last_unstale":"2024-01-15T17:58:56.027235+0000","last_undegraded":"2024-01-15T17:58:56.027235+0000","last_fullsized":"2024-01-15T17:58:56.027235+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:12:53.884858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.780840+0000","last_change":"2024-01-15T17:58:53.769315+0000","last_active":"2024-01-15T17:58:54.780840+0000","last_peered":"2024-01-15T17:58:54.780840+0000","last_clean":"2024-01-15T17:58:54.780840+0000","last_became_active":"2024-01-15T17:58:53.769028+0000","last_became_peered":"2024-01-15T17:58:53.769028+0000","last_unstale":"2024-01-15T17:58:54.780840+0000","last_undegraded":"2024-01-15T17:58:54.780840+0000","last_fullsized":"2024-01-15T17:58:54.780840+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:32.342923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.961136+0000","last_change":"2024-01-15T17:58:54.805397+0000","last_active":"2024-01-15T17:58:55.961136+0000","last_peered":"2024-01-15T17:58:55.961136+0000","last_clean":"2024-01-15T17:58:55.961136+0000","last_became_active":"2024-01-15T17:58:54.805199+0000","last_became_peered":"2024-01-15T17:58:54.805199+0000","last_unstale":"2024-01-15T17:58:55.961136+0000","last_undegraded":"2024-01-15T17:58:55.961136+0000","last_fullsized":"2024-01-15T17:58:55.961136+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:16:45.310193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.776563+0000","last_change":"2024-01-15T17:58:53.778012+0000","last_active":"2024-01-15T17:58:54.776563+0000","last_peered":"2024-01-15T17:58:54.776563+0000","last_clean":"2024-01-15T17:58:54.776563+0000","last_became_active":"2024-01-15T17:58:53.776887+0000","last_became_peered":"2024-01-15T17:58:53.776887+0000","last_unstale":"2024-01-15T17:58:54.776563+0000","last_undegraded":"2024-01-15T17:58:54.776563+0000","last_fullsized":"2024-01-15T17:58:54.776563+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:11.691647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.805445+0000","last_change":"2024-01-15T17:58:54.799708+0000","last_active":"2024-01-15T17:58:55.805445+0000","last_peered":"2024-01-15T17:58:55.805445+0000","last_clean":"2024-01-15T17:58:55.805445+0000","last_became_active":"2024-01-15T17:58:54.798984+0000","last_became_peered":"2024-01-15T17:58:54.798984+0000","last_unstale":"2024-01-15T17:58:55.805445+0000","last_undegraded":"2024-01-15T17:58:55.805445+0000","last_fullsized":"2024-01-15T17:58:55.805445+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:44.162291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.780551+0000","last_change":"2024-01-15T17:58:53.786507+0000","last_active":"2024-01-15T17:58:54.780551+0000","last_peered":"2024-01-15T17:58:54.780551+0000","last_clean":"2024-01-15T17:58:54.780551+0000","last_became_active":"2024-01-15T17:58:53.786154+0000","last_became_peered":"2024-01-15T17:58:53.786154+0000","last_unstale":"2024-01-15T17:58:54.780551+0000","last_undegraded":"2024-01-15T17:58:54.780551+0000","last_fullsized":"2024-01-15T17:58:54.780551+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:46.674180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.799929+0000","last_change":"2024-01-15T17:58:54.802830+0000","last_active":"2024-01-15T17:58:55.799929+0000","last_peered":"2024-01-15T17:58:55.799929+0000","last_clean":"2024-01-15T17:58:55.799929+0000","last_became_active":"2024-01-15T17:58:54.802572+0000","last_became_peered":"2024-01-15T17:58:54.802572+0000","last_unstale":"2024-01-15T17:58:55.799929+0000","last_undegraded":"2024-01-15T17:58:55.799929+0000","last_fullsized":"2024-01-15T17:58:55.799929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:31:14.000701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.775154+0000","last_change":"2024-01-15T17:58:53.779129+0000","last_active":"2024-01-15T17:58:54.775154+0000","last_peered":"2024-01-15T17:58:54.775154+0000","last_clean":"2024-01-15T17:58:54.775154+0000","last_became_active":"2024-01-15T17:58:53.778863+0000","last_became_peered":"2024-01-15T17:58:53.778863+0000","last_unstale":"2024-01-15T17:58:54.775154+0000","last_undegraded":"2024-01-15T17:58:54.775154+0000","last_fullsized":"2024-01-15T17:58:54.775154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.931485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.799670+0000","last_change":"2024-01-15T17:58:54.813160+0000","last_active":"2024-01-15T17:58:55.799670+0000","last_peered":"2024-01-15T17:58:55.799670+0000","last_clean":"2024-01-15T17:58:55.799670+0000","last_became_active":"2024-01-15T17:58:54.812614+0000","last_became_peered":"2024-01-15T17:58:54.812614+0000","last_unstale":"2024-01-15T17:58:55.799670+0000","last_undegraded":"2024-01-15T17:58:55.799670+0000","last_fullsized":"2024-01-15T17:58:55.799670+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:45.071743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.819899+0000","last_change":"2024-01-15T17:58:53.782469+0000","last_active":"2024-01-15T17:58:54.819899+0000","last_peered":"2024-01-15T17:58:54.819899+0000","last_clean":"2024-01-15T17:58:54.819899+0000","last_became_active":"2024-01-15T17:58:53.782219+0000","last_became_peered":"2024-01-15T17:58:53.782219+0000","last_unstale":"2024-01-15T17:58:54.819899+0000","last_undegraded":"2024-01-15T17:58:54.819899+0000","last_fullsized":"2024-01-15T17:58:54.819899+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:44.489457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.804647+0000","last_change":"2024-01-15T17:58:54.802292+0000","last_active":"2024-01-15T17:58:55.804647+0000","last_peered":"2024-01-15T17:58:55.804647+0000","last_clean":"2024-01-15T17:58:55.804647+0000","last_became_active":"2024-01-15T17:58:54.801993+0000","last_became_peered":"2024-01-15T17:58:54.801993+0000","last_unstale":"2024-01-15T17:58:55.804647+0000","last_undegraded":"2024-01-15T17:58:55.804647+0000","last_fullsized":"2024-01-15T17:58:55.804647+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:31.587532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":18,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.821293+0000","last_change":"2024-01-15T17:58:53.778698+0000","last_active":"2024-01-15T17:58:54.821293+0000","last_peered":"2024-01-15T17:58:54.821293+0000","last_clean":"2024-01-15T17:58:54.821293+0000","last_became_active":"2024-01-15T17:58:53.778416+0000","last_became_peered":"2024-01-15T17:58:53.778416+0000","last_unstale":"2024-01-15T17:58:54.821293+0000","last_undegraded":"2024-01-15T17:58:54.821293+0000","last_fullsized":"2024-01-15T17:58:54.821293+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:10:15.901079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.961525+0000","last_change":"2024-01-15T17:58:54.809255+0000","last_active":"2024-01-15T17:58:55.961525+0000","last_peered":"2024-01-15T17:58:55.961525+0000","last_clean":"2024-01-15T17:58:55.961525+0000","last_became_active":"2024-01-15T17:58:54.808988+0000","last_became_peered":"2024-01-15T17:58:54.808988+0000","last_unstale":"2024-01-15T17:58:55.961525+0000","last_undegraded":"2024-01-15T17:58:55.961525+0000","last_fullsized":"2024-01-15T17:58:55.961525+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:52.594611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.811674+0000","last_change":"2024-01-15T17:58:53.780478+0000","last_active":"2024-01-15T17:58:54.811674+0000","last_peered":"2024-01-15T17:58:54.811674+0000","last_clean":"2024-01-15T17:58:54.811674+0000","last_became_active":"2024-01-15T17:58:53.777564+0000","last_became_peered":"2024-01-15T17:58:53.777564+0000","last_unstale":"2024-01-15T17:58:54.811674+0000","last_undegraded":"2024-01-15T17:58:54.811674+0000","last_fullsized":"2024-01-15T17:58:54.811674+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:02:14.461512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.799383+0000","last_change":"2024-01-15T17:58:54.797690+0000","last_active":"2024-01-15T17:58:55.799383+0000","last_peered":"2024-01-15T17:58:55.799383+0000","last_clean":"2024-01-15T17:58:55.799383+0000","last_became_active":"2024-01-15T17:58:54.797349+0000","last_became_peered":"2024-01-15T17:58:54.797349+0000","last_unstale":"2024-01-15T17:58:55.799383+0000","last_undegraded":"2024-01-15T17:58:55.799383+0000","last_fullsized":"2024-01-15T17:58:55.799383+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:44.162291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.791305+0000","last_change":"2024-01-15T17:58:53.777976+0000","last_active":"2024-01-15T17:58:54.791305+0000","last_peered":"2024-01-15T17:58:54.791305+0000","last_clean":"2024-01-15T17:58:54.791305+0000","last_became_active":"2024-01-15T17:58:53.777658+0000","last_became_peered":"2024-01-15T17:58:53.777658+0000","last_unstale":"2024-01-15T17:58:54.791305+0000","last_undegraded":"2024-01-15T17:58:54.791305+0000","last_fullsized":"2024-01-15T17:58:54.791305+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:57.822622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.802708+0000","last_change":"2024-01-15T17:58:54.816039+0000","last_active":"2024-01-15T17:58:55.802708+0000","last_peered":"2024-01-15T17:58:55.802708+0000","last_clean":"2024-01-15T17:58:55.802708+0000","last_became_active":"2024-01-15T17:58:54.815054+0000","last_became_peered":"2024-01-15T17:58:54.815054+0000","last_unstale":"2024-01-15T17:58:55.802708+0000","last_undegraded":"2024-01-15T17:58:55.802708+0000","last_fullsized":"2024-01-15T17:58:55.802708+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:20:31.587532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.780073+0000","last_change":"2024-01-15T17:58:53.782063+0000","last_active":"2024-01-15T17:58:54.780073+0000","last_peered":"2024-01-15T17:58:54.780073+0000","last_clean":"2024-01-15T17:58:54.780073+0000","last_became_active":"2024-01-15T17:58:53.781761+0000","last_became_peered":"2024-01-15T17:58:53.781761+0000","last_unstale":"2024-01-15T17:58:54.780073+0000","last_undegraded":"2024-01-15T17:58:54.780073+0000","last_fullsized":"2024-01-15T17:58:54.780073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:34:59.133052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.028225+0000","last_change":"2024-01-15T17:58:54.817390+0000","last_active":"2024-01-15T17:58:56.028225+0000","last_peered":"2024-01-15T17:58:56.028225+0000","last_clean":"2024-01-15T17:58:56.028225+0000","last_became_active":"2024-01-15T17:58:54.816469+0000","last_became_peered":"2024-01-15T17:58:54.816469+0000","last_unstale":"2024-01-15T17:58:56.028225+0000","last_undegraded":"2024-01-15T17:58:56.028225+0000","last_fullsized":"2024-01-15T17:58:56.028225+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:06.670278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":20,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.819318+0000","last_change":"2024-01-15T17:58:53.775656+0000","last_active":"2024-01-15T17:58:54.819318+0000","last_peered":"2024-01-15T17:58:54.819318+0000","last_clean":"2024-01-15T17:58:54.819318+0000","last_became_active":"2024-01-15T17:58:53.775310+0000","last_became_peered":"2024-01-15T17:58:53.775310+0000","last_unstale":"2024-01-15T17:58:54.819318+0000","last_undegraded":"2024-01-15T17:58:54.819318+0000","last_fullsized":"2024-01-15T17:58:54.819318+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:35.431055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.028910+0000","last_change":"2024-01-15T17:58:54.801748+0000","last_active":"2024-01-15T17:58:56.028910+0000","last_peered":"2024-01-15T17:58:56.028910+0000","last_clean":"2024-01-15T17:58:56.028910+0000","last_became_active":"2024-01-15T17:58:54.801434+0000","last_became_peered":"2024-01-15T17:58:54.801434+0000","last_unstale":"2024-01-15T17:58:56.028910+0000","last_undegraded":"2024-01-15T17:58:56.028910+0000","last_fullsized":"2024-01-15T17:58:56.028910+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:50.904464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.810817+0000","last_change":"2024-01-15T17:58:53.779965+0000","last_active":"2024-01-15T17:58:54.810817+0000","last_peered":"2024-01-15T17:58:54.810817+0000","last_clean":"2024-01-15T17:58:54.810817+0000","last_became_active":"2024-01-15T17:58:53.779597+0000","last_became_peered":"2024-01-15T17:58:53.779597+0000","last_unstale":"2024-01-15T17:58:54.810817+0000","last_undegraded":"2024-01-15T17:58:54.810817+0000","last_fullsized":"2024-01-15T17:58:54.810817+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:06:50.158507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.806162+0000","last_change":"2024-01-15T17:58:54.805945+0000","last_active":"2024-01-15T17:58:55.806162+0000","last_peered":"2024-01-15T17:58:55.806162+0000","last_clean":"2024-01-15T17:58:55.806162+0000","last_became_active":"2024-01-15T17:58:54.805653+0000","last_became_peered":"2024-01-15T17:58:54.805653+0000","last_unstale":"2024-01-15T17:58:55.806162+0000","last_undegraded":"2024-01-15T17:58:55.806162+0000","last_fullsized":"2024-01-15T17:58:55.806162+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:45.071743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.811115+0000","last_change":"2024-01-15T17:58:53.774956+0000","last_active":"2024-01-15T17:58:54.811115+0000","last_peered":"2024-01-15T17:58:54.811115+0000","last_clean":"2024-01-15T17:58:54.811115+0000","last_became_active":"2024-01-15T17:58:53.774680+0000","last_became_peered":"2024-01-15T17:58:53.774680+0000","last_unstale":"2024-01-15T17:58:54.811115+0000","last_undegraded":"2024-01-15T17:58:54.811115+0000","last_fullsized":"2024-01-15T17:58:54.811115+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:40.251201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.961984+0000","last_change":"2024-01-15T17:58:54.803748+0000","last_active":"2024-01-15T17:58:55.961984+0000","last_peered":"2024-01-15T17:58:55.961984+0000","last_clean":"2024-01-15T17:58:55.961984+0000","last_became_active":"2024-01-15T17:58:54.803488+0000","last_became_peered":"2024-01-15T17:58:54.803488+0000","last_unstale":"2024-01-15T17:58:55.961984+0000","last_undegraded":"2024-01-15T17:58:55.961984+0000","last_fullsized":"2024-01-15T17:58:55.961984+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:35:00.145297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.811471+0000","last_change":"2024-01-15T17:58:53.772674+0000","last_active":"2024-01-15T17:58:54.811471+0000","last_peered":"2024-01-15T17:58:54.811471+0000","last_clean":"2024-01-15T17:58:54.811471+0000","last_became_active":"2024-01-15T17:58:53.769008+0000","last_became_peered":"2024-01-15T17:58:53.769008+0000","last_unstale":"2024-01-15T17:58:54.811471+0000","last_undegraded":"2024-01-15T17:58:54.811471+0000","last_fullsized":"2024-01-15T17:58:54.811471+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:37.435123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027313+0000","last_change":"2024-01-15T17:58:54.807766+0000","last_active":"2024-01-15T17:58:56.027313+0000","last_peered":"2024-01-15T17:58:56.027313+0000","last_clean":"2024-01-15T17:58:56.027313+0000","last_became_active":"2024-01-15T17:58:54.807259+0000","last_became_peered":"2024-01-15T17:58:54.807259+0000","last_unstale":"2024-01-15T17:58:56.027313+0000","last_undegraded":"2024-01-15T17:58:56.027313+0000","last_fullsized":"2024-01-15T17:58:56.027313+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:02:15.473757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.779152+0000","last_change":"2024-01-15T17:58:53.779480+0000","last_active":"2024-01-15T17:58:54.779152+0000","last_peered":"2024-01-15T17:58:54.779152+0000","last_clean":"2024-01-15T17:58:54.779152+0000","last_became_active":"2024-01-15T17:58:53.778673+0000","last_became_peered":"2024-01-15T17:58:53.778673+0000","last_unstale":"2024-01-15T17:58:54.779152+0000","last_undegraded":"2024-01-15T17:58:54.779152+0000","last_fullsized":"2024-01-15T17:58:54.779152+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:37:54.751284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.026826+0000","last_change":"2024-01-15T17:58:54.811132+0000","last_active":"2024-01-15T17:58:56.026826+0000","last_peered":"2024-01-15T17:58:56.026826+0000","last_clean":"2024-01-15T17:58:56.026826+0000","last_became_active":"2024-01-15T17:58:54.804961+0000","last_became_peered":"2024-01-15T17:58:54.804961+0000","last_unstale":"2024-01-15T17:58:56.026826+0000","last_undegraded":"2024-01-15T17:58:56.026826+0000","last_fullsized":"2024-01-15T17:58:56.026826+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:32:04.925485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.812160+0000","last_change":"2024-01-15T17:58:53.780996+0000","last_active":"2024-01-15T17:58:54.812160+0000","last_peered":"2024-01-15T17:58:54.812160+0000","last_clean":"2024-01-15T17:58:54.812160+0000","last_became_active":"2024-01-15T17:58:53.778816+0000","last_became_peered":"2024-01-15T17:58:53.778816+0000","last_unstale":"2024-01-15T17:58:54.812160+0000","last_undegraded":"2024-01-15T17:58:54.812160+0000","last_fullsized":"2024-01-15T17:58:54.812160+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:44:19.764727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.026940+0000","last_change":"2024-01-15T17:58:54.797670+0000","last_active":"2024-01-15T17:58:56.026940+0000","last_peered":"2024-01-15T17:58:56.026940+0000","last_clean":"2024-01-15T17:58:56.026940+0000","last_became_active":"2024-01-15T17:58:54.796944+0000","last_became_peered":"2024-01-15T17:58:54.796944+0000","last_unstale":"2024-01-15T17:58:56.026940+0000","last_undegraded":"2024-01-15T17:58:56.026940+0000","last_fullsized":"2024-01-15T17:58:56.026940+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:22.881615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.804527+0000","last_change":"2024-01-15T17:58:53.784613+0000","last_active":"2024-01-15T17:58:54.804527+0000","last_peered":"2024-01-15T17:58:54.804527+0000","last_clean":"2024-01-15T17:58:54.804527+0000","last_became_active":"2024-01-15T17:58:53.784295+0000","last_became_peered":"2024-01-15T17:58:53.784295+0000","last_unstale":"2024-01-15T17:58:54.804527+0000","last_undegraded":"2024-01-15T17:58:54.804527+0000","last_fullsized":"2024-01-15T17:58:54.804527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:16:51.863117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.026575+0000","last_change":"2024-01-15T17:58:54.788650+0000","last_active":"2024-01-15T17:58:56.026575+0000","last_peered":"2024-01-15T17:58:56.026575+0000","last_clean":"2024-01-15T17:58:56.026575+0000","last_became_active":"2024-01-15T17:58:54.788371+0000","last_became_peered":"2024-01-15T17:58:54.788371+0000","last_unstale":"2024-01-15T17:58:56.026575+0000","last_undegraded":"2024-01-15T17:58:56.026575+0000","last_fullsized":"2024-01-15T17:58:56.026575+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:24:07.230989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.779544+0000","last_change":"2024-01-15T17:58:53.785047+0000","last_active":"2024-01-15T17:58:54.779544+0000","last_peered":"2024-01-15T17:58:54.779544+0000","last_clean":"2024-01-15T17:58:54.779544+0000","last_became_active":"2024-01-15T17:58:53.784573+0000","last_became_peered":"2024-01-15T17:58:53.784573+0000","last_unstale":"2024-01-15T17:58:54.779544+0000","last_undegraded":"2024-01-15T17:58:54.779544+0000","last_fullsized":"2024-01-15T17:58:54.779544+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:48:06.420797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.803115+0000","last_change":"2024-01-15T17:58:54.807918+0000","last_active":"2024-01-15T17:58:55.803115+0000","last_peered":"2024-01-15T17:58:55.803115+0000","last_clean":"2024-01-15T17:58:55.803115+0000","last_became_active":"2024-01-15T17:58:54.807313+0000","last_became_peered":"2024-01-15T17:58:54.807313+0000","last_unstale":"2024-01-15T17:58:55.803115+0000","last_undegraded":"2024-01-15T17:58:55.803115+0000","last_fullsized":"2024-01-15T17:58:55.803115+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:45:46.963255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.783086+0000","last_change":"2024-01-15T17:58:53.782707+0000","last_active":"2024-01-15T17:58:54.783086+0000","last_peered":"2024-01-15T17:58:54.783086+0000","last_clean":"2024-01-15T17:58:54.783086+0000","last_became_active":"2024-01-15T17:58:53.782425+0000","last_became_peered":"2024-01-15T17:58:53.782425+0000","last_unstale":"2024-01-15T17:58:54.783086+0000","last_undegraded":"2024-01-15T17:58:54.783086+0000","last_fullsized":"2024-01-15T17:58:54.783086+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:38.360943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027415+0000","last_change":"2024-01-15T17:58:54.816313+0000","last_active":"2024-01-15T17:58:56.027415+0000","last_peered":"2024-01-15T17:58:56.027415+0000","last_clean":"2024-01-15T17:58:56.027415+0000","last_became_active":"2024-01-15T17:58:54.813992+0000","last_became_peered":"2024-01-15T17:58:54.813992+0000","last_unstale":"2024-01-15T17:58:56.027415+0000","last_undegraded":"2024-01-15T17:58:56.027415+0000","last_fullsized":"2024-01-15T17:58:56.027415+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:18:31.436515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.816069+0000","last_change":"2024-01-15T17:58:53.776979+0000","last_active":"2024-01-15T17:58:54.816069+0000","last_peered":"2024-01-15T17:58:54.816069+0000","last_clean":"2024-01-15T17:58:54.816069+0000","last_became_active":"2024-01-15T17:58:53.776609+0000","last_became_peered":"2024-01-15T17:58:53.776609+0000","last_unstale":"2024-01-15T17:58:54.816069+0000","last_undegraded":"2024-01-15T17:58:54.816069+0000","last_fullsized":"2024-01-15T17:58:54.816069+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:23.202096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.806340+0000","last_change":"2024-01-15T17:58:54.800389+0000","last_active":"2024-01-15T17:58:55.806340+0000","last_peered":"2024-01-15T17:58:55.806340+0000","last_clean":"2024-01-15T17:58:55.806340+0000","last_became_active":"2024-01-15T17:58:54.800108+0000","last_became_peered":"2024-01-15T17:58:54.800108+0000","last_unstale":"2024-01-15T17:58:55.806340+0000","last_undegraded":"2024-01-15T17:58:55.806340+0000","last_fullsized":"2024-01-15T17:58:55.806340+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:31:14.000701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"35'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.956257+0000","last_change":"2024-01-15T17:58:53.777358+0000","last_active":"2024-01-15T17:58:55.956257+0000","last_peered":"2024-01-15T17:58:55.956257+0000","last_clean":"2024-01-15T17:58:55.956257+0000","last_became_active":"2024-01-15T17:58:53.777077+0000","last_became_peered":"2024-01-15T17:58:53.777077+0000","last_unstale":"2024-01-15T17:58:55.956257+0000","last_undegraded":"2024-01-15T17:58:55.956257+0000","last_fullsized":"2024-01-15T17:58:55.956257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:42:57.822622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.955949+0000","last_change":"2024-01-15T17:58:54.796889+0000","last_active":"2024-01-15T17:58:55.955949+0000","last_peered":"2024-01-15T17:58:55.955949+0000","last_clean":"2024-01-15T17:58:55.955949+0000","last_became_active":"2024-01-15T17:58:54.795325+0000","last_became_peered":"2024-01-15T17:58:54.795325+0000","last_unstale":"2024-01-15T17:58:55.955949+0000","last_undegraded":"2024-01-15T17:58:55.955949+0000","last_fullsized":"2024-01-15T17:58:55.955949+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:06:51.170752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.791660+0000","last_change":"2024-01-15T17:58:53.775372+0000","last_active":"2024-01-15T17:58:54.791660+0000","last_peered":"2024-01-15T17:58:54.791660+0000","last_clean":"2024-01-15T17:58:54.791660+0000","last_became_active":"2024-01-15T17:58:53.775056+0000","last_became_peered":"2024-01-15T17:58:53.775056+0000","last_unstale":"2024-01-15T17:58:54.791660+0000","last_undegraded":"2024-01-15T17:58:54.791660+0000","last_fullsized":"2024-01-15T17:58:54.791660+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:23.480750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.029769+0000","last_change":"2024-01-15T17:58:54.811369+0000","last_active":"2024-01-15T17:58:56.029769+0000","last_peered":"2024-01-15T17:58:56.029769+0000","last_clean":"2024-01-15T17:58:56.029769+0000","last_became_active":"2024-01-15T17:58:54.811043+0000","last_became_peered":"2024-01-15T17:58:54.811043+0000","last_unstale":"2024-01-15T17:58:56.029769+0000","last_undegraded":"2024-01-15T17:58:56.029769+0000","last_fullsized":"2024-01-15T17:58:56.029769+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:36:24.214341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.773490+0000","last_change":"2024-01-15T17:58:53.770121+0000","last_active":"2024-01-15T17:58:54.773490+0000","last_peered":"2024-01-15T17:58:54.773490+0000","last_clean":"2024-01-15T17:58:54.773490+0000","last_became_active":"2024-01-15T17:58:53.769805+0000","last_became_peered":"2024-01-15T17:58:53.769805+0000","last_unstale":"2024-01-15T17:58:54.773490+0000","last_undegraded":"2024-01-15T17:58:54.773490+0000","last_fullsized":"2024-01-15T17:58:54.773490+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:40:50.242725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.955603+0000","last_change":"2024-01-15T17:58:54.809106+0000","last_active":"2024-01-15T17:58:55.955603+0000","last_peered":"2024-01-15T17:58:55.955603+0000","last_clean":"2024-01-15T17:58:55.955603+0000","last_became_active":"2024-01-15T17:58:54.807292+0000","last_became_peered":"2024-01-15T17:58:54.807292+0000","last_unstale":"2024-01-15T17:58:55.955603+0000","last_undegraded":"2024-01-15T17:58:55.955603+0000","last_fullsized":"2024-01-15T17:58:55.955603+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:09:24.492996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.791333+0000","last_change":"2024-01-15T17:58:53.776755+0000","last_active":"2024-01-15T17:58:54.791333+0000","last_peered":"2024-01-15T17:58:54.791333+0000","last_clean":"2024-01-15T17:58:54.791333+0000","last_became_active":"2024-01-15T17:58:53.776423+0000","last_became_peered":"2024-01-15T17:58:53.776423+0000","last_unstale":"2024-01-15T17:58:54.791333+0000","last_undegraded":"2024-01-15T17:58:54.791333+0000","last_fullsized":"2024-01-15T17:58:54.791333+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:46:05.658033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.798531+0000","last_change":"2024-01-15T17:58:54.791515+0000","last_active":"2024-01-15T17:58:55.798531+0000","last_peered":"2024-01-15T17:58:55.798531+0000","last_clean":"2024-01-15T17:58:55.798531+0000","last_became_active":"2024-01-15T17:58:54.791212+0000","last_became_peered":"2024-01-15T17:58:54.791212+0000","last_unstale":"2024-01-15T17:58:55.798531+0000","last_undegraded":"2024-01-15T17:58:55.798531+0000","last_fullsized":"2024-01-15T17:58:55.798531+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:12:58.006337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.780717+0000","last_change":"2024-01-15T17:58:53.770125+0000","last_active":"2024-01-15T17:58:54.780717+0000","last_peered":"2024-01-15T17:58:54.780717+0000","last_clean":"2024-01-15T17:58:54.780717+0000","last_became_active":"2024-01-15T17:58:53.769709+0000","last_became_peered":"2024-01-15T17:58:53.769709+0000","last_unstale":"2024-01-15T17:58:54.780717+0000","last_undegraded":"2024-01-15T17:58:54.780717+0000","last_fullsized":"2024-01-15T17:58:54.780717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:00.272884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:56.027936+0000","last_change":"2024-01-15T17:58:54.794275+0000","last_active":"2024-01-15T17:58:56.027936+0000","last_peered":"2024-01-15T17:58:56.027936+0000","last_clean":"2024-01-15T17:58:56.027936+0000","last_became_active":"2024-01-15T17:58:54.793845+0000","last_became_peered":"2024-01-15T17:58:54.793845+0000","last_unstale":"2024-01-15T17:58:56.027936+0000","last_undegraded":"2024-01-15T17:58:56.027936+0000","last_fullsized":"2024-01-15T17:58:56.027936+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:25.845223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.778413+0000","last_change":"2024-01-15T17:58:53.786548+0000","last_active":"2024-01-15T17:58:54.778413+0000","last_peered":"2024-01-15T17:58:54.778413+0000","last_clean":"2024-01-15T17:58:54.778413+0000","last_became_active":"2024-01-15T17:58:53.786220+0000","last_became_peered":"2024-01-15T17:58:53.786220+0000","last_unstale":"2024-01-15T17:58:54.778413+0000","last_undegraded":"2024-01-15T17:58:54.778413+0000","last_fullsized":"2024-01-15T17:58:54.778413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:02:38.360943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-15T17:58:55.803554+0000","last_change":"2024-01-15T17:58:54.808170+0000","last_active":"2024-01-15T17:58:55.803554+0000","last_peered":"2024-01-15T17:58:55.803554+0000","last_clean":"2024-01-15T17:58:55.803554+0000","last_became_active":"2024-01-15T17:58:54.807755+0000","last_became_peered":"2024-01-15T17:58:54.807755+0000","last_unstale":"2024-01-15T17:58:55.803554+0000","last_undegraded":"2024-01-15T17:58:55.803554+0000","last_fullsized":"2024-01-15T17:58:55.803554+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:53.739527+0000","last_clean_scrub_stamp":"2024-01-15T17:58:53.739527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:45.071743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.806467+0000","last_change":"2024-01-15T17:58:53.783339+0000","last_active":"2024-01-15T17:58:54.806467+0000","last_peered":"2024-01-15T17:58:54.806467+0000","last_clean":"2024-01-15T17:58:54.806467+0000","last_became_active":"2024-01-15T17:58:53.783028+0000","last_became_peered":"2024-01-15T17:58:53.783028+0000","last_unstale":"2024-01-15T17:58:54.806467+0000","last_undegraded":"2024-01-15T17:58:54.806467+0000","last_fullsized":"2024-01-15T17:58:54.806467+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:58:52.727281+0000","last_clean_scrub_stamp":"2024-01-15T17:58:52.727281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:19.366878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":175,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-15T17:58:54.796567+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T17:58:54.796567+0000","last_peered":"2024-01-15T17:58:54.796567+0000","last_clean":"2024-01-15T17:58:54.796567+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T17:58:54.796567+0000","last_undegraded":"2024-01-15T17:58:54.796567+0000","last_fullsized":"2024-01-15T17:58:54.796567+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":991232,"data_stored":744942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.093,"15min":1.093},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.4379999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.6970000000000001,"5min":7.6970000000000001,"15min":7.6970000000000001},"last":1.0900000000000001}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":3.4140000000000001,"5min":3.4140000000000001,"15min":3.4140000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.5640000000000001},"last":1.167}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.6299999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":7.5940000000000003,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":0.95099999999999996}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":4.8739999999999997,"5min":4.8739999999999997,"15min":4.8739999999999997},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.2929999999999999}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":7.4809999999999999,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.2619999999999996,"5min":4.2619999999999996,"15min":4.2619999999999996},"last":1.4570000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.5389999999999999,"15min":1.5389999999999999},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":6.0049999999999999,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":7.798,"5min":7.798,"15min":7.798},"last":1.2390000000000001}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.7769999999999999,"15min":1.7769999999999999},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":7.968,"5min":7.968,"15min":7.968},"last":1.405},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4710000000000001,"15min":1.4710000000000001},"min":{"1min":0.95099999999999996,"5min":0.95099999999999996,"15min":0.95099999999999996},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":1.325}]}]},{"osd":6,"up_from":14,"seq":60129542165,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.73,"5min":1.73,"15min":1.73},"last":1.46},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.7,"5min":1.7,"15min":1.7},"last":1.032}]},{"osd":1,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.22,"15min":1.22},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.554},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":2.4159999999999999,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.91600000000000004}]},{"osd":2,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":3.415,"5min":3.415,"15min":3.415},"last":1.091},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.7529999999999999,"5min":1.7529999999999999,"15min":1.7529999999999999},"last":1.276}]},{"osd":3,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.244},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":1.504}]},{"osd":4,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":2.8450000000000002,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.621},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.1459999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.3490000000000002,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.6990000000000001}]},{"osd":7,"last update":"Mon Jan 15 17:58:27 2024","interfaces":[{"interface":"back","average":{"1min":1.708,"5min":1.708,"15min":1.708},"min":{"1min":1.073,"5min":1.073,"15min":1.073},"max":{"1min":5.6449999999999996,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.423},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.6000000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542167,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":155156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.123,"15min":1.123},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.53,"5min":1.53,"15min":1.53},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.728,"5min":1.728,"15min":1.728},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3540000000000001,"15min":1.3540000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.59099999999999997}]},{"osd":3,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.242,"15min":1.242},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.0659999999999998,"5min":2.0659999999999998,"15min":2.0659999999999998},"last":0.622},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":0.64500000000000002}]},{"osd":4,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.29,"15min":1.29},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":0.498}]},{"osd":7,"last update":"Mon Jan 15 17:58:28 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.266,"5min":2.266,"15min":2.266},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.1920000000000002,"5min":2.1920000000000002,"15min":2.1920000000000002},"last":0.69799999999999995}]}]},{"osd":4,"up_from":14,"seq":60129542166,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27420,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":401408,"data_stored":154588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.5700000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.101,"15min":1.101},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.696,"5min":1.696,"15min":1.696},"last":1.113}]},{"osd":1,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.151,"15min":1.151},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.379},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.21}]},{"osd":2,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"last":1.3540000000000001}]},{"osd":3,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.321,"15min":1.321},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.996,"5min":1.996,"15min":1.996},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3140000000000001,"15min":1.3140000000000001},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.0430000000000001,"5min":2.0430000000000001,"15min":2.0430000000000001},"last":1.073}]},{"osd":5,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":1.47},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.5169999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"min":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"max":{"1min":1.98,"5min":1.98,"15min":1.98},"last":1.417}]},{"osd":7,"last update":"Mon Jan 15 17:58:25 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.528,"15min":1.528},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.9750000000000001,"5min":1.9750000000000001,"15min":1.9750000000000001},"last":1.248}]}]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":156118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":1.325},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":1.03}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.996,"15min":0.996},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.768,"5min":1.768,"15min":1.768},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":2.6890000000000001,"5min":2.6890000000000001,"15min":2.6890000000000001},"last":1.732}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.1099999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":3.1219999999999999,"5min":3.1219999999999999,"15min":3.1219999999999999},"last":1.6100000000000001}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":2.0409999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.7130000000000001}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1789999999999998,"5min":2.1789999999999998,"15min":2.1789999999999998},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.3090000000000002,"5min":3.3090000000000002,"15min":3.3090000000000002},"last":1.827}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.274},{"interface":"front","average":{"1min":1.502,"5min":1.502,"15min":1.502},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.653}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":3.4359999999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.528}]}]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31576,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340264,"statfs":{"total":96636764160,"available":96604430336,"internally_reserved":0,"allocated":397312,"data_stored":154554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":1.0940000000000001}]},{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":2.5150000000000001,"5min":2.5150000000000001,"15min":2.5150000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.198,"5min":2.198,"15min":2.198},"last":0.84899999999999998}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"last":1.278}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.579,"15min":1.579},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":3.7029999999999998,"5min":3.7029999999999998,"15min":3.7029999999999998},"last":1.446},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.163}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.5129999999999999,"5min":2.5129999999999999,"15min":2.5129999999999999},"last":1.194},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"max":{"1min":2.4700000000000002,"5min":2.4700000000000002,"15min":2.4700000000000002},"last":1.6619999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.675,"15min":1.675},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":2.8410000000000002,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":2.7559999999999998,"5min":2.7559999999999998,"15min":2.7559999999999998},"last":1.25}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":1.6830000000000001,"15min":1.6830000000000001},"min":{"1min":1.149,"5min":1.149,"15min":1.149},"max":{"1min":2.613,"5min":2.613,"15min":2.613},"last":1.333},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.369}]}]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":409600,"data_stored":156118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":1.9390000000000001},"last":0.753}]},{"osd":2,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.073,"5min":1.073,"15min":1.073},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":2.5470000000000002,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.81799999999999995}]},{"osd":3,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.3320000000000001}]},{"osd":4,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.9670000000000001,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":0.872},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.4870000000000001},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.677}]},{"osd":5,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.5409999999999999,"15min":1.5409999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.6760000000000002,"5min":2.6760000000000002,"15min":2.6760000000000002},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.1589999999999998,"5min":3.1589999999999998,"15min":3.1589999999999998},"last":1.1839999999999999}]},{"osd":6,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.5920000000000001,"15min":1.5920000000000001},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.7970000000000002,"5min":2.7970000000000002,"15min":2.7970000000000002},"last":1.748}]},{"osd":7,"last update":"Mon Jan 15 17:58:26 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.486,"15min":1.486},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":2.8929999999999998,"5min":2.8929999999999998,"15min":2.8929999999999998},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.619}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":987136,"data_stored":744852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.6659999999999999,"5min":1.6659999999999999,"15min":1.6659999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.0329999999999999}]},{"osd":2,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.874},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":1.7809999999999999},"last":0.97699999999999998}]},{"osd":3,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":2.2090000000000001,"5min":2.2090000000000001,"15min":2.2090000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.9380000000000002,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.2849999999999999}]},{"osd":4,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.391,"15min":1.391},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.248},{"interface":"front","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":1.1839999999999999}]},{"osd":5,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.407,"15min":1.407},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"max":{"1min":1.9419999999999999,"5min":1.9419999999999999,"15min":1.9419999999999999},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.458,"5min":1.458,"15min":1.458},"min":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":1.657}]},{"osd":6,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.472,"15min":1.472},"min":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.425},{"interface":"front","average":{"1min":1.508,"5min":1.508,"15min":1.508},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.6520000000000001,"5min":2.6520000000000001,"15min":2.6520000000000001},"last":1.1379999999999999}]},{"osd":7,"last update":"Mon Jan 15 17:58:24 2024","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.591,"15min":1.591},"min":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"max":{"1min":2.5950000000000002,"5min":2.5950000000000002,"15min":2.5950000000000002},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.456,"15min":1.456},"min":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.1219999999999999,"5min":2.1219999999999999,"15min":2.1219999999999999},"last":1.0489999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T17:59:04.906 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T17:59:04.906 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T17:59:05.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.172+0000 7fbc39200640 1 Processor -- start 2024-01-15T17:59:05.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.173+0000 7fbc39200640 1 -- start start 2024-01-15T17:59:05.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.174+0000 7fbc39200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc340cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.174+0000 7fbc39200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 0x7fbc340b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.174+0000 7fbc39200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc340d0af0 con 0x7fbc340cfac0 2024-01-15T17:59:05.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.174+0000 7fbc39200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc340ce7b0 con 0x7fbc340d1d40 2024-01-15T17:59:05.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.174+0000 7fbc39200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340cdbe0 con 0x7fbc341609d0 2024-01-15T17:59:05.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.175+0000 7fbc32a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 0x7fbc340b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.175+0000 7fbc33400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc340cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.175+0000 7fbc32a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 0x7fbc340b3fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:33306/0 (socket says 172.21.15.2:33306) 2024-01-15T17:59:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.175+0000 7fbc33e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fbc340cfac0 0x7fbc340cff70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:33786/0 (socket says 172.21.15.2:33786) 2024-01-15T17:59:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.175+0000 7fbc32a00640 1 -- 172.21.15.2:0/1884182728 learned_addr learned my addr 172.21.15.2:0/1884182728 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:05.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.176+0000 7fbc33400640 1 -- 172.21.15.2:0/1884182728 >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 msgr2=0x7fbc340b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:05.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.176+0000 7fbc33400640 1 --2- 172.21.15.2:0/1884182728 >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 0x7fbc340b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.176+0000 7fbc33400640 1 -- 172.21.15.2:0/1884182728 >> v1:172.21.15.137:6789/0 conn(0x7fbc340cfac0 legacy=0x7fbc340cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.176+0000 7fbc33400640 1 -- 172.21.15.2:0/1884182728 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc340cde80 con 0x7fbc341609d0 2024-01-15T17:59:05.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.176+0000 7fbc32a00640 1 --2- 172.21.15.2:0/1884182728 >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 0x7fbc340b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T17:59:05.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.178+0000 7fbc33400640 1 --2- 172.21.15.2:0/1884182728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc340cec40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009d90 tx=0x7fbc2402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=961f644d75690c6f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:05.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.178+0000 7fbc32000640 1 -- 172.21.15.2:0/1884182728 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2402cc50 con 0x7fbc341609d0 2024-01-15T17:59:05.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.179+0000 7fbc32000640 1 -- 172.21.15.2:0/1884182728 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2402cdb0 con 0x7fbc341609d0 2024-01-15T17:59:05.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.179+0000 7fbc32000640 1 -- 172.21.15.2:0/1884182728 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24034660 con 0x7fbc341609d0 2024-01-15T17:59:05.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.179+0000 7fbc39200640 1 -- 172.21.15.2:0/1884182728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 msgr2=0x7fbc340cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.179+0000 7fbc39200640 1 --2- 172.21.15.2:0/1884182728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc340cec40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009d90 tx=0x7fbc2402bea0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.180+0000 7fbc39200640 1 -- 172.21.15.2:0/1884182728 shutdown_connections 2024-01-15T17:59:05.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.180+0000 7fbc39200640 1 --2- 172.21.15.2:0/1884182728 >> v2:172.21.15.137:3300/0 conn(0x7fbc340d1d40 0x7fbc340b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.180+0000 7fbc39200640 1 --2- 172.21.15.2:0/1884182728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc340cec40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.180+0000 7fbc39200640 1 -- 172.21.15.2:0/1884182728 >> 172.21.15.2:0/1884182728 conn(0x7fbc3409c3e0 msgr2=0x7fbc340d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:05.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.180+0000 7fbc39200640 1 -- 172.21.15.2:0/1884182728 shutdown_connections 2024-01-15T17:59:05.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.180+0000 7fbc39200640 1 -- 172.21.15.2:0/1884182728 wait complete. 2024-01-15T17:59:05.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.182+0000 7fbc39200640 1 Processor -- start 2024-01-15T17:59:05.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.182+0000 7fbc39200640 1 -- start start 2024-01-15T17:59:05.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc39200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 0x7fbc340d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc39200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc340d1d40 0x7fbc3414c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc39200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc3414c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc39200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340cba30 con 0x7fbc340cfac0 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc39200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340cae20 con 0x7fbc340d1d40 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc39200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340cb100 con 0x7fbc341609d0 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc33400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 0x7fbc340d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc33400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 0x7fbc340d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52558/0 (socket says 172.21.15.2:52558) 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.184+0000 7fbc33400640 1 -- 172.21.15.2:0/3240877809 learned_addr learned my addr 172.21.15.2:0/3240877809 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc32a00640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc340d1d40 0x7fbc3414c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33e00640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc3414c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33400640 1 -- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 msgr2=0x7fbc3414c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33400640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc3414c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33400640 1 -- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc340d1d40 msgr2=0x7fbc3414c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33400640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc340d1d40 0x7fbc3414c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33400640 1 -- 172.21.15.2:0/3240877809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c0094d0 con 0x7fbc340cfac0 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc33400640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 0x7fbc340d0ef0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009d90 tx=0x7fbc24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.185+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2403e610 con 0x7fbc340cfac0 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.186+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2403e790 con 0x7fbc340cfac0 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.186+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2403eb40 con 0x7fbc340cfac0 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.186+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc24009a70 con 0x7fbc340cfac0 2024-01-15T17:59:05.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.186+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc340cb6c0 con 0x7fbc340cfac0 2024-01-15T17:59:05.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.187+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc340b4780 con 0x7fbc340cfac0 2024-01-15T17:59:05.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.188+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbc2403b050 con 0x7fbc340cfac0 2024-01-15T17:59:05.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.189+0000 7fbc2b400640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc0c06b2c0 0x7fbc0c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.189+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc240ac4f0 con 0x7fbc340cfac0 2024-01-15T17:59:05.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.189+0000 7fbc32a00640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc0c06b2c0 0x7fbc0c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.190+0000 7fbc32a00640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc0c06b2c0 0x7fbc0c06d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc340c6560 tx=0x7fbc1c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:05.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.193+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc240772c0 con 0x7fbc340cfac0 2024-01-15T17:59:05.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.458+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fbc34074770 con 0x7fbc340cfac0 2024-01-15T17:59:05.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.463+0000 7fbc2b400640 1 -- 172.21.15.2:0/3240877809 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7fbc2407a8e0 con 0x7fbc340cfac0 2024-01-15T17:59:05.466 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T17:59:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T17:59:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T17:59:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T17:59:05.467 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T17:59:05.467 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T17:59:05.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc0c06b2c0 msgr2=0x7fbc0c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc0c06b2c0 0x7fbc0c06d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc340c6560 tx=0x7fbc1c008040 comp rx=0 tx=0).stop 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 msgr2=0x7fbc340d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 0x7fbc340d0ef0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fbc24009d90 tx=0x7fbc24004a40 comp rx=0 tx=0).stop 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 shutdown_connections 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc0c06b2c0 0x7fbc0c06d780 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc341609d0 0x7fbc3414c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc340d1d40 0x7fbc3414c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 --2- 172.21.15.2:0/3240877809 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc340cfac0 0x7fbc340d0ef0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 >> 172.21.15.2:0/3240877809 conn(0x7fbc3409c3e0 msgr2=0x7fbc340b5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:05.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 shutdown_connections 2024-01-15T17:59:05.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.466+0000 7fbc39200640 1 -- 172.21.15.2:0/3240877809 wait complete. 2024-01-15T17:59:05.491 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T17:59:05.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.705+0000 7fdf43200640 1 Processor -- start 2024-01-15T17:59:05.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.705+0000 7fdf43200640 1 -- start start 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf43200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf3c15c0e0 0x7fdf3c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf43200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c15cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf43200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf3c15cb00 con 0x7fdf3c15b7d0 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf43200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf3c0ccec0 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf43200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c0cd1a0 con 0x7fdf3c06cb80 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf3be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf3c15c0e0 0x7fdf3c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf3be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf3c15c0e0 0x7fdf3c0cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52572/0 (socket says 172.21.15.2:52572) 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.707+0000 7fdf3be00640 1 -- 172.21.15.2:0/3805097150 learned_addr learned my addr 172.21.15.2:0/3805097150 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.708+0000 7fdf41200640 1 --2- 172.21.15.2:0/3805097150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c15cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.708+0000 7fdf3be00640 1 -- 172.21.15.2:0/3805097150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 msgr2=0x7fdf3c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.708+0000 7fdf3be00640 1 --2- 172.21.15.2:0/3805097150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c15cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.708+0000 7fdf3be00640 1 -- 172.21.15.2:0/3805097150 >> v1:172.21.15.137:6789/0 conn(0x7fdf3c15b7d0 legacy=0x7fdf3c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.708+0000 7fdf3be00640 1 -- 172.21.15.2:0/3805097150 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c0cd480 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.709+0000 7fdf3be00640 1 --2- 172.21.15.2:0/3805097150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c0cc5c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009870 tx=0x7fdf2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d735a758aa2e7f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.709+0000 7fdf3aa00640 1 -- 172.21.15.2:0/3805097150 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c031640 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.709+0000 7fdf3aa00640 1 -- 172.21.15.2:0/3805097150 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2c0317a0 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.709+0000 7fdf3aa00640 1 -- 172.21.15.2:0/3805097150 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2c031b30 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 -- 172.21.15.2:0/3805097150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 msgr2=0x7fdf3c0cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 --2- 172.21.15.2:0/3805097150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c0cc5c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c009870 tx=0x7fdf2c02b8f0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 -- 172.21.15.2:0/3805097150 shutdown_connections 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 --2- 172.21.15.2:0/3805097150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c15cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 --2- 172.21.15.2:0/3805097150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c0cc5c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 -- 172.21.15.2:0/3805097150 >> 172.21.15.2:0/3805097150 conn(0x7fdf3c06da20 msgr2=0x7fdf3c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 -- 172.21.15.2:0/3805097150 shutdown_connections 2024-01-15T17:59:05.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.710+0000 7fdf43200640 1 -- 172.21.15.2:0/3805097150 wait complete. 2024-01-15T17:59:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.712+0000 7fdf43200640 1 Processor -- start 2024-01-15T17:59:05.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.712+0000 7fdf43200640 1 -- start start 2024-01-15T17:59:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf43200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c14dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf43200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3c15b7d0 0x7fdf3c14f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf43200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf43200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c154160 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf43200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c1f62a0 con 0x7fdf3c15b7d0 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf43200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3c1f6580 con 0x7fdf3c06cb80 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf41200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf41200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c14fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52576/0 (socket says 172.21.15.2:52576) 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.713+0000 7fdf41200640 1 -- 172.21.15.2:0/4266473171 learned_addr learned my addr 172.21.15.2:0/4266473171 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf3be00640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c14dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf3b400640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3c15b7d0 0x7fdf3c14f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf41200640 1 -- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 msgr2=0x7fdf3c14dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf41200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c14dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf41200640 1 -- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3c15b7d0 msgr2=0x7fdf3c14f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf41200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3c15b7d0 0x7fdf3c14f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf41200640 1 -- 172.21.15.2:0/4266473171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0094d0 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf41200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c14fd00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400cd70 tx=0x7fdf3400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34003b90 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf34003cf0 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c1f6860 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.714+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf34004240 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.715+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c14e1a0 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.716+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c1f6d80 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.717+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdf340043a0 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.718+0000 7fdf33e00640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf1406b2c0 0x7fdf1406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.718+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf34012070 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.718+0000 7fdf3be00640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf1406b2c0 0x7fdf1406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:05.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.720+0000 7fdf3be00640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf1406b2c0 0x7fdf1406d780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c036000 tx=0x7fdf2c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:05.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.722+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf3c1f6d80 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.955+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf3c1f6d80 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.956+0000 7fdf33e00640 1 -- 172.21.15.2:0/4266473171 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4904 (secure 0 0 0) 0x7fdf3c1f6d80 con 0x7fdf3c15c0e0 2024-01-15T17:59:05.960 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T17:59:05.960 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/122233456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":122233456},{"type":"v1","addr":"172.21.15.137:6837","nonce":122233456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1272974394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":1272974394},{"type":"v1","addr":"172.21.15.2:6835","nonce":1272974394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/84168043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":84168043},{"type":"v1","addr":"172.21.15.137:6835","nonce":84168043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T17:58:54.222210+0000","modified":"2024-01-15T17:58:57.297868+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4954":{"gid":4954,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.2:6834/2859107199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2859107199},{"type":"v1","addr":"172.21.15.2:6834","nonce":2859107199}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-15T17:59:05.960 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 16 2024-01-15T17:59:05.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf1406b2c0 msgr2=0x7fdf1406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf1406b2c0 0x7fdf1406d780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdf2c036000 tx=0x7fdf2c03c040 comp rx=0 tx=0).stop 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 msgr2=0x7fdf3c14fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c14fd00 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fdf3400cd70 tx=0x7fdf3400a660 comp rx=0 tx=0).stop 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 shutdown_connections 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf1406b2c0 0x7fdf1406d780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf3c15c0e0 0x7fdf3c14fd00 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf3c15b7d0 0x7fdf3c14f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 --2- 172.21.15.2:0/4266473171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf3c06cb80 0x7fdf3c14dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 >> 172.21.15.2:0/4266473171 conn(0x7fdf3c06da20 msgr2=0x7fdf3c0b3af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 shutdown_connections 2024-01-15T17:59:05.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:05.959+0000 7fdf43200640 1 -- 172.21.15.2:0/4266473171 wait complete. 2024-01-15T17:59:06.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T17:58:54.222210+0000', 'modified': '2024-01-15T17:58:57.297868+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4954': {'gid': 4954, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.2:6834/2859107199', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6832', 'nonce': 2859107199}, {'type': 'v1', 'addr': '172.21.15.2:6834', 'nonce': 2859107199}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T17:59:06.386 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T17:59:06.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-15T17:59:06.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.613+0000 7f5eb5400640 1 Processor -- start 2024-01-15T17:59:06.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.613+0000 7f5eb5400640 1 -- start start 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.615+0000 7f5eb5400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5eb0144b80 0x7f5eb00cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.615+0000 7f5eb5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 0x7f5eb015b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.615+0000 7f5eb5400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb00b35a0 con 0x7f5eb00b30a0 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.615+0000 7f5eb5400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5eb00b4220 con 0x7f5eb0144b80 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.615+0000 7f5eb5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb0144f60 con 0x7f5eb006cfc0 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.616+0000 7f5eade00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5eb00b30a0 0x7f5eb006c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56874/0 (socket says 172.21.15.2:56874) 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.616+0000 7f5eade00640 1 -- 172.21.15.2:0/2452268311 learned_addr learned my addr 172.21.15.2:0/2452268311 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.616+0000 7f5eaf200640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 0x7f5eb015b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.616+0000 7f5eae800640 1 --2- 172.21.15.2:0/2452268311 >> v2:172.21.15.137:3300/0 conn(0x7f5eb0144b80 0x7f5eb00cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.616+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234414428 0 0) 0x7f5eb00b35a0 con 0x7f5eb00b30a0 2024-01-15T17:59:06.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.616+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e90003520 con 0x7f5eb00b30a0 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368633222 0 0) 0x7f5e90003520 con 0x7f5eb00b30a0 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 msgr2=0x7f5eb015b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 0x7f5eb015b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 >> v2:172.21.15.137:3300/0 conn(0x7f5eb0144b80 msgr2=0x7f5eb00cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 --2- 172.21.15.2:0/2452268311 >> v2:172.21.15.137:3300/0 conn(0x7f5eb0144b80 0x7f5eb00cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00b3880 con 0x7f5eb00b30a0 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5e98001670 con 0x7f5eb00b30a0 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 >> v1:172.21.15.2:6789/0 conn(0x7f5eb00b30a0 legacy=0x7f5eb006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 0x7f5e90003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e90003f40 0x7f5e900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90004d00 0x7f5e90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb00b4220 con 0x7f5e90003f40 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb0144f60 con 0x7f5e90004d00 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9000ba30 con 0x7f5eb006cfc0 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5eade00640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e90003f40 0x7f5e900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.617+0000 7f5eae800640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 0x7f5e90003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.618+0000 7f5eade00640 1 -- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 msgr2=0x7f5e90003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.618+0000 7f5eade00640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb006cfc0 0x7f5e90003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.618+0000 7f5eade00640 1 -- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90004d00 msgr2=0x7f5e90005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.618+0000 7f5eade00640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90004d00 0x7f5e90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.618+0000 7f5eade00640 1 -- 172.21.15.2:0/2452268311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e90003520 con 0x7f5e90003f40 2024-01-15T17:59:06.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.619+0000 7f5eade00640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e90003f40 0x7f5e900047c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c110 tx=0x7f5e900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f617f86969799f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.619+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98005ad0 con 0x7f5e90003f40 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.619+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98005c50 con 0x7f5e90003f40 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.619+0000 7f5ead400640 1 -- 172.21.15.2:0/2452268311 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9802c830 con 0x7f5e90003f40 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eb5400640 1 -- 172.21.15.2:0/2452268311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e90003f40 msgr2=0x7f5e900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eb5400640 1 --2- 172.21.15.2:0/2452268311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e90003f40 0x7f5e900047c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c110 tx=0x7f5e900057f0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eaf200640 1 -- 172.21.15.2:0/2452268311 reap_dead start 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eb5400640 1 -- 172.21.15.2:0/2452268311 shutdown_connections 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eb5400640 1 -- 172.21.15.2:0/2452268311 >> 172.21.15.2:0/2452268311 conn(0x7f5eb00b46d0 msgr2=0x7f5eb00b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eb5400640 1 -- 172.21.15.2:0/2452268311 shutdown_connections 2024-01-15T17:59:06.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.620+0000 7f5eb5400640 1 -- 172.21.15.2:0/2452268311 wait complete. 2024-01-15T17:59:06.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.622+0000 7f5eb5400640 1 Processor -- start 2024-01-15T17:59:06.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.622+0000 7f5eb5400640 1 -- start start 2024-01-15T17:59:06.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eb5400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90003f40 0x7f5eb01f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eb5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb013aee0 0x7f5eb01e1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eb5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 0x7f5eb01411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eb5400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb01e2150 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eb5400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb0141eb0 con 0x7f5e90003f40 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eb5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb013b6e0 con 0x7f5eb013aee0 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eaf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 0x7f5eb01411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.623+0000 7f5eaf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 0x7f5eb01411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52598/0 (socket says 172.21.15.2:52598) 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 -- 172.21.15.2:0/4145000792 learned_addr learned my addr 172.21.15.2:0/4145000792 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eae800640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90003f40 0x7f5eb01f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eade00640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb013aee0 0x7f5eb01e1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 -- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb013aee0 msgr2=0x7f5eb01e1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb013aee0 0x7f5eb01e1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 -- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90003f40 msgr2=0x7f5eb01f3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90003f40 0x7f5eb01f3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 -- 172.21.15.2:0/4145000792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb013b9c0 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.624+0000 7f5eaf200640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 0x7f5eb01411c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00ae9a0 tx=0x7f5ea800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.625+0000 7f5ea6a00640 1 -- 172.21.15.2:0/4145000792 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8010040 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.625+0000 7f5ea6a00640 1 -- 172.21.15.2:0/4145000792 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea800bae0 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.625+0000 7f5ea6a00640 1 -- 172.21.15.2:0/4145000792 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea800be70 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.625+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb013bca0 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.625+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb013b360 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.627+0000 7f5ea6a00640 1 -- 172.21.15.2:0/4145000792 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5ea801e410 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.629+0000 7f5ea6a00640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e8806b2c0 0x7f5e8806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.629+0000 7f5ea6a00640 1 -- 172.21.15.2:0/4145000792 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ea8089df0 con 0x7f5eb01e1cd0 2024-01-15T17:59:06.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.629+0000 7f5eae800640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e8806b2c0 0x7f5e8806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.629+0000 7f5eb7fd8740 1 Processor -- start 2024-01-15T17:59:06.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 -- start start 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf16474ea0 0x55cf16475280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 0x55cf164740a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf164745e0 0x55cf164b8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf163d3880 con 0x55cf1646fba0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf163e2ae0 con 0x55cf164745e0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.630+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf163e01d0 con 0x55cf16474ea0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eae800640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e8806b2c0 0x7f5e8806d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009a10 tx=0x7f5ea0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 0x55cf164740a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 0x55cf164740a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52612/0 (socket says 172.21.15.2:52612) 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 -- 172.21.15.2:0/1741338545 learned_addr learned my addr 172.21.15.2:0/1741338545 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eae800640 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf16474ea0 0x55cf16475280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eaf200640 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf164745e0 0x55cf164b8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 -- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf16474ea0 msgr2=0x55cf16475280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf16474ea0 0x55cf16475280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 -- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf164745e0 msgr2=0x55cf164b8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf164745e0 0x55cf164b8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eade00640 1 -- 172.21.15.2:0/1741338545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf163a3860 con 0x55cf1646fba0 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.631+0000 7f5eaf200640 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf164745e0 0x55cf164b8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.632+0000 7f5eade00640 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 0x55cf164740a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c230 tx=0x7f5e9802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bba8eb1faf4c0bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.632+0000 7f5e9f400640 1 -- 172.21.15.2:0/1741338545 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9803c040 con 0x55cf1646fba0 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.632+0000 7f5e9f400640 1 -- 172.21.15.2:0/1741338545 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98001630 con 0x55cf1646fba0 2024-01-15T17:59:06.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.632+0000 7f5e9f400640 1 -- 172.21.15.2:0/1741338545 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9802ca50 con 0x55cf1646fba0 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.632+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/1741338545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 msgr2=0x55cf164740a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.632+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 0x55cf164740a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c230 tx=0x7f5e9802e000 comp rx=0 tx=0).stop 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/1741338545 shutdown_connections 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf164745e0 0x55cf164b8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf1646fba0 0x55cf164740a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/1741338545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf16474ea0 0x55cf16475280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/1741338545 >> 172.21.15.2:0/1741338545 conn(0x55cf16436c70 msgr2=0x55cf164ba990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/1741338545 shutdown_connections 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/1741338545 wait complete. 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.633+0000 7f5eb7fd8740 1 -- start start 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.634+0000 7f5eb7fd8740 1 Processor -- start 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 0x55cf16499dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf1649a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 0x55cf1649a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf163dbac0 con 0x55cf164745e0 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf163782f0 con 0x55cf16474ea0 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf16395c70 con 0x55cf1646fba0 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eb7fd8740 10 client.0 _ll_register_callbacks cb 0x55cf1639a3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf1649a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf1649a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52616/0 (socket says 172.21.15.2:52616) 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 -- 172.21.15.2:0/168662078 learned_addr learned my addr 172.21.15.2:0/168662078 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 -- 172.21.15.2:0/168662078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 msgr2=0x55cf16499dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 0x55cf16499dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 -- 172.21.15.2:0/168662078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 msgr2=0x55cf1649a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 0x55cf1649a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.635+0000 7f5eade00640 1 -- 172.21.15.2:0/168662078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e90003520 con 0x55cf164745e0 2024-01-15T17:59:06.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5eade00640 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf1649a310 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e98045040 tx=0x7f5e980019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba032eef65f6413 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5e97e00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T17:59:06.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5e97e00640 1 -- 172.21.15.2:0/168662078 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9802cc40 con 0x55cf164745e0 2024-01-15T17:59:06.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf163d4aa0 con 0x55cf164745e0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5e97e00640 1 -- 172.21.15.2:0/168662078 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e98001470 con 0x55cf164745e0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5e97e00640 1 -- 172.21.15.2:0/168662078 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e9802cda0 con 0x55cf164745e0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5eb7fd8740 10 client.5382 fetch_fsmap learned FSMap version 16 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.636+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55cf163da140 con 0x55cf164745e0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5e97e00640 1 -- 172.21.15.2:0/168662078 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cf163d4aa0 con 0x55cf164745e0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5e97e00640 1 -- 172.21.15.2:0/168662078 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5e98007070 con 0x55cf164745e0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eb7fd8740 10 client.5382 fetch_fsmap finished waiting for FSMap version 16 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eb7fd8740 10 client.5382 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eb7fd8740 20 client.5382 populate_metadata read hostname 'smithi002' 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf164bf9d0 0x55cf164d7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eb7fd8740 4 client.5382 mds_command: new command op to 4954 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 --> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf163d47c0 con 0x55cf164bf9d0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.637+0000 7f5eae800640 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf164bf9d0 0x55cf164d7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.638+0000 7f5eae800640 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf164bf9d0 0x55cf164d7c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1025a6716f0e8386 server_cookie=4e28834852e5633a in_seq=0 out_seq=0 2024-01-15T17:59:06.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.638+0000 7f5e97e00640 10 client.5382 ms_handle_connect on v2:172.21.15.2:6832/2859107199 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5e97e00640 1 -- 172.21.15.2:0/168662078 <== mds.0 v2:172.21.15.2:6832/2859107199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cf163d47c0 con 0x55cf164bf9d0 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5e97e00640 10 client.5382 handle_command_reply: tid=0 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 client.5382 shutdown 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 msgr2=0x55cf1649a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf1649a310 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f5e98045040 tx=0x7f5e980019a0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 shutdown_connections 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf164bf9d0 0x55cf164d7c40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 0x55cf1649a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf1649a310 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/168662078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 0x55cf16499dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5e97e00640 0 client.5382 ms_handle_reset on v2:172.21.15.2:6832/2859107199 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 >> 172.21.15.2:0/168662078 conn(0x55cf16436c70 msgr2=0x55cf16475c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 shutdown_connections 2024-01-15T17:59:06.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.639+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/168662078 wait complete. 2024-01-15T17:59:06.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.640+0000 7f5eb7fd8740 20 client.5382 trim_cache size 0 max 16384 2024-01-15T17:59:06.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.657+0000 7f5eb7fd8740 1 Processor -- start 2024-01-15T17:59:06.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.657+0000 7f5eb7fd8740 1 -- start start 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 0x55cf16475280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf164c1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf1613ef20 con 0x55cf164745e0 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf164beb80 con 0x55cf16474ea0 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf164708e0 con 0x55cf1646fba0 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf164c1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf164c1970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52630/0 (socket says 172.21.15.2:52630) 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 -- 172.21.15.2:0/2744619792 learned_addr learned my addr 172.21.15.2:0/2744619792 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 -- 172.21.15.2:0/2744619792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 msgr2=0x55cf16475280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 0x55cf16475280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 -- 172.21.15.2:0/2744619792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 msgr2=0x55cf1646eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.658+0000 7f5eade00640 1 -- 172.21.15.2:0/2744619792 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf16377b80 con 0x55cf164745e0 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.659+0000 7f5eade00640 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf164c1970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5e980078a0 tx=0x7f5e98049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bb861bffd5bf05 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.659+0000 7f5e9fe00640 1 -- 172.21.15.2:0/2744619792 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98001470 con 0x55cf164745e0 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.659+0000 7f5e9fe00640 1 -- 172.21.15.2:0/2744619792 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e980074e0 con 0x55cf164745e0 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.659+0000 7f5e9fe00640 1 -- 172.21.15.2:0/2744619792 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9802cc40 con 0x55cf164745e0 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.659+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/2744619792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 msgr2=0x55cf164c1970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.659+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf164c1970 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f5e980078a0 tx=0x7f5e98049000 comp rx=0 tx=0).stop 2024-01-15T17:59:06.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/2744619792 shutdown_connections 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf164745e0 0x55cf164c1970 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/2744619792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf1646fba0 0x55cf16475280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/2744619792 >> 172.21.15.2:0/2744619792 conn(0x55cf16436c70 msgr2=0x55cf164bfdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/2744619792 shutdown_connections 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/2744619792 wait complete. 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.660+0000 7f5eb7fd8740 1 -- start start 2024-01-15T17:59:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.661+0000 7f5eb7fd8740 1 Processor -- start 2024-01-15T17:59:06.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf1646fba0 0x55cf1649f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf164745e0 0x55cf164c1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf164c2250 con 0x55cf16474ea0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf1649a720 con 0x55cf1646fba0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf16498e30 con 0x55cf164745e0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eb7fd8740 10 client.0 _ll_register_callbacks cb 0x7f5e90004670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52634/0 (socket says 172.21.15.2:52634) 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 -- 172.21.15.2:0/4134693157 learned_addr learned my addr 172.21.15.2:0/4134693157 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eae800640 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf1646fba0 0x55cf1649f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 -- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf164745e0 msgr2=0x55cf164c1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf164745e0 0x55cf164c1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 -- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf1646fba0 msgr2=0x55cf1649f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf1646fba0 0x55cf1649f820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.662+0000 7f5eaf200640 1 -- 172.21.15.2:0/4134693157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf163da140 con 0x55cf16474ea0 2024-01-15T17:59:06.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eaf200640 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800f010 tx=0x7f5ea8058920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0ec4a1f6e615fc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5e97e00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5e97e00640 1 -- 172.21.15.2:0/4134693157 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea8070220 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5e97e00640 1 -- 172.21.15.2:0/4134693157 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ea8060c20 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e98001630 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5e97e00640 1 -- 172.21.15.2:0/4134693157 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ea8088050 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eb7fd8740 10 client.5388 fetch_fsmap learned FSMap version 16 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55cf1639d170 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5e97e00640 1 -- 172.21.15.2:0/4134693157 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98001630 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5e97e00640 1 -- 172.21.15.2:0/4134693157 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5ea8060790 con 0x55cf16474ea0 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eb7fd8740 10 client.5388 fetch_fsmap finished waiting for FSMap version 16 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eb7fd8740 10 client.5388 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.663+0000 7f5eb7fd8740 20 client.5388 populate_metadata read hostname 'smithi002' 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.664+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf16499e00 0x55cf164bdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.664+0000 7f5eb7fd8740 4 client.5388 mds_command: new command op to 4954 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T17:59:06.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.664+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 --> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55cf160dc6c0 con 0x55cf16499e00 2024-01-15T17:59:06.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.664+0000 7f5eae800640 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf16499e00 0x55cf164bdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.664+0000 7f5eae800640 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf16499e00 0x55cf164bdad0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37017195a38bd67 server_cookie=7b534ea0c752e449 in_seq=0 out_seq=0 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.664+0000 7f5e97e00640 10 client.5388 ms_handle_connect on v2:172.21.15.2:6832/2859107199 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5e97e00640 1 -- 172.21.15.2:0/4134693157 <== mds.0 v2:172.21.15.2:6832/2859107199 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55cf160dc6c0 con 0x55cf16499e00 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5e97e00640 10 client.5388 handle_command_reply: tid=0 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 client.5388 shutdown 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 msgr2=0x55cf1646eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f5ea800f010 tx=0x7f5ea8058920 comp rx=0 tx=0).stop 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 shutdown_connections 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55cf16474ea0 0x55cf1646eb80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55cf164745e0 0x55cf164c1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.2:6832/2859107199,v1:172.21.15.2:6834/2859107199] conn(0x55cf16499e00 0x55cf164bdad0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 --2- 172.21.15.2:0/4134693157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55cf1646fba0 0x55cf1649f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 >> 172.21.15.2:0/4134693157 conn(0x55cf16436c70 msgr2=0x55cf16473790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5e97e00640 0 client.5388 ms_handle_reset on v2:172.21.15.2:6832/2859107199 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 shutdown_connections 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 1 -- 172.21.15.2:0/4134693157 wait complete. 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.665+0000 7f5eb7fd8740 20 client.5388 trim_cache size 0 max 16384 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4954, 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T17:59:06.671 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 11.894287113000001, 2024-01-15T17:59:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 16, 2024-01-15T17:59:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 35, 2024-01-15T17:59:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 35, 2024-01-15T17:59:06.672 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 20.717787094999998 2024-01-15T17:59:06.672 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e8806b2c0 msgr2=0x7f5e8806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e8806b2c0 0x7f5e8806d780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009a10 tx=0x7f5ea0008040 comp rx=0 tx=0).stop 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 msgr2=0x7f5eb01411c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 0x7f5eb01411c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00ae9a0 tx=0x7f5ea800b480 comp rx=0 tx=0).stop 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 shutdown_connections 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e8806b2c0 0x7f5e8806d780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb01e1cd0 0x7f5eb01411c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5eb013aee0 0x7f5eb01e1790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 --2- 172.21.15.2:0/4145000792 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e90003f40 0x7f5eb01f3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 >> 172.21.15.2:0/4145000792 conn(0x7f5eb00b46d0 msgr2=0x7f5eb0143c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 shutdown_connections 2024-01-15T17:59:06.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.666+0000 7f5eb5400640 1 -- 172.21.15.2:0/4145000792 wait complete. 2024-01-15T17:59:06.687 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T17:59:06.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.889+0000 7f1be0e00640 1 Processor -- start 2024-01-15T17:59:06.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.890+0000 7f1be0e00640 1 -- start start 2024-01-15T17:59:06.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.891+0000 7f1be0e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bdc143810 0x7f1bdc06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1be0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc06e010 0x7f1bdc0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1be0e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1bdc0b2080 0x7f1bdc0700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1be0e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc0b3520 con 0x7f1bdc143810 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1be0e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bdc06e470 con 0x7f1bdc0b2080 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1be0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc0ccbe0 con 0x7f1bdc06e010 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bdc143810 0x7f1bdc06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bdc143810 0x7f1bdc06dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52650/0 (socket says 172.21.15.2:52650) 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 -- 172.21.15.2:0/1317219571 learned_addr learned my addr 172.21.15.2:0/1317219571 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bd3e00640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc06e010 0x7f1bdc0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 -- 172.21.15.2:0/1317219571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc06e010 msgr2=0x7f1bdc0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc06e010 0x7f1bdc0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 -- 172.21.15.2:0/1317219571 >> v2:172.21.15.137:3300/0 conn(0x7f1bdc0b2080 msgr2=0x7f1bdc0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 --2- 172.21.15.2:0/1317219571 >> v2:172.21.15.137:3300/0 conn(0x7f1bdc0b2080 0x7f1bdc0700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bdb400640 1 -- 172.21.15.2:0/1317219571 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc143bf0 con 0x7f1bdc143810 2024-01-15T17:59:06.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.892+0000 7f1bd3e00640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc06e010 0x7f1bdc0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.894+0000 7f1bdb400640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc143810 0x7f1bdc06dad0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0ccb50 tx=0x7f1bcc02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b96daec0ba3605a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.894+0000 7f1bdaa00640 1 -- 172.21.15.2:0/1317219571 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc0317f0 con 0x7f1bdc143810 2024-01-15T17:59:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.894+0000 7f1bdaa00640 1 -- 172.21.15.2:0/1317219571 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc031950 con 0x7f1bdc143810 2024-01-15T17:59:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.894+0000 7f1bdaa00640 1 -- 172.21.15.2:0/1317219571 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc031ce0 con 0x7f1bdc143810 2024-01-15T17:59:06.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 -- 172.21.15.2:0/1317219571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc143810 msgr2=0x7f1bdc06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc143810 0x7f1bdc06dad0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1bdc0ccb50 tx=0x7f1bcc02b890 comp rx=0 tx=0).stop 2024-01-15T17:59:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 -- 172.21.15.2:0/1317219571 shutdown_connections 2024-01-15T17:59:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 --2- 172.21.15.2:0/1317219571 >> v2:172.21.15.137:3300/0 conn(0x7f1bdc0b2080 0x7f1bdc0700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc06e010 0x7f1bdc0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 --2- 172.21.15.2:0/1317219571 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc143810 0x7f1bdc06dad0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 -- 172.21.15.2:0/1317219571 >> 172.21.15.2:0/1317219571 conn(0x7f1bdc0b0840 msgr2=0x7f1bdc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 -- 172.21.15.2:0/1317219571 shutdown_connections 2024-01-15T17:59:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.895+0000 7f1be0e00640 1 -- 172.21.15.2:0/1317219571 wait complete. 2024-01-15T17:59:06.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.897+0000 7f1be0e00640 1 Processor -- start 2024-01-15T17:59:06.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.897+0000 7f1be0e00640 1 -- start start 2024-01-15T17:59:06.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1be0e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 0x7f1bdc0c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1be0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc0b2080 0x7f1bdc0acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1be0e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bdc143810 0x7f1bdc1401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1be0e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc140ff0 con 0x7f1bdc06e010 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1be0e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc1392d0 con 0x7f1bdc143810 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1be0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bdc13b230 con 0x7f1bdc0b2080 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1bdb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 0x7f1bdc0c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1bdb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 0x7f1bdc0c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52656/0 (socket says 172.21.15.2:52656) 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.898+0000 7f1bdb400640 1 -- 172.21.15.2:0/2271653711 learned_addr learned my addr 172.21.15.2:0/2271653711 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bdb400640 1 -- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc0b2080 msgr2=0x7f1bdc0acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bdb400640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc0b2080 0x7f1bdc0acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bdb400640 1 -- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bdc143810 msgr2=0x7f1bdc1401a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bdb400640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bdc143810 0x7f1bdc1401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bdb400640 1 -- 172.21.15.2:0/2271653711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0094d0 con 0x7f1bdc06e010 2024-01-15T17:59:06.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bdb400640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 0x7f1bdc0c3ab0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc02b8d0 tx=0x7f1bcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc032950 con 0x7f1bdc06e010 2024-01-15T17:59:06.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc032ab0 con 0x7f1bdc06e010 2024-01-15T17:59:06.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bdc140d10 con 0x7f1bdc06e010 2024-01-15T17:59:06.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc032e60 con 0x7f1bdc06e010 2024-01-15T17:59:06.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.899+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bdc1397c0 con 0x7f1bdc06e010 2024-01-15T17:59:06.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.900+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bdc074770 con 0x7f1bdc06e010 2024-01-15T17:59:06.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.902+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1bcc04e450 con 0x7f1bdc06e010 2024-01-15T17:59:06.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.903+0000 7f1bd3400640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bb406b2f0 0x7f1bb406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:06.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.903+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bcc0abfd0 con 0x7f1bdc06e010 2024-01-15T17:59:06.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.903+0000 7f1bd3e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bb406b2f0 0x7f1bb406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:06.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.904+0000 7f1bd3e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bb406b2f0 0x7f1bb406d7b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4006fd0 tx=0x7f1bc4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:06.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:06.907+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bcc076e40 con 0x7f1bdc06e010 2024-01-15T17:59:07.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.140+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1bdc074060 con 0x7f1bdc06e010 2024-01-15T17:59:07.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.141+0000 7f1bd3400640 1 -- 172.21.15.2:0/2271653711 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1bcc07a460 con 0x7f1bdc06e010 2024-01-15T17:59:07.145 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.144+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bb406b2f0 msgr2=0x7f1bb406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.144+0000 7f1be0e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bb406b2f0 0x7f1bb406d7b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4006fd0 tx=0x7f1bc4008040 comp rx=0 tx=0).stop 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.144+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 msgr2=0x7f1bdc0c3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.144+0000 7f1be0e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 0x7f1bdc0c3ab0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc02b8d0 tx=0x7f1bcc0046c0 comp rx=0 tx=0).stop 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 shutdown_connections 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bb406b2f0 0x7f1bb406d7b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:07.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bdc143810 0x7f1bdc1401a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:07.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bdc0b2080 0x7f1bdc0acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:07.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 --2- 172.21.15.2:0/2271653711 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bdc06e010 0x7f1bdc0c3ab0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:07.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 >> 172.21.15.2:0/2271653711 conn(0x7f1bdc0b0840 msgr2=0x7f1bdc0cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:07.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 shutdown_connections 2024-01-15T17:59:07.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:07.145+0000 7f1be0e00640 1 -- 172.21.15.2:0/2271653711 wait complete. 2024-01-15T17:59:07.167 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T17:59:07.167 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T17:59:07.167 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T17:59:07.168 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T17:59:07.168 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T17:59:07.168 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T17:59:07.168 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T17:59:07.168 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T17:59:07.168 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T17:59:07.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:07.169 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T17:59:07.221 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:59:07.221 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:59:07.221 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:59:07.221 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:07.222 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42535sec preferred_lft 42535sec 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.223 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T17:59:07.224 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:07.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:07.226 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T17:59:07.319 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T17:59:07.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:07.324 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T17:59:07.382 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T17:59:07.382 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T17:59:07.382 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T17:59:07.382 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T17:59:07.382 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T17:59:07.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:07.383 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T17:59:07.383 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T17:59:07.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T17:59:07.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T17:59:07.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T17:59:07.383 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T17:59:07.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:07.576 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T17:59:07.629 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T17:59:07.629 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T17:59:07.629 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T17:59:07.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:07.630 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T17:59:07.727 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T17:59:07.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:59:07.727 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T17:59:07.784 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T17:59:07.784 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:59:07.785 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T17:59:07.835 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:59:07.935 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:59:07.995 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:59:07.996 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:59:07.997 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:59:07.998 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:59:08.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:08.000 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:59:08.056 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T17:59:08.056 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T17:59:08.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T17:59:08.147 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T17:59:08.194 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:59:08.195 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:59:08.196 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T17:59:08.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:08.198 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T17:59:08.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T17:59:08.262+0000 7f5d7c35c480 -1 init, newargv = 0x55a9df77bca0 newargc=15 2024-01-15T17:59:08.265 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[49225]: starting ceph client 2024-01-15T17:59:08.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[49225]: starting fuse 2024-01-15T17:59:08.287 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T17:59:08.287 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T17:59:08.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T17:59:08.287 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T17:59:08.287 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T17:59:08.288 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T17:59:08.289 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T17:59:08.290 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T17:59:08.290 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T17:59:08.493 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.49225.asok 2024-01-15T17:59:08.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T17:59:08.511 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49225.asok 2024-01-15T17:59:08.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:08.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49225.asok status 2024-01-15T17:59:08.758 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T17:59:08.758 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T17:59:08.758 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T17:59:08.758 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T17:59:08.758 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T17:59:08.758 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "49225", 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5400, 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T17:59:08.759 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 5400 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 4185218873 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T17:59:08.760 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 4185218873 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.5400 192.168.0.1:0/4185218873", 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/4185218873", 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 16, 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 35, 2024-01-15T17:59:08.761 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T17:59:08.762 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T17:59:08.762 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T17:59:08.762 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T17:59:08.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:08.774 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T17:59:08.774 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T17:59:08.829 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T17:59:08.829 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T17:59:08.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:08.830 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T17:59:08.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T17:59:09.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.156+0000 7f6ab6e00640 1 Processor -- start 2024-01-15T17:59:09.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.156+0000 7f6ab6e00640 1 -- start start 2024-01-15T17:59:09.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.157+0000 7f6ab6e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6ab015c0e0 0x7f6ab00cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:09.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.157+0000 7f6ab6e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ab006cb80 0x7f6ab015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:09.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.157+0000 7f6ab6e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab015cb00 con 0x7f6ab015b7d0 2024-01-15T17:59:09.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.157+0000 7f6ab6e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00ccec0 con 0x7f6ab006cb80 2024-01-15T17:59:09.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.157+0000 7f6ab6e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab00cd1a0 con 0x7f6ab015c0e0 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ab006cb80 0x7f6ab015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ab006cb80 0x7f6ab015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52660/0 (socket says 172.21.15.2:52660) 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 -- 172.21.15.2:0/3635679402 learned_addr learned my addr 172.21.15.2:0/3635679402 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6aafe00640 1 --2- 172.21.15.2:0/3635679402 >> v2:172.21.15.137:3300/0 conn(0x7f6ab015c0e0 0x7f6ab00cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 -- 172.21.15.2:0/3635679402 >> v2:172.21.15.137:3300/0 conn(0x7f6ab015c0e0 msgr2=0x7f6ab00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 --2- 172.21.15.2:0/3635679402 >> v2:172.21.15.137:3300/0 conn(0x7f6ab015c0e0 0x7f6ab00cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 -- 172.21.15.2:0/3635679402 >> v1:172.21.15.2:6789/0 conn(0x7f6ab015b7d0 legacy=0x7f6ab006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6ab4e00640 1 -- 172.21.15.2:0/3635679402 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab00cd480 con 0x7f6ab006cb80 2024-01-15T17:59:09.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.158+0000 7f6aafe00640 1 --2- 172.21.15.2:0/3635679402 >> v2:172.21.15.137:3300/0 conn(0x7f6ab015c0e0 0x7f6ab00cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.159+0000 7f6ab4e00640 1 --2- 172.21.15.2:0/3635679402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab006cb80 0x7f6ab015cf90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00725f0 tx=0x7f6aa802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59244dc7585049b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6aaea00640 1 -- 172.21.15.2:0/3635679402 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa802cb10 con 0x7f6ab006cb80 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6aaea00640 1 -- 172.21.15.2:0/3635679402 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa802cc70 con 0x7f6ab006cb80 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6aaea00640 1 -- 172.21.15.2:0/3635679402 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80343f0 con 0x7f6ab006cb80 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 -- 172.21.15.2:0/3635679402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab006cb80 msgr2=0x7f6ab015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/3635679402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab006cb80 0x7f6ab015cf90 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00725f0 tx=0x7f6aa802bf70 comp rx=0 tx=0).stop 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 -- 172.21.15.2:0/3635679402 shutdown_connections 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/3635679402 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab006cb80 0x7f6ab015cf90 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/3635679402 >> v2:172.21.15.137:3300/0 conn(0x7f6ab015c0e0 0x7f6ab00cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 -- 172.21.15.2:0/3635679402 >> 172.21.15.2:0/3635679402 conn(0x7f6ab006da20 msgr2=0x7f6ab006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:09.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 -- 172.21.15.2:0/3635679402 shutdown_connections 2024-01-15T17:59:09.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.160+0000 7f6ab6e00640 1 -- 172.21.15.2:0/3635679402 wait complete. 2024-01-15T17:59:09.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.162+0000 7f6ab6e00640 1 Processor -- start 2024-01-15T17:59:09.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.162+0000 7f6ab6e00640 1 -- start start 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.163+0000 7f6ab6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ab006cb80 0x7f6ab014ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.163+0000 7f6ab6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 0x7f6ab00b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.163+0000 7f6ab6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ab015c0e0 0x7f6ab01f2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.163+0000 7f6ab6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00b2a50 con 0x7f6ab015b7d0 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.163+0000 7f6ab6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00b2cf0 con 0x7f6ab015c0e0 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.163+0000 7f6ab6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab00b2fd0 con 0x7f6ab006cb80 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 0x7f6ab00b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:09.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 0x7f6ab00b2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52664/0 (socket says 172.21.15.2:52664) 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 -- 172.21.15.2:0/945608316 learned_addr learned my addr 172.21.15.2:0/945608316 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 -- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ab006cb80 msgr2=0x7f6ab014ad60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ab006cb80 0x7f6ab014ad60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6ab4e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ab015c0e0 0x7f6ab01f2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 -- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ab015c0e0 msgr2=0x7f6ab01f2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ab015c0e0 0x7f6ab01f2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 -- 172.21.15.2:0/945608316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa8009b30 con 0x7f6ab015b7d0 2024-01-15T17:59:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.164+0000 7f6aaf400640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 0x7f6ab00b2510 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6a9800c450 tx=0x7f6a9800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:09.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.165+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a980052d0 con 0x7f6ab015b7d0 2024-01-15T17:59:09.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.165+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a98005430 con 0x7f6ab015b7d0 2024-01-15T17:59:09.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.165+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98012870 con 0x7f6ab015b7d0 2024-01-15T17:59:09.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.165+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0070380 con 0x7f6ab015b7d0 2024-01-15T17:59:09.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.165+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0070940 con 0x7f6ab015b7d0 2024-01-15T17:59:09.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.166+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab00b31f0 con 0x7f6ab015b7d0 2024-01-15T17:59:09.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.167+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6a980129d0 con 0x7f6ab015b7d0 2024-01-15T17:59:09.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.168+0000 7f6aa7e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a8806b2c0 0x7f6a8806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:09.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.168+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a980897b0 con 0x7f6ab015b7d0 2024-01-15T17:59:09.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.169+0000 7f6aafe00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a8806b2c0 0x7f6a8806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.170+0000 7f6aafe00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a8806b2c0 0x7f6a8806d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00b21b0 tx=0x7f6aa0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:09.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.174+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a98054580 con 0x7f6ab015b7d0 2024-01-15T17:59:09.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.410+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6ab00686f0 con 0x7f6ab015b7d0 2024-01-15T17:59:09.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.411+0000 7f6aa7e00640 1 -- 172.21.15.2:0/945608316 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6a98057ba0 con 0x7f6ab015b7d0 2024-01-15T17:59:09.414 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T17:59:09.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a8806b2c0 msgr2=0x7f6a8806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a8806b2c0 0x7f6a8806d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00b21b0 tx=0x7f6aa0008040 comp rx=0 tx=0).stop 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 msgr2=0x7f6ab00b2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 0x7f6ab00b2510 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f6a9800c450 tx=0x7f6a9800c910 comp rx=0 tx=0).stop 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 shutdown_connections 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a8806b2c0 0x7f6a8806d780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ab015c0e0 0x7f6ab01f2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ab015b7d0 0x7f6ab00b2510 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 --2- 172.21.15.2:0/945608316 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ab006cb80 0x7f6ab014ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.414+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 >> 172.21.15.2:0/945608316 conn(0x7f6ab006da20 msgr2=0x7f6ab006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.415+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 shutdown_connections 2024-01-15T17:59:09.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:09.415+0000 7f6ab6e00640 1 -- 172.21.15.2:0/945608316 wait complete. 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T17:59:09.437 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T17:59:09.438 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T17:59:09.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:09.438 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:59:09.495 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42521sec preferred_lft 42521sec 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T17:59:09.496 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T17:59:09.497 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T17:59:09.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:09.498 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T17:59:09.578 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T17:59:09.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:09.583 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T17:59:09.635 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T17:59:09.685 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T17:59:09.685 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T17:59:09.685 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T17:59:09.685 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T17:59:09.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:09.685 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T17:59:09.685 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T17:59:09.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T17:59:09.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T17:59:09.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T17:59:09.685 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T17:59:09.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:09.866 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T17:59:09.915 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T17:59:09.915 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T17:59:09.916 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T17:59:09.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:09.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T17:59:10.011 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T17:59:10.011 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:59:10.011 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T17:59:10.066 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T17:59:10.066 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T17:59:10.066 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T17:59:10.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:59:10.210 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:59:10.265 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:59:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:59:10.267 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:59:10.268 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:59:10.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:10.269 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:59:10.318 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T17:59:10.319 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T17:59:10.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T17:59:10.394 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T17:59:10.434 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T17:59:10.434 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T17:59:10.434 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T17:59:10.434 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T17:59:10.435 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T17:59:10.436 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:59:10.437 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T17:59:10.437 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:59:10.437 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T17:59:10.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:10.438 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T17:59:10.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T17:59:10.503+0000 7f8ba7341480 -1 init, newargv = 0x559d5f55adc0 newargc=15 2024-01-15T17:59:10.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[42662]: starting ceph client 2024-01-15T17:59:10.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[42662]: starting fuse 2024-01-15T17:59:10.525 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T17:59:10.525 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T17:59:10.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T17:59:10.525 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T17:59:10.525 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:59:10.526 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T17:59:10.527 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T17:59:10.705 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.42662.asok 2024-01-15T17:59:10.710 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T17:59:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T17:59:10.721 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42662.asok 2024-01-15T17:59:10.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:10.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42662.asok status 2024-01-15T17:59:10.925 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T17:59:10.925 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T17:59:10.925 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "42662", 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 4805, 2024-01-15T17:59:10.926 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 4805 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 487616397 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:59:10.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 487616397 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.4805 192.168.0.1:0/487616397", 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/487616397", 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 16, 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 35, 2024-01-15T17:59:10.928 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T17:59:10.929 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T17:59:10.929 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T17:59:10.929 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T17:59:10.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:10.939 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T17:59:10.940 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T17:59:10.989 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T17:59:10.990 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T17:59:10.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:10.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T17:59:11.079 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T17:59:11.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.311+0000 7fd6a1c00640 1 Processor -- start 2024-01-15T17:59:11.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.311+0000 7fd6a1c00640 1 -- start start 2024-01-15T17:59:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.313+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 0x7fd69c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.313+0000 7fd6a1c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c071090 con 0x7fd69c1447d0 2024-01-15T17:59:11.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.313+0000 7fd6a1c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69c06d160 con 0x7fd69c0720d0 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.313+0000 7fd6a1c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c071430 con 0x7fd69c143d20 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.314+0000 7fd6a1200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd69c1447d0 0x7fd69c0cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56890/0 (socket says 172.21.15.2:56890) 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.314+0000 7fd6a1200640 1 -- 172.21.15.2:0/2213413228 learned_addr learned my addr 172.21.15.2:0/2213413228 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.314+0000 7fd69be00640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 0x7fd69c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.314+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725018697 0 0) 0x7fd69c071090 con 0x7fd69c1447d0 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.314+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd67c003520 con 0x7fd69c1447d0 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912220905 0 0) 0x7fd67c003520 con 0x7fd69c1447d0 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 msgr2=0x7fd69c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 0x7fd69c15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:11.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 >> v1:172.21.15.137:6789/0 conn(0x7fd69c0720d0 legacy=0x7fd69c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c144560 con 0x7fd69c1447d0 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd6940019f0 con 0x7fd69c1447d0 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 >> v1:172.21.15.2:6789/0 conn(0x7fd69c1447d0 legacy=0x7fd69c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd67c0039a0 0x7fd67c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 0x7fd67c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd67c004d00 0x7fd67c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c071430 con 0x7fd67c004d00 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c00bb10 con 0x7fd67c0039a0 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c00bcd0 con 0x7fd69c143d20 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.315+0000 7fd6a1200640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd67c004d00 0x7fd67c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd69be00640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd67c0039a0 0x7fd67c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd693e00640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 0x7fd67c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd6a1200640 1 -- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 msgr2=0x7fd67c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd6a1200640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd69c143d20 0x7fd67c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd6a1200640 1 -- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd67c0039a0 msgr2=0x7fd67c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd6a1200640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd67c0039a0 0x7fd67c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:11.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.316+0000 7fd6a1200640 1 -- 172.21.15.2:0/2213413228 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c003520 con 0x7fd67c004d00 2024-01-15T17:59:11.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.317+0000 7fd6a1200640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd67c004d00 0x7fd67c005180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd69400a760 tx=0x7fd67c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e471f63a5a0a636 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:11.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.317+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694034630 con 0x7fd67c004d00 2024-01-15T17:59:11.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69402f070 con 0x7fd67c004d00 2024-01-15T17:59:11.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd69b400640 1 -- 172.21.15.2:0/2213413228 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694034a90 con 0x7fd67c004d00 2024-01-15T17:59:11.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1c00640 1 -- 172.21.15.2:0/2213413228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd67c004d00 msgr2=0x7fd67c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:11.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/2213413228 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd67c004d00 0x7fd67c005180 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd69400a760 tx=0x7fd67c0057f0 comp rx=0 tx=0).stop 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1200640 1 -- 172.21.15.2:0/2213413228 reap_dead start 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1c00640 1 -- 172.21.15.2:0/2213413228 shutdown_connections 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1c00640 1 -- 172.21.15.2:0/2213413228 >> 172.21.15.2:0/2213413228 conn(0x7fd69c0b0840 msgr2=0x7fd69c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1c00640 1 -- 172.21.15.2:0/2213413228 shutdown_connections 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.318+0000 7fd6a1c00640 1 -- 172.21.15.2:0/2213413228 wait complete. 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.321+0000 7fd6a1c00640 1 Processor -- start 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.321+0000 7fd6a1c00640 1 -- start start 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.322+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd67c004d00 0x7fd69c140fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.322+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd69c0c3c00 0x7fd69c141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.322+0000 7fd6a1c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 0x7fd69c1f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.322+0000 7fd6a1c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c1422b0 con 0x7fd69c15a7d0 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.322+0000 7fd6a1c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c0c4080 con 0x7fd69c0c3c00 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.322+0000 7fd6a1c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c140b10 con 0x7fd67c004d00 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 0x7fd69c1f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 0x7fd69c1f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52696/0 (socket says 172.21.15.2:52696) 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 -- 172.21.15.2:0/1632209139 learned_addr learned my addr 172.21.15.2:0/1632209139 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd69be00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd67c004d00 0x7fd69c140fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd693e00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd69c0c3c00 0x7fd69c141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 -- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd67c004d00 msgr2=0x7fd69c140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd67c004d00 0x7fd69c140fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 -- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd69c0c3c00 msgr2=0x7fd69c141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:11.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd69c0c3c00 0x7fd69c141ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd6a1200640 1 -- 172.21.15.2:0/1632209139 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c142010 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.323+0000 7fd69be00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd67c004d00 0x7fd69c140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.324+0000 7fd6a1200640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 0x7fd69c1f2e40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd69400a760 tx=0x7fd694001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.324+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694034630 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.324+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd694034790 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.324+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694034b00 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.324+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c1401e0 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.324+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd69c1407a0 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.325+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd69c074770 con 0x7fd69c15a7d0 2024-01-15T17:59:11.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.326+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd694031030 con 0x7fd69c15a7d0 2024-01-15T17:59:11.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.328+0000 7fd698c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67406b2c0 0x7fd67406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:11.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.328+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6940abee0 con 0x7fd69c15a7d0 2024-01-15T17:59:11.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.329+0000 7fd69be00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67406b2c0 0x7fd67406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:11.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.330+0000 7fd69be00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67406b2c0 0x7fd67406d780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009980 tx=0x7fd68c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:11.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.333+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd694076e60 con 0x7fd69c15a7d0 2024-01-15T17:59:11.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:11.572+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd69c074770 con 0x7fd69c15a7d0 2024-01-15T17:59:12.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.474+0000 7fd698c00640 1 -- 172.21.15.2:0/1632209139 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 136+0+0 (secure 0 0 0) 0x7fd69407a480 con 0x7fd69c15a7d0 2024-01-15T17:59:12.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67406b2c0 msgr2=0x7fd67406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67406b2c0 0x7fd67406d780 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009980 tx=0x7fd68c008040 comp rx=0 tx=0).stop 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 msgr2=0x7fd69c1f2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 0x7fd69c1f2e40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd69400a760 tx=0x7fd694001cb0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 shutdown_connections 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd67406b2c0 0x7fd67406d780 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd69c15a7d0 0x7fd69c1f2e40 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd69c0c3c00 0x7fd69c141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 --2- 172.21.15.2:0/1632209139 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd67c004d00 0x7fd69c140fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 >> 172.21.15.2:0/1632209139 conn(0x7fd69c0b0840 msgr2=0x7fd69c06d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:12.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.477+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 shutdown_connections 2024-01-15T17:59:12.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.478+0000 7fd6a1c00640 1 -- 172.21.15.2:0/1632209139 wait complete. 2024-01-15T17:59:12.506 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T17:59:12.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.726+0000 7ff7b5a00640 1 Processor -- start 2024-01-15T17:59:12.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.727+0000 7ff7b5a00640 1 -- start start 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff7b0160e50 0x7ff7b00cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff7b00ce3b0 0x7ff7b015d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b00cb3a0 con 0x7ff7b006d350 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b00cb680 con 0x7ff7b00ce3b0 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b00cb960 con 0x7ff7b0160e50 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff7b00ce3b0 0x7ff7b015d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff7b00ce3b0 0x7ff7b015d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52702/0 (socket says 172.21.15.2:52702) 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 -- 172.21.15.2:0/1795051063 learned_addr learned my addr 172.21.15.2:0/1795051063 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:12.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7a7e00640 1 --2- 172.21.15.2:0/1795051063 >> v2:172.21.15.137:3300/0 conn(0x7ff7b0160e50 0x7ff7b00cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 -- 172.21.15.2:0/1795051063 >> v2:172.21.15.137:3300/0 conn(0x7ff7b0160e50 msgr2=0x7ff7b00cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 --2- 172.21.15.2:0/1795051063 >> v2:172.21.15.137:3300/0 conn(0x7ff7b0160e50 0x7ff7b00cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 -- 172.21.15.2:0/1795051063 >> v1:172.21.15.137:6789/0 conn(0x7ff7b006d350 legacy=0x7ff7b006d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.729+0000 7ff7b5000640 1 -- 172.21.15.2:0/1795051063 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b00cbc40 con 0x7ff7b00ce3b0 2024-01-15T17:59:12.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.730+0000 7ff7a7e00640 1 --2- 172.21.15.2:0/1795051063 >> v2:172.21.15.137:3300/0 conn(0x7ff7b0160e50 0x7ff7b00cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.731+0000 7ff7b5000640 1 --2- 172.21.15.2:0/1795051063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015d960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff7b004f7a0 tx=0x7ff7a802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7e2d2c379a5876 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.731+0000 7ff7af400640 1 -- 172.21.15.2:0/1795051063 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a802ca80 con 0x7ff7b00ce3b0 2024-01-15T17:59:12.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.731+0000 7ff7af400640 1 -- 172.21.15.2:0/1795051063 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a802cbe0 con 0x7ff7b00ce3b0 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.731+0000 7ff7af400640 1 -- 172.21.15.2:0/1795051063 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a80343f0 con 0x7ff7b00ce3b0 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 -- 172.21.15.2:0/1795051063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b00ce3b0 msgr2=0x7ff7b015d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/1795051063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015d960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7ff7b004f7a0 tx=0x7ff7a802bee0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 -- 172.21.15.2:0/1795051063 shutdown_connections 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/1795051063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015d960 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/1795051063 >> v2:172.21.15.137:3300/0 conn(0x7ff7b0160e50 0x7ff7b00cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 -- 172.21.15.2:0/1795051063 >> 172.21.15.2:0/1795051063 conn(0x7ff7b00b0840 msgr2=0x7ff7b00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 -- 172.21.15.2:0/1795051063 shutdown_connections 2024-01-15T17:59:12.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.732+0000 7ff7b5a00640 1 -- 172.21.15.2:0/1795051063 wait complete. 2024-01-15T17:59:12.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.734+0000 7ff7b5a00640 1 Processor -- start 2024-01-15T17:59:12.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.734+0000 7ff7b5a00640 1 -- start start 2024-01-15T17:59:12.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff7b006d350 0x7ff7b0153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 0x7ff7b01460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b015a180 con 0x7ff7b0160e50 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b015d960 con 0x7ff7b00ce3b0 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b015dc40 con 0x7ff7b006d350 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 0x7ff7b01460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 0x7ff7b01460f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52706/0 (socket says 172.21.15.2:52706) 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.735+0000 7ff7b5000640 1 -- 172.21.15.2:0/2129837142 learned_addr learned my addr 172.21.15.2:0/2129837142 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:12.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7afe00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff7b006d350 0x7ff7b0153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7a7e00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7b5000640 1 -- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff7b006d350 msgr2=0x7ff7b0153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7b5000640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff7b006d350 0x7ff7b0153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7b5000640 1 -- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7b00ce3b0 msgr2=0x7ff7b015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7b5000640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7b5000640 1 -- 172.21.15.2:0/2129837142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a8009b30 con 0x7ff7b0160e50 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7afe00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff7b006d350 0x7ff7b0153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7b5000640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 0x7ff7b01460f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8036040 tx=0x7ff7a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.736+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a803e450 con 0x7ff7b0160e50 2024-01-15T17:59:12.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.737+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a803e5d0 con 0x7ff7b0160e50 2024-01-15T17:59:12.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.737+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a803e980 con 0x7ff7b0160e50 2024-01-15T17:59:12.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.737+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b015df20 con 0x7ff7b0160e50 2024-01-15T17:59:12.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.737+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b015ec40 con 0x7ff7b0160e50 2024-01-15T17:59:12.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.738+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b006bfa0 con 0x7ff7b0160e50 2024-01-15T17:59:12.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.739+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff7a802cd50 con 0x7ff7b0160e50 2024-01-15T17:59:12.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.740+0000 7ff7acc00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff78806b2c0 0x7ff78806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:12.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.740+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7a80ac8e0 con 0x7ff7b0160e50 2024-01-15T17:59:12.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.740+0000 7ff7afe00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff78806b2c0 0x7ff78806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:12.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.741+0000 7ff7afe00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff78806b2c0 0x7ff78806d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0007fd0 tx=0x7ff7a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:12.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.744+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7a80776b0 con 0x7ff7b0160e50 2024-01-15T17:59:12.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.965+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7b015a000 con 0x7ff78806b2c0 2024-01-15T17:59:12.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.970+0000 7ff7acc00640 1 -- 172.21.15.2:0/2129837142 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff7a0001360 con 0x7ff78806b2c0 2024-01-15T17:59:12.974 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T17:59:12.974 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T17:59:12.974 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T17:59:12.974 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T17:59:12.974 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T17:59:12.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff78806b2c0 msgr2=0x7ff78806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff78806b2c0 0x7ff78806d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0007fd0 tx=0x7ff7a0009040 comp rx=0 tx=0).stop 2024-01-15T17:59:12.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 msgr2=0x7ff7b01460f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:12.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 0x7ff7b01460f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8036040 tx=0x7ff7a8004a40 comp rx=0 tx=0).stop 2024-01-15T17:59:12.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 shutdown_connections 2024-01-15T17:59:12.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff78806b2c0 0x7ff78806d780 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff7b0160e50 0x7ff7b01460f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff7b00ce3b0 0x7ff7b015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 --2- 172.21.15.2:0/2129837142 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff7b006d350 0x7ff7b0153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:12.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.972+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 >> 172.21.15.2:0/2129837142 conn(0x7ff7b00b0840 msgr2=0x7ff7b006d730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:12.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.973+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 shutdown_connections 2024-01-15T17:59:12.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:12.973+0000 7ff7b5a00640 1 -- 172.21.15.2:0/2129837142 wait complete. 2024-01-15T17:59:12.997 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T17:59:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.220+0000 7fdc94a00640 1 Processor -- start 2024-01-15T17:59:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.221+0000 7fdc94a00640 1 -- start start 2024-01-15T17:59:13.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.222+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900cde90 0x7fdc90162660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.222+0000 7fdc94a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdc900c9210 0x7fdc900c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.222+0000 7fdc94a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc900723f0 con 0x7fdc90070150 2024-01-15T17:59:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.222+0000 7fdc94a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc900c9be0 con 0x7fdc900c9210 2024-01-15T17:59:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.222+0000 7fdc94a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc90162ba0 con 0x7fdc900cde90 2024-01-15T17:59:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc85c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdc90070150 0x7fdc900cad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48872/0 (socket says 172.21.15.2:48872) 2024-01-15T17:59:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc85c00640 1 -- 172.21.15.2:0/246374642 learned_addr learned my addr 172.21.15.2:0/246374642 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8f200640 1 --2- 172.21.15.2:0/246374642 >> v2:172.21.15.2:3300/0 conn(0x7fdc900c9210 0x7fdc900c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8e800640 1 --2- 172.21.15.2:0/246374642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900cde90 0x7fdc90162660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8de00640 1 -- 172.21.15.2:0/246374642 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363993822 0 0) 0x7fdc900723f0 con 0x7fdc90070150 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8de00640 1 -- 172.21.15.2:0/246374642 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc70003520 con 0x7fdc90070150 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8f200640 1 -- 172.21.15.2:0/246374642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900cde90 msgr2=0x7fdc90162660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8f200640 1 --2- 172.21.15.2:0/246374642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900cde90 0x7fdc90162660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8f200640 1 -- 172.21.15.2:0/246374642 >> v1:172.21.15.2:6789/0 conn(0x7fdc90070150 legacy=0x7fdc900cad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.223+0000 7fdc8f200640 1 -- 172.21.15.2:0/246374642 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc900ca8d0 con 0x7fdc900c9210 2024-01-15T17:59:13.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.224+0000 7fdc8f200640 1 --2- 172.21.15.2:0/246374642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900c9210 0x7fdc900c96a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009d90 tx=0x7fdc8802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eec97d982186dd02 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:13.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc8de00640 1 -- 172.21.15.2:0/246374642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8802cbc0 con 0x7fdc900c9210 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc8de00640 1 -- 172.21.15.2:0/246374642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8802cd20 con 0x7fdc900c9210 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc8de00640 1 -- 172.21.15.2:0/246374642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc88034660 con 0x7fdc900c9210 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 -- 172.21.15.2:0/246374642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900c9210 msgr2=0x7fdc900c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 --2- 172.21.15.2:0/246374642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900c9210 0x7fdc900c96a0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009d90 tx=0x7fdc8802be10 comp rx=0 tx=0).stop 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 -- 172.21.15.2:0/246374642 shutdown_connections 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 --2- 172.21.15.2:0/246374642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900c9210 0x7fdc900c96a0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 --2- 172.21.15.2:0/246374642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900cde90 0x7fdc90162660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 -- 172.21.15.2:0/246374642 >> 172.21.15.2:0/246374642 conn(0x7fdc900cebc0 msgr2=0x7fdc900cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.225+0000 7fdc94a00640 1 -- 172.21.15.2:0/246374642 shutdown_connections 2024-01-15T17:59:13.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.226+0000 7fdc94a00640 1 -- 172.21.15.2:0/246374642 wait complete. 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.227+0000 7fdc94a00640 1 Processor -- start 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.227+0000 7fdc94a00640 1 -- start start 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.228+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc90070150 0x7fdc90153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.228+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900c9210 0x7fdc900b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.228+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 0x7fdc900b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.228+0000 7fdc94a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc901538a0 con 0x7fdc900cde90 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.228+0000 7fdc94a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900b2420 con 0x7fdc90070150 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.228+0000 7fdc94a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc900b2700 con 0x7fdc900c9210 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 0x7fdc900b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 0x7fdc900b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55422/0 (socket says 172.21.15.2:55422) 2024-01-15T17:59:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 -- 172.21.15.2:0/891240363 learned_addr learned my addr 172.21.15.2:0/891240363 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc85c00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900c9210 0x7fdc900b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8e800640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc90070150 0x7fdc90153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 -- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900c9210 msgr2=0x7fdc900b1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900c9210 0x7fdc900b1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 -- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc90070150 msgr2=0x7fdc90153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc90070150 0x7fdc90153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 -- 172.21.15.2:0/891240363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc88009a70 con 0x7fdc900cde90 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8e800640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc90070150 0x7fdc90153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.229+0000 7fdc8f200640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 0x7fdc900b36d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009d90 tx=0x7fdc88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.230+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8803e610 con 0x7fdc900cde90 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.230+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8802ce90 con 0x7fdc900cde90 2024-01-15T17:59:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.230+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc8803e9f0 con 0x7fdc900cde90 2024-01-15T17:59:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.230+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc900b29e0 con 0x7fdc900cde90 2024-01-15T17:59:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.230+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc90154250 con 0x7fdc900cde90 2024-01-15T17:59:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.231+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc90074770 con 0x7fdc900cde90 2024-01-15T17:59:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.232+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdc8803b050 con 0x7fdc900cde90 2024-01-15T17:59:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.233+0000 7fdc87400640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc6806b2c0 0x7fdc6806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.233+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc880ac390 con 0x7fdc900cde90 2024-01-15T17:59:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.234+0000 7fdc8e800640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc6806b2c0 0x7fdc6806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.235+0000 7fdc8e800640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc6806b2c0 0x7fdc6806d780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdc80006fd0 tx=0x7fdc80008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:13.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.238+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc880771f0 con 0x7fdc900cde90 2024-01-15T17:59:13.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.435+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdc9006e220 con 0x7fdc900cde90 2024-01-15T17:59:13.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.436+0000 7fdc87400640 1 -- 172.21.15.2:0/891240363 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdc88009450 con 0x7fdc900cde90 2024-01-15T17:59:13.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc6806b2c0 msgr2=0x7fdc6806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc6806b2c0 0x7fdc6806d780 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdc80006fd0 tx=0x7fdc80008040 comp rx=0 tx=0).stop 2024-01-15T17:59:13.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 msgr2=0x7fdc900b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 0x7fdc900b36d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fdc88009d90 tx=0x7fdc88004a40 comp rx=0 tx=0).stop 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 shutdown_connections 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc6806b2c0 0x7fdc6806d780 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc900cde90 0x7fdc900b36d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc900c9210 0x7fdc900b1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 --2- 172.21.15.2:0/891240363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc90070150 0x7fdc90153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 >> 172.21.15.2:0/891240363 conn(0x7fdc900cebc0 msgr2=0x7fdc900ad1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 shutdown_connections 2024-01-15T17:59:13.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.439+0000 7fdc94a00640 1 -- 172.21.15.2:0/891240363 wait complete. 2024-01-15T17:59:13.461 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T17:59:13.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.673+0000 7f2cc2600640 1 Processor -- start 2024-01-15T17:59:13.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.674+0000 7f2cc2600640 1 -- start start 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.675+0000 7f2cc2600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2cbc143d20 0x7f2cbc15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.675+0000 7f2cc2600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.675+0000 7f2cc2600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2cbc1447d0 0x7f2cbc0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.675+0000 7f2cc2600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cbc071430 con 0x7f2cbc1447d0 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc2600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cbc144100 con 0x7f2cbc143d20 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc2600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc071090 con 0x7f2cbc0720d0 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2cbc1447d0 0x7f2cbc0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2cbc1447d0 0x7f2cbc0cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55434/0 (socket says 172.21.15.2:55434) 2024-01-15T17:59:13.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 -- 172.21.15.2:0/1781488901 learned_addr learned my addr 172.21.15.2:0/1781488901 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1200640 1 --2- 172.21.15.2:0/1781488901 >> v2:172.21.15.137:3300/0 conn(0x7f2cbc143d20 0x7f2cbc15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cb3c00640 1 --2- 172.21.15.2:0/1781488901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 -- 172.21.15.2:0/1781488901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 msgr2=0x7f2cbc15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 --2- 172.21.15.2:0/1781488901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 -- 172.21.15.2:0/1781488901 >> v2:172.21.15.137:3300/0 conn(0x7f2cbc143d20 msgr2=0x7f2cbc15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 --2- 172.21.15.2:0/1781488901 >> v2:172.21.15.137:3300/0 conn(0x7f2cbc143d20 0x7f2cbc15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.676+0000 7f2cc1c00640 1 -- 172.21.15.2:0/1781488901 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc144560 con 0x7f2cbc1447d0 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.678+0000 7f2cc1c00640 1 --2- 172.21.15.2:0/1781488901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc0cd330 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc06e600 tx=0x7f2cb402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41e0528cd54f355 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.678+0000 7f2cbbe00640 1 -- 172.21.15.2:0/1781488901 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb402c9e0 con 0x7f2cbc1447d0 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.678+0000 7f2cbbe00640 1 -- 172.21.15.2:0/1781488901 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb402cb40 con 0x7f2cbc1447d0 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.678+0000 7f2cbbe00640 1 -- 172.21.15.2:0/1781488901 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40343f0 con 0x7f2cbc1447d0 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 -- 172.21.15.2:0/1781488901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc1447d0 msgr2=0x7f2cbc0cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 --2- 172.21.15.2:0/1781488901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc0cd330 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc06e600 tx=0x7f2cb402be40 comp rx=0 tx=0).stop 2024-01-15T17:59:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 -- 172.21.15.2:0/1781488901 shutdown_connections 2024-01-15T17:59:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 --2- 172.21.15.2:0/1781488901 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc0cd330 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 --2- 172.21.15.2:0/1781488901 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 --2- 172.21.15.2:0/1781488901 >> v2:172.21.15.137:3300/0 conn(0x7f2cbc143d20 0x7f2cbc15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 -- 172.21.15.2:0/1781488901 >> 172.21.15.2:0/1781488901 conn(0x7f2cbc0b0840 msgr2=0x7f2cbc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 -- 172.21.15.2:0/1781488901 shutdown_connections 2024-01-15T17:59:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.679+0000 7f2cc2600640 1 -- 172.21.15.2:0/1781488901 wait complete. 2024-01-15T17:59:13.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.681+0000 7f2cc2600640 1 Processor -- start 2024-01-15T17:59:13.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.681+0000 7f2cc2600640 1 -- start start 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc2600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc13a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 0x7f2cbc13de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc2600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc15a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc2600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc13e350 con 0x7f2cbc143d20 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc2600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc13e580 con 0x7f2cbc1447d0 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc2600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cbc13e860 con 0x7f2cbc0720d0 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cb3c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 0x7f2cbc13de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cb3c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 0x7f2cbc13de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55448/0 (socket says 172.21.15.2:55448) 2024-01-15T17:59:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cb3c00640 1 -- 172.21.15.2:0/3206941967 learned_addr learned my addr 172.21.15.2:0/3206941967 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc1c00640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc15a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.683+0000 7f2cc1200640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc13a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb3c00640 1 -- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 msgr2=0x7f2cbc13a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb3c00640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc13a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb3c00640 1 -- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cbc1447d0 msgr2=0x7f2cbc15a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb3c00640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc15a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb3c00640 1 -- 172.21.15.2:0/3206941967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb4009b30 con 0x7f2cbc143d20 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb3c00640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 0x7f2cbc13de10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400d800 tx=0x7f2ca400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca40096a0 con 0x7f2cbc143d20 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca4009800 con 0x7f2cbc143d20 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cbc13eb40 con 0x7f2cbc143d20 2024-01-15T17:59:13.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.684+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca4009990 con 0x7f2cbc143d20 2024-01-15T17:59:13.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.685+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cbc13f100 con 0x7f2cbc143d20 2024-01-15T17:59:13.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.686+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cbc074770 con 0x7f2cbc143d20 2024-01-15T17:59:13.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.687+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ca4009af0 con 0x7f2cbc143d20 2024-01-15T17:59:13.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.687+0000 7f2cb9600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c9406b2f0 0x7f2c9406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:13.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.688+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ca4089fe0 con 0x7f2cbc143d20 2024-01-15T17:59:13.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.688+0000 7f2cc1200640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c9406b2f0 0x7f2c9406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:13.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.690+0000 7f2cc1200640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c9406b2f0 0x7f2c9406d7b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2cac006fd0 tx=0x7f2cac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.692+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ca4054e60 con 0x7f2cbc143d20 2024-01-15T17:59:13.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:13.913+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cbc15fa00 con 0x7f2c9406b2f0 2024-01-15T17:59:13.927 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:59:13.924+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T17:59:13.927 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:59:13.924+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T17:59:13.927 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:59:13.924+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T17:59:13.928 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:59:13.924+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T17:59:13.928 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T17:59:13.924+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T17:59:14.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.002+0000 7f2cb9600640 1 -- 172.21.15.2:0/3206941967 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2cac00f860 con 0x7f2c9406b2f0 2024-01-15T17:59:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c9406b2f0 msgr2=0x7f2c9406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c9406b2f0 0x7f2c9406d7b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2cac006fd0 tx=0x7f2cac008040 comp rx=0 tx=0).stop 2024-01-15T17:59:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 msgr2=0x7f2cbc13de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 0x7f2cbc13de10 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f2ca400d800 tx=0x7f2ca400dcc0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 shutdown_connections 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c9406b2f0 0x7f2c9406d7b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2cbc1447d0 0x7f2cbc15a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2cbc143d20 0x7f2cbc13de10 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 --2- 172.21.15.2:0/3206941967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2cbc0720d0 0x7f2cbc13a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.005+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 >> 172.21.15.2:0/3206941967 conn(0x7f2cbc0b0840 msgr2=0x7f2cbc0afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.006+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 shutdown_connections 2024-01-15T17:59:14.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.006+0000 7f2cc2600640 1 -- 172.21.15.2:0/3206941967 wait complete. 2024-01-15T17:59:14.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T17:59:14.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.252+0000 7fbbb3e00640 1 Processor -- start 2024-01-15T17:59:14.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.252+0000 7fbbb3e00640 1 -- start start 2024-01-15T17:59:14.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.254+0000 7fbbb3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbbb415d270 0x7fbbb40b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.254+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 0x7fbbb406cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.254+0000 7fbbb3e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbb406c5e0 con 0x7fbbb4161890 2024-01-15T17:59:14.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.254+0000 7fbbb3e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbb406c8c0 con 0x7fbbb415d270 2024-01-15T17:59:14.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.254+0000 7fbbb3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb4071c00 con 0x7fbbb4161370 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb2000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbbb4161890 0x7fbbb4160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48884/0 (socket says 172.21.15.2:48884) 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb2000640 1 -- 172.21.15.2:0/2414085778 learned_addr learned my addr 172.21.15.2:0/2414085778 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb2a00640 1 --2- 172.21.15.2:0/2414085778 >> v2:172.21.15.137:3300/0 conn(0x7fbbb415d270 0x7fbbb40b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb3400640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 0x7fbbb406cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502208931 0 0) 0x7fbbb406c5e0 con 0x7fbbb4161890 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb94003520 con 0x7fbbb4161890 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.255+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3728145665 0 0) 0x7fbb94003520 con 0x7fbbb4161890 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 msgr2=0x7fbbb406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 0x7fbbb406cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 >> v2:172.21.15.137:3300/0 conn(0x7fbbb415d270 msgr2=0x7fbbb40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 --2- 172.21.15.2:0/2414085778 >> v2:172.21.15.137:3300/0 conn(0x7fbbb415d270 0x7fbbb40b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb4071ee0 con 0x7fbbb4161890 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fbb9c001670 con 0x7fbbb4161890 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 >> v1:172.21.15.2:6789/0 conn(0x7fbbb4161890 legacy=0x7fbbb4160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb94003970 0x7fbb94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb940047c0 0x7fbb94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 0x7fbb94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb406c8c0 con 0x7fbb940047c0 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb4071c00 con 0x7fbb94003970 2024-01-15T17:59:14.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9400bab0 con 0x7fbbb4161370 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb2000640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb940047c0 0x7fbb94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb2a00640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb94003970 0x7fbb94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.256+0000 7fbbb3400640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 0x7fbb94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.257+0000 7fbbb2000640 1 -- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 msgr2=0x7fbb94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.257+0000 7fbbb2000640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb4161370 0x7fbb94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.257+0000 7fbbb2000640 1 -- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb94003970 msgr2=0x7fbb94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.257+0000 7fbbb2000640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb94003970 0x7fbb94003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.257+0000 7fbbb2000640 1 -- 172.21.15.2:0/2414085778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb94003520 con 0x7fbb940047c0 2024-01-15T17:59:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.258+0000 7fbbb2000640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb940047c0 0x7fbb94004c40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00c0e0 tx=0x7fbb940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c15444d201adc5e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.258+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c005ad0 con 0x7fbb940047c0 2024-01-15T17:59:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.258+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9c005c30 con 0x7fbb940047c0 2024-01-15T17:59:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.258+0000 7fbbb1600640 1 -- 172.21.15.2:0/2414085778 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb9c02c7d0 con 0x7fbb940047c0 2024-01-15T17:59:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3e00640 1 -- 172.21.15.2:0/2414085778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb940047c0 msgr2=0x7fbb94004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/2414085778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb940047c0 0x7fbb94004c40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c00c0e0 tx=0x7fbb940057f0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3400640 1 -- 172.21.15.2:0/2414085778 reap_dead start 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3e00640 1 -- 172.21.15.2:0/2414085778 shutdown_connections 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3e00640 1 -- 172.21.15.2:0/2414085778 >> 172.21.15.2:0/2414085778 conn(0x7fbbb40b0840 msgr2=0x7fbbb40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3e00640 1 -- 172.21.15.2:0/2414085778 shutdown_connections 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.259+0000 7fbbb3e00640 1 -- 172.21.15.2:0/2414085778 wait complete. 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.261+0000 7fbbb3e00640 1 Processor -- start 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.261+0000 7fbbb3e00640 1 -- start start 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb940047c0 0x7fbbb41546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb40cdb90 0x7fbbb40ce010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 0x7fbbb415dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb415e010 con 0x7fbbb415d650 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41f3930 con 0x7fbb940047c0 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb41f2d20 con 0x7fbbb40cdb90 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 0x7fbbb415dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 0x7fbbb415dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55468/0 (socket says 172.21.15.2:55468) 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.262+0000 7fbbb3400640 1 -- 172.21.15.2:0/3346865131 learned_addr learned my addr 172.21.15.2:0/3346865131 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb2a00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb940047c0 0x7fbbb41546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3400640 1 -- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb40cdb90 msgr2=0x7fbbb40ce010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3400640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb40cdb90 0x7fbbb40ce010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3400640 1 -- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb940047c0 msgr2=0x7fbbb41546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3400640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb940047c0 0x7fbbb41546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3400640 1 -- 172.21.15.2:0/3346865131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb41f3000 con 0x7fbbb415d650 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb2a00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb940047c0 0x7fbbb41546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3400640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 0x7fbbb415dad0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbbb406fef0 tx=0x7fbbac0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac00e070 con 0x7fbbb415d650 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbac011ed0 con 0x7fbbb415d650 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb41f32e0 con 0x7fbbb415d650 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.263+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbac003990 con 0x7fbbb415d650 2024-01-15T17:59:14.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.264+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb41f29d0 con 0x7fbbb415d650 2024-01-15T17:59:14.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.265+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb4074770 con 0x7fbbb415d650 2024-01-15T17:59:14.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.266+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbbac003b10 con 0x7fbbb415d650 2024-01-15T17:59:14.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.267+0000 7fbbab400640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb8c06b2f0 0x7fbb8c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:14.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.267+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbac08a280 con 0x7fbbb415d650 2024-01-15T17:59:14.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.268+0000 7fbbb2a00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb8c06b2f0 0x7fbb8c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:14.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.269+0000 7fbbb2a00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb8c06b2f0 0x7fbb8c06d7b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009a20 tx=0x7fbba4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:14.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.272+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbbac055050 con 0x7fbbb415d650 2024-01-15T17:59:14.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.504+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbb406cd50 con 0x7fbb8c06b2f0 2024-01-15T17:59:14.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.518+0000 7fbbab400640 1 -- 172.21.15.2:0/3346865131 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fbba4013960 con 0x7fbb8c06b2f0 2024-01-15T17:59:14.522 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f 2024-01-15T17:59:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.520+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb8c06b2f0 msgr2=0x7fbb8c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.520+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb8c06b2f0 0x7fbb8c06d7b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbba4009a20 tx=0x7fbba4008040 comp rx=0 tx=0).stop 2024-01-15T17:59:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.520+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 msgr2=0x7fbbb415dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.520+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 0x7fbbb415dad0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbbb406fef0 tx=0x7fbbac0115a0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 shutdown_connections 2024-01-15T17:59:14.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb8c06b2f0 0x7fbb8c06d7b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbbb415d650 0x7fbbb415dad0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbbb40cdb90 0x7fbbb40ce010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 --2- 172.21.15.2:0/3346865131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb940047c0 0x7fbbb41546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 >> 172.21.15.2:0/3346865131 conn(0x7fbbb40b0840 msgr2=0x7fbbb415f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 shutdown_connections 2024-01-15T17:59:14.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:14.521+0000 7fbbb3e00640 1 -- 172.21.15.2:0/3346865131 wait complete. 2024-01-15T17:59:14.546 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f 2024-01-15T17:59:14.546 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:59:14.546 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:59:14.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:14.604 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T17:59:29.137 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:59:29.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:59:31.678 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T17:59:31.678 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T17:59:31.678 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.9927 s, 63.2 MB/s 2024-01-15T17:59:31.680 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:59:31.680 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:59:31.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:31.736 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T17:59:46.244 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T17:59:46.245 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T17:59:46.245 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.4487 s, 74.3 MB/s 2024-01-15T17:59:46.246 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T17:59:46.247 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T17:59:46.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T17:59:46.301 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T17:59:59.275 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:59:59.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T17:59:59.656 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T17:59:59.656 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T17:59:59.656 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.3015 s, 80.7 MB/s 2024-01-15T17:59:59.658 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T17:59:59.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.872+0000 7f7be4600640 1 Processor -- start 2024-01-15T17:59:59.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.873+0000 7f7be4600640 1 -- start start 2024-01-15T17:59:59.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.874+0000 7f7be4600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7bdc144c00 0x7f7bdc0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.874+0000 7f7be4600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7bdc0cdd70 0x7f7bdc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.874+0000 7f7be4600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc15c450 0x7f7bdc06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.874+0000 7f7be4600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc0ce230 con 0x7f7bdc144c00 2024-01-15T17:59:59.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.874+0000 7f7be4600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc0ce510 con 0x7f7bdc0cdd70 2024-01-15T17:59:59.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.874+0000 7f7be4600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc0ce7f0 con 0x7f7bdc15c450 2024-01-15T17:59:59.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7bdc144c00 0x7f7bdc0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7bdc144c00 0x7f7bdc0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54834/0 (socket says 172.21.15.2:54834) 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 -- 172.21.15.2:0/1766929244 learned_addr learned my addr 172.21.15.2:0/1766929244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be2600640 1 --2- 172.21.15.2:0/1766929244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc15c450 0x7f7bdc06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1200640 1 --2- 172.21.15.2:0/1766929244 >> v2:172.21.15.137:3300/0 conn(0x7f7bdc0cdd70 0x7f7bdc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 -- 172.21.15.2:0/1766929244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc15c450 msgr2=0x7f7bdc06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 --2- 172.21.15.2:0/1766929244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc15c450 0x7f7bdc06ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 -- 172.21.15.2:0/1766929244 >> v2:172.21.15.137:3300/0 conn(0x7f7bdc0cdd70 msgr2=0x7f7bdc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 --2- 172.21.15.2:0/1766929244 >> v2:172.21.15.137:3300/0 conn(0x7f7bdc0cdd70 0x7f7bdc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1c00640 1 -- 172.21.15.2:0/1766929244 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc0705f0 con 0x7f7bdc144c00 2024-01-15T17:59:59.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.875+0000 7f7be1200640 1 --2- 172.21.15.2:0/1766929244 >> v2:172.21.15.137:3300/0 conn(0x7f7bdc0cdd70 0x7f7bdc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T17:59:59.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.877+0000 7f7be1c00640 1 --2- 172.21.15.2:0/1766929244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc144c00 0x7f7bdc0cd830 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc06f9f0 tx=0x7f7bcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6617514af15891a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.877+0000 7f7bdbe00640 1 -- 172.21.15.2:0/1766929244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc031870 con 0x7f7bdc144c00 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.877+0000 7f7bdbe00640 1 -- 172.21.15.2:0/1766929244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc0319d0 con 0x7f7bdc144c00 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.877+0000 7f7bdbe00640 1 -- 172.21.15.2:0/1766929244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc031d60 con 0x7f7bdc144c00 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.877+0000 7f7be4600640 1 -- 172.21.15.2:0/1766929244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc144c00 msgr2=0x7f7bdc0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.877+0000 7f7be4600640 1 --2- 172.21.15.2:0/1766929244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc144c00 0x7f7bdc0cd830 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc06f9f0 tx=0x7f7bcc02b910 comp rx=0 tx=0).stop 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 -- 172.21.15.2:0/1766929244 shutdown_connections 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 --2- 172.21.15.2:0/1766929244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc15c450 0x7f7bdc06ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 --2- 172.21.15.2:0/1766929244 >> v2:172.21.15.137:3300/0 conn(0x7f7bdc0cdd70 0x7f7bdc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 --2- 172.21.15.2:0/1766929244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc144c00 0x7f7bdc0cd830 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 -- 172.21.15.2:0/1766929244 >> 172.21.15.2:0/1766929244 conn(0x7f7bdc0b45f0 msgr2=0x7f7bdc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 -- 172.21.15.2:0/1766929244 shutdown_connections 2024-01-15T17:59:59.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.878+0000 7f7be4600640 1 -- 172.21.15.2:0/1766929244 wait complete. 2024-01-15T17:59:59.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.880+0000 7f7be4600640 1 Processor -- start 2024-01-15T17:59:59.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.880+0000 7f7be4600640 1 -- start start 2024-01-15T17:59:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be4600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bdc0cdd70 0x7f7bdc0b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be4600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc144c00 0x7f7bdc1435b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be4600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 0x7f7bdc143af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be4600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc159060 con 0x7f7bdc15c450 2024-01-15T17:59:59.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be4600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc144030 con 0x7f7bdc0cdd70 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be4600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1583e0 con 0x7f7bdc144c00 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 0x7f7bdc143af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 0x7f7bdc143af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54842/0 (socket says 172.21.15.2:54842) 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 -- 172.21.15.2:0/279112681 learned_addr learned my addr 172.21.15.2:0/279112681 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be1c00640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bdc0cdd70 0x7f7bdc0b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be1200640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc144c00 0x7f7bdc1435b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 -- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc144c00 msgr2=0x7f7bdc1435b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc144c00 0x7f7bdc1435b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 -- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bdc0cdd70 msgr2=0x7f7bdc0b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bdc0cdd70 0x7f7bdc0b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T17:59:59.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.881+0000 7f7be2600640 1 -- 172.21.15.2:0/279112681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc0094d0 con 0x7f7bdc15c450 2024-01-15T17:59:59.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.882+0000 7f7be2600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 0x7f7bdc143af0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009fd0 tx=0x7f7bd400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:59.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.882+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4014420 con 0x7f7bdc15c450 2024-01-15T17:59:59.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.882+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd4014580 con 0x7f7bdc15c450 2024-01-15T17:59:59.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.882+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd4014930 con 0x7f7bdc15c450 2024-01-15T17:59:59.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.882+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc1586c0 con 0x7f7bdc15c450 2024-01-15T17:59:59.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.882+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc158c80 con 0x7f7bdc15c450 2024-01-15T17:59:59.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.884+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc1591a0 con 0x7f7bdc15c450 2024-01-15T17:59:59.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.885+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f7bd4014a90 con 0x7f7bdc15c450 2024-01-15T17:59:59.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.886+0000 7f7bd9600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bb406b350 0x7f7bb406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T17:59:59.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.886+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7bd408a4c0 con 0x7f7bdc15c450 2024-01-15T17:59:59.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.886+0000 7f7be1c00640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bb406b350 0x7f7bb406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T17:59:59.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.887+0000 7f7be1c00640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bb406b350 0x7f7bb406d810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc036000 tx=0x7f7bcc03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T17:59:59.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T17:59:59.891+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bdc1591a0 con 0x7f7bdc15c450 2024-01-15T18:00:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.090+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f7bdc1591a0 con 0x7f7bdc15c450 2024-01-15T18:00:00.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.097+0000 7f7bd9600640 1 -- 172.21.15.2:0/279112681 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7f7bdc1591a0 con 0x7f7bdc15c450 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bb406b350 msgr2=0x7f7bb406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bb406b350 0x7f7bb406d810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc036000 tx=0x7f7bcc03c040 comp rx=0 tx=0).stop 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 msgr2=0x7f7bdc143af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 0x7f7bdc143af0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f7bd4009fd0 tx=0x7f7bd400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 shutdown_connections 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bb406b350 0x7f7bb406d810 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bdc15c450 0x7f7bdc143af0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bdc144c00 0x7f7bdc1435b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 --2- 172.21.15.2:0/279112681 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bdc0cdd70 0x7f7bdc0b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.100+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 >> 172.21.15.2:0/279112681 conn(0x7f7bdc0b45f0 msgr2=0x7f7bdc070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.101+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 shutdown_connections 2024-01-15T18:00:00.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.101+0000 7f7be4600640 1 -- 172.21.15.2:0/279112681 wait complete. 2024-01-15T18:00:00.124 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:00:00.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.376+0000 7f772f200640 1 Processor -- start 2024-01-15T18:00:00.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.376+0000 7f772f200640 1 -- start start 2024-01-15T18:00:00.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77280720d0 0x7f772806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772f200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7728071090 con 0x7f7728143d20 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7728071430 con 0x7f77280720d0 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7728144100 con 0x7f772815f260 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772e800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77280720d0 0x7f772806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772806c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:49456/0 (socket says 172.21.15.2:49456) 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.378+0000 7f772d400640 1 -- 172.21.15.2:0/2464463453 learned_addr learned my addr 172.21.15.2:0/2464463453 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:00.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.379+0000 7f772d400640 1 -- 172.21.15.2:0/2464463453 >> v2:172.21.15.137:3300/0 conn(0x7f77280720d0 msgr2=0x7f772806c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.379+0000 7f772d400640 1 --2- 172.21.15.2:0/2464463453 >> v2:172.21.15.137:3300/0 conn(0x7f77280720d0 0x7f772806c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.379+0000 7f772d400640 1 -- 172.21.15.2:0/2464463453 >> v1:172.21.15.137:6789/0 conn(0x7f7728143d20 legacy=0x7f772815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.379+0000 7f772d400640 1 -- 172.21.15.2:0/2464463453 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7728144560 con 0x7f772815f260 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.380+0000 7f772d400640 1 --2- 172.21.15.2:0/2464463453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772806c150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7710009870 tx=0x7f771002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44b953f433896d9f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.381+0000 7f772ca00640 1 -- 172.21.15.2:0/2464463453 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77100317c0 con 0x7f772815f260 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.381+0000 7f772ca00640 1 -- 172.21.15.2:0/2464463453 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7710031920 con 0x7f772815f260 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.381+0000 7f772ca00640 1 -- 172.21.15.2:0/2464463453 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710031cb0 con 0x7f772815f260 2024-01-15T18:00:00.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 -- 172.21.15.2:0/2464463453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 msgr2=0x7f772806c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 --2- 172.21.15.2:0/2464463453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772806c150 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f7710009870 tx=0x7f771002b860 comp rx=0 tx=0).stop 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 -- 172.21.15.2:0/2464463453 shutdown_connections 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 --2- 172.21.15.2:0/2464463453 >> v2:172.21.15.137:3300/0 conn(0x7f77280720d0 0x7f772806c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 --2- 172.21.15.2:0/2464463453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772806c150 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 -- 172.21.15.2:0/2464463453 >> 172.21.15.2:0/2464463453 conn(0x7f77280b0840 msgr2=0x7f77280b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 -- 172.21.15.2:0/2464463453 shutdown_connections 2024-01-15T18:00:00.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.382+0000 7f772f200640 1 -- 172.21.15.2:0/2464463453 wait complete. 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.384+0000 7f772f200640 1 Processor -- start 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.385+0000 7f772f200640 1 -- start start 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 0x7f7728158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7728143d20 0x7f772815d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7728132230 con 0x7f77280720d0 2024-01-15T18:00:00.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280a8590 con 0x7f7728143d20 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77281324d0 con 0x7f772815f260 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 0x7f7728158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 0x7f7728158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54856/0 (socket says 172.21.15.2:54856) 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.386+0000 7f772de00640 1 -- 172.21.15.2:0/2475429904 learned_addr learned my addr 172.21.15.2:0/2475429904 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772e800640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772d400640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7728143d20 0x7f772815d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772de00640 1 -- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 msgr2=0x7f772815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772de00640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772815dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772de00640 1 -- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7728143d20 msgr2=0x7f772815d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772de00640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7728143d20 0x7f772815d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772de00640 1 -- 172.21.15.2:0/2475429904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77100094d0 con 0x7f77280720d0 2024-01-15T18:00:00.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f772de00640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 0x7f7728158f40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f771800c450 tx=0x7f771800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:00.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.387+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77180052d0 con 0x7f77280720d0 2024-01-15T18:00:00.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.388+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7718005430 con 0x7f77280720d0 2024-01-15T18:00:00.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.388+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718012870 con 0x7f77280720d0 2024-01-15T18:00:00.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.388+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77281327b0 con 0x7f77280720d0 2024-01-15T18:00:00.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.388+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77281f1960 con 0x7f77280720d0 2024-01-15T18:00:00.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.389+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7728074770 con 0x7f77280720d0 2024-01-15T18:00:00.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.390+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f77180129d0 con 0x7f77280720d0 2024-01-15T18:00:00.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.391+0000 7f7726000640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f770006b350 0x7f770006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.391+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77180896a0 con 0x7f77280720d0 2024-01-15T18:00:00.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.391+0000 7f772d400640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f770006b350 0x7f770006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.393+0000 7f772d400640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f770006b350 0x7f770006d810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7710036000 tx=0x7f771003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:00.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.395+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77180545b0 con 0x7f77280720d0 2024-01-15T18:00:00.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.695+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f772815fa00 con 0x7f770006b350 2024-01-15T18:00:00.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.726+0000 7f7726000640 1 -- 172.21.15.2:0/2475429904 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7710031b10 con 0x7f770006b350 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f770006b350 msgr2=0x7f770006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f770006b350 0x7f770006d810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7710036000 tx=0x7f771003d040 comp rx=0 tx=0).stop 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 msgr2=0x7f7728158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 0x7f7728158f40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f771800c450 tx=0x7f771800c910 comp rx=0 tx=0).stop 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 shutdown_connections 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f770006b350 0x7f770006d810 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772815f260 0x7f772815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7728143d20 0x7f772815d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 --2- 172.21.15.2:0/2475429904 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f77280720d0 0x7f7728158f40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 >> 172.21.15.2:0/2475429904 conn(0x7f77280b0840 msgr2=0x7f77280ae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:00.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 shutdown_connections 2024-01-15T18:00:00.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.729+0000 7f772f200640 1 -- 172.21.15.2:0/2475429904 wait complete. 2024-01-15T18:00:00.753 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:00:00.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.967+0000 7f51d7e00640 1 Processor -- start 2024-01-15T18:00:00.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.967+0000 7f51d7e00640 1 -- start start 2024-01-15T18:00:00.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.969+0000 7f51d7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f51d8143d20 0x7f51d815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.969+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.969+0000 7f51d7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51d81447d0 0x7f51d80cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.969+0000 7f51d7e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d8071430 con 0x7f51d81447d0 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.969+0000 7f51d7e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d8144100 con 0x7f51d8143d20 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.969+0000 7f51d7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d8071090 con 0x7f51d80720d0 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.970+0000 7f51d7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51d81447d0 0x7f51d80cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.970+0000 7f51d7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51d81447d0 0x7f51d80cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54866/0 (socket says 172.21.15.2:54866) 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.970+0000 7f51d7400640 1 -- 172.21.15.2:0/2894543203 learned_addr learned my addr 172.21.15.2:0/2894543203 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.970+0000 7f51d6a00640 1 --2- 172.21.15.2:0/2894543203 >> v2:172.21.15.137:3300/0 conn(0x7f51d8143d20 0x7f51d815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.970+0000 7f51cfe00640 1 --2- 172.21.15.2:0/2894543203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.971+0000 7f51d7400640 1 -- 172.21.15.2:0/2894543203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 msgr2=0x7f51d815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:00.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.971+0000 7f51d7400640 1 --2- 172.21.15.2:0/2894543203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.971+0000 7f51d7400640 1 -- 172.21.15.2:0/2894543203 >> v2:172.21.15.137:3300/0 conn(0x7f51d8143d20 msgr2=0x7f51d815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:00.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.971+0000 7f51d7400640 1 --2- 172.21.15.2:0/2894543203 >> v2:172.21.15.137:3300/0 conn(0x7f51d8143d20 0x7f51d815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.971+0000 7f51d7400640 1 -- 172.21.15.2:0/2894543203 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d8144560 con 0x7f51d81447d0 2024-01-15T18:00:00.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.972+0000 7f51d7400640 1 --2- 172.21.15.2:0/2894543203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d80cd330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f51d806e600 tx=0x7f51d002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbc9db38138a15a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:00.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.973+0000 7f51d6000640 1 -- 172.21.15.2:0/2894543203 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d002c9e0 con 0x7f51d81447d0 2024-01-15T18:00:00.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.973+0000 7f51d6000640 1 -- 172.21.15.2:0/2894543203 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d002cb40 con 0x7f51d81447d0 2024-01-15T18:00:00.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.973+0000 7f51d6000640 1 -- 172.21.15.2:0/2894543203 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d00343f0 con 0x7f51d81447d0 2024-01-15T18:00:00.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 -- 172.21.15.2:0/2894543203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 msgr2=0x7f51d80cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2894543203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d80cd330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f51d806e600 tx=0x7f51d002be40 comp rx=0 tx=0).stop 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 -- 172.21.15.2:0/2894543203 shutdown_connections 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2894543203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d80cd330 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2894543203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2894543203 >> v2:172.21.15.137:3300/0 conn(0x7f51d8143d20 0x7f51d815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 -- 172.21.15.2:0/2894543203 >> 172.21.15.2:0/2894543203 conn(0x7f51d80b0840 msgr2=0x7f51d80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 -- 172.21.15.2:0/2894543203 shutdown_connections 2024-01-15T18:00:00.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.974+0000 7f51d7e00640 1 -- 172.21.15.2:0/2894543203 wait complete. 2024-01-15T18:00:00.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.976+0000 7f51d7e00640 1 Processor -- start 2024-01-15T18:00:00.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.976+0000 7f51d7e00640 1 -- start start 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d80720d0 0x7f51d815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d8143d20 0x7f51d813e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d815cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d813cbf0 con 0x7f51d81447d0 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d813ced0 con 0x7f51d80720d0 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d813d1b0 con 0x7f51d8143d20 2024-01-15T18:00:00.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d815cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d815cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54878/0 (socket says 172.21.15.2:54878) 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.977+0000 7f51d7400640 1 -- 172.21.15.2:0/3050657962 learned_addr learned my addr 172.21.15.2:0/3050657962 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51cfe00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d8143d20 0x7f51d813e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d6a00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d80720d0 0x7f51d815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d7400640 1 -- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d8143d20 msgr2=0x7f51d813e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d7400640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d8143d20 0x7f51d813e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d7400640 1 -- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d80720d0 msgr2=0x7f51d815a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d7400640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d80720d0 0x7f51d815a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d7400640 1 -- 172.21.15.2:0/3050657962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0009b30 con 0x7f51d81447d0 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51cfe00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d8143d20 0x7f51d813e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:00.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d6a00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d80720d0 0x7f51d815a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:00.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51d7400640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d815cf50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f51d0036040 tx=0x7f51d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:00.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.978+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0040670 con 0x7f51d81447d0 2024-01-15T18:00:00.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.979+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d00407d0 con 0x7f51d81447d0 2024-01-15T18:00:00.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.979+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d813d490 con 0x7f51d81447d0 2024-01-15T18:00:00.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.979+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0040bc0 con 0x7f51d81447d0 2024-01-15T18:00:00.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.979+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d815ac10 con 0x7f51d81447d0 2024-01-15T18:00:00.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.980+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d81f0e20 con 0x7f51d81447d0 2024-01-15T18:00:00.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.981+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f51d0040d20 con 0x7f51d81447d0 2024-01-15T18:00:00.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.982+0000 7f51cea00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006b350 0x7f51b006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:00.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.982+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51d00acd60 con 0x7f51d81447d0 2024-01-15T18:00:00.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.982+0000 7f51d6a00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006b350 0x7f51b006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:00.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.984+0000 7f51d6a00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006b350 0x7f51b006d810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009a20 tx=0x7f51c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:00.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:00.986+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51d81f0e20 con 0x7f51d81447d0 2024-01-15T18:00:01.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.209+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f51d813d040 con 0x7f51b006b350 2024-01-15T18:00:01.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.221+0000 7f51cea00640 1 -- 172.21.15.2:0/3050657962 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f51d813d040 con 0x7f51b006b350 2024-01-15T18:00:01.224 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f 2024-01-15T18:00:01.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.222+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006b350 msgr2=0x7f51b006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.222+0000 7f51d7e00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006b350 0x7f51b006d810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009a20 tx=0x7f51c8008040 comp rx=0 tx=0).stop 2024-01-15T18:00:01.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.222+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 msgr2=0x7f51d815cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.222+0000 7f51d7e00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d815cf50 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f51d0036040 tx=0x7f51d0004a40 comp rx=0 tx=0).stop 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 shutdown_connections 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006b350 0x7f51b006d810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d81447d0 0x7f51d815cf50 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d8143d20 0x7f51d813e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 --2- 172.21.15.2:0/3050657962 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d80720d0 0x7f51d815a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 >> 172.21.15.2:0/3050657962 conn(0x7f51d80b0840 msgr2=0x7f51d80ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 shutdown_connections 2024-01-15T18:00:01.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.223+0000 7f51d7e00640 1 -- 172.21.15.2:0/3050657962 wait complete. 2024-01-15T18:00:01.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:01.246 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f) 2024-01-15T18:00:01.305 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/8c291746-2d37-494c-a6fd-79c083bba71f 2024-01-15T18:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:00:01.306 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:01.307 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:00:01.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T18:00:01.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.535+0000 7fa7a5000640 1 Processor -- start 2024-01-15T18:00:01.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.535+0000 7fa7a5000640 1 -- start start 2024-01-15T18:00:01.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.537+0000 7fa7a5000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa7a015ae80 0x7fa7a00acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:01.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.537+0000 7fa7a5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa7a006c8f0 0x7fa7a006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:01.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.537+0000 7fa7a5000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7a006d290 con 0x7fa7a00b2430 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.537+0000 7fa7a5000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a015e840 con 0x7fa7a006c8f0 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.537+0000 7fa7a5000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7a015dbc0 con 0x7fa7a015ae80 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.537+0000 7fa79d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa7a006c8f0 0x7fa7a006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa7a006c8f0 0x7fa7a006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54894/0 (socket says 172.21.15.2:54894) 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79d400640 1 -- 172.21.15.2:0/4140628586 learned_addr learned my addr 172.21.15.2:0/4140628586 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79de00640 1 --2- 172.21.15.2:0/4140628586 >> v2:172.21.15.137:3300/0 conn(0x7fa7a015ae80 0x7fa7a00acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79d400640 1 -- 172.21.15.2:0/4140628586 >> v2:172.21.15.137:3300/0 conn(0x7fa7a015ae80 msgr2=0x7fa7a00acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79d400640 1 --2- 172.21.15.2:0/4140628586 >> v2:172.21.15.137:3300/0 conn(0x7fa7a015ae80 0x7fa7a00acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79d400640 1 -- 172.21.15.2:0/4140628586 >> v1:172.21.15.137:6789/0 conn(0x7fa7a00b2430 legacy=0x7fa7a00b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.538+0000 7fa79d400640 1 -- 172.21.15.2:0/4140628586 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a015dea0 con 0x7fa7a006c8f0 2024-01-15T18:00:01.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.540+0000 7fa79d400640 1 --2- 172.21.15.2:0/4140628586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a006cd50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa7a006e660 tx=0x7fa78802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64f9a2c3ee75087a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:01.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.540+0000 7fa79ca00640 1 -- 172.21.15.2:0/4140628586 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788030860 con 0x7fa7a006c8f0 2024-01-15T18:00:01.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.540+0000 7fa79ca00640 1 -- 172.21.15.2:0/4140628586 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7880309c0 con 0x7fa7a006c8f0 2024-01-15T18:00:01.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.540+0000 7fa79ca00640 1 -- 172.21.15.2:0/4140628586 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788030d50 con 0x7fa7a006c8f0 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.540+0000 7fa7a5000640 1 -- 172.21.15.2:0/4140628586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 msgr2=0x7fa7a006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.540+0000 7fa7a5000640 1 --2- 172.21.15.2:0/4140628586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a006cd50 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa7a006e660 tx=0x7fa78802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.541+0000 7fa7a5000640 1 -- 172.21.15.2:0/4140628586 shutdown_connections 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.541+0000 7fa7a5000640 1 --2- 172.21.15.2:0/4140628586 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a006cd50 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.541+0000 7fa7a5000640 1 --2- 172.21.15.2:0/4140628586 >> v2:172.21.15.137:3300/0 conn(0x7fa7a015ae80 0x7fa7a00acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.541+0000 7fa7a5000640 1 -- 172.21.15.2:0/4140628586 >> 172.21.15.2:0/4140628586 conn(0x7fa7a00b45f0 msgr2=0x7fa7a00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.541+0000 7fa7a5000640 1 -- 172.21.15.2:0/4140628586 shutdown_connections 2024-01-15T18:00:01.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.541+0000 7fa7a5000640 1 -- 172.21.15.2:0/4140628586 wait complete. 2024-01-15T18:00:01.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.542+0000 7fa7a5000640 1 Processor -- start 2024-01-15T18:00:01.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.543+0000 7fa7a5000640 1 -- start start 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa7a5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a014ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa7a5000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7a00b2430 0x7fa7a00aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa7a5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7a015ae80 0x7fa7a015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa7a5000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00af920 con 0x7fa7a006c8f0 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa7a5000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00af360 con 0x7fa7a00b2430 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa7a5000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a00af640 con 0x7fa7a015ae80 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a014ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:01.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a014ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54900/0 (socket says 172.21.15.2:54900) 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 -- 172.21.15.2:0/2823457442 learned_addr learned my addr 172.21.15.2:0/2823457442 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 -- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7a015ae80 msgr2=0x7fa7a015c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7a015ae80 0x7fa7a015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79d400640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7a00b2430 0x7fa7a00aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 -- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7a00b2430 msgr2=0x7fa7a00aea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7a00b2430 0x7fa7a00aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.544+0000 7fa79de00640 1 -- 172.21.15.2:0/2823457442 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7880094d0 con 0x7fa7a006c8f0 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.545+0000 7fa79de00640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a014ba00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa79000e870 tx=0x7fa79000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.545+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7900094f0 con 0x7fa7a006c8f0 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.545+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79000c940 con 0x7fa7a006c8f0 2024-01-15T18:00:01.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.545+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79000ccf0 con 0x7fa7a006c8f0 2024-01-15T18:00:01.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.545+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a00afe90 con 0x7fa7a006c8f0 2024-01-15T18:00:01.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.545+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a014bf40 con 0x7fa7a006c8f0 2024-01-15T18:00:01.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.546+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a00686f0 con 0x7fa7a006c8f0 2024-01-15T18:00:01.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.547+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fa790013470 con 0x7fa7a006c8f0 2024-01-15T18:00:01.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.548+0000 7fa796000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa77806b350 0x7fa77806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:01.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.548+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa790089e10 con 0x7fa7a006c8f0 2024-01-15T18:00:01.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.549+0000 7fa79d400640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa77806b350 0x7fa77806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:01.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.550+0000 7fa79d400640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa77806b350 0x7fa77806d810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa788036000 tx=0x7fa78803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:01.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.552+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa79008d030 con 0x7fa7a006c8f0 2024-01-15T18:00:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.780+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7a00af7b0 con 0x7fa77806b350 2024-01-15T18:00:01.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.938+0000 7fa796000640 1 -- 172.21.15.2:0/2823457442 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa7a00af7b0 con 0x7fa77806b350 2024-01-15T18:00:01.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa77806b350 msgr2=0x7fa77806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa77806b350 0x7fa77806d810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa788036000 tx=0x7fa78803b040 comp rx=0 tx=0).stop 2024-01-15T18:00:01.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 msgr2=0x7fa7a014ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:01.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a014ba00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fa79000e870 tx=0x7fa79000ed30 comp rx=0 tx=0).stop 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 shutdown_connections 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa77806b350 0x7fa77806d810 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7a015ae80 0x7fa7a015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7a00b2430 0x7fa7a00aea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 --2- 172.21.15.2:0/2823457442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7a006c8f0 0x7fa7a014ba00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 >> 172.21.15.2:0/2823457442 conn(0x7fa7a00b45f0 msgr2=0x7fa7a00704c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 shutdown_connections 2024-01-15T18:00:01.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:01.942+0000 7fa7a5000640 1 -- 172.21.15.2:0/2823457442 wait complete. 2024-01-15T18:00:01.949 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.947+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.950 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.947+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.950 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.947+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.950 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.947+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.950 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.947+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.970 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:00:01.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:01.982 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.979+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.982 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.979+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.982 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.979+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.982 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.979+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:01.982 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:01.979+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:02.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.183+0000 7fbb43e00640 1 Processor -- start 2024-01-15T18:00:02.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.184+0000 7fbb43e00640 1 -- start start 2024-01-15T18:00:02.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.185+0000 7fbb43e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbb3c0702c0 0x7fbb3c0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:02.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.185+0000 7fbb43e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c15dd70 con 0x7fbb3c1439f0 2024-01-15T18:00:02.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.185+0000 7fbb43e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb3c0711b0 con 0x7fbb3c06c8f0 2024-01-15T18:00:02.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.185+0000 7fbb43e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb3c143dd0 con 0x7fbb3c0702c0 2024-01-15T18:00:02.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.186+0000 7fbb41400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbb3c1439f0 0x7fbb3c0cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54726/0 (socket says 172.21.15.2:54726) 2024-01-15T18:00:02.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.186+0000 7fbb41400640 1 -- 172.21.15.2:0/2512624798 learned_addr learned my addr 172.21.15.2:0/2512624798 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:02.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.186+0000 7fbb40a00640 1 --2- 172.21.15.2:0/2512624798 >> v2:172.21.15.137:3300/0 conn(0x7fbb3c0702c0 0x7fbb3c0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.186+0000 7fbb3be00640 1 -- 172.21.15.2:0/2512624798 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4176355432 0 0) 0x7fbb3c15dd70 con 0x7fbb3c1439f0 2024-01-15T18:00:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.186+0000 7fbb3be00640 1 -- 172.21.15.2:0/2512624798 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb1c005730 con 0x7fbb3c1439f0 2024-01-15T18:00:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.187+0000 7fbb40a00640 1 -- 172.21.15.2:0/2512624798 >> v1:172.21.15.137:6789/0 conn(0x7fbb3c06c8f0 legacy=0x7fbb3c06ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.187+0000 7fbb40a00640 1 -- 172.21.15.2:0/2512624798 >> v1:172.21.15.2:6789/0 conn(0x7fbb3c1439f0 legacy=0x7fbb3c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.187+0000 7fbb3be00640 1 -- 172.21.15.2:0/2512624798 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2672171364 0 0) 0x7fbb1c005730 con 0x7fbb3c1439f0 2024-01-15T18:00:02.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.187+0000 7fbb40a00640 1 -- 172.21.15.2:0/2512624798 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c070910 con 0x7fbb3c0702c0 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.188+0000 7fbb40a00640 1 --2- 172.21.15.2:0/2512624798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c0b3420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009d90 tx=0x7fbb2402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29192443bed3b377 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.189+0000 7fbb3be00640 1 -- 172.21.15.2:0/2512624798 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2c001670 con 0x7fbb3c0702c0 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.189+0000 7fbb3be00640 1 -- 172.21.15.2:0/2512624798 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2402c940 con 0x7fbb3c0702c0 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.189+0000 7fbb3be00640 1 -- 172.21.15.2:0/2512624798 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2402ccd0 con 0x7fbb3c0702c0 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.189+0000 7fbb43e00640 1 -- 172.21.15.2:0/2512624798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0702c0 msgr2=0x7fbb3c0b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.189+0000 7fbb43e00640 1 --2- 172.21.15.2:0/2512624798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c0b3420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009d90 tx=0x7fbb2402be10 comp rx=0 tx=0).stop 2024-01-15T18:00:02.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.190+0000 7fbb43e00640 1 -- 172.21.15.2:0/2512624798 shutdown_connections 2024-01-15T18:00:02.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.190+0000 7fbb43e00640 1 --2- 172.21.15.2:0/2512624798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c0b3420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:02.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.190+0000 7fbb43e00640 1 -- 172.21.15.2:0/2512624798 >> 172.21.15.2:0/2512624798 conn(0x7fbb3c0b45f0 msgr2=0x7fbb3c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:02.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.190+0000 7fbb43e00640 1 -- 172.21.15.2:0/2512624798 shutdown_connections 2024-01-15T18:00:02.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.190+0000 7fbb43e00640 1 -- 172.21.15.2:0/2512624798 wait complete. 2024-01-15T18:00:02.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.191+0000 7fbb43e00640 1 Processor -- start 2024-01-15T18:00:02.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.192+0000 7fbb43e00640 1 -- start start 2024-01-15T18:00:02.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c06c8f0 0x7fbb3c159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb43e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c1439f0 0x7fbb3c0adca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb43e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c1e7650 con 0x7fbb3c0702c0 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb43e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c15a100 con 0x7fbb3c1439f0 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb43e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3c142ab0 con 0x7fbb3c06c8f0 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c142570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54910/0 (socket says 172.21.15.2:54910) 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 -- 172.21.15.2:0/1533175436 learned_addr learned my addr 172.21.15.2:0/1533175436 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:02.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 -- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c06c8f0 msgr2=0x7fbb3c159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb41400640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c06c8f0 0x7fbb3c159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c06c8f0 0x7fbb3c159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 -- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c1439f0 msgr2=0x7fbb3c0adca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c1439f0 0x7fbb3c0adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.193+0000 7fbb40a00640 1 -- 172.21.15.2:0/1533175436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24009a70 con 0x7fbb3c0702c0 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.194+0000 7fbb40a00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c142570 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbb24037040 tx=0x7fbb24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.194+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24009450 con 0x7fbb3c0702c0 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.194+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24003990 con 0x7fbb3c0702c0 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.194+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3c1e69d0 con 0x7fbb3c0702c0 2024-01-15T18:00:02.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.194+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24003d40 con 0x7fbb3c0702c0 2024-01-15T18:00:02.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.194+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3c1e6f90 con 0x7fbb3c0702c0 2024-01-15T18:00:02.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.195+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3c0b4260 con 0x7fbb3c0702c0 2024-01-15T18:00:02.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.197+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fbb2403b050 con 0x7fbb3c0702c0 2024-01-15T18:00:02.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.197+0000 7fbb39600640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb1406b350 0x7fbb1406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:02.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.198+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb240ac1b0 con 0x7fbb3c0702c0 2024-01-15T18:00:02.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.198+0000 7fbb41400640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb1406b350 0x7fbb1406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:02.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.199+0000 7fbb41400640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb1406b350 0x7fbb1406d810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c001380 tx=0x7fbb2c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:02.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.202+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb24076fc0 con 0x7fbb3c0702c0 2024-01-15T18:00:02.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:02.437+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb3c070ed0 con 0x7fbb1406b350 2024-01-15T18:00:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:03.510+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fbb24077390 con 0x7fbb3c0702c0 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.979+0000 7fbb39600640 1 -- 172.21.15.2:0/1533175436 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fbb2c0058d0 con 0x7fbb1406b350 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:05.984 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:05.985 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:05.985 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.982+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb1406b350 msgr2=0x7fbb1406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.982+0000 7fbb43e00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb1406b350 0x7fbb1406d810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c001380 tx=0x7fbb2c009040 comp rx=0 tx=0).stop 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.982+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 msgr2=0x7fbb3c142570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.982+0000 7fbb43e00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c142570 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbb24037040 tx=0x7fbb24004a40 comp rx=0 tx=0).stop 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 shutdown_connections 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbb1406b350 0x7fbb1406d810 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbb3c1439f0 0x7fbb3c0adca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbb3c0702c0 0x7fbb3c142570 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 --2- 172.21.15.2:0/1533175436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbb3c06c8f0 0x7fbb3c159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 >> 172.21.15.2:0/1533175436 conn(0x7fbb3c0b45f0 msgr2=0x7fbb3c0b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 shutdown_connections 2024-01-15T18:00:05.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:05.983+0000 7fbb43e00640 1 -- 172.21.15.2:0/1533175436 wait complete. 2024-01-15T18:00:06.009 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:00:06.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.219+0000 7f907d600640 1 Processor -- start 2024-01-15T18:00:06.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.219+0000 7f907d600640 1 -- start start 2024-01-15T18:00:06.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f907d600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f907815e340 0x7f90780cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f907d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f90780cdfa0 0x7f9078070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f907d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90780aeab0 con 0x7f907813ce30 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f907d600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780ce6d0 con 0x7f90780cdfa0 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f907d600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90780ce9b0 con 0x7f907815e340 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f9077400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f90780cdfa0 0x7f9078070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f9077400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f90780cdfa0 0x7f9078070ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38430/0 (socket says 172.21.15.2:38430) 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.221+0000 7f9077400640 1 -- 172.21.15.2:0/1983350137 learned_addr learned my addr 172.21.15.2:0/1983350137 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.222+0000 7f9076000640 1 --2- 172.21.15.2:0/1983350137 >> v2:172.21.15.137:3300/0 conn(0x7f907815e340 0x7f90780cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.222+0000 7f9077400640 1 -- 172.21.15.2:0/1983350137 >> v2:172.21.15.137:3300/0 conn(0x7f907815e340 msgr2=0x7f90780cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.222+0000 7f9077400640 1 --2- 172.21.15.2:0/1983350137 >> v2:172.21.15.137:3300/0 conn(0x7f907815e340 0x7f90780cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.222+0000 7f9077400640 1 -- 172.21.15.2:0/1983350137 >> v1:172.21.15.137:6789/0 conn(0x7f907813ce30 legacy=0x7f907815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:06.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.222+0000 7f9077400640 1 -- 172.21.15.2:0/1983350137 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9078070380 con 0x7f90780cdfa0 2024-01-15T18:00:06.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.223+0000 7f9077400640 1 --2- 172.21.15.2:0/1983350137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078070ad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90780721d0 tx=0x7f907002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dfd60e153d3d349 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:06.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.223+0000 7f9075600640 1 -- 172.21.15.2:0/1983350137 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070031870 con 0x7f90780cdfa0 2024-01-15T18:00:06.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.223+0000 7f9075600640 1 -- 172.21.15.2:0/1983350137 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90700319d0 con 0x7f90780cdfa0 2024-01-15T18:00:06.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.223+0000 7f9075600640 1 -- 172.21.15.2:0/1983350137 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9070031d60 con 0x7f90780cdfa0 2024-01-15T18:00:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 -- 172.21.15.2:0/1983350137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 msgr2=0x7f9078070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 --2- 172.21.15.2:0/1983350137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078070ad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f90780721d0 tx=0x7f907002b910 comp rx=0 tx=0).stop 2024-01-15T18:00:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 -- 172.21.15.2:0/1983350137 shutdown_connections 2024-01-15T18:00:06.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 --2- 172.21.15.2:0/1983350137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078070ad0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 --2- 172.21.15.2:0/1983350137 >> v2:172.21.15.137:3300/0 conn(0x7f907815e340 0x7f90780cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 -- 172.21.15.2:0/1983350137 >> 172.21.15.2:0/1983350137 conn(0x7f907806da20 msgr2=0x7f907806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 -- 172.21.15.2:0/1983350137 shutdown_connections 2024-01-15T18:00:06.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.224+0000 7f907d600640 1 -- 172.21.15.2:0/1983350137 wait complete. 2024-01-15T18:00:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.226+0000 7f907d600640 1 Processor -- start 2024-01-15T18:00:06.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.226+0000 7f907d600640 1 -- start start 2024-01-15T18:00:06.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f907d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f907d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907813ce30 0x7f907813df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f907d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907815e340 0x7f90780b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f907d600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907813e9d0 con 0x7f90780cdfa0 2024-01-15T18:00:06.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f907d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907813ecb0 con 0x7f907815e340 2024-01-15T18:00:06.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f907d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907813ef90 con 0x7f907813ce30 2024-01-15T18:00:06.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f9076a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078137890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f9076a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078137890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38444/0 (socket says 172.21.15.2:38444) 2024-01-15T18:00:06.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.227+0000 7f9076a00640 1 -- 172.21.15.2:0/1631693210 learned_addr learned my addr 172.21.15.2:0/1631693210 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076000640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907813ce30 0x7f907813df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9077400640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907815e340 0x7f90780b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076a00640 1 -- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907813ce30 msgr2=0x7f907813df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076a00640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907813ce30 0x7f907813df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076a00640 1 -- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907815e340 msgr2=0x7f90780b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076a00640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907815e340 0x7f90780b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076a00640 1 -- 172.21.15.2:0/1631693210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700094d0 con 0x7f90780cdfa0 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9077400640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907815e340 0x7f90780b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f9076a00640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078137890 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f906800c430 tx=0x7f906800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068011790 con 0x7f90780cdfa0 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90680118f0 con 0x7f90780cdfa0 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.228+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907813f270 con 0x7f90780cdfa0 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.229+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068011ce0 con 0x7f90780cdfa0 2024-01-15T18:00:06.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.229+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907813f5e0 con 0x7f90780cdfa0 2024-01-15T18:00:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.230+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90780686f0 con 0x7f90780cdfa0 2024-01-15T18:00:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.231+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f906801c460 con 0x7f90780cdfa0 2024-01-15T18:00:06.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.232+0000 7f906ea00640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905006b4a0 0x7f905006d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.232+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9068089e70 con 0x7f90780cdfa0 2024-01-15T18:00:06.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.232+0000 7f9076000640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905006b4a0 0x7f905006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.234+0000 7f9076000640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905006b4a0 0x7f905006d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9078138220 tx=0x7f9060003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:06.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.237+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9068054a60 con 0x7f90780cdfa0 2024-01-15T18:00:06.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.508+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f90781515b0 con 0x7f90780cdfa0 2024-01-15T18:00:06.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.508+0000 7f906ea00640 1 -- 172.21.15.2:0/1631693210 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1855 (secure 0 0 0) 0x7f9068058080 con 0x7f90780cdfa0 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:00:06.513 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 174, 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 35, 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:00:06.514 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:00:06.515 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 796, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3222094292, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 6690045952, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 766404067328, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 85, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 69889460, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 0, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 19 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:00:06.516 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 17, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 2, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 2, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 4954 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.517 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 8, 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T17:59:59.425218+0000", 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": {} 2024-01-15T18:00:06.518 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905006b4a0 msgr2=0x7f905006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905006b4a0 0x7f905006d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9078138220 tx=0x7f9060003990 comp rx=0 tx=0).stop 2024-01-15T18:00:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 msgr2=0x7f9078137890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078137890 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f906800c430 tx=0x7f906800c8f0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 shutdown_connections 2024-01-15T18:00:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905006b4a0 0x7f905006d960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907815e340 0x7f90780b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907813ce30 0x7f907813df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 --2- 172.21.15.2:0/1631693210 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90780cdfa0 0x7f9078137890 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.511+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 >> 172.21.15.2:0/1631693210 conn(0x7f907806da20 msgr2=0x7f9078071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.512+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 shutdown_connections 2024-01-15T18:00:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.512+0000 7f907d600640 1 -- 172.21.15.2:0/1631693210 wait complete. 2024-01-15T18:00:06.534 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:00:06.534 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:06.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.742+0000 7f0620a00640 1 Processor -- start 2024-01-15T18:00:06.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.742+0000 7f0620a00640 1 -- start start 2024-01-15T18:00:06.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.744+0000 7f0620a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.744+0000 7f0620a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f061806ce30 0x7f0618070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.744+0000 7f0620a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180ce230 con 0x7f0618144c00 2024-01-15T18:00:06.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.744+0000 7f0620a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f061806d2c0 con 0x7f061806ce30 2024-01-15T18:00:06.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.744+0000 7f0620a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06180ce510 con 0x7f061815c450 2024-01-15T18:00:06.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:44188/0 (socket says 172.21.15.2:44188) 2024-01-15T18:00:06.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061d600640 1 -- 172.21.15.2:0/3749156641 learned_addr learned my addr 172.21.15.2:0/3749156641 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:06.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061ea00640 1 --2- 172.21.15.2:0/3749156641 >> v2:172.21.15.137:3300/0 conn(0x7f061806ce30 0x7f0618070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061ea00640 1 -- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 msgr2=0x7f061806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:06.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061ea00640 1 --2- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061ea00640 1 -- 172.21.15.2:0/3749156641 >> v1:172.21.15.137:6789/0 conn(0x7f0618144c00 legacy=0x7f06180cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.745+0000 7f061ea00640 1 -- 172.21.15.2:0/3749156641 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06180ce7f0 con 0x7f061806ce30 2024-01-15T18:00:06.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.747+0000 7f061ea00640 1 --2- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f0618070b10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0610009d90 tx=0x7f061002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce866de3fd1ad43d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:06.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.747+0000 7f061cc00640 1 -- 172.21.15.2:0/3749156641 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061002cc60 con 0x7f061806ce30 2024-01-15T18:00:06.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f061cc00640 1 -- 172.21.15.2:0/3749156641 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061002cdc0 con 0x7f061806ce30 2024-01-15T18:00:06.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f061cc00640 1 -- 172.21.15.2:0/3749156641 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0610034660 con 0x7f061806ce30 2024-01-15T18:00:06.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 -- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 msgr2=0x7f0618070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 --2- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f0618070b10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0610009d90 tx=0x7f061002beb0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 -- 172.21.15.2:0/3749156641 shutdown_connections 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 --2- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f0618070b10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 --2- 172.21.15.2:0/3749156641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 -- 172.21.15.2:0/3749156641 >> 172.21.15.2:0/3749156641 conn(0x7f06180b45f0 msgr2=0x7f06180b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 -- 172.21.15.2:0/3749156641 shutdown_connections 2024-01-15T18:00:06.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.748+0000 7f0620a00640 1 -- 172.21.15.2:0/3749156641 wait complete. 2024-01-15T18:00:06.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.750+0000 7f0620a00640 1 Processor -- start 2024-01-15T18:00:06.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.750+0000 7f0620a00640 1 -- start start 2024-01-15T18:00:06.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f0620a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f06180b13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f0620a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 0x7f061813da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f0620a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f0620a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061813d5e0 con 0x7f0618144c00 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f0620a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06180afac0 con 0x7f061806ce30 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f0620a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06180af7e0 con 0x7f061815c450 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f061d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 0x7f061813da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f061d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 0x7f061813da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38456/0 (socket says 172.21.15.2:38456) 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.751+0000 7f061d600640 1 -- 172.21.15.2:0/844425501 learned_addr learned my addr 172.21.15.2:0/844425501 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061e000640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f06180b13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061ea00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061d600640 1 -- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 msgr2=0x7f061815d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061d600640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061815d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061d600640 1 -- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 msgr2=0x7f06180b13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061d600640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f06180b13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061d600640 1 -- 172.21.15.2:0/844425501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0610009a70 con 0x7f0618144c00 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.752+0000 7f061d600640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 0x7f061813da70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f060000e840 tx=0x7f060000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.753+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000c8e0 con 0x7f0618144c00 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.753+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060000ca40 con 0x7f0618144c00 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.753+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f060000cdf0 con 0x7f0618144c00 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.753+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f061813cf90 con 0x7f0618144c00 2024-01-15T18:00:06.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.753+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06180b1930 con 0x7f0618144c00 2024-01-15T18:00:06.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.754+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061813d790 con 0x7f0618144c00 2024-01-15T18:00:06.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.755+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f0600013470 con 0x7f0618144c00 2024-01-15T18:00:06.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.756+0000 7f0616000640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f05f006b4a0 0x7f05f006d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:06.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.756+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0600089fb0 con 0x7f0618144c00 2024-01-15T18:00:06.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.756+0000 7f061e000640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f05f006b4a0 0x7f05f006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:06.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.757+0000 7f061e000640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f05f006b4a0 0x7f05f006d960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0608007fd0 tx=0x7f0608009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:06.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.760+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f061813d790 con 0x7f0618144c00 2024-01-15T18:00:07.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:06.996+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f06180cead0 con 0x7f05f006b4a0 2024-01-15T18:00:07.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.011+0000 7f0616000640 1 -- 172.21.15.2:0/844425501 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f06080012c0 con 0x7f05f006b4a0 2024-01-15T18:00:07.015 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:07.015 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:07.015 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 0.0, 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:00:07.016 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "0/3", 2024-01-15T18:00:07.017 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:07.017 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:07.017 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:07.017 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:07.017 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f05f006b4a0 msgr2=0x7f05f006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f05f006b4a0 0x7f05f006d960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0608007fd0 tx=0x7f0608009040 comp rx=0 tx=0).stop 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 msgr2=0x7f061813da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 0x7f061813da70 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f060000e840 tx=0x7f060000ed00 comp rx=0 tx=0).stop 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 shutdown_connections 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f05f006b4a0 0x7f05f006d960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f061815c450 0x7f061815d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:07.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0618144c00 0x7f061813da70 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:07.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 --2- 172.21.15.2:0/844425501 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f061806ce30 0x7f06180b13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:07.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.013+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 >> 172.21.15.2:0/844425501 conn(0x7f06180b45f0 msgr2=0x7f06180b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:07.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.014+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 shutdown_connections 2024-01-15T18:00:07.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:07.014+0000 7f0620a00640 1 -- 172.21.15.2:0/844425501 wait complete. 2024-01-15T18:00:07.036 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:08.037 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:08.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.263+0000 7fbfffe00640 1 Processor -- start 2024-01-15T18:00:08.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.264+0000 7fbfffe00640 1 -- start start 2024-01-15T18:00:08.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.266+0000 7fbfffe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc0000d3090 0x7fc0000d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:08.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.266+0000 7fbfffe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d1680 0x7fc0000d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:08.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.266+0000 7fbfffe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0000d3a60 con 0x7fc00015d340 2024-01-15T18:00:08.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.266+0000 7fbfffe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0000ce430 con 0x7fc0000d3090 2024-01-15T18:00:08.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.266+0000 7fbfffe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000b3de0 con 0x7fc0000d1680 2024-01-15T18:00:08.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.267+0000 7fbffea00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fc00015d340 0x7fc0000cdd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:50072/0 (socket says 172.21.15.2:50072) 2024-01-15T18:00:08.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.267+0000 7fbffea00640 1 -- 172.21.15.2:0/3160511571 learned_addr learned my addr 172.21.15.2:0/3160511571 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:08.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.267+0000 7fbffe000640 1 --2- 172.21.15.2:0/3160511571 >> v2:172.21.15.137:3300/0 conn(0x7fc0000d3090 0x7fc0000d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:08.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.267+0000 7fbfff400640 1 --2- 172.21.15.2:0/3160511571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d1680 0x7fc0000d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:08.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.268+0000 7fbffd600640 1 -- 172.21.15.2:0/3160511571 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558879557 0 0) 0x7fc0000d3a60 con 0x7fc00015d340 2024-01-15T18:00:08.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.268+0000 7fbffd600640 1 -- 172.21.15.2:0/3160511571 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfe0003520 con 0x7fc00015d340 2024-01-15T18:00:08.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.268+0000 7fbfff400640 1 -- 172.21.15.2:0/3160511571 >> v2:172.21.15.137:3300/0 conn(0x7fc0000d3090 msgr2=0x7fc0000d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:08.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.268+0000 7fbfff400640 1 --2- 172.21.15.2:0/3160511571 >> v2:172.21.15.137:3300/0 conn(0x7fc0000d3090 0x7fc0000d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.268+0000 7fbfff400640 1 -- 172.21.15.2:0/3160511571 >> v1:172.21.15.137:6789/0 conn(0x7fc00015d340 legacy=0x7fc0000cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:08.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.268+0000 7fbfff400640 1 -- 172.21.15.2:0/3160511571 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0000b40c0 con 0x7fc0000d1680 2024-01-15T18:00:08.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.272+0000 7fbfff400640 1 --2- 172.21.15.2:0/3160511571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d1680 0x7fc0000d1b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009d90 tx=0x7fbff802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2693c3f6c43b3a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.273+0000 7fbffd600640 1 -- 172.21.15.2:0/3160511571 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff802cc50 con 0x7fc0000d1680 2024-01-15T18:00:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.273+0000 7fbffd600640 1 -- 172.21.15.2:0/3160511571 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff802cdb0 con 0x7fc0000d1680 2024-01-15T18:00:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.273+0000 7fbffd600640 1 -- 172.21.15.2:0/3160511571 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff8034640 con 0x7fc0000d1680 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 -- 172.21.15.2:0/3160511571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d1680 msgr2=0x7fc0000d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 --2- 172.21.15.2:0/3160511571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d1680 0x7fc0000d1b10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbff8009d90 tx=0x7fbff802bea0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 -- 172.21.15.2:0/3160511571 shutdown_connections 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 --2- 172.21.15.2:0/3160511571 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d1680 0x7fc0000d1b10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 --2- 172.21.15.2:0/3160511571 >> v2:172.21.15.137:3300/0 conn(0x7fc0000d3090 0x7fc0000d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 -- 172.21.15.2:0/3160511571 >> 172.21.15.2:0/3160511571 conn(0x7fc00008caf0 msgr2=0x7fc00009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:08.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 -- 172.21.15.2:0/3160511571 shutdown_connections 2024-01-15T18:00:08.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.274+0000 7fbfffe00640 1 -- 172.21.15.2:0/3160511571 wait complete. 2024-01-15T18:00:08.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.277+0000 7fbfffe00640 1 Processor -- start 2024-01-15T18:00:08.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.277+0000 7fbfffe00640 1 -- start start 2024-01-15T18:00:08.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfffe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0000d1680 0x7fc00015da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:08.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfffe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d3090 0x7fc0001f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:08.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfffe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 0x7fc000158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:08.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfffe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0000c6c60 con 0x7fc00015d340 2024-01-15T18:00:08.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfffe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0001f2b50 con 0x7fc0000d1680 2024-01-15T18:00:08.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfffe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc000160550 con 0x7fc0000d3090 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.278+0000 7fbfff400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 0x7fc000158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 0x7fc000158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38466/0 (socket says 172.21.15.2:38466) 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 -- 172.21.15.2:0/2996544978 learned_addr learned my addr 172.21.15.2:0/2996544978 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbffe000640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d3090 0x7fc0001f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 -- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d3090 msgr2=0x7fc0001f2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d3090 0x7fc0001f2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 -- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0000d1680 msgr2=0x7fc00015da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0000d1680 0x7fc00015da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 -- 172.21.15.2:0/2996544978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff8009a70 con 0x7fc00015d340 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.279+0000 7fbfff400640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 0x7fc000158960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbff802c4b0 tx=0x7fbff8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.280+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff803e610 con 0x7fc00015d340 2024-01-15T18:00:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.280+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff803e770 con 0x7fc00015d340 2024-01-15T18:00:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.280+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff803eb60 con 0x7fc00015d340 2024-01-15T18:00:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.280+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0000c9600 con 0x7fc00015d340 2024-01-15T18:00:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.280+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0000c70f0 con 0x7fc00015d340 2024-01-15T18:00:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.281+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0000c9b20 con 0x7fc00015d340 2024-01-15T18:00:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.282+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fbff803b050 con 0x7fc00015d340 2024-01-15T18:00:08.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.283+0000 7fbff6a00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbfd806b470 0x7fbfd806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:08.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.283+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbff80abfa0 con 0x7fc00015d340 2024-01-15T18:00:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.288+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbff80dee90 con 0x7fc00015d340 2024-01-15T18:00:08.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.292+0000 7fbffea00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbfd806b470 0x7fbfd806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:08.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.293+0000 7fbffea00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbfd806b470 0x7fbfd806d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbff0001610 tx=0x7fbff0007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:08.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.640+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0000d2050 con 0x7fbfd806b470 2024-01-15T18:00:08.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.659+0000 7fbff6a00640 1 -- 172.21.15.2:0/2996544978 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fbff00058b0 con 0x7fbfd806b470 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 0.0, 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "0/3", 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:08.664 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:08.665 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:08.665 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.662+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbfd806b470 msgr2=0x7fbfd806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.662+0000 7fbfffe00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbfd806b470 0x7fbfd806d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbff0001610 tx=0x7fbff0007930 comp rx=0 tx=0).stop 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.662+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 msgr2=0x7fc000158960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.662+0000 7fbfffe00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 0x7fc000158960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbff802c4b0 tx=0x7fbff8004a40 comp rx=0 tx=0).stop 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.663+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 shutdown_connections 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.663+0000 7fbfffe00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbfd806b470 0x7fbfd806d930 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbff0001610 tx=0x7fbff0007930 comp rx=0 tx=0).stop 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.663+0000 7fbfffe00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc00015d340 0x7fc000158960 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.663+0000 7fbfffe00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0000d3090 0x7fc0001f2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.663+0000 7fbfffe00640 1 --2- 172.21.15.2:0/2996544978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc0000d1680 0x7fc00015da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:08.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.663+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 >> 172.21.15.2:0/2996544978 conn(0x7fc00008caf0 msgr2=0x7fc0000cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:08.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.664+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 shutdown_connections 2024-01-15T18:00:08.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:08.664+0000 7fbfffe00640 1 -- 172.21.15.2:0/2996544978 wait complete. 2024-01-15T18:00:08.690 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:09.692 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.006+0000 7fbac6e00640 1 Processor -- start 2024-01-15T18:00:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.007+0000 7fbac6e00640 1 -- start start 2024-01-15T18:00:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.010+0000 7fbac6e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbac014afd0 0x7fbac00b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.013+0000 7fbabfe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbac014afd0 0x7fbac00b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.015+0000 7fbabfe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbac014afd0 0x7fbac00b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:58704/0 (socket says 172.21.15.2:58704) 2024-01-15T18:00:10.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.016+0000 7fbac6e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00b25f0 con 0x7fbac014b3b0 2024-01-15T18:00:10.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.016+0000 7fbac6e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00b28d0 con 0x7fbac006d000 2024-01-15T18:00:10.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.016+0000 7fbac6e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac0070ff0 con 0x7fbac014afd0 2024-01-15T18:00:10.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.017+0000 7fbabf400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbac014b3b0 0x7fbac006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:43812/0 (socket says 172.21.15.2:43812) 2024-01-15T18:00:10.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.017+0000 7fbabf400640 1 -- 172.21.15.2:0/3368148106 learned_addr learned my addr 172.21.15.2:0/3368148106 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:10.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.018+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34444294 0 0) 0x7fbac00b25f0 con 0x7fbac014b3b0 2024-01-15T18:00:10.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.018+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0003520 con 0x7fbac014b3b0 2024-01-15T18:00:10.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751917775 0 0) 0x7fbaa0003520 con 0x7fbac014b3b0 2024-01-15T18:00:10.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 >> v2:172.21.15.137:3300/0 conn(0x7fbac014afd0 msgr2=0x7fbac00b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:10.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 --2- 172.21.15.2:0/3368148106 >> v2:172.21.15.137:3300/0 conn(0x7fbac014afd0 0x7fbac00b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 >> v1:172.21.15.137:6789/0 conn(0x7fbac006d000 legacy=0x7fbac00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac00b2bb0 con 0x7fbac014b3b0 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fbaa8001670 con 0x7fbac014b3b0 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 >> v1:172.21.15.2:6789/0 conn(0x7fbac014b3b0 legacy=0x7fbac006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa0003970 0x7fbaa0003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbaa0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa0005180 0x7fbaa000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0070ff0 con 0x7fbaa00047c0 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa000c080 con 0x7fbaa0003970 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.019+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa000c240 con 0x7fbaa0005180 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.020+0000 7fbabf400640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbaa0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.021+0000 7fbabfe00640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa0003970 0x7fbaa0003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.021+0000 7fbabf400640 1 -- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa0005180 msgr2=0x7fbaa000b980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.021+0000 7fbabf400640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa0005180 0x7fbaa000b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.021+0000 7fbabf400640 1 -- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa0003970 msgr2=0x7fbaa0003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.021+0000 7fbabf400640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa0003970 0x7fbaa0003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.021+0000 7fbabf400640 1 -- 172.21.15.2:0/3368148106 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa0003520 con 0x7fbaa00047c0 2024-01-15T18:00:10.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.022+0000 7fbabf400640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbaa0004c40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800c160 tx=0x7fbaa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e8d27d8505c619 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:10.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.023+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8005ad0 con 0x7fbaa00047c0 2024-01-15T18:00:10.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.023+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa8005c30 con 0x7fbaa00047c0 2024-01-15T18:00:10.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.023+0000 7fbabea00640 1 -- 172.21.15.2:0/3368148106 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa802c7e0 con 0x7fbaa00047c0 2024-01-15T18:00:10.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.024+0000 7fbac6e00640 1 -- 172.21.15.2:0/3368148106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 msgr2=0x7fbaa0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.024+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3368148106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbaa0004c40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800c160 tx=0x7fbaa00057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.025+0000 7fbac4e00640 1 -- 172.21.15.2:0/3368148106 reap_dead start 2024-01-15T18:00:10.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.028+0000 7fbac6e00640 1 -- 172.21.15.2:0/3368148106 shutdown_connections 2024-01-15T18:00:10.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.028+0000 7fbac6e00640 1 -- 172.21.15.2:0/3368148106 >> 172.21.15.2:0/3368148106 conn(0x7fbac00b45f0 msgr2=0x7fbac00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:10.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.030+0000 7fbac6e00640 1 -- 172.21.15.2:0/3368148106 shutdown_connections 2024-01-15T18:00:10.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.030+0000 7fbac6e00640 1 -- 172.21.15.2:0/3368148106 wait complete. 2024-01-15T18:00:10.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.033+0000 7fbac6e00640 1 Processor -- start 2024-01-15T18:00:10.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.034+0000 7fbac6e00640 1 -- start start 2024-01-15T18:00:10.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbac6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbac0140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbac6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00b0560 0x7fbac00cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbac6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac01466b0 0x7fbac0146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbac6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00aed00 con 0x7fbaa00047c0 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbac6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0147070 con 0x7fbac00b0560 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbac6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac01414f0 con 0x7fbac01466b0 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbabfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbac0140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.035+0000 7fbabfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbac0140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38478/0 (socket says 172.21.15.2:38478) 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.036+0000 7fbabfe00640 1 -- 172.21.15.2:0/3222602216 learned_addr learned my addr 172.21.15.2:0/3222602216 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.036+0000 7fbabfe00640 1 -- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac01466b0 msgr2=0x7fbac0146b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.036+0000 7fbabfe00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac01466b0 0x7fbac0146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.036+0000 7fbabfe00640 1 -- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00b0560 msgr2=0x7fbac00cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.036+0000 7fbabfe00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00b0560 0x7fbac00cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.036+0000 7fbabfe00640 1 -- 172.21.15.2:0/3222602216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac01417d0 con 0x7fbaa00047c0 2024-01-15T18:00:10.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.037+0000 7fbabfe00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbac0140eb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbab00116d0 tx=0x7fbab0011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:10.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.037+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000f9c0 con 0x7fbaa00047c0 2024-01-15T18:00:10.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.037+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac0141ab0 con 0x7fbaa00047c0 2024-01-15T18:00:10.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.037+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac00b09e0 con 0x7fbaa00047c0 2024-01-15T18:00:10.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.037+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000fb20 con 0x7fbaa00047c0 2024-01-15T18:00:10.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.037+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab000fe90 con 0x7fbaa00047c0 2024-01-15T18:00:10.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.039+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba90005250 con 0x7fbaa00047c0 2024-01-15T18:00:10.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.041+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fbab0015460 con 0x7fbaa00047c0 2024-01-15T18:00:10.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.042+0000 7fbab7e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba9806b490 0x7fba9806d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:10.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.042+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbab008a790 con 0x7fbaa00047c0 2024-01-15T18:00:10.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.043+0000 7fbabf400640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba9806b490 0x7fba9806d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:10.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.044+0000 7fbabf400640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba9806b490 0x7fba9806d950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8034000 tx=0x7fbaa8007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:10.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.050+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbab00554f0 con 0x7fbaa00047c0 2024-01-15T18:00:10.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.364+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fba90000bc0 con 0x7fba9806b490 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.378+0000 7fbab7e00640 1 -- 172.21.15.2:0/3222602216 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fba90000bc0 con 0x7fba9806b490 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:10.382 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:10.383 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba9806b490 msgr2=0x7fba9806d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba9806b490 0x7fba9806d950 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8034000 tx=0x7fbaa8007000 comp rx=0 tx=0).stop 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 msgr2=0x7fbac0140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbac0140eb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fbab00116d0 tx=0x7fbab0011b90 comp rx=0 tx=0).stop 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 shutdown_connections 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba9806b490 0x7fba9806d950 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbac01466b0 0x7fbac0146b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbac00b0560 0x7fbac00cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 --2- 172.21.15.2:0/3222602216 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa00047c0 0x7fbac0140eb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.380+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 >> 172.21.15.2:0/3222602216 conn(0x7fbac00b45f0 msgr2=0x7fbac006acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.381+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 shutdown_connections 2024-01-15T18:00:10.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:10.381+0000 7fbac6e00640 1 -- 172.21.15.2:0/3222602216 wait complete. 2024-01-15T18:00:10.402 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:11.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:11.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.636+0000 7f4ff1400640 1 Processor -- start 2024-01-15T18:00:11.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.636+0000 7f4ff1400640 1 -- start start 2024-01-15T18:00:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4ff1400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4fec0b2210 0x7f4fec0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4ff1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 0x7f4fec0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4ff1400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fec0d1250 con 0x7f4fec0d1870 2024-01-15T18:00:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4ff1400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fec0ce6d0 con 0x7f4fec0b2210 2024-01-15T18:00:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4ff1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0ce8e0 con 0x7f4fec0d1c50 2024-01-15T18:00:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4febe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4fec0d1870 0x7f4fec15d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:43818/0 (socket says 172.21.15.2:43818) 2024-01-15T18:00:11.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.638+0000 7f4febe00640 1 -- 172.21.15.2:0/4180679431 learned_addr learned my addr 172.21.15.2:0/4180679431 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4fe3e00640 1 --2- 172.21.15.2:0/4180679431 >> v2:172.21.15.137:3300/0 conn(0x7f4fec0b2210 0x7f4fec0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4ff0a00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 0x7f4fec0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061100212 0 0) 0x7f4fec0d1250 con 0x7f4fec0d1870 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fcc003520 con 0x7f4fec0d1870 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4213385325 0 0) 0x7f4fcc003520 con 0x7f4fec0d1870 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 msgr2=0x7f4fec0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 0x7f4fec0b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 >> v2:172.21.15.137:3300/0 conn(0x7f4fec0b2210 msgr2=0x7f4fec0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 --2- 172.21.15.2:0/4180679431 >> v2:172.21.15.137:3300/0 conn(0x7f4fec0b2210 0x7f4fec0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0cebc0 con 0x7f4fec0d1870 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.639+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4fdc001670 con 0x7f4fec0d1870 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 >> v1:172.21.15.2:6789/0 conn(0x7f4fec0d1870 legacy=0x7f4fec15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc003970 0x7f4fcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 0x7f4fcc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc004d00 0x7f4fcc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0ce6d0 con 0x7f4fcc003970 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0ce8e0 con 0x7f4fcc004d00 2024-01-15T18:00:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc00b980 con 0x7f4fec0d1c50 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4febe00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc003970 0x7f4fcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4ff0a00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc004d00 0x7f4fcc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4febe00640 1 -- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 msgr2=0x7f4fcc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4fe3e00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 0x7f4fcc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4febe00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fec0d1c50 0x7f4fcc0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4febe00640 1 -- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc004d00 msgr2=0x7f4fcc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4febe00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc004d00 0x7f4fcc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4febe00640 1 -- 172.21.15.2:0/4180679431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc003520 con 0x7f4fcc003970 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.640+0000 7f4ff0a00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc004d00 0x7f4fcc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.641+0000 7f4febe00640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc003970 0x7f4fcc003d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00c240 tx=0x7f4fcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1c82860b095164 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc005ad0 con 0x7f4fcc003970 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc005c30 con 0x7f4fcc003970 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4feb400640 1 -- 172.21.15.2:0/4180679431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc02c8d0 con 0x7f4fcc003970 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4ff1400640 1 -- 172.21.15.2:0/4180679431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc003970 msgr2=0x7f4fcc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4ff1400640 1 --2- 172.21.15.2:0/4180679431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc003970 0x7f4fcc003d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc00c240 tx=0x7f4fcc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4ff0a00640 1 -- 172.21.15.2:0/4180679431 reap_dead start 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.642+0000 7f4ff1400640 1 -- 172.21.15.2:0/4180679431 shutdown_connections 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.643+0000 7f4ff1400640 1 -- 172.21.15.2:0/4180679431 >> 172.21.15.2:0/4180679431 conn(0x7f4fec0d30b0 msgr2=0x7f4fec0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.643+0000 7f4ff1400640 1 -- 172.21.15.2:0/4180679431 shutdown_connections 2024-01-15T18:00:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.643+0000 7f4ff1400640 1 -- 172.21.15.2:0/4180679431 wait complete. 2024-01-15T18:00:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.644+0000 7f4ff1400640 1 Processor -- start 2024-01-15T18:00:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 -- start start 2024-01-15T18:00:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc003970 0x7f4fec1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fec0b8260 0x7f4fec0b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 0x7f4fec0b6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0b8c20 con 0x7f4fec0b5c00 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0b65c0 con 0x7f4fec0b8260 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.645+0000 7f4ff1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fec0b6860 con 0x7f4fcc003970 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.646+0000 7f4ff0a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 0x7f4fec0b6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.646+0000 7f4febe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc003970 0x7f4fec1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.646+0000 7f4ff0a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 0x7f4fec0b6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38494/0 (socket says 172.21.15.2:38494) 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.646+0000 7f4ff0a00640 1 -- 172.21.15.2:0/355764873 learned_addr learned my addr 172.21.15.2:0/355764873 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.646+0000 7f4fe3e00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fec0b8260 0x7f4fec0b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff0a00640 1 -- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc003970 msgr2=0x7f4fec1523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff0a00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc003970 0x7f4fec1523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff0a00640 1 -- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fec0b8260 msgr2=0x7f4fec0b86e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff0a00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fec0b8260 0x7f4fec0b86e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff0a00640 1 -- 172.21.15.2:0/355764873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fec0b6b40 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff0a00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 0x7f4fec0b6080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b9160 tx=0x7f4fe400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe4017400 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe4011040 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fe40177e0 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fec0b6e20 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.647+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fec0b73e0 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.648+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fec0b7600 con 0x7f4fec0b5c00 2024-01-15T18:00:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.649+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f4fe4017940 con 0x7f4fec0b5c00 2024-01-15T18:00:11.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.650+0000 7f4fe8c00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fc406b470 0x7f4fc406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:11.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.650+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fe408a380 con 0x7f4fec0b5c00 2024-01-15T18:00:11.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.651+0000 7f4febe00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fc406b470 0x7f4fc406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:11.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.652+0000 7f4febe00640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fc406b470 0x7f4fc406d930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc034000 tx=0x7f4fdc007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:11.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.655+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fe4054f20 con 0x7f4fec0b5c00 2024-01-15T18:00:11.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.900+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fec0d0f20 con 0x7f4fc406b470 2024-01-15T18:00:11.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.913+0000 7f4fe8c00640 1 -- 172.21.15.2:0/355764873 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f4fdc0013b0 con 0x7f4fc406b470 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:11.918 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:11.919 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:11.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fc406b470 msgr2=0x7f4fc406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fc406b470 0x7f4fc406d930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc034000 tx=0x7f4fdc007000 comp rx=0 tx=0).stop 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 msgr2=0x7f4fec0b6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 0x7f4fec0b6080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f4fec0b9160 tx=0x7f4fe400f4a0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 shutdown_connections 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fc406b470 0x7f4fc406d930 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fec0b5c00 0x7f4fec0b6080 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fec0b8260 0x7f4fec0b86e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 --2- 172.21.15.2:0/355764873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc003970 0x7f4fec1523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 >> 172.21.15.2:0/355764873 conn(0x7f4fec0d30b0 msgr2=0x7f4fec0d2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.915+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 shutdown_connections 2024-01-15T18:00:11.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:11.916+0000 7f4ff1400640 1 -- 172.21.15.2:0/355764873 wait complete. 2024-01-15T18:00:11.938 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:12.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:13.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.209+0000 7f03d7e00640 1 Processor -- start 2024-01-15T18:00:13.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.209+0000 7f03d7e00640 1 -- start start 2024-01-15T18:00:13.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.210+0000 7f03d7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 0x7f03d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:13.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.210+0000 7f03d7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03d00b3420 0x7f03d00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:13.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.210+0000 7f03d7e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d015dd70 con 0x7f03d00ccc70 2024-01-15T18:00:13.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.210+0000 7f03d7e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d00b3880 con 0x7f03d00b3420 2024-01-15T18:00:13.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.210+0000 7f03d7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00711b0 con 0x7f03d01439f0 2024-01-15T18:00:13.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.211+0000 7f03d4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03d00b3420 0x7f03d00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:13.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.211+0000 7f03d4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03d00b3420 0x7f03d00cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:46462/0 (socket says 172.21.15.2:46462) 2024-01-15T18:00:13.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.211+0000 7f03d4a00640 1 -- 172.21.15.2:0/1524259172 learned_addr learned my addr 172.21.15.2:0/1524259172 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:13.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.212+0000 7f03d4a00640 1 -- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 msgr2=0x7f03d00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:13.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.213+0000 7f03d4a00640 1 --2- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 0x7f03d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.213+0000 7f03d4a00640 1 -- 172.21.15.2:0/1524259172 >> v1:172.21.15.137:6789/0 conn(0x7f03d00ccc70 legacy=0x7f03d006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:13.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.213+0000 7f03d4a00640 1 -- 172.21.15.2:0/1524259172 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0143dd0 con 0x7f03d00b3420 2024-01-15T18:00:13.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.215+0000 7f03d4a00640 1 --2- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d00cc730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009870 tx=0x7f03b802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=476968f795304de4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.218+0000 7f03cfe00640 1 -- 172.21.15.2:0/1524259172 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031640 con 0x7f03d00b3420 2024-01-15T18:00:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.218+0000 7f03cfe00640 1 -- 172.21.15.2:0/1524259172 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b80317a0 con 0x7f03d00b3420 2024-01-15T18:00:13.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.218+0000 7f03cfe00640 1 -- 172.21.15.2:0/1524259172 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8031b30 con 0x7f03d00b3420 2024-01-15T18:00:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.219+0000 7f03d7e00640 1 -- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 msgr2=0x7f03d00cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.219+0000 7f03d7e00640 1 --2- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d00cc730 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f03b8009870 tx=0x7f03b802b8f0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.220+0000 7f03d7e00640 1 -- 172.21.15.2:0/1524259172 shutdown_connections 2024-01-15T18:00:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.220+0000 7f03d7e00640 1 --2- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d00cc730 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.220+0000 7f03d7e00640 1 --2- 172.21.15.2:0/1524259172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 0x7f03d00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.220+0000 7f03d7e00640 1 -- 172.21.15.2:0/1524259172 >> 172.21.15.2:0/1524259172 conn(0x7f03d00b45f0 msgr2=0x7f03d00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.220+0000 7f03d7e00640 1 -- 172.21.15.2:0/1524259172 shutdown_connections 2024-01-15T18:00:13.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.220+0000 7f03d7e00640 1 -- 172.21.15.2:0/1524259172 wait complete. 2024-01-15T18:00:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.223+0000 7f03d7e00640 1 Processor -- start 2024-01-15T18:00:13.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.223+0000 7f03d7e00640 1 -- start start 2024-01-15T18:00:13.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d015b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 0x7f03d01f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 0x7f03d013a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d013a4e0 con 0x7f03d00ccc70 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00aeb10 con 0x7f03d00b3420 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d00aedf0 con 0x7f03d01439f0 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.224+0000 7f03d4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 0x7f03d01f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 0x7f03d01f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58450/0 (socket says 172.21.15.2:58450) 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 -- 172.21.15.2:0/3420814244 learned_addr learned my addr 172.21.15.2:0/3420814244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d5400640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d015b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:13.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 -- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 msgr2=0x7f03d013a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 0x7f03d013a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 -- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 msgr2=0x7f03d015b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d015b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.225+0000 7f03d4a00640 1 -- 172.21.15.2:0/3420814244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b80094d0 con 0x7f03d00ccc70 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.226+0000 7f03d4a00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 0x7f03d01f7050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f03b802b930 tx=0x7f03b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.226+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b803ac60 con 0x7f03d00ccc70 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.226+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b803ade0 con 0x7f03d00ccc70 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.226+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d00af0d0 con 0x7f03d00ccc70 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.226+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8044630 con 0x7f03d00ccc70 2024-01-15T18:00:13.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.226+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d01f7590 con 0x7f03d00ccc70 2024-01-15T18:00:13.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.228+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d00af5f0 con 0x7f03d00ccc70 2024-01-15T18:00:13.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.230+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f03b802f050 con 0x7f03d00ccc70 2024-01-15T18:00:13.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.231+0000 7f03cd600640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03a806b470 0x7f03a806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:13.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.231+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03b80ac050 con 0x7f03d00ccc70 2024-01-15T18:00:13.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.234+0000 7f03d5400640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03a806b470 0x7f03a806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.235+0000 7f03d5400640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03a806b470 0x7f03a806d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b810 tx=0x7f03c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:13.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.239+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03d00af5f0 con 0x7f03d00ccc70 2024-01-15T18:00:13.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.576+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f03d0070ed0 con 0x7f03a806b470 2024-01-15T18:00:13.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.604+0000 7f03cd600640 1 -- 172.21.15.2:0/3420814244 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f03c000b400 con 0x7f03a806b470 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:13.609 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:13.610 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:13.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.611+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03a806b470 msgr2=0x7f03a806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:13.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.611+0000 7f03d7e00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03a806b470 0x7f03a806d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f03c000b810 tx=0x7f03c0008040 comp rx=0 tx=0).stop 2024-01-15T18:00:13.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.611+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 msgr2=0x7f03d01f7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:13.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.611+0000 7f03d7e00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 0x7f03d01f7050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f03b802b930 tx=0x7f03b80046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.612+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 shutdown_connections 2024-01-15T18:00:13.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.612+0000 7f03d7e00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03a806b470 0x7f03a806d930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.612+0000 7f03d7e00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03d01439f0 0x7f03d013a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.612+0000 7f03d7e00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03d00ccc70 0x7f03d01f7050 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.612+0000 7f03d7e00640 1 --2- 172.21.15.2:0/3420814244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03d00b3420 0x7f03d015b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:13.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.612+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 >> 172.21.15.2:0/3420814244 conn(0x7f03d00b45f0 msgr2=0x7f03d00b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:13.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.613+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 shutdown_connections 2024-01-15T18:00:13.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:13.613+0000 7f03d7e00640 1 -- 172.21.15.2:0/3420814244 wait complete. 2024-01-15T18:00:13.644 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:14.645 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:14.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.880+0000 7f5155800640 1 Processor -- start 2024-01-15T18:00:14.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.881+0000 7f5155800640 1 -- start start 2024-01-15T18:00:14.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.882+0000 7f5155800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515015c450 0x7f515006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:14.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.882+0000 7f5155800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f515006ce30 0x7f5150070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:14.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.882+0000 7f5155800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51500ce230 con 0x7f5150144c00 2024-01-15T18:00:14.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.882+0000 7f5155800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515006d2c0 con 0x7f515006ce30 2024-01-15T18:00:14.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.882+0000 7f5155800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500ce510 con 0x7f515015c450 2024-01-15T18:00:14.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.882+0000 7f514ea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5150144c00 0x7f51500cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50570/0 (socket says 172.21.15.2:50570) 2024-01-15T18:00:14.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514ea00640 1 -- 172.21.15.2:0/2296112422 learned_addr learned my addr 172.21.15.2:0/2296112422 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:14.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514f400640 1 --2- 172.21.15.2:0/2296112422 >> v2:172.21.15.2:3300/0 conn(0x7f515006ce30 0x7f5150070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:14.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514d600640 1 -- 172.21.15.2:0/2296112422 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862320211 0 0) 0x7f51500ce230 con 0x7f5150144c00 2024-01-15T18:00:14.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514d600640 1 -- 172.21.15.2:0/2296112422 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5130003520 con 0x7f5150144c00 2024-01-15T18:00:14.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514e000640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515015c450 0x7f515006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:14.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514f400640 1 -- 172.21.15.2:0/2296112422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515015c450 msgr2=0x7f515006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:14.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514f400640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515015c450 0x7f515006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:14.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514f400640 1 -- 172.21.15.2:0/2296112422 >> v1:172.21.15.2:6789/0 conn(0x7f5150144c00 legacy=0x7f51500cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:14.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514f400640 1 -- 172.21.15.2:0/2296112422 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500ce7f0 con 0x7f515006ce30 2024-01-15T18:00:14.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.883+0000 7f514e000640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515015c450 0x7f515006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:14.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.884+0000 7f514f400640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515006ce30 0x7f5150070b10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f51480097a0 tx=0x7f514802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0a038850b3ea49 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:14.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f514d600640 1 -- 172.21.15.2:0/2296112422 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514802d040 con 0x7f515006ce30 2024-01-15T18:00:14.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f514d600640 1 -- 172.21.15.2:0/2296112422 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5148031810 con 0x7f515006ce30 2024-01-15T18:00:14.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f514d600640 1 -- 172.21.15.2:0/2296112422 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148031ba0 con 0x7f515006ce30 2024-01-15T18:00:14.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 -- 172.21.15.2:0/2296112422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515006ce30 msgr2=0x7f5150070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:14.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515006ce30 0x7f5150070b10 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f51480097a0 tx=0x7f514802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:00:14.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 -- 172.21.15.2:0/2296112422 shutdown_connections 2024-01-15T18:00:14.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515006ce30 0x7f5150070b10 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:14.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 --2- 172.21.15.2:0/2296112422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515015c450 0x7f515006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:14.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 -- 172.21.15.2:0/2296112422 >> 172.21.15.2:0/2296112422 conn(0x7f51500b45f0 msgr2=0x7f51500b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:14.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 -- 172.21.15.2:0/2296112422 shutdown_connections 2024-01-15T18:00:14.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.886+0000 7f5155800640 1 -- 172.21.15.2:0/2296112422 wait complete. 2024-01-15T18:00:14.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.887+0000 7f5155800640 1 Processor -- start 2024-01-15T18:00:14.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 -- start start 2024-01-15T18:00:14.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515006ce30 0x7f51501591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:14.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5150144c00 0x7f51500b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:14.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 0x7f51500af0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:14.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515013fb10 con 0x7f515015c450 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51501e3bf0 con 0x7f5150144c00 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.888+0000 7f5155800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515013f270 con 0x7f515006ce30 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 0x7f51500af0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 0x7f51500af0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58466/0 (socket says 172.21.15.2:58466) 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 -- 172.21.15.2:0/1810014312 learned_addr learned my addr 172.21.15.2:0/1810014312 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514ea00640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515006ce30 0x7f51501591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514e000640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5150144c00 0x7f51500b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 -- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515006ce30 msgr2=0x7f51501591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515006ce30 0x7f51501591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 -- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5150144c00 msgr2=0x7f51500b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:14.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5150144c00 0x7f51500b1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:14.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.889+0000 7f514f400640 1 -- 172.21.15.2:0/1810014312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5148009400 con 0x7f515015c450 2024-01-15T18:00:14.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.890+0000 7f514f400640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 0x7f51500af0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f51480098c0 tx=0x7f51480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.890+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514803ace0 con 0x7f515015c450 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.890+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514803ae60 con 0x7f515015c450 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.890+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515013f550 con 0x7f515015c450 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.890+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5148042630 con 0x7f515015c450 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.890+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51501596f0 con 0x7f515015c450 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.891+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51501e39c0 con 0x7f515015c450 2024-01-15T18:00:14.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.892+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f514804b3b0 con 0x7f515015c450 2024-01-15T18:00:14.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.894+0000 7f5146a00640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f512806b470 0x7f512806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:14.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.894+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51480ac180 con 0x7f515015c450 2024-01-15T18:00:14.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.894+0000 7f514ea00640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f512806b470 0x7f512806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:14.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.895+0000 7f514ea00640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f512806b470 0x7f512806d930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f51400011e0 tx=0x7f5140009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:14.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:14.898+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51501e39c0 con 0x7f515015c450 2024-01-15T18:00:15.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.131+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f51500cead0 con 0x7f512806b470 2024-01-15T18:00:15.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.149+0000 7f5146a00640 1 -- 172.21.15.2:0/1810014312 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f5140001370 con 0x7f512806b470 2024-01-15T18:00:15.152 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:15.152 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:15.153 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:15.154 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:15.154 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:15.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.151+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f512806b470 msgr2=0x7f512806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:15.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.151+0000 7f5155800640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f512806b470 0x7f512806d930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f51400011e0 tx=0x7f5140009040 comp rx=0 tx=0).stop 2024-01-15T18:00:15.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.151+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 msgr2=0x7f51500af0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:15.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 0x7f51500af0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f51480098c0 tx=0x7f51480046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:15.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 shutdown_connections 2024-01-15T18:00:15.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f512806b470 0x7f512806d930 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:15.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f515015c450 0x7f51500af0e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:15.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5150144c00 0x7f51500b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:15.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 --2- 172.21.15.2:0/1810014312 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f515006ce30 0x7f51501591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:15.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 >> 172.21.15.2:0/1810014312 conn(0x7f51500b45f0 msgr2=0x7f51501e67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:15.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 shutdown_connections 2024-01-15T18:00:15.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:15.152+0000 7f5155800640 1 -- 172.21.15.2:0/1810014312 wait complete. 2024-01-15T18:00:15.175 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:16.176 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:16.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.413+0000 7f759ee00640 1 Processor -- start 2024-01-15T18:00:16.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.413+0000 7f759ee00640 1 -- start start 2024-01-15T18:00:16.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759ee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7598160830 0x7f75980ad750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759ee00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f75980b4ae0 0x7f759806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759ee00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75980af3a0 con 0x7f759806a9f0 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759ee00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7598161d50 con 0x7f75980b4ae0 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759ee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759806d1f0 con 0x7f7598160830 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f75980b4ae0 0x7f759806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f75980b4ae0 0x7f759806a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58472/0 (socket says 172.21.15.2:58472) 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759d000640 1 -- 172.21.15.2:0/3381282310 learned_addr learned my addr 172.21.15.2:0/3381282310 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759d000640 1 -- 172.21.15.2:0/3381282310 >> v2:172.21.15.137:3300/0 conn(0x7f7598160830 msgr2=0x7f75980ad750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759d000640 1 --2- 172.21.15.2:0/3381282310 >> v2:172.21.15.137:3300/0 conn(0x7f7598160830 0x7f75980ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.415+0000 7f759d000640 1 -- 172.21.15.2:0/3381282310 >> v1:172.21.15.2:6789/0 conn(0x7f759806a9f0 legacy=0x7f75980711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:16.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.416+0000 7f759d000640 1 -- 172.21.15.2:0/3381282310 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980b31c0 con 0x7f75980b4ae0 2024-01-15T18:00:16.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.417+0000 7f759d000640 1 --2- 172.21.15.2:0/3381282310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75980b4ae0 0x7f759806a4b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7598070e80 tx=0x7f758002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e57dde8bbf5c7b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:16.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.417+0000 7f7597e00640 1 -- 172.21.15.2:0/3381282310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580004230 con 0x7f75980b4ae0 2024-01-15T18:00:16.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.417+0000 7f7597e00640 1 -- 172.21.15.2:0/3381282310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7580004390 con 0x7f75980b4ae0 2024-01-15T18:00:16.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.417+0000 7f7597e00640 1 -- 172.21.15.2:0/3381282310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7580003810 con 0x7f75980b4ae0 2024-01-15T18:00:16.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 -- 172.21.15.2:0/3381282310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75980b4ae0 msgr2=0x7f759806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:16.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 --2- 172.21.15.2:0/3381282310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75980b4ae0 0x7f759806a4b0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7598070e80 tx=0x7f758002b8f0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 -- 172.21.15.2:0/3381282310 shutdown_connections 2024-01-15T18:00:16.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 --2- 172.21.15.2:0/3381282310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75980b4ae0 0x7f759806a4b0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 --2- 172.21.15.2:0/3381282310 >> v2:172.21.15.137:3300/0 conn(0x7f7598160830 0x7f75980ad750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 -- 172.21.15.2:0/3381282310 >> 172.21.15.2:0/3381282310 conn(0x7f759806b5c0 msgr2=0x7f759806b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:16.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 -- 172.21.15.2:0/3381282310 shutdown_connections 2024-01-15T18:00:16.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.418+0000 7f759ee00640 1 -- 172.21.15.2:0/3381282310 wait complete. 2024-01-15T18:00:16.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.420+0000 7f759ee00640 1 Processor -- start 2024-01-15T18:00:16.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.420+0000 7f759ee00640 1 -- start start 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759ee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759806a9f0 0x7f759815ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759ee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75980b4ae0 0x7f7598150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759ee00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 0x7f7598150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759ee00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7598150fb0 con 0x7f7598160830 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759ee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759815b2b0 con 0x7f75980b4ae0 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759ee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759815b500 con 0x7f759806a9f0 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 0x7f7598150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:16.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 0x7f7598150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58476/0 (socket says 172.21.15.2:58476) 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759e400640 1 -- 172.21.15.2:0/1554483967 learned_addr learned my addr 172.21.15.2:0/1554483967 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759d000640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75980b4ae0 0x7f7598150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.421+0000 7f759da00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759806a9f0 0x7f759815ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759e400640 1 -- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759806a9f0 msgr2=0x7f759815ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759e400640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759806a9f0 0x7f759815ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759e400640 1 -- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75980b4ae0 msgr2=0x7f7598150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759e400640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75980b4ae0 0x7f7598150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759e400640 1 -- 172.21.15.2:0/1554483967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7580009460 con 0x7f7598160830 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759da00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759806a9f0 0x7f759815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759e400640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 0x7f7598150a70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7590001eb0 tx=0x7f759000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:16.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75900052d0 con 0x7f7598160830 2024-01-15T18:00:16.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7590005430 con 0x7f7598160830 2024-01-15T18:00:16.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7590003810 con 0x7f7598160830 2024-01-15T18:00:16.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759815b7e0 con 0x7f7598160830 2024-01-15T18:00:16.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.422+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759815bda0 con 0x7f7598160830 2024-01-15T18:00:16.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.424+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75981542d0 con 0x7f7598160830 2024-01-15T18:00:16.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.424+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f7590003970 con 0x7f7598160830 2024-01-15T18:00:16.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.425+0000 7f7595600640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757006b470 0x7f757006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:16.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.425+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7590089530 con 0x7f7598160830 2024-01-15T18:00:16.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.426+0000 7f759da00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757006b470 0x7f757006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:16.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.427+0000 7f759da00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757006b470 0x7f757006d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7588007eb0 tx=0x7f75880073c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:16.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.430+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75981542d0 con 0x7f7598160830 2024-01-15T18:00:16.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.767+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f75980b3a60 con 0x7f757006b470 2024-01-15T18:00:16.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.814+0000 7f7595600640 1 -- 172.21.15.2:0/1554483967 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f758800cdd0 con 0x7f757006b470 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:00:16.818 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:16.819 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:16.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757006b470 msgr2=0x7f757006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757006b470 0x7f757006d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7588007eb0 tx=0x7f75880073c0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 msgr2=0x7f7598150a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 0x7f7598150a70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7590001eb0 tx=0x7f759000c890 comp rx=0 tx=0).stop 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 shutdown_connections 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757006b470 0x7f757006d930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7598160830 0x7f7598150a70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75980b4ae0 0x7f7598150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 --2- 172.21.15.2:0/1554483967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759806a9f0 0x7f759815ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:16.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.817+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 >> 172.21.15.2:0/1554483967 conn(0x7f759806b5c0 msgr2=0x7f75980b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:16.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.820+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 shutdown_connections 2024-01-15T18:00:16.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:16.820+0000 7f759ee00640 1 -- 172.21.15.2:0/1554483967 wait complete. 2024-01-15T18:00:16.848 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:00:17.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:00:18.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.105+0000 7fb2eee00640 1 Processor -- start 2024-01-15T18:00:18.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.106+0000 7fb2eee00640 1 -- start start 2024-01-15T18:00:18.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.107+0000 7fb2eee00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb2e814afd0 0x7fb2e80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.107+0000 7fb2eee00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2e814b3b0 0x7fb2e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.107+0000 7fb2eee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2e80b25f0 con 0x7fb2e806ce30 2024-01-15T18:00:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.107+0000 7fb2eee00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e8070ff0 con 0x7fb2e814b3b0 2024-01-15T18:00:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.107+0000 7fb2eee00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e806d2a0 con 0x7fb2e814afd0 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.108+0000 7fb2e7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2e814b3b0 0x7fb2e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.108+0000 7fb2e7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2e814b3b0 0x7fb2e806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58482/0 (socket says 172.21.15.2:58482) 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.108+0000 7fb2e7400640 1 -- 172.21.15.2:0/980422803 learned_addr learned my addr 172.21.15.2:0/980422803 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.108+0000 7fb2e7e00640 1 --2- 172.21.15.2:0/980422803 >> v2:172.21.15.137:3300/0 conn(0x7fb2e814afd0 0x7fb2e80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.109+0000 7fb2e7400640 1 -- 172.21.15.2:0/980422803 >> v2:172.21.15.137:3300/0 conn(0x7fb2e814afd0 msgr2=0x7fb2e80b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.109+0000 7fb2e7400640 1 --2- 172.21.15.2:0/980422803 >> v2:172.21.15.137:3300/0 conn(0x7fb2e814afd0 0x7fb2e80b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.109+0000 7fb2e7400640 1 -- 172.21.15.2:0/980422803 >> v1:172.21.15.137:6789/0 conn(0x7fb2e806ce30 legacy=0x7fb2e80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.109+0000 7fb2e7400640 1 -- 172.21.15.2:0/980422803 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80b28d0 con 0x7fb2e814b3b0 2024-01-15T18:00:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.110+0000 7fb2e7400640 1 --2- 172.21.15.2:0/980422803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814b3b0 0x7fb2e806c8f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb2e806f9f0 tx=0x7fb2d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861b8398d84d68af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.111+0000 7fb2e6a00640 1 -- 172.21.15.2:0/980422803 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00317e0 con 0x7fb2e814b3b0 2024-01-15T18:00:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.111+0000 7fb2e6a00640 1 -- 172.21.15.2:0/980422803 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d0031940 con 0x7fb2e814b3b0 2024-01-15T18:00:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.111+0000 7fb2e6a00640 1 -- 172.21.15.2:0/980422803 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0031cd0 con 0x7fb2e814b3b0 2024-01-15T18:00:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 -- 172.21.15.2:0/980422803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814b3b0 msgr2=0x7fb2e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 --2- 172.21.15.2:0/980422803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814b3b0 0x7fb2e806c8f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb2e806f9f0 tx=0x7fb2d002b880 comp rx=0 tx=0).stop 2024-01-15T18:00:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 -- 172.21.15.2:0/980422803 shutdown_connections 2024-01-15T18:00:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 --2- 172.21.15.2:0/980422803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814b3b0 0x7fb2e806c8f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 --2- 172.21.15.2:0/980422803 >> v2:172.21.15.137:3300/0 conn(0x7fb2e814afd0 0x7fb2e80b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 -- 172.21.15.2:0/980422803 >> 172.21.15.2:0/980422803 conn(0x7fb2e80b45f0 msgr2=0x7fb2e80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:18.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 -- 172.21.15.2:0/980422803 shutdown_connections 2024-01-15T18:00:18.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.112+0000 7fb2eee00640 1 -- 172.21.15.2:0/980422803 wait complete. 2024-01-15T18:00:18.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.115+0000 7fb2eee00640 1 Processor -- start 2024-01-15T18:00:18.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.115+0000 7fb2eee00640 1 -- start start 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2eee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2e806ce30 0x7fb2e8144310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2eee00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 0x7fb2e81f8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2eee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2e814b3b0 0x7fb2e80cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2eee00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80a8460 con 0x7fb2e814afd0 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2eee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80a86d0 con 0x7fb2e806ce30 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2eee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e80cd0c0 con 0x7fb2e814b3b0 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 0x7fb2e81f8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2e7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 0x7fb2e81f8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58492/0 (socket says 172.21.15.2:58492) 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.117+0000 7fb2e7400640 1 -- 172.21.15.2:0/1366903516 learned_addr learned my addr 172.21.15.2:0/1366903516 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2ece00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2e814b3b0 0x7fb2e80cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2e7400640 1 -- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2e814b3b0 msgr2=0x7fb2e80cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2e7400640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2e814b3b0 0x7fb2e80cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2e7400640 1 -- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2e806ce30 msgr2=0x7fb2e8144310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2e7400640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2e806ce30 0x7fb2e8144310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2e7400640 1 -- 172.21.15.2:0/1366903516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d00094d0 con 0x7fb2e814afd0 2024-01-15T18:00:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.118+0000 7fb2e7400640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 0x7fb2e81f8090 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000c00 tx=0x7fb2d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.119+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00329c0 con 0x7fb2e814afd0 2024-01-15T18:00:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.119+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d0032b40 con 0x7fb2e814afd0 2024-01-15T18:00:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.119+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00453f0 con 0x7fb2e814afd0 2024-01-15T18:00:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.119+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e80cd3a0 con 0x7fb2e814afd0 2024-01-15T18:00:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.119+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e80ce030 con 0x7fb2e814afd0 2024-01-15T18:00:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.121+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e80ce270 con 0x7fb2e814afd0 2024-01-15T18:00:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.122+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fb2d004d3f0 con 0x7fb2e814afd0 2024-01-15T18:00:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.123+0000 7fb2dfe00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c006b4c0 0x7fb2c006d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.124+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2d00af120 con 0x7fb2e814afd0 2024-01-15T18:00:18.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.128+0000 7fb2e7e00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c006b4c0 0x7fb2c006d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.129+0000 7fb2e7e00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c006b4c0 0x7fb2c006d980 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800a890 tx=0x7fb2d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:18.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.130+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2e80ce270 con 0x7fb2e814afd0 2024-01-15T18:00:18.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.530+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2e80b2e90 con 0x7fb2c006b4c0 2024-01-15T18:00:18.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.555+0000 7fb2dfe00640 1 -- 172.21.15.2:0/1366903516 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fb2d800ae70 con 0x7fb2c006b4c0 2024-01-15T18:00:18.558 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:18.558 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:00:18.558 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:00:18.558 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:18.558 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:18.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.557+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c006b4c0 msgr2=0x7fb2c006d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.557+0000 7fb2eee00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c006b4c0 0x7fb2c006d980 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800a890 tx=0x7fb2d8008040 comp rx=0 tx=0).stop 2024-01-15T18:00:18.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.557+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 msgr2=0x7fb2e81f8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.557+0000 7fb2eee00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 0x7fb2e81f8090 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0000c00 tx=0x7fb2d00046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 shutdown_connections 2024-01-15T18:00:18.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c006b4c0 0x7fb2c006d980 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2e814b3b0 0x7fb2e80cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2e814afd0 0x7fb2e81f8090 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 --2- 172.21.15.2:0/1366903516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2e806ce30 0x7fb2e8144310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 >> 172.21.15.2:0/1366903516 conn(0x7fb2e80b45f0 msgr2=0x7fb2e80b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 shutdown_connections 2024-01-15T18:00:18.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.558+0000 7fb2eee00640 1 -- 172.21.15.2:0/1366903516 wait complete. 2024-01-15T18:00:18.580 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:00:18.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.773+0000 7f893d800640 1 Processor -- start 2024-01-15T18:00:18.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.773+0000 7f893d800640 1 -- start start 2024-01-15T18:00:18.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f893d800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f893806d350 0x7f893806d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f893d800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f89380ce580 0x7f893815d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f893d800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893806dc70 con 0x7f8938160e50 2024-01-15T18:00:18.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f893d800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380cb3a0 con 0x7f893806d350 2024-01-15T18:00:18.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f893d800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380cb680 con 0x7f89380ce580 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f8937e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f893806d350 0x7f893806d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f8937400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f8938160e50 0x7f89380cde70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50596/0 (socket says 172.21.15.2:50596) 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.775+0000 7f8937400640 1 -- 172.21.15.2:0/2163910686 learned_addr learned my addr 172.21.15.2:0/2163910686 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.776+0000 7f893ce00640 1 --2- 172.21.15.2:0/2163910686 >> v2:172.21.15.137:3300/0 conn(0x7f89380ce580 0x7f893815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.776+0000 7f8937e00640 1 -- 172.21.15.2:0/2163910686 >> v2:172.21.15.137:3300/0 conn(0x7f89380ce580 msgr2=0x7f893815d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.776+0000 7f8937e00640 1 --2- 172.21.15.2:0/2163910686 >> v2:172.21.15.137:3300/0 conn(0x7f89380ce580 0x7f893815d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.776+0000 7f8937e00640 1 -- 172.21.15.2:0/2163910686 >> v1:172.21.15.2:6789/0 conn(0x7f8938160e50 legacy=0x7f89380cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.776+0000 7f8937e00640 1 -- 172.21.15.2:0/2163910686 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380cb960 con 0x7f893806d350 2024-01-15T18:00:18.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.776+0000 7f8936a00640 1 -- 172.21.15.2:0/2163910686 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431039727 0 0) 0x7f893806dc70 con 0x7f8938160e50 2024-01-15T18:00:18.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.777+0000 7f8937e00640 1 --2- 172.21.15.2:0/2163910686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f893806d350 0x7f893806d730 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8938070060 tx=0x7f892802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ba05aa10ac5772 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.777+0000 7f8936a00640 1 -- 172.21.15.2:0/2163910686 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928031830 con 0x7f893806d350 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.777+0000 7f8936a00640 1 -- 172.21.15.2:0/2163910686 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8928031990 con 0x7f893806d350 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f8936a00640 1 -- 172.21.15.2:0/2163910686 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928031d20 con 0x7f893806d350 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 -- 172.21.15.2:0/2163910686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f893806d350 msgr2=0x7f893806d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 --2- 172.21.15.2:0/2163910686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f893806d350 0x7f893806d730 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8938070060 tx=0x7f892802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 -- 172.21.15.2:0/2163910686 shutdown_connections 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 --2- 172.21.15.2:0/2163910686 >> v2:172.21.15.137:3300/0 conn(0x7f89380ce580 0x7f893815d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 --2- 172.21.15.2:0/2163910686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f893806d350 0x7f893806d730 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 -- 172.21.15.2:0/2163910686 >> 172.21.15.2:0/2163910686 conn(0x7f89380b0840 msgr2=0x7f89380b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 -- 172.21.15.2:0/2163910686 shutdown_connections 2024-01-15T18:00:18.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.778+0000 7f893d800640 1 -- 172.21.15.2:0/2163910686 wait complete. 2024-01-15T18:00:18.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.780+0000 7f893d800640 1 Processor -- start 2024-01-15T18:00:18.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.780+0000 7f893d800640 1 -- start start 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.781+0000 7f893d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f893806d350 0x7f8938141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.781+0000 7f893d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89380ce580 0x7f8938151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.781+0000 7f893d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 0x7f893815f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.781+0000 7f893d800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8938152f20 con 0x7f8938160e50 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.781+0000 7f893d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8938152310 con 0x7f89380ce580 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.781+0000 7f893d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89381525f0 con 0x7f893806d350 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 0x7f893815f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 0x7f893815f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58502/0 (socket says 172.21.15.2:58502) 2024-01-15T18:00:18.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 -- 172.21.15.2:0/1256481370 learned_addr learned my addr 172.21.15.2:0/1256481370 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f8937e00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f893806d350 0x7f8938141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f8937400640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89380ce580 0x7f8938151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 -- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f893806d350 msgr2=0x7f8938141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f893806d350 0x7f8938141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 -- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89380ce580 msgr2=0x7f8938151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89380ce580 0x7f8938151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f893ce00640 1 -- 172.21.15.2:0/1256481370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8928009400 con 0x7f8938160e50 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.782+0000 7f8937400640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89380ce580 0x7f8938151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.783+0000 7f893ce00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 0x7f893815f710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8930007e40 tx=0x7f893000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.783+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8930014620 con 0x7f8938160e50 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.783+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893000ee90 con 0x7f8938160e50 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.783+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89300149e0 con 0x7f8938160e50 2024-01-15T18:00:18.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.783+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89381528d0 con 0x7f8938160e50 2024-01-15T18:00:18.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.783+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89381533b0 con 0x7f8938160e50 2024-01-15T18:00:18.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.784+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893806bfa0 con 0x7f8938160e50 2024-01-15T18:00:18.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.785+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f8930014b40 con 0x7f8938160e50 2024-01-15T18:00:18.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.786+0000 7f892fe00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f891006b470 0x7f891006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:18.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.786+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f893008a930 con 0x7f8938160e50 2024-01-15T18:00:18.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.787+0000 7f8937e00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f891006b470 0x7f891006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:18.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.788+0000 7f8937e00640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f891006b470 0x7f891006d930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8928036000 tx=0x7f8928008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:18.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:18.791+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8930055480 con 0x7f8938160e50 2024-01-15T18:00:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.015+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f89380cbf20 con 0x7f891006b470 2024-01-15T18:00:19.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.018+0000 7f892fe00640 1 -- 172.21.15.2:0/1256481370 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8928031b00 con 0x7f891006b470 2024-01-15T18:00:19.022 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:00:19.022 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:19.022 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:00:19.022 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:19.022 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f891006b470 msgr2=0x7f891006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f891006b470 0x7f891006d930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8928036000 tx=0x7f8928008030 comp rx=0 tx=0).stop 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 msgr2=0x7f893815f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 0x7f893815f710 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8930007e40 tx=0x7f893000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 shutdown_connections 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f891006b470 0x7f891006d930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8938160e50 0x7f893815f710 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89380ce580 0x7f8938151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 --2- 172.21.15.2:0/1256481370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f893806d350 0x7f8938141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.021+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 >> 172.21.15.2:0/1256481370 conn(0x7f89380b0840 msgr2=0x7f89380ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.022+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 shutdown_connections 2024-01-15T18:00:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.022+0000 7f893d800640 1 -- 172.21.15.2:0/1256481370 wait complete. 2024-01-15T18:00:19.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:00:19.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.265+0000 7f66ba800640 1 Processor -- start 2024-01-15T18:00:19.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.265+0000 7f66ba800640 1 -- start start 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66ba800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 0x7f66b406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66ba800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66b40720d0 0x7f66b406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66ba800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b4071090 con 0x7f66b4143d20 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66ba800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b4071430 con 0x7f66b40720d0 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66ba800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4144100 con 0x7f66b415f260 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66b9400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f66b4143d20 0x7f66b415eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50604/0 (socket says 172.21.15.2:50604) 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.267+0000 7f66b9400640 1 -- 172.21.15.2:0/648550069 learned_addr learned my addr 172.21.15.2:0/648550069 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:19.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b9e00640 1 --2- 172.21.15.2:0/648550069 >> v2:172.21.15.2:3300/0 conn(0x7f66b40720d0 0x7f66b406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b8a00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 0x7f66b406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96998473 0 0) 0x7f66b4071090 con 0x7f66b4143d20 2024-01-15T18:00:19.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6694003520 con 0x7f66b4143d20 2024-01-15T18:00:19.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743893331 0 0) 0x7f6694003520 con 0x7f66b4143d20 2024-01-15T18:00:19.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 msgr2=0x7f66b406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:19.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 0x7f66b406c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 >> v2:172.21.15.2:3300/0 conn(0x7f66b40720d0 msgr2=0x7f66b406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:19.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 --2- 172.21.15.2:0/648550069 >> v2:172.21.15.2:3300/0 conn(0x7f66b40720d0 0x7f66b406c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b4144560 con 0x7f66b4143d20 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.268+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f66a4001670 con 0x7f66b4143d20 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 >> v1:172.21.15.2:6789/0 conn(0x7f66b4143d20 legacy=0x7f66b415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 0x7f6694005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4071430 con 0x7f66940047c0 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b4144100 con 0x7f6694003970 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669400ba60 con 0x7f66b415f260 2024-01-15T18:00:19.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b9e00640 1 --2- 172.21.15.2:0/648550069 >> v2:172.21.15.2:3300/0 conn(0x7f66b40720d0 0x7f66b406c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b8a00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f6694004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b9400640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b8a00640 1 -- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 msgr2=0x7f6694005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b8a00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b415f260 0x7f6694005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b8a00640 1 -- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6694003970 msgr2=0x7f6694003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.269+0000 7f66b8a00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6694003970 0x7f6694003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.270+0000 7f66b8a00640 1 -- 172.21.15.2:0/648550069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6694003520 con 0x7f66940047c0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66b8a00640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f6694004c40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f669c009a90 tx=0x7f66940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d90dba48c4707ce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c02c6b0 con 0x7f66940047c0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669c02c810 con 0x7f66940047c0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66b3e00640 1 -- 172.21.15.2:0/648550069 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669c02cba0 con 0x7f66940047c0 2024-01-15T18:00:19.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66ba800640 1 -- 172.21.15.2:0/648550069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 msgr2=0x7f6694004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66ba800640 1 --2- 172.21.15.2:0/648550069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f6694004c40 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f669c009a90 tx=0x7f66940057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.271+0000 7f66b9e00640 1 -- 172.21.15.2:0/648550069 reap_dead start 2024-01-15T18:00:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.272+0000 7f66ba800640 1 -- 172.21.15.2:0/648550069 shutdown_connections 2024-01-15T18:00:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.272+0000 7f66ba800640 1 -- 172.21.15.2:0/648550069 >> 172.21.15.2:0/648550069 conn(0x7f66b40b0840 msgr2=0x7f66b40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.272+0000 7f66ba800640 1 -- 172.21.15.2:0/648550069 shutdown_connections 2024-01-15T18:00:19.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.272+0000 7f66ba800640 1 -- 172.21.15.2:0/648550069 wait complete. 2024-01-15T18:00:19.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.274+0000 7f66ba800640 1 Processor -- start 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.274+0000 7f66ba800640 1 -- start start 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66ba800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f66b41f6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66ba800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66b415b5a0 0x7f66b40ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66ba800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b40ade00 0x7f66b41f1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66ba800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b415c630 con 0x7f66940047c0 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66ba800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b40ae290 con 0x7f66b415b5a0 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66ba800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b41f6c40 con 0x7f66b40ade00 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f66b41f6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f66b41f6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58522/0 (socket says 172.21.15.2:58522) 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 -- 172.21.15.2:0/2294219973 learned_addr learned my addr 172.21.15.2:0/2294219973 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:19.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b8a00640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66b415b5a0 0x7f66b40ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 -- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b40ade00 msgr2=0x7f66b41f1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b40ade00 0x7f66b41f1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 -- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66b415b5a0 msgr2=0x7f66b40ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66b415b5a0 0x7f66b40ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.275+0000 7f66b9400640 1 -- 172.21.15.2:0/2294219973 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b415bab0 con 0x7f66940047c0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.276+0000 7f66b9400640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f66b41f6700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f66a400e700 tx=0x7f66a400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.276+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a400ee10 con 0x7f66940047c0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.276+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a400c7f0 con 0x7f66940047c0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.276+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a400cba0 con 0x7f66940047c0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.276+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b415bd90 con 0x7f66940047c0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.276+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b415c350 con 0x7f66940047c0 2024-01-15T18:00:19.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.277+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b40c4e40 con 0x7f66940047c0 2024-01-15T18:00:19.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.278+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f66a400cd00 con 0x7f66940047c0 2024-01-15T18:00:19.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.279+0000 7f66b1600640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668c06b470 0x7f668c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.279+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66a4089eb0 con 0x7f66940047c0 2024-01-15T18:00:19.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.280+0000 7f66b8a00640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668c06b470 0x7f668c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.281+0000 7f66b8a00640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668c06b470 0x7f668c06d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f669c036000 tx=0x7f669c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:19.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.284+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66a4054a00 con 0x7f66940047c0 2024-01-15T18:00:19.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.504+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66b415fa00 con 0x7f668c06b470 2024-01-15T18:00:19.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.515+0000 7f66b1600640 1 -- 172.21.15.2:0/2294219973 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f669c004110 con 0x7f668c06b470 2024-01-15T18:00:19.518 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:19.518 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:00:19.518 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:19.519 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:00:19.519 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:19.519 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:19.519 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1clone1" 2024-01-15T18:00:19.519 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:19.519 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:00:19.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668c06b470 msgr2=0x7f668c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668c06b470 0x7f668c06d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f669c036000 tx=0x7f669c030000 comp rx=0 tx=0).stop 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 msgr2=0x7f66b41f6700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f66b41f6700 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f66a400e700 tx=0x7f66a400ebc0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 shutdown_connections 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f668c06b470 0x7f668c06d930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66b40ade00 0x7f66b41f1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66b415b5a0 0x7f66b40ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 --2- 172.21.15.2:0/2294219973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66940047c0 0x7f66b41f6700 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 >> 172.21.15.2:0/2294219973 conn(0x7f66b40b0840 msgr2=0x7f66b406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 shutdown_connections 2024-01-15T18:00:19.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.518+0000 7f66ba800640 1 -- 172.21.15.2:0/2294219973 wait complete. 2024-01-15T18:00:19.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:00:19.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.762+0000 7f1682800640 1 Processor -- start 2024-01-15T18:00:19.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.762+0000 7f1682800640 1 -- start start 2024-01-15T18:00:19.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1682800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f167c06d000 0x7f167c0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1682800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f167c0b2430 0x7f167c0b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1682800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f167c15dbc0 con 0x7f167c15ae80 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1682800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c15e840 con 0x7f167c06d000 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1682800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f167c15dea0 con 0x7f167c0b2430 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1673e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f167c06d000 0x7f167c0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1673e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f167c06d000 0x7f167c0acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58532/0 (socket says 172.21.15.2:58532) 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f1673e00640 1 -- 172.21.15.2:0/1311935 learned_addr learned my addr 172.21.15.2:0/1311935 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.764+0000 7f167b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f167c15ae80 0x7f167c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50614/0 (socket says 172.21.15.2:50614) 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.765+0000 7f167be00640 1 --2- 172.21.15.2:0/1311935 >> v2:172.21.15.137:3300/0 conn(0x7f167c0b2430 0x7f167c0b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.765+0000 7f1673e00640 1 -- 172.21.15.2:0/1311935 >> v2:172.21.15.137:3300/0 conn(0x7f167c0b2430 msgr2=0x7f167c0b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:19.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.765+0000 7f1673e00640 1 --2- 172.21.15.2:0/1311935 >> v2:172.21.15.137:3300/0 conn(0x7f167c0b2430 0x7f167c0b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.765+0000 7f1673e00640 1 -- 172.21.15.2:0/1311935 >> v1:172.21.15.2:6789/0 conn(0x7f167c15ae80 legacy=0x7f167c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.765+0000 7f1673e00640 1 -- 172.21.15.2:0/1311935 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c15e180 con 0x7f167c06d000 2024-01-15T18:00:19.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.765+0000 7f167be00640 1 --2- 172.21.15.2:0/1311935 >> v2:172.21.15.137:3300/0 conn(0x7f167c0b2430 0x7f167c0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:19.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.766+0000 7f1673e00640 1 --2- 172.21.15.2:0/1311935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c06d000 0x7f167c0acd00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f167c0cf9a0 tx=0x7f166402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d7e8f898a1cd85 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:19.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.766+0000 7f167aa00640 1 -- 172.21.15.2:0/1311935 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664030860 con 0x7f167c06d000 2024-01-15T18:00:19.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.766+0000 7f167aa00640 1 -- 172.21.15.2:0/1311935 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16640309c0 con 0x7f167c06d000 2024-01-15T18:00:19.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.766+0000 7f167aa00640 1 -- 172.21.15.2:0/1311935 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1664030d50 con 0x7f167c06d000 2024-01-15T18:00:19.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 -- 172.21.15.2:0/1311935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c06d000 msgr2=0x7f167c0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 --2- 172.21.15.2:0/1311935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c06d000 0x7f167c0acd00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f167c0cf9a0 tx=0x7f166402b9a0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 -- 172.21.15.2:0/1311935 shutdown_connections 2024-01-15T18:00:19.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 --2- 172.21.15.2:0/1311935 >> v2:172.21.15.137:3300/0 conn(0x7f167c0b2430 0x7f167c0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 --2- 172.21.15.2:0/1311935 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c06d000 0x7f167c0acd00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 -- 172.21.15.2:0/1311935 >> 172.21.15.2:0/1311935 conn(0x7f167c0b45f0 msgr2=0x7f167c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:19.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 -- 172.21.15.2:0/1311935 shutdown_connections 2024-01-15T18:00:19.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.767+0000 7f1682800640 1 -- 172.21.15.2:0/1311935 wait complete. 2024-01-15T18:00:19.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.769+0000 7f1682800640 1 Processor -- start 2024-01-15T18:00:19.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.769+0000 7f1682800640 1 -- start start 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f1682800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f167c06d000 0x7f167c141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f1682800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f167c0b2430 0x7f167c0b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f1682800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 0x7f167c1598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f1682800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c141550 con 0x7f167c15ae80 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f1682800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c158800 con 0x7f167c06d000 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f1682800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167c158ae0 con 0x7f167c0b2430 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f167be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 0x7f167c1598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f167be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 0x7f167c1598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58544/0 (socket says 172.21.15.2:58544) 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.770+0000 7f167be00640 1 -- 172.21.15.2:0/575366798 learned_addr learned my addr 172.21.15.2:0/575366798 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167b400640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f167c06d000 0x7f167c141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f1673e00640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f167c0b2430 0x7f167c0b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167be00640 1 -- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f167c0b2430 msgr2=0x7f167c0b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167be00640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f167c0b2430 0x7f167c0b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167be00640 1 -- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f167c06d000 msgr2=0x7f167c141010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167be00640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f167c06d000 0x7f167c141010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167be00640 1 -- 172.21.15.2:0/575366798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640094d0 con 0x7f167c15ae80 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f1673e00640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f167c0b2430 0x7f167c0b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.771+0000 7f167be00640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 0x7f167c1598a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f167400cef0 tx=0x7f167400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.772+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167400d840 con 0x7f167c15ae80 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.772+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167400d9c0 con 0x7f167c15ae80 2024-01-15T18:00:19.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.772+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167400dd70 con 0x7f167c15ae80 2024-01-15T18:00:19.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.772+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c158dc0 con 0x7f167c15ae80 2024-01-15T18:00:19.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.772+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c159380 con 0x7f167c15ae80 2024-01-15T18:00:19.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.773+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f167c0686f0 con 0x7f167c15ae80 2024-01-15T18:00:19.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.774+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f167401e3b0 con 0x7f167c15ae80 2024-01-15T18:00:19.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.775+0000 7f1673400640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165406b4f0 0x7f165406d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:19.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.775+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f167408a9e0 con 0x7f167c15ae80 2024-01-15T18:00:19.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.776+0000 7f167b400640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165406b4f0 0x7f165406d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:19.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.777+0000 7f167b400640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165406b4f0 0x7f165406d9b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f166c000f60 tx=0x7f166c007c80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:19.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:19.779+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1674055670 con 0x7f167c15ae80 2024-01-15T18:00:20.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.006+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f167c0b2d90 con 0x7f165406b4f0 2024-01-15T18:00:20.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.039+0000 7f1673400640 1 -- 172.21.15.2:0/575366798 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f167c0b2d90 con 0x7f165406b4f0 2024-01-15T18:00:20.043 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:20.043 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:00:20.043 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:20.043 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:00:20.043 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:20.043 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165406b4f0 msgr2=0x7f165406d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165406b4f0 0x7f165406d9b0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f166c000f60 tx=0x7f166c007c80 comp rx=0 tx=0).stop 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 msgr2=0x7f167c1598a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 0x7f167c1598a0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f167400cef0 tx=0x7f167400a710 comp rx=0 tx=0).stop 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 shutdown_connections 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f165406b4f0 0x7f165406d9b0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f167c15ae80 0x7f167c1598a0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f167c0b2430 0x7f167c0b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 --2- 172.21.15.2:0/575366798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f167c06d000 0x7f167c141010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 >> 172.21.15.2:0/575366798 conn(0x7f167c0b45f0 msgr2=0x7f167c070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:20.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 shutdown_connections 2024-01-15T18:00:20.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.043+0000 7f1682800640 1 -- 172.21.15.2:0/575366798 wait complete. 2024-01-15T18:00:20.067 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:00:20.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.295+0000 7f7046c00640 1 Processor -- start 2024-01-15T18:00:20.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.295+0000 7f7046c00640 1 -- start start 2024-01-15T18:00:20.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.297+0000 7f7046c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 0x7f7040071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.297+0000 7f7046c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704006c500 con 0x7f704006da80 2024-01-15T18:00:20.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.297+0000 7f7046c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704015b8a0 con 0x7f70400704c0 2024-01-15T18:00:20.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.297+0000 7f7046c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704015c4b0 con 0x7f70400ae260 2024-01-15T18:00:20.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.297+0000 7f7044e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f704006da80 0x7f704006df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50628/0 (socket says 172.21.15.2:50628) 2024-01-15T18:00:20.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.297+0000 7f7044e00640 1 -- 172.21.15.2:0/1398993136 learned_addr learned my addr 172.21.15.2:0/1398993136 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:20.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f7045800640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 0x7f7040071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872861165 0 0) 0x7f7028001110 con 0x7f704006da80 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7020003520 con 0x7f704006da80 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429047824 0 0) 0x7f7020003520 con 0x7f704006da80 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 msgr2=0x7f7040071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 0x7f7040071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 >> v1:172.21.15.137:6789/0 conn(0x7f70400704c0 legacy=0x7f7040070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704015bb80 con 0x7f704006da80 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7028001800 con 0x7f704006da80 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 >> v1:172.21.15.2:6789/0 conn(0x7f704006da80 legacy=0x7f704006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70200039a0 0x7f7020003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70200047c0 0x7f7020004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 0x7f70200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704015c4b0 con 0x7f70200039a0 2024-01-15T18:00:20.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f702000bab0 con 0x7f70200047c0 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.298+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702000bc70 con 0x7f70400ae260 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7045800640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70200039a0 0x7f7020003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7044e00640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70200047c0 0x7f7020004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7045800640 1 -- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 msgr2=0x7f70200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7045800640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70400ae260 0x7f70200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7045800640 1 -- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70200047c0 msgr2=0x7f7020004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7045800640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70200047c0 0x7f7020004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.299+0000 7f7045800640 1 -- 172.21.15.2:0/1398993136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020003520 con 0x7f70200039a0 2024-01-15T18:00:20.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.300+0000 7f7045800640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70200039a0 0x7f7020003dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7030009fc0 tx=0x7f70200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8137e486741af0d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:20.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.300+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703002cc60 con 0x7f70200039a0 2024-01-15T18:00:20.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.300+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703000f040 con 0x7f70200039a0 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.300+0000 7f703fe00640 1 -- 172.21.15.2:0/1398993136 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030034440 con 0x7f70200039a0 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046c00640 1 -- 172.21.15.2:0/1398993136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70200039a0 msgr2=0x7f7020003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046c00640 1 --2- 172.21.15.2:0/1398993136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70200039a0 0x7f7020003dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f7030009fc0 tx=0x7f70200057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046200640 1 -- 172.21.15.2:0/1398993136 reap_dead start 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046c00640 1 -- 172.21.15.2:0/1398993136 shutdown_connections 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046c00640 1 -- 172.21.15.2:0/1398993136 >> 172.21.15.2:0/1398993136 conn(0x7f70400cebc0 msgr2=0x7f70400cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046c00640 1 -- 172.21.15.2:0/1398993136 shutdown_connections 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.301+0000 7f7046c00640 1 -- 172.21.15.2:0/1398993136 wait complete. 2024-01-15T18:00:20.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.303+0000 7f7046c00640 1 Processor -- start 2024-01-15T18:00:20.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.303+0000 7f7046c00640 1 -- start start 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7046c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70200039a0 0x7f7040150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7046c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 0x7f7040152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7046c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70400cd600 0x7f70400cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7046c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70400cdfc0 con 0x7f70401f8130 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7046c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70401f8630 con 0x7f70400cd600 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7046c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7040152b60 con 0x7f70200039a0 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7044e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 0x7f7040152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7044e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 0x7f7040152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58568/0 (socket says 172.21.15.2:58568) 2024-01-15T18:00:20.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.304+0000 7f7044e00640 1 -- 172.21.15.2:0/3741535007 learned_addr learned my addr 172.21.15.2:0/3741535007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7045800640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70200039a0 0x7f7040150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7044e00640 1 -- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70200039a0 msgr2=0x7f7040150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7044e00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70200039a0 0x7f7040150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7044e00640 1 -- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70400cd600 msgr2=0x7f70400cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7046200640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70400cd600 0x7f70400cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7044e00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70400cd600 0x7f70400cda80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7044e00640 1 -- 172.21.15.2:0/3741535007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7040152e40 con 0x7f70401f8130 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.305+0000 7f7044e00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 0x7f7040152260 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f702800d840 tx=0x7f702800dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.306+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280074e0 con 0x7f70401f8130 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.306+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7028007640 con 0x7f70401f8130 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.306+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7040153120 con 0x7f70401f8130 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.306+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70280049e0 con 0x7f70401f8130 2024-01-15T18:00:20.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.306+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70400b35d0 con 0x7f70401f8130 2024-01-15T18:00:20.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.307+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7040153340 con 0x7f70401f8130 2024-01-15T18:00:20.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.308+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f7028004b40 con 0x7f70401f8130 2024-01-15T18:00:20.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.309+0000 7f703d600640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f701806b4a0 0x7f701806d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.309+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7028089bf0 con 0x7f70401f8130 2024-01-15T18:00:20.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.310+0000 7f7045800640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f701806b4a0 0x7f701806d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.311+0000 7f7045800640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f701806b4a0 0x7f701806d960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7030036000 tx=0x7f703002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.313+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70280546c0 con 0x7f70401f8130 2024-01-15T18:00:20.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.543+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f704015ae00 con 0x7f701806b4a0 2024-01-15T18:00:20.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.609+0000 7f703d600640 1 -- 172.21.15.2:0/3741535007 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f704015ae00 con 0x7f701806b4a0 2024-01-15T18:00:20.612 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:20.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.612+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f701806b4a0 msgr2=0x7f701806d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.612+0000 7f7046c00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f701806b4a0 0x7f701806d960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7030036000 tx=0x7f703002e000 comp rx=0 tx=0).stop 2024-01-15T18:00:20.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.612+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 msgr2=0x7f7040152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.612+0000 7f7046c00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 0x7f7040152260 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f702800d840 tx=0x7f702800dd00 comp rx=0 tx=0).stop 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 shutdown_connections 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f701806b4a0 0x7f701806d960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70400cd600 0x7f70400cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70401f8130 0x7f7040152260 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 --2- 172.21.15.2:0/3741535007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70200039a0 0x7f7040150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 >> 172.21.15.2:0/3741535007 conn(0x7f70400cebc0 msgr2=0x7f704015a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 shutdown_connections 2024-01-15T18:00:20.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.613+0000 7f7046c00640 1 -- 172.21.15.2:0/3741535007 wait complete. 2024-01-15T18:00:20.638 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:00:20.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.855+0000 7f506e400640 1 Processor -- start 2024-01-15T18:00:20.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.855+0000 7f506e400640 1 -- start start 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f506e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50680b2e30 0x7f50680ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f506e400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50680cf000 0x7f50680b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f506e400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506806a4b0 con 0x7f506815c130 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f506e400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5068155b20 con 0x7f50680cf000 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f506e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50680cf470 con 0x7f50680b2e30 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f5067e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50680cf000 0x7f50680b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f5067e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50680cf000 0x7f50680b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58576/0 (socket says 172.21.15.2:58576) 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.857+0000 7f5067e00640 1 -- 172.21.15.2:0/974870226 learned_addr learned my addr 172.21.15.2:0/974870226 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:20.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.858+0000 7f505fe00640 1 --2- 172.21.15.2:0/974870226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50680b2e30 0x7f50680ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.858+0000 7f5067e00640 1 -- 172.21.15.2:0/974870226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50680b2e30 msgr2=0x7f50680ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.858+0000 7f5067e00640 1 --2- 172.21.15.2:0/974870226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50680b2e30 0x7f50680ceac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.858+0000 7f5067e00640 1 -- 172.21.15.2:0/974870226 >> v1:172.21.15.137:6789/0 conn(0x7f506815c130 legacy=0x7f50680b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.858+0000 7f5067e00640 1 -- 172.21.15.2:0/974870226 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506815ed80 con 0x7f50680cf000 2024-01-15T18:00:20.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.860+0000 7f5067e00640 1 --2- 172.21.15.2:0/974870226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680cf000 0x7f50680b4670 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5060009e50 tx=0x7f506002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a33940097af354 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.860+0000 7f5066a00640 1 -- 172.21.15.2:0/974870226 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506002ca70 con 0x7f50680cf000 2024-01-15T18:00:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.860+0000 7f5066a00640 1 -- 172.21.15.2:0/974870226 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506002cbd0 con 0x7f50680cf000 2024-01-15T18:00:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.860+0000 7f5066a00640 1 -- 172.21.15.2:0/974870226 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600343f0 con 0x7f50680cf000 2024-01-15T18:00:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 -- 172.21.15.2:0/974870226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680cf000 msgr2=0x7f50680b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 --2- 172.21.15.2:0/974870226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680cf000 0x7f50680b4670 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f5060009e50 tx=0x7f506002bed0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 -- 172.21.15.2:0/974870226 shutdown_connections 2024-01-15T18:00:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 --2- 172.21.15.2:0/974870226 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680cf000 0x7f50680b4670 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 --2- 172.21.15.2:0/974870226 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50680b2e30 0x7f50680ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 -- 172.21.15.2:0/974870226 >> 172.21.15.2:0/974870226 conn(0x7f50680cc170 msgr2=0x7f50680cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:20.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 -- 172.21.15.2:0/974870226 shutdown_connections 2024-01-15T18:00:20.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.861+0000 7f506e400640 1 -- 172.21.15.2:0/974870226 wait complete. 2024-01-15T18:00:20.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.863+0000 7f506e400640 1 Processor -- start 2024-01-15T18:00:20.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.863+0000 7f506e400640 1 -- start start 2024-01-15T18:00:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.864+0000 7f506e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 0x7f50681460c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.864+0000 7f506e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50680cf000 0x7f5068146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.864+0000 7f506e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f506815c130 0x7f50680cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.864+0000 7f506e400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f506815d320 con 0x7f50680b2e30 2024-01-15T18:00:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.864+0000 7f506e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068147060 con 0x7f50680cf000 2024-01-15T18:00:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.864+0000 7f506e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506815d5c0 con 0x7f506815c130 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 0x7f50681460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 0x7f50681460c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58590/0 (socket says 172.21.15.2:58590) 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 -- 172.21.15.2:0/3037888270 learned_addr learned my addr 172.21.15.2:0/3037888270 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f505fe00640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50680cf000 0x7f5068146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067e00640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f506815c130 0x7f50680cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 -- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f506815c130 msgr2=0x7f50680cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f506815c130 0x7f50680cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 -- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50680cf000 msgr2=0x7f5068146b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50680cf000 0x7f5068146b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:20.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.865+0000 7f5067400640 1 -- 172.21.15.2:0/3037888270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5060009b30 con 0x7f50680b2e30 2024-01-15T18:00:20.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.866+0000 7f5067400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 0x7f50681460c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f505800c400 tx=0x7f505800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:20.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.866+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50580052d0 con 0x7f50680b2e30 2024-01-15T18:00:20.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.866+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058005450 con 0x7f50680b2e30 2024-01-15T18:00:20.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.866+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506815d8a0 con 0x7f50680b2e30 2024-01-15T18:00:20.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.866+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058007260 con 0x7f50680b2e30 2024-01-15T18:00:20.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.866+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f506815de60 con 0x7f50680b2e30 2024-01-15T18:00:20.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.867+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50680686f0 con 0x7f50680b2e30 2024-01-15T18:00:20.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.868+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f50580073c0 con 0x7f50680b2e30 2024-01-15T18:00:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.870+0000 7f505f400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504006b470 0x7f504006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.870+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50580892d0 con 0x7f50680b2e30 2024-01-15T18:00:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.870+0000 7f505fe00640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504006b470 0x7f504006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:20.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.872+0000 7f505fe00640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504006b470 0x7f504006d930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5068153750 tx=0x7f5050008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:20.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:20.874+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5058053ed0 con 0x7f50680b2e30 2024-01-15T18:00:21.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.101+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f506815f620 con 0x7f504006b470 2024-01-15T18:00:21.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.156+0000 7f505f400640 1 -- 172.21.15.2:0/3037888270 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5050004020 con 0x7f504006b470 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504006b470 msgr2=0x7f504006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504006b470 0x7f504006d930 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5068153750 tx=0x7f5050008040 comp rx=0 tx=0).stop 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 msgr2=0x7f50681460c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 0x7f50681460c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f505800c400 tx=0x7f505800c8c0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 shutdown_connections 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504006b470 0x7f504006d930 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.159+0000 7f506e400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f506815c130 0x7f50680cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.160+0000 7f506e400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50680cf000 0x7f5068146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.160+0000 7f506e400640 1 --2- 172.21.15.2:0/3037888270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50680b2e30 0x7f50681460c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.160+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 >> 172.21.15.2:0/3037888270 conn(0x7f50680cc170 msgr2=0x7f506815a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:21.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.160+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 shutdown_connections 2024-01-15T18:00:21.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.160+0000 7f506e400640 1 -- 172.21.15.2:0/3037888270 wait complete. 2024-01-15T18:00:21.167 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.163+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.167 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.163+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.167 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.163+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.167 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.163+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.167 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.163+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.186 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T18:00:21.198 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.194+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.198 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.194+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.198 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.194+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.194+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:21.194+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:21.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.390+0000 7f2051600640 1 Processor -- start 2024-01-15T18:00:21.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.390+0000 7f2051600640 1 -- start start 2024-01-15T18:00:21.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f2051600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f204c146ba0 0x7f204c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f2051600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f204c0b2ba0 0x7f204c0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f2051600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c15d9a0 0x7f204c15de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f2051600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204c146f80 con 0x7f204c0b2ba0 2024-01-15T18:00:21.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f2051600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204c06f440 con 0x7f204c146ba0 2024-01-15T18:00:21.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f2051600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0b3520 con 0x7f204c15d9a0 2024-01-15T18:00:21.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f204a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f204c0b2ba0 0x7f204c0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f204a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f204c0b2ba0 0x7f204c0b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58594/0 (socket says 172.21.15.2:58594) 2024-01-15T18:00:21.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.392+0000 7f204a000640 1 -- 172.21.15.2:0/3427781272 learned_addr learned my addr 172.21.15.2:0/3427781272 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204b400640 1 --2- 172.21.15.2:0/3427781272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c15d9a0 0x7f204c15de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204aa00640 1 --2- 172.21.15.2:0/3427781272 >> v2:172.21.15.137:3300/0 conn(0x7f204c146ba0 0x7f204c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204a000640 1 -- 172.21.15.2:0/3427781272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c15d9a0 msgr2=0x7f204c15de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204a000640 1 --2- 172.21.15.2:0/3427781272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c15d9a0 0x7f204c15de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204a000640 1 -- 172.21.15.2:0/3427781272 >> v2:172.21.15.137:3300/0 conn(0x7f204c146ba0 msgr2=0x7f204c0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204a000640 1 --2- 172.21.15.2:0/3427781272 >> v2:172.21.15.137:3300/0 conn(0x7f204c146ba0 0x7f204c0d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204a000640 1 -- 172.21.15.2:0/3427781272 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c15e370 con 0x7f204c0b2ba0 2024-01-15T18:00:21.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.393+0000 7f204aa00640 1 --2- 172.21.15.2:0/3427781272 >> v2:172.21.15.137:3300/0 conn(0x7f204c146ba0 0x7f204c0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:21.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.394+0000 7f204a000640 1 --2- 172.21.15.2:0/3427781272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c0b2ba0 0x7f204c0b2fe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f204c06e430 tx=0x7f203402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d6e5bfc5b0150b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:21.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2049600640 1 -- 172.21.15.2:0/3427781272 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20340317c0 con 0x7f204c0b2ba0 2024-01-15T18:00:21.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2049600640 1 -- 172.21.15.2:0/3427781272 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2034031920 con 0x7f204c0b2ba0 2024-01-15T18:00:21.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2049600640 1 -- 172.21.15.2:0/3427781272 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034031cb0 con 0x7f204c0b2ba0 2024-01-15T18:00:21.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2051600640 1 -- 172.21.15.2:0/3427781272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c0b2ba0 msgr2=0x7f204c0b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2051600640 1 --2- 172.21.15.2:0/3427781272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c0b2ba0 0x7f204c0b2fe0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f204c06e430 tx=0x7f203402b860 comp rx=0 tx=0).stop 2024-01-15T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2051600640 1 -- 172.21.15.2:0/3427781272 shutdown_connections 2024-01-15T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2051600640 1 --2- 172.21.15.2:0/3427781272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c15d9a0 0x7f204c15de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.395+0000 7f2051600640 1 --2- 172.21.15.2:0/3427781272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c0b2ba0 0x7f204c0b2fe0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.396+0000 7f2051600640 1 --2- 172.21.15.2:0/3427781272 >> v2:172.21.15.137:3300/0 conn(0x7f204c146ba0 0x7f204c0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.396+0000 7f2051600640 1 -- 172.21.15.2:0/3427781272 >> 172.21.15.2:0/3427781272 conn(0x7f204c0b6430 msgr2=0x7f204c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:21.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.396+0000 7f2051600640 1 -- 172.21.15.2:0/3427781272 shutdown_connections 2024-01-15T18:00:21.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.396+0000 7f2051600640 1 -- 172.21.15.2:0/3427781272 wait complete. 2024-01-15T18:00:21.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.398+0000 7f2051600640 1 Processor -- start 2024-01-15T18:00:21.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.398+0000 7f2051600640 1 -- start start 2024-01-15T18:00:21.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.399+0000 7f2051600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c0b2ba0 0x7f204c1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.399+0000 7f2051600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 0x7f204c0caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.399+0000 7f2051600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f204c15d9a0 0x7f204c0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.399+0000 7f2051600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c0ca830 con 0x7f204c146ba0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.399+0000 7f2051600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1f48d0 con 0x7f204c15d9a0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.399+0000 7f2051600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204c1f71f0 con 0x7f204c0b2ba0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 0x7f204c0caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 0x7f204c0caff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58606/0 (socket says 172.21.15.2:58606) 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 -- 172.21.15.2:0/4101755824 learned_addr learned my addr 172.21.15.2:0/4101755824 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204aa00640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c0b2ba0 0x7f204c1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204b400640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f204c15d9a0 0x7f204c0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 -- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c0b2ba0 msgr2=0x7f204c1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c0b2ba0 0x7f204c1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 -- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f204c15d9a0 msgr2=0x7f204c0cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f204c15d9a0 0x7f204c0cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204a000640 1 -- 172.21.15.2:0/4101755824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20340094d0 con 0x7f204c146ba0 2024-01-15T18:00:21.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.400+0000 7f204b400640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f204c15d9a0 0x7f204c0cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:21.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.401+0000 7f204a000640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 0x7f204c0caff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f20340097c0 tx=0x7f20340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:21.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.401+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034032860 con 0x7f204c146ba0 2024-01-15T18:00:21.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.401+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20340329e0 con 0x7f204c146ba0 2024-01-15T18:00:21.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.401+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2034032d90 con 0x7f204c146ba0 2024-01-15T18:00:21.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.401+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c1f3fa0 con 0x7f204c146ba0 2024-01-15T18:00:21.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.401+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c1f4560 con 0x7f204c146ba0 2024-01-15T18:00:21.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.402+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c1f4a40 con 0x7f204c146ba0 2024-01-15T18:00:21.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.403+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f203404d3f0 con 0x7f204c146ba0 2024-01-15T18:00:21.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.404+0000 7f2042a00640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f202406b470 0x7f202406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.405+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20340abca0 con 0x7f204c146ba0 2024-01-15T18:00:21.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.405+0000 7f204aa00640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f202406b470 0x7f202406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.407+0000 7f204aa00640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f202406b470 0x7f202406d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f204c0caa00 tx=0x7f203c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:21.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.409+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2034076880 con 0x7f204c146ba0 2024-01-15T18:00:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.514+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 8) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f2034076c50 con 0x7f204c146ba0 2024-01-15T18:00:21.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.650+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f204c0d0f80 con 0x7f202406b470 2024-01-15T18:00:21.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.686+0000 7f2042a00640 1 -- 172.21.15.2:0/4101755824 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f203c00a1f0 con 0x7f202406b470 2024-01-15T18:00:21.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f202406b470 msgr2=0x7f202406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f202406b470 0x7f202406d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f204c0caa00 tx=0x7f203c008040 comp rx=0 tx=0).stop 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 msgr2=0x7f204c0caff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 0x7f204c0caff0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f20340097c0 tx=0x7f20340046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 shutdown_connections 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f202406b470 0x7f202406d930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f204c15d9a0 0x7f204c0cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f204c146ba0 0x7f204c0caff0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 --2- 172.21.15.2:0/4101755824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f204c0b2ba0 0x7f204c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.689+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 >> 172.21.15.2:0/4101755824 conn(0x7f204c0b6430 msgr2=0x7f204c0b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.690+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 shutdown_connections 2024-01-15T18:00:21.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.690+0000 7f2051600640 1 -- 172.21.15.2:0/4101755824 wait complete. 2024-01-15T18:00:21.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:21.715 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:00:21.766 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-01-15T18:00:21.767 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:00:21.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.965+0000 7f02ec800640 1 Processor -- start 2024-01-15T18:00:21.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.965+0000 7f02ec800640 1 -- start start 2024-01-15T18:00:21.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.967+0000 7f02ec800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 0x7f02e40b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.967+0000 7f02ec800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e415dd70 con 0x7f02e41439f0 2024-01-15T18:00:21.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.967+0000 7f02ec800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02e40711b0 con 0x7f02e40702c0 2024-01-15T18:00:21.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.967+0000 7f02ec800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4143dd0 con 0x7f02e406d000 2024-01-15T18:00:21.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.968+0000 7f02e9e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f02e41439f0 0x7f02e40cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50642/0 (socket says 172.21.15.2:50642) 2024-01-15T18:00:21.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.968+0000 7f02e9e00640 1 -- 172.21.15.2:0/1198198261 learned_addr learned my addr 172.21.15.2:0/1198198261 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:21.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.968+0000 7f02ea800640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 0x7f02e40b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.968+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263623591 0 0) 0x7f02e415dd70 con 0x7f02e41439f0 2024-01-15T18:00:21.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.968+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c4003520 con 0x7f02e41439f0 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1607804776 0 0) 0x7f02c4003520 con 0x7f02e41439f0 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 msgr2=0x7f02e40b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 0x7f02e40b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 >> v1:172.21.15.137:6789/0 conn(0x7f02e40702c0 legacy=0x7f02e406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4070910 con 0x7f02e41439f0 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f02d40019f0 con 0x7f02e41439f0 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 >> v1:172.21.15.2:6789/0 conn(0x7f02e41439f0 legacy=0x7f02e40cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 0x7f02c40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c4003f70 0x7f02c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4143dd0 con 0x7f02c4004d00 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c400bac0 con 0x7f02c4003f70 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c400de90 con 0x7f02e406d000 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02ea800640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.969+0000 7f02e9e00640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 0x7f02c40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.970+0000 7f02e9400640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c4003f70 0x7f02c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.970+0000 7f02ea800640 1 -- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 msgr2=0x7f02c40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.970+0000 7f02ea800640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e406d000 0x7f02c40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.970+0000 7f02ea800640 1 -- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c4003f70 msgr2=0x7f02c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.970+0000 7f02ea800640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c4003f70 0x7f02c40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.970+0000 7f02ea800640 1 -- 172.21.15.2:0/1198198261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c4003520 con 0x7f02c4004d00 2024-01-15T18:00:21.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.971+0000 7f02ea800640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02c4005180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009a90 tx=0x7f02c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf0f2a229d1b700 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:21.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.971+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc02c680 con 0x7f02c4004d00 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc02c7e0 con 0x7f02c4004d00 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02e8a00640 1 -- 172.21.15.2:0/1198198261 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc02cb70 con 0x7f02c4004d00 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ec800640 1 -- 172.21.15.2:0/1198198261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 msgr2=0x7f02c4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ec800640 1 --2- 172.21.15.2:0/1198198261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02c4005180 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009a90 tx=0x7f02c40057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ea800640 1 -- 172.21.15.2:0/1198198261 reap_dead start 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ec800640 1 -- 172.21.15.2:0/1198198261 shutdown_connections 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ec800640 1 -- 172.21.15.2:0/1198198261 >> 172.21.15.2:0/1198198261 conn(0x7f02e40b45f0 msgr2=0x7f02e40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ec800640 1 -- 172.21.15.2:0/1198198261 shutdown_connections 2024-01-15T18:00:21.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.972+0000 7f02ec800640 1 -- 172.21.15.2:0/1198198261 wait complete. 2024-01-15T18:00:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.974+0000 7f02ec800640 1 Processor -- start 2024-01-15T18:00:21.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.975+0000 7f02ec800640 1 -- start start 2024-01-15T18:00:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02ec800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02e41592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02ec800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02e40b1830 0x7f02e4139bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02ec800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e413a100 0x7f02e41f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02ec800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e413a580 con 0x7f02c4004d00 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02ec800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e4138330 con 0x7f02e40b1830 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02ec800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e41eea60 con 0x7f02e413a100 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02e9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02e41592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02e9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02e41592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58634/0 (socket says 172.21.15.2:58634) 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.976+0000 7f02e9e00640 1 -- 172.21.15.2:0/3795025078 learned_addr learned my addr 172.21.15.2:0/3795025078 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02ea800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e413a100 0x7f02e41f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9400640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02e40b1830 0x7f02e4139bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9e00640 1 -- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e413a100 msgr2=0x7f02e41f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9e00640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e413a100 0x7f02e41f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9e00640 1 -- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02e40b1830 msgr2=0x7f02e4139bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9e00640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02e40b1830 0x7f02e4139bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9e00640 1 -- 172.21.15.2:0/3795025078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4159810 con 0x7f02c4004d00 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02ea800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e413a100 0x7f02e41f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e9e00640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02e41592d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f02d40079d0 tx=0x7f02d400a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d40133f0 con 0x7f02c4004d00 2024-01-15T18:00:21.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02d4013550 con 0x7f02c4004d00 2024-01-15T18:00:21.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02d4013900 con 0x7f02c4004d00 2024-01-15T18:00:21.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.977+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e4138580 con 0x7f02c4004d00 2024-01-15T18:00:21.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.978+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e4138b40 con 0x7f02c4004d00 2024-01-15T18:00:21.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.979+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e40686f0 con 0x7f02c4004d00 2024-01-15T18:00:21.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.980+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 8) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f02d4013a60 con 0x7f02c4004d00 2024-01-15T18:00:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.981+0000 7f02e2000640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02bc06b5e0 0x7f02bc06daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.981+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f02d408ae20 con 0x7f02c4004d00 2024-01-15T18:00:21.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.981+0000 7f02e9400640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02bc06b5e0 0x7f02bc06daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:21.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.982+0000 7f02e9400640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02bc06b5e0 0x7f02bc06daa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f02cc007fd0 tx=0x7f02cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:21.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:21.985+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02d40558f0 con 0x7f02c4004d00 2024-01-15T18:00:22.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:22.223+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f02e4070ed0 con 0x7f02bc06b5e0 2024-01-15T18:00:22.276 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T18:00:22.273+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:00:22.372 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:00:24.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:24.314+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f02d4090020 con 0x7f02c4004d00 2024-01-15T18:00:25.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.324+0000 7f02e2000640 1 -- 172.21.15.2:0/3795025078 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f02cc00d050 con 0x7f02bc06b5e0 2024-01-15T18:00:25.330 INFO:teuthology.orchestra.run.smithi002.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:00:25.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.328+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02bc06b5e0 msgr2=0x7f02bc06daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:25.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.328+0000 7f02ec800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02bc06b5e0 0x7f02bc06daa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f02cc007fd0 tx=0x7f02cc009040 comp rx=0 tx=0).stop 2024-01-15T18:00:25.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.328+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 msgr2=0x7f02e41592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:25.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.328+0000 7f02ec800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02e41592d0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f02d40079d0 tx=0x7f02d400a640 comp rx=0 tx=0).stop 2024-01-15T18:00:25.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 shutdown_connections 2024-01-15T18:00:25.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02bc06b5e0 0x7f02bc06daa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02e413a100 0x7f02e41f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02e40b1830 0x7f02e4139bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 --2- 172.21.15.2:0/3795025078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c4004d00 0x7f02e41592d0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 >> 172.21.15.2:0/3795025078 conn(0x7f02e40b45f0 msgr2=0x7f02e415be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.329+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 shutdown_connections 2024-01-15T18:00:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.330+0000 7f02ec800640 1 -- 172.21.15.2:0/3795025078 wait complete. 2024-01-15T18:00:25.363 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:00:25.363 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:00:25.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:00:25.598 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:00:25.599 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:00:25.599 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:25.656 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:25.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.881+0000 7fd15b200640 1 Processor -- start 2024-01-15T18:00:25.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.881+0000 7fd15b200640 1 -- start start 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd15b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd15c06b830 0x7fd15c0cb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd15b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c0ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd15b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15c0caae0 con 0x7fd15c06eb00 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd15b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15c06dd40 con 0x7fd15c06b830 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd15b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c06e020 con 0x7fd15c06bc10 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd159e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd15c06b830 0x7fd15c0cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd159e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd15c06b830 0x7fd15c0cb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35494/0 (socket says 172.21.15.2:35494) 2024-01-15T18:00:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.883+0000 7fd159e00640 1 -- 172.21.15.2:0/1330260579 learned_addr learned my addr 172.21.15.2:0/1330260579 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:25.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.884+0000 7fd159e00640 1 -- 172.21.15.2:0/1330260579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 msgr2=0x7fd15c0ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:25.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.884+0000 7fd159e00640 1 --2- 172.21.15.2:0/1330260579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c0ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.884+0000 7fd159e00640 1 -- 172.21.15.2:0/1330260579 >> v1:172.21.15.137:6789/0 conn(0x7fd15c06eb00 legacy=0x7fd15c0cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:25.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.884+0000 7fd159e00640 1 -- 172.21.15.2:0/1330260579 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c06e300 con 0x7fd15c06b830 2024-01-15T18:00:25.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.885+0000 7fd159e00640 1 --2- 172.21.15.2:0/1330260579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c0cb770 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009870 tx=0x7fd14c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf7542fd995a583 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:25.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.885+0000 7fd159400640 1 -- 172.21.15.2:0/1330260579 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0317c0 con 0x7fd15c06b830 2024-01-15T18:00:25.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.885+0000 7fd159400640 1 -- 172.21.15.2:0/1330260579 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c031920 con 0x7fd15c06b830 2024-01-15T18:00:25.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.885+0000 7fd159400640 1 -- 172.21.15.2:0/1330260579 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c031cb0 con 0x7fd15c06b830 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 -- 172.21.15.2:0/1330260579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 msgr2=0x7fd15c0cb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 --2- 172.21.15.2:0/1330260579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c0cb770 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009870 tx=0x7fd14c02b860 comp rx=0 tx=0).stop 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 -- 172.21.15.2:0/1330260579 shutdown_connections 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 --2- 172.21.15.2:0/1330260579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c0ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 --2- 172.21.15.2:0/1330260579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c0cb770 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 -- 172.21.15.2:0/1330260579 >> 172.21.15.2:0/1330260579 conn(0x7fd15c0cbfc0 msgr2=0x7fd15c0cc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 -- 172.21.15.2:0/1330260579 shutdown_connections 2024-01-15T18:00:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.886+0000 7fd15b200640 1 -- 172.21.15.2:0/1330260579 wait complete. 2024-01-15T18:00:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.888+0000 7fd15b200640 1 Processor -- start 2024-01-15T18:00:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.888+0000 7fd15b200640 1 -- start start 2024-01-15T18:00:25.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd15b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c154260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd15b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c1f1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd15b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c06eb00 0x7fd15c153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd15b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1f1a60 con 0x7fd15c06b830 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd15b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c1548f0 con 0x7fd15c06eb00 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd15b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15c154bd0 con 0x7fd15c06bc10 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.889+0000 7fd159e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c154260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c154260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35504/0 (socket says 172.21.15.2:35504) 2024-01-15T18:00:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 -- 172.21.15.2:0/234954461 learned_addr learned my addr 172.21.15.2:0/234954461 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 -- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 msgr2=0x7fd15c1f1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd15a800640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c06eb00 0x7fd15c153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd153e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c1f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c1f1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 -- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c06eb00 msgr2=0x7fd15c153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c06eb00 0x7fd15c153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 -- 172.21.15.2:0/234954461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c0094d0 con 0x7fd15c06b830 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd153e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c1f1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd15a800640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c06eb00 0x7fd15c153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:25.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.890+0000 7fd159e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c154260 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009870 tx=0x7fd14c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:25.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.891+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c032860 con 0x7fd15c06b830 2024-01-15T18:00:25.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.891+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14c0329c0 con 0x7fd15c06b830 2024-01-15T18:00:25.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.891+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14c032d50 con 0x7fd15c06b830 2024-01-15T18:00:25.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.891+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd15c154eb0 con 0x7fd15c06b830 2024-01-15T18:00:25.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.891+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd15c155190 con 0x7fd15c06b830 2024-01-15T18:00:25.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.892+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15c0c89f0 con 0x7fd15c06b830 2024-01-15T18:00:25.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.893+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd14c04d3f0 con 0x7fd15c06b830 2024-01-15T18:00:25.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.894+0000 7fd152000640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd13406b500 0x7fd13406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:25.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.894+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd14c0abce0 con 0x7fd15c06b830 2024-01-15T18:00:25.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.895+0000 7fd153e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd13406b500 0x7fd13406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:25.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.896+0000 7fd153e00640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd13406b500 0x7fd13406d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd144006fd0 tx=0x7fd144008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:25.899+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd15c0c89f0 con 0x7fd15c06b830 2024-01-15T18:00:26.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.135+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd15c154d10 con 0x7fd15c06b830 2024-01-15T18:00:26.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.136+0000 7fd152000640 1 -- 172.21.15.2:0/234954461 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+12967 (secure 0 0 0) 0x7fd14c0a3e60 con 0x7fd15c06b830 2024-01-15T18:00:26.140 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:26.140 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":39,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:25.305089+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6834/2859107199":"2024-01-16T18:00:22.271924+0000","172.21.15.2:6832/2859107199":"2024-01-16T18:00:22.271924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:26.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.140+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd13406b500 msgr2=0x7fd13406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.140+0000 7fd15b200640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd13406b500 0x7fd13406d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fd144006fd0 tx=0x7fd144008040 comp rx=0 tx=0).stop 2024-01-15T18:00:26.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.140+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 msgr2=0x7fd15c154260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.140+0000 7fd15b200640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c154260 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009870 tx=0x7fd14c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.142+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 shutdown_connections 2024-01-15T18:00:26.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.142+0000 7fd15b200640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd13406b500 0x7fd13406d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.142+0000 7fd15b200640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd15c06eb00 0x7fd15c153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.142+0000 7fd15b200640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd15c06bc10 0x7fd15c1f1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.142+0000 7fd15b200640 1 --2- 172.21.15.2:0/234954461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd15c06b830 0x7fd15c154260 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.142+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 >> 172.21.15.2:0/234954461 conn(0x7fd15c0cbfc0 msgr2=0x7fd15c15ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:26.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.144+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 shutdown_connections 2024-01-15T18:00:26.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.144+0000 7fd15b200640 1 -- 172.21.15.2:0/234954461 wait complete. 2024-01-15T18:00:26.178 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:00:26.179 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:00:26.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.457+0000 7fb023c00640 1 Processor -- start 2024-01-15T18:00:26.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.457+0000 7fb023c00640 1 -- start start 2024-01-15T18:00:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb023c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c0b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb023c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb01c06a4b0 0x7fb01c06a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb023c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c06f060 con 0x7fb01c0af6a0 2024-01-15T18:00:26.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb023c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb01c06ae80 con 0x7fb01c06a4b0 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb023c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c15d8b0 con 0x7fb01c15d4d0 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb01be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb01c0af6a0 0x7fb01c06f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56278/0 (socket says 172.21.15.2:56278) 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb01be00640 1 -- 172.21.15.2:0/2076893737 learned_addr learned my addr 172.21.15.2:0/2076893737 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb021c00640 1 --2- 172.21.15.2:0/2076893737 >> v2:172.21.15.2:3300/0 conn(0x7fb01c06a4b0 0x7fb01c06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb021200640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c0b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.459+0000 7fb01b400640 1 -- 172.21.15.2:0/2076893737 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128676635 0 0) 0x7fb01c06f060 con 0x7fb01c0af6a0 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.460+0000 7fb01b400640 1 -- 172.21.15.2:0/2076893737 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc003520 con 0x7fb01c0af6a0 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.460+0000 7fb021c00640 1 -- 172.21.15.2:0/2076893737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 msgr2=0x7fb01c0b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:26.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.460+0000 7fb021c00640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c0b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.460+0000 7fb021c00640 1 -- 172.21.15.2:0/2076893737 >> v1:172.21.15.2:6789/0 conn(0x7fb01c0af6a0 legacy=0x7fb01c06f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.460+0000 7fb021c00640 1 -- 172.21.15.2:0/2076893737 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0aef50 con 0x7fb01c06a4b0 2024-01-15T18:00:26.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.460+0000 7fb021200640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c0b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.461+0000 7fb021c00640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c06a940 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb014009dc0 tx=0x7fb01402bd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d4d2fac37e648 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.461+0000 7fb01b400640 1 -- 172.21.15.2:0/2076893737 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01402cb20 con 0x7fb01c06a4b0 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.461+0000 7fb01b400640 1 -- 172.21.15.2:0/2076893737 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01402cc80 con 0x7fb01c06a4b0 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.461+0000 7fb01b400640 1 -- 172.21.15.2:0/2076893737 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014034440 con 0x7fb01c06a4b0 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.461+0000 7fb023c00640 1 -- 172.21.15.2:0/2076893737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 msgr2=0x7fb01c06a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.461+0000 7fb023c00640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c06a940 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb014009dc0 tx=0x7fb01402bd70 comp rx=0 tx=0).stop 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.462+0000 7fb023c00640 1 -- 172.21.15.2:0/2076893737 shutdown_connections 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.462+0000 7fb023c00640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c06a940 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.462+0000 7fb023c00640 1 --2- 172.21.15.2:0/2076893737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c0b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.462+0000 7fb023c00640 1 -- 172.21.15.2:0/2076893737 >> 172.21.15.2:0/2076893737 conn(0x7fb01c0b1d70 msgr2=0x7fb01c0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.462+0000 7fb023c00640 1 -- 172.21.15.2:0/2076893737 shutdown_connections 2024-01-15T18:00:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.462+0000 7fb023c00640 1 -- 172.21.15.2:0/2076893737 wait complete. 2024-01-15T18:00:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.463+0000 7fb023c00640 1 Processor -- start 2024-01-15T18:00:26.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.463+0000 7fb023c00640 1 -- start start 2024-01-15T18:00:26.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb023c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c14d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb023c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb01c0af6a0 0x7fb01c15f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb023c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c15f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb023c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c15dd80 con 0x7fb01c06a4b0 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb023c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c15fcd0 con 0x7fb01c0af6a0 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb023c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c15ffb0 con 0x7fb01c15d4d0 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.464+0000 7fb021200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c14d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c14d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35522/0 (socket says 172.21.15.2:35522) 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 -- 172.21.15.2:0/1620659486 learned_addr learned my addr 172.21.15.2:0/1620659486 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 -- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 msgr2=0x7fb01c15f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c15f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c15f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 -- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb01c0af6a0 msgr2=0x7fb01c15f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb01be00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb01c0af6a0 0x7fb01c15f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb01c0af6a0 0x7fb01c15f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 -- 172.21.15.2:0/1620659486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb014009a70 con 0x7fb01c06a4b0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb021200640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c14d940 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00c7f0 tx=0x7fb00c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.465+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c007820 con 0x7fb01c06a4b0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.466+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c007980 con 0x7fb01c06a4b0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.466+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c007cf0 con 0x7fb01c06a4b0 2024-01-15T18:00:26.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.466+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c160290 con 0x7fb01c06a4b0 2024-01-15T18:00:26.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.466+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c160850 con 0x7fb01c06a4b0 2024-01-15T18:00:26.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.467+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c067f30 con 0x7fb01c06a4b0 2024-01-15T18:00:26.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.468+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb00c0113f0 con 0x7fb01c06a4b0 2024-01-15T18:00:26.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.469+0000 7fb018c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faff406b500 0x7faff406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.469+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb00c08a140 con 0x7fb01c06a4b0 2024-01-15T18:00:26.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.470+0000 7fb01be00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faff406b500 0x7faff406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.471+0000 7fb01be00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faff406b500 0x7faff406d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb004005880 tx=0x7fb004009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:26.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.473+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb00c054e10 con 0x7fb01c06a4b0 2024-01-15T18:00:26.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.656+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb01c0686f0 con 0x7fb01c06a4b0 2024-01-15T18:00:26.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.657+0000 7fb018c00640 1 -- 172.21.15.2:0/1620659486 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb00c058430 con 0x7fb01c06a4b0 2024-01-15T18:00:26.660 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faff406b500 msgr2=0x7faff406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faff406b500 0x7faff406d9c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb004005880 tx=0x7fb004009040 comp rx=0 tx=0).stop 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 msgr2=0x7fb01c14d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c14d940 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb00c00c7f0 tx=0x7fb00c00ccb0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 shutdown_connections 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faff406b500 0x7faff406d9c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb01c15d4d0 0x7fb01c15f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb01c0af6a0 0x7fb01c15f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 --2- 172.21.15.2:0/1620659486 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb01c06a4b0 0x7fb01c14d940 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.659+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 >> 172.21.15.2:0/1620659486 conn(0x7fb01c0b1d70 msgr2=0x7fb01c0b0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:26.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.660+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 shutdown_connections 2024-01-15T18:00:26.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.660+0000 7fb023c00640 1 -- 172.21.15.2:0/1620659486 wait complete. 2024-01-15T18:00:26.680 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:00:26.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.880+0000 7f3fe7200640 1 Processor -- start 2024-01-15T18:00:26.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.880+0000 7f3fe7200640 1 -- start start 2024-01-15T18:00:26.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fe7200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3fe80ccb80 0x7f3fe806c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fe7200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe8071340 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fe7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe806c670 con 0x7f3fe80cb620 2024-01-15T18:00:26.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fe7200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe815bd50 con 0x7f3fe80ccb80 2024-01-15T18:00:26.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fe5e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3fe80ce1d0 0x7f3fe815b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56284/0 (socket says 172.21.15.2:56284) 2024-01-15T18:00:26.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fe5e00640 1 -- 172.21.15.2:0/1940323836 learned_addr learned my addr 172.21.15.2:0/1940323836 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:26.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.882+0000 7f3fdfe00640 1 --2- 172.21.15.2:0/1940323836 >> v2:172.21.15.2:3300/0 conn(0x7f3fe80ccb80 0x7f3fe806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.883+0000 7f3fdfe00640 1 -- 172.21.15.2:0/1940323836 >> v1:172.21.15.137:6789/0 conn(0x7f3fe80cb620 legacy=0x7f3fe80cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.883+0000 7f3fdfe00640 1 -- 172.21.15.2:0/1940323836 >> v1:172.21.15.2:6789/0 conn(0x7f3fe80ce1d0 legacy=0x7f3fe815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.883+0000 7f3fdfe00640 1 -- 172.21.15.2:0/1940323836 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe815bfb0 con 0x7f3fe80ccb80 2024-01-15T18:00:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.884+0000 7f3fdfe00640 1 --2- 172.21.15.2:0/1940323836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ccb80 0x7f3fe806c130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097a0 tx=0x7f3fd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc2058b56ec259b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.884+0000 7f3fe5400640 1 -- 172.21.15.2:0/1940323836 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0031830 con 0x7f3fe80ccb80 2024-01-15T18:00:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.884+0000 7f3fe5400640 1 -- 172.21.15.2:0/1940323836 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd0031990 con 0x7f3fe80ccb80 2024-01-15T18:00:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.884+0000 7f3fe5400640 1 -- 172.21.15.2:0/1940323836 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0031d20 con 0x7f3fe80ccb80 2024-01-15T18:00:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 -- 172.21.15.2:0/1940323836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ccb80 msgr2=0x7f3fe806c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 --2- 172.21.15.2:0/1940323836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ccb80 0x7f3fe806c130 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097a0 tx=0x7f3fd002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 -- 172.21.15.2:0/1940323836 shutdown_connections 2024-01-15T18:00:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 --2- 172.21.15.2:0/1940323836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ccb80 0x7f3fe806c130 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 -- 172.21.15.2:0/1940323836 >> 172.21.15.2:0/1940323836 conn(0x7f3fe80b0840 msgr2=0x7f3fe80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 -- 172.21.15.2:0/1940323836 shutdown_connections 2024-01-15T18:00:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.885+0000 7f3fe7200640 1 -- 172.21.15.2:0/1940323836 wait complete. 2024-01-15T18:00:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.886+0000 7f3fe7200640 1 Processor -- start 2024-01-15T18:00:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.887+0000 7f3fe7200640 1 -- start start 2024-01-15T18:00:26.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.887+0000 7f3fe7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fe80cb620 0x7f3fe8154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.887+0000 7f3fe7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fe80ccb80 0x7f3fe81e4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 0x7f3fe8152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe8152e90 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe8153cb0 con 0x7f3fe80cb620 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe81530a0 con 0x7f3fe80ccb80 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 0x7f3fe8152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 0x7f3fe8152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35526/0 (socket says 172.21.15.2:35526) 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe6800640 1 -- 172.21.15.2:0/3863740414 learned_addr learned my addr 172.21.15.2:0/3863740414 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fdfe00640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fe80ccb80 0x7f3fe81e4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.888+0000 7f3fe5e00640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fe80cb620 0x7f3fe8154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe6800640 1 -- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fe80ccb80 msgr2=0x7f3fe81e4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe6800640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fe80ccb80 0x7f3fe81e4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe6800640 1 -- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fe80cb620 msgr2=0x7f3fe8154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe6800640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fe80cb620 0x7f3fe8154140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe6800640 1 -- 172.21.15.2:0/3863740414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0009400 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe6800640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 0x7f3fe8152950 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c7b0 tx=0x7f3fe000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe0004b60 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe0004cc0 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe000aa90 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe8153380 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.889+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe8153940 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.890+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe806bfa0 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.892+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3fe000abf0 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.893+0000 7f3fde000640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fc006b500 0x7f3fc006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:26.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.893+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f3fe008a190 con 0x7f3fe80ce1d0 2024-01-15T18:00:26.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.894+0000 7f3fe5e00640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fc006b500 0x7f3fc006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:26.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.895+0000 7f3fe5e00640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fc006b500 0x7f3fc006d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8005880 tx=0x7f3fd8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:26.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:26.898+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3fe0054ee0 con 0x7f3fe80ce1d0 2024-01-15T18:00:27.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.099+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3fe8074770 con 0x7f3fe80ce1d0 2024-01-15T18:00:27.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.100+0000 7f3fde000640 1 -- 172.21.15.2:0/3863740414 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v39) v1 ==== 80+0+120 (secure 0 0 0) 0x7f3fe0058500 con 0x7f3fe80ce1d0 2024-01-15T18:00:27.103 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/2859107199 2024-01-16T18:00:22.271924+0000 2024-01-15T18:00:27.103 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6832/2859107199 2024-01-16T18:00:22.271924+0000 2024-01-15T18:00:27.104 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:00:27.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fc006b500 msgr2=0x7f3fc006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fc006b500 0x7f3fc006d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8005880 tx=0x7f3fd8009040 comp rx=0 tx=0).stop 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 msgr2=0x7f3fe8152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 0x7f3fe8152950 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3fe000c7b0 tx=0x7f3fe000cc70 comp rx=0 tx=0).stop 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 shutdown_connections 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fc006b500 0x7f3fc006d9c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3fe80ce1d0 0x7f3fe8152950 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3fe80ccb80 0x7f3fe81e4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 --2- 172.21.15.2:0/3863740414 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3fe80cb620 0x7f3fe8154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 >> 172.21.15.2:0/3863740414 conn(0x7f3fe80b0840 msgr2=0x7f3fe8071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 shutdown_connections 2024-01-15T18:00:27.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.103+0000 7f3fe7200640 1 -- 172.21.15.2:0/3863740414 wait complete. 2024-01-15T18:00:27.122 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:00:27.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:27.122 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:27.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[49225]: fuse finished with error 0 and tester_r 0 2024-01-15T18:00:27.201 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:00:27.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:27.202 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:00:27.418 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:00:27.418 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:27.476 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:27.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.694+0000 7f9ac3200640 1 Processor -- start 2024-01-15T18:00:27.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.695+0000 7f9ac3200640 1 -- start start 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.696+0000 7f9ac3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9ac4141880 0x7f9ac4070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.696+0000 7f9ac3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac415f0e0 0x7f9ac40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.696+0000 7f9ac3200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac406dae0 con 0x7f9ac415ec60 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.696+0000 7f9ac3200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac4071f20 con 0x7f9ac4141880 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.696+0000 7f9ac3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac406c6b0 con 0x7f9ac415f0e0 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9ac4141880 0x7f9ac4070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9ac4141880 0x7f9ac4070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35528/0 (socket says 172.21.15.2:35528) 2024-01-15T18:00:27.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9abbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9ac415ec60 0x7f9ac40cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56288/0 (socket says 172.21.15.2:56288) 2024-01-15T18:00:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 -- 172.21.15.2:0/1355933428 learned_addr learned my addr 172.21.15.2:0/1355933428 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 -- 172.21.15.2:0/1355933428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac415f0e0 msgr2=0x7f9ac40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 --2- 172.21.15.2:0/1355933428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac415f0e0 0x7f9ac40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 -- 172.21.15.2:0/1355933428 >> v1:172.21.15.2:6789/0 conn(0x7f9ac415ec60 legacy=0x7f9ac40cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.697+0000 7f9ac1e00640 1 -- 172.21.15.2:0/1355933428 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac415f570 con 0x7f9ac4141880 2024-01-15T18:00:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.698+0000 7f9ac1e00640 1 --2- 172.21.15.2:0/1355933428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac4141880 0x7f9ac4070fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40098a0 tx=0x7f9ab402b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523568f89828dc41 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac1400640 1 -- 172.21.15.2:0/1355933428 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4031640 con 0x7f9ac4141880 2024-01-15T18:00:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac1400640 1 -- 172.21.15.2:0/1355933428 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab40317a0 con 0x7f9ac4141880 2024-01-15T18:00:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac1400640 1 -- 172.21.15.2:0/1355933428 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab4031b30 con 0x7f9ac4141880 2024-01-15T18:00:27.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 -- 172.21.15.2:0/1355933428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac4141880 msgr2=0x7f9ac4070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 --2- 172.21.15.2:0/1355933428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac4141880 0x7f9ac4070fc0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40098a0 tx=0x7f9ab402b8e0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 -- 172.21.15.2:0/1355933428 shutdown_connections 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 --2- 172.21.15.2:0/1355933428 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac415f0e0 0x7f9ac40cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 --2- 172.21.15.2:0/1355933428 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac4141880 0x7f9ac4070fc0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 -- 172.21.15.2:0/1355933428 >> 172.21.15.2:0/1355933428 conn(0x7f9ac40b0840 msgr2=0x7f9ac40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 -- 172.21.15.2:0/1355933428 shutdown_connections 2024-01-15T18:00:27.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.699+0000 7f9ac3200640 1 -- 172.21.15.2:0/1355933428 wait complete. 2024-01-15T18:00:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.701+0000 7f9ac3200640 1 Processor -- start 2024-01-15T18:00:27.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.701+0000 7f9ac3200640 1 -- start start 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac4141880 0x7f9ac41f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac415ec60 0x7f9ac40ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 0x7f9ac40acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4140ac0 con 0x7f9ac415f0e0 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4159920 con 0x7f9ac415ec60 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac415a980 con 0x7f9ac4141880 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.702+0000 7f9ac2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 0x7f9ac40acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:27.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 0x7f9ac40acdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35544/0 (socket says 172.21.15.2:35544) 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 -- 172.21.15.2:0/2063579924 learned_addr learned my addr 172.21.15.2:0/2063579924 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac1e00640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac4141880 0x7f9ac41f32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 -- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac4141880 msgr2=0x7f9ac41f32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac4141880 0x7f9ac41f32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 -- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac415ec60 msgr2=0x7f9ac40ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac415ec60 0x7f9ac40ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 -- 172.21.15.2:0/2063579924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab40094d0 con 0x7f9ac415f0e0 2024-01-15T18:00:27.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.703+0000 7f9ac2800640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 0x7f9ac40acdb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00b970 tx=0x7f9abc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:27.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.704+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00cb40 con 0x7f9ac415f0e0 2024-01-15T18:00:27.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.704+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc00ccc0 con 0x7f9ac415f0e0 2024-01-15T18:00:27.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.704+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc014680 con 0x7f9ac415f0e0 2024-01-15T18:00:27.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.704+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac415ac60 con 0x7f9ac415f0e0 2024-01-15T18:00:27.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.704+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac415b220 con 0x7f9ac415f0e0 2024-01-15T18:00:27.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.705+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac415b440 con 0x7f9ac415f0e0 2024-01-15T18:00:27.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.706+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9abc01d3b0 con 0x7f9ac415f0e0 2024-01-15T18:00:27.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.707+0000 7f9aba000640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a9c06b500 0x7f9a9c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:27.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.707+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9abc089ca0 con 0x7f9ac415f0e0 2024-01-15T18:00:27.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.707+0000 7f9ac1e00640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a9c06b500 0x7f9a9c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:27.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.708+0000 7f9ac1e00640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a9c06b500 0x7f9a9c06d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4036000 tx=0x7f9ab403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:27.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.711+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9abc054a80 con 0x7f9ac415f0e0 2024-01-15T18:00:27.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.901+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ac4074770 con 0x7f9ac415f0e0 2024-01-15T18:00:27.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.903+0000 7f9aba000640 1 -- 172.21.15.2:0/2063579924 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+12967 (secure 0 0 0) 0x7f9abc019020 con 0x7f9ac415f0e0 2024-01-15T18:00:27.907 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:27.907 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":39,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:25.305089+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6834/2859107199":"2024-01-16T18:00:22.271924+0000","172.21.15.2:6832/2859107199":"2024-01-16T18:00:22.271924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:27.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a9c06b500 msgr2=0x7f9a9c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a9c06b500 0x7f9a9c06d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4036000 tx=0x7f9ab403c040 comp rx=0 tx=0).stop 2024-01-15T18:00:27.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 msgr2=0x7f9ac40acdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:27.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 0x7f9ac40acdb0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00b970 tx=0x7f9abc00be30 comp rx=0 tx=0).stop 2024-01-15T18:00:27.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 shutdown_connections 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a9c06b500 0x7f9a9c06d9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac415f0e0 0x7f9ac40acdb0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac415ec60 0x7f9ac40ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 --2- 172.21.15.2:0/2063579924 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac4141880 0x7f9ac41f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 >> 172.21.15.2:0/2063579924 conn(0x7f9ac40b0840 msgr2=0x7f9ac40ce2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 shutdown_connections 2024-01-15T18:00:27.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:27.906+0000 7f9ac3200640 1 -- 172.21.15.2:0/2063579924 wait complete. 2024-01-15T18:00:27.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:00:27.927 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:00:28.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.139+0000 7f6431200640 1 Processor -- start 2024-01-15T18:00:28.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.139+0000 7f6431200640 1 -- start start 2024-01-15T18:00:28.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f6431200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f642c143620 0x7f642c0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f6431200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f642c071940 0x7f642c0cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f6431200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0b2fa0 con 0x7f642c0b2a90 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f6431200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0b3280 con 0x7f642c143620 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f6431200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c0b3560 con 0x7f642c071940 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f642b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f642c143620 0x7f642c0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f642b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f642c143620 0x7f642c0b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35552/0 (socket says 172.21.15.2:35552) 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f642b400640 1 -- 172.21.15.2:0/2820127731 learned_addr learned my addr 172.21.15.2:0/2820127731 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f642be00640 1 --2- 172.21.15.2:0/2820127731 >> v2:172.21.15.137:3300/0 conn(0x7f642c071940 0x7f642c0cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.141+0000 7f642b400640 1 -- 172.21.15.2:0/2820127731 >> v2:172.21.15.137:3300/0 conn(0x7f642c071940 msgr2=0x7f642c0cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.142+0000 7f642b400640 1 --2- 172.21.15.2:0/2820127731 >> v2:172.21.15.137:3300/0 conn(0x7f642c071940 0x7f642c0cc3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.142+0000 7f642b400640 1 -- 172.21.15.2:0/2820127731 >> v1:172.21.15.137:6789/0 conn(0x7f642c0b2a90 legacy=0x7f642c071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:28.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.142+0000 7f642b400640 1 -- 172.21.15.2:0/2820127731 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c0b3840 con 0x7f642c143620 2024-01-15T18:00:28.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.143+0000 7f642b400640 1 --2- 172.21.15.2:0/2820127731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c0b2550 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f642c06fc80 tx=0x7f641c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b46701aade33bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:28.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.143+0000 7f642aa00640 1 -- 172.21.15.2:0/2820127731 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c031870 con 0x7f642c143620 2024-01-15T18:00:28.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.143+0000 7f642aa00640 1 -- 172.21.15.2:0/2820127731 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c0319d0 con 0x7f642c143620 2024-01-15T18:00:28.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.143+0000 7f642aa00640 1 -- 172.21.15.2:0/2820127731 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c031d60 con 0x7f642c143620 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 -- 172.21.15.2:0/2820127731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 msgr2=0x7f642c0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 --2- 172.21.15.2:0/2820127731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c0b2550 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f642c06fc80 tx=0x7f641c02b910 comp rx=0 tx=0).stop 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 -- 172.21.15.2:0/2820127731 shutdown_connections 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 --2- 172.21.15.2:0/2820127731 >> v2:172.21.15.137:3300/0 conn(0x7f642c071940 0x7f642c0cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 --2- 172.21.15.2:0/2820127731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c0b2550 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 -- 172.21.15.2:0/2820127731 >> 172.21.15.2:0/2820127731 conn(0x7f642c0b0840 msgr2=0x7f642c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 -- 172.21.15.2:0/2820127731 shutdown_connections 2024-01-15T18:00:28.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.144+0000 7f6431200640 1 -- 172.21.15.2:0/2820127731 wait complete. 2024-01-15T18:00:28.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.146+0000 7f6431200640 1 Processor -- start 2024-01-15T18:00:28.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.146+0000 7f6431200640 1 -- start start 2024-01-15T18:00:28.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6431200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c071940 0x7f642c1550b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6431200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c0b2a90 0x7f642c0ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6431200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c13a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6431200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c153ce0 con 0x7f642c143620 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6431200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1562f0 con 0x7f642c0b2a90 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6431200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0ae340 con 0x7f642c071940 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f642be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c13a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f642be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c13a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35564/0 (socket says 172.21.15.2:35564) 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f642be00640 1 -- 172.21.15.2:0/1698620289 learned_addr learned my addr 172.21.15.2:0/1698620289 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.147+0000 7f6423e00640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c0b2a90 0x7f642c0ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642be00640 1 -- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c071940 msgr2=0x7f642c1550b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642b400640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c071940 0x7f642c1550b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642be00640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c071940 0x7f642c1550b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642be00640 1 -- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c0b2a90 msgr2=0x7f642c0ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642be00640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c0b2a90 0x7f642c0ade00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642be00640 1 -- 172.21.15.2:0/1698620289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0094d0 con 0x7f642c143620 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f6423e00640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c0b2a90 0x7f642c0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f642be00640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c13a520 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f642400cd70 tx=0x7f642400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424003b90 con 0x7f642c143620 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424003cf0 con 0x7f642c143620 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c155770 con 0x7f642c143620 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424004240 con 0x7f642c143620 2024-01-15T18:00:28.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.148+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c155d30 con 0x7f642c143620 2024-01-15T18:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.149+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c074770 con 0x7f642c143620 2024-01-15T18:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.150+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f64240043a0 con 0x7f642c143620 2024-01-15T18:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.151+0000 7f6423400640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406b550 0x7f640406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.151+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6424012070 con 0x7f642c143620 2024-01-15T18:00:28.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.152+0000 7f642b400640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406b550 0x7f640406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.153+0000 7f642b400640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406b550 0x7f640406da10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f641c036000 tx=0x7f641c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:28.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.155+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6424055450 con 0x7f642c143620 2024-01-15T18:00:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.348+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f642c06e3e0 con 0x7f642c143620 2024-01-15T18:00:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.349+0000 7f6423400640 1 -- 172.21.15.2:0/1698620289 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6424058a70 con 0x7f642c143620 2024-01-15T18:00:28.352 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:00:28.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406b550 msgr2=0x7f640406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406b550 0x7f640406da10 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f641c036000 tx=0x7f641c03c040 comp rx=0 tx=0).stop 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 msgr2=0x7f642c13a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c13a520 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f642400cd70 tx=0x7f642400a660 comp rx=0 tx=0).stop 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 shutdown_connections 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406b550 0x7f640406da10 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c143620 0x7f642c13a520 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c0b2a90 0x7f642c0ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 --2- 172.21.15.2:0/1698620289 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c071940 0x7f642c1550b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 >> 172.21.15.2:0/1698620289 conn(0x7f642c0b0840 msgr2=0x7f642c0ccc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 shutdown_connections 2024-01-15T18:00:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.352+0000 7f6431200640 1 -- 172.21.15.2:0/1698620289 wait complete. 2024-01-15T18:00:28.374 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:00:28.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.592+0000 7f3b8c400640 1 Processor -- start 2024-01-15T18:00:28.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.593+0000 7f3b8c400640 1 -- start start 2024-01-15T18:00:28.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.594+0000 7f3b8c400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b841439e0 0x7f3b8406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.594+0000 7f3b8c400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3b840cc0a0 0x7f3b84070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.594+0000 7f3b8c400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b84143dc0 con 0x7f3b8406cdf0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.594+0000 7f3b8c400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8406d280 con 0x7f3b840cc0a0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.594+0000 7f3b8c400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b8415b620 con 0x7f3b841439e0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.595+0000 7f3b8a400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3b840cc0a0 0x7f3b84070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.595+0000 7f3b89000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3b8406cdf0 0x7f3b840cb990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56298/0 (socket says 172.21.15.2:56298) 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.595+0000 7f3b89000640 1 -- 172.21.15.2:0/1407013473 learned_addr learned my addr 172.21.15.2:0/1407013473 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.595+0000 7f3b89a00640 1 --2- 172.21.15.2:0/1407013473 >> v2:172.21.15.137:3300/0 conn(0x7f3b841439e0 0x7f3b8406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.595+0000 7f3b83e00640 1 -- 172.21.15.2:0/1407013473 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076337727 0 0) 0x7f3b84143dc0 con 0x7f3b8406cdf0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.595+0000 7f3b83e00640 1 -- 172.21.15.2:0/1407013473 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b64005730 con 0x7f3b8406cdf0 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.596+0000 7f3b8a400640 1 -- 172.21.15.2:0/1407013473 >> v2:172.21.15.137:3300/0 conn(0x7f3b841439e0 msgr2=0x7f3b8406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.596+0000 7f3b8a400640 1 --2- 172.21.15.2:0/1407013473 >> v2:172.21.15.137:3300/0 conn(0x7f3b841439e0 0x7f3b8406c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.596+0000 7f3b8a400640 1 -- 172.21.15.2:0/1407013473 >> v1:172.21.15.2:6789/0 conn(0x7f3b8406cdf0 legacy=0x7f3b840cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.596+0000 7f3b8a400640 1 -- 172.21.15.2:0/1407013473 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840cd9f0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.597+0000 7f3b8a400640 1 --2- 172.21.15.2:0/1407013473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b84070380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3b8406d400 tx=0x7f3b7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd17c684539cec6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.597+0000 7f3b83e00640 1 -- 172.21.15.2:0/1407013473 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c02cc80 con 0x7f3b840cc0a0 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.597+0000 7f3b83e00640 1 -- 172.21.15.2:0/1407013473 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7c02cde0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.597+0000 7f3b83e00640 1 -- 172.21.15.2:0/1407013473 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7c034660 con 0x7f3b840cc0a0 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 -- 172.21.15.2:0/1407013473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 msgr2=0x7f3b84070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 --2- 172.21.15.2:0/1407013473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b84070380 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f3b8406d400 tx=0x7f3b7c02bed0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 -- 172.21.15.2:0/1407013473 shutdown_connections 2024-01-15T18:00:28.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 --2- 172.21.15.2:0/1407013473 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b84070380 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 --2- 172.21.15.2:0/1407013473 >> v2:172.21.15.137:3300/0 conn(0x7f3b841439e0 0x7f3b8406c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 -- 172.21.15.2:0/1407013473 >> 172.21.15.2:0/1407013473 conn(0x7f3b840b46d0 msgr2=0x7f3b840b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:28.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 -- 172.21.15.2:0/1407013473 shutdown_connections 2024-01-15T18:00:28.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.598+0000 7f3b8c400640 1 -- 172.21.15.2:0/1407013473 wait complete. 2024-01-15T18:00:28.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.600+0000 7f3b8c400640 1 Processor -- start 2024-01-15T18:00:28.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.600+0000 7f3b8c400640 1 -- start start 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b8c400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b8406cdf0 0x7f3b8413f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b8c400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b840c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b8c400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b841439e0 0x7f3b84140870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b8c400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b840c75a0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b8c400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841f62f0 con 0x7f3b8406cdf0 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b8c400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841f65d0 con 0x7f3b841439e0 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b840c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b840c6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35576/0 (socket says 172.21.15.2:35576) 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 -- 172.21.15.2:0/568734961 learned_addr learned my addr 172.21.15.2:0/568734961 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 -- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b841439e0 msgr2=0x7f3b84140870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b841439e0 0x7f3b84140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 -- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b8406cdf0 msgr2=0x7f3b8413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b8406cdf0 0x7f3b8413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.601+0000 7f3b89000640 1 -- 172.21.15.2:0/568734961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7c009a70 con 0x7f3b840cc0a0 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.602+0000 7f3b89000640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b840c6bc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00c500 tx=0x7f3b6c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.602+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c005ce0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.602+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b6c005e40 con 0x7f3b840cc0a0 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.602+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c007330 con 0x7f3b840cc0a0 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.602+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b841f68b0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.602+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b841f70b0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.603+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8415da50 con 0x7f3b840cc0a0 2024-01-15T18:00:28.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.604+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3b6c007550 con 0x7f3b840cc0a0 2024-01-15T18:00:28.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.605+0000 7f3b81600640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5c06b500 0x7f3b5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:28.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.605+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f3b6c0893c0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.606+0000 7f3b89a00640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5c06b500 0x7f3b5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:28.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.607+0000 7f3b89a00640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5c06b500 0x7f3b5c06d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3b74006a90 tx=0x7f3b74006a20 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:28.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.610+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b6c054260 con 0x7f3b840cc0a0 2024-01-15T18:00:28.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.819+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3b841f6dd0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.820+0000 7f3b81600640 1 -- 172.21.15.2:0/568734961 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v39) v1 ==== 80+0+120 (secure 0 0 0) 0x7f3b841f6dd0 con 0x7f3b840cc0a0 2024-01-15T18:00:28.823 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/2859107199 2024-01-16T18:00:22.271924+0000 2024-01-15T18:00:28.823 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6832/2859107199 2024-01-16T18:00:22.271924+0000 2024-01-15T18:00:28.823 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:00:28.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5c06b500 msgr2=0x7f3b5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5c06b500 0x7f3b5c06d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f3b74006a90 tx=0x7f3b74006a20 comp rx=0 tx=0).stop 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 msgr2=0x7f3b840c6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b840c6bc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f3b6c00c500 tx=0x7f3b6c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 shutdown_connections 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5c06b500 0x7f3b5c06d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b841439e0 0x7f3b84140870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b840cc0a0 0x7f3b840c6bc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 --2- 172.21.15.2:0/568734961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b8406cdf0 0x7f3b8413f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 >> 172.21.15.2:0/568734961 conn(0x7f3b840b46d0 msgr2=0x7f3b840716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 shutdown_connections 2024-01-15T18:00:28.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:28.823+0000 7f3b8c400640 1 -- 172.21.15.2:0/568734961 wait complete. 2024-01-15T18:00:28.846 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:00:28.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:28.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:28.943 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[42662]: fuse finished with error 0 and tester_r 0 2024-01-15T18:00:28.949 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:00:28.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:28.949 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:00:29.133 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:00:29.134 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:29.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:00:29.193 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:00:29.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:29.193 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:00:29.248 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:00:29.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:00:29.250 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:00:29.250 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:29.307 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:00:29.307 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:00:29.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:29.307 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:00:29.360 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:00:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:00:29.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:00:29.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:00:29.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:00:29.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.575+0000 7f5b28c00640 1 Processor -- start 2024-01-15T18:00:29.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.575+0000 7f5b28c00640 1 -- start start 2024-01-15T18:00:29.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b28c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b240cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b28c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5b2415b7d0 0x7f5b2406c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b28c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5b2415c4c0 0x7f5b2406c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b28c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2415cb00 con 0x7f5b2415b7d0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b28c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b240ccec0 con 0x7f5b2415c4c0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b28c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240cd1a0 con 0x7f5b2415c0e0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b1be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5b2415b7d0 0x7f5b2406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b1be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5b2415b7d0 0x7f5b2406c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35584/0 (socket says 172.21.15.2:35584) 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b1be00640 1 -- 172.21.15.2:0/823638417 learned_addr learned my addr 172.21.15.2:0/823638417 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b21e00640 1 --2- 172.21.15.2:0/823638417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b240cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b1be00640 1 -- 172.21.15.2:0/823638417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 msgr2=0x7f5b240cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b1be00640 1 --2- 172.21.15.2:0/823638417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b240cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b22800640 1 --2- 172.21.15.2:0/823638417 >> v2:172.21.15.137:3300/0 conn(0x7f5b2415c4c0 0x7f5b2406c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.577+0000 7f5b1be00640 1 -- 172.21.15.2:0/823638417 >> v2:172.21.15.137:3300/0 conn(0x7f5b2415c4c0 msgr2=0x7f5b2406c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.578+0000 7f5b1be00640 1 --2- 172.21.15.2:0/823638417 >> v2:172.21.15.137:3300/0 conn(0x7f5b2415c4c0 0x7f5b2406c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.578+0000 7f5b1be00640 1 -- 172.21.15.2:0/823638417 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240cd480 con 0x7f5b2415b7d0 2024-01-15T18:00:29.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.578+0000 7f5b22800640 1 --2- 172.21.15.2:0/823638417 >> v2:172.21.15.137:3300/0 conn(0x7f5b2415c4c0 0x7f5b2406c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:29.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.579+0000 7f5b1be00640 1 --2- 172.21.15.2:0/823638417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415b7d0 0x7f5b2406c470 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5b240cc230 tx=0x7f5b0c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70461f2a56397d32 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:29.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.579+0000 7f5b21400640 1 -- 172.21.15.2:0/823638417 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c031860 con 0x7f5b2415b7d0 2024-01-15T18:00:29.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.579+0000 7f5b21400640 1 -- 172.21.15.2:0/823638417 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0c0319c0 con 0x7f5b2415b7d0 2024-01-15T18:00:29.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.579+0000 7f5b21400640 1 -- 172.21.15.2:0/823638417 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c031d50 con 0x7f5b2415b7d0 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 -- 172.21.15.2:0/823638417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415b7d0 msgr2=0x7f5b2406c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 --2- 172.21.15.2:0/823638417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415b7d0 0x7f5b2406c470 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5b240cc230 tx=0x7f5b0c02b900 comp rx=0 tx=0).stop 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 -- 172.21.15.2:0/823638417 shutdown_connections 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 --2- 172.21.15.2:0/823638417 >> v2:172.21.15.137:3300/0 conn(0x7f5b2415c4c0 0x7f5b2406c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 --2- 172.21.15.2:0/823638417 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415b7d0 0x7f5b2406c470 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 --2- 172.21.15.2:0/823638417 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b240cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 -- 172.21.15.2:0/823638417 >> 172.21.15.2:0/823638417 conn(0x7f5b2406da20 msgr2=0x7f5b2406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 -- 172.21.15.2:0/823638417 shutdown_connections 2024-01-15T18:00:29.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.580+0000 7f5b28c00640 1 -- 172.21.15.2:0/823638417 wait complete. 2024-01-15T18:00:29.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.582+0000 7f5b28c00640 1 Processor -- start 2024-01-15T18:00:29.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.582+0000 7f5b28c00640 1 -- start start 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b28c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2415b7d0 0x7f5b241490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b28c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b24146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b28c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 0x7f5b24147910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b28c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24148c20 con 0x7f5b2415c4c0 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b28c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b24148010 con 0x7f5b2415b7d0 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b28c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b241482f0 con 0x7f5b2415c0e0 2024-01-15T18:00:29.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b22800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 0x7f5b24147910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b22800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 0x7f5b24147910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35596/0 (socket says 172.21.15.2:35596) 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.583+0000 7f5b22800640 1 -- 172.21.15.2:0/1966794807 learned_addr learned my addr 172.21.15.2:0/1966794807 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b21e00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2415b7d0 0x7f5b241490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b1be00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b24146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b22800640 1 -- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 msgr2=0x7f5b24146bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b22800640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b24146bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b22800640 1 -- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2415b7d0 msgr2=0x7f5b241490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b22800640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2415b7d0 0x7f5b241490b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b22800640 1 -- 172.21.15.2:0/1966794807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c0094d0 con 0x7f5b2415c4c0 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b1be00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b24146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:29.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.584+0000 7f5b22800640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 0x7f5b24147910 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00cd50 tx=0x7f5b1c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:29.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.585+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00f680 con 0x7f5b2415c4c0 2024-01-15T18:00:29.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.585+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c00f800 con 0x7f5b2415c4c0 2024-01-15T18:00:29.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.585+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b241485d0 con 0x7f5b2415c4c0 2024-01-15T18:00:29.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.585+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c00fbb0 con 0x7f5b2415c4c0 2024-01-15T18:00:29.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.585+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2414c050 con 0x7f5b2415c4c0 2024-01-15T18:00:29.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.586+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b24148d90 con 0x7f5b2415c4c0 2024-01-15T18:00:29.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.587+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5b1c00b040 con 0x7f5b2415c4c0 2024-01-15T18:00:29.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.588+0000 7f5b1a000640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5afc06b500 0x7f5afc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:29.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.588+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5b1c08a840 con 0x7f5b2415c4c0 2024-01-15T18:00:29.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.588+0000 7f5b21e00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5afc06b500 0x7f5afc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:29.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.590+0000 7f5b21e00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5afc06b500 0x7f5afc06d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5b14006fd0 tx=0x7f5b14008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:29.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.592+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b1c055510 con 0x7f5b2415c4c0 2024-01-15T18:00:29.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.828+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b240686f0 con 0x7f5b2415c4c0 2024-01-15T18:00:29.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.829+0000 7f5b1a000640 1 -- 172.21.15.2:0/1966794807 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3687 (secure 0 0 0) 0x7f5b1c058b30 con 0x7f5b2415c4c0 2024-01-15T18:00:29.832 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:29.833 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/122233456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":122233456},{"type":"v1","addr":"172.21.15.137:6837","nonce":122233456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1272974394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":1272974394},{"type":"v1","addr":"172.21.15.2:6835","nonce":1272974394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/84168043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":84168043},{"type":"v1","addr":"172.21.15.137:6835","nonce":84168043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5601,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/645061371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":645061371},{"type":"v1","addr":"172.21.15.2:6834","nonce":645061371}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-01-15T18:00:29.833 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 19 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5afc06b500 msgr2=0x7f5afc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5afc06b500 0x7f5afc06d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5b14006fd0 tx=0x7f5b14008040 comp rx=0 tx=0).stop 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 msgr2=0x7f5b24147910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 0x7f5b24147910 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00cd50 tx=0x7f5b1c00a650 comp rx=0 tx=0).stop 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 shutdown_connections 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5afc06b500 0x7f5afc06d9c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5b2415c4c0 0x7f5b24147910 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5b2415c0e0 0x7f5b24146bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 --2- 172.21.15.2:0/1966794807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5b2415b7d0 0x7f5b241490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:29.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 >> 172.21.15.2:0/1966794807 conn(0x7f5b2406da20 msgr2=0x7f5b2415cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:29.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.832+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 shutdown_connections 2024-01-15T18:00:29.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:29.833+0000 7f5b28c00640 1 -- 172.21.15.2:0/1966794807 wait complete. 2024-01-15T18:00:29.855 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:00:30.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.066+0000 7fecbda00640 1 Processor -- start 2024-01-15T18:00:30.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.067+0000 7fecbda00640 1 -- start start 2024-01-15T18:00:30.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.068+0000 7fecbda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1870 0x7fecb80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.068+0000 7fecbda00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fecb80b2210 0x7fecb815d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.068+0000 7fecbda00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb80d1250 con 0x7fecb80d1c50 2024-01-15T18:00:30.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.068+0000 7fecbda00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb80ce6d0 con 0x7fecb80b2210 2024-01-15T18:00:30.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.068+0000 7fecbda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80ce8e0 con 0x7fecb80d1870 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.069+0000 7fecb7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1870 0x7fecb80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.069+0000 7fecbd000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fecb80d1c50 0x7fecb80cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56304/0 (socket says 172.21.15.2:56304) 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.069+0000 7fecbd000640 1 -- 172.21.15.2:0/2352484973 learned_addr learned my addr 172.21.15.2:0/2352484973 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.069+0000 7fecb7400640 1 --2- 172.21.15.2:0/2352484973 >> v2:172.21.15.137:3300/0 conn(0x7fecb80b2210 0x7fecb815d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.070+0000 7fecb6a00640 1 -- 172.21.15.2:0/2352484973 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57755980 0 0) 0x7fecb80d1250 con 0x7fecb80d1c50 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.070+0000 7fecb6a00640 1 -- 172.21.15.2:0/2352484973 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec98003520 con 0x7fecb80d1c50 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.070+0000 7fecb7e00640 1 -- 172.21.15.2:0/2352484973 >> v2:172.21.15.137:3300/0 conn(0x7fecb80b2210 msgr2=0x7fecb815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.070+0000 7fecb7e00640 1 --2- 172.21.15.2:0/2352484973 >> v2:172.21.15.137:3300/0 conn(0x7fecb80b2210 0x7fecb815d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.070+0000 7fecb7e00640 1 -- 172.21.15.2:0/2352484973 >> v1:172.21.15.2:6789/0 conn(0x7fecb80d1c50 legacy=0x7fecb80cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.070+0000 7fecb7e00640 1 -- 172.21.15.2:0/2352484973 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb80cebc0 con 0x7fecb80d1870 2024-01-15T18:00:30.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.072+0000 7fecb7e00640 1 --2- 172.21.15.2:0/2352484973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1870 0x7fecb80d3bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feca80097a0 tx=0x7feca802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6b1f9b8823ef522 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.072+0000 7fecb6a00640 1 -- 172.21.15.2:0/2352484973 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca802d040 con 0x7fecb80d1870 2024-01-15T18:00:30.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.072+0000 7fecb6a00640 1 -- 172.21.15.2:0/2352484973 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca8031810 con 0x7fecb80d1870 2024-01-15T18:00:30.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.072+0000 7fecb6a00640 1 -- 172.21.15.2:0/2352484973 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca8031ba0 con 0x7fecb80d1870 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 -- 172.21.15.2:0/2352484973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1870 msgr2=0x7fecb80d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 --2- 172.21.15.2:0/2352484973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1870 0x7fecb80d3bd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7feca80097a0 tx=0x7feca802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 -- 172.21.15.2:0/2352484973 shutdown_connections 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 --2- 172.21.15.2:0/2352484973 >> v2:172.21.15.137:3300/0 conn(0x7fecb80b2210 0x7fecb815d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 --2- 172.21.15.2:0/2352484973 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1870 0x7fecb80d3bd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 -- 172.21.15.2:0/2352484973 >> 172.21.15.2:0/2352484973 conn(0x7fecb80d30b0 msgr2=0x7fecb80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 -- 172.21.15.2:0/2352484973 shutdown_connections 2024-01-15T18:00:30.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.073+0000 7fecbda00640 1 -- 172.21.15.2:0/2352484973 wait complete. 2024-01-15T18:00:30.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.075+0000 7fecbda00640 1 Processor -- start 2024-01-15T18:00:30.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.075+0000 7fecbda00640 1 -- start start 2024-01-15T18:00:30.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecbda00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 0x7fecb81f37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecbda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fecb80d1870 0x7fecb80a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecbda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1c50 0x7fecb80cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecbda00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b7d60 con 0x7fecb80b2210 2024-01-15T18:00:30.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecbda00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b8040 con 0x7fecb80d1870 2024-01-15T18:00:30.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecbda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b8320 con 0x7fecb80d1c50 2024-01-15T18:00:30.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecb7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 0x7fecb81f37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecb7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 0x7fecb81f37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35600/0 (socket says 172.21.15.2:35600) 2024-01-15T18:00:30.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.076+0000 7fecb7e00640 1 -- 172.21.15.2:0/3636552652 learned_addr learned my addr 172.21.15.2:0/3636552652 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:30.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7400640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fecb80d1870 0x7fecb80a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecbd000640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1c50 0x7fecb80cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7e00640 1 -- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1c50 msgr2=0x7fecb80cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7e00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1c50 0x7fecb80cfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7e00640 1 -- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fecb80d1870 msgr2=0x7fecb80a04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7e00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fecb80d1870 0x7fecb80a04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7e00640 1 -- 172.21.15.2:0/3636552652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca8009400 con 0x7fecb80b2210 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecbd000640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1c50 0x7fecb80cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7400640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fecb80d1870 0x7fecb80a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.077+0000 7fecb7e00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 0x7fecb81f37b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7feca80098c0 tx=0x7feca80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.078+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca803ace0 con 0x7fecb80b2210 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.078+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca803ae60 con 0x7fecb80b2210 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.078+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca8044640 con 0x7fecb80b2210 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.078+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb81f3290 con 0x7fecb80b2210 2024-01-15T18:00:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.078+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb80b8c80 con 0x7fecb80b2210 2024-01-15T18:00:30.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.079+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb8074770 con 0x7fecb80b2210 2024-01-15T18:00:30.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.080+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7feca804d440 con 0x7fecb80b2210 2024-01-15T18:00:30.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.081+0000 7fecafe00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec9006b500 0x7fec9006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.081+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7feca80ac020 con 0x7fecb80b2210 2024-01-15T18:00:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.081+0000 7fecb7400640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec9006b500 0x7fec9006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.083+0000 7fecb7400640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec9006b500 0x7fec9006d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fecb80b0160 tx=0x7feca0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.085+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feca8076d10 con 0x7fecb80b2210 2024-01-15T18:00:30.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.285+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fecb80d2e90 con 0x7fecb80b2210 2024-01-15T18:00:30.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.290+0000 7fecafe00640 1 -- 172.21.15.2:0/3636552652 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v12) v1 ==== 120+0+0 (secure 0 0 0) 0x7feca807a330 con 0x7fecb80b2210 2024-01-15T18:00:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec9006b500 msgr2=0x7fec9006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec9006b500 0x7fec9006d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fecb80b0160 tx=0x7feca0008040 comp rx=0 tx=0).stop 2024-01-15T18:00:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 msgr2=0x7fecb81f37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 0x7fecb81f37b0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7feca80098c0 tx=0x7feca80046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 shutdown_connections 2024-01-15T18:00:30.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec9006b500 0x7fec9006d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fecb80d1c50 0x7fecb80cfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fecb80d1870 0x7fecb80a04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 --2- 172.21.15.2:0/3636552652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fecb80b2210 0x7fecb81f37b0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 >> 172.21.15.2:0/3636552652 conn(0x7fecb80d30b0 msgr2=0x7fecb80d2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 shutdown_connections 2024-01-15T18:00:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.294+0000 7fecbda00640 1 -- 172.21.15.2:0/3636552652 wait complete. 2024-01-15T18:00:30.316 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:00:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.526+0000 7f6a2ec00640 1 Processor -- start 2024-01-15T18:00:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.526+0000 7f6a2ec00640 1 -- start start 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2ec00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6a2815e0b0 0x7f6a280d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2ec00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2806f440 con 0x7f6a280b32b0 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2ec00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2815e5f0 con 0x7f6a28146ba0 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2ec00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a28146f80 con 0x7f6a2815e0b0 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2cc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6a2815e0b0 0x7f6a280d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2cc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6a2815e0b0 0x7f6a280d1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35616/0 (socket says 172.21.15.2:35616) 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a27400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6a280b32b0 0x7f6a2815d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56320/0 (socket says 172.21.15.2:56320) 2024-01-15T18:00:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.528+0000 7f6a2cc00640 1 -- 172.21.15.2:0/3686247263 learned_addr learned my addr 172.21.15.2:0/3686247263 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:30.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.529+0000 7f6a2cc00640 1 -- 172.21.15.2:0/3686247263 >> v1:172.21.15.137:6789/0 conn(0x7f6a28146ba0 legacy=0x7f6a280b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:30.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.529+0000 7f6a2cc00640 1 -- 172.21.15.2:0/3686247263 >> v1:172.21.15.2:6789/0 conn(0x7f6a280b32b0 legacy=0x7f6a2815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.529+0000 7f6a2cc00640 1 -- 172.21.15.2:0/3686247263 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2815e8d0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.530+0000 7f6a2cc00640 1 --2- 172.21.15.2:0/3686247263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a280d1a60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009870 tx=0x7f6a2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93317a7addfcbb15 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.530+0000 7f6a26a00640 1 -- 172.21.15.2:0/3686247263 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20031870 con 0x7f6a2815e0b0 2024-01-15T18:00:30.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.530+0000 7f6a26a00640 1 -- 172.21.15.2:0/3686247263 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a200319d0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.530+0000 7f6a26a00640 1 -- 172.21.15.2:0/3686247263 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20031d60 con 0x7f6a2815e0b0 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 -- 172.21.15.2:0/3686247263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 msgr2=0x7f6a280d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/3686247263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a280d1a60 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009870 tx=0x7f6a2002b910 comp rx=0 tx=0).stop 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 -- 172.21.15.2:0/3686247263 shutdown_connections 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/3686247263 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a280d1a60 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 -- 172.21.15.2:0/3686247263 >> 172.21.15.2:0/3686247263 conn(0x7f6a280b6430 msgr2=0x7f6a280b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 -- 172.21.15.2:0/3686247263 shutdown_connections 2024-01-15T18:00:30.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.531+0000 7f6a2ec00640 1 -- 172.21.15.2:0/3686247263 wait complete. 2024-01-15T18:00:30.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.532+0000 7f6a2ec00640 1 Processor -- start 2024-01-15T18:00:30.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.533+0000 7f6a2ec00640 1 -- start start 2024-01-15T18:00:30.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a280b32b0 0x7f6a28145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a28146ba0 0x7f6a280cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2ec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a2815ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2ec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a28135650 con 0x7f6a2815e0b0 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280a85e0 con 0x7f6a280b32b0 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a280a88a0 con 0x7f6a28146ba0 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a2815ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a2815ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35624/0 (socket says 172.21.15.2:35624) 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 -- 172.21.15.2:0/1251678938 learned_addr learned my addr 172.21.15.2:0/1251678938 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a27e00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a280b32b0 0x7f6a28145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a27400640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a28146ba0 0x7f6a280cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 -- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a28146ba0 msgr2=0x7f6a280cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a28146ba0 0x7f6a280cd380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 -- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a280b32b0 msgr2=0x7f6a28145560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a280b32b0 0x7f6a28145560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.534+0000 7f6a2cc00640 1 -- 172.21.15.2:0/1251678938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200094d0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.535+0000 7f6a2cc00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a2815ad00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6a2002b950 tx=0x7f6a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.535+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20031800 con 0x7f6a2815e0b0 2024-01-15T18:00:30.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.535+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a200443f0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.535+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a200447a0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.535+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a28135000 con 0x7f6a2815e0b0 2024-01-15T18:00:30.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.535+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2815b240 con 0x7f6a2815e0b0 2024-01-15T18:00:30.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.537+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a281357c0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.537+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f6a2004d450 con 0x7f6a2815e0b0 2024-01-15T18:00:30.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.538+0000 7f6a1fe00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a0006b500 0x7f6a0006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.538+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6a2002d040 con 0x7f6a2815e0b0 2024-01-15T18:00:30.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.539+0000 7f6a27e00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a0006b500 0x7f6a0006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.540+0000 7f6a27e00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a0006b500 0x7f6a0006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6a280d0c40 tx=0x7f6a18009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.542+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a20077220 con 0x7f6a2815e0b0 2024-01-15T18:00:30.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.737+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6a280686f0 con 0x7f6a2815e0b0 2024-01-15T18:00:30.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.744+0000 7f6a1fe00640 1 -- 172.21.15.2:0/1251678938 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6a2007a840 con 0x7f6a2815e0b0 2024-01-15T18:00:30.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a0006b500 msgr2=0x7f6a0006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a0006b500 0x7f6a0006d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6a280d0c40 tx=0x7f6a18009040 comp rx=0 tx=0).stop 2024-01-15T18:00:30.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 msgr2=0x7f6a2815ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a2815ad00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6a2002b950 tx=0x7f6a200046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 shutdown_connections 2024-01-15T18:00:30.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a0006b500 0x7f6a0006d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a2815e0b0 0x7f6a2815ad00 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a28146ba0 0x7f6a280cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 --2- 172.21.15.2:0/1251678938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a280b32b0 0x7f6a28145560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 >> 172.21.15.2:0/1251678938 conn(0x7f6a280b6430 msgr2=0x7f6a2815d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:30.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 shutdown_connections 2024-01-15T18:00:30.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.747+0000 7f6a2ec00640 1 -- 172.21.15.2:0/1251678938 wait complete. 2024-01-15T18:00:30.769 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-15T18:00:30.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.985+0000 7fa533400640 1 Processor -- start 2024-01-15T18:00:30.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.985+0000 7fa533400640 1 -- start start 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa533400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 0x7fa52c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa533400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa52c06cad0 0x7fa52c06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa533400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c0716d0 con 0x7fa52c141880 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa533400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52c071f20 con 0x7fa52c06cad0 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa533400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c06c6b0 con 0x7fa52c15ec60 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa532000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa52c141880 0x7fa52c070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56330/0 (socket says 172.21.15.2:56330) 2024-01-15T18:00:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.987+0000 7fa532000640 1 -- 172.21.15.2:0/1609860504 learned_addr learned my addr 172.21.15.2:0/1609860504 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa531600640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 0x7fa52c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa532a00640 1 --2- 172.21.15.2:0/1609860504 >> v2:172.21.15.137:3300/0 conn(0x7fa52c06cad0 0x7fa52c06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770790063 0 0) 0x7fa52c0716d0 con 0x7fa52c141880 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa50c003520 con 0x7fa52c141880 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3764978904 0 0) 0x7fa50c003520 con 0x7fa52c141880 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 msgr2=0x7fa52c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 0x7fa52c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 >> v2:172.21.15.137:3300/0 conn(0x7fa52c06cad0 msgr2=0x7fa52c06cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 --2- 172.21.15.2:0/1609860504 >> v2:172.21.15.137:3300/0 conn(0x7fa52c06cad0 0x7fa52c06cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c071960 con 0x7fa52c141880 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fa5240019f0 con 0x7fa52c141880 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 >> v1:172.21.15.2:6789/0 conn(0x7fa52c141880 legacy=0x7fa52c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 0x7fa50c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.988+0000 7fa530c00640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa50c003ef0 0x7fa50c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa530c00640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c004d00 0x7fa50c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c071f20 con 0x7fa50c003ef0 2024-01-15T18:00:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c06c6b0 con 0x7fa50c004d00 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50c00b9d0 con 0x7fa52c15ec60 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa531600640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa50c003ef0 0x7fa50c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa532000640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 0x7fa50c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa532a00640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c004d00 0x7fa50c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa531600640 1 -- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 msgr2=0x7fa50c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa531600640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15ec60 0x7fa50c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa531600640 1 -- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c004d00 msgr2=0x7fa50c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa531600640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c004d00 0x7fa50c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.989+0000 7fa531600640 1 -- 172.21.15.2:0/1609860504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50c003520 con 0x7fa50c003ef0 2024-01-15T18:00:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.990+0000 7fa531600640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa50c003ef0 0x7fa50c0047c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa514009af0 tx=0x7fa50c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d421cde99e276c14 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51402c6b0 con 0x7fa50c003ef0 2024-01-15T18:00:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51402c810 con 0x7fa50c003ef0 2024-01-15T18:00:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa530c00640 1 -- 172.21.15.2:0/1609860504 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51402cba0 con 0x7fa50c003ef0 2024-01-15T18:00:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa533400640 1 -- 172.21.15.2:0/1609860504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa50c003ef0 msgr2=0x7fa50c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa533400640 1 --2- 172.21.15.2:0/1609860504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa50c003ef0 0x7fa50c0047c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fa514009af0 tx=0x7fa50c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa532a00640 1 -- 172.21.15.2:0/1609860504 reap_dead start 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa533400640 1 -- 172.21.15.2:0/1609860504 shutdown_connections 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa533400640 1 -- 172.21.15.2:0/1609860504 >> 172.21.15.2:0/1609860504 conn(0x7fa52c0b0840 msgr2=0x7fa52c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.991+0000 7fa533400640 1 -- 172.21.15.2:0/1609860504 shutdown_connections 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.992+0000 7fa533400640 1 -- 172.21.15.2:0/1609860504 wait complete. 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.993+0000 7fa533400640 1 Processor -- start 2024-01-15T18:00:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.993+0000 7fa533400640 1 -- start start 2024-01-15T18:00:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.994+0000 7fa533400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c003ef0 0x7fa52c0a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.994+0000 7fa533400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 0x7fa52c1578a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.994+0000 7fa533400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15cdb0 0x7fa52c0aec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.994+0000 7fa533400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c15bd10 con 0x7fa52c1406f0 2024-01-15T18:00:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.994+0000 7fa533400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c15bff0 con 0x7fa50c003ef0 2024-01-15T18:00:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.994+0000 7fa533400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c15c2d0 con 0x7fa52c15cdb0 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 0x7fa52c1578a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 0x7fa52c1578a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35648/0 (socket says 172.21.15.2:35648) 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 -- 172.21.15.2:0/602039848 learned_addr learned my addr 172.21.15.2:0/602039848 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa532000640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c003ef0 0x7fa52c0a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa532a00640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15cdb0 0x7fa52c0aec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 -- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15cdb0 msgr2=0x7fa52c0aec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15cdb0 0x7fa52c0aec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 -- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c003ef0 msgr2=0x7fa52c0a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c003ef0 0x7fa52c0a8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:30.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.995+0000 7fa531600640 1 -- 172.21.15.2:0/602039848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c15c5b0 con 0x7fa52c1406f0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.996+0000 7fa531600640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 0x7fa52c1578a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa514037040 tx=0x7fa514004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.996+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514003810 con 0x7fa52c1406f0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.996+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514003970 con 0x7fa52c1406f0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.996+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514003ce0 con 0x7fa52c1406f0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.996+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c15c890 con 0x7fa52c1406f0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.996+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c15b990 con 0x7fa52c1406f0 2024-01-15T18:00:31.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.997+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c15c410 con 0x7fa52c1406f0 2024-01-15T18:00:31.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.998+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa514034440 con 0x7fa52c1406f0 2024-01-15T18:00:31.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.999+0000 7fa52a000640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa50406b530 0x7fa50406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:30.999+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fa5140abea0 con 0x7fa52c1406f0 2024-01-15T18:00:31.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.000+0000 7fa532000640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa50406b530 0x7fa50406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.001+0000 7fa532000640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa50406b530 0x7fa50406d9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa5240013d0 tx=0x7fa524001830 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:31.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.003+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa52c15c410 con 0x7fa52c1406f0 2024-01-15T18:00:31.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.274+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7fa52c15c410 con 0x7fa52c1406f0 2024-01-15T18:00:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.395+0000 7fa52a000640 1 -- 172.21.15.2:0/602039848 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v159) v1 ==== 152+0+0 (secure 0 0 0) 0x7fa52c15c410 con 0x7fa52c1406f0 2024-01-15T18:00:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa50406b530 msgr2=0x7fa50406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa50406b530 0x7fa50406d9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa5240013d0 tx=0x7fa524001830 comp rx=0 tx=0).stop 2024-01-15T18:00:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 msgr2=0x7fa52c1578a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 0x7fa52c1578a0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fa514037040 tx=0x7fa514004730 comp rx=0 tx=0).stop 2024-01-15T18:00:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 shutdown_connections 2024-01-15T18:00:31.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa50406b530 0x7fa50406d9f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa52c15cdb0 0x7fa52c0aec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa52c1406f0 0x7fa52c1578a0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 --2- 172.21.15.2:0/602039848 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa50c003ef0 0x7fa52c0a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 >> 172.21.15.2:0/602039848 conn(0x7fa52c0b0840 msgr2=0x7fa52c15f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 shutdown_connections 2024-01-15T18:00:31.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.399+0000 7fa533400640 1 -- 172.21.15.2:0/602039848 wait complete. 2024-01-15T18:00:31.426 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-01-15T18:00:31.426 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-15T18:00:31.426 INFO:tasks.cephfs_test_runner:test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:00:31.427 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.652+0000 7fc12cc00640 1 Processor -- start 2024-01-15T18:00:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.652+0000 7fc12cc00640 1 -- start start 2024-01-15T18:00:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.653+0000 7fc12cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc12806bfd0 0x7fc12806c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.653+0000 7fc12cc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc1280cc170 0x7fc1280cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.653+0000 7fc12cc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1280cbce0 con 0x7fc12806b7d0 2024-01-15T18:00:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.653+0000 7fc12cc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12806c8f0 con 0x7fc1280cc170 2024-01-15T18:00:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.653+0000 7fc12cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1280d03f0 con 0x7fc12806bfd0 2024-01-15T18:00:31.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc1280cc170 0x7fc1280cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc126a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc12806b7d0 0x7fc1280cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56336/0 (socket says 172.21.15.2:56336) 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc126a00640 1 -- 172.21.15.2:0/4269682572 learned_addr learned my addr 172.21.15.2:0/4269682572 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc1280cc170 0x7fc1280cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35652/0 (socket says 172.21.15.2:35652) 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127400640 1 --2- 172.21.15.2:0/4269682572 >> v2:172.21.15.137:3300/0 conn(0x7fc12806bfd0 0x7fc12806c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127e00640 1 -- 172.21.15.2:0/4269682572 >> v2:172.21.15.137:3300/0 conn(0x7fc12806bfd0 msgr2=0x7fc12806c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127e00640 1 --2- 172.21.15.2:0/4269682572 >> v2:172.21.15.137:3300/0 conn(0x7fc12806bfd0 0x7fc12806c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127e00640 1 -- 172.21.15.2:0/4269682572 >> v1:172.21.15.2:6789/0 conn(0x7fc12806b7d0 legacy=0x7fc1280cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc127e00640 1 -- 172.21.15.2:0/4269682572 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1280cb690 con 0x7fc1280cc170 2024-01-15T18:00:31.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.654+0000 7fc126000640 1 -- 172.21.15.2:0/4269682572 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504465724 0 0) 0x7fc1280cbce0 con 0x7fc12806b7d0 2024-01-15T18:00:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.656+0000 7fc127e00640 1 --2- 172.21.15.2:0/4269682572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1280cc170 0x7fc1280cc600 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc1280715a0 tx=0x7fc12002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c698ff17b475b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.656+0000 7fc126000640 1 -- 172.21.15.2:0/4269682572 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12002cb60 con 0x7fc1280cc170 2024-01-15T18:00:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.656+0000 7fc126000640 1 -- 172.21.15.2:0/4269682572 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12002ccc0 con 0x7fc1280cc170 2024-01-15T18:00:31.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.656+0000 7fc126000640 1 -- 172.21.15.2:0/4269682572 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120034660 con 0x7fc1280cc170 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.656+0000 7fc12cc00640 1 -- 172.21.15.2:0/4269682572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1280cc170 msgr2=0x7fc1280cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.656+0000 7fc12cc00640 1 --2- 172.21.15.2:0/4269682572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1280cc170 0x7fc1280cc600 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fc1280715a0 tx=0x7fc12002bdb0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.657+0000 7fc12cc00640 1 -- 172.21.15.2:0/4269682572 shutdown_connections 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.657+0000 7fc12cc00640 1 --2- 172.21.15.2:0/4269682572 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1280cc170 0x7fc1280cc600 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.657+0000 7fc12cc00640 1 --2- 172.21.15.2:0/4269682572 >> v2:172.21.15.137:3300/0 conn(0x7fc12806bfd0 0x7fc12806c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.657+0000 7fc12cc00640 1 -- 172.21.15.2:0/4269682572 >> 172.21.15.2:0/4269682572 conn(0x7fc1280d0ec0 msgr2=0x7fc1280d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.657+0000 7fc12cc00640 1 -- 172.21.15.2:0/4269682572 shutdown_connections 2024-01-15T18:00:31.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.657+0000 7fc12cc00640 1 -- 172.21.15.2:0/4269682572 wait complete. 2024-01-15T18:00:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.658+0000 7fc12cc00640 1 Processor -- start 2024-01-15T18:00:31.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.658+0000 7fc12cc00640 1 -- start start 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.659+0000 7fc12cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 0x7fc1280b1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.659+0000 7fc12cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc12806bfd0 0x7fc1281548e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.659+0000 7fc12cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1280cc170 0x7fc1280c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.659+0000 7fc12cc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280c8e90 con 0x7fc12806b7d0 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.659+0000 7fc12cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280b05c0 con 0x7fc1280cc170 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.659+0000 7fc12cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1280b08a0 con 0x7fc12806bfd0 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 0x7fc1280b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 0x7fc1280b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35662/0 (socket says 172.21.15.2:35662) 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 -- 172.21.15.2:0/2523120658 learned_addr learned my addr 172.21.15.2:0/2523120658 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc126a00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc12806bfd0 0x7fc1281548e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127e00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1280cc170 0x7fc1280c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 -- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc12806bfd0 msgr2=0x7fc1281548e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc12806bfd0 0x7fc1281548e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 -- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1280cc170 msgr2=0x7fc1280c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1280cc170 0x7fc1280c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127400640 1 -- 172.21.15.2:0/2523120658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120009a70 con 0x7fc12806b7d0 2024-01-15T18:00:31.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.660+0000 7fc127e00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1280cc170 0x7fc1280c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:31.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.661+0000 7fc127400640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 0x7fc1280b1660 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc11800d800 tx=0x7fc11800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:31.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.661+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800b6c0 con 0x7fc12806b7d0 2024-01-15T18:00:31.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.661+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11800b820 con 0x7fc12806b7d0 2024-01-15T18:00:31.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.661+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1280b0b80 con 0x7fc12806b7d0 2024-01-15T18:00:31.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.661+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800bbb0 con 0x7fc12806b7d0 2024-01-15T18:00:31.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.661+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1280b1140 con 0x7fc12806b7d0 2024-01-15T18:00:31.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.662+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc128074770 con 0x7fc12806b7d0 2024-01-15T18:00:31.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.663+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc11800bd10 con 0x7fc12806b7d0 2024-01-15T18:00:31.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.664+0000 7fc11f400640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc10006b500 0x7fc10006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:31.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.664+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc11808a280 con 0x7fc12806b7d0 2024-01-15T18:00:31.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.665+0000 7fc126a00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc10006b500 0x7fc10006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:31.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.666+0000 7fc126a00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc10006b500 0x7fc10006d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc1280b7150 tx=0x7fc110007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:31.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.668+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc118055060 con 0x7fc12806b7d0 2024-01-15T18:00:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.860+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc12806bdb0 con 0x7fc12806b7d0 2024-01-15T18:00:31.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.862+0000 7fc11f400640 1 -- 172.21.15.2:0/2523120658 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+12967 (secure 0 0 0) 0x7fc118058680 con 0x7fc12806b7d0 2024-01-15T18:00:31.865 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:31.865 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":39,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:25.305089+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6834/2859107199":"2024-01-16T18:00:22.271924+0000","172.21.15.2:6832/2859107199":"2024-01-16T18:00:22.271924+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:31.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc10006b500 msgr2=0x7fc10006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc10006b500 0x7fc10006d9c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc1280b7150 tx=0x7fc110007b50 comp rx=0 tx=0).stop 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 msgr2=0x7fc1280b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 0x7fc1280b1660 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fc11800d800 tx=0x7fc11800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 shutdown_connections 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc10006b500 0x7fc10006d9c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1280cc170 0x7fc1280c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc12806bfd0 0x7fc1281548e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 --2- 172.21.15.2:0/2523120658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc12806b7d0 0x7fc1280b1660 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 >> 172.21.15.2:0/2523120658 conn(0x7fc1280d0ec0 msgr2=0x7fc1280cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 shutdown_connections 2024-01-15T18:00:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:31.865+0000 7fc12cc00640 1 -- 172.21.15.2:0/2523120658 wait complete. 2024-01-15T18:00:31.887 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:00:31.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:00:32.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.104+0000 7f809a200640 1 Processor -- start 2024-01-15T18:00:32.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.104+0000 7f809a200640 1 -- start start 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.105+0000 7f809a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f809415c450 0x7f809406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.105+0000 7f809a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f809406ce30 0x7f8094070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.105+0000 7f809a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80940ce230 con 0x7f8094144c00 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.105+0000 7f809a200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809406d2c0 con 0x7f809415c450 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.105+0000 7f809a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80940ce510 con 0x7f809406ce30 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f809415c450 0x7f809406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f809415c450 0x7f809406c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35664/0 (socket says 172.21.15.2:35664) 2024-01-15T18:00:32.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 -- 172.21.15.2:0/88843731 learned_addr learned my addr 172.21.15.2:0/88843731 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8093e00640 1 --2- 172.21.15.2:0/88843731 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f809406ce30 0x7f8094070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 -- 172.21.15.2:0/88843731 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f809406ce30 msgr2=0x7f8094070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 --2- 172.21.15.2:0/88843731 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f809406ce30 0x7f8094070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 -- 172.21.15.2:0/88843731 >> v1:172.21.15.137:6789/0 conn(0x7f8094144c00 legacy=0x7f80940cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.106+0000 7f8092a00640 1 -- 172.21.15.2:0/88843731 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80940ce7f0 con 0x7f809415c450 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.107+0000 7f8092a00640 1 --2- 172.21.15.2:0/88843731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f809406c8f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f807c009870 tx=0x7f807c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf16b2ae7a05ac5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:32.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f8092000640 1 -- 172.21.15.2:0/88843731 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c031640 con 0x7f809415c450 2024-01-15T18:00:32.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f8092000640 1 -- 172.21.15.2:0/88843731 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807c0317a0 con 0x7f809415c450 2024-01-15T18:00:32.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f8092000640 1 -- 172.21.15.2:0/88843731 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807c031b30 con 0x7f809415c450 2024-01-15T18:00:32.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 -- 172.21.15.2:0/88843731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 msgr2=0x7f809406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 --2- 172.21.15.2:0/88843731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f809406c8f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f807c009870 tx=0x7f807c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 -- 172.21.15.2:0/88843731 shutdown_connections 2024-01-15T18:00:32.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 --2- 172.21.15.2:0/88843731 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f809406ce30 0x7f8094070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 --2- 172.21.15.2:0/88843731 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f809406c8f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 -- 172.21.15.2:0/88843731 >> 172.21.15.2:0/88843731 conn(0x7f80940b45f0 msgr2=0x7f80940b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:32.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.108+0000 7f809a200640 1 -- 172.21.15.2:0/88843731 shutdown_connections 2024-01-15T18:00:32.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.109+0000 7f809a200640 1 -- 172.21.15.2:0/88843731 wait complete. 2024-01-15T18:00:32.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.110+0000 7f809a200640 1 Processor -- start 2024-01-15T18:00:32.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.110+0000 7f809a200640 1 -- start start 2024-01-15T18:00:32.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.111+0000 7f809a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f809406ce30 0x7f80940b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.111+0000 7f809a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8094144c00 0x7f809414a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.111+0000 7f809a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f8094143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.111+0000 7f809a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8094141580 con 0x7f809415c450 2024-01-15T18:00:32.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.111+0000 7f809a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80940b0720 con 0x7f809406ce30 2024-01-15T18:00:32.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.111+0000 7f809a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80940b0a00 con 0x7f8094144c00 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f8094143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f8094143290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35674/0 (socket says 172.21.15.2:35674) 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 -- 172.21.15.2:0/877868457 learned_addr learned my addr 172.21.15.2:0/877868457 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8092a00640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8094144c00 0x7f809414a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093400640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f809406ce30 0x7f80940b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 -- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8094144c00 msgr2=0x7f809414a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8094144c00 0x7f809414a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 -- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f809406ce30 msgr2=0x7f80940b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f809406ce30 0x7f80940b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.112+0000 7f8093e00640 1 -- 172.21.15.2:0/877868457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c0094d0 con 0x7f809415c450 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.113+0000 7f8093e00640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f8094143290 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f808c007e40 tx=0x7f808c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.113+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c014620 con 0x7f809415c450 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.113+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c00ee90 con 0x7f809415c450 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.113+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80940b0ce0 con 0x7f809415c450 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.113+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c0149e0 con 0x7f809415c450 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.113+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80940b12a0 con 0x7f809415c450 2024-01-15T18:00:32.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.114+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80940686f0 con 0x7f809415c450 2024-01-15T18:00:32.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.115+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f808c014b40 con 0x7f809415c450 2024-01-15T18:00:32.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.117+0000 7f808b400640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f806c06b500 0x7f806c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.117+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f808c08a700 con 0x7f809415c450 2024-01-15T18:00:32.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.118+0000 7f8093400640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f806c06b500 0x7f806c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.119+0000 7f8093400640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f806c06b500 0x7f806c06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8084004ea0 tx=0x7f8084009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:32.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.121+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f808c055520 con 0x7f809415c450 2024-01-15T18:00:32.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.313+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8094071fb0 con 0x7f809415c450 2024-01-15T18:00:32.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.314+0000 7f808b400640 1 -- 172.21.15.2:0/877868457 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f808c058b40 con 0x7f809415c450 2024-01-15T18:00:32.317 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f806c06b500 msgr2=0x7f806c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f806c06b500 0x7f806c06d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f8084004ea0 tx=0x7f8084009040 comp rx=0 tx=0).stop 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 msgr2=0x7f8094143290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f8094143290 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f808c007e40 tx=0x7f808c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 shutdown_connections 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f806c06b500 0x7f806c06d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f809415c450 0x7f8094143290 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8094144c00 0x7f809414a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 --2- 172.21.15.2:0/877868457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f809406ce30 0x7f80940b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.317+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 >> 172.21.15.2:0/877868457 conn(0x7f80940b45f0 msgr2=0x7f809415b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.318+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 shutdown_connections 2024-01-15T18:00:32.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.318+0000 7f809a200640 1 -- 172.21.15.2:0/877868457 wait complete. 2024-01-15T18:00:32.339 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-15T18:00:32.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.560+0000 7fc4c3200640 1 Processor -- start 2024-01-15T18:00:32.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.561+0000 7fc4c3200640 1 -- start start 2024-01-15T18:00:32.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.562+0000 7fc4c3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 0x7fc4c415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.562+0000 7fc4c3200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc4c406cad0 0x7fc4c406cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.562+0000 7fc4c3200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c40716d0 con 0x7fc4c4141880 2024-01-15T18:00:32.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.562+0000 7fc4c3200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c4071f20 con 0x7fc4c406cad0 2024-01-15T18:00:32.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.562+0000 7fc4c3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c406c6b0 con 0x7fc4c415ec60 2024-01-15T18:00:32.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c1e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc4c4141880 0x7fc4c4070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56338/0 (socket says 172.21.15.2:56338) 2024-01-15T18:00:32.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c1e00640 1 -- 172.21.15.2:0/1301802781 learned_addr learned my addr 172.21.15.2:0/1301802781 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4250572999 0 0) 0x7fc4c40716d0 con 0x7fc4c4141880 2024-01-15T18:00:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a4003520 con 0x7fc4c4141880 2024-01-15T18:00:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c1400640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 0x7fc4c415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c2800640 1 --2- 172.21.15.2:0/1301802781 >> v2:172.21.15.137:3300/0 conn(0x7fc4c406cad0 0x7fc4c406cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.563+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 882515321 0 0) 0x7fc4a4003520 con 0x7fc4c4141880 2024-01-15T18:00:32.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 msgr2=0x7fc4c415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 0x7fc4c415f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 >> v2:172.21.15.137:3300/0 conn(0x7fc4c406cad0 msgr2=0x7fc4c406cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 --2- 172.21.15.2:0/1301802781 >> v2:172.21.15.137:3300/0 conn(0x7fc4c406cad0 0x7fc4c406cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4071960 con 0x7fc4c4141880 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc4b4001670 con 0x7fc4c4141880 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 >> v1:172.21.15.2:6789/0 conn(0x7fc4c4141880 legacy=0x7fc4c4070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 0x7fc4a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c4071f20 con 0x7fc4a40047c0 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c406c6b0 con 0x7fc4a4003970 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a400bae0 con 0x7fc4c415ec60 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1400640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1e00640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1400640 1 -- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 msgr2=0x7fc4a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:32.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1400640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c415ec60 0x7fc4a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1400640 1 -- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4a4003970 msgr2=0x7fc4a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:32.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1400640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4a4003970 0x7fc4a4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.564+0000 7fc4c1400640 1 -- 172.21.15.2:0/1301802781 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4003520 con 0x7fc4a40047c0 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.565+0000 7fc4c1400640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac007330 tx=0x7fc4a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e0fa73ff3476d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac02ccb0 con 0x7fc4a40047c0 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac02ce10 con 0x7fc4a40047c0 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c0a00640 1 -- 172.21.15.2:0/1301802781 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac034660 con 0x7fc4a40047c0 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c3200640 1 -- 172.21.15.2:0/1301802781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 msgr2=0x7fc4a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c3200640 1 --2- 172.21.15.2:0/1301802781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4a4004c40 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac007330 tx=0x7fc4a40057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c2800640 1 -- 172.21.15.2:0/1301802781 reap_dead start 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c3200640 1 -- 172.21.15.2:0/1301802781 shutdown_connections 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c3200640 1 -- 172.21.15.2:0/1301802781 >> 172.21.15.2:0/1301802781 conn(0x7fc4c40b0840 msgr2=0x7fc4c40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.566+0000 7fc4c3200640 1 -- 172.21.15.2:0/1301802781 shutdown_connections 2024-01-15T18:00:32.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.567+0000 7fc4c3200640 1 -- 172.21.15.2:0/1301802781 wait complete. 2024-01-15T18:00:32.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.568+0000 7fc4c3200640 1 Processor -- start 2024-01-15T18:00:32.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.568+0000 7fc4c3200640 1 -- start start 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.569+0000 7fc4c3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4c413d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.569+0000 7fc4c3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c413dad0 0x7fc4c40cf250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.569+0000 7fc4c3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4c40cf790 0x7fc4c413f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.569+0000 7fc4c3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40ad590 con 0x7fc4a40047c0 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.569+0000 7fc4c3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40ac910 con 0x7fc4c40cf790 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.569+0000 7fc4c3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40acbf0 con 0x7fc4c413dad0 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4c413d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4c413d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35702/0 (socket says 172.21.15.2:35702) 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 -- 172.21.15.2:0/198096156 learned_addr learned my addr 172.21.15.2:0/198096156 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1400640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c413dad0 0x7fc4c40cf250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 -- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c413dad0 msgr2=0x7fc4c40cf250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c413dad0 0x7fc4c40cf250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 -- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4c40cf790 msgr2=0x7fc4c413f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:32.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4c40cf790 0x7fc4c413f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:32.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 -- 172.21.15.2:0/198096156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c40aced0 con 0x7fc4a40047c0 2024-01-15T18:00:32.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.570+0000 7fc4c1e00640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4c413d250 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400d8e0 tx=0x7fc4b400dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:32.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.571+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40074e0 con 0x7fc4a40047c0 2024-01-15T18:00:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.571+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4007640 con 0x7fc4a40047c0 2024-01-15T18:00:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.571+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4004a20 con 0x7fc4a40047c0 2024-01-15T18:00:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.571+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c40ad1b0 con 0x7fc4a40047c0 2024-01-15T18:00:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.571+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c413f6b0 con 0x7fc4a40047c0 2024-01-15T18:00:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.572+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c4074770 con 0x7fc4a40047c0 2024-01-15T18:00:32.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.573+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc4b4004b80 con 0x7fc4a40047c0 2024-01-15T18:00:32.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.574+0000 7fc4ba000640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc49c06b500 0x7fc49c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:32.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.574+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc4b4089900 con 0x7fc4a40047c0 2024-01-15T18:00:32.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.574+0000 7fc4c1400640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc49c06b500 0x7fc49c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:32.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.576+0000 7fc4c1400640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc49c06b500 0x7fc49c06d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac036000 tx=0x7fc4ac030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:32.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.579+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4b40546a0 con 0x7fc4a40047c0 2024-01-15T18:00:32.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:32.844+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7fc4c406e1c0 con 0x7fc4a40047c0 2024-01-15T18:00:33.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.409+0000 7fc4ba000640 1 -- 172.21.15.2:0/198096156 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v161) v1 ==== 156+0+0 (secure 0 0 0) 0x7fc4b4019020 con 0x7fc4a40047c0 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.412+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc49c06b500 msgr2=0x7fc49c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.412+0000 7fc4c3200640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc49c06b500 0x7fc49c06d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac036000 tx=0x7fc4ac030000 comp rx=0 tx=0).stop 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.412+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 msgr2=0x7fc4c413d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.412+0000 7fc4c3200640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4c413d250 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400d8e0 tx=0x7fc4b400dda0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 shutdown_connections 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc49c06b500 0x7fc49c06d9c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4c40cf790 0x7fc4c413f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4c413dad0 0x7fc4c40cf250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 --2- 172.21.15.2:0/198096156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4a40047c0 0x7fc4c413d250 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 >> 172.21.15.2:0/198096156 conn(0x7fc4c40b0840 msgr2=0x7fc4c40710f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 shutdown_connections 2024-01-15T18:00:33.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.413+0000 7fc4c3200640 1 -- 172.21.15.2:0/198096156 wait complete. 2024-01-15T18:00:33.438 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:00:33.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.662+0000 7fb2f8200640 1 Processor -- start 2024-01-15T18:00:33.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.662+0000 7fb2f8200640 1 -- start start 2024-01-15T18:00:33.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f8200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2f015ae80 0x7fb2f00acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f8200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb2f006c8f0 0x7fb2f006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f8200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00b2430 0x7fb2f00b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f8200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f015e840 con 0x7fb2f015ae80 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f8200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f006d290 con 0x7fb2f006c8f0 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f8200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f015dbc0 con 0x7fb2f00b2430 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f5800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2f015ae80 0x7fb2f00acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f5800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2f015ae80 0x7fb2f00acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58490/0 (socket says 172.21.15.2:58490) 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f5800640 1 -- 172.21.15.2:0/3651576004 learned_addr learned my addr 172.21.15.2:0/3651576004 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f4e00640 1 --2- 172.21.15.2:0/3651576004 >> v2:172.21.15.137:3300/0 conn(0x7fb2f006c8f0 0x7fb2f006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f5800640 1 -- 172.21.15.2:0/3651576004 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00b2430 msgr2=0x7fb2f00b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.665+0000 7fb2f6200640 1 --2- 172.21.15.2:0/3651576004 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00b2430 0x7fb2f00b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.666+0000 7fb2f5800640 1 --2- 172.21.15.2:0/3651576004 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00b2430 0x7fb2f00b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.666+0000 7fb2f5800640 1 -- 172.21.15.2:0/3651576004 >> v2:172.21.15.137:3300/0 conn(0x7fb2f006c8f0 msgr2=0x7fb2f006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:33.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.666+0000 7fb2f5800640 1 --2- 172.21.15.2:0/3651576004 >> v2:172.21.15.137:3300/0 conn(0x7fb2f006c8f0 0x7fb2f006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.666+0000 7fb2f5800640 1 -- 172.21.15.2:0/3651576004 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f015dea0 con 0x7fb2f015ae80 2024-01-15T18:00:33.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.667+0000 7fb2f5800640 1 --2- 172.21.15.2:0/3651576004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f00acd00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00b4130 tx=0x7fb2e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9b8bd56788827d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:33.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.667+0000 7fb2efe00640 1 -- 172.21.15.2:0/3651576004 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0031870 con 0x7fb2f015ae80 2024-01-15T18:00:33.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.667+0000 7fb2efe00640 1 -- 172.21.15.2:0/3651576004 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e00319d0 con 0x7fb2f015ae80 2024-01-15T18:00:33.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.667+0000 7fb2efe00640 1 -- 172.21.15.2:0/3651576004 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0031d60 con 0x7fb2f015ae80 2024-01-15T18:00:33.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 -- 172.21.15.2:0/3651576004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 msgr2=0x7fb2f00acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 --2- 172.21.15.2:0/3651576004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f00acd00 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb2f00b4130 tx=0x7fb2e002b910 comp rx=0 tx=0).stop 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 -- 172.21.15.2:0/3651576004 shutdown_connections 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 --2- 172.21.15.2:0/3651576004 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f00b2430 0x7fb2f00b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 --2- 172.21.15.2:0/3651576004 >> v2:172.21.15.137:3300/0 conn(0x7fb2f006c8f0 0x7fb2f006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 --2- 172.21.15.2:0/3651576004 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f00acd00 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 -- 172.21.15.2:0/3651576004 >> 172.21.15.2:0/3651576004 conn(0x7fb2f00b45f0 msgr2=0x7fb2f00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 -- 172.21.15.2:0/3651576004 shutdown_connections 2024-01-15T18:00:33.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.668+0000 7fb2f8200640 1 -- 172.21.15.2:0/3651576004 wait complete. 2024-01-15T18:00:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.670+0000 7fb2f8200640 1 Processor -- start 2024-01-15T18:00:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.670+0000 7fb2f8200640 1 -- start start 2024-01-15T18:00:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f8200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f006c8f0 0x7fb2f014b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f8200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f00b2430 0x7fb2f00b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f8200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f0153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f8200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f014e350 con 0x7fb2f015ae80 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f8200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f014d2b0 con 0x7fb2f00b2430 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f8200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f014d590 con 0x7fb2f006c8f0 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f6200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f0153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f6200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f0153e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58504/0 (socket says 172.21.15.2:58504) 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f6200640 1 -- 172.21.15.2:0/4246067738 learned_addr learned my addr 172.21.15.2:0/4246067738 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f4e00640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f00b2430 0x7fb2f00b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.671+0000 7fb2f5800640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f006c8f0 0x7fb2f014b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f6200640 1 -- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f006c8f0 msgr2=0x7fb2f014b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f6200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f006c8f0 0x7fb2f014b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f6200640 1 -- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f00b2430 msgr2=0x7fb2f00b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f6200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f00b2430 0x7fb2f00b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f6200640 1 -- 172.21.15.2:0/4246067738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00094d0 con 0x7fb2f015ae80 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f6200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f0153e70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800cd70 tx=0x7fb2e800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8003b90 con 0x7fb2f015ae80 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e8003cf0 con 0x7fb2f015ae80 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e8004240 con 0x7fb2f015ae80 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.672+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f014d870 con 0x7fb2f015ae80 2024-01-15T18:00:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.673+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f014de30 con 0x7fb2f015ae80 2024-01-15T18:00:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.674+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0066f30 con 0x7fb2f015ae80 2024-01-15T18:00:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.675+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb2e80043a0 con 0x7fb2f015ae80 2024-01-15T18:00:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.676+0000 7fb2ed600640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806b500 0x7fb2c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.676+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb2e8012070 con 0x7fb2f015ae80 2024-01-15T18:00:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.676+0000 7fb2f5800640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806b500 0x7fb2c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.677+0000 7fb2f5800640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806b500 0x7fb2c806d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0036000 tx=0x7fb2e003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:33.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.680+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2e80554f0 con 0x7fb2f015ae80 2024-01-15T18:00:33.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.874+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb2f00686f0 con 0x7fb2f015ae80 2024-01-15T18:00:33.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.881+0000 7fb2ed600640 1 -- 172.21.15.2:0/4246067738 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb2e8058b10 con 0x7fb2f015ae80 2024-01-15T18:00:33.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.884+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806b500 msgr2=0x7fb2c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.884+0000 7fb2f8200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806b500 0x7fb2c806d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0036000 tx=0x7fb2e003c040 comp rx=0 tx=0).stop 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 msgr2=0x7fb2f0153e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f0153e70 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fb2e800cd70 tx=0x7fb2e800a660 comp rx=0 tx=0).stop 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 shutdown_connections 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806b500 0x7fb2c806d9c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f015ae80 0x7fb2f0153e70 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f00b2430 0x7fb2f00b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 --2- 172.21.15.2:0/4246067738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f006c8f0 0x7fb2f014b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 >> 172.21.15.2:0/4246067738 conn(0x7fb2f00b45f0 msgr2=0x7fb2f0070f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:33.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 shutdown_connections 2024-01-15T18:00:33.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:33.885+0000 7fb2f8200640 1 -- 172.21.15.2:0/4246067738 wait complete. 2024-01-15T18:00:33.908 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:00:33.908 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:00:34.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:00:34.148 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:00:34.148 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:00:34.148 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:34.209 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:00:34.209 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:00:34.209 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:34.268 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:00:34.268 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:00:34.269 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:34.331 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:00:34.331 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:00:34.331 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:34.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:00:34.388 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:00:34.388 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:00:34.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.614+0000 7f9dfcc00640 1 Processor -- start 2024-01-15T18:00:34.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.614+0000 7f9dfcc00640 1 -- start start 2024-01-15T18:00:34.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.615+0000 7f9dfcc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9df815d340 0x7f9df80cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.615+0000 7f9dfcc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df80d3090 0x7f9df80d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.615+0000 7f9dfcc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9df80d1680 0x7f9df80d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.615+0000 7f9dfcc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df80d3a30 con 0x7f9df80d1680 2024-01-15T18:00:34.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.615+0000 7f9dfcc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df80b3de0 con 0x7f9df815d340 2024-01-15T18:00:34.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.615+0000 7f9dfcc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80b40c0 con 0x7f9df80d3090 2024-01-15T18:00:34.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9df80d1680 0x7f9df80d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9df80d1680 0x7f9df80d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58506/0 (socket says 172.21.15.2:58506) 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 -- 172.21.15.2:0/4258646833 learned_addr learned my addr 172.21.15.2:0/4258646833 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7400640 1 --2- 172.21.15.2:0/4258646833 >> v2:172.21.15.137:3300/0 conn(0x7f9df815d340 0x7f9df80cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df6a00640 1 --2- 172.21.15.2:0/4258646833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df80d3090 0x7f9df80d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 -- 172.21.15.2:0/4258646833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df80d3090 msgr2=0x7f9df80d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 --2- 172.21.15.2:0/4258646833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df80d3090 0x7f9df80d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 -- 172.21.15.2:0/4258646833 >> v2:172.21.15.137:3300/0 conn(0x7f9df815d340 msgr2=0x7f9df80cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 --2- 172.21.15.2:0/4258646833 >> v2:172.21.15.137:3300/0 conn(0x7f9df815d340 0x7f9df80cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.616+0000 7f9df7e00640 1 -- 172.21.15.2:0/4258646833 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80d0e80 con 0x7f9df80d1680 2024-01-15T18:00:34.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.618+0000 7f9df7e00640 1 --2- 172.21.15.2:0/4258646833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80d1b10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9df808cf70 tx=0x7f9df002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408d5e2c800f472d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:34.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.618+0000 7f9df6000640 1 -- 172.21.15.2:0/4258646833 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df002ca70 con 0x7f9df80d1680 2024-01-15T18:00:34.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.618+0000 7f9df6000640 1 -- 172.21.15.2:0/4258646833 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df002cbd0 con 0x7f9df80d1680 2024-01-15T18:00:34.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.618+0000 7f9df6000640 1 -- 172.21.15.2:0/4258646833 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df00343f0 con 0x7f9df80d1680 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 -- 172.21.15.2:0/4258646833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 msgr2=0x7f9df80d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/4258646833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80d1b10 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f9df808cf70 tx=0x7f9df002bed0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 -- 172.21.15.2:0/4258646833 shutdown_connections 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/4258646833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80d1b10 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/4258646833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df80d3090 0x7f9df80d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/4258646833 >> v2:172.21.15.137:3300/0 conn(0x7f9df815d340 0x7f9df80cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 -- 172.21.15.2:0/4258646833 >> 172.21.15.2:0/4258646833 conn(0x7f9df808caf0 msgr2=0x7f9df809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 -- 172.21.15.2:0/4258646833 shutdown_connections 2024-01-15T18:00:34.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.619+0000 7f9dfcc00640 1 -- 172.21.15.2:0/4258646833 wait complete. 2024-01-15T18:00:34.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.621+0000 7f9dfcc00640 1 Processor -- start 2024-01-15T18:00:34.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.621+0000 7f9dfcc00640 1 -- start start 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9dfcc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80cfbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9dfcc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9df80d3090 0x7f9df80cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9dfcc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df815d340 0x7f9df80ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9dfcc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df8149f60 con 0x7f9df80d1680 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9dfcc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df80c9b50 con 0x7f9df80d3090 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9dfcc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df81495c0 con 0x7f9df815d340 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9df7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80cfbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9df7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80cfbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58516/0 (socket says 172.21.15.2:58516) 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.622+0000 7f9df7400640 1 -- 172.21.15.2:0/1574517523 learned_addr learned my addr 172.21.15.2:0/1574517523 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df6a00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9df80d3090 0x7f9df80cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7e00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df815d340 0x7f9df80ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7400640 1 -- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df815d340 msgr2=0x7f9df80ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7400640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df815d340 0x7f9df80ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7400640 1 -- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9df80d3090 msgr2=0x7f9df80cc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:34.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7400640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9df80d3090 0x7f9df80cc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7400640 1 -- 172.21.15.2:0/1574517523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0009b30 con 0x7f9df80d1680 2024-01-15T18:00:34.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df6a00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9df80d3090 0x7f9df80cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:34.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9df7400640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80cfbd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9de800d830 tx=0x7f9de800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:34.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800b6c0 con 0x7f9df80d1680 2024-01-15T18:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de800b820 con 0x7f9df80d1680 2024-01-15T18:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de800bbb0 con 0x7f9df80d1680 2024-01-15T18:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.623+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df81498a0 con 0x7f9df80d1680 2024-01-15T18:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.624+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df81492a0 con 0x7f9df80d1680 2024-01-15T18:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.625+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df8074770 con 0x7f9df80d1680 2024-01-15T18:00:34.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.626+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9de801e460 con 0x7f9df80d1680 2024-01-15T18:00:34.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.627+0000 7f9def400640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9dd006b500 0x7f9dd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:34.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.627+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9de808a490 con 0x7f9df80d1680 2024-01-15T18:00:34.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.628+0000 7f9df6a00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9dd006b500 0x7f9dd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:34.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.629+0000 7f9df6a00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9dd006b500 0x7f9dd006d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006fd0 tx=0x7f9de0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:34.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.632+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9de8055280 con 0x7f9df80d1680 2024-01-15T18:00:34.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.863+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df80678e0 con 0x7f9df80d1680 2024-01-15T18:00:34.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.864+0000 7f9def400640 1 -- 172.21.15.2:0/1574517523 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3687 (secure 0 0 0) 0x7f9de80588a0 con 0x7f9df80d1680 2024-01-15T18:00:34.868 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:34.868 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4724,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/122233456","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6836","nonce":122233456},{"type":"v1","addr":"172.21.15.137:6837","nonce":122233456}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4730,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1272974394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":1272974394},{"type":"v1","addr":"172.21.15.2:6835","nonce":1272974394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4948,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6835/84168043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":84168043},{"type":"v1","addr":"172.21.15.137:6835","nonce":84168043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5601,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/645061371","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":645061371},{"type":"v1","addr":"172.21.15.2:6834","nonce":645061371}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-01-15T18:00:34.868 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 19 2024-01-15T18:00:34.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9dd006b500 msgr2=0x7f9dd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:34.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9dd006b500 0x7f9dd006d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9de0006fd0 tx=0x7f9de0008040 comp rx=0 tx=0).stop 2024-01-15T18:00:34.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 msgr2=0x7f9df80cfbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80cfbd0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9de800d830 tx=0x7f9de800dcf0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 shutdown_connections 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9dd006b500 0x7f9dd006d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9df815d340 0x7f9df80ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9df80d3090 0x7f9df80cc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 --2- 172.21.15.2:0/1574517523 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9df80d1680 0x7f9df80cfbd0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 >> 172.21.15.2:0/1574517523 conn(0x7f9df808caf0 msgr2=0x7f9df806b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 shutdown_connections 2024-01-15T18:00:34.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:34.867+0000 7f9dfcc00640 1 -- 172.21.15.2:0/1574517523 wait complete. 2024-01-15T18:00:34.890 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:00:34.890 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:00:34.890 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:00:34.890 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:00:34.890 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:00:34.890 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:00:34.890 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:00:34.890 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:00:34.891 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:00:34.891 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:00:34.891 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:00:34.891 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:00:34.891 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:00:34.891 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:00:34.891 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:00:34.891 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:00:34.908 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:00:34.908 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:00:34.909 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:00:34.909 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:00:34.912 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:00:34.912 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:00:34.924 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:00:34.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:00:34.956 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:00:34.956 DEBUG:teuthology.parallel:result is None 2024-01-15T18:00:34.963 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:00:34.963 DEBUG:teuthology.parallel:result is None 2024-01-15T18:00:34.970 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:00:34.971 DEBUG:teuthology.parallel:result is None 2024-01-15T18:00:34.977 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:00:34.977 DEBUG:teuthology.parallel:result is None 2024-01-15T18:00:34.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:00:35.112 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:00:35.118 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:00:35.131 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T18:00:35.131 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:00:35.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.160+0000 7f3353c00640 1 Processor -- start 2024-01-15T18:00:35.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.160+0000 7f3353c00640 1 -- start start 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.161+0000 7f3353c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c144b80 0x7f334c0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.161+0000 7f3353c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f334c06cfc0 0x7f334c15b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.161+0000 7f3353c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f334c0b35a0 con 0x7f334c0b30a0 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.161+0000 7f3353c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334c0b4220 con 0x7f334c06cfc0 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.161+0000 7f3353c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c144f60 con 0x7f334c144b80 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f334c06cfc0 0x7f334c15b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f334c06cfc0 0x7f334c15b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58546/0 (socket says 172.21.15.2:58546) 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 -- 172.21.15.2:0/473386575 learned_addr learned my addr 172.21.15.2:0/473386575 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351200640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c144b80 0x7f334c0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 -- 172.21.15.2:0/473386575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c144b80 msgr2=0x7f334c0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:35.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c144b80 0x7f334c0cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 -- 172.21.15.2:0/473386575 >> v1:172.21.15.137:6789/0 conn(0x7f334c0b30a0 legacy=0x7f334c06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351c00640 1 -- 172.21.15.2:0/473386575 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c0b3880 con 0x7f334c06cfc0 2024-01-15T18:00:35.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.162+0000 7f3351200640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c144b80 0x7f334c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:00:35.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.163+0000 7f3351c00640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c15b9e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f334c1450a0 tx=0x7f334402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b32060df5904e30 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:35.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.163+0000 7f334b400640 1 -- 172.21.15.2:0/473386575 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334402ca50 con 0x7f334c06cfc0 2024-01-15T18:00:35.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.163+0000 7f334b400640 1 -- 172.21.15.2:0/473386575 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334402cbb0 con 0x7f334c06cfc0 2024-01-15T18:00:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.163+0000 7f334b400640 1 -- 172.21.15.2:0/473386575 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33440343f0 con 0x7f334c06cfc0 2024-01-15T18:00:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 -- 172.21.15.2:0/473386575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 msgr2=0x7f334c15b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c15b9e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f334c1450a0 tx=0x7f334402beb0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 -- 172.21.15.2:0/473386575 shutdown_connections 2024-01-15T18:00:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c15b9e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 --2- 172.21.15.2:0/473386575 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c144b80 0x7f334c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 -- 172.21.15.2:0/473386575 >> 172.21.15.2:0/473386575 conn(0x7f334c0b46d0 msgr2=0x7f334c0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:35.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 -- 172.21.15.2:0/473386575 shutdown_connections 2024-01-15T18:00:35.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.164+0000 7f3353c00640 1 -- 172.21.15.2:0/473386575 wait complete. 2024-01-15T18:00:35.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.166+0000 7f3353c00640 1 Processor -- start 2024-01-15T18:00:35.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.166+0000 7f3353c00640 1 -- start start 2024-01-15T18:00:35.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3353c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c137bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3353c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c0b30a0 0x7f334c15ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3353c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334c144b80 0x7f334c13fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3353c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1380f0 con 0x7f334c06cfc0 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3353c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c15c5f0 con 0x7f334c144b80 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3353c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f334c15b9e0 con 0x7f334c0b30a0 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3351200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c137bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3351200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c137bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58558/0 (socket says 172.21.15.2:58558) 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.167+0000 7f3351200640 1 -- 172.21.15.2:0/3753053351 learned_addr learned my addr 172.21.15.2:0/3753053351 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334c144b80 0x7f334c13fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f334be00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c0b30a0 0x7f334c15ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351200640 1 -- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c0b30a0 msgr2=0x7f334c15ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351200640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c0b30a0 0x7f334c15ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351200640 1 -- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334c144b80 msgr2=0x7f334c13fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351200640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334c144b80 0x7f334c13fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351200640 1 -- 172.21.15.2:0/3753053351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009b30 con 0x7f334c06cfc0 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334c144b80 0x7f334c13fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f334be00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c0b30a0 0x7f334c15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:35.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3351200640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c137bb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f333c00d7d0 tx=0x7f333c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c004030 con 0x7f334c06cfc0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333c004190 con 0x7f334c06cfc0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333c004520 con 0x7f334c06cfc0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.168+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c15bcc0 con 0x7f334c06cfc0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.169+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c15c280 con 0x7f334c06cfc0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.170+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c0686f0 con 0x7f334c06cfc0 2024-01-15T18:00:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.171+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f333c00b6c0 con 0x7f334c06cfc0 2024-01-15T18:00:35.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.172+0000 7f3348c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f332406b530 0x7f332406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.172+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f333c089ed0 con 0x7f334c06cfc0 2024-01-15T18:00:35.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.173+0000 7f334be00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f332406b530 0x7f332406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.173+0000 7f334be00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f332406b530 0x7f332406d9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f334c153150 tx=0x7f3334009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:35.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.177+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f333c054c20 con 0x7f334c06cfc0 2024-01-15T18:00:35.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.384+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f334c0cb980 con 0x7f334c06cfc0 2024-01-15T18:00:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.443+0000 7f3348c00640 1 -- 172.21.15.2:0/3753053351 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v41) v1 ==== 97+0+0 (secure 0 0 0) 0x7f333c058240 con 0x7f334c06cfc0 2024-01-15T18:00:35.446 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T18:00:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f332406b530 msgr2=0x7f332406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f332406b530 0x7f332406d9f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f334c153150 tx=0x7f3334009040 comp rx=0 tx=0).stop 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 msgr2=0x7f334c137bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c137bb0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f333c00d7d0 tx=0x7f333c00dc90 comp rx=0 tx=0).stop 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 shutdown_connections 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f332406b530 0x7f332406d9f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334c144b80 0x7f334c13fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334c0b30a0 0x7f334c15ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 --2- 172.21.15.2:0/3753053351 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334c06cfc0 0x7f334c137bb0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 >> 172.21.15.2:0/3753053351 conn(0x7f334c0b46d0 msgr2=0x7f334c0cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 shutdown_connections 2024-01-15T18:00:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.447+0000 7f3353c00640 1 -- 172.21.15.2:0/3753053351 wait complete. 2024-01-15T18:00:35.469 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:00:35.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.656+0000 7fc2c2c00640 1 Processor -- start 2024-01-15T18:00:35.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.657+0000 7fc2c2c00640 1 -- start start 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.659+0000 7fc2c2c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2bc15c0e0 0x7fc2bc0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.659+0000 7fc2c2c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc2bc15b7d0 0x7fc2bc06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.659+0000 7fc2c2c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2bc15cb00 con 0x7fc2bc15c4c0 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.659+0000 7fc2c2c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0ccec0 con 0x7fc2bc15b7d0 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.659+0000 7fc2c2c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2bc0cd1a0 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.659+0000 7fc2b3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc2bc15b7d0 0x7fc2bc06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2b3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc2bc15b7d0 0x7fc2bc06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58572/0 (socket says 172.21.15.2:58572) 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2b3200640 1 -- 172.21.15.2:0/4012040492 learned_addr learned my addr 172.21.15.2:0/4012040492 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:35.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2bbe00640 1 --2- 172.21.15.2:0/4012040492 >> v2:172.21.15.137:3300/0 conn(0x7fc2bc15c0e0 0x7fc2bc0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2b3200640 1 -- 172.21.15.2:0/4012040492 >> v2:172.21.15.137:3300/0 conn(0x7fc2bc15c0e0 msgr2=0x7fc2bc0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:35.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2b3200640 1 --2- 172.21.15.2:0/4012040492 >> v2:172.21.15.137:3300/0 conn(0x7fc2bc15c0e0 0x7fc2bc0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2b3200640 1 -- 172.21.15.2:0/4012040492 >> v1:172.21.15.137:6789/0 conn(0x7fc2bc15c4c0 legacy=0x7fc2bc15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:35.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.660+0000 7fc2b3200640 1 -- 172.21.15.2:0/4012040492 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc0cd480 con 0x7fc2bc15b7d0 2024-01-15T18:00:35.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.661+0000 7fc2b3200640 1 --2- 172.21.15.2:0/4012040492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15b7d0 0x7fc2bc06c470 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0725f0 tx=0x7fc2a402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61473994110f0ccf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:35.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.661+0000 7fc2bb400640 1 -- 172.21.15.2:0/4012040492 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4031890 con 0x7fc2bc15b7d0 2024-01-15T18:00:35.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.661+0000 7fc2bb400640 1 -- 172.21.15.2:0/4012040492 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a40319f0 con 0x7fc2bc15b7d0 2024-01-15T18:00:35.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2bb400640 1 -- 172.21.15.2:0/4012040492 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4031d80 con 0x7fc2bc15b7d0 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 -- 172.21.15.2:0/4012040492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15b7d0 msgr2=0x7fc2bc06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/4012040492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15b7d0 0x7fc2bc06c470 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc0725f0 tx=0x7fc2a402b930 comp rx=0 tx=0).stop 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 -- 172.21.15.2:0/4012040492 shutdown_connections 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/4012040492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15b7d0 0x7fc2bc06c470 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/4012040492 >> v2:172.21.15.137:3300/0 conn(0x7fc2bc15c0e0 0x7fc2bc0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 -- 172.21.15.2:0/4012040492 >> 172.21.15.2:0/4012040492 conn(0x7fc2bc06da20 msgr2=0x7fc2bc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 -- 172.21.15.2:0/4012040492 shutdown_connections 2024-01-15T18:00:35.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.662+0000 7fc2c2c00640 1 -- 172.21.15.2:0/4012040492 wait complete. 2024-01-15T18:00:35.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.664+0000 7fc2c2c00640 1 Processor -- start 2024-01-15T18:00:35.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.664+0000 7fc2c2c00640 1 -- start start 2024-01-15T18:00:35.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2c2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc15b7d0 0x7fc2bc147540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2c2c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 0x7fc2bc0713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2c2c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc15c4c0 0x7fc2bc071910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2c2c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc148490 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2c2c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b2440 con 0x7fc2bc15c4c0 2024-01-15T18:00:35.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2c2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2bc0b2720 con 0x7fc2bc15b7d0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2b3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 0x7fc2bc0713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2b3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 0x7fc2bc0713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58574/0 (socket says 172.21.15.2:58574) 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.665+0000 7fc2b3200640 1 -- 172.21.15.2:0/680949132 learned_addr learned my addr 172.21.15.2:0/680949132 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b3200640 1 -- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc15b7d0 msgr2=0x7fc2bc147540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b3200640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc15b7d0 0x7fc2bc147540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b3200640 1 -- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc15c4c0 msgr2=0x7fc2bc071910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b3200640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc15c4c0 0x7fc2bc071910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b3200640 1 -- 172.21.15.2:0/680949132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a40094d0 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b3200640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 0x7fc2bc0713d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4000c00 tx=0x7fc2a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a403ac60 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a403adc0 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4044610 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2bc0b2a00 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.666+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2bc147a80 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.667+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2bc148600 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.668+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc2a404d450 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.669+0000 7fc2b8c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29406b500 0x7fc29406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:35.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.669+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fc2a40ac500 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.670+0000 7fc2bbe00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29406b500 0x7fc29406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:35.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.671+0000 7fc2bbe00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29406b500 0x7fc29406d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac006fd0 tx=0x7fc2ac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:35.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.673+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2a4077190 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.938+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc2bc0686f0 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.939+0000 7fc2b8c00640 1 -- 172.21.15.2:0/680949132 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc2a407a7b0 con 0x7fc2bc15c0e0 2024-01-15T18:00:35.942 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.943 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.944 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:00:35.945 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T18:00:35.946 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.947 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.948 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.949 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.950 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow r", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:00:35.951 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.952 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.953 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:35.954 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T18:00:35.955 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29406b500 msgr2=0x7fc29406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29406b500 0x7fc29406d9c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac006fd0 tx=0x7fc2ac008040 comp rx=0 tx=0).stop 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 msgr2=0x7fc2bc0713d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 0x7fc2bc0713d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4000c00 tx=0x7fc2a40046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 shutdown_connections 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29406b500 0x7fc29406d9c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2bc15c4c0 0x7fc2bc071910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2bc15c0e0 0x7fc2bc0713d0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 --2- 172.21.15.2:0/680949132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2bc15b7d0 0x7fc2bc147540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:35.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 >> 172.21.15.2:0/680949132 conn(0x7fc2bc06da20 msgr2=0x7fc2bc152a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:35.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 shutdown_connections 2024-01-15T18:00:35.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:35.942+0000 7fc2c2c00640 1 -- 172.21.15.2:0/680949132 wait complete. 2024-01-15T18:00:35.964 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:36.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.182+0000 7f5d8e600640 1 Processor -- start 2024-01-15T18:00:36.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.182+0000 7f5d8e600640 1 -- start start 2024-01-15T18:00:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 0x7f5d8806d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8e600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5d8806da60 0x7f5d88161c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8e600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d880b1c70 con 0x7f5d881621d0 2024-01-15T18:00:36.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8e600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d880b28f0 con 0x7f5d8806da60 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b1f50 con 0x7f5d8814af80 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8dc00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5d881621d0 0x7f5d880700d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38608/0 (socket says 172.21.15.2:38608) 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.184+0000 7f5d8dc00640 1 -- 172.21.15.2:0/3582662459 learned_addr learned my addr 172.21.15.2:0/3582662459 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d8d200640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 0x7f5d8806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87e00640 1 --2- 172.21.15.2:0/3582662459 >> v2:172.21.15.137:3300/0 conn(0x7f5d8806da60 0x7f5d88161c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830372383 0 0) 0x7f5d880b1c70 con 0x7f5d881621d0 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d68003520 con 0x7f5d881621d0 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 299833426 0 0) 0x7f5d68003520 con 0x7f5d881621d0 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 msgr2=0x7f5d8806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 0x7f5d8806d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 >> v2:172.21.15.137:3300/0 conn(0x7f5d8806da60 msgr2=0x7f5d88161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 --2- 172.21.15.2:0/3582662459 >> v2:172.21.15.137:3300/0 conn(0x7f5d8806da60 0x7f5d88161c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880b2230 con 0x7f5d881621d0 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5d800019f0 con 0x7f5d881621d0 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 >> v1:172.21.15.2:6789/0 conn(0x7f5d881621d0 legacy=0x7f5d880700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d68003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 0x7f5d680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d68004d00 0x7f5d68005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b28f0 con 0x7f5d68003970 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880b1f50 con 0x7f5d68004d00 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.185+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6800b980 con 0x7f5d8814af80 2024-01-15T18:00:36.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8d200640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d68003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d87e00640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 0x7f5d680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8dc00640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d68004d00 0x7f5d68005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8d200640 1 -- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 msgr2=0x7f5d680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8d200640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d8814af80 0x7f5d680047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8d200640 1 -- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d68004d00 msgr2=0x7f5d68005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8d200640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d68004d00 0x7f5d68005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8d200640 1 -- 172.21.15.2:0/3582662459 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68003520 con 0x7f5d68003970 2024-01-15T18:00:36.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.186+0000 7f5d8dc00640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d68004d00 0x7f5d68005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.187+0000 7f5d8d200640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d68003d90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009ac0 tx=0x7f5d680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70db9e4201794116 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7802c6b0 con 0x7f5d68003970 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7802c810 con 0x7f5d68003970 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d87400640 1 -- 172.21.15.2:0/3582662459 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7802cba0 con 0x7f5d68003970 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d8e600640 1 -- 172.21.15.2:0/3582662459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 msgr2=0x7f5d68003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d8e600640 1 --2- 172.21.15.2:0/3582662459 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d68003d90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009ac0 tx=0x7f5d680057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d8dc00640 1 -- 172.21.15.2:0/3582662459 reap_dead start 2024-01-15T18:00:36.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d8e600640 1 -- 172.21.15.2:0/3582662459 shutdown_connections 2024-01-15T18:00:36.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d8e600640 1 -- 172.21.15.2:0/3582662459 >> 172.21.15.2:0/3582662459 conn(0x7f5d880b0840 msgr2=0x7f5d880b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:36.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.188+0000 7f5d8e600640 1 -- 172.21.15.2:0/3582662459 shutdown_connections 2024-01-15T18:00:36.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.189+0000 7f5d8e600640 1 -- 172.21.15.2:0/3582662459 wait complete. 2024-01-15T18:00:36.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.190+0000 7f5d8e600640 1 Processor -- start 2024-01-15T18:00:36.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 -- start start 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d88142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d881f69d0 0x7f5d8813fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d881403c0 0x7f5d88140840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d88146310 con 0x7f5d68003970 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d881428f0 con 0x7f5d881f69d0 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.191+0000 7f5d8e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d881f7280 con 0x7f5d881403c0 2024-01-15T18:00:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d88142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d88142200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58596/0 (socket says 172.21.15.2:58596) 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 -- 172.21.15.2:0/2919520702 learned_addr learned my addr 172.21.15.2:0/2919520702 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d87e00640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d881f69d0 0x7f5d8813fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8dc00640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d881403c0 0x7f5d88140840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 -- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d881403c0 msgr2=0x7f5d88140840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d881403c0 0x7f5d88140840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 -- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d881f69d0 msgr2=0x7f5d8813fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d881f69d0 0x7f5d8813fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.192+0000 7f5d8d200640 1 -- 172.21.15.2:0/2919520702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d881f7560 con 0x7f5d68003970 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.193+0000 7f5d8d200640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d88142200 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5d78037040 tx=0x7f5d78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.193+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78003810 con 0x7f5d68003970 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.193+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78003970 con 0x7f5d68003970 2024-01-15T18:00:36.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.193+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78003ce0 con 0x7f5d68003970 2024-01-15T18:00:36.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.193+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d881f7840 con 0x7f5d68003970 2024-01-15T18:00:36.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.193+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d881f8040 con 0x7f5d68003970 2024-01-15T18:00:36.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.195+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d88074770 con 0x7f5d68003970 2024-01-15T18:00:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.195+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5d78034440 con 0x7f5d68003970 2024-01-15T18:00:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.197+0000 7f5d84c00640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6006b530 0x7f5d6006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.197+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f5d780abc80 con 0x7f5d68003970 2024-01-15T18:00:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.197+0000 7f5d87e00640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6006b530 0x7f5d6006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.198+0000 7f5d87e00640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6006b530 0x7f5d6006d9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5d88145ce0 tx=0x7f5d70008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:36.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.201+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d78076bf0 con 0x7f5d68003970 2024-01-15T18:00:36.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.397+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d8806ea40 con 0x7f5d68003970 2024-01-15T18:00:36.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.398+0000 7f5d84c00640 1 -- 172.21.15.2:0/2919520702 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+12840 (secure 0 0 0) 0x7f5d7804c090 con 0x7f5d68003970 2024-01-15T18:00:36.401 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:36.401 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":43,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:35.458655+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6006b530 msgr2=0x7f5d6006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6006b530 0x7f5d6006d9f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5d88145ce0 tx=0x7f5d70008040 comp rx=0 tx=0).stop 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 msgr2=0x7f5d88142200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d88142200 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5d78037040 tx=0x7f5d78004730 comp rx=0 tx=0).stop 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 shutdown_connections 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6006b530 0x7f5d6006d9f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d881403c0 0x7f5d88140840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d881f69d0 0x7f5d8813fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 --2- 172.21.15.2:0/2919520702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d68003970 0x7f5d88142200 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 >> 172.21.15.2:0/2919520702 conn(0x7f5d880b0840 msgr2=0x7f5d880b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 shutdown_connections 2024-01-15T18:00:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.400+0000 7f5d8e600640 1 -- 172.21.15.2:0/2919520702 wait complete. 2024-01-15T18:00:36.424 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:00:36.424 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:00:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.632+0000 7f13dd600640 1 Processor -- start 2024-01-15T18:00:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.632+0000 7f13dd600640 1 -- start start 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.634+0000 7f13dd600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f13d8155960 0x7f13d80cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.634+0000 7f13dd600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d815c440 con 0x7f13d80cfa50 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.634+0000 7f13dd600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d815c720 con 0x7f13d8071410 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.634+0000 7f13dd600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d80ced10 con 0x7f13d8155960 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.635+0000 7f13d6000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f13d80cfa50 0x7f13d815b900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38610/0 (socket says 172.21.15.2:38610) 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.635+0000 7f13d6000640 1 -- 172.21.15.2:0/630946807 learned_addr learned my addr 172.21.15.2:0/630946807 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.635+0000 7f13d6a00640 1 --2- 172.21.15.2:0/630946807 >> v2:172.21.15.137:3300/0 conn(0x7f13d8155960 0x7f13d80cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.635+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106042737 0 0) 0x7f13d815c440 con 0x7f13d80cfa50 2024-01-15T18:00:36.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.635+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b8003520 con 0x7f13d80cfa50 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2364764622 0 0) 0x7f13b8003520 con 0x7f13d80cfa50 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 >> v2:172.21.15.137:3300/0 conn(0x7f13d8155960 msgr2=0x7f13d80cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 --2- 172.21.15.2:0/630946807 >> v2:172.21.15.137:3300/0 conn(0x7f13d8155960 0x7f13d80cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 >> v1:172.21.15.137:6789/0 conn(0x7f13d8071410 legacy=0x7f13d806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d80ceff0 con 0x7f13d80cfa50 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f13c0001670 con 0x7f13d80cfa50 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 >> v1:172.21.15.2:6789/0 conn(0x7f13d80cfa50 legacy=0x7f13d815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b80047c0 0x7f13b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13b8005180 0x7f13b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d80ced10 con 0x7f13b8003970 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b800c020 con 0x7f13b80047c0 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b800c1e0 con 0x7f13b8005180 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d6a00640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.636+0000 7f13d7400640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13b8005180 0x7f13b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6a00640 1 -- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13b8005180 msgr2=0x7f13b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6000640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b80047c0 0x7f13b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6a00640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13b8005180 0x7f13b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6a00640 1 -- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b80047c0 msgr2=0x7f13b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6a00640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b80047c0 0x7f13b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6a00640 1 -- 172.21.15.2:0/630946807 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b8003520 con 0x7f13b8003970 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d7400640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13b8005180 0x7f13b800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d6000640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13b80047c0 0x7f13b8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.637+0000 7f13d7400640 1 -- 172.21.15.2:0/630946807 reap_dead start 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.638+0000 7f13d6a00640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13b8003d90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009ac0 tx=0x7f13b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee423e7e452450f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.638+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c802c6b0 con 0x7f13b8003970 2024-01-15T18:00:36.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.638+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c802c810 con 0x7f13b8003970 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.638+0000 7f13d5600640 1 -- 172.21.15.2:0/630946807 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c802cba0 con 0x7f13b8003970 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 -- 172.21.15.2:0/630946807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 msgr2=0x7f13b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13b8003d90 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f13c8009ac0 tx=0x7f13b80057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 -- 172.21.15.2:0/630946807 shutdown_connections 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 --2- 172.21.15.2:0/630946807 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13b8003d90 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 -- 172.21.15.2:0/630946807 >> 172.21.15.2:0/630946807 conn(0x7f13d80b1c70 msgr2=0x7f13d80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 -- 172.21.15.2:0/630946807 shutdown_connections 2024-01-15T18:00:36.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.639+0000 7f13dd600640 1 -- 172.21.15.2:0/630946807 wait complete. 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.641+0000 7f13dd600640 1 Processor -- start 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.641+0000 7f13dd600640 1 -- start start 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13dd600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13d8154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13dd600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d8135460 0x7f13d8140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13dd600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d8140a60 0x7f13d8140ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13dd600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d81f1ef0 con 0x7f13b8003970 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13dd600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d8155260 con 0x7f13d8140a60 2024-01-15T18:00:36.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13dd600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d81f1270 con 0x7f13d8135460 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13d6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13d8154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13d6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13d8154d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58614/0 (socket says 172.21.15.2:58614) 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.642+0000 7f13d6a00640 1 -- 172.21.15.2:0/4037165766 learned_addr learned my addr 172.21.15.2:0/4037165766 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13d6a00640 1 -- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d8135460 msgr2=0x7f13d8140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13d6a00640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d8135460 0x7f13d8140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13d6a00640 1 -- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d8140a60 msgr2=0x7f13d8140ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13d6a00640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d8140a60 0x7f13d8140ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13d6a00640 1 -- 172.21.15.2:0/4037165766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d81f1550 con 0x7f13b8003970 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13d6a00640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13d8154d20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f13c8037040 tx=0x7f13c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.643+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8003810 con 0x7f13b8003970 2024-01-15T18:00:36.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.644+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c8003970 con 0x7f13b8003970 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.644+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c8003d00 con 0x7f13b8003970 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.644+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d81f1830 con 0x7f13b8003970 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.644+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d81f0f60 con 0x7f13b8003970 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.645+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d8067450 con 0x7f13b8003970 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.646+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f13c8034440 con 0x7f13b8003970 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.647+0000 7f13cea00640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13b006b530 0x7f13b006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:36.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.647+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f13c80abe00 con 0x7f13b8003970 2024-01-15T18:00:36.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.648+0000 7f13d6000640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13b006b530 0x7f13b006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:36.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.649+0000 7f13d6000640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13b006b530 0x7f13b006d9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f13c0001200 tx=0x7f13c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:36.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.651+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13c8076e00 con 0x7f13b8003970 2024-01-15T18:00:36.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.844+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13d80686f0 con 0x7f13b8003970 2024-01-15T18:00:36.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.845+0000 7f13cea00640 1 -- 172.21.15.2:0/4037165766 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7f13c807a420 con 0x7f13b8003970 2024-01-15T18:00:36.849 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:00:36.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13b006b530 msgr2=0x7f13b006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13b006b530 0x7f13b006d9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f13c0001200 tx=0x7f13c0009040 comp rx=0 tx=0).stop 2024-01-15T18:00:36.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 msgr2=0x7f13d8154d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:36.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13d8154d20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f13c8037040 tx=0x7f13c8004730 comp rx=0 tx=0).stop 2024-01-15T18:00:36.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 shutdown_connections 2024-01-15T18:00:36.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13b006b530 0x7f13b006d9f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13d8140a60 0x7f13d8140ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13d8135460 0x7f13d8140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 --2- 172.21.15.2:0/4037165766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13b8003970 0x7f13d8154d20 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:36.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 >> 172.21.15.2:0/4037165766 conn(0x7f13d80b1c70 msgr2=0x7f13d806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:36.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 shutdown_connections 2024-01-15T18:00:36.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:36.848+0000 7f13dd600640 1 -- 172.21.15.2:0/4037165766 wait complete. 2024-01-15T18:00:36.871 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:00:37.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.091+0000 7fc1b6e00640 1 Processor -- start 2024-01-15T18:00:37.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.091+0000 7fc1b6e00640 1 -- start start 2024-01-15T18:00:37.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.093+0000 7fc1b6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b015db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.093+0000 7fc1b6e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 0x7fc1b0166d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.093+0000 7fc1b6e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1b00b2e70 con 0x7fc1b015d750 2024-01-15T18:00:37.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.093+0000 7fc1b6e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1b006c430 con 0x7fc1b015e070 2024-01-15T18:00:37.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.093+0000 7fc1b6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b00b30c0 con 0x7fc1b00ad960 2024-01-15T18:00:37.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b015db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 0x7fc1b0166d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b015db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:45872/0 (socket says 172.21.15.2:45872) 2024-01-15T18:00:37.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 0x7fc1b0166d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:57968/0 (socket says 172.21.15.2:57968) 2024-01-15T18:00:37.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b5000640 1 -- 172.21.15.2:0/3831182472 learned_addr learned my addr 172.21.15.2:0/3831182472 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:37.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b5000640 1 -- 172.21.15.2:0/3831182472 >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 msgr2=0x7fc1b0166d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:37.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.094+0000 7fc1b5000640 1 --2- 172.21.15.2:0/3831182472 >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 0x7fc1b0166d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.095+0000 7fc1b5000640 1 -- 172.21.15.2:0/3831182472 >> v1:172.21.15.137:6789/0 conn(0x7fc1b015d750 legacy=0x7fc1b006cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.095+0000 7fc1b5000640 1 -- 172.21.15.2:0/3831182472 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b00b33a0 con 0x7fc1b00ad960 2024-01-15T18:00:37.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.095+0000 7fc1b6400640 1 --2- 172.21.15.2:0/3831182472 >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 0x7fc1b0166d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:00:37.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.096+0000 7fc1b5000640 1 --2- 172.21.15.2:0/3831182472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b015db30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc198009870 tx=0x7fc19802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f7f1a36c5297383 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:37.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.096+0000 7fc1afe00640 1 -- 172.21.15.2:0/3831182472 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198030840 con 0x7fc1b00ad960 2024-01-15T18:00:37.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.096+0000 7fc1afe00640 1 -- 172.21.15.2:0/3831182472 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1980309a0 con 0x7fc1b00ad960 2024-01-15T18:00:37.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.096+0000 7fc1afe00640 1 -- 172.21.15.2:0/3831182472 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198030d30 con 0x7fc1b00ad960 2024-01-15T18:00:37.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 -- 172.21.15.2:0/3831182472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 msgr2=0x7fc1b015db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/3831182472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b015db30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc198009870 tx=0x7fc19802b980 comp rx=0 tx=0).stop 2024-01-15T18:00:37.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 -- 172.21.15.2:0/3831182472 shutdown_connections 2024-01-15T18:00:37.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/3831182472 >> v2:172.21.15.137:3300/0 conn(0x7fc1b015e070 0x7fc1b0166d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/3831182472 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b015db30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 -- 172.21.15.2:0/3831182472 >> 172.21.15.2:0/3831182472 conn(0x7fc1b00b0840 msgr2=0x7fc1b00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:37.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 -- 172.21.15.2:0/3831182472 shutdown_connections 2024-01-15T18:00:37.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.097+0000 7fc1b6e00640 1 -- 172.21.15.2:0/3831182472 wait complete. 2024-01-15T18:00:37.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.099+0000 7fc1b6e00640 1 Processor -- start 2024-01-15T18:00:37.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.099+0000 7fc1b6e00640 1 -- start start 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.100+0000 7fc1b6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b0155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.100+0000 7fc1b6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 0x7fc1b0144560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.100+0000 7fc1b6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b015e070 0x7fc1b00af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.100+0000 7fc1b6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0153fb0 con 0x7fc1b015d750 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.100+0000 7fc1b6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0154290 con 0x7fc1b015e070 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.100+0000 7fc1b6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b0154570 con 0x7fc1b00ad960 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 0x7fc1b0144560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 0x7fc1b0144560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58620/0 (socket says 172.21.15.2:58620) 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 -- 172.21.15.2:0/2311250759 learned_addr learned my addr 172.21.15.2:0/2311250759 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:37.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5a00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b0155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b6400640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b015e070 0x7fc1b00af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 -- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 msgr2=0x7fc1b0155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b0155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 -- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b015e070 msgr2=0x7fc1b00af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b015e070 0x7fc1b00af330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b5000640 1 -- 172.21.15.2:0/2311250759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1a8009a70 con 0x7fc1b015d750 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.101+0000 7fc1b6400640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b015e070 0x7fc1b00af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:00:37.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.102+0000 7fc1b5000640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 0x7fc1b0144560 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc198000c00 tx=0x7fc198004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:37.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.102+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198039a90 con 0x7fc1b015d750 2024-01-15T18:00:37.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.102+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc198039c10 con 0x7fc1b015d750 2024-01-15T18:00:37.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.102+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc198043410 con 0x7fc1b015d750 2024-01-15T18:00:37.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.102+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1980094d0 con 0x7fc1b015d750 2024-01-15T18:00:37.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.102+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1b0154b30 con 0x7fc1b015d750 2024-01-15T18:00:37.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.103+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1b0154d20 con 0x7fc1b015d750 2024-01-15T18:00:37.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.104+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc19802f390 con 0x7fc1b015d750 2024-01-15T18:00:37.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.105+0000 7fc1ad600640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc18806b500 0x7fc18806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.105+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fc1980abe90 con 0x7fc1b015d750 2024-01-15T18:00:37.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.106+0000 7fc1b5a00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc18806b500 0x7fc18806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.107+0000 7fc1b5a00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc18806b500 0x7fc18806d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0001640 tx=0x7fc1a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:37.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.109+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc198076cd0 con 0x7fc1b015d750 2024-01-15T18:00:37.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.305+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc1b0074770 con 0x7fc1b015d750 2024-01-15T18:00:37.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.306+0000 7fc1ad600640 1 -- 172.21.15.2:0/2311250759 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v43) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc198048060 con 0x7fc1b015d750 2024-01-15T18:00:37.309 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:37.309 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:00:37.310 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:37.310 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T18:00:37.310 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T18:00:37.310 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:37.310 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:00:37.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.308+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc18806b500 msgr2=0x7fc18806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.308+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc18806b500 0x7fc18806d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc1a0001640 tx=0x7fc1a0009040 comp rx=0 tx=0).stop 2024-01-15T18:00:37.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.308+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 msgr2=0x7fc1b0144560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.308+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 0x7fc1b0144560 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fc198000c00 tx=0x7fc198004730 comp rx=0 tx=0).stop 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 shutdown_connections 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc18806b500 0x7fc18806d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc1b015e070 0x7fc1b00af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc1b015d750 0x7fc1b0144560 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 --2- 172.21.15.2:0/2311250759 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc1b00ad960 0x7fc1b0155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 >> 172.21.15.2:0/2311250759 conn(0x7fc1b00b0840 msgr2=0x7fc1b00708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 shutdown_connections 2024-01-15T18:00:37.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.309+0000 7fc1b6e00640 1 -- 172.21.15.2:0/2311250759 wait complete. 2024-01-15T18:00:37.330 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:00:37.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:00:37.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.553+0000 7f1b43200640 1 Processor -- start 2024-01-15T18:00:37.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.554+0000 7f1b43200640 1 -- start start 2024-01-15T18:00:37.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.555+0000 7f1b43200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 0x7f1b440ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.555+0000 7f1b43200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b44071b40 con 0x7f1b44143620 2024-01-15T18:00:37.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.555+0000 7f1b43200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b440b2fa0 con 0x7f1b44071230 2024-01-15T18:00:37.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.555+0000 7f1b43200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440b3280 con 0x7f1b440716b0 2024-01-15T18:00:37.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.555+0000 7f1b41e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1b44143620 0x7f1b440b2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38614/0 (socket says 172.21.15.2:38614) 2024-01-15T18:00:37.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41e00640 1 -- 172.21.15.2:0/4041600128 learned_addr learned my addr 172.21.15.2:0/4041600128 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:37.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b42800640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 0x7f1b440ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512751406 0 0) 0x7f1b44071b40 con 0x7f1b44143620 2024-01-15T18:00:37.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24003520 con 0x7f1b44143620 2024-01-15T18:00:37.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634181335 0 0) 0x7f1b24003520 con 0x7f1b44143620 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 msgr2=0x7f1b440ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41400640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 0x7f1b440ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.556+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 >> v1:172.21.15.137:6789/0 conn(0x7f1b44071230 legacy=0x7f1b440cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b440b3560 con 0x7f1b44143620 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1b34001670 con 0x7f1b44143620 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 >> v1:172.21.15.2:6789/0 conn(0x7f1b44143620 legacy=0x7f1b440b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 0x7f1b240039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b24004d00 0x7f1b24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b440b3280 con 0x7f1b24003f20 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2400ba60 con 0x7f1b24004d00 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2400de30 con 0x7f1b440716b0 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b3be00640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b41e00640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 0x7f1b240039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b42800640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b24004d00 0x7f1b24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b3be00640 1 -- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 msgr2=0x7f1b240039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b3be00640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b440716b0 0x7f1b240039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b3be00640 1 -- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b24004d00 msgr2=0x7f1b24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.557+0000 7f1b3be00640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b24004d00 0x7f1b24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.558+0000 7f1b3be00640 1 -- 172.21.15.2:0/4041600128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b24003520 con 0x7f1b24003f20 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b3be00640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b240047c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c001750 tx=0x7f1b240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8d57000a61ecfa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c0052d0 con 0x7f1b24003f20 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c02e050 con 0x7f1b24003f20 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b41400640 1 -- 172.21.15.2:0/4041600128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c003830 con 0x7f1b24003f20 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b43200640 1 -- 172.21.15.2:0/4041600128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 msgr2=0x7f1b240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b43200640 1 --2- 172.21.15.2:0/4041600128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b240047c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c001750 tx=0x7f1b240057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b42800640 1 -- 172.21.15.2:0/4041600128 reap_dead start 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b43200640 1 -- 172.21.15.2:0/4041600128 shutdown_connections 2024-01-15T18:00:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.559+0000 7f1b43200640 1 -- 172.21.15.2:0/4041600128 >> 172.21.15.2:0/4041600128 conn(0x7f1b440b0840 msgr2=0x7f1b440b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:37.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.560+0000 7f1b43200640 1 -- 172.21.15.2:0/4041600128 shutdown_connections 2024-01-15T18:00:37.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.560+0000 7f1b43200640 1 -- 172.21.15.2:0/4041600128 wait complete. 2024-01-15T18:00:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.561+0000 7f1b43200640 1 Processor -- start 2024-01-15T18:00:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 -- start start 2024-01-15T18:00:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b44137ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b44158fb0 0x7f1b441404f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b44140a30 0x7f1b44157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b44140eb0 con 0x7f1b24003f20 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4413af20 con 0x7f1b44158fb0 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.562+0000 7f1b43200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b44138410 con 0x7f1b44140a30 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b44137ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b44137ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58636/0 (socket says 172.21.15.2:58636) 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 -- 172.21.15.2:0/1265445529 learned_addr learned my addr 172.21.15.2:0/1265445529 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b3be00640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b44158fb0 0x7f1b441404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b42800640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b44140a30 0x7f1b44157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 -- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b44140a30 msgr2=0x7f1b44157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b44140a30 0x7f1b44157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 -- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b44158fb0 msgr2=0x7f1b441404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b44158fb0 0x7f1b441404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b41e00640 1 -- 172.21.15.2:0/1265445529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44157f10 con 0x7f1b24003f20 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.563+0000 7f1b42800640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b44140a30 0x7f1b44157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.564+0000 7f1b41e00640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b44137ed0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e820 tx=0x7f1b3400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.564+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400eed0 con 0x7f1b24003f20 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.564+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3400c870 con 0x7f1b24003f20 2024-01-15T18:00:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.564+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3400cc00 con 0x7f1b24003f20 2024-01-15T18:00:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.564+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b441581f0 con 0x7f1b24003f20 2024-01-15T18:00:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.564+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b441587b0 con 0x7f1b24003f20 2024-01-15T18:00:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.565+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b44074770 con 0x7f1b24003f20 2024-01-15T18:00:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.566+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f1b3401d460 con 0x7f1b24003f20 2024-01-15T18:00:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.567+0000 7f1b3a000640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1c06b530 0x7f1b1c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.567+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f1b34089bb0 con 0x7f1b24003f20 2024-01-15T18:00:37.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.568+0000 7f1b3be00640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1c06b530 0x7f1b1c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:37.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.569+0000 7f1b3be00640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1c06b530 0x7f1b1c06d9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c033000 tx=0x7f1b2c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:37.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.572+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b34054ab0 con 0x7f1b24003f20 2024-01-15T18:00:37.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:37.775+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1b440b1ce0 con 0x7f1b24003f20 2024-01-15T18:00:38.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.468+0000 7f1b3a000640 1 -- 172.21.15.2:0/1265445529 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v44) v1 ==== 138+0+0 (secure 0 0 0) 0x7f1b340580d0 con 0x7f1b24003f20 2024-01-15T18:00:38.471 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1c06b530 msgr2=0x7f1b1c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1c06b530 0x7f1b1c06d9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c033000 tx=0x7f1b2c009030 comp rx=0 tx=0).stop 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 msgr2=0x7f1b44137ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b44137ed0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b3400e820 tx=0x7f1b3400ece0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 shutdown_connections 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1c06b530 0x7f1b1c06d9f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b44140a30 0x7f1b44157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b44158fb0 0x7f1b441404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 --2- 172.21.15.2:0/1265445529 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b24003f20 0x7f1b44137ed0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.473+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 >> 172.21.15.2:0/1265445529 conn(0x7f1b440b0840 msgr2=0x7f1b441520e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:38.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.474+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 shutdown_connections 2024-01-15T18:00:38.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.474+0000 7f1b43200640 1 -- 172.21.15.2:0/1265445529 wait complete. 2024-01-15T18:00:38.504 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:00:38.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.717+0000 7f5c91c00640 1 Processor -- start 2024-01-15T18:00:38.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.717+0000 7f5c91c00640 1 -- start start 2024-01-15T18:00:38.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c91c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5c8c143620 0x7f5c8c0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:38.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c91c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0b2fa0 con 0x7f5c8c0b2a90 2024-01-15T18:00:38.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c91c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0b3280 con 0x7f5c8c071940 2024-01-15T18:00:38.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c91c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8c0b3560 con 0x7f5c8c143620 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c8be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5c8c143620 0x7f5c8c0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c8b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5c8c0b2a90 0x7f5c8c071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38630/0 (socket says 172.21.15.2:38630) 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.719+0000 7f5c8b400640 1 -- 172.21.15.2:0/3494084086 learned_addr learned my addr 172.21.15.2:0/3494084086 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.720+0000 7f5c8be00640 1 -- 172.21.15.2:0/3494084086 >> v1:172.21.15.137:6789/0 conn(0x7f5c8c071940 legacy=0x7f5c8c0cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.720+0000 7f5c8be00640 1 -- 172.21.15.2:0/3494084086 >> v1:172.21.15.2:6789/0 conn(0x7f5c8c0b2a90 legacy=0x7f5c8c071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.720+0000 7f5c8aa00640 1 -- 172.21.15.2:0/3494084086 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948757351 0 0) 0x7f5c8c0b2fa0 con 0x7f5c8c0b2a90 2024-01-15T18:00:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.720+0000 7f5c8be00640 1 -- 172.21.15.2:0/3494084086 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0b3840 con 0x7f5c8c143620 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.721+0000 7f5c8be00640 1 --2- 172.21.15.2:0/3494084086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c0b2550 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0097a0 tx=0x7f5c7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f404d3770425e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.721+0000 7f5c8aa00640 1 -- 172.21.15.2:0/3494084086 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c031830 con 0x7f5c8c143620 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.721+0000 7f5c8aa00640 1 -- 172.21.15.2:0/3494084086 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7c031990 con 0x7f5c8c143620 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.721+0000 7f5c8aa00640 1 -- 172.21.15.2:0/3494084086 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c031d20 con 0x7f5c8c143620 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 -- 172.21.15.2:0/3494084086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 msgr2=0x7f5c8c0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 --2- 172.21.15.2:0/3494084086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c0b2550 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0097a0 tx=0x7f5c7c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 -- 172.21.15.2:0/3494084086 shutdown_connections 2024-01-15T18:00:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 --2- 172.21.15.2:0/3494084086 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c0b2550 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 -- 172.21.15.2:0/3494084086 >> 172.21.15.2:0/3494084086 conn(0x7f5c8c0b0840 msgr2=0x7f5c8c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 -- 172.21.15.2:0/3494084086 shutdown_connections 2024-01-15T18:00:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.722+0000 7f5c91c00640 1 -- 172.21.15.2:0/3494084086 wait complete. 2024-01-15T18:00:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.724+0000 7f5c91c00640 1 Processor -- start 2024-01-15T18:00:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.724+0000 7f5c91c00640 1 -- start start 2024-01-15T18:00:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c8c071940 0x7f5c8c13e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c8c0b2a90 0x7f5c8c1f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c13d4c0 con 0x7f5c8c143620 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c13c8b0 con 0x7f5c8c071940 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8c13cb90 con 0x7f5c8c0b2a90 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.725+0000 7f5c91200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c13d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c13d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58654/0 (socket says 172.21.15.2:58654) 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 -- 172.21.15.2:0/669482810 learned_addr learned my addr 172.21.15.2:0/669482810 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c8b400640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c8c0b2a90 0x7f5c8c1f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c8be00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c8c071940 0x7f5c8c13e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 -- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c8c0b2a90 msgr2=0x7f5c8c1f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c8c0b2a90 0x7f5c8c1f0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 -- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c8c071940 msgr2=0x7f5c8c13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c8c071940 0x7f5c8c13e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 -- 172.21.15.2:0/669482810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c009400 con 0x7f5c8c143620 2024-01-15T18:00:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.726+0000 7f5c91200640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c13d950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400c700 tx=0x7f5c8400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.727+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400aa50 con 0x7f5c8c143620 2024-01-15T18:00:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.727+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c8400abb0 con 0x7f5c8c143620 2024-01-15T18:00:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.727+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c8400ce10 con 0x7f5c8c143620 2024-01-15T18:00:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.727+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c13ce70 con 0x7f5c8c143620 2024-01-15T18:00:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.727+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c13ed50 con 0x7f5c8c143620 2024-01-15T18:00:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.728+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c13d670 con 0x7f5c8c143620 2024-01-15T18:00:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.730+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5c84006b50 con 0x7f5c8c143620 2024-01-15T18:00:38.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.731+0000 7f5c83e00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c6406b500 0x7f5c6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:38.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.731+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f5c8408a660 con 0x7f5c8c143620 2024-01-15T18:00:38.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.732+0000 7f5c8be00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c6406b500 0x7f5c6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.733+0000 7f5c8be00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c6406b500 0x7f5c6406d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c036000 tx=0x7f5c7c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:38.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.735+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c84055310 con 0x7f5c8c143620 2024-01-15T18:00:38.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:38.930+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f5c8c13d670 con 0x7f5c8c143620 2024-01-15T18:00:39.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.481+0000 7f5c83e00640 1 -- 172.21.15.2:0/669482810 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v45) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5c8c13d670 con 0x7f5c8c143620 2024-01-15T18:00:39.485 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T18:00:39.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.486+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c6406b500 msgr2=0x7f5c6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:39.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.486+0000 7f5c91c00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c6406b500 0x7f5c6406d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c036000 tx=0x7f5c7c03c040 comp rx=0 tx=0).stop 2024-01-15T18:00:39.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.486+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 msgr2=0x7f5c8c13d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:39.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.486+0000 7f5c91c00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c13d950 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5c8400c700 tx=0x7f5c8400cbc0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 shutdown_connections 2024-01-15T18:00:39.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c6406b500 0x7f5c6406d9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c8c143620 0x7f5c8c13d950 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c8c0b2a90 0x7f5c8c1f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 --2- 172.21.15.2:0/669482810 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c8c071940 0x7f5c8c13e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 >> 172.21.15.2:0/669482810 conn(0x7f5c8c0b0840 msgr2=0x7f5c8c158c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:39.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.487+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 shutdown_connections 2024-01-15T18:00:39.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.488+0000 7f5c91c00640 1 -- 172.21.15.2:0/669482810 wait complete. 2024-01-15T18:00:39.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:00:39.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.708+0000 7f032da00640 1 Processor -- start 2024-01-15T18:00:39.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.709+0000 7f032da00640 1 -- start start 2024-01-15T18:00:39.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.710+0000 7f032da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0328144c00 0x7f03280cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.710+0000 7f032da00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03280cdd70 0x7f032806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.710+0000 7f032da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f032815c450 0x7f032806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.710+0000 7f032da00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280ce230 con 0x7f032815c450 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.710+0000 7f032da00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03280ce510 con 0x7f03280cdd70 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.710+0000 7f032da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280ce7f0 con 0x7f0328144c00 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f032815c450 0x7f032806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f032815c450 0x7f032806ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58666/0 (socket says 172.21.15.2:58666) 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 -- 172.21.15.2:0/2923574132 learned_addr learned my addr 172.21.15.2:0/2923574132 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0326a00640 1 --2- 172.21.15.2:0/2923574132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0328144c00 0x7f03280cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f031fe00640 1 --2- 172.21.15.2:0/2923574132 >> v2:172.21.15.137:3300/0 conn(0x7f03280cdd70 0x7f032806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 -- 172.21.15.2:0/2923574132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0328144c00 msgr2=0x7f03280cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 --2- 172.21.15.2:0/2923574132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0328144c00 0x7f03280cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 -- 172.21.15.2:0/2923574132 >> v2:172.21.15.137:3300/0 conn(0x7f03280cdd70 msgr2=0x7f032806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 --2- 172.21.15.2:0/2923574132 >> v2:172.21.15.137:3300/0 conn(0x7f03280cdd70 0x7f032806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.711+0000 7f0327400640 1 -- 172.21.15.2:0/2923574132 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03280705f0 con 0x7f032815c450 2024-01-15T18:00:39.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.713+0000 7f0327400640 1 --2- 172.21.15.2:0/2923574132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032815c450 0x7f032806ce30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f032806d420 tx=0x7f032002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5119aec895de793e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:39.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.713+0000 7f0326000640 1 -- 172.21.15.2:0/2923574132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f032002ca70 con 0x7f032815c450 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.713+0000 7f0326000640 1 -- 172.21.15.2:0/2923574132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f032002cbd0 con 0x7f032815c450 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.713+0000 7f0326000640 1 -- 172.21.15.2:0/2923574132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03200343f0 con 0x7f032815c450 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.713+0000 7f032da00640 1 -- 172.21.15.2:0/2923574132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032815c450 msgr2=0x7f032806ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.713+0000 7f032da00640 1 --2- 172.21.15.2:0/2923574132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032815c450 0x7f032806ce30 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f032806d420 tx=0x7f032002bed0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 -- 172.21.15.2:0/2923574132 shutdown_connections 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 --2- 172.21.15.2:0/2923574132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f032815c450 0x7f032806ce30 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 --2- 172.21.15.2:0/2923574132 >> v2:172.21.15.137:3300/0 conn(0x7f03280cdd70 0x7f032806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 --2- 172.21.15.2:0/2923574132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0328144c00 0x7f03280cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 -- 172.21.15.2:0/2923574132 >> 172.21.15.2:0/2923574132 conn(0x7f03280b45f0 msgr2=0x7f03280b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 -- 172.21.15.2:0/2923574132 shutdown_connections 2024-01-15T18:00:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.714+0000 7f032da00640 1 -- 172.21.15.2:0/2923574132 wait complete. 2024-01-15T18:00:39.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.716+0000 7f032da00640 1 Processor -- start 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.716+0000 7f032da00640 1 -- start start 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f032da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 0x7f03280acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f032da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328144c00 0x7f0328159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f032da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f032815c450 0x7f0328140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f032da00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03281f37d0 con 0x7f03280cdd70 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f032da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03281f2c60 con 0x7f0328144c00 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f032da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03280ad590 con 0x7f032815c450 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f0326a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 0x7f03280acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f0326a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 0x7f03280acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58682/0 (socket says 172.21.15.2:58682) 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f0326a00640 1 -- 172.21.15.2:0/1367613740 learned_addr learned my addr 172.21.15.2:0/1367613740 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f031fe00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328144c00 0x7f0328159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f0327400640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f032815c450 0x7f0328140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.717+0000 7f0326a00640 1 -- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f032815c450 msgr2=0x7f0328140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f0326a00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f032815c450 0x7f0328140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f0326a00640 1 -- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328144c00 msgr2=0x7f0328159390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f0326a00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328144c00 0x7f0328159390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f0326a00640 1 -- 172.21.15.2:0/1367613740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0320009b30 con 0x7f03280cdd70 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f031fe00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328144c00 0x7f0328159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f0326a00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 0x7f03280acd40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f031800d830 tx=0x7f031800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031800b6c0 con 0x7f03280cdd70 2024-01-15T18:00:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031800b820 con 0x7f03280cdd70 2024-01-15T18:00:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f031800bbb0 con 0x7f03280cdd70 2024-01-15T18:00:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03281f3bf0 con 0x7f03280cdd70 2024-01-15T18:00:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.718+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03281f33f0 con 0x7f03280cdd70 2024-01-15T18:00:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.720+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03280686f0 con 0x7f03280cdd70 2024-01-15T18:00:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.721+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f031801e460 con 0x7f03280cdd70 2024-01-15T18:00:39.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.722+0000 7f031ea00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f030006b500 0x7f030006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:39.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.722+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f031808a540 con 0x7f03280cdd70 2024-01-15T18:00:39.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.723+0000 7f031fe00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f030006b500 0x7f030006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:39.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.724+0000 7f031fe00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f030006b500 0x7f030006d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f03280accd0 tx=0x7f0310008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:39.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.728+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0318055120 con 0x7f03280cdd70 2024-01-15T18:00:39.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:39.955+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f03281f3bf0 con 0x7f03280cdd70 2024-01-15T18:00:40.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.513+0000 7f031ea00640 1 -- 172.21.15.2:0/1367613740 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v22) v1 ==== 172+0+0 (secure 0 0 0) 0x7f03281f3bf0 con 0x7f03280cdd70 2024-01-15T18:00:40.520 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 6 and data pool 7 2024-01-15T18:00:40.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.520+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f030006b500 msgr2=0x7f030006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.520+0000 7f032da00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f030006b500 0x7f030006d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f03280accd0 tx=0x7f0310008040 comp rx=0 tx=0).stop 2024-01-15T18:00:40.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.520+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 msgr2=0x7f03280acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.520+0000 7f032da00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 0x7f03280acd40 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f031800d830 tx=0x7f031800dcf0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.527+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 shutdown_connections 2024-01-15T18:00:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.528+0000 7f032da00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f030006b500 0x7f030006d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.528+0000 7f032da00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f032815c450 0x7f0328140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.528+0000 7f032da00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0328144c00 0x7f0328159390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.528+0000 7f032da00640 1 --2- 172.21.15.2:0/1367613740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03280cdd70 0x7f03280acd40 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.528+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 >> 172.21.15.2:0/1367613740 conn(0x7f03280b45f0 msgr2=0x7f0328071a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:40.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.528+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 shutdown_connections 2024-01-15T18:00:40.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.529+0000 7f032da00640 1 -- 172.21.15.2:0/1367613740 wait complete. 2024-01-15T18:00:40.562 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:40.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.735+0000 7f5351600640 1 Processor -- start 2024-01-15T18:00:40.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.735+0000 7f5351600640 1 -- start start 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f5351600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f534c15b7d0 0x7f534c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f5351600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f534c15c4c0 0x7f534c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f5351600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f534c15cb00 con 0x7f534c15c0e0 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f5351600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534c0cccd0 con 0x7f534c15c4c0 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f5351600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f534c0ccec0 con 0x7f534c15b7d0 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f534c15c4c0 0x7f534c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f534c15c4c0 0x7f534c06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58686/0 (socket says 172.21.15.2:58686) 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534b400640 1 -- 172.21.15.2:0/3883919057 learned_addr learned my addr 172.21.15.2:0/3883919057 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534a000640 1 --2- 172.21.15.2:0/3883919057 >> v2:172.21.15.137:3300/0 conn(0x7f534c15b7d0 0x7f534c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534b400640 1 -- 172.21.15.2:0/3883919057 >> v2:172.21.15.137:3300/0 conn(0x7f534c15b7d0 msgr2=0x7f534c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534b400640 1 --2- 172.21.15.2:0/3883919057 >> v2:172.21.15.137:3300/0 conn(0x7f534c15b7d0 0x7f534c06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.737+0000 7f534b400640 1 -- 172.21.15.2:0/3883919057 >> v1:172.21.15.137:6789/0 conn(0x7f534c15c0e0 legacy=0x7f534c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.738+0000 7f534b400640 1 -- 172.21.15.2:0/3883919057 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f534c0cd1a0 con 0x7f534c15c4c0 2024-01-15T18:00:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f534b400640 1 --2- 172.21.15.2:0/3883919057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c06c9b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f534c072830 tx=0x7f534402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94533d9c7899486d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5349600640 1 -- 172.21.15.2:0/3883919057 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344031870 con 0x7f534c15c4c0 2024-01-15T18:00:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5349600640 1 -- 172.21.15.2:0/3883919057 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53440319d0 con 0x7f534c15c4c0 2024-01-15T18:00:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5349600640 1 -- 172.21.15.2:0/3883919057 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344031d60 con 0x7f534c15c4c0 2024-01-15T18:00:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5351600640 1 -- 172.21.15.2:0/3883919057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 msgr2=0x7f534c06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5351600640 1 --2- 172.21.15.2:0/3883919057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c06c9b0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f534c072830 tx=0x7f534402b910 comp rx=0 tx=0).stop 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5351600640 1 -- 172.21.15.2:0/3883919057 shutdown_connections 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5351600640 1 --2- 172.21.15.2:0/3883919057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c06c9b0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5351600640 1 --2- 172.21.15.2:0/3883919057 >> v2:172.21.15.137:3300/0 conn(0x7f534c15b7d0 0x7f534c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.739+0000 7f5351600640 1 -- 172.21.15.2:0/3883919057 >> 172.21.15.2:0/3883919057 conn(0x7f534c06da20 msgr2=0x7f534c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.740+0000 7f5351600640 1 -- 172.21.15.2:0/3883919057 shutdown_connections 2024-01-15T18:00:40.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.740+0000 7f5351600640 1 -- 172.21.15.2:0/3883919057 wait complete. 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.741+0000 7f5351600640 1 Processor -- start 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.741+0000 7f5351600640 1 -- start start 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f5351600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f534c15b7d0 0x7f534c13ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f5351600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f534c15c0e0 0x7f534c152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f5351600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f5351600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c153ab0 con 0x7f534c15c4c0 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f5351600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534c153e60 con 0x7f534c15c0e0 2024-01-15T18:00:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f5351600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534c13bfa0 con 0x7f534c15b7d0 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.742+0000 7f534b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58698/0 (socket says 172.21.15.2:58698) 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 -- 172.21.15.2:0/3438114320 learned_addr learned my addr 172.21.15.2:0/3438114320 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534a000640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f534c15c0e0 0x7f534c152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534aa00640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f534c15b7d0 0x7f534c13ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 -- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f534c15b7d0 msgr2=0x7f534c13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f534c15b7d0 0x7f534c13ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 -- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f534c15c0e0 msgr2=0x7f534c152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f534c15c0e0 0x7f534c152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 -- 172.21.15.2:0/3438114320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53440094d0 con 0x7f534c15c4c0 2024-01-15T18:00:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534a000640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f534c15c0e0 0x7f534c152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.743+0000 7f534b400640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c149db0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f53440090b0 tx=0x7f53440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.744+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53440443f0 con 0x7f534c15c4c0 2024-01-15T18:00:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.744+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5344044570 con 0x7f534c15c4c0 2024-01-15T18:00:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.744+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5344044920 con 0x7f534c15c4c0 2024-01-15T18:00:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.744+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f534c153210 con 0x7f534c15c4c0 2024-01-15T18:00:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.744+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f534c1537d0 con 0x7f534c15c4c0 2024-01-15T18:00:40.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.745+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f534c153c30 con 0x7f534c15c4c0 2024-01-15T18:00:40.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.746+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f5344044a80 con 0x7f534c15c4c0 2024-01-15T18:00:40.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.747+0000 7f5342a00640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f532406b500 0x7f532406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:40.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.747+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f534402d040 con 0x7f534c15c4c0 2024-01-15T18:00:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.748+0000 7f534aa00640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f532406b500 0x7f532406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.749+0000 7f534aa00640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f532406b500 0x7f532406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f533c007fd0 tx=0x7f533c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.751+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f534c153c30 con 0x7f534c15c4c0 2024-01-15T18:00:40.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.945+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f534c153c30 con 0x7f534c15c4c0 2024-01-15T18:00:40.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.947+0000 7f5342a00640 1 -- 172.21.15.2:0/3438114320 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16509 (secure 0 0 0) 0x7f534c153c30 con 0x7f534c15c4c0 2024-01-15T18:00:40.952 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:40.953 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":46,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:39.958396+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:00:37.830306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-15T18:00:38.982411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:40.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f532406b500 msgr2=0x7f532406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f532406b500 0x7f532406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f533c007fd0 tx=0x7f533c009040 comp rx=0 tx=0).stop 2024-01-15T18:00:40.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 msgr2=0x7f534c149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:40.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c149db0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f53440090b0 tx=0x7f53440046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 shutdown_connections 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f532406b500 0x7f532406d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f534c15c4c0 0x7f534c149db0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f534c15c0e0 0x7f534c152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 --2- 172.21.15.2:0/3438114320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f534c15b7d0 0x7f534c13ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 >> 172.21.15.2:0/3438114320 conn(0x7f534c06da20 msgr2=0x7f534c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 shutdown_connections 2024-01-15T18:00:40.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:40.950+0000 7f5351600640 1 -- 172.21.15.2:0/3438114320 wait complete. 2024-01-15T18:00:40.975 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.192+0000 7fec38400640 1 Processor -- start 2024-01-15T18:00:41.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.192+0000 7fec38400640 1 -- start start 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec38400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec3015c0e0 0x7fec300cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec38400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec3015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec38400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3015cb00 con 0x7fec3015b7d0 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec38400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec300ccec0 con 0x7fec3015c0e0 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec38400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec300cd1a0 con 0x7fec3006cb80 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec35000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fec3015b7d0 0x7fec3006c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38636/0 (socket says 172.21.15.2:38636) 2024-01-15T18:00:41.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec35000640 1 -- 172.21.15.2:0/715103873 learned_addr learned my addr 172.21.15.2:0/715103873 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec35a00640 1 --2- 172.21.15.2:0/715103873 >> v2:172.21.15.137:3300/0 conn(0x7fec3015c0e0 0x7fec300cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.194+0000 7fec36400640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec3015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632937609 0 0) 0x7fec3015cb00 con 0x7fec3015b7d0 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec10003520 con 0x7fec3015b7d0 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1366220860 0 0) 0x7fec10003520 con 0x7fec3015b7d0 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 msgr2=0x7fec3015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec3015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 >> v2:172.21.15.137:3300/0 conn(0x7fec3015c0e0 msgr2=0x7fec300cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 --2- 172.21.15.2:0/715103873 >> v2:172.21.15.137:3300/0 conn(0x7fec3015c0e0 0x7fec300cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec300cd480 con 0x7fec3015b7d0 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fec18001670 con 0x7fec3015b7d0 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 >> v1:172.21.15.2:6789/0 conn(0x7fec3015b7d0 legacy=0x7fec3006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec10003970 0x7fec10003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec10005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec300ccec0 con 0x7fec100047c0 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec300cd1a0 con 0x7fec10003970 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1000b980 con 0x7fec3006cb80 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.195+0000 7fec35000640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec36400640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec10005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec35a00640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec10003970 0x7fec10003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec35000640 1 -- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 msgr2=0x7fec10005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec35000640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec10005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec35000640 1 -- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec10003970 msgr2=0x7fec10003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec35000640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec10003970 0x7fec10003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec35000640 1 -- 172.21.15.2:0/715103873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec10003520 con 0x7fec100047c0 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.196+0000 7fec36400640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec3006cb80 0x7fec10005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.197+0000 7fec35000640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec10004c40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fec1800c240 tx=0x7fec100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88b0f55e10b85eed server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.197+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec18005ad0 con 0x7fec100047c0 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.197+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec18005c30 con 0x7fec100047c0 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.197+0000 7fec2fe00640 1 -- 172.21.15.2:0/715103873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1802c8d0 con 0x7fec100047c0 2024-01-15T18:00:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec38400640 1 -- 172.21.15.2:0/715103873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 msgr2=0x7fec10004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec38400640 1 --2- 172.21.15.2:0/715103873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec10004c40 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fec1800c240 tx=0x7fec100057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec36400640 1 -- 172.21.15.2:0/715103873 reap_dead start 2024-01-15T18:00:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec38400640 1 -- 172.21.15.2:0/715103873 shutdown_connections 2024-01-15T18:00:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec38400640 1 -- 172.21.15.2:0/715103873 >> 172.21.15.2:0/715103873 conn(0x7fec3006da20 msgr2=0x7fec3006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec38400640 1 -- 172.21.15.2:0/715103873 shutdown_connections 2024-01-15T18:00:41.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.198+0000 7fec38400640 1 -- 172.21.15.2:0/715103873 wait complete. 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.200+0000 7fec38400640 1 Processor -- start 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.200+0000 7fec38400640 1 -- start start 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec38400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec30157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec38400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec30155d00 0x7fec30150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec38400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec30151030 0x7fec301f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec38400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec30157760 con 0x7fec100047c0 2024-01-15T18:00:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec38400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec301514b0 con 0x7fec30155d00 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec38400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec30152a50 con 0x7fec30151030 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec35a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec30157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec35a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec30157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58730/0 (socket says 172.21.15.2:58730) 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec35a00640 1 -- 172.21.15.2:0/3924417261 learned_addr learned my addr 172.21.15.2:0/3924417261 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec35a00640 1 -- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec30151030 msgr2=0x7fec301f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.201+0000 7fec35a00640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec30151030 0x7fec301f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec35a00640 1 -- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec30155d00 msgr2=0x7fec30150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec35a00640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec30155d00 0x7fec30150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec35a00640 1 -- 172.21.15.2:0/3924417261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30152d30 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec35a00640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec30157220 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fec200108e0 tx=0x7fec20010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2000ea80 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2000ebe0 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2001c400 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec30153010 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.202+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec301535d0 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.204+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec300686f0 con 0x7fec100047c0 2024-01-15T18:00:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.204+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fec2001c560 con 0x7fec100047c0 2024-01-15T18:00:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.205+0000 7fec2d600640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec0806b500 0x7fec0806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.206+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fec2008a5f0 con 0x7fec100047c0 2024-01-15T18:00:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.206+0000 7fec35000640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec0806b500 0x7fec0806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.207+0000 7fec35000640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec0806b500 0x7fec0806d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec18034000 tx=0x7fec1802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:41.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.210+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec20055110 con 0x7fec100047c0 2024-01-15T18:00:41.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.410+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec3015ea20 con 0x7fec100047c0 2024-01-15T18:00:41.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.413+0000 7fec2d600640 1 -- 172.21.15.2:0/3924417261 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16509 (secure 0 0 0) 0x7fec20058730 con 0x7fec100047c0 2024-01-15T18:00:41.418 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:41.418 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":46,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:39.958396+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:00:37.830306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-15T18:00:38.982411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:41.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec0806b500 msgr2=0x7fec0806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec0806b500 0x7fec0806d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fec18034000 tx=0x7fec1802e000 comp rx=0 tx=0).stop 2024-01-15T18:00:41.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 msgr2=0x7fec30157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec30157220 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fec200108e0 tx=0x7fec20010da0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 shutdown_connections 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec0806b500 0x7fec0806d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec30151030 0x7fec301f1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec30155d00 0x7fec30150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.416+0000 7fec38400640 1 --2- 172.21.15.2:0/3924417261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec100047c0 0x7fec30157220 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.417+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 >> 172.21.15.2:0/3924417261 conn(0x7fec3006da20 msgr2=0x7fec3015ec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.417+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 shutdown_connections 2024-01-15T18:00:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.417+0000 7fec38400640 1 -- 172.21.15.2:0/3924417261 wait complete. 2024-01-15T18:00:41.440 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:00:41.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.655+0000 7f7863e00640 1 Processor -- start 2024-01-15T18:00:41.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.655+0000 7f7863e00640 1 -- start start 2024-01-15T18:00:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.656+0000 7f7863e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 0x7f785c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f7863e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c0b25f0 con 0x7f785c14b3b0 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f7863e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785c0b28d0 con 0x7f785c06d000 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f7863e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c070ff0 con 0x7f785c14afd0 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f7860a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f785c14b3b0 0x7f785c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38638/0 (socket says 172.21.15.2:38638) 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f7860a00640 1 -- 172.21.15.2:0/2400082413 learned_addr learned my addr 172.21.15.2:0/2400082413 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f7861400640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 0x7f785c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.657+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830410062 0 0) 0x7f785c0b25f0 con 0x7f785c14b3b0 2024-01-15T18:00:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f783c003520 con 0x7f785c14b3b0 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359093264 0 0) 0x7f783c003520 con 0x7f785c14b3b0 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 msgr2=0x7f785c0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 0x7f785c0b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 >> v1:172.21.15.137:6789/0 conn(0x7f785c06d000 legacy=0x7f785c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c0b2bb0 con 0x7f785c14b3b0 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7844001670 con 0x7f785c14b3b0 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 >> v1:172.21.15.2:6789/0 conn(0x7f785c14b3b0 legacy=0x7f785c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 0x7f783c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f783c003f70 0x7f783c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f783c004d00 0x7f783c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c070ff0 con 0x7f783c004d00 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f783c00bb70 con 0x7f783c003f70 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f783c00bd30 con 0x7f785c14afd0 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.658+0000 7f7861e00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f783c004d00 0x7f783c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7861400640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 0x7f783c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7860a00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f783c003f70 0x7f783c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7861e00640 1 -- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 msgr2=0x7f783c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7861e00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f785c14afd0 0x7f783c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7861e00640 1 -- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f783c003f70 msgr2=0x7f783c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7861e00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f783c003f70 0x7f783c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.659+0000 7f7861e00640 1 -- 172.21.15.2:0/2400082413 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c003520 con 0x7f783c004d00 2024-01-15T18:00:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.660+0000 7f7861e00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f783c004d00 0x7f783c005180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7854001e50 tx=0x7f783c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5965af9ce07b641 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.660+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854003b70 con 0x7f783c004d00 2024-01-15T18:00:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.660+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854003cd0 con 0x7f783c004d00 2024-01-15T18:00:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.660+0000 7f785be00640 1 -- 172.21.15.2:0/2400082413 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854005310 con 0x7f783c004d00 2024-01-15T18:00:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7863e00640 1 -- 172.21.15.2:0/2400082413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f783c004d00 msgr2=0x7f783c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7863e00640 1 --2- 172.21.15.2:0/2400082413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f783c004d00 0x7f783c005180 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f7854001e50 tx=0x7f783c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7861e00640 1 -- 172.21.15.2:0/2400082413 reap_dead start 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7863e00640 1 -- 172.21.15.2:0/2400082413 shutdown_connections 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7863e00640 1 -- 172.21.15.2:0/2400082413 >> 172.21.15.2:0/2400082413 conn(0x7f785c0b45f0 msgr2=0x7f785c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7863e00640 1 -- 172.21.15.2:0/2400082413 shutdown_connections 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.661+0000 7f7863e00640 1 -- 172.21.15.2:0/2400082413 wait complete. 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.663+0000 7f7863e00640 1 Processor -- start 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.663+0000 7f7863e00640 1 -- start start 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7863e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f783c004d00 0x7f785c1e7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7863e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 0x7f785c1460a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7863e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f785c1e3910 0x7f785c1e3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7863e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1e42d0 con 0x7f785c1e5a10 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7863e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1e5fd0 con 0x7f785c1e3910 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7863e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785c1e62b0 con 0x7f783c004d00 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7860a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 0x7f785c1460a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7860a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 0x7f785c1460a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58750/0 (socket says 172.21.15.2:58750) 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.664+0000 7f7860a00640 1 -- 172.21.15.2:0/2987199783 learned_addr learned my addr 172.21.15.2:0/2987199783 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7861400640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f783c004d00 0x7f785c1e7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7860a00640 1 -- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f783c004d00 msgr2=0x7f785c1e7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7860a00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f783c004d00 0x7f785c1e7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7860a00640 1 -- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f785c1e3910 msgr2=0x7f785c1e3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7860a00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f785c1e3910 0x7f785c1e3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7860a00640 1 -- 172.21.15.2:0/2987199783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c1e6590 con 0x7f785c1e5a10 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7860a00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 0x7f785c1460a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f784400e7d0 tx=0x7f784400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784400c7f0 con 0x7f785c1e5a10 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784400c950 con 0x7f785c1e5a10 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784400cd00 con 0x7f785c1e5a10 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.665+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785c1e6870 con 0x7f785c1e5a10 2024-01-15T18:00:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.666+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f785c1e7fc0 con 0x7f785c1e5a10 2024-01-15T18:00:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.667+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785c1f76d0 con 0x7f785c1e5a10 2024-01-15T18:00:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.668+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f784401d460 con 0x7f785c1e5a10 2024-01-15T18:00:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.669+0000 7f7859600640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f783406b500 0x7f783406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.669+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7844089fd0 con 0x7f785c1e5a10 2024-01-15T18:00:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.669+0000 7f7861400640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f783406b500 0x7f783406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.670+0000 7f7861400640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f783406b500 0x7f783406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f784c009950 tx=0x7f784c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:41.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.673+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f785c1f76d0 con 0x7f785c1e5a10 2024-01-15T18:00:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:41.903+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f785c1f76d0 con 0x7f785c1e5a10 2024-01-15T18:00:42.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.524+0000 7f7859600640 1 -- 172.21.15.2:0/2987199783 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v25) v1 ==== 118+0+0 (secure 0 0 0) 0x7f785c1f76d0 con 0x7f785c1e5a10 2024-01-15T18:00:42.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.527+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f783406b500 msgr2=0x7f783406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.527+0000 7f7863e00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f783406b500 0x7f783406d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f784c009950 tx=0x7f784c008040 comp rx=0 tx=0).stop 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 msgr2=0x7f785c1460a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 0x7f785c1460a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f784400e7d0 tx=0x7f784400ec90 comp rx=0 tx=0).stop 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 shutdown_connections 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f783406b500 0x7f783406d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f785c1e3910 0x7f785c1e3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f785c1e5a10 0x7f785c1460a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 --2- 172.21.15.2:0/2987199783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f783c004d00 0x7f785c1e7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 >> 172.21.15.2:0/2987199783 conn(0x7f785c0b45f0 msgr2=0x7f785c0ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:42.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 shutdown_connections 2024-01-15T18:00:42.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.528+0000 7f7863e00640 1 -- 172.21.15.2:0/2987199783 wait complete. 2024-01-15T18:00:42.554 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:00:42.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:00:42.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.777+0000 7fee02800640 1 Processor -- start 2024-01-15T18:00:42.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.777+0000 7fee02800640 1 -- start start 2024-01-15T18:00:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.779+0000 7fee02800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7fedfc071c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.779+0000 7fee02800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedfc0ce880 con 0x7fedfc06abc0 2024-01-15T18:00:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.779+0000 7fee02800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedfc0ca610 con 0x7fedfc06d250 2024-01-15T18:00:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.779+0000 7fee02800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc072160 con 0x7fedfc06f2b0 2024-01-15T18:00:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.779+0000 7fee01e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fedfc06abc0 0x7fedfc0ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38650/0 (socket says 172.21.15.2:38650) 2024-01-15T18:00:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.779+0000 7fee01e00640 1 -- 172.21.15.2:0/1332889336 learned_addr learned my addr 172.21.15.2:0/1332889336 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fee01400640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7fedfc071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216646301 0 0) 0x7fedfc0ce880 con 0x7fedfc06abc0 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feddc003520 con 0x7fedfc06abc0 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196714489 0 0) 0x7feddc003520 con 0x7fedfc06abc0 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 msgr2=0x7fedfc071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7fedfc071c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 >> v1:172.21.15.137:6789/0 conn(0x7fedfc06d250 legacy=0x7fedfc06a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc0cdc70 con 0x7fedfc06abc0 2024-01-15T18:00:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fedf40019f0 con 0x7fedfc06abc0 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 >> v1:172.21.15.2:6789/0 conn(0x7fedfc06abc0 legacy=0x7fedfc0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7feddc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7feddc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feddc004d00 0x7feddc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc072160 con 0x7feddc0039a0 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feddc00ba60 con 0x7feddc004d00 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.780+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc00bc20 con 0x7fedfc06f2b0 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01400640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7feddc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01e00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feddc004d00 0x7feddc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee00a00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7feddc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01400640 1 -- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 msgr2=0x7feddc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01400640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7feddc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01400640 1 -- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feddc004d00 msgr2=0x7feddc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01400640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feddc004d00 0x7feddc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee01400640 1 -- 172.21.15.2:0/1332889336 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc003520 con 0x7feddc0039a0 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.781+0000 7fee00a00640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc06f2b0 0x7feddc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.782+0000 7fee01400640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7feddc003dc0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fedec009a90 tx=0x7feddc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa2a3f87eee50ae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec02c680 con 0x7feddc0039a0 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec02c7e0 con 0x7feddc0039a0 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fedfbe00640 1 -- 172.21.15.2:0/1332889336 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec02cb70 con 0x7feddc0039a0 2024-01-15T18:00:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee02800640 1 -- 172.21.15.2:0/1332889336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 msgr2=0x7feddc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee02800640 1 --2- 172.21.15.2:0/1332889336 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7feddc003dc0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fedec009a90 tx=0x7feddc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee01e00640 1 -- 172.21.15.2:0/1332889336 reap_dead start 2024-01-15T18:00:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee02800640 1 -- 172.21.15.2:0/1332889336 shutdown_connections 2024-01-15T18:00:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee02800640 1 -- 172.21.15.2:0/1332889336 >> 172.21.15.2:0/1332889336 conn(0x7fedfc0cfd30 msgr2=0x7fedfc0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee02800640 1 -- 172.21.15.2:0/1332889336 shutdown_connections 2024-01-15T18:00:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.783+0000 7fee02800640 1 -- 172.21.15.2:0/1332889336 wait complete. 2024-01-15T18:00:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.785+0000 7fee02800640 1 Processor -- start 2024-01-15T18:00:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.785+0000 7fee02800640 1 -- start start 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.786+0000 7fee02800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7fedfc15abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.786+0000 7fee02800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedfc0c7fd0 0x7fedfc0ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee02800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc0add30 0x7fedfc0b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee02800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0ae1e0 con 0x7feddc0039a0 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee02800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0c5710 con 0x7fedfc0c7fd0 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee02800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedfc0c59f0 con 0x7fedfc0add30 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7fedfc15abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7fedfc15abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58772/0 (socket says 172.21.15.2:58772) 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 -- 172.21.15.2:0/1385066755 learned_addr learned my addr 172.21.15.2:0/1385066755 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee00a00640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedfc0c7fd0 0x7fedfc0ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01e00640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc0add30 0x7fedfc0b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 -- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc0add30 msgr2=0x7fedfc0b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc0add30 0x7fedfc0b2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 -- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedfc0c7fd0 msgr2=0x7fedfc0ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedfc0c7fd0 0x7fedfc0ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.787+0000 7fee01400640 1 -- 172.21.15.2:0/1385066755 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedfc0c5cd0 con 0x7feddc0039a0 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.788+0000 7fee01400640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7fedfc15abb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fedec02c220 tx=0x7fedec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.788+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec003810 con 0x7feddc0039a0 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.788+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedec003970 con 0x7feddc0039a0 2024-01-15T18:00:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.788+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedec003ce0 con 0x7feddc0039a0 2024-01-15T18:00:42.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.788+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedfc0c5fb0 con 0x7feddc0039a0 2024-01-15T18:00:42.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.788+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedfc0afb90 con 0x7feddc0039a0 2024-01-15T18:00:42.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.789+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedfc0c6140 con 0x7feddc0039a0 2024-01-15T18:00:42.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.790+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fedec034440 con 0x7feddc0039a0 2024-01-15T18:00:42.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.791+0000 7fedf9600640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedd406b530 0x7fedd406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:42.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.791+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fedec0abe40 con 0x7feddc0039a0 2024-01-15T18:00:42.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.792+0000 7fee00a00640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedd406b530 0x7fedd406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:42.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.793+0000 7fee00a00640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedd406b530 0x7fedd406d9f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0c7f60 tx=0x7fede4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:42.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:42.796+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fedec0769d0 con 0x7feddc0039a0 2024-01-15T18:00:43.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.036+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fedfc0ae140 con 0x7feddc0039a0 2024-01-15T18:00:43.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.535+0000 7fedf9600640 1 -- 172.21.15.2:0/1385066755 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fedec0044e0 con 0x7feddc0039a0 2024-01-15T18:00:43.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedd406b530 msgr2=0x7fedd406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:43.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedd406b530 0x7fedd406d9f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fedfc0c7f60 tx=0x7fede4009040 comp rx=0 tx=0).stop 2024-01-15T18:00:43.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 msgr2=0x7fedfc15abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:43.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7fedfc15abb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fedec02c220 tx=0x7fedec004730 comp rx=0 tx=0).stop 2024-01-15T18:00:43.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 shutdown_connections 2024-01-15T18:00:43.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fedd406b530 0x7fedd406d9f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fedfc0add30 0x7fedfc0b2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fedfc0c7fd0 0x7fedfc0ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 --2- 172.21.15.2:0/1385066755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feddc0039a0 0x7fedfc15abb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 >> 172.21.15.2:0/1385066755 conn(0x7fedfc0cfd30 msgr2=0x7fedfc0cb980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:43.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 shutdown_connections 2024-01-15T18:00:43.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.539+0000 7fee02800640 1 -- 172.21.15.2:0/1385066755 wait complete. 2024-01-15T18:00:43.565 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:00:43.565 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:00:43.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.784+0000 7f7d37e00640 1 Processor -- start 2024-01-15T18:00:43.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.784+0000 7f7d37e00640 1 -- start start 2024-01-15T18:00:43.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d37e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7d38141880 0x7f7d38070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d37e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7d3815ec60 0x7f7d3815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d37e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d380cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d37e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d38071f20 con 0x7f7d38141880 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d37e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3806c6b0 con 0x7f7d3815ec60 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d37e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3815f620 con 0x7f7d380cd490 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d36a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7d38141880 0x7f7d38070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d36a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7d38141880 0x7f7d38070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58894/0 (socket says 172.21.15.2:58894) 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.786+0000 7f7d36a00640 1 -- 172.21.15.2:0/3665948053 learned_addr learned my addr 172.21.15.2:0/3665948053 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d36000640 1 --2- 172.21.15.2:0/3665948053 >> v2:172.21.15.137:3300/0 conn(0x7f7d3815ec60 0x7f7d3815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d37400640 1 --2- 172.21.15.2:0/3665948053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d380cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d36a00640 1 -- 172.21.15.2:0/3665948053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 msgr2=0x7f7d380cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:43.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d36a00640 1 --2- 172.21.15.2:0/3665948053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d380cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d36a00640 1 -- 172.21.15.2:0/3665948053 >> v2:172.21.15.137:3300/0 conn(0x7f7d3815ec60 msgr2=0x7f7d3815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:43.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d36a00640 1 --2- 172.21.15.2:0/3665948053 >> v2:172.21.15.137:3300/0 conn(0x7f7d3815ec60 0x7f7d3815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.787+0000 7f7d36a00640 1 -- 172.21.15.2:0/3665948053 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38071960 con 0x7f7d38141880 2024-01-15T18:00:43.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.789+0000 7f7d36a00640 1 --2- 172.21.15.2:0/3665948053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d38141880 0x7f7d38070fc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7d380722d0 tx=0x7f7d2802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f87216fc37b59 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.789+0000 7f7d35600640 1 -- 172.21.15.2:0/3665948053 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280317e0 con 0x7f7d38141880 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.789+0000 7f7d35600640 1 -- 172.21.15.2:0/3665948053 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d28031940 con 0x7f7d38141880 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.789+0000 7f7d35600640 1 -- 172.21.15.2:0/3665948053 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28031cd0 con 0x7f7d38141880 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 -- 172.21.15.2:0/3665948053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d38141880 msgr2=0x7f7d38070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3665948053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d38141880 0x7f7d38070fc0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f7d380722d0 tx=0x7f7d2802b880 comp rx=0 tx=0).stop 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 -- 172.21.15.2:0/3665948053 shutdown_connections 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3665948053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d380cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3665948053 >> v2:172.21.15.137:3300/0 conn(0x7f7d3815ec60 0x7f7d3815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3665948053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d38141880 0x7f7d38070fc0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 -- 172.21.15.2:0/3665948053 >> 172.21.15.2:0/3665948053 conn(0x7f7d380b0840 msgr2=0x7f7d380b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:43.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 -- 172.21.15.2:0/3665948053 shutdown_connections 2024-01-15T18:00:43.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.790+0000 7f7d37e00640 1 -- 172.21.15.2:0/3665948053 wait complete. 2024-01-15T18:00:43.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.792+0000 7f7d37e00640 1 Processor -- start 2024-01-15T18:00:43.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.792+0000 7f7d37e00640 1 -- start start 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d38142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38141880 0x7f7d38153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 0x7f7d3813cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d38156a60 con 0x7f7d3815ec60 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381427f0 con 0x7f7d38141880 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d381442f0 con 0x7f7d380cd490 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 0x7f7d3813cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 0x7f7d3813cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58902/0 (socket says 172.21.15.2:58902) 2024-01-15T18:00:43.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d37400640 1 -- 172.21.15.2:0/3157882833 learned_addr learned my addr 172.21.15.2:0/3157882833 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d36a00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d38142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.793+0000 7f7d36000640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38141880 0x7f7d38153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37400640 1 -- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 msgr2=0x7f7d38142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37400640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d38142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37400640 1 -- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38141880 msgr2=0x7f7d38153570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37400640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38141880 0x7f7d38153570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37400640 1 -- 172.21.15.2:0/3157882833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280094d0 con 0x7f7d3815ec60 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37400640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 0x7f7d3813cfa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000d5d0 tx=0x7f7d3000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30014620 con 0x7f7d3815ec60 2024-01-15T18:00:43.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d300147a0 con 0x7f7d3815ec60 2024-01-15T18:00:43.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d38153ab0 con 0x7f7d3815ec60 2024-01-15T18:00:43.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30014b50 con 0x7f7d3815ec60 2024-01-15T18:00:43.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.794+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d38142130 con 0x7f7d3815ec60 2024-01-15T18:00:43.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.796+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d38074770 con 0x7f7d3815ec60 2024-01-15T18:00:43.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.796+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7d30014cb0 con 0x7f7d3815ec60 2024-01-15T18:00:43.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.797+0000 7f7d2ea00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d1006b530 0x7f7d1006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:43.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.797+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7d3008a3a0 con 0x7f7d3815ec60 2024-01-15T18:00:43.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.798+0000 7f7d36a00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d1006b530 0x7f7d1006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:43.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.799+0000 7f7d36a00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d1006b530 0x7f7d1006d9f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d28036000 tx=0x7f7d2803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:43.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:43.802+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d30054e80 con 0x7f7d3815ec60 2024-01-15T18:00:44.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.034+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3813e5a0 con 0x7f7d3815ec60 2024-01-15T18:00:44.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.035+0000 7f7d2ea00640 1 -- 172.21.15.2:0/3157882833 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f7d300584a0 con 0x7f7d3815ec60 2024-01-15T18:00:44.040 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:44.040 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1873476051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1873476051},{"type":"v1","addr":"172.21.15.2:6835","nonce":1873476051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1702488249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1702488249},{"type":"v1","addr":"172.21.15.137:6836","nonce":1702488249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5251,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/4131491202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":4131491202},{"type":"v1","addr":"172.21.15.137:6837","nonce":4131491202}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:00:39.959212+0000","modified":"2024-01-15T18:00:43.038891+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/4162570290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":4162570290},{"type":"v1","addr":"172.21.15.2:6833","nonce":4162570290}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-15T18:00:44.040 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d1006b530 msgr2=0x7f7d1006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d1006b530 0x7f7d1006d9f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f7d28036000 tx=0x7f7d2803d040 comp rx=0 tx=0).stop 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 msgr2=0x7f7d3813cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 0x7f7d3813cfa0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f7d3000d5d0 tx=0x7f7d3000da90 comp rx=0 tx=0).stop 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 shutdown_connections 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d1006b530 0x7f7d1006d9f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d3815ec60 0x7f7d3813cfa0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d38141880 0x7f7d38153570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 --2- 172.21.15.2:0/3157882833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d380cd490 0x7f7d38142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 >> 172.21.15.2:0/3157882833 conn(0x7f7d380b0840 msgr2=0x7f7d380cc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.038+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 shutdown_connections 2024-01-15T18:00:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.039+0000 7f7d37e00640 1 -- 172.21.15.2:0/3157882833 wait complete. 2024-01-15T18:00:44.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:00:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.277+0000 7fdf82e00640 1 Processor -- start 2024-01-15T18:00:44.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.277+0000 7fdf82e00640 1 -- start start 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.279+0000 7fdf82e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf7c14af80 0x7fdf7c06d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.279+0000 7fdf82e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c0700d0 0x7fdf7c070520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.279+0000 7fdf82e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c070a60 con 0x7fdf7c06da60 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.279+0000 7fdf82e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c0b28f0 con 0x7fdf7c14af80 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.279+0000 7fdf82e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0b1c70 con 0x7fdf7c0700d0 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf81a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf7c14af80 0x7fdf7c06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf82400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c0700d0 0x7fdf7c070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf81a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf7c14af80 0x7fdf7c06d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:52780/0 (socket says 172.21.15.2:52780) 2024-01-15T18:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf81a00640 1 -- 172.21.15.2:0/815652650 learned_addr learned my addr 172.21.15.2:0/815652650 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf82400640 1 -- 172.21.15.2:0/815652650 >> v2:172.21.15.137:3300/0 conn(0x7fdf7c14af80 msgr2=0x7fdf7c06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:44.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf82400640 1 --2- 172.21.15.2:0/815652650 >> v2:172.21.15.137:3300/0 conn(0x7fdf7c14af80 0x7fdf7c06d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf7be00640 1 -- 172.21.15.2:0/815652650 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416194297 0 0) 0x7fdf64001110 con 0x7fdf7c06da60 2024-01-15T18:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf82400640 1 -- 172.21.15.2:0/815652650 >> v1:172.21.15.137:6789/0 conn(0x7fdf7c06da60 legacy=0x7fdf7c161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.280+0000 7fdf82400640 1 -- 172.21.15.2:0/815652650 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0b1f50 con 0x7fdf7c0700d0 2024-01-15T18:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.282+0000 7fdf82400640 1 --2- 172.21.15.2:0/815652650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c0700d0 0x7fdf7c070520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009d90 tx=0x7fdf7402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d171fb535e91c7f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.282+0000 7fdf7be00640 1 -- 172.21.15.2:0/815652650 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7402cbd0 con 0x7fdf7c0700d0 2024-01-15T18:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.282+0000 7fdf7be00640 1 -- 172.21.15.2:0/815652650 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7402cd30 con 0x7fdf7c0700d0 2024-01-15T18:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.282+0000 7fdf7be00640 1 -- 172.21.15.2:0/815652650 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7402cea0 con 0x7fdf7c0700d0 2024-01-15T18:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 -- 172.21.15.2:0/815652650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c0700d0 msgr2=0x7fdf7c070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 --2- 172.21.15.2:0/815652650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c0700d0 0x7fdf7c070520 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009d90 tx=0x7fdf7402be20 comp rx=0 tx=0).stop 2024-01-15T18:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 -- 172.21.15.2:0/815652650 shutdown_connections 2024-01-15T18:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 --2- 172.21.15.2:0/815652650 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c0700d0 0x7fdf7c070520 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 --2- 172.21.15.2:0/815652650 >> v2:172.21.15.137:3300/0 conn(0x7fdf7c14af80 0x7fdf7c06d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 -- 172.21.15.2:0/815652650 >> 172.21.15.2:0/815652650 conn(0x7fdf7c0b0840 msgr2=0x7fdf7c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:44.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.283+0000 7fdf82e00640 1 -- 172.21.15.2:0/815652650 shutdown_connections 2024-01-15T18:00:44.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.284+0000 7fdf82e00640 1 -- 172.21.15.2:0/815652650 wait complete. 2024-01-15T18:00:44.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.285+0000 7fdf82e00640 1 Processor -- start 2024-01-15T18:00:44.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.285+0000 7fdf82e00640 1 -- start start 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.286+0000 7fdf82e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c06da60 0x7fdf7c144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.286+0000 7fdf82e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf7c0700d0 0x7fdf7c141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.286+0000 7fdf82e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 0x7fdf7c165260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.286+0000 7fdf82e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c141dc0 con 0x7fdf7c14af80 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.286+0000 7fdf82e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c1441d0 con 0x7fdf7c0700d0 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.286+0000 7fdf82e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c1444b0 con 0x7fdf7c06da60 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 0x7fdf7c165260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 0x7fdf7c165260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58906/0 (socket says 172.21.15.2:58906) 2024-01-15T18:00:44.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 -- 172.21.15.2:0/3965177057 learned_addr learned my addr 172.21.15.2:0/3965177057 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf81000640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf7c0700d0 0x7fdf7c141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf81a00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c06da60 0x7fdf7c144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 -- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c06da60 msgr2=0x7fdf7c144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c06da60 0x7fdf7c144cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 -- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf7c0700d0 msgr2=0x7fdf7c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf7c0700d0 0x7fdf7c141450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf82400640 1 -- 172.21.15.2:0/3965177057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf74009a70 con 0x7fdf7c14af80 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.287+0000 7fdf81000640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf7c0700d0 0x7fdf7c141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.288+0000 7fdf82400640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 0x7fdf7c165260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009d90 tx=0x7fdf74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:44.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.288+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7402cea0 con 0x7fdf7c14af80 2024-01-15T18:00:44.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.288+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c144790 con 0x7fdf7c14af80 2024-01-15T18:00:44.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.288+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf74040650 con 0x7fdf7c14af80 2024-01-15T18:00:44.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.288+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74040a00 con 0x7fdf7c14af80 2024-01-15T18:00:44.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.288+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c1420a0 con 0x7fdf7c14af80 2024-01-15T18:00:44.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.289+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c074770 con 0x7fdf7c14af80 2024-01-15T18:00:44.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.290+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fdf74040b60 con 0x7fdf7c14af80 2024-01-15T18:00:44.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.291+0000 7fdf79600640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5406b500 0x7fdf5406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.291+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdf740acd30 con 0x7fdf7c14af80 2024-01-15T18:00:44.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.292+0000 7fdf81a00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5406b500 0x7fdf5406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.293+0000 7fdf81a00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5406b500 0x7fdf5406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009b60 tx=0x7fdf6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:44.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.295+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf74077790 con 0x7fdf7c14af80 2024-01-15T18:00:44.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.490+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf7c0b04e0 con 0x7fdf7c14af80 2024-01-15T18:00:44.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.492+0000 7fdf79600640 1 -- 172.21.15.2:0/3965177057 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16496 (secure 0 0 0) 0x7fdf74009450 con 0x7fdf7c14af80 2024-01-15T18:00:44.497 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:44.497 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":47,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:00:41.487021+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:00:37.830306+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-15T18:00:38.982411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:00:44.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.494+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5406b500 msgr2=0x7fdf5406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5406b500 0x7fdf5406d9c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009b60 tx=0x7fdf6c008040 comp rx=0 tx=0).stop 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 msgr2=0x7fdf7c165260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 0x7fdf7c165260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fdf74009d90 tx=0x7fdf74004a40 comp rx=0 tx=0).stop 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 shutdown_connections 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf5406b500 0x7fdf5406d9c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf7c14af80 0x7fdf7c165260 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf7c0700d0 0x7fdf7c141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 --2- 172.21.15.2:0/3965177057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf7c06da60 0x7fdf7c144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 >> 172.21.15.2:0/3965177057 conn(0x7fdf7c0b0840 msgr2=0x7fdf7c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:44.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 shutdown_connections 2024-01-15T18:00:44.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.495+0000 7fdf82e00640 1 -- 172.21.15.2:0/3965177057 wait complete. 2024-01-15T18:00:44.518 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:00:44.519 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:00:44.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.735+0000 7effcbe00640 1 Processor -- start 2024-01-15T18:00:44.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.735+0000 7effcbe00640 1 -- start start 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.736+0000 7effcbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 0x7effcc0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.736+0000 7effcbe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7effcc0d1d90 0x7effcc0d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.736+0000 7effcbe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effcc0d3a30 con 0x7effcc0d3090 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.736+0000 7effcbe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effcc0b3de0 con 0x7effcc0d1d90 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.736+0000 7effcbe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0b40c0 con 0x7effcc15d340 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.737+0000 7effca000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7effcc0d3090 0x7effcc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55948/0 (socket says 172.21.15.2:55948) 2024-01-15T18:00:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.737+0000 7effca000640 1 -- 172.21.15.2:0/2265007334 learned_addr learned my addr 172.21.15.2:0/2265007334 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.737+0000 7effcb400640 1 --2- 172.21.15.2:0/2265007334 >> v2:172.21.15.137:3300/0 conn(0x7effcc0d1d90 0x7effcc0d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.737+0000 7effcaa00640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 0x7effcc0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.737+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885872150 0 0) 0x7effcc0d3a30 con 0x7effcc0d3090 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.737+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effac003520 con 0x7effcc0d3090 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3354191164 0 0) 0x7effac003520 con 0x7effcc0d3090 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 msgr2=0x7effcc0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 0x7effcc0cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 >> v2:172.21.15.137:3300/0 conn(0x7effcc0d1d90 msgr2=0x7effcc0d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 --2- 172.21.15.2:0/2265007334 >> v2:172.21.15.137:3300/0 conn(0x7effcc0d1d90 0x7effcc0d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc0d0e80 con 0x7effcc0d3090 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7effb4001670 con 0x7effcc0d3090 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 >> v1:172.21.15.2:6789/0 conn(0x7effcc0d3090 legacy=0x7effcc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 0x7effac003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effac003f40 0x7effac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effac004d00 0x7effac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0b3de0 con 0x7effac004d00 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0b40c0 con 0x7effac003f40 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac00b980 con 0x7effcc15d340 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effcb400640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effac004d00 0x7effac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effcaa00640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 0x7effac003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.738+0000 7effca000640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effac003f40 0x7effac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.739+0000 7effcb400640 1 -- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 msgr2=0x7effac003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.739+0000 7effcb400640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effcc15d340 0x7effac003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.739+0000 7effcb400640 1 -- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effac003f40 msgr2=0x7effac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.739+0000 7effcb400640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effac003f40 0x7effac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.739+0000 7effcb400640 1 -- 172.21.15.2:0/2265007334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac003520 con 0x7effac004d00 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.739+0000 7effca000640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effac003f40 0x7effac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effcb400640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effac004d00 0x7effac005180 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7effc400e170 tx=0x7effac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcd205b1cda32e73 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc402cc20 con 0x7effac004d00 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc400f040 con 0x7effac004d00 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effc9600640 1 -- 172.21.15.2:0/2265007334 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4034440 con 0x7effac004d00 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effcbe00640 1 -- 172.21.15.2:0/2265007334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effac004d00 msgr2=0x7effac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effcbe00640 1 --2- 172.21.15.2:0/2265007334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effac004d00 0x7effac005180 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7effc400e170 tx=0x7effac0057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.740+0000 7effcb400640 1 -- 172.21.15.2:0/2265007334 reap_dead start 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.741+0000 7effcbe00640 1 -- 172.21.15.2:0/2265007334 shutdown_connections 2024-01-15T18:00:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.741+0000 7effcbe00640 1 -- 172.21.15.2:0/2265007334 >> 172.21.15.2:0/2265007334 conn(0x7effcc08caf0 msgr2=0x7effcc09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:44.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.741+0000 7effcbe00640 1 -- 172.21.15.2:0/2265007334 shutdown_connections 2024-01-15T18:00:44.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.741+0000 7effcbe00640 1 -- 172.21.15.2:0/2265007334 wait complete. 2024-01-15T18:00:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.742+0000 7effcbe00640 1 Processor -- start 2024-01-15T18:00:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.743+0000 7effcbe00640 1 -- start start 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.743+0000 7effcbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effac004d00 0x7effcc14d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.743+0000 7effcbe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effcc1577b0 0x7effcc160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.743+0000 7effcbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 0x7effcc1f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcbe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc160840 con 0x7effcc1f33a0 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcbe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0cb6d0 con 0x7effcc1577b0 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcbe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc0cb9b0 con 0x7effac004d00 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 0x7effcc1f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 0x7effcc1f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58920/0 (socket says 172.21.15.2:58920) 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 -- 172.21.15.2:0/4015735569 learned_addr learned my addr 172.21.15.2:0/4015735569 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effca000640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effcc1577b0 0x7effcc160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcaa00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effac004d00 0x7effcc14d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 -- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effac004d00 msgr2=0x7effcc14d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effac004d00 0x7effcc14d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 -- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effcc1577b0 msgr2=0x7effcc160300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effcc1577b0 0x7effcc160300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcb400640 1 -- 172.21.15.2:0/4015735569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc0cbc90 con 0x7effcc1f33a0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.744+0000 7effcaa00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effac004d00 0x7effcc14d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.745+0000 7effcb400640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 0x7effcc1f3830 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7effc400e170 tx=0x7effc4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.745+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4043070 con 0x7effcc1f33a0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.745+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc402de90 con 0x7effcc1f33a0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.745+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4034d90 con 0x7effcc1f33a0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.745+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effcc0cbf70 con 0x7effcc1f33a0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.745+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effcc0cb2f0 con 0x7effcc1f33a0 2024-01-15T18:00:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.746+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effcc0cbaf0 con 0x7effcc1f33a0 2024-01-15T18:00:44.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.747+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7effc403e440 con 0x7effcc1f33a0 2024-01-15T18:00:44.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.748+0000 7effc2a00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effa406b500 0x7effa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:44.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.749+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7effc40abcc0 con 0x7effcc1f33a0 2024-01-15T18:00:44.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.749+0000 7effcaa00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effa406b500 0x7effa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:44.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.750+0000 7effcaa00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effa406b500 0x7effa406d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7effbc009a00 tx=0x7effbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:44.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.753+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effcc0cbaf0 con 0x7effcc1f33a0 2024-01-15T18:00:44.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.947+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7effcc0cdd20 con 0x7effa406b500 2024-01-15T18:00:44.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.974+0000 7effc2a00640 1 -- 172.21.15.2:0/4015735569 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274764 (secure 0 0 0) 0x7effbc012950 con 0x7effa406b500 2024-01-15T18:00:44.983 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:44.984 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effa406b500 msgr2=0x7effa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effa406b500 0x7effa406d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7effbc009a00 tx=0x7effbc008040 comp rx=0 tx=0).stop 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 msgr2=0x7effcc1f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 0x7effcc1f3830 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7effc400e170 tx=0x7effc4009750 comp rx=0 tx=0).stop 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 shutdown_connections 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effa406b500 0x7effa406d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effcc1f33a0 0x7effcc1f3830 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effcc1577b0 0x7effcc160300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 --2- 172.21.15.2:0/4015735569 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effac004d00 0x7effcc14d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 >> 172.21.15.2:0/4015735569 conn(0x7effcc08caf0 msgr2=0x7effcc073310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 shutdown_connections 2024-01-15T18:00:44.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:44.980+0000 7effcbe00640 1 -- 172.21.15.2:0/4015735569 wait complete. 2024-01-15T18:00:45.005 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-01-15T18:00:43.440521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":262920,"kb_used_data":5160,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754711800,"statfs":{"total":773094113280,"available":772824883200,"internally_reserved":0,"allocated":5283840,"data_stored":2820272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":263835185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.969529"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478890+0000","last_change":"2024-01-15T18:00:38.478890+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478890+0000","last_undegraded":"2024-01-15T18:00:38.478890+0000","last_fullsized":"2024-01-15T18:00:38.478890+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.475942+0000","last_change":"2024-01-15T18:00:38.475942+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.475942+0000","last_undegraded":"2024-01-15T18:00:38.475942+0000","last_fullsized":"2024-01-15T18:00:38.475942+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.480205+0000","last_change":"2024-01-15T18:00:38.480205+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.480205+0000","last_undegraded":"2024-01-15T18:00:38.480205+0000","last_fullsized":"2024-01-15T18:00:38.480205+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483872+0000","last_change":"2024-01-15T18:00:38.483872+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483872+0000","last_undegraded":"2024-01-15T18:00:38.483872+0000","last_fullsized":"2024-01-15T18:00:38.483872+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.479327+0000","last_change":"2024-01-15T18:00:38.479327+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.479327+0000","last_undegraded":"2024-01-15T18:00:38.479327+0000","last_fullsized":"2024-01-15T18:00:38.479327+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.476208+0000","last_change":"2024-01-15T18:00:38.476208+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.476208+0000","last_undegraded":"2024-01-15T18:00:38.476208+0000","last_fullsized":"2024-01-15T18:00:38.476208+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.476983+0000","last_change":"2024-01-15T18:00:38.476983+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.476983+0000","last_undegraded":"2024-01-15T18:00:38.476983+0000","last_fullsized":"2024-01-15T18:00:38.476983+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.486432+0000","last_change":"2024-01-15T18:00:38.486432+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.486432+0000","last_undegraded":"2024-01-15T18:00:38.486432+0000","last_fullsized":"2024-01-15T18:00:38.486432+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.474716+0000","last_change":"2024-01-15T18:00:38.474716+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.474716+0000","last_undegraded":"2024-01-15T18:00:38.474716+0000","last_fullsized":"2024-01-15T18:00:38.474716+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483465+0000","last_change":"2024-01-15T18:00:38.483465+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483465+0000","last_undegraded":"2024-01-15T18:00:38.483465+0000","last_fullsized":"2024-01-15T18:00:38.483465+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.484623+0000","last_change":"2024-01-15T18:00:38.484623+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.484623+0000","last_undegraded":"2024-01-15T18:00:38.484623+0000","last_fullsized":"2024-01-15T18:00:38.484623+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.474435+0000","last_change":"2024-01-15T18:00:38.474435+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.474435+0000","last_undegraded":"2024-01-15T18:00:38.474435+0000","last_fullsized":"2024-01-15T18:00:38.474435+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478865+0000","last_change":"2024-01-15T18:00:38.478865+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478865+0000","last_undegraded":"2024-01-15T18:00:38.478865+0000","last_fullsized":"2024-01-15T18:00:38.478865+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483660+0000","last_change":"2024-01-15T18:00:38.483660+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483660+0000","last_undegraded":"2024-01-15T18:00:38.483660+0000","last_fullsized":"2024-01-15T18:00:38.483660+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.477627+0000","last_change":"2024-01-15T18:00:38.477627+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.477627+0000","last_undegraded":"2024-01-15T18:00:38.477627+0000","last_fullsized":"2024-01-15T18:00:38.477627+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.484599+0000","last_change":"2024-01-15T18:00:38.484599+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.484599+0000","last_undegraded":"2024-01-15T18:00:38.484599+0000","last_fullsized":"2024-01-15T18:00:38.484599+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.474349+0000","last_change":"2024-01-15T18:00:38.474349+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.474349+0000","last_undegraded":"2024-01-15T18:00:38.474349+0000","last_fullsized":"2024-01-15T18:00:38.474349+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.474719+0000","last_change":"2024-01-15T18:00:38.474719+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.474719+0000","last_undegraded":"2024-01-15T18:00:38.474719+0000","last_fullsized":"2024-01-15T18:00:38.474719+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.476423+0000","last_change":"2024-01-15T18:00:38.476423+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.476423+0000","last_undegraded":"2024-01-15T18:00:38.476423+0000","last_fullsized":"2024-01-15T18:00:38.476423+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478164+0000","last_change":"2024-01-15T18:00:38.478164+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478164+0000","last_undegraded":"2024-01-15T18:00:38.478164+0000","last_fullsized":"2024-01-15T18:00:38.478164+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478215+0000","last_change":"2024-01-15T18:00:38.478215+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478215+0000","last_undegraded":"2024-01-15T18:00:38.478215+0000","last_fullsized":"2024-01-15T18:00:38.478215+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.477710+0000","last_change":"2024-01-15T18:00:38.477710+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.477710+0000","last_undegraded":"2024-01-15T18:00:38.477710+0000","last_fullsized":"2024-01-15T18:00:38.477710+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.482993+0000","last_change":"2024-01-15T18:00:38.482993+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.482993+0000","last_undegraded":"2024-01-15T18:00:38.482993+0000","last_fullsized":"2024-01-15T18:00:38.482993+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483447+0000","last_change":"2024-01-15T18:00:38.483447+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483447+0000","last_undegraded":"2024-01-15T18:00:38.483447+0000","last_fullsized":"2024-01-15T18:00:38.483447+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478042+0000","last_change":"2024-01-15T18:00:38.478042+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478042+0000","last_undegraded":"2024-01-15T18:00:38.478042+0000","last_fullsized":"2024-01-15T18:00:38.478042+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483820+0000","last_change":"2024-01-15T18:00:38.483820+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483820+0000","last_undegraded":"2024-01-15T18:00:38.483820+0000","last_fullsized":"2024-01-15T18:00:38.483820+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478244+0000","last_change":"2024-01-15T18:00:38.478244+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478244+0000","last_undegraded":"2024-01-15T18:00:38.478244+0000","last_fullsized":"2024-01-15T18:00:38.478244+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483708+0000","last_change":"2024-01-15T18:00:38.483708+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483708+0000","last_undegraded":"2024-01-15T18:00:38.483708+0000","last_fullsized":"2024-01-15T18:00:38.483708+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.474704+0000","last_change":"2024-01-15T18:00:38.474704+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.474704+0000","last_undegraded":"2024-01-15T18:00:38.474704+0000","last_fullsized":"2024-01-15T18:00:38.474704+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.478042+0000","last_change":"2024-01-15T18:00:38.478042+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.478042+0000","last_undegraded":"2024-01-15T18:00:38.478042+0000","last_fullsized":"2024-01-15T18:00:38.478042+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.483461+0000","last_change":"2024-01-15T18:00:38.483461+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.483461+0000","last_undegraded":"2024-01-15T18:00:38.483461+0000","last_fullsized":"2024-01-15T18:00:38.483461+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:00:39.469222+0000","last_change":"2024-01-15T18:00:39.469222+0000","last_active":"2024-01-15T18:00:39.469222+0000","last_peered":"2024-01-15T18:00:39.469222+0000","last_clean":"2024-01-15T18:00:39.469222+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:39.469222+0000","last_undegraded":"2024-01-15T18:00:39.469222+0000","last_fullsized":"2024-01-15T18:00:39.469222+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1,"reported_epoch":44,"state":"creating+peering","last_fresh":"2024-01-15T18:00:38.477152+0000","last_change":"2024-01-15T18:00:38.477152+0000","last_active":"2024-01-15T18:00:38.453564+0000","last_peered":"2024-01-15T18:00:38.453564+0000","last_clean":"2024-01-15T18:00:38.453564+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:00:38.477152+0000","last_undegraded":"2024-01-15T18:00:38.477152+0000","last_fullsized":"2024-01-15T18:00:38.477152+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":184,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-01-15T18:00:36.631330+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:00:36.631330+0000","last_peered":"2024-01-15T18:00:36.631330+0000","last_clean":"2024-01-15T18:00:36.631330+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:00:36.631330+0000","last_undegraded":"2024-01-15T18:00:36.631330+0000","last_fullsized":"2024-01-15T18:00:36.631330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1105920,"data_stored":795310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0009999999999999,"5min":1.2969999999999999,"15min":1.161},"min":{"1min":0.72599999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":12.468999999999999,"5min":12.468999999999999,"15min":12.468999999999999},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.486,"15min":1.454},"min":{"1min":0.78000000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":12.496,"5min":12.496,"15min":12.496},"last":1.2649999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0369999999999999,"5min":1.3,"15min":1.1659999999999999},"min":{"1min":0.93899999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":14.497,"5min":14.497,"15min":14.497},"last":1.29},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.1759999999999999,"15min":1.1299999999999999},"min":{"1min":0.70099999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":4.2190000000000003,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.518}]},{"osd":2,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.208,"15min":1.1399999999999999},"min":{"1min":0.82299999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.3179999999999996,"5min":4.3179999999999996,"15min":4.3179999999999996},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.377,"15min":1.4450000000000001},"min":{"1min":0.72099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6059999999999999,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.0660000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":2.028,"5min":1.4610000000000001,"15min":1.397},"min":{"1min":0.90800000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":14.622,"5min":14.622,"15min":14.622},"last":1.117},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.379,"15min":1.347},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":4.2690000000000001,"5min":4.2690000000000001,"15min":4.2690000000000001},"last":1.6220000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.4810000000000001,"15min":1.577},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.794,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":0.95199999999999996},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.5309999999999999,"15min":1.4379999999999999},"min":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":1.1990000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.51,"15min":1.5289999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":4.3620000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.389},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.5720000000000001,"15min":1.6619999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.0169999999999999,"5min":7.798,"15min":7.798},"last":1.167}]},{"osd":6,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6870000000000001,"15min":1.7470000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":3.6800000000000002,"5min":7.968,"15min":7.968},"last":1.3440000000000001},{"interface":"front","average":{"1min":2.1850000000000001,"5min":1.6080000000000001,"15min":1.516},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":14.804,"5min":14.804,"15min":14.804},"last":1.0169999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":204942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.109,"15min":1.071},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5740000000000001,"5min":5.0439999999999996,"15min":5.0439999999999996},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.1259999999999999,"15min":1.125},"min":{"1min":0.68899999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":4.6210000000000004,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":0.76800000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2390000000000001,"15min":1.226},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8200000000000001,"5min":1.972,"15min":1.972},"last":1.647},{"interface":"front","average":{"1min":1.137,"5min":1.2809999999999999,"15min":1.3260000000000001},"min":{"1min":0.72499999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.7230000000000001,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":0.92900000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2609999999999999,"15min":1.2709999999999999},"min":{"1min":0.86099999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.669,"5min":3.415,"15min":3.415},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.294,"15min":1.276},"min":{"1min":1.04,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.413}]},{"osd":3,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.381,"15min":1.359},"min":{"1min":0.875,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0099999999999998,"5min":4.6559999999999997,"15min":4.6559999999999997},"last":1.226},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.413,"15min":1.3919999999999999},"min":{"1min":0.92600000000000005,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.992,"5min":4.7199999999999998,"15min":4.7199999999999998},"last":1.133}]},{"osd":4,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.329,"15min":1.337},"min":{"1min":0.77800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.859,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.498},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.4370000000000001,"15min":1.4870000000000001},"min":{"1min":0.56399999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.8500000000000001,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.5449999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.335,"15min":1.357},"min":{"1min":0.80300000000000005,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.881,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.042},{"interface":"front","average":{"1min":1.536,"5min":1.5049999999999999,"15min":1.498},"min":{"1min":0.90200000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.925,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.5820000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.607,"15min":1.6739999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.9339999999999999,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.456},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5880000000000001,"15min":1.6499999999999999},"min":{"1min":1.02,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9790000000000001,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.383}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":204942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.143,"15min":1.139},"min":{"1min":0.79100000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5760000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.675,"5min":1.347,"15min":1.264},"min":{"1min":0.66700000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.75900000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2130000000000001,"15min":1.153},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.6440000000000001,"5min":4.806,"15min":4.806},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.179,"15min":1.1180000000000001},"min":{"1min":0.72699999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6910000000000001,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":1.4850000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3580000000000001,"15min":1.327},"min":{"1min":1.0620000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.4220000000000002,"5min":5.516,"15min":5.516},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.4430000000000001,"15min":1.383},"min":{"1min":0.81499999999999995,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":7.1340000000000003,"5min":7.1340000000000003,"15min":7.1340000000000003},"last":1.0800000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.274,"15min":1.252},"min":{"1min":0.83599999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.4890000000000001,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.82199999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.999}]},{"osd":4,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.3580000000000001,"15min":1.3120000000000001},"min":{"1min":0.83199999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":7.2439999999999998,"5min":7.2439999999999998,"15min":7.2439999999999998},"last":1.385},{"interface":"front","average":{"1min":1.748,"5min":1.5429999999999999,"15min":1.4870000000000001},"min":{"1min":1.008,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.399,"5min":7.399,"15min":7.399},"last":1.258}]},{"osd":6,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3280000000000001,"15min":1.371},"min":{"1min":0.84199999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5940000000000001,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.319},{"interface":"front","average":{"1min":1.361,"5min":1.472,"15min":1.488},"min":{"1min":0.91600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.7849999999999999,"5min":4.9699999999999998,"15min":4.9699999999999998},"last":0.90600000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.6759999999999999,"15min":1.6279999999999999},"min":{"1min":1.167,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.546,"15min":1.552},"min":{"1min":1.161,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.8100000000000001,"5min":5.0910000000000002,"15min":5.0910000000000002},"last":1.419}]}]},{"osd":4,"up_from":14,"seq":60129542186,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":204942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":2.9700000000000002,"5min":1.458,"15min":1.2010000000000001},"min":{"1min":0.84299999999999997,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":21.643000000000001,"5min":21.643000000000001,"15min":21.643000000000001},"last":1.423},{"interface":"front","average":{"1min":1.3,"5min":1.1399999999999999,"15min":1.1140000000000001},"min":{"1min":0.78600000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":1.042}]},{"osd":1,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.208,"15min":1.1699999999999999},"min":{"1min":0.72899999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.2300000000000004,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.224},{"interface":"front","average":{"1min":3.052,"5min":1.5940000000000001,"15min":1.3240000000000001},"min":{"1min":0.99299999999999999,"5min":0.754,"15min":0.754},"max":{"1min":21.440000000000001,"5min":21.440000000000001,"15min":21.440000000000001},"last":1.3620000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.216,"15min":1.1559999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":1.399},{"interface":"front","average":{"1min":3.0289999999999999,"5min":1.6399999999999999,"15min":1.3939999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":22.350999999999999,"5min":22.350999999999999,"15min":22.350999999999999},"last":1.0589999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.4910000000000001,"15min":1.377},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":5.2169999999999996,"5min":14.657999999999999,"15min":14.657999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.631,"5min":1.4159999999999999,"15min":1.3480000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":4.4720000000000004,"5min":5.1879999999999997,"15min":5.1879999999999997},"last":1.448}]},{"osd":5,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":1.7649999999999999,"15min":1.5029999999999999},"min":{"1min":0.90800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":21.285,"5min":21.285,"15min":21.285},"last":1.47},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.357,"15min":1.385},"min":{"1min":0.64400000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":0.88400000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.399,"15min":1.399},"min":{"1min":0.61799999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0979999999999999,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.097},{"interface":"front","average":{"1min":3.1960000000000002,"5min":1.8380000000000001,"15min":1.6100000000000001},"min":{"1min":0.82199999999999995,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":21.404,"5min":21.404,"15min":21.404},"last":1.266}]},{"osd":7,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5,"15min":1.518},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.597,"5min":2.597,"15min":2.597},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.734,"5min":1.661,"15min":1.5569999999999999},"min":{"1min":0.88500000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.391,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":1.325}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35840,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336000,"statfs":{"total":96636764160,"available":96600064000,"internally_reserved":0,"allocated":512000,"data_stored":204942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":0.89800000000000002,"15min":0.81599999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.9169999999999998,"5min":4.9169999999999998,"15min":4.9169999999999998},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.88300000000000001,"15min":0.878},"min":{"1min":0.42499999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.383,"5min":1.512,"15min":1.512},"last":1.387}]},{"osd":1,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":0.995,"15min":0.995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.792},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.1519999999999999,"15min":1.1259999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.097,"15min":1.075},"min":{"1min":0.58699999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.962,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.653},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.2569999999999999,"15min":1.2150000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":5.0910000000000002,"5min":5.0910000000000002,"15min":5.0910000000000002},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2509999999999999,"15min":1.2549999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":1.458},{"interface":"front","average":{"1min":1.7869999999999999,"5min":1.4430000000000001,"15min":1.3620000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":5.6310000000000002,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.917}]},{"osd":5,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.4079999999999999,"15min":1.3520000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.718,"5min":2.718,"15min":2.718},"last":1.73},{"interface":"front","average":{"1min":4.1989999999999998,"5min":1.9890000000000001,"15min":1.6040000000000001},"min":{"1min":0.746,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":40.384999999999998,"5min":40.384999999999998,"15min":40.384999999999998},"last":1.194}]},{"osd":6,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.3380000000000001,"15min":1.298},"min":{"1min":0.90900000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.74,"5min":1.5589999999999999,"15min":1.5209999999999999},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"last":1.8520000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.623,"15min":1.5920000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":4.9950000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.476,"15min":1.4259999999999999},"min":{"1min":1.054,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":4.6600000000000001,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":1.601}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":204942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.1160000000000001},"min":{"1min":0.63100000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.2480000000000002,"5min":3.2480000000000002,"15min":3.2480000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.101,"5min":1.1359999999999999,"15min":1.21},"min":{"1min":0.68200000000000005,"5min":0.313,"15min":0.313},"max":{"1min":1.998,"5min":2.343,"15min":2.343},"last":0.59099999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.0429999999999999,"15min":1.0069999999999999},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.499},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.254},"min":{"1min":0.78100000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8540000000000001,"5min":2.198,"15min":2.198},"last":0.63400000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.214,"15min":1.1870000000000001},"min":{"1min":0.746,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":4.1349999999999998,"5min":4.1349999999999998,"15min":4.1349999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.1719999999999999,"15min":1.147},"min":{"1min":0.72399999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.379,"5min":3.379,"15min":3.379},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.786,"5min":1.5620000000000001,"15min":1.573},"min":{"1min":1.0840000000000001,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":7.0640000000000001,"5min":7.0640000000000001,"15min":7.0640000000000001},"last":1.123},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4019999999999999,"15min":1.4199999999999999},"min":{"1min":1.0960000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.089,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.95399999999999996}]},{"osd":5,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.5009999999999999,"15min":1.4730000000000001},"min":{"1min":0.92700000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":4.2850000000000001,"5min":4.2850000000000001,"15min":4.2850000000000001},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.534,"15min":1.5489999999999999},"min":{"1min":0.84399999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":3.5430000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.1699999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.6180000000000001,"15min":1.6559999999999999},"min":{"1min":1.0089999999999999,"5min":0.87,"15min":0.87},"max":{"1min":2.827,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.5189999999999999,"15min":1.4850000000000001},"min":{"1min":0.86899999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.4210000000000003,"5min":4.4210000000000003,"15min":4.4210000000000003},"last":0.90700000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.633,"15min":1.6659999999999999},"min":{"1min":0.94499999999999995,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.214,"5min":2.613,"15min":2.613},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.8879999999999999,"5min":1.631,"15min":1.593},"min":{"1min":1.181,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":4.9349999999999996,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":1.0780000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35840,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94336000,"statfs":{"total":96636764160,"available":96600064000,"internally_reserved":0,"allocated":512000,"data_stored":204942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.90000000000000002,"15min":0.88400000000000001},"min":{"1min":0.53000000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8919999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.165,"5min":1.101,"15min":1.077},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.782,"5min":3.3159999999999998,"15min":3.3159999999999998},"last":0.61699999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.107,"15min":0.97599999999999998},"min":{"1min":0.39200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":7.6479999999999997,"5min":7.6479999999999997,"15min":7.6479999999999997},"last":0.873},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0600000000000001,"15min":1.0680000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.6259999999999999,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.339,"15min":1.264},"min":{"1min":0.53300000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":7.8659999999999997,"5min":7.8659999999999997,"15min":7.8659999999999997},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.361,"15min":1.262},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.226,"5min":14.071,"15min":14.071},"last":0.83099999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.397,"15min":1.399},"min":{"1min":0.80800000000000005,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1030000000000002,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.833,"5min":1.55,"15min":1.508},"min":{"1min":0.81899999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":11.516,"5min":11.516,"15min":11.516},"last":1.1839999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.5660000000000001,"15min":1.5489999999999999},"min":{"1min":0.86199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"last":1.012},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5049999999999999,"15min":1.427},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":8.0730000000000004,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":1.2729999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.889,"5min":1.5700000000000001,"15min":1.52},"min":{"1min":0.78600000000000003,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":8.0670000000000002,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":1.403},{"interface":"front","average":{"1min":1.454,"5min":1.5800000000000001,"15min":1.5880000000000001},"min":{"1min":0.88900000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.5110000000000001,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":1.113}]},{"osd":7,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.5900000000000001,"15min":1.52},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.9980000000000002,"5min":7.9980000000000002,"15min":7.9980000000000002},"last":1.3320000000000001},{"interface":"front","average":{"1min":2.1389999999999998,"5min":1.679,"15min":1.5980000000000001},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"last":1.512}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32260,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339580,"statfs":{"total":96636764160,"available":96603729920,"internally_reserved":0,"allocated":1105920,"data_stored":795310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.85599999999999998,"15min":0.82299999999999995},"min":{"1min":0.55800000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.391,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.016,"15min":1.026},"min":{"1min":0.45900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.0389999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.69199999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.663,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.278,"15min":1.1870000000000001},"min":{"1min":0.59699999999999998,"5min":0.502,"15min":0.502},"max":{"1min":10.942,"5min":10.942,"15min":10.942},"last":0.98799999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.054,"15min":1.036},"min":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":1.407},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.25,"15min":1.28},"min":{"1min":0.66300000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.736,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.4690000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.405,"15min":1.395},"min":{"1min":0.80400000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":1.258},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2949999999999999,"15min":1.323},"min":{"1min":0.82199999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.526}]},{"osd":5,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3480000000000001,"15min":1.387},"min":{"1min":0.80300000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.452,"15min":1.456},"min":{"1min":1.073,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.72}]},{"osd":6,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.46,"15min":1.468},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":1.9039999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.514,"5min":1.4970000000000001,"15min":1.504},"min":{"1min":0.95299999999999996,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.9780000000000002,"5min":2.9780000000000002,"15min":2.9780000000000002},"last":1.5700000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.556,"15min":1.579},"min":{"1min":0.89500000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":3.1240000000000001,"5min":3.1240000000000001,"15min":3.1240000000000001},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.4359999999999999,"15min":1.4490000000000001},"min":{"1min":0.95799999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.7789999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:00:45.010 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:00:45.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.274+0000 7fb2d4a00640 1 Processor -- start 2024-01-15T18:00:45.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.274+0000 7fb2d4a00640 1 -- start start 2024-01-15T18:00:45.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.275+0000 7fb2d4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2cc0d2f10 0x7fb2cc0d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.275+0000 7fb2d4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb2cc15d670 0x7fb2cc0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.275+0000 7fb2d4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2cc0ce930 con 0x7fb2cc0ce2b0 2024-01-15T18:00:45.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.275+0000 7fb2d4a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0ce740 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.275+0000 7fb2d4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2cc0cec10 con 0x7fb2cc15d670 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2cc0d2f10 0x7fb2cc0d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2cc0d2f10 0x7fb2cc0d32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58924/0 (socket says 172.21.15.2:58924) 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 -- 172.21.15.2:0/1836329330 learned_addr learned my addr 172.21.15.2:0/1836329330 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d1600640 1 --2- 172.21.15.2:0/1836329330 >> v2:172.21.15.137:3300/0 conn(0x7fb2cc15d670 0x7fb2cc0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 -- 172.21.15.2:0/1836329330 >> v2:172.21.15.137:3300/0 conn(0x7fb2cc15d670 msgr2=0x7fb2cc0cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 --2- 172.21.15.2:0/1836329330 >> v2:172.21.15.137:3300/0 conn(0x7fb2cc15d670 0x7fb2cc0cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 -- 172.21.15.2:0/1836329330 >> v1:172.21.15.137:6789/0 conn(0x7fb2cc0ce2b0 legacy=0x7fb2cc0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:45.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.276+0000 7fb2d2000640 1 -- 172.21.15.2:0/1836329330 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2cc0ceef0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.278+0000 7fb2d2000640 1 --2- 172.21.15.2:0/1836329330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc0d32f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0d4030 tx=0x7fb2b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47280cf690373257 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:45.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.278+0000 7fb2d0c00640 1 -- 172.21.15.2:0/1836329330 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4031640 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.278+0000 7fb2d0c00640 1 -- 172.21.15.2:0/1836329330 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b40317a0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.278+0000 7fb2d0c00640 1 -- 172.21.15.2:0/1836329330 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b4031b30 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1836329330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 msgr2=0x7fb2cc0d32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1836329330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc0d32f0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc0d4030 tx=0x7fb2b402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1836329330 shutdown_connections 2024-01-15T18:00:45.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1836329330 >> v2:172.21.15.137:3300/0 conn(0x7fb2cc15d670 0x7fb2cc0cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1836329330 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc0d32f0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1836329330 >> 172.21.15.2:0/1836329330 conn(0x7fb2cc08cc30 msgr2=0x7fb2cc09c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:45.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1836329330 shutdown_connections 2024-01-15T18:00:45.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.279+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1836329330 wait complete. 2024-01-15T18:00:45.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.282+0000 7fb2d4a00640 1 Processor -- start 2024-01-15T18:00:45.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.282+0000 7fb2d4a00640 1 -- start start 2024-01-15T18:00:45.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.283+0000 7fb2d4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2cc0ce2b0 0x7fb2cc0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.283+0000 7fb2d4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc1f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.283+0000 7fb2d4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2cc0cb130 0x7fb2cc1f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.283+0000 7fb2d4a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc0cd2c0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.283+0000 7fb2d4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc159c00 con 0x7fb2cc0cb130 2024-01-15T18:00:45.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.283+0000 7fb2d4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2cc152fa0 con 0x7fb2cc0ce2b0 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc1f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc1f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58934/0 (socket says 172.21.15.2:58934) 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 -- 172.21.15.2:0/1587379007 learned_addr learned my addr 172.21.15.2:0/1587379007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d2000640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2cc0ce2b0 0x7fb2cc0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 -- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2cc0ce2b0 msgr2=0x7fb2cc0ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2cc0ce2b0 0x7fb2cc0ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 -- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2cc0cb130 msgr2=0x7fb2cc1f3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2cc0cb130 0x7fb2cc1f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 -- 172.21.15.2:0/1587379007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b40094d0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.284+0000 7fb2d1600640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc1f2f40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00d800 tx=0x7fb2bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.285+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00b660 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.285+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2bc00b7c0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.285+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2cc1f3b90 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.285+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2bc00bb50 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.285+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2cc0b6f70 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.286+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2cc0686f0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.287+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb2bc00bcb0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.288+0000 7fb2ca000640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2a406b500 0x7fb2a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.288+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb2bc08a5d0 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.289+0000 7fb2d2000640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2a406b500 0x7fb2a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.290+0000 7fb2d2000640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2a406b500 0x7fb2a406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4036000 tx=0x7fb2b403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:45.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.292+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2bc055020 con 0x7fb2cc0d2f10 2024-01-15T18:00:45.454 INFO:tasks.ceph.mds.c.smithi002.stderr:2024-01-15T18:00:45.450+0000 7fab1aa00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:00:45.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.486+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2cc0d1070 con 0x7fb2a406b500 2024-01-15T18:00:45.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.510+0000 7fb2ca000640 1 -- 172.21.15.2:0/1587379007 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286878 (secure 0 0 0) 0x7fb2b4031a10 con 0x7fb2a406b500 2024-01-15T18:00:45.519 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:45.520 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:00:45.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2a406b500 msgr2=0x7fb2a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2a406b500 0x7fb2a406d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4036000 tx=0x7fb2b403c040 comp rx=0 tx=0).stop 2024-01-15T18:00:45.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 msgr2=0x7fb2cc1f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc1f2f40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb2bc00d800 tx=0x7fb2bc00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 shutdown_connections 2024-01-15T18:00:45.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2a406b500 0x7fb2a406d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2cc0cb130 0x7fb2cc1f3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2cc0d2f10 0x7fb2cc1f2f40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 --2- 172.21.15.2:0/1587379007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2cc0ce2b0 0x7fb2cc0ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 >> 172.21.15.2:0/1587379007 conn(0x7fb2cc08cc30 msgr2=0x7fb2cc15e890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:45.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 shutdown_connections 2024-01-15T18:00:45.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.516+0000 7fb2d4a00640 1 -- 172.21.15.2:0/1587379007 wait complete. 2024-01-15T18:00:45.540 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-01-15T18:00:45.441121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":267456,"kb_used_data":5512,"kb_used_omap":12,"kb_used_meta":261811,"kb_avail":754707264,"statfs":{"total":773094113280,"available":772820238336,"internally_reserved":0,"allocated":5644288,"data_stored":2980500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":268095025},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001863"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.499230+0000","last_change":"2024-01-15T18:00:40.522730+0000","last_active":"2024-01-15T18:00:41.499230+0000","last_peered":"2024-01-15T18:00:41.499230+0000","last_clean":"2024-01-15T18:00:41.499230+0000","last_became_active":"2024-01-15T18:00:40.522355+0000","last_became_peered":"2024-01-15T18:00:40.522355+0000","last_unstale":"2024-01-15T18:00:41.499230+0000","last_undegraded":"2024-01-15T18:00:41.499230+0000","last_fullsized":"2024-01-15T18:00:41.499230+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:58.207865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.498801+0000","last_change":"2024-01-15T18:00:40.524588+0000","last_active":"2024-01-15T18:00:41.498801+0000","last_peered":"2024-01-15T18:00:41.498801+0000","last_clean":"2024-01-15T18:00:41.498801+0000","last_became_active":"2024-01-15T18:00:40.521141+0000","last_became_peered":"2024-01-15T18:00:40.521141+0000","last_unstale":"2024-01-15T18:00:41.498801+0000","last_undegraded":"2024-01-15T18:00:41.498801+0000","last_fullsized":"2024-01-15T18:00:41.498801+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:14.146565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510350+0000","last_change":"2024-01-15T18:00:40.517392+0000","last_active":"2024-01-15T18:00:41.510350+0000","last_peered":"2024-01-15T18:00:41.510350+0000","last_clean":"2024-01-15T18:00:41.510350+0000","last_became_active":"2024-01-15T18:00:40.517084+0000","last_became_peered":"2024-01-15T18:00:40.517084+0000","last_unstale":"2024-01-15T18:00:41.510350+0000","last_undegraded":"2024-01-15T18:00:41.510350+0000","last_fullsized":"2024-01-15T18:00:41.510350+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:09:01.870297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508720+0000","last_change":"2024-01-15T18:00:40.531375+0000","last_active":"2024-01-15T18:00:41.508720+0000","last_peered":"2024-01-15T18:00:41.508720+0000","last_clean":"2024-01-15T18:00:41.508720+0000","last_became_active":"2024-01-15T18:00:40.531103+0000","last_became_peered":"2024-01-15T18:00:40.531103+0000","last_unstale":"2024-01-15T18:00:41.508720+0000","last_undegraded":"2024-01-15T18:00:41.508720+0000","last_fullsized":"2024-01-15T18:00:41.508720+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:12:52.568572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.498617+0000","last_change":"2024-01-15T18:00:40.519912+0000","last_active":"2024-01-15T18:00:41.498617+0000","last_peered":"2024-01-15T18:00:41.498617+0000","last_clean":"2024-01-15T18:00:41.498617+0000","last_became_active":"2024-01-15T18:00:40.518835+0000","last_became_peered":"2024-01-15T18:00:40.518835+0000","last_unstale":"2024-01-15T18:00:41.498617+0000","last_undegraded":"2024-01-15T18:00:41.498617+0000","last_fullsized":"2024-01-15T18:00:41.498617+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:40.512803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510379+0000","last_change":"2024-01-15T18:00:40.524753+0000","last_active":"2024-01-15T18:00:41.510379+0000","last_peered":"2024-01-15T18:00:41.510379+0000","last_clean":"2024-01-15T18:00:41.510379+0000","last_became_active":"2024-01-15T18:00:40.524492+0000","last_became_peered":"2024-01-15T18:00:40.524492+0000","last_unstale":"2024-01-15T18:00:41.510379+0000","last_undegraded":"2024-01-15T18:00:41.510379+0000","last_fullsized":"2024-01-15T18:00:41.510379+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:47:56.572359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.497881+0000","last_change":"2024-01-15T18:00:40.535920+0000","last_active":"2024-01-15T18:00:41.497881+0000","last_peered":"2024-01-15T18:00:41.497881+0000","last_clean":"2024-01-15T18:00:41.497881+0000","last_became_active":"2024-01-15T18:00:40.535579+0000","last_became_peered":"2024-01-15T18:00:40.535579+0000","last_unstale":"2024-01-15T18:00:41.497881+0000","last_undegraded":"2024-01-15T18:00:41.497881+0000","last_fullsized":"2024-01-15T18:00:41.497881+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:20.616823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.507067+0000","last_change":"2024-01-15T18:00:40.541556+0000","last_active":"2024-01-15T18:00:41.507067+0000","last_peered":"2024-01-15T18:00:41.507067+0000","last_clean":"2024-01-15T18:00:41.507067+0000","last_became_active":"2024-01-15T18:00:40.541223+0000","last_became_peered":"2024-01-15T18:00:40.541223+0000","last_unstale":"2024-01-15T18:00:41.507067+0000","last_undegraded":"2024-01-15T18:00:41.507067+0000","last_fullsized":"2024-01-15T18:00:41.507067+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:35:04.041978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510249+0000","last_change":"2024-01-15T18:00:40.540213+0000","last_active":"2024-01-15T18:00:41.510249+0000","last_peered":"2024-01-15T18:00:41.510249+0000","last_clean":"2024-01-15T18:00:41.510249+0000","last_became_active":"2024-01-15T18:00:40.539890+0000","last_became_peered":"2024-01-15T18:00:40.539890+0000","last_unstale":"2024-01-15T18:00:41.510249+0000","last_undegraded":"2024-01-15T18:00:41.510249+0000","last_fullsized":"2024-01-15T18:00:41.510249+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:01.109601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509410+0000","last_change":"2024-01-15T18:00:40.536065+0000","last_active":"2024-01-15T18:00:41.509410+0000","last_peered":"2024-01-15T18:00:41.509410+0000","last_clean":"2024-01-15T18:00:41.509410+0000","last_became_active":"2024-01-15T18:00:40.535823+0000","last_became_peered":"2024-01-15T18:00:40.535823+0000","last_unstale":"2024-01-15T18:00:41.509410+0000","last_undegraded":"2024-01-15T18:00:41.509410+0000","last_fullsized":"2024-01-15T18:00:41.509410+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:55.469803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508881+0000","last_change":"2024-01-15T18:00:40.543968+0000","last_active":"2024-01-15T18:00:41.508881+0000","last_peered":"2024-01-15T18:00:41.508881+0000","last_clean":"2024-01-15T18:00:41.508881+0000","last_became_active":"2024-01-15T18:00:40.543755+0000","last_became_peered":"2024-01-15T18:00:40.543755+0000","last_unstale":"2024-01-15T18:00:41.508881+0000","last_undegraded":"2024-01-15T18:00:41.508881+0000","last_fullsized":"2024-01-15T18:00:41.508881+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:48:29.445626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.498545+0000","last_change":"2024-01-15T18:00:40.516094+0000","last_active":"2024-01-15T18:00:41.498545+0000","last_peered":"2024-01-15T18:00:41.498545+0000","last_clean":"2024-01-15T18:00:41.498545+0000","last_became_active":"2024-01-15T18:00:40.515809+0000","last_became_peered":"2024-01-15T18:00:40.515809+0000","last_unstale":"2024-01-15T18:00:41.498545+0000","last_undegraded":"2024-01-15T18:00:41.498545+0000","last_fullsized":"2024-01-15T18:00:41.498545+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:58.279254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.511409+0000","last_change":"2024-01-15T18:00:40.524434+0000","last_active":"2024-01-15T18:00:41.511409+0000","last_peered":"2024-01-15T18:00:41.511409+0000","last_clean":"2024-01-15T18:00:41.511409+0000","last_became_active":"2024-01-15T18:00:40.524160+0000","last_became_peered":"2024-01-15T18:00:40.524160+0000","last_unstale":"2024-01-15T18:00:41.511409+0000","last_undegraded":"2024-01-15T18:00:41.511409+0000","last_fullsized":"2024-01-15T18:00:41.511409+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:11.522127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508523+0000","last_change":"2024-01-15T18:00:40.539479+0000","last_active":"2024-01-15T18:00:41.508523+0000","last_peered":"2024-01-15T18:00:41.508523+0000","last_clean":"2024-01-15T18:00:41.508523+0000","last_became_active":"2024-01-15T18:00:40.539168+0000","last_became_peered":"2024-01-15T18:00:40.539168+0000","last_unstale":"2024-01-15T18:00:41.508523+0000","last_undegraded":"2024-01-15T18:00:41.508523+0000","last_fullsized":"2024-01-15T18:00:41.508523+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:29:06.896113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.507142+0000","last_change":"2024-01-15T18:00:40.523064+0000","last_active":"2024-01-15T18:00:41.507142+0000","last_peered":"2024-01-15T18:00:41.507142+0000","last_clean":"2024-01-15T18:00:41.507142+0000","last_became_active":"2024-01-15T18:00:40.522763+0000","last_became_peered":"2024-01-15T18:00:40.522763+0000","last_unstale":"2024-01-15T18:00:41.507142+0000","last_undegraded":"2024-01-15T18:00:41.507142+0000","last_fullsized":"2024-01-15T18:00:41.507142+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:57:48.973872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508468+0000","last_change":"2024-01-15T18:00:40.543405+0000","last_active":"2024-01-15T18:00:41.508468+0000","last_peered":"2024-01-15T18:00:41.508468+0000","last_clean":"2024-01-15T18:00:41.508468+0000","last_became_active":"2024-01-15T18:00:40.542475+0000","last_became_peered":"2024-01-15T18:00:40.542475+0000","last_unstale":"2024-01-15T18:00:41.508468+0000","last_undegraded":"2024-01-15T18:00:41.508468+0000","last_fullsized":"2024-01-15T18:00:41.508468+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:14:16.048272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.511744+0000","last_change":"2024-01-15T18:00:40.522340+0000","last_active":"2024-01-15T18:00:41.511744+0000","last_peered":"2024-01-15T18:00:41.511744+0000","last_clean":"2024-01-15T18:00:41.511744+0000","last_became_active":"2024-01-15T18:00:40.521343+0000","last_became_peered":"2024-01-15T18:00:40.521343+0000","last_unstale":"2024-01-15T18:00:41.511744+0000","last_undegraded":"2024-01-15T18:00:41.511744+0000","last_fullsized":"2024-01-15T18:00:41.511744+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:15.757291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510365+0000","last_change":"2024-01-15T18:00:40.523867+0000","last_active":"2024-01-15T18:00:41.510365+0000","last_peered":"2024-01-15T18:00:41.510365+0000","last_clean":"2024-01-15T18:00:41.510365+0000","last_became_active":"2024-01-15T18:00:40.523562+0000","last_became_peered":"2024-01-15T18:00:40.523562+0000","last_unstale":"2024-01-15T18:00:41.510365+0000","last_undegraded":"2024-01-15T18:00:41.510365+0000","last_fullsized":"2024-01-15T18:00:41.510365+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:24:06.129386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.512319+0000","last_change":"2024-01-15T18:00:40.543315+0000","last_active":"2024-01-15T18:00:41.512319+0000","last_peered":"2024-01-15T18:00:41.512319+0000","last_clean":"2024-01-15T18:00:41.512319+0000","last_became_active":"2024-01-15T18:00:40.542663+0000","last_became_peered":"2024-01-15T18:00:40.542663+0000","last_unstale":"2024-01-15T18:00:41.512319+0000","last_undegraded":"2024-01-15T18:00:41.512319+0000","last_fullsized":"2024-01-15T18:00:41.512319+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:18:18.970904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.512068+0000","last_change":"2024-01-15T18:00:40.523124+0000","last_active":"2024-01-15T18:00:41.512068+0000","last_peered":"2024-01-15T18:00:41.512068+0000","last_clean":"2024-01-15T18:00:41.512068+0000","last_became_active":"2024-01-15T18:00:40.522800+0000","last_became_peered":"2024-01-15T18:00:40.522800+0000","last_unstale":"2024-01-15T18:00:41.512068+0000","last_undegraded":"2024-01-15T18:00:41.512068+0000","last_fullsized":"2024-01-15T18:00:41.512068+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:25:03.446513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509814+0000","last_change":"2024-01-15T18:00:40.528671+0000","last_active":"2024-01-15T18:00:41.509814+0000","last_peered":"2024-01-15T18:00:41.509814+0000","last_clean":"2024-01-15T18:00:41.509814+0000","last_became_active":"2024-01-15T18:00:40.528264+0000","last_became_peered":"2024-01-15T18:00:40.528264+0000","last_unstale":"2024-01-15T18:00:41.509814+0000","last_undegraded":"2024-01-15T18:00:41.509814+0000","last_fullsized":"2024-01-15T18:00:41.509814+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:01:05.156661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508515+0000","last_change":"2024-01-15T18:00:40.535185+0000","last_active":"2024-01-15T18:00:41.508515+0000","last_peered":"2024-01-15T18:00:41.508515+0000","last_clean":"2024-01-15T18:00:41.508515+0000","last_became_active":"2024-01-15T18:00:40.534648+0000","last_became_peered":"2024-01-15T18:00:40.534648+0000","last_unstale":"2024-01-15T18:00:41.508515+0000","last_undegraded":"2024-01-15T18:00:41.508515+0000","last_fullsized":"2024-01-15T18:00:41.508515+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:28:46.125224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508995+0000","last_change":"2024-01-15T18:00:40.524208+0000","last_active":"2024-01-15T18:00:41.508995+0000","last_peered":"2024-01-15T18:00:41.508995+0000","last_clean":"2024-01-15T18:00:41.508995+0000","last_became_active":"2024-01-15T18:00:40.523961+0000","last_became_peered":"2024-01-15T18:00:40.523961+0000","last_unstale":"2024-01-15T18:00:41.508995+0000","last_undegraded":"2024-01-15T18:00:41.508995+0000","last_fullsized":"2024-01-15T18:00:41.508995+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:54.976496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508557+0000","last_change":"2024-01-15T18:00:40.526036+0000","last_active":"2024-01-15T18:00:41.508557+0000","last_peered":"2024-01-15T18:00:41.508557+0000","last_clean":"2024-01-15T18:00:41.508557+0000","last_became_active":"2024-01-15T18:00:40.525725+0000","last_became_peered":"2024-01-15T18:00:40.525725+0000","last_unstale":"2024-01-15T18:00:41.508557+0000","last_undegraded":"2024-01-15T18:00:41.508557+0000","last_fullsized":"2024-01-15T18:00:41.508557+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:04.065968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.512409+0000","last_change":"2024-01-15T18:00:40.539725+0000","last_active":"2024-01-15T18:00:41.512409+0000","last_peered":"2024-01-15T18:00:41.512409+0000","last_clean":"2024-01-15T18:00:41.512409+0000","last_became_active":"2024-01-15T18:00:40.539395+0000","last_became_peered":"2024-01-15T18:00:40.539395+0000","last_unstale":"2024-01-15T18:00:41.512409+0000","last_undegraded":"2024-01-15T18:00:41.512409+0000","last_fullsized":"2024-01-15T18:00:41.512409+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:07.592394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.512033+0000","last_change":"2024-01-15T18:00:40.520712+0000","last_active":"2024-01-15T18:00:41.512033+0000","last_peered":"2024-01-15T18:00:41.512033+0000","last_clean":"2024-01-15T18:00:41.512033+0000","last_became_active":"2024-01-15T18:00:40.519329+0000","last_became_peered":"2024-01-15T18:00:40.519329+0000","last_unstale":"2024-01-15T18:00:41.512033+0000","last_undegraded":"2024-01-15T18:00:41.512033+0000","last_fullsized":"2024-01-15T18:00:41.512033+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:11.768958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510744+0000","last_change":"2024-01-15T18:00:40.508342+0000","last_active":"2024-01-15T18:00:41.510744+0000","last_peered":"2024-01-15T18:00:41.510744+0000","last_clean":"2024-01-15T18:00:41.510744+0000","last_became_active":"2024-01-15T18:00:40.507878+0000","last_became_peered":"2024-01-15T18:00:40.507878+0000","last_unstale":"2024-01-15T18:00:41.510744+0000","last_undegraded":"2024-01-15T18:00:41.510744+0000","last_fullsized":"2024-01-15T18:00:41.510744+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:06:15.793280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.499059+0000","last_change":"2024-01-15T18:00:40.513840+0000","last_active":"2024-01-15T18:00:41.499059+0000","last_peered":"2024-01-15T18:00:41.499059+0000","last_clean":"2024-01-15T18:00:41.499059+0000","last_became_active":"2024-01-15T18:00:40.513537+0000","last_became_peered":"2024-01-15T18:00:40.513537+0000","last_unstale":"2024-01-15T18:00:41.499059+0000","last_undegraded":"2024-01-15T18:00:41.499059+0000","last_fullsized":"2024-01-15T18:00:41.499059+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:54:05.109529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509291+0000","last_change":"2024-01-15T18:00:40.540362+0000","last_active":"2024-01-15T18:00:41.509291+0000","last_peered":"2024-01-15T18:00:41.509291+0000","last_clean":"2024-01-15T18:00:41.509291+0000","last_became_active":"2024-01-15T18:00:40.540007+0000","last_became_peered":"2024-01-15T18:00:40.540007+0000","last_unstale":"2024-01-15T18:00:41.509291+0000","last_undegraded":"2024-01-15T18:00:41.509291+0000","last_fullsized":"2024-01-15T18:00:41.509291+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:28:20.637350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510790+0000","last_change":"2024-01-15T18:00:40.520719+0000","last_active":"2024-01-15T18:00:41.510790+0000","last_peered":"2024-01-15T18:00:41.510790+0000","last_clean":"2024-01-15T18:00:41.510790+0000","last_became_active":"2024-01-15T18:00:40.520419+0000","last_became_peered":"2024-01-15T18:00:40.520419+0000","last_unstale":"2024-01-15T18:00:41.510790+0000","last_undegraded":"2024-01-15T18:00:41.510790+0000","last_fullsized":"2024-01-15T18:00:41.510790+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:07.838155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.507482+0000","last_change":"2024-01-15T18:00:40.528384+0000","last_active":"2024-01-15T18:00:41.507482+0000","last_peered":"2024-01-15T18:00:41.507482+0000","last_clean":"2024-01-15T18:00:41.507482+0000","last_became_active":"2024-01-15T18:00:40.527900+0000","last_became_peered":"2024-01-15T18:00:40.527900+0000","last_unstale":"2024-01-15T18:00:41.507482+0000","last_undegraded":"2024-01-15T18:00:41.507482+0000","last_fullsized":"2024-01-15T18:00:41.507482+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:49.916377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.511002+0000","last_change":"2024-01-15T18:00:40.535474+0000","last_active":"2024-01-15T18:00:41.511002+0000","last_peered":"2024-01-15T18:00:41.511002+0000","last_clean":"2024-01-15T18:00:41.511002+0000","last_became_active":"2024-01-15T18:00:40.535095+0000","last_became_peered":"2024-01-15T18:00:40.535095+0000","last_unstale":"2024-01-15T18:00:41.511002+0000","last_undegraded":"2024-01-15T18:00:41.511002+0000","last_fullsized":"2024-01-15T18:00:41.511002+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:37.595741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.493609+0000","last_change":"2024-01-15T18:00:40.530750+0000","last_active":"2024-01-15T18:00:41.493609+0000","last_peered":"2024-01-15T18:00:41.493609+0000","last_clean":"2024-01-15T18:00:41.493609+0000","last_became_active":"2024-01-15T18:00:40.530419+0000","last_became_peered":"2024-01-15T18:00:40.530419+0000","last_unstale":"2024-01-15T18:00:41.493609+0000","last_undegraded":"2024-01-15T18:00:41.493609+0000","last_fullsized":"2024-01-15T18:00:41.493609+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:30.235229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"46'4","reported_seq":16,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.545216+0000","last_change":"2024-01-15T18:00:39.517990+0000","last_active":"2024-01-15T18:00:40.545216+0000","last_peered":"2024-01-15T18:00:40.545216+0000","last_clean":"2024-01-15T18:00:40.545216+0000","last_became_active":"2024-01-15T18:00:39.517570+0000","last_became_peered":"2024-01-15T18:00:39.517570+0000","last_unstale":"2024-01-15T18:00:40.545216+0000","last_undegraded":"2024-01-15T18:00:40.545216+0000","last_fullsized":"2024-01-15T18:00:40.545216+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:57.192206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.512426+0000","last_change":"2024-01-15T18:00:40.520961+0000","last_active":"2024-01-15T18:00:41.512426+0000","last_peered":"2024-01-15T18:00:41.512426+0000","last_clean":"2024-01-15T18:00:41.512426+0000","last_became_active":"2024-01-15T18:00:40.520490+0000","last_became_peered":"2024-01-15T18:00:40.520490+0000","last_unstale":"2024-01-15T18:00:41.512426+0000","last_undegraded":"2024-01-15T18:00:41.512426+0000","last_fullsized":"2024-01-15T18:00:41.512426+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:35.172106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.534109+0000","last_change":"2024-01-15T18:00:39.515276+0000","last_active":"2024-01-15T18:00:40.534109+0000","last_peered":"2024-01-15T18:00:40.534109+0000","last_clean":"2024-01-15T18:00:40.534109+0000","last_became_active":"2024-01-15T18:00:39.513765+0000","last_became_peered":"2024-01-15T18:00:39.513765+0000","last_unstale":"2024-01-15T18:00:40.534109+0000","last_undegraded":"2024-01-15T18:00:40.534109+0000","last_fullsized":"2024-01-15T18:00:40.534109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:32:45.652100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.498874+0000","last_change":"2024-01-15T18:00:40.522705+0000","last_active":"2024-01-15T18:00:41.498874+0000","last_peered":"2024-01-15T18:00:41.498874+0000","last_clean":"2024-01-15T18:00:41.498874+0000","last_became_active":"2024-01-15T18:00:40.520753+0000","last_became_peered":"2024-01-15T18:00:40.520753+0000","last_unstale":"2024-01-15T18:00:41.498874+0000","last_undegraded":"2024-01-15T18:00:41.498874+0000","last_fullsized":"2024-01-15T18:00:41.498874+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:55:34.858304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.544466+0000","last_change":"2024-01-15T18:00:39.509549+0000","last_active":"2024-01-15T18:00:40.544466+0000","last_peered":"2024-01-15T18:00:40.544466+0000","last_clean":"2024-01-15T18:00:40.544466+0000","last_became_active":"2024-01-15T18:00:39.509194+0000","last_became_peered":"2024-01-15T18:00:39.509194+0000","last_unstale":"2024-01-15T18:00:40.544466+0000","last_undegraded":"2024-01-15T18:00:40.544466+0000","last_fullsized":"2024-01-15T18:00:40.544466+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:18:17.955245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.493464+0000","last_change":"2024-01-15T18:00:40.529649+0000","last_active":"2024-01-15T18:00:41.493464+0000","last_peered":"2024-01-15T18:00:41.493464+0000","last_clean":"2024-01-15T18:00:41.493464+0000","last_became_active":"2024-01-15T18:00:40.529364+0000","last_became_peered":"2024-01-15T18:00:40.529364+0000","last_unstale":"2024-01-15T18:00:41.493464+0000","last_undegraded":"2024-01-15T18:00:41.493464+0000","last_fullsized":"2024-01-15T18:00:41.493464+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:35.172106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.541201+0000","last_change":"2024-01-15T18:00:39.508458+0000","last_active":"2024-01-15T18:00:40.541201+0000","last_peered":"2024-01-15T18:00:40.541201+0000","last_clean":"2024-01-15T18:00:40.541201+0000","last_became_active":"2024-01-15T18:00:39.506831+0000","last_became_peered":"2024-01-15T18:00:39.506831+0000","last_unstale":"2024-01-15T18:00:40.541201+0000","last_undegraded":"2024-01-15T18:00:40.541201+0000","last_fullsized":"2024-01-15T18:00:40.541201+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:29:00.093942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.511006+0000","last_change":"2024-01-15T18:00:40.518246+0000","last_active":"2024-01-15T18:00:41.511006+0000","last_peered":"2024-01-15T18:00:41.511006+0000","last_clean":"2024-01-15T18:00:41.511006+0000","last_became_active":"2024-01-15T18:00:40.517352+0000","last_became_peered":"2024-01-15T18:00:40.517352+0000","last_unstale":"2024-01-15T18:00:41.511006+0000","last_undegraded":"2024-01-15T18:00:41.511006+0000","last_fullsized":"2024-01-15T18:00:41.511006+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:56.220484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.546473+0000","last_change":"2024-01-15T18:00:39.514220+0000","last_active":"2024-01-15T18:00:40.546473+0000","last_peered":"2024-01-15T18:00:40.546473+0000","last_clean":"2024-01-15T18:00:40.546473+0000","last_became_active":"2024-01-15T18:00:39.513892+0000","last_became_peered":"2024-01-15T18:00:39.513892+0000","last_unstale":"2024-01-15T18:00:40.546473+0000","last_undegraded":"2024-01-15T18:00:40.546473+0000","last_fullsized":"2024-01-15T18:00:40.546473+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:56.204810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509410+0000","last_change":"2024-01-15T18:00:40.542401+0000","last_active":"2024-01-15T18:00:41.509410+0000","last_peered":"2024-01-15T18:00:41.509410+0000","last_clean":"2024-01-15T18:00:41.509410+0000","last_became_active":"2024-01-15T18:00:40.542205+0000","last_became_peered":"2024-01-15T18:00:40.542205+0000","last_unstale":"2024-01-15T18:00:41.509410+0000","last_undegraded":"2024-01-15T18:00:41.509410+0000","last_fullsized":"2024-01-15T18:00:41.509410+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:04:24.924931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.534682+0000","last_change":"2024-01-15T18:00:39.508316+0000","last_active":"2024-01-15T18:00:40.534682+0000","last_peered":"2024-01-15T18:00:40.534682+0000","last_clean":"2024-01-15T18:00:40.534682+0000","last_became_active":"2024-01-15T18:00:39.507756+0000","last_became_peered":"2024-01-15T18:00:39.507756+0000","last_unstale":"2024-01-15T18:00:40.534682+0000","last_undegraded":"2024-01-15T18:00:40.534682+0000","last_fullsized":"2024-01-15T18:00:40.534682+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:09:53.171406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.507169+0000","last_change":"2024-01-15T18:00:40.535474+0000","last_active":"2024-01-15T18:00:41.507169+0000","last_peered":"2024-01-15T18:00:41.507169+0000","last_clean":"2024-01-15T18:00:41.507169+0000","last_became_active":"2024-01-15T18:00:40.535195+0000","last_became_peered":"2024-01-15T18:00:40.535195+0000","last_unstale":"2024-01-15T18:00:41.507169+0000","last_undegraded":"2024-01-15T18:00:41.507169+0000","last_fullsized":"2024-01-15T18:00:41.507169+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:54:55.248776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.497369+0000","last_change":"2024-01-15T18:00:39.502847+0000","last_active":"2024-01-15T18:00:40.497369+0000","last_peered":"2024-01-15T18:00:40.497369+0000","last_clean":"2024-01-15T18:00:40.497369+0000","last_became_active":"2024-01-15T18:00:39.502372+0000","last_became_peered":"2024-01-15T18:00:39.502372+0000","last_unstale":"2024-01-15T18:00:40.497369+0000","last_undegraded":"2024-01-15T18:00:40.497369+0000","last_fullsized":"2024-01-15T18:00:40.497369+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:35.933547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508755+0000","last_change":"2024-01-15T18:00:40.526585+0000","last_active":"2024-01-15T18:00:41.508755+0000","last_peered":"2024-01-15T18:00:41.508755+0000","last_clean":"2024-01-15T18:00:41.508755+0000","last_became_active":"2024-01-15T18:00:40.526277+0000","last_became_peered":"2024-01-15T18:00:40.526277+0000","last_unstale":"2024-01-15T18:00:41.508755+0000","last_undegraded":"2024-01-15T18:00:41.508755+0000","last_fullsized":"2024-01-15T18:00:41.508755+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:02.860732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.515620+0000","last_change":"2024-01-15T18:00:39.504496+0000","last_active":"2024-01-15T18:00:40.515620+0000","last_peered":"2024-01-15T18:00:40.515620+0000","last_clean":"2024-01-15T18:00:40.515620+0000","last_became_active":"2024-01-15T18:00:39.504064+0000","last_became_peered":"2024-01-15T18:00:39.504064+0000","last_unstale":"2024-01-15T18:00:40.515620+0000","last_undegraded":"2024-01-15T18:00:40.515620+0000","last_fullsized":"2024-01-15T18:00:40.515620+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:38:34.156448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510905+0000","last_change":"2024-01-15T18:00:40.523873+0000","last_active":"2024-01-15T18:00:41.510905+0000","last_peered":"2024-01-15T18:00:41.510905+0000","last_clean":"2024-01-15T18:00:41.510905+0000","last_became_active":"2024-01-15T18:00:40.523591+0000","last_became_peered":"2024-01-15T18:00:40.523591+0000","last_unstale":"2024-01-15T18:00:41.510905+0000","last_undegraded":"2024-01-15T18:00:41.510905+0000","last_fullsized":"2024-01-15T18:00:41.510905+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:49:51.816919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.504255+0000","last_change":"2024-01-15T18:00:39.514446+0000","last_active":"2024-01-15T18:00:40.504255+0000","last_peered":"2024-01-15T18:00:40.504255+0000","last_clean":"2024-01-15T18:00:40.504255+0000","last_became_active":"2024-01-15T18:00:39.514162+0000","last_became_peered":"2024-01-15T18:00:39.514162+0000","last_unstale":"2024-01-15T18:00:40.504255+0000","last_undegraded":"2024-01-15T18:00:40.504255+0000","last_fullsized":"2024-01-15T18:00:40.504255+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:59:19.318480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.499540+0000","last_change":"2024-01-15T18:00:40.524710+0000","last_active":"2024-01-15T18:00:41.499540+0000","last_peered":"2024-01-15T18:00:41.499540+0000","last_clean":"2024-01-15T18:00:41.499540+0000","last_became_active":"2024-01-15T18:00:40.524335+0000","last_became_peered":"2024-01-15T18:00:40.524335+0000","last_unstale":"2024-01-15T18:00:41.499540+0000","last_undegraded":"2024-01-15T18:00:41.499540+0000","last_fullsized":"2024-01-15T18:00:41.499540+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:45:25.343641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.511703+0000","last_change":"2024-01-15T18:00:39.514145+0000","last_active":"2024-01-15T18:00:40.511703+0000","last_peered":"2024-01-15T18:00:40.511703+0000","last_clean":"2024-01-15T18:00:40.511703+0000","last_became_active":"2024-01-15T18:00:39.511885+0000","last_became_peered":"2024-01-15T18:00:39.511885+0000","last_unstale":"2024-01-15T18:00:40.511703+0000","last_undegraded":"2024-01-15T18:00:40.511703+0000","last_fullsized":"2024-01-15T18:00:40.511703+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:28:19.621691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.513228+0000","last_change":"2024-01-15T18:00:40.522452+0000","last_active":"2024-01-15T18:00:41.513228+0000","last_peered":"2024-01-15T18:00:41.513228+0000","last_clean":"2024-01-15T18:00:41.513228+0000","last_became_active":"2024-01-15T18:00:40.522194+0000","last_became_peered":"2024-01-15T18:00:40.522194+0000","last_unstale":"2024-01-15T18:00:41.513228+0000","last_undegraded":"2024-01-15T18:00:41.513228+0000","last_fullsized":"2024-01-15T18:00:41.513228+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:35:46.548763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"46'3","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.541909+0000","last_change":"2024-01-15T18:00:39.519070+0000","last_active":"2024-01-15T18:00:40.541909+0000","last_peered":"2024-01-15T18:00:40.541909+0000","last_clean":"2024-01-15T18:00:40.541909+0000","last_became_active":"2024-01-15T18:00:39.518768+0000","last_became_peered":"2024-01-15T18:00:39.518768+0000","last_unstale":"2024-01-15T18:00:40.541909+0000","last_undegraded":"2024-01-15T18:00:40.541909+0000","last_fullsized":"2024-01-15T18:00:40.541909+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:57.263596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.498667+0000","last_change":"2024-01-15T18:00:40.531578+0000","last_active":"2024-01-15T18:00:41.498667+0000","last_peered":"2024-01-15T18:00:41.498667+0000","last_clean":"2024-01-15T18:00:41.498667+0000","last_became_active":"2024-01-15T18:00:40.531264+0000","last_became_peered":"2024-01-15T18:00:40.531264+0000","last_unstale":"2024-01-15T18:00:41.498667+0000","last_undegraded":"2024-01-15T18:00:41.498667+0000","last_fullsized":"2024-01-15T18:00:41.498667+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:57.220468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"46'6","reported_seq":18,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.550376+0000","last_change":"2024-01-15T18:00:39.506750+0000","last_active":"2024-01-15T18:00:40.550376+0000","last_peered":"2024-01-15T18:00:40.550376+0000","last_clean":"2024-01-15T18:00:40.550376+0000","last_became_active":"2024-01-15T18:00:39.506360+0000","last_became_peered":"2024-01-15T18:00:39.506360+0000","last_unstale":"2024-01-15T18:00:40.550376+0000","last_undegraded":"2024-01-15T18:00:40.550376+0000","last_fullsized":"2024-01-15T18:00:40.550376+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:02:44.094210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509317+0000","last_change":"2024-01-15T18:00:40.542961+0000","last_active":"2024-01-15T18:00:41.509317+0000","last_peered":"2024-01-15T18:00:41.509317+0000","last_clean":"2024-01-15T18:00:41.509317+0000","last_became_active":"2024-01-15T18:00:40.542638+0000","last_became_peered":"2024-01-15T18:00:40.542638+0000","last_unstale":"2024-01-15T18:00:41.509317+0000","last_undegraded":"2024-01-15T18:00:41.509317+0000","last_fullsized":"2024-01-15T18:00:41.509317+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:05:41.896280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.545112+0000","last_change":"2024-01-15T18:00:39.516161+0000","last_active":"2024-01-15T18:00:40.545112+0000","last_peered":"2024-01-15T18:00:40.545112+0000","last_clean":"2024-01-15T18:00:40.545112+0000","last_became_active":"2024-01-15T18:00:39.515883+0000","last_became_peered":"2024-01-15T18:00:39.515883+0000","last_unstale":"2024-01-15T18:00:40.545112+0000","last_undegraded":"2024-01-15T18:00:40.545112+0000","last_fullsized":"2024-01-15T18:00:40.545112+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:17:36.522397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509787+0000","last_change":"2024-01-15T18:00:40.520404+0000","last_active":"2024-01-15T18:00:41.509787+0000","last_peered":"2024-01-15T18:00:41.509787+0000","last_clean":"2024-01-15T18:00:41.509787+0000","last_became_active":"2024-01-15T18:00:40.520068+0000","last_became_peered":"2024-01-15T18:00:40.520068+0000","last_unstale":"2024-01-15T18:00:41.509787+0000","last_undegraded":"2024-01-15T18:00:41.509787+0000","last_fullsized":"2024-01-15T18:00:41.509787+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:47:36.949205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.514305+0000","last_change":"2024-01-15T18:00:39.513367+0000","last_active":"2024-01-15T18:00:40.514305+0000","last_peered":"2024-01-15T18:00:40.514305+0000","last_clean":"2024-01-15T18:00:40.514305+0000","last_became_active":"2024-01-15T18:00:39.511580+0000","last_became_peered":"2024-01-15T18:00:39.511580+0000","last_unstale":"2024-01-15T18:00:40.514305+0000","last_undegraded":"2024-01-15T18:00:40.514305+0000","last_fullsized":"2024-01-15T18:00:40.514305+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:55:53.777752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.507073+0000","last_change":"2024-01-15T18:00:40.520977+0000","last_active":"2024-01-15T18:00:41.507073+0000","last_peered":"2024-01-15T18:00:41.507073+0000","last_clean":"2024-01-15T18:00:41.507073+0000","last_became_active":"2024-01-15T18:00:40.520059+0000","last_became_peered":"2024-01-15T18:00:40.520059+0000","last_unstale":"2024-01-15T18:00:41.507073+0000","last_undegraded":"2024-01-15T18:00:41.507073+0000","last_fullsized":"2024-01-15T18:00:41.507073+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:06:49.377106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.515920+0000","last_change":"2024-01-15T18:00:39.516318+0000","last_active":"2024-01-15T18:00:40.515920+0000","last_peered":"2024-01-15T18:00:40.515920+0000","last_clean":"2024-01-15T18:00:40.515920+0000","last_became_active":"2024-01-15T18:00:39.515263+0000","last_became_peered":"2024-01-15T18:00:39.515263+0000","last_unstale":"2024-01-15T18:00:40.515920+0000","last_undegraded":"2024-01-15T18:00:40.515920+0000","last_fullsized":"2024-01-15T18:00:40.515920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:19.601165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.497939+0000","last_change":"2024-01-15T18:00:40.529338+0000","last_active":"2024-01-15T18:00:41.497939+0000","last_peered":"2024-01-15T18:00:41.497939+0000","last_clean":"2024-01-15T18:00:41.497939+0000","last_became_active":"2024-01-15T18:00:40.529039+0000","last_became_peered":"2024-01-15T18:00:40.529039+0000","last_unstale":"2024-01-15T18:00:41.497939+0000","last_undegraded":"2024-01-15T18:00:41.497939+0000","last_fullsized":"2024-01-15T18:00:41.497939+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:18.706163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"46'8","reported_seq":20,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.548111+0000","last_change":"2024-01-15T18:00:39.508572+0000","last_active":"2024-01-15T18:00:40.548111+0000","last_peered":"2024-01-15T18:00:40.548111+0000","last_clean":"2024-01-15T18:00:40.548111+0000","last_became_active":"2024-01-15T18:00:39.508209+0000","last_became_peered":"2024-01-15T18:00:39.508209+0000","last_unstale":"2024-01-15T18:00:40.548111+0000","last_undegraded":"2024-01-15T18:00:40.548111+0000","last_fullsized":"2024-01-15T18:00:40.548111+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:06.576736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.514638+0000","last_change":"2024-01-15T18:00:40.506517+0000","last_active":"2024-01-15T18:00:41.514638+0000","last_peered":"2024-01-15T18:00:41.514638+0000","last_clean":"2024-01-15T18:00:41.514638+0000","last_became_active":"2024-01-15T18:00:40.505782+0000","last_became_peered":"2024-01-15T18:00:40.505782+0000","last_unstale":"2024-01-15T18:00:41.514638+0000","last_undegraded":"2024-01-15T18:00:41.514638+0000","last_fullsized":"2024-01-15T18:00:41.514638+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:48.989893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.548774+0000","last_change":"2024-01-15T18:00:39.509196+0000","last_active":"2024-01-15T18:00:40.548774+0000","last_peered":"2024-01-15T18:00:40.548774+0000","last_clean":"2024-01-15T18:00:40.548774+0000","last_became_active":"2024-01-15T18:00:39.508876+0000","last_became_peered":"2024-01-15T18:00:39.508876+0000","last_unstale":"2024-01-15T18:00:40.548774+0000","last_undegraded":"2024-01-15T18:00:40.548774+0000","last_fullsized":"2024-01-15T18:00:40.548774+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:33:46.372699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510389+0000","last_change":"2024-01-15T18:00:40.537861+0000","last_active":"2024-01-15T18:00:41.510389+0000","last_peered":"2024-01-15T18:00:41.510389+0000","last_clean":"2024-01-15T18:00:41.510389+0000","last_became_active":"2024-01-15T18:00:40.537616+0000","last_became_peered":"2024-01-15T18:00:40.537616+0000","last_unstale":"2024-01-15T18:00:41.510389+0000","last_undegraded":"2024-01-15T18:00:41.510389+0000","last_fullsized":"2024-01-15T18:00:41.510389+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:12.122988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.545553+0000","last_change":"2024-01-15T18:00:39.516905+0000","last_active":"2024-01-15T18:00:40.545553+0000","last_peered":"2024-01-15T18:00:40.545553+0000","last_clean":"2024-01-15T18:00:40.545553+0000","last_became_active":"2024-01-15T18:00:39.515972+0000","last_became_peered":"2024-01-15T18:00:39.515972+0000","last_unstale":"2024-01-15T18:00:40.545553+0000","last_undegraded":"2024-01-15T18:00:40.545553+0000","last_fullsized":"2024-01-15T18:00:40.545553+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:31:02.723112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508898+0000","last_change":"2024-01-15T18:00:40.528027+0000","last_active":"2024-01-15T18:00:41.508898+0000","last_peered":"2024-01-15T18:00:41.508898+0000","last_clean":"2024-01-15T18:00:41.508898+0000","last_became_active":"2024-01-15T18:00:40.527370+0000","last_became_peered":"2024-01-15T18:00:40.527370+0000","last_unstale":"2024-01-15T18:00:41.508898+0000","last_undegraded":"2024-01-15T18:00:41.508898+0000","last_fullsized":"2024-01-15T18:00:41.508898+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:59.404348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"46'1","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.549000+0000","last_change":"2024-01-15T18:00:39.506992+0000","last_active":"2024-01-15T18:00:40.549000+0000","last_peered":"2024-01-15T18:00:40.549000+0000","last_clean":"2024-01-15T18:00:40.549000+0000","last_became_active":"2024-01-15T18:00:39.506698+0000","last_became_peered":"2024-01-15T18:00:39.506698+0000","last_unstale":"2024-01-15T18:00:40.549000+0000","last_undegraded":"2024-01-15T18:00:40.549000+0000","last_fullsized":"2024-01-15T18:00:40.549000+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:54.454145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.511754+0000","last_change":"2024-01-15T18:00:40.539702+0000","last_active":"2024-01-15T18:00:41.511754+0000","last_peered":"2024-01-15T18:00:41.511754+0000","last_clean":"2024-01-15T18:00:41.511754+0000","last_became_active":"2024-01-15T18:00:40.539388+0000","last_became_peered":"2024-01-15T18:00:40.539388+0000","last_unstale":"2024-01-15T18:00:41.511754+0000","last_undegraded":"2024-01-15T18:00:41.511754+0000","last_fullsized":"2024-01-15T18:00:41.511754+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:30:12.526641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.515856+0000","last_change":"2024-01-15T18:00:39.517063+0000","last_active":"2024-01-15T18:00:40.515856+0000","last_peered":"2024-01-15T18:00:40.515856+0000","last_clean":"2024-01-15T18:00:40.515856+0000","last_became_active":"2024-01-15T18:00:39.516826+0000","last_became_peered":"2024-01-15T18:00:39.516826+0000","last_unstale":"2024-01-15T18:00:40.515856+0000","last_undegraded":"2024-01-15T18:00:40.515856+0000","last_fullsized":"2024-01-15T18:00:40.515856+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:55:53.777752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510563+0000","last_change":"2024-01-15T18:00:40.540022+0000","last_active":"2024-01-15T18:00:41.510563+0000","last_peered":"2024-01-15T18:00:41.510563+0000","last_clean":"2024-01-15T18:00:41.510563+0000","last_became_active":"2024-01-15T18:00:40.534138+0000","last_became_peered":"2024-01-15T18:00:40.534138+0000","last_unstale":"2024-01-15T18:00:41.510563+0000","last_undegraded":"2024-01-15T18:00:41.510563+0000","last_fullsized":"2024-01-15T18:00:41.510563+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:29:32.583572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.546677+0000","last_change":"2024-01-15T18:00:39.512612+0000","last_active":"2024-01-15T18:00:40.546677+0000","last_peered":"2024-01-15T18:00:40.546677+0000","last_clean":"2024-01-15T18:00:40.546677+0000","last_became_active":"2024-01-15T18:00:39.512336+0000","last_became_peered":"2024-01-15T18:00:39.512336+0000","last_unstale":"2024-01-15T18:00:40.546677+0000","last_undegraded":"2024-01-15T18:00:40.546677+0000","last_fullsized":"2024-01-15T18:00:40.546677+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:17.690504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.493151+0000","last_change":"2024-01-15T18:00:40.533358+0000","last_active":"2024-01-15T18:00:41.493151+0000","last_peered":"2024-01-15T18:00:41.493151+0000","last_clean":"2024-01-15T18:00:41.493151+0000","last_became_active":"2024-01-15T18:00:40.533043+0000","last_became_peered":"2024-01-15T18:00:40.533043+0000","last_unstale":"2024-01-15T18:00:41.493151+0000","last_undegraded":"2024-01-15T18:00:41.493151+0000","last_fullsized":"2024-01-15T18:00:41.493151+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:45:07.592394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.500535+0000","last_change":"2024-01-15T18:00:39.515148+0000","last_active":"2024-01-15T18:00:40.500535+0000","last_peered":"2024-01-15T18:00:40.500535+0000","last_clean":"2024-01-15T18:00:40.500535+0000","last_became_active":"2024-01-15T18:00:39.513738+0000","last_became_peered":"2024-01-15T18:00:39.513738+0000","last_unstale":"2024-01-15T18:00:40.500535+0000","last_undegraded":"2024-01-15T18:00:40.500535+0000","last_fullsized":"2024-01-15T18:00:40.500535+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:07:11.107330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.511307+0000","last_change":"2024-01-15T18:00:40.524160+0000","last_active":"2024-01-15T18:00:41.511307+0000","last_peered":"2024-01-15T18:00:41.511307+0000","last_clean":"2024-01-15T18:00:41.511307+0000","last_became_active":"2024-01-15T18:00:40.523871+0000","last_became_peered":"2024-01-15T18:00:40.523871+0000","last_unstale":"2024-01-15T18:00:41.511307+0000","last_undegraded":"2024-01-15T18:00:41.511307+0000","last_fullsized":"2024-01-15T18:00:41.511307+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:16:52.822975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.507728+0000","last_change":"2024-01-15T18:00:39.511826+0000","last_active":"2024-01-15T18:00:40.507728+0000","last_peered":"2024-01-15T18:00:40.507728+0000","last_clean":"2024-01-15T18:00:40.507728+0000","last_became_active":"2024-01-15T18:00:39.511410+0000","last_became_peered":"2024-01-15T18:00:39.511410+0000","last_unstale":"2024-01-15T18:00:40.507728+0000","last_undegraded":"2024-01-15T18:00:40.507728+0000","last_fullsized":"2024-01-15T18:00:40.507728+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:23:03.050309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.507748+0000","last_change":"2024-01-15T18:00:40.540315+0000","last_active":"2024-01-15T18:00:41.507748+0000","last_peered":"2024-01-15T18:00:41.507748+0000","last_clean":"2024-01-15T18:00:41.507748+0000","last_became_active":"2024-01-15T18:00:40.540013+0000","last_became_peered":"2024-01-15T18:00:40.540013+0000","last_unstale":"2024-01-15T18:00:41.507748+0000","last_undegraded":"2024-01-15T18:00:41.507748+0000","last_fullsized":"2024-01-15T18:00:41.507748+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:11.974368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.511855+0000","last_change":"2024-01-15T18:00:39.510206+0000","last_active":"2024-01-15T18:00:40.511855+0000","last_peered":"2024-01-15T18:00:40.511855+0000","last_clean":"2024-01-15T18:00:40.511855+0000","last_became_active":"2024-01-15T18:00:39.509532+0000","last_became_peered":"2024-01-15T18:00:39.509532+0000","last_unstale":"2024-01-15T18:00:40.511855+0000","last_undegraded":"2024-01-15T18:00:40.511855+0000","last_fullsized":"2024-01-15T18:00:40.511855+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:17:36.522397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.512820+0000","last_change":"2024-01-15T18:00:40.543493+0000","last_active":"2024-01-15T18:00:41.512820+0000","last_peered":"2024-01-15T18:00:41.512820+0000","last_clean":"2024-01-15T18:00:41.512820+0000","last_became_active":"2024-01-15T18:00:40.543173+0000","last_became_peered":"2024-01-15T18:00:40.543173+0000","last_unstale":"2024-01-15T18:00:41.512820+0000","last_undegraded":"2024-01-15T18:00:41.512820+0000","last_fullsized":"2024-01-15T18:00:41.512820+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:30.235229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.518109+0000","last_change":"2024-01-15T18:00:39.512357+0000","last_active":"2024-01-15T18:00:40.518109+0000","last_peered":"2024-01-15T18:00:40.518109+0000","last_clean":"2024-01-15T18:00:40.518109+0000","last_became_active":"2024-01-15T18:00:39.512044+0000","last_became_peered":"2024-01-15T18:00:39.512044+0000","last_unstale":"2024-01-15T18:00:40.518109+0000","last_undegraded":"2024-01-15T18:00:40.518109+0000","last_fullsized":"2024-01-15T18:00:40.518109+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:52.658564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.508984+0000","last_change":"2024-01-15T18:00:40.516642+0000","last_active":"2024-01-15T18:00:41.508984+0000","last_peered":"2024-01-15T18:00:41.508984+0000","last_clean":"2024-01-15T18:00:41.508984+0000","last_became_active":"2024-01-15T18:00:40.516249+0000","last_became_peered":"2024-01-15T18:00:40.516249+0000","last_unstale":"2024-01-15T18:00:41.508984+0000","last_undegraded":"2024-01-15T18:00:41.508984+0000","last_fullsized":"2024-01-15T18:00:41.508984+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:24.949220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.509895+0000","last_change":"2024-01-15T18:00:39.519218+0000","last_active":"2024-01-15T18:00:40.509895+0000","last_peered":"2024-01-15T18:00:40.509895+0000","last_clean":"2024-01-15T18:00:40.509895+0000","last_became_active":"2024-01-15T18:00:39.519033+0000","last_became_peered":"2024-01-15T18:00:39.519033+0000","last_unstale":"2024-01-15T18:00:40.509895+0000","last_undegraded":"2024-01-15T18:00:40.509895+0000","last_fullsized":"2024-01-15T18:00:40.509895+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:31:39.497145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.498954+0000","last_change":"2024-01-15T18:00:40.519612+0000","last_active":"2024-01-15T18:00:41.498954+0000","last_peered":"2024-01-15T18:00:41.498954+0000","last_clean":"2024-01-15T18:00:41.498954+0000","last_became_active":"2024-01-15T18:00:40.519318+0000","last_became_peered":"2024-01-15T18:00:40.519318+0000","last_unstale":"2024-01-15T18:00:41.498954+0000","last_undegraded":"2024-01-15T18:00:41.498954+0000","last_fullsized":"2024-01-15T18:00:41.498954+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:53.674222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.528704+0000","last_change":"2024-01-15T18:00:39.507987+0000","last_active":"2024-01-15T18:00:40.528704+0000","last_peered":"2024-01-15T18:00:40.528704+0000","last_clean":"2024-01-15T18:00:40.528704+0000","last_became_active":"2024-01-15T18:00:39.507629+0000","last_became_peered":"2024-01-15T18:00:39.507629+0000","last_unstale":"2024-01-15T18:00:40.528704+0000","last_undegraded":"2024-01-15T18:00:40.528704+0000","last_fullsized":"2024-01-15T18:00:40.528704+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:30:11.510983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510740+0000","last_change":"2024-01-15T18:00:40.545641+0000","last_active":"2024-01-15T18:00:41.510740+0000","last_peered":"2024-01-15T18:00:41.510740+0000","last_clean":"2024-01-15T18:00:41.510740+0000","last_became_active":"2024-01-15T18:00:40.544067+0000","last_became_peered":"2024-01-15T18:00:40.544067+0000","last_unstale":"2024-01-15T18:00:41.510740+0000","last_undegraded":"2024-01-15T18:00:41.510740+0000","last_fullsized":"2024-01-15T18:00:41.510740+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:19:34.578327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.508439+0000","last_change":"2024-01-15T18:00:39.518334+0000","last_active":"2024-01-15T18:00:40.508439+0000","last_peered":"2024-01-15T18:00:40.508439+0000","last_clean":"2024-01-15T18:00:40.508439+0000","last_became_active":"2024-01-15T18:00:39.517862+0000","last_became_peered":"2024-01-15T18:00:39.517862+0000","last_unstale":"2024-01-15T18:00:40.508439+0000","last_undegraded":"2024-01-15T18:00:40.508439+0000","last_fullsized":"2024-01-15T18:00:40.508439+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:10.816091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510172+0000","last_change":"2024-01-15T18:00:40.538563+0000","last_active":"2024-01-15T18:00:41.510172+0000","last_peered":"2024-01-15T18:00:41.510172+0000","last_clean":"2024-01-15T18:00:41.510172+0000","last_became_active":"2024-01-15T18:00:40.538339+0000","last_became_peered":"2024-01-15T18:00:40.538339+0000","last_unstale":"2024-01-15T18:00:41.510172+0000","last_undegraded":"2024-01-15T18:00:41.510172+0000","last_fullsized":"2024-01-15T18:00:41.510172+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:07:21.478998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.500270+0000","last_change":"2024-01-15T18:00:39.512639+0000","last_active":"2024-01-15T18:00:40.500270+0000","last_peered":"2024-01-15T18:00:40.500270+0000","last_clean":"2024-01-15T18:00:40.500270+0000","last_became_active":"2024-01-15T18:00:39.512093+0000","last_became_peered":"2024-01-15T18:00:39.512093+0000","last_unstale":"2024-01-15T18:00:40.500270+0000","last_undegraded":"2024-01-15T18:00:40.500270+0000","last_fullsized":"2024-01-15T18:00:40.500270+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:41:42.738042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.496800+0000","last_change":"2024-01-15T18:00:40.532115+0000","last_active":"2024-01-15T18:00:41.496800+0000","last_peered":"2024-01-15T18:00:41.496800+0000","last_clean":"2024-01-15T18:00:41.496800+0000","last_became_active":"2024-01-15T18:00:40.531859+0000","last_became_peered":"2024-01-15T18:00:40.531859+0000","last_unstale":"2024-01-15T18:00:41.496800+0000","last_undegraded":"2024-01-15T18:00:41.496800+0000","last_fullsized":"2024-01-15T18:00:41.496800+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:23:11.016208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.517136+0000","last_change":"2024-01-15T18:00:39.513197+0000","last_active":"2024-01-15T18:00:40.517136+0000","last_peered":"2024-01-15T18:00:40.517136+0000","last_clean":"2024-01-15T18:00:40.517136+0000","last_became_active":"2024-01-15T18:00:39.512891+0000","last_became_peered":"2024-01-15T18:00:39.512891+0000","last_unstale":"2024-01-15T18:00:40.517136+0000","last_undegraded":"2024-01-15T18:00:40.517136+0000","last_fullsized":"2024-01-15T18:00:40.517136+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:05:40.880621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.510138+0000","last_change":"2024-01-15T18:00:40.523202+0000","last_active":"2024-01-15T18:00:41.510138+0000","last_peered":"2024-01-15T18:00:41.510138+0000","last_clean":"2024-01-15T18:00:41.510138+0000","last_became_active":"2024-01-15T18:00:40.522992+0000","last_became_peered":"2024-01-15T18:00:40.522992+0000","last_unstale":"2024-01-15T18:00:41.510138+0000","last_undegraded":"2024-01-15T18:00:41.510138+0000","last_fullsized":"2024-01-15T18:00:41.510138+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:10.152633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.515787+0000","last_change":"2024-01-15T18:00:39.510678+0000","last_active":"2024-01-15T18:00:40.515787+0000","last_peered":"2024-01-15T18:00:40.515787+0000","last_clean":"2024-01-15T18:00:40.515787+0000","last_became_active":"2024-01-15T18:00:39.510396+0000","last_became_peered":"2024-01-15T18:00:39.510396+0000","last_unstale":"2024-01-15T18:00:40.515787+0000","last_undegraded":"2024-01-15T18:00:40.515787+0000","last_fullsized":"2024-01-15T18:00:40.515787+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:29.219570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-15T18:00:41.509860+0000","last_change":"2024-01-15T18:00:40.521418+0000","last_active":"2024-01-15T18:00:41.509860+0000","last_peered":"2024-01-15T18:00:41.509860+0000","last_clean":"2024-01-15T18:00:41.509860+0000","last_became_active":"2024-01-15T18:00:40.521140+0000","last_became_peered":"2024-01-15T18:00:40.521140+0000","last_unstale":"2024-01-15T18:00:41.509860+0000","last_undegraded":"2024-01-15T18:00:41.509860+0000","last_fullsized":"2024-01-15T18:00:41.509860+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:39.469222+0000","last_clean_scrub_stamp":"2024-01-15T18:00:39.469222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:11.831750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"46'2","reported_seq":14,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.546732+0000","last_change":"2024-01-15T18:00:39.509579+0000","last_active":"2024-01-15T18:00:40.546732+0000","last_peered":"2024-01-15T18:00:40.546732+0000","last_clean":"2024-01-15T18:00:40.546732+0000","last_became_active":"2024-01-15T18:00:39.509216+0000","last_became_peered":"2024-01-15T18:00:39.509216+0000","last_unstale":"2024-01-15T18:00:40.546732+0000","last_undegraded":"2024-01-15T18:00:40.546732+0000","last_fullsized":"2024-01-15T18:00:40.546732+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:00:38.453564+0000","last_clean_scrub_stamp":"2024-01-15T18:00:38.453564+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:23.933562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":187,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-15T18:00:40.512058+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:00:40.512058+0000","last_peered":"2024-01-15T18:00:40.512058+0000","last_clean":"2024-01-15T18:00:40.512058+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:00:40.512058+0000","last_undegraded":"2024-01-15T18:00:40.512058+0000","last_fullsized":"2024-01-15T18:00:40.512058+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542186,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1146880,"data_stored":814826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0009999999999999,"5min":1.2969999999999999,"15min":1.161},"min":{"1min":0.72599999999999998,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":12.468999999999999,"5min":12.468999999999999,"15min":12.468999999999999},"last":1.254},{"interface":"front","average":{"1min":1.8680000000000001,"5min":1.486,"15min":1.454},"min":{"1min":0.78000000000000003,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":12.496,"5min":12.496,"15min":12.496},"last":1.4039999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":2.0369999999999999,"5min":1.3,"15min":1.1659999999999999},"min":{"1min":0.93899999999999995,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":14.497,"5min":14.497,"15min":14.497},"last":1.286},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.1759999999999999,"15min":1.1299999999999999},"min":{"1min":0.70099999999999996,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":4.2190000000000003,"5min":4.2190000000000003,"15min":4.2190000000000003},"last":1.3120000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.208,"15min":1.1399999999999999},"min":{"1min":0.82299999999999995,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":4.3179999999999996,"5min":4.3179999999999996,"15min":4.3179999999999996},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.377,"15min":1.4450000000000001},"min":{"1min":0.72099999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6059999999999999,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.141}]},{"osd":3,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":2.028,"5min":1.4610000000000001,"15min":1.397},"min":{"1min":0.90800000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":14.622,"5min":14.622,"15min":14.622},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.379,"15min":1.347},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":4.2690000000000001,"5min":4.2690000000000001,"15min":4.2690000000000001},"last":1.1719999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.4810000000000001,"15min":1.577},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.794,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":2.1339999999999999,"5min":1.5309999999999999,"15min":1.4379999999999999},"min":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":1.2150000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.51,"15min":1.5289999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":4.3620000000000001,"5min":6.0049999999999999,"15min":6.0049999999999999},"last":1.377},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.5720000000000001,"15min":1.6619999999999999},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.0169999999999999,"5min":7.798,"15min":7.798},"last":1.1990000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6870000000000001,"15min":1.7470000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":3.6800000000000002,"5min":7.968,"15min":7.968},"last":1.4690000000000001},{"interface":"front","average":{"1min":2.1850000000000001,"5min":1.6080000000000001,"15min":1.516},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":14.804,"5min":14.804,"15min":14.804},"last":1.0980000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":557056,"data_stored":225040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.109,"15min":1.071},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.5740000000000001,"5min":5.0439999999999996,"15min":5.0439999999999996},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.1259999999999999,"15min":1.125},"min":{"1min":0.68899999999999995,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":4.6210000000000004,"5min":4.6210000000000004,"15min":4.6210000000000004},"last":1.4159999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2390000000000001,"15min":1.226},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8200000000000001,"5min":1.972,"15min":1.972},"last":0.83199999999999996},{"interface":"front","average":{"1min":1.137,"5min":1.2809999999999999,"15min":1.3260000000000001},"min":{"1min":0.72499999999999998,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.7230000000000001,"5min":2.4159999999999999,"15min":2.4159999999999999},"last":1.107}]},{"osd":2,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2609999999999999,"15min":1.2709999999999999},"min":{"1min":0.86099999999999999,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.669,"5min":3.415,"15min":3.415},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.294,"15min":1.276},"min":{"1min":1.04,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.381}]},{"osd":3,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.381,"15min":1.359},"min":{"1min":0.875,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.0099999999999998,"5min":4.6559999999999997,"15min":4.6559999999999997},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.413,"15min":1.3919999999999999},"min":{"1min":0.92600000000000005,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.992,"5min":4.7199999999999998,"15min":4.7199999999999998},"last":1.3149999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.329,"15min":1.337},"min":{"1min":0.77800000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.859,"5min":2.8450000000000002,"15min":2.8450000000000002},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.4370000000000001,"15min":1.4870000000000001},"min":{"1min":0.56399999999999995,"5min":0.501,"15min":0.501},"max":{"1min":1.8500000000000001,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.744}]},{"osd":5,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.335,"15min":1.357},"min":{"1min":0.80300000000000005,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.881,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.5049999999999999,"15min":1.498},"min":{"1min":0.90200000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.925,"5min":2.3490000000000002,"15min":2.3490000000000002},"last":1.5640000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:00:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.607,"15min":1.6739999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.9339999999999999,"5min":5.6449999999999996,"15min":5.6449999999999996},"last":1.847},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5880000000000001,"15min":1.6499999999999999},"min":{"1min":1.02,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.9790000000000001,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":1.6040000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542188,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":552960,"data_stored":224438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.143,"15min":1.139},"min":{"1min":0.79100000000000004,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.5760000000000001,"5min":1.6539999999999999,"15min":1.6539999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.675,"5min":1.347,"15min":1.264},"min":{"1min":0.66700000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":7.2949999999999999,"5min":7.2949999999999999,"15min":7.2949999999999999},"last":0.76900000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.2130000000000001,"15min":1.153},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.6440000000000001,"5min":4.806,"15min":4.806},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.274,"5min":1.179,"15min":1.1180000000000001},"min":{"1min":0.72699999999999998,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.6910000000000001,"5min":4.8970000000000002,"15min":4.8970000000000002},"last":0.72999999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.3580000000000001,"15min":1.327},"min":{"1min":1.0620000000000001,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.4220000000000002,"5min":5.516,"15min":5.516},"last":1.504},{"interface":"front","average":{"1min":1.6679999999999999,"5min":1.4430000000000001,"15min":1.383},"min":{"1min":0.81499999999999995,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":7.1340000000000003,"5min":7.1340000000000003,"15min":7.1340000000000003},"last":1.226}]},{"osd":3,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.274,"15min":1.252},"min":{"1min":0.83599999999999997,"5min":0.622,"15min":0.622},"max":{"1min":1.4890000000000001,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.82199999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":1.611}]},{"osd":4,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.3580000000000001,"15min":1.3120000000000001},"min":{"1min":0.83199999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":7.2439999999999998,"5min":7.2439999999999998,"15min":7.2439999999999998},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.748,"5min":1.5429999999999999,"15min":1.4870000000000001},"min":{"1min":1.008,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":7.399,"5min":7.399,"15min":7.399},"last":1.397}]},{"osd":6,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.2230000000000001,"5min":1.3280000000000001,"15min":1.371},"min":{"1min":0.84199999999999997,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.5940000000000001,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.462},{"interface":"front","average":{"1min":1.361,"5min":1.472,"15min":1.488},"min":{"1min":0.91600000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.7849999999999999,"5min":4.9699999999999998,"15min":4.9699999999999998},"last":1.6799999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:00:31 2024","interfaces":[{"interface":"back","average":{"1min":1.907,"5min":1.6759999999999999,"15min":1.6279999999999999},"min":{"1min":1.167,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.546,"15min":1.552},"min":{"1min":1.161,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.8100000000000001,"5min":5.0910000000000002,"15min":5.0910000000000002},"last":1.639}]}]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31736,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340104,"statfs":{"total":96636764160,"available":96604266496,"internally_reserved":0,"allocated":561152,"data_stored":225980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":2.9700000000000002,"5min":1.458,"15min":1.2010000000000001},"min":{"1min":0.84299999999999997,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":21.643000000000001,"5min":21.643000000000001,"15min":21.643000000000001},"last":1.335},{"interface":"front","average":{"1min":1.3,"5min":1.1399999999999999,"15min":1.1140000000000001},"min":{"1min":0.78600000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":0.96499999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.208,"15min":1.1699999999999999},"min":{"1min":0.72899999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.2300000000000004,"5min":4.2300000000000004,"15min":4.2300000000000004},"last":1.129},{"interface":"front","average":{"1min":3.052,"5min":1.5940000000000001,"15min":1.3240000000000001},"min":{"1min":0.99299999999999999,"5min":0.754,"15min":0.754},"max":{"1min":21.440000000000001,"5min":21.440000000000001,"15min":21.440000000000001},"last":1.075}]},{"osd":2,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.216,"15min":1.1559999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":1.4179999999999999},{"interface":"front","average":{"1min":3.0289999999999999,"5min":1.6399999999999999,"15min":1.3939999999999999},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":22.350999999999999,"5min":22.350999999999999,"15min":22.350999999999999},"last":1.175}]},{"osd":3,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.4910000000000001,"15min":1.377},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":5.2169999999999996,"5min":14.657999999999999,"15min":14.657999999999999},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.631,"5min":1.4159999999999999,"15min":1.3480000000000001},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":4.4720000000000004,"5min":5.1879999999999997,"15min":5.1879999999999997},"last":0.88100000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":3.3980000000000001,"5min":1.7649999999999999,"15min":1.5029999999999999},"min":{"1min":0.90800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":21.285,"5min":21.285,"15min":21.285},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.357,"15min":1.385},"min":{"1min":0.64400000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.379,"5min":2.379,"15min":2.379},"last":1.212}]},{"osd":6,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.399,"15min":1.399},"min":{"1min":0.61799999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0979999999999999,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.5249999999999999},{"interface":"front","average":{"1min":3.1960000000000002,"5min":1.8380000000000001,"15min":1.6100000000000001},"min":{"1min":0.82199999999999995,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":21.404,"5min":21.404,"15min":21.404},"last":1.3919999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:00:29 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5,"15min":1.518},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.597,"5min":2.597,"15min":2.597},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.734,"5min":1.661,"15min":1.5569999999999999},"min":{"1min":0.88500000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.391,"5min":9.6639999999999997,"15min":9.6639999999999997},"last":1.6220000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542188,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35892,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335948,"statfs":{"total":96636764160,"available":96600010752,"internally_reserved":0,"allocated":561152,"data_stored":225006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":36174280},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":0.89800000000000002,"15min":0.81599999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.9169999999999998,"5min":4.9169999999999998,"15min":4.9169999999999998},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.88300000000000001,"15min":0.878},"min":{"1min":0.42499999999999999,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.383,"5min":1.512,"15min":1.512},"last":0.64200000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":0.995,"15min":0.995},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":2.008,"5min":2.008,"15min":2.008},"last":1.345},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.1519999999999999,"15min":1.1259999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":0.90700000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.155,"5min":1.097,"15min":1.075},"min":{"1min":0.58699999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.962,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.2569999999999999,"15min":1.2150000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":5.0910000000000002,"5min":5.0910000000000002,"15min":5.0910000000000002},"last":1.087}]},{"osd":4,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.2509999999999999,"15min":1.2549999999999999},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.7869999999999999,"5min":1.4430000000000001,"15min":1.3620000000000001},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":5.6310000000000002,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.585}]},{"osd":5,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.4079999999999999,"15min":1.3520000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.718,"5min":2.718,"15min":2.718},"last":1.708},{"interface":"front","average":{"1min":4.1989999999999998,"5min":1.9890000000000001,"15min":1.6040000000000001},"min":{"1min":0.746,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":40.384999999999998,"5min":40.384999999999998,"15min":40.384999999999998},"last":1.2050000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.3380000000000001,"15min":1.298},"min":{"1min":0.90900000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.74,"5min":1.5589999999999999,"15min":1.5209999999999999},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":5.5659999999999998,"5min":5.5659999999999998,"15min":5.5659999999999998},"last":1.6359999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.623,"15min":1.5920000000000001},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":4.9950000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":1.409},{"interface":"front","average":{"1min":1.7190000000000001,"5min":1.476,"15min":1.4259999999999999},"min":{"1min":1.054,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":4.6600000000000001,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":1.657}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35900,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335940,"statfs":{"total":96636764160,"available":96600002560,"internally_reserved":0,"allocated":565248,"data_stored":226002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0720000000000001,"15min":1.1160000000000001},"min":{"1min":0.63100000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":3.2480000000000002,"5min":3.2480000000000002,"15min":3.2480000000000002},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.1359999999999999,"15min":1.21},"min":{"1min":0.68200000000000005,"5min":0.313,"15min":0.313},"max":{"1min":1.998,"5min":2.343,"15min":2.343},"last":1.159}]},{"osd":1,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.0429999999999999,"15min":1.0069999999999999},"min":{"1min":0.499,"5min":0.379,"15min":0.379},"max":{"1min":3.9470000000000001,"5min":3.9470000000000001,"15min":3.9470000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.254},"min":{"1min":0.78100000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.8540000000000001,"5min":2.198,"15min":2.198},"last":0.996}]},{"osd":3,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.214,"15min":1.1870000000000001},"min":{"1min":0.746,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":4.1349999999999998,"5min":4.1349999999999998,"15min":4.1349999999999998},"last":1.131},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.1719999999999999,"15min":1.147},"min":{"1min":0.72399999999999998,"5min":0.442,"15min":0.442},"max":{"1min":3.379,"5min":3.379,"15min":3.379},"last":1.2170000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.786,"5min":1.5620000000000001,"15min":1.573},"min":{"1min":1.0840000000000001,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":7.0640000000000001,"5min":7.0640000000000001,"15min":7.0640000000000001},"last":1.399},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.4019999999999999,"15min":1.4199999999999999},"min":{"1min":1.0960000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.089,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.504}]},{"osd":5,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.5009999999999999,"15min":1.4730000000000001},"min":{"1min":0.92700000000000005,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":4.2850000000000001,"5min":4.2850000000000001,"15min":4.2850000000000001},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.534,"15min":1.5489999999999999},"min":{"1min":0.84399999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":3.5430000000000001,"5min":3.5430000000000001,"15min":3.5430000000000001},"last":1.5680000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.6180000000000001,"15min":1.6559999999999999},"min":{"1min":1.0089999999999999,"5min":0.87,"15min":0.87},"max":{"1min":2.827,"5min":2.8410000000000002,"15min":2.8410000000000002},"last":1.339},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.5189999999999999,"15min":1.4850000000000001},"min":{"1min":0.86899999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":4.4210000000000003,"5min":4.4210000000000003,"15min":4.4210000000000003},"last":1.623}]},{"osd":7,"last update":"Mon Jan 15 18:00:25 2024","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.633,"15min":1.6659999999999999},"min":{"1min":0.94499999999999995,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":2.214,"5min":2.613,"15min":2.613},"last":1.671},{"interface":"front","average":{"1min":1.8879999999999999,"5min":1.631,"15min":1.593},"min":{"1min":1.181,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":4.9349999999999996,"5min":4.9349999999999996,"15min":4.9349999999999996},"last":1.28}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35884,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335956,"statfs":{"total":96636764160,"available":96600018944,"internally_reserved":0,"allocated":552960,"data_stored":224438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":0.90000000000000002,"15min":0.88400000000000001},"min":{"1min":0.53000000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.8919999999999999,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.165,"5min":1.101,"15min":1.077},"min":{"1min":0.41899999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.782,"5min":3.3159999999999998,"15min":3.3159999999999998},"last":1.2929999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.736,"5min":1.107,"15min":0.97599999999999998},"min":{"1min":0.39200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":7.6479999999999997,"5min":7.6479999999999997,"15min":7.6479999999999997},"last":0.876},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0600000000000001,"15min":1.0680000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.6259999999999999,"5min":2.5470000000000002,"15min":2.5470000000000002},"last":0.94499999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8149999999999999,"5min":1.339,"15min":1.264},"min":{"1min":0.53300000000000003,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":7.8659999999999997,"5min":7.8659999999999997,"15min":7.8659999999999997},"last":1.48},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.361,"15min":1.262},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":2.226,"5min":14.071,"15min":14.071},"last":1.1100000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.397,"15min":1.399},"min":{"1min":0.80800000000000005,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.1030000000000002,"5min":2.9670000000000001,"15min":2.9670000000000001},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.833,"5min":1.55,"15min":1.508},"min":{"1min":0.81899999999999995,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":11.516,"5min":11.516,"15min":11.516},"last":1.421}]},{"osd":5,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.5660000000000001,"15min":1.5489999999999999},"min":{"1min":0.86199999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":7.8159999999999998,"5min":7.8159999999999998,"15min":7.8159999999999998},"last":1.526},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.5049999999999999,"15min":1.427},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":8.0730000000000004,"5min":8.0730000000000004,"15min":8.0730000000000004},"last":1.008}]},{"osd":6,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.889,"5min":1.5700000000000001,"15min":1.52},"min":{"1min":0.78600000000000003,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":8.0670000000000002,"5min":8.0670000000000002,"15min":8.0670000000000002},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.5800000000000001,"15min":1.5880000000000001},"min":{"1min":0.88900000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":2.5110000000000001,"5min":3.5249999999999999,"15min":3.5249999999999999},"last":1.6479999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:00:30 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.5900000000000001,"15min":1.52},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":7.9980000000000002,"5min":7.9980000000000002,"15min":7.9980000000000002},"last":1.1899999999999999},{"interface":"front","average":{"1min":2.1389999999999998,"5min":1.679,"15min":1.5980000000000001},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":8.1940000000000008,"5min":8.1940000000000008,"15min":8.1940000000000008},"last":1.2609999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32300,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339540,"statfs":{"total":96636764160,"available":96603688960,"internally_reserved":0,"allocated":1146880,"data_stored":814770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.85599999999999998,"15min":0.82299999999999995},"min":{"1min":0.55800000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.391,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.016,"15min":1.026},"min":{"1min":0.45900000000000002,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.335}]},{"osd":2,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.69199999999999995,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.663,"5min":1.9610000000000001,"15min":1.9610000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.278,"15min":1.1870000000000001},"min":{"1min":0.59699999999999998,"5min":0.502,"15min":0.502},"max":{"1min":10.942,"5min":10.942,"15min":10.942},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.054,"15min":1.036},"min":{"1min":0.55900000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":7.0949999999999998,"5min":7.0949999999999998,"15min":7.0949999999999998},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.1719999999999999,"5min":1.25,"15min":1.28},"min":{"1min":0.66300000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.736,"5min":2.9380000000000002,"15min":2.9380000000000002},"last":1.2769999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.405,"15min":1.395},"min":{"1min":0.80400000000000005,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":5.1859999999999999,"5min":5.1859999999999999,"15min":5.1859999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2949999999999999,"15min":1.323},"min":{"1min":0.82199999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":2.6800000000000002,"5min":2.6800000000000002,"15min":2.6800000000000002},"last":0.88700000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.3480000000000001,"15min":1.387},"min":{"1min":0.80300000000000005,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.169},{"interface":"front","average":{"1min":1.484,"5min":1.452,"15min":1.456},"min":{"1min":1.073,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.409}]},{"osd":6,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.46,"15min":1.468},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":1.9039999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.53},{"interface":"front","average":{"1min":1.514,"5min":1.4970000000000001,"15min":1.504},"min":{"1min":0.95299999999999996,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":2.9780000000000002,"5min":2.9780000000000002,"15min":2.9780000000000002},"last":1.4670000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:00:26 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.556,"15min":1.579},"min":{"1min":0.89500000000000002,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":3.1240000000000001,"5min":3.1240000000000001,"15min":3.1240000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.4359999999999999,"15min":1.4490000000000001},"min":{"1min":0.95799999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.3319999999999999,"5min":3.3319999999999999,"15min":3.3319999999999999},"last":1.119}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:00:45.545 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:00:45.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:00:45.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.813+0000 7f68b8c00640 1 Processor -- start 2024-01-15T18:00:45.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.813+0000 7f68b8c00640 1 -- start start 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b8c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f68b415c450 0x7f68b406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f68b4070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b8c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40ce230 con 0x7f68b4144c00 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b8c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68b406d2c0 con 0x7f68b415c450 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40ce510 con 0x7f68b406ce30 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b1e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f68b4144c00 0x7f68b40cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55952/0 (socket says 172.21.15.2:55952) 2024-01-15T18:00:45.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.815+0000 7f68b1e00640 1 -- 172.21.15.2:0/1845562083 learned_addr learned my addr 172.21.15.2:0/1845562083 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:45.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68abe00640 1 --2- 172.21.15.2:0/1845562083 >> v2:172.21.15.137:3300/0 conn(0x7f68b415c450 0x7f68b406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b2800640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f68b4070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308664077 0 0) 0x7f68b40ce230 con 0x7f68b4144c00 2024-01-15T18:00:45.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894003520 con 0x7f68b4144c00 2024-01-15T18:00:45.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3067217099 0 0) 0x7f6894003520 con 0x7f68b4144c00 2024-01-15T18:00:45.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 msgr2=0x7f68b4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:45.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f68b4070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 >> v2:172.21.15.137:3300/0 conn(0x7f68b415c450 msgr2=0x7f68b406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:45.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 --2- 172.21.15.2:0/1845562083 >> v2:172.21.15.137:3300/0 conn(0x7f68b415c450 0x7f68b406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.816+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b40ce7f0 con 0x7f68b4144c00 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f68a4001670 con 0x7f68b4144c00 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 >> v1:172.21.15.2:6789/0 conn(0x7f68b4144c00 legacy=0x7f68b40cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f6894003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6894003f40 0x7f68940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f6894005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b406d2c0 con 0x7f6894004d00 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b40ce510 con 0x7f6894003f40 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689400b980 con 0x7f68b406ce30 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b2800640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f6894005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68abe00640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6894003f40 0x7f68940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b2800640 1 -- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 msgr2=0x7f6894003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.817+0000 7f68b1e00640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f6894003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.818+0000 7f68b2800640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f6894003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.818+0000 7f68b2800640 1 -- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6894003f40 msgr2=0x7f68940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.818+0000 7f68b2800640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6894003f40 0x7f68940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.818+0000 7f68b2800640 1 -- 172.21.15.2:0/1845562083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894003520 con 0x7f6894004d00 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.818+0000 7f68abe00640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6894003f40 0x7f68940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.818+0000 7f68b1e00640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b406ce30 0x7f6894003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.819+0000 7f68b2800640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f6894005180 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00e170 tx=0x7f68940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af75b4cdf3b35c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.819+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac02cbf0 con 0x7f6894004d00 2024-01-15T18:00:45.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.819+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac00f040 con 0x7f6894004d00 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.819+0000 7f68b1400640 1 -- 172.21.15.2:0/1845562083 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac034440 con 0x7f6894004d00 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b8c00640 1 -- 172.21.15.2:0/1845562083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 msgr2=0x7f6894005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b8c00640 1 --2- 172.21.15.2:0/1845562083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f6894005180 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00e170 tx=0x7f68940057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b2800640 1 -- 172.21.15.2:0/1845562083 reap_dead start 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b8c00640 1 -- 172.21.15.2:0/1845562083 shutdown_connections 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b8c00640 1 -- 172.21.15.2:0/1845562083 >> 172.21.15.2:0/1845562083 conn(0x7f68b40b45f0 msgr2=0x7f68b40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b8c00640 1 -- 172.21.15.2:0/1845562083 shutdown_connections 2024-01-15T18:00:45.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.820+0000 7f68b8c00640 1 -- 172.21.15.2:0/1845562083 wait complete. 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.822+0000 7f68b8c00640 1 Processor -- start 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.822+0000 7f68b8c00640 1 -- start start 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f68b4141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68b4157a60 0x7f68b413ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b413f280 0x7f68b4140d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b8c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b41f3b50 con 0x7f6894004d00 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b41f2f40 con 0x7f68b4157a60 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b41f3220 con 0x7f68b413f280 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f68b4141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f68b4141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58944/0 (socket says 172.21.15.2:58944) 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 -- 172.21.15.2:0/225241363 learned_addr learned my addr 172.21.15.2:0/225241363 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b2800640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b413f280 0x7f68b4140d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68abe00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68b4157a60 0x7f68b413ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 -- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b413f280 msgr2=0x7f68b4140d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b413f280 0x7f68b4140d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 -- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68b4157a60 msgr2=0x7f68b413ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68b4157a60 0x7f68b413ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.823+0000 7f68b1e00640 1 -- 172.21.15.2:0/225241363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b41f3500 con 0x7f6894004d00 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.824+0000 7f68b1e00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f68b4141ee0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f68a400d8d0 tx=0x7f68a400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:45.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.824+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a40074e0 con 0x7f6894004d00 2024-01-15T18:00:45.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.824+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a4007640 con 0x7f6894004d00 2024-01-15T18:00:45.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.824+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68a40049e0 con 0x7f6894004d00 2024-01-15T18:00:45.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.824+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b41f37e0 con 0x7f6894004d00 2024-01-15T18:00:45.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.824+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b4157ee0 con 0x7f6894004d00 2024-01-15T18:00:45.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.826+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b41f3970 con 0x7f6894004d00 2024-01-15T18:00:45.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.826+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f68a4004b40 con 0x7f6894004d00 2024-01-15T18:00:45.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.828+0000 7f68aa000640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f688c06b530 0x7f688c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:45.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.828+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f68a4089bd0 con 0x7f6894004d00 2024-01-15T18:00:45.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.828+0000 7f68abe00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f688c06b530 0x7f688c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:45.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.829+0000 7f68abe00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f688c06b530 0x7f688c06d9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f68b4072650 tx=0x7f689c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:45.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:45.832+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68a40545b0 con 0x7f6894004d00 2024-01-15T18:00:46.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.096+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f68b40686f0 con 0x7f6894004d00 2024-01-15T18:00:46.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.101+0000 7f68aa000640 1 -- 172.21.15.2:0/225241363 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f68a4019020 con 0x7f6894004d00 2024-01-15T18:00:46.106 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T18:00:46.106 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T18:00:46.106 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T18:00:46.106 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T18:00:46.106 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:00:46.107 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f688c06b530 msgr2=0x7f688c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f688c06b530 0x7f688c06d9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f68b4072650 tx=0x7f689c008040 comp rx=0 tx=0).stop 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 msgr2=0x7f68b4141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f68b4141ee0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f68a400d8d0 tx=0x7f68a400dd90 comp rx=0 tx=0).stop 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 shutdown_connections 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f688c06b530 0x7f688c06d9f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68b413f280 0x7f68b4140d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68b4157a60 0x7f68b413ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 --2- 172.21.15.2:0/225241363 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6894004d00 0x7f68b4141ee0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 >> 172.21.15.2:0/225241363 conn(0x7f68b40b45f0 msgr2=0x7f68b40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.104+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 shutdown_connections 2024-01-15T18:00:46.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.105+0000 7f68b8c00640 1 -- 172.21.15.2:0/225241363 wait complete. 2024-01-15T18:00:46.129 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:00:46.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.346+0000 7f38e6200640 1 Processor -- start 2024-01-15T18:00:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.346+0000 7f38e6200640 1 -- start start 2024-01-15T18:00:46.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.348+0000 7f38e6200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f38e015d340 0x7f38e00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:46.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.348+0000 7f38e6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 0x7f38e00d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:46.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.348+0000 7f38e6200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e00d3a60 con 0x7f38e00d3090 2024-01-15T18:00:46.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.348+0000 7f38e6200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e00b3de0 con 0x7f38e015d340 2024-01-15T18:00:46.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.348+0000 7f38e6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00b40c0 con 0x7f38e00d1d90 2024-01-15T18:00:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.349+0000 7f38e5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 0x7f38e00d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.349+0000 7f38e4e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f38e015d340 0x7f38e00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.349+0000 7f38e5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 0x7f38e00d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:57194/0 (socket says 172.21.15.2:57194) 2024-01-15T18:00:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.349+0000 7f38e5800640 1 -- 172.21.15.2:0/2324944850 learned_addr learned my addr 172.21.15.2:0/2324944850 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.349+0000 7f38d7e00640 1 --1- 172.21.15.2:0/2324944850 >> v1:172.21.15.137:6789/0 conn(0x7f38e00d3090 0x7f38e00d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:33930/0 (socket says 172.21.15.2:33930) 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.349+0000 7f38e4e00640 1 --2- 172.21.15.2:0/2324944850 >> v2:172.21.15.137:3300/0 conn(0x7f38e015d340 0x7f38e00cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:52874/0 (socket says 172.21.15.2:52874) 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.350+0000 7f38e4e00640 1 -- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 msgr2=0x7f38e00d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.350+0000 7f38e4e00640 1 --2- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 0x7f38e00d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.350+0000 7f38e4e00640 1 -- 172.21.15.2:0/2324944850 >> v1:172.21.15.137:6789/0 conn(0x7f38e00d3090 legacy=0x7f38e00d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.350+0000 7f38e4e00640 1 -- 172.21.15.2:0/2324944850 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e00d0e80 con 0x7f38e015d340 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.350+0000 7f38dfe00640 1 -- 172.21.15.2:0/2324944850 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063417133 0 0) 0x7f38e00d3a60 con 0x7f38e00d3090 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.350+0000 7f38e5800640 1 --2- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 0x7f38e00d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:00:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.352+0000 7f38e4e00640 1 --2- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e015d340 0x7f38e00cdd20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f38d00097a0 tx=0x7f38d002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a08c3f5c2f1ddf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38dfe00640 1 -- 172.21.15.2:0/2324944850 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d002d040 con 0x7f38e015d340 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38dfe00640 1 -- 172.21.15.2:0/2324944850 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d0031810 con 0x7f38e015d340 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38dfe00640 1 -- 172.21.15.2:0/2324944850 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0031ba0 con 0x7f38e015d340 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38e6200640 1 -- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e015d340 msgr2=0x7f38e00cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38e6200640 1 --2- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e015d340 0x7f38e00cdd20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f38d00097a0 tx=0x7f38d002b8b0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38e6200640 1 -- 172.21.15.2:0/2324944850 shutdown_connections 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.353+0000 7f38e6200640 1 --2- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d1d90 0x7f38e00d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.354+0000 7f38e6200640 1 --2- 172.21.15.2:0/2324944850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e015d340 0x7f38e00cdd20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.354+0000 7f38e6200640 1 -- 172.21.15.2:0/2324944850 >> 172.21.15.2:0/2324944850 conn(0x7f38e008caf0 msgr2=0x7f38e009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.354+0000 7f38e6200640 1 -- 172.21.15.2:0/2324944850 shutdown_connections 2024-01-15T18:00:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.354+0000 7f38e6200640 1 -- 172.21.15.2:0/2324944850 wait complete. 2024-01-15T18:00:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.355+0000 7f38e6200640 1 Processor -- start 2024-01-15T18:00:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 -- start start 2024-01-15T18:00:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e00d1d90 0x7f38e01f7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d3090 0x7f38e0146c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 0x7f38e015b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e01467e0 con 0x7f38e015d340 2024-01-15T18:00:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e01f7590 con 0x7f38e00d1d90 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.356+0000 7f38e6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e00cd060 con 0x7f38e00d3090 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 0x7f38e015b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 0x7f38e015b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58960/0 (socket says 172.21.15.2:58960) 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 -- 172.21.15.2:0/339892244 learned_addr learned my addr 172.21.15.2:0/339892244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e4e00640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e00d1d90 0x7f38e01f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38d7e00640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d3090 0x7f38e0146c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 -- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d3090 msgr2=0x7f38e0146c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d3090 0x7f38e0146c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 -- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e00d1d90 msgr2=0x7f38e01f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e00d1d90 0x7f38e01f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.357+0000 7f38e5800640 1 -- 172.21.15.2:0/339892244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d8009b30 con 0x7f38e015d340 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.358+0000 7f38e5800640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 0x7f38e015b270 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f38d800efc0 tx=0x7f38d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.358+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d8018070 con 0x7f38e015d340 2024-01-15T18:00:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.358+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d800f040 con 0x7f38e015d340 2024-01-15T18:00:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.358+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d0009400 con 0x7f38e015d340 2024-01-15T18:00:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.358+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d800ce00 con 0x7f38e015d340 2024-01-15T18:00:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.358+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e00ccd80 con 0x7f38e015d340 2024-01-15T18:00:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.360+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e0146990 con 0x7f38e015d340 2024-01-15T18:00:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.360+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f38d8014480 con 0x7f38e015d340 2024-01-15T18:00:46.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.362+0000 7f38dd600640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38b806b530 0x7f38b806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:46.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.362+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f38d808a2e0 con 0x7f38e015d340 2024-01-15T18:00:46.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.363+0000 7f38e4e00640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38b806b530 0x7f38b806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:46.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.364+0000 7f38e4e00640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38b806b530 0x7f38b806d9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f38d0036000 tx=0x7f38d003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:46.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.366+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38e0146990 con 0x7f38e015d340 2024-01-15T18:00:46.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.600+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38e0074770 con 0x7f38e015d340 2024-01-15T18:00:46.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.601+0000 7f38dd600640 1 -- 172.21.15.2:0/339892244 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f38d8054d50 con 0x7f38e015d340 2024-01-15T18:00:46.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:00:46.606 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":26,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1873476051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1873476051},{"type":"v1","addr":"172.21.15.2:6835","nonce":1873476051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1702488249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1702488249},{"type":"v1","addr":"172.21.15.137:6836","nonce":1702488249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5251,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/4131491202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":4131491202},{"type":"v1","addr":"172.21.15.137:6837","nonce":4131491202}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:00:39.959212+0000","modified":"2024-01-15T18:00:43.038891+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5721},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5721":{"gid":5721,"name":"c","rank":0,"incarnation":22,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6833/4162570290","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":4162570290},{"type":"v1","addr":"172.21.15.2:6833","nonce":4162570290}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-15T18:00:46.606 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 26 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38b806b530 msgr2=0x7f38b806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38b806b530 0x7f38b806d9f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f38d0036000 tx=0x7f38d003c040 comp rx=0 tx=0).stop 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 msgr2=0x7f38e015b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 0x7f38e015b270 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f38d800efc0 tx=0x7f38d800c490 comp rx=0 tx=0).stop 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 shutdown_connections 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38b806b530 0x7f38b806d9f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38e015d340 0x7f38e015b270 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38e00d3090 0x7f38e0146c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 --2- 172.21.15.2:0/339892244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38e00d1d90 0x7f38e01f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 >> 172.21.15.2:0/339892244 conn(0x7f38e008caf0 msgr2=0x7f38e00b4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:46.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 shutdown_connections 2024-01-15T18:00:46.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:46.604+0000 7f38e6200640 1 -- 172.21.15.2:0/339892244 wait complete. 2024-01-15T18:00:47.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:00:39.959212+0000', 'modified': '2024-01-15T18:00:43.038891+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5721}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5721': {'gid': 5721, 'name': 'c', 'rank': 0, 'incarnation': 22, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.2:6833/4162570290', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6832', 'nonce': 4162570290}, {'type': 'v1', 'addr': '172.21.15.2:6833', 'nonce': 4162570290}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:00:47.031 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:00:47.031 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-15T18:00:47.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.261+0000 7f79f3200640 1 Processor -- start 2024-01-15T18:00:47.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.261+0000 7f79f3200640 1 -- start start 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f79f4160e50 0x7f79f40cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f415d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f3200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f40cb3a0 con 0x7f79f406d350 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f3200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f40cb680 con 0x7f79f4160e50 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f40cb960 con 0x7f79f40ce3b0 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f79f4160e50 0x7f79f40cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f79f4160e50 0x7f79f40cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58970/0 (socket says 172.21.15.2:58970) 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f1400640 1 -- 172.21.15.2:0/2269880492 learned_addr learned my addr 172.21.15.2:0/2269880492 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f2800640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f415d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f1400640 1 -- 172.21.15.2:0/2269880492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 msgr2=0x7f79f415d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.263+0000 7f79f1400640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f415d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.264+0000 7f79f1400640 1 -- 172.21.15.2:0/2269880492 >> v1:172.21.15.137:6789/0 conn(0x7f79f406d350 legacy=0x7f79f406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.264+0000 7f79f1400640 1 -- 172.21.15.2:0/2269880492 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f40cbc40 con 0x7f79f4160e50 2024-01-15T18:00:47.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.264+0000 7f79f2800640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f415d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:47.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.265+0000 7f79f1400640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40cde70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009870 tx=0x7f79dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c0aeba971eaaac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.265+0000 7f79f0a00640 1 -- 172.21.15.2:0/2269880492 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc031640 con 0x7f79f4160e50 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.265+0000 7f79f0a00640 1 -- 172.21.15.2:0/2269880492 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc0317a0 con 0x7f79f4160e50 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.265+0000 7f79f0a00640 1 -- 172.21.15.2:0/2269880492 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79dc031b30 con 0x7f79f4160e50 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 -- 172.21.15.2:0/2269880492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 msgr2=0x7f79f40cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40cde70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f79dc009870 tx=0x7f79dc02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 -- 172.21.15.2:0/2269880492 shutdown_connections 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f415d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 --2- 172.21.15.2:0/2269880492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40cde70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 -- 172.21.15.2:0/2269880492 >> 172.21.15.2:0/2269880492 conn(0x7f79f40b0840 msgr2=0x7f79f40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 -- 172.21.15.2:0/2269880492 shutdown_connections 2024-01-15T18:00:47.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.266+0000 7f79f3200640 1 -- 172.21.15.2:0/2269880492 wait complete. 2024-01-15T18:00:47.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.268+0000 7f79f3200640 1 Processor -- start 2024-01-15T18:00:47.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.268+0000 7f79f3200640 1 -- start start 2024-01-15T18:00:47.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79f406d350 0x7f79f41520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f4070370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f4071e20 con 0x7f79f4160e50 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f414e570 con 0x7f79f406d350 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f415dbc0 con 0x7f79f40ce3b0 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:58986/0 (socket says 172.21.15.2:58986) 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.270+0000 7f79f2800640 1 -- 172.21.15.2:0/4226781704 learned_addr learned my addr 172.21.15.2:0/4226781704 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f1e00640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79f406d350 0x7f79f41520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f1400640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f4070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f2800640 1 -- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 msgr2=0x7f79f4070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f2800640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f4070370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f2800640 1 -- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79f406d350 msgr2=0x7f79f41520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f2800640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79f406d350 0x7f79f41520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f2800640 1 -- 172.21.15.2:0/4226781704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc0094d0 con 0x7f79f4160e50 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f2800640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40708b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00cef0 tx=0x7f79ec00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79ea000640 1 -- 172.21.15.2:0/4226781704 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00d840 con 0x7f79f4160e50 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79ea000640 1 -- 172.21.15.2:0/4226781704 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec00d9c0 con 0x7f79f4160e50 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79ea000640 1 -- 172.21.15.2:0/4226781704 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec00dd70 con 0x7f79f4160e50 2024-01-15T18:00:47.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.271+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f415d890 con 0x7f79f4160e50 2024-01-15T18:00:47.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.272+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f4071b40 con 0x7f79f4160e50 2024-01-15T18:00:47.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.275+0000 7f79ea000640 1 -- 172.21.15.2:0/4226781704 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f79ec01e3b0 con 0x7f79f4160e50 2024-01-15T18:00:47.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.276+0000 7f79ea000640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79cc06b500 0x7f79cc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.276+0000 7f79ea000640 1 -- 172.21.15.2:0/4226781704 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f79ec08ab50 con 0x7f79f4160e50 2024-01-15T18:00:47.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.277+0000 7f79fb9e4740 1 Processor -- start 2024-01-15T18:00:47.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.277+0000 7f79f1e00640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79cc06b500 0x7f79cc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.277+0000 7f79fb9e4740 1 -- start start 2024-01-15T18:00:47.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819acf070 0x55a819acf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ace270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 0x55a819b12730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819a2d880 con 0x55a819ace7b0 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819a3cae0 con 0x55a819ac9d70 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a819a3a1d0 con 0x55a819acf070 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79f1e00640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79cc06b500 0x7f79cc06d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f79e4003d90 tx=0x7f79e4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 0x55a819b12730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 0x55a819b12730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59002/0 (socket says 172.21.15.2:59002) 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.278+0000 7f79f2800640 1 -- 172.21.15.2:0/3621837304 learned_addr learned my addr 172.21.15.2:0/3621837304 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f1e00640 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819acf070 0x55a819acf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f1400640 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ace270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f2800640 1 -- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819acf070 msgr2=0x55a819acf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f2800640 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819acf070 0x55a819acf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f2800640 1 -- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 msgr2=0x55a819ace270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f2800640 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ace270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f2800640 1 -- 172.21.15.2:0/3621837304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8199fd860 con 0x55a819ace7b0 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79f2800640 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 0x55a819b12730 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0592b0 tx=0x7f79ec055790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c007fb3197e853c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79e2a00640 1 -- 172.21.15.2:0/3621837304 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec061620 con 0x55a819ace7b0 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79e2a00640 1 -- 172.21.15.2:0/3621837304 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec05d170 con 0x55a819ace7b0 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.279+0000 7f79e2a00640 1 -- 172.21.15.2:0/3621837304 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec058810 con 0x55a819ace7b0 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 -- 172.21.15.2:0/3621837304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 msgr2=0x55a819b12730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 0x55a819b12730 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0592b0 tx=0x7f79ec055790 comp rx=0 tx=0).stop 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 -- 172.21.15.2:0/3621837304 shutdown_connections 2024-01-15T18:00:47.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ace7b0 0x55a819b12730 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ace270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/3621837304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819acf070 0x55a819acf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 -- 172.21.15.2:0/3621837304 >> 172.21.15.2:0/3621837304 conn(0x55a819a90c70 msgr2=0x55a819b14b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 -- 172.21.15.2:0/3621837304 shutdown_connections 2024-01-15T18:00:47.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.280+0000 7f79fb9e4740 1 -- 172.21.15.2:0/3621837304 wait complete. 2024-01-15T18:00:47.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.281+0000 7f79fb9e4740 1 -- start start 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.281+0000 7f79fb9e4740 1 Processor -- start 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819af7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819af9d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819afa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819a35ac0 con 0x55a819acf070 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8199d22f0 con 0x55a819ac9d70 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a8199efc70 con 0x55a819ace7b0 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.282+0000 7f79fb9e4740 10 client.0 _ll_register_callbacks cb 0x55a8199f43b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819afa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819afa2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59018/0 (socket says 172.21.15.2:59018) 2024-01-15T18:00:47.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 -- 172.21.15.2:0/169754819 learned_addr learned my addr 172.21.15.2:0/169754819 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f1400640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819af9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f1e00640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819af7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 -- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 msgr2=0x55a819af9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819af9d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 -- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 msgr2=0x55a819af7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819af7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 -- 172.21.15.2:0/169754819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f4071b40 con 0x55a819acf070 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79f2800640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819afa2b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0594e0 tx=0x7f79ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5681cc856d2ece6c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.283+0000 7f79db400640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79db400640 1 -- 172.21.15.2:0/169754819 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec061620 con 0x55a819acf070 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79db400640 1 -- 172.21.15.2:0/169754819 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f79ec061190 con 0x55a819acf070 2024-01-15T18:00:47.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a819a2eaa0 con 0x55a819acf070 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79db400640 1 -- 172.21.15.2:0/169754819 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79ec061350 con 0x55a819acf070 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 10 client.5850 fetch_fsmap learned FSMap version 26 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55a819a34140 con 0x55a819acf070 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79db400640 1 -- 172.21.15.2:0/169754819 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a819a2eaa0 con 0x55a819acf070 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79db400640 1 -- 172.21.15.2:0/169754819 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f79ec008040 con 0x55a819acf070 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 10 client.5850 fetch_fsmap finished waiting for FSMap version 26 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 10 client.5850 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 20 client.5850 populate_metadata read hostname 'smithi002' 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819af6060 0x55a819b32240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 4 client.5850 mds_command: new command op to 5721 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.284+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 --> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a819a2e7c0 con 0x55a819af6060 2024-01-15T18:00:47.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.285+0000 7f79f1e00640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819af6060 0x55a819b32240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.285+0000 7f79f1e00640 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819af6060 0x55a819b32240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60d156ad1bdf4090 server_cookie=10437fedbdf9d57c in_seq=0 out_seq=0 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.285+0000 7f79db400640 10 client.5850 ms_handle_connect on v2:172.21.15.2:6832/4162570290 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79db400640 1 -- 172.21.15.2:0/169754819 <== mds.0 v2:172.21.15.2:6832/4162570290 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a819a2e7c0 con 0x55a819af6060 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79db400640 10 client.5850 handle_command_reply: tid=0 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 client.5850 shutdown 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 msgr2=0x55a819afa2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819afa2b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0594e0 tx=0x7f79ec004a40 comp rx=0 tx=0).stop 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 shutdown_connections 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819af6060 0x55a819b32240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819afa2b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819af9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/169754819 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819af7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79db400640 0 client.5850 ms_handle_reset on v2:172.21.15.2:6832/4162570290 2024-01-15T18:00:47.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.286+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 >> 172.21.15.2:0/169754819 conn(0x55a819a90c70 msgr2=0x55a819b17ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.287+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 shutdown_connections 2024-01-15T18:00:47.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.287+0000 7f79fb9e4740 1 -- 172.21.15.2:0/169754819 wait complete. 2024-01-15T18:00:47.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.287+0000 7f79fb9e4740 20 client.5850 trim_cache size 0 max 16384 2024-01-15T18:00:47.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.304+0000 7f79fb9e4740 1 Processor -- start 2024-01-15T18:00:47.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.304+0000 7f79fb9e4740 1 -- start start 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ac8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819ac9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819af9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819798f20 con 0x55a819acf070 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819afb960 con 0x55a819ac9d70 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a8199e9e30 con 0x55a819ace7b0 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819af9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819af9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59024/0 (socket says 172.21.15.2:59024) 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.305+0000 7f79f2800640 1 -- 172.21.15.2:0/367268279 learned_addr learned my addr 172.21.15.2:0/367268279 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79f2800640 1 -- 172.21.15.2:0/367268279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 msgr2=0x55a819ac9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79f2800640 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819ac9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79f2800640 1 -- 172.21.15.2:0/367268279 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 msgr2=0x55a819ac8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79f2800640 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ac8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79f2800640 1 -- 172.21.15.2:0/367268279 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a8199d1b80 con 0x55a819acf070 2024-01-15T18:00:47.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79f2800640 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819af9d20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f79ec059020 tx=0x7f79ec084070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db1338b229956e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79e3400640 1 -- 172.21.15.2:0/367268279 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec058e60 con 0x55a819acf070 2024-01-15T18:00:47.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79e3400640 1 -- 172.21.15.2:0/367268279 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec088e00 con 0x55a819acf070 2024-01-15T18:00:47.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.306+0000 7f79e3400640 1 -- 172.21.15.2:0/367268279 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec061190 con 0x55a819acf070 2024-01-15T18:00:47.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 -- 172.21.15.2:0/367268279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 msgr2=0x55a819af9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819af9d20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f79ec059020 tx=0x7f79ec084070 comp rx=0 tx=0).stop 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 -- 172.21.15.2:0/367268279 shutdown_connections 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819acf070 0x55a819af9d20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819ac9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/367268279 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819ac9d70 0x55a819ac8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 -- 172.21.15.2:0/367268279 >> 172.21.15.2:0/367268279 conn(0x55a819a90c70 msgr2=0x55a819acf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 -- 172.21.15.2:0/367268279 shutdown_connections 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.307+0000 7f79fb9e4740 1 -- 172.21.15.2:0/367268279 wait complete. 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.308+0000 7f79fb9e4740 1 -- start start 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.308+0000 7f79fb9e4740 1 Processor -- start 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 0x55a819b87170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819b32240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819acf070 0x55a819af9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819b32ab0 con 0x55a819ac9d70 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a819af7750 con 0x55a819acf070 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a819b23410 con 0x55a819ace7b0 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79fb9e4740 10 client.0 _ll_register_callbacks cb 0x7f79ec085070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 0x55a819b87170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 0x55a819b87170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59032/0 (socket says 172.21.15.2:59032) 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 -- 172.21.15.2:0/2285522672 learned_addr learned my addr 172.21.15.2:0/2285522672 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 -- 172.21.15.2:0/2285522672 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 msgr2=0x55a819b32240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819b32240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 -- 172.21.15.2:0/2285522672 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819acf070 msgr2=0x55a819af9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819acf070 0x55a819af9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.309+0000 7f79f1e00640 1 -- 172.21.15.2:0/2285522672 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a819a34140 con 0x55a819ac9d70 2024-01-15T18:00:47.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79f1e00640 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 0x55a819b87170 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55a819af7590 tx=0x7f79e4003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7791edd43fb7ec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79db400640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:00:47.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79db400640 1 -- 172.21.15.2:0/2285522672 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e4014870 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79db400640 1 -- 172.21.15.2:0/2285522672 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f79e40149d0 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79ec05d170 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79db400640 1 -- 172.21.15.2:0/2285522672 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79e4014b90 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79fb9e4740 10 client.5856 fetch_fsmap learned FSMap version 26 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.310+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=2+}) v3 -- 0x55a8199f7170 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79db400640 1 -- 172.21.15.2:0/2285522672 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec05d170 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79db400640 1 -- 172.21.15.2:0/2285522672 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 26) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f79e4014cf0 con 0x55a819ac9d70 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79fb9e4740 10 client.5856 fetch_fsmap finished waiting for FSMap version 26 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79fb9e4740 10 client.5856 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79fb9e4740 20 client.5856 populate_metadata read hostname 'smithi002' 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819b13f10 0x55a819b142f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79fb9e4740 4 client.5856 mds_command: new command op to 5721 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 --> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a8197366c0 con 0x55a819b13f10 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.311+0000 7f79f1400640 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819b13f10 0x55a819b142f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79f1400640 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819b13f10 0x55a819b142f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80aad8871e31fb8c server_cookie=e2974f20ed3e3b74 in_seq=0 out_seq=0 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79db400640 10 client.5856 ms_handle_connect on v2:172.21.15.2:6832/4162570290 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79db400640 1 -- 172.21.15.2:0/2285522672 <== mds.0 v2:172.21.15.2:6832/4162570290 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a8197366c0 con 0x55a819b13f10 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79db400640 10 client.5856 handle_command_reply: tid=0 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 client.5856 shutdown 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 msgr2=0x55a819b87170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 0x55a819b87170 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x55a819af7590 tx=0x7f79e4003760 comp rx=0 tx=0).stop 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 shutdown_connections 2024-01-15T18:00:47.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:6832/4162570290,v1:172.21.15.2:6833/4162570290] conn(0x55a819b13f10 0x55a819b142f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55a819acf070 0x55a819af9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55a819ace7b0 0x55a819b32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 --2- 172.21.15.2:0/2285522672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55a819ac9d70 0x55a819b87170 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79db400640 0 client.5856 ms_handle_reset on v2:172.21.15.2:6832/4162570290 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 >> 172.21.15.2:0/2285522672 conn(0x55a819a90c70 msgr2=0x55a819afae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 shutdown_connections 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 1 -- 172.21.15.2:0/2285522672 wait complete. 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.312+0000 7f79fb9e4740 20 client.5856 trim_cache size 0 max 16384 2024-01-15T18:00:47.321 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5721, 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 6.7995243309999998, 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 26, 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 47, 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 47, 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 12.199618212000001 2024-01-15T18:00:47.322 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:47.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79cc06b500 msgr2=0x7f79cc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79cc06b500 0x7f79cc06d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f79e4003d90 tx=0x7f79e4009040 comp rx=0 tx=0).stop 2024-01-15T18:00:47.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 msgr2=0x7f79f40708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40708b0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00cef0 tx=0x7f79ec00a710 comp rx=0 tx=0).stop 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 shutdown_connections 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79cc06b500 0x7f79cc06d9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79f4160e50 0x7f79f40708b0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79f40ce3b0 0x7f79f4070370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 --2- 172.21.15.2:0/4226781704 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79f406d350 0x7f79f41520f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 >> 172.21.15.2:0/4226781704 conn(0x7f79f40b0840 msgr2=0x7f79f40b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 shutdown_connections 2024-01-15T18:00:47.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.313+0000 7f79f3200640 1 -- 172.21.15.2:0/4226781704 wait complete. 2024-01-15T18:00:47.335 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:00:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.553+0000 7f42db600640 1 Processor -- start 2024-01-15T18:00:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.554+0000 7f42db600640 1 -- start start 2024-01-15T18:00:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42db600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42d406cb80 0x7f42d4166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42db600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d415cb00 con 0x7f42d415b7d0 2024-01-15T18:00:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42db600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d40ccec0 con 0x7f42d415c0e0 2024-01-15T18:00:47.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42db600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d40cccd0 con 0x7f42d406cb80 2024-01-15T18:00:47.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42cbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f42d415b7d0 0x7f42d406c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55966/0 (socket says 172.21.15.2:55966) 2024-01-15T18:00:47.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42cbe00640 1 -- 172.21.15.2:0/4145321408 learned_addr learned my addr 172.21.15.2:0/4145321408 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:47.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.556+0000 7f42d9600640 1 --2- 172.21.15.2:0/4145321408 >> v2:172.21.15.2:3300/0 conn(0x7f42d406cb80 0x7f42d4166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.557+0000 7f42d9600640 1 -- 172.21.15.2:0/4145321408 >> v1:172.21.15.137:6789/0 conn(0x7f42d415c0e0 legacy=0x7f42d40cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:47.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.557+0000 7f42d9600640 1 -- 172.21.15.2:0/4145321408 >> v1:172.21.15.2:6789/0 conn(0x7f42d415b7d0 legacy=0x7f42d406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.557+0000 7f42d9600640 1 -- 172.21.15.2:0/4145321408 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d40cd1a0 con 0x7f42d406cb80 2024-01-15T18:00:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.558+0000 7f42d9600640 1 --2- 172.21.15.2:0/4145321408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d4166070 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009d90 tx=0x7f42cc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f968aea9a9a74d1c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.558+0000 7f42d3e00640 1 -- 172.21.15.2:0/4145321408 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc02cc60 con 0x7f42d406cb80 2024-01-15T18:00:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.558+0000 7f42d3e00640 1 -- 172.21.15.2:0/4145321408 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc02cdc0 con 0x7f42d406cb80 2024-01-15T18:00:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.558+0000 7f42d3e00640 1 -- 172.21.15.2:0/4145321408 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc034660 con 0x7f42d406cb80 2024-01-15T18:00:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 -- 172.21.15.2:0/4145321408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 msgr2=0x7f42d4166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 --2- 172.21.15.2:0/4145321408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d4166070 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009d90 tx=0x7f42cc02beb0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 -- 172.21.15.2:0/4145321408 shutdown_connections 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 --2- 172.21.15.2:0/4145321408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d4166070 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 -- 172.21.15.2:0/4145321408 >> 172.21.15.2:0/4145321408 conn(0x7f42d406da20 msgr2=0x7f42d406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 -- 172.21.15.2:0/4145321408 shutdown_connections 2024-01-15T18:00:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.559+0000 7f42db600640 1 -- 172.21.15.2:0/4145321408 wait complete. 2024-01-15T18:00:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.561+0000 7f42db600640 1 Processor -- start 2024-01-15T18:00:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.562+0000 7f42db600640 1 -- start start 2024-01-15T18:00:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42db600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d413f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42db600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d415b7d0 0x7f42d41552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42db600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415c0e0 0x7f42d40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42db600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d41f76c0 con 0x7f42d406cb80 2024-01-15T18:00:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42db600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d41f6ab0 con 0x7f42d415b7d0 2024-01-15T18:00:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42db600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d41f6d90 con 0x7f42d415c0e0 2024-01-15T18:00:47.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42d8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d413f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42d8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d413f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59048/0 (socket says 172.21.15.2:59048) 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.563+0000 7f42d8c00640 1 -- 172.21.15.2:0/1679654869 learned_addr learned my addr 172.21.15.2:0/1679654869 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d8c00640 1 -- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415c0e0 msgr2=0x7f42d40b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d8c00640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415c0e0 0x7f42d40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d8c00640 1 -- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d415b7d0 msgr2=0x7f42d41552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d8c00640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d415b7d0 0x7f42d41552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d8c00640 1 -- 172.21.15.2:0/1679654869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc009a70 con 0x7f42d406cb80 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d8c00640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d413f7e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f42c400b3f0 tx=0x7f42c400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4011830 con 0x7f42d406cb80 2024-01-15T18:00:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c4011990 con 0x7f42d406cb80 2024-01-15T18:00:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4011d40 con 0x7f42d406cb80 2024-01-15T18:00:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.564+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d41f7070 con 0x7f42d406cb80 2024-01-15T18:00:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.565+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d413fd20 con 0x7f42d406cb80 2024-01-15T18:00:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.566+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d40686f0 con 0x7f42d406cb80 2024-01-15T18:00:47.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.567+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f42c401a400 con 0x7f42d406cb80 2024-01-15T18:00:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.568+0000 7f42d1600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42ac06b550 0x7f42ac06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.568+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f42c4089de0 con 0x7f42d406cb80 2024-01-15T18:00:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.568+0000 7f42cbe00640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42ac06b550 0x7f42ac06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.569+0000 7f42cbe00640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42ac06b550 0x7f42ac06da10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f42d41f6970 tx=0x7f42bc007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:47.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.572+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42c40549b0 con 0x7f42d406cb80 2024-01-15T18:00:47.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.805+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f42d415e800 con 0x7f42d406cb80 2024-01-15T18:00:47.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.806+0000 7f42d1600640 1 -- 172.21.15.2:0/1679654869 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7f42c4015070 con 0x7f42d406cb80 2024-01-15T18:00:47.810 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42ac06b550 msgr2=0x7f42ac06da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42ac06b550 0x7f42ac06da10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f42d41f6970 tx=0x7f42bc007a40 comp rx=0 tx=0).stop 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 msgr2=0x7f42d413f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d413f7e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f42c400b3f0 tx=0x7f42c400b8b0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 shutdown_connections 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42ac06b550 0x7f42ac06da10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42d415c0e0 0x7f42d40b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42d415b7d0 0x7f42d41552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 --2- 172.21.15.2:0/1679654869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42d406cb80 0x7f42d413f7e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:47.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.808+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 >> 172.21.15.2:0/1679654869 conn(0x7f42d406da20 msgr2=0x7f42d40703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:47.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.809+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 shutdown_connections 2024-01-15T18:00:47.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:47.809+0000 7f42db600640 1 -- 172.21.15.2:0/1679654869 wait complete. 2024-01-15T18:00:47.832 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:00:47.832 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:00:47.833 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:00:47.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:47.834 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T18:00:47.889 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:00:47.889 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:00:47.889 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:00:47.889 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T18:00:47.890 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42434sec preferred_lft 42434sec 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T18:00:47.891 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T18:00:47.892 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:47.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:47.893 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:00:47.987 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T18:00:47.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:47.992 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T18:00:48.047 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T18:00:48.047 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:00:48.047 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T18:00:48.047 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T18:00:48.047 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:00:48.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:48.048 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T18:00:48.048 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T18:00:48.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:00:48.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:00:48.049 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:00:48.049 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T18:00:48.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:48.249 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:00:48.303 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:00:48.303 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:00:48.303 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:00:48.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:48.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T18:00:48.404 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:00:48.404 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:00:48.404 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:48.459 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:00:48.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:00:48.459 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:48.512 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:00:48.608 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:00:48.664 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:00:48.664 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:00:48.664 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:00:48.664 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:00:48.665 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:00:48.666 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:48.667 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:48.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:48.668 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:00:48.723 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:00:48.723 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:00:48.729 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:00:48.813 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:00:48.859 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:00:48.860 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:00:48.860 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:00:48.860 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:00:48.860 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:00:48.860 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:00:48.860 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:00:48.861 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:00:48.861 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:00:48.861 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:00:48.861 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:00:48.861 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:00:48.861 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:00:48.862 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:00:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:00:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:00:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:00:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:00:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:00:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:00:48.864 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:00:48.864 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:00:48.864 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:00:48.864 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:48.864 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:48.864 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:48.865 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:48.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:48.866 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:00:48.926 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T18:00:48.922+0000 7fce21a1f480 -1 init, newargv = 0x55d3026f5b70 newargc=15 2024-01-15T18:00:48.926 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[52707]: starting ceph client 2024-01-15T18:00:48.935 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[52707]: starting fuse 2024-01-15T18:00:50.007 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:00:50.114 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:00:50.175 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:00:50.176 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:00:50.177 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:50.178 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:00:50.178 INFO:teuthology.orchestra.run.smithi002.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:50.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:50.179 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:00:50.235 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T18:00:50.235 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T18:00:50.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:00:50.235 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T18:00:50.235 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T18:00:50.235 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T18:00:50.235 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:00:50.236 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T18:00:50.237 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T18:00:50.424 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.52707.asok 2024-01-15T18:00:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:00:50.444 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52707.asok 2024-01-15T18:00:50.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:50.444 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52707.asok status 2024-01-15T18:00:50.692 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "52707", 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T18:00:50.693 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5865, 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 5865 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:00:50.694 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 4245497230 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 4245497230 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.5865 192.168.0.1:0/4245497230", 2024-01-15T18:00:50.695 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/4245497230", 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 26, 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 47, 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T18:00:50.696 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:00:50.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:50.708 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T18:00:50.708 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:50.764 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T18:00:50.764 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:50.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:50.765 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:00:50.865 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:00:51.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.082+0000 7fe547000640 1 Processor -- start 2024-01-15T18:00:51.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.083+0000 7fe547000640 1 -- start start 2024-01-15T18:00:51.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.084+0000 7fe547000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe54015d340 0x7fe5400cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:51.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.084+0000 7fe547000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 0x7fe5400d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:51.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.084+0000 7fe547000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5400d3a60 con 0x7fe5400d3090 2024-01-15T18:00:51.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.084+0000 7fe547000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5400b3de0 con 0x7fe54015d340 2024-01-15T18:00:51.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.084+0000 7fe547000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400b40c0 con 0x7fe5400d1d90 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.085+0000 7fe546600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 0x7fe5400d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.085+0000 7fe545c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe54015d340 0x7fe5400cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.085+0000 7fe546600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 0x7fe5400d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:57270/0 (socket says 172.21.15.2:57270) 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.085+0000 7fe545c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe54015d340 0x7fe5400cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:52954/0 (socket says 172.21.15.2:52954) 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.085+0000 7fe546600640 1 -- 172.21.15.2:0/2034809800 learned_addr learned my addr 172.21.15.2:0/2034809800 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.086+0000 7fe545c00640 1 -- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 msgr2=0x7fe5400d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.086+0000 7fe545c00640 1 --2- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 0x7fe5400d3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.086+0000 7fe545c00640 1 -- 172.21.15.2:0/2034809800 >> v1:172.21.15.137:6789/0 conn(0x7fe5400d3090 legacy=0x7fe5400d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.086+0000 7fe545c00640 1 -- 172.21.15.2:0/2034809800 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400d0e80 con 0x7fe54015d340 2024-01-15T18:00:51.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.086+0000 7fe546600640 1 --2- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 0x7fe5400d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:00:51.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.087+0000 7fe545c00640 1 --2- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe5400cdd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5300097a0 tx=0x7fe53002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e58cfe2d00d63d9b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:51.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe53fe00640 1 -- 172.21.15.2:0/2034809800 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53002d040 con 0x7fe54015d340 2024-01-15T18:00:51.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe53fe00640 1 -- 172.21.15.2:0/2034809800 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe530031810 con 0x7fe54015d340 2024-01-15T18:00:51.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe53fe00640 1 -- 172.21.15.2:0/2034809800 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530031ba0 con 0x7fe54015d340 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe547000640 1 -- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 msgr2=0x7fe5400cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe547000640 1 --2- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe5400cdd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe5300097a0 tx=0x7fe53002b8b0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe547000640 1 -- 172.21.15.2:0/2034809800 shutdown_connections 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe547000640 1 --2- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d1d90 0x7fe5400d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe547000640 1 --2- 172.21.15.2:0/2034809800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe5400cdd20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.088+0000 7fe547000640 1 -- 172.21.15.2:0/2034809800 >> 172.21.15.2:0/2034809800 conn(0x7fe54008caf0 msgr2=0x7fe54009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.089+0000 7fe547000640 1 -- 172.21.15.2:0/2034809800 shutdown_connections 2024-01-15T18:00:51.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.089+0000 7fe547000640 1 -- 172.21.15.2:0/2034809800 wait complete. 2024-01-15T18:00:51.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.091+0000 7fe547000640 1 Processor -- start 2024-01-15T18:00:51.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.091+0000 7fe547000640 1 -- start start 2024-01-15T18:00:51.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.092+0000 7fe547000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 0x7fe5401602a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:51.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.092+0000 7fe547000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d3090 0x7fe54015da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.092+0000 7fe547000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe54015df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.092+0000 7fe547000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c8050 con 0x7fe5400d1d90 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.092+0000 7fe547000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c73d0 con 0x7fe54015d340 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.092+0000 7fe547000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c76b0 con 0x7fe5400d3090 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 0x7fe5401602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 0x7fe5401602a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59058/0 (socket says 172.21.15.2:59058) 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 -- 172.21.15.2:0/2053287748 learned_addr learned my addr 172.21.15.2:0/2053287748 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545200640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d3090 0x7fe54015da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe546600640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe54015df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 -- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d3090 msgr2=0x7fe54015da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:51.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d3090 0x7fe54015da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 -- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 msgr2=0x7fe54015df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:51.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe54015df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.093+0000 7fe545c00640 1 -- 172.21.15.2:0/2053287748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe538009b30 con 0x7fe5400d1d90 2024-01-15T18:00:51.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.094+0000 7fe545c00640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 0x7fe5401602a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe53002b8f0 tx=0x7fe5300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:51.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.094+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530006e10 con 0x7fe5400d1d90 2024-01-15T18:00:51.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.094+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53003ad60 con 0x7fe5400d1d90 2024-01-15T18:00:51.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.094+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe530009400 con 0x7fe5400d1d90 2024-01-15T18:00:51.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.094+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530044630 con 0x7fe5400d1d90 2024-01-15T18:00:51.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.094+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5400c7c70 con 0x7fe5400d1d90 2024-01-15T18:00:51.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.096+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5400c8190 con 0x7fe5400d1d90 2024-01-15T18:00:51.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.097+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe530044790 con 0x7fe5400d1d90 2024-01-15T18:00:51.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.099+0000 7fe53d600640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51806b530 0x7fe51806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:51.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.099+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe5300ac350 con 0x7fe5400d1d90 2024-01-15T18:00:51.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.099+0000 7fe545200640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51806b530 0x7fe51806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:51.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.100+0000 7fe545200640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51806b530 0x7fe51806d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe528007fd0 tx=0x7fe528009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:51.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.104+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5400c8190 con 0x7fe5400d1d90 2024-01-15T18:00:51.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.337+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe5400c8190 con 0x7fe5400d1d90 2024-01-15T18:00:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.338+0000 7fe53d600640 1 -- 172.21.15.2:0/2053287748 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe5400c8190 con 0x7fe5400d1d90 2024-01-15T18:00:51.343 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51806b530 msgr2=0x7fe51806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51806b530 0x7fe51806d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fe528007fd0 tx=0x7fe528009040 comp rx=0 tx=0).stop 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 msgr2=0x7fe5401602a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 0x7fe5401602a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe53002b8f0 tx=0x7fe5300046c0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 shutdown_connections 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51806b530 0x7fe51806d9f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54015d340 0x7fe54015df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5400d3090 0x7fe54015da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 --2- 172.21.15.2:0/2053287748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5400d1d90 0x7fe5401602a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:51.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 >> 172.21.15.2:0/2053287748 conn(0x7fe54008caf0 msgr2=0x7fe54006b370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:51.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 shutdown_connections 2024-01-15T18:00:51.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:51.341+0000 7fe547000640 1 -- 172.21.15.2:0/2053287748 wait complete. 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:00:51.364 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:00:51.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:51.365 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T18:00:51.425 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T18:00:51.426 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42419sec preferred_lft 42419sec 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.427 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.428 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:00:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:00:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T18:00:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:00:51.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:51.430 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:00:51.519 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T18:00:51.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:51.523 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T18:00:51.576 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T18:00:51.576 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:00:51.576 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:00:51.576 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:00:51.576 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:00:51.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:51.577 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T18:00:51.577 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T18:00:51.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:00:51.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:00:51.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:00:51.577 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T18:00:51.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:51.757 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:00:51.808 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:00:51.808 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:00:51.808 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:00:51.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:51.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T18:00:51.896 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:00:51.896 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:00:51.896 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:51.948 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:00:51.948 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:00:51.948 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:52.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:00:52.093 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:00:52.150 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:00:52.150 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:00:52.150 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:00:52.151 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:00:52.152 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:00:52.153 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:00:52.153 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:00:52.153 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:00:52.153 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:00:52.153 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:00:52.153 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:00:52.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:52.154 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:00:52.205 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:00:52.205 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:00:52.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:00:52.287 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:00:52.334 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:00:52.335 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:00:52.336 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:00:52.337 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:00:52.337 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:00:52.337 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:00:52.337 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:00:52.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:52.338 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:00:52.397 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T18:00:52.394+0000 7fc8f700e480 -1 init, newargv = 0x562efb9a1910 newargc=15 2024-01-15T18:00:52.398 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[43390]: starting ceph client 2024-01-15T18:00:52.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[43390]: starting fuse 2024-01-15T18:00:52.426 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T18:00:52.426 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T18:00:52.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:00:52.426 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T18:00:52.426 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T18:00:52.427 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T18:00:52.428 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T18:00:52.429 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:00:52.429 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:00:52.429 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T18:00:52.429 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T18:00:52.613 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.43390.asok 2024-01-15T18:00:52.617 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T18:00:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:00:52.630 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43390.asok 2024-01-15T18:00:52.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:52.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43390.asok status 2024-01-15T18:00:52.862 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T18:00:52.862 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T18:00:52.862 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "43390", 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5057, 2024-01-15T18:00:52.863 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5057 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3801861346 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:00:52.864 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3801861346 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5057 192.168.0.1:0/3801861346", 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/3801861346", 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 26, 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 47, 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T18:00:52.865 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T18:00:52.866 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T18:00:52.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:52.877 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T18:00:52.877 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:52.929 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T18:00:52.929 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:52.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:52.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:00:53.020 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:00:53.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.252+0000 7f4c91e00640 1 Processor -- start 2024-01-15T18:00:53.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.252+0000 7f4c91e00640 1 -- start start 2024-01-15T18:00:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.254+0000 7f4c91e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4c8c15e0b0 0x7f4c8c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.254+0000 7f4c91e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c06f440 con 0x7f4c8c146ba0 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.254+0000 7f4c91e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8c15e5f0 con 0x7f4c8c0b32b0 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.254+0000 7f4c91e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c146f80 con 0x7f4c8c15e0b0 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.254+0000 7f4c8aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4c8c146ba0 0x7f4c8c0b2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52876/0 (socket says 172.21.15.2:52876) 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.254+0000 7f4c8aa00640 1 -- 172.21.15.2:0/276489173 learned_addr learned my addr 172.21.15.2:0/276489173 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.255+0000 7f4c8b400640 1 --2- 172.21.15.2:0/276489173 >> v2:172.21.15.2:3300/0 conn(0x7f4c8c15e0b0 0x7f4c8c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.255+0000 7f4c8a000640 1 -- 172.21.15.2:0/276489173 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380179936 0 0) 0x7f4c8c06f440 con 0x7f4c8c146ba0 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.255+0000 7f4c8a000640 1 -- 172.21.15.2:0/276489173 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c6c003520 con 0x7f4c8c146ba0 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.255+0000 7f4c8b400640 1 -- 172.21.15.2:0/276489173 >> v1:172.21.15.137:6789/0 conn(0x7f4c8c0b32b0 legacy=0x7f4c8c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.255+0000 7f4c8b400640 1 -- 172.21.15.2:0/276489173 >> v1:172.21.15.2:6789/0 conn(0x7f4c8c146ba0 legacy=0x7f4c8c0b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.255+0000 7f4c8b400640 1 -- 172.21.15.2:0/276489173 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c15e8d0 con 0x7f4c8c15e0b0 2024-01-15T18:00:53.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c8b400640 1 --2- 172.21.15.2:0/276489173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c15e0b0 0x7f4c8c0d1a60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009d90 tx=0x7f4c8402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb11bb2766128dd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:53.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c8a000640 1 -- 172.21.15.2:0/276489173 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c8402cc60 con 0x7f4c8c15e0b0 2024-01-15T18:00:53.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c8a000640 1 -- 172.21.15.2:0/276489173 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8402cdc0 con 0x7f4c8c15e0b0 2024-01-15T18:00:53.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c8a000640 1 -- 172.21.15.2:0/276489173 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c84034660 con 0x7f4c8c15e0b0 2024-01-15T18:00:53.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c91e00640 1 -- 172.21.15.2:0/276489173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c15e0b0 msgr2=0x7f4c8c0d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c91e00640 1 --2- 172.21.15.2:0/276489173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c15e0b0 0x7f4c8c0d1a60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4c84009d90 tx=0x7f4c8402beb0 comp rx=0 tx=0).stop 2024-01-15T18:00:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c91e00640 1 -- 172.21.15.2:0/276489173 shutdown_connections 2024-01-15T18:00:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c91e00640 1 --2- 172.21.15.2:0/276489173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c15e0b0 0x7f4c8c0d1a60 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.257+0000 7f4c91e00640 1 -- 172.21.15.2:0/276489173 >> 172.21.15.2:0/276489173 conn(0x7f4c8c0b6430 msgr2=0x7f4c8c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.258+0000 7f4c91e00640 1 -- 172.21.15.2:0/276489173 shutdown_connections 2024-01-15T18:00:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.258+0000 7f4c91e00640 1 -- 172.21.15.2:0/276489173 wait complete. 2024-01-15T18:00:53.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.259+0000 7f4c91e00640 1 Processor -- start 2024-01-15T18:00:53.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.259+0000 7f4c91e00640 1 -- start start 2024-01-15T18:00:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.260+0000 7f4c91e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c8c0b32b0 0x7f4c8c0cf870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.260+0000 7f4c91e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 0x7f4c8c0cfdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.260+0000 7f4c91e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c8c15e0b0 0x7f4c8c0cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.260+0000 7f4c91e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c1359e0 con 0x7f4c8c146ba0 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.260+0000 7f4c91e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c132ab0 con 0x7f4c8c0b32b0 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.260+0000 7f4c91e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0cc320 con 0x7f4c8c15e0b0 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 0x7f4c8c0cfdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 0x7f4c8c0cfdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42808/0 (socket says 172.21.15.2:42808) 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 -- 172.21.15.2:0/2175713872 learned_addr learned my addr 172.21.15.2:0/2175713872 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c8b400640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c8c15e0b0 0x7f4c8c0cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:53.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 -- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c8c15e0b0 msgr2=0x7f4c8c0cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c8c15e0b0 0x7f4c8c0cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 -- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c8c0b32b0 msgr2=0x7f4c8c0cf870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c8aa00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c8c0b32b0 0x7f4c8c0cf870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c8c0b32b0 0x7f4c8c0cf870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c83e00640 1 -- 172.21.15.2:0/2175713872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c84009a70 con 0x7f4c8c146ba0 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.261+0000 7f4c8aa00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c8c0b32b0 0x7f4c8c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.262+0000 7f4c83e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 0x7f4c8c0cfdb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c450 tx=0x7f4c7400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.262+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c740052d0 con 0x7f4c8c146ba0 2024-01-15T18:00:53.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.262+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74005430 con 0x7f4c8c146ba0 2024-01-15T18:00:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.262+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c0cc600 con 0x7f4c8c146ba0 2024-01-15T18:00:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.262+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c74012870 con 0x7f4c8c146ba0 2024-01-15T18:00:53.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.262+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c0ccbc0 con 0x7f4c8c146ba0 2024-01-15T18:00:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.263+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c06b130 con 0x7f4c8c146ba0 2024-01-15T18:00:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.264+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4c740129d0 con 0x7f4c8c146ba0 2024-01-15T18:00:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.265+0000 7f4c83400640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4c6406b500 0x7f4c6406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.266+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4c74089bf0 con 0x7f4c8c146ba0 2024-01-15T18:00:53.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.266+0000 7f4c8aa00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4c6406b500 0x7f4c6406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:53.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.268+0000 7f4c8aa00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4c6406b500 0x7f4c6406d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c007fd0 tx=0x7f4c7c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:53.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.271+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c74054640 con 0x7f4c8c146ba0 2024-01-15T18:00:53.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:53.503+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4c8c0686f0 con 0x7f4c8c146ba0 2024-01-15T18:00:54.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.510+0000 7f4c83400640 1 -- 172.21.15.2:0/2175713872 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f4c74057c60 con 0x7f4c8c146ba0 2024-01-15T18:00:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4c6406b500 msgr2=0x7f4c6406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4c6406b500 0x7f4c6406d9c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c007fd0 tx=0x7f4c7c009040 comp rx=0 tx=0).stop 2024-01-15T18:00:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 msgr2=0x7f4c8c0cfdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 0x7f4c8c0cfdb0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400c450 tx=0x7f4c7400c910 comp rx=0 tx=0).stop 2024-01-15T18:00:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 shutdown_connections 2024-01-15T18:00:54.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4c6406b500 0x7f4c6406d9c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4c8c15e0b0 0x7f4c8c0cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4c8c146ba0 0x7f4c8c0cfdb0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 --2- 172.21.15.2:0/2175713872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4c8c0b32b0 0x7f4c8c0cf870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.515+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 >> 172.21.15.2:0/2175713872 conn(0x7f4c8c0b6430 msgr2=0x7f4c8c0b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:54.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.516+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 shutdown_connections 2024-01-15T18:00:54.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.516+0000 7f4c91e00640 1 -- 172.21.15.2:0/2175713872 wait complete. 2024-01-15T18:00:54.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:00:54.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.759+0000 7fe180e00640 1 Processor -- start 2024-01-15T18:00:54.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.760+0000 7fe180e00640 1 -- start start 2024-01-15T18:00:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.761+0000 7fe180e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 0x7fe17c0d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.761+0000 7fe180e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe17c15d610 0x7fe17c0d3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.761+0000 7fe180e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17c0d3fe0 con 0x7fe17c0cfe80 2024-01-15T18:00:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.761+0000 7fe180e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe17c0b3200 con 0x7fe17c15d610 2024-01-15T18:00:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.761+0000 7fe180e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0d2070 con 0x7fe17c0cfaa0 2024-01-15T18:00:54.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.762+0000 7fe17be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe17c0cfe80 0x7fe17c0cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52888/0 (socket says 172.21.15.2:52888) 2024-01-15T18:00:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.762+0000 7fe17be00640 1 -- 172.21.15.2:0/2790942130 learned_addr learned my addr 172.21.15.2:0/2790942130 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.762+0000 7fe17aa00640 1 --2- 172.21.15.2:0/2790942130 >> v2:172.21.15.137:3300/0 conn(0x7fe17c15d610 0x7fe17c0d3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.762+0000 7fe17b400640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 0x7fe17c0d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.762+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778559059 0 0) 0x7fe174001110 con 0x7fe17c0cfe80 2024-01-15T18:00:54.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.762+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe15c003520 con 0x7fe17c0cfe80 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 982844369 0 0) 0x7fe15c003520 con 0x7fe17c0cfe80 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 msgr2=0x7fe17c0d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 0x7fe17c0d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 >> v2:172.21.15.137:3300/0 conn(0x7fe17c15d610 msgr2=0x7fe17c0d3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 --2- 172.21.15.2:0/2790942130 >> v2:172.21.15.137:3300/0 conn(0x7fe17c15d610 0x7fe17c0d3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c0d1d90 con 0x7fe17c0cfe80 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe174001800 con 0x7fe17c0cfe80 2024-01-15T18:00:54.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 >> v1:172.21.15.2:6789/0 conn(0x7fe17c0cfe80 legacy=0x7fe17c0cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c003970 0x7fe15c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c0047c0 0x7fe15c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 0x7fe15c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0b3200 con 0x7fe15c003970 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0d2070 con 0x7fe15c0047c0 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c00b980 con 0x7fe17c0cfaa0 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17b400640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c003970 0x7fe15c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17aa00640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c0047c0 0x7fe15c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17b400640 1 -- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 msgr2=0x7fe15c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17b400640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cfaa0 0x7fe15c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17b400640 1 -- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c0047c0 msgr2=0x7fe15c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:54.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17b400640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c0047c0 0x7fe15c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.763+0000 7fe17b400640 1 -- 172.21.15.2:0/2790942130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c003520 con 0x7fe15c003970 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.765+0000 7fe17b400640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c003970 0x7fe15c003d90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009ac0 tx=0x7fe15c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=871dca3b4d1afc31 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.765+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c02c6d0 con 0x7fe15c003970 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.765+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c02c830 con 0x7fe15c003970 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.765+0000 7fe17a000640 1 -- 172.21.15.2:0/2790942130 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c02cbc0 con 0x7fe15c003970 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe180e00640 1 -- 172.21.15.2:0/2790942130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c003970 msgr2=0x7fe15c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe180e00640 1 --2- 172.21.15.2:0/2790942130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe15c003970 0x7fe15c003d90 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fe16c009ac0 tx=0x7fe15c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe17be00640 1 -- 172.21.15.2:0/2790942130 reap_dead start 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe180e00640 1 -- 172.21.15.2:0/2790942130 shutdown_connections 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe180e00640 1 -- 172.21.15.2:0/2790942130 >> 172.21.15.2:0/2790942130 conn(0x7fe17c09c3b0 msgr2=0x7fe17c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe180e00640 1 -- 172.21.15.2:0/2790942130 shutdown_connections 2024-01-15T18:00:54.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.766+0000 7fe180e00640 1 -- 172.21.15.2:0/2790942130 wait complete. 2024-01-15T18:00:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.768+0000 7fe180e00640 1 Processor -- start 2024-01-15T18:00:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.768+0000 7fe180e00640 1 -- start start 2024-01-15T18:00:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe180e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c003970 0x7fe17c0c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe180e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 0x7fe17c0ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe180e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cd300 0x7fe17c14a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe180e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0c9320 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe180e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0cc930 con 0x7fe15c003970 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe180e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c0cbd20 con 0x7fe17c0cd300 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 0x7fe17c0ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 0x7fe17c0ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42834/0 (socket says 172.21.15.2:42834) 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 -- 172.21.15.2:0/2789749722 learned_addr learned my addr 172.21.15.2:0/2789749722 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 -- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cd300 msgr2=0x7fe17c14a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cd300 0x7fe17c14a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 -- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c003970 msgr2=0x7fe17c0c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c003970 0x7fe17c0c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.769+0000 7fe17aa00640 1 -- 172.21.15.2:0/2789749722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c0cc000 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.770+0000 7fe17aa00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 0x7fe17c0ccdc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe16400f400 tx=0x7fe16400f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:54.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.770+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164016620 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.770+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe164016780 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.770+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe164016b30 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.770+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c0cc2e0 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.770+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c1594b0 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.771+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c0d0af0 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.772+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe164016c90 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.773+0000 7fe173400640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe15406b550 0x7fe15406da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:54.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.773+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe16408b240 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.774+0000 7fe17b400640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe15406b550 0x7fe15406da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:54.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.775+0000 7fe17b400640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe15406b550 0x7fe15406da10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe16c036000 tx=0x7fe16c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:54.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.778+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe164055c00 con 0x7fe17c0cb7f0 2024-01-15T18:00:54.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.995+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe17c0d0330 con 0x7fe15406b550 2024-01-15T18:00:55.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:54.999+0000 7fe173400640 1 -- 172.21.15.2:0/2789749722 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe16c004040 con 0x7fe15406b550 2024-01-15T18:00:55.003 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:00:55.003 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:00:55.003 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:00:55.003 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:00:55.003 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe15406b550 msgr2=0x7fe15406da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe15406b550 0x7fe15406da10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe16c036000 tx=0x7fe16c02e000 comp rx=0 tx=0).stop 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 msgr2=0x7fe17c0ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 0x7fe17c0ccdc0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe16400f400 tx=0x7fe16400f8c0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 shutdown_connections 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe15406b550 0x7fe15406da10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe17c0cd300 0x7fe17c14a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe17c0cb7f0 0x7fe17c0ccdc0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 --2- 172.21.15.2:0/2789749722 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe15c003970 0x7fe17c0c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 >> 172.21.15.2:0/2789749722 conn(0x7fe17c09c3b0 msgr2=0x7fe17c0cdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 shutdown_connections 2024-01-15T18:00:55.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.001+0000 7fe180e00640 1 -- 172.21.15.2:0/2789749722 wait complete. 2024-01-15T18:00:55.026 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:00:55.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.252+0000 7f43b9000640 1 Processor -- start 2024-01-15T18:00:55.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.252+0000 7f43b9000640 1 -- start start 2024-01-15T18:00:55.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.253+0000 7f43b9000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43b40b4c70 0x7f43b40d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.253+0000 7f43b9000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f43b40b8c30 0x7f43b40d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.253+0000 7f43b9000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b40d1900 con 0x7f43b415d730 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.253+0000 7f43b9000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40d1be0 con 0x7f43b40b8c30 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.253+0000 7f43b9000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b40d1ec0 con 0x7f43b40b4c70 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.254+0000 7f43b2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f43b40b8c30 0x7f43b40d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.254+0000 7f43b2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f43b40b8c30 0x7f43b40d29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42840/0 (socket says 172.21.15.2:42840) 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.254+0000 7f43b2800640 1 -- 172.21.15.2:0/78077730 learned_addr learned my addr 172.21.15.2:0/78077730 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.254+0000 7f43b1e00640 1 --2- 172.21.15.2:0/78077730 >> v2:172.21.15.137:3300/0 conn(0x7f43b40b4c70 0x7f43b40d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.255+0000 7f43b2800640 1 -- 172.21.15.2:0/78077730 >> v2:172.21.15.137:3300/0 conn(0x7f43b40b4c70 msgr2=0x7f43b40d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.255+0000 7f43b2800640 1 --2- 172.21.15.2:0/78077730 >> v2:172.21.15.137:3300/0 conn(0x7f43b40b4c70 0x7f43b40d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.255+0000 7f43b2800640 1 -- 172.21.15.2:0/78077730 >> v1:172.21.15.137:6789/0 conn(0x7f43b415d730 legacy=0x7f43b40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:55.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.255+0000 7f43b2800640 1 -- 172.21.15.2:0/78077730 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b40d21a0 con 0x7f43b40b8c30 2024-01-15T18:00:55.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.256+0000 7f43b2800640 1 --2- 172.21.15.2:0/78077730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b40d29a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f43b40b5c40 tx=0x7f43ac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb82b79097bb1072 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b0a00640 1 -- 172.21.15.2:0/78077730 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac031870 con 0x7f43b40b8c30 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b0a00640 1 -- 172.21.15.2:0/78077730 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ac0319d0 con 0x7f43b40b8c30 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b0a00640 1 -- 172.21.15.2:0/78077730 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ac031d60 con 0x7f43b40b8c30 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b9000640 1 -- 172.21.15.2:0/78077730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 msgr2=0x7f43b40d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b9000640 1 --2- 172.21.15.2:0/78077730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b40d29a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f43b40b5c40 tx=0x7f43ac02b910 comp rx=0 tx=0).stop 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b9000640 1 -- 172.21.15.2:0/78077730 shutdown_connections 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b9000640 1 --2- 172.21.15.2:0/78077730 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b40d29a0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b9000640 1 --2- 172.21.15.2:0/78077730 >> v2:172.21.15.137:3300/0 conn(0x7f43b40b4c70 0x7f43b40d1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.257+0000 7f43b9000640 1 -- 172.21.15.2:0/78077730 >> 172.21.15.2:0/78077730 conn(0x7f43b4072210 msgr2=0x7f43b40cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:55.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.258+0000 7f43b9000640 1 -- 172.21.15.2:0/78077730 shutdown_connections 2024-01-15T18:00:55.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.258+0000 7f43b9000640 1 -- 172.21.15.2:0/78077730 wait complete. 2024-01-15T18:00:55.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.259+0000 7f43b9000640 1 Processor -- start 2024-01-15T18:00:55.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.259+0000 7f43b9000640 1 -- start start 2024-01-15T18:00:55.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.260+0000 7f43b9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43b40b4c70 0x7f43b40cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.260+0000 7f43b9000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b4157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.260+0000 7f43b9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43b415d730 0x7f43b415f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.260+0000 7f43b9000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b41605a0 con 0x7f43b40b8c30 2024-01-15T18:00:55.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.260+0000 7f43b9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40cbc20 con 0x7f43b40b4c70 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.260+0000 7f43b9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b40cb310 con 0x7f43b415d730 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b4157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b4157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42856/0 (socket says 172.21.15.2:42856) 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 -- 172.21.15.2:0/3912062735 learned_addr learned my addr 172.21.15.2:0/3912062735 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1e00640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43b40b4c70 0x7f43b40cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 -- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43b415d730 msgr2=0x7f43b415f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b2800640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43b415d730 0x7f43b415f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43b415d730 0x7f43b415f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 -- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43b40b4c70 msgr2=0x7f43b40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43b40b4c70 0x7f43b40cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1400640 1 -- 172.21.15.2:0/3912062735 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ac0094d0 con 0x7f43b40b8c30 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.261+0000 7f43b1e00640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43b40b4c70 0x7f43b40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.262+0000 7f43b1400640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b4157af0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f439c00b3c0 tx=0x7f439c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.262+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c011830 con 0x7f43b40b8c30 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.262+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f439c011990 con 0x7f43b40b8c30 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.262+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f439c011d40 con 0x7f43b40b8c30 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.262+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b40d3060 con 0x7f43b40b8c30 2024-01-15T18:00:55.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.262+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b40cb840 con 0x7f43b40b8c30 2024-01-15T18:00:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.263+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b40686f0 con 0x7f43b40b8c30 2024-01-15T18:00:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.264+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f439c01b460 con 0x7f43b40b8c30 2024-01-15T18:00:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.266+0000 7f43aa000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f438c06b500 0x7f438c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.266+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f439c089be0 con 0x7f43b40b8c30 2024-01-15T18:00:55.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.266+0000 7f43b1e00640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f438c06b500 0x7f438c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.268+0000 7f43b1e00640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f438c06b500 0x7f438c06d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f43a4007330 tx=0x7f43a400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.270+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f439c054630 con 0x7f43b40b8c30 2024-01-15T18:00:55.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.466+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f43b40b29f0 con 0x7f43b40b8c30 2024-01-15T18:00:55.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.467+0000 7f43aa000640 1 -- 172.21.15.2:0/3912062735 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f439c057c50 con 0x7f43b40b8c30 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.469+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f438c06b500 msgr2=0x7f438c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.469+0000 7f43b9000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f438c06b500 0x7f438c06d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f43a4007330 tx=0x7f43a400b040 comp rx=0 tx=0).stop 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.469+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 msgr2=0x7f43b4157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.469+0000 7f43b9000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b4157af0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f439c00b3c0 tx=0x7f439c00b880 comp rx=0 tx=0).stop 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 shutdown_connections 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f438c06b500 0x7f438c06d9c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43b415d730 0x7f43b415f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43b40b8c30 0x7f43b4157af0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 --2- 172.21.15.2:0/3912062735 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43b40b4c70 0x7f43b40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 >> 172.21.15.2:0/3912062735 conn(0x7f43b4072210 msgr2=0x7f43b41582b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:55.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 shutdown_connections 2024-01-15T18:00:55.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.470+0000 7f43b9000640 1 -- 172.21.15.2:0/3912062735 wait complete. 2024-01-15T18:00:55.494 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:00:55.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.705+0000 7f9770000640 1 Processor -- start 2024-01-15T18:00:55.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.705+0000 7f9770000640 1 -- start start 2024-01-15T18:00:55.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.707+0000 7f9770000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 0x7f97680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.707+0000 7f9770000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f97680b3420 0x7f97680cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.707+0000 7f9770000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976815dd70 con 0x7f97680ccc70 2024-01-15T18:00:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.707+0000 7f9770000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97680b3880 con 0x7f97680b3420 2024-01-15T18:00:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.707+0000 7f9770000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97680711b0 con 0x7f97681439f0 2024-01-15T18:00:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.707+0000 7f976e000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f97680ccc70 0x7f976806c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52900/0 (socket says 172.21.15.2:52900) 2024-01-15T18:00:55.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.708+0000 7f976e000640 1 -- 172.21.15.2:0/1913349014 learned_addr learned my addr 172.21.15.2:0/1913349014 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.708+0000 7f976cc00640 1 --2- 172.21.15.2:0/1913349014 >> v2:172.21.15.137:3300/0 conn(0x7f97680b3420 0x7f97680cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.708+0000 7f976d600640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 0x7f97680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.708+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551937305 0 0) 0x7f976815dd70 con 0x7f97680ccc70 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.708+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9748005730 con 0x7f97680ccc70 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3660478510 0 0) 0x7f9748005730 con 0x7f97680ccc70 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 msgr2=0x7f97680702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:55.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 0x7f97680702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 >> v2:172.21.15.137:3300/0 conn(0x7f97680b3420 msgr2=0x7f97680cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 --2- 172.21.15.2:0/1913349014 >> v2:172.21.15.137:3300/0 conn(0x7f97680b3420 0x7f97680cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9768143dd0 con 0x7f97680ccc70 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f97600019f0 con 0x7f97680ccc70 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 >> v1:172.21.15.2:6789/0 conn(0x7f97680ccc70 legacy=0x7f976806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9748005b80 0x7f9748005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97480069d0 0x7f9748006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 0x7f9748007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680b3880 con 0x7f97480069d0 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680711b0 con 0x7f9748005b80 2024-01-15T18:00:55.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974800db90 con 0x7f97681439f0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f976cc00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97480069d0 0x7f9748006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.709+0000 7f976d600640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9748005b80 0x7f9748005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976e000640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 0x7f9748007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976cc00640 1 -- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 msgr2=0x7f9748007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976cc00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97681439f0 0x7f9748007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976cc00640 1 -- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9748005b80 msgr2=0x7f9748005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976cc00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9748005b80 0x7f9748005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976cc00640 1 -- 172.21.15.2:0/1913349014 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9748005730 con 0x7f97480069d0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.710+0000 7f976d600640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9748005b80 0x7f9748005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.711+0000 7f976cc00640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97480069d0 0x7f9748006e50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9750009ac0 tx=0x7f9748007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6d7ba799ecacbf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.711+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975002c710 con 0x7f97480069d0 2024-01-15T18:00:55.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.711+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975002c870 con 0x7f97480069d0 2024-01-15T18:00:55.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.711+0000 7f9767e00640 1 -- 172.21.15.2:0/1913349014 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975002cc00 con 0x7f97480069d0 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f9770000640 1 -- 172.21.15.2:0/1913349014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97480069d0 msgr2=0x7f9748006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f9770000640 1 --2- 172.21.15.2:0/1913349014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97480069d0 0x7f9748006e50 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9750009ac0 tx=0x7f9748007a00 comp rx=0 tx=0).stop 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f976e000640 1 -- 172.21.15.2:0/1913349014 reap_dead start 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f9770000640 1 -- 172.21.15.2:0/1913349014 shutdown_connections 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f9770000640 1 -- 172.21.15.2:0/1913349014 >> 172.21.15.2:0/1913349014 conn(0x7f97680b45f0 msgr2=0x7f97680b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f9770000640 1 -- 172.21.15.2:0/1913349014 shutdown_connections 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.712+0000 7f9770000640 1 -- 172.21.15.2:0/1913349014 wait complete. 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.714+0000 7f9770000640 1 Processor -- start 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.714+0000 7f9770000640 1 -- start start 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f9770000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97480069d0 0x7f97681f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f9770000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 0x7f976815d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f9770000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97680b06f0 0x7f97680b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f9770000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97680b10b0 con 0x7f976815d040 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f9770000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976813ed10 con 0x7f97680b06f0 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f9770000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976813e100 con 0x7f97480069d0 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f976cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 0x7f976815d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f976cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 0x7f976815d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42872/0 (socket says 172.21.15.2:42872) 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f976cc00640 1 -- 172.21.15.2:0/1897002450 learned_addr learned my addr 172.21.15.2:0/1897002450 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.715+0000 7f976e000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97680b06f0 0x7f97680b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976cc00640 1 -- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97480069d0 msgr2=0x7f97681f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976d600640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97480069d0 0x7f97681f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976cc00640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97480069d0 0x7f97681f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976cc00640 1 -- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97680b06f0 msgr2=0x7f97680b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976cc00640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97680b06f0 0x7f97680b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976cc00640 1 -- 172.21.15.2:0/1897002450 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976813e3e0 con 0x7f976815d040 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976e000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97680b06f0 0x7f97680b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976d600640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97480069d0 0x7f97681f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f976cc00640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 0x7f976815d4c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9750037040 tx=0x7f9750004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750003810 con 0x7f976815d040 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975002de90 con 0x7f976815d040 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976813e6c0 con 0x7f976815d040 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750003b70 con 0x7f976815d040 2024-01-15T18:00:55.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.716+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97681e3670 con 0x7f976815d040 2024-01-15T18:00:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.718+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97680686f0 con 0x7f976815d040 2024-01-15T18:00:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.719+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9750003cd0 con 0x7f976815d040 2024-01-15T18:00:55.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.719+0000 7f9765600640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f974006b530 0x7f974006d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.720+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f97500abfe0 con 0x7f976815d040 2024-01-15T18:00:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.720+0000 7f976d600640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f974006b530 0x7f974006d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:55.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.722+0000 7f976d600640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f974006b530 0x7f974006d9f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f97580099c0 tx=0x7f9758008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:55.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.724+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9750076b40 con 0x7f976815d040 2024-01-15T18:00:55.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:55.944+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f97680b10b0 con 0x7f974006b530 2024-01-15T18:00:55.958 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:55.955+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:55.958 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:55.955+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:55.958 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:55.955+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:55.958 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:55.955+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:55.958 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:00:55.955+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:00:56.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.031+0000 7f9765600640 1 -- 172.21.15.2:0/1897002450 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f97680b10b0 con 0x7f974006b530 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f974006b530 msgr2=0x7f974006d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f974006b530 0x7f974006d9f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f97580099c0 tx=0x7f9758008040 comp rx=0 tx=0).stop 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 msgr2=0x7f976815d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 0x7f976815d4c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9750037040 tx=0x7f9750004730 comp rx=0 tx=0).stop 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 shutdown_connections 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f974006b530 0x7f974006d9f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97680b06f0 0x7f97680b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f976815d040 0x7f976815d4c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 --2- 172.21.15.2:0/1897002450 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f97480069d0 0x7f97681f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.035+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 >> 172.21.15.2:0/1897002450 conn(0x7f97680b45f0 msgr2=0x7f9768142ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.036+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 shutdown_connections 2024-01-15T18:00:56.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.036+0000 7f9770000640 1 -- 172.21.15.2:0/1897002450 wait complete. 2024-01-15T18:00:56.062 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:00:56.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.282+0000 7f99cac00640 1 Processor -- start 2024-01-15T18:00:56.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.282+0000 7f99cac00640 1 -- start start 2024-01-15T18:00:56.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.283+0000 7f99cac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f99c40d1c50 0x7f99c40d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99cac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c40d1250 con 0x7f99c40b2210 2024-01-15T18:00:56.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99cac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c40ce8e0 con 0x7f99c40d1870 2024-01-15T18:00:56.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99cac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c40ce6d0 con 0x7f99c40d1c50 2024-01-15T18:00:56.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99c8e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f99c40b2210 0x7f99c40b26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52912/0 (socket says 172.21.15.2:52912) 2024-01-15T18:00:56.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99c8e00640 1 -- 172.21.15.2:0/3078938515 learned_addr learned my addr 172.21.15.2:0/3078938515 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99ca200640 1 --2- 172.21.15.2:0/3078938515 >> v2:172.21.15.137:3300/0 conn(0x7f99c40d1c50 0x7f99c40d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056557061 0 0) 0x7f99c40d1250 con 0x7f99c40b2210 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.284+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a4003520 con 0x7f99c40b2210 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573649829 0 0) 0x7f99a4003520 con 0x7f99c40b2210 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 >> v2:172.21.15.137:3300/0 conn(0x7f99c40d1c50 msgr2=0x7f99c40d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 --2- 172.21.15.2:0/3078938515 >> v2:172.21.15.137:3300/0 conn(0x7f99c40d1c50 0x7f99c40d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 >> v1:172.21.15.137:6789/0 conn(0x7f99c40d1870 legacy=0x7f99c415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c40cebc0 con 0x7f99c40b2210 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f99ac001670 con 0x7f99c40b2210 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 >> v1:172.21.15.2:6789/0 conn(0x7f99c40b2210 legacy=0x7f99c40b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99a40039c0 0x7f99a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99a40047c0 0x7f99a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40ce6d0 con 0x7f99a4005180 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a400c080 con 0x7f99a40039c0 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a400c240 con 0x7f99a40047c0 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.285+0000 7f99ca200640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99c9800640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99a40039c0 0x7f99a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99ca200640 1 -- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99a40047c0 msgr2=0x7f99a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99ca200640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99a40047c0 0x7f99a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99ca200640 1 -- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99a40039c0 msgr2=0x7f99a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99ca200640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99a40039c0 0x7f99a4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99ca200640 1 -- 172.21.15.2:0/3078938515 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a4003520 con 0x7f99a4005180 2024-01-15T18:00:56.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.286+0000 7f99c9800640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99a40039c0 0x7f99a4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:56.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.287+0000 7f99ca200640 1 -- 172.21.15.2:0/3078938515 reap_dead start 2024-01-15T18:00:56.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.287+0000 7f99ca200640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99a400b980 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009fc0 tx=0x7f99a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59d9ac0b5bf70096 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:56.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.287+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc02cc60 con 0x7f99a4005180 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.287+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc00f040 con 0x7f99a4005180 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.287+0000 7f99c3e00640 1 -- 172.21.15.2:0/3078938515 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc003990 con 0x7f99a4005180 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 -- 172.21.15.2:0/3078938515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 msgr2=0x7f99a400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99a400b980 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009fc0 tx=0x7f99a40057f0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 -- 172.21.15.2:0/3078938515 shutdown_connections 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 --2- 172.21.15.2:0/3078938515 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99a400b980 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 -- 172.21.15.2:0/3078938515 >> 172.21.15.2:0/3078938515 conn(0x7f99c40d30b0 msgr2=0x7f99c40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 -- 172.21.15.2:0/3078938515 shutdown_connections 2024-01-15T18:00:56.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.288+0000 7f99cac00640 1 -- 172.21.15.2:0/3078938515 wait complete. 2024-01-15T18:00:56.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.290+0000 7f99cac00640 1 Processor -- start 2024-01-15T18:00:56.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.290+0000 7f99cac00640 1 -- start start 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99cac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99c41f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99cac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99c414c140 0x7f99c414c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99cac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99c40c8db0 0x7f99c40c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99cac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40c7a70 con 0x7f99a4005180 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99cac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40c7d10 con 0x7f99c414c140 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99cac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40c7ff0 con 0x7f99c40c8db0 2024-01-15T18:00:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99c9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99c41f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99c9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99c41f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42896/0 (socket says 172.21.15.2:42896) 2024-01-15T18:00:56.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.291+0000 7f99c9800640 1 -- 172.21.15.2:0/2014419411 learned_addr learned my addr 172.21.15.2:0/2014419411 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c8e00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99c414c140 0x7f99c414c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99ca200640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99c40c8db0 0x7f99c40c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c9800640 1 -- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99c40c8db0 msgr2=0x7f99c40c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c9800640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99c40c8db0 0x7f99c40c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c9800640 1 -- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99c414c140 msgr2=0x7f99c414c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c9800640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99c414c140 0x7f99c414c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c9800640 1 -- 172.21.15.2:0/2014419411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c40c82d0 con 0x7f99a4005180 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99ca200640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99c40c8db0 0x7f99c40c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c8e00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99c414c140 0x7f99c414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c9800640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99c41f6510 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f99b400d770 tx=0x7f99b400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b4005270 con 0x7f99a4005180 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b40053d0 con 0x7f99a4005180 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b400b8d0 con 0x7f99a4005180 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.292+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c40c85b0 con 0x7f99a4005180 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.293+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c41e1bc0 con 0x7f99a4005180 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.294+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c41e1d50 con 0x7f99a4005180 2024-01-15T18:00:56.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.295+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f99b400ba30 con 0x7f99a4005180 2024-01-15T18:00:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.296+0000 7f99c1600640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f999c06b500 0x7f999c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:00:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.296+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99b408a110 con 0x7f99a4005180 2024-01-15T18:00:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.296+0000 7f99c8e00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f999c06b500 0x7f999c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:00:56.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.297+0000 7f99c8e00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f999c06b500 0x7f999c06d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0011e0 tx=0x7f99ac009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:00:56.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.300+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99c41e1d50 con 0x7f99a4005180 2024-01-15T18:00:56.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.535+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f99c40b4620 con 0x7f999c06b500 2024-01-15T18:00:56.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.547+0000 7f99c1600640 1 -- 172.21.15.2:0/2014419411 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f99c40b4620 con 0x7f999c06b500 2024-01-15T18:00:56.551 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30 2024-01-15T18:00:56.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f999c06b500 msgr2=0x7f999c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f999c06b500 0x7f999c06d9c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0011e0 tx=0x7f99ac009040 comp rx=0 tx=0).stop 2024-01-15T18:00:56.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 msgr2=0x7f99c41f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:00:56.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99c41f6510 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f99b400d770 tx=0x7f99b400dc30 comp rx=0 tx=0).stop 2024-01-15T18:00:56.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 shutdown_connections 2024-01-15T18:00:56.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f999c06b500 0x7f999c06d9c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f99c40c8db0 0x7f99c40c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f99c414c140 0x7f99c414c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 --2- 172.21.15.2:0/2014419411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f99a4005180 0x7f99c41f6510 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:00:56.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 >> 172.21.15.2:0/2014419411 conn(0x7f99c40d30b0 msgr2=0x7f99c406c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:00:56.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 shutdown_connections 2024-01-15T18:00:56.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:00:56.550+0000 7f99cac00640 1 -- 172.21.15.2:0/2014419411 wait complete. 2024-01-15T18:00:56.574 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30 2024-01-15T18:00:56.574 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:00:56.574 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:00:56.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:00:56.627 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:00:59.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:00:59.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:01:12.666 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:01:12.666 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:01:12.666 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.9613 s, 67.3 MB/s 2024-01-15T18:01:12.668 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:01:12.668 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:01:12.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:01:12.724 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:01:26.777 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:01:26.777 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:01:26.777 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.991 s, 76.7 MB/s 2024-01-15T18:01:26.779 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:01:26.779 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:01:26.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:01:26.836 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:01:29.688 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:01:29.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:01:40.091 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:01:40.091 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:01:40.091 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.1899 s, 81.4 MB/s 2024-01-15T18:01:40.094 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:01:40.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.321+0000 7f7dd1c00640 1 Processor -- start 2024-01-15T18:01:40.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.322+0000 7f7dd1c00640 1 -- start start 2024-01-15T18:01:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.323+0000 7f7dd1c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7dcc0cddd0 0x7f7dcc0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.323+0000 7f7dd1c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dcc0ceb40 con 0x7f7dcc0cf920 2024-01-15T18:01:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.323+0000 7f7dd1c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dcc0cee20 con 0x7f7dcc15d120 2024-01-15T18:01:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.323+0000 7f7dd1c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cf100 con 0x7f7dcc0cddd0 2024-01-15T18:01:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dc3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7dcc0cf920 0x7f7dcc0cfd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33092/0 (socket says 172.21.15.2:33092) 2024-01-15T18:01:40.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dc3e00640 1 -- 172.21.15.2:0/4235263889 learned_addr learned my addr 172.21.15.2:0/4235263889 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:01:40.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dd1200640 1 --2- 172.21.15.2:0/4235263889 >> v2:172.21.15.137:3300/0 conn(0x7f7dcc0cddd0 0x7f7dcc0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300877642 0 0) 0x7f7dcc0ceb40 con 0x7f7dcc0cf920 2024-01-15T18:01:40.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac003520 con 0x7f7dcc0cf920 2024-01-15T18:01:40.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939117198 0 0) 0x7f7dac003520 con 0x7f7dcc0cf920 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 >> v2:172.21.15.137:3300/0 conn(0x7f7dcc0cddd0 msgr2=0x7f7dcc0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 --2- 172.21.15.2:0/4235263889 >> v2:172.21.15.137:3300/0 conn(0x7f7dcc0cddd0 0x7f7dcc0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 >> v1:172.21.15.137:6789/0 conn(0x7f7dcc15d120 legacy=0x7f7dcc0b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0d0d30 con 0x7f7dcc0cf920 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.324+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7db4001670 con 0x7f7dcc0cf920 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 >> v1:172.21.15.2:6789/0 conn(0x7f7dcc0cf920 legacy=0x7f7dcc0cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dac003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac0047c0 0x7f7dac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac005180 0x7f7dac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cf100 con 0x7f7dac003970 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac00c0b0 con 0x7f7dac0047c0 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac00c270 con 0x7f7dac005180 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dac003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dc3e00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac0047c0 0x7f7dac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dd1200640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac005180 0x7f7dac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcbe00640 1 -- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac005180 msgr2=0x7f7dac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac005180 0x7f7dac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcbe00640 1 -- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac0047c0 msgr2=0x7f7dac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dac0047c0 0x7f7dac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.325+0000 7f7dcbe00640 1 -- 172.21.15.2:0/4235263889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac003520 con 0x7f7dac003970 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.326+0000 7f7dd1200640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dac005180 0x7f7dac00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.326+0000 7f7dd1200640 1 -- 172.21.15.2:0/4235263889 reap_dead start 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.327+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dac003d70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc003d80 tx=0x7f7dac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5209549e7d00ec10 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.327+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc032600 con 0x7f7dac003970 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.327+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc02e050 con 0x7f7dac003970 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.327+0000 7f7dcb400640 1 -- 172.21.15.2:0/4235263889 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0329e0 con 0x7f7dac003970 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.327+0000 7f7dd1c00640 1 -- 172.21.15.2:0/4235263889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 msgr2=0x7f7dac003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.327+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dac003d70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc003d80 tx=0x7f7dac0057f0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.328+0000 7f7dd1c00640 1 -- 172.21.15.2:0/4235263889 shutdown_connections 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.328+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/4235263889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dac003d70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.328+0000 7f7dd1c00640 1 -- 172.21.15.2:0/4235263889 >> 172.21.15.2:0/4235263889 conn(0x7f7dcc09c3e0 msgr2=0x7f7dcc0d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:40.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.328+0000 7f7dd1c00640 1 -- 172.21.15.2:0/4235263889 shutdown_connections 2024-01-15T18:01:40.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.328+0000 7f7dd1c00640 1 -- 172.21.15.2:0/4235263889 wait complete. 2024-01-15T18:01:40.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.330+0000 7f7dd1c00640 1 Processor -- start 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.330+0000 7f7dd1c00640 1 -- start start 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dd1c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dcc159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dd1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dcc0cc510 0x7f7dcc0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dd1c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dcc0b5bc0 0x7f7dcc0b6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dd1c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0b6580 con 0x7f7dac003970 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dd1c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0b88e0 con 0x7f7dcc0b5bc0 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dd1c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0cceb0 con 0x7f7dcc0cc510 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dcbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dcc159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dcbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dcc159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51618/0 (socket says 172.21.15.2:51618) 2024-01-15T18:01:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.331+0000 7f7dcbe00640 1 -- 172.21.15.2:0/299782690 learned_addr learned my addr 172.21.15.2:0/299782690 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dcbe00640 1 -- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dcc0cc510 msgr2=0x7f7dcc0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dcc0cc510 0x7f7dcc0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dd1200640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dcc0b5bc0 0x7f7dcc0b6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dcbe00640 1 -- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dcc0b5bc0 msgr2=0x7f7dcc0b6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dcc0b5bc0 0x7f7dcc0b6040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dcbe00640 1 -- 172.21.15.2:0/299782690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0b73b0 con 0x7f7dac003970 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dcbe00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dcc159150 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc035040 tx=0x7f7dbc0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc03c610 con 0x7f7dac003970 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc03c770 con 0x7f7dac003970 2024-01-15T18:01:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc03cb00 con 0x7f7dac003970 2024-01-15T18:01:40.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dcc159690 con 0x7f7dac003970 2024-01-15T18:01:40.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.332+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dcc0b6a80 con 0x7f7dac003970 2024-01-15T18:01:40.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.334+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dcc074770 con 0x7f7dac003970 2024-01-15T18:01:40.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.335+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f7dbc039050 con 0x7f7dac003970 2024-01-15T18:01:40.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.336+0000 7f7dc8c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7da406b590 0x7f7da406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.336+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7dbc0ab9e0 con 0x7f7dac003970 2024-01-15T18:01:40.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.336+0000 7f7dc3e00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7da406b590 0x7f7da406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.338+0000 7f7dc3e00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7da406b590 0x7f7da406da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0c64a0 tx=0x7f7db40011a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:40.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.342+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7dbc0762c0 con 0x7f7dac003970 2024-01-15T18:01:40.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.540+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f7dcc06cc50 con 0x7f7dac003970 2024-01-15T18:01:40.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.545+0000 7f7dc8c00640 1 -- 172.21.15.2:0/299782690 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7f7dbc037030 con 0x7f7dac003970 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7da406b590 msgr2=0x7f7da406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7da406b590 0x7f7da406da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc0c64a0 tx=0x7f7db40011a0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 msgr2=0x7f7dcc159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dcc159150 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc035040 tx=0x7f7dbc0036f0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 shutdown_connections 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7da406b590 0x7f7da406da50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7dcc0b5bc0 0x7f7dcc0b6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7dcc0cc510 0x7f7dcc0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 --2- 172.21.15.2:0/299782690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7dac003970 0x7f7dcc159150 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.548+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 >> 172.21.15.2:0/299782690 conn(0x7f7dcc09c3e0 msgr2=0x7f7dcc15c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:40.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.549+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 shutdown_connections 2024-01-15T18:01:40.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.549+0000 7f7dd1c00640 1 -- 172.21.15.2:0/299782690 wait complete. 2024-01-15T18:01:40.573 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:01:40.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.779+0000 7fe731c00640 1 Processor -- start 2024-01-15T18:01:40.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.780+0000 7fe731c00640 1 -- start start 2024-01-15T18:01:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe731c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe72c143d20 0x7fe72c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe731c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe731c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe72c1447d0 0x7fe72c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe731c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c071430 con 0x7fe72c143d20 2024-01-15T18:01:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe731c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe72c144100 con 0x7fe72c1447d0 2024-01-15T18:01:40.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe731c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c071090 con 0x7fe72c0720d0 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe72be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe72c143d20 0x7fe72c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe72be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe72c143d20 0x7fe72c15eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51624/0 (socket says 172.21.15.2:51624) 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.781+0000 7fe72be00640 1 -- 172.21.15.2:0/1394639737 learned_addr learned my addr 172.21.15.2:0/1394639737 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe72b400640 1 --2- 172.21.15.2:0/1394639737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe731200640 1 --2- 172.21.15.2:0/1394639737 >> v2:172.21.15.137:3300/0 conn(0x7fe72c1447d0 0x7fe72c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe72be00640 1 -- 172.21.15.2:0/1394639737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 msgr2=0x7fe72c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe72be00640 1 --2- 172.21.15.2:0/1394639737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c15f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe72be00640 1 -- 172.21.15.2:0/1394639737 >> v2:172.21.15.137:3300/0 conn(0x7fe72c1447d0 msgr2=0x7fe72c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe72be00640 1 --2- 172.21.15.2:0/1394639737 >> v2:172.21.15.137:3300/0 conn(0x7fe72c1447d0 0x7fe72c0cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.782+0000 7fe72be00640 1 -- 172.21.15.2:0/1394639737 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe72c144560 con 0x7fe72c143d20 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.783+0000 7fe72be00640 1 --2- 172.21.15.2:0/1394639737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c15eb50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe72c06e0a0 tx=0x7fe71c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b28dc2122a963a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.783+0000 7fe72aa00640 1 -- 172.21.15.2:0/1394639737 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c02c9e0 con 0x7fe72c143d20 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.783+0000 7fe72aa00640 1 -- 172.21.15.2:0/1394639737 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe71c02cb40 con 0x7fe72c143d20 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.783+0000 7fe72aa00640 1 -- 172.21.15.2:0/1394639737 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe71c0343f0 con 0x7fe72c143d20 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 -- 172.21.15.2:0/1394639737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 msgr2=0x7fe72c15eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 --2- 172.21.15.2:0/1394639737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c15eb50 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe72c06e0a0 tx=0x7fe71c02be40 comp rx=0 tx=0).stop 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 -- 172.21.15.2:0/1394639737 shutdown_connections 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 --2- 172.21.15.2:0/1394639737 >> v2:172.21.15.137:3300/0 conn(0x7fe72c1447d0 0x7fe72c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 --2- 172.21.15.2:0/1394639737 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 --2- 172.21.15.2:0/1394639737 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c15eb50 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 -- 172.21.15.2:0/1394639737 >> 172.21.15.2:0/1394639737 conn(0x7fe72c0b0840 msgr2=0x7fe72c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:40.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 -- 172.21.15.2:0/1394639737 shutdown_connections 2024-01-15T18:01:40.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.784+0000 7fe731c00640 1 -- 172.21.15.2:0/1394639737 wait complete. 2024-01-15T18:01:40.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.786+0000 7fe731c00640 1 Processor -- start 2024-01-15T18:01:40.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.786+0000 7fe731c00640 1 -- start start 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe731c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c0cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe731c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c0c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe731c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe72c1447d0 0x7fe72c0c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe731c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c136420 con 0x7fe72c143d20 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe731c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c135e60 con 0x7fe72c1447d0 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe731c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c136140 con 0x7fe72c0720d0 2024-01-15T18:01:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe72b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c0c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe72b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c0c6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51630/0 (socket says 172.21.15.2:51630) 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.787+0000 7fe72b400640 1 -- 172.21.15.2:0/1720541939 learned_addr learned my addr 172.21.15.2:0/1720541939 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72be00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c0cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe731200640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe72c1447d0 0x7fe72c0c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72b400640 1 -- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 msgr2=0x7fe72c0cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72b400640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c0cbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72b400640 1 -- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe72c1447d0 msgr2=0x7fe72c0c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72b400640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe72c1447d0 0x7fe72c0c6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72b400640 1 -- 172.21.15.2:0/1720541939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71c009b30 con 0x7fe72c143d20 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe72b400640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c0c6630 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe71400c570 tx=0x7fe71400ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714011800 con 0x7fe72c143d20 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe714011960 con 0x7fe72c143d20 2024-01-15T18:01:40.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe714011d10 con 0x7fe72c143d20 2024-01-15T18:01:40.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.788+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c15c670 con 0x7fe72c143d20 2024-01-15T18:01:40.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.789+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c133460 con 0x7fe72c143d20 2024-01-15T18:01:40.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.790+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c15cb90 con 0x7fe72c143d20 2024-01-15T18:01:40.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.791+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fe714018360 con 0x7fe72c143d20 2024-01-15T18:01:40.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.791+0000 7fe723e00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe70406b590 0x7fe70406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:40.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.792+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe71408a150 con 0x7fe72c143d20 2024-01-15T18:01:40.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.792+0000 7fe72be00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe70406b590 0x7fe70406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:40.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.793+0000 7fe72be00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe70406b590 0x7fe70406da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe71c035000 tx=0x7fe71c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:40.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.796+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe72c15cb90 con 0x7fe72c143d20 2024-01-15T18:01:40.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.992+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fe72c15cb90 con 0x7fe72c143d20 2024-01-15T18:01:41.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:40.997+0000 7fe723e00640 1 -- 172.21.15.2:0/1720541939 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v16)=0 v16) v1 ==== 135+0+0 (secure 0 0 0) 0x7fe72c15cb90 con 0x7fe72c143d20 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe70406b590 msgr2=0x7fe70406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe70406b590 0x7fe70406da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe71c035000 tx=0x7fe71c030000 comp rx=0 tx=0).stop 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 msgr2=0x7fe72c0c6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c0c6630 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fe71400c570 tx=0x7fe71400ca30 comp rx=0 tx=0).stop 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 shutdown_connections 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe70406b590 0x7fe70406da50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe72c1447d0 0x7fe72c0c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe72c143d20 0x7fe72c0c6630 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 --2- 172.21.15.2:0/1720541939 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe72c0720d0 0x7fe72c0cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 >> 172.21.15.2:0/1720541939 conn(0x7fe72c0b0840 msgr2=0x7fe72c0ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:41.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.000+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 shutdown_connections 2024-01-15T18:01:41.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.001+0000 7fe731c00640 1 -- 172.21.15.2:0/1720541939 wait complete. 2024-01-15T18:01:41.024 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:01:41.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.252+0000 7fc3e7200640 1 Processor -- start 2024-01-15T18:01:41.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.253+0000 7fc3e7200640 1 -- start start 2024-01-15T18:01:41.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.254+0000 7fc3e7200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc3e8143d20 0x7fc3e815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.254+0000 7fc3e7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e80720d0 0x7fc3e815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.254+0000 7fc3e7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc3e81447d0 0x7fc3e80cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.254+0000 7fc3e7200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e8071430 con 0x7fc3e8143d20 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.254+0000 7fc3e7200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e8144100 con 0x7fc3e81447d0 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.254+0000 7fc3e7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8071090 con 0x7fc3e80720d0 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc3e8143d20 0x7fc3e815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc3e8143d20 0x7fc3e815eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51644/0 (socket says 172.21.15.2:51644) 2024-01-15T18:01:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 -- 172.21.15.2:0/1510255321 learned_addr learned my addr 172.21.15.2:0/1510255321 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5400640 1 --2- 172.21.15.2:0/1510255321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e80720d0 0x7fc3e815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 -- 172.21.15.2:0/1510255321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e80720d0 msgr2=0x7fc3e815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 --2- 172.21.15.2:0/1510255321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e80720d0 0x7fc3e815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e6800640 1 --2- 172.21.15.2:0/1510255321 >> v2:172.21.15.137:3300/0 conn(0x7fc3e81447d0 0x7fc3e80cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 -- 172.21.15.2:0/1510255321 >> v2:172.21.15.137:3300/0 conn(0x7fc3e81447d0 msgr2=0x7fc3e80cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 --2- 172.21.15.2:0/1510255321 >> v2:172.21.15.137:3300/0 conn(0x7fc3e81447d0 0x7fc3e80cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e5e00640 1 -- 172.21.15.2:0/1510255321 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8144560 con 0x7fc3e8143d20 2024-01-15T18:01:41.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.255+0000 7fc3e6800640 1 --2- 172.21.15.2:0/1510255321 >> v2:172.21.15.137:3300/0 conn(0x7fc3e81447d0 0x7fc3e80cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.257+0000 7fc3e5e00640 1 --2- 172.21.15.2:0/1510255321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e8143d20 0x7fc3e815eb50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc3e806e0a0 tx=0x7fc3d802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43753fc67fb00f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.257+0000 7fc3e4a00640 1 -- 172.21.15.2:0/1510255321 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d802c9f0 con 0x7fc3e8143d20 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.257+0000 7fc3e4a00640 1 -- 172.21.15.2:0/1510255321 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d802cb50 con 0x7fc3e8143d20 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.257+0000 7fc3e4a00640 1 -- 172.21.15.2:0/1510255321 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d80343f0 con 0x7fc3e8143d20 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.257+0000 7fc3e7200640 1 -- 172.21.15.2:0/1510255321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e8143d20 msgr2=0x7fc3e815eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.257+0000 7fc3e7200640 1 --2- 172.21.15.2:0/1510255321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e8143d20 0x7fc3e815eb50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc3e806e0a0 tx=0x7fc3d802be50 comp rx=0 tx=0).stop 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 -- 172.21.15.2:0/1510255321 shutdown_connections 2024-01-15T18:01:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 --2- 172.21.15.2:0/1510255321 >> v2:172.21.15.137:3300/0 conn(0x7fc3e81447d0 0x7fc3e80cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 --2- 172.21.15.2:0/1510255321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e80720d0 0x7fc3e815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 --2- 172.21.15.2:0/1510255321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e8143d20 0x7fc3e815eb50 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 -- 172.21.15.2:0/1510255321 >> 172.21.15.2:0/1510255321 conn(0x7fc3e80b0840 msgr2=0x7fc3e80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:41.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 -- 172.21.15.2:0/1510255321 shutdown_connections 2024-01-15T18:01:41.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.258+0000 7fc3e7200640 1 -- 172.21.15.2:0/1510255321 wait complete. 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.260+0000 7fc3e7200640 1 Processor -- start 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.260+0000 7fc3e7200640 1 -- start start 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3e80720d0 0x7fc3e815ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e8143d20 0x7fc3e8142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 0x7fc3e813fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e815b190 con 0x7fc3e81447d0 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8142f80 con 0x7fc3e80720d0 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e81401c0 con 0x7fc3e8143d20 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 0x7fc3e813fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 0x7fc3e813fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51648/0 (socket says 172.21.15.2:51648) 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e6800640 1 -- 172.21.15.2:0/353669685 learned_addr learned my addr 172.21.15.2:0/353669685 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e5e00640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3e80720d0 0x7fc3e815ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.261+0000 7fc3e5400640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e8143d20 0x7fc3e8142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e6800640 1 -- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e8143d20 msgr2=0x7fc3e8142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e6800640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e8143d20 0x7fc3e8142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e6800640 1 -- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3e80720d0 msgr2=0x7fc3e815ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e6800640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3e80720d0 0x7fc3e815ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e6800640 1 -- 172.21.15.2:0/353669685 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d8009b30 con 0x7fc3e81447d0 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e6800640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 0x7fc3e813fc80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000c7f0 tx=0x7fc3e000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0004230 con 0x7fc3e81447d0 2024-01-15T18:01:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0004390 con 0x7fc3e81447d0 2024-01-15T18:01:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0007860 con 0x7fc3e81447d0 2024-01-15T18:01:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.262+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e81403f0 con 0x7fc3e81447d0 2024-01-15T18:01:41.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.263+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e81409b0 con 0x7fc3e81447d0 2024-01-15T18:01:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.264+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e8074770 con 0x7fc3e81447d0 2024-01-15T18:01:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.265+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fc3e0007bd0 con 0x7fc3e81447d0 2024-01-15T18:01:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.266+0000 7fc3de000640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc3c006b590 0x7fc3c006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.266+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc3e008a5b0 con 0x7fc3e81447d0 2024-01-15T18:01:41.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.267+0000 7fc3e5e00640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc3c006b590 0x7fc3c006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.268+0000 7fc3e5e00640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc3c006b590 0x7fc3c006da50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8035000 tx=0x7fc3d8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:41.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.270+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3e0054fe0 con 0x7fc3e81447d0 2024-01-15T18:01:41.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.498+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3e815f720 con 0x7fc3c006b590 2024-01-15T18:01:41.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.534+0000 7fc3de000640 1 -- 172.21.15.2:0/353669685 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc3d803f440 con 0x7fc3c006b590 2024-01-15T18:01:41.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.536+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc3c006b590 msgr2=0x7fc3c006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.536+0000 7fc3e7200640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc3c006b590 0x7fc3c006da50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8035000 tx=0x7fc3d8030000 comp rx=0 tx=0).stop 2024-01-15T18:01:41.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 msgr2=0x7fc3e813fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 0x7fc3e813fc80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000c7f0 tx=0x7fc3e000ccb0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 shutdown_connections 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc3c006b590 0x7fc3c006da50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc3e81447d0 0x7fc3e813fc80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc3e8143d20 0x7fc3e8142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 --2- 172.21.15.2:0/353669685 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc3e80720d0 0x7fc3e815ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 >> 172.21.15.2:0/353669685 conn(0x7fc3e80b0840 msgr2=0x7fc3e815d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 shutdown_connections 2024-01-15T18:01:41.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.537+0000 7fc3e7200640 1 -- 172.21.15.2:0/353669685 wait complete. 2024-01-15T18:01:41.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:01:41.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.781+0000 7f7fd0400640 1 Processor -- start 2024-01-15T18:01:41.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.781+0000 7f7fd0400640 1 -- start start 2024-01-15T18:01:41.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.783+0000 7f7fd0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 0x7f7fc80acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.783+0000 7f7fd0400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc815dbc0 con 0x7f7fc80b2b40 2024-01-15T18:01:41.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.783+0000 7f7fd0400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc815dea0 con 0x7f7fc806c8f0 2024-01-15T18:01:41.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.783+0000 7f7fd0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc815e840 con 0x7f7fc815ae80 2024-01-15T18:01:41.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fce400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7fc80b2b40 0x7f7fc8161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33098/0 (socket says 172.21.15.2:33098) 2024-01-15T18:01:41.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fce400640 1 -- 172.21.15.2:0/3406501621 learned_addr learned my addr 172.21.15.2:0/3406501621 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fcda00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 0x7f7fc80acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583755054 0 0) 0x7f7fc815dbc0 con 0x7f7fc80b2b40 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa8003520 con 0x7f7fc80b2b40 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359375348 0 0) 0x7f7fc815dea0 con 0x7f7fc806c8f0 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.784+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc815dbc0 con 0x7f7fc806c8f0 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2261683893 0 0) 0x7f7fa8003520 con 0x7f7fc80b2b40 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 msgr2=0x7f7fc80acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:41.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 0x7f7fc80acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 >> v1:172.21.15.137:6789/0 conn(0x7f7fc806c8f0 legacy=0x7f7fc80b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc815e180 con 0x7f7fc80b2b40 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7fc00019f0 con 0x7f7fc80b2b40 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 >> v1:172.21.15.2:6789/0 conn(0x7f7fc80b2b40 legacy=0x7f7fc8161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fa8003b30 0x7f7fa8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 0x7f7fa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa8004d00 0x7f7fa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc815e840 con 0x7f7fa8003b30 2024-01-15T18:01:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa800ba30 con 0x7f7fa8004d00 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa800bbf0 con 0x7f7fc815ae80 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.785+0000 7f7fcda00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fa8003b30 0x7f7fa8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fcd000640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 0x7f7fa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fcda00640 1 -- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 msgr2=0x7f7fa80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fcda00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fc815ae80 0x7f7fa80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fce400640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa8004d00 0x7f7fa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fcda00640 1 -- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa8004d00 msgr2=0x7f7fa8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fcda00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fa8004d00 0x7f7fa8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.786+0000 7f7fcda00640 1 -- 172.21.15.2:0/3406501621 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa8003520 con 0x7f7fa8003b30 2024-01-15T18:01:41.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.787+0000 7f7fcda00640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fa8003b30 0x7f7fa8003f30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009ac0 tx=0x7f7fa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6b7824358dd40b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:41.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.787+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb802c6b0 con 0x7f7fa8003b30 2024-01-15T18:01:41.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.787+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fb802c810 con 0x7f7fa8003b30 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.787+0000 7f7fc7e00640 1 -- 172.21.15.2:0/3406501621 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fb802cba0 con 0x7f7fa8003b30 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fd0400640 1 -- 172.21.15.2:0/3406501621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fa8003b30 msgr2=0x7f7fa8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fd0400640 1 --2- 172.21.15.2:0/3406501621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fa8003b30 0x7f7fa8003f30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8009ac0 tx=0x7f7fa80057f0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fce400640 1 -- 172.21.15.2:0/3406501621 reap_dead start 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fd0400640 1 -- 172.21.15.2:0/3406501621 shutdown_connections 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fd0400640 1 -- 172.21.15.2:0/3406501621 >> 172.21.15.2:0/3406501621 conn(0x7f7fc80b45f0 msgr2=0x7f7fc80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fd0400640 1 -- 172.21.15.2:0/3406501621 shutdown_connections 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.788+0000 7f7fd0400640 1 -- 172.21.15.2:0/3406501621 wait complete. 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.790+0000 7f7fd0400640 1 Processor -- start 2024-01-15T18:01:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.790+0000 7f7fd0400640 1 -- start start 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fd0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fa8003b30 0x7f7fc80b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fd0400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fc814f180 0x7f7fc814ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fd0400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 0x7f7fc814cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fd0400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814b690 con 0x7f7fc814c730 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fd0400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814b970 con 0x7f7fc814f180 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fd0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fc814bc50 con 0x7f7fa8003b30 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fce400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 0x7f7fc814cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fce400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 0x7f7fc814cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51668/0 (socket says 172.21.15.2:51668) 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.791+0000 7f7fce400640 1 -- 172.21.15.2:0/1297876222 learned_addr learned my addr 172.21.15.2:0/1297876222 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fcda00640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fa8003b30 0x7f7fc80b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fcd000640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fc814f180 0x7f7fc814ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fce400640 1 -- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fa8003b30 msgr2=0x7f7fc80b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fce400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fa8003b30 0x7f7fc80b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fce400640 1 -- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fc814f180 msgr2=0x7f7fc814ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fce400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fc814f180 0x7f7fc814ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fce400640 1 -- 172.21.15.2:0/1297876222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc814bf30 con 0x7f7fc814c730 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fce400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 0x7f7fc814cbe0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7fc806e660 tx=0x7f7fc000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc0004d60 con 0x7f7fc814c730 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc000bb20 con 0x7f7fc814c730 2024-01-15T18:01:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc000bed0 con 0x7f7fc814c730 2024-01-15T18:01:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.792+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc814c210 con 0x7f7fc814c730 2024-01-15T18:01:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.793+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc814b260 con 0x7f7fc814c730 2024-01-15T18:01:41.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.794+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc80686f0 con 0x7f7fc814c730 2024-01-15T18:01:41.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.795+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f7fc0007070 con 0x7f7fc814c730 2024-01-15T18:01:41.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.796+0000 7f7fc5600640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fa006b590 0x7f7fa006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:41.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.796+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7fc008a5c0 con 0x7f7fc814c730 2024-01-15T18:01:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.796+0000 7f7fcda00640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fa006b590 0x7f7fa006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.798+0000 7f7fcda00640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fa006b590 0x7f7fa006da50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8036000 tx=0x7f7fb8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:41.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:41.800+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fc0054ee0 con 0x7f7fc814c730 2024-01-15T18:01:42.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.022+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fc814b480 con 0x7f7fa006b590 2024-01-15T18:01:42.038 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30 2024-01-15T18:01:42.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.033+0000 7f7fc5600640 1 -- 172.21.15.2:0/1297876222 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7fc814b480 con 0x7f7fa006b590 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fa006b590 msgr2=0x7f7fa006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fa006b590 0x7f7fa006da50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f7fb8036000 tx=0x7f7fb8030000 comp rx=0 tx=0).stop 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 msgr2=0x7f7fc814cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 0x7f7fc814cbe0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f7fc806e660 tx=0x7f7fc000b4a0 comp rx=0 tx=0).stop 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 shutdown_connections 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fa006b590 0x7f7fa006da50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fc814c730 0x7f7fc814cbe0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:42.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fc814f180 0x7f7fc814ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:42.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 --2- 172.21.15.2:0/1297876222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fa8003b30 0x7f7fc80b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:42.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 >> 172.21.15.2:0/1297876222 conn(0x7f7fc80b45f0 msgr2=0x7f7fc80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:42.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 shutdown_connections 2024-01-15T18:01:42.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:42.036+0000 7f7fd0400640 1 -- 172.21.15.2:0/1297876222 wait complete. 2024-01-15T18:01:42.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:01:42.060 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30) 2024-01-15T18:01:42.121 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30 2024-01-15T18:01:42.121 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:01:42.121 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:01:42.122 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:01:44.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:01:44.124 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30) 2024-01-15T18:01:44.194 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30 2024-01-15T18:01:44.194 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:01:44.194 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:01:44.196 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:01:46.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:01:46.197 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30) 2024-01-15T18:01:46.267 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/191300f5-a4bf-4f8e-a7c1-e584b3a86b30 2024-01-15T18:01:46.267 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:01:46.267 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:01:46.269 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:01:46.269 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone22401 2024-01-15T18:01:46.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.501+0000 7f6f40a00640 1 Processor -- start 2024-01-15T18:01:46.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.501+0000 7f6f40a00640 1 -- start start 2024-01-15T18:01:46.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.503+0000 7f6f40a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6f3c143d20 0x7f6f3c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.503+0000 7f6f40a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c0720d0 0x7f6f3c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.503+0000 7f6f40a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f3c071090 con 0x7f6f3c1447d0 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.503+0000 7f6f40a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f3c071430 con 0x7f6f3c143d20 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.503+0000 7f6f40a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c144100 con 0x7f6f3c0720d0 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.504+0000 7f6f3b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6f3c143d20 0x7f6f3c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.504+0000 7f6f3be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6f3c1447d0 0x7f6f3c0cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54512/0 (socket says 172.21.15.2:54512) 2024-01-15T18:01:46.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.504+0000 7f6f3be00640 1 -- 172.21.15.2:0/2204821940 learned_addr learned my addr 172.21.15.2:0/2204821940 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:01:46.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.504+0000 7f6f3aa00640 1 --2- 172.21.15.2:0/2204821940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c0720d0 0x7f6f3c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:46.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.504+0000 7f6f3a000640 1 -- 172.21.15.2:0/2204821940 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613541080 0 0) 0x7f6f3c071090 con 0x7f6f3c1447d0 2024-01-15T18:01:46.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.504+0000 7f6f3a000640 1 -- 172.21.15.2:0/2204821940 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f1c003520 con 0x7f6f3c1447d0 2024-01-15T18:01:46.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.505+0000 7f6f3b400640 1 -- 172.21.15.2:0/2204821940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c0720d0 msgr2=0x7f6f3c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:46.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.505+0000 7f6f3b400640 1 --2- 172.21.15.2:0/2204821940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c0720d0 0x7f6f3c15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.505+0000 7f6f3b400640 1 -- 172.21.15.2:0/2204821940 >> v1:172.21.15.2:6789/0 conn(0x7f6f3c1447d0 legacy=0x7f6f3c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:46.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.505+0000 7f6f3b400640 1 -- 172.21.15.2:0/2204821940 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f3c06d160 con 0x7f6f3c143d20 2024-01-15T18:01:46.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.506+0000 7f6f3b400640 1 --2- 172.21.15.2:0/2204821940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c143d20 0x7f6f3c15eb50 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0097a0 tx=0x7f6f2c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e11204df47ca7b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:46.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.507+0000 7f6f3a000640 1 -- 172.21.15.2:0/2204821940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c02e070 con 0x7f6f3c143d20 2024-01-15T18:01:46.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.507+0000 7f6f3a000640 1 -- 172.21.15.2:0/2204821940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c0315a0 con 0x7f6f3c143d20 2024-01-15T18:01:46.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.507+0000 7f6f3a000640 1 -- 172.21.15.2:0/2204821940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c031930 con 0x7f6f3c143d20 2024-01-15T18:01:46.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.507+0000 7f6f40a00640 1 -- 172.21.15.2:0/2204821940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c143d20 msgr2=0x7f6f3c15eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:46.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.507+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2204821940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c143d20 0x7f6f3c15eb50 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c0097a0 tx=0x7f6f2c02b840 comp rx=0 tx=0).stop 2024-01-15T18:01:46.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.508+0000 7f6f40a00640 1 -- 172.21.15.2:0/2204821940 shutdown_connections 2024-01-15T18:01:46.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.508+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2204821940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c0720d0 0x7f6f3c15f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.508+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2204821940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c143d20 0x7f6f3c15eb50 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.508+0000 7f6f40a00640 1 -- 172.21.15.2:0/2204821940 >> 172.21.15.2:0/2204821940 conn(0x7f6f3c0b0840 msgr2=0x7f6f3c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:46.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.508+0000 7f6f40a00640 1 -- 172.21.15.2:0/2204821940 shutdown_connections 2024-01-15T18:01:46.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.508+0000 7f6f40a00640 1 -- 172.21.15.2:0/2204821940 wait complete. 2024-01-15T18:01:46.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.510+0000 7f6f40a00640 1 Processor -- start 2024-01-15T18:01:46.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.510+0000 7f6f40a00640 1 -- start start 2024-01-15T18:01:46.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f40a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f3c0720d0 0x7f6f3c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:46.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f40a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c143d20 0x7f6f3c0c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:46.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f40a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 0x7f6f3c0c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f40a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c0c3ff0 con 0x7f6f3c1447d0 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f40a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c0c4220 con 0x7f6f3c0720d0 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f40a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3c0c4500 con 0x7f6f3c143d20 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 0x7f6f3c0c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 0x7f6f3c0c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41250/0 (socket says 172.21.15.2:41250) 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f3be00640 1 -- 172.21.15.2:0/2243518430 learned_addr learned my addr 172.21.15.2:0/2243518430 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.511+0000 7f6f3aa00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c143d20 0x7f6f3c0c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f3be00640 1 -- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c143d20 msgr2=0x7f6f3c0c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f3be00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c143d20 0x7f6f3c0c3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f3be00640 1 -- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f3c0720d0 msgr2=0x7f6f3c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f3be00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f3c0720d0 0x7f6f3c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f3be00640 1 -- 172.21.15.2:0/2243518430 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c009400 con 0x7f6f3c1447d0 2024-01-15T18:01:46.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f3be00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 0x7f6f3c0c8b90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400cb40 tx=0x7f6f3400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:46.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3400aca0 con 0x7f6f3c1447d0 2024-01-15T18:01:46.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3400ae00 con 0x7f6f3c1447d0 2024-01-15T18:01:46.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f3c0c47e0 con 0x7f6f3c1447d0 2024-01-15T18:01:46.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f34012620 con 0x7f6f3c1447d0 2024-01-15T18:01:46.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.512+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3c0c4da0 con 0x7f6f3c1447d0 2024-01-15T18:01:46.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.514+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3c0c4f90 con 0x7f6f3c1447d0 2024-01-15T18:01:46.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.514+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f6f34007070 con 0x7f6f3c1447d0 2024-01-15T18:01:46.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.515+0000 7f6f33400640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6f1406b590 0x7f6f1406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:46.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.515+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6f3408a160 con 0x7f6f3c1447d0 2024-01-15T18:01:46.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.516+0000 7f6f3b400640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6f1406b590 0x7f6f1406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:46.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.517+0000 7f6f3b400640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6f1406b590 0x7f6f1406da50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c036000 tx=0x7f6f2c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:46.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.520+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f34054b90 con 0x7f6f3c1447d0 2024-01-15T18:01:46.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.750+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone22401", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f3c15f720 con 0x7f6f1406b590 2024-01-15T18:01:46.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.911+0000 7f6f33400640 1 -- 172.21.15.2:0/2243518430 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f2c031970 con 0x7f6f1406b590 2024-01-15T18:01:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6f1406b590 msgr2=0x7f6f1406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6f1406b590 0x7f6f1406da50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c036000 tx=0x7f6f2c030000 comp rx=0 tx=0).stop 2024-01-15T18:01:46.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 msgr2=0x7f6f3c0c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 0x7f6f3c0c8b90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6f3400cb40 tx=0x7f6f3400a060 comp rx=0 tx=0).stop 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 shutdown_connections 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6f1406b590 0x7f6f1406da50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f3c1447d0 0x7f6f3c0c8b90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f3c143d20 0x7f6f3c0c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 --2- 172.21.15.2:0/2243518430 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f3c0720d0 0x7f6f3c0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 >> 172.21.15.2:0/2243518430 conn(0x7f6f3c0b0840 msgr2=0x7f6f3c0716a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 shutdown_connections 2024-01-15T18:01:46.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:46.915+0000 7f6f40a00640 1 -- 172.21.15.2:0/2243518430 wait complete. 2024-01-15T18:01:46.923 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.919+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.923 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.920+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.923 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.920+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.923 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.920+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.923 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.920+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.941 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone22402 2024-01-15T18:01:46.947 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.944+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.947 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.944+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.947 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.944+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.948 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.944+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:46.948 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:01:46.944+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:01:47.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.162+0000 7f332b600640 1 Processor -- start 2024-01-15T18:01:47.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.163+0000 7f332b600640 1 -- start start 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.164+0000 7f332b600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3324144c00 0x7f33240cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.164+0000 7f332b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f332406d000 0x7f3324070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.164+0000 7f332b600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33240ce230 con 0x7f33240cdd70 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.164+0000 7f332b600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33240ce510 con 0x7f3324144c00 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.164+0000 7f332b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33240ce7f0 con 0x7f332406d000 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3324144c00 0x7f33240cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3324144c00 0x7f33240cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41256/0 (socket says 172.21.15.2:41256) 2024-01-15T18:01:47.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 -- 172.21.15.2:0/3167778356 learned_addr learned my addr 172.21.15.2:0/3167778356 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:47.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3329600640 1 --2- 172.21.15.2:0/3167778356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f332406d000 0x7f3324070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 -- 172.21.15.2:0/3167778356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f332406d000 msgr2=0x7f3324070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:47.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 --2- 172.21.15.2:0/3167778356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f332406d000 0x7f3324070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 -- 172.21.15.2:0/3167778356 >> v1:172.21.15.137:6789/0 conn(0x7f33240cdd70 legacy=0x7f332406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:47.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.165+0000 7f3328c00640 1 -- 172.21.15.2:0/3167778356 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33240705f0 con 0x7f3324144c00 2024-01-15T18:01:47.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.166+0000 7f3328c00640 1 --2- 172.21.15.2:0/3167778356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33240cd830 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3314009870 tx=0x7f331402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fe7c074108694d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f3323e00640 1 -- 172.21.15.2:0/3167778356 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314031640 con 0x7f3324144c00 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f3323e00640 1 -- 172.21.15.2:0/3167778356 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33140317a0 con 0x7f3324144c00 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f3323e00640 1 -- 172.21.15.2:0/3167778356 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314031b30 con 0x7f3324144c00 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f332b600640 1 -- 172.21.15.2:0/3167778356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 msgr2=0x7f33240cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f332b600640 1 --2- 172.21.15.2:0/3167778356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33240cd830 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f3314009870 tx=0x7f331402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f332b600640 1 -- 172.21.15.2:0/3167778356 shutdown_connections 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f332b600640 1 --2- 172.21.15.2:0/3167778356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f332406d000 0x7f3324070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f332b600640 1 --2- 172.21.15.2:0/3167778356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33240cd830 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.167+0000 7f332b600640 1 -- 172.21.15.2:0/3167778356 >> 172.21.15.2:0/3167778356 conn(0x7f33240b45f0 msgr2=0x7f33240b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:47.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.168+0000 7f332b600640 1 -- 172.21.15.2:0/3167778356 shutdown_connections 2024-01-15T18:01:47.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.168+0000 7f332b600640 1 -- 172.21.15.2:0/3167778356 wait complete. 2024-01-15T18:01:47.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.169+0000 7f332b600640 1 Processor -- start 2024-01-15T18:01:47.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.170+0000 7f332b600640 1 -- start start 2024-01-15T18:01:47.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.170+0000 7f332b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f332406d000 0x7f332415d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.170+0000 7f332b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33240cdd70 0x7f3324159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.170+0000 7f332b600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33241f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.170+0000 7f332b600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332415d730 con 0x7f3324144c00 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f332b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332413bd50 con 0x7f332406d000 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f332b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33240b0660 con 0x7f33240cdd70 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33241f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33241f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41258/0 (socket says 172.21.15.2:41258) 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 -- 172.21.15.2:0/2009845648 learned_addr learned my addr 172.21.15.2:0/2009845648 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 -- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33240cdd70 msgr2=0x7f3324159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33240cdd70 0x7f3324159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 -- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f332406d000 msgr2=0x7f332415d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f332406d000 0x7f332415d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.171+0000 7f3329600640 1 -- 172.21.15.2:0/2009845648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33140094d0 con 0x7f3324144c00 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.172+0000 7f3329600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33241f24d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f331c00cd70 tx=0x7f331c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.172+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c003b90 con 0x7f3324144c00 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.172+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c003cf0 con 0x7f3324144c00 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.172+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331c004240 con 0x7f3324144c00 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.172+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33240b0940 con 0x7f3324144c00 2024-01-15T18:01:47.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.172+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33240b0f00 con 0x7f3324144c00 2024-01-15T18:01:47.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.173+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33240b14e0 con 0x7f3324144c00 2024-01-15T18:01:47.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.174+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f331c0043a0 con 0x7f3324144c00 2024-01-15T18:01:47.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.175+0000 7f3321600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32fc06b590 0x7f32fc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.175+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f331c012070 con 0x7f3324144c00 2024-01-15T18:01:47.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.176+0000 7f3328c00640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32fc06b590 0x7f32fc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.177+0000 7f3328c00640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32fc06b590 0x7f32fc06da50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3314036000 tx=0x7f3314030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:47.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.180+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f331c055570 con 0x7f3324144c00 2024-01-15T18:01:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.407+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone22402", "target": ["mon-mgr", ""]}) v1 -- 0x7f33240b0b10 con 0x7f32fc06b590 2024-01-15T18:01:47.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.580+0000 7f3321600640 1 -- 172.21.15.2:0/2009845648 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33240b0b10 con 0x7f32fc06b590 2024-01-15T18:01:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.583+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32fc06b590 msgr2=0x7f32fc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.583+0000 7f332b600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32fc06b590 0x7f32fc06da50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3314036000 tx=0x7f3314030000 comp rx=0 tx=0).stop 2024-01-15T18:01:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.583+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 msgr2=0x7f33241f24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.583+0000 7f332b600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33241f24d0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f331c00cd70 tx=0x7f331c00a660 comp rx=0 tx=0).stop 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 shutdown_connections 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32fc06b590 0x7f32fc06da50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3324144c00 0x7f33241f24d0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33240cdd70 0x7f3324159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 --2- 172.21.15.2:0/2009845648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f332406d000 0x7f332415d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 >> 172.21.15.2:0/2009845648 conn(0x7f33240b45f0 msgr2=0x7f332415b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 shutdown_connections 2024-01-15T18:01:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.584+0000 7f332b600640 1 -- 172.21.15.2:0/2009845648 wait complete. 2024-01-15T18:01:47.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone22403 2024-01-15T18:01:47.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.831+0000 7f3b85200640 1 Processor -- start 2024-01-15T18:01:47.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.831+0000 7f3b85200640 1 -- start start 2024-01-15T18:01:47.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.833+0000 7f3b85200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800b2210 0x7f3b800d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.833+0000 7f3b85200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b800d1c50 0x7f3b800b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.833+0000 7f3b85200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b800d1250 con 0x7f3b800d1870 2024-01-15T18:01:47.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.833+0000 7f3b85200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b800ce6d0 con 0x7f3b800d1c50 2024-01-15T18:01:47.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.833+0000 7f3b85200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800ce8e0 con 0x7f3b800b2210 2024-01-15T18:01:47.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.834+0000 7f3b7fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b800d1c50 0x7f3b800b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.834+0000 7f3b7fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3b800d1c50 0x7f3b800b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:51990/0 (socket says 172.21.15.2:51990) 2024-01-15T18:01:47.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.834+0000 7f3b7fe00640 1 -- 172.21.15.2:0/2230804445 learned_addr learned my addr 172.21.15.2:0/2230804445 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:47.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.834+0000 7f3b7ea00640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800b2210 0x7f3b800d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.835+0000 7f3b7fe00640 1 -- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800b2210 msgr2=0x7f3b800d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:47.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.835+0000 7f3b7fe00640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800b2210 0x7f3b800d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.835+0000 7f3b7fe00640 1 -- 172.21.15.2:0/2230804445 >> v1:172.21.15.137:6789/0 conn(0x7f3b800d1870 legacy=0x7f3b8015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.835+0000 7f3b7fe00640 1 -- 172.21.15.2:0/2230804445 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b800cebc0 con 0x7f3b800d1c50 2024-01-15T18:01:47.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.835+0000 7f3b7ea00640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800b2210 0x7f3b800d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:01:47.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.835+0000 7f3b7e000640 1 -- 172.21.15.2:0/2230804445 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211498616 0 0) 0x7f3b800d1250 con 0x7f3b800d1870 2024-01-15T18:01:47.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.836+0000 7f3b7fe00640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b800b26f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009e50 tx=0x7f3b7802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c48b13a17b42ab47 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:47.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b7e000640 1 -- 172.21.15.2:0/2230804445 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7802cb00 con 0x7f3b800d1c50 2024-01-15T18:01:47.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b7e000640 1 -- 172.21.15.2:0/2230804445 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7802cc60 con 0x7f3b800d1c50 2024-01-15T18:01:47.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b7e000640 1 -- 172.21.15.2:0/2230804445 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b780343f0 con 0x7f3b800d1c50 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b85200640 1 -- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 msgr2=0x7f3b800b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b85200640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b800b26f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f3b78009e50 tx=0x7f3b7802bf60 comp rx=0 tx=0).stop 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b85200640 1 -- 172.21.15.2:0/2230804445 shutdown_connections 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b85200640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b800b26f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b85200640 1 --2- 172.21.15.2:0/2230804445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800b2210 0x7f3b800d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.837+0000 7f3b85200640 1 -- 172.21.15.2:0/2230804445 >> 172.21.15.2:0/2230804445 conn(0x7f3b800d30b0 msgr2=0x7f3b800d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.838+0000 7f3b85200640 1 -- 172.21.15.2:0/2230804445 shutdown_connections 2024-01-15T18:01:47.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.838+0000 7f3b85200640 1 -- 172.21.15.2:0/2230804445 wait complete. 2024-01-15T18:01:47.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.839+0000 7f3b85200640 1 Processor -- start 2024-01-15T18:01:47.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.839+0000 7f3b85200640 1 -- start start 2024-01-15T18:01:47.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.840+0000 7f3b85200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 0x7f3b800c95c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.840+0000 7f3b85200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800d1870 0x7f3b801f7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.840+0000 7f3b85200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b8014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.840+0000 7f3b85200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800c9b00 con 0x7f3b800b2210 2024-01-15T18:01:47.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.840+0000 7f3b85200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b800b7400 con 0x7f3b800d1c50 2024-01-15T18:01:47.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.840+0000 7f3b85200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b80160b80 con 0x7f3b800d1870 2024-01-15T18:01:47.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 0x7f3b800c95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b8014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 0x7f3b800c95c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41272/0 (socket says 172.21.15.2:41272) 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7f400640 1 -- 172.21.15.2:0/2237173360 learned_addr learned my addr 172.21.15.2:0/2237173360 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7ea00640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800d1870 0x7f3b801f7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7f400640 1 -- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800d1870 msgr2=0x7f3b801f7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7f400640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800d1870 0x7f3b801f7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.841+0000 7f3b7f400640 1 -- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 msgr2=0x7f3b8014d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b7f400640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b8014d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b7f400640 1 -- 172.21.15.2:0/2237173360 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b68009400 con 0x7f3b800b2210 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b7f400640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 0x7f3b800c95c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000c460 tx=0x7f3b7000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b70005480 con 0x7f3b800b2210 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b700035f0 con 0x7f3b800b2210 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b78009b30 con 0x7f3b800b2210 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b700039a0 con 0x7f3b800b2210 2024-01-15T18:01:47.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.842+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b801f7690 con 0x7f3b800b2210 2024-01-15T18:01:47.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.844+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8014d830 con 0x7f3b800b2210 2024-01-15T18:01:47.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.845+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f3b70003b00 con 0x7f3b800b2210 2024-01-15T18:01:47.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.846+0000 7f3b77400640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5806b590 0x7f3b5806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:47.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.846+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3b70089b50 con 0x7f3b800b2210 2024-01-15T18:01:47.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.846+0000 7f3b7ea00640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5806b590 0x7f3b5806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:47.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.847+0000 7f3b7ea00640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5806b590 0x7f3b5806da50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3b80149dd0 tx=0x7f3b68008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:47.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:47.851+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b70054610 con 0x7f3b800b2210 2024-01-15T18:01:48.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:48.085+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone22403", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b80074770 con 0x7f3b5806b590 2024-01-15T18:01:48.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:48.425+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f3b700549e0 con 0x7f3b800b2210 2024-01-15T18:01:50.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.300+0000 7f3b77400640 1 -- 172.21.15.2:0/2237173360 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3b80074770 con 0x7f3b5806b590 2024-01-15T18:01:50.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5806b590 msgr2=0x7f3b5806da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5806b590 0x7f3b5806da50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3b80149dd0 tx=0x7f3b68008040 comp rx=0 tx=0).stop 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 msgr2=0x7f3b800c95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 0x7f3b800c95c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000c460 tx=0x7f3b7000c920 comp rx=0 tx=0).stop 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 shutdown_connections 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3b5806b590 0x7f3b5806da50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3b800d1c50 0x7f3b8014d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3b800d1870 0x7f3b801f7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.303+0000 7f3b85200640 1 --2- 172.21.15.2:0/2237173360 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3b800b2210 0x7f3b800c95c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.304+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 >> 172.21.15.2:0/2237173360 conn(0x7f3b800d30b0 msgr2=0x7f3b8015c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.304+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 shutdown_connections 2024-01-15T18:01:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.304+0000 7f3b85200640 1 -- 172.21.15.2:0/2237173360 wait complete. 2024-01-15T18:01:50.329 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone22404 2024-01-15T18:01:50.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.555+0000 7f443b800640 1 Processor -- start 2024-01-15T18:01:50.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.556+0000 7f443b800640 1 -- start start 2024-01-15T18:01:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.557+0000 7f443b800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f443406d000 0x7f44340b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.557+0000 7f443b800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443415dd70 con 0x7f44341439f0 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.557+0000 7f443b800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44340711b0 con 0x7f44340702c0 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.557+0000 7f443b800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4434143dd0 con 0x7f443406d000 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4439800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f443406d000 0x7f44340b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4438e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f44341439f0 0x7f44340cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54528/0 (socket says 172.21.15.2:54528) 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4438e00640 1 -- 172.21.15.2:0/384913221 learned_addr learned my addr 172.21.15.2:0/384913221 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4439800640 1 -- 172.21.15.2:0/384913221 >> v1:172.21.15.137:6789/0 conn(0x7f44340702c0 legacy=0x7f443406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4439800640 1 -- 172.21.15.2:0/384913221 >> v1:172.21.15.2:6789/0 conn(0x7f44341439f0 legacy=0x7f44340cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:50.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4439800640 1 -- 172.21.15.2:0/384913221 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434070910 con 0x7f443406d000 2024-01-15T18:01:50.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.558+0000 7f4433400640 1 -- 172.21.15.2:0/384913221 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946160844 0 0) 0x7f443415dd70 con 0x7f44341439f0 2024-01-15T18:01:50.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.559+0000 7f4439800640 1 --2- 172.21.15.2:0/384913221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443406d000 0x7f44340b3420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f442c0097a0 tx=0x7f442c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15420e2cf286db4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:50.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.560+0000 7f4433400640 1 -- 172.21.15.2:0/384913221 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c031830 con 0x7f443406d000 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.560+0000 7f4433400640 1 -- 172.21.15.2:0/384913221 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c031990 con 0x7f443406d000 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.560+0000 7f4433400640 1 -- 172.21.15.2:0/384913221 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c031d20 con 0x7f443406d000 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.560+0000 7f443b800640 1 -- 172.21.15.2:0/384913221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443406d000 msgr2=0x7f44340b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.560+0000 7f443b800640 1 --2- 172.21.15.2:0/384913221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443406d000 0x7f44340b3420 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f442c0097a0 tx=0x7f442c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.561+0000 7f443b800640 1 -- 172.21.15.2:0/384913221 shutdown_connections 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.561+0000 7f443b800640 1 --2- 172.21.15.2:0/384913221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443406d000 0x7f44340b3420 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.561+0000 7f443b800640 1 -- 172.21.15.2:0/384913221 >> 172.21.15.2:0/384913221 conn(0x7f44340b45f0 msgr2=0x7f44340b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.561+0000 7f443b800640 1 -- 172.21.15.2:0/384913221 shutdown_connections 2024-01-15T18:01:50.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.561+0000 7f443b800640 1 -- 172.21.15.2:0/384913221 wait complete. 2024-01-15T18:01:50.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.563+0000 7f443b800640 1 Processor -- start 2024-01-15T18:01:50.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.563+0000 7f443b800640 1 -- start start 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.564+0000 7f443b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443406d000 0x7f4434156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.564+0000 7f443b800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 0x7f443413a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.564+0000 7f443b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44341439f0 0x7f443415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.564+0000 7f443b800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44341591e0 con 0x7f44340702c0 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.564+0000 7f443b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4434158610 con 0x7f44341439f0 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.564+0000 7f443b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44341588b0 con 0x7f443406d000 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 0x7f443413a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 0x7f443413a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41292/0 (socket says 172.21.15.2:41292) 2024-01-15T18:01:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 -- 172.21.15.2:0/223374956 learned_addr learned my addr 172.21.15.2:0/223374956 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4439800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44341439f0 0x7f443415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4438e00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443406d000 0x7f4434156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 -- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443406d000 msgr2=0x7f4434156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443406d000 0x7f4434156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 -- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44341439f0 msgr2=0x7f443415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44341439f0 0x7f443415c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.565+0000 7f4433e00640 1 -- 172.21.15.2:0/223374956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c009400 con 0x7f44340702c0 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.566+0000 7f4433e00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 0x7f443413a670 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f442400c7c0 tx=0x7f442400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.566+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442400aa50 con 0x7f44340702c0 2024-01-15T18:01:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.566+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442400abb0 con 0x7f44340702c0 2024-01-15T18:01:50.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.566+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4434158b90 con 0x7f44340702c0 2024-01-15T18:01:50.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.566+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424012420 con 0x7f44340702c0 2024-01-15T18:01:50.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.566+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443413d810 con 0x7f44340702c0 2024-01-15T18:01:50.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.567+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44340686f0 con 0x7f44340702c0 2024-01-15T18:01:50.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.568+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f4424012580 con 0x7f44340702c0 2024-01-15T18:01:50.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.569+0000 7f4430c00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f440c06b6b0 0x7f440c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:50.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.569+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f442408a660 con 0x7f44340702c0 2024-01-15T18:01:50.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.570+0000 7f4438e00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f440c06b6b0 0x7f440c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:50.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.571+0000 7f4438e00640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f440c06b6b0 0x7f440c06db70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f441c000f60 tx=0x7f441c007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:50.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.574+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4424054e50 con 0x7f44340702c0 2024-01-15T18:01:50.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:50.800+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone22404", "target": ["mon-mgr", ""]}) v1 -- 0x7f4434070ed0 con 0x7f440c06b6b0 2024-01-15T18:01:55.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.313+0000 7f4430c00640 1 -- 172.21.15.2:0/223374956 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f441c007270 con 0x7f440c06b6b0 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f440c06b6b0 msgr2=0x7f440c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f440c06b6b0 0x7f440c06db70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f441c000f60 tx=0x7f441c007c00 comp rx=0 tx=0).stop 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 msgr2=0x7f443413a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 0x7f443413a670 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f442400c7c0 tx=0x7f442400cc80 comp rx=0 tx=0).stop 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 shutdown_connections 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f440c06b6b0 0x7f440c06db70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44341439f0 0x7f443415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44340702c0 0x7f443413a670 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 --2- 172.21.15.2:0/223374956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443406d000 0x7f4434156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 >> 172.21.15.2:0/223374956 conn(0x7f44340b45f0 msgr2=0x7f4434139c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:55.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 shutdown_connections 2024-01-15T18:01:55.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.317+0000 7f443b800640 1 -- 172.21.15.2:0/223374956 wait complete. 2024-01-15T18:01:55.344 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:01:55.344 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22401 2024-01-15T18:01:55.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.550+0000 7f2518c00640 1 Processor -- start 2024-01-15T18:01:55.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.551+0000 7f2518c00640 1 -- start start 2024-01-15T18:01:55.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.552+0000 7f2518c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2514144c00 0x7f25140cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:55.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.552+0000 7f2518c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140cdd70 0x7f251406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.552+0000 7f2518c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140ce230 con 0x7f251415c450 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.552+0000 7f2518c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25140ce510 con 0x7f2514144c00 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.552+0000 7f2518c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25140ce7f0 con 0x7f25140cdd70 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.553+0000 7f2512800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f251415c450 0x7f25140b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40880/0 (socket says 172.21.15.2:40880) 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.553+0000 7f2512800640 1 -- 172.21.15.2:0/3984080024 learned_addr learned my addr 172.21.15.2:0/3984080024 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.553+0000 7f2511e00640 1 --2- 172.21.15.2:0/3984080024 >> v2:172.21.15.2:3300/0 conn(0x7f2514144c00 0x7f25140cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.553+0000 7f2511400640 1 --2- 172.21.15.2:0/3984080024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140cdd70 0x7f251406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.553+0000 7f2510a00640 1 -- 172.21.15.2:0/3984080024 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216537829 0 0) 0x7f25140ce230 con 0x7f251415c450 2024-01-15T18:01:55.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.553+0000 7f2510a00640 1 -- 172.21.15.2:0/3984080024 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f4003520 con 0x7f251415c450 2024-01-15T18:01:55.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.554+0000 7f2511e00640 1 -- 172.21.15.2:0/3984080024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140cdd70 msgr2=0x7f251406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:01:55.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.554+0000 7f2511e00640 1 --2- 172.21.15.2:0/3984080024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140cdd70 0x7f251406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.554+0000 7f2511e00640 1 -- 172.21.15.2:0/3984080024 >> v1:172.21.15.2:6789/0 conn(0x7f251415c450 legacy=0x7f25140b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:55.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.554+0000 7f2511e00640 1 -- 172.21.15.2:0/3984080024 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25140705f0 con 0x7f2514144c00 2024-01-15T18:01:55.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.555+0000 7f2511e00640 1 --2- 172.21.15.2:0/3984080024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2514144c00 0x7f25140cd830 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f25040097a0 tx=0x7f250402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5460d2196f812505 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:55.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.555+0000 7f2510a00640 1 -- 172.21.15.2:0/3984080024 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f250402d040 con 0x7f2514144c00 2024-01-15T18:01:55.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.555+0000 7f2510a00640 1 -- 172.21.15.2:0/3984080024 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2504031810 con 0x7f2514144c00 2024-01-15T18:01:55.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.555+0000 7f2510a00640 1 -- 172.21.15.2:0/3984080024 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504031b80 con 0x7f2514144c00 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 -- 172.21.15.2:0/3984080024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2514144c00 msgr2=0x7f25140cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 --2- 172.21.15.2:0/3984080024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2514144c00 0x7f25140cd830 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f25040097a0 tx=0x7f250402b8b0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 -- 172.21.15.2:0/3984080024 shutdown_connections 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 --2- 172.21.15.2:0/3984080024 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f25140cdd70 0x7f251406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 --2- 172.21.15.2:0/3984080024 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2514144c00 0x7f25140cd830 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 -- 172.21.15.2:0/3984080024 >> 172.21.15.2:0/3984080024 conn(0x7f25140b45f0 msgr2=0x7f25140b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 -- 172.21.15.2:0/3984080024 shutdown_connections 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.556+0000 7f2518c00640 1 -- 172.21.15.2:0/3984080024 wait complete. 2024-01-15T18:01:55.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.558+0000 7f2518c00640 1 Processor -- start 2024-01-15T18:01:55.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.558+0000 7f2518c00640 1 -- start start 2024-01-15T18:01:55.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2518c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 0x7f2514141120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2518c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2514144c00 0x7f2514158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2518c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f251415c450 0x7f2514158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2518c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141592c0 con 0x7f25140cdd70 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2518c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25141f3830 con 0x7f2514144c00 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2518c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25141f3b10 con 0x7f251415c450 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2511e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 0x7f2514141120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2511e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 0x7f2514141120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59076/0 (socket says 172.21.15.2:59076) 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.559+0000 7f2511e00640 1 -- 172.21.15.2:0/2352848707 learned_addr learned my addr 172.21.15.2:0/2352848707 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2512800640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f251415c450 0x7f2514158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:55.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511400640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2514144c00 0x7f2514158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511e00640 1 -- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f251415c450 msgr2=0x7f2514158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511e00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f251415c450 0x7f2514158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511e00640 1 -- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2514144c00 msgr2=0x7f2514158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511e00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2514144c00 0x7f2514158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511e00640 1 -- 172.21.15.2:0/2352848707 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2504009400 con 0x7f25140cdd70 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511400640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2514144c00 0x7f2514158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.560+0000 7f2511e00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 0x7f2514141120 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2504000c00 tx=0x7f25040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.561+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504003820 con 0x7f25140cdd70 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.561+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f250403f070 con 0x7f25140cdd70 2024-01-15T18:01:55.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.561+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2504003bc0 con 0x7f25140cdd70 2024-01-15T18:01:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.561+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25141f3df0 con 0x7f25140cdd70 2024-01-15T18:01:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.561+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25141f43b0 con 0x7f25140cdd70 2024-01-15T18:01:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.562+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25140686f0 con 0x7f25140cdd70 2024-01-15T18:01:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.563+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f2504003d20 con 0x7f25140cdd70 2024-01-15T18:01:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.564+0000 7f250a000640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24ec06b6e0 0x7f24ec06dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:01:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.564+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2504041070 con 0x7f25140cdd70 2024-01-15T18:01:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.565+0000 7f2511400640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24ec06b6e0 0x7f24ec06dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:01:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.566+0000 7f2511400640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24ec06b6e0 0x7f24ec06dba0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f251415a960 tx=0x7f24fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:01:55.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.569+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25040771a0 con 0x7f25140cdd70 2024-01-15T18:01:55.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:01:55.798+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22401", "target": ["mon-mgr", ""]}) v1 -- 0x7f251406ce30 con 0x7f24ec06b6e0 2024-01-15T18:01:59.816 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:01:59.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.158+0000 7f250a000640 1 -- 172.21.15.2:0/2352848707 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f24fc0095d0 con 0x7f24ec06b6e0 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:02:00.162 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:00.163 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:00.163 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24ec06b6e0 msgr2=0x7f24ec06dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24ec06b6e0 0x7f24ec06dba0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f251415a960 tx=0x7f24fc008040 comp rx=0 tx=0).stop 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 msgr2=0x7f2514141120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 0x7f2514141120 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f2504000c00 tx=0x7f25040046c0 comp rx=0 tx=0).stop 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 shutdown_connections 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24ec06b6e0 0x7f24ec06dba0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f251415c450 0x7f2514158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2514144c00 0x7f2514158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 --2- 172.21.15.2:0/2352848707 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f25140cdd70 0x7f2514141120 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.162+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 >> 172.21.15.2:0/2352848707 conn(0x7f25140b45f0 msgr2=0x7f2514142060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:00.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.163+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 shutdown_connections 2024-01-15T18:02:00.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:00.163+0000 7f2518c00640 1 -- 172.21.15.2:0/2352848707 wait complete. 2024-01-15T18:02:00.190 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:02:01.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22401 2024-01-15T18:02:01.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.429+0000 7f3aa7600640 1 Processor -- start 2024-01-15T18:02:01.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.429+0000 7f3aa7600640 1 -- start start 2024-01-15T18:02:01.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa7600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3aa015c450 0x7f3aa006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa7600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3aa0070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa7600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa00ce230 con 0x7f3aa0144c00 2024-01-15T18:02:01.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa7600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa006d2c0 con 0x7f3aa015c450 2024-01-15T18:02:01.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa7600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00ce510 con 0x7f3aa006ce30 2024-01-15T18:02:01.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa4c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3aa0144c00 0x7f3aa00cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40890/0 (socket says 172.21.15.2:40890) 2024-01-15T18:02:01.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa4c00640 1 -- 172.21.15.2:0/2169386934 learned_addr learned my addr 172.21.15.2:0/2169386934 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3aa5600640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3aa0070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.431+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/2169386934 >> v2:172.21.15.137:3300/0 conn(0x7f3aa015c450 0x7f3aa006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047585838 0 0) 0x7f3aa00ce230 con 0x7f3aa0144c00 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a80003520 con 0x7f3aa0144c00 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 64277874 0 0) 0x7f3a80003520 con 0x7f3aa0144c00 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 msgr2=0x7f3aa0070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3aa0070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 >> v2:172.21.15.137:3300/0 conn(0x7f3aa015c450 msgr2=0x7f3aa006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 --2- 172.21.15.2:0/2169386934 >> v2:172.21.15.137:3300/0 conn(0x7f3aa015c450 0x7f3aa006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa00ce7f0 con 0x7f3aa0144c00 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3a90001670 con 0x7f3aa0144c00 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 >> v1:172.21.15.2:6789/0 conn(0x7f3aa0144c00 legacy=0x7f3aa00cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3a80003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a80003ef0 0x7f3a800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a80004d00 0x7f3a80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa006d2c0 con 0x7f3a80003ef0 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00ce510 con 0x7f3a80004d00 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8000b9d0 con 0x7f3aa006ce30 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/2169386934 >> v2:172.21.15.137:3300/0 conn(0x7f3aa015c450 0x7f3aa006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.432+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a80003ef0 0x7f3a800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3aa4c00640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3a80003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3aa5600640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a80004d00 0x7f3a80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3a9fe00640 1 -- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 msgr2=0x7f3a80003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3a80003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3a9fe00640 1 -- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a80004d00 msgr2=0x7f3a80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3a80004d00 0x7f3a80005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3a9fe00640 1 -- 172.21.15.2:0/2169386934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80003520 con 0x7f3a80003ef0 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.433+0000 7f3aa4c00640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3aa006ce30 0x7f3a80003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:01.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.434+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a80003ef0 0x7f3a800047c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3a880098a0 tx=0x7f3a800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcc9135c770abd13 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:01.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.434+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88037600 con 0x7f3a80003ef0 2024-01-15T18:02:01.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.434+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a88037760 con 0x7f3a80003ef0 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.434+0000 7f3a9f400640 1 -- 172.21.15.2:0/2169386934 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88037ad0 con 0x7f3a80003ef0 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa7600640 1 -- 172.21.15.2:0/2169386934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a80003ef0 msgr2=0x7f3a800047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa7600640 1 --2- 172.21.15.2:0/2169386934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3a80003ef0 0x7f3a800047c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f3a880098a0 tx=0x7f3a800057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa5600640 1 -- 172.21.15.2:0/2169386934 reap_dead start 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa7600640 1 -- 172.21.15.2:0/2169386934 shutdown_connections 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa7600640 1 -- 172.21.15.2:0/2169386934 >> 172.21.15.2:0/2169386934 conn(0x7f3aa00b45f0 msgr2=0x7f3aa00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa7600640 1 -- 172.21.15.2:0/2169386934 shutdown_connections 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.435+0000 7f3aa7600640 1 -- 172.21.15.2:0/2169386934 wait complete. 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.437+0000 7f3aa7600640 1 Processor -- start 2024-01-15T18:02:01.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.437+0000 7f3aa7600640 1 -- start start 2024-01-15T18:02:01.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.438+0000 7f3aa7600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a80003ef0 0x7f3aa01f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.438+0000 7f3aa7600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 0x7f3aa0130200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.438+0000 7f3aa7600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3aa0130740 0x7f3aa0155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.438+0000 7f3aa7600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa01f13f0 con 0x7f3aa01397f0 2024-01-15T18:02:01.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.438+0000 7f3aa7600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00b0580 con 0x7f3aa0130740 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.438+0000 7f3aa7600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa00b0860 con 0x7f3a80003ef0 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 0x7f3aa0130200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 0x7f3aa0130200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59102/0 (socket says 172.21.15.2:59102) 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 -- 172.21.15.2:0/1373748311 learned_addr learned my addr 172.21.15.2:0/1373748311 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3aa4c00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a80003ef0 0x7f3aa01f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3aa5600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3aa0130740 0x7f3aa0155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 -- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a80003ef0 msgr2=0x7f3aa01f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a80003ef0 0x7f3aa01f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 -- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3aa0130740 msgr2=0x7f3aa0155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3aa0130740 0x7f3aa0155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3a9fe00640 1 -- 172.21.15.2:0/1373748311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa0130bc0 con 0x7f3aa01397f0 2024-01-15T18:02:01.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.439+0000 7f3aa5600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3aa0130740 0x7f3aa0155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.440+0000 7f3a9fe00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 0x7f3aa0130200 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3a8802bff0 tx=0x7f3a88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.440+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8802f980 con 0x7f3aa01397f0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.440+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8802fae0 con 0x7f3aa01397f0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.440+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8802fe90 con 0x7f3aa01397f0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.440+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa0157870 con 0x7f3aa01397f0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.440+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa00cc410 con 0x7f3aa01397f0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.441+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa00686f0 con 0x7f3aa01397f0 2024-01-15T18:02:01.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.442+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f3a88042440 con 0x7f3aa01397f0 2024-01-15T18:02:01.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.443+0000 7f3a9cc00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a7806b6b0 0x7f3a7806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.443+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a8803e040 con 0x7f3aa01397f0 2024-01-15T18:02:01.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.444+0000 7f3aa4c00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a7806b6b0 0x7f3a7806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.445+0000 7f3aa4c00640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a7806b6b0 0x7f3a7806db70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a90001eb0 tx=0x7f3a90009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:01.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.448+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a88076f10 con 0x7f3aa01397f0 2024-01-15T18:02:01.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.687+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22401", "target": ["mon-mgr", ""]}) v1 -- 0x7f3aa01f13f0 con 0x7f3a7806b6b0 2024-01-15T18:02:01.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.710+0000 7f3a9cc00640 1 -- 172.21.15.2:0/1373748311 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f3aa01f13f0 con 0x7f3a7806b6b0 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 0.0, 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:02:01.714 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "0/3", 2024-01-15T18:02:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:02:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:02:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:01.715 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:01.715 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a7806b6b0 msgr2=0x7f3a7806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a7806b6b0 0x7f3a7806db70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3a90001eb0 tx=0x7f3a90009040 comp rx=0 tx=0).stop 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 msgr2=0x7f3aa0130200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 0x7f3aa0130200 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f3a8802bff0 tx=0x7f3a88004730 comp rx=0 tx=0).stop 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 shutdown_connections 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3a7806b6b0 0x7f3a7806db70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3aa0130740 0x7f3aa0155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3aa01397f0 0x7f3aa0130200 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 --2- 172.21.15.2:0/1373748311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3a80003ef0 0x7f3aa01f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 >> 172.21.15.2:0/1373748311 conn(0x7f3aa00b45f0 msgr2=0x7f3aa0142740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.713+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 shutdown_connections 2024-01-15T18:02:01.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.714+0000 7f3aa7600640 1 -- 172.21.15.2:0/1373748311 wait complete. 2024-01-15T18:02:01.737 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:01.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.951+0000 7fbddf400640 1 Processor -- start 2024-01-15T18:02:01.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.952+0000 7fbddf400640 1 -- start start 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.953+0000 7fbddf400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbdd813ce30 0x7fbdd815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.953+0000 7fbddf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 0x7fbdd8070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.953+0000 7fbddf400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd80aeab0 con 0x7fbdd815e170 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.953+0000 7fbddf400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd815e620 con 0x7fbdd813ce30 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.953+0000 7fbddf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80ce6d0 con 0x7fbdd80ce170 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.954+0000 7fbdcfe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbdd815e170 0x7fbdd80cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40900/0 (socket says 172.21.15.2:40900) 2024-01-15T18:02:01.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.954+0000 7fbdcfe00640 1 -- 172.21.15.2:0/3416342709 learned_addr learned my addr 172.21.15.2:0/3416342709 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.954+0000 7fbddca00640 1 --2- 172.21.15.2:0/3416342709 >> v2:172.21.15.137:3300/0 conn(0x7fbdd813ce30 0x7fbdd815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.954+0000 7fbddd400640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 0x7fbdd8070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.954+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453592065 0 0) 0x7fbdd80aeab0 con 0x7fbdd815e170 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.954+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb8003520 con 0x7fbdd815e170 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839476237 0 0) 0x7fbdb8003520 con 0x7fbdd815e170 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 msgr2=0x7fbdd8070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 0x7fbdd8070ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 >> v2:172.21.15.137:3300/0 conn(0x7fbdd813ce30 msgr2=0x7fbdd815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 --2- 172.21.15.2:0/3416342709 >> v2:172.21.15.137:3300/0 conn(0x7fbdd813ce30 0x7fbdd815dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8070380 con 0x7fbdd815e170 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fbdc0001690 con 0x7fbdd815e170 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 >> v1:172.21.15.2:6789/0 conn(0x7fbdd815e170 legacy=0x7fbdd80cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdb8003970 0x7fbdb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 0x7fbdb80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8004d00 0x7fbdb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd815e620 con 0x7fbdb8003970 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80ce6d0 con 0x7fbdb8004d00 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb800db90 con 0x7fbdd80ce170 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.955+0000 7fbddca00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdb8003970 0x7fbdb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbdcfe00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 0x7fbdb80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbddca00640 1 -- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 msgr2=0x7fbdb80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbddd400640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8004d00 0x7fbdb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbddca00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd80ce170 0x7fbdb80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbddca00640 1 -- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8004d00 msgr2=0x7fbdb8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:01.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbddca00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8004d00 0x7fbdb8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.956+0000 7fbddca00640 1 -- 172.21.15.2:0/3416342709 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb8003520 con 0x7fbdb8003970 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.957+0000 7fbddca00640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdb8003970 0x7fbdb8003d90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009ac0 tx=0x7fbdb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46d1bd8acc9ff8c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.957+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc802c710 con 0x7fbdb8003970 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.957+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc802c870 con 0x7fbdb8003970 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.957+0000 7fbdd7e00640 1 -- 172.21.15.2:0/3416342709 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc802cc00 con 0x7fbdb8003970 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddf400640 1 -- 172.21.15.2:0/3416342709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdb8003970 msgr2=0x7fbdb8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddf400640 1 --2- 172.21.15.2:0/3416342709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdb8003970 0x7fbdb8003d90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8009ac0 tx=0x7fbdb80057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddd400640 1 -- 172.21.15.2:0/3416342709 reap_dead start 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddf400640 1 -- 172.21.15.2:0/3416342709 shutdown_connections 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddf400640 1 -- 172.21.15.2:0/3416342709 >> 172.21.15.2:0/3416342709 conn(0x7fbdd806da20 msgr2=0x7fbdd806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddf400640 1 -- 172.21.15.2:0/3416342709 shutdown_connections 2024-01-15T18:02:01.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.958+0000 7fbddf400640 1 -- 172.21.15.2:0/3416342709 wait complete. 2024-01-15T18:02:01.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.960+0000 7fbddf400640 1 Processor -- start 2024-01-15T18:02:01.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.960+0000 7fbddf400640 1 -- start start 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbddf400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8003970 0x7fbdd813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbddf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 0x7fbdd8137030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbddf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd8137570 0x7fbdd80aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbddf400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd813fad0 con 0x7fbdd81367f0 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbddf400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd80cc670 con 0x7fbdb8003970 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbddf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd81379f0 con 0x7fbdd8137570 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbdcfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 0x7fbdd8137030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbdcfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 0x7fbdd8137030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59122/0 (socket says 172.21.15.2:59122) 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.961+0000 7fbdcfe00640 1 -- 172.21.15.2:0/157540239 learned_addr learned my addr 172.21.15.2:0/157540239 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbddca00640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8003970 0x7fbdd813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbddd400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd8137570 0x7fbdd80aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdcfe00640 1 -- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd8137570 msgr2=0x7fbdd80aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdcfe00640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd8137570 0x7fbdd80aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdcfe00640 1 -- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8003970 msgr2=0x7fbdd813fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdcfe00640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8003970 0x7fbdd813fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdcfe00640 1 -- 172.21.15.2:0/157540239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd80afc70 con 0x7fbdd81367f0 2024-01-15T18:02:01.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbddd400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd8137570 0x7fbdd80aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdcfe00640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 0x7fbdd8137030 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000d770 tx=0x7fbdc000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc00074e0 con 0x7fbdd81367f0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc0007640 con 0x7fbdd81367f0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0004a00 con 0x7fbdd81367f0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.962+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd8140430 con 0x7fbdd81367f0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.963+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd813f410 con 0x7fbdd81367f0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.964+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd80686f0 con 0x7fbdd81367f0 2024-01-15T18:02:01.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.965+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7fbdc0004b60 con 0x7fbdd81367f0 2024-01-15T18:02:01.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.965+0000 7fbdd5600640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb006b6e0 0x7fbdb006dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:01.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.966+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbdc0089c20 con 0x7fbdd81367f0 2024-01-15T18:02:01.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.966+0000 7fbddca00640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb006b6e0 0x7fbdb006dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:01.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.968+0000 7fbddca00640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb006b6e0 0x7fbdb006dba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8036000 tx=0x7fbdc8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:01.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:01.971+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdc0054390 con 0x7fbdd81367f0 2024-01-15T18:02:02.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.231+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbdd813a730 con 0x7fbdd81367f0 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.232+0000 7fbdd5600640 1 -- 172.21.15.2:0/157540239 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2093 (secure 0 0 0) 0x7fbdc0019090 con 0x7fbdd81367f0 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:02.236 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:02.237 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 289, 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:02.239 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 800, 2024-01-15T18:02:02.240 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3222207988, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 6721552384, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 766372560896, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 597, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 18595, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 0, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 5 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:02.241 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:02.242 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:02.243 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:02.244 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb006b6e0 msgr2=0x7fbdb006dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb006b6e0 0x7fbdb006dba0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbdc8036000 tx=0x7fbdc8030000 comp rx=0 tx=0).stop 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 msgr2=0x7fbdd8137030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 0x7fbdd8137030 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbdc000d770 tx=0x7fbdc000dc30 comp rx=0 tx=0).stop 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 shutdown_connections 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb006b6e0 0x7fbdb006dba0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbdd8137570 0x7fbdd80aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbdd81367f0 0x7fbdd8137030 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 --2- 172.21.15.2:0/157540239 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbdb8003970 0x7fbdd813fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 >> 172.21.15.2:0/157540239 conn(0x7fbdd806da20 msgr2=0x7fbdd813af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 shutdown_connections 2024-01-15T18:02:02.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:02.235+0000 7fbddf400640 1 -- 172.21.15.2:0/157540239 wait complete. 2024-01-15T18:02:04.257 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:04.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.553+0000 7ff15d200640 1 Processor -- start 2024-01-15T18:02:04.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.553+0000 7ff15d200640 1 -- start start 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff15d200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff15815d340 0x7ff1580cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff15d200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1580b3de0 con 0x7ff1580d3090 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff15d200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1580b40c0 con 0x7ff1580d1d90 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff15d200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1580d0e80 con 0x7ff15815d340 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff156a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7ff1580d3090 0x7ff1580d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50336/0 (socket says 172.21.15.2:50336) 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff156a00640 1 -- 172.21.15.2:0/3954868444 learned_addr learned my addr 172.21.15.2:0/3954868444 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.556+0000 7ff157400640 1 --2- 172.21.15.2:0/3954868444 >> v2:172.21.15.2:3300/0 conn(0x7ff15815d340 0x7ff1580cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.557+0000 7ff157400640 1 -- 172.21.15.2:0/3954868444 >> v1:172.21.15.137:6789/0 conn(0x7ff1580d1d90 legacy=0x7ff1580b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.557+0000 7ff157400640 1 -- 172.21.15.2:0/3954868444 >> v1:172.21.15.2:6789/0 conn(0x7ff1580d3090 legacy=0x7ff1580d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.557+0000 7ff157400640 1 -- 172.21.15.2:0/3954868444 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1580d1160 con 0x7ff15815d340 2024-01-15T18:02:04.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.558+0000 7ff157400640 1 --2- 172.21.15.2:0/3954868444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580cdd20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff1480097a0 tx=0x7ff14802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e1079083ebdcdc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:04.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.559+0000 7ff156000640 1 -- 172.21.15.2:0/3954868444 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148030710 con 0x7ff15815d340 2024-01-15T18:02:04.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.559+0000 7ff156000640 1 -- 172.21.15.2:0/3954868444 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14802e050 con 0x7ff15815d340 2024-01-15T18:02:04.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.559+0000 7ff156000640 1 -- 172.21.15.2:0/3954868444 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148030a90 con 0x7ff15815d340 2024-01-15T18:02:04.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.560+0000 7ff15d200640 1 -- 172.21.15.2:0/3954868444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 msgr2=0x7ff1580cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:04.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.560+0000 7ff15d200640 1 --2- 172.21.15.2:0/3954868444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580cdd20 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff1480097a0 tx=0x7ff14802b710 comp rx=0 tx=0).stop 2024-01-15T18:02:04.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.560+0000 7ff15d200640 1 -- 172.21.15.2:0/3954868444 shutdown_connections 2024-01-15T18:02:04.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.560+0000 7ff15d200640 1 --2- 172.21.15.2:0/3954868444 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580cdd20 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:04.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.560+0000 7ff15d200640 1 -- 172.21.15.2:0/3954868444 >> 172.21.15.2:0/3954868444 conn(0x7ff15808caf0 msgr2=0x7ff15809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:04.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.561+0000 7ff15d200640 1 -- 172.21.15.2:0/3954868444 shutdown_connections 2024-01-15T18:02:04.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.561+0000 7ff15d200640 1 -- 172.21.15.2:0/3954868444 wait complete. 2024-01-15T18:02:04.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.563+0000 7ff15d200640 1 Processor -- start 2024-01-15T18:02:04.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.563+0000 7ff15d200640 1 -- start start 2024-01-15T18:02:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff15d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1580d1d90 0x7ff1580c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff15d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1580d3090 0x7ff1580b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:04.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff15d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580b13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff15d200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580cab30 con 0x7ff15815d340 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff15d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1581f3ed0 con 0x7ff1580d3090 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff15d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580ca200 con 0x7ff1580d1d90 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.564+0000 7ff157e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580b13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.565+0000 7ff157e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580b13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60820/0 (socket says 172.21.15.2:60820) 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.565+0000 7ff157e00640 1 -- 172.21.15.2:0/3389486131 learned_addr learned my addr 172.21.15.2:0/3389486131 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.565+0000 7ff157e00640 1 -- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1580d1d90 msgr2=0x7ff1580c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.565+0000 7ff157e00640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1580d1d90 0x7ff1580c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.566+0000 7ff157e00640 1 -- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1580d3090 msgr2=0x7ff1580b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.566+0000 7ff157e00640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1580d3090 0x7ff1580b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.566+0000 7ff157e00640 1 -- 172.21.15.2:0/3389486131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148009400 con 0x7ff15815d340 2024-01-15T18:02:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.566+0000 7ff157e00640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580b13c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff15000c780 tx=0x7ff15000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.566+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150003d80 con 0x7ff15815d340 2024-01-15T18:02:04.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.566+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1580ca4e0 con 0x7ff15815d340 2024-01-15T18:02:04.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.567+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15000ce90 con 0x7ff15815d340 2024-01-15T18:02:04.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.567+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1500052b0 con 0x7ff15815d340 2024-01-15T18:02:04.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.567+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1580c8410 con 0x7ff15815d340 2024-01-15T18:02:04.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.568+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff158074770 con 0x7ff15815d340 2024-01-15T18:02:04.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.569+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7ff150005410 con 0x7ff15815d340 2024-01-15T18:02:04.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.570+0000 7ff14f400640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff13006b6b0 0x7ff13006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:04.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.571+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff150089c00 con 0x7ff15815d340 2024-01-15T18:02:04.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.578+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff150054430 con 0x7ff15815d340 2024-01-15T18:02:04.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.606+0000 7ff157400640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff13006b6b0 0x7ff13006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:04.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:04.616+0000 7ff157400640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff13006b6b0 0x7ff13006db70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff148034000 tx=0x7ff14802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:05.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.073+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff1580cac70 con 0x7ff15815d340 2024-01-15T18:02:05.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.074+0000 7ff14f400640 1 -- 172.21.15.2:0/3389486131 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2092 (secure 0 0 0) 0x7ff150017020 con 0x7ff15815d340 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:05.078 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:05.079 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 292, 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.080 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:05.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 800, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3222207988, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 6721552384, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 766372560896, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:05.082 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 426, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 9809, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 0, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 3 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:05.083 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:05.084 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:05.086 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:05.087 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:05.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff13006b6b0 msgr2=0x7ff13006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:05.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff13006b6b0 0x7ff13006db70 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff148034000 tx=0x7ff14802f000 comp rx=0 tx=0).stop 2024-01-15T18:02:05.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 msgr2=0x7ff1580b13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:05.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580b13c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff15000c780 tx=0x7ff15000cc40 comp rx=0 tx=0).stop 2024-01-15T18:02:05.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 shutdown_connections 2024-01-15T18:02:05.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff13006b6b0 0x7ff13006db70 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:05.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff15815d340 0x7ff1580b13c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:05.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff1580d3090 0x7ff1580b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:05.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 --2- 172.21.15.2:0/3389486131 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff1580d1d90 0x7ff1580c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:05.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 >> 172.21.15.2:0/3389486131 conn(0x7ff15808caf0 msgr2=0x7ff15806c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:05.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 shutdown_connections 2024-01-15T18:02:05.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:05.077+0000 7ff15d200640 1 -- 172.21.15.2:0/3389486131 wait complete. 2024-01-15T18:02:05.266 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:05.263+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:05.266 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:05.263+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:05.266 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:05.263+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:05.266 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:05.263+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:05.266 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:05.263+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:07.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:07.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.344+0000 7facfee00640 1 Processor -- start 2024-01-15T18:02:07.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.345+0000 7facfee00640 1 -- start start 2024-01-15T18:02:07.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.346+0000 7facfee00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7facf8159690 0x7facf815d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.346+0000 7facfee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf80ce5c0 0x7facf815d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.346+0000 7facfee00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf806cd30 con 0x7facf815dd00 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.346+0000 7facfee00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facf80b3e20 con 0x7facf8159690 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.346+0000 7facfee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf8070d60 con 0x7facf80ce5c0 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfda00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7facf8159690 0x7facf815d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfe400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7facf815dd00 0x7facf80b28a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50340/0 (socket says 172.21.15.2:50340) 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfe400640 1 -- 172.21.15.2:0/655398733 learned_addr learned my addr 172.21.15.2:0/655398733 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:07.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfd000640 1 --2- 172.21.15.2:0/655398733 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf80ce5c0 0x7facf815d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:07.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facf7e00640 1 -- 172.21.15.2:0/655398733 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973640170 0 0) 0x7facf806cd30 con 0x7facf815dd00 2024-01-15T18:02:07.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facf7e00640 1 -- 172.21.15.2:0/655398733 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd8003520 con 0x7facf815dd00 2024-01-15T18:02:07.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfda00640 1 -- 172.21.15.2:0/655398733 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf80ce5c0 msgr2=0x7facf815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:07.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfda00640 1 --2- 172.21.15.2:0/655398733 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf80ce5c0 0x7facf815d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfda00640 1 -- 172.21.15.2:0/655398733 >> v1:172.21.15.2:6789/0 conn(0x7facf815dd00 legacy=0x7facf80b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:07.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.347+0000 7facfda00640 1 -- 172.21.15.2:0/655398733 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf8071040 con 0x7facf8159690 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.348+0000 7facfda00640 1 --2- 172.21.15.2:0/655398733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf8159690 0x7facf815d280 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7face80097a0 tx=0x7face802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c6b7ea23919929 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facf7e00640 1 -- 172.21.15.2:0/655398733 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face802d040 con 0x7facf8159690 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facf7e00640 1 -- 172.21.15.2:0/655398733 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face8031810 con 0x7facf8159690 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facf7e00640 1 -- 172.21.15.2:0/655398733 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8031ba0 con 0x7facf8159690 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 -- 172.21.15.2:0/655398733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf8159690 msgr2=0x7facf815d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 --2- 172.21.15.2:0/655398733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf8159690 0x7facf815d280 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7face80097a0 tx=0x7face802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 -- 172.21.15.2:0/655398733 shutdown_connections 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 --2- 172.21.15.2:0/655398733 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf80ce5c0 0x7facf815d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 --2- 172.21.15.2:0/655398733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf8159690 0x7facf815d280 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 -- 172.21.15.2:0/655398733 >> 172.21.15.2:0/655398733 conn(0x7facf80b0840 msgr2=0x7facf80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:07.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.349+0000 7facfee00640 1 -- 172.21.15.2:0/655398733 shutdown_connections 2024-01-15T18:02:07.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.350+0000 7facfee00640 1 -- 172.21.15.2:0/655398733 wait complete. 2024-01-15T18:02:07.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.351+0000 7facfee00640 1 Processor -- start 2024-01-15T18:02:07.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.351+0000 7facfee00640 1 -- start start 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.352+0000 7facfee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf80ce5c0 0x7facf81fa070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.352+0000 7facfee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf8159690 0x7facf814d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.352+0000 7facfee00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 0x7facf814e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.352+0000 7facfee00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf815a520 con 0x7facf815dd00 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.352+0000 7facfee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf814d850 con 0x7facf80ce5c0 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.352+0000 7facfee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf814db30 con 0x7facf8159690 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 0x7facf814e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 0x7facf814e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60848/0 (socket says 172.21.15.2:60848) 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 -- 172.21.15.2:0/78715896 learned_addr learned my addr 172.21.15.2:0/78715896 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfd000640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf8159690 0x7facf814d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:07.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfda00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf80ce5c0 0x7facf81fa070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 -- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf8159690 msgr2=0x7facf814d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf8159690 0x7facf814d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 -- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf80ce5c0 msgr2=0x7facf81fa070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf80ce5c0 0x7facf81fa070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 -- 172.21.15.2:0/78715896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face8009400 con 0x7facf815dd00 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.353+0000 7facfe400640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 0x7facf814e5f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7facf000ca70 tx=0x7facf000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.354+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf000abd0 con 0x7facf815dd00 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.354+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf000ad30 con 0x7facf815dd00 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.354+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf814de10 con 0x7facf815dd00 2024-01-15T18:02:07.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.354+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0012600 con 0x7facf815dd00 2024-01-15T18:02:07.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.354+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf8158a50 con 0x7facf815dd00 2024-01-15T18:02:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.355+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf815a690 con 0x7facf815dd00 2024-01-15T18:02:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.356+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7facf0007070 con 0x7facf815dd00 2024-01-15T18:02:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.357+0000 7facf5600640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7facd006b740 0x7facd006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.357+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7facf008a2f0 con 0x7facf815dd00 2024-01-15T18:02:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.357+0000 7facfda00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7facd006b740 0x7facd006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:07.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.359+0000 7facfda00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7facd006b740 0x7facd006dc00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7face8036000 tx=0x7face803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:07.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.361+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facf0054ac0 con 0x7facf815dd00 2024-01-15T18:02:07.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.632+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7facf8074770 con 0x7facf815dd00 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.633+0000 7facf5600640 1 -- 172.21.15.2:0/78715896 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2106 (secure 0 0 0) 0x7facf00580e0 con 0x7facf815dd00 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:07.637 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 295, 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:07.638 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:07.639 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1117, 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 4551906636, 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 8877621248, 2024-01-15T18:02:07.640 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 764216492032, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 120194750, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 110779707, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 29, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 32 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:07.641 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:07.642 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:07.643 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:07.644 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:02:07.644 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:02:07.644 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:07.644 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:07.644 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:07.644 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7facd006b740 msgr2=0x7facd006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7facd006b740 0x7facd006dc00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7face8036000 tx=0x7face803c040 comp rx=0 tx=0).stop 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 msgr2=0x7facf814e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 0x7facf814e5f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7facf000ca70 tx=0x7facf000cf30 comp rx=0 tx=0).stop 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 shutdown_connections 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7facd006b740 0x7facd006dc00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7facf815dd00 0x7facf814e5f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7facf8159690 0x7facf814d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 --2- 172.21.15.2:0/78715896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7facf80ce5c0 0x7facf81fa070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 >> 172.21.15.2:0/78715896 conn(0x7facf80b0840 msgr2=0x7facf806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.635+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 shutdown_connections 2024-01-15T18:02:07.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:07.636+0000 7facfee00640 1 -- 172.21.15.2:0/78715896 wait complete. 2024-01-15T18:02:09.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:09.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.963+0000 7f9c12e00640 1 Processor -- start 2024-01-15T18:02:09.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.964+0000 7f9c12e00640 1 -- start start 2024-01-15T18:02:09.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.968+0000 7f9c12e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9c0c0c3fe0 0x7f9c0c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:09.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.968+0000 7f9c12e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c0c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:09.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.968+0000 7f9c12e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c09aa00 con 0x7f9c0c0700b0 2024-01-15T18:02:09.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.968+0000 7f9c12e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c09a720 con 0x7f9c0c0c3fe0 2024-01-15T18:02:09.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.969+0000 7f9c12e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c06e3b0 con 0x7f9c0c161a90 2024-01-15T18:02:09.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.970+0000 7f9c0b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c0c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:09.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.970+0000 7f9c0b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c0c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:39552/0 (socket says 172.21.15.2:39552) 2024-01-15T18:02:09.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.970+0000 7f9c0b400640 1 -- 172.21.15.2:0/2762415045 learned_addr learned my addr 172.21.15.2:0/2762415045 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:09.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.972+0000 7f9c0be00640 1 --2- 172.21.15.2:0/2762415045 >> v2:172.21.15.137:3300/0 conn(0x7f9c0c0c3fe0 0x7f9c0c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:09.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.973+0000 7f9c0be00640 1 -- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 msgr2=0x7f9c0c0c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:09.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.973+0000 7f9c0be00640 1 --2- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c0c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:09.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.973+0000 7f9c0be00640 1 -- 172.21.15.2:0/2762415045 >> v1:172.21.15.137:6789/0 conn(0x7f9c0c0700b0 legacy=0x7f9c0c1653b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:09.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.973+0000 7f9c0be00640 1 -- 172.21.15.2:0/2762415045 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c06e690 con 0x7f9c0c0c3fe0 2024-01-15T18:02:09.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.973+0000 7f9c0b400640 1 --2- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:02:09.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.977+0000 7f9c0be00640 1 --2- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 0x7f9c0c0b04b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0097a0 tx=0x7f9bfc02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b21bcac9b6fcb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:09.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.978+0000 7f9c0aa00640 1 -- 172.21.15.2:0/2762415045 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc0316f0 con 0x7f9c0c0c3fe0 2024-01-15T18:02:09.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.979+0000 7f9c0aa00640 1 -- 172.21.15.2:0/2762415045 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc031850 con 0x7f9c0c0c3fe0 2024-01-15T18:02:09.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.979+0000 7f9c0aa00640 1 -- 172.21.15.2:0/2762415045 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc031be0 con 0x7f9c0c0c3fe0 2024-01-15T18:02:09.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 -- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 msgr2=0x7f9c0c0b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 --2- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 0x7f9c0c0b04b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc0097a0 tx=0x7f9bfc02b790 comp rx=0 tx=0).stop 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 -- 172.21.15.2:0/2762415045 shutdown_connections 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 --2- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c0c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 --2- 172.21.15.2:0/2762415045 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 0x7f9c0c0b04b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 -- 172.21.15.2:0/2762415045 >> 172.21.15.2:0/2762415045 conn(0x7f9c0c0b2b00 msgr2=0x7f9c0c0b2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 -- 172.21.15.2:0/2762415045 shutdown_connections 2024-01-15T18:02:09.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.980+0000 7f9c12e00640 1 -- 172.21.15.2:0/2762415045 wait complete. 2024-01-15T18:02:09.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.984+0000 7f9c12e00640 1 Processor -- start 2024-01-15T18:02:09.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.984+0000 7f9c12e00640 1 -- start start 2024-01-15T18:02:09.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.986+0000 7f9c12e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 0x7f9c0c162120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:09.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.986+0000 7f9c12e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 0x7f9c0c15f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:09.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.986+0000 7f9c12e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c14be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:09.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.986+0000 7f9c12e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c15fc80 con 0x7f9c0c0700b0 2024-01-15T18:02:09.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.986+0000 7f9c12e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c15ff60 con 0x7f9c0c0c3fe0 2024-01-15T18:02:09.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.986+0000 7f9c12e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c160240 con 0x7f9c0c161a90 2024-01-15T18:02:09.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.987+0000 7f9c0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 0x7f9c0c162120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:09.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.987+0000 7f9c0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 0x7f9c0c162120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60860/0 (socket says 172.21.15.2:60860) 2024-01-15T18:02:09.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.987+0000 7f9c0be00640 1 -- 172.21.15.2:0/3869708998 learned_addr learned my addr 172.21.15.2:0/3869708998 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:09.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.988+0000 7f9c10e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c14be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:09.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.988+0000 7f9c0be00640 1 -- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 msgr2=0x7f9c0c14be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:09.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.988+0000 7f9c0be00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c14be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:09.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.988+0000 7f9c0be00640 1 -- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 msgr2=0x7f9c0c15f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:09.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.988+0000 7f9c0be00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 0x7f9c0c15f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:09.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.988+0000 7f9c0be00640 1 -- 172.21.15.2:0/3869708998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf40094d0 con 0x7f9c0c0700b0 2024-01-15T18:02:09.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.990+0000 7f9c0be00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 0x7f9c0c162120 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc000c00 tx=0x7f9bfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:09.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.990+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc034680 con 0x7f9c0c0700b0 2024-01-15T18:02:09.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.990+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc0347e0 con 0x7f9c0c0700b0 2024-01-15T18:02:09.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.990+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bfc034b90 con 0x7f9c0c0700b0 2024-01-15T18:02:09.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.991+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bfc009400 con 0x7f9c0c0700b0 2024-01-15T18:02:09.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.991+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c160800 con 0x7f9c0c0700b0 2024-01-15T18:02:09.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.992+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c160d20 con 0x7f9c0c0700b0 2024-01-15T18:02:09.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.994+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7f9bfc03b040 con 0x7f9c0c0700b0 2024-01-15T18:02:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.996+0000 7f9c03e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9be406b740 0x7f9be406dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:09.996+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9bfc0aba40 con 0x7f9c0c0700b0 2024-01-15T18:02:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.007+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bfc076080 con 0x7f9c0c0700b0 2024-01-15T18:02:10.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.025+0000 7f9c0b400640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9be406b740 0x7f9be406dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:10.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.028+0000 7f9c0b400640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9be406b740 0x7f9be406dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4006f40 tx=0x7f9bf4006e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:10.371 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:10.366+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:10.371 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:10.366+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:10.371 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:10.366+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:10.371 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:10.366+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:10.371 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:10.366+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:10.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.509+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9c0c0686f0 con 0x7f9c0c0700b0 2024-01-15T18:02:10.519 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:10.519 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:10.519 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:10.519 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:10.519 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:10.520 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 298, 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.521 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.522 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1117, 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 4551906636, 2024-01-15T18:02:10.523 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 8877621248, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 764216492032, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 120178143, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 110760201, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 29, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 31 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:10.524 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:10.525 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:10.526 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:10.527 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:10.528 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-15T18:02:10.528 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.22220000624656677, 2024-01-15T18:02:10.528 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:10.528 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:10.528 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:10.528 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:10.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.512+0000 7f9c03e00640 1 -- 172.21.15.2:0/3869708998 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2142 (secure 0 0 0) 0x7f9bfc04d870 con 0x7f9c0c0700b0 2024-01-15T18:02:10.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.517+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9be406b740 msgr2=0x7f9be406dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:10.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.517+0000 7f9c12e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9be406b740 0x7f9be406dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4006f40 tx=0x7f9bf4006e90 comp rx=0 tx=0).stop 2024-01-15T18:02:10.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.517+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 msgr2=0x7f9c0c162120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.517+0000 7f9c12e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 0x7f9c0c162120 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9bfc000c00 tx=0x7f9bfc0046c0 comp rx=0 tx=0).stop 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 shutdown_connections 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9be406b740 0x7f9be406dc00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9c0c161a90 0x7f9c0c14be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9c0c0c3fe0 0x7f9c0c15f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 --2- 172.21.15.2:0/3869708998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9c0c0700b0 0x7f9c0c162120 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 >> 172.21.15.2:0/3869708998 conn(0x7f9c0c0b2b00 msgr2=0x7f9c0c06cbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 shutdown_connections 2024-01-15T18:02:10.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:10.518+0000 7f9c12e00640 1 -- 172.21.15.2:0/3869708998 wait complete. 2024-01-15T18:02:12.552 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:12.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.773+0000 7fb1e0400640 1 Processor -- start 2024-01-15T18:02:12.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.773+0000 7fb1e0400640 1 -- start start 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.775+0000 7fb1e0400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb1d80b2f90 0x7fb1d8160e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.775+0000 7fb1e0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d815d890 0x7fb1d8070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.775+0000 7fb1e0400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb1d80b3370 0x7fb1d806e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.775+0000 7fb1e0400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d8070fb0 con 0x7fb1d80b3370 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.775+0000 7fb1e0400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d8146ed0 con 0x7fb1d80b2f90 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.775+0000 7fb1e0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d8161430 con 0x7fb1d815d890 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb1d80b3370 0x7fb1d806e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb1d80b3370 0x7fb1d806e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60870/0 (socket says 172.21.15.2:60870) 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 -- 172.21.15.2:0/3018374578 learned_addr learned my addr 172.21.15.2:0/3018374578 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:12.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1dd000640 1 --2- 172.21.15.2:0/3018374578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d815d890 0x7fb1d8070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1dda00640 1 --2- 172.21.15.2:0/3018374578 >> v2:172.21.15.137:3300/0 conn(0x7fb1d80b2f90 0x7fb1d8160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 -- 172.21.15.2:0/3018374578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d815d890 msgr2=0x7fb1d8070a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:12.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 --2- 172.21.15.2:0/3018374578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d815d890 0x7fb1d8070a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 -- 172.21.15.2:0/3018374578 >> v2:172.21.15.137:3300/0 conn(0x7fb1d80b2f90 msgr2=0x7fb1d8160e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:12.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 --2- 172.21.15.2:0/3018374578 >> v2:172.21.15.137:3300/0 conn(0x7fb1d80b2f90 0x7fb1d8160e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.776+0000 7fb1de400640 1 -- 172.21.15.2:0/3018374578 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d8161710 con 0x7fb1d80b3370 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.778+0000 7fb1de400640 1 --2- 172.21.15.2:0/3018374578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d806e750 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80707d0 tx=0x7fb1d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76ed4a7dcdb7c4a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.778+0000 7fb1d7e00640 1 -- 172.21.15.2:0/3018374578 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0031870 con 0x7fb1d80b3370 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.778+0000 7fb1d7e00640 1 -- 172.21.15.2:0/3018374578 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d00319d0 con 0x7fb1d80b3370 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.778+0000 7fb1d7e00640 1 -- 172.21.15.2:0/3018374578 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d0031d60 con 0x7fb1d80b3370 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 -- 172.21.15.2:0/3018374578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 msgr2=0x7fb1d806e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3018374578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d806e750 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb1d80707d0 tx=0x7fb1d002b910 comp rx=0 tx=0).stop 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 -- 172.21.15.2:0/3018374578 shutdown_connections 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3018374578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d806e750 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3018374578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d815d890 0x7fb1d8070a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3018374578 >> v2:172.21.15.137:3300/0 conn(0x7fb1d80b2f90 0x7fb1d8160e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 -- 172.21.15.2:0/3018374578 >> 172.21.15.2:0/3018374578 conn(0x7fb1d80663d0 msgr2=0x7fb1d80667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:12.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 -- 172.21.15.2:0/3018374578 shutdown_connections 2024-01-15T18:02:12.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.779+0000 7fb1e0400640 1 -- 172.21.15.2:0/3018374578 wait complete. 2024-01-15T18:02:12.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.782+0000 7fb1e0400640 1 Processor -- start 2024-01-15T18:02:12.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.782+0000 7fb1e0400640 1 -- start start 2024-01-15T18:02:12.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1e0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d80b2f90 0x7fb1d80cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1e0400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d81605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1e0400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1d815d890 0x7fb1d80cc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1e0400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d8160b00 con 0x7fb1d80b3370 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1e0400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d8160cf0 con 0x7fb1d815d890 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1e0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d80cacf0 con 0x7fb1d80b2f90 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1dd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d81605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1dd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d81605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60878/0 (socket says 172.21.15.2:60878) 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.783+0000 7fb1dd000640 1 -- 172.21.15.2:0/3205261435 learned_addr learned my addr 172.21.15.2:0/3205261435 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dda00640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d80b2f90 0x7fb1d80cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1de400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1d815d890 0x7fb1d80cc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dd000640 1 -- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d80b2f90 msgr2=0x7fb1d80cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dd000640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d80b2f90 0x7fb1d80cb180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dd000640 1 -- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1d815d890 msgr2=0x7fb1d80cc9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dd000640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1d815d890 0x7fb1d80cc9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dd000640 1 -- 172.21.15.2:0/3205261435 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d00094d0 con 0x7fb1d80b3370 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1de400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1d815d890 0x7fb1d80cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1dd000640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d81605c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000d620 tx=0x7fb1c000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.784+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000b620 con 0x7fb1d80b3370 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.785+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c000b780 con 0x7fb1d80b3370 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.785+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d80ccef0 con 0x7fb1d80b3370 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.785+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000bb30 con 0x7fb1d80b3370 2024-01-15T18:02:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.785+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d80ca6a0 con 0x7fb1d80b3370 2024-01-15T18:02:12.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.786+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d80cae80 con 0x7fb1d80b3370 2024-01-15T18:02:12.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.787+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fb1c000bc90 con 0x7fb1d80b3370 2024-01-15T18:02:12.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.788+0000 7fb1d5600640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1b006b7d0 0x7fb1b006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:12.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.788+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb1c008a870 con 0x7fb1d80b3370 2024-01-15T18:02:12.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.788+0000 7fb1dda00640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1b006b7d0 0x7fb1b006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:12.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.790+0000 7fb1dda00640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1b006b7d0 0x7fb1b006dc90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1d81f7180 tx=0x7fb1c80039b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:12.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:12.793+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1c0055080 con 0x7fb1d80b3370 2024-01-15T18:02:13.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.055+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb1d80686f0 con 0x7fb1d80b3370 2024-01-15T18:02:13.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.057+0000 7fb1d5600640 1 -- 172.21.15.2:0/3205261435 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2143 (secure 0 0 0) 0x7fb1c00586a0 con 0x7fb1d80b3370 2024-01-15T18:02:13.061 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:13.061 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:13.061 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:13.062 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 300, 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:13.063 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:13.064 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:13.065 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1592, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 6544309737, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 12648591360, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 760445521920, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 288190535, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 276687260, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 69, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 74 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.066 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:13.067 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:13.068 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:13.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.22220000624656677, 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:13.070 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1b006b7d0 msgr2=0x7fb1b006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1b006b7d0 0x7fb1b006dc90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb1d81f7180 tx=0x7fb1c80039b0 comp rx=0 tx=0).stop 2024-01-15T18:02:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 msgr2=0x7fb1d81605c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:13.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d81605c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000d620 tx=0x7fb1c000dae0 comp rx=0 tx=0).stop 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 shutdown_connections 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1b006b7d0 0x7fb1b006dc90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb1d815d890 0x7fb1d80cc9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1d80b3370 0x7fb1d81605c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 --2- 172.21.15.2:0/3205261435 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1d80b2f90 0x7fb1d80cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 >> 172.21.15.2:0/3205261435 conn(0x7fb1d80663d0 msgr2=0x7fb1d80b1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 shutdown_connections 2024-01-15T18:02:13.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:13.059+0000 7fb1e0400640 1 -- 172.21.15.2:0/3205261435 wait complete. 2024-01-15T18:02:15.084 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:15.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.358+0000 7f745bc00640 1 Processor -- start 2024-01-15T18:02:15.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.359+0000 7f745bc00640 1 -- start start 2024-01-15T18:02:15.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.361+0000 7f745bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f745413ce30 0x7f745415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:15.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.361+0000 7f745bc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f74540ce170 0x7f745406c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.361+0000 7f745bc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74540aeab0 con 0x7f745415e170 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.361+0000 7f745bc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745415e5f0 con 0x7f74540ce170 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.361+0000 7f745bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74540ce6d0 con 0x7f745413ce30 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.362+0000 7f7459c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f74540ce170 0x7f745406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.362+0000 7f7459c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f74540ce170 0x7f745406c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45282/0 (socket says 172.21.15.2:45282) 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.362+0000 7f7459c00640 1 -- 172.21.15.2:0/122767979 learned_addr learned my addr 172.21.15.2:0/122767979 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.362+0000 7f7459c00640 1 -- 172.21.15.2:0/122767979 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f745413ce30 msgr2=0x7f745415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.362+0000 7f7459c00640 1 --2- 172.21.15.2:0/122767979 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f745413ce30 0x7f745415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.362+0000 7f7459c00640 1 -- 172.21.15.2:0/122767979 >> v1:172.21.15.137:6789/0 conn(0x7f745415e170 legacy=0x7f74540cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:15.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.363+0000 7f7459c00640 1 -- 172.21.15.2:0/122767979 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7454070380 con 0x7f74540ce170 2024-01-15T18:02:15.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.364+0000 7f7459c00640 1 --2- 172.21.15.2:0/122767979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74540ce170 0x7f745406c470 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f744c009e50 tx=0x7f744c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc59ffcefff6a6f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:15.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.365+0000 7f7453e00640 1 -- 172.21.15.2:0/122767979 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c02ca70 con 0x7f74540ce170 2024-01-15T18:02:15.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.365+0000 7f7453e00640 1 -- 172.21.15.2:0/122767979 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c02cbd0 con 0x7f74540ce170 2024-01-15T18:02:15.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.365+0000 7f7453e00640 1 -- 172.21.15.2:0/122767979 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c0343f0 con 0x7f74540ce170 2024-01-15T18:02:15.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.369+0000 7f745bc00640 1 -- 172.21.15.2:0/122767979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74540ce170 msgr2=0x7f745406c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:15.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.369+0000 7f745bc00640 1 --2- 172.21.15.2:0/122767979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74540ce170 0x7f745406c470 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f744c009e50 tx=0x7f744c02bed0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.371+0000 7f745bc00640 1 -- 172.21.15.2:0/122767979 shutdown_connections 2024-01-15T18:02:15.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.371+0000 7f745bc00640 1 --2- 172.21.15.2:0/122767979 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74540ce170 0x7f745406c470 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.371+0000 7f745bc00640 1 --2- 172.21.15.2:0/122767979 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f745413ce30 0x7f745415dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.371+0000 7f745bc00640 1 -- 172.21.15.2:0/122767979 >> 172.21.15.2:0/122767979 conn(0x7f745406da20 msgr2=0x7f745406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:15.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.374+0000 7f745bc00640 1 -- 172.21.15.2:0/122767979 shutdown_connections 2024-01-15T18:02:15.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.374+0000 7f745bc00640 1 -- 172.21.15.2:0/122767979 wait complete. 2024-01-15T18:02:15.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.378+0000 7f745bc00640 1 Processor -- start 2024-01-15T18:02:15.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.378+0000 7f745bc00640 1 -- start start 2024-01-15T18:02:15.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.380+0000 7f745bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74540ce170 0x7f74541412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.380+0000 7f745bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f745413ce30 0x7f7454131fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.380+0000 7f745bc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 0x7f74540b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.380+0000 7f745bc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7454153d10 con 0x7f745415e170 2024-01-15T18:02:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.380+0000 7f745bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7454142100 con 0x7f745413ce30 2024-01-15T18:02:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.380+0000 7f745bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74540b0ac0 con 0x7f74540ce170 2024-01-15T18:02:15.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.381+0000 7f7459c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 0x7f74540b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.381+0000 7f7459c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 0x7f74540b14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45288/0 (socket says 172.21.15.2:45288) 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.381+0000 7f7459c00640 1 -- 172.21.15.2:0/1656729633 learned_addr learned my addr 172.21.15.2:0/1656729633 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.382+0000 7f7459c00640 1 -- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74540ce170 msgr2=0x7f74541412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.382+0000 7f7459200640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74540ce170 0x7f74541412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.382+0000 7f7459c00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74540ce170 0x7f74541412f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.382+0000 7f7459c00640 1 -- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f745413ce30 msgr2=0x7f7454131fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.382+0000 7f7459c00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f745413ce30 0x7f7454131fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.382+0000 7f7459c00640 1 -- 172.21.15.2:0/1656729633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c009b30 con 0x7f745415e170 2024-01-15T18:02:15.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.383+0000 7f7459200640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74540ce170 0x7f74541412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.383+0000 7f7459c00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 0x7f74540b14d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f744c036040 tx=0x7f744c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:15.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.384+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c03e450 con 0x7f745415e170 2024-01-15T18:02:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.384+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744c03e5b0 con 0x7f745415e170 2024-01-15T18:02:15.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.384+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744c03e960 con 0x7f745415e170 2024-01-15T18:02:15.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.386+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7454141b40 con 0x7f745415e170 2024-01-15T18:02:15.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.386+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74540b0cd0 con 0x7f745415e170 2024-01-15T18:02:15.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.389+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74540b11c0 con 0x7f745415e170 2024-01-15T18:02:15.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.390+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f744c03eac0 con 0x7f745415e170 2024-01-15T18:02:15.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.391+0000 7f7451600640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f742c06b820 0x7f742c06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:15.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.392+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f744c0ad260 con 0x7f745415e170 2024-01-15T18:02:15.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.397+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f744c077b20 con 0x7f745415e170 2024-01-15T18:02:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.412+0000 7f7459200640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f742c06b820 0x7f742c06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:15.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.413+0000 7f7459200640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f742c06b820 0x7f742c06dce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f745415da30 tx=0x7f7444008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:15.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.933+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f74540686f0 con 0x7f745415e170 2024-01-15T18:02:15.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.934+0000 7f7451600640 1 -- 172.21.15.2:0/1656729633 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2147 (secure 0 0 0) 0x7f744c043880 con 0x7f745415e170 2024-01-15T18:02:15.938 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:15.938 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:15.939 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 303, 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:15.940 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.941 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2185, 2024-01-15T18:02:15.942 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 9031652989, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 17933643776, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 755160469504, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 497146086, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 483560546, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 119, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 125 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:15.943 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:15.944 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.945 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.41667500138282776, 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:15.946 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:15.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f742c06b820 msgr2=0x7f742c06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:15.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f742c06b820 0x7f742c06dce0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f745415da30 tx=0x7f7444008040 comp rx=0 tx=0).stop 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 msgr2=0x7f74540b14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 0x7f74540b14d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f744c036040 tx=0x7f744c004a40 comp rx=0 tx=0).stop 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 shutdown_connections 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f742c06b820 0x7f742c06dce0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f745415e170 0x7f74540b14d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f745413ce30 0x7f7454131fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 --2- 172.21.15.2:0/1656729633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74540ce170 0x7f74541412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 >> 172.21.15.2:0/1656729633 conn(0x7f745406da20 msgr2=0x7f745406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 shutdown_connections 2024-01-15T18:02:15.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:15.938+0000 7f745bc00640 1 -- 172.21.15.2:0/1656729633 wait complete. 2024-01-15T18:02:17.966 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.272+0000 7f049ba00640 1 Processor -- start 2024-01-15T18:02:18.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.272+0000 7f049ba00640 1 -- start start 2024-01-15T18:02:18.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f049ba00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0494162440 0x7f049406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f049ba00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0494071cb0 con 0x7f04940b3c70 2024-01-15T18:02:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f049ba00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0494070a70 con 0x7f049406c470 2024-01-15T18:02:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f049ba00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0494071610 con 0x7f0494162440 2024-01-15T18:02:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f0499a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0494162440 0x7f049406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f0493e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f049406c470 0x7f0494161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:50984/0 (socket says 172.21.15.2:50984) 2024-01-15T18:02:18.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.275+0000 7f0493e00640 1 -- 172.21.15.2:0/1532046392 learned_addr learned my addr 172.21.15.2:0/1532046392 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:18.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.277+0000 7f0499a00640 1 -- 172.21.15.2:0/1532046392 >> v1:172.21.15.137:6789/0 conn(0x7f049406c470 legacy=0x7f0494161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.277+0000 7f0499a00640 1 -- 172.21.15.2:0/1532046392 >> v1:172.21.15.2:6789/0 conn(0x7f04940b3c70 legacy=0x7f0494070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.277+0000 7f0499a00640 1 -- 172.21.15.2:0/1532046392 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494162960 con 0x7f0494162440 2024-01-15T18:02:18.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.277+0000 7f0493400640 1 -- 172.21.15.2:0/1532046392 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613944927 0 0) 0x7f0494071cb0 con 0x7f04940b3c70 2024-01-15T18:02:18.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.279+0000 7f0499a00640 1 --2- 172.21.15.2:0/1532046392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0494162440 0x7f049406c8f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f048c009d90 tx=0x7f048c02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c759c9f69264a0b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:18.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.280+0000 7f0493400640 1 -- 172.21.15.2:0/1532046392 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c02ccf0 con 0x7f0494162440 2024-01-15T18:02:18.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.280+0000 7f0493400640 1 -- 172.21.15.2:0/1532046392 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f048c02ce50 con 0x7f0494162440 2024-01-15T18:02:18.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.280+0000 7f0493400640 1 -- 172.21.15.2:0/1532046392 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f048c034660 con 0x7f0494162440 2024-01-15T18:02:18.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.281+0000 7f049ba00640 1 -- 172.21.15.2:0/1532046392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0494162440 msgr2=0x7f049406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.281+0000 7f049ba00640 1 --2- 172.21.15.2:0/1532046392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0494162440 0x7f049406c8f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f048c009d90 tx=0x7f048c02bf40 comp rx=0 tx=0).stop 2024-01-15T18:02:18.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.282+0000 7f049ba00640 1 -- 172.21.15.2:0/1532046392 shutdown_connections 2024-01-15T18:02:18.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.282+0000 7f049ba00640 1 --2- 172.21.15.2:0/1532046392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0494162440 0x7f049406c8f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.282+0000 7f049ba00640 1 -- 172.21.15.2:0/1532046392 >> 172.21.15.2:0/1532046392 conn(0x7f049406da20 msgr2=0x7f049406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:18.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.282+0000 7f049ba00640 1 -- 172.21.15.2:0/1532046392 shutdown_connections 2024-01-15T18:02:18.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.283+0000 7f049ba00640 1 -- 172.21.15.2:0/1532046392 wait complete. 2024-01-15T18:02:18.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.288+0000 7f049ba00640 1 Processor -- start 2024-01-15T18:02:18.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.288+0000 7f049ba00640 1 -- start start 2024-01-15T18:02:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.289+0000 7f049ba00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 0x7f049414c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.290+0000 7f049ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04940b3c70 0x7f0494153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.290+0000 7f049ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0494162440 0x7f04940abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.290+0000 7f049ba00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0494153650 con 0x7f049406c470 2024-01-15T18:02:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.290+0000 7f049ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04940a9760 con 0x7f0494162440 2024-01-15T18:02:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.290+0000 7f049ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04941fb5d0 con 0x7f04940b3c70 2024-01-15T18:02:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.290+0000 7f0499000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 0x7f049414c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:18.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.291+0000 7f0499000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 0x7f049414c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45308/0 (socket says 172.21.15.2:45308) 2024-01-15T18:02:18.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.291+0000 7f0499000640 1 -- 172.21.15.2:0/142643738 learned_addr learned my addr 172.21.15.2:0/142643738 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.292+0000 7f0493e00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04940b3c70 0x7f0494153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.292+0000 7f0499000640 1 -- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04940b3c70 msgr2=0x7f0494153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.292+0000 7f0499000640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04940b3c70 0x7f0494153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.292+0000 7f0499000640 1 -- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0494162440 msgr2=0x7f04940abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.292+0000 7f0499000640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0494162440 0x7f04940abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.292+0000 7f0499000640 1 -- 172.21.15.2:0/142643738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048c009a70 con 0x7f049406c470 2024-01-15T18:02:18.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.293+0000 7f0499000640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 0x7f049414c9d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c450 tx=0x7f047c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:18.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.294+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c005480 con 0x7f049406c470 2024-01-15T18:02:18.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.294+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c0035f0 con 0x7f049406c470 2024-01-15T18:02:18.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.295+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c0039c0 con 0x7f049406c470 2024-01-15T18:02:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.298+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04941fa810 con 0x7f049406c470 2024-01-15T18:02:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.298+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04941fadd0 con 0x7f049406c470 2024-01-15T18:02:18.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.304+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04940686f0 con 0x7f049406c470 2024-01-15T18:02:18.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.312+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f047c003b20 con 0x7f049406c470 2024-01-15T18:02:18.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.313+0000 7f0490c00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f046c06b7d0 0x7f046c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:18.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.314+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f047c089fa0 con 0x7f049406c470 2024-01-15T18:02:18.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.322+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f047c0547b0 con 0x7f049406c470 2024-01-15T18:02:18.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.371+0000 7f0493e00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f046c06b7d0 0x7f046c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:18.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.409+0000 7f0493e00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f046c06b7d0 0x7f046c06dc90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0484005880 tx=0x7f0484009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:18.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.774+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0494164ef0 con 0x7f049406c470 2024-01-15T18:02:18.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.775+0000 7f0490c00640 1 -- 172.21.15.2:0/142643738 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2145 (secure 0 0 0) 0x7f047c057dd0 con 0x7f049406c470 2024-01-15T18:02:18.780 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:18.780 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:18.780 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:18.780 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:18.780 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:18.780 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:18.781 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 306, 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:18.782 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:18.783 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:18.784 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2185, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 9031652989, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 17933643776, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 755160469504, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:18.785 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 376936288, 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 372768818, 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 90, 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 95 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:18.786 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:18.787 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:18.788 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:18.789 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "10fed56b-6fdc-4f75-b7e8-f012a44889a9": { 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 16s)", 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.41667500138282776, 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:02:18.790 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:18.791 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:18.791 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.782+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f046c06b7d0 msgr2=0x7f046c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.782+0000 7f049ba00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f046c06b7d0 0x7f046c06dc90 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0484005880 tx=0x7f0484009040 comp rx=0 tx=0).stop 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.782+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 msgr2=0x7f049414c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.782+0000 7f049ba00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 0x7f049414c9d0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f047c00c450 tx=0x7f047c00c910 comp rx=0 tx=0).stop 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.783+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 shutdown_connections 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.783+0000 7f049ba00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f046c06b7d0 0x7f046c06dc90 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.783+0000 7f049ba00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0494162440 0x7f04940abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.783+0000 7f049ba00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04940b3c70 0x7f0494153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.783+0000 7f049ba00640 1 --2- 172.21.15.2:0/142643738 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f049406c470 0x7f049414c9d0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:18.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.784+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 >> 172.21.15.2:0/142643738 conn(0x7f049406da20 msgr2=0x7f0494161d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:18.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.784+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 shutdown_connections 2024-01-15T18:02:18.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:18.784+0000 7f049ba00640 1 -- 172.21.15.2:0/142643738 wait complete. 2024-01-15T18:02:20.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:02:21.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.047+0000 7ff06b000640 1 Processor -- start 2024-01-15T18:02:21.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.047+0000 7ff06b000640 1 -- start start 2024-01-15T18:02:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.048+0000 7ff06b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0640cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff06b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff0640d1d90 0x7ff0640d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff06b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640d3a30 con 0x7ff0640d3090 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff06b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0640b3de0 con 0x7ff0640d1d90 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff06b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640b40c0 con 0x7ff06415d340 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff069c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0640cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff069c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0640cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:59100/0 (socket says 172.21.15.2:59100) 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff069c00640 1 -- 172.21.15.2:0/2749390935 learned_addr learned my addr 172.21.15.2:0/2749390935 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.049+0000 7ff06a600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff0640d1d90 0x7ff0640d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.050+0000 7ff069c00640 1 -- 172.21.15.2:0/2749390935 >> v2:172.21.15.137:3300/0 conn(0x7ff0640d1d90 msgr2=0x7ff0640d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.050+0000 7ff069c00640 1 --2- 172.21.15.2:0/2749390935 >> v2:172.21.15.137:3300/0 conn(0x7ff0640d1d90 0x7ff0640d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.050+0000 7ff069c00640 1 -- 172.21.15.2:0/2749390935 >> v1:172.21.15.137:6789/0 conn(0x7ff0640d3090 legacy=0x7ff0640d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.050+0000 7ff069c00640 1 -- 172.21.15.2:0/2749390935 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640d0e80 con 0x7ff06415d340 2024-01-15T18:02:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.050+0000 7ff06a600640 1 --2- 172.21.15.2:0/2749390935 >> v2:172.21.15.137:3300/0 conn(0x7ff0640d1d90 0x7ff0640d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:02:21.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.052+0000 7ff069c00640 1 --2- 172.21.15.2:0/2749390935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0640cdd20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff054009870 tx=0x7ff05402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b41306fbca15bebf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff063e00640 1 -- 172.21.15.2:0/2749390935 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054030840 con 0x7ff06415d340 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff063e00640 1 -- 172.21.15.2:0/2749390935 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0540309a0 con 0x7ff06415d340 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff063e00640 1 -- 172.21.15.2:0/2749390935 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054030d30 con 0x7ff06415d340 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff06b000640 1 -- 172.21.15.2:0/2749390935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 msgr2=0x7ff0640cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff06b000640 1 --2- 172.21.15.2:0/2749390935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0640cdd20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff054009870 tx=0x7ff05402b980 comp rx=0 tx=0).stop 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff06b000640 1 -- 172.21.15.2:0/2749390935 shutdown_connections 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff06b000640 1 --2- 172.21.15.2:0/2749390935 >> v2:172.21.15.137:3300/0 conn(0x7ff0640d1d90 0x7ff0640d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff06b000640 1 --2- 172.21.15.2:0/2749390935 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0640cdd20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.053+0000 7ff06b000640 1 -- 172.21.15.2:0/2749390935 >> 172.21.15.2:0/2749390935 conn(0x7ff06408caf0 msgr2=0x7ff06409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.054+0000 7ff06b000640 1 -- 172.21.15.2:0/2749390935 shutdown_connections 2024-01-15T18:02:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.054+0000 7ff06b000640 1 -- 172.21.15.2:0/2749390935 wait complete. 2024-01-15T18:02:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.055+0000 7ff06b000640 1 Processor -- start 2024-01-15T18:02:21.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.055+0000 7ff06b000640 1 -- start start 2024-01-15T18:02:21.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.056+0000 7ff06b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0640d1d90 0x7ff0640b7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.056+0000 7ff06b000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 0x7ff0640cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.056+0000 7ff06b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0641502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.056+0000 7ff06b000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640b6e40 con 0x7ff0640d3090 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.056+0000 7ff06b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640b7dd0 con 0x7ff0640d1d90 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.056+0000 7ff06b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff064150d60 con 0x7ff06415d340 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 0x7ff0640cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 0x7ff0640cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45322/0 (socket says 172.21.15.2:45322) 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 -- 172.21.15.2:0/1652634732 learned_addr learned my addr 172.21.15.2:0/1652634732 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff06a600640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0641502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069c00640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0640d1d90 0x7ff0640b7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 -- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 msgr2=0x7ff0641502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0641502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 -- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0640d1d90 msgr2=0x7ff0640b7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0640d1d90 0x7ff0640b7890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff069200640 1 -- 172.21.15.2:0/1652634732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c009a70 con 0x7ff0640d3090 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.057+0000 7ff06a600640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0641502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.058+0000 7ff069200640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 0x7ff0640cce60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c400 tx=0x7ff04c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.058+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0052d0 con 0x7ff0640d3090 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.058+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff04c005430 con 0x7ff0640d3090 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.058+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04c003830 con 0x7ff0640d3090 2024-01-15T18:02:21.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.058+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0540094d0 con 0x7ff0640d3090 2024-01-15T18:02:21.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.058+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0640b7090 con 0x7ff0640d3090 2024-01-15T18:02:21.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.059+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff064074770 con 0x7ff0640d3090 2024-01-15T18:02:21.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.060+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7ff04c003990 con 0x7ff0640d3090 2024-01-15T18:02:21.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.061+0000 7ff061600640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff03c06b7d0 0x7ff03c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.061+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff04c089950 con 0x7ff0640d3090 2024-01-15T18:02:21.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.062+0000 7ff069c00640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff03c06b7d0 0x7ff03c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.063+0000 7ff069c00640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff03c06b7d0 0x7ff03c06dc90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff054036000 tx=0x7ff054008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:21.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.066+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff04c054160 con 0x7ff0640d3090 2024-01-15T18:02:21.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.320+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff0640678e0 con 0x7ff0640d3090 2024-01-15T18:02:21.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.321+0000 7ff061600640 1 -- 172.21.15.2:0/1652634732 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1867 (secure 0 0 0) 0x7ff04c017760 con 0x7ff0640d3090 2024-01-15T18:02:21.325 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:21.325 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:21.325 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:02:21.325 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:02:21.325 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 309, 2024-01-15T18:02:21.326 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 49, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:02:21.327 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2223, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 9191036541, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 18697007104, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 754397106176, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:02:21.328 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 389864820, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 386046219, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 93, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 99 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 27, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 3, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 3, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:02:21.329 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 5721 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.330 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:02:21.331 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 10, 2024-01-15T18:02:21.331 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:01:45.465843+0000", 2024-01-15T18:02:21.331 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:02:21.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:21.331 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": {} 2024-01-15T18:02:21.331 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.323+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff03c06b7d0 msgr2=0x7ff03c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.323+0000 7ff06b000640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff03c06b7d0 0x7ff03c06dc90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff054036000 tx=0x7ff054008030 comp rx=0 tx=0).stop 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 msgr2=0x7ff0640cce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 0x7ff0640cce60 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff04c00c400 tx=0x7ff04c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 shutdown_connections 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff03c06b7d0 0x7ff03c06dc90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff06415d340 0x7ff0641502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff0640d3090 0x7ff0640cce60 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 --2- 172.21.15.2:0/1652634732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff0640d1d90 0x7ff0640b7890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 >> 172.21.15.2:0/1652634732 conn(0x7ff06408caf0 msgr2=0x7ff0640b1d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 shutdown_connections 2024-01-15T18:02:21.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.324+0000 7ff06b000640 1 -- 172.21.15.2:0/1652634732 wait complete. 2024-01-15T18:02:21.344 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:02:21.345 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22401 2024-01-15T18:02:21.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.551+0000 7f4b83a00640 1 Processor -- start 2024-01-15T18:02:21.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.551+0000 7f4b83a00640 1 -- start start 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b83a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b7c15c450 0x7f4b7c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b83a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 0x7f4b7c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b83a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b7c0ce230 con 0x7f4b7c144c00 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b83a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b7c06d2c0 con 0x7f4b7c15c450 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b83a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0ce510 con 0x7f4b7c06ce30 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b81000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4b7c144c00 0x7f4b7c0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39272/0 (socket says 172.21.15.2:39272) 2024-01-15T18:02:21.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.553+0000 7f4b81000640 1 -- 172.21.15.2:0/309806039 learned_addr learned my addr 172.21.15.2:0/309806039 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b81a00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 0x7f4b7c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082218846 0 0) 0x7f4b7c0ce230 con 0x7f4b7c144c00 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5c003520 con 0x7f4b7c144c00 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7be00640 1 --2- 172.21.15.2:0/309806039 >> v2:172.21.15.137:3300/0 conn(0x7f4b7c15c450 0x7f4b7c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 980325761 0 0) 0x7f4b5c003520 con 0x7f4b7c144c00 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 msgr2=0x7f4b7c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 0x7f4b7c070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 >> v2:172.21.15.137:3300/0 conn(0x7f4b7c15c450 msgr2=0x7f4b7c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 --2- 172.21.15.2:0/309806039 >> v2:172.21.15.137:3300/0 conn(0x7f4b7c15c450 0x7f4b7c06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0ce7f0 con 0x7f4b7c144c00 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4b6c001670 con 0x7f4b7c144c00 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 >> v1:172.21.15.2:6789/0 conn(0x7f4b7c144c00 legacy=0x7f4b7c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 0x7f4b5c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c003ef0 0x7f4b5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b5c004d00 0x7f4b5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c06d2c0 con 0x7f4b5c004d00 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c0ce510 con 0x7f4b5c003ef0 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.554+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c00b9d0 con 0x7f4b7c06ce30 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81a00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b5c004d00 0x7f4b5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81000640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 0x7f4b5c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b7be00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c003ef0 0x7f4b5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81a00640 1 -- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 msgr2=0x7f4b5c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81a00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c06ce30 0x7f4b5c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81a00640 1 -- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c003ef0 msgr2=0x7f4b5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81a00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c003ef0 0x7f4b5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.555+0000 7f4b81a00640 1 -- 172.21.15.2:0/309806039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c003520 con 0x7f4b5c004d00 2024-01-15T18:02:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.556+0000 7f4b81a00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b5c004d00 0x7f4b5c005180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009fc0 tx=0x7f4b5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed78c99f7458bc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:21.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7402cc20 con 0x7f4b5c004d00 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7400f040 con 0x7f4b5c004d00 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b7b400640 1 -- 172.21.15.2:0/309806039 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b74034440 con 0x7f4b5c004d00 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b83a00640 1 -- 172.21.15.2:0/309806039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b5c004d00 msgr2=0x7f4b5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b83a00640 1 --2- 172.21.15.2:0/309806039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b5c004d00 0x7f4b5c005180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009fc0 tx=0x7f4b5c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b81a00640 1 -- 172.21.15.2:0/309806039 reap_dead start 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b83a00640 1 -- 172.21.15.2:0/309806039 shutdown_connections 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b83a00640 1 -- 172.21.15.2:0/309806039 >> 172.21.15.2:0/309806039 conn(0x7f4b7c0b45f0 msgr2=0x7f4b7c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b83a00640 1 -- 172.21.15.2:0/309806039 shutdown_connections 2024-01-15T18:02:21.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.557+0000 7f4b83a00640 1 -- 172.21.15.2:0/309806039 wait complete. 2024-01-15T18:02:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.560+0000 7f4b83a00640 1 Processor -- start 2024-01-15T18:02:21.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.560+0000 7f4b83a00640 1 -- start start 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.561+0000 7f4b83a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c004d00 0x7f4b7c154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.561+0000 7f4b83a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 0x7f4b7c13b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.561+0000 7f4b83a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c1560d0 0x7f4b7c158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.561+0000 7f4b83a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c1531c0 con 0x7f4b7c152d40 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.561+0000 7f4b83a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c1533b0 con 0x7f4b5c004d00 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.561+0000 7f4b83a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7c153690 con 0x7f4b7c1560d0 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 0x7f4b7c13b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 0x7f4b7c13b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45334/0 (socket says 172.21.15.2:45334) 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 -- 172.21.15.2:0/2921670427 learned_addr learned my addr 172.21.15.2:0/2921670427 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:21.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b81000640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c004d00 0x7f4b7c154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b81a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c1560d0 0x7f4b7c158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 -- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c1560d0 msgr2=0x7f4b7c158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c1560d0 0x7f4b7c158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 -- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c004d00 msgr2=0x7f4b7c154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c004d00 0x7f4b7c154450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.562+0000 7f4b7be00640 1 -- 172.21.15.2:0/2921670427 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c153970 con 0x7f4b7c152d40 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.563+0000 7f4b7be00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 0x7f4b7c13b870 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400f5a0 tx=0x7f4b6400fa60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.563+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64014870 con 0x7f4b7c152d40 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.563+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b640149d0 con 0x7f4b7c152d40 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.563+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b64014d40 con 0x7f4b7c152d40 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.563+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7c153c50 con 0x7f4b7c152d40 2024-01-15T18:02:21.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.563+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7c156550 con 0x7f4b7c152d40 2024-01-15T18:02:21.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.564+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7c0686f0 con 0x7f4b7c152d40 2024-01-15T18:02:21.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.565+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f4b64015410 con 0x7f4b7c152d40 2024-01-15T18:02:21.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.566+0000 7f4b78c00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b5406b7d0 0x7f4b5406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:21.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.566+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4b6408ae40 con 0x7f4b7c152d40 2024-01-15T18:02:21.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.567+0000 7f4b81000640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b5406b7d0 0x7f4b5406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:21.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.568+0000 7f4b81000640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b5406b7d0 0x7f4b5406dc90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c001eb0 tx=0x7f4b6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.571+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b640555c0 con 0x7f4b7c152d40 2024-01-15T18:02:21.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.810+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22401", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b7c0cead0 con 0x7f4b5406b7d0 2024-01-15T18:02:21.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.941+0000 7f4b78c00640 1 -- 172.21.15.2:0/2921670427 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f4b6c005890 con 0x7f4b5406b7d0 2024-01-15T18:02:21.945 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:21.945 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:21.945 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:02:21.945 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:21.945 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:21.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b5406b7d0 msgr2=0x7f4b5406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b5406b7d0 0x7f4b5406dc90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c001eb0 tx=0x7f4b6c009040 comp rx=0 tx=0).stop 2024-01-15T18:02:21.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 msgr2=0x7f4b7c13b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:21.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 0x7f4b7c13b870 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f4b6400f5a0 tx=0x7f4b6400fa60 comp rx=0 tx=0).stop 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 shutdown_connections 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b5406b7d0 0x7f4b5406dc90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b7c1560d0 0x7f4b7c158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b7c152d40 0x7f4b7c13b870 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 --2- 172.21.15.2:0/2921670427 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b5c004d00 0x7f4b7c154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.945+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 >> 172.21.15.2:0/2921670427 conn(0x7f4b7c0b45f0 msgr2=0x7f4b7c143200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.946+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 shutdown_connections 2024-01-15T18:02:21.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:21.946+0000 7f4b83a00640 1 -- 172.21.15.2:0/2921670427 wait complete. 2024-01-15T18:02:21.981 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:02:21.981 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22402 2024-01-15T18:02:22.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.277+0000 7f8ab2c00640 1 Processor -- start 2024-01-15T18:02:22.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.277+0000 7f8ab2c00640 1 -- start start 2024-01-15T18:02:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.279+0000 7f8ab2c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8aac15c450 0x7f8aac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.279+0000 7f8ab2c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8aac06ce30 0x7f8aac070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.279+0000 7f8ab2c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac06d290 con 0x7f8aac144c00 2024-01-15T18:02:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.279+0000 7f8ab2c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0ce230 con 0x7f8aac15c450 2024-01-15T18:02:22.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.279+0000 7f8ab2c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac0ce510 con 0x7f8aac06ce30 2024-01-15T18:02:22.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.281+0000 7f8aa3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8aac15c450 0x7f8aac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.281+0000 7f8aabe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8aac144c00 0x7f8aac0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:51000/0 (socket says 172.21.15.2:51000) 2024-01-15T18:02:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.281+0000 7f8aabe00640 1 -- 172.21.15.2:0/2334886621 learned_addr learned my addr 172.21.15.2:0/2334886621 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.281+0000 7f8ab0c00640 1 --2- 172.21.15.2:0/2334886621 >> v2:172.21.15.137:3300/0 conn(0x7f8aac06ce30 0x7f8aac070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:22.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.282+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2334886621 >> v2:172.21.15.137:3300/0 conn(0x7f8aac06ce30 msgr2=0x7f8aac070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:22.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.282+0000 7f8aa3e00640 1 --2- 172.21.15.2:0/2334886621 >> v2:172.21.15.137:3300/0 conn(0x7f8aac06ce30 0x7f8aac070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:22.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.282+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2334886621 >> v1:172.21.15.137:6789/0 conn(0x7f8aac144c00 legacy=0x7f8aac0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:22.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.282+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2334886621 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac0ce7f0 con 0x7f8aac15c450 2024-01-15T18:02:22.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.284+0000 7f8aa3e00640 1 --2- 172.21.15.2:0/2334886621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac15c450 0x7f8aac06c8f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8aac15ec50 tx=0x7f8a9c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b421107f7d02c950 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.285+0000 7f8aab400640 1 -- 172.21.15.2:0/2334886621 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c031830 con 0x7f8aac15c450 2024-01-15T18:02:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.285+0000 7f8aab400640 1 -- 172.21.15.2:0/2334886621 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c031990 con 0x7f8aac15c450 2024-01-15T18:02:22.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.285+0000 7f8aab400640 1 -- 172.21.15.2:0/2334886621 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c031d20 con 0x7f8aac15c450 2024-01-15T18:02:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.287+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2334886621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac15c450 msgr2=0x7f8aac06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.287+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2334886621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac15c450 0x7f8aac06c8f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f8aac15ec50 tx=0x7f8a9c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:02:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.288+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2334886621 shutdown_connections 2024-01-15T18:02:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.288+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2334886621 >> v2:172.21.15.137:3300/0 conn(0x7f8aac06ce30 0x7f8aac070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.288+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2334886621 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac15c450 0x7f8aac06c8f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:22.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.288+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2334886621 >> 172.21.15.2:0/2334886621 conn(0x7f8aac0b45f0 msgr2=0x7f8aac0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:22.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.288+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2334886621 shutdown_connections 2024-01-15T18:02:22.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.288+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2334886621 wait complete. 2024-01-15T18:02:22.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.292+0000 7f8ab2c00640 1 Processor -- start 2024-01-15T18:02:22.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.292+0000 7f8ab2c00640 1 -- start start 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.294+0000 7f8ab2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aac06ce30 0x7f8aac0ae940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.294+0000 7f8ab2c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 0x7f8aac139be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.294+0000 7f8ab2c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aac15c450 0x7f8aac0b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8aa3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 0x7f8aac139be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8aabe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aac06ce30 0x7f8aac0ae940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8aa3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 0x7f8aac139be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45350/0 (socket says 172.21.15.2:45350) 2024-01-15T18:02:22.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8ab2c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac13d5f0 con 0x7f8aac144c00 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8ab2c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac13f1d0 con 0x7f8aac15c450 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8ab2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac13c700 con 0x7f8aac06ce30 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.295+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2133577003 learned_addr learned my addr 172.21.15.2:0/2133577003 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.296+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aac06ce30 msgr2=0x7f8aac0ae940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.296+0000 7f8aa3e00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aac06ce30 0x7f8aac0ae940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.296+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aac15c450 msgr2=0x7f8aac0b09e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.296+0000 7f8aa3e00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aac15c450 0x7f8aac0b09e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:22.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.296+0000 7f8aa3e00640 1 -- 172.21.15.2:0/2133577003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c009400 con 0x7f8aac144c00 2024-01-15T18:02:22.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.297+0000 7f8aa3e00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 0x7f8aac139be0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c000c00 tx=0x7f8a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:22.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.298+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c03aa60 con 0x7f8aac144c00 2024-01-15T18:02:22.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.298+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c03abe0 con 0x7f8aac144c00 2024-01-15T18:02:22.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.298+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c0443f0 con 0x7f8aac144c00 2024-01-15T18:02:22.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.298+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac13c9e0 con 0x7f8aac144c00 2024-01-15T18:02:22.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.298+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac13cfa0 con 0x7f8aac144c00 2024-01-15T18:02:22.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.301+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac13d760 con 0x7f8aac144c00 2024-01-15T18:02:22.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.302+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f8a9c04d450 con 0x7f8aac144c00 2024-01-15T18:02:22.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.304+0000 7f8aa8c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8a8406b7d0 0x7f8a8406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:22.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.304+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8a9c0ac580 con 0x7f8aac144c00 2024-01-15T18:02:22.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.311+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a9c076cc0 con 0x7f8aac144c00 2024-01-15T18:02:22.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.314+0000 7f8aabe00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8a8406b7d0 0x7f8a8406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:22.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.334+0000 7f8aabe00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8a8406b7d0 0x7f8a8406dc90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8a94005880 tx=0x7f8a94009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:22.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:22.791+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22402", "target": ["mon-mgr", ""]}) v1 -- 0x7f8aac0cead0 con 0x7f8a8406b7d0 2024-01-15T18:02:23.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.090+0000 7f8aa8c00640 1 -- 172.21.15.2:0/2133577003 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f8a94001d10 con 0x7f8a8406b7d0 2024-01-15T18:02:23.095 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:23.095 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:23.096 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:02:23.096 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:23.096 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:23.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8a8406b7d0 msgr2=0x7f8a8406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8a8406b7d0 0x7f8a8406dc90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8a94005880 tx=0x7f8a94009040 comp rx=0 tx=0).stop 2024-01-15T18:02:23.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 msgr2=0x7f8aac139be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 0x7f8aac139be0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c000c00 tx=0x7f8a9c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 shutdown_connections 2024-01-15T18:02:23.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8a8406b7d0 0x7f8a8406dc90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8aac15c450 0x7f8aac0b09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8aac144c00 0x7f8aac139be0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 --2- 172.21.15.2:0/2133577003 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8aac06ce30 0x7f8aac0ae940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 >> 172.21.15.2:0/2133577003 conn(0x7f8aac0b45f0 msgr2=0x7f8aac142520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:23.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 shutdown_connections 2024-01-15T18:02:23.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.097+0000 7f8ab2c00640 1 -- 172.21.15.2:0/2133577003 wait complete. 2024-01-15T18:02:23.143 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:02:23.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22403 2024-01-15T18:02:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.431+0000 7f3d21600640 1 Processor -- start 2024-01-15T18:02:23.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.431+0000 7f3d21600640 1 -- start start 2024-01-15T18:02:23.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d21600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3d1c15c450 0x7f3d1c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d21600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 0x7f3d1c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d21600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d1c0ce230 con 0x7f3d1c144c00 2024-01-15T18:02:23.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d21600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d1c06d2c0 con 0x7f3d1c15c450 2024-01-15T18:02:23.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d21600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0ce510 con 0x7f3d1c06ce30 2024-01-15T18:02:23.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d1aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3d1c144c00 0x7f3d1c0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54014/0 (socket says 172.21.15.2:54014) 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d1aa00640 1 -- 172.21.15.2:0/2954367155 learned_addr learned my addr 172.21.15.2:0/2954367155 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d1b400640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 0x7f3d1c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.433+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699140272 0 0) 0x7f3d1c0ce230 con 0x7f3d1c144c00 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cfc003520 con 0x7f3d1c144c00 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863437012 0 0) 0x7f3cfc003520 con 0x7f3d1c144c00 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 msgr2=0x7f3d1c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 0x7f3d1c070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 >> v2:172.21.15.137:3300/0 conn(0x7f3d1c15c450 msgr2=0x7f3d1c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 --2- 172.21.15.2:0/2954367155 >> v2:172.21.15.137:3300/0 conn(0x7f3d1c15c450 0x7f3d1c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0ce7f0 con 0x7f3d1c144c00 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3d0c001670 con 0x7f3d1c144c00 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 >> v1:172.21.15.2:6789/0 conn(0x7f3d1c144c00 legacy=0x7f3d1c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 0x7f3cfc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc003fa0 0x7f3cfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc004d00 0x7f3cfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c06d2c0 con 0x7f3cfc003fa0 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0ce510 con 0x7f3cfc004d00 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.434+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc00bb90 con 0x7f3d1c06ce30 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d13e00640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc003fa0 0x7f3cfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:23.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d1aa00640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 0x7f3cfc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:23.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d13e00640 1 -- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 msgr2=0x7f3cfc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:23.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d13e00640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d1c06ce30 0x7f3cfc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d13e00640 1 -- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc004d00 msgr2=0x7f3cfc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:23.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d13e00640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cfc004d00 0x7f3cfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.435+0000 7f3d13e00640 1 -- 172.21.15.2:0/2954367155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c0ce7f0 con 0x7f3cfc003fa0 2024-01-15T18:02:23.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.436+0000 7f3d13e00640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc003fa0 0x7f3cfc0047c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009860 tx=0x7f3cfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b895669bdc83 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:23.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.437+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d04032600 con 0x7f3cfc003fa0 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.437+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d0402e050 con 0x7f3cfc003fa0 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.437+0000 7f3d1a000640 1 -- 172.21.15.2:0/2954367155 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d040329c0 con 0x7f3cfc003fa0 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.437+0000 7f3d21600640 1 -- 172.21.15.2:0/2954367155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc003fa0 msgr2=0x7f3cfc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.437+0000 7f3d21600640 1 --2- 172.21.15.2:0/2954367155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cfc003fa0 0x7f3cfc0047c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009860 tx=0x7f3cfc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.437+0000 7f3d1b400640 1 -- 172.21.15.2:0/2954367155 reap_dead start 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.438+0000 7f3d21600640 1 -- 172.21.15.2:0/2954367155 shutdown_connections 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.438+0000 7f3d21600640 1 -- 172.21.15.2:0/2954367155 >> 172.21.15.2:0/2954367155 conn(0x7f3d1c0b45f0 msgr2=0x7f3d1c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.438+0000 7f3d21600640 1 -- 172.21.15.2:0/2954367155 shutdown_connections 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.438+0000 7f3d21600640 1 -- 172.21.15.2:0/2954367155 wait complete. 2024-01-15T18:02:23.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.440+0000 7f3d21600640 1 Processor -- start 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.440+0000 7f3d21600640 1 -- start start 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d21600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cfc003fa0 0x7f3d1c0b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d21600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d1c0af890 0x7f3d1c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d21600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 0x7f3d1c1f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d21600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c159690 con 0x7f3d1c159210 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d21600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0aeb60 con 0x7f3d1c0af890 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d21600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1c0af400 con 0x7f3cfc003fa0 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 0x7f3d1c1f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 0x7f3d1c1f25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34624/0 (socket says 172.21.15.2:34624) 2024-01-15T18:02:23.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 -- 172.21.15.2:0/29586716 learned_addr learned my addr 172.21.15.2:0/29586716 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 -- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cfc003fa0 msgr2=0x7f3d1c0b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cfc003fa0 0x7f3d1c0b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 -- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d1c0af890 msgr2=0x7f3d1c158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d1c0af890 0x7f3d1c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.441+0000 7f3d1b400640 1 -- 172.21.15.2:0/29586716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c138c00 con 0x7f3d1c159210 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.442+0000 7f3d1b400640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 0x7f3d1c1f25b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3d14007cc0 tx=0x7f3d1400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.442+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1400edc0 con 0x7f3d1c159210 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.442+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d14014420 con 0x7f3d1c159210 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.442+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d140147d0 con 0x7f3d1c159210 2024-01-15T18:02:23.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.442+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c1598b0 con 0x7f3d1c159210 2024-01-15T18:02:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.442+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c0af090 con 0x7f3d1c159210 2024-01-15T18:02:23.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.444+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d1c0686f0 con 0x7f3d1c159210 2024-01-15T18:02:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.445+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f3d14014930 con 0x7f3d1c159210 2024-01-15T18:02:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.446+0000 7f3d12a00640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cf406b7d0 0x7f3cf406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:23.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.446+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3d1408a910 con 0x7f3d1c159210 2024-01-15T18:02:23.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.452+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d14055120 con 0x7f3d1c159210 2024-01-15T18:02:23.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.465+0000 7f3d1aa00640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cf406b7d0 0x7f3cf406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:23.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.468+0000 7f3d1aa00640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cf406b7d0 0x7f3cf406dc90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c001050 tx=0x7f3d0c000f70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:23.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.854+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22403", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d1c0cead0 con 0x7f3cf406b7d0 2024-01-15T18:02:23.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.930+0000 7f3d12a00640 1 -- 172.21.15.2:0/29586716 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f3d0c00e8b0 con 0x7f3cf406b7d0 2024-01-15T18:02:23.934 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:02:23.935 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:23.936 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cf406b7d0 msgr2=0x7f3cf406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cf406b7d0 0x7f3cf406dc90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c001050 tx=0x7f3d0c000f70 comp rx=0 tx=0).stop 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 msgr2=0x7f3d1c1f25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 0x7f3d1c1f25b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3d14007cc0 tx=0x7f3d1400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 shutdown_connections 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cf406b7d0 0x7f3cf406dc90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d1c159210 0x7f3d1c1f25b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d1c0af890 0x7f3d1c158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.932+0000 7f3d21600640 1 --2- 172.21.15.2:0/29586716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cfc003fa0 0x7f3d1c0b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:23.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.933+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 >> 172.21.15.2:0/29586716 conn(0x7f3d1c0b45f0 msgr2=0x7f3d1c070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:23.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.933+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 shutdown_connections 2024-01-15T18:02:23.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:23.933+0000 7f3d21600640 1 -- 172.21.15.2:0/29586716 wait complete. 2024-01-15T18:02:23.969 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:02:24.970 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22403 2024-01-15T18:02:25.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.270+0000 7f4003400640 1 Processor -- start 2024-01-15T18:02:25.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.271+0000 7f4003400640 1 -- start start 2024-01-15T18:02:25.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4003400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 0x7f3ffc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:25.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4003400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3ffc06ce30 0x7f3ffc070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:25.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4003400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc0ce230 con 0x7f3ffc144c00 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4003400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ffc06d2c0 con 0x7f3ffc06ce30 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4003400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc0ce510 con 0x7f3ffc15c450 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f3ff3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 0x7f3ffc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4001400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3ffc06ce30 0x7f3ffc070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4000a00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f3ffc144c00 0x7f3ffc0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:56150/0 (socket says 172.21.15.2:56150) 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f4000a00640 1 -- 172.21.15.2:0/1500067508 learned_addr learned my addr 172.21.15.2:0/1500067508 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.273+0000 7f3ff3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 0x7f3ffc06c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:43094/0 (socket says 172.21.15.2:43094) 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.274+0000 7f3ffbe00640 1 -- 172.21.15.2:0/1500067508 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919000389 0 0) 0x7f3ffc0ce230 con 0x7f3ffc144c00 2024-01-15T18:02:25.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.274+0000 7f3ffbe00640 1 -- 172.21.15.2:0/1500067508 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fdc003520 con 0x7f3ffc144c00 2024-01-15T18:02:25.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.274+0000 7f3ff3e00640 1 -- 172.21.15.2:0/1500067508 >> v2:172.21.15.137:3300/0 conn(0x7f3ffc06ce30 msgr2=0x7f3ffc070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:25.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.274+0000 7f3ff3e00640 1 --2- 172.21.15.2:0/1500067508 >> v2:172.21.15.137:3300/0 conn(0x7f3ffc06ce30 0x7f3ffc070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:25.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.274+0000 7f3ff3e00640 1 -- 172.21.15.2:0/1500067508 >> v1:172.21.15.137:6789/0 conn(0x7f3ffc144c00 legacy=0x7f3ffc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:25.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.274+0000 7f3ff3e00640 1 -- 172.21.15.2:0/1500067508 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc0ce7f0 con 0x7f3ffc15c450 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.275+0000 7f3ff3e00640 1 --2- 172.21.15.2:0/1500067508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 0x7f3ffc06c8f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40097a0 tx=0x7f3fe402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=251da207647b1464 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.275+0000 7f3ffbe00640 1 -- 172.21.15.2:0/1500067508 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe402d040 con 0x7f3ffc15c450 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.275+0000 7f3ffbe00640 1 -- 172.21.15.2:0/1500067508 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe4031810 con 0x7f3ffc15c450 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.275+0000 7f3ffbe00640 1 -- 172.21.15.2:0/1500067508 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fe4031ba0 con 0x7f3ffc15c450 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 -- 172.21.15.2:0/1500067508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 msgr2=0x7f3ffc06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 --2- 172.21.15.2:0/1500067508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 0x7f3ffc06c8f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3fe40097a0 tx=0x7f3fe402b8b0 comp rx=0 tx=0).stop 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 -- 172.21.15.2:0/1500067508 shutdown_connections 2024-01-15T18:02:25.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 --2- 172.21.15.2:0/1500067508 >> v2:172.21.15.137:3300/0 conn(0x7f3ffc06ce30 0x7f3ffc070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:25.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 --2- 172.21.15.2:0/1500067508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc15c450 0x7f3ffc06c8f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:25.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 -- 172.21.15.2:0/1500067508 >> 172.21.15.2:0/1500067508 conn(0x7f3ffc0b45f0 msgr2=0x7f3ffc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:25.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 -- 172.21.15.2:0/1500067508 shutdown_connections 2024-01-15T18:02:25.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.276+0000 7f4003400640 1 -- 172.21.15.2:0/1500067508 wait complete. 2024-01-15T18:02:25.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.279+0000 7f4003400640 1 Processor -- start 2024-01-15T18:02:25.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.279+0000 7f4003400640 1 -- start start 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4003400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ffc06ce30 0x7f3ffc13bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4003400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc144c00 0x7f3ffc138680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4003400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 0x7f3ffc0b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4003400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc0afb70 con 0x7f3ffc15c450 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4003400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc0afd50 con 0x7f3ffc06ce30 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4003400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc0b0030 con 0x7f3ffc144c00 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4001400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 0x7f3ffc0b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:25.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4000a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ffc06ce30 0x7f3ffc13bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:25.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f3ff3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc144c00 0x7f3ffc138680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:25.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4001400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 0x7f3ffc0b0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34626/0 (socket says 172.21.15.2:34626) 2024-01-15T18:02:25.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.280+0000 7f4001400640 1 -- 172.21.15.2:0/2497542060 learned_addr learned my addr 172.21.15.2:0/2497542060 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:25.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f4001400640 1 -- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc144c00 msgr2=0x7f3ffc138680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:25.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f4001400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc144c00 0x7f3ffc138680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f4001400640 1 -- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ffc06ce30 msgr2=0x7f3ffc13bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f4001400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ffc06ce30 0x7f3ffc13bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f4001400640 1 -- 172.21.15.2:0/2497542060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe4009400 con 0x7f3ffc15c450 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f4001400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 0x7f3ffc0b0df0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400efc0 tx=0x7f3ff400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.281+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4014480 con 0x7f3ffc15c450 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.282+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff400f040 con 0x7f3ffc15c450 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.282+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff4014820 con 0x7f3ffc15c450 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.282+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc0b0310 con 0x7f3ffc15c450 2024-01-15T18:02:25.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.282+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc0b08d0 con 0x7f3ffc15c450 2024-01-15T18:02:25.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.283+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ffc0686f0 con 0x7f3ffc15c450 2024-01-15T18:02:25.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.284+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f3ff4014980 con 0x7f3ffc15c450 2024-01-15T18:02:25.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.285+0000 7f3ff9600640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fd406b7d0 0x7f3fd406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:25.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.285+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ff408af60 con 0x7f3ffc15c450 2024-01-15T18:02:25.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.285+0000 7f4000a00640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fd406b7d0 0x7f3fd406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:25.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.286+0000 7f4000a00640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fd406b7d0 0x7f3fd406dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0011e0 tx=0x7f3fec007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:25.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.290+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ff4055710 con 0x7f3ffc15c450 2024-01-15T18:02:25.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:25.532+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22403", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ffc0cead0 con 0x7f3fd406b7d0 2024-01-15T18:02:26.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.390+0000 7f3ff9600640 1 -- 172.21.15.2:0/2497542060 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f3fec006b20 con 0x7f3fd406b7d0 2024-01-15T18:02:26.393 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:26.394 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:26.394 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:02:26.394 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:26.394 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fd406b7d0 msgr2=0x7f3fd406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fd406b7d0 0x7f3fd406dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3fec0011e0 tx=0x7f3fec007d30 comp rx=0 tx=0).stop 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 msgr2=0x7f3ffc0b0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 0x7f3ffc0b0df0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400efc0 tx=0x7f3ff400c490 comp rx=0 tx=0).stop 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 shutdown_connections 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3fd406b7d0 0x7f3fd406dc90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3ffc15c450 0x7f3ffc0b0df0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3ffc144c00 0x7f3ffc138680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 --2- 172.21.15.2:0/2497542060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3ffc06ce30 0x7f3ffc13bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.392+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 >> 172.21.15.2:0/2497542060 conn(0x7f3ffc0b45f0 msgr2=0x7f3ffc071940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.393+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 shutdown_connections 2024-01-15T18:02:26.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.393+0000 7f4003400640 1 -- 172.21.15.2:0/2497542060 wait complete. 2024-01-15T18:02:26.418 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:02:26.418 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22404 2024-01-15T18:02:26.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.634+0000 7f492b200640 1 Processor -- start 2024-01-15T18:02:26.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.636+0000 7f492b200640 1 -- start start 2024-01-15T18:02:26.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.636+0000 7f492b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f492c071230 0x7f492c0716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:26.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.636+0000 7f492b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0b2fa0 con 0x7f492c0cc3b0 2024-01-15T18:02:26.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.636+0000 7f492b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492c0b3280 con 0x7f492c143620 2024-01-15T18:02:26.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.636+0000 7f492b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492c071bf0 con 0x7f492c071230 2024-01-15T18:02:26.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.637+0000 7f492a800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f492c0cc3b0 0x7f492c0cc820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54028/0 (socket says 172.21.15.2:54028) 2024-01-15T18:02:26.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.637+0000 7f492a800640 1 -- 172.21.15.2:0/2315130376 learned_addr learned my addr 172.21.15.2:0/2315130376 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:26.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.637+0000 7f4923e00640 1 --2- 172.21.15.2:0/2315130376 >> v2:172.21.15.137:3300/0 conn(0x7f492c071230 0x7f492c0716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:26.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.638+0000 7f4929400640 1 -- 172.21.15.2:0/2315130376 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070775704 0 0) 0x7f492c0b2fa0 con 0x7f492c0cc3b0 2024-01-15T18:02:26.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.638+0000 7f4929400640 1 -- 172.21.15.2:0/2315130376 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c005730 con 0x7f492c0cc3b0 2024-01-15T18:02:26.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.638+0000 7f4923e00640 1 -- 172.21.15.2:0/2315130376 >> v1:172.21.15.137:6789/0 conn(0x7f492c143620 legacy=0x7f492c0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.638+0000 7f4923e00640 1 -- 172.21.15.2:0/2315130376 >> v1:172.21.15.2:6789/0 conn(0x7f492c0cc3b0 legacy=0x7f492c0cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.638+0000 7f4923e00640 1 -- 172.21.15.2:0/2315130376 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c0b3560 con 0x7f492c071230 2024-01-15T18:02:26.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.639+0000 7f4923e00640 1 --2- 172.21.15.2:0/2315130376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c0716b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f49140097a0 tx=0x7f491402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae3b23c020665568 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:26.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.640+0000 7f4929400640 1 -- 172.21.15.2:0/2315130376 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914030610 con 0x7f492c071230 2024-01-15T18:02:26.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.640+0000 7f4929400640 1 -- 172.21.15.2:0/2315130376 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4914030770 con 0x7f492c071230 2024-01-15T18:02:26.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.640+0000 7f4929400640 1 -- 172.21.15.2:0/2315130376 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914030b00 con 0x7f492c071230 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 -- 172.21.15.2:0/2315130376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 msgr2=0x7f492c0716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 --2- 172.21.15.2:0/2315130376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c0716b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f49140097a0 tx=0x7f491402b960 comp rx=0 tx=0).stop 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 -- 172.21.15.2:0/2315130376 shutdown_connections 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 --2- 172.21.15.2:0/2315130376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c0716b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 -- 172.21.15.2:0/2315130376 >> 172.21.15.2:0/2315130376 conn(0x7f492c0b0840 msgr2=0x7f492c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 -- 172.21.15.2:0/2315130376 shutdown_connections 2024-01-15T18:02:26.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.641+0000 7f492b200640 1 -- 172.21.15.2:0/2315130376 wait complete. 2024-01-15T18:02:26.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.644+0000 7f492b200640 1 Processor -- start 2024-01-15T18:02:26.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.644+0000 7f492b200640 1 -- start start 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f492b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c13f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f492b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 0x7f492c157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f492b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f492c143620 0x7f492c1e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f492b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c13ccc0 con 0x7f492c0cc3b0 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f492b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c13cfa0 con 0x7f492c071230 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f492b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f492c13d280 con 0x7f492c143620 2024-01-15T18:02:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f4923e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 0x7f492c157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:26.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f4923e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 0x7f492c157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34638/0 (socket says 172.21.15.2:34638) 2024-01-15T18:02:26.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.645+0000 7f4923e00640 1 -- 172.21.15.2:0/2512078659 learned_addr learned my addr 172.21.15.2:0/2512078659 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4929e00640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c13f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4923e00640 1 -- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f492c143620 msgr2=0x7f492c1e4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4923e00640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f492c143620 0x7f492c1e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4923e00640 1 -- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 msgr2=0x7f492c13f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4923e00640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c13f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4923e00640 1 -- 172.21.15.2:0/2512078659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4914009400 con 0x7f492c0cc3b0 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4923e00640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 0x7f492c157950 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f491402bf70 tx=0x7f4914004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.646+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491403e040 con 0x7f492c0cc3b0 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.647+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4914039a40 con 0x7f492c0cc3b0 2024-01-15T18:02:26.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.647+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914039df0 con 0x7f492c0cc3b0 2024-01-15T18:02:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.647+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c13d560 con 0x7f492c0cc3b0 2024-01-15T18:02:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.647+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c13f6b0 con 0x7f492c0cc3b0 2024-01-15T18:02:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.648+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c13da80 con 0x7f492c0cc3b0 2024-01-15T18:02:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.649+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f4914043390 con 0x7f492c0cc3b0 2024-01-15T18:02:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.651+0000 7f4922000640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f490406b7d0 0x7f490406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:26.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.651+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f49140ac660 con 0x7f492c0cc3b0 2024-01-15T18:02:26.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.651+0000 7f4929e00640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f490406b7d0 0x7f490406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:26.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.652+0000 7f4929e00640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f490406b7d0 0x7f490406dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f491c007fd0 tx=0x7f491c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:26.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.656+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f492c13da80 con 0x7f492c0cc3b0 2024-01-15T18:02:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.886+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22404", "target": ["mon-mgr", ""]}) v1 -- 0x7f492c143a00 con 0x7f490406b7d0 2024-01-15T18:02:26.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.910+0000 7f4922000640 1 -- 172.21.15.2:0/2512078659 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f491c001200 con 0x7f490406b7d0 2024-01-15T18:02:26.913 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:26.913 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:26.913 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:26.914 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f490406b7d0 msgr2=0x7f490406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f490406b7d0 0x7f490406dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f491c007fd0 tx=0x7f491c009040 comp rx=0 tx=0).stop 2024-01-15T18:02:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 msgr2=0x7f492c157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 0x7f492c157950 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f491402bf70 tx=0x7f4914004730 comp rx=0 tx=0).stop 2024-01-15T18:02:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 shutdown_connections 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f490406b7d0 0x7f490406dc90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f492c143620 0x7f492c1e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f492c0cc3b0 0x7f492c157950 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 --2- 172.21.15.2:0/2512078659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f492c071230 0x7f492c13f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 >> 172.21.15.2:0/2512078659 conn(0x7f492c0b0840 msgr2=0x7f492c142950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 shutdown_connections 2024-01-15T18:02:26.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:26.912+0000 7f492b200640 1 -- 172.21.15.2:0/2512078659 wait complete. 2024-01-15T18:02:26.936 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:02:27.937 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22404 2024-01-15T18:02:28.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.246+0000 7f2e50400640 1 Processor -- start 2024-01-15T18:02:28.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.247+0000 7f2e50400640 1 -- start start 2024-01-15T18:02:28.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.249+0000 7f2e50400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e48144c00 0x7f2e480cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.249+0000 7f2e50400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e480cdd70 0x7f2e4806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.249+0000 7f2e50400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2e4815c450 0x7f2e4806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.249+0000 7f2e50400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e480ce230 con 0x7f2e48144c00 2024-01-15T18:02:28.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.249+0000 7f2e50400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e480ce510 con 0x7f2e4815c450 2024-01-15T18:02:28.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.249+0000 7f2e50400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e480ce7f0 con 0x7f2e480cdd70 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.250+0000 7f2e4e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2e4815c450 0x7f2e4806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.250+0000 7f2e4da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e48144c00 0x7f2e480cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.250+0000 7f2e4e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2e4815c450 0x7f2e4806ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:34608/0 (socket says 172.21.15.2:34608) 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.250+0000 7f2e4da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e48144c00 0x7f2e480cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34648/0 (socket says 172.21.15.2:34648) 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.250+0000 7f2e4e400640 1 -- 172.21.15.2:0/2483699814 learned_addr learned my addr 172.21.15.2:0/2483699814 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.250+0000 7f2e4d000640 1 --2- 172.21.15.2:0/2483699814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e480cdd70 0x7f2e4806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.251+0000 7f2e4da00640 1 -- 172.21.15.2:0/2483699814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e480cdd70 msgr2=0x7f2e4806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.251+0000 7f2e4da00640 1 --2- 172.21.15.2:0/2483699814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e480cdd70 0x7f2e4806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.251+0000 7f2e4da00640 1 -- 172.21.15.2:0/2483699814 >> v2:172.21.15.137:3300/0 conn(0x7f2e4815c450 msgr2=0x7f2e4806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.251+0000 7f2e4da00640 1 --2- 172.21.15.2:0/2483699814 >> v2:172.21.15.137:3300/0 conn(0x7f2e4815c450 0x7f2e4806ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.251+0000 7f2e4da00640 1 -- 172.21.15.2:0/2483699814 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480705f0 con 0x7f2e48144c00 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.252+0000 7f2e4da00640 1 --2- 172.21.15.2:0/2483699814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e48144c00 0x7f2e480cd830 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2e480cf090 tx=0x7f2e4002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14bf33a4d6b2b3a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.254+0000 7f2e47e00640 1 -- 172.21.15.2:0/2483699814 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4002cbe0 con 0x7f2e48144c00 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.254+0000 7f2e47e00640 1 -- 172.21.15.2:0/2483699814 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4002cd40 con 0x7f2e48144c00 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.254+0000 7f2e47e00640 1 -- 172.21.15.2:0/2483699814 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e40034660 con 0x7f2e48144c00 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.255+0000 7f2e50400640 1 -- 172.21.15.2:0/2483699814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e48144c00 msgr2=0x7f2e480cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.255+0000 7f2e50400640 1 --2- 172.21.15.2:0/2483699814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e48144c00 0x7f2e480cd830 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2e480cf090 tx=0x7f2e4002be30 comp rx=0 tx=0).stop 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.256+0000 7f2e50400640 1 -- 172.21.15.2:0/2483699814 shutdown_connections 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.256+0000 7f2e50400640 1 --2- 172.21.15.2:0/2483699814 >> v2:172.21.15.137:3300/0 conn(0x7f2e4815c450 0x7f2e4806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.256+0000 7f2e50400640 1 --2- 172.21.15.2:0/2483699814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e480cdd70 0x7f2e4806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.256+0000 7f2e50400640 1 --2- 172.21.15.2:0/2483699814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e48144c00 0x7f2e480cd830 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.256+0000 7f2e50400640 1 -- 172.21.15.2:0/2483699814 >> 172.21.15.2:0/2483699814 conn(0x7f2e480b45f0 msgr2=0x7f2e480b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:28.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.257+0000 7f2e50400640 1 -- 172.21.15.2:0/2483699814 shutdown_connections 2024-01-15T18:02:28.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.257+0000 7f2e50400640 1 -- 172.21.15.2:0/2483699814 wait complete. 2024-01-15T18:02:28.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.260+0000 7f2e50400640 1 Processor -- start 2024-01-15T18:02:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.262+0000 7f2e50400640 1 -- start start 2024-01-15T18:02:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.263+0000 7f2e50400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e480cdd70 0x7f2e4816cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.263+0000 7f2e50400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 0x7f2e4813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.263+0000 7f2e50400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e4813cc40 0x7f2e481ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.263+0000 7f2e50400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4816d180 con 0x7f2e4815c450 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.263+0000 7f2e50400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4813a2b0 con 0x7f2e480cdd70 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.263+0000 7f2e50400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4813a590 con 0x7f2e4813cc40 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.264+0000 7f2e4d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 0x7f2e4813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.264+0000 7f2e4d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 0x7f2e4813c700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34650/0 (socket says 172.21.15.2:34650) 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.264+0000 7f2e4d000640 1 -- 172.21.15.2:0/3563925567 learned_addr learned my addr 172.21.15.2:0/3563925567 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.265+0000 7f2e4d000640 1 -- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e4813cc40 msgr2=0x7f2e481ffa60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.265+0000 7f2e4d000640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e4813cc40 0x7f2e481ffa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.265+0000 7f2e4d000640 1 -- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e480cdd70 msgr2=0x7f2e4816cc40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.265+0000 7f2e4d000640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e480cdd70 0x7f2e4816cc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.265+0000 7f2e4d000640 1 -- 172.21.15.2:0/3563925567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e40009a70 con 0x7f2e4815c450 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.265+0000 7f2e4d000640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 0x7f2e4813c700 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800d8f0 tx=0x7f2e3800ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:28.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.266+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800b6c0 con 0x7f2e4815c450 2024-01-15T18:02:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.266+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3800b820 con 0x7f2e4815c450 2024-01-15T18:02:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.266+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3800bb90 con 0x7f2e4815c450 2024-01-15T18:02:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.270+0000 7f2e50400640 1 -- 172.21.15.2:0/3563925567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e4813a870 con 0x7f2e4815c450 2024-01-15T18:02:28.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.271+0000 7f2e50400640 1 -- 172.21.15.2:0/3563925567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e4813ae30 con 0x7f2e4815c450 2024-01-15T18:02:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.273+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e480686f0 con 0x7f2e4815c450 2024-01-15T18:02:28.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.274+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f2e3800bcf0 con 0x7f2e4815c450 2024-01-15T18:02:28.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.276+0000 7f2e45600640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e2008d810 0x7f2e2008fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:28.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.276+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2e3808a280 con 0x7f2e4815c450 2024-01-15T18:02:28.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.277+0000 7f2e4da00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e2008d810 0x7f2e2008fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:28.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.280+0000 7f2e4da00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e2008d810 0x7f2e2008fcd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2e40036000 tx=0x7f2e40030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:28.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.288+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e3808f060 con 0x7f2e4815c450 2024-01-15T18:02:28.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.675+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22404", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e480af090 con 0x7f2e2008d810 2024-01-15T18:02:28.709 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:02:28.710 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:02:28.711 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:28.711 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:28.711 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:28.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.706+0000 7f2e45600640 1 -- 172.21.15.2:0/3563925567 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f2e480af090 con 0x7f2e2008d810 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e2008d810 msgr2=0x7f2e2008fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e2008d810 0x7f2e2008fcd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2e40036000 tx=0x7f2e40030000 comp rx=0 tx=0).stop 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 msgr2=0x7f2e4813c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 0x7f2e4813c700 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800d8f0 tx=0x7f2e3800ddb0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 shutdown_connections 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e2008d810 0x7f2e2008fcd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e4813cc40 0x7f2e481ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e4815c450 0x7f2e4813c700 secure :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2e3800d8f0 tx=0x7f2e3800ddb0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 --2- 172.21.15.2:0/3563925567 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e480cdd70 0x7f2e4816cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.711+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 >> 172.21.15.2:0/3563925567 conn(0x7f2e480b45f0 msgr2=0x7f2e480aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.712+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 shutdown_connections 2024-01-15T18:02:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:28.712+0000 7f2e3ea00640 1 -- 172.21.15.2:0/3563925567 wait complete. 2024-01-15T18:02:28.756 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:02:29.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone22404 2024-01-15T18:02:29.958 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:02:29.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:02:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.982+0000 7f8822e00640 1 Processor -- start 2024-01-15T18:02:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.983+0000 7f8822e00640 1 -- start start 2024-01-15T18:02:29.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.984+0000 7f8822e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f881c1439f0 0x7f881c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.984+0000 7f8822e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f881c0b3420 0x7f881c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.984+0000 7f8822e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c0ccc70 0x7f881c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.984+0000 7f8822e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881c15dd70 con 0x7f881c0b3420 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.984+0000 7f8822e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f881c0b38b0 con 0x7f881c1439f0 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.984+0000 7f8822e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881c0711b0 con 0x7f881c0ccc70 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f881c0b3420 0x7f881c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f881c0b3420 0x7f881c0cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34662/0 (socket says 172.21.15.2:34662) 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 -- 172.21.15.2:0/2977401615 learned_addr learned my addr 172.21.15.2:0/2977401615 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8820e00640 1 --2- 172.21.15.2:0/2977401615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c0ccc70 0x7f881c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f881be00640 1 --2- 172.21.15.2:0/2977401615 >> v2:172.21.15.137:3300/0 conn(0x7f881c1439f0 0x7f881c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 -- 172.21.15.2:0/2977401615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c0ccc70 msgr2=0x7f881c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 --2- 172.21.15.2:0/2977401615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c0ccc70 0x7f881c06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 -- 172.21.15.2:0/2977401615 >> v2:172.21.15.137:3300/0 conn(0x7f881c1439f0 msgr2=0x7f881c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 --2- 172.21.15.2:0/2977401615 >> v2:172.21.15.137:3300/0 conn(0x7f881c1439f0 0x7f881c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.985+0000 7f8813200640 1 -- 172.21.15.2:0/2977401615 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c143dd0 con 0x7f881c0b3420 2024-01-15T18:02:29.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.987+0000 7f8813200640 1 --2- 172.21.15.2:0/2977401615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0b3420 0x7f881c0cc730 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f881c0b28d0 tx=0x7f880402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46911b1aa6d1b424 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:29.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.987+0000 7f881b400640 1 -- 172.21.15.2:0/2977401615 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880402ec80 con 0x7f881c0b3420 2024-01-15T18:02:29.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.987+0000 7f881b400640 1 -- 172.21.15.2:0/2977401615 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880402ede0 con 0x7f881c0b3420 2024-01-15T18:02:29.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.987+0000 7f881b400640 1 -- 172.21.15.2:0/2977401615 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8804036610 con 0x7f881c0b3420 2024-01-15T18:02:29.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 -- 172.21.15.2:0/2977401615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0b3420 msgr2=0x7f881c0cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:29.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 --2- 172.21.15.2:0/2977401615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0b3420 0x7f881c0cc730 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f881c0b28d0 tx=0x7f880402e0e0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 -- 172.21.15.2:0/2977401615 shutdown_connections 2024-01-15T18:02:29.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 --2- 172.21.15.2:0/2977401615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c0ccc70 0x7f881c06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 --2- 172.21.15.2:0/2977401615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0b3420 0x7f881c0cc730 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 --2- 172.21.15.2:0/2977401615 >> v2:172.21.15.137:3300/0 conn(0x7f881c1439f0 0x7f881c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 -- 172.21.15.2:0/2977401615 >> 172.21.15.2:0/2977401615 conn(0x7f881c0b45f0 msgr2=0x7f881c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:29.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 -- 172.21.15.2:0/2977401615 shutdown_connections 2024-01-15T18:02:29.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.988+0000 7f8822e00640 1 -- 172.21.15.2:0/2977401615 wait complete. 2024-01-15T18:02:29.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.990+0000 7f8822e00640 1 Processor -- start 2024-01-15T18:02:29.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.990+0000 7f8822e00640 1 -- start start 2024-01-15T18:02:29.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8822e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f881c0b3420 0x7f881c1381e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8822e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 0x7f881c1f3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8822e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c1439f0 0x7f881c1f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8822e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c1f4130 con 0x7f881c0ccc70 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8822e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c139c70 con 0x7f881c0b3420 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8822e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f881c139f10 con 0x7f881c1439f0 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.991+0000 7f8813200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 0x7f881c1f3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 0x7f881c1f3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34668/0 (socket says 172.21.15.2:34668) 2024-01-15T18:02:29.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 -- 172.21.15.2:0/2731615193 learned_addr learned my addr 172.21.15.2:0/2731615193 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:29.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f881be00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f881c0b3420 0x7f881c1381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:29.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 -- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c1439f0 msgr2=0x7f881c1f0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:29.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c1439f0 0x7f881c1f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 -- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f881c0b3420 msgr2=0x7f881c1381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f881c0b3420 0x7f881c1381e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 -- 172.21.15.2:0/2731615193 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8804009b30 con 0x7f881c0ccc70 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f881be00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f881c0b3420 0x7f881c1381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.992+0000 7f8813200640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 0x7f881c1f3bf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8804000f80 tx=0x7f8804004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.993+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880402ec80 con 0x7f881c0ccc70 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.993+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8804009510 con 0x7f881c0ccc70 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.993+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f880403e610 con 0x7f881c0ccc70 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.993+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881c13a1f0 con 0x7f881c0ccc70 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.993+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881c13a7b0 con 0x7f881c0ccc70 2024-01-15T18:02:29.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.994+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881c0686f0 con 0x7f881c0ccc70 2024-01-15T18:02:29.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.995+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f880403e770 con 0x7f881c0ccc70 2024-01-15T18:02:29.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.996+0000 7f8818c00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87f406b7d0 0x7f87f406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:29.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.996+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f88040acbc0 con 0x7f881c0ccc70 2024-01-15T18:02:29.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.997+0000 7f881be00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87f406b7d0 0x7f87f406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:29.998+0000 7f881be00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87f406b7d0 0x7f87f406dc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f880c006fd0 tx=0x7f880c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:30.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.001+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8804077350 con 0x7f881c0ccc70 2024-01-15T18:02:30.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.237+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone22404", "target": ["mon-mgr", ""]}) v1 -- 0x7f881c070ed0 con 0x7f87f406b7d0 2024-01-15T18:02:30.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.312+0000 7f8818c00640 1 -- 172.21.15.2:0/2731615193 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f880c009530 con 0x7f87f406b7d0 2024-01-15T18:02:30.316 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:02:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:02:30.316 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:30.316 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:30.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87f406b7d0 msgr2=0x7f87f406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87f406b7d0 0x7f87f406dc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f880c006fd0 tx=0x7f880c008040 comp rx=0 tx=0).stop 2024-01-15T18:02:30.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 msgr2=0x7f881c1f3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 0x7f881c1f3bf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8804000f80 tx=0x7f8804004a40 comp rx=0 tx=0).stop 2024-01-15T18:02:30.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 shutdown_connections 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87f406b7d0 0x7f87f406dc90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f881c1439f0 0x7f881c1f0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f881c0ccc70 0x7f881c1f3bf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 --2- 172.21.15.2:0/2731615193 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f881c0b3420 0x7f881c1381e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 >> 172.21.15.2:0/2731615193 conn(0x7f881c0b45f0 msgr2=0x7f881c0b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 shutdown_connections 2024-01-15T18:02:30.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.316+0000 7f8822e00640 1 -- 172.21.15.2:0/2731615193 wait complete. 2024-01-15T18:02:30.339 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:02:30.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.553+0000 7f2d58a00640 1 Processor -- start 2024-01-15T18:02:30.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.553+0000 7f2d58a00640 1 -- start start 2024-01-15T18:02:30.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.554+0000 7f2d58a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2d501439e0 0x7f2d5006c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:30.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.554+0000 7f2d58a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2d5006cdf0 0x7f2d500cb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d58a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d50143dc0 con 0x7f2d500cbed0 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d58a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5006d280 con 0x7f2d5006cdf0 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d58a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d5015b620 con 0x7f2d501439e0 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2d5006cdf0 0x7f2d500cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2d5006cdf0 0x7f2d500cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34678/0 (socket says 172.21.15.2:34678) 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 -- 172.21.15.2:0/1348504120 learned_addr learned my addr 172.21.15.2:0/1348504120 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d56000640 1 --2- 172.21.15.2:0/1348504120 >> v2:172.21.15.137:3300/0 conn(0x7f2d501439e0 0x7f2d5006c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 -- 172.21.15.2:0/1348504120 >> v2:172.21.15.137:3300/0 conn(0x7f2d501439e0 msgr2=0x7f2d5006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 --2- 172.21.15.2:0/1348504120 >> v2:172.21.15.137:3300/0 conn(0x7f2d501439e0 0x7f2d5006c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 -- 172.21.15.2:0/1348504120 >> v1:172.21.15.137:6789/0 conn(0x7f2d500cbed0 legacy=0x7f2d500b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:30.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.555+0000 7f2d55600640 1 -- 172.21.15.2:0/1348504120 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d500cd9f0 con 0x7f2d5006cdf0 2024-01-15T18:02:30.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.557+0000 7f2d55600640 1 --2- 172.21.15.2:0/1348504120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d5006cdf0 0x7f2d500cb990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2d5006d400 tx=0x7f2d3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ffb852f93dc82bf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:30.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.557+0000 7f2d54c00640 1 -- 172.21.15.2:0/1348504120 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38030860 con 0x7f2d5006cdf0 2024-01-15T18:02:30.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.557+0000 7f2d54c00640 1 -- 172.21.15.2:0/1348504120 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d380309c0 con 0x7f2d5006cdf0 2024-01-15T18:02:30.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.557+0000 7f2d54c00640 1 -- 172.21.15.2:0/1348504120 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38030d50 con 0x7f2d5006cdf0 2024-01-15T18:02:30.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 -- 172.21.15.2:0/1348504120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d5006cdf0 msgr2=0x7f2d500cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 --2- 172.21.15.2:0/1348504120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d5006cdf0 0x7f2d500cb990 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f2d5006d400 tx=0x7f2d3802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 -- 172.21.15.2:0/1348504120 shutdown_connections 2024-01-15T18:02:30.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 --2- 172.21.15.2:0/1348504120 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d5006cdf0 0x7f2d500cb990 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 --2- 172.21.15.2:0/1348504120 >> v2:172.21.15.137:3300/0 conn(0x7f2d501439e0 0x7f2d5006c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 -- 172.21.15.2:0/1348504120 >> 172.21.15.2:0/1348504120 conn(0x7f2d500b46d0 msgr2=0x7f2d500b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:30.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 -- 172.21.15.2:0/1348504120 shutdown_connections 2024-01-15T18:02:30.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.558+0000 7f2d58a00640 1 -- 172.21.15.2:0/1348504120 wait complete. 2024-01-15T18:02:30.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.560+0000 7f2d58a00640 1 Processor -- start 2024-01-15T18:02:30.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.560+0000 7f2d58a00640 1 -- start start 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d58a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d5006cdf0 0x7f2d5013a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d58a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d500cbed0 0x7f2d500a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d58a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 0x7f2d500a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d58a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5013bdc0 con 0x7f2d501439e0 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d58a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5013ad40 con 0x7f2d5006cdf0 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d58a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5013b020 con 0x7f2d500cbed0 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d56a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 0x7f2d500a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d56a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 0x7f2d500a89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34688/0 (socket says 172.21.15.2:34688) 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.561+0000 7f2d56a00640 1 -- 172.21.15.2:0/683596506 learned_addr learned my addr 172.21.15.2:0/683596506 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d55600640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d500cbed0 0x7f2d500a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56000640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d5006cdf0 0x7f2d5013a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56a00640 1 -- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d500cbed0 msgr2=0x7f2d500a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d500cbed0 0x7f2d500a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56a00640 1 -- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d5006cdf0 msgr2=0x7f2d5013a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d5006cdf0 0x7f2d5013a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56a00640 1 -- 172.21.15.2:0/683596506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d380094d0 con 0x7f2d501439e0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d56a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 0x7f2d500a89a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800c7a0 tx=0x7f2d4800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.562+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003b70 con 0x7f2d501439e0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.563+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d48003cf0 con 0x7f2d501439e0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.563+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003ea0 con 0x7f2d501439e0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.563+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5013b300 con 0x7f2d501439e0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.563+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5013b8c0 con 0x7f2d501439e0 2024-01-15T18:02:30.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.564+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d5013ba50 con 0x7f2d501439e0 2024-01-15T18:02:30.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.565+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f2d4800aa90 con 0x7f2d501439e0 2024-01-15T18:02:30.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.566+0000 7f2d4e000640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2d2806b7d0 0x7f2d2806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:30.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.566+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2d4808a2e0 con 0x7f2d501439e0 2024-01-15T18:02:30.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.567+0000 7f2d56000640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2d2806b7d0 0x7f2d2806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:30.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.568+0000 7f2d56000640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2d2806b7d0 0x7f2d2806dc90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2d40006fd0 tx=0x7f2d40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:30.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.571+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d48054a60 con 0x7f2d501439e0 2024-01-15T18:02:30.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.794+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d500cc2d0 con 0x7f2d2806b7d0 2024-01-15T18:02:30.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.798+0000 7f2d4e000640 1 -- 172.21.15.2:0/683596506 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2d4000f860 con 0x7f2d2806b7d0 2024-01-15T18:02:30.801 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:02:30.801 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:30.802 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:02:30.802 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:30.802 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:02:30.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2d2806b7d0 msgr2=0x7f2d2806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2d2806b7d0 0x7f2d2806dc90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2d40006fd0 tx=0x7f2d40008040 comp rx=0 tx=0).stop 2024-01-15T18:02:30.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 msgr2=0x7f2d500a89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:30.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 0x7f2d500a89a0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800c7a0 tx=0x7f2d4800cc60 comp rx=0 tx=0).stop 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 shutdown_connections 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2d2806b7d0 0x7f2d2806dc90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2d501439e0 0x7f2d500a89a0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2d500cbed0 0x7f2d500a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 --2- 172.21.15.2:0/683596506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2d5006cdf0 0x7f2d5013a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 >> 172.21.15.2:0/683596506 conn(0x7f2d500b46d0 msgr2=0x7f2d500b1490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 shutdown_connections 2024-01-15T18:02:30.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:30.801+0000 7f2d58a00640 1 -- 172.21.15.2:0/683596506 wait complete. 2024-01-15T18:02:30.822 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:02:31.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.042+0000 7f8309400640 1 Processor -- start 2024-01-15T18:02:31.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.042+0000 7f8309400640 1 -- start start 2024-01-15T18:02:31.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.044+0000 7f8309400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 0x7f8304165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.044+0000 7f8309400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040ce230 con 0x7f830415c450 2024-01-15T18:02:31.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.044+0000 7f8309400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83040ce510 con 0x7f8304144c00 2024-01-15T18:02:31.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.044+0000 7f8309400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83040ce7f0 con 0x7f830406d000 2024-01-15T18:02:31.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.045+0000 7f82fbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f830415c450 0x7f830406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54032/0 (socket says 172.21.15.2:54032) 2024-01-15T18:02:31.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.045+0000 7f82fbe00640 1 -- 172.21.15.2:0/877423313 learned_addr learned my addr 172.21.15.2:0/877423313 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.045+0000 7f8303200640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 0x7f8304165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.045+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321720058 0 0) 0x7f83040ce230 con 0x7f830415c450 2024-01-15T18:02:31.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.045+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82e4003520 con 0x7f830415c450 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1822479593 0 0) 0x7f83040ce510 con 0x7f8304144c00 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83040ce230 con 0x7f8304144c00 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 590244490 0 0) 0x7f82e4003520 con 0x7f830415c450 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 msgr2=0x7f8304165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 0x7f8304165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 >> v1:172.21.15.137:6789/0 conn(0x7f8304144c00 legacy=0x7f83040cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040705f0 con 0x7f830415c450 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f82ec001670 con 0x7f830415c450 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 >> v1:172.21.15.2:6789/0 conn(0x7f830415c450 legacy=0x7f830406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e4003b30 0x7f82e4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 0x7f82e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82e4004d00 0x7f82e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83040ce7f0 con 0x7f82e4004d00 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e400ba60 con 0x7f82e4003b30 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e400bc20 con 0x7f830406d000 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8303200640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82e4004d00 0x7f82e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.046+0000 7f8302800640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e4003b30 0x7f82e4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.047+0000 7f82fbe00640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 0x7f82e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.047+0000 7f8303200640 1 -- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 msgr2=0x7f82e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.047+0000 7f8303200640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f830406d000 0x7f82e40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.047+0000 7f8303200640 1 -- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e4003b30 msgr2=0x7f82e4003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:31.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.047+0000 7f8303200640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f82e4003b30 0x7f82e4003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.047+0000 7f8303200640 1 -- 172.21.15.2:0/877423313 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e4003520 con 0x7f82e4004d00 2024-01-15T18:02:31.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.048+0000 7f8303200640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82e4004d00 0x7f82e4005180 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009fc0 tx=0x7f82e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d2169bbc4ec044 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.048+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc02cc20 con 0x7f82e4004d00 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.048+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00f040 con 0x7f82e4004d00 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.048+0000 7f8301e00640 1 -- 172.21.15.2:0/877423313 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc034440 con 0x7f82e4004d00 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8309400640 1 -- 172.21.15.2:0/877423313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82e4004d00 msgr2=0x7f82e4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8309400640 1 --2- 172.21.15.2:0/877423313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82e4004d00 0x7f82e4005180 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f82fc009fc0 tx=0x7f82e40057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8303200640 1 -- 172.21.15.2:0/877423313 reap_dead start 2024-01-15T18:02:31.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8309400640 1 -- 172.21.15.2:0/877423313 shutdown_connections 2024-01-15T18:02:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8309400640 1 -- 172.21.15.2:0/877423313 >> 172.21.15.2:0/877423313 conn(0x7f83040b45f0 msgr2=0x7f83040b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8309400640 1 -- 172.21.15.2:0/877423313 shutdown_connections 2024-01-15T18:02:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.049+0000 7f8309400640 1 -- 172.21.15.2:0/877423313 wait complete. 2024-01-15T18:02:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.051+0000 7f8309400640 1 Processor -- start 2024-01-15T18:02:31.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.051+0000 7f8309400640 1 -- start start 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8309400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8304158b90 0x7f830413e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8309400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f830415d980 0x7f830415de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8309400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 0x7f83040accf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8309400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830413b140 con 0x7f83040ac870 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8309400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8304139ea0 con 0x7f830415d980 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8309400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830415e340 con 0x7f8304158b90 2024-01-15T18:02:31.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8303200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 0x7f83040accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8303200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 0x7f83040accf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34712/0 (socket says 172.21.15.2:34712) 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.052+0000 7f8303200640 1 -- 172.21.15.2:0/3397467266 learned_addr learned my addr 172.21.15.2:0/3397467266 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8302800640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8304158b90 0x7f830413e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f82fbe00640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f830415d980 0x7f830415de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8303200640 1 -- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8304158b90 msgr2=0x7f830413e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8303200640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8304158b90 0x7f830413e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8303200640 1 -- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f830415d980 msgr2=0x7f830415de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8303200640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f830415d980 0x7f830415de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8303200640 1 -- 172.21.15.2:0/3397467266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83040b2120 con 0x7f83040ac870 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8302800640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8304158b90 0x7f830413e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f82fbe00640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f830415d980 0x7f830415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:31.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.053+0000 7f8303200640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 0x7f83040accf0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f82fc002230 tx=0x7f82fc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.054+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc034ad0 con 0x7f83040ac870 2024-01-15T18:02:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.054+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc034c30 con 0x7f83040ac870 2024-01-15T18:02:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.054+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc003990 con 0x7f83040ac870 2024-01-15T18:02:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.054+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83041f7320 con 0x7f83040ac870 2024-01-15T18:02:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.054+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f830413a390 con 0x7f83040ac870 2024-01-15T18:02:31.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.055+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83040686f0 con 0x7f83040ac870 2024-01-15T18:02:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.056+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f82fc03e440 con 0x7f83040ac870 2024-01-15T18:02:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.057+0000 7f82faa00640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f82dc06b800 0x7f82dc06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.057+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f82fc0ac320 con 0x7f83040ac870 2024-01-15T18:02:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.058+0000 7f8302800640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f82dc06b800 0x7f82dc06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.059+0000 7f8302800640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f82dc06b800 0x7f82dc06dcc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f82f40035f0 tx=0x7f82f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:31.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.062+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82fc076b70 con 0x7f83040ac870 2024-01-15T18:02:31.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.289+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83040cead0 con 0x7f82dc06b800 2024-01-15T18:02:31.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.303+0000 7f82faa00640 1 -- 172.21.15.2:0/3397467266 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+234 (secure 0 0 0) 0x7f82f4001650 con 0x7f82dc06b800 2024-01-15T18:02:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:31.306 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:02:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:02:31.306 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone22404" 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone22403" 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone22401" 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone22402" 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:31.307 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:02:31.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.305+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f82dc06b800 msgr2=0x7f82dc06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.305+0000 7f8309400640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f82dc06b800 0x7f82dc06dcc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f82f40035f0 tx=0x7f82f4009040 comp rx=0 tx=0).stop 2024-01-15T18:02:31.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.305+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 msgr2=0x7f83040accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.305+0000 7f8309400640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 0x7f83040accf0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f82fc002230 tx=0x7f82fc009750 comp rx=0 tx=0).stop 2024-01-15T18:02:31.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 shutdown_connections 2024-01-15T18:02:31.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f82dc06b800 0x7f82dc06dcc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83040ac870 0x7f83040accf0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f830415d980 0x7f830415de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 --2- 172.21.15.2:0/3397467266 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8304158b90 0x7f830413e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 >> 172.21.15.2:0/3397467266 conn(0x7f83040b45f0 msgr2=0x7f83041421f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:31.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 shutdown_connections 2024-01-15T18:02:31.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.306+0000 7f8309400640 1 -- 172.21.15.2:0/3397467266 wait complete. 2024-01-15T18:02:31.330 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:02:31.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.557+0000 7f036be00640 1 Processor -- start 2024-01-15T18:02:31.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.557+0000 7f036be00640 1 -- start start 2024-01-15T18:02:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.558+0000 7f036be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f036c0720d0 0x7f036c06c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.558+0000 7f036be00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c071090 con 0x7f036c15f260 2024-01-15T18:02:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.559+0000 7f036be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036c144560 con 0x7f036c143d20 2024-01-15T18:02:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.559+0000 7f036be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036c071430 con 0x7f036c0720d0 2024-01-15T18:02:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.559+0000 7f036a000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f036c15f260 0x7f036c06c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54048/0 (socket says 172.21.15.2:54048) 2024-01-15T18:02:31.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.559+0000 7f036a000640 1 -- 172.21.15.2:0/3445558734 learned_addr learned my addr 172.21.15.2:0/3445558734 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.559+0000 7f036b400640 1 --2- 172.21.15.2:0/3445558734 >> v2:172.21.15.137:3300/0 conn(0x7f036c0720d0 0x7f036c06c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564736103 0 0) 0x7f036c071090 con 0x7f036c15f260 2024-01-15T18:02:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f034c003520 con 0x7f036c15f260 2024-01-15T18:02:31.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823189953 0 0) 0x7f036c144560 con 0x7f036c143d20 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f036c071090 con 0x7f036c143d20 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1933320179 0 0) 0x7f034c003520 con 0x7f036c15f260 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 >> v2:172.21.15.137:3300/0 conn(0x7f036c0720d0 msgr2=0x7f036c06c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 --2- 172.21.15.2:0/3445558734 >> v2:172.21.15.137:3300/0 conn(0x7f036c0720d0 0x7f036c06c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 >> v1:172.21.15.137:6789/0 conn(0x7f036c143d20 legacy=0x7f036c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c144280 con 0x7f036c15f260 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f0354001670 con 0x7f036c15f260 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 >> v1:172.21.15.2:6789/0 conn(0x7f036c15f260 legacy=0x7f036c06c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f034c003b50 0x7f034c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c0047c0 0x7f034c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f034c0051d0 0x7f034c00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c071430 con 0x7f034c003b50 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c010510 con 0x7f034c0051d0 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.560+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c0106d0 con 0x7f034c0047c0 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036aa00640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f034c003b50 0x7f034c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036b400640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f034c0051d0 0x7f034c00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036a000640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c0047c0 0x7f034c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036aa00640 1 -- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c0047c0 msgr2=0x7f034c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036aa00640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c0047c0 0x7f034c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036aa00640 1 -- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f034c0051d0 msgr2=0x7f034c00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036aa00640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f034c0051d0 0x7f034c00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036aa00640 1 -- 172.21.15.2:0/3445558734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c003520 con 0x7f034c003b50 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036b400640 1 -- 172.21.15.2:0/3445558734 reap_dead start 2024-01-15T18:02:31.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.561+0000 7f036a000640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c0047c0 0x7f034c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.562+0000 7f036aa00640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f034c003b50 0x7f034c003f50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f035c001fd0 tx=0x7f034c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68712dbd9b20e4a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.562+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c003b70 con 0x7f034c003b50 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035c003cd0 con 0x7f034c003b50 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f0369600640 1 -- 172.21.15.2:0/3445558734 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035c0052f0 con 0x7f034c003b50 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 -- 172.21.15.2:0/3445558734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f034c003b50 msgr2=0x7f034c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f034c003b50 0x7f034c003f50 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f035c001fd0 tx=0x7f034c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 -- 172.21.15.2:0/3445558734 shutdown_connections 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 --2- 172.21.15.2:0/3445558734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f034c003b50 0x7f034c003f50 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 -- 172.21.15.2:0/3445558734 >> 172.21.15.2:0/3445558734 conn(0x7f036c0b0840 msgr2=0x7f036c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:31.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 -- 172.21.15.2:0/3445558734 shutdown_connections 2024-01-15T18:02:31.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.563+0000 7f036be00640 1 -- 172.21.15.2:0/3445558734 wait complete. 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.565+0000 7f036be00640 1 Processor -- start 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.565+0000 7f036be00640 1 -- start start 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.566+0000 7f036be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c003b50 0x7f036c15b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.566+0000 7f036be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 0x7f036c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.566+0000 7f036be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036c0b2520 0x7f036c0b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.566+0000 7f036be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c14aef0 con 0x7f036c14d320 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.566+0000 7f036be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f036c13ae60 con 0x7f036c0b2520 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.566+0000 7f036be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f036c0b2ee0 con 0x7f034c003b50 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 0x7f036c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 0x7f036c15bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34734/0 (socket says 172.21.15.2:34734) 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 -- 172.21.15.2:0/2564335157 learned_addr learned my addr 172.21.15.2:0/2564335157 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036aa00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c003b50 0x7f036c15b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036b400640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036c0b2520 0x7f036c0b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 -- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c003b50 msgr2=0x7f036c15b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c003b50 0x7f036c15b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 -- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036c0b2520 msgr2=0x7f036c0b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036c0b2520 0x7f036c0b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 -- 172.21.15.2:0/2564335157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c14b160 con 0x7f036c14d320 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036aa00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c003b50 0x7f036c15b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.567+0000 7f036a000640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 0x7f036c15bd70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0354000fb0 tx=0x7f035400d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.568+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03540074e0 con 0x7f036c14d320 2024-01-15T18:02:31.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.568+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0354007640 con 0x7f036c14d320 2024-01-15T18:02:31.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.568+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03540049e0 con 0x7f036c14d320 2024-01-15T18:02:31.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.568+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036c15e6c0 con 0x7f036c14d320 2024-01-15T18:02:31.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.568+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f036c15c2b0 con 0x7f036c14d320 2024-01-15T18:02:31.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.569+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f036c15ebe0 con 0x7f036c14d320 2024-01-15T18:02:31.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.570+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f0354004b40 con 0x7f036c14d320 2024-01-15T18:02:31.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.571+0000 7f0362a00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f034406b820 0x7f034406dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:31.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.571+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0354089ec0 con 0x7f036c14d320 2024-01-15T18:02:31.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.571+0000 7f036aa00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f034406b820 0x7f034406dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:31.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.572+0000 7f036aa00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f034406b820 0x7f034406dce0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f035c034000 tx=0x7f035c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:31.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.575+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f036c15ebe0 con 0x7f036c14d320 2024-01-15T18:02:31.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:31.802+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f036c071850 con 0x7f034406b820 2024-01-15T18:02:35.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.149+0000 7f0362a00640 1 -- 172.21.15.2:0/2564335157 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f035c006b50 con 0x7f034406b820 2024-01-15T18:02:35.153 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:35.153 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:02:35.153 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:35.153 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:02:35.153 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:35.153 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:02:35.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.152+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f034406b820 msgr2=0x7f034406dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.152+0000 7f036be00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f034406b820 0x7f034406dce0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f035c034000 tx=0x7f035c009030 comp rx=0 tx=0).stop 2024-01-15T18:02:35.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.152+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 msgr2=0x7f036c15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.152+0000 7f036be00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 0x7f036c15bd70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f0354000fb0 tx=0x7f035400d9b0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 shutdown_connections 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f034406b820 0x7f034406dce0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036c0b2520 0x7f036c0b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f036c14d320 0x7f036c15bd70 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 --2- 172.21.15.2:0/2564335157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f034c003b50 0x7f036c15b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 >> 172.21.15.2:0/2564335157 conn(0x7f036c0b0840 msgr2=0x7f036c143050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 shutdown_connections 2024-01-15T18:02:35.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.153+0000 7f036be00640 1 -- 172.21.15.2:0/2564335157 wait complete. 2024-01-15T18:02:35.177 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:02:35.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.402+0000 7f91a0800640 1 Processor -- start 2024-01-15T18:02:35.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.402+0000 7f91a0800640 1 -- start start 2024-01-15T18:02:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.404+0000 7f91a0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 0x7f91980cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.404+0000 7f91a0800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91980cd9f0 con 0x7f919806c920 2024-01-15T18:02:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.404+0000 7f91a0800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919806aec0 con 0x7f91980cd2c0 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.404+0000 7f91a0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919806eb20 con 0x7f919806d410 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.405+0000 7f919de00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f919806c920 0x7f919806cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52982/0 (socket says 172.21.15.2:52982) 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.405+0000 7f919de00640 1 -- 172.21.15.2:0/374091197 learned_addr learned my addr 172.21.15.2:0/374091197 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.405+0000 7f919d400640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 0x7f91980cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.405+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410858980 0 0) 0x7f91980cd9f0 con 0x7f919806c920 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.405+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9178003520 con 0x7f919806c920 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 577459687 0 0) 0x7f9178003520 con 0x7f919806c920 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 msgr2=0x7f91980cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 0x7f91980cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 >> v1:172.21.15.137:6789/0 conn(0x7f91980cd2c0 legacy=0x7f9198166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919806b1a0 con 0x7f919806c920 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9188001670 con 0x7f919806c920 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 >> v1:172.21.15.2:6789/0 conn(0x7f919806c920 legacy=0x7f919806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91780039a0 0x7f9178003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f9178004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 0x7f91780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919806eb20 con 0x7f91780047c0 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917800bae0 con 0x7f91780039a0 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917800bca0 con 0x7f919806d410 2024-01-15T18:02:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.406+0000 7f919d400640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f9178004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.407+0000 7f919d400640 1 -- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 msgr2=0x7f91780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.407+0000 7f919d400640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919806d410 0x7f91780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.407+0000 7f919d400640 1 -- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91780039a0 msgr2=0x7f9178003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.407+0000 7f919d400640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91780039a0 0x7f9178003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.407+0000 7f919d400640 1 -- 172.21.15.2:0/374091197 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9178003520 con 0x7f91780047c0 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.408+0000 7f919d400640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f9178004c40 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9180009ac0 tx=0x7f91780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8a50c84554523d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.408+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918002c710 con 0x7f91780047c0 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.408+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918002c870 con 0x7f91780047c0 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.408+0000 7f919ca00640 1 -- 172.21.15.2:0/374091197 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918002cc00 con 0x7f91780047c0 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f91a0800640 1 -- 172.21.15.2:0/374091197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 msgr2=0x7f9178004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f91a0800640 1 --2- 172.21.15.2:0/374091197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f9178004c40 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f9180009ac0 tx=0x7f91780057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f919e800640 1 -- 172.21.15.2:0/374091197 reap_dead start 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f91a0800640 1 -- 172.21.15.2:0/374091197 shutdown_connections 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f91a0800640 1 -- 172.21.15.2:0/374091197 >> 172.21.15.2:0/374091197 conn(0x7f91980cc170 msgr2=0x7f91980cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f91a0800640 1 -- 172.21.15.2:0/374091197 shutdown_connections 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.409+0000 7f91a0800640 1 -- 172.21.15.2:0/374091197 wait complete. 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.411+0000 7f91a0800640 1 Processor -- start 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.411+0000 7f91a0800640 1 -- start start 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f91a0800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f91980b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f91a0800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91980b3510 0x7f9198144af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f91a0800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919815cb70 0x7f9198159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f91a0800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198145030 con 0x7f91780047c0 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f91a0800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91980b3a60 con 0x7f91980b3510 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f91a0800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91980b3d40 con 0x7f919815cb70 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f919de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f91980b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f919de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f91980b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49490/0 (socket says 172.21.15.2:49490) 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.412+0000 7f919de00640 1 -- 172.21.15.2:0/1468675504 learned_addr learned my addr 172.21.15.2:0/1468675504 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919e800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919815cb70 0x7f9198159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919d400640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91980b3510 0x7f9198144af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919de00640 1 -- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919815cb70 msgr2=0x7f9198159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919de00640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919815cb70 0x7f9198159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919de00640 1 -- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91980b3510 msgr2=0x7f9198144af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919de00640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91980b3510 0x7f9198144af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919de00640 1 -- 172.21.15.2:0/1468675504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980b4020 con 0x7f91780047c0 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919e800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919815cb70 0x7f9198159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f919de00640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f91980b4b00 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f91880011e0 tx=0x7f918800ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91880074e0 con 0x7f91780047c0 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9188007660 con 0x7f91780047c0 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9188004a20 con 0x7f91780047c0 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.413+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91980b4300 con 0x7f91780047c0 2024-01-15T18:02:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.414+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f919815cff0 con 0x7f91780047c0 2024-01-15T18:02:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.414+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91980b4820 con 0x7f91780047c0 2024-01-15T18:02:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.416+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f9188004b80 con 0x7f91780047c0 2024-01-15T18:02:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.416+0000 7f9196000640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f917006b800 0x7f917006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.417+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9188089900 con 0x7f91780047c0 2024-01-15T18:02:35.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.417+0000 7f919d400640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f917006b800 0x7f917006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.418+0000 7f919d400640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f917006b800 0x7f917006dcc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9180036000 tx=0x7f9180030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.421+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91980b4820 con 0x7f91780047c0 2024-01-15T18:02:35.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.650+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f919815d210 con 0x7f917006b800 2024-01-15T18:02:35.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.720+0000 7f9196000640 1 -- 172.21.15.2:0/1468675504 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f919815d210 con 0x7f917006b800 2024-01-15T18:02:35.724 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f917006b800 msgr2=0x7f917006dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f917006b800 0x7f917006dcc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f9180036000 tx=0x7f9180030000 comp rx=0 tx=0).stop 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 msgr2=0x7f91980b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f91980b4b00 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f91880011e0 tx=0x7f918800ccd0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 shutdown_connections 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f917006b800 0x7f917006dcc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f919815cb70 0x7f9198159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91980b3510 0x7f9198144af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 --2- 172.21.15.2:0/1468675504 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91780047c0 0x7f91980b4b00 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 >> 172.21.15.2:0/1468675504 conn(0x7f91980cc170 msgr2=0x7f9198154f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:35.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 shutdown_connections 2024-01-15T18:02:35.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.725+0000 7f91a0800640 1 -- 172.21.15.2:0/1468675504 wait complete. 2024-01-15T18:02:35.751 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:02:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.969+0000 7f21b1400640 1 Processor -- start 2024-01-15T18:02:35.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.969+0000 7f21b1400640 1 -- start start 2024-01-15T18:02:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21b1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21b1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f21ac06ce30 0x7f21ac070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21b1400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21ac0ce230 con 0x7f21ac144c00 2024-01-15T18:02:35.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21b1400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21ac06d2c0 con 0x7f21ac06ce30 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21b1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0ce510 con 0x7f21ac15c450 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21ab200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f21ac06ce30 0x7f21ac070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21ab200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f21ac06ce30 0x7f21ac070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49502/0 (socket says 172.21.15.2:49502) 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.971+0000 7f21ab200640 1 -- 172.21.15.2:0/1424395612 learned_addr learned my addr 172.21.15.2:0/1424395612 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.972+0000 7f21a9e00640 1 --2- 172.21.15.2:0/1424395612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.972+0000 7f21ab200640 1 -- 172.21.15.2:0/1424395612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 msgr2=0x7f21ac06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.972+0000 7f21ab200640 1 --2- 172.21.15.2:0/1424395612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.972+0000 7f21ab200640 1 -- 172.21.15.2:0/1424395612 >> v1:172.21.15.137:6789/0 conn(0x7f21ac144c00 legacy=0x7f21ac0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.972+0000 7f21ab200640 1 -- 172.21.15.2:0/1424395612 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac0ce7f0 con 0x7f21ac06ce30 2024-01-15T18:02:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.973+0000 7f21ab200640 1 --2- 172.21.15.2:0/1424395612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac06ce30 0x7f21ac070b10 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009e50 tx=0x7f21a402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c53fd77ab17bd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.973+0000 7f21a9400640 1 -- 172.21.15.2:0/1424395612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a402cb00 con 0x7f21ac06ce30 2024-01-15T18:02:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21a9400640 1 -- 172.21.15.2:0/1424395612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a402cc60 con 0x7f21ac06ce30 2024-01-15T18:02:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21a9400640 1 -- 172.21.15.2:0/1424395612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a40343f0 con 0x7f21ac06ce30 2024-01-15T18:02:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 -- 172.21.15.2:0/1424395612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac06ce30 msgr2=0x7f21ac070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 --2- 172.21.15.2:0/1424395612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac06ce30 0x7f21ac070b10 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f21a4009e50 tx=0x7f21a402bf60 comp rx=0 tx=0).stop 2024-01-15T18:02:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 -- 172.21.15.2:0/1424395612 shutdown_connections 2024-01-15T18:02:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 --2- 172.21.15.2:0/1424395612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac06ce30 0x7f21ac070b10 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 --2- 172.21.15.2:0/1424395612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 -- 172.21.15.2:0/1424395612 >> 172.21.15.2:0/1424395612 conn(0x7f21ac0b45f0 msgr2=0x7f21ac0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 -- 172.21.15.2:0/1424395612 shutdown_connections 2024-01-15T18:02:35.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.974+0000 7f21b1400640 1 -- 172.21.15.2:0/1424395612 wait complete. 2024-01-15T18:02:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.976+0000 7f21b1400640 1 Processor -- start 2024-01-15T18:02:35.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.976+0000 7f21b1400640 1 -- start start 2024-01-15T18:02:35.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.977+0000 7f21b1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21ac06ce30 0x7f21ac14b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.977+0000 7f21b1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 0x7f21ac13fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.977+0000 7f21b1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac1401c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.977+0000 7f21b1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac0af930 con 0x7f21ac144c00 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.977+0000 7f21b1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac15a3a0 con 0x7f21ac06ce30 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.977+0000 7f21b1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac13ed40 con 0x7f21ac15c450 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 0x7f21ac13fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 0x7f21ac13fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49506/0 (socket says 172.21.15.2:49506) 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 -- 172.21.15.2:0/1632038241 learned_addr learned my addr 172.21.15.2:0/1632038241 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21aa800640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21ac06ce30 0x7f21ac14b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21ab200640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac1401c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 -- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 msgr2=0x7f21ac1401c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac1401c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 -- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21ac06ce30 msgr2=0x7f21ac14b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21ac06ce30 0x7f21ac14b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 -- 172.21.15.2:0/1632038241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a4009b30 con 0x7f21ac144c00 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.978+0000 7f21a9e00640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 0x7f21ac13fc80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f219400d5d0 tx=0x7f219400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.979+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21940125f0 con 0x7f21ac144c00 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.979+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2194012770 con 0x7f21ac144c00 2024-01-15T18:02:35.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.979+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194012b20 con 0x7f21ac144c00 2024-01-15T18:02:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.979+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21ac13f020 con 0x7f21ac144c00 2024-01-15T18:02:35.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.979+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21ac0af550 con 0x7f21ac144c00 2024-01-15T18:02:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.980+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21ac0cfe20 con 0x7f21ac144c00 2024-01-15T18:02:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.981+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f2194012c80 con 0x7f21ac144c00 2024-01-15T18:02:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.982+0000 7f21a2a00640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f218406b7d0 0x7f218406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.982+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f219408a9a0 con 0x7f21ac144c00 2024-01-15T18:02:35.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.983+0000 7f21aa800640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f218406b7d0 0x7f218406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:35.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.984+0000 7f21aa800640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f218406b7d0 0x7f218406dc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f219c000f60 tx=0x7f219c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:35.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:35.987+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2194055120 con 0x7f21ac144c00 2024-01-15T18:02:36.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.205+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f21ac0cead0 con 0x7f218406b7d0 2024-01-15T18:02:36.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.250+0000 7f21a2a00640 1 -- 172.21.15.2:0/1632038241 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f219c001c70 con 0x7f218406b7d0 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f218406b7d0 msgr2=0x7f218406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f218406b7d0 0x7f218406dc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f219c000f60 tx=0x7f219c009040 comp rx=0 tx=0).stop 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 msgr2=0x7f21ac13fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 0x7f21ac13fc80 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f219400d5d0 tx=0x7f219400da90 comp rx=0 tx=0).stop 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 shutdown_connections 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f218406b7d0 0x7f218406dc90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f21ac15c450 0x7f21ac1401c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f21ac144c00 0x7f21ac13fc80 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 --2- 172.21.15.2:0/1632038241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f21ac06ce30 0x7f21ac14b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.253+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 >> 172.21.15.2:0/1632038241 conn(0x7f21ac0b45f0 msgr2=0x7f21ac071430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.254+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 shutdown_connections 2024-01-15T18:02:36.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.254+0000 7f21b1400640 1 -- 172.21.15.2:0/1632038241 wait complete. 2024-01-15T18:02:36.264 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.260+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.264 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.260+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.264 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.260+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.264 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.260+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.264 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.260+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.278 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone22404 2024-01-15T18:02:36.293 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.290+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.294 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.290+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.294 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.290+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.294 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.290+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.294 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:02:36.290+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:02:36.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.495+0000 7f4ec5200640 1 Processor -- start 2024-01-15T18:02:36.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.495+0000 7f4ec5200640 1 -- start start 2024-01-15T18:02:36.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.496+0000 7f4ec5200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4ec00b2210 0x7f4ec00d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:36.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ec5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ec00d1c50 0x7f4ec00b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:36.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ec5200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec00d1250 con 0x7f4ec00d1870 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ec5200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec00ce6d0 con 0x7f4ec00b2210 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ec5200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec00ce8e0 con 0x7f4ec00d1c50 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ebf400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4ec00d1870 0x7f4ec015d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52992/0 (socket says 172.21.15.2:52992) 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ebf400640 1 -- 172.21.15.2:0/2501466651 learned_addr learned my addr 172.21.15.2:0/2501466651 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ebea00640 1 --2- 172.21.15.2:0/2501466651 >> v2:172.21.15.2:3300/0 conn(0x7f4ec00b2210 0x7f4ec00d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ebe000640 1 -- 172.21.15.2:0/2501466651 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976989669 0 0) 0x7f4ec00d1250 con 0x7f4ec00d1870 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.497+0000 7f4ebe000640 1 -- 172.21.15.2:0/2501466651 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea0003520 con 0x7f4ec00d1870 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.498+0000 7f4ebfe00640 1 --2- 172.21.15.2:0/2501466651 >> v2:172.21.15.137:3300/0 conn(0x7f4ec00d1c50 0x7f4ec00b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.498+0000 7f4ebea00640 1 -- 172.21.15.2:0/2501466651 >> v2:172.21.15.137:3300/0 conn(0x7f4ec00d1c50 msgr2=0x7f4ec00b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:36.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.498+0000 7f4ebea00640 1 --2- 172.21.15.2:0/2501466651 >> v2:172.21.15.137:3300/0 conn(0x7f4ec00d1c50 0x7f4ec00b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.498+0000 7f4ebea00640 1 -- 172.21.15.2:0/2501466651 >> v1:172.21.15.2:6789/0 conn(0x7f4ec00d1870 legacy=0x7f4ec015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.498+0000 7f4ebea00640 1 -- 172.21.15.2:0/2501466651 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00cebc0 con 0x7f4ec00b2210 2024-01-15T18:02:36.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.498+0000 7f4ebfe00640 1 --2- 172.21.15.2:0/2501466651 >> v2:172.21.15.137:3300/0 conn(0x7f4ec00d1c50 0x7f4ec00b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:36.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.499+0000 7f4ebea00640 1 --2- 172.21.15.2:0/2501466651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec00d3bd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4ec008d7e0 tx=0x7f4ea802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23aac552b256e93b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:36.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.499+0000 7f4ebe000640 1 -- 172.21.15.2:0/2501466651 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8031830 con 0x7f4ec00b2210 2024-01-15T18:02:36.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.499+0000 7f4ebe000640 1 -- 172.21.15.2:0/2501466651 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea8031990 con 0x7f4ec00b2210 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.499+0000 7f4ebe000640 1 -- 172.21.15.2:0/2501466651 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea8031d20 con 0x7f4ec00b2210 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 -- 172.21.15.2:0/2501466651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 msgr2=0x7f4ec00d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 --2- 172.21.15.2:0/2501466651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec00d3bd0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4ec008d7e0 tx=0x7f4ea802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 -- 172.21.15.2:0/2501466651 shutdown_connections 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 --2- 172.21.15.2:0/2501466651 >> v2:172.21.15.137:3300/0 conn(0x7f4ec00d1c50 0x7f4ec00b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 --2- 172.21.15.2:0/2501466651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec00d3bd0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 -- 172.21.15.2:0/2501466651 >> 172.21.15.2:0/2501466651 conn(0x7f4ec00d30b0 msgr2=0x7f4ec00d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 -- 172.21.15.2:0/2501466651 shutdown_connections 2024-01-15T18:02:36.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.500+0000 7f4ec5200640 1 -- 172.21.15.2:0/2501466651 wait complete. 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.502+0000 7f4ec5200640 1 Processor -- start 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.502+0000 7f4ec5200640 1 -- start start 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ec5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ec5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec00d1870 0x7f4ec015f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ec5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00d1c50 0x7f4ec00c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ec5200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec01604b0 con 0x7f4ec00b2210 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ec5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec015f830 con 0x7f4ec00d1c50 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ec5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec015fb10 con 0x7f4ec00d1870 2024-01-15T18:02:36.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec015ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49528/0 (socket says 172.21.15.2:49528) 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 -- 172.21.15.2:0/548368117 learned_addr learned my addr 172.21.15.2:0/548368117 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebfe00640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00d1c50 0x7f4ec00c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 -- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec00d1870 msgr2=0x7f4ec015f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec00d1870 0x7f4ec015f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 -- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00d1c50 msgr2=0x7f4ec00c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00d1c50 0x7f4ec00c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.503+0000 7f4ebf400640 1 -- 172.21.15.2:0/548368117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea8009400 con 0x7f4ec00b2210 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.504+0000 7f4ebf400640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec015ecc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c500 tx=0x7f4eb000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.504+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0005ce0 con 0x7f4ec00b2210 2024-01-15T18:02:36.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.504+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb0005e40 con 0x7f4ec00b2210 2024-01-15T18:02:36.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.504+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0007330 con 0x7f4ec00b2210 2024-01-15T18:02:36.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.504+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec015fdf0 con 0x7f4ec00b2210 2024-01-15T18:02:36.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.504+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec0152260 con 0x7f4ec00b2210 2024-01-15T18:02:36.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.505+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec0074770 con 0x7f4ec00b2210 2024-01-15T18:02:36.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.506+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f4eb0007490 con 0x7f4ec00b2210 2024-01-15T18:02:36.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.507+0000 7f4eb7400640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e9806b820 0x7f4e9806dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:36.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.507+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4eb0089a70 con 0x7f4ec00b2210 2024-01-15T18:02:36.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.508+0000 7f4ebea00640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e9806b820 0x7f4e9806dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:36.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.509+0000 7f4ebea00640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e9806b820 0x7f4e9806dce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8036000 tx=0x7f4ea8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:36.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.516+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4eb0057e50 con 0x7f4ec00b2210 2024-01-15T18:02:36.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.739+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone22404", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ec00d0f20 con 0x7f4e9806b820 2024-01-15T18:02:36.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.781+0000 7f4eb7400640 1 -- 172.21.15.2:0/548368117 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4ea8031b00 con 0x7f4e9806b820 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e9806b820 msgr2=0x7f4e9806dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e9806b820 0x7f4e9806dce0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8036000 tx=0x7f4ea8030000 comp rx=0 tx=0).stop 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 msgr2=0x7f4ec015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec015ecc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f4eb000c500 tx=0x7f4eb000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 shutdown_connections 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e9806b820 0x7f4e9806dce0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ec00d1c50 0x7f4ec00c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ec00d1870 0x7f4ec015f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 --2- 172.21.15.2:0/548368117 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ec00b2210 0x7f4ec015ecc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:36.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 >> 172.21.15.2:0/548368117 conn(0x7f4ec00d30b0 msgr2=0x7f4ec015c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:36.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 shutdown_connections 2024-01-15T18:02:36.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.784+0000 7f4ec5200640 1 -- 172.21.15.2:0/548368117 wait complete. 2024-01-15T18:02:36.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone22403 2024-01-15T18:02:36.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.992+0000 7f6a07200640 1 Processor -- start 2024-01-15T18:02:36.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.993+0000 7f6a07200640 1 -- start start 2024-01-15T18:02:37.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.995+0000 7f6a07200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6a08160800 0x7f6a080cd900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.995+0000 7f6a07200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a080d0ba0 con 0x7f6a080cde40 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.995+0000 7f6a07200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0806d250 con 0x7f6a0806abc0 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.995+0000 7f6a07200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a08160be0 con 0x7f6a08160800 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.995+0000 7f6a05400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6a080cde40 0x7f6a0806a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53004/0 (socket says 172.21.15.2:53004) 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.995+0000 7f6a05400640 1 -- 172.21.15.2:0/569059025 learned_addr learned my addr 172.21.15.2:0/569059025 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a05e00640 1 --2- 172.21.15.2:0/569059025 >> v2:172.21.15.137:3300/0 conn(0x7f6a08160800 0x7f6a080cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208608638 0 0) 0x7f6a080d0ba0 con 0x7f6a080cde40 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e8003520 con 0x7f6a080cde40 2024-01-15T18:02:37.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3803545249 0 0) 0x7f69e8003520 con 0x7f6a080cde40 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 >> v2:172.21.15.137:3300/0 conn(0x7f6a08160800 msgr2=0x7f6a080cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a04a00640 1 --2- 172.21.15.2:0/569059025 >> v2:172.21.15.137:3300/0 conn(0x7f6a08160800 0x7f6a080cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.996+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 >> v1:172.21.15.137:6789/0 conn(0x7f6a0806abc0 legacy=0x7f6a08160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a080712b0 con 0x7f6a080cde40 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f69f0001670 con 0x7f6a080cde40 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 >> v1:172.21.15.2:6789/0 conn(0x7f6a080cde40 legacy=0x7f6a0806a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69e80039c0 0x7f69e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69e80047c0 0x7f69e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f69e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e800c170 con 0x7f69e80052a0 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e800c2e0 con 0x7f69e80039c0 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e800c4a0 con 0x7f69e80047c0 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a06800640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f69e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a05400640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69e80047c0 0x7f69e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a05e00640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69e80039c0 0x7f69e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a06800640 1 -- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69e80047c0 msgr2=0x7f69e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a06800640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69e80047c0 0x7f69e8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a06800640 1 -- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69e80039c0 msgr2=0x7f69e8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a06800640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f69e80039c0 0x7f69e8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a06800640 1 -- 172.21.15.2:0/569059025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e8003520 con 0x7f69e80052a0 2024-01-15T18:02:37.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.997+0000 7f6a05400640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f69e80047c0 0x7f69e8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:37.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.999+0000 7f6a06800640 1 -- 172.21.15.2:0/569059025 reap_dead start 2024-01-15T18:02:37.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.999+0000 7f6a06800640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f69e800b980 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6a00001750 tx=0x7f69e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c157665f9ccc6b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:37.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.999+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a000052d0 con 0x7f69e80052a0 2024-01-15T18:02:37.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.999+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0002e050 con 0x7f69e80052a0 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:36.999+0000 7f6a04a00640 1 -- 172.21.15.2:0/569059025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a00003830 con 0x7f69e80052a0 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 -- 172.21.15.2:0/569059025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 msgr2=0x7f69e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f69e800b980 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f6a00001750 tx=0x7f69e80057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 -- 172.21.15.2:0/569059025 shutdown_connections 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 --2- 172.21.15.2:0/569059025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f69e800b980 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 -- 172.21.15.2:0/569059025 >> 172.21.15.2:0/569059025 conn(0x7f6a080cfd30 msgr2=0x7f6a080d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 -- 172.21.15.2:0/569059025 shutdown_connections 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.000+0000 7f6a07200640 1 -- 172.21.15.2:0/569059025 wait complete. 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.002+0000 7f6a07200640 1 Processor -- start 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.002+0000 7f6a07200640 1 -- start start 2024-01-15T18:02:37.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a07200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f6a080c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a07200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a080c7340 0x7f6a0806e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a07200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a08152d30 0x7f6a0806e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a07200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a08154fe0 con 0x7f69e80052a0 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a07200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a081543d0 con 0x7f6a08152d30 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a07200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a081546b0 con 0x7f6a080c7340 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a05e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f6a080c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a05e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f6a080c6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49546/0 (socket says 172.21.15.2:49546) 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.003+0000 7f6a05e00640 1 -- 172.21.15.2:0/4034708181 learned_addr learned my addr 172.21.15.2:0/4034708181 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a06800640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a08152d30 0x7f6a0806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05400640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a080c7340 0x7f6a0806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05e00640 1 -- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a080c7340 msgr2=0x7f6a0806e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05e00640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a080c7340 0x7f6a0806e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05e00640 1 -- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a08152d30 msgr2=0x7f6a0806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05e00640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a08152d30 0x7f6a0806e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05e00640 1 -- 172.21.15.2:0/4034708181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08154990 con 0x7f69e80052a0 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05400640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a080c7340 0x7f6a0806e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f6a05e00640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f6a080c6e00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f69f800f380 tx=0x7f69f800f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.004+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f8017600 con 0x7f69e80052a0 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.005+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f8011040 con 0x7f69e80052a0 2024-01-15T18:02:37.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.005+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a08154c70 con 0x7f69e80052a0 2024-01-15T18:02:37.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.005+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80179a0 con 0x7f69e80052a0 2024-01-15T18:02:37.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.005+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a08155650 con 0x7f69e80052a0 2024-01-15T18:02:37.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.006+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a08154e30 con 0x7f69e80052a0 2024-01-15T18:02:37.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.007+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f69f80203b0 con 0x7f69e80052a0 2024-01-15T18:02:37.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.008+0000 7f69fe000640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f69e006b800 0x7f69e006dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.008+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f69f808ac90 con 0x7f69e80052a0 2024-01-15T18:02:37.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.009+0000 7f6a05400640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f69e006b800 0x7f69e006dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.010+0000 7f6a05400640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f69e006b800 0x7f69e006dcc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f69f00017a0 tx=0x7f69f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:37.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.013+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69f8055300 con 0x7f69e80052a0 2024-01-15T18:02:37.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.231+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone22403", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a08154820 con 0x7f69e006b800 2024-01-15T18:02:37.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.276+0000 7f69fe000640 1 -- 172.21.15.2:0/4034708181 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6a08154820 con 0x7f69e006b800 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f69e006b800 msgr2=0x7f69e006dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f69e006b800 0x7f69e006dcc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f69f00017a0 tx=0x7f69f0009040 comp rx=0 tx=0).stop 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 msgr2=0x7f6a080c6e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f6a080c6e00 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f69f800f380 tx=0x7f69f800f840 comp rx=0 tx=0).stop 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 shutdown_connections 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f69e006b800 0x7f69e006dcc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a08152d30 0x7f6a0806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a080c7340 0x7f6a0806e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 --2- 172.21.15.2:0/4034708181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f69e80052a0 0x7f6a080c6e00 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 >> 172.21.15.2:0/4034708181 conn(0x7f6a080cfd30 msgr2=0x7f6a080cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:37.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 shutdown_connections 2024-01-15T18:02:37.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.280+0000 7f6a07200640 1 -- 172.21.15.2:0/4034708181 wait complete. 2024-01-15T18:02:37.303 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone22401 2024-01-15T18:02:37.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.518+0000 7fd90ae00640 1 Processor -- start 2024-01-15T18:02:37.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.518+0000 7fd90ae00640 1 -- start start 2024-01-15T18:02:37.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd90ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90415c450 0x7fd90406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd90ae00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd90406ce30 0x7fd904070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd90ae00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9040ce230 con 0x7fd904144c00 2024-01-15T18:02:37.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd90ae00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90406d2c0 con 0x7fd90406ce30 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd90ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ce510 con 0x7fd90415c450 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd908e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd90406ce30 0x7fd904070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd908e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd90406ce30 0x7fd904070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49554/0 (socket says 172.21.15.2:49554) 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.520+0000 7fd908e00640 1 -- 172.21.15.2:0/1016290231 learned_addr learned my addr 172.21.15.2:0/1016290231 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.521+0000 7fd908e00640 1 -- 172.21.15.2:0/1016290231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90415c450 msgr2=0x7fd90406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.521+0000 7fd908e00640 1 --2- 172.21.15.2:0/1016290231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90415c450 0x7fd90406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.521+0000 7fd908e00640 1 -- 172.21.15.2:0/1016290231 >> v1:172.21.15.137:6789/0 conn(0x7fd904144c00 legacy=0x7fd9040cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:37.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.521+0000 7fd908e00640 1 -- 172.21.15.2:0/1016290231 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9040ce7f0 con 0x7fd90406ce30 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.522+0000 7fd908e00640 1 --2- 172.21.15.2:0/1016290231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd90406ce30 0x7fd904070b10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009e50 tx=0x7fd8fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c3776bd09e27ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.522+0000 7fd902a00640 1 -- 172.21.15.2:0/1016290231 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc02ca70 con 0x7fd90406ce30 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.522+0000 7fd902a00640 1 -- 172.21.15.2:0/1016290231 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc02cbd0 con 0x7fd90406ce30 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.522+0000 7fd902a00640 1 -- 172.21.15.2:0/1016290231 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc0343f0 con 0x7fd90406ce30 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 -- 172.21.15.2:0/1016290231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd90406ce30 msgr2=0x7fd904070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 --2- 172.21.15.2:0/1016290231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd90406ce30 0x7fd904070b10 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009e50 tx=0x7fd8fc02bed0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 -- 172.21.15.2:0/1016290231 shutdown_connections 2024-01-15T18:02:37.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 --2- 172.21.15.2:0/1016290231 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd90406ce30 0x7fd904070b10 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 --2- 172.21.15.2:0/1016290231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90415c450 0x7fd90406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 -- 172.21.15.2:0/1016290231 >> 172.21.15.2:0/1016290231 conn(0x7fd9040b45f0 msgr2=0x7fd9040b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:37.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 -- 172.21.15.2:0/1016290231 shutdown_connections 2024-01-15T18:02:37.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.523+0000 7fd90ae00640 1 -- 172.21.15.2:0/1016290231 wait complete. 2024-01-15T18:02:37.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.525+0000 7fd90ae00640 1 Processor -- start 2024-01-15T18:02:37.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.525+0000 7fd90ae00640 1 -- start start 2024-01-15T18:02:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.526+0000 7fd90ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90406ce30 0x7fd904157b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.526+0000 7fd90ae00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 0x7fd904156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.526+0000 7fd90ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd90415c450 0x7fd9041419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.526+0000 7fd90ae00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ae340 con 0x7fd904144c00 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.526+0000 7fd90ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90413c8a0 con 0x7fd90415c450 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.526+0000 7fd90ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90413cb80 con 0x7fd90406ce30 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 0x7fd904156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 0x7fd904156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49558/0 (socket says 172.21.15.2:49558) 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 -- 172.21.15.2:0/4042351331 learned_addr learned my addr 172.21.15.2:0/4042351331 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd908e00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd90415c450 0x7fd9041419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903e00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90406ce30 0x7fd904157b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 -- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90406ce30 msgr2=0x7fd904157b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90406ce30 0x7fd904157b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 -- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd90415c450 msgr2=0x7fd9041419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd90415c450 0x7fd9041419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.527+0000 7fd903400640 1 -- 172.21.15.2:0/4042351331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009b30 con 0x7fd904144c00 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.528+0000 7fd903400640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 0x7fd904156a70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00b410 tx=0x7fd8ec00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.528+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011830 con 0x7fd904144c00 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.528+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec0119b0 con 0x7fd904144c00 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.528+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec011d60 con 0x7fd904144c00 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.528+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90413ce60 con 0x7fd904144c00 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.528+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9041577d0 con 0x7fd904144c00 2024-01-15T18:02:37.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.529+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90413d380 con 0x7fd904144c00 2024-01-15T18:02:37.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.530+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd8ec01a400 con 0x7fd904144c00 2024-01-15T18:02:37.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.531+0000 7fd8fbe00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:37.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.531+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd8ec08a610 con 0x7fd904144c00 2024-01-15T18:02:37.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.532+0000 7fd903e00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:37.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.533+0000 7fd903e00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4004ea0 tx=0x7fd8f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:37.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.536+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8ec054d90 con 0x7fd904144c00 2024-01-15T18:02:37.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.761+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone22401", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9040cead0 con 0x7fd8dc06b7d0 2024-01-15T18:02:37.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.806+0000 7fd8fbe00640 1 -- 172.21.15.2:0/4042351331 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd8f4005420 con 0x7fd8dc06b7d0 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8dc06b7d0 msgr2=0x7fd8dc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4004ea0 tx=0x7fd8f4009040 comp rx=0 tx=0).stop 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 msgr2=0x7fd904156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 0x7fd904156a70 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00b410 tx=0x7fd8ec00b8d0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 shutdown_connections 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8dc06b7d0 0x7fd8dc06dc90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd90415c450 0x7fd9041419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd904144c00 0x7fd904156a70 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 --2- 172.21.15.2:0/4042351331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd90406ce30 0x7fd904157b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 >> 172.21.15.2:0/4042351331 conn(0x7fd9040b45f0 msgr2=0x7fd9040714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 shutdown_connections 2024-01-15T18:02:37.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:37.810+0000 7fd90ae00640 1 -- 172.21.15.2:0/4042351331 wait complete. 2024-01-15T18:02:37.837 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone22402 2024-01-15T18:02:38.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.053+0000 7fcba2800640 1 Processor -- start 2024-01-15T18:02:38.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.054+0000 7fcba2800640 1 -- start start 2024-01-15T18:02:38.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.055+0000 7fcba2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb9c146ba0 0x7fcb9c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:38.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.055+0000 7fcba2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c15e0b0 0x7fcb9c0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:38.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcba2800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c06f440 con 0x7fcb9c0b2ba0 2024-01-15T18:02:38.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcba2800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c146f80 con 0x7fcb9c146ba0 2024-01-15T18:02:38.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcba2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0b3520 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb9c146ba0 0x7fcb9c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb9c146ba0 0x7fcb9c0d1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49568/0 (socket says 172.21.15.2:49568) 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 -- 172.21.15.2:0/173864081 learned_addr learned my addr 172.21.15.2:0/173864081 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9be00640 1 --2- 172.21.15.2:0/173864081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c15e0b0 0x7fcb9c0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 -- 172.21.15.2:0/173864081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c15e0b0 msgr2=0x7fcb9c0b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 --2- 172.21.15.2:0/173864081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c15e0b0 0x7fcb9c0b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 -- 172.21.15.2:0/173864081 >> v1:172.21.15.2:6789/0 conn(0x7fcb9c0b2ba0 legacy=0x7fcb9c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:38.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.056+0000 7fcb9b400640 1 -- 172.21.15.2:0/173864081 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c15e5f0 con 0x7fcb9c146ba0 2024-01-15T18:02:38.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.058+0000 7fcb9b400640 1 --2- 172.21.15.2:0/173864081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c146ba0 0x7fcb9c0d1a60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009870 tx=0x7fcb8c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50e689de5b320f66 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:38.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.058+0000 7fcb9a000640 1 -- 172.21.15.2:0/173864081 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c031640 con 0x7fcb9c146ba0 2024-01-15T18:02:38.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.058+0000 7fcb9a000640 1 -- 172.21.15.2:0/173864081 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c0317a0 con 0x7fcb9c146ba0 2024-01-15T18:02:38.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.058+0000 7fcb9a000640 1 -- 172.21.15.2:0/173864081 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c031b30 con 0x7fcb9c146ba0 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 -- 172.21.15.2:0/173864081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c146ba0 msgr2=0x7fcb9c0d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 --2- 172.21.15.2:0/173864081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c146ba0 0x7fcb9c0d1a60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c009870 tx=0x7fcb8c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 -- 172.21.15.2:0/173864081 shutdown_connections 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 --2- 172.21.15.2:0/173864081 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c15e0b0 0x7fcb9c0b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 --2- 172.21.15.2:0/173864081 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c146ba0 0x7fcb9c0d1a60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 -- 172.21.15.2:0/173864081 >> 172.21.15.2:0/173864081 conn(0x7fcb9c0b6430 msgr2=0x7fcb9c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 -- 172.21.15.2:0/173864081 shutdown_connections 2024-01-15T18:02:38.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.059+0000 7fcba2800640 1 -- 172.21.15.2:0/173864081 wait complete. 2024-01-15T18:02:38.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.061+0000 7fcba2800640 1 Processor -- start 2024-01-15T18:02:38.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.061+0000 7fcba2800640 1 -- start start 2024-01-15T18:02:38.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcba2800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0b2ba0 0x7fcb9c0cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcba2800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c146ba0 0x7fcb9c0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcba2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 0x7fcb9c1427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcba2800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0cea60 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcba2800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0cb3f0 con 0x7fcb9c0b2ba0 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcba2800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0cecb0 con 0x7fcb9c146ba0 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcb9be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 0x7fcb9c1427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcb9be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 0x7fcb9c1427a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49574/0 (socket says 172.21.15.2:49574) 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcb9be00640 1 -- 172.21.15.2:0/1060176538 learned_addr learned my addr 172.21.15.2:0/1060176538 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.062+0000 7fcb9b400640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0b2ba0 0x7fcb9c0cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9aa00640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c146ba0 0x7fcb9c0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9be00640 1 -- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c146ba0 msgr2=0x7fcb9c0caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9be00640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c146ba0 0x7fcb9c0caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9be00640 1 -- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0b2ba0 msgr2=0x7fcb9c0cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9be00640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0b2ba0 0x7fcb9c0cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9be00640 1 -- 172.21.15.2:0/1060176538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0094d0 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9b400640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0b2ba0 0x7fcb9c0cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb9be00640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 0x7fcb9c1427a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400d600 tx=0x7fcb9400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb940125f0 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb94012750 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb94012b00 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.063+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c0cef90 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.064+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c0cf550 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.065+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c0686f0 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.066+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fcb94012c60 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.067+0000 7fcb93400640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb7406b7d0 0x7fcb7406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:38.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.067+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcb9408a920 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.068+0000 7fcb9b400640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb7406b7d0 0x7fcb7406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:38.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.069+0000 7fcb9b400640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb7406b7d0 0x7fcb7406dc90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c036000 tx=0x7fcb8c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:38.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.072+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb94055130 con 0x7fcb9c15e0b0 2024-01-15T18:02:38.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.295+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone22402", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb9c0b4e50 con 0x7fcb7406b7d0 2024-01-15T18:02:38.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.349+0000 7fcb93400640 1 -- 172.21.15.2:0/1060176538 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcb9c0b4e50 con 0x7fcb7406b7d0 2024-01-15T18:02:38.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.352+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb7406b7d0 msgr2=0x7fcb7406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.352+0000 7fcba2800640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb7406b7d0 0x7fcb7406dc90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c036000 tx=0x7fcb8c03c040 comp rx=0 tx=0).stop 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 msgr2=0x7fcb9c1427a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 0x7fcb9c1427a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400d600 tx=0x7fcb9400dac0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 shutdown_connections 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb7406b7d0 0x7fcb7406dc90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb9c15e0b0 0x7fcb9c1427a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb9c146ba0 0x7fcb9c0caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 --2- 172.21.15.2:0/1060176538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb9c0b2ba0 0x7fcb9c0cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 >> 172.21.15.2:0/1060176538 conn(0x7fcb9c0b6430 msgr2=0x7fcb9c0b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 shutdown_connections 2024-01-15T18:02:38.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:38.353+0000 7fcba2800640 1 -- 172.21.15.2:0/1060176538 wait complete. 2024-01-15T18:02:38.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:38.379 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:02:38.443 INFO:teuthology.orchestra.run.smithi002.stdout:3 2024-01-15T18:02:43.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:43.445 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:02:43.513 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-01-15T18:02:43.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:02:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.749+0000 7f41cea00640 1 Processor -- start 2024-01-15T18:02:43.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.749+0000 7f41cea00640 1 -- start start 2024-01-15T18:02:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c81439f0 0x7f41c80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f41c80b3420 0x7f41c80cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cea00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c815dd70 con 0x7f41c80ccc70 2024-01-15T18:02:43.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cea00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c80b3880 con 0x7f41c80b3420 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80711b0 con 0x7f41c81439f0 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41c7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f41c80b3420 0x7f41c80cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cca00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f41c80ccc70 0x7f41c806c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40864/0 (socket says 172.21.15.2:40864) 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.751+0000 7f41cca00640 1 -- 172.21.15.2:0/53147475 learned_addr learned my addr 172.21.15.2:0/53147475 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.752+0000 7f41c7e00640 1 --2- 172.21.15.2:0/53147475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c81439f0 0x7f41c80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.752+0000 7f41c7400640 1 -- 172.21.15.2:0/53147475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c81439f0 msgr2=0x7f41c80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.752+0000 7f41c7400640 1 --2- 172.21.15.2:0/53147475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c81439f0 0x7f41c80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.752+0000 7f41c7400640 1 -- 172.21.15.2:0/53147475 >> v1:172.21.15.2:6789/0 conn(0x7f41c80ccc70 legacy=0x7f41c806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:43.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.752+0000 7f41c7400640 1 -- 172.21.15.2:0/53147475 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c8143dd0 con 0x7f41c80b3420 2024-01-15T18:02:43.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.752+0000 7f41c6a00640 1 -- 172.21.15.2:0/53147475 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883634255 0 0) 0x7f41c815dd70 con 0x7f41c80ccc70 2024-01-15T18:02:43.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.753+0000 7f41c7400640 1 --2- 172.21.15.2:0/53147475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80b3420 0x7f41c80cc730 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f41b00097a0 tx=0x7f41b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e3b3f704eb2984 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:43.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.753+0000 7f41c6a00640 1 -- 172.21.15.2:0/53147475 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0031830 con 0x7f41c80b3420 2024-01-15T18:02:43.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41c6a00640 1 -- 172.21.15.2:0/53147475 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b0031990 con 0x7f41c80b3420 2024-01-15T18:02:43.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41c6a00640 1 -- 172.21.15.2:0/53147475 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0031d20 con 0x7f41c80b3420 2024-01-15T18:02:43.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 -- 172.21.15.2:0/53147475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80b3420 msgr2=0x7f41c80cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:43.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 --2- 172.21.15.2:0/53147475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80b3420 0x7f41c80cc730 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f41b00097a0 tx=0x7f41b002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:02:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 -- 172.21.15.2:0/53147475 shutdown_connections 2024-01-15T18:02:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 --2- 172.21.15.2:0/53147475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80b3420 0x7f41c80cc730 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 --2- 172.21.15.2:0/53147475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c81439f0 0x7f41c80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 -- 172.21.15.2:0/53147475 >> 172.21.15.2:0/53147475 conn(0x7f41c80b45f0 msgr2=0x7f41c80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 -- 172.21.15.2:0/53147475 shutdown_connections 2024-01-15T18:02:43.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.754+0000 7f41cea00640 1 -- 172.21.15.2:0/53147475 wait complete. 2024-01-15T18:02:43.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.756+0000 7f41cea00640 1 Processor -- start 2024-01-15T18:02:43.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.756+0000 7f41cea00640 1 -- start start 2024-01-15T18:02:43.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.757+0000 7f41cea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c80b3420 0x7f41c81ef8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:43.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.757+0000 7f41cea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 0x7f41c80ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:43.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.757+0000 7f41cea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41c81439f0 0x7f41c815c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:43.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.757+0000 7f41cea00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c81efe30 con 0x7f41c80ccc70 2024-01-15T18:02:43.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.757+0000 7f41cea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80a8800 con 0x7f41c81439f0 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.757+0000 7f41cea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c80a8ae0 con 0x7f41c80b3420 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 0x7f41c80ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 0x7f41c80ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36814/0 (socket says 172.21.15.2:36814) 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 -- 172.21.15.2:0/785222793 learned_addr learned my addr 172.21.15.2:0/785222793 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41cca00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41c81439f0 0x7f41c815c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7e00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c80b3420 0x7f41c81ef8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 -- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c80b3420 msgr2=0x7f41c81ef8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c80b3420 0x7f41c81ef8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 -- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41c81439f0 msgr2=0x7f41c815c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41c81439f0 0x7f41c815c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:43.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.758+0000 7f41c7400640 1 -- 172.21.15.2:0/785222793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b0009400 con 0x7f41c80ccc70 2024-01-15T18:02:43.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.759+0000 7f41c7400640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 0x7f41c80ca740 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f41b002bee0 tx=0x7f41b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:43.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.759+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b003a6f0 con 0x7f41c80ccc70 2024-01-15T18:02:43.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.759+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b003a870 con 0x7f41c80ccc70 2024-01-15T18:02:43.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.759+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b003ac20 con 0x7f41c80ccc70 2024-01-15T18:02:43.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.759+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c80a8dc0 con 0x7f41c80ccc70 2024-01-15T18:02:43.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.759+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c80a9380 con 0x7f41c80ccc70 2024-01-15T18:02:43.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.760+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c80a9f40 con 0x7f41c80ccc70 2024-01-15T18:02:43.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.761+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f41b004c3b0 con 0x7f41c80ccc70 2024-01-15T18:02:43.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.762+0000 7f41bfe00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41a006b940 0x7f41a006de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:43.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.762+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f41b00ac6b0 con 0x7f41c80ccc70 2024-01-15T18:02:43.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.763+0000 7f41c7e00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41a006b940 0x7f41a006de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:43.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.764+0000 7f41c7e00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41a006b940 0x7f41a006de00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f41b8006fd0 tx=0x7f41b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:43.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.767+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41c80a9f40 con 0x7f41c80ccc70 2024-01-15T18:02:43.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:43.985+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c80b5220 con 0x7f41a006b940 2024-01-15T18:02:44.036 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T18:02:44.032+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:02:44.138 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:02:46.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:46.071+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f41b0076f70 con 0x7f41c80ccc70 2024-01-15T18:02:47.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.092+0000 7f41bfe00640 1 -- 172.21.15.2:0/785222793 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f41c80b5220 con 0x7f41a006b940 2024-01-15T18:02:47.096 INFO:teuthology.orchestra.run.smithi002.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41a006b940 msgr2=0x7f41a006de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41a006b940 0x7f41a006de00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f41b8006fd0 tx=0x7f41b8008040 comp rx=0 tx=0).stop 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 msgr2=0x7f41c80ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 0x7f41c80ca740 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f41b002bee0 tx=0x7f41b00046c0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 shutdown_connections 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41a006b940 0x7f41a006de00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41c81439f0 0x7f41c815c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41c80ccc70 0x7f41c80ca740 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 --2- 172.21.15.2:0/785222793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41c80b3420 0x7f41c81ef8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 >> 172.21.15.2:0/785222793 conn(0x7f41c80b45f0 msgr2=0x7f41c80aeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:47.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 shutdown_connections 2024-01-15T18:02:47.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.096+0000 7f41cea00640 1 -- 172.21.15.2:0/785222793 wait complete. 2024-01-15T18:02:47.123 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:02:47.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:02:47.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:02:47.355 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:02:47.356 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:02:47.356 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:47.406 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:02:47.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.645+0000 7feae6600640 1 Processor -- start 2024-01-15T18:02:47.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.645+0000 7feae6600640 1 -- start start 2024-01-15T18:02:47.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feae6600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feae015c0e0 0x7feae00cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feae6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015b7d0 0x7feae006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feae6600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feae015c4c0 0x7feae006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feae6600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae015cb00 con 0x7feae015c4c0 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feae6600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae00ccec0 con 0x7feae015c0e0 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feae6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae00cd1a0 con 0x7feae015b7d0 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feadfe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feae015c4c0 0x7feae006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feadfe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feae015c4c0 0x7feae006c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36826/0 (socket says 172.21.15.2:36826) 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.647+0000 7feadfe00640 1 -- 172.21.15.2:0/592511189 learned_addr learned my addr 172.21.15.2:0/592511189 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7feadf400640 1 --2- 172.21.15.2:0/592511189 >> v2:172.21.15.137:3300/0 conn(0x7feae015c0e0 0x7feae00cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7fead7e00640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015b7d0 0x7feae006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7feadfe00640 1 -- 172.21.15.2:0/592511189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015b7d0 msgr2=0x7feae006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:47.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7feadfe00640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015b7d0 0x7feae006c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7feadfe00640 1 -- 172.21.15.2:0/592511189 >> v2:172.21.15.137:3300/0 conn(0x7feae015c0e0 msgr2=0x7feae00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:47.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7feadfe00640 1 --2- 172.21.15.2:0/592511189 >> v2:172.21.15.137:3300/0 conn(0x7feae015c0e0 0x7feae00cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7feadfe00640 1 -- 172.21.15.2:0/592511189 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae00cd480 con 0x7feae015c4c0 2024-01-15T18:02:47.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.648+0000 7fead7e00640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015b7d0 0x7feae006c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:47.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.649+0000 7feadfe00640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015c4c0 0x7feae006c9b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7feae00725f0 tx=0x7fead802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ff532baf5f0c4d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feadea00640 1 -- 172.21.15.2:0/592511189 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead802ca70 con 0x7feae015c4c0 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feadea00640 1 -- 172.21.15.2:0/592511189 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead802cbd0 con 0x7feae015c4c0 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feadea00640 1 -- 172.21.15.2:0/592511189 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80343f0 con 0x7feae015c4c0 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 -- 172.21.15.2:0/592511189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015c4c0 msgr2=0x7feae006c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015c4c0 0x7feae006c9b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7feae00725f0 tx=0x7fead802bed0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 -- 172.21.15.2:0/592511189 shutdown_connections 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015c4c0 0x7feae006c9b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 --2- 172.21.15.2:0/592511189 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015b7d0 0x7feae006c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 --2- 172.21.15.2:0/592511189 >> v2:172.21.15.137:3300/0 conn(0x7feae015c0e0 0x7feae00cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.650+0000 7feae6600640 1 -- 172.21.15.2:0/592511189 >> 172.21.15.2:0/592511189 conn(0x7feae006da20 msgr2=0x7feae006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:47.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.651+0000 7feae6600640 1 -- 172.21.15.2:0/592511189 shutdown_connections 2024-01-15T18:02:47.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.651+0000 7feae6600640 1 -- 172.21.15.2:0/592511189 wait complete. 2024-01-15T18:02:47.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.652+0000 7feae6600640 1 Processor -- start 2024-01-15T18:02:47.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.652+0000 7feae6600640 1 -- start start 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.653+0000 7feae6600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 0x7feae013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.653+0000 7feae6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015c0e0 0x7feae0071df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.653+0000 7feae6600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feae015c4c0 0x7feae01485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.653+0000 7feae6600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00acd60 con 0x7feae015b7d0 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.653+0000 7feae6600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0070bf0 con 0x7feae015c4c0 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.653+0000 7feae6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae0070ed0 con 0x7feae015c0e0 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 0x7feae013eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 0x7feae013eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36842/0 (socket says 172.21.15.2:36842) 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 -- 172.21.15.2:0/2389938649 learned_addr learned my addr 172.21.15.2:0/2389938649 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7fead7e00640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015c0e0 0x7feae0071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadfe00640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feae015c4c0 0x7feae01485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 -- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015c0e0 msgr2=0x7feae0071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015c0e0 0x7feae0071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 -- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feae015c4c0 msgr2=0x7feae01485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feae015c4c0 0x7feae01485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadf400640 1 -- 172.21.15.2:0/2389938649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead8009b30 con 0x7feae015b7d0 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.654+0000 7feadfe00640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feae015c4c0 0x7feae01485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.655+0000 7feadf400640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 0x7feae013eb00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fead000d800 tx=0x7fead000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.655+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead000b6c0 con 0x7feae015b7d0 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.655+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead000b820 con 0x7feae015b7d0 2024-01-15T18:02:47.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.655+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead000bbb0 con 0x7feae015b7d0 2024-01-15T18:02:47.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.655+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae00ac9f0 con 0x7feae015b7d0 2024-01-15T18:02:47.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.655+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae01f3ed0 con 0x7feae015b7d0 2024-01-15T18:02:47.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.656+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae006d210 con 0x7feae015b7d0 2024-01-15T18:02:47.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.657+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fead000bd10 con 0x7feae015b7d0 2024-01-15T18:02:47.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.658+0000 7fead7400640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feab806b890 0x7feab806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:47.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.659+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fead008a940 con 0x7feae015b7d0 2024-01-15T18:02:47.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.659+0000 7fead7e00640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feab806b890 0x7feab806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:47.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.660+0000 7fead7e00640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feab806b890 0x7feab806dd50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feae0148220 tx=0x7feac8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:47.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.663+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fead00551c0 con 0x7feae015b7d0 2024-01-15T18:02:47.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.863+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feae00686f0 con 0x7feae015b7d0 2024-01-15T18:02:47.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.865+0000 7fead7400640 1 -- 172.21.15.2:0/2389938649 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+13011 (secure 0 0 0) 0x7fead00587e0 con 0x7feae015b7d0 2024-01-15T18:02:47.869 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:47.869 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":53,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:02:47.056364+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6833/4162570290":"2024-01-16T18:02:44.032475+0000","172.21.15.2:6832/4162570290":"2024-01-16T18:02:44.032475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:02:47.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.867+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feab806b890 msgr2=0x7feab806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.867+0000 7feae6600640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feab806b890 0x7feab806dd50 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7feae0148220 tx=0x7feac8008040 comp rx=0 tx=0).stop 2024-01-15T18:02:47.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.867+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 msgr2=0x7feae013eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.867+0000 7feae6600640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 0x7feae013eb00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fead000d800 tx=0x7fead000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 shutdown_connections 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feab806b890 0x7feab806dd50 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feae015c4c0 0x7feae01485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feae015c0e0 0x7feae0071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 --2- 172.21.15.2:0/2389938649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feae015b7d0 0x7feae013eb00 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 >> 172.21.15.2:0/2389938649 conn(0x7feae006da20 msgr2=0x7feae0150fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 shutdown_connections 2024-01-15T18:02:47.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:47.868+0000 7feae6600640 1 -- 172.21.15.2:0/2389938649 wait complete. 2024-01-15T18:02:47.891 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:02:47.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:02:48.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.114+0000 7ffabcc00640 1 Processor -- start 2024-01-15T18:02:48.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.115+0000 7ffabcc00640 1 -- start start 2024-01-15T18:02:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.116+0000 7ffabcc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffab80b2210 0x7ffab80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.116+0000 7ffabcc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80d1250 con 0x7ffab80d1c50 2024-01-15T18:02:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.116+0000 7ffabcc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffab80ce8e0 con 0x7ffab80d1870 2024-01-15T18:02:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.116+0000 7ffabcc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffab80ce6d0 con 0x7ffab80b2210 2024-01-15T18:02:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.116+0000 7ffab6a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffab80b2210 0x7ffab80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.117+0000 7ffab6a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffab80b2210 0x7ffab80d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36858/0 (socket says 172.21.15.2:36858) 2024-01-15T18:02:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.117+0000 7ffab6a00640 1 -- 172.21.15.2:0/131527795 learned_addr learned my addr 172.21.15.2:0/131527795 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.117+0000 7ffab6a00640 1 -- 172.21.15.2:0/131527795 >> v1:172.21.15.137:6789/0 conn(0x7ffab80d1870 legacy=0x7ffab815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.117+0000 7ffab6a00640 1 -- 172.21.15.2:0/131527795 >> v1:172.21.15.2:6789/0 conn(0x7ffab80d1c50 legacy=0x7ffab80b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.117+0000 7ffab6a00640 1 -- 172.21.15.2:0/131527795 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab80cebc0 con 0x7ffab80b2210 2024-01-15T18:02:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.118+0000 7ffab6a00640 1 --2- 172.21.15.2:0/131527795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80b2210 0x7ffab80d3bd0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009870 tx=0x7ffaa002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a5fa15d8f4eeea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffab6000640 1 -- 172.21.15.2:0/131527795 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0030850 con 0x7ffab80b2210 2024-01-15T18:02:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffab6000640 1 -- 172.21.15.2:0/131527795 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa00309b0 con 0x7ffab80b2210 2024-01-15T18:02:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffab6000640 1 -- 172.21.15.2:0/131527795 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0030d40 con 0x7ffab80b2210 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffabcc00640 1 -- 172.21.15.2:0/131527795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80b2210 msgr2=0x7ffab80d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffabcc00640 1 --2- 172.21.15.2:0/131527795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80b2210 0x7ffab80d3bd0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009870 tx=0x7ffaa002b990 comp rx=0 tx=0).stop 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffabcc00640 1 -- 172.21.15.2:0/131527795 shutdown_connections 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffabcc00640 1 --2- 172.21.15.2:0/131527795 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80b2210 0x7ffab80d3bd0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffabcc00640 1 -- 172.21.15.2:0/131527795 >> 172.21.15.2:0/131527795 conn(0x7ffab80d30b0 msgr2=0x7ffab80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.119+0000 7ffabcc00640 1 -- 172.21.15.2:0/131527795 shutdown_connections 2024-01-15T18:02:48.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.120+0000 7ffabcc00640 1 -- 172.21.15.2:0/131527795 wait complete. 2024-01-15T18:02:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.121+0000 7ffabcc00640 1 Processor -- start 2024-01-15T18:02:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.121+0000 7ffabcc00640 1 -- start start 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.122+0000 7ffabcc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffab80b2210 0x7ffab81534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.122+0000 7ffabcc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 0x7ffab809d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.122+0000 7ffabcc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffab80d1c50 0x7ffab809da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.122+0000 7ffabcc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab809df90 con 0x7ffab80d1870 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.122+0000 7ffabcc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab81575a0 con 0x7ffab80b2210 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.122+0000 7ffabcc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffab80a0410 con 0x7ffab80d1c50 2024-01-15T18:02:48.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 0x7ffab809d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 0x7ffab809d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36866/0 (socket says 172.21.15.2:36866) 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 -- 172.21.15.2:0/2449313926 learned_addr learned my addr 172.21.15.2:0/2449313926 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab7400640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffab80b2210 0x7ffab81534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab7e00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffab80d1c50 0x7ffab809da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 -- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffab80d1c50 msgr2=0x7ffab809da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffab80d1c50 0x7ffab809da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 -- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffab80b2210 msgr2=0x7ffab81534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffab80b2210 0x7ffab81534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab6a00640 1 -- 172.21.15.2:0/2449313926 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa00094d0 con 0x7ffab80d1870 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab7400640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffab80b2210 0x7ffab81534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.123+0000 7ffab7e00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffab80d1c50 0x7ffab809da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:48.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.124+0000 7ffab6a00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 0x7ffab809d510 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009870 tx=0x7ffaa0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.124+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa0030800 con 0x7ffab80d1870 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.124+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa0043400 con 0x7ffab80d1870 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.124+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab80a06f0 con 0x7ffab80d1870 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.124+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa00437b0 con 0x7ffab80d1870 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.124+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab80a0cb0 con 0x7ffab80d1870 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.125+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab8157710 con 0x7ffab80d1870 2024-01-15T18:02:48.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.126+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ffaa002e070 con 0x7ffab80d1870 2024-01-15T18:02:48.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.127+0000 7ffaaf400640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa9006b860 0x7ffa9006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.127+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7ffaa003e070 con 0x7ffab80d1870 2024-01-15T18:02:48.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.128+0000 7ffab7400640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa9006b860 0x7ffa9006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.129+0000 7ffab7400640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa9006b860 0x7ffa9006dd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8003eb0 tx=0x7ffaa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:48.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.132+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffaa0077c80 con 0x7ffab80d1870 2024-01-15T18:02:48.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.326+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffab8074770 con 0x7ffab80d1870 2024-01-15T18:02:48.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.327+0000 7ffaaf400640 1 -- 172.21.15.2:0/2449313926 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffaa007b2a0 con 0x7ffab80d1870 2024-01-15T18:02:48.330 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:02:48.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa9006b860 msgr2=0x7ffa9006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa9006b860 0x7ffa9006dd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffaa8003eb0 tx=0x7ffaa8009040 comp rx=0 tx=0).stop 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 msgr2=0x7ffab809d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 0x7ffab809d510 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7ffaa0009870 tx=0x7ffaa0004730 comp rx=0 tx=0).stop 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 shutdown_connections 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffa9006b860 0x7ffa9006dd20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffab80d1c50 0x7ffab809da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffab80d1870 0x7ffab809d510 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 --2- 172.21.15.2:0/2449313926 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffab80b2210 0x7ffab81534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 >> 172.21.15.2:0/2449313926 conn(0x7ffab80d30b0 msgr2=0x7ffab815cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 shutdown_connections 2024-01-15T18:02:48.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.330+0000 7ffabcc00640 1 -- 172.21.15.2:0/2449313926 wait complete. 2024-01-15T18:02:48.354 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:02:48.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.577+0000 7fe39dc00640 1 Processor -- start 2024-01-15T18:02:48.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.577+0000 7fe39dc00640 1 -- start start 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.578+0000 7fe39dc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe3980ce1d0 0x7fe39806c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.578+0000 7fe39dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe3980cba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.578+0000 7fe39dc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe398071340 con 0x7fe3980ccb80 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.578+0000 7fe39dc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39806c670 con 0x7fe3980ce1d0 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.578+0000 7fe39dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39815bd50 con 0x7fe3980cb620 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.579+0000 7fe38f200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe3980ccb80 0x7fe39815b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40882/0 (socket says 172.21.15.2:40882) 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.579+0000 7fe38f200640 1 -- 172.21.15.2:0/3654693313 learned_addr learned my addr 172.21.15.2:0/3654693313 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:48.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.579+0000 7fe397e00640 1 --2- 172.21.15.2:0/3654693313 >> v2:172.21.15.2:3300/0 conn(0x7fe3980ce1d0 0x7fe39806c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.579+0000 7fe39d200640 1 --2- 172.21.15.2:0/3654693313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe3980cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.579+0000 7fe397400640 1 -- 172.21.15.2:0/3654693313 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355058441 0 0) 0x7fe398071340 con 0x7fe3980ccb80 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.580+0000 7fe397400640 1 -- 172.21.15.2:0/3654693313 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe378003520 con 0x7fe3980ccb80 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.580+0000 7fe397e00640 1 -- 172.21.15.2:0/3654693313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 msgr2=0x7fe3980cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.580+0000 7fe397e00640 1 --2- 172.21.15.2:0/3654693313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe3980cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.580+0000 7fe397e00640 1 -- 172.21.15.2:0/3654693313 >> v1:172.21.15.2:6789/0 conn(0x7fe3980ccb80 legacy=0x7fe39815b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.580+0000 7fe397e00640 1 -- 172.21.15.2:0/3654693313 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980cbf90 con 0x7fe3980ce1d0 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.581+0000 7fe397e00640 1 --2- 172.21.15.2:0/3654693313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ce1d0 0x7fe39806c130 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe3880097a0 tx=0x7fe38802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94905cc0d133f6b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.581+0000 7fe397400640 1 -- 172.21.15.2:0/3654693313 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38802d040 con 0x7fe3980ce1d0 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.581+0000 7fe397400640 1 -- 172.21.15.2:0/3654693313 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe388031810 con 0x7fe3980ce1d0 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.581+0000 7fe397400640 1 -- 172.21.15.2:0/3654693313 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388031ba0 con 0x7fe3980ce1d0 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 -- 172.21.15.2:0/3654693313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ce1d0 msgr2=0x7fe39806c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 --2- 172.21.15.2:0/3654693313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ce1d0 0x7fe39806c130 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe3880097a0 tx=0x7fe38802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 -- 172.21.15.2:0/3654693313 shutdown_connections 2024-01-15T18:02:48.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 --2- 172.21.15.2:0/3654693313 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe3980cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 --2- 172.21.15.2:0/3654693313 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ce1d0 0x7fe39806c130 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 -- 172.21.15.2:0/3654693313 >> 172.21.15.2:0/3654693313 conn(0x7fe3980b0840 msgr2=0x7fe3980b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:48.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 -- 172.21.15.2:0/3654693313 shutdown_connections 2024-01-15T18:02:48.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.582+0000 7fe39dc00640 1 -- 172.21.15.2:0/3654693313 wait complete. 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.584+0000 7fe39dc00640 1 Processor -- start 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.584+0000 7fe39dc00640 1 -- start start 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe398144090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39dc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 0x7fe3980afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3980ce1d0 0x7fe3980b0030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39dc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39815aad0 con 0x7fe3980ccb80 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe398159a30 con 0x7fe3980ce1d0 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe398159d10 con 0x7fe3980cb620 2024-01-15T18:02:48.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 0x7fe3980afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 0x7fe3980afaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36882/0 (socket says 172.21.15.2:36882) 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 -- 172.21.15.2:0/2204569040 learned_addr learned my addr 172.21.15.2:0/2204569040 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe397e00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe398144090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39d200640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3980ce1d0 0x7fe3980b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 -- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 msgr2=0x7fe398144090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe398144090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 -- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3980ce1d0 msgr2=0x7fe3980b0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3980ce1d0 0x7fe3980b0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe38f200640 1 -- 172.21.15.2:0/2204569040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009400 con 0x7fe3980ccb80 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.585+0000 7fe39d200640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3980ce1d0 0x7fe3980b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:48.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.586+0000 7fe38f200640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 0x7fe3980afaf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c500 tx=0x7fe38000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.586+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380005ce0 con 0x7fe3980ccb80 2024-01-15T18:02:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.586+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe380005e40 con 0x7fe3980ccb80 2024-01-15T18:02:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.586+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe398159ff0 con 0x7fe3980ccb80 2024-01-15T18:02:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.586+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380007330 con 0x7fe3980ccb80 2024-01-15T18:02:48.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.586+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39815a5b0 con 0x7fe3980ccb80 2024-01-15T18:02:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.588+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39806bfa0 con 0x7fe3980ccb80 2024-01-15T18:02:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.588+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe380007550 con 0x7fe3980ccb80 2024-01-15T18:02:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.590+0000 7fe394c00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe37006b860 0x7fe37006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.590+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fe380089a10 con 0x7fe3980ccb80 2024-01-15T18:02:48.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.590+0000 7fe397e00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe37006b860 0x7fe37006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:48.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.591+0000 7fe397e00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe37006b860 0x7fe37006dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe388036000 tx=0x7fe38803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:48.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.593+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3800543e0 con 0x7fe3980ccb80 2024-01-15T18:02:48.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.800+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe39815a7a0 con 0x7fe3980ccb80 2024-01-15T18:02:48.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.801+0000 7fe394c00640 1 -- 172.21.15.2:0/2204569040 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v53) v1 ==== 80+0+120 (secure 0 0 0) 0x7fe380017080 con 0x7fe3980ccb80 2024-01-15T18:02:48.804 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6833/4162570290 2024-01-16T18:02:44.032475+0000 2024-01-15T18:02:48.804 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6832/4162570290 2024-01-16T18:02:44.032475+0000 2024-01-15T18:02:48.805 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:02:48.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe37006b860 msgr2=0x7fe37006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe37006b860 0x7fe37006dd20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe388036000 tx=0x7fe38803c040 comp rx=0 tx=0).stop 2024-01-15T18:02:48.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 msgr2=0x7fe3980afaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:48.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 0x7fe3980afaf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fe38000c500 tx=0x7fe38000c9c0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 shutdown_connections 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe37006b860 0x7fe37006dd20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3980ce1d0 0x7fe3980b0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3980ccb80 0x7fe3980afaf0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 --2- 172.21.15.2:0/2204569040 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3980cb620 0x7fe398144090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 >> 172.21.15.2:0/2204569040 conn(0x7fe3980b0840 msgr2=0x7fe3980aea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.804+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 shutdown_connections 2024-01-15T18:02:48.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:48.805+0000 7fe39dc00640 1 -- 172.21.15.2:0/2204569040 wait complete. 2024-01-15T18:02:48.826 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:02:48.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:48.827 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:02:48.925 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[52707]: fuse finished with error 0 and tester_r 0 2024-01-15T18:02:48.928 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:02:48.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:48.928 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:02:49.134 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:02:49.135 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:49.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:02:49.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.413+0000 7f035fe00640 1 Processor -- start 2024-01-15T18:02:49.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.413+0000 7f035fe00640 1 -- start start 2024-01-15T18:02:49.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0360141880 0x7f0360070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f036015f0e0 0x7f03600cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035fe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f036006dae0 con 0x7f036015ec60 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035fe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0360071f20 con 0x7f0360141880 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f036006c6b0 con 0x7f036015f0e0 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0360141880 0x7f0360070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035e000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f036015ec60 0x7f03600cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40892/0 (socket says 172.21.15.2:40892) 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.415+0000 7f035e000640 1 -- 172.21.15.2:0/198674027 learned_addr learned my addr 172.21.15.2:0/198674027 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035f400640 1 --2- 172.21.15.2:0/198674027 >> v2:172.21.15.137:3300/0 conn(0x7f036015f0e0 0x7f03600cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035d600640 1 -- 172.21.15.2:0/198674027 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144801556 0 0) 0x7f036006dae0 con 0x7f036015ec60 2024-01-15T18:02:49.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035d600640 1 -- 172.21.15.2:0/198674027 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0340003520 con 0x7f036015ec60 2024-01-15T18:02:49.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035ea00640 1 -- 172.21.15.2:0/198674027 >> v2:172.21.15.137:3300/0 conn(0x7f036015f0e0 msgr2=0x7f03600cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:49.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035ea00640 1 --2- 172.21.15.2:0/198674027 >> v2:172.21.15.137:3300/0 conn(0x7f036015f0e0 0x7f03600cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035ea00640 1 -- 172.21.15.2:0/198674027 >> v1:172.21.15.2:6789/0 conn(0x7f036015ec60 legacy=0x7f03600cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035ea00640 1 -- 172.21.15.2:0/198674027 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0360071960 con 0x7f0360141880 2024-01-15T18:02:49.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.416+0000 7f035f400640 1 --2- 172.21.15.2:0/198674027 >> v2:172.21.15.137:3300/0 conn(0x7f036015f0e0 0x7f03600cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.417+0000 7f035ea00640 1 --2- 172.21.15.2:0/198674027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f0360070fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0360070060 tx=0x7f035002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a0bae34c1873fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.418+0000 7f035d600640 1 -- 172.21.15.2:0/198674027 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035002cbf0 con 0x7f0360141880 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.418+0000 7f035d600640 1 -- 172.21.15.2:0/198674027 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f035002cd50 con 0x7f0360141880 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.418+0000 7f035d600640 1 -- 172.21.15.2:0/198674027 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350034660 con 0x7f0360141880 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.418+0000 7f035fe00640 1 -- 172.21.15.2:0/198674027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 msgr2=0x7f0360070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.418+0000 7f035fe00640 1 --2- 172.21.15.2:0/198674027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f0360070fc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f0360070060 tx=0x7f035002be40 comp rx=0 tx=0).stop 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.419+0000 7f035fe00640 1 -- 172.21.15.2:0/198674027 shutdown_connections 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.419+0000 7f035fe00640 1 --2- 172.21.15.2:0/198674027 >> v2:172.21.15.137:3300/0 conn(0x7f036015f0e0 0x7f03600cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.419+0000 7f035fe00640 1 --2- 172.21.15.2:0/198674027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f0360070fc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.419+0000 7f035fe00640 1 -- 172.21.15.2:0/198674027 >> 172.21.15.2:0/198674027 conn(0x7f03600b0840 msgr2=0x7f03600b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:49.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.419+0000 7f035fe00640 1 -- 172.21.15.2:0/198674027 shutdown_connections 2024-01-15T18:02:49.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.419+0000 7f035fe00640 1 -- 172.21.15.2:0/198674027 wait complete. 2024-01-15T18:02:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.421+0000 7f035fe00640 1 Processor -- start 2024-01-15T18:02:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.421+0000 7f035fe00640 1 -- start start 2024-01-15T18:02:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f03600c6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036015ec60 0x7f03600c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f036015f0e0 0x7f03600c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03600c7fd0 con 0x7f0360141880 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03600c7400 con 0x7f036015ec60 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03600c76a0 con 0x7f036015f0e0 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f03600c6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f03600c6b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36896/0 (socket says 172.21.15.2:36896) 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035ea00640 1 -- 172.21.15.2:0/2921115806 learned_addr learned my addr 172.21.15.2:0/2921115806 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.422+0000 7f035e000640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036015ec60 0x7f03600c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035ea00640 1 -- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f036015f0e0 msgr2=0x7f03600c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035f400640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f036015f0e0 0x7f03600c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035ea00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f036015f0e0 0x7f03600c8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035ea00640 1 -- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036015ec60 msgr2=0x7f03600c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035ea00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036015ec60 0x7f03600c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035ea00640 1 -- 172.21.15.2:0/2921115806 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0350009a70 con 0x7f0360141880 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035ea00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f03600c6b30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0350037040 tx=0x7f0350004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350040650 con 0x7f0360141880 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03500407b0 con 0x7f0360141880 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0350040b60 con 0x7f0360141880 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03600c7980 con 0x7f0360141880 2024-01-15T18:02:49.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.423+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03600c4a00 con 0x7f0360141880 2024-01-15T18:02:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.425+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03600c4b90 con 0x7f0360141880 2024-01-15T18:02:49.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.426+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f0350040cc0 con 0x7f0360141880 2024-01-15T18:02:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.426+0000 7f0356a00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f033806b860 0x7f033806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.427+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f03500ad160 con 0x7f0360141880 2024-01-15T18:02:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.427+0000 7f035e000640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f033806b860 0x7f033806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.428+0000 7f035e000640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f033806b860 0x7f033806dd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03600c5670 tx=0x7f0348009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:49.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.431+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0350077b90 con 0x7f0360141880 2024-01-15T18:02:49.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.633+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0360074770 con 0x7f0360141880 2024-01-15T18:02:49.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.636+0000 7f0356a00640 1 -- 172.21.15.2:0/2921115806 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+13011 (secure 0 0 0) 0x7f035007b1b0 con 0x7f0360141880 2024-01-15T18:02:49.639 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:49.640 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":53,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:02:47.056364+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6833/4162570290":"2024-01-16T18:02:44.032475+0000","172.21.15.2:6832/4162570290":"2024-01-16T18:02:44.032475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:02:49.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.638+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f033806b860 msgr2=0x7f033806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.638+0000 7f035fe00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f033806b860 0x7f033806dd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f03600c5670 tx=0x7f0348009040 comp rx=0 tx=0).stop 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.638+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 msgr2=0x7f03600c6b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.638+0000 7f035fe00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f03600c6b30 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f0350037040 tx=0x7f0350004a40 comp rx=0 tx=0).stop 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 shutdown_connections 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f033806b860 0x7f033806dd20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f036015f0e0 0x7f03600c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f036015ec60 0x7f03600c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 --2- 172.21.15.2:0/2921115806 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0360141880 0x7f03600c6b30 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 >> 172.21.15.2:0/2921115806 conn(0x7f03600b0840 msgr2=0x7f03600ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:49.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 shutdown_connections 2024-01-15T18:02:49.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.639+0000 7f035fe00640 1 -- 172.21.15.2:0/2921115806 wait complete. 2024-01-15T18:02:49.664 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:02:49.664 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:02:49.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.890+0000 7f1950e00640 1 Processor -- start 2024-01-15T18:02:49.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.890+0000 7f1950e00640 1 -- start start 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.891+0000 7f1950e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 0x7f194c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.891+0000 7f1950e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f194c0d1c50 0x7f194c0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f1950e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194c0d1250 con 0x7f194c0d1870 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f1950e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c0ce6d0 con 0x7f194c0d1c50 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f1950e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0ce8e0 con 0x7f194c0b2210 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f194b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f194c0d1870 0x7f194c15d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40894/0 (socket says 172.21.15.2:40894) 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f194b400640 1 -- 172.21.15.2:0/2404333692 learned_addr learned my addr 172.21.15.2:0/2404333692 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:49.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f194aa00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 0x7f194c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f194be00640 1 --2- 172.21.15.2:0/2404333692 >> v2:172.21.15.137:3300/0 conn(0x7f194c0d1c50 0x7f194c0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 519724757 0 0) 0x7f194c0d1250 con 0x7f194c0d1870 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.892+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192c003520 con 0x7f194c0d1870 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 612446411 0 0) 0x7f192c003520 con 0x7f194c0d1870 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 msgr2=0x7f194c0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 0x7f194c0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 >> v2:172.21.15.137:3300/0 conn(0x7f194c0d1c50 msgr2=0x7f194c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 --2- 172.21.15.2:0/2404333692 >> v2:172.21.15.137:3300/0 conn(0x7f194c0d1c50 0x7f194c0b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c0cebc0 con 0x7f194c0d1870 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f193c001670 con 0x7f194c0d1870 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 >> v1:172.21.15.2:6789/0 conn(0x7f194c0d1870 legacy=0x7f194c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 0x7f192c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c003ef0 0x7f192c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f192c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0ce6d0 con 0x7f192c004d00 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0ce8e0 con 0x7f192c003ef0 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c00b980 con 0x7f194c0b2210 2024-01-15T18:02:49.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.893+0000 7f194be00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f192c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194aa00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c003ef0 0x7f192c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194b400640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 0x7f192c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194be00640 1 -- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 msgr2=0x7f192c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194be00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c0b2210 0x7f192c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194be00640 1 -- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c003ef0 msgr2=0x7f192c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194be00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f192c003ef0 0x7f192c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.894+0000 7f194be00640 1 -- 172.21.15.2:0/2404333692 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c003520 con 0x7f192c004d00 2024-01-15T18:02:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.895+0000 7f194be00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f192c005180 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1944009f90 tx=0x7f192c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bc804b602513ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:49.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.895+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194402cc00 con 0x7f192c004d00 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.895+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194400f040 con 0x7f192c004d00 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.895+0000 7f194a000640 1 -- 172.21.15.2:0/2404333692 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944034440 con 0x7f192c004d00 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f1950e00640 1 -- 172.21.15.2:0/2404333692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 msgr2=0x7f192c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f1950e00640 1 --2- 172.21.15.2:0/2404333692 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f192c005180 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f1944009f90 tx=0x7f192c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f194be00640 1 -- 172.21.15.2:0/2404333692 reap_dead start 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f1950e00640 1 -- 172.21.15.2:0/2404333692 shutdown_connections 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f1950e00640 1 -- 172.21.15.2:0/2404333692 >> 172.21.15.2:0/2404333692 conn(0x7f194c0d30b0 msgr2=0x7f194c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f1950e00640 1 -- 172.21.15.2:0/2404333692 shutdown_connections 2024-01-15T18:02:49.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.896+0000 7f1950e00640 1 -- 172.21.15.2:0/2404333692 wait complete. 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.898+0000 7f1950e00640 1 Processor -- start 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.898+0000 7f1950e00640 1 -- start start 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.899+0000 7f1950e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f194c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.899+0000 7f1950e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f194c1e7210 0x7f194c160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.899+0000 7f1950e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c160560 0x7f194c1e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.899+0000 7f1950e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0c71d0 con 0x7f192c004d00 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.899+0000 7f1950e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c15ec50 con 0x7f194c1e7210 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.899+0000 7f1950e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0c6be0 con 0x7f194c160560 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f194c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f194c15e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36926/0 (socket says 172.21.15.2:36926) 2024-01-15T18:02:49.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 -- 172.21.15.2:0/1447305088 learned_addr learned my addr 172.21.15.2:0/1447305088 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194aa00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f194c1e7210 0x7f194c160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194be00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c160560 0x7f194c1e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 -- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c160560 msgr2=0x7f194c1e5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c160560 0x7f194c1e5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 -- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f194c1e7210 msgr2=0x7f194c160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f194c1e7210 0x7f194c160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 -- 172.21.15.2:0/1447305088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c160a00 con 0x7f192c004d00 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194be00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c160560 0x7f194c1e5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:49.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.900+0000 7f194b400640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f194c15e710 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f193c00e8f0 tx=0x7f193c00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:49.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.901+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c017070 con 0x7f192c004d00 2024-01-15T18:02:49.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.901+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c00c8d0 con 0x7f192c004d00 2024-01-15T18:02:49.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.901+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c00ca60 con 0x7f192c004d00 2024-01-15T18:02:49.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.901+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c160ce0 con 0x7f192c004d00 2024-01-15T18:02:49.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.901+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c1e63c0 con 0x7f192c004d00 2024-01-15T18:02:49.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.902+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c074770 con 0x7f192c004d00 2024-01-15T18:02:49.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.903+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f193c00cbc0 con 0x7f192c004d00 2024-01-15T18:02:49.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.904+0000 7f1943400640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f192406b860 0x7f192406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:49.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.904+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f193c089c10 con 0x7f192c004d00 2024-01-15T18:02:49.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.905+0000 7f194aa00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f192406b860 0x7f192406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:49.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.906+0000 7f194aa00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f192406b860 0x7f192406dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f194c06cd80 tx=0x7f1934008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:49.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:49.908+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f193c054490 con 0x7f192c004d00 2024-01-15T18:02:50.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.103+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f194c0d2e90 con 0x7f192c004d00 2024-01-15T18:02:50.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.104+0000 7f1943400640 1 -- 172.21.15.2:0/1447305088 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7f193c057ab0 con 0x7f192c004d00 2024-01-15T18:02:50.107 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:02:50.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.106+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f192406b860 msgr2=0x7f192406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.106+0000 7f1950e00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f192406b860 0x7f192406dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f194c06cd80 tx=0x7f1934008040 comp rx=0 tx=0).stop 2024-01-15T18:02:50.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.106+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 msgr2=0x7f194c15e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.106+0000 7f1950e00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f194c15e710 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f193c00e8f0 tx=0x7f193c00edb0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 shutdown_connections 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f192406b860 0x7f192406dd20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f194c160560 0x7f194c1e5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f194c1e7210 0x7f194c160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 --2- 172.21.15.2:0/1447305088 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f192c004d00 0x7f194c15e710 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 >> 172.21.15.2:0/1447305088 conn(0x7f194c0d30b0 msgr2=0x7f194c0cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 shutdown_connections 2024-01-15T18:02:50.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.107+0000 7f1950e00640 1 -- 172.21.15.2:0/1447305088 wait complete. 2024-01-15T18:02:50.129 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:02:50.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.340+0000 7ff81aa00640 1 Processor -- start 2024-01-15T18:02:50.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.341+0000 7ff81aa00640 1 -- start start 2024-01-15T18:02:50.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.342+0000 7ff81aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff8140d2de0 0x7ff8140d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:50.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.342+0000 7ff81aa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff8140d0ed0 0x7ff8140d1340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:50.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.342+0000 7ff81aa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8140b30e0 con 0x7ff81415d200 2024-01-15T18:02:50.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.342+0000 7ff81aa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8140d3780 con 0x7ff8140d2de0 2024-01-15T18:02:50.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.342+0000 7ff81aa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8140d1ca0 con 0x7ff8140d0ed0 2024-01-15T18:02:50.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff8140d2de0 0x7ff8140d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:50.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff8140d2de0 0x7ff8140d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36934/0 (socket says 172.21.15.2:36934) 2024-01-15T18:02:50.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 -- 172.21.15.2:0/2061461991 learned_addr learned my addr 172.21.15.2:0/2061461991 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:50.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 -- 172.21.15.2:0/2061461991 >> v2:172.21.15.137:3300/0 conn(0x7ff8140d0ed0 msgr2=0x7ff8140d1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:50.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff81a000640 1 --2- 172.21.15.2:0/2061461991 >> v2:172.21.15.137:3300/0 conn(0x7ff8140d0ed0 0x7ff8140d1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:50.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 --2- 172.21.15.2:0/2061461991 >> v2:172.21.15.137:3300/0 conn(0x7ff8140d0ed0 0x7ff8140d1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 -- 172.21.15.2:0/2061461991 >> v1:172.21.15.137:6789/0 conn(0x7ff81415d200 legacy=0x7ff8140cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:50.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff818c00640 1 -- 172.21.15.2:0/2061461991 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8140d1f80 con 0x7ff8140d2de0 2024-01-15T18:02:50.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.343+0000 7ff81a000640 1 --2- 172.21.15.2:0/2061461991 >> v2:172.21.15.137:3300/0 conn(0x7ff8140d0ed0 0x7ff8140d1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:50.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.344+0000 7ff818c00640 1 --2- 172.21.15.2:0/2061461991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff8140d3240 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff81408c370 tx=0x7ff7fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8696c91e908313ea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:50.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff813e00640 1 -- 172.21.15.2:0/2061461991 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc031870 con 0x7ff8140d2de0 2024-01-15T18:02:50.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff813e00640 1 -- 172.21.15.2:0/2061461991 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc0319d0 con 0x7ff8140d2de0 2024-01-15T18:02:50.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff813e00640 1 -- 172.21.15.2:0/2061461991 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc031d60 con 0x7ff8140d2de0 2024-01-15T18:02:50.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 -- 172.21.15.2:0/2061461991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 msgr2=0x7ff8140d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2061461991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff8140d3240 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff81408c370 tx=0x7ff7fc02b910 comp rx=0 tx=0).stop 2024-01-15T18:02:50.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 -- 172.21.15.2:0/2061461991 shutdown_connections 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2061461991 >> v2:172.21.15.137:3300/0 conn(0x7ff8140d0ed0 0x7ff8140d1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2061461991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff8140d3240 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 -- 172.21.15.2:0/2061461991 >> 172.21.15.2:0/2061461991 conn(0x7ff81408caf0 msgr2=0x7ff81409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.345+0000 7ff81aa00640 1 -- 172.21.15.2:0/2061461991 shutdown_connections 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.346+0000 7ff81aa00640 1 -- 172.21.15.2:0/2061461991 wait complete. 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.347+0000 7ff81aa00640 1 Processor -- start 2024-01-15T18:02:50.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.347+0000 7ff81aa00640 1 -- start start 2024-01-15T18:02:50.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.348+0000 7ff81aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8140d0ed0 0x7ff814157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.348+0000 7ff81aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff81415f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.348+0000 7ff81aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81415d200 0x7ff8140cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.348+0000 7ff81aa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140c6a00 con 0x7ff8140d2de0 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.348+0000 7ff81aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140c6c70 con 0x7ff8140d0ed0 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.348+0000 7ff81aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8140c6f50 con 0x7ff81415d200 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff81415f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff81415f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36938/0 (socket says 172.21.15.2:36938) 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 -- 172.21.15.2:0/2891192607 learned_addr learned my addr 172.21.15.2:0/2891192607 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:50.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff819600640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8140d0ed0 0x7ff814157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 -- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81415d200 msgr2=0x7ff8140cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81415d200 0x7ff8140cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 -- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8140d0ed0 msgr2=0x7ff814157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8140d0ed0 0x7ff814157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 -- 172.21.15.2:0/2891192607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc0094d0 con 0x7ff8140d2de0 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff819600640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8140d0ed0 0x7ff814157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.349+0000 7ff818c00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff81415f8d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc000c00 tx=0x7ff7fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.350+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc031840 con 0x7ff8140d2de0 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.350+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc0443f0 con 0x7ff8140d2de0 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.350+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8140c7230 con 0x7ff8140d2de0 2024-01-15T18:02:50.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.350+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc0447a0 con 0x7ff8140d2de0 2024-01-15T18:02:50.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.350+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff814160390 con 0x7ff8140d2de0 2024-01-15T18:02:50.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.351+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8140c7440 con 0x7ff8140d2de0 2024-01-15T18:02:50.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.352+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ff7fc04d450 con 0x7ff8140d2de0 2024-01-15T18:02:50.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.352+0000 7ff811600640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7ec06b860 0x7ff7ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:50.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.353+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7ff7fc02d040 con 0x7ff8140d2de0 2024-01-15T18:02:50.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.353+0000 7ff819600640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7ec06b860 0x7ff7ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:50.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.354+0000 7ff819600640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7ec06b860 0x7ff7ec06dd20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff81415cac0 tx=0x7ff804009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:50.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.357+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7fc077440 con 0x7ff8140d2de0 2024-01-15T18:02:50.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.563+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff814074770 con 0x7ff8140d2de0 2024-01-15T18:02:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.564+0000 7ff811600640 1 -- 172.21.15.2:0/2891192607 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v53) v1 ==== 80+0+120 (secure 0 0 0) 0x7ff7fc07aa60 con 0x7ff8140d2de0 2024-01-15T18:02:50.567 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6833/4162570290 2024-01-16T18:02:44.032475+0000 2024-01-15T18:02:50.568 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6832/4162570290 2024-01-16T18:02:44.032475+0000 2024-01-15T18:02:50.568 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:02:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.566+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7ec06b860 msgr2=0x7ff7ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.566+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7ec06b860 0x7ff7ec06dd20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff81415cac0 tx=0x7ff804009040 comp rx=0 tx=0).stop 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.566+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 msgr2=0x7ff81415f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.566+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff81415f8d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc000c00 tx=0x7ff7fc0046c0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 shutdown_connections 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7ec06b860 0x7ff7ec06dd20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff81415d200 0x7ff8140cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff8140d2de0 0x7ff81415f8d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 --2- 172.21.15.2:0/2891192607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8140d0ed0 0x7ff814157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 >> 172.21.15.2:0/2891192607 conn(0x7ff81408caf0 msgr2=0x7ff8140afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 shutdown_connections 2024-01-15T18:02:50.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:50.567+0000 7ff81aa00640 1 -- 172.21.15.2:0/2891192607 wait complete. 2024-01-15T18:02:50.591 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:02:50.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:50.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:02:50.686 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[43390]: fuse finished with error 0 and tester_r 0 2024-01-15T18:02:50.690 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:02:50.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:50.690 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:02:50.874 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:02:50.874 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:50.935 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:02:50.935 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:02:50.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:50.935 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:02:50.990 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:02:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:02:50.992 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:02:50.992 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:51.049 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:02:51.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:02:51.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:02:51.049 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:02:51.100 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:02:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:02:51.101 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:02:51.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.324+0000 7fdfebe00640 1 Processor -- start 2024-01-15T18:02:51.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.325+0000 7fdfebe00640 1 -- start start 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.326+0000 7fdfebe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdfec15d340 0x7fdfec0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.326+0000 7fdfebe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfec0b3de0 con 0x7fdfec0d3090 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.326+0000 7fdfebe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfec0b40c0 con 0x7fdfec0d1d90 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.326+0000 7fdfebe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfec0d0e80 con 0x7fdfec15d340 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.326+0000 7fdfeaa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdfec15d340 0x7fdfec0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.327+0000 7fdfeaa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdfec15d340 0x7fdfec0cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36954/0 (socket says 172.21.15.2:36954) 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.327+0000 7fdfeaa00640 1 -- 172.21.15.2:0/375894455 learned_addr learned my addr 172.21.15.2:0/375894455 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:51.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.327+0000 7fdfeaa00640 1 -- 172.21.15.2:0/375894455 >> v1:172.21.15.137:6789/0 conn(0x7fdfec0d1d90 legacy=0x7fdfec0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:51.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.327+0000 7fdfeaa00640 1 -- 172.21.15.2:0/375894455 >> v1:172.21.15.2:6789/0 conn(0x7fdfec0d3090 legacy=0x7fdfec0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.327+0000 7fdfeaa00640 1 -- 172.21.15.2:0/375894455 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfec0d1160 con 0x7fdfec15d340 2024-01-15T18:02:51.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.327+0000 7fdfe9600640 1 -- 172.21.15.2:0/375894455 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012641902 0 0) 0x7fdfec0b3de0 con 0x7fdfec0d3090 2024-01-15T18:02:51.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.328+0000 7fdfeaa00640 1 --2- 172.21.15.2:0/375894455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0cdd20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009870 tx=0x7fdfdc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7a2cb14c2fdf129 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:51.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.329+0000 7fdfe9600640 1 -- 172.21.15.2:0/375894455 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc030850 con 0x7fdfec15d340 2024-01-15T18:02:51.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.329+0000 7fdfe9600640 1 -- 172.21.15.2:0/375894455 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc0309b0 con 0x7fdfec15d340 2024-01-15T18:02:51.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.329+0000 7fdfe9600640 1 -- 172.21.15.2:0/375894455 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc030d40 con 0x7fdfec15d340 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.329+0000 7fdfebe00640 1 -- 172.21.15.2:0/375894455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 msgr2=0x7fdfec0cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.329+0000 7fdfebe00640 1 --2- 172.21.15.2:0/375894455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0cdd20 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009870 tx=0x7fdfdc02b990 comp rx=0 tx=0).stop 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.330+0000 7fdfebe00640 1 -- 172.21.15.2:0/375894455 shutdown_connections 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.330+0000 7fdfebe00640 1 --2- 172.21.15.2:0/375894455 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0cdd20 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.330+0000 7fdfebe00640 1 -- 172.21.15.2:0/375894455 >> 172.21.15.2:0/375894455 conn(0x7fdfec08caf0 msgr2=0x7fdfec09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.330+0000 7fdfebe00640 1 -- 172.21.15.2:0/375894455 shutdown_connections 2024-01-15T18:02:51.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.330+0000 7fdfebe00640 1 -- 172.21.15.2:0/375894455 wait complete. 2024-01-15T18:02:51.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.332+0000 7fdfebe00640 1 Processor -- start 2024-01-15T18:02:51.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.332+0000 7fdfebe00640 1 -- start start 2024-01-15T18:02:51.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfec0d1d90 0x7fdfec158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfebe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfec0d3090 0x7fdfec1f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0b4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfebe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec0c8a10 con 0x7fdfec15d340 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfebe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec1f3270 con 0x7fdfec0d3090 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfebe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfec0c7e00 con 0x7fdfec0d1d90 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfeb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0b4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfeb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0b4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36970/0 (socket says 172.21.15.2:36970) 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.333+0000 7fdfeb400640 1 -- 172.21.15.2:0/436211719 learned_addr learned my addr 172.21.15.2:0/436211719 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfea000640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfec0d3090 0x7fdfec1f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeaa00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfec0d1d90 0x7fdfec158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeb400640 1 -- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfec0d1d90 msgr2=0x7fdfec158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeb400640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfec0d1d90 0x7fdfec158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeb400640 1 -- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfec0d3090 msgr2=0x7fdfec1f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeb400640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfec0d3090 0x7fdfec1f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeb400640 1 -- 172.21.15.2:0/436211719 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc0094d0 con 0x7fdfec15d340 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfeb400640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0b4c30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400c390 tx=0x7fdfe400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.334+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe40035f0 con 0x7fdfec15d340 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.335+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe4003750 con 0x7fdfec15d340 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.335+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfec0c7b20 con 0x7fdfec15d340 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.335+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe4003b00 con 0x7fdfec15d340 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.335+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfec0c8630 con 0x7fdfec15d340 2024-01-15T18:02:51.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.336+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfec0c8b50 con 0x7fdfec15d340 2024-01-15T18:02:51.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.337+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fdfe4003c60 con 0x7fdfec15d340 2024-01-15T18:02:51.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.338+0000 7fdfe3400640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfc406b890 0x7fdfc406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.339+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fdfe4089900 con 0x7fdfec15d340 2024-01-15T18:02:51.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.339+0000 7fdfeaa00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfc406b890 0x7fdfc406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.340+0000 7fdfeaa00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfc406b890 0x7fdfc406dd50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc036000 tx=0x7fdfdc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:51.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.343+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdfec0c8b50 con 0x7fdfec15d340 2024-01-15T18:02:51.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.578+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfec0c8b50 con 0x7fdfec15d340 2024-01-15T18:02:51.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.580+0000 7fdfe3400640 1 -- 172.21.15.2:0/436211719 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fdfec0c8b50 con 0x7fdfec15d340 2024-01-15T18:02:51.584 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:51.584 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1873476051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1873476051},{"type":"v1","addr":"172.21.15.2:6835","nonce":1873476051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1702488249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1702488249},{"type":"v1","addr":"172.21.15.137:6836","nonce":1702488249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5251,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/4131491202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":4131491202},{"type":"v1","addr":"172.21.15.137:6837","nonce":4131491202}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6165,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/3020321570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3020321570},{"type":"v1","addr":"172.21.15.2:6833","nonce":3020321570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2024-01-15T18:02:51.585 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 29 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfc406b890 msgr2=0x7fdfc406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfc406b890 0x7fdfc406dd50 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc036000 tx=0x7fdfdc03b040 comp rx=0 tx=0).stop 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 msgr2=0x7fdfec0b4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0b4c30 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdfe400c390 tx=0x7fdfe400c850 comp rx=0 tx=0).stop 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 shutdown_connections 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfc406b890 0x7fdfc406dd50 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.582+0000 7fdfebe00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdfec15d340 0x7fdfec0b4c30 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.583+0000 7fdfebe00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdfec0d3090 0x7fdfec1f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.583+0000 7fdfebe00640 1 --2- 172.21.15.2:0/436211719 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdfec0d1d90 0x7fdfec158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.583+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 >> 172.21.15.2:0/436211719 conn(0x7fdfec08caf0 msgr2=0x7fdfec0b21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:51.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.583+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 shutdown_connections 2024-01-15T18:02:51.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.583+0000 7fdfebe00640 1 -- 172.21.15.2:0/436211719 wait complete. 2024-01-15T18:02:51.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-15T18:02:51.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.815+0000 7fdf5b600640 1 Processor -- start 2024-01-15T18:02:51.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.815+0000 7fdf5b600640 1 -- start start 2024-01-15T18:02:51.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.817+0000 7fdf5b600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf54144c00 0x7fdf540cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.817+0000 7fdf5b600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdf5406d000 0x7fdf54070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.817+0000 7fdf5b600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540ce230 con 0x7fdf540cdd70 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.817+0000 7fdf5b600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540ce510 con 0x7fdf54144c00 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.817+0000 7fdf5b600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf540ce7f0 con 0x7fdf5406d000 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf54144c00 0x7fdf540cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdf54144c00 0x7fdf540cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36978/0 (socket says 172.21.15.2:36978) 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 -- 172.21.15.2:0/715625746 learned_addr learned my addr 172.21.15.2:0/715625746 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf59600640 1 --2- 172.21.15.2:0/715625746 >> v2:172.21.15.137:3300/0 conn(0x7fdf5406d000 0x7fdf54070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 -- 172.21.15.2:0/715625746 >> v2:172.21.15.137:3300/0 conn(0x7fdf5406d000 msgr2=0x7fdf54070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 --2- 172.21.15.2:0/715625746 >> v2:172.21.15.137:3300/0 conn(0x7fdf5406d000 0x7fdf54070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 -- 172.21.15.2:0/715625746 >> v1:172.21.15.137:6789/0 conn(0x7fdf540cdd70 legacy=0x7fdf5406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:51.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.818+0000 7fdf58c00640 1 -- 172.21.15.2:0/715625746 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540705f0 con 0x7fdf54144c00 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.819+0000 7fdf58c00640 1 --2- 172.21.15.2:0/715625746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf540cd830 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdf5415ec80 tx=0x7fdf4402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adb98c702a951e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.820+0000 7fdf53400640 1 -- 172.21.15.2:0/715625746 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4402ca80 con 0x7fdf54144c00 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.820+0000 7fdf53400640 1 -- 172.21.15.2:0/715625746 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4402cbe0 con 0x7fdf54144c00 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.820+0000 7fdf53400640 1 -- 172.21.15.2:0/715625746 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf440343f0 con 0x7fdf54144c00 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 -- 172.21.15.2:0/715625746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 msgr2=0x7fdf540cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 --2- 172.21.15.2:0/715625746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf540cd830 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fdf5415ec80 tx=0x7fdf4402bee0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 -- 172.21.15.2:0/715625746 shutdown_connections 2024-01-15T18:02:51.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 --2- 172.21.15.2:0/715625746 >> v2:172.21.15.137:3300/0 conn(0x7fdf5406d000 0x7fdf54070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 --2- 172.21.15.2:0/715625746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf540cd830 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 -- 172.21.15.2:0/715625746 >> 172.21.15.2:0/715625746 conn(0x7fdf540b45f0 msgr2=0x7fdf540b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:51.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 -- 172.21.15.2:0/715625746 shutdown_connections 2024-01-15T18:02:51.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.821+0000 7fdf5b600640 1 -- 172.21.15.2:0/715625746 wait complete. 2024-01-15T18:02:51.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.823+0000 7fdf5b600640 1 Processor -- start 2024-01-15T18:02:51.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.823+0000 7fdf5b600640 1 -- start start 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf5b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf5406d000 0x7fdf5413b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf5b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf540cdd70 0x7fdf54155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf5b600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf541f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf5b600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf540a9160 con 0x7fdf54144c00 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf5b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf541f47d0 con 0x7fdf540cdd70 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf5b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf541f4ab0 con 0x7fdf5406d000 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf541f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf541f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36992/0 (socket says 172.21.15.2:36992) 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf59600640 1 -- 172.21.15.2:0/2288643051 learned_addr learned my addr 172.21.15.2:0/2288643051 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf58c00640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf5406d000 0x7fdf5413b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.824+0000 7fdf53e00640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf540cdd70 0x7fdf54155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf59600640 1 -- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf5406d000 msgr2=0x7fdf5413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf59600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf5406d000 0x7fdf5413b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf59600640 1 -- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf540cdd70 msgr2=0x7fdf54155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf59600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf540cdd70 0x7fdf54155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf59600640 1 -- 172.21.15.2:0/2288643051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf44009b30 con 0x7fdf54144c00 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf53e00640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf540cdd70 0x7fdf54155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf59600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf541f5870 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00d5d0 tx=0x7fdf4c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c0125f0 con 0x7fdf54144c00 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4c012750 con 0x7fdf54144c00 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c012b00 con 0x7fdf54144c00 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf541f4d90 con 0x7fdf54144c00 2024-01-15T18:02:51.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.825+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf541f5350 con 0x7fdf54144c00 2024-01-15T18:02:51.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.827+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf540686f0 con 0x7fdf54144c00 2024-01-15T18:02:51.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.828+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fdf4c012c60 con 0x7fdf54144c00 2024-01-15T18:02:51.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.829+0000 7fdf50c00640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf2c06b890 0x7fdf2c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:51.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.829+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fdf4c08a8f0 con 0x7fdf54144c00 2024-01-15T18:02:51.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.829+0000 7fdf58c00640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf2c06b890 0x7fdf2c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:51.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.830+0000 7fdf58c00640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf2c06b890 0x7fdf2c06dd50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf44035000 tx=0x7fdf44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:51.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:51.833+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf4c055170 con 0x7fdf54144c00 2024-01-15T18:02:52.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.032+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fdf5415b290 con 0x7fdf54144c00 2024-01-15T18:02:52.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.038+0000 7fdf50c00640 1 -- 172.21.15.2:0/2288643051 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v17) v1 ==== 115+0+0 (secure 0 0 0) 0x7fdf4c058790 con 0x7fdf54144c00 2024-01-15T18:02:52.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf2c06b890 msgr2=0x7fdf2c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf2c06b890 0x7fdf2c06dd50 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fdf44035000 tx=0x7fdf44030000 comp rx=0 tx=0).stop 2024-01-15T18:02:52.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 msgr2=0x7fdf541f5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf541f5870 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00d5d0 tx=0x7fdf4c00da90 comp rx=0 tx=0).stop 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 shutdown_connections 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf2c06b890 0x7fdf2c06dd50 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf54144c00 0x7fdf541f5870 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf540cdd70 0x7fdf54155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 --2- 172.21.15.2:0/2288643051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf5406d000 0x7fdf5413b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 >> 172.21.15.2:0/2288643051 conn(0x7fdf540b45f0 msgr2=0x7fdf5415b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 shutdown_connections 2024-01-15T18:02:52.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.041+0000 7fdf5b600640 1 -- 172.21.15.2:0/2288643051 wait complete. 2024-01-15T18:02:52.065 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:02:52.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.282+0000 7f42fca00640 1 Processor -- start 2024-01-15T18:02:52.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.282+0000 7f42fca00640 1 -- start start 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42fca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42f815d270 0x7f42f80b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42fca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f42f8161890 0x7f42f8160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42fca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f81611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42fca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f806c5e0 con 0x7f42f815d270 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42fca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42f806c8c0 con 0x7f42f8161890 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42fca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8071c00 con 0x7f42f80b2b60 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42f7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42f815d270 0x7f42f80b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42f7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42f815d270 0x7f42f80b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37002/0 (socket says 172.21.15.2:37002) 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.284+0000 7f42f7400640 1 -- 172.21.15.2:0/628589059 learned_addr learned my addr 172.21.15.2:0/628589059 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:52.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.285+0000 7f42f7400640 1 -- 172.21.15.2:0/628589059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 msgr2=0x7f42f81611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.285+0000 7f42f7400640 1 --2- 172.21.15.2:0/628589059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f81611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.285+0000 7f42f7400640 1 -- 172.21.15.2:0/628589059 >> v2:172.21.15.137:3300/0 conn(0x7f42f8161890 msgr2=0x7f42f8160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.285+0000 7f42f7400640 1 --2- 172.21.15.2:0/628589059 >> v2:172.21.15.137:3300/0 conn(0x7f42f8161890 0x7f42f8160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.285+0000 7f42f7400640 1 -- 172.21.15.2:0/628589059 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8071ee0 con 0x7f42f815d270 2024-01-15T18:02:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.286+0000 7f42f7400640 1 --2- 172.21.15.2:0/628589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f815d270 0x7f42f80b2620 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f42f80ceba0 tx=0x7f42e802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d68c50d1d51816 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:52.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.286+0000 7f42f6a00640 1 -- 172.21.15.2:0/628589059 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8031750 con 0x7f42f815d270 2024-01-15T18:02:52.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.286+0000 7f42f6a00640 1 -- 172.21.15.2:0/628589059 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e80318b0 con 0x7f42f815d270 2024-01-15T18:02:52.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.286+0000 7f42f6a00640 1 -- 172.21.15.2:0/628589059 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8031c40 con 0x7f42f815d270 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 -- 172.21.15.2:0/628589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f815d270 msgr2=0x7f42f80b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 --2- 172.21.15.2:0/628589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f815d270 0x7f42f80b2620 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f42f80ceba0 tx=0x7f42e802b7f0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 -- 172.21.15.2:0/628589059 shutdown_connections 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 --2- 172.21.15.2:0/628589059 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f81611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 --2- 172.21.15.2:0/628589059 >> v2:172.21.15.137:3300/0 conn(0x7f42f8161890 0x7f42f8160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 --2- 172.21.15.2:0/628589059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f815d270 0x7f42f80b2620 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 -- 172.21.15.2:0/628589059 >> 172.21.15.2:0/628589059 conn(0x7f42f80b0840 msgr2=0x7f42f80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 -- 172.21.15.2:0/628589059 shutdown_connections 2024-01-15T18:02:52.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.287+0000 7f42fca00640 1 -- 172.21.15.2:0/628589059 wait complete. 2024-01-15T18:02:52.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.289+0000 7f42fca00640 1 Processor -- start 2024-01-15T18:02:52.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.289+0000 7f42fca00640 1 -- start start 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42fca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f8159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42fca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42f815d270 0x7f42f8152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42fca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 0x7f42f80ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42fca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80cc620 con 0x7f42f8161890 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42fca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f8152d70 con 0x7f42f815d270 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42fca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42f80ad6f0 con 0x7f42f80b2b60 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42f7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 0x7f42f80ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42f7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 0x7f42f80ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37010/0 (socket says 172.21.15.2:37010) 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.290+0000 7f42f7e00640 1 -- 172.21.15.2:0/4123889772 learned_addr learned my addr 172.21.15.2:0/4123889772 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7400640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f8159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42efe00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42f815d270 0x7f42f8152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7e00640 1 -- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 msgr2=0x7f42f8159f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7e00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f8159f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7e00640 1 -- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42f815d270 msgr2=0x7f42f8152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7e00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42f815d270 0x7f42f8152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7e00640 1 -- 172.21.15.2:0/4123889772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e80094d0 con 0x7f42f8161890 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42efe00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42f815d270 0x7f42f8152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42f7e00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 0x7f42f80ac8e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f42f000ba10 tx=0x7f42f000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f000cb40 con 0x7f42f8161890 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f000cca0 con 0x7f42f8161890 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f80cbf60 con 0x7f42f8161890 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.291+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f0014680 con 0x7f42f8161890 2024-01-15T18:02:52.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.292+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f80ad9d0 con 0x7f42f8161890 2024-01-15T18:02:52.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.293+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42f80cc760 con 0x7f42f8161890 2024-01-15T18:02:52.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.294+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f42f001d3b0 con 0x7f42f8161890 2024-01-15T18:02:52.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.295+0000 7f42ef400640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42d006b860 0x7f42d006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.295+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f42f008a250 con 0x7f42f8161890 2024-01-15T18:02:52.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.295+0000 7f42f7400640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42d006b860 0x7f42d006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.297+0000 7f42f7400640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42d006b860 0x7f42d006dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f42e8035000 tx=0x7f42e8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:52.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.299+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42f80cc760 con 0x7f42f8161890 2024-01-15T18:02:52.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.494+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f42f80cc760 con 0x7f42f8161890 2024-01-15T18:02:52.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.501+0000 7f42ef400640 1 -- 172.21.15.2:0/4123889772 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v18) v1 ==== 120+0+0 (secure 0 0 0) 0x7f42f80cc760 con 0x7f42f8161890 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42d006b860 msgr2=0x7f42d006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42d006b860 0x7f42d006dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f42e8035000 tx=0x7f42e8008030 comp rx=0 tx=0).stop 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 msgr2=0x7f42f80ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 0x7f42f80ac8e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f42f000ba10 tx=0x7f42f000bed0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 shutdown_connections 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42d006b860 0x7f42d006dd20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42f8161890 0x7f42f80ac8e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42f815d270 0x7f42f8152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 --2- 172.21.15.2:0/4123889772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42f80b2b60 0x7f42f8159f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 >> 172.21.15.2:0/4123889772 conn(0x7f42f80b0840 msgr2=0x7f42f80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:52.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 shutdown_connections 2024-01-15T18:02:52.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.504+0000 7f42fca00640 1 -- 172.21.15.2:0/4123889772 wait complete. 2024-01-15T18:02:52.530 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:02:52.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.748+0000 7f6e0d800640 1 Processor -- start 2024-01-15T18:02:52.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.748+0000 7f6e0d800640 1 -- start start 2024-01-15T18:02:52.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e0d800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6e0815ae80 0x7f6e080acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e0d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e0806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e0d800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e0806d290 con 0x7f6e0806c8f0 2024-01-15T18:02:52.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e0d800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0815e840 con 0x7f6e0815ae80 2024-01-15T18:02:52.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e0d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0815dbc0 con 0x7f6e080b2b40 2024-01-15T18:02:52.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e06a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6e0815ae80 0x7f6e080acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e06a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6e0815ae80 0x7f6e080acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37026/0 (socket says 172.21.15.2:37026) 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.750+0000 7f6e06a00640 1 -- 172.21.15.2:0/3723048680 learned_addr learned my addr 172.21.15.2:0/3723048680 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.751+0000 7f6e07400640 1 --2- 172.21.15.2:0/3723048680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e0806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.751+0000 7f6e06a00640 1 -- 172.21.15.2:0/3723048680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 msgr2=0x7f6e0806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.751+0000 7f6e06a00640 1 --2- 172.21.15.2:0/3723048680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e0806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.751+0000 7f6e06a00640 1 -- 172.21.15.2:0/3723048680 >> v1:172.21.15.137:6789/0 conn(0x7f6e0806c8f0 legacy=0x7f6e080b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:52.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.751+0000 7f6e06a00640 1 -- 172.21.15.2:0/3723048680 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0815dea0 con 0x7f6e0815ae80 2024-01-15T18:02:52.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.752+0000 7f6e06a00640 1 --2- 172.21.15.2:0/3723048680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0815ae80 0x7f6e080acd00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009870 tx=0x7f6df802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac0ea5b53b0a42ff server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:52.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.752+0000 7f6e05600640 1 -- 172.21.15.2:0/3723048680 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8030840 con 0x7f6e0815ae80 2024-01-15T18:02:52.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.752+0000 7f6e05600640 1 -- 172.21.15.2:0/3723048680 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df80309a0 con 0x7f6e0815ae80 2024-01-15T18:02:52.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.752+0000 7f6e05600640 1 -- 172.21.15.2:0/3723048680 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8030d30 con 0x7f6e0815ae80 2024-01-15T18:02:52.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 -- 172.21.15.2:0/3723048680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0815ae80 msgr2=0x7f6e080acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 --2- 172.21.15.2:0/3723048680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0815ae80 0x7f6e080acd00 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f6df8009870 tx=0x7f6df802b980 comp rx=0 tx=0).stop 2024-01-15T18:02:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 -- 172.21.15.2:0/3723048680 shutdown_connections 2024-01-15T18:02:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 --2- 172.21.15.2:0/3723048680 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e0806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 --2- 172.21.15.2:0/3723048680 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0815ae80 0x7f6e080acd00 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 -- 172.21.15.2:0/3723048680 >> 172.21.15.2:0/3723048680 conn(0x7f6e080b45f0 msgr2=0x7f6e080b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 -- 172.21.15.2:0/3723048680 shutdown_connections 2024-01-15T18:02:52.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.753+0000 7f6e0d800640 1 -- 172.21.15.2:0/3723048680 wait complete. 2024-01-15T18:02:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.755+0000 7f6e0d800640 1 Processor -- start 2024-01-15T18:02:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.755+0000 7f6e0d800640 1 -- start start 2024-01-15T18:02:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e0d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 0x7f6e080aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e0d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e080b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e0d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e0815ae80 0x7f6e080b0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e0d800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e080b0fd0 con 0x7f6e0806c8f0 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e0d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0815fc00 con 0x7f6e0815ae80 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e0d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e08160460 con 0x7f6e080b2b40 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 0x7f6e080aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e06a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 0x7f6e080aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37042/0 (socket says 172.21.15.2:37042) 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.756+0000 7f6e06a00640 1 -- 172.21.15.2:0/1484693834 learned_addr learned my addr 172.21.15.2:0/1484693834 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06000640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e080b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e07400640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e0815ae80 0x7f6e080b0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06a00640 1 -- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 msgr2=0x7f6e080b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06a00640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e080b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06a00640 1 -- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e0815ae80 msgr2=0x7f6e080b0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06a00640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e0815ae80 0x7f6e080b0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06a00640 1 -- 172.21.15.2:0/1484693834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df80094d0 con 0x7f6e0806c8f0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.757+0000 7f6e06a00640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 0x7f6e080aea60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6df8000c00 tx=0x7f6df8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.758+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8030810 con 0x7f6e0806c8f0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.758+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df8039d90 con 0x7f6e0806c8f0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.758+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df8043670 con 0x7f6e0806c8f0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.758+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0815fea0 con 0x7f6e0806c8f0 2024-01-15T18:02:52.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.758+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e08160230 con 0x7f6e0806c8f0 2024-01-15T18:02:52.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.759+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e08160030 con 0x7f6e0806c8f0 2024-01-15T18:02:52.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.760+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6df802f390 con 0x7f6e0806c8f0 2024-01-15T18:02:52.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.761+0000 7f6dfea00640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6de006b860 0x7f6de006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:52.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.761+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f6df80ac770 con 0x7f6e0806c8f0 2024-01-15T18:02:52.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.762+0000 7f6e06000640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6de006b860 0x7f6de006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:52.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.763+0000 7f6e06000640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6de006b860 0x7f6de006dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6e080b4b20 tx=0x7f6df0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:52.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.766+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6df8076f70 con 0x7f6e0806c8f0 2024-01-15T18:02:52.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.960+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6e080af0c0 con 0x7f6e0806c8f0 2024-01-15T18:02:52.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.965+0000 7f6dfea00640 1 -- 172.21.15.2:0/1484693834 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6df807a590 con 0x7f6e0806c8f0 2024-01-15T18:02:52.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6de006b860 msgr2=0x7f6de006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6de006b860 0x7f6de006dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6e080b4b20 tx=0x7f6df0009040 comp rx=0 tx=0).stop 2024-01-15T18:02:52.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 msgr2=0x7f6e080aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:52.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 0x7f6e080aea60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6df8000c00 tx=0x7f6df8004730 comp rx=0 tx=0).stop 2024-01-15T18:02:52.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 shutdown_connections 2024-01-15T18:02:52.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6de006b860 0x7f6de006dd20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e0815ae80 0x7f6e080b0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e080b2b40 0x7f6e080b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 --2- 172.21.15.2:0/1484693834 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e0806c8f0 0x7f6e080aea60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 >> 172.21.15.2:0/1484693834 conn(0x7f6e080b45f0 msgr2=0x7f6e08070350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 shutdown_connections 2024-01-15T18:02:52.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:52.968+0000 7f6e0d800640 1 -- 172.21.15.2:0/1484693834 wait complete. 2024-01-15T18:02:52.994 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-15T18:02:53.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.204+0000 7f1f0a200640 1 Processor -- start 2024-01-15T18:02:53.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.204+0000 7f1f0a200640 1 -- start start 2024-01-15T18:02:53.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.205+0000 7f1f0a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415c0e0 0x7f1f040cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.205+0000 7f1f0a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1f0406cb80 0x7f1f0415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.205+0000 7f1f0a200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0415cb00 con 0x7f1f0415b7d0 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.205+0000 7f1f0a200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040ccec0 con 0x7f1f0406cb80 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.205+0000 7f1f0a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040cd1a0 con 0x7f1f0415c0e0 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f03e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1f0406cb80 0x7f1f0415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f02a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1f0415b7d0 0x7f1f0406c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56942/0 (socket says 172.21.15.2:56942) 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f02a00640 1 -- 172.21.15.2:0/460726137 learned_addr learned my addr 172.21.15.2:0/460726137 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:53.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f03400640 1 --2- 172.21.15.2:0/460726137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415c0e0 0x7f1f040cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:53.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f03e00640 1 -- 172.21.15.2:0/460726137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415c0e0 msgr2=0x7f1f040cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:53.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f03e00640 1 --2- 172.21.15.2:0/460726137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415c0e0 0x7f1f040cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:53.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f03e00640 1 -- 172.21.15.2:0/460726137 >> v1:172.21.15.2:6789/0 conn(0x7f1f0415b7d0 legacy=0x7f1f0406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:53.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f02000640 1 -- 172.21.15.2:0/460726137 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418363043 0 0) 0x7f1f0415cb00 con 0x7f1f0415b7d0 2024-01-15T18:02:53.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.206+0000 7f1f03e00640 1 -- 172.21.15.2:0/460726137 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040cd480 con 0x7f1f0406cb80 2024-01-15T18:02:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.207+0000 7f1f03e00640 1 --2- 172.21.15.2:0/460726137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f0415cf90 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0097a0 tx=0x7f1efc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be6760c0706749a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f02000640 1 -- 172.21.15.2:0/460726137 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc031830 con 0x7f1f0406cb80 2024-01-15T18:02:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f02000640 1 -- 172.21.15.2:0/460726137 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc031990 con 0x7f1f0406cb80 2024-01-15T18:02:53.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f02000640 1 -- 172.21.15.2:0/460726137 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc031d20 con 0x7f1f0406cb80 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 -- 172.21.15.2:0/460726137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 msgr2=0x7f1f0415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 --2- 172.21.15.2:0/460726137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f0415cf90 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0097a0 tx=0x7f1efc02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 -- 172.21.15.2:0/460726137 shutdown_connections 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 --2- 172.21.15.2:0/460726137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f0415cf90 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 --2- 172.21.15.2:0/460726137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415c0e0 0x7f1f040cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 -- 172.21.15.2:0/460726137 >> 172.21.15.2:0/460726137 conn(0x7f1f0406da20 msgr2=0x7f1f0406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.208+0000 7f1f0a200640 1 -- 172.21.15.2:0/460726137 shutdown_connections 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.209+0000 7f1f0a200640 1 -- 172.21.15.2:0/460726137 wait complete. 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.210+0000 7f1f0a200640 1 Processor -- start 2024-01-15T18:02:53.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.210+0000 7f1f0a200640 1 -- start start 2024-01-15T18:02:53.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f0a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f04153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f0a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415b7d0 0x7f1f04154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f0a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f0415c0e0 0x7f1f040b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f0a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040b2f90 con 0x7f1f0406cb80 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f0a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0414b340 con 0x7f1f0415c0e0 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f0a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f040b29d0 con 0x7f1f0415b7d0 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.211+0000 7f1f03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f04153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f04153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45120/0 (socket says 172.21.15.2:45120) 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 -- 172.21.15.2:0/3560369168 learned_addr learned my addr 172.21.15.2:0/3560369168 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:53.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f02a00640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415b7d0 0x7f1f04154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03e00640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f0415c0e0 0x7f1f040b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 -- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415b7d0 msgr2=0x7f1f04154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415b7d0 0x7f1f04154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 -- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f0415c0e0 msgr2=0x7f1f040b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f0415c0e0 0x7f1f040b1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 -- 172.21.15.2:0/3560369168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc009400 con 0x7f1f0406cb80 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f02a00640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415b7d0 0x7f1f04154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.212+0000 7f1f03400640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f04153120 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007e40 tx=0x7f1ef400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:53.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.213+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4014620 con 0x7f1f0406cb80 2024-01-15T18:02:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.213+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef400ee90 con 0x7f1f0406cb80 2024-01-15T18:02:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.213+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40149c0 con 0x7f1f0406cb80 2024-01-15T18:02:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.213+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f040b2cb0 con 0x7f1f0406cb80 2024-01-15T18:02:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.213+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f04152c00 con 0x7f1f0406cb80 2024-01-15T18:02:53.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.214+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f04152780 con 0x7f1f0406cb80 2024-01-15T18:02:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.215+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1ef4014b20 con 0x7f1f0406cb80 2024-01-15T18:02:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.216+0000 7f1efb400640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1edc06b860 0x7f1edc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.216+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f1ef408ad00 con 0x7f1f0406cb80 2024-01-15T18:02:53.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.217+0000 7f1f02a00640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1edc06b860 0x7f1edc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:53.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.218+0000 7f1f02a00640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1edc06b860 0x7f1edc06dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1f0406c2b0 tx=0x7f1eec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:53.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.220+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f04152780 con 0x7f1f0406cb80 2024-01-15T18:02:53.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:53.488+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f1f04152780 con 0x7f1f0406cb80 2024-01-15T18:02:54.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.438+0000 7f1efb400640 1 -- 172.21.15.2:0/3560369168 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v264) v1 ==== 153+0+0 (secure 0 0 0) 0x7f1f04152780 con 0x7f1f0406cb80 2024-01-15T18:02:54.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1edc06b860 msgr2=0x7f1edc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1edc06b860 0x7f1edc06dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1f0406c2b0 tx=0x7f1eec009040 comp rx=0 tx=0).stop 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 msgr2=0x7f1f04153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f04153120 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4007e40 tx=0x7f1ef400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 shutdown_connections 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1edc06b860 0x7f1edc06dd20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f0415c0e0 0x7f1f040b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f0415b7d0 0x7f1f04154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 --2- 172.21.15.2:0/3560369168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f0406cb80 0x7f1f04153120 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 >> 172.21.15.2:0/3560369168 conn(0x7f1f0406da20 msgr2=0x7f1f0406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:54.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 shutdown_connections 2024-01-15T18:02:54.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.442+0000 7f1f0a200640 1 -- 172.21.15.2:0/3560369168 wait complete. 2024-01-15T18:02:54.469 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-15T18:02:54.469 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-15T18:02:54.469 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:02:54.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:02:54.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.679+0000 7f3903000640 1 Processor -- start 2024-01-15T18:02:54.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.679+0000 7f3903000640 1 -- start start 2024-01-15T18:02:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3903000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f38fc143810 0x7f38fc06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:54.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3903000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f38fc0b2250 0x7f38fc0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3903000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38fc06e470 con 0x7f38fc06e010 2024-01-15T18:02:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3903000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38fc0b3520 con 0x7f38fc143810 2024-01-15T18:02:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3903000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38fc0ccbe0 con 0x7f38fc0b2250 2024-01-15T18:02:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3901c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f38fc143810 0x7f38fc06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3901c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f38fc143810 0x7f38fc06dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45134/0 (socket says 172.21.15.2:45134) 2024-01-15T18:02:54.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.681+0000 7f3901c00640 1 -- 172.21.15.2:0/601613306 learned_addr learned my addr 172.21.15.2:0/601613306 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:54.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.682+0000 7f3902600640 1 --2- 172.21.15.2:0/601613306 >> v2:172.21.15.137:3300/0 conn(0x7f38fc0b2250 0x7f38fc0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:54.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.682+0000 7f3901c00640 1 -- 172.21.15.2:0/601613306 >> v2:172.21.15.137:3300/0 conn(0x7f38fc0b2250 msgr2=0x7f38fc0cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:54.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.682+0000 7f3901c00640 1 --2- 172.21.15.2:0/601613306 >> v2:172.21.15.137:3300/0 conn(0x7f38fc0b2250 0x7f38fc0cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.682+0000 7f3901c00640 1 -- 172.21.15.2:0/601613306 >> v1:172.21.15.2:6789/0 conn(0x7f38fc06e010 legacy=0x7f38fc0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.682+0000 7f3901c00640 1 -- 172.21.15.2:0/601613306 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38fc143bf0 con 0x7f38fc143810 2024-01-15T18:02:54.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.682+0000 7f38fbe00640 1 -- 172.21.15.2:0/601613306 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606924445 0 0) 0x7f38e4001110 con 0x7f38fc06e010 2024-01-15T18:02:54.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.683+0000 7f3901c00640 1 --2- 172.21.15.2:0/601613306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc143810 0x7f38fc06dad0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0b17e0 tx=0x7f38ec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6abe60344671014 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f38fbe00640 1 -- 172.21.15.2:0/601613306 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec02ca80 con 0x7f38fc143810 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f38fbe00640 1 -- 172.21.15.2:0/601613306 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ec02cbe0 con 0x7f38fc143810 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f38fbe00640 1 -- 172.21.15.2:0/601613306 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ec0343f0 con 0x7f38fc143810 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 -- 172.21.15.2:0/601613306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc143810 msgr2=0x7f38fc06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 --2- 172.21.15.2:0/601613306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc143810 0x7f38fc06dad0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0b17e0 tx=0x7f38ec02bee0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 -- 172.21.15.2:0/601613306 shutdown_connections 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 --2- 172.21.15.2:0/601613306 >> v2:172.21.15.137:3300/0 conn(0x7f38fc0b2250 0x7f38fc0cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 --2- 172.21.15.2:0/601613306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc143810 0x7f38fc06dad0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 -- 172.21.15.2:0/601613306 >> 172.21.15.2:0/601613306 conn(0x7f38fc0b0840 msgr2=0x7f38fc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 -- 172.21.15.2:0/601613306 shutdown_connections 2024-01-15T18:02:54.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.684+0000 7f3903000640 1 -- 172.21.15.2:0/601613306 wait complete. 2024-01-15T18:02:54.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.686+0000 7f3903000640 1 Processor -- start 2024-01-15T18:02:54.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.686+0000 7f3903000640 1 -- start start 2024-01-15T18:02:54.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.687+0000 7f3903000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38fc06e010 0x7f38fc164880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:54.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.687+0000 7f3903000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 0x7f38fc155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.687+0000 7f3903000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38fc143810 0x7f38fc06d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.687+0000 7f3903000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc164dc0 con 0x7f38fc0b2250 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.687+0000 7f3903000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc156930 con 0x7f38fc06e010 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.687+0000 7f3903000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38fc1595f0 con 0x7f38fc143810 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 0x7f38fc155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 0x7f38fc155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45150/0 (socket says 172.21.15.2:45150) 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 -- 172.21.15.2:0/2493966815 learned_addr learned my addr 172.21.15.2:0/2493966815 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901c00640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38fc06e010 0x7f38fc164880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3902600640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38fc143810 0x7f38fc06d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 -- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38fc143810 msgr2=0x7f38fc06d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38fc143810 0x7f38fc06d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 -- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38fc06e010 msgr2=0x7f38fc164880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38fc06e010 0x7f38fc164880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 -- 172.21.15.2:0/2493966815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ec009b30 con 0x7f38fc0b2250 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901c00640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38fc06e010 0x7f38fc164880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.688+0000 7f3901200640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 0x7f38fc155ba0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38e400c410 tx=0x7f38e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.689+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e40035f0 con 0x7f38fc0b2250 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.689+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e4003770 con 0x7f38fc0b2250 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.689+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e4003b20 con 0x7f38fc0b2250 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.689+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38fc167220 con 0x7f38fc0b2250 2024-01-15T18:02:54.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.689+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38fc1677e0 con 0x7f38fc0b2250 2024-01-15T18:02:54.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.690+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38fc074770 con 0x7f38fc0b2250 2024-01-15T18:02:54.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.691+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f38e4003c80 con 0x7f38fc0b2250 2024-01-15T18:02:54.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.692+0000 7f38f9600640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38d406b860 0x7f38d406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:54.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.692+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f38e4089880 con 0x7f38fc0b2250 2024-01-15T18:02:54.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.692+0000 7f3901c00640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38d406b860 0x7f38d406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:54.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.694+0000 7f3901c00640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38d406b860 0x7f38d406dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f38ec035000 tx=0x7f38ec030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:54.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.696+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f38e4054100 con 0x7f38fc0b2250 2024-01-15T18:02:54.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.892+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38fc06e470 con 0x7f38fc0b2250 2024-01-15T18:02:54.898 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:54.898 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":53,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:02:47.056364+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6833/4162570290":"2024-01-16T18:02:44.032475+0000","172.21.15.2:6832/4162570290":"2024-01-16T18:02:44.032475+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:02:54.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.893+0000 7f38f9600640 1 -- 172.21.15.2:0/2493966815 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+13011 (secure 0 0 0) 0x7f38fc06e470 con 0x7f38fc0b2250 2024-01-15T18:02:54.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.896+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38d406b860 msgr2=0x7f38d406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.896+0000 7f3903000640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38d406b860 0x7f38d406dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f38ec035000 tx=0x7f38ec030000 comp rx=0 tx=0).stop 2024-01-15T18:02:54.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.896+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 msgr2=0x7f38fc155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.896+0000 7f3903000640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 0x7f38fc155ba0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f38e400c410 tx=0x7f38e400c8d0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 shutdown_connections 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f38d406b860 0x7f38d406dd20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38fc143810 0x7f38fc06d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38fc0b2250 0x7f38fc155ba0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 --2- 172.21.15.2:0/2493966815 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38fc06e010 0x7f38fc164880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 >> 172.21.15.2:0/2493966815 conn(0x7f38fc0b0840 msgr2=0x7f38fc0af9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 shutdown_connections 2024-01-15T18:02:54.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:54.897+0000 7f3903000640 1 -- 172.21.15.2:0/2493966815 wait complete. 2024-01-15T18:02:54.922 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:02:54.922 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:02:55.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.145+0000 7fe787400640 1 Processor -- start 2024-01-15T18:02:55.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.145+0000 7fe787400640 1 -- start start 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe787400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe78006d000 0x7fe7800b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe787400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78014b3b0 0x7fe7800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe787400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7800b25f0 con 0x7fe78014afd0 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe787400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe780070ff0 con 0x7fe78006d000 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe787400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7800b28d0 con 0x7fe78014b3b0 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe78006d000 0x7fe7800b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe78006d000 0x7fe7800b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45162/0 (socket says 172.21.15.2:45162) 2024-01-15T18:02:55.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 -- 172.21.15.2:0/2705241936 learned_addr learned my addr 172.21.15.2:0/2705241936 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:55.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 -- 172.21.15.2:0/2705241936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78014b3b0 msgr2=0x7fe7800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:55.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 --2- 172.21.15.2:0/2705241936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78014b3b0 0x7fe7800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 -- 172.21.15.2:0/2705241936 >> v1:172.21.15.137:6789/0 conn(0x7fe78014afd0 legacy=0x7fe78006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:55.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.148+0000 7fe777c00640 1 -- 172.21.15.2:0/2705241936 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800b2bb0 con 0x7fe78006d000 2024-01-15T18:02:55.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.150+0000 7fe777c00640 1 --2- 172.21.15.2:0/2705241936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78006d000 0x7fe7800b33b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe768009870 tx=0x7fe76802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40971a1c7e0b6a97 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.150+0000 7fe77fe00640 1 -- 172.21.15.2:0/2705241936 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680317c0 con 0x7fe78006d000 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.150+0000 7fe77fe00640 1 -- 172.21.15.2:0/2705241936 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe768031920 con 0x7fe78006d000 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.150+0000 7fe77fe00640 1 -- 172.21.15.2:0/2705241936 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768031cb0 con 0x7fe78006d000 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 -- 172.21.15.2:0/2705241936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78006d000 msgr2=0x7fe7800b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 --2- 172.21.15.2:0/2705241936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78006d000 0x7fe7800b33b0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe768009870 tx=0x7fe76802b860 comp rx=0 tx=0).stop 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 -- 172.21.15.2:0/2705241936 shutdown_connections 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 --2- 172.21.15.2:0/2705241936 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78014b3b0 0x7fe7800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 --2- 172.21.15.2:0/2705241936 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78006d000 0x7fe7800b33b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 -- 172.21.15.2:0/2705241936 >> 172.21.15.2:0/2705241936 conn(0x7fe7800b45f0 msgr2=0x7fe7800b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:55.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 -- 172.21.15.2:0/2705241936 shutdown_connections 2024-01-15T18:02:55.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.151+0000 7fe787400640 1 -- 172.21.15.2:0/2705241936 wait complete. 2024-01-15T18:02:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.153+0000 7fe787400640 1 Processor -- start 2024-01-15T18:02:55.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.153+0000 7fe787400640 1 -- start start 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe787400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78006d000 0x7fe7800c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe787400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe78014afd0 0x7fe7800ce110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe787400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 0x7fe7800cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe787400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7800cd090 con 0x7fe78014b3b0 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe787400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7800cd370 con 0x7fe78014afd0 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe787400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7800cd650 con 0x7fe78006d000 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 0x7fe7800cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 0x7fe7800cc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45166/0 (socket says 172.21.15.2:45166) 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 -- 172.21.15.2:0/2825072667 learned_addr learned my addr 172.21.15.2:0/2825072667 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe784a00640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78006d000 0x7fe7800c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe777c00640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe78014afd0 0x7fe7800ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 -- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78006d000 msgr2=0x7fe7800c40d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78006d000 0x7fe7800c40d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 -- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe78014afd0 msgr2=0x7fe7800ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe78014afd0 0x7fe7800ce110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe785400640 1 -- 172.21.15.2:0/2825072667 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680094d0 con 0x7fe78014b3b0 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.154+0000 7fe784a00640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78006d000 0x7fe7800c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.155+0000 7fe785400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 0x7fe7800cc6b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe77800b410 tx=0x7fe77800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.155+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778011830 con 0x7fe78014b3b0 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.155+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe778011990 con 0x7fe78014b3b0 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.155+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe778011d40 con 0x7fe78014b3b0 2024-01-15T18:02:55.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.155+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7800cd930 con 0x7fe78014b3b0 2024-01-15T18:02:55.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.155+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7800c4610 con 0x7fe78014b3b0 2024-01-15T18:02:55.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.156+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7800686f0 con 0x7fe78014b3b0 2024-01-15T18:02:55.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.157+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe77801a400 con 0x7fe78014b3b0 2024-01-15T18:02:55.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.158+0000 7fe77d600640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe75806b860 0x7fe75806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.158+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fe77808a3e0 con 0x7fe78014b3b0 2024-01-15T18:02:55.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.159+0000 7fe784a00640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe75806b860 0x7fe75806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.160+0000 7fe784a00640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe75806b860 0x7fe75806dd20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe7800cfc10 tx=0x7fe770009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:55.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.162+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe778054d40 con 0x7fe78014b3b0 2024-01-15T18:02:55.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.352+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7800cfc90 con 0x7fe78014b3b0 2024-01-15T18:02:55.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.353+0000 7fe77d600640 1 -- 172.21.15.2:0/2825072667 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v53) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe778058360 con 0x7fe78014b3b0 2024-01-15T18:02:55.357 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:02:55.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe75806b860 msgr2=0x7fe75806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe75806b860 0x7fe75806dd20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fe7800cfc10 tx=0x7fe770009040 comp rx=0 tx=0).stop 2024-01-15T18:02:55.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 msgr2=0x7fe7800cc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 0x7fe7800cc6b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe77800b410 tx=0x7fe77800b810 comp rx=0 tx=0).stop 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 shutdown_connections 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe75806b860 0x7fe75806dd20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe78014b3b0 0x7fe7800cc6b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe78014afd0 0x7fe7800ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 --2- 172.21.15.2:0/2825072667 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe78006d000 0x7fe7800c40d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.355+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 >> 172.21.15.2:0/2825072667 conn(0x7fe7800b45f0 msgr2=0x7fe780147420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.356+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 shutdown_connections 2024-01-15T18:02:55.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.356+0000 7fe787400640 1 -- 172.21.15.2:0/2825072667 wait complete. 2024-01-15T18:02:55.379 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-15T18:02:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.598+0000 7f244e200640 1 Processor -- start 2024-01-15T18:02:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.598+0000 7f244e200640 1 -- start start 2024-01-15T18:02:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f244e200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f244815c0e0 0x7f24480cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f244e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244806cb80 0x7f244815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f244e200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244815cb00 con 0x7f244815b7d0 2024-01-15T18:02:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f244e200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24480ccec0 con 0x7f244815c0e0 2024-01-15T18:02:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f244e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24480cd1a0 con 0x7f244806cb80 2024-01-15T18:02:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f2447400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f244815c0e0 0x7f24480cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f2447400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f244815c0e0 0x7f24480cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45176/0 (socket says 172.21.15.2:45176) 2024-01-15T18:02:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f2447400640 1 -- 172.21.15.2:0/2342325173 learned_addr learned my addr 172.21.15.2:0/2342325173 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.600+0000 7f2446a00640 1 --1- 172.21.15.2:0/2342325173 >> v1:172.21.15.2:6789/0 conn(0x7f244815b7d0 0x7f244806c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56950/0 (socket says 172.21.15.2:56950) 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.601+0000 7f2447e00640 1 --2- 172.21.15.2:0/2342325173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244806cb80 0x7f244815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.601+0000 7f2447400640 1 -- 172.21.15.2:0/2342325173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244806cb80 msgr2=0x7f244815cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.601+0000 7f2447400640 1 --2- 172.21.15.2:0/2342325173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244806cb80 0x7f244815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.601+0000 7f2447400640 1 -- 172.21.15.2:0/2342325173 >> v1:172.21.15.2:6789/0 conn(0x7f244815b7d0 legacy=0x7f244806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.601+0000 7f2446000640 1 -- 172.21.15.2:0/2342325173 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921142833 0 0) 0x7f244815cb00 con 0x7f244815b7d0 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.601+0000 7f2447400640 1 -- 172.21.15.2:0/2342325173 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24480cd480 con 0x7f244815c0e0 2024-01-15T18:02:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.602+0000 7f2447400640 1 --2- 172.21.15.2:0/2342325173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815c0e0 0x7f24480cc5c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2438009870 tx=0x7f243802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e3f1d284a5d70 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.602+0000 7f2446000640 1 -- 172.21.15.2:0/2342325173 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438030840 con 0x7f244815c0e0 2024-01-15T18:02:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.602+0000 7f2446000640 1 -- 172.21.15.2:0/2342325173 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24380309a0 con 0x7f244815c0e0 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.602+0000 7f2446000640 1 -- 172.21.15.2:0/2342325173 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2438030d30 con 0x7f244815c0e0 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 -- 172.21.15.2:0/2342325173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815c0e0 msgr2=0x7f24480cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 --2- 172.21.15.2:0/2342325173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815c0e0 0x7f24480cc5c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f2438009870 tx=0x7f243802b980 comp rx=0 tx=0).stop 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 -- 172.21.15.2:0/2342325173 shutdown_connections 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 --2- 172.21.15.2:0/2342325173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244806cb80 0x7f244815cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 --2- 172.21.15.2:0/2342325173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815c0e0 0x7f24480cc5c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 -- 172.21.15.2:0/2342325173 >> 172.21.15.2:0/2342325173 conn(0x7f244806da20 msgr2=0x7f244806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 -- 172.21.15.2:0/2342325173 shutdown_connections 2024-01-15T18:02:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.603+0000 7f244e200640 1 -- 172.21.15.2:0/2342325173 wait complete. 2024-01-15T18:02:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.605+0000 7f244e200640 1 Processor -- start 2024-01-15T18:02:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.605+0000 7f244e200640 1 -- start start 2024-01-15T18:02:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f244e200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f244806cb80 0x7f2448151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f244e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 0x7f24481f6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f244e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244815c0e0 0x7f24480a8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f244e200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24481f3fc0 con 0x7f244815b7d0 2024-01-15T18:02:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f244e200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24481f3440 con 0x7f244806cb80 2024-01-15T18:02:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f244e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24481f3720 con 0x7f244815c0e0 2024-01-15T18:02:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f2446a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 0x7f24481f6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f2446a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 0x7f24481f6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45188/0 (socket says 172.21.15.2:45188) 2024-01-15T18:02:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.606+0000 7f2446a00640 1 -- 172.21.15.2:0/3166801511 learned_addr learned my addr 172.21.15.2:0/3166801511 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:55.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2447e00640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244815c0e0 0x7f24480a8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2447400640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f244806cb80 0x7f2448151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2446a00640 1 -- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244815c0e0 msgr2=0x7f24480a8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2446a00640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244815c0e0 0x7f24480a8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2446a00640 1 -- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f244806cb80 msgr2=0x7f2448151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2446a00640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f244806cb80 0x7f2448151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2446a00640 1 -- 172.21.15.2:0/3166801511 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24380094d0 con 0x7f244815b7d0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2447400640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f244806cb80 0x7f2448151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f2446a00640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 0x7f24481f6ff0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f243000c4d0 tx=0x7f243000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24300074e0 con 0x7f244815b7d0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2430007660 con 0x7f244815b7d0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2430004a20 con 0x7f244815b7d0 2024-01-15T18:02:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.607+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24481f3a00 con 0x7f244815b7d0 2024-01-15T18:02:55.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.608+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24480a8e70 con 0x7f244815b7d0 2024-01-15T18:02:55.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.609+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24480686f0 con 0x7f244815b7d0 2024-01-15T18:02:55.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.610+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f2430004b80 con 0x7f244815b7d0 2024-01-15T18:02:55.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.611+0000 7f243f400640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f242006b860 0x7f242006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:55.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.611+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7f24300893e0 con 0x7f244815b7d0 2024-01-15T18:02:55.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.611+0000 7f2447400640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f242006b860 0x7f242006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:55.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.613+0000 7f2447400640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f242006b860 0x7f242006dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2438036000 tx=0x7f243803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:55.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.616+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2430053dd0 con 0x7f244815b7d0 2024-01-15T18:02:55.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:55.877+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f24480a8710 con 0x7f244815b7d0 2024-01-15T18:02:56.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.450+0000 7f243f400640 1 -- 172.21.15.2:0/3166801511 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v266) v1 ==== 157+0+0 (secure 0 0 0) 0x7f24300573f0 con 0x7f244815b7d0 2024-01-15T18:02:56.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f242006b860 msgr2=0x7f242006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f242006b860 0x7f242006dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2438036000 tx=0x7f243803b040 comp rx=0 tx=0).stop 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 msgr2=0x7f24481f6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 0x7f24481f6ff0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f243000c4d0 tx=0x7f243000c990 comp rx=0 tx=0).stop 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 shutdown_connections 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f242006b860 0x7f242006dd20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f244815c0e0 0x7f24480a8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f244815b7d0 0x7f24481f6ff0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 --2- 172.21.15.2:0/3166801511 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f244806cb80 0x7f2448151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.455+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 >> 172.21.15.2:0/3166801511 conn(0x7f244806da20 msgr2=0x7f244815efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:56.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.456+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 shutdown_connections 2024-01-15T18:02:56.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.456+0000 7f244e200640 1 -- 172.21.15.2:0/3166801511 wait complete. 2024-01-15T18:02:56.482 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:02:56.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.700+0000 7fac26e00640 1 Processor -- start 2024-01-15T18:02:56.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.700+0000 7fac26e00640 1 -- start start 2024-01-15T18:02:56.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac26e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fac2015b900 0x7fac2015bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:56.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac26e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2015c440 con 0x7fac20155960 2024-01-15T18:02:56.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac26e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2015c720 con 0x7fac20071410 2024-01-15T18:02:56.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac26e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac200ced10 con 0x7fac2015b900 2024-01-15T18:02:56.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac1f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fac2015b900 0x7fac2015bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:56.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac1f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fac2015b900 0x7fac2015bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45198/0 (socket says 172.21.15.2:45198) 2024-01-15T18:02:56.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.702+0000 7fac1f400640 1 -- 172.21.15.2:0/2383069018 learned_addr learned my addr 172.21.15.2:0/2383069018 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:56.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.703+0000 7fac1f400640 1 -- 172.21.15.2:0/2383069018 >> v1:172.21.15.137:6789/0 conn(0x7fac20071410 legacy=0x7fac2006a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:56.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.703+0000 7fac1f400640 1 -- 172.21.15.2:0/2383069018 >> v1:172.21.15.2:6789/0 conn(0x7fac20155960 legacy=0x7fac200cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.703+0000 7fac1f400640 1 -- 172.21.15.2:0/2383069018 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac200ceff0 con 0x7fac2015b900 2024-01-15T18:02:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.704+0000 7fac1f400640 1 --2- 172.21.15.2:0/2383069018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac2015bdb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fac08009870 tx=0x7fac0802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55101c2ad5138348 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.704+0000 7fac1ea00640 1 -- 172.21.15.2:0/2383069018 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac08030860 con 0x7fac2015b900 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.704+0000 7fac1ea00640 1 -- 172.21.15.2:0/2383069018 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac080309c0 con 0x7fac2015b900 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.704+0000 7fac1ea00640 1 -- 172.21.15.2:0/2383069018 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac08030d50 con 0x7fac2015b900 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 -- 172.21.15.2:0/2383069018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 msgr2=0x7fac2015bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 --2- 172.21.15.2:0/2383069018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac2015bdb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fac08009870 tx=0x7fac0802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 -- 172.21.15.2:0/2383069018 shutdown_connections 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 --2- 172.21.15.2:0/2383069018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac2015bdb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 -- 172.21.15.2:0/2383069018 >> 172.21.15.2:0/2383069018 conn(0x7fac200b1c70 msgr2=0x7fac200b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 -- 172.21.15.2:0/2383069018 shutdown_connections 2024-01-15T18:02:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.705+0000 7fac26e00640 1 -- 172.21.15.2:0/2383069018 wait complete. 2024-01-15T18:02:56.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.707+0000 7fac26e00640 1 Processor -- start 2024-01-15T18:02:56.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.707+0000 7fac26e00640 1 -- start start 2024-01-15T18:02:56.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.708+0000 7fac26e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac20071410 0x7fac20154a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:56.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.708+0000 7fac26e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac20155960 0x7fac20142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:56.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.708+0000 7fac26e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac20143010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:56.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.708+0000 7fac26e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2013fa50 con 0x7fac2015b900 2024-01-15T18:02:56.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.708+0000 7fac26e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2013f1f0 con 0x7fac20155960 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.708+0000 7fac26e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac2013f4d0 con 0x7fac20071410 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac20143010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac20143010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45202/0 (socket says 172.21.15.2:45202) 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 -- 172.21.15.2:0/1867568140 learned_addr learned my addr 172.21.15.2:0/1867568140 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac1f400640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac20155960 0x7fac20142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac1fe00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac20071410 0x7fac20154a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 -- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac20071410 msgr2=0x7fac20154a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac20071410 0x7fac20154a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 -- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac20155960 msgr2=0x7fac20142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac20155960 0x7fac20142ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.709+0000 7fac24e00640 1 -- 172.21.15.2:0/1867568140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac080094d0 con 0x7fac2015b900 2024-01-15T18:02:56.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.710+0000 7fac24e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac20143010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c780 tx=0x7fac1800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:56.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.710+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18004b60 con 0x7fac2015b900 2024-01-15T18:02:56.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.710+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac1800ce90 con 0x7fac2015b900 2024-01-15T18:02:56.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.710+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac1800a850 con 0x7fac2015b900 2024-01-15T18:02:56.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.710+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2013fca0 con 0x7fac2015b900 2024-01-15T18:02:56.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.710+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac20154fb0 con 0x7fac2015b900 2024-01-15T18:02:56.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.712+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac201401c0 con 0x7fac2015b900 2024-01-15T18:02:56.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.712+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fac1800a9b0 con 0x7fac2015b900 2024-01-15T18:02:56.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.713+0000 7fac17e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabf806b8b0 0x7fabf806dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:56.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.713+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fac1808a4c0 con 0x7fac2015b900 2024-01-15T18:02:56.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.714+0000 7fac1fe00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabf806b8b0 0x7fabf806dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:56.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.715+0000 7fac1fe00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabf806b8b0 0x7fabf806dd70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fac10000f60 tx=0x7fac10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:56.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.718+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac201401c0 con 0x7fac2015b900 2024-01-15T18:02:56.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.921+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fac201401c0 con 0x7fac2015b900 2024-01-15T18:02:56.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.928+0000 7fac17e00640 1 -- 172.21.15.2:0/1867568140 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7fac201401c0 con 0x7fac2015b900 2024-01-15T18:02:56.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.930+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabf806b8b0 msgr2=0x7fabf806dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.930+0000 7fac26e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabf806b8b0 0x7fabf806dd70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fac10000f60 tx=0x7fac10009040 comp rx=0 tx=0).stop 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.930+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 msgr2=0x7fac20143010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.930+0000 7fac26e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac20143010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fac1800c780 tx=0x7fac1800cc40 comp rx=0 tx=0).stop 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 shutdown_connections 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fabf806b8b0 0x7fabf806dd70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac2015b900 0x7fac20143010 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac20155960 0x7fac20142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 --2- 172.21.15.2:0/1867568140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac20071410 0x7fac20154a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 >> 172.21.15.2:0/1867568140 conn(0x7fac200b1c70 msgr2=0x7fac200cf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 shutdown_connections 2024-01-15T18:02:56.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:56.931+0000 7fac26e00640 1 -- 172.21.15.2:0/1867568140 wait complete. 2024-01-15T18:02:56.953 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:02:56.953 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:02:57.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:02:57.165 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:02:57.165 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:02:57.165 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:57.223 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:02:57.224 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:02:57.224 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:57.281 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:02:57.282 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:02:57.282 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:57.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:02:57.344 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:02:57.345 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:02:57.401 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:02:57.401 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:02:57.401 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:02:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.626+0000 7fe575e00640 1 Processor -- start 2024-01-15T18:02:57.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.626+0000 7fe575e00640 1 -- start start 2024-01-15T18:02:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.627+0000 7fe575e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe57015b7d0 0x7fe57006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe575e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe57015c4c0 0x7fe57006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe575e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe57015cb00 con 0x7fe57015c0e0 2024-01-15T18:02:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe575e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5700cccd0 con 0x7fe57015c4c0 2024-01-15T18:02:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe575e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5700ccec0 con 0x7fe57015b7d0 2024-01-15T18:02:57.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe57015c4c0 0x7fe57006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe57015c4c0 0x7fe57006c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45212/0 (socket says 172.21.15.2:45212) 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 -- 172.21.15.2:0/856482746 learned_addr learned my addr 172.21.15.2:0/856482746 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe567e00640 1 --2- 172.21.15.2:0/856482746 >> v2:172.21.15.137:3300/0 conn(0x7fe57015b7d0 0x7fe57006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 -- 172.21.15.2:0/856482746 >> v2:172.21.15.137:3300/0 conn(0x7fe57015b7d0 msgr2=0x7fe57006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 --2- 172.21.15.2:0/856482746 >> v2:172.21.15.137:3300/0 conn(0x7fe57015b7d0 0x7fe57006c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 -- 172.21.15.2:0/856482746 >> v1:172.21.15.137:6789/0 conn(0x7fe57015c0e0 legacy=0x7fe5700cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:57.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.628+0000 7fe56f400640 1 -- 172.21.15.2:0/856482746 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5700cd1a0 con 0x7fe57015c4c0 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.630+0000 7fe56f400640 1 --2- 172.21.15.2:0/856482746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57006c9b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe570072830 tx=0x7fe56802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393e5fccf70e5646 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.630+0000 7fe56e000640 1 -- 172.21.15.2:0/856482746 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56802ca80 con 0x7fe57015c4c0 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.630+0000 7fe56e000640 1 -- 172.21.15.2:0/856482746 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56802cbe0 con 0x7fe57015c4c0 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.630+0000 7fe56e000640 1 -- 172.21.15.2:0/856482746 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5680343f0 con 0x7fe57015c4c0 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.630+0000 7fe575e00640 1 -- 172.21.15.2:0/856482746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 msgr2=0x7fe57006c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.630+0000 7fe575e00640 1 --2- 172.21.15.2:0/856482746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57006c9b0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fe570072830 tx=0x7fe56802bee0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.631+0000 7fe575e00640 1 -- 172.21.15.2:0/856482746 shutdown_connections 2024-01-15T18:02:57.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.631+0000 7fe575e00640 1 --2- 172.21.15.2:0/856482746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57006c9b0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.631+0000 7fe575e00640 1 --2- 172.21.15.2:0/856482746 >> v2:172.21.15.137:3300/0 conn(0x7fe57015b7d0 0x7fe57006c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.631+0000 7fe575e00640 1 -- 172.21.15.2:0/856482746 >> 172.21.15.2:0/856482746 conn(0x7fe57006da20 msgr2=0x7fe57006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.631+0000 7fe575e00640 1 -- 172.21.15.2:0/856482746 shutdown_connections 2024-01-15T18:02:57.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.631+0000 7fe575e00640 1 -- 172.21.15.2:0/856482746 wait complete. 2024-01-15T18:02:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.632+0000 7fe575e00640 1 Processor -- start 2024-01-15T18:02:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.633+0000 7fe575e00640 1 -- start start 2024-01-15T18:02:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.633+0000 7fe575e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe57015b7d0 0x7fe5701e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe575e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015c0e0 0x7fe57014cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe575e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57014d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:57.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe575e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5701e9520 con 0x7fe57015c4c0 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe575e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5700b3870 con 0x7fe57015c0e0 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe575e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe570153010 con 0x7fe57015b7d0 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57014d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57014d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45216/0 (socket says 172.21.15.2:45216) 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 -- 172.21.15.2:0/3103906092 learned_addr learned my addr 172.21.15.2:0/3103906092 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe567e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015c0e0 0x7fe57014cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56ea00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe57015b7d0 0x7fe5701e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 -- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe57015b7d0 msgr2=0x7fe5701e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe57015b7d0 0x7fe5701e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 -- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015c0e0 msgr2=0x7fe57014cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015c0e0 0x7fe57014cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.634+0000 7fe56f400640 1 -- 172.21.15.2:0/3103906092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568009b30 con 0x7fe57015c4c0 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.635+0000 7fe56f400640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57014d300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe568036040 tx=0x7fe568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.635+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe568009510 con 0x7fe57015c4c0 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.635+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56803e450 con 0x7fe57015c4c0 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.635+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56803e7e0 con 0x7fe57015c4c0 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.635+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5701532f0 con 0x7fe57015c4c0 2024-01-15T18:02:57.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.635+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5700b2960 con 0x7fe57015c4c0 2024-01-15T18:02:57.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.636+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe570153810 con 0x7fe57015c4c0 2024-01-15T18:02:57.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.637+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe56802cd50 con 0x7fe57015c4c0 2024-01-15T18:02:57.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.638+0000 7fe567400640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b890 0x7fe54806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:57.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.638+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fe5680acbf0 con 0x7fe57015c4c0 2024-01-15T18:02:57.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.639+0000 7fe56ea00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b890 0x7fe54806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:57.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.640+0000 7fe56ea00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b890 0x7fe54806dd50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe560004ea0 tx=0x7fe560009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:57.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.643+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe570153810 con 0x7fe57015c4c0 2024-01-15T18:02:57.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.873+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe570153810 con 0x7fe57015c4c0 2024-01-15T18:02:57.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.874+0000 7fe567400640 1 -- 172.21.15.2:0/3103906092 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fe568043e90 con 0x7fe57015c4c0 2024-01-15T18:02:57.879 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:57.879 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5239,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1873476051","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1873476051},{"type":"v1","addr":"172.21.15.2:6835","nonce":1873476051}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5248,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1702488249","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1702488249},{"type":"v1","addr":"172.21.15.137:6836","nonce":1702488249}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":5251,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/4131491202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":4131491202},{"type":"v1","addr":"172.21.15.137:6837","nonce":4131491202}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":21},{"gid":6165,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/3020321570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3020321570},{"type":"v1","addr":"172.21.15.2:6833","nonce":3020321570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29}],"filesystems":[]} 2024-01-15T18:02:57.879 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 29 2024-01-15T18:02:57.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b890 msgr2=0x7fe54806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:57.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b890 0x7fe54806dd50 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fe560004ea0 tx=0x7fe560009040 comp rx=0 tx=0).stop 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 msgr2=0x7fe57014d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57014d300 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fe568036040 tx=0x7fe568004a40 comp rx=0 tx=0).stop 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 shutdown_connections 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe54806b890 0x7fe54806dd50 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe57015c4c0 0x7fe57014d300 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe57015c0e0 0x7fe57014cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 --2- 172.21.15.2:0/3103906092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe57015b7d0 0x7fe5701e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 >> 172.21.15.2:0/3103906092 conn(0x7fe57006da20 msgr2=0x7fe57015acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 shutdown_connections 2024-01-15T18:02:57.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:57.877+0000 7fe575e00640 1 -- 172.21.15.2:0/3103906092 wait complete. 2024-01-15T18:02:57.901 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:02:57.901 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:02:57.902 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:02:57.902 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:02:57.902 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:02:57.902 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:02:57.902 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:02:57.902 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:02:57.902 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:02:57.902 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:02:57.903 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:02:57.903 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:02:57.903 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:02:57.903 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:02:57.903 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:02:57.903 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:02:57.921 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:02:57.921 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:02:57.922 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:02:57.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:02:57.924 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:02:57.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:02:57.925 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:02:57.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:02:57.970 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:02:57.970 DEBUG:teuthology.parallel:result is None 2024-01-15T18:02:57.972 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:02:57.973 DEBUG:teuthology.parallel:result is None 2024-01-15T18:02:57.977 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:02:57.977 DEBUG:teuthology.parallel:result is None 2024-01-15T18:02:57.978 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:02:57.978 DEBUG:teuthology.parallel:result is None 2024-01-15T18:02:57.979 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:02:58.130 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:02:58.132 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:02:58.134 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:02:58.134 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T18:02:58.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.168+0000 7fe123200640 1 Processor -- start 2024-01-15T18:02:58.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.168+0000 7fe123200640 1 -- start start 2024-01-15T18:02:58.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.169+0000 7fe123200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe11c143850 0x7fe11c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe123200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe11c06cfc0 0x7fe11c071450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe123200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe11c15e970 con 0x7fe11c160f60 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe123200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c1619b0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe123200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe11c161c90 con 0x7fe11c143850 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe11c06cfc0 0x7fe11c071450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe11c06cfc0 0x7fe11c071450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45264/0 (socket says 172.21.15.2:45264) 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 -- 172.21.15.2:0/410178398 learned_addr learned my addr 172.21.15.2:0/410178398 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 -- 172.21.15.2:0/410178398 >> v2:172.21.15.137:3300/0 conn(0x7fe11c143850 msgr2=0x7fe11c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe11be00640 1 --2- 172.21.15.2:0/410178398 >> v2:172.21.15.137:3300/0 conn(0x7fe11c143850 0x7fe11c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 --2- 172.21.15.2:0/410178398 >> v2:172.21.15.137:3300/0 conn(0x7fe11c143850 0x7fe11c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 -- 172.21.15.2:0/410178398 >> v1:172.21.15.137:6789/0 conn(0x7fe11c160f60 legacy=0x7fe11c06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe121200640 1 -- 172.21.15.2:0/410178398 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe11c15dc30 con 0x7fe11c06cfc0 2024-01-15T18:02:58.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.170+0000 7fe11be00640 1 --2- 172.21.15.2:0/410178398 >> v2:172.21.15.137:3300/0 conn(0x7fe11c143850 0x7fe11c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.171+0000 7fe121200640 1 --2- 172.21.15.2:0/410178398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c071450 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0acfb0 tx=0x7fe11402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431bdb59a74d7fce server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe11b400640 1 -- 172.21.15.2:0/410178398 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11402c9f0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe11b400640 1 -- 172.21.15.2:0/410178398 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe11402cb50 con 0x7fe11c06cfc0 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe11b400640 1 -- 172.21.15.2:0/410178398 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1140343f0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe123200640 1 -- 172.21.15.2:0/410178398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 msgr2=0x7fe11c071450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe123200640 1 --2- 172.21.15.2:0/410178398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c071450 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe11c0acfb0 tx=0x7fe11402be50 comp rx=0 tx=0).stop 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe123200640 1 -- 172.21.15.2:0/410178398 shutdown_connections 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe123200640 1 --2- 172.21.15.2:0/410178398 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c071450 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe123200640 1 --2- 172.21.15.2:0/410178398 >> v2:172.21.15.137:3300/0 conn(0x7fe11c143850 0x7fe11c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.172+0000 7fe123200640 1 -- 172.21.15.2:0/410178398 >> 172.21.15.2:0/410178398 conn(0x7fe11c0b46d0 msgr2=0x7fe11c0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.173+0000 7fe123200640 1 -- 172.21.15.2:0/410178398 shutdown_connections 2024-01-15T18:02:58.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.173+0000 7fe123200640 1 -- 172.21.15.2:0/410178398 wait complete. 2024-01-15T18:02:58.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.174+0000 7fe123200640 1 Processor -- start 2024-01-15T18:02:58.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.174+0000 7fe123200640 1 -- start start 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.175+0000 7fe123200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c15eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.175+0000 7fe123200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c143850 0x7fe11c13b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.175+0000 7fe123200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe11c160f60 0x7fe11c13cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.175+0000 7fe123200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c0b18a0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.175+0000 7fe123200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c13da80 con 0x7fe11c143850 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.175+0000 7fe123200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11c13dd60 con 0x7fe11c160f60 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c15eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c15eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45276/0 (socket says 172.21.15.2:45276) 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 -- 172.21.15.2:0/524314025 learned_addr learned my addr 172.21.15.2:0/524314025 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 -- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe11c160f60 msgr2=0x7fe11c13cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe121200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe11c160f60 0x7fe11c13cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe113200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c143850 0x7fe11c13b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe11c160f60 0x7fe11c13cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 -- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c143850 msgr2=0x7fe11c13b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c143850 0x7fe11c13b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 -- 172.21.15.2:0/524314025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe114009b30 con 0x7fe11c06cfc0 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe121200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe11c160f60 0x7fe11c13cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe113200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c143850 0x7fe11c13b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.176+0000 7fe11be00640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c15eda0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c7f0 tx=0x7fe10c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.177+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c007a30 con 0x7fe11c06cfc0 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.177+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe10c007b90 con 0x7fe11c06cfc0 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.177+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10c00ce40 con 0x7fe11c06cfc0 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.177+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11c0afe60 con 0x7fe11c06cfc0 2024-01-15T18:02:58.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.177+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11c13e040 con 0x7fe11c06cfc0 2024-01-15T18:02:58.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.178+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe11c0686f0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.179+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe10c0113f0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.180+0000 7fe118c00640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0f406b890 0x7fe0f406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.180+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4999+0+0 (secure 0 0 0) 0x7fe10c08a590 con 0x7fe11c06cfc0 2024-01-15T18:02:58.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.180+0000 7fe113200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0f406b890 0x7fe0f406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.181+0000 7fe113200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0f406b890 0x7fe0f406dd50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe11c1f07f0 tx=0x7fe104008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:58.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.184+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe10c054d90 con 0x7fe11c06cfc0 2024-01-15T18:02:58.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.389+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe11c067780 con 0x7fe11c06cfc0 2024-01-15T18:02:58.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.477+0000 7fe118c00640 1 -- 172.21.15.2:0/524314025 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v55) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe10c0583b0 con 0x7fe11c06cfc0 2024-01-15T18:02:58.481 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T18:02:58.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0f406b890 msgr2=0x7fe0f406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0f406b890 0x7fe0f406dd50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe11c1f07f0 tx=0x7fe104008040 comp rx=0 tx=0).stop 2024-01-15T18:02:58.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 msgr2=0x7fe11c15eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c15eda0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe10c00c7f0 tx=0x7fe10c00cbf0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 shutdown_connections 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0f406b890 0x7fe0f406dd50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe11c160f60 0x7fe11c13cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe11c143850 0x7fe11c13b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 --2- 172.21.15.2:0/524314025 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe11c06cfc0 0x7fe11c15eda0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 >> 172.21.15.2:0/524314025 conn(0x7fe11c0b46d0 msgr2=0x7fe11c142a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 shutdown_connections 2024-01-15T18:02:58.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.481+0000 7fe123200640 1 -- 172.21.15.2:0/524314025 wait complete. 2024-01-15T18:02:58.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:02:58.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.724+0000 7f7260600640 1 Processor -- start 2024-01-15T18:02:58.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.725+0000 7f7260600640 1 -- start start 2024-01-15T18:02:58.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.726+0000 7f7260600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f725815d5d0 0x7f72580d2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.726+0000 7f7260600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580d2780 con 0x7f72580b4cd0 2024-01-15T18:02:58.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.726+0000 7f7260600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580d1b70 con 0x7f72580b8ed0 2024-01-15T18:02:58.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.726+0000 7f7260600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72580d1980 con 0x7f725815d5d0 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.727+0000 7f725e600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f72580b4cd0 0x7f72580b50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56974/0 (socket says 172.21.15.2:56974) 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.727+0000 7f725e600640 1 -- 172.21.15.2:0/998689615 learned_addr learned my addr 172.21.15.2:0/998689615 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.727+0000 7f725dc00640 1 --2- 172.21.15.2:0/998689615 >> v2:172.21.15.2:3300/0 conn(0x7f725815d5d0 0x7f72580d2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.727+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152035950 0 0) 0x7f72580d2780 con 0x7f72580b4cd0 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.727+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7238003520 con 0x7f72580b4cd0 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408185070 0 0) 0x7f72580d1b70 con 0x7f72580b8ed0 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72580d2780 con 0x7f72580b8ed0 2024-01-15T18:02:58.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2901164598 0 0) 0x7f7238003520 con 0x7f72580b4cd0 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 >> v2:172.21.15.2:3300/0 conn(0x7f725815d5d0 msgr2=0x7f72580d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 --2- 172.21.15.2:0/998689615 >> v2:172.21.15.2:3300/0 conn(0x7f725815d5d0 0x7f72580d2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 >> v1:172.21.15.137:6789/0 conn(0x7f72580b8ed0 legacy=0x7f72580d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580d1e50 con 0x7f72580b4cd0 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7250001670 con 0x7f72580b4cd0 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 >> v1:172.21.15.2:6789/0 conn(0x7f72580b4cd0 legacy=0x7f72580b50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238003b00 0x7f7238003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72380047c0 0x7f7238004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7238005180 0x7f723800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723800bec0 con 0x7f7238005180 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723800c030 con 0x7f72380047c0 2024-01-15T18:02:58.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.728+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723800c1f0 con 0x7f7238003b00 2024-01-15T18:02:58.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.729+0000 7f725dc00640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238003b00 0x7f7238003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.729+0000 7f725e600640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7238005180 0x7f723800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.729+0000 7f725d200640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72380047c0 0x7f7238004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.730+0000 7f725e600640 1 -- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238003b00 msgr2=0x7f7238003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.730+0000 7f725e600640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238003b00 0x7f7238003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.730+0000 7f725e600640 1 -- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72380047c0 msgr2=0x7f7238004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.730+0000 7f725e600640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72380047c0 0x7f7238004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.730+0000 7f725e600640 1 -- 172.21.15.2:0/998689615 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7238003520 con 0x7f7238005180 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.730+0000 7f725d200640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72380047c0 0x7f7238004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:02:58.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.731+0000 7f725e600640 1 -- 172.21.15.2:0/998689615 reap_dead start 2024-01-15T18:02:58.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.731+0000 7f725e600640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7238005180 0x7f723800b980 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f725000c160 tx=0x7f72380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4e0b85e27512214 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:58.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.731+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250005ad0 con 0x7f7238005180 2024-01-15T18:02:58.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.731+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7250005c70 con 0x7f7238005180 2024-01-15T18:02:58.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.731+0000 7f7257e00640 1 -- 172.21.15.2:0/998689615 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725002c830 con 0x7f7238005180 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.732+0000 7f7260600640 1 -- 172.21.15.2:0/998689615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7238005180 msgr2=0x7f723800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.732+0000 7f7260600640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7238005180 0x7f723800b980 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f725000c160 tx=0x7f72380057f0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.733+0000 7f7260600640 1 -- 172.21.15.2:0/998689615 shutdown_connections 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.733+0000 7f7260600640 1 --2- 172.21.15.2:0/998689615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7238005180 0x7f723800b980 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.733+0000 7f7260600640 1 -- 172.21.15.2:0/998689615 >> 172.21.15.2:0/998689615 conn(0x7f7258072210 msgr2=0x7f72580cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.733+0000 7f7260600640 1 -- 172.21.15.2:0/998689615 shutdown_connections 2024-01-15T18:02:58.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.733+0000 7f7260600640 1 -- 172.21.15.2:0/998689615 wait complete. 2024-01-15T18:02:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.735+0000 7f7260600640 1 Processor -- start 2024-01-15T18:02:58.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.735+0000 7f7260600640 1 -- start start 2024-01-15T18:02:58.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f7260600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238005180 0x7f72580b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f7260600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f725815ea00 0x7f725815ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f7260600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 0x7f72581f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f7260600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725815f3c0 con 0x7f72581f2e80 2024-01-15T18:02:58.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f7260600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72581f3bf0 con 0x7f725815ea00 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f7260600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7258156370 con 0x7f7238005180 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f725e600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 0x7f72581f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f725e600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 0x7f72581f3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45302/0 (socket says 172.21.15.2:45302) 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f725e600640 1 -- 172.21.15.2:0/2154463285 learned_addr learned my addr 172.21.15.2:0/2154463285 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f725d200640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f725815ea00 0x7f725815ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.736+0000 7f725dc00640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238005180 0x7f72580b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f725e600640 1 -- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238005180 msgr2=0x7f72580b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f725e600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238005180 0x7f72580b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f725e600640 1 -- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f725815ea00 msgr2=0x7f725815ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f725e600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f725815ea00 0x7f725815ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f725e600640 1 -- 172.21.15.2:0/2154463285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7258156610 con 0x7f72581f2e80 2024-01-15T18:02:58.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f725e600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 0x7f72581f3300 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7250000c00 tx=0x7f7250001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:58.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.737+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7250039070 con 0x7f72581f2e80 2024-01-15T18:02:58.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.738+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725003a040 con 0x7f72581f2e80 2024-01-15T18:02:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.738+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72500072b0 con 0x7f72581f2e80 2024-01-15T18:02:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.738+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72581568f0 con 0x7f72581f2e80 2024-01-15T18:02:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.738+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7258156eb0 con 0x7f72581f2e80 2024-01-15T18:02:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.739+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72580cf3e0 con 0x7f72581f2e80 2024-01-15T18:02:58.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.740+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7250032070 con 0x7f72581f2e80 2024-01-15T18:02:58.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.741+0000 7f7255600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f723006b890 0x7f723006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:58.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.741+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f72500abc60 con 0x7f72581f2e80 2024-01-15T18:02:58.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.741+0000 7f725dc00640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f723006b890 0x7f723006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:58.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.742+0000 7f725dc00640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f723006b890 0x7f723006dd50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7248009fd0 tx=0x7f724800b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:58.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:58.745+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7250076690 con 0x7f72581f2e80 2024-01-15T18:02:59.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.007+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f72581570a0 con 0x7f72581f2e80 2024-01-15T18:02:59.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.008+0000 7f7255600640 1 -- 172.21.15.2:0/2154463285 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7250079cb0 con 0x7f72581f2e80 2024-01-15T18:02:59.012 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.013 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:02:59.014 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.015 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.016 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.017 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.018 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.019 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.020 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T18:02:59.021 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T18:02:59.022 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.023 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.024 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow r", 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:02:59.025 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.026 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.027 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.028 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T18:02:59.029 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T18:02:59.030 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:02:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f723006b890 msgr2=0x7f723006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f723006b890 0x7f723006dd50 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7248009fd0 tx=0x7f724800b040 comp rx=0 tx=0).stop 2024-01-15T18:02:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 msgr2=0x7f72581f3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 0x7f72581f3300 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f7250000c00 tx=0x7f7250001980 comp rx=0 tx=0).stop 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 shutdown_connections 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f723006b890 0x7f723006dd50 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72581f2e80 0x7f72581f3300 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f725815ea00 0x7f725815ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 --2- 172.21.15.2:0/2154463285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7238005180 0x7f72580b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.010+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 >> 172.21.15.2:0/2154463285 conn(0x7f7258072210 msgr2=0x7f72580cffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.011+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 shutdown_connections 2024-01-15T18:02:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.011+0000 7f7260600640 1 -- 172.21.15.2:0/2154463285 wait complete. 2024-01-15T18:02:59.035 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:02:59.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.249+0000 7f17a6000640 1 Processor -- start 2024-01-15T18:02:59.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.249+0000 7f17a6000640 1 -- start start 2024-01-15T18:02:59.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a6000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17a015f0e0 0x7f17a006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a6000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a00716d0 con 0x7f17a0141880 2024-01-15T18:02:59.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a6000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17a0071960 con 0x7f17a015ec60 2024-01-15T18:02:59.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a6000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17a0071f20 con 0x7f17a015f0e0 2024-01-15T18:02:59.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a5600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17a015f0e0 0x7f17a006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a4c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f17a0141880 0x7f17a0070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56980/0 (socket says 172.21.15.2:56980) 2024-01-15T18:02:59.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.251+0000 7f17a4c00640 1 -- 172.21.15.2:0/834005884 learned_addr learned my addr 172.21.15.2:0/834005884 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:02:59.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.252+0000 7f17a5600640 1 -- 172.21.15.2:0/834005884 >> v1:172.21.15.137:6789/0 conn(0x7f17a015ec60 legacy=0x7f17a006cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:59.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.252+0000 7f17a5600640 1 -- 172.21.15.2:0/834005884 >> v1:172.21.15.2:6789/0 conn(0x7f17a0141880 legacy=0x7f17a0070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.252+0000 7f17a5600640 1 -- 172.21.15.2:0/834005884 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a0071c40 con 0x7f17a015f0e0 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.253+0000 7f17a5600640 1 --2- 172.21.15.2:0/834005884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a015f0e0 0x7f17a006d1e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1798009d90 tx=0x7f179802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9f8b862a86f70d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.253+0000 7f179f400640 1 -- 172.21.15.2:0/834005884 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179802cbd0 con 0x7f17a015f0e0 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.253+0000 7f179f400640 1 -- 172.21.15.2:0/834005884 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179802cd30 con 0x7f17a015f0e0 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.253+0000 7f179f400640 1 -- 172.21.15.2:0/834005884 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179802cea0 con 0x7f17a015f0e0 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 -- 172.21.15.2:0/834005884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a015f0e0 msgr2=0x7f17a006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 --2- 172.21.15.2:0/834005884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a015f0e0 0x7f17a006d1e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f1798009d90 tx=0x7f179802be20 comp rx=0 tx=0).stop 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 -- 172.21.15.2:0/834005884 shutdown_connections 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 --2- 172.21.15.2:0/834005884 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a015f0e0 0x7f17a006d1e0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 -- 172.21.15.2:0/834005884 >> 172.21.15.2:0/834005884 conn(0x7f17a00b0840 msgr2=0x7f17a00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:59.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 -- 172.21.15.2:0/834005884 shutdown_connections 2024-01-15T18:02:59.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.254+0000 7f17a6000640 1 -- 172.21.15.2:0/834005884 wait complete. 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.256+0000 7f17a6000640 1 Processor -- start 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.256+0000 7f17a6000640 1 -- start start 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 0x7f17a00ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a6000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a015ec60 0x7f17a00b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a6000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a015f0e0 0x7f17a0156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a6000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a006c6b0 con 0x7f17a0141880 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a6000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b2b50 con 0x7f17a015ec60 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a6000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17a00b2da0 con 0x7f17a015f0e0 2024-01-15T18:02:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 0x7f17a00ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 0x7f17a00ad7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45330/0 (socket says 172.21.15.2:45330) 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 -- 172.21.15.2:0/4159178020 learned_addr learned my addr 172.21.15.2:0/4159178020 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 -- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a015f0e0 msgr2=0x7f17a0156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a015f0e0 0x7f17a0156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 -- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a015ec60 msgr2=0x7f17a00b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f179fe00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a015ec60 0x7f17a00b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a015ec60 0x7f17a00b2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.257+0000 7f17a4c00640 1 -- 172.21.15.2:0/4159178020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1798009a70 con 0x7f17a0141880 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.258+0000 7f17a4c00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 0x7f17a00ad7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f179000c4d0 tx=0x7f179000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.258+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790005ce0 con 0x7f17a0141880 2024-01-15T18:02:59.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.258+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1790005e40 con 0x7f17a0141880 2024-01-15T18:02:59.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.258+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1790007270 con 0x7f17a0141880 2024-01-15T18:02:59.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.258+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a01570d0 con 0x7f17a0141880 2024-01-15T18:02:59.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.258+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a00b37e0 con 0x7f17a0141880 2024-01-15T18:02:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.259+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a00b4e10 con 0x7f17a0141880 2024-01-15T18:02:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.260+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f17900073d0 con 0x7f17a0141880 2024-01-15T18:02:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.261+0000 7f179cc00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f177806b860 0x7f177806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.261+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f1790089720 con 0x7f17a0141880 2024-01-15T18:02:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.262+0000 7f179fe00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f177806b860 0x7f177806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.263+0000 7f179fe00640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f177806b860 0x7f177806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f17a00ad490 tx=0x7f1788009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:59.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.266+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1790054130 con 0x7f17a0141880 2024-01-15T18:02:59.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.455+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17a0074770 con 0x7f17a0141880 2024-01-15T18:02:59.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.457+0000 7f179cc00640 1 -- 172.21.15.2:0/4159178020 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+12884 (secure 0 0 0) 0x7f1790057750 con 0x7f17a0141880 2024-01-15T18:02:59.461 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:02:59.462 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":57,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:02:58.492316+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:02:59.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f177806b860 msgr2=0x7f177806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f177806b860 0x7f177806dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f17a00ad490 tx=0x7f1788009040 comp rx=0 tx=0).stop 2024-01-15T18:02:59.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 msgr2=0x7f17a00ad7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 0x7f17a00ad7a0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f179000c4d0 tx=0x7f179000c990 comp rx=0 tx=0).stop 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 shutdown_connections 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f177806b860 0x7f177806dd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17a015f0e0 0x7f17a0156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17a015ec60 0x7f17a00b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 --2- 172.21.15.2:0/4159178020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17a0141880 0x7f17a00ad7a0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.459+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 >> 172.21.15.2:0/4159178020 conn(0x7f17a00b0840 msgr2=0x7f17a00af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.460+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 shutdown_connections 2024-01-15T18:02:59.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.460+0000 7f17a6000640 1 -- 172.21.15.2:0/4159178020 wait complete. 2024-01-15T18:02:59.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:02:59.484 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:02:59.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.686+0000 7fba27800640 1 Processor -- start 2024-01-15T18:02:59.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.687+0000 7fba27800640 1 -- start start 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.688+0000 7fba27800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fba20144b90 0x7fba200b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.688+0000 7fba27800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fba2006e830 0x7fba2006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.688+0000 7fba27800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba20070430 con 0x7fba2015b840 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.688+0000 7fba27800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2006f1e0 con 0x7fba20144b90 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.688+0000 7fba27800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba200b4760 con 0x7fba2006e830 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fba20144b90 0x7fba200b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fba20144b90 0x7fba200b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45332/0 (socket says 172.21.15.2:45332) 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 -- 172.21.15.2:0/858481016 learned_addr learned my addr 172.21.15.2:0/858481016 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba25800640 1 --2- 172.21.15.2:0/858481016 >> v2:172.21.15.137:3300/0 conn(0x7fba2006e830 0x7fba2006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 -- 172.21.15.2:0/858481016 >> v2:172.21.15.137:3300/0 conn(0x7fba2006e830 msgr2=0x7fba2006eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:59.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 --2- 172.21.15.2:0/858481016 >> v2:172.21.15.137:3300/0 conn(0x7fba2006e830 0x7fba2006eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 -- 172.21.15.2:0/858481016 >> v1:172.21.15.137:6789/0 conn(0x7fba2015b840 legacy=0x7fba2006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:02:59.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba24e00640 1 -- 172.21.15.2:0/858481016 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20144f70 con 0x7fba20144b90 2024-01-15T18:02:59.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.689+0000 7fba25800640 1 --2- 172.21.15.2:0/858481016 >> v2:172.21.15.137:3300/0 conn(0x7fba2006e830 0x7fba2006eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.690+0000 7fba24e00640 1 --2- 172.21.15.2:0/858481016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba20144b90 0x7fba200b4220 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fba20155d00 tx=0x7fba1002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2f1777aaa093cb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.691+0000 7fba1f400640 1 -- 172.21.15.2:0/858481016 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1002ca80 con 0x7fba20144b90 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.691+0000 7fba1f400640 1 -- 172.21.15.2:0/858481016 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1002cbe0 con 0x7fba20144b90 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.691+0000 7fba1f400640 1 -- 172.21.15.2:0/858481016 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba100343f0 con 0x7fba20144b90 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.691+0000 7fba27800640 1 -- 172.21.15.2:0/858481016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba20144b90 msgr2=0x7fba200b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.691+0000 7fba27800640 1 --2- 172.21.15.2:0/858481016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba20144b90 0x7fba200b4220 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fba20155d00 tx=0x7fba1002bee0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.692+0000 7fba27800640 1 -- 172.21.15.2:0/858481016 shutdown_connections 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.692+0000 7fba27800640 1 --2- 172.21.15.2:0/858481016 >> v2:172.21.15.137:3300/0 conn(0x7fba2006e830 0x7fba2006eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.692+0000 7fba27800640 1 --2- 172.21.15.2:0/858481016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba20144b90 0x7fba200b4220 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.692+0000 7fba27800640 1 -- 172.21.15.2:0/858481016 >> 172.21.15.2:0/858481016 conn(0x7fba200b2c10 msgr2=0x7fba200b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:59.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.692+0000 7fba27800640 1 -- 172.21.15.2:0/858481016 shutdown_connections 2024-01-15T18:02:59.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.692+0000 7fba27800640 1 -- 172.21.15.2:0/858481016 wait complete. 2024-01-15T18:02:59.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.694+0000 7fba27800640 1 Processor -- start 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.694+0000 7fba27800640 1 -- start start 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba27800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fba2006e830 0x7fba20131830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba27800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba20144b90 0x7fba2015efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba27800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 0x7fba200d11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba27800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba20155950 con 0x7fba2015b840 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba27800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba201f32f0 con 0x7fba20144b90 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba27800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba20154cd0 con 0x7fba2006e830 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba25800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 0x7fba200d11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba25800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 0x7fba200d11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45344/0 (socket says 172.21.15.2:45344) 2024-01-15T18:02:59.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba25800640 1 -- 172.21.15.2:0/3720182578 learned_addr learned my addr 172.21.15.2:0/3720182578 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba24e00640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fba2006e830 0x7fba20131830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba25800640 1 -- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fba2006e830 msgr2=0x7fba20131830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.695+0000 7fba25800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fba2006e830 0x7fba20131830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba25800640 1 -- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba20144b90 msgr2=0x7fba2015efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba25800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba20144b90 0x7fba2015efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba25800640 1 -- 172.21.15.2:0/3720182578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba10009b30 con 0x7fba2015b840 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba25800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 0x7fba200d11e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fba1800c7f0 tx=0x7fba1800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18004230 con 0x7fba2015b840 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba18004390 con 0x7fba2015b840 2024-01-15T18:02:59.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba18007860 con 0x7fba2015b840 2024-01-15T18:02:59.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba20154fb0 con 0x7fba2015b840 2024-01-15T18:02:59.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.696+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba20155570 con 0x7fba2015b840 2024-01-15T18:02:59.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.697+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20155a90 con 0x7fba2015b840 2024-01-15T18:02:59.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.698+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fba18007bd0 con 0x7fba2015b840 2024-01-15T18:02:59.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.700+0000 7fba1cc00640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb9f806b860 0x7fb9f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:02:59.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.700+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4913+0+0 (secure 0 0 0) 0x7fba1808a8a0 con 0x7fba2015b840 2024-01-15T18:02:59.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.700+0000 7fba24e00640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb9f806b860 0x7fb9f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:02:59.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.701+0000 7fba24e00640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb9f806b860 0x7fb9f806dd20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fba10035000 tx=0x7fba10030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:02:59.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.704+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba20155a90 con 0x7fba2015b840 2024-01-15T18:02:59.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.901+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fba20155a90 con 0x7fba2015b840 2024-01-15T18:02:59.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.902+0000 7fba1cc00640 1 -- 172.21.15.2:0/3720182578 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7fba20155a90 con 0x7fba2015b840 2024-01-15T18:02:59.906 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb9f806b860 msgr2=0x7fb9f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb9f806b860 0x7fb9f806dd20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fba10035000 tx=0x7fba10030000 comp rx=0 tx=0).stop 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 msgr2=0x7fba200d11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 0x7fba200d11e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fba1800c7f0 tx=0x7fba1800ccb0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 shutdown_connections 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb9f806b860 0x7fb9f806dd20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fba2015b840 0x7fba200d11e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fba20144b90 0x7fba2015efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 --2- 172.21.15.2:0/3720182578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fba2006e830 0x7fba20131830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:02:59.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 >> 172.21.15.2:0/3720182578 conn(0x7fba200b2c10 msgr2=0x7fba200b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:02:59.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.904+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 shutdown_connections 2024-01-15T18:02:59.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:02:59.905+0000 7fba27800640 1 -- 172.21.15.2:0/3720182578 wait complete. 2024-01-15T18:02:59.928 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:03:00.090 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:03:00.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:03:00.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.140+0000 7f5861e00640 1 Processor -- start 2024-01-15T18:03:00.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.140+0000 7f5861e00640 1 -- start start 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5861e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f585c15d390 0x7f585c0ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5861e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c0d3bb0 0x7f585c0d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5861e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f585c0cfa50 0x7f585c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5861e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585c0d03c0 con 0x7f585c15d390 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5861e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585c0cf0f0 con 0x7f585c0cfa50 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5861e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585c0d0e80 con 0x7f585c0d3bb0 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f585aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f585c15d390 0x7f585c0ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f585aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f585c15d390 0x7f585c0ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45348/0 (socket says 172.21.15.2:45348) 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f585aa00640 1 -- 172.21.15.2:0/2856543794 learned_addr learned my addr 172.21.15.2:0/2856543794 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f585b400640 1 --2- 172.21.15.2:0/2856543794 >> v2:172.21.15.137:3300/0 conn(0x7f585c0cfa50 0x7f585c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.142+0000 7f5853e00640 1 --2- 172.21.15.2:0/2856543794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c0d3bb0 0x7f585c0d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.143+0000 7f585aa00640 1 -- 172.21.15.2:0/2856543794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c0d3bb0 msgr2=0x7f585c0d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.143+0000 7f585aa00640 1 --2- 172.21.15.2:0/2856543794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c0d3bb0 0x7f585c0d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.143+0000 7f585aa00640 1 -- 172.21.15.2:0/2856543794 >> v2:172.21.15.137:3300/0 conn(0x7f585c0cfa50 msgr2=0x7f585c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.143+0000 7f585aa00640 1 --2- 172.21.15.2:0/2856543794 >> v2:172.21.15.137:3300/0 conn(0x7f585c0cfa50 0x7f585c0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.143+0000 7f585aa00640 1 -- 172.21.15.2:0/2856543794 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585c0d1160 con 0x7f585c15d390 2024-01-15T18:03:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.144+0000 7f585aa00640 1 --2- 172.21.15.2:0/2856543794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c15d390 0x7f585c0ce8a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f585c0c6c20 tx=0x7f584c02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0630e938d3bb93 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.145+0000 7f585a000640 1 -- 172.21.15.2:0/2856543794 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c0317d0 con 0x7f585c15d390 2024-01-15T18:03:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.145+0000 7f585a000640 1 -- 172.21.15.2:0/2856543794 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c031930 con 0x7f585c15d390 2024-01-15T18:03:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.145+0000 7f585a000640 1 -- 172.21.15.2:0/2856543794 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c031cc0 con 0x7f585c15d390 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.145+0000 7f5861e00640 1 -- 172.21.15.2:0/2856543794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c15d390 msgr2=0x7f585c0ce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.145+0000 7f5861e00640 1 --2- 172.21.15.2:0/2856543794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c15d390 0x7f585c0ce8a0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f585c0c6c20 tx=0x7f584c02b870 comp rx=0 tx=0).stop 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 -- 172.21.15.2:0/2856543794 shutdown_connections 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 --2- 172.21.15.2:0/2856543794 >> v2:172.21.15.137:3300/0 conn(0x7f585c0cfa50 0x7f585c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 --2- 172.21.15.2:0/2856543794 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c0d3bb0 0x7f585c0d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 --2- 172.21.15.2:0/2856543794 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c15d390 0x7f585c0ce8a0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 -- 172.21.15.2:0/2856543794 >> 172.21.15.2:0/2856543794 conn(0x7f585c09c560 msgr2=0x7f585c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 -- 172.21.15.2:0/2856543794 shutdown_connections 2024-01-15T18:03:00.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.146+0000 7f5861e00640 1 -- 172.21.15.2:0/2856543794 wait complete. 2024-01-15T18:03:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.148+0000 7f5861e00640 1 Processor -- start 2024-01-15T18:03:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.148+0000 7f5861e00640 1 -- start start 2024-01-15T18:03:00.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5861e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 0x7f585c09ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5861e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f585c0d3bb0 0x7f585c0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5861e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c15d390 0x7f585c09cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5861e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c154ec0 con 0x7f585c0cfa50 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5861e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c09dc20 con 0x7f585c0d3bb0 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5861e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f585c09df00 con 0x7f585c15d390 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 0x7f585c09ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 0x7f585c09ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45358/0 (socket says 172.21.15.2:45358) 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 -- 172.21.15.2:0/889313155 learned_addr learned my addr 172.21.15.2:0/889313155 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f5853e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f585c0d3bb0 0x7f585c0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585b400640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c15d390 0x7f585c09cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 -- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c15d390 msgr2=0x7f585c09cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c15d390 0x7f585c09cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 -- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f585c0d3bb0 msgr2=0x7f585c0b6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f585c0d3bb0 0x7f585c0b6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.149+0000 7f585aa00640 1 -- 172.21.15.2:0/889313155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c0094d0 con 0x7f585c0cfa50 2024-01-15T18:03:00.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.150+0000 7f585aa00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 0x7f585c09ecc0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f584c000c00 tx=0x7f584c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.150+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c032870 con 0x7f585c0cfa50 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.150+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c0329d0 con 0x7f585c0cfa50 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.150+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585c09e1e0 con 0x7f585c0cfa50 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.150+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c032d80 con 0x7f585c0cfa50 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.150+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585c09e7a0 con 0x7f585c0cfa50 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.152+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585c09e990 con 0x7f585c0cfa50 2024-01-15T18:03:00.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.152+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f584c030070 con 0x7f585c0cfa50 2024-01-15T18:03:00.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.154+0000 7f5853400640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f583406b860 0x7f583406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.154+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f584c0ac220 con 0x7f585c0cfa50 2024-01-15T18:03:00.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.155+0000 7f5853e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f583406b860 0x7f583406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.156+0000 7f5853e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f583406b860 0x7f583406dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f585c1f6770 tx=0x7f5844009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:00.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.158+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f584c076ac0 con 0x7f585c0cfa50 2024-01-15T18:03:00.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.356+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f585c0686f0 con 0x7f585c0cfa50 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.357+0000 7f5853400640 1 -- 172.21.15.2:0/889313155 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+65 (secure 0 0 0) 0x7f584c031ed0 con 0x7f585c0cfa50 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T18:03:00.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:03:00.363 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f583406b860 msgr2=0x7f583406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f583406b860 0x7f583406dd20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f585c1f6770 tx=0x7f5844009190 comp rx=0 tx=0).stop 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 msgr2=0x7f585c09ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 0x7f585c09ecc0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f584c000c00 tx=0x7f584c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 shutdown_connections 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f583406b860 0x7f583406dd20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f585c15d390 0x7f585c09cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f585c0d3bb0 0x7f585c0b6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 --2- 172.21.15.2:0/889313155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f585c0cfa50 0x7f585c09ecc0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 >> 172.21.15.2:0/889313155 conn(0x7f585c09c560 msgr2=0x7f585c0b7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:00.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.360+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 shutdown_connections 2024-01-15T18:03:00.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.361+0000 7f5861e00640 1 -- 172.21.15.2:0/889313155 wait complete. 2024-01-15T18:03:00.384 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:03:00.384 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:03:00.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.604+0000 7f01b4a00640 1 Processor -- start 2024-01-15T18:03:00.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.604+0000 7f01b4a00640 1 -- start start 2024-01-15T18:03:00.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.606+0000 7f01b4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f01b006c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.606+0000 7f01b4a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0071090 con 0x7f01b0143d20 2024-01-15T18:03:00.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.606+0000 7f01b4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01b0144560 con 0x7f01b015f260 2024-01-15T18:03:00.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.606+0000 7f01b4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0071430 con 0x7f01b00720d0 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.606+0000 7f01ae800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f01b0143d20 0x7f01b015eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56982/0 (socket says 172.21.15.2:56982) 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.606+0000 7f01ae800640 1 -- 172.21.15.2:0/2092303885 learned_addr learned my addr 172.21.15.2:0/2092303885 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481138308 0 0) 0x7f01b0071090 con 0x7f01b0143d20 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003520 con 0x7f01b0143d20 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01af200640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f01b006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1064958148 0 0) 0x7f0190003520 con 0x7f01b0143d20 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 msgr2=0x7f01b006c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f01b006c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 >> v1:172.21.15.137:6789/0 conn(0x7f01b015f260 legacy=0x7f01b006c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b0144280 con 0x7f01b0143d20 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.607+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f01a0001670 con 0x7f01b0143d20 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 >> v1:172.21.15.2:6789/0 conn(0x7f01b0143d20 legacy=0x7f01b015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01900039a0 0x7f0190003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f019000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b0071430 con 0x7f01900047c0 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01900055c0 con 0x7f01900039a0 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019000bec0 con 0x7f01b00720d0 2024-01-15T18:03:00.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01a7e00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f0190004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01ae800640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01900039a0 0x7f0190003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01af200640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f019000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01a7e00640 1 -- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 msgr2=0x7f019000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01a7e00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f019000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01a7e00640 1 -- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01900039a0 msgr2=0x7f0190003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01a7e00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01900039a0 0x7f0190003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01a7e00640 1 -- 172.21.15.2:0/2092303885 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190003520 con 0x7f01900047c0 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.608+0000 7f01af200640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b00720d0 0x7f019000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:00.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01a7e00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f0190004c40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0198001750 tx=0x7f01900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4249e5d4f9f9b92a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:00.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980052d0 con 0x7f01900047c0 2024-01-15T18:03:00.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019802e050 con 0x7f01900047c0 2024-01-15T18:03:00.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01ade00640 1 -- 172.21.15.2:0/2092303885 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01980035f0 con 0x7f01900047c0 2024-01-15T18:03:00.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01b4a00640 1 -- 172.21.15.2:0/2092303885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 msgr2=0x7f0190004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01b4a00640 1 --2- 172.21.15.2:0/2092303885 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f0190004c40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f0198001750 tx=0x7f01900057f0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.610+0000 7f01af200640 1 -- 172.21.15.2:0/2092303885 reap_dead start 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.611+0000 7f01b4a00640 1 -- 172.21.15.2:0/2092303885 shutdown_connections 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.611+0000 7f01b4a00640 1 -- 172.21.15.2:0/2092303885 >> 172.21.15.2:0/2092303885 conn(0x7f01b00b0840 msgr2=0x7f01b00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.611+0000 7f01b4a00640 1 -- 172.21.15.2:0/2092303885 shutdown_connections 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.611+0000 7f01b4a00640 1 -- 172.21.15.2:0/2092303885 wait complete. 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.612+0000 7f01b4a00640 1 Processor -- start 2024-01-15T18:03:00.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.613+0000 7f01b4a00640 1 -- start start 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.613+0000 7f01b4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f01b006c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01b4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b015c2b0 0x7f01b015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01b4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01b015d560 0x7f01b014df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01b4a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b015dae0 con 0x7f01900047c0 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01b4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b015b4f0 con 0x7f01b015d560 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01b4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01b015b7d0 con 0x7f01b015c2b0 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f01b006c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f01b006c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45364/0 (socket says 172.21.15.2:45364) 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 -- 172.21.15.2:0/1763652991 learned_addr learned my addr 172.21.15.2:0/1763652991 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:00.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 -- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b015c2b0 msgr2=0x7f01b015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b015c2b0 0x7f01b015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 -- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01b015d560 msgr2=0x7f01b014df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01af200640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01b015d560 0x7f01b014df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01b015d560 0x7f01b014df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.614+0000 7f01ae800640 1 -- 172.21.15.2:0/1763652991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b015bab0 con 0x7f01900047c0 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.615+0000 7f01ae800640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f01b006c640 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f01a0001eb0 tx=0x7f01a000ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.615+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a000c5f0 con 0x7f01900047c0 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.615+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a000c770 con 0x7f01900047c0 2024-01-15T18:03:00.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.615+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a000cb20 con 0x7f01900047c0 2024-01-15T18:03:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.615+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b015bd90 con 0x7f01900047c0 2024-01-15T18:03:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.615+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b006cb80 con 0x7f01900047c0 2024-01-15T18:03:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.616+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b006cd70 con 0x7f01900047c0 2024-01-15T18:03:00.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.617+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f01a000cc80 con 0x7f01900047c0 2024-01-15T18:03:00.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.618+0000 7f01a6a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f018806b890 0x7f018806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:00.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.618+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4913+0+0 (secure 0 0 0) 0x7f01a008a250 con 0x7f01900047c0 2024-01-15T18:03:00.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.619+0000 7f01a7e00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f018806b890 0x7f018806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:00.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.620+0000 7f01a7e00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f018806b890 0x7f018806dd50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0198033000 tx=0x7f019802f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:00.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.622+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01a0054b20 con 0x7f01900047c0 2024-01-15T18:03:00.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:00.818+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f01b015bf80 con 0x7f01900047c0 2024-01-15T18:03:01.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.501+0000 7f01a6a00640 1 -- 172.21.15.2:0/1763652991 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v58) v1 ==== 138+0+0 (secure 0 0 0) 0x7f01a0058140 con 0x7f01900047c0 2024-01-15T18:03:01.506 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f018806b890 msgr2=0x7f018806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f018806b890 0x7f018806dd50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0198033000 tx=0x7f019802f000 comp rx=0 tx=0).stop 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 msgr2=0x7f01b006c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f01b006c640 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f01a0001eb0 tx=0x7f01a000ead0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 shutdown_connections 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f018806b890 0x7f018806dd50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f01b015d560 0x7f01b014df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01b015c2b0 0x7f01b015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 --2- 172.21.15.2:0/1763652991 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01900047c0 0x7f01b006c640 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 >> 172.21.15.2:0/1763652991 conn(0x7f01b00b0840 msgr2=0x7f01b01413d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:01.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 shutdown_connections 2024-01-15T18:03:01.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.505+0000 7f01b4a00640 1 -- 172.21.15.2:0/1763652991 wait complete. 2024-01-15T18:03:01.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:03:01.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.753+0000 7fcf92600640 1 Processor -- start 2024-01-15T18:03:01.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.754+0000 7fcf92600640 1 -- start start 2024-01-15T18:03:01.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf92600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcf8c0d1870 0x7fcf8c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf92600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0b2210 0x7fcf8c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf92600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcf8c0d1c50 0x7fcf8c15d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf92600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8c0ce6d0 con 0x7fcf8c0d1870 2024-01-15T18:03:01.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf92600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf8c15db60 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf92600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c0d1250 con 0x7fcf8c0b2210 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf91200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcf8c0d1870 0x7fcf8c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf91200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcf8c0d1870 0x7fcf8c0d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45372/0 (socket says 172.21.15.2:45372) 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.755+0000 7fcf91200640 1 -- 172.21.15.2:0/1337182592 learned_addr learned my addr 172.21.15.2:0/1337182592 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.756+0000 7fcf91200640 1 -- 172.21.15.2:0/1337182592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0b2210 msgr2=0x7fcf8c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.756+0000 7fcf91200640 1 --2- 172.21.15.2:0/1337182592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0b2210 0x7fcf8c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.756+0000 7fcf91200640 1 -- 172.21.15.2:0/1337182592 >> v2:172.21.15.137:3300/0 conn(0x7fcf8c0d1c50 msgr2=0x7fcf8c15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.756+0000 7fcf91200640 1 --2- 172.21.15.2:0/1337182592 >> v2:172.21.15.137:3300/0 conn(0x7fcf8c0d1c50 0x7fcf8c15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.756+0000 7fcf91200640 1 -- 172.21.15.2:0/1337182592 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf8c0ce8e0 con 0x7fcf8c0d1870 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.757+0000 7fcf91200640 1 --2- 172.21.15.2:0/1337182592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1870 0x7fcf8c0d3bd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c161120 tx=0x7fcf7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d86cd886c2471a83 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.757+0000 7fcf8b400640 1 -- 172.21.15.2:0/1337182592 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c02ca70 con 0x7fcf8c0d1870 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.757+0000 7fcf8b400640 1 -- 172.21.15.2:0/1337182592 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c02cbd0 con 0x7fcf8c0d1870 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.757+0000 7fcf8b400640 1 -- 172.21.15.2:0/1337182592 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c0343f0 con 0x7fcf8c0d1870 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.757+0000 7fcf92600640 1 -- 172.21.15.2:0/1337182592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1870 msgr2=0x7fcf8c0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.757+0000 7fcf92600640 1 --2- 172.21.15.2:0/1337182592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1870 0x7fcf8c0d3bd0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c161120 tx=0x7fcf7c02bed0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 -- 172.21.15.2:0/1337182592 shutdown_connections 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 --2- 172.21.15.2:0/1337182592 >> v2:172.21.15.137:3300/0 conn(0x7fcf8c0d1c50 0x7fcf8c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 --2- 172.21.15.2:0/1337182592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0b2210 0x7fcf8c15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 --2- 172.21.15.2:0/1337182592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1870 0x7fcf8c0d3bd0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 -- 172.21.15.2:0/1337182592 >> 172.21.15.2:0/1337182592 conn(0x7fcf8c0d30b0 msgr2=0x7fcf8c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:01.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 -- 172.21.15.2:0/1337182592 shutdown_connections 2024-01-15T18:03:01.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.758+0000 7fcf92600640 1 -- 172.21.15.2:0/1337182592 wait complete. 2024-01-15T18:03:01.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.760+0000 7fcf92600640 1 Processor -- start 2024-01-15T18:03:01.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.760+0000 7fcf92600640 1 -- start start 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.761+0000 7fcf92600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf8c0b2210 0x7fcf8c1f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.761+0000 7fcf92600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0d1870 0x7fcf8c1f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.761+0000 7fcf92600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 0x7fcf8c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.761+0000 7fcf92600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c1f3ea0 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.761+0000 7fcf92600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c160760 con 0x7fcf8c0b2210 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.761+0000 7fcf92600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8c09d550 con 0x7fcf8c0d1870 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 0x7fcf8c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 0x7fcf8c09ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45374/0 (socket says 172.21.15.2:45374) 2024-01-15T18:03:01.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 -- 172.21.15.2:0/274472980 learned_addr learned my addr 172.21.15.2:0/274472980 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:01.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf8be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0d1870 0x7fcf8c1f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:01.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91200640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf8c0b2210 0x7fcf8c1f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 -- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0d1870 msgr2=0x7fcf8c1f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0d1870 0x7fcf8c1f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 -- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf8c0b2210 msgr2=0x7fcf8c1f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf8c0b2210 0x7fcf8c1f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 -- 172.21.15.2:0/274472980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c009b30 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.762+0000 7fcf91c00640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 0x7fcf8c09ef80 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400b410 tx=0x7fcf8400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.763+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84011830 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.763+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf840119b0 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.763+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84011d60 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.763+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8c09d830 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.763+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8c09ddf0 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.764+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8c074770 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.765+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fcf8401a400 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.766+0000 7fcf88c00640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf6406b860 0x7fcf6406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:01.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.767+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5283+0+0 (secure 0 0 0) 0x7fcf8408a400 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.767+0000 7fcf91200640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf6406b860 0x7fcf6406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:01.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.768+0000 7fcf91200640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf6406b860 0x7fcf6406dd20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c035000 tx=0x7fcf7c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:01.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.772+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf84054c30 con 0x7fcf8c0d1c50 2024-01-15T18:03:01.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:01.973+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcf8c0d2e90 con 0x7fcf8c0d1c50 2024-01-15T18:03:02.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.506+0000 7fcf88c00640 1 -- 172.21.15.2:0/274472980 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v59) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcf84058250 con 0x7fcf8c0d1c50 2024-01-15T18:03:02.512 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T18:03:02.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.511+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf6406b860 msgr2=0x7fcf6406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:02.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.511+0000 7fcf92600640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf6406b860 0x7fcf6406dd20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c035000 tx=0x7fcf7c030000 comp rx=0 tx=0).stop 2024-01-15T18:03:02.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.511+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 msgr2=0x7fcf8c09ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:02.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.511+0000 7fcf92600640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 0x7fcf8c09ef80 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcf8400b410 tx=0x7fcf8400b810 comp rx=0 tx=0).stop 2024-01-15T18:03:02.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.512+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 shutdown_connections 2024-01-15T18:03:02.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.512+0000 7fcf92600640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf6406b860 0x7fcf6406dd20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.512+0000 7fcf92600640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf8c0d1c50 0x7fcf8c09ef80 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.512+0000 7fcf92600640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf8c0d1870 0x7fcf8c1f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.512+0000 7fcf92600640 1 --2- 172.21.15.2:0/274472980 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf8c0b2210 0x7fcf8c1f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.512+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 >> 172.21.15.2:0/274472980 conn(0x7fcf8c0d30b0 msgr2=0x7fcf8c0b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:02.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.513+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 shutdown_connections 2024-01-15T18:03:02.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.513+0000 7fcf92600640 1 -- 172.21.15.2:0/274472980 wait complete. 2024-01-15T18:03:02.557 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:03:02.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.739+0000 7f1d40200640 1 Processor -- start 2024-01-15T18:03:02.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.740+0000 7f1d40200640 1 -- start start 2024-01-15T18:03:02.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.741+0000 7f1d40200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1d380b3a50 0x7f1d38161650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.741+0000 7f1d40200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 0x7f1d3806e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.741+0000 7f1d40200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d381611c0 con 0x7f1d3815d4a0 2024-01-15T18:03:02.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.741+0000 7f1d40200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3815e920 con 0x7f1d380b3a50 2024-01-15T18:03:02.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.741+0000 7f1d40200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3815e080 con 0x7f1d38161b90 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.742+0000 7f1d3d800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1d3815d4a0 0x7f1d3815d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56984/0 (socket says 172.21.15.2:56984) 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.742+0000 7f1d3d800640 1 -- 172.21.15.2:0/992768147 learned_addr learned my addr 172.21.15.2:0/992768147 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.742+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564822885 0 0) 0x7f1d381611c0 con 0x7f1d3815d4a0 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.742+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/992768147 >> v2:172.21.15.137:3300/0 conn(0x7f1d380b3a50 0x7f1d38161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.742+0000 7f1d3e200640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 0x7f1d3806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.742+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d18003520 con 0x7f1d3815d4a0 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084316407 0 0) 0x7f1d18003520 con 0x7f1d3815d4a0 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 msgr2=0x7f1d3806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 0x7f1d3806e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 >> v2:172.21.15.137:3300/0 conn(0x7f1d380b3a50 msgr2=0x7f1d38161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 --2- 172.21.15.2:0/992768147 >> v2:172.21.15.137:3300/0 conn(0x7f1d380b3a50 0x7f1d38161650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3815e360 con 0x7f1d3815d4a0 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1d28001670 con 0x7f1d3815d4a0 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 >> v1:172.21.15.2:6789/0 conn(0x7f1d3815d4a0 legacy=0x7f1d3815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d18003970 0x7f1d18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 0x7f1d18005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3815e920 con 0x7f1d180047c0 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3815e080 con 0x7f1d18003970 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1800ba00 con 0x7f1d38161b90 2024-01-15T18:03:02.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.743+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3e200640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 0x7f1d18005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3d800640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d18003970 0x7f1d18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3ce00640 1 -- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 msgr2=0x7f1d18005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d38161b90 0x7f1d18005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3ce00640 1 -- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d18003970 msgr2=0x7f1d18003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d18003970 0x7f1d18003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.744+0000 7f1d3ce00640 1 -- 172.21.15.2:0/992768147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d18003520 con 0x7f1d180047c0 2024-01-15T18:03:02.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d18004c40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ac0 tx=0x7f1d180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f2416c8f819dfe2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2002c710 con 0x7f1d180047c0 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2002c870 con 0x7f1d180047c0 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d37e00640 1 -- 172.21.15.2:0/992768147 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2002cc00 con 0x7f1d180047c0 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d40200640 1 -- 172.21.15.2:0/992768147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 msgr2=0x7f1d18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d40200640 1 --2- 172.21.15.2:0/992768147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d18004c40 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ac0 tx=0x7f1d180057f0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.745+0000 7f1d3e200640 1 -- 172.21.15.2:0/992768147 reap_dead start 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.746+0000 7f1d40200640 1 -- 172.21.15.2:0/992768147 shutdown_connections 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.746+0000 7f1d40200640 1 -- 172.21.15.2:0/992768147 >> 172.21.15.2:0/992768147 conn(0x7f1d380b1c70 msgr2=0x7f1d380b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:02.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.746+0000 7f1d40200640 1 -- 172.21.15.2:0/992768147 shutdown_connections 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.746+0000 7f1d40200640 1 -- 172.21.15.2:0/992768147 wait complete. 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.747+0000 7f1d40200640 1 Processor -- start 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.748+0000 7f1d40200640 1 -- start start 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.748+0000 7f1d40200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d381580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.748+0000 7f1d40200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d380ac870 0x7f1d380af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d40200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d380aded0 0x7f1d380ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d40200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d380ae9b0 con 0x7f1d180047c0 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d40200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d380ada40 con 0x7f1d380ac870 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d40200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d380ace30 con 0x7f1d380aded0 2024-01-15T18:03:02.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d381580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d381580a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45402/0 (socket says 172.21.15.2:45402) 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 -- 172.21.15.2:0/2292847828 learned_addr learned my addr 172.21.15.2:0/2292847828 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3e200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d380aded0 0x7f1d380ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d380ac870 0x7f1d380af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 -- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d380aded0 msgr2=0x7f1d380ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d380aded0 0x7f1d380ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 -- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d380ac870 msgr2=0x7f1d380af550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d380ac870 0x7f1d380af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.749+0000 7f1d3d800640 1 -- 172.21.15.2:0/2292847828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d380ad110 con 0x7f1d180047c0 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.750+0000 7f1d3d800640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d381580a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800d7a0 tx=0x7f1d2800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:02.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.750+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d280074e0 con 0x7f1d180047c0 2024-01-15T18:03:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.750+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d28007660 con 0x7f1d180047c0 2024-01-15T18:03:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.750+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d380ad3f0 con 0x7f1d180047c0 2024-01-15T18:03:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.750+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d28004a20 con 0x7f1d180047c0 2024-01-15T18:03:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.750+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d380ae350 con 0x7f1d180047c0 2024-01-15T18:03:02.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.751+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d380b53d0 con 0x7f1d180047c0 2024-01-15T18:03:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.753+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f1d28004b80 con 0x7f1d180047c0 2024-01-15T18:03:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.754+0000 7f1d35600640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d1006b890 0x7f1d1006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.754+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5665+0+0 (secure 0 0 0) 0x7f1d2808a3b0 con 0x7f1d180047c0 2024-01-15T18:03:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.754+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d1006b890 0x7f1d1006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:02.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.756+0000 7f1d3ce00640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d1006b890 0x7f1d1006dd50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ac0 tx=0x7f1d20030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:02.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.760+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d28054880 con 0x7f1d180047c0 2024-01-15T18:03:02.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:02.992+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1d380686f0 con 0x7f1d180047c0 2024-01-15T18:03:03.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.527+0000 7f1d35600640 1 -- 172.21.15.2:0/2292847828 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v32) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1d28057ea0 con 0x7f1d180047c0 2024-01-15T18:03:03.532 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 8 and data pool 9 2024-01-15T18:03:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.533+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d1006b890 msgr2=0x7f1d1006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.533+0000 7f1d40200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d1006b890 0x7f1d1006dd50 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ac0 tx=0x7f1d20030000 comp rx=0 tx=0).stop 2024-01-15T18:03:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.533+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 msgr2=0x7f1d381580a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.533+0000 7f1d40200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d381580a0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f1d2800d7a0 tx=0x7f1d2800dc60 comp rx=0 tx=0).stop 2024-01-15T18:03:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.534+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 shutdown_connections 2024-01-15T18:03:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.534+0000 7f1d40200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d1006b890 0x7f1d1006dd50 secure :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1d20009ac0 tx=0x7f1d20030000 comp rx=0 tx=0).stop 2024-01-15T18:03:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.534+0000 7f1d40200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d380aded0 0x7f1d380ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.534+0000 7f1d40200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d380ac870 0x7f1d380af550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.534+0000 7f1d40200640 1 --2- 172.21.15.2:0/2292847828 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d180047c0 0x7f1d381580a0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.534+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 >> 172.21.15.2:0/2292847828 conn(0x7f1d380b1c70 msgr2=0x7f1d380b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.535+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 shutdown_connections 2024-01-15T18:03:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.536+0000 7f1d40200640 1 -- 172.21.15.2:0/2292847828 wait complete. 2024-01-15T18:03:03.579 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:03:03.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.758+0000 7f2442200640 1 Processor -- start 2024-01-15T18:03:03.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.758+0000 7f2442200640 1 -- start start 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.759+0000 7f2442200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c0e0 0x7f243c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.759+0000 7f2442200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f243c15b7d0 0x7f243c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.759+0000 7f2442200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f243c15cb00 con 0x7f243c15c4c0 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.759+0000 7f2442200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f243c0ccec0 con 0x7f243c15b7d0 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.759+0000 7f2442200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243c0cd1a0 con 0x7f243c15c0e0 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f2432800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f243c15b7d0 0x7f243c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f243be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f243c15c4c0 0x7f243c15ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38092/0 (socket says 172.21.15.2:38092) 2024-01-15T18:03:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f243be00640 1 -- 172.21.15.2:0/1636394988 learned_addr learned my addr 172.21.15.2:0/1636394988 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f243b400640 1 --2- 172.21.15.2:0/1636394988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c0e0 0x7f243c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f2432800640 1 -- 172.21.15.2:0/1636394988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c0e0 msgr2=0x7f243c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f2432800640 1 --2- 172.21.15.2:0/1636394988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c0e0 0x7f243c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f2432800640 1 -- 172.21.15.2:0/1636394988 >> v1:172.21.15.2:6789/0 conn(0x7f243c15c4c0 legacy=0x7f243c15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.760+0000 7f2432800640 1 -- 172.21.15.2:0/1636394988 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c0cd480 con 0x7f243c15b7d0 2024-01-15T18:03:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.761+0000 7f2432800640 1 --2- 172.21.15.2:0/1636394988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c06c470 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f24240097a0 tx=0x7f242402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=835d15a99b0051c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f243aa00640 1 -- 172.21.15.2:0/1636394988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242402d040 con 0x7f243c15b7d0 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f243aa00640 1 -- 172.21.15.2:0/1636394988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2424031810 con 0x7f243c15b7d0 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f243aa00640 1 -- 172.21.15.2:0/1636394988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424031ba0 con 0x7f243c15b7d0 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f2442200640 1 -- 172.21.15.2:0/1636394988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 msgr2=0x7f243c06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f2442200640 1 --2- 172.21.15.2:0/1636394988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c06c470 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f24240097a0 tx=0x7f242402b8b0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f2442200640 1 -- 172.21.15.2:0/1636394988 shutdown_connections 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.762+0000 7f2442200640 1 --2- 172.21.15.2:0/1636394988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c06c470 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.763+0000 7f2442200640 1 --2- 172.21.15.2:0/1636394988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c0e0 0x7f243c0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.763+0000 7f2442200640 1 -- 172.21.15.2:0/1636394988 >> 172.21.15.2:0/1636394988 conn(0x7f243c06da20 msgr2=0x7f243c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.763+0000 7f2442200640 1 -- 172.21.15.2:0/1636394988 shutdown_connections 2024-01-15T18:03:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.763+0000 7f2442200640 1 -- 172.21.15.2:0/1636394988 wait complete. 2024-01-15T18:03:03.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.764+0000 7f2442200640 1 Processor -- start 2024-01-15T18:03:03.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 -- start start 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c145900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f243c15c0e0 0x7f243c152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c4c0 0x7f243c1e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c153d40 con 0x7f243c15b7d0 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c153130 con 0x7f243c15c0e0 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.765+0000 7f2442200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243c153410 con 0x7f243c15c4c0 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c145900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c145900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56458/0 (socket says 172.21.15.2:56458) 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 -- 172.21.15.2:0/3939384784 learned_addr learned my addr 172.21.15.2:0/3939384784 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:03.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f2432800640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f243c15c0e0 0x7f243c152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243be00640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c4c0 0x7f243c1e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 -- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c4c0 msgr2=0x7f243c1e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c4c0 0x7f243c1e6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 -- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f243c15c0e0 msgr2=0x7f243c152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f243c15c0e0 0x7f243c152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f243b400640 1 -- 172.21.15.2:0/3939384784 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424009400 con 0x7f243c15b7d0 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.766+0000 7f2432800640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f243c15c0e0 0x7f243c152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.767+0000 7f243b400640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c145900 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f242c00d800 tx=0x7f242c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.767+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c00b6c0 con 0x7f243c15b7d0 2024-01-15T18:03:03.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.767+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f242c00b820 con 0x7f243c15b7d0 2024-01-15T18:03:03.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.767+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243c1536f0 con 0x7f243c15b7d0 2024-01-15T18:03:03.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.767+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242c00bbb0 con 0x7f243c15b7d0 2024-01-15T18:03:03.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.767+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f243c155880 con 0x7f243c15b7d0 2024-01-15T18:03:03.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.768+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f243c0686f0 con 0x7f243c15b7d0 2024-01-15T18:03:03.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.770+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f242c00bd10 con 0x7f243c15b7d0 2024-01-15T18:03:03.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.770+0000 7f2433e00640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f241406b860 0x7f241406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:03.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.771+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f242c08ac80 con 0x7f243c15b7d0 2024-01-15T18:03:03.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.771+0000 7f2432800640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f241406b860 0x7f241406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:03.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.773+0000 7f2432800640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f241406b860 0x7f241406dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2424036000 tx=0x7f242403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:03.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.775+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f242c055200 con 0x7f243c15b7d0 2024-01-15T18:03:03.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.971+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f243c06b260 con 0x7f243c15b7d0 2024-01-15T18:03:03.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.972+0000 7f2433e00640 1 -- 172.21.15.2:0/3939384784 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16605 (secure 0 0 0) 0x7f242c058820 con 0x7f243c15b7d0 2024-01-15T18:03:03.978 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:03.978 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":60,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:03:02.994259+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:03:00.876169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-15T18:03:02.030400+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f241406b860 msgr2=0x7f241406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f241406b860 0x7f241406dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2424036000 tx=0x7f242403c040 comp rx=0 tx=0).stop 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 msgr2=0x7f243c145900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c145900 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f242c00d800 tx=0x7f242c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 shutdown_connections 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f241406b860 0x7f241406dd20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f243c15c4c0 0x7f243c1e6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f243c15c0e0 0x7f243c152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 --2- 172.21.15.2:0/3939384784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f243c15b7d0 0x7f243c145900 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:03.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.975+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 >> 172.21.15.2:0/3939384784 conn(0x7f243c06da20 msgr2=0x7f243c15acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:03.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.976+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 shutdown_connections 2024-01-15T18:03:03.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:03.976+0000 7f2442200640 1 -- 172.21.15.2:0/3939384784 wait complete. 2024-01-15T18:03:03.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:03:04.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.211+0000 7fda82600640 1 Processor -- start 2024-01-15T18:03:04.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.211+0000 7fda82600640 1 -- start start 2024-01-15T18:03:04.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.212+0000 7fda82600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0cfad0 0x7fda7c0cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda82600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fda7c0d37f0 0x7fda7c0cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda82600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda7c0c6e40 con 0x7fda7c0d3c70 2024-01-15T18:03:04.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda82600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda7c0d2090 con 0x7fda7c0d37f0 2024-01-15T18:03:04.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda82600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c0d03f0 con 0x7fda7c0cfad0 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda7be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fda7c0d37f0 0x7fda7c0cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda7be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fda7c0d37f0 0x7fda7c0cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56464/0 (socket says 172.21.15.2:56464) 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda7be00640 1 -- 172.21.15.2:0/1228481595 learned_addr learned my addr 172.21.15.2:0/1228481595 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.213+0000 7fda81200640 1 --2- 172.21.15.2:0/1228481595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0cfad0 0x7fda7c0cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.214+0000 7fda7be00640 1 -- 172.21.15.2:0/1228481595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0cfad0 msgr2=0x7fda7c0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.214+0000 7fda7be00640 1 --2- 172.21.15.2:0/1228481595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0cfad0 0x7fda7c0cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.214+0000 7fda7be00640 1 -- 172.21.15.2:0/1228481595 >> v1:172.21.15.137:6789/0 conn(0x7fda7c0d3c70 legacy=0x7fda7c0ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.214+0000 7fda7be00640 1 -- 172.21.15.2:0/1228481595 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda7c0d17f0 con 0x7fda7c0d37f0 2024-01-15T18:03:04.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.215+0000 7fda7be00640 1 --2- 172.21.15.2:0/1228481595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d37f0 0x7fda7c0cdbd0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fda64009870 tx=0x7fda6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383f9d484fe04d74 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:04.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.215+0000 7fda7b400640 1 -- 172.21.15.2:0/1228481595 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64031640 con 0x7fda7c0d37f0 2024-01-15T18:03:04.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.215+0000 7fda7b400640 1 -- 172.21.15.2:0/1228481595 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda640317a0 con 0x7fda7c0d37f0 2024-01-15T18:03:04.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.215+0000 7fda7b400640 1 -- 172.21.15.2:0/1228481595 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64031b30 con 0x7fda7c0d37f0 2024-01-15T18:03:04.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 -- 172.21.15.2:0/1228481595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d37f0 msgr2=0x7fda7c0cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 --2- 172.21.15.2:0/1228481595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d37f0 0x7fda7c0cdbd0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fda64009870 tx=0x7fda6402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 -- 172.21.15.2:0/1228481595 shutdown_connections 2024-01-15T18:03:04.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 --2- 172.21.15.2:0/1228481595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d37f0 0x7fda7c0cdbd0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 --2- 172.21.15.2:0/1228481595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0cfad0 0x7fda7c0cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 -- 172.21.15.2:0/1228481595 >> 172.21.15.2:0/1228481595 conn(0x7fda7c09c3b0 msgr2=0x7fda7c0d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:04.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 -- 172.21.15.2:0/1228481595 shutdown_connections 2024-01-15T18:03:04.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.216+0000 7fda82600640 1 -- 172.21.15.2:0/1228481595 wait complete. 2024-01-15T18:03:04.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.218+0000 7fda82600640 1 Processor -- start 2024-01-15T18:03:04.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.218+0000 7fda82600640 1 -- start start 2024-01-15T18:03:04.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda82600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda7c0cfad0 0x7fda7c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda82600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0d37f0 0x7fda7c0b6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda82600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 0x7fda7c0b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda82600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c1600f0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda82600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c0cd120 con 0x7fda7c0cfad0 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda82600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda7c15f4e0 con 0x7fda7c0d37f0 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda81c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 0x7fda7c0b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda81c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 0x7fda7c0b7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56474/0 (socket says 172.21.15.2:56474) 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda81c00640 1 -- 172.21.15.2:0/1813743431 learned_addr learned my addr 172.21.15.2:0/1813743431 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda7be00640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0d37f0 0x7fda7c0b6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.219+0000 7fda81200640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda7c0cfad0 0x7fda7c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81c00640 1 -- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0d37f0 msgr2=0x7fda7c0b6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81c00640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0d37f0 0x7fda7c0b6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81c00640 1 -- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda7c0cfad0 msgr2=0x7fda7c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81c00640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda7c0cfad0 0x7fda7c0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81c00640 1 -- 172.21.15.2:0/1813743431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640094d0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81200640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda7c0cfad0 0x7fda7c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda81c00640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 0x7fda7c0b7390 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fda7400c6b0 tx=0x7fda7400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74003b70 con 0x7fda7c0d3c70 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda74003cf0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74003ea0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7c15f7c0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.220+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7c15fd80 con 0x7fda7c0d3c70 2024-01-15T18:03:04.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.222+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7c15ff10 con 0x7fda7c0d3c70 2024-01-15T18:03:04.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.222+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fda74005310 con 0x7fda7c0d3c70 2024-01-15T18:03:04.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.224+0000 7fda78c00640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda5406b860 0x7fda5406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.224+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fda7408a3a0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.225+0000 7fda81200640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda5406b860 0x7fda5406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.226+0000 7fda81200640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda5406b860 0x7fda5406dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fda6c006fd0 tx=0x7fda6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:04.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.228+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda740549f0 con 0x7fda7c0d3c70 2024-01-15T18:03:04.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.422+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda7c074770 con 0x7fda7c0d3c70 2024-01-15T18:03:04.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.424+0000 7fda78c00640 1 -- 172.21.15.2:0/1813743431 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16605 (secure 0 0 0) 0x7fda74058010 con 0x7fda7c0d3c70 2024-01-15T18:03:04.429 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:04.429 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":60,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:03:02.994259+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:03:00.876169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-15T18:03:02.030400+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:03:04.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.426+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda5406b860 msgr2=0x7fda5406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.426+0000 7fda82600640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda5406b860 0x7fda5406dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fda6c006fd0 tx=0x7fda6c008040 comp rx=0 tx=0).stop 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.426+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 msgr2=0x7fda7c0b7390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.426+0000 7fda82600640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 0x7fda7c0b7390 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fda7400c6b0 tx=0x7fda7400cb70 comp rx=0 tx=0).stop 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 shutdown_connections 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda5406b860 0x7fda5406dd20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda7c0d3c70 0x7fda7c0b7390 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda7c0d37f0 0x7fda7c0b6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 --2- 172.21.15.2:0/1813743431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda7c0cfad0 0x7fda7c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 >> 172.21.15.2:0/1813743431 conn(0x7fda7c09c3b0 msgr2=0x7fda7c0cf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:04.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 shutdown_connections 2024-01-15T18:03:04.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.427+0000 7fda82600640 1 -- 172.21.15.2:0/1813743431 wait complete. 2024-01-15T18:03:04.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:03:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.664+0000 7fbcee000640 1 Processor -- start 2024-01-15T18:03:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.664+0000 7fbcee000640 1 -- start start 2024-01-15T18:03:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcee000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbce8071230 0x7fbce80716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcee000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbce80cc3b0 0x7fbce80cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcee000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbce80b2fa0 con 0x7fbce8143620 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcee000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce8071bf0 con 0x7fbce8071230 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcee000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbce80b3280 con 0x7fbce80cc3b0 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcecc00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbce8143620 0x7fbce80b2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38100/0 (socket says 172.21.15.2:38100) 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbcecc00640 1 -- 172.21.15.2:0/1428730014 learned_addr learned my addr 172.21.15.2:0/1428730014 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbce7e00640 1 --2- 172.21.15.2:0/1428730014 >> v2:172.21.15.2:3300/0 conn(0x7fbce8071230 0x7fbce80716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.666+0000 7fbce7400640 1 -- 172.21.15.2:0/1428730014 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972106475 0 0) 0x7fbce80b2fa0 con 0x7fbce8143620 2024-01-15T18:03:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.667+0000 7fbced600640 1 --2- 172.21.15.2:0/1428730014 >> v2:172.21.15.137:3300/0 conn(0x7fbce80cc3b0 0x7fbce80cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.667+0000 7fbce7400640 1 -- 172.21.15.2:0/1428730014 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcc8005730 con 0x7fbce8143620 2024-01-15T18:03:04.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.667+0000 7fbce7e00640 1 -- 172.21.15.2:0/1428730014 >> v2:172.21.15.137:3300/0 conn(0x7fbce80cc3b0 msgr2=0x7fbce80cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:04.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.667+0000 7fbce7e00640 1 --2- 172.21.15.2:0/1428730014 >> v2:172.21.15.137:3300/0 conn(0x7fbce80cc3b0 0x7fbce80cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.667+0000 7fbce7e00640 1 -- 172.21.15.2:0/1428730014 >> v1:172.21.15.2:6789/0 conn(0x7fbce8143620 legacy=0x7fbce80b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.667+0000 7fbce7e00640 1 -- 172.21.15.2:0/1428730014 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce80b3560 con 0x7fbce8071230 2024-01-15T18:03:04.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.668+0000 7fbce7e00640 1 --2- 172.21.15.2:0/1428730014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8071230 0x7fbce80716b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbce806fc80 tx=0x7fbcd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72cc93ecded6c782 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:04.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.668+0000 7fbce7400640 1 -- 172.21.15.2:0/1428730014 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0031830 con 0x7fbce8071230 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.668+0000 7fbce7400640 1 -- 172.21.15.2:0/1428730014 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd0031990 con 0x7fbce8071230 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.668+0000 7fbce7400640 1 -- 172.21.15.2:0/1428730014 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0031d20 con 0x7fbce8071230 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 -- 172.21.15.2:0/1428730014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8071230 msgr2=0x7fbce80716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 --2- 172.21.15.2:0/1428730014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8071230 0x7fbce80716b0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbce806fc80 tx=0x7fbcd002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 -- 172.21.15.2:0/1428730014 shutdown_connections 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 --2- 172.21.15.2:0/1428730014 >> v2:172.21.15.137:3300/0 conn(0x7fbce80cc3b0 0x7fbce80cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 --2- 172.21.15.2:0/1428730014 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8071230 0x7fbce80716b0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 -- 172.21.15.2:0/1428730014 >> 172.21.15.2:0/1428730014 conn(0x7fbce80b0840 msgr2=0x7fbce80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 -- 172.21.15.2:0/1428730014 shutdown_connections 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.669+0000 7fbcee000640 1 -- 172.21.15.2:0/1428730014 wait complete. 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.671+0000 7fbcee000640 1 Processor -- start 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.671+0000 7fbcee000640 1 -- start start 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcee000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbce8071230 0x7fbce81579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcee000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbce80cc3b0 0x7fbce80cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcee000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 0x7fbce8141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcee000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce80ce8d0 con 0x7fbce8143620 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcee000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbce80cc820 con 0x7fbce80cc3b0 2024-01-15T18:03:04.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcee000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbce813bd70 con 0x7fbce8071230 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbced600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 0x7fbce8141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbced600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 0x7fbce8141ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56490/0 (socket says 172.21.15.2:56490) 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbced600640 1 -- 172.21.15.2:0/1516532013 learned_addr learned my addr 172.21.15.2:0/1516532013 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbce7e00640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbce80cc3b0 0x7fbce80cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.672+0000 7fbcecc00640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbce8071230 0x7fbce81579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbced600640 1 -- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbce8071230 msgr2=0x7fbce81579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbced600640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbce8071230 0x7fbce81579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbced600640 1 -- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbce80cc3b0 msgr2=0x7fbce80cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbced600640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbce80cc3b0 0x7fbce80cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:04.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbced600640 1 -- 172.21.15.2:0/1516532013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd0009400 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbced600640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 0x7fbce8141ee0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbce0007e40 tx=0x7fbce000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0014620 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce000ee90 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce00149e0 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce80ccac0 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.673+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbce80cd080 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.675+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbce8074770 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.675+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fbce0014b40 con 0x7fbce8143620 2024-01-15T18:03:04.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.677+0000 7fbce4c00640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbcc006b860 0x7fbcc006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:04.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.677+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fbce008afa0 con 0x7fbce8143620 2024-01-15T18:03:04.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.678+0000 7fbcecc00640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbcc006b860 0x7fbcc006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:04.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.679+0000 7fbcecc00640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbcc006b860 0x7fbcc006dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80011e0 tx=0x7fbcd8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:04.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.682+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbce00555f0 con 0x7fbce8143620 2024-01-15T18:03:04.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:04.913+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbce80cf820 con 0x7fbce8143620 2024-01-15T18:03:05.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.566+0000 7fbce4c00640 1 -- 172.21.15.2:0/1516532013 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbce0058c10 con 0x7fbce8143620 2024-01-15T18:03:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbcc006b860 msgr2=0x7fbcc006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbcc006b860 0x7fbcc006dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80011e0 tx=0x7fbcd8009040 comp rx=0 tx=0).stop 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 msgr2=0x7fbce8141ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 0x7fbce8141ee0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbce0007e40 tx=0x7fbce000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 shutdown_connections 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbcc006b860 0x7fbcc006dd20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbce8143620 0x7fbce8141ee0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbce80cc3b0 0x7fbce80cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 --2- 172.21.15.2:0/1516532013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbce8071230 0x7fbce81579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.570+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 >> 172.21.15.2:0/1516532013 conn(0x7fbce80b0840 msgr2=0x7fbce8159800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.571+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 shutdown_connections 2024-01-15T18:03:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.571+0000 7fbcee000640 1 -- 172.21.15.2:0/1516532013 wait complete. 2024-01-15T18:03:05.598 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:03:05.598 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:03:05.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.817+0000 7fbde5800640 1 Processor -- start 2024-01-15T18:03:05.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.817+0000 7fbde5800640 1 -- start start 2024-01-15T18:03:05.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbde5800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbde00720d0 0x7fbde006c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:05.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbde5800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde0071090 con 0x7fbde0143d20 2024-01-15T18:03:05.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbde5800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde0144560 con 0x7fbde015f260 2024-01-15T18:03:05.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbde5800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde0071430 con 0x7fbde00720d0 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbde4e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbde00720d0 0x7fbde006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbddfe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbde0143d20 0x7fbde015eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38102/0 (socket says 172.21.15.2:38102) 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.819+0000 7fbddfe00640 1 -- 172.21.15.2:0/2034283295 learned_addr learned my addr 172.21.15.2:0/2034283295 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.820+0000 7fbde4e00640 1 -- 172.21.15.2:0/2034283295 >> v1:172.21.15.137:6789/0 conn(0x7fbde015f260 legacy=0x7fbde006c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.820+0000 7fbde4e00640 1 -- 172.21.15.2:0/2034283295 >> v1:172.21.15.2:6789/0 conn(0x7fbde0143d20 legacy=0x7fbde015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.820+0000 7fbde4e00640 1 -- 172.21.15.2:0/2034283295 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde0144280 con 0x7fbde00720d0 2024-01-15T18:03:05.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.820+0000 7fbddea00640 1 -- 172.21.15.2:0/2034283295 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1700195147 0 0) 0x7fbde0071090 con 0x7fbde0143d20 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.821+0000 7fbde4e00640 1 --2- 172.21.15.2:0/2034283295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde00720d0 0x7fbde006c860 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80097a0 tx=0x7fbdd802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94baf728aabbdfa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbddea00640 1 -- 172.21.15.2:0/2034283295 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd802e070 con 0x7fbde00720d0 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbddea00640 1 -- 172.21.15.2:0/2034283295 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd80315a0 con 0x7fbde00720d0 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbddea00640 1 -- 172.21.15.2:0/2034283295 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8031930 con 0x7fbde00720d0 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 -- 172.21.15.2:0/2034283295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde00720d0 msgr2=0x7fbde006c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 --2- 172.21.15.2:0/2034283295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde00720d0 0x7fbde006c860 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80097a0 tx=0x7fbdd802b840 comp rx=0 tx=0).stop 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 -- 172.21.15.2:0/2034283295 shutdown_connections 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 --2- 172.21.15.2:0/2034283295 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde00720d0 0x7fbde006c860 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 -- 172.21.15.2:0/2034283295 >> 172.21.15.2:0/2034283295 conn(0x7fbde00b0840 msgr2=0x7fbde00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:05.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 -- 172.21.15.2:0/2034283295 shutdown_connections 2024-01-15T18:03:05.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.822+0000 7fbde5800640 1 -- 172.21.15.2:0/2034283295 wait complete. 2024-01-15T18:03:05.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.824+0000 7fbde5800640 1 Processor -- start 2024-01-15T18:03:05.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.824+0000 7fbde5800640 1 -- start start 2024-01-15T18:03:05.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.825+0000 7fbde5800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbde00720d0 0x7fbde00a89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.825+0000 7fbde5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 0x7fbde015d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.825+0000 7fbde5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbde015f260 0x7fbde013d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.825+0000 7fbde5800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde015e4e0 con 0x7fbde0143d20 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.825+0000 7fbde5800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00ad6b0 con 0x7fbde00720d0 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.825+0000 7fbde5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde015da30 con 0x7fbde015f260 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 0x7fbde015d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 0x7fbde015d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56516/0 (socket says 172.21.15.2:56516) 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 -- 172.21.15.2:0/2717021173 learned_addr learned my addr 172.21.15.2:0/2717021173 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddfe00640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbde00720d0 0x7fbde00a89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:05.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 -- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbde015f260 msgr2=0x7fbde013d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbde4e00640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbde015f260 0x7fbde013d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbde015f260 0x7fbde013d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 -- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbde00720d0 msgr2=0x7fbde00a89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbde00720d0 0x7fbde00a89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddf400640 1 -- 172.21.15.2:0/2717021173 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8009400 con 0x7fbde0143d20 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.826+0000 7fbddfe00640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbde00720d0 0x7fbde00a89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.827+0000 7fbddf400640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 0x7fbde015d4f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800b420 tx=0x7fbdc800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.827+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8011830 con 0x7fbde0143d20 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.827+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8011990 con 0x7fbde0143d20 2024-01-15T18:03:05.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.827+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8011d40 con 0x7fbde0143d20 2024-01-15T18:03:05.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.827+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde013dc00 con 0x7fbde0143d20 2024-01-15T18:03:05.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.827+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde015dc40 con 0x7fbde0143d20 2024-01-15T18:03:05.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.828+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde0074770 con 0x7fbde0143d20 2024-01-15T18:03:05.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.829+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fbdc801b460 con 0x7fbde0143d20 2024-01-15T18:03:05.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.830+0000 7fbdd7e00640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb806b860 0x7fbdb806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:05.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.830+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fbdc808a210 con 0x7fbde0143d20 2024-01-15T18:03:05.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.831+0000 7fbddfe00640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb806b860 0x7fbdb806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:05.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.832+0000 7fbddfe00640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb806b860 0x7fbdb806dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00012e0 tx=0x7fbdd0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:05.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:05.835+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdc80547d0 con 0x7fbde0143d20 2024-01-15T18:03:06.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.069+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbde00cc620 con 0x7fbde0143d20 2024-01-15T18:03:06.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.569+0000 7fbdd7e00640 1 -- 172.21.15.2:0/2717021173 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbdc8057df0 con 0x7fbde0143d20 2024-01-15T18:03:06.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb806b860 msgr2=0x7fbdb806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:06.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb806b860 0x7fbdb806dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbdd00012e0 tx=0x7fbdd0009040 comp rx=0 tx=0).stop 2024-01-15T18:03:06.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 msgr2=0x7fbde015d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:06.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 0x7fbde015d4f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800b420 tx=0x7fbdc800b8e0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 shutdown_connections 2024-01-15T18:03:06.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbdb806b860 0x7fbdb806dd20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbde015f260 0x7fbde013d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbde0143d20 0x7fbde015d4f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 --2- 172.21.15.2:0/2717021173 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbde00720d0 0x7fbde00a89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.573+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 >> 172.21.15.2:0/2717021173 conn(0x7fbde00b0840 msgr2=0x7fbde0142c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:06.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.574+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 shutdown_connections 2024-01-15T18:03:06.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.574+0000 7fbde5800640 1 -- 172.21.15.2:0/2717021173 wait complete. 2024-01-15T18:03:06.599 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:03:06.600 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:03:06.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.819+0000 7f5d93200640 1 Processor -- start 2024-01-15T18:03:06.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.819+0000 7f5d93200640 1 -- start start 2024-01-15T18:03:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.821+0000 7f5d93200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5d9406e1e0 0x7f5d940b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.821+0000 7f5d93200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d940cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.821+0000 7f5d93200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d940ccbe0 con 0x7f5d94143810 2024-01-15T18:03:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.821+0000 7f5d93200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d940b3520 con 0x7f5d9406e1e0 2024-01-15T18:03:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.821+0000 7f5d93200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d94143bf0 con 0x7f5d940b2080 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d940cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d940cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:36076/0 (socket says 172.21.15.2:36076) 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 -- 172.21.15.2:0/2642195173 learned_addr learned my addr 172.21.15.2:0/2642195173 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d89200640 1 --2- 172.21.15.2:0/2642195173 >> v2:172.21.15.137:3300/0 conn(0x7f5d9406e1e0 0x7f5d940b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 -- 172.21.15.2:0/2642195173 >> v2:172.21.15.137:3300/0 conn(0x7f5d9406e1e0 msgr2=0x7f5d940b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 --2- 172.21.15.2:0/2642195173 >> v2:172.21.15.137:3300/0 conn(0x7f5d9406e1e0 0x7f5d940b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 -- 172.21.15.2:0/2642195173 >> v1:172.21.15.137:6789/0 conn(0x7f5d94143810 legacy=0x7f5d9406dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:06.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.822+0000 7f5d92800640 1 -- 172.21.15.2:0/2642195173 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940b2ac0 con 0x7f5d940b2080 2024-01-15T18:03:06.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.824+0000 7f5d92800640 1 --2- 172.21.15.2:0/2642195173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d940cd070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009870 tx=0x7f5d8c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eedb426dbe63034b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.824+0000 7f5d91400640 1 -- 172.21.15.2:0/2642195173 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c031640 con 0x7f5d940b2080 2024-01-15T18:03:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.824+0000 7f5d91400640 1 -- 172.21.15.2:0/2642195173 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c0317a0 con 0x7f5d940b2080 2024-01-15T18:03:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.824+0000 7f5d91400640 1 -- 172.21.15.2:0/2642195173 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c031b30 con 0x7f5d940b2080 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 -- 172.21.15.2:0/2642195173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 msgr2=0x7f5d940cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 --2- 172.21.15.2:0/2642195173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d940cd070 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c009870 tx=0x7f5d8c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 -- 172.21.15.2:0/2642195173 shutdown_connections 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 --2- 172.21.15.2:0/2642195173 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d940cd070 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 --2- 172.21.15.2:0/2642195173 >> v2:172.21.15.137:3300/0 conn(0x7f5d9406e1e0 0x7f5d940b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 -- 172.21.15.2:0/2642195173 >> 172.21.15.2:0/2642195173 conn(0x7f5d940b0840 msgr2=0x7f5d940b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 -- 172.21.15.2:0/2642195173 shutdown_connections 2024-01-15T18:03:06.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.825+0000 7f5d93200640 1 -- 172.21.15.2:0/2642195173 wait complete. 2024-01-15T18:03:06.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.827+0000 7f5d93200640 1 Processor -- start 2024-01-15T18:03:06.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.827+0000 7f5d93200640 1 -- start start 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d93200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 0x7f5d941367d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d93200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d94141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d93200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d94143810 0x7f5d9413ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d93200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9413b300 con 0x7f5d9406e1e0 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d93200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9413b5e0 con 0x7f5d94143810 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d93200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d9413b8c0 con 0x7f5d940b2080 2024-01-15T18:03:06.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d91e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 0x7f5d941367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d91e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 0x7f5d941367d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56520/0 (socket says 172.21.15.2:56520) 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.828+0000 7f5d91e00640 1 -- 172.21.15.2:0/3618870754 learned_addr learned my addr 172.21.15.2:0/3618870754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d92800640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d94143810 0x7f5d9413ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d89200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d94141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d91e00640 1 -- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 msgr2=0x7f5d94141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d91e00640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d94141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d91e00640 1 -- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d94143810 msgr2=0x7f5d9413ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d91e00640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d94143810 0x7f5d9413ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d91e00640 1 -- 172.21.15.2:0/3618870754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c0094d0 con 0x7f5d9406e1e0 2024-01-15T18:03:06.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d89200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d94141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:06.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d91e00640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 0x7f5d941367d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400c730 tx=0x7f5d8400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:06.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d84003b70 con 0x7f5d9406e1e0 2024-01-15T18:03:06.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d84003cf0 con 0x7f5d9406e1e0 2024-01-15T18:03:06.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.829+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d84003ea0 con 0x7f5d9406e1e0 2024-01-15T18:03:06.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.830+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d9413bba0 con 0x7f5d9406e1e0 2024-01-15T18:03:06.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.830+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d9413c3a0 con 0x7f5d9406e1e0 2024-01-15T18:03:06.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.831+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d94074770 con 0x7f5d9406e1e0 2024-01-15T18:03:06.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.832+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f5d8400aa90 con 0x7f5d9406e1e0 2024-01-15T18:03:06.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.833+0000 7f5d8aa00640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6c06b860 0x7f5d6c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:06.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.833+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5d8408a4f0 con 0x7f5d9406e1e0 2024-01-15T18:03:06.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.833+0000 7f5d89200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6c06b860 0x7f5d6c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:06.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.834+0000 7f5d89200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6c06b860 0x7f5d6c06dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5d94136d10 tx=0x7f5d7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:06.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:06.838+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d84054a70 con 0x7f5d9406e1e0 2024-01-15T18:03:07.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.070+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d940ad3f0 con 0x7f5d9406e1e0 2024-01-15T18:03:07.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.071+0000 7f5d8aa00640 1 -- 172.21.15.2:0/3618870754 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4910 (secure 0 0 0) 0x7f5d84058090 con 0x7f5d9406e1e0 2024-01-15T18:03:07.075 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:07.076 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1133084700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1133084700},{"type":"v1","addr":"172.21.15.137:6836","nonce":1133084700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5614,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/302473054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":302473054},{"type":"v1","addr":"172.21.15.137:6837","nonce":302473054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2347293827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2347293827},{"type":"v1","addr":"172.21.15.2:6833","nonce":2347293827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:03:02.994614+0000","modified":"2024-01-15T18:03:06.071657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6312},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6312":{"gid":6312,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6835/2855093754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2855093754},{"type":"v1","addr":"172.21.15.2:6835","nonce":2855093754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-15T18:03:07.076 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 36 2024-01-15T18:03:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6c06b860 msgr2=0x7f5d6c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6c06b860 0x7f5d6c06dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5d94136d10 tx=0x7f5d7c008040 comp rx=0 tx=0).stop 2024-01-15T18:03:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 msgr2=0x7f5d941367d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 0x7f5d941367d0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f5d8400c730 tx=0x7f5d8400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 shutdown_connections 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d6c06b860 0x7f5d6c06dd20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d94143810 0x7f5d9413ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d940b2080 0x7f5d94141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 --2- 172.21.15.2:0/3618870754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d9406e1e0 0x7f5d941367d0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 >> 172.21.15.2:0/3618870754 conn(0x7f5d940b0840 msgr2=0x7f5d940ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 shutdown_connections 2024-01-15T18:03:07.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.074+0000 7f5d93200640 1 -- 172.21.15.2:0/3618870754 wait complete. 2024-01-15T18:03:07.098 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:03:07.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.296+0000 7f2df7e00640 1 Processor -- start 2024-01-15T18:03:07.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.296+0000 7f2df7e00640 1 -- start start 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 0x7f2df815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2df806cad0 0x7f2df806cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df7e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2df80716d0 con 0x7f2df8141880 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df7e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2df8071f20 con 0x7f2df806cad0 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df806c6b0 con 0x7f2df815ec60 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df6a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2df8141880 0x7f2df8070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38104/0 (socket says 172.21.15.2:38104) 2024-01-15T18:03:07.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df6a00640 1 -- 172.21.15.2:0/340457546 learned_addr learned my addr 172.21.15.2:0/340457546 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df7400640 1 --2- 172.21.15.2:0/340457546 >> v2:172.21.15.137:3300/0 conn(0x7f2df806cad0 0x7f2df806cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.298+0000 7f2df6000640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 0x7f2df815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569345982 0 0) 0x7f2df80716d0 con 0x7f2df8141880 2024-01-15T18:03:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dd8003520 con 0x7f2df8141880 2024-01-15T18:03:07.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766189735 0 0) 0x7f2dd8003520 con 0x7f2df8141880 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 msgr2=0x7f2df815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 0x7f2df815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 >> v2:172.21.15.137:3300/0 conn(0x7f2df806cad0 msgr2=0x7f2df806cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 --2- 172.21.15.2:0/340457546 >> v2:172.21.15.137:3300/0 conn(0x7f2df806cad0 0x7f2df806cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8071960 con 0x7f2df8141880 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2df00019f0 con 0x7f2df8141880 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 >> v1:172.21.15.2:6789/0 conn(0x7f2df8141880 legacy=0x7f2df8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 0x7f2dd8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2dd8003f40 0x7f2dd80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2dd8004d00 0x7f2dd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8071f20 con 0x7f2dd8003f40 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df806c6b0 con 0x7f2dd8004d00 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.299+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dd800ba30 con 0x7f2df815ec60 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6000640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2dd8003f40 0x7f2dd80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6a00640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 0x7f2dd8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6000640 1 -- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 msgr2=0x7f2dd8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6000640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2df815ec60 0x7f2dd8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6000640 1 -- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2dd8004d00 msgr2=0x7f2dd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df7400640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2dd8004d00 0x7f2dd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6000640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2dd8004d00 0x7f2dd8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.300+0000 7f2df6000640 1 -- 172.21.15.2:0/340457546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd8003520 con 0x7f2dd8003f40 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df6000640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2dd8003f40 0x7f2dd80047c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009ac0 tx=0x7f2dd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fcf99d4cdd326 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de802c6b0 con 0x7f2dd8003f40 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de802c810 con 0x7f2dd8003f40 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df5600640 1 -- 172.21.15.2:0/340457546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de802cba0 con 0x7f2dd8003f40 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df7e00640 1 -- 172.21.15.2:0/340457546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2dd8003f40 msgr2=0x7f2dd80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df7e00640 1 --2- 172.21.15.2:0/340457546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2dd8003f40 0x7f2dd80047c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f2de8009ac0 tx=0x7f2dd80057f0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df7400640 1 -- 172.21.15.2:0/340457546 reap_dead start 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df7e00640 1 -- 172.21.15.2:0/340457546 shutdown_connections 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.302+0000 7f2df7e00640 1 -- 172.21.15.2:0/340457546 >> 172.21.15.2:0/340457546 conn(0x7f2df80b0840 msgr2=0x7f2df80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:07.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.303+0000 7f2df7e00640 1 -- 172.21.15.2:0/340457546 shutdown_connections 2024-01-15T18:03:07.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.303+0000 7f2df7e00640 1 -- 172.21.15.2:0/340457546 wait complete. 2024-01-15T18:03:07.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.304+0000 7f2df7e00640 1 Processor -- start 2024-01-15T18:03:07.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.305+0000 7f2df7e00640 1 -- start start 2024-01-15T18:03:07.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2dd8003f40 0x7f2df81f4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 0x7f2df8155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2df815d570 0x7f2df80ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8155770 con 0x7f2df80b1c40 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8155fd0 con 0x7f2df815d570 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df80b21e0 con 0x7f2dd8003f40 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 0x7f2df8155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 0x7f2df8155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56534/0 (socket says 172.21.15.2:56534) 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 -- 172.21.15.2:0/2540488456 learned_addr learned my addr 172.21.15.2:0/2540488456 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6a00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2dd8003f40 0x7f2df81f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df7400640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2df815d570 0x7f2df80ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 -- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2dd8003f40 msgr2=0x7f2df81f4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2dd8003f40 0x7f2df81f4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 -- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2df815d570 msgr2=0x7f2df80ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2df815d570 0x7f2df80ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6000640 1 -- 172.21.15.2:0/2540488456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df80b24c0 con 0x7f2df80b1c40 2024-01-15T18:03:07.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.306+0000 7f2df6a00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2dd8003f40 0x7f2df81f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:07.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.307+0000 7f2df6000640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 0x7f2df8155a90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2de8037040 tx=0x7f2de8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:07.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.307+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8003810 con 0x7f2df80b1c40 2024-01-15T18:03:07.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.307+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de8003970 con 0x7f2df80b1c40 2024-01-15T18:03:07.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.307+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df80b27a0 con 0x7f2df80b1c40 2024-01-15T18:03:07.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.307+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de8003ce0 con 0x7f2df80b1c40 2024-01-15T18:03:07.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.307+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df80acdb0 con 0x7f2df80b1c40 2024-01-15T18:03:07.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.309+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df81f35e0 con 0x7f2df80b1c40 2024-01-15T18:03:07.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.310+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f2de8047450 con 0x7f2df80b1c40 2024-01-15T18:03:07.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.310+0000 7f2deea00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2dd006b8e0 0x7f2dd006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.311+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f2de80ac7d0 con 0x7f2df80b1c40 2024-01-15T18:03:07.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.311+0000 7f2df6a00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2dd006b8e0 0x7f2dd006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.312+0000 7f2df6a00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2dd006b8e0 0x7f2dd006dda0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2df806e0a0 tx=0x7f2df0001290 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:07.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.315+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2df81f35e0 con 0x7f2df80b1c40 2024-01-15T18:03:07.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.513+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2df81f35e0 con 0x7f2df80b1c40 2024-01-15T18:03:07.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.516+0000 7f2deea00640 1 -- 172.21.15.2:0/2540488456 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f2de80a90e0 con 0x7f2df80b1c40 2024-01-15T18:03:07.521 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:07.521 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":61,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:03:04.545306+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:03:00.876169+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-15T18:03:02.030400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2dd006b8e0 msgr2=0x7f2dd006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2dd006b8e0 0x7f2dd006dda0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2df806e0a0 tx=0x7f2df0001290 comp rx=0 tx=0).stop 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 msgr2=0x7f2df8155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 0x7f2df8155a90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f2de8037040 tx=0x7f2de8004730 comp rx=0 tx=0).stop 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 shutdown_connections 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2dd006b8e0 0x7f2dd006dda0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2df815d570 0x7f2df80ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2df80b1c40 0x7f2df8155a90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 --2- 172.21.15.2:0/2540488456 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2dd8003f40 0x7f2df81f4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 >> 172.21.15.2:0/2540488456 conn(0x7f2df80b0840 msgr2=0x7f2df8144520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 shutdown_connections 2024-01-15T18:03:07.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.519+0000 7f2df7e00640 1 -- 172.21.15.2:0/2540488456 wait complete. 2024-01-15T18:03:07.543 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:03:07.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:03:07.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.748+0000 7fb127e00640 1 Processor -- start 2024-01-15T18:03:07.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.748+0000 7fb127e00640 1 -- start start 2024-01-15T18:03:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb127e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb12015c0e0 0x7fb1200cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb127e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015b7d0 0x7fb12006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb127e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb12015cb00 con 0x7fb12015c4c0 2024-01-15T18:03:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb127e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1200ccec0 con 0x7fb12015c0e0 2024-01-15T18:03:07.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb127e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1200cd1a0 con 0x7fb12015b7d0 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb125400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb12015c0e0 0x7fb1200cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb125e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb12015c4c0 0x7fb12015ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38110/0 (socket says 172.21.15.2:38110) 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.750+0000 7fb125e00640 1 -- 172.21.15.2:0/849943867 learned_addr learned my addr 172.21.15.2:0/849943867 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.751+0000 7fb124a00640 1 --2- 172.21.15.2:0/849943867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015b7d0 0x7fb12006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.751+0000 7fb125400640 1 -- 172.21.15.2:0/849943867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015b7d0 msgr2=0x7fb12006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.751+0000 7fb125400640 1 --2- 172.21.15.2:0/849943867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015b7d0 0x7fb12006c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.751+0000 7fb125400640 1 -- 172.21.15.2:0/849943867 >> v1:172.21.15.2:6789/0 conn(0x7fb12015c4c0 legacy=0x7fb12015ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.751+0000 7fb125400640 1 -- 172.21.15.2:0/849943867 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200cd480 con 0x7fb12015c0e0 2024-01-15T18:03:07.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.752+0000 7fb125400640 1 --2- 172.21.15.2:0/849943867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015c0e0 0x7fb1200cc5c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb1100097a0 tx=0x7fb11002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b60ece4434afa3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb11fe00640 1 -- 172.21.15.2:0/849943867 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11002d040 con 0x7fb12015c0e0 2024-01-15T18:03:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb11fe00640 1 -- 172.21.15.2:0/849943867 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb110031810 con 0x7fb12015c0e0 2024-01-15T18:03:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb11fe00640 1 -- 172.21.15.2:0/849943867 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb110031ba0 con 0x7fb12015c0e0 2024-01-15T18:03:07.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb127e00640 1 -- 172.21.15.2:0/849943867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015c0e0 msgr2=0x7fb1200cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb127e00640 1 --2- 172.21.15.2:0/849943867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015c0e0 0x7fb1200cc5c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb1100097a0 tx=0x7fb11002b8b0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb127e00640 1 -- 172.21.15.2:0/849943867 shutdown_connections 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb127e00640 1 --2- 172.21.15.2:0/849943867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015b7d0 0x7fb12006c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb127e00640 1 --2- 172.21.15.2:0/849943867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015c0e0 0x7fb1200cc5c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.753+0000 7fb127e00640 1 -- 172.21.15.2:0/849943867 >> 172.21.15.2:0/849943867 conn(0x7fb12006da20 msgr2=0x7fb12006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.754+0000 7fb127e00640 1 -- 172.21.15.2:0/849943867 shutdown_connections 2024-01-15T18:03:07.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.754+0000 7fb127e00640 1 -- 172.21.15.2:0/849943867 wait complete. 2024-01-15T18:03:07.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.755+0000 7fb127e00640 1 Processor -- start 2024-01-15T18:03:07.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.756+0000 7fb127e00640 1 -- start start 2024-01-15T18:03:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.756+0000 7fb127e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 0x7fb12014bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.756+0000 7fb127e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb12015c0e0 0x7fb1201531f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.756+0000 7fb127e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015c4c0 0x7fb120154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb127e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb120154dc0 con 0x7fb12015b7d0 2024-01-15T18:03:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb127e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb120152710 con 0x7fb12015c0e0 2024-01-15T18:03:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb127e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1201529f0 con 0x7fb12015c4c0 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 0x7fb12014bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 0x7fb12014bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56560/0 (socket says 172.21.15.2:56560) 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 -- 172.21.15.2:0/3541917388 learned_addr learned my addr 172.21.15.2:0/3541917388 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 -- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015c4c0 msgr2=0x7fb120154450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015c4c0 0x7fb120154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 -- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb12015c0e0 msgr2=0x7fb1201531f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb12015c0e0 0x7fb1201531f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.757+0000 7fb125400640 1 -- 172.21.15.2:0/3541917388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110009400 con 0x7fb12015b7d0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.758+0000 7fb125400640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 0x7fb12014bdf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb1100098c0 tx=0x7fb1100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.758+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb11003ad60 con 0x7fb12015b7d0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.758+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb110044400 con 0x7fb12015b7d0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.758+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1100447b0 con 0x7fb12015b7d0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.758+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb120152cd0 con 0x7fb12015b7d0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.758+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb120153730 con 0x7fb12015b7d0 2024-01-15T18:03:07.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.759+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb120152ec0 con 0x7fb12015b7d0 2024-01-15T18:03:07.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.760+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb11004d440 con 0x7fb12015b7d0 2024-01-15T18:03:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.761+0000 7fb11d600640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0f806b860 0x7fb0f806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.761+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb1100acb20 con 0x7fb12015b7d0 2024-01-15T18:03:07.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.762+0000 7fb124a00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0f806b860 0x7fb0f806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:07.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.763+0000 7fb124a00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0f806b860 0x7fb0f806dd20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb108006fd0 tx=0x7fb108008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:07.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.766+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1100770b0 con 0x7fb12015b7d0 2024-01-15T18:03:07.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.955+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1200686f0 con 0x7fb0f806b860 2024-01-15T18:03:07.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.983+0000 7fb11d600640 1 -- 172.21.15.2:0/3541917388 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287225 (secure 0 0 0) 0x7fb1200686f0 con 0x7fb0f806b860 2024-01-15T18:03:07.991 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:07.992 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0f806b860 msgr2=0x7fb0f806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0f806b860 0x7fb0f806dd20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb108006fd0 tx=0x7fb108008040 comp rx=0 tx=0).stop 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 msgr2=0x7fb12014bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 0x7fb12014bdf0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fb1100098c0 tx=0x7fb1100046c0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 shutdown_connections 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0f806b860 0x7fb0f806dd20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb12015c4c0 0x7fb120154450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb12015c0e0 0x7fb1201531f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 --2- 172.21.15.2:0/3541917388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb12015b7d0 0x7fb12014bdf0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.988+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 >> 172.21.15.2:0/3541917388 conn(0x7fb12006da20 msgr2=0x7fb12015e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:07.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.989+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 shutdown_connections 2024-01-15T18:03:07.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:07.989+0000 7fb127e00640 1 -- 172.21.15.2:0/3541917388 wait complete. 2024-01-15T18:03:08.013 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-01-15T18:03:07.515568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310360,"kb_used_data":6816,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754664360,"statfs":{"total":773094113280,"available":772776304640,"internally_reserved":0,"allocated":6979584,"data_stored":3589532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":310693425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002320"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561769+0000","last_change":"2024-01-15T18:03:03.556818+0000","last_active":"2024-01-15T18:03:03.561769+0000","last_peered":"2024-01-15T18:03:03.561769+0000","last_clean":"2024-01-15T18:03:03.561769+0000","last_became_active":"2024-01-15T18:03:03.556532+0000","last_became_peered":"2024-01-15T18:03:03.556532+0000","last_unstale":"2024-01-15T18:03:03.561769+0000","last_undegraded":"2024-01-15T18:03:03.561769+0000","last_fullsized":"2024-01-15T18:03:03.561769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:04.576059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572671+0000","last_change":"2024-01-15T18:03:03.564151+0000","last_active":"2024-01-15T18:03:03.572671+0000","last_peered":"2024-01-15T18:03:03.572671+0000","last_clean":"2024-01-15T18:03:03.572671+0000","last_became_active":"2024-01-15T18:03:03.563896+0000","last_became_peered":"2024-01-15T18:03:03.563896+0000","last_unstale":"2024-01-15T18:03:03.572671+0000","last_undegraded":"2024-01-15T18:03:03.572671+0000","last_fullsized":"2024-01-15T18:03:03.572671+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:26.579782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575880+0000","last_change":"2024-01-15T18:03:03.569042+0000","last_active":"2024-01-15T18:03:03.575880+0000","last_peered":"2024-01-15T18:03:03.575880+0000","last_clean":"2024-01-15T18:03:03.575880+0000","last_became_active":"2024-01-15T18:03:03.568645+0000","last_became_peered":"2024-01-15T18:03:03.568645+0000","last_unstale":"2024-01-15T18:03:03.575880+0000","last_undegraded":"2024-01-15T18:03:03.575880+0000","last_fullsized":"2024-01-15T18:03:03.575880+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:52:26.067882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561651+0000","last_change":"2024-01-15T18:03:03.557155+0000","last_active":"2024-01-15T18:03:03.561651+0000","last_peered":"2024-01-15T18:03:03.561651+0000","last_clean":"2024-01-15T18:03:03.561651+0000","last_became_active":"2024-01-15T18:03:03.556824+0000","last_became_peered":"2024-01-15T18:03:03.556824+0000","last_unstale":"2024-01-15T18:03:03.561651+0000","last_undegraded":"2024-01-15T18:03:03.561651+0000","last_fullsized":"2024-01-15T18:03:03.561651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:52:26.067882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572356+0000","last_change":"2024-01-15T18:03:03.569272+0000","last_active":"2024-01-15T18:03:03.572356+0000","last_peered":"2024-01-15T18:03:03.572356+0000","last_clean":"2024-01-15T18:03:03.572356+0000","last_became_active":"2024-01-15T18:03:03.564615+0000","last_became_peered":"2024-01-15T18:03:03.564615+0000","last_unstale":"2024-01-15T18:03:03.572356+0000","last_undegraded":"2024-01-15T18:03:03.572356+0000","last_fullsized":"2024-01-15T18:03:03.572356+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:08:10.493793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550644+0000","last_change":"2024-01-15T18:03:03.547527+0000","last_active":"2024-01-15T18:03:03.550644+0000","last_peered":"2024-01-15T18:03:03.550644+0000","last_clean":"2024-01-15T18:03:03.550644+0000","last_became_active":"2024-01-15T18:03:03.547216+0000","last_became_peered":"2024-01-15T18:03:03.547216+0000","last_unstale":"2024-01-15T18:03:03.550644+0000","last_undegraded":"2024-01-15T18:03:03.550644+0000","last_fullsized":"2024-01-15T18:03:03.550644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:24:23.035552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562349+0000","last_change":"2024-01-15T18:03:03.561460+0000","last_active":"2024-01-15T18:03:04.562349+0000","last_peered":"2024-01-15T18:03:04.562349+0000","last_clean":"2024-01-15T18:03:04.562349+0000","last_became_active":"2024-01-15T18:03:03.561027+0000","last_became_peered":"2024-01-15T18:03:03.561027+0000","last_unstale":"2024-01-15T18:03:04.562349+0000","last_undegraded":"2024-01-15T18:03:04.562349+0000","last_fullsized":"2024-01-15T18:03:04.562349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:22.576177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572258+0000","last_change":"2024-01-15T18:03:03.560223+0000","last_active":"2024-01-15T18:03:03.572258+0000","last_peered":"2024-01-15T18:03:03.572258+0000","last_clean":"2024-01-15T18:03:03.572258+0000","last_became_active":"2024-01-15T18:03:03.559717+0000","last_became_peered":"2024-01-15T18:03:03.559717+0000","last_unstale":"2024-01-15T18:03:03.572258+0000","last_undegraded":"2024-01-15T18:03:03.572258+0000","last_fullsized":"2024-01-15T18:03:03.572258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:58.728010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.565550+0000","last_change":"2024-01-15T18:03:03.554478+0000","last_active":"2024-01-15T18:03:04.565550+0000","last_peered":"2024-01-15T18:03:04.565550+0000","last_clean":"2024-01-15T18:03:04.565550+0000","last_became_active":"2024-01-15T18:03:03.554176+0000","last_became_peered":"2024-01-15T18:03:03.554176+0000","last_unstale":"2024-01-15T18:03:04.565550+0000","last_undegraded":"2024-01-15T18:03:04.565550+0000","last_fullsized":"2024-01-15T18:03:04.565550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:14.918999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.571746+0000","last_change":"2024-01-15T18:03:03.566156+0000","last_active":"2024-01-15T18:03:03.571746+0000","last_peered":"2024-01-15T18:03:03.571746+0000","last_clean":"2024-01-15T18:03:03.571746+0000","last_became_active":"2024-01-15T18:03:03.565907+0000","last_became_peered":"2024-01-15T18:03:03.565907+0000","last_unstale":"2024-01-15T18:03:03.571746+0000","last_undegraded":"2024-01-15T18:03:03.571746+0000","last_fullsized":"2024-01-15T18:03:03.571746+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:11.929487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.574306+0000","last_change":"2024-01-15T18:03:03.562279+0000","last_active":"2024-01-15T18:03:03.574306+0000","last_peered":"2024-01-15T18:03:03.574306+0000","last_clean":"2024-01-15T18:03:03.574306+0000","last_became_active":"2024-01-15T18:03:03.562008+0000","last_became_peered":"2024-01-15T18:03:03.562008+0000","last_unstale":"2024-01-15T18:03:03.574306+0000","last_undegraded":"2024-01-15T18:03:03.574306+0000","last_fullsized":"2024-01-15T18:03:03.574306+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.005801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578643+0000","last_change":"2024-01-15T18:03:03.575980+0000","last_active":"2024-01-15T18:03:03.578643+0000","last_peered":"2024-01-15T18:03:03.578643+0000","last_clean":"2024-01-15T18:03:03.578643+0000","last_became_active":"2024-01-15T18:03:03.575748+0000","last_became_peered":"2024-01-15T18:03:03.575748+0000","last_unstale":"2024-01-15T18:03:03.578643+0000","last_undegraded":"2024-01-15T18:03:03.578643+0000","last_fullsized":"2024-01-15T18:03:03.578643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:24:23.035552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.583267+0000","last_change":"2024-01-15T18:03:03.580017+0000","last_active":"2024-01-15T18:03:03.583267+0000","last_peered":"2024-01-15T18:03:03.583267+0000","last_clean":"2024-01-15T18:03:03.583267+0000","last_became_active":"2024-01-15T18:03:03.579710+0000","last_became_peered":"2024-01-15T18:03:03.579710+0000","last_unstale":"2024-01-15T18:03:03.583267+0000","last_undegraded":"2024-01-15T18:03:03.583267+0000","last_fullsized":"2024-01-15T18:03:03.583267+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:16.679701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562467+0000","last_change":"2024-01-15T18:03:03.556480+0000","last_active":"2024-01-15T18:03:04.562467+0000","last_peered":"2024-01-15T18:03:04.562467+0000","last_clean":"2024-01-15T18:03:04.562467+0000","last_became_active":"2024-01-15T18:03:03.555437+0000","last_became_peered":"2024-01-15T18:03:03.555437+0000","last_unstale":"2024-01-15T18:03:04.562467+0000","last_undegraded":"2024-01-15T18:03:04.562467+0000","last_fullsized":"2024-01-15T18:03:04.562467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:09.221734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562872+0000","last_change":"2024-01-15T18:03:03.563820+0000","last_active":"2024-01-15T18:03:04.562872+0000","last_peered":"2024-01-15T18:03:04.562872+0000","last_clean":"2024-01-15T18:03:04.562872+0000","last_became_active":"2024-01-15T18:03:03.563532+0000","last_became_peered":"2024-01-15T18:03:03.563532+0000","last_unstale":"2024-01-15T18:03:04.562872+0000","last_undegraded":"2024-01-15T18:03:04.562872+0000","last_fullsized":"2024-01-15T18:03:04.562872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:02.869083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580227+0000","last_change":"2024-01-15T18:03:03.570114+0000","last_active":"2024-01-15T18:03:03.580227+0000","last_peered":"2024-01-15T18:03:03.580227+0000","last_clean":"2024-01-15T18:03:03.580227+0000","last_became_active":"2024-01-15T18:03:03.569810+0000","last_became_peered":"2024-01-15T18:03:03.569810+0000","last_unstale":"2024-01-15T18:03:03.580227+0000","last_undegraded":"2024-01-15T18:03:03.580227+0000","last_fullsized":"2024-01-15T18:03:03.580227+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:34.021909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575113+0000","last_change":"2024-01-15T18:03:03.572668+0000","last_active":"2024-01-15T18:03:03.575113+0000","last_peered":"2024-01-15T18:03:03.575113+0000","last_clean":"2024-01-15T18:03:03.575113+0000","last_became_active":"2024-01-15T18:03:03.568800+0000","last_became_peered":"2024-01-15T18:03:03.568800+0000","last_unstale":"2024-01-15T18:03:03.575113+0000","last_undegraded":"2024-01-15T18:03:03.575113+0000","last_fullsized":"2024-01-15T18:03:03.575113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:49:06.674946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581492+0000","last_change":"2024-01-15T18:03:03.575426+0000","last_active":"2024-01-15T18:03:03.581492+0000","last_peered":"2024-01-15T18:03:03.581492+0000","last_clean":"2024-01-15T18:03:03.581492+0000","last_became_active":"2024-01-15T18:03:03.575237+0000","last_became_peered":"2024-01-15T18:03:03.575237+0000","last_unstale":"2024-01-15T18:03:03.581492+0000","last_undegraded":"2024-01-15T18:03:03.581492+0000","last_fullsized":"2024-01-15T18:03:03.581492+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:38.275323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.562718+0000","last_change":"2024-01-15T18:03:03.558267+0000","last_active":"2024-01-15T18:03:03.562718+0000","last_peered":"2024-01-15T18:03:03.562718+0000","last_clean":"2024-01-15T18:03:03.562718+0000","last_became_active":"2024-01-15T18:03:03.557947+0000","last_became_peered":"2024-01-15T18:03:03.557947+0000","last_unstale":"2024-01-15T18:03:03.562718+0000","last_undegraded":"2024-01-15T18:03:03.562718+0000","last_fullsized":"2024-01-15T18:03:03.562718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:24.128195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572616+0000","last_change":"2024-01-15T18:03:03.555727+0000","last_active":"2024-01-15T18:03:03.572616+0000","last_peered":"2024-01-15T18:03:03.572616+0000","last_clean":"2024-01-15T18:03:03.572616+0000","last_became_active":"2024-01-15T18:03:03.553483+0000","last_became_peered":"2024-01-15T18:03:03.553483+0000","last_unstale":"2024-01-15T18:03:03.572616+0000","last_undegraded":"2024-01-15T18:03:03.572616+0000","last_fullsized":"2024-01-15T18:03:03.572616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:23.222355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581359+0000","last_change":"2024-01-15T18:03:03.573290+0000","last_active":"2024-01-15T18:03:03.581359+0000","last_peered":"2024-01-15T18:03:03.581359+0000","last_clean":"2024-01-15T18:03:03.581359+0000","last_became_active":"2024-01-15T18:03:03.572347+0000","last_became_peered":"2024-01-15T18:03:03.572347+0000","last_unstale":"2024-01-15T18:03:03.581359+0000","last_undegraded":"2024-01-15T18:03:03.581359+0000","last_fullsized":"2024-01-15T18:03:03.581359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.947434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573186+0000","last_change":"2024-01-15T18:03:03.565599+0000","last_active":"2024-01-15T18:03:03.573186+0000","last_peered":"2024-01-15T18:03:03.573186+0000","last_clean":"2024-01-15T18:03:03.573186+0000","last_became_active":"2024-01-15T18:03:03.565087+0000","last_became_peered":"2024-01-15T18:03:03.565087+0000","last_unstale":"2024-01-15T18:03:03.573186+0000","last_undegraded":"2024-01-15T18:03:03.573186+0000","last_fullsized":"2024-01-15T18:03:03.573186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:26.100036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581806+0000","last_change":"2024-01-15T18:03:03.578999+0000","last_active":"2024-01-15T18:03:03.581806+0000","last_peered":"2024-01-15T18:03:03.581806+0000","last_clean":"2024-01-15T18:03:03.581806+0000","last_became_active":"2024-01-15T18:03:03.577347+0000","last_became_peered":"2024-01-15T18:03:03.577347+0000","last_unstale":"2024-01-15T18:03:03.581806+0000","last_undegraded":"2024-01-15T18:03:03.581806+0000","last_fullsized":"2024-01-15T18:03:03.581806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:13.272751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.583255+0000","last_change":"2024-01-15T18:03:03.578538+0000","last_active":"2024-01-15T18:03:03.583255+0000","last_peered":"2024-01-15T18:03:03.583255+0000","last_clean":"2024-01-15T18:03:03.583255+0000","last_became_active":"2024-01-15T18:03:03.578110+0000","last_became_peered":"2024-01-15T18:03:03.578110+0000","last_unstale":"2024-01-15T18:03:03.583255+0000","last_undegraded":"2024-01-15T18:03:03.583255+0000","last_fullsized":"2024-01-15T18:03:03.583255+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:12.548993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572325+0000","last_change":"2024-01-15T18:03:03.568945+0000","last_active":"2024-01-15T18:03:03.572325+0000","last_peered":"2024-01-15T18:03:03.572325+0000","last_clean":"2024-01-15T18:03:03.572325+0000","last_became_active":"2024-01-15T18:03:03.563481+0000","last_became_peered":"2024-01-15T18:03:03.563481+0000","last_unstale":"2024-01-15T18:03:03.572325+0000","last_undegraded":"2024-01-15T18:03:03.572325+0000","last_fullsized":"2024-01-15T18:03:03.572325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:37.019971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.562564+0000","last_change":"2024-01-15T18:03:03.559469+0000","last_active":"2024-01-15T18:03:03.562564+0000","last_peered":"2024-01-15T18:03:03.562564+0000","last_clean":"2024-01-15T18:03:03.562564+0000","last_became_active":"2024-01-15T18:03:03.559155+0000","last_became_peered":"2024-01-15T18:03:03.559155+0000","last_unstale":"2024-01-15T18:03:03.562564+0000","last_undegraded":"2024-01-15T18:03:03.562564+0000","last_fullsized":"2024-01-15T18:03:03.562564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:51.286116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.582039+0000","last_change":"2024-01-15T18:03:03.571321+0000","last_active":"2024-01-15T18:03:03.582039+0000","last_peered":"2024-01-15T18:03:03.582039+0000","last_clean":"2024-01-15T18:03:03.582039+0000","last_became_active":"2024-01-15T18:03:03.570991+0000","last_became_peered":"2024-01-15T18:03:03.570991+0000","last_unstale":"2024-01-15T18:03:03.582039+0000","last_undegraded":"2024-01-15T18:03:03.582039+0000","last_fullsized":"2024-01-15T18:03:03.582039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:47.816541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.569168+0000","last_change":"2024-01-15T18:03:03.566629+0000","last_active":"2024-01-15T18:03:03.569168+0000","last_peered":"2024-01-15T18:03:03.569168+0000","last_clean":"2024-01-15T18:03:03.569168+0000","last_became_active":"2024-01-15T18:03:03.566305+0000","last_became_peered":"2024-01-15T18:03:03.566305+0000","last_unstale":"2024-01-15T18:03:03.569168+0000","last_undegraded":"2024-01-15T18:03:03.569168+0000","last_fullsized":"2024-01-15T18:03:03.569168+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:15.787221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573301+0000","last_change":"2024-01-15T18:03:03.571334+0000","last_active":"2024-01-15T18:03:03.573301+0000","last_peered":"2024-01-15T18:03:03.573301+0000","last_clean":"2024-01-15T18:03:03.573301+0000","last_became_active":"2024-01-15T18:03:03.570330+0000","last_became_peered":"2024-01-15T18:03:03.570330+0000","last_unstale":"2024-01-15T18:03:03.573301+0000","last_undegraded":"2024-01-15T18:03:03.573301+0000","last_fullsized":"2024-01-15T18:03:03.573301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:13.175551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573825+0000","last_change":"2024-01-15T18:03:03.568407+0000","last_active":"2024-01-15T18:03:03.573825+0000","last_peered":"2024-01-15T18:03:03.573825+0000","last_clean":"2024-01-15T18:03:03.573825+0000","last_became_active":"2024-01-15T18:03:03.567608+0000","last_became_peered":"2024-01-15T18:03:03.567608+0000","last_unstale":"2024-01-15T18:03:03.573825+0000","last_undegraded":"2024-01-15T18:03:03.573825+0000","last_fullsized":"2024-01-15T18:03:03.573825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:24:23.035552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.569734+0000","last_change":"2024-01-15T18:03:03.568365+0000","last_active":"2024-01-15T18:03:03.569734+0000","last_peered":"2024-01-15T18:03:03.569734+0000","last_clean":"2024-01-15T18:03:03.569734+0000","last_became_active":"2024-01-15T18:03:03.563420+0000","last_became_peered":"2024-01-15T18:03:03.563420+0000","last_unstale":"2024-01-15T18:03:03.569734+0000","last_undegraded":"2024-01-15T18:03:03.569734+0000","last_fullsized":"2024-01-15T18:03:03.569734+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:54.246039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578720+0000","last_change":"2024-01-15T18:03:03.558405+0000","last_active":"2024-01-15T18:03:03.578720+0000","last_peered":"2024-01-15T18:03:03.578720+0000","last_clean":"2024-01-15T18:03:03.578720+0000","last_became_active":"2024-01-15T18:03:03.558094+0000","last_became_peered":"2024-01-15T18:03:03.558094+0000","last_unstale":"2024-01-15T18:03:03.578720+0000","last_undegraded":"2024-01-15T18:03:03.578720+0000","last_fullsized":"2024-01-15T18:03:03.578720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:52:26.067882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581673+0000","last_change":"2024-01-15T18:03:03.571621+0000","last_active":"2024-01-15T18:03:03.581673+0000","last_peered":"2024-01-15T18:03:03.581673+0000","last_clean":"2024-01-15T18:03:03.581673+0000","last_became_active":"2024-01-15T18:03:03.571353+0000","last_became_peered":"2024-01-15T18:03:03.571353+0000","last_unstale":"2024-01-15T18:03:03.581673+0000","last_undegraded":"2024-01-15T18:03:03.581673+0000","last_fullsized":"2024-01-15T18:03:03.581673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:13.272751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"60'4","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580382+0000","last_change":"2024-01-15T18:03:02.546211+0000","last_active":"2024-01-15T18:03:03.580382+0000","last_peered":"2024-01-15T18:03:03.580382+0000","last_clean":"2024-01-15T18:03:03.580382+0000","last_became_active":"2024-01-15T18:03:02.545986+0000","last_became_peered":"2024-01-15T18:03:02.545986+0000","last_unstale":"2024-01-15T18:03:03.580382+0000","last_undegraded":"2024-01-15T18:03:03.580382+0000","last_fullsized":"2024-01-15T18:03:03.580382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:15:49.720738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575291+0000","last_change":"2024-01-15T18:03:03.573544+0000","last_active":"2024-01-15T18:03:03.575291+0000","last_peered":"2024-01-15T18:03:03.575291+0000","last_clean":"2024-01-15T18:03:03.575291+0000","last_became_active":"2024-01-15T18:03:03.573282+0000","last_became_peered":"2024-01-15T18:03:03.573282+0000","last_unstale":"2024-01-15T18:03:03.575291+0000","last_undegraded":"2024-01-15T18:03:03.575291+0000","last_fullsized":"2024-01-15T18:03:03.575291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:33.954665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.576416+0000","last_change":"2024-01-15T18:03:02.534875+0000","last_active":"2024-01-15T18:03:03.576416+0000","last_peered":"2024-01-15T18:03:03.576416+0000","last_clean":"2024-01-15T18:03:03.576416+0000","last_became_active":"2024-01-15T18:03:02.534561+0000","last_became_peered":"2024-01-15T18:03:02.534561+0000","last_unstale":"2024-01-15T18:03:03.576416+0000","last_undegraded":"2024-01-15T18:03:03.576416+0000","last_fullsized":"2024-01-15T18:03:03.576416+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:10.880631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579828+0000","last_change":"2024-01-15T18:03:03.561303+0000","last_active":"2024-01-15T18:03:03.579828+0000","last_peered":"2024-01-15T18:03:03.579828+0000","last_clean":"2024-01-15T18:03:03.579828+0000","last_became_active":"2024-01-15T18:03:03.560997+0000","last_became_peered":"2024-01-15T18:03:03.560997+0000","last_unstale":"2024-01-15T18:03:03.579828+0000","last_undegraded":"2024-01-15T18:03:03.579828+0000","last_fullsized":"2024-01-15T18:03:03.579828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:22.070448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579711+0000","last_change":"2024-01-15T18:03:02.541272+0000","last_active":"2024-01-15T18:03:03.579711+0000","last_peered":"2024-01-15T18:03:03.579711+0000","last_clean":"2024-01-15T18:03:03.579711+0000","last_became_active":"2024-01-15T18:03:02.541029+0000","last_became_peered":"2024-01-15T18:03:02.541029+0000","last_unstale":"2024-01-15T18:03:03.579711+0000","last_undegraded":"2024-01-15T18:03:03.579711+0000","last_fullsized":"2024-01-15T18:03:03.579711+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:02.371466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579941+0000","last_change":"2024-01-15T18:03:03.563676+0000","last_active":"2024-01-15T18:03:03.579941+0000","last_peered":"2024-01-15T18:03:03.579941+0000","last_clean":"2024-01-15T18:03:03.579941+0000","last_became_active":"2024-01-15T18:03:03.563223+0000","last_became_peered":"2024-01-15T18:03:03.563223+0000","last_unstale":"2024-01-15T18:03:03.579941+0000","last_undegraded":"2024-01-15T18:03:03.579941+0000","last_fullsized":"2024-01-15T18:03:03.579941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:34.769159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581443+0000","last_change":"2024-01-15T18:03:02.548803+0000","last_active":"2024-01-15T18:03:03.581443+0000","last_peered":"2024-01-15T18:03:03.581443+0000","last_clean":"2024-01-15T18:03:03.581443+0000","last_became_active":"2024-01-15T18:03:02.547942+0000","last_became_peered":"2024-01-15T18:03:02.547942+0000","last_unstale":"2024-01-15T18:03:03.581443+0000","last_undegraded":"2024-01-15T18:03:03.581443+0000","last_fullsized":"2024-01-15T18:03:03.581443+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:04:40.709592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573165+0000","last_change":"2024-01-15T18:03:03.565344+0000","last_active":"2024-01-15T18:03:03.573165+0000","last_peered":"2024-01-15T18:03:03.573165+0000","last_clean":"2024-01-15T18:03:03.573165+0000","last_became_active":"2024-01-15T18:03:03.564730+0000","last_became_peered":"2024-01-15T18:03:03.564730+0000","last_unstale":"2024-01-15T18:03:03.573165+0000","last_undegraded":"2024-01-15T18:03:03.573165+0000","last_fullsized":"2024-01-15T18:03:03.573165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:31:28.167754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"60'1","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.564808+0000","last_change":"2024-01-15T18:03:02.537087+0000","last_active":"2024-01-15T18:03:04.564808+0000","last_peered":"2024-01-15T18:03:04.564808+0000","last_clean":"2024-01-15T18:03:04.564808+0000","last_became_active":"2024-01-15T18:03:02.536776+0000","last_became_peered":"2024-01-15T18:03:02.536776+0000","last_unstale":"2024-01-15T18:03:04.564808+0000","last_undegraded":"2024-01-15T18:03:04.564808+0000","last_fullsized":"2024-01-15T18:03:04.564808+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:22:36.038520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.557849+0000","last_change":"2024-01-15T18:03:03.553987+0000","last_active":"2024-01-15T18:03:03.557849+0000","last_peered":"2024-01-15T18:03:03.557849+0000","last_clean":"2024-01-15T18:03:03.557849+0000","last_became_active":"2024-01-15T18:03:03.552722+0000","last_became_peered":"2024-01-15T18:03:03.552722+0000","last_unstale":"2024-01-15T18:03:03.557849+0000","last_undegraded":"2024-01-15T18:03:03.557849+0000","last_fullsized":"2024-01-15T18:03:03.557849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:46.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579378+0000","last_change":"2024-01-15T18:03:02.539551+0000","last_active":"2024-01-15T18:03:03.579378+0000","last_peered":"2024-01-15T18:03:03.579378+0000","last_clean":"2024-01-15T18:03:03.579378+0000","last_became_active":"2024-01-15T18:03:02.539260+0000","last_became_peered":"2024-01-15T18:03:02.539260+0000","last_unstale":"2024-01-15T18:03:03.579378+0000","last_undegraded":"2024-01-15T18:03:03.579378+0000","last_fullsized":"2024-01-15T18:03:03.579378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:46.977934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580677+0000","last_change":"2024-01-15T18:03:03.575030+0000","last_active":"2024-01-15T18:03:03.580677+0000","last_peered":"2024-01-15T18:03:03.580677+0000","last_clean":"2024-01-15T18:03:03.580677+0000","last_became_active":"2024-01-15T18:03:03.574283+0000","last_became_peered":"2024-01-15T18:03:03.574283+0000","last_unstale":"2024-01-15T18:03:03.580677+0000","last_undegraded":"2024-01-15T18:03:03.580677+0000","last_fullsized":"2024-01-15T18:03:03.580677+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:19.009008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550759+0000","last_change":"2024-01-15T18:03:02.545792+0000","last_active":"2024-01-15T18:03:03.550759+0000","last_peered":"2024-01-15T18:03:03.550759+0000","last_clean":"2024-01-15T18:03:03.550759+0000","last_became_active":"2024-01-15T18:03:02.545242+0000","last_became_peered":"2024-01-15T18:03:02.545242+0000","last_unstale":"2024-01-15T18:03:03.550759+0000","last_undegraded":"2024-01-15T18:03:03.550759+0000","last_fullsized":"2024-01-15T18:03:03.550759+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:02.371466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575658+0000","last_change":"2024-01-15T18:03:03.573048+0000","last_active":"2024-01-15T18:03:03.575658+0000","last_peered":"2024-01-15T18:03:03.575658+0000","last_clean":"2024-01-15T18:03:03.575658+0000","last_became_active":"2024-01-15T18:03:03.568637+0000","last_became_peered":"2024-01-15T18:03:03.568637+0000","last_unstale":"2024-01-15T18:03:03.575658+0000","last_undegraded":"2024-01-15T18:03:03.575658+0000","last_fullsized":"2024-01-15T18:03:03.575658+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:52:04.192561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.563305+0000","last_change":"2024-01-15T18:03:02.538617+0000","last_active":"2024-01-15T18:03:04.563305+0000","last_peered":"2024-01-15T18:03:04.563305+0000","last_clean":"2024-01-15T18:03:04.563305+0000","last_became_active":"2024-01-15T18:03:02.538293+0000","last_became_peered":"2024-01-15T18:03:02.538293+0000","last_unstale":"2024-01-15T18:03:04.563305+0000","last_undegraded":"2024-01-15T18:03:04.563305+0000","last_fullsized":"2024-01-15T18:03:04.563305+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:46.832961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.565397+0000","last_change":"2024-01-15T18:03:03.563735+0000","last_active":"2024-01-15T18:03:04.565397+0000","last_peered":"2024-01-15T18:03:04.565397+0000","last_clean":"2024-01-15T18:03:04.565397+0000","last_became_active":"2024-01-15T18:03:03.563450+0000","last_became_peered":"2024-01-15T18:03:03.563450+0000","last_unstale":"2024-01-15T18:03:04.565397+0000","last_undegraded":"2024-01-15T18:03:04.565397+0000","last_fullsized":"2024-01-15T18:03:04.565397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:04.905534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.555343+0000","last_change":"2024-01-15T18:03:02.544434+0000","last_active":"2024-01-15T18:03:03.555343+0000","last_peered":"2024-01-15T18:03:03.555343+0000","last_clean":"2024-01-15T18:03:03.555343+0000","last_became_active":"2024-01-15T18:03:02.543980+0000","last_became_peered":"2024-01-15T18:03:02.543980+0000","last_unstale":"2024-01-15T18:03:03.555343+0000","last_undegraded":"2024-01-15T18:03:03.555343+0000","last_fullsized":"2024-01-15T18:03:03.555343+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:33.755676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581828+0000","last_change":"2024-01-15T18:03:03.577261+0000","last_active":"2024-01-15T18:03:03.581828+0000","last_peered":"2024-01-15T18:03:03.581828+0000","last_clean":"2024-01-15T18:03:03.581828+0000","last_became_active":"2024-01-15T18:03:03.576662+0000","last_became_peered":"2024-01-15T18:03:03.576662+0000","last_unstale":"2024-01-15T18:03:03.581828+0000","last_undegraded":"2024-01-15T18:03:03.581828+0000","last_fullsized":"2024-01-15T18:03:03.581828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:03.384948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.542911+0000","last_change":"2024-01-15T18:03:02.531700+0000","last_active":"2024-01-15T18:03:03.542911+0000","last_peered":"2024-01-15T18:03:03.542911+0000","last_clean":"2024-01-15T18:03:03.542911+0000","last_became_active":"2024-01-15T18:03:02.531004+0000","last_became_peered":"2024-01-15T18:03:02.531004+0000","last_unstale":"2024-01-15T18:03:03.542911+0000","last_undegraded":"2024-01-15T18:03:03.542911+0000","last_fullsized":"2024-01-15T18:03:03.542911+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:01:24.526602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572095+0000","last_change":"2024-01-15T18:03:03.564631+0000","last_active":"2024-01-15T18:03:03.572095+0000","last_peered":"2024-01-15T18:03:03.572095+0000","last_clean":"2024-01-15T18:03:03.572095+0000","last_became_active":"2024-01-15T18:03:03.564186+0000","last_became_peered":"2024-01-15T18:03:03.564186+0000","last_unstale":"2024-01-15T18:03:03.572095+0000","last_undegraded":"2024-01-15T18:03:03.572095+0000","last_fullsized":"2024-01-15T18:03:03.572095+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:40.840335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"60'3","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580774+0000","last_change":"2024-01-15T18:03:02.543881+0000","last_active":"2024-01-15T18:03:03.580774+0000","last_peered":"2024-01-15T18:03:03.580774+0000","last_clean":"2024-01-15T18:03:03.580774+0000","last_became_active":"2024-01-15T18:03:02.543581+0000","last_became_peered":"2024-01-15T18:03:02.543581+0000","last_unstale":"2024-01-15T18:03:03.580774+0000","last_undegraded":"2024-01-15T18:03:03.580774+0000","last_fullsized":"2024-01-15T18:03:03.580774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:21.056965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581727+0000","last_change":"2024-01-15T18:03:03.576760+0000","last_active":"2024-01-15T18:03:03.581727+0000","last_peered":"2024-01-15T18:03:03.581727+0000","last_clean":"2024-01-15T18:03:03.581727+0000","last_became_active":"2024-01-15T18:03:03.576184+0000","last_became_peered":"2024-01-15T18:03:03.576184+0000","last_unstale":"2024-01-15T18:03:03.581727+0000","last_undegraded":"2024-01-15T18:03:03.581727+0000","last_fullsized":"2024-01-15T18:03:03.581727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:51.286116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"60'6","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.565425+0000","last_change":"2024-01-15T18:03:02.541543+0000","last_active":"2024-01-15T18:03:04.565425+0000","last_peered":"2024-01-15T18:03:04.565425+0000","last_clean":"2024-01-15T18:03:04.565425+0000","last_became_active":"2024-01-15T18:03:02.541266+0000","last_became_peered":"2024-01-15T18:03:02.541266+0000","last_unstale":"2024-01-15T18:03:04.565425+0000","last_undegraded":"2024-01-15T18:03:04.565425+0000","last_fullsized":"2024-01-15T18:03:04.565425+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:46.401878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.567798+0000","last_change":"2024-01-15T18:03:03.566237+0000","last_active":"2024-01-15T18:03:03.567798+0000","last_peered":"2024-01-15T18:03:03.567798+0000","last_clean":"2024-01-15T18:03:03.567798+0000","last_became_active":"2024-01-15T18:03:03.565767+0000","last_became_peered":"2024-01-15T18:03:03.565767+0000","last_unstale":"2024-01-15T18:03:03.567798+0000","last_undegraded":"2024-01-15T18:03:03.567798+0000","last_fullsized":"2024-01-15T18:03:03.567798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:47.991417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575141+0000","last_change":"2024-01-15T18:03:02.532962+0000","last_active":"2024-01-15T18:03:03.575141+0000","last_peered":"2024-01-15T18:03:03.575141+0000","last_clean":"2024-01-15T18:03:03.575141+0000","last_became_active":"2024-01-15T18:03:02.530931+0000","last_became_peered":"2024-01-15T18:03:02.530931+0000","last_unstale":"2024-01-15T18:03:03.575141+0000","last_undegraded":"2024-01-15T18:03:03.575141+0000","last_fullsized":"2024-01-15T18:03:03.575141+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:42:00.049457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581382+0000","last_change":"2024-01-15T18:03:03.573574+0000","last_active":"2024-01-15T18:03:03.581382+0000","last_peered":"2024-01-15T18:03:03.581382+0000","last_clean":"2024-01-15T18:03:03.581382+0000","last_became_active":"2024-01-15T18:03:03.572751+0000","last_became_peered":"2024-01-15T18:03:03.572751+0000","last_unstale":"2024-01-15T18:03:03.581382+0000","last_undegraded":"2024-01-15T18:03:03.581382+0000","last_fullsized":"2024-01-15T18:03:03.581382+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:56.271555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.562539+0000","last_change":"2024-01-15T18:03:02.547172+0000","last_active":"2024-01-15T18:03:03.562539+0000","last_peered":"2024-01-15T18:03:03.562539+0000","last_clean":"2024-01-15T18:03:03.562539+0000","last_became_active":"2024-01-15T18:03:02.546843+0000","last_became_peered":"2024-01-15T18:03:02.546843+0000","last_unstale":"2024-01-15T18:03:03.562539+0000","last_undegraded":"2024-01-15T18:03:03.562539+0000","last_fullsized":"2024-01-15T18:03:03.562539+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:28.593520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562334+0000","last_change":"2024-01-15T18:03:03.574733+0000","last_active":"2024-01-15T18:03:04.562334+0000","last_peered":"2024-01-15T18:03:04.562334+0000","last_clean":"2024-01-15T18:03:04.562334+0000","last_became_active":"2024-01-15T18:03:03.573270+0000","last_became_peered":"2024-01-15T18:03:03.573270+0000","last_unstale":"2024-01-15T18:03:04.562334+0000","last_undegraded":"2024-01-15T18:03:04.562334+0000","last_fullsized":"2024-01-15T18:03:04.562334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:29:02.501394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550900+0000","last_change":"2024-01-15T18:03:02.547792+0000","last_active":"2024-01-15T18:03:03.550900+0000","last_peered":"2024-01-15T18:03:03.550900+0000","last_clean":"2024-01-15T18:03:03.550900+0000","last_became_active":"2024-01-15T18:03:02.545716+0000","last_became_peered":"2024-01-15T18:03:02.545716+0000","last_unstale":"2024-01-15T18:03:03.550900+0000","last_undegraded":"2024-01-15T18:03:03.550900+0000","last_fullsized":"2024-01-15T18:03:03.550900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:33.755676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.583477+0000","last_change":"2024-01-15T18:03:03.580549+0000","last_active":"2024-01-15T18:03:03.583477+0000","last_peered":"2024-01-15T18:03:03.583477+0000","last_clean":"2024-01-15T18:03:03.583477+0000","last_became_active":"2024-01-15T18:03:03.580261+0000","last_became_peered":"2024-01-15T18:03:03.580261+0000","last_unstale":"2024-01-15T18:03:03.583477+0000","last_undegraded":"2024-01-15T18:03:03.583477+0000","last_fullsized":"2024-01-15T18:03:03.583477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:44:05.698832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"60'8","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581627+0000","last_change":"2024-01-15T18:03:02.549962+0000","last_active":"2024-01-15T18:03:03.581627+0000","last_peered":"2024-01-15T18:03:03.581627+0000","last_clean":"2024-01-15T18:03:03.581627+0000","last_became_active":"2024-01-15T18:03:02.549571+0000","last_became_peered":"2024-01-15T18:03:02.549571+0000","last_unstale":"2024-01-15T18:03:03.581627+0000","last_undegraded":"2024-01-15T18:03:03.581627+0000","last_fullsized":"2024-01-15T18:03:03.581627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:58.440091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581717+0000","last_change":"2024-01-15T18:03:03.576245+0000","last_active":"2024-01-15T18:03:03.581717+0000","last_peered":"2024-01-15T18:03:03.581717+0000","last_clean":"2024-01-15T18:03:03.581717+0000","last_became_active":"2024-01-15T18:03:03.573742+0000","last_became_peered":"2024-01-15T18:03:03.573742+0000","last_unstale":"2024-01-15T18:03:03.581717+0000","last_undegraded":"2024-01-15T18:03:03.581717+0000","last_fullsized":"2024-01-15T18:03:03.581717+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:15:50.734220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575043+0000","last_change":"2024-01-15T18:03:02.533913+0000","last_active":"2024-01-15T18:03:03.575043+0000","last_peered":"2024-01-15T18:03:03.575043+0000","last_clean":"2024-01-15T18:03:03.575043+0000","last_became_active":"2024-01-15T18:03:02.533631+0000","last_became_peered":"2024-01-15T18:03:02.533631+0000","last_unstale":"2024-01-15T18:03:03.575043+0000","last_undegraded":"2024-01-15T18:03:03.575043+0000","last_fullsized":"2024-01-15T18:03:03.575043+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:35:37.181184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.574892+0000","last_change":"2024-01-15T18:03:03.570140+0000","last_active":"2024-01-15T18:03:03.574892+0000","last_peered":"2024-01-15T18:03:03.574892+0000","last_clean":"2024-01-15T18:03:03.574892+0000","last_became_active":"2024-01-15T18:03:03.569749+0000","last_became_peered":"2024-01-15T18:03:03.569749+0000","last_unstale":"2024-01-15T18:03:03.574892+0000","last_undegraded":"2024-01-15T18:03:03.574892+0000","last_fullsized":"2024-01-15T18:03:03.574892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:38.275323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579915+0000","last_change":"2024-01-15T18:03:02.530979+0000","last_active":"2024-01-15T18:03:03.579915+0000","last_peered":"2024-01-15T18:03:03.579915+0000","last_clean":"2024-01-15T18:03:03.579915+0000","last_became_active":"2024-01-15T18:03:02.530656+0000","last_became_peered":"2024-01-15T18:03:02.530656+0000","last_unstale":"2024-01-15T18:03:03.579915+0000","last_undegraded":"2024-01-15T18:03:03.579915+0000","last_fullsized":"2024-01-15T18:03:03.579915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:55.258072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572639+0000","last_change":"2024-01-15T18:03:03.563213+0000","last_active":"2024-01-15T18:03:03.572639+0000","last_peered":"2024-01-15T18:03:03.572639+0000","last_clean":"2024-01-15T18:03:03.572639+0000","last_became_active":"2024-01-15T18:03:03.562880+0000","last_became_peered":"2024-01-15T18:03:03.562880+0000","last_unstale":"2024-01-15T18:03:03.572639+0000","last_undegraded":"2024-01-15T18:03:03.572639+0000","last_fullsized":"2024-01-15T18:03:03.572639+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:22.205672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.577084+0000","last_change":"2024-01-15T18:03:02.548504+0000","last_active":"2024-01-15T18:03:03.577084+0000","last_peered":"2024-01-15T18:03:03.577084+0000","last_clean":"2024-01-15T18:03:03.577084+0000","last_became_active":"2024-01-15T18:03:02.548153+0000","last_became_peered":"2024-01-15T18:03:02.548153+0000","last_unstale":"2024-01-15T18:03:03.577084+0000","last_undegraded":"2024-01-15T18:03:03.577084+0000","last_fullsized":"2024-01-15T18:03:03.577084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:44:04.685350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581661+0000","last_change":"2024-01-15T18:03:03.576221+0000","last_active":"2024-01-15T18:03:03.581661+0000","last_peered":"2024-01-15T18:03:03.581661+0000","last_clean":"2024-01-15T18:03:03.581661+0000","last_became_active":"2024-01-15T18:03:03.575870+0000","last_became_peered":"2024-01-15T18:03:03.575870+0000","last_unstale":"2024-01-15T18:03:03.581661+0000","last_undegraded":"2024-01-15T18:03:03.581661+0000","last_fullsized":"2024-01-15T18:03:03.581661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:47.991417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.535336+0000","last_change":"2024-01-15T18:03:02.531412+0000","last_active":"2024-01-15T18:03:03.535336+0000","last_peered":"2024-01-15T18:03:03.535336+0000","last_clean":"2024-01-15T18:03:03.535336+0000","last_became_active":"2024-01-15T18:03:02.530878+0000","last_became_peered":"2024-01-15T18:03:02.530878+0000","last_unstale":"2024-01-15T18:03:03.535336+0000","last_undegraded":"2024-01-15T18:03:03.535336+0000","last_fullsized":"2024-01-15T18:03:03.535336+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:37.261840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.564699+0000","last_change":"2024-01-15T18:03:03.559879+0000","last_active":"2024-01-15T18:03:03.564699+0000","last_peered":"2024-01-15T18:03:03.564699+0000","last_clean":"2024-01-15T18:03:03.564699+0000","last_became_active":"2024-01-15T18:03:03.559149+0000","last_became_peered":"2024-01-15T18:03:03.559149+0000","last_unstale":"2024-01-15T18:03:03.564699+0000","last_undegraded":"2024-01-15T18:03:03.564699+0000","last_fullsized":"2024-01-15T18:03:03.564699+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.947434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575808+0000","last_change":"2024-01-15T18:03:02.536266+0000","last_active":"2024-01-15T18:03:03.575808+0000","last_peered":"2024-01-15T18:03:03.575808+0000","last_clean":"2024-01-15T18:03:03.575808+0000","last_became_active":"2024-01-15T18:03:02.535944+0000","last_became_peered":"2024-01-15T18:03:02.535944+0000","last_unstale":"2024-01-15T18:03:03.575808+0000","last_undegraded":"2024-01-15T18:03:03.575808+0000","last_fullsized":"2024-01-15T18:03:03.575808+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:06:00.779130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.582236+0000","last_change":"2024-01-15T18:03:03.570664+0000","last_active":"2024-01-15T18:03:03.582236+0000","last_peered":"2024-01-15T18:03:03.582236+0000","last_clean":"2024-01-15T18:03:03.582236+0000","last_became_active":"2024-01-15T18:03:03.570381+0000","last_became_peered":"2024-01-15T18:03:03.570381+0000","last_unstale":"2024-01-15T18:03:03.582236+0000","last_undegraded":"2024-01-15T18:03:03.582236+0000","last_fullsized":"2024-01-15T18:03:03.582236+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:59.453574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.552498+0000","last_change":"2024-01-15T18:03:02.529702+0000","last_active":"2024-01-15T18:03:03.552498+0000","last_peered":"2024-01-15T18:03:03.552498+0000","last_clean":"2024-01-15T18:03:03.552498+0000","last_became_active":"2024-01-15T18:03:02.529384+0000","last_became_peered":"2024-01-15T18:03:02.529384+0000","last_unstale":"2024-01-15T18:03:03.552498+0000","last_undegraded":"2024-01-15T18:03:03.552498+0000","last_fullsized":"2024-01-15T18:03:03.552498+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:17.995525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.582008+0000","last_change":"2024-01-15T18:03:03.574453+0000","last_active":"2024-01-15T18:03:03.582008+0000","last_peered":"2024-01-15T18:03:03.582008+0000","last_clean":"2024-01-15T18:03:03.582008+0000","last_became_active":"2024-01-15T18:03:03.574068+0000","last_became_peered":"2024-01-15T18:03:03.574068+0000","last_unstale":"2024-01-15T18:03:03.582008+0000","last_undegraded":"2024-01-15T18:03:03.582008+0000","last_fullsized":"2024-01-15T18:03:03.582008+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:51.286116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.552667+0000","last_change":"2024-01-15T18:03:02.546511+0000","last_active":"2024-01-15T18:03:03.552667+0000","last_peered":"2024-01-15T18:03:03.552667+0000","last_clean":"2024-01-15T18:03:03.552667+0000","last_became_active":"2024-01-15T18:03:02.546101+0000","last_became_peered":"2024-01-15T18:03:02.546101+0000","last_unstale":"2024-01-15T18:03:03.552667+0000","last_undegraded":"2024-01-15T18:03:03.552667+0000","last_fullsized":"2024-01-15T18:03:03.552667+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:55.258072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.567262+0000","last_change":"2024-01-15T18:03:03.563991+0000","last_active":"2024-01-15T18:03:03.567262+0000","last_peered":"2024-01-15T18:03:03.567262+0000","last_clean":"2024-01-15T18:03:03.567262+0000","last_became_active":"2024-01-15T18:03:03.562781+0000","last_became_peered":"2024-01-15T18:03:03.562781+0000","last_unstale":"2024-01-15T18:03:03.567262+0000","last_undegraded":"2024-01-15T18:03:03.567262+0000","last_fullsized":"2024-01-15T18:03:03.567262+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:22.070448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578532+0000","last_change":"2024-01-15T18:03:02.532628+0000","last_active":"2024-01-15T18:03:03.578532+0000","last_peered":"2024-01-15T18:03:03.578532+0000","last_clean":"2024-01-15T18:03:03.578532+0000","last_became_active":"2024-01-15T18:03:02.532254+0000","last_became_peered":"2024-01-15T18:03:02.532254+0000","last_unstale":"2024-01-15T18:03:03.578532+0000","last_undegraded":"2024-01-15T18:03:03.578532+0000","last_fullsized":"2024-01-15T18:03:03.578532+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:11.535510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.574872+0000","last_change":"2024-01-15T18:03:03.569997+0000","last_active":"2024-01-15T18:03:03.574872+0000","last_peered":"2024-01-15T18:03:03.574872+0000","last_clean":"2024-01-15T18:03:03.574872+0000","last_became_active":"2024-01-15T18:03:03.569125+0000","last_became_peered":"2024-01-15T18:03:03.569125+0000","last_unstale":"2024-01-15T18:03:03.574872+0000","last_undegraded":"2024-01-15T18:03:03.574872+0000","last_fullsized":"2024-01-15T18:03:03.574872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:15:50.734220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.537561+0000","last_change":"2024-01-15T18:03:02.544503+0000","last_active":"2024-01-15T18:03:03.537561+0000","last_peered":"2024-01-15T18:03:03.537561+0000","last_clean":"2024-01-15T18:03:03.537561+0000","last_became_active":"2024-01-15T18:03:02.544227+0000","last_became_peered":"2024-01-15T18:03:02.544227+0000","last_unstale":"2024-01-15T18:03:03.537561+0000","last_undegraded":"2024-01-15T18:03:03.537561+0000","last_fullsized":"2024-01-15T18:03:03.537561+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:04:40.709592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572073+0000","last_change":"2024-01-15T18:03:03.564455+0000","last_active":"2024-01-15T18:03:03.572073+0000","last_peered":"2024-01-15T18:03:03.572073+0000","last_clean":"2024-01-15T18:03:03.572073+0000","last_became_active":"2024-01-15T18:03:03.563837+0000","last_became_peered":"2024-01-15T18:03:03.563837+0000","last_unstale":"2024-01-15T18:03:03.572073+0000","last_undegraded":"2024-01-15T18:03:03.572073+0000","last_fullsized":"2024-01-15T18:03:03.572073+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:54.826330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561586+0000","last_change":"2024-01-15T18:03:02.543415+0000","last_active":"2024-01-15T18:03:03.561586+0000","last_peered":"2024-01-15T18:03:03.561586+0000","last_clean":"2024-01-15T18:03:03.561586+0000","last_became_active":"2024-01-15T18:03:02.543113+0000","last_became_peered":"2024-01-15T18:03:02.543113+0000","last_unstale":"2024-01-15T18:03:03.561586+0000","last_undegraded":"2024-01-15T18:03:03.561586+0000","last_fullsized":"2024-01-15T18:03:03.561586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:47.609801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579921+0000","last_change":"2024-01-15T18:03:03.563481+0000","last_active":"2024-01-15T18:03:03.579921+0000","last_peered":"2024-01-15T18:03:03.579921+0000","last_clean":"2024-01-15T18:03:03.579921+0000","last_became_active":"2024-01-15T18:03:03.562880+0000","last_became_peered":"2024-01-15T18:03:03.562880+0000","last_unstale":"2024-01-15T18:03:03.579921+0000","last_undegraded":"2024-01-15T18:03:03.579921+0000","last_fullsized":"2024-01-15T18:03:03.579921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.947434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.555786+0000","last_change":"2024-01-15T18:03:02.549187+0000","last_active":"2024-01-15T18:03:03.555786+0000","last_peered":"2024-01-15T18:03:03.555786+0000","last_clean":"2024-01-15T18:03:03.555786+0000","last_became_active":"2024-01-15T18:03:02.548364+0000","last_became_peered":"2024-01-15T18:03:02.548364+0000","last_unstale":"2024-01-15T18:03:03.555786+0000","last_undegraded":"2024-01-15T18:03:03.555786+0000","last_fullsized":"2024-01-15T18:03:03.555786+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:12.259268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.571818+0000","last_change":"2024-01-15T18:03:03.555932+0000","last_active":"2024-01-15T18:03:03.571818+0000","last_peered":"2024-01-15T18:03:03.571818+0000","last_clean":"2024-01-15T18:03:03.571818+0000","last_became_active":"2024-01-15T18:03:03.555615+0000","last_became_peered":"2024-01-15T18:03:03.555615+0000","last_unstale":"2024-01-15T18:03:03.571818+0000","last_undegraded":"2024-01-15T18:03:03.571818+0000","last_fullsized":"2024-01-15T18:03:03.571818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:35.978962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.552248+0000","last_change":"2024-01-15T18:03:02.543613+0000","last_active":"2024-01-15T18:03:03.552248+0000","last_peered":"2024-01-15T18:03:03.552248+0000","last_clean":"2024-01-15T18:03:03.552248+0000","last_became_active":"2024-01-15T18:03:02.543206+0000","last_became_peered":"2024-01-15T18:03:02.543206+0000","last_unstale":"2024-01-15T18:03:03.552248+0000","last_undegraded":"2024-01-15T18:03:03.552248+0000","last_fullsized":"2024-01-15T18:03:03.552248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:47:35.101403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581574+0000","last_change":"2024-01-15T18:03:03.576012+0000","last_active":"2024-01-15T18:03:03.581574+0000","last_peered":"2024-01-15T18:03:03.581574+0000","last_clean":"2024-01-15T18:03:03.581574+0000","last_became_active":"2024-01-15T18:03:03.573946+0000","last_became_peered":"2024-01-15T18:03:03.573946+0000","last_unstale":"2024-01-15T18:03:03.581574+0000","last_undegraded":"2024-01-15T18:03:03.581574+0000","last_fullsized":"2024-01-15T18:03:03.581574+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:04:41.723075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.564626+0000","last_change":"2024-01-15T18:03:02.534835+0000","last_active":"2024-01-15T18:03:04.564626+0000","last_peered":"2024-01-15T18:03:04.564626+0000","last_clean":"2024-01-15T18:03:04.564626+0000","last_became_active":"2024-01-15T18:03:02.533194+0000","last_became_peered":"2024-01-15T18:03:02.533194+0000","last_unstale":"2024-01-15T18:03:04.564626+0000","last_undegraded":"2024-01-15T18:03:04.564626+0000","last_fullsized":"2024-01-15T18:03:04.564626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:52:07.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572040+0000","last_change":"2024-01-15T18:03:03.567136+0000","last_active":"2024-01-15T18:03:03.572040+0000","last_peered":"2024-01-15T18:03:03.572040+0000","last_clean":"2024-01-15T18:03:03.572040+0000","last_became_active":"2024-01-15T18:03:03.566863+0000","last_became_peered":"2024-01-15T18:03:03.566863+0000","last_unstale":"2024-01-15T18:03:03.572040+0000","last_undegraded":"2024-01-15T18:03:03.572040+0000","last_fullsized":"2024-01-15T18:03:03.572040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:58.996025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.553947+0000","last_change":"2024-01-15T18:03:02.532855+0000","last_active":"2024-01-15T18:03:03.553947+0000","last_peered":"2024-01-15T18:03:03.553947+0000","last_clean":"2024-01-15T18:03:03.553947+0000","last_became_active":"2024-01-15T18:03:02.532576+0000","last_became_peered":"2024-01-15T18:03:02.532576+0000","last_unstale":"2024-01-15T18:03:03.553947+0000","last_undegraded":"2024-01-15T18:03:03.553947+0000","last_fullsized":"2024-01-15T18:03:03.553947+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:15.666218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578626+0000","last_change":"2024-01-15T18:03:03.575811+0000","last_active":"2024-01-15T18:03:03.578626+0000","last_peered":"2024-01-15T18:03:03.578626+0000","last_clean":"2024-01-15T18:03:03.578626+0000","last_became_active":"2024-01-15T18:03:03.574055+0000","last_became_peered":"2024-01-15T18:03:03.574055+0000","last_unstale":"2024-01-15T18:03:03.578626+0000","last_undegraded":"2024-01-15T18:03:03.578626+0000","last_fullsized":"2024-01-15T18:03:03.578626+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:55:09.343429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550113+0000","last_change":"2024-01-15T18:03:02.543820+0000","last_active":"2024-01-15T18:03:03.550113+0000","last_peered":"2024-01-15T18:03:03.550113+0000","last_clean":"2024-01-15T18:03:03.550113+0000","last_became_active":"2024-01-15T18:03:02.543309+0000","last_became_peered":"2024-01-15T18:03:02.543309+0000","last_unstale":"2024-01-15T18:03:03.550113+0000","last_undegraded":"2024-01-15T18:03:03.550113+0000","last_fullsized":"2024-01-15T18:03:03.550113+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:34.446304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.577403+0000","last_change":"2024-01-15T18:03:03.575523+0000","last_active":"2024-01-15T18:03:03.577403+0000","last_peered":"2024-01-15T18:03:03.577403+0000","last_clean":"2024-01-15T18:03:03.577403+0000","last_became_active":"2024-01-15T18:03:03.575221+0000","last_became_peered":"2024-01-15T18:03:03.575221+0000","last_unstale":"2024-01-15T18:03:03.577403+0000","last_undegraded":"2024-01-15T18:03:03.577403+0000","last_fullsized":"2024-01-15T18:03:03.577403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:22.491470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561353+0000","last_change":"2024-01-15T18:03:02.541780+0000","last_active":"2024-01-15T18:03:03.561353+0000","last_peered":"2024-01-15T18:03:03.561353+0000","last_clean":"2024-01-15T18:03:03.561353+0000","last_became_active":"2024-01-15T18:03:02.541337+0000","last_became_peered":"2024-01-15T18:03:02.541337+0000","last_unstale":"2024-01-15T18:03:03.561353+0000","last_undegraded":"2024-01-15T18:03:03.561353+0000","last_fullsized":"2024-01-15T18:03:03.561353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:41.075647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":200,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-15T18:03:02.530450+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:03:02.530450+0000","last_peered":"2024-01-15T18:03:02.530450+0000","last_clean":"2024-01-15T18:03:02.530450+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:03:02.530450+0000","last_undegraded":"2024-01-15T18:03:02.530450+0000","last_fullsized":"2024-01-15T18:03:02.530450+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542214,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1318912,"data_stored":891032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2400000000000002,"5min":1.5549999999999999,"15min":1.2470000000000001},"min":{"1min":0.622,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":14.114000000000001,"5min":14.114000000000001,"15min":14.114000000000001},"last":1.206},{"interface":"front","average":{"1min":1.75,"5min":1.52,"15min":1.4650000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":6.3440000000000003,"5min":12.496,"15min":12.496},"last":1.3520000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.413,"15min":1.204},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.0229999999999997,"5min":14.497,"15min":14.497},"last":1.1359999999999999},{"interface":"front","average":{"1min":2.2440000000000002,"5min":1.444,"15min":1.2190000000000001},"min":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":12.565,"5min":12.565,"15min":12.565},"last":1.4059999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":1.631,"15min":1.2809999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":12.449,"5min":13.801,"15min":13.801},"last":1.272},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.3680000000000001,"15min":1.4419999999999999},"min":{"1min":0.61499999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.9669999999999996,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.2410000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.782,"5min":1.5409999999999999,"15min":1.4239999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":6.3879999999999999,"5min":14.622,"15min":14.622},"last":0.75600000000000001},{"interface":"front","average":{"1min":2.3620000000000001,"5min":1.5940000000000001,"15min":1.419},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":14.042999999999999,"5min":14.042999999999999,"15min":14.042999999999999},"last":1.0489999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.4810000000000001,"15min":1.577},"min":{"1min":0.66400000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":7.3529999999999998,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.625,"5min":1.5740000000000001,"15min":1.452},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":4.3929999999999998,"5min":14.711,"15min":14.711},"last":1.3140000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":1.6839999999999999,"15min":1.587},"min":{"1min":0.47499999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":12.518000000000001,"5min":12.518000000000001,"15min":12.518000000000001},"last":1.466},{"interface":"front","average":{"1min":1.734,"5min":1.5229999999999999,"15min":1.6459999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.2910000000000004,"5min":7.798,"15min":7.798},"last":1.492}]},{"osd":6,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":9.4589999999999996,"5min":3.1880000000000002,"15min":2.2469999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":53.664999999999999,"5min":53.664999999999999,"15min":53.664999999999999},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.6539999999999999,"15min":1.532},"min":{"1min":0.52400000000000002,"5min":0.44,"15min":0.44},"max":{"1min":4.0970000000000004,"5min":14.804,"15min":14.804},"last":1.3879999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40200,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331640,"statfs":{"total":96636764160,"available":96595599360,"internally_reserved":0,"allocated":716800,"data_stored":300504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.224,"15min":1.1100000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":5.0890000000000004,"5min":5.0890000000000004,"15min":5.0890000000000004},"last":1.046},{"interface":"front","average":{"1min":2.7250000000000001,"5min":1.46,"15min":1.236},"min":{"1min":0.80700000000000005,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":21.565000000000001,"5min":21.565000000000001,"15min":21.565000000000001},"last":1.5660000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.3260000000000001,"15min":1.2549999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":5.4169999999999998,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.766,"5min":1.3640000000000001,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"last":1.141}]},{"osd":2,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":1.7010000000000001,"15min":1.417},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":34.670000000000002,"5min":34.670000000000002,"15min":34.670000000000002},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.819,"5min":1.425,"15min":1.3200000000000001},"min":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":5.125,"5min":5.125,"15min":5.125},"last":1.4550000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.5209999999999999,"15min":1.4059999999999999},"min":{"1min":0.85399999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":1.4970000000000001},{"interface":"front","average":{"1min":3.4580000000000002,"5min":1.8520000000000001,"15min":1.538},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":34.725000000000001,"5min":34.725000000000001,"15min":34.725000000000001},"last":1.097}]},{"osd":4,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.454,"15min":1.3779999999999999},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":8.5489999999999995,"5min":8.5489999999999995,"15min":8.5489999999999995},"last":1.371},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.5089999999999999,"15min":1.5109999999999999},"min":{"1min":0.63200000000000001,"5min":0.501,"15min":0.501},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.90200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":3.3900000000000001,"5min":1.7649999999999999,"15min":1.5009999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":34.798000000000002,"5min":34.798000000000002,"15min":34.798000000000002},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.5940000000000001,"15min":1.528},"min":{"1min":0.59699999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.0620000000000003,"5min":7.0620000000000003,"15min":7.0620000000000003},"last":1.278}]},{"osd":7,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":1.6040000000000001,"15min":1.673},"min":{"1min":0.72699999999999998,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":8.2590000000000003,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":1.331},{"interface":"front","average":{"1min":3.669,"5min":1.968,"15min":1.776},"min":{"1min":0.61799999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":34.762999999999998,"5min":34.762999999999998,"15min":34.762999999999998},"last":1.417}]}]},{"osd":5,"up_from":14,"seq":60129542216,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":301156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":1.274,"15min":1.1830000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":10.207000000000001,"5min":10.207000000000001,"15min":10.207000000000001},"last":1.252},{"interface":"front","average":{"1min":2.6789999999999998,"5min":1.6299999999999999,"15min":1.3580000000000001},"min":{"1min":0.77600000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":1.1479999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2450000000000001,"15min":1.163},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":3.8690000000000002,"5min":4.806,"15min":4.806},"last":1.544},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.3089999999999999,"15min":1.161},"min":{"1min":0.66700000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":5.4630000000000001,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":1.3140000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.375,"15min":1.333},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":3.9129999999999998,"5min":5.516,"15min":5.516},"last":1.425},{"interface":"front","average":{"1min":2.1520000000000001,"5min":1.5940000000000001,"15min":1.4339999999999999},"min":{"1min":0.80800000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":10.327999999999999,"5min":10.327999999999999,"15min":10.327999999999999},"last":1.353}]},{"osd":3,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":2.1190000000000002,"5min":1.45,"15min":1.3109999999999999},"min":{"1min":0.70299999999999996,"5min":0.622,"15min":0.622},"max":{"1min":10.249000000000001,"5min":10.249000000000001,"15min":10.249000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":2.0350000000000001,"5min":1.522,"15min":1.4339999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":9.3130000000000006,"5min":9.3130000000000006,"15min":9.3130000000000006},"last":1.6020000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9729999999999999,"5min":1.6779999999999999,"15min":1.419},"min":{"1min":0.65800000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":25.776,"5min":25.776,"15min":25.776},"last":0.98899999999999999},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.696,"15min":1.538},"min":{"1min":0.63800000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.231,"5min":10.231,"15min":10.231},"last":1.081}]},{"osd":6,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":3.1339999999999999,"5min":1.6619999999999999,"15min":1.482},"min":{"1min":0.97699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.202},{"interface":"front","average":{"1min":3.9790000000000001,"5min":2.0569999999999999,"15min":1.6830000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":36.087000000000003,"5min":36.087000000000003,"15min":36.087000000000003},"last":1.0409999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":3.2610000000000001,"5min":1.992,"15min":1.734},"min":{"1min":0.89900000000000002,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":12.734,"5min":12.734,"15min":12.734},"last":1.482},{"interface":"front","average":{"1min":2.8769999999999998,"5min":1.8049999999999999,"15min":1.6379999999999999},"min":{"1min":0.81599999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":11.609,"5min":11.609,"15min":11.609},"last":1.119}]}]},{"osd":4,"up_from":14,"seq":60129542215,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40220,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331620,"statfs":{"total":96636764160,"available":96595578880,"internally_reserved":0,"allocated":729088,"data_stored":302116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":1.663,"15min":1.2689999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":18.673999999999999,"5min":21.643000000000001,"15min":21.643000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.786,"5min":1.286,"15min":1.1619999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":6.4989999999999997,"5min":6.4989999999999997,"15min":6.4989999999999997},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.1139999999999999,"5min":1.4239999999999999,"15min":1.242},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":12.738,"5min":12.738,"15min":12.738},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.214,"5min":1.617,"15min":1.331},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.9119999999999999,"5min":21.440000000000001,"15min":21.440000000000001},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5030000000000001,"5min":1.524,"15min":1.258},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":12.974,"5min":12.974,"15min":12.974},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.294,"5min":1.6499999999999999,"15min":1.3979999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.9850000000000003,"5min":22.350999999999999,"15min":22.350999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.4209999999999998,"5min":1.7,"15min":1.4470000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":15.093,"5min":15.093,"15min":15.093},"last":0.72199999999999998},{"interface":"front","average":{"1min":3.169,"5min":1.8149999999999999,"15min":1.4810000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":19.486000000000001,"5min":19.486000000000001,"15min":19.486000000000001},"last":0.73599999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.2210000000000001,"5min":1.9279999999999999,"15min":1.5569999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":11.731,"5min":21.285,"15min":21.285},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.262,"5min":1.5009999999999999,"15min":1.4330000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":18.289999999999999,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":3.0209999999999999,"5min":1.73,"15min":1.51},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":17.733000000000001,"5min":17.733000000000001,"15min":17.733000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":3.278,"5min":2.2000000000000002,"15min":1.7310000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":28.797000000000001,"5min":28.797000000000001,"15min":28.797000000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.5980000000000001,"15min":1.5509999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.2210000000000001,"5min":7.6429999999999998,"15min":7.6429999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":3.032,"5min":1.9810000000000001,"15min":1.663},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":22.751000000000001,"5min":22.751000000000001,"15min":22.751000000000001},"last":0.76800000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542216,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44364,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327476,"statfs":{"total":96636764160,"available":96591335424,"internally_reserved":0,"allocated":716800,"data_stored":300504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44693960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":0.83099999999999996},"min":{"1min":0.50600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.246,"5min":4.9169999999999998,"15min":4.9169999999999998},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.054,"5min":0.98599999999999999,"15min":0.91200000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.841,"5min":5.0529999999999999,"15min":5.0529999999999999},"last":0.73699999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1180000000000001,"15min":1.036},"min":{"1min":0.59799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":5.7060000000000004,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.1359999999999999,"15min":1.1200000000000001},"min":{"1min":0.67700000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6619999999999999,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":1.131}]},{"osd":2,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.1579999999999999,"15min":1.0960000000000001},"min":{"1min":0.55200000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.839,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.272,"15min":1.22},"min":{"1min":0.58799999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.1219999999999999,"5min":5.0910000000000002,"15min":5.0910000000000002},"last":1.712}]},{"osd":4,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.421,"15min":1.3120000000000001},"min":{"1min":0.75800000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.75,"5min":17.082000000000001,"15min":17.082000000000001},"last":1.546},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.478,"15min":1.3740000000000001},"min":{"1min":0.82199999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.839,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.4390000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.506,"15min":1.3839999999999999},"min":{"1min":1.016,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.4399999999999999,"5min":2.718,"15min":2.718},"last":1.181},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.9930000000000001,"15min":1.605},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.3530000000000002,"5min":40.384999999999998,"15min":40.384999999999998},"last":1.778}]},{"osd":6,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4059999999999999,"15min":1.321},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4199999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5900000000000001,"15min":1.5309999999999999},"min":{"1min":0.88100000000000001,"5min":0.873,"15min":0.873},"max":{"1min":2.4020000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"last":1.5660000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.7070000000000001,"15min":1.6200000000000001},"min":{"1min":0.84099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":10.202999999999999},"last":1.825},{"interface":"front","average":{"1min":2.0350000000000001,"5min":1.6259999999999999,"15min":1.476},"min":{"1min":1.056,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.3999999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542215,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44376,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327464,"statfs":{"total":96636764160,"available":96591323136,"internally_reserved":0,"allocated":724992,"data_stored":300560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.022,"15min":1.099},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":6.101,"5min":6.101,"15min":6.101},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.022,"15min":1.1719999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.1779999999999999,"15min":1.052},"min":{"1min":0.434,"5min":0.379,"15min":0.379},"max":{"1min":10.807,"5min":10.807,"15min":10.807},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.155,"15min":1.23},"min":{"1min":0.497,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.403,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.359,"15min":1.2350000000000001},"min":{"1min":0.48699999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.627},{"interface":"front","average":{"1min":1.365,"5min":1.216,"15min":1.1619999999999999},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":7.2599999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.5489999999999999,"15min":1.569},"min":{"1min":0.76800000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":7.3209999999999997,"5min":7.3209999999999997,"15min":7.3209999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.8089999999999999,"5min":1.4750000000000001,"15min":1.444},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.7150000000000001,"15min":1.5449999999999999},"min":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":11.016999999999999,"5min":11.016999999999999,"15min":11.016999999999999},"last":0.879},{"interface":"front","average":{"1min":3.5579999999999998,"5min":1.929,"15min":1.681},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":37.020000000000003,"5min":37.020000000000003,"15min":37.020000000000003},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":3.012,"5min":1.8440000000000001,"15min":1.7310000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":23.762,"5min":23.762,"15min":23.762},"last":0.72099999999999997},{"interface":"front","average":{"1min":4.6059999999999999,"5min":2.1600000000000001,"15min":1.698},"min":{"1min":1.0389999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":39.926000000000002,"5min":39.926000000000002,"15min":39.926000000000002},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":3.6640000000000001,"5min":1.998,"15min":1.788},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":52.767000000000003,"5min":52.767000000000003,"15min":52.767000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.4580000000000002,"5min":1.7989999999999999,"15min":1.649},"min":{"1min":1.052,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":13.055999999999999,"5min":13.055999999999999,"15min":13.055999999999999},"last":0.83499999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542215,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40208,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":720896,"data_stored":300594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":0.96699999999999997,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.5939999999999999,"5min":3.5939999999999999,"15min":3.5939999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.173,"5min":1.1379999999999999,"15min":1.0900000000000001},"min":{"1min":0.44500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.3079999999999998,"5min":3.3159999999999998,"15min":3.3159999999999998},"last":1.24}]},{"osd":2,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.254,"15min":1.0249999999999999},"min":{"1min":0.44700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":6.8899999999999997,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":1.097},{"interface":"front","average":{"1min":1.667,"5min":1.1819999999999999,"15min":1.109},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":16.795999999999999,"5min":16.795999999999999,"15min":16.795999999999999},"last":1.052}]},{"osd":3,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.3180000000000001,"15min":1.2569999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6619999999999999,"5min":7.8659999999999997,"15min":7.8659999999999997},"last":1.171},{"interface":"front","average":{"1min":1.25,"5min":1.369,"15min":1.2649999999999999},"min":{"1min":0.68300000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.99,"5min":14.071,"15min":14.071},"last":1.679}]},{"osd":4,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.4610000000000001,"15min":1.421},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":7.3239999999999998,"5min":7.3239999999999998,"15min":7.3239999999999998},"last":1.306},{"interface":"front","average":{"1min":2.976,"5min":1.8460000000000001,"15min":1.6060000000000001},"min":{"1min":1.196,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":22.204000000000001,"5min":22.204000000000001,"15min":22.204000000000001},"last":2.0750000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":1.72,"15min":1.6000000000000001},"min":{"1min":0.76000000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":14.727,"5min":14.727,"15min":14.727},"last":1.95},{"interface":"front","average":{"1min":3.2290000000000001,"5min":1.881,"15min":1.5529999999999999},"min":{"1min":0.58799999999999997,"5min":0.499,"15min":0.499},"max":{"1min":19.998999999999999,"5min":19.998999999999999,"15min":19.998999999999999},"last":1.3660000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.7050000000000001,"15min":1.5649999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.0030000000000001,"5min":10.089,"15min":10.089},"last":1.4870000000000001},{"interface":"front","average":{"1min":3.407,"5min":1.962,"15min":1.7150000000000001},"min":{"1min":0.80200000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":27.579999999999998,"5min":27.579999999999998,"15min":27.579999999999998},"last":2.1579999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.8029999999999999,"15min":1.591},"min":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":9.7780000000000005,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.944,"5min":1.764,"15min":1.6259999999999999},"min":{"1min":0.66700000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":8.0749999999999993,"5min":8.1940000000000008,"15min":8.1940000000000008},"last":1.5900000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542214,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1327104,"data_stored":893066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.1819999999999999,"15min":0.93200000000000005},"min":{"1min":0.437,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.9060000000000006,"5min":17.207000000000001,"15min":17.207000000000001},"last":1.393},{"interface":"front","average":{"1min":1.228,"5min":1.056,"15min":1.04},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.0910000000000002,"5min":3.056,"15min":3.056},"last":0.84799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.52600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.093},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3120000000000001,"15min":1.198},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.917,"5min":10.942,"15min":10.942},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.1619999999999999,"15min":1.0720000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":8.9949999999999992,"5min":8.9949999999999992,"15min":8.9949999999999992},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.3300000000000001,"15min":1.306},"min":{"1min":0.81599999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":5.8360000000000003,"5min":5.8360000000000003,"15min":5.8360000000000003},"last":1.4870000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":2.9460000000000002,"5min":1.7210000000000001,"15min":1.5009999999999999},"min":{"1min":0.97199999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":28.631,"5min":28.631,"15min":28.631},"last":1.046},{"interface":"front","average":{"1min":2.137,"5min":1.4570000000000001,"15min":1.377},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":15.968999999999999,"5min":15.968999999999999,"15min":15.968999999999999},"last":1.2609999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":2.5619999999999998,"5min":1.647,"15min":1.4870000000000001},"min":{"1min":0.70299999999999996,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":9.1059999999999999,"5min":9.1059999999999999,"15min":9.1059999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.712,"15min":1.542},"min":{"1min":1.3140000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":8.4369999999999994},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9089999999999998,"5min":1.976,"15min":1.6399999999999999},"min":{"1min":1.1870000000000001,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":36.390999999999998,"5min":36.390999999999998,"15min":36.390999999999998},"last":1.3240000000000001},{"interface":"front","average":{"1min":3.7349999999999999,"5min":1.9490000000000001,"15min":1.655},"min":{"1min":1.0660000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":36.207000000000001,"5min":36.207000000000001,"15min":36.207000000000001},"last":1.4359999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":2.641,"5min":1.7789999999999999,"15min":1.653},"min":{"1min":1.0609999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":12.164,"5min":12.164,"15min":12.164},"last":1.5760000000000001},{"interface":"front","average":{"1min":2.0640000000000001,"5min":1.577,"15min":1.496},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":9.0749999999999993,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":1.544}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:03:08.018 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:03:08.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.276+0000 7fc274a00640 1 Processor -- start 2024-01-15T18:03:08.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.277+0000 7fc274a00640 1 -- start start 2024-01-15T18:03:08.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.278+0000 7fc274a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc270143810 0x7fc27006dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.278+0000 7fc274a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc27006e010 0x7fc2700b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.278+0000 7fc274a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2700b2080 0x7fc2700700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.278+0000 7fc274a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2700b3520 con 0x7fc270143810 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.278+0000 7fc274a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27006e470 con 0x7fc27006e010 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.278+0000 7fc274a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700ccbe0 con 0x7fc2700b2080 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc270143810 0x7fc27006dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc270143810 0x7fc27006dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56562/0 (socket says 172.21.15.2:56562) 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 -- 172.21.15.2:0/1419572574 learned_addr learned my addr 172.21.15.2:0/1419572574 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 -- 172.21.15.2:0/1419572574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2700b2080 msgr2=0x7fc2700700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 --2- 172.21.15.2:0/1419572574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2700b2080 0x7fc2700700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 -- 172.21.15.2:0/1419572574 >> v2:172.21.15.137:3300/0 conn(0x7fc27006e010 msgr2=0x7fc2700b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 --2- 172.21.15.2:0/1419572574 >> v2:172.21.15.137:3300/0 conn(0x7fc27006e010 0x7fc2700b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.279+0000 7fc26f400640 1 -- 172.21.15.2:0/1419572574 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270143bf0 con 0x7fc270143810 2024-01-15T18:03:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.280+0000 7fc26f400640 1 --2- 172.21.15.2:0/1419572574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27006dad0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc2700cbba0 tx=0x7fc26002b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38d4fe636bfd817 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.281+0000 7fc26ea00640 1 -- 172.21.15.2:0/1419572574 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2600317f0 con 0x7fc270143810 2024-01-15T18:03:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.281+0000 7fc26ea00640 1 -- 172.21.15.2:0/1419572574 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc260031950 con 0x7fc270143810 2024-01-15T18:03:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.281+0000 7fc26ea00640 1 -- 172.21.15.2:0/1419572574 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc260031ce0 con 0x7fc270143810 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.281+0000 7fc274a00640 1 -- 172.21.15.2:0/1419572574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 msgr2=0x7fc27006dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.281+0000 7fc274a00640 1 --2- 172.21.15.2:0/1419572574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27006dad0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fc2700cbba0 tx=0x7fc26002b890 comp rx=0 tx=0).stop 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 -- 172.21.15.2:0/1419572574 shutdown_connections 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 --2- 172.21.15.2:0/1419572574 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2700b2080 0x7fc2700700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 --2- 172.21.15.2:0/1419572574 >> v2:172.21.15.137:3300/0 conn(0x7fc27006e010 0x7fc2700b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 --2- 172.21.15.2:0/1419572574 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27006dad0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 -- 172.21.15.2:0/1419572574 >> 172.21.15.2:0/1419572574 conn(0x7fc2700b0840 msgr2=0x7fc2700b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 -- 172.21.15.2:0/1419572574 shutdown_connections 2024-01-15T18:03:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.282+0000 7fc274a00640 1 -- 172.21.15.2:0/1419572574 wait complete. 2024-01-15T18:03:08.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.283+0000 7fc274a00640 1 Processor -- start 2024-01-15T18:03:08.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 -- start start 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc27006e010 0x7fc2700c4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2700b2080 0x7fc270165d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27015f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc270166240 con 0x7fc270143810 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700c5bd0 con 0x7fc2700b2080 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.284+0000 7fc274a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2700c4b30 con 0x7fc27006e010 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27015f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27015f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56578/0 (socket says 172.21.15.2:56578) 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 -- 172.21.15.2:0/2919415408 learned_addr learned my addr 172.21.15.2:0/2919415408 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26f400640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc27006e010 0x7fc2700c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc266800640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2700b2080 0x7fc270165d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 -- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc27006e010 msgr2=0x7fc2700c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc27006e010 0x7fc2700c4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 -- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2700b2080 msgr2=0x7fc270165d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2700b2080 0x7fc270165d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.285+0000 7fc26fe00640 1 -- 172.21.15.2:0/2919415408 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600094d0 con 0x7fc270143810 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.286+0000 7fc26fe00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27015f340 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc26800b970 tx=0x7fc26800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.286+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26800cb40 con 0x7fc270143810 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.286+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26800ccc0 con 0x7fc270143810 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.286+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268014680 con 0x7fc270143810 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.286+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2700c4e10 con 0x7fc270143810 2024-01-15T18:03:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.286+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2700c53d0 con 0x7fc270143810 2024-01-15T18:03:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.287+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc270074770 con 0x7fc270143810 2024-01-15T18:03:08.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.288+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fc26801d3b0 con 0x7fc270143810 2024-01-15T18:03:08.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.289+0000 7fc267e00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc24806b860 0x7fc24806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.289+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fc26808a590 con 0x7fc270143810 2024-01-15T18:03:08.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.290+0000 7fc26f400640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc24806b860 0x7fc24806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.291+0000 7fc26f400640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc24806b860 0x7fc24806dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc260036000 tx=0x7fc26003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.294+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc268054be0 con 0x7fc270143810 2024-01-15T18:03:08.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.490+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2700cc590 con 0x7fc24806b860 2024-01-15T18:03:08.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.514+0000 7fc267e00640 1 -- 172.21.15.2:0/2919415408 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287225 (secure 0 0 0) 0x7fc260031bc0 con 0x7fc24806b860 2024-01-15T18:03:08.522 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:08.524 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:03:08.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.519+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc24806b860 msgr2=0x7fc24806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.519+0000 7fc274a00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc24806b860 0x7fc24806dd20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc260036000 tx=0x7fc26003d040 comp rx=0 tx=0).stop 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 msgr2=0x7fc27015f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27015f340 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fc26800b970 tx=0x7fc26800be30 comp rx=0 tx=0).stop 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 shutdown_connections 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc24806b860 0x7fc24806dd20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc270143810 0x7fc27015f340 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2700b2080 0x7fc270165d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 --2- 172.21.15.2:0/2919415408 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc27006e010 0x7fc2700c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 >> 172.21.15.2:0/2919415408 conn(0x7fc2700b0840 msgr2=0x7fc270160270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 shutdown_connections 2024-01-15T18:03:08.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.520+0000 7fc274a00640 1 -- 172.21.15.2:0/2919415408 wait complete. 2024-01-15T18:03:08.546 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2024-01-15T18:03:07.515568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":122,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":310360,"kb_used_data":6816,"kb_used_omap":12,"kb_used_meta":303411,"kb_avail":754664360,"statfs":{"total":773094113280,"available":772776304640,"internally_reserved":0,"allocated":6979584,"data_stored":3589532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":310693425},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002320"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561769+0000","last_change":"2024-01-15T18:03:03.556818+0000","last_active":"2024-01-15T18:03:03.561769+0000","last_peered":"2024-01-15T18:03:03.561769+0000","last_clean":"2024-01-15T18:03:03.561769+0000","last_became_active":"2024-01-15T18:03:03.556532+0000","last_became_peered":"2024-01-15T18:03:03.556532+0000","last_unstale":"2024-01-15T18:03:03.561769+0000","last_undegraded":"2024-01-15T18:03:03.561769+0000","last_fullsized":"2024-01-15T18:03:03.561769+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:04.576059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572671+0000","last_change":"2024-01-15T18:03:03.564151+0000","last_active":"2024-01-15T18:03:03.572671+0000","last_peered":"2024-01-15T18:03:03.572671+0000","last_clean":"2024-01-15T18:03:03.572671+0000","last_became_active":"2024-01-15T18:03:03.563896+0000","last_became_peered":"2024-01-15T18:03:03.563896+0000","last_unstale":"2024-01-15T18:03:03.572671+0000","last_undegraded":"2024-01-15T18:03:03.572671+0000","last_fullsized":"2024-01-15T18:03:03.572671+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:26.579782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575880+0000","last_change":"2024-01-15T18:03:03.569042+0000","last_active":"2024-01-15T18:03:03.575880+0000","last_peered":"2024-01-15T18:03:03.575880+0000","last_clean":"2024-01-15T18:03:03.575880+0000","last_became_active":"2024-01-15T18:03:03.568645+0000","last_became_peered":"2024-01-15T18:03:03.568645+0000","last_unstale":"2024-01-15T18:03:03.575880+0000","last_undegraded":"2024-01-15T18:03:03.575880+0000","last_fullsized":"2024-01-15T18:03:03.575880+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:52:26.067882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561651+0000","last_change":"2024-01-15T18:03:03.557155+0000","last_active":"2024-01-15T18:03:03.561651+0000","last_peered":"2024-01-15T18:03:03.561651+0000","last_clean":"2024-01-15T18:03:03.561651+0000","last_became_active":"2024-01-15T18:03:03.556824+0000","last_became_peered":"2024-01-15T18:03:03.556824+0000","last_unstale":"2024-01-15T18:03:03.561651+0000","last_undegraded":"2024-01-15T18:03:03.561651+0000","last_fullsized":"2024-01-15T18:03:03.561651+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:52:26.067882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572356+0000","last_change":"2024-01-15T18:03:03.569272+0000","last_active":"2024-01-15T18:03:03.572356+0000","last_peered":"2024-01-15T18:03:03.572356+0000","last_clean":"2024-01-15T18:03:03.572356+0000","last_became_active":"2024-01-15T18:03:03.564615+0000","last_became_peered":"2024-01-15T18:03:03.564615+0000","last_unstale":"2024-01-15T18:03:03.572356+0000","last_undegraded":"2024-01-15T18:03:03.572356+0000","last_fullsized":"2024-01-15T18:03:03.572356+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:08:10.493793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550644+0000","last_change":"2024-01-15T18:03:03.547527+0000","last_active":"2024-01-15T18:03:03.550644+0000","last_peered":"2024-01-15T18:03:03.550644+0000","last_clean":"2024-01-15T18:03:03.550644+0000","last_became_active":"2024-01-15T18:03:03.547216+0000","last_became_peered":"2024-01-15T18:03:03.547216+0000","last_unstale":"2024-01-15T18:03:03.550644+0000","last_undegraded":"2024-01-15T18:03:03.550644+0000","last_fullsized":"2024-01-15T18:03:03.550644+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:24:23.035552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562349+0000","last_change":"2024-01-15T18:03:03.561460+0000","last_active":"2024-01-15T18:03:04.562349+0000","last_peered":"2024-01-15T18:03:04.562349+0000","last_clean":"2024-01-15T18:03:04.562349+0000","last_became_active":"2024-01-15T18:03:03.561027+0000","last_became_peered":"2024-01-15T18:03:03.561027+0000","last_unstale":"2024-01-15T18:03:04.562349+0000","last_undegraded":"2024-01-15T18:03:04.562349+0000","last_fullsized":"2024-01-15T18:03:04.562349+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:22.576177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572258+0000","last_change":"2024-01-15T18:03:03.560223+0000","last_active":"2024-01-15T18:03:03.572258+0000","last_peered":"2024-01-15T18:03:03.572258+0000","last_clean":"2024-01-15T18:03:03.572258+0000","last_became_active":"2024-01-15T18:03:03.559717+0000","last_became_peered":"2024-01-15T18:03:03.559717+0000","last_unstale":"2024-01-15T18:03:03.572258+0000","last_undegraded":"2024-01-15T18:03:03.572258+0000","last_fullsized":"2024-01-15T18:03:03.572258+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:56:58.728010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.565550+0000","last_change":"2024-01-15T18:03:03.554478+0000","last_active":"2024-01-15T18:03:04.565550+0000","last_peered":"2024-01-15T18:03:04.565550+0000","last_clean":"2024-01-15T18:03:04.565550+0000","last_became_active":"2024-01-15T18:03:03.554176+0000","last_became_peered":"2024-01-15T18:03:03.554176+0000","last_unstale":"2024-01-15T18:03:04.565550+0000","last_undegraded":"2024-01-15T18:03:04.565550+0000","last_fullsized":"2024-01-15T18:03:04.565550+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:14.918999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.571746+0000","last_change":"2024-01-15T18:03:03.566156+0000","last_active":"2024-01-15T18:03:03.571746+0000","last_peered":"2024-01-15T18:03:03.571746+0000","last_clean":"2024-01-15T18:03:03.571746+0000","last_became_active":"2024-01-15T18:03:03.565907+0000","last_became_peered":"2024-01-15T18:03:03.565907+0000","last_unstale":"2024-01-15T18:03:03.571746+0000","last_undegraded":"2024-01-15T18:03:03.571746+0000","last_fullsized":"2024-01-15T18:03:03.571746+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:11.929487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.574306+0000","last_change":"2024-01-15T18:03:03.562279+0000","last_active":"2024-01-15T18:03:03.574306+0000","last_peered":"2024-01-15T18:03:03.574306+0000","last_clean":"2024-01-15T18:03:03.574306+0000","last_became_active":"2024-01-15T18:03:03.562008+0000","last_became_peered":"2024-01-15T18:03:03.562008+0000","last_unstale":"2024-01-15T18:03:03.574306+0000","last_undegraded":"2024-01-15T18:03:03.574306+0000","last_fullsized":"2024-01-15T18:03:03.574306+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:41.005801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578643+0000","last_change":"2024-01-15T18:03:03.575980+0000","last_active":"2024-01-15T18:03:03.578643+0000","last_peered":"2024-01-15T18:03:03.578643+0000","last_clean":"2024-01-15T18:03:03.578643+0000","last_became_active":"2024-01-15T18:03:03.575748+0000","last_became_peered":"2024-01-15T18:03:03.575748+0000","last_unstale":"2024-01-15T18:03:03.578643+0000","last_undegraded":"2024-01-15T18:03:03.578643+0000","last_fullsized":"2024-01-15T18:03:03.578643+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:24:23.035552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.583267+0000","last_change":"2024-01-15T18:03:03.580017+0000","last_active":"2024-01-15T18:03:03.583267+0000","last_peered":"2024-01-15T18:03:03.583267+0000","last_clean":"2024-01-15T18:03:03.583267+0000","last_became_active":"2024-01-15T18:03:03.579710+0000","last_became_peered":"2024-01-15T18:03:03.579710+0000","last_unstale":"2024-01-15T18:03:03.583267+0000","last_undegraded":"2024-01-15T18:03:03.583267+0000","last_fullsized":"2024-01-15T18:03:03.583267+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:16.679701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562467+0000","last_change":"2024-01-15T18:03:03.556480+0000","last_active":"2024-01-15T18:03:04.562467+0000","last_peered":"2024-01-15T18:03:04.562467+0000","last_clean":"2024-01-15T18:03:04.562467+0000","last_became_active":"2024-01-15T18:03:03.555437+0000","last_became_peered":"2024-01-15T18:03:03.555437+0000","last_unstale":"2024-01-15T18:03:04.562467+0000","last_undegraded":"2024-01-15T18:03:04.562467+0000","last_fullsized":"2024-01-15T18:03:04.562467+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:09.221734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562872+0000","last_change":"2024-01-15T18:03:03.563820+0000","last_active":"2024-01-15T18:03:04.562872+0000","last_peered":"2024-01-15T18:03:04.562872+0000","last_clean":"2024-01-15T18:03:04.562872+0000","last_became_active":"2024-01-15T18:03:03.563532+0000","last_became_peered":"2024-01-15T18:03:03.563532+0000","last_unstale":"2024-01-15T18:03:04.562872+0000","last_undegraded":"2024-01-15T18:03:04.562872+0000","last_fullsized":"2024-01-15T18:03:04.562872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:11:02.869083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580227+0000","last_change":"2024-01-15T18:03:03.570114+0000","last_active":"2024-01-15T18:03:03.580227+0000","last_peered":"2024-01-15T18:03:03.580227+0000","last_clean":"2024-01-15T18:03:03.580227+0000","last_became_active":"2024-01-15T18:03:03.569810+0000","last_became_peered":"2024-01-15T18:03:03.569810+0000","last_unstale":"2024-01-15T18:03:03.580227+0000","last_undegraded":"2024-01-15T18:03:03.580227+0000","last_fullsized":"2024-01-15T18:03:03.580227+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:49:34.021909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575113+0000","last_change":"2024-01-15T18:03:03.572668+0000","last_active":"2024-01-15T18:03:03.575113+0000","last_peered":"2024-01-15T18:03:03.575113+0000","last_clean":"2024-01-15T18:03:03.575113+0000","last_became_active":"2024-01-15T18:03:03.568800+0000","last_became_peered":"2024-01-15T18:03:03.568800+0000","last_unstale":"2024-01-15T18:03:03.575113+0000","last_undegraded":"2024-01-15T18:03:03.575113+0000","last_fullsized":"2024-01-15T18:03:03.575113+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:49:06.674946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581492+0000","last_change":"2024-01-15T18:03:03.575426+0000","last_active":"2024-01-15T18:03:03.581492+0000","last_peered":"2024-01-15T18:03:03.581492+0000","last_clean":"2024-01-15T18:03:03.581492+0000","last_became_active":"2024-01-15T18:03:03.575237+0000","last_became_peered":"2024-01-15T18:03:03.575237+0000","last_unstale":"2024-01-15T18:03:03.581492+0000","last_undegraded":"2024-01-15T18:03:03.581492+0000","last_fullsized":"2024-01-15T18:03:03.581492+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:38.275323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.562718+0000","last_change":"2024-01-15T18:03:03.558267+0000","last_active":"2024-01-15T18:03:03.562718+0000","last_peered":"2024-01-15T18:03:03.562718+0000","last_clean":"2024-01-15T18:03:03.562718+0000","last_became_active":"2024-01-15T18:03:03.557947+0000","last_became_peered":"2024-01-15T18:03:03.557947+0000","last_unstale":"2024-01-15T18:03:03.562718+0000","last_undegraded":"2024-01-15T18:03:03.562718+0000","last_fullsized":"2024-01-15T18:03:03.562718+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:24.128195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572616+0000","last_change":"2024-01-15T18:03:03.555727+0000","last_active":"2024-01-15T18:03:03.572616+0000","last_peered":"2024-01-15T18:03:03.572616+0000","last_clean":"2024-01-15T18:03:03.572616+0000","last_became_active":"2024-01-15T18:03:03.553483+0000","last_became_peered":"2024-01-15T18:03:03.553483+0000","last_unstale":"2024-01-15T18:03:03.572616+0000","last_undegraded":"2024-01-15T18:03:03.572616+0000","last_fullsized":"2024-01-15T18:03:03.572616+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:23.222355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581359+0000","last_change":"2024-01-15T18:03:03.573290+0000","last_active":"2024-01-15T18:03:03.581359+0000","last_peered":"2024-01-15T18:03:03.581359+0000","last_clean":"2024-01-15T18:03:03.581359+0000","last_became_active":"2024-01-15T18:03:03.572347+0000","last_became_peered":"2024-01-15T18:03:03.572347+0000","last_unstale":"2024-01-15T18:03:03.581359+0000","last_undegraded":"2024-01-15T18:03:03.581359+0000","last_fullsized":"2024-01-15T18:03:03.581359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.947434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573186+0000","last_change":"2024-01-15T18:03:03.565599+0000","last_active":"2024-01-15T18:03:03.573186+0000","last_peered":"2024-01-15T18:03:03.573186+0000","last_clean":"2024-01-15T18:03:03.573186+0000","last_became_active":"2024-01-15T18:03:03.565087+0000","last_became_peered":"2024-01-15T18:03:03.565087+0000","last_unstale":"2024-01-15T18:03:03.573186+0000","last_undegraded":"2024-01-15T18:03:03.573186+0000","last_fullsized":"2024-01-15T18:03:03.573186+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:20:26.100036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581806+0000","last_change":"2024-01-15T18:03:03.578999+0000","last_active":"2024-01-15T18:03:03.581806+0000","last_peered":"2024-01-15T18:03:03.581806+0000","last_clean":"2024-01-15T18:03:03.581806+0000","last_became_active":"2024-01-15T18:03:03.577347+0000","last_became_peered":"2024-01-15T18:03:03.577347+0000","last_unstale":"2024-01-15T18:03:03.581806+0000","last_undegraded":"2024-01-15T18:03:03.581806+0000","last_fullsized":"2024-01-15T18:03:03.581806+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:13.272751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.583255+0000","last_change":"2024-01-15T18:03:03.578538+0000","last_active":"2024-01-15T18:03:03.583255+0000","last_peered":"2024-01-15T18:03:03.583255+0000","last_clean":"2024-01-15T18:03:03.583255+0000","last_became_active":"2024-01-15T18:03:03.578110+0000","last_became_peered":"2024-01-15T18:03:03.578110+0000","last_unstale":"2024-01-15T18:03:03.583255+0000","last_undegraded":"2024-01-15T18:03:03.583255+0000","last_fullsized":"2024-01-15T18:03:03.583255+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:12.548993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572325+0000","last_change":"2024-01-15T18:03:03.568945+0000","last_active":"2024-01-15T18:03:03.572325+0000","last_peered":"2024-01-15T18:03:03.572325+0000","last_clean":"2024-01-15T18:03:03.572325+0000","last_became_active":"2024-01-15T18:03:03.563481+0000","last_became_peered":"2024-01-15T18:03:03.563481+0000","last_unstale":"2024-01-15T18:03:03.572325+0000","last_undegraded":"2024-01-15T18:03:03.572325+0000","last_fullsized":"2024-01-15T18:03:03.572325+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:10:37.019971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.562564+0000","last_change":"2024-01-15T18:03:03.559469+0000","last_active":"2024-01-15T18:03:03.562564+0000","last_peered":"2024-01-15T18:03:03.562564+0000","last_clean":"2024-01-15T18:03:03.562564+0000","last_became_active":"2024-01-15T18:03:03.559155+0000","last_became_peered":"2024-01-15T18:03:03.559155+0000","last_unstale":"2024-01-15T18:03:03.562564+0000","last_undegraded":"2024-01-15T18:03:03.562564+0000","last_fullsized":"2024-01-15T18:03:03.562564+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:51.286116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.582039+0000","last_change":"2024-01-15T18:03:03.571321+0000","last_active":"2024-01-15T18:03:03.582039+0000","last_peered":"2024-01-15T18:03:03.582039+0000","last_clean":"2024-01-15T18:03:03.582039+0000","last_became_active":"2024-01-15T18:03:03.570991+0000","last_became_peered":"2024-01-15T18:03:03.570991+0000","last_unstale":"2024-01-15T18:03:03.582039+0000","last_undegraded":"2024-01-15T18:03:03.582039+0000","last_fullsized":"2024-01-15T18:03:03.582039+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:47.816541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.569168+0000","last_change":"2024-01-15T18:03:03.566629+0000","last_active":"2024-01-15T18:03:03.569168+0000","last_peered":"2024-01-15T18:03:03.569168+0000","last_clean":"2024-01-15T18:03:03.569168+0000","last_became_active":"2024-01-15T18:03:03.566305+0000","last_became_peered":"2024-01-15T18:03:03.566305+0000","last_unstale":"2024-01-15T18:03:03.569168+0000","last_undegraded":"2024-01-15T18:03:03.569168+0000","last_fullsized":"2024-01-15T18:03:03.569168+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:51:15.787221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573301+0000","last_change":"2024-01-15T18:03:03.571334+0000","last_active":"2024-01-15T18:03:03.573301+0000","last_peered":"2024-01-15T18:03:03.573301+0000","last_clean":"2024-01-15T18:03:03.573301+0000","last_became_active":"2024-01-15T18:03:03.570330+0000","last_became_peered":"2024-01-15T18:03:03.570330+0000","last_unstale":"2024-01-15T18:03:03.573301+0000","last_undegraded":"2024-01-15T18:03:03.573301+0000","last_fullsized":"2024-01-15T18:03:03.573301+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:13.175551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573825+0000","last_change":"2024-01-15T18:03:03.568407+0000","last_active":"2024-01-15T18:03:03.573825+0000","last_peered":"2024-01-15T18:03:03.573825+0000","last_clean":"2024-01-15T18:03:03.573825+0000","last_became_active":"2024-01-15T18:03:03.567608+0000","last_became_peered":"2024-01-15T18:03:03.567608+0000","last_unstale":"2024-01-15T18:03:03.573825+0000","last_undegraded":"2024-01-15T18:03:03.573825+0000","last_fullsized":"2024-01-15T18:03:03.573825+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:24:23.035552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.569734+0000","last_change":"2024-01-15T18:03:03.568365+0000","last_active":"2024-01-15T18:03:03.569734+0000","last_peered":"2024-01-15T18:03:03.569734+0000","last_clean":"2024-01-15T18:03:03.569734+0000","last_became_active":"2024-01-15T18:03:03.563420+0000","last_became_peered":"2024-01-15T18:03:03.563420+0000","last_unstale":"2024-01-15T18:03:03.569734+0000","last_undegraded":"2024-01-15T18:03:03.569734+0000","last_fullsized":"2024-01-15T18:03:03.569734+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:54.246039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578720+0000","last_change":"2024-01-15T18:03:03.558405+0000","last_active":"2024-01-15T18:03:03.578720+0000","last_peered":"2024-01-15T18:03:03.578720+0000","last_clean":"2024-01-15T18:03:03.578720+0000","last_became_active":"2024-01-15T18:03:03.558094+0000","last_became_peered":"2024-01-15T18:03:03.558094+0000","last_unstale":"2024-01-15T18:03:03.578720+0000","last_undegraded":"2024-01-15T18:03:03.578720+0000","last_fullsized":"2024-01-15T18:03:03.578720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:52:26.067882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581673+0000","last_change":"2024-01-15T18:03:03.571621+0000","last_active":"2024-01-15T18:03:03.581673+0000","last_peered":"2024-01-15T18:03:03.581673+0000","last_clean":"2024-01-15T18:03:03.581673+0000","last_became_active":"2024-01-15T18:03:03.571353+0000","last_became_peered":"2024-01-15T18:03:03.571353+0000","last_unstale":"2024-01-15T18:03:03.581673+0000","last_undegraded":"2024-01-15T18:03:03.581673+0000","last_fullsized":"2024-01-15T18:03:03.581673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:13.272751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"60'4","reported_seq":15,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580382+0000","last_change":"2024-01-15T18:03:02.546211+0000","last_active":"2024-01-15T18:03:03.580382+0000","last_peered":"2024-01-15T18:03:03.580382+0000","last_clean":"2024-01-15T18:03:03.580382+0000","last_became_active":"2024-01-15T18:03:02.545986+0000","last_became_peered":"2024-01-15T18:03:02.545986+0000","last_unstale":"2024-01-15T18:03:03.580382+0000","last_undegraded":"2024-01-15T18:03:03.580382+0000","last_fullsized":"2024-01-15T18:03:03.580382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:15:49.720738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575291+0000","last_change":"2024-01-15T18:03:03.573544+0000","last_active":"2024-01-15T18:03:03.575291+0000","last_peered":"2024-01-15T18:03:03.575291+0000","last_clean":"2024-01-15T18:03:03.575291+0000","last_became_active":"2024-01-15T18:03:03.573282+0000","last_became_peered":"2024-01-15T18:03:03.573282+0000","last_unstale":"2024-01-15T18:03:03.575291+0000","last_undegraded":"2024-01-15T18:03:03.575291+0000","last_fullsized":"2024-01-15T18:03:03.575291+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:33.954665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.576416+0000","last_change":"2024-01-15T18:03:02.534875+0000","last_active":"2024-01-15T18:03:03.576416+0000","last_peered":"2024-01-15T18:03:03.576416+0000","last_clean":"2024-01-15T18:03:03.576416+0000","last_became_active":"2024-01-15T18:03:02.534561+0000","last_became_peered":"2024-01-15T18:03:02.534561+0000","last_unstale":"2024-01-15T18:03:03.576416+0000","last_undegraded":"2024-01-15T18:03:03.576416+0000","last_fullsized":"2024-01-15T18:03:03.576416+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:10.880631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579828+0000","last_change":"2024-01-15T18:03:03.561303+0000","last_active":"2024-01-15T18:03:03.579828+0000","last_peered":"2024-01-15T18:03:03.579828+0000","last_clean":"2024-01-15T18:03:03.579828+0000","last_became_active":"2024-01-15T18:03:03.560997+0000","last_became_peered":"2024-01-15T18:03:03.560997+0000","last_unstale":"2024-01-15T18:03:03.579828+0000","last_undegraded":"2024-01-15T18:03:03.579828+0000","last_fullsized":"2024-01-15T18:03:03.579828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:22.070448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579711+0000","last_change":"2024-01-15T18:03:02.541272+0000","last_active":"2024-01-15T18:03:03.579711+0000","last_peered":"2024-01-15T18:03:03.579711+0000","last_clean":"2024-01-15T18:03:03.579711+0000","last_became_active":"2024-01-15T18:03:02.541029+0000","last_became_peered":"2024-01-15T18:03:02.541029+0000","last_unstale":"2024-01-15T18:03:03.579711+0000","last_undegraded":"2024-01-15T18:03:03.579711+0000","last_fullsized":"2024-01-15T18:03:03.579711+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:02.371466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579941+0000","last_change":"2024-01-15T18:03:03.563676+0000","last_active":"2024-01-15T18:03:03.579941+0000","last_peered":"2024-01-15T18:03:03.579941+0000","last_clean":"2024-01-15T18:03:03.579941+0000","last_became_active":"2024-01-15T18:03:03.563223+0000","last_became_peered":"2024-01-15T18:03:03.563223+0000","last_unstale":"2024-01-15T18:03:03.579941+0000","last_undegraded":"2024-01-15T18:03:03.579941+0000","last_fullsized":"2024-01-15T18:03:03.579941+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:34.769159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581443+0000","last_change":"2024-01-15T18:03:02.548803+0000","last_active":"2024-01-15T18:03:03.581443+0000","last_peered":"2024-01-15T18:03:03.581443+0000","last_clean":"2024-01-15T18:03:03.581443+0000","last_became_active":"2024-01-15T18:03:02.547942+0000","last_became_peered":"2024-01-15T18:03:02.547942+0000","last_unstale":"2024-01-15T18:03:03.581443+0000","last_undegraded":"2024-01-15T18:03:03.581443+0000","last_fullsized":"2024-01-15T18:03:03.581443+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:04:40.709592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.573165+0000","last_change":"2024-01-15T18:03:03.565344+0000","last_active":"2024-01-15T18:03:03.573165+0000","last_peered":"2024-01-15T18:03:03.573165+0000","last_clean":"2024-01-15T18:03:03.573165+0000","last_became_active":"2024-01-15T18:03:03.564730+0000","last_became_peered":"2024-01-15T18:03:03.564730+0000","last_unstale":"2024-01-15T18:03:03.573165+0000","last_undegraded":"2024-01-15T18:03:03.573165+0000","last_fullsized":"2024-01-15T18:03:03.573165+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:31:28.167754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"60'1","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.564808+0000","last_change":"2024-01-15T18:03:02.537087+0000","last_active":"2024-01-15T18:03:04.564808+0000","last_peered":"2024-01-15T18:03:04.564808+0000","last_clean":"2024-01-15T18:03:04.564808+0000","last_became_active":"2024-01-15T18:03:02.536776+0000","last_became_peered":"2024-01-15T18:03:02.536776+0000","last_unstale":"2024-01-15T18:03:04.564808+0000","last_undegraded":"2024-01-15T18:03:04.564808+0000","last_fullsized":"2024-01-15T18:03:04.564808+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:22:36.038520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.557849+0000","last_change":"2024-01-15T18:03:03.553987+0000","last_active":"2024-01-15T18:03:03.557849+0000","last_peered":"2024-01-15T18:03:03.557849+0000","last_clean":"2024-01-15T18:03:03.557849+0000","last_became_active":"2024-01-15T18:03:03.552722+0000","last_became_peered":"2024-01-15T18:03:03.552722+0000","last_unstale":"2024-01-15T18:03:03.557849+0000","last_undegraded":"2024-01-15T18:03:03.557849+0000","last_fullsized":"2024-01-15T18:03:03.557849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:34:46.885448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579378+0000","last_change":"2024-01-15T18:03:02.539551+0000","last_active":"2024-01-15T18:03:03.579378+0000","last_peered":"2024-01-15T18:03:03.579378+0000","last_clean":"2024-01-15T18:03:03.579378+0000","last_became_active":"2024-01-15T18:03:02.539260+0000","last_became_peered":"2024-01-15T18:03:02.539260+0000","last_unstale":"2024-01-15T18:03:03.579378+0000","last_undegraded":"2024-01-15T18:03:03.579378+0000","last_fullsized":"2024-01-15T18:03:03.579378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:46.977934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580677+0000","last_change":"2024-01-15T18:03:03.575030+0000","last_active":"2024-01-15T18:03:03.580677+0000","last_peered":"2024-01-15T18:03:03.580677+0000","last_clean":"2024-01-15T18:03:03.580677+0000","last_became_active":"2024-01-15T18:03:03.574283+0000","last_became_peered":"2024-01-15T18:03:03.574283+0000","last_unstale":"2024-01-15T18:03:03.580677+0000","last_undegraded":"2024-01-15T18:03:03.580677+0000","last_fullsized":"2024-01-15T18:03:03.580677+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:19.009008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550759+0000","last_change":"2024-01-15T18:03:02.545792+0000","last_active":"2024-01-15T18:03:03.550759+0000","last_peered":"2024-01-15T18:03:03.550759+0000","last_clean":"2024-01-15T18:03:03.550759+0000","last_became_active":"2024-01-15T18:03:02.545242+0000","last_became_peered":"2024-01-15T18:03:02.545242+0000","last_unstale":"2024-01-15T18:03:03.550759+0000","last_undegraded":"2024-01-15T18:03:03.550759+0000","last_fullsized":"2024-01-15T18:03:03.550759+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:02.371466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575658+0000","last_change":"2024-01-15T18:03:03.573048+0000","last_active":"2024-01-15T18:03:03.575658+0000","last_peered":"2024-01-15T18:03:03.575658+0000","last_clean":"2024-01-15T18:03:03.575658+0000","last_became_active":"2024-01-15T18:03:03.568637+0000","last_became_peered":"2024-01-15T18:03:03.568637+0000","last_unstale":"2024-01-15T18:03:03.575658+0000","last_undegraded":"2024-01-15T18:03:03.575658+0000","last_fullsized":"2024-01-15T18:03:03.575658+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:52:04.192561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.563305+0000","last_change":"2024-01-15T18:03:02.538617+0000","last_active":"2024-01-15T18:03:04.563305+0000","last_peered":"2024-01-15T18:03:04.563305+0000","last_clean":"2024-01-15T18:03:04.563305+0000","last_became_active":"2024-01-15T18:03:02.538293+0000","last_became_peered":"2024-01-15T18:03:02.538293+0000","last_unstale":"2024-01-15T18:03:04.563305+0000","last_undegraded":"2024-01-15T18:03:04.563305+0000","last_fullsized":"2024-01-15T18:03:04.563305+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:22:46.832961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.565397+0000","last_change":"2024-01-15T18:03:03.563735+0000","last_active":"2024-01-15T18:03:04.565397+0000","last_peered":"2024-01-15T18:03:04.565397+0000","last_clean":"2024-01-15T18:03:04.565397+0000","last_became_active":"2024-01-15T18:03:03.563450+0000","last_became_peered":"2024-01-15T18:03:03.563450+0000","last_unstale":"2024-01-15T18:03:04.565397+0000","last_undegraded":"2024-01-15T18:03:04.565397+0000","last_fullsized":"2024-01-15T18:03:04.565397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:04.905534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.555343+0000","last_change":"2024-01-15T18:03:02.544434+0000","last_active":"2024-01-15T18:03:03.555343+0000","last_peered":"2024-01-15T18:03:03.555343+0000","last_clean":"2024-01-15T18:03:03.555343+0000","last_became_active":"2024-01-15T18:03:02.543980+0000","last_became_peered":"2024-01-15T18:03:02.543980+0000","last_unstale":"2024-01-15T18:03:03.555343+0000","last_undegraded":"2024-01-15T18:03:03.555343+0000","last_fullsized":"2024-01-15T18:03:03.555343+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:33.755676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581828+0000","last_change":"2024-01-15T18:03:03.577261+0000","last_active":"2024-01-15T18:03:03.581828+0000","last_peered":"2024-01-15T18:03:03.581828+0000","last_clean":"2024-01-15T18:03:03.581828+0000","last_became_active":"2024-01-15T18:03:03.576662+0000","last_became_peered":"2024-01-15T18:03:03.576662+0000","last_unstale":"2024-01-15T18:03:03.581828+0000","last_undegraded":"2024-01-15T18:03:03.581828+0000","last_fullsized":"2024-01-15T18:03:03.581828+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:03.384948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.542911+0000","last_change":"2024-01-15T18:03:02.531700+0000","last_active":"2024-01-15T18:03:03.542911+0000","last_peered":"2024-01-15T18:03:03.542911+0000","last_clean":"2024-01-15T18:03:03.542911+0000","last_became_active":"2024-01-15T18:03:02.531004+0000","last_became_peered":"2024-01-15T18:03:02.531004+0000","last_unstale":"2024-01-15T18:03:03.542911+0000","last_undegraded":"2024-01-15T18:03:03.542911+0000","last_fullsized":"2024-01-15T18:03:03.542911+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:01:24.526602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572095+0000","last_change":"2024-01-15T18:03:03.564631+0000","last_active":"2024-01-15T18:03:03.572095+0000","last_peered":"2024-01-15T18:03:03.572095+0000","last_clean":"2024-01-15T18:03:03.572095+0000","last_became_active":"2024-01-15T18:03:03.564186+0000","last_became_peered":"2024-01-15T18:03:03.564186+0000","last_unstale":"2024-01-15T18:03:03.572095+0000","last_undegraded":"2024-01-15T18:03:03.572095+0000","last_fullsized":"2024-01-15T18:03:03.572095+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:40.840335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"60'3","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.580774+0000","last_change":"2024-01-15T18:03:02.543881+0000","last_active":"2024-01-15T18:03:03.580774+0000","last_peered":"2024-01-15T18:03:03.580774+0000","last_clean":"2024-01-15T18:03:03.580774+0000","last_became_active":"2024-01-15T18:03:02.543581+0000","last_became_peered":"2024-01-15T18:03:02.543581+0000","last_unstale":"2024-01-15T18:03:03.580774+0000","last_undegraded":"2024-01-15T18:03:03.580774+0000","last_fullsized":"2024-01-15T18:03:03.580774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:21.056965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581727+0000","last_change":"2024-01-15T18:03:03.576760+0000","last_active":"2024-01-15T18:03:03.581727+0000","last_peered":"2024-01-15T18:03:03.581727+0000","last_clean":"2024-01-15T18:03:03.581727+0000","last_became_active":"2024-01-15T18:03:03.576184+0000","last_became_peered":"2024-01-15T18:03:03.576184+0000","last_unstale":"2024-01-15T18:03:03.581727+0000","last_undegraded":"2024-01-15T18:03:03.581727+0000","last_fullsized":"2024-01-15T18:03:03.581727+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:51.286116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"60'6","reported_seq":18,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.565425+0000","last_change":"2024-01-15T18:03:02.541543+0000","last_active":"2024-01-15T18:03:04.565425+0000","last_peered":"2024-01-15T18:03:04.565425+0000","last_clean":"2024-01-15T18:03:04.565425+0000","last_became_active":"2024-01-15T18:03:02.541266+0000","last_became_peered":"2024-01-15T18:03:02.541266+0000","last_unstale":"2024-01-15T18:03:04.565425+0000","last_undegraded":"2024-01-15T18:03:04.565425+0000","last_fullsized":"2024-01-15T18:03:04.565425+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:46.401878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.567798+0000","last_change":"2024-01-15T18:03:03.566237+0000","last_active":"2024-01-15T18:03:03.567798+0000","last_peered":"2024-01-15T18:03:03.567798+0000","last_clean":"2024-01-15T18:03:03.567798+0000","last_became_active":"2024-01-15T18:03:03.565767+0000","last_became_peered":"2024-01-15T18:03:03.565767+0000","last_unstale":"2024-01-15T18:03:03.567798+0000","last_undegraded":"2024-01-15T18:03:03.567798+0000","last_fullsized":"2024-01-15T18:03:03.567798+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:47.991417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575141+0000","last_change":"2024-01-15T18:03:02.532962+0000","last_active":"2024-01-15T18:03:03.575141+0000","last_peered":"2024-01-15T18:03:03.575141+0000","last_clean":"2024-01-15T18:03:03.575141+0000","last_became_active":"2024-01-15T18:03:02.530931+0000","last_became_peered":"2024-01-15T18:03:02.530931+0000","last_unstale":"2024-01-15T18:03:03.575141+0000","last_undegraded":"2024-01-15T18:03:03.575141+0000","last_fullsized":"2024-01-15T18:03:03.575141+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:42:00.049457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581382+0000","last_change":"2024-01-15T18:03:03.573574+0000","last_active":"2024-01-15T18:03:03.581382+0000","last_peered":"2024-01-15T18:03:03.581382+0000","last_clean":"2024-01-15T18:03:03.581382+0000","last_became_active":"2024-01-15T18:03:03.572751+0000","last_became_peered":"2024-01-15T18:03:03.572751+0000","last_unstale":"2024-01-15T18:03:03.581382+0000","last_undegraded":"2024-01-15T18:03:03.581382+0000","last_fullsized":"2024-01-15T18:03:03.581382+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:56.271555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.562539+0000","last_change":"2024-01-15T18:03:02.547172+0000","last_active":"2024-01-15T18:03:03.562539+0000","last_peered":"2024-01-15T18:03:03.562539+0000","last_clean":"2024-01-15T18:03:03.562539+0000","last_became_active":"2024-01-15T18:03:02.546843+0000","last_became_peered":"2024-01-15T18:03:02.546843+0000","last_unstale":"2024-01-15T18:03:03.562539+0000","last_undegraded":"2024-01-15T18:03:03.562539+0000","last_fullsized":"2024-01-15T18:03:03.562539+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:28.593520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.562334+0000","last_change":"2024-01-15T18:03:03.574733+0000","last_active":"2024-01-15T18:03:04.562334+0000","last_peered":"2024-01-15T18:03:04.562334+0000","last_clean":"2024-01-15T18:03:04.562334+0000","last_became_active":"2024-01-15T18:03:03.573270+0000","last_became_peered":"2024-01-15T18:03:03.573270+0000","last_unstale":"2024-01-15T18:03:04.562334+0000","last_undegraded":"2024-01-15T18:03:04.562334+0000","last_fullsized":"2024-01-15T18:03:04.562334+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:29:02.501394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550900+0000","last_change":"2024-01-15T18:03:02.547792+0000","last_active":"2024-01-15T18:03:03.550900+0000","last_peered":"2024-01-15T18:03:03.550900+0000","last_clean":"2024-01-15T18:03:03.550900+0000","last_became_active":"2024-01-15T18:03:02.545716+0000","last_became_peered":"2024-01-15T18:03:02.545716+0000","last_unstale":"2024-01-15T18:03:03.550900+0000","last_undegraded":"2024-01-15T18:03:03.550900+0000","last_fullsized":"2024-01-15T18:03:03.550900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:33.755676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.583477+0000","last_change":"2024-01-15T18:03:03.580549+0000","last_active":"2024-01-15T18:03:03.583477+0000","last_peered":"2024-01-15T18:03:03.583477+0000","last_clean":"2024-01-15T18:03:03.583477+0000","last_became_active":"2024-01-15T18:03:03.580261+0000","last_became_peered":"2024-01-15T18:03:03.580261+0000","last_unstale":"2024-01-15T18:03:03.583477+0000","last_undegraded":"2024-01-15T18:03:03.583477+0000","last_fullsized":"2024-01-15T18:03:03.583477+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:44:05.698832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"60'8","reported_seq":19,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581627+0000","last_change":"2024-01-15T18:03:02.549962+0000","last_active":"2024-01-15T18:03:03.581627+0000","last_peered":"2024-01-15T18:03:03.581627+0000","last_clean":"2024-01-15T18:03:03.581627+0000","last_became_active":"2024-01-15T18:03:02.549571+0000","last_became_peered":"2024-01-15T18:03:02.549571+0000","last_unstale":"2024-01-15T18:03:03.581627+0000","last_undegraded":"2024-01-15T18:03:03.581627+0000","last_fullsized":"2024-01-15T18:03:03.581627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:58.440091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581717+0000","last_change":"2024-01-15T18:03:03.576245+0000","last_active":"2024-01-15T18:03:03.581717+0000","last_peered":"2024-01-15T18:03:03.581717+0000","last_clean":"2024-01-15T18:03:03.581717+0000","last_became_active":"2024-01-15T18:03:03.573742+0000","last_became_peered":"2024-01-15T18:03:03.573742+0000","last_unstale":"2024-01-15T18:03:03.581717+0000","last_undegraded":"2024-01-15T18:03:03.581717+0000","last_fullsized":"2024-01-15T18:03:03.581717+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:15:50.734220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575043+0000","last_change":"2024-01-15T18:03:02.533913+0000","last_active":"2024-01-15T18:03:03.575043+0000","last_peered":"2024-01-15T18:03:03.575043+0000","last_clean":"2024-01-15T18:03:03.575043+0000","last_became_active":"2024-01-15T18:03:02.533631+0000","last_became_peered":"2024-01-15T18:03:02.533631+0000","last_unstale":"2024-01-15T18:03:03.575043+0000","last_undegraded":"2024-01-15T18:03:03.575043+0000","last_fullsized":"2024-01-15T18:03:03.575043+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:35:37.181184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.574892+0000","last_change":"2024-01-15T18:03:03.570140+0000","last_active":"2024-01-15T18:03:03.574892+0000","last_peered":"2024-01-15T18:03:03.574892+0000","last_clean":"2024-01-15T18:03:03.574892+0000","last_became_active":"2024-01-15T18:03:03.569749+0000","last_became_peered":"2024-01-15T18:03:03.569749+0000","last_unstale":"2024-01-15T18:03:03.574892+0000","last_undegraded":"2024-01-15T18:03:03.574892+0000","last_fullsized":"2024-01-15T18:03:03.574892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:38.275323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579915+0000","last_change":"2024-01-15T18:03:02.530979+0000","last_active":"2024-01-15T18:03:03.579915+0000","last_peered":"2024-01-15T18:03:03.579915+0000","last_clean":"2024-01-15T18:03:03.579915+0000","last_became_active":"2024-01-15T18:03:02.530656+0000","last_became_peered":"2024-01-15T18:03:02.530656+0000","last_unstale":"2024-01-15T18:03:03.579915+0000","last_undegraded":"2024-01-15T18:03:03.579915+0000","last_fullsized":"2024-01-15T18:03:03.579915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:55.258072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572639+0000","last_change":"2024-01-15T18:03:03.563213+0000","last_active":"2024-01-15T18:03:03.572639+0000","last_peered":"2024-01-15T18:03:03.572639+0000","last_clean":"2024-01-15T18:03:03.572639+0000","last_became_active":"2024-01-15T18:03:03.562880+0000","last_became_peered":"2024-01-15T18:03:03.562880+0000","last_unstale":"2024-01-15T18:03:03.572639+0000","last_undegraded":"2024-01-15T18:03:03.572639+0000","last_fullsized":"2024-01-15T18:03:03.572639+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:22.205672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"60'1","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.577084+0000","last_change":"2024-01-15T18:03:02.548504+0000","last_active":"2024-01-15T18:03:03.577084+0000","last_peered":"2024-01-15T18:03:03.577084+0000","last_clean":"2024-01-15T18:03:03.577084+0000","last_became_active":"2024-01-15T18:03:02.548153+0000","last_became_peered":"2024-01-15T18:03:02.548153+0000","last_unstale":"2024-01-15T18:03:03.577084+0000","last_undegraded":"2024-01-15T18:03:03.577084+0000","last_fullsized":"2024-01-15T18:03:03.577084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:44:04.685350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581661+0000","last_change":"2024-01-15T18:03:03.576221+0000","last_active":"2024-01-15T18:03:03.581661+0000","last_peered":"2024-01-15T18:03:03.581661+0000","last_clean":"2024-01-15T18:03:03.581661+0000","last_became_active":"2024-01-15T18:03:03.575870+0000","last_became_peered":"2024-01-15T18:03:03.575870+0000","last_unstale":"2024-01-15T18:03:03.581661+0000","last_undegraded":"2024-01-15T18:03:03.581661+0000","last_fullsized":"2024-01-15T18:03:03.581661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:47.991417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.535336+0000","last_change":"2024-01-15T18:03:02.531412+0000","last_active":"2024-01-15T18:03:03.535336+0000","last_peered":"2024-01-15T18:03:03.535336+0000","last_clean":"2024-01-15T18:03:03.535336+0000","last_became_active":"2024-01-15T18:03:02.530878+0000","last_became_peered":"2024-01-15T18:03:02.530878+0000","last_unstale":"2024-01-15T18:03:03.535336+0000","last_undegraded":"2024-01-15T18:03:03.535336+0000","last_fullsized":"2024-01-15T18:03:03.535336+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:05:37.261840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.564699+0000","last_change":"2024-01-15T18:03:03.559879+0000","last_active":"2024-01-15T18:03:03.564699+0000","last_peered":"2024-01-15T18:03:03.564699+0000","last_clean":"2024-01-15T18:03:03.564699+0000","last_became_active":"2024-01-15T18:03:03.559149+0000","last_became_peered":"2024-01-15T18:03:03.559149+0000","last_unstale":"2024-01-15T18:03:03.564699+0000","last_undegraded":"2024-01-15T18:03:03.564699+0000","last_fullsized":"2024-01-15T18:03:03.564699+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.947434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.575808+0000","last_change":"2024-01-15T18:03:02.536266+0000","last_active":"2024-01-15T18:03:03.575808+0000","last_peered":"2024-01-15T18:03:03.575808+0000","last_clean":"2024-01-15T18:03:03.575808+0000","last_became_active":"2024-01-15T18:03:02.535944+0000","last_became_peered":"2024-01-15T18:03:02.535944+0000","last_unstale":"2024-01-15T18:03:03.575808+0000","last_undegraded":"2024-01-15T18:03:03.575808+0000","last_fullsized":"2024-01-15T18:03:03.575808+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:06:00.779130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.582236+0000","last_change":"2024-01-15T18:03:03.570664+0000","last_active":"2024-01-15T18:03:03.582236+0000","last_peered":"2024-01-15T18:03:03.582236+0000","last_clean":"2024-01-15T18:03:03.582236+0000","last_became_active":"2024-01-15T18:03:03.570381+0000","last_became_peered":"2024-01-15T18:03:03.570381+0000","last_unstale":"2024-01-15T18:03:03.582236+0000","last_undegraded":"2024-01-15T18:03:03.582236+0000","last_fullsized":"2024-01-15T18:03:03.582236+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:07:59.453574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.552498+0000","last_change":"2024-01-15T18:03:02.529702+0000","last_active":"2024-01-15T18:03:03.552498+0000","last_peered":"2024-01-15T18:03:03.552498+0000","last_clean":"2024-01-15T18:03:03.552498+0000","last_became_active":"2024-01-15T18:03:02.529384+0000","last_became_peered":"2024-01-15T18:03:02.529384+0000","last_unstale":"2024-01-15T18:03:03.552498+0000","last_undegraded":"2024-01-15T18:03:03.552498+0000","last_fullsized":"2024-01-15T18:03:03.552498+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:17.995525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.582008+0000","last_change":"2024-01-15T18:03:03.574453+0000","last_active":"2024-01-15T18:03:03.582008+0000","last_peered":"2024-01-15T18:03:03.582008+0000","last_clean":"2024-01-15T18:03:03.582008+0000","last_became_active":"2024-01-15T18:03:03.574068+0000","last_became_peered":"2024-01-15T18:03:03.574068+0000","last_unstale":"2024-01-15T18:03:03.582008+0000","last_undegraded":"2024-01-15T18:03:03.582008+0000","last_fullsized":"2024-01-15T18:03:03.582008+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:51.286116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.552667+0000","last_change":"2024-01-15T18:03:02.546511+0000","last_active":"2024-01-15T18:03:03.552667+0000","last_peered":"2024-01-15T18:03:03.552667+0000","last_clean":"2024-01-15T18:03:03.552667+0000","last_became_active":"2024-01-15T18:03:02.546101+0000","last_became_peered":"2024-01-15T18:03:02.546101+0000","last_unstale":"2024-01-15T18:03:03.552667+0000","last_undegraded":"2024-01-15T18:03:03.552667+0000","last_fullsized":"2024-01-15T18:03:03.552667+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:55.258072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.567262+0000","last_change":"2024-01-15T18:03:03.563991+0000","last_active":"2024-01-15T18:03:03.567262+0000","last_peered":"2024-01-15T18:03:03.567262+0000","last_clean":"2024-01-15T18:03:03.567262+0000","last_became_active":"2024-01-15T18:03:03.562781+0000","last_became_peered":"2024-01-15T18:03:03.562781+0000","last_unstale":"2024-01-15T18:03:03.567262+0000","last_undegraded":"2024-01-15T18:03:03.567262+0000","last_fullsized":"2024-01-15T18:03:03.567262+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:25:22.070448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"60'2","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578532+0000","last_change":"2024-01-15T18:03:02.532628+0000","last_active":"2024-01-15T18:03:03.578532+0000","last_peered":"2024-01-15T18:03:03.578532+0000","last_clean":"2024-01-15T18:03:03.578532+0000","last_became_active":"2024-01-15T18:03:02.532254+0000","last_became_peered":"2024-01-15T18:03:02.532254+0000","last_unstale":"2024-01-15T18:03:03.578532+0000","last_undegraded":"2024-01-15T18:03:03.578532+0000","last_fullsized":"2024-01-15T18:03:03.578532+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:04:11.535510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.574872+0000","last_change":"2024-01-15T18:03:03.569997+0000","last_active":"2024-01-15T18:03:03.574872+0000","last_peered":"2024-01-15T18:03:03.574872+0000","last_clean":"2024-01-15T18:03:03.574872+0000","last_became_active":"2024-01-15T18:03:03.569125+0000","last_became_peered":"2024-01-15T18:03:03.569125+0000","last_unstale":"2024-01-15T18:03:03.574872+0000","last_undegraded":"2024-01-15T18:03:03.574872+0000","last_fullsized":"2024-01-15T18:03:03.574872+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:15:50.734220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.537561+0000","last_change":"2024-01-15T18:03:02.544503+0000","last_active":"2024-01-15T18:03:03.537561+0000","last_peered":"2024-01-15T18:03:03.537561+0000","last_clean":"2024-01-15T18:03:03.537561+0000","last_became_active":"2024-01-15T18:03:02.544227+0000","last_became_peered":"2024-01-15T18:03:02.544227+0000","last_unstale":"2024-01-15T18:03:03.537561+0000","last_undegraded":"2024-01-15T18:03:03.537561+0000","last_fullsized":"2024-01-15T18:03:03.537561+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:04:40.709592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572073+0000","last_change":"2024-01-15T18:03:03.564455+0000","last_active":"2024-01-15T18:03:03.572073+0000","last_peered":"2024-01-15T18:03:03.572073+0000","last_clean":"2024-01-15T18:03:03.572073+0000","last_became_active":"2024-01-15T18:03:03.563837+0000","last_became_peered":"2024-01-15T18:03:03.563837+0000","last_unstale":"2024-01-15T18:03:03.572073+0000","last_undegraded":"2024-01-15T18:03:03.572073+0000","last_fullsized":"2024-01-15T18:03:03.572073+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:54.826330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561586+0000","last_change":"2024-01-15T18:03:02.543415+0000","last_active":"2024-01-15T18:03:03.561586+0000","last_peered":"2024-01-15T18:03:03.561586+0000","last_clean":"2024-01-15T18:03:03.561586+0000","last_became_active":"2024-01-15T18:03:02.543113+0000","last_became_peered":"2024-01-15T18:03:02.543113+0000","last_unstale":"2024-01-15T18:03:03.561586+0000","last_undegraded":"2024-01-15T18:03:03.561586+0000","last_fullsized":"2024-01-15T18:03:03.561586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:58:47.609801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.579921+0000","last_change":"2024-01-15T18:03:03.563481+0000","last_active":"2024-01-15T18:03:03.579921+0000","last_peered":"2024-01-15T18:03:03.579921+0000","last_clean":"2024-01-15T18:03:03.579921+0000","last_became_active":"2024-01-15T18:03:03.562880+0000","last_became_peered":"2024-01-15T18:03:03.562880+0000","last_unstale":"2024-01-15T18:03:03.579921+0000","last_undegraded":"2024-01-15T18:03:03.579921+0000","last_fullsized":"2024-01-15T18:03:03.579921+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:03.947434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.555786+0000","last_change":"2024-01-15T18:03:02.549187+0000","last_active":"2024-01-15T18:03:03.555786+0000","last_peered":"2024-01-15T18:03:03.555786+0000","last_clean":"2024-01-15T18:03:03.555786+0000","last_became_active":"2024-01-15T18:03:02.548364+0000","last_became_peered":"2024-01-15T18:03:02.548364+0000","last_unstale":"2024-01-15T18:03:03.555786+0000","last_undegraded":"2024-01-15T18:03:03.555786+0000","last_fullsized":"2024-01-15T18:03:03.555786+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:12.259268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.571818+0000","last_change":"2024-01-15T18:03:03.555932+0000","last_active":"2024-01-15T18:03:03.571818+0000","last_peered":"2024-01-15T18:03:03.571818+0000","last_clean":"2024-01-15T18:03:03.571818+0000","last_became_active":"2024-01-15T18:03:03.555615+0000","last_became_peered":"2024-01-15T18:03:03.555615+0000","last_unstale":"2024-01-15T18:03:03.571818+0000","last_undegraded":"2024-01-15T18:03:03.571818+0000","last_fullsized":"2024-01-15T18:03:03.571818+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:35.978962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.552248+0000","last_change":"2024-01-15T18:03:02.543613+0000","last_active":"2024-01-15T18:03:03.552248+0000","last_peered":"2024-01-15T18:03:03.552248+0000","last_clean":"2024-01-15T18:03:03.552248+0000","last_became_active":"2024-01-15T18:03:02.543206+0000","last_became_peered":"2024-01-15T18:03:02.543206+0000","last_unstale":"2024-01-15T18:03:03.552248+0000","last_undegraded":"2024-01-15T18:03:03.552248+0000","last_fullsized":"2024-01-15T18:03:03.552248+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:47:35.101403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.581574+0000","last_change":"2024-01-15T18:03:03.576012+0000","last_active":"2024-01-15T18:03:03.581574+0000","last_peered":"2024-01-15T18:03:03.581574+0000","last_clean":"2024-01-15T18:03:03.581574+0000","last_became_active":"2024-01-15T18:03:03.573946+0000","last_became_peered":"2024-01-15T18:03:03.573946+0000","last_unstale":"2024-01-15T18:03:03.581574+0000","last_undegraded":"2024-01-15T18:03:03.581574+0000","last_fullsized":"2024-01-15T18:03:03.581574+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:04:41.723075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-01-15T18:03:04.564626+0000","last_change":"2024-01-15T18:03:02.534835+0000","last_active":"2024-01-15T18:03:04.564626+0000","last_peered":"2024-01-15T18:03:04.564626+0000","last_clean":"2024-01-15T18:03:04.564626+0000","last_became_active":"2024-01-15T18:03:02.533194+0000","last_became_peered":"2024-01-15T18:03:02.533194+0000","last_unstale":"2024-01-15T18:03:04.564626+0000","last_undegraded":"2024-01-15T18:03:04.564626+0000","last_fullsized":"2024-01-15T18:03:04.564626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:52:07.900093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.572040+0000","last_change":"2024-01-15T18:03:03.567136+0000","last_active":"2024-01-15T18:03:03.572040+0000","last_peered":"2024-01-15T18:03:03.572040+0000","last_clean":"2024-01-15T18:03:03.572040+0000","last_became_active":"2024-01-15T18:03:03.566863+0000","last_became_peered":"2024-01-15T18:03:03.566863+0000","last_unstale":"2024-01-15T18:03:03.572040+0000","last_undegraded":"2024-01-15T18:03:03.572040+0000","last_fullsized":"2024-01-15T18:03:03.572040+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:21:58.996025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.553947+0000","last_change":"2024-01-15T18:03:02.532855+0000","last_active":"2024-01-15T18:03:03.553947+0000","last_peered":"2024-01-15T18:03:03.553947+0000","last_clean":"2024-01-15T18:03:03.553947+0000","last_became_active":"2024-01-15T18:03:02.532576+0000","last_became_peered":"2024-01-15T18:03:02.532576+0000","last_unstale":"2024-01-15T18:03:03.553947+0000","last_undegraded":"2024-01-15T18:03:03.553947+0000","last_fullsized":"2024-01-15T18:03:03.553947+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:10:15.666218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.578626+0000","last_change":"2024-01-15T18:03:03.575811+0000","last_active":"2024-01-15T18:03:03.578626+0000","last_peered":"2024-01-15T18:03:03.578626+0000","last_clean":"2024-01-15T18:03:03.578626+0000","last_became_active":"2024-01-15T18:03:03.574055+0000","last_became_peered":"2024-01-15T18:03:03.574055+0000","last_unstale":"2024-01-15T18:03:03.578626+0000","last_undegraded":"2024-01-15T18:03:03.578626+0000","last_fullsized":"2024-01-15T18:03:03.578626+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:55:09.343429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.550113+0000","last_change":"2024-01-15T18:03:02.543820+0000","last_active":"2024-01-15T18:03:03.550113+0000","last_peered":"2024-01-15T18:03:03.550113+0000","last_clean":"2024-01-15T18:03:03.550113+0000","last_became_active":"2024-01-15T18:03:02.543309+0000","last_became_peered":"2024-01-15T18:03:02.543309+0000","last_unstale":"2024-01-15T18:03:03.550113+0000","last_undegraded":"2024-01-15T18:03:03.550113+0000","last_fullsized":"2024-01-15T18:03:03.550113+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:37:34.446304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.577403+0000","last_change":"2024-01-15T18:03:03.575523+0000","last_active":"2024-01-15T18:03:03.577403+0000","last_peered":"2024-01-15T18:03:03.577403+0000","last_clean":"2024-01-15T18:03:03.577403+0000","last_became_active":"2024-01-15T18:03:03.575221+0000","last_became_peered":"2024-01-15T18:03:03.575221+0000","last_unstale":"2024-01-15T18:03:03.577403+0000","last_undegraded":"2024-01-15T18:03:03.577403+0000","last_fullsized":"2024-01-15T18:03:03.577403+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:02.502189+0000","last_clean_scrub_stamp":"2024-01-15T18:03:02.502189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:06:22.491470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-15T18:03:03.561353+0000","last_change":"2024-01-15T18:03:02.541780+0000","last_active":"2024-01-15T18:03:03.561353+0000","last_peered":"2024-01-15T18:03:03.561353+0000","last_clean":"2024-01-15T18:03:03.561353+0000","last_became_active":"2024-01-15T18:03:02.541337+0000","last_became_peered":"2024-01-15T18:03:02.541337+0000","last_unstale":"2024-01-15T18:03:03.561353+0000","last_undegraded":"2024-01-15T18:03:03.561353+0000","last_fullsized":"2024-01-15T18:03:03.561353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:03:01.488706+0000","last_clean_scrub_stamp":"2024-01-15T18:03:01.488706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:41.075647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":200,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-15T18:03:02.530450+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:03:02.530450+0000","last_peered":"2024-01-15T18:03:02.530450+0000","last_clean":"2024-01-15T18:03:02.530450+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:03:02.530450+0000","last_undegraded":"2024-01-15T18:03:02.530450+0000","last_fullsized":"2024-01-15T18:03:02.530450+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542214,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36628,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335212,"statfs":{"total":96636764160,"available":96599257088,"internally_reserved":0,"allocated":1318912,"data_stored":891032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2400000000000002,"5min":1.5549999999999999,"15min":1.2470000000000001},"min":{"1min":0.622,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":14.114000000000001,"5min":14.114000000000001,"15min":14.114000000000001},"last":1.206},{"interface":"front","average":{"1min":1.75,"5min":1.52,"15min":1.4650000000000001},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":6.3440000000000003,"5min":12.496,"15min":12.496},"last":1.3520000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.413,"15min":1.204},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.0229999999999997,"5min":14.497,"15min":14.497},"last":1.1359999999999999},{"interface":"front","average":{"1min":2.2440000000000002,"5min":1.444,"15min":1.2190000000000001},"min":{"1min":0.69899999999999995,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":12.565,"5min":12.565,"15min":12.565},"last":1.4059999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":1.631,"15min":1.2809999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":12.449,"5min":13.801,"15min":13.801},"last":1.272},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.3680000000000001,"15min":1.4419999999999999},"min":{"1min":0.61499999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":4.9669999999999996,"5min":7.5940000000000003,"15min":7.5940000000000003},"last":1.2410000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.782,"5min":1.5409999999999999,"15min":1.4239999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":6.3879999999999999,"5min":14.622,"15min":14.622},"last":0.75600000000000001},{"interface":"front","average":{"1min":2.3620000000000001,"5min":1.5940000000000001,"15min":1.419},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":14.042999999999999,"5min":14.042999999999999,"15min":14.042999999999999},"last":1.0489999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.4810000000000001,"15min":1.577},"min":{"1min":0.66400000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":7.3529999999999998,"5min":7.4809999999999999,"15min":7.4809999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.625,"5min":1.5740000000000001,"15min":1.452},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":4.3929999999999998,"5min":14.711,"15min":14.711},"last":1.3140000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":2.4940000000000002,"5min":1.6839999999999999,"15min":1.587},"min":{"1min":0.47499999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":12.518000000000001,"5min":12.518000000000001,"15min":12.518000000000001},"last":1.466},{"interface":"front","average":{"1min":1.734,"5min":1.5229999999999999,"15min":1.6459999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":6.2910000000000004,"5min":7.798,"15min":7.798},"last":1.492}]},{"osd":6,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":9.4589999999999996,"5min":3.1880000000000002,"15min":2.2469999999999999},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":53.664999999999999,"5min":53.664999999999999,"15min":53.664999999999999},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.6539999999999999,"15min":1.532},"min":{"1min":0.52400000000000002,"5min":0.44,"15min":0.44},"max":{"1min":4.0970000000000004,"5min":14.804,"15min":14.804},"last":1.3879999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542214,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40200,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331640,"statfs":{"total":96636764160,"available":96595599360,"internally_reserved":0,"allocated":716800,"data_stored":300504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.573,"5min":1.224,"15min":1.1100000000000001},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":5.0890000000000004,"5min":5.0890000000000004,"15min":5.0890000000000004},"last":1.046},{"interface":"front","average":{"1min":2.7250000000000001,"5min":1.46,"15min":1.236},"min":{"1min":0.80700000000000005,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":21.565000000000001,"5min":21.565000000000001,"15min":21.565000000000001},"last":1.5660000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.3260000000000001,"15min":1.2549999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":5.4169999999999998,"5min":5.4169999999999998,"15min":5.4169999999999998},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.766,"5min":1.3640000000000001,"15min":1.3540000000000001},"min":{"1min":0.80500000000000005,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":7.8109999999999999},"last":1.141}]},{"osd":2,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":3.4060000000000001,"5min":1.7010000000000001,"15min":1.417},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":34.670000000000002,"5min":34.670000000000002,"15min":34.670000000000002},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.819,"5min":1.425,"15min":1.3200000000000001},"min":{"1min":0.73499999999999999,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":5.125,"5min":5.125,"15min":5.125},"last":1.4550000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":1.5209999999999999,"15min":1.4059999999999999},"min":{"1min":0.85399999999999998,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":7.7309999999999999,"5min":7.7309999999999999,"15min":7.7309999999999999},"last":1.4970000000000001},{"interface":"front","average":{"1min":3.4580000000000002,"5min":1.8520000000000001,"15min":1.538},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":34.725000000000001,"5min":34.725000000000001,"15min":34.725000000000001},"last":1.097}]},{"osd":4,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8100000000000001,"5min":1.454,"15min":1.3779999999999999},"min":{"1min":0.52200000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":8.5489999999999995,"5min":8.5489999999999995,"15min":8.5489999999999995},"last":1.371},{"interface":"front","average":{"1min":1.7989999999999999,"5min":1.5089999999999999,"15min":1.5109999999999999},"min":{"1min":0.63200000000000001,"5min":0.501,"15min":0.501},"max":{"1min":7.7729999999999997,"5min":7.7729999999999997,"15min":7.7729999999999997},"last":0.90200000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":3.3900000000000001,"5min":1.7649999999999999,"15min":1.5009999999999999},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":34.798000000000002,"5min":34.798000000000002,"15min":34.798000000000002},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.8660000000000001,"5min":1.5940000000000001,"15min":1.528},"min":{"1min":0.59699999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":7.0620000000000003,"5min":7.0620000000000003,"15min":7.0620000000000003},"last":1.278}]},{"osd":7,"last update":"Mon Jan 15 18:02:40 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":1.6040000000000001,"15min":1.673},"min":{"1min":0.72699999999999998,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":8.2590000000000003,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":1.331},{"interface":"front","average":{"1min":3.669,"5min":1.968,"15min":1.776},"min":{"1min":0.61799999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":34.762999999999998,"5min":34.762999999999998,"15min":34.762999999999998},"last":1.417}]}]},{"osd":5,"up_from":14,"seq":60129542216,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36048,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335792,"statfs":{"total":96636764160,"available":96599851008,"internally_reserved":0,"allocated":724992,"data_stored":301156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.8089999999999999,"5min":1.274,"15min":1.1830000000000001},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":10.207000000000001,"5min":10.207000000000001,"15min":10.207000000000001},"last":1.252},{"interface":"front","average":{"1min":2.6789999999999998,"5min":1.6299999999999999,"15min":1.3580000000000001},"min":{"1min":0.77600000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":14.711,"5min":14.711,"15min":14.711},"last":1.1479999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2450000000000001,"15min":1.163},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":3.8690000000000002,"5min":4.806,"15min":4.806},"last":1.544},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.3089999999999999,"15min":1.161},"min":{"1min":0.66700000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":5.4630000000000001,"5min":5.4630000000000001,"15min":5.4630000000000001},"last":1.3140000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.375,"15min":1.333},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":3.9129999999999998,"5min":5.516,"15min":5.516},"last":1.425},{"interface":"front","average":{"1min":2.1520000000000001,"5min":1.5940000000000001,"15min":1.4339999999999999},"min":{"1min":0.80800000000000005,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":10.327999999999999,"5min":10.327999999999999,"15min":10.327999999999999},"last":1.353}]},{"osd":3,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":2.1190000000000002,"5min":1.45,"15min":1.3109999999999999},"min":{"1min":0.70299999999999996,"5min":0.622,"15min":0.622},"max":{"1min":10.249000000000001,"5min":10.249000000000001,"15min":10.249000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":2.0350000000000001,"5min":1.522,"15min":1.4339999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":9.3130000000000006,"5min":9.3130000000000006,"15min":9.3130000000000006},"last":1.6020000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":2.9729999999999999,"5min":1.6779999999999999,"15min":1.419},"min":{"1min":0.65800000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":25.776,"5min":25.776,"15min":25.776},"last":0.98899999999999999},{"interface":"front","average":{"1min":2.2450000000000001,"5min":1.696,"15min":1.538},"min":{"1min":0.63800000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":10.231,"5min":10.231,"15min":10.231},"last":1.081}]},{"osd":6,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":3.1339999999999999,"5min":1.6619999999999999,"15min":1.482},"min":{"1min":0.97699999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":11.162000000000001,"5min":11.162000000000001,"15min":11.162000000000001},"last":1.202},{"interface":"front","average":{"1min":3.9790000000000001,"5min":2.0569999999999999,"15min":1.6830000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":36.087000000000003,"5min":36.087000000000003,"15min":36.087000000000003},"last":1.0409999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:02:36 2024","interfaces":[{"interface":"back","average":{"1min":3.2610000000000001,"5min":1.992,"15min":1.734},"min":{"1min":0.89900000000000002,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":12.734,"5min":12.734,"15min":12.734},"last":1.482},{"interface":"front","average":{"1min":2.8769999999999998,"5min":1.8049999999999999,"15min":1.6379999999999999},"min":{"1min":0.81599999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":11.609,"5min":11.609,"15min":11.609},"last":1.119}]}]},{"osd":4,"up_from":14,"seq":60129542215,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40220,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331620,"statfs":{"total":96636764160,"available":96595578880,"internally_reserved":0,"allocated":729088,"data_stored":302116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.008,"5min":1.663,"15min":1.2689999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":18.673999999999999,"5min":21.643000000000001,"15min":21.643000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.786,"5min":1.286,"15min":1.1619999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":6.4989999999999997,"5min":6.4989999999999997,"15min":6.4989999999999997},"last":0.755}]},{"osd":1,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.1139999999999999,"5min":1.4239999999999999,"15min":1.242},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":12.738,"5min":12.738,"15min":12.738},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.214,"5min":1.617,"15min":1.331},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":4.9119999999999999,"5min":21.440000000000001,"15min":21.440000000000001},"last":0.81899999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.5030000000000001,"5min":1.524,"15min":1.258},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":12.974,"5min":12.974,"15min":12.974},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.294,"5min":1.6499999999999999,"15min":1.3979999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":5.9850000000000003,"5min":22.350999999999999,"15min":22.350999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.4209999999999998,"5min":1.7,"15min":1.4470000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":15.093,"5min":15.093,"15min":15.093},"last":0.72199999999999998},{"interface":"front","average":{"1min":3.169,"5min":1.8149999999999999,"15min":1.4810000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":19.486000000000001,"5min":19.486000000000001,"15min":19.486000000000001},"last":0.73599999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":2.2210000000000001,"5min":1.9279999999999999,"15min":1.5569999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":11.731,"5min":21.285,"15min":21.285},"last":0.61099999999999999},{"interface":"front","average":{"1min":2.262,"5min":1.5009999999999999,"15min":1.4330000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":18.289999999999999,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":3.0209999999999999,"5min":1.73,"15min":1.51},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":17.733000000000001,"5min":17.733000000000001,"15min":17.733000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":3.278,"5min":2.2000000000000002,"15min":1.7310000000000001},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":28.797000000000001,"5min":28.797000000000001,"15min":28.797000000000001},"last":0.78500000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:02:35 2024","interfaces":[{"interface":"back","average":{"1min":1.849,"5min":1.5980000000000001,"15min":1.5509999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":6.2210000000000001,"5min":7.6429999999999998,"15min":7.6429999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":3.032,"5min":1.9810000000000001,"15min":1.663},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":22.751000000000001,"5min":22.751000000000001,"15min":22.751000000000001},"last":0.76800000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542216,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44364,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327476,"statfs":{"total":96636764160,"available":96591335424,"internally_reserved":0,"allocated":716800,"data_stored":300504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44693960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.94299999999999995,"15min":0.83099999999999996},"min":{"1min":0.50600000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":2.246,"5min":4.9169999999999998,"15min":4.9169999999999998},"last":0.95199999999999996},{"interface":"front","average":{"1min":1.054,"5min":0.98599999999999999,"15min":0.91200000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.841,"5min":5.0529999999999999,"15min":5.0529999999999999},"last":0.73699999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.1180000000000001,"15min":1.036},"min":{"1min":0.59799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":5.7060000000000004,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.1359999999999999,"15min":1.1200000000000001},"min":{"1min":0.67700000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.6619999999999999,"5min":4.7889999999999997,"15min":4.7889999999999997},"last":1.131}]},{"osd":2,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.1579999999999999,"15min":1.0960000000000001},"min":{"1min":0.55200000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.839,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.272,"15min":1.22},"min":{"1min":0.58799999999999997,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.1219999999999999,"5min":5.0910000000000002,"15min":5.0910000000000002},"last":1.712}]},{"osd":4,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.421,"15min":1.3120000000000001},"min":{"1min":0.75800000000000001,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.75,"5min":17.082000000000001,"15min":17.082000000000001},"last":1.546},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.478,"15min":1.3740000000000001},"min":{"1min":0.82199999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.839,"5min":5.6310000000000002,"15min":5.6310000000000002},"last":1.4390000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.506,"15min":1.3839999999999999},"min":{"1min":1.016,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.4399999999999999,"5min":2.718,"15min":2.718},"last":1.181},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.9930000000000001,"15min":1.605},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.3530000000000002,"5min":40.384999999999998,"15min":40.384999999999998},"last":1.778}]},{"osd":6,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4059999999999999,"15min":1.321},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":2.4199999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5900000000000001,"15min":1.5309999999999999},"min":{"1min":0.88100000000000001,"5min":0.873,"15min":0.873},"max":{"1min":2.4020000000000001,"5min":5.5659999999999998,"15min":5.5659999999999998},"last":1.5660000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:02:27 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.7070000000000001,"15min":1.6200000000000001},"min":{"1min":0.84099999999999997,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":10.202999999999999},"last":1.825},{"interface":"front","average":{"1min":2.0350000000000001,"5min":1.6259999999999999,"15min":1.476},"min":{"1min":1.056,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":9.0389999999999997,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.3999999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542215,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44376,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94327464,"statfs":{"total":96636764160,"available":96591323136,"internally_reserved":0,"allocated":724992,"data_stored":300560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.022,"15min":1.099},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":6.101,"5min":6.101,"15min":6.101},"last":0.35099999999999998},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.022,"15min":1.1719999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.8740000000000001,"5min":2.8740000000000001,"15min":2.8740000000000001},"last":0.38700000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.1779999999999999,"15min":1.052},"min":{"1min":0.434,"5min":0.379,"15min":0.379},"max":{"1min":10.807,"5min":10.807,"15min":10.807},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.155,"15min":1.23},"min":{"1min":0.497,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":2.403,"5min":2.4049999999999998,"15min":2.4049999999999998},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.359,"15min":1.2350000000000001},"min":{"1min":0.48699999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.627},{"interface":"front","average":{"1min":1.365,"5min":1.216,"15min":1.1619999999999999},"min":{"1min":0.47799999999999998,"5min":0.442,"15min":0.442},"max":{"1min":7.2599999999999998,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":0.44700000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":1.7849999999999999,"5min":1.5489999999999999,"15min":1.569},"min":{"1min":0.76800000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":7.3209999999999997,"5min":7.3209999999999997,"15min":7.3209999999999997},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.8089999999999999,"5min":1.4750000000000001,"15min":1.444},"min":{"1min":0.58899999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":6.4290000000000003,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":2.5880000000000001,"5min":1.7150000000000001,"15min":1.5449999999999999},"min":{"1min":0.76500000000000001,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":11.016999999999999,"5min":11.016999999999999,"15min":11.016999999999999},"last":0.879},{"interface":"front","average":{"1min":3.5579999999999998,"5min":1.929,"15min":1.681},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":37.020000000000003,"5min":37.020000000000003,"15min":37.020000000000003},"last":0.59199999999999997}]},{"osd":6,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":3.012,"5min":1.8440000000000001,"15min":1.7310000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":23.762,"5min":23.762,"15min":23.762},"last":0.72099999999999997},{"interface":"front","average":{"1min":4.6059999999999999,"5min":2.1600000000000001,"15min":1.698},"min":{"1min":1.0389999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":39.926000000000002,"5min":39.926000000000002,"15min":39.926000000000002},"last":0.78800000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:02:31 2024","interfaces":[{"interface":"back","average":{"1min":3.6640000000000001,"5min":1.998,"15min":1.788},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":52.767000000000003,"5min":52.767000000000003,"15min":52.767000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":2.4580000000000002,"5min":1.7989999999999999,"15min":1.649},"min":{"1min":1.052,"5min":0.89100000000000001,"15min":0.89100000000000001},"max":{"1min":13.055999999999999,"5min":13.055999999999999,"15min":13.055999999999999},"last":0.83499999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542215,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40208,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331632,"statfs":{"total":96636764160,"available":96595591168,"internally_reserved":0,"allocated":720896,"data_stored":300594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":0.96699999999999997,"15min":0.90600000000000003},"min":{"1min":0.622,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.5939999999999999,"5min":3.5939999999999999,"15min":3.5939999999999999},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.173,"5min":1.1379999999999999,"15min":1.0900000000000001},"min":{"1min":0.44500000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.3079999999999998,"5min":3.3159999999999998,"15min":3.3159999999999998},"last":1.24}]},{"osd":2,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.254,"15min":1.0249999999999999},"min":{"1min":0.44700000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":6.8899999999999997,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":1.097},{"interface":"front","average":{"1min":1.667,"5min":1.1819999999999999,"15min":1.109},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":16.795999999999999,"5min":16.795999999999999,"15min":16.795999999999999},"last":1.052}]},{"osd":3,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.3180000000000001,"15min":1.2569999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.6619999999999999,"5min":7.8659999999999997,"15min":7.8659999999999997},"last":1.171},{"interface":"front","average":{"1min":1.25,"5min":1.369,"15min":1.2649999999999999},"min":{"1min":0.68300000000000005,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.99,"5min":14.071,"15min":14.071},"last":1.679}]},{"osd":4,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.4610000000000001,"15min":1.421},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":7.3239999999999998,"5min":7.3239999999999998,"15min":7.3239999999999998},"last":1.306},{"interface":"front","average":{"1min":2.976,"5min":1.8460000000000001,"15min":1.6060000000000001},"min":{"1min":1.196,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":22.204000000000001,"5min":22.204000000000001,"15min":22.204000000000001},"last":2.0750000000000002}]},{"osd":5,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":1.72,"15min":1.6000000000000001},"min":{"1min":0.76000000000000001,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":14.727,"5min":14.727,"15min":14.727},"last":1.95},{"interface":"front","average":{"1min":3.2290000000000001,"5min":1.881,"15min":1.5529999999999999},"min":{"1min":0.58799999999999997,"5min":0.499,"15min":0.499},"max":{"1min":19.998999999999999,"5min":19.998999999999999,"15min":19.998999999999999},"last":1.3660000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.7050000000000001,"15min":1.5649999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":7.0030000000000001,"5min":10.089,"15min":10.089},"last":1.4870000000000001},{"interface":"front","average":{"1min":3.407,"5min":1.962,"15min":1.7150000000000001},"min":{"1min":0.80200000000000005,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":27.579999999999998,"5min":27.579999999999998,"15min":27.579999999999998},"last":2.1579999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:02:34 2024","interfaces":[{"interface":"back","average":{"1min":2.5070000000000001,"5min":1.8029999999999999,"15min":1.591},"min":{"1min":0.63700000000000001,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":9.7780000000000005,"5min":9.7780000000000005,"15min":9.7780000000000005},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.944,"5min":1.764,"15min":1.6259999999999999},"min":{"1min":0.66700000000000004,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":8.0749999999999993,"5min":8.1940000000000008,"15min":8.1940000000000008},"last":1.5900000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542214,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28316,"kb_used_data":1296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343524,"statfs":{"total":96636764160,"available":96607768576,"internally_reserved":0,"allocated":1327104,"data_stored":893066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.1819999999999999,"15min":0.93200000000000005},"min":{"1min":0.437,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":8.9060000000000006,"5min":17.207000000000001,"15min":17.207000000000001},"last":1.393},{"interface":"front","average":{"1min":1.228,"5min":1.056,"15min":1.04},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":2.0910000000000002,"5min":3.056,"15min":3.056},"last":0.84799999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.103,"15min":1.0169999999999999},"min":{"1min":0.52600000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":3.7789999999999999,"5min":3.7789999999999999,"15min":3.7789999999999999},"last":1.093},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.3120000000000001,"15min":1.198},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.917,"5min":10.942,"15min":10.942},"last":0.97799999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.1619999999999999,"15min":1.0720000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":8.9949999999999992,"5min":8.9949999999999992,"15min":8.9949999999999992},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.3300000000000001,"15min":1.306},"min":{"1min":0.81599999999999995,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":5.8360000000000003,"5min":5.8360000000000003,"15min":5.8360000000000003},"last":1.4870000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":2.9460000000000002,"5min":1.7210000000000001,"15min":1.5009999999999999},"min":{"1min":0.97199999999999998,"5min":0.69199999999999995,"15min":0.69199999999999995},"max":{"1min":28.631,"5min":28.631,"15min":28.631},"last":1.046},{"interface":"front","average":{"1min":2.137,"5min":1.4570000000000001,"15min":1.377},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":15.968999999999999,"5min":15.968999999999999,"15min":15.968999999999999},"last":1.2609999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":2.5619999999999998,"5min":1.647,"15min":1.4870000000000001},"min":{"1min":0.70299999999999996,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":9.1059999999999999,"5min":9.1059999999999999,"15min":9.1059999999999999},"last":1.3440000000000001},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.712,"15min":1.542},"min":{"1min":1.3140000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":8.4369999999999994,"5min":8.4369999999999994,"15min":8.4369999999999994},"last":1.6200000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":3.9089999999999998,"5min":1.976,"15min":1.6399999999999999},"min":{"1min":1.1870000000000001,"5min":0.96399999999999997,"15min":0.96399999999999997},"max":{"1min":36.390999999999998,"5min":36.390999999999998,"15min":36.390999999999998},"last":1.3240000000000001},{"interface":"front","average":{"1min":3.7349999999999999,"5min":1.9490000000000001,"15min":1.655},"min":{"1min":1.0660000000000001,"5min":0.85199999999999998,"15min":0.85199999999999998},"max":{"1min":36.207000000000001,"5min":36.207000000000001,"15min":36.207000000000001},"last":1.4359999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:02:29 2024","interfaces":[{"interface":"back","average":{"1min":2.641,"5min":1.7789999999999999,"15min":1.653},"min":{"1min":1.0609999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":12.164,"5min":12.164,"15min":12.164},"last":1.5760000000000001},{"interface":"front","average":{"1min":2.0640000000000001,"5min":1.577,"15min":1.496},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":9.0749999999999993,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":1.544}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:03:08.551 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:03:08.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:03:08.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.812+0000 7fea2aa00640 1 Processor -- start 2024-01-15T18:03:08.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.812+0000 7fea2aa00640 1 -- start start 2024-01-15T18:03:08.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fea240b3fa0 0x7fea240d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2aa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea240d0af0 con 0x7fea241609d0 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2aa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea240ce7b0 con 0x7fea240b3fa0 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea240cdbe0 con 0x7fea240d1d40 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fea240b3fa0 0x7fea240d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fea240b3fa0 0x7fea240d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56594/0 (socket says 172.21.15.2:56594) 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea2a000640 1 -- 172.21.15.2:0/3011015425 learned_addr learned my addr 172.21.15.2:0/3011015425 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.814+0000 7fea28c00640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.815+0000 7fea2a000640 1 -- 172.21.15.2:0/3011015425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 msgr2=0x7fea240cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.815+0000 7fea2a000640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.815+0000 7fea2a000640 1 -- 172.21.15.2:0/3011015425 >> v1:172.21.15.137:6789/0 conn(0x7fea241609d0 legacy=0x7fea240cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:08.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.815+0000 7fea2a000640 1 -- 172.21.15.2:0/3011015425 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea240cde80 con 0x7fea240b3fa0 2024-01-15T18:03:08.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.815+0000 7fea28c00640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:08.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.816+0000 7fea2a000640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea240b3fa0 0x7fea240d01d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009e50 tx=0x7fea1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db499d120eb1b98 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:08.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.816+0000 7fea23e00640 1 -- 172.21.15.2:0/3011015425 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c02ca70 con 0x7fea240b3fa0 2024-01-15T18:03:08.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.816+0000 7fea23e00640 1 -- 172.21.15.2:0/3011015425 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1c02cbd0 con 0x7fea240b3fa0 2024-01-15T18:03:08.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.816+0000 7fea23e00640 1 -- 172.21.15.2:0/3011015425 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c0343f0 con 0x7fea240b3fa0 2024-01-15T18:03:08.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 -- 172.21.15.2:0/3011015425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea240b3fa0 msgr2=0x7fea240d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea240b3fa0 0x7fea240d01d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009e50 tx=0x7fea1c02bed0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 -- 172.21.15.2:0/3011015425 shutdown_connections 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea240b3fa0 0x7fea240d01d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 --2- 172.21.15.2:0/3011015425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 -- 172.21.15.2:0/3011015425 >> 172.21.15.2:0/3011015425 conn(0x7fea2409c3e0 msgr2=0x7fea240d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 -- 172.21.15.2:0/3011015425 shutdown_connections 2024-01-15T18:03:08.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.817+0000 7fea2aa00640 1 -- 172.21.15.2:0/3011015425 wait complete. 2024-01-15T18:03:08.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.819+0000 7fea2aa00640 1 Processor -- start 2024-01-15T18:03:08.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.819+0000 7fea2aa00640 1 -- start start 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.820+0000 7fea2aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea240b3fa0 0x7fea2414c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.820+0000 7fea2aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240b6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.820+0000 7fea2aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 0x7fea240cfaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.820+0000 7fea2aa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2414cd40 con 0x7fea241609d0 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.820+0000 7fea2aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2414d180 con 0x7fea240b3fa0 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.820+0000 7fea2aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2414d420 con 0x7fea240d1d40 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 0x7fea240cfaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 0x7fea240cfaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56608/0 (socket says 172.21.15.2:56608) 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 -- 172.21.15.2:0/2307522598 learned_addr learned my addr 172.21.15.2:0/2307522598 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:08.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea28c00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240b6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea29600640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea240b3fa0 0x7fea2414c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 -- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 msgr2=0x7fea240b6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240b6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 -- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea240b3fa0 msgr2=0x7fea2414c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea240b3fa0 0x7fea2414c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 -- 172.21.15.2:0/2307522598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1c009b30 con 0x7fea241609d0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea28c00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea29600640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea240b3fa0 0x7fea2414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.821+0000 7fea2a000640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 0x7fea240cfaa0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009e50 tx=0x7fea1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.822+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c009510 con 0x7fea241609d0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.822+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea1c03e450 con 0x7fea241609d0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.822+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1c03e800 con 0x7fea241609d0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.822+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2414d700 con 0x7fea241609d0 2024-01-15T18:03:08.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.822+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2414dcc0 con 0x7fea241609d0 2024-01-15T18:03:08.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.823+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea240b4780 con 0x7fea241609d0 2024-01-15T18:03:08.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.824+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fea1c03e960 con 0x7fea241609d0 2024-01-15T18:03:08.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.825+0000 7fea21600640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9fc06b8e0 0x7fe9fc06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:08.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.825+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fea1c0ad3b0 con 0x7fea241609d0 2024-01-15T18:03:08.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.826+0000 7fea29600640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9fc06b8e0 0x7fe9fc06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:08.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.827+0000 7fea29600640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9fc06b8e0 0x7fe9fc06dda0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fea14003eb0 tx=0x7fea14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:08.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:08.830+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea1c077af0 con 0x7fea241609d0 2024-01-15T18:03:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.100+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fea240d06c0 con 0x7fea241609d0 2024-01-15T18:03:09.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.106+0000 7fea21600640 1 -- 172.21.15.2:0/2307522598 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7fea1c07b110 con 0x7fea241609d0 2024-01-15T18:03:09.110 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T18:03:09.110 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T18:03:09.111 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T18:03:09.111 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T18:03:09.111 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:03:09.111 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.108+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9fc06b8e0 msgr2=0x7fe9fc06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.108+0000 7fea2aa00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9fc06b8e0 0x7fe9fc06dda0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fea14003eb0 tx=0x7fea14009040 comp rx=0 tx=0).stop 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.108+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 msgr2=0x7fea240cfaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.108+0000 7fea2aa00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 0x7fea240cfaa0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009e50 tx=0x7fea1c004a40 comp rx=0 tx=0).stop 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 shutdown_connections 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9fc06b8e0 0x7fe9fc06dda0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fea241609d0 0x7fea240cfaa0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fea240d1d40 0x7fea240b6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 --2- 172.21.15.2:0/2307522598 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fea240b3fa0 0x7fea2414c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 >> 172.21.15.2:0/2307522598 conn(0x7fea2409c3e0 msgr2=0x7fea240b2910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:09.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 shutdown_connections 2024-01-15T18:03:09.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.109+0000 7fea2aa00640 1 -- 172.21.15.2:0/2307522598 wait complete. 2024-01-15T18:03:09.133 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:03:09.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.341+0000 7feb2b000640 1 Processor -- start 2024-01-15T18:03:09.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.341+0000 7feb2b000640 1 -- start start 2024-01-15T18:03:09.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.343+0000 7feb2b000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feb2415c0e0 0x7feb240cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.343+0000 7feb2b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7feb2406cb80 0x7feb2415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.343+0000 7feb2b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2415cb00 con 0x7feb2415b7d0 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.343+0000 7feb2b000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb240ccec0 con 0x7feb2415c0e0 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.343+0000 7feb2b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb240cd1a0 con 0x7feb2406cb80 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.343+0000 7feb23e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feb2415c0e0 0x7feb240cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb23e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feb2415c0e0 0x7feb240cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56624/0 (socket says 172.21.15.2:56624) 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb23e00640 1 -- 172.21.15.2:0/3099282748 learned_addr learned my addr 172.21.15.2:0/3099282748 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb29000640 1 --2- 172.21.15.2:0/3099282748 >> v2:172.21.15.137:3300/0 conn(0x7feb2406cb80 0x7feb2415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb23e00640 1 -- 172.21.15.2:0/3099282748 >> v2:172.21.15.137:3300/0 conn(0x7feb2406cb80 msgr2=0x7feb2415cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:09.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb23e00640 1 --2- 172.21.15.2:0/3099282748 >> v2:172.21.15.137:3300/0 conn(0x7feb2406cb80 0x7feb2415cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb23e00640 1 -- 172.21.15.2:0/3099282748 >> v1:172.21.15.137:6789/0 conn(0x7feb2415b7d0 legacy=0x7feb2406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:09.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.344+0000 7feb23e00640 1 -- 172.21.15.2:0/3099282748 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240cd480 con 0x7feb2415c0e0 2024-01-15T18:03:09.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.345+0000 7feb23e00640 1 --2- 172.21.15.2:0/3099282748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2415c0e0 0x7feb240cc5c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feb24152750 tx=0x7feb1402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642d7c293bb08dda server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:09.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.345+0000 7feb22a00640 1 -- 172.21.15.2:0/3099282748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14031870 con 0x7feb2415c0e0 2024-01-15T18:03:09.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb22a00640 1 -- 172.21.15.2:0/3099282748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb140319d0 con 0x7feb2415c0e0 2024-01-15T18:03:09.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb22a00640 1 -- 172.21.15.2:0/3099282748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14031d60 con 0x7feb2415c0e0 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 -- 172.21.15.2:0/3099282748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2415c0e0 msgr2=0x7feb240cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 --2- 172.21.15.2:0/3099282748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2415c0e0 0x7feb240cc5c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7feb24152750 tx=0x7feb1402b910 comp rx=0 tx=0).stop 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 -- 172.21.15.2:0/3099282748 shutdown_connections 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 --2- 172.21.15.2:0/3099282748 >> v2:172.21.15.137:3300/0 conn(0x7feb2406cb80 0x7feb2415cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 --2- 172.21.15.2:0/3099282748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2415c0e0 0x7feb240cc5c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 -- 172.21.15.2:0/3099282748 >> 172.21.15.2:0/3099282748 conn(0x7feb2406da20 msgr2=0x7feb2406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 -- 172.21.15.2:0/3099282748 shutdown_connections 2024-01-15T18:03:09.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.346+0000 7feb2b000640 1 -- 172.21.15.2:0/3099282748 wait complete. 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.348+0000 7feb2b000640 1 Processor -- start 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.348+0000 7feb2b000640 1 -- start start 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.349+0000 7feb2b000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 0x7feb24151270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.349+0000 7feb2b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb2415b7d0 0x7feb24152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.349+0000 7feb2b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb2415c0e0 0x7feb24152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.349+0000 7feb2b000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb24151d10 con 0x7feb2406cb80 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.349+0000 7feb2b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb24153e90 con 0x7feb2415b7d0 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.349+0000 7feb2b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb241517b0 con 0x7feb2415c0e0 2024-01-15T18:03:09.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 0x7feb24151270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:09.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 0x7feb24151270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56626/0 (socket says 172.21.15.2:56626) 2024-01-15T18:03:09.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 -- 172.21.15.2:0/90125902 learned_addr learned my addr 172.21.15.2:0/90125902 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:09.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb29000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb2415c0e0 0x7feb24152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23400640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb2415b7d0 0x7feb24152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 -- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb2415c0e0 msgr2=0x7feb24152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb2415c0e0 0x7feb24152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 -- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb2415b7d0 msgr2=0x7feb24152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb2415b7d0 0x7feb24152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.350+0000 7feb23e00640 1 -- 172.21.15.2:0/90125902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140094d0 con 0x7feb2406cb80 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.351+0000 7feb23e00640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 0x7feb24151270 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7feb14000c00 tx=0x7feb140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.351+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb1403abe0 con 0x7feb2406cb80 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.351+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1403ad40 con 0x7feb2406cb80 2024-01-15T18:03:09.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.351+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb14044610 con 0x7feb2406cb80 2024-01-15T18:03:09.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.351+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb24153280 con 0x7feb2406cb80 2024-01-15T18:03:09.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.351+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb24153840 con 0x7feb2406cb80 2024-01-15T18:03:09.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.352+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb240686f0 con 0x7feb2406cb80 2024-01-15T18:03:09.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.353+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7feb1404d450 con 0x7feb2406cb80 2024-01-15T18:03:09.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.354+0000 7feb1be00640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feafc06b860 0x7feafc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:09.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.354+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7feb1402d040 con 0x7feb2406cb80 2024-01-15T18:03:09.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.355+0000 7feb23400640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feafc06b860 0x7feafc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:09.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.356+0000 7feb23400640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feafc06b860 0x7feafc06dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feb24155770 tx=0x7feb0c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:09.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.359+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb140772d0 con 0x7feb2406cb80 2024-01-15T18:03:09.482 INFO:tasks.ceph.mds.c.smithi002.stderr:2024-01-15T18:03:09.477+0000 7f2066800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:03:09.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.596+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb24151fc0 con 0x7feb2406cb80 2024-01-15T18:03:09.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.597+0000 7feb1be00640 1 -- 172.21.15.2:0/90125902 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+4910 (secure 0 0 0) 0x7feb14035090 con 0x7feb2406cb80 2024-01-15T18:03:09.603 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:03:09.603 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":36,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1133084700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1133084700},{"type":"v1","addr":"172.21.15.137:6836","nonce":1133084700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5614,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/302473054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":302473054},{"type":"v1","addr":"172.21.15.137:6837","nonce":302473054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2347293827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2347293827},{"type":"v1","addr":"172.21.15.2:6833","nonce":2347293827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:03:02.994614+0000","modified":"2024-01-15T18:03:06.071657+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6312},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6312":{"gid":6312,"name":"c","rank":0,"incarnation":32,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6835/2855093754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2855093754},{"type":"v1","addr":"172.21.15.2:6835","nonce":2855093754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-15T18:03:09.603 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 36 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.600+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feafc06b860 msgr2=0x7feafc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.600+0000 7feb2b000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feafc06b860 0x7feafc06dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7feb24155770 tx=0x7feb0c009040 comp rx=0 tx=0).stop 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 msgr2=0x7feb24151270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 0x7feb24151270 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7feb14000c00 tx=0x7feb140046c0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 shutdown_connections 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feafc06b860 0x7feafc06dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feb2415c0e0 0x7feb24152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feb2415b7d0 0x7feb24152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 --2- 172.21.15.2:0/90125902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feb2406cb80 0x7feb24151270 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 >> 172.21.15.2:0/90125902 conn(0x7feb2406da20 msgr2=0x7feb240b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 shutdown_connections 2024-01-15T18:03:09.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:09.601+0000 7feb2b000640 1 -- 172.21.15.2:0/90125902 wait complete. 2024-01-15T18:03:10.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 36, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:03:02.994614+0000', 'modified': '2024-01-15T18:03:06.071657+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6312}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6312': {'gid': 6312, 'name': 'c', 'rank': 0, 'incarnation': 32, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.2:6835/2855093754', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6834', 'nonce': 2855093754}, {'type': 'v1', 'addr': '172.21.15.2:6835', 'nonce': 2855093754}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:03:10.027 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:03:10.028 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-15T18:03:10.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.255+0000 7f4075c00640 1 Processor -- start 2024-01-15T18:03:10.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.255+0000 7f4075c00640 1 -- start start 2024-01-15T18:03:10.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f4075c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f40700ccb80 0x7f407006c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f4075c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700cb620 0x7f40700cba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f4075c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4070071340 con 0x7f40700ce1d0 2024-01-15T18:03:10.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f4075c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407006c670 con 0x7f40700ccb80 2024-01-15T18:03:10.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f4075c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407015bd50 con 0x7f40700cb620 2024-01-15T18:03:10.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f406fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f40700ce1d0 0x7f407015b640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38118/0 (socket says 172.21.15.2:38118) 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f406fe00640 1 -- 172.21.15.2:0/1987312534 learned_addr learned my addr 172.21.15.2:0/1987312534 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.256+0000 7f406f400640 1 --2- 172.21.15.2:0/1987312534 >> v2:172.21.15.2:3300/0 conn(0x7f40700ccb80 0x7f407006c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f4075200640 1 --2- 172.21.15.2:0/1987312534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700cb620 0x7f40700cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f406ea00640 1 -- 172.21.15.2:0/1987312534 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159770883 0 0) 0x7f4070071340 con 0x7f40700ce1d0 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f406ea00640 1 -- 172.21.15.2:0/1987312534 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4050003520 con 0x7f40700ce1d0 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f406f400640 1 -- 172.21.15.2:0/1987312534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700cb620 msgr2=0x7f40700cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f406f400640 1 --2- 172.21.15.2:0/1987312534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700cb620 0x7f40700cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f406f400640 1 -- 172.21.15.2:0/1987312534 >> v1:172.21.15.2:6789/0 conn(0x7f40700ce1d0 legacy=0x7f407015b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.257+0000 7f406f400640 1 -- 172.21.15.2:0/1987312534 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40700cbf90 con 0x7f40700ccb80 2024-01-15T18:03:10.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.258+0000 7f406f400640 1 --2- 172.21.15.2:0/1987312534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f407006c130 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f40600097a0 tx=0x7f406002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1de6fccbd821d74b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f406ea00640 1 -- 172.21.15.2:0/1987312534 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060031830 con 0x7f40700ccb80 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f406ea00640 1 -- 172.21.15.2:0/1987312534 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4060031990 con 0x7f40700ccb80 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f406ea00640 1 -- 172.21.15.2:0/1987312534 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060031d20 con 0x7f40700ccb80 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 -- 172.21.15.2:0/1987312534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 msgr2=0x7f407006c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 --2- 172.21.15.2:0/1987312534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f407006c130 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f40600097a0 tx=0x7f406002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 -- 172.21.15.2:0/1987312534 shutdown_connections 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 --2- 172.21.15.2:0/1987312534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700cb620 0x7f40700cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 --2- 172.21.15.2:0/1987312534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f407006c130 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 -- 172.21.15.2:0/1987312534 >> 172.21.15.2:0/1987312534 conn(0x7f40700b0840 msgr2=0x7f40700b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 -- 172.21.15.2:0/1987312534 shutdown_connections 2024-01-15T18:03:10.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.259+0000 7f4075c00640 1 -- 172.21.15.2:0/1987312534 wait complete. 2024-01-15T18:03:10.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.261+0000 7f4075c00640 1 Processor -- start 2024-01-15T18:03:10.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.261+0000 7f4075c00640 1 -- start start 2024-01-15T18:03:10.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.262+0000 7f4075c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40700cb620 0x7f407014c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.262+0000 7f4075c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f40700ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.262+0000 7f4075c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700ce1d0 0x7f40700b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.262+0000 7f4075c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407014cc50 con 0x7f40700ccb80 2024-01-15T18:03:10.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.262+0000 7f4075c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40700ae210 con 0x7f40700cb620 2024-01-15T18:03:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.262+0000 7f4075c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40700ad170 con 0x7f40700ce1d0 2024-01-15T18:03:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f40700ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f40700ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56640/0 (socket says 172.21.15.2:56640) 2024-01-15T18:03:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 -- 172.21.15.2:0/4245195971 learned_addr learned my addr 172.21.15.2:0/4245195971 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406fe00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40700cb620 0x7f407014c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f4075200640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700ce1d0 0x7f40700b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 -- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700ce1d0 msgr2=0x7f40700b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700ce1d0 0x7f40700b28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 -- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40700cb620 msgr2=0x7f407014c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40700cb620 0x7f407014c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 -- 172.21.15.2:0/4245195971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4060009400 con 0x7f40700ccb80 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406fe00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40700cb620 0x7f407014c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:10.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.263+0000 7f406f400640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f40700ac870 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f406002bee0 tx=0x7f40600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.264+0000 7f4067e00640 1 -- 172.21.15.2:0/4245195971 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406003a6f0 con 0x7f40700ccb80 2024-01-15T18:03:10.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.264+0000 7f4067e00640 1 -- 172.21.15.2:0/4245195971 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406003a870 con 0x7f40700ccb80 2024-01-15T18:03:10.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.264+0000 7f4067e00640 1 -- 172.21.15.2:0/4245195971 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406003ac20 con 0x7f40700ccb80 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.264+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40700ad450 con 0x7f40700ccb80 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.264+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40700ada10 con 0x7f40700ccb80 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.267+0000 7f4067e00640 1 -- 172.21.15.2:0/4245195971 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4060030380 con 0x7f40700ccb80 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.268+0000 7f4067e00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f404806b860 0x7f404806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.268+0000 7f4067e00640 1 -- 172.21.15.2:0/4245195971 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f40600ac100 con 0x7f40700ccb80 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.268+0000 7f406fe00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f404806b860 0x7f404806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.268+0000 7f4079e24740 1 Processor -- start 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.269+0000 7f4079e24740 1 -- start start 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd05dd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f406fe00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f404806b860 0x7f404806dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f40580011e0 tx=0x7f4058009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd05dc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4079e24740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd0620730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4079e24740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd053b880 con 0x55afd05dc7b0 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd054aae0 con 0x55afd05dd070 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55afd05481d0 con 0x55afd05d7d70 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd0620730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd0620730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56656/0 (socket says 172.21.15.2:56656) 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 -- 172.21.15.2:0/753657013 learned_addr learned my addr 172.21.15.2:0/753657013 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f406f400640 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd05dc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f406fe00640 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd05dd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 -- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 msgr2=0x55afd05dc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd05dc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 -- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 msgr2=0x55afd05dd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd05dd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f4075200640 1 -- 172.21.15.2:0/753657013 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afd050b860 con 0x55afd05dc7b0 2024-01-15T18:03:10.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.270+0000 7f406fe00640 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd05dd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.271+0000 7f4075200640 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd0620730 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4068009fa0 tx=0x7f40680077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7298e5316179fa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.271+0000 7f4064c00640 1 -- 172.21.15.2:0/753657013 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068013780 con 0x55afd05dc7b0 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.271+0000 7f4064c00640 1 -- 172.21.15.2:0/753657013 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406800f050 con 0x55afd05dc7b0 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.271+0000 7f4064c00640 1 -- 172.21.15.2:0/753657013 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068013b20 con 0x55afd05dc7b0 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 -- 172.21.15.2:0/753657013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 msgr2=0x55afd0620730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd0620730 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4068009fa0 tx=0x7f40680077c0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 -- 172.21.15.2:0/753657013 shutdown_connections 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd0620730 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd05dc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 --2- 172.21.15.2:0/753657013 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd05dd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 -- 172.21.15.2:0/753657013 >> 172.21.15.2:0/753657013 conn(0x55afd059ec70 msgr2=0x55afd0622b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 -- 172.21.15.2:0/753657013 shutdown_connections 2024-01-15T18:03:10.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.272+0000 7f4079e24740 1 -- 172.21.15.2:0/753657013 wait complete. 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.273+0000 7f4079e24740 1 -- start start 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.273+0000 7f4079e24740 1 Processor -- start 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0606490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd06299d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd0629f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd0543ac0 con 0x55afd05dd070 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd04e02f0 con 0x55afd05dc7b0 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55afd04fdc70 con 0x55afd05d7d70 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4079e24740 10 client.0 _ll_register_callbacks cb 0x55afd05023b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4075200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd0629f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4075200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd0629f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56672/0 (socket says 172.21.15.2:56672) 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f4075200640 1 -- 172.21.15.2:0/1285869011 learned_addr learned my addr 172.21.15.2:0/1285869011 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:10.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f406f400640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd06299d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.274+0000 7f406fe00640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0606490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4075200640 1 -- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 msgr2=0x55afd0606490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4075200640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0606490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4075200640 1 -- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 msgr2=0x55afd06299d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4075200640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd06299d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4075200640 1 -- 172.21.15.2:0/1285869011 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4068009ba0 con 0x55afd05dd070 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f406fe00640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0606490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4075200640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd0629f10 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4068000f80 tx=0x7f4068004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f8474e94447a7f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f405d600640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f405d600640 1 -- 172.21.15.2:0/1285869011 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40680045f0 con 0x55afd05dd070 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f405d600640 1 -- 172.21.15.2:0/1285869011 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4068004750 con 0x55afd05dd070 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.275+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55afd053caa0 con 0x55afd05dd070 2024-01-15T18:03:10.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f405d600640 1 -- 172.21.15.2:0/1285869011 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4068003e60 con 0x55afd05dd070 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 10 client.6450 fetch_fsmap learned FSMap version 36 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55afd0542140 con 0x55afd05dd070 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f405d600640 1 -- 172.21.15.2:0/1285869011 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55afd053caa0 con 0x55afd05dd070 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f405d600640 1 -- 172.21.15.2:0/1285869011 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4068013580 con 0x55afd05dd070 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 10 client.6450 fetch_fsmap finished waiting for FSMap version 36 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 10 client.6450 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 20 client.6450 populate_metadata read hostname 'smithi002' 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd0602fa0 0x55afd063fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 4 client.6450 mds_command: new command op to 6312 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 --> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afd053c7c0 con 0x55afd0602fa0 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.276+0000 7f406fe00640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd0602fa0 0x55afd063fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.277+0000 7f406fe00640 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd0602fa0 0x55afd063fc70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b758231619712f9d server_cookie=6c0c451802e8f803 in_seq=0 out_seq=0 2024-01-15T18:03:10.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.277+0000 7f405d600640 10 client.6450 ms_handle_connect on v2:172.21.15.2:6834/2855093754 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f405d600640 1 -- 172.21.15.2:0/1285869011 <== mds.0 v2:172.21.15.2:6834/2855093754 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55afd053c7c0 con 0x55afd0602fa0 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f405d600640 10 client.6450 handle_command_reply: tid=0 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 client.6450 shutdown 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 msgr2=0x55afd0629f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd0629f10 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4068000f80 tx=0x7f4068004ab0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 shutdown_connections 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd0602fa0 0x55afd063fc70 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd0629f10 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd06299d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 --2- 172.21.15.2:0/1285869011 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0606490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f405d600640 0 client.6450 ms_handle_reset on v2:172.21.15.2:6834/2855093754 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 >> 172.21.15.2:0/1285869011 conn(0x55afd059ec70 msgr2=0x55afd061fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 shutdown_connections 2024-01-15T18:03:10.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.278+0000 7f4079e24740 1 -- 172.21.15.2:0/1285869011 wait complete. 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.279+0000 7f4079e24740 20 client.6450 trim_cache size 0 max 16384 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.296+0000 7f4079e24740 1 Processor -- start 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.296+0000 7f4079e24740 1 -- start start 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0629080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd0622b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4079e24740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd06230a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4079e24740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd02a6f20 con 0x55afd05dd070 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd0600db0 con 0x55afd05dc7b0 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55afd06353f0 con 0x55afd05d7d70 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd06230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd06230a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56680/0 (socket says 172.21.15.2:56680) 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 -- 172.21.15.2:0/49787043 learned_addr learned my addr 172.21.15.2:0/49787043 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:10.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 -- 172.21.15.2:0/49787043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 msgr2=0x55afd0629080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0629080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 -- 172.21.15.2:0/49787043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 msgr2=0x55afd0622b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd0622b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.297+0000 7f4075200640 1 -- 172.21.15.2:0/49787043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afd04dfb80 con 0x55afd05dd070 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.298+0000 7f4075200640 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd06230a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f406800b000 tx=0x7f4068021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e1e7cfb8ee834d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.298+0000 7f4065600640 1 -- 172.21.15.2:0/49787043 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068009370 con 0x55afd05dd070 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.298+0000 7f4065600640 1 -- 172.21.15.2:0/49787043 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068023070 con 0x55afd05dd070 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.298+0000 7f4065600640 1 -- 172.21.15.2:0/49787043 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40680090c0 con 0x55afd05dd070 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 -- 172.21.15.2:0/49787043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 msgr2=0x55afd06230a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd06230a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f406800b000 tx=0x7f4068021070 comp rx=0 tx=0).stop 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 -- 172.21.15.2:0/49787043 shutdown_connections 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dd070 0x55afd06230a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dc7b0 0x55afd0622b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 --2- 172.21.15.2:0/49787043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd0629080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 -- 172.21.15.2:0/49787043 >> 172.21.15.2:0/49787043 conn(0x55afd059ec70 msgr2=0x55afd060b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 -- 172.21.15.2:0/49787043 shutdown_connections 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 -- 172.21.15.2:0/49787043 wait complete. 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.299+0000 7f4079e24740 1 -- start start 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.300+0000 7f4079e24740 1 Processor -- start 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd060b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd060ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd068cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd068e180 con 0x55afd05dc7b0 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd0629cd0 con 0x55afd05dd070 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x55afd0644740 con 0x55afd05d7d70 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f4079e24740 10 client.0 _ll_register_callbacks cb 0x7f4068009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:03:10.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd060ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd060ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56684/0 (socket says 172.21.15.2:56684) 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 -- 172.21.15.2:0/190435554 learned_addr learned my addr 172.21.15.2:0/190435554 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406fe00640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd060b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 -- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 msgr2=0x55afd060b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd060b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 -- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 msgr2=0x55afd068cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd068cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406f400640 1 -- 172.21.15.2:0/190435554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afd0542140 con 0x55afd05dc7b0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.301+0000 7f406fe00640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd060b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f406f400640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd060ba60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f406007a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343c2856e208e707 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f405d600640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f405d600640 1 -- 172.21.15.2:0/190435554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4060091070 con 0x55afd05dc7b0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f405d600640 1 -- 172.21.15.2:0/190435554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4060082570 con 0x55afd05dc7b0 2024-01-15T18:03:10.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f406800f050 con 0x55afd05dc7b0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f405d600640 1 -- 172.21.15.2:0/190435554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40600a9b40 con 0x55afd05dc7b0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 10 client.6456 fetch_fsmap learned FSMap version 36 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55afd0505170 con 0x55afd05dc7b0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f405d600640 1 -- 172.21.15.2:0/190435554 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406800f050 con 0x55afd05dc7b0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f405d600640 1 -- 172.21.15.2:0/190435554 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 36) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4060076980 con 0x55afd05dc7b0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 10 client.6456 fetch_fsmap finished waiting for FSMap version 36 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 10 client.6456 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 20 client.6456 populate_metadata read hostname 'smithi002' 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.302+0000 7f4079e24740 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd068d7d0 0x55afd060a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f4079e24740 4 client.6456 mds_command: new command op to 6312 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 --> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55afd02446c0 con 0x55afd068d7d0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f406fe00640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd068d7d0 0x55afd060a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f406fe00640 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd068d7d0 0x55afd060a370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=836a6af1f17ed0a6 server_cookie=a14405406c5bf71b in_seq=0 out_seq=0 2024-01-15T18:03:10.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f405d600640 10 client.6456 ms_handle_connect on v2:172.21.15.2:6834/2855093754 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f405d600640 1 -- 172.21.15.2:0/190435554 <== mds.0 v2:172.21.15.2:6834/2855093754 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55afd02446c0 con 0x55afd068d7d0 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f405d600640 10 client.6456 handle_command_reply: tid=0 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.303+0000 7f4079e24740 1 client.6456 shutdown 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 msgr2=0x55afd060ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd060ba60 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f4060000c00 tx=0x7f406007a270 comp rx=0 tx=0).stop 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 shutdown_connections 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:6834/2855093754,v1:172.21.15.2:6835/2855093754] conn(0x55afd068d7d0 0x55afd060a370 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x55afd05dd070 0x55afd068cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x55afd05dc7b0 0x55afd060ba60 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 --2- 172.21.15.2:0/190435554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x55afd05d7d70 0x55afd060b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f405d600640 0 client.6456 ms_handle_reset on v2:172.21.15.2:6834/2855093754 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 >> 172.21.15.2:0/190435554 conn(0x55afd059ec70 msgr2=0x55afd0628de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 shutdown_connections 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 1 -- 172.21.15.2:0/190435554 wait complete. 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.304+0000 7f4079e24740 20 client.6456 trim_cache size 0 max 16384 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 6312, 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 6.7749151779999996, 2024-01-15T18:03:10.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 36, 2024-01-15T18:03:10.343 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 61, 2024-01-15T18:03:10.343 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 61, 2024-01-15T18:03:10.343 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 12.168329233 2024-01-15T18:03:10.343 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:03:10.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f404806b860 msgr2=0x7f404806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f404806b860 0x7f404806dd20 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f40580011e0 tx=0x7f4058009040 comp rx=0 tx=0).stop 2024-01-15T18:03:10.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 msgr2=0x7f40700ac870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f40700ac870 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f406002bee0 tx=0x7f40600046c0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 shutdown_connections 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f404806b860 0x7f404806dd20 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40700ce1d0 0x7f40700b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f40700ccb80 0x7f40700ac870 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 --2- 172.21.15.2:0/4245195971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f40700cb620 0x7f407014c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 >> 172.21.15.2:0/4245195971 conn(0x7f40700b0840 msgr2=0x7f40700717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 shutdown_connections 2024-01-15T18:03:10.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.305+0000 7f4075c00640 1 -- 172.21.15.2:0/4245195971 wait complete. 2024-01-15T18:03:10.345 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:03:10.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.568+0000 7fda99c00640 1 Processor -- start 2024-01-15T18:03:10.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.568+0000 7fda99c00640 1 -- start start 2024-01-15T18:03:10.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.570+0000 7fda99c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda940b3c70 0x7fda94070f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.570+0000 7fda99c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.570+0000 7fda99c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda94071cb0 con 0x7fda9406c470 2024-01-15T18:03:10.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.570+0000 7fda99c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9406ce30 con 0x7fda940b3c70 2024-01-15T18:03:10.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.570+0000 7fda99c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda94070a70 con 0x7fda94162440 2024-01-15T18:03:10.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda93400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda92a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda940b3c70 0x7fda94070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda8be00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fda9406c470 0x7fda94161d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:37534/0 (socket says 172.21.15.2:37534) 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda93400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9406c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:36230/0 (socket says 172.21.15.2:36230) 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda8be00640 1 -- 172.21.15.2:0/842692585 learned_addr learned my addr 172.21.15.2:0/842692585 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda92a00640 1 -- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 msgr2=0x7fda9406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda92a00640 1 --2- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda92a00640 1 -- 172.21.15.2:0/842692585 >> v1:172.21.15.137:6789/0 conn(0x7fda9406c470 legacy=0x7fda94161d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.571+0000 7fda92a00640 1 -- 172.21.15.2:0/842692585 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94162960 con 0x7fda940b3c70 2024-01-15T18:03:10.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.573+0000 7fda92a00640 1 --2- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda940b3c70 0x7fda94070f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fda840097a0 tx=0x7fda8402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c82e66ec9c1823b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.573+0000 7fda92000640 1 -- 172.21.15.2:0/842692585 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8402d040 con 0x7fda940b3c70 2024-01-15T18:03:10.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda92000640 1 -- 172.21.15.2:0/842692585 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda84031810 con 0x7fda940b3c70 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda92000640 1 -- 172.21.15.2:0/842692585 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84031ba0 con 0x7fda940b3c70 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda99c00640 1 -- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda940b3c70 msgr2=0x7fda94070f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda99c00640 1 --2- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda940b3c70 0x7fda94070f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fda840097a0 tx=0x7fda8402b8b0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda99c00640 1 -- 172.21.15.2:0/842692585 shutdown_connections 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda99c00640 1 --2- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda99c00640 1 --2- 172.21.15.2:0/842692585 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda940b3c70 0x7fda94070f00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.574+0000 7fda99c00640 1 -- 172.21.15.2:0/842692585 >> 172.21.15.2:0/842692585 conn(0x7fda9406da20 msgr2=0x7fda9406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.575+0000 7fda99c00640 1 -- 172.21.15.2:0/842692585 shutdown_connections 2024-01-15T18:03:10.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.575+0000 7fda99c00640 1 -- 172.21.15.2:0/842692585 wait complete. 2024-01-15T18:03:10.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.577+0000 7fda99c00640 1 Processor -- start 2024-01-15T18:03:10.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.577+0000 7fda99c00640 1 -- start start 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.578+0000 7fda99c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda9406c470 0x7fda941f7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.578+0000 7fda99c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 0x7fda9415ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.578+0000 7fda99c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9415b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.578+0000 7fda99c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9415b6f0 con 0x7fda940b3c70 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.578+0000 7fda99c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda940a8570 con 0x7fda9406c470 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.578+0000 7fda99c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda940a8850 con 0x7fda94162440 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 0x7fda9415ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 0x7fda9415ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56696/0 (socket says 172.21.15.2:56696) 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 -- 172.21.15.2:0/1510248929 learned_addr learned my addr 172.21.15.2:0/1510248929 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda92a00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda9406c470 0x7fda941f7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda93400640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9415b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 -- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 msgr2=0x7fda9415b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9415b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 -- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda9406c470 msgr2=0x7fda941f7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda9406c470 0x7fda941f7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda8be00640 1 -- 172.21.15.2:0/1510248929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84009400 con 0x7fda940b3c70 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.579+0000 7fda93400640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9415b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:10.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.580+0000 7fda8be00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 0x7fda9415ac70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c480 tx=0x7fda7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.582+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c005ce0 con 0x7fda940b3c70 2024-01-15T18:03:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.582+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c005e40 con 0x7fda940b3c70 2024-01-15T18:03:10.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.582+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c0072f0 con 0x7fda940b3c70 2024-01-15T18:03:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.582+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda940a8b30 con 0x7fda940b3c70 2024-01-15T18:03:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.582+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda940a90f0 con 0x7fda940b3c70 2024-01-15T18:03:10.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.583+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda940686f0 con 0x7fda940b3c70 2024-01-15T18:03:10.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.584+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fda7c007510 con 0x7fda940b3c70 2024-01-15T18:03:10.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.585+0000 7fda8aa00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6c06b860 0x7fda6c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:10.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.585+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fda7c089b50 con 0x7fda940b3c70 2024-01-15T18:03:10.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.586+0000 7fda92a00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6c06b860 0x7fda6c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:10.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.587+0000 7fda92a00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6c06b860 0x7fda6c06dd20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fda84036000 tx=0x7fda8403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:10.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.590+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda7c054210 con 0x7fda940b3c70 2024-01-15T18:03:10.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.820+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fda94163070 con 0x7fda940b3c70 2024-01-15T18:03:10.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.821+0000 7fda8aa00640 1 -- 172.21.15.2:0/1510248929 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7fda7c017070 con 0x7fda940b3c70 2024-01-15T18:03:10.825 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6c06b860 msgr2=0x7fda6c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6c06b860 0x7fda6c06dd20 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fda84036000 tx=0x7fda8403c040 comp rx=0 tx=0).stop 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 msgr2=0x7fda9415ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 0x7fda9415ac70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00c480 tx=0x7fda7c00c940 comp rx=0 tx=0).stop 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 shutdown_connections 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6c06b860 0x7fda6c06dd20 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda94162440 0x7fda9415b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda940b3c70 0x7fda9415ac70 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 --2- 172.21.15.2:0/1510248929 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda9406c470 0x7fda941f7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 >> 172.21.15.2:0/1510248929 conn(0x7fda9406da20 msgr2=0x7fda9415f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 shutdown_connections 2024-01-15T18:03:10.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:10.824+0000 7fda99c00640 1 -- 172.21.15.2:0/1510248929 wait complete. 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:03:10.848 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:03:10.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:10.849 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T18:03:10.902 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:03:10.902 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:03:10.902 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:03:10.902 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.902 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T18:03:10.902 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T18:03:10.903 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42291sec preferred_lft 42291sec 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.904 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T18:03:10.905 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:10.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:10.906 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:03:10.994 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T18:03:10.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:11.000 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T18:03:11.054 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T18:03:11.055 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:03:11.055 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T18:03:11.055 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T18:03:11.055 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:03:11.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:11.056 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T18:03:11.056 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T18:03:11.056 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:03:11.056 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:03:11.056 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:03:11.057 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T18:03:11.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:11.254 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:03:11.311 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:03:11.311 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:03:11.311 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:03:11.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:11.311 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T18:03:11.413 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:03:11.413 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:03:11.413 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:03:11.469 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:03:11.469 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:03:11.469 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:03:11.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:03:11.628 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:03:11.686 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:03:11.686 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:03:11.686 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:03:11.687 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:03:11.688 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:03:11.689 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:03:11.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:11.691 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:03:11.746 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:03:11.746 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:03:11.754 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:03:11.834 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:03:11.878 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:03:11.878 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:03:11.879 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:03:11.880 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:03:11.881 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:03:11.882 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:03:11.882 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:03:11.882 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:03:11.882 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:03:11.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:11.883 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:03:11.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T18:03:11.943+0000 7f20d72bb480 -1 init, newargv = 0x5651186fbc10 newargc=15 2024-01-15T18:03:11.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[56990]: starting ceph client 2024-01-15T18:03:11.956 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[56990]: starting fuse 2024-01-15T18:03:11.971 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T18:03:11.971 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T18:03:11.971 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T18:03:11.972 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:03:11.973 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T18:03:11.974 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T18:03:12.171 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.56990.asok 2024-01-15T18:03:12.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:03:12.189 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56990.asok 2024-01-15T18:03:12.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:12.190 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56990.asok status 2024-01-15T18:03:12.437 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "56990", 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T18:03:12.438 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 6465, 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 6465 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:03:12.439 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2326950611 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2326950611 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:03:12.440 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.6465 192.168.0.1:0/2326950611", 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/2326950611", 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 36, 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 61, 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T18:03:12.441 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:03:12.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:12.453 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T18:03:12.454 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:03:12.511 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T18:03:12.511 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:03:12.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:12.511 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:03:12.621 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:03:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.837+0000 7fb91a000640 1 Processor -- start 2024-01-15T18:03:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.838+0000 7fb91a000640 1 -- start start 2024-01-15T18:03:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.839+0000 7fb91a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb91415ec60 0x7fb91415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.839+0000 7fb91a000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140716d0 con 0x7fb91406cad0 2024-01-15T18:03:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.839+0000 7fb91a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb914071960 con 0x7fb914141880 2024-01-15T18:03:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.839+0000 7fb91a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb914071f20 con 0x7fb91415ec60 2024-01-15T18:03:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.840+0000 7fb919600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb91406cad0 0x7fb91406cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38128/0 (socket says 172.21.15.2:38128) 2024-01-15T18:03:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.840+0000 7fb919600640 1 -- 172.21.15.2:0/2989165453 learned_addr learned my addr 172.21.15.2:0/2989165453 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:12.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.840+0000 7fb90be00640 1 --2- 172.21.15.2:0/2989165453 >> v2:172.21.15.137:3300/0 conn(0x7fb91415ec60 0x7fb91415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.840+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539944089 0 0) 0x7fb9140716d0 con 0x7fb91406cad0 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.840+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f4003520 con 0x7fb91406cad0 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459689086 0 0) 0x7fb8f4003520 con 0x7fb91406cad0 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 >> v2:172.21.15.137:3300/0 conn(0x7fb91415ec60 msgr2=0x7fb91415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 --2- 172.21.15.2:0/2989165453 >> v2:172.21.15.137:3300/0 conn(0x7fb91415ec60 0x7fb91415f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 >> v1:172.21.15.137:6789/0 conn(0x7fb914141880 legacy=0x7fb914070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb914071c40 con 0x7fb91406cad0 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb90c0019f0 con 0x7fb91406cad0 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 >> v1:172.21.15.2:6789/0 conn(0x7fb91406cad0 legacy=0x7fb91406cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40039c0 0x7fb8f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8f40047c0 0x7fb8f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8f40051d0 0x7fb8f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914071f20 con 0x7fb8f40051d0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f400c120 con 0x7fb8f40039c0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f400c2e0 con 0x7fb8f40047c0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.841+0000 7fb919600640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8f40051d0 0x7fb8f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb90be00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8f40047c0 0x7fb8f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb919600640 1 -- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8f40047c0 msgr2=0x7fb8f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb918c00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40039c0 0x7fb8f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb919600640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8f40047c0 0x7fb8f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb919600640 1 -- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40039c0 msgr2=0x7fb8f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb919600640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40039c0 0x7fb8f4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb919600640 1 -- 172.21.15.2:0/2989165453 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4003520 con 0x7fb8f40051d0 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb918c00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40039c0 0x7fb8f4003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.842+0000 7fb90be00640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb8f40047c0 0x7fb8f4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:12.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.843+0000 7fb919600640 1 -- 172.21.15.2:0/2989165453 reap_dead start 2024-01-15T18:03:12.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.843+0000 7fb919600640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8f40051d0 0x7fb8f400b980 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb90c002230 tx=0x7fb8f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101460f91383c51f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:12.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.843+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c034630 con 0x7fb8f40051d0 2024-01-15T18:03:12.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.843+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90c02f070 con 0x7fb8f40051d0 2024-01-15T18:03:12.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.843+0000 7fb913e00640 1 -- 172.21.15.2:0/2989165453 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90c0349d0 con 0x7fb8f40051d0 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 -- 172.21.15.2:0/2989165453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8f40051d0 msgr2=0x7fb8f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8f40051d0 0x7fb8f400b980 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fb90c002230 tx=0x7fb8f40057f0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 -- 172.21.15.2:0/2989165453 shutdown_connections 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 --2- 172.21.15.2:0/2989165453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb8f40051d0 0x7fb8f400b980 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 -- 172.21.15.2:0/2989165453 >> 172.21.15.2:0/2989165453 conn(0x7fb9140b0840 msgr2=0x7fb9140b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 -- 172.21.15.2:0/2989165453 shutdown_connections 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.844+0000 7fb91a000640 1 -- 172.21.15.2:0/2989165453 wait complete. 2024-01-15T18:03:12.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.846+0000 7fb91a000640 1 Processor -- start 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.846+0000 7fb91a000640 1 -- start start 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb91a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40051d0 0x7fb9140b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb91a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 0x7fb914143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb91a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb914153ab0 0x7fb9140b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb91a000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91406c6b0 con 0x7fb9141409f0 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb91a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141556b0 con 0x7fb8f40051d0 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb91a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141f6950 con 0x7fb914153ab0 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.847+0000 7fb90be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 0x7fb914143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 0x7fb914143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56724/0 (socket says 172.21.15.2:56724) 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 -- 172.21.15.2:0/1448897593 learned_addr learned my addr 172.21.15.2:0/1448897593 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb919600640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb914153ab0 0x7fb9140b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 -- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb914153ab0 msgr2=0x7fb9140b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb918c00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40051d0 0x7fb9140b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb914153ab0 0x7fb9140b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 -- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40051d0 msgr2=0x7fb9140b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40051d0 0x7fb9140b1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 -- 172.21.15.2:0/1448897593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9141f6c30 con 0x7fb9141409f0 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb918c00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40051d0 0x7fb9140b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.848+0000 7fb90be00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 0x7fb914143600 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0108b0 tx=0x7fb8fc010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.849+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc00eb60 con 0x7fb9141409f0 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.849+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8fc00ecc0 con 0x7fb9141409f0 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.849+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8fc01c400 con 0x7fb9141409f0 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.849+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9141f6f10 con 0x7fb9141409f0 2024-01-15T18:03:12.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.849+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb914143b40 con 0x7fb9141409f0 2024-01-15T18:03:12.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.850+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb914155870 con 0x7fb9141409f0 2024-01-15T18:03:12.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.851+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb8fc01c560 con 0x7fb9141409f0 2024-01-15T18:03:12.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.852+0000 7fb911600640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb8ec06b860 0x7fb8ec06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:12.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.852+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb8fc08ad00 con 0x7fb9141409f0 2024-01-15T18:03:12.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.853+0000 7fb918c00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb8ec06b860 0x7fb8ec06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:12.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.854+0000 7fb918c00640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb8ec06b860 0x7fb8ec06dd20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb904001690 tx=0x7fb904009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:12.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:12.857+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb8fc055330 con 0x7fb9141409f0 2024-01-15T18:03:13.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.091+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb914074770 con 0x7fb9141409f0 2024-01-15T18:03:13.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.092+0000 7fb911600640 1 -- 172.21.15.2:0/1448897593 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v36) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb8fc058950 con 0x7fb9141409f0 2024-01-15T18:03:13.095 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:03:13.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.094+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb8ec06b860 msgr2=0x7fb8ec06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.094+0000 7fb91a000640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb8ec06b860 0x7fb8ec06dd20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fb904001690 tx=0x7fb904009040 comp rx=0 tx=0).stop 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.094+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 msgr2=0x7fb914143600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.094+0000 7fb91a000640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 0x7fb914143600 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fb8fc0108b0 tx=0x7fb8fc010d70 comp rx=0 tx=0).stop 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 shutdown_connections 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb8ec06b860 0x7fb8ec06dd20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb914153ab0 0x7fb9140b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb9141409f0 0x7fb914143600 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 --2- 172.21.15.2:0/1448897593 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb8f40051d0 0x7fb9140b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 >> 172.21.15.2:0/1448897593 conn(0x7fb9140b0840 msgr2=0x7fb9140cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 shutdown_connections 2024-01-15T18:03:13.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:13.095+0000 7fb91a000640 1 -- 172.21.15.2:0/1448897593 wait complete. 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:03:13.116 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:03:13.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.117 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T18:03:13.175 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:03:13.175 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:03:13.175 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:03:13.175 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.175 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T18:03:13.175 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:13.176 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42277sec preferred_lft 42277sec 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.177 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T18:03:13.178 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:03:13.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.179 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:03:13.271 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T18:03:13.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.276 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T18:03:13.330 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T18:03:13.330 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:03:13.330 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:03:13.330 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:03:13.330 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:03:13.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.331 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T18:03:13.331 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T18:03:13.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:03:13.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:03:13.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:03:13.331 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T18:03:13.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.514 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:03:13.560 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:03:13.561 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:03:13.562 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:03:13.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T18:03:13.701 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:03:13.702 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:03:13.702 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:03:13.756 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:03:13.756 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:03:13.756 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:03:13.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:03:13.901 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:03:13.958 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:03:13.958 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:03:13.958 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:03:13.958 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:03:13.958 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:03:13.959 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:03:13.960 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:03:13.961 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:03:13.962 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:03:13.962 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:03:13.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:13.963 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:03:14.015 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:03:14.015 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:03:14.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:03:14.098 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:03:14.139 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:03:14.139 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:03:14.139 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:03:14.139 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:03:14.139 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:03:14.140 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:03:14.141 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:03:14.142 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:03:14.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:14.144 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:03:14.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T18:03:14.194+0000 7f8051c0c480 -1 init, newargv = 0x56273ea32c00 newargc=15 2024-01-15T18:03:14.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[44205]: starting ceph client 2024-01-15T18:03:14.209 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[44205]: starting fuse 2024-01-15T18:03:14.225 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T18:03:14.226 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:03:14.226 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:03:14.227 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T18:03:14.228 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T18:03:14.228 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T18:03:14.228 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:03:14.228 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:03:14.228 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T18:03:14.228 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T18:03:14.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.44205.asok 2024-01-15T18:03:14.429 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T18:03:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:03:14.443 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44205.asok 2024-01-15T18:03:14.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:14.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44205.asok status 2024-01-15T18:03:14.677 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T18:03:14.677 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T18:03:14.677 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:03:14.677 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:03:14.677 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T18:03:14.677 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "44205", 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5692, 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T18:03:14.678 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5692 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2007617815 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:03:14.679 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2007617815 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5692 192.168.0.1:0/2007617815", 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/2007617815", 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 36, 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 61, 2024-01-15T18:03:14.680 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:03:14.681 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T18:03:14.681 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T18:03:14.681 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T18:03:14.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:14.692 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T18:03:14.692 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:03:14.744 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T18:03:14.745 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:03:14.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:14.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:03:14.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:03:15.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.068+0000 7f376be00640 1 Processor -- start 2024-01-15T18:03:15.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.069+0000 7f376be00640 1 -- start start 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.070+0000 7f376be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f376c0b3cb0 0x7f376c0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.070+0000 7f376be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f376c0b37e0 0x7f376c0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.070+0000 7f376be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376c0d3bd0 con 0x7f376c0cfa50 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.071+0000 7f376be00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376c0d3eb0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.071+0000 7f376be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376c0d18a0 con 0x7f376c0b37e0 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.071+0000 7f376a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f376c0b3cb0 0x7f376c0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.071+0000 7f376a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f376c0b3cb0 0x7f376c0cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33362/0 (socket says 172.21.15.2:33362) 2024-01-15T18:03:15.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.071+0000 7f376a000640 1 -- 172.21.15.2:0/2285274432 learned_addr learned my addr 172.21.15.2:0/2285274432 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:15.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.071+0000 7f376b400640 1 --2- 172.21.15.2:0/2285274432 >> v2:172.21.15.137:3300/0 conn(0x7f376c0b37e0 0x7f376c0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.072+0000 7f376a000640 1 -- 172.21.15.2:0/2285274432 >> v2:172.21.15.137:3300/0 conn(0x7f376c0b37e0 msgr2=0x7f376c0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:15.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.072+0000 7f376a000640 1 --2- 172.21.15.2:0/2285274432 >> v2:172.21.15.137:3300/0 conn(0x7f376c0b37e0 0x7f376c0ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.072+0000 7f376a000640 1 -- 172.21.15.2:0/2285274432 >> v1:172.21.15.137:6789/0 conn(0x7f376c0cfa50 legacy=0x7f376c0b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:15.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.072+0000 7f376a000640 1 -- 172.21.15.2:0/2285274432 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376c0d1b80 con 0x7f376c0b3cb0 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.073+0000 7f376a000640 1 --2- 172.21.15.2:0/2285274432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0cdec0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f376c0d4190 tx=0x7f375402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebd762b76403219 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.073+0000 7f3769600640 1 -- 172.21.15.2:0/2285274432 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754031870 con 0x7f376c0b3cb0 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.073+0000 7f3769600640 1 -- 172.21.15.2:0/2285274432 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37540319d0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.073+0000 7f3769600640 1 -- 172.21.15.2:0/2285274432 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3754031d60 con 0x7f376c0b3cb0 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 -- 172.21.15.2:0/2285274432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 msgr2=0x7f376c0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 --2- 172.21.15.2:0/2285274432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0cdec0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f376c0d4190 tx=0x7f375402b910 comp rx=0 tx=0).stop 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 -- 172.21.15.2:0/2285274432 shutdown_connections 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 --2- 172.21.15.2:0/2285274432 >> v2:172.21.15.137:3300/0 conn(0x7f376c0b37e0 0x7f376c0ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 --2- 172.21.15.2:0/2285274432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0cdec0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 -- 172.21.15.2:0/2285274432 >> 172.21.15.2:0/2285274432 conn(0x7f376c0d30b0 msgr2=0x7f376c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:15.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 -- 172.21.15.2:0/2285274432 shutdown_connections 2024-01-15T18:03:15.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.074+0000 7f376be00640 1 -- 172.21.15.2:0/2285274432 wait complete. 2024-01-15T18:03:15.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.076+0000 7f376be00640 1 Processor -- start 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.076+0000 7f376be00640 1 -- start start 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f376c0b37e0 0x7f376c1f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f376c0cfa50 0x7f376c0b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c09fd60 con 0x7f376c0b3cb0 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376c09f0e0 con 0x7f376c0b37e0 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376c09f3c0 con 0x7f376c0cfa50 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0b4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33372/0 (socket says 172.21.15.2:33372) 2024-01-15T18:03:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 -- 172.21.15.2:0/4212932602 learned_addr learned my addr 172.21.15.2:0/4212932602 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376aa00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f376c0b37e0 0x7f376c1f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 -- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f376c0cfa50 msgr2=0x7f376c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f376c0cfa50 0x7f376c0b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 -- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f376c0b37e0 msgr2=0x7f376c1f5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f376c0b37e0 0x7f376c1f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.077+0000 7f376a000640 1 -- 172.21.15.2:0/4212932602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540094d0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.078+0000 7f376a000640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0b4b30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f375402b950 tx=0x7f37540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.078+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375403ad30 con 0x7f376c0b3cb0 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.078+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37540443f0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.078+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37540447a0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.078+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376c09f6a0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.078+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376c0b5070 con 0x7f376c0b3cb0 2024-01-15T18:03:15.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.080+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376c09fea0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.081+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f375404d450 con 0x7f376c0b3cb0 2024-01-15T18:03:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.082+0000 7f3762a00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f374406b8b0 0x7f374406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.082+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f375402d040 con 0x7f376c0b3cb0 2024-01-15T18:03:15.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.083+0000 7f376aa00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f374406b8b0 0x7f374406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.084+0000 7f376aa00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f374406b8b0 0x7f374406dd70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f375c00a750 tx=0x7f375c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:15.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.087+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3754077290 con 0x7f376c0b3cb0 2024-01-15T18:03:15.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.315+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f376c09fea0 con 0x7f376c0b3cb0 2024-01-15T18:03:15.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.370+0000 7f3762a00640 1 -- 172.21.15.2:0/4212932602 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 136+0+0 (secure 0 0 0) 0x7f3754035020 con 0x7f376c0b3cb0 2024-01-15T18:03:15.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f374406b8b0 msgr2=0x7f374406dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f374406b8b0 0x7f374406dd70 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f375c00a750 tx=0x7f375c008040 comp rx=0 tx=0).stop 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 msgr2=0x7f376c0b4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0b4b30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f375402b950 tx=0x7f37540046c0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 shutdown_connections 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f374406b8b0 0x7f374406dd70 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f376c0cfa50 0x7f376c0b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f376c0b3cb0 0x7f376c0b4b30 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 --2- 172.21.15.2:0/4212932602 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f376c0b37e0 0x7f376c1f5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 >> 172.21.15.2:0/4212932602 conn(0x7f376c0d30b0 msgr2=0x7f376c0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:15.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 shutdown_connections 2024-01-15T18:03:15.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.373+0000 7f376be00640 1 -- 172.21.15.2:0/4212932602 wait complete. 2024-01-15T18:03:15.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:03:15.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.619+0000 7f6af5600640 1 Processor -- start 2024-01-15T18:03:15.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.619+0000 7f6af5600640 1 -- start start 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.620+0000 7f6af5600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6af00d2c40 0x7f6af00b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.620+0000 7f6af5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00b6180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.620+0000 7f6af5600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af00d18f0 con 0x7f6af015d540 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.620+0000 7f6af5600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af00b5600 con 0x7f6af00d2c40 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.620+0000 7f6af5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00d1ba0 con 0x7f6af00b5180 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6af00d2c40 0x7f6af00b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6af00d2c40 0x7f6af00b4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33384/0 (socket says 172.21.15.2:33384) 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aeea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6af015d540 0x7f6af00d11e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33686/0 (socket says 172.21.15.2:33686) 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 -- 172.21.15.2:0/3978959769 learned_addr learned my addr 172.21.15.2:0/3978959769 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:15.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aef400640 1 --2- 172.21.15.2:0/3978959769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00b6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 -- 172.21.15.2:0/3978959769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 msgr2=0x7f6af00b6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:15.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 --2- 172.21.15.2:0/3978959769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00b6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 -- 172.21.15.2:0/3978959769 >> v1:172.21.15.2:6789/0 conn(0x7f6af015d540 legacy=0x7f6af00d11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.621+0000 7f6aee000640 1 -- 172.21.15.2:0/3978959769 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af00d1e80 con 0x7f6af00d2c40 2024-01-15T18:03:15.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.623+0000 7f6aee000640 1 --2- 172.21.15.2:0/3978959769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00b4c40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f698a09f6253dbb0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:15.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.623+0000 7f6aed600640 1 -- 172.21.15.2:0/3978959769 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802f830 con 0x7f6af00d2c40 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.623+0000 7f6aed600640 1 -- 172.21.15.2:0/3978959769 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad802f990 con 0x7f6af00d2c40 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.623+0000 7f6aed600640 1 -- 172.21.15.2:0/3978959769 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802fd20 con 0x7f6af00d2c40 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.623+0000 7f6af5600640 1 -- 172.21.15.2:0/3978959769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 msgr2=0x7f6af00b4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.623+0000 7f6af5600640 1 --2- 172.21.15.2:0/3978959769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00b4c40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009870 tx=0x7f6ad802ba10 comp rx=0 tx=0).stop 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.624+0000 7f6af5600640 1 -- 172.21.15.2:0/3978959769 shutdown_connections 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.624+0000 7f6af5600640 1 --2- 172.21.15.2:0/3978959769 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00b6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.624+0000 7f6af5600640 1 --2- 172.21.15.2:0/3978959769 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00b4c40 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.624+0000 7f6af5600640 1 -- 172.21.15.2:0/3978959769 >> 172.21.15.2:0/3978959769 conn(0x7f6af0072210 msgr2=0x7f6af00cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.624+0000 7f6af5600640 1 -- 172.21.15.2:0/3978959769 shutdown_connections 2024-01-15T18:03:15.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.624+0000 7f6af5600640 1 -- 172.21.15.2:0/3978959769 wait complete. 2024-01-15T18:03:15.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.625+0000 7f6af5600640 1 Processor -- start 2024-01-15T18:03:15.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.626+0000 7f6af5600640 1 -- start start 2024-01-15T18:03:15.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.626+0000 7f6af5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00c9730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.626+0000 7f6af5600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.626+0000 7f6af5600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6af015d540 0x7f6af01f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6af5600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af015d920 con 0x7f6af00d2c40 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6af5600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00cc190 con 0x7f6af015d540 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6af5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af00cc420 con 0x7f6af00b5180 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33396/0 (socket says 172.21.15.2:33396) 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 -- 172.21.15.2:0/4224277997 learned_addr learned my addr 172.21.15.2:0/4224277997 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aeea00640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 -- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 msgr2=0x7f6af00c9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00c9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 -- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6af015d540 msgr2=0x7f6af01f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6af015d540 0x7f6af01f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.627+0000 7f6aee000640 1 -- 172.21.15.2:0/4224277997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad80094d0 con 0x7f6af00d2c40 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.628+0000 7f6aee000640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00cbc50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.628+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad802f800 con 0x7f6af00d2c40 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.628+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad8041400 con 0x7f6af00d2c40 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.628+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad80417d0 con 0x7f6af00d2c40 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.628+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af00cc700 con 0x7f6af00d2c40 2024-01-15T18:03:15.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.628+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af00cccc0 con 0x7f6af00d2c40 2024-01-15T18:03:15.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.629+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af00686f0 con 0x7f6af00d2c40 2024-01-15T18:03:15.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.630+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6ad8041930 con 0x7f6af00d2c40 2024-01-15T18:03:15.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.631+0000 7f6ae6a00640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ac806b860 0x7f6ac806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:15.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.632+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6ad803c070 con 0x7f6af00d2c40 2024-01-15T18:03:15.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.632+0000 7f6aeea00640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ac806b860 0x7f6ac806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:15.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.633+0000 7f6aeea00640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ac806b860 0x7f6ac806dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00012e0 tx=0x7f6ae0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:15.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.636+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ad8077b90 con 0x7f6af00d2c40 2024-01-15T18:03:15.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.857+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6af00d2720 con 0x7f6ac806b860 2024-01-15T18:03:15.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.860+0000 7f6ae6a00640 1 -- 172.21.15.2:0/4224277997 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6ae0005b90 con 0x7f6ac806b860 2024-01-15T18:03:15.864 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:03:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:03:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:03:15.864 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:03:15.864 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:03:15.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ac806b860 msgr2=0x7f6ac806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ac806b860 0x7f6ac806dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00012e0 tx=0x7f6ae0009040 comp rx=0 tx=0).stop 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 msgr2=0x7f6af00cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00cbc50 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8000c00 tx=0x7f6ad8004730 comp rx=0 tx=0).stop 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 shutdown_connections 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6ac806b860 0x7f6ac806dd20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6af015d540 0x7f6af01f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6af00d2c40 0x7f6af00cbc50 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 --2- 172.21.15.2:0/4224277997 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6af00b5180 0x7f6af00c9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:15.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 >> 172.21.15.2:0/4224277997 conn(0x7f6af0072210 msgr2=0x7f6af00b6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:15.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 shutdown_connections 2024-01-15T18:03:15.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:15.863+0000 7f6af5600640 1 -- 172.21.15.2:0/4224277997 wait complete. 2024-01-15T18:03:15.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:03:16.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.110+0000 7f6329000640 1 Processor -- start 2024-01-15T18:03:16.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.110+0000 7f6329000640 1 -- start start 2024-01-15T18:03:16.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.111+0000 7f6329000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 0x7f6324071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.111+0000 7f6329000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f632406da80 0x7f632406df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.111+0000 7f6329000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632406c500 con 0x7f632415eab0 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.111+0000 7f6329000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632415c4b0 con 0x7f632406da80 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.111+0000 7f6329000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240714d0 con 0x7f63240ae260 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.112+0000 7f6323400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 0x7f6324071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.112+0000 7f631be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f632406da80 0x7f632406df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.112+0000 7f6323400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 0x7f6324071ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:42384/0 (socket says 172.21.15.2:42384) 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.112+0000 7f631be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f632406da80 0x7f632406df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:57954/0 (socket says 172.21.15.2:57954) 2024-01-15T18:03:16.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.112+0000 7f6323400640 1 -- 172.21.15.2:0/659877291 learned_addr learned my addr 172.21.15.2:0/659877291 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:16.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.113+0000 7f631be00640 1 -- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 msgr2=0x7f6324071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:16.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.113+0000 7f631be00640 1 --2- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 0x7f6324071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.113+0000 7f631be00640 1 -- 172.21.15.2:0/659877291 >> v1:172.21.15.137:6789/0 conn(0x7f632415eab0 legacy=0x7f632415ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.113+0000 7f631be00640 1 -- 172.21.15.2:0/659877291 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632415b8a0 con 0x7f632406da80 2024-01-15T18:03:16.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.113+0000 7f6323400640 1 --2- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 0x7f6324071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:03:16.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.115+0000 7f631be00640 1 --2- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632406da80 0x7f632406df00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f630c0097a0 tx=0x7f630c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98acb56602c209a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:16.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.115+0000 7f6322a00640 1 -- 172.21.15.2:0/659877291 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c031780 con 0x7f632406da80 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.115+0000 7f6322a00640 1 -- 172.21.15.2:0/659877291 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c0318e0 con 0x7f632406da80 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.115+0000 7f6322a00640 1 -- 172.21.15.2:0/659877291 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c031c70 con 0x7f632406da80 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 -- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632406da80 msgr2=0x7f632406df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 --2- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632406da80 0x7f632406df00 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f630c0097a0 tx=0x7f630c02b820 comp rx=0 tx=0).stop 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 -- 172.21.15.2:0/659877291 shutdown_connections 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 --2- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632406da80 0x7f632406df00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 --2- 172.21.15.2:0/659877291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63240ae260 0x7f6324071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 -- 172.21.15.2:0/659877291 >> 172.21.15.2:0/659877291 conn(0x7f63240cebc0 msgr2=0x7f63240cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:16.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 -- 172.21.15.2:0/659877291 shutdown_connections 2024-01-15T18:03:16.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.116+0000 7f6329000640 1 -- 172.21.15.2:0/659877291 wait complete. 2024-01-15T18:03:16.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.117+0000 7f6329000640 1 Processor -- start 2024-01-15T18:03:16.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.118+0000 7f6329000640 1 -- start start 2024-01-15T18:03:16.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6329000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 0x7f6324149810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6329000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63240ae260 0x7f6324157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6329000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632415eab0 0x7f6324157650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6329000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63240b2480 con 0x7f632406da80 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6329000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324155210 con 0x7f63240ae260 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6329000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240b1ae0 con 0x7f632415eab0 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 0x7f6324149810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 0x7f6324149810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33398/0 (socket says 172.21.15.2:33398) 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 -- 172.21.15.2:0/2785746049 learned_addr learned my addr 172.21.15.2:0/2785746049 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 -- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632415eab0 msgr2=0x7f6324157650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f631be00640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63240ae260 0x7f6324157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323e00640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632415eab0 0x7f6324157650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632415eab0 0x7f6324157650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 -- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63240ae260 msgr2=0x7f6324157110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63240ae260 0x7f6324157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.119+0000 7f6323400640 1 -- 172.21.15.2:0/2785746049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63140094d0 con 0x7f632406da80 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.120+0000 7f6323400640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 0x7f6324149810 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f631400e9d0 tx=0x7f631400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.120+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631400ca60 con 0x7f632406da80 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.120+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631400cbc0 con 0x7f632406da80 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.120+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314013470 con 0x7f632406da80 2024-01-15T18:03:16.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.120+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630c009400 con 0x7f632406da80 2024-01-15T18:03:16.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.120+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240b20a0 con 0x7f632406da80 2024-01-15T18:03:16.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.121+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632406d610 con 0x7f632406da80 2024-01-15T18:03:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.122+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f63140135d0 con 0x7f632406da80 2024-01-15T18:03:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.123+0000 7f631b400640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62fc06b860 0x7f62fc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.123+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f631408a9a0 con 0x7f632406da80 2024-01-15T18:03:16.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.124+0000 7f631be00640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62fc06b860 0x7f62fc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.125+0000 7f631be00640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62fc06b860 0x7f62fc06dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f630c036000 tx=0x7f630c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:16.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.128+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6314054fb0 con 0x7f632406da80 2024-01-15T18:03:16.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.313+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6324074770 con 0x7f632406da80 2024-01-15T18:03:16.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.314+0000 7f631b400640 1 -- 172.21.15.2:0/2785746049 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f63140170c0 con 0x7f632406da80 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62fc06b860 msgr2=0x7f62fc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62fc06b860 0x7f62fc06dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f630c036000 tx=0x7f630c03d040 comp rx=0 tx=0).stop 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 msgr2=0x7f6324149810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 0x7f6324149810 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f631400e9d0 tx=0x7f631400ee90 comp rx=0 tx=0).stop 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 shutdown_connections 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62fc06b860 0x7f62fc06dd20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632415eab0 0x7f6324157650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63240ae260 0x7f6324157110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 --2- 172.21.15.2:0/2785746049 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632406da80 0x7f6324149810 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 >> 172.21.15.2:0/2785746049 conn(0x7f63240cebc0 msgr2=0x7f63240719b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:16.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 shutdown_connections 2024-01-15T18:03:16.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.316+0000 7f6329000640 1 -- 172.21.15.2:0/2785746049 wait complete. 2024-01-15T18:03:16.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:03:16.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.561+0000 7f5f1e800640 1 Processor -- start 2024-01-15T18:03:16.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.561+0000 7f5f1e800640 1 -- start start 2024-01-15T18:03:16.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f1e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5f1806d410 0x7f5f180cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f1e800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f180cd9f0 con 0x7f5f1806c920 2024-01-15T18:03:16.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f1e800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1806aec0 con 0x7f5f180cd2c0 2024-01-15T18:03:16.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f1e800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1806eb20 con 0x7f5f1806d410 2024-01-15T18:03:16.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f17400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5f1806c920 0x7f5f1806cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33688/0 (socket says 172.21.15.2:33688) 2024-01-15T18:03:16.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f17400640 1 -- 172.21.15.2:0/3914839940 learned_addr learned my addr 172.21.15.2:0/3914839940 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:03:16.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.563+0000 7f5f16a00640 1 --2- 172.21.15.2:0/3914839940 >> v2:172.21.15.2:3300/0 conn(0x7f5f1806d410 0x7f5f180cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.564+0000 7f5f16a00640 1 -- 172.21.15.2:0/3914839940 >> v1:172.21.15.137:6789/0 conn(0x7f5f180cd2c0 legacy=0x7f5f18166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:16.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.564+0000 7f5f16a00640 1 -- 172.21.15.2:0/3914839940 >> v1:172.21.15.2:6789/0 conn(0x7f5f1806c920 legacy=0x7f5f1806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.564+0000 7f5f16a00640 1 -- 172.21.15.2:0/3914839940 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1806b1a0 con 0x7f5f1806d410 2024-01-15T18:03:16.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.564+0000 7f5f16000640 1 -- 172.21.15.2:0/3914839940 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582231052 0 0) 0x7f5f180cd9f0 con 0x7f5f1806c920 2024-01-15T18:03:16.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.565+0000 7f5f16a00640 1 --2- 172.21.15.2:0/3914839940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f180cf050 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009d90 tx=0x7f5f0002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696857da5d1fe907 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:16.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.565+0000 7f5f16000640 1 -- 172.21.15.2:0/3914839940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0002cbd0 con 0x7f5f1806d410 2024-01-15T18:03:16.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.565+0000 7f5f16000640 1 -- 172.21.15.2:0/3914839940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0002cd30 con 0x7f5f1806d410 2024-01-15T18:03:16.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.565+0000 7f5f16000640 1 -- 172.21.15.2:0/3914839940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0002cea0 con 0x7f5f1806d410 2024-01-15T18:03:16.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 -- 172.21.15.2:0/3914839940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 msgr2=0x7f5f180cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 --2- 172.21.15.2:0/3914839940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f180cf050 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f5f00009d90 tx=0x7f5f0002be20 comp rx=0 tx=0).stop 2024-01-15T18:03:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 -- 172.21.15.2:0/3914839940 shutdown_connections 2024-01-15T18:03:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 --2- 172.21.15.2:0/3914839940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f180cf050 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 -- 172.21.15.2:0/3914839940 >> 172.21.15.2:0/3914839940 conn(0x7f5f180cc170 msgr2=0x7f5f180cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 -- 172.21.15.2:0/3914839940 shutdown_connections 2024-01-15T18:03:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.566+0000 7f5f1e800640 1 -- 172.21.15.2:0/3914839940 wait complete. 2024-01-15T18:03:16.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.568+0000 7f5f1e800640 1 Processor -- start 2024-01-15T18:03:16.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.568+0000 7f5f1e800640 1 -- start start 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f1e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f1806c920 0x7f5f18145500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f1e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f18147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f1e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f180cd2c0 0x7f5f18148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f1e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18145a40 con 0x7f5f1806d410 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f1e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f181590e0 con 0x7f5f1806c920 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f1e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f181488e0 con 0x7f5f180cd2c0 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f16a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f18147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f16a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f18147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33402/0 (socket says 172.21.15.2:33402) 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.569+0000 7f5f16a00640 1 -- 172.21.15.2:0/1968055587 learned_addr learned my addr 172.21.15.2:0/1968055587 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f17400640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f1806c920 0x7f5f18145500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f16a00640 1 -- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f180cd2c0 msgr2=0x7f5f18148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f17e00640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f180cd2c0 0x7f5f18148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f16a00640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f180cd2c0 0x7f5f18148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f16a00640 1 -- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f1806c920 msgr2=0x7f5f18145500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f16a00640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f1806c920 0x7f5f18145500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f16a00640 1 -- 172.21.15.2:0/1968055587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f00009a70 con 0x7f5f1806d410 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f17e00640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f180cd2c0 0x7f5f18148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.570+0000 7f5f16a00640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f18147d20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5f00037040 tx=0x7f5f00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.571+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0002cea0 con 0x7f5f1806d410 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.571+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0003e610 con 0x7f5f1806d410 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.571+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0003e9c0 con 0x7f5f1806d410 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.571+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f18148bc0 con 0x7f5f1806d410 2024-01-15T18:03:16.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.571+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f18145fc0 con 0x7f5f1806d410 2024-01-15T18:03:16.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.572+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f180686f0 con 0x7f5f1806d410 2024-01-15T18:03:16.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.573+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f5f0003b050 con 0x7f5f1806d410 2024-01-15T18:03:16.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.574+0000 7f5f0f400640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ef006b860 0x7f5ef006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:16.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.574+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5f000ad030 con 0x7f5f1806d410 2024-01-15T18:03:16.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.575+0000 7f5f17400640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ef006b860 0x7f5ef006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:16.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.576+0000 7f5f17400640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ef006b860 0x7f5ef006dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5f08000f60 tx=0x7f5f08007c00 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:16.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.578+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f000775b0 con 0x7f5f1806d410 2024-01-15T18:03:16.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.798+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f180cd6d0 con 0x7f5ef006b860 2024-01-15T18:03:16.813 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:03:16.809+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:03:16.813 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:03:16.809+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:03:16.813 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:03:16.809+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:03:16.813 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:03:16.809+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:03:16.813 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:03:16.809+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:03:16.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.890+0000 7f5f0f400640 1 -- 172.21.15.2:0/1968055587 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5f08007270 con 0x7f5ef006b860 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ef006b860 msgr2=0x7f5ef006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ef006b860 0x7f5ef006dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5f08000f60 tx=0x7f5f08007c00 comp rx=0 tx=0).stop 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 msgr2=0x7f5f18147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f18147d20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5f00037040 tx=0x7f5f00004a40 comp rx=0 tx=0).stop 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 shutdown_connections 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ef006b860 0x7f5ef006dd20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f180cd2c0 0x7f5f18148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f1806d410 0x7f5f18147d20 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 --2- 172.21.15.2:0/1968055587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f1806c920 0x7f5f18145500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.894+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 >> 172.21.15.2:0/1968055587 conn(0x7f5f180cc170 msgr2=0x7f5f180af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.895+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 shutdown_connections 2024-01-15T18:03:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:16.895+0000 7f5f1e800640 1 -- 172.21.15.2:0/1968055587 wait complete. 2024-01-15T18:03:16.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:03:17.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.143+0000 7f7215200640 1 Processor -- start 2024-01-15T18:03:17.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.143+0000 7f7215200640 1 -- start start 2024-01-15T18:03:17.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.144+0000 7f7215200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7210160980 0x7f72100cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:17.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.144+0000 7f7215200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f72100b4040 0x7f72100cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:17.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.144+0000 7f7215200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72100ce7b0 con 0x7f72100d01a0 2024-01-15T18:03:17.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.144+0000 7f7215200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100cf010 con 0x7f72100b4040 2024-01-15T18:03:17.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.144+0000 7f7215200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72100d05b0 con 0x7f7210160980 2024-01-15T18:03:17.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f72100b4040 0x7f72100cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:17.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f72100b4040 0x7f72100cfc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33412/0 (socket says 172.21.15.2:33412) 2024-01-15T18:03:17.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720ea00640 1 -- 172.21.15.2:0/3749960162 learned_addr learned my addr 172.21.15.2:0/3749960162 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:17.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f72100d01a0 0x7f72100d0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33704/0 (socket says 172.21.15.2:33704) 2024-01-15T18:03:17.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720f400640 1 --2- 172.21.15.2:0/3749960162 >> v2:172.21.15.137:3300/0 conn(0x7f7210160980 0x7f72100cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:17.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720ea00640 1 -- 172.21.15.2:0/3749960162 >> v2:172.21.15.137:3300/0 conn(0x7f7210160980 msgr2=0x7f72100cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:03:17.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720ea00640 1 --2- 172.21.15.2:0/3749960162 >> v2:172.21.15.137:3300/0 conn(0x7f7210160980 0x7f72100cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.145+0000 7f720ea00640 1 -- 172.21.15.2:0/3749960162 >> v1:172.21.15.2:6789/0 conn(0x7f72100d01a0 legacy=0x7f72100d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:17.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.146+0000 7f720e000640 1 -- 172.21.15.2:0/3749960162 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800393521 0 0) 0x7f72100ce7b0 con 0x7f72100d01a0 2024-01-15T18:03:17.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.146+0000 7f720ea00640 1 -- 172.21.15.2:0/3749960162 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100cea50 con 0x7f72100b4040 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.147+0000 7f720ea00640 1 --2- 172.21.15.2:0/3749960162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72100b4040 0x7f72100cfc60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f72100d4060 tx=0x7f71f802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28b1ed22d001937 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.147+0000 7f720e000640 1 -- 172.21.15.2:0/3749960162 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f802cb20 con 0x7f72100b4040 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.147+0000 7f720e000640 1 -- 172.21.15.2:0/3749960162 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f802cc80 con 0x7f72100b4040 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.147+0000 7f720e000640 1 -- 172.21.15.2:0/3749960162 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80343f0 con 0x7f72100b4040 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 -- 172.21.15.2:0/3749960162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72100b4040 msgr2=0x7f72100cfc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 --2- 172.21.15.2:0/3749960162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72100b4040 0x7f72100cfc60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f72100d4060 tx=0x7f71f802bf80 comp rx=0 tx=0).stop 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 -- 172.21.15.2:0/3749960162 shutdown_connections 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 --2- 172.21.15.2:0/3749960162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72100b4040 0x7f72100cfc60 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 --2- 172.21.15.2:0/3749960162 >> v2:172.21.15.137:3300/0 conn(0x7f7210160980 0x7f72100cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 -- 172.21.15.2:0/3749960162 >> 172.21.15.2:0/3749960162 conn(0x7f72100d2ff0 msgr2=0x7f72100d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:17.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 -- 172.21.15.2:0/3749960162 shutdown_connections 2024-01-15T18:03:17.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.148+0000 7f7215200640 1 -- 172.21.15.2:0/3749960162 wait complete. 2024-01-15T18:03:17.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.150+0000 7f7215200640 1 Processor -- start 2024-01-15T18:03:17.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.150+0000 7f7215200640 1 -- start start 2024-01-15T18:03:17.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f7215200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72100b4040 0x7f72100cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f7215200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72100d01a0 0x7f7210152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f7215200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 0x7f72100b7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f7215200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100ca0c0 con 0x7f7210160980 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f7215200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72101531a0 con 0x7f72100b4040 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f7215200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72100ca310 con 0x7f72100d01a0 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 0x7f72100b7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 0x7f72100b7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33424/0 (socket says 172.21.15.2:33424) 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 -- 172.21.15.2:0/1981959883 learned_addr learned my addr 172.21.15.2:0/1981959883 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720f400640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72100b4040 0x7f72100cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:17.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 -- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72100d01a0 msgr2=0x7f7210152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72100d01a0 0x7f7210152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 -- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72100b4040 msgr2=0x7f72100cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72100b4040 0x7f72100cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.151+0000 7f720fe00640 1 -- 172.21.15.2:0/1981959883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f8009b30 con 0x7f7210160980 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.152+0000 7f720fe00640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 0x7f72100b7a20 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7208001eb0 tx=0x7f720800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.152+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72080074e0 con 0x7f7210160980 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.152+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7208007640 con 0x7f7210160980 2024-01-15T18:03:17.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.152+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208004a00 con 0x7f7210160980 2024-01-15T18:03:17.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.152+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72100ca5f0 con 0x7f7210160980 2024-01-15T18:03:17.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.152+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72100cabb0 con 0x7f7210160980 2024-01-15T18:03:17.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.153+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72100cada0 con 0x7f7210160980 2024-01-15T18:03:17.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.154+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7208004b60 con 0x7f7210160980 2024-01-15T18:03:17.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.155+0000 7f7207400640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f71e806b860 0x7f71e806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:03:17.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.155+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f720808a4c0 con 0x7f7210160980 2024-01-15T18:03:17.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.156+0000 7f720f400640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f71e806b860 0x7f71e806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:03:17.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.157+0000 7f720f400640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f71e806b860 0x7f71e806dd20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7200006fd0 tx=0x7f7200008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:03:17.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.160+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72080584d0 con 0x7f7210160980 2024-01-15T18:03:17.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.386+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f72100ca480 con 0x7f71e806b860 2024-01-15T18:03:17.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.398+0000 7f7207400640 1 -- 172.21.15.2:0/1981959883 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f72100ca480 con 0x7f71e806b860 2024-01-15T18:03:17.401 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f71e806b860 msgr2=0x7f71e806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f71e806b860 0x7f71e806dd20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7200006fd0 tx=0x7f7200008040 comp rx=0 tx=0).stop 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 msgr2=0x7f72100b7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 0x7f72100b7a20 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7208001eb0 tx=0x7f720800c870 comp rx=0 tx=0).stop 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 shutdown_connections 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f71e806b860 0x7f71e806dd20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7210160980 0x7f72100b7a20 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72100d01a0 0x7f7210152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 --2- 172.21.15.2:0/1981959883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72100b4040 0x7f72100cb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:03:17.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 >> 172.21.15.2:0/1981959883 conn(0x7f72100d2ff0 msgr2=0x7f72100cf3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:03:17.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 shutdown_connections 2024-01-15T18:03:17.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:03:17.401+0000 7f7215200640 1 -- 172.21.15.2:0/1981959883 wait complete. 2024-01-15T18:03:17.423 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6 2024-01-15T18:03:17.423 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:03:17.423 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:03:17.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:17.469 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:03:30.230 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:03:30.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:03:33.380 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:03:33.381 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:03:33.381 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.7843 s, 68.0 MB/s 2024-01-15T18:03:33.382 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:03:33.383 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:03:33.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:33.442 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:03:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:03:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:03:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.6105 s, 73.5 MB/s 2024-01-15T18:03:48.118 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:03:48.118 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:03:48.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:03:48.175 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:04:00.368 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:00.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:02.382 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:04:02.382 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:04:02.382 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.1455 s, 75.9 MB/s 2024-01-15T18:04:02.384 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:04:02.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.606+0000 7f440de00640 1 Processor -- start 2024-01-15T18:04:02.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.607+0000 7f440de00640 1 -- start start 2024-01-15T18:04:02.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.608+0000 7f440de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4408070820 0x7f4408161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:02.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.608+0000 7f440de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4408072100 0x7f440806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:02.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.608+0000 7f440de00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44081620d0 con 0x7f440814b160 2024-01-15T18:04:02.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.608+0000 7f440de00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080b35b0 con 0x7f4408072100 2024-01-15T18:04:02.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.608+0000 7f440de00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44080b1510 con 0x7f4408070820 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440ca00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f440814b160 0x7f4408070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45310/0 (socket says 172.21.15.2:45310) 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440ca00640 1 -- 172.21.15.2:0/1870143585 learned_addr learned my addr 172.21.15.2:0/1870143585 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440d400640 1 --2- 172.21.15.2:0/1870143585 >> v2:172.21.15.2:3300/0 conn(0x7f4408072100 0x7f440806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f4407e00640 1 --2- 172.21.15.2:0/1870143585 >> v2:172.21.15.137:3300/0 conn(0x7f4408070820 0x7f4408161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f4407400640 1 -- 172.21.15.2:0/1870143585 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 983414683 0 0) 0x7f44081620d0 con 0x7f440814b160 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f4407400640 1 -- 172.21.15.2:0/1870143585 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8003520 con 0x7f440814b160 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440d400640 1 -- 172.21.15.2:0/1870143585 >> v2:172.21.15.137:3300/0 conn(0x7f4408070820 msgr2=0x7f4408161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440d400640 1 --2- 172.21.15.2:0/1870143585 >> v2:172.21.15.137:3300/0 conn(0x7f4408070820 0x7f4408161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440d400640 1 -- 172.21.15.2:0/1870143585 >> v1:172.21.15.2:6789/0 conn(0x7f440814b160 legacy=0x7f4408070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:02.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.609+0000 7f440d400640 1 -- 172.21.15.2:0/1870143585 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44080b32d0 con 0x7f4408072100 2024-01-15T18:04:02.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.611+0000 7f440d400640 1 --2- 172.21.15.2:0/1870143585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4408072100 0x7f440806cdd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f440806e040 tx=0x7f440002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee64bf4a770631c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:02.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.611+0000 7f4407400640 1 -- 172.21.15.2:0/1870143585 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440002cc80 con 0x7f4408072100 2024-01-15T18:04:02.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.611+0000 7f4407400640 1 -- 172.21.15.2:0/1870143585 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440002cde0 con 0x7f4408072100 2024-01-15T18:04:02.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.611+0000 7f4407400640 1 -- 172.21.15.2:0/1870143585 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400034660 con 0x7f4408072100 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.611+0000 7f440de00640 1 -- 172.21.15.2:0/1870143585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4408072100 msgr2=0x7f440806cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.611+0000 7f440de00640 1 --2- 172.21.15.2:0/1870143585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4408072100 0x7f440806cdd0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f440806e040 tx=0x7f440002bed0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.612+0000 7f440de00640 1 -- 172.21.15.2:0/1870143585 shutdown_connections 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.612+0000 7f440de00640 1 --2- 172.21.15.2:0/1870143585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4408072100 0x7f440806cdd0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.612+0000 7f440de00640 1 --2- 172.21.15.2:0/1870143585 >> v2:172.21.15.137:3300/0 conn(0x7f4408070820 0x7f4408161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.612+0000 7f440de00640 1 -- 172.21.15.2:0/1870143585 >> 172.21.15.2:0/1870143585 conn(0x7f44080b0840 msgr2=0x7f44080b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.612+0000 7f440de00640 1 -- 172.21.15.2:0/1870143585 shutdown_connections 2024-01-15T18:04:02.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.612+0000 7f440de00640 1 -- 172.21.15.2:0/1870143585 wait complete. 2024-01-15T18:04:02.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.613+0000 7f440de00640 1 Processor -- start 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 -- start start 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4408070820 0x7f44081f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4408072100 0x7f44080a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 0x7f44080af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44081f8ea0 con 0x7f440814b160 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44081f8290 con 0x7f4408072100 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.614+0000 7f440de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44081f8570 con 0x7f4408070820 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 0x7f44080af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:02.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 0x7f44080af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47962/0 (socket says 172.21.15.2:47962) 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 -- 172.21.15.2:0/1316661710 learned_addr learned my addr 172.21.15.2:0/1316661710 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f4407e00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4408072100 0x7f44080a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 -- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4408070820 msgr2=0x7f44081f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440ca00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4408070820 0x7f44081f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4408070820 0x7f44081f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 -- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4408072100 msgr2=0x7f44080a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4408072100 0x7f44080a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f440d400640 1 -- 172.21.15.2:0/1316661710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4400009a70 con 0x7f440814b160 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.615+0000 7f4407e00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4408072100 0x7f44080a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.616+0000 7f440d400640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 0x7f44080af9a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4400009e80 tx=0x7f4400004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:02.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.616+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400040650 con 0x7f440814b160 2024-01-15T18:04:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.616+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44000407b0 con 0x7f440814b160 2024-01-15T18:04:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.616+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400040b60 con 0x7f440814b160 2024-01-15T18:04:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.616+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44081f8850 con 0x7f440814b160 2024-01-15T18:04:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.616+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44081f74a0 con 0x7f440814b160 2024-01-15T18:04:02.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.617+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44081f9010 con 0x7f440814b160 2024-01-15T18:04:02.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.618+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f4400040cc0 con 0x7f440814b160 2024-01-15T18:04:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.619+0000 7f4404c00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f43e006b920 0x7f43e006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.619+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f44000ad1a0 con 0x7f440814b160 2024-01-15T18:04:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.620+0000 7f440ca00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f43e006b920 0x7f43e006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:02.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.621+0000 7f440ca00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f43e006b920 0x7f43e006dde0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f43f80011e0 tx=0x7f43f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:02.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.624+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44000dfe90 con 0x7f440814b160 2024-01-15T18:04:02.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.816+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f4408074770 con 0x7f440814b160 2024-01-15T18:04:02.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.822+0000 7f4404c00640 1 -- 172.21.15.2:0/1316661710 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v21)=0 v21) v1 ==== 131+0+0 (secure 0 0 0) 0x7f4400077650 con 0x7f440814b160 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.825+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f43e006b920 msgr2=0x7f43e006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.825+0000 7f440de00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f43e006b920 0x7f43e006dde0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f43f80011e0 tx=0x7f43f8009040 comp rx=0 tx=0).stop 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.825+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 msgr2=0x7f44080af9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.825+0000 7f440de00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 0x7f44080af9a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f4400009e80 tx=0x7f4400004a40 comp rx=0 tx=0).stop 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 shutdown_connections 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f43e006b920 0x7f43e006dde0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f440814b160 0x7f44080af9a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4408072100 0x7f44080a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 --2- 172.21.15.2:0/1316661710 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4408070820 0x7f44081f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 >> 172.21.15.2:0/1316661710 conn(0x7f44080b0840 msgr2=0x7f440806ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:02.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 shutdown_connections 2024-01-15T18:04:02.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:02.826+0000 7f440de00640 1 -- 172.21.15.2:0/1316661710 wait complete. 2024-01-15T18:04:02.847 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:04:03.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.059+0000 7f270a200640 1 Processor -- start 2024-01-15T18:04:03.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.060+0000 7f270a200640 1 -- start start 2024-01-15T18:04:03.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.061+0000 7f270a200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2704143d20 0x7f270415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.061+0000 7f270a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f27040720d0 0x7f270415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.061+0000 7f270a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2704071090 con 0x7f27041447d0 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.061+0000 7f270a200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2704071430 con 0x7f27040720d0 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.061+0000 7f270a200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2704144100 con 0x7f2704143d20 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f27040720d0 0x7f270415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f27040720d0 0x7f270415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39024/0 (socket says 172.21.15.2:39024) 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 -- 172.21.15.2:0/97404685 learned_addr learned my addr 172.21.15.2:0/97404685 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f2708e00640 1 --2- 172.21.15.2:0/97404685 >> v2:172.21.15.137:3300/0 conn(0x7f2704143d20 0x7f270415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 -- 172.21.15.2:0/97404685 >> v2:172.21.15.137:3300/0 conn(0x7f2704143d20 msgr2=0x7f270415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:03.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 --2- 172.21.15.2:0/97404685 >> v2:172.21.15.137:3300/0 conn(0x7f2704143d20 0x7f270415eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 -- 172.21.15.2:0/97404685 >> v1:172.21.15.137:6789/0 conn(0x7f27041447d0 legacy=0x7f27040cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:03.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f26fbe00640 1 -- 172.21.15.2:0/97404685 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f270406d160 con 0x7f27040720d0 2024-01-15T18:04:03.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.062+0000 7f2708e00640 1 --2- 172.21.15.2:0/97404685 >> v2:172.21.15.137:3300/0 conn(0x7f2704143d20 0x7f270415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:03.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.063+0000 7f26fbe00640 1 --2- 172.21.15.2:0/97404685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270415f090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f270406e600 tx=0x7f26ec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db8cfaa31b058f75 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:03.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f2703e00640 1 -- 172.21.15.2:0/97404685 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec0317e0 con 0x7f27040720d0 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f2703e00640 1 -- 172.21.15.2:0/97404685 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26ec031940 con 0x7f27040720d0 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f2703e00640 1 -- 172.21.15.2:0/97404685 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26ec031cd0 con 0x7f27040720d0 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f270a200640 1 -- 172.21.15.2:0/97404685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 msgr2=0x7f270415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f270a200640 1 --2- 172.21.15.2:0/97404685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270415f090 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f270406e600 tx=0x7f26ec02b880 comp rx=0 tx=0).stop 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f270a200640 1 -- 172.21.15.2:0/97404685 shutdown_connections 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f270a200640 1 --2- 172.21.15.2:0/97404685 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270415f090 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f270a200640 1 --2- 172.21.15.2:0/97404685 >> v2:172.21.15.137:3300/0 conn(0x7f2704143d20 0x7f270415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.064+0000 7f270a200640 1 -- 172.21.15.2:0/97404685 >> 172.21.15.2:0/97404685 conn(0x7f27040b0840 msgr2=0x7f27040b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:03.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.065+0000 7f270a200640 1 -- 172.21.15.2:0/97404685 shutdown_connections 2024-01-15T18:04:03.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.065+0000 7f270a200640 1 -- 172.21.15.2:0/97404685 wait complete. 2024-01-15T18:04:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.066+0000 7f270a200640 1 Processor -- start 2024-01-15T18:04:03.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.066+0000 7f270a200640 1 -- start start 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.067+0000 7f270a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270413fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.067+0000 7f270a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2704143d20 0x7f270415a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.067+0000 7f270a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27041447d0 0x7f270415c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.067+0000 7f270a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040cbf00 con 0x7f27040720d0 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.067+0000 7f270a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27040c3ab0 con 0x7f27041447d0 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.067+0000 7f270a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f270415ac70 con 0x7f2704143d20 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270413fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270413fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39032/0 (socket says 172.21.15.2:39032) 2024-01-15T18:04:03.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 -- 172.21.15.2:0/1368216384 learned_addr learned my addr 172.21.15.2:0/1368216384 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 -- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2704143d20 msgr2=0x7f270415a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2704143d20 0x7f270415a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 -- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27041447d0 msgr2=0x7f270415c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27041447d0 0x7f270415c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.068+0000 7f2708e00640 1 -- 172.21.15.2:0/1368216384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec0094d0 con 0x7f27040720d0 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.069+0000 7f2708e00640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270413fab0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c7f0 tx=0x7f26f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.069+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f4007a30 con 0x7f27040720d0 2024-01-15T18:04:03.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.069+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f4007b90 con 0x7f27040720d0 2024-01-15T18:04:03.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.069+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f400ce40 con 0x7f27040720d0 2024-01-15T18:04:03.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.069+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27041401b0 con 0x7f27040720d0 2024-01-15T18:04:03.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.069+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2704140770 con 0x7f27040720d0 2024-01-15T18:04:03.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.070+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704074770 con 0x7f27040720d0 2024-01-15T18:04:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.071+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f26f40113f0 con 0x7f27040720d0 2024-01-15T18:04:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.072+0000 7f2701600640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26dc06b8f0 0x7f26dc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.072+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f26f408ab50 con 0x7f27040720d0 2024-01-15T18:04:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.073+0000 7f26fbe00640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26dc06b8f0 0x7f26dc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.074+0000 7f26fbe00640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26dc06b8f0 0x7f26dc06ddb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036000 tx=0x7f26ec030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:03.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.076+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26f4054e60 con 0x7f27040720d0 2024-01-15T18:04:03.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.274+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f2704071ab0 con 0x7f27040720d0 2024-01-15T18:04:03.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.279+0000 7f2701600640 1 -- 172.21.15.2:0/1368216384 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f26f401f020 con 0x7f27040720d0 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26dc06b8f0 msgr2=0x7f26dc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26dc06b8f0 0x7f26dc06ddb0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f26ec036000 tx=0x7f26ec030000 comp rx=0 tx=0).stop 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 msgr2=0x7f270413fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270413fab0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f26f400c7f0 tx=0x7f26f400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 shutdown_connections 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26dc06b8f0 0x7f26dc06ddb0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27041447d0 0x7f270415c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2704143d20 0x7f270415a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 --2- 172.21.15.2:0/1368216384 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27040720d0 0x7f270413fab0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 >> 172.21.15.2:0/1368216384 conn(0x7f27040b0840 msgr2=0x7f27041414d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 shutdown_connections 2024-01-15T18:04:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.283+0000 7f270a200640 1 -- 172.21.15.2:0/1368216384 wait complete. 2024-01-15T18:04:03.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:04:03.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.529+0000 7f06cf000640 1 Processor -- start 2024-01-15T18:04:03.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.529+0000 7f06cf000640 1 -- start start 2024-01-15T18:04:03.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06cf000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f06c80d2e10 0x7f06c80d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06cf000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f06c80d17f0 0x7f06c80b62e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06cf000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c815d8b0 con 0x7f06c80b4cf0 2024-01-15T18:04:03.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06cf000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80d2980 con 0x7f06c80d2e10 2024-01-15T18:04:03.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06cf000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c80d1d70 con 0x7f06c80d17f0 2024-01-15T18:04:03.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f06c80d2e10 0x7f06c80d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f06c80d2e10 0x7f06c80d12b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39048/0 (socket says 172.21.15.2:39048) 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7400640 1 -- 172.21.15.2:0/158819850 learned_addr learned my addr 172.21.15.2:0/158819850 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7e00640 1 --1- 172.21.15.2:0/158819850 >> v1:172.21.15.2:6789/0 conn(0x7f06c80b4cf0 0x7f06c80b50d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41634/0 (socket says 172.21.15.2:41634) 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7400640 1 -- 172.21.15.2:0/158819850 >> v2:172.21.15.137:3300/0 conn(0x7f06c80d17f0 msgr2=0x7f06c80b62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06cd000640 1 --2- 172.21.15.2:0/158819850 >> v2:172.21.15.137:3300/0 conn(0x7f06c80d17f0 0x7f06c80b62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7400640 1 --2- 172.21.15.2:0/158819850 >> v2:172.21.15.137:3300/0 conn(0x7f06c80d17f0 0x7f06c80b62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c7400640 1 -- 172.21.15.2:0/158819850 >> v1:172.21.15.2:6789/0 conn(0x7f06c80b4cf0 legacy=0x7f06c80b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.532+0000 7f06c7400640 1 -- 172.21.15.2:0/158819850 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c80d2050 con 0x7f06c80d2e10 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.531+0000 7f06c6a00640 1 -- 172.21.15.2:0/158819850 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415879972 0 0) 0x7f06c815d8b0 con 0x7f06c80b4cf0 2024-01-15T18:04:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.532+0000 7f06cd000640 1 --2- 172.21.15.2:0/158819850 >> v2:172.21.15.137:3300/0 conn(0x7f06c80d17f0 0x7f06c80b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.533+0000 7f06c7400640 1 --2- 172.21.15.2:0/158819850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d2e10 0x7f06c80d12b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ca090 tx=0x7f06b002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55f89b419150c437 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.533+0000 7f06c6a00640 1 -- 172.21.15.2:0/158819850 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b002f850 con 0x7f06c80d2e10 2024-01-15T18:04:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.533+0000 7f06c6a00640 1 -- 172.21.15.2:0/158819850 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b002f9b0 con 0x7f06c80d2e10 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.533+0000 7f06c6a00640 1 -- 172.21.15.2:0/158819850 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b002fd40 con 0x7f06c80d2e10 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 -- 172.21.15.2:0/158819850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d2e10 msgr2=0x7f06c80d12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 --2- 172.21.15.2:0/158819850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d2e10 0x7f06c80d12b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f06c80ca090 tx=0x7f06b002ba30 comp rx=0 tx=0).stop 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 -- 172.21.15.2:0/158819850 shutdown_connections 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 --2- 172.21.15.2:0/158819850 >> v2:172.21.15.137:3300/0 conn(0x7f06c80d17f0 0x7f06c80b62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 --2- 172.21.15.2:0/158819850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d2e10 0x7f06c80d12b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 -- 172.21.15.2:0/158819850 >> 172.21.15.2:0/158819850 conn(0x7f06c8072210 msgr2=0x7f06c80cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 -- 172.21.15.2:0/158819850 shutdown_connections 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.534+0000 7f06cf000640 1 -- 172.21.15.2:0/158819850 wait complete. 2024-01-15T18:04:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.536+0000 7f06cf000640 1 Processor -- start 2024-01-15T18:04:03.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.536+0000 7f06cf000640 1 -- start start 2024-01-15T18:04:03.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cf000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06c80b4cf0 0x7f06c80ccd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cf000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 0x7f06c8154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cf000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06c80d2e10 0x7f06c815fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cf000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c815f0d0 con 0x7f06c80d17f0 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cf000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c80ca500 con 0x7f06c80d2e10 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cf000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c8157fb0 con 0x7f06c80b4cf0 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 0x7f06c8154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 0x7f06c8154820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39060/0 (socket says 172.21.15.2:39060) 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 -- 172.21.15.2:0/3475827492 learned_addr learned my addr 172.21.15.2:0/3475827492 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cd000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06c80d2e10 0x7f06c815fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7e00640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06c80b4cf0 0x7f06c80ccd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 -- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06c80b4cf0 msgr2=0x7f06c80ccd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06c80b4cf0 0x7f06c80ccd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 -- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06c80d2e10 msgr2=0x7f06c815fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06c80d2e10 0x7f06c815fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06c7400640 1 -- 172.21.15.2:0/3475827492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00094d0 con 0x7f06c80d17f0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.537+0000 7f06cd000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06c80d2e10 0x7f06c815fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.538+0000 7f06c7400640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 0x7f06c8154820 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f06b002ba70 tx=0x7f06b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.538+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0037cf0 con 0x7f06c80d17f0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.538+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b0037e50 con 0x7f06c80d17f0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.538+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c815e830 con 0x7f06c80d17f0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.538+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b0041640 con 0x7f06c80d17f0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.538+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c815edf0 con 0x7f06c80d17f0 2024-01-15T18:04:03.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.539+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c80686f0 con 0x7f06c80d17f0 2024-01-15T18:04:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.541+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f06b00417a0 con 0x7f06c80d17f0 2024-01-15T18:04:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.541+0000 7f06bfe00640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f06a006b920 0x7f06a006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.542+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f06b003c070 con 0x7f06c80d17f0 2024-01-15T18:04:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.542+0000 7f06c7e00640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f06a006b920 0x7f06a006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.544+0000 7f06c7e00640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f06a006b920 0x7f06a006dde0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f06b8007fd0 tx=0x7f06b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:03.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.546+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06b0077a70 con 0x7f06c80d17f0 2024-01-15T18:04:03.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.782+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f06c80b8d10 con 0x7f06a006b920 2024-01-15T18:04:03.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.818+0000 7f06bfe00640 1 -- 172.21.15.2:0/3475827492 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f06b8001350 con 0x7f06a006b920 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.820+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f06a006b920 msgr2=0x7f06a006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.820+0000 7f06cf000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f06a006b920 0x7f06a006dde0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f06b8007fd0 tx=0x7f06b8009040 comp rx=0 tx=0).stop 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 msgr2=0x7f06c8154820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 0x7f06c8154820 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f06b002ba70 tx=0x7f06b0004730 comp rx=0 tx=0).stop 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 shutdown_connections 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f06a006b920 0x7f06a006dde0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f06c80d2e10 0x7f06c815fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f06c80d17f0 0x7f06c8154820 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 --2- 172.21.15.2:0/3475827492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f06c80b4cf0 0x7f06c80ccd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:03.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 >> 172.21.15.2:0/3475827492 conn(0x7f06c8072210 msgr2=0x7f06c80b7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:03.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 shutdown_connections 2024-01-15T18:04:03.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:03.821+0000 7f06cf000640 1 -- 172.21.15.2:0/3475827492 wait complete. 2024-01-15T18:04:03.842 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:04:04.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.055+0000 7f7fd9400640 1 Processor -- start 2024-01-15T18:04:04.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.055+0000 7f7fd9400640 1 -- start start 2024-01-15T18:04:04.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd9400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7fd4144c00 0x7f7fd40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:04.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd9400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7fd40cdd70 0x7f7fd406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:04.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd9400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd40ce230 con 0x7f7fd415c450 2024-01-15T18:04:04.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd9400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd40ce510 con 0x7f7fd4144c00 2024-01-15T18:04:04.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd9400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd40ce7f0 con 0x7f7fd40cdd70 2024-01-15T18:04:04.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7fd4144c00 0x7f7fd40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:04.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7fd4144c00 0x7f7fd40cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39066/0 (socket says 172.21.15.2:39066) 2024-01-15T18:04:04.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.057+0000 7f7fd2800640 1 -- 172.21.15.2:0/3429598726 learned_addr learned my addr 172.21.15.2:0/3429598726 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:04.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.058+0000 7f7fd2800640 1 -- 172.21.15.2:0/3429598726 >> v2:172.21.15.137:3300/0 conn(0x7f7fd40cdd70 msgr2=0x7f7fd406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:04.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.058+0000 7f7fd2800640 1 --2- 172.21.15.2:0/3429598726 >> v2:172.21.15.137:3300/0 conn(0x7f7fd40cdd70 0x7f7fd406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.058+0000 7f7fd2800640 1 -- 172.21.15.2:0/3429598726 >> v1:172.21.15.137:6789/0 conn(0x7f7fd415c450 legacy=0x7f7fd40b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:04.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.058+0000 7f7fd2800640 1 -- 172.21.15.2:0/3429598726 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd40705f0 con 0x7f7fd4144c00 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.059+0000 7f7fd2800640 1 --2- 172.21.15.2:0/3429598726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd4144c00 0x7f7fd40cd830 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7fd406f9f0 tx=0x7f7fc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72ace14f9d49a13 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.059+0000 7f7fd1e00640 1 -- 172.21.15.2:0/3429598726 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4031870 con 0x7f7fd4144c00 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.059+0000 7f7fd1e00640 1 -- 172.21.15.2:0/3429598726 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc40319d0 con 0x7f7fd4144c00 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.059+0000 7f7fd1e00640 1 -- 172.21.15.2:0/3429598726 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4031d60 con 0x7f7fd4144c00 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 -- 172.21.15.2:0/3429598726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd4144c00 msgr2=0x7f7fd40cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3429598726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd4144c00 0x7f7fd40cd830 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f7fd406f9f0 tx=0x7f7fc402b910 comp rx=0 tx=0).stop 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 -- 172.21.15.2:0/3429598726 shutdown_connections 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3429598726 >> v2:172.21.15.137:3300/0 conn(0x7f7fd40cdd70 0x7f7fd406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3429598726 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd4144c00 0x7f7fd40cd830 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 -- 172.21.15.2:0/3429598726 >> 172.21.15.2:0/3429598726 conn(0x7f7fd40b45f0 msgr2=0x7f7fd40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 -- 172.21.15.2:0/3429598726 shutdown_connections 2024-01-15T18:04:04.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.060+0000 7f7fd9400640 1 -- 172.21.15.2:0/3429598726 wait complete. 2024-01-15T18:04:04.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.063+0000 7f7fd9400640 1 Processor -- start 2024-01-15T18:04:04.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.063+0000 7f7fd9400640 1 -- start start 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.064+0000 7f7fd9400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fd40cdd70 0x7f7fd40a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.064+0000 7f7fd9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fd4144c00 0x7f7fd40b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.064+0000 7f7fd9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 0x7f7fd4140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.064+0000 7f7fd9400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40b1ff0 con 0x7f7fd415c450 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.064+0000 7f7fd9400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40a8b60 con 0x7f7fd40cdd70 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.064+0000 7f7fd9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd40b1300 con 0x7f7fd4144c00 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 0x7f7fd4140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 0x7f7fd4140d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39074/0 (socket says 172.21.15.2:39074) 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 -- 172.21.15.2:0/3871595757 learned_addr learned my addr 172.21.15.2:0/3871595757 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fc9c00640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fd4144c00 0x7f7fd40b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd2800640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fd40cdd70 0x7f7fd40a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 -- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fd4144c00 msgr2=0x7f7fd40b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fd4144c00 0x7f7fd40b05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 -- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fd40cdd70 msgr2=0x7f7fd40a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fd40cdd70 0x7f7fd40a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.065+0000 7f7fd3200640 1 -- 172.21.15.2:0/3871595757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc40094d0 con 0x7f7fd415c450 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.066+0000 7f7fd3200640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 0x7f7fd4140d80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00b410 tx=0x7f7fcc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.066+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc011830 con 0x7f7fd415c450 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.066+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc011990 con 0x7f7fd415c450 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.066+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc011d40 con 0x7f7fd415c450 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.066+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd40b0b20 con 0x7f7fd415c450 2024-01-15T18:04:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.066+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd40b1810 con 0x7f7fd415c450 2024-01-15T18:04:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.067+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd40686f0 con 0x7f7fd415c450 2024-01-15T18:04:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.068+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f7fcc01a400 con 0x7f7fd415c450 2024-01-15T18:04:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.069+0000 7f7fcb400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fac06b920 0x7f7fac06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.070+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f7fcc08a600 con 0x7f7fd415c450 2024-01-15T18:04:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.070+0000 7f7fd2800640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fac06b920 0x7f7fac06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.071+0000 7f7fd2800640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fac06b920 0x7f7fac06dde0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4036000 tx=0x7f7fc403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:04.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.074+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fcc054bf0 con 0x7f7fd415c450 2024-01-15T18:04:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.298+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fd406ce30 con 0x7f7fac06b920 2024-01-15T18:04:04.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.309+0000 7f7fcb400640 1 -- 172.21.15.2:0/3871595757 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7fc4031c40 con 0x7f7fac06b920 2024-01-15T18:04:04.312 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fac06b920 msgr2=0x7f7fac06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fac06b920 0x7f7fac06dde0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4036000 tx=0x7f7fc403c040 comp rx=0 tx=0).stop 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 msgr2=0x7f7fd4140d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 0x7f7fd4140d80 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00b410 tx=0x7f7fcc00b8d0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 shutdown_connections 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7fac06b920 0x7f7fac06dde0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7fd415c450 0x7f7fd4140d80 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7fd4144c00 0x7f7fd40b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 --2- 172.21.15.2:0/3871595757 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7fd40cdd70 0x7f7fd40a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:04.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 >> 172.21.15.2:0/3871595757 conn(0x7f7fd40b45f0 msgr2=0x7f7fd415c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:04.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 shutdown_connections 2024-01-15T18:04:04.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:04.312+0000 7f7fd9400640 1 -- 172.21.15.2:0/3871595757 wait complete. 2024-01-15T18:04:04.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:04:04.332 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6) 2024-01-15T18:04:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6 2024-01-15T18:04:04.392 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:04:04.393 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:04.394 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:04:06.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:04:06.395 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6) 2024-01-15T18:04:06.465 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6 2024-01-15T18:04:06.465 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:04:06.465 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:06.466 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:04:08.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:04:08.468 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6) 2024-01-15T18:04:08.534 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/02019c34-e09f-4394-9559-da0a5f6889d6 2024-01-15T18:04:08.534 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:04:08.534 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:08.535 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:04:08.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27421 2024-01-15T18:04:08.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.763+0000 7f1f90c00640 1 Processor -- start 2024-01-15T18:04:08.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.763+0000 7f1f90c00640 1 -- start start 2024-01-15T18:04:08.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f90c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1f8c0716b0 0x7f1f8c0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f90c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0b2fa0 con 0x7f1f8c143620 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f90c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0b3280 con 0x7f1f8c071230 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f90c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8c0b3560 con 0x7f1f8c0716b0 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f8b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1f8c143620 0x7f1f8c0b2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41638/0 (socket says 172.21.15.2:41638) 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f8b400640 1 -- 172.21.15.2:0/3703054431 learned_addr learned my addr 172.21.15.2:0/3703054431 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.765+0000 7f1f8be00640 1 --2- 172.21.15.2:0/3703054431 >> v2:172.21.15.2:3300/0 conn(0x7f1f8c0716b0 0x7f1f8c0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.766+0000 7f1f8a000640 1 -- 172.21.15.2:0/3703054431 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362355096 0 0) 0x7f1f8c0b2fa0 con 0x7f1f8c143620 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.766+0000 7f1f8a000640 1 -- 172.21.15.2:0/3703054431 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c003520 con 0x7f1f8c143620 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.766+0000 7f1f8be00640 1 -- 172.21.15.2:0/3703054431 >> v1:172.21.15.137:6789/0 conn(0x7f1f8c071230 legacy=0x7f1f8c0cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.766+0000 7f1f8be00640 1 -- 172.21.15.2:0/3703054431 >> v1:172.21.15.2:6789/0 conn(0x7f1f8c143620 legacy=0x7f1f8c0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:08.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.766+0000 7f1f8be00640 1 -- 172.21.15.2:0/3703054431 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0b3840 con 0x7f1f8c0716b0 2024-01-15T18:04:08.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.767+0000 7f1f8be00640 1 --2- 172.21.15.2:0/3703054431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c0ccac0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009d90 tx=0x7f1f8402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8a3b32364006178 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:08.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.767+0000 7f1f8a000640 1 -- 172.21.15.2:0/3703054431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8402cc60 con 0x7f1f8c0716b0 2024-01-15T18:04:08.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f8a000640 1 -- 172.21.15.2:0/3703054431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8402cdc0 con 0x7f1f8c0716b0 2024-01-15T18:04:08.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f8a000640 1 -- 172.21.15.2:0/3703054431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84034660 con 0x7f1f8c0716b0 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 -- 172.21.15.2:0/3703054431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c0716b0 msgr2=0x7f1f8c0ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 --2- 172.21.15.2:0/3703054431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c0ccac0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1f84009d90 tx=0x7f1f8402beb0 comp rx=0 tx=0).stop 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 -- 172.21.15.2:0/3703054431 shutdown_connections 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 --2- 172.21.15.2:0/3703054431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c0ccac0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 -- 172.21.15.2:0/3703054431 >> 172.21.15.2:0/3703054431 conn(0x7f1f8c0b0840 msgr2=0x7f1f8c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 -- 172.21.15.2:0/3703054431 shutdown_connections 2024-01-15T18:04:08.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.768+0000 7f1f90c00640 1 -- 172.21.15.2:0/3703054431 wait complete. 2024-01-15T18:04:08.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.770+0000 7f1f90c00640 1 Processor -- start 2024-01-15T18:04:08.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.770+0000 7f1f90c00640 1 -- start start 2024-01-15T18:04:08.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f90c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 0x7f1f8c0ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:08.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f90c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c13c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:08.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f90c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f8c143620 0x7f1f8c1f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:08.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f90c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c13c650 con 0x7f1f8c071230 2024-01-15T18:04:08.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f90c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c1f6080 con 0x7f1f8c0716b0 2024-01-15T18:04:08.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f90c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8c13c8a0 con 0x7f1f8c143620 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.771+0000 7f1f8b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 0x7f1f8c0ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 0x7f1f8c0ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39092/0 (socket says 172.21.15.2:39092) 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 -- 172.21.15.2:0/709362378 learned_addr learned my addr 172.21.15.2:0/709362378 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 -- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f8c143620 msgr2=0x7f1f8c1f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f8c143620 0x7f1f8c1f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 -- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f8c0716b0 msgr2=0x7f1f8c13c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8aa00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c13c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c13c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 -- 172.21.15.2:0/709362378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84009a70 con 0x7f1f8c071230 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.772+0000 7f1f8b400640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 0x7f1f8c0ad790 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00c410 tx=0x7f1f7c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:08.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.773+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c005ce0 con 0x7f1f8c071230 2024-01-15T18:04:08.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.773+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7c005e40 con 0x7f1f8c071230 2024-01-15T18:04:08.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.773+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7c0072f0 con 0x7f1f8c071230 2024-01-15T18:04:08.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.773+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c13cb80 con 0x7f1f8c071230 2024-01-15T18:04:08.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.773+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c13d140 con 0x7f1f8c071230 2024-01-15T18:04:08.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.774+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c074770 con 0x7f1f8c071230 2024-01-15T18:04:08.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.775+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f1f7c007510 con 0x7f1f8c071230 2024-01-15T18:04:08.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.776+0000 7f1f83400640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f6406b8f0 0x7f1f6406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:08.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.776+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f1f7c089e70 con 0x7f1f8c071230 2024-01-15T18:04:08.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.776+0000 7f1f8aa00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f6406b8f0 0x7f1f6406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:08.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.777+0000 7f1f8aa00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f6406b8f0 0x7f1f6406ddb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0acf60 tx=0x7f1f74009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:08.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:08.781+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f7c0543e0 con 0x7f1f8c071230 2024-01-15T18:04:09.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.012+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27421", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f8c143a00 con 0x7f1f6406b8f0 2024-01-15T18:04:09.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.165+0000 7f1f83400640 1 -- 172.21.15.2:0/709362378 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1f74003980 con 0x7f1f6406b8f0 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f6406b8f0 msgr2=0x7f1f6406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f6406b8f0 0x7f1f6406ddb0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1f8c0acf60 tx=0x7f1f74009040 comp rx=0 tx=0).stop 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 msgr2=0x7f1f8c0ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 0x7f1f8c0ad790 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c00c410 tx=0x7f1f7c00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 shutdown_connections 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f6406b8f0 0x7f1f6406ddb0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.169+0000 7f1f90c00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f8c143620 0x7f1f8c1f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.170+0000 7f1f90c00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f8c0716b0 0x7f1f8c13c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.170+0000 7f1f90c00640 1 --2- 172.21.15.2:0/709362378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f8c071230 0x7f1f8c0ad790 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.170+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 >> 172.21.15.2:0/709362378 conn(0x7f1f8c0b0840 msgr2=0x7f1f8c06d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.170+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 shutdown_connections 2024-01-15T18:04:09.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.170+0000 7f1f90c00640 1 -- 172.21.15.2:0/709362378 wait complete. 2024-01-15T18:04:09.175 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.171+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.175 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.171+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.175 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.171+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.175 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.171+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.175 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.171+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.197 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27422 2024-01-15T18:04:09.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.201+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.201+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.201+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.206 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.201+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.206 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:09.201+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:09.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.416+0000 7f41e3000640 1 Processor -- start 2024-01-15T18:04:09.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.416+0000 7f41e3000640 1 -- start start 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e3000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f41dc06c8f0 0x7f41dc06cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e3000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc15dd70 con 0x7f41dc1439f0 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e3000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc070720 con 0x7f41dc06c8f0 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc06d2a0 con 0x7f41dc0702c0 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e1000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f41dc06c8f0 0x7f41dc06cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:09.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e1000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f41dc06c8f0 0x7f41dc06cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39096/0 (socket says 172.21.15.2:39096) 2024-01-15T18:04:09.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.418+0000 7f41e1000640 1 -- 172.21.15.2:0/2200304095 learned_addr learned my addr 172.21.15.2:0/2200304095 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:09.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.419+0000 7f41db400640 1 --2- 172.21.15.2:0/2200304095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:09.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.419+0000 7f41e1000640 1 -- 172.21.15.2:0/2200304095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 msgr2=0x7f41dc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:09.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.419+0000 7f41e1000640 1 --2- 172.21.15.2:0/2200304095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc0b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.419+0000 7f41e1000640 1 -- 172.21.15.2:0/2200304095 >> v1:172.21.15.137:6789/0 conn(0x7f41dc1439f0 legacy=0x7f41dc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:09.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.419+0000 7f41e1000640 1 -- 172.21.15.2:0/2200304095 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc0711b0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.420+0000 7f41e1000640 1 --2- 172.21.15.2:0/2200304095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc06cd60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f41d40098a0 tx=0x7f41d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffd488cdb5a9601 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:09.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.420+0000 7f41daa00640 1 -- 172.21.15.2:0/2200304095 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4031640 con 0x7f41dc06c8f0 2024-01-15T18:04:09.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.420+0000 7f41daa00640 1 -- 172.21.15.2:0/2200304095 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d40317a0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.420+0000 7f41daa00640 1 -- 172.21.15.2:0/2200304095 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d4031b30 con 0x7f41dc06c8f0 2024-01-15T18:04:09.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 -- 172.21.15.2:0/2200304095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 msgr2=0x7f41dc06cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 --2- 172.21.15.2:0/2200304095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc06cd60 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f41d40098a0 tx=0x7f41d402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 -- 172.21.15.2:0/2200304095 shutdown_connections 2024-01-15T18:04:09.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 --2- 172.21.15.2:0/2200304095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc06cd60 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 --2- 172.21.15.2:0/2200304095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 -- 172.21.15.2:0/2200304095 >> 172.21.15.2:0/2200304095 conn(0x7f41dc0b45f0 msgr2=0x7f41dc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 -- 172.21.15.2:0/2200304095 shutdown_connections 2024-01-15T18:04:09.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.421+0000 7f41e3000640 1 -- 172.21.15.2:0/2200304095 wait complete. 2024-01-15T18:04:09.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.423+0000 7f41e3000640 1 Processor -- start 2024-01-15T18:04:09.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.423+0000 7f41e3000640 1 -- start start 2024-01-15T18:04:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41e3000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41e3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc1410e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:09.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41e3000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41dc1439f0 0x7f41dc13f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41e3000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc140c50 con 0x7f41dc06c8f0 2024-01-15T18:04:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41e3000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc157fe0 con 0x7f41dc1439f0 2024-01-15T18:04:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41e3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc140040 con 0x7f41dc0702c0 2024-01-15T18:04:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39102/0 (socket says 172.21.15.2:39102) 2024-01-15T18:04:09.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.424+0000 7f41dbe00640 1 -- 172.21.15.2:0/2878248172 learned_addr learned my addr 172.21.15.2:0/2878248172 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41dbe00640 1 -- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 msgr2=0x7f41dc1410e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41dbe00640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc1410e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41dbe00640 1 -- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41dc1439f0 msgr2=0x7f41dc13f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41dbe00640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41dc1439f0 0x7f41dc13f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41dbe00640 1 -- 172.21.15.2:0/2878248172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40094d0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41dbe00640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc157aa0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00ba80 tx=0x7f41cc00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00cc00 con 0x7f41dc06c8f0 2024-01-15T18:04:09.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc00cd60 con 0x7f41dc06c8f0 2024-01-15T18:04:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc003bb0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.425+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc140320 con 0x7f41dc06c8f0 2024-01-15T18:04:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.426+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc1408e0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.427+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc0686f0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.428+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f41cc014440 con 0x7f41dc06c8f0 2024-01-15T18:04:09.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.428+0000 7f41d3e00640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41b406b8f0 0x7f41b406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:09.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.429+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f41cc08a5e0 con 0x7f41dc06c8f0 2024-01-15T18:04:09.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.429+0000 7f41db400640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41b406b8f0 0x7f41b406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:09.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.430+0000 7f41db400640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41b406b8f0 0x7f41b406ddb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006fd0 tx=0x7f41c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:09.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.433+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41cc054a90 con 0x7f41dc06c8f0 2024-01-15T18:04:09.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.656+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f41cc054e60 con 0x7f41dc06c8f0 2024-01-15T18:04:09.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.669+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27422", "target": ["mon-mgr", ""]}) v1 -- 0x7f41dc0cbb40 con 0x7f41b406b8f0 2024-01-15T18:04:09.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.829+0000 7f41d3e00640 1 -- 172.21.15.2:0/2878248172 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f41dc0cbb40 con 0x7f41b406b8f0 2024-01-15T18:04:09.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.832+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41b406b8f0 msgr2=0x7f41b406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.832+0000 7f41e3000640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41b406b8f0 0x7f41b406ddb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f41c4006fd0 tx=0x7f41c4008040 comp rx=0 tx=0).stop 2024-01-15T18:04:09.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.832+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 msgr2=0x7f41dc157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:09.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.832+0000 7f41e3000640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc157aa0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00ba80 tx=0x7f41cc00bf40 comp rx=0 tx=0).stop 2024-01-15T18:04:09.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 shutdown_connections 2024-01-15T18:04:09.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41b406b8f0 0x7f41b406ddb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41dc1439f0 0x7f41dc13f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41dc0702c0 0x7f41dc1410e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 --2- 172.21.15.2:0/2878248172 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41dc06c8f0 0x7f41dc157aa0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:09.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 >> 172.21.15.2:0/2878248172 conn(0x7f41dc0b45f0 msgr2=0x7f41dc1e2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:09.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 shutdown_connections 2024-01-15T18:04:09.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:09.833+0000 7f41e3000640 1 -- 172.21.15.2:0/2878248172 wait complete. 2024-01-15T18:04:09.859 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27423 2024-01-15T18:04:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.070+0000 7f1de3e00640 1 Processor -- start 2024-01-15T18:04:10.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.071+0000 7f1de3e00640 1 -- start start 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.072+0000 7f1de3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1de415f260 0x7f1de406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.072+0000 7f1de3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1de40720d0 0x7f1de406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.072+0000 7f1de3e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de4071090 con 0x7f1de4143d20 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.072+0000 7f1de3e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de4071430 con 0x7f1de40720d0 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.072+0000 7f1de3e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de4144100 con 0x7f1de415f260 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de2a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1de4143d20 0x7f1de415eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41644/0 (socket says 172.21.15.2:41644) 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de2a00640 1 -- 172.21.15.2:0/3015400348 learned_addr learned my addr 172.21.15.2:0/3015400348 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:10.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de3400640 1 --2- 172.21.15.2:0/3015400348 >> v2:172.21.15.2:3300/0 conn(0x7f1de40720d0 0x7f1de406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:10.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1ddbe00640 1 --2- 172.21.15.2:0/3015400348 >> v2:172.21.15.137:3300/0 conn(0x7f1de415f260 0x7f1de406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:10.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de3400640 1 -- 172.21.15.2:0/3015400348 >> v2:172.21.15.137:3300/0 conn(0x7f1de415f260 msgr2=0x7f1de406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de3400640 1 --2- 172.21.15.2:0/3015400348 >> v2:172.21.15.137:3300/0 conn(0x7f1de415f260 0x7f1de406c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de3400640 1 -- 172.21.15.2:0/3015400348 >> v1:172.21.15.2:6789/0 conn(0x7f1de4143d20 legacy=0x7f1de415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de3400640 1 -- 172.21.15.2:0/3015400348 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4144560 con 0x7f1de40720d0 2024-01-15T18:04:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1ddbe00640 1 --2- 172.21.15.2:0/3015400348 >> v2:172.21.15.137:3300/0 conn(0x7f1de415f260 0x7f1de406c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.073+0000 7f1de2000640 1 -- 172.21.15.2:0/3015400348 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657408336 0 0) 0x7f1de4071090 con 0x7f1de4143d20 2024-01-15T18:04:10.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.074+0000 7f1de3400640 1 --2- 172.21.15.2:0/3015400348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de40720d0 0x7f1de406c690 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1de404f900 tx=0x7f1ddc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44285d77db80778d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:10.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de2000640 1 -- 172.21.15.2:0/3015400348 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc02cbe0 con 0x7f1de40720d0 2024-01-15T18:04:10.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de2000640 1 -- 172.21.15.2:0/3015400348 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc02cd40 con 0x7f1de40720d0 2024-01-15T18:04:10.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de2000640 1 -- 172.21.15.2:0/3015400348 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc034660 con 0x7f1de40720d0 2024-01-15T18:04:10.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de3e00640 1 -- 172.21.15.2:0/3015400348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de40720d0 msgr2=0x7f1de406c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de3e00640 1 --2- 172.21.15.2:0/3015400348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de40720d0 0x7f1de406c690 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f1de404f900 tx=0x7f1ddc02be30 comp rx=0 tx=0).stop 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de3e00640 1 -- 172.21.15.2:0/3015400348 shutdown_connections 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de3e00640 1 --2- 172.21.15.2:0/3015400348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de40720d0 0x7f1de406c690 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de3e00640 1 --2- 172.21.15.2:0/3015400348 >> v2:172.21.15.137:3300/0 conn(0x7f1de415f260 0x7f1de406c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.075+0000 7f1de3e00640 1 -- 172.21.15.2:0/3015400348 >> 172.21.15.2:0/3015400348 conn(0x7f1de40b0840 msgr2=0x7f1de40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.076+0000 7f1de3e00640 1 -- 172.21.15.2:0/3015400348 shutdown_connections 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.076+0000 7f1de3e00640 1 -- 172.21.15.2:0/3015400348 wait complete. 2024-01-15T18:04:10.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.077+0000 7f1de3e00640 1 Processor -- start 2024-01-15T18:04:10.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.077+0000 7f1de3e00640 1 -- start start 2024-01-15T18:04:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.078+0000 7f1de3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de40720d0 0x7f1de415cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.078+0000 7f1de3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de4143d20 0x7f1de41f05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.078+0000 7f1de3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 0x7f1de415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.078+0000 7f1de3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de41f1620 con 0x7f1de415f260 2024-01-15T18:04:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.078+0000 7f1de3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de415d4b0 con 0x7f1de4143d20 2024-01-15T18:04:10.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.078+0000 7f1de3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de41f0b20 con 0x7f1de40720d0 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 0x7f1de415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 0x7f1de415a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:39120/0 (socket says 172.21.15.2:39120) 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 -- 172.21.15.2:0/64680522 learned_addr learned my addr 172.21.15.2:0/64680522 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1ddbe00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de4143d20 0x7f1de41f05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 -- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de40720d0 msgr2=0x7f1de415cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de40720d0 0x7f1de415cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 -- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de4143d20 msgr2=0x7f1de41f05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de4143d20 0x7f1de41f05e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1de3400640 1 -- 172.21.15.2:0/64680522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc009a70 con 0x7f1de415f260 2024-01-15T18:04:10.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.079+0000 7f1ddbe00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de4143d20 0x7f1de41f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:10.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.080+0000 7f1de3400640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 0x7f1de415a2a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02c440 tx=0x7f1ddc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:10.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.080+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc040650 con 0x7f1de415f260 2024-01-15T18:04:10.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.080+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ddc0407d0 con 0x7f1de415f260 2024-01-15T18:04:10.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.080+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc040b80 con 0x7f1de415f260 2024-01-15T18:04:10.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.080+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de415a7e0 con 0x7f1de415f260 2024-01-15T18:04:10.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.080+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de41f1060 con 0x7f1de415f260 2024-01-15T18:04:10.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.081+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de4074770 con 0x7f1de415f260 2024-01-15T18:04:10.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.082+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f1ddc040ce0 con 0x7f1de415f260 2024-01-15T18:04:10.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.083+0000 7f1ddaa00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dbc06ba10 0x7f1dbc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:10.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.083+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f1ddc0ae090 con 0x7f1de415f260 2024-01-15T18:04:10.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.084+0000 7f1de2a00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dbc06ba10 0x7f1dbc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:10.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.085+0000 7f1de2a00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dbc06ba10 0x7f1dbc06ded0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4007fd0 tx=0x7f1dd4007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:10.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.088+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ddc0ab510 con 0x7f1de415f260 2024-01-15T18:04:10.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:10.322+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27423", "target": ["mon-mgr", ""]}) v1 -- 0x7f1de415f720 con 0x7f1dbc06ba10 2024-01-15T18:04:13.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.329+0000 7f1ddaa00640 1 -- 172.21.15.2:0/64680522 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1dd40070d0 con 0x7f1dbc06ba10 2024-01-15T18:04:13.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dbc06ba10 msgr2=0x7f1dbc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:13.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dbc06ba10 0x7f1dbc06ded0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4007fd0 tx=0x7f1dd4007920 comp rx=0 tx=0).stop 2024-01-15T18:04:13.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 msgr2=0x7f1de415a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:13.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 0x7f1de415a2a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02c440 tx=0x7f1ddc004a40 comp rx=0 tx=0).stop 2024-01-15T18:04:13.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 shutdown_connections 2024-01-15T18:04:13.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dbc06ba10 0x7f1dbc06ded0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de415f260 0x7f1de415a2a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de4143d20 0x7f1de41f05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 --2- 172.21.15.2:0/64680522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de40720d0 0x7f1de415cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 >> 172.21.15.2:0/64680522 conn(0x7f1de40b0840 msgr2=0x7f1de40aeb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:13.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 shutdown_connections 2024-01-15T18:04:13.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.333+0000 7f1de3e00640 1 -- 172.21.15.2:0/64680522 wait complete. 2024-01-15T18:04:13.362 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27424 2024-01-15T18:04:13.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.586+0000 7f5078e00640 1 Processor -- start 2024-01-15T18:04:13.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.587+0000 7f5078e00640 1 -- start start 2024-01-15T18:04:13.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.588+0000 7f5078e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1870 0x7f50740d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:13.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.588+0000 7f5078e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50740d1c50 0x7f50740b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:13.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.588+0000 7f5078e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740d1250 con 0x7f50740b2210 2024-01-15T18:04:13.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.588+0000 7f5078e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50740ce6d0 con 0x7f50740d1c50 2024-01-15T18:04:13.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.588+0000 7f5078e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50740ce8e0 con 0x7f50740d1870 2024-01-15T18:04:13.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50740d1c50 0x7f50740b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f50740d1c50 0x7f50740b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37980/0 (socket says 172.21.15.2:37980) 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 -- 172.21.15.2:0/4191115822 learned_addr learned my addr 172.21.15.2:0/4191115822 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 -- 172.21.15.2:0/4191115822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1870 msgr2=0x7f50740d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073400640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1870 0x7f50740d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1870 0x7f50740d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 -- 172.21.15.2:0/4191115822 >> v1:172.21.15.137:6789/0 conn(0x7f50740b2210 legacy=0x7f507415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073e00640 1 -- 172.21.15.2:0/4191115822 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740cebc0 con 0x7f50740d1c50 2024-01-15T18:04:13.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.589+0000 7f5073400640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1870 0x7f50740d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:13.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.590+0000 7f5073e00640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740d1c50 0x7f50740b26f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f506c009870 tx=0x7f506c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c709a6b9f91ebc6b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:13.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.591+0000 7f5072000640 1 -- 172.21.15.2:0/4191115822 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c031640 con 0x7f50740d1c50 2024-01-15T18:04:13.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.591+0000 7f5072000640 1 -- 172.21.15.2:0/4191115822 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f506c0317a0 con 0x7f50740d1c50 2024-01-15T18:04:13.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.591+0000 7f5072000640 1 -- 172.21.15.2:0/4191115822 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506c031b30 con 0x7f50740d1c50 2024-01-15T18:04:13.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.591+0000 7f5078e00640 1 -- 172.21.15.2:0/4191115822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740d1c50 msgr2=0x7f50740b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.591+0000 7f5078e00640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740d1c50 0x7f50740b26f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f506c009870 tx=0x7f506c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.592+0000 7f5078e00640 1 -- 172.21.15.2:0/4191115822 shutdown_connections 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.592+0000 7f5078e00640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740d1c50 0x7f50740b26f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.592+0000 7f5078e00640 1 --2- 172.21.15.2:0/4191115822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1870 0x7f50740d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.592+0000 7f5078e00640 1 -- 172.21.15.2:0/4191115822 >> 172.21.15.2:0/4191115822 conn(0x7f50740d30b0 msgr2=0x7f50740d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.592+0000 7f5078e00640 1 -- 172.21.15.2:0/4191115822 shutdown_connections 2024-01-15T18:04:13.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.592+0000 7f5078e00640 1 -- 172.21.15.2:0/4191115822 wait complete. 2024-01-15T18:04:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.593+0000 7f5078e00640 1 Processor -- start 2024-01-15T18:04:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.594+0000 7f5078e00640 1 -- start start 2024-01-15T18:04:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5078e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 0x7f507415f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5078e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50740d1870 0x7f507414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:13.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5078e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1c50 0x7f507414c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5078e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5074160c50 con 0x7f50740b2210 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5078e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741603b0 con 0x7f50740d1870 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5078e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5074160690 con 0x7f50740d1c50 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 0x7f507415f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 0x7f507415f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37994/0 (socket says 172.21.15.2:37994) 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 -- 172.21.15.2:0/35627270 learned_addr learned my addr 172.21.15.2:0/35627270 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5072a00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50740d1870 0x7f507414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1c50 0x7f507414c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 -- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1c50 msgr2=0x7f507414c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1c50 0x7f507414c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 -- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50740d1870 msgr2=0x7f507414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50740d1870 0x7f507414c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.595+0000 7f5073400640 1 -- 172.21.15.2:0/35627270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0094d0 con 0x7f50740b2210 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.596+0000 7f5073400640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 0x7f507415f980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f506400c7f0 tx=0x7f506400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.596+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064007a30 con 0x7f50740b2210 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.596+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5064007b90 con 0x7f50740b2210 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.596+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506400ce40 con 0x7f50740b2210 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.596+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5074160970 con 0x7f50740b2210 2024-01-15T18:04:13.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.596+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50740b3910 con 0x7f50740b2210 2024-01-15T18:04:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.597+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5074160d90 con 0x7f50740b2210 2024-01-15T18:04:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.598+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f50640113f0 con 0x7f50740b2210 2024-01-15T18:04:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.599+0000 7f506b400640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06ba10 0x7f504c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.599+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f506408b050 con 0x7f50740b2210 2024-01-15T18:04:13.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.600+0000 7f5072a00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06ba10 0x7f504c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:13.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.601+0000 7f5072a00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06ba10 0x7f504c06ded0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f50740b4360 tx=0x7f505c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:13.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.604+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5064088450 con 0x7f50740b2210 2024-01-15T18:04:13.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:13.833+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27424", "target": ["mon-mgr", ""]}) v1 -- 0x7f50740d0f20 con 0x7f504c06ba10 2024-01-15T18:04:18.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.329+0000 7f506b400640 1 -- 172.21.15.2:0/35627270 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f505c005110 con 0x7f504c06ba10 2024-01-15T18:04:18.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06ba10 msgr2=0x7f504c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06ba10 0x7f504c06ded0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f50740b4360 tx=0x7f505c009040 comp rx=0 tx=0).stop 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 msgr2=0x7f507415f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 0x7f507415f980 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f506400c7f0 tx=0x7f506400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 shutdown_connections 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06ba10 0x7f504c06ded0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50740d1c50 0x7f507414c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50740d1870 0x7f507414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 --2- 172.21.15.2:0/35627270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50740b2210 0x7f507415f980 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 >> 172.21.15.2:0/35627270 conn(0x7f50740d30b0 msgr2=0x7f50740d3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 shutdown_connections 2024-01-15T18:04:18.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.333+0000 7f5078e00640 1 -- 172.21.15.2:0/35627270 wait complete. 2024-01-15T18:04:18.364 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27425 2024-01-15T18:04:18.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.589+0000 7efcf3200640 1 Processor -- start 2024-01-15T18:04:18.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.590+0000 7efcf3200640 1 -- start start 2024-01-15T18:04:18.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.591+0000 7efcf3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf4143d20 0x7efcf415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:18.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.591+0000 7efcf3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efcf41447d0 0x7efcf415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:18.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.591+0000 7efcf3200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf4071090 con 0x7efcf40720d0 2024-01-15T18:04:18.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.591+0000 7efcf3200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf4071430 con 0x7efcf41447d0 2024-01-15T18:04:18.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.591+0000 7efcf3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf4144100 con 0x7efcf4143d20 2024-01-15T18:04:18.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efcf41447d0 0x7efcf415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efcf41447d0 0x7efcf415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38008/0 (socket says 172.21.15.2:38008) 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 -- 172.21.15.2:0/290010401 learned_addr learned my addr 172.21.15.2:0/290010401 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf1e00640 1 --2- 172.21.15.2:0/290010401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf4143d20 0x7efcf415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 -- 172.21.15.2:0/290010401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf4143d20 msgr2=0x7efcf415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 --2- 172.21.15.2:0/290010401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf4143d20 0x7efcf415eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 -- 172.21.15.2:0/290010401 >> v1:172.21.15.137:6789/0 conn(0x7efcf40720d0 legacy=0x7efcf40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.592+0000 7efcf2800640 1 -- 172.21.15.2:0/290010401 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf406d160 con 0x7efcf41447d0 2024-01-15T18:04:18.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.593+0000 7efcf2800640 1 --2- 172.21.15.2:0/290010401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf415f090 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7efcec009e50 tx=0x7efcec02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2cc6ed78bec17 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:18.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.594+0000 7efcf1400640 1 -- 172.21.15.2:0/290010401 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec02c9f0 con 0x7efcf41447d0 2024-01-15T18:04:18.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.594+0000 7efcf1400640 1 -- 172.21.15.2:0/290010401 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec02cb50 con 0x7efcf41447d0 2024-01-15T18:04:18.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.594+0000 7efcf1400640 1 -- 172.21.15.2:0/290010401 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0343f0 con 0x7efcf41447d0 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.594+0000 7efcf3200640 1 -- 172.21.15.2:0/290010401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 msgr2=0x7efcf415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.594+0000 7efcf3200640 1 --2- 172.21.15.2:0/290010401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf415f090 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7efcec009e50 tx=0x7efcec02be50 comp rx=0 tx=0).stop 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.595+0000 7efcf3200640 1 -- 172.21.15.2:0/290010401 shutdown_connections 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.595+0000 7efcf3200640 1 --2- 172.21.15.2:0/290010401 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf415f090 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.595+0000 7efcf3200640 1 --2- 172.21.15.2:0/290010401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf4143d20 0x7efcf415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.595+0000 7efcf3200640 1 -- 172.21.15.2:0/290010401 >> 172.21.15.2:0/290010401 conn(0x7efcf40b0840 msgr2=0x7efcf40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.595+0000 7efcf3200640 1 -- 172.21.15.2:0/290010401 shutdown_connections 2024-01-15T18:04:18.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.595+0000 7efcf3200640 1 -- 172.21.15.2:0/290010401 wait complete. 2024-01-15T18:04:18.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.596+0000 7efcf3200640 1 Processor -- start 2024-01-15T18:04:18.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.597+0000 7efcf3200640 1 -- start start 2024-01-15T18:04:18.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf40720d0 0x7efcf413e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:18.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcf4143d20 0x7efcf413dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:18.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:18.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf413d920 con 0x7efcf41447d0 2024-01-15T18:04:18.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf413e2d0 con 0x7efcf4143d20 2024-01-15T18:04:18.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf413eb80 con 0x7efcf40720d0 2024-01-15T18:04:18.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:18.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf4159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38014/0 (socket says 172.21.15.2:38014) 2024-01-15T18:04:18.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf2800640 1 -- 172.21.15.2:0/4147784283 learned_addr learned my addr 172.21.15.2:0/4147784283 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:18.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcebe00640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcf4143d20 0x7efcf413dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:18.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf1e00640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf40720d0 0x7efcf413e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf2800640 1 -- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf40720d0 msgr2=0x7efcf413e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf2800640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf40720d0 0x7efcf413e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.598+0000 7efcf2800640 1 -- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcf4143d20 msgr2=0x7efcf413dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcf2800640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcf4143d20 0x7efcf413dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcf2800640 1 -- 172.21.15.2:0/4147784283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec009b30 con 0x7efcf41447d0 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcf2800640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf4159b50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7efcec036040 tx=0x7efcec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec040670 con 0x7efcf41447d0 2024-01-15T18:04:18.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec0407f0 con 0x7efcf41447d0 2024-01-15T18:04:18.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec040ba0 con 0x7efcf41447d0 2024-01-15T18:04:18.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.599+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf413ee60 con 0x7efcf41447d0 2024-01-15T18:04:18.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.600+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf415b5a0 con 0x7efcf41447d0 2024-01-15T18:04:18.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.601+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf4074770 con 0x7efcf41447d0 2024-01-15T18:04:18.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.601+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7efcec040d00 con 0x7efcf41447d0 2024-01-15T18:04:18.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.602+0000 7efcea000640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efccc06ba10 0x7efccc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:18.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.602+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7efcec0adea0 con 0x7efcf41447d0 2024-01-15T18:04:18.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.603+0000 7efcf1e00640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efccc06ba10 0x7efccc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:18.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.604+0000 7efcf1e00640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efccc06ba10 0x7efccc06ded0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efce4006fd0 tx=0x7efce4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:18.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.607+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcec0ab320 con 0x7efcf41447d0 2024-01-15T18:04:18.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:18.836+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7efcf415f720 con 0x7efccc06ba10 2024-01-15T18:04:23.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.326+0000 7efcea000640 1 -- 172.21.15.2:0/4147784283 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efce400f860 con 0x7efccc06ba10 2024-01-15T18:04:23.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efccc06ba10 msgr2=0x7efccc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:23.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efccc06ba10 0x7efccc06ded0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efce4006fd0 tx=0x7efce4008040 comp rx=0 tx=0).stop 2024-01-15T18:04:23.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 msgr2=0x7efcf4159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf4159b50 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7efcec036040 tx=0x7efcec004a40 comp rx=0 tx=0).stop 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 shutdown_connections 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efccc06ba10 0x7efccc06ded0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efcf41447d0 0x7efcf4159b50 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efcf4143d20 0x7efcf413dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 --2- 172.21.15.2:0/4147784283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efcf40720d0 0x7efcf413e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.330+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 >> 172.21.15.2:0/4147784283 conn(0x7efcf40b0840 msgr2=0x7efcf40ae7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.331+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 shutdown_connections 2024-01-15T18:04:23.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.331+0000 7efcf3200640 1 -- 172.21.15.2:0/4147784283 wait complete. 2024-01-15T18:04:23.357 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27426 2024-01-15T18:04:23.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.584+0000 7f6e95c00640 1 Processor -- start 2024-01-15T18:04:23.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.584+0000 7f6e95c00640 1 -- start start 2024-01-15T18:04:23.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e95c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6e90144b90 0x7f6e900b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:23.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e95c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e9006ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:23.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e95c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9006f1b0 con 0x7f6e9015b840 2024-01-15T18:04:23.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e95c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e90070430 con 0x7f6e90144b90 2024-01-15T18:04:23.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e95c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900b4760 con 0x7f6e9006e830 2024-01-15T18:04:23.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e8ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6e90144b90 0x7f6e900b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:23.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e87e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6e9015b840 0x7f6e9006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35476/0 (socket says 172.21.15.2:35476) 2024-01-15T18:04:23.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.586+0000 7f6e87e00640 1 -- 172.21.15.2:0/3579986633 learned_addr learned my addr 172.21.15.2:0/3579986633 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8f400640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e9006ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8ea00640 1 -- 172.21.15.2:0/3579986633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 msgr2=0x7f6e9006ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8ea00640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e9006ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8ea00640 1 -- 172.21.15.2:0/3579986633 >> v1:172.21.15.2:6789/0 conn(0x7f6e9015b840 legacy=0x7f6e9006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8e000640 1 -- 172.21.15.2:0/3579986633 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318222867 0 0) 0x7f6e9006f1b0 con 0x7f6e9015b840 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8ea00640 1 -- 172.21.15.2:0/3579986633 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90144f70 con 0x7f6e90144b90 2024-01-15T18:04:23.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.587+0000 7f6e8f400640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e9006ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.588+0000 7f6e8ea00640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e90144b90 0x7f6e900b4220 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6e800097a0 tx=0x7f6e8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f8907319c6fa7d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e8e000640 1 -- 172.21.15.2:0/3579986633 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80031830 con 0x7f6e90144b90 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e8e000640 1 -- 172.21.15.2:0/3579986633 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e80031990 con 0x7f6e90144b90 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e8e000640 1 -- 172.21.15.2:0/3579986633 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80031d20 con 0x7f6e90144b90 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 -- 172.21.15.2:0/3579986633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e90144b90 msgr2=0x7f6e900b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e90144b90 0x7f6e900b4220 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6e800097a0 tx=0x7f6e8002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 -- 172.21.15.2:0/3579986633 shutdown_connections 2024-01-15T18:04:23.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e9006ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 --2- 172.21.15.2:0/3579986633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e90144b90 0x7f6e900b4220 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 -- 172.21.15.2:0/3579986633 >> 172.21.15.2:0/3579986633 conn(0x7f6e900b2c10 msgr2=0x7f6e900b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:23.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 -- 172.21.15.2:0/3579986633 shutdown_connections 2024-01-15T18:04:23.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.589+0000 7f6e95c00640 1 -- 172.21.15.2:0/3579986633 wait complete. 2024-01-15T18:04:23.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.591+0000 7f6e95c00640 1 Processor -- start 2024-01-15T18:04:23.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.591+0000 7f6e95c00640 1 -- start start 2024-01-15T18:04:23.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.592+0000 7f6e95c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e900b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:23.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.592+0000 7f6e95c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e90144b90 0x7f6e90143300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:23.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.592+0000 7f6e95c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 0x7f6e90151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:23.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.592+0000 7f6e95c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90138c20 con 0x7f6e9015b840 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.592+0000 7f6e95c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e901520c0 con 0x7f6e90144b90 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.592+0000 7f6e95c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90137fa0 con 0x7f6e9006e830 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 0x7f6e90151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 0x7f6e90151b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38030/0 (socket says 172.21.15.2:38030) 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 -- 172.21.15.2:0/4279161644 learned_addr learned my addr 172.21.15.2:0/4279161644 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8ea00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e900b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e87e00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e90144b90 0x7f6e90143300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 -- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 msgr2=0x7f6e900b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e900b2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 -- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e90144b90 msgr2=0x7f6e90143300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e90144b90 0x7f6e90143300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.593+0000 7f6e8f400640 1 -- 172.21.15.2:0/4279161644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80009400 con 0x7f6e9015b840 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.594+0000 7f6e8f400640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 0x7f6e90151b80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800cd50 tx=0x7f6e8800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.594+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800f680 con 0x7f6e9015b840 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.594+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8800f800 con 0x7f6e9015b840 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.594+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8800fbb0 con 0x7f6e9015b840 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.594+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e90138280 con 0x7f6e9015b840 2024-01-15T18:04:23.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.594+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e90138840 con 0x7f6e9015b840 2024-01-15T18:04:23.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.595+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e90138d60 con 0x7f6e9015b840 2024-01-15T18:04:23.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.596+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f6e8800b040 con 0x7f6e9015b840 2024-01-15T18:04:23.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.597+0000 7f6e86a00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806ba10 0x7f6e6806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:23.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.598+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6e8808b4e0 con 0x7f6e9015b840 2024-01-15T18:04:23.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.598+0000 7f6e8ea00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806ba10 0x7f6e6806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:23.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.599+0000 7f6e8ea00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806ba10 0x7f6e6806ded0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e80036000 tx=0x7f6e8003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:23.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.602+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e90138d60 con 0x7f6e9015b840 2024-01-15T18:04:23.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:23.835+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27426", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e90070c50 con 0x7f6e6806ba10 2024-01-15T18:04:24.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.002+0000 7f6e86a00640 1 -- 172.21.15.2:0/4279161644 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6e8003a420 con 0x7f6e6806ba10 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806ba10 msgr2=0x7f6e6806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806ba10 0x7f6e6806ded0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6e80036000 tx=0x7f6e8003c040 comp rx=0 tx=0).stop 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 msgr2=0x7f6e90151b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 0x7f6e90151b80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800cd50 tx=0x7f6e8800a650 comp rx=0 tx=0).stop 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 shutdown_connections 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e6806ba10 0x7f6e6806ded0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e9015b840 0x7f6e90151b80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e90144b90 0x7f6e90143300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 --2- 172.21.15.2:0/4279161644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e9006e830 0x7f6e900b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 >> 172.21.15.2:0/4279161644 conn(0x7f6e900b2c10 msgr2=0x7f6e900b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 shutdown_connections 2024-01-15T18:04:24.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.006+0000 7f6e95c00640 1 -- 172.21.15.2:0/4279161644 wait complete. 2024-01-15T18:04:24.030 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone27427 2024-01-15T18:04:24.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.251+0000 7f7c79c00640 1 Processor -- start 2024-01-15T18:04:24.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.252+0000 7f7c79c00640 1 -- start start 2024-01-15T18:04:24.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.253+0000 7f7c79c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 0x7f7c740b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.253+0000 7f7c79c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c74162110 con 0x7f7c740cfe30 2024-01-15T18:04:24.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.253+0000 7f7c79c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c740d3bd0 con 0x7f7c740b3cb0 2024-01-15T18:04:24.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.253+0000 7f7c79c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740d3eb0 con 0x7f7c740cfa50 2024-01-15T18:04:24.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.254+0000 7f7c79200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7c740cfe30 0x7f7c74161a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35482/0 (socket says 172.21.15.2:35482) 2024-01-15T18:04:24.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.254+0000 7f7c79200640 1 -- 172.21.15.2:0/2225464740 learned_addr learned my addr 172.21.15.2:0/2225464740 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:24.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.254+0000 7f7c73e00640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 0x7f7c740b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.254+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703751084 0 0) 0x7f7c74162110 con 0x7f7c740cfe30 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.254+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c54003520 con 0x7f7c740cfe30 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 56436613 0 0) 0x7f7c54003520 con 0x7f7c740cfe30 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 msgr2=0x7f7c740b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 0x7f7c740b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 >> v1:172.21.15.137:6789/0 conn(0x7f7c740b3cb0 legacy=0x7f7c741612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740d18a0 con 0x7f7c740cfe30 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7c6c0019f0 con 0x7f7c740cfe30 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 >> v1:172.21.15.2:6789/0 conn(0x7f7c740cfe30 legacy=0x7f7c74161a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c540039a0 0x7f7c54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 0x7f7c540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c54004d00 0x7f7c54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740d3eb0 con 0x7f7c54004d00 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5400ba60 con 0x7f7c540039a0 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5400bc20 con 0x7f7c740cfa50 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c79200640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c54004d00 0x7f7c54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.255+0000 7f7c6be00640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 0x7f7c540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.256+0000 7f7c73e00640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c540039a0 0x7f7c54003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.256+0000 7f7c79200640 1 -- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 msgr2=0x7f7c540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.256+0000 7f7c79200640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740cfa50 0x7f7c540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.256+0000 7f7c79200640 1 -- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c540039a0 msgr2=0x7f7c54003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.256+0000 7f7c79200640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c540039a0 0x7f7c54003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.256+0000 7f7c79200640 1 -- 172.21.15.2:0/2225464740 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54003520 con 0x7f7c54004d00 2024-01-15T18:04:24.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.257+0000 7f7c79200640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c54004d00 0x7f7c54005180 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c002230 tx=0x7f7c540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4330b9a5db16a5db server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:24.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.257+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c034630 con 0x7f7c54004d00 2024-01-15T18:04:24.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.257+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c02f070 con 0x7f7c54004d00 2024-01-15T18:04:24.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.257+0000 7f7c73400640 1 -- 172.21.15.2:0/2225464740 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c034a10 con 0x7f7c54004d00 2024-01-15T18:04:24.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79c00640 1 -- 172.21.15.2:0/2225464740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c54004d00 msgr2=0x7f7c54005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79c00640 1 --2- 172.21.15.2:0/2225464740 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c54004d00 0x7f7c54005180 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c002230 tx=0x7f7c540057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79200640 1 -- 172.21.15.2:0/2225464740 reap_dead start 2024-01-15T18:04:24.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79c00640 1 -- 172.21.15.2:0/2225464740 shutdown_connections 2024-01-15T18:04:24.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79c00640 1 -- 172.21.15.2:0/2225464740 >> 172.21.15.2:0/2225464740 conn(0x7f7c740d30b0 msgr2=0x7f7c740d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:24.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79c00640 1 -- 172.21.15.2:0/2225464740 shutdown_connections 2024-01-15T18:04:24.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.258+0000 7f7c79c00640 1 -- 172.21.15.2:0/2225464740 wait complete. 2024-01-15T18:04:24.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.260+0000 7f7c79c00640 1 Processor -- start 2024-01-15T18:04:24.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.260+0000 7f7c79c00640 1 -- start start 2024-01-15T18:04:24.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c79c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c54004d00 0x7f7c740cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c79c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 0x7f7c7414c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c79c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740b5c30 0x7f7c740b60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c79c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7414ca90 con 0x7f7c7414c0d0 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c79c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740b68d0 con 0x7f7c54004d00 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c79c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740b6bb0 con 0x7f7c740b5c30 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 0x7f7c7414c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 0x7f7c7414c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38048/0 (socket says 172.21.15.2:38048) 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.261+0000 7f7c6be00640 1 -- 172.21.15.2:0/625751027 learned_addr learned my addr 172.21.15.2:0/625751027 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c79200640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740b5c30 0x7f7c740b60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c73e00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c54004d00 0x7f7c740cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c6be00640 1 -- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740b5c30 msgr2=0x7f7c740b60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c6be00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740b5c30 0x7f7c740b60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c6be00640 1 -- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c54004d00 msgr2=0x7f7c740cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c6be00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c54004d00 0x7f7c740cbd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c6be00640 1 -- 172.21.15.2:0/625751027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740b6e90 con 0x7f7c7414c0d0 2024-01-15T18:04:24.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c79200640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740b5c30 0x7f7c740b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c6be00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 0x7f7c7414c550 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00e4d0 tx=0x7f7c5c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c0053a0 con 0x7f7c7414c0d0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5c0035f0 con 0x7f7c7414c0d0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c740b7170 con 0x7f7c7414c0d0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5c003980 con 0x7f7c7414c0d0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.262+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c740b65f0 con 0x7f7c7414c0d0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.263+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c74074770 con 0x7f7c7414c0d0 2024-01-15T18:04:24.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.265+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f7c5c003ae0 con 0x7f7c7414c0d0 2024-01-15T18:04:24.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.265+0000 7f7c70c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c4c06ba10 0x7f7c4c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.266+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f7c5c08abb0 con 0x7f7c7414c0d0 2024-01-15T18:04:24.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.266+0000 7f7c73e00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c4c06ba10 0x7f7c4c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.267+0000 7f7c73e00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c4c06ba10 0x7f7c4c06ded0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c741f6d60 tx=0x7f7c64008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:24.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.270+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c5c088030 con 0x7f7c7414c0d0 2024-01-15T18:04:24.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.499+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone27427", "target": ["mon-mgr", ""]}) v1 -- 0x7f7c7415e590 con 0x7f7c4c06ba10 2024-01-15T18:04:24.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.679+0000 7f7c70c00640 1 -- 172.21.15.2:0/625751027 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7c7415e590 con 0x7f7c4c06ba10 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c4c06ba10 msgr2=0x7f7c4c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c4c06ba10 0x7f7c4c06ded0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c741f6d60 tx=0x7f7c64008040 comp rx=0 tx=0).stop 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 msgr2=0x7f7c7414c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 0x7f7c7414c550 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c00e4d0 tx=0x7f7c5c00e990 comp rx=0 tx=0).stop 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 shutdown_connections 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c4c06ba10 0x7f7c4c06ded0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7c740b5c30 0x7f7c740b60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7c7414c0d0 0x7f7c7414c550 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 --2- 172.21.15.2:0/625751027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7c54004d00 0x7f7c740cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 >> 172.21.15.2:0/625751027 conn(0x7f7c740d30b0 msgr2=0x7f7c740b2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 shutdown_connections 2024-01-15T18:04:24.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.683+0000 7f7c79c00640 1 -- 172.21.15.2:0/625751027 wait complete. 2024-01-15T18:04:24.711 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:04:24.711 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27421 2024-01-15T18:04:24.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.929+0000 7f9561000640 1 Processor -- start 2024-01-15T18:04:24.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.930+0000 7f9561000640 1 -- start start 2024-01-15T18:04:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.931+0000 7f9561000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 0x7f955c0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f9561000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c071b40 con 0x7f955c071230 2024-01-15T18:04:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f9561000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955c0b2fa0 con 0x7f955c143620 2024-01-15T18:04:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f9561000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0b3280 con 0x7f955c0716b0 2024-01-15T18:04:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f9552800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f955c071230 0x7f955c0cc3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35486/0 (socket says 172.21.15.2:35486) 2024-01-15T18:04:24.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f9552800640 1 -- 172.21.15.2:0/3742492285 learned_addr learned my addr 172.21.15.2:0/3742492285 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f955be00640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 0x7f955c0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420749695 0 0) 0x7f955c071b40 con 0x7f955c071230 2024-01-15T18:04:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.932+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f953c003520 con 0x7f955c071230 2024-01-15T18:04:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2433813543 0 0) 0x7f953c003520 con 0x7f955c071230 2024-01-15T18:04:24.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 msgr2=0x7f955c0ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 0x7f955c0ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 >> v1:172.21.15.137:6789/0 conn(0x7f955c143620 legacy=0x7f955c0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c0b3560 con 0x7f955c071230 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9544001670 con 0x7f955c071230 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 >> v1:172.21.15.2:6789/0 conn(0x7f955c071230 legacy=0x7f955c0cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f953c0039a0 0x7f953c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f953c0047c0 0x7f953c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 0x7f953c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0b3280 con 0x7f953c0047c0 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c00ba60 con 0x7f953c0039a0 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f953c00de30 con 0x7f955c0716b0 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.933+0000 7f9552800640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f953c0047c0 0x7f953c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f955b400640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f953c0039a0 0x7f953c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f955be00640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 0x7f953c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f9552800640 1 -- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 msgr2=0x7f953c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f9552800640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f955c0716b0 0x7f953c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f9552800640 1 -- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f953c0039a0 msgr2=0x7f953c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f9552800640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f953c0039a0 0x7f953c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f9552800640 1 -- 172.21.15.2:0/3742492285 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c003520 con 0x7f953c0047c0 2024-01-15T18:04:24.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.934+0000 7f955b400640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f953c0039a0 0x7f953c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.935+0000 7f9552800640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f953c0047c0 0x7f953c004c40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9544001eb0 tx=0x7f953c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da87eaa1b85e297a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.935+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9544005ad0 con 0x7f953c0047c0 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.935+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9544005c50 con 0x7f953c0047c0 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.935+0000 7f955aa00640 1 -- 172.21.15.2:0/3742492285 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954402c7c0 con 0x7f953c0047c0 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f9561000640 1 -- 172.21.15.2:0/3742492285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f953c0047c0 msgr2=0x7f953c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f9561000640 1 --2- 172.21.15.2:0/3742492285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f953c0047c0 0x7f953c004c40 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9544001eb0 tx=0x7f953c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f955be00640 1 -- 172.21.15.2:0/3742492285 reap_dead start 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f9561000640 1 -- 172.21.15.2:0/3742492285 shutdown_connections 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f9561000640 1 -- 172.21.15.2:0/3742492285 >> 172.21.15.2:0/3742492285 conn(0x7f955c0b0840 msgr2=0x7f955c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f9561000640 1 -- 172.21.15.2:0/3742492285 shutdown_connections 2024-01-15T18:04:24.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.936+0000 7f9561000640 1 -- 172.21.15.2:0/3742492285 wait complete. 2024-01-15T18:04:24.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.938+0000 7f9561000640 1 Processor -- start 2024-01-15T18:04:24.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.938+0000 7f9561000640 1 -- start start 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.939+0000 7f9561000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f953c0047c0 0x7f955c0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.939+0000 7f9561000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f955c0cef10 0x7f955c1398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.939+0000 7f9561000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 0x7f955c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.939+0000 7f9561000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c0ce2b0 con 0x7f955c1540d0 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.939+0000 7f9561000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f955c139e00 con 0x7f955c0cef10 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.939+0000 7f9561000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955c153030 con 0x7f953c0047c0 2024-01-15T18:04:24.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 0x7f955c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 0x7f955c06d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38074/0 (socket says 172.21.15.2:38074) 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 -- 172.21.15.2:0/3190669554 learned_addr learned my addr 172.21.15.2:0/3190669554 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955b400640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f953c0047c0 0x7f955c0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f9552800640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f955c0cef10 0x7f955c1398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 -- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f953c0047c0 msgr2=0x7f955c0cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f953c0047c0 0x7f955c0cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 -- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f955c0cef10 msgr2=0x7f955c1398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f955c0cef10 0x7f955c1398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:24.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 -- 172.21.15.2:0/3190669554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955c153310 con 0x7f955c1540d0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.940+0000 7f955be00640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 0x7f955c06d030 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f955c06bb40 tx=0x7f955400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.941+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554017400 con 0x7f955c1540d0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.941+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9554011040 con 0x7f955c1540d0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.941+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95540177c0 con 0x7f955c1540d0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.941+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f955c1535f0 con 0x7f955c1540d0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.941+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f955c153bb0 con 0x7f955c1540d0 2024-01-15T18:04:24.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.942+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f955c074770 con 0x7f955c1540d0 2024-01-15T18:04:24.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.943+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f9554017920 con 0x7f955c1540d0 2024-01-15T18:04:24.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.944+0000 7f9553e00640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f953406ba10 0x7f953406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:24.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.945+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f955408b380 con 0x7f955c1540d0 2024-01-15T18:04:24.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.945+0000 7f955b400640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f953406ba10 0x7f953406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:24.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.946+0000 7f955b400640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f953406ba10 0x7f953406ded0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f954c001790 tx=0x7f954c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:24.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:24.949+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95540886f0 con 0x7f955c1540d0 2024-01-15T18:04:25.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.182+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27421", "target": ["mon-mgr", ""]}) v1 -- 0x7f955c143a00 con 0x7f953406ba10 2024-01-15T18:04:25.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.204+0000 7f9553e00640 1 -- 172.21.15.2:0/3190669554 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f954c00d050 con 0x7f953406ba10 2024-01-15T18:04:25.208 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:25.208 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:25.208 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:04:25.208 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:04:25.208 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 0.0, 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "0/3", 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:25.209 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.206+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f953406ba10 msgr2=0x7f953406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.206+0000 7f9561000640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f953406ba10 0x7f953406ded0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f954c001790 tx=0x7f954c009040 comp rx=0 tx=0).stop 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.206+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 msgr2=0x7f955c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.206+0000 7f9561000640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 0x7f955c06d030 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f955c06bb40 tx=0x7f955400f4a0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 shutdown_connections 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f953406ba10 0x7f953406ded0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f955c1540d0 0x7f955c06d030 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f955c0cef10 0x7f955c1398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 --2- 172.21.15.2:0/3190669554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f953c0047c0 0x7f955c0cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 >> 172.21.15.2:0/3190669554 conn(0x7f955c0b0840 msgr2=0x7f955c0af860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:25.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 shutdown_connections 2024-01-15T18:04:25.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.207+0000 7f9561000640 1 -- 172.21.15.2:0/3190669554 wait complete. 2024-01-15T18:04:25.230 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:04:25.230 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27427 2024-01-15T18:04:25.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.456+0000 7f2644400640 1 Processor -- start 2024-01-15T18:04:25.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.456+0000 7f2644400640 1 -- start start 2024-01-15T18:04:25.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2644400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f263c15ae80 0x7f263c0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2644400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c15dbc0 con 0x7f263c0b2b40 2024-01-15T18:04:25.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2644400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263c15dea0 con 0x7f263c06c8f0 2024-01-15T18:04:25.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2644400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f263c15e840 con 0x7f263c15ae80 2024-01-15T18:04:25.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2642400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f263c0b2b40 0x7f263c161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35496/0 (socket says 172.21.15.2:35496) 2024-01-15T18:04:25.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2642400640 1 -- 172.21.15.2:0/2618934065 learned_addr learned my addr 172.21.15.2:0/2618934065 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:25.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.458+0000 7f2641a00640 1 --2- 172.21.15.2:0/2618934065 >> v2:172.21.15.137:3300/0 conn(0x7f263c15ae80 0x7f263c0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364279288 0 0) 0x7f263c15dbc0 con 0x7f263c0b2b40 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c003520 con 0x7f263c0b2b40 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242776826 0 0) 0x7f261c003520 con 0x7f263c0b2b40 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 >> v2:172.21.15.137:3300/0 conn(0x7f263c15ae80 msgr2=0x7f263c0acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 --2- 172.21.15.2:0/2618934065 >> v2:172.21.15.137:3300/0 conn(0x7f263c15ae80 0x7f263c0acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 >> v1:172.21.15.137:6789/0 conn(0x7f263c06c8f0 legacy=0x7f263c0b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c15e180 con 0x7f263c0b2b40 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f26340019f0 con 0x7f263c0b2b40 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 >> v1:172.21.15.2:6789/0 conn(0x7f263c0b2b40 legacy=0x7f263c161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0039c0 0x7f261c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f261c0047c0 0x7f261c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f261c005180 0x7f261c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c15e840 con 0x7f261c0047c0 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00c080 con 0x7f261c0039c0 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.459+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f261c00c240 con 0x7f261c005180 2024-01-15T18:04:25.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641000640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f261c0047c0 0x7f261c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641a00640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0039c0 0x7f261c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2642400640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f261c005180 0x7f261c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641000640 1 -- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f261c005180 msgr2=0x7f261c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641000640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f261c005180 0x7f261c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641000640 1 -- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0039c0 msgr2=0x7f261c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641000640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0039c0 0x7f261c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2641000640 1 -- 172.21.15.2:0/2618934065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261c003520 con 0x7f261c0047c0 2024-01-15T18:04:25.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.460+0000 7f2642400640 1 -- 172.21.15.2:0/2618934065 reap_dead start 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.461+0000 7f2641000640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f261c0047c0 0x7f261c004c40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2624001750 tx=0x7f261c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e46ed4f1bf04dec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.461+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240052d0 con 0x7f261c0047c0 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262402e050 con 0x7f261c0047c0 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f263be00640 1 -- 172.21.15.2:0/2618934065 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2624003830 con 0x7f261c0047c0 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 -- 172.21.15.2:0/2618934065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f261c0047c0 msgr2=0x7f261c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f261c0047c0 0x7f261c004c40 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2624001750 tx=0x7f261c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 -- 172.21.15.2:0/2618934065 shutdown_connections 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 --2- 172.21.15.2:0/2618934065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f261c0047c0 0x7f261c004c40 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 -- 172.21.15.2:0/2618934065 >> 172.21.15.2:0/2618934065 conn(0x7f263c0b45f0 msgr2=0x7f263c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 -- 172.21.15.2:0/2618934065 shutdown_connections 2024-01-15T18:04:25.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.462+0000 7f2644400640 1 -- 172.21.15.2:0/2618934065 wait complete. 2024-01-15T18:04:25.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.464+0000 7f2644400640 1 Processor -- start 2024-01-15T18:04:25.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.464+0000 7f2644400640 1 -- start start 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2644400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0047c0 0x7f263c1f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2644400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 0x7f263c14c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2644400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f263c0b1860 0x7f263c14c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2644400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c14e440 con 0x7f263c0b13e0 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2644400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263c1f3f30 con 0x7f261c0047c0 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2644400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263c1f74a0 con 0x7f263c0b1860 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.465+0000 7f2641000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 0x7f263c14c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 0x7f263c14c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38098/0 (socket says 172.21.15.2:38098) 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 -- 172.21.15.2:0/935916058 learned_addr learned my addr 172.21.15.2:0/935916058 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641a00640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0047c0 0x7f263c1f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2642400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f263c0b1860 0x7f263c14c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 -- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f263c0b1860 msgr2=0x7f263c14c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f263c0b1860 0x7f263c14c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 -- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0047c0 msgr2=0x7f263c1f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0047c0 0x7f263c1f6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 -- 172.21.15.2:0/935916058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263c1f33b0 con 0x7f263c0b13e0 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641a00640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0047c0 0x7f263c1f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.466+0000 7f2641000640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 0x7f263c14c490 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2624000c00 tx=0x7f2624001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.467+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240076c0 con 0x7f263c0b13e0 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.467+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2624007a30 con 0x7f263c0b13e0 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.467+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2624006be0 con 0x7f263c0b13e0 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.467+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263c1f3690 con 0x7f263c0b13e0 2024-01-15T18:04:25.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.467+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263c1f3c50 con 0x7f263c0b13e0 2024-01-15T18:04:25.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.468+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f263c0686f0 con 0x7f263c0b13e0 2024-01-15T18:04:25.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.469+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f2624006d40 con 0x7f263c0b13e0 2024-01-15T18:04:25.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.470+0000 7f2639600640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f261406ba10 0x7f261406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:25.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.470+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f26240aa130 con 0x7f263c0b13e0 2024-01-15T18:04:25.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.471+0000 7f2641a00640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f261406ba10 0x7f261406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:25.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.472+0000 7f2641a00640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f261406ba10 0x7f261406ded0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f263c0717d0 tx=0x7f262c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:25.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.474+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26240a9e40 con 0x7f263c0b13e0 2024-01-15T18:04:25.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.780+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27427", "target": ["mon-mgr", ""]}) v1 -- 0x7f263c0acd00 con 0x7f261406ba10 2024-01-15T18:04:25.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.821+0000 7f2639600640 1 -- 172.21.15.2:0/935916058 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f262c009710 con 0x7f261406ba10 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:25.829 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:25.830 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:25.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f261406ba10 msgr2=0x7f261406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f261406ba10 0x7f261406ded0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f263c0717d0 tx=0x7f262c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:25.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 msgr2=0x7f263c14c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:25.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 0x7f263c14c490 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f2624000c00 tx=0x7f2624001120 comp rx=0 tx=0).stop 2024-01-15T18:04:25.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 shutdown_connections 2024-01-15T18:04:25.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f261406ba10 0x7f261406ded0 secure :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f263c0717d0 tx=0x7f262c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:25.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f263c0b1860 0x7f263c14c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f263c0b13e0 0x7f263c14c490 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 --2- 172.21.15.2:0/935916058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f261c0047c0 0x7f263c1f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:25.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 >> 172.21.15.2:0/935916058 conn(0x7f263c0b45f0 msgr2=0x7f263c0ad090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:25.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.835+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 shutdown_connections 2024-01-15T18:04:25.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:25.836+0000 7f2644400640 1 -- 172.21.15.2:0/935916058 wait complete. 2024-01-15T18:04:25.889 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:26.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.150+0000 7fe9e9600640 1 Processor -- start 2024-01-15T18:04:26.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.150+0000 7fe9e9600640 1 -- start start 2024-01-15T18:04:26.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.152+0000 7fe9e9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e415d390 0x7fe9e40ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:26.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.152+0000 7fe9e9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe9e40d3bb0 0x7fe9e40d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:26.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.153+0000 7fe9e9600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e40cf0f0 con 0x7fe9e40cfa50 2024-01-15T18:04:26.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.153+0000 7fe9e9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e40d0e80 con 0x7fe9e40d3bb0 2024-01-15T18:04:26.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.153+0000 7fe9e9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e40d1160 con 0x7fe9e415d390 2024-01-15T18:04:26.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.153+0000 7fe9e2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e415d390 0x7fe9e40ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:26.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.154+0000 7fe9e2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e415d390 0x7fe9e40ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:45040/0 (socket says 172.21.15.2:45040) 2024-01-15T18:04:26.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.154+0000 7fe9e2a00640 1 -- 172.21.15.2:0/1667849990 learned_addr learned my addr 172.21.15.2:0/1667849990 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:26.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.155+0000 7fe9e2000640 1 --2- 172.21.15.2:0/1667849990 >> v2:172.21.15.137:3300/0 conn(0x7fe9e40d3bb0 0x7fe9e40d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:26.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.156+0000 7fe9e1600640 1 -- 172.21.15.2:0/1667849990 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981731720 0 0) 0x7fe9e40cf0f0 con 0x7fe9e40cfa50 2024-01-15T18:04:26.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.156+0000 7fe9e2000640 1 -- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e415d390 msgr2=0x7fe9e40ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:26.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.156+0000 7fe9e2000640 1 --2- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e415d390 0x7fe9e40ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.156+0000 7fe9e2000640 1 -- 172.21.15.2:0/1667849990 >> v1:172.21.15.137:6789/0 conn(0x7fe9e40cfa50 legacy=0x7fe9e40b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:26.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.156+0000 7fe9e2000640 1 -- 172.21.15.2:0/1667849990 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e40d1440 con 0x7fe9e40d3bb0 2024-01-15T18:04:26.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.159+0000 7fe9e2000640 1 --2- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40d3bb0 0x7fe9e40d1c40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0097a0 tx=0x7fe9cc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1099aeae70d01c9f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:26.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.163+0000 7fe9e1600640 1 -- 172.21.15.2:0/1667849990 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc031780 con 0x7fe9e40d3bb0 2024-01-15T18:04:26.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.163+0000 7fe9e1600640 1 -- 172.21.15.2:0/1667849990 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc0318e0 con 0x7fe9e40d3bb0 2024-01-15T18:04:26.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.163+0000 7fe9e1600640 1 -- 172.21.15.2:0/1667849990 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc031c70 con 0x7fe9e40d3bb0 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 -- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40d3bb0 msgr2=0x7fe9e40d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 --2- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40d3bb0 0x7fe9e40d1c40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc0097a0 tx=0x7fe9cc02b820 comp rx=0 tx=0).stop 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 -- 172.21.15.2:0/1667849990 shutdown_connections 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 --2- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40d3bb0 0x7fe9e40d1c40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 --2- 172.21.15.2:0/1667849990 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e415d390 0x7fe9e40ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 -- 172.21.15.2:0/1667849990 >> 172.21.15.2:0/1667849990 conn(0x7fe9e409c560 msgr2=0x7fe9e40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.164+0000 7fe9e9600640 1 -- 172.21.15.2:0/1667849990 shutdown_connections 2024-01-15T18:04:26.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.165+0000 7fe9e9600640 1 -- 172.21.15.2:0/1667849990 wait complete. 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.168+0000 7fe9e9600640 1 Processor -- start 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.168+0000 7fe9e9600640 1 -- start start 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40cfa50 0x7fe9e4152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e40d3bb0 0x7fe9e415f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 0x7fe9e406bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e9600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4153970 con 0x7fe9e415d390 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e41573e0 con 0x7fe9e40cfa50 2024-01-15T18:04:26.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4153450 con 0x7fe9e40d3bb0 2024-01-15T18:04:26.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 0x7fe9e406bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:26.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 0x7fe9e406bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38108/0 (socket says 172.21.15.2:38108) 2024-01-15T18:04:26.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.170+0000 7fe9e3400640 1 -- 172.21.15.2:0/620421644 learned_addr learned my addr 172.21.15.2:0/620421644 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:26.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.171+0000 7fe9e3400640 1 -- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e40d3bb0 msgr2=0x7fe9e415f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:26.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.171+0000 7fe9e3400640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e40d3bb0 0x7fe9e415f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.171+0000 7fe9e3400640 1 -- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40cfa50 msgr2=0x7fe9e4152f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:26.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.171+0000 7fe9e3400640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40cfa50 0x7fe9e4152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.171+0000 7fe9e3400640 1 -- 172.21.15.2:0/620421644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc009400 con 0x7fe9e415d390 2024-01-15T18:04:26.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.172+0000 7fe9e3400640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 0x7fe9e406bfb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc001560 tx=0x7fe9dc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:26.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.172+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc003da0 con 0x7fe9e415d390 2024-01-15T18:04:26.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.173+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9dc005070 con 0x7fe9e415d390 2024-01-15T18:04:26.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.173+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e4157630 con 0x7fe9e415d390 2024-01-15T18:04:26.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.173+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc005420 con 0x7fe9e415d390 2024-01-15T18:04:26.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.173+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e4157bf0 con 0x7fe9e415d390 2024-01-15T18:04:26.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.174+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e40686f0 con 0x7fe9e415d390 2024-01-15T18:04:26.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.176+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fe9dc005580 con 0x7fe9e415d390 2024-01-15T18:04:26.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.177+0000 7fe9daa00640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9bc06ba10 0x7fe9bc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:26.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.178+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe9dc08a600 con 0x7fe9e415d390 2024-01-15T18:04:26.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.184+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9dc087b10 con 0x7fe9e415d390 2024-01-15T18:04:26.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.186+0000 7fe9e2a00640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9bc06ba10 0x7fe9bc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:26.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.189+0000 7fe9e2a00640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9bc06ba10 0x7fe9bc06ded0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4006f40 tx=0x7fe9d4006e90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:26.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.675+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe9e40d3920 con 0x7fe9e415d390 2024-01-15T18:04:26.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.676+0000 7fe9daa00640 1 -- 172.21.15.2:0/620421644 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fe9dc017090 con 0x7fe9e415d390 2024-01-15T18:04:26.680 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:26.680 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:26.680 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:26.681 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 434, 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:26.682 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:26.683 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 807, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3222347785, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 6737293312, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 766356819968, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 852, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 25504, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 0, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 7 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:26.684 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:26.685 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:26.686 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:26.687 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:26.688 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:26.688 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9bc06ba10 msgr2=0x7fe9bc06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9bc06ba10 0x7fe9bc06ded0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4006f40 tx=0x7fe9d4006e90 comp rx=0 tx=0).stop 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 msgr2=0x7fe9e406bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 0x7fe9e406bfb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc001560 tx=0x7fe9dc00cc70 comp rx=0 tx=0).stop 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 shutdown_connections 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9bc06ba10 0x7fe9bc06ded0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e415d390 0x7fe9e406bfb0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e40d3bb0 0x7fe9e415f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 --2- 172.21.15.2:0/620421644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9e40cfa50 0x7fe9e4152f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.679+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 >> 172.21.15.2:0/620421644 conn(0x7fe9e409c560 msgr2=0x7fe9e40cf310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:26.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.680+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 shutdown_connections 2024-01-15T18:04:26.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:26.680+0000 7fe9e9600640 1 -- 172.21.15.2:0/620421644 wait complete. 2024-01-15T18:04:27.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:28.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.029+0000 7f5f8be00640 1 Processor -- start 2024-01-15T18:04:28.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.029+0000 7f5f8be00640 1 -- start start 2024-01-15T18:04:28.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f8be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5f8c15d340 0x7f5f8c0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f8be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c0d1d90 0x7f5f8c0d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f8be00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8c0d3a60 con 0x7f5f8c0d3090 2024-01-15T18:04:28.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f8be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b3de0 con 0x7f5f8c15d340 2024-01-15T18:04:28.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f8be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b40c0 con 0x7f5f8c0d1d90 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f83e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5f8c0d3090 0x7f5f8c0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35512/0 (socket says 172.21.15.2:35512) 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.031+0000 7f5f83e00640 1 -- 172.21.15.2:0/2687508016 learned_addr learned my addr 172.21.15.2:0/2687508016 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8aa00640 1 --2- 172.21.15.2:0/2687508016 >> v2:172.21.15.137:3300/0 conn(0x7f5f8c15d340 0x7f5f8c0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053753439 0 0) 0x7f5f8c0d3a60 con 0x7f5f8c0d3090 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c003520 con 0x7f5f8c0d3090 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3668423035 0 0) 0x7f5f6c003520 con 0x7f5f8c0d3090 2024-01-15T18:04:28.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c0d1d90 msgr2=0x7f5f8c0d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c0d1d90 0x7f5f8c0d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 >> v2:172.21.15.137:3300/0 conn(0x7f5f8c15d340 msgr2=0x7f5f8c0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 --2- 172.21.15.2:0/2687508016 >> v2:172.21.15.137:3300/0 conn(0x7f5f8c15d340 0x7f5f8c0cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0d0e80 con 0x7f5f8c0d3090 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5f74001670 con 0x7f5f8c0d3090 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 >> v1:172.21.15.2:6789/0 conn(0x7f5f8c0d3090 legacy=0x7f5f8c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.032+0000 7f5f8a000640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c0d1d90 0x7f5f6c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8a000640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f6c003f40 0x7f5f6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8a000640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b3de0 con 0x7f5f6c004d00 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b40c0 con 0x7f5f6c003f40 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c00bb10 con 0x7f5f8c0d1d90 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8b400640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:28.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f83e00640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f6c003f40 0x7f5f6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:28.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8b400640 1 -- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c0d1d90 msgr2=0x7f5f6c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:28.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8b400640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c0d1d90 0x7f5f6c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8b400640 1 -- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f6c003f40 msgr2=0x7f5f6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:28.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.033+0000 7f5f8b400640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f6c003f40 0x7f5f6c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.034+0000 7f5f8b400640 1 -- 172.21.15.2:0/2687508016 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c003520 con 0x7f5f6c004d00 2024-01-15T18:04:28.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.035+0000 7f5f8b400640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f6c005180 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009e10 tx=0x7f5f6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8692c167ffa0289 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.035+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8402cd50 con 0x7f5f6c004d00 2024-01-15T18:04:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.035+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84034440 con 0x7f5f6c004d00 2024-01-15T18:04:28.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.035+0000 7f5f8a000640 1 -- 172.21.15.2:0/2687508016 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84034830 con 0x7f5f6c004d00 2024-01-15T18:04:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.036+0000 7f5f8be00640 1 -- 172.21.15.2:0/2687508016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 msgr2=0x7f5f6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.036+0000 7f5f8be00640 1 --2- 172.21.15.2:0/2687508016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f6c005180 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009e10 tx=0x7f5f6c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.037+0000 7f5f8b400640 1 -- 172.21.15.2:0/2687508016 reap_dead start 2024-01-15T18:04:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.037+0000 7f5f8be00640 1 -- 172.21.15.2:0/2687508016 shutdown_connections 2024-01-15T18:04:28.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.037+0000 7f5f8be00640 1 -- 172.21.15.2:0/2687508016 >> 172.21.15.2:0/2687508016 conn(0x7f5f8c08caf0 msgr2=0x7f5f8c09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:28.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.038+0000 7f5f8be00640 1 -- 172.21.15.2:0/2687508016 shutdown_connections 2024-01-15T18:04:28.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.038+0000 7f5f8be00640 1 -- 172.21.15.2:0/2687508016 wait complete. 2024-01-15T18:04:28.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.042+0000 7f5f8be00640 1 Processor -- start 2024-01-15T18:04:28.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.042+0000 7f5f8be00640 1 -- start start 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f8c0b08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f8c0b1f60 0x7f5f8c157680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c1f39b0 0x7f5f8c0c7550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c157bc0 con 0x7f5f6c004d00 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b0ec0 con 0x7f5f8c0b1f60 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c0b11a0 con 0x7f5f8c1f39b0 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f8c0b08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:28.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f8c0b08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38116/0 (socket says 172.21.15.2:38116) 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.043+0000 7f5f8aa00640 1 -- 172.21.15.2:0/138724325 learned_addr learned my addr 172.21.15.2:0/138724325 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8aa00640 1 -- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c1f39b0 msgr2=0x7f5f8c0c7550 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8aa00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c1f39b0 0x7f5f8c0c7550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8aa00640 1 -- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f8c0b1f60 msgr2=0x7f5f8c157680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8aa00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f8c0b1f60 0x7f5f8c157680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8aa00640 1 -- 172.21.15.2:0/138724325 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c0b1480 con 0x7f5f6c004d00 2024-01-15T18:04:28.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8aa00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f8c0b08e0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00e7c0 tx=0x7f5f7c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:28.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00eed0 con 0x7f5f6c004d00 2024-01-15T18:04:28.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c00c8c0 con 0x7f5f6c004d00 2024-01-15T18:04:28.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00cc70 con 0x7f5f6c004d00 2024-01-15T18:04:28.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.044+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c0b1760 con 0x7f5f6c004d00 2024-01-15T18:04:28.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.045+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c1f3e60 con 0x7f5f6c004d00 2024-01-15T18:04:28.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.046+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c074770 con 0x7f5f6c004d00 2024-01-15T18:04:28.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.047+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f5f7c013470 con 0x7f5f6c004d00 2024-01-15T18:04:28.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.048+0000 7f5f83400640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f6406ba60 0x7f5f6406df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:28.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.048+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5f7c08aec0 con 0x7f5f6c004d00 2024-01-15T18:04:28.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.050+0000 7f5f83e00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f6406ba60 0x7f5f6406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:28.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.053+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f7c088340 con 0x7f5f6c004d00 2024-01-15T18:04:28.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.054+0000 7f5f83e00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f6406ba60 0x7f5f6406df20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5f74001eb0 tx=0x7f5f74001e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:28.217 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:28.213+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:28.217 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:28.213+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:28.218 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:28.213+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:28.218 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:28.213+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:28.218 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:28.214+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:28.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.516+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5f8c0678e0 con 0x7f5f6c004d00 2024-01-15T18:04:28.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.517+0000 7f5f83400640 1 -- 172.21.15.2:0/138724325 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7f5f7c021940 con 0x7f5f6c004d00 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.522 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:28.523 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 436, 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:28.524 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:28.525 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 807, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3222347785, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 6737293312, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 766356819968, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 597, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 17826, 2024-01-15T18:04:28.526 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 0, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 4 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:28.527 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:28.528 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.529 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:28.530 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:28.531 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-15T18:04:28.531 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:04:28.531 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:28.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:28.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:28.531 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:28.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f6406ba60 msgr2=0x7f5f6406df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:28.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f6406ba60 0x7f5f6406df20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f5f74001eb0 tx=0x7f5f74001e40 comp rx=0 tx=0).stop 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 msgr2=0x7f5f8c0b08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f8c0b08e0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00e7c0 tx=0x7f5f7c00ec80 comp rx=0 tx=0).stop 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 shutdown_connections 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f6406ba60 0x7f5f6406df20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5f8c1f39b0 0x7f5f8c0c7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5f8c0b1f60 0x7f5f8c157680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 --2- 172.21.15.2:0/138724325 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5f6c004d00 0x7f5f8c0b08e0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 >> 172.21.15.2:0/138724325 conn(0x7f5f8c08caf0 msgr2=0x7f5f8c161490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.520+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 shutdown_connections 2024-01-15T18:04:28.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:28.521+0000 7f5f8be00640 1 -- 172.21.15.2:0/138724325 wait complete. 2024-01-15T18:04:30.508 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:30.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:04:30.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.853+0000 7f5bef200640 1 Processor -- start 2024-01-15T18:04:30.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.854+0000 7f5bef200640 1 -- start start 2024-01-15T18:04:30.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5bef200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5be806c5e0 0x7f5be8146630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:30.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5bef200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be81616f0 con 0x7f5be81460b0 2024-01-15T18:04:30.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5bef200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be8146c60 con 0x7f5be8161b80 2024-01-15T18:04:30.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5bef200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be8146f40 con 0x7f5be806c5e0 2024-01-15T18:04:30.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5be7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5be81460b0 0x7f5be806bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35528/0 (socket says 172.21.15.2:35528) 2024-01-15T18:04:30.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5be7e00640 1 -- 172.21.15.2:0/4064264474 learned_addr learned my addr 172.21.15.2:0/4064264474 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:30.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.856+0000 7f5be7400640 1 --2- 172.21.15.2:0/4064264474 >> v2:172.21.15.2:3300/0 conn(0x7f5be806c5e0 0x7f5be8146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:30.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.857+0000 7f5be7400640 1 -- 172.21.15.2:0/4064264474 >> v1:172.21.15.137:6789/0 conn(0x7f5be8161b80 legacy=0x7f5be80cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:30.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.857+0000 7f5be7400640 1 -- 172.21.15.2:0/4064264474 >> v1:172.21.15.2:6789/0 conn(0x7f5be81460b0 legacy=0x7f5be806bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:30.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.857+0000 7f5be6a00640 1 -- 172.21.15.2:0/4064264474 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334393607 0 0) 0x7f5be81616f0 con 0x7f5be81460b0 2024-01-15T18:04:30.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.857+0000 7f5be7400640 1 -- 172.21.15.2:0/4064264474 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be81610a0 con 0x7f5be806c5e0 2024-01-15T18:04:30.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.859+0000 7f5be7400640 1 --2- 172.21.15.2:0/4064264474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be806c5e0 0x7f5be8146630 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00097a0 tx=0x7f5bd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ac0b048d614457 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:30.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.859+0000 7f5be6a00640 1 -- 172.21.15.2:0/4064264474 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0031830 con 0x7f5be806c5e0 2024-01-15T18:04:30.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.859+0000 7f5be6a00640 1 -- 172.21.15.2:0/4064264474 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd0031990 con 0x7f5be806c5e0 2024-01-15T18:04:30.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.859+0000 7f5be6a00640 1 -- 172.21.15.2:0/4064264474 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0031d20 con 0x7f5be806c5e0 2024-01-15T18:04:30.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.861+0000 7f5bef200640 1 -- 172.21.15.2:0/4064264474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be806c5e0 msgr2=0x7f5be8146630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:30.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.861+0000 7f5bef200640 1 --2- 172.21.15.2:0/4064264474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be806c5e0 0x7f5be8146630 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f5bd00097a0 tx=0x7f5bd002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:04:30.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.862+0000 7f5bef200640 1 -- 172.21.15.2:0/4064264474 shutdown_connections 2024-01-15T18:04:30.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.862+0000 7f5bef200640 1 --2- 172.21.15.2:0/4064264474 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be806c5e0 0x7f5be8146630 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:30.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.862+0000 7f5bef200640 1 -- 172.21.15.2:0/4064264474 >> 172.21.15.2:0/4064264474 conn(0x7f5be806a8d0 msgr2=0x7f5be80661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:30.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.862+0000 7f5bef200640 1 -- 172.21.15.2:0/4064264474 shutdown_connections 2024-01-15T18:04:30.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.862+0000 7f5bef200640 1 -- 172.21.15.2:0/4064264474 wait complete. 2024-01-15T18:04:30.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.864+0000 7f5bef200640 1 Processor -- start 2024-01-15T18:04:30.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.864+0000 7f5bef200640 1 -- start start 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5bef200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5be806c5e0 0x7f5be815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5bef200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 0x7f5be81452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5bef200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5be8161b80 0x7f5be80b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5bef200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8145800 con 0x7f5be81460b0 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5bef200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be8141ef0 con 0x7f5be8161b80 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5bef200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be81e8ce0 con 0x7f5be806c5e0 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.865+0000 7f5be7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 0x7f5be81452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 0x7f5be81452c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38138/0 (socket says 172.21.15.2:38138) 2024-01-15T18:04:30.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 -- 172.21.15.2:0/2208724337 learned_addr learned my addr 172.21.15.2:0/2208724337 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5bed200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5be8161b80 0x7f5be80b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 -- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5be806c5e0 msgr2=0x7f5be815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5be806c5e0 0x7f5be815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 -- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5be8161b80 msgr2=0x7f5be80b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:30.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5be8161b80 0x7f5be80b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:30.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.866+0000 7f5be7400640 1 -- 172.21.15.2:0/2208724337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd0009400 con 0x7f5be81460b0 2024-01-15T18:04:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.867+0000 7f5be7400640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 0x7f5be81452c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5bd002bee0 tx=0x7f5bd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.868+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd003ba70 con 0x7f5be81460b0 2024-01-15T18:04:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.868+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd003bbd0 con 0x7f5be81460b0 2024-01-15T18:04:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.868+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be81e8f30 con 0x7f5be81460b0 2024-01-15T18:04:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.868+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd00453f0 con 0x7f5be81460b0 2024-01-15T18:04:30.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.868+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be81e94f0 con 0x7f5be81460b0 2024-01-15T18:04:30.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.870+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be80686f0 con 0x7f5be81460b0 2024-01-15T18:04:30.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.872+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 18) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f5bd0045550 con 0x7f5be81460b0 2024-01-15T18:04:30.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.873+0000 7f5bdfe00640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bc006baa0 0x7f5bc006df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:30.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.873+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5bd00aa560 con 0x7f5be81460b0 2024-01-15T18:04:30.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.877+0000 7f5be7e00640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bc006baa0 0x7f5bc006df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:30.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.880+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bd0076fd0 con 0x7f5be81460b0 2024-01-15T18:04:30.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:30.904+0000 7f5be7e00640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bc006baa0 0x7f5bc006df60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8007fd0 tx=0x7f5bd8007920 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:31.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.347+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5be80b09a0 con 0x7f5be81460b0 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.348+0000 7f5bdfe00640 1 -- 172.21.15.2:0/2208724337 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2421 (secure 0 0 0) 0x7f5bd007a5f0 con 0x7f5be81460b0 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:31.353 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:31.354 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 439, 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:31.355 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.356 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1073, 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 4346421222, 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 8991395840, 2024-01-15T18:04:31.357 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 764102717440, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 87349199, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 93655322, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 21, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 27 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:31.358 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:31.359 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:31.360 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:31.361 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.22220000624656677, 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 49s)", 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:31.362 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.350+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bc006baa0 msgr2=0x7f5bc006df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.350+0000 7f5bef200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bc006baa0 0x7f5bc006df60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f5bd8007fd0 tx=0x7f5bd8007920 comp rx=0 tx=0).stop 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.350+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 msgr2=0x7f5be81452c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.350+0000 7f5bef200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 0x7f5be81452c0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f5bd002bee0 tx=0x7f5bd00046c0 comp rx=0 tx=0).stop 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 shutdown_connections 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bc006baa0 0x7f5bc006df60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5be8161b80 0x7f5be80b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5be81460b0 0x7f5be81452c0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 --2- 172.21.15.2:0/2208724337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5be806c5e0 0x7f5be815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:31.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 >> 172.21.15.2:0/2208724337 conn(0x7f5be806a8d0 msgr2=0x7f5be80d0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:31.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 shutdown_connections 2024-01-15T18:04:31.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:31.351+0000 7f5bef200640 1 -- 172.21.15.2:0/2208724337 wait complete. 2024-01-15T18:04:33.247 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:33.243+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:33.247 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:33.243+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:33.248 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:33.243+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:33.248 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:33.243+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:33.248 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:04:33.243+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:04:33.378 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.672+0000 7f4d57e00640 1 Processor -- start 2024-01-15T18:04:33.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.672+0000 7f4d57e00640 1 -- start start 2024-01-15T18:04:33.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.674+0000 7f4d57e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4d500d2f10 0x7f4d500d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.674+0000 7f4d57e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d500ce480 0x7f4d500b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.674+0000 7f4d57e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d500ce930 con 0x7f4d5015d670 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.674+0000 7f4d57e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d500cec10 con 0x7f4d500ce480 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.674+0000 7f4d57e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d500ceef0 con 0x7f4d500d2f10 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.675+0000 7f4d54a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4d5015d670 0x7f4d500cdd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48808/0 (socket says 172.21.15.2:48808) 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.675+0000 7f4d54a00640 1 -- 172.21.15.2:0/2434653098 learned_addr learned my addr 172.21.15.2:0/2434653098 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:33.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.675+0000 7f4d55e00640 1 --2- 172.21.15.2:0/2434653098 >> v2:172.21.15.2:3300/0 conn(0x7f4d500ce480 0x7f4d500b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d4fe00640 1 -- 172.21.15.2:0/2434653098 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612678246 0 0) 0x7f4d500ce930 con 0x7f4d5015d670 2024-01-15T18:04:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d4fe00640 1 -- 172.21.15.2:0/2434653098 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003520 con 0x7f4d5015d670 2024-01-15T18:04:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d55e00640 1 -- 172.21.15.2:0/2434653098 >> v2:172.21.15.137:3300/0 conn(0x7f4d500d2f10 msgr2=0x7f4d500d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d55400640 1 --2- 172.21.15.2:0/2434653098 >> v2:172.21.15.137:3300/0 conn(0x7f4d500d2f10 0x7f4d500d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d55e00640 1 --2- 172.21.15.2:0/2434653098 >> v2:172.21.15.137:3300/0 conn(0x7f4d500d2f10 0x7f4d500d32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d4fe00640 1 -- 172.21.15.2:0/2434653098 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2199350266 0 0) 0x7f4d30003520 con 0x7f4d5015d670 2024-01-15T18:04:33.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d55e00640 1 -- 172.21.15.2:0/2434653098 >> v1:172.21.15.2:6789/0 conn(0x7f4d5015d670 legacy=0x7f4d500cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.676+0000 7f4d55e00640 1 -- 172.21.15.2:0/2434653098 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500cf1d0 con 0x7f4d500ce480 2024-01-15T18:04:33.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.678+0000 7f4d55e00640 1 --2- 172.21.15.2:0/2434653098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b2420 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4d500d4030 tx=0x7f4d4802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ed9f1403199f2a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.678+0000 7f4d4fe00640 1 -- 172.21.15.2:0/2434653098 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38001670 con 0x7f4d500ce480 2024-01-15T18:04:33.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.678+0000 7f4d4fe00640 1 -- 172.21.15.2:0/2434653098 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4802c8c0 con 0x7f4d500ce480 2024-01-15T18:04:33.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.678+0000 7f4d4fe00640 1 -- 172.21.15.2:0/2434653098 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4802cc50 con 0x7f4d500ce480 2024-01-15T18:04:33.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.679+0000 7f4d57e00640 1 -- 172.21.15.2:0/2434653098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 msgr2=0x7f4d500b2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.679+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2434653098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b2420 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f4d500d4030 tx=0x7f4d4802bd90 comp rx=0 tx=0).stop 2024-01-15T18:04:33.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.680+0000 7f4d57e00640 1 -- 172.21.15.2:0/2434653098 shutdown_connections 2024-01-15T18:04:33.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.680+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2434653098 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b2420 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.680+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2434653098 >> v2:172.21.15.137:3300/0 conn(0x7f4d500d2f10 0x7f4d500d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.680+0000 7f4d57e00640 1 -- 172.21.15.2:0/2434653098 >> 172.21.15.2:0/2434653098 conn(0x7f4d5008cc30 msgr2=0x7f4d5009c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:33.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.680+0000 7f4d57e00640 1 -- 172.21.15.2:0/2434653098 shutdown_connections 2024-01-15T18:04:33.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.680+0000 7f4d57e00640 1 -- 172.21.15.2:0/2434653098 wait complete. 2024-01-15T18:04:33.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.684+0000 7f4d57e00640 1 Processor -- start 2024-01-15T18:04:33.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.684+0000 7f4d57e00640 1 -- start start 2024-01-15T18:04:33.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.686+0000 7f4d57e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.686+0000 7f4d57e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 0x7f4d500cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.686+0000 7f4d57e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d500cad20 0x7f4d500ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.686+0000 7f4d57e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d500b7460 con 0x7f4d500ce480 2024-01-15T18:04:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.686+0000 7f4d57e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d500cd270 con 0x7f4d500cad20 2024-01-15T18:04:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.686+0000 7f4d57e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d500cb1e0 con 0x7f4d500d2f10 2024-01-15T18:04:33.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.687+0000 7f4d55400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.687+0000 7f4d54a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 0x7f4d500cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.688+0000 7f4d54a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 0x7f4d500cc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:56068/0 (socket says 172.21.15.2:56068) 2024-01-15T18:04:33.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.688+0000 7f4d54a00640 1 -- 172.21.15.2:0/2177105776 learned_addr learned my addr 172.21.15.2:0/2177105776 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.690+0000 7f4d54a00640 1 -- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d500cad20 msgr2=0x7f4d500ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.690+0000 7f4d54a00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d500cad20 0x7f4d500ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.690+0000 7f4d54a00640 1 -- 172.21.15.2:0/2177105776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 msgr2=0x7f4d500b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.690+0000 7f4d54a00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:33.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.690+0000 7f4d54a00640 1 -- 172.21.15.2:0/2177105776 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d48009a70 con 0x7f4d500d2f10 2024-01-15T18:04:33.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.692+0000 7f4d54a00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 0x7f4d500cc7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c780 tx=0x7f4d3800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:33.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.695+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38005ad0 con 0x7f4d500d2f10 2024-01-15T18:04:33.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.695+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d38005c50 con 0x7f4d500d2f10 2024-01-15T18:04:33.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.695+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38007070 con 0x7f4d500d2f10 2024-01-15T18:04:33.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.695+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d500c9710 con 0x7f4d500d2f10 2024-01-15T18:04:33.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.695+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d500c9cd0 con 0x7f4d500d2f10 2024-01-15T18:04:33.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.697+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d500b75a0 con 0x7f4d500d2f10 2024-01-15T18:04:33.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.704+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 18) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f4d380071d0 con 0x7f4d500d2f10 2024-01-15T18:04:33.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.705+0000 7f4d4d600640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d2806bb20 0x7f4d2806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:33.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.706+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f4d38088900 con 0x7f4d500d2f10 2024-01-15T18:04:33.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.713+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d38055370 con 0x7f4d500d2f10 2024-01-15T18:04:33.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.716+0000 7f4d55400640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d2806bb20 0x7f4d2806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:33.740+0000 7f4d55400640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d2806bb20 0x7f4d2806dfe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000a970 tx=0x7f4d40008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:34.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.197+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4d500b75a0 con 0x7f4d500d2f10 2024-01-15T18:04:34.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.199+0000 7f4d4d600640 1 -- 172.21.15.2:0/2177105776 <== mon.2 v2:172.21.15.137:3301/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2424 (secure 0 0 0) 0x7f4d500b75a0 con 0x7f4d500d2f10 2024-01-15T18:04:34.203 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:34.203 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:34.203 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:34.203 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:34.203 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 441, 2024-01-15T18:04:34.204 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:34.205 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1316, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 5369947929, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 11042693120, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 762051420160, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 178863977, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 178887005, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 43, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 50 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:34.206 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:34.207 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.208 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (5s)\n [======......................] (remaining: 18s)", 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.22220000624656677, 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 49s)", 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:34.209 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d2806bb20 msgr2=0x7f4d2806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d2806bb20 0x7f4d2806dfe0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4d4000a970 tx=0x7f4d40008040 comp rx=0 tx=0).stop 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 msgr2=0x7f4d500cc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 0x7f4d500cc7f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c780 tx=0x7f4d3800cc40 comp rx=0 tx=0).stop 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 shutdown_connections 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d2806bb20 0x7f4d2806dfe0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d500cad20 0x7f4d500ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d500d2f10 0x7f4d500cc7f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2177105776 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d500ce480 0x7f4d500b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:34.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 >> 172.21.15.2:0/2177105776 conn(0x7f4d5008cc30 msgr2=0x7f4d5006b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:34.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 shutdown_connections 2024-01-15T18:04:34.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:34.204+0000 7f4d57e00640 1 -- 172.21.15.2:0/2177105776 wait complete. 2024-01-15T18:04:36.239 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:36.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.544+0000 7f3172e00640 1 Processor -- start 2024-01-15T18:04:36.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.545+0000 7f3172e00640 1 -- start start 2024-01-15T18:04:36.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.549+0000 7f3172e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316c1608b0 0x7f316c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.549+0000 7f3172e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f316c0d39b0 0x7f316c0b3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.549+0000 7f3172e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f316c0d2150 con 0x7f316c0d1a50 2024-01-15T18:04:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.549+0000 7f3172e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f316c0b34d0 con 0x7f316c0d39b0 2024-01-15T18:04:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.549+0000 7f3172e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0d1ed0 con 0x7f316c1608b0 2024-01-15T18:04:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.550+0000 7f316b400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f316c0d1a50 0x7f316c0d32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:46642/0 (socket says 172.21.15.2:46642) 2024-01-15T18:04:36.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.550+0000 7f316b400640 1 -- 172.21.15.2:0/1837478894 learned_addr learned my addr 172.21.15.2:0/1837478894 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.551+0000 7f316be00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316c1608b0 0x7f316c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.551+0000 7f3170e00640 1 --2- 172.21.15.2:0/1837478894 >> v2:172.21.15.137:3300/0 conn(0x7f316c0d39b0 0x7f316c0b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093675531 0 0) 0x7f316c0d2150 con 0x7f316c0d1a50 2024-01-15T18:04:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f314c00bd30 con 0x7f316c0d1a50 2024-01-15T18:04:36.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2665099119 0 0) 0x7f314c00bd30 con 0x7f316c0d1a50 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316c1608b0 msgr2=0x7f316c0b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316c1608b0 0x7f316c0b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 >> v2:172.21.15.137:3300/0 conn(0x7f316c0d39b0 msgr2=0x7f316c0b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 --2- 172.21.15.2:0/1837478894 >> v2:172.21.15.137:3300/0 conn(0x7f316c0d39b0 0x7f316c0b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c160c90 con 0x7f316c0d1a50 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.552+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3154001670 con 0x7f316c0d1a50 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 >> v1:172.21.15.137:6789/0 conn(0x7f316c0d1a50 legacy=0x7f316c0d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00c180 0x7f314c00c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f314c00cfd0 0x7f314c00d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f314c00d990 0x7f314c009470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0b34d0 con 0x7f314c00d990 2024-01-15T18:04:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0d1ed0 con 0x7f314c00cfd0 2024-01-15T18:04:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c009b60 con 0x7f314c00c180 2024-01-15T18:04:36.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.553+0000 7f316be00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f316c1608b0 0x7f316c0b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:36.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.555+0000 7f316b400640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f314c00cfd0 0x7f314c00d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.556+0000 7f316be00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00c180 0x7f314c00c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.556+0000 7f3170e00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f314c00d990 0x7f314c009470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.558+0000 7f3170e00640 1 -- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00c180 msgr2=0x7f314c00c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.558+0000 7f3170e00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00c180 0x7f314c00c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.558+0000 7f3170e00640 1 -- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f314c00cfd0 msgr2=0x7f314c00d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.558+0000 7f3170e00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f314c00cfd0 0x7f314c00d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.558+0000 7f3170e00640 1 -- 172.21.15.2:0/1837478894 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c00bd30 con 0x7f314c00d990 2024-01-15T18:04:36.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.558+0000 7f316b400640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f314c00cfd0 0x7f314c00d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:04:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.561+0000 7f3170e00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f314c00d990 0x7f314c009470 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f316c0d3dc0 tx=0x7f314c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d948d63114d3672 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.561+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316402cc80 con 0x7f314c00d990 2024-01-15T18:04:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.561+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316402cde0 con 0x7f314c00d990 2024-01-15T18:04:36.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.561+0000 7f316aa00640 1 -- 172.21.15.2:0/1837478894 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3164034640 con 0x7f314c00d990 2024-01-15T18:04:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.563+0000 7f3172e00640 1 -- 172.21.15.2:0/1837478894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f314c00d990 msgr2=0x7f314c009470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.563+0000 7f3172e00640 1 --2- 172.21.15.2:0/1837478894 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f314c00d990 0x7f314c009470 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f316c0d3dc0 tx=0x7f314c00e000 comp rx=0 tx=0).stop 2024-01-15T18:04:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.564+0000 7f3170e00640 1 -- 172.21.15.2:0/1837478894 reap_dead start 2024-01-15T18:04:36.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.564+0000 7f3172e00640 1 -- 172.21.15.2:0/1837478894 shutdown_connections 2024-01-15T18:04:36.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.564+0000 7f3172e00640 1 -- 172.21.15.2:0/1837478894 >> 172.21.15.2:0/1837478894 conn(0x7f316c072660 msgr2=0x7f316c09c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:36.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.564+0000 7f3172e00640 1 -- 172.21.15.2:0/1837478894 shutdown_connections 2024-01-15T18:04:36.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.564+0000 7f3172e00640 1 -- 172.21.15.2:0/1837478894 wait complete. 2024-01-15T18:04:36.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.570+0000 7f3172e00640 1 Processor -- start 2024-01-15T18:04:36.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.570+0000 7f3172e00640 1 -- start start 2024-01-15T18:04:36.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.572+0000 7f3172e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00d990 0x7f316c14fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.572+0000 7f3172e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316c1503f0 0x7f316c0b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.572+0000 7f3172e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 0x7f316c154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.572+0000 7f3172e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c155170 con 0x7f316c1547a0 2024-01-15T18:04:36.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.572+0000 7f3172e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0b0c00 con 0x7f316c1503f0 2024-01-15T18:04:36.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.572+0000 7f3172e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f316c0b0ee0 con 0x7f314c00d990 2024-01-15T18:04:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.573+0000 7f3170e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 0x7f316c154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.573+0000 7f3170e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 0x7f316c154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35880/0 (socket says 172.21.15.2:35880) 2024-01-15T18:04:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.573+0000 7f3170e00640 1 -- 172.21.15.2:0/3040374632 learned_addr learned my addr 172.21.15.2:0/3040374632 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.573+0000 7f316be00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00d990 0x7f316c14fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.573+0000 7f316b400640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316c1503f0 0x7f316c0b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.573+0000 7f3170e00640 1 -- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00d990 msgr2=0x7f316c14fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3170e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00d990 0x7f316c14fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3170e00640 1 -- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316c1503f0 msgr2=0x7f316c0b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3170e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316c1503f0 0x7f316c0b6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3170e00640 1 -- 172.21.15.2:0/3040374632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f316c0b11c0 con 0x7f316c1547a0 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f316b400640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316c1503f0 0x7f316c0b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:04:36.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f316be00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00d990 0x7f316c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3170e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 0x7f316c154c30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f3164037040 tx=0x7f3164004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316403e640 con 0x7f316c1547a0 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f316403e7c0 con 0x7f316c1547a0 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316c0b14a0 con 0x7f316c1547a0 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f316403eb70 con 0x7f316c1547a0 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.575+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316c0b1ca0 con 0x7f316c1547a0 2024-01-15T18:04:36.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.577+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316c0b19c0 con 0x7f316c1547a0 2024-01-15T18:04:36.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.578+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f316403ecd0 con 0x7f316c1547a0 2024-01-15T18:04:36.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.580+0000 7f3163e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f314406bb30 0x7f314406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:36.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.580+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f31640aab90 con 0x7f316c1547a0 2024-01-15T18:04:36.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.590+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f316c0b19c0 con 0x7f316c1547a0 2024-01-15T18:04:36.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.598+0000 7f316be00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f314406bb30 0x7f314406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:36.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:36.629+0000 7f316be00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f314406bb30 0x7f314406dff0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f316c0a1410 tx=0x7f315c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:37.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.069+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f316c0b19c0 con 0x7f316c1547a0 2024-01-15T18:04:37.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.070+0000 7f3163e00640 1 -- 172.21.15.2:0/3040374632 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f3164039990 con 0x7f316c1547a0 2024-01-15T18:04:37.074 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:37.074 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:37.074 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:37.075 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 444, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:37.076 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1986, 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 8180274645, 2024-01-15T18:04:37.077 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 16005808128, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 757088305152, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 420268394, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 412968053, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 101, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 109 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:37.078 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:37.079 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 16s)", 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.41667500138282776, 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 37s)", 2024-01-15T18:04:37.080 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.23810000717639923, 2024-01-15T18:04:37.081 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:37.081 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:37.081 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:37.081 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f314406bb30 msgr2=0x7f314406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f314406bb30 0x7f314406dff0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f316c0a1410 tx=0x7f315c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 msgr2=0x7f316c154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 0x7f316c154c30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f3164037040 tx=0x7f3164004ab0 comp rx=0 tx=0).stop 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 shutdown_connections 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f314406bb30 0x7f314406dff0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f316c1547a0 0x7f316c154c30 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f316c1503f0 0x7f316c0b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 --2- 172.21.15.2:0/3040374632 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f314c00d990 0x7f316c14fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.073+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 >> 172.21.15.2:0/3040374632 conn(0x7f316c072660 msgr2=0x7f316c0ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.074+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 shutdown_connections 2024-01-15T18:04:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:37.074+0000 7f3172e00640 1 -- 172.21.15.2:0/3040374632 wait complete. 2024-01-15T18:04:39.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:39.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.435+0000 7f6331400640 1 Processor -- start 2024-01-15T18:04:39.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.435+0000 7f6331400640 1 -- start start 2024-01-15T18:04:39.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.437+0000 7f6331400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f632c06bfd0 0x7f632c06c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.437+0000 7f6331400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c0cbce0 con 0x7f632c06b7d0 2024-01-15T18:04:39.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.437+0000 7f6331400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c06c8f0 con 0x7f632c0cc170 2024-01-15T18:04:39.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.437+0000 7f6331400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c0d03f0 con 0x7f632c06bfd0 2024-01-15T18:04:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.438+0000 7f632b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f632c06b7d0 0x7f632c0cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48812/0 (socket says 172.21.15.2:48812) 2024-01-15T18:04:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.438+0000 7f632b400640 1 -- 172.21.15.2:0/1502564528 learned_addr learned my addr 172.21.15.2:0/1502564528 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.438+0000 7f632be00640 1 --2- 172.21.15.2:0/1502564528 >> v2:172.21.15.137:3300/0 conn(0x7f632c06bfd0 0x7f632c06c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.438+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059806076 0 0) 0x7f632c0cbce0 con 0x7f632c06b7d0 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.438+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f630c003520 con 0x7f632c06b7d0 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2318952440 0 0) 0x7f630c003520 con 0x7f632c06b7d0 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 >> v2:172.21.15.137:3300/0 conn(0x7f632c06bfd0 msgr2=0x7f632c06c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 --2- 172.21.15.2:0/1502564528 >> v2:172.21.15.137:3300/0 conn(0x7f632c06bfd0 0x7f632c06c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 >> v1:172.21.15.137:6789/0 conn(0x7f632c0cc170 legacy=0x7f632c0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0cb690 con 0x7f632c06b7d0 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6314001670 con 0x7f632c06b7d0 2024-01-15T18:04:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 >> v1:172.21.15.2:6789/0 conn(0x7f632c06b7d0 legacy=0x7f632c0cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c003970 0x7f630c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c0047c0 0x7f630c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c0052a0 0x7f630c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0d03f0 con 0x7f630c0047c0 2024-01-15T18:04:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630c00c1f0 con 0x7f630c0052a0 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.439+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630c00c3b0 con 0x7f630c003970 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.440+0000 7f632b400640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c0047c0 0x7f630c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.440+0000 7f632b400640 1 -- 172.21.15.2:0/1502564528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c003970 msgr2=0x7f630c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.440+0000 7f632b400640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c003970 0x7f630c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.440+0000 7f632b400640 1 -- 172.21.15.2:0/1502564528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c0052a0 msgr2=0x7f630c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.440+0000 7f632b400640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f630c0052a0 0x7f630c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.440+0000 7f632b400640 1 -- 172.21.15.2:0/1502564528 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0cb690 con 0x7f630c0047c0 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.441+0000 7f6330a00640 1 -- 172.21.15.2:0/1502564528 reap_dead start 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.441+0000 7f632b400640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c0047c0 0x7f630c004c40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f631400c260 tx=0x7f630c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7df6f2bdc7c1e75 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.442+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631402cb20 con 0x7f630c0047c0 2024-01-15T18:04:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.442+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631402cc80 con 0x7f630c0047c0 2024-01-15T18:04:39.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.442+0000 7f632aa00640 1 -- 172.21.15.2:0/1502564528 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140058d0 con 0x7f630c0047c0 2024-01-15T18:04:39.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 -- 172.21.15.2:0/1502564528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c0047c0 msgr2=0x7f630c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:39.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c0047c0 0x7f630c004c40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f631400c260 tx=0x7f630c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 -- 172.21.15.2:0/1502564528 shutdown_connections 2024-01-15T18:04:39.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 --2- 172.21.15.2:0/1502564528 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f630c0047c0 0x7f630c004c40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 -- 172.21.15.2:0/1502564528 >> 172.21.15.2:0/1502564528 conn(0x7f632c0d0ec0 msgr2=0x7f632c0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:39.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 -- 172.21.15.2:0/1502564528 shutdown_connections 2024-01-15T18:04:39.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.444+0000 7f6331400640 1 -- 172.21.15.2:0/1502564528 wait complete. 2024-01-15T18:04:39.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.447+0000 7f6331400640 1 Processor -- start 2024-01-15T18:04:39.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.447+0000 7f6331400640 1 -- start start 2024-01-15T18:04:39.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6331400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c0047c0 0x7f632c1f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6331400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c157140 0x7f632c0ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6331400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 0x7f632c14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6331400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c158510 con 0x7f632c14cef0 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6331400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c14d8e0 con 0x7f632c157140 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6331400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c157890 con 0x7f630c0047c0 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6330a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 0x7f632c14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6330a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 0x7f632c14d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35886/0 (socket says 172.21.15.2:35886) 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.448+0000 7f6330a00640 1 -- 172.21.15.2:0/2118727741 learned_addr learned my addr 172.21.15.2:0/2118727741 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.449+0000 7f6330a00640 1 -- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c0047c0 msgr2=0x7f632c1f2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:39.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.449+0000 7f6330a00640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c0047c0 0x7f632c1f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.449+0000 7f6330a00640 1 -- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c157140 msgr2=0x7f632c0ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:39.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.449+0000 7f6330a00640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c157140 0x7f632c0ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.449+0000 7f6330a00640 1 -- 172.21.15.2:0/2118727741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c157b70 con 0x7f632c14cef0 2024-01-15T18:04:39.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.449+0000 7f6330a00640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 0x7f632c14d3a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f632c072010 tx=0x7f632400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:39.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.450+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63240070d0 con 0x7f632c14cef0 2024-01-15T18:04:39.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.450+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6324007230 con 0x7f632c14cef0 2024-01-15T18:04:39.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.450+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c157e50 con 0x7f632c14cef0 2024-01-15T18:04:39.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.450+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63240075e0 con 0x7f632c14cef0 2024-01-15T18:04:39.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.450+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c1575a0 con 0x7f632c14cef0 2024-01-15T18:04:39.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.451+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c158650 con 0x7f632c14cef0 2024-01-15T18:04:39.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.452+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6324007740 con 0x7f632c14cef0 2024-01-15T18:04:39.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.453+0000 7f6323e00640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406bb30 0x7f630406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:39.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.453+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6324087b00 con 0x7f632c14cef0 2024-01-15T18:04:39.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.459+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f632c158650 con 0x7f632c14cef0 2024-01-15T18:04:39.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.484+0000 7f632be00640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406bb30 0x7f630406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:39.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.487+0000 7f632be00640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406bb30 0x7f630406dff0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f631c009850 tx=0x7f631c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.916+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f632c158650 con 0x7f632c14cef0 2024-01-15T18:04:39.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.917+0000 7f6323e00640 1 -- 172.21.15.2:0/2118727741 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f632c158650 con 0x7f632c14cef0 2024-01-15T18:04:39.922 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:39.922 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:39.923 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 447, 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:39.924 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:39.925 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2099, 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 8654231098, 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 17322815488, 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 755771297792, 2024-01-15T18:04:39.926 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 466623504, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 452327461, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 112, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 116 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:39.927 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:39.928 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (11s)\n [===========.................] (remaining: 16s)", 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.41667500138282776, 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (11s)\n [======......................] (remaining: 37s)", 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.23810000717639923, 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:39.931 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:39.931 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406bb30 msgr2=0x7f630406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406bb30 0x7f630406dff0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f631c009850 tx=0x7f631c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 msgr2=0x7f632c14d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 0x7f632c14d3a0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f632c072010 tx=0x7f632400c8e0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 shutdown_connections 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406bb30 0x7f630406dff0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c14cef0 0x7f632c14d3a0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c157140 0x7f632c0ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 --2- 172.21.15.2:0/2118727741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f630c0047c0 0x7f632c1f2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:39.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 >> 172.21.15.2:0/2118727741 conn(0x7f632c0d0ec0 msgr2=0x7f632c06ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:39.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 shutdown_connections 2024-01-15T18:04:39.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:39.920+0000 7f6331400640 1 -- 172.21.15.2:0/2118727741 wait complete. 2024-01-15T18:04:41.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:42.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.190+0000 7fceaac00640 1 Processor -- start 2024-01-15T18:04:42.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.191+0000 7fceaac00640 1 -- start start 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.192+0000 7fceaac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 0x7fcea415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.192+0000 7fceaac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcea40ce170 0x7fcea4070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.192+0000 7fceaac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40aeab0 con 0x7fcea415e170 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.192+0000 7fceaac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea415e5f0 con 0x7fcea40ce170 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.192+0000 7fceaac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40ce6d0 con 0x7fcea413ce30 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.193+0000 7fce9be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcea415e170 0x7fcea40cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48816/0 (socket says 172.21.15.2:48816) 2024-01-15T18:04:42.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.193+0000 7fce9be00640 1 -- 172.21.15.2:0/1116258554 learned_addr learned my addr 172.21.15.2:0/1116258554 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:42.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.193+0000 7fcea3e00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 0x7fcea415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.193+0000 7fcea8c00640 1 --2- 172.21.15.2:0/1116258554 >> v2:172.21.15.137:3300/0 conn(0x7fcea40ce170 0x7fcea4070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373292442 0 0) 0x7fcea40aeab0 con 0x7fcea415e170 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84003520 con 0x7fcea415e170 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 772845721 0 0) 0x7fce84003520 con 0x7fcea415e170 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 msgr2=0x7fcea415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 0x7fcea415dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 >> v2:172.21.15.137:3300/0 conn(0x7fcea40ce170 msgr2=0x7fcea4070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 --2- 172.21.15.2:0/1116258554 >> v2:172.21.15.137:3300/0 conn(0x7fcea40ce170 0x7fcea4070ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea4070380 con 0x7fcea415e170 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fce8c001670 con 0x7fcea415e170 2024-01-15T18:04:42.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 >> v1:172.21.15.2:6789/0 conn(0x7fcea415e170 legacy=0x7fcea40cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce84003970 0x7fce84003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fce84004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 0x7fce84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415e5f0 con 0x7fce840047c0 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea40ce6d0 con 0x7fce84003970 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.194+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8400b980 con 0x7fcea413ce30 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.195+0000 7fce9be00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fce84004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.195+0000 7fcea3e00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce84003970 0x7fce84003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.195+0000 7fcea8c00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 0x7fce84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.196+0000 7fce9be00640 1 -- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 msgr2=0x7fce84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:42.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.196+0000 7fce9be00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea413ce30 0x7fce84005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.196+0000 7fce9be00640 1 -- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce84003970 msgr2=0x7fce84003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.196+0000 7fce9be00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce84003970 0x7fce84003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.196+0000 7fce9be00640 1 -- 172.21.15.2:0/1116258554 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84003520 con 0x7fce840047c0 2024-01-15T18:04:42.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.197+0000 7fce9be00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fce84004c40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c160 tx=0x7fce840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee4265bcd95e496 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:42.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.200+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c005ad0 con 0x7fce840047c0 2024-01-15T18:04:42.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.200+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c005c30 con 0x7fce840047c0 2024-01-15T18:04:42.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.200+0000 7fcea3400640 1 -- 172.21.15.2:0/1116258554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c02ca50 con 0x7fce840047c0 2024-01-15T18:04:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.205+0000 7fceaac00640 1 -- 172.21.15.2:0/1116258554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 msgr2=0x7fce84004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.205+0000 7fceaac00640 1 --2- 172.21.15.2:0/1116258554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fce84004c40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00c160 tx=0x7fce840057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.205+0000 7fcea8c00640 1 -- 172.21.15.2:0/1116258554 reap_dead start 2024-01-15T18:04:42.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.206+0000 7fceaac00640 1 -- 172.21.15.2:0/1116258554 shutdown_connections 2024-01-15T18:04:42.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.206+0000 7fceaac00640 1 -- 172.21.15.2:0/1116258554 >> 172.21.15.2:0/1116258554 conn(0x7fcea406da20 msgr2=0x7fcea406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:42.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.206+0000 7fceaac00640 1 -- 172.21.15.2:0/1116258554 shutdown_connections 2024-01-15T18:04:42.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.206+0000 7fceaac00640 1 -- 172.21.15.2:0/1116258554 wait complete. 2024-01-15T18:04:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.210+0000 7fceaac00640 1 Processor -- start 2024-01-15T18:04:42.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.210+0000 7fceaac00640 1 -- start start 2024-01-15T18:04:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.212+0000 7fceaac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fcea413b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.212+0000 7fceaac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea41590f0 0x7fcea4159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.212+0000 7fceaac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea415ca20 0x7fcea415ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fceaac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4159ab0 con 0x7fce840047c0 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fceaac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415b2a0 con 0x7fcea415ca20 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fceaac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea415d410 con 0x7fcea41590f0 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fcea3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fcea413b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fcea3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fcea413b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:35904/0 (socket says 172.21.15.2:35904) 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fcea3e00640 1 -- 172.21.15.2:0/1652826659 learned_addr learned my addr 172.21.15.2:0/1652826659 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.213+0000 7fcea8c00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea415ca20 0x7fcea415ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.214+0000 7fcea3e00640 1 -- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea41590f0 msgr2=0x7fcea4159570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.214+0000 7fcea3e00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea41590f0 0x7fcea4159570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.214+0000 7fcea3e00640 1 -- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea415ca20 msgr2=0x7fcea415ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.214+0000 7fcea3e00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea415ca20 0x7fcea415ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.214+0000 7fcea3e00640 1 -- 172.21.15.2:0/1652826659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea415d6e0 con 0x7fce840047c0 2024-01-15T18:04:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.216+0000 7fcea3e00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fcea413b8d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fce94011910 tx=0x7fce94011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.216+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9400faf0 con 0x7fce840047c0 2024-01-15T18:04:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.217+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9400fc50 con 0x7fce840047c0 2024-01-15T18:04:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.217+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94014440 con 0x7fce840047c0 2024-01-15T18:04:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.217+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea41f67d0 con 0x7fce840047c0 2024-01-15T18:04:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.217+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea41f6d90 con 0x7fce840047c0 2024-01-15T18:04:42.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.219+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea40686f0 con 0x7fce840047c0 2024-01-15T18:04:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.220+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fce940145a0 con 0x7fce840047c0 2024-01-15T18:04:42.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.221+0000 7fcea0c00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce7c06bbc0 0x7fce7c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:42.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.221+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fce94088d30 con 0x7fce840047c0 2024-01-15T18:04:42.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.222+0000 7fce9be00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce7c06bbc0 0x7fce7c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.223+0000 7fce9be00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce7c06bbc0 0x7fce7c06e080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fce8c001050 tx=0x7fce8c007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:42.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.226+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce94055420 con 0x7fce840047c0 2024-01-15T18:04:42.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.524+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcea406af30 con 0x7fce840047c0 2024-01-15T18:04:42.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.525+0000 7fcea0c00640 1 -- 172.21.15.2:0/1652826659 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2430 (secure 0 0 0) 0x7fce94058a40 con 0x7fce840047c0 2024-01-15T18:04:42.529 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:42.529 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:42.529 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:42.529 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:42.529 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:42.530 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 450, 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:42.531 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:42.532 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2638, 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 10919271140, 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 21410054144, 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 751684059136, 2024-01-15T18:04:42.533 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 563837166, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 547113520, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 134, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 142 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:42.534 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.535 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:42.536 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 58.332% (17s)\n [================............] (remaining: 12s)", 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.58332502841949463, 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:42.537 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:42.538 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 33.333% (17s)\n [=========...................] (remaining: 35s)", 2024-01-15T18:04:42.538 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.33332857489585876, 2024-01-15T18:04:42.538 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:42.538 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:42.538 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:42.538 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:42.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce7c06bbc0 msgr2=0x7fce7c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:42.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce7c06bbc0 0x7fce7c06e080 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fce8c001050 tx=0x7fce8c007000 comp rx=0 tx=0).stop 2024-01-15T18:04:42.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 msgr2=0x7fcea413b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:42.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fcea413b8d0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fce94011910 tx=0x7fce94011dd0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 shutdown_connections 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce7c06bbc0 0x7fce7c06e080 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcea415ca20 0x7fcea415ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcea41590f0 0x7fcea4159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 --2- 172.21.15.2:0/1652826659 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce840047c0 0x7fcea413b8d0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 >> 172.21.15.2:0/1652826659 conn(0x7fcea406da20 msgr2=0x7fcea40aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.528+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 shutdown_connections 2024-01-15T18:04:42.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:42.529+0000 7fceaac00640 1 -- 172.21.15.2:0/1652826659 wait complete. 2024-01-15T18:04:44.552 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.893+0000 7f46b2c00640 1 Processor -- start 2024-01-15T18:04:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.893+0000 7f46b2c00640 1 -- start start 2024-01-15T18:04:44.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.894+0000 7f46b2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15c0e0 0x7f46ac0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.894+0000 7f46b2c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f46ac15b7d0 0x7f46ac06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.894+0000 7f46b2c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ac15cb00 con 0x7f46ac15c4c0 2024-01-15T18:04:44.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.894+0000 7f46b2c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ac0ccec0 con 0x7f46ac15b7d0 2024-01-15T18:04:44.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.894+0000 7f46b2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac0cd1a0 con 0x7f46ac15c0e0 2024-01-15T18:04:44.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.895+0000 7f46ab400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f46ac15b7d0 0x7f46ac06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.895+0000 7f46ab400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f46ac15b7d0 0x7f46ac06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56080/0 (socket says 172.21.15.2:56080) 2024-01-15T18:04:44.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.895+0000 7f46ab400640 1 -- 172.21.15.2:0/3890594684 learned_addr learned my addr 172.21.15.2:0/3890594684 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:44.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.896+0000 7f46ab400640 1 -- 172.21.15.2:0/3890594684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15c0e0 msgr2=0x7f46ac0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:44.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.896+0000 7f46ab400640 1 --2- 172.21.15.2:0/3890594684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15c0e0 0x7f46ac0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.896+0000 7f46ab400640 1 -- 172.21.15.2:0/3890594684 >> v1:172.21.15.137:6789/0 conn(0x7f46ac15c4c0 legacy=0x7f46ac15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:44.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.896+0000 7f46ab400640 1 -- 172.21.15.2:0/3890594684 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac0cd480 con 0x7f46ac15b7d0 2024-01-15T18:04:44.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.897+0000 7f46ab400640 1 --2- 172.21.15.2:0/3890594684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15b7d0 0x7f46ac06c470 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4694009870 tx=0x7f469402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c89476dc73c08f79 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:44.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.898+0000 7f46aaa00640 1 -- 172.21.15.2:0/3890594684 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694031640 con 0x7f46ac15b7d0 2024-01-15T18:04:44.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.898+0000 7f46aaa00640 1 -- 172.21.15.2:0/3890594684 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46940317a0 con 0x7f46ac15b7d0 2024-01-15T18:04:44.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.898+0000 7f46aaa00640 1 -- 172.21.15.2:0/3890594684 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4694031b30 con 0x7f46ac15b7d0 2024-01-15T18:04:44.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 -- 172.21.15.2:0/3890594684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15b7d0 msgr2=0x7f46ac06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 --2- 172.21.15.2:0/3890594684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15b7d0 0x7f46ac06c470 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4694009870 tx=0x7f469402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 -- 172.21.15.2:0/3890594684 shutdown_connections 2024-01-15T18:04:44.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 --2- 172.21.15.2:0/3890594684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15b7d0 0x7f46ac06c470 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 --2- 172.21.15.2:0/3890594684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15c0e0 0x7f46ac0cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 -- 172.21.15.2:0/3890594684 >> 172.21.15.2:0/3890594684 conn(0x7f46ac06da20 msgr2=0x7f46ac06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:44.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 -- 172.21.15.2:0/3890594684 shutdown_connections 2024-01-15T18:04:44.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.899+0000 7f46b2c00640 1 -- 172.21.15.2:0/3890594684 wait complete. 2024-01-15T18:04:44.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.902+0000 7f46b2c00640 1 Processor -- start 2024-01-15T18:04:44.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.903+0000 7f46b2c00640 1 -- start start 2024-01-15T18:04:44.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.904+0000 7f46b2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15b7d0 0x7f46ac14e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.904+0000 7f46b2c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ac15c0e0 0x7f46ac14ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.904+0000 7f46b2c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 0x7f46ac14fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.904+0000 7f46b2c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac1506f0 con 0x7f46ac15c4c0 2024-01-15T18:04:44.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.904+0000 7f46b2c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac14cf90 con 0x7f46ac15c0e0 2024-01-15T18:04:44.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.904+0000 7f46b2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ac152f30 con 0x7f46ac15b7d0 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.905+0000 7f46b0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 0x7f46ac14fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.905+0000 7f46b0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 0x7f46ac14fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56094/0 (socket says 172.21.15.2:56094) 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.905+0000 7f46b0c00640 1 -- 172.21.15.2:0/2529843677 learned_addr learned my addr 172.21.15.2:0/2529843677 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46b0c00640 1 -- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15b7d0 msgr2=0x7f46ac14e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46b0c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15b7d0 0x7f46ac14e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46b0c00640 1 -- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ac15c0e0 msgr2=0x7f46ac14ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46b0c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ac15c0e0 0x7f46ac14ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46b0c00640 1 -- 172.21.15.2:0/2529843677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46940094d0 con 0x7f46ac15c4c0 2024-01-15T18:04:44.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46b0c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 0x7f46ac14fa10 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f46a400b970 tx=0x7f46a400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:44.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.906+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a400cae0 con 0x7f46ac15c4c0 2024-01-15T18:04:44.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.907+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a400cc40 con 0x7f46ac15c4c0 2024-01-15T18:04:44.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.907+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a4014440 con 0x7f46ac15c4c0 2024-01-15T18:04:44.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.907+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ac150410 con 0x7f46ac15c4c0 2024-01-15T18:04:44.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.907+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ac152bc0 con 0x7f46ac15c4c0 2024-01-15T18:04:44.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.909+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ac152720 con 0x7f46ac15c4c0 2024-01-15T18:04:44.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.911+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f46a401d3b0 con 0x7f46ac15c4c0 2024-01-15T18:04:44.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.913+0000 7f46a3e00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f468406bc00 0x7f468406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:44.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.914+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f46a4088410 con 0x7f46ac15c4c0 2024-01-15T18:04:44.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.923+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46a4054b00 con 0x7f46ac15c4c0 2024-01-15T18:04:44.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.948+0000 7f46abe00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f468406bc00 0x7f468406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:44.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:44.977+0000 7f46abe00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f468406bc00 0x7f468406e0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f469c009710 tx=0x7f469c006c90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:45.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.397+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f46ac0686f0 con 0x7f46ac15c4c0 2024-01-15T18:04:45.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.397+0000 7f46a3e00640 1 -- 172.21.15.2:0/2529843677 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7f46a4019090 con 0x7f46ac15c4c0 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:45.402 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:45.403 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 453, 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.404 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:45.405 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2638, 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 10919271140, 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 21410054144, 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 751684059136, 2024-01-15T18:04:45.406 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 472300241, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 461851140, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 112, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 118 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:45.407 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:45.408 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.409 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (18s)\n [==============..............] (remaining: 18s)", 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.5, 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:45.410 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:45.411 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 42.857% (18s)\n [============................] (remaining: 25s)", 2024-01-15T18:04:45.411 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.4285714328289032, 2024-01-15T18:04:45.411 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:45.411 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:45.411 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:45.411 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:45.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f468406bc00 msgr2=0x7f468406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:45.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f468406bc00 0x7f468406e0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f469c009710 tx=0x7f469c006c90 comp rx=0 tx=0).stop 2024-01-15T18:04:45.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 msgr2=0x7f46ac14fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 0x7f46ac14fa10 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f46a400b970 tx=0x7f46a400be30 comp rx=0 tx=0).stop 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 shutdown_connections 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f468406bc00 0x7f468406e0c0 secure :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f469c009710 tx=0x7f469c006c90 comp rx=0 tx=0).stop 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46ac15c4c0 0x7f46ac14fa10 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46ac15c0e0 0x7f46ac14ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 --2- 172.21.15.2:0/2529843677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46ac15b7d0 0x7f46ac14e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.401+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 >> 172.21.15.2:0/2529843677 conn(0x7f46ac06da20 msgr2=0x7f46ac070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.402+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 shutdown_connections 2024-01-15T18:04:45.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:45.402+0000 7f46b2c00640 1 -- 172.21.15.2:0/2529843677 wait complete. 2024-01-15T18:04:47.436 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:47.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.746+0000 7f1a42200640 1 Processor -- start 2024-01-15T18:04:47.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.746+0000 7f1a42200640 1 -- start start 2024-01-15T18:04:47.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.750+0000 7f1a42200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c15a310 0x7f1a3c0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.750+0000 7f1a42200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c0ce390 con 0x7f1a3c071710 2024-01-15T18:04:47.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.750+0000 7f1a42200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c0ce670 con 0x7f1a3c159f30 2024-01-15T18:04:47.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.750+0000 7f1a42200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c071b80 con 0x7f1a3c15a310 2024-01-15T18:04:47.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.751+0000 7f1a40e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f1a3c159f30 0x7f1a3c0cd8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:50832/0 (socket says 172.21.15.2:50832) 2024-01-15T18:04:47.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.751+0000 7f1a40e00640 1 -- 172.21.15.2:0/358136480 learned_addr learned my addr 172.21.15.2:0/358136480 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:47.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.751+0000 7f1a3be00640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c15a310 0x7f1a3c0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.751+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925189204 0 0) 0x7f1a3c0ce390 con 0x7f1a3c071710 2024-01-15T18:04:47.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.751+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a1c003520 con 0x7f1a3c071710 2024-01-15T18:04:47.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137199690 0 0) 0x7f1a1c003520 con 0x7f1a3c071710 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c15a310 msgr2=0x7f1a3c0711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c15a310 0x7f1a3c0711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 >> v1:172.21.15.137:6789/0 conn(0x7f1a3c159f30 legacy=0x7f1a3c0cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0adf60 con 0x7f1a3c071710 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1a2c001670 con 0x7f1a3c071710 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 >> v1:172.21.15.2:6789/0 conn(0x7f1a3c071710 legacy=0x7f1a3c0ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a1c0047c0 0x7f1a1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a1c005180 0x7f1a1c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c071b80 con 0x7f1a1c0039a0 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c00e0d0 con 0x7f1a1c0047c0 2024-01-15T18:04:47.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1c005950 con 0x7f1a1c005180 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.752+0000 7f1a40e00640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a3be00640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a1c0047c0 0x7f1a1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a41800640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a1c005180 0x7f1a1c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a40e00640 1 -- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a1c005180 msgr2=0x7f1a1c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a40e00640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a1c005180 0x7f1a1c00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a40e00640 1 -- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a1c0047c0 msgr2=0x7f1a1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a40e00640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a1c0047c0 0x7f1a1c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:47.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.753+0000 7f1a40e00640 1 -- 172.21.15.2:0/358136480 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0adf60 con 0x7f1a1c0039a0 2024-01-15T18:04:47.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.755+0000 7f1a40e00640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a1c003dc0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400c190 tx=0x7f1a1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=106f5cddbcb68309 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.755+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3402ca50 con 0x7f1a1c0039a0 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.755+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3402cbb0 con 0x7f1a1c0039a0 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.755+0000 7f1a3b400640 1 -- 172.21.15.2:0/358136480 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34035470 con 0x7f1a1c0039a0 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a42200640 1 -- 172.21.15.2:0/358136480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 msgr2=0x7f1a1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a42200640 1 --2- 172.21.15.2:0/358136480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a1c003dc0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400c190 tx=0x7f1a1c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a41800640 1 -- 172.21.15.2:0/358136480 reap_dead start 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a42200640 1 -- 172.21.15.2:0/358136480 shutdown_connections 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a42200640 1 -- 172.21.15.2:0/358136480 >> 172.21.15.2:0/358136480 conn(0x7f1a3c0cfd20 msgr2=0x7f1a3c0d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a42200640 1 -- 172.21.15.2:0/358136480 shutdown_connections 2024-01-15T18:04:47.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.756+0000 7f1a42200640 1 -- 172.21.15.2:0/358136480 wait complete. 2024-01-15T18:04:47.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.758+0000 7f1a42200640 1 Processor -- start 2024-01-15T18:04:47.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.758+0000 7f1a42200640 1 -- start start 2024-01-15T18:04:47.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a42200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a3c0c5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a42200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a3c0cce70 0x7f1a3c14e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a42200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c06e350 0x7f1a3c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a42200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c06ed10 con 0x7f1a1c0039a0 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a42200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0c6df0 con 0x7f1a3c0cce70 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a42200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0c5d10 con 0x7f1a3c06e350 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a40e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a3c0c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a40e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a3c0c5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56116/0 (socket says 172.21.15.2:56116) 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.760+0000 7f1a40e00640 1 -- 172.21.15.2:0/1250197267 learned_addr learned my addr 172.21.15.2:0/1250197267 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a3be00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a3c0cce70 0x7f1a3c14e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a40e00640 1 -- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c06e350 msgr2=0x7f1a3c06e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a40e00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c06e350 0x7f1a3c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a40e00640 1 -- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a3c0cce70 msgr2=0x7f1a3c14e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a40e00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a3c0cce70 0x7f1a3c14e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a40e00640 1 -- 172.21.15.2:0/1250197267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0c5ff0 con 0x7f1a1c0039a0 2024-01-15T18:04:47.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.761+0000 7f1a3be00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a3c0cce70 0x7f1a3c14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:47.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.764+0000 7f1a40e00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a3c0c5700 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1a3402c4c0 tx=0x7f1a340014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:47.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.764+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34035880 con 0x7f1a1c0039a0 2024-01-15T18:04:47.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.764+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a340359e0 con 0x7f1a1c0039a0 2024-01-15T18:04:47.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.764+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a34035d90 con 0x7f1a1c0039a0 2024-01-15T18:04:47.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.765+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a3c0c62d0 con 0x7f1a1c0039a0 2024-01-15T18:04:47.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.765+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c14e540 con 0x7f1a1c0039a0 2024-01-15T18:04:47.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.766+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c14e700 con 0x7f1a1c0039a0 2024-01-15T18:04:47.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.768+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f1a34032070 con 0x7f1a1c0039a0 2024-01-15T18:04:47.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.769+0000 7f1a38c00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a1406bb30 0x7f1a1406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:47.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.769+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f1a340ac6c0 con 0x7f1a1c0039a0 2024-01-15T18:04:47.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.773+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a34076600 con 0x7f1a1c0039a0 2024-01-15T18:04:47.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.792+0000 7f1a3be00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a1406bb30 0x7f1a1406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:47.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:47.805+0000 7f1a3be00640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a1406bb30 0x7f1a1406dff0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0c6940 tx=0x7f1a24008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:48.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.240+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1a3c074770 con 0x7f1a1c0039a0 2024-01-15T18:04:48.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.243+0000 7f1a38c00640 1 -- 172.21.15.2:0/1250197267 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f1a34045070 con 0x7f1a1c0039a0 2024-01-15T18:04:48.248 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:48.249 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 455, 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:48.250 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:48.251 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:48.252 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 3316, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 13754822528, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 27242061824, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 745852051456, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 703438686, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 697894583, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 168, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 179 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:48.253 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:48.254 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:48.255 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "2 ongoing cloning jobs; Avg progress = 66.67% (21s)\n [==================..........] (remaining: 10s)", 2024-01-15T18:04:48.256 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.66670000553131104, 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 47.62% (21s)\n [=============...............] (remaining: 24s)", 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.47620001435279846, 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:48.257 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:48.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.246+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a1406bb30 msgr2=0x7f1a1406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:48.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.246+0000 7f1a42200640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a1406bb30 0x7f1a1406dff0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c0c6940 tx=0x7f1a24008040 comp rx=0 tx=0).stop 2024-01-15T18:04:48.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.247+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 msgr2=0x7f1a3c0c5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.247+0000 7f1a42200640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a3c0c5700 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1a3402c4c0 tx=0x7f1a340014a0 comp rx=0 tx=0).stop 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 shutdown_connections 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a1406bb30 0x7f1a1406dff0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a3c06e350 0x7f1a3c06e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a3c0cce70 0x7f1a3c14e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 --2- 172.21.15.2:0/1250197267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a1c0039a0 0x7f1a3c0c5700 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 >> 172.21.15.2:0/1250197267 conn(0x7f1a3c0cfd20 msgr2=0x7f1a3c0cb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.249+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 shutdown_connections 2024-01-15T18:04:48.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:48.250+0000 7f1a42200640 1 -- 172.21.15.2:0/1250197267 wait complete. 2024-01-15T18:04:50.275 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:04:50.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.499+0000 7f523b200640 1 Processor -- start 2024-01-15T18:04:50.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.499+0000 7f523b200640 1 -- start start 2024-01-15T18:04:50.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f523b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f523c0d1000 0x7f523c0d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f523b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c0b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f523b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c0b5e60 con 0x7f523c15d310 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f523b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c0d1e30 con 0x7f523c0d1000 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f523b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c15d6f0 con 0x7f523c0d19a0 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f5239400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f523c0d1000 0x7f523c0d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f5239400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f523c0d1000 0x7f523c0d1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56124/0 (socket says 172.21.15.2:56124) 2024-01-15T18:04:50.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.501+0000 7f5239400640 1 -- 172.21.15.2:0/2495229581 learned_addr learned my addr 172.21.15.2:0/2495229581 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.502+0000 7f523a800640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c0b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.502+0000 7f5239400640 1 -- 172.21.15.2:0/2495229581 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 msgr2=0x7f523c0b65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.502+0000 7f5239400640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c0b65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.502+0000 7f5239400640 1 -- 172.21.15.2:0/2495229581 >> v1:172.21.15.137:6789/0 conn(0x7f523c15d310 legacy=0x7f523c0d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.502+0000 7f5239400640 1 -- 172.21.15.2:0/2495229581 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c0ce760 con 0x7f523c0d1000 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.502+0000 7f523a800640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c0b65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:50.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.503+0000 7f5239400640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0d1460 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5224009870 tx=0x7f522402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5149c0663d03ae8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:50.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f5238a00640 1 -- 172.21.15.2:0/2495229581 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224031640 con 0x7f523c0d1000 2024-01-15T18:04:50.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f5238a00640 1 -- 172.21.15.2:0/2495229581 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52240317a0 con 0x7f523c0d1000 2024-01-15T18:04:50.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f5238a00640 1 -- 172.21.15.2:0/2495229581 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5224031b30 con 0x7f523c0d1000 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f523b200640 1 -- 172.21.15.2:0/2495229581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 msgr2=0x7f523c0d1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f523b200640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0d1460 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f5224009870 tx=0x7f522402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f523b200640 1 -- 172.21.15.2:0/2495229581 shutdown_connections 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f523b200640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c0b65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f523b200640 1 --2- 172.21.15.2:0/2495229581 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0d1460 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.504+0000 7f523b200640 1 -- 172.21.15.2:0/2495229581 >> 172.21.15.2:0/2495229581 conn(0x7f523c09c3b0 msgr2=0x7f523c0d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.505+0000 7f523b200640 1 -- 172.21.15.2:0/2495229581 shutdown_connections 2024-01-15T18:04:50.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.505+0000 7f523b200640 1 -- 172.21.15.2:0/2495229581 wait complete. 2024-01-15T18:04:50.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.506+0000 7f523b200640 1 Processor -- start 2024-01-15T18:04:50.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.506+0000 7f523b200640 1 -- start start 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.507+0000 7f523b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.507+0000 7f523b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c14ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.507+0000 7f523b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f523c15d310 0x7f523c06b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.507+0000 7f523b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1488f0 con 0x7f523c0d1000 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.507+0000 7f523b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c14b1a0 con 0x7f523c15d310 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.507+0000 7f523b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c0ccf90 con 0x7f523c0d19a0 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0c9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56136/0 (socket says 172.21.15.2:56136) 2024-01-15T18:04:50.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 -- 172.21.15.2:0/457082241 learned_addr learned my addr 172.21.15.2:0/457082241 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239400640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c14ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 -- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 msgr2=0x7f523c14ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c14ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 -- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f523c15d310 msgr2=0x7f523c06b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f523a800640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f523c15d310 0x7f523c06b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f523c15d310 0x7f523c06b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f5239e00640 1 -- 172.21.15.2:0/457082241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52240094d0 con 0x7f523c0d1000 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.508+0000 7f523a800640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f523c15d310 0x7f523c06b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.509+0000 7f5239e00640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0c9520 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f523400c730 tx=0x7f523400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.509+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234003b70 con 0x7f523c0d1000 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.509+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5234003cf0 con 0x7f523c0d1000 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.509+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5234003ea0 con 0x7f523c0d1000 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.509+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c0cd270 con 0x7f523c0d1000 2024-01-15T18:04:50.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.509+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c148580 con 0x7f523c0d1000 2024-01-15T18:04:50.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.510+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c074770 con 0x7f523c0d1000 2024-01-15T18:04:50.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.511+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f523400aa90 con 0x7f523c0d1000 2024-01-15T18:04:50.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.512+0000 7f5232000640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f521406bb30 0x7f521406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:50.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.512+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f5234088440 con 0x7f523c0d1000 2024-01-15T18:04:50.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.513+0000 7f5239400640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f521406bb30 0x7f521406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:50.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.514+0000 7f5239400640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f521406bb30 0x7f521406dff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5224036000 tx=0x7f522403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:50.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.517+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5234054b30 con 0x7f523c0d1000 2024-01-15T18:04:50.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.785+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f523c0b5700 con 0x7f523c0d1000 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.786+0000 7f5232000640 1 -- 172.21.15.2:0/457082241 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2404 (secure 0 0 0) 0x7f5234058150 con 0x7f523c0d1000 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:04:50.789 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 458, 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:04:50.790 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 61, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:50.791 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 3417, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 14174253824, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 28542357504, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 744551755776, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 500988111, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 498921945, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 120, 2024-01-15T18:04:50.792 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 128 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 37, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 4, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 4, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "c", 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6312 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:50.793 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 12, 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:04:09.541206+0000", 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.794 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "a874df04-6948-4dbf-8c1f-785e30dda030": { 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Subvolume \"subvol_clone27427\" has been 100.0% cloned (25s)\n [============================] ", 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 1, 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "b10040f0-fae7-4cd3-89c4-ee341111390b": { 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (25s)\n [================............] (remaining: 19s)", 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.57142859697341919, 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:50.795 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:50.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.788+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f521406bb30 msgr2=0x7f521406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:50.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.788+0000 7f523b200640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f521406bb30 0x7f521406dff0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f5224036000 tx=0x7f522403c040 comp rx=0 tx=0).stop 2024-01-15T18:04:50.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.788+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 msgr2=0x7f523c0c9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:50.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.788+0000 7f523b200640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0c9520 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f523400c730 tx=0x7f523400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 shutdown_connections 2024-01-15T18:04:50.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f521406bb30 0x7f521406dff0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f523c15d310 0x7f523c06b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f523c0d19a0 0x7f523c14ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 --2- 172.21.15.2:0/457082241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f523c0d1000 0x7f523c0c9520 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 >> 172.21.15.2:0/457082241 conn(0x7f523c09c3b0 msgr2=0x7f523c161610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 shutdown_connections 2024-01-15T18:04:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:50.789+0000 7f523b200640 1 -- 172.21.15.2:0/457082241 wait complete. 2024-01-15T18:04:52.813 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:04:52.813 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27421 2024-01-15T18:04:53.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.042+0000 7fa393200640 1 Processor -- start 2024-01-15T18:04:53.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.043+0000 7fa393200640 1 -- start start 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa393200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa39415d270 0x7fa3940b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa393200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa394160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa393200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3940b2b60 0x7fa3941611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa393200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39406c5e0 con 0x7fa3940b2b60 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa393200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39406c8c0 con 0x7fa39415d270 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa393200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa394071c00 con 0x7fa394161890 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa392800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3940b2b60 0x7fa3941611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa392800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3940b2b60 0x7fa3941611a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50276/0 (socket says 172.21.15.2:50276) 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.044+0000 7fa392800640 1 -- 172.21.15.2:0/337563053 learned_addr learned my addr 172.21.15.2:0/337563053 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa38be00640 1 --2- 172.21.15.2:0/337563053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa394160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa392800640 1 -- 172.21.15.2:0/337563053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 msgr2=0x7fa394160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa392800640 1 --2- 172.21.15.2:0/337563053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa394160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa391e00640 1 --2- 172.21.15.2:0/337563053 >> v2:172.21.15.137:3300/0 conn(0x7fa39415d270 0x7fa3940b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa392800640 1 -- 172.21.15.2:0/337563053 >> v2:172.21.15.137:3300/0 conn(0x7fa39415d270 msgr2=0x7fa3940b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa392800640 1 --2- 172.21.15.2:0/337563053 >> v2:172.21.15.137:3300/0 conn(0x7fa39415d270 0x7fa3940b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa392800640 1 -- 172.21.15.2:0/337563053 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa394071ee0 con 0x7fa3940b2b60 2024-01-15T18:04:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.045+0000 7fa391e00640 1 --2- 172.21.15.2:0/337563053 >> v2:172.21.15.137:3300/0 conn(0x7fa39415d270 0x7fa3940b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.046+0000 7fa392800640 1 --2- 172.21.15.2:0/337563053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa3941611a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa39406bbf0 tx=0x7fa38c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9b05a9eeb964f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa391400640 1 -- 172.21.15.2:0/337563053 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c02c950 con 0x7fa3940b2b60 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa391400640 1 -- 172.21.15.2:0/337563053 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c02cab0 con 0x7fa3940b2b60 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa391400640 1 -- 172.21.15.2:0/337563053 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c02ce40 con 0x7fa3940b2b60 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 -- 172.21.15.2:0/337563053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 msgr2=0x7fa3941611a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 --2- 172.21.15.2:0/337563053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa3941611a0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fa39406bbf0 tx=0x7fa38c02bdb0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 -- 172.21.15.2:0/337563053 shutdown_connections 2024-01-15T18:04:53.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 --2- 172.21.15.2:0/337563053 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa3941611a0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 --2- 172.21.15.2:0/337563053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa394160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 --2- 172.21.15.2:0/337563053 >> v2:172.21.15.137:3300/0 conn(0x7fa39415d270 0x7fa3940b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 -- 172.21.15.2:0/337563053 >> 172.21.15.2:0/337563053 conn(0x7fa3940b0840 msgr2=0x7fa3940b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:53.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 -- 172.21.15.2:0/337563053 shutdown_connections 2024-01-15T18:04:53.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.047+0000 7fa393200640 1 -- 172.21.15.2:0/337563053 wait complete. 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.049+0000 7fa393200640 1 Processor -- start 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.049+0000 7fa393200640 1 -- start start 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.050+0000 7fa393200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa394160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.050+0000 7fa393200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa39415d270 0x7fa394161130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.050+0000 7fa393200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa3941472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.050+0000 7fa393200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa394157dc0 con 0x7fa3940b2b60 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.050+0000 7fa393200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3940c3f70 con 0x7fa39415d270 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.050+0000 7fa393200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3941571b0 con 0x7fa394161890 2024-01-15T18:04:53.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa394160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa394160bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50278/0 (socket says 172.21.15.2:50278) 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 -- 172.21.15.2:0/1224828875 learned_addr learned my addr 172.21.15.2:0/1224828875 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa392800640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa3941472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 -- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 msgr2=0x7fa3941472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa3941472a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 -- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa39415d270 msgr2=0x7fa394161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa38be00640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa39415d270 0x7fa394161130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa39415d270 0x7fa394161130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.051+0000 7fa391e00640 1 -- 172.21.15.2:0/1224828875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c009b30 con 0x7fa3940b2b60 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.052+0000 7fa391e00640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa394160bf0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fa38400c7c0 tx=0x7fa38400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:53.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.052+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa384007a30 con 0x7fa3940b2b60 2024-01-15T18:04:53.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.052+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa384007b90 con 0x7fa3940b2b60 2024-01-15T18:04:53.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.052+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa394157490 con 0x7fa3940b2b60 2024-01-15T18:04:53.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.052+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38400ce70 con 0x7fa3940b2b60 2024-01-15T18:04:53.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.052+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa394157a50 con 0x7fa3940b2b60 2024-01-15T18:04:53.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.053+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa394074770 con 0x7fa3940b2b60 2024-01-15T18:04:53.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.054+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa3840113f0 con 0x7fa3940b2b60 2024-01-15T18:04:53.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.055+0000 7fa38a000640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa36c06bb30 0x7fa36c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:53.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.055+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fa384088790 con 0x7fa3940b2b60 2024-01-15T18:04:53.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.056+0000 7fa38be00640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa36c06bb30 0x7fa36c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:53.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.057+0000 7fa38be00640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa36c06bb30 0x7fa36c06dff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa39414b6b0 tx=0x7fa37c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:53.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.060+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa384054e80 con 0x7fa3940b2b60 2024-01-15T18:04:53.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:53.294+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27421", "target": ["mon-mgr", ""]}) v1 -- 0x7fa394157f00 con 0x7fa36c06bb30 2024-01-15T18:04:54.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.950+0000 7fa38a000640 1 -- 172.21.15.2:0/1224828875 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fa37c00c040 con 0x7fa36c06bb30 2024-01-15T18:04:54.954 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:54.954 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:54.955 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:04:54.955 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:54.955 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.953+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa36c06bb30 msgr2=0x7fa36c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.953+0000 7fa393200640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa36c06bb30 0x7fa36c06dff0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa39414b6b0 tx=0x7fa37c008040 comp rx=0 tx=0).stop 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 msgr2=0x7fa394160bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa394160bf0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fa38400c7c0 tx=0x7fa38400cc80 comp rx=0 tx=0).stop 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 shutdown_connections 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa36c06bb30 0x7fa36c06dff0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa394161890 0x7fa3941472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa39415d270 0x7fa394161130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 --2- 172.21.15.2:0/1224828875 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3940b2b60 0x7fa394160bf0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:54.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 >> 172.21.15.2:0/1224828875 conn(0x7fa3940b0840 msgr2=0x7fa39415e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:54.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.954+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 shutdown_connections 2024-01-15T18:04:54.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:54.955+0000 7fa393200640 1 -- 172.21.15.2:0/1224828875 wait complete. 2024-01-15T18:04:54.985 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:04:54.985 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27422 2024-01-15T18:04:55.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.260+0000 7f00a1400640 1 Processor -- start 2024-01-15T18:04:55.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.260+0000 7f00a1400640 1 -- start start 2024-01-15T18:04:55.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f00a1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f009c0d1270 0x7f009c0d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f00a1400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0d1900 con 0x7f009c0b4c70 2024-01-15T18:04:55.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f00a1400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f009c0d1be0 con 0x7f009c15d730 2024-01-15T18:04:55.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f00a1400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f009c0d1700 con 0x7f009c0d1270 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f009a800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f009c0b4c70 0x7f009c0b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44118/0 (socket says 172.21.15.2:44118) 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f009a800640 1 -- 172.21.15.2:0/4187588177 learned_addr learned my addr 172.21.15.2:0/4187588177 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.262+0000 7f009b200640 1 --2- 172.21.15.2:0/4187588177 >> v2:172.21.15.2:3300/0 conn(0x7f009c0d1270 0x7f009c0d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.263+0000 7f0099e00640 1 -- 172.21.15.2:0/4187588177 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159393649 0 0) 0x7f009c0d1900 con 0x7f009c0b4c70 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.263+0000 7f0099e00640 1 -- 172.21.15.2:0/4187588177 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f007c003520 con 0x7f009c0b4c70 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.263+0000 7f009b200640 1 -- 172.21.15.2:0/4187588177 >> v1:172.21.15.137:6789/0 conn(0x7f009c15d730 legacy=0x7f009c0b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.263+0000 7f009b200640 1 -- 172.21.15.2:0/4187588177 >> v1:172.21.15.2:6789/0 conn(0x7f009c0b4c70 legacy=0x7f009c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.263+0000 7f009b200640 1 -- 172.21.15.2:0/4187588177 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f009c0d1ec0 con 0x7f009c0d1270 2024-01-15T18:04:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.265+0000 7f009b200640 1 --2- 172.21.15.2:0/4187588177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c0d29a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0094009d90 tx=0x7f009402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884eed8b4a61228a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.266+0000 7f0099e00640 1 -- 172.21.15.2:0/4187588177 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009402cc60 con 0x7f009c0d1270 2024-01-15T18:04:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.266+0000 7f0099e00640 1 -- 172.21.15.2:0/4187588177 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009402cdc0 con 0x7f009c0d1270 2024-01-15T18:04:55.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.266+0000 7f0099e00640 1 -- 172.21.15.2:0/4187588177 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094034660 con 0x7f009c0d1270 2024-01-15T18:04:55.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 -- 172.21.15.2:0/4187588177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 msgr2=0x7f009c0d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 --2- 172.21.15.2:0/4187588177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c0d29a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0094009d90 tx=0x7f009402beb0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 -- 172.21.15.2:0/4187588177 shutdown_connections 2024-01-15T18:04:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 --2- 172.21.15.2:0/4187588177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c0d29a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 -- 172.21.15.2:0/4187588177 >> 172.21.15.2:0/4187588177 conn(0x7f009c072210 msgr2=0x7f009c0cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 -- 172.21.15.2:0/4187588177 shutdown_connections 2024-01-15T18:04:55.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.268+0000 7f00a1400640 1 -- 172.21.15.2:0/4187588177 wait complete. 2024-01-15T18:04:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.270+0000 7f00a1400640 1 Processor -- start 2024-01-15T18:04:55.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.271+0000 7f00a1400640 1 -- start start 2024-01-15T18:04:55.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f00a1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c0b4c70 0x7f009c14e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f00a1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c15b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f00a1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c15d730 0x7f009c15baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f00a1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c14eb80 con 0x7f009c0d1270 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f00a1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009c1f6410 con 0x7f009c15d730 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f00a1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009c1f66b0 con 0x7f009c0b4c70 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f0091c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c15b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f0091c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c15b5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50308/0 (socket says 172.21.15.2:50308) 2024-01-15T18:04:55.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.272+0000 7f0091c00640 1 -- 172.21.15.2:0/3034307587 learned_addr learned my addr 172.21.15.2:0/3034307587 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f009a800640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c0b4c70 0x7f009c14e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f009b200640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c15d730 0x7f009c15baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f0091c00640 1 -- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c0b4c70 msgr2=0x7f009c14e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f0091c00640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c0b4c70 0x7f009c14e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f0091c00640 1 -- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c15d730 msgr2=0x7f009c15baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f0091c00640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c15d730 0x7f009c15baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.273+0000 7f0091c00640 1 -- 172.21.15.2:0/3034307587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094009a70 con 0x7f009c0d1270 2024-01-15T18:04:55.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.274+0000 7f0091c00640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c15b5b0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f008400b3c0 tx=0x7f008400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.275+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084011830 con 0x7f009c0d1270 2024-01-15T18:04:55.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.275+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084011990 con 0x7f009c0d1270 2024-01-15T18:04:55.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.275+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084011d40 con 0x7f009c0d1270 2024-01-15T18:04:55.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.275+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f009c0d3060 con 0x7f009c0d1270 2024-01-15T18:04:55.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.276+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009c1f6c70 con 0x7f009c0d1270 2024-01-15T18:04:55.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.277+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c005250 con 0x7f009c0d1270 2024-01-15T18:04:55.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.279+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f008401b460 con 0x7f009c0d1270 2024-01-15T18:04:55.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.280+0000 7f0093400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f007406bb30 0x7f007406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:55.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.281+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f00840883f0 con 0x7f009c0d1270 2024-01-15T18:04:55.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.286+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0084054ae0 con 0x7f009c0d1270 2024-01-15T18:04:55.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.295+0000 7f009a800640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f007406bb30 0x7f007406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:55.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.297+0000 7f009a800640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f007406bb30 0x7f007406dff0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f008c0011e0 tx=0x7f008c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.646+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27422", "target": ["mon-mgr", ""]}) v1 -- 0x7f006c000bc0 con 0x7f007406bb30 2024-01-15T18:04:55.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.971+0000 7f0093400640 1 -- 172.21.15.2:0/3034307587 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f006c000bc0 con 0x7f007406bb30 2024-01-15T18:04:55.974 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:55.974 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:55.974 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:04:55.974 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:55.975 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f007406bb30 msgr2=0x7f007406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f007406bb30 0x7f007406dff0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f008c0011e0 tx=0x7f008c009040 comp rx=0 tx=0).stop 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 msgr2=0x7f009c15b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c15b5b0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f008400b3c0 tx=0x7f008400b880 comp rx=0 tx=0).stop 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 shutdown_connections 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f007406bb30 0x7f007406dff0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f009c15d730 0x7f009c15baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f009c0d1270 0x7f009c15b5b0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 --2- 172.21.15.2:0/3034307587 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f009c0b4c70 0x7f009c14e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:55.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.975+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 >> 172.21.15.2:0/3034307587 conn(0x7f009c072210 msgr2=0x7f009c0b6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:55.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.977+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 shutdown_connections 2024-01-15T18:04:55.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:55.977+0000 7f00a1400640 1 -- 172.21.15.2:0/3034307587 wait complete. 2024-01-15T18:04:56.013 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:04:56.013 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27423 2024-01-15T18:04:56.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.309+0000 7fe6bf400640 1 Processor -- start 2024-01-15T18:04:56.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.309+0000 7fe6bf400640 1 -- start start 2024-01-15T18:04:56.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.311+0000 7fe6bf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe6b806d410 0x7fe6b80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.311+0000 7fe6bf400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe6b80cd2c0 0x7fe6b8166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.311+0000 7fe6bf400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b80cd9f0 con 0x7fe6b806c920 2024-01-15T18:04:56.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.311+0000 7fe6bf400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b806eb20 con 0x7fe6b806d410 2024-01-15T18:04:56.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.311+0000 7fe6bf400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b806aec0 con 0x7fe6b80cd2c0 2024-01-15T18:04:56.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.312+0000 7fe6b7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe6b806d410 0x7fe6b80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.312+0000 7fe6b7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe6b806d410 0x7fe6b80cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50312/0 (socket says 172.21.15.2:50312) 2024-01-15T18:04:56.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.312+0000 7fe6b7e00640 1 -- 172.21.15.2:0/3938569462 learned_addr learned my addr 172.21.15.2:0/3938569462 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:56.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.312+0000 7fe6b7e00640 1 -- 172.21.15.2:0/3938569462 >> v2:172.21.15.137:3300/0 conn(0x7fe6b80cd2c0 msgr2=0x7fe6b8166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:56.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.313+0000 7fe6b7e00640 1 --2- 172.21.15.2:0/3938569462 >> v2:172.21.15.137:3300/0 conn(0x7fe6b80cd2c0 0x7fe6b8166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.313+0000 7fe6b7e00640 1 -- 172.21.15.2:0/3938569462 >> v1:172.21.15.137:6789/0 conn(0x7fe6b806c920 legacy=0x7fe6b806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:56.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.313+0000 7fe6b7e00640 1 -- 172.21.15.2:0/3938569462 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b806b1a0 con 0x7fe6b806d410 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.314+0000 7fe6b7e00640 1 --2- 172.21.15.2:0/3938569462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b80cf050 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80ce320 tx=0x7fe6a002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba65d139fba8efc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.314+0000 7fe6b7400640 1 -- 172.21.15.2:0/3938569462 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00317e0 con 0x7fe6b806d410 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.314+0000 7fe6b7400640 1 -- 172.21.15.2:0/3938569462 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0031940 con 0x7fe6b806d410 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.314+0000 7fe6b7400640 1 -- 172.21.15.2:0/3938569462 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a0031cd0 con 0x7fe6b806d410 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 -- 172.21.15.2:0/3938569462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 msgr2=0x7fe6b80cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 --2- 172.21.15.2:0/3938569462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b80cf050 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80ce320 tx=0x7fe6a002b880 comp rx=0 tx=0).stop 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 -- 172.21.15.2:0/3938569462 shutdown_connections 2024-01-15T18:04:56.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 --2- 172.21.15.2:0/3938569462 >> v2:172.21.15.137:3300/0 conn(0x7fe6b80cd2c0 0x7fe6b8166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 --2- 172.21.15.2:0/3938569462 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b80cf050 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 -- 172.21.15.2:0/3938569462 >> 172.21.15.2:0/3938569462 conn(0x7fe6b80cc170 msgr2=0x7fe6b80cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:56.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 -- 172.21.15.2:0/3938569462 shutdown_connections 2024-01-15T18:04:56.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.315+0000 7fe6bf400640 1 -- 172.21.15.2:0/3938569462 wait complete. 2024-01-15T18:04:56.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.318+0000 7fe6bf400640 1 Processor -- start 2024-01-15T18:04:56.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.318+0000 7fe6bf400640 1 -- start start 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6bf400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6b806c920 0x7fe6b80b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6bf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b81523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6bf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6b80cd2c0 0x7fe6b8154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6bf400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8152910 con 0x7fe6b806d410 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6bf400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8154fa0 con 0x7fe6b806c920 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6bf400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b8155280 con 0x7fe6b80cd2c0 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b81523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b81523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50318/0 (socket says 172.21.15.2:50318) 2024-01-15T18:04:56.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.320+0000 7fe6b7e00640 1 -- 172.21.15.2:0/118568405 learned_addr learned my addr 172.21.15.2:0/118568405 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:56.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6bca00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6b806c920 0x7fe6b80b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6b7e00640 1 -- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6b80cd2c0 msgr2=0x7fe6b8154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6b7e00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6b80cd2c0 0x7fe6b8154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6b7e00640 1 -- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6b806c920 msgr2=0x7fe6b80b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6b7e00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6b806c920 0x7fe6b80b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6b7e00640 1 -- 172.21.15.2:0/118568405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a00094d0 con 0x7fe6b806d410 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6bca00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6b806c920 0x7fe6b80b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.321+0000 7fe6b7e00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b81523d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe6a002b8c0 tx=0x7fe6a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.322+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00329c0 con 0x7fe6b806d410 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.322+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a0032b40 con 0x7fe6b806d410 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.322+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a00453f0 con 0x7fe6b806d410 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.322+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b8155560 con 0x7fe6b806d410 2024-01-15T18:04:56.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.323+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b8155b20 con 0x7fe6b806d410 2024-01-15T18:04:56.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.324+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fe6a0045550 con 0x7fe6b806d410 2024-01-15T18:04:56.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.325+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b8155d10 con 0x7fe6b806d410 2024-01-15T18:04:56.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.326+0000 7fe6b4c00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe69006bb30 0x7fe69006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:56.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.326+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe6a00ad330 con 0x7fe6b806d410 2024-01-15T18:04:56.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.327+0000 7fe6bca00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe69006bb30 0x7fe69006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:56.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.328+0000 7fe6bca00640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe69006bb30 0x7fe69006dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800a810 tx=0x7fe6a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:56.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.331+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6a00b2060 con 0x7fe6b806d410 2024-01-15T18:04:56.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.693+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27423", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6b8156b10 con 0x7fe69006bb30 2024-01-15T18:04:56.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.854+0000 7fe6b4c00640 1 -- 172.21.15.2:0/118568405 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fe6b8156b10 con 0x7fe69006bb30 2024-01-15T18:04:56.858 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:56.858 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:56.858 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:04:56.858 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:56.858 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.859+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe69006bb30 msgr2=0x7fe69006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.859+0000 7fe6bf400640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe69006bb30 0x7fe69006dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe6a800a810 tx=0x7fe6a8008040 comp rx=0 tx=0).stop 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.859+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 msgr2=0x7fe6b81523d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.859+0000 7fe6bf400640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b81523d0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fe6a002b8c0 tx=0x7fe6a00046c0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 shutdown_connections 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe69006bb30 0x7fe69006dff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6b80cd2c0 0x7fe6b8154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6b806d410 0x7fe6b81523d0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 --2- 172.21.15.2:0/118568405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6b806c920 0x7fe6b80b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:56.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 >> 172.21.15.2:0/118568405 conn(0x7fe6b80cc170 msgr2=0x7fe6b815b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:56.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 shutdown_connections 2024-01-15T18:04:56.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:56.860+0000 7fe6bf400640 1 -- 172.21.15.2:0/118568405 wait complete. 2024-01-15T18:04:56.888 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:04:56.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27424 2024-01-15T18:04:57.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.074+0000 7f7241600640 1 Processor -- start 2024-01-15T18:04:57.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.074+0000 7f7241600640 1 -- start start 2024-01-15T18:04:57.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.076+0000 7f7241600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f723c15ae80 0x7f723c0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.076+0000 7f7241600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c06c8f0 0x7f723c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.076+0000 7f7241600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f723c06d290 con 0x7f723c0b2430 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.076+0000 7f7241600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f723c15e840 con 0x7f723c15ae80 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.076+0000 7f7241600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c15dbc0 con 0x7f723c06c8f0 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.076+0000 7f723aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f723c15ae80 0x7f723c0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f723aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f723c15ae80 0x7f723c0acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50326/0 (socket says 172.21.15.2:50326) 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f723aa00640 1 -- 172.21.15.2:0/454021863 learned_addr learned my addr 172.21.15.2:0/454021863 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f7231e00640 1 --2- 172.21.15.2:0/454021863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c06c8f0 0x7f723c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f723aa00640 1 -- 172.21.15.2:0/454021863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c06c8f0 msgr2=0x7f723c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f723aa00640 1 --2- 172.21.15.2:0/454021863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c06c8f0 0x7f723c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f723aa00640 1 -- 172.21.15.2:0/454021863 >> v1:172.21.15.137:6789/0 conn(0x7f723c0b2430 legacy=0x7f723c0b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:57.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.077+0000 7f723aa00640 1 -- 172.21.15.2:0/454021863 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c15dea0 con 0x7f723c15ae80 2024-01-15T18:04:57.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.078+0000 7f723aa00640 1 --2- 172.21.15.2:0/454021863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c15ae80 0x7f723c0acd00 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f722c009870 tx=0x7f722c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e7d746f82eee3d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f723a000640 1 -- 172.21.15.2:0/454021863 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c030840 con 0x7f723c15ae80 2024-01-15T18:04:57.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f723a000640 1 -- 172.21.15.2:0/454021863 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c0309a0 con 0x7f723c15ae80 2024-01-15T18:04:57.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f723a000640 1 -- 172.21.15.2:0/454021863 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c030d30 con 0x7f723c15ae80 2024-01-15T18:04:57.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f7241600640 1 -- 172.21.15.2:0/454021863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c15ae80 msgr2=0x7f723c0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f7241600640 1 --2- 172.21.15.2:0/454021863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c15ae80 0x7f723c0acd00 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f722c009870 tx=0x7f722c02b980 comp rx=0 tx=0).stop 2024-01-15T18:04:57.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f7241600640 1 -- 172.21.15.2:0/454021863 shutdown_connections 2024-01-15T18:04:57.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f7241600640 1 --2- 172.21.15.2:0/454021863 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c06c8f0 0x7f723c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f7241600640 1 --2- 172.21.15.2:0/454021863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c15ae80 0x7f723c0acd00 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.079+0000 7f7241600640 1 -- 172.21.15.2:0/454021863 >> 172.21.15.2:0/454021863 conn(0x7f723c0b45f0 msgr2=0x7f723c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:57.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.080+0000 7f7241600640 1 -- 172.21.15.2:0/454021863 shutdown_connections 2024-01-15T18:04:57.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.080+0000 7f7241600640 1 -- 172.21.15.2:0/454021863 wait complete. 2024-01-15T18:04:57.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.081+0000 7f7241600640 1 Processor -- start 2024-01-15T18:04:57.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.081+0000 7f7241600640 1 -- start start 2024-01-15T18:04:57.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.082+0000 7f7241600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f723c06c8f0 0x7f723c1445c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.082+0000 7f7241600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 0x7f723c15cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.082+0000 7f7241600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c15ae80 0x7f723c1405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.082+0000 7f7241600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c151f60 con 0x7f723c0b2430 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.082+0000 7f7241600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c14f1e0 con 0x7f723c06c8f0 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.082+0000 7f7241600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f723c14f4c0 con 0x7f723c15ae80 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 0x7f723c15cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 0x7f723c15cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50330/0 (socket says 172.21.15.2:50330) 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 -- 172.21.15.2:0/3311575063 learned_addr learned my addr 172.21.15.2:0/3311575063 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f723b400640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c15ae80 0x7f723c1405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f723aa00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f723c06c8f0 0x7f723c1445c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 -- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c15ae80 msgr2=0x7f723c1405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c15ae80 0x7f723c1405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 -- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f723c06c8f0 msgr2=0x7f723c1445c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f723c06c8f0 0x7f723c1445c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f7231e00640 1 -- 172.21.15.2:0/3311575063 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722c0094d0 con 0x7f723c0b2430 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.083+0000 7f723aa00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f723c06c8f0 0x7f723c1445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.084+0000 7f7231e00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 0x7f723c15cef0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f722400d800 tx=0x7f722400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.084+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722400b6c0 con 0x7f723c0b2430 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.084+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722400b820 con 0x7f723c0b2430 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.084+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722400bbb0 con 0x7f723c0b2430 2024-01-15T18:04:57.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.084+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723c151910 con 0x7f723c0b2430 2024-01-15T18:04:57.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.084+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723c14f7a0 con 0x7f723c0b2430 2024-01-15T18:04:57.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.085+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723c0686f0 con 0x7f723c0b2430 2024-01-15T18:04:57.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.086+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f722400bd10 con 0x7f723c0b2430 2024-01-15T18:04:57.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.087+0000 7f7233400640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f721406bb30 0x7f721406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.087+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f7224088ba0 con 0x7f723c0b2430 2024-01-15T18:04:57.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.088+0000 7f723aa00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f721406bb30 0x7f721406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.089+0000 7f723aa00640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f721406bb30 0x7f721406dff0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f722c036000 tx=0x7f722c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.092+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7224055290 con 0x7f723c0b2430 2024-01-15T18:04:57.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.365+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27424", "target": ["mon-mgr", ""]}) v1 -- 0x7f723c15e460 con 0x7f721406bb30 2024-01-15T18:04:57.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.488+0000 7f7233400640 1 -- 172.21.15.2:0/3311575063 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f722c030b90 con 0x7f721406bb30 2024-01-15T18:04:57.492 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:57.492 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:57.492 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:04:57.492 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:57.493 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f721406bb30 msgr2=0x7f721406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f721406bb30 0x7f721406dff0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f722c036000 tx=0x7f722c03b040 comp rx=0 tx=0).stop 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 msgr2=0x7f723c15cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 0x7f723c15cef0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f722400d800 tx=0x7f722400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 shutdown_connections 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f721406bb30 0x7f721406dff0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f723c15ae80 0x7f723c1405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f723c0b2430 0x7f723c15cef0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 --2- 172.21.15.2:0/3311575063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f723c06c8f0 0x7f723c1445c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 >> 172.21.15.2:0/3311575063 conn(0x7f723c0b45f0 msgr2=0x7f723c0accc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 shutdown_connections 2024-01-15T18:04:57.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.492+0000 7f7241600640 1 -- 172.21.15.2:0/3311575063 wait complete. 2024-01-15T18:04:57.532 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:04:57.532 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27425 2024-01-15T18:04:57.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.829+0000 7fb102c00640 1 Processor -- start 2024-01-15T18:04:57.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.830+0000 7fb102c00640 1 -- start start 2024-01-15T18:04:57.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.833+0000 7fb102c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0cf520 0x7fb0fc0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.833+0000 7fb102c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb0fc0b2a00 0x7fb0fc0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.833+0000 7fb102c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0fc0b2e90 con 0x7fb0fc14b280 2024-01-15T18:04:57.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.833+0000 7fb102c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0fc06ec40 con 0x7fb0fc0b2a00 2024-01-15T18:04:57.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.833+0000 7fb102c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc0cbca0 con 0x7fb0fc0cf520 2024-01-15T18:04:57.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.834+0000 7fb0fbe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb0fc14b280 0x7fb0fc0cee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:38168/0 (socket says 172.21.15.2:38168) 2024-01-15T18:04:57.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.834+0000 7fb0fbe00640 1 -- 172.21.15.2:0/3374796660 learned_addr learned my addr 172.21.15.2:0/3374796660 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:04:57.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.834+0000 7fb0fb400640 1 --2- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0cf520 0x7fb0fc0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.835+0000 7fb100c00640 1 --2- 172.21.15.2:0/3374796660 >> v2:172.21.15.137:3300/0 conn(0x7fb0fc0b2a00 0x7fb0fc0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.836+0000 7fb0faa00640 1 -- 172.21.15.2:0/3374796660 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77720005 0 0) 0x7fb0e4001110 con 0x7fb0fc14b280 2024-01-15T18:04:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.837+0000 7fb100c00640 1 -- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0cf520 msgr2=0x7fb0fc0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.837+0000 7fb100c00640 1 --2- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0cf520 0x7fb0fc0b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.837+0000 7fb100c00640 1 -- 172.21.15.2:0/3374796660 >> v1:172.21.15.137:6789/0 conn(0x7fb0fc14b280 legacy=0x7fb0fc0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.837+0000 7fb100c00640 1 -- 172.21.15.2:0/3374796660 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0fc0cb970 con 0x7fb0fc0b2a00 2024-01-15T18:04:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.839+0000 7fb100c00640 1 --2- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0b2a00 0x7fb0fc0cc130 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009dc0 tx=0x7fb0f402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=830a3ac49d09dc42 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.839+0000 7fb0faa00640 1 -- 172.21.15.2:0/3374796660 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f402cc80 con 0x7fb0fc0b2a00 2024-01-15T18:04:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.839+0000 7fb0faa00640 1 -- 172.21.15.2:0/3374796660 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f402cde0 con 0x7fb0fc0b2a00 2024-01-15T18:04:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.839+0000 7fb0faa00640 1 -- 172.21.15.2:0/3374796660 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f4034660 con 0x7fb0fc0b2a00 2024-01-15T18:04:57.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.841+0000 7fb102c00640 1 -- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0b2a00 msgr2=0x7fb0fc0cc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.841+0000 7fb102c00640 1 --2- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0b2a00 0x7fb0fc0cc130 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009dc0 tx=0x7fb0f402bed0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.842+0000 7fb102c00640 1 -- 172.21.15.2:0/3374796660 shutdown_connections 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.842+0000 7fb102c00640 1 --2- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0b2a00 0x7fb0fc0cc130 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.842+0000 7fb102c00640 1 --2- 172.21.15.2:0/3374796660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0cf520 0x7fb0fc0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.842+0000 7fb102c00640 1 -- 172.21.15.2:0/3374796660 >> 172.21.15.2:0/3374796660 conn(0x7fb0fc06c4d0 msgr2=0x7fb0fc06c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.842+0000 7fb102c00640 1 -- 172.21.15.2:0/3374796660 shutdown_connections 2024-01-15T18:04:57.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.842+0000 7fb102c00640 1 -- 172.21.15.2:0/3374796660 wait complete. 2024-01-15T18:04:57.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.846+0000 7fb102c00640 1 Processor -- start 2024-01-15T18:04:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.846+0000 7fb102c00640 1 -- start start 2024-01-15T18:04:57.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb102c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0b2a00 0x7fb0fc1f8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb102c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0cf520 0x7fb0fc161180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb102c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 0x7fb0fc0c55d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb102c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc142ad0 con 0x7fb0fc14b280 2024-01-15T18:04:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb102c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc1616c0 con 0x7fb0fc0cf520 2024-01-15T18:04:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb102c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0fc1f7f60 con 0x7fb0fc0b2a00 2024-01-15T18:04:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.848+0000 7fb100c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 0x7fb0fc0c55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.849+0000 7fb100c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 0x7fb0fc0c55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50334/0 (socket says 172.21.15.2:50334) 2024-01-15T18:04:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.849+0000 7fb100c00640 1 -- 172.21.15.2:0/666500457 learned_addr learned my addr 172.21.15.2:0/666500457 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.849+0000 7fb0fbe00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0b2a00 0x7fb0fc1f8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.850+0000 7fb100c00640 1 -- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0b2a00 msgr2=0x7fb0fc1f8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.850+0000 7fb100c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0b2a00 0x7fb0fc1f8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.850+0000 7fb100c00640 1 -- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0cf520 msgr2=0x7fb0fc161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.850+0000 7fb100c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0cf520 0x7fb0fc161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.850+0000 7fb100c00640 1 -- 172.21.15.2:0/666500457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f4009a70 con 0x7fb0fc14b280 2024-01-15T18:04:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.850+0000 7fb100c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 0x7fb0fc0c55d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009ee0 tx=0x7fb0f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.851+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f403e610 con 0x7fb0fc14b280 2024-01-15T18:04:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.851+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f403e770 con 0x7fb0fc14b280 2024-01-15T18:04:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.851+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f403eb20 con 0x7fb0fc14b280 2024-01-15T18:04:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.851+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0fc142cb0 con 0x7fb0fc14b280 2024-01-15T18:04:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.852+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0fc143270 con 0x7fb0fc14b280 2024-01-15T18:04:57.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.854+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0fc143490 con 0x7fb0fc14b280 2024-01-15T18:04:57.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.858+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb0f403b050 con 0x7fb0fc14b280 2024-01-15T18:04:57.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.861+0000 7fb0f3e00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0d406bc00 0x7fb0d406e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.862+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb0f40aacd0 con 0x7fb0fc14b280 2024-01-15T18:04:57.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.862+0000 7fb0fbe00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0d406bc00 0x7fb0d406e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:57.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.865+0000 7fb0fbe00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0d406bc00 0x7fb0d406e0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000f60 tx=0x7fb0e4007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:57.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:57.867+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0f4077470 con 0x7fb0fc14b280 2024-01-15T18:04:58.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.260+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0fc0b3690 con 0x7fb0d406bc00 2024-01-15T18:04:58.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.290+0000 7fb0f3e00640 1 -- 172.21.15.2:0/666500457 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fb0e4007300 con 0x7fb0d406bc00 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:04:58.294 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:04:58.295 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:04:58.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0d406bc00 msgr2=0x7fb0d406e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0d406bc00 0x7fb0d406e0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4000f60 tx=0x7fb0e4007e40 comp rx=0 tx=0).stop 2024-01-15T18:04:58.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 msgr2=0x7fb0fc0c55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 0x7fb0fc0c55d0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4009ee0 tx=0x7fb0f4004a40 comp rx=0 tx=0).stop 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 shutdown_connections 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb0d406bc00 0x7fb0d406e0c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0fc14b280 0x7fb0fc0c55d0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0fc0cf520 0x7fb0fc161180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 --2- 172.21.15.2:0/666500457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0fc0b2a00 0x7fb0fc1f8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 >> 172.21.15.2:0/666500457 conn(0x7fb0fc06c4d0 msgr2=0x7fb0fc0ccad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.293+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 shutdown_connections 2024-01-15T18:04:58.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:58.294+0000 7fb102c00640 1 -- 172.21.15.2:0/666500457 wait complete. 2024-01-15T18:04:58.326 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:04:59.326 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27425 2024-01-15T18:04:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.619+0000 7fb0bc200640 1 Processor -- start 2024-01-15T18:04:59.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.619+0000 7fb0bc200640 1 -- start start 2024-01-15T18:04:59.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.621+0000 7fb0bc200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb0b40b4c70 0x7fb0b40d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.621+0000 7fb0bc200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b40b8c30 0x7fb0b40d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.621+0000 7fb0bc200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0b40d1900 con 0x7fb0b415d730 2024-01-15T18:04:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.621+0000 7fb0bc200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0b40d1be0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.621+0000 7fb0bc200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b40d1ec0 con 0x7fb0b40b8c30 2024-01-15T18:04:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.622+0000 7fb0b9800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb0b40b4c70 0x7fb0b40d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.622+0000 7fb0b9800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb0b40b4c70 0x7fb0b40d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50344/0 (socket says 172.21.15.2:50344) 2024-01-15T18:04:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.622+0000 7fb0b8e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb0b415d730 0x7fb0b40b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44124/0 (socket says 172.21.15.2:44124) 2024-01-15T18:04:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.622+0000 7fb0b9800640 1 -- 172.21.15.2:0/3257979723 learned_addr learned my addr 172.21.15.2:0/3257979723 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:59.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.622+0000 7fb0ba200640 1 --2- 172.21.15.2:0/3257979723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b40b8c30 0x7fb0b40d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.623+0000 7fb0b9800640 1 -- 172.21.15.2:0/3257979723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b40b8c30 msgr2=0x7fb0b40d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:04:59.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.623+0000 7fb0b9800640 1 --2- 172.21.15.2:0/3257979723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b40b8c30 0x7fb0b40d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.624+0000 7fb0b9800640 1 -- 172.21.15.2:0/3257979723 >> v1:172.21.15.2:6789/0 conn(0x7fb0b415d730 legacy=0x7fb0b40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.624+0000 7fb0b9800640 1 -- 172.21.15.2:0/3257979723 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b40d21a0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.626+0000 7fb0b9800640 1 --2- 172.21.15.2:0/3257979723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b40d1270 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009e50 tx=0x7fb0a402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a97ee9def073732 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.627+0000 7fb0b3e00640 1 -- 172.21.15.2:0/3257979723 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a402cb20 con 0x7fb0b40b4c70 2024-01-15T18:04:59.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.627+0000 7fb0b3e00640 1 -- 172.21.15.2:0/3257979723 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a402cc80 con 0x7fb0b40b4c70 2024-01-15T18:04:59.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.627+0000 7fb0b3e00640 1 -- 172.21.15.2:0/3257979723 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40343f0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.628+0000 7fb0bc200640 1 -- 172.21.15.2:0/3257979723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 msgr2=0x7fb0b40d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.628+0000 7fb0bc200640 1 --2- 172.21.15.2:0/3257979723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b40d1270 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4009e50 tx=0x7fb0a402bf80 comp rx=0 tx=0).stop 2024-01-15T18:04:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.628+0000 7fb0bc200640 1 -- 172.21.15.2:0/3257979723 shutdown_connections 2024-01-15T18:04:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.628+0000 7fb0bc200640 1 --2- 172.21.15.2:0/3257979723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b40b8c30 0x7fb0b40d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.628+0000 7fb0bc200640 1 --2- 172.21.15.2:0/3257979723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b40d1270 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.629+0000 7fb0bc200640 1 -- 172.21.15.2:0/3257979723 >> 172.21.15.2:0/3257979723 conn(0x7fb0b4072210 msgr2=0x7fb0b40cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:04:59.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.630+0000 7fb0bc200640 1 -- 172.21.15.2:0/3257979723 shutdown_connections 2024-01-15T18:04:59.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.631+0000 7fb0bc200640 1 -- 172.21.15.2:0/3257979723 wait complete. 2024-01-15T18:04:59.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.634+0000 7fb0bc200640 1 Processor -- start 2024-01-15T18:04:59.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.635+0000 7fb0bc200640 1 -- start start 2024-01-15T18:04:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.636+0000 7fb0bc200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b415f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0bc200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0b40b8c30 0x7fb0b40ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0bc200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b415d730 0x7fb0b41f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0bc200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b4143280 con 0x7fb0b40b4c70 2024-01-15T18:04:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0bc200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b415fdf0 con 0x7fb0b40b8c30 2024-01-15T18:04:59.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0bc200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b41600d0 con 0x7fb0b415d730 2024-01-15T18:04:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0b9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b415f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0b9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b415f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50350/0 (socket says 172.21.15.2:50350) 2024-01-15T18:04:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.637+0000 7fb0b9800640 1 -- 172.21.15.2:0/2175535910 learned_addr learned my addr 172.21.15.2:0/2175535910 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:04:59.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.638+0000 7fb0b9800640 1 -- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b415d730 msgr2=0x7fb0b41f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.638+0000 7fb0b9800640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b415d730 0x7fb0b41f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.638+0000 7fb0b9800640 1 -- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0b40b8c30 msgr2=0x7fb0b40ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:04:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.638+0000 7fb0b9800640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0b40b8c30 0x7fb0b40ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:04:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.638+0000 7fb0b9800640 1 -- 172.21.15.2:0/2175535910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4009b30 con 0x7fb0b40b4c70 2024-01-15T18:04:59.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.639+0000 7fb0b9800640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b415f540 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4036040 tx=0x7fb0a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.639+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40043f0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.639+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a4004570 con 0x7fb0b40b4c70 2024-01-15T18:04:59.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.639+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a403e450 con 0x7fb0b40b4c70 2024-01-15T18:04:59.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.640+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0b40d3060 con 0x7fb0b40b4c70 2024-01-15T18:04:59.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.640+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0b4160690 con 0x7fb0b40b4c70 2024-01-15T18:04:59.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.641+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0b40686f0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.643+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb0a403e5b0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.644+0000 7fb0b1600640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb08c06bb80 0x7fb08c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:04:59.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.644+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb0a40aada0 con 0x7fb0b40b4c70 2024-01-15T18:04:59.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.648+0000 7fb0b8e00640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb08c06bb80 0x7fb08c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:04:59.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.649+0000 7fb0b8e00640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb08c06bb80 0x7fb08c06e040 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40ceb00 tx=0x7fb09c0079c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:04:59.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:04:59.650+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb0a4077490 con 0x7fb0b40b4c70 2024-01-15T18:05:00.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.057+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0b40d02b0 con 0x7fb08c06bb80 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.138+0000 7fb0b1600640 1 -- 172.21.15.2:0/2175535910 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fb0b40d02b0 con 0x7fb08c06bb80 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:05:00.141 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:00.142 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.143+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb08c06bb80 msgr2=0x7fb08c06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.143+0000 7fb0bc200640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb08c06bb80 0x7fb08c06e040 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb0b40ceb00 tx=0x7fb09c0079c0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.143+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 msgr2=0x7fb0b415f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:00.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.143+0000 7fb0bc200640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b415f540 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4036040 tx=0x7fb0a4004a40 comp rx=0 tx=0).stop 2024-01-15T18:05:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.149+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 shutdown_connections 2024-01-15T18:05:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.149+0000 7fb0bc200640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb08c06bb80 0x7fb08c06e040 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.149+0000 7fb0bc200640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb0b415d730 0x7fb0b41f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.149+0000 7fb0bc200640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb0b40b8c30 0x7fb0b40ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.149+0000 7fb0bc200640 1 --2- 172.21.15.2:0/2175535910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb0b40b4c70 0x7fb0b415f540 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:00.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.149+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 >> 172.21.15.2:0/2175535910 conn(0x7fb0b4072210 msgr2=0x7fb0b40b74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.150+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 shutdown_connections 2024-01-15T18:05:00.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:00.150+0000 7fb0bc200640 1 -- 172.21.15.2:0/2175535910 wait complete. 2024-01-15T18:05:00.205 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:05:00.680 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:00.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:01.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27425 2024-01-15T18:05:01.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.506+0000 7fd088400640 1 Processor -- start 2024-01-15T18:05:01.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.507+0000 7fd088400640 1 -- start start 2024-01-15T18:05:01.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.510+0000 7fd088400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.510+0000 7fd088400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd08014b3b0 0x7fd08006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.510+0000 7fd088400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd08006ce30 0x7fd0800702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.510+0000 7fd088400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd080070ff0 con 0x7fd08014b3b0 2024-01-15T18:05:01.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.510+0000 7fd088400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08006d270 con 0x7fd08006ce30 2024-01-15T18:05:01.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.510+0000 7fd088400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800b25f0 con 0x7fd08014afd0 2024-01-15T18:05:01.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.512+0000 7fd086400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd08006ce30 0x7fd0800702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.512+0000 7fd085000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd08014b3b0 0x7fd08006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.513+0000 7fd086400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd08006ce30 0x7fd0800702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:34138/0 (socket says 172.21.15.2:34138) 2024-01-15T18:05:01.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.513+0000 7fd086400640 1 -- 172.21.15.2:0/873923046 learned_addr learned my addr 172.21.15.2:0/873923046 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:01.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.515+0000 7fd085a00640 1 --2- 172.21.15.2:0/873923046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.516+0000 7fd085000640 1 -- 172.21.15.2:0/873923046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 msgr2=0x7fd0800b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:01.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.516+0000 7fd085000640 1 --2- 172.21.15.2:0/873923046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.516+0000 7fd085000640 1 -- 172.21.15.2:0/873923046 >> v2:172.21.15.137:3300/0 conn(0x7fd08006ce30 msgr2=0x7fd0800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:01.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.516+0000 7fd085000640 1 --2- 172.21.15.2:0/873923046 >> v2:172.21.15.137:3300/0 conn(0x7fd08006ce30 0x7fd0800702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.516+0000 7fd085000640 1 -- 172.21.15.2:0/873923046 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800b28d0 con 0x7fd08014b3b0 2024-01-15T18:05:01.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.518+0000 7fd085000640 1 --2- 172.21.15.2:0/873923046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08014b3b0 0x7fd08006c8f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd0800b4390 tx=0x7fd07002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb8fa3293833d98 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.520+0000 7fd07fe00640 1 -- 172.21.15.2:0/873923046 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07002f050 con 0x7fd08014b3b0 2024-01-15T18:05:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.520+0000 7fd07fe00640 1 -- 172.21.15.2:0/873923046 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070031700 con 0x7fd08014b3b0 2024-01-15T18:05:01.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.520+0000 7fd07fe00640 1 -- 172.21.15.2:0/873923046 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070031a90 con 0x7fd08014b3b0 2024-01-15T18:05:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.521+0000 7fd088400640 1 -- 172.21.15.2:0/873923046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08014b3b0 msgr2=0x7fd08006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.521+0000 7fd088400640 1 --2- 172.21.15.2:0/873923046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08014b3b0 0x7fd08006c8f0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fd0800b4390 tx=0x7fd07002b7a0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.522+0000 7fd088400640 1 -- 172.21.15.2:0/873923046 shutdown_connections 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.522+0000 7fd088400640 1 --2- 172.21.15.2:0/873923046 >> v2:172.21.15.137:3300/0 conn(0x7fd08006ce30 0x7fd0800702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.522+0000 7fd088400640 1 --2- 172.21.15.2:0/873923046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08014b3b0 0x7fd08006c8f0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.522+0000 7fd088400640 1 --2- 172.21.15.2:0/873923046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.522+0000 7fd088400640 1 -- 172.21.15.2:0/873923046 >> 172.21.15.2:0/873923046 conn(0x7fd0800b45f0 msgr2=0x7fd0800b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.522+0000 7fd088400640 1 -- 172.21.15.2:0/873923046 shutdown_connections 2024-01-15T18:05:01.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.523+0000 7fd088400640 1 -- 172.21.15.2:0/873923046 wait complete. 2024-01-15T18:05:01.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.527+0000 7fd088400640 1 Processor -- start 2024-01-15T18:05:01.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.527+0000 7fd088400640 1 -- start start 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.530+0000 7fd088400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 0x7fd08015f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.530+0000 7fd088400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.530+0000 7fd088400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd08014b3b0 0x7fd0800c5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.530+0000 7fd088400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08015f8d0 con 0x7fd08006ce30 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.530+0000 7fd088400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c47d0 con 0x7fd08014b3b0 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.530+0000 7fd088400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0800c4ab0 con 0x7fd08014afd0 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.531+0000 7fd085a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 0x7fd08015f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.531+0000 7fd085a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 0x7fd08015f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50364/0 (socket says 172.21.15.2:50364) 2024-01-15T18:05:01.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.531+0000 7fd085a00640 1 -- 172.21.15.2:0/360820919 learned_addr learned my addr 172.21.15.2:0/360820919 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:01.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085000640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085a00640 1 -- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 msgr2=0x7fd0800c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085a00640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085a00640 1 -- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd08014b3b0 msgr2=0x7fd0800c5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:01.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085a00640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd08014b3b0 0x7fd0800c5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:01.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085a00640 1 -- 172.21.15.2:0/360820919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd070009400 con 0x7fd08006ce30 2024-01-15T18:05:01.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.532+0000 7fd085000640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:01.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.533+0000 7fd085a00640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 0x7fd08015f390 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd06800d800 tx=0x7fd06800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:01.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.533+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06800b6c0 con 0x7fd08006ce30 2024-01-15T18:05:01.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.534+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06800b820 con 0x7fd08006ce30 2024-01-15T18:05:01.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.534+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0800c4d90 con 0x7fd08006ce30 2024-01-15T18:05:01.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.534+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06800bb90 con 0x7fd08006ce30 2024-01-15T18:05:01.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.535+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0800c5350 con 0x7fd08006ce30 2024-01-15T18:05:01.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.537+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fd06800bcf0 con 0x7fd08006ce30 2024-01-15T18:05:01.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.537+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0800686f0 con 0x7fd08006ce30 2024-01-15T18:05:01.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.540+0000 7fd07d600640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd05806bb60 0x7fd05806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:01.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.542+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fd06808f170 con 0x7fd08006ce30 2024-01-15T18:05:01.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.544+0000 7fd085000640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd05806bb60 0x7fd05806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:01.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.546+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd068090060 con 0x7fd08006ce30 2024-01-15T18:05:01.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:01.561+0000 7fd085000640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd05806bb60 0x7fd05806e020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd070035000 tx=0x7fd070008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:02.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.015+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0800c4c20 con 0x7fd05806bb60 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:05:02.063 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:02.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.058+0000 7fd07d600640 1 -- 172.21.15.2:0/360820919 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fd0800c4c20 con 0x7fd05806bb60 2024-01-15T18:05:02.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.063+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd05806bb60 msgr2=0x7fd05806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.063+0000 7fd088400640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd05806bb60 0x7fd05806e020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd070035000 tx=0x7fd070008030 comp rx=0 tx=0).stop 2024-01-15T18:05:02.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.063+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 msgr2=0x7fd08015f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:02.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.063+0000 7fd088400640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 0x7fd08015f390 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fd06800d800 tx=0x7fd06800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.064+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 shutdown_connections 2024-01-15T18:05:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.064+0000 7fd088400640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd05806bb60 0x7fd05806e020 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.064+0000 7fd088400640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd08014b3b0 0x7fd0800c5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.064+0000 7fd088400640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd08014afd0 0x7fd0800c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.064+0000 7fd088400640 1 --2- 172.21.15.2:0/360820919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd08006ce30 0x7fd08015f390 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.064+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 >> 172.21.15.2:0/360820919 conn(0x7fd0800b45f0 msgr2=0x7fd0800cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:02.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.065+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 shutdown_connections 2024-01-15T18:05:02.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:02.065+0000 7fd088400640 1 -- 172.21.15.2:0/360820919 wait complete. 2024-01-15T18:05:02.098 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:05:03.098 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27425 2024-01-15T18:05:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.399+0000 7f81e9400640 1 Processor -- start 2024-01-15T18:05:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.400+0000 7f81e9400640 1 -- start start 2024-01-15T18:05:03.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.401+0000 7f81e9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e40d1270 0x7f81e40d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.401+0000 7f81e9400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40d1900 con 0x7f81e415d730 2024-01-15T18:05:03.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.401+0000 7f81e9400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e40d1be0 con 0x7f81e40b4c70 2024-01-15T18:05:03.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.401+0000 7f81e9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e40d1700 con 0x7f81e40d1270 2024-01-15T18:05:03.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.402+0000 7f81e1e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f81e415d730 0x7f81e40b8c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57054/0 (socket says 172.21.15.2:57054) 2024-01-15T18:05:03.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.402+0000 7f81e3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e40d1270 0x7f81e40d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.402+0000 7f81e1e00640 1 -- 172.21.15.2:0/2576398284 learned_addr learned my addr 172.21.15.2:0/2576398284 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.402+0000 7f81e2800640 1 --1- 172.21.15.2:0/2576398284 >> v1:172.21.15.137:6789/0 conn(0x7f81e40b4c70 0x7f81e40b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:56034/0 (socket says 172.21.15.2:56034) 2024-01-15T18:05:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.403+0000 7f81e1400640 1 -- 172.21.15.2:0/2576398284 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148439243 0 0) 0x7f81e40d1900 con 0x7f81e415d730 2024-01-15T18:05:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.403+0000 7f81e1400640 1 -- 172.21.15.2:0/2576398284 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c4003520 con 0x7f81e415d730 2024-01-15T18:05:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.403+0000 7f81e3200640 1 -- 172.21.15.2:0/2576398284 >> v1:172.21.15.137:6789/0 conn(0x7f81e40b4c70 legacy=0x7f81e40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.403+0000 7f81e3200640 1 -- 172.21.15.2:0/2576398284 >> v1:172.21.15.2:6789/0 conn(0x7f81e415d730 legacy=0x7f81e40b8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.403+0000 7f81e3200640 1 -- 172.21.15.2:0/2576398284 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e40d1ec0 con 0x7f81e40d1270 2024-01-15T18:05:03.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.407+0000 7f81e3200640 1 --2- 172.21.15.2:0/2576398284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e40d1270 0x7f81e40d29a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009dc0 tx=0x7f81dc02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f8c79b4c0e49866 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.408+0000 7f81e1400640 1 -- 172.21.15.2:0/2576398284 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc02ccc0 con 0x7f81e40d1270 2024-01-15T18:05:03.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.408+0000 7f81e1400640 1 -- 172.21.15.2:0/2576398284 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81dc02ce20 con 0x7f81e40d1270 2024-01-15T18:05:03.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.408+0000 7f81e1400640 1 -- 172.21.15.2:0/2576398284 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81dc034660 con 0x7f81e40d1270 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.409+0000 7f81e9400640 1 -- 172.21.15.2:0/2576398284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e40d1270 msgr2=0x7f81e40d29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.409+0000 7f81e9400640 1 --2- 172.21.15.2:0/2576398284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e40d1270 0x7f81e40d29a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f81dc009dc0 tx=0x7f81dc02bf10 comp rx=0 tx=0).stop 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.410+0000 7f81e9400640 1 -- 172.21.15.2:0/2576398284 shutdown_connections 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.410+0000 7f81e9400640 1 --2- 172.21.15.2:0/2576398284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e40d1270 0x7f81e40d29a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.410+0000 7f81e9400640 1 -- 172.21.15.2:0/2576398284 >> 172.21.15.2:0/2576398284 conn(0x7f81e4072210 msgr2=0x7f81e40cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.410+0000 7f81e9400640 1 -- 172.21.15.2:0/2576398284 shutdown_connections 2024-01-15T18:05:03.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.410+0000 7f81e9400640 1 -- 172.21.15.2:0/2576398284 wait complete. 2024-01-15T18:05:03.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.413+0000 7f81e9400640 1 Processor -- start 2024-01-15T18:05:03.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.413+0000 7f81e9400640 1 -- start start 2024-01-15T18:05:03.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.414+0000 7f81e9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 0x7f81e41594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.414+0000 7f81e9400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81e40d1270 0x7f81e40c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.414+0000 7f81e9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e415d730 0x7f81e40c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.414+0000 7f81e9400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e41f78b0 con 0x7f81e40b4c70 2024-01-15T18:05:03.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.414+0000 7f81e9400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e415e560 con 0x7f81e40d1270 2024-01-15T18:05:03.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.414+0000 7f81e9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e415f380 con 0x7f81e415d730 2024-01-15T18:05:03.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.415+0000 7f81e2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 0x7f81e41594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.415+0000 7f81e2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 0x7f81e41594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52024/0 (socket says 172.21.15.2:52024) 2024-01-15T18:05:03.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.415+0000 7f81e2800640 1 -- 172.21.15.2:0/2518766774 learned_addr learned my addr 172.21.15.2:0/2518766774 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:03.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.415+0000 7f81e3200640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e415d730 0x7f81e40c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.415+0000 7f81e1e00640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81e40d1270 0x7f81e40c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.416+0000 7f81e2800640 1 -- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e415d730 msgr2=0x7f81e40c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.416+0000 7f81e2800640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e415d730 0x7f81e40c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.416+0000 7f81e2800640 1 -- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81e40d1270 msgr2=0x7f81e40c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.416+0000 7f81e2800640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81e40d1270 0x7f81e40c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.416+0000 7f81e2800640 1 -- 172.21.15.2:0/2518766774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81dc009a70 con 0x7f81e40b4c70 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.416+0000 7f81e2800640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 0x7f81e41594b0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c4d0 tx=0x7f81d400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.417+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d4007480 con 0x7f81e40b4c70 2024-01-15T18:05:03.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.417+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d40075e0 con 0x7f81e40b4c70 2024-01-15T18:05:03.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.417+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d40049e0 con 0x7f81e40b4c70 2024-01-15T18:05:03.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.417+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e40d3060 con 0x7f81e40b4c70 2024-01-15T18:05:03.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.418+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e4071230 con 0x7f81e40b4c70 2024-01-15T18:05:03.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.420+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f81d4004b40 con 0x7f81e40b4c70 2024-01-15T18:05:03.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.420+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e40b3b40 con 0x7f81e40b4c70 2024-01-15T18:05:03.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.422+0000 7f81daa00640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f81bc06bb30 0x7f81bc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:03.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.422+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f81d4087620 con 0x7f81e40b4c70 2024-01-15T18:05:03.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.428+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f81d4053d90 con 0x7f81e40b4c70 2024-01-15T18:05:03.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.430+0000 7f81e1e00640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f81bc06bb30 0x7f81bc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:03.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.432+0000 7f81e1e00640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f81bc06bb30 0x7f81bc06dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f81e415e4f0 tx=0x7f81cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:03.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.818+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7f81e40d2ee0 con 0x7f81bc06bb30 2024-01-15T18:05:03.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.881+0000 7f81daa00640 1 -- 172.21.15.2:0/2518766774 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f81cc001d50 con 0x7f81bc06bb30 2024-01-15T18:05:03.887 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:03.887 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:03.888 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f81bc06bb30 msgr2=0x7f81bc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f81bc06bb30 0x7f81bc06dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f81e415e4f0 tx=0x7f81cc009040 comp rx=0 tx=0).stop 2024-01-15T18:05:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 msgr2=0x7f81e41594b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 0x7f81e41594b0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f81d400c4d0 tx=0x7f81d400c990 comp rx=0 tx=0).stop 2024-01-15T18:05:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 shutdown_connections 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f81bc06bb30 0x7f81bc06dff0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f81e415d730 0x7f81e40c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f81e40d1270 0x7f81e40c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 --2- 172.21.15.2:0/2518766774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f81e40b4c70 0x7f81e41594b0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.886+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 >> 172.21.15.2:0/2518766774 conn(0x7f81e4072210 msgr2=0x7f81e40b3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.887+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 shutdown_connections 2024-01-15T18:05:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:03.887+0000 7f81e9400640 1 -- 172.21.15.2:0/2518766774 wait complete. 2024-01-15T18:05:03.925 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:05:04.926 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27425 2024-01-15T18:05:05.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.226+0000 7f144fa00640 1 Processor -- start 2024-01-15T18:05:05.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.226+0000 7f144fa00640 1 -- start start 2024-01-15T18:05:05.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.227+0000 7f144fa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f14480b1290 0x7f14480ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.227+0000 7f144fa00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14480cf330 con 0x7f144806e3b0 2024-01-15T18:05:05.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.228+0000 7f144fa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1448070c50 con 0x7f144815ca60 2024-01-15T18:05:05.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.228+0000 7f144fa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1448070f30 con 0x7f14480b1290 2024-01-15T18:05:05.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.228+0000 7f143fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f14480b1290 0x7f14480ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.228+0000 7f143fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f14480b1290 0x7f14480ce270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52040/0 (socket says 172.21.15.2:52040) 2024-01-15T18:05:05.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.229+0000 7f143fe00640 1 -- 172.21.15.2:0/2615140238 learned_addr learned my addr 172.21.15.2:0/2615140238 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:05.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.230+0000 7f143fe00640 1 -- 172.21.15.2:0/2615140238 >> v1:172.21.15.137:6789/0 conn(0x7f144815ca60 legacy=0x7f14480b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:05.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.230+0000 7f1447e00640 1 -- 172.21.15.2:0/2615140238 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635630730 0 0) 0x7f14480cf330 con 0x7f144806e3b0 2024-01-15T18:05:05.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.230+0000 7f143fe00640 1 -- 172.21.15.2:0/2615140238 >> v1:172.21.15.2:6789/0 conn(0x7f144806e3b0 legacy=0x7f14480b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.230+0000 7f143fe00640 1 -- 172.21.15.2:0/2615140238 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14480cf5b0 con 0x7f14480b1290 2024-01-15T18:05:05.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.233+0000 7f143fe00640 1 --2- 172.21.15.2:0/2615140238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14480b1290 0x7f14480ce270 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1438009870 tx=0x7f143802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928515ecf4b2ff58 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.234+0000 7f1447e00640 1 -- 172.21.15.2:0/2615140238 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438031870 con 0x7f14480b1290 2024-01-15T18:05:05.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.234+0000 7f1447e00640 1 -- 172.21.15.2:0/2615140238 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14380319d0 con 0x7f14480b1290 2024-01-15T18:05:05.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.234+0000 7f1447e00640 1 -- 172.21.15.2:0/2615140238 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438031870 con 0x7f14480b1290 2024-01-15T18:05:05.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.236+0000 7f144fa00640 1 -- 172.21.15.2:0/2615140238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14480b1290 msgr2=0x7f14480ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.236+0000 7f144fa00640 1 --2- 172.21.15.2:0/2615140238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14480b1290 0x7f14480ce270 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1438009870 tx=0x7f143802b910 comp rx=0 tx=0).stop 2024-01-15T18:05:05.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.237+0000 7f144fa00640 1 -- 172.21.15.2:0/2615140238 shutdown_connections 2024-01-15T18:05:05.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.237+0000 7f144fa00640 1 --2- 172.21.15.2:0/2615140238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14480b1290 0x7f14480ce270 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.237+0000 7f144fa00640 1 -- 172.21.15.2:0/2615140238 >> 172.21.15.2:0/2615140238 conn(0x7f14480b45f0 msgr2=0x7f14480b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.239+0000 7f144fa00640 1 -- 172.21.15.2:0/2615140238 shutdown_connections 2024-01-15T18:05:05.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.239+0000 7f144fa00640 1 -- 172.21.15.2:0/2615140238 wait complete. 2024-01-15T18:05:05.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.241+0000 7f144fa00640 1 Processor -- start 2024-01-15T18:05:05.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.241+0000 7f144fa00640 1 -- start start 2024-01-15T18:05:05.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.243+0000 7f144fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 0x7f1448154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.243+0000 7f144fa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 0x7f144815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.243+0000 7f144fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14481560b0 0x7f144815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.243+0000 7f144fa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448150a30 con 0x7f144815ca60 2024-01-15T18:05:05.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.243+0000 7f144fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448157e90 con 0x7f14481560b0 2024-01-15T18:05:05.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.243+0000 7f144fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14481f89c0 con 0x7f144806e3b0 2024-01-15T18:05:05.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.244+0000 7f144d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 0x7f1448154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.244+0000 7f143fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 0x7f144815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.244+0000 7f144d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 0x7f1448154690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:41438/0 (socket says 172.21.15.2:41438) 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.244+0000 7f144d000640 1 -- 172.21.15.2:0/3977934092 learned_addr learned my addr 172.21.15.2:0/3977934092 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.244+0000 7f143fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 0x7f144815b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52056/0 (socket says 172.21.15.2:52056) 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.244+0000 7f144da00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14481560b0 0x7f144815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f143fe00640 1 -- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 msgr2=0x7f1448154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f143fe00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 0x7f1448154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f143fe00640 1 -- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14481560b0 msgr2=0x7f144815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f143fe00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14481560b0 0x7f144815bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f143fe00640 1 -- 172.21.15.2:0/3977934092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14380094d0 con 0x7f144815ca60 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f144d000640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 0x7f1448154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.245+0000 7f143fe00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 0x7f144815b5e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1438033000 tx=0x7f14380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.246+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1438003a20 con 0x7f144815ca60 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.246+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1438003b80 con 0x7f144815ca60 2024-01-15T18:05:05.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.246+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143803a420 con 0x7f144815ca60 2024-01-15T18:05:05.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.246+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1448154bd0 con 0x7f144815ca60 2024-01-15T18:05:05.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.247+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14481f86e0 con 0x7f144815ca60 2024-01-15T18:05:05.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.249+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14480686f0 con 0x7f144815ca60 2024-01-15T18:05:05.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.250+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f143803a580 con 0x7f144815ca60 2024-01-15T18:05:05.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.252+0000 7f1445600640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f142006bc00 0x7f142006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:05.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.252+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f143802d040 con 0x7f144815ca60 2024-01-15T18:05:05.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.254+0000 7f144d000640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f142006bc00 0x7f142006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:05.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.256+0000 7f144d000640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f142006bc00 0x7f142006e0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f14300099f0 tx=0x7f1430008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:05.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.261+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1438077420 con 0x7f144815ca60 2024-01-15T18:05:05.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.638+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7f14480b3e70 con 0x7f142006bc00 2024-01-15T18:05:05.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.697+0000 7f1445600640 1 -- 172.21.15.2:0/3977934092 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f14300095e0 con 0x7f142006bc00 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:05:05.702 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:05.703 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:05.703 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:05.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.705+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f142006bc00 msgr2=0x7f142006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.705+0000 7f144fa00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f142006bc00 0x7f142006e0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f14300099f0 tx=0x7f1430008040 comp rx=0 tx=0).stop 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.705+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 msgr2=0x7f144815b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.705+0000 7f144fa00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 0x7f144815b5e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1438033000 tx=0x7f14380046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.706+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 shutdown_connections 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.706+0000 7f144fa00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f142006bc00 0x7f142006e0c0 secure :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f14300099f0 tx=0x7f1430008040 comp rx=0 tx=0).stop 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.706+0000 7f144fa00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f14481560b0 0x7f144815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.706+0000 7f144fa00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f144815ca60 0x7f144815b5e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.706+0000 7f144fa00640 1 --2- 172.21.15.2:0/3977934092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f144806e3b0 0x7f1448154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.706+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 >> 172.21.15.2:0/3977934092 conn(0x7f14480b45f0 msgr2=0x7f1448152d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.707+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 shutdown_connections 2024-01-15T18:05:05.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:05.707+0000 7f144fa00640 1 -- 172.21.15.2:0/3977934092 wait complete. 2024-01-15T18:05:05.748 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:05:06.749 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27425 2024-01-15T18:05:07.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.071+0000 7f247f000640 1 Processor -- start 2024-01-15T18:05:07.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.072+0000 7f247f000640 1 -- start start 2024-01-15T18:05:07.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f247f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247815c450 0x7f247806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f247f000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f247806ce30 0x7f2478070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f247f000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24780ce230 con 0x7f2478144c00 2024-01-15T18:05:07.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f247f000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247806d2c0 con 0x7f247806ce30 2024-01-15T18:05:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f247f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780ce510 con 0x7f247815c450 2024-01-15T18:05:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f2477e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2478144c00 0x7f24780cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57062/0 (socket says 172.21.15.2:57062) 2024-01-15T18:05:07.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.073+0000 7f2477e00640 1 -- 172.21.15.2:0/1817312352 learned_addr learned my addr 172.21.15.2:0/1817312352 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737336282 0 0) 0x7f24780ce230 con 0x7f2478144c00 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2458003520 con 0x7f2478144c00 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242432477 0 0) 0x7f2458003520 con 0x7f2478144c00 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f247d000640 1 --2- 172.21.15.2:0/1817312352 >> v2:172.21.15.137:3300/0 conn(0x7f247806ce30 0x7f2478070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247815c450 msgr2=0x7f247806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247815c450 0x7f247806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 >> v2:172.21.15.137:3300/0 conn(0x7f247806ce30 msgr2=0x7f2478070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 --2- 172.21.15.2:0/1817312352 >> v2:172.21.15.137:3300/0 conn(0x7f247806ce30 0x7f2478070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24780ce7f0 con 0x7f2478144c00 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2468001670 con 0x7f2478144c00 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 >> v1:172.21.15.2:6789/0 conn(0x7f2478144c00 legacy=0x7f24780cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247815c450 0x7f2458003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2458004080 0x7f24580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2458004d50 0x7f24580051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247806d2c0 con 0x7f2458004080 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780ce510 con 0x7f2458004d50 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245800bc40 con 0x7f247815c450 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f247d000640 1 --2- 172.21.15.2:0/1817312352 >> v2:172.21.15.137:3300/0 conn(0x7f247806ce30 0x7f2478070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.074+0000 7f246fe00640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2458004080 0x7f24580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.075+0000 7f247d000640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2458004d50 0x7f24580051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.075+0000 7f246fe00640 1 -- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247815c450 msgr2=0x7f2458003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.075+0000 7f246fe00640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f247815c450 0x7f2458003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.075+0000 7f246fe00640 1 -- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2458004d50 msgr2=0x7f24580051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.075+0000 7f246fe00640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2458004d50 0x7f24580051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.075+0000 7f246fe00640 1 -- 172.21.15.2:0/1817312352 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458003520 con 0x7f2458004080 2024-01-15T18:05:07.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.076+0000 7f246fe00640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2458004080 0x7f24580047c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2460009860 tx=0x7f24580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ab9cf842f0f9af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.076+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2460032600 con 0x7f2458004080 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.076+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246002e050 con 0x7f2458004080 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f2477400640 1 -- 172.21.15.2:0/1817312352 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24600329c0 con 0x7f2458004080 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247f000640 1 -- 172.21.15.2:0/1817312352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2458004080 msgr2=0x7f24580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247f000640 1 --2- 172.21.15.2:0/1817312352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2458004080 0x7f24580047c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f2460009860 tx=0x7f24580057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247d000640 1 -- 172.21.15.2:0/1817312352 reap_dead start 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247f000640 1 -- 172.21.15.2:0/1817312352 shutdown_connections 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247f000640 1 -- 172.21.15.2:0/1817312352 >> 172.21.15.2:0/1817312352 conn(0x7f24780b45f0 msgr2=0x7f24780b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247f000640 1 -- 172.21.15.2:0/1817312352 shutdown_connections 2024-01-15T18:05:07.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.077+0000 7f247f000640 1 -- 172.21.15.2:0/1817312352 wait complete. 2024-01-15T18:05:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.079+0000 7f247f000640 1 Processor -- start 2024-01-15T18:05:07.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.079+0000 7f247f000640 1 -- start start 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.080+0000 7f247f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2458004080 0x7f24781f9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.080+0000 7f247f000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 0x7f24781383e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.080+0000 7f247f000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24780afc90 0x7f2478138920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.080+0000 7f247f000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2478139d50 con 0x7f24780af810 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.080+0000 7f247f000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24780b1ae0 con 0x7f24780afc90 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.080+0000 7f247f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24780b2c00 con 0x7f2458004080 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 0x7f24781383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 0x7f24781383e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52074/0 (socket says 172.21.15.2:52074) 2024-01-15T18:05:07.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 -- 172.21.15.2:0/4188269775 learned_addr learned my addr 172.21.15.2:0/4188269775 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 -- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2458004080 msgr2=0x7f24781f9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2458004080 0x7f24781f9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 -- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24780afc90 msgr2=0x7f2478138920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24780afc90 0x7f2478138920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 -- 172.21.15.2:0/4188269775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2478156060 con 0x7f24780af810 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.081+0000 7f246fe00640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 0x7f24781383e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2460000c00 tx=0x7f24600047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.082+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246003d890 con 0x7f24780af810 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.082+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246003d9f0 con 0x7f24780af810 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.082+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246003dd80 con 0x7f24780af810 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.082+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24780b1ff0 con 0x7f24780af810 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.082+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24780b25b0 con 0x7f24780af810 2024-01-15T18:05:07.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.083+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24780686f0 con 0x7f24780af810 2024-01-15T18:05:07.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.084+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f2460039040 con 0x7f24780af810 2024-01-15T18:05:07.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.085+0000 7f2474c00640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f245006bb60 0x7f245006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:07.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.085+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f24600ac670 con 0x7f24780af810 2024-01-15T18:05:07.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.089+0000 7f2477e00640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f245006bb60 0x7f245006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:07.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.092+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2460076930 con 0x7f24780af810 2024-01-15T18:05:07.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.125+0000 7f2477e00640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f245006bb60 0x7f245006e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2468000f60 tx=0x7f24680015c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:07.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.464+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7f24780cead0 con 0x7f245006bb60 2024-01-15T18:05:07.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.892+0000 7f2474c00640 1 -- 172.21.15.2:0/4188269775 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f2468007110 con 0x7f245006bb60 2024-01-15T18:05:07.897 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:07.897 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:07.897 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:05:07.897 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:07.897 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:07.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.897+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f245006bb60 msgr2=0x7f245006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.897+0000 7f247f000640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f245006bb60 0x7f245006e020 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2468000f60 tx=0x7f24680015c0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.897+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 msgr2=0x7f24781383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:07.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.897+0000 7f247f000640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 0x7f24781383e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f2460000c00 tx=0x7f24600047d0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 shutdown_connections 2024-01-15T18:05:07.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f245006bb60 0x7f245006e020 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24780afc90 0x7f2478138920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24780af810 0x7f24781383e0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 --2- 172.21.15.2:0/4188269775 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2458004080 0x7f24781f9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:07.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 >> 172.21.15.2:0/4188269775 conn(0x7f24780b45f0 msgr2=0x7f247815c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:07.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 shutdown_connections 2024-01-15T18:05:07.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:07.898+0000 7f247f000640 1 -- 172.21.15.2:0/4188269775 wait complete. 2024-01-15T18:05:07.932 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:05:07.932 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27426 2024-01-15T18:05:08.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.269+0000 7f83e6a00640 1 Processor -- start 2024-01-15T18:05:08.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.270+0000 7f83e6a00640 1 -- start start 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.272+0000 7f83e6a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f83e00b2ba0 0x7f83e00b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.272+0000 7f83e6a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e00cfec0 con 0x7f83e0144b80 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.272+0000 7f83e6a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e015f410 con 0x7f83e015b860 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.272+0000 7f83e6a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e00b34f0 con 0x7f83e00b2ba0 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.273+0000 7f83dfe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f83e0144b80 0x7f83e015ed00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57072/0 (socket says 172.21.15.2:57072) 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.273+0000 7f83dfe00640 1 -- 172.21.15.2:0/765955432 learned_addr learned my addr 172.21.15.2:0/765955432 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:08.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.273+0000 7f83e4a00640 1 --2- 172.21.15.2:0/765955432 >> v2:172.21.15.2:3300/0 conn(0x7f83e00b2ba0 0x7f83e00b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.276+0000 7f83dea00640 1 -- 172.21.15.2:0/765955432 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376294917 0 0) 0x7f83e00cfec0 con 0x7f83e0144b80 2024-01-15T18:05:08.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.277+0000 7f83dea00640 1 -- 172.21.15.2:0/765955432 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c0003520 con 0x7f83e0144b80 2024-01-15T18:05:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.278+0000 7f83dea00640 1 -- 172.21.15.2:0/765955432 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1507845419 0 0) 0x7f83e015f410 con 0x7f83e015b860 2024-01-15T18:05:08.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.278+0000 7f83e4a00640 1 -- 172.21.15.2:0/765955432 >> v1:172.21.15.137:6789/0 conn(0x7f83e015b860 legacy=0x7f83e00b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.278+0000 7f83e4a00640 1 -- 172.21.15.2:0/765955432 >> v1:172.21.15.2:6789/0 conn(0x7f83e0144b80 legacy=0x7f83e015ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.278+0000 7f83e4a00640 1 -- 172.21.15.2:0/765955432 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e00b4620 con 0x7f83e00b2ba0 2024-01-15T18:05:08.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.281+0000 7f83e4a00640 1 --2- 172.21.15.2:0/765955432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e00b2ba0 0x7f83e00b2fb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009d90 tx=0x7f83d802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccc609c0883fb5ae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.282+0000 7f83dea00640 1 -- 172.21.15.2:0/765955432 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d802ccf0 con 0x7f83e00b2ba0 2024-01-15T18:05:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.282+0000 7f83dea00640 1 -- 172.21.15.2:0/765955432 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d802ce50 con 0x7f83e00b2ba0 2024-01-15T18:05:08.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.282+0000 7f83dea00640 1 -- 172.21.15.2:0/765955432 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8034640 con 0x7f83e00b2ba0 2024-01-15T18:05:08.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.283+0000 7f83e6a00640 1 -- 172.21.15.2:0/765955432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e00b2ba0 msgr2=0x7f83e00b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.283+0000 7f83e6a00640 1 --2- 172.21.15.2:0/765955432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e00b2ba0 0x7f83e00b2fb0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009d90 tx=0x7f83d802bf40 comp rx=0 tx=0).stop 2024-01-15T18:05:08.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.283+0000 7f83e6a00640 1 -- 172.21.15.2:0/765955432 shutdown_connections 2024-01-15T18:05:08.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.283+0000 7f83e6a00640 1 --2- 172.21.15.2:0/765955432 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e00b2ba0 0x7f83e00b2fb0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.283+0000 7f83e6a00640 1 -- 172.21.15.2:0/765955432 >> 172.21.15.2:0/765955432 conn(0x7f83e00b6430 msgr2=0x7f83e00b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.284+0000 7f83e6a00640 1 -- 172.21.15.2:0/765955432 shutdown_connections 2024-01-15T18:05:08.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.284+0000 7f83e6a00640 1 -- 172.21.15.2:0/765955432 wait complete. 2024-01-15T18:05:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.287+0000 7f83e6a00640 1 Processor -- start 2024-01-15T18:05:08.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.287+0000 7f83e6a00640 1 -- start start 2024-01-15T18:05:08.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.290+0000 7f83e6a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e00b2ba0 0x7f83e015e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.291+0000 7f83e6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 0x7f83e015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.292+0000 7f83e6a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e015b860 0x7f83e00b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.292+0000 7f83e6a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0155490 con 0x7f83e0144b80 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.292+0000 7f83e6a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e015d270 con 0x7f83e00b2ba0 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.292+0000 7f83e6a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e015d550 con 0x7f83e015b860 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.293+0000 7f83dfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e00b2ba0 0x7f83e015e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.293+0000 7f83df400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 0x7f83e015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.293+0000 7f83df400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 0x7f83e015c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52082/0 (socket says 172.21.15.2:52082) 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.293+0000 7f83df400640 1 -- 172.21.15.2:0/2588510765 learned_addr learned my addr 172.21.15.2:0/2588510765 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:08.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.293+0000 7f83e4a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e015b860 0x7f83e00b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.294+0000 7f83df400640 1 -- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e015b860 msgr2=0x7f83e00b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.294+0000 7f83df400640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e015b860 0x7f83e00b11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.294+0000 7f83df400640 1 -- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e00b2ba0 msgr2=0x7f83e015e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.294+0000 7f83df400640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e00b2ba0 0x7f83e015e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.294+0000 7f83df400640 1 -- 172.21.15.2:0/2588510765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8009a70 con 0x7f83e0144b80 2024-01-15T18:05:08.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.294+0000 7f83df400640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 0x7f83e015c8b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f83d000c450 tx=0x7f83d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.295+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0005480 con 0x7f83e0144b80 2024-01-15T18:05:08.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.295+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e015d830 con 0x7f83e0144b80 2024-01-15T18:05:08.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.295+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e015ddf0 con 0x7f83e0144b80 2024-01-15T18:05:08.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.296+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e015e000 con 0x7f83e0144b80 2024-01-15T18:05:08.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.298+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d0001160 con 0x7f83e0144b80 2024-01-15T18:05:08.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.298+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d00037f0 con 0x7f83e0144b80 2024-01-15T18:05:08.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.298+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f83d00039d0 con 0x7f83e0144b80 2024-01-15T18:05:08.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.299+0000 7f83d7e00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83b806bbe0 0x7f83b806e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:08.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.299+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f83d0087ba0 con 0x7f83e0144b80 2024-01-15T18:05:08.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.300+0000 7f83dfe00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83b806bbe0 0x7f83b806e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:08.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.302+0000 7f83dfe00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83b806bbe0 0x7f83b806e0a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f83c80011e0 tx=0x7f83c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:08.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.306+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83d0057dd0 con 0x7f83e0144b80 2024-01-15T18:05:08.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.700+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27426", "target": ["mon-mgr", ""]}) v1 -- 0x7f83e00b4810 con 0x7f83b806bbe0 2024-01-15T18:05:08.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.737+0000 7f83d7e00640 1 -- 172.21.15.2:0/2588510765 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f83e00b4810 con 0x7f83b806bbe0 2024-01-15T18:05:08.741 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:05:08.742 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:08.743 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:08.743 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83b806bbe0 msgr2=0x7f83b806e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83b806bbe0 0x7f83b806e0a0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f83c80011e0 tx=0x7f83c8009040 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 msgr2=0x7f83e015c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 0x7f83e015c8b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f83d000c450 tx=0x7f83d000c910 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 shutdown_connections 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83b806bbe0 0x7f83b806e0a0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e015b860 0x7f83e00b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e0144b80 0x7f83e015c8b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 --2- 172.21.15.2:0/2588510765 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e00b2ba0 0x7f83e015e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:08.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 >> 172.21.15.2:0/2588510765 conn(0x7f83e00b6430 msgr2=0x7f83e00b3e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:08.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 shutdown_connections 2024-01-15T18:05:08.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:08.740+0000 7f83e6a00640 1 -- 172.21.15.2:0/2588510765 wait complete. 2024-01-15T18:05:08.772 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:05:09.773 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27426 2024-01-15T18:05:10.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.001+0000 7fc7ffe00640 1 Processor -- start 2024-01-15T18:05:10.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.001+0000 7fc7ffe00640 1 -- start start 2024-01-15T18:05:10.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7ffe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc80014b160 0x7fc800161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7ffe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc800072100 0x7fc800070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8000b2060 0x7fc800070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7ffe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8000b35b0 con 0x7fc80014b160 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7ffe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8000b1510 con 0x7fc800072100 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001620d0 con 0x7fc8000b2060 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7fea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc80014b160 0x7fc800161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7fea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc80014b160 0x7fc800161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52090/0 (socket says 172.21.15.2:52090) 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.003+0000 7fc7fea00640 1 -- 172.21.15.2:0/1866042278 learned_addr learned my addr 172.21.15.2:0/1866042278 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7ff400640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8000b2060 0x7fc800070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fe000640 1 --2- 172.21.15.2:0/1866042278 >> v2:172.21.15.137:3300/0 conn(0x7fc800072100 0x7fc800070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fea00640 1 -- 172.21.15.2:0/1866042278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8000b2060 msgr2=0x7fc800070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fea00640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8000b2060 0x7fc800070650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fea00640 1 -- 172.21.15.2:0/1866042278 >> v2:172.21.15.137:3300/0 conn(0x7fc800072100 msgr2=0x7fc800070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fea00640 1 --2- 172.21.15.2:0/1866042278 >> v2:172.21.15.137:3300/0 conn(0x7fc800072100 0x7fc800070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fea00640 1 -- 172.21.15.2:0/1866042278 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000b32d0 con 0x7fc80014b160 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7ff400640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8000b2060 0x7fc800070650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:10.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.004+0000 7fc7fe000640 1 --2- 172.21.15.2:0/1866042278 >> v2:172.21.15.137:3300/0 conn(0x7fc800072100 0x7fc800070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.005+0000 7fc7fea00640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc80014b160 0x7fc800161b90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc80006e040 tx=0x7fc7f002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e856bcc18de0fd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.005+0000 7fc7fd600640 1 -- 172.21.15.2:0/1866042278 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0031870 con 0x7fc80014b160 2024-01-15T18:05:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.005+0000 7fc7fd600640 1 -- 172.21.15.2:0/1866042278 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f00319d0 con 0x7fc80014b160 2024-01-15T18:05:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.005+0000 7fc7fd600640 1 -- 172.21.15.2:0/1866042278 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0031d60 con 0x7fc80014b160 2024-01-15T18:05:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 -- 172.21.15.2:0/1866042278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc80014b160 msgr2=0x7fc800161b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc80014b160 0x7fc800161b90 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fc80006e040 tx=0x7fc7f002b910 comp rx=0 tx=0).stop 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 -- 172.21.15.2:0/1866042278 shutdown_connections 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc8000b2060 0x7fc800070650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/1866042278 >> v2:172.21.15.137:3300/0 conn(0x7fc800072100 0x7fc800070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/1866042278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc80014b160 0x7fc800161b90 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 -- 172.21.15.2:0/1866042278 >> 172.21.15.2:0/1866042278 conn(0x7fc8000b0840 msgr2=0x7fc8000b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 -- 172.21.15.2:0/1866042278 shutdown_connections 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.006+0000 7fc7ffe00640 1 -- 172.21.15.2:0/1866042278 wait complete. 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.008+0000 7fc7ffe00640 1 Processor -- start 2024-01-15T18:05:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.008+0000 7fc7ffe00640 1 -- start start 2024-01-15T18:05:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 0x7fc8000a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8000b2060 0x7fc8000ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7ffe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80014b160 0x7fc8000adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001488f0 con 0x7fc800072100 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000ae2e0 con 0x7fc8000b2060 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7ffe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8000a9390 con 0x7fc80014b160 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7fea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 0x7fc8000a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7fea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 0x7fc8000a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52106/0 (socket says 172.21.15.2:52106) 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.009+0000 7fc7fea00640 1 -- 172.21.15.2:0/2928445998 learned_addr learned my addr 172.21.15.2:0/2928445998 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7fea00640 1 -- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80014b160 msgr2=0x7fc8000adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7fea00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80014b160 0x7fc8000adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7fea00640 1 -- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8000b2060 msgr2=0x7fc8000ad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7fea00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8000b2060 0x7fc8000ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7fea00640 1 -- 172.21.15.2:0/2928445998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f00094d0 con 0x7fc800072100 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7fea00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 0x7fc8000a8e50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0000c00 tx=0x7fc7f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0031860 con 0x7fc800072100 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7f00443f0 con 0x7fc800072100 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f00447a0 con 0x7fc800072100 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.010+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8001542f0 con 0x7fc800072100 2024-01-15T18:05:10.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.011+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8001548b0 con 0x7fc800072100 2024-01-15T18:05:10.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.012+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc800074770 con 0x7fc800072100 2024-01-15T18:05:10.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.012+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fc7f0044900 con 0x7fc800072100 2024-01-15T18:05:10.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.013+0000 7fc7f6a00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7d806bb30 0x7fc7d806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.013+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fc7f002d040 con 0x7fc800072100 2024-01-15T18:05:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.014+0000 7fc7fe000640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7d806bb30 0x7fc7d806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.015+0000 7fc7fe000640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7d806bb30 0x7fc7d806dff0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8006b10 tx=0x7fc7e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.017+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7f0077680 con 0x7fc800072100 2024-01-15T18:05:10.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.267+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27426", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8001629a0 con 0x7fc7d806bb30 2024-01-15T18:05:10.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.367+0000 7fc7f6a00640 1 -- 172.21.15.2:0/2928445998 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fc8001629a0 con 0x7fc7d806bb30 2024-01-15T18:05:10.372 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:10.372 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:10.372 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:05:10.372 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:10.372 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.371+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7d806bb30 msgr2=0x7fc7d806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.371+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7d806bb30 0x7fc7d806dff0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8006b10 tx=0x7fc7e8008040 comp rx=0 tx=0).stop 2024-01-15T18:05:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.371+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 msgr2=0x7fc8000a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.371+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 0x7fc8000a8e50 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0000c00 tx=0x7fc7f00046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 shutdown_connections 2024-01-15T18:05:10.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7d806bb30 0x7fc7d806dff0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc80014b160 0x7fc8000adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc8000b2060 0x7fc8000ad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 --2- 172.21.15.2:0/2928445998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc800072100 0x7fc8000a8e50 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 >> 172.21.15.2:0/2928445998 conn(0x7fc8000b0840 msgr2=0x7fc800162290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 shutdown_connections 2024-01-15T18:05:10.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.372+0000 7fc7ffe00640 1 -- 172.21.15.2:0/2928445998 wait complete. 2024-01-15T18:05:10.410 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:05:10.410 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27427 2024-01-15T18:05:10.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.688+0000 7f3864600640 1 Processor -- start 2024-01-15T18:05:10.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.688+0000 7f3864600640 1 -- start start 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3864600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f385c0adb10 0x7f385c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3864600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 0x7f385c070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3864600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385c06cd00 con 0x7f385c0721f0 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3864600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385c0ae430 con 0x7f385c0adb10 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3864600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c144150 con 0x7f385c1448e0 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3862600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f385c0721f0 0x7f385c06d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57076/0 (socket says 172.21.15.2:57076) 2024-01-15T18:05:10.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.691+0000 7f3862600640 1 -- 172.21.15.2:0/1578974251 learned_addr learned my addr 172.21.15.2:0/1578974251 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.693+0000 7f3861c00640 1 --2- 172.21.15.2:0/1578974251 >> v2:172.21.15.137:3300/0 conn(0x7f385c0adb10 0x7f385c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.693+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553459407 0 0) 0x7f385c06cd00 con 0x7f385c0721f0 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.693+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f383c003520 con 0x7f385c0721f0 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.693+0000 7f3861200640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 0x7f385c070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2101773748 0 0) 0x7f383c003520 con 0x7f385c0721f0 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 msgr2=0x7f385c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 0x7f385c070230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 >> v2:172.21.15.137:3300/0 conn(0x7f385c0adb10 msgr2=0x7f385c0adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 --2- 172.21.15.2:0/1578974251 >> v2:172.21.15.137:3300/0 conn(0x7f385c0adb10 0x7f385c0adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c0abed0 con 0x7f385c0721f0 2024-01-15T18:05:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f38540019f0 con 0x7f385c0721f0 2024-01-15T18:05:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 >> v1:172.21.15.2:6789/0 conn(0x7f385c0721f0 legacy=0x7f385c06d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f383c003970 0x7f383c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f383c0047c0 0x7f383c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 0x7f383c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c0ae430 con 0x7f383c0047c0 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c144150 con 0x7f383c003970 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.694+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f383c00bab0 con 0x7f385c1448e0 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.696+0000 7f3861200640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f383c0047c0 0x7f383c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.697+0000 7f3861c00640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f383c003970 0x7f383c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.699+0000 7f3861200640 1 -- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 msgr2=0x7f383c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.699+0000 7f3861200640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f385c1448e0 0x7f383c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.699+0000 7f3861200640 1 -- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f383c003970 msgr2=0x7f383c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.699+0000 7f3861200640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f383c003970 0x7f383c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.699+0000 7f3861200640 1 -- 172.21.15.2:0/1578974251 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c003520 con 0x7f383c0047c0 2024-01-15T18:05:10.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.701+0000 7f3861200640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f383c0047c0 0x7f383c004c40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f384400ba00 tx=0x7f383c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64adba2d007e8786 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.702+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844032620 con 0x7f383c0047c0 2024-01-15T18:05:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.702+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3844032780 con 0x7f383c0047c0 2024-01-15T18:05:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.702+0000 7f385be00640 1 -- 172.21.15.2:0/1578974251 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3844032b50 con 0x7f383c0047c0 2024-01-15T18:05:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.703+0000 7f3864600640 1 -- 172.21.15.2:0/1578974251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f383c0047c0 msgr2=0x7f383c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.703+0000 7f3864600640 1 --2- 172.21.15.2:0/1578974251 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f383c0047c0 0x7f383c004c40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f384400ba00 tx=0x7f383c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.703+0000 7f3862600640 1 -- 172.21.15.2:0/1578974251 reap_dead start 2024-01-15T18:05:10.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.706+0000 7f3864600640 1 -- 172.21.15.2:0/1578974251 shutdown_connections 2024-01-15T18:05:10.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.706+0000 7f3864600640 1 -- 172.21.15.2:0/1578974251 >> 172.21.15.2:0/1578974251 conn(0x7f385c0ced60 msgr2=0x7f385c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:10.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.707+0000 7f3864600640 1 -- 172.21.15.2:0/1578974251 shutdown_connections 2024-01-15T18:05:10.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.707+0000 7f3864600640 1 -- 172.21.15.2:0/1578974251 wait complete. 2024-01-15T18:05:10.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.715+0000 7f3864600640 1 Processor -- start 2024-01-15T18:05:10.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.715+0000 7f3864600640 1 -- start start 2024-01-15T18:05:10.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.717+0000 7f3864600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 0x7f385c15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.717+0000 7f3864600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f385c15a7b0 0x7f385c099a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.717+0000 7f3864600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f385c15d570 0x7f385c0aef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.717+0000 7f3864600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c15bdd0 con 0x7f385c15d570 2024-01-15T18:05:10.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.717+0000 7f3864600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c09a2e0 con 0x7f385c15a7b0 2024-01-15T18:05:10.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.717+0000 7f3864600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385c09a5c0 con 0x7f383c0047c0 2024-01-15T18:05:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.718+0000 7f3861c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 0x7f385c15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.718+0000 7f3861c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 0x7f385c15b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:41506/0 (socket says 172.21.15.2:41506) 2024-01-15T18:05:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.718+0000 7f3861c00640 1 -- 172.21.15.2:0/281071966 learned_addr learned my addr 172.21.15.2:0/281071966 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.718+0000 7f3861200640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f385c15a7b0 0x7f385c099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.718+0000 7f3862600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f385c15d570 0x7f385c0aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.720+0000 7f3862600640 1 -- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 msgr2=0x7f385c15b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.720+0000 7f3862600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 0x7f385c15b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.720+0000 7f3862600640 1 -- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f385c15a7b0 msgr2=0x7f385c099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:10.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.721+0000 7f3862600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f385c15a7b0 0x7f385c099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:10.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.722+0000 7f3862600640 1 -- 172.21.15.2:0/281071966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c09a8a0 con 0x7f385c15d570 2024-01-15T18:05:10.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.722+0000 7f3861200640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f385c15a7b0 0x7f385c099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:10.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.722+0000 7f3861c00640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 0x7f385c15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:10.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.723+0000 7f3862600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f385c15d570 0x7f385c0aef40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f383c00bc40 tx=0x7f385400ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.724+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400a610 con 0x7f385c15d570 2024-01-15T18:05:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.724+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385400a7b0 con 0x7f385c15d570 2024-01-15T18:05:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.724+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385400ab60 con 0x7f385c15d570 2024-01-15T18:05:10.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.724+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c13f390 con 0x7f385c15d570 2024-01-15T18:05:10.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.725+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c099f40 con 0x7f385c15d570 2024-01-15T18:05:10.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.726+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385c13f8b0 con 0x7f385c15d570 2024-01-15T18:05:10.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.728+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f385400acc0 con 0x7f385c15d570 2024-01-15T18:05:10.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.732+0000 7f3859600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383406bb80 0x7f383406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:10.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.732+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f38540881f0 con 0x7f385c15d570 2024-01-15T18:05:10.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.735+0000 7f3861c00640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383406bb80 0x7f383406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:10.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.737+0000 7f3861c00640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383406bb80 0x7f383406e040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f384c00bbb0 tx=0x7f384c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:10.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:10.739+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f385c13f8b0 con 0x7f385c15d570 2024-01-15T18:05:11.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.138+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27427", "target": ["mon-mgr", ""]}) v1 -- 0x7f385c144390 con 0x7f383406bb80 2024-01-15T18:05:11.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.179+0000 7f3859600640 1 -- 172.21.15.2:0/281071966 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f384c00b7a0 con 0x7f383406bb80 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:05:11.185 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:05:11.186 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:05:11.186 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:05:11.186 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:05:11.186 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:11.186 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:11.186 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:11.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383406bb80 msgr2=0x7f383406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383406bb80 0x7f383406e040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f384c00bbb0 tx=0x7f384c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:11.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f385c15d570 msgr2=0x7f385c0aef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:11.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f385c15d570 0x7f385c0aef40 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f383c00bc40 tx=0x7f385400ccd0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 shutdown_connections 2024-01-15T18:05:11.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383406bb80 0x7f383406e040 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f385c15d570 0x7f385c0aef40 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f385c15a7b0 0x7f385c099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 --2- 172.21.15.2:0/281071966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f383c0047c0 0x7f385c15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:11.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.184+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 >> 172.21.15.2:0/281071966 conn(0x7f385c0ced60 msgr2=0x7f385c0b0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:11.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.186+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 shutdown_connections 2024-01-15T18:05:11.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:11.186+0000 7f3864600640 1 -- 172.21.15.2:0/281071966 wait complete. 2024-01-15T18:05:11.230 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-15T18:05:12.231 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone27427 2024-01-15T18:05:12.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.452+0000 7fe317200640 1 Processor -- start 2024-01-15T18:05:12.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.452+0000 7fe317200640 1 -- start start 2024-01-15T18:05:12.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.454+0000 7fe317200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe31806dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.454+0000 7fe317200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe318070880 con 0x7fe318070110 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.454+0000 7fe317200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3180b2b50 con 0x7fe318161210 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.454+0000 7fe317200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe318159d10 con 0x7fe318070b20 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.455+0000 7fe315400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe318070110 0x7fe318070f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57090/0 (socket says 172.21.15.2:57090) 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.455+0000 7fe315400640 1 -- 172.21.15.2:0/3838371892 learned_addr learned my addr 172.21.15.2:0/3838371892 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.455+0000 7fe315e00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe31806dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.455+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662801909 0 0) 0x7fe318070880 con 0x7fe318070110 2024-01-15T18:05:12.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.455+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f8003440 con 0x7fe318070110 2024-01-15T18:05:12.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 998304135 0 0) 0x7fe2f8003440 con 0x7fe318070110 2024-01-15T18:05:12.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 msgr2=0x7fe31806dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe31806dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 >> v1:172.21.15.137:6789/0 conn(0x7fe318161210 legacy=0x7fe318163600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318159ff0 con 0x7fe318070110 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe300001670 con 0x7fe318070110 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 >> v1:172.21.15.2:6789/0 conn(0x7fe318070110 legacy=0x7fe318070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f8003a90 0x7fe2f8003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe2f800acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe2f800d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318159d10 con 0x7fe2f8008850 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f800dc70 con 0x7fe2f8003a90 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f800de30 con 0x7fe318070b20 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.456+0000 7fe315400640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe2f800acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315400640 1 -- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 msgr2=0x7fe2f800d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315e00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f8003a90 0x7fe2f8003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe316800640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe2f800d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315400640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe2f800d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315400640 1 -- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f8003a90 msgr2=0x7fe2f8003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:12.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315400640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f8003a90 0x7fe2f8003eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315400640 1 -- 172.21.15.2:0/3838371892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f8003440 con 0x7fe2f8008850 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe316800640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318070b20 0x7fe2f800d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.457+0000 7fe315e00640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe2f8003a90 0x7fe2f8003eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.458+0000 7fe315400640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe2f800acc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c0e0 tx=0x7fe2f80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a17c00b4223702dd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.458+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300005ad0 con 0x7fe2f8008850 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.458+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe300005c30 con 0x7fe2f8008850 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.458+0000 7fe314a00640 1 -- 172.21.15.2:0/3838371892 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30002c7e0 con 0x7fe2f8008850 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe317200640 1 -- 172.21.15.2:0/3838371892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 msgr2=0x7fe2f800acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe317200640 1 --2- 172.21.15.2:0/3838371892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe2f800acc0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c0e0 tx=0x7fe2f80036f0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe316800640 1 -- 172.21.15.2:0/3838371892 reap_dead start 2024-01-15T18:05:12.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe317200640 1 -- 172.21.15.2:0/3838371892 shutdown_connections 2024-01-15T18:05:12.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe317200640 1 -- 172.21.15.2:0/3838371892 >> 172.21.15.2:0/3838371892 conn(0x7fe3180b0840 msgr2=0x7fe3180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:12.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe317200640 1 -- 172.21.15.2:0/3838371892 shutdown_connections 2024-01-15T18:05:12.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.459+0000 7fe317200640 1 -- 172.21.15.2:0/3838371892 wait complete. 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.461+0000 7fe317200640 1 Processor -- start 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.461+0000 7fe317200640 1 -- start start 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.462+0000 7fe317200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe3181542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.462+0000 7fe317200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3180cd7f0 0x7fe318150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.462+0000 7fe317200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318152020 0x7fe3181524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.462+0000 7fe317200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318152a10 con 0x7fe2f8008850 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.462+0000 7fe317200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe318154930 con 0x7fe3180cd7f0 2024-01-15T18:05:12.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.462+0000 7fe317200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe318154c10 con 0x7fe318152020 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe3181542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe3181542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52142/0 (socket says 172.21.15.2:52142) 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 -- 172.21.15.2:0/4015326331 learned_addr learned my addr 172.21.15.2:0/4015326331 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe316800640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318152020 0x7fe3181524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 -- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318152020 msgr2=0x7fe3181524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318152020 0x7fe3181524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 -- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3180cd7f0 msgr2=0x7fe318150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3180cd7f0 0x7fe318150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:12.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 -- 172.21.15.2:0/4015326331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318154ef0 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe316800640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318152020 0x7fe3181524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.463+0000 7fe315e00640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe3181542c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe3080108b0 tx=0x7fe308010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.464+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30800eb60 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.464+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30800ecc0 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.464+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30801c400 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.464+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3181551d0 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.464+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3180cdc70 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.465+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3180cde30 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.466+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fe30801c560 con 0x7fe2f8008850 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.467+0000 7fe30e000640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f006bb60 0x7fe2f006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:12.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.468+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe308088bb0 con 0x7fe2f8008850 2024-01-15T18:05:12.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.468+0000 7fe315400640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f006bb60 0x7fe2f006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:12.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.469+0000 7fe315400640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f006bb60 0x7fe2f006e020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe300034000 tx=0x7fe300007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:12.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.472+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3080553b0 con 0x7fe2f8008850 2024-01-15T18:05:12.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:12.704+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone27427", "target": ["mon-mgr", ""]}) v1 -- 0x7fe318070590 con 0x7fe2f006bb60 2024-01-15T18:05:13.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.171+0000 7fe30e000640 1 -- 172.21.15.2:0/4015326331 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fe3000013b0 con 0x7fe2f006bb60 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:13.176 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.174+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f006bb60 msgr2=0x7fe2f006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.174+0000 7fe317200640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f006bb60 0x7fe2f006e020 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe300034000 tx=0x7fe300007000 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.174+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 msgr2=0x7fe3181542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.174+0000 7fe317200640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe3181542c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fe3080108b0 tx=0x7fe308010d70 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 shutdown_connections 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f006bb60 0x7fe2f006e020 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe318152020 0x7fe3181524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3180cd7f0 0x7fe318150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 --2- 172.21.15.2:0/4015326331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe2f8008850 0x7fe3181542c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 >> 172.21.15.2:0/4015326331 conn(0x7fe3180b0840 msgr2=0x7fe318158c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 shutdown_connections 2024-01-15T18:05:13.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.175+0000 7fe317200640 1 -- 172.21.15.2:0/4015326331 wait complete. 2024-01-15T18:05:13.201 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:05:13.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.418+0000 7f6d1b800640 1 Processor -- start 2024-01-15T18:05:13.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.418+0000 7f6d1b800640 1 -- start start 2024-01-15T18:05:13.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d1b800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d14144b80 0x7f6d140cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d1b800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6d140b30a0 0x7f6d1406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d1b800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d1406d220 con 0x7f6d1406cdf0 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d1b800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d140b4220 con 0x7f6d140b30a0 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d1b800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d14144f60 con 0x7f6d14144b80 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d13e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6d140b30a0 0x7f6d1406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d19800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6d1406cdf0 0x7f6d140b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51550/0 (socket says 172.21.15.2:51550) 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d19800640 1 -- 172.21.15.2:0/3453610930 learned_addr learned my addr 172.21.15.2:0/3453610930 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d18e00640 1 --2- 172.21.15.2:0/3453610930 >> v2:172.21.15.137:3300/0 conn(0x7f6d14144b80 0x7f6d140cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d13e00640 1 -- 172.21.15.2:0/3453610930 >> v2:172.21.15.137:3300/0 conn(0x7f6d14144b80 msgr2=0x7f6d140cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d13e00640 1 --2- 172.21.15.2:0/3453610930 >> v2:172.21.15.137:3300/0 conn(0x7f6d14144b80 0x7f6d140cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.420+0000 7f6d13e00640 1 -- 172.21.15.2:0/3453610930 >> v1:172.21.15.2:6789/0 conn(0x7f6d1406cdf0 legacy=0x7f6d140b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.421+0000 7f6d13e00640 1 -- 172.21.15.2:0/3453610930 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d140b35a0 con 0x7f6d140b30a0 2024-01-15T18:05:13.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.421+0000 7f6d13400640 1 -- 172.21.15.2:0/3453610930 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122566702 0 0) 0x7f6d1406d220 con 0x7f6d1406cdf0 2024-01-15T18:05:13.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.422+0000 7f6d13e00640 1 --2- 172.21.15.2:0/3453610930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d140b30a0 0x7f6d1406c8b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6d14074060 tx=0x7f6cfc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9363023e44c7044b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.422+0000 7f6d13400640 1 -- 172.21.15.2:0/3453610930 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc02e070 con 0x7f6d140b30a0 2024-01-15T18:05:13.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.422+0000 7f6d13400640 1 -- 172.21.15.2:0/3453610930 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cfc0315a0 con 0x7f6d140b30a0 2024-01-15T18:05:13.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.422+0000 7f6d13400640 1 -- 172.21.15.2:0/3453610930 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cfc031930 con 0x7f6d140b30a0 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 -- 172.21.15.2:0/3453610930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d140b30a0 msgr2=0x7f6d1406c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3453610930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d140b30a0 0x7f6d1406c8b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f6d14074060 tx=0x7f6cfc02b840 comp rx=0 tx=0).stop 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 -- 172.21.15.2:0/3453610930 shutdown_connections 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3453610930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d140b30a0 0x7f6d1406c8b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3453610930 >> v2:172.21.15.137:3300/0 conn(0x7f6d14144b80 0x7f6d140cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 -- 172.21.15.2:0/3453610930 >> 172.21.15.2:0/3453610930 conn(0x7f6d140b46d0 msgr2=0x7f6d140b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 -- 172.21.15.2:0/3453610930 shutdown_connections 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.423+0000 7f6d1b800640 1 -- 172.21.15.2:0/3453610930 wait complete. 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.425+0000 7f6d1b800640 1 Processor -- start 2024-01-15T18:05:13.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.425+0000 7f6d1b800640 1 -- start start 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d1b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1406cdf0 0x7f6d141f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d1b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140b30a0 0x7f6d14140190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d1b800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 0x7f6d141f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d1b800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1415cfa0 con 0x7f6d14144b80 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d1b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1415d280 con 0x7f6d140b30a0 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d1b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1415d560 con 0x7f6d1406cdf0 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d19800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 0x7f6d141f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d19800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 0x7f6d141f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53100/0 (socket says 172.21.15.2:53100) 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d19800640 1 -- 172.21.15.2:0/3074497534 learned_addr learned my addr 172.21.15.2:0/3074497534 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:13.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d18e00640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1406cdf0 0x7f6d141f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.426+0000 7f6d13e00640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140b30a0 0x7f6d14140190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d19800640 1 -- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1406cdf0 msgr2=0x7f6d141f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d19800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1406cdf0 0x7f6d141f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d19800640 1 -- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140b30a0 msgr2=0x7f6d14140190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d19800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140b30a0 0x7f6d14140190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d19800640 1 -- 172.21.15.2:0/3074497534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cfc009400 con 0x7f6d14144b80 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d18e00640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1406cdf0 0x7f6d141f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d19800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 0x7f6d141f3bd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c480 tx=0x7f6d0c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c005ce0 con 0x7f6d14144b80 2024-01-15T18:05:13.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0c005e60 con 0x7f6d14144b80 2024-01-15T18:05:13.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0c0072b0 con 0x7f6d14144b80 2024-01-15T18:05:13.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1415d840 con 0x7f6d14144b80 2024-01-15T18:05:13.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.427+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d141f3970 con 0x7f6d14144b80 2024-01-15T18:05:13.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.429+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1415dd60 con 0x7f6d14144b80 2024-01-15T18:05:13.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.430+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6d0c007410 con 0x7f6d14144b80 2024-01-15T18:05:13.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.431+0000 7f6d10c00640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6cec06bb80 0x7f6cec06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.431+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6d0c087c50 con 0x7f6d14144b80 2024-01-15T18:05:13.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.432+0000 7f6d18e00640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6cec06bb80 0x7f6cec06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.433+0000 7f6d18e00640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6cec06bb80 0x7f6cec06e040 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009f50 tx=0x7f6d0400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.436+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d0c054340 con 0x7f6d14144b80 2024-01-15T18:05:13.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.656+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d140b3e40 con 0x7f6cec06bb80 2024-01-15T18:05:13.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.660+0000 7f6d10c00640 1 -- 172.21.15.2:0/3074497534 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6d040096f0 con 0x7f6cec06bb80 2024-01-15T18:05:13.664 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:05:13.664 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:13.665 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:05:13.665 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:13.665 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6cec06bb80 msgr2=0x7f6cec06e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6cec06bb80 0x7f6cec06e040 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009f50 tx=0x7f6d0400b040 comp rx=0 tx=0).stop 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 msgr2=0x7f6d141f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 0x7f6d141f3bd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c00c480 tx=0x7f6d0c00c940 comp rx=0 tx=0).stop 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 shutdown_connections 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6cec06bb80 0x7f6cec06e040 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d14144b80 0x7f6d141f3bd0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d140b30a0 0x7f6d14140190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 --2- 172.21.15.2:0/3074497534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d1406cdf0 0x7f6d141f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 >> 172.21.15.2:0/3074497534 conn(0x7f6d140b46d0 msgr2=0x7f6d14143d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 shutdown_connections 2024-01-15T18:05:13.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.663+0000 7f6d1b800640 1 -- 172.21.15.2:0/3074497534 wait complete. 2024-01-15T18:05:13.689 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:05:13.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.905+0000 7fa779800640 1 Processor -- start 2024-01-15T18:05:13.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.906+0000 7fa779800640 1 -- start start 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.907+0000 7fa779800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 0x7fa7740716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.907+0000 7fa779800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa7740cc3b0 0x7fa7740cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.907+0000 7fa779800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7740b2fa0 con 0x7fa774143620 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.907+0000 7fa779800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa774071bf0 con 0x7fa7740cc3b0 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.907+0000 7fa779800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740b3280 con 0x7fa774071230 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.908+0000 7fa76be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 0x7fa7740716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.908+0000 7fa76be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 0x7fa7740716b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:41504/0 (socket says 172.21.15.2:41504) 2024-01-15T18:05:13.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.908+0000 7fa76be00640 1 -- 172.21.15.2:0/3026024288 learned_addr learned my addr 172.21.15.2:0/3026024288 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:13.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.908+0000 7fa778e00640 1 --2- 172.21.15.2:0/3026024288 >> v2:172.21.15.137:3300/0 conn(0x7fa7740cc3b0 0x7fa7740cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.909+0000 7fa76be00640 1 -- 172.21.15.2:0/3026024288 >> v2:172.21.15.137:3300/0 conn(0x7fa7740cc3b0 msgr2=0x7fa7740cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:13.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.909+0000 7fa76be00640 1 --2- 172.21.15.2:0/3026024288 >> v2:172.21.15.137:3300/0 conn(0x7fa7740cc3b0 0x7fa7740cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.909+0000 7fa76be00640 1 -- 172.21.15.2:0/3026024288 >> v1:172.21.15.137:6789/0 conn(0x7fa774143620 legacy=0x7fa7740b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.909+0000 7fa76be00640 1 -- 172.21.15.2:0/3026024288 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7740b3560 con 0x7fa774071230 2024-01-15T18:05:13.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.909+0000 7fa778e00640 1 --2- 172.21.15.2:0/3026024288 >> v2:172.21.15.137:3300/0 conn(0x7fa7740cc3b0 0x7fa7740cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.911+0000 7fa76be00640 1 --2- 172.21.15.2:0/3026024288 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 0x7fa7740716b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00bb10 tx=0x7fa75c02e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50f467ec77d62e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.911+0000 7fa773400640 1 -- 172.21.15.2:0/3026024288 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c036440 con 0x7fa774071230 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.911+0000 7fa773400640 1 -- 172.21.15.2:0/3026024288 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa75c0365a0 con 0x7fa774071230 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.911+0000 7fa773400640 1 -- 172.21.15.2:0/3026024288 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa75c036930 con 0x7fa774071230 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 -- 172.21.15.2:0/3026024288 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 msgr2=0x7fa7740716b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 --2- 172.21.15.2:0/3026024288 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 0x7fa7740716b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa75c00bb10 tx=0x7fa75c02e150 comp rx=0 tx=0).stop 2024-01-15T18:05:13.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 -- 172.21.15.2:0/3026024288 shutdown_connections 2024-01-15T18:05:13.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 --2- 172.21.15.2:0/3026024288 >> v2:172.21.15.137:3300/0 conn(0x7fa7740cc3b0 0x7fa7740cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 --2- 172.21.15.2:0/3026024288 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa774071230 0x7fa7740716b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 -- 172.21.15.2:0/3026024288 >> 172.21.15.2:0/3026024288 conn(0x7fa7740b0840 msgr2=0x7fa7740b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:13.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 -- 172.21.15.2:0/3026024288 shutdown_connections 2024-01-15T18:05:13.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.912+0000 7fa779800640 1 -- 172.21.15.2:0/3026024288 wait complete. 2024-01-15T18:05:13.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.914+0000 7fa779800640 1 Processor -- start 2024-01-15T18:05:13.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.914+0000 7fa779800640 1 -- start start 2024-01-15T18:05:13.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.915+0000 7fa779800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa774071230 0x7fa7741f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.915+0000 7fa779800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7740cc3b0 0x7fa7740cea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.915+0000 7fa779800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 0x7fa774141080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.915+0000 7fa779800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa774155a20 con 0x7fa774143620 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.915+0000 7fa779800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740ce590 con 0x7fa774071230 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.915+0000 7fa779800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7740cd9c0 con 0x7fa7740cc3b0 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 0x7fa774141080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 0x7fa774141080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53112/0 (socket says 172.21.15.2:53112) 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 -- 172.21.15.2:0/3945829522 learned_addr learned my addr 172.21.15.2:0/3945829522 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:13.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa76be00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7740cc3b0 0x7fa7740cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa773e00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa774071230 0x7fa7741f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 -- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7740cc3b0 msgr2=0x7fa7740cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7740cc3b0 0x7fa7740cea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 -- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa774071230 msgr2=0x7fa7741f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa774071230 0x7fa7741f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa778e00640 1 -- 172.21.15.2:0/3945829522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa76c009a70 con 0x7fa774143620 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.916+0000 7fa76be00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7740cc3b0 0x7fa7740cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.917+0000 7fa778e00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 0x7fa774141080 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00efc0 tx=0x7fa76c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.917+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c009d70 con 0x7fa774143620 2024-01-15T18:05:13.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.917+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa76c00f040 con 0x7fa774143620 2024-01-15T18:05:13.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.917+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa76c003990 con 0x7fa774143620 2024-01-15T18:05:13.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.917+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c00b770 con 0x7fa774143620 2024-01-15T18:05:13.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.917+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7740cdf40 con 0x7fa774143620 2024-01-15T18:05:13.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.918+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7740ce700 con 0x7fa774143620 2024-01-15T18:05:13.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.919+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa76c0144a0 con 0x7fa774143620 2024-01-15T18:05:13.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.920+0000 7fa770c00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa74c06bb60 0x7fa74c06e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:13.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.920+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fa76c089200 con 0x7fa774143620 2024-01-15T18:05:13.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.921+0000 7fa773e00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa74c06bb60 0x7fa74c06e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:13.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.922+0000 7fa773e00640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa74c06bb60 0x7fa74c06e020 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa764007fd0 tx=0x7fa764009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:13.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:13.924+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa76c055950 con 0x7fa774143620 2024-01-15T18:05:14.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.144+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa774143a00 con 0x7fa74c06bb60 2024-01-15T18:05:14.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.164+0000 7fa770c00640 1 -- 172.21.15.2:0/3945829522 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7fa7640050d0 con 0x7fa74c06bb60 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.169 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27424" 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27425" 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27427" 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.170 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27423" 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27421" 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27426" 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:14.171 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone27422" 2024-01-15T18:05:14.172 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:14.172 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:05:14.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.166+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa74c06bb60 msgr2=0x7fa74c06e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.166+0000 7fa779800640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa74c06bb60 0x7fa74c06e020 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa764007fd0 tx=0x7fa764009040 comp rx=0 tx=0).stop 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 msgr2=0x7fa774141080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 0x7fa774141080 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00efc0 tx=0x7fa76c009f90 comp rx=0 tx=0).stop 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 shutdown_connections 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa74c06bb60 0x7fa74c06e020 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa774143620 0x7fa774141080 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7740cc3b0 0x7fa7740cea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 --2- 172.21.15.2:0/3945829522 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa774071230 0x7fa7741f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 >> 172.21.15.2:0/3945829522 conn(0x7fa7740b0840 msgr2=0x7fa7740b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:14.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 shutdown_connections 2024-01-15T18:05:14.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.167+0000 7fa779800640 1 -- 172.21.15.2:0/3945829522 wait complete. 2024-01-15T18:05:14.192 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:05:14.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.419+0000 7f4647800640 1 Processor -- start 2024-01-15T18:05:14.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.419+0000 7f4647800640 1 -- start start 2024-01-15T18:05:14.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.421+0000 7f4647800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 0x7f46400cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.421+0000 7f4647800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46400ce230 con 0x7f46400cdd70 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.421+0000 7f4647800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46400ce510 con 0x7f464006d000 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.421+0000 7f4647800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46400ce7f0 con 0x7f4640144c00 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f4637e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f46400cdd70 0x7f464006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51554/0 (socket says 172.21.15.2:51554) 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f4637e00640 1 -- 172.21.15.2:0/1350700539 learned_addr learned my addr 172.21.15.2:0/1350700539 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f4644e00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 0x7f46400cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2980980697 0 0) 0x7f46400ce230 con 0x7f46400cdd70 2024-01-15T18:05:14.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4620003520 con 0x7f46400cdd70 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1725357166 0 0) 0x7f4620003520 con 0x7f46400cdd70 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 msgr2=0x7f46400cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f463fe00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 0x7f46400cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.422+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 >> v1:172.21.15.137:6789/0 conn(0x7f464006d000 legacy=0x7f4640070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46400705f0 con 0x7f46400cdd70 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4628001670 con 0x7f46400cdd70 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 >> v1:172.21.15.2:6789/0 conn(0x7f46400cdd70 legacy=0x7f464006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46200039a0 0x7f4620003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 0x7f46200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4620004d00 0x7f4620005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400ce7f0 con 0x7f46200039a0 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462000ba30 con 0x7f4620004d00 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462000bbf0 con 0x7f4640144c00 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f4644e00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46200039a0 0x7f4620003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f4645800640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4620004d00 0x7f4620005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f4637e00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 0x7f46200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.423+0000 7f4644e00640 1 -- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 msgr2=0x7f46200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.424+0000 7f4644e00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4640144c00 0x7f46200047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.424+0000 7f4644e00640 1 -- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4620004d00 msgr2=0x7f4620005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:14.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.424+0000 7f4644e00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4620004d00 0x7f4620005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.424+0000 7f4644e00640 1 -- 172.21.15.2:0/1350700539 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4620003520 con 0x7f46200039a0 2024-01-15T18:05:14.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.424+0000 7f4645800640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4620004d00 0x7f4620005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:14.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.425+0000 7f4644e00640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46200039a0 0x7f4620003dc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4630009a90 tx=0x7f46200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6153c1c1ff3d50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:14.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.425+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463002c6b0 con 0x7f46200039a0 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.425+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463002c810 con 0x7f46200039a0 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.425+0000 7f463fe00640 1 -- 172.21.15.2:0/1350700539 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463002cba0 con 0x7f46200039a0 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4647800640 1 -- 172.21.15.2:0/1350700539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46200039a0 msgr2=0x7f4620003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4647800640 1 --2- 172.21.15.2:0/1350700539 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46200039a0 0x7f4620003dc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4630009a90 tx=0x7f46200057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4645800640 1 -- 172.21.15.2:0/1350700539 reap_dead start 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4647800640 1 -- 172.21.15.2:0/1350700539 shutdown_connections 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4647800640 1 -- 172.21.15.2:0/1350700539 >> 172.21.15.2:0/1350700539 conn(0x7f46400b45f0 msgr2=0x7f46400b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4647800640 1 -- 172.21.15.2:0/1350700539 shutdown_connections 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.426+0000 7f4647800640 1 -- 172.21.15.2:0/1350700539 wait complete. 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.428+0000 7f4647800640 1 Processor -- start 2024-01-15T18:05:14.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.428+0000 7f4647800640 1 -- start start 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4647800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46200039a0 0x7f4640155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4647800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46400aeb20 0x7f46400b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4647800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 0x7f464013bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4647800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400af350 con 0x7f46400b0930 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4647800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46400af630 con 0x7f46200039a0 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4647800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46400af910 con 0x7f46400aeb20 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4645800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 0x7f464013bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4645800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 0x7f464013bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53128/0 (socket says 172.21.15.2:53128) 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.429+0000 7f4645800640 1 -- 172.21.15.2:0/3111466292 learned_addr learned my addr 172.21.15.2:0/3111466292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:14.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4644e00640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46200039a0 0x7f4640155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4637e00640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46400aeb20 0x7f46400b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4645800640 1 -- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46400aeb20 msgr2=0x7f46400b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4645800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46400aeb20 0x7f46400b03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4645800640 1 -- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46200039a0 msgr2=0x7f4640155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4645800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46200039a0 0x7f4640155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4645800640 1 -- 172.21.15.2:0/3111466292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46400afbf0 con 0x7f46400b0930 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4644e00640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46200039a0 0x7f4640155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4645800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 0x7f464013bf30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f46400b1c90 tx=0x7f463800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:14.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638006d50 con 0x7f46400b0930 2024-01-15T18:05:14.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638003970 con 0x7f46400b0930 2024-01-15T18:05:14.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638003d60 con 0x7f46400b0930 2024-01-15T18:05:14.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.430+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46400afed0 con 0x7f46400b0930 2024-01-15T18:05:14.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.431+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464013c470 con 0x7f46400b0930 2024-01-15T18:05:14.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.432+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46400b00c0 con 0x7f46400b0930 2024-01-15T18:05:14.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.433+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4638005070 con 0x7f46400b0930 2024-01-15T18:05:14.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.433+0000 7f463d600640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f461806bb30 0x7f461806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:14.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.434+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f46380885c0 con 0x7f46400b0930 2024-01-15T18:05:14.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.434+0000 7f4644e00640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f461806bb30 0x7f461806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:14.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.435+0000 7f4644e00640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f461806bb30 0x7f461806dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4630036000 tx=0x7f4630030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:14.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.437+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4638054c20 con 0x7f46400b0930 2024-01-15T18:05:14.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:14.666+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46400af7a0 con 0x7f461806bb30 2024-01-15T18:05:18.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.171+0000 7f463d600640 1 -- 172.21.15.2:0/3111466292 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f46400af7a0 con 0x7f461806bb30 2024-01-15T18:05:18.176 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:18.176 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:05:18.176 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:18.176 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:05:18.176 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:18.176 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:05:18.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.174+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f461806bb30 msgr2=0x7f461806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.174+0000 7f4647800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f461806bb30 0x7f461806dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4630036000 tx=0x7f4630030000 comp rx=0 tx=0).stop 2024-01-15T18:05:18.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.174+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 msgr2=0x7f464013bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.174+0000 7f4647800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 0x7f464013bf30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f46400b1c90 tx=0x7f463800dc00 comp rx=0 tx=0).stop 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 shutdown_connections 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f461806bb30 0x7f461806dff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f46400b0930 0x7f464013bf30 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f46400aeb20 0x7f46400b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 --2- 172.21.15.2:0/3111466292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f46200039a0 0x7f4640155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 >> 172.21.15.2:0/3111466292 conn(0x7f46400b45f0 msgr2=0x7f4640142090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 shutdown_connections 2024-01-15T18:05:18.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.175+0000 7f4647800640 1 -- 172.21.15.2:0/3111466292 wait complete. 2024-01-15T18:05:18.204 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:05:18.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.439+0000 7fe018800640 1 Processor -- start 2024-01-15T18:05:18.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.439+0000 7fe018800640 1 -- start start 2024-01-15T18:05:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe018800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 0x7fe01006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe018800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0100ce230 con 0x7fe010144c00 2024-01-15T18:05:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe018800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0100ce510 con 0x7fe01006ce30 2024-01-15T18:05:18.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe018800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01006d2c0 con 0x7fe01015c450 2024-01-15T18:05:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe015e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe010144c00 0x7fe0100cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51562/0 (socket says 172.21.15.2:51562) 2024-01-15T18:05:18.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe015e00640 1 -- 172.21.15.2:0/426790127 learned_addr learned my addr 172.21.15.2:0/426790127 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.441+0000 7fe015400640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 0x7fe01006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615968943 0 0) 0x7fe0100ce230 con 0x7fe010144c00 2024-01-15T18:05:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdff0003520 con 0x7fe010144c00 2024-01-15T18:05:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 280504740 0 0) 0x7fdff0003520 con 0x7fe010144c00 2024-01-15T18:05:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 msgr2=0x7fe01006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:18.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 0x7fe01006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 >> v1:172.21.15.137:6789/0 conn(0x7fe01006ce30 legacy=0x7fe010070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0100ce7f0 con 0x7fe010144c00 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe0000019f0 con 0x7fe010144c00 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 >> v1:172.21.15.2:6789/0 conn(0x7fe010144c00 legacy=0x7fe0100cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdff00039a0 0x7fdff0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00047c0 0x7fdff0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 0x7fdff0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe01006d2c0 con 0x7fdff00039a0 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdff000ba60 con 0x7fdff00047c0 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.442+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdff000bc20 con 0x7fe01015c450 2024-01-15T18:05:18.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015e00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdff00039a0 0x7fdff0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015400640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00047c0 0x7fdff0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015e00640 1 -- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 msgr2=0x7fdff0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015e00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe01015c450 0x7fdff0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015e00640 1 -- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00047c0 msgr2=0x7fdff0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015e00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00047c0 0x7fdff0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.443+0000 7fe015e00640 1 -- 172.21.15.2:0/426790127 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff0003520 con 0x7fdff00039a0 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.444+0000 7fe015e00640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdff00039a0 0x7fdff0003dc0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe000002230 tx=0x7fdff00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3f52f13dc36123 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000034630 con 0x7fdff00039a0 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00002f070 con 0x7fdff00039a0 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe014a00640 1 -- 172.21.15.2:0/426790127 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe000034a10 con 0x7fdff00039a0 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe018800640 1 -- 172.21.15.2:0/426790127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdff00039a0 msgr2=0x7fdff0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe018800640 1 --2- 172.21.15.2:0/426790127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdff00039a0 0x7fdff0003dc0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe000002230 tx=0x7fdff00057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe016800640 1 -- 172.21.15.2:0/426790127 reap_dead start 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe018800640 1 -- 172.21.15.2:0/426790127 shutdown_connections 2024-01-15T18:05:18.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe018800640 1 -- 172.21.15.2:0/426790127 >> 172.21.15.2:0/426790127 conn(0x7fe0100b45f0 msgr2=0x7fe0100b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:18.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe018800640 1 -- 172.21.15.2:0/426790127 shutdown_connections 2024-01-15T18:05:18.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.445+0000 7fe018800640 1 -- 172.21.15.2:0/426790127 wait complete. 2024-01-15T18:05:18.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.447+0000 7fe018800640 1 Processor -- start 2024-01-15T18:05:18.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.447+0000 7fe018800640 1 -- start start 2024-01-15T18:05:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe018800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00039a0 0x7fe0101f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe018800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 0x7fe010153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe018800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe010157d20 0x7fe0101544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe018800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010138a10 con 0x7fe0101578a0 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe018800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe010154a10 con 0x7fdff00039a0 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe018800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01013a2d0 con 0x7fe010157d20 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.448+0000 7fe015400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 0x7fe010153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 0x7fe010153f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53142/0 (socket says 172.21.15.2:53142) 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 -- 172.21.15.2:0/3957854543 learned_addr learned my addr 172.21.15.2:0/3957854543 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe016800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe010157d20 0x7fe0101544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 -- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe010157d20 msgr2=0x7fe0101544d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe010157d20 0x7fe0101544d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 -- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00039a0 msgr2=0x7fe0101f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015e00640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00039a0 0x7fe0101f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00039a0 0x7fe0101f4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 -- 172.21.15.2:0/3957854543 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe010151e40 con 0x7fe0101578a0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe016800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe010157d20 0x7fe0101544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe015400640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 0x7fe010153f90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdff8011610 tx=0x7fdff8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.449+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8004220 con 0x7fe0101578a0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.450+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff8004380 con 0x7fe0101578a0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.450+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff800f8f0 con 0x7fe0101578a0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.450+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe010152120 con 0x7fe0101578a0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.450+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0101526e0 con 0x7fe0101578a0 2024-01-15T18:05:18.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.451+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0100cf350 con 0x7fe0101578a0 2024-01-15T18:05:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.452+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fdff800fa50 con 0x7fe0101578a0 2024-01-15T18:05:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.453+0000 7fe00e000640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfe806bb80 0x7fdfe806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.453+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fdff8088760 con 0x7fe0101578a0 2024-01-15T18:05:18.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.453+0000 7fe015e00640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfe806bb80 0x7fdfe806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:18.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.455+0000 7fe015e00640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfe806bb80 0x7fdfe806e040 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe000034000 tx=0x7fe00002f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:18.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.458+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdff8054e50 con 0x7fe0101578a0 2024-01-15T18:05:18.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.702+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0100cead0 con 0x7fdfe806bb80 2024-01-15T18:05:18.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.764+0000 7fe00e000640 1 -- 172.21.15.2:0/3957854543 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe000004c70 con 0x7fdfe806bb80 2024-01-15T18:05:18.768 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:18.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfe806bb80 msgr2=0x7fdfe806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfe806bb80 0x7fdfe806e040 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe000034000 tx=0x7fe00002f000 comp rx=0 tx=0).stop 2024-01-15T18:05:18.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 msgr2=0x7fe010153f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 0x7fe010153f90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fdff8011610 tx=0x7fdff8011ad0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 shutdown_connections 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdfe806bb80 0x7fdfe806e040 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe010157d20 0x7fe0101544d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0101578a0 0x7fe010153f90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 --2- 172.21.15.2:0/3957854543 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdff00039a0 0x7fe0101f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 >> 172.21.15.2:0/3957854543 conn(0x7fe0100b45f0 msgr2=0x7fe01015d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:18.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 shutdown_connections 2024-01-15T18:05:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:18.767+0000 7fe018800640 1 -- 172.21.15.2:0/3957854543 wait complete. 2024-01-15T18:05:18.794 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:05:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.010+0000 7f1691200640 1 Processor -- start 2024-01-15T18:05:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.010+0000 7f1691200640 1 -- start start 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.012+0000 7f1691200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f168c161890 0x7f168c160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.012+0000 7f1691200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f168c1611a0 0x7f168c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.012+0000 7f1691200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f168c1615f0 con 0x7f168c15d270 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.012+0000 7f1691200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168c06c5e0 con 0x7f168c161890 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.012+0000 7f1691200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f168c06c8c0 con 0x7f168c1611a0 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f168c15d270 0x7f168c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51570/0 (socket says 172.21.15.2:51570) 2024-01-15T18:05:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168b400640 1 -- 172.21.15.2:0/993964866 learned_addr learned my addr 172.21.15.2:0/993964866 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168aa00640 1 --2- 172.21.15.2:0/993964866 >> v2:172.21.15.2:3300/0 conn(0x7f168c161890 0x7f168c160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168be00640 1 --2- 172.21.15.2:0/993964866 >> v2:172.21.15.137:3300/0 conn(0x7f168c1611a0 0x7f168c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168a000640 1 -- 172.21.15.2:0/993964866 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638979407 0 0) 0x7f167c001110 con 0x7f168c15d270 2024-01-15T18:05:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168a000640 1 -- 172.21.15.2:0/993964866 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166c003520 con 0x7f168c15d270 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168aa00640 1 -- 172.21.15.2:0/993964866 >> v2:172.21.15.137:3300/0 conn(0x7f168c1611a0 msgr2=0x7f168c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168aa00640 1 --2- 172.21.15.2:0/993964866 >> v2:172.21.15.137:3300/0 conn(0x7f168c1611a0 0x7f168c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168aa00640 1 -- 172.21.15.2:0/993964866 >> v1:172.21.15.2:6789/0 conn(0x7f168c15d270 legacy=0x7f168c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.013+0000 7f168aa00640 1 -- 172.21.15.2:0/993964866 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c071c00 con 0x7f168c161890 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.015+0000 7f168aa00640 1 --2- 172.21.15.2:0/993964866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c161890 0x7f168c160c60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f168c06fed0 tx=0x7f167402b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f502e1cdcc657d01 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.015+0000 7f168a000640 1 -- 172.21.15.2:0/993964866 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674031710 con 0x7f168c161890 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.015+0000 7f168a000640 1 -- 172.21.15.2:0/993964866 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167402f050 con 0x7f168c161890 2024-01-15T18:05:19.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.015+0000 7f168a000640 1 -- 172.21.15.2:0/993964866 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1674031a90 con 0x7f168c161890 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 -- 172.21.15.2:0/993964866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c161890 msgr2=0x7f168c160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 --2- 172.21.15.2:0/993964866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c161890 0x7f168c160c60 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f168c06fed0 tx=0x7f167402b7b0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 -- 172.21.15.2:0/993964866 shutdown_connections 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 --2- 172.21.15.2:0/993964866 >> v2:172.21.15.137:3300/0 conn(0x7f168c1611a0 0x7f168c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 --2- 172.21.15.2:0/993964866 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c161890 0x7f168c160c60 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 -- 172.21.15.2:0/993964866 >> 172.21.15.2:0/993964866 conn(0x7f168c0b0840 msgr2=0x7f168c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 -- 172.21.15.2:0/993964866 shutdown_connections 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.016+0000 7f1691200640 1 -- 172.21.15.2:0/993964866 wait complete. 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.018+0000 7f1691200640 1 Processor -- start 2024-01-15T18:05:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.018+0000 7f1691200640 1 -- start start 2024-01-15T18:05:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f1691200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168c15d270 0x7f168c151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f1691200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 0x7f168c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f1691200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f168c161890 0x7f168c15a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f1691200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c151e60 con 0x7f168c1611a0 2024-01-15T18:05:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f1691200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168c0ae1d0 con 0x7f168c161890 2024-01-15T18:05:19.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f1691200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c1520a0 con 0x7f168c15d270 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 0x7f168c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 0x7f168c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53158/0 (socket says 172.21.15.2:53158) 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 -- 172.21.15.2:0/576427148 learned_addr learned my addr 172.21.15.2:0/576427148 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168b400640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168c15d270 0x7f168c151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168be00640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f168c161890 0x7f168c15a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 -- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168c15d270 msgr2=0x7f168c151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168c15d270 0x7f168c151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 -- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f168c161890 msgr2=0x7f168c15a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f168c161890 0x7f168c15a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.019+0000 7f168aa00640 1 -- 172.21.15.2:0/576427148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1674009400 con 0x7f168c1611a0 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.020+0000 7f168aa00640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 0x7f168c15a090 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f167402b7f0 tx=0x7f16740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:19.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.020+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167403c810 con 0x7f168c1611a0 2024-01-15T18:05:19.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.020+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167403c970 con 0x7f168c1611a0 2024-01-15T18:05:19.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.020+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f167403cd00 con 0x7f168c1611a0 2024-01-15T18:05:19.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.020+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f168c152380 con 0x7f168c1611a0 2024-01-15T18:05:19.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.021+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f168c0ae470 con 0x7f168c1611a0 2024-01-15T18:05:19.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.022+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f168c074770 con 0x7f168c1611a0 2024-01-15T18:05:19.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.023+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f167403a040 con 0x7f168c1611a0 2024-01-15T18:05:19.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.024+0000 7f1683400640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f166406bb60 0x7f166406e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.024+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f16740a9e30 con 0x7f168c1611a0 2024-01-15T18:05:19.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.024+0000 7f168b400640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f166406bb60 0x7f166406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.025+0000 7f168b400640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f166406bb60 0x7f166406e020 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f167c007fd0 tx=0x7f167c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:19.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.028+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1674076520 con 0x7f168c1611a0 2024-01-15T18:05:19.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.250+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f168c0b2d30 con 0x7f166406bb60 2024-01-15T18:05:19.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.296+0000 7f1683400640 1 -- 172.21.15.2:0/576427148 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f167c001d10 con 0x7f166406bb60 2024-01-15T18:05:19.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f166406bb60 msgr2=0x7f166406e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f166406bb60 0x7f166406e020 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f167c007fd0 tx=0x7f167c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 msgr2=0x7f168c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 0x7f168c15a090 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f167402b7f0 tx=0x7f16740046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 shutdown_connections 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f166406bb60 0x7f166406e020 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f168c161890 0x7f168c15a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f168c1611a0 0x7f168c15a090 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 --2- 172.21.15.2:0/576427148 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f168c15d270 0x7f168c151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 >> 172.21.15.2:0/576427148 conn(0x7f168c0b0840 msgr2=0x7f168c15dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 shutdown_connections 2024-01-15T18:05:19.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.299+0000 7f1691200640 1 -- 172.21.15.2:0/576427148 wait complete. 2024-01-15T18:05:19.308 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.304+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.308 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.304+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.308 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.304+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.308 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.304+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.308 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.304+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.325 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27424 2024-01-15T18:05:19.340 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.336+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.341 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.336+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.341 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.336+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.341 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.336+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.341 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:19.336+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:19.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.524+0000 7f8876000640 1 Processor -- start 2024-01-15T18:05:19.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.524+0000 7f8876000640 1 -- start start 2024-01-15T18:05:19.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.525+0000 7f8876000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870143d20 0x7f887015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.525+0000 7f8876000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f88700720d0 0x7f887015f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.525+0000 7f8876000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f88701447d0 0x7f88700cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f8876000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8870071430 con 0x7f88700720d0 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f8876000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8870144100 con 0x7f88701447d0 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f8876000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8870071090 con 0x7f8870143d20 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f88700720d0 0x7f887015f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f88700720d0 0x7f887015f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53172/0 (socket says 172.21.15.2:53172) 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 -- 172.21.15.2:0/3238044140 learned_addr learned my addr 172.21.15.2:0/3238044140 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f8874c00640 1 --2- 172.21.15.2:0/3238044140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870143d20 0x7f887015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 -- 172.21.15.2:0/3238044140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870143d20 msgr2=0x7f887015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:19.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 --2- 172.21.15.2:0/3238044140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870143d20 0x7f887015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 -- 172.21.15.2:0/3238044140 >> v2:172.21.15.137:3300/0 conn(0x7f88701447d0 msgr2=0x7f88700cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:19.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f8875600640 1 --2- 172.21.15.2:0/3238044140 >> v2:172.21.15.137:3300/0 conn(0x7f88701447d0 0x7f88700cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 --2- 172.21.15.2:0/3238044140 >> v2:172.21.15.137:3300/0 conn(0x7f88701447d0 0x7f88700cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.526+0000 7f886fe00640 1 -- 172.21.15.2:0/3238044140 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8870144560 con 0x7f88700720d0 2024-01-15T18:05:19.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.528+0000 7f886fe00640 1 --2- 172.21.15.2:0/3238044140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88700720d0 0x7f887015f090 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f887006e0a0 tx=0x7f885802b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df8d7d6e062bfba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:19.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.528+0000 7f886f400640 1 -- 172.21.15.2:0/3238044140 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88580317d0 con 0x7f88700720d0 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.528+0000 7f886f400640 1 -- 172.21.15.2:0/3238044140 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8858031930 con 0x7f88700720d0 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.528+0000 7f886f400640 1 -- 172.21.15.2:0/3238044140 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8858031cc0 con 0x7f88700720d0 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 -- 172.21.15.2:0/3238044140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88700720d0 msgr2=0x7f887015f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 --2- 172.21.15.2:0/3238044140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88700720d0 0x7f887015f090 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f887006e0a0 tx=0x7f885802b870 comp rx=0 tx=0).stop 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 -- 172.21.15.2:0/3238044140 shutdown_connections 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 --2- 172.21.15.2:0/3238044140 >> v2:172.21.15.137:3300/0 conn(0x7f88701447d0 0x7f88700cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 --2- 172.21.15.2:0/3238044140 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88700720d0 0x7f887015f090 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 --2- 172.21.15.2:0/3238044140 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8870143d20 0x7f887015eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 -- 172.21.15.2:0/3238044140 >> 172.21.15.2:0/3238044140 conn(0x7f88700b0840 msgr2=0x7f88700b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:19.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 -- 172.21.15.2:0/3238044140 shutdown_connections 2024-01-15T18:05:19.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.529+0000 7f8876000640 1 -- 172.21.15.2:0/3238044140 wait complete. 2024-01-15T18:05:19.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.531+0000 7f8876000640 1 Processor -- start 2024-01-15T18:05:19.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.531+0000 7f8876000640 1 -- start start 2024-01-15T18:05:19.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8876000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88700720d0 0x7f88700cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8876000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8870143d20 0x7f88700a9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8876000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 0x7f88700ae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8876000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887015af50 con 0x7f88701447d0 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8876000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88700cc050 con 0x7f8870143d20 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8876000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887012f1d0 con 0x7f88700720d0 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8875600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 0x7f88700ae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8875600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 0x7f88700ae1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53176/0 (socket says 172.21.15.2:53176) 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.532+0000 7f8875600640 1 -- 172.21.15.2:0/3573451433 learned_addr learned my addr 172.21.15.2:0/3573451433 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8874c00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88700720d0 0x7f88700cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8875600640 1 -- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88700720d0 msgr2=0x7f88700cbb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f886fe00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8870143d20 0x7f88700a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8875600640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88700720d0 0x7f88700cbb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8875600640 1 -- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8870143d20 msgr2=0x7f88700a9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8875600640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8870143d20 0x7f88700a9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8875600640 1 -- 172.21.15.2:0/3573451433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88580094d0 con 0x7f88701447d0 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8874c00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88700720d0 0x7f88700cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f886fe00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8870143d20 0x7f88700a9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.533+0000 7f8875600640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 0x7f88700ae1a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f886800cda0 tx=0x7f886800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.534+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868003b90 con 0x7f88701447d0 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.534+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8868003cf0 con 0x7f88701447d0 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.534+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868004240 con 0x7f88701447d0 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.534+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88700aa7c0 con 0x7f88701447d0 2024-01-15T18:05:19.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.534+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88700aad80 con 0x7f88701447d0 2024-01-15T18:05:19.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.535+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8870074770 con 0x7f88701447d0 2024-01-15T18:05:19.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.536+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f88680043a0 con 0x7f88701447d0 2024-01-15T18:05:19.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.537+0000 7f886cc00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f884806bb30 0x7f884806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:19.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.537+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f8868012070 con 0x7f88701447d0 2024-01-15T18:05:19.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.537+0000 7f8874c00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f884806bb30 0x7f884806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:19.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.539+0000 7f8874c00640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f884806bb30 0x7f884806dff0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8860006fd0 tx=0x7f8860008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:19.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.541+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8868055670 con 0x7f88701447d0 2024-01-15T18:05:19.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.701+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f8868055a40 con 0x7f88701447d0 2024-01-15T18:05:19.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.784+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27424", "target": ["mon-mgr", ""]}) v1 -- 0x7f887015fa00 con 0x7f884806bb30 2024-01-15T18:05:19.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.829+0000 7f886cc00640 1 -- 172.21.15.2:0/3573451433 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8860004030 con 0x7f884806bb30 2024-01-15T18:05:19.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f884806bb30 msgr2=0x7f884806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f884806bb30 0x7f884806dff0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8860006fd0 tx=0x7f8860008040 comp rx=0 tx=0).stop 2024-01-15T18:05:19.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 msgr2=0x7f88700ae1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:19.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 0x7f88700ae1a0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f886800cda0 tx=0x7f886800a660 comp rx=0 tx=0).stop 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 shutdown_connections 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f884806bb30 0x7f884806dff0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88701447d0 0x7f88700ae1a0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8870143d20 0x7f88700a9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 --2- 172.21.15.2:0/3573451433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88700720d0 0x7f88700cbb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.832+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 >> 172.21.15.2:0/3573451433 conn(0x7f88700b0840 msgr2=0x7f88700afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.833+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 shutdown_connections 2024-01-15T18:05:19.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:19.833+0000 7f8876000640 1 -- 172.21.15.2:0/3573451433 wait complete. 2024-01-15T18:05:19.858 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27425 2024-01-15T18:05:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.058+0000 7f6aaa800640 1 Processor -- start 2024-01-15T18:05:20.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.058+0000 7f6aaa800640 1 -- start start 2024-01-15T18:05:20.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aaa800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6aa4143d20 0x7f6aa415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aaa800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 0x7f6aa415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aaa800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa4071090 con 0x7f6aa40720d0 2024-01-15T18:05:20.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aaa800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa4071430 con 0x7f6aa4143d20 2024-01-15T18:05:20.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aaa800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa4144100 con 0x7f6aa41447d0 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aa9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 0x7f6aa415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aa9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 0x7f6aa415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:41594/0 (socket says 172.21.15.2:41594) 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aa9e00640 1 -- 172.21.15.2:0/3205033121 learned_addr learned my addr 172.21.15.2:0/3205033121 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.060+0000 7f6aa9400640 1 --2- 172.21.15.2:0/3205033121 >> v2:172.21.15.137:3300/0 conn(0x7f6aa4143d20 0x7f6aa415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.061+0000 7f6aa9e00640 1 -- 172.21.15.2:0/3205033121 >> v2:172.21.15.137:3300/0 conn(0x7f6aa4143d20 msgr2=0x7f6aa415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.061+0000 7f6aa9e00640 1 --2- 172.21.15.2:0/3205033121 >> v2:172.21.15.137:3300/0 conn(0x7f6aa4143d20 0x7f6aa415eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.061+0000 7f6aa9e00640 1 -- 172.21.15.2:0/3205033121 >> v1:172.21.15.137:6789/0 conn(0x7f6aa40720d0 legacy=0x7f6aa40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.061+0000 7f6aa9e00640 1 -- 172.21.15.2:0/3205033121 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa406d160 con 0x7f6aa41447d0 2024-01-15T18:05:20.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.062+0000 7f6aa9e00640 1 --2- 172.21.15.2:0/3205033121 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 0x7f6aa415f090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009e50 tx=0x7f6a9c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=107289d1dda29f3a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.063+0000 7f6aa3e00640 1 -- 172.21.15.2:0/3205033121 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c02ca70 con 0x7f6aa41447d0 2024-01-15T18:05:20.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.063+0000 7f6aa3e00640 1 -- 172.21.15.2:0/3205033121 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c02cbd0 con 0x7f6aa41447d0 2024-01-15T18:05:20.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.063+0000 7f6aa3e00640 1 -- 172.21.15.2:0/3205033121 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0343f0 con 0x7f6aa41447d0 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 -- 172.21.15.2:0/3205033121 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 msgr2=0x7f6aa415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 --2- 172.21.15.2:0/3205033121 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 0x7f6aa415f090 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009e50 tx=0x7f6a9c02bed0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 -- 172.21.15.2:0/3205033121 shutdown_connections 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 --2- 172.21.15.2:0/3205033121 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa41447d0 0x7f6aa415f090 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 --2- 172.21.15.2:0/3205033121 >> v2:172.21.15.137:3300/0 conn(0x7f6aa4143d20 0x7f6aa415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 -- 172.21.15.2:0/3205033121 >> 172.21.15.2:0/3205033121 conn(0x7f6aa40b0840 msgr2=0x7f6aa40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 -- 172.21.15.2:0/3205033121 shutdown_connections 2024-01-15T18:05:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.064+0000 7f6aaa800640 1 -- 172.21.15.2:0/3205033121 wait complete. 2024-01-15T18:05:20.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.066+0000 7f6aaa800640 1 Processor -- start 2024-01-15T18:05:20.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.066+0000 7f6aaa800640 1 -- start start 2024-01-15T18:05:20.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aaa800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa40720d0 0x7f6aa4135430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aaa800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 0x7f6aa41f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aaa800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6aa41447d0 0x7f6aa415e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aaa800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa4135970 con 0x7f6aa4143d20 2024-01-15T18:05:20.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aaa800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40b22d0 con 0x7f6aa41447d0 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aaa800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa40b25b0 con 0x7f6aa40720d0 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 0x7f6aa41f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 0x7f6aa41f2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53186/0 (socket says 172.21.15.2:53186) 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 -- 172.21.15.2:0/2643404969 learned_addr learned my addr 172.21.15.2:0/2643404969 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa9400640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa40720d0 0x7f6aa4135430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa9e00640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6aa41447d0 0x7f6aa415e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 -- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa40720d0 msgr2=0x7f6aa4135430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa40720d0 0x7f6aa4135430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 -- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6aa41447d0 msgr2=0x7f6aa415e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6aa41447d0 0x7f6aa415e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.067+0000 7f6aa8a00640 1 -- 172.21.15.2:0/2643404969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c009b30 con 0x7f6aa4143d20 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.068+0000 7f6aa8a00640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 0x7f6aa41f2e30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c400 tx=0x7f6a8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.068+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0052d0 con 0x7f6aa4143d20 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.068+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c005430 con 0x7f6aa4143d20 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.068+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c003830 con 0x7f6aa4143d20 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.068+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa40b2890 con 0x7f6aa4143d20 2024-01-15T18:05:20.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.068+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa41f3370 con 0x7f6aa4143d20 2024-01-15T18:05:20.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.069+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa40b2db0 con 0x7f6aa4143d20 2024-01-15T18:05:20.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.070+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f6a8c003990 con 0x7f6aa4143d20 2024-01-15T18:05:20.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.071+0000 7f6aa1600640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a7c06bc50 0x7f6a7c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.071+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f6a8c087d20 con 0x7f6aa4143d20 2024-01-15T18:05:20.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.072+0000 7f6aa9400640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a7c06bc50 0x7f6a7c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.073+0000 7f6aa9400640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a7c06bc50 0x7f6a7c06e110 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009a30 tx=0x7f6a94008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.076+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6aa40b2db0 con 0x7f6aa4143d20 2024-01-15T18:05:20.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.295+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27425", "target": ["mon-mgr", ""]}) v1 -- 0x7f6aa415f720 con 0x7f6a7c06bc50 2024-01-15T18:05:20.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.335+0000 7f6aa1600640 1 -- 172.21.15.2:0/2643404969 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6a940119a0 con 0x7f6a7c06bc50 2024-01-15T18:05:20.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a7c06bc50 msgr2=0x7f6a7c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a7c06bc50 0x7f6a7c06e110 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6a94009a30 tx=0x7f6a94008040 comp rx=0 tx=0).stop 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 msgr2=0x7f6aa41f2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 0x7f6aa41f2e30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00c400 tx=0x7f6a8c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 shutdown_connections 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a7c06bc50 0x7f6a7c06e110 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6aa41447d0 0x7f6aa415e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6aa4143d20 0x7f6aa41f2e30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 --2- 172.21.15.2:0/2643404969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6aa40720d0 0x7f6aa4135430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.337+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 >> 172.21.15.2:0/2643404969 conn(0x7f6aa40b0840 msgr2=0x7f6aa40aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.338+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 shutdown_connections 2024-01-15T18:05:20.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.338+0000 7f6aaa800640 1 -- 172.21.15.2:0/2643404969 wait complete. 2024-01-15T18:05:20.364 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27427 2024-01-15T18:05:20.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.572+0000 7f9656e00640 1 Processor -- start 2024-01-15T18:05:20.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.572+0000 7f9656e00640 1 -- start start 2024-01-15T18:05:20.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.574+0000 7f9656e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f96500cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.574+0000 7f9656e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f96500cdd70 0x7f965006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.574+0000 7f9656e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96500ce230 con 0x7f965015c450 2024-01-15T18:05:20.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.574+0000 7f9656e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96500ce510 con 0x7f96500cdd70 2024-01-15T18:05:20.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.574+0000 7f9656e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96500ce7f0 con 0x7f9650144c00 2024-01-15T18:05:20.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f964fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f96500cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f964fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f96500cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:41610/0 (socket says 172.21.15.2:41610) 2024-01-15T18:05:20.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f964fe00640 1 -- 172.21.15.2:0/2095646117 learned_addr learned my addr 172.21.15.2:0/2095646117 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:20.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f9647200640 1 --2- 172.21.15.2:0/2095646117 >> v2:172.21.15.137:3300/0 conn(0x7f96500cdd70 0x7f965006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f964fe00640 1 -- 172.21.15.2:0/2095646117 >> v2:172.21.15.137:3300/0 conn(0x7f96500cdd70 msgr2=0x7f965006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:20.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f964fe00640 1 --2- 172.21.15.2:0/2095646117 >> v2:172.21.15.137:3300/0 conn(0x7f96500cdd70 0x7f965006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.575+0000 7f964fe00640 1 -- 172.21.15.2:0/2095646117 >> v1:172.21.15.137:6789/0 conn(0x7f965015c450 legacy=0x7f96500b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.576+0000 7f964fe00640 1 -- 172.21.15.2:0/2095646117 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96500705f0 con 0x7f9650144c00 2024-01-15T18:05:20.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.577+0000 7f964fe00640 1 --2- 172.21.15.2:0/2095646117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f96500cd830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9640009e50 tx=0x7f964002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f14b3b2bbf0a685d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.578+0000 7f964f400640 1 -- 172.21.15.2:0/2095646117 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964002cb00 con 0x7f9650144c00 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.578+0000 7f964f400640 1 -- 172.21.15.2:0/2095646117 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964002cc60 con 0x7f9650144c00 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.578+0000 7f964f400640 1 -- 172.21.15.2:0/2095646117 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96400343f0 con 0x7f9650144c00 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 -- 172.21.15.2:0/2095646117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 msgr2=0x7f96500cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 --2- 172.21.15.2:0/2095646117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f96500cd830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9640009e50 tx=0x7f964002bf60 comp rx=0 tx=0).stop 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 -- 172.21.15.2:0/2095646117 shutdown_connections 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 --2- 172.21.15.2:0/2095646117 >> v2:172.21.15.137:3300/0 conn(0x7f96500cdd70 0x7f965006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 --2- 172.21.15.2:0/2095646117 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f96500cd830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 -- 172.21.15.2:0/2095646117 >> 172.21.15.2:0/2095646117 conn(0x7f96500b45f0 msgr2=0x7f96500b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:20.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 -- 172.21.15.2:0/2095646117 shutdown_connections 2024-01-15T18:05:20.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.579+0000 7f9656e00640 1 -- 172.21.15.2:0/2095646117 wait complete. 2024-01-15T18:05:20.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.581+0000 7f9656e00640 1 Processor -- start 2024-01-15T18:05:20.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.581+0000 7f9656e00640 1 -- start start 2024-01-15T18:05:20.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f9656e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 0x7f9650138f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f9656e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f965013ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f9656e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f965015c450 0x7f96500aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f9656e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96501f2db0 con 0x7f96500cdd70 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f9656e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650139570 con 0x7f965015c450 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f9656e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9650139850 con 0x7f9650144c00 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f964fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 0x7f9650138f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f964fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 0x7f9650138f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53190/0 (socket says 172.21.15.2:53190) 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.582+0000 7f964fe00640 1 -- 172.21.15.2:0/4003083018 learned_addr learned my addr 172.21.15.2:0/4003083018 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:20.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f9647200640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f965013ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f964fe00640 1 -- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 msgr2=0x7f965013ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f9654e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f965015c450 0x7f96500aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f964fe00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f965013ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f964fe00640 1 -- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f965015c450 msgr2=0x7f96500aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f964fe00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f965015c450 0x7f96500aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f964fe00640 1 -- 172.21.15.2:0/4003083018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640009b30 con 0x7f96500cdd70 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f9654e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f965015c450 0x7f96500aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.583+0000 7f964fe00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 0x7f9650138f90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9640000f80 tx=0x7f9640004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.584+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640009510 con 0x7f96500cdd70 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.584+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964003e450 con 0x7f96500cdd70 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.584+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964003e800 con 0x7f96500cdd70 2024-01-15T18:05:20.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.584+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9650139b30 con 0x7f96500cdd70 2024-01-15T18:05:20.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.584+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f965013a0f0 con 0x7f96500cdd70 2024-01-15T18:05:20.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.585+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96500686f0 con 0x7f96500cdd70 2024-01-15T18:05:20.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.586+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f964003e960 con 0x7f96500cdd70 2024-01-15T18:05:20.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.587+0000 7f964cc00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f962806bc50 0x7f962806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:20.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.587+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f96400ab490 con 0x7f96500cdd70 2024-01-15T18:05:20.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.587+0000 7f9647200640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f962806bc50 0x7f962806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:20.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.589+0000 7f9647200640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f962806bc50 0x7f962806e110 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f96501f63a0 tx=0x7f9638008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:20.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.592+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9640077bf0 con 0x7f96500cdd70 2024-01-15T18:05:20.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.817+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27427", "target": ["mon-mgr", ""]}) v1 -- 0x7f965006ce30 con 0x7f962806bc50 2024-01-15T18:05:20.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.864+0000 7f964cc00640 1 -- 172.21.15.2:0/4003083018 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9638010840 con 0x7f962806bc50 2024-01-15T18:05:20.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f962806bc50 msgr2=0x7f962806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f962806bc50 0x7f962806e110 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f96501f63a0 tx=0x7f9638008040 comp rx=0 tx=0).stop 2024-01-15T18:05:20.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 msgr2=0x7f9650138f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 0x7f9650138f90 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f9640000f80 tx=0x7f9640004a40 comp rx=0 tx=0).stop 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 shutdown_connections 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f962806bc50 0x7f962806e110 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f965015c450 0x7f96500aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9650144c00 0x7f965013ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 --2- 172.21.15.2:0/4003083018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96500cdd70 0x7f9650138f90 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 >> 172.21.15.2:0/4003083018 conn(0x7f96500b45f0 msgr2=0x7f9650140b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 shutdown_connections 2024-01-15T18:05:20.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:20.867+0000 7f9656e00640 1 -- 172.21.15.2:0/4003083018 wait complete. 2024-01-15T18:05:20.895 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27423 2024-01-15T18:05:21.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.111+0000 7fb5b1400640 1 Processor -- start 2024-01-15T18:05:21.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.111+0000 7fb5b1400640 1 -- start start 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.113+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 0x7fb5ac15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.113+0000 7fb5b1400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb5ac1447d0 0x7fb5ac15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.113+0000 7fb5b1400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac071090 con 0x7fb5ac0720d0 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.113+0000 7fb5b1400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac071430 con 0x7fb5ac1447d0 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.113+0000 7fb5b1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac144100 con 0x7fb5ac143d20 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.114+0000 7fb5a3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb5ac0720d0 0x7fb5ac0cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51576/0 (socket says 172.21.15.2:51576) 2024-01-15T18:05:21.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.114+0000 7fb5a3e00640 1 -- 172.21.15.2:0/3929072747 learned_addr learned my addr 172.21.15.2:0/3929072747 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:21.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.114+0000 7fb5abe00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 0x7fb5ac15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.114+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861146092 0 0) 0x7fb5ac071090 con 0x7fb5ac0720d0 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.114+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb58c003520 con 0x7fb5ac0720d0 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.114+0000 7fb5b0a00640 1 --2- 172.21.15.2:0/3929072747 >> v2:172.21.15.137:3300/0 conn(0x7fb5ac1447d0 0x7fb5ac15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3840356114 0 0) 0x7fb58c003520 con 0x7fb5ac0720d0 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 msgr2=0x7fb5ac15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 0x7fb5ac15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 >> v2:172.21.15.137:3300/0 conn(0x7fb5ac1447d0 msgr2=0x7fb5ac15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 --2- 172.21.15.2:0/3929072747 >> v2:172.21.15.137:3300/0 conn(0x7fb5ac1447d0 0x7fb5ac15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac06d160 con 0x7fb5ac0720d0 2024-01-15T18:05:21.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb594001670 con 0x7fb5ac0720d0 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 >> v1:172.21.15.2:6789/0 conn(0x7fb5ac0720d0 legacy=0x7fb5ac0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb58c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb58c0047c0 0x7fb58c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 0x7fb58c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac071430 con 0x7fb58c003970 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac144100 con 0x7fb58c0047c0 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58c00ba80 con 0x7fb5ac143d20 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5abe00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb58c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5a3e00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb58c0047c0 0x7fb58c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.115+0000 7fb5b0a00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 0x7fb58c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.116+0000 7fb5abe00640 1 -- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 msgr2=0x7fb58c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.116+0000 7fb5abe00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac143d20 0x7fb58c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.116+0000 7fb5abe00640 1 -- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb58c0047c0 msgr2=0x7fb58c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.116+0000 7fb5abe00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb58c0047c0 0x7fb58c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.116+0000 7fb5abe00640 1 -- 172.21.15.2:0/3929072747 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c003520 con 0x7fb58c003970 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.117+0000 7fb5abe00640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb58c003d90 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009ac0 tx=0x7fb58c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388bb4fc560043ca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.117+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c02c660 con 0x7fb58c003970 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.117+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c02c7c0 con 0x7fb58c003970 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.117+0000 7fb5ab400640 1 -- 172.21.15.2:0/3929072747 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c02cb50 con 0x7fb58c003970 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b1400640 1 -- 172.21.15.2:0/3929072747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 msgr2=0x7fb58c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b1400640 1 --2- 172.21.15.2:0/3929072747 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb58c003d90 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009ac0 tx=0x7fb58c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b0a00640 1 -- 172.21.15.2:0/3929072747 reap_dead start 2024-01-15T18:05:21.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b1400640 1 -- 172.21.15.2:0/3929072747 shutdown_connections 2024-01-15T18:05:21.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b1400640 1 -- 172.21.15.2:0/3929072747 >> 172.21.15.2:0/3929072747 conn(0x7fb5ac0b0840 msgr2=0x7fb5ac0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:21.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b1400640 1 -- 172.21.15.2:0/3929072747 shutdown_connections 2024-01-15T18:05:21.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.118+0000 7fb5b1400640 1 -- 172.21.15.2:0/3929072747 wait complete. 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.120+0000 7fb5b1400640 1 Processor -- start 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.120+0000 7fb5b1400640 1 -- start start 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb5ac0c5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac15dc10 0x7fb5ac1413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5b1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5ac141920 0x7fb5ac0c6040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5b1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0c4ef0 con 0x7fb58c003970 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5b1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0c58c0 con 0x7fb5ac141920 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5b1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac0c45c0 con 0x7fb5ac15dc10 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5abe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb5ac0c5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5abe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb5ac0c5380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53214/0 (socket says 172.21.15.2:53214) 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.121+0000 7fb5abe00640 1 -- 172.21.15.2:0/1793093268 learned_addr learned my addr 172.21.15.2:0/1793093268 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5a3e00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac15dc10 0x7fb5ac1413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5abe00640 1 -- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac15dc10 msgr2=0x7fb5ac1413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5abe00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac15dc10 0x7fb5ac1413e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5abe00640 1 -- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5ac141920 msgr2=0x7fb5ac0c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5abe00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5ac141920 0x7fb5ac0c6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5abe00640 1 -- 172.21.15.2:0/1793093268 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0c48a0 con 0x7fb58c003970 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5abe00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb5ac0c5380 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb59c02c200 tx=0x7fb59c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c003810 con 0x7fb58c003970 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c003970 con 0x7fb58c003970 2024-01-15T18:05:21.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb59c003ce0 con 0x7fb58c003970 2024-01-15T18:05:21.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac0c4b80 con 0x7fb58c003970 2024-01-15T18:05:21.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.122+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac0c6d30 con 0x7fb58c003970 2024-01-15T18:05:21.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.123+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac074770 con 0x7fb58c003970 2024-01-15T18:05:21.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.125+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fb59c034440 con 0x7fb58c003970 2024-01-15T18:05:21.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.125+0000 7fb5a8c00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb58406bc50 0x7fb58406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.126+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fb59c0aa550 con 0x7fb58c003970 2024-01-15T18:05:21.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.126+0000 7fb5a3e00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb58406bc50 0x7fb58406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.127+0000 7fb5a3e00640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb58406bc50 0x7fb58406e110 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac15e4d0 tx=0x7fb594009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:21.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.130+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb59c076e40 con 0x7fb58c003970 2024-01-15T18:05:21.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.359+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27423", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5ac15f720 con 0x7fb58406bc50 2024-01-15T18:05:21.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.410+0000 7fb5a8c00640 1 -- 172.21.15.2:0/1793093268 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb59400d040 con 0x7fb58406bc50 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb58406bc50 msgr2=0x7fb58406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb58406bc50 0x7fb58406e110 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac15e4d0 tx=0x7fb594009040 comp rx=0 tx=0).stop 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 msgr2=0x7fb5ac0c5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb5ac0c5380 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb59c02c200 tx=0x7fb59c004730 comp rx=0 tx=0).stop 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 shutdown_connections 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb58406bc50 0x7fb58406e110 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb5ac141920 0x7fb5ac0c6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb5ac15dc10 0x7fb5ac1413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 --2- 172.21.15.2:0/1793093268 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb58c003970 0x7fb5ac0c5380 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 >> 172.21.15.2:0/1793093268 conn(0x7fb5ac0b0840 msgr2=0x7fb5ac06d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:21.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 shutdown_connections 2024-01-15T18:05:21.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.414+0000 7fb5b1400640 1 -- 172.21.15.2:0/1793093268 wait complete. 2024-01-15T18:05:21.440 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27421 2024-01-15T18:05:21.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.641+0000 7f1373e00640 1 Processor -- start 2024-01-15T18:05:21.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.641+0000 7f1373e00640 1 -- start start 2024-01-15T18:05:21.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1373e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f13740d1870 0x7f13740d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1373e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f13740d1c50 0x7f13740b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1373e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13740d1250 con 0x7f13740b2210 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1373e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13740ce6d0 con 0x7f13740d1870 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1373e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13740ce8e0 con 0x7f13740d1c50 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1372a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f13740d1870 0x7f13740d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1372a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f13740d1870 0x7f13740d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53230/0 (socket says 172.21.15.2:53230) 2024-01-15T18:05:21.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.643+0000 7f1372a00640 1 -- 172.21.15.2:0/1169283029 learned_addr learned my addr 172.21.15.2:0/1169283029 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:21.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.644+0000 7f1372a00640 1 -- 172.21.15.2:0/1169283029 >> v2:172.21.15.137:3300/0 conn(0x7f13740d1c50 msgr2=0x7f13740b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:21.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.644+0000 7f1373400640 1 --2- 172.21.15.2:0/1169283029 >> v2:172.21.15.137:3300/0 conn(0x7f13740d1c50 0x7f13740b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.644+0000 7f1372a00640 1 --2- 172.21.15.2:0/1169283029 >> v2:172.21.15.137:3300/0 conn(0x7f13740d1c50 0x7f13740b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.644+0000 7f1372a00640 1 -- 172.21.15.2:0/1169283029 >> v1:172.21.15.137:6789/0 conn(0x7f13740b2210 legacy=0x7f137415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:21.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.644+0000 7f1372a00640 1 -- 172.21.15.2:0/1169283029 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13740cebc0 con 0x7f13740d1870 2024-01-15T18:05:21.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.644+0000 7f1373400640 1 --2- 172.21.15.2:0/1169283029 >> v2:172.21.15.137:3300/0 conn(0x7f13740d1c50 0x7f13740b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:21.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.645+0000 7f1372a00640 1 --2- 172.21.15.2:0/1169283029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13740d3bd0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f137408d7e0 tx=0x7f136402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd496543c60be102 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:21.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.646+0000 7f1372000640 1 -- 172.21.15.2:0/1169283029 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136402ca80 con 0x7f13740d1870 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.646+0000 7f1372000640 1 -- 172.21.15.2:0/1169283029 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136402cbe0 con 0x7f13740d1870 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.646+0000 7f1372000640 1 -- 172.21.15.2:0/1169283029 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13640343f0 con 0x7f13740d1870 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.646+0000 7f1373e00640 1 -- 172.21.15.2:0/1169283029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 msgr2=0x7f13740d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.646+0000 7f1373e00640 1 --2- 172.21.15.2:0/1169283029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13740d3bd0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f137408d7e0 tx=0x7f136402bee0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.646+0000 7f1373e00640 1 -- 172.21.15.2:0/1169283029 shutdown_connections 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.647+0000 7f1373e00640 1 --2- 172.21.15.2:0/1169283029 >> v2:172.21.15.137:3300/0 conn(0x7f13740d1c50 0x7f13740b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.647+0000 7f1373e00640 1 --2- 172.21.15.2:0/1169283029 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13740d3bd0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.647+0000 7f1373e00640 1 -- 172.21.15.2:0/1169283029 >> 172.21.15.2:0/1169283029 conn(0x7f13740d30b0 msgr2=0x7f13740d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:21.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.647+0000 7f1373e00640 1 -- 172.21.15.2:0/1169283029 shutdown_connections 2024-01-15T18:05:21.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.647+0000 7f1373e00640 1 -- 172.21.15.2:0/1169283029 wait complete. 2024-01-15T18:05:21.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.650+0000 7f1373e00640 1 Processor -- start 2024-01-15T18:05:21.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.650+0000 7f1373e00640 1 -- start start 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1373e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13740b2210 0x7f137414b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1373e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13741f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1373e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13740d1c50 0x7f137414c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1373e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137415ed60 con 0x7f13740d1870 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1373e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137415f040 con 0x7f13740d1c50 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1373e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137415f320 con 0x7f13740b2210 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1369e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13741f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1369e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13741f6e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53232/0 (socket says 172.21.15.2:53232) 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.652+0000 7f1369e00640 1 -- 172.21.15.2:0/1235859933 learned_addr learned my addr 172.21.15.2:0/1235859933 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1372a00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13740b2210 0x7f137414b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1369e00640 1 -- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13740b2210 msgr2=0x7f137414b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1369e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13740b2210 0x7f137414b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1369e00640 1 -- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13740d1c50 msgr2=0x7f137414c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1369e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13740d1c50 0x7f137414c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1369e00640 1 -- 172.21.15.2:0/1235859933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1364009b30 con 0x7f13740d1870 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1369e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13741f6e20 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c400 tx=0x7f135c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c011790 con 0x7f13740d1870 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c011930 con 0x7f13740d1870 2024-01-15T18:05:21.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f135c011ce0 con 0x7f13740d1870 2024-01-15T18:05:21.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.653+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137415f600 con 0x7f13740d1870 2024-01-15T18:05:21.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.654+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137414bd30 con 0x7f13740d1870 2024-01-15T18:05:21.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.655+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137415fb20 con 0x7f13740d1870 2024-01-15T18:05:21.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.658+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f135c01c460 con 0x7f13740d1870 2024-01-15T18:05:21.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.659+0000 7f136b400640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f134c06bc50 0x7f134c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:21.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.659+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f135c088fa0 con 0x7f13740d1870 2024-01-15T18:05:21.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.660+0000 7f1372a00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f134c06bc50 0x7f134c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:21.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.661+0000 7f1372a00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f134c06bc50 0x7f134c06e110 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1364035000 tx=0x7f1364030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:21.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.662+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f135c055660 con 0x7f13740d1870 2024-01-15T18:05:22.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:21.997+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27421", "target": ["mon-mgr", ""]}) v1 -- 0x7f13740d0f20 con 0x7f134c06bc50 2024-01-15T18:05:22.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.025+0000 7f136b400640 1 -- 172.21.15.2:0/1235859933 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1364040450 con 0x7f134c06bc50 2024-01-15T18:05:22.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f134c06bc50 msgr2=0x7f134c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f134c06bc50 0x7f134c06e110 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1364035000 tx=0x7f1364030000 comp rx=0 tx=0).stop 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 msgr2=0x7f13741f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13741f6e20 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c400 tx=0x7f135c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 shutdown_connections 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f134c06bc50 0x7f134c06e110 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f13740d1c50 0x7f137414c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f13740d1870 0x7f13741f6e20 secure :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f135c00c400 tx=0x7f135c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 --2- 172.21.15.2:0/1235859933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13740b2210 0x7f137414b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.029+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 >> 172.21.15.2:0/1235859933 conn(0x7f13740d30b0 msgr2=0x7f13740b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:22.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.031+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 shutdown_connections 2024-01-15T18:05:22.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.031+0000 7f1373e00640 1 -- 172.21.15.2:0/1235859933 wait complete. 2024-01-15T18:05:22.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27426 2024-01-15T18:05:22.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.352+0000 7fe54a200640 1 Processor -- start 2024-01-15T18:05:22.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.353+0000 7fe54a200640 1 -- start start 2024-01-15T18:05:22.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.354+0000 7fe54a200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe54406cfc0 0x7fe544157090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe54a200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54415a800 con 0x7fe5440b28b0 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe54a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5440b4230 con 0x7fe5440cf350 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe54a200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5440cba50 con 0x7fe54406cfc0 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe542a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe5440b28b0 0x7fe544070b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51580/0 (socket says 172.21.15.2:51580) 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe542a00640 1 -- 172.21.15.2:0/1942313774 learned_addr learned my addr 172.21.15.2:0/1942313774 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe543400640 1 --2- 172.21.15.2:0/1942313774 >> v2:172.21.15.137:3300/0 conn(0x7fe54406cfc0 0x7fe544157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.355+0000 7fe542000640 1 -- 172.21.15.2:0/1942313774 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 913802422 0 0) 0x7fe54415a800 con 0x7fe5440b28b0 2024-01-15T18:05:22.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.356+0000 7fe542000640 1 -- 172.21.15.2:0/1942313774 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe524005730 con 0x7fe5440b28b0 2024-01-15T18:05:22.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.356+0000 7fe543400640 1 -- 172.21.15.2:0/1942313774 >> v1:172.21.15.137:6789/0 conn(0x7fe5440cf350 legacy=0x7fe54406d860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:05:22.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.356+0000 7fe543400640 1 -- 172.21.15.2:0/1942313774 >> v1:172.21.15.2:6789/0 conn(0x7fe5440b28b0 legacy=0x7fe544070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.356+0000 7fe543400640 1 -- 172.21.15.2:0/1942313774 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe544159bf0 con 0x7fe54406cfc0 2024-01-15T18:05:22.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.358+0000 7fe543400640 1 --2- 172.21.15.2:0/1942313774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54406cfc0 0x7fe544157090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe5340097a0 tx=0x7fe53402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d0d61fc3f03ee89 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:22.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.359+0000 7fe542000640 1 -- 172.21.15.2:0/1942313774 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534030610 con 0x7fe54406cfc0 2024-01-15T18:05:22.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.359+0000 7fe542000640 1 -- 172.21.15.2:0/1942313774 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe534030770 con 0x7fe54406cfc0 2024-01-15T18:05:22.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.359+0000 7fe542000640 1 -- 172.21.15.2:0/1942313774 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534030b00 con 0x7fe54406cfc0 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.360+0000 7fe54a200640 1 -- 172.21.15.2:0/1942313774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54406cfc0 msgr2=0x7fe544157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.360+0000 7fe54a200640 1 --2- 172.21.15.2:0/1942313774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54406cfc0 0x7fe544157090 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe5340097a0 tx=0x7fe53402b960 comp rx=0 tx=0).stop 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.360+0000 7fe54a200640 1 -- 172.21.15.2:0/1942313774 shutdown_connections 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.360+0000 7fe54a200640 1 --2- 172.21.15.2:0/1942313774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe54406cfc0 0x7fe544157090 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.360+0000 7fe54a200640 1 -- 172.21.15.2:0/1942313774 >> 172.21.15.2:0/1942313774 conn(0x7fe54406c4d0 msgr2=0x7fe54406c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.361+0000 7fe54a200640 1 -- 172.21.15.2:0/1942313774 shutdown_connections 2024-01-15T18:05:22.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.361+0000 7fe54a200640 1 -- 172.21.15.2:0/1942313774 wait complete. 2024-01-15T18:05:22.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.364+0000 7fe54a200640 1 Processor -- start 2024-01-15T18:05:22.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.364+0000 7fe54a200640 1 -- start start 2024-01-15T18:05:22.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe54a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 0x7fe544154ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:22.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe54a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5440b28b0 0x7fe544150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:22.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe54a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5440cf350 0x7fe5440b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe54a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe544150780 con 0x7fe54406cfc0 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe54a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5440b0b00 con 0x7fe5440cf350 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe54a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe544153e40 con 0x7fe5440b28b0 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe543400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 0x7fe544154ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe543400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 0x7fe544154ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53248/0 (socket says 172.21.15.2:53248) 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe543400640 1 -- 172.21.15.2:0/1589792864 learned_addr learned my addr 172.21.15.2:0/1589792864 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe542a00640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5440b28b0 0x7fe544150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:22.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.366+0000 7fe543e00640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5440cf350 0x7fe5440b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.367+0000 7fe543400640 1 -- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5440b28b0 msgr2=0x7fe544150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.367+0000 7fe543400640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5440b28b0 0x7fe544150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.367+0000 7fe543400640 1 -- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5440cf350 msgr2=0x7fe5440b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.367+0000 7fe543400640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5440cf350 0x7fe5440b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.367+0000 7fe543400640 1 -- 172.21.15.2:0/1589792864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe534009400 con 0x7fe54406cfc0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.368+0000 7fe543400640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 0x7fe544154ee0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe5340098c0 tx=0x7fe534004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.368+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53403e040 con 0x7fe54406cfc0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.368+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe534039a40 con 0x7fe54406cfc0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.368+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe544154120 con 0x7fe54406cfc0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.369+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534039df0 con 0x7fe54406cfc0 2024-01-15T18:05:22.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.369+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5441546e0 con 0x7fe54406cfc0 2024-01-15T18:05:22.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.370+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5440686f0 con 0x7fe54406cfc0 2024-01-15T18:05:22.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.372+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fe53404d3b0 con 0x7fe54406cfc0 2024-01-15T18:05:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.373+0000 7fe53b400640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51c06bca0 0x7fe51c06e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:22.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.373+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fe5340ad6e0 con 0x7fe54406cfc0 2024-01-15T18:05:22.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.376+0000 7fe542a00640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51c06bca0 0x7fe51c06e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:22.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.377+0000 7fe542a00640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51c06bca0 0x7fe51c06e160 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe544156df0 tx=0x7fe52c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:22.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.379+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5340773e0 con 0x7fe54406cfc0 2024-01-15T18:05:22.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.692+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27426", "target": ["mon-mgr", ""]}) v1 -- 0x7fe544066fd0 con 0x7fe51c06bca0 2024-01-15T18:05:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.732+0000 7fe53b400640 1 -- 172.21.15.2:0/1589792864 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe544066fd0 con 0x7fe51c06bca0 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51c06bca0 msgr2=0x7fe51c06e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51c06bca0 0x7fe51c06e160 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fe544156df0 tx=0x7fe52c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 msgr2=0x7fe544154ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 0x7fe544154ee0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe5340098c0 tx=0x7fe534004730 comp rx=0 tx=0).stop 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 shutdown_connections 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe51c06bca0 0x7fe51c06e160 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5440cf350 0x7fe5440b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5440b28b0 0x7fe544150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 --2- 172.21.15.2:0/1589792864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe54406cfc0 0x7fe544154ee0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 >> 172.21.15.2:0/1589792864 conn(0x7fe54406c4d0 msgr2=0x7fe5440707d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 shutdown_connections 2024-01-15T18:05:22.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:22.734+0000 7fe54a200640 1 -- 172.21.15.2:0/1589792864 wait complete. 2024-01-15T18:05:22.759 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone27422 2024-01-15T18:05:23.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.039+0000 7fd93e600640 1 Processor -- start 2024-01-15T18:05:23.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.040+0000 7fd93e600640 1 -- start start 2024-01-15T18:05:23.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.041+0000 7fd93e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93806d000 0x7fd9380b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.041+0000 7fd93e600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd93814b3b0 0x7fd9380702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.041+0000 7fd93e600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9380b25f0 con 0x7fd93814afd0 2024-01-15T18:05:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.041+0000 7fd93e600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd938070ff0 con 0x7fd93814b3b0 2024-01-15T18:05:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.041+0000 7fd93e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9380b28d0 con 0x7fd93806d000 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.042+0000 7fd937400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd93814afd0 0x7fd93806c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56918/0 (socket says 172.21.15.2:56918) 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.042+0000 7fd937400640 1 -- 172.21.15.2:0/1285814322 learned_addr learned my addr 172.21.15.2:0/1285814322 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.042+0000 7fd937e00640 1 --2- 172.21.15.2:0/1285814322 >> v2:172.21.15.137:3300/0 conn(0x7fd93814b3b0 0x7fd9380702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.043+0000 7fd936a00640 1 -- 172.21.15.2:0/1285814322 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583696012 0 0) 0x7fd9380b25f0 con 0x7fd93814afd0 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.043+0000 7fd936a00640 1 -- 172.21.15.2:0/1285814322 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd918003520 con 0x7fd93814afd0 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.043+0000 7fd937e00640 1 -- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93806d000 msgr2=0x7fd9380b33b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.043+0000 7fd937e00640 1 --2- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93806d000 0x7fd9380b33b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.043+0000 7fd937e00640 1 -- 172.21.15.2:0/1285814322 >> v1:172.21.15.2:6789/0 conn(0x7fd93814afd0 legacy=0x7fd93806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.043+0000 7fd937e00640 1 -- 172.21.15.2:0/1285814322 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9380b2bb0 con 0x7fd93814b3b0 2024-01-15T18:05:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.045+0000 7fd937e00640 1 --2- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93814b3b0 0x7fd9380702c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd930009d90 tx=0x7fd93002bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31e812002e4f81a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.045+0000 7fd936a00640 1 -- 172.21.15.2:0/1285814322 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93002cb60 con 0x7fd93814b3b0 2024-01-15T18:05:23.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.045+0000 7fd936a00640 1 -- 172.21.15.2:0/1285814322 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93002ccc0 con 0x7fd93814b3b0 2024-01-15T18:05:23.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.045+0000 7fd936a00640 1 -- 172.21.15.2:0/1285814322 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930034660 con 0x7fd93814b3b0 2024-01-15T18:05:23.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.046+0000 7fd93e600640 1 -- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93814b3b0 msgr2=0x7fd9380702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.046+0000 7fd93e600640 1 --2- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93814b3b0 0x7fd9380702c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fd930009d90 tx=0x7fd93002bdb0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.046+0000 7fd93e600640 1 -- 172.21.15.2:0/1285814322 shutdown_connections 2024-01-15T18:05:23.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.046+0000 7fd93e600640 1 --2- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93814b3b0 0x7fd9380702c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.046+0000 7fd93e600640 1 --2- 172.21.15.2:0/1285814322 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93806d000 0x7fd9380b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.046+0000 7fd93e600640 1 -- 172.21.15.2:0/1285814322 >> 172.21.15.2:0/1285814322 conn(0x7fd9380b45f0 msgr2=0x7fd9380b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:23.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.048+0000 7fd93e600640 1 -- 172.21.15.2:0/1285814322 shutdown_connections 2024-01-15T18:05:23.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.048+0000 7fd93e600640 1 -- 172.21.15.2:0/1285814322 wait complete. 2024-01-15T18:05:23.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.053+0000 7fd93e600640 1 Processor -- start 2024-01-15T18:05:23.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.054+0000 7fd93e600640 1 -- start start 2024-01-15T18:05:23.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.057+0000 7fd93e600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93806d000 0x7fd9381fa900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.057+0000 7fd93e600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 0x7fd9381f79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.057+0000 7fd93e600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93814b3b0 0x7fd938155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.057+0000 7fd93e600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938155d40 con 0x7fd93814afd0 2024-01-15T18:05:23.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.057+0000 7fd93e600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938156020 con 0x7fd93806d000 2024-01-15T18:05:23.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.057+0000 7fd93e600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd938156300 con 0x7fd93814b3b0 2024-01-15T18:05:23.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.058+0000 7fd92fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 0x7fd9381f79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.058+0000 7fd92fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 0x7fd9381f79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40076/0 (socket says 172.21.15.2:40076) 2024-01-15T18:05:23.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.058+0000 7fd92fe00640 1 -- 172.21.15.2:0/1173787842 learned_addr learned my addr 172.21.15.2:0/1173787842 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:23.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.059+0000 7fd937400640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93806d000 0x7fd9381fa900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.059+0000 7fd92fe00640 1 -- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93814b3b0 msgr2=0x7fd938155360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.059+0000 7fd92fe00640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93814b3b0 0x7fd938155360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.059+0000 7fd92fe00640 1 -- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93806d000 msgr2=0x7fd9381fa900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.059+0000 7fd92fe00640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93806d000 0x7fd9381fa900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.059+0000 7fd92fe00640 1 -- 172.21.15.2:0/1173787842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd930009a70 con 0x7fd93814afd0 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.060+0000 7fd92fe00640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 0x7fd9381f79f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd92000b3e0 tx=0x7fd92000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.061+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920011830 con 0x7fd93814afd0 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.061+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9200119b0 con 0x7fd93814afd0 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.061+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd920011d60 con 0x7fd93814afd0 2024-01-15T18:05:23.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.061+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9381565e0 con 0x7fd93814afd0 2024-01-15T18:05:23.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.061+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381f7f30 con 0x7fd93814afd0 2024-01-15T18:05:23.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.064+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938156b00 con 0x7fd93814afd0 2024-01-15T18:05:23.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.065+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fd92001a400 con 0x7fd93814afd0 2024-01-15T18:05:23.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.067+0000 7fd92f400640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd91006bca0 0x7fd91006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.067+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5765+0+0 (secure 0 0 0) 0x7fd9200887b0 con 0x7fd93814afd0 2024-01-15T18:05:23.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.067+0000 7fd937400640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd91006bca0 0x7fd91006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.070+0000 7fd937400640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd91006bca0 0x7fd91006e160 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd9380ce4f0 tx=0x7fd928007b80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.075+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd938156b00 con 0x7fd93814afd0 2024-01-15T18:05:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.501+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone27422", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9380702c0 con 0x7fd91006bca0 2024-01-15T18:05:23.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.533+0000 7fd92f400640 1 -- 172.21.15.2:0/1173787842 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd9380702c0 con 0x7fd91006bca0 2024-01-15T18:05:23.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.536+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd91006bca0 msgr2=0x7fd91006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.536+0000 7fd93e600640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd91006bca0 0x7fd91006e160 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd9380ce4f0 tx=0x7fd928007b80 comp rx=0 tx=0).stop 2024-01-15T18:05:23.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.536+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 msgr2=0x7fd9381f79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.536+0000 7fd93e600640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 0x7fd9381f79f0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd92000b3e0 tx=0x7fd92000b8a0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.537+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 shutdown_connections 2024-01-15T18:05:23.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.537+0000 7fd93e600640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd91006bca0 0x7fd91006e160 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.537+0000 7fd93e600640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd93814b3b0 0x7fd938155360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.537+0000 7fd93e600640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd93814afd0 0x7fd9381f79f0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.537+0000 7fd93e600640 1 --2- 172.21.15.2:0/1173787842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd93806d000 0x7fd9381fa900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.537+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 >> 172.21.15.2:0/1173787842 conn(0x7fd9380b45f0 msgr2=0x7fd938164a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:23.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.538+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 shutdown_connections 2024-01-15T18:05:23.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.538+0000 7fd93e600640 1 -- 172.21.15.2:0/1173787842 wait complete. 2024-01-15T18:05:23.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:23.571 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:05:23.659 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-01-15T18:05:23.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:05:23.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.952+0000 7fa748000640 1 Processor -- start 2024-01-15T18:05:23.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.954+0000 7fa748000640 1 -- start start 2024-01-15T18:05:23.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.957+0000 7fa748000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa74014afd0 0x7fa7400b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.957+0000 7fa748000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74006d000 0x7fa7400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.957+0000 7fa748000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7400b25f0 con 0x7fa74014b3b0 2024-01-15T18:05:23.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.957+0000 7fa748000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa740070ff0 con 0x7fa74014afd0 2024-01-15T18:05:23.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa744c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa74014b3b0 0x7fa74006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56928/0 (socket says 172.21.15.2:56928) 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa746000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74006d000 0x7fa7400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa744c00640 1 -- 172.21.15.2:0/1722155194 learned_addr learned my addr 172.21.15.2:0/1722155194 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa745600640 1 --2- 172.21.15.2:0/1722155194 >> v2:172.21.15.137:3300/0 conn(0x7fa74014afd0 0x7fa7400b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa748000640 1 -- 172.21.15.2:0/1722155194 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400b28d0 con 0x7fa74006d000 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa745600640 1 -- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74006d000 msgr2=0x7fa7400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa745600640 1 --2- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74006d000 0x7fa7400702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa745600640 1 -- 172.21.15.2:0/1722155194 >> v1:172.21.15.2:6789/0 conn(0x7fa74014b3b0 legacy=0x7fa74006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa73fe00640 1 -- 172.21.15.2:0/1722155194 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484191881 0 0) 0x7fa7400b25f0 con 0x7fa74014b3b0 2024-01-15T18:05:23.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.958+0000 7fa745600640 1 -- 172.21.15.2:0/1722155194 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400b2bb0 con 0x7fa74014afd0 2024-01-15T18:05:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.961+0000 7fa745600640 1 --2- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74014afd0 0x7fa7400b33b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa7300097a0 tx=0x7fa73002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d8efe2d576773af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.961+0000 7fa73fe00640 1 -- 172.21.15.2:0/1722155194 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730031780 con 0x7fa74014afd0 2024-01-15T18:05:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.961+0000 7fa73fe00640 1 -- 172.21.15.2:0/1722155194 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7300318e0 con 0x7fa74014afd0 2024-01-15T18:05:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.961+0000 7fa73fe00640 1 -- 172.21.15.2:0/1722155194 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730031c70 con 0x7fa74014afd0 2024-01-15T18:05:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.962+0000 7fa748000640 1 -- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74014afd0 msgr2=0x7fa7400b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.962+0000 7fa748000640 1 --2- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74014afd0 0x7fa7400b33b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa7300097a0 tx=0x7fa73002b820 comp rx=0 tx=0).stop 2024-01-15T18:05:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.963+0000 7fa748000640 1 -- 172.21.15.2:0/1722155194 shutdown_connections 2024-01-15T18:05:23.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.963+0000 7fa748000640 1 --2- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74006d000 0x7fa7400702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.963+0000 7fa748000640 1 --2- 172.21.15.2:0/1722155194 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74014afd0 0x7fa7400b33b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.963+0000 7fa748000640 1 -- 172.21.15.2:0/1722155194 >> 172.21.15.2:0/1722155194 conn(0x7fa7400b45f0 msgr2=0x7fa7400b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.963+0000 7fa748000640 1 -- 172.21.15.2:0/1722155194 shutdown_connections 2024-01-15T18:05:23.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.963+0000 7fa748000640 1 -- 172.21.15.2:0/1722155194 wait complete. 2024-01-15T18:05:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.965+0000 7fa748000640 1 Processor -- start 2024-01-15T18:05:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.966+0000 7fa748000640 1 -- start start 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.967+0000 7fa748000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74006d000 0x7fa7401fa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.967+0000 7fa748000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74014afd0 0x7fa7400aac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.967+0000 7fa748000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74014b3b0 0x7fa7400b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.967+0000 7fa748000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401fb4a0 con 0x7fa74014b3b0 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.967+0000 7fa748000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400ab7d0 con 0x7fa74006d000 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.967+0000 7fa748000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7401fac00 con 0x7fa74014afd0 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.968+0000 7fa745600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74006d000 0x7fa7401fa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.968+0000 7fa744c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74014afd0 0x7fa7400aac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.968+0000 7fa746000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74014b3b0 0x7fa7400b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.968+0000 7fa746000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74014b3b0 0x7fa7400b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40086/0 (socket says 172.21.15.2:40086) 2024-01-15T18:05:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.968+0000 7fa746000640 1 -- 172.21.15.2:0/3599540647 learned_addr learned my addr 172.21.15.2:0/3599540647 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.969+0000 7fa745600640 1 -- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74014afd0 msgr2=0x7fa7400aac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.969+0000 7fa745600640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74014afd0 0x7fa7400aac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.969+0000 7fa745600640 1 -- 172.21.15.2:0/3599540647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74014b3b0 msgr2=0x7fa7400b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.969+0000 7fa745600640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74014b3b0 0x7fa7400b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.969+0000 7fa745600640 1 -- 172.21.15.2:0/3599540647 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa730009400 con 0x7fa74006d000 2024-01-15T18:05:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.970+0000 7fa745600640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74006d000 0x7fa7401fa540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa730033000 tx=0x7fa7300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:23.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.970+0000 7fa744c00640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74014afd0 0x7fa7400aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:23.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.971+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730032820 con 0x7fa74006d000 2024-01-15T18:05:23.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.972+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa730032980 con 0x7fa74006d000 2024-01-15T18:05:23.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.972+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa730032d30 con 0x7fa74006d000 2024-01-15T18:05:23.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.972+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7401faee0 con 0x7fa74006d000 2024-01-15T18:05:23.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.972+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7400a8b90 con 0x7fa74006d000 2024-01-15T18:05:23.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.974+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7400686f0 con 0x7fa74006d000 2024-01-15T18:05:23.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.978+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fa730005990 con 0x7fa74006d000 2024-01-15T18:05:23.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.986+0000 7fa73d600640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa71806bc80 0x7fa71806e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:23.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.986+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fa7300aa450 con 0x7fa74006d000 2024-01-15T18:05:23.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.987+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7300ad210 con 0x7fa74006d000 2024-01-15T18:05:23.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.988+0000 7fa744c00640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa71806bc80 0x7fa71806e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:23.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:23.989+0000 7fa744c00640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa71806bc80 0x7fa71806e140 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa7400a89f0 tx=0x7fa728007ca0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:24.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:24.403+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7400cfc90 con 0x7fa71806bc80 2024-01-15T18:05:24.438 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T18:05:24.435+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:05:24.538 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:05:26.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:26.475+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fa730076ee0 con 0x7fa74006d000 2024-01-15T18:05:27.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.492+0000 7fa73d600640 1 -- 172.21.15.2:0/3599540647 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fa7400cfc90 con 0x7fa71806bc80 2024-01-15T18:05:27.496 INFO:teuthology.orchestra.run.smithi002.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:05:27.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa71806bc80 msgr2=0x7fa71806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa71806bc80 0x7fa71806e140 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa7400a89f0 tx=0x7fa728007ca0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74006d000 msgr2=0x7fa7401fa540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74006d000 0x7fa7401fa540 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa730033000 tx=0x7fa7300046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 shutdown_connections 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa71806bc80 0x7fa71806e140 secure :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa7400a89f0 tx=0x7fa728007ca0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa74014b3b0 0x7fa7400b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa74014afd0 0x7fa7400aac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 --2- 172.21.15.2:0/3599540647 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa74006d000 0x7fa7401fa540 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:27.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 >> 172.21.15.2:0/3599540647 conn(0x7fa7400b45f0 msgr2=0x7fa740148420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:27.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.497+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 shutdown_connections 2024-01-15T18:05:27.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:27.498+0000 7fa748000640 1 -- 172.21.15.2:0/3599540647 wait complete. 2024-01-15T18:05:27.533 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:05:27.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:05:27.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:05:27.806 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:05:27.806 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:05:27.806 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:27.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:28.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.101+0000 7f5e11c00640 1 Processor -- start 2024-01-15T18:05:28.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.101+0000 7f5e11c00640 1 -- start start 2024-01-15T18:05:28.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.102+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 0x7f5e0c06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e11c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5e0c0b2250 0x7f5e0c0cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e11c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c06e470 con 0x7f5e0c06e010 2024-01-15T18:05:28.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e11c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c0b3520 con 0x7f5e0c0b2250 2024-01-15T18:05:28.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e11c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0ccbe0 con 0x7f5e0c143810 2024-01-15T18:05:28.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e03e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5e0c06e010 0x7f5e0c0b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56952/0 (socket says 172.21.15.2:56952) 2024-01-15T18:05:28.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e03e00640 1 -- 172.21.15.2:0/155384611 learned_addr learned my addr 172.21.15.2:0/155384611 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:28.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e11200640 1 --2- 172.21.15.2:0/155384611 >> v2:172.21.15.137:3300/0 conn(0x7f5e0c0b2250 0x7f5e0c0cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e0be00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 0x7f5e0c06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.103+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170945431 0 0) 0x7f5df4001110 con 0x7f5e0c06e010 2024-01-15T18:05:28.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec003520 con 0x7f5e0c06e010 2024-01-15T18:05:28.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780993942 0 0) 0x7f5dec003520 con 0x7f5e0c06e010 2024-01-15T18:05:28.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 msgr2=0x7f5e0c06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 0x7f5e0c06dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 >> v2:172.21.15.137:3300/0 conn(0x7f5e0c0b2250 msgr2=0x7f5e0c0cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 --2- 172.21.15.2:0/155384611 >> v2:172.21.15.137:3300/0 conn(0x7f5e0c0b2250 0x7f5e0c0cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c143bf0 con 0x7f5e0c06e010 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5df4001800 con 0x7f5e0c06e010 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 >> v1:172.21.15.2:6789/0 conn(0x7f5e0c06e010 legacy=0x7f5e0c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 0x7f5dec003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5dec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dec004d00 0x7f5dec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0b3520 con 0x7f5dec003ef0 2024-01-15T18:05:28.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c0ccbe0 con 0x7f5dec004d00 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec00dbe0 con 0x7f5e0c143810 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.104+0000 7f5e03e00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5dec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e0be00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 0x7f5dec003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e11200640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dec004d00 0x7f5dec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e03e00640 1 -- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 msgr2=0x7f5dec003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e03e00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c143810 0x7f5dec003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e03e00640 1 -- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dec004d00 msgr2=0x7f5dec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e03e00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dec004d00 0x7f5dec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e03e00640 1 -- 172.21.15.2:0/155384611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec003520 con 0x7f5dec003ef0 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.105+0000 7f5e11200640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5dec004d00 0x7f5dec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.106+0000 7f5e03e00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5dec0047c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5df400c240 tx=0x7f5dec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ace5e0899da069 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.106+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4005ad0 con 0x7f5dec003ef0 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.106+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df4005c50 con 0x7f5dec003ef0 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.106+0000 7f5e0b400640 1 -- 172.21.15.2:0/155384611 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df402c8d0 con 0x7f5dec003ef0 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11c00640 1 -- 172.21.15.2:0/155384611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 msgr2=0x7f5dec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11c00640 1 --2- 172.21.15.2:0/155384611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5dec0047c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f5df400c240 tx=0x7f5dec0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11200640 1 -- 172.21.15.2:0/155384611 reap_dead start 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11c00640 1 -- 172.21.15.2:0/155384611 shutdown_connections 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11c00640 1 -- 172.21.15.2:0/155384611 >> 172.21.15.2:0/155384611 conn(0x7f5e0c0b0840 msgr2=0x7f5e0c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11c00640 1 -- 172.21.15.2:0/155384611 shutdown_connections 2024-01-15T18:05:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.107+0000 7f5e11c00640 1 -- 172.21.15.2:0/155384611 wait complete. 2024-01-15T18:05:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.109+0000 7f5e11c00640 1 Processor -- start 2024-01-15T18:05:28.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.109+0000 7f5e11c00640 1 -- start start 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5e0c1de590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e0c1dfdc0 0x7f5e0c1dc460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c1dc9a0 0x7f5e0c1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e11c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1df9a0 con 0x7f5dec003ef0 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e11c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1dce50 con 0x7f5e0c1dfdc0 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e11c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1dead0 con 0x7f5e0c1dc9a0 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.110+0000 7f5e0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5e0c1de590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5e0c1de590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40106/0 (socket says 172.21.15.2:40106) 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 -- 172.21.15.2:0/3178985328 learned_addr learned my addr 172.21.15.2:0/3178985328 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:28.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e11200640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c1dc9a0 0x7f5e0c1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 -- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c1dc9a0 msgr2=0x7f5e0c1f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c1dc9a0 0x7f5e0c1f2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e03e00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e0c1dfdc0 0x7f5e0c1dc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 -- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e0c1dfdc0 msgr2=0x7f5e0c1dc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e0c1dfdc0 0x7f5e0c1dc460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 -- 172.21.15.2:0/3178985328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c1ded20 con 0x7f5dec003ef0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.111+0000 7f5e0be00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5e0c1de590 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc011630 tx=0x7f5dfc011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.112+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00f920 con 0x7f5dec003ef0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.112+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dfc00fa80 con 0x7f5dec003ef0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.112+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c1df000 con 0x7f5dec003ef0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.112+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dfc00fdf0 con 0x7f5dec003ef0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.112+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1df5c0 con 0x7f5dec003ef0 2024-01-15T18:05:28.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.113+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e0c074770 con 0x7f5dec003ef0 2024-01-15T18:05:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.114+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5dfc015460 con 0x7f5dec003ef0 2024-01-15T18:05:28.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.115+0000 7f5e08c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5de406bbc0 0x7f5de406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.115+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f5dfc088ca0 con 0x7f5dec003ef0 2024-01-15T18:05:28.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.116+0000 7f5e03e00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5de406bbc0 0x7f5de406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.117+0000 7f5e03e00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5de406bbc0 0x7f5de406e080 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5df4034000 tx=0x7f5df4007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.120+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5dfc055560 con 0x7f5dec003ef0 2024-01-15T18:05:28.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.382+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e0c0cf7f0 con 0x7f5dec003ef0 2024-01-15T18:05:28.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.385+0000 7f5e08c00640 1 -- 172.21.15.2:0/3178985328 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+13055 (secure 0 0 0) 0x7f5dfc058b80 con 0x7f5dec003ef0 2024-01-15T18:05:28.392 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:28.393 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":65,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:27.460948+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/2855093754":"2024-01-16T18:05:24.435299+0000","172.21.15.2:6834/2855093754":"2024-01-16T18:05:24.435299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:28.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.389+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5de406bbc0 msgr2=0x7f5de406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5de406bbc0 0x7f5de406e080 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f5df4034000 tx=0x7f5df4007000 comp rx=0 tx=0).stop 2024-01-15T18:05:28.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 msgr2=0x7f5e0c1de590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5e0c1de590 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc011630 tx=0x7f5dfc011af0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 shutdown_connections 2024-01-15T18:05:28.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5de406bbc0 0x7f5de406e080 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e0c1dc9a0 0x7f5e0c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e0c1dfdc0 0x7f5e0c1dc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 --2- 172.21.15.2:0/3178985328 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5dec003ef0 0x7f5e0c1de590 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 >> 172.21.15.2:0/3178985328 conn(0x7f5e0c0b0840 msgr2=0x7f5e0c0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 shutdown_connections 2024-01-15T18:05:28.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.390+0000 7f5e11c00640 1 -- 172.21.15.2:0/3178985328 wait complete. 2024-01-15T18:05:28.422 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:28.422 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:28.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.709+0000 7fc26b400640 1 Processor -- start 2024-01-15T18:05:28.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.710+0000 7fc26b400640 1 -- start start 2024-01-15T18:05:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.711+0000 7fc26b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2641439f0 0x7fc2640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.711+0000 7fc26b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26406c8f0 0x7fc26406cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.711+0000 7fc26b400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc26406d280 con 0x7fc2640b3420 2024-01-15T18:05:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.711+0000 7fc26b400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc26415dd70 con 0x7fc2641439f0 2024-01-15T18:05:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.711+0000 7fc26b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2640b38b0 con 0x7fc26406c8f0 2024-01-15T18:05:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc269400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26406c8f0 0x7fc26406cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc263e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fc2640b3420 0x7fc2640cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:52498/0 (socket says 172.21.15.2:52498) 2024-01-15T18:05:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc263e00640 1 -- 172.21.15.2:0/2395461053 learned_addr learned my addr 172.21.15.2:0/2395461053 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc268a00640 1 --2- 172.21.15.2:0/2395461053 >> v2:172.21.15.137:3300/0 conn(0x7fc2641439f0 0x7fc2640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc263400640 1 -- 172.21.15.2:0/2395461053 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940422748 0 0) 0x7fc24c001110 con 0x7fc2640b3420 2024-01-15T18:05:28.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc263400640 1 -- 172.21.15.2:0/2395461053 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc244003520 con 0x7fc2640b3420 2024-01-15T18:05:28.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc269400640 1 -- 172.21.15.2:0/2395461053 >> v2:172.21.15.137:3300/0 conn(0x7fc2641439f0 msgr2=0x7fc2640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:28.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc269400640 1 --2- 172.21.15.2:0/2395461053 >> v2:172.21.15.137:3300/0 conn(0x7fc2641439f0 0x7fc2640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc269400640 1 -- 172.21.15.2:0/2395461053 >> v1:172.21.15.137:6789/0 conn(0x7fc2640b3420 legacy=0x7fc2640cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.712+0000 7fc269400640 1 -- 172.21.15.2:0/2395461053 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640711b0 con 0x7fc26406c8f0 2024-01-15T18:05:28.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.714+0000 7fc269400640 1 --2- 172.21.15.2:0/2395461053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26406c8f0 0x7fc26406cd40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009d90 tx=0x7fc25c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ad4a8ee33eb4e07 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.714+0000 7fc263400640 1 -- 172.21.15.2:0/2395461053 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c02cc50 con 0x7fc26406c8f0 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.714+0000 7fc263400640 1 -- 172.21.15.2:0/2395461053 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c02cdb0 con 0x7fc26406c8f0 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.714+0000 7fc263400640 1 -- 172.21.15.2:0/2395461053 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c034660 con 0x7fc26406c8f0 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 -- 172.21.15.2:0/2395461053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26406c8f0 msgr2=0x7fc26406cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 --2- 172.21.15.2:0/2395461053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26406c8f0 0x7fc26406cd40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009d90 tx=0x7fc25c02bea0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 -- 172.21.15.2:0/2395461053 shutdown_connections 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 --2- 172.21.15.2:0/2395461053 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc26406c8f0 0x7fc26406cd40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 --2- 172.21.15.2:0/2395461053 >> v2:172.21.15.137:3300/0 conn(0x7fc2641439f0 0x7fc2640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 -- 172.21.15.2:0/2395461053 >> 172.21.15.2:0/2395461053 conn(0x7fc2640b45f0 msgr2=0x7fc2640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 -- 172.21.15.2:0/2395461053 shutdown_connections 2024-01-15T18:05:28.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.715+0000 7fc26b400640 1 -- 172.21.15.2:0/2395461053 wait complete. 2024-01-15T18:05:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.717+0000 7fc26b400640 1 Processor -- start 2024-01-15T18:05:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.717+0000 7fc26b400640 1 -- start start 2024-01-15T18:05:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc26b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26406c8f0 0x7fc26413b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc26b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 0x7fc2640cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc26b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2641439f0 0x7fc2640cf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc26b400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26413eb60 con 0x7fc2640b3420 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc26b400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26413dfe0 con 0x7fc26406c8f0 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc26b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26413e2c0 con 0x7fc2641439f0 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc263e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 0x7fc2640cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc263e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 0x7fc2640cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40122/0 (socket says 172.21.15.2:40122) 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.718+0000 7fc263e00640 1 -- 172.21.15.2:0/3201688959 learned_addr learned my addr 172.21.15.2:0/3201688959 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc269400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2641439f0 0x7fc2640cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc268a00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26406c8f0 0x7fc26413b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc263e00640 1 -- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2641439f0 msgr2=0x7fc2640cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc263e00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2641439f0 0x7fc2640cf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc263e00640 1 -- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26406c8f0 msgr2=0x7fc26413b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc263e00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26406c8f0 0x7fc26413b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc263e00640 1 -- 172.21.15.2:0/3201688959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c009a70 con 0x7fc2640b3420 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc269400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2641439f0 0x7fc2640cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc263e00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 0x7fc2640cece0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c730 tx=0x7fc24c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c005ab0 con 0x7fc2640b3420 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc24c005c10 con 0x7fc2640b3420 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24c001d50 con 0x7fc2640b3420 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.719+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26413e5a0 con 0x7fc2640b3420 2024-01-15T18:05:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.720+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26413be90 con 0x7fc2640b3420 2024-01-15T18:05:28.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.721+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2640686f0 con 0x7fc2640b3420 2024-01-15T18:05:28.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.722+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc24c007070 con 0x7fc2640b3420 2024-01-15T18:05:28.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.723+0000 7fc260c00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23c06bbf0 0x7fc23c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:28.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.723+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fc24c087cf0 con 0x7fc2640b3420 2024-01-15T18:05:28.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.724+0000 7fc268a00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23c06bbf0 0x7fc23c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:28.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.725+0000 7fc268a00640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23c06bbf0 0x7fc23c06e0b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc254009a30 tx=0x7fc254008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:28.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.726+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc24c0545f0 con 0x7fc2640b3420 2024-01-15T18:05:28.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.911+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc26406d280 con 0x7fc2640b3420 2024-01-15T18:05:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.912+0000 7fc260c00640 1 -- 172.21.15.2:0/3201688959 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc26406d280 con 0x7fc2640b3420 2024-01-15T18:05:28.915 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23c06bbf0 msgr2=0x7fc23c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23c06bbf0 0x7fc23c06e0b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc254009a30 tx=0x7fc254008040 comp rx=0 tx=0).stop 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 msgr2=0x7fc2640cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 0x7fc2640cece0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fc24c00c730 tx=0x7fc24c00cbf0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 shutdown_connections 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc23c06bbf0 0x7fc23c06e0b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2641439f0 0x7fc2640cf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2640b3420 0x7fc2640cece0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 --2- 172.21.15.2:0/3201688959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc26406c8f0 0x7fc26413b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 >> 172.21.15.2:0/3201688959 conn(0x7fc2640b45f0 msgr2=0x7fc264071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 shutdown_connections 2024-01-15T18:05:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:28.914+0000 7fc26b400640 1 -- 172.21.15.2:0/3201688959 wait complete. 2024-01-15T18:05:28.936 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:29.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.142+0000 7f5804800640 1 Processor -- start 2024-01-15T18:05:29.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.143+0000 7f5804800640 1 -- start start 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.144+0000 7f5804800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57fc144b80 0x7f57fc0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.144+0000 7f5804800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc0b30a0 0x7f57fc06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.144+0000 7f5804800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57fc0b35a0 con 0x7f57fc06cdf0 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.144+0000 7f5804800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc0b4220 con 0x7f57fc144b80 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.144+0000 7f5804800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc06d280 con 0x7f57fc0b30a0 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57fc144b80 0x7f57fc0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57fc144b80 0x7f57fc0cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40138/0 (socket says 172.21.15.2:40138) 2024-01-15T18:05:29.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 -- 172.21.15.2:0/1615577841 learned_addr learned my addr 172.21.15.2:0/1615577841 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801400640 1 --2- 172.21.15.2:0/1615577841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc0b30a0 0x7f57fc06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 -- 172.21.15.2:0/1615577841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc0b30a0 msgr2=0x7f57fc06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 --2- 172.21.15.2:0/1615577841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc0b30a0 0x7f57fc06c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 -- 172.21.15.2:0/1615577841 >> v1:172.21.15.2:6789/0 conn(0x7f57fc06cdf0 legacy=0x7f57fc0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.145+0000 7f5801e00640 1 -- 172.21.15.2:0/1615577841 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc0b3880 con 0x7f57fc144b80 2024-01-15T18:05:29.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.146+0000 7f5801e00640 1 --2- 172.21.15.2:0/1615577841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc144b80 0x7f57fc0cbae0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009870 tx=0x7f57ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc7a04f4ebe7aa2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5800a00640 1 -- 172.21.15.2:0/1615577841 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec031640 con 0x7f57fc144b80 2024-01-15T18:05:29.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5800a00640 1 -- 172.21.15.2:0/1615577841 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ec0317a0 con 0x7f57fc144b80 2024-01-15T18:05:29.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5800a00640 1 -- 172.21.15.2:0/1615577841 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ec031b30 con 0x7f57fc144b80 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5804800640 1 -- 172.21.15.2:0/1615577841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc144b80 msgr2=0x7f57fc0cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5804800640 1 --2- 172.21.15.2:0/1615577841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc144b80 0x7f57fc0cbae0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009870 tx=0x7f57ec02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5804800640 1 -- 172.21.15.2:0/1615577841 shutdown_connections 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5804800640 1 --2- 172.21.15.2:0/1615577841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc0b30a0 0x7f57fc06c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5804800640 1 --2- 172.21.15.2:0/1615577841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc144b80 0x7f57fc0cbae0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.147+0000 7f5804800640 1 -- 172.21.15.2:0/1615577841 >> 172.21.15.2:0/1615577841 conn(0x7f57fc0b46d0 msgr2=0x7f57fc0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.148+0000 7f5804800640 1 -- 172.21.15.2:0/1615577841 shutdown_connections 2024-01-15T18:05:29.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.148+0000 7f5804800640 1 -- 172.21.15.2:0/1615577841 wait complete. 2024-01-15T18:05:29.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.149+0000 7f5804800640 1 Processor -- start 2024-01-15T18:05:29.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.149+0000 7f5804800640 1 -- start start 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.150+0000 7f5804800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc06cdf0 0x7f57fc13ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.150+0000 7f5804800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 0x7f57fc13a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.150+0000 7f5804800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57fc144b80 0x7f57fc13c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.150+0000 7f5804800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc13cd30 con 0x7f57fc0b30a0 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.150+0000 7f5804800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc13d010 con 0x7f57fc144b80 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.150+0000 7f5804800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc13d2f0 con 0x7f57fc06cdf0 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 0x7f57fc13a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 0x7f57fc13a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40140/0 (socket says 172.21.15.2:40140) 2024-01-15T18:05:29.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 -- 172.21.15.2:0/1601767963 learned_addr learned my addr 172.21.15.2:0/1601767963 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:29.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801e00640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc06cdf0 0x7f57fc13ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 -- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc06cdf0 msgr2=0x7f57fc13ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc06cdf0 0x7f57fc13ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 -- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57fc144b80 msgr2=0x7f57fc13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57fc144b80 0x7f57fc13c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 -- 172.21.15.2:0/1601767963 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec0094d0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.151+0000 7f5801400640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 0x7f57fc13a850 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f57e400d800 tx=0x7f57e400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.152+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e400b6c0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.152+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e400b820 con 0x7f57fc0b30a0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.152+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e400bbb0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.152+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc13d5d0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.152+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc13e310 con 0x7f57fc0b30a0 2024-01-15T18:05:29.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.153+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc13e4d0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.154+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f57e400bd10 con 0x7f57fc0b30a0 2024-01-15T18:05:29.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.155+0000 7f57fa000640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57d406bbf0 0x7f57d406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.155+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f57e4088c60 con 0x7f57fc0b30a0 2024-01-15T18:05:29.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.155+0000 7f5801e00640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57d406bbf0 0x7f57d406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.157+0000 7f5801e00640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57d406bbf0 0x7f57d406e0b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f57ec036000 tx=0x7f57ec03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.159+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57e40552a0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.358+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f57fc0686f0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.359+0000 7f57fa000640 1 -- 172.21.15.2:0/1601767963 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v65) v1 ==== 80+0+120 (secure 0 0 0) 0x7f57e40588c0 con 0x7f57fc0b30a0 2024-01-15T18:05:29.362 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6835/2855093754 2024-01-16T18:05:24.435299+0000 2024-01-15T18:05:29.363 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/2855093754 2024-01-16T18:05:24.435299+0000 2024-01-15T18:05:29.363 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:05:29.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57d406bbf0 msgr2=0x7f57d406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57d406bbf0 0x7f57d406e0b0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f57ec036000 tx=0x7f57ec03c040 comp rx=0 tx=0).stop 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 msgr2=0x7f57fc13a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 0x7f57fc13a850 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f57e400d800 tx=0x7f57e400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 shutdown_connections 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f57d406bbf0 0x7f57d406e0b0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57fc144b80 0x7f57fc13c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57fc0b30a0 0x7f57fc13a850 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 --2- 172.21.15.2:0/1601767963 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57fc06cdf0 0x7f57fc13ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.361+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 >> 172.21.15.2:0/1601767963 conn(0x7f57fc0b46d0 msgr2=0x7f57fc0af8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.362+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 shutdown_connections 2024-01-15T18:05:29.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.362+0000 7f5804800640 1 -- 172.21.15.2:0/1601767963 wait complete. 2024-01-15T18:05:29.383 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:05:29.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:29.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:29.483 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:05:29.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:29.483 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:05:29.494 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[56990]: fuse finished with error 0 and tester_r 0 2024-01-15T18:05:29.691 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:05:29.691 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:29.747 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:29.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.973+0000 7f03fda00640 1 Processor -- start 2024-01-15T18:05:29.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.973+0000 7f03fda00640 1 -- start start 2024-01-15T18:05:29.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03fda00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f03f813ce30 0x7f03f815dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03fda00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f03f815e170 0x7f03f80cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03fda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f80cdfa0 0x7f03f813f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03fda00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f815e620 con 0x7f03f813ce30 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03fda00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f80aeab0 con 0x7f03f815e170 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03fda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f8070380 con 0x7f03f80cdfa0 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03f6a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f03f813ce30 0x7f03f815dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03f6a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f03f813ce30 0x7f03f815dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40146/0 (socket says 172.21.15.2:40146) 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.975+0000 7f03f6a00640 1 -- 172.21.15.2:0/65045754 learned_addr learned my addr 172.21.15.2:0/65045754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:29.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.976+0000 7f03ede00640 1 --2- 172.21.15.2:0/65045754 >> v2:172.21.15.137:3300/0 conn(0x7f03f815e170 0x7f03f80cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.976+0000 7f03f6a00640 1 -- 172.21.15.2:0/65045754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f80cdfa0 msgr2=0x7f03f813f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:29.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.976+0000 7f03f6a00640 1 --2- 172.21.15.2:0/65045754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f80cdfa0 0x7f03f813f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.976+0000 7f03f6a00640 1 -- 172.21.15.2:0/65045754 >> v2:172.21.15.137:3300/0 conn(0x7f03f815e170 msgr2=0x7f03f80cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:29.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.976+0000 7f03f6a00640 1 --2- 172.21.15.2:0/65045754 >> v2:172.21.15.137:3300/0 conn(0x7f03f815e170 0x7f03f80cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.976+0000 7f03f6a00640 1 -- 172.21.15.2:0/65045754 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f80ce6d0 con 0x7f03f813ce30 2024-01-15T18:05:29.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.977+0000 7f03f6a00640 1 --2- 172.21.15.2:0/65045754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f813ce30 0x7f03f815dc30 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f03f80ae550 tx=0x7f03e802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b9ea5678ab6718 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.977+0000 7f03f6000640 1 -- 172.21.15.2:0/65045754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e802ee90 con 0x7f03f813ce30 2024-01-15T18:05:29.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.977+0000 7f03f6000640 1 -- 172.21.15.2:0/65045754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8036440 con 0x7f03f813ce30 2024-01-15T18:05:29.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.977+0000 7f03f6000640 1 -- 172.21.15.2:0/65045754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e80367d0 con 0x7f03f813ce30 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 -- 172.21.15.2:0/65045754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f813ce30 msgr2=0x7f03f815dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 --2- 172.21.15.2:0/65045754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f813ce30 0x7f03f815dc30 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f03f80ae550 tx=0x7f03e802e0e0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 -- 172.21.15.2:0/65045754 shutdown_connections 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 --2- 172.21.15.2:0/65045754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f80cdfa0 0x7f03f813f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 --2- 172.21.15.2:0/65045754 >> v2:172.21.15.137:3300/0 conn(0x7f03f815e170 0x7f03f80cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 --2- 172.21.15.2:0/65045754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f813ce30 0x7f03f815dc30 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 -- 172.21.15.2:0/65045754 >> 172.21.15.2:0/65045754 conn(0x7f03f806da20 msgr2=0x7f03f806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 -- 172.21.15.2:0/65045754 shutdown_connections 2024-01-15T18:05:29.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.978+0000 7f03fda00640 1 -- 172.21.15.2:0/65045754 wait complete. 2024-01-15T18:05:29.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.980+0000 7f03fda00640 1 Processor -- start 2024-01-15T18:05:29.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.980+0000 7f03fda00640 1 -- start start 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03fda00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 0x7f03f81f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03fda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f813ce30 0x7f03f813d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03fda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03f815e170 0x7f03f813ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03fda00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f813db10 con 0x7f03f80cdfa0 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03fda00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f813ddf0 con 0x7f03f815e170 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03fda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f813e0d0 con 0x7f03f813ce30 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03f6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 0x7f03f81f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03f6a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 0x7f03f81f2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40152/0 (socket says 172.21.15.2:40152) 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.981+0000 7f03f6a00640 1 -- 172.21.15.2:0/232230970 learned_addr learned my addr 172.21.15.2:0/232230970 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03f6a00640 1 -- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f813ce30 msgr2=0x7f03f813d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03f6a00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f813ce30 0x7f03f813d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03f6a00640 1 -- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03f815e170 msgr2=0x7f03f813ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03f6a00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03f815e170 0x7f03f813ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03f6a00640 1 -- 172.21.15.2:0/232230970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e800b770 con 0x7f03f80cdfa0 2024-01-15T18:05:29.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03f6a00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 0x7f03f81f2d40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f03e802e120 tx=0x7f03e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e80348d0 con 0x7f03f80cdfa0 2024-01-15T18:05:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e8034a50 con 0x7f03f80cdfa0 2024-01-15T18:05:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e8034e00 con 0x7f03f80cdfa0 2024-01-15T18:05:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.982+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f813e3b0 con 0x7f03f80cdfa0 2024-01-15T18:05:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.983+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f81f3280 con 0x7f03f80cdfa0 2024-01-15T18:05:29.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.984+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f813e8d0 con 0x7f03f80cdfa0 2024-01-15T18:05:29.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.985+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f03e80403e0 con 0x7f03f80cdfa0 2024-01-15T18:05:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.985+0000 7f03ef400640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03d006bbc0 0x7f03d006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.986+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f03e80aa9d0 con 0x7f03f80cdfa0 2024-01-15T18:05:29.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.986+0000 7f03ede00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03d006bbc0 0x7f03d006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:29.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.987+0000 7f03ede00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03d006bbc0 0x7f03d006e080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f03e0006fd0 tx=0x7f03e0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:29.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:29.990+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f03f813e8d0 con 0x7f03f80cdfa0 2024-01-15T18:05:30.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.186+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03f813e8d0 con 0x7f03f80cdfa0 2024-01-15T18:05:30.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.187+0000 7f03ef400640 1 -- 172.21.15.2:0/232230970 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+13055 (secure 0 0 0) 0x7f03f813e8d0 con 0x7f03f80cdfa0 2024-01-15T18:05:30.191 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:30.191 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":65,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:27.460948+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/2855093754":"2024-01-16T18:05:24.435299+0000","172.21.15.2:6834/2855093754":"2024-01-16T18:05:24.435299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:30.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.190+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03d006bbc0 msgr2=0x7f03d006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.190+0000 7f03fda00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03d006bbc0 0x7f03d006e080 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f03e0006fd0 tx=0x7f03e0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:30.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.190+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 msgr2=0x7f03f81f2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.190+0000 7f03fda00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 0x7f03f81f2d40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f03e802e120 tx=0x7f03e80046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 shutdown_connections 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f03d006bbc0 0x7f03d006e080 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f03f815e170 0x7f03f813ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f03f813ce30 0x7f03f813d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 --2- 172.21.15.2:0/232230970 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f03f80cdfa0 0x7f03f81f2d40 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 >> 172.21.15.2:0/232230970 conn(0x7f03f806da20 msgr2=0x7f03f8151760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 shutdown_connections 2024-01-15T18:05:30.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.191+0000 7f03fda00640 1 -- 172.21.15.2:0/232230970 wait complete. 2024-01-15T18:05:30.215 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:30.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:30.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.437+0000 7f24e2000640 1 Processor -- start 2024-01-15T18:05:30.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.437+0000 7f24e2000640 1 -- start start 2024-01-15T18:05:30.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.438+0000 7f24e2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 0x7f24dc0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.438+0000 7f24e2000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc0d1250 con 0x7f24dc0d1c50 2024-01-15T18:05:30.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.438+0000 7f24e2000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc0ce8e0 con 0x7f24dc0d1870 2024-01-15T18:05:30.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.438+0000 7f24e2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0ce6d0 con 0x7f24dc0b2210 2024-01-15T18:05:30.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.438+0000 7f24e1600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f24dc0d1c50 0x7f24dc0b26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56964/0 (socket says 172.21.15.2:56964) 2024-01-15T18:05:30.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.438+0000 7f24e1600640 1 -- 172.21.15.2:0/1409008167 learned_addr learned my addr 172.21.15.2:0/1409008167 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24dbe00640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 0x7f24dc0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164495402 0 0) 0x7f24dc0d1250 con 0x7f24dc0d1c50 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24bc003520 con 0x7f24dc0d1c50 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 868173010 0 0) 0x7f24bc003520 con 0x7f24dc0d1c50 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 msgr2=0x7f24dc0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 0x7f24dc0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 >> v1:172.21.15.137:6789/0 conn(0x7f24dc0d1870 legacy=0x7f24dc15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0cebc0 con 0x7f24dc0d1c50 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f24d40019f0 con 0x7f24dc0d1c50 2024-01-15T18:05:30.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 >> v1:172.21.15.2:6789/0 conn(0x7f24dc0d1c50 legacy=0x7f24dc0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 0x7f24bc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.439+0000 7f24db400640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24bc003f20 0x7f24bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24db400640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24bc004d00 0x7f24bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0ce6d0 con 0x7f24bc003f20 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc00ba60 con 0x7f24bc004d00 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24bc00bc20 con 0x7f24dc0b2210 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24dbe00640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24bc003f20 0x7f24bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24e0c00640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 0x7f24bc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24e1600640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24bc004d00 0x7f24bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24dbe00640 1 -- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 msgr2=0x7f24bc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24dbe00640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24dc0b2210 0x7f24bc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24dbe00640 1 -- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24bc004d00 msgr2=0x7f24bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24dbe00640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24bc004d00 0x7f24bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24dbe00640 1 -- 172.21.15.2:0/1409008167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24bc003520 con 0x7f24bc003f20 2024-01-15T18:05:30.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.440+0000 7f24e1600640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24bc004d00 0x7f24bc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:30.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.441+0000 7f24dbe00640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24bc003f20 0x7f24bc0047c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009a90 tx=0x7f24bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a930900341bde49a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c402c680 con 0x7f24bc003f20 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c402c7e0 con 0x7f24bc003f20 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24db400640 1 -- 172.21.15.2:0/1409008167 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c402cb90 con 0x7f24bc003f20 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24e2000640 1 -- 172.21.15.2:0/1409008167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24bc003f20 msgr2=0x7f24bc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24e2000640 1 --2- 172.21.15.2:0/1409008167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24bc003f20 0x7f24bc0047c0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009a90 tx=0x7f24bc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24e1600640 1 -- 172.21.15.2:0/1409008167 reap_dead start 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24e2000640 1 -- 172.21.15.2:0/1409008167 shutdown_connections 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.442+0000 7f24e2000640 1 -- 172.21.15.2:0/1409008167 >> 172.21.15.2:0/1409008167 conn(0x7f24dc0d30b0 msgr2=0x7f24dc0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.443+0000 7f24e2000640 1 -- 172.21.15.2:0/1409008167 shutdown_connections 2024-01-15T18:05:30.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.443+0000 7f24e2000640 1 -- 172.21.15.2:0/1409008167 wait complete. 2024-01-15T18:05:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.444+0000 7f24e2000640 1 Processor -- start 2024-01-15T18:05:30.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.444+0000 7f24e2000640 1 -- start start 2024-01-15T18:05:30.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.445+0000 7f24e2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24bc003f20 0x7f24dc15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.445+0000 7f24e2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24dc09e0f0 0x7f24dc15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.445+0000 7f24e2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 0x7f24dc14f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.445+0000 7f24e2000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc15f0b0 con 0x7f24dc160680 2024-01-15T18:05:30.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.445+0000 7f24e2000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc15e1b0 con 0x7f24dc09e0f0 2024-01-15T18:05:30.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.445+0000 7f24e2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc15e430 con 0x7f24bc003f20 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 0x7f24dc14f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 0x7f24dc14f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40166/0 (socket says 172.21.15.2:40166) 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 -- 172.21.15.2:0/2675470128 learned_addr learned my addr 172.21.15.2:0/2675470128 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e0c00640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24bc003f20 0x7f24dc15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 -- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24bc003f20 msgr2=0x7f24dc15b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24bc003f20 0x7f24dc15b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24dbe00640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24dc09e0f0 0x7f24dc15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 -- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24dc09e0f0 msgr2=0x7f24dc15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24dc09e0f0 0x7f24dc15dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.446+0000 7f24e1600640 1 -- 172.21.15.2:0/2675470128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc15e710 con 0x7f24dc160680 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.447+0000 7f24e1600640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 0x7f24dc14f0e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f24d40012e0 tx=0x7f24d400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.447+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d400bb80 con 0x7f24dc160680 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.447+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d400bce0 con 0x7f24dc160680 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.447+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4007330 con 0x7f24dc160680 2024-01-15T18:05:30.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.447+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc15e9f0 con 0x7f24dc160680 2024-01-15T18:05:30.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.447+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc09e570 con 0x7f24dc160680 2024-01-15T18:05:30.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.448+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc15f1f0 con 0x7f24dc160680 2024-01-15T18:05:30.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.449+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f24d4007490 con 0x7f24dc160680 2024-01-15T18:05:30.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.450+0000 7f24d8c00640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24b406bc10 0x7f24b406e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.450+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f24d4088ab0 con 0x7f24dc160680 2024-01-15T18:05:30.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.451+0000 7f24e0c00640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24b406bc10 0x7f24b406e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.452+0000 7f24e0c00640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24b406bc10 0x7f24b406e0d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0ce660 tx=0x7f24cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.454+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24dc15f1f0 con 0x7f24dc160680 2024-01-15T18:05:30.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.654+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24dc15f1f0 con 0x7f24dc160680 2024-01-15T18:05:30.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.655+0000 7f24d8c00640 1 -- 172.21.15.2:0/2675470128 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24dc15f1f0 con 0x7f24dc160680 2024-01-15T18:05:30.658 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24b406bc10 msgr2=0x7f24b406e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24b406bc10 0x7f24b406e0d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f24dc0ce660 tx=0x7f24cc009040 comp rx=0 tx=0).stop 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 msgr2=0x7f24dc14f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 0x7f24dc14f0e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f24d40012e0 tx=0x7f24d400b4a0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 shutdown_connections 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24b406bc10 0x7f24b406e0d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24dc160680 0x7f24dc14f0e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24dc09e0f0 0x7f24dc15dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 --2- 172.21.15.2:0/2675470128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24bc003f20 0x7f24dc15b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.658+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 >> 172.21.15.2:0/2675470128 conn(0x7f24dc0d30b0 msgr2=0x7f24dc0b2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.659+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 shutdown_connections 2024-01-15T18:05:30.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.659+0000 7f24e2000640 1 -- 172.21.15.2:0/2675470128 wait complete. 2024-01-15T18:05:30.681 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:05:30.855 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:30.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:05:30.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.899+0000 7f374b200640 1 Processor -- start 2024-01-15T18:05:30.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.899+0000 7f374b200640 1 -- start start 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.900+0000 7f374b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f374c15d200 0x7f374c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.900+0000 7f374b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c0d1380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.900+0000 7f374b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f374c0b30e0 con 0x7f374c0d2de0 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.900+0000 7f374b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f374c0ce180 con 0x7f374c15d200 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.900+0000 7f374b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c0d1ca0 con 0x7f374c0d0ed0 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3749e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f374c15d200 0x7f374c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3743e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f374c0d2de0 0x7f374c0d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56978/0 (socket says 172.21.15.2:56978) 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3743e00640 1 -- 172.21.15.2:0/3387594291 learned_addr learned my addr 172.21.15.2:0/3387594291 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:30.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f374a800640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c0d1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3749400640 1 -- 172.21.15.2:0/3387594291 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307499037 0 0) 0x7f374c0b30e0 con 0x7f374c0d2de0 2024-01-15T18:05:30.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3749e00640 1 -- 172.21.15.2:0/3387594291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 msgr2=0x7f374c0d1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:30.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3749e00640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c0d1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3749e00640 1 -- 172.21.15.2:0/3387594291 >> v1:172.21.15.2:6789/0 conn(0x7f374c0d2de0 legacy=0x7f374c0d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f3749e00640 1 -- 172.21.15.2:0/3387594291 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374c0d1f80 con 0x7f374c15d200 2024-01-15T18:05:30.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.901+0000 7f374a800640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c0d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:30.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.902+0000 7f3749e00640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c0cdc40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f373c0097a0 tx=0x7f373c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b8e5ab86fa8186 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.903+0000 7f3749400640 1 -- 172.21.15.2:0/3387594291 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c02d040 con 0x7f374c15d200 2024-01-15T18:05:30.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.903+0000 7f3749400640 1 -- 172.21.15.2:0/3387594291 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c031810 con 0x7f374c15d200 2024-01-15T18:05:30.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.903+0000 7f3749400640 1 -- 172.21.15.2:0/3387594291 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c031ba0 con 0x7f374c15d200 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.903+0000 7f374b200640 1 -- 172.21.15.2:0/3387594291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 msgr2=0x7f374c0cdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.903+0000 7f374b200640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c0cdc40 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f373c0097a0 tx=0x7f373c02b8b0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.904+0000 7f374b200640 1 -- 172.21.15.2:0/3387594291 shutdown_connections 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.904+0000 7f374b200640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c0d1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.904+0000 7f374b200640 1 --2- 172.21.15.2:0/3387594291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c0cdc40 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.904+0000 7f374b200640 1 -- 172.21.15.2:0/3387594291 >> 172.21.15.2:0/3387594291 conn(0x7f374c08caf0 msgr2=0x7f374c09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:30.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.904+0000 7f374b200640 1 -- 172.21.15.2:0/3387594291 shutdown_connections 2024-01-15T18:05:30.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.904+0000 7f374b200640 1 -- 172.21.15.2:0/3387594291 wait complete. 2024-01-15T18:05:30.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.906+0000 7f374b200640 1 Processor -- start 2024-01-15T18:05:30.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.907+0000 7f374b200640 1 -- start start 2024-01-15T18:05:30.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c15aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f374c0d2de0 0x7f374c1543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c15ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c15bb50 con 0x7f374c15d200 2024-01-15T18:05:30.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374c159c40 con 0x7f374c0d2de0 2024-01-15T18:05:30.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374c159f20 con 0x7f374c0d0ed0 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c15ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c15ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40186/0 (socket says 172.21.15.2:40186) 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.908+0000 7f374a800640 1 -- 172.21.15.2:0/3018184345 learned_addr learned my addr 172.21.15.2:0/3018184345 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f3749e00640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c15aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f3743e00640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f374c0d2de0 0x7f374c1543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f374a800640 1 -- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 msgr2=0x7f374c15aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f374a800640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c15aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f374a800640 1 -- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f374c0d2de0 msgr2=0x7f374c1543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f374a800640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f374c0d2de0 0x7f374c1543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f374a800640 1 -- 172.21.15.2:0/3018184345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c009400 con 0x7f374c15d200 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f3749e00640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c15aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:30.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.909+0000 7f374a800640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c15ee70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3744007e70 tx=0x7f374400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.910+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3744003b90 con 0x7f374c15d200 2024-01-15T18:05:30.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.910+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374400f040 con 0x7f374c15d200 2024-01-15T18:05:30.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.910+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f374c15a200 con 0x7f374c15d200 2024-01-15T18:05:30.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.910+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374400de80 con 0x7f374c15d200 2024-01-15T18:05:30.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.910+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f374c1576b0 con 0x7f374c15d200 2024-01-15T18:05:30.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.911+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f374c15bcc0 con 0x7f374c15d200 2024-01-15T18:05:30.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.913+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f3744004030 con 0x7f374c15d200 2024-01-15T18:05:30.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.914+0000 7f3742000640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f372406bbf0 0x7f372406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:30.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.914+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f37440888a0 con 0x7f374c15d200 2024-01-15T18:05:30.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.915+0000 7f3749e00640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f372406bbf0 0x7f372406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:30.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.915+0000 7f3749e00640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f372406bbf0 0x7f372406e0b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f373c036000 tx=0x7f373c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:30.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:30.918+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f374c15bcc0 con 0x7f374c15d200 2024-01-15T18:05:31.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.128+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f374c15a720 con 0x7f374c15d200 2024-01-15T18:05:31.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.129+0000 7f3742000640 1 -- 172.21.15.2:0/3018184345 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v65) v1 ==== 80+0+120 (secure 0 0 0) 0x7f374c15a720 con 0x7f374c15d200 2024-01-15T18:05:31.132 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6835/2855093754 2024-01-16T18:05:24.435299+0000 2024-01-15T18:05:31.133 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/2855093754 2024-01-16T18:05:24.435299+0000 2024-01-15T18:05:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:05:31.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f372406bbf0 msgr2=0x7f372406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f372406bbf0 0x7f372406e0b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f373c036000 tx=0x7f373c03c040 comp rx=0 tx=0).stop 2024-01-15T18:05:31.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 msgr2=0x7f374c15ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c15ee70 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f3744007e70 tx=0x7f374400d4a0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 shutdown_connections 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f372406bbf0 0x7f372406e0b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f374c15d200 0x7f374c15ee70 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f374c0d2de0 0x7f374c1543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 --2- 172.21.15.2:0/3018184345 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f374c0d0ed0 0x7f374c15aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 >> 172.21.15.2:0/3018184345 conn(0x7f374c08caf0 msgr2=0x7f374c0cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 shutdown_connections 2024-01-15T18:05:31.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.132+0000 7f374b200640 1 -- 172.21.15.2:0/3018184345 wait complete. 2024-01-15T18:05:31.157 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:05:31.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:31.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:31.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[44205]: fuse finished with error 0 and tester_r 0 2024-01-15T18:05:31.254 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:05:31.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:31.255 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:05:31.435 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:05:31.436 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:31.492 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:05:31.492 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:05:31.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:31.492 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:05:31.548 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:05:31.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:05:31.550 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:05:31.550 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:31.606 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:05:31.606 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:05:31.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:31.607 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:05:31.657 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:05:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:05:31.659 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:31.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.891+0000 7eff64a00640 1 Processor -- start 2024-01-15T18:05:31.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.891+0000 7eff64a00640 1 -- start start 2024-01-15T18:05:31.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff64a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7eff5c15c0e0 0x7eff5c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff64a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 0x7eff5c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff64a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff5c15cb00 con 0x7eff5c15c4c0 2024-01-15T18:05:31.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff64a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff5c0ccec0 con 0x7eff5c15c0e0 2024-01-15T18:05:31.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff64a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c0cd1a0 con 0x7eff5c15b7d0 2024-01-15T18:05:31.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff62a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7eff5c15c4c0 0x7eff5c15ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56986/0 (socket says 172.21.15.2:56986) 2024-01-15T18:05:31.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff62a00640 1 -- 172.21.15.2:0/2947536618 learned_addr learned my addr 172.21.15.2:0/2947536618 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff62000640 1 --2- 172.21.15.2:0/2947536618 >> v2:172.21.15.137:3300/0 conn(0x7eff5c15c0e0 0x7eff5c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.893+0000 7eff61600640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 0x7eff5c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113094186 0 0) 0x7eff5c15cb00 con 0x7eff5c15c4c0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff3c003520 con 0x7eff5c15c4c0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541443109 0 0) 0x7eff3c003520 con 0x7eff5c15c4c0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 msgr2=0x7eff5c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 0x7eff5c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 >> v2:172.21.15.137:3300/0 conn(0x7eff5c15c0e0 msgr2=0x7eff5c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 --2- 172.21.15.2:0/2947536618 >> v2:172.21.15.137:3300/0 conn(0x7eff5c15c0e0 0x7eff5c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c0cd480 con 0x7eff5c15c4c0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7eff540019f0 con 0x7eff5c15c4c0 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 >> v1:172.21.15.2:6789/0 conn(0x7eff5c15c4c0 legacy=0x7eff5c15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff3c003970 0x7eff3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 0x7eff3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c0ccec0 con 0x7eff3c004d00 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c0cd1a0 con 0x7eff3c003970 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.894+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c00b980 con 0x7eff5c15b7d0 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62a00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62000640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff3c003970 0x7eff3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62a00640 1 -- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 msgr2=0x7eff3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62a00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c15b7d0 0x7eff3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62a00640 1 -- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff3c003970 msgr2=0x7eff3c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62a00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff3c003970 0x7eff3c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.895+0000 7eff62a00640 1 -- 172.21.15.2:0/2947536618 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c003520 con 0x7eff3c004d00 2024-01-15T18:05:31.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.896+0000 7eff62a00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7eff5400a760 tx=0x7eff3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b57eeece6d674c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.896+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54034630 con 0x7eff3c004d00 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.896+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5402f070 con 0x7eff3c004d00 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.896+0000 7eff60c00640 1 -- 172.21.15.2:0/2947536618 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff540349d0 con 0x7eff3c004d00 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff64a00640 1 -- 172.21.15.2:0/2947536618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 msgr2=0x7eff3c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff64a00640 1 --2- 172.21.15.2:0/2947536618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7eff5400a760 tx=0x7eff3c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff62a00640 1 -- 172.21.15.2:0/2947536618 reap_dead start 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff64a00640 1 -- 172.21.15.2:0/2947536618 shutdown_connections 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff64a00640 1 -- 172.21.15.2:0/2947536618 >> 172.21.15.2:0/2947536618 conn(0x7eff5c06da20 msgr2=0x7eff5c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff64a00640 1 -- 172.21.15.2:0/2947536618 shutdown_connections 2024-01-15T18:05:31.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.897+0000 7eff64a00640 1 -- 172.21.15.2:0/2947536618 wait complete. 2024-01-15T18:05:31.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.899+0000 7eff64a00640 1 Processor -- start 2024-01-15T18:05:31.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.899+0000 7eff64a00640 1 -- start start 2024-01-15T18:05:31.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff64a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff5c071980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff64a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff5c0afdb0 0x7eff5c13a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff64a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c0b0230 0x7eff5c13ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff64a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c14a260 con 0x7eff3c004d00 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff64a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c14c160 con 0x7eff5c0afdb0 2024-01-15T18:05:31.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff64a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5c071ec0 con 0x7eff5c0b0230 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff62000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff5c071980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.900+0000 7eff62000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff5c071980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40198/0 (socket says 172.21.15.2:40198) 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 -- 172.21.15.2:0/793485234 learned_addr learned my addr 172.21.15.2:0/793485234 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 -- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c0b0230 msgr2=0x7eff5c13ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c0b0230 0x7eff5c13ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 -- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff5c0afdb0 msgr2=0x7eff5c13a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff61600640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff5c0afdb0 0x7eff5c13a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff5c0afdb0 0x7eff5c13a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 -- 172.21.15.2:0/793485234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5c149ca0 con 0x7eff3c004d00 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff61600640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff5c0afdb0 0x7eff5c13a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.901+0000 7eff62000640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff5c071980 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7eff4c011680 tx=0x7eff4c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.902+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c00f960 con 0x7eff3c004d00 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.902+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c00fac0 con 0x7eff3c004d00 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.902+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5c149f80 con 0x7eff3c004d00 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.902+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c00fe30 con 0x7eff3c004d00 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.902+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff5c14c650 con 0x7eff3c004d00 2024-01-15T18:05:31.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.903+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5c0686f0 con 0x7eff3c004d00 2024-01-15T18:05:31.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.904+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7eff4c015460 con 0x7eff3c004d00 2024-01-15T18:05:31.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.905+0000 7eff5a000640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff3406bbc0 0x7eff3406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:31.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.905+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7eff4c088d00 con 0x7eff3c004d00 2024-01-15T18:05:31.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.906+0000 7eff61600640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff3406bbc0 0x7eff3406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:31.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.907+0000 7eff61600640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff3406bbc0 0x7eff3406e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7eff5c15b2f0 tx=0x7eff44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:31.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:31.909+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff4c0555c0 con 0x7eff3c004d00 2024-01-15T18:05:32.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.145+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff5c06b260 con 0x7eff3c004d00 2024-01-15T18:05:32.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.146+0000 7eff5a000640 1 -- 172.21.15.2:0/793485234 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+3696 (secure 0 0 0) 0x7eff4c058be0 con 0x7eff3c004d00 2024-01-15T18:05:32.150 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:32.150 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1133084700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1133084700},{"type":"v1","addr":"172.21.15.137:6836","nonce":1133084700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5614,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/302473054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":302473054},{"type":"v1","addr":"172.21.15.137:6837","nonce":302473054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2347293827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2347293827},{"type":"v1","addr":"172.21.15.2:6833","nonce":2347293827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6867,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1512269903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1512269903},{"type":"v1","addr":"172.21.15.2:6835","nonce":1512269903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2024-01-15T18:05:32.150 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 39 2024-01-15T18:05:32.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff3406bbc0 msgr2=0x7eff3406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff3406bbc0 0x7eff3406e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7eff5c15b2f0 tx=0x7eff44008040 comp rx=0 tx=0).stop 2024-01-15T18:05:32.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 msgr2=0x7eff5c071980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff5c071980 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7eff4c011680 tx=0x7eff4c011b40 comp rx=0 tx=0).stop 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 shutdown_connections 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff3406bbc0 0x7eff3406e080 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5c0b0230 0x7eff5c13ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff5c0afdb0 0x7eff5c13a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 --2- 172.21.15.2:0/793485234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff3c004d00 0x7eff5c071980 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.149+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 >> 172.21.15.2:0/793485234 conn(0x7eff5c06da20 msgr2=0x7eff5c15e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.150+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 shutdown_connections 2024-01-15T18:05:32.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.150+0000 7eff64a00640 1 -- 172.21.15.2:0/793485234 wait complete. 2024-01-15T18:05:32.171 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-15T18:05:32.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.394+0000 7f318a200640 1 Processor -- start 2024-01-15T18:05:32.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.395+0000 7f318a200640 1 -- start start 2024-01-15T18:05:32.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f318a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3184143d20 0x7f318415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f318a200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f31841447d0 0x7f318415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f318a200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3184071090 con 0x7f31840720d0 2024-01-15T18:05:32.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f318a200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3184071430 con 0x7f3184143d20 2024-01-15T18:05:32.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f318a200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3184144100 con 0x7f31841447d0 2024-01-15T18:05:32.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f317be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f31840720d0 0x7f31840cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56996/0 (socket says 172.21.15.2:56996) 2024-01-15T18:05:32.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f317be00640 1 -- 172.21.15.2:0/152703661 learned_addr learned my addr 172.21.15.2:0/152703661 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:32.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.396+0000 7f3188e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3184143d20 0x7f318415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.397+0000 7f3189800640 1 --2- 172.21.15.2:0/152703661 >> v2:172.21.15.137:3300/0 conn(0x7f31841447d0 0x7f318415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.397+0000 7f3188e00640 1 -- 172.21.15.2:0/152703661 >> v2:172.21.15.137:3300/0 conn(0x7f31841447d0 msgr2=0x7f318415f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.397+0000 7f3188e00640 1 --2- 172.21.15.2:0/152703661 >> v2:172.21.15.137:3300/0 conn(0x7f31841447d0 0x7f318415f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.397+0000 7f3188e00640 1 -- 172.21.15.2:0/152703661 >> v1:172.21.15.2:6789/0 conn(0x7f31840720d0 legacy=0x7f31840cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.397+0000 7f3188e00640 1 -- 172.21.15.2:0/152703661 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318406d160 con 0x7f3184143d20 2024-01-15T18:05:32.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.397+0000 7f3183e00640 1 -- 172.21.15.2:0/152703661 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603750263 0 0) 0x7f3184071090 con 0x7f31840720d0 2024-01-15T18:05:32.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.398+0000 7f3188e00640 1 --2- 172.21.15.2:0/152703661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3184143d20 0x7f318415eb50 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3184072a20 tx=0x7f317402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3cf232ad91f2aa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.398+0000 7f3183e00640 1 -- 172.21.15.2:0/152703661 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317402e070 con 0x7f3184143d20 2024-01-15T18:05:32.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.398+0000 7f3183e00640 1 -- 172.21.15.2:0/152703661 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31740315a0 con 0x7f3184143d20 2024-01-15T18:05:32.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.398+0000 7f3183e00640 1 -- 172.21.15.2:0/152703661 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3174031930 con 0x7f3184143d20 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 -- 172.21.15.2:0/152703661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3184143d20 msgr2=0x7f318415eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 --2- 172.21.15.2:0/152703661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3184143d20 0x7f318415eb50 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f3184072a20 tx=0x7f317402b840 comp rx=0 tx=0).stop 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 -- 172.21.15.2:0/152703661 shutdown_connections 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 --2- 172.21.15.2:0/152703661 >> v2:172.21.15.137:3300/0 conn(0x7f31841447d0 0x7f318415f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 --2- 172.21.15.2:0/152703661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3184143d20 0x7f318415eb50 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 -- 172.21.15.2:0/152703661 >> 172.21.15.2:0/152703661 conn(0x7f31840b0840 msgr2=0x7f31840b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 -- 172.21.15.2:0/152703661 shutdown_connections 2024-01-15T18:05:32.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.399+0000 7f318a200640 1 -- 172.21.15.2:0/152703661 wait complete. 2024-01-15T18:05:32.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.401+0000 7f318a200640 1 Processor -- start 2024-01-15T18:05:32.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.401+0000 7f318a200640 1 -- start start 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f318a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31840720d0 0x7f3184136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f318a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3184143d20 0x7f31841f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f318a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 0x7f31841f2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f318a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f318415af50 con 0x7f31841447d0 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f318a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3184136e70 con 0x7f31840720d0 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f318a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31841f3a80 con 0x7f3184143d20 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.402+0000 7f3189800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 0x7f31841f2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 0x7f31841f2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40212/0 (socket says 172.21.15.2:40212) 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 -- 172.21.15.2:0/3036999096 learned_addr learned my addr 172.21.15.2:0/3036999096 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3188e00640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31840720d0 0x7f3184136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f317be00640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3184143d20 0x7f31841f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 -- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3184143d20 msgr2=0x7f31841f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3184143d20 0x7f31841f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 -- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31840720d0 msgr2=0x7f3184136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31840720d0 0x7f3184136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 -- 172.21.15.2:0/3036999096 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3174009400 con 0x7f31841447d0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3189800640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 0x7f31841f2850 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f317c007e40 tx=0x7f317c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.403+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c014620 con 0x7f31841447d0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.404+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f317c00ee90 con 0x7f31841447d0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.404+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f317c0149e0 con 0x7f31841447d0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.404+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31841f2f00 con 0x7f31841447d0 2024-01-15T18:05:32.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.404+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31841f34c0 con 0x7f31841447d0 2024-01-15T18:05:32.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.405+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3184074770 con 0x7f31841447d0 2024-01-15T18:05:32.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.406+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f317c014b40 con 0x7f31841447d0 2024-01-15T18:05:32.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.407+0000 7f3181600640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f315c06bbc0 0x7f315c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.407+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f317c088e60 con 0x7f31841447d0 2024-01-15T18:05:32.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.408+0000 7f3188e00640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f315c06bbc0 0x7f315c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.409+0000 7f3188e00640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f315c06bbc0 0x7f315c06e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3174036000 tx=0x7f317403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.411+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f317c055720 con 0x7f31841447d0 2024-01-15T18:05:32.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.614+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f31840ad4a0 con 0x7f31841447d0 2024-01-15T18:05:32.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.619+0000 7f3181600640 1 -- 172.21.15.2:0/3036999096 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f317c058d40 con 0x7f31841447d0 2024-01-15T18:05:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.622+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f315c06bbc0 msgr2=0x7f315c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.622+0000 7f318a200640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f315c06bbc0 0x7f315c06e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f3174036000 tx=0x7f317403d040 comp rx=0 tx=0).stop 2024-01-15T18:05:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.622+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 msgr2=0x7f31841f2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.622+0000 7f318a200640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 0x7f31841f2850 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f317c007e40 tx=0x7f317c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 shutdown_connections 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f315c06bbc0 0x7f315c06e080 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31841447d0 0x7f31841f2850 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3184143d20 0x7f31841f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 --2- 172.21.15.2:0/3036999096 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31840720d0 0x7f3184136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 >> 172.21.15.2:0/3036999096 conn(0x7f31840b0840 msgr2=0x7f31840cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 shutdown_connections 2024-01-15T18:05:32.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.623+0000 7f318a200640 1 -- 172.21.15.2:0/3036999096 wait complete. 2024-01-15T18:05:32.647 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:05:32.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.872+0000 7fbc4ac00640 1 Processor -- start 2024-01-15T18:05:32.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.872+0000 7fbc4ac00640 1 -- start start 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc4ac00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbc440d1870 0x7fbc440d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc4ac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fbc440d1c50 0x7fbc440b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc4ac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440d1250 con 0x7fbc440b2210 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc4ac00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc440ce6d0 con 0x7fbc440d1870 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc4ac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc440ce8e0 con 0x7fbc440d1c50 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc49800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbc440d1870 0x7fbc440d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc48e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbc440b2210 0x7fbc4415d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57004/0 (socket says 172.21.15.2:57004) 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.874+0000 7fbc48e00640 1 -- 172.21.15.2:0/4108192016 learned_addr learned my addr 172.21.15.2:0/4108192016 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.875+0000 7fbc49800640 1 -- 172.21.15.2:0/4108192016 >> v2:172.21.15.137:3300/0 conn(0x7fbc440d1c50 msgr2=0x7fbc440b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.875+0000 7fbc4a200640 1 --2- 172.21.15.2:0/4108192016 >> v2:172.21.15.137:3300/0 conn(0x7fbc440d1c50 0x7fbc440b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.875+0000 7fbc49800640 1 --2- 172.21.15.2:0/4108192016 >> v2:172.21.15.137:3300/0 conn(0x7fbc440d1c50 0x7fbc440b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.875+0000 7fbc49800640 1 -- 172.21.15.2:0/4108192016 >> v1:172.21.15.2:6789/0 conn(0x7fbc440b2210 legacy=0x7fbc4415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.875+0000 7fbc43e00640 1 -- 172.21.15.2:0/4108192016 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479901053 0 0) 0x7fbc440d1250 con 0x7fbc440b2210 2024-01-15T18:05:32.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.875+0000 7fbc49800640 1 -- 172.21.15.2:0/4108192016 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc440cebc0 con 0x7fbc440d1870 2024-01-15T18:05:32.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.876+0000 7fbc49800640 1 --2- 172.21.15.2:0/4108192016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440d1870 0x7fbc440d3bd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbc4408d7e0 tx=0x7fbc2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed228be7399a3d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.876+0000 7fbc43e00640 1 -- 172.21.15.2:0/4108192016 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c031830 con 0x7fbc440d1870 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.876+0000 7fbc43e00640 1 -- 172.21.15.2:0/4108192016 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c031990 con 0x7fbc440d1870 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.876+0000 7fbc43e00640 1 -- 172.21.15.2:0/4108192016 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c031d20 con 0x7fbc440d1870 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 -- 172.21.15.2:0/4108192016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440d1870 msgr2=0x7fbc440d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/4108192016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440d1870 0x7fbc440d3bd0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fbc4408d7e0 tx=0x7fbc2c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 -- 172.21.15.2:0/4108192016 shutdown_connections 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/4108192016 >> v2:172.21.15.137:3300/0 conn(0x7fbc440d1c50 0x7fbc440b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/4108192016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440d1870 0x7fbc440d3bd0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 -- 172.21.15.2:0/4108192016 >> 172.21.15.2:0/4108192016 conn(0x7fbc440d30b0 msgr2=0x7fbc440d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 -- 172.21.15.2:0/4108192016 shutdown_connections 2024-01-15T18:05:32.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.877+0000 7fbc4ac00640 1 -- 172.21.15.2:0/4108192016 wait complete. 2024-01-15T18:05:32.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.879+0000 7fbc4ac00640 1 Processor -- start 2024-01-15T18:05:32.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.879+0000 7fbc4ac00640 1 -- start start 2024-01-15T18:05:32.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc4ac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 0x7fbc441f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc4ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc440d1870 0x7fbc441f38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc4ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc440d1c50 0x7fbc440c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc4ac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc440c7a00 con 0x7fbc440b2210 2024-01-15T18:05:32.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc4ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc441f0900 con 0x7fbc440d1870 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc4ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc441f3e30 con 0x7fbc440d1c50 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc49800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 0x7fbc441f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc49800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 0x7fbc441f03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40230/0 (socket says 172.21.15.2:40230) 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.880+0000 7fbc49800640 1 -- 172.21.15.2:0/1812262584 learned_addr learned my addr 172.21.15.2:0/1812262584 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc48e00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc440d1870 0x7fbc441f38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc49800640 1 -- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc440d1c50 msgr2=0x7fbc440c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc49800640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc440d1c50 0x7fbc440c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc49800640 1 -- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc440d1870 msgr2=0x7fbc441f38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc49800640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc440d1870 0x7fbc441f38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc49800640 1 -- 172.21.15.2:0/1812262584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2c009400 con 0x7fbc440b2210 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc48e00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc440d1870 0x7fbc441f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc49800640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 0x7fbc441f03c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c000c00 tx=0x7fbc2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c004230 con 0x7fbc440b2210 2024-01-15T18:05:32.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c004390 con 0x7fbc440b2210 2024-01-15T18:05:32.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc440c6960 con 0x7fbc440b2210 2024-01-15T18:05:32.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.881+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c03a920 con 0x7fbc440b2210 2024-01-15T18:05:32.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.882+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc440c6f20 con 0x7fbc440b2210 2024-01-15T18:05:32.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.883+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44074770 con 0x7fbc440b2210 2024-01-15T18:05:32.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.884+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fbc2c03aa80 con 0x7fbc440b2210 2024-01-15T18:05:32.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.885+0000 7fbc41600640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc1c06bbc0 0x7fbc1c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:32.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.885+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fbc2c0aabc0 con 0x7fbc440b2210 2024-01-15T18:05:32.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.885+0000 7fbc48e00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc1c06bbc0 0x7fbc1c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:32.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.887+0000 7fbc48e00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc1c06bbc0 0x7fbc1c06e080 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbc441608f0 tx=0x7fbc34009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:32.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:32.890+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc2c0b1070 con 0x7fbc440b2210 2024-01-15T18:05:33.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.090+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fbc440d2e90 con 0x7fbc440b2210 2024-01-15T18:05:33.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.096+0000 7fbc41600640 1 -- 172.21.15.2:0/1812262584 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v24) v1 ==== 120+0+0 (secure 0 0 0) 0x7fbc2c077480 con 0x7fbc440b2210 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc1c06bbc0 msgr2=0x7fbc1c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc1c06bbc0 0x7fbc1c06e080 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbc441608f0 tx=0x7fbc34009040 comp rx=0 tx=0).stop 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 msgr2=0x7fbc441f03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 0x7fbc441f03c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c000c00 tx=0x7fbc2c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 shutdown_connections 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbc1c06bbc0 0x7fbc1c06e080 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbc440d1c50 0x7fbc440c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbc440d1870 0x7fbc441f38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 --2- 172.21.15.2:0/1812262584 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbc440b2210 0x7fbc441f03c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 >> 172.21.15.2:0/1812262584 conn(0x7fbc440d30b0 msgr2=0x7fbc440cdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 shutdown_connections 2024-01-15T18:05:33.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.100+0000 7fbc4ac00640 1 -- 172.21.15.2:0/1812262584 wait complete. 2024-01-15T18:05:33.122 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:05:33.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.343+0000 7f1a81200640 1 Processor -- start 2024-01-15T18:05:33.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.344+0000 7f1a81200640 1 -- start start 2024-01-15T18:05:33.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.345+0000 7f1a81200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 0x7f1a7c0b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.345+0000 7f1a81200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1a7c0cfe30 0x7f1a7c0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.345+0000 7f1a81200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c0d3bd0 con 0x7f1a7c0b3cb0 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.345+0000 7f1a81200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a7c0b3b50 con 0x7f1a7c0cfe30 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.345+0000 7f1a81200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0d3eb0 con 0x7f1a7c0cfa50 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.346+0000 7f1a73e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1a7c0b3cb0 0x7f1a7c1612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57480/0 (socket says 172.21.15.2:57480) 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.346+0000 7f1a73e00640 1 -- 172.21.15.2:0/4153666233 learned_addr learned my addr 172.21.15.2:0/4153666233 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.346+0000 7f1a7be00640 1 --2- 172.21.15.2:0/4153666233 >> v2:172.21.15.137:3300/0 conn(0x7f1a7c0cfe30 0x7f1a7c0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.346+0000 7f1a7b400640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 0x7f1a7c0b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.346+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 431322953 0 0) 0x7f1a7c0d3bd0 con 0x7f1a7c0b3cb0 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.346+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c003520 con 0x7f1a7c0b3cb0 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2404141591 0 0) 0x7f1a5c003520 con 0x7f1a7c0b3cb0 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 msgr2=0x7f1a7c0b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 0x7f1a7c0b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 >> v2:172.21.15.137:3300/0 conn(0x7f1a7c0cfe30 msgr2=0x7f1a7c0b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 --2- 172.21.15.2:0/4153666233 >> v2:172.21.15.137:3300/0 conn(0x7f1a7c0cfe30 0x7f1a7c0b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c0d18a0 con 0x7f1a7c0b3cb0 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1a64001670 con 0x7f1a7c0b3cb0 2024-01-15T18:05:33.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 >> v1:172.21.15.2:6789/0 conn(0x7f1a7c0b3cb0 legacy=0x7f1a7c1612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 0x7f1a5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0b3b50 con 0x7f1a5c004d00 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c0d3eb0 con 0x7f1a5c003970 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c00b980 con 0x7f1a7c0cfa50 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7be00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.347+0000 7f1a7b400640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a73e00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 0x7f1a5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a7be00640 1 -- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 msgr2=0x7f1a5c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a7be00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c0cfa50 0x7f1a5c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a7be00640 1 -- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a5c003970 msgr2=0x7f1a5c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a7be00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a7be00640 1 -- 172.21.15.2:0/4153666233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c003520 con 0x7f1a5c004d00 2024-01-15T18:05:33.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.348+0000 7f1a7b400640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a5c003970 0x7f1a5c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:33.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.349+0000 7f1a7be00640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a5c005180 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009fc0 tx=0x7f1a5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f3674ae639a209 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.349+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7402cc60 con 0x7f1a5c004d00 2024-01-15T18:05:33.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.349+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7400f040 con 0x7f1a5c004d00 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.349+0000 7f1a7aa00640 1 -- 172.21.15.2:0/4153666233 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a74034440 con 0x7f1a5c004d00 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a81200640 1 -- 172.21.15.2:0/4153666233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 msgr2=0x7f1a5c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a81200640 1 --2- 172.21.15.2:0/4153666233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a5c005180 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f1a74009fc0 tx=0x7f1a5c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a7be00640 1 -- 172.21.15.2:0/4153666233 reap_dead start 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a81200640 1 -- 172.21.15.2:0/4153666233 shutdown_connections 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a81200640 1 -- 172.21.15.2:0/4153666233 >> 172.21.15.2:0/4153666233 conn(0x7f1a7c0d30b0 msgr2=0x7f1a7c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a81200640 1 -- 172.21.15.2:0/4153666233 shutdown_connections 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.350+0000 7f1a81200640 1 -- 172.21.15.2:0/4153666233 wait complete. 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.352+0000 7f1a81200640 1 Processor -- start 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.352+0000 7f1a81200640 1 -- start start 2024-01-15T18:05:33.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a81200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a7c1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a81200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c152d80 0x7f1a7c0ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a81200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c153200 0x7f1a7c0ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a81200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c1543e0 con 0x7f1a5c004d00 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a81200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c1537d0 con 0x7f1a7c152d80 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a81200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7c153ab0 con 0x7f1a7c153200 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a7b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a7c1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a7b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a7c1f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36234/0 (socket says 172.21.15.2:36234) 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.353+0000 7f1a7b400640 1 -- 172.21.15.2:0/2548869394 learned_addr learned my addr 172.21.15.2:0/2548869394 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a73e00640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c152d80 0x7f1a7c0ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7be00640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c153200 0x7f1a7c0ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7b400640 1 -- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c153200 msgr2=0x7f1a7c0ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7b400640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c153200 0x7f1a7c0ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7b400640 1 -- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c152d80 msgr2=0x7f1a7c0ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7b400640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c152d80 0x7f1a7c0ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7b400640 1 -- 172.21.15.2:0/2548869394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c153d90 con 0x7f1a5c004d00 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7be00640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c153200 0x7f1a7c0ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.354+0000 7f1a7b400640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a7c1f2520 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c011540 tx=0x7f1a6c011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.355+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c015870 con 0x7f1a5c004d00 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.355+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c0159d0 con 0x7f1a5c004d00 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.355+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c015d40 con 0x7f1a5c004d00 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.355+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c154070 con 0x7f1a5c004d00 2024-01-15T18:05:33.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.355+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c1f2a60 con 0x7f1a5c004d00 2024-01-15T18:05:33.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.356+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7c074770 con 0x7f1a5c004d00 2024-01-15T18:05:33.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.357+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1a6c00f460 con 0x7f1a5c004d00 2024-01-15T18:05:33.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.358+0000 7f1a73400640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a5406bbc0 0x7f1a5406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.358+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f1a6c088cf0 con 0x7f1a5c004d00 2024-01-15T18:05:33.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.359+0000 7f1a73e00640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a5406bbc0 0x7f1a5406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.360+0000 7f1a73e00640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a5406bbc0 0x7f1a5406e080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0b8330 tx=0x7f1a64009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.362+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a6c055530 con 0x7f1a5c004d00 2024-01-15T18:05:33.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.558+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1a7c072880 con 0x7f1a5c004d00 2024-01-15T18:05:33.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.565+0000 7f1a73400640 1 -- 172.21.15.2:0/2548869394 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1a6c058b50 con 0x7f1a5c004d00 2024-01-15T18:05:33.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a5406bbc0 msgr2=0x7f1a5406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a5406bbc0 0x7f1a5406e080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f1a7c0b8330 tx=0x7f1a64009040 comp rx=0 tx=0).stop 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 msgr2=0x7f1a7c1f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a7c1f2520 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c011540 tx=0x7f1a6c011a00 comp rx=0 tx=0).stop 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 shutdown_connections 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1a5406bbc0 0x7f1a5406e080 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1a7c153200 0x7f1a7c0ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1a7c152d80 0x7f1a7c0ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 --2- 172.21.15.2:0/2548869394 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1a5c004d00 0x7f1a7c1f2520 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 >> 172.21.15.2:0/2548869394 conn(0x7f1a7c0d30b0 msgr2=0x7f1a7c0b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 shutdown_connections 2024-01-15T18:05:33.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.568+0000 7f1a81200640 1 -- 172.21.15.2:0/2548869394 wait complete. 2024-01-15T18:05:33.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-15T18:05:33.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.810+0000 7fc685a00640 1 Processor -- start 2024-01-15T18:05:33.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.810+0000 7fc685a00640 1 -- start start 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc685a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc68015ec60 0x7fc68015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc685a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc68006cad0 0x7fc68006cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc685a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6800716d0 con 0x7fc680141880 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc685a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc680071f20 con 0x7fc68015ec60 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc685a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68006c6b0 con 0x7fc68006cad0 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc67f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc68015ec60 0x7fc68015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc67f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc68015ec60 0x7fc68015f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36240/0 (socket says 172.21.15.2:36240) 2024-01-15T18:05:33.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.812+0000 7fc67f400640 1 -- 172.21.15.2:0/4290472406 learned_addr learned my addr 172.21.15.2:0/4290472406 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.813+0000 7fc685000640 1 --2- 172.21.15.2:0/4290472406 >> v2:172.21.15.137:3300/0 conn(0x7fc68006cad0 0x7fc68006cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.813+0000 7fc67f400640 1 -- 172.21.15.2:0/4290472406 >> v2:172.21.15.137:3300/0 conn(0x7fc68006cad0 msgr2=0x7fc68006cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.813+0000 7fc67f400640 1 --2- 172.21.15.2:0/4290472406 >> v2:172.21.15.137:3300/0 conn(0x7fc68006cad0 0x7fc68006cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.813+0000 7fc67f400640 1 -- 172.21.15.2:0/4290472406 >> v1:172.21.15.137:6789/0 conn(0x7fc680141880 legacy=0x7fc680070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.813+0000 7fc67f400640 1 -- 172.21.15.2:0/4290472406 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680071960 con 0x7fc68015ec60 2024-01-15T18:05:33.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.813+0000 7fc685000640 1 --2- 172.21.15.2:0/4290472406 >> v2:172.21.15.137:3300/0 conn(0x7fc68006cad0 0x7fc68006cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:33.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.814+0000 7fc67f400640 1 --2- 172.21.15.2:0/4290472406 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc68015f0e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc680070060 tx=0x7fc66802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31680b1d85c40501 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc67ea00640 1 -- 172.21.15.2:0/4290472406 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6680317e0 con 0x7fc68015ec60 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc67ea00640 1 -- 172.21.15.2:0/4290472406 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc668031940 con 0x7fc68015ec60 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc67ea00640 1 -- 172.21.15.2:0/4290472406 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc668031cd0 con 0x7fc68015ec60 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 -- 172.21.15.2:0/4290472406 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 msgr2=0x7fc68015f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 --2- 172.21.15.2:0/4290472406 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc68015f0e0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fc680070060 tx=0x7fc66802b880 comp rx=0 tx=0).stop 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 -- 172.21.15.2:0/4290472406 shutdown_connections 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 --2- 172.21.15.2:0/4290472406 >> v2:172.21.15.137:3300/0 conn(0x7fc68006cad0 0x7fc68006cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 --2- 172.21.15.2:0/4290472406 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc68015f0e0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 -- 172.21.15.2:0/4290472406 >> 172.21.15.2:0/4290472406 conn(0x7fc6800b0840 msgr2=0x7fc6800b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 -- 172.21.15.2:0/4290472406 shutdown_connections 2024-01-15T18:05:33.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.815+0000 7fc685a00640 1 -- 172.21.15.2:0/4290472406 wait complete. 2024-01-15T18:05:33.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.817+0000 7fc685a00640 1 Processor -- start 2024-01-15T18:05:33.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.817+0000 7fc685a00640 1 -- start start 2024-01-15T18:05:33.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.818+0000 7fc685a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc68006cad0 0x7fc68013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.818+0000 7fc685a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc680141880 0x7fc6801e8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.818+0000 7fc685a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc6801e9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.818+0000 7fc685a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6801f79b0 con 0x7fc68015ec60 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.818+0000 7fc685a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc680140110 con 0x7fc680141880 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.818+0000 7fc685a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6801f76d0 con 0x7fc68006cad0 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc6801e9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc6801e9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36250/0 (socket says 172.21.15.2:36250) 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 -- 172.21.15.2:0/3467851644 learned_addr learned my addr 172.21.15.2:0/3467851644 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc67fe00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc68006cad0 0x7fc68013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 -- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc68006cad0 msgr2=0x7fc68013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc68006cad0 0x7fc68013df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 -- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc680141880 msgr2=0x7fc6801e8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc67f400640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc680141880 0x7fc6801e8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc680141880 0x7fc6801e8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:33.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.819+0000 7fc685000640 1 -- 172.21.15.2:0/3467851644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680094d0 con 0x7fc68015ec60 2024-01-15T18:05:33.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.820+0000 7fc685000640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc6801e9440 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc67800cef0 tx=0x7fc67800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.820+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67800d840 con 0x7fc68015ec60 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.820+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67800d9c0 con 0x7fc68015ec60 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.820+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc67800dd70 con 0x7fc68015ec60 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.820+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6801595a0 con 0x7fc68015ec60 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.820+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc680159b60 con 0x7fc68015ec60 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.821+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc680159400 con 0x7fc68015ec60 2024-01-15T18:05:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.822+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fc67801e3b0 con 0x7fc68015ec60 2024-01-15T18:05:33.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.823+0000 7fc677e00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc65806bbc0 0x7fc65806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:33.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.823+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fc678089180 con 0x7fc68015ec60 2024-01-15T18:05:33.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.824+0000 7fc67fe00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc65806bbc0 0x7fc65806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:33.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.825+0000 7fc67fe00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc65806bbc0 0x7fc65806e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc670004ea0 tx=0x7fc670009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:33.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:33.827+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc680159400 con 0x7fc68015ec60 2024-01-15T18:05:34.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.090+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7fc680159400 con 0x7fc68015ec60 2024-01-15T18:05:34.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.531+0000 7fc677e00640 1 -- 172.21.15.2:0/3467851644 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v379) v1 ==== 154+0+0 (secure 0 0 0) 0x7fc680159400 con 0x7fc68015ec60 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.534+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc65806bbc0 msgr2=0x7fc65806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.534+0000 7fc685a00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc65806bbc0 0x7fc65806e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc670004ea0 tx=0x7fc670009040 comp rx=0 tx=0).stop 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.534+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 msgr2=0x7fc6801e9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.534+0000 7fc685a00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc6801e9440 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fc67800cef0 tx=0x7fc67800a710 comp rx=0 tx=0).stop 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 shutdown_connections 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc65806bbc0 0x7fc65806e080 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc68015ec60 0x7fc6801e9440 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc680141880 0x7fc6801e8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 --2- 172.21.15.2:0/3467851644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc68006cad0 0x7fc68013df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 >> 172.21.15.2:0/3467851644 conn(0x7fc6800b0840 msgr2=0x7fc6800ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 shutdown_connections 2024-01-15T18:05:34.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.535+0000 7fc685a00640 1 -- 172.21.15.2:0/3467851644 wait complete. 2024-01-15T18:05:34.559 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-15T18:05:34.559 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-15T18:05:34.560 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:05:34.560 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:34.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.769+0000 7fd8c1a00640 1 Processor -- start 2024-01-15T18:05:34.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.770+0000 7fd8c1a00640 1 -- start start 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.771+0000 7fd8c1a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc14af80 0x7fd8bc06d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.771+0000 7fd8c1a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd8bc0700d0 0x7fd8bc070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.771+0000 7fd8c1a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8bc0b1c70 con 0x7fd8bc06da60 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.771+0000 7fd8c1a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8bc0b28f0 con 0x7fd8bc0700d0 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.771+0000 7fd8c1a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc070a80 con 0x7fd8bc14af80 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd8bc0700d0 0x7fd8bc070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd8bc0700d0 0x7fd8bc070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36256/0 (socket says 172.21.15.2:36256) 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 -- 172.21.15.2:0/272577467 learned_addr learned my addr 172.21.15.2:0/272577467 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8bbe00640 1 --2- 172.21.15.2:0/272577467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc14af80 0x7fd8bc06d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 -- 172.21.15.2:0/272577467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc14af80 msgr2=0x7fd8bc06d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:34.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 --2- 172.21.15.2:0/272577467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc14af80 0x7fd8bc06d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 -- 172.21.15.2:0/272577467 >> v1:172.21.15.137:6789/0 conn(0x7fd8bc06da60 legacy=0x7fd8bc161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:34.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.772+0000 7fd8c1000640 1 -- 172.21.15.2:0/272577467 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8bc0b1f50 con 0x7fd8bc0700d0 2024-01-15T18:05:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.774+0000 7fd8c1000640 1 --2- 172.21.15.2:0/272577467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc0700d0 0x7fd8bc070540 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009e80 tx=0x7fd8b402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c064ae88226d6126 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.774+0000 7fd8bb400640 1 -- 172.21.15.2:0/272577467 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b402c9c0 con 0x7fd8bc0700d0 2024-01-15T18:05:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.774+0000 7fd8bb400640 1 -- 172.21.15.2:0/272577467 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b402cb20 con 0x7fd8bc0700d0 2024-01-15T18:05:34.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.774+0000 7fd8bb400640 1 -- 172.21.15.2:0/272577467 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b40343f0 con 0x7fd8bc0700d0 2024-01-15T18:05:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 -- 172.21.15.2:0/272577467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc0700d0 msgr2=0x7fd8bc070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/272577467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc0700d0 0x7fd8bc070540 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009e80 tx=0x7fd8b402be20 comp rx=0 tx=0).stop 2024-01-15T18:05:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 -- 172.21.15.2:0/272577467 shutdown_connections 2024-01-15T18:05:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/272577467 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc0700d0 0x7fd8bc070540 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/272577467 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc14af80 0x7fd8bc06d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 -- 172.21.15.2:0/272577467 >> 172.21.15.2:0/272577467 conn(0x7fd8bc0b0840 msgr2=0x7fd8bc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 -- 172.21.15.2:0/272577467 shutdown_connections 2024-01-15T18:05:34.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.775+0000 7fd8c1a00640 1 -- 172.21.15.2:0/272577467 wait complete. 2024-01-15T18:05:34.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.777+0000 7fd8c1a00640 1 Processor -- start 2024-01-15T18:05:34.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.777+0000 7fd8c1a00640 1 -- start start 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8c1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 0x7fd8bc153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8c1a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc0700d0 0x7fd8bc153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8c1a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8bc14af80 0x7fd8bc1fb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8c1a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc155050 con 0x7fd8bc06da60 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8c1a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc153960 con 0x7fd8bc14af80 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8c1a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8bc1fb5a0 con 0x7fd8bc0700d0 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8bbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 0x7fd8bc153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8bbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 0x7fd8bc153e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36262/0 (socket says 172.21.15.2:36262) 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8bbe00640 1 -- 172.21.15.2:0/3192027822 learned_addr learned my addr 172.21.15.2:0/3192027822 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.778+0000 7fd8b3e00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc0700d0 0x7fd8bc153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8bbe00640 1 -- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc0700d0 msgr2=0x7fd8bc153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8bbe00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc0700d0 0x7fd8bc153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8c1000640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8bc14af80 0x7fd8bc1fb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8bbe00640 1 -- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8bc14af80 msgr2=0x7fd8bc1fb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8bbe00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8bc14af80 0x7fd8bc1fb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8bbe00640 1 -- 172.21.15.2:0/3192027822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b4009b30 con 0x7fd8bc06da60 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8b3e00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc0700d0 0x7fd8bc153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8bbe00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 0x7fd8bc153e60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00d800 tx=0x7fd8ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00b6c0 con 0x7fd8bc06da60 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ac00b820 con 0x7fd8bc06da60 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8bc154440 con 0x7fd8bc06da60 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ac00bbb0 con 0x7fd8bc06da60 2024-01-15T18:05:34.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.779+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8bc154a00 con 0x7fd8bc06da60 2024-01-15T18:05:34.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.781+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8bc074770 con 0x7fd8bc06da60 2024-01-15T18:05:34.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.782+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd8ac00bd10 con 0x7fd8bc06da60 2024-01-15T18:05:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.783+0000 7fd8b8c00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd89406bbc0 0x7fd89406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:34.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.783+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fd8ac088c00 con 0x7fd8bc06da60 2024-01-15T18:05:34.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.784+0000 7fd8b3e00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd89406bbc0 0x7fd89406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:34.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.785+0000 7fd8b3e00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd89406bbc0 0x7fd89406e080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0b0690 tx=0x7fd8a4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:34.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.787+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8ac0552d0 con 0x7fd8bc06da60 2024-01-15T18:05:34.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.986+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8bc14a5a0 con 0x7fd8bc06da60 2024-01-15T18:05:34.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.988+0000 7fd8b8c00640 1 -- 172.21.15.2:0/3192027822 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+13055 (secure 0 0 0) 0x7fd8ac0588f0 con 0x7fd8bc06da60 2024-01-15T18:05:34.992 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:34.992 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":65,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:27.460948+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/2855093754":"2024-01-16T18:05:24.435299+0000","172.21.15.2:6834/2855093754":"2024-01-16T18:05:24.435299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd89406bbc0 msgr2=0x7fd89406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd89406bbc0 0x7fd89406e080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc0b0690 tx=0x7fd8a4008040 comp rx=0 tx=0).stop 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 msgr2=0x7fd8bc153e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 0x7fd8bc153e60 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac00d800 tx=0x7fd8ac00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 shutdown_connections 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd89406bbc0 0x7fd89406e080 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8bc14af80 0x7fd8bc1fb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd8bc0700d0 0x7fd8bc153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 --2- 172.21.15.2:0/3192027822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8bc06da60 0x7fd8bc153e60 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 >> 172.21.15.2:0/3192027822 conn(0x7fd8bc0b0840 msgr2=0x7fd8bc162660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 shutdown_connections 2024-01-15T18:05:34.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:34.991+0000 7fd8c1a00640 1 -- 172.21.15.2:0/3192027822 wait complete. 2024-01-15T18:05:35.015 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:35.015 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:35.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.239+0000 7fece7000640 1 Processor -- start 2024-01-15T18:05:35.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.240+0000 7fece7000640 1 -- start start 2024-01-15T18:05:35.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.241+0000 7fece7000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fece01439f0 0x7fece00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.241+0000 7fece7000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fece00b3420 0x7fece00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.241+0000 7fece7000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00ccc70 0x7fece006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.241+0000 7fece7000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece015dd70 con 0x7fece00b3420 2024-01-15T18:05:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.241+0000 7fece7000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece00b38b0 con 0x7fece01439f0 2024-01-15T18:05:35.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.241+0000 7fece7000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece00711b0 con 0x7fece00ccc70 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fece00b3420 0x7fece00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fece00b3420 0x7fece00cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36268/0 (socket says 172.21.15.2:36268) 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 -- 172.21.15.2:0/1036701651 learned_addr learned my addr 172.21.15.2:0/1036701651 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fece5000640 1 --2- 172.21.15.2:0/1036701651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00ccc70 0x7fece006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdfe00640 1 --2- 172.21.15.2:0/1036701651 >> v2:172.21.15.137:3300/0 conn(0x7fece01439f0 0x7fece00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 -- 172.21.15.2:0/1036701651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00ccc70 msgr2=0x7fece006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 --2- 172.21.15.2:0/1036701651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00ccc70 0x7fece006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 -- 172.21.15.2:0/1036701651 >> v2:172.21.15.137:3300/0 conn(0x7fece01439f0 msgr2=0x7fece00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 --2- 172.21.15.2:0/1036701651 >> v2:172.21.15.137:3300/0 conn(0x7fece01439f0 0x7fece00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.242+0000 7fecdf400640 1 -- 172.21.15.2:0/1036701651 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0143dd0 con 0x7fece00b3420 2024-01-15T18:05:35.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.243+0000 7fecdf400640 1 --2- 172.21.15.2:0/1036701651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece00b3420 0x7fece00cc730 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fece00b28d0 tx=0x7fecc802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1368678909295e29 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.244+0000 7fecdea00640 1 -- 172.21.15.2:0/1036701651 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc802ee90 con 0x7fece00b3420 2024-01-15T18:05:35.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.244+0000 7fecdea00640 1 -- 172.21.15.2:0/1036701651 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc8036440 con 0x7fece00b3420 2024-01-15T18:05:35.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.244+0000 7fecdea00640 1 -- 172.21.15.2:0/1036701651 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc80367d0 con 0x7fece00b3420 2024-01-15T18:05:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.244+0000 7fece7000640 1 -- 172.21.15.2:0/1036701651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece00b3420 msgr2=0x7fece00cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.244+0000 7fece7000640 1 --2- 172.21.15.2:0/1036701651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece00b3420 0x7fece00cc730 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fece00b28d0 tx=0x7fecc802e0e0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 -- 172.21.15.2:0/1036701651 shutdown_connections 2024-01-15T18:05:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 --2- 172.21.15.2:0/1036701651 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00ccc70 0x7fece006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 --2- 172.21.15.2:0/1036701651 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece00b3420 0x7fece00cc730 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 --2- 172.21.15.2:0/1036701651 >> v2:172.21.15.137:3300/0 conn(0x7fece01439f0 0x7fece00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 -- 172.21.15.2:0/1036701651 >> 172.21.15.2:0/1036701651 conn(0x7fece00b45f0 msgr2=0x7fece00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 -- 172.21.15.2:0/1036701651 shutdown_connections 2024-01-15T18:05:35.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.245+0000 7fece7000640 1 -- 172.21.15.2:0/1036701651 wait complete. 2024-01-15T18:05:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.246+0000 7fece7000640 1 Processor -- start 2024-01-15T18:05:35.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.247+0000 7fece7000640 1 -- start start 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.247+0000 7fece7000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00b3420 0x7fece0159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.247+0000 7fece7000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fece00ccc70 0x7fece00ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece7000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 0x7fece00ae880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece7000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00acbe0 con 0x7fece01439f0 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece7000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b1310 con 0x7fece00ccc70 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece7000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece00b15f0 con 0x7fece00b3420 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 0x7fece00ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 0x7fece00ae880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36280/0 (socket says 172.21.15.2:36280) 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 -- 172.21.15.2:0/3725327162 learned_addr learned my addr 172.21.15.2:0/3725327162 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fecdfe00640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00b3420 0x7fece0159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fecdf400640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fece00ccc70 0x7fece00ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 -- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00b3420 msgr2=0x7fece0159950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00b3420 0x7fece0159950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 -- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fece00ccc70 msgr2=0x7fece00ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fece00ccc70 0x7fece00ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.248+0000 7fece5000640 1 -- 172.21.15.2:0/3725327162 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc800b770 con 0x7fece01439f0 2024-01-15T18:05:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.249+0000 7fece5000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 0x7fece00ae880 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fecd800eed0 tx=0x7fecd800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.249+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8010040 con 0x7fece01439f0 2024-01-15T18:05:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.249+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd8014400 con 0x7fece01439f0 2024-01-15T18:05:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.249+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece00b18d0 con 0x7fece01439f0 2024-01-15T18:05:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.249+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8009ea0 con 0x7fece01439f0 2024-01-15T18:05:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.249+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece0159e90 con 0x7fece01439f0 2024-01-15T18:05:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.250+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece00acd50 con 0x7fece01439f0 2024-01-15T18:05:35.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.251+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fecd80147b0 con 0x7fece01439f0 2024-01-15T18:05:35.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.252+0000 7fecd7e00640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fecb806bbc0 0x7fecb806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.252+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fecd808b3a0 con 0x7fece01439f0 2024-01-15T18:05:35.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.253+0000 7fecdfe00640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fecb806bbc0 0x7fecb806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.254+0000 7fecdfe00640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fecb806bbc0 0x7fecb806e080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009a30 tx=0x7fecd0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.256+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fecd8055480 con 0x7fece01439f0 2024-01-15T18:05:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.448+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fece00686f0 con 0x7fece01439f0 2024-01-15T18:05:35.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.449+0000 7fecd7e00640 1 -- 172.21.15.2:0/3725327162 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fecd80180a0 con 0x7fece01439f0 2024-01-15T18:05:35.453 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:05:35.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fecb806bbc0 msgr2=0x7fecb806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fecb806bbc0 0x7fecb806e080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009a30 tx=0x7fecd0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:35.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 msgr2=0x7fece00ae880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 0x7fece00ae880 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fecd800eed0 tx=0x7fecd800b480 comp rx=0 tx=0).stop 2024-01-15T18:05:35.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 shutdown_connections 2024-01-15T18:05:35.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fecb806bbc0 0x7fecb806e080 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fece01439f0 0x7fece00ae880 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fece00ccc70 0x7fece00ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 --2- 172.21.15.2:0/3725327162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fece00b3420 0x7fece0159950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 >> 172.21.15.2:0/3725327162 conn(0x7fece00b45f0 msgr2=0x7fece01426f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 shutdown_connections 2024-01-15T18:05:35.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.452+0000 7fece7000640 1 -- 172.21.15.2:0/3725327162 wait complete. 2024-01-15T18:05:35.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-15T18:05:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.696+0000 7f1da4000640 1 Processor -- start 2024-01-15T18:05:35.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.697+0000 7f1da4000640 1 -- start start 2024-01-15T18:05:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.698+0000 7f1da4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d9c146dd0 0x7f1d9c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.698+0000 7f1da4000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c06cdf0 con 0x7f1d9c06c9b0 2024-01-15T18:05:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.698+0000 7f1da4000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9c0ac870 con 0x7f1d9c1469f0 2024-01-15T18:05:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.698+0000 7f1da4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c15d970 con 0x7f1d9c146dd0 2024-01-15T18:05:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.699+0000 7f1da2000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1d9c06c9b0 0x7f1d9c0b34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57488/0 (socket says 172.21.15.2:57488) 2024-01-15T18:05:35.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.699+0000 7f1da2000640 1 -- 172.21.15.2:0/584578883 learned_addr learned my addr 172.21.15.2:0/584578883 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:35.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.699+0000 7f1da0c00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d9c146dd0 0x7f1d9c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.699+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766679741 0 0) 0x7f1d9c06cdf0 con 0x7f1d9c06c9b0 2024-01-15T18:05:35.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.699+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d7c003520 con 0x7f1d9c06c9b0 2024-01-15T18:05:35.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 296504084 0 0) 0x7f1d7c003520 con 0x7f1d9c06c9b0 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d9c146dd0 msgr2=0x7f1d9c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d9c146dd0 0x7f1d9c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 >> v1:172.21.15.137:6789/0 conn(0x7f1d9c1469f0 legacy=0x7f1d9c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c15dc30 con 0x7f1d9c06c9b0 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1d94001670 con 0x7f1d9c06c9b0 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 >> v1:172.21.15.2:6789/0 conn(0x7f1d9c06c9b0 legacy=0x7f1d9c0b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0039a0 0x7f1d7c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d7c004350 0x7f1d7c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c005020 0x7f1d7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c15d970 con 0x7f1d7c004350 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c00bec0 con 0x7f1d7c005020 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c00c050 con 0x7f1d7c0039a0 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.700+0000 7f1da0c00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d7c004350 0x7f1d7c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da1600640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0039a0 0x7f1d7c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da2000640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c005020 0x7f1d7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da0c00640 1 -- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0039a0 msgr2=0x7f1d7c003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da0c00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0039a0 0x7f1d7c003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da0c00640 1 -- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c005020 msgr2=0x7f1d7c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da0c00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c005020 0x7f1d7c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da0c00640 1 -- 172.21.15.2:0/584578883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d7c003520 con 0x7f1d7c004350 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da2000640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d7c005020 0x7f1d7c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:35.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.701+0000 7f1da1600640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c0039a0 0x7f1d7c003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:35.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.702+0000 7f1da0c00640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d7c004350 0x7f1d7c004820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009fc0 tx=0x7f1d7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1292580b415baa87 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.702+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d8402cbf0 con 0x7f1d7c004350 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.702+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8400f040 con 0x7f1d7c004350 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.702+0000 7f1d9be00640 1 -- 172.21.15.2:0/584578883 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d84034440 con 0x7f1d7c004350 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da4000640 1 -- 172.21.15.2:0/584578883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d7c004350 msgr2=0x7f1d7c004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da4000640 1 --2- 172.21.15.2:0/584578883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d7c004350 0x7f1d7c004820 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009fc0 tx=0x7f1d7c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da2000640 1 -- 172.21.15.2:0/584578883 reap_dead start 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da4000640 1 -- 172.21.15.2:0/584578883 shutdown_connections 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da4000640 1 -- 172.21.15.2:0/584578883 >> 172.21.15.2:0/584578883 conn(0x7f1d9c06da20 msgr2=0x7f1d9c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da4000640 1 -- 172.21.15.2:0/584578883 shutdown_connections 2024-01-15T18:05:35.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.703+0000 7f1da4000640 1 -- 172.21.15.2:0/584578883 wait complete. 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.705+0000 7f1da4000640 1 Processor -- start 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.705+0000 7f1da4000640 1 -- start start 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c004350 0x7f1d9c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d9c13e1a0 0x7f1d9c071c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da4000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 0x7f1d9c0afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da4000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c13cbe0 con 0x7f1d9c072160 2024-01-15T18:05:35.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c0725e0 con 0x7f1d9c13e1a0 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9c1434b0 con 0x7f1d7c004350 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 0x7f1d9c0afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 0x7f1d9c0afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36292/0 (socket says 172.21.15.2:36292) 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.706+0000 7f1da2000640 1 -- 172.21.15.2:0/1035135042 learned_addr learned my addr 172.21.15.2:0/1035135042 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da1600640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c004350 0x7f1d9c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da2000640 1 -- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c004350 msgr2=0x7f1d9c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da2000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c004350 0x7f1d9c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da0c00640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d9c13e1a0 0x7f1d9c071c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da2000640 1 -- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d9c13e1a0 msgr2=0x7f1d9c071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da2000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d9c13e1a0 0x7f1d9c071c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da2000640 1 -- 172.21.15.2:0/1035135042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9c13c060 con 0x7f1d9c072160 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da1600640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c004350 0x7f1d9c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da0c00640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d9c13e1a0 0x7f1d9c071c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:35.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da2000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 0x7f1d9c0afc90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c06e560 tx=0x7f1d9400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94007440 con 0x7f1d9c072160 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d940075a0 con 0x7f1d9c072160 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d94004a00 con 0x7f1d9c072160 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.707+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d9c13c340 con 0x7f1d9c072160 2024-01-15T18:05:35.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.708+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d9c13c900 con 0x7f1d9c072160 2024-01-15T18:05:35.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.709+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d9c0686f0 con 0x7f1d9c072160 2024-01-15T18:05:35.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.710+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1d94004b60 con 0x7f1d9c072160 2024-01-15T18:05:35.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.710+0000 7f1d99600640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d7406bbf0 0x7f1d7406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:35.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.711+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f1d94088060 con 0x7f1d9c072160 2024-01-15T18:05:35.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.711+0000 7f1da1600640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d7406bbf0 0x7f1d7406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:35.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.712+0000 7f1da1600640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d7406bbf0 0x7f1d7406e0b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0016f0 tx=0x7f1d8c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:35.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.715+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d94054980 con 0x7f1d9c072160 2024-01-15T18:05:35.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:35.979+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7f1d9c0b3770 con 0x7f1d9c072160 2024-01-15T18:05:36.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.546+0000 7f1d99600640 1 -- 172.21.15.2:0/1035135042 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v381) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1d94057fa0 con 0x7f1d9c072160 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d7406bbf0 msgr2=0x7f1d7406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d7406bbf0 0x7f1d7406e0b0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f1d8c0016f0 tx=0x7f1d8c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 msgr2=0x7f1d9c0afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 0x7f1d9c0afc90 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c06e560 tx=0x7f1d9400d9d0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 shutdown_connections 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1d7406bbf0 0x7f1d7406e0b0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1d9c072160 0x7f1d9c0afc90 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1d9c13e1a0 0x7f1d9c071c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 --2- 172.21.15.2:0/1035135042 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1d7c004350 0x7f1d9c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 >> 172.21.15.2:0/1035135042 conn(0x7f1d9c06da20 msgr2=0x7f1d9c15fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 shutdown_connections 2024-01-15T18:05:36.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.550+0000 7f1da4000640 1 -- 172.21.15.2:0/1035135042 wait complete. 2024-01-15T18:05:36.576 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:05:36.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.795+0000 7fb7f7c00640 1 Processor -- start 2024-01-15T18:05:36.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.796+0000 7fb7f7c00640 1 -- start start 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.798+0000 7fb7f7c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f015ae80 0x7fb7f00acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.798+0000 7fb7f7c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb7f00b2b40 0x7fb7f006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.798+0000 7fb7f7c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f006d290 con 0x7fb7f006c8f0 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.798+0000 7fb7f7c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f015e840 con 0x7fb7f00b2b40 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.798+0000 7fb7f7c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f015dbc0 con 0x7fb7f015ae80 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb7f00b2b40 0x7fb7f006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f015ae80 0x7fb7f00acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7efe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb7f006c8f0 0x7fb7f00b2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:59164/0 (socket says 172.21.15.2:59164) 2024-01-15T18:05:36.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7efe00640 1 -- 172.21.15.2:0/1934288062 learned_addr learned my addr 172.21.15.2:0/1934288062 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:36.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5c00640 1 --2- 172.21.15.2:0/1934288062 >> v2:172.21.15.2:3300/0 conn(0x7fb7f00b2b40 0x7fb7f006cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36296/0 (socket says 172.21.15.2:36296) 2024-01-15T18:05:36.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5c00640 1 -- 172.21.15.2:0/1934288062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f015ae80 msgr2=0x7fb7f00acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:36.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5c00640 1 --2- 172.21.15.2:0/1934288062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f015ae80 0x7fb7f00acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5c00640 1 -- 172.21.15.2:0/1934288062 >> v1:172.21.15.137:6789/0 conn(0x7fb7f006c8f0 legacy=0x7fb7f00b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5c00640 1 -- 172.21.15.2:0/1934288062 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f015dea0 con 0x7fb7f00b2b40 2024-01-15T18:05:36.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.799+0000 7fb7f5200640 1 --2- 172.21.15.2:0/1934288062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f015ae80 0x7fb7f00acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:36.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.801+0000 7fb7f5c00640 1 --2- 172.21.15.2:0/1934288062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f00b2b40 0x7fb7f006cd50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009d90 tx=0x7fb7e802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c66bfd049dc8926 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.801+0000 7fb7ef400640 1 -- 172.21.15.2:0/1934288062 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e802cc50 con 0x7fb7f00b2b40 2024-01-15T18:05:36.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.801+0000 7fb7ef400640 1 -- 172.21.15.2:0/1934288062 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e802cdb0 con 0x7fb7f00b2b40 2024-01-15T18:05:36.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.801+0000 7fb7ef400640 1 -- 172.21.15.2:0/1934288062 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8034660 con 0x7fb7f00b2b40 2024-01-15T18:05:36.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 -- 172.21.15.2:0/1934288062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f00b2b40 msgr2=0x7fb7f006cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/1934288062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f00b2b40 0x7fb7f006cd50 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009d90 tx=0x7fb7e802bea0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 -- 172.21.15.2:0/1934288062 shutdown_connections 2024-01-15T18:05:36.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/1934288062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f00b2b40 0x7fb7f006cd50 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/1934288062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f015ae80 0x7fb7f00acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 -- 172.21.15.2:0/1934288062 >> 172.21.15.2:0/1934288062 conn(0x7fb7f00b45f0 msgr2=0x7fb7f00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:36.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 -- 172.21.15.2:0/1934288062 shutdown_connections 2024-01-15T18:05:36.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.802+0000 7fb7f7c00640 1 -- 172.21.15.2:0/1934288062 wait complete. 2024-01-15T18:05:36.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.804+0000 7fb7f7c00640 1 Processor -- start 2024-01-15T18:05:36.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.804+0000 7fb7f7c00640 1 -- start start 2024-01-15T18:05:36.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f7c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 0x7fb7f00b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f7c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f00b2b40 0x7fb7f01f06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f7c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7f015ae80 0x7fb7f014c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f7c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f014d460 con 0x7fb7f006c8f0 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f7c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00afde0 con 0x7fb7f015ae80 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f7c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f014cea0 con 0x7fb7f00b2b40 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 0x7fb7f00b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 0x7fb7f00b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36300/0 (socket says 172.21.15.2:36300) 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f5200640 1 -- 172.21.15.2:0/3708794236 learned_addr learned my addr 172.21.15.2:0/3708794236 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7f5200640 1 -- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f00b2b40 msgr2=0x7fb7f01f06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.805+0000 7fb7efe00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f00b2b40 0x7fb7f01f06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7f015ae80 0x7fb7f014c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5200640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f00b2b40 0x7fb7f01f06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5200640 1 -- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7f015ae80 msgr2=0x7fb7f014c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5200640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7f015ae80 0x7fb7f014c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:36.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5200640 1 -- 172.21.15.2:0/3708794236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e8009a70 con 0x7fb7f006c8f0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7f015ae80 0x7fb7f014c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f5200640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 0x7fb7f00b07f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000d7d0 tx=0x7fb7e000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0004030 con 0x7fb7f006c8f0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e0004190 con 0x7fb7f006c8f0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0004520 con 0x7fb7f006c8f0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f014d180 con 0x7fb7f006c8f0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.806+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f00b02d0 con 0x7fb7f006c8f0 2024-01-15T18:05:36.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.808+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f00686f0 con 0x7fb7f006c8f0 2024-01-15T18:05:36.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.808+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb7e00096a0 con 0x7fb7f006c8f0 2024-01-15T18:05:36.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.809+0000 7fb7ecc00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb7c806bbc0 0x7fb7c806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:36.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.809+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fb7e00886b0 con 0x7fb7f006c8f0 2024-01-15T18:05:36.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.810+0000 7fb7efe00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb7c806bbc0 0x7fb7c806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:36.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.811+0000 7fb7efe00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb7c806bbc0 0x7fb7c806e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb7f006b490 tx=0x7fb7d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:36.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:36.814+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb7e0054f70 con 0x7fb7f006c8f0 2024-01-15T18:05:37.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.007+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb7f00ad770 con 0x7fb7f006c8f0 2024-01-15T18:05:37.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.015+0000 7fb7ecc00640 1 -- 172.21.15.2:0/3708794236 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb7e0058590 con 0x7fb7f006c8f0 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.017+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb7c806bbc0 msgr2=0x7fb7c806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.017+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb7c806bbc0 0x7fb7c806e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fb7f006b490 tx=0x7fb7d8009040 comp rx=0 tx=0).stop 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.017+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 msgr2=0x7fb7f00b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.017+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 0x7fb7f00b07f0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fb7e000d7d0 tx=0x7fb7e000dc90 comp rx=0 tx=0).stop 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 shutdown_connections 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb7c806bbc0 0x7fb7c806e080 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7f015ae80 0x7fb7f014c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb7f00b2b40 0x7fb7f01f06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 --2- 172.21.15.2:0/3708794236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb7f006c8f0 0x7fb7f00b07f0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 >> 172.21.15.2:0/3708794236 conn(0x7fb7f00b45f0 msgr2=0x7fb7f00ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 shutdown_connections 2024-01-15T18:05:37.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.018+0000 7fb7f7c00640 1 -- 172.21.15.2:0/3708794236 wait complete. 2024-01-15T18:05:37.040 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:05:37.041 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:05:37.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:05:37.278 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:05:37.278 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:05:37.279 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:37.337 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:05:37.337 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:05:37.337 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:37.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:05:37.395 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:05:37.395 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:37.453 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:05:37.453 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:05:37.453 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:37.509 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:05:37.509 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:05:37.509 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.734+0000 7f57c3e00640 1 Processor -- start 2024-01-15T18:05:37.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.735+0000 7f57c3e00640 1 -- start start 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.736+0000 7f57c3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f57c4141880 0x7f57c4070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.736+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 0x7f57c40cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.736+0000 7f57c3e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c406dae0 con 0x7f57c415ec60 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.736+0000 7f57c3e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c4071f20 con 0x7f57c4141880 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.736+0000 7f57c3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c406c6b0 con 0x7f57c415f0e0 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.737+0000 7f57b9e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f57c415ec60 0x7f57c40cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57504/0 (socket says 172.21.15.2:57504) 2024-01-15T18:05:37.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.737+0000 7f57b9e00640 1 -- 172.21.15.2:0/1015453838 learned_addr learned my addr 172.21.15.2:0/1015453838 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.737+0000 7f57c2a00640 1 --2- 172.21.15.2:0/1015453838 >> v2:172.21.15.137:3300/0 conn(0x7f57c4141880 0x7f57c4070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.737+0000 7f57c3400640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 0x7f57c40cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.737+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601924330 0 0) 0x7f57c406dae0 con 0x7f57c415ec60 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.737+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a4003520 con 0x7f57c415ec60 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441135348 0 0) 0x7f57a4003520 con 0x7f57c415ec60 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 msgr2=0x7f57c40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 0x7f57c40cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 >> v2:172.21.15.137:3300/0 conn(0x7f57c4141880 msgr2=0x7f57c4070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 --2- 172.21.15.2:0/1015453838 >> v2:172.21.15.137:3300/0 conn(0x7f57c4141880 0x7f57c4070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c415f570 con 0x7f57c415ec60 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f57ac001670 con 0x7f57c415ec60 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 >> v1:172.21.15.2:6789/0 conn(0x7f57c415ec60 legacy=0x7f57c40cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 0x7f57a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57a4003f70 0x7f57a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57a4004d00 0x7f57a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c4071f20 con 0x7f57a4004d00 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c406c6b0 con 0x7f57a4003f70 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a400ba60 con 0x7f57c415f0e0 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c3400640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57a4004d00 0x7f57a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.738+0000 7f57c2a00640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 0x7f57a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57b9e00640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57a4003f70 0x7f57a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57c3400640 1 -- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 msgr2=0x7f57a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57c3400640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415f0e0 0x7f57a40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57c3400640 1 -- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57a4003f70 msgr2=0x7f57a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57c3400640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57a4003f70 0x7f57a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57c3400640 1 -- 172.21.15.2:0/1015453838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4003520 con 0x7f57a4004d00 2024-01-15T18:05:37.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.739+0000 7f57b9e00640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57a4003f70 0x7f57a40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.740+0000 7f57c3400640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57a4004d00 0x7f57a4005180 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009ac0 tx=0x7f57a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df6e70d019cde55 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.740+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc02c6b0 con 0x7f57a4004d00 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.740+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc02c810 con 0x7f57a4004d00 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.740+0000 7f57c2000640 1 -- 172.21.15.2:0/1015453838 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc02cba0 con 0x7f57a4004d00 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3e00640 1 -- 172.21.15.2:0/1015453838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57a4004d00 msgr2=0x7f57a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3e00640 1 --2- 172.21.15.2:0/1015453838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57a4004d00 0x7f57a4005180 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f57bc009ac0 tx=0x7f57a40057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3400640 1 -- 172.21.15.2:0/1015453838 reap_dead start 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3e00640 1 -- 172.21.15.2:0/1015453838 shutdown_connections 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3e00640 1 -- 172.21.15.2:0/1015453838 >> 172.21.15.2:0/1015453838 conn(0x7f57c40b0840 msgr2=0x7f57c40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3e00640 1 -- 172.21.15.2:0/1015453838 shutdown_connections 2024-01-15T18:05:37.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.741+0000 7f57c3e00640 1 -- 172.21.15.2:0/1015453838 wait complete. 2024-01-15T18:05:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.743+0000 7f57c3e00640 1 Processor -- start 2024-01-15T18:05:37.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.743+0000 7f57c3e00640 1 -- start start 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57a4004d00 0x7f57c40b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c40cdca0 0x7f57c415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 0x7f57c413fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c40b28c0 con 0x7f57c40ce120 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c40b1930 con 0x7f57c40cdca0 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c415b490 con 0x7f57a4004d00 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 0x7f57c413fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 0x7f57c413fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36322/0 (socket says 172.21.15.2:36322) 2024-01-15T18:05:37.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 -- 172.21.15.2:0/3608991774 learned_addr learned my addr 172.21.15.2:0/3608991774 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57b9e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c40cdca0 0x7f57c415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c2a00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57a4004d00 0x7f57c40b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 -- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57a4004d00 msgr2=0x7f57c40b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57a4004d00 0x7f57c40b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 -- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c40cdca0 msgr2=0x7f57c415aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.744+0000 7f57c3400640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c40cdca0 0x7f57c415aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57c3400640 1 -- 172.21.15.2:0/3608991774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c415b770 con 0x7f57c40ce120 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57c2a00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57a4004d00 0x7f57c40b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:37.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57c3400640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 0x7f57c413fa20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f57bc037040 tx=0x7f57bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc02cd90 con 0x7f57c40ce120 2024-01-15T18:05:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57bc003610 con 0x7f57c40ce120 2024-01-15T18:05:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c40b1b80 con 0x7f57c40ce120 2024-01-15T18:05:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0039c0 con 0x7f57c40ce120 2024-01-15T18:05:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.745+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c41407d0 con 0x7f57c40ce120 2024-01-15T18:05:37.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.747+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c40b20a0 con 0x7f57c40ce120 2024-01-15T18:05:37.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.747+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f57bc003b20 con 0x7f57c40ce120 2024-01-15T18:05:37.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.748+0000 7f57bbe00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06bbf0 0x7f579c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:37.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.748+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f57bc0a9580 con 0x7f57c40ce120 2024-01-15T18:05:37.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.749+0000 7f57c2a00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06bbf0 0x7f579c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:37.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.750+0000 7f57c2a00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06bbf0 0x7f579c06e0b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f57c41e99e0 tx=0x7f57b400b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:37.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.753+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57c40b20a0 con 0x7f57c40ce120 2024-01-15T18:05:37.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.986+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c40b20a0 con 0x7f57c40ce120 2024-01-15T18:05:37.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.987+0000 7f57bbe00640 1 -- 172.21.15.2:0/3608991774 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f57bc039e80 con 0x7f57c40ce120 2024-01-15T18:05:37.991 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:37.991 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5611,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1133084700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1133084700},{"type":"v1","addr":"172.21.15.137:6836","nonce":1133084700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":5614,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/302473054","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":302473054},{"type":"v1","addr":"172.21.15.137:6837","nonce":302473054}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":31},{"gid":6303,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2347293827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2347293827},{"type":"v1","addr":"172.21.15.2:6833","nonce":2347293827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":6867,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/1512269903","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":1512269903},{"type":"v1","addr":"172.21.15.2:6835","nonce":1512269903}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2024-01-15T18:05:37.991 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 39 2024-01-15T18:05:37.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06bbf0 msgr2=0x7f579c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06bbf0 0x7f579c06e0b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f57c41e99e0 tx=0x7f57b400b040 comp rx=0 tx=0).stop 2024-01-15T18:05:37.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 msgr2=0x7f57c413fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:37.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 0x7f57c413fa20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f57bc037040 tx=0x7f57bc004730 comp rx=0 tx=0).stop 2024-01-15T18:05:37.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 shutdown_connections 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06bbf0 0x7f579c06e0b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c40ce120 0x7f57c413fa20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.989+0000 7f57c3e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c40cdca0 0x7f57c415aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.990+0000 7f57c3e00640 1 --2- 172.21.15.2:0/3608991774 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57a4004d00 0x7f57c40b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.990+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 >> 172.21.15.2:0/3608991774 conn(0x7f57c40b0840 msgr2=0x7f57c40af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.990+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 shutdown_connections 2024-01-15T18:05:37.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:37.990+0000 7f57c3e00640 1 -- 172.21.15.2:0/3608991774 wait complete. 2024-01-15T18:05:38.014 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:05:38.014 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:05:38.014 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:05:38.015 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:38.015 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:05:38.015 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:05:38.015 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:05:38.015 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:38.015 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:05:38.015 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:05:38.016 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:05:38.016 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:38.016 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:05:38.016 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:05:38.016 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:05:38.016 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:05:38.032 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:05:38.032 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:05:38.033 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:05:38.033 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:05:38.038 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:05:38.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:05:38.047 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:05:38.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:05:38.079 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:05:38.080 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:38.085 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:05:38.086 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:38.094 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:05:38.094 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:38.100 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:05:38.100 DEBUG:teuthology.parallel:result is None 2024-01-15T18:05:38.100 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:05:38.246 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:05:38.252 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:05:38.262 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T18:05:38.262 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:05:38.264 INFO:tasks.ceph.mds.b.smithi137.stderr:2024-01-15T18:05:38.261+0000 7fa51f800640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-01-15T18:05:38.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.283+0000 7f526e800640 1 Processor -- start 2024-01-15T18:05:38.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.283+0000 7f526e800640 1 -- start start 2024-01-15T18:05:38.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f526e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52681615d0 0x7f52680ce110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f526e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f52680b18c0 0x7f526806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f526e800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52680b3010 con 0x7f52680ce810 2024-01-15T18:05:38.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f526e800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52680b0de0 con 0x7f52680b18c0 2024-01-15T18:05:38.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f526e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52680b10c0 con 0x7f52681615d0 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f525fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f52680b18c0 0x7f526806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f525fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f52680b18c0 0x7f526806c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36358/0 (socket says 172.21.15.2:36358) 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f525fe00640 1 -- 172.21.15.2:0/2886659491 learned_addr learned my addr 172.21.15.2:0/2886659491 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f5267400640 1 --2- 172.21.15.2:0/2886659491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52681615d0 0x7f52680ce110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f525fe00640 1 -- 172.21.15.2:0/2886659491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52681615d0 msgr2=0x7f52680ce110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.285+0000 7f525fe00640 1 --2- 172.21.15.2:0/2886659491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52681615d0 0x7f52680ce110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.286+0000 7f525fe00640 1 -- 172.21.15.2:0/2886659491 >> v1:172.21.15.137:6789/0 conn(0x7f52680ce810 legacy=0x7f5268070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.286+0000 7f525fe00640 1 -- 172.21.15.2:0/2886659491 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680b13a0 con 0x7f52680b18c0 2024-01-15T18:05:38.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.287+0000 7f525fe00640 1 --2- 172.21.15.2:0/2886659491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680b18c0 0x7f526806c8b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f5250009870 tx=0x7f525002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=334ad85f205be05f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.287+0000 7f5266a00640 1 -- 172.21.15.2:0/2886659491 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250030840 con 0x7f52680b18c0 2024-01-15T18:05:38.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.287+0000 7f5266a00640 1 -- 172.21.15.2:0/2886659491 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52500309a0 con 0x7f52680b18c0 2024-01-15T18:05:38.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.287+0000 7f5266a00640 1 -- 172.21.15.2:0/2886659491 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250030d30 con 0x7f52680b18c0 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.287+0000 7f526e800640 1 -- 172.21.15.2:0/2886659491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680b18c0 msgr2=0x7f526806c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.287+0000 7f526e800640 1 --2- 172.21.15.2:0/2886659491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680b18c0 0x7f526806c8b0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f5250009870 tx=0x7f525002b980 comp rx=0 tx=0).stop 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.288+0000 7f526e800640 1 -- 172.21.15.2:0/2886659491 shutdown_connections 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.288+0000 7f526e800640 1 --2- 172.21.15.2:0/2886659491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680b18c0 0x7f526806c8b0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.288+0000 7f526e800640 1 --2- 172.21.15.2:0/2886659491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52681615d0 0x7f52680ce110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.288+0000 7f526e800640 1 -- 172.21.15.2:0/2886659491 >> 172.21.15.2:0/2886659491 conn(0x7f52680b46d0 msgr2=0x7f52680b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.288+0000 7f526e800640 1 -- 172.21.15.2:0/2886659491 shutdown_connections 2024-01-15T18:05:38.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.288+0000 7f526e800640 1 -- 172.21.15.2:0/2886659491 wait complete. 2024-01-15T18:05:38.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.290+0000 7f526e800640 1 Processor -- start 2024-01-15T18:05:38.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.290+0000 7f526e800640 1 -- start start 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f526e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680b18c0 0x7f526815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f526e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 0x7f526815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f526e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52681615d0 0x7f526815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f526e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52681f9190 con 0x7f52680ce810 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f526e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f526815a8a0 con 0x7f52681615d0 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f526e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52681f8510 con 0x7f52680b18c0 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 0x7f526815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 0x7f526815bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36360/0 (socket says 172.21.15.2:36360) 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 -- 172.21.15.2:0/1020416292 learned_addr learned my addr 172.21.15.2:0/1020416292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f5267400640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680b18c0 0x7f526815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 -- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680b18c0 msgr2=0x7f526815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680b18c0 0x7f526815a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 -- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52681615d0 msgr2=0x7f526815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52681615d0 0x7f526815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.291+0000 7f525fe00640 1 -- 172.21.15.2:0/1020416292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52500094d0 con 0x7f52680ce810 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.292+0000 7f525fe00640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 0x7f526815bb40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5250009870 tx=0x7f5250004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.292+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250039c90 con 0x7f52680ce810 2024-01-15T18:05:38.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.292+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5250039df0 con 0x7f52680ce810 2024-01-15T18:05:38.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.292+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250043640 con 0x7f52680ce810 2024-01-15T18:05:38.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.292+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52681f87f0 con 0x7f52680ce810 2024-01-15T18:05:38.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.292+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52681f8db0 con 0x7f52680ce810 2024-01-15T18:05:38.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.294+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52681f92d0 con 0x7f52680ce810 2024-01-15T18:05:38.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.295+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f52500437a0 con 0x7f52680ce810 2024-01-15T18:05:38.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.296+0000 7f525f400640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bc10 0x7f524006e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.296+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f525003e070 con 0x7f52680ce810 2024-01-15T18:05:38.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.296+0000 7f5267400640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bc10 0x7f524006e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.297+0000 7f5267400640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bc10 0x7f524006e0d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5258009a30 tx=0x7f5258008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.299+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5250078710 con 0x7f52680ce810 2024-01-15T18:05:38.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.507+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f52681f92d0 con 0x7f52680ce810 2024-01-15T18:05:38.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.577+0000 7f525f400640 1 -- 172.21.15.2:0/1020416292 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7f52681f92d0 con 0x7f52680ce810 2024-01-15T18:05:38.581 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bc10 msgr2=0x7f524006e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bc10 0x7f524006e0d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5258009a30 tx=0x7f5258008040 comp rx=0 tx=0).stop 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 msgr2=0x7f526815bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 0x7f526815bb40 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f5250009870 tx=0x7f5250004730 comp rx=0 tx=0).stop 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 shutdown_connections 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bc10 0x7f524006e0d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52681615d0 0x7f526815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52680ce810 0x7f526815bb40 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 --2- 172.21.15.2:0/1020416292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680b18c0 0x7f526815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.582+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 >> 172.21.15.2:0/1020416292 conn(0x7f52680b46d0 msgr2=0x7f52680b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.583+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 shutdown_connections 2024-01-15T18:05:38.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.583+0000 7f526e800640 1 -- 172.21.15.2:0/1020416292 wait complete. 2024-01-15T18:05:38.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:05:38.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.818+0000 7fe360800640 1 Processor -- start 2024-01-15T18:05:38.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.818+0000 7fe360800640 1 -- start start 2024-01-15T18:05:38.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.820+0000 7fe360800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe35815c0e0 0x7fe3580cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.820+0000 7fe360800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815b7d0 0x7fe35806c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.820+0000 7fe360800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe35815cb00 con 0x7fe35815c4c0 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.820+0000 7fe360800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3580ccec0 con 0x7fe35815c0e0 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.820+0000 7fe360800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3580cd1a0 con 0x7fe35815b7d0 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.820+0000 7fe35d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815b7d0 0x7fe35806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe35815c0e0 0x7fe3580cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815b7d0 0x7fe35806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:42416/0 (socket says 172.21.15.2:42416) 2024-01-15T18:05:38.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35d400640 1 -- 172.21.15.2:0/3729738064 learned_addr learned my addr 172.21.15.2:0/3729738064 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:38.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35de00640 1 -- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815b7d0 msgr2=0x7fe35806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:38.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35de00640 1 --2- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815b7d0 0x7fe35806c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35de00640 1 -- 172.21.15.2:0/3729738064 >> v1:172.21.15.137:6789/0 conn(0x7fe35815c4c0 legacy=0x7fe35815ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.821+0000 7fe35de00640 1 -- 172.21.15.2:0/3729738064 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3580cd480 con 0x7fe35815c0e0 2024-01-15T18:05:38.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.823+0000 7fe35de00640 1 --2- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c0e0 0x7fe3580cc5c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe3480097a0 tx=0x7fe34802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92e4bbdb8242022b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.823+0000 7fe35ca00640 1 -- 172.21.15.2:0/3729738064 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34802d040 con 0x7fe35815c0e0 2024-01-15T18:05:38.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.823+0000 7fe35ca00640 1 -- 172.21.15.2:0/3729738064 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348031810 con 0x7fe35815c0e0 2024-01-15T18:05:38.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.823+0000 7fe35ca00640 1 -- 172.21.15.2:0/3729738064 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348031ba0 con 0x7fe35815c0e0 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 -- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c0e0 msgr2=0x7fe3580cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 --2- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c0e0 0x7fe3580cc5c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fe3480097a0 tx=0x7fe34802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 -- 172.21.15.2:0/3729738064 shutdown_connections 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 --2- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815b7d0 0x7fe35806c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 --2- 172.21.15.2:0/3729738064 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c0e0 0x7fe3580cc5c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 -- 172.21.15.2:0/3729738064 >> 172.21.15.2:0/3729738064 conn(0x7fe35806da20 msgr2=0x7fe35806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 -- 172.21.15.2:0/3729738064 shutdown_connections 2024-01-15T18:05:38.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.824+0000 7fe360800640 1 -- 172.21.15.2:0/3729738064 wait complete. 2024-01-15T18:05:38.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.827+0000 7fe360800640 1 Processor -- start 2024-01-15T18:05:38.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.827+0000 7fe360800640 1 -- start start 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe360800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 0x7fe3581550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe360800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815c0e0 0x7fe3581e6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe360800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c4c0 0x7fe3580b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe360800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3581e70d0 con 0x7fe35815b7d0 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe360800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3581e6800 con 0x7fe35815c4c0 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe360800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3581e5000 con 0x7fe35815c0e0 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.828+0000 7fe35de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 0x7fe3581550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 0x7fe3581550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36364/0 (socket says 172.21.15.2:36364) 2024-01-15T18:05:38.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 -- 172.21.15.2:0/611997403 learned_addr learned my addr 172.21.15.2:0/611997403 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:38.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 -- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815c0e0 msgr2=0x7fe3581e6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815c0e0 0x7fe3581e6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 -- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c4c0 msgr2=0x7fe3580b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c4c0 0x7fe3580b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 -- 172.21.15.2:0/611997403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348009400 con 0x7fe35815b7d0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.829+0000 7fe35de00640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 0x7fe3581550e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe3480098c0 tx=0x7fe3480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.830+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34803ace0 con 0x7fe35815b7d0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.830+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34803ae60 con 0x7fe35815b7d0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.830+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe348044640 con 0x7fe35815b7d0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.830+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3581e52e0 con 0x7fe35815b7d0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.830+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3581e58a0 con 0x7fe35815b7d0 2024-01-15T18:05:38.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.831+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3580cbf70 con 0x7fe35815b7d0 2024-01-15T18:05:38.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.832+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe34804d440 con 0x7fe35815b7d0 2024-01-15T18:05:38.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.833+0000 7fe356000640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe33006bbc0 0x7fe33006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:38.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.834+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fe3480aa730 con 0x7fe35815b7d0 2024-01-15T18:05:38.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.834+0000 7fe35d400640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe33006bbc0 0x7fe33006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:38.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.835+0000 7fe35d400640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe33006bbc0 0x7fe33006e080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe3400094a0 tx=0x7fe3400093f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:38.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:38.838+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe348076ff0 con 0x7fe35815b7d0 2024-01-15T18:05:39.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.101+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe3580686f0 con 0x7fe35815b7d0 2024-01-15T18:05:39.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.102+0000 7fe356000640 1 -- 172.21.15.2:0/611997403 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe348035090 con 0x7fe35815b7d0 2024-01-15T18:05:39.105 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:39.105 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.106 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.107 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.108 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T18:05:39.109 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T18:05:39.110 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.111 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:05:39.112 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow r", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T18:05:39.114 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:05:39.116 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:05:39.117 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.117 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:39.117 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T18:05:39.117 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe33006bbc0 msgr2=0x7fe33006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe33006bbc0 0x7fe33006e080 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fe3400094a0 tx=0x7fe3400093f0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 msgr2=0x7fe3581550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 0x7fe3581550e0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fe3480098c0 tx=0x7fe3480046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 shutdown_connections 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe33006bbc0 0x7fe33006e080 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe35815c4c0 0x7fe3580b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe35815c0e0 0x7fe3581e6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 --2- 172.21.15.2:0/611997403 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe35815b7d0 0x7fe3581550e0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 >> 172.21.15.2:0/611997403 conn(0x7fe35806da20 msgr2=0x7fe35815f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 shutdown_connections 2024-01-15T18:05:39.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.105+0000 7fe360800640 1 -- 172.21.15.2:0/611997403 wait complete. 2024-01-15T18:05:39.128 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.348+0000 7f16b4a00640 1 Processor -- start 2024-01-15T18:05:39.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.348+0000 7f16b4a00640 1 -- start start 2024-01-15T18:05:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f16ac15b7d0 0x7f16ac06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c4c0 0x7f16ac06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac15cb00 con 0x7f16ac15c0e0 2024-01-15T18:05:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b4a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac0cccd0 con 0x7f16ac15b7d0 2024-01-15T18:05:39.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0ccec0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f16ac15b7d0 0x7f16ac06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f16ac15b7d0 0x7f16ac06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36380/0 (socket says 172.21.15.2:36380) 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b1600640 1 -- 172.21.15.2:0/2694431555 learned_addr learned my addr 172.21.15.2:0/2694431555 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.350+0000 7f16b2a00640 1 --2- 172.21.15.2:0/2694431555 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c4c0 0x7f16ac06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.351+0000 7f16b1600640 1 -- 172.21.15.2:0/2694431555 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c4c0 msgr2=0x7f16ac06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.351+0000 7f16b1600640 1 --2- 172.21.15.2:0/2694431555 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c4c0 0x7f16ac06c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.351+0000 7f16b1600640 1 -- 172.21.15.2:0/2694431555 >> v1:172.21.15.137:6789/0 conn(0x7f16ac15c0e0 legacy=0x7f16ac0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.351+0000 7f16b1600640 1 -- 172.21.15.2:0/2694431555 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0cd1a0 con 0x7f16ac15b7d0 2024-01-15T18:05:39.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.352+0000 7f16b1600640 1 --2- 172.21.15.2:0/2694431555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15b7d0 0x7f16ac06c470 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1694009870 tx=0x7f169402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8404f3ccb23271a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.352+0000 7f16b0c00640 1 -- 172.21.15.2:0/2694431555 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694031640 con 0x7f16ac15b7d0 2024-01-15T18:05:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b0c00640 1 -- 172.21.15.2:0/2694431555 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16940317a0 con 0x7f16ac15b7d0 2024-01-15T18:05:39.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b0c00640 1 -- 172.21.15.2:0/2694431555 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1694031b30 con 0x7f16ac15b7d0 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 -- 172.21.15.2:0/2694431555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15b7d0 msgr2=0x7f16ac06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2694431555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15b7d0 0x7f16ac06c470 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f1694009870 tx=0x7f169402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 -- 172.21.15.2:0/2694431555 shutdown_connections 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2694431555 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c4c0 0x7f16ac06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2694431555 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15b7d0 0x7f16ac06c470 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 -- 172.21.15.2:0/2694431555 >> 172.21.15.2:0/2694431555 conn(0x7f16ac06da20 msgr2=0x7f16ac06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.353+0000 7f16b4a00640 1 -- 172.21.15.2:0/2694431555 shutdown_connections 2024-01-15T18:05:39.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.354+0000 7f16b4a00640 1 -- 172.21.15.2:0/2694431555 wait complete. 2024-01-15T18:05:39.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.355+0000 7f16b4a00640 1 Processor -- start 2024-01-15T18:05:39.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.355+0000 7f16b4a00640 1 -- start start 2024-01-15T18:05:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.356+0000 7f16b4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16ac15b7d0 0x7f16ac13a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.356+0000 7f16b4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c0e0 0x7f16ac142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.356+0000 7f16b4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 0x7f16ac142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.356+0000 7f16b4a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1437b0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.356+0000 7f16b4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac13dfa0 con 0x7f16ac15b7d0 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.356+0000 7f16b4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac145010 con 0x7f16ac15c0e0 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 0x7f16ac142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 0x7f16ac142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36384/0 (socket says 172.21.15.2:36384) 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 -- 172.21.15.2:0/2536125643 learned_addr learned my addr 172.21.15.2:0/2536125643 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b1600640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c0e0 0x7f16ac142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2000640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16ac15b7d0 0x7f16ac13a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 -- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c0e0 msgr2=0x7f16ac142020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c0e0 0x7f16ac142020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 -- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16ac15b7d0 msgr2=0x7f16ac13a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16ac15b7d0 0x7f16ac13a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.357+0000 7f16b2a00640 1 -- 172.21.15.2:0/2536125643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16940094d0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.358+0000 7f16b2a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 0x7f16ac142560 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f16a4007e60 tx=0x7f16a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.358+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4014420 con 0x7f16ac15c4c0 2024-01-15T18:05:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.358+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a40145a0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.358+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a4014950 con 0x7f16ac15c4c0 2024-01-15T18:05:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.358+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac1452f0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.358+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac1458b0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.359+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac0686f0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.361+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f16a4014ab0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.362+0000 7f16aa000640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f168406bbc0 0x7f168406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.362+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f16a408b3a0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.362+0000 7f16b2000640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f168406bbc0 0x7f168406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.363+0000 7f16b2000640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f168406bbc0 0x7f168406e080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f16ac144a20 tx=0x7f169c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.367+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16a40556c0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.561+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ac06b260 con 0x7f16ac15c4c0 2024-01-15T18:05:39.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.563+0000 7f16aa000640 1 -- 172.21.15.2:0/2536125643 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+12928 (secure 0 0 0) 0x7f16a4058ce0 con 0x7f16ac15c4c0 2024-01-15T18:05:39.566 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:39.566 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":69,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:38.590010+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f168406bbc0 msgr2=0x7f168406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f168406bbc0 0x7f168406e080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f16ac144a20 tx=0x7f169c009040 comp rx=0 tx=0).stop 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 msgr2=0x7f16ac142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 0x7f16ac142560 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f16a4007e60 tx=0x7f16a400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 shutdown_connections 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f168406bbc0 0x7f168406e080 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16ac15c4c0 0x7f16ac142560 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f16ac15c0e0 0x7f16ac142020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 --2- 172.21.15.2:0/2536125643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16ac15b7d0 0x7f16ac13a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.565+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 >> 172.21.15.2:0/2536125643 conn(0x7f16ac06da20 msgr2=0x7f16ac0afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.566+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 shutdown_connections 2024-01-15T18:05:39.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.566+0000 7f16b4a00640 1 -- 172.21.15.2:0/2536125643 wait complete. 2024-01-15T18:05:39.590 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:05:39.590 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:05:39.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.812+0000 7fbd67200640 1 Processor -- start 2024-01-15T18:05:39.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.812+0000 7fbd67200640 1 -- start start 2024-01-15T18:05:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.814+0000 7fbd67200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd6815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.814+0000 7fbd67200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd68071090 con 0x7fbd681447d0 2024-01-15T18:05:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.814+0000 7fbd67200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6806d160 con 0x7fbd680720d0 2024-01-15T18:05:39.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.814+0000 7fbd67200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68071430 con 0x7fbd68143d20 2024-01-15T18:05:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.814+0000 7fbd66800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbd681447d0 0x7fbd680cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57532/0 (socket says 172.21.15.2:57532) 2024-01-15T18:05:39.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.814+0000 7fbd66800640 1 -- 172.21.15.2:0/577484938 learned_addr learned my addr 172.21.15.2:0/577484938 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd65e00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd6815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746150546 0 0) 0x7fbd68071090 con 0x7fbd681447d0 2024-01-15T18:05:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd48003520 con 0x7fbd681447d0 2024-01-15T18:05:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168032974 0 0) 0x7fbd48003520 con 0x7fbd681447d0 2024-01-15T18:05:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 msgr2=0x7fbd6815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd6815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 >> v1:172.21.15.137:6789/0 conn(0x7fbd680720d0 legacy=0x7fbd680cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd68144560 con 0x7fbd681447d0 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fbd600019f0 con 0x7fbd681447d0 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.815+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 >> v1:172.21.15.2:6789/0 conn(0x7fbd681447d0 legacy=0x7fbd680cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd64a00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd480039a0 0x7fbd48003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd64a00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd64a00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd48004d00 0x7fbd48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68071430 con 0x7fbd480039a0 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800ba30 con 0x7fbd48004d00 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4800bbf0 con 0x7fbd68143d20 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65e00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd480039a0 0x7fbd48003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65400640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd66800640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd48004d00 0x7fbd48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65e00640 1 -- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 msgr2=0x7fbd480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65e00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65e00640 1 -- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd48004d00 msgr2=0x7fbd48005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65e00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd48004d00 0x7fbd48005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65e00640 1 -- 172.21.15.2:0/577484938 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd48003520 con 0x7fbd480039a0 2024-01-15T18:05:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd66800640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd48004d00 0x7fbd48005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:39.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.816+0000 7fbd65400640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd68143d20 0x7fbd480047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd65e00640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd480039a0 0x7fbd48003d80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009ac0 tx=0x7fbd480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a192a2d43db84034 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5802c6b0 con 0x7fbd480039a0 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5802c810 con 0x7fbd480039a0 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd64a00640 1 -- 172.21.15.2:0/577484938 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5802cba0 con 0x7fbd480039a0 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd67200640 1 -- 172.21.15.2:0/577484938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd480039a0 msgr2=0x7fbd48003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd67200640 1 --2- 172.21.15.2:0/577484938 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd480039a0 0x7fbd48003d80 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fbd58009ac0 tx=0x7fbd480057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.818+0000 7fbd66800640 1 -- 172.21.15.2:0/577484938 reap_dead start 2024-01-15T18:05:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.819+0000 7fbd67200640 1 -- 172.21.15.2:0/577484938 shutdown_connections 2024-01-15T18:05:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.819+0000 7fbd67200640 1 -- 172.21.15.2:0/577484938 >> 172.21.15.2:0/577484938 conn(0x7fbd680b0840 msgr2=0x7fbd680b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.819+0000 7fbd67200640 1 -- 172.21.15.2:0/577484938 shutdown_connections 2024-01-15T18:05:39.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.819+0000 7fbd67200640 1 -- 172.21.15.2:0/577484938 wait complete. 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.820+0000 7fbd67200640 1 Processor -- start 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.821+0000 7fbd67200640 1 -- start start 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.821+0000 7fbd67200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd480039a0 0x7fbd681f1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.821+0000 7fbd67200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 0x7fbd680ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.821+0000 7fbd67200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd680ab290 0x7fbd680a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.821+0000 7fbd67200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680ace80 con 0x7fbd680aae50 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd67200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680a8460 con 0x7fbd480039a0 2024-01-15T18:05:39.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd67200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd680a8740 con 0x7fbd680ab290 2024-01-15T18:05:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 0x7fbd680ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 0x7fbd680ac2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36404/0 (socket says 172.21.15.2:36404) 2024-01-15T18:05:39.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 -- 172.21.15.2:0/705685137 learned_addr learned my addr 172.21.15.2:0/705685137 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd66800640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd680ab290 0x7fbd680a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65e00640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd480039a0 0x7fbd681f1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 -- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd680ab290 msgr2=0x7fbd680a97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd680ab290 0x7fbd680a97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 -- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd480039a0 msgr2=0x7fbd681f1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd480039a0 0x7fbd681f1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65400640 1 -- 172.21.15.2:0/705685137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd680a8a20 con 0x7fbd680aae50 2024-01-15T18:05:39.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.822+0000 7fbd65e00640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd480039a0 0x7fbd681f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.823+0000 7fbd65400640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 0x7fbd680ac2e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000d7f0 tx=0x7fbd5000dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.823+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50017070 con 0x7fbd680aae50 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.823+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50005280 con 0x7fbd680aae50 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.823+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd680a8d00 con 0x7fbd680aae50 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.823+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000dea0 con 0x7fbd680aae50 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.823+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd680a92c0 con 0x7fbd680aae50 2024-01-15T18:05:39.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.824+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd68071e40 con 0x7fbd680aae50 2024-01-15T18:05:39.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.825+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fbd50007030 con 0x7fbd680aae50 2024-01-15T18:05:39.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.826+0000 7fbd5e000640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd4006bbc0 0x7fbd4006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.826+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fbd50087ed0 con 0x7fbd680aae50 2024-01-15T18:05:39.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.827+0000 7fbd65e00640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd4006bbc0 0x7fbd4006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:39.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.828+0000 7fbd65e00640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd4006bbc0 0x7fbd4006e080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbd58036000 tx=0x7fbd58030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:39.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:39.831+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd50054790 con 0x7fbd680aae50 2024-01-15T18:05:40.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.027+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd68074770 con 0x7fbd680aae50 2024-01-15T18:05:40.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.028+0000 7fbd5e000640 1 -- 172.21.15.2:0/705685137 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd50057db0 con 0x7fbd680aae50 2024-01-15T18:05:40.032 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:05:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd4006bbc0 msgr2=0x7fbd4006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd4006bbc0 0x7fbd4006e080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fbd58036000 tx=0x7fbd58030000 comp rx=0 tx=0).stop 2024-01-15T18:05:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 msgr2=0x7fbd680ac2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 0x7fbd680ac2e0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000d7f0 tx=0x7fbd5000dcb0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 shutdown_connections 2024-01-15T18:05:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbd4006bbc0 0x7fbd4006e080 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbd680ab290 0x7fbd680a97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbd680aae50 0x7fbd680ac2e0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 --2- 172.21.15.2:0/705685137 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbd480039a0 0x7fbd681f1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 >> 172.21.15.2:0/705685137 conn(0x7fbd680b0840 msgr2=0x7fbd6815eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 shutdown_connections 2024-01-15T18:05:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.031+0000 7fbd67200640 1 -- 172.21.15.2:0/705685137 wait complete. 2024-01-15T18:05:40.055 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:05:40.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.264+0000 7f3d39a00640 1 Processor -- start 2024-01-15T18:05:40.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.265+0000 7f3d39a00640 1 -- start start 2024-01-15T18:05:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.266+0000 7f3d39a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d34166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d39a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3415cb00 con 0x7f3d3415c0e0 2024-01-15T18:05:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d39a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d340ccec0 con 0x7f3d3415b7d0 2024-01-15T18:05:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d39a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d340cccd0 con 0x7f3d3406cb80 2024-01-15T18:05:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d32a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3d3415c0e0 0x7f3d340cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57536/0 (socket says 172.21.15.2:57536) 2024-01-15T18:05:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d32a00640 1 -- 172.21.15.2:0/1704337583 learned_addr learned my addr 172.21.15.2:0/1704337583 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:40.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d33400640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d34166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772902652 0 0) 0x7f3d3415cb00 con 0x7f3d3415c0e0 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.267+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d14003520 con 0x7f3d3415c0e0 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 978290654 0 0) 0x7f3d14003520 con 0x7f3d3415c0e0 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 msgr2=0x7f3d34166070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d34166070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 >> v1:172.21.15.137:6789/0 conn(0x7f3d3415b7d0 legacy=0x7f3d3406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d340cd1a0 con 0x7f3d3415c0e0 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3d24001670 con 0x7f3d3415c0e0 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 >> v1:172.21.15.2:6789/0 conn(0x7f3d3415c0e0 legacy=0x7f3d340cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d140039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d14003f20 0x7f3d140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d14004d00 0x7f3d14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1400ba30 con 0x7f3d14004d00 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1400bba0 con 0x7f3d14003f20 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d1400bd60 con 0x7f3d3406cb80 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.268+0000 7f3d33400640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d14004d00 0x7f3d14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d32a00640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d140039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d32000640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d14003f20 0x7f3d140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d33400640 1 -- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 msgr2=0x7f3d140039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d33400640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d140039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d33400640 1 -- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d14003f20 msgr2=0x7f3d140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d33400640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d14003f20 0x7f3d140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d33400640 1 -- 172.21.15.2:0/1704337583 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d14003520 con 0x7f3d14004d00 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d32000640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d14003f20 0x7f3d140047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.269+0000 7f3d32a00640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d3406cb80 0x7f3d140039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:40.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.270+0000 7f3d33400640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d14004d00 0x7f3d14005180 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009fc0 tx=0x7f3d140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b6d5a65433821 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.270+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c02cbf0 con 0x7f3d14004d00 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.270+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2c00f040 con 0x7f3d14004d00 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.270+0000 7f3d31600640 1 -- 172.21.15.2:0/1704337583 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c034440 con 0x7f3d14004d00 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d39a00640 1 -- 172.21.15.2:0/1704337583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d14004d00 msgr2=0x7f3d14005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d39a00640 1 --2- 172.21.15.2:0/1704337583 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d14004d00 0x7f3d14005180 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c009fc0 tx=0x7f3d140057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d33400640 1 -- 172.21.15.2:0/1704337583 reap_dead start 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d39a00640 1 -- 172.21.15.2:0/1704337583 shutdown_connections 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d39a00640 1 -- 172.21.15.2:0/1704337583 >> 172.21.15.2:0/1704337583 conn(0x7f3d3406da20 msgr2=0x7f3d3406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d39a00640 1 -- 172.21.15.2:0/1704337583 shutdown_connections 2024-01-15T18:05:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.271+0000 7f3d39a00640 1 -- 172.21.15.2:0/1704337583 wait complete. 2024-01-15T18:05:40.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.273+0000 7f3d39a00640 1 Processor -- start 2024-01-15T18:05:40.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.273+0000 7f3d39a00640 1 -- start start 2024-01-15T18:05:40.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d39a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d14004d00 0x7f3d34146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d39a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 0x7f3d340b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d39a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d3415a590 0x7f3d340b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d39a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3415b100 con 0x7f3d3415a130 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d39a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3415ab40 con 0x7f3d3415a590 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d39a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3415ae20 con 0x7f3d14004d00 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d32000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 0x7f3d340b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d32000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 0x7f3d340b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36430/0 (socket says 172.21.15.2:36430) 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.274+0000 7f3d32000640 1 -- 172.21.15.2:0/520940728 learned_addr learned my addr 172.21.15.2:0/520940728 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d14004d00 0x7f3d34146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d33400640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d3415a590 0x7f3d340b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32000640 1 -- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d14004d00 msgr2=0x7f3d34146830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32000640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d14004d00 0x7f3d34146830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32000640 1 -- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d3415a590 msgr2=0x7f3d340b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32000640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d3415a590 0x7f3d340b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32000640 1 -- 172.21.15.2:0/520940728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d34070bd0 con 0x7f3d3415a130 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d14004d00 0x7f3d34146830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d33400640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d3415a590 0x7f3d340b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d32000640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 0x7f3d340b33d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00d7a0 tx=0x7f3d1c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c017070 con 0x7f3d3415a130 2024-01-15T18:05:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c005070 con 0x7f3d3415a130 2024-01-15T18:05:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.275+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d34070eb0 con 0x7f3d3415a130 2024-01-15T18:05:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.276+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c005420 con 0x7f3d3415a130 2024-01-15T18:05:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.276+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d341475f0 con 0x7f3d3415a130 2024-01-15T18:05:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.277+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d34070a40 con 0x7f3d3415a130 2024-01-15T18:05:40.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.278+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f3d1c007030 con 0x7f3d3415a130 2024-01-15T18:05:40.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.279+0000 7f3d2aa00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d0c06bbc0 0x7f3d0c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.280+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f3d1c087d50 con 0x7f3d3415a130 2024-01-15T18:05:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.280+0000 7f3d32a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d0c06bbc0 0x7f3d0c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.281+0000 7f3d32a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d0c06bbc0 0x7f3d0c06e080 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3d240017a0 tx=0x7f3d24009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.284+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d1c054670 con 0x7f3d3415a130 2024-01-15T18:05:40.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.482+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3d340686f0 con 0x7f3d3415a130 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.482+0000 7f3d2aa00640 1 -- 172.21.15.2:0/520940728 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v69) v1 ==== 84+0+65 (secure 0 0 0) 0x7f3d1c08a020 con 0x7f3d3415a130 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T18:05:40.486 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:40.487 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d0c06bbc0 msgr2=0x7f3d0c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d0c06bbc0 0x7f3d0c06e080 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f3d240017a0 tx=0x7f3d24009040 comp rx=0 tx=0).stop 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 msgr2=0x7f3d340b33d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 0x7f3d340b33d0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c00d7a0 tx=0x7f3d1c00dc60 comp rx=0 tx=0).stop 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 shutdown_connections 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d0c06bbc0 0x7f3d0c06e080 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d3415a590 0x7f3d340b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d3415a130 0x7f3d340b33d0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 --2- 172.21.15.2:0/520940728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d14004d00 0x7f3d34146830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 >> 172.21.15.2:0/520940728 conn(0x7f3d3406da20 msgr2=0x7f3d340af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 shutdown_connections 2024-01-15T18:05:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.485+0000 7f3d39a00640 1 -- 172.21.15.2:0/520940728 wait complete. 2024-01-15T18:05:40.508 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:05:40.508 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:05:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.738+0000 7f6e7dc00640 1 Processor -- start 2024-01-15T18:05:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.739+0000 7f6e7dc00640 1 -- start start 2024-01-15T18:05:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.740+0000 7f6e7dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e780d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.740+0000 7f6e7dc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780d1250 con 0x7f6e780d1870 2024-01-15T18:05:40.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.740+0000 7f6e7dc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780ce8e0 con 0x7f6e780d1c50 2024-01-15T18:05:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.740+0000 7f6e7dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780ce6d0 con 0x7f6e780b2210 2024-01-15T18:05:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.740+0000 7f6e77e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6e780d1870 0x7f6e7815d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57544/0 (socket says 172.21.15.2:57544) 2024-01-15T18:05:40.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.740+0000 7f6e77e00640 1 -- 172.21.15.2:0/3367777308 learned_addr learned my addr 172.21.15.2:0/3367777308 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e77400640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e780d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 337367215 0 0) 0x7f6e780d1250 con 0x7f6e780d1870 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e58003520 con 0x7f6e780d1870 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067246898 0 0) 0x7f6e58003520 con 0x7f6e780d1870 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 msgr2=0x7f6e780d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e780d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 >> v1:172.21.15.137:6789/0 conn(0x7f6e780d1c50 legacy=0x7f6e780b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780cebc0 con 0x7f6e780d1870 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6e680019f0 con 0x7f6e780d1870 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 >> v1:172.21.15.2:6789/0 conn(0x7f6e780d1870 legacy=0x7f6e7815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e580039a0 0x7f6e58003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780ce6d0 con 0x7f6e580047c0 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5800ba30 con 0x7f6e580039a0 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.741+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5800bbf0 con 0x7f6e780b2210 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77400640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e7d200640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77e00640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e580039a0 0x7f6e58003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77400640 1 -- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 msgr2=0x7f6e58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77400640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e58005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77400640 1 -- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e580039a0 msgr2=0x7f6e58003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77400640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e580039a0 0x7f6e58003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e77400640 1 -- 172.21.15.2:0/3367777308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58003520 con 0x7f6e580047c0 2024-01-15T18:05:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.742+0000 7f6e7d200640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b2210 0x7f6e58005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.743+0000 7f6e77400640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e58004c40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6e780b2650 tx=0x7f6e580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25404ddb2c6e3bd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.743+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6002c630 con 0x7f6e580047c0 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.743+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6002c790 con 0x7f6e580047c0 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.743+0000 7f6e76a00640 1 -- 172.21.15.2:0/3367777308 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6002cb40 con 0x7f6e580047c0 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7dc00640 1 -- 172.21.15.2:0/3367777308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 msgr2=0x7f6e58004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/3367777308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e58004c40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6e780b2650 tx=0x7f6e580057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7d200640 1 -- 172.21.15.2:0/3367777308 reap_dead start 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7dc00640 1 -- 172.21.15.2:0/3367777308 shutdown_connections 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7dc00640 1 -- 172.21.15.2:0/3367777308 >> 172.21.15.2:0/3367777308 conn(0x7f6e780d30b0 msgr2=0x7f6e780d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7dc00640 1 -- 172.21.15.2:0/3367777308 shutdown_connections 2024-01-15T18:05:40.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.744+0000 7f6e7dc00640 1 -- 172.21.15.2:0/3367777308 wait complete. 2024-01-15T18:05:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.746+0000 7f6e7dc00640 1 Processor -- start 2024-01-15T18:05:40.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.746+0000 7f6e7dc00640 1 -- start start 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e7dc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e780b8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e7dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b5c00 0x7f6e78155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e7dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e7809eee0 0x7f6e7809f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e7dc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7809f8a0 con 0x7f6e580047c0 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e7dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780b6f80 con 0x7f6e7809eee0 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e7dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780b6370 con 0x7f6e780b5c00 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e77e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e780b8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e77e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e780b8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36448/0 (socket says 172.21.15.2:36448) 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.747+0000 7f6e77e00640 1 -- 172.21.15.2:0/1247265181 learned_addr learned my addr 172.21.15.2:0/1247265181 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77400640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b5c00 0x7f6e78155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e7d200640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e7809eee0 0x7f6e7809f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77e00640 1 -- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b5c00 msgr2=0x7f6e78155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77e00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b5c00 0x7f6e78155700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77e00640 1 -- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e7809eee0 msgr2=0x7f6e7809f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77e00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e7809eee0 0x7f6e7809f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77e00640 1 -- 172.21.15.2:0/1247265181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780b6650 con 0x7f6e580047c0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e77e00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e780b8920 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6e680021a0 tx=0x7f6e6800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68004d60 con 0x7f6e580047c0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e680070d0 con 0x7f6e580047c0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68007440 con 0x7f6e580047c0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.748+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e780b6930 con 0x7f6e580047c0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.749+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e78155c40 con 0x7f6e580047c0 2024-01-15T18:05:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.750+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e78074770 con 0x7f6e580047c0 2024-01-15T18:05:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.751+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6e680075a0 con 0x7f6e580047c0 2024-01-15T18:05:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.752+0000 7f6e6fe00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e5006bbc0 0x7f6e5006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.752+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f6e68088310 con 0x7f6e580047c0 2024-01-15T18:05:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.752+0000 7f6e77400640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e5006bbc0 0x7f6e5006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.754+0000 7f6e77400640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e5006bbc0 0x7f6e5006e080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6e60036000 tx=0x7f6e60030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:40.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.756+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e68054ce0 con 0x7f6e580047c0 2024-01-15T18:05:40.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:40.952+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6e780d2e90 con 0x7f6e580047c0 2024-01-15T18:05:41.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.604+0000 7f6e6fe00640 1 -- 172.21.15.2:0/1247265181 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v70) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6e68019020 con 0x7f6e580047c0 2024-01-15T18:05:41.608 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T18:05:41.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.608+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e5006bbc0 msgr2=0x7f6e5006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.608+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e5006bbc0 0x7f6e5006e080 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6e60036000 tx=0x7f6e60030000 comp rx=0 tx=0).stop 2024-01-15T18:05:41.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 msgr2=0x7f6e780b8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e780b8920 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6e680021a0 tx=0x7f6e6800b560 comp rx=0 tx=0).stop 2024-01-15T18:05:41.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 shutdown_connections 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6e5006bbc0 0x7f6e5006e080 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6e7809eee0 0x7f6e7809f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6e780b5c00 0x7f6e78155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 --2- 172.21.15.2:0/1247265181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6e580047c0 0x7f6e780b8920 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.609+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 >> 172.21.15.2:0/1247265181 conn(0x7f6e780d30b0 msgr2=0x7f6e780d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.610+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 shutdown_connections 2024-01-15T18:05:41.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.610+0000 7f6e7dc00640 1 -- 172.21.15.2:0/1247265181 wait complete. 2024-01-15T18:05:41.639 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:05:41.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.846+0000 7f61a8a00640 1 Processor -- start 2024-01-15T18:05:41.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.846+0000 7f61a8a00640 1 -- start start 2024-01-15T18:05:41.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.848+0000 7f61a8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a415f260 0x7f61a406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.848+0000 7f61a8a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f61a40720d0 0x7f61a406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.848+0000 7f61a8a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a4071090 con 0x7f61a4143d20 2024-01-15T18:05:41.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.848+0000 7f61a8a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a4071430 con 0x7f61a40720d0 2024-01-15T18:05:41.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.848+0000 7f61a8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4144100 con 0x7f61a415f260 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.848+0000 7f61a3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f61a40720d0 0x7f61a406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f61a40720d0 0x7f61a406c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36452/0 (socket says 172.21.15.2:36452) 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a3200640 1 -- 172.21.15.2:0/3125543431 learned_addr learned my addr 172.21.15.2:0/3125543431 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a1e00640 1 --2- 172.21.15.2:0/3125543431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a415f260 0x7f61a406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a3200640 1 -- 172.21.15.2:0/3125543431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a415f260 msgr2=0x7f61a406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a3200640 1 --2- 172.21.15.2:0/3125543431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a415f260 0x7f61a406c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a3200640 1 -- 172.21.15.2:0/3125543431 >> v1:172.21.15.137:6789/0 conn(0x7f61a4143d20 legacy=0x7f61a415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.849+0000 7f61a3200640 1 -- 172.21.15.2:0/3125543431 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4144560 con 0x7f61a40720d0 2024-01-15T18:05:41.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.850+0000 7f61a3200640 1 --2- 172.21.15.2:0/3125543431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a40720d0 0x7f61a406c690 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c02b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886046fcb02a11eb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.851+0000 7f61a1400640 1 -- 172.21.15.2:0/3125543431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c031740 con 0x7f61a40720d0 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.851+0000 7f61a1400640 1 -- 172.21.15.2:0/3125543431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619c0318a0 con 0x7f61a40720d0 2024-01-15T18:05:41.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.851+0000 7f61a1400640 1 -- 172.21.15.2:0/3125543431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619c031c30 con 0x7f61a40720d0 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.851+0000 7f61a8a00640 1 -- 172.21.15.2:0/3125543431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a40720d0 msgr2=0x7f61a406c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.851+0000 7f61a8a00640 1 --2- 172.21.15.2:0/3125543431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a40720d0 0x7f61a406c690 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f619c009870 tx=0x7f619c02b7e0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.852+0000 7f61a8a00640 1 -- 172.21.15.2:0/3125543431 shutdown_connections 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.852+0000 7f61a8a00640 1 --2- 172.21.15.2:0/3125543431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a40720d0 0x7f61a406c690 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.852+0000 7f61a8a00640 1 --2- 172.21.15.2:0/3125543431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a415f260 0x7f61a406c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.852+0000 7f61a8a00640 1 -- 172.21.15.2:0/3125543431 >> 172.21.15.2:0/3125543431 conn(0x7f61a40b0840 msgr2=0x7f61a40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.852+0000 7f61a8a00640 1 -- 172.21.15.2:0/3125543431 shutdown_connections 2024-01-15T18:05:41.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.852+0000 7f61a8a00640 1 -- 172.21.15.2:0/3125543431 wait complete. 2024-01-15T18:05:41.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.853+0000 7f61a8a00640 1 Processor -- start 2024-01-15T18:05:41.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.854+0000 7f61a8a00640 1 -- start start 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a40720d0 0x7f61a413d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 0x7f61a415bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a8a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a415f260 0x7f61a4142900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a8a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a415d8a0 con 0x7f61a4143d20 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a8a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a415c800 con 0x7f61a415f260 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a415cae0 con 0x7f61a40720d0 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 0x7f61a415bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 0x7f61a415bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36458/0 (socket says 172.21.15.2:36458) 2024-01-15T18:05:41.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a1e00640 1 -- 172.21.15.2:0/1407916952 learned_addr learned my addr 172.21.15.2:0/1407916952 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a3200640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a415f260 0x7f61a4142900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a2800640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a40720d0 0x7f61a413d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a1e00640 1 -- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a40720d0 msgr2=0x7f61a413d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.855+0000 7f61a1e00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a40720d0 0x7f61a413d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a1e00640 1 -- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a415f260 msgr2=0x7f61a4142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a1e00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a415f260 0x7f61a4142900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a1e00640 1 -- 172.21.15.2:0/1407916952 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0094d0 con 0x7f61a4143d20 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a3200640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a415f260 0x7f61a4142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a2800640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a40720d0 0x7f61a413d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:41.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a1e00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 0x7f61a415bfb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f618c007ef0 tx=0x7f618c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c014620 con 0x7f61a4143d20 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c014780 con 0x7f61a4143d20 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c014b30 con 0x7f61a4143d20 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a415cdc0 con 0x7f61a4143d20 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.856+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a415d380 con 0x7f61a4143d20 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.857+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a4074770 con 0x7f61a4143d20 2024-01-15T18:05:41.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.859+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f618c014c90 con 0x7f61a4143d20 2024-01-15T18:05:41.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.859+0000 7f619aa00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617c06bbf0 0x7f617c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:41.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.860+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5311+0+0 (secure 0 0 0) 0x7f618c088f90 con 0x7f61a4143d20 2024-01-15T18:05:41.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.860+0000 7f61a2800640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617c06bbf0 0x7f617c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:41.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.861+0000 7f61a2800640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617c06bbf0 0x7f617c06e0b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6194003eb0 tx=0x7f6194009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:41.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:41.864+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f618c0557c0 con 0x7f61a4143d20 2024-01-15T18:05:42.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.064+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f61a40cf6c0 con 0x7f61a4143d20 2024-01-15T18:05:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.610+0000 7f619aa00640 1 -- 172.21.15.2:0/1407916952 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v71) v1 ==== 170+0+0 (secure 0 0 0) 0x7f618c058de0 con 0x7f61a4143d20 2024-01-15T18:05:42.614 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T18:05:42.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617c06bbf0 msgr2=0x7f617c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617c06bbf0 0x7f617c06e0b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6194003eb0 tx=0x7f6194009040 comp rx=0 tx=0).stop 2024-01-15T18:05:42.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 msgr2=0x7f61a415bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 0x7f61a415bfb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f618c007ef0 tx=0x7f618c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 shutdown_connections 2024-01-15T18:05:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617c06bbf0 0x7f617c06e0b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a415f260 0x7f61a4142900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a4143d20 0x7f61a415bfb0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 --2- 172.21.15.2:0/1407916952 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a40720d0 0x7f61a413d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.614+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 >> 172.21.15.2:0/1407916952 conn(0x7f61a40b0840 msgr2=0x7f61a40aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.615+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 shutdown_connections 2024-01-15T18:05:42.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.615+0000 7f61a8a00640 1 -- 172.21.15.2:0/1407916952 wait complete. 2024-01-15T18:05:42.652 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.836+0000 7f02cd000640 1 Processor -- start 2024-01-15T18:05:42.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.837+0000 7f02cd000640 1 -- start start 2024-01-15T18:05:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.839+0000 7f02cd000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02c815ae80 0x7f02c80acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.839+0000 7f02cd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c806c8f0 0x7f02c806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.839+0000 7f02cd000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02c806d290 con 0x7f02c80b2430 2024-01-15T18:05:42.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.839+0000 7f02cd000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02c815e840 con 0x7f02c815ae80 2024-01-15T18:05:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.839+0000 7f02cd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c815dbc0 con 0x7f02c806c8f0 2024-01-15T18:05:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c806c8f0 0x7f02c806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02c815ae80 0x7f02c80acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c6800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f02c80b2430 0x7f02c80b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:59214/0 (socket says 172.21.15.2:59214) 2024-01-15T18:05:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c806c8f0 0x7f02c806cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:42562/0 (socket says 172.21.15.2:42562) 2024-01-15T18:05:42.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c6800640 1 -- 172.21.15.2:0/3782423092 learned_addr learned my addr 172.21.15.2:0/3782423092 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:42.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5e00640 1 -- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c806c8f0 msgr2=0x7f02c806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:42.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5e00640 1 --2- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c806c8f0 0x7f02c806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5e00640 1 -- 172.21.15.2:0/3782423092 >> v1:172.21.15.137:6789/0 conn(0x7f02c80b2430 legacy=0x7f02c80b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.840+0000 7f02c5e00640 1 -- 172.21.15.2:0/3782423092 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c815dea0 con 0x7f02c815ae80 2024-01-15T18:05:42.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.842+0000 7f02c5e00640 1 --2- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c815ae80 0x7f02c80acd00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f02b80097a0 tx=0x7f02b802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18883cecce9086a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02c4a00640 1 -- 172.21.15.2:0/3782423092 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b802d040 con 0x7f02c815ae80 2024-01-15T18:05:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02c4a00640 1 -- 172.21.15.2:0/3782423092 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b8031810 con 0x7f02c815ae80 2024-01-15T18:05:42.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02c4a00640 1 -- 172.21.15.2:0/3782423092 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8031ba0 con 0x7f02c815ae80 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 -- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c815ae80 msgr2=0x7f02c80acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 --2- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c815ae80 0x7f02c80acd00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f02b80097a0 tx=0x7f02b802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 -- 172.21.15.2:0/3782423092 shutdown_connections 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 --2- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c806c8f0 0x7f02c806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 --2- 172.21.15.2:0/3782423092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c815ae80 0x7f02c80acd00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 -- 172.21.15.2:0/3782423092 >> 172.21.15.2:0/3782423092 conn(0x7f02c80b45f0 msgr2=0x7f02c80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 -- 172.21.15.2:0/3782423092 shutdown_connections 2024-01-15T18:05:42.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.843+0000 7f02cd000640 1 -- 172.21.15.2:0/3782423092 wait complete. 2024-01-15T18:05:42.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.845+0000 7f02cd000640 1 Processor -- start 2024-01-15T18:05:42.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.845+0000 7f02cd000640 1 -- start start 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.846+0000 7f02cd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 0x7f02c81f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.846+0000 7f02cd000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c80b2430 0x7f02c80af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.846+0000 7f02cd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c815ae80 0x7f02c815c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.846+0000 7f02cd000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c814e110 con 0x7f02c806c8f0 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.846+0000 7f02cd000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02c80b0080 con 0x7f02c80b2430 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.846+0000 7f02cd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c814d770 con 0x7f02c815ae80 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 0x7f02c81f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 0x7f02c81f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36468/0 (socket says 172.21.15.2:36468) 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 -- 172.21.15.2:0/1826506728 learned_addr learned my addr 172.21.15.2:0/1826506728 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c6800640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c815ae80 0x7f02c815c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5400640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c80b2430 0x7f02c80af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 -- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c815ae80 msgr2=0x7f02c815c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c815ae80 0x7f02c815c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 -- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c80b2430 msgr2=0x7f02c80af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:42.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c80b2430 0x7f02c80af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:42.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.847+0000 7f02c5e00640 1 -- 172.21.15.2:0/1826506728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b8009400 con 0x7f02c806c8f0 2024-01-15T18:05:42.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.848+0000 7f02c5e00640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 0x7f02c81f5f70 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000c00 tx=0x7f02b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.848+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8006e10 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.848+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b803ad60 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.848+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b8044630 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.848+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02c814da50 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.848+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02c80afbe0 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.849+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02c814e250 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.851+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f02b804d450 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.852+0000 7f02be000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02a006bbc0 0x7f02a006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.853+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f02b80aa8c0 con 0x7f02c806c8f0 2024-01-15T18:05:42.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.853+0000 7f02c5400640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02a006bbc0 0x7f02a006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:42.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.854+0000 7f02c5400640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02a006bbc0 0x7f02a006e080 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f02c815d8b0 tx=0x7f02b000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:42.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:42.858+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02b80772e0 con 0x7f02c806c8f0 2024-01-15T18:05:43.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.091+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f02c814e250 con 0x7f02c806c8f0 2024-01-15T18:05:43.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.650+0000 7f02be000640 1 -- 172.21.15.2:0/1826506728 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v42) v1 ==== 174+0+0 (secure 0 0 0) 0x7f02c814e250 con 0x7f02c806c8f0 2024-01-15T18:05:43.657 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 10 and data pool 11 2024-01-15T18:05:43.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.657+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02a006bbc0 msgr2=0x7f02a006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.657+0000 7f02cd000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02a006bbc0 0x7f02a006e080 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f02c815d8b0 tx=0x7f02b000b040 comp rx=0 tx=0).stop 2024-01-15T18:05:43.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.657+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 msgr2=0x7f02c81f5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.657+0000 7f02cd000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 0x7f02c81f5f70 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f02b8000c00 tx=0x7f02b80046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.659+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 shutdown_connections 2024-01-15T18:05:43.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.659+0000 7f02cd000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02a006bbc0 0x7f02a006e080 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.659+0000 7f02cd000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02c815ae80 0x7f02c815c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.659+0000 7f02cd000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02c80b2430 0x7f02c80af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.659+0000 7f02cd000640 1 --2- 172.21.15.2:0/1826506728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02c806c8f0 0x7f02c81f5f70 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.659+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 >> 172.21.15.2:0/1826506728 conn(0x7f02c80b45f0 msgr2=0x7f02c80ad900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.660+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 shutdown_connections 2024-01-15T18:05:43.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.660+0000 7f02cd000640 1 -- 172.21.15.2:0/1826506728 wait complete. 2024-01-15T18:05:43.697 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.874+0000 7fdd57800640 1 Processor -- start 2024-01-15T18:05:43.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.874+0000 7fdd57800640 1 -- start start 2024-01-15T18:05:43.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.875+0000 7fdd57800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdd50146dd0 0x7fdd5006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.875+0000 7fdd57800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500ac870 con 0x7fdd501469f0 2024-01-15T18:05:43.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd57800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5015d970 con 0x7fdd5006c9b0 2024-01-15T18:05:43.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd57800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5006ce20 con 0x7fdd50146dd0 2024-01-15T18:05:43.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd54e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdd501469f0 0x7fdd50071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38972/0 (socket says 172.21.15.2:38972) 2024-01-15T18:05:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd54e00640 1 -- 172.21.15.2:0/3671538184 learned_addr learned my addr 172.21.15.2:0/3671538184 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd4fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdd50146dd0 0x7fdd5006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd4fe00640 1 -- 172.21.15.2:0/3671538184 >> v1:172.21.15.137:6789/0 conn(0x7fdd5006c9b0 legacy=0x7fdd500b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd4fe00640 1 -- 172.21.15.2:0/3671538184 >> v1:172.21.15.2:6789/0 conn(0x7fdd501469f0 legacy=0x7fdd50071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.876+0000 7fdd4fe00640 1 -- 172.21.15.2:0/3671538184 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5015dc30 con 0x7fdd50146dd0 2024-01-15T18:05:43.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.878+0000 7fdd4fe00640 1 --2- 172.21.15.2:0/3671538184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50146dd0 0x7fdd5006c470 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fdd380097a0 tx=0x7fdd3802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c97b6e7c6a93c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.878+0000 7fdd4f400640 1 -- 172.21.15.2:0/3671538184 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38031830 con 0x7fdd50146dd0 2024-01-15T18:05:43.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.878+0000 7fdd4f400640 1 -- 172.21.15.2:0/3671538184 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd38031990 con 0x7fdd50146dd0 2024-01-15T18:05:43.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.878+0000 7fdd4f400640 1 -- 172.21.15.2:0/3671538184 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd38031d20 con 0x7fdd50146dd0 2024-01-15T18:05:43.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.878+0000 7fdd57800640 1 -- 172.21.15.2:0/3671538184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50146dd0 msgr2=0x7fdd5006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.878+0000 7fdd57800640 1 --2- 172.21.15.2:0/3671538184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50146dd0 0x7fdd5006c470 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fdd380097a0 tx=0x7fdd3802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.879+0000 7fdd57800640 1 -- 172.21.15.2:0/3671538184 shutdown_connections 2024-01-15T18:05:43.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.879+0000 7fdd57800640 1 --2- 172.21.15.2:0/3671538184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50146dd0 0x7fdd5006c470 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.879+0000 7fdd57800640 1 -- 172.21.15.2:0/3671538184 >> 172.21.15.2:0/3671538184 conn(0x7fdd5006da20 msgr2=0x7fdd5006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:43.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.879+0000 7fdd57800640 1 -- 172.21.15.2:0/3671538184 shutdown_connections 2024-01-15T18:05:43.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.879+0000 7fdd57800640 1 -- 172.21.15.2:0/3671538184 wait complete. 2024-01-15T18:05:43.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.880+0000 7fdd57800640 1 Processor -- start 2024-01-15T18:05:43.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 -- start start 2024-01-15T18:05:43.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd5006c9b0 0x7fdd5013b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd501469f0 0x7fdd5015c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 0x7fdd5014f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5013ae70 con 0x7fdd50139d50 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5015ce70 con 0x7fdd501469f0 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.881+0000 7fdd57800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5013a1f0 con 0x7fdd5006c9b0 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 0x7fdd5014f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 0x7fdd5014f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53148/0 (socket says 172.21.15.2:53148) 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 -- 172.21.15.2:0/918789311 learned_addr learned my addr 172.21.15.2:0/918789311 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd54e00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd5006c9b0 0x7fdd5013b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd4fe00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd501469f0 0x7fdd5015c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 -- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd5006c9b0 msgr2=0x7fdd5013b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd5006c9b0 0x7fdd5013b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 -- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd501469f0 msgr2=0x7fdd5015c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd501469f0 0x7fdd5015c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd55800640 1 -- 172.21.15.2:0/918789311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd38009400 con 0x7fdd50139d50 2024-01-15T18:05:43.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd4fe00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd501469f0 0x7fdd5015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.882+0000 7fdd54e00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd5006c9b0 0x7fdd5013b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.883+0000 7fdd55800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 0x7fdd5014f460 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b970 tx=0x7fdd4800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.883+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4800cb40 con 0x7fdd50139d50 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.883+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4800cca0 con 0x7fdd50139d50 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.883+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd48014680 con 0x7fdd50139d50 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.883+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5013a4d0 con 0x7fdd50139d50 2024-01-15T18:05:43.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.883+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5013aa90 con 0x7fdd50139d50 2024-01-15T18:05:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.884+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5013afb0 con 0x7fdd50139d50 2024-01-15T18:05:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.885+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fdd4801d3b0 con 0x7fdd50139d50 2024-01-15T18:05:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.886+0000 7fdd4cc00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd2806bc10 0x7fdd2806e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:43.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.886+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fdd48088530 con 0x7fdd50139d50 2024-01-15T18:05:43.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.887+0000 7fdd54e00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd2806bc10 0x7fdd2806e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:43.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.888+0000 7fdd54e00640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd2806bc10 0x7fdd2806e0d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdd40007fd0 tx=0x7fdd40009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:43.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:43.890+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd48054c00 con 0x7fdd50139d50 2024-01-15T18:05:44.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.084+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd500686f0 con 0x7fdd50139d50 2024-01-15T18:05:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.087+0000 7fdd4cc00640 1 -- 172.21.15.2:0/918789311 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16652 (secure 0 0 0) 0x7fdd48019090 con 0x7fdd50139d50 2024-01-15T18:05:44.091 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:44.091 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":72,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:43.093143+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:05:41.005847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-15T18:05:42.117627+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd2806bc10 msgr2=0x7fdd2806e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd2806bc10 0x7fdd2806e0d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdd40007fd0 tx=0x7fdd40009040 comp rx=0 tx=0).stop 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 msgr2=0x7fdd5014f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 0x7fdd5014f460 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fdd4800b970 tx=0x7fdd4800be30 comp rx=0 tx=0).stop 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 shutdown_connections 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd2806bc10 0x7fdd2806e0d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd50139d50 0x7fdd5014f460 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd501469f0 0x7fdd5015c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 --2- 172.21.15.2:0/918789311 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd5006c9b0 0x7fdd5013b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 >> 172.21.15.2:0/918789311 conn(0x7fdd5006da20 msgr2=0x7fdd50071c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 shutdown_connections 2024-01-15T18:05:44.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.089+0000 7fdd57800640 1 -- 172.21.15.2:0/918789311 wait complete. 2024-01-15T18:05:44.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:44.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.329+0000 7f6be5800640 1 Processor -- start 2024-01-15T18:05:44.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.329+0000 7f6be5800640 1 -- start start 2024-01-15T18:05:44.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6be5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be01578d0 0x7f6be015d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6be5800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6be00b31c0 0x7f6be006da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6be5800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be00b3640 con 0x7f6be006dfd0 2024-01-15T18:05:44.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6be5800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00ae180 con 0x7f6be00b31c0 2024-01-15T18:05:44.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6be5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be015de90 con 0x7f6be01578d0 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6bd7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6be00b31c0 0x7f6be006da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6bd7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6be00b31c0 0x7f6be006da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53156/0 (socket says 172.21.15.2:53156) 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6bd7e00640 1 -- 172.21.15.2:0/3819141728 learned_addr learned my addr 172.21.15.2:0/3819141728 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.331+0000 7f6bdea00640 1 --2- 172.21.15.2:0/3819141728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be01578d0 0x7f6be015d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.332+0000 7f6bd7e00640 1 -- 172.21.15.2:0/3819141728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be01578d0 msgr2=0x7f6be015d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.332+0000 7f6bd7e00640 1 --2- 172.21.15.2:0/3819141728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be01578d0 0x7f6be015d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.332+0000 7f6bd7e00640 1 -- 172.21.15.2:0/3819141728 >> v1:172.21.15.137:6789/0 conn(0x7f6be006dfd0 legacy=0x7f6be006b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.332+0000 7f6bd7e00640 1 -- 172.21.15.2:0/3819141728 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be015e170 con 0x7f6be00b31c0 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.333+0000 7f6bd7e00640 1 --2- 172.21.15.2:0/3819141728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be00b31c0 0x7f6be006da90 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009870 tx=0x7f6bc802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aeb46a8cd2c5c1f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.333+0000 7f6bde000640 1 -- 172.21.15.2:0/3819141728 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc80317c0 con 0x7f6be00b31c0 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.333+0000 7f6bde000640 1 -- 172.21.15.2:0/3819141728 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc8031920 con 0x7f6be00b31c0 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.333+0000 7f6bde000640 1 -- 172.21.15.2:0/3819141728 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc8031cb0 con 0x7f6be00b31c0 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 -- 172.21.15.2:0/3819141728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be00b31c0 msgr2=0x7f6be006da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 --2- 172.21.15.2:0/3819141728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be00b31c0 0x7f6be006da90 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8009870 tx=0x7f6bc802b860 comp rx=0 tx=0).stop 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 -- 172.21.15.2:0/3819141728 shutdown_connections 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 --2- 172.21.15.2:0/3819141728 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be00b31c0 0x7f6be006da90 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 --2- 172.21.15.2:0/3819141728 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be01578d0 0x7f6be015d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 -- 172.21.15.2:0/3819141728 >> 172.21.15.2:0/3819141728 conn(0x7f6be0070fc0 msgr2=0x7f6be00713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 -- 172.21.15.2:0/3819141728 shutdown_connections 2024-01-15T18:05:44.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.334+0000 7f6be5800640 1 -- 172.21.15.2:0/3819141728 wait complete. 2024-01-15T18:05:44.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.335+0000 7f6be5800640 1 Processor -- start 2024-01-15T18:05:44.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.336+0000 7f6be5800640 1 -- start start 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.336+0000 7f6be5800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6be006dfd0 0x7f6be0149180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.336+0000 7f6be5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be00b31c0 0x7f6be015bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6be5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 0x7f6be015c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6be5800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be01541b0 con 0x7f6be01578d0 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6be5800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be01525d0 con 0x7f6be006dfd0 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6be5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0153530 con 0x7f6be00b31c0 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 0x7f6be015c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 0x7f6be015c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53168/0 (socket says 172.21.15.2:53168) 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 -- 172.21.15.2:0/3976406561 learned_addr learned my addr 172.21.15.2:0/3976406561 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bd7e00640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be00b31c0 0x7f6be015bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdea00640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6be006dfd0 0x7f6be0149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 -- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be00b31c0 msgr2=0x7f6be015bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be00b31c0 0x7f6be015bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 -- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6be006dfd0 msgr2=0x7f6be0149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6be006dfd0 0x7f6be0149180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bdf400640 1 -- 172.21.15.2:0/3976406561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc80094d0 con 0x7f6be01578d0 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.337+0000 7f6bd7e00640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be00b31c0 0x7f6be015bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.338+0000 7f6bdf400640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 0x7f6be015c4b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c730 tx=0x7f6bd800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.338+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8003b70 con 0x7f6be01578d0 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.338+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd8003cd0 con 0x7f6be01578d0 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.338+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd8005310 con 0x7f6be01578d0 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.338+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be0153810 con 0x7f6be01578d0 2024-01-15T18:05:44.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.338+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be0153dd0 con 0x7f6be01578d0 2024-01-15T18:05:44.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.339+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be01542f0 con 0x7f6be01578d0 2024-01-15T18:05:44.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.340+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6bd8005470 con 0x7f6be01578d0 2024-01-15T18:05:44.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.341+0000 7f6bd6a00640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bb806bc10 0x7f6bb806e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.341+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f6bd8088210 con 0x7f6be01578d0 2024-01-15T18:05:44.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.342+0000 7f6bdea00640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bb806bc10 0x7f6bb806e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.343+0000 7f6bdea00640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bb806bc10 0x7f6bb806e0d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006fd0 tx=0x7f6bd0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.346+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6be01542f0 con 0x7f6be01578d0 2024-01-15T18:05:44.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.543+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6be01542f0 con 0x7f6be01578d0 2024-01-15T18:05:44.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.545+0000 7f6bd6a00640 1 -- 172.21.15.2:0/3976406561 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f6be01542f0 con 0x7f6be01578d0 2024-01-15T18:05:44.549 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:44.549 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":72,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:43.093143+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:05:41.005847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-15T18:05:42.117627+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:44.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bb806bc10 msgr2=0x7f6bb806e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bb806bc10 0x7f6bb806e0d0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6bd0006fd0 tx=0x7f6bd0008040 comp rx=0 tx=0).stop 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 msgr2=0x7f6be015c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 0x7f6be015c4b0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6bd800c730 tx=0x7f6bd800cb30 comp rx=0 tx=0).stop 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 shutdown_connections 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bb806bc10 0x7f6bb806e0d0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6be01578d0 0x7f6be015c4b0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6be00b31c0 0x7f6be015bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 --2- 172.21.15.2:0/3976406561 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6be006dfd0 0x7f6be0149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.548+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 >> 172.21.15.2:0/3976406561 conn(0x7f6be0070fc0 msgr2=0x7f6be006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.549+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 shutdown_connections 2024-01-15T18:05:44.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.549+0000 7f6be5800640 1 -- 172.21.15.2:0/3976406561 wait complete. 2024-01-15T18:05:44.570 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:05:44.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.784+0000 7f435f200640 1 Processor -- start 2024-01-15T18:05:44.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.784+0000 7f435f200640 1 -- start start 2024-01-15T18:05:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4360143d20 0x7f436015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 0x7f436015f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435f200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4360071090 con 0x7f43600720d0 2024-01-15T18:05:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4360071430 con 0x7f4360143d20 2024-01-15T18:05:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4360144100 con 0x7f43601447d0 2024-01-15T18:05:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435d400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f43600720d0 0x7f43600cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38974/0 (socket says 172.21.15.2:38974) 2024-01-15T18:05:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.786+0000 7f435d400640 1 -- 172.21.15.2:0/4284232433 learned_addr learned my addr 172.21.15.2:0/4284232433 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435e800640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 0x7f436015f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435de00640 1 --2- 172.21.15.2:0/4284232433 >> v2:172.21.15.137:3300/0 conn(0x7f4360143d20 0x7f436015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279455477 0 0) 0x7f4360071090 con 0x7f43600720d0 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4340003520 con 0x7f43600720d0 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3264492698 0 0) 0x7f4340003520 con 0x7f43600720d0 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 msgr2=0x7f436015f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 0x7f436015f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 >> v2:172.21.15.137:3300/0 conn(0x7f4360143d20 msgr2=0x7f436015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 --2- 172.21.15.2:0/4284232433 >> v2:172.21.15.137:3300/0 conn(0x7f4360143d20 0x7f436015eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436006d160 con 0x7f43600720d0 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4348001670 con 0x7f43600720d0 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 >> v1:172.21.15.2:6789/0 conn(0x7f43600720d0 legacy=0x7f43600cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4340003970 0x7f4340003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f4340004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 0x7f4340005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4360071430 con 0x7f43400047c0 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4360144100 con 0x7f4340003970 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.787+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434000ba00 con 0x7f43601447d0 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435d400640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f4340004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435e800640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 0x7f4340005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435de00640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4340003970 0x7f4340003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435d400640 1 -- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 msgr2=0x7f4340005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435d400640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43601447d0 0x7f4340005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435d400640 1 -- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4340003970 msgr2=0x7f4340003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435d400640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4340003970 0x7f4340003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.788+0000 7f435d400640 1 -- 172.21.15.2:0/4284232433 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4340003520 con 0x7f43400047c0 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.789+0000 7f435d400640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f4340004c40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f434800c110 tx=0x7f43400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af7088bb4c98060 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.789+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348005ad0 con 0x7f43400047c0 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.789+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4348005c50 con 0x7f43400047c0 2024-01-15T18:05:44.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.789+0000 7f435ca00640 1 -- 172.21.15.2:0/4284232433 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f434802c830 con 0x7f43400047c0 2024-01-15T18:05:44.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435f200640 1 -- 172.21.15.2:0/4284232433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 msgr2=0x7f4340004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435f200640 1 --2- 172.21.15.2:0/4284232433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f4340004c40 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f434800c110 tx=0x7f43400057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435e800640 1 -- 172.21.15.2:0/4284232433 reap_dead start 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435f200640 1 -- 172.21.15.2:0/4284232433 shutdown_connections 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435f200640 1 -- 172.21.15.2:0/4284232433 >> 172.21.15.2:0/4284232433 conn(0x7f43600b0840 msgr2=0x7f43600b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435f200640 1 -- 172.21.15.2:0/4284232433 shutdown_connections 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.790+0000 7f435f200640 1 -- 172.21.15.2:0/4284232433 wait complete. 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.792+0000 7f435f200640 1 Processor -- start 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.792+0000 7f435f200640 1 -- start start 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f43601f4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43600c52b0 0x7f436015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43600c5730 0x7f436015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43601f4c90 con 0x7f43400047c0 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4360137d50 con 0x7f43600c5730 2024-01-15T18:05:44.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43600c3f20 con 0x7f43600c52b0 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f43601f4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f43601f4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53178/0 (socket says 172.21.15.2:53178) 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.793+0000 7f435de00640 1 -- 172.21.15.2:0/1205374564 learned_addr learned my addr 172.21.15.2:0/1205374564 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435de00640 1 -- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43600c52b0 msgr2=0x7f436015a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435de00640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43600c52b0 0x7f436015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435de00640 1 -- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43600c5730 msgr2=0x7f436015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435de00640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43600c5730 0x7f436015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435de00640 1 -- 172.21.15.2:0/1205374564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43600c4200 con 0x7f43400047c0 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435de00640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f43601f4750 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4350010890 tx=0x7f4350010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435000eb60 con 0x7f43400047c0 2024-01-15T18:05:44.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f435000ecc0 con 0x7f43400047c0 2024-01-15T18:05:44.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435001c400 con 0x7f43400047c0 2024-01-15T18:05:44.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43600c44e0 con 0x7f43400047c0 2024-01-15T18:05:44.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.794+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43600c4aa0 con 0x7f43400047c0 2024-01-15T18:05:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.795+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4360074770 con 0x7f43400047c0 2024-01-15T18:05:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.796+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f435001c560 con 0x7f43400047c0 2024-01-15T18:05:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.797+0000 7f4356000640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f433806bbc0 0x7f433806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.797+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f43500889e0 con 0x7f43400047c0 2024-01-15T18:05:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.798+0000 7f435d400640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f433806bbc0 0x7f433806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:44.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.799+0000 7f435d400640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f433806bbc0 0x7f433806e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f434800c110 tx=0x7f434802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:44.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:44.802+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4350055350 con 0x7f43400047c0 2024-01-15T18:05:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.041+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f43600afca0 con 0x7f43400047c0 2024-01-15T18:05:45.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.665+0000 7f4356000640 1 -- 172.21.15.2:0/1205374564 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4350058970 con 0x7f43400047c0 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f433806bbc0 msgr2=0x7f433806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f433806bbc0 0x7f433806e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f434800c110 tx=0x7f434802e000 comp rx=0 tx=0).stop 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 msgr2=0x7f43601f4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f43601f4750 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f4350010890 tx=0x7f4350010d50 comp rx=0 tx=0).stop 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 shutdown_connections 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f433806bbc0 0x7f433806e080 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.669+0000 7f435f200640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43600c5730 0x7f436015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.670+0000 7f435f200640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43600c52b0 0x7f436015a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.670+0000 7f435f200640 1 --2- 172.21.15.2:0/1205374564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43400047c0 0x7f43601f4750 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.670+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 >> 172.21.15.2:0/1205374564 conn(0x7f43600b0840 msgr2=0x7f436015b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.670+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 shutdown_connections 2024-01-15T18:05:45.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.670+0000 7f435f200640 1 -- 172.21.15.2:0/1205374564 wait complete. 2024-01-15T18:05:45.695 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:05:45.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:05:45.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.907+0000 7f5885800640 1 Processor -- start 2024-01-15T18:05:45.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.907+0000 7f5885800640 1 -- start start 2024-01-15T18:05:45.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f5885800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f588006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f5885800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f588006ce30 0x7f5880070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f5885800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800ce230 con 0x7f5880144c00 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f5885800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f588006d2c0 con 0x7f588006ce30 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f5885800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58800ce510 con 0x7f588015c450 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f587ea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5880144c00 0x7f58800cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38980/0 (socket says 172.21.15.2:38980) 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f587ea00640 1 -- 172.21.15.2:0/1801165369 learned_addr learned my addr 172.21.15.2:0/1801165369 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f587f400640 1 --2- 172.21.15.2:0/1801165369 >> v2:172.21.15.2:3300/0 conn(0x7f588006ce30 0x7f5880070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.909+0000 7f587e000640 1 --2- 172.21.15.2:0/1801165369 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f588006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.910+0000 7f587d600640 1 -- 172.21.15.2:0/1801165369 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902608417 0 0) 0x7f58800ce230 con 0x7f5880144c00 2024-01-15T18:05:45.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.910+0000 7f587d600640 1 -- 172.21.15.2:0/1801165369 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5860003520 con 0x7f5880144c00 2024-01-15T18:05:45.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.910+0000 7f587f400640 1 -- 172.21.15.2:0/1801165369 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 msgr2=0x7f588006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:45.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.910+0000 7f587f400640 1 --2- 172.21.15.2:0/1801165369 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f588006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.910+0000 7f587f400640 1 -- 172.21.15.2:0/1801165369 >> v1:172.21.15.2:6789/0 conn(0x7f5880144c00 legacy=0x7f58800cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.910+0000 7f587f400640 1 -- 172.21.15.2:0/1801165369 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58800ce7f0 con 0x7f588006ce30 2024-01-15T18:05:45.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.911+0000 7f587f400640 1 --2- 172.21.15.2:0/1801165369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588006ce30 0x7f5880070b10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f5878009d90 tx=0x7f587802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b9753d829e2c84 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f587d600640 1 -- 172.21.15.2:0/1801165369 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587802cc50 con 0x7f588006ce30 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f587d600640 1 -- 172.21.15.2:0/1801165369 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587802cdb0 con 0x7f588006ce30 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f587d600640 1 -- 172.21.15.2:0/1801165369 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5878034640 con 0x7f588006ce30 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 -- 172.21.15.2:0/1801165369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588006ce30 msgr2=0x7f5880070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 --2- 172.21.15.2:0/1801165369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588006ce30 0x7f5880070b10 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f5878009d90 tx=0x7f587802bea0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 -- 172.21.15.2:0/1801165369 shutdown_connections 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 --2- 172.21.15.2:0/1801165369 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f588006ce30 0x7f5880070b10 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 --2- 172.21.15.2:0/1801165369 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f588006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 -- 172.21.15.2:0/1801165369 >> 172.21.15.2:0/1801165369 conn(0x7f58800b45f0 msgr2=0x7f58800b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:45.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 -- 172.21.15.2:0/1801165369 shutdown_connections 2024-01-15T18:05:45.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.912+0000 7f5885800640 1 -- 172.21.15.2:0/1801165369 wait complete. 2024-01-15T18:05:45.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.914+0000 7f5885800640 1 Processor -- start 2024-01-15T18:05:45.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.914+0000 7f5885800640 1 -- start start 2024-01-15T18:05:45.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.915+0000 7f5885800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f588006ce30 0x7f5880137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.915+0000 7f5885800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 0x7f58800af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.915+0000 7f5885800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f58801f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.915+0000 7f5885800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58801429b0 con 0x7f5880144c00 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.915+0000 7f5885800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58800b1580 con 0x7f588006ce30 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.915+0000 7f5885800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5880141e30 con 0x7f588015c450 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 0x7f58800af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 0x7f58800af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53192/0 (socket says 172.21.15.2:53192) 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 -- 172.21.15.2:0/3035957108 learned_addr learned my addr 172.21.15.2:0/3035957108 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587ea00640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f588006ce30 0x7f5880137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587f400640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f58801f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 -- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 msgr2=0x7f58801f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f58801f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 -- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f588006ce30 msgr2=0x7f5880137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f588006ce30 0x7f5880137b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.916+0000 7f587e000640 1 -- 172.21.15.2:0/3035957108 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878009a70 con 0x7f5880144c00 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.917+0000 7f587e000640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 0x7f58800af410 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f586800d5f0 tx=0x7f586800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.917+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58680125f0 con 0x7f5880144c00 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.917+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5868012750 con 0x7f5880144c00 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.917+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5868012b00 con 0x7f5880144c00 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.917+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5880142110 con 0x7f5880144c00 2024-01-15T18:05:45.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.917+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58801426d0 con 0x7f5880144c00 2024-01-15T18:05:45.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.918+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5880067660 con 0x7f5880144c00 2024-01-15T18:05:45.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.919+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5868012c60 con 0x7f5880144c00 2024-01-15T18:05:45.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.920+0000 7f5876a00640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f585806bbf0 0x7f585806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:45.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.920+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f5868088b30 con 0x7f5880144c00 2024-01-15T18:05:45.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.921+0000 7f587ea00640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f585806bbf0 0x7f585806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:45.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.922+0000 7f587ea00640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f585806bbf0 0x7f585806e0b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5870005880 tx=0x7f5870009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:45.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:45.925+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5868055200 con 0x7f5880144c00 2024-01-15T18:05:46.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.160+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f58800686f0 con 0x7f5880144c00 2024-01-15T18:05:46.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.676+0000 7f5876a00640 1 -- 172.21.15.2:0/3035957108 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5868058820 con 0x7f5880144c00 2024-01-15T18:05:46.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.679+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f585806bbf0 msgr2=0x7f585806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.679+0000 7f5885800640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f585806bbf0 0x7f585806e0b0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f5870005880 tx=0x7f5870009040 comp rx=0 tx=0).stop 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.679+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 msgr2=0x7f58800af410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.679+0000 7f5885800640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 0x7f58800af410 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f586800d5f0 tx=0x7f586800dab0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 shutdown_connections 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f585806bbf0 0x7f585806e0b0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f588015c450 0x7f58801f65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5880144c00 0x7f58800af410 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 --2- 172.21.15.2:0/3035957108 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f588006ce30 0x7f5880137b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 >> 172.21.15.2:0/3035957108 conn(0x7f58800b45f0 msgr2=0x7f58800b34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 shutdown_connections 2024-01-15T18:05:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.680+0000 7f5885800640 1 -- 172.21.15.2:0/3035957108 wait complete. 2024-01-15T18:05:46.706 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:05:46.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:46.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.926+0000 7f31d0e00640 1 Processor -- start 2024-01-15T18:05:46.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.927+0000 7f31d0e00640 1 -- start start 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.928+0000 7f31d0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 0x7f31cc0c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.928+0000 7f31d0e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc0c6e40 con 0x7f31cc0d3c70 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.928+0000 7f31d0e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31cc0d2090 con 0x7f31cc0cfad0 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.928+0000 7f31d0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0d17f0 con 0x7f31cc0d37f0 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31cbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f31cc0d3c70 0x7f31cc0c7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38992/0 (socket says 172.21.15.2:38992) 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31cbe00640 1 -- 172.21.15.2:0/1395749463 learned_addr learned my addr 172.21.15.2:0/1395749463 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:46.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31c3e00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 0x7f31cc0c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172938305 0 0) 0x7f31cc0c6e40 con 0x7f31cc0d3c70 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31ac003520 con 0x7f31cc0d3c70 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 443930099 0 0) 0x7f31ac003520 con 0x7f31cc0d3c70 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 msgr2=0x7f31cc0c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31caa00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 0x7f31cc0c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.929+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 >> v1:172.21.15.137:6789/0 conn(0x7f31cc0cfad0 legacy=0x7f31cc0cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0d1ad0 con 0x7f31cc0d3c70 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f31c40019f0 con 0x7f31cc0d3c70 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 >> v1:172.21.15.2:6789/0 conn(0x7f31cc0d3c70 legacy=0x7f31cc0c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 0x7f31ac0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac003f20 0x7f31ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31ac004d00 0x7f31ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0d17f0 con 0x7f31ac004d00 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac00bb10 con 0x7f31ac003f20 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31ac00bcd0 con 0x7f31cc0d37f0 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31cbe00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31ac004d00 0x7f31ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31cb400640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 0x7f31ac0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.930+0000 7f31c3e00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac003f20 0x7f31ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.931+0000 7f31cbe00640 1 -- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 msgr2=0x7f31ac0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.931+0000 7f31cbe00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0d37f0 0x7f31ac0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.931+0000 7f31cbe00640 1 -- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac003f20 msgr2=0x7f31ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.931+0000 7f31cbe00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac003f20 0x7f31ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.931+0000 7f31cbe00640 1 -- 172.21.15.2:0/1395749463 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac003520 con 0x7f31ac004d00 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.932+0000 7f31cbe00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31ac004d00 0x7f31ac005180 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f31c400a830 tx=0x7f31ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b1c05772d06eb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.932+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40072d0 con 0x7f31ac004d00 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.932+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c402f070 con 0x7f31ac004d00 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.932+0000 7f31caa00640 1 -- 172.21.15.2:0/1395749463 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c40076b0 con 0x7f31ac004d00 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31d0e00640 1 -- 172.21.15.2:0/1395749463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31ac004d00 msgr2=0x7f31ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31d0e00640 1 --2- 172.21.15.2:0/1395749463 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31ac004d00 0x7f31ac005180 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f31c400a830 tx=0x7f31ac0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31cbe00640 1 -- 172.21.15.2:0/1395749463 reap_dead start 2024-01-15T18:05:46.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31d0e00640 1 -- 172.21.15.2:0/1395749463 shutdown_connections 2024-01-15T18:05:46.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31d0e00640 1 -- 172.21.15.2:0/1395749463 >> 172.21.15.2:0/1395749463 conn(0x7f31cc09c3b0 msgr2=0x7f31cc0d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:46.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31d0e00640 1 -- 172.21.15.2:0/1395749463 shutdown_connections 2024-01-15T18:05:46.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.933+0000 7f31d0e00640 1 -- 172.21.15.2:0/1395749463 wait complete. 2024-01-15T18:05:46.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.935+0000 7f31d0e00640 1 Processor -- start 2024-01-15T18:05:46.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.935+0000 7f31d0e00640 1 -- start start 2024-01-15T18:05:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31d0e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac004d00 0x7f31cc15d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31d0e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 0x7f31cc159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31d0e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0b6e50 0x7f31cc0b7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31d0e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0b7840 con 0x7f31cc15e290 2024-01-15T18:05:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31d0e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc1f29e0 con 0x7f31ac004d00 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31d0e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc0cb260 con 0x7f31cc0b6e50 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31c3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 0x7f31cc159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31c3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 0x7f31cc159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53210/0 (socket says 172.21.15.2:53210) 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31c3e00640 1 -- 172.21.15.2:0/820341051 learned_addr learned my addr 172.21.15.2:0/820341051 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31cbe00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0b6e50 0x7f31cc0b7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.936+0000 7f31cb400640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac004d00 0x7f31cc15d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3e00640 1 -- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0b6e50 msgr2=0x7f31cc0b7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0b6e50 0x7f31cc0b7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3e00640 1 -- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac004d00 msgr2=0x7f31cc15d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac004d00 0x7f31cc15d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3e00640 1 -- 172.21.15.2:0/820341051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc0cb540 con 0x7f31cc15e290 2024-01-15T18:05:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 0x7f31cc159770 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f31b4011600 tx=0x7f31b4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b400f920 con 0x7f31cc15e290 2024-01-15T18:05:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b400fa80 con 0x7f31cc15e290 2024-01-15T18:05:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc0cb820 con 0x7f31cc15e290 2024-01-15T18:05:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b400fdf0 con 0x7f31cc15e290 2024-01-15T18:05:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.937+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc0a0bc0 con 0x7f31cc15e290 2024-01-15T18:05:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.939+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31cc074770 con 0x7f31cc15e290 2024-01-15T18:05:46.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.940+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f31b4015460 con 0x7f31cc15e290 2024-01-15T18:05:46.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.941+0000 7f31c3400640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f31a406bc40 0x7f31a406e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:46.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.941+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f31b4088cf0 con 0x7f31cc15e290 2024-01-15T18:05:46.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.942+0000 7f31cb400640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f31a406bc40 0x7f31a406e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:46.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.943+0000 7f31cb400640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f31a406bc40 0x7f31a406e100 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f31bc005490 tx=0x7f31bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:46.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:46.945+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31b40555b0 con 0x7f31cc15e290 2024-01-15T18:05:47.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.175+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31cc0b8c20 con 0x7f31cc15e290 2024-01-15T18:05:47.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.176+0000 7f31c3400640 1 -- 172.21.15.2:0/820341051 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f31b4058bd0 con 0x7f31cc15e290 2024-01-15T18:05:47.181 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:47.181 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5573,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2894571444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2894571444},{"type":"v1","addr":"172.21.15.137:6836","nonce":2894571444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5980,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/2585344206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2585344206},{"type":"v1","addr":"172.21.15.137:6837","nonce":2585344206}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2342931858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2342931858},{"type":"v1","addr":"172.21.15.2:6833","nonce":2342931858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:05:43.093928+0000","modified":"2024-01-15T18:05:46.161925+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6993":{"gid":6993,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6835/4035045426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4035045426},{"type":"v1","addr":"172.21.15.2:6835","nonce":4035045426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-15T18:05:47.181 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 46 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f31a406bc40 msgr2=0x7f31a406e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f31a406bc40 0x7f31a406e100 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f31bc005490 tx=0x7f31bc009040 comp rx=0 tx=0).stop 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 msgr2=0x7f31cc159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 0x7f31cc159770 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f31b4011600 tx=0x7f31b4011ac0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 shutdown_connections 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f31a406bc40 0x7f31a406e100 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f31cc0b6e50 0x7f31cc0b7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f31cc15e290 0x7f31cc159770 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.179+0000 7f31d0e00640 1 --2- 172.21.15.2:0/820341051 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f31ac004d00 0x7f31cc15d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.180+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 >> 172.21.15.2:0/820341051 conn(0x7f31cc09c3b0 msgr2=0x7f31cc15c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.180+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 shutdown_connections 2024-01-15T18:05:47.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.180+0000 7f31d0e00640 1 -- 172.21.15.2:0/820341051 wait complete. 2024-01-15T18:05:47.204 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:05:47.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.422+0000 7f0d96e00640 1 Processor -- start 2024-01-15T18:05:47.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.422+0000 7f0d96e00640 1 -- start start 2024-01-15T18:05:47.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.424+0000 7f0d96e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0d90071ae0 0x7f0d90164fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.424+0000 7f0d96e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d900aeab0 con 0x7f0d9013ce30 2024-01-15T18:05:47.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.424+0000 7f0d96e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d90070380 con 0x7f0d9015e340 2024-01-15T18:05:47.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.424+0000 7f0d96e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d900ce6d0 con 0x7f0d90071ae0 2024-01-15T18:05:47.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.424+0000 7f0d8fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0d9013ce30 0x7f0d9015dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38996/0 (socket says 172.21.15.2:38996) 2024-01-15T18:05:47.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.424+0000 7f0d8fe00640 1 -- 172.21.15.2:0/4085535521 learned_addr learned my addr 172.21.15.2:0/4085535521 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:47.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d94e00640 1 --2- 172.21.15.2:0/4085535521 >> v2:172.21.15.137:3300/0 conn(0x7f0d90071ae0 0x7f0d90164fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321908195 0 0) 0x7f0d900aeab0 con 0x7f0d9013ce30 2024-01-15T18:05:47.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d70003520 con 0x7f0d9013ce30 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125015989 0 0) 0x7f0d70003520 con 0x7f0d9013ce30 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 >> v2:172.21.15.137:3300/0 conn(0x7f0d90071ae0 msgr2=0x7f0d90164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 --2- 172.21.15.2:0/4085535521 >> v2:172.21.15.137:3300/0 conn(0x7f0d90071ae0 0x7f0d90164fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 >> v1:172.21.15.137:6789/0 conn(0x7f0d9015e340 legacy=0x7f0d900cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900ce9b0 con 0x7f0d9013ce30 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f0d80001670 con 0x7f0d9013ce30 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 >> v1:172.21.15.2:6789/0 conn(0x7f0d9013ce30 legacy=0x7f0d9015dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d70003970 0x7f0d70003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d700047c0 0x7f0d70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d70005180 0x7f0d7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900ce6d0 con 0x7f0d70003970 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7000c020 con 0x7f0d700047c0 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.425+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7000c1e0 con 0x7f0d70005180 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d70003970 0x7f0d70003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8f400640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d700047c0 0x7f0d70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8fe00640 1 -- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d70005180 msgr2=0x7f0d7000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d70005180 0x7f0d7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8fe00640 1 -- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d700047c0 msgr2=0x7f0d70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d700047c0 0x7f0d70004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8fe00640 1 -- 172.21.15.2:0/4085535521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d70003520 con 0x7f0d70003970 2024-01-15T18:05:47.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d94e00640 1 -- 172.21.15.2:0/4085535521 reap_dead start 2024-01-15T18:05:47.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.426+0000 7f0d8f400640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d700047c0 0x7f0d70004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.427+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d70003970 0x7f0d70003d90 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000c240 tx=0x7f0d700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c875c979b828c7bb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80005ad0 con 0x7f0d70003970 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d80005c50 con 0x7f0d70003970 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d8ea00640 1 -- 172.21.15.2:0/4085535521 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8002c870 con 0x7f0d70003970 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 -- 172.21.15.2:0/4085535521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d70003970 msgr2=0x7f0d70003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d70003970 0x7f0d70003d90 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f0d8000c240 tx=0x7f0d700057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 -- 172.21.15.2:0/4085535521 shutdown_connections 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 --2- 172.21.15.2:0/4085535521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d70003970 0x7f0d70003d90 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 -- 172.21.15.2:0/4085535521 >> 172.21.15.2:0/4085535521 conn(0x7f0d9006da20 msgr2=0x7f0d9006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 -- 172.21.15.2:0/4085535521 shutdown_connections 2024-01-15T18:05:47.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.428+0000 7f0d96e00640 1 -- 172.21.15.2:0/4085535521 wait complete. 2024-01-15T18:05:47.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.430+0000 7f0d96e00640 1 Processor -- start 2024-01-15T18:05:47.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.430+0000 7f0d96e00640 1 -- start start 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d96e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70003970 0x7f0d901590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d96e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 0x7f0d901f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d96e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d901f2860 0x7f0d90137dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d96e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d90137670 con 0x7f0d901511e0 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d96e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900aed40 con 0x7f0d70003970 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d96e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900af550 con 0x7f0d901f2860 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 0x7f0d901f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 0x7f0d901f39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53240/0 (socket says 172.21.15.2:53240) 2024-01-15T18:05:47.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.431+0000 7f0d8f400640 1 -- 172.21.15.2:0/3569196536 learned_addr learned my addr 172.21.15.2:0/3569196536 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:47.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70003970 0x7f0d901590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8f400640 1 -- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d901f2860 msgr2=0x7f0d90137dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8f400640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d901f2860 0x7f0d90137dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8f400640 1 -- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70003970 msgr2=0x7f0d901590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8f400640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70003970 0x7f0d901590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8f400640 1 -- 172.21.15.2:0/3569196536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900aef90 con 0x7f0d901511e0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.432+0000 7f0d8f400640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 0x7f0d901f39c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800e4b0 tx=0x7f0d7800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.433+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78007e90 con 0x7f0d901511e0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.433+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d780052e0 con 0x7f0d901511e0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.433+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78003830 con 0x7f0d901511e0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.433+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d900af270 con 0x7f0d901511e0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.433+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d901f34a0 con 0x7f0d901511e0 2024-01-15T18:05:47.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.434+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d901f3660 con 0x7f0d901511e0 2024-01-15T18:05:47.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.435+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f0d78003990 con 0x7f0d901511e0 2024-01-15T18:05:47.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.436+0000 7f0d87e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d6806bbc0 0x7f0d6806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.436+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f0d78087b90 con 0x7f0d901511e0 2024-01-15T18:05:47.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.436+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d6806bbc0 0x7f0d6806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.438+0000 7f0d8fe00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d6806bbc0 0x7f0d6806e080 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0d80034000 tx=0x7f0d80009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.440+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d78054260 con 0x7f0d901511e0 2024-01-15T18:05:47.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.636+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d900686f0 con 0x7f0d901511e0 2024-01-15T18:05:47.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.638+0000 7f0d87e00640 1 -- 172.21.15.2:0/3569196536 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f0d78057880 con 0x7f0d901511e0 2024-01-15T18:05:47.643 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:47.643 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":73,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:05:44.650653+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:05:41.005847+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-15T18:05:42.117627+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:05:47.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d6806bbc0 msgr2=0x7f0d6806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d6806bbc0 0x7f0d6806e080 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0d80034000 tx=0x7f0d80009030 comp rx=0 tx=0).stop 2024-01-15T18:05:47.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 msgr2=0x7f0d901f39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 0x7f0d901f39c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800e4b0 tx=0x7f0d7800e970 comp rx=0 tx=0).stop 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 shutdown_connections 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0d6806bbc0 0x7f0d6806e080 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d901f2860 0x7f0d90137dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d901511e0 0x7f0d901f39c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 --2- 172.21.15.2:0/3569196536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d70003970 0x7f0d901590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 >> 172.21.15.2:0/3569196536 conn(0x7f0d9006da20 msgr2=0x7f0d9015a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 shutdown_connections 2024-01-15T18:05:47.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.641+0000 7f0d96e00640 1 -- 172.21.15.2:0/3569196536 wait complete. 2024-01-15T18:05:47.666 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:05:47.666 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:05:47.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.876+0000 7f8122a00640 1 Processor -- start 2024-01-15T18:05:47.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.876+0000 7f8122a00640 1 -- start start 2024-01-15T18:05:47.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.878+0000 7f8122a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f811c0ad960 0x7f811c15db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.878+0000 7f8122a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c0b2e70 con 0x7f811c15e070 2024-01-15T18:05:47.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.878+0000 7f8122a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c06c430 con 0x7f811c15d750 2024-01-15T18:05:47.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.878+0000 7f8122a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811c0b30c0 con 0x7f811c0ad960 2024-01-15T18:05:47.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.879+0000 7f8122000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f811c15e070 0x7f811c166d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39006/0 (socket says 172.21.15.2:39006) 2024-01-15T18:05:47.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.879+0000 7f8122000640 1 -- 172.21.15.2:0/4229089590 learned_addr learned my addr 172.21.15.2:0/4229089590 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:47.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.879+0000 7f8120c00640 1 --2- 172.21.15.2:0/4229089590 >> v2:172.21.15.137:3300/0 conn(0x7f811c0ad960 0x7f811c15db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.879+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164406201 0 0) 0x7f811c0b2e70 con 0x7f811c15e070 2024-01-15T18:05:47.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.879+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc003520 con 0x7f811c15e070 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1781047464 0 0) 0x7f80fc003520 con 0x7f811c15e070 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 >> v2:172.21.15.137:3300/0 conn(0x7f811c0ad960 msgr2=0x7f811c15db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 --2- 172.21.15.2:0/4229089590 >> v2:172.21.15.137:3300/0 conn(0x7f811c0ad960 0x7f811c15db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 >> v1:172.21.15.137:6789/0 conn(0x7f811c15d750 legacy=0x7f811c06cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c0b33a0 con 0x7f811c15e070 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f81140019f0 con 0x7f811c15e070 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 >> v1:172.21.15.2:6789/0 conn(0x7f811c15e070 legacy=0x7f811c166d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f80fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80fc0047c0 0x7f80fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80fc0051d0 0x7f80fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c0b30c0 con 0x7f80fc0039c0 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00c120 con 0x7f80fc0047c0 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00c2e0 con 0x7f80fc0051d0 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8121600640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f80fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8120c00640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80fc0047c0 0x7f80fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8121600640 1 -- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80fc0051d0 msgr2=0x7f80fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8121600640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f80fc0051d0 0x7f80fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8121600640 1 -- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80fc0047c0 msgr2=0x7f80fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8121600640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f80fc0047c0 0x7f80fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.880+0000 7f8121600640 1 -- 172.21.15.2:0/4229089590 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc003520 con 0x7f80fc0039c0 2024-01-15T18:05:47.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.881+0000 7f8122000640 1 -- 172.21.15.2:0/4229089590 reap_dead start 2024-01-15T18:05:47.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.882+0000 7f8121600640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f80fc003de0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f810c001720 tx=0x7f80fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd95605edd94529 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.882+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c0052d0 con 0x7f80fc0039c0 2024-01-15T18:05:47.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.882+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c02e050 con 0x7f80fc0039c0 2024-01-15T18:05:47.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.882+0000 7f811be00640 1 -- 172.21.15.2:0/4229089590 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c003830 con 0x7f80fc0039c0 2024-01-15T18:05:47.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.882+0000 7f8122a00640 1 -- 172.21.15.2:0/4229089590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 msgr2=0x7f80fc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.882+0000 7f8122a00640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f80fc003de0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f810c001720 tx=0x7f80fc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.883+0000 7f8122a00640 1 -- 172.21.15.2:0/4229089590 shutdown_connections 2024-01-15T18:05:47.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.883+0000 7f8122a00640 1 --2- 172.21.15.2:0/4229089590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f80fc003de0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.883+0000 7f8122a00640 1 -- 172.21.15.2:0/4229089590 >> 172.21.15.2:0/4229089590 conn(0x7f811c0b0840 msgr2=0x7f811c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:47.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.883+0000 7f8122a00640 1 -- 172.21.15.2:0/4229089590 shutdown_connections 2024-01-15T18:05:47.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.883+0000 7f8122a00640 1 -- 172.21.15.2:0/4229089590 wait complete. 2024-01-15T18:05:47.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.885+0000 7f8122a00640 1 Processor -- start 2024-01-15T18:05:47.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.885+0000 7f8122a00640 1 -- start start 2024-01-15T18:05:47.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f811c147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f811c06c5c0 0x7f811c06ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f811c153e90 0x7f811c154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c06cf80 con 0x7f80fc0039c0 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c155850 con 0x7f811c06c5c0 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811c147c70 con 0x7f811c153e90 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8121600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f811c147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8121600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f811c147730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53264/0 (socket says 172.21.15.2:53264) 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8121600640 1 -- 172.21.15.2:0/330553177 learned_addr learned my addr 172.21.15.2:0/330553177 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8120c00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f811c06c5c0 0x7f811c06ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.886+0000 7f8122000640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f811c153e90 0x7f811c154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8121600640 1 -- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f811c153e90 msgr2=0x7f811c154310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8121600640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f811c153e90 0x7f811c154310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8121600640 1 -- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f811c06c5c0 msgr2=0x7f811c06ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8121600640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f811c06c5c0 0x7f811c06ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8121600640 1 -- 172.21.15.2:0/330553177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c154c40 con 0x7f80fc0039c0 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8121600640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f811c147730 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f810c005040 tx=0x7f810c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c0076c0 con 0x7f80fc0039c0 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c0069e0 con 0x7f80fc0039c0 2024-01-15T18:05:47.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811c154f20 con 0x7f80fc0039c0 2024-01-15T18:05:47.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c006d90 con 0x7f80fc0039c0 2024-01-15T18:05:47.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.887+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811c1554e0 con 0x7f80fc0039c0 2024-01-15T18:05:47.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.888+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811c074770 con 0x7f80fc0039c0 2024-01-15T18:05:47.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.890+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f810c0035f0 con 0x7f80fc0039c0 2024-01-15T18:05:47.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.891+0000 7f8119600640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80f406bbf0 0x7f80f406e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:47.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.891+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f810c0aa2a0 con 0x7f80fc0039c0 2024-01-15T18:05:47.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.891+0000 7f8120c00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80f406bbf0 0x7f80f406e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:47.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.893+0000 7f8120c00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80f406bbf0 0x7f80f406e0b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f811c06d180 tx=0x7f8104008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:47.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:47.896+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f810c076c80 con 0x7f80fc0039c0 2024-01-15T18:05:48.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.085+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f811c06cf80 con 0x7f80f406bbf0 2024-01-15T18:05:48.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.112+0000 7f8119600640 1 -- 172.21.15.2:0/330553177 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287001 (secure 0 0 0) 0x7f811c06cf80 con 0x7f80f406bbf0 2024-01-15T18:05:48.121 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:48.122 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80f406bbf0 msgr2=0x7f80f406e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80f406bbf0 0x7f80f406e0b0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f811c06d180 tx=0x7f8104008040 comp rx=0 tx=0).stop 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 msgr2=0x7f811c147730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f811c147730 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f810c005040 tx=0x7f810c001120 comp rx=0 tx=0).stop 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 shutdown_connections 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80f406bbf0 0x7f80f406e0b0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f811c153e90 0x7f811c154310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f811c06c5c0 0x7f811c06ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 --2- 172.21.15.2:0/330553177 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f80fc0039c0 0x7f811c147730 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 >> 172.21.15.2:0/330553177 conn(0x7f811c0b0840 msgr2=0x7f811c0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 shutdown_connections 2024-01-15T18:05:48.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.118+0000 7f8122a00640 1 -- 172.21.15.2:0/330553177 wait complete. 2024-01-15T18:05:48.144 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-01-15T18:05:47.598048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431000,"kb_used_data":8040,"kb_used_omap":12,"kb_used_meta":422835,"kb_avail":754543720,"statfs":{"total":773094113280,"available":772652769280,"internally_reserved":0,"allocated":8232960,"data_stored":4153876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":432983601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.990460"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.678714+0000","last_change":"2024-01-15T18:05:43.675850+0000","last_active":"2024-01-15T18:05:43.678714+0000","last_peered":"2024-01-15T18:05:43.678714+0000","last_clean":"2024-01-15T18:05:43.678714+0000","last_became_active":"2024-01-15T18:05:43.675545+0000","last_became_peered":"2024-01-15T18:05:43.675545+0000","last_unstale":"2024-01-15T18:05:43.678714+0000","last_undegraded":"2024-01-15T18:05:43.678714+0000","last_fullsized":"2024-01-15T18:05:43.678714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:17.390435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674671+0000","last_change":"2024-01-15T18:05:43.671148+0000","last_active":"2024-01-15T18:05:43.674671+0000","last_peered":"2024-01-15T18:05:43.674671+0000","last_clean":"2024-01-15T18:05:43.674671+0000","last_became_active":"2024-01-15T18:05:43.669822+0000","last_became_peered":"2024-01-15T18:05:43.669822+0000","last_unstale":"2024-01-15T18:05:43.674671+0000","last_undegraded":"2024-01-15T18:05:43.674671+0000","last_fullsized":"2024-01-15T18:05:43.674671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:28:37.233984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668840+0000","last_change":"2024-01-15T18:05:43.663373+0000","last_active":"2024-01-15T18:05:43.668840+0000","last_peered":"2024-01-15T18:05:43.668840+0000","last_clean":"2024-01-15T18:05:43.668840+0000","last_became_active":"2024-01-15T18:05:43.662253+0000","last_became_peered":"2024-01-15T18:05:43.662253+0000","last_unstale":"2024-01-15T18:05:43.668840+0000","last_undegraded":"2024-01-15T18:05:43.668840+0000","last_fullsized":"2024-01-15T18:05:43.668840+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:57:40.803431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679100+0000","last_change":"2024-01-15T18:05:43.676852+0000","last_active":"2024-01-15T18:05:43.679100+0000","last_peered":"2024-01-15T18:05:43.679100+0000","last_clean":"2024-01-15T18:05:43.679100+0000","last_became_active":"2024-01-15T18:05:43.676556+0000","last_became_peered":"2024-01-15T18:05:43.676556+0000","last_unstale":"2024-01-15T18:05:43.679100+0000","last_undegraded":"2024-01-15T18:05:43.679100+0000","last_fullsized":"2024-01-15T18:05:43.679100+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.998414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677284+0000","last_change":"2024-01-15T18:05:43.670491+0000","last_active":"2024-01-15T18:05:43.677284+0000","last_peered":"2024-01-15T18:05:43.677284+0000","last_clean":"2024-01-15T18:05:43.677284+0000","last_became_active":"2024-01-15T18:05:43.670177+0000","last_became_peered":"2024-01-15T18:05:43.670177+0000","last_unstale":"2024-01-15T18:05:43.677284+0000","last_undegraded":"2024-01-15T18:05:43.677284+0000","last_fullsized":"2024-01-15T18:05:43.677284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:13:03.781810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677535+0000","last_change":"2024-01-15T18:05:43.672718+0000","last_active":"2024-01-15T18:05:43.677535+0000","last_peered":"2024-01-15T18:05:43.677535+0000","last_clean":"2024-01-15T18:05:43.677535+0000","last_became_active":"2024-01-15T18:05:43.671441+0000","last_became_peered":"2024-01-15T18:05:43.671441+0000","last_unstale":"2024-01-15T18:05:43.677535+0000","last_undegraded":"2024-01-15T18:05:43.677535+0000","last_fullsized":"2024-01-15T18:05:43.677535+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:49.267617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.663544+0000","last_change":"2024-01-15T18:05:43.661992+0000","last_active":"2024-01-15T18:05:43.663544+0000","last_peered":"2024-01-15T18:05:43.663544+0000","last_clean":"2024-01-15T18:05:43.663544+0000","last_became_active":"2024-01-15T18:05:43.657977+0000","last_became_peered":"2024-01-15T18:05:43.657977+0000","last_unstale":"2024-01-15T18:05:43.663544+0000","last_undegraded":"2024-01-15T18:05:43.663544+0000","last_fullsized":"2024-01-15T18:05:43.663544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:42.285026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680972+0000","last_change":"2024-01-15T18:05:43.677758+0000","last_active":"2024-01-15T18:05:43.680972+0000","last_peered":"2024-01-15T18:05:43.680972+0000","last_clean":"2024-01-15T18:05:43.680972+0000","last_became_active":"2024-01-15T18:05:43.677034+0000","last_became_peered":"2024-01-15T18:05:43.677034+0000","last_unstale":"2024-01-15T18:05:43.680972+0000","last_undegraded":"2024-01-15T18:05:43.680972+0000","last_fullsized":"2024-01-15T18:05:43.680972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:11.538253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680450+0000","last_change":"2024-01-15T18:05:43.674797+0000","last_active":"2024-01-15T18:05:43.680450+0000","last_peered":"2024-01-15T18:05:43.680450+0000","last_clean":"2024-01-15T18:05:43.680450+0000","last_became_active":"2024-01-15T18:05:43.673775+0000","last_became_peered":"2024-01-15T18:05:43.673775+0000","last_unstale":"2024-01-15T18:05:43.680450+0000","last_undegraded":"2024-01-15T18:05:43.680450+0000","last_fullsized":"2024-01-15T18:05:43.680450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:26.111871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.669040+0000","last_change":"2024-01-15T18:05:43.666742+0000","last_active":"2024-01-15T18:05:43.669040+0000","last_peered":"2024-01-15T18:05:43.669040+0000","last_clean":"2024-01-15T18:05:43.669040+0000","last_became_active":"2024-01-15T18:05:43.662605+0000","last_became_peered":"2024-01-15T18:05:43.662605+0000","last_unstale":"2024-01-15T18:05:43.669040+0000","last_undegraded":"2024-01-15T18:05:43.669040+0000","last_fullsized":"2024-01-15T18:05:43.669040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:20:24.155396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675215+0000","last_change":"2024-01-15T18:05:43.672860+0000","last_active":"2024-01-15T18:05:43.675215+0000","last_peered":"2024-01-15T18:05:43.675215+0000","last_clean":"2024-01-15T18:05:43.675215+0000","last_became_active":"2024-01-15T18:05:43.672561+0000","last_became_peered":"2024-01-15T18:05:43.672561+0000","last_unstale":"2024-01-15T18:05:43.675215+0000","last_undegraded":"2024-01-15T18:05:43.675215+0000","last_fullsized":"2024-01-15T18:05:43.675215+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:37.136328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674984+0000","last_change":"2024-01-15T18:05:43.665889+0000","last_active":"2024-01-15T18:05:43.674984+0000","last_peered":"2024-01-15T18:05:43.674984+0000","last_clean":"2024-01-15T18:05:43.674984+0000","last_became_active":"2024-01-15T18:05:43.665618+0000","last_became_peered":"2024-01-15T18:05:43.665618+0000","last_unstale":"2024-01-15T18:05:43.674984+0000","last_undegraded":"2024-01-15T18:05:43.674984+0000","last_fullsized":"2024-01-15T18:05:43.674984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:29.813139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.672912+0000","last_change":"2024-01-15T18:05:43.671154+0000","last_active":"2024-01-15T18:05:43.672912+0000","last_peered":"2024-01-15T18:05:43.672912+0000","last_clean":"2024-01-15T18:05:43.672912+0000","last_became_active":"2024-01-15T18:05:43.670186+0000","last_became_peered":"2024-01-15T18:05:43.670186+0000","last_unstale":"2024-01-15T18:05:43.672912+0000","last_undegraded":"2024-01-15T18:05:43.672912+0000","last_fullsized":"2024-01-15T18:05:43.672912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:54.618059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680628+0000","last_change":"2024-01-15T18:05:43.675004+0000","last_active":"2024-01-15T18:05:43.680628+0000","last_peered":"2024-01-15T18:05:43.680628+0000","last_clean":"2024-01-15T18:05:43.680628+0000","last_became_active":"2024-01-15T18:05:43.674586+0000","last_became_peered":"2024-01-15T18:05:43.674586+0000","last_unstale":"2024-01-15T18:05:43.680628+0000","last_undegraded":"2024-01-15T18:05:43.680628+0000","last_fullsized":"2024-01-15T18:05:43.680628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:23.989128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668906+0000","last_change":"2024-01-15T18:05:43.667864+0000","last_active":"2024-01-15T18:05:43.668906+0000","last_peered":"2024-01-15T18:05:43.668906+0000","last_clean":"2024-01-15T18:05:43.668906+0000","last_became_active":"2024-01-15T18:05:43.667561+0000","last_became_peered":"2024-01-15T18:05:43.667561+0000","last_unstale":"2024-01-15T18:05:43.668906+0000","last_undegraded":"2024-01-15T18:05:43.668906+0000","last_fullsized":"2024-01-15T18:05:43.668906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:32.434900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.683217+0000","last_change":"2024-01-15T18:05:43.677601+0000","last_active":"2024-01-15T18:05:43.683217+0000","last_peered":"2024-01-15T18:05:43.683217+0000","last_clean":"2024-01-15T18:05:43.683217+0000","last_became_active":"2024-01-15T18:05:43.677285+0000","last_became_peered":"2024-01-15T18:05:43.677285+0000","last_unstale":"2024-01-15T18:05:43.683217+0000","last_undegraded":"2024-01-15T18:05:43.683217+0000","last_fullsized":"2024-01-15T18:05:43.683217+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:57.612507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.681241+0000","last_change":"2024-01-15T18:05:43.675229+0000","last_active":"2024-01-15T18:05:43.681241+0000","last_peered":"2024-01-15T18:05:43.681241+0000","last_clean":"2024-01-15T18:05:43.681241+0000","last_became_active":"2024-01-15T18:05:43.670691+0000","last_became_peered":"2024-01-15T18:05:43.670691+0000","last_unstale":"2024-01-15T18:05:43.681241+0000","last_undegraded":"2024-01-15T18:05:43.681241+0000","last_fullsized":"2024-01-15T18:05:43.681241+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:16:26.052298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679024+0000","last_change":"2024-01-15T18:05:43.676489+0000","last_active":"2024-01-15T18:05:43.679024+0000","last_peered":"2024-01-15T18:05:43.679024+0000","last_clean":"2024-01-15T18:05:43.679024+0000","last_became_active":"2024-01-15T18:05:43.676199+0000","last_became_peered":"2024-01-15T18:05:43.676199+0000","last_unstale":"2024-01-15T18:05:43.679024+0000","last_undegraded":"2024-01-15T18:05:43.679024+0000","last_fullsized":"2024-01-15T18:05:43.679024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:52.224215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680943+0000","last_change":"2024-01-15T18:05:43.678865+0000","last_active":"2024-01-15T18:05:43.680943+0000","last_peered":"2024-01-15T18:05:43.680943+0000","last_clean":"2024-01-15T18:05:43.680943+0000","last_became_active":"2024-01-15T18:05:43.678328+0000","last_became_peered":"2024-01-15T18:05:43.678328+0000","last_unstale":"2024-01-15T18:05:43.680943+0000","last_undegraded":"2024-01-15T18:05:43.680943+0000","last_fullsized":"2024-01-15T18:05:43.680943+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:17.016743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.659937+0000","last_change":"2024-01-15T18:05:43.656831+0000","last_active":"2024-01-15T18:05:43.659937+0000","last_peered":"2024-01-15T18:05:43.659937+0000","last_clean":"2024-01-15T18:05:43.659937+0000","last_became_active":"2024-01-15T18:05:43.656552+0000","last_became_peered":"2024-01-15T18:05:43.656552+0000","last_unstale":"2024-01-15T18:05:43.659937+0000","last_undegraded":"2024-01-15T18:05:43.659937+0000","last_fullsized":"2024-01-15T18:05:43.659937+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:48:59.746981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668945+0000","last_change":"2024-01-15T18:05:43.660514+0000","last_active":"2024-01-15T18:05:43.668945+0000","last_peered":"2024-01-15T18:05:43.668945+0000","last_clean":"2024-01-15T18:05:43.668945+0000","last_became_active":"2024-01-15T18:05:43.660188+0000","last_became_peered":"2024-01-15T18:05:43.660188+0000","last_unstale":"2024-01-15T18:05:43.668945+0000","last_undegraded":"2024-01-15T18:05:43.668945+0000","last_fullsized":"2024-01-15T18:05:43.668945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:51.462560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668239+0000","last_change":"2024-01-15T18:05:43.661939+0000","last_active":"2024-01-15T18:05:43.668239+0000","last_peered":"2024-01-15T18:05:43.668239+0000","last_clean":"2024-01-15T18:05:43.668239+0000","last_became_active":"2024-01-15T18:05:43.661468+0000","last_became_peered":"2024-01-15T18:05:43.661468+0000","last_unstale":"2024-01-15T18:05:43.668239+0000","last_undegraded":"2024-01-15T18:05:43.668239+0000","last_fullsized":"2024-01-15T18:05:43.668239+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:00.184025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.670031+0000","last_change":"2024-01-15T18:05:43.664403+0000","last_active":"2024-01-15T18:05:43.670031+0000","last_peered":"2024-01-15T18:05:43.670031+0000","last_clean":"2024-01-15T18:05:43.670031+0000","last_became_active":"2024-01-15T18:05:43.663745+0000","last_became_peered":"2024-01-15T18:05:43.663745+0000","last_unstale":"2024-01-15T18:05:43.670031+0000","last_undegraded":"2024-01-15T18:05:43.670031+0000","last_fullsized":"2024-01-15T18:05:43.670031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:48:59.746981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679765+0000","last_change":"2024-01-15T18:05:43.677291+0000","last_active":"2024-01-15T18:05:43.679765+0000","last_peered":"2024-01-15T18:05:43.679765+0000","last_clean":"2024-01-15T18:05:43.679765+0000","last_became_active":"2024-01-15T18:05:43.677001+0000","last_became_peered":"2024-01-15T18:05:43.677001+0000","last_unstale":"2024-01-15T18:05:43.679765+0000","last_undegraded":"2024-01-15T18:05:43.679765+0000","last_fullsized":"2024-01-15T18:05:43.679765+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:28.191729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668332+0000","last_change":"2024-01-15T18:05:43.664395+0000","last_active":"2024-01-15T18:05:43.668332+0000","last_peered":"2024-01-15T18:05:43.668332+0000","last_clean":"2024-01-15T18:05:43.668332+0000","last_became_active":"2024-01-15T18:05:43.664036+0000","last_became_peered":"2024-01-15T18:05:43.664036+0000","last_unstale":"2024-01-15T18:05:43.668332+0000","last_undegraded":"2024-01-15T18:05:43.668332+0000","last_fullsized":"2024-01-15T18:05:43.668332+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:26.160525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.676598+0000","last_change":"2024-01-15T18:05:43.673812+0000","last_active":"2024-01-15T18:05:43.676598+0000","last_peered":"2024-01-15T18:05:43.676598+0000","last_clean":"2024-01-15T18:05:43.676598+0000","last_became_active":"2024-01-15T18:05:43.673506+0000","last_became_peered":"2024-01-15T18:05:43.673506+0000","last_unstale":"2024-01-15T18:05:43.676598+0000","last_undegraded":"2024-01-15T18:05:43.676598+0000","last_fullsized":"2024-01-15T18:05:43.676598+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:32.434900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.672872+0000","last_change":"2024-01-15T18:05:43.666518+0000","last_active":"2024-01-15T18:05:43.672872+0000","last_peered":"2024-01-15T18:05:43.672872+0000","last_clean":"2024-01-15T18:05:43.672872+0000","last_became_active":"2024-01-15T18:05:43.666127+0000","last_became_peered":"2024-01-15T18:05:43.666127+0000","last_unstale":"2024-01-15T18:05:43.672872+0000","last_undegraded":"2024-01-15T18:05:43.672872+0000","last_fullsized":"2024-01-15T18:05:43.672872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:41.284860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679029+0000","last_change":"2024-01-15T18:05:43.675917+0000","last_active":"2024-01-15T18:05:43.679029+0000","last_peered":"2024-01-15T18:05:43.679029+0000","last_clean":"2024-01-15T18:05:43.679029+0000","last_became_active":"2024-01-15T18:05:43.675641+0000","last_became_peered":"2024-01-15T18:05:43.675641+0000","last_unstale":"2024-01-15T18:05:43.679029+0000","last_undegraded":"2024-01-15T18:05:43.679029+0000","last_fullsized":"2024-01-15T18:05:43.679029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:20.927398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.671868+0000","last_change":"2024-01-15T18:05:43.665148+0000","last_active":"2024-01-15T18:05:43.671868+0000","last_peered":"2024-01-15T18:05:43.671868+0000","last_clean":"2024-01-15T18:05:43.671868+0000","last_became_active":"2024-01-15T18:05:43.664873+0000","last_became_peered":"2024-01-15T18:05:43.664873+0000","last_unstale":"2024-01-15T18:05:43.671868+0000","last_undegraded":"2024-01-15T18:05:43.671868+0000","last_fullsized":"2024-01-15T18:05:43.671868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:34.891192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.676641+0000","last_change":"2024-01-15T18:05:43.674794+0000","last_active":"2024-01-15T18:05:43.676641+0000","last_peered":"2024-01-15T18:05:43.676641+0000","last_clean":"2024-01-15T18:05:43.676641+0000","last_became_active":"2024-01-15T18:05:43.673468+0000","last_became_peered":"2024-01-15T18:05:43.673468+0000","last_unstale":"2024-01-15T18:05:43.676641+0000","last_undegraded":"2024-01-15T18:05:43.676641+0000","last_fullsized":"2024-01-15T18:05:43.676641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:37.136328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.671103+0000","last_change":"2024-01-15T18:05:43.667583+0000","last_active":"2024-01-15T18:05:43.671103+0000","last_peered":"2024-01-15T18:05:43.671103+0000","last_clean":"2024-01-15T18:05:43.671103+0000","last_became_active":"2024-01-15T18:05:43.667317+0000","last_became_peered":"2024-01-15T18:05:43.667317+0000","last_unstale":"2024-01-15T18:05:43.671103+0000","last_undegraded":"2024-01-15T18:05:43.671103+0000","last_fullsized":"2024-01-15T18:05:43.671103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:40.878771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679354+0000","last_change":"2024-01-15T18:05:43.674945+0000","last_active":"2024-01-15T18:05:43.679354+0000","last_peered":"2024-01-15T18:05:43.679354+0000","last_clean":"2024-01-15T18:05:43.679354+0000","last_became_active":"2024-01-15T18:05:43.674367+0000","last_became_peered":"2024-01-15T18:05:43.674367+0000","last_unstale":"2024-01-15T18:05:43.679354+0000","last_undegraded":"2024-01-15T18:05:43.679354+0000","last_fullsized":"2024-01-15T18:05:43.679354+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.998414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680175+0000","last_change":"2024-01-15T18:05:43.678756+0000","last_active":"2024-01-15T18:05:43.680175+0000","last_peered":"2024-01-15T18:05:43.680175+0000","last_clean":"2024-01-15T18:05:43.680175+0000","last_became_active":"2024-01-15T18:05:43.678334+0000","last_became_peered":"2024-01-15T18:05:43.678334+0000","last_unstale":"2024-01-15T18:05:43.680175+0000","last_undegraded":"2024-01-15T18:05:43.680175+0000","last_fullsized":"2024-01-15T18:05:43.680175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:13.141022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"72'4","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.687563+0000","last_change":"2024-01-15T18:05:42.647533+0000","last_active":"2024-01-15T18:05:43.687563+0000","last_peered":"2024-01-15T18:05:43.687563+0000","last_clean":"2024-01-15T18:05:43.687563+0000","last_became_active":"2024-01-15T18:05:42.646160+0000","last_became_peered":"2024-01-15T18:05:42.646160+0000","last_unstale":"2024-01-15T18:05:43.687563+0000","last_undegraded":"2024-01-15T18:05:43.687563+0000","last_fullsized":"2024-01-15T18:05:43.687563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:08:51.487836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680431+0000","last_change":"2024-01-15T18:05:43.674056+0000","last_active":"2024-01-15T18:05:43.680431+0000","last_peered":"2024-01-15T18:05:43.680431+0000","last_clean":"2024-01-15T18:05:43.680431+0000","last_became_active":"2024-01-15T18:05:43.670293+0000","last_became_peered":"2024-01-15T18:05:43.670293+0000","last_unstale":"2024-01-15T18:05:43.680431+0000","last_undegraded":"2024-01-15T18:05:43.680431+0000","last_fullsized":"2024-01-15T18:05:43.680431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:46.968578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.685599+0000","last_change":"2024-01-15T18:05:42.632765+0000","last_active":"2024-01-15T18:05:43.685599+0000","last_peered":"2024-01-15T18:05:43.685599+0000","last_clean":"2024-01-15T18:05:43.685599+0000","last_became_active":"2024-01-15T18:05:42.632474+0000","last_became_peered":"2024-01-15T18:05:42.632474+0000","last_unstale":"2024-01-15T18:05:43.685599+0000","last_undegraded":"2024-01-15T18:05:43.685599+0000","last_fullsized":"2024-01-15T18:05:43.685599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:21.746449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.682665+0000","last_change":"2024-01-15T18:05:43.678962+0000","last_active":"2024-01-15T18:05:43.682665+0000","last_peered":"2024-01-15T18:05:43.682665+0000","last_clean":"2024-01-15T18:05:43.682665+0000","last_became_active":"2024-01-15T18:05:43.678461+0000","last_became_peered":"2024-01-15T18:05:43.678461+0000","last_unstale":"2024-01-15T18:05:43.682665+0000","last_undegraded":"2024-01-15T18:05:43.682665+0000","last_fullsized":"2024-01-15T18:05:43.682665+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:12:55.296555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.686204+0000","last_change":"2024-01-15T18:05:42.644508+0000","last_active":"2024-01-15T18:05:43.686204+0000","last_peered":"2024-01-15T18:05:43.686204+0000","last_clean":"2024-01-15T18:05:43.686204+0000","last_became_active":"2024-01-15T18:05:42.644174+0000","last_became_peered":"2024-01-15T18:05:42.644174+0000","last_unstale":"2024-01-15T18:05:43.686204+0000","last_undegraded":"2024-01-15T18:05:43.686204+0000","last_fullsized":"2024-01-15T18:05:43.686204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:45.433793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675194+0000","last_change":"2024-01-15T18:05:43.670585+0000","last_active":"2024-01-15T18:05:43.675194+0000","last_peered":"2024-01-15T18:05:43.675194+0000","last_clean":"2024-01-15T18:05:43.675194+0000","last_became_active":"2024-01-15T18:05:43.669332+0000","last_became_peered":"2024-01-15T18:05:43.669332+0000","last_unstale":"2024-01-15T18:05:43.675194+0000","last_undegraded":"2024-01-15T18:05:43.675194+0000","last_fullsized":"2024-01-15T18:05:43.675194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:19.643686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688428+0000","last_change":"2024-01-15T18:05:42.648112+0000","last_active":"2024-01-15T18:05:43.688428+0000","last_peered":"2024-01-15T18:05:43.688428+0000","last_clean":"2024-01-15T18:05:43.688428+0000","last_became_active":"2024-01-15T18:05:42.647761+0000","last_became_peered":"2024-01-15T18:05:42.647761+0000","last_unstale":"2024-01-15T18:05:43.688428+0000","last_undegraded":"2024-01-15T18:05:43.688428+0000","last_fullsized":"2024-01-15T18:05:43.688428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:50.052265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.669342+0000","last_change":"2024-01-15T18:05:43.663667+0000","last_active":"2024-01-15T18:05:43.669342+0000","last_peered":"2024-01-15T18:05:43.669342+0000","last_clean":"2024-01-15T18:05:43.669342+0000","last_became_active":"2024-01-15T18:05:43.662601+0000","last_became_peered":"2024-01-15T18:05:43.662601+0000","last_unstale":"2024-01-15T18:05:43.669342+0000","last_undegraded":"2024-01-15T18:05:43.669342+0000","last_fullsized":"2024-01-15T18:05:43.669342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:51:48.516632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.687488+0000","last_change":"2024-01-15T18:05:42.642739+0000","last_active":"2024-01-15T18:05:43.687488+0000","last_peered":"2024-01-15T18:05:43.687488+0000","last_clean":"2024-01-15T18:05:43.687488+0000","last_became_active":"2024-01-15T18:05:42.642345+0000","last_became_peered":"2024-01-15T18:05:42.642345+0000","last_unstale":"2024-01-15T18:05:43.687488+0000","last_undegraded":"2024-01-15T18:05:43.687488+0000","last_fullsized":"2024-01-15T18:05:43.687488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:54.080792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668966+0000","last_change":"2024-01-15T18:05:43.664935+0000","last_active":"2024-01-15T18:05:43.668966+0000","last_peered":"2024-01-15T18:05:43.668966+0000","last_clean":"2024-01-15T18:05:43.668966+0000","last_became_active":"2024-01-15T18:05:43.661915+0000","last_became_peered":"2024-01-15T18:05:43.661915+0000","last_unstale":"2024-01-15T18:05:43.668966+0000","last_undegraded":"2024-01-15T18:05:43.668966+0000","last_fullsized":"2024-01-15T18:05:43.668966+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:40.831307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688766+0000","last_change":"2024-01-15T18:05:42.641270+0000","last_active":"2024-01-15T18:05:43.688766+0000","last_peered":"2024-01-15T18:05:43.688766+0000","last_clean":"2024-01-15T18:05:43.688766+0000","last_became_active":"2024-01-15T18:05:42.639544+0000","last_became_peered":"2024-01-15T18:05:42.639544+0000","last_unstale":"2024-01-15T18:05:43.688766+0000","last_undegraded":"2024-01-15T18:05:43.688766+0000","last_fullsized":"2024-01-15T18:05:43.688766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:03:38.120276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677405+0000","last_change":"2024-01-15T18:05:43.670901+0000","last_active":"2024-01-15T18:05:43.677405+0000","last_peered":"2024-01-15T18:05:43.677405+0000","last_clean":"2024-01-15T18:05:43.677405+0000","last_became_active":"2024-01-15T18:05:43.670525+0000","last_became_peered":"2024-01-15T18:05:43.670525+0000","last_unstale":"2024-01-15T18:05:43.677405+0000","last_undegraded":"2024-01-15T18:05:43.677405+0000","last_fullsized":"2024-01-15T18:05:43.677405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:51:16.047771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.637181+0000","last_change":"2024-01-15T18:05:42.631403+0000","last_active":"2024-01-15T18:05:43.637181+0000","last_peered":"2024-01-15T18:05:43.637181+0000","last_clean":"2024-01-15T18:05:43.637181+0000","last_became_active":"2024-01-15T18:05:42.629617+0000","last_became_peered":"2024-01-15T18:05:42.629617+0000","last_unstale":"2024-01-15T18:05:43.637181+0000","last_undegraded":"2024-01-15T18:05:43.637181+0000","last_fullsized":"2024-01-15T18:05:43.637181+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:27:19.291541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668887+0000","last_change":"2024-01-15T18:05:43.667281+0000","last_active":"2024-01-15T18:05:43.668887+0000","last_peered":"2024-01-15T18:05:43.668887+0000","last_clean":"2024-01-15T18:05:43.668887+0000","last_became_active":"2024-01-15T18:05:43.664866+0000","last_became_peered":"2024-01-15T18:05:43.664866+0000","last_unstale":"2024-01-15T18:05:43.668887+0000","last_undegraded":"2024-01-15T18:05:43.668887+0000","last_fullsized":"2024-01-15T18:05:43.668887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:28.191729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.656238+0000","last_change":"2024-01-15T18:05:42.638734+0000","last_active":"2024-01-15T18:05:43.656238+0000","last_peered":"2024-01-15T18:05:43.656238+0000","last_clean":"2024-01-15T18:05:43.656238+0000","last_became_active":"2024-01-15T18:05:42.638109+0000","last_became_peered":"2024-01-15T18:05:42.638109+0000","last_unstale":"2024-01-15T18:05:43.656238+0000","last_undegraded":"2024-01-15T18:05:43.656238+0000","last_fullsized":"2024-01-15T18:05:43.656238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:55.437694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668645+0000","last_change":"2024-01-15T18:05:43.662916+0000","last_active":"2024-01-15T18:05:43.668645+0000","last_peered":"2024-01-15T18:05:43.668645+0000","last_clean":"2024-01-15T18:05:43.668645+0000","last_became_active":"2024-01-15T18:05:43.662392+0000","last_became_peered":"2024-01-15T18:05:43.662392+0000","last_unstale":"2024-01-15T18:05:43.668645+0000","last_undegraded":"2024-01-15T18:05:43.668645+0000","last_fullsized":"2024-01-15T18:05:43.668645+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:48:59.746981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.646999+0000","last_change":"2024-01-15T18:05:42.630224+0000","last_active":"2024-01-15T18:05:43.646999+0000","last_peered":"2024-01-15T18:05:43.646999+0000","last_clean":"2024-01-15T18:05:43.646999+0000","last_became_active":"2024-01-15T18:05:42.629918+0000","last_became_peered":"2024-01-15T18:05:42.629918+0000","last_unstale":"2024-01-15T18:05:43.646999+0000","last_undegraded":"2024-01-15T18:05:43.646999+0000","last_fullsized":"2024-01-15T18:05:43.646999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:50.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680062+0000","last_change":"2024-01-15T18:05:43.677656+0000","last_active":"2024-01-15T18:05:43.680062+0000","last_peered":"2024-01-15T18:05:43.680062+0000","last_clean":"2024-01-15T18:05:43.680062+0000","last_became_active":"2024-01-15T18:05:43.677375+0000","last_became_peered":"2024-01-15T18:05:43.677375+0000","last_unstale":"2024-01-15T18:05:43.680062+0000","last_undegraded":"2024-01-15T18:05:43.680062+0000","last_fullsized":"2024-01-15T18:05:43.680062+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:40.482537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.665905+0000","last_change":"2024-01-15T18:05:42.642561+0000","last_active":"2024-01-15T18:05:43.665905+0000","last_peered":"2024-01-15T18:05:43.665905+0000","last_clean":"2024-01-15T18:05:43.665905+0000","last_became_active":"2024-01-15T18:05:42.641744+0000","last_became_peered":"2024-01-15T18:05:42.641744+0000","last_unstale":"2024-01-15T18:05:43.665905+0000","last_undegraded":"2024-01-15T18:05:43.665905+0000","last_fullsized":"2024-01-15T18:05:43.665905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:43:35.993670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.678650+0000","last_change":"2024-01-15T18:05:43.673134+0000","last_active":"2024-01-15T18:05:43.678650+0000","last_peered":"2024-01-15T18:05:43.678650+0000","last_clean":"2024-01-15T18:05:43.678650+0000","last_became_active":"2024-01-15T18:05:43.672347+0000","last_became_peered":"2024-01-15T18:05:43.672347+0000","last_unstale":"2024-01-15T18:05:43.678650+0000","last_undegraded":"2024-01-15T18:05:43.678650+0000","last_fullsized":"2024-01-15T18:05:43.678650+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:26:10.727244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"72'3","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688442+0000","last_change":"2024-01-15T18:05:42.629845+0000","last_active":"2024-01-15T18:05:43.688442+0000","last_peered":"2024-01-15T18:05:43.688442+0000","last_clean":"2024-01-15T18:05:43.688442+0000","last_became_active":"2024-01-15T18:05:42.629526+0000","last_became_peered":"2024-01-15T18:05:42.629526+0000","last_unstale":"2024-01-15T18:05:43.688442+0000","last_undegraded":"2024-01-15T18:05:43.688442+0000","last_fullsized":"2024-01-15T18:05:43.688442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:52.355190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.667131+0000","last_change":"2024-01-15T18:05:43.662932+0000","last_active":"2024-01-15T18:05:43.667131+0000","last_peered":"2024-01-15T18:05:43.667131+0000","last_clean":"2024-01-15T18:05:43.667131+0000","last_became_active":"2024-01-15T18:05:43.662646+0000","last_became_peered":"2024-01-15T18:05:43.662646+0000","last_unstale":"2024-01-15T18:05:43.667131+0000","last_undegraded":"2024-01-15T18:05:43.667131+0000","last_fullsized":"2024-01-15T18:05:43.667131+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:59:16.414665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"72'6","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.689531+0000","last_change":"2024-01-15T18:05:42.638256+0000","last_active":"2024-01-15T18:05:43.689531+0000","last_peered":"2024-01-15T18:05:43.689531+0000","last_clean":"2024-01-15T18:05:43.689531+0000","last_became_active":"2024-01-15T18:05:42.637935+0000","last_became_peered":"2024-01-15T18:05:42.637935+0000","last_unstale":"2024-01-15T18:05:43.689531+0000","last_undegraded":"2024-01-15T18:05:43.689531+0000","last_fullsized":"2024-01-15T18:05:43.689531+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:10.523858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677105+0000","last_change":"2024-01-15T18:05:43.673787+0000","last_active":"2024-01-15T18:05:43.677105+0000","last_peered":"2024-01-15T18:05:43.677105+0000","last_clean":"2024-01-15T18:05:43.677105+0000","last_became_active":"2024-01-15T18:05:43.673490+0000","last_became_peered":"2024-01-15T18:05:43.673490+0000","last_unstale":"2024-01-15T18:05:43.677105+0000","last_undegraded":"2024-01-15T18:05:43.677105+0000","last_fullsized":"2024-01-15T18:05:43.677105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:32.434900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.685442+0000","last_change":"2024-01-15T18:05:42.641185+0000","last_active":"2024-01-15T18:05:43.685442+0000","last_peered":"2024-01-15T18:05:43.685442+0000","last_clean":"2024-01-15T18:05:43.685442+0000","last_became_active":"2024-01-15T18:05:42.640872+0000","last_became_peered":"2024-01-15T18:05:42.640872+0000","last_unstale":"2024-01-15T18:05:43.685442+0000","last_undegraded":"2024-01-15T18:05:43.685442+0000","last_fullsized":"2024-01-15T18:05:43.685442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:56:19.406839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.669334+0000","last_change":"2024-01-15T18:05:43.667185+0000","last_active":"2024-01-15T18:05:43.669334+0000","last_peered":"2024-01-15T18:05:43.669334+0000","last_clean":"2024-01-15T18:05:43.669334+0000","last_became_active":"2024-01-15T18:05:43.666749+0000","last_became_peered":"2024-01-15T18:05:43.666749+0000","last_unstale":"2024-01-15T18:05:43.669334+0000","last_undegraded":"2024-01-15T18:05:43.669334+0000","last_fullsized":"2024-01-15T18:05:43.669334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:28.191729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.666045+0000","last_change":"2024-01-15T18:05:42.641169+0000","last_active":"2024-01-15T18:05:43.666045+0000","last_peered":"2024-01-15T18:05:43.666045+0000","last_clean":"2024-01-15T18:05:43.666045+0000","last_became_active":"2024-01-15T18:05:42.640910+0000","last_became_peered":"2024-01-15T18:05:42.640910+0000","last_unstale":"2024-01-15T18:05:43.666045+0000","last_undegraded":"2024-01-15T18:05:43.666045+0000","last_fullsized":"2024-01-15T18:05:43.666045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:53.872641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674136+0000","last_change":"2024-01-15T18:05:43.672335+0000","last_active":"2024-01-15T18:05:43.674136+0000","last_peered":"2024-01-15T18:05:43.674136+0000","last_clean":"2024-01-15T18:05:43.674136+0000","last_became_active":"2024-01-15T18:05:43.671991+0000","last_became_peered":"2024-01-15T18:05:43.671991+0000","last_unstale":"2024-01-15T18:05:43.674136+0000","last_undegraded":"2024-01-15T18:05:43.674136+0000","last_fullsized":"2024-01-15T18:05:43.674136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:13.141022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.640679+0000","last_change":"2024-01-15T18:05:42.634255+0000","last_active":"2024-01-15T18:05:43.640679+0000","last_peered":"2024-01-15T18:05:43.640679+0000","last_clean":"2024-01-15T18:05:43.640679+0000","last_became_active":"2024-01-15T18:05:42.632600+0000","last_became_peered":"2024-01-15T18:05:42.632600+0000","last_unstale":"2024-01-15T18:05:43.640679+0000","last_undegraded":"2024-01-15T18:05:43.640679+0000","last_fullsized":"2024-01-15T18:05:43.640679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:21.031022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677921+0000","last_change":"2024-01-15T18:05:43.673024+0000","last_active":"2024-01-15T18:05:43.677921+0000","last_peered":"2024-01-15T18:05:43.677921+0000","last_clean":"2024-01-15T18:05:43.677921+0000","last_became_active":"2024-01-15T18:05:43.669764+0000","last_became_peered":"2024-01-15T18:05:43.669764+0000","last_unstale":"2024-01-15T18:05:43.677921+0000","last_undegraded":"2024-01-15T18:05:43.677921+0000","last_fullsized":"2024-01-15T18:05:43.677921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:45:52.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"72'8","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.689856+0000","last_change":"2024-01-15T18:05:42.641745+0000","last_active":"2024-01-15T18:05:43.689856+0000","last_peered":"2024-01-15T18:05:43.689856+0000","last_clean":"2024-01-15T18:05:43.689856+0000","last_became_active":"2024-01-15T18:05:42.641369+0000","last_became_peered":"2024-01-15T18:05:42.641369+0000","last_unstale":"2024-01-15T18:05:43.689856+0000","last_undegraded":"2024-01-15T18:05:43.689856+0000","last_fullsized":"2024-01-15T18:05:43.689856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:55:42.282640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.667593+0000","last_change":"2024-01-15T18:05:43.659383+0000","last_active":"2024-01-15T18:05:43.667593+0000","last_peered":"2024-01-15T18:05:43.667593+0000","last_clean":"2024-01-15T18:05:43.667593+0000","last_became_active":"2024-01-15T18:05:43.658770+0000","last_became_peered":"2024-01-15T18:05:43.658770+0000","last_unstale":"2024-01-15T18:05:43.667593+0000","last_undegraded":"2024-01-15T18:05:43.667593+0000","last_fullsized":"2024-01-15T18:05:43.667593+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:15.409561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688172+0000","last_change":"2024-01-15T18:05:42.647173+0000","last_active":"2024-01-15T18:05:43.688172+0000","last_peered":"2024-01-15T18:05:43.688172+0000","last_clean":"2024-01-15T18:05:43.688172+0000","last_became_active":"2024-01-15T18:05:42.646897+0000","last_became_peered":"2024-01-15T18:05:42.646897+0000","last_unstale":"2024-01-15T18:05:43.688172+0000","last_undegraded":"2024-01-15T18:05:43.688172+0000","last_fullsized":"2024-01-15T18:05:43.688172+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:13:50.534577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675206+0000","last_change":"2024-01-15T18:05:43.671557+0000","last_active":"2024-01-15T18:05:43.675206+0000","last_peered":"2024-01-15T18:05:43.675206+0000","last_clean":"2024-01-15T18:05:43.675206+0000","last_became_active":"2024-01-15T18:05:43.671181+0000","last_became_peered":"2024-01-15T18:05:43.671181+0000","last_unstale":"2024-01-15T18:05:43.675206+0000","last_undegraded":"2024-01-15T18:05:43.675206+0000","last_fullsized":"2024-01-15T18:05:43.675206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:08.605428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.686282+0000","last_change":"2024-01-15T18:05:42.648070+0000","last_active":"2024-01-15T18:05:43.686282+0000","last_peered":"2024-01-15T18:05:43.686282+0000","last_clean":"2024-01-15T18:05:43.686282+0000","last_became_active":"2024-01-15T18:05:42.647154+0000","last_became_peered":"2024-01-15T18:05:42.647154+0000","last_unstale":"2024-01-15T18:05:43.686282+0000","last_undegraded":"2024-01-15T18:05:43.686282+0000","last_fullsized":"2024-01-15T18:05:43.686282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:39.583721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675004+0000","last_change":"2024-01-15T18:05:43.667405+0000","last_active":"2024-01-15T18:05:43.675004+0000","last_peered":"2024-01-15T18:05:43.675004+0000","last_clean":"2024-01-15T18:05:43.675004+0000","last_became_active":"2024-01-15T18:05:43.665200+0000","last_became_peered":"2024-01-15T18:05:43.665200+0000","last_unstale":"2024-01-15T18:05:43.675004+0000","last_undegraded":"2024-01-15T18:05:43.675004+0000","last_fullsized":"2024-01-15T18:05:43.675004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:16:02.770737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.689883+0000","last_change":"2024-01-15T18:05:42.650648+0000","last_active":"2024-01-15T18:05:43.689883+0000","last_peered":"2024-01-15T18:05:43.689883+0000","last_clean":"2024-01-15T18:05:43.689883+0000","last_became_active":"2024-01-15T18:05:42.649790+0000","last_became_peered":"2024-01-15T18:05:42.649790+0000","last_unstale":"2024-01-15T18:05:43.689883+0000","last_undegraded":"2024-01-15T18:05:43.689883+0000","last_fullsized":"2024-01-15T18:05:43.689883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:14:04.306129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.681688+0000","last_change":"2024-01-15T18:05:43.677178+0000","last_active":"2024-01-15T18:05:43.681688+0000","last_peered":"2024-01-15T18:05:43.681688+0000","last_clean":"2024-01-15T18:05:43.681688+0000","last_became_active":"2024-01-15T18:05:43.674372+0000","last_became_peered":"2024-01-15T18:05:43.674372+0000","last_unstale":"2024-01-15T18:05:43.681688+0000","last_undegraded":"2024-01-15T18:05:43.681688+0000","last_fullsized":"2024-01-15T18:05:43.681688+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:34.551767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.659854+0000","last_change":"2024-01-15T18:05:42.648607+0000","last_active":"2024-01-15T18:05:43.659854+0000","last_peered":"2024-01-15T18:05:43.659854+0000","last_clean":"2024-01-15T18:05:43.659854+0000","last_became_active":"2024-01-15T18:05:42.648153+0000","last_became_peered":"2024-01-15T18:05:42.648153+0000","last_unstale":"2024-01-15T18:05:43.659854+0000","last_undegraded":"2024-01-15T18:05:43.659854+0000","last_fullsized":"2024-01-15T18:05:43.659854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:05.055793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.670898+0000","last_change":"2024-01-15T18:05:43.666345+0000","last_active":"2024-01-15T18:05:43.670898+0000","last_peered":"2024-01-15T18:05:43.670898+0000","last_clean":"2024-01-15T18:05:43.670898+0000","last_became_active":"2024-01-15T18:05:43.666032+0000","last_became_peered":"2024-01-15T18:05:43.666032+0000","last_unstale":"2024-01-15T18:05:43.670898+0000","last_undegraded":"2024-01-15T18:05:43.670898+0000","last_fullsized":"2024-01-15T18:05:43.670898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:43:38.909893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.666122+0000","last_change":"2024-01-15T18:05:42.640152+0000","last_active":"2024-01-15T18:05:43.666122+0000","last_peered":"2024-01-15T18:05:43.666122+0000","last_clean":"2024-01-15T18:05:43.666122+0000","last_became_active":"2024-01-15T18:05:42.639183+0000","last_became_peered":"2024-01-15T18:05:42.639183+0000","last_unstale":"2024-01-15T18:05:43.666122+0000","last_undegraded":"2024-01-15T18:05:43.666122+0000","last_fullsized":"2024-01-15T18:05:43.666122+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:18:26.930742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679057+0000","last_change":"2024-01-15T18:05:43.677096+0000","last_active":"2024-01-15T18:05:43.679057+0000","last_peered":"2024-01-15T18:05:43.679057+0000","last_clean":"2024-01-15T18:05:43.679057+0000","last_became_active":"2024-01-15T18:05:43.676730+0000","last_became_peered":"2024-01-15T18:05:43.676730+0000","last_unstale":"2024-01-15T18:05:43.679057+0000","last_undegraded":"2024-01-15T18:05:43.679057+0000","last_fullsized":"2024-01-15T18:05:43.679057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:53.547860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.641071+0000","last_change":"2024-01-15T18:05:42.638308+0000","last_active":"2024-01-15T18:05:43.641071+0000","last_peered":"2024-01-15T18:05:43.641071+0000","last_clean":"2024-01-15T18:05:43.641071+0000","last_became_active":"2024-01-15T18:05:42.637580+0000","last_became_peered":"2024-01-15T18:05:42.637580+0000","last_unstale":"2024-01-15T18:05:43.641071+0000","last_undegraded":"2024-01-15T18:05:43.641071+0000","last_fullsized":"2024-01-15T18:05:43.641071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:53.603664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.673677+0000","last_change":"2024-01-15T18:05:43.669038+0000","last_active":"2024-01-15T18:05:43.673677+0000","last_peered":"2024-01-15T18:05:43.673677+0000","last_clean":"2024-01-15T18:05:43.673677+0000","last_became_active":"2024-01-15T18:05:43.668736+0000","last_became_peered":"2024-01-15T18:05:43.668736+0000","last_unstale":"2024-01-15T18:05:43.673677+0000","last_undegraded":"2024-01-15T18:05:43.673677+0000","last_fullsized":"2024-01-15T18:05:43.673677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:54.034157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.650694+0000","last_change":"2024-01-15T18:05:42.636058+0000","last_active":"2024-01-15T18:05:43.650694+0000","last_peered":"2024-01-15T18:05:43.650694+0000","last_clean":"2024-01-15T18:05:43.650694+0000","last_became_active":"2024-01-15T18:05:42.634995+0000","last_became_peered":"2024-01-15T18:05:42.634995+0000","last_unstale":"2024-01-15T18:05:43.650694+0000","last_undegraded":"2024-01-15T18:05:43.650694+0000","last_fullsized":"2024-01-15T18:05:43.650694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:18.629291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.673727+0000","last_change":"2024-01-15T18:05:43.671119+0000","last_active":"2024-01-15T18:05:43.673727+0000","last_peered":"2024-01-15T18:05:43.673727+0000","last_clean":"2024-01-15T18:05:43.673727+0000","last_became_active":"2024-01-15T18:05:43.669879+0000","last_became_peered":"2024-01-15T18:05:43.669879+0000","last_unstale":"2024-01-15T18:05:43.673727+0000","last_undegraded":"2024-01-15T18:05:43.673727+0000","last_fullsized":"2024-01-15T18:05:43.673727+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:37.136328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.687897+0000","last_change":"2024-01-15T18:05:42.645829+0000","last_active":"2024-01-15T18:05:43.687897+0000","last_peered":"2024-01-15T18:05:43.687897+0000","last_clean":"2024-01-15T18:05:43.687897+0000","last_became_active":"2024-01-15T18:05:42.645555+0000","last_became_peered":"2024-01-15T18:05:42.645555+0000","last_unstale":"2024-01-15T18:05:43.687897+0000","last_undegraded":"2024-01-15T18:05:43.687897+0000","last_fullsized":"2024-01-15T18:05:43.687897+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:28.798744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668193+0000","last_change":"2024-01-15T18:05:43.662334+0000","last_active":"2024-01-15T18:05:43.668193+0000","last_peered":"2024-01-15T18:05:43.668193+0000","last_clean":"2024-01-15T18:05:43.668193+0000","last_became_active":"2024-01-15T18:05:43.661858+0000","last_became_peered":"2024-01-15T18:05:43.661858+0000","last_unstale":"2024-01-15T18:05:43.668193+0000","last_undegraded":"2024-01-15T18:05:43.668193+0000","last_fullsized":"2024-01-15T18:05:43.668193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:19.643686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.646443+0000","last_change":"2024-01-15T18:05:42.632413+0000","last_active":"2024-01-15T18:05:43.646443+0000","last_peered":"2024-01-15T18:05:43.646443+0000","last_clean":"2024-01-15T18:05:43.646443+0000","last_became_active":"2024-01-15T18:05:42.632094+0000","last_became_peered":"2024-01-15T18:05:42.632094+0000","last_unstale":"2024-01-15T18:05:43.646443+0000","last_undegraded":"2024-01-15T18:05:43.646443+0000","last_fullsized":"2024-01-15T18:05:43.646443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:12.126627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680213+0000","last_change":"2024-01-15T18:05:43.678180+0000","last_active":"2024-01-15T18:05:43.680213+0000","last_peered":"2024-01-15T18:05:43.680213+0000","last_clean":"2024-01-15T18:05:43.680213+0000","last_became_active":"2024-01-15T18:05:43.677877+0000","last_became_peered":"2024-01-15T18:05:43.677877+0000","last_unstale":"2024-01-15T18:05:43.680213+0000","last_undegraded":"2024-01-15T18:05:43.680213+0000","last_fullsized":"2024-01-15T18:05:43.680213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:01:39.810030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.651746+0000","last_change":"2024-01-15T18:05:42.632767+0000","last_active":"2024-01-15T18:05:43.651746+0000","last_peered":"2024-01-15T18:05:43.651746+0000","last_clean":"2024-01-15T18:05:43.651746+0000","last_became_active":"2024-01-15T18:05:42.632423+0000","last_became_peered":"2024-01-15T18:05:42.632423+0000","last_unstale":"2024-01-15T18:05:43.651746+0000","last_undegraded":"2024-01-15T18:05:43.651746+0000","last_fullsized":"2024-01-15T18:05:43.651746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:00:54.056977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.671429+0000","last_change":"2024-01-15T18:05:43.668793+0000","last_active":"2024-01-15T18:05:43.671429+0000","last_peered":"2024-01-15T18:05:43.671429+0000","last_clean":"2024-01-15T18:05:43.671429+0000","last_became_active":"2024-01-15T18:05:43.668352+0000","last_became_peered":"2024-01-15T18:05:43.668352+0000","last_unstale":"2024-01-15T18:05:43.671429+0000","last_undegraded":"2024-01-15T18:05:43.671429+0000","last_fullsized":"2024-01-15T18:05:43.671429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:22.760844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.652502+0000","last_change":"2024-01-15T18:05:42.626386+0000","last_active":"2024-01-15T18:05:43.652502+0000","last_peered":"2024-01-15T18:05:43.652502+0000","last_clean":"2024-01-15T18:05:43.652502+0000","last_became_active":"2024-01-15T18:05:42.626113+0000","last_became_peered":"2024-01-15T18:05:42.626113+0000","last_unstale":"2024-01-15T18:05:43.652502+0000","last_undegraded":"2024-01-15T18:05:43.652502+0000","last_fullsized":"2024-01-15T18:05:43.652502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:05.055793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668188+0000","last_change":"2024-01-15T18:05:43.664055+0000","last_active":"2024-01-15T18:05:43.668188+0000","last_peered":"2024-01-15T18:05:43.668188+0000","last_clean":"2024-01-15T18:05:43.668188+0000","last_became_active":"2024-01-15T18:05:43.663752+0000","last_became_peered":"2024-01-15T18:05:43.663752+0000","last_unstale":"2024-01-15T18:05:43.668188+0000","last_undegraded":"2024-01-15T18:05:43.668188+0000","last_fullsized":"2024-01-15T18:05:43.668188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:38.345675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.656060+0000","last_change":"2024-01-15T18:05:42.633521+0000","last_active":"2024-01-15T18:05:43.656060+0000","last_peered":"2024-01-15T18:05:43.656060+0000","last_clean":"2024-01-15T18:05:43.656060+0000","last_became_active":"2024-01-15T18:05:42.633251+0000","last_became_peered":"2024-01-15T18:05:42.633251+0000","last_unstale":"2024-01-15T18:05:43.656060+0000","last_undegraded":"2024-01-15T18:05:43.656060+0000","last_fullsized":"2024-01-15T18:05:43.656060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:21.796601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680743+0000","last_change":"2024-01-15T18:05:43.678142+0000","last_active":"2024-01-15T18:05:43.680743+0000","last_peered":"2024-01-15T18:05:43.680743+0000","last_clean":"2024-01-15T18:05:43.680743+0000","last_became_active":"2024-01-15T18:05:43.677866+0000","last_became_peered":"2024-01-15T18:05:43.677866+0000","last_unstale":"2024-01-15T18:05:43.680743+0000","last_undegraded":"2024-01-15T18:05:43.680743+0000","last_fullsized":"2024-01-15T18:05:43.680743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:52.224215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.637189+0000","last_change":"2024-01-15T18:05:42.643050+0000","last_active":"2024-01-15T18:05:43.637189+0000","last_peered":"2024-01-15T18:05:43.637189+0000","last_clean":"2024-01-15T18:05:43.637189+0000","last_became_active":"2024-01-15T18:05:42.642447+0000","last_became_peered":"2024-01-15T18:05:42.642447+0000","last_unstale":"2024-01-15T18:05:43.637189+0000","last_undegraded":"2024-01-15T18:05:43.637189+0000","last_fullsized":"2024-01-15T18:05:43.637189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:27.496701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674805+0000","last_change":"2024-01-15T18:05:43.663963+0000","last_active":"2024-01-15T18:05:43.674805+0000","last_peered":"2024-01-15T18:05:43.674805+0000","last_clean":"2024-01-15T18:05:43.674805+0000","last_became_active":"2024-01-15T18:05:43.663679+0000","last_became_peered":"2024-01-15T18:05:43.663679+0000","last_unstale":"2024-01-15T18:05:43.674805+0000","last_undegraded":"2024-01-15T18:05:43.674805+0000","last_fullsized":"2024-01-15T18:05:43.674805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:13:51.548972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.643653+0000","last_change":"2024-01-15T18:05:42.641523+0000","last_active":"2024-01-15T18:05:43.643653+0000","last_peered":"2024-01-15T18:05:43.643653+0000","last_clean":"2024-01-15T18:05:43.643653+0000","last_became_active":"2024-01-15T18:05:42.641199+0000","last_became_peered":"2024-01-15T18:05:42.641199+0000","last_unstale":"2024-01-15T18:05:43.643653+0000","last_undegraded":"2024-01-15T18:05:43.643653+0000","last_fullsized":"2024-01-15T18:05:43.643653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:29.964431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680735+0000","last_change":"2024-01-15T18:05:43.677001+0000","last_active":"2024-01-15T18:05:43.680735+0000","last_peered":"2024-01-15T18:05:43.680735+0000","last_clean":"2024-01-15T18:05:43.680735+0000","last_became_active":"2024-01-15T18:05:43.675396+0000","last_became_peered":"2024-01-15T18:05:43.675396+0000","last_unstale":"2024-01-15T18:05:43.680735+0000","last_undegraded":"2024-01-15T18:05:43.680735+0000","last_fullsized":"2024-01-15T18:05:43.680735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:16:02.770737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668123+0000","last_change":"2024-01-15T18:05:42.648523+0000","last_active":"2024-01-15T18:05:43.668123+0000","last_peered":"2024-01-15T18:05:43.668123+0000","last_clean":"2024-01-15T18:05:43.668123+0000","last_became_active":"2024-01-15T18:05:42.648246+0000","last_became_peered":"2024-01-15T18:05:42.648246+0000","last_unstale":"2024-01-15T18:05:43.668123+0000","last_undegraded":"2024-01-15T18:05:43.668123+0000","last_fullsized":"2024-01-15T18:05:43.668123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:22.607805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.673096+0000","last_change":"2024-01-15T18:05:43.669680+0000","last_active":"2024-01-15T18:05:43.673096+0000","last_peered":"2024-01-15T18:05:43.673096+0000","last_clean":"2024-01-15T18:05:43.673096+0000","last_became_active":"2024-01-15T18:05:43.669373+0000","last_became_peered":"2024-01-15T18:05:43.669373+0000","last_unstale":"2024-01-15T18:05:43.673096+0000","last_undegraded":"2024-01-15T18:05:43.673096+0000","last_fullsized":"2024-01-15T18:05:43.673096+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:19:08.869684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688290+0000","last_change":"2024-01-15T18:05:42.647945+0000","last_active":"2024-01-15T18:05:43.688290+0000","last_peered":"2024-01-15T18:05:43.688290+0000","last_clean":"2024-01-15T18:05:43.688290+0000","last_became_active":"2024-01-15T18:05:42.647645+0000","last_became_peered":"2024-01-15T18:05:42.647645+0000","last_unstale":"2024-01-15T18:05:43.688290+0000","last_undegraded":"2024-01-15T18:05:43.688290+0000","last_fullsized":"2024-01-15T18:05:43.688290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:05:57.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":211,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-15T18:05:42.642168+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:05:42.642168+0000","last_peered":"2024-01-15T18:05:42.642168+0000","last_clean":"2024-01-15T18:05:42.642168+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:05:42.642168+0000","last_undegraded":"2024-01-15T18:05:42.642168+0000","last_fullsized":"2024-01-15T18:05:42.642168+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542246,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50668,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321172,"statfs":{"total":96636764160,"available":96584880128,"internally_reserved":0,"allocated":1474560,"data_stored":962024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.7030000000000001,"15min":1.304},"min":{"1min":0.84599999999999997,"5min":0.622,"15min":0.53600000000000003},"max":{"1min":9.1379999999999999,"5min":14.114000000000001,"15min":14.114000000000001},"last":1.407},{"interface":"front","average":{"1min":1.73,"5min":1.601,"15min":1.4790000000000001},"min":{"1min":1.0569999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.0289999999999999,"5min":12.496,"15min":12.496},"last":1.1679999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.4590000000000001,"15min":1.224},"min":{"1min":0.67300000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.9210000000000003,"5min":14.497,"15min":14.497},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.748,"5min":1.6399999999999999,"15min":1.292},"min":{"1min":0.70599999999999996,"5min":0.69899999999999995,"15min":0.59499999999999997},"max":{"1min":8.5980000000000008,"5min":12.565,"15min":12.565},"last":1.0289999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":2.3690000000000002,"5min":1.9139999999999999,"15min":1.3879999999999999},"min":{"1min":0.88200000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":12.709,"5min":13.801,"15min":13.801},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.96,"5min":1.5049999999999999,"15min":1.468},"min":{"1min":0.91300000000000003,"5min":0.61499999999999999,"15min":0.58699999999999997},"max":{"1min":7.0810000000000004,"5min":7.0810000000000004,"15min":7.5940000000000003},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.758,"5min":1.6439999999999999,"15min":1.446},"min":{"1min":0.73099999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":5.6280000000000001,"5min":14.622,"15min":14.622},"last":1.079},{"interface":"front","average":{"1min":2.2149999999999999,"5min":1.8520000000000001,"15min":1.502},"min":{"1min":0.76000000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":16.088999999999999,"5min":16.088999999999999,"15min":16.088999999999999},"last":1.5609999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.5369999999999999,"15min":1.579},"min":{"1min":0.85899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":8.3870000000000005,"5min":8.3870000000000005,"15min":8.3870000000000005},"last":1.452},{"interface":"front","average":{"1min":1.948,"5min":1.6899999999999999,"15min":1.488},"min":{"1min":1.04,"5min":0.438,"15min":0.438},"max":{"1min":7.2489999999999997,"5min":14.711,"15min":14.711},"last":1.234}]},{"osd":5,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.7589999999999999,"15min":1.599},"min":{"1min":0.67000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":7.6079999999999997,"5min":12.518000000000001,"15min":12.518000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.9470000000000001,"5min":1.607,"15min":1.653},"min":{"1min":0.69699999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.1219999999999999,"5min":7.1219999999999999,"15min":7.798},"last":1.5269999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":3.3199999999999998,"15min":2.2730000000000001},"min":{"1min":1.2689999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":8.7620000000000005,"5min":53.664999999999999,"15min":53.664999999999999},"last":1.5800000000000001},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.8580000000000001,"15min":1.5980000000000001},"min":{"1min":1.028,"5min":0.44,"15min":0.44},"max":{"1min":14.220000000000001,"5min":14.804,"15min":14.804},"last":1.355}]}]},{"osd":6,"up_from":14,"seq":60129542246,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50088,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321752,"statfs":{"total":96636764160,"available":96585474048,"internally_reserved":0,"allocated":880640,"data_stored":371132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2549999999999999,"15min":1.139},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":5.1189999999999998,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.611,"15min":1.2869999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":9.548,"5min":21.565000000000001,"15min":21.565000000000001},"last":1.327}]},{"osd":1,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.425,"15min":1.294},"min":{"1min":0.53300000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":6.343,"5min":6.343,"15min":6.343},"last":1.052},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.639,"15min":1.423},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":9.6389999999999993,"5min":13.984999999999999,"15min":13.984999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.7929999999999999,"15min":1.4430000000000001},"min":{"1min":0.79800000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":7.0030000000000001,"5min":34.670000000000002,"15min":34.670000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.494,"15min":1.3520000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":6.585,"5min":6.585,"15min":6.585},"last":0.93200000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.7969999999999999,"15min":1.5089999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":16.855,"5min":16.855,"15min":16.855},"last":1.448},{"interface":"front","average":{"1min":1.7370000000000001,"5min":2.0779999999999998,"15min":1.6240000000000001},"min":{"1min":0.628,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":8.0139999999999993,"5min":34.725000000000001,"15min":34.725000000000001},"last":1.008}]},{"osd":4,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":1.597,"15min":1.4219999999999999},"min":{"1min":0.56299999999999994,"5min":0.52200000000000002,"15min":0.38400000000000001},"max":{"1min":6.5220000000000002,"5min":8.5489999999999995,"15min":8.5489999999999995},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.8420000000000001,"5min":1.583,"15min":1.5109999999999999},"min":{"1min":0.63600000000000001,"5min":0.56599999999999995,"15min":0.501},"max":{"1min":10.118,"5min":10.118,"15min":10.118},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.242,"5min":2.1269999999999998,"15min":1.6100000000000001},"min":{"1min":0.71199999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":7.9500000000000002,"5min":34.798000000000002,"15min":34.798000000000002},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.96,"5min":1.673,"15min":1.5580000000000001},"min":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.47899999999999998},"max":{"1min":10.077,"5min":10.077,"15min":10.077},"last":1.4219999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0099999999999998,"5min":1.7130000000000001,"15min":1.6759999999999999},"min":{"1min":0.78900000000000003,"5min":0.72699999999999998,"15min":0.64000000000000001},"max":{"1min":10.302,"5min":10.302,"15min":10.302},"last":1.5129999999999999},{"interface":"front","average":{"1min":2.113,"5min":2.2869999999999999,"15min":1.8520000000000001},"min":{"1min":0.84599999999999997,"5min":0.61799999999999999,"15min":0.52600000000000002},"max":{"1min":8.0519999999999996,"5min":34.762999999999998,"15min":34.762999999999998},"last":1.1899999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542248,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":876544,"data_stored":371130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.786,"5min":1.7070000000000001,"15min":1.329},"min":{"1min":0.77500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.077999999999999,"5min":26.077999999999999,"15min":26.077999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.6100000000000001,"15min":1.393},"min":{"1min":0.875,"5min":0.71099999999999997,"15min":0.52400000000000002},"max":{"1min":3.2330000000000001,"5min":14.711,"15min":14.711},"last":1.472}]},{"osd":1,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.6040000000000001,"15min":1.3129999999999999},"min":{"1min":0.81200000000000006,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":26,"5min":26,"15min":26},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.4970000000000001,"15min":1.2549999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":3.9239999999999999,"5min":12.619,"15min":12.619},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":3.0819999999999999,"5min":1.7869999999999999,"15min":1.4850000000000001},"min":{"1min":0.73299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":26.234999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.575,"15min":1.4570000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.59099999999999997},"max":{"1min":3.4990000000000001,"5min":10.327999999999999,"15min":10.327999999999999},"last":1.091}]},{"osd":3,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.7090000000000001,"15min":1.4079999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":6.3259999999999996,"5min":12.722,"15min":12.722},"last":0.92700000000000005},{"interface":"front","average":{"1min":3.2770000000000001,"5min":2.0249999999999999,"15min":1.601},"min":{"1min":0.92500000000000004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":26.134,"5min":26.134,"15min":26.134},"last":1.2250000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.7190000000000001,"15min":1.4550000000000001},"min":{"1min":0.88100000000000001,"5min":0.60599999999999998,"15min":0.55900000000000005},"max":{"1min":5.0570000000000004,"5min":25.776,"15min":25.776},"last":0.98599999999999999},{"interface":"front","average":{"1min":2.617,"5min":1.944,"15min":1.6479999999999999},"min":{"1min":0.999,"5min":0.63800000000000001,"15min":0.47799999999999998},"max":{"1min":16.414000000000001,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.385}]},{"osd":6,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.8600000000000001,"15min":1.5269999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":4.7039999999999997,"5min":12.769,"15min":12.769},"last":1.026},{"interface":"front","average":{"1min":2.8279999999999998,"5min":2.4039999999999999,"15min":1.7909999999999999},"min":{"1min":0.49199999999999999,"5min":0.435,"15min":0.435},"max":{"1min":26.186,"5min":36.087000000000003,"15min":36.087000000000003},"last":1.51}]},{"osd":7,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":2.1139999999999999,"15min":1.798},"min":{"1min":0.90200000000000002,"5min":0.745,"15min":0.68000000000000005},"max":{"1min":3.339,"5min":13.706,"15min":13.706},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.621,"5min":2.1669999999999998,"15min":1.756},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":4.7489999999999997,"5min":21.728000000000002,"15min":21.728000000000002},"last":1.266}]}]},{"osd":4,"up_from":14,"seq":60129542247,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50088,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321752,"statfs":{"total":96636764160,"available":96585474048,"internally_reserved":0,"allocated":876544,"data_stored":371110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.3939999999999999,"15min":1.3080000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6599999999999999,"5min":18.673999999999999,"15min":21.643000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.325,"5min":1.4850000000000001,"15min":1.242},"min":{"1min":0.46200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":8.9109999999999996,"5min":11.196,"15min":11.196},"last":1.0409999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.415,"15min":1.258},"min":{"1min":0.41499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.3499999999999996,"5min":12.738,"15min":12.738},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.431,"15min":1.405},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":10.411,"5min":15.096,"15min":21.440000000000001},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.595,"15min":1.3120000000000001},"min":{"1min":0.66500000000000004,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6929999999999996,"5min":12.974,"15min":12.974},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.498,"15min":1.47},"min":{"1min":0.52500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":7.9169999999999998,"5min":16.393000000000001,"15min":22.350999999999999},"last":1.468}]},{"osd":3,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.653,"15min":1.488},"min":{"1min":0.56699999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.9369999999999998,"5min":15.093,"15min":15.093},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.863,"15min":1.5309999999999999},"min":{"1min":0.71699999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.4039999999999999,"5min":19.486000000000001,"15min":19.486000000000001},"last":1.3859999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":2.1320000000000001,"15min":1.756},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.29,"5min":54.938000000000002,"15min":54.938000000000002},"last":1.403},{"interface":"front","average":{"1min":1.361,"5min":1.665,"15min":1.4730000000000001},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8369999999999997,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.93799999999999994}]},{"osd":6,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.74,"15min":1.5129999999999999},"min":{"1min":0.495,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.1399999999999997,"5min":17.733000000000001,"15min":17.733000000000001},"last":1.331},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.99,"15min":1.7749999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":4.2409999999999997,"5min":28.797000000000001,"15min":28.797000000000001},"last":1.292}]},{"osd":7,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":2.113,"15min":1.7130000000000001},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":21.100000000000001,"5min":21.100000000000001,"15min":21.100000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.98,"15min":1.7150000000000001},"min":{"1min":0.62,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":5.726,"5min":22.751000000000001,"15min":22.751000000000001},"last":1.4350000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58412,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313428,"statfs":{"total":96636764160,"available":96576950272,"internally_reserved":0,"allocated":880640,"data_stored":371152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.3169999999999999,"15min":0.997},"min":{"1min":0.47099999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":36.408000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.165,"15min":0.97499999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":5.0190000000000001,"5min":5.0529999999999999,"15min":5.0529999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.236,"15min":1.0760000000000001},"min":{"1min":0.45300000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.226,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.095,"5min":1.222,"15min":1.1619999999999999},"min":{"1min":0.51600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":1.3959999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.2470000000000001,"15min":1.1359999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.7650000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":1.155},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2350000000000001,"15min":1.228},"min":{"1min":0.57399999999999995,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.1680000000000001,"5min":2.7650000000000001,"15min":5.0910000000000002},"last":1.526}]},{"osd":4,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.554,"15min":1.3540000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":5.0970000000000004,"5min":17.082000000000001,"15min":17.082000000000001},"last":1.256},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.5229999999999999,"15min":1.429},"min":{"1min":0.82899999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":5.6310000000000002},"last":1.49}]},{"osd":5,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.625,"15min":1.452},"min":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":5.351,"5min":5.351,"15min":5.351},"last":1.2210000000000001},{"interface":"front","average":{"1min":3.698,"5min":1.897,"15min":1.766},"min":{"1min":0.67900000000000005,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":38.832999999999998,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.593}]},{"osd":6,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.516,"15min":1.3779999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":5.1840000000000002,"5min":5.1840000000000002,"15min":5.1840000000000002},"last":1.619},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.857,"15min":1.639},"min":{"1min":1.024,"5min":0.747,"15min":0.747},"max":{"1min":5.3079999999999998,"5min":16.195,"15min":16.195},"last":1.5620000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":4.0190000000000001,"5min":2.149,"15min":1.7829999999999999},"min":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":24.146999999999998,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.681},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.724,"15min":1.534},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":5.1429999999999998,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.764}]}]},{"osd":2,"up_from":14,"seq":60129542247,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54260,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317580,"statfs":{"total":96636764160,"available":96581201920,"internally_reserved":0,"allocated":884736,"data_stored":372652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.014,"15min":1.0740000000000001},"min":{"1min":0.371,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.9199999999999999,"5min":6.101,"15min":6.101},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0600000000000001,"15min":1.1479999999999999},"min":{"1min":0.33700000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":4.0830000000000002,"15min":4.0830000000000002},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.2150000000000001,"15min":1.083},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":2.3929999999999998,"5min":10.807,"15min":10.807},"last":0.752},{"interface":"front","average":{"1min":1.282,"5min":1.1739999999999999,"15min":1.222},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2370000000000001,"5min":4.2370000000000001,"15min":4.2370000000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":2.6899999999999999,"5min":1.8340000000000001,"15min":1.407},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":25.891999999999999,"5min":25.891999999999999,"15min":25.891999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2529999999999999,"15min":1.1870000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1899999999999999,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.5900000000000001,"15min":1.577},"min":{"1min":0.66100000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":9.0679999999999996,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.706,"5min":1.6200000000000001,"15min":1.484},"min":{"1min":0.79400000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.2080000000000002,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":1.0860000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":2.2989999999999999,"5min":1.9330000000000001,"15min":1.631},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.8380000000000001,"5min":2.052,"15min":1.714},"min":{"1min":0.71699999999999997,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":8.5250000000000004,"5min":37.020000000000003,"15min":37.020000000000003},"last":1.248}]},{"osd":6,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.8340000000000001,"15min":1.7090000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.0129999999999999,"5min":23.762,"15min":23.762},"last":1.145},{"interface":"front","average":{"1min":1.7110000000000001,"5min":2.2170000000000001,"15min":1.734},"min":{"1min":0.85899999999999999,"5min":0.78800000000000003,"15min":0.73499999999999999},"max":{"1min":3.879,"5min":39.926000000000002,"15min":39.926000000000002},"last":0.879}]},{"osd":7,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":3.6459999999999999,"5min":2.4180000000000001,"15min":1.911},"min":{"1min":0.85999999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":19.986000000000001,"5min":52.767000000000003,"15min":52.767000000000003},"last":1.1799999999999999},{"interface":"front","average":{"1min":2.3399999999999999,"5min":1.9450000000000001,"15min":1.7170000000000001},"min":{"1min":0.88100000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":10.987,"5min":13.055999999999999,"15min":13.055999999999999},"last":0.91400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542247,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58424,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313416,"statfs":{"total":96636764160,"available":96576937984,"internally_reserved":0,"allocated":892928,"data_stored":373268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.99399999999999999,"15min":0.92300000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.9119999999999999,"5min":3.5939999999999999,"15min":3.5939999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":2.089,"5min":1.381,"15min":1.1819999999999999},"min":{"1min":0.48999999999999999,"5min":0.39800000000000002,"15min":0.35299999999999998},"max":{"1min":19.823,"5min":19.823,"15min":19.823},"last":1.278}]},{"osd":2,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.149,"15min":1.0549999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.679,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.266,"15min":1.133},"min":{"1min":0.44800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.6309999999999998,"5min":16.795999999999999,"15min":16.795999999999999},"last":1.2}]},{"osd":3,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.2729999999999999,"15min":1.2789999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.7169999999999996,"5min":4.7169999999999996,"15min":7.8659999999999997},"last":1.0289999999999999},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.4399999999999999,"15min":1.3380000000000001},"min":{"1min":0.52900000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":19.733000000000001,"5min":19.733000000000001,"15min":19.733000000000001},"last":1.762}]},{"osd":4,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.5449999999999999,"15min":1.448},"min":{"1min":0.70599999999999996,"5min":0.497,"15min":0.497},"max":{"1min":2.6840000000000002,"5min":7.3239999999999998,"15min":7.3239999999999998},"last":1.4490000000000001},{"interface":"front","average":{"1min":2.2639999999999998,"5min":1.9830000000000001,"15min":1.673},"min":{"1min":0.627,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":19.925999999999998,"5min":22.204000000000001,"15min":22.204000000000001},"last":1.5209999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.9970000000000001,"15min":1.7010000000000001},"min":{"1min":0.78600000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":20.012,"5min":20.012,"15min":20.012},"last":1.623},{"interface":"front","average":{"1min":1.419,"5min":1.909,"15min":1.601},"min":{"1min":0.80600000000000005,"5min":0.58799999999999997,"15min":0.499},"max":{"1min":2.6190000000000002,"5min":19.998999999999999,"15min":19.998999999999999},"last":0.995}]},{"osd":6,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.7370000000000001,"15min":1.6000000000000001},"min":{"1min":0.70799999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.6160000000000001,"5min":10.089,"15min":10.089},"last":1.3400000000000001},{"interface":"front","average":{"1min":2.6880000000000002,"5min":2.2469999999999999,"15min":1.806},"min":{"1min":0.67500000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":20.064,"5min":27.579999999999998,"15min":27.579999999999998},"last":1.6990000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":2.0569999999999999,"15min":1.7110000000000001},"min":{"1min":0.72999999999999998,"5min":0.505,"15min":0.505},"max":{"1min":6.9160000000000004,"5min":19.170000000000002,"15min":19.170000000000002},"last":1.129},{"interface":"front","average":{"1min":1.901,"5min":1.849,"15min":1.6950000000000001},"min":{"1min":0.76100000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":10.416,"5min":10.416,"15min":10.416},"last":1.3879999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58980,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312860,"statfs":{"total":96636764160,"available":96576368640,"internally_reserved":0,"allocated":1466368,"data_stored":961408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.4019999999999999,"15min":1.022},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5700000000000001,"5min":17.207000000000001,"15min":17.207000000000001},"last":1.198},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.4450000000000001,"15min":1.1639999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3720000000000001,"5min":35.137999999999998,"15min":35.137999999999998},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.464,"15min":1.153},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.1120000000000001,"5min":12.565,"15min":12.565},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.5089999999999999,"15min":1.3089999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.5820000000000001,"5min":35.183,"15min":35.183},"last":0.71399999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.232,"15min":1.1040000000000001},"min":{"1min":0.498,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.7759999999999998,"5min":8.9949999999999992,"15min":8.9949999999999992},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.9119999999999999,"15min":1.4850000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3980000000000001,"5min":25.372,"15min":25.372},"last":1.3089999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":2.0830000000000002,"15min":1.6259999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.5609999999999999,"5min":35.223999999999997,"15min":35.223999999999997},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.677,"15min":1.4359999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.3610000000000002,"5min":15.968999999999999,"15min":15.968999999999999},"last":0.96899999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.7470000000000001,"15min":1.5},"min":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.3450000000000002,"5min":9.1059999999999999,"15min":9.1059999999999999},"last":1.171},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.9339999999999999,"15min":1.615},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.5739999999999998,"5min":12.622999999999999,"15min":12.622999999999999},"last":1.458}]},{"osd":6,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":2.4369999999999998,"15min":1.7889999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":8.3569999999999993,"5min":36.390999999999998,"15min":36.390999999999998},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":2.5449999999999999,"15min":1.8500000000000001},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":8.3829999999999991,"5min":36.207000000000001,"15min":36.207000000000001},"last":0.997}]},{"osd":7,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":2.798,"5min":2.5019999999999998,"15min":1.883},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":24.556999999999999,"5min":35.636000000000003,"15min":35.636000000000003},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.7569999999999999,"15min":1.5489999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":3.4319999999999999,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":1.2390000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:05:48.149 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:05:48.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.407+0000 7fca4b000640 1 Processor -- start 2024-01-15T18:05:48.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.407+0000 7fca4b000640 1 -- start start 2024-01-15T18:05:48.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca4b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca441505d0 0x7fca440b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca4b000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca4415aa70 0x7fca44070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca4b000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fca440710c0 0x7fca440b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca4b000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca440b29d0 con 0x7fca4415aa70 2024-01-15T18:05:48.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca4b000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca440b2cb0 con 0x7fca440710c0 2024-01-15T18:05:48.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca4b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca440b2f90 con 0x7fca441505d0 2024-01-15T18:05:48.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca3be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca4415aa70 0x7fca44070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca3be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca4415aa70 0x7fca44070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53268/0 (socket says 172.21.15.2:53268) 2024-01-15T18:05:48.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.409+0000 7fca3be00640 1 -- 172.21.15.2:0/2761363853 learned_addr learned my addr 172.21.15.2:0/2761363853 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:48.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca49000640 1 --2- 172.21.15.2:0/2761363853 >> v2:172.21.15.137:3300/0 conn(0x7fca440710c0 0x7fca440b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca43e00640 1 --2- 172.21.15.2:0/2761363853 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca441505d0 0x7fca440b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca3be00640 1 -- 172.21.15.2:0/2761363853 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca441505d0 msgr2=0x7fca440b20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca3be00640 1 --2- 172.21.15.2:0/2761363853 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca441505d0 0x7fca440b20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca3be00640 1 -- 172.21.15.2:0/2761363853 >> v2:172.21.15.137:3300/0 conn(0x7fca440710c0 msgr2=0x7fca440b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca3be00640 1 --2- 172.21.15.2:0/2761363853 >> v2:172.21.15.137:3300/0 conn(0x7fca440710c0 0x7fca440b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.410+0000 7fca3be00640 1 -- 172.21.15.2:0/2761363853 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca44070380 con 0x7fca4415aa70 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.411+0000 7fca3be00640 1 --2- 172.21.15.2:0/2761363853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca4415aa70 0x7fca44070b80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fca4406d300 tx=0x7fca2c02deb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2124669cb5086408 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.411+0000 7fca43400640 1 -- 172.21.15.2:0/2761363853 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c02ec60 con 0x7fca4415aa70 2024-01-15T18:05:48.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.411+0000 7fca43400640 1 -- 172.21.15.2:0/2761363853 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c032070 con 0x7fca4415aa70 2024-01-15T18:05:48.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.411+0000 7fca43400640 1 -- 172.21.15.2:0/2761363853 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c038400 con 0x7fca4415aa70 2024-01-15T18:05:48.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 -- 172.21.15.2:0/2761363853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca4415aa70 msgr2=0x7fca44070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 --2- 172.21.15.2:0/2761363853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca4415aa70 0x7fca44070b80 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fca4406d300 tx=0x7fca2c02deb0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 -- 172.21.15.2:0/2761363853 shutdown_connections 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 --2- 172.21.15.2:0/2761363853 >> v2:172.21.15.137:3300/0 conn(0x7fca440710c0 0x7fca440b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 --2- 172.21.15.2:0/2761363853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca4415aa70 0x7fca44070b80 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 --2- 172.21.15.2:0/2761363853 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca441505d0 0x7fca440b20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 -- 172.21.15.2:0/2761363853 >> 172.21.15.2:0/2761363853 conn(0x7fca4406da20 msgr2=0x7fca4406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 -- 172.21.15.2:0/2761363853 shutdown_connections 2024-01-15T18:05:48.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.412+0000 7fca4b000640 1 -- 172.21.15.2:0/2761363853 wait complete. 2024-01-15T18:05:48.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.415+0000 7fca4b000640 1 Processor -- start 2024-01-15T18:05:48.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.415+0000 7fca4b000640 1 -- start start 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca4b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca440710c0 0x7fca44150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca4b000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 0x7fca441f2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca4b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca4415aa70 0x7fca44158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca4b000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca441f3620 con 0x7fca441505d0 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca4b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca441f3900 con 0x7fca440710c0 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca4b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca441f3be0 con 0x7fca4415aa70 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.416+0000 7fca3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 0x7fca441f2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 0x7fca441f2d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53274/0 (socket says 172.21.15.2:53274) 2024-01-15T18:05:48.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 -- 172.21.15.2:0/3982146872 learned_addr learned my addr 172.21.15.2:0/3982146872 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:48.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca49000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca4415aa70 0x7fca44158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca43e00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca440710c0 0x7fca44150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 -- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca4415aa70 msgr2=0x7fca44158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca4415aa70 0x7fca44158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 -- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca440710c0 msgr2=0x7fca44150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca440710c0 0x7fca44150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 -- 172.21.15.2:0/3982146872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2c00b760 con 0x7fca441505d0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca49000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca4415aa70 0x7fca44158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca43e00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca440710c0 0x7fca44150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.417+0000 7fca3be00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 0x7fca441f2d90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fca2c03a040 tx=0x7fca2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.418+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c038da0 con 0x7fca441505d0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.418+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c042410 con 0x7fca441505d0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.418+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c0427c0 con 0x7fca441505d0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.418+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca441f3ec0 con 0x7fca441505d0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.418+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca441f32d0 con 0x7fca441505d0 2024-01-15T18:05:48.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.419+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca441f43e0 con 0x7fca441505d0 2024-01-15T18:05:48.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.420+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fca2c0373b0 con 0x7fca441505d0 2024-01-15T18:05:48.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.421+0000 7fca40c00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca1c06bbf0 0x7fca1c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.421+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fca2c0aa0a0 con 0x7fca441505d0 2024-01-15T18:05:48.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.422+0000 7fca43e00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca1c06bbf0 0x7fca1c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.423+0000 7fca43e00640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca1c06bbf0 0x7fca1c06e0b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fca34006fd0 tx=0x7fca34008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.426+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca441f43e0 con 0x7fca441505d0 2024-01-15T18:05:48.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.624+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca441509b0 con 0x7fca1c06bbf0 2024-01-15T18:05:48.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.647+0000 7fca40c00640 1 -- 172.21.15.2:0/3982146872 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287001 (secure 0 0 0) 0x7fca34009e10 con 0x7fca1c06bbf0 2024-01-15T18:05:48.655 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:48.656 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.652+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca1c06bbf0 msgr2=0x7fca1c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.652+0000 7fca4b000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca1c06bbf0 0x7fca1c06e0b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fca34006fd0 tx=0x7fca34008040 comp rx=0 tx=0).stop 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.652+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 msgr2=0x7fca441f2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.652+0000 7fca4b000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 0x7fca441f2d90 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fca2c03a040 tx=0x7fca2c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 shutdown_connections 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca1c06bbf0 0x7fca1c06e0b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca4415aa70 0x7fca44158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca441505d0 0x7fca441f2d90 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 --2- 172.21.15.2:0/3982146872 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca440710c0 0x7fca44150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 >> 172.21.15.2:0/3982146872 conn(0x7fca4406da20 msgr2=0x7fca4406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 shutdown_connections 2024-01-15T18:05:48.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.653+0000 7fca4b000640 1 -- 172.21.15.2:0/3982146872 wait complete. 2024-01-15T18:05:48.678 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-01-15T18:05:47.598048+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":103,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":431000,"kb_used_data":8040,"kb_used_omap":12,"kb_used_meta":422835,"kb_avail":754543720,"statfs":{"total":773094113280,"available":772652769280,"internally_reserved":0,"allocated":8232960,"data_stored":4153876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":432983601},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.990460"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.678714+0000","last_change":"2024-01-15T18:05:43.675850+0000","last_active":"2024-01-15T18:05:43.678714+0000","last_peered":"2024-01-15T18:05:43.678714+0000","last_clean":"2024-01-15T18:05:43.678714+0000","last_became_active":"2024-01-15T18:05:43.675545+0000","last_became_peered":"2024-01-15T18:05:43.675545+0000","last_unstale":"2024-01-15T18:05:43.678714+0000","last_undegraded":"2024-01-15T18:05:43.678714+0000","last_fullsized":"2024-01-15T18:05:43.678714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:54:17.390435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674671+0000","last_change":"2024-01-15T18:05:43.671148+0000","last_active":"2024-01-15T18:05:43.674671+0000","last_peered":"2024-01-15T18:05:43.674671+0000","last_clean":"2024-01-15T18:05:43.674671+0000","last_became_active":"2024-01-15T18:05:43.669822+0000","last_became_peered":"2024-01-15T18:05:43.669822+0000","last_unstale":"2024-01-15T18:05:43.674671+0000","last_undegraded":"2024-01-15T18:05:43.674671+0000","last_fullsized":"2024-01-15T18:05:43.674671+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:28:37.233984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668840+0000","last_change":"2024-01-15T18:05:43.663373+0000","last_active":"2024-01-15T18:05:43.668840+0000","last_peered":"2024-01-15T18:05:43.668840+0000","last_clean":"2024-01-15T18:05:43.668840+0000","last_became_active":"2024-01-15T18:05:43.662253+0000","last_became_peered":"2024-01-15T18:05:43.662253+0000","last_unstale":"2024-01-15T18:05:43.668840+0000","last_undegraded":"2024-01-15T18:05:43.668840+0000","last_fullsized":"2024-01-15T18:05:43.668840+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:57:40.803431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679100+0000","last_change":"2024-01-15T18:05:43.676852+0000","last_active":"2024-01-15T18:05:43.679100+0000","last_peered":"2024-01-15T18:05:43.679100+0000","last_clean":"2024-01-15T18:05:43.679100+0000","last_became_active":"2024-01-15T18:05:43.676556+0000","last_became_peered":"2024-01-15T18:05:43.676556+0000","last_unstale":"2024-01-15T18:05:43.679100+0000","last_undegraded":"2024-01-15T18:05:43.679100+0000","last_fullsized":"2024-01-15T18:05:43.679100+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.998414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677284+0000","last_change":"2024-01-15T18:05:43.670491+0000","last_active":"2024-01-15T18:05:43.677284+0000","last_peered":"2024-01-15T18:05:43.677284+0000","last_clean":"2024-01-15T18:05:43.677284+0000","last_became_active":"2024-01-15T18:05:43.670177+0000","last_became_peered":"2024-01-15T18:05:43.670177+0000","last_unstale":"2024-01-15T18:05:43.677284+0000","last_undegraded":"2024-01-15T18:05:43.677284+0000","last_fullsized":"2024-01-15T18:05:43.677284+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:13:03.781810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677535+0000","last_change":"2024-01-15T18:05:43.672718+0000","last_active":"2024-01-15T18:05:43.677535+0000","last_peered":"2024-01-15T18:05:43.677535+0000","last_clean":"2024-01-15T18:05:43.677535+0000","last_became_active":"2024-01-15T18:05:43.671441+0000","last_became_peered":"2024-01-15T18:05:43.671441+0000","last_unstale":"2024-01-15T18:05:43.677535+0000","last_undegraded":"2024-01-15T18:05:43.677535+0000","last_fullsized":"2024-01-15T18:05:43.677535+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:49.267617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.663544+0000","last_change":"2024-01-15T18:05:43.661992+0000","last_active":"2024-01-15T18:05:43.663544+0000","last_peered":"2024-01-15T18:05:43.663544+0000","last_clean":"2024-01-15T18:05:43.663544+0000","last_became_active":"2024-01-15T18:05:43.657977+0000","last_became_peered":"2024-01-15T18:05:43.657977+0000","last_unstale":"2024-01-15T18:05:43.663544+0000","last_undegraded":"2024-01-15T18:05:43.663544+0000","last_fullsized":"2024-01-15T18:05:43.663544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:30:42.285026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680972+0000","last_change":"2024-01-15T18:05:43.677758+0000","last_active":"2024-01-15T18:05:43.680972+0000","last_peered":"2024-01-15T18:05:43.680972+0000","last_clean":"2024-01-15T18:05:43.680972+0000","last_became_active":"2024-01-15T18:05:43.677034+0000","last_became_peered":"2024-01-15T18:05:43.677034+0000","last_unstale":"2024-01-15T18:05:43.680972+0000","last_undegraded":"2024-01-15T18:05:43.680972+0000","last_fullsized":"2024-01-15T18:05:43.680972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:11.538253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680450+0000","last_change":"2024-01-15T18:05:43.674797+0000","last_active":"2024-01-15T18:05:43.680450+0000","last_peered":"2024-01-15T18:05:43.680450+0000","last_clean":"2024-01-15T18:05:43.680450+0000","last_became_active":"2024-01-15T18:05:43.673775+0000","last_became_peered":"2024-01-15T18:05:43.673775+0000","last_unstale":"2024-01-15T18:05:43.680450+0000","last_undegraded":"2024-01-15T18:05:43.680450+0000","last_fullsized":"2024-01-15T18:05:43.680450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:00:26.111871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.669040+0000","last_change":"2024-01-15T18:05:43.666742+0000","last_active":"2024-01-15T18:05:43.669040+0000","last_peered":"2024-01-15T18:05:43.669040+0000","last_clean":"2024-01-15T18:05:43.669040+0000","last_became_active":"2024-01-15T18:05:43.662605+0000","last_became_peered":"2024-01-15T18:05:43.662605+0000","last_unstale":"2024-01-15T18:05:43.669040+0000","last_undegraded":"2024-01-15T18:05:43.669040+0000","last_fullsized":"2024-01-15T18:05:43.669040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:20:24.155396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675215+0000","last_change":"2024-01-15T18:05:43.672860+0000","last_active":"2024-01-15T18:05:43.675215+0000","last_peered":"2024-01-15T18:05:43.675215+0000","last_clean":"2024-01-15T18:05:43.675215+0000","last_became_active":"2024-01-15T18:05:43.672561+0000","last_became_peered":"2024-01-15T18:05:43.672561+0000","last_unstale":"2024-01-15T18:05:43.675215+0000","last_undegraded":"2024-01-15T18:05:43.675215+0000","last_fullsized":"2024-01-15T18:05:43.675215+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:37.136328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674984+0000","last_change":"2024-01-15T18:05:43.665889+0000","last_active":"2024-01-15T18:05:43.674984+0000","last_peered":"2024-01-15T18:05:43.674984+0000","last_clean":"2024-01-15T18:05:43.674984+0000","last_became_active":"2024-01-15T18:05:43.665618+0000","last_became_peered":"2024-01-15T18:05:43.665618+0000","last_unstale":"2024-01-15T18:05:43.674984+0000","last_undegraded":"2024-01-15T18:05:43.674984+0000","last_fullsized":"2024-01-15T18:05:43.674984+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:29.813139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.672912+0000","last_change":"2024-01-15T18:05:43.671154+0000","last_active":"2024-01-15T18:05:43.672912+0000","last_peered":"2024-01-15T18:05:43.672912+0000","last_clean":"2024-01-15T18:05:43.672912+0000","last_became_active":"2024-01-15T18:05:43.670186+0000","last_became_peered":"2024-01-15T18:05:43.670186+0000","last_unstale":"2024-01-15T18:05:43.672912+0000","last_undegraded":"2024-01-15T18:05:43.672912+0000","last_fullsized":"2024-01-15T18:05:43.672912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:54.618059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680628+0000","last_change":"2024-01-15T18:05:43.675004+0000","last_active":"2024-01-15T18:05:43.680628+0000","last_peered":"2024-01-15T18:05:43.680628+0000","last_clean":"2024-01-15T18:05:43.680628+0000","last_became_active":"2024-01-15T18:05:43.674586+0000","last_became_peered":"2024-01-15T18:05:43.674586+0000","last_unstale":"2024-01-15T18:05:43.680628+0000","last_undegraded":"2024-01-15T18:05:43.680628+0000","last_fullsized":"2024-01-15T18:05:43.680628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:24:23.989128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668906+0000","last_change":"2024-01-15T18:05:43.667864+0000","last_active":"2024-01-15T18:05:43.668906+0000","last_peered":"2024-01-15T18:05:43.668906+0000","last_clean":"2024-01-15T18:05:43.668906+0000","last_became_active":"2024-01-15T18:05:43.667561+0000","last_became_peered":"2024-01-15T18:05:43.667561+0000","last_unstale":"2024-01-15T18:05:43.668906+0000","last_undegraded":"2024-01-15T18:05:43.668906+0000","last_fullsized":"2024-01-15T18:05:43.668906+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:32.434900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.683217+0000","last_change":"2024-01-15T18:05:43.677601+0000","last_active":"2024-01-15T18:05:43.683217+0000","last_peered":"2024-01-15T18:05:43.683217+0000","last_clean":"2024-01-15T18:05:43.683217+0000","last_became_active":"2024-01-15T18:05:43.677285+0000","last_became_peered":"2024-01-15T18:05:43.677285+0000","last_unstale":"2024-01-15T18:05:43.683217+0000","last_undegraded":"2024-01-15T18:05:43.683217+0000","last_fullsized":"2024-01-15T18:05:43.683217+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:32:57.612507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.681241+0000","last_change":"2024-01-15T18:05:43.675229+0000","last_active":"2024-01-15T18:05:43.681241+0000","last_peered":"2024-01-15T18:05:43.681241+0000","last_clean":"2024-01-15T18:05:43.681241+0000","last_became_active":"2024-01-15T18:05:43.670691+0000","last_became_peered":"2024-01-15T18:05:43.670691+0000","last_unstale":"2024-01-15T18:05:43.681241+0000","last_undegraded":"2024-01-15T18:05:43.681241+0000","last_fullsized":"2024-01-15T18:05:43.681241+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:16:26.052298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679024+0000","last_change":"2024-01-15T18:05:43.676489+0000","last_active":"2024-01-15T18:05:43.679024+0000","last_peered":"2024-01-15T18:05:43.679024+0000","last_clean":"2024-01-15T18:05:43.679024+0000","last_became_active":"2024-01-15T18:05:43.676199+0000","last_became_peered":"2024-01-15T18:05:43.676199+0000","last_unstale":"2024-01-15T18:05:43.679024+0000","last_undegraded":"2024-01-15T18:05:43.679024+0000","last_fullsized":"2024-01-15T18:05:43.679024+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:52.224215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680943+0000","last_change":"2024-01-15T18:05:43.678865+0000","last_active":"2024-01-15T18:05:43.680943+0000","last_peered":"2024-01-15T18:05:43.680943+0000","last_clean":"2024-01-15T18:05:43.680943+0000","last_became_active":"2024-01-15T18:05:43.678328+0000","last_became_peered":"2024-01-15T18:05:43.678328+0000","last_unstale":"2024-01-15T18:05:43.680943+0000","last_undegraded":"2024-01-15T18:05:43.680943+0000","last_fullsized":"2024-01-15T18:05:43.680943+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:15:17.016743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.659937+0000","last_change":"2024-01-15T18:05:43.656831+0000","last_active":"2024-01-15T18:05:43.659937+0000","last_peered":"2024-01-15T18:05:43.659937+0000","last_clean":"2024-01-15T18:05:43.659937+0000","last_became_active":"2024-01-15T18:05:43.656552+0000","last_became_peered":"2024-01-15T18:05:43.656552+0000","last_unstale":"2024-01-15T18:05:43.659937+0000","last_undegraded":"2024-01-15T18:05:43.659937+0000","last_fullsized":"2024-01-15T18:05:43.659937+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:48:59.746981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668945+0000","last_change":"2024-01-15T18:05:43.660514+0000","last_active":"2024-01-15T18:05:43.668945+0000","last_peered":"2024-01-15T18:05:43.668945+0000","last_clean":"2024-01-15T18:05:43.668945+0000","last_became_active":"2024-01-15T18:05:43.660188+0000","last_became_peered":"2024-01-15T18:05:43.660188+0000","last_unstale":"2024-01-15T18:05:43.668945+0000","last_undegraded":"2024-01-15T18:05:43.668945+0000","last_fullsized":"2024-01-15T18:05:43.668945+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:37:51.462560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668239+0000","last_change":"2024-01-15T18:05:43.661939+0000","last_active":"2024-01-15T18:05:43.668239+0000","last_peered":"2024-01-15T18:05:43.668239+0000","last_clean":"2024-01-15T18:05:43.668239+0000","last_became_active":"2024-01-15T18:05:43.661468+0000","last_became_peered":"2024-01-15T18:05:43.661468+0000","last_unstale":"2024-01-15T18:05:43.668239+0000","last_undegraded":"2024-01-15T18:05:43.668239+0000","last_fullsized":"2024-01-15T18:05:43.668239+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:00.184025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.670031+0000","last_change":"2024-01-15T18:05:43.664403+0000","last_active":"2024-01-15T18:05:43.670031+0000","last_peered":"2024-01-15T18:05:43.670031+0000","last_clean":"2024-01-15T18:05:43.670031+0000","last_became_active":"2024-01-15T18:05:43.663745+0000","last_became_peered":"2024-01-15T18:05:43.663745+0000","last_unstale":"2024-01-15T18:05:43.670031+0000","last_undegraded":"2024-01-15T18:05:43.670031+0000","last_fullsized":"2024-01-15T18:05:43.670031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:48:59.746981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679765+0000","last_change":"2024-01-15T18:05:43.677291+0000","last_active":"2024-01-15T18:05:43.679765+0000","last_peered":"2024-01-15T18:05:43.679765+0000","last_clean":"2024-01-15T18:05:43.679765+0000","last_became_active":"2024-01-15T18:05:43.677001+0000","last_became_peered":"2024-01-15T18:05:43.677001+0000","last_unstale":"2024-01-15T18:05:43.679765+0000","last_undegraded":"2024-01-15T18:05:43.679765+0000","last_fullsized":"2024-01-15T18:05:43.679765+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:28.191729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668332+0000","last_change":"2024-01-15T18:05:43.664395+0000","last_active":"2024-01-15T18:05:43.668332+0000","last_peered":"2024-01-15T18:05:43.668332+0000","last_clean":"2024-01-15T18:05:43.668332+0000","last_became_active":"2024-01-15T18:05:43.664036+0000","last_became_peered":"2024-01-15T18:05:43.664036+0000","last_unstale":"2024-01-15T18:05:43.668332+0000","last_undegraded":"2024-01-15T18:05:43.668332+0000","last_fullsized":"2024-01-15T18:05:43.668332+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:53:26.160525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.676598+0000","last_change":"2024-01-15T18:05:43.673812+0000","last_active":"2024-01-15T18:05:43.676598+0000","last_peered":"2024-01-15T18:05:43.676598+0000","last_clean":"2024-01-15T18:05:43.676598+0000","last_became_active":"2024-01-15T18:05:43.673506+0000","last_became_peered":"2024-01-15T18:05:43.673506+0000","last_unstale":"2024-01-15T18:05:43.676598+0000","last_undegraded":"2024-01-15T18:05:43.676598+0000","last_fullsized":"2024-01-15T18:05:43.676598+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:32.434900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.672872+0000","last_change":"2024-01-15T18:05:43.666518+0000","last_active":"2024-01-15T18:05:43.672872+0000","last_peered":"2024-01-15T18:05:43.672872+0000","last_clean":"2024-01-15T18:05:43.672872+0000","last_became_active":"2024-01-15T18:05:43.666127+0000","last_became_peered":"2024-01-15T18:05:43.666127+0000","last_unstale":"2024-01-15T18:05:43.672872+0000","last_undegraded":"2024-01-15T18:05:43.672872+0000","last_fullsized":"2024-01-15T18:05:43.672872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:41.284860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679029+0000","last_change":"2024-01-15T18:05:43.675917+0000","last_active":"2024-01-15T18:05:43.679029+0000","last_peered":"2024-01-15T18:05:43.679029+0000","last_clean":"2024-01-15T18:05:43.679029+0000","last_became_active":"2024-01-15T18:05:43.675641+0000","last_became_peered":"2024-01-15T18:05:43.675641+0000","last_unstale":"2024-01-15T18:05:43.679029+0000","last_undegraded":"2024-01-15T18:05:43.679029+0000","last_fullsized":"2024-01-15T18:05:43.679029+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:48:20.927398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.671868+0000","last_change":"2024-01-15T18:05:43.665148+0000","last_active":"2024-01-15T18:05:43.671868+0000","last_peered":"2024-01-15T18:05:43.671868+0000","last_clean":"2024-01-15T18:05:43.671868+0000","last_became_active":"2024-01-15T18:05:43.664873+0000","last_became_peered":"2024-01-15T18:05:43.664873+0000","last_unstale":"2024-01-15T18:05:43.671868+0000","last_undegraded":"2024-01-15T18:05:43.671868+0000","last_fullsized":"2024-01-15T18:05:43.671868+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:11:34.891192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.676641+0000","last_change":"2024-01-15T18:05:43.674794+0000","last_active":"2024-01-15T18:05:43.676641+0000","last_peered":"2024-01-15T18:05:43.676641+0000","last_clean":"2024-01-15T18:05:43.676641+0000","last_became_active":"2024-01-15T18:05:43.673468+0000","last_became_peered":"2024-01-15T18:05:43.673468+0000","last_unstale":"2024-01-15T18:05:43.676641+0000","last_undegraded":"2024-01-15T18:05:43.676641+0000","last_fullsized":"2024-01-15T18:05:43.676641+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:37.136328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.671103+0000","last_change":"2024-01-15T18:05:43.667583+0000","last_active":"2024-01-15T18:05:43.671103+0000","last_peered":"2024-01-15T18:05:43.671103+0000","last_clean":"2024-01-15T18:05:43.671103+0000","last_became_active":"2024-01-15T18:05:43.667317+0000","last_became_peered":"2024-01-15T18:05:43.667317+0000","last_unstale":"2024-01-15T18:05:43.671103+0000","last_undegraded":"2024-01-15T18:05:43.671103+0000","last_fullsized":"2024-01-15T18:05:43.671103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:02:40.878771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679354+0000","last_change":"2024-01-15T18:05:43.674945+0000","last_active":"2024-01-15T18:05:43.679354+0000","last_peered":"2024-01-15T18:05:43.679354+0000","last_clean":"2024-01-15T18:05:43.679354+0000","last_became_active":"2024-01-15T18:05:43.674367+0000","last_became_peered":"2024-01-15T18:05:43.674367+0000","last_unstale":"2024-01-15T18:05:43.679354+0000","last_undegraded":"2024-01-15T18:05:43.679354+0000","last_fullsized":"2024-01-15T18:05:43.679354+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:55.998414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680175+0000","last_change":"2024-01-15T18:05:43.678756+0000","last_active":"2024-01-15T18:05:43.680175+0000","last_peered":"2024-01-15T18:05:43.680175+0000","last_clean":"2024-01-15T18:05:43.680175+0000","last_became_active":"2024-01-15T18:05:43.678334+0000","last_became_peered":"2024-01-15T18:05:43.678334+0000","last_unstale":"2024-01-15T18:05:43.680175+0000","last_undegraded":"2024-01-15T18:05:43.680175+0000","last_fullsized":"2024-01-15T18:05:43.680175+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:13.141022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"72'4","reported_seq":15,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.687563+0000","last_change":"2024-01-15T18:05:42.647533+0000","last_active":"2024-01-15T18:05:43.687563+0000","last_peered":"2024-01-15T18:05:43.687563+0000","last_clean":"2024-01-15T18:05:43.687563+0000","last_became_active":"2024-01-15T18:05:42.646160+0000","last_became_peered":"2024-01-15T18:05:42.646160+0000","last_unstale":"2024-01-15T18:05:43.687563+0000","last_undegraded":"2024-01-15T18:05:43.687563+0000","last_fullsized":"2024-01-15T18:05:43.687563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:08:51.487836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680431+0000","last_change":"2024-01-15T18:05:43.674056+0000","last_active":"2024-01-15T18:05:43.680431+0000","last_peered":"2024-01-15T18:05:43.680431+0000","last_clean":"2024-01-15T18:05:43.680431+0000","last_became_active":"2024-01-15T18:05:43.670293+0000","last_became_peered":"2024-01-15T18:05:43.670293+0000","last_unstale":"2024-01-15T18:05:43.680431+0000","last_undegraded":"2024-01-15T18:05:43.680431+0000","last_fullsized":"2024-01-15T18:05:43.680431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:46.968578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.685599+0000","last_change":"2024-01-15T18:05:42.632765+0000","last_active":"2024-01-15T18:05:43.685599+0000","last_peered":"2024-01-15T18:05:43.685599+0000","last_clean":"2024-01-15T18:05:43.685599+0000","last_became_active":"2024-01-15T18:05:42.632474+0000","last_became_peered":"2024-01-15T18:05:42.632474+0000","last_unstale":"2024-01-15T18:05:43.685599+0000","last_undegraded":"2024-01-15T18:05:43.685599+0000","last_fullsized":"2024-01-15T18:05:43.685599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:21.746449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.682665+0000","last_change":"2024-01-15T18:05:43.678962+0000","last_active":"2024-01-15T18:05:43.682665+0000","last_peered":"2024-01-15T18:05:43.682665+0000","last_clean":"2024-01-15T18:05:43.682665+0000","last_became_active":"2024-01-15T18:05:43.678461+0000","last_became_peered":"2024-01-15T18:05:43.678461+0000","last_unstale":"2024-01-15T18:05:43.682665+0000","last_undegraded":"2024-01-15T18:05:43.682665+0000","last_fullsized":"2024-01-15T18:05:43.682665+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:12:55.296555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.686204+0000","last_change":"2024-01-15T18:05:42.644508+0000","last_active":"2024-01-15T18:05:43.686204+0000","last_peered":"2024-01-15T18:05:43.686204+0000","last_clean":"2024-01-15T18:05:43.686204+0000","last_became_active":"2024-01-15T18:05:42.644174+0000","last_became_peered":"2024-01-15T18:05:42.644174+0000","last_unstale":"2024-01-15T18:05:43.686204+0000","last_undegraded":"2024-01-15T18:05:43.686204+0000","last_fullsized":"2024-01-15T18:05:43.686204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:45.433793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675194+0000","last_change":"2024-01-15T18:05:43.670585+0000","last_active":"2024-01-15T18:05:43.675194+0000","last_peered":"2024-01-15T18:05:43.675194+0000","last_clean":"2024-01-15T18:05:43.675194+0000","last_became_active":"2024-01-15T18:05:43.669332+0000","last_became_peered":"2024-01-15T18:05:43.669332+0000","last_unstale":"2024-01-15T18:05:43.675194+0000","last_undegraded":"2024-01-15T18:05:43.675194+0000","last_fullsized":"2024-01-15T18:05:43.675194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:19.643686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688428+0000","last_change":"2024-01-15T18:05:42.648112+0000","last_active":"2024-01-15T18:05:43.688428+0000","last_peered":"2024-01-15T18:05:43.688428+0000","last_clean":"2024-01-15T18:05:43.688428+0000","last_became_active":"2024-01-15T18:05:42.647761+0000","last_became_peered":"2024-01-15T18:05:42.647761+0000","last_unstale":"2024-01-15T18:05:43.688428+0000","last_undegraded":"2024-01-15T18:05:43.688428+0000","last_fullsized":"2024-01-15T18:05:43.688428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:04:50.052265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.669342+0000","last_change":"2024-01-15T18:05:43.663667+0000","last_active":"2024-01-15T18:05:43.669342+0000","last_peered":"2024-01-15T18:05:43.669342+0000","last_clean":"2024-01-15T18:05:43.669342+0000","last_became_active":"2024-01-15T18:05:43.662601+0000","last_became_peered":"2024-01-15T18:05:43.662601+0000","last_unstale":"2024-01-15T18:05:43.669342+0000","last_undegraded":"2024-01-15T18:05:43.669342+0000","last_fullsized":"2024-01-15T18:05:43.669342+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:51:48.516632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.687488+0000","last_change":"2024-01-15T18:05:42.642739+0000","last_active":"2024-01-15T18:05:43.687488+0000","last_peered":"2024-01-15T18:05:43.687488+0000","last_clean":"2024-01-15T18:05:43.687488+0000","last_became_active":"2024-01-15T18:05:42.642345+0000","last_became_peered":"2024-01-15T18:05:42.642345+0000","last_unstale":"2024-01-15T18:05:43.687488+0000","last_undegraded":"2024-01-15T18:05:43.687488+0000","last_fullsized":"2024-01-15T18:05:43.687488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:42:54.080792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668966+0000","last_change":"2024-01-15T18:05:43.664935+0000","last_active":"2024-01-15T18:05:43.668966+0000","last_peered":"2024-01-15T18:05:43.668966+0000","last_clean":"2024-01-15T18:05:43.668966+0000","last_became_active":"2024-01-15T18:05:43.661915+0000","last_became_peered":"2024-01-15T18:05:43.661915+0000","last_unstale":"2024-01-15T18:05:43.668966+0000","last_undegraded":"2024-01-15T18:05:43.668966+0000","last_fullsized":"2024-01-15T18:05:43.668966+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:59:40.831307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688766+0000","last_change":"2024-01-15T18:05:42.641270+0000","last_active":"2024-01-15T18:05:43.688766+0000","last_peered":"2024-01-15T18:05:43.688766+0000","last_clean":"2024-01-15T18:05:43.688766+0000","last_became_active":"2024-01-15T18:05:42.639544+0000","last_became_peered":"2024-01-15T18:05:42.639544+0000","last_unstale":"2024-01-15T18:05:43.688766+0000","last_undegraded":"2024-01-15T18:05:43.688766+0000","last_fullsized":"2024-01-15T18:05:43.688766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:03:38.120276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677405+0000","last_change":"2024-01-15T18:05:43.670901+0000","last_active":"2024-01-15T18:05:43.677405+0000","last_peered":"2024-01-15T18:05:43.677405+0000","last_clean":"2024-01-15T18:05:43.677405+0000","last_became_active":"2024-01-15T18:05:43.670525+0000","last_became_peered":"2024-01-15T18:05:43.670525+0000","last_unstale":"2024-01-15T18:05:43.677405+0000","last_undegraded":"2024-01-15T18:05:43.677405+0000","last_fullsized":"2024-01-15T18:05:43.677405+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:51:16.047771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.637181+0000","last_change":"2024-01-15T18:05:42.631403+0000","last_active":"2024-01-15T18:05:43.637181+0000","last_peered":"2024-01-15T18:05:43.637181+0000","last_clean":"2024-01-15T18:05:43.637181+0000","last_became_active":"2024-01-15T18:05:42.629617+0000","last_became_peered":"2024-01-15T18:05:42.629617+0000","last_unstale":"2024-01-15T18:05:43.637181+0000","last_undegraded":"2024-01-15T18:05:43.637181+0000","last_fullsized":"2024-01-15T18:05:43.637181+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:27:19.291541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668887+0000","last_change":"2024-01-15T18:05:43.667281+0000","last_active":"2024-01-15T18:05:43.668887+0000","last_peered":"2024-01-15T18:05:43.668887+0000","last_clean":"2024-01-15T18:05:43.668887+0000","last_became_active":"2024-01-15T18:05:43.664866+0000","last_became_peered":"2024-01-15T18:05:43.664866+0000","last_unstale":"2024-01-15T18:05:43.668887+0000","last_undegraded":"2024-01-15T18:05:43.668887+0000","last_fullsized":"2024-01-15T18:05:43.668887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:28.191729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.656238+0000","last_change":"2024-01-15T18:05:42.638734+0000","last_active":"2024-01-15T18:05:43.656238+0000","last_peered":"2024-01-15T18:05:43.656238+0000","last_clean":"2024-01-15T18:05:43.656238+0000","last_became_active":"2024-01-15T18:05:42.638109+0000","last_became_peered":"2024-01-15T18:05:42.638109+0000","last_unstale":"2024-01-15T18:05:43.656238+0000","last_undegraded":"2024-01-15T18:05:43.656238+0000","last_fullsized":"2024-01-15T18:05:43.656238+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:55.437694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668645+0000","last_change":"2024-01-15T18:05:43.662916+0000","last_active":"2024-01-15T18:05:43.668645+0000","last_peered":"2024-01-15T18:05:43.668645+0000","last_clean":"2024-01-15T18:05:43.668645+0000","last_became_active":"2024-01-15T18:05:43.662392+0000","last_became_peered":"2024-01-15T18:05:43.662392+0000","last_unstale":"2024-01-15T18:05:43.668645+0000","last_undegraded":"2024-01-15T18:05:43.668645+0000","last_fullsized":"2024-01-15T18:05:43.668645+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:48:59.746981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.646999+0000","last_change":"2024-01-15T18:05:42.630224+0000","last_active":"2024-01-15T18:05:43.646999+0000","last_peered":"2024-01-15T18:05:43.646999+0000","last_clean":"2024-01-15T18:05:43.646999+0000","last_became_active":"2024-01-15T18:05:42.629918+0000","last_became_peered":"2024-01-15T18:05:42.629918+0000","last_unstale":"2024-01-15T18:05:43.646999+0000","last_undegraded":"2024-01-15T18:05:43.646999+0000","last_fullsized":"2024-01-15T18:05:43.646999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:50.961139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680062+0000","last_change":"2024-01-15T18:05:43.677656+0000","last_active":"2024-01-15T18:05:43.680062+0000","last_peered":"2024-01-15T18:05:43.680062+0000","last_clean":"2024-01-15T18:05:43.680062+0000","last_became_active":"2024-01-15T18:05:43.677375+0000","last_became_peered":"2024-01-15T18:05:43.677375+0000","last_unstale":"2024-01-15T18:05:43.680062+0000","last_undegraded":"2024-01-15T18:05:43.680062+0000","last_fullsized":"2024-01-15T18:05:43.680062+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:40.482537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.665905+0000","last_change":"2024-01-15T18:05:42.642561+0000","last_active":"2024-01-15T18:05:43.665905+0000","last_peered":"2024-01-15T18:05:43.665905+0000","last_clean":"2024-01-15T18:05:43.665905+0000","last_became_active":"2024-01-15T18:05:42.641744+0000","last_became_peered":"2024-01-15T18:05:42.641744+0000","last_unstale":"2024-01-15T18:05:43.665905+0000","last_undegraded":"2024-01-15T18:05:43.665905+0000","last_fullsized":"2024-01-15T18:05:43.665905+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:43:35.993670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.678650+0000","last_change":"2024-01-15T18:05:43.673134+0000","last_active":"2024-01-15T18:05:43.678650+0000","last_peered":"2024-01-15T18:05:43.678650+0000","last_clean":"2024-01-15T18:05:43.678650+0000","last_became_active":"2024-01-15T18:05:43.672347+0000","last_became_peered":"2024-01-15T18:05:43.672347+0000","last_unstale":"2024-01-15T18:05:43.678650+0000","last_undegraded":"2024-01-15T18:05:43.678650+0000","last_fullsized":"2024-01-15T18:05:43.678650+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:26:10.727244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"72'3","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688442+0000","last_change":"2024-01-15T18:05:42.629845+0000","last_active":"2024-01-15T18:05:43.688442+0000","last_peered":"2024-01-15T18:05:43.688442+0000","last_clean":"2024-01-15T18:05:43.688442+0000","last_became_active":"2024-01-15T18:05:42.629526+0000","last_became_peered":"2024-01-15T18:05:42.629526+0000","last_unstale":"2024-01-15T18:05:43.688442+0000","last_undegraded":"2024-01-15T18:05:43.688442+0000","last_fullsized":"2024-01-15T18:05:43.688442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:52.355190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.667131+0000","last_change":"2024-01-15T18:05:43.662932+0000","last_active":"2024-01-15T18:05:43.667131+0000","last_peered":"2024-01-15T18:05:43.667131+0000","last_clean":"2024-01-15T18:05:43.667131+0000","last_became_active":"2024-01-15T18:05:43.662646+0000","last_became_peered":"2024-01-15T18:05:43.662646+0000","last_unstale":"2024-01-15T18:05:43.667131+0000","last_undegraded":"2024-01-15T18:05:43.667131+0000","last_fullsized":"2024-01-15T18:05:43.667131+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:59:16.414665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"72'6","reported_seq":17,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.689531+0000","last_change":"2024-01-15T18:05:42.638256+0000","last_active":"2024-01-15T18:05:43.689531+0000","last_peered":"2024-01-15T18:05:43.689531+0000","last_clean":"2024-01-15T18:05:43.689531+0000","last_became_active":"2024-01-15T18:05:42.637935+0000","last_became_peered":"2024-01-15T18:05:42.637935+0000","last_unstale":"2024-01-15T18:05:43.689531+0000","last_undegraded":"2024-01-15T18:05:43.689531+0000","last_fullsized":"2024-01-15T18:05:43.689531+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:17:10.523858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677105+0000","last_change":"2024-01-15T18:05:43.673787+0000","last_active":"2024-01-15T18:05:43.677105+0000","last_peered":"2024-01-15T18:05:43.677105+0000","last_clean":"2024-01-15T18:05:43.677105+0000","last_became_active":"2024-01-15T18:05:43.673490+0000","last_became_peered":"2024-01-15T18:05:43.673490+0000","last_unstale":"2024-01-15T18:05:43.677105+0000","last_undegraded":"2024-01-15T18:05:43.677105+0000","last_fullsized":"2024-01-15T18:05:43.677105+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:32.434900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.685442+0000","last_change":"2024-01-15T18:05:42.641185+0000","last_active":"2024-01-15T18:05:43.685442+0000","last_peered":"2024-01-15T18:05:43.685442+0000","last_clean":"2024-01-15T18:05:43.685442+0000","last_became_active":"2024-01-15T18:05:42.640872+0000","last_became_peered":"2024-01-15T18:05:42.640872+0000","last_unstale":"2024-01-15T18:05:43.685442+0000","last_undegraded":"2024-01-15T18:05:43.685442+0000","last_fullsized":"2024-01-15T18:05:43.685442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:56:19.406839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.669334+0000","last_change":"2024-01-15T18:05:43.667185+0000","last_active":"2024-01-15T18:05:43.669334+0000","last_peered":"2024-01-15T18:05:43.669334+0000","last_clean":"2024-01-15T18:05:43.669334+0000","last_became_active":"2024-01-15T18:05:43.666749+0000","last_became_peered":"2024-01-15T18:05:43.666749+0000","last_unstale":"2024-01-15T18:05:43.669334+0000","last_undegraded":"2024-01-15T18:05:43.669334+0000","last_fullsized":"2024-01-15T18:05:43.669334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:28.191729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.666045+0000","last_change":"2024-01-15T18:05:42.641169+0000","last_active":"2024-01-15T18:05:43.666045+0000","last_peered":"2024-01-15T18:05:43.666045+0000","last_clean":"2024-01-15T18:05:43.666045+0000","last_became_active":"2024-01-15T18:05:42.640910+0000","last_became_peered":"2024-01-15T18:05:42.640910+0000","last_unstale":"2024-01-15T18:05:43.666045+0000","last_undegraded":"2024-01-15T18:05:43.666045+0000","last_fullsized":"2024-01-15T18:05:43.666045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:53.872641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674136+0000","last_change":"2024-01-15T18:05:43.672335+0000","last_active":"2024-01-15T18:05:43.674136+0000","last_peered":"2024-01-15T18:05:43.674136+0000","last_clean":"2024-01-15T18:05:43.674136+0000","last_became_active":"2024-01-15T18:05:43.671991+0000","last_became_peered":"2024-01-15T18:05:43.671991+0000","last_unstale":"2024-01-15T18:05:43.674136+0000","last_undegraded":"2024-01-15T18:05:43.674136+0000","last_fullsized":"2024-01-15T18:05:43.674136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:13.141022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.640679+0000","last_change":"2024-01-15T18:05:42.634255+0000","last_active":"2024-01-15T18:05:43.640679+0000","last_peered":"2024-01-15T18:05:43.640679+0000","last_clean":"2024-01-15T18:05:43.640679+0000","last_became_active":"2024-01-15T18:05:42.632600+0000","last_became_peered":"2024-01-15T18:05:42.632600+0000","last_unstale":"2024-01-15T18:05:43.640679+0000","last_undegraded":"2024-01-15T18:05:43.640679+0000","last_fullsized":"2024-01-15T18:05:43.640679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:27:21.031022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.677921+0000","last_change":"2024-01-15T18:05:43.673024+0000","last_active":"2024-01-15T18:05:43.677921+0000","last_peered":"2024-01-15T18:05:43.677921+0000","last_clean":"2024-01-15T18:05:43.677921+0000","last_became_active":"2024-01-15T18:05:43.669764+0000","last_became_peered":"2024-01-15T18:05:43.669764+0000","last_unstale":"2024-01-15T18:05:43.677921+0000","last_undegraded":"2024-01-15T18:05:43.677921+0000","last_fullsized":"2024-01-15T18:05:43.677921+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:45:52.668007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"72'8","reported_seq":19,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.689856+0000","last_change":"2024-01-15T18:05:42.641745+0000","last_active":"2024-01-15T18:05:43.689856+0000","last_peered":"2024-01-15T18:05:43.689856+0000","last_clean":"2024-01-15T18:05:43.689856+0000","last_became_active":"2024-01-15T18:05:42.641369+0000","last_became_peered":"2024-01-15T18:05:42.641369+0000","last_unstale":"2024-01-15T18:05:43.689856+0000","last_undegraded":"2024-01-15T18:05:43.689856+0000","last_fullsized":"2024-01-15T18:05:43.689856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:55:42.282640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.667593+0000","last_change":"2024-01-15T18:05:43.659383+0000","last_active":"2024-01-15T18:05:43.667593+0000","last_peered":"2024-01-15T18:05:43.667593+0000","last_clean":"2024-01-15T18:05:43.667593+0000","last_became_active":"2024-01-15T18:05:43.658770+0000","last_became_peered":"2024-01-15T18:05:43.658770+0000","last_unstale":"2024-01-15T18:05:43.667593+0000","last_undegraded":"2024-01-15T18:05:43.667593+0000","last_fullsized":"2024-01-15T18:05:43.667593+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:27:15.409561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688172+0000","last_change":"2024-01-15T18:05:42.647173+0000","last_active":"2024-01-15T18:05:43.688172+0000","last_peered":"2024-01-15T18:05:43.688172+0000","last_clean":"2024-01-15T18:05:43.688172+0000","last_became_active":"2024-01-15T18:05:42.646897+0000","last_became_peered":"2024-01-15T18:05:42.646897+0000","last_unstale":"2024-01-15T18:05:43.688172+0000","last_undegraded":"2024-01-15T18:05:43.688172+0000","last_fullsized":"2024-01-15T18:05:43.688172+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:13:50.534577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675206+0000","last_change":"2024-01-15T18:05:43.671557+0000","last_active":"2024-01-15T18:05:43.675206+0000","last_peered":"2024-01-15T18:05:43.675206+0000","last_clean":"2024-01-15T18:05:43.675206+0000","last_became_active":"2024-01-15T18:05:43.671181+0000","last_became_peered":"2024-01-15T18:05:43.671181+0000","last_unstale":"2024-01-15T18:05:43.675206+0000","last_undegraded":"2024-01-15T18:05:43.675206+0000","last_fullsized":"2024-01-15T18:05:43.675206+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:08.605428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.686282+0000","last_change":"2024-01-15T18:05:42.648070+0000","last_active":"2024-01-15T18:05:43.686282+0000","last_peered":"2024-01-15T18:05:43.686282+0000","last_clean":"2024-01-15T18:05:43.686282+0000","last_became_active":"2024-01-15T18:05:42.647154+0000","last_became_peered":"2024-01-15T18:05:42.647154+0000","last_unstale":"2024-01-15T18:05:43.686282+0000","last_undegraded":"2024-01-15T18:05:43.686282+0000","last_fullsized":"2024-01-15T18:05:43.686282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:39.583721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.675004+0000","last_change":"2024-01-15T18:05:43.667405+0000","last_active":"2024-01-15T18:05:43.675004+0000","last_peered":"2024-01-15T18:05:43.675004+0000","last_clean":"2024-01-15T18:05:43.675004+0000","last_became_active":"2024-01-15T18:05:43.665200+0000","last_became_peered":"2024-01-15T18:05:43.665200+0000","last_unstale":"2024-01-15T18:05:43.675004+0000","last_undegraded":"2024-01-15T18:05:43.675004+0000","last_fullsized":"2024-01-15T18:05:43.675004+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:16:02.770737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"72'1","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.689883+0000","last_change":"2024-01-15T18:05:42.650648+0000","last_active":"2024-01-15T18:05:43.689883+0000","last_peered":"2024-01-15T18:05:43.689883+0000","last_clean":"2024-01-15T18:05:43.689883+0000","last_became_active":"2024-01-15T18:05:42.649790+0000","last_became_peered":"2024-01-15T18:05:42.649790+0000","last_unstale":"2024-01-15T18:05:43.689883+0000","last_undegraded":"2024-01-15T18:05:43.689883+0000","last_fullsized":"2024-01-15T18:05:43.689883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:14:04.306129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.681688+0000","last_change":"2024-01-15T18:05:43.677178+0000","last_active":"2024-01-15T18:05:43.681688+0000","last_peered":"2024-01-15T18:05:43.681688+0000","last_clean":"2024-01-15T18:05:43.681688+0000","last_became_active":"2024-01-15T18:05:43.674372+0000","last_became_peered":"2024-01-15T18:05:43.674372+0000","last_unstale":"2024-01-15T18:05:43.681688+0000","last_undegraded":"2024-01-15T18:05:43.681688+0000","last_fullsized":"2024-01-15T18:05:43.681688+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:34.551767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.659854+0000","last_change":"2024-01-15T18:05:42.648607+0000","last_active":"2024-01-15T18:05:43.659854+0000","last_peered":"2024-01-15T18:05:43.659854+0000","last_clean":"2024-01-15T18:05:43.659854+0000","last_became_active":"2024-01-15T18:05:42.648153+0000","last_became_peered":"2024-01-15T18:05:42.648153+0000","last_unstale":"2024-01-15T18:05:43.659854+0000","last_undegraded":"2024-01-15T18:05:43.659854+0000","last_fullsized":"2024-01-15T18:05:43.659854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:05.055793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.670898+0000","last_change":"2024-01-15T18:05:43.666345+0000","last_active":"2024-01-15T18:05:43.670898+0000","last_peered":"2024-01-15T18:05:43.670898+0000","last_clean":"2024-01-15T18:05:43.670898+0000","last_became_active":"2024-01-15T18:05:43.666032+0000","last_became_peered":"2024-01-15T18:05:43.666032+0000","last_unstale":"2024-01-15T18:05:43.670898+0000","last_undegraded":"2024-01-15T18:05:43.670898+0000","last_fullsized":"2024-01-15T18:05:43.670898+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:43:38.909893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.666122+0000","last_change":"2024-01-15T18:05:42.640152+0000","last_active":"2024-01-15T18:05:43.666122+0000","last_peered":"2024-01-15T18:05:43.666122+0000","last_clean":"2024-01-15T18:05:43.666122+0000","last_became_active":"2024-01-15T18:05:42.639183+0000","last_became_peered":"2024-01-15T18:05:42.639183+0000","last_unstale":"2024-01-15T18:05:43.666122+0000","last_undegraded":"2024-01-15T18:05:43.666122+0000","last_fullsized":"2024-01-15T18:05:43.666122+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:18:26.930742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.679057+0000","last_change":"2024-01-15T18:05:43.677096+0000","last_active":"2024-01-15T18:05:43.679057+0000","last_peered":"2024-01-15T18:05:43.679057+0000","last_clean":"2024-01-15T18:05:43.679057+0000","last_became_active":"2024-01-15T18:05:43.676730+0000","last_became_peered":"2024-01-15T18:05:43.676730+0000","last_unstale":"2024-01-15T18:05:43.679057+0000","last_undegraded":"2024-01-15T18:05:43.679057+0000","last_fullsized":"2024-01-15T18:05:43.679057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:53.547860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.641071+0000","last_change":"2024-01-15T18:05:42.638308+0000","last_active":"2024-01-15T18:05:43.641071+0000","last_peered":"2024-01-15T18:05:43.641071+0000","last_clean":"2024-01-15T18:05:43.641071+0000","last_became_active":"2024-01-15T18:05:42.637580+0000","last_became_peered":"2024-01-15T18:05:42.637580+0000","last_unstale":"2024-01-15T18:05:43.641071+0000","last_undegraded":"2024-01-15T18:05:43.641071+0000","last_fullsized":"2024-01-15T18:05:43.641071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:53.603664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.673677+0000","last_change":"2024-01-15T18:05:43.669038+0000","last_active":"2024-01-15T18:05:43.673677+0000","last_peered":"2024-01-15T18:05:43.673677+0000","last_clean":"2024-01-15T18:05:43.673677+0000","last_became_active":"2024-01-15T18:05:43.668736+0000","last_became_peered":"2024-01-15T18:05:43.668736+0000","last_unstale":"2024-01-15T18:05:43.673677+0000","last_undegraded":"2024-01-15T18:05:43.673677+0000","last_fullsized":"2024-01-15T18:05:43.673677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:41:54.034157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.650694+0000","last_change":"2024-01-15T18:05:42.636058+0000","last_active":"2024-01-15T18:05:43.650694+0000","last_peered":"2024-01-15T18:05:43.650694+0000","last_clean":"2024-01-15T18:05:43.650694+0000","last_became_active":"2024-01-15T18:05:42.634995+0000","last_became_peered":"2024-01-15T18:05:42.634995+0000","last_unstale":"2024-01-15T18:05:43.650694+0000","last_undegraded":"2024-01-15T18:05:43.650694+0000","last_fullsized":"2024-01-15T18:05:43.650694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:18.629291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.673727+0000","last_change":"2024-01-15T18:05:43.671119+0000","last_active":"2024-01-15T18:05:43.673727+0000","last_peered":"2024-01-15T18:05:43.673727+0000","last_clean":"2024-01-15T18:05:43.673727+0000","last_became_active":"2024-01-15T18:05:43.669879+0000","last_became_peered":"2024-01-15T18:05:43.669879+0000","last_unstale":"2024-01-15T18:05:43.673727+0000","last_undegraded":"2024-01-15T18:05:43.673727+0000","last_fullsized":"2024-01-15T18:05:43.673727+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:37.136328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.687897+0000","last_change":"2024-01-15T18:05:42.645829+0000","last_active":"2024-01-15T18:05:43.687897+0000","last_peered":"2024-01-15T18:05:43.687897+0000","last_clean":"2024-01-15T18:05:43.687897+0000","last_became_active":"2024-01-15T18:05:42.645555+0000","last_became_peered":"2024-01-15T18:05:42.645555+0000","last_unstale":"2024-01-15T18:05:43.687897+0000","last_undegraded":"2024-01-15T18:05:43.687897+0000","last_fullsized":"2024-01-15T18:05:43.687897+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:41:28.798744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668193+0000","last_change":"2024-01-15T18:05:43.662334+0000","last_active":"2024-01-15T18:05:43.668193+0000","last_peered":"2024-01-15T18:05:43.668193+0000","last_clean":"2024-01-15T18:05:43.668193+0000","last_became_active":"2024-01-15T18:05:43.661858+0000","last_became_peered":"2024-01-15T18:05:43.661858+0000","last_unstale":"2024-01-15T18:05:43.668193+0000","last_undegraded":"2024-01-15T18:05:43.668193+0000","last_fullsized":"2024-01-15T18:05:43.668193+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:51:19.643686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.646443+0000","last_change":"2024-01-15T18:05:42.632413+0000","last_active":"2024-01-15T18:05:43.646443+0000","last_peered":"2024-01-15T18:05:43.646443+0000","last_clean":"2024-01-15T18:05:43.646443+0000","last_became_active":"2024-01-15T18:05:42.632094+0000","last_became_peered":"2024-01-15T18:05:42.632094+0000","last_unstale":"2024-01-15T18:05:43.646443+0000","last_undegraded":"2024-01-15T18:05:43.646443+0000","last_fullsized":"2024-01-15T18:05:43.646443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:12.126627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680213+0000","last_change":"2024-01-15T18:05:43.678180+0000","last_active":"2024-01-15T18:05:43.680213+0000","last_peered":"2024-01-15T18:05:43.680213+0000","last_clean":"2024-01-15T18:05:43.680213+0000","last_became_active":"2024-01-15T18:05:43.677877+0000","last_became_peered":"2024-01-15T18:05:43.677877+0000","last_unstale":"2024-01-15T18:05:43.680213+0000","last_undegraded":"2024-01-15T18:05:43.680213+0000","last_fullsized":"2024-01-15T18:05:43.680213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:01:39.810030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.651746+0000","last_change":"2024-01-15T18:05:42.632767+0000","last_active":"2024-01-15T18:05:43.651746+0000","last_peered":"2024-01-15T18:05:43.651746+0000","last_clean":"2024-01-15T18:05:43.651746+0000","last_became_active":"2024-01-15T18:05:42.632423+0000","last_became_peered":"2024-01-15T18:05:42.632423+0000","last_unstale":"2024-01-15T18:05:43.651746+0000","last_undegraded":"2024-01-15T18:05:43.651746+0000","last_fullsized":"2024-01-15T18:05:43.651746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:00:54.056977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.671429+0000","last_change":"2024-01-15T18:05:43.668793+0000","last_active":"2024-01-15T18:05:43.671429+0000","last_peered":"2024-01-15T18:05:43.671429+0000","last_clean":"2024-01-15T18:05:43.671429+0000","last_became_active":"2024-01-15T18:05:43.668352+0000","last_became_peered":"2024-01-15T18:05:43.668352+0000","last_unstale":"2024-01-15T18:05:43.671429+0000","last_undegraded":"2024-01-15T18:05:43.671429+0000","last_fullsized":"2024-01-15T18:05:43.671429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:22.760844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.652502+0000","last_change":"2024-01-15T18:05:42.626386+0000","last_active":"2024-01-15T18:05:43.652502+0000","last_peered":"2024-01-15T18:05:43.652502+0000","last_clean":"2024-01-15T18:05:43.652502+0000","last_became_active":"2024-01-15T18:05:42.626113+0000","last_became_peered":"2024-01-15T18:05:42.626113+0000","last_unstale":"2024-01-15T18:05:43.652502+0000","last_undegraded":"2024-01-15T18:05:43.652502+0000","last_fullsized":"2024-01-15T18:05:43.652502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:05.055793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668188+0000","last_change":"2024-01-15T18:05:43.664055+0000","last_active":"2024-01-15T18:05:43.668188+0000","last_peered":"2024-01-15T18:05:43.668188+0000","last_clean":"2024-01-15T18:05:43.668188+0000","last_became_active":"2024-01-15T18:05:43.663752+0000","last_became_peered":"2024-01-15T18:05:43.663752+0000","last_unstale":"2024-01-15T18:05:43.668188+0000","last_undegraded":"2024-01-15T18:05:43.668188+0000","last_fullsized":"2024-01-15T18:05:43.668188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:09:38.345675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.656060+0000","last_change":"2024-01-15T18:05:42.633521+0000","last_active":"2024-01-15T18:05:43.656060+0000","last_peered":"2024-01-15T18:05:43.656060+0000","last_clean":"2024-01-15T18:05:43.656060+0000","last_became_active":"2024-01-15T18:05:42.633251+0000","last_became_peered":"2024-01-15T18:05:42.633251+0000","last_unstale":"2024-01-15T18:05:43.656060+0000","last_undegraded":"2024-01-15T18:05:43.656060+0000","last_fullsized":"2024-01-15T18:05:43.656060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:25:21.796601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680743+0000","last_change":"2024-01-15T18:05:43.678142+0000","last_active":"2024-01-15T18:05:43.680743+0000","last_peered":"2024-01-15T18:05:43.680743+0000","last_clean":"2024-01-15T18:05:43.680743+0000","last_became_active":"2024-01-15T18:05:43.677866+0000","last_became_peered":"2024-01-15T18:05:43.677866+0000","last_unstale":"2024-01-15T18:05:43.680743+0000","last_undegraded":"2024-01-15T18:05:43.680743+0000","last_fullsized":"2024-01-15T18:05:43.680743+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:52.224215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.637189+0000","last_change":"2024-01-15T18:05:42.643050+0000","last_active":"2024-01-15T18:05:43.637189+0000","last_peered":"2024-01-15T18:05:43.637189+0000","last_clean":"2024-01-15T18:05:43.637189+0000","last_became_active":"2024-01-15T18:05:42.642447+0000","last_became_peered":"2024-01-15T18:05:42.642447+0000","last_unstale":"2024-01-15T18:05:43.637189+0000","last_undegraded":"2024-01-15T18:05:43.637189+0000","last_fullsized":"2024-01-15T18:05:43.637189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:27.496701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.674805+0000","last_change":"2024-01-15T18:05:43.663963+0000","last_active":"2024-01-15T18:05:43.674805+0000","last_peered":"2024-01-15T18:05:43.674805+0000","last_clean":"2024-01-15T18:05:43.674805+0000","last_became_active":"2024-01-15T18:05:43.663679+0000","last_became_peered":"2024-01-15T18:05:43.663679+0000","last_unstale":"2024-01-15T18:05:43.674805+0000","last_undegraded":"2024-01-15T18:05:43.674805+0000","last_fullsized":"2024-01-15T18:05:43.674805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:13:51.548972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.643653+0000","last_change":"2024-01-15T18:05:42.641523+0000","last_active":"2024-01-15T18:05:43.643653+0000","last_peered":"2024-01-15T18:05:43.643653+0000","last_clean":"2024-01-15T18:05:43.643653+0000","last_became_active":"2024-01-15T18:05:42.641199+0000","last_became_peered":"2024-01-15T18:05:42.641199+0000","last_unstale":"2024-01-15T18:05:43.643653+0000","last_undegraded":"2024-01-15T18:05:43.643653+0000","last_fullsized":"2024-01-15T18:05:43.643653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:44:29.964431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.680735+0000","last_change":"2024-01-15T18:05:43.677001+0000","last_active":"2024-01-15T18:05:43.680735+0000","last_peered":"2024-01-15T18:05:43.680735+0000","last_clean":"2024-01-15T18:05:43.680735+0000","last_became_active":"2024-01-15T18:05:43.675396+0000","last_became_peered":"2024-01-15T18:05:43.675396+0000","last_unstale":"2024-01-15T18:05:43.680735+0000","last_undegraded":"2024-01-15T18:05:43.680735+0000","last_fullsized":"2024-01-15T18:05:43.680735+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:16:02.770737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.668123+0000","last_change":"2024-01-15T18:05:42.648523+0000","last_active":"2024-01-15T18:05:43.668123+0000","last_peered":"2024-01-15T18:05:43.668123+0000","last_clean":"2024-01-15T18:05:43.668123+0000","last_became_active":"2024-01-15T18:05:42.648246+0000","last_became_peered":"2024-01-15T18:05:42.648246+0000","last_unstale":"2024-01-15T18:05:43.668123+0000","last_undegraded":"2024-01-15T18:05:43.668123+0000","last_fullsized":"2024-01-15T18:05:43.668123+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:48:22.607805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.673096+0000","last_change":"2024-01-15T18:05:43.669680+0000","last_active":"2024-01-15T18:05:43.673096+0000","last_peered":"2024-01-15T18:05:43.673096+0000","last_clean":"2024-01-15T18:05:43.673096+0000","last_became_active":"2024-01-15T18:05:43.669373+0000","last_became_peered":"2024-01-15T18:05:43.669373+0000","last_unstale":"2024-01-15T18:05:43.673096+0000","last_undegraded":"2024-01-15T18:05:43.673096+0000","last_fullsized":"2024-01-15T18:05:43.673096+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:42.604373+0000","last_clean_scrub_stamp":"2024-01-15T18:05:42.604373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:19:08.869684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"72'2","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-15T18:05:43.688290+0000","last_change":"2024-01-15T18:05:42.647945+0000","last_active":"2024-01-15T18:05:43.688290+0000","last_peered":"2024-01-15T18:05:43.688290+0000","last_clean":"2024-01-15T18:05:43.688290+0000","last_became_active":"2024-01-15T18:05:42.647645+0000","last_became_peered":"2024-01-15T18:05:42.647645+0000","last_unstale":"2024-01-15T18:05:43.688290+0000","last_undegraded":"2024-01-15T18:05:43.688290+0000","last_fullsized":"2024-01-15T18:05:43.688290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:05:41.589978+0000","last_clean_scrub_stamp":"2024-01-15T18:05:41.589978+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:05:57.002432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":211,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-15T18:05:42.642168+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:05:42.642168+0000","last_peered":"2024-01-15T18:05:42.642168+0000","last_clean":"2024-01-15T18:05:42.642168+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:05:42.642168+0000","last_undegraded":"2024-01-15T18:05:42.642168+0000","last_fullsized":"2024-01-15T18:05:42.642168+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542246,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50668,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321172,"statfs":{"total":96636764160,"available":96584880128,"internally_reserved":0,"allocated":1474560,"data_stored":962024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.7030000000000001,"15min":1.304},"min":{"1min":0.84599999999999997,"5min":0.622,"15min":0.53600000000000003},"max":{"1min":9.1379999999999999,"5min":14.114000000000001,"15min":14.114000000000001},"last":1.407},{"interface":"front","average":{"1min":1.73,"5min":1.601,"15min":1.4790000000000001},"min":{"1min":1.0569999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":7.0289999999999999,"5min":12.496,"15min":12.496},"last":1.1679999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.4590000000000001,"15min":1.224},"min":{"1min":0.67300000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":4.9210000000000003,"5min":14.497,"15min":14.497},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.748,"5min":1.6399999999999999,"15min":1.292},"min":{"1min":0.70599999999999996,"5min":0.69899999999999995,"15min":0.59499999999999997},"max":{"1min":8.5980000000000008,"5min":12.565,"15min":12.565},"last":1.0289999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":2.3690000000000002,"5min":1.9139999999999999,"15min":1.3879999999999999},"min":{"1min":0.88200000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":12.709,"5min":13.801,"15min":13.801},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.96,"5min":1.5049999999999999,"15min":1.468},"min":{"1min":0.91300000000000003,"5min":0.61499999999999999,"15min":0.58699999999999997},"max":{"1min":7.0810000000000004,"5min":7.0810000000000004,"15min":7.5940000000000003},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.758,"5min":1.6439999999999999,"15min":1.446},"min":{"1min":0.73099999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":5.6280000000000001,"5min":14.622,"15min":14.622},"last":1.079},{"interface":"front","average":{"1min":2.2149999999999999,"5min":1.8520000000000001,"15min":1.502},"min":{"1min":0.76000000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":16.088999999999999,"5min":16.088999999999999,"15min":16.088999999999999},"last":1.5609999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.899,"5min":1.5369999999999999,"15min":1.579},"min":{"1min":0.85899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":8.3870000000000005,"5min":8.3870000000000005,"15min":8.3870000000000005},"last":1.452},{"interface":"front","average":{"1min":1.948,"5min":1.6899999999999999,"15min":1.488},"min":{"1min":1.04,"5min":0.438,"15min":0.438},"max":{"1min":7.2489999999999997,"5min":14.711,"15min":14.711},"last":1.234}]},{"osd":5,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7250000000000001,"5min":1.7589999999999999,"15min":1.599},"min":{"1min":0.67000000000000004,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":7.6079999999999997,"5min":12.518000000000001,"15min":12.518000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.9470000000000001,"5min":1.607,"15min":1.653},"min":{"1min":0.69699999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.1219999999999999,"5min":7.1219999999999999,"15min":7.798},"last":1.5269999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:04:44 2024","interfaces":[{"interface":"back","average":{"1min":2.339,"5min":3.3199999999999998,"15min":2.2730000000000001},"min":{"1min":1.2689999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":8.7620000000000005,"5min":53.664999999999999,"15min":53.664999999999999},"last":1.5800000000000001},{"interface":"front","average":{"1min":2.4140000000000001,"5min":1.8580000000000001,"15min":1.5980000000000001},"min":{"1min":1.028,"5min":0.44,"15min":0.44},"max":{"1min":14.220000000000001,"5min":14.804,"15min":14.804},"last":1.355}]}]},{"osd":6,"up_from":14,"seq":60129542246,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50088,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321752,"statfs":{"total":96636764160,"available":96585474048,"internally_reserved":0,"allocated":880640,"data_stored":371132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2549999999999999,"15min":1.139},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":5.1189999999999998,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.7,"5min":1.611,"15min":1.2869999999999999},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":9.548,"5min":21.565000000000001,"15min":21.565000000000001},"last":1.327}]},{"osd":1,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.425,"15min":1.294},"min":{"1min":0.53300000000000003,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":6.343,"5min":6.343,"15min":6.343},"last":1.052},{"interface":"front","average":{"1min":1.7010000000000001,"5min":1.639,"15min":1.423},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":9.6389999999999993,"5min":13.984999999999999,"15min":13.984999999999999},"last":0.83499999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":1.7929999999999999,"15min":1.4430000000000001},"min":{"1min":0.79800000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":7.0030000000000001,"5min":34.670000000000002,"15min":34.670000000000002},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.494,"15min":1.3520000000000001},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":6.585,"5min":6.585,"15min":6.585},"last":0.93200000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.1739999999999999,"5min":1.7969999999999999,"15min":1.5089999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":16.855,"5min":16.855,"15min":16.855},"last":1.448},{"interface":"front","average":{"1min":1.7370000000000001,"5min":2.0779999999999998,"15min":1.6240000000000001},"min":{"1min":0.628,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":8.0139999999999993,"5min":34.725000000000001,"15min":34.725000000000001},"last":1.008}]},{"osd":4,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":1.597,"15min":1.4219999999999999},"min":{"1min":0.56299999999999994,"5min":0.52200000000000002,"15min":0.38400000000000001},"max":{"1min":6.5220000000000002,"5min":8.5489999999999995,"15min":8.5489999999999995},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.8420000000000001,"5min":1.583,"15min":1.5109999999999999},"min":{"1min":0.63600000000000001,"5min":0.56599999999999995,"15min":0.501},"max":{"1min":10.118,"5min":10.118,"15min":10.118},"last":1.5669999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.242,"5min":2.1269999999999998,"15min":1.6100000000000001},"min":{"1min":0.71199999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":7.9500000000000002,"5min":34.798000000000002,"15min":34.798000000000002},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.96,"5min":1.673,"15min":1.5580000000000001},"min":{"1min":0.66000000000000003,"5min":0.59699999999999998,"15min":0.47899999999999998},"max":{"1min":10.077,"5min":10.077,"15min":10.077},"last":1.4219999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:05:43 2024","interfaces":[{"interface":"back","average":{"1min":2.0099999999999998,"5min":1.7130000000000001,"15min":1.6759999999999999},"min":{"1min":0.78900000000000003,"5min":0.72699999999999998,"15min":0.64000000000000001},"max":{"1min":10.302,"5min":10.302,"15min":10.302},"last":1.5129999999999999},{"interface":"front","average":{"1min":2.113,"5min":2.2869999999999999,"15min":1.8520000000000001},"min":{"1min":0.84599999999999997,"5min":0.61799999999999999,"15min":0.52600000000000002},"max":{"1min":8.0519999999999996,"5min":34.762999999999998,"15min":34.762999999999998},"last":1.1899999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542248,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50080,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321760,"statfs":{"total":96636764160,"available":96585482240,"internally_reserved":0,"allocated":876544,"data_stored":371130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.786,"5min":1.7070000000000001,"15min":1.329},"min":{"1min":0.77500000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":26.077999999999999,"5min":26.077999999999999,"15min":26.077999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.6100000000000001,"15min":1.393},"min":{"1min":0.875,"5min":0.71099999999999997,"15min":0.52400000000000002},"max":{"1min":3.2330000000000001,"5min":14.711,"15min":14.711},"last":1.472}]},{"osd":1,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.6659999999999999,"5min":1.6040000000000001,"15min":1.3129999999999999},"min":{"1min":0.81200000000000006,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":26,"5min":26,"15min":26},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.413,"5min":1.4970000000000001,"15min":1.2549999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":3.9239999999999999,"5min":12.619,"15min":12.619},"last":0.61099999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":3.0819999999999999,"5min":1.7869999999999999,"15min":1.4850000000000001},"min":{"1min":0.73299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":26.234999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.538,"5min":1.575,"15min":1.4570000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.59099999999999997},"max":{"1min":3.4990000000000001,"5min":10.327999999999999,"15min":10.327999999999999},"last":1.091}]},{"osd":3,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.7090000000000001,"15min":1.4079999999999999},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":6.3259999999999996,"5min":12.722,"15min":12.722},"last":0.92700000000000005},{"interface":"front","average":{"1min":3.2770000000000001,"5min":2.0249999999999999,"15min":1.601},"min":{"1min":0.92500000000000004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":26.134,"5min":26.134,"15min":26.134},"last":1.2250000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.7190000000000001,"15min":1.4550000000000001},"min":{"1min":0.88100000000000001,"5min":0.60599999999999998,"15min":0.55900000000000005},"max":{"1min":5.0570000000000004,"5min":25.776,"15min":25.776},"last":0.98599999999999999},{"interface":"front","average":{"1min":2.617,"5min":1.944,"15min":1.6479999999999999},"min":{"1min":0.999,"5min":0.63800000000000001,"15min":0.47799999999999998},"max":{"1min":16.414000000000001,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.385}]},{"osd":6,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.8600000000000001,"15min":1.5269999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":4.7039999999999997,"5min":12.769,"15min":12.769},"last":1.026},{"interface":"front","average":{"1min":2.8279999999999998,"5min":2.4039999999999999,"15min":1.7909999999999999},"min":{"1min":0.49199999999999999,"5min":0.435,"15min":0.435},"max":{"1min":26.186,"5min":36.087000000000003,"15min":36.087000000000003},"last":1.51}]},{"osd":7,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.774,"5min":2.1139999999999999,"15min":1.798},"min":{"1min":0.90200000000000002,"5min":0.745,"15min":0.68000000000000005},"max":{"1min":3.339,"5min":13.706,"15min":13.706},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.621,"5min":2.1669999999999998,"15min":1.756},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":4.7489999999999997,"5min":21.728000000000002,"15min":21.728000000000002},"last":1.266}]}]},{"osd":4,"up_from":14,"seq":60129542247,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50088,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321752,"statfs":{"total":96636764160,"available":96585474048,"internally_reserved":0,"allocated":876544,"data_stored":371110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":1.3939999999999999,"15min":1.3080000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6599999999999999,"5min":18.673999999999999,"15min":21.643000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.325,"5min":1.4850000000000001,"15min":1.242},"min":{"1min":0.46200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":8.9109999999999996,"5min":11.196,"15min":11.196},"last":1.0409999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.415,"15min":1.258},"min":{"1min":0.41499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":5.3499999999999996,"5min":12.738,"15min":12.738},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.431,"15min":1.405},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":10.411,"5min":15.096,"15min":21.440000000000001},"last":0.85199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.595,"15min":1.3120000000000001},"min":{"1min":0.66500000000000004,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":6.6929999999999996,"5min":12.974,"15min":12.974},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.498,"15min":1.47},"min":{"1min":0.52500000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":7.9169999999999998,"5min":16.393000000000001,"15min":22.350999999999999},"last":1.468}]},{"osd":3,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.653,"15min":1.488},"min":{"1min":0.56699999999999995,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":3.9369999999999998,"5min":15.093,"15min":15.093},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.863,"15min":1.5309999999999999},"min":{"1min":0.71699999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":5.4039999999999999,"5min":19.486000000000001,"15min":19.486000000000001},"last":1.3859999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":2.1320000000000001,"15min":1.756},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.29,"5min":54.938000000000002,"15min":54.938000000000002},"last":1.403},{"interface":"front","average":{"1min":1.361,"5min":1.665,"15min":1.4730000000000001},"min":{"1min":0.42399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.8369999999999997,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.93799999999999994}]},{"osd":6,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.74,"15min":1.5129999999999999},"min":{"1min":0.495,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.1399999999999997,"5min":17.733000000000001,"15min":17.733000000000001},"last":1.331},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.99,"15min":1.7749999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":4.2409999999999997,"5min":28.797000000000001,"15min":28.797000000000001},"last":1.292}]},{"osd":7,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":2.113,"15min":1.7130000000000001},"min":{"1min":0.46999999999999997,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":21.100000000000001,"5min":21.100000000000001,"15min":21.100000000000001},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.98,"15min":1.7150000000000001},"min":{"1min":0.62,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":5.726,"5min":22.751000000000001,"15min":22.751000000000001},"last":1.4350000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542248,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58412,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313428,"statfs":{"total":96636764160,"available":96576950272,"internally_reserved":0,"allocated":880640,"data_stored":371152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58915272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":3.0539999999999998,"5min":1.3169999999999999,"15min":0.997},"min":{"1min":0.47099999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":36.408000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.165,"15min":0.97499999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":5.0190000000000001,"5min":5.0529999999999999,"15min":5.0529999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.236,"15min":1.0760000000000001},"min":{"1min":0.45300000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.226,"5min":8.1359999999999992,"15min":8.1359999999999992},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.095,"5min":1.222,"15min":1.1619999999999999},"min":{"1min":0.51600000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":2.1339999999999999,"5min":7.2169999999999996,"15min":7.2169999999999996},"last":1.3959999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.2470000000000001,"15min":1.1359999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.7650000000000001,"5min":3.1699999999999999,"15min":3.1699999999999999},"last":1.155},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2350000000000001,"15min":1.228},"min":{"1min":0.57399999999999995,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.1680000000000001,"5min":2.7650000000000001,"15min":5.0910000000000002},"last":1.526}]},{"osd":4,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":1.554,"15min":1.3540000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":5.0970000000000004,"5min":17.082000000000001,"15min":17.082000000000001},"last":1.256},{"interface":"front","average":{"1min":1.8109999999999999,"5min":1.5229999999999999,"15min":1.429},"min":{"1min":0.82899999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":5.6310000000000002},"last":1.49}]},{"osd":5,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8280000000000001,"5min":1.625,"15min":1.452},"min":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.65000000000000002},"max":{"1min":5.351,"5min":5.351,"15min":5.351},"last":1.2210000000000001},{"interface":"front","average":{"1min":3.698,"5min":1.897,"15min":1.766},"min":{"1min":0.67900000000000005,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":38.832999999999998,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.593}]},{"osd":6,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.516,"15min":1.3779999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":5.1840000000000002,"5min":5.1840000000000002,"15min":5.1840000000000002},"last":1.619},{"interface":"front","average":{"1min":2.0600000000000001,"5min":1.857,"15min":1.639},"min":{"1min":1.024,"5min":0.747,"15min":0.747},"max":{"1min":5.3079999999999998,"5min":16.195,"15min":16.195},"last":1.5620000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":4.0190000000000001,"5min":2.149,"15min":1.7829999999999999},"min":{"1min":0.76000000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":24.146999999999998,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.681},{"interface":"front","average":{"1min":2.0800000000000001,"5min":1.724,"15min":1.534},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":5.1429999999999998,"5min":9.0389999999999997,"15min":9.0389999999999997},"last":1.764}]}]},{"osd":2,"up_from":14,"seq":60129542247,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54260,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317580,"statfs":{"total":96636764160,"available":96581201920,"internally_reserved":0,"allocated":884736,"data_stored":372652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.014,"15min":1.0740000000000001},"min":{"1min":0.371,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":3.9199999999999999,"5min":6.101,"15min":6.101},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0600000000000001,"15min":1.1479999999999999},"min":{"1min":0.33700000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.0510000000000002,"5min":4.0830000000000002,"15min":4.0830000000000002},"last":0.51700000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":1.2150000000000001,"15min":1.083},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":2.3929999999999998,"5min":10.807,"15min":10.807},"last":0.752},{"interface":"front","average":{"1min":1.282,"5min":1.1739999999999999,"15min":1.222},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2370000000000001,"5min":4.2370000000000001,"15min":4.2370000000000001},"last":0.78100000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":2.6899999999999999,"5min":1.8340000000000001,"15min":1.407},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":25.891999999999999,"5min":25.891999999999999,"15min":25.891999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.2529999999999999,"15min":1.1870000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":3.1899999999999999,"5min":7.2599999999999998,"15min":7.2599999999999998},"last":0.57599999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":1.5900000000000001,"15min":1.577},"min":{"1min":0.66100000000000003,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":9.0679999999999996,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.706,"5min":1.6200000000000001,"15min":1.484},"min":{"1min":0.79400000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":3.2080000000000002,"5min":6.4290000000000003,"15min":6.4290000000000003},"last":1.0860000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":2.2989999999999999,"5min":1.9330000000000001,"15min":1.631},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":11.071999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.8380000000000001,"5min":2.052,"15min":1.714},"min":{"1min":0.71699999999999997,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":8.5250000000000004,"5min":37.020000000000003,"15min":37.020000000000003},"last":1.248}]},{"osd":6,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.8340000000000001,"15min":1.7090000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":3.0129999999999999,"5min":23.762,"15min":23.762},"last":1.145},{"interface":"front","average":{"1min":1.7110000000000001,"5min":2.2170000000000001,"15min":1.734},"min":{"1min":0.85899999999999999,"5min":0.78800000000000003,"15min":0.73499999999999999},"max":{"1min":3.879,"5min":39.926000000000002,"15min":39.926000000000002},"last":0.879}]},{"osd":7,"last update":"Mon Jan 15 18:05:35 2024","interfaces":[{"interface":"back","average":{"1min":3.6459999999999999,"5min":2.4180000000000001,"15min":1.911},"min":{"1min":0.85999999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":19.986000000000001,"5min":52.767000000000003,"15min":52.767000000000003},"last":1.1799999999999999},{"interface":"front","average":{"1min":2.3399999999999999,"5min":1.9450000000000001,"15min":1.7170000000000001},"min":{"1min":0.88100000000000001,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":10.987,"5min":13.055999999999999,"15min":13.055999999999999},"last":0.91400000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542247,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58424,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313416,"statfs":{"total":96636764160,"available":96576937984,"internally_reserved":0,"allocated":892928,"data_stored":373268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.99399999999999999,"15min":0.92300000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.9119999999999999,"5min":3.5939999999999999,"15min":3.5939999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":2.089,"5min":1.381,"15min":1.1819999999999999},"min":{"1min":0.48999999999999999,"5min":0.39800000000000002,"15min":0.35299999999999998},"max":{"1min":19.823,"5min":19.823,"15min":19.823},"last":1.278}]},{"osd":2,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":1.149,"15min":1.0549999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.679,"5min":7.6680000000000001,"15min":7.6680000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.266,"15min":1.133},"min":{"1min":0.44800000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.6309999999999998,"5min":16.795999999999999,"15min":16.795999999999999},"last":1.2}]},{"osd":3,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.2729999999999999,"15min":1.2789999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":4.7169999999999996,"5min":4.7169999999999996,"15min":7.8659999999999997},"last":1.0289999999999999},{"interface":"front","average":{"1min":2.1419999999999999,"5min":1.4399999999999999,"15min":1.3380000000000001},"min":{"1min":0.52900000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":19.733000000000001,"5min":19.733000000000001,"15min":19.733000000000001},"last":1.762}]},{"osd":4,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.5449999999999999,"15min":1.448},"min":{"1min":0.70599999999999996,"5min":0.497,"15min":0.497},"max":{"1min":2.6840000000000002,"5min":7.3239999999999998,"15min":7.3239999999999998},"last":1.4490000000000001},{"interface":"front","average":{"1min":2.2639999999999998,"5min":1.9830000000000001,"15min":1.673},"min":{"1min":0.627,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":19.925999999999998,"5min":22.204000000000001,"15min":22.204000000000001},"last":1.5209999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.9970000000000001,"15min":1.7010000000000001},"min":{"1min":0.78600000000000003,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":20.012,"5min":20.012,"15min":20.012},"last":1.623},{"interface":"front","average":{"1min":1.419,"5min":1.909,"15min":1.601},"min":{"1min":0.80600000000000005,"5min":0.58799999999999997,"15min":0.499},"max":{"1min":2.6190000000000002,"5min":19.998999999999999,"15min":19.998999999999999},"last":0.995}]},{"osd":6,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.7370000000000001,"15min":1.6000000000000001},"min":{"1min":0.70799999999999996,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.6160000000000001,"5min":10.089,"15min":10.089},"last":1.3400000000000001},{"interface":"front","average":{"1min":2.6880000000000002,"5min":2.2469999999999999,"15min":1.806},"min":{"1min":0.67500000000000004,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":20.064,"5min":27.579999999999998,"15min":27.579999999999998},"last":1.6990000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:05:42 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":2.0569999999999999,"15min":1.7110000000000001},"min":{"1min":0.72999999999999998,"5min":0.505,"15min":0.505},"max":{"1min":6.9160000000000004,"5min":19.170000000000002,"15min":19.170000000000002},"last":1.129},{"interface":"front","average":{"1min":1.901,"5min":1.849,"15min":1.6950000000000001},"min":{"1min":0.76100000000000001,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":10.416,"5min":10.416,"15min":10.416},"last":1.3879999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58980,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312860,"statfs":{"total":96636764160,"available":96576368640,"internally_reserved":0,"allocated":1466368,"data_stored":961408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":1.4019999999999999,"15min":1.022},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5700000000000001,"5min":17.207000000000001,"15min":17.207000000000001},"last":1.198},{"interface":"front","average":{"1min":0.80200000000000005,"5min":1.4450000000000001,"15min":1.1639999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.3720000000000001,"5min":35.137999999999998,"15min":35.137999999999998},"last":0.58699999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.464,"15min":1.153},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.1120000000000001,"5min":12.565,"15min":12.565},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.5089999999999999,"15min":1.3089999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.5820000000000001,"5min":35.183,"15min":35.183},"last":0.71399999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":1.232,"15min":1.1040000000000001},"min":{"1min":0.498,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":3.7759999999999998,"5min":8.9949999999999992,"15min":8.9949999999999992},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.9119999999999999,"15min":1.4850000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":2.3980000000000001,"5min":25.372,"15min":25.372},"last":1.3089999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":2.0830000000000002,"15min":1.6259999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.5609999999999999,"5min":35.223999999999997,"15min":35.223999999999997},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.677,"15min":1.4359999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.3610000000000002,"5min":15.968999999999999,"15min":15.968999999999999},"last":0.96899999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.7470000000000001,"15min":1.5},"min":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":2.3450000000000002,"5min":9.1059999999999999,"15min":9.1059999999999999},"last":1.171},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.9339999999999999,"15min":1.615},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.5739999999999998,"5min":12.622999999999999,"15min":12.622999999999999},"last":1.458}]},{"osd":6,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":1.9199999999999999,"5min":2.4369999999999998,"15min":1.7889999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":8.3569999999999993,"5min":36.390999999999998,"15min":36.390999999999998},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":2.5449999999999999,"15min":1.8500000000000001},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":8.3829999999999991,"5min":36.207000000000001,"15min":36.207000000000001},"last":0.997}]},{"osd":7,"last update":"Mon Jan 15 18:05:40 2024","interfaces":[{"interface":"back","average":{"1min":2.798,"5min":2.5019999999999998,"15min":1.883},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":24.556999999999999,"5min":35.636000000000003,"15min":35.636000000000003},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.7569999999999999,"15min":1.5489999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":3.4319999999999999,"5min":9.0749999999999993,"15min":9.0749999999999993},"last":1.2390000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:05:48.682 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:05:48.682 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:05:48.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.952+0000 7fd663e00640 1 Processor -- start 2024-01-15T18:05:48.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.953+0000 7fd663e00640 1 -- start start 2024-01-15T18:05:48.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd663e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd66415e1d0 0x7fd6640c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd663e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6640c9750 con 0x7fd6640c7ef0 2024-01-15T18:05:48.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd663e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6640c99b0 con 0x7fd6640709d0 2024-01-15T18:05:48.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd663e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66415e680 con 0x7fd66415e1d0 2024-01-15T18:05:48.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd662a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd6640c7ef0 0x7fd6640c82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39010/0 (socket says 172.21.15.2:39010) 2024-01-15T18:05:48.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd662a00640 1 -- 172.21.15.2:0/3574892270 learned_addr learned my addr 172.21.15.2:0/3574892270 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:48.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.954+0000 7fd659e00640 1 --2- 172.21.15.2:0/3574892270 >> v2:172.21.15.2:3300/0 conn(0x7fd66415e1d0 0x7fd6640c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.955+0000 7fd662000640 1 -- 172.21.15.2:0/3574892270 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937742090 0 0) 0x7fd6640c9750 con 0x7fd6640c7ef0 2024-01-15T18:05:48.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.955+0000 7fd662000640 1 -- 172.21.15.2:0/3574892270 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd644003520 con 0x7fd6640c7ef0 2024-01-15T18:05:48.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.955+0000 7fd659e00640 1 -- 172.21.15.2:0/3574892270 >> v1:172.21.15.137:6789/0 conn(0x7fd6640709d0 legacy=0x7fd664070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.955+0000 7fd659e00640 1 -- 172.21.15.2:0/3574892270 >> v1:172.21.15.2:6789/0 conn(0x7fd6640c7ef0 legacy=0x7fd6640c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.955+0000 7fd659e00640 1 -- 172.21.15.2:0/3574892270 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6640c9c90 con 0x7fd66415e1d0 2024-01-15T18:05:48.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.957+0000 7fd659e00640 1 --2- 172.21.15.2:0/3574892270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd66415e1d0 0x7fd6640c9210 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0097a0 tx=0x7fd64c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcab5f977bb0cac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.957+0000 7fd662000640 1 -- 172.21.15.2:0/3574892270 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c030610 con 0x7fd66415e1d0 2024-01-15T18:05:48.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.957+0000 7fd662000640 1 -- 172.21.15.2:0/3574892270 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd64c030770 con 0x7fd66415e1d0 2024-01-15T18:05:48.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.957+0000 7fd662000640 1 -- 172.21.15.2:0/3574892270 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c030ae0 con 0x7fd66415e1d0 2024-01-15T18:05:48.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 -- 172.21.15.2:0/3574892270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd66415e1d0 msgr2=0x7fd6640c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 --2- 172.21.15.2:0/3574892270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd66415e1d0 0x7fd6640c9210 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fd64c0097a0 tx=0x7fd64c02b960 comp rx=0 tx=0).stop 2024-01-15T18:05:48.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 -- 172.21.15.2:0/3574892270 shutdown_connections 2024-01-15T18:05:48.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 --2- 172.21.15.2:0/3574892270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd66415e1d0 0x7fd6640c9210 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 -- 172.21.15.2:0/3574892270 >> 172.21.15.2:0/3574892270 conn(0x7fd6640cebc0 msgr2=0x7fd6640cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:48.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 -- 172.21.15.2:0/3574892270 shutdown_connections 2024-01-15T18:05:48.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.958+0000 7fd663e00640 1 -- 172.21.15.2:0/3574892270 wait complete. 2024-01-15T18:05:48.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.960+0000 7fd663e00640 1 Processor -- start 2024-01-15T18:05:48.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.960+0000 7fd663e00640 1 -- start start 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd663e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 0x7fd6640b4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd663e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6640c7ef0 0x7fd6640b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd663e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66415e1d0 0x7fd66409c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd663e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6640b47d0 con 0x7fd6640709d0 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd663e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66409cee0 con 0x7fd66415e1d0 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd663e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66409cc00 con 0x7fd6640c7ef0 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd662a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 0x7fd6640b4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd662a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 0x7fd6640b4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53290/0 (socket says 172.21.15.2:53290) 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd662a00640 1 -- 172.21.15.2:0/2617928068 learned_addr learned my addr 172.21.15.2:0/2617928068 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd662a00640 1 -- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6640c7ef0 msgr2=0x7fd6640b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.961+0000 7fd662a00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6640c7ef0 0x7fd6640b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd662a00640 1 -- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66415e1d0 msgr2=0x7fd66409c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd662a00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66415e1d0 0x7fd66409c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:48.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd662a00640 1 -- 172.21.15.2:0/2617928068 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64c009400 con 0x7fd6640709d0 2024-01-15T18:05:48.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd662a00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 0x7fd6640b4290 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c4d0 tx=0x7fd65400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654005ce0 con 0x7fd6640709d0 2024-01-15T18:05:48.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd654005e40 con 0x7fd6640709d0 2024-01-15T18:05:48.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654007270 con 0x7fd6640709d0 2024-01-15T18:05:48.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6640b1a70 con 0x7fd6640709d0 2024-01-15T18:05:48.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.962+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6640b2030 con 0x7fd6640709d0 2024-01-15T18:05:48.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.964+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6640b2250 con 0x7fd6640709d0 2024-01-15T18:05:48.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.965+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd6540073d0 con 0x7fd6640709d0 2024-01-15T18:05:48.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.966+0000 7fd65be00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd63c06bbc0 0x7fd63c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:48.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.966+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fd6540879c0 con 0x7fd6640709d0 2024-01-15T18:05:48.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.966+0000 7fd659e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd63c06bbc0 0x7fd63c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:48.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.967+0000 7fd659e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd63c06bbc0 0x7fd63c06e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd64c036000 tx=0x7fd64c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:48.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:48.970+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd654054330 con 0x7fd6640709d0 2024-01-15T18:05:49.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.232+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd664154800 con 0x7fd6640709d0 2024-01-15T18:05:49.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.237+0000 7fd65be00640 1 -- 172.21.15.2:0/2617928068 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd654057950 con 0x7fd6640709d0 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:05:49.242 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T18:05:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.240+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd63c06bbc0 msgr2=0x7fd63c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.240+0000 7fd663e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd63c06bbc0 0x7fd63c06e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fd64c036000 tx=0x7fd64c030000 comp rx=0 tx=0).stop 2024-01-15T18:05:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 msgr2=0x7fd6640b4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 0x7fd6640b4290 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fd65400c4d0 tx=0x7fd65400c990 comp rx=0 tx=0).stop 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 shutdown_connections 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd63c06bbc0 0x7fd63c06e080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd66415e1d0 0x7fd66409c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6640c7ef0 0x7fd6640b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 --2- 172.21.15.2:0/2617928068 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6640709d0 0x7fd6640b4290 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 >> 172.21.15.2:0/2617928068 conn(0x7fd6640cebc0 msgr2=0x7fd6641540f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 shutdown_connections 2024-01-15T18:05:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.241+0000 7fd663e00640 1 -- 172.21.15.2:0/2617928068 wait complete. 2024-01-15T18:05:49.262 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:05:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.481+0000 7f90e8c00640 1 Processor -- start 2024-01-15T18:05:49.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.481+0000 7f90e8c00640 1 -- start start 2024-01-15T18:05:49.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e8c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f90e415d750 0x7f90e406cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e8c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f90e40ad960 0x7f90e415e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e8c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e40b2e70 con 0x7f90e406d050 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e8c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e406c430 con 0x7f90e40ad960 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e8c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e40b30c0 con 0x7f90e415d750 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90dbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f90e406d050 0x7f90e415db30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39022/0 (socket says 172.21.15.2:39022) 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90dbe00640 1 -- 172.21.15.2:0/1194862664 learned_addr learned my addr 172.21.15.2:0/1194862664 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e3e00640 1 --2- 172.21.15.2:0/1194862664 >> v2:172.21.15.2:3300/0 conn(0x7f90e40ad960 0x7f90e415e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.483+0000 7f90e3400640 1 --2- 172.21.15.2:0/1194862664 >> v2:172.21.15.137:3300/0 conn(0x7f90e415d750 0x7f90e406cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e2a00640 1 -- 172.21.15.2:0/1194862664 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941656915 0 0) 0x7f90e40b2e70 con 0x7f90e406d050 2024-01-15T18:05:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e2a00640 1 -- 172.21.15.2:0/1194862664 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90c4003520 con 0x7f90e406d050 2024-01-15T18:05:49.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e3e00640 1 -- 172.21.15.2:0/1194862664 >> v2:172.21.15.137:3300/0 conn(0x7f90e415d750 msgr2=0x7f90e406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e3e00640 1 --2- 172.21.15.2:0/1194862664 >> v2:172.21.15.137:3300/0 conn(0x7f90e415d750 0x7f90e406cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e3e00640 1 -- 172.21.15.2:0/1194862664 >> v1:172.21.15.2:6789/0 conn(0x7f90e406d050 legacy=0x7f90e415db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e3e00640 1 -- 172.21.15.2:0/1194862664 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e40b33a0 con 0x7f90e40ad960 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.484+0000 7f90e3400640 1 --2- 172.21.15.2:0/1194862664 >> v2:172.21.15.137:3300/0 conn(0x7f90e415d750 0x7f90e406cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.485+0000 7f90e3e00640 1 --2- 172.21.15.2:0/1194862664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e415e240 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f90e4067e00 tx=0x7f90dc02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12c68536838dba2a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.485+0000 7f90e2a00640 1 -- 172.21.15.2:0/1194862664 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc02cc70 con 0x7f90e40ad960 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.485+0000 7f90e2a00640 1 -- 172.21.15.2:0/1194862664 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90dc02cdd0 con 0x7f90e40ad960 2024-01-15T18:05:49.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.485+0000 7f90e2a00640 1 -- 172.21.15.2:0/1194862664 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc034660 con 0x7f90e40ad960 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 -- 172.21.15.2:0/1194862664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 msgr2=0x7f90e415e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1194862664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e415e240 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f90e4067e00 tx=0x7f90dc02bec0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 -- 172.21.15.2:0/1194862664 shutdown_connections 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1194862664 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e415e240 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1194862664 >> v2:172.21.15.137:3300/0 conn(0x7f90e415d750 0x7f90e406cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 -- 172.21.15.2:0/1194862664 >> 172.21.15.2:0/1194862664 conn(0x7f90e40b0840 msgr2=0x7f90e40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 -- 172.21.15.2:0/1194862664 shutdown_connections 2024-01-15T18:05:49.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.486+0000 7f90e8c00640 1 -- 172.21.15.2:0/1194862664 wait complete. 2024-01-15T18:05:49.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.488+0000 7f90e8c00640 1 Processor -- start 2024-01-15T18:05:49.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.488+0000 7f90e8c00640 1 -- start start 2024-01-15T18:05:49.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90e8c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90e406d050 0x7f90e414ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90e8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e414b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90e8c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90e415d750 0x7f90e414edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90e8c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40af4f0 con 0x7f90e40ad960 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90e8c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40af7d0 con 0x7f90e406d050 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90e8c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e40afab0 con 0x7f90e415d750 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e414b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e414b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53304/0 (socket says 172.21.15.2:53304) 2024-01-15T18:05:49.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.489+0000 7f90dbe00640 1 -- 172.21.15.2:0/1405490487 learned_addr learned my addr 172.21.15.2:0/1405490487 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:49.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90e3e00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90e415d750 0x7f90e414edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90e3400640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90e406d050 0x7f90e414ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90dbe00640 1 -- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90e415d750 msgr2=0x7f90e414edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90dbe00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90e415d750 0x7f90e414edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90dbe00640 1 -- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90e406d050 msgr2=0x7f90e414ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90dbe00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90e406d050 0x7f90e414ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90dbe00640 1 -- 172.21.15.2:0/1405490487 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc009a70 con 0x7f90e40ad960 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90e3400640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90e406d050 0x7f90e414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90e3e00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90e415d750 0x7f90e414edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.490+0000 7f90dbe00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e414b350 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00c410 tx=0x7f90cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.491+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc005ce0 con 0x7f90e40ad960 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.491+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc005e40 con 0x7f90e40ad960 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.491+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc0072f0 con 0x7f90e40ad960 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.491+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e40afd90 con 0x7f90e40ad960 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.491+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e414f4d0 con 0x7f90e40ad960 2024-01-15T18:05:49.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.492+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e40cae90 con 0x7f90e40ad960 2024-01-15T18:05:49.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.493+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f90cc007510 con 0x7f90e40ad960 2024-01-15T18:05:49.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.494+0000 7f90db400640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f90bc06bbc0 0x7f90bc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:49.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.494+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f90cc087ad0 con 0x7f90e40ad960 2024-01-15T18:05:49.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.495+0000 7f90e3400640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f90bc06bbc0 0x7f90bc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:49.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.496+0000 7f90e3400640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f90bc06bbc0 0x7f90bc06e080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f90d4006fd0 tx=0x7f90d4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:49.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.499+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90e40cae90 con 0x7f90e40ad960 2024-01-15T18:05:49.583 INFO:tasks.ceph.mds.a.smithi002.stderr:2024-01-15T18:05:49.579+0000 7f1746000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:05:49.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.727+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90e40cae90 con 0x7f90e40ad960 2024-01-15T18:05:49.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.728+0000 7f90db400640 1 -- 172.21.15.2:0/1405490487 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f90e40cae90 con 0x7f90e40ad960 2024-01-15T18:05:49.732 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:05:49.732 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5573,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2894571444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2894571444},{"type":"v1","addr":"172.21.15.137:6836","nonce":2894571444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5980,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/2585344206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2585344206},{"type":"v1","addr":"172.21.15.137:6837","nonce":2585344206}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2342931858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2342931858},{"type":"v1","addr":"172.21.15.2:6833","nonce":2342931858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:05:43.093928+0000","modified":"2024-01-15T18:05:46.161925+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6993},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6993":{"gid":6993,"name":"a","rank":0,"incarnation":42,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6835/4035045426","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":4035045426},{"type":"v1","addr":"172.21.15.2:6835","nonce":4035045426}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-15T18:05:49.733 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 46 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f90bc06bbc0 msgr2=0x7f90bc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f90bc06bbc0 0x7f90bc06e080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f90d4006fd0 tx=0x7f90d4008040 comp rx=0 tx=0).stop 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 msgr2=0x7f90e414b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e414b350 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f90cc00c410 tx=0x7f90cc00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 shutdown_connections 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f90bc06bbc0 0x7f90bc06e080 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f90e415d750 0x7f90e414edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f90e40ad960 0x7f90e414b350 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 --2- 172.21.15.2:0/1405490487 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f90e406d050 0x7f90e414ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 >> 172.21.15.2:0/1405490487 conn(0x7f90e40b0840 msgr2=0x7f90e4150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 shutdown_connections 2024-01-15T18:05:49.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:49.731+0000 7f90e8c00640 1 -- 172.21.15.2:0/1405490487 wait complete. 2024-01-15T18:05:50.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:05:43.093928+0000', 'modified': '2024-01-15T18:05:46.161925+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6993}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6993': {'gid': 6993, 'name': 'a', 'rank': 0, 'incarnation': 42, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.2:6835/4035045426', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6834', 'nonce': 4035045426}, {'type': 'v1', 'addr': '172.21.15.2:6835', 'nonce': 4035045426}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:05:50.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:05:50.158 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-01-15T18:05:50.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.380+0000 7fdaf4400640 1 Processor -- start 2024-01-15T18:05:50.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.380+0000 7fdaf4400640 1 -- start start 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf4400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdaec144b80 0x7fdaec0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec0b30a0 0x7fdaec06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf4400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaec0b35a0 con 0x7fdaec06cdf0 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf4400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaec0b4220 con 0x7fdaec144b80 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec06d280 con 0x7fdaec0b30a0 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf1a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdaec144b80 0x7fdaec0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf1a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdaec144b80 0x7fdaec0cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53306/0 (socket says 172.21.15.2:53306) 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.382+0000 7fdaf1a00640 1 -- 172.21.15.2:0/3695195813 learned_addr learned my addr 172.21.15.2:0/3695195813 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.383+0000 7fdaf1a00640 1 -- 172.21.15.2:0/3695195813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec0b30a0 msgr2=0x7fdaec06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.383+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3695195813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec0b30a0 0x7fdaec06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.383+0000 7fdaf1a00640 1 -- 172.21.15.2:0/3695195813 >> v1:172.21.15.137:6789/0 conn(0x7fdaec06cdf0 legacy=0x7fdaec0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.383+0000 7fdaf1a00640 1 -- 172.21.15.2:0/3695195813 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaec0b3880 con 0x7fdaec144b80 2024-01-15T18:05:50.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.384+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3695195813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec144b80 0x7fdaec0cbae0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009870 tx=0x7fdadc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1977412362ce5a89 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.384+0000 7fdaebe00640 1 -- 172.21.15.2:0/3695195813 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0317c0 con 0x7fdaec144b80 2024-01-15T18:05:50.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaebe00640 1 -- 172.21.15.2:0/3695195813 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc031920 con 0x7fdaec144b80 2024-01-15T18:05:50.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaebe00640 1 -- 172.21.15.2:0/3695195813 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc031cb0 con 0x7fdaec144b80 2024-01-15T18:05:50.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 -- 172.21.15.2:0/3695195813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec144b80 msgr2=0x7fdaec0cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 --2- 172.21.15.2:0/3695195813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec144b80 0x7fdaec0cbae0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fdadc009870 tx=0x7fdadc02b860 comp rx=0 tx=0).stop 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 -- 172.21.15.2:0/3695195813 shutdown_connections 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 --2- 172.21.15.2:0/3695195813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec0b30a0 0x7fdaec06c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 --2- 172.21.15.2:0/3695195813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec144b80 0x7fdaec0cbae0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 -- 172.21.15.2:0/3695195813 >> 172.21.15.2:0/3695195813 conn(0x7fdaec0b46d0 msgr2=0x7fdaec0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 -- 172.21.15.2:0/3695195813 shutdown_connections 2024-01-15T18:05:50.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.385+0000 7fdaf4400640 1 -- 172.21.15.2:0/3695195813 wait complete. 2024-01-15T18:05:50.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.387+0000 7fdaf4400640 1 Processor -- start 2024-01-15T18:05:50.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.387+0000 7fdaf4400640 1 -- start start 2024-01-15T18:05:50.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.388+0000 7fdaf4400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdaec06cdf0 0x7fdaec0ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.388+0000 7fdaf4400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 0x7fdaec13e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.388+0000 7fdaf4400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec144b80 0x7fdaec13d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.388+0000 7fdaf4400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0ad9f0 con 0x7fdaec0b30a0 2024-01-15T18:05:50.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.388+0000 7fdaf4400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0acd70 con 0x7fdaec06cdf0 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.388+0000 7fdaf4400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaec0ad050 con 0x7fdaec144b80 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 0x7fdaec13e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 0x7fdaec13e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53322/0 (socket says 172.21.15.2:53322) 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 -- 172.21.15.2:0/1168111293 learned_addr learned my addr 172.21.15.2:0/1168111293 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdaec06cdf0 0x7fdaec0ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf2400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec144b80 0x7fdaec13d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 -- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec144b80 msgr2=0x7fdaec13d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec144b80 0x7fdaec13d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 -- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdaec06cdf0 msgr2=0x7fdaec0ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdaec06cdf0 0x7fdaec0ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 -- 172.21.15.2:0/1168111293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc0094d0 con 0x7fdaec0b30a0 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdaec06cdf0 0x7fdaec0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:50.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf2400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec144b80 0x7fdaec13d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:50.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.389+0000 7fdaf1000640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 0x7fdaec13e0b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fdad400b4a0 tx=0x7fdad400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.390+0000 7fdae9600640 1 -- 172.21.15.2:0/1168111293 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4010610 con 0x7fdaec0b30a0 2024-01-15T18:05:50.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.390+0000 7fdae9600640 1 -- 172.21.15.2:0/1168111293 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad4010770 con 0x7fdaec0b30a0 2024-01-15T18:05:50.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.390+0000 7fdae9600640 1 -- 172.21.15.2:0/1168111293 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4010b20 con 0x7fdaec0b30a0 2024-01-15T18:05:50.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.390+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaec0ad330 con 0x7fdaec0b30a0 2024-01-15T18:05:50.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.390+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec15ad00 con 0x7fdaec0b30a0 2024-01-15T18:05:50.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.392+0000 7fdae9600640 1 -- 172.21.15.2:0/1168111293 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fdad4010c80 con 0x7fdaec0b30a0 2024-01-15T18:05:50.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.393+0000 7fdae9600640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdac406bbc0 0x7fdac406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.393+0000 7fdae9600640 1 -- 172.21.15.2:0/1168111293 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fdad40882c0 con 0x7fdaec0b30a0 2024-01-15T18:05:50.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.394+0000 7fdaf71fc740 1 Processor -- start 2024-01-15T18:05:50.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.394+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdac406bbc0 0x7fdac406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.394+0000 7fdaf71fc740 1 -- start start 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 0x5572ee151280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee14bba0 0x5572ee1500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee1505e0 0x5572ee194560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee0af880 con 0x5572ee150ea0 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee0beae0 con 0x5572ee1505e0 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572ee0bc1d0 con 0x5572ee14bba0 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdac406bbc0 0x7fdac406e080 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdadc036000 tx=0x7fdadc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 0x5572ee151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 0x5572ee151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53330/0 (socket says 172.21.15.2:53330) 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf1a00640 1 -- 172.21.15.2:0/1828253310 learned_addr learned my addr 172.21.15.2:0/1828253310 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf1000640 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee14bba0 0x5572ee1500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.395+0000 7fdaf2400640 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee1505e0 0x5572ee194560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf1a00640 1 -- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee14bba0 msgr2=0x5572ee1500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee14bba0 0x5572ee1500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf1a00640 1 -- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee1505e0 msgr2=0x5572ee194560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee1505e0 0x5572ee194560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf1a00640 1 -- 172.21.15.2:0/1828253310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572ee07f860 con 0x5572ee150ea0 2024-01-15T18:05:50.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf2400640 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee1505e0 0x5572ee194560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 0x5572ee151280 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fdadc034040 tx=0x7fdadc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241a96d28addf8ae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdae2000640 1 -- 172.21.15.2:0/1828253310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc031da0 con 0x5572ee150ea0 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdae2000640 1 -- 172.21.15.2:0/1828253310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc031f00 con 0x5572ee150ea0 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.396+0000 7fdae2000640 1 -- 172.21.15.2:0/1828253310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc04e400 con 0x5572ee150ea0 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 -- 172.21.15.2:0/1828253310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 msgr2=0x5572ee151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 0x5572ee151280 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fdadc034040 tx=0x7fdadc030000 comp rx=0 tx=0).stop 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 -- 172.21.15.2:0/1828253310 shutdown_connections 2024-01-15T18:05:50.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee1505e0 0x5572ee194560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee14bba0 0x5572ee1500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/1828253310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee150ea0 0x5572ee151280 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 -- 172.21.15.2:0/1828253310 >> 172.21.15.2:0/1828253310 conn(0x5572ee112c70 msgr2=0x5572ee196990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 -- 172.21.15.2:0/1828253310 shutdown_connections 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.397+0000 7fdaf71fc740 1 -- 172.21.15.2:0/1828253310 wait complete. 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.398+0000 7fdaf71fc740 1 -- start start 2024-01-15T18:05:50.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.398+0000 7fdaf71fc740 1 Processor -- start 2024-01-15T18:05:50.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee16d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee1713e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee0b7ac0 con 0x5572ee1505e0 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee0542f0 con 0x5572ee14bba0 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572ee071c70 con 0x5572ee150ea0 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf71fc740 10 client.0 _ll_register_callbacks cb 0x5572ee0763b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53336/0 (socket says 172.21.15.2:53336) 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.399+0000 7fdaf1000640 1 -- 172.21.15.2:0/3963128076 learned_addr learned my addr 172.21.15.2:0/3963128076 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1000640 1 -- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 msgr2=0x5572ee1713e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1000640 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee1713e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1000640 1 -- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 msgr2=0x5572ee16d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee16d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1000640 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee16d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1000640 1 -- 172.21.15.2:0/3963128076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaec0b3880 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf1000640 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1984d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fdad400be60 tx=0x7fdad4058790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaae7d42a08cff95 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdadaa00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdadaa00640 1 -- 172.21.15.2:0/3963128076 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad40606c0 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572ee0b0aa0 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.400+0000 7fdadaa00640 1 -- 172.21.15.2:0/3963128076 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fdad406eb50 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdadaa00640 1 -- 172.21.15.2:0/3963128076 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdad406ed10 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 10 client.7125 fetch_fsmap learned FSMap version 46 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x5572ee0b6140 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdadaa00640 1 -- 172.21.15.2:0/3963128076 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5572ee0b0aa0 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdadaa00640 1 -- 172.21.15.2:0/3963128076 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 46) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdad4054440 con 0x5572ee1505e0 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 10 client.7125 fetch_fsmap finished waiting for FSMap version 46 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 10 client.7125 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 20 client.7125 populate_metadata read hostname 'smithi002' 2024-01-15T18:05:50.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee194ef0 0x5572ee16dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 4 client.7125 mds_command: new command op to 6993 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:05:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 --> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572ee0b07c0 con 0x5572ee194ef0 2024-01-15T18:05:50.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.401+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee194ef0 0x5572ee16dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.402+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee194ef0 0x5572ee16dad0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7458332743adbf81 server_cookie=37bb08b8d187ecf1 in_seq=0 out_seq=0 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.402+0000 7fdadaa00640 10 client.7125 ms_handle_connect on v2:172.21.15.2:6834/4035045426 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdadaa00640 1 -- 172.21.15.2:0/3963128076 <== mds.0 v2:172.21.15.2:6834/4035045426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5572ee0b07c0 con 0x5572ee194ef0 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdadaa00640 10 client.7125 handle_command_reply: tid=0 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 client.7125 shutdown 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 msgr2=0x5572ee1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1984d0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fdad400be60 tx=0x7fdad4058790 comp rx=0 tx=0).stop 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 shutdown_connections 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee194ef0 0x5572ee16dad0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee1713e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1984d0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3963128076 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee16d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdadaa00640 0 client.7125 ms_handle_reset on v2:172.21.15.2:6834/4035045426 2024-01-15T18:05:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 >> 172.21.15.2:0/3963128076 conn(0x5572ee112c70 msgr2=0x5572ee196990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 shutdown_connections 2024-01-15T18:05:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.403+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3963128076 wait complete. 2024-01-15T18:05:50.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.404+0000 7fdaf71fc740 20 client.7125 trim_cache size 0 max 16384 2024-01-15T18:05:50.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.420+0000 7fdaf71fc740 1 Processor -- start 2024-01-15T18:05:50.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.421+0000 7fdaf71fc740 1 -- start start 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.421+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee196990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.421+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1939c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.421+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee16ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ede1af20 con 0x5572ee1505e0 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee0b9a40 con 0x5572ee14bba0 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572ee06bd90 con 0x5572ee150ea0 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1939c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1939c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53346/0 (socket says 172.21.15.2:53346) 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 -- 172.21.15.2:0/642070830 learned_addr learned my addr 172.21.15.2:0/642070830 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf2400640 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee16ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee196990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 -- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 msgr2=0x5572ee16ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee16ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 -- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 msgr2=0x5572ee196990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee196990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.422+0000 7fdaf1000640 1 -- 172.21.15.2:0/642070830 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572ee053b80 con 0x5572ee1505e0 2024-01-15T18:05:50.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.423+0000 7fdaf1000640 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1939c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fdad400f020 tx=0x7fdad401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ebbe8e98abbeab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.423+0000 7fdae2a00640 1 -- 172.21.15.2:0/642070830 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad40604c0 con 0x5572ee1505e0 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.423+0000 7fdae2a00640 1 -- 172.21.15.2:0/642070830 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad4060620 con 0x5572ee1505e0 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.423+0000 7fdae2a00640 1 -- 172.21.15.2:0/642070830 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4060790 con 0x5572ee1505e0 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.423+0000 7fdaf71fc740 1 -- 172.21.15.2:0/642070830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 msgr2=0x5572ee1939c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.423+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1939c0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fdad400f020 tx=0x7fdad401b000 comp rx=0 tx=0).stop 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 -- 172.21.15.2:0/642070830 shutdown_connections 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee16ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee1939c0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/642070830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee196990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 -- 172.21.15.2:0/642070830 >> 172.21.15.2:0/642070830 conn(0x5572ee112c70 msgr2=0x5572ee14ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 -- 172.21.15.2:0/642070830 shutdown_connections 2024-01-15T18:05:50.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 -- 172.21.15.2:0/642070830 wait complete. 2024-01-15T18:05:50.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.424+0000 7fdaf71fc740 1 -- start start 2024-01-15T18:05:50.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.425+0000 7fdaf71fc740 1 Processor -- start 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee200180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee151280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee151fe0 con 0x5572ee1505e0 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572ee17fbf0 con 0x5572ee14bba0 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572ee16ef80 con 0x5572ee150ea0 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf71fc740 10 client.0 _ll_register_callbacks cb 0x7fdad4058650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:05:50.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53348/0 (socket says 172.21.15.2:53348) 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 -- 172.21.15.2:0/3856253350 learned_addr learned my addr 172.21.15.2:0/3856253350 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee200180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 -- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 msgr2=0x5572ee1517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 -- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 msgr2=0x5572ee200180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee200180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1000640 1 -- 172.21.15.2:0/3856253350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572ee0b6140 con 0x5572ee1505e0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.426+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee200180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdadaa00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdaf1000640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee151280 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdad4054b50 tx=0x7fdad4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a46cd59138ca5cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdadaa00640 1 -- 172.21.15.2:0/3856253350 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4060790 con 0x5572ee1505e0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdadaa00640 1 -- 172.21.15.2:0/3856253350 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fdad4005dd0 con 0x5572ee1505e0 2024-01-15T18:05:50.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdadc031f00 con 0x5572ee1505e0 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdadaa00640 1 -- 172.21.15.2:0/3856253350 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdad40604c0 con 0x5572ee1505e0 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdaf71fc740 10 client.7131 fetch_fsmap learned FSMap version 46 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=2+}) v3 -- 0x5572ee079170 con 0x5572ee1505e0 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdadaa00640 1 -- 172.21.15.2:0/3856253350 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc031f00 con 0x5572ee1505e0 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.427+0000 7fdadaa00640 1 -- 172.21.15.2:0/3856253350 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 46) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdad4078020 con 0x5572ee1505e0 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf71fc740 10 client.7131 fetch_fsmap finished waiting for FSMap version 46 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf71fc740 10 client.7131 resolve_mds: resolved name 'a' to daemon mds.a 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf71fc740 20 client.7131 populate_metadata read hostname 'smithi002' 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee203d20 0x5572ee204100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf71fc740 4 client.7131 mds_command: new command op to 6993 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 --> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5572eddb86c0 con 0x5572ee203d20 2024-01-15T18:05:50.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee203d20 0x5572ee204100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdaf1a00640 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee203d20 0x5572ee204100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92630d0e005251bf server_cookie=e799cdd6a6d0107 in_seq=0 out_seq=0 2024-01-15T18:05:50.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.428+0000 7fdadaa00640 10 client.7131 ms_handle_connect on v2:172.21.15.2:6834/4035045426 2024-01-15T18:05:50.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdadaa00640 1 -- 172.21.15.2:0/3856253350 <== mds.0 v2:172.21.15.2:6834/4035045426 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5572eddb86c0 con 0x5572ee203d20 2024-01-15T18:05:50.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdadaa00640 10 client.7131 handle_command_reply: tid=0 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 client.7131 shutdown 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 msgr2=0x5572ee151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee151280 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fdad4054b50 tx=0x7fdad4000bc0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 shutdown_connections 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:6834/4035045426,v1:172.21.15.2:6835/4035045426] conn(0x5572ee203d20 0x5572ee204100 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x5572ee150ea0 0x5572ee1517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x5572ee1505e0 0x5572ee151280 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 --2- 172.21.15.2:0/3856253350 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x5572ee14bba0 0x5572ee200180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdadaa00640 0 client.7131 ms_handle_reset on v2:172.21.15.2:6834/4035045426 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 >> 172.21.15.2:0/3856253350 conn(0x5572ee112c70 msgr2=0x5572ee193bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 shutdown_connections 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 1 -- 172.21.15.2:0/3856253350 wait complete. 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.429+0000 7fdaf71fc740 20 client.7131 trim_cache size 0 max 16384 2024-01-15T18:05:50.436 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 6993, 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 6.7758907160000001, 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 46, 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 73, 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 73, 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 12.175442865000001 2024-01-15T18:05:50.437 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdac406bbc0 msgr2=0x7fdac406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdac406bbc0 0x7fdac406e080 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdadc036000 tx=0x7fdadc03d040 comp rx=0 tx=0).stop 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 msgr2=0x7fdaec13e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 0x7fdaec13e0b0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fdad400b4a0 tx=0x7fdad400b960 comp rx=0 tx=0).stop 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 shutdown_connections 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdac406bbc0 0x7fdac406e080 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdaec144b80 0x7fdaec13d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdaec0b30a0 0x7fdaec13e0b0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 --2- 172.21.15.2:0/1168111293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdaec06cdf0 0x7fdaec0ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 >> 172.21.15.2:0/1168111293 conn(0x7fdaec0b46d0 msgr2=0x7fdaec0afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 shutdown_connections 2024-01-15T18:05:50.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.430+0000 7fdaf4400640 1 -- 172.21.15.2:0/1168111293 wait complete. 2024-01-15T18:05:50.451 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:05:50.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.661+0000 7f1c89800640 1 Processor -- start 2024-01-15T18:05:50.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.661+0000 7f1c89800640 1 -- start start 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c89800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1c8415d340 0x7f1c840cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c89800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1c840d3090 0x7f1c840d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c89800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c840d1680 0x7f1c840d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c89800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c840d3a60 con 0x7f1c8415d340 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c89800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c840b3de0 con 0x7f1c840d3090 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c89800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840b40c0 con 0x7f1c840d1680 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c83e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1c8415d340 0x7f1c840cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c83e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1c8415d340 0x7f1c840cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53362/0 (socket says 172.21.15.2:53362) 2024-01-15T18:05:50.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.663+0000 7f1c83e00640 1 -- 172.21.15.2:0/2632813376 learned_addr learned my addr 172.21.15.2:0/2632813376 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.664+0000 7f1c83e00640 1 -- 172.21.15.2:0/2632813376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c840d1680 msgr2=0x7f1c840d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.664+0000 7f1c83e00640 1 --2- 172.21.15.2:0/2632813376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c840d1680 0x7f1c840d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.664+0000 7f1c83e00640 1 -- 172.21.15.2:0/2632813376 >> v2:172.21.15.137:3300/0 conn(0x7f1c840d3090 msgr2=0x7f1c840d3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:50.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.664+0000 7f1c83e00640 1 --2- 172.21.15.2:0/2632813376 >> v2:172.21.15.137:3300/0 conn(0x7f1c840d3090 0x7f1c840d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.664+0000 7f1c83e00640 1 -- 172.21.15.2:0/2632813376 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c840d0e80 con 0x7f1c8415d340 2024-01-15T18:05:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.665+0000 7f1c83e00640 1 --2- 172.21.15.2:0/2632813376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c8415d340 0x7f1c840cdd20 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f1c8408c560 tx=0x7f1c7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2586f0eae8c26d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.665+0000 7f1c82a00640 1 -- 172.21.15.2:0/2632813376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74031870 con 0x7f1c8415d340 2024-01-15T18:05:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.665+0000 7f1c82a00640 1 -- 172.21.15.2:0/2632813376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c740319d0 con 0x7f1c8415d340 2024-01-15T18:05:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.665+0000 7f1c82a00640 1 -- 172.21.15.2:0/2632813376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74031d60 con 0x7f1c8415d340 2024-01-15T18:05:50.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 -- 172.21.15.2:0/2632813376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c8415d340 msgr2=0x7f1c840cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 --2- 172.21.15.2:0/2632813376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c8415d340 0x7f1c840cdd20 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f1c8408c560 tx=0x7f1c7402b910 comp rx=0 tx=0).stop 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 -- 172.21.15.2:0/2632813376 shutdown_connections 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 --2- 172.21.15.2:0/2632813376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c840d1680 0x7f1c840d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 --2- 172.21.15.2:0/2632813376 >> v2:172.21.15.137:3300/0 conn(0x7f1c840d3090 0x7f1c840d3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 --2- 172.21.15.2:0/2632813376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c8415d340 0x7f1c840cdd20 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 -- 172.21.15.2:0/2632813376 >> 172.21.15.2:0/2632813376 conn(0x7f1c8408caf0 msgr2=0x7f1c8409c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 -- 172.21.15.2:0/2632813376 shutdown_connections 2024-01-15T18:05:50.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.666+0000 7f1c89800640 1 -- 172.21.15.2:0/2632813376 wait complete. 2024-01-15T18:05:50.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.668+0000 7f1c89800640 1 Processor -- start 2024-01-15T18:05:50.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.668+0000 7f1c89800640 1 -- start start 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c89800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 0x7f1c840b86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c89800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c840d3090 0x7f1c8415e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c89800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c8415d340 0x7f1c8415ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c89800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840c8b10 con 0x7f1c840d1680 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c89800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840c8df0 con 0x7f1c840d3090 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c89800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c840c90d0 con 0x7f1c8415d340 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 0x7f1c840b86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 0x7f1c840b86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53368/0 (socket says 172.21.15.2:53368) 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 -- 172.21.15.2:0/4262877867 learned_addr learned my addr 172.21.15.2:0/4262877867 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 -- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c8415d340 msgr2=0x7f1c8415ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c8415d340 0x7f1c8415ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 -- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c840d3090 msgr2=0x7f1c8415e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c840d3090 0x7f1c8415e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.669+0000 7f1c83e00640 1 -- 172.21.15.2:0/4262877867 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c740094d0 con 0x7f1c840d1680 2024-01-15T18:05:50.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.670+0000 7f1c83e00640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 0x7f1c840b86a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1c74000c00 tx=0x7f1c740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.670+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74031840 con 0x7f1c840d1680 2024-01-15T18:05:50.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.670+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c740443f0 con 0x7f1c840d1680 2024-01-15T18:05:50.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.670+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c740447a0 con 0x7f1c840d1680 2024-01-15T18:05:50.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.670+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c840c93b0 con 0x7f1c840d1680 2024-01-15T18:05:50.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.670+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c840c87a0 con 0x7f1c840d1680 2024-01-15T18:05:50.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.671+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c840c98d0 con 0x7f1c840d1680 2024-01-15T18:05:50.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.673+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1c7404d450 con 0x7f1c840d1680 2024-01-15T18:05:50.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.673+0000 7f1c7be00640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5c06bbc0 0x7f1c5c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:50.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.674+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1c7402d040 con 0x7f1c840d1680 2024-01-15T18:05:50.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.674+0000 7f1c83400640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5c06bbc0 0x7f1c5c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:50.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.675+0000 7f1c83400640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5c06bbc0 0x7f1c5c06e080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1c8415cd60 tx=0x7f1c6c006bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:50.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.678+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c840c98d0 con 0x7f1c840d1680 2024-01-15T18:05:50.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.901+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1c840c98d0 con 0x7f1c840d1680 2024-01-15T18:05:50.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.901+0000 7f1c7be00640 1 -- 172.21.15.2:0/4262877867 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1c840c98d0 con 0x7f1c840d1680 2024-01-15T18:05:50.906 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5c06bbc0 msgr2=0x7f1c5c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5c06bbc0 0x7f1c5c06e080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f1c8415cd60 tx=0x7f1c6c006bf0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 msgr2=0x7f1c840b86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 0x7f1c840b86a0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1c74000c00 tx=0x7f1c740046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 shutdown_connections 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5c06bbc0 0x7f1c5c06e080 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c8415d340 0x7f1c8415ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c840d3090 0x7f1c8415e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 --2- 172.21.15.2:0/4262877867 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c840d1680 0x7f1c840b86a0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:50.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 >> 172.21.15.2:0/4262877867 conn(0x7f1c8408caf0 msgr2=0x7f1c8406b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:50.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.904+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 shutdown_connections 2024-01-15T18:05:50.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:50.905+0000 7f1c89800640 1 -- 172.21.15.2:0/4262877867 wait complete. 2024-01-15T18:05:50.928 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:05:50.928 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:05:50.928 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T18:05:50.928 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:05:50.929 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:50.929 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:05:50.929 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:05:50.929 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:05:50.929 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:05:50.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:50.929 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.984 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T18:05:50.985 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 42131sec preferred_lft 42131sec 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.986 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:50.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:50.988 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:05:51.082 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T18:05:51.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:51.087 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T18:05:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T18:05:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:05:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T18:05:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T18:05:51.141 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:05:51.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:51.143 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T18:05:51.143 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T18:05:51.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:05:51.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:05:51.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:05:51.143 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T18:05:51.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:51.334 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:05:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:05:51.389 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:05:51.390 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:05:51.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:51.390 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T18:05:51.492 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:05:51.492 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:51.493 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:51.552 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:05:51.553 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:51.553 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:51.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:05:51.713 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:05:51.768 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:05:51.769 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:51.770 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:51.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:51.771 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:05:51.827 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:05:51.827 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:05:51.834 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:05:51.917 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:51.959 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:05:51.959 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:05:51.960 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:05:51.961 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:05:51.962 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:51.962 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:05:51.962 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:51.962 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:05:51.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:51.963 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:05:52.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T18:05:52.025+0000 7f260b238480 -1 init, newargv = 0x5564e97617e0 newargc=15 2024-01-15T18:05:52.029 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[61789]: starting ceph client 2024-01-15T18:05:52.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[61789]: starting fuse 2024-01-15T18:05:52.048 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T18:05:52.048 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T18:05:52.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:05:52.048 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:05:52.049 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T18:05:52.050 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T18:05:52.250 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.61789.asok 2024-01-15T18:05:52.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:05:52.267 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61789.asok 2024-01-15T18:05:52.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:52.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61789.asok status 2024-01-15T18:05:52.515 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:05:52.515 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T18:05:52.515 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:05:52.515 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:05:52.515 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T18:05:52.515 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "61789", 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 7143, 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T18:05:52.516 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 7143 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3347292353 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:05:52.517 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 3347292353 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.7143 192.168.0.1:0/3347292353", 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/3347292353", 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 46, 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 73, 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T18:05:52.518 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:05:52.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:52.531 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T18:05:52.531 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:52.583 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T18:05:52.583 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:52.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:52.584 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:05:52.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:05:52.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.908+0000 7efd91800640 1 Processor -- start 2024-01-15T18:05:52.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.908+0000 7efd91800640 1 -- start start 2024-01-15T18:05:52.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.909+0000 7efd91800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efd8c13ce30 0x7efd8c15dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.909+0000 7efd91800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c15e170 0x7efd8c0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.909+0000 7efd91800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8c0aeab0 con 0x7efd8c0cdfa0 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.909+0000 7efd91800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8c15e5f0 con 0x7efd8c13ce30 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.909+0000 7efd91800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0ce410 con 0x7efd8c15e170 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.910+0000 7efd8b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7efd8c0cdfa0 0x7efd8c13f720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39032/0 (socket says 172.21.15.2:39032) 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.910+0000 7efd8b400640 1 -- 172.21.15.2:0/396505186 learned_addr learned my addr 172.21.15.2:0/396505186 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.910+0000 7efd8aa00640 1 --2- 172.21.15.2:0/396505186 >> v2:172.21.15.137:3300/0 conn(0x7efd8c13ce30 0x7efd8c15dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.910+0000 7efd81e00640 1 --2- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c15e170 0x7efd8c0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.910+0000 7efd8a000640 1 -- 172.21.15.2:0/396505186 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183481731 0 0) 0x7efd8c0aeab0 con 0x7efd8c0cdfa0 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.911+0000 7efd8a000640 1 -- 172.21.15.2:0/396505186 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd6c003520 con 0x7efd8c0cdfa0 2024-01-15T18:05:52.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.911+0000 7efd8aa00640 1 -- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c15e170 msgr2=0x7efd8c0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:52.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.911+0000 7efd8aa00640 1 --2- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c15e170 0x7efd8c0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.911+0000 7efd8a000640 1 -- 172.21.15.2:0/396505186 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647977711 0 0) 0x7efd6c003520 con 0x7efd8c0cdfa0 2024-01-15T18:05:52.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.911+0000 7efd8aa00640 1 -- 172.21.15.2:0/396505186 >> v1:172.21.15.2:6789/0 conn(0x7efd8c0cdfa0 legacy=0x7efd8c13f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.911+0000 7efd8aa00640 1 -- 172.21.15.2:0/396505186 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c070380 con 0x7efd8c13ce30 2024-01-15T18:05:52.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.912+0000 7efd8aa00640 1 --2- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c13ce30 0x7efd8c15dc30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0097a0 tx=0x7efd7c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ec3f97c2d523b4b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.913+0000 7efd8a000640 1 -- 172.21.15.2:0/396505186 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd840019f0 con 0x7efd8c13ce30 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.913+0000 7efd8a000640 1 -- 172.21.15.2:0/396505186 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7c02e050 con 0x7efd8c13ce30 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.913+0000 7efd8a000640 1 -- 172.21.15.2:0/396505186 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7c030710 con 0x7efd8c13ce30 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.913+0000 7efd91800640 1 -- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c13ce30 msgr2=0x7efd8c15dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.913+0000 7efd91800640 1 --2- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c13ce30 0x7efd8c15dc30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0097a0 tx=0x7efd7c02b790 comp rx=0 tx=0).stop 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.914+0000 7efd91800640 1 -- 172.21.15.2:0/396505186 shutdown_connections 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.914+0000 7efd91800640 1 --2- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c15e170 0x7efd8c0cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.914+0000 7efd91800640 1 --2- 172.21.15.2:0/396505186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c13ce30 0x7efd8c15dc30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.914+0000 7efd91800640 1 -- 172.21.15.2:0/396505186 >> 172.21.15.2:0/396505186 conn(0x7efd8c06da20 msgr2=0x7efd8c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.914+0000 7efd91800640 1 -- 172.21.15.2:0/396505186 shutdown_connections 2024-01-15T18:05:52.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.914+0000 7efd91800640 1 -- 172.21.15.2:0/396505186 wait complete. 2024-01-15T18:05:52.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.916+0000 7efd91800640 1 Processor -- start 2024-01-15T18:05:52.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.916+0000 7efd91800640 1 -- start start 2024-01-15T18:05:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd91800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c0cdfa0 0x7efd8c1f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd91800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 0x7efd8c1f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd91800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c15e170 0x7efd8c13f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd91800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c13d2b0 con 0x7efd8c13ce30 2024-01-15T18:05:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd91800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c13fc70 con 0x7efd8c15e170 2024-01-15T18:05:52.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd91800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c13d520 con 0x7efd8c0cdfa0 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 0x7efd8c1f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 0x7efd8c1f3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54698/0 (socket says 172.21.15.2:54698) 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 -- 172.21.15.2:0/2375282262 learned_addr learned my addr 172.21.15.2:0/2375282262 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd8b400640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c15e170 0x7efd8c13f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd8aa00640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c0cdfa0 0x7efd8c1f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 -- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c0cdfa0 msgr2=0x7efd8c1f02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c0cdfa0 0x7efd8c1f02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 -- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c15e170 msgr2=0x7efd8c13f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c15e170 0x7efd8c13f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.917+0000 7efd81e00640 1 -- 172.21.15.2:0/2375282262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c009400 con 0x7efd8c13ce30 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.918+0000 7efd81e00640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 0x7efd8c1f3850 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7efd7400e9b0 tx=0x7efd7400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.918+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74009670 con 0x7efd8c13ce30 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.918+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd7400c9a0 con 0x7efd8c13ce30 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.918+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400cd50 con 0x7efd8c13ce30 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.918+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8c13d800 con 0x7efd8c13ce30 2024-01-15T18:05:52.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.918+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8c13ff10 con 0x7efd8c13ce30 2024-01-15T18:05:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.920+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8c0686f0 con 0x7efd8c13ce30 2024-01-15T18:05:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.921+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7efd74013470 con 0x7efd8c13ce30 2024-01-15T18:05:52.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.922+0000 7efd83400640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efd6406bbc0 0x7efd6406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:52.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.922+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7efd740888b0 con 0x7efd8c13ce30 2024-01-15T18:05:52.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.923+0000 7efd8aa00640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efd6406bbc0 0x7efd6406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:52.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.924+0000 7efd8aa00640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efd6406bbc0 0x7efd6406e080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efd7c034000 tx=0x7efd7c02f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:52.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:52.927+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd74054f80 con 0x7efd8c13ce30 2024-01-15T18:05:53.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.166+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efd8c0724d0 con 0x7efd8c13ce30 2024-01-15T18:05:53.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.167+0000 7efd83400640 1 -- 172.21.15.2:0/2375282262 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v46) v1 ==== 53+0+73 (secure 0 0 0) 0x7efd740585a0 con 0x7efd8c13ce30 2024-01-15T18:05:53.172 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:05:53.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efd6406bbc0 msgr2=0x7efd6406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efd6406bbc0 0x7efd6406e080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7efd7c034000 tx=0x7efd7c02f000 comp rx=0 tx=0).stop 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 msgr2=0x7efd8c1f3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 0x7efd8c1f3850 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7efd7400e9b0 tx=0x7efd7400ee70 comp rx=0 tx=0).stop 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 shutdown_connections 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efd6406bbc0 0x7efd6406e080 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.170+0000 7efd91800640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd8c15e170 0x7efd8c13f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.171+0000 7efd91800640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd8c13ce30 0x7efd8c1f3850 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.171+0000 7efd91800640 1 --2- 172.21.15.2:0/2375282262 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd8c0cdfa0 0x7efd8c1f02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.171+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 >> 172.21.15.2:0/2375282262 conn(0x7efd8c06da20 msgr2=0x7efd8c0aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:53.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.171+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 shutdown_connections 2024-01-15T18:05:53.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:53.171+0000 7efd91800640 1 -- 172.21.15.2:0/2375282262 wait complete. 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:05:53.195 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:05:53.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:53.196 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T18:05:53.256 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 42117sec preferred_lft 42117sec 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T18:05:53.258 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T18:05:53.259 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:05:53.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:53.261 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:05:53.354 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T18:05:53.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:53.359 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T18:05:53.411 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T18:05:53.411 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:05:53.411 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:05:53.411 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:05:53.411 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:05:53.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:53.411 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T18:05:53.411 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T18:05:53.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:05:53.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:05:53.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:05:53.412 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T18:05:53.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:53.598 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:05:53.652 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:05:53.652 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:05:53.653 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:05:53.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:53.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T18:05:53.749 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:05:53.749 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:53.749 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:53.803 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:05:53.803 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:05:53.803 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:53.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:05:53.947 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:05:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:05:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:05:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:05:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:05:54.002 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:54.003 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:05:54.004 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:54.005 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:54.005 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:54.005 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:54.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:54.006 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:05:54.056 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:05:54.056 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:05:54.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:05:54.137 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:05:54.182 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:05:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:05:54.184 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:05:54.185 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:54.186 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:05:54.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:54.187 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:05:54.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T18:05:54.247+0000 7f6fa8841480 -1 init, newargv = 0x55744ce42ba0 newargc=15 2024-01-15T18:05:54.251 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45006]: starting ceph client 2024-01-15T18:05:54.262 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45006]: starting fuse 2024-01-15T18:05:54.274 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T18:05:54.274 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T18:05:54.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:05:54.274 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T18:05:54.274 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T18:05:54.274 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T18:05:54.275 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:05:54.276 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T18:05:54.277 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T18:05:54.465 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.45006.asok 2024-01-15T18:05:54.469 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T18:05:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:05:54.482 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45006.asok 2024-01-15T18:05:54.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:54.482 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45006.asok status 2024-01-15T18:05:54.717 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "45006", 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T18:05:54.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 5651, 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 5651 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:05:54.719 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1526666068 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1526666068 2024-01-15T18:05:54.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.5651 192.168.0.1:0/1526666068", 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/1526666068", 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 46, 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 73, 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T18:05:54.721 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T18:05:54.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:54.734 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T18:05:54.734 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:54.788 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T18:05:54.788 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:54.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:54.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:05:54.880 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:05:55.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.116+0000 7f9bb0a00640 1 Processor -- start 2024-01-15T18:05:55.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.116+0000 7f9bb0a00640 1 -- start start 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9bb0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac143d20 0x7f9bac15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9bb0a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bac0720d0 0x7f9bac15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9bb0a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9bac1447d0 0x7f9bac0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9bb0a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac071430 con 0x7f9bac0720d0 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9bb0a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bac144100 con 0x7f9bac1447d0 2024-01-15T18:05:55.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9bb0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac071090 con 0x7f9bac143d20 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9baaa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bac0720d0 0x7f9bac15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9baaa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bac0720d0 0x7f9bac15f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54702/0 (socket says 172.21.15.2:54702) 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.118+0000 7f9baaa00640 1 -- 172.21.15.2:0/4087609420 learned_addr learned my addr 172.21.15.2:0/4087609420 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9babe00640 1 --2- 172.21.15.2:0/4087609420 >> v2:172.21.15.137:3300/0 conn(0x7f9bac1447d0 0x7f9bac0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9bab400640 1 --2- 172.21.15.2:0/4087609420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac143d20 0x7f9bac15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9baaa00640 1 -- 172.21.15.2:0/4087609420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac143d20 msgr2=0x7f9bac15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9baaa00640 1 --2- 172.21.15.2:0/4087609420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac143d20 0x7f9bac15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9baaa00640 1 -- 172.21.15.2:0/4087609420 >> v2:172.21.15.137:3300/0 conn(0x7f9bac1447d0 msgr2=0x7f9bac0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9baaa00640 1 --2- 172.21.15.2:0/4087609420 >> v2:172.21.15.137:3300/0 conn(0x7f9bac1447d0 0x7f9bac0cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.119+0000 7f9baaa00640 1 -- 172.21.15.2:0/4087609420 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bac144560 con 0x7f9bac0720d0 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.120+0000 7f9baaa00640 1 --2- 172.21.15.2:0/4087609420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac0720d0 0x7f9bac15f090 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9bac06e0a0 tx=0x7f9b9402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b506254a79aea7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.121+0000 7f9baa000640 1 -- 172.21.15.2:0/4087609420 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b940317e0 con 0x7f9bac0720d0 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.121+0000 7f9baa000640 1 -- 172.21.15.2:0/4087609420 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94031940 con 0x7f9bac0720d0 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.121+0000 7f9baa000640 1 -- 172.21.15.2:0/4087609420 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94031cd0 con 0x7f9bac0720d0 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.121+0000 7f9bb0a00640 1 -- 172.21.15.2:0/4087609420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac0720d0 msgr2=0x7f9bac15f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.121+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/4087609420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac0720d0 0x7f9bac15f090 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f9bac06e0a0 tx=0x7f9b9402b880 comp rx=0 tx=0).stop 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 -- 172.21.15.2:0/4087609420 shutdown_connections 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/4087609420 >> v2:172.21.15.137:3300/0 conn(0x7f9bac1447d0 0x7f9bac0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/4087609420 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac0720d0 0x7f9bac15f090 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/4087609420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac143d20 0x7f9bac15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 -- 172.21.15.2:0/4087609420 >> 172.21.15.2:0/4087609420 conn(0x7f9bac0b0840 msgr2=0x7f9bac0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 -- 172.21.15.2:0/4087609420 shutdown_connections 2024-01-15T18:05:55.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.122+0000 7f9bb0a00640 1 -- 172.21.15.2:0/4087609420 wait complete. 2024-01-15T18:05:55.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.123+0000 7f9bb0a00640 1 Processor -- start 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.124+0000 7f9bb0a00640 1 -- start start 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bb0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bac0720d0 0x7f9bac133420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bb0a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 0x7f9bac15c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bb0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac1447d0 0x7f9bac0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bb0a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac15cf90 con 0x7f9bac143d20 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bb0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac15d270 con 0x7f9bac0720d0 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bb0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bac15d550 con 0x7f9bac1447d0 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 0x7f9bac15c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 0x7f9bac15c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54706/0 (socket says 172.21.15.2:54706) 2024-01-15T18:05:55.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 -- 172.21.15.2:0/3162111835 learned_addr learned my addr 172.21.15.2:0/3162111835 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9bab400640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bac0720d0 0x7f9bac133420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9babe00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac1447d0 0x7f9bac0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 -- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac1447d0 msgr2=0x7f9bac0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac1447d0 0x7f9bac0c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 -- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bac0720d0 msgr2=0x7f9bac133420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bac0720d0 0x7f9bac133420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.125+0000 7f9baaa00640 1 -- 172.21.15.2:0/3162111835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b940094d0 con 0x7f9bac143d20 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.126+0000 7f9baaa00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 0x7f9bac15c540 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9b940090b0 tx=0x7f9b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.126+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94032880 con 0x7f9bac143d20 2024-01-15T18:05:55.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.126+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b94032a00 con 0x7f9bac143d20 2024-01-15T18:05:55.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.126+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94032db0 con 0x7f9bac143d20 2024-01-15T18:05:55.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.126+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bac15d830 con 0x7f9bac143d20 2024-01-15T18:05:55.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.126+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bac139d50 con 0x7f9bac143d20 2024-01-15T18:05:55.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.128+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bac15dd50 con 0x7f9bac143d20 2024-01-15T18:05:55.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.129+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f9b9404d3f0 con 0x7f9bac143d20 2024-01-15T18:05:55.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.129+0000 7f9ba3400640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b8406bbc0 0x7f9b8406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.130+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f9b940aa270 con 0x7f9bac143d20 2024-01-15T18:05:55.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.130+0000 7f9bab400640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b8406bbc0 0x7f9b8406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.131+0000 7f9bab400640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b8406bbc0 0x7f9b8406e080 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c006fd0 tx=0x7f9b9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.134+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bac15dd50 con 0x7f9bac143d20 2024-01-15T18:05:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.369+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9bac15dd50 con 0x7f9bac143d20 2024-01-15T18:05:55.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.560+0000 7f9ba3400640 1 -- 172.21.15.2:0/3162111835 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 136+0+0 (secure 0 0 0) 0x7f9bac15dd50 con 0x7f9bac143d20 2024-01-15T18:05:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b8406bbc0 msgr2=0x7f9b8406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b8406bbc0 0x7f9b8406e080 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c006fd0 tx=0x7f9b9c008040 comp rx=0 tx=0).stop 2024-01-15T18:05:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 msgr2=0x7f9bac15c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 0x7f9bac15c540 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f9b940090b0 tx=0x7f9b940046c0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 shutdown_connections 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b8406bbc0 0x7f9b8406e080 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bac1447d0 0x7f9bac0c67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bac143d20 0x7f9bac15c540 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 --2- 172.21.15.2:0/3162111835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bac0720d0 0x7f9bac133420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 >> 172.21.15.2:0/3162111835 conn(0x7f9bac0b0840 msgr2=0x7f9bac142fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 shutdown_connections 2024-01-15T18:05:55.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.563+0000 7f9bb0a00640 1 -- 172.21.15.2:0/3162111835 wait complete. 2024-01-15T18:05:55.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:05:55.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.814+0000 7fcfbca00640 1 Processor -- start 2024-01-15T18:05:55.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.815+0000 7fcfbca00640 1 -- start start 2024-01-15T18:05:55.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.816+0000 7fcfbca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcfb806d950 0x7fcfb80cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.816+0000 7fcfbca00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb80cbcd0 con 0x7fcfb815e8f0 2024-01-15T18:05:55.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.816+0000 7fcfbca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb8143d80 con 0x7fcfb81439a0 2024-01-15T18:05:55.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.816+0000 7fcfbca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb80b2250 con 0x7fcfb806d950 2024-01-15T18:05:55.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.816+0000 7fcfadc00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcfb815e8f0 0x7fcfb806d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33666/0 (socket says 172.21.15.2:33666) 2024-01-15T18:05:55.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.816+0000 7fcfadc00640 1 -- 172.21.15.2:0/389755932 learned_addr learned my addr 172.21.15.2:0/389755932 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263831390 0 0) 0x7fcfb80cbcd0 con 0x7fcfb815e8f0 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb7200640 1 --2- 172.21.15.2:0/389755932 >> v2:172.21.15.137:3300/0 conn(0x7fcfb806d950 0x7fcfb80cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf98003520 con 0x7fcfb815e8f0 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1019903735 0 0) 0x7fcf98003520 con 0x7fcfb815e8f0 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 >> v2:172.21.15.137:3300/0 conn(0x7fcfb806d950 msgr2=0x7fcfb80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 --2- 172.21.15.2:0/389755932 >> v2:172.21.15.137:3300/0 conn(0x7fcfb806d950 0x7fcfb80cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 >> v1:172.21.15.137:6789/0 conn(0x7fcfb81439a0 legacy=0x7fcfb80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb80b2500 con 0x7fcfb815e8f0 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fcfa0001670 con 0x7fcfb815e8f0 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 >> v1:172.21.15.2:6789/0 conn(0x7fcfb815e8f0 legacy=0x7fcfb806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf98003970 0x7fcf98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf980047c0 0x7fcf98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.817+0000 7fcfb5e00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf98005180 0x7fcf9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb80b2250 con 0x7fcf980047c0 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9800c0d0 con 0x7fcf98005180 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9800c290 con 0x7fcf98003970 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfadc00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf980047c0 0x7fcf98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfb6800640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf98003970 0x7fcf98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfb7200640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf98005180 0x7fcf9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfadc00640 1 -- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf98003970 msgr2=0x7fcf98003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfadc00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf98003970 0x7fcf98003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfadc00640 1 -- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf98005180 msgr2=0x7fcf9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:55.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfadc00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf98005180 0x7fcf9800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfadc00640 1 -- 172.21.15.2:0/389755932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf98003520 con 0x7fcf980047c0 2024-01-15T18:05:55.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.818+0000 7fcfb7200640 1 -- 172.21.15.2:0/389755932 reap_dead start 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.819+0000 7fcfadc00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf980047c0 0x7fcf98004c40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcf980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=651a23f099f424f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0005ad0 con 0x7fcf980047c0 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa0005c30 con 0x7fcf980047c0 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfb5e00640 1 -- 172.21.15.2:0/389755932 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa002caf0 con 0x7fcf980047c0 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 -- 172.21.15.2:0/389755932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf980047c0 msgr2=0x7fcf98004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf980047c0 0x7fcf98004c40 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcf980057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 -- 172.21.15.2:0/389755932 shutdown_connections 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 --2- 172.21.15.2:0/389755932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf980047c0 0x7fcf98004c40 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 -- 172.21.15.2:0/389755932 >> 172.21.15.2:0/389755932 conn(0x7fcfb80b0840 msgr2=0x7fcfb80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 -- 172.21.15.2:0/389755932 shutdown_connections 2024-01-15T18:05:55.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.820+0000 7fcfbca00640 1 -- 172.21.15.2:0/389755932 wait complete. 2024-01-15T18:05:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.822+0000 7fcfbca00640 1 Processor -- start 2024-01-15T18:05:55.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.822+0000 7fcfbca00640 1 -- start start 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.823+0000 7fcfbca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf980047c0 0x7fcfb81f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.823+0000 7fcfbca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 0x7fcfb81ecf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.823+0000 7fcfbca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb80ae7b0 0x7fcfb80aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.823+0000 7fcfbca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb81f2f40 con 0x7fcfb8070ae0 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.823+0000 7fcfbca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb8070f60 con 0x7fcf980047c0 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.823+0000 7fcfbca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb81ed4b0 con 0x7fcfb80ae7b0 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 0x7fcfb81ecf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 0x7fcfb81ecf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54718/0 (socket says 172.21.15.2:54718) 2024-01-15T18:05:55.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 -- 172.21.15.2:0/3813582130 learned_addr learned my addr 172.21.15.2:0/3813582130 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:55.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfb7200640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb80ae7b0 0x7fcfb80aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfb6800640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf980047c0 0x7fcfb81f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 -- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb80ae7b0 msgr2=0x7fcfb80aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb80ae7b0 0x7fcfb80aec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 -- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf980047c0 msgr2=0x7fcfb81f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf980047c0 0x7fcfb81f6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 -- 172.21.15.2:0/3813582130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb81f74b0 con 0x7fcfb8070ae0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.824+0000 7fcfadc00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 0x7fcfb81ecf70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcfa002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.825+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa003c040 con 0x7fcfb8070ae0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.825+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa002ccc0 con 0x7fcfb8070ae0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.825+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0007290 con 0x7fcfb8070ae0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.825+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb80ae340 con 0x7fcfb8070ae0 2024-01-15T18:05:55.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.825+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb81f6ef0 con 0x7fcfb8070ae0 2024-01-15T18:05:55.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.826+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb80cea10 con 0x7fcfb8070ae0 2024-01-15T18:05:55.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.827+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fcfa0032070 con 0x7fcfb8070ae0 2024-01-15T18:05:55.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.828+0000 7fcfaf400640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf9006bbc0 0x7fcf9006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:55.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.828+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fcfa00a9960 con 0x7fcfb8070ae0 2024-01-15T18:05:55.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.829+0000 7fcfb6800640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf9006bbc0 0x7fcf9006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:55.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.830+0000 7fcfb6800640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf9006bbc0 0x7fcf9006e080 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8007fd0 tx=0x7fcfa8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:55.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:55.832+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfa0076280 con 0x7fcfb8070ae0 2024-01-15T18:05:56.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.051+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfb80ccb10 con 0x7fcf9006bbc0 2024-01-15T18:05:56.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.055+0000 7fcfaf400640 1 -- 172.21.15.2:0/3813582130 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcfb80ccb10 con 0x7fcf9006bbc0 2024-01-15T18:05:56.059 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:05:56.059 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:05:56.059 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:05:56.059 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:05:56.059 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf9006bbc0 msgr2=0x7fcf9006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf9006bbc0 0x7fcf9006e080 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8007fd0 tx=0x7fcfa8009040 comp rx=0 tx=0).stop 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 msgr2=0x7fcfb81ecf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 0x7fcfb81ecf70 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000c240 tx=0x7fcfa002e070 comp rx=0 tx=0).stop 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 shutdown_connections 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf9006bbc0 0x7fcf9006e080 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb80ae7b0 0x7fcfb80aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb8070ae0 0x7fcfb81ecf70 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 --2- 172.21.15.2:0/3813582130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf980047c0 0x7fcfb81f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 >> 172.21.15.2:0/3813582130 conn(0x7fcfb80b0840 msgr2=0x7fcfb80cf480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 shutdown_connections 2024-01-15T18:05:56.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.058+0000 7fcfbca00640 1 -- 172.21.15.2:0/3813582130 wait complete. 2024-01-15T18:05:56.082 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:05:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.291+0000 7f35cfe00640 1 Processor -- start 2024-01-15T18:05:56.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.291+0000 7f35cfe00640 1 -- start start 2024-01-15T18:05:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.292+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 0x7f35d006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.292+0000 7f35cfe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d0071660 con 0x7f35d00c92a0 2024-01-15T18:05:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.292+0000 7f35cfe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35d00ca810 con 0x7f35d0159e60 2024-01-15T18:05:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.292+0000 7f35cfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d006e520 con 0x7f35d006db30 2024-01-15T18:05:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.293+0000 7f35c7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f35d00c92a0 0x7f35d00ca100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33674/0 (socket says 172.21.15.2:33674) 2024-01-15T18:05:56.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.293+0000 7f35c7e00640 1 -- 172.21.15.2:0/3510761021 learned_addr learned my addr 172.21.15.2:0/3510761021 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.293+0000 7f35cf400640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 0x7f35d006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.293+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245848118 0 0) 0x7f35d0071660 con 0x7f35d00c92a0 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.293+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b0003520 con 0x7f35d00c92a0 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2837592104 0 0) 0x7f35b0003520 con 0x7f35d00c92a0 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 msgr2=0x7f35d006dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 0x7f35d006dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 >> v1:172.21.15.137:6789/0 conn(0x7f35d0159e60 legacy=0x7f35d0071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d015a240 con 0x7f35d00c92a0 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f35b8001670 con 0x7f35d00c92a0 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 >> v1:172.21.15.2:6789/0 conn(0x7f35d00c92a0 legacy=0x7f35d00ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b00039a0 0x7f35b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35b00047c0 0x7f35b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 0x7f35b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d006e520 con 0x7f35b00047c0 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b000ba60 con 0x7f35b00039a0 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b000de30 con 0x7f35d006db30 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35c7e00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35b00047c0 0x7f35b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35cea00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b00039a0 0x7f35b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35c7e00640 1 -- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 msgr2=0x7f35b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:56.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35c7e00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35d006db30 0x7f35b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35c7e00640 1 -- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b00039a0 msgr2=0x7f35b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:56.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35c7e00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b00039a0 0x7f35b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.294+0000 7f35c7e00640 1 -- 172.21.15.2:0/3510761021 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b0003520 con 0x7f35b00047c0 2024-01-15T18:05:56.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.295+0000 7f35cea00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35b00039a0 0x7f35b0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35c7e00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35b00047c0 0x7f35b0004c40 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f35d00c96c0 tx=0x7f35b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5600b4c0b8b4619b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b8005ad0 con 0x7f35b00047c0 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b8005c30 con 0x7f35b00047c0 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35ce000640 1 -- 172.21.15.2:0/3510761021 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b802c830 con 0x7f35b00047c0 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35cfe00640 1 -- 172.21.15.2:0/3510761021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35b00047c0 msgr2=0x7f35b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3510761021 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35b00047c0 0x7f35b0004c40 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f35d00c96c0 tx=0x7f35b00057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.296+0000 7f35cf400640 1 -- 172.21.15.2:0/3510761021 reap_dead start 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.297+0000 7f35cfe00640 1 -- 172.21.15.2:0/3510761021 shutdown_connections 2024-01-15T18:05:56.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.297+0000 7f35cfe00640 1 -- 172.21.15.2:0/3510761021 >> 172.21.15.2:0/3510761021 conn(0x7f35d00cebc0 msgr2=0x7f35d00cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.297+0000 7f35cfe00640 1 -- 172.21.15.2:0/3510761021 shutdown_connections 2024-01-15T18:05:56.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.297+0000 7f35cfe00640 1 -- 172.21.15.2:0/3510761021 wait complete. 2024-01-15T18:05:56.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.298+0000 7f35cfe00640 1 Processor -- start 2024-01-15T18:05:56.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.299+0000 7f35cfe00640 1 -- start start 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b00047c0 0x7f35d0147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d006ca20 0x7f35d006cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 0x7f35d009c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cfe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0156bd0 con 0x7f35d009bec0 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d006d3e0 con 0x7f35d006ca20 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d0156de0 con 0x7f35b00047c0 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 0x7f35d009c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 0x7f35d009c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54742/0 (socket says 172.21.15.2:54742) 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 -- 172.21.15.2:0/3810054546 learned_addr learned my addr 172.21.15.2:0/3810054546 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cea00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b00047c0 0x7f35d0147480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35c7e00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d006ca20 0x7f35d006cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 -- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b00047c0 msgr2=0x7f35d0147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b00047c0 0x7f35d0147480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 -- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d006ca20 msgr2=0x7f35d006cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d006ca20 0x7f35d006cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35cf400640 1 -- 172.21.15.2:0/3810054546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d01570c0 con 0x7f35d009bec0 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.300+0000 7f35c7e00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d006ca20 0x7f35d006cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.301+0000 7f35cf400640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 0x7f35d009c320 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f35d0067e00 tx=0x7f35c800e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.301+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8013680 con 0x7f35d009bec0 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.301+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35c80137e0 con 0x7f35d009bec0 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.301+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c8013b90 con 0x7f35d009bec0 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.301+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d01573a0 con 0x7f35d009bec0 2024-01-15T18:05:56.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.301+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d01479c0 con 0x7f35d009bec0 2024-01-15T18:05:56.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.302+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d0073b50 con 0x7f35d009bec0 2024-01-15T18:05:56.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.303+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f35c8013cf0 con 0x7f35d009bec0 2024-01-15T18:05:56.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.304+0000 7f35c7400640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f35a806bbf0 0x7f35a806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.304+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f35c8087fe0 con 0x7f35d009bec0 2024-01-15T18:05:56.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.305+0000 7f35cea00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f35a806bbf0 0x7f35a806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.306+0000 7f35cea00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f35a806bbf0 0x7f35a806e0b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35c0001e40 tx=0x7f35c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.308+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f35d0073b50 con 0x7f35d009bec0 2024-01-15T18:05:56.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.501+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f35d0073b50 con 0x7f35d009bec0 2024-01-15T18:05:56.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.502+0000 7f35c7400640 1 -- 172.21.15.2:0/3810054546 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f35d0073b50 con 0x7f35d009bec0 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f35a806bbf0 msgr2=0x7f35a806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f35a806bbf0 0x7f35a806e0b0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f35c0001e40 tx=0x7f35c0009040 comp rx=0 tx=0).stop 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 msgr2=0x7f35d009c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 0x7f35d009c320 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f35d0067e00 tx=0x7f35c800e9f0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 shutdown_connections 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f35a806bbf0 0x7f35a806e0b0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35d009bec0 0x7f35d009c320 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35d006ca20 0x7f35d006cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 --2- 172.21.15.2:0/3810054546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35b00047c0 0x7f35d0147480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.504+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 >> 172.21.15.2:0/3810054546 conn(0x7f35d00cebc0 msgr2=0x7f35d00aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.505+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 shutdown_connections 2024-01-15T18:05:56.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.505+0000 7f35cfe00640 1 -- 172.21.15.2:0/3810054546 wait complete. 2024-01-15T18:05:56.528 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:05:56.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.745+0000 7f0962200640 1 Processor -- start 2024-01-15T18:05:56.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.745+0000 7f0962200640 1 -- start start 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.746+0000 7f0962200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.746+0000 7f0962200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f095c06d000 0x7f095c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.747+0000 7f0962200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0ce230 con 0x7f095c0cdd70 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.747+0000 7f0962200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c0ce510 con 0x7f095c06d000 2024-01-15T18:05:56.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.747+0000 7f0962200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0ce7f0 con 0x7f095c144c00 2024-01-15T18:05:56.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.747+0000 7f095b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.747+0000 7f095b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:43854/0 (socket says 172.21.15.2:43854) 2024-01-15T18:05:56.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.747+0000 7f095b400640 1 -- 172.21.15.2:0/3492317787 learned_addr learned my addr 172.21.15.2:0/3492317787 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:56.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.748+0000 7f095be00640 1 --2- 172.21.15.2:0/3492317787 >> v2:172.21.15.137:3300/0 conn(0x7f095c06d000 0x7f095c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.748+0000 7f095b400640 1 -- 172.21.15.2:0/3492317787 >> v2:172.21.15.137:3300/0 conn(0x7f095c06d000 msgr2=0x7f095c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:56.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.748+0000 7f095b400640 1 --2- 172.21.15.2:0/3492317787 >> v2:172.21.15.137:3300/0 conn(0x7f095c06d000 0x7f095c070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.748+0000 7f095b400640 1 -- 172.21.15.2:0/3492317787 >> v1:172.21.15.137:6789/0 conn(0x7f095c0cdd70 legacy=0x7f095c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.748+0000 7f095aa00640 1 -- 172.21.15.2:0/3492317787 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 62483010 0 0) 0x7f095c0ce230 con 0x7f095c0cdd70 2024-01-15T18:05:56.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.748+0000 7f095b400640 1 -- 172.21.15.2:0/3492317787 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c0705f0 con 0x7f095c144c00 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.750+0000 7f095b400640 1 --2- 172.21.15.2:0/3492317787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0cd830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f094c009870 tx=0x7f094c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=555306d82681e08e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.750+0000 7f095aa00640 1 -- 172.21.15.2:0/3492317787 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c030840 con 0x7f095c144c00 2024-01-15T18:05:56.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.750+0000 7f095aa00640 1 -- 172.21.15.2:0/3492317787 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c0309a0 con 0x7f095c144c00 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.750+0000 7f095aa00640 1 -- 172.21.15.2:0/3492317787 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c030d30 con 0x7f095c144c00 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 -- 172.21.15.2:0/3492317787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 msgr2=0x7f095c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 --2- 172.21.15.2:0/3492317787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0cd830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f094c009870 tx=0x7f094c02b980 comp rx=0 tx=0).stop 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 -- 172.21.15.2:0/3492317787 shutdown_connections 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 --2- 172.21.15.2:0/3492317787 >> v2:172.21.15.137:3300/0 conn(0x7f095c06d000 0x7f095c070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 --2- 172.21.15.2:0/3492317787 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0cd830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 -- 172.21.15.2:0/3492317787 >> 172.21.15.2:0/3492317787 conn(0x7f095c0b45f0 msgr2=0x7f095c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 -- 172.21.15.2:0/3492317787 shutdown_connections 2024-01-15T18:05:56.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.751+0000 7f0962200640 1 -- 172.21.15.2:0/3492317787 wait complete. 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.753+0000 7f0962200640 1 Processor -- start 2024-01-15T18:05:56.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.753+0000 7f0962200640 1 -- start start 2024-01-15T18:05:56.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0962200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f095c06d000 0x7f095c13c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0962200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 0x7f095c0a9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0962200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0962200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0af9e0 con 0x7f095c0cdd70 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0962200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c1598f0 con 0x7f095c06d000 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0962200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0b1830 con 0x7f095c144c00 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0953e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 0x7f095c0a9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0953e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 0x7f095c0a9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54746/0 (socket says 172.21.15.2:54746) 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.754+0000 7f0953e00640 1 -- 172.21.15.2:0/2978053998 learned_addr learned my addr 172.21.15.2:0/2978053998 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f095be00640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f095b400640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f095c06d000 0x7f095c13c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953e00640 1 -- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 msgr2=0x7f095c0711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953e00640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0711d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953e00640 1 -- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f095c06d000 msgr2=0x7f095c13c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953e00640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f095c06d000 0x7f095c13c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953e00640 1 -- 172.21.15.2:0/2978053998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094c0094d0 con 0x7f095c0cdd70 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f095b400640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f095c06d000 0x7f095c13c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953e00640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 0x7f095c0a9e40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f094400c4b0 tx=0x7f094400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.755+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944005480 con 0x7f095c0cdd70 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.756+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09440035f0 con 0x7f095c0cdd70 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.756+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c0afc30 con 0x7f095c0cdd70 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.756+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09440039e0 con 0x7f095c0cdd70 2024-01-15T18:05:56.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.756+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c0b01f0 con 0x7f095c0cdd70 2024-01-15T18:05:56.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.757+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c0b0410 con 0x7f095c0cdd70 2024-01-15T18:05:56.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.758+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f0944003b40 con 0x7f095c0cdd70 2024-01-15T18:05:56.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.759+0000 7f0953400640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f093406bbc0 0x7f093406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:56.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.759+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f0944087a50 con 0x7f095c0cdd70 2024-01-15T18:05:56.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.759+0000 7f095b400640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f093406bbc0 0x7f093406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:56.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.761+0000 7f095b400640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f093406bbc0 0x7f093406e080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f094c036000 tx=0x7f094c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:56.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.766+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0944054350 con 0x7f095c0cdd70 2024-01-15T18:05:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:56.986+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f095c0686f0 con 0x7f093406bbc0 2024-01-15T18:05:57.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:56.997+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:57.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:56.997+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:57.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:56.997+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:57.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:56.997+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:57.002 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:05:56.997+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:05:57.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.062+0000 7f0953400640 1 -- 172.21.15.2:0/2978053998 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f095c0686f0 con 0x7f093406bbc0 2024-01-15T18:05:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f093406bbc0 msgr2=0x7f093406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f093406bbc0 0x7f093406e080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f094c036000 tx=0x7f094c03b040 comp rx=0 tx=0).stop 2024-01-15T18:05:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 msgr2=0x7f095c0a9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 0x7f095c0a9e40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f094400c4b0 tx=0x7f094400c970 comp rx=0 tx=0).stop 2024-01-15T18:05:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 shutdown_connections 2024-01-15T18:05:57.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f093406bbc0 0x7f093406e080 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f095c144c00 0x7f095c0711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f095c0cdd70 0x7f095c0a9e40 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 --2- 172.21.15.2:0/2978053998 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f095c06d000 0x7f095c13c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 >> 172.21.15.2:0/2978053998 conn(0x7f095c0b45f0 msgr2=0x7f095c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 shutdown_connections 2024-01-15T18:05:57.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.065+0000 7f0962200640 1 -- 172.21.15.2:0/2978053998 wait complete. 2024-01-15T18:05:57.090 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:05:57.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.309+0000 7f1b3be00640 1 Processor -- start 2024-01-15T18:05:57.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.309+0000 7f1b3be00640 1 -- start start 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b3c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1b3c1447d0 0x7f1b3c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3be00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c071090 con 0x7f1b3c0720d0 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c071430 con 0x7f1b3c1447d0 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c144100 con 0x7f1b3c143d20 2024-01-15T18:05:57.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3a000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1b3c0720d0 0x7f1b3c0cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33688/0 (socket says 172.21.15.2:33688) 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3a000640 1 -- 172.21.15.2:0/3653083125 learned_addr learned my addr 172.21.15.2:0/3653083125 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3b400640 1 --2- 172.21.15.2:0/3653083125 >> v2:172.21.15.137:3300/0 conn(0x7f1b3c1447d0 0x7f1b3c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.311+0000 7f1b3aa00640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b3c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798087508 0 0) 0x7f1b3c071090 con 0x7f1b3c0720d0 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b1c003520 con 0x7f1b3c0720d0 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3322566936 0 0) 0x7f1b1c003520 con 0x7f1b3c0720d0 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 msgr2=0x7f1b3c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b3c15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 >> v2:172.21.15.137:3300/0 conn(0x7f1b3c1447d0 msgr2=0x7f1b3c15f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 --2- 172.21.15.2:0/3653083125 >> v2:172.21.15.137:3300/0 conn(0x7f1b3c1447d0 0x7f1b3c15f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c06d160 con 0x7f1b3c0720d0 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1b24001670 con 0x7f1b3c0720d0 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 >> v1:172.21.15.2:6789/0 conn(0x7f1b3c0720d0 legacy=0x7f1b3c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1c003970 0x7f1b1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c071430 con 0x7f1b1c0047c0 2024-01-15T18:05:57.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c144100 con 0x7f1b1c003970 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c00b980 con 0x7f1b3c143d20 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.312+0000 7f1b3a000640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3a000640 1 -- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 msgr2=0x7f1b1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3aa00640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1c003970 0x7f1b1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3b400640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3a000640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b1c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3a000640 1 -- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1c003970 msgr2=0x7f1b1c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3a000640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1c003970 0x7f1b1c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3a000640 1 -- 172.21.15.2:0/3653083125 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b1c003520 con 0x7f1b1c0047c0 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3b400640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c143d20 0x7f1b1c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:57.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.313+0000 7f1b3aa00640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1c003970 0x7f1b1c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.314+0000 7f1b3a000640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c270 tx=0x7f1b1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62c71337ec62a21b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.314+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2402cae0 con 0x7f1b1c0047c0 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.314+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2402cc40 con 0x7f1b1c0047c0 2024-01-15T18:05:57.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.314+0000 7f1b39600640 1 -- 172.21.15.2:0/3653083125 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240058d0 con 0x7f1b1c0047c0 2024-01-15T18:05:57.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3be00640 1 -- 172.21.15.2:0/3653083125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 msgr2=0x7f1b1c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3be00640 1 --2- 172.21.15.2:0/3653083125 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b1c004c40 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c270 tx=0x7f1b1c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3b400640 1 -- 172.21.15.2:0/3653083125 reap_dead start 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3be00640 1 -- 172.21.15.2:0/3653083125 shutdown_connections 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3be00640 1 -- 172.21.15.2:0/3653083125 >> 172.21.15.2:0/3653083125 conn(0x7f1b3c0b0840 msgr2=0x7f1b3c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3be00640 1 -- 172.21.15.2:0/3653083125 shutdown_connections 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.315+0000 7f1b3be00640 1 -- 172.21.15.2:0/3653083125 wait complete. 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.317+0000 7f1b3be00640 1 Processor -- start 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.317+0000 7f1b3be00640 1 -- start start 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b3c0cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b3c1f32b0 0x7f1b3c1342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c134810 0x7f1b3c15da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c15af80 con 0x7f1b1c0047c0 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0cbcc0 con 0x7f1b3c1f32b0 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c15c960 con 0x7f1b3c134810 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b3c0cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b3c0cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54766/0 (socket says 172.21.15.2:54766) 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.318+0000 7f1b3aa00640 1 -- 172.21.15.2:0/771387506 learned_addr learned my addr 172.21.15.2:0/771387506 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3b400640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c134810 0x7f1b3c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3aa00640 1 -- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c134810 msgr2=0x7f1b3c15da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3aa00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c134810 0x7f1b3c15da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3a000640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b3c1f32b0 0x7f1b3c1342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3aa00640 1 -- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b3c1f32b0 msgr2=0x7f1b3c1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3aa00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b3c1f32b0 0x7f1b3c1342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3aa00640 1 -- 172.21.15.2:0/771387506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c15cc40 con 0x7f1b1c0047c0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3b400640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c134810 0x7f1b3c15da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3aa00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b3c0cb780 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0108b0 tx=0x7f1b2c010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00eb60 con 0x7f1b1c0047c0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2c00ecc0 con 0x7f1b1c0047c0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.319+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c15cf20 con 0x7f1b1c0047c0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.320+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c01c400 con 0x7f1b1c0047c0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.320+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c15d4e0 con 0x7f1b1c0047c0 2024-01-15T18:05:57.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.321+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c074770 con 0x7f1b1c0047c0 2024-01-15T18:05:57.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.322+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1b2c01c560 con 0x7f1b1c0047c0 2024-01-15T18:05:57.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.323+0000 7f1b32a00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1406bbc0 0x7f1b1406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:05:57.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.323+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1b2c088a30 con 0x7f1b1c0047c0 2024-01-15T18:05:57.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.323+0000 7f1b3a000640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1406bbc0 0x7f1b1406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:05:57.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.324+0000 7f1b3a000640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1406bbc0 0x7f1b1406e080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1b24034000 tx=0x7f1b24007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:05:57.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.327+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1b2c0553a0 con 0x7f1b1c0047c0 2024-01-15T18:05:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.559+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b3c15f720 con 0x7f1b1406bbc0 2024-01-15T18:05:57.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.571+0000 7f1b32a00640 1 -- 172.21.15.2:0/771387506 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1b2402c8e0 con 0x7f1b1406bbc0 2024-01-15T18:05:57.576 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25 2024-01-15T18:05:57.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.573+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1406bbc0 msgr2=0x7f1b1406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.573+0000 7f1b3be00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1406bbc0 0x7f1b1406e080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1b24034000 tx=0x7f1b24007000 comp rx=0 tx=0).stop 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.573+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 msgr2=0x7f1b3c0cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.573+0000 7f1b3be00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b3c0cb780 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0108b0 tx=0x7f1b2c010d70 comp rx=0 tx=0).stop 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 shutdown_connections 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b1406bbc0 0x7f1b1406e080 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b3c134810 0x7f1b3c15da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b3c1f32b0 0x7f1b3c1342d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 --2- 172.21.15.2:0/771387506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1c0047c0 0x7f1b3c0cb780 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 >> 172.21.15.2:0/771387506 conn(0x7f1b3c0b0840 msgr2=0x7f1b3c15b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 shutdown_connections 2024-01-15T18:05:57.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:05:57.574+0000 7f1b3be00640 1 -- 172.21.15.2:0/771387506 wait complete. 2024-01-15T18:05:57.599 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25 2024-01-15T18:05:57.599 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:05:57.599 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:05:57.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:05:57.655 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:06:00.999 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:01.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:13.839 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:06:13.839 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:06:13.839 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.101 s, 66.7 MB/s 2024-01-15T18:06:13.841 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:06:13.841 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:06:13.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:13.896 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:06:28.937 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:06:28.937 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:06:28.937 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.9802 s, 71.7 MB/s 2024-01-15T18:06:28.940 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:06:28.940 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:06:28.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:28.994 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:06:31.143 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:31.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:06:43.797 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:06:43.798 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:06:43.798 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.7434 s, 72.8 MB/s 2024-01-15T18:06:43.800 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 10 2024-01-15T18:06:44.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.025+0000 7f883ec00640 1 Processor -- start 2024-01-15T18:06:44.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.026+0000 7f883ec00640 1 -- start start 2024-01-15T18:06:44.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.027+0000 7f883ec00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f883815f260 0x7f883806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.027+0000 7f883ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f883806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.027+0000 7f883ec00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8838071090 con 0x7f8838143d20 2024-01-15T18:06:44.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.027+0000 7f883ec00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8838071430 con 0x7f883815f260 2024-01-15T18:06:44.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.027+0000 7f883ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8838144100 con 0x7f88380720d0 2024-01-15T18:06:44.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.028+0000 7f883e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f883806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.028+0000 7f883e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f883806c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:44244/0 (socket says 172.21.15.2:44244) 2024-01-15T18:06:44.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.028+0000 7f883e200640 1 -- 172.21.15.2:0/3276250977 learned_addr learned my addr 172.21.15.2:0/3276250977 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:44.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.028+0000 7f883ce00640 1 --2- 172.21.15.2:0/3276250977 >> v2:172.21.15.137:3300/0 conn(0x7f883815f260 0x7f883806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.029+0000 7f883e200640 1 -- 172.21.15.2:0/3276250977 >> v2:172.21.15.137:3300/0 conn(0x7f883815f260 msgr2=0x7f883806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:44.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.029+0000 7f883e200640 1 --2- 172.21.15.2:0/3276250977 >> v2:172.21.15.137:3300/0 conn(0x7f883815f260 0x7f883806c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.029+0000 7f883e200640 1 -- 172.21.15.2:0/3276250977 >> v1:172.21.15.137:6789/0 conn(0x7f8838143d20 legacy=0x7f883815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.029+0000 7f883e200640 1 -- 172.21.15.2:0/3276250977 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8838144560 con 0x7f88380720d0 2024-01-15T18:06:44.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.029+0000 7f883ce00640 1 --2- 172.21.15.2:0/3276250977 >> v2:172.21.15.137:3300/0 conn(0x7f883815f260 0x7f883806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:44.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.030+0000 7f883e200640 1 --2- 172.21.15.2:0/3276250977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f883806c690 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f883002be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9edf2ca6205e8a23 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.031+0000 7f8837e00640 1 -- 172.21.15.2:0/3276250977 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883002c9f0 con 0x7f88380720d0 2024-01-15T18:06:44.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.031+0000 7f8837e00640 1 -- 172.21.15.2:0/3276250977 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883002cb50 con 0x7f88380720d0 2024-01-15T18:06:44.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.031+0000 7f8837e00640 1 -- 172.21.15.2:0/3276250977 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88300343f0 con 0x7f88380720d0 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.031+0000 7f883ec00640 1 -- 172.21.15.2:0/3276250977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 msgr2=0x7f883806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.031+0000 7f883ec00640 1 --2- 172.21.15.2:0/3276250977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f883806c690 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f883002be50 comp rx=0 tx=0).stop 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.032+0000 7f883ec00640 1 -- 172.21.15.2:0/3276250977 shutdown_connections 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.032+0000 7f883ec00640 1 --2- 172.21.15.2:0/3276250977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f883806c690 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.032+0000 7f883ec00640 1 --2- 172.21.15.2:0/3276250977 >> v2:172.21.15.137:3300/0 conn(0x7f883815f260 0x7f883806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.032+0000 7f883ec00640 1 -- 172.21.15.2:0/3276250977 >> 172.21.15.2:0/3276250977 conn(0x7f88380b0840 msgr2=0x7f88380b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.032+0000 7f883ec00640 1 -- 172.21.15.2:0/3276250977 shutdown_connections 2024-01-15T18:06:44.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.032+0000 7f883ec00640 1 -- 172.21.15.2:0/3276250977 wait complete. 2024-01-15T18:06:44.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.033+0000 7f883ec00640 1 Processor -- start 2024-01-15T18:06:44.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.034+0000 7f883ec00640 1 -- start start 2024-01-15T18:06:44.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.034+0000 7f883ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f88381f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8838143d20 0x7f8838135340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883ec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 0x7f8838154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883ec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380b26f0 con 0x7f883815f260 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88380b1dc0 con 0x7f8838143d20 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88380b20a0 con 0x7f88380720d0 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 0x7f8838154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 0x7f8838154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51056/0 (socket says 172.21.15.2:51056) 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 -- 172.21.15.2:0/2049003084 learned_addr learned my addr 172.21.15.2:0/2049003084 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883d800640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f88381f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883ce00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8838143d20 0x7f8838135340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 -- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 msgr2=0x7f88381f2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f88381f2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 -- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8838143d20 msgr2=0x7f8838135340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.035+0000 7f883e200640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8838143d20 0x7f8838135340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f883e200640 1 -- 172.21.15.2:0/2049003084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8820009400 con 0x7f883815f260 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f883e200640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 0x7f8838154550 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f8830004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8830009510 con 0x7f883815f260 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f883003e450 con 0x7f883815f260 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8830009b30 con 0x7f883815f260 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f883003e800 con 0x7f883815f260 2024-01-15T18:06:44.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.036+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88381f32e0 con 0x7f883815f260 2024-01-15T18:06:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.038+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88380b2860 con 0x7f883815f260 2024-01-15T18:06:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.039+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f883003e960 con 0x7f883815f260 2024-01-15T18:06:44.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.039+0000 7f8835600640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f881006bc50 0x7f881006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.040+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f88300ab3d0 con 0x7f883815f260 2024-01-15T18:06:44.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.040+0000 7f883d800640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f881006bc50 0x7f881006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.041+0000 7f883d800640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f881006bc50 0x7f881006e110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8828000f60 tx=0x7f8828009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.044+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88300b3020 con 0x7f883815f260 2024-01-15T18:06:44.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.237+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f8838074770 con 0x7f883815f260 2024-01-15T18:06:44.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.245+0000 7f8835600640 1 -- 172.21.15.2:0/2049003084 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f8830077a90 con 0x7f883815f260 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f881006bc50 msgr2=0x7f881006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f881006bc50 0x7f881006e110 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8828000f60 tx=0x7f8828009040 comp rx=0 tx=0).stop 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 msgr2=0x7f8838154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 0x7f8838154550 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f8830009e50 tx=0x7f8830004a40 comp rx=0 tx=0).stop 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 shutdown_connections 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f881006bc50 0x7f881006e110 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f883815f260 0x7f8838154550 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8838143d20 0x7f8838135340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 --2- 172.21.15.2:0/2049003084 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88380720d0 0x7f88381f2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 >> 172.21.15.2:0/2049003084 conn(0x7f88380b0840 msgr2=0x7f883815b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:44.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 shutdown_connections 2024-01-15T18:06:44.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.248+0000 7f883ec00640 1 -- 172.21.15.2:0/2049003084 wait complete. 2024-01-15T18:06:44.273 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:06:44.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.499+0000 7f1645800640 1 Processor -- start 2024-01-15T18:06:44.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.500+0000 7f1645800640 1 -- start start 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.501+0000 7f1645800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 0x7f16400cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.501+0000 7f1645800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f16400cdd70 0x7f164006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.501+0000 7f1645800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16400ce230 con 0x7f164015c450 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.501+0000 7f1645800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16400ce510 con 0x7f16400cdd70 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.501+0000 7f1645800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16400ce7f0 con 0x7f1640144c00 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.502+0000 7f163ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 0x7f16400cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.502+0000 7f163e000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f16400cdd70 0x7f164006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.502+0000 7f163ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 0x7f16400cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:44266/0 (socket says 172.21.15.2:44266) 2024-01-15T18:06:44.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.502+0000 7f163ea00640 1 -- 172.21.15.2:0/4103708018 learned_addr learned my addr 172.21.15.2:0/4103708018 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:44.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.503+0000 7f163e000640 1 -- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 msgr2=0x7f16400cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:44.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.503+0000 7f163e000640 1 --2- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 0x7f16400cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.503+0000 7f163e000640 1 -- 172.21.15.2:0/4103708018 >> v1:172.21.15.137:6789/0 conn(0x7f164015c450 legacy=0x7f16400b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.503+0000 7f163e000640 1 -- 172.21.15.2:0/4103708018 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16400705f0 con 0x7f16400cdd70 2024-01-15T18:06:44.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.503+0000 7f163ea00640 1 --2- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 0x7f16400cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:06:44.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.505+0000 7f163e000640 1 --2- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400cdd70 0x7f164006c8f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f16280097a0 tx=0x7f162802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66d16780e0805e5a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.505+0000 7f163d600640 1 -- 172.21.15.2:0/4103708018 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f162802d040 con 0x7f16400cdd70 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.505+0000 7f163d600640 1 -- 172.21.15.2:0/4103708018 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1628031810 con 0x7f16400cdd70 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.505+0000 7f163d600640 1 -- 172.21.15.2:0/4103708018 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628031ba0 con 0x7f16400cdd70 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 -- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400cdd70 msgr2=0x7f164006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 --2- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400cdd70 0x7f164006c8f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f16280097a0 tx=0x7f162802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 -- 172.21.15.2:0/4103708018 shutdown_connections 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 --2- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f16400cdd70 0x7f164006c8f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 --2- 172.21.15.2:0/4103708018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1640144c00 0x7f16400cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 -- 172.21.15.2:0/4103708018 >> 172.21.15.2:0/4103708018 conn(0x7f16400b45f0 msgr2=0x7f16400b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 -- 172.21.15.2:0/4103708018 shutdown_connections 2024-01-15T18:06:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.506+0000 7f1645800640 1 -- 172.21.15.2:0/4103708018 wait complete. 2024-01-15T18:06:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.508+0000 7f1645800640 1 Processor -- start 2024-01-15T18:06:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.509+0000 7f1645800640 1 -- start start 2024-01-15T18:06:44.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f1645800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 0x7f16401e59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f1645800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640144c00 0x7f164013ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f1645800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164015c450 0x7f16400b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f1645800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16401e72e0 con 0x7f16400cdd70 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f1645800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16401e6240 con 0x7f1640144c00 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f1645800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16401e6520 con 0x7f164015c450 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f163ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 0x7f16401e59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.510+0000 7f163ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 0x7f16401e59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51068/0 (socket says 172.21.15.2:51068) 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 -- 172.21.15.2:0/1079801066 learned_addr learned my addr 172.21.15.2:0/1079801066 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163e000640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640144c00 0x7f164013ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163f400640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164015c450 0x7f16400b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 -- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164015c450 msgr2=0x7f16400b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164015c450 0x7f16400b0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 -- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640144c00 msgr2=0x7f164013ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640144c00 0x7f164013ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 -- 172.21.15.2:0/1079801066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16300094d0 con 0x7f16400cdd70 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.511+0000 7f163ea00640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 0x7f16401e59f0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f163000e9d0 tx=0x7f163000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.512+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163000ca60 con 0x7f16400cdd70 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.512+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163000cbc0 con 0x7f16400cdd70 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.512+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1630013470 con 0x7f16400cdd70 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.512+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1628009400 con 0x7f16400cdd70 2024-01-15T18:06:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.512+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16401e6ae0 con 0x7f16400cdd70 2024-01-15T18:06:44.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.513+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16400686f0 con 0x7f16400cdd70 2024-01-15T18:06:44.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.514+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f16300135d0 con 0x7f16400cdd70 2024-01-15T18:06:44.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.515+0000 7f1636a00640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f161806bc50 0x7f161806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:44.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.515+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f1630088a20 con 0x7f16400cdd70 2024-01-15T18:06:44.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.516+0000 7f163e000640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f161806bc50 0x7f161806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.517+0000 7f163e000640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f161806bc50 0x7f161806e110 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1628036000 tx=0x7f162803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:44.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.519+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16300550e0 con 0x7f16400cdd70 2024-01-15T18:06:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.746+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f16401e6690 con 0x7f161806bc50 2024-01-15T18:06:44.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.780+0000 7f1636a00640 1 -- 172.21.15.2:0/1079801066 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f16401e6690 con 0x7f161806bc50 2024-01-15T18:06:44.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f161806bc50 msgr2=0x7f161806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f161806bc50 0x7f161806e110 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1628036000 tx=0x7f162803c040 comp rx=0 tx=0).stop 2024-01-15T18:06:44.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 msgr2=0x7f16401e59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 0x7f16401e59f0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f163000e9d0 tx=0x7f163000ee90 comp rx=0 tx=0).stop 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 shutdown_connections 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f161806bc50 0x7f161806e110 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f164015c450 0x7f16400b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1640144c00 0x7f164013ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 --2- 172.21.15.2:0/1079801066 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f16400cdd70 0x7f16401e59f0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 >> 172.21.15.2:0/1079801066 conn(0x7f16400b45f0 msgr2=0x7f16401406d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:44.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 shutdown_connections 2024-01-15T18:06:44.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:44.783+0000 7f1645800640 1 -- 172.21.15.2:0/1079801066 wait complete. 2024-01-15T18:06:44.806 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:06:45.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.025+0000 7fa91e000640 1 Processor -- start 2024-01-15T18:06:45.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.026+0000 7fa91e000640 1 -- start start 2024-01-15T18:06:45.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.027+0000 7fa91e000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa91815c130 0x7fa9180b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.027+0000 7fa91e000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91806a4b0 con 0x7fa9180cf1d0 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.027+0000 7fa91e000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa91815ed80 con 0x7fa9180b2c60 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.027+0000 7fa91e000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa918155b20 con 0x7fa91815c130 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.028+0000 7fa917e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa9180cf1d0 0x7fa9180b4670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45858/0 (socket says 172.21.15.2:45858) 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.028+0000 7fa917e00640 1 -- 172.21.15.2:0/2569763905 learned_addr learned my addr 172.21.15.2:0/2569763905 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.028+0000 7fa917400640 1 --2- 172.21.15.2:0/2569763905 >> v2:172.21.15.137:3300/0 conn(0x7fa91815c130 0x7fa9180b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.028+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477357934 0 0) 0x7fa91806a4b0 con 0x7fa9180cf1d0 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.028+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f8003520 con 0x7fa9180cf1d0 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3155231718 0 0) 0x7fa8f8003520 con 0x7fa9180cf1d0 2024-01-15T18:06:45.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 >> v2:172.21.15.137:3300/0 conn(0x7fa91815c130 msgr2=0x7fa9180b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 --2- 172.21.15.2:0/2569763905 >> v2:172.21.15.137:3300/0 conn(0x7fa91815c130 0x7fa9180b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 >> v1:172.21.15.137:6789/0 conn(0x7fa9180b2c60 legacy=0x7fa9180ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91815f060 con 0x7fa9180cf1d0 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fa910001670 con 0x7fa9180cf1d0 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 >> v1:172.21.15.2:6789/0 conn(0x7fa9180cf1d0 legacy=0x7fa9180b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8f80039c0 0x7fa8f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8f80047c0 0x7fa8f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80051d0 0x7fa8f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa918155b20 con 0x7fa8f80039c0 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f800c120 con 0x7fa8f80047c0 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f800c2e0 con 0x7fa8f80051d0 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.029+0000 7fa917400640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8f80039c0 0x7fa8f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa90e800640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8f80047c0 0x7fa8f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa917400640 1 -- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80051d0 msgr2=0x7fa8f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa917400640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80051d0 0x7fa8f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa917400640 1 -- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8f80047c0 msgr2=0x7fa8f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:45.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa917400640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa8f80047c0 0x7fa8f8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa917400640 1 -- 172.21.15.2:0/2569763905 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f8003520 con 0x7fa8f80039c0 2024-01-15T18:06:45.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.030+0000 7fa917e00640 1 -- 172.21.15.2:0/2569763905 reap_dead start 2024-01-15T18:06:45.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.031+0000 7fa917400640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8f80039c0 0x7fa8f8003de0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa908009f90 tx=0x7fa8f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17c04a3e4393cb95 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.031+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90802cbd0 con 0x7fa8f80039c0 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.031+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90800f040 con 0x7fa8f80039c0 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.031+0000 7fa916a00640 1 -- 172.21.15.2:0/2569763905 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa908034440 con 0x7fa8f80039c0 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 -- 172.21.15.2:0/2569763905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8f80039c0 msgr2=0x7fa8f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8f80039c0 0x7fa8f8003de0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa908009f90 tx=0x7fa8f80057f0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 -- 172.21.15.2:0/2569763905 shutdown_connections 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 --2- 172.21.15.2:0/2569763905 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8f80039c0 0x7fa8f8003de0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 -- 172.21.15.2:0/2569763905 >> 172.21.15.2:0/2569763905 conn(0x7fa9180cc170 msgr2=0x7fa9180cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 -- 172.21.15.2:0/2569763905 shutdown_connections 2024-01-15T18:06:45.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.032+0000 7fa91e000640 1 -- 172.21.15.2:0/2569763905 wait complete. 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.034+0000 7fa91e000640 1 Processor -- start 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.034+0000 7fa91e000640 1 -- start start 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa91e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80039c0 0x7fa918145b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa91e000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa91806c920 0x7fa9180ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa91e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 0x7fa9180abe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa91e000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa918146040 con 0x7fa91806cda0 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa91e000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa91806d300 con 0x7fa91806c920 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa91e000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91806d5e0 con 0x7fa8f80039c0 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa917e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 0x7fa9180abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa917e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 0x7fa9180abe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51084/0 (socket says 172.21.15.2:51084) 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.035+0000 7fa917e00640 1 -- 172.21.15.2:0/147342992 learned_addr learned my addr 172.21.15.2:0/147342992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa90e800640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa91806c920 0x7fa9180ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917400640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80039c0 0x7fa918145b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917e00640 1 -- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80039c0 msgr2=0x7fa918145b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917e00640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80039c0 0x7fa918145b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917e00640 1 -- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa91806c920 msgr2=0x7fa9180ab920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917e00640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa91806c920 0x7fa9180ab920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917e00640 1 -- 172.21.15.2:0/147342992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa91806d8c0 con 0x7fa91806cda0 2024-01-15T18:06:45.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa90e800640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa91806c920 0x7fa9180ab920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:45.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa917e00640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 0x7fa9180abe60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa9180cd270 tx=0x7fa91000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:45.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9100074e0 con 0x7fa91806cda0 2024-01-15T18:06:45.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa910007640 con 0x7fa91806cda0 2024-01-15T18:06:45.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa910004a20 con 0x7fa91806cda0 2024-01-15T18:06:45.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.036+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa91806dba0 con 0x7fa91806cda0 2024-01-15T18:06:45.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.037+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa91814c560 con 0x7fa91806cda0 2024-01-15T18:06:45.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.038+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa91806e0c0 con 0x7fa91806cda0 2024-01-15T18:06:45.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.039+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fa910004b80 con 0x7fa91806cda0 2024-01-15T18:06:45.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.040+0000 7fa90fe00640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa8f006bc50 0x7fa8f006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:45.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.040+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fa910087c30 con 0x7fa91806cda0 2024-01-15T18:06:45.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.040+0000 7fa917400640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa8f006bc50 0x7fa8f006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:45.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.041+0000 7fa917400640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa8f006bc50 0x7fa8f006e110 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa908036000 tx=0x7fa90802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:45.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.044+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa91806e0c0 con 0x7fa91806cda0 2024-01-15T18:06:45.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.267+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa91814c780 con 0x7fa8f006bc50 2024-01-15T18:06:45.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.279+0000 7fa90fe00640 1 -- 172.21.15.2:0/147342992 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa91814c780 con 0x7fa8f006bc50 2024-01-15T18:06:45.284 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25 2024-01-15T18:06:45.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa8f006bc50 msgr2=0x7fa8f006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa8f006bc50 0x7fa8f006e110 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fa908036000 tx=0x7fa90802e000 comp rx=0 tx=0).stop 2024-01-15T18:06:45.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 msgr2=0x7fa9180abe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 0x7fa9180abe60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fa9180cd270 tx=0x7fa91000c960 comp rx=0 tx=0).stop 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 shutdown_connections 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa8f006bc50 0x7fa8f006e110 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.281+0000 7fa91e000640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa91806cda0 0x7fa9180abe60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.282+0000 7fa91e000640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa91806c920 0x7fa9180ab920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.282+0000 7fa91e000640 1 --2- 172.21.15.2:0/147342992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8f80039c0 0x7fa918145b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.282+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 >> 172.21.15.2:0/147342992 conn(0x7fa9180cc170 msgr2=0x7fa9180ce8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.282+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 shutdown_connections 2024-01-15T18:06:45.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:45.282+0000 7fa91e000640 1 -- 172.21.15.2:0/147342992 wait complete. 2024-01-15T18:06:45.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:45.307 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25) 2024-01-15T18:06:45.373 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25 2024-01-15T18:06:45.373 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:06:45.373 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:06:45.374 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:06:47.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:47.376 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25) 2024-01-15T18:06:47.445 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25 2024-01-15T18:06:47.445 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:06:47.445 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:06:47.446 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:06:49.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:06:49.448 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25) 2024-01-15T18:06:49.516 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/bfa2739d-eafa-45cb-a7f5-a017ba0afd25 2024-01-15T18:06:49.516 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:06:49.516 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:06:49.517 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:06:49.517 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47161 2024-01-15T18:06:49.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.748+0000 7f676b400640 1 Processor -- start 2024-01-15T18:06:49.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.749+0000 7f676b400640 1 -- start start 2024-01-15T18:06:49.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.750+0000 7f676b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 0x7f6764164750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.750+0000 7f676b400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6764071250 con 0x7f6764150890 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.750+0000 7f676b400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6764071530 con 0x7f67640b1290 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.750+0000 7f676b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67640b2ae0 con 0x7f6764070c70 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.750+0000 7f6768a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6764150890 0x7f67640b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45874/0 (socket says 172.21.15.2:45874) 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6768a00640 1 -- 172.21.15.2:0/1900366099 learned_addr learned my addr 172.21.15.2:0/1900366099 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6769400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 0x7f6764164750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097457082 0 0) 0x7f6764071250 con 0x7f6764150890 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6744003440 con 0x7f6764150890 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766372911 0 0) 0x7f6744003440 con 0x7f6764150890 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 msgr2=0x7f6764164750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 0x7f6764164750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 >> v1:172.21.15.137:6789/0 conn(0x7f67640b1290 legacy=0x7f6764070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67640b2dc0 con 0x7f6764150890 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.751+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6754001670 con 0x7f6764150890 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 >> v1:172.21.15.2:6789/0 conn(0x7f6764150890 legacy=0x7f67640b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 0x7f6744003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f674400a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f674400aff0 0x7f674400d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67640b2ae0 con 0x7f6744004060 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674400dd60 con 0x7f674400aff0 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674400df20 con 0x7f6764070c70 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763e00640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f674400a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6769400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f674400aff0 0x7f674400d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:49.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6768a00640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 0x7f6744003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:49.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763e00640 1 -- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 msgr2=0x7f6744003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:49.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763e00640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6764070c70 0x7f6744003a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763e00640 1 -- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f674400aff0 msgr2=0x7f674400d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:49.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.752+0000 7f6763e00640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f674400aff0 0x7f674400d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.753+0000 7f6763e00640 1 -- 172.21.15.2:0/1900366099 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744003440 con 0x7f6744004060 2024-01-15T18:06:49.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.753+0000 7f6769400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f674400aff0 0x7f674400d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.754+0000 7f6763e00640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f674400a860 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f674c001e50 tx=0x7f67440036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ff811d58ee7adb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.754+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c003b70 con 0x7f6744004060 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.754+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674c003cd0 con 0x7f6744004060 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.754+0000 7f6763400640 1 -- 172.21.15.2:0/1900366099 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674c005310 con 0x7f6744004060 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f676b400640 1 -- 172.21.15.2:0/1900366099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 msgr2=0x7f674400a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f676b400640 1 --2- 172.21.15.2:0/1900366099 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f674400a860 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f674c001e50 tx=0x7f67440036f0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f6769400640 1 -- 172.21.15.2:0/1900366099 reap_dead start 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f676b400640 1 -- 172.21.15.2:0/1900366099 shutdown_connections 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f676b400640 1 -- 172.21.15.2:0/1900366099 >> 172.21.15.2:0/1900366099 conn(0x7f67640b45f0 msgr2=0x7f67640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f676b400640 1 -- 172.21.15.2:0/1900366099 shutdown_connections 2024-01-15T18:06:49.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.755+0000 7f676b400640 1 -- 172.21.15.2:0/1900366099 wait complete. 2024-01-15T18:06:49.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.757+0000 7f676b400640 1 Processor -- start 2024-01-15T18:06:49.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.757+0000 7f676b400640 1 -- start start 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f676b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f67641419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f676b400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f676414ea90 0x7f676414f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f676b400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f676414fae0 0x7f67640cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f676b400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676414ff70 con 0x7f6744004060 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f676b400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6764153bb0 con 0x7f676414ea90 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f676b400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6764153310 con 0x7f676414fae0 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f6768a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f67641419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f6768a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f67641419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51096/0 (socket says 172.21.15.2:51096) 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.758+0000 7f6768a00640 1 -- 172.21.15.2:0/454740124 learned_addr learned my addr 172.21.15.2:0/454740124 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6768a00640 1 -- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f676414fae0 msgr2=0x7f67640cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6768a00640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f676414fae0 0x7f67640cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6768a00640 1 -- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f676414ea90 msgr2=0x7f676414f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6768a00640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f676414ea90 0x7f676414f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6768a00640 1 -- 172.21.15.2:0/454740124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67641535f0 con 0x7f6744004060 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6768a00640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f67641419d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f675400d6f0 tx=0x7f675400dbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67540072e0 con 0x7f6744004060 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6754007440 con 0x7f6744004060 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67540047e0 con 0x7f6744004060 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.759+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67641538d0 con 0x7f6744004060 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.760+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6764152fc0 con 0x7f6744004060 2024-01-15T18:06:49.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.761+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67640cc170 con 0x7f6744004060 2024-01-15T18:06:49.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.762+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f6754004940 con 0x7f6744004060 2024-01-15T18:06:49.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.762+0000 7f6760c00640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f673c06bc50 0x7f673c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:49.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.763+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6754088090 con 0x7f6744004060 2024-01-15T18:06:49.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.763+0000 7f6763e00640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f673c06bc50 0x7f673c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:49.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.764+0000 7f6763e00640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f673c06bc50 0x7f673c06e110 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f674c034000 tx=0x7f674c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:49.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.767+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67640cc170 con 0x7f6744004060 2024-01-15T18:06:49.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:49.995+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47161", "target": ["mon-mgr", ""]}) v1 -- 0x7f67640cc170 con 0x7f673c06bc50 2024-01-15T18:06:50.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.185+0000 7f6760c00640 1 -- 172.21.15.2:0/454740124 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f67640cc170 con 0x7f673c06bc50 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.187+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f673c06bc50 msgr2=0x7f673c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.187+0000 7f676b400640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f673c06bc50 0x7f673c06e110 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f674c034000 tx=0x7f674c030000 comp rx=0 tx=0).stop 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.187+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 msgr2=0x7f67641419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.187+0000 7f676b400640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f67641419d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f675400d6f0 tx=0x7f675400dbb0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.187+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 shutdown_connections 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.187+0000 7f676b400640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f673c06bc50 0x7f673c06e110 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.188+0000 7f676b400640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f676414fae0 0x7f67640cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.188+0000 7f676b400640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f676414ea90 0x7f676414f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.188+0000 7f676b400640 1 --2- 172.21.15.2:0/454740124 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6744004060 0x7f67641419d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.188+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 >> 172.21.15.2:0/454740124 conn(0x7f67640b45f0 msgr2=0x7f67640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.188+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 shutdown_connections 2024-01-15T18:06:50.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.188+0000 7f676b400640 1 -- 172.21.15.2:0/454740124 wait complete. 2024-01-15T18:06:50.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.194+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.194+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.194+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.194+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.199 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.194+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47162 2024-01-15T18:06:50.228 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.223+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.228 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.223+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.228 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.223+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.228 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.223+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.228 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:06:50.223+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:06:50.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.435+0000 7f9cd6200640 1 Processor -- start 2024-01-15T18:06:50.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.435+0000 7f9cd6200640 1 -- start start 2024-01-15T18:06:50.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cd6200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9cd00b2210 0x7f9cd00d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cd6200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd00d1250 con 0x7f9cd00d1870 2024-01-15T18:06:50.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cd6200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cd00ce8e0 con 0x7f9cd00d1c50 2024-01-15T18:06:50.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cd6200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cd00ce6d0 con 0x7f9cd00b2210 2024-01-15T18:06:50.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cd4e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9cd00d1870 0x7f9cd015d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45884/0 (socket says 172.21.15.2:45884) 2024-01-15T18:06:50.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cd4e00640 1 -- 172.21.15.2:0/2913452452 learned_addr learned my addr 172.21.15.2:0/2913452452 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:06:50.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.437+0000 7f9cc7e00640 1 --2- 172.21.15.2:0/2913452452 >> v2:172.21.15.137:3300/0 conn(0x7f9cd00b2210 0x7f9cd00d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 634331074 0 0) 0x7f9cd00d1250 con 0x7f9cd00d1870 2024-01-15T18:06:50.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb0003520 con 0x7f9cd00d1870 2024-01-15T18:06:50.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2622516123 0 0) 0x7f9cb0003520 con 0x7f9cd00d1870 2024-01-15T18:06:50.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 >> v2:172.21.15.137:3300/0 conn(0x7f9cd00b2210 msgr2=0x7f9cd00d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:50.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 --2- 172.21.15.2:0/2913452452 >> v2:172.21.15.137:3300/0 conn(0x7f9cd00b2210 0x7f9cd00d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 >> v1:172.21.15.137:6789/0 conn(0x7f9cd00d1c50 legacy=0x7f9cd00b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd00cebc0 con 0x7f9cd00d1870 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9cc0001670 con 0x7f9cd00d1870 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 >> v1:172.21.15.2:6789/0 conn(0x7f9cd00d1870 legacy=0x7f9cd015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0003970 0x7f9cb0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cb00047c0 0x7f9cb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cb0005180 0x7f9cb000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd00ce6d0 con 0x7f9cb0005180 2024-01-15T18:06:50.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb000c020 con 0x7f9cb0003970 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.438+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb000c1e0 con 0x7f9cb00047c0 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd5800640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cb0005180 0x7f9cb000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd4e00640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0003970 0x7f9cb0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cc7e00640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cb00047c0 0x7f9cb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd5800640 1 -- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cb00047c0 msgr2=0x7f9cb0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd5800640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cb00047c0 0x7f9cb0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd5800640 1 -- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0003970 msgr2=0x7f9cb0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd5800640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0003970 0x7f9cb0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.439+0000 7f9cd5800640 1 -- 172.21.15.2:0/2913452452 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb0003520 con 0x7f9cb0005180 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.440+0000 7f9cd5800640 1 -- 172.21.15.2:0/2913452452 reap_dead start 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.440+0000 7f9cd5800640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cb0005180 0x7f9cb000b980 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8001e50 tx=0x7f9cb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1429c168b20af9b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.440+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8003b70 con 0x7f9cb0005180 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.440+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc8003cf0 con 0x7f9cb0005180 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.440+0000 7f9ccfe00640 1 -- 172.21.15.2:0/2913452452 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8005310 con 0x7f9cb0005180 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 -- 172.21.15.2:0/2913452452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cb0005180 msgr2=0x7f9cb000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cb0005180 0x7f9cb000b980 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8001e50 tx=0x7f9cb00057f0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 -- 172.21.15.2:0/2913452452 shutdown_connections 2024-01-15T18:06:50.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 --2- 172.21.15.2:0/2913452452 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cb0005180 0x7f9cb000b980 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 -- 172.21.15.2:0/2913452452 >> 172.21.15.2:0/2913452452 conn(0x7f9cd00d30b0 msgr2=0x7f9cd00d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:50.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 -- 172.21.15.2:0/2913452452 shutdown_connections 2024-01-15T18:06:50.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.441+0000 7f9cd6200640 1 -- 172.21.15.2:0/2913452452 wait complete. 2024-01-15T18:06:50.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 Processor -- start 2024-01-15T18:06:50.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 -- start start 2024-01-15T18:06:50.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0005180 0x7f9cd01f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cd015af10 0x7f9cd015d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 0x7f9cd015d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd01f7380 con 0x7f9cd015b370 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd015e900 con 0x7f9cb0005180 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cd015ebe0 con 0x7f9cd015af10 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.444+0000 7f9cd5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 0x7f9cd015d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 0x7f9cd015d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51102/0 (socket says 172.21.15.2:51102) 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 -- 172.21.15.2:0/771568854 learned_addr learned my addr 172.21.15.2:0/771568854 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd4e00640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0005180 0x7f9cd01f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 -- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cd015af10 msgr2=0x7f9cd015d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cd015af10 0x7f9cd015d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 -- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0005180 msgr2=0x7f9cd01f6e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0005180 0x7f9cd01f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 -- 172.21.15.2:0/771568854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd015eec0 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.445+0000 7f9cd5800640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 0x7f9cd015d8c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8001f70 tx=0x7f9cc80014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.446+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc8006d50 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.446+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc8007030 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.446+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd015f1a0 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.446+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc80073c0 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.446+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cd015f760 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.447+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cd0066d20 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.448+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f9cc8007520 con 0x7f9cd015b370 2024-01-15T18:06:50.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.449+0000 7f9ccd600640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ca806bc50 0x7f9ca806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:50.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.449+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f9cc80aa5b0 con 0x7f9cd015b370 2024-01-15T18:06:50.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.449+0000 7f9cd4e00640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ca806bc50 0x7f9ca806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:50.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.450+0000 7f9cd4e00640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ca806bc50 0x7f9ca806e110 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0001200 tx=0x7f9cc0001150 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:50.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.454+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cc8076c70 con 0x7f9cd015b370 2024-01-15T18:06:50.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.685+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47162", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cd01f7380 con 0x7f9ca806bc50 2024-01-15T18:06:50.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.871+0000 7f9ccd600640 1 -- 172.21.15.2:0/771568854 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9cd01f7380 con 0x7f9ca806bc50 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ca806bc50 msgr2=0x7f9ca806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ca806bc50 0x7f9ca806e110 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9cc0001200 tx=0x7f9cc0001150 comp rx=0 tx=0).stop 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 msgr2=0x7f9cd015d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 0x7f9cd015d8c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f9cc8001f70 tx=0x7f9cc80014a0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 shutdown_connections 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ca806bc50 0x7f9ca806e110 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cd015b370 0x7f9cd015d8c0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cd015af10 0x7f9cd015d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 --2- 172.21.15.2:0/771568854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cb0005180 0x7f9cd01f6e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:50.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 >> 172.21.15.2:0/771568854 conn(0x7f9cd00d30b0 msgr2=0x7f9cd006c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:50.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 shutdown_connections 2024-01-15T18:06:50.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:50.874+0000 7f9cd6200640 1 -- 172.21.15.2:0/771568854 wait complete. 2024-01-15T18:06:50.902 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47163 2024-01-15T18:06:51.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.126+0000 7f4236e00640 1 Processor -- start 2024-01-15T18:06:51.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.126+0000 7f4236e00640 1 -- start start 2024-01-15T18:06:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4236e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f423015d910 0x7f423015dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4236e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42300b3840 0x7f4230099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4236e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300cf530 con 0x7f423015eda0 2024-01-15T18:06:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4236e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42300cf810 con 0x7f42300b3840 2024-01-15T18:06:51.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4236e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42300cfaf0 con 0x7f423015d910 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4234e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42300b3840 0x7f4230099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4234e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f42300b3840 0x7f4230099a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51110/0 (socket says 172.21.15.2:51110) 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.128+0000 7f4234e00640 1 -- 172.21.15.2:0/3939277757 learned_addr learned my addr 172.21.15.2:0/3939277757 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.129+0000 7f422fe00640 1 --2- 172.21.15.2:0/3939277757 >> v2:172.21.15.137:3300/0 conn(0x7f423015d910 0x7f423015dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.129+0000 7f4234e00640 1 -- 172.21.15.2:0/3939277757 >> v2:172.21.15.137:3300/0 conn(0x7f423015d910 msgr2=0x7f423015dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.129+0000 7f4234e00640 1 --2- 172.21.15.2:0/3939277757 >> v2:172.21.15.137:3300/0 conn(0x7f423015d910 0x7f423015dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.129+0000 7f4234e00640 1 -- 172.21.15.2:0/3939277757 >> v1:172.21.15.137:6789/0 conn(0x7f423015eda0 legacy=0x7f42300d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:51.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.129+0000 7f4234e00640 1 -- 172.21.15.2:0/3939277757 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300cfdd0 con 0x7f42300b3840 2024-01-15T18:06:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.130+0000 7f4234e00640 1 --2- 172.21.15.2:0/3939277757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42300b3840 0x7f4230099a00 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4230071b10 tx=0x7f422802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee3e0a829d59bf5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.130+0000 7f422ea00640 1 -- 172.21.15.2:0/3939277757 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422802c9f0 con 0x7f42300b3840 2024-01-15T18:06:51.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.130+0000 7f422ea00640 1 -- 172.21.15.2:0/3939277757 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422802cb50 con 0x7f42300b3840 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.130+0000 7f422ea00640 1 -- 172.21.15.2:0/3939277757 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42280343f0 con 0x7f42300b3840 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 -- 172.21.15.2:0/3939277757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42300b3840 msgr2=0x7f4230099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 --2- 172.21.15.2:0/3939277757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42300b3840 0x7f4230099a00 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4230071b10 tx=0x7f422802be50 comp rx=0 tx=0).stop 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 -- 172.21.15.2:0/3939277757 shutdown_connections 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 --2- 172.21.15.2:0/3939277757 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42300b3840 0x7f4230099a00 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 --2- 172.21.15.2:0/3939277757 >> v2:172.21.15.137:3300/0 conn(0x7f423015d910 0x7f423015dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 -- 172.21.15.2:0/3939277757 >> 172.21.15.2:0/3939277757 conn(0x7f42300b2b10 msgr2=0x7f42300b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 -- 172.21.15.2:0/3939277757 shutdown_connections 2024-01-15T18:06:51.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.131+0000 7f4236e00640 1 -- 172.21.15.2:0/3939277757 wait complete. 2024-01-15T18:06:51.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.134+0000 7f4236e00640 1 Processor -- start 2024-01-15T18:06:51.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.134+0000 7f4236e00640 1 -- start start 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.135+0000 7f4236e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42300b3840 0x7f42300cae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.135+0000 7f4236e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f423015d910 0x7f42300b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.135+0000 7f4236e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 0x7f42300b5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.135+0000 7f4236e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300b6920 con 0x7f423015eda0 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.135+0000 7f4236e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42300b63a0 con 0x7f42300b3840 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.135+0000 7f4236e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42300b6640 con 0x7f423015d910 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 0x7f42300b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 0x7f42300b5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51118/0 (socket says 172.21.15.2:51118) 2024-01-15T18:06:51.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 -- 172.21.15.2:0/3278986001 learned_addr learned my addr 172.21.15.2:0/3278986001 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f422f400640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f423015d910 0x7f42300b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 -- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f423015d910 msgr2=0x7f42300b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f423015d910 0x7f42300b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 -- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42300b3840 msgr2=0x7f42300cae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42300b3840 0x7f42300cae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.136+0000 7f4234e00640 1 -- 172.21.15.2:0/3278986001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4228009b30 con 0x7f423015eda0 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.137+0000 7f4234e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 0x7f42300b5480 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4228036040 tx=0x7f4228004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.137+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4228009510 con 0x7f423015eda0 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.137+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f422803e450 con 0x7f423015eda0 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.137+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f422803e7e0 con 0x7f423015eda0 2024-01-15T18:06:51.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.137+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42300c8200 con 0x7f423015eda0 2024-01-15T18:06:51.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.137+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42300c87c0 con 0x7f423015eda0 2024-01-15T18:06:51.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.138+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4230074850 con 0x7f423015eda0 2024-01-15T18:06:51.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.139+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f422803e940 con 0x7f423015eda0 2024-01-15T18:06:51.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.140+0000 7f4227e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f420806bcd0 0x7f420806e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:51.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.141+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f42280ab1f0 con 0x7f423015eda0 2024-01-15T18:06:51.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.141+0000 7f422fe00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f420806bcd0 0x7f420806e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:51.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.142+0000 7f422fe00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f420806bcd0 0x7f420806e190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4220006fd0 tx=0x7f4220008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:51.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.145+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42280778b0 con 0x7f423015eda0 2024-01-15T18:06:51.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.372+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47163", "target": ["mon-mgr", ""]}) v1 -- 0x7f42300b6ce0 con 0x7f420806bcd0 2024-01-15T18:06:51.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:51.753+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f4228077c80 con 0x7f423015eda0 2024-01-15T18:06:53.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.453+0000 7f4227e00640 1 -- 172.21.15.2:0/3278986001 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f42300b6ce0 con 0x7f420806bcd0 2024-01-15T18:06:53.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f420806bcd0 msgr2=0x7f420806e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f420806bcd0 0x7f420806e190 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f4220006fd0 tx=0x7f4220008040 comp rx=0 tx=0).stop 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 msgr2=0x7f42300b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 0x7f42300b5480 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4228036040 tx=0x7f4228004a40 comp rx=0 tx=0).stop 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 shutdown_connections 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f420806bcd0 0x7f420806e190 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f423015eda0 0x7f42300b5480 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f423015d910 0x7f42300b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 --2- 172.21.15.2:0/3278986001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f42300b3840 0x7f42300cae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.457+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 >> 172.21.15.2:0/3278986001 conn(0x7f42300b2b10 msgr2=0x7f42300cd690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:53.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.458+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 shutdown_connections 2024-01-15T18:06:53.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.458+0000 7f4236e00640 1 -- 172.21.15.2:0/3278986001 wait complete. 2024-01-15T18:06:53.484 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47164 2024-01-15T18:06:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.710+0000 7eff5b600640 1 Processor -- start 2024-01-15T18:06:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.710+0000 7eff5b600640 1 -- start start 2024-01-15T18:06:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.711+0000 7eff5b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 0x7eff5406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.711+0000 7eff5b600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff54144f70 con 0x7eff54144b90 2024-01-15T18:06:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.711+0000 7eff5b600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff54070690 con 0x7eff5406e830 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.711+0000 7eff5b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff54070970 con 0x7eff5415b840 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.711+0000 7eff58c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7eff54144b90 0x7eff540b4220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53210/0 (socket says 172.21.15.2:53210) 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.711+0000 7eff58c00640 1 -- 172.21.15.2:0/578183800 learned_addr learned my addr 172.21.15.2:0/578183800 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138553641 0 0) 0x7eff54144f70 con 0x7eff54144b90 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34003520 con 0x7eff54144b90 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53e00640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 0x7eff5406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164237811 0 0) 0x7eff34003520 con 0x7eff54144b90 2024-01-15T18:06:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 msgr2=0x7eff5406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 0x7eff5406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 >> v1:172.21.15.137:6789/0 conn(0x7eff5406e830 legacy=0x7eff540d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff54070c50 con 0x7eff54144b90 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7eff44001670 con 0x7eff54144b90 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 >> v1:172.21.15.2:6789/0 conn(0x7eff54144b90 legacy=0x7eff540b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.712+0000 7eff53400640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340039a0 0x7eff34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53400640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff340047c0 0x7eff34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53400640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 0x7eff340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff54070970 con 0x7eff340047c0 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3400bab0 con 0x7eff340039a0 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3400bc70 con 0x7eff5415b840 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53e00640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff340047c0 0x7eff34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff58c00640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340039a0 0x7eff34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff59600640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 0x7eff340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53e00640 1 -- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 msgr2=0x7eff340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53e00640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff5415b840 0x7eff340051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53e00640 1 -- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340039a0 msgr2=0x7eff34003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:06:53.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53e00640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340039a0 0x7eff34003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.713+0000 7eff53e00640 1 -- 172.21.15.2:0/578183800 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff34003520 con 0x7eff340047c0 2024-01-15T18:06:53.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.714+0000 7eff53e00640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff340047c0 0x7eff34004c40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00ba30 tx=0x7eff340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3367db30012fa3e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.715+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c032640 con 0x7eff340047c0 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.715+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3c0327a0 con 0x7eff340047c0 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.715+0000 7eff53400640 1 -- 172.21.15.2:0/578183800 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c032b30 con 0x7eff340047c0 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.715+0000 7eff5b600640 1 -- 172.21.15.2:0/578183800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff340047c0 msgr2=0x7eff34004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.715+0000 7eff5b600640 1 --2- 172.21.15.2:0/578183800 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff340047c0 0x7eff34004c40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7eff3c00ba30 tx=0x7eff340057f0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.715+0000 7eff59600640 1 -- 172.21.15.2:0/578183800 reap_dead start 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.716+0000 7eff5b600640 1 -- 172.21.15.2:0/578183800 shutdown_connections 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.716+0000 7eff5b600640 1 -- 172.21.15.2:0/578183800 >> 172.21.15.2:0/578183800 conn(0x7eff540b2c10 msgr2=0x7eff540b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:53.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.716+0000 7eff5b600640 1 -- 172.21.15.2:0/578183800 shutdown_connections 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.716+0000 7eff5b600640 1 -- 172.21.15.2:0/578183800 wait complete. 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.717+0000 7eff5b600640 1 Processor -- start 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.718+0000 7eff5b600640 1 -- start start 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff5b600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340047c0 0x7eff541e1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff5b600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff540b19b0 0x7eff54151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff5b600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 0x7eff54151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff5b600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff541e4b30 con 0x7eff540b1e30 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff5b600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff541e4e10 con 0x7eff340047c0 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff5b600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff541e50f0 con 0x7eff540b19b0 2024-01-15T18:06:53.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 0x7eff54151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 0x7eff54151e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54212/0 (socket says 172.21.15.2:54212) 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 -- 172.21.15.2:0/1426099799 learned_addr learned my addr 172.21.15.2:0/1426099799 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff58c00640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340047c0 0x7eff541e1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff53e00640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff540b19b0 0x7eff54151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 -- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff540b19b0 msgr2=0x7eff54151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff540b19b0 0x7eff54151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 -- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340047c0 msgr2=0x7eff541e1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340047c0 0x7eff541e1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:53.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff59600640 1 -- 172.21.15.2:0/1426099799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff541e53d0 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.719+0000 7eff53e00640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff540b19b0 0x7eff54151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.720+0000 7eff59600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 0x7eff54151e70 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7eff54071100 tx=0x7eff4c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.720+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c003b70 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.720+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c003cd0 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.720+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff541545d0 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.720+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c0052b0 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.720+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff54154a60 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.721+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff541e55c0 con 0x7eff540b1e30 2024-01-15T18:06:53.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.722+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7eff4c005410 con 0x7eff540b1e30 2024-01-15T18:06:53.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.723+0000 7eff50c00640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff2c06bd70 0x7eff2c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:53.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.723+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7eff4c0881b0 con 0x7eff540b1e30 2024-01-15T18:06:53.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.724+0000 7eff58c00640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff2c06bd70 0x7eff2c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:53.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.725+0000 7eff58c00640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff2c06bd70 0x7eff2c06e230 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7eff44001eb0 tx=0x7eff44009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:53.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.728+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff4c0548a0 con 0x7eff540b1e30 2024-01-15T18:06:53.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:53.964+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7eff540d0920 con 0x7eff2c06bd70 2024-01-15T18:06:58.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.452+0000 7eff50c00640 1 -- 172.21.15.2:0/1426099799 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7eff440108f0 con 0x7eff2c06bd70 2024-01-15T18:06:58.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff2c06bd70 msgr2=0x7eff2c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff2c06bd70 0x7eff2c06e230 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7eff44001eb0 tx=0x7eff44009040 comp rx=0 tx=0).stop 2024-01-15T18:06:58.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 msgr2=0x7eff54151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 0x7eff54151e70 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7eff54071100 tx=0x7eff4c00dc00 comp rx=0 tx=0).stop 2024-01-15T18:06:58.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 shutdown_connections 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff2c06bd70 0x7eff2c06e230 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff540b1e30 0x7eff54151e70 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff540b19b0 0x7eff54151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 --2- 172.21.15.2:0/1426099799 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff340047c0 0x7eff541e1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 >> 172.21.15.2:0/1426099799 conn(0x7eff540b2c10 msgr2=0x7eff540b5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 shutdown_connections 2024-01-15T18:06:58.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.457+0000 7eff5b600640 1 -- 172.21.15.2:0/1426099799 wait complete. 2024-01-15T18:06:58.489 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47165 2024-01-15T18:06:58.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.708+0000 7f5202a00640 1 Processor -- start 2024-01-15T18:06:58.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.709+0000 7f5202a00640 1 -- start start 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.710+0000 7f5202a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51fc0b0b80 0x7f51fc0b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.710+0000 7f5202a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc071250 con 0x7f51fc070560 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.710+0000 7f5202a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51fc071530 con 0x7f51fc150890 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.710+0000 7f5202a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51fc0b1570 con 0x7f51fc0b0b80 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f5200a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f51fc070560 0x7f51fc0709b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53218/0 (socket says 172.21.15.2:53218) 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f5200a00640 1 -- 172.21.15.2:0/245148492 learned_addr learned my addr 172.21.15.2:0/245148492 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f51fb400640 1 --2- 172.21.15.2:0/245148492 >> v2:172.21.15.2:3300/0 conn(0x7f51fc0b0b80 0x7f51fc0b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:58.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f51faa00640 1 -- 172.21.15.2:0/245148492 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814271561 0 0) 0x7f51fc071250 con 0x7f51fc070560 2024-01-15T18:06:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f51faa00640 1 -- 172.21.15.2:0/245148492 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51dc005730 con 0x7f51fc070560 2024-01-15T18:06:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f51fb400640 1 -- 172.21.15.2:0/245148492 >> v1:172.21.15.137:6789/0 conn(0x7f51fc150890 legacy=0x7f51fc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f51fb400640 1 -- 172.21.15.2:0/245148492 >> v1:172.21.15.2:6789/0 conn(0x7f51fc070560 legacy=0x7f51fc0709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.711+0000 7f51fb400640 1 -- 172.21.15.2:0/245148492 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51fc0b2ae0 con 0x7f51fc0b0b80 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.712+0000 7f51fb400640 1 --2- 172.21.15.2:0/245148492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc0b1030 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f51e40097a0 tx=0x7f51e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a178d0417f86d38 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.713+0000 7f51faa00640 1 -- 172.21.15.2:0/245148492 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4031830 con 0x7f51fc0b0b80 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.713+0000 7f51faa00640 1 -- 172.21.15.2:0/245148492 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e4031990 con 0x7f51fc0b0b80 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.713+0000 7f51faa00640 1 -- 172.21.15.2:0/245148492 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4031d20 con 0x7f51fc0b0b80 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.713+0000 7f5202a00640 1 -- 172.21.15.2:0/245148492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 msgr2=0x7f51fc0b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.713+0000 7f5202a00640 1 --2- 172.21.15.2:0/245148492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc0b1030 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f51e40097a0 tx=0x7f51e402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.714+0000 7f5202a00640 1 -- 172.21.15.2:0/245148492 shutdown_connections 2024-01-15T18:06:58.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.714+0000 7f5202a00640 1 --2- 172.21.15.2:0/245148492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc0b1030 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.714+0000 7f5202a00640 1 -- 172.21.15.2:0/245148492 >> 172.21.15.2:0/245148492 conn(0x7f51fc0b45f0 msgr2=0x7f51fc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:06:58.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.714+0000 7f5202a00640 1 -- 172.21.15.2:0/245148492 shutdown_connections 2024-01-15T18:06:58.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.714+0000 7f5202a00640 1 -- 172.21.15.2:0/245148492 wait complete. 2024-01-15T18:06:58.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.715+0000 7f5202a00640 1 Processor -- start 2024-01-15T18:06:58.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.716+0000 7f5202a00640 1 -- start start 2024-01-15T18:06:58.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.716+0000 7f5202a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51fc070560 0x7f51fc1eea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:58.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5202a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:58.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5202a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51fc150890 0x7f51fc15b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:58.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5202a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc141190 con 0x7f51fc0b0b80 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5202a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc15c560 con 0x7f51fc150890 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5202a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51fc149910 con 0x7f51fc070560 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc15b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54232/0 (socket says 172.21.15.2:54232) 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 -- 172.21.15.2:0/1036888055 learned_addr learned my addr 172.21.15.2:0/1036888055 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fbe00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51fc070560 0x7f51fc1eea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5200a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51fc150890 0x7f51fc15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 -- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51fc070560 msgr2=0x7f51fc1eea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51fc070560 0x7f51fc1eea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 -- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51fc150890 msgr2=0x7f51fc15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:06:58.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51fc150890 0x7f51fc15b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f51fb400640 1 -- 172.21.15.2:0/1036888055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e4009400 con 0x7f51fc0b0b80 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.717+0000 7f5200a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51fc150890 0x7f51fc15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.718+0000 7f51fb400640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc15b8a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f51e402bee0 tx=0x7f51e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.718+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e403a6f0 con 0x7f51fc0b0b80 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.718+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e403a870 con 0x7f51fc0b0b80 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.718+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51fc15bfa0 con 0x7f51fc0b0b80 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.718+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e403ac20 con 0x7f51fc0b0b80 2024-01-15T18:06:58.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.718+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51fc149b20 con 0x7f51fc0b0b80 2024-01-15T18:06:58.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.720+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51fc0686f0 con 0x7f51fc0b0b80 2024-01-15T18:06:58.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.720+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f51e404c3b0 con 0x7f51fc0b0b80 2024-01-15T18:06:58.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.722+0000 7f51f3e00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51d406bd70 0x7f51d406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:06:58.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.722+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f51e40aa3b0 con 0x7f51fc0b0b80 2024-01-15T18:06:58.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.723+0000 7f51fbe00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51d406bd70 0x7f51d406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:06:58.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.724+0000 7f51fbe00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51d406bd70 0x7f51d406e230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f51ec000f60 tx=0x7f51ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:06:58.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.727+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51e4076b60 con 0x7f51fc0b0b80 2024-01-15T18:06:58.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:06:58.959+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47165", "target": ["mon-mgr", ""]}) v1 -- 0x7f51fc0cedf0 con 0x7f51d406bd70 2024-01-15T18:07:01.291 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:01.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:03.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.455+0000 7f51f3e00640 1 -- 172.21.15.2:0/1036888055 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f51fc0cedf0 con 0x7f51d406bd70 2024-01-15T18:07:03.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.458+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51d406bd70 msgr2=0x7f51d406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.458+0000 7f5202a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51d406bd70 0x7f51d406e230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f51ec000f60 tx=0x7f51ec009040 comp rx=0 tx=0).stop 2024-01-15T18:07:03.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.458+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 msgr2=0x7f51fc15b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.458+0000 7f5202a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc15b8a0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f51e402bee0 tx=0x7f51e40046c0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 shutdown_connections 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51d406bd70 0x7f51d406e230 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51fc150890 0x7f51fc15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51fc0b0b80 0x7f51fc15b8a0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 --2- 172.21.15.2:0/1036888055 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51fc070560 0x7f51fc1eea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 >> 172.21.15.2:0/1036888055 conn(0x7f51fc0b45f0 msgr2=0x7f51fc0733a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 shutdown_connections 2024-01-15T18:07:03.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.459+0000 7f5202a00640 1 -- 172.21.15.2:0/1036888055 wait complete. 2024-01-15T18:07:03.487 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47166 2024-01-15T18:07:03.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.714+0000 7f6f23e00640 1 Processor -- start 2024-01-15T18:07:03.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.714+0000 7f6f23e00640 1 -- start start 2024-01-15T18:07:03.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6f240720d0 0x7f6f2406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f24071090 con 0x7f6f24143d20 2024-01-15T18:07:03.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f24071430 con 0x7f6f240720d0 2024-01-15T18:07:03.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24144100 con 0x7f6f2415f260 2024-01-15T18:07:03.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f22a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6f24143d20 0x7f6f2415eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52014/0 (socket says 172.21.15.2:52014) 2024-01-15T18:07:03.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f22a00640 1 -- 172.21.15.2:0/1791382902 learned_addr learned my addr 172.21.15.2:0/1791382902 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:03.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23400640 1 --2- 172.21.15.2:0/1791382902 >> v2:172.21.15.2:3300/0 conn(0x7f6f240720d0 0x7f6f2406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f1be00640 1 --2- 172.21.15.2:0/1791382902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f22000640 1 -- 172.21.15.2:0/1791382902 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711062261 0 0) 0x7f6f24071090 con 0x7f6f24143d20 2024-01-15T18:07:03.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f22000640 1 -- 172.21.15.2:0/1791382902 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f04003520 con 0x7f6f24143d20 2024-01-15T18:07:03.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23400640 1 -- 172.21.15.2:0/1791382902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 msgr2=0x7f6f2406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:03.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23400640 1 --2- 172.21.15.2:0/1791382902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2406c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23400640 1 -- 172.21.15.2:0/1791382902 >> v1:172.21.15.2:6789/0 conn(0x7f6f24143d20 legacy=0x7f6f2415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.716+0000 7f6f23400640 1 -- 172.21.15.2:0/1791382902 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24144560 con 0x7f6f240720d0 2024-01-15T18:07:03.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.718+0000 7f6f23400640 1 --2- 172.21.15.2:0/1791382902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2406c690 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009d90 tx=0x7f6f1c02bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f34e719778f384f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.718+0000 7f6f22000640 1 -- 172.21.15.2:0/1791382902 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c02cb30 con 0x7f6f240720d0 2024-01-15T18:07:03.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.718+0000 7f6f22000640 1 -- 172.21.15.2:0/1791382902 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f1c02cc90 con 0x7f6f240720d0 2024-01-15T18:07:03.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.718+0000 7f6f22000640 1 -- 172.21.15.2:0/1791382902 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1c034440 con 0x7f6f240720d0 2024-01-15T18:07:03.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.718+0000 7f6f23e00640 1 -- 172.21.15.2:0/1791382902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 msgr2=0x7f6f2406c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.718+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1791382902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2406c690 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009d90 tx=0x7f6f1c02bd80 comp rx=0 tx=0).stop 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.719+0000 7f6f23e00640 1 -- 172.21.15.2:0/1791382902 shutdown_connections 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.719+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1791382902 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2406c690 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.719+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1791382902 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2406c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.719+0000 7f6f23e00640 1 -- 172.21.15.2:0/1791382902 >> 172.21.15.2:0/1791382902 conn(0x7f6f240b0840 msgr2=0x7f6f240b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.719+0000 7f6f23e00640 1 -- 172.21.15.2:0/1791382902 shutdown_connections 2024-01-15T18:07:03.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.719+0000 7f6f23e00640 1 -- 172.21.15.2:0/1791382902 wait complete. 2024-01-15T18:07:03.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.720+0000 7f6f23e00640 1 Processor -- start 2024-01-15T18:07:03.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.721+0000 7f6f23e00640 1 -- start start 2024-01-15T18:07:03.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.721+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2413ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.721+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f24143d20 0x7f6f2415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2413d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f23e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24140ca0 con 0x7f6f240720d0 2024-01-15T18:07:03.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f23e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2415a820 con 0x7f6f24143d20 2024-01-15T18:07:03.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f23e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241406e0 con 0x7f6f2415f260 2024-01-15T18:07:03.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2413ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2413ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46392/0 (socket says 172.21.15.2:46392) 2024-01-15T18:07:03.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 -- 172.21.15.2:0/1135428490 learned_addr learned my addr 172.21.15.2:0/1135428490 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 -- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 msgr2=0x7f6f2413d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2413d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 -- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f24143d20 msgr2=0x7f6f2415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f24143d20 0x7f6f2415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.722+0000 7f6f22a00640 1 -- 172.21.15.2:0/1135428490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c009a70 con 0x7f6f240720d0 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.723+0000 7f6f22a00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2413ffe0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c530 tx=0x7f6f1400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.723+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14005ce0 con 0x7f6f240720d0 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.723+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14005e40 con 0x7f6f240720d0 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.723+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f140072b0 con 0x7f6f240720d0 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.723+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f241409c0 con 0x7f6f240720d0 2024-01-15T18:07:03.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.723+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f2415ba20 con 0x7f6f240720d0 2024-01-15T18:07:03.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.724+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2415b5a0 con 0x7f6f240720d0 2024-01-15T18:07:03.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.725+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f6f14007410 con 0x7f6f240720d0 2024-01-15T18:07:03.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.726+0000 7f6f1aa00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6efc06bd70 0x7f6efc06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:03.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.726+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6f14087d60 con 0x7f6f240720d0 2024-01-15T18:07:03.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.727+0000 7f6f1be00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6efc06bd70 0x7f6efc06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:03.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.728+0000 7f6f1be00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6efc06bd70 0x7f6efc06e230 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6f2415ab80 tx=0x7f6f0c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:03.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.732+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f2415b5a0 con 0x7f6f240720d0 2024-01-15T18:07:03.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:03.962+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47166", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f240b3850 con 0x7f6efc06bd70 2024-01-15T18:07:04.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.125+0000 7f6f1aa00640 1 -- 172.21.15.2:0/1135428490 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f240b3850 con 0x7f6efc06bd70 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6efc06bd70 msgr2=0x7f6efc06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6efc06bd70 0x7f6efc06e230 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6f2415ab80 tx=0x7f6f0c008040 comp rx=0 tx=0).stop 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 msgr2=0x7f6f2413ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2413ffe0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400c530 tx=0x7f6f1400c9f0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 shutdown_connections 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6efc06bd70 0x7f6efc06e230 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6f2415f260 0x7f6f2413d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6f24143d20 0x7f6f2415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 --2- 172.21.15.2:0/1135428490 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6f240720d0 0x7f6f2413ffe0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 >> 172.21.15.2:0/1135428490 conn(0x7f6f240b0840 msgr2=0x7f6f2406c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 shutdown_connections 2024-01-15T18:07:04.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.129+0000 7f6f23e00640 1 -- 172.21.15.2:0/1135428490 wait complete. 2024-01-15T18:07:04.154 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47167 2024-01-15T18:07:04.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.377+0000 7f7ca8a00640 1 Processor -- start 2024-01-15T18:07:04.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.377+0000 7f7ca8a00640 1 -- start start 2024-01-15T18:07:04.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.379+0000 7f7ca8a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7ca4143620 0x7f7ca40b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:04.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.379+0000 7f7ca8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca4071940 0x7f7ca40cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:04.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.379+0000 7f7ca8a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca40b2fa0 con 0x7f7ca40b2a90 2024-01-15T18:07:04.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.379+0000 7f7ca8a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca40b3280 con 0x7f7ca4143620 2024-01-15T18:07:04.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.379+0000 7f7ca8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca40b3560 con 0x7f7ca4071940 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7ca4143620 0x7f7ca40b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca2a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7ca40b2a90 0x7f7ca4071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52024/0 (socket says 172.21.15.2:52024) 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca2a00640 1 -- 172.21.15.2:0/4035580308 learned_addr learned my addr 172.21.15.2:0/4035580308 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca3e00640 1 --2- 172.21.15.2:0/4035580308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca4071940 0x7f7ca40cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca3400640 1 -- 172.21.15.2:0/4035580308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca4071940 msgr2=0x7f7ca40cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca3400640 1 --2- 172.21.15.2:0/4035580308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca4071940 0x7f7ca40cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca3400640 1 -- 172.21.15.2:0/4035580308 >> v1:172.21.15.2:6789/0 conn(0x7f7ca40b2a90 legacy=0x7f7ca4071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca3400640 1 -- 172.21.15.2:0/4035580308 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca40b3840 con 0x7f7ca4143620 2024-01-15T18:07:04.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.380+0000 7f7ca2000640 1 -- 172.21.15.2:0/4035580308 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971366344 0 0) 0x7f7ca40b2fa0 con 0x7f7ca40b2a90 2024-01-15T18:07:04.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.381+0000 7f7ca3400640 1 --2- 172.21.15.2:0/4035580308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40b2550 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f7c940097a0 tx=0x7f7c9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7890a508de7ec67 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:04.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.382+0000 7f7ca2000640 1 -- 172.21.15.2:0/4035580308 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94031830 con 0x7f7ca4143620 2024-01-15T18:07:04.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.382+0000 7f7ca2000640 1 -- 172.21.15.2:0/4035580308 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c94031990 con 0x7f7ca4143620 2024-01-15T18:07:04.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.382+0000 7f7ca2000640 1 -- 172.21.15.2:0/4035580308 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c94031d20 con 0x7f7ca4143620 2024-01-15T18:07:04.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.382+0000 7f7ca8a00640 1 -- 172.21.15.2:0/4035580308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 msgr2=0x7f7ca40b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.382+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/4035580308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40b2550 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f7c940097a0 tx=0x7f7c9402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.383+0000 7f7ca8a00640 1 -- 172.21.15.2:0/4035580308 shutdown_connections 2024-01-15T18:07:04.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.383+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/4035580308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca4071940 0x7f7ca40cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.383+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/4035580308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40b2550 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.383+0000 7f7ca8a00640 1 -- 172.21.15.2:0/4035580308 >> 172.21.15.2:0/4035580308 conn(0x7f7ca40b0840 msgr2=0x7f7ca40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:04.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.383+0000 7f7ca8a00640 1 -- 172.21.15.2:0/4035580308 shutdown_connections 2024-01-15T18:07:04.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.383+0000 7f7ca8a00640 1 -- 172.21.15.2:0/4035580308 wait complete. 2024-01-15T18:07:04.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.384+0000 7f7ca8a00640 1 Processor -- start 2024-01-15T18:07:04.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 -- start start 2024-01-15T18:07:04.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ca4071940 0x7f7ca4154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:04.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca40b2a90 0x7f7ca4136510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:04.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40cecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca413afb0 con 0x7f7ca4143620 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca4135a20 con 0x7f7ca4071940 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.385+0000 7f7ca8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca4140680 con 0x7f7ca40b2a90 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40cecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40cecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46398/0 (socket says 172.21.15.2:46398) 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 -- 172.21.15.2:0/1247761388 learned_addr learned my addr 172.21.15.2:0/1247761388 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3400640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ca4071940 0x7f7ca4154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca2a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca40b2a90 0x7f7ca4136510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 -- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca40b2a90 msgr2=0x7f7ca4136510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca40b2a90 0x7f7ca4136510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 -- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ca4071940 msgr2=0x7f7ca4154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ca4071940 0x7f7ca4154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.386+0000 7f7ca3e00640 1 -- 172.21.15.2:0/1247761388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c94009400 con 0x7f7ca4143620 2024-01-15T18:07:04.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.387+0000 7f7ca3e00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40cecc0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c007de0 tx=0x7f7c9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.387+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c014620 con 0x7f7ca4143620 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.387+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c9c00ee90 con 0x7f7ca4143620 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.387+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c9c0149e0 con 0x7f7ca4143620 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.387+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca4135c70 con 0x7f7ca4143620 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.387+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca4136230 con 0x7f7ca4143620 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.388+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca406c2e0 con 0x7f7ca4143620 2024-01-15T18:07:04.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.389+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f7c9c014b40 con 0x7f7ca4143620 2024-01-15T18:07:04.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.390+0000 7f7c9b400640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c7c06bd70 0x7f7c7c06e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:04.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.391+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f7c9c088f80 con 0x7f7ca4143620 2024-01-15T18:07:04.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.391+0000 7f7ca3400640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c7c06bd70 0x7f7c7c06e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:04.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.392+0000 7f7ca3400640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c7c06bd70 0x7f7c7c06e230 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7c94036000 tx=0x7f7c9403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:04.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.395+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7c9c055840 con 0x7f7ca4143620 2024-01-15T18:07:04.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.624+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47167", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ca4143a00 con 0x7f7c7c06bd70 2024-01-15T18:07:04.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.804+0000 7f7c9b400640 1 -- 172.21.15.2:0/1247761388 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7c94031b00 con 0x7f7c7c06bd70 2024-01-15T18:07:04.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c7c06bd70 msgr2=0x7f7c7c06e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c7c06bd70 0x7f7c7c06e230 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7c94036000 tx=0x7f7c9403c040 comp rx=0 tx=0).stop 2024-01-15T18:07:04.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 msgr2=0x7f7ca40cecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:04.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40cecc0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f7c9c007de0 tx=0x7f7c9c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 shutdown_connections 2024-01-15T18:07:04.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7c7c06bd70 0x7f7c7c06e230 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ca4143620 0x7f7ca40cecc0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ca40b2a90 0x7f7ca4136510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 --2- 172.21.15.2:0/1247761388 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ca4071940 0x7f7ca4154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:04.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 >> 172.21.15.2:0/1247761388 conn(0x7f7ca40b0840 msgr2=0x7f7ca406d070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:04.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 shutdown_connections 2024-01-15T18:07:04.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:04.808+0000 7f7ca8a00640 1 -- 172.21.15.2:0/1247761388 wait complete. 2024-01-15T18:07:04.837 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:07:04.838 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47164 2024-01-15T18:07:05.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.059+0000 7fb2f5400640 1 Processor -- start 2024-01-15T18:07:05.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.059+0000 7fb2f5400640 1 -- start start 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2f5400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2f00720d0 0x7fb2f006c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2f5400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f0071090 con 0x7fb2f0143d20 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2f5400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2f0144560 con 0x7fb2f015f260 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2f5400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2f0071430 con 0x7fb2f00720d0 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2efe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb2f0143d20 0x7fb2f015eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52034/0 (socket says 172.21.15.2:52034) 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2f4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2f00720d0 0x7fb2f006c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:05.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.061+0000 7fb2efe00640 1 -- 172.21.15.2:0/2216936199 learned_addr learned my addr 172.21.15.2:0/2216936199 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:05.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.062+0000 7fb2f4a00640 1 -- 172.21.15.2:0/2216936199 >> v1:172.21.15.137:6789/0 conn(0x7fb2f015f260 legacy=0x7fb2f006c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:05.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.062+0000 7fb2f4a00640 1 -- 172.21.15.2:0/2216936199 >> v1:172.21.15.2:6789/0 conn(0x7fb2f0143d20 legacy=0x7fb2f015eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:05.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.062+0000 7fb2f4a00640 1 -- 172.21.15.2:0/2216936199 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0144280 con 0x7fb2f00720d0 2024-01-15T18:07:05.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.063+0000 7fb2f4a00640 1 --2- 172.21.15.2:0/2216936199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f006c860 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009d90 tx=0x7fb2e802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecf78f892e1cad4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:05.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.063+0000 7fb2ef400640 1 -- 172.21.15.2:0/2216936199 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e802cbd0 con 0x7fb2f00720d0 2024-01-15T18:07:05.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.063+0000 7fb2ef400640 1 -- 172.21.15.2:0/2216936199 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e802cd30 con 0x7fb2f00720d0 2024-01-15T18:07:05.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.063+0000 7fb2ef400640 1 -- 172.21.15.2:0/2216936199 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e802cea0 con 0x7fb2f00720d0 2024-01-15T18:07:05.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 -- 172.21.15.2:0/2216936199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 msgr2=0x7fb2f006c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:05.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 --2- 172.21.15.2:0/2216936199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f006c860 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8009d90 tx=0x7fb2e802be20 comp rx=0 tx=0).stop 2024-01-15T18:07:05.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 -- 172.21.15.2:0/2216936199 shutdown_connections 2024-01-15T18:07:05.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 --2- 172.21.15.2:0/2216936199 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f006c860 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 -- 172.21.15.2:0/2216936199 >> 172.21.15.2:0/2216936199 conn(0x7fb2f00b0840 msgr2=0x7fb2f00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:05.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 -- 172.21.15.2:0/2216936199 shutdown_connections 2024-01-15T18:07:05.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.064+0000 7fb2f5400640 1 -- 172.21.15.2:0/2216936199 wait complete. 2024-01-15T18:07:05.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.066+0000 7fb2f5400640 1 Processor -- start 2024-01-15T18:07:05.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.066+0000 7fb2f5400640 1 -- start start 2024-01-15T18:07:05.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2f5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f0156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:05.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2f5400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0143d20 0x7fb2f015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2f5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f015f260 0x7fb2f01565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2f5400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f015ddc0 con 0x7fb2f00720d0 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2f5400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0156b30 con 0x7fb2f0143d20 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2f5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f015d140 con 0x7fb2f015f260 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f0156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2efe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f0156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46408/0 (socket says 172.21.15.2:46408) 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.067+0000 7fb2efe00640 1 -- 172.21.15.2:0/1426812475 learned_addr learned my addr 172.21.15.2:0/1426812475 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2f4a00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f015f260 0x7fb2f01565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:05.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2efe00640 1 -- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f015f260 msgr2=0x7fb2f01565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2e7e00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0143d20 0x7fb2f015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2efe00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f015f260 0x7fb2f01565f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2efe00640 1 -- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0143d20 msgr2=0x7fb2f015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2efe00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0143d20 0x7fb2f015e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2efe00640 1 -- 172.21.15.2:0/1426812475 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e8009a70 con 0x7fb2f00720d0 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2efe00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f0156d80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000c4a0 tx=0x7fb2e000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0005ce0 con 0x7fb2f00720d0 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2e0005e40 con 0x7fb2f00720d0 2024-01-15T18:07:05.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2e0007270 con 0x7fb2f00720d0 2024-01-15T18:07:05.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f015d420 con 0x7fb2f00720d0 2024-01-15T18:07:05.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.068+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f015d9e0 con 0x7fb2f00720d0 2024-01-15T18:07:05.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.070+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0074770 con 0x7fb2f00720d0 2024-01-15T18:07:05.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.071+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fb2e0007490 con 0x7fb2f00720d0 2024-01-15T18:07:05.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.071+0000 7fb2ecc00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806bd70 0x7fb2c806e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:05.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.072+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fb2e008a720 con 0x7fb2f00720d0 2024-01-15T18:07:05.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.072+0000 7fb2e7e00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806bd70 0x7fb2c806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:05.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.073+0000 7fb2e7e00640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806bd70 0x7fb2c806e230 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb2f006e9e0 tx=0x7fb2d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:05.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.076+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2e00542f0 con 0x7fb2f00720d0 2024-01-15T18:07:05.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.308+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2f0071850 con 0x7fb2c806bd70 2024-01-15T18:07:05.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.328+0000 7fb2ecc00640 1 -- 172.21.15.2:0/1426812475 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fb2d800a380 con 0x7fb2c806bd70 2024-01-15T18:07:05.332 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:07:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:07:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:07:05.332 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:07:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:07:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:07:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:05.333 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:05.333 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:05.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806bd70 msgr2=0x7fb2c806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:05.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806bd70 0x7fb2c806e230 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fb2f006e9e0 tx=0x7fb2d8008040 comp rx=0 tx=0).stop 2024-01-15T18:07:05.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 msgr2=0x7fb2f0156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:05.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f0156d80 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fb2e000c4a0 tx=0x7fb2e000c960 comp rx=0 tx=0).stop 2024-01-15T18:07:05.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 shutdown_connections 2024-01-15T18:07:05.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2c806bd70 0x7fb2c806e230 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2f015f260 0x7fb2f01565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2f0143d20 0x7fb2f015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 --2- 172.21.15.2:0/1426812475 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2f00720d0 0x7fb2f0156d80 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:05.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 >> 172.21.15.2:0/1426812475 conn(0x7fb2f00b0840 msgr2=0x7fb2f00ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:05.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 shutdown_connections 2024-01-15T18:07:05.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:05.331+0000 7fb2f5400640 1 -- 172.21.15.2:0/1426812475 wait complete. 2024-01-15T18:07:05.354 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:07:06.356 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47164 2024-01-15T18:07:06.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.586+0000 7fa2d6e00640 1 Processor -- start 2024-01-15T18:07:06.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.587+0000 7fa2d6e00640 1 -- start start 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.588+0000 7fa2d6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d0071230 0x7fa2d00716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.588+0000 7fa2d6e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa2d00cc3b0 0x7fa2d00cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.588+0000 7fa2d6e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d00b2fa0 con 0x7fa2d0143620 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.588+0000 7fa2d6e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d0071bf0 con 0x7fa2d00cc3b0 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.588+0000 7fa2d6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d00b3280 con 0x7fa2d0071230 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa2d00cc3b0 0x7fa2d00cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa2d00cc3b0 0x7fa2d00cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46422/0 (socket says 172.21.15.2:46422) 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 -- 172.21.15.2:0/4009324937 learned_addr learned my addr 172.21.15.2:0/4009324937 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d5000640 1 --2- 172.21.15.2:0/4009324937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d0071230 0x7fa2d00716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:06.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 -- 172.21.15.2:0/4009324937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d0071230 msgr2=0x7fa2d00716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:06.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 --2- 172.21.15.2:0/4009324937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d0071230 0x7fa2d00716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 -- 172.21.15.2:0/4009324937 >> v1:172.21.15.137:6789/0 conn(0x7fa2d0143620 legacy=0x7fa2d00b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.589+0000 7fa2d6400640 1 -- 172.21.15.2:0/4009324937 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d00b3560 con 0x7fa2d00cc3b0 2024-01-15T18:07:06.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.590+0000 7fa2d6400640 1 --2- 172.21.15.2:0/4009324937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d00cc3b0 0x7fa2d00cc840 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009e50 tx=0x7fa2c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca59a543dd7ef1b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:06.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.591+0000 7fa2cfe00640 1 -- 172.21.15.2:0/4009324937 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c802cb00 con 0x7fa2d00cc3b0 2024-01-15T18:07:06.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.591+0000 7fa2cfe00640 1 -- 172.21.15.2:0/4009324937 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c802cc60 con 0x7fa2d00cc3b0 2024-01-15T18:07:06.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.591+0000 7fa2cfe00640 1 -- 172.21.15.2:0/4009324937 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c80343f0 con 0x7fa2d00cc3b0 2024-01-15T18:07:06.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.591+0000 7fa2d6e00640 1 -- 172.21.15.2:0/4009324937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d00cc3b0 msgr2=0x7fa2d00cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.591+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/4009324937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d00cc3b0 0x7fa2d00cc840 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009e50 tx=0x7fa2c802bf60 comp rx=0 tx=0).stop 2024-01-15T18:07:06.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.592+0000 7fa2d6e00640 1 -- 172.21.15.2:0/4009324937 shutdown_connections 2024-01-15T18:07:06.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.592+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/4009324937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d00cc3b0 0x7fa2d00cc840 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.592+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/4009324937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d0071230 0x7fa2d00716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.592+0000 7fa2d6e00640 1 -- 172.21.15.2:0/4009324937 >> 172.21.15.2:0/4009324937 conn(0x7fa2d00b0840 msgr2=0x7fa2d00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:06.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.592+0000 7fa2d6e00640 1 -- 172.21.15.2:0/4009324937 shutdown_connections 2024-01-15T18:07:06.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.592+0000 7fa2d6e00640 1 -- 172.21.15.2:0/4009324937 wait complete. 2024-01-15T18:07:06.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.593+0000 7fa2d6e00640 1 Processor -- start 2024-01-15T18:07:06.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.594+0000 7fa2d6e00640 1 -- start start 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d0071230 0x7fa2d013d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d00cc3b0 0x7fa2d00add40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 0x7fa2d013e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d013ac40 con 0x7fa2d0143620 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d013ebe0 con 0x7fa2d0071230 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d0166f70 con 0x7fa2d00cc3b0 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 0x7fa2d013e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 0x7fa2d013e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46436/0 (socket says 172.21.15.2:46436) 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 -- 172.21.15.2:0/3358614687 learned_addr learned my addr 172.21.15.2:0/3358614687 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d5a00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d0071230 0x7fa2d013d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d5000640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d00cc3b0 0x7fa2d00add40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 -- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d00cc3b0 msgr2=0x7fa2d00add40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d00cc3b0 0x7fa2d00add40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 -- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d0071230 msgr2=0x7fa2d013d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d0071230 0x7fa2d013d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.595+0000 7fa2d6400640 1 -- 172.21.15.2:0/3358614687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c8009b30 con 0x7fa2d0143620 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.596+0000 7fa2d6400640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 0x7fa2d013e6a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8036040 tx=0x7fa2c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.596+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8009510 con 0x7fa2d0143620 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.596+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c803e450 con 0x7fa2d0143620 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.596+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d0140e60 con 0x7fa2d0143620 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.596+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c803e800 con 0x7fa2d0143620 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.596+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d0141420 con 0x7fa2d0143620 2024-01-15T18:07:06.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.598+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d0074770 con 0x7fa2d0143620 2024-01-15T18:07:06.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.598+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fa2c803e960 con 0x7fa2d0143620 2024-01-15T18:07:06.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.599+0000 7fa2cd600640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa2a806bd70 0x7fa2a806e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:06.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.599+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fa2c80ab220 con 0x7fa2d0143620 2024-01-15T18:07:06.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.600+0000 7fa2d5a00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa2a806bd70 0x7fa2a806e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:06.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.601+0000 7fa2d5a00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa2a806bd70 0x7fa2a806e230 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0000f60 tx=0x7fa2c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:06.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.604+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2c8077b40 con 0x7fa2d0143620 2024-01-15T18:07:06.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.837+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2d00b02e0 con 0x7fa2a806bd70 2024-01-15T18:07:06.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.855+0000 7fa2cd600640 1 -- 172.21.15.2:0/3358614687 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fa2d00b02e0 con 0x7fa2a806bd70 2024-01-15T18:07:06.858 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:07:06.858 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:06.859 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa2a806bd70 msgr2=0x7fa2a806e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa2a806bd70 0x7fa2a806e230 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa2c0000f60 tx=0x7fa2c0009040 comp rx=0 tx=0).stop 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 msgr2=0x7fa2d013e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 0x7fa2d013e6a0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8036040 tx=0x7fa2c8004a40 comp rx=0 tx=0).stop 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 shutdown_connections 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa2a806bd70 0x7fa2a806e230 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa2d0143620 0x7fa2d013e6a0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa2d00cc3b0 0x7fa2d00add40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 --2- 172.21.15.2:0/3358614687 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa2d0071230 0x7fa2d013d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:06.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.857+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 >> 172.21.15.2:0/3358614687 conn(0x7fa2d00b0840 msgr2=0x7fa2d006d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:06.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.858+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 shutdown_connections 2024-01-15T18:07:06.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:06.858+0000 7fa2d6e00640 1 -- 172.21.15.2:0/3358614687 wait complete. 2024-01-15T18:07:06.882 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:07:07.882 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47164 2024-01-15T18:07:08.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.111+0000 7f9143600640 1 Processor -- start 2024-01-15T18:07:08.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.112+0000 7f9143600640 1 -- start start 2024-01-15T18:07:08.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.113+0000 7f9143600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f913c14afd0 0x7f913c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.113+0000 7f9143600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 0x7f913c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.113+0000 7f9143600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0b25f0 con 0x7f913c14b3b0 2024-01-15T18:07:08.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.113+0000 7f9143600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c070ff0 con 0x7f913c14afd0 2024-01-15T18:07:08.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.113+0000 7f9143600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0b28d0 con 0x7f913c06d000 2024-01-15T18:07:08.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.114+0000 7f9133e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f913c14b3b0 0x7f913c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52038/0 (socket says 172.21.15.2:52038) 2024-01-15T18:07:08.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.114+0000 7f9133e00640 1 -- 172.21.15.2:0/1991080957 learned_addr learned my addr 172.21.15.2:0/1991080957 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:08.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.114+0000 7f9140c00640 1 --2- 172.21.15.2:0/1991080957 >> v2:172.21.15.137:3300/0 conn(0x7f913c14afd0 0x7f913c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.114+0000 7f9141600640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 0x7f913c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242087793 0 0) 0x7f913c0b25f0 con 0x7f913c14b3b0 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f911c003520 con 0x7f913c14b3b0 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2520031617 0 0) 0x7f911c003520 con 0x7f913c14b3b0 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 msgr2=0x7f913c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 0x7f913c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 >> v2:172.21.15.137:3300/0 conn(0x7f913c14afd0 msgr2=0x7f913c0b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 --2- 172.21.15.2:0/1991080957 >> v2:172.21.15.137:3300/0 conn(0x7f913c14afd0 0x7f913c0b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0b2bb0 con 0x7f913c14b3b0 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9124001670 con 0x7f913c14b3b0 2024-01-15T18:07:08.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 >> v1:172.21.15.2:6789/0 conn(0x7f913c14b3b0 legacy=0x7f913c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c003970 0x7f911c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 0x7f911c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c070ff0 con 0x7f911c0047c0 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0b28d0 con 0x7f911c003970 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c00ba30 con 0x7f913c06d000 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.115+0000 7f9133e00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9140c00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c003970 0x7f911c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9141600640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 0x7f911c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9133e00640 1 -- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 msgr2=0x7f911c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9133e00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c06d000 0x7f911c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9133e00640 1 -- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c003970 msgr2=0x7f911c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9133e00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c003970 0x7f911c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9133e00640 1 -- 172.21.15.2:0/1991080957 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c003520 con 0x7f911c0047c0 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.116+0000 7f9140c00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c003970 0x7f911c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.117+0000 7f9133e00640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f912400c0e0 tx=0x7f911c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c298b53d64e5f2ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.117+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124005ad0 con 0x7f911c0047c0 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.117+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124005c50 con 0x7f911c0047c0 2024-01-15T18:07:08.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.117+0000 7f913be00640 1 -- 172.21.15.2:0/1991080957 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124007070 con 0x7f911c0047c0 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9143600640 1 -- 172.21.15.2:0/1991080957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 msgr2=0x7f911c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9143600640 1 --2- 172.21.15.2:0/1991080957 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f912400c0e0 tx=0x7f911c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9141600640 1 -- 172.21.15.2:0/1991080957 reap_dead start 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9143600640 1 -- 172.21.15.2:0/1991080957 shutdown_connections 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9143600640 1 -- 172.21.15.2:0/1991080957 >> 172.21.15.2:0/1991080957 conn(0x7f913c0b45f0 msgr2=0x7f913c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9143600640 1 -- 172.21.15.2:0/1991080957 shutdown_connections 2024-01-15T18:07:08.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.118+0000 7f9143600640 1 -- 172.21.15.2:0/1991080957 wait complete. 2024-01-15T18:07:08.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.120+0000 7f9143600640 1 Processor -- start 2024-01-15T18:07:08.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.120+0000 7f9143600640 1 -- start start 2024-01-15T18:07:08.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9143600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f913c1d78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9143600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c0c40d0 0x7f913c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9143600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c0b0f80 0x7f913c0b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9143600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0b1940 con 0x7f911c0047c0 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9143600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c1d7f60 con 0x7f913c0b0f80 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9143600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c1d8240 con 0x7f913c0c40d0 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9140c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f913c1d78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.121+0000 7f9140c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f913c1d78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46462/0 (socket says 172.21.15.2:46462) 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 -- 172.21.15.2:0/3787225145 learned_addr learned my addr 172.21.15.2:0/3787225145 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9133e00640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c0c40d0 0x7f913c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9141600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c0b0f80 0x7f913c0b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 -- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c0c40d0 msgr2=0x7f913c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c0c40d0 0x7f913c0ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 -- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c0b0f80 msgr2=0x7f913c0b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c0b0f80 0x7f913c0b1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 -- 172.21.15.2:0/3787225145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c1d8520 con 0x7f911c0047c0 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.122+0000 7f9140c00640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f913c1d78a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f912c010920 tx=0x7f912c010de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.123+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c00ec20 con 0x7f911c0047c0 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.123+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c00ed80 con 0x7f911c0047c0 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.123+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c01c640 con 0x7f911c0047c0 2024-01-15T18:07:08.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.123+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c1d8800 con 0x7f911c0047c0 2024-01-15T18:07:08.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.123+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c0c4550 con 0x7f911c0047c0 2024-01-15T18:07:08.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.124+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c0686f0 con 0x7f911c0047c0 2024-01-15T18:07:08.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.125+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f912c01c7a0 con 0x7f911c0047c0 2024-01-15T18:07:08.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.126+0000 7f9139600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406bda0 0x7f911406e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:08.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.126+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f912c088de0 con 0x7f911c0047c0 2024-01-15T18:07:08.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.127+0000 7f9133e00640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406bda0 0x7f911406e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:08.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.128+0000 7f9133e00640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406bda0 0x7f911406e260 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9124034000 tx=0x7f9124007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:08.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.131+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f912c055520 con 0x7f911c0047c0 2024-01-15T18:07:08.299 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:08.295+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:08.299 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:08.295+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:08.299 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:08.295+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:08.300 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:08.295+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:08.300 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:08.295+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:08.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:08.371+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7f913c0cfc90 con 0x7f911406bda0 2024-01-15T18:07:10.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:10.593+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 24) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f912c0558f0 con 0x7f911c0047c0 2024-01-15T18:07:13.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.282+0000 7f9139600640 1 -- 172.21.15.2:0/3787225145 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f913c0cfc90 con 0x7f911406bda0 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:13.286 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:13.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406bda0 msgr2=0x7f911406e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:13.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406bda0 0x7f911406e260 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f9124034000 tx=0x7f9124007000 comp rx=0 tx=0).stop 2024-01-15T18:07:13.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 msgr2=0x7f913c1d78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:13.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f913c1d78a0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f912c010920 tx=0x7f912c010de0 comp rx=0 tx=0).stop 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 shutdown_connections 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406bda0 0x7f911406e260 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c0b0f80 0x7f913c0b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c0c40d0 0x7f913c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 --2- 172.21.15.2:0/3787225145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f913c1d78a0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.286+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 >> 172.21.15.2:0/3787225145 conn(0x7f913c0b45f0 msgr2=0x7f913c1e8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.287+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 shutdown_connections 2024-01-15T18:07:13.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:13.287+0000 7f9143600640 1 -- 172.21.15.2:0/3787225145 wait complete. 2024-01-15T18:07:13.305 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:13.302+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:13.305 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:13.302+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:13.305 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:13.302+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:13.305 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:13.302+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:13.305 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:13.302+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:07:13.320 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:07:14.320 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47164 2024-01-15T18:07:14.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.640+0000 7f8fcb200640 1 Processor -- start 2024-01-15T18:07:14.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.640+0000 7f8fcb200640 1 -- start start 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fcb200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8fcc0d1a30 0x7f8fcc0cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fcb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8fcc0b1a10 0x7f8fcc0c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fc9e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f8fcc0b1520 0x7f8fcc0cecf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57760/0 (socket says 172.21.15.2:57760) 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fc9e00640 1 -- 172.21.15.2:0/1519872390 learned_addr learned my addr 172.21.15.2:0/1519872390 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fc9400640 1 --2- 172.21.15.2:0/1519872390 >> v2:172.21.15.2:3300/0 conn(0x7f8fcc0d1a30 0x7f8fcc0cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc15edc0 con 0x7f8fcc0b1520 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0cfa70 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.643+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc0cdfb0 con 0x7f8fcc0b1a10 2024-01-15T18:07:14.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fca800640 1 --2- 172.21.15.2:0/1519872390 >> v2:172.21.15.137:3300/0 conn(0x7f8fcc0b1a10 0x7f8fcc0c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fc8a00640 1 -- 172.21.15.2:0/1519872390 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525360100 0 0) 0x7f8fcc15edc0 con 0x7f8fcc0b1520 2024-01-15T18:07:14.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fc8a00640 1 -- 172.21.15.2:0/1519872390 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fac003520 con 0x7f8fcc0b1520 2024-01-15T18:07:14.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fc9400640 1 -- 172.21.15.2:0/1519872390 >> v2:172.21.15.137:3300/0 conn(0x7f8fcc0b1a10 msgr2=0x7f8fcc0c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:14.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fc9400640 1 --2- 172.21.15.2:0/1519872390 >> v2:172.21.15.137:3300/0 conn(0x7f8fcc0b1a10 0x7f8fcc0c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fc9400640 1 -- 172.21.15.2:0/1519872390 >> v1:172.21.15.2:6789/0 conn(0x7f8fcc0b1520 legacy=0x7f8fcc0cecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.644+0000 7f8fc9400640 1 -- 172.21.15.2:0/1519872390 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0ce290 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.646+0000 7f8fc9400640 1 --2- 172.21.15.2:0/1519872390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc0cf400 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0b2970 tx=0x7f8fb402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acf3feb6d9eabc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.646+0000 7f8fc8a00640 1 -- 172.21.15.2:0/1519872390 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb402e070 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.646+0000 7f8fc8a00640 1 -- 172.21.15.2:0/1519872390 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb40315a0 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.646+0000 7f8fc8a00640 1 -- 172.21.15.2:0/1519872390 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4031930 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 msgr2=0x7f8fcc0cf400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 --2- 172.21.15.2:0/1519872390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc0cf400 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc0b2970 tx=0x7f8fb402b840 comp rx=0 tx=0).stop 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 shutdown_connections 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 --2- 172.21.15.2:0/1519872390 >> v2:172.21.15.137:3300/0 conn(0x7f8fcc0b1a10 0x7f8fcc0c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 --2- 172.21.15.2:0/1519872390 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc0cf400 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 >> 172.21.15.2:0/1519872390 conn(0x7f8fcc0d0f10 msgr2=0x7f8fcc0d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 shutdown_connections 2024-01-15T18:07:14.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.647+0000 7f8fcb200640 1 -- 172.21.15.2:0/1519872390 wait complete. 2024-01-15T18:07:14.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.651+0000 7f8fcb200640 1 Processor -- start 2024-01-15T18:07:14.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.651+0000 7f8fcb200640 1 -- start start 2024-01-15T18:07:14.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fcb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8fcc0b1520 0x7f8fcc0cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fcb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fcc0b1a10 0x7f8fcc0c9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fcb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc15d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fcb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0b5e10 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fcb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0c9840 con 0x7f8fcc0b1a10 2024-01-15T18:07:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fcb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc0c9ac0 con 0x7f8fcc0b1520 2024-01-15T18:07:14.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fca800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc15d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fca800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc15d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53260/0 (socket says 172.21.15.2:53260) 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.653+0000 7f8fca800640 1 -- 172.21.15.2:0/3547745943 learned_addr learned my addr 172.21.15.2:0/3547745943 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fc9e00640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8fcc0b1520 0x7f8fcc0cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fca800640 1 -- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8fcc0b1520 msgr2=0x7f8fcc0cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fca800640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8fcc0b1520 0x7f8fcc0cab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fca800640 1 -- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fcc0b1a10 msgr2=0x7f8fcc0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fca800640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fcc0b1a10 0x7f8fcc0c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:14.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fca800640 1 -- 172.21.15.2:0/3547745943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb4009400 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.654+0000 7f8fca800640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc15d000 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400d5d0 tx=0x7f8fc400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.655+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40125f0 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.655+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4012770 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.655+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc0c9da0 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.655+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4012b20 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.655+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc0ca360 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.657+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc074770 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.658+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 24) v1 ==== 90559+0+0 (secure 0 0 0) 0x7f8fc4012c80 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.660+0000 7f8fc2000640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8fa406be50 0x7f8fa406e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:14.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.660+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8fc40894c0 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.663+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8fc4055dc0 con 0x7f8fcc0d1a30 2024-01-15T18:07:14.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.666+0000 7f8fc9e00640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8fa406be50 0x7f8fa406e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:14.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.669+0000 7f8fc9e00640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8fa406be50 0x7f8fa406e310 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc001560 tx=0x7f8fbc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:14.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:14.721+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8fc4056190 con 0x7f8fcc0d1a30 2024-01-15T18:07:15.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.045+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fcc0b20f0 con 0x7f8fa406be50 2024-01-15T18:07:15.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.096+0000 7f8fc2000640 1 -- 172.21.15.2:0/3547745943 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f8fbc001930 con 0x7f8fa406be50 2024-01-15T18:07:15.102 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:07:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:07:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:07:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:07:15.102 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 0.0, 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "0/3", 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:15.103 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:15.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.103+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8fa406be50 msgr2=0x7f8fa406e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.103+0000 7f8fcb200640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8fa406be50 0x7f8fa406e310 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc001560 tx=0x7f8fbc009040 comp rx=0 tx=0).stop 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.103+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 msgr2=0x7f8fcc15d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.103+0000 7f8fcb200640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc15d000 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400d5d0 tx=0x7f8fc400da90 comp rx=0 tx=0).stop 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.103+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 shutdown_connections 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.103+0000 7f8fcb200640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8fa406be50 0x7f8fa406e310 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.104+0000 7f8fcb200640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8fcc0d1a30 0x7f8fcc15d000 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.104+0000 7f8fcb200640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8fcc0b1a10 0x7f8fcc0c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.104+0000 7f8fcb200640 1 --2- 172.21.15.2:0/3547745943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8fcc0b1520 0x7f8fcc0cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.104+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 >> 172.21.15.2:0/3547745943 conn(0x7f8fcc0d0f10 msgr2=0x7f8fcc15e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:15.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.104+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 shutdown_connections 2024-01-15T18:07:15.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.104+0000 7f8fcb200640 1 -- 172.21.15.2:0/3547745943 wait complete. 2024-01-15T18:07:15.138 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:07:15.138 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47167 2024-01-15T18:07:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.416+0000 7f87fa200640 1 Processor -- start 2024-01-15T18:07:15.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.416+0000 7f87fa200640 1 -- start start 2024-01-15T18:07:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.417+0000 7f87fa200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f87f415d730 0x7f87f40d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.417+0000 7f87fa200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b8c30 0x7f87f40d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.417+0000 7f87fa200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f40d1900 con 0x7f87f40b4c70 2024-01-15T18:07:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.417+0000 7f87fa200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f40d1be0 con 0x7f87f415d730 2024-01-15T18:07:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.417+0000 7f87fa200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40d1ec0 con 0x7f87f40b8c30 2024-01-15T18:07:15.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f87f415d730 0x7f87f40d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f87f415d730 0x7f87f40d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53272/0 (socket says 172.21.15.2:53272) 2024-01-15T18:07:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 -- 172.21.15.2:0/2116738672 learned_addr learned my addr 172.21.15.2:0/2116738672 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 -- 172.21.15.2:0/2116738672 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b8c30 msgr2=0x7f87f40d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 --2- 172.21.15.2:0/2116738672 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b8c30 0x7f87f40d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 -- 172.21.15.2:0/2116738672 >> v1:172.21.15.137:6789/0 conn(0x7f87f40b4c70 legacy=0x7f87f40b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:15.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.418+0000 7f87f2a00640 1 -- 172.21.15.2:0/2116738672 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40d21a0 con 0x7f87f415d730 2024-01-15T18:07:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.419+0000 7f87f2a00640 1 --2- 172.21.15.2:0/2116738672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f40d1270 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009870 tx=0x7f87dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ae6115de18ba84 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87f2000640 1 -- 172.21.15.2:0/2116738672 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc031640 con 0x7f87f415d730 2024-01-15T18:07:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87f2000640 1 -- 172.21.15.2:0/2116738672 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc0317a0 con 0x7f87f415d730 2024-01-15T18:07:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87f2000640 1 -- 172.21.15.2:0/2116738672 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc031b30 con 0x7f87f415d730 2024-01-15T18:07:15.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 -- 172.21.15.2:0/2116738672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 msgr2=0x7f87f40d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 --2- 172.21.15.2:0/2116738672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f40d1270 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f87dc009870 tx=0x7f87dc02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 -- 172.21.15.2:0/2116738672 shutdown_connections 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 --2- 172.21.15.2:0/2116738672 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b8c30 0x7f87f40d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 --2- 172.21.15.2:0/2116738672 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f40d1270 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 -- 172.21.15.2:0/2116738672 >> 172.21.15.2:0/2116738672 conn(0x7f87f4072210 msgr2=0x7f87f40cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.420+0000 7f87fa200640 1 -- 172.21.15.2:0/2116738672 shutdown_connections 2024-01-15T18:07:15.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.421+0000 7f87fa200640 1 -- 172.21.15.2:0/2116738672 wait complete. 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.422+0000 7f87fa200640 1 Processor -- start 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.422+0000 7f87fa200640 1 -- start start 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.423+0000 7f87fa200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b4c70 0x7f87f40c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.423+0000 7f87fa200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87f40b8c30 0x7f87f4153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.423+0000 7f87fa200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f4154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.423+0000 7f87fa200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4160640 con 0x7f87f415d730 2024-01-15T18:07:15.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.423+0000 7f87fa200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40cc810 con 0x7f87f40b8c30 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.423+0000 7f87fa200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f415fd10 con 0x7f87f40b4c70 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f4154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f4154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53276/0 (socket says 172.21.15.2:53276) 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 -- 172.21.15.2:0/4182413845 learned_addr learned my addr 172.21.15.2:0/4182413845 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 -- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b4c70 msgr2=0x7f87f40c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b4c70 0x7f87f40c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 -- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87f40b8c30 msgr2=0x7f87f4153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87f40b8c30 0x7f87f4153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.424+0000 7f87f3e00640 1 -- 172.21.15.2:0/4182413845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc0094d0 con 0x7f87f415d730 2024-01-15T18:07:15.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.425+0000 7f87f3e00640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f4154520 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b970 tx=0x7f87ec00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.425+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00cb40 con 0x7f87f415d730 2024-01-15T18:07:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.425+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec00ccc0 con 0x7f87f415d730 2024-01-15T18:07:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.425+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f40d3060 con 0x7f87f415d730 2024-01-15T18:07:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.425+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec014680 con 0x7f87f415d730 2024-01-15T18:07:15.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.425+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f41602d0 con 0x7f87f415d730 2024-01-15T18:07:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.427+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f40686f0 con 0x7f87f415d730 2024-01-15T18:07:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.427+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f87ec01d3b0 con 0x7f87f415d730 2024-01-15T18:07:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.428+0000 7f87eb400640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87cc06be90 0x7f87cc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:15.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.428+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f87ec088810 con 0x7f87f415d730 2024-01-15T18:07:15.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.431+0000 7f87f3400640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87cc06be90 0x7f87cc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:15.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.435+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87ec0588d0 con 0x7f87f415d730 2024-01-15T18:07:15.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.436+0000 7f87f3400640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87cc06be90 0x7f87cc06e350 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f87e4001560 tx=0x7f87e4007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:15.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.848+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47167", "target": ["mon-mgr", ""]}) v1 -- 0x7f87f40d2ee0 con 0x7f87cc06be90 2024-01-15T18:07:15.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.922+0000 7f87eb400640 1 -- 172.21.15.2:0/4182413845 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f87e40011f0 con 0x7f87cc06be90 2024-01-15T18:07:15.927 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:07:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:07:15.927 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:07:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:07:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:07:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:07:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:15.928 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:15.928 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:15.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87cc06be90 msgr2=0x7f87cc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87cc06be90 0x7f87cc06e350 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f87e4001560 tx=0x7f87e4007990 comp rx=0 tx=0).stop 2024-01-15T18:07:15.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 msgr2=0x7f87f4154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:15.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f4154520 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b970 tx=0x7f87ec00be30 comp rx=0 tx=0).stop 2024-01-15T18:07:15.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 shutdown_connections 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f87cc06be90 0x7f87cc06e350 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87f415d730 0x7f87f4154520 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87f40b8c30 0x7f87f4153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.925+0000 7f87fa200640 1 --2- 172.21.15.2:0/4182413845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87f40b4c70 0x7f87f40c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.926+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 >> 172.21.15.2:0/4182413845 conn(0x7f87f4072210 msgr2=0x7f87f40b66e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.926+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 shutdown_connections 2024-01-15T18:07:15.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:15.926+0000 7f87fa200640 1 -- 172.21.15.2:0/4182413845 wait complete. 2024-01-15T18:07:15.956 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:07:16.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.215+0000 7f336f200640 1 Processor -- start 2024-01-15T18:07:16.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.215+0000 7f336f200640 1 -- start start 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f33700cd900 0x7f33701605f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3370160b30 0x7f33700cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336f200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33700d0930 con 0x7f33700ce220 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336f200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3370160fc0 con 0x7f33700cd900 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33700b2380 con 0x7f3370160b30 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f33700cd900 0x7f33701605f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f33700cd900 0x7f33701605f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53278/0 (socket says 172.21.15.2:53278) 2024-01-15T18:07:16.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.217+0000 7f336de00640 1 -- 172.21.15.2:0/3458551596 learned_addr learned my addr 172.21.15.2:0/3458551596 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:16.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.218+0000 7f336de00640 1 -- 172.21.15.2:0/3458551596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3370160b30 msgr2=0x7f33700cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:16.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.218+0000 7f336de00640 1 --2- 172.21.15.2:0/3458551596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3370160b30 0x7f33700cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.218+0000 7f336de00640 1 -- 172.21.15.2:0/3458551596 >> v1:172.21.15.137:6789/0 conn(0x7f33700ce220 legacy=0x7f337006b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:16.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.218+0000 7f336de00640 1 -- 172.21.15.2:0/3458551596 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33700d0090 con 0x7f33700cd900 2024-01-15T18:07:16.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.219+0000 7f336de00640 1 --2- 172.21.15.2:0/3458551596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33700cd900 0x7f33701605f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3360009870 tx=0x7f336002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e48607baa4d73b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.219+0000 7f336ca00640 1 -- 172.21.15.2:0/3458551596 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33600317c0 con 0x7f33700cd900 2024-01-15T18:07:16.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.219+0000 7f336ca00640 1 -- 172.21.15.2:0/3458551596 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3360031920 con 0x7f33700cd900 2024-01-15T18:07:16.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.219+0000 7f336ca00640 1 -- 172.21.15.2:0/3458551596 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360031cb0 con 0x7f33700cd900 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.220+0000 7f336f200640 1 -- 172.21.15.2:0/3458551596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33700cd900 msgr2=0x7f33701605f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.220+0000 7f336f200640 1 --2- 172.21.15.2:0/3458551596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33700cd900 0x7f33701605f0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f3360009870 tx=0x7f336002b860 comp rx=0 tx=0).stop 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.221+0000 7f336f200640 1 -- 172.21.15.2:0/3458551596 shutdown_connections 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.221+0000 7f336f200640 1 --2- 172.21.15.2:0/3458551596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3370160b30 0x7f33700cdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.221+0000 7f336f200640 1 --2- 172.21.15.2:0/3458551596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f33700cd900 0x7f33701605f0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.221+0000 7f336f200640 1 -- 172.21.15.2:0/3458551596 >> 172.21.15.2:0/3458551596 conn(0x7f33700d0e10 msgr2=0x7f33700d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.221+0000 7f336f200640 1 -- 172.21.15.2:0/3458551596 shutdown_connections 2024-01-15T18:07:16.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.221+0000 7f336f200640 1 -- 172.21.15.2:0/3458551596 wait complete. 2024-01-15T18:07:16.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.223+0000 7f336f200640 1 Processor -- start 2024-01-15T18:07:16.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.224+0000 7f336f200640 1 -- start start 2024-01-15T18:07:16.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.225+0000 7f336f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33700cd900 0x7f3370150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.225+0000 7f336f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33700ce220 0x7f33701513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.225+0000 7f336f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 0x7f33700ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.225+0000 7f336f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33701537d0 con 0x7f3370160b30 2024-01-15T18:07:16.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.225+0000 7f336f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33700c7ee0 con 0x7f33700ce220 2024-01-15T18:07:16.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.225+0000 7f336f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33700c5a40 con 0x7f33700cd900 2024-01-15T18:07:16.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 0x7f33700ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 0x7f33700ca900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53294/0 (socket says 172.21.15.2:53294) 2024-01-15T18:07:16.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 -- 172.21.15.2:0/4249620644 learned_addr learned my addr 172.21.15.2:0/4249620644 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:16.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 -- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33700cd900 msgr2=0x7f3370150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33700cd900 0x7f3370150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 -- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33700ce220 msgr2=0x7f33701513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33700ce220 0x7f33701513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 -- 172.21.15.2:0/4249620644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33600094d0 con 0x7f3370160b30 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.226+0000 7f336e800640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 0x7f33700ca900 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f336800b990 tx=0x7f336800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.227+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336800cba0 con 0x7f3370160b30 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.227+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336800cd00 con 0x7f3370160b30 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.227+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33700c8130 con 0x7f3370160b30 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.227+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368014680 con 0x7f3370160b30 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.227+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33700c86f0 con 0x7f3370160b30 2024-01-15T18:07:16.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.228+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337006ca70 con 0x7f3370160b30 2024-01-15T18:07:16.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.229+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f336801d3b0 con 0x7f3370160b30 2024-01-15T18:07:16.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.230+0000 7f3366000640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f334806be90 0x7f334806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:16.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.230+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f3368088810 con 0x7f3370160b30 2024-01-15T18:07:16.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.234+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3368054e70 con 0x7f3370160b30 2024-01-15T18:07:16.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.244+0000 7f336de00640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f334806be90 0x7f334806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:16.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.267+0000 7f336de00640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f334806be90 0x7f334806e350 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3360036000 tx=0x7f3360030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:16.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.638+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f33700c8910 con 0x7f3370160b30 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.638+0000 7f3366000640 1 -- 172.21.15.2:0/4249620644 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2347 (secure 0 0 0) 0x7f3368019020 con 0x7f3370160b30 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:16.643 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 604, 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.644 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:16.645 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 935, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3771936734, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 7376293888, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 765717819392, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 54503030, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 45793241, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 14, 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 18 2024-01-15T18:07:16.646 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:16.647 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:16.648 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:16.649 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:16.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f334806be90 msgr2=0x7f334806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f334806be90 0x7f334806e350 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f3360036000 tx=0x7f3360030000 comp rx=0 tx=0).stop 2024-01-15T18:07:16.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 msgr2=0x7f33700ca900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:16.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 0x7f33700ca900 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f336800b990 tx=0x7f336800be50 comp rx=0 tx=0).stop 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 shutdown_connections 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f334806be90 0x7f334806e350 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3370160b30 0x7f33700ca900 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f33700ce220 0x7f33701513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 --2- 172.21.15.2:0/4249620644 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f33700cd900 0x7f3370150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.641+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 >> 172.21.15.2:0/4249620644 conn(0x7f33700d0e10 msgr2=0x7f33700cbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.642+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 shutdown_connections 2024-01-15T18:07:16.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:16.642+0000 7f336f200640 1 -- 172.21.15.2:0/4249620644 wait complete. 2024-01-15T18:07:17.674 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:17.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.896+0000 7fe0e5400640 1 Processor -- start 2024-01-15T18:07:17.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.896+0000 7fe0e5400640 1 -- start start 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.897+0000 7fe0e5400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe0e015c0e0 0x7fe0e00cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.897+0000 7fe0e5400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe0e015b7d0 0x7fe0e006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.897+0000 7fe0e5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c4c0 0x7fe0e006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.897+0000 7fe0e5400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e015cb00 con 0x7fe0e015b7d0 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.897+0000 7fe0e5400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e00ccec0 con 0x7fe0e015c0e0 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.897+0000 7fe0e5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e00cd1a0 con 0x7fe0e015c4c0 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe0e015b7d0 0x7fe0e006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe0e015b7d0 0x7fe0e006c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53300/0 (socket says 172.21.15.2:53300) 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 -- 172.21.15.2:0/2079065045 learned_addr learned my addr 172.21.15.2:0/2079065045 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0de800640 1 --2- 172.21.15.2:0/2079065045 >> v2:172.21.15.137:3300/0 conn(0x7fe0e015c0e0 0x7fe0e00cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0df200640 1 --2- 172.21.15.2:0/2079065045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c4c0 0x7fe0e006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 -- 172.21.15.2:0/2079065045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c4c0 msgr2=0x7fe0e006c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/2079065045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c4c0 0x7fe0e006c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 -- 172.21.15.2:0/2079065045 >> v2:172.21.15.137:3300/0 conn(0x7fe0e015c0e0 msgr2=0x7fe0e00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/2079065045 >> v2:172.21.15.137:3300/0 conn(0x7fe0e015c0e0 0x7fe0e00cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.898+0000 7fe0d5c00640 1 -- 172.21.15.2:0/2079065045 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00cd480 con 0x7fe0e015b7d0 2024-01-15T18:07:17.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.899+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/2079065045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e006c470 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00725f0 tx=0x7fe0c802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13dcb3bd2dc601a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0dde00640 1 -- 172.21.15.2:0/2079065045 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8031870 con 0x7fe0e015b7d0 2024-01-15T18:07:17.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0dde00640 1 -- 172.21.15.2:0/2079065045 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c80319d0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0dde00640 1 -- 172.21.15.2:0/2079065045 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0c8031d60 con 0x7fe0e015b7d0 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 -- 172.21.15.2:0/2079065045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 msgr2=0x7fe0e006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 --2- 172.21.15.2:0/2079065045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e006c470 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00725f0 tx=0x7fe0c802b910 comp rx=0 tx=0).stop 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 -- 172.21.15.2:0/2079065045 shutdown_connections 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 --2- 172.21.15.2:0/2079065045 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c4c0 0x7fe0e006c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 --2- 172.21.15.2:0/2079065045 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e006c470 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 --2- 172.21.15.2:0/2079065045 >> v2:172.21.15.137:3300/0 conn(0x7fe0e015c0e0 0x7fe0e00cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.900+0000 7fe0e5400640 1 -- 172.21.15.2:0/2079065045 >> 172.21.15.2:0/2079065045 conn(0x7fe0e006da20 msgr2=0x7fe0e006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.901+0000 7fe0e5400640 1 -- 172.21.15.2:0/2079065045 shutdown_connections 2024-01-15T18:07:17.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.901+0000 7fe0e5400640 1 -- 172.21.15.2:0/2079065045 wait complete. 2024-01-15T18:07:17.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.902+0000 7fe0e5400640 1 Processor -- start 2024-01-15T18:07:17.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.902+0000 7fe0e5400640 1 -- start start 2024-01-15T18:07:17.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.903+0000 7fe0e5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e01479b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.903+0000 7fe0e5400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c0e0 0x7fe0e0146e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.903+0000 7fe0e5400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0e015c4c0 0x7fe0e01f2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.903+0000 7fe0e5400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e01492a0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.903+0000 7fe0e5400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e014acf0 con 0x7fe0e015c4c0 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.903+0000 7fe0e5400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e014aa10 con 0x7fe0e015c0e0 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e01479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e01479b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53310/0 (socket says 172.21.15.2:53310) 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 -- 172.21.15.2:0/1413096484 learned_addr learned my addr 172.21.15.2:0/1413096484 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0df200640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0e015c4c0 0x7fe0e01f2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c0e0 0x7fe0e0146e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 -- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c0e0 msgr2=0x7fe0e0146e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c0e0 0x7fe0e0146e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 -- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0e015c4c0 msgr2=0x7fe0e01f2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0e015c4c0 0x7fe0e01f2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0de800640 1 -- 172.21.15.2:0/1413096484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c80094d0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.904+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c0e0 0x7fe0e0146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.905+0000 7fe0de800640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e01479b0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000d800 tx=0x7fe0d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.905+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00096a0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.905+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d0009800 con 0x7fe0e015b7d0 2024-01-15T18:07:17.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.905+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e00b29a0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.905+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d0009b90 con 0x7fe0e015b7d0 2024-01-15T18:07:17.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.905+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e00b2f60 con 0x7fe0e015b7d0 2024-01-15T18:07:17.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.906+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e00686f0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.907+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe0d0009cf0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.908+0000 7fe0d7400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0b806be90 0x7fe0b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:17.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.908+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fe0d0088bc0 con 0x7fe0e015b7d0 2024-01-15T18:07:17.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.909+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0b806be90 0x7fe0b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:17.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.910+0000 7fe0d5c00640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0b806be90 0x7fe0b806e350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8036000 tx=0x7fe0c803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:17.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:17.912+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0d0055220 con 0x7fe0e015b7d0 2024-01-15T18:07:18.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.178+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe0e006b260 con 0x7fe0e015b7d0 2024-01-15T18:07:18.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.179+0000 7fe0d7400640 1 -- 172.21.15.2:0/1413096484 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2347 (secure 0 0 0) 0x7fe0d000b440 con 0x7fe0e015b7d0 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:18.184 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 605, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:18.185 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 935, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3771936734, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 7376293888, 2024-01-15T18:07:18.186 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 765717819392, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 54503030, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 45793241, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 14, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 18 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:18.187 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:18.188 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:18.189 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:18.190 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-15T18:07:18.190 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0, 2024-01-15T18:07:18.190 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:18.190 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:18.190 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:18.190 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0b806be90 msgr2=0x7fe0b806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0b806be90 0x7fe0b806e350 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe0c8036000 tx=0x7fe0c803c040 comp rx=0 tx=0).stop 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 msgr2=0x7fe0e01479b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e01479b0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fe0d000d800 tx=0x7fe0d000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 shutdown_connections 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe0b806be90 0x7fe0b806e350 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0e015c4c0 0x7fe0e01f2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0e015c0e0 0x7fe0e0146e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 --2- 172.21.15.2:0/1413096484 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0e015b7d0 0x7fe0e01479b0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 >> 172.21.15.2:0/1413096484 conn(0x7fe0e006da20 msgr2=0x7fe0e01512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 shutdown_connections 2024-01-15T18:07:18.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:18.182+0000 7fe0e5400640 1 -- 172.21.15.2:0/1413096484 wait complete. 2024-01-15T18:07:19.208 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:19.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.425+0000 7f713b200640 1 Processor -- start 2024-01-15T18:07:19.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.425+0000 7f713b200640 1 -- start start 2024-01-15T18:07:19.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.427+0000 7f713b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15a180 0x7f713c06e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.427+0000 7f713b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f713c06ea20 0x7f713c071280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.427+0000 7f713b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f713c0b35c0 con 0x7f713c0717c0 2024-01-15T18:07:19.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.427+0000 7f713b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f713c071c50 con 0x7f713c06ea20 2024-01-15T18:07:19.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.427+0000 7f713b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c06ef30 con 0x7f713c15a180 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f713c06ea20 0x7f713c071280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f713c06ea20 0x7f713c071280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:34554/0 (socket says 172.21.15.2:34554) 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 -- 172.21.15.2:0/1746234208 learned_addr learned my addr 172.21.15.2:0/1746234208 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139e00640 1 --2- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15a180 0x7f713c06e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 -- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15a180 msgr2=0x7f713c06e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 --2- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15a180 0x7f713c06e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 -- 172.21.15.2:0/1746234208 >> v1:172.21.15.137:6789/0 conn(0x7f713c0717c0 legacy=0x7f713c06d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.428+0000 7f7139400640 1 -- 172.21.15.2:0/1746234208 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0b2f70 con 0x7f713c06ea20 2024-01-15T18:07:19.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.430+0000 7f7139400640 1 --2- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c071280 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7124009870 tx=0x7f712402db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cc7ab459e5cc3d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:19.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.430+0000 7f7138a00640 1 -- 172.21.15.2:0/1746234208 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712402e940 con 0x7f713c06ea20 2024-01-15T18:07:19.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f7138a00640 1 -- 172.21.15.2:0/1746234208 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712402eaa0 con 0x7f713c06ea20 2024-01-15T18:07:19.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f7138a00640 1 -- 172.21.15.2:0/1746234208 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712402ee30 con 0x7f713c06ea20 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 -- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 msgr2=0x7f713c071280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 --2- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c071280 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7124009870 tx=0x7f712402db90 comp rx=0 tx=0).stop 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 -- 172.21.15.2:0/1746234208 shutdown_connections 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 --2- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c071280 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 --2- 172.21.15.2:0/1746234208 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c15a180 0x7f713c06e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 -- 172.21.15.2:0/1746234208 >> 172.21.15.2:0/1746234208 conn(0x7f713c0cfd60 msgr2=0x7f713c0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 -- 172.21.15.2:0/1746234208 shutdown_connections 2024-01-15T18:07:19.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.431+0000 7f713b200640 1 -- 172.21.15.2:0/1746234208 wait complete. 2024-01-15T18:07:19.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.433+0000 7f713b200640 1 Processor -- start 2024-01-15T18:07:19.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.433+0000 7f713b200640 1 -- start start 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.434+0000 7f713b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.434+0000 7f713b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c0717c0 0x7f713c13d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.434+0000 7f713b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 0x7f713c13b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.434+0000 7f713b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c13dfc0 con 0x7f713c13db10 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.434+0000 7f713b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c13cd30 con 0x7f713c06ea20 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.434+0000 7f713b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c14eec0 con 0x7f713c0717c0 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 0x7f713c13b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 0x7f713c13b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53316/0 (socket says 172.21.15.2:53316) 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 -- 172.21.15.2:0/3882168221 learned_addr learned my addr 172.21.15.2:0/3882168221 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:19.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f7139e00640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f7139400640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c0717c0 0x7f713c13d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 -- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c0717c0 msgr2=0x7f713c13d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c0717c0 0x7f713c13d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 -- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 msgr2=0x7f713c158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 -- 172.21.15.2:0/3882168221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240094d0 con 0x7f713c13db10 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.435+0000 7f713a800640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 0x7f713c13b540 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f713400c730 tx=0x7f713400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.436+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134003b70 con 0x7f713c13db10 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.436+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7134003cf0 con 0x7f713c13db10 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.436+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713c13c120 con 0x7f713c13db10 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.436+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7134005310 con 0x7f713c13db10 2024-01-15T18:07:19.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.436+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713c13c6e0 con 0x7f713c13db10 2024-01-15T18:07:19.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.437+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713c13cee0 con 0x7f713c13db10 2024-01-15T18:07:19.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.438+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7134005680 con 0x7f713c13db10 2024-01-15T18:07:19.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.439+0000 7f7132000640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f711406bec0 0x7f711406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:19.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.439+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f7134088810 con 0x7f713c13db10 2024-01-15T18:07:19.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.440+0000 7f7139e00640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f711406bec0 0x7f711406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:19.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.441+0000 7f7139e00640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f711406bec0 0x7f711406e380 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f712c009a30 tx=0x7f712c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:19.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.443+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f713c13cee0 con 0x7f713c13db10 2024-01-15T18:07:19.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.713+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f713c13cee0 con 0x7f713c13db10 2024-01-15T18:07:19.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.714+0000 7f7132000640 1 -- 172.21.15.2:0/3882168221 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f713c13cee0 con 0x7f713c13db10 2024-01-15T18:07:19.718 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:19.718 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:19.719 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 607, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:19.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1216, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 4954758099, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 10320764928, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 762773348352, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 139057102, 2024-01-15T18:07:19.721 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 144315232, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 33, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 40 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:19.722 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.723 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.16664999723434448, 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:19.724 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:19.725 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:19.725 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:19.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f711406bec0 msgr2=0x7f711406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f711406bec0 0x7f711406e380 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f712c009a30 tx=0x7f712c008040 comp rx=0 tx=0).stop 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 msgr2=0x7f713c13b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 0x7f713c13b540 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f713400c730 tx=0x7f713400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 shutdown_connections 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f711406bec0 0x7f711406e380 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.716+0000 7f713b200640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f713c13db10 0x7f713c13b540 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.717+0000 7f713b200640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f713c0717c0 0x7f713c13d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.717+0000 7f713b200640 1 --2- 172.21.15.2:0/3882168221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f713c06ea20 0x7f713c158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.717+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 >> 172.21.15.2:0/3882168221 conn(0x7f713c0cfd60 msgr2=0x7f713c0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.717+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 shutdown_connections 2024-01-15T18:07:19.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:19.717+0000 7f713b200640 1 -- 172.21.15.2:0/3882168221 wait complete. 2024-01-15T18:07:20.741 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:20.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.960+0000 7f91cce00640 1 Processor -- start 2024-01-15T18:07:20.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.961+0000 7f91cce00640 1 -- start start 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91cce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f91c80d3090 0x7f91c80d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91cce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d1680 0x7f91c80d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91cce00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91c80d3a60 con 0x7f91c815d340 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91cce00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91c80ce430 con 0x7f91c80d3090 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91cce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c80b3de0 con 0x7f91c80d1680 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91c6a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f91c80d3090 0x7f91c80d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.962+0000 7f91c7400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f91c815d340 0x7f91c80cdd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57764/0 (socket says 172.21.15.2:57764) 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c7400640 1 -- 172.21.15.2:0/2685912321 learned_addr learned my addr 172.21.15.2:0/2685912321 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c7e00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d1680 0x7f91c80d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c6a00640 1 -- 172.21.15.2:0/2685912321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d1680 msgr2=0x7f91c80d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:20.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c6a00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d1680 0x7f91c80d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c6a00640 1 -- 172.21.15.2:0/2685912321 >> v1:172.21.15.2:6789/0 conn(0x7f91c815d340 legacy=0x7f91c80cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c6a00640 1 -- 172.21.15.2:0/2685912321 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91c80b40c0 con 0x7f91c80d3090 2024-01-15T18:07:20.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c6000640 1 -- 172.21.15.2:0/2685912321 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 188187613 0 0) 0x7f91c80d3a60 con 0x7f91c815d340 2024-01-15T18:07:20.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.963+0000 7f91c7e00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d1680 0x7f91c80d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.964+0000 7f91c6a00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d3090 0x7f91c80d3520 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009d90 tx=0x7f91b002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7983d6c9973a416 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.964+0000 7f91c6000640 1 -- 172.21.15.2:0/2685912321 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b002cc50 con 0x7f91c80d3090 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.964+0000 7f91c6000640 1 -- 172.21.15.2:0/2685912321 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b002cdb0 con 0x7f91c80d3090 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.964+0000 7f91c6000640 1 -- 172.21.15.2:0/2685912321 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0034660 con 0x7f91c80d3090 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 -- 172.21.15.2:0/2685912321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d3090 msgr2=0x7f91c80d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d3090 0x7f91c80d3520 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f91b0009d90 tx=0x7f91b002bea0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 -- 172.21.15.2:0/2685912321 shutdown_connections 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d1680 0x7f91c80d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 --2- 172.21.15.2:0/2685912321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d3090 0x7f91c80d3520 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 -- 172.21.15.2:0/2685912321 >> 172.21.15.2:0/2685912321 conn(0x7f91c808caf0 msgr2=0x7f91c809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:20.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 -- 172.21.15.2:0/2685912321 shutdown_connections 2024-01-15T18:07:20.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.965+0000 7f91cce00640 1 -- 172.21.15.2:0/2685912321 wait complete. 2024-01-15T18:07:20.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.967+0000 7f91cce00640 1 Processor -- start 2024-01-15T18:07:20.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.967+0000 7f91cce00640 1 -- start start 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91cce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 0x7f91c815fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91cce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d3090 0x7f91c81f6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91cce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91c815d340 0x7f91c80b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91cce00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c81f6620 con 0x7f91c80d1680 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91cce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91c81601a0 con 0x7f91c815d340 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91cce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91c8160480 con 0x7f91c80d3090 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 0x7f91c815fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91c7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 0x7f91c815fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53332/0 (socket says 172.21.15.2:53332) 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.968+0000 7f91c7400640 1 -- 172.21.15.2:0/3886330988 learned_addr learned my addr 172.21.15.2:0/3886330988 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c7400640 1 -- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d3090 msgr2=0x7f91c81f6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c6a00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d3090 0x7f91c81f6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c7400640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d3090 0x7f91c81f6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c7400640 1 -- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91c815d340 msgr2=0x7f91c80b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c7400640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91c815d340 0x7f91c80b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c7400640 1 -- 172.21.15.2:0/3886330988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b0009a70 con 0x7f91c80d1680 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91c7400640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 0x7f91c815fb70 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f91b800c4d0 tx=0x7f91b800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b80074e0 con 0x7f91c80d1680 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b8007660 con 0x7f91c80d1680 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b8004a20 con 0x7f91c80d1680 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91c8160760 con 0x7f91c80d1680 2024-01-15T18:07:20.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.969+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91c81f6ff0 con 0x7f91c80d1680 2024-01-15T18:07:20.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.971+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91c8074770 con 0x7f91c80d1680 2024-01-15T18:07:20.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.971+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f91b8004b80 con 0x7f91c80d1680 2024-01-15T18:07:20.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.973+0000 7f91bf400640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91a006be90 0x7f91a006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:20.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.973+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f91b8087ac0 con 0x7f91c80d1680 2024-01-15T18:07:20.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.973+0000 7f91c6a00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91a006be90 0x7f91a006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:20.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.975+0000 7f91c6a00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91a006be90 0x7f91a006e350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f91b0036000 tx=0x7f91b0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:20.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:20.977+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91b8054060 con 0x7f91c80d1680 2024-01-15T18:07:21.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.240+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f91c80678e0 con 0x7f91c80d1680 2024-01-15T18:07:21.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.241+0000 7f91bf400640 1 -- 172.21.15.2:0/3886330988 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f91b8018090 con 0x7f91c80d1680 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:21.246 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 608, 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:21.247 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:21.248 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1216, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 4954758099, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 10320764928, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 762773348352, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 139057102, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 144315232, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 33, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 40 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:21.249 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:21.250 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:21.251 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.16664999723434448, 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:21.252 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91a006be90 msgr2=0x7f91a006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91a006be90 0x7f91a006e350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f91b0036000 tx=0x7f91b0030000 comp rx=0 tx=0).stop 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 msgr2=0x7f91c815fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 0x7f91c815fb70 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f91b800c4d0 tx=0x7f91b800c990 comp rx=0 tx=0).stop 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 shutdown_connections 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91a006be90 0x7f91a006e350 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91c815d340 0x7f91c80b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91c80d3090 0x7f91c81f6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:21.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 --2- 172.21.15.2:0/3886330988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91c80d1680 0x7f91c815fb70 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:21.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 >> 172.21.15.2:0/3886330988 conn(0x7f91c808caf0 msgr2=0x7f91c80cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:21.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 shutdown_connections 2024-01-15T18:07:21.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:21.244+0000 7f91cce00640 1 -- 172.21.15.2:0/3886330988 wait complete. 2024-01-15T18:07:22.270 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:22.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.488+0000 7fc7f1400640 1 Processor -- start 2024-01-15T18:07:22.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.488+0000 7fc7f1400640 1 -- start start 2024-01-15T18:07:22.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7f1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc7ec143810 0x7fc7ec06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7f1400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc7ec06e010 0x7fc7ec0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7f1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec0b2080 0x7fc7ec0700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7f1400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec0b3520 con 0x7fc7ec143810 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7f1400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec06e470 con 0x7fc7ec06e010 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7f1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec0ccbe0 con 0x7fc7ec0b2080 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc7ec143810 0x7fc7ec06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc7ec143810 0x7fc7ec06dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53336/0 (socket says 172.21.15.2:53336) 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 -- 172.21.15.2:0/1252727272 learned_addr learned my addr 172.21.15.2:0/1252727272 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:22.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 -- 172.21.15.2:0/1252727272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec0b2080 msgr2=0x7fc7ec0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:22.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/1252727272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec0b2080 0x7fc7ec0700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 -- 172.21.15.2:0/1252727272 >> v2:172.21.15.137:3300/0 conn(0x7fc7ec06e010 msgr2=0x7fc7ec0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:22.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.490+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/1252727272 >> v2:172.21.15.137:3300/0 conn(0x7fc7ec06e010 0x7fc7ec0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.491+0000 7fc7ebe00640 1 -- 172.21.15.2:0/1252727272 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec143bf0 con 0x7fc7ec143810 2024-01-15T18:07:22.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.492+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/1252727272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec143810 0x7fc7ec06dad0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0cbba0 tx=0x7fc7dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9093c5db692582a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:22.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.492+0000 7fc7eaa00640 1 -- 172.21.15.2:0/1252727272 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0317e0 con 0x7fc7ec143810 2024-01-15T18:07:22.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.492+0000 7fc7eaa00640 1 -- 172.21.15.2:0/1252727272 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc031940 con 0x7fc7ec143810 2024-01-15T18:07:22.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.492+0000 7fc7eaa00640 1 -- 172.21.15.2:0/1252727272 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc031cd0 con 0x7fc7ec143810 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 -- 172.21.15.2:0/1252727272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec143810 msgr2=0x7fc7ec06dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 --2- 172.21.15.2:0/1252727272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec143810 0x7fc7ec06dad0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7fc7ec0cbba0 tx=0x7fc7dc02b880 comp rx=0 tx=0).stop 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 -- 172.21.15.2:0/1252727272 shutdown_connections 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 --2- 172.21.15.2:0/1252727272 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec0b2080 0x7fc7ec0700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 --2- 172.21.15.2:0/1252727272 >> v2:172.21.15.137:3300/0 conn(0x7fc7ec06e010 0x7fc7ec0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 --2- 172.21.15.2:0/1252727272 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec143810 0x7fc7ec06dad0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 -- 172.21.15.2:0/1252727272 >> 172.21.15.2:0/1252727272 conn(0x7fc7ec0b0840 msgr2=0x7fc7ec0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 -- 172.21.15.2:0/1252727272 shutdown_connections 2024-01-15T18:07:22.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.493+0000 7fc7f1400640 1 -- 172.21.15.2:0/1252727272 wait complete. 2024-01-15T18:07:22.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.495+0000 7fc7f1400640 1 Processor -- start 2024-01-15T18:07:22.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.495+0000 7fc7f1400640 1 -- start start 2024-01-15T18:07:22.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7f1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7ec06e010 0x7fc7ec135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7f1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 0x7fc7ec1e61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7f1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec143810 0x7fc7ec1e0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7f1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec135670 con 0x7fc7ec0b2080 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7f1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec1e4e70 con 0x7fc7ec06e010 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7f1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec1e5110 con 0x7fc7ec143810 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7eb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 0x7fc7ec1e61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7eb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 0x7fc7ec1e61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53352/0 (socket says 172.21.15.2:53352) 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.496+0000 7fc7eb400640 1 -- 172.21.15.2:0/3635319967 learned_addr learned my addr 172.21.15.2:0/3635319967 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7ec06e010 0x7fc7ec135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7eb400640 1 -- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec143810 msgr2=0x7fc7ec1e0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7eb400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec143810 0x7fc7ec1e0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7eb400640 1 -- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7ec06e010 msgr2=0x7fc7ec135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7eb400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7ec06e010 0x7fc7ec135130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7eb400640 1 -- 172.21.15.2:0/3635319967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0094d0 con 0x7fc7ec0b2080 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7ec06e010 0x7fc7ec135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:22.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7eb400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 0x7fc7ec1e61b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b4a0 tx=0x7fc7d400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4010610 con 0x7fc7ec0b2080 2024-01-15T18:07:22.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d4010770 con 0x7fc7ec0b2080 2024-01-15T18:07:22.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.497+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec1e53f0 con 0x7fc7ec0b2080 2024-01-15T18:07:22.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.498+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4010b20 con 0x7fc7ec0b2080 2024-01-15T18:07:22.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.498+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec1e59b0 con 0x7fc7ec0b2080 2024-01-15T18:07:22.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.499+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ec0b24f0 con 0x7fc7ec0b2080 2024-01-15T18:07:22.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.500+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc7d4010c80 con 0x7fc7ec0b2080 2024-01-15T18:07:22.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.501+0000 7fc7e3e00640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7c406be90 0x7fc7c406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:22.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.501+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fc7d4088510 con 0x7fc7ec0b2080 2024-01-15T18:07:22.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.501+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7c406be90 0x7fc7c406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:22.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.503+0000 7fc7ebe00640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7c406be90 0x7fc7c406e350 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc036000 tx=0x7fc7dc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:22.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.505+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7d4054ae0 con 0x7fc7ec0b2080 2024-01-15T18:07:22.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.775+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc7ec074770 con 0x7fc7ec0b2080 2024-01-15T18:07:22.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.776+0000 7fc7e3e00640 1 -- 172.21.15.2:0/3635319967 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fc7d401e020 con 0x7fc7ec0b2080 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:22.781 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 610, 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:22.782 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:22.783 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1315, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 5369994296, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 11188985856, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 761905127424, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 178883091, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 178892388, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 43, 2024-01-15T18:07:22.784 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 46 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:22.785 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:22.786 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.16664999723434448, 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:22.787 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:22.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7c406be90 msgr2=0x7fc7c406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7c406be90 0x7fc7c406e350 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc036000 tx=0x7fc7dc03d040 comp rx=0 tx=0).stop 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 msgr2=0x7fc7ec1e61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 0x7fc7ec1e61b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b4a0 tx=0x7fc7d400b960 comp rx=0 tx=0).stop 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 shutdown_connections 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc7c406be90 0x7fc7c406e350 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7ec143810 0x7fc7ec1e0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7ec0b2080 0x7fc7ec1e61b0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 --2- 172.21.15.2:0/3635319967 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7ec06e010 0x7fc7ec135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 >> 172.21.15.2:0/3635319967 conn(0x7fc7ec0b0840 msgr2=0x7fc7ec06c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.779+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 shutdown_connections 2024-01-15T18:07:22.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:22.780+0000 7fc7f1400640 1 -- 172.21.15.2:0/3635319967 wait complete. 2024-01-15T18:07:23.804 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:24.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.079+0000 7f9bc6800640 1 Processor -- start 2024-01-15T18:07:24.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.080+0000 7f9bc6800640 1 -- start start 2024-01-15T18:07:24.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.081+0000 7f9bc6800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bc013ce30 0x7f9bc015dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.081+0000 7f9bc6800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.081+0000 7f9bc6800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc00aeab0 con 0x7f9bc015e170 2024-01-15T18:07:24.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.081+0000 7f9bc6800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc015e620 con 0x7f9bc013ce30 2024-01-15T18:07:24.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.081+0000 7f9bc6800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00ce6d0 con 0x7f9bc00ce170 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bc013ce30 0x7f9bc015dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bb6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9bc015e170 0x7f9bc00cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48672/0 (socket says 172.21.15.2:48672) 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bb6800640 1 -- 172.21.15.2:0/458525553 learned_addr learned my addr 172.21.15.2:0/458525553 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbfe00640 1 --2- 172.21.15.2:0/458525553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbf400640 1 -- 172.21.15.2:0/458525553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 msgr2=0x7f9bc0070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbf400640 1 --2- 172.21.15.2:0/458525553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0070ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbf400640 1 -- 172.21.15.2:0/458525553 >> v1:172.21.15.2:6789/0 conn(0x7f9bc015e170 legacy=0x7f9bc00cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbf400640 1 -- 172.21.15.2:0/458525553 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc0070380 con 0x7f9bc013ce30 2024-01-15T18:07:24.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.082+0000 7f9bbea00640 1 -- 172.21.15.2:0/458525553 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319406218 0 0) 0x7f9bc00aeab0 con 0x7f9bc015e170 2024-01-15T18:07:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.084+0000 7f9bbf400640 1 --2- 172.21.15.2:0/458525553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc013ce30 0x7f9bc015dc30 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00097a0 tx=0x7f9bb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f445c75a5132bc5c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.085+0000 7f9bbea00640 1 -- 172.21.15.2:0/458525553 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0031830 con 0x7f9bc013ce30 2024-01-15T18:07:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.085+0000 7f9bbea00640 1 -- 172.21.15.2:0/458525553 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb0031990 con 0x7f9bc013ce30 2024-01-15T18:07:24.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.085+0000 7f9bbea00640 1 -- 172.21.15.2:0/458525553 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0031d20 con 0x7f9bc013ce30 2024-01-15T18:07:24.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 -- 172.21.15.2:0/458525553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc013ce30 msgr2=0x7f9bc015dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 --2- 172.21.15.2:0/458525553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc013ce30 0x7f9bc015dc30 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f9bb00097a0 tx=0x7f9bb002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 -- 172.21.15.2:0/458525553 shutdown_connections 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 --2- 172.21.15.2:0/458525553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 --2- 172.21.15.2:0/458525553 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc013ce30 0x7f9bc015dc30 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 -- 172.21.15.2:0/458525553 >> 172.21.15.2:0/458525553 conn(0x7f9bc006da20 msgr2=0x7f9bc006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 -- 172.21.15.2:0/458525553 shutdown_connections 2024-01-15T18:07:24.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.086+0000 7f9bc6800640 1 -- 172.21.15.2:0/458525553 wait complete. 2024-01-15T18:07:24.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.088+0000 7f9bc6800640 1 Processor -- start 2024-01-15T18:07:24.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.089+0000 7f9bc6800640 1 -- start start 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bc6800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bc6800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc013ce30 0x7f9bc0141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bc6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 0x7f9bc01f30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bc6800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc013f1c0 con 0x7f9bc015e170 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bc6800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc01418a0 con 0x7f9bc013ce30 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bc6800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc0141b40 con 0x7f9bc00ce170 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bbfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 0x7f9bc01f30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bbfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 0x7f9bc01f30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57354/0 (socket says 172.21.15.2:57354) 2024-01-15T18:07:24.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.090+0000 7f9bbfe00640 1 -- 172.21.15.2:0/3646449020 learned_addr learned my addr 172.21.15.2:0/3646449020 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:24.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.091+0000 7f9bbfe00640 1 -- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 msgr2=0x7f9bc0138450 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.091+0000 7f9bbfe00640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0138450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.091+0000 7f9bbfe00640 1 -- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc013ce30 msgr2=0x7f9bc0141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.091+0000 7f9bbfe00640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc013ce30 0x7f9bc0141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.091+0000 7f9bbfe00640 1 -- 172.21.15.2:0/3646449020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb0009400 con 0x7f9bc015e170 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.091+0000 7f9bbfe00640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 0x7f9bc01f30f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007e90 tx=0x7f9bb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.092+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8014420 con 0x7f9bc015e170 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.092+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb80145a0 con 0x7f9bc015e170 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.092+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8014950 con 0x7f9bc015e170 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.092+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc0143030 con 0x7f9bc015e170 2024-01-15T18:07:24.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.092+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc01435f0 con 0x7f9bc015e170 2024-01-15T18:07:24.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.094+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc01437b0 con 0x7f9bc015e170 2024-01-15T18:07:24.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.096+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9bb8014ab0 con 0x7f9bc015e170 2024-01-15T18:07:24.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.097+0000 7f9bb7e00640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b9806be90 0x7f9b9806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:24.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.097+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f9bb8088e80 con 0x7f9bc015e170 2024-01-15T18:07:24.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.108+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bb80557a0 con 0x7f9bc015e170 2024-01-15T18:07:24.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.114+0000 7f9bbf400640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b9806be90 0x7f9b9806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:24.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.119+0000 7f9bbf400640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b9806be90 0x7f9b9806e350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0036000 tx=0x7f9bb0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:24.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.536+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9bc00686f0 con 0x7f9bc015e170 2024-01-15T18:07:24.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.541+0000 7f9bb7e00640 1 -- 172.21.15.2:0/3646449020 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f9bb8058dc0 con 0x7f9bc015e170 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:24.549 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 612, 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:24.550 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:24.551 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1315, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 5369994296, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 11188985856, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 761905127424, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 178883091, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 178892388, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 43, 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 46 2024-01-15T18:07:24.552 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:24.553 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:24.554 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.16664999723434448, 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:24.555 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.547+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b9806be90 msgr2=0x7f9b9806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.547+0000 7f9bc6800640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b9806be90 0x7f9b9806e350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0036000 tx=0x7f9bb0008030 comp rx=0 tx=0).stop 2024-01-15T18:07:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 msgr2=0x7f9bc01f30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:24.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 0x7f9bc01f30f0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007e90 tx=0x7f9bb800e4b0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 shutdown_connections 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9b9806be90 0x7f9b9806e350 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bc015e170 0x7f9bc01f30f0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bc013ce30 0x7f9bc0141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 --2- 172.21.15.2:0/3646449020 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bc00ce170 0x7f9bc0138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.548+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 >> 172.21.15.2:0/3646449020 conn(0x7f9bc006da20 msgr2=0x7f9bc015c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.549+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 shutdown_connections 2024-01-15T18:07:24.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:24.551+0000 7f9bc6800640 1 -- 172.21.15.2:0/3646449020 wait complete. 2024-01-15T18:07:25.593 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:25.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.872+0000 7fce7de00640 1 Processor -- start 2024-01-15T18:07:25.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.873+0000 7fce7de00640 1 -- start start 2024-01-15T18:07:25.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.875+0000 7fce7de00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fce7815aa10 0x7fce780b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.875+0000 7fce7de00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce78150c50 con 0x7fce7806b070 2024-01-15T18:07:25.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.875+0000 7fce7de00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce780b29d0 con 0x7fce7806abc0 2024-01-15T18:07:25.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.875+0000 7fce7de00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce780b2cb0 con 0x7fce7815aa10 2024-01-15T18:07:25.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.876+0000 7fce76a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fce7815aa10 0x7fce780b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.876+0000 7fce76a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fce7815aa10 0x7fce780b20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:60992/0 (socket says 172.21.15.2:60992) 2024-01-15T18:07:25.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.876+0000 7fce76a00640 1 -- 172.21.15.2:0/1007674148 learned_addr learned my addr 172.21.15.2:0/1007674148 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:25.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.877+0000 7fce76a00640 1 -- 172.21.15.2:0/1007674148 >> v1:172.21.15.137:6789/0 conn(0x7fce7806abc0 legacy=0x7fce78150540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.877+0000 7fce76a00640 1 -- 172.21.15.2:0/1007674148 >> v1:172.21.15.2:6789/0 conn(0x7fce7806b070 legacy=0x7fce78070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.877+0000 7fce75600640 1 -- 172.21.15.2:0/1007674148 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540808707 0 0) 0x7fce78150c50 con 0x7fce7806b070 2024-01-15T18:07:25.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.877+0000 7fce76a00640 1 -- 172.21.15.2:0/1007674148 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce780b2f90 con 0x7fce7815aa10 2024-01-15T18:07:25.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.879+0000 7fce76a00640 1 --2- 172.21.15.2:0/1007674148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 0x7fce780b20d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce70009e50 tx=0x7fce7002c010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9eaa25bdee819137 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.879+0000 7fce75600640 1 -- 172.21.15.2:0/1007674148 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7002cbb0 con 0x7fce7815aa10 2024-01-15T18:07:25.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.879+0000 7fce75600640 1 -- 172.21.15.2:0/1007674148 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7002cd10 con 0x7fce7815aa10 2024-01-15T18:07:25.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.879+0000 7fce75600640 1 -- 172.21.15.2:0/1007674148 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce70034610 con 0x7fce7815aa10 2024-01-15T18:07:25.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.880+0000 7fce7de00640 1 -- 172.21.15.2:0/1007674148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 msgr2=0x7fce780b20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.880+0000 7fce7de00640 1 --2- 172.21.15.2:0/1007674148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 0x7fce780b20d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fce70009e50 tx=0x7fce7002c010 comp rx=0 tx=0).stop 2024-01-15T18:07:25.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.881+0000 7fce7de00640 1 -- 172.21.15.2:0/1007674148 shutdown_connections 2024-01-15T18:07:25.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.881+0000 7fce7de00640 1 --2- 172.21.15.2:0/1007674148 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 0x7fce780b20d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.881+0000 7fce7de00640 1 -- 172.21.15.2:0/1007674148 >> 172.21.15.2:0/1007674148 conn(0x7fce7806da20 msgr2=0x7fce7806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:25.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.881+0000 7fce7de00640 1 -- 172.21.15.2:0/1007674148 shutdown_connections 2024-01-15T18:07:25.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.881+0000 7fce7de00640 1 -- 172.21.15.2:0/1007674148 wait complete. 2024-01-15T18:07:25.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.883+0000 7fce7de00640 1 Processor -- start 2024-01-15T18:07:25.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.884+0000 7fce7de00640 1 -- start start 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.885+0000 7fce7de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 0x7fce780ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.885+0000 7fce7de00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce7806b070 0x7fce7813d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.885+0000 7fce7de00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 0x7fce7813d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.885+0000 7fce7de00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7813da80 con 0x7fce7806abc0 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.885+0000 7fce7de00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce78160ac0 con 0x7fce7815aa10 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.885+0000 7fce7de00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce780b1a80 con 0x7fce7806b070 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.886+0000 7fce76a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 0x7fce780ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.886+0000 7fce76a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 0x7fce780ae9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57358/0 (socket says 172.21.15.2:57358) 2024-01-15T18:07:25.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.886+0000 7fce76a00640 1 -- 172.21.15.2:0/1623357951 learned_addr learned my addr 172.21.15.2:0/1623357951 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.887+0000 7fce76a00640 1 -- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce7806b070 msgr2=0x7fce7813d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.887+0000 7fce76a00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce7806b070 0x7fce7813d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.887+0000 7fce76a00640 1 -- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 msgr2=0x7fce7813d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.887+0000 7fce76a00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 0x7fce7813d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:25.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.887+0000 7fce76a00640 1 -- 172.21.15.2:0/1623357951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70009b30 con 0x7fce7806abc0 2024-01-15T18:07:25.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.887+0000 7fce76a00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 0x7fce780ae9a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fce70009e50 tx=0x7fce70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.889+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7003e450 con 0x7fce7806abc0 2024-01-15T18:07:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.889+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7003e5f0 con 0x7fce7806abc0 2024-01-15T18:07:25.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.889+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7003e9a0 con 0x7fce7806abc0 2024-01-15T18:07:25.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.889+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce780b14c0 con 0x7fce7806abc0 2024-01-15T18:07:25.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.890+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce78160750 con 0x7fce7806abc0 2024-01-15T18:07:25.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.892+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce780686f0 con 0x7fce7806abc0 2024-01-15T18:07:25.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.895+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fce7003eb00 con 0x7fce7806abc0 2024-01-15T18:07:25.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.897+0000 7fce6f400640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce5006be90 0x7fce5006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:25.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.897+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fce700ab630 con 0x7fce7806abc0 2024-01-15T18:07:25.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.905+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce70078010 con 0x7fce7806abc0 2024-01-15T18:07:25.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.911+0000 7fce76000640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce5006be90 0x7fce5006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:25.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:25.915+0000 7fce76000640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce5006be90 0x7fce5006e350 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fce68000f60 tx=0x7fce680079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:26.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.332+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fce78071c30 con 0x7fce7806abc0 2024-01-15T18:07:26.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.333+0000 7fce6f400640 1 -- 172.21.15.2:0/1623357951 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fce7003edb0 con 0x7fce7806abc0 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:26.338 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 614, 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:26.339 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:26.340 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1493, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 6116657144, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 11956678656, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 761137434624, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 250859866, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 241093431, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 60, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 63 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:26.341 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:26.342 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:26.343 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-15T18:07:26.344 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.16664999723434448, 2024-01-15T18:07:26.344 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:26.344 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:26.344 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:26.344 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce5006be90 msgr2=0x7fce5006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce5006be90 0x7fce5006e350 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fce68000f60 tx=0x7fce680079b0 comp rx=0 tx=0).stop 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 msgr2=0x7fce780ae9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 0x7fce780ae9a0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fce70009e50 tx=0x7fce70004a40 comp rx=0 tx=0).stop 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 shutdown_connections 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce5006be90 0x7fce5006e350 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce7815aa10 0x7fce7813d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce7806b070 0x7fce7813d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 --2- 172.21.15.2:0/1623357951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce7806abc0 0x7fce780ae9a0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 >> 172.21.15.2:0/1623357951 conn(0x7fce7806da20 msgr2=0x7fce7806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 shutdown_connections 2024-01-15T18:07:26.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:26.336+0000 7fce7de00640 1 -- 172.21.15.2:0/1623357951 wait complete. 2024-01-15T18:07:27.360 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:27.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.618+0000 7fe078000640 1 Processor -- start 2024-01-15T18:07:27.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.618+0000 7fe078000640 1 -- start start 2024-01-15T18:07:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.621+0000 7fe078000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe07015e340 0x7fe0700cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.621+0000 7fe078000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0700aeab0 con 0x7fe07013ce30 2024-01-15T18:07:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.621+0000 7fe078000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe070070380 con 0x7fe0700cdfa0 2024-01-15T18:07:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.621+0000 7fe078000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0700ce6d0 con 0x7fe07015e340 2024-01-15T18:07:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.622+0000 7fe075600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe07013ce30 0x7fe07015dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48678/0 (socket says 172.21.15.2:48678) 2024-01-15T18:07:27.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.622+0000 7fe075600640 1 -- 172.21.15.2:0/3682404015 learned_addr learned my addr 172.21.15.2:0/3682404015 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:27.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730346784 0 0) 0x7fe0700aeab0 con 0x7fe07013ce30 2024-01-15T18:07:27.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe050003520 con 0x7fe07013ce30 2024-01-15T18:07:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4116635644 0 0) 0x7fe050003520 con 0x7fe07013ce30 2024-01-15T18:07:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 >> v2:172.21.15.137:3300/0 conn(0x7fe07015e340 msgr2=0x7fe0700cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 --2- 172.21.15.2:0/3682404015 >> v2:172.21.15.137:3300/0 conn(0x7fe07015e340 0x7fe0700cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 >> v1:172.21.15.137:6789/0 conn(0x7fe0700cdfa0 legacy=0x7fe070070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0700ce9b0 con 0x7fe07013ce30 2024-01-15T18:07:27.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe060001670 con 0x7fe07013ce30 2024-01-15T18:07:27.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.623+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 >> v1:172.21.15.2:6789/0 conn(0x7fe07013ce30 legacy=0x7fe07015dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:27.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe06fe00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe050003970 0x7fe050003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe06fe00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0500047c0 0x7fe050004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe06fe00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0500052a0 0x7fe05000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0700ce6d0 con 0x7fe0500047c0 2024-01-15T18:07:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05000c1a0 con 0x7fe0500052a0 2024-01-15T18:07:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05000c360 con 0x7fe050003970 2024-01-15T18:07:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe076000640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0500052a0 0x7fe05000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.624+0000 7fe074c00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0500047c0 0x7fe050004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:27.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.625+0000 7fe075600640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe050003970 0x7fe050003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:27.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.625+0000 7fe074c00640 1 -- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe050003970 msgr2=0x7fe050003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:27.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.625+0000 7fe074c00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe050003970 0x7fe050003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.625+0000 7fe074c00640 1 -- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0500052a0 msgr2=0x7fe05000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:27.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.625+0000 7fe074c00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe0500052a0 0x7fe05000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.626+0000 7fe074c00640 1 -- 172.21.15.2:0/3682404015 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0700ce9b0 con 0x7fe0500047c0 2024-01-15T18:07:27.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.626+0000 7fe076000640 1 -- 172.21.15.2:0/3682404015 reap_dead start 2024-01-15T18:07:27.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.631+0000 7fe074c00640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0500047c0 0x7fe050004c40 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fe058006fd0 tx=0x7fe0500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a36d79ba436703 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:27.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.631+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058032600 con 0x7fe0500047c0 2024-01-15T18:07:27.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.631+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05802e050 con 0x7fe0500047c0 2024-01-15T18:07:27.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.631+0000 7fe06fe00640 1 -- 172.21.15.2:0/3682404015 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0580329c0 con 0x7fe0500047c0 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.632+0000 7fe078000640 1 -- 172.21.15.2:0/3682404015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0500047c0 msgr2=0x7fe050004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.632+0000 7fe078000640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0500047c0 0x7fe050004c40 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fe058006fd0 tx=0x7fe0500057f0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.632+0000 7fe078000640 1 -- 172.21.15.2:0/3682404015 shutdown_connections 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.632+0000 7fe078000640 1 --2- 172.21.15.2:0/3682404015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe0500047c0 0x7fe050004c40 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.632+0000 7fe078000640 1 -- 172.21.15.2:0/3682404015 >> 172.21.15.2:0/3682404015 conn(0x7fe07006da20 msgr2=0x7fe07006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.632+0000 7fe078000640 1 -- 172.21.15.2:0/3682404015 shutdown_connections 2024-01-15T18:07:27.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.633+0000 7fe078000640 1 -- 172.21.15.2:0/3682404015 wait complete. 2024-01-15T18:07:27.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.636+0000 7fe078000640 1 Processor -- start 2024-01-15T18:07:27.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.636+0000 7fe078000640 1 -- start start 2024-01-15T18:07:27.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe078000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0500047c0 0x7fe070143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe078000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe070132690 0x7fe070131680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe078000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 0x7fe070131bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe078000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe070150650 con 0x7fe07013be40 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe078000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07014fe30 con 0x7fe070132690 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe078000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe07013c2d0 con 0x7fe0500047c0 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe076000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 0x7fe070131bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe076000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 0x7fe070131bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57388/0 (socket says 172.21.15.2:57388) 2024-01-15T18:07:27.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.637+0000 7fe076000640 1 -- 172.21.15.2:0/4068838190 learned_addr learned my addr 172.21.15.2:0/4068838190 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe076000640 1 -- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0500047c0 msgr2=0x7fe070143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe076000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0500047c0 0x7fe070143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe076000640 1 -- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe070132690 msgr2=0x7fe070131680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe076000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe070132690 0x7fe070131680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe076000640 1 -- 172.21.15.2:0/4068838190 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0701508f0 con 0x7fe07013be40 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe076000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 0x7fe070131bc0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe0700cc890 tx=0x7fe06800fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:27.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068006b30 con 0x7fe07013be40 2024-01-15T18:07:27.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe068006cb0 con 0x7fe07013be40 2024-01-15T18:07:27.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068003bb0 con 0x7fe07013be40 2024-01-15T18:07:27.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.638+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe070150bd0 con 0x7fe07013be40 2024-01-15T18:07:27.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.639+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe070150190 con 0x7fe07013be40 2024-01-15T18:07:27.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.640+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe070151540 con 0x7fe07013be40 2024-01-15T18:07:27.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.641+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe068003d10 con 0x7fe07013be40 2024-01-15T18:07:27.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.642+0000 7fe06d600640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe04806be90 0x7fe04806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:27.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.642+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fe0680877a0 con 0x7fe07013be40 2024-01-15T18:07:27.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.647+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe068057890 con 0x7fe07013be40 2024-01-15T18:07:27.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.658+0000 7fe075600640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe04806be90 0x7fe04806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:27.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:27.661+0000 7fe075600640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe04806be90 0x7fe04806e350 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe070153450 tx=0x7fe060000f70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:28.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.107+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe07015eb40 con 0x7fe07013be40 2024-01-15T18:07:28.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.108+0000 7fe06d600640 1 -- 172.21.15.2:0/4068838190 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fe068019090 con 0x7fe07013be40 2024-01-15T18:07:28.112 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:28.112 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:28.113 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 615, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:28.114 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1493, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 6116657144, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 11956678656, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 761137434624, 2024-01-15T18:07:28.115 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 250859866, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 241093431, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 60, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 63 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:28.116 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:28.117 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.095228567719459534, 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:28.118 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-15T18:07:28.119 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.16664999723434448, 2024-01-15T18:07:28.119 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:28.119 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:28.119 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:28.119 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe04806be90 msgr2=0x7fe04806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe04806be90 0x7fe04806e350 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fe070153450 tx=0x7fe060000f70 comp rx=0 tx=0).stop 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 msgr2=0x7fe070131bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 0x7fe070131bc0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fe0700cc890 tx=0x7fe06800fc00 comp rx=0 tx=0).stop 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 shutdown_connections 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe04806be90 0x7fe04806e350 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe07013be40 0x7fe070131bc0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe070132690 0x7fe070131680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 --2- 172.21.15.2:0/4068838190 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe0500047c0 0x7fe070143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 >> 172.21.15.2:0/4068838190 conn(0x7fe07006da20 msgr2=0x7fe070152b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 shutdown_connections 2024-01-15T18:07:28.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:28.111+0000 7fe078000640 1 -- 172.21.15.2:0/4068838190 wait complete. 2024-01-15T18:07:29.146 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:29.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.458+0000 7f4fd3200640 1 Processor -- start 2024-01-15T18:07:29.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.459+0000 7f4fd3200640 1 -- start start 2024-01-15T18:07:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.461+0000 7f4fd3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc0b3c70 0x7f4fcc070f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.461+0000 7f4fd3200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4fcc06c470 0x7f4fcc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.461+0000 7f4fd3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4fcc161d30 0x7f4fcc1621c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.461+0000 7f4fd3200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fcc06ce30 con 0x7f4fcc161d30 2024-01-15T18:07:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.461+0000 7f4fd3200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fcc071cb0 con 0x7f4fcc06c470 2024-01-15T18:07:29.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.461+0000 7f4fd3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc070a70 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.462+0000 7f4fd1200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4fcc161d30 0x7f4fcc1621c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.462+0000 7f4fd1200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4fcc161d30 0x7f4fcc1621c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57390/0 (socket says 172.21.15.2:57390) 2024-01-15T18:07:29.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.462+0000 7f4fd1200640 1 -- 172.21.15.2:0/2694321922 learned_addr learned my addr 172.21.15.2:0/2694321922 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.463+0000 7f4fc3200640 1 --2- 172.21.15.2:0/2694321922 >> v2:172.21.15.137:3300/0 conn(0x7f4fcc06c470 0x7f4fcc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.463+0000 7f4fd1200640 1 -- 172.21.15.2:0/2694321922 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc0b3c70 msgr2=0x7f4fcc070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.463+0000 7f4fd1200640 1 --2- 172.21.15.2:0/2694321922 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc0b3c70 0x7f4fcc070f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.463+0000 7f4fd1200640 1 -- 172.21.15.2:0/2694321922 >> v2:172.21.15.137:3300/0 conn(0x7f4fcc06c470 msgr2=0x7f4fcc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.463+0000 7f4fd1200640 1 --2- 172.21.15.2:0/2694321922 >> v2:172.21.15.137:3300/0 conn(0x7f4fcc06c470 0x7f4fcc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.463+0000 7f4fd1200640 1 -- 172.21.15.2:0/2694321922 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fcc162700 con 0x7f4fcc161d30 2024-01-15T18:07:29.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.466+0000 7f4fd1200640 1 --2- 172.21.15.2:0/2694321922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc161d30 0x7f4fcc1621c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc06e590 tx=0x7f4fc402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fd4ba510f01075 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.467+0000 7f4fcb400640 1 -- 172.21.15.2:0/2694321922 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc402ca70 con 0x7f4fcc161d30 2024-01-15T18:07:29.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.467+0000 7f4fcb400640 1 -- 172.21.15.2:0/2694321922 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc402cbd0 con 0x7f4fcc161d30 2024-01-15T18:07:29.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.467+0000 7f4fcb400640 1 -- 172.21.15.2:0/2694321922 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40343f0 con 0x7f4fcc161d30 2024-01-15T18:07:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.469+0000 7f4fd3200640 1 -- 172.21.15.2:0/2694321922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc161d30 msgr2=0x7f4fcc1621c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.469+0000 7f4fd3200640 1 --2- 172.21.15.2:0/2694321922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc161d30 0x7f4fcc1621c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc06e590 tx=0x7f4fc402bed0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.472+0000 7f4fd3200640 1 -- 172.21.15.2:0/2694321922 shutdown_connections 2024-01-15T18:07:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.472+0000 7f4fd3200640 1 --2- 172.21.15.2:0/2694321922 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc161d30 0x7f4fcc1621c0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.473+0000 7f4fd3200640 1 --2- 172.21.15.2:0/2694321922 >> v2:172.21.15.137:3300/0 conn(0x7f4fcc06c470 0x7f4fcc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.473+0000 7f4fd3200640 1 --2- 172.21.15.2:0/2694321922 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc0b3c70 0x7f4fcc070f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.473+0000 7f4fd3200640 1 -- 172.21.15.2:0/2694321922 >> 172.21.15.2:0/2694321922 conn(0x7f4fcc06da20 msgr2=0x7f4fcc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:29.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.474+0000 7f4fd3200640 1 -- 172.21.15.2:0/2694321922 shutdown_connections 2024-01-15T18:07:29.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.475+0000 7f4fd3200640 1 -- 172.21.15.2:0/2694321922 wait complete. 2024-01-15T18:07:29.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.480+0000 7f4fd3200640 1 Processor -- start 2024-01-15T18:07:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.481+0000 7f4fd3200640 1 -- start start 2024-01-15T18:07:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.483+0000 7f4fd3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc06c470 0x7f4fcc0ab730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.483+0000 7f4fd3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 0x7f4fcc153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.483+0000 7f4fd3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc161d30 0x7f4fcc160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.483+0000 7f4fd3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0abc70 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.483+0000 7f4fd3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc15c7c0 con 0x7f4fcc161d30 2024-01-15T18:07:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.483+0000 7f4fd3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fcc0aa690 con 0x7f4fcc06c470 2024-01-15T18:07:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.485+0000 7f4fc3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 0x7f4fcc153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.486+0000 7f4fc3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 0x7f4fcc153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57398/0 (socket says 172.21.15.2:57398) 2024-01-15T18:07:29.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.486+0000 7f4fc3200640 1 -- 172.21.15.2:0/3609305684 learned_addr learned my addr 172.21.15.2:0/3609305684 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:29.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.487+0000 7f4fd1200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc161d30 0x7f4fcc160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.488+0000 7f4fcbe00640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc06c470 0x7f4fcc0ab730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.488+0000 7f4fc3200640 1 -- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc06c470 msgr2=0x7f4fcc0ab730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.488+0000 7f4fc3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc06c470 0x7f4fcc0ab730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.488+0000 7f4fc3200640 1 -- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc161d30 msgr2=0x7f4fcc160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.488+0000 7f4fc3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc161d30 0x7f4fcc160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.488+0000 7f4fc3200640 1 -- 172.21.15.2:0/3609305684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4009b30 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.489+0000 7f4fc3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 0x7f4fcc153360 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400d800 tx=0x7f4fb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.490+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400b6c0 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.490+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb400b820 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.490+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400bbb0 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.490+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fcc0aa970 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.490+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fcc0aaf30 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.493+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fcc0686f0 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.497+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4fb400bd10 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.500+0000 7f4fc8c00640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fa406bee0 0x7f4fa406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:29.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.500+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4fb4088d50 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.509+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4fb40553f0 con 0x7f4fcc0b3c70 2024-01-15T18:07:29.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.519+0000 7f4fcbe00640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fa406bee0 0x7f4fa406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:29.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:29.528+0000 7f4fcbe00640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fa406bee0 0x7f4fa406e3a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009890 tx=0x7f4fbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:30.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.003+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4fcc06ca10 con 0x7f4fcc0b3c70 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.003+0000 7f4fc8c00640 1 -- 172.21.15.2:0/3609305684 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f4fb4058a10 con 0x7f4fcc0b3c70 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:30.008 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 617, 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:30.009 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1851, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 7618305483, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 15098523648, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 757995589632, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 320039049, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 320405582, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 76, 2024-01-15T18:07:30.011 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 82 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:30.012 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:30.013 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.19045713543891907, 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.33329999446868896, 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:30.014 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:30.015 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:30.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.006+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fa406bee0 msgr2=0x7f4fa406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.006+0000 7f4fd3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fa406bee0 0x7f4fa406e3a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc009890 tx=0x7f4fbc008040 comp rx=0 tx=0).stop 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.006+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 msgr2=0x7f4fcc153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.006+0000 7f4fd3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 0x7f4fcc153360 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400d800 tx=0x7f4fb400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 shutdown_connections 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4fa406bee0 0x7f4fa406e3a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4fcc161d30 0x7f4fcc160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4fcc0b3c70 0x7f4fcc153360 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 --2- 172.21.15.2:0/3609305684 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4fcc06c470 0x7f4fcc0ab730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 >> 172.21.15.2:0/3609305684 conn(0x7f4fcc06da20 msgr2=0x7f4fcc0cde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 shutdown_connections 2024-01-15T18:07:30.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:30.007+0000 7f4fd3200640 1 -- 172.21.15.2:0/3609305684 wait complete. 2024-01-15T18:07:31.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:31.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.252+0000 7fcb47a00640 1 Processor -- start 2024-01-15T18:07:31.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.253+0000 7fcb47a00640 1 -- start start 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb47a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcb4015c0e0 0x7fcb400cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb47a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb4006cb80 0x7fcb4015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb47a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb4015cb00 con 0x7fcb4015b7d0 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb47a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb400ccec0 con 0x7fcb4006cb80 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb47a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb400cd1a0 con 0x7fcb4015c0e0 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb45a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb4006cb80 0x7fcb4015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb45a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcb4006cb80 0x7fcb4015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57410/0 (socket says 172.21.15.2:57410) 2024-01-15T18:07:31.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.255+0000 7fcb45a00640 1 -- 172.21.15.2:0/3784669925 learned_addr learned my addr 172.21.15.2:0/3784669925 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:31.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.256+0000 7fcb45a00640 1 -- 172.21.15.2:0/3784669925 >> v2:172.21.15.137:3300/0 conn(0x7fcb4015c0e0 msgr2=0x7fcb400cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:31.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.256+0000 7fcb45000640 1 --2- 172.21.15.2:0/3784669925 >> v2:172.21.15.137:3300/0 conn(0x7fcb4015c0e0 0x7fcb400cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:31.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.256+0000 7fcb45a00640 1 --2- 172.21.15.2:0/3784669925 >> v2:172.21.15.137:3300/0 conn(0x7fcb4015c0e0 0x7fcb400cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.256+0000 7fcb45a00640 1 -- 172.21.15.2:0/3784669925 >> v1:172.21.15.137:6789/0 conn(0x7fcb4015b7d0 legacy=0x7fcb4006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:31.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.256+0000 7fcb45a00640 1 -- 172.21.15.2:0/3784669925 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb400cd480 con 0x7fcb4006cb80 2024-01-15T18:07:31.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.256+0000 7fcb45000640 1 --2- 172.21.15.2:0/3784669925 >> v2:172.21.15.137:3300/0 conn(0x7fcb4015c0e0 0x7fcb400cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.257+0000 7fcb45a00640 1 --2- 172.21.15.2:0/3784669925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4006cb80 0x7fcb4015cf90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fcb400725f0 tx=0x7fcb3802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15965f60b0468cd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.257+0000 7fcb3f400640 1 -- 172.21.15.2:0/3784669925 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38031890 con 0x7fcb4006cb80 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb3f400640 1 -- 172.21.15.2:0/3784669925 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb380319f0 con 0x7fcb4006cb80 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb3f400640 1 -- 172.21.15.2:0/3784669925 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38031d80 con 0x7fcb4006cb80 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 -- 172.21.15.2:0/3784669925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4006cb80 msgr2=0x7fcb4015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 --2- 172.21.15.2:0/3784669925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4006cb80 0x7fcb4015cf90 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fcb400725f0 tx=0x7fcb3802b930 comp rx=0 tx=0).stop 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 -- 172.21.15.2:0/3784669925 shutdown_connections 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 --2- 172.21.15.2:0/3784669925 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4006cb80 0x7fcb4015cf90 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 --2- 172.21.15.2:0/3784669925 >> v2:172.21.15.137:3300/0 conn(0x7fcb4015c0e0 0x7fcb400cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 -- 172.21.15.2:0/3784669925 >> 172.21.15.2:0/3784669925 conn(0x7fcb4006da20 msgr2=0x7fcb4006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 -- 172.21.15.2:0/3784669925 shutdown_connections 2024-01-15T18:07:31.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.258+0000 7fcb47a00640 1 -- 172.21.15.2:0/3784669925 wait complete. 2024-01-15T18:07:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.260+0000 7fcb47a00640 1 Processor -- start 2024-01-15T18:07:31.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.260+0000 7fcb47a00640 1 -- start start 2024-01-15T18:07:31.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.261+0000 7fcb47a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb4006cb80 0x7fcb4014d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.261+0000 7fcb47a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb4015b7d0 0x7fcb40153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.261+0000 7fcb47a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 0x7fcb4015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.261+0000 7fcb47a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4014db50 con 0x7fcb4015c0e0 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.261+0000 7fcb47a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb400b4140 con 0x7fcb4006cb80 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.261+0000 7fcb47a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb400b3e60 con 0x7fcb4015b7d0 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 0x7fcb4015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 0x7fcb4015a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57424/0 (socket says 172.21.15.2:57424) 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 -- 172.21.15.2:0/78672339 learned_addr learned my addr 172.21.15.2:0/78672339 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45000640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb4006cb80 0x7fcb4014d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb3fe00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb4015b7d0 0x7fcb40153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 -- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb4015b7d0 msgr2=0x7fcb40153360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb4015b7d0 0x7fcb40153360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 -- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb4006cb80 msgr2=0x7fcb4014d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:31.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb4006cb80 0x7fcb4014d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.262+0000 7fcb45a00640 1 -- 172.21.15.2:0/78672339 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb380094d0 con 0x7fcb4015c0e0 2024-01-15T18:07:31.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.263+0000 7fcb45a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 0x7fcb4015a0e0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcb3802b970 tx=0x7fcb380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.263+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3803ae10 con 0x7fcb4015c0e0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.263+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb380443f0 con 0x7fcb4015c0e0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.263+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb380447a0 con 0x7fcb4015c0e0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.263+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb40152880 con 0x7fcb4015c0e0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.263+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb40152e40 con 0x7fcb4015c0e0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.264+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb40153060 con 0x7fcb4015c0e0 2024-01-15T18:07:31.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.265+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fcb38044900 con 0x7fcb4015c0e0 2024-01-15T18:07:31.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.266+0000 7fcb3cc00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb1806be90 0x7fcb1806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:31.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.266+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fcb380ab130 con 0x7fcb4015c0e0 2024-01-15T18:07:31.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.267+0000 7fcb45000640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb1806be90 0x7fcb1806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:31.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.268+0000 7fcb45000640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb1806be90 0x7fcb1806e350 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcb300075b0 tx=0x7fcb30007540 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:31.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.271+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb380778b0 con 0x7fcb4015c0e0 2024-01-15T18:07:31.445 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:31.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:07:31.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.536+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcb400686f0 con 0x7fcb4015c0e0 2024-01-15T18:07:31.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.537+0000 7fcb3cc00640 1 -- 172.21.15.2:0/78672339 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fcb38035020 con 0x7fcb4015c0e0 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:31.542 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 619, 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:31.543 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:31.544 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 1851, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 7618305483, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 15098523648, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 757995589632, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 320039049, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 320405582, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 76, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 82 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:31.545 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:31.546 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:31.547 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 97s)", 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.19045713543891907, 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.33329999446868896, 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:31.548 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:31.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb1806be90 msgr2=0x7fcb1806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:31.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb1806be90 0x7fcb1806e350 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcb300075b0 tx=0x7fcb30007540 comp rx=0 tx=0).stop 2024-01-15T18:07:31.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 msgr2=0x7fcb4015a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:31.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 0x7fcb4015a0e0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fcb3802b970 tx=0x7fcb380046c0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 shutdown_connections 2024-01-15T18:07:31.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcb1806be90 0x7fcb1806e350 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb4015c0e0 0x7fcb4015a0e0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb4015b7d0 0x7fcb40153360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 --2- 172.21.15.2:0/78672339 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb4006cb80 0x7fcb4014d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:31.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 >> 172.21.15.2:0/78672339 conn(0x7fcb4006da20 msgr2=0x7fcb4006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:31.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 shutdown_connections 2024-01-15T18:07:31.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:31.540+0000 7fcb47a00640 1 -- 172.21.15.2:0/78672339 wait complete. 2024-01-15T18:07:32.564 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:32.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.789+0000 7f772c800640 1 Processor -- start 2024-01-15T18:07:32.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.789+0000 7f772c800640 1 -- start start 2024-01-15T18:07:32.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.791+0000 7f772c800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f772413ce30 0x7f772415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.791+0000 7f772c800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77240aeab0 con 0x7f772415e170 2024-01-15T18:07:32.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.791+0000 7f772c800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7724070380 con 0x7f77240ce170 2024-01-15T18:07:32.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.791+0000 7f772c800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f772415e620 con 0x7f772413ce30 2024-01-15T18:07:32.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.791+0000 7f7729400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f772415e170 0x7f77240cda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48688/0 (socket says 172.21.15.2:48688) 2024-01-15T18:07:32.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.791+0000 7f7729400640 1 -- 172.21.15.2:0/3920244383 learned_addr learned my addr 172.21.15.2:0/3920244383 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:32.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7729e00640 1 --2- 172.21.15.2:0/3920244383 >> v2:172.21.15.137:3300/0 conn(0x7f772413ce30 0x7f772415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996715555 0 0) 0x7f77240aeab0 con 0x7f772415e170 2024-01-15T18:07:32.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704005730 con 0x7f772415e170 2024-01-15T18:07:32.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1591779571 0 0) 0x7f7704005730 con 0x7f772415e170 2024-01-15T18:07:32.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 >> v2:172.21.15.137:3300/0 conn(0x7f772413ce30 msgr2=0x7f772415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:32.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 --2- 172.21.15.2:0/3920244383 >> v2:172.21.15.137:3300/0 conn(0x7f772413ce30 0x7f772415dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 >> v1:172.21.15.137:6789/0 conn(0x7f77240ce170 legacy=0x7f7724070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77240ce6d0 con 0x7f772415e170 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f770c001670 con 0x7f772415e170 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 >> v1:172.21.15.2:6789/0 conn(0x7f772415e170 legacy=0x7f77240cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7704005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77040069d0 0x7f7704006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7704007390 0x7f770400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772415e620 con 0x7f7704005bd0 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770400e290 con 0x7f7704007390 2024-01-15T18:07:32.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.792+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770400e450 con 0x7f77040069d0 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f7729e00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7704005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f772a800640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7704007390 0x7f770400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f7729e00640 1 -- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77040069d0 msgr2=0x7f7704006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f7729e00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f77040069d0 0x7f7704006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f7729e00640 1 -- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7704007390 msgr2=0x7f770400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f7729e00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7704007390 0x7f770400db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f7729e00640 1 -- 172.21.15.2:0/3920244383 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704005730 con 0x7f7704005bd0 2024-01-15T18:07:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.793+0000 7f772a800640 1 -- 172.21.15.2:0/3920244383 reap_dead start 2024-01-15T18:07:32.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.794+0000 7f7729e00640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7704005ff0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7714009ac0 tx=0x7f7704007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5624b6f1b414d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:32.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771402c6b0 con 0x7f7704005bd0 2024-01-15T18:07:32.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771402c810 con 0x7f7704005bd0 2024-01-15T18:07:32.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f7728a00640 1 -- 172.21.15.2:0/3920244383 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771402cba0 con 0x7f7704005bd0 2024-01-15T18:07:32.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f772c800640 1 -- 172.21.15.2:0/3920244383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 msgr2=0x7f7704005ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f772c800640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7704005ff0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f7714009ac0 tx=0x7f7704007a00 comp rx=0 tx=0).stop 2024-01-15T18:07:32.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f772c800640 1 -- 172.21.15.2:0/3920244383 shutdown_connections 2024-01-15T18:07:32.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f772c800640 1 --2- 172.21.15.2:0/3920244383 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7704005ff0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.795+0000 7f772c800640 1 -- 172.21.15.2:0/3920244383 >> 172.21.15.2:0/3920244383 conn(0x7f772406da20 msgr2=0x7f772406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:32.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.796+0000 7f772c800640 1 -- 172.21.15.2:0/3920244383 shutdown_connections 2024-01-15T18:07:32.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.796+0000 7f772c800640 1 -- 172.21.15.2:0/3920244383 wait complete. 2024-01-15T18:07:32.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.797+0000 7f772c800640 1 Processor -- start 2024-01-15T18:07:32.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.798+0000 7f772c800640 1 -- start start 2024-01-15T18:07:32.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772c800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7724139280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772c800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772413ded0 0x7f7724138450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772c800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772413f7a0 0x7f772413fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772c800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772413c510 con 0x7f7704005bd0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772c800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772413e350 con 0x7f772413ded0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772c800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7724138990 con 0x7f772413f7a0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7724139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7724139280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57430/0 (socket says 172.21.15.2:57430) 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 -- 172.21.15.2:0/109616661 learned_addr learned my addr 172.21.15.2:0/109616661 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729400640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772413ded0 0x7f7724138450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 -- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772413f7a0 msgr2=0x7f772413fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f772a800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772413f7a0 0x7f772413fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772413f7a0 0x7f772413fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 -- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772413ded0 msgr2=0x7f7724138450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772413ded0 0x7f7724138450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729e00640 1 -- 172.21.15.2:0/109616661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7724153ea0 con 0x7f7704005bd0 2024-01-15T18:07:32.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.799+0000 7f7729400640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772413ded0 0x7f7724138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.800+0000 7f7729e00640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7724139280 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7714009ac0 tx=0x7f7714004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.800+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714003810 con 0x7f7704005bd0 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.800+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7714003970 con 0x7f7704005bd0 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.800+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77241e4530 con 0x7f7704005bd0 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.800+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714003d00 con 0x7f7704005bd0 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.800+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77241e4af0 con 0x7f7704005bd0 2024-01-15T18:07:32.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.801+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77241e4390 con 0x7f7704005bd0 2024-01-15T18:07:32.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.802+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7714040450 con 0x7f7704005bd0 2024-01-15T18:07:32.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.803+0000 7f7722000640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76fc06bec0 0x7f76fc06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:32.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.803+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f77140aa710 con 0x7f7704005bd0 2024-01-15T18:07:32.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.804+0000 7f7729400640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76fc06bec0 0x7f76fc06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:32.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.805+0000 7f7729400640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76fc06bec0 0x7f76fc06e380 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f770c0017a0 tx=0x7f770c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:32.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:32.809+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77241e4390 con 0x7f7704005bd0 2024-01-15T18:07:33.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.071+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f77241e4390 con 0x7f7704005bd0 2024-01-15T18:07:33.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.073+0000 7f7722000640 1 -- 172.21.15.2:0/109616661 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2409 (secure 0 0 0) 0x7f7714044980 con 0x7f7704005bd0 2024-01-15T18:07:33.077 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:33.077 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:33.077 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:33.077 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:33.078 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 620, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.079 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 2269, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 9371524555, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 18449772544, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 754644340736, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:33.080 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 378730619, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 367914436, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 90, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 92 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:33.081 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:33.082 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 28.571% (26s)\n [========....................] (remaining: 66s)", 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.28571429848670959, 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (26s)\n [==============..............] (remaining: 26s)", 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.5, 2024-01-15T18:07:33.083 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:33.084 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:33.084 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:33.084 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:33.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.075+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76fc06bec0 msgr2=0x7f76fc06e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.075+0000 7f772c800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76fc06bec0 0x7f76fc06e380 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f770c0017a0 tx=0x7f770c009040 comp rx=0 tx=0).stop 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.075+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 msgr2=0x7f7724139280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.075+0000 7f772c800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7724139280 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f7714009ac0 tx=0x7f7714004730 comp rx=0 tx=0).stop 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 shutdown_connections 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76fc06bec0 0x7f76fc06e380 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f772413f7a0 0x7f772413fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f772413ded0 0x7f7724138450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 --2- 172.21.15.2:0/109616661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7704005bd0 0x7f7724139280 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 >> 172.21.15.2:0/109616661 conn(0x7f772406da20 msgr2=0x7f7724150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 shutdown_connections 2024-01-15T18:07:33.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:33.076+0000 7f772c800640 1 -- 172.21.15.2:0/109616661 wait complete. 2024-01-15T18:07:34.102 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47168 2024-01-15T18:07:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.344+0000 7eff32200640 1 Processor -- start 2024-01-15T18:07:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.344+0000 7eff32200640 1 -- start start 2024-01-15T18:07:34.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff32200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7eff2c15c450 0x7eff2c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:34.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff32200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c06ce30 0x7eff2c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:34.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff32200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c0ce230 con 0x7eff2c144c00 2024-01-15T18:07:34.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff32200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2c06d2c0 con 0x7eff2c15c450 2024-01-15T18:07:34.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff32200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c0ce510 con 0x7eff2c06ce30 2024-01-15T18:07:34.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7eff2c15c450 0x7eff2c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:34.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7eff2c15c450 0x7eff2c06c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59920/0 (socket says 172.21.15.2:59920) 2024-01-15T18:07:34.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 -- 172.21.15.2:0/608366874 learned_addr learned my addr 172.21.15.2:0/608366874 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:34.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff2be00640 1 --2- 172.21.15.2:0/608366874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c06ce30 0x7eff2c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:34.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 -- 172.21.15.2:0/608366874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c06ce30 msgr2=0x7eff2c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:34.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 --2- 172.21.15.2:0/608366874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c06ce30 0x7eff2c070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 -- 172.21.15.2:0/608366874 >> v1:172.21.15.137:6789/0 conn(0x7eff2c144c00 legacy=0x7eff2c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.347+0000 7eff23e00640 1 -- 172.21.15.2:0/608366874 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c0ce7f0 con 0x7eff2c15c450 2024-01-15T18:07:34.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.348+0000 7eff23e00640 1 --2- 172.21.15.2:0/608366874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c06c8f0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7eff14009870 tx=0x7eff1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=637d1002415954df server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:34.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff2aa00640 1 -- 172.21.15.2:0/608366874 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14030840 con 0x7eff2c15c450 2024-01-15T18:07:34.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff2aa00640 1 -- 172.21.15.2:0/608366874 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff140309a0 con 0x7eff2c15c450 2024-01-15T18:07:34.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff2aa00640 1 -- 172.21.15.2:0/608366874 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14030d30 con 0x7eff2c15c450 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 -- 172.21.15.2:0/608366874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 msgr2=0x7eff2c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 --2- 172.21.15.2:0/608366874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c06c8f0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7eff14009870 tx=0x7eff1402b980 comp rx=0 tx=0).stop 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 -- 172.21.15.2:0/608366874 shutdown_connections 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 --2- 172.21.15.2:0/608366874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c06ce30 0x7eff2c070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 --2- 172.21.15.2:0/608366874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c06c8f0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 -- 172.21.15.2:0/608366874 >> 172.21.15.2:0/608366874 conn(0x7eff2c0b45f0 msgr2=0x7eff2c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 -- 172.21.15.2:0/608366874 shutdown_connections 2024-01-15T18:07:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.349+0000 7eff32200640 1 -- 172.21.15.2:0/608366874 wait complete. 2024-01-15T18:07:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.351+0000 7eff32200640 1 Processor -- start 2024-01-15T18:07:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.351+0000 7eff32200640 1 -- start start 2024-01-15T18:07:34.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.352+0000 7eff32200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff2c06ce30 0x7eff2c0b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.352+0000 7eff32200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c144c00 0x7eff2c0b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.352+0000 7eff32200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c0b1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.352+0000 7eff32200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c142420 con 0x7eff2c15c450 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.352+0000 7eff32200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c15dd90 con 0x7eff2c06ce30 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.352+0000 7eff32200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c0b1640 con 0x7eff2c144c00 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c0b1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c0b1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59936/0 (socket says 172.21.15.2:59936) 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 -- 172.21.15.2:0/40127873 learned_addr learned my addr 172.21.15.2:0/40127873 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff23e00640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c144c00 0x7eff2c0b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2b400640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff2c06ce30 0x7eff2c0b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 -- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c144c00 msgr2=0x7eff2c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c144c00 0x7eff2c0b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 -- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff2c06ce30 msgr2=0x7eff2c0b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff2c06ce30 0x7eff2c0b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2be00640 1 -- 172.21.15.2:0/40127873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140094d0 con 0x7eff2c15c450 2024-01-15T18:07:34.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.353+0000 7eff2b400640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff2c06ce30 0x7eff2c0b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:34.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.354+0000 7eff2be00640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c0b1100 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7eff24007e40 tx=0x7eff2400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:34.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.354+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24014620 con 0x7eff2c15c450 2024-01-15T18:07:34.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.354+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2400ee90 con 0x7eff2c15c450 2024-01-15T18:07:34.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.354+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240149e0 con 0x7eff2c15c450 2024-01-15T18:07:34.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.354+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2c141810 con 0x7eff2c15c450 2024-01-15T18:07:34.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.354+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2c141dd0 con 0x7eff2c15c450 2024-01-15T18:07:34.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.355+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2c142590 con 0x7eff2c15c450 2024-01-15T18:07:34.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.356+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7eff24014b40 con 0x7eff2c15c450 2024-01-15T18:07:34.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.357+0000 7eff23400640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff0406be90 0x7eff0406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:34.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.357+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7eff24088fd0 con 0x7eff2c15c450 2024-01-15T18:07:34.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.358+0000 7eff2b400640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff0406be90 0x7eff0406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:34.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.359+0000 7eff2b400640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff0406be90 0x7eff0406e350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff1c001700 tx=0x7eff1c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:34.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.362+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff24055890 con 0x7eff2c15c450 2024-01-15T18:07:34.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.601+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47168", "target": ["mon-mgr", ""]}) v1 -- 0x7eff2c0cead0 con 0x7eff0406be90 2024-01-15T18:07:34.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.770+0000 7eff23400640 1 -- 172.21.15.2:0/40127873 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7eff1c001ce0 con 0x7eff0406be90 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff0406be90 msgr2=0x7eff0406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff0406be90 0x7eff0406e350 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7eff1c001700 tx=0x7eff1c009040 comp rx=0 tx=0).stop 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 msgr2=0x7eff2c0b1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c0b1100 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7eff24007e40 tx=0x7eff2400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 shutdown_connections 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff0406be90 0x7eff0406e350 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff2c15c450 0x7eff2c0b1100 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff2c144c00 0x7eff2c0b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 --2- 172.21.15.2:0/40127873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff2c06ce30 0x7eff2c0b0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 >> 172.21.15.2:0/40127873 conn(0x7eff2c0b45f0 msgr2=0x7eff2c143f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 shutdown_connections 2024-01-15T18:07:34.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:34.773+0000 7eff32200640 1 -- 172.21.15.2:0/40127873 wait complete. 2024-01-15T18:07:34.798 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone47169 2024-01-15T18:07:35.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.024+0000 7f0244600640 1 Processor -- start 2024-01-15T18:07:35.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.024+0000 7f0244600640 1 -- start start 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0244600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f023c0b4c70 0x7f023c0d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0244600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023c0d1900 con 0x7f023c15d730 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0244600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f023c0d1be0 con 0x7f023c0b8c30 2024-01-15T18:07:35.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0244600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f023c0d30b0 con 0x7f023c0b4c70 2024-01-15T18:07:35.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0241200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f023c15d730 0x7f023c0b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41592/0 (socket says 172.21.15.2:41592) 2024-01-15T18:07:35.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0241200640 1 -- 172.21.15.2:0/3207842380 learned_addr learned my addr 172.21.15.2:0/3207842380 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.026+0000 7f0241c00640 1 --2- 172.21.15.2:0/3207842380 >> v2:172.21.15.137:3300/0 conn(0x7f023c0b4c70 0x7f023c0d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2599902627 0 0) 0x7f023c0d1900 con 0x7f023c15d730 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f021c003520 con 0x7f023c15d730 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 550794055 0 0) 0x7f021c003520 con 0x7f023c15d730 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 >> v2:172.21.15.137:3300/0 conn(0x7f023c0b4c70 msgr2=0x7f023c0d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 --2- 172.21.15.2:0/3207842380 >> v2:172.21.15.137:3300/0 conn(0x7f023c0b4c70 0x7f023c0d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 >> v1:172.21.15.137:6789/0 conn(0x7f023c0b8c30 legacy=0x7f023c0d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c0d1ec0 con 0x7f023c15d730 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f0224001670 con 0x7f023c15d730 2024-01-15T18:07:35.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 >> v1:172.21.15.2:6789/0 conn(0x7f023c15d730 legacy=0x7f023c0b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021c003970 0x7f021c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021c0047c0 0x7f021c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c005180 0x7f021c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c0d30b0 con 0x7f021c003970 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021c00c020 con 0x7f021c005180 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021c00c1e0 con 0x7f021c0047c0 2024-01-15T18:07:35.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.027+0000 7f0241c00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021c003970 0x7f021c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0242600640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c005180 0x7f021c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0241c00640 1 -- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021c0047c0 msgr2=0x7f021c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0241c00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021c0047c0 0x7f021c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0241c00640 1 -- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c005180 msgr2=0x7f021c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0241c00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c005180 0x7f021c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0241c00640 1 -- 172.21.15.2:0/3207842380 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c003520 con 0x7f021c003970 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0242600640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c005180 0x7f021c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.028+0000 7f0242600640 1 -- 172.21.15.2:0/3207842380 reap_dead start 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.029+0000 7f0241c00640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021c003970 0x7f021c003d90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f022c009ac0 tx=0x7f021c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f635a096a07053 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.029+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c02c6b0 con 0x7f021c003970 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.029+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022c02c810 con 0x7f021c003970 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.029+0000 7f023be00640 1 -- 172.21.15.2:0/3207842380 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022c02cba0 con 0x7f021c003970 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 -- 172.21.15.2:0/3207842380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021c003970 msgr2=0x7f021c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021c003970 0x7f021c003d90 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f022c009ac0 tx=0x7f021c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 -- 172.21.15.2:0/3207842380 shutdown_connections 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 --2- 172.21.15.2:0/3207842380 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021c003970 0x7f021c003d90 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 -- 172.21.15.2:0/3207842380 >> 172.21.15.2:0/3207842380 conn(0x7f023c072210 msgr2=0x7f023c0cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 -- 172.21.15.2:0/3207842380 shutdown_connections 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.030+0000 7f0244600640 1 -- 172.21.15.2:0/3207842380 wait complete. 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.032+0000 7f0244600640 1 Processor -- start 2024-01-15T18:07:35.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.032+0000 7f0244600640 1 -- start start 2024-01-15T18:07:35.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0244600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c003970 0x7f023c156240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0244600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f023c152730 0x7f023c150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0244600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 0x7f023c0cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0244600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c151060 con 0x7f023c0cc490 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0244600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023c0cce50 con 0x7f021c003970 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0244600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023c1568d0 con 0x7f023c152730 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0242600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 0x7f023c0cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0242600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 0x7f023c0cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59962/0 (socket says 172.21.15.2:59962) 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.033+0000 7f0242600640 1 -- 172.21.15.2:0/1769454039 learned_addr learned my addr 172.21.15.2:0/1769454039 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0241200640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f023c152730 0x7f023c150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0241c00640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c003970 0x7f023c156240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0242600640 1 -- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f023c152730 msgr2=0x7f023c150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0242600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f023c152730 0x7f023c150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0242600640 1 -- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c003970 msgr2=0x7f023c156240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0242600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c003970 0x7f023c156240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0242600640 1 -- 172.21.15.2:0/1769454039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c0c9330 con 0x7f023c0cc490 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0241c00640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c003970 0x7f023c156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:35.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0242600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 0x7f023c0cc910 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f023c0cda10 tx=0x7f023400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234003d80 con 0x7f023c0cc490 2024-01-15T18:07:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0234005070 con 0x7f023c0cc490 2024-01-15T18:07:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.034+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0234005420 con 0x7f023c0cc490 2024-01-15T18:07:35.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.035+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023c152490 con 0x7f023c0cc490 2024-01-15T18:07:35.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.035+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023c0c98b0 con 0x7f023c0cc490 2024-01-15T18:07:35.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.036+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f023c15c580 con 0x7f023c0cc490 2024-01-15T18:07:35.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.037+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0234005580 con 0x7f023c0cc490 2024-01-15T18:07:35.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.038+0000 7f0239600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f021406be90 0x7f021406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.038+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0234087d90 con 0x7f023c0cc490 2024-01-15T18:07:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.038+0000 7f0241c00640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f021406be90 0x7f021406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.040+0000 7f0241c00640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f021406be90 0x7f021406e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f022c036000 tx=0x7f022c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.042+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02340546b0 con 0x7f023c0cc490 2024-01-15T18:07:35.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.272+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone47169", "target": ["mon-mgr", ""]}) v1 -- 0x7f023c0b90c0 con 0x7f021406be90 2024-01-15T18:07:35.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.471+0000 7f0239600640 1 -- 172.21.15.2:0/1769454039 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f022c0041c0 con 0x7f021406be90 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f021406be90 msgr2=0x7f021406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f021406be90 0x7f021406e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f022c036000 tx=0x7f022c030000 comp rx=0 tx=0).stop 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 msgr2=0x7f023c0cc910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 0x7f023c0cc910 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f023c0cda10 tx=0x7f023400d9d0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 shutdown_connections 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f021406be90 0x7f021406e350 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f023c0cc490 0x7f023c0cc910 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f023c152730 0x7f023c150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 --2- 172.21.15.2:0/1769454039 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f021c003970 0x7f023c156240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 >> 172.21.15.2:0/1769454039 conn(0x7f023c072210 msgr2=0x7f023c0b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 shutdown_connections 2024-01-15T18:07:35.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.475+0000 7f0244600640 1 -- 172.21.15.2:0/1769454039 wait complete. 2024-01-15T18:07:35.499 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471610 2024-01-15T18:07:35.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.720+0000 7f971c800640 1 Processor -- start 2024-01-15T18:07:35.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.720+0000 7f971c800640 1 -- start start 2024-01-15T18:07:35.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f971c800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f97140702c0 0x7f97140b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f971c800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f971406c8f0 0x7f971406ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f971c800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f971415dd70 con 0x7f97141439f0 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f971c800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971406d230 con 0x7f97140702c0 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f971c800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97140711b0 con 0x7f971406c8f0 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f9719400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f97140702c0 0x7f97140b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f9719400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f97140702c0 0x7f97140b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59966/0 (socket says 172.21.15.2:59966) 2024-01-15T18:07:35.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.722+0000 7f9719400640 1 -- 172.21.15.2:0/198565431 learned_addr learned my addr 172.21.15.2:0/198565431 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:35.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.723+0000 7f971a800640 1 --2- 172.21.15.2:0/198565431 >> v2:172.21.15.137:3300/0 conn(0x7f971406c8f0 0x7f971406ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.723+0000 7f9719400640 1 -- 172.21.15.2:0/198565431 >> v2:172.21.15.137:3300/0 conn(0x7f971406c8f0 msgr2=0x7f971406ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.723+0000 7f9719400640 1 --2- 172.21.15.2:0/198565431 >> v2:172.21.15.137:3300/0 conn(0x7f971406c8f0 0x7f971406ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.723+0000 7f9719400640 1 -- 172.21.15.2:0/198565431 >> v1:172.21.15.137:6789/0 conn(0x7f97141439f0 legacy=0x7f97140cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:35.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.723+0000 7f9719400640 1 -- 172.21.15.2:0/198565431 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9714143dd0 con 0x7f97140702c0 2024-01-15T18:07:35.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.724+0000 7f9719400640 1 --2- 172.21.15.2:0/198565431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97140b3420 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f97140b4310 tx=0x7f96fc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=514b54840c075424 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.725+0000 7f9718a00640 1 -- 172.21.15.2:0/198565431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc02ca80 con 0x7f97140702c0 2024-01-15T18:07:35.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.725+0000 7f9718a00640 1 -- 172.21.15.2:0/198565431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc02cbe0 con 0x7f97140702c0 2024-01-15T18:07:35.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.725+0000 7f9718a00640 1 -- 172.21.15.2:0/198565431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc0343f0 con 0x7f97140702c0 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.725+0000 7f971c800640 1 -- 172.21.15.2:0/198565431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 msgr2=0x7f97140b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.725+0000 7f971c800640 1 --2- 172.21.15.2:0/198565431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97140b3420 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f97140b4310 tx=0x7f96fc02bee0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.725+0000 7f971c800640 1 -- 172.21.15.2:0/198565431 shutdown_connections 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.726+0000 7f971c800640 1 --2- 172.21.15.2:0/198565431 >> v2:172.21.15.137:3300/0 conn(0x7f971406c8f0 0x7f971406ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.726+0000 7f971c800640 1 --2- 172.21.15.2:0/198565431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97140b3420 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.726+0000 7f971c800640 1 -- 172.21.15.2:0/198565431 >> 172.21.15.2:0/198565431 conn(0x7f97140b45f0 msgr2=0x7f97140b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:35.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.726+0000 7f971c800640 1 -- 172.21.15.2:0/198565431 shutdown_connections 2024-01-15T18:07:35.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.726+0000 7f971c800640 1 -- 172.21.15.2:0/198565431 wait complete. 2024-01-15T18:07:35.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.727+0000 7f971c800640 1 Processor -- start 2024-01-15T18:07:35.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.728+0000 7f971c800640 1 -- start start 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.728+0000 7f971c800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f971406c8f0 0x7f971415a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f971c800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97141399d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f971c800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97141439f0 0x7f97140b03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f971c800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971415ac30 con 0x7f97140702c0 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f971c800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971415af10 con 0x7f97141439f0 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f971c800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971415b1f0 con 0x7f971406c8f0 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f9719400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97141399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f9719400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97141399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59970/0 (socket says 172.21.15.2:59970) 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f9719400640 1 -- 172.21.15.2:0/978373203 learned_addr learned my addr 172.21.15.2:0/978373203 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f9719e00640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f971406c8f0 0x7f971415a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.729+0000 7f971a800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97141439f0 0x7f97140b03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9719400640 1 -- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f971406c8f0 msgr2=0x7f971415a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9719400640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f971406c8f0 0x7f971415a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9719400640 1 -- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97141439f0 msgr2=0x7f97140b03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9719400640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97141439f0 0x7f97140b03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9719400640 1 -- 172.21.15.2:0/978373203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc009b30 con 0x7f97140702c0 2024-01-15T18:07:35.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9719400640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97141399d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f96fc000f80 tx=0x7f96fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc009510 con 0x7f97140702c0 2024-01-15T18:07:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc03e450 con 0x7f97140702c0 2024-01-15T18:07:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc03e800 con 0x7f97140702c0 2024-01-15T18:07:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.730+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971415b4d0 con 0x7f97140702c0 2024-01-15T18:07:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.731+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9714139f10 con 0x7f97140702c0 2024-01-15T18:07:35.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.732+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97140686f0 con 0x7f97140702c0 2024-01-15T18:07:35.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.733+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f96fc02cd50 con 0x7f97140702c0 2024-01-15T18:07:35.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.734+0000 7f9712000640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96ec06be90 0x7f96ec06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:35.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.734+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f96fc0ab130 con 0x7f97140702c0 2024-01-15T18:07:35.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.735+0000 7f9719e00640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96ec06be90 0x7f96ec06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:35.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.736+0000 7f9719e00640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96ec06be90 0x7f96ec06e350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9704008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:35.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.739+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96fc0779f0 con 0x7f97140702c0 2024-01-15T18:07:35.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:35.972+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471610", "target": ["mon-mgr", ""]}) v1 -- 0x7f9714070ed0 con 0x7f96ec06be90 2024-01-15T18:07:36.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.149+0000 7f9712000640 1 -- 172.21.15.2:0/978373203 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f970400bd80 con 0x7f96ec06be90 2024-01-15T18:07:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.152+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96ec06be90 msgr2=0x7f96ec06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.152+0000 7f971c800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96ec06be90 0x7f96ec06e350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f9704006fd0 tx=0x7f9704008040 comp rx=0 tx=0).stop 2024-01-15T18:07:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.152+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 msgr2=0x7f97141399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.152+0000 7f971c800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97141399d0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f96fc000f80 tx=0x7f96fc004a40 comp rx=0 tx=0).stop 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.152+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 shutdown_connections 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.152+0000 7f971c800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96ec06be90 0x7f96ec06e350 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.153+0000 7f971c800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f97141439f0 0x7f97140b03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.153+0000 7f971c800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f97140702c0 0x7f97141399d0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.153+0000 7f971c800640 1 --2- 172.21.15.2:0/978373203 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f971406c8f0 0x7f971415a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.153+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 >> 172.21.15.2:0/978373203 conn(0x7f97140b45f0 msgr2=0x7f97140aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.153+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 shutdown_connections 2024-01-15T18:07:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.153+0000 7f971c800640 1 -- 172.21.15.2:0/978373203 wait complete. 2024-01-15T18:07:36.180 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471611 2024-01-15T18:07:36.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.399+0000 7fe9e6600640 1 Processor -- start 2024-01-15T18:07:36.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.399+0000 7fe9e6600640 1 -- start start 2024-01-15T18:07:36.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e6600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe9e00716b0 0x7fe9e00ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e6600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00b2fa0 con 0x7fe9e0143620 2024-01-15T18:07:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e6600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e00b3280 con 0x7fe9e0071230 2024-01-15T18:07:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e6600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e00b3560 con 0x7fe9e00716b0 2024-01-15T18:07:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e5200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe9e0143620 0x7fe9e00b2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41596/0 (socket says 172.21.15.2:41596) 2024-01-15T18:07:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e5200640 1 -- 172.21.15.2:0/344901660 learned_addr learned my addr 172.21.15.2:0/344901660 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:36.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.401+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/344901660 >> v2:172.21.15.137:3300/0 conn(0x7fe9e00716b0 0x7fe9e00ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182786805 0 0) 0x7fe9e00b2fa0 con 0x7fe9e0143620 2024-01-15T18:07:36.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0003520 con 0x7fe9e0143620 2024-01-15T18:07:36.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 796975518 0 0) 0x7fe9c0003520 con 0x7fe9e0143620 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 >> v2:172.21.15.137:3300/0 conn(0x7fe9e00716b0 msgr2=0x7fe9e00ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 --2- 172.21.15.2:0/344901660 >> v2:172.21.15.137:3300/0 conn(0x7fe9e00716b0 0x7fe9e00ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 >> v1:172.21.15.137:6789/0 conn(0x7fe9e0071230 legacy=0x7fe9e00cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e00b3840 con 0x7fe9e0143620 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fe9d00019f0 con 0x7fe9e0143620 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 >> v1:172.21.15.2:6789/0 conn(0x7fe9e0143620 legacy=0x7fe9e00b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9c0003970 0x7fe9c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c00047c0 0x7fe9c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9c0005180 0x7fe9c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e00b3560 con 0x7fe9c0005180 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c000c020 con 0x7fe9c00047c0 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.402+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c000c1e0 con 0x7fe9c0003970 2024-01-15T18:07:36.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9c0005180 0x7fe9c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9dfe00640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c00047c0 0x7fe9c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5200640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9c0003970 0x7fe9c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5c00640 1 -- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9c0003970 msgr2=0x7fe9c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9c0003970 0x7fe9c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5c00640 1 -- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c00047c0 msgr2=0x7fe9c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c00047c0 0x7fe9c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.403+0000 7fe9e5c00640 1 -- 172.21.15.2:0/344901660 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0003520 con 0x7fe9c0005180 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.404+0000 7fe9e5c00640 1 -- 172.21.15.2:0/344901660 reap_dead start 2024-01-15T18:07:36.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.404+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9c0005180 0x7fe9c000b980 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009650 tx=0x7fe9c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d92d64aa01a58666 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:36.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.404+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8004020 con 0x7fe9c0005180 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.404+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d8004180 con 0x7fe9c0005180 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9df400640 1 -- 172.21.15.2:0/344901660 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d802e920 con 0x7fe9c0005180 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 -- 172.21.15.2:0/344901660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9c0005180 msgr2=0x7fe9c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9c0005180 0x7fe9c000b980 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009650 tx=0x7fe9c00057f0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 -- 172.21.15.2:0/344901660 shutdown_connections 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 --2- 172.21.15.2:0/344901660 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9c0005180 0x7fe9c000b980 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 -- 172.21.15.2:0/344901660 >> 172.21.15.2:0/344901660 conn(0x7fe9e00b0840 msgr2=0x7fe9e00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 -- 172.21.15.2:0/344901660 shutdown_connections 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.405+0000 7fe9e6600640 1 -- 172.21.15.2:0/344901660 wait complete. 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.407+0000 7fe9e6600640 1 Processor -- start 2024-01-15T18:07:36.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.407+0000 7fe9e6600640 1 -- start start 2024-01-15T18:07:36.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9e6600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c0005180 0x7fe9e015cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9e6600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 0x7fe9e00af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9e6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e0158fc0 0x7fe9e00aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9e6600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0157f60 con 0x7fe9e015d110 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9e6600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e0158200 con 0x7fe9c0005180 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9e6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e01584e0 con 0x7fe9e0158fc0 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.408+0000 7fe9dfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 0x7fe9e00af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 0x7fe9e00af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59974/0 (socket says 172.21.15.2:59974) 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 -- 172.21.15.2:0/229850793 learned_addr learned my addr 172.21.15.2:0/229850793 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e0158fc0 0x7fe9e00aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9e5200640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c0005180 0x7fe9e015cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 -- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e0158fc0 msgr2=0x7fe9e00aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e0158fc0 0x7fe9e00aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 -- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c0005180 msgr2=0x7fe9e015cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c0005180 0x7fe9e015cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 -- 172.21.15.2:0/229850793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e01587c0 con 0x7fe9e015d110 2024-01-15T18:07:36.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9e5c00640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e0158fc0 0x7fe9e00aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9e5200640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c0005180 0x7fe9e015cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.409+0000 7fe9dfe00640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 0x7fe9e00af9c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800e480 tx=0x7fe9c800e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.410+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c80052e0 con 0x7fe9e015d110 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.410+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c8005440 con 0x7fe9e015d110 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.410+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e0158aa0 con 0x7fe9e015d110 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.410+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8003830 con 0x7fe9e015d110 2024-01-15T18:07:36.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.410+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e01f6590 con 0x7fe9e015d110 2024-01-15T18:07:36.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.411+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e0073b20 con 0x7fe9e015d110 2024-01-15T18:07:36.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.412+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe9c8003990 con 0x7fe9e015d110 2024-01-15T18:07:36.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.413+0000 7fe9dcc00640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9b806be90 0x7fe9b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:36.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.413+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fe9c80885d0 con 0x7fe9e015d110 2024-01-15T18:07:36.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.414+0000 7fe9e5200640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9b806be90 0x7fe9b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:36.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.415+0000 7fe9e5200640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9b806be90 0x7fe9b806e350 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0001580 tx=0x7fe9d00014f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:36.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.417+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9c8054c30 con 0x7fe9e015d110 2024-01-15T18:07:36.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:36.745+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471611", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9e0143a00 con 0x7fe9b806be90 2024-01-15T18:07:37.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.286+0000 7fe9dcc00640 1 -- 172.21.15.2:0/229850793 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe9d0002040 con 0x7fe9b806be90 2024-01-15T18:07:37.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9b806be90 msgr2=0x7fe9b806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:37.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9b806be90 0x7fe9b806e350 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe9d0001580 tx=0x7fe9d00014f0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 msgr2=0x7fe9e00af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:37.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 0x7fe9e00af9c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800e480 tx=0x7fe9c800e940 comp rx=0 tx=0).stop 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 shutdown_connections 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe9b806be90 0x7fe9b806e350 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9e0158fc0 0x7fe9e00aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe9e015d110 0x7fe9e00af9c0 secure :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe9c800e480 tx=0x7fe9c800e940 comp rx=0 tx=0).stop 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 --2- 172.21.15.2:0/229850793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe9c0005180 0x7fe9e015cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.292+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 >> 172.21.15.2:0/229850793 conn(0x7fe9e00b0840 msgr2=0x7fe9e01597c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.293+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 shutdown_connections 2024-01-15T18:07:37.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.293+0000 7fe9e6600640 1 -- 172.21.15.2:0/229850793 wait complete. 2024-01-15T18:07:37.326 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471612 2024-01-15T18:07:37.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.617+0000 7f00afa00640 1 Processor -- start 2024-01-15T18:07:37.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.618+0000 7f00afa00640 1 -- start start 2024-01-15T18:07:37.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.620+0000 7f00afa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f00a814afd0 0x7f00a80b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:37.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.621+0000 7f00afa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 0x7f00a806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:37.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.621+0000 7f00afa00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a806d270 con 0x7f00a806ce30 2024-01-15T18:07:37.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.621+0000 7f00afa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a8070ff0 con 0x7f00a814afd0 2024-01-15T18:07:37.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.621+0000 7f00afa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a80b25f0 con 0x7f00a814b3b0 2024-01-15T18:07:37.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.622+0000 7f00ada00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f00a806ce30 0x7f00a80702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41612/0 (socket says 172.21.15.2:41612) 2024-01-15T18:07:37.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.622+0000 7f00ada00640 1 -- 172.21.15.2:0/3054978910 learned_addr learned my addr 172.21.15.2:0/3054978910 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:37.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.622+0000 7f00ad000640 1 --2- 172.21.15.2:0/3054978910 >> v2:172.21.15.2:3300/0 conn(0x7f00a814afd0 0x7f00a80b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:37.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.623+0000 7f00ad000640 1 -- 172.21.15.2:0/3054978910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 msgr2=0x7f00a806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:37.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.623+0000 7f00ad000640 1 --2- 172.21.15.2:0/3054978910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 0x7f00a806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.623+0000 7f00ad000640 1 -- 172.21.15.2:0/3054978910 >> v1:172.21.15.2:6789/0 conn(0x7f00a806ce30 legacy=0x7f00a80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:37.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.623+0000 7f00ad000640 1 -- 172.21.15.2:0/3054978910 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a80b28d0 con 0x7f00a814afd0 2024-01-15T18:07:37.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.625+0000 7f00ad000640 1 --2- 172.21.15.2:0/3054978910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a80b33b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f00980097a0 tx=0x7f009802b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a4a8d8d001eaa8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:37.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.625+0000 7f00a7400640 1 -- 172.21.15.2:0/3054978910 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098031780 con 0x7f00a814afd0 2024-01-15T18:07:37.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.625+0000 7f00a7400640 1 -- 172.21.15.2:0/3054978910 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00980318e0 con 0x7f00a814afd0 2024-01-15T18:07:37.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.625+0000 7f00a7400640 1 -- 172.21.15.2:0/3054978910 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0098031c70 con 0x7f00a814afd0 2024-01-15T18:07:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.626+0000 7f00afa00640 1 -- 172.21.15.2:0/3054978910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 msgr2=0x7f00a80b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:37.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.626+0000 7f00afa00640 1 --2- 172.21.15.2:0/3054978910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a80b33b0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f00980097a0 tx=0x7f009802b820 comp rx=0 tx=0).stop 2024-01-15T18:07:37.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.627+0000 7f00afa00640 1 -- 172.21.15.2:0/3054978910 shutdown_connections 2024-01-15T18:07:37.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.627+0000 7f00afa00640 1 --2- 172.21.15.2:0/3054978910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 0x7f00a806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.627+0000 7f00afa00640 1 --2- 172.21.15.2:0/3054978910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a80b33b0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.627+0000 7f00afa00640 1 -- 172.21.15.2:0/3054978910 >> 172.21.15.2:0/3054978910 conn(0x7f00a80b45f0 msgr2=0x7f00a80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:37.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.627+0000 7f00afa00640 1 -- 172.21.15.2:0/3054978910 shutdown_connections 2024-01-15T18:07:37.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.628+0000 7f00afa00640 1 -- 172.21.15.2:0/3054978910 wait complete. 2024-01-15T18:07:37.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.631+0000 7f00afa00640 1 Processor -- start 2024-01-15T18:07:37.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.631+0000 7f00afa00640 1 -- start start 2024-01-15T18:07:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.633+0000 7f00afa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00a806ce30 0x7f00a81fb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.633+0000 7f00afa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a8140600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.634+0000 7f00afa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 0x7f00a81610c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.634+0000 7f00afa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a8144a50 con 0x7f00a814afd0 2024-01-15T18:07:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.634+0000 7f00afa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a8140b40 con 0x7f00a806ce30 2024-01-15T18:07:37.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.634+0000 7f00afa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a8161600 con 0x7f00a814b3b0 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.635+0000 7f00a7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a8140600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.635+0000 7f00a7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a8140600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:59994/0 (socket says 172.21.15.2:59994) 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.635+0000 7f00a7e00640 1 -- 172.21.15.2:0/2792467824 learned_addr learned my addr 172.21.15.2:0/2792467824 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.635+0000 7f00ad000640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00a806ce30 0x7f00a81fb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.636+0000 7f00a7e00640 1 -- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 msgr2=0x7f00a81610c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.636+0000 7f00a7e00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 0x7f00a81610c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.636+0000 7f00a7e00640 1 -- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00a806ce30 msgr2=0x7f00a81fb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.636+0000 7f00a7e00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00a806ce30 0x7f00a81fb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.636+0000 7f00a7e00640 1 -- 172.21.15.2:0/2792467824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0098009400 con 0x7f00a814afd0 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.636+0000 7f00a7e00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a8140600 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f009000b450 tx=0x7f009000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:37.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.637+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090011830 con 0x7f00a814afd0 2024-01-15T18:07:37.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.637+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a81618a0 con 0x7f00a814afd0 2024-01-15T18:07:37.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.637+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0090011990 con 0x7f00a814afd0 2024-01-15T18:07:37.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.637+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0090011d40 con 0x7f00a814afd0 2024-01-15T18:07:37.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.637+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a8161e60 con 0x7f00a814afd0 2024-01-15T18:07:37.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.639+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a80b1a20 con 0x7f00a814afd0 2024-01-15T18:07:37.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.640+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f009001a400 con 0x7f00a814afd0 2024-01-15T18:07:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.642+0000 7f00a4c00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f008006be90 0x7f008006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:37.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.642+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0090088a70 con 0x7f00a814afd0 2024-01-15T18:07:37.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.644+0000 7f00ad000640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f008006be90 0x7f008006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:37.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.649+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0090055040 con 0x7f00a814afd0 2024-01-15T18:07:37.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:37.656+0000 7f00ad000640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f008006be90 0x7f008006e350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0098036000 tx=0x7f009803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:38.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.032+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471612", "target": ["mon-mgr", ""]}) v1 -- 0x7f00a80b2e90 con 0x7f008006be90 2024-01-15T18:07:38.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.509+0000 7f00a4c00640 1 -- 172.21.15.2:0/2792467824 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0098031ad0 con 0x7f008006be90 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f008006be90 msgr2=0x7f008006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f008006be90 0x7f008006e350 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0098036000 tx=0x7f009803d040 comp rx=0 tx=0).stop 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 msgr2=0x7f00a8140600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a8140600 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f009000b450 tx=0x7f009000b910 comp rx=0 tx=0).stop 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 shutdown_connections 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f008006be90 0x7f008006e350 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00a814b3b0 0x7f00a81610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00a814afd0 0x7f00a8140600 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 --2- 172.21.15.2:0/2792467824 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00a806ce30 0x7f00a81fb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.514+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 >> 172.21.15.2:0/2792467824 conn(0x7f00a80b45f0 msgr2=0x7f00a81491e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:38.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.515+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 shutdown_connections 2024-01-15T18:07:38.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.515+0000 7f00afa00640 1 -- 172.21.15.2:0/2792467824 wait complete. 2024-01-15T18:07:38.543 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471613 2024-01-15T18:07:38.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.834+0000 7fcb1ce00640 1 Processor -- start 2024-01-15T18:07:38.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.835+0000 7fcb1ce00640 1 -- start start 2024-01-15T18:07:38.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.837+0000 7fcb1ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb1806d000 0x7fcb180b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.837+0000 7fcb1ce00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb180b25f0 con 0x7fcb1814afd0 2024-01-15T18:07:38.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.837+0000 7fcb1ce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb180b28d0 con 0x7fcb1814b3b0 2024-01-15T18:07:38.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.837+0000 7fcb1ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb18070ff0 con 0x7fcb1806d000 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.838+0000 7fcb15e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcb1814afd0 0x7fcb1806c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41618/0 (socket says 172.21.15.2:41618) 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.838+0000 7fcb15e00640 1 -- 172.21.15.2:0/593486153 learned_addr learned my addr 172.21.15.2:0/593486153 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.838+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025936349 0 0) 0x7fcb180b25f0 con 0x7fcb1814afd0 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.838+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf8003520 con 0x7fcb1814afd0 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 941846923 0 0) 0x7fcaf8003520 con 0x7fcb1814afd0 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb1806d000 msgr2=0x7fcb180b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb1806d000 0x7fcb180b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 >> v1:172.21.15.137:6789/0 conn(0x7fcb1814b3b0 legacy=0x7fcb180702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb180b2bb0 con 0x7fcb1814afd0 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fcb08001670 con 0x7fcb1814afd0 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 >> v1:172.21.15.2:6789/0 conn(0x7fcb1814afd0 legacy=0x7fcb1806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaf80039a0 0x7fcaf8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb1806d000 0x7fcaf80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaf8004d70 0x7fcaf80051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb18070ff0 con 0x7fcaf8004d70 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800bc80 con 0x7fcaf80039a0 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf800be40 con 0x7fcb1806d000 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.839+0000 7fcb16800640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaf8004d70 0x7fcaf80051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.840+0000 7fcb15e00640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaf80039a0 0x7fcaf8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.840+0000 7fcb16800640 1 -- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb1806d000 msgr2=0x7fcaf80047c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:07:38.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.840+0000 7fcb16800640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcb1806d000 0x7fcaf80047c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.840+0000 7fcb16800640 1 -- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaf80039a0 msgr2=0x7fcaf8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:38.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.840+0000 7fcb16800640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcaf80039a0 0x7fcaf8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.840+0000 7fcb16800640 1 -- 172.21.15.2:0/593486153 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf8003520 con 0x7fcaf8004d70 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.841+0000 7fcb16800640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaf8004d70 0x7fcaf80051f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcb10001d40 tx=0x7fcaf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b1a4815087d5b96 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.841+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb10003b90 con 0x7fcaf8004d70 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.841+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10003cf0 con 0x7fcaf8004d70 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.841+0000 7fcb14a00640 1 -- 172.21.15.2:0/593486153 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb100052b0 con 0x7fcaf8004d70 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.842+0000 7fcb1ce00640 1 -- 172.21.15.2:0/593486153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaf8004d70 msgr2=0x7fcaf80051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.842+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/593486153 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcaf8004d70 0x7fcaf80051f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7fcb10001d40 tx=0x7fcaf80057f0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.842+0000 7fcb16800640 1 -- 172.21.15.2:0/593486153 reap_dead start 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.843+0000 7fcb1ce00640 1 -- 172.21.15.2:0/593486153 shutdown_connections 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.843+0000 7fcb1ce00640 1 -- 172.21.15.2:0/593486153 >> 172.21.15.2:0/593486153 conn(0x7fcb180b45f0 msgr2=0x7fcb180b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:38.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.843+0000 7fcb1ce00640 1 -- 172.21.15.2:0/593486153 shutdown_connections 2024-01-15T18:07:38.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.843+0000 7fcb1ce00640 1 -- 172.21.15.2:0/593486153 wait complete. 2024-01-15T18:07:38.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.845+0000 7fcb1ce00640 1 Processor -- start 2024-01-15T18:07:38.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.845+0000 7fcb1ce00640 1 -- start start 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb1ce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaf8004d70 0x7fcb181f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb1ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 0x7fcb180a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb1ce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb180a93a0 0x7fcb181464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb1ce00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb181fa740 con 0x7fcb18144fb0 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb1ce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb18145440 con 0x7fcb180a93a0 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb1ce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb18145720 con 0x7fcaf8004d70 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb15400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 0x7fcb180a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb15400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 0x7fcb180a8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60026/0 (socket says 172.21.15.2:60026) 2024-01-15T18:07:38.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.846+0000 7fcb15400640 1 -- 172.21.15.2:0/215286244 learned_addr learned my addr 172.21.15.2:0/215286244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:38.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb15400640 1 -- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaf8004d70 msgr2=0x7fcb181f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:38.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb15400640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaf8004d70 0x7fcb181f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb15400640 1 -- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb180a93a0 msgr2=0x7fcb181464e0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-15T18:07:38.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb15400640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb180a93a0 0x7fcb181464e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:38.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb15400640 1 -- 172.21.15.2:0/215286244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb18145a00 con 0x7fcb18144fb0 2024-01-15T18:07:38.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb15400640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 0x7fcb180a8e60 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000b430 tx=0x7fcb0000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:38.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00011830 con 0x7fcb18144fb0 2024-01-15T18:07:38.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb00011990 con 0x7fcb18144fb0 2024-01-15T18:07:38.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00011d40 con 0x7fcb18144fb0 2024-01-15T18:07:38.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.847+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb18145ce0 con 0x7fcb18144fb0 2024-01-15T18:07:38.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.848+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb18146a20 con 0x7fcb18144fb0 2024-01-15T18:07:38.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.850+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fcb0001a400 con 0x7fcb18144fb0 2024-01-15T18:07:38.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.850+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb18146200 con 0x7fcb18144fb0 2024-01-15T18:07:38.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.851+0000 7fcb0e000640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaf006bee0 0x7fcaf006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:38.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.851+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fcb000888c0 con 0x7fcb18144fb0 2024-01-15T18:07:38.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.856+0000 7fcb15e00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaf006bee0 0x7fcaf006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:38.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.856+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb18146200 con 0x7fcb18144fb0 2024-01-15T18:07:38.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:38.866+0000 7fcb15e00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaf006bee0 0x7fcaf006e3a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcb08000f60 tx=0x7fcb08000ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:39.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.282+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471613", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb18146200 con 0x7fcaf006bee0 2024-01-15T18:07:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.643+0000 7fcb0e000640 1 -- 172.21.15.2:0/215286244 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcb18146200 con 0x7fcaf006bee0 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaf006bee0 msgr2=0x7fcaf006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaf006bee0 0x7fcaf006e3a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fcb08000f60 tx=0x7fcb08000ed0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 msgr2=0x7fcb180a8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 0x7fcb180a8e60 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000b430 tx=0x7fcb0000b8f0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 shutdown_connections 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcaf006bee0 0x7fcaf006e3a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcb180a93a0 0x7fcb181464e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcb18144fb0 0x7fcb180a8e60 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 --2- 172.21.15.2:0/215286244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcaf8004d70 0x7fcb181f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 >> 172.21.15.2:0/215286244 conn(0x7fcb180b45f0 msgr2=0x7fcb180b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 shutdown_connections 2024-01-15T18:07:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.646+0000 7fcb1ce00640 1 -- 172.21.15.2:0/215286244 wait complete. 2024-01-15T18:07:39.668 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471614 2024-01-15T18:07:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.875+0000 7f6486800640 1 Processor -- start 2024-01-15T18:07:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.876+0000 7f6486800640 1 -- start start 2024-01-15T18:07:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6486800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f64800d1870 0x7f64800d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6486800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f64800b2210 0x7f648015d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6486800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1c50 0x7f648015d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6486800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64800ce6d0 con 0x7f64800b2210 2024-01-15T18:07:39.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6486800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648015db60 con 0x7f64800d1870 2024-01-15T18:07:39.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6486800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64800d1250 con 0x7f64800d1c50 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.877+0000 7f6484a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f64800b2210 0x7f648015d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f64800b2210 0x7f648015d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60032/0 (socket says 172.21.15.2:60032) 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 -- 172.21.15.2:0/3354708626 learned_addr learned my addr 172.21.15.2:0/3354708626 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6485400640 1 --2- 172.21.15.2:0/3354708626 >> v2:172.21.15.137:3300/0 conn(0x7f64800d1870 0x7f64800d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6485e00640 1 --2- 172.21.15.2:0/3354708626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1c50 0x7f648015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 -- 172.21.15.2:0/3354708626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1c50 msgr2=0x7f648015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 --2- 172.21.15.2:0/3354708626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1c50 0x7f648015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 -- 172.21.15.2:0/3354708626 >> v2:172.21.15.137:3300/0 conn(0x7f64800d1870 msgr2=0x7f64800d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 --2- 172.21.15.2:0/3354708626 >> v2:172.21.15.137:3300/0 conn(0x7f64800d1870 0x7f64800d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.878+0000 7f6484a00640 1 -- 172.21.15.2:0/3354708626 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64800ce8e0 con 0x7f64800b2210 2024-01-15T18:07:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.880+0000 7f6484a00640 1 --2- 172.21.15.2:0/3354708626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648015d0e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6480067890 tx=0x7f646802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938fbb74b47b71ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.880+0000 7f647fe00640 1 -- 172.21.15.2:0/3354708626 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468031870 con 0x7f64800b2210 2024-01-15T18:07:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.880+0000 7f647fe00640 1 -- 172.21.15.2:0/3354708626 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64680319d0 con 0x7f64800b2210 2024-01-15T18:07:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.880+0000 7f647fe00640 1 -- 172.21.15.2:0/3354708626 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6468031d60 con 0x7f64800b2210 2024-01-15T18:07:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 -- 172.21.15.2:0/3354708626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 msgr2=0x7f648015d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 --2- 172.21.15.2:0/3354708626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648015d0e0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f6480067890 tx=0x7f646802b910 comp rx=0 tx=0).stop 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 -- 172.21.15.2:0/3354708626 shutdown_connections 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 --2- 172.21.15.2:0/3354708626 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1c50 0x7f648015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 --2- 172.21.15.2:0/3354708626 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648015d0e0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 --2- 172.21.15.2:0/3354708626 >> v2:172.21.15.137:3300/0 conn(0x7f64800d1870 0x7f64800d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 -- 172.21.15.2:0/3354708626 >> 172.21.15.2:0/3354708626 conn(0x7f64800d30b0 msgr2=0x7f64800d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 -- 172.21.15.2:0/3354708626 shutdown_connections 2024-01-15T18:07:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.881+0000 7f6486800640 1 -- 172.21.15.2:0/3354708626 wait complete. 2024-01-15T18:07:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.882+0000 7f6486800640 1 Processor -- start 2024-01-15T18:07:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.883+0000 7f6486800640 1 -- start start 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.883+0000 7f6486800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648014a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.883+0000 7f6486800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1870 0x7f64800b5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6486800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64800d1c50 0x7f64800b6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6486800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480159980 con 0x7f64800b2210 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6486800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64800cb1d0 con 0x7f64800d1c50 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6486800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64800cb420 con 0x7f64800d1870 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648014a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648014a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60040/0 (socket says 172.21.15.2:60040) 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 -- 172.21.15.2:0/1424069434 learned_addr learned my addr 172.21.15.2:0/1424069434 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 -- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1870 msgr2=0x7f64800b5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1870 0x7f64800b5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 -- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64800d1c50 msgr2=0x7f64800b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64800d1c50 0x7f64800b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 -- 172.21.15.2:0/1424069434 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64680094d0 con 0x7f64800b2210 2024-01-15T18:07:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.884+0000 7f6485400640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648014a550 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f647000d800 tx=0x7f647000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.885+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700096a0 con 0x7f64800b2210 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.885+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6470009800 con 0x7f64800b2210 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.885+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470009b90 con 0x7f64800b2210 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.885+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64800cb700 con 0x7f64800b2210 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.885+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64800cbcc0 con 0x7f64800b2210 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.886+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480074770 con 0x7f64800b2210 2024-01-15T18:07:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.887+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6470009cf0 con 0x7f64800b2210 2024-01-15T18:07:39.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.888+0000 7f647d600640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f645806bec0 0x7f645806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:39.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.888+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6470088c70 con 0x7f64800b2210 2024-01-15T18:07:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.888+0000 7f6484a00640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f645806bec0 0x7f645806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:39.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.890+0000 7f6484a00640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f645806bec0 0x7f645806e380 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6468036000 tx=0x7f6468030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:39.892+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64700552d0 con 0x7f64800b2210 2024-01-15T18:07:40.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.141+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471614", "target": ["mon-mgr", ""]}) v1 -- 0x7f64800d0f20 con 0x7f645806bec0 2024-01-15T18:07:40.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.693+0000 7f647d600640 1 -- 172.21.15.2:0/1424069434 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6468031bc0 con 0x7f645806bec0 2024-01-15T18:07:40.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f645806bec0 msgr2=0x7f645806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:40.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f645806bec0 0x7f645806e380 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6468036000 tx=0x7f6468030000 comp rx=0 tx=0).stop 2024-01-15T18:07:40.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 msgr2=0x7f648014a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:40.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648014a550 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f647000d800 tx=0x7f647000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 shutdown_connections 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f645806bec0 0x7f645806e380 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64800d1c50 0x7f64800b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64800d1870 0x7f64800b5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 --2- 172.21.15.2:0/1424069434 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64800b2210 0x7f648014a550 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.697+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 >> 172.21.15.2:0/1424069434 conn(0x7f64800d30b0 msgr2=0x7f64800b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.698+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 shutdown_connections 2024-01-15T18:07:40.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:40.698+0000 7f6486800640 1 -- 172.21.15.2:0/1424069434 wait complete. 2024-01-15T18:07:40.724 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471615 2024-01-15T18:07:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.010+0000 7f694be00640 1 Processor -- start 2024-01-15T18:07:41.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.011+0000 7f694be00640 1 -- start start 2024-01-15T18:07:41.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.013+0000 7f694be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f694c15d120 0x7f694c0b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:41.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.013+0000 7f694be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cddd0 0x7f694c0ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:41.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.013+0000 7f694be00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c0ceb40 con 0x7f694c0cf920 2024-01-15T18:07:41.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.013+0000 7f694be00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f694c0cf100 con 0x7f694c15d120 2024-01-15T18:07:41.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.013+0000 7f694be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0cee20 con 0x7f694c0cddd0 2024-01-15T18:07:41.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.014+0000 7f694aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f694c15d120 0x7f694c0b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:41.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.014+0000 7f694aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f694c15d120 0x7f694c0b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60050/0 (socket says 172.21.15.2:60050) 2024-01-15T18:07:41.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.014+0000 7f694aa00640 1 -- 172.21.15.2:0/1635326531 learned_addr learned my addr 172.21.15.2:0/1635326531 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:41.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.015+0000 7f694aa00640 1 -- 172.21.15.2:0/1635326531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cddd0 msgr2=0x7f694c0ce260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:07:41.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.015+0000 7f694aa00640 1 --2- 172.21.15.2:0/1635326531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cddd0 0x7f694c0ce260 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.015+0000 7f694aa00640 1 -- 172.21.15.2:0/1635326531 >> v1:172.21.15.2:6789/0 conn(0x7f694c0cf920 legacy=0x7f694c0cfd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:41.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.015+0000 7f694aa00640 1 -- 172.21.15.2:0/1635326531 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c0d0d30 con 0x7f694c15d120 2024-01-15T18:07:41.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.017+0000 7f694aa00640 1 --2- 172.21.15.2:0/1635326531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0b2800 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f6934009870 tx=0x7f693402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=accb55b0ffa31699 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:41.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.018+0000 7f6949600640 1 -- 172.21.15.2:0/1635326531 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934031870 con 0x7f694c15d120 2024-01-15T18:07:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.018+0000 7f6949600640 1 -- 172.21.15.2:0/1635326531 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69340319d0 con 0x7f694c15d120 2024-01-15T18:07:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.018+0000 7f6949600640 1 -- 172.21.15.2:0/1635326531 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6934031d60 con 0x7f694c15d120 2024-01-15T18:07:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.019+0000 7f694be00640 1 -- 172.21.15.2:0/1635326531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 msgr2=0x7f694c0b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:41.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.019+0000 7f694be00640 1 --2- 172.21.15.2:0/1635326531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0b2800 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f6934009870 tx=0x7f693402b910 comp rx=0 tx=0).stop 2024-01-15T18:07:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.020+0000 7f694be00640 1 -- 172.21.15.2:0/1635326531 shutdown_connections 2024-01-15T18:07:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.020+0000 7f694be00640 1 --2- 172.21.15.2:0/1635326531 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cddd0 0x7f694c0ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.020+0000 7f694be00640 1 --2- 172.21.15.2:0/1635326531 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0b2800 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.020+0000 7f694be00640 1 -- 172.21.15.2:0/1635326531 >> 172.21.15.2:0/1635326531 conn(0x7f694c09c3e0 msgr2=0x7f694c0d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:41.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.020+0000 7f694be00640 1 -- 172.21.15.2:0/1635326531 shutdown_connections 2024-01-15T18:07:41.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.020+0000 7f694be00640 1 -- 172.21.15.2:0/1635326531 wait complete. 2024-01-15T18:07:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.022+0000 7f694be00640 1 Processor -- start 2024-01-15T18:07:41.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.023+0000 7f694be00640 1 -- start start 2024-01-15T18:07:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.024+0000 7f694be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694c0cddd0 0x7f694c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cf920 0x7f694c0c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:41.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0c9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0cba20 con 0x7f694c15d120 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694c1f7430 con 0x7f694c0cddd0 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c0cd200 con 0x7f694c0cf920 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0c9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0c9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60052/0 (socket says 172.21.15.2:60052) 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.025+0000 7f694b400640 1 -- 172.21.15.2:0/1657570657 learned_addr learned my addr 172.21.15.2:0/1657570657 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.026+0000 7f694a000640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cf920 0x7f694c0c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.026+0000 7f694b400640 1 -- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cf920 msgr2=0x7f694c0c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.026+0000 7f694b400640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cf920 0x7f694c0c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.026+0000 7f694b400640 1 -- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694c0cddd0 msgr2=0x7f694c0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.026+0000 7f694b400640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694c0cddd0 0x7f694c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.026+0000 7f694b400640 1 -- 172.21.15.2:0/1657570657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69340094d0 con 0x7f694c15d120 2024-01-15T18:07:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.027+0000 7f694b400640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0c9240 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f694400b990 tx=0x7f694400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.027+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694400cba0 con 0x7f694c15d120 2024-01-15T18:07:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.027+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c0cbf00 con 0x7f694c15d120 2024-01-15T18:07:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.027+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694400cd00 con 0x7f694c15d120 2024-01-15T18:07:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.027+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944014680 con 0x7f694c15d120 2024-01-15T18:07:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.027+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c0cc4c0 con 0x7f694c15d120 2024-01-15T18:07:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.029+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c074770 con 0x7f694c15d120 2024-01-15T18:07:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.030+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f694401d3b0 con 0x7f694c15d120 2024-01-15T18:07:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.032+0000 7f6942a00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f692406be90 0x7f692406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:41.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.033+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f69440887e0 con 0x7f694c15d120 2024-01-15T18:07:41.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.041+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6944054e40 con 0x7f694c15d120 2024-01-15T18:07:41.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.049+0000 7f694aa00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f692406be90 0x7f692406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:41.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.055+0000 7f694aa00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f692406be90 0x7f692406e350 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6934036000 tx=0x7f6934030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.438+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471615", "target": ["mon-mgr", ""]}) v1 -- 0x7f694c0b2d40 con 0x7f692406be90 2024-01-15T18:07:41.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.888+0000 7f6942a00640 1 -- 172.21.15.2:0/1657570657 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f693403e070 con 0x7f692406be90 2024-01-15T18:07:41.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f692406be90 msgr2=0x7f692406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:41.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f692406be90 0x7f692406e350 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6934036000 tx=0x7f6934030000 comp rx=0 tx=0).stop 2024-01-15T18:07:41.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 msgr2=0x7f694c0c9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:41.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0c9240 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f694400b990 tx=0x7f694400be50 comp rx=0 tx=0).stop 2024-01-15T18:07:41.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 shutdown_connections 2024-01-15T18:07:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f692406be90 0x7f692406e350 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f694c15d120 0x7f694c0c9240 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f694c0cf920 0x7f694c0c8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 --2- 172.21.15.2:0/1657570657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f694c0cddd0 0x7f694c0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 >> 172.21.15.2:0/1657570657 conn(0x7f694c09c3e0 msgr2=0x7f694c0b3440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 shutdown_connections 2024-01-15T18:07:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:41.892+0000 7f694be00640 1 -- 172.21.15.2:0/1657570657 wait complete. 2024-01-15T18:07:41.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471616 2024-01-15T18:07:42.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.212+0000 7f61a5800640 1 Processor -- start 2024-01-15T18:07:42.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.212+0000 7f61a5800640 1 -- start start 2024-01-15T18:07:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.213+0000 7f61a5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a0070110 0x7f61a0164940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.213+0000 7f61a5800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a006e030 con 0x7f61a0070b20 2024-01-15T18:07:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.213+0000 7f61a5800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a0070880 con 0x7f61a006dbb0 2024-01-15T18:07:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.213+0000 7f61a5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00b2b50 con 0x7f61a0070110 2024-01-15T18:07:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.214+0000 7f619fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f61a0070b20 0x7f61a0070f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41624/0 (socket says 172.21.15.2:41624) 2024-01-15T18:07:42.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.214+0000 7f619fe00640 1 -- 172.21.15.2:0/247710058 learned_addr learned my addr 172.21.15.2:0/247710058 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.214+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788426058 0 0) 0x7f6190001110 con 0x7f61a0070b20 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.214+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6180003520 con 0x7f61a0070b20 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3093225095 0 0) 0x7f6180003520 con 0x7f61a0070b20 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a0070110 msgr2=0x7f61a0164940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a0070110 0x7f61a0164940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 >> v1:172.21.15.137:6789/0 conn(0x7f61a006dbb0 legacy=0x7f61a0161210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a0159d10 con 0x7f61a0070b20 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6190001800 con 0x7f61a0070b20 2024-01-15T18:07:42.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 >> v1:172.21.15.2:6789/0 conn(0x7f61a0070b20 legacy=0x7f61a0070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61800039a0 0x7f6180003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61800047c0 0x7f6180004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a0070110 0x7f618000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00b2b50 con 0x7f61800047c0 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6180005500 con 0x7f61800039a0 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f618000bec0 con 0x7f61a0070110 2024-01-15T18:07:42.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.215+0000 7f619f400640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61800047c0 0x7f6180004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.216+0000 7f619f400640 1 -- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a0070110 msgr2=0x7f618000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.216+0000 7f619f400640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61a0070110 0x7f618000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.216+0000 7f619f400640 1 -- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61800039a0 msgr2=0x7f6180003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.216+0000 7f619fe00640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61800039a0 0x7f6180003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.216+0000 7f619f400640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61800039a0 0x7f6180003da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:42.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.216+0000 7f619f400640 1 -- 172.21.15.2:0/247710058 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6180003520 con 0x7f61800047c0 2024-01-15T18:07:42.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.217+0000 7f619f400640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61800047c0 0x7f6180004c40 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6188003db0 tx=0x7f61800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5069e8d070c156da server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:42.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.218+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188032600 con 0x7f61800047c0 2024-01-15T18:07:42.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.218+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618802e050 con 0x7f61800047c0 2024-01-15T18:07:42.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.218+0000 7f619ea00640 1 -- 172.21.15.2:0/247710058 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188032a20 con 0x7f61800047c0 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a5800640 1 -- 172.21.15.2:0/247710058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61800047c0 msgr2=0x7f6180004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a5800640 1 --2- 172.21.15.2:0/247710058 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61800047c0 0x7f6180004c40 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f6188003db0 tx=0x7f61800057f0 comp rx=0 tx=0).stop 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a4e00640 1 -- 172.21.15.2:0/247710058 reap_dead start 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a5800640 1 -- 172.21.15.2:0/247710058 shutdown_connections 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a5800640 1 -- 172.21.15.2:0/247710058 >> 172.21.15.2:0/247710058 conn(0x7f61a00b0840 msgr2=0x7f61a00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a5800640 1 -- 172.21.15.2:0/247710058 shutdown_connections 2024-01-15T18:07:42.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.219+0000 7f61a5800640 1 -- 172.21.15.2:0/247710058 wait complete. 2024-01-15T18:07:42.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.221+0000 7f61a5800640 1 Processor -- start 2024-01-15T18:07:42.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.221+0000 7f61a5800640 1 -- start start 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61800047c0 0x7f61a0140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a5800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a0070f00 0x7f61a0156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 0x7f61a0156950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a5800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00ccbd0 con 0x7f61a01451f0 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a5800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00cd100 con 0x7f61a0070f00 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a0153a70 con 0x7f61800047c0 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 0x7f61a0156950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 0x7f61a0156950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60056/0 (socket says 172.21.15.2:60056) 2024-01-15T18:07:42.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.222+0000 7f61a4e00640 1 -- 172.21.15.2:0/1304737750 learned_addr learned my addr 172.21.15.2:0/1304737750 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:42.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f61a4e00640 1 -- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61800047c0 msgr2=0x7f61a0140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:42.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f61a4e00640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61800047c0 0x7f61a0140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:42.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f61a4e00640 1 -- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a0070f00 msgr2=0x7f61a0156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:42.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f61a4e00640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a0070f00 0x7f61a0156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f61a4e00640 1 -- 172.21.15.2:0/1304737750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00cce20 con 0x7f61a01451f0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f61a4e00640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 0x7f61a0156950 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f61a006d7d0 tx=0x7f619800ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.223+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619800aa20 con 0x7f61a01451f0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.224+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619800aba0 con 0x7f61a01451f0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.224+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198012400 con 0x7f61a01451f0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.224+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a0144cd0 con 0x7f61a01451f0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.224+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a0156e90 con 0x7f61a01451f0 2024-01-15T18:07:42.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.225+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a006e430 con 0x7f61a01451f0 2024-01-15T18:07:42.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.226+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6198007070 con 0x7f61a01451f0 2024-01-15T18:07:42.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.227+0000 7f6197e00640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617806be90 0x7f617806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:42.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.227+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f61980883c0 con 0x7f61a01451f0 2024-01-15T18:07:42.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.235+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6198054a20 con 0x7f61a01451f0 2024-01-15T18:07:42.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.247+0000 7f619fe00640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617806be90 0x7f617806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.253+0000 7f619fe00640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617806be90 0x7f617806e350 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f61900012b0 tx=0x7f6190009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:42.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:42.601+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471616", "target": ["mon-mgr", ""]}) v1 -- 0x7f61a0153d50 con 0x7f617806be90 2024-01-15T18:07:43.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.149+0000 7f6197e00640 1 -- 172.21.15.2:0/1304737750 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f61900070d0 con 0x7f617806be90 2024-01-15T18:07:43.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.152+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617806be90 msgr2=0x7f617806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:43.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.152+0000 7f61a5800640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617806be90 0x7f617806e350 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f61900012b0 tx=0x7f6190009040 comp rx=0 tx=0).stop 2024-01-15T18:07:43.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.152+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 msgr2=0x7f61a0156950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:43.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.152+0000 7f61a5800640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 0x7f61a0156950 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f61a006d7d0 tx=0x7f619800ccf0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 shutdown_connections 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f617806be90 0x7f617806e350 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61a01451f0 0x7f61a0156950 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61a0070f00 0x7f61a0156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 --2- 172.21.15.2:0/1304737750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61800047c0 0x7f61a0140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 >> 172.21.15.2:0/1304737750 conn(0x7f61a00b0840 msgr2=0x7f61a00b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 shutdown_connections 2024-01-15T18:07:43.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.153+0000 7f61a5800640 1 -- 172.21.15.2:0/1304737750 wait complete. 2024-01-15T18:07:43.177 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471617 2024-01-15T18:07:43.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.378+0000 7eff78600640 1 Processor -- start 2024-01-15T18:07:43.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.378+0000 7eff78600640 1 -- start start 2024-01-15T18:07:43.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.379+0000 7eff78600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7eff7014afd0 0x7eff700b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.379+0000 7eff78600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7eff7014b3b0 0x7eff7006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.379+0000 7eff78600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff700b25f0 con 0x7eff7006ce30 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.379+0000 7eff78600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff70070ff0 con 0x7eff7014b3b0 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.379+0000 7eff78600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff7006d2a0 con 0x7eff7014afd0 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7eff7014b3b0 0x7eff7006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7eff7014b3b0 0x7eff7006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56674/0 (socket says 172.21.15.2:56674) 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 -- 172.21.15.2:0/1636473306 learned_addr learned my addr 172.21.15.2:0/1636473306 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75c00640 1 --2- 172.21.15.2:0/1636473306 >> v2:172.21.15.137:3300/0 conn(0x7eff7014afd0 0x7eff700b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 -- 172.21.15.2:0/1636473306 >> v2:172.21.15.137:3300/0 conn(0x7eff7014afd0 msgr2=0x7eff700b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 --2- 172.21.15.2:0/1636473306 >> v2:172.21.15.137:3300/0 conn(0x7eff7014afd0 0x7eff700b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 -- 172.21.15.2:0/1636473306 >> v1:172.21.15.137:6789/0 conn(0x7eff7006ce30 legacy=0x7eff700702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:43.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.380+0000 7eff75200640 1 -- 172.21.15.2:0/1636473306 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff700b28d0 con 0x7eff7014b3b0 2024-01-15T18:07:43.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.381+0000 7eff75200640 1 --2- 172.21.15.2:0/1636473306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff7006c8f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7eff7006f9f0 tx=0x7eff5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1792dd27f79fbf65 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:43.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff6fe00640 1 -- 172.21.15.2:0/1636473306 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff580317e0 con 0x7eff7014b3b0 2024-01-15T18:07:43.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff6fe00640 1 -- 172.21.15.2:0/1636473306 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff58031940 con 0x7eff7014b3b0 2024-01-15T18:07:43.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff6fe00640 1 -- 172.21.15.2:0/1636473306 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff58031cd0 con 0x7eff7014b3b0 2024-01-15T18:07:43.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff78600640 1 -- 172.21.15.2:0/1636473306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 msgr2=0x7eff7006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:43.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff78600640 1 --2- 172.21.15.2:0/1636473306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff7006c8f0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7eff7006f9f0 tx=0x7eff5802b880 comp rx=0 tx=0).stop 2024-01-15T18:07:43.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff78600640 1 -- 172.21.15.2:0/1636473306 shutdown_connections 2024-01-15T18:07:43.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff78600640 1 --2- 172.21.15.2:0/1636473306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff7006c8f0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff78600640 1 --2- 172.21.15.2:0/1636473306 >> v2:172.21.15.137:3300/0 conn(0x7eff7014afd0 0x7eff700b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.382+0000 7eff78600640 1 -- 172.21.15.2:0/1636473306 >> 172.21.15.2:0/1636473306 conn(0x7eff700b45f0 msgr2=0x7eff700b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:43.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.383+0000 7eff78600640 1 -- 172.21.15.2:0/1636473306 shutdown_connections 2024-01-15T18:07:43.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.383+0000 7eff78600640 1 -- 172.21.15.2:0/1636473306 wait complete. 2024-01-15T18:07:43.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.384+0000 7eff78600640 1 Processor -- start 2024-01-15T18:07:43.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.384+0000 7eff78600640 1 -- start start 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.385+0000 7eff78600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7006ce30 0x7eff700a8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.385+0000 7eff78600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff7014afd0 0x7eff700afbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.385+0000 7eff78600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff700cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.385+0000 7eff78600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff700b0fb0 con 0x7eff7014b3b0 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.385+0000 7eff78600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff700a8c50 con 0x7eff7014afd0 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.385+0000 7eff78600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff700b00f0 con 0x7eff7006ce30 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff700cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff700cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56682/0 (socket says 172.21.15.2:56682) 2024-01-15T18:07:43.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 -- 172.21.15.2:0/2908689179 learned_addr learned my addr 172.21.15.2:0/2908689179 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff75200640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff7014afd0 0x7eff700afbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff75c00640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7006ce30 0x7eff700a8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 -- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7006ce30 msgr2=0x7eff700a8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7006ce30 0x7eff700a8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 -- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff7014afd0 msgr2=0x7eff700afbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff7014afd0 0x7eff700afbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.386+0000 7eff76600640 1 -- 172.21.15.2:0/2908689179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff580094d0 con 0x7eff7014b3b0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.387+0000 7eff76600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff700cdf00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7eff6800c780 tx=0x7eff6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.387+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68003d80 con 0x7eff7014b3b0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.387+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6800ce30 con 0x7eff7014b3b0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.387+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff68005290 con 0x7eff7014b3b0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.387+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff700b0390 con 0x7eff7014b3b0 2024-01-15T18:07:43.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.387+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff700b0bd0 con 0x7eff7014b3b0 2024-01-15T18:07:43.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.388+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff700686f0 con 0x7eff7014b3b0 2024-01-15T18:07:43.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.389+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7eff680053f0 con 0x7eff7014b3b0 2024-01-15T18:07:43.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.390+0000 7eff6d600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff4806be90 0x7eff4806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:43.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.390+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7eff68087c40 con 0x7eff7014b3b0 2024-01-15T18:07:43.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.391+0000 7eff75c00640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff4806be90 0x7eff4806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:43.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.392+0000 7eff75c00640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff4806be90 0x7eff4806e350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7eff60006fd0 tx=0x7eff60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:43.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.395+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff68054300 con 0x7eff7014b3b0 2024-01-15T18:07:43.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:43.624+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471617", "target": ["mon-mgr", ""]}) v1 -- 0x7eff700b0780 con 0x7eff4806be90 2024-01-15T18:07:48.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.568+0000 7eff6d600640 1 -- 172.21.15.2:0/2908689179 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7eff700b0780 con 0x7eff4806be90 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff4806be90 msgr2=0x7eff4806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff4806be90 0x7eff4806e350 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7eff60006fd0 tx=0x7eff60008040 comp rx=0 tx=0).stop 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 msgr2=0x7eff700cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff700cdf00 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7eff6800c780 tx=0x7eff6800cc40 comp rx=0 tx=0).stop 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 shutdown_connections 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7eff4806be90 0x7eff4806e350 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7eff7014b3b0 0x7eff700cdf00 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7eff7014afd0 0x7eff700afbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 --2- 172.21.15.2:0/2908689179 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7eff7006ce30 0x7eff700a8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 >> 172.21.15.2:0/2908689179 conn(0x7eff700b45f0 msgr2=0x7eff70148f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 shutdown_connections 2024-01-15T18:07:48.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.572+0000 7eff78600640 1 -- 172.21.15.2:0/2908689179 wait complete. 2024-01-15T18:07:48.598 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471618 2024-01-15T18:07:48.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.817+0000 7f7742c00640 1 Processor -- start 2024-01-15T18:07:48.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.817+0000 7f7742c00640 1 -- start start 2024-01-15T18:07:48.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.819+0000 7f7742c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:48.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.819+0000 7f7742c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f773c0cdd70 0x7f773c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:48.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.819+0000 7f7742c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f773c0ce230 con 0x7f773c15c450 2024-01-15T18:07:48.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.819+0000 7f7742c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f773c0ce510 con 0x7f773c0cdd70 2024-01-15T18:07:48.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.819+0000 7f7742c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0ce7f0 con 0x7f773c144c00 2024-01-15T18:07:48.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.820+0000 7f773b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f773c0cdd70 0x7f773c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:48.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.820+0000 7f773be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:48.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.820+0000 7f773b400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f773c0cdd70 0x7f773c06c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:45960/0 (socket says 172.21.15.2:45960) 2024-01-15T18:07:48.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.820+0000 7f773b400640 1 -- 172.21.15.2:0/3291309934 learned_addr learned my addr 172.21.15.2:0/3291309934 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:48.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.820+0000 7f773be00640 1 --2- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:58442/0 (socket says 172.21.15.2:58442) 2024-01-15T18:07:48.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.821+0000 7f773b400640 1 -- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 msgr2=0x7f773c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:48.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.821+0000 7f773b400640 1 --2- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.821+0000 7f773b400640 1 -- 172.21.15.2:0/3291309934 >> v1:172.21.15.137:6789/0 conn(0x7f773c15c450 legacy=0x7f773c0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:48.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.821+0000 7f773b400640 1 -- 172.21.15.2:0/3291309934 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c0705f0 con 0x7f773c0cdd70 2024-01-15T18:07:48.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.822+0000 7f773b400640 1 --2- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c0cdd70 0x7f773c06c8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7724009870 tx=0x7f772402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49352c2cdefb1db3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:48.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.823+0000 7f773aa00640 1 -- 172.21.15.2:0/3291309934 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724030840 con 0x7f773c0cdd70 2024-01-15T18:07:48.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.823+0000 7f773aa00640 1 -- 172.21.15.2:0/3291309934 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77240309a0 con 0x7f773c0cdd70 2024-01-15T18:07:48.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.823+0000 7f773aa00640 1 -- 172.21.15.2:0/3291309934 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724030d30 con 0x7f773c0cdd70 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.823+0000 7f7742c00640 1 -- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c0cdd70 msgr2=0x7f773c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.823+0000 7f7742c00640 1 --2- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c0cdd70 0x7f773c06c8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7724009870 tx=0x7f772402b980 comp rx=0 tx=0).stop 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.824+0000 7f7742c00640 1 -- 172.21.15.2:0/3291309934 shutdown_connections 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.824+0000 7f7742c00640 1 --2- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c0cdd70 0x7f773c06c8f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.824+0000 7f7742c00640 1 --2- 172.21.15.2:0/3291309934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.824+0000 7f7742c00640 1 -- 172.21.15.2:0/3291309934 >> 172.21.15.2:0/3291309934 conn(0x7f773c0b45f0 msgr2=0x7f773c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:48.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.824+0000 7f7742c00640 1 -- 172.21.15.2:0/3291309934 shutdown_connections 2024-01-15T18:07:48.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.824+0000 7f7742c00640 1 -- 172.21.15.2:0/3291309934 wait complete. 2024-01-15T18:07:48.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.825+0000 7f7742c00640 1 Processor -- start 2024-01-15T18:07:48.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.826+0000 7f7742c00640 1 -- start start 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.826+0000 7f7742c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 0x7f773c140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.826+0000 7f7742c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c1f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.826+0000 7f7742c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c15c450 0x7f773c141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f7742c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c140ba0 con 0x7f773c0cdd70 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f7742c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f773c1f3470 con 0x7f773c15c450 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f7742c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f773c0b0620 con 0x7f773c144c00 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 0x7f773c140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 0x7f773c140660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56698/0 (socket says 172.21.15.2:56698) 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 -- 172.21.15.2:0/309171934 learned_addr learned my addr 172.21.15.2:0/309171934 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:48.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773b400640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c1f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f7740c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c15c450 0x7f773c141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 -- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 msgr2=0x7f773c1f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c1f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 -- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c15c450 msgr2=0x7f773c141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c15c450 0x7f773c141c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f773be00640 1 -- 172.21.15.2:0/309171934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77240094d0 con 0x7f773c0cdd70 2024-01-15T18:07:48.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.827+0000 7f7740c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c15c450 0x7f773c141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.828+0000 7f773be00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 0x7f773c140660 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f772c00e9e0 tx=0x7f772c00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.828+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c009670 con 0x7f773c0cdd70 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.828+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c00c9d0 con 0x7f773c0cdd70 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.828+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f773c0b0900 con 0x7f773c0cdd70 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.828+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c00cd80 con 0x7f773c0cdd70 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.828+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773c0b0ec0 con 0x7f773c0cdd70 2024-01-15T18:07:48.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.830+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773c0cd7f0 con 0x7f773c0cdd70 2024-01-15T18:07:48.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.831+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f772c013470 con 0x7f773c0cdd70 2024-01-15T18:07:48.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.832+0000 7f7733e00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771406be90 0x7f771406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:48.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.832+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f772c088a40 con 0x7f773c0cdd70 2024-01-15T18:07:48.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.833+0000 7f773b400640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771406be90 0x7f771406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:48.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.834+0000 7f773b400640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771406be90 0x7f771406e350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7724036000 tx=0x7f7724008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:48.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:48.837+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f772c055100 con 0x7f773c0cdd70 2024-01-15T18:07:49.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.070+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471618", "target": ["mon-mgr", ""]}) v1 -- 0x7f773c06ce30 con 0x7f771406be90 2024-01-15T18:07:49.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.238+0000 7f7733e00640 1 -- 172.21.15.2:0/309171934 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7724030b90 con 0x7f771406be90 2024-01-15T18:07:49.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771406be90 msgr2=0x7f771406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771406be90 0x7f771406e350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7724036000 tx=0x7f7724008030 comp rx=0 tx=0).stop 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 msgr2=0x7f773c140660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 0x7f773c140660 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f772c00e9e0 tx=0x7f772c00eea0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 shutdown_connections 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771406be90 0x7f771406e350 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f773c15c450 0x7f773c141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f773c144c00 0x7f773c1f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 --2- 172.21.15.2:0/309171934 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f773c0cdd70 0x7f773c140660 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 >> 172.21.15.2:0/309171934 conn(0x7f773c0b45f0 msgr2=0x7f773c15a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.241+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 shutdown_connections 2024-01-15T18:07:49.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.242+0000 7f7742c00640 1 -- 172.21.15.2:0/309171934 wait complete. 2024-01-15T18:07:49.267 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471619 2024-01-15T18:07:49.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.494+0000 7f0ad9400640 1 Processor -- start 2024-01-15T18:07:49.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.494+0000 7f0ad9400640 1 -- start start 2024-01-15T18:07:49.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.495+0000 7f0ad9400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0ad415c450 0x7f0ad406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:49.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.496+0000 7f0ad9400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad406d290 con 0x7f0ad4144c00 2024-01-15T18:07:49.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.496+0000 7f0ad9400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ad40ce230 con 0x7f0ad406ce30 2024-01-15T18:07:49.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.496+0000 7f0ad9400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ad40ce510 con 0x7f0ad415c450 2024-01-15T18:07:49.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.496+0000 7f0ad2800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0ad4144c00 0x7f0ad40cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60148/0 (socket says 172.21.15.2:60148) 2024-01-15T18:07:49.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.496+0000 7f0ad2800640 1 -- 172.21.15.2:0/1587522825 learned_addr learned my addr 172.21.15.2:0/1587522825 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:49.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.496+0000 7f0ad1e00640 1 --2- 172.21.15.2:0/1587522825 >> v2:172.21.15.137:3300/0 conn(0x7f0ad415c450 0x7f0ad406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:49.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.497+0000 7f0ad1400640 1 -- 172.21.15.2:0/1587522825 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692784753 0 0) 0x7f0ad406d290 con 0x7f0ad4144c00 2024-01-15T18:07:49.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.497+0000 7f0ad1400640 1 -- 172.21.15.2:0/1587522825 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ab4003520 con 0x7f0ad4144c00 2024-01-15T18:07:49.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.497+0000 7f0ad1400640 1 -- 172.21.15.2:0/1587522825 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 322659119 0 0) 0x7f0ab4003520 con 0x7f0ad4144c00 2024-01-15T18:07:49.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.497+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1587522825 >> v1:172.21.15.137:6789/0 conn(0x7f0ad406ce30 legacy=0x7f0ad4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.497+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1587522825 >> v1:172.21.15.2:6789/0 conn(0x7f0ad4144c00 legacy=0x7f0ad40cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.497+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1587522825 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad40ce7f0 con 0x7f0ad415c450 2024-01-15T18:07:49.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.499+0000 7f0ad1e00640 1 --2- 172.21.15.2:0/1587522825 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad406c8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab4846097ebdd821 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:49.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.499+0000 7f0ad1400640 1 -- 172.21.15.2:0/1587522825 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc02f050 con 0x7f0ad415c450 2024-01-15T18:07:49.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.499+0000 7f0ad1400640 1 -- 172.21.15.2:0/1587522825 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc031500 con 0x7f0ad415c450 2024-01-15T18:07:49.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.499+0000 7f0ad1400640 1 -- 172.21.15.2:0/1587522825 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4001670 con 0x7f0ad415c450 2024-01-15T18:07:49.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 -- 172.21.15.2:0/1587522825 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 msgr2=0x7f0ad406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1587522825 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad406c8f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02b820 comp rx=0 tx=0).stop 2024-01-15T18:07:49.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 -- 172.21.15.2:0/1587522825 shutdown_connections 2024-01-15T18:07:49.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1587522825 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad406c8f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 -- 172.21.15.2:0/1587522825 >> 172.21.15.2:0/1587522825 conn(0x7f0ad40b45f0 msgr2=0x7f0ad40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:49.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 -- 172.21.15.2:0/1587522825 shutdown_connections 2024-01-15T18:07:49.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.500+0000 7f0ad9400640 1 -- 172.21.15.2:0/1587522825 wait complete. 2024-01-15T18:07:49.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.502+0000 7f0ad9400640 1 Processor -- start 2024-01-15T18:07:49.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.502+0000 7f0ad9400640 1 -- start start 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ad406ce30 0x7f0ad4154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 0x7f0ad4156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad9400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad4156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad9400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41578d0 con 0x7f0ad4144c00 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad9400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad413dd00 con 0x7f0ad415c450 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41570b0 con 0x7f0ad406ce30 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 0x7f0ad4156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 0x7f0ad4156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56714/0 (socket says 172.21.15.2:56714) 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1880436438 learned_addr learned my addr 172.21.15.2:0/1880436438 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ad406ce30 msgr2=0x7f0ad4154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ad406ce30 0x7f0ad4154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 msgr2=0x7f0ad4156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:49.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad3200640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad4156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:49.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.503+0000 7f0ad1e00640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad4156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:49.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0ad1e00640 1 -- 172.21.15.2:0/1880436438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc009400 con 0x7f0ad4144c00 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0ad1e00640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 0x7f0ad4156630 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0098c0 tx=0x7f0abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc004230 con 0x7f0ad4144c00 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc004390 con 0x7f0ad4144c00 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc03e950 con 0x7f0ad4144c00 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4138e90 con 0x7f0ad4144c00 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.504+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41575f0 con 0x7f0ad4144c00 2024-01-15T18:07:49.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.505+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad40686f0 con 0x7f0ad4144c00 2024-01-15T18:07:49.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.506+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0abc03a040 con 0x7f0ad4144c00 2024-01-15T18:07:49.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.507+0000 7f0acaa00640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0aac06be90 0x7f0aac06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:49.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.507+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0abc0aa650 con 0x7f0ad4144c00 2024-01-15T18:07:49.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.508+0000 7f0ad2800640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0aac06be90 0x7f0aac06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:49.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.509+0000 7f0ad2800640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0aac06be90 0x7f0aac06e350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4001e80 tx=0x7f0ac40010a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:49.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.512+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0abc076cb0 con 0x7f0ad4144c00 2024-01-15T18:07:49.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:49.762+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471619", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ad413de70 con 0x7f0aac06be90 2024-01-15T18:07:50.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.214+0000 7f0acaa00640 1 -- 172.21.15.2:0/1880436438 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0ad413de70 con 0x7f0aac06be90 2024-01-15T18:07:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.218+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0aac06be90 msgr2=0x7f0aac06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.218+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0aac06be90 0x7f0aac06e350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f0ac4001e80 tx=0x7f0ac40010a0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 msgr2=0x7f0ad4156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 0x7f0ad4156630 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0098c0 tx=0x7f0abc0046c0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 shutdown_connections 2024-01-15T18:07:50.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0aac06be90 0x7f0aac06e350 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ad415c450 0x7f0ad4156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ad4144c00 0x7f0ad4156630 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 --2- 172.21.15.2:0/1880436438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ad406ce30 0x7f0ad4154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 >> 172.21.15.2:0/1880436438 conn(0x7f0ad40b45f0 msgr2=0x7f0ad40715f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 shutdown_connections 2024-01-15T18:07:50.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.219+0000 7f0ad9400640 1 -- 172.21.15.2:0/1880436438 wait complete. 2024-01-15T18:07:50.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone471620 2024-01-15T18:07:50.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.550+0000 7f2923e00640 1 Processor -- start 2024-01-15T18:07:50.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.551+0000 7f2923e00640 1 -- start start 2024-01-15T18:07:50.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.552+0000 7f2923e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f291c1439f0 0x7f291c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:50.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.552+0000 7f2923e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f291c0b3420 0x7f291c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:50.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.552+0000 7f2923e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c15dd70 con 0x7f291c0ccc70 2024-01-15T18:07:50.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.552+0000 7f2923e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c0b38b0 con 0x7f291c1439f0 2024-01-15T18:07:50.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.552+0000 7f2923e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c0711b0 con 0x7f291c0b3420 2024-01-15T18:07:50.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.553+0000 7f2921400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f291c1439f0 0x7f291c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:50.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.553+0000 7f2921400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f291c1439f0 0x7f291c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56720/0 (socket says 172.21.15.2:56720) 2024-01-15T18:07:50.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.553+0000 7f2921400640 1 -- 172.21.15.2:0/3723413556 learned_addr learned my addr 172.21.15.2:0/3723413556 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:50.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.554+0000 7f2920a00640 1 --2- 172.21.15.2:0/3723413556 >> v2:172.21.15.137:3300/0 conn(0x7f291c0b3420 0x7f291c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:50.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.554+0000 7f2921400640 1 -- 172.21.15.2:0/3723413556 >> v2:172.21.15.137:3300/0 conn(0x7f291c0b3420 msgr2=0x7f291c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:50.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.554+0000 7f2921400640 1 --2- 172.21.15.2:0/3723413556 >> v2:172.21.15.137:3300/0 conn(0x7f291c0b3420 0x7f291c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.554+0000 7f2921400640 1 -- 172.21.15.2:0/3723413556 >> v1:172.21.15.137:6789/0 conn(0x7f291c0ccc70 legacy=0x7f291c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:50.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.554+0000 7f2921400640 1 -- 172.21.15.2:0/3723413556 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c143dd0 con 0x7f291c1439f0 2024-01-15T18:07:50.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.554+0000 7f2920a00640 1 --2- 172.21.15.2:0/3723413556 >> v2:172.21.15.137:3300/0 conn(0x7f291c0b3420 0x7f291c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:50.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.556+0000 7f2921400640 1 --2- 172.21.15.2:0/3723413556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c1439f0 0x7f291c0702c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f291c072670 tx=0x7f290c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16bd97d1177e2e64 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:50.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.557+0000 7f291be00640 1 -- 172.21.15.2:0/3723413556 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c031870 con 0x7f291c1439f0 2024-01-15T18:07:50.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.557+0000 7f291be00640 1 -- 172.21.15.2:0/3723413556 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c0319d0 con 0x7f291c1439f0 2024-01-15T18:07:50.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.557+0000 7f291be00640 1 -- 172.21.15.2:0/3723413556 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c031d60 con 0x7f291c1439f0 2024-01-15T18:07:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 -- 172.21.15.2:0/3723413556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c1439f0 msgr2=0x7f291c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 --2- 172.21.15.2:0/3723413556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c1439f0 0x7f291c0702c0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f291c072670 tx=0x7f290c02b910 comp rx=0 tx=0).stop 2024-01-15T18:07:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 -- 172.21.15.2:0/3723413556 shutdown_connections 2024-01-15T18:07:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 --2- 172.21.15.2:0/3723413556 >> v2:172.21.15.137:3300/0 conn(0x7f291c0b3420 0x7f291c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 --2- 172.21.15.2:0/3723413556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c1439f0 0x7f291c0702c0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 -- 172.21.15.2:0/3723413556 >> 172.21.15.2:0/3723413556 conn(0x7f291c0b45f0 msgr2=0x7f291c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.558+0000 7f2923e00640 1 -- 172.21.15.2:0/3723413556 shutdown_connections 2024-01-15T18:07:50.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.559+0000 7f2923e00640 1 -- 172.21.15.2:0/3723413556 wait complete. 2024-01-15T18:07:50.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.561+0000 7f2923e00640 1 Processor -- start 2024-01-15T18:07:50.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.561+0000 7f2923e00640 1 -- start start 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.562+0000 7f2923e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f291c0b3420 0x7f291c1e7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2923e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 0x7f291c1e78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2923e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f291c1439f0 0x7f291c1f7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2923e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c15b6e0 con 0x7f291c0ccc70 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2923e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c143190 con 0x7f291c1439f0 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2923e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c15aa60 con 0x7f291c0b3420 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 0x7f291c1e78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 0x7f291c1e78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56730/0 (socket says 172.21.15.2:56730) 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 -- 172.21.15.2:0/2389508547 learned_addr learned my addr 172.21.15.2:0/2389508547 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2921400640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f291c0b3420 0x7f291c1e7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 -- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f291c0b3420 msgr2=0x7f291c1e7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:50.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f291c0b3420 0x7f291c1e7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 -- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f291c1439f0 msgr2=0x7f291c1f7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:50.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f291c1439f0 0x7f291c1f7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:50.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.563+0000 7f2920a00640 1 -- 172.21.15.2:0/2389508547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0094d0 con 0x7f291c0ccc70 2024-01-15T18:07:50.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.564+0000 7f2920a00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 0x7f291c1e78c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f290400c820 tx=0x7f290400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:50.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.564+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29040079d0 con 0x7f291c0ccc70 2024-01-15T18:07:50.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.564+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2904007b30 con 0x7f291c0ccc70 2024-01-15T18:07:50.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.564+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290400ce70 con 0x7f291c0ccc70 2024-01-15T18:07:50.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.564+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f291c15ad40 con 0x7f291c0ccc70 2024-01-15T18:07:50.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.565+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c15b300 con 0x7f291c0ccc70 2024-01-15T18:07:50.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.567+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c15b820 con 0x7f291c0ccc70 2024-01-15T18:07:50.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.568+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f29040113f0 con 0x7f291c0ccc70 2024-01-15T18:07:50.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.569+0000 7f2919600640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f28f406be90 0x7f28f406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:50.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.569+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f29040885d0 con 0x7f291c0ccc70 2024-01-15T18:07:50.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.573+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f291c15b820 con 0x7f291c0ccc70 2024-01-15T18:07:50.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.573+0000 7f2921400640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f28f406be90 0x7f28f406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:50.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.577+0000 7f2921400640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f28f406be90 0x7f28f406e350 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f290c036000 tx=0x7f290c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:50.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:50.925+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone471620", "target": ["mon-mgr", ""]}) v1 -- 0x7f291c0b4de0 con 0x7f28f406be90 2024-01-15T18:07:51.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.302+0000 7f2919600640 1 -- 172.21.15.2:0/2389508547 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f291c0b4de0 con 0x7f28f406be90 2024-01-15T18:07:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f28f406be90 msgr2=0x7f28f406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f28f406be90 0x7f28f406e350 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f290c036000 tx=0x7f290c03c040 comp rx=0 tx=0).stop 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 msgr2=0x7f291c1e78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 0x7f291c1e78c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f290400c820 tx=0x7f290400cc20 comp rx=0 tx=0).stop 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 shutdown_connections 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f28f406be90 0x7f28f406e350 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f291c1439f0 0x7f291c1f7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f291c0ccc70 0x7f291c1e78c0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 --2- 172.21.15.2:0/2389508547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f291c0b3420 0x7f291c1e7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 >> 172.21.15.2:0/2389508547 conn(0x7f291c0b45f0 msgr2=0x7f291c0cb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 shutdown_connections 2024-01-15T18:07:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.306+0000 7f2923e00640 1 -- 172.21.15.2:0/2389508547 wait complete. 2024-01-15T18:07:51.336 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-15T18:07:51.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.626+0000 7fcc1c200640 1 Processor -- start 2024-01-15T18:07:51.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.626+0000 7fcc1c200640 1 -- start start 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc1c200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc14071ae0 0x7fcc14164fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc1c200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc140aeab0 con 0x7fcc1413ce30 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc1c200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc14070380 con 0x7fcc1415e340 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc1c200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc140ce6d0 con 0x7fcc14071ae0 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc1a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc14071ae0 0x7fcc14164fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc19800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc1413ce30 0x7fcc1415dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60156/0 (socket says 172.21.15.2:60156) 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc1a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc14071ae0 0x7fcc14164fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56732/0 (socket says 172.21.15.2:56732) 2024-01-15T18:07:51.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.628+0000 7fcc19800640 1 -- 172.21.15.2:0/4111650035 learned_addr learned my addr 172.21.15.2:0/4111650035 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:51.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.629+0000 7fcc1a200640 1 -- 172.21.15.2:0/4111650035 >> v1:172.21.15.137:6789/0 conn(0x7fcc1415e340 legacy=0x7fcc140cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:51.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.629+0000 7fcc1a200640 1 -- 172.21.15.2:0/4111650035 >> v1:172.21.15.2:6789/0 conn(0x7fcc1413ce30 legacy=0x7fcc1415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:51.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.629+0000 7fcc1a200640 1 -- 172.21.15.2:0/4111650035 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc140ce9b0 con 0x7fcc14071ae0 2024-01-15T18:07:51.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.630+0000 7fcc1a200640 1 --2- 172.21.15.2:0/4111650035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc14164fc0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48d77916d140d551 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:51.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.630+0000 7fcc13e00640 1 -- 172.21.15.2:0/4111650035 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c02cc60 con 0x7fcc14071ae0 2024-01-15T18:07:51.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.630+0000 7fcc13e00640 1 -- 172.21.15.2:0/4111650035 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c02cdc0 con 0x7fcc14071ae0 2024-01-15T18:07:51.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.630+0000 7fcc13e00640 1 -- 172.21.15.2:0/4111650035 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c034660 con 0x7fcc14071ae0 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.631+0000 7fcc1c200640 1 -- 172.21.15.2:0/4111650035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 msgr2=0x7fcc14164fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.631+0000 7fcc1c200640 1 --2- 172.21.15.2:0/4111650035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc14164fc0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c009d90 tx=0x7fcc0c02beb0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.632+0000 7fcc1c200640 1 -- 172.21.15.2:0/4111650035 shutdown_connections 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.632+0000 7fcc1c200640 1 --2- 172.21.15.2:0/4111650035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc14164fc0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.632+0000 7fcc1c200640 1 -- 172.21.15.2:0/4111650035 >> 172.21.15.2:0/4111650035 conn(0x7fcc1406da20 msgr2=0x7fcc1406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.632+0000 7fcc1c200640 1 -- 172.21.15.2:0/4111650035 shutdown_connections 2024-01-15T18:07:51.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.632+0000 7fcc1c200640 1 -- 172.21.15.2:0/4111650035 wait complete. 2024-01-15T18:07:51.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.634+0000 7fcc1c200640 1 Processor -- start 2024-01-15T18:07:51.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.634+0000 7fcc1c200640 1 -- start start 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc1c200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc141f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc1c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc1413ce30 0x7fcc1415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc1c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc1415e340 0x7fcc1415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc1c200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1415bcd0 con 0x7fcc14071ae0 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc1c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140b04f0 con 0x7fcc1415e340 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc1c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140b07d0 con 0x7fcc1413ce30 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.635+0000 7fcc19800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc141f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc141f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56740/0 (socket says 172.21.15.2:56740) 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 -- 172.21.15.2:0/2491419038 learned_addr learned my addr 172.21.15.2:0/2491419038 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:51.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc1a200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc1415e340 0x7fcc1415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc18e00640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc1413ce30 0x7fcc1415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 -- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc1413ce30 msgr2=0x7fcc1415a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc1413ce30 0x7fcc1415a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 -- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc1415e340 msgr2=0x7fcc1415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc1415e340 0x7fcc1415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.636+0000 7fcc19800640 1 -- 172.21.15.2:0/2491419038 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c009a70 con 0x7fcc14071ae0 2024-01-15T18:07:51.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.637+0000 7fcc19800640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc141f6c60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c4a0 tx=0x7fcc0400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:51.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.637+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04005ce0 con 0x7fcc14071ae0 2024-01-15T18:07:51.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.637+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc04005e40 con 0x7fcc14071ae0 2024-01-15T18:07:51.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.637+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc04007270 con 0x7fcc14071ae0 2024-01-15T18:07:51.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.637+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1415b8f0 con 0x7fcc14071ae0 2024-01-15T18:07:51.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.637+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc141f71a0 con 0x7fcc14071ae0 2024-01-15T18:07:51.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.639+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc1415be10 con 0x7fcc14071ae0 2024-01-15T18:07:51.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.640+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fcc04007490 con 0x7fcc14071ae0 2024-01-15T18:07:51.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.641+0000 7fcc11600640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbec06be90 0x7fcbec06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:51.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.641+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fcc04087c70 con 0x7fcc14071ae0 2024-01-15T18:07:51.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.646+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc040543e0 con 0x7fcc14071ae0 2024-01-15T18:07:51.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.657+0000 7fcc18e00640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbec06be90 0x7fcbec06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:51.706+0000 7fcc18e00640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbec06be90 0x7fcbec06e350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc006fd0 tx=0x7fcbfc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:52.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.061+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcc140686f0 con 0x7fcc14071ae0 2024-01-15T18:07:52.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.063+0000 7fcc11600640 1 -- 172.21.15.2:0/2491419038 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7fcc04017990 con 0x7fcc14071ae0 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:07:52.068 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 639, 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 75, 2024-01-15T18:07:52.069 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:52.070 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 3381, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 13960834501, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 28376379392, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 744717733888, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 248753574, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 256829107, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 60, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 71 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 47, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 5, 2024-01-15T18:07:52.071 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 5, 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "a", 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 6993 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.072 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 14, 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:06:49.623999+0000", 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.073 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": { 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "a54b10cc-43b3-4535-9b9c-6282fb405232": { 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "15 ongoing+pending cloning jobs; Avg progress = 22.223% (39s)\n [======......................] (remaining: 2m)", 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.22222666442394257, 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "d5ce14e9-7bd2-441e-847f-17d203843f41": { 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "4 ongoing cloning jobs; Avg progress = 83.335% (39s)\n [=======================.....] (remaining: 7s)", 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": 0.83335000276565552, 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: "add_to_ceph_s": true 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:07:52.074 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbec06be90 msgr2=0x7fcbec06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbec06be90 0x7fcbec06e350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc006fd0 tx=0x7fcbfc008040 comp rx=0 tx=0).stop 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 msgr2=0x7fcc141f6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc141f6c60 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fcc0400c4a0 tx=0x7fcc0400c960 comp rx=0 tx=0).stop 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 shutdown_connections 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcbec06be90 0x7fcbec06e350 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc1415e340 0x7fcc1415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc1413ce30 0x7fcc1415a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 --2- 172.21.15.2:0/2491419038 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc14071ae0 0x7fcc141f6c60 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 >> 172.21.15.2:0/2491419038 conn(0x7fcc1406da20 msgr2=0x7fcc1406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:52.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 shutdown_connections 2024-01-15T18:07:52.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.067+0000 7fcc1c200640 1 -- 172.21.15.2:0/2491419038 wait complete. 2024-01-15T18:07:52.100 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-01-15T18:07:52.100 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.22222666442394257; onpen_p = 0.2857142984867096 2024-01-15T18:07:52.100 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47161 2024-01-15T18:07:52.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.408+0000 7fc092e00640 1 Processor -- start 2024-01-15T18:07:52.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.409+0000 7fc092e00640 1 -- start start 2024-01-15T18:07:52.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.410+0000 7fc092e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c0ce010 0x7fc08c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:52.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.410+0000 7fc092e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc08c06a9f0 0x7fc08c160da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:52.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.410+0000 7fc092e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c0d0ba0 con 0x7fc08c160800 2024-01-15T18:07:52.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.411+0000 7fc092e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c160be0 con 0x7fc08c06a9f0 2024-01-15T18:07:52.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.411+0000 7fc092e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c06ae20 con 0x7fc08c0ce010 2024-01-15T18:07:52.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.411+0000 7fc091a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc08c160800 0x7fc08c0cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60170/0 (socket says 172.21.15.2:60170) 2024-01-15T18:07:52.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.411+0000 7fc091a00640 1 -- 172.21.15.2:0/3175200639 learned_addr learned my addr 172.21.15.2:0/3175200639 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:52.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.411+0000 7fc091000640 1 --2- 172.21.15.2:0/3175200639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c0ce010 0x7fc08c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.411+0000 7fc092400640 1 --2- 172.21.15.2:0/3175200639 >> v2:172.21.15.137:3300/0 conn(0x7fc08c06a9f0 0x7fc08c160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc08be00640 1 -- 172.21.15.2:0/3175200639 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 711753131 0 0) 0x7fc08c0d0ba0 con 0x7fc08c160800 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc08be00640 1 -- 172.21.15.2:0/3175200639 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc06c003520 con 0x7fc08c160800 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc091000640 1 -- 172.21.15.2:0/3175200639 >> v2:172.21.15.137:3300/0 conn(0x7fc08c06a9f0 msgr2=0x7fc08c160da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc091000640 1 --2- 172.21.15.2:0/3175200639 >> v2:172.21.15.137:3300/0 conn(0x7fc08c06a9f0 0x7fc08c160da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc091000640 1 -- 172.21.15.2:0/3175200639 >> v1:172.21.15.2:6789/0 conn(0x7fc08c160800 legacy=0x7fc08c0cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc091000640 1 -- 172.21.15.2:0/3175200639 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c06d250 con 0x7fc08c0ce010 2024-01-15T18:07:52.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.412+0000 7fc092400640 1 --2- 172.21.15.2:0/3175200639 >> v2:172.21.15.137:3300/0 conn(0x7fc08c06a9f0 0x7fc08c160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:07:52.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.414+0000 7fc091000640 1 --2- 172.21.15.2:0/3175200639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c0ce010 0x7fc08c06a4b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc0740097a0 tx=0x7fc07402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b27a78ed434b416b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:52.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.415+0000 7fc08be00640 1 -- 172.21.15.2:0/3175200639 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07402f050 con 0x7fc08c0ce010 2024-01-15T18:07:52.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.415+0000 7fc08be00640 1 -- 172.21.15.2:0/3175200639 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc074031700 con 0x7fc08c0ce010 2024-01-15T18:07:52.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.415+0000 7fc08be00640 1 -- 172.21.15.2:0/3175200639 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc074031a90 con 0x7fc08c0ce010 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 -- 172.21.15.2:0/3175200639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c0ce010 msgr2=0x7fc08c06a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 --2- 172.21.15.2:0/3175200639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c0ce010 0x7fc08c06a4b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc0740097a0 tx=0x7fc07402b7a0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 -- 172.21.15.2:0/3175200639 shutdown_connections 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 --2- 172.21.15.2:0/3175200639 >> v2:172.21.15.137:3300/0 conn(0x7fc08c06a9f0 0x7fc08c160da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 --2- 172.21.15.2:0/3175200639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c0ce010 0x7fc08c06a4b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 -- 172.21.15.2:0/3175200639 >> 172.21.15.2:0/3175200639 conn(0x7fc08c0cfd30 msgr2=0x7fc08c0d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 -- 172.21.15.2:0/3175200639 shutdown_connections 2024-01-15T18:07:52.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.416+0000 7fc092e00640 1 -- 172.21.15.2:0/3175200639 wait complete. 2024-01-15T18:07:52.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.419+0000 7fc092e00640 1 Processor -- start 2024-01-15T18:07:52.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.419+0000 7fc092e00640 1 -- start start 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.421+0000 7fc092e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc08c06a9f0 0x7fc08c0c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.421+0000 7fc092e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c160800 0x7fc08c156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.421+0000 7fc092e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc08c159da0 0x7fc08c1592d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.421+0000 7fc092e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0c7e10 con 0x7fc08c159da0 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.421+0000 7fc092e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c158e40 con 0x7fc08c06a9f0 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.421+0000 7fc092e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0af3b0 con 0x7fc08c160800 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.422+0000 7fc092400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc08c159da0 0x7fc08c1592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.422+0000 7fc092400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc08c159da0 0x7fc08c1592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:56752/0 (socket says 172.21.15.2:56752) 2024-01-15T18:07:52.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.422+0000 7fc092400640 1 -- 172.21.15.2:0/3384934883 learned_addr learned my addr 172.21.15.2:0/3384934883 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:52.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.422+0000 7fc091a00640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc08c06a9f0 0x7fc08c0c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.424+0000 7fc092400640 1 -- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c160800 msgr2=0x7fc08c156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:52.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.424+0000 7fc092400640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc08c160800 0x7fc08c156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.424+0000 7fc092400640 1 -- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc08c06a9f0 msgr2=0x7fc08c0c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.424+0000 7fc092400640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc08c06a9f0 0x7fc08c0c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.424+0000 7fc092400640 1 -- 172.21.15.2:0/3384934883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084009a70 con 0x7fc08c159da0 2024-01-15T18:07:52.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.425+0000 7fc092400640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc08c159da0 0x7fc08c1592d0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fc08400efc0 tx=0x7fc084009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:52.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.425+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08400cde0 con 0x7fc08c159da0 2024-01-15T18:07:52.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.426+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08400f040 con 0x7fc08c159da0 2024-01-15T18:07:52.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.426+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084003bd0 con 0x7fc08c159da0 2024-01-15T18:07:52.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.426+0000 7fc092e00640 1 -- 172.21.15.2:0/3384934883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc074009400 con 0x7fc08c159da0 2024-01-15T18:07:52.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.426+0000 7fc092e00640 1 -- 172.21.15.2:0/3384934883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c1587f0 con 0x7fc08c159da0 2024-01-15T18:07:52.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.427+0000 7fc092e00640 1 -- 172.21.15.2:0/3384934883 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c0cedb0 con 0x7fc08c159da0 2024-01-15T18:07:52.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.429+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc084014480 con 0x7fc08c159da0 2024-01-15T18:07:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.430+0000 7fc089600640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc06406bee0 0x7fc06406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:52.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.430+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fc084089a00 con 0x7fc08c159da0 2024-01-15T18:07:52.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.434+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0840562c0 con 0x7fc08c159da0 2024-01-15T18:07:52.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.466+0000 7fc091a00640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc06406bee0 0x7fc06406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:52.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.501+0000 7fc091a00640 1 --2- 172.21.15.2:0/3384934883 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc06406bee0 0x7fc06406e3a0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc07c000f60 tx=0x7fc07c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:52.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.771+0000 7fc092e00640 1 -- 172.21.15.2:0/3384934883 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47161", "target": ["mon-mgr", ""]}) v1 -- 0x7fc08c071520 con 0x7fc06406bee0 2024-01-15T18:07:52.805 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:52.800+0000 7fe20c000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:52.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:52.802+0000 7fc089600640 1 -- 172.21.15.2:0/3384934883 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fc07c0058d0 con 0x7fc06406bee0 2024-01-15T18:07:52.805 INFO:teuthology.orchestra.run.smithi002.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-15T18:07:52.825 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47162 2024-01-15T18:07:53.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.030+0000 7f856be00640 1 Processor -- start 2024-01-15T18:07:53.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.030+0000 7f856be00640 1 -- start start 2024-01-15T18:07:53.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.031+0000 7f856be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f856c0d1870 0x7f856c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.031+0000 7f856be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.031+0000 7f856be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856c0d1250 con 0x7f856c0d1c50 2024-01-15T18:07:53.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.031+0000 7f856be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856c0ce6d0 con 0x7f856c0d1870 2024-01-15T18:07:53.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.031+0000 7f856be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0ce8e0 con 0x7f856c0b2210 2024-01-15T18:07:53.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856aa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f856c0d1870 0x7f856c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856b400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f856c0d1c50 0x7f856c0cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:45758/0 (socket says 172.21.15.2:45758) 2024-01-15T18:07:53.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856b400640 1 -- 172.21.15.2:0/3959384830 learned_addr learned my addr 172.21.15.2:0/3959384830 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:53.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856aa00640 1 -- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 msgr2=0x7f856c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:53.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856aa00640 1 --2- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c15d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.032+0000 7f856aa00640 1 -- 172.21.15.2:0/3959384830 >> v1:172.21.15.137:6789/0 conn(0x7f856c0d1c50 legacy=0x7f856c0cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.033+0000 7f856aa00640 1 -- 172.21.15.2:0/3959384830 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0cebc0 con 0x7f856c0d1870 2024-01-15T18:07:53.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.034+0000 7f856aa00640 1 --2- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 0x7f856c0d3bd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f855c0097a0 tx=0x7f855c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16de98da849057c5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:53.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.034+0000 7f8569600640 1 -- 172.21.15.2:0/3959384830 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c02d040 con 0x7f856c0d1870 2024-01-15T18:07:53.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.034+0000 7f8569600640 1 -- 172.21.15.2:0/3959384830 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855c031810 con 0x7f856c0d1870 2024-01-15T18:07:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.034+0000 7f8569600640 1 -- 172.21.15.2:0/3959384830 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855c031ba0 con 0x7f856c0d1870 2024-01-15T18:07:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 -- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 msgr2=0x7f856c0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 --2- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 0x7f856c0d3bd0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f855c0097a0 tx=0x7f855c02b8b0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 -- 172.21.15.2:0/3959384830 shutdown_connections 2024-01-15T18:07:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 --2- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c15d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 --2- 172.21.15.2:0/3959384830 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 0x7f856c0d3bd0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 -- 172.21.15.2:0/3959384830 >> 172.21.15.2:0/3959384830 conn(0x7f856c0d30b0 msgr2=0x7f856c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 -- 172.21.15.2:0/3959384830 shutdown_connections 2024-01-15T18:07:53.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.035+0000 7f856be00640 1 -- 172.21.15.2:0/3959384830 wait complete. 2024-01-15T18:07:53.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.037+0000 7f856be00640 1 Processor -- start 2024-01-15T18:07:53.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.037+0000 7f856be00640 1 -- start start 2024-01-15T18:07:53.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 0x7f856c0b6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856c0d1c50 0x7f856c156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c152b00 con 0x7f856c0d1c50 2024-01-15T18:07:53.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0b6a40 con 0x7f856c0d1870 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856c0c97a0 con 0x7f856c0b2210 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856c0d1c50 0x7f856c156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856c0d1c50 0x7f856c156700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53932/0 (socket says 172.21.15.2:53932) 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.038+0000 7f856b400640 1 -- 172.21.15.2:0/2456411147 learned_addr learned my addr 172.21.15.2:0/2456411147 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856aa00640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856a000640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 0x7f856c0b6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856b400640 1 -- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 msgr2=0x7f856c143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856b400640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856c0b2210 0x7f856c143140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856b400640 1 -- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 msgr2=0x7f856c0b6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856b400640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f856c0d1870 0x7f856c0b6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856b400640 1 -- 172.21.15.2:0/2456411147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c009400 con 0x7f856c0d1c50 2024-01-15T18:07:53.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856b400640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856c0d1c50 0x7f856c156700 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f856400caa0 tx=0x7f856400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:53.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f856400ac00 con 0x7f856c0d1c50 2024-01-15T18:07:53.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856400ad60 con 0x7f856c0d1c50 2024-01-15T18:07:53.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8564012600 con 0x7f856c0d1c50 2024-01-15T18:07:53.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.039+0000 7f856be00640 1 -- 172.21.15.2:0/2456411147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c159500 con 0x7f856c0d1c50 2024-01-15T18:07:53.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.040+0000 7f856be00640 1 -- 172.21.15.2:0/2456411147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c0cae90 con 0x7f856c0d1c50 2024-01-15T18:07:53.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.041+0000 7f856be00640 1 -- 172.21.15.2:0/2456411147 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c0caa10 con 0x7f856c0d1c50 2024-01-15T18:07:53.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.041+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8564007070 con 0x7f856c0d1c50 2024-01-15T18:07:53.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.042+0000 7f8562a00640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f854406be90 0x7f854406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.042+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f85640885a0 con 0x7f856c0d1c50 2024-01-15T18:07:53.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.043+0000 7f856aa00640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f854406be90 0x7f854406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.044+0000 7f856aa00640 1 --2- 172.21.15.2:0/2456411147 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f854406be90 0x7f854406e350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f855c036000 tx=0x7f855c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:53.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.047+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f856c0caa10 con 0x7f856c0d1c50 2024-01-15T18:07:53.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.284+0000 7f856be00640 1 -- 172.21.15.2:0/2456411147 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47162", "target": ["mon-mgr", ""]}) v1 -- 0x7f856c0d0f20 con 0x7f854406be90 2024-01-15T18:07:53.472 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:53.468+0000 7fe20c000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:53.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.469+0000 7f8562a00640 1 -- 172.21.15.2:0/2456411147 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f855c031b00 con 0x7f854406be90 2024-01-15T18:07:53.473 INFO:teuthology.orchestra.run.smithi002.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:53.488 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-15T18:07:53.489 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47163 2024-01-15T18:07:53.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.700+0000 7f2e65c00640 1 Processor -- start 2024-01-15T18:07:53.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.700+0000 7f2e65c00640 1 -- start start 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e65c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e6006d250 0x7f2e6006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e65c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2e6006a9f0 0x7f2e600ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e65c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e600ce880 con 0x7f2e6006f2b0 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e65c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6006ae60 con 0x7f2e6006d250 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e65c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e600ca610 con 0x7f2e6006a9f0 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e5f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e6006d250 0x7f2e6006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e5f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e6006d250 0x7f2e6006a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53948/0 (socket says 172.21.15.2:53948) 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.702+0000 7f2e5f400640 1 -- 172.21.15.2:0/309549733 learned_addr learned my addr 172.21.15.2:0/309549733 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:53.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.703+0000 7f2e65200640 1 --2- 172.21.15.2:0/309549733 >> v2:172.21.15.137:3300/0 conn(0x7f2e6006a9f0 0x7f2e600ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.703+0000 7f2e5f400640 1 -- 172.21.15.2:0/309549733 >> v2:172.21.15.137:3300/0 conn(0x7f2e6006a9f0 msgr2=0x7f2e600ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:53.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.703+0000 7f2e5f400640 1 --2- 172.21.15.2:0/309549733 >> v2:172.21.15.137:3300/0 conn(0x7f2e6006a9f0 0x7f2e600ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.703+0000 7f2e5f400640 1 -- 172.21.15.2:0/309549733 >> v1:172.21.15.137:6789/0 conn(0x7f2e6006f2b0 legacy=0x7f2e60071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:53.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.703+0000 7f2e5f400640 1 -- 172.21.15.2:0/309549733 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e600cdc70 con 0x7f2e6006d250 2024-01-15T18:07:53.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.704+0000 7f2e5f400640 1 --2- 172.21.15.2:0/309549733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006d250 0x7f2e6006a4b0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f2e600d0850 tx=0x7f2e4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb49cb41465ac745 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:53.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.704+0000 7f2e5ea00640 1 -- 172.21.15.2:0/309549733 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48031870 con 0x7f2e6006d250 2024-01-15T18:07:53.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.704+0000 7f2e5ea00640 1 -- 172.21.15.2:0/309549733 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e480319d0 con 0x7f2e6006d250 2024-01-15T18:07:53.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.704+0000 7f2e5ea00640 1 -- 172.21.15.2:0/309549733 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48031d60 con 0x7f2e6006d250 2024-01-15T18:07:53.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 -- 172.21.15.2:0/309549733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006d250 msgr2=0x7f2e6006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 --2- 172.21.15.2:0/309549733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006d250 0x7f2e6006a4b0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f2e600d0850 tx=0x7f2e4802b910 comp rx=0 tx=0).stop 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 -- 172.21.15.2:0/309549733 shutdown_connections 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 --2- 172.21.15.2:0/309549733 >> v2:172.21.15.137:3300/0 conn(0x7f2e6006a9f0 0x7f2e600ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 --2- 172.21.15.2:0/309549733 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006d250 0x7f2e6006a4b0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 -- 172.21.15.2:0/309549733 >> 172.21.15.2:0/309549733 conn(0x7f2e600cfd30 msgr2=0x7f2e600d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 -- 172.21.15.2:0/309549733 shutdown_connections 2024-01-15T18:07:53.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.705+0000 7f2e65c00640 1 -- 172.21.15.2:0/309549733 wait complete. 2024-01-15T18:07:53.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.707+0000 7f2e65c00640 1 Processor -- start 2024-01-15T18:07:53.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.707+0000 7f2e65c00640 1 -- start start 2024-01-15T18:07:53.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.708+0000 7f2e65c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6006a9f0 0x7f2e600b5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.708+0000 7f2e65c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e6006d250 0x7f2e60137870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.708+0000 7f2e65c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006f2b0 0x7f2e60137db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.708+0000 7f2e65c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e60135980 con 0x7f2e6006f2b0 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.708+0000 7f2e65c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e60135c60 con 0x7f2e6006a9f0 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.708+0000 7f2e65c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e60135f40 con 0x7f2e6006d250 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006f2b0 0x7f2e60137db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006f2b0 0x7f2e60137db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53958/0 (socket says 172.21.15.2:53958) 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 -- 172.21.15.2:0/2072064250 learned_addr learned my addr 172.21.15.2:0/2072064250 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e5fe00640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6006a9f0 0x7f2e600b5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e5f400640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e6006d250 0x7f2e60137870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 -- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e6006d250 msgr2=0x7f2e60137870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e6006d250 0x7f2e60137870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 -- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6006a9f0 msgr2=0x7f2e600b5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:53.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6006a9f0 0x7f2e600b5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:53.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 -- 172.21.15.2:0/2072064250 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480094d0 con 0x7f2e6006f2b0 2024-01-15T18:07:53.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e5fe00640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6006a9f0 0x7f2e600b5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:07:53.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.709+0000 7f2e65200640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6006f2b0 0x7f2e60137db0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800c550 tx=0x7f2e5800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.710+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58004220 con 0x7f2e6006f2b0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.710+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e58004380 con 0x7f2e6006f2b0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.710+0000 7f2e65c00640 1 -- 172.21.15.2:0/2072064250 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e60136220 con 0x7f2e6006f2b0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.710+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58003830 con 0x7f2e6006f2b0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.710+0000 7f2e65c00640 1 -- 172.21.15.2:0/2072064250 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e60143ab0 con 0x7f2e6006f2b0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.711+0000 7f2e65c00640 1 -- 172.21.15.2:0/2072064250 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e60074770 con 0x7f2e6006f2b0 2024-01-15T18:07:53.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.712+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2e58003990 con 0x7f2e6006f2b0 2024-01-15T18:07:53.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.713+0000 7f2e57e00640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e3806be90 0x7f2e3806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:53.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.713+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f2e580882b0 con 0x7f2e6006f2b0 2024-01-15T18:07:53.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.714+0000 7f2e5fe00640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e3806be90 0x7f2e3806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:53.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.715+0000 7f2e5fe00640 1 --2- 172.21.15.2:0/2072064250 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e3806be90 0x7f2e3806e350 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2e50004520 tx=0x7f2e50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:53.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.718+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e58054b70 con 0x7f2e6006f2b0 2024-01-15T18:07:53.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.950+0000 7f2e65c00640 1 -- 172.21.15.2:0/2072064250 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47163", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e60135dd0 con 0x7f2e3806be90 2024-01-15T18:07:53.976 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:53.971+0000 7fe20c000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:53.973+0000 7f2e57e00640 1 -- 172.21.15.2:0/2072064250 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f2e60135dd0 con 0x7f2e3806be90 2024-01-15T18:07:53.977 INFO:teuthology.orchestra.run.smithi002.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:53.988 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-15T18:07:53.989 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47164 2024-01-15T18:07:54.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.209+0000 7f268ac00640 1 Processor -- start 2024-01-15T18:07:54.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.209+0000 7f268ac00640 1 -- start start 2024-01-15T18:07:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f268ac00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2684144c00 0x7f26840cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f268ac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26840ce230 con 0x7f268406d000 2024-01-15T18:07:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f268ac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26840ce510 con 0x7f26840cdd70 2024-01-15T18:07:54.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f268ac00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26840ce7f0 con 0x7f2684144c00 2024-01-15T18:07:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f2683e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2684144c00 0x7f26840cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f2688c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f268406d000 0x7f2684070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:58390/0 (socket says 172.21.15.2:58390) 2024-01-15T18:07:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.211+0000 7f2688c00640 1 -- 172.21.15.2:0/917894673 learned_addr learned my addr 172.21.15.2:0/917894673 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.212+0000 7f2683e00640 1 -- 172.21.15.2:0/917894673 >> v1:172.21.15.137:6789/0 conn(0x7f26840cdd70 legacy=0x7f268406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.212+0000 7f2683e00640 1 -- 172.21.15.2:0/917894673 >> v1:172.21.15.2:6789/0 conn(0x7f268406d000 legacy=0x7f2684070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.212+0000 7f2683e00640 1 -- 172.21.15.2:0/917894673 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840705f0 con 0x7f2684144c00 2024-01-15T18:07:54.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.212+0000 7f2682a00640 1 -- 172.21.15.2:0/917894673 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947385670 0 0) 0x7f26840ce230 con 0x7f268406d000 2024-01-15T18:07:54.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.213+0000 7f2683e00640 1 --2- 172.21.15.2:0/917894673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f26840cd830 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f26740097a0 tx=0x7f267402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6a254ac7c7fdbb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:54.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.213+0000 7f2682a00640 1 -- 172.21.15.2:0/917894673 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674031830 con 0x7f2684144c00 2024-01-15T18:07:54.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.213+0000 7f2682a00640 1 -- 172.21.15.2:0/917894673 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2674031990 con 0x7f2684144c00 2024-01-15T18:07:54.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.213+0000 7f2682a00640 1 -- 172.21.15.2:0/917894673 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674031d20 con 0x7f2684144c00 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 -- 172.21.15.2:0/917894673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 msgr2=0x7f26840cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 --2- 172.21.15.2:0/917894673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f26840cd830 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f26740097a0 tx=0x7f267402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 -- 172.21.15.2:0/917894673 shutdown_connections 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 --2- 172.21.15.2:0/917894673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f26840cd830 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 -- 172.21.15.2:0/917894673 >> 172.21.15.2:0/917894673 conn(0x7f26840b45f0 msgr2=0x7f26840b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 -- 172.21.15.2:0/917894673 shutdown_connections 2024-01-15T18:07:54.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.214+0000 7f268ac00640 1 -- 172.21.15.2:0/917894673 wait complete. 2024-01-15T18:07:54.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.216+0000 7f268ac00640 1 Processor -- start 2024-01-15T18:07:54.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.216+0000 7f268ac00640 1 -- start start 2024-01-15T18:07:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.217+0000 7f268ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f268406d000 0x7f2684156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.217+0000 7f268ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26840cdd70 0x7f26840b13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.217+0000 7f268ac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f268412fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.217+0000 7f268ac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268412f8e0 con 0x7f2684144c00 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.217+0000 7f268ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268412ec60 con 0x7f268406d000 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.217+0000 7f268ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268412ef40 con 0x7f26840cdd70 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f268412fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f268412fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53978/0 (socket says 172.21.15.2:53978) 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 -- 172.21.15.2:0/1239562880 learned_addr learned my addr 172.21.15.2:0/1239562880 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2683e00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f268406d000 0x7f2684156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2683400640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26840cdd70 0x7f26840b13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 -- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26840cdd70 msgr2=0x7f26840b13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26840cdd70 0x7f26840b13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 -- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f268406d000 msgr2=0x7f2684156f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f268406d000 0x7f2684156f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.218+0000 7f2688c00640 1 -- 172.21.15.2:0/1239562880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2674009400 con 0x7f2684144c00 2024-01-15T18:07:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.219+0000 7f2688c00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2684144c00 0x7f268412fd00 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f267c001560 tx=0x7f267c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.219+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00a640 con 0x7f2684144c00 2024-01-15T18:07:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.219+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00a7a0 con 0x7f2684144c00 2024-01-15T18:07:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.219+0000 7f268ac00640 1 -- 172.21.15.2:0/1239562880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268412f220 con 0x7f2684144c00 2024-01-15T18:07:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.219+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00ab30 con 0x7f2684144c00 2024-01-15T18:07:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.219+0000 7f268ac00640 1 -- 172.21.15.2:0/1239562880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26841574d0 con 0x7f2684144c00 2024-01-15T18:07:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.220+0000 7f268ac00640 1 -- 172.21.15.2:0/1239562880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268412fa20 con 0x7f2684144c00 2024-01-15T18:07:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.221+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f267c00ac90 con 0x7f2684144c00 2024-01-15T18:07:54.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.222+0000 7f267be00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f265c06be90 0x7f265c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.222+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f267c088050 con 0x7f2684144c00 2024-01-15T18:07:54.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.223+0000 7f2683e00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f265c06be90 0x7f265c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.227 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.224+0000 7f2683e00640 1 --2- 172.21.15.2:0/1239562880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f265c06be90 0x7f265c06e350 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2674036000 tx=0x7f267403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:54.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.226+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f268412fa20 con 0x7f2684144c00 2024-01-15T18:07:54.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.462+0000 7f268ac00640 1 -- 172.21.15.2:0/1239562880 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7f26840b3c70 con 0x7f265c06be90 2024-01-15T18:07:54.505 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:07:54.500+0000 7fe20c000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:54.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.501+0000 7f267be00640 1 -- 172.21.15.2:0/1239562880 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f26840b3c70 con 0x7f265c06be90 2024-01-15T18:07:54.505 INFO:teuthology.orchestra.run.smithi002.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-15T18:07:54.521 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-15T18:07:54.521 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47165 2024-01-15T18:07:54.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.742+0000 7f7c03c00640 1 Processor -- start 2024-01-15T18:07:54.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.743+0000 7f7c03c00640 1 -- start start 2024-01-15T18:07:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.744+0000 7f7c03c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7bfc144c00 0x7f7bfc0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.744+0000 7f7c03c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7bfc06d000 0x7f7bfc070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.744+0000 7f7c03c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc0ce230 con 0x7f7bfc0cdd70 2024-01-15T18:07:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.744+0000 7f7c03c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc0ce510 con 0x7f7bfc06d000 2024-01-15T18:07:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.744+0000 7f7c03c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc0ce7f0 con 0x7f7bfc144c00 2024-01-15T18:07:54.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7bfbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7bfc0cdd70 0x7f7bfc06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:58396/0 (socket says 172.21.15.2:58396) 2024-01-15T18:07:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7bfbe00640 1 -- 172.21.15.2:0/1364982234 learned_addr learned my addr 172.21.15.2:0/1364982234 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:07:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7c01c00640 1 --2- 172.21.15.2:0/1364982234 >> v2:172.21.15.2:3300/0 conn(0x7f7bfc06d000 0x7f7bfc070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7c01200640 1 --2- 172.21.15.2:0/1364982234 >> v2:172.21.15.137:3300/0 conn(0x7f7bfc144c00 0x7f7bfc0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7bfb400640 1 -- 172.21.15.2:0/1364982234 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457017182 0 0) 0x7f7bfc0ce230 con 0x7f7bfc0cdd70 2024-01-15T18:07:54.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7bfb400640 1 -- 172.21.15.2:0/1364982234 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bdc003520 con 0x7f7bfc0cdd70 2024-01-15T18:07:54.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7c01c00640 1 -- 172.21.15.2:0/1364982234 >> v2:172.21.15.137:3300/0 conn(0x7f7bfc144c00 msgr2=0x7f7bfc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:07:54.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7c01c00640 1 --2- 172.21.15.2:0/1364982234 >> v2:172.21.15.137:3300/0 conn(0x7f7bfc144c00 0x7f7bfc0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7c01c00640 1 -- 172.21.15.2:0/1364982234 >> v1:172.21.15.2:6789/0 conn(0x7f7bfc0cdd70 legacy=0x7f7bfc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.745+0000 7f7c01c00640 1 -- 172.21.15.2:0/1364982234 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc0705f0 con 0x7f7bfc06d000 2024-01-15T18:07:54.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.746+0000 7f7c01200640 1 --2- 172.21.15.2:0/1364982234 >> v2:172.21.15.137:3300/0 conn(0x7f7bfc144c00 0x7f7bfc0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.747+0000 7f7c01c00640 1 --2- 172.21.15.2:0/1364982234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc070b10 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0b1c60 tx=0x7f7bf402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c760fc8843477e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.747+0000 7f7bfb400640 1 -- 172.21.15.2:0/1364982234 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf402cc70 con 0x7f7bfc06d000 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.747+0000 7f7bfb400640 1 -- 172.21.15.2:0/1364982234 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf402cdd0 con 0x7f7bfc06d000 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.747+0000 7f7bfb400640 1 -- 172.21.15.2:0/1364982234 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4034660 con 0x7f7bfc06d000 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.747+0000 7f7c03c00640 1 -- 172.21.15.2:0/1364982234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 msgr2=0x7f7bfc070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.747+0000 7f7c03c00640 1 --2- 172.21.15.2:0/1364982234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc070b10 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0b1c60 tx=0x7f7bf402bec0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.748+0000 7f7c03c00640 1 -- 172.21.15.2:0/1364982234 shutdown_connections 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.748+0000 7f7c03c00640 1 --2- 172.21.15.2:0/1364982234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc070b10 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.748+0000 7f7c03c00640 1 --2- 172.21.15.2:0/1364982234 >> v2:172.21.15.137:3300/0 conn(0x7f7bfc144c00 0x7f7bfc0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.748+0000 7f7c03c00640 1 -- 172.21.15.2:0/1364982234 >> 172.21.15.2:0/1364982234 conn(0x7f7bfc0b45f0 msgr2=0x7f7bfc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:07:54.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.748+0000 7f7c03c00640 1 -- 172.21.15.2:0/1364982234 shutdown_connections 2024-01-15T18:07:54.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.748+0000 7f7c03c00640 1 -- 172.21.15.2:0/1364982234 wait complete. 2024-01-15T18:07:54.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.749+0000 7f7c03c00640 1 Processor -- start 2024-01-15T18:07:54.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.750+0000 7f7c03c00640 1 -- start start 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c03c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c03c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0cdd70 0x7f7bfc13e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c03c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc144c00 0x7f7bfc13eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c03c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc0b0640 con 0x7f7bfc06d000 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c03c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc13f1a0 con 0x7f7bfc144c00 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c03c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc159420 con 0x7f7bfc0cdd70 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc158ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54008/0 (socket says 172.21.15.2:54008) 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 -- 172.21.15.2:0/3148489781 learned_addr learned my addr 172.21.15.2:0/3148489781 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 -- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0cdd70 msgr2=0x7f7bfc13e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0cdd70 0x7f7bfc13e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 -- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc144c00 msgr2=0x7f7bfc13eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc144c00 0x7f7bfc13eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.751+0000 7f7c01200640 1 -- 172.21.15.2:0/3148489781 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf4009a70 con 0x7f7bfc06d000 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.752+0000 7f7c01200640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc158ee0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00d7d0 tx=0x7f7bec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.752+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec004030 con 0x7f7bfc06d000 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.752+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec004190 con 0x7f7bfc06d000 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.752+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bec004520 con 0x7f7bfc06d000 2024-01-15T18:07:54.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.752+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bfc0b1470 con 0x7f7bfc06d000 2024-01-15T18:07:54.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.752+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bfc0aff80 con 0x7f7bfc06d000 2024-01-15T18:07:54.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.754+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bfc0686f0 con 0x7f7bfc06d000 2024-01-15T18:07:54.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.754+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7bec0096a0 con 0x7f7bfc06d000 2024-01-15T18:07:54.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.756+0000 7f7bf8c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bd406be90 0x7f7bd406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:07:54.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.756+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f7bec088900 con 0x7f7bfc06d000 2024-01-15T18:07:54.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.756+0000 7f7bfbe00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bd406be90 0x7f7bd406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:07:54.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.757+0000 7f7bfbe00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bd406be90 0x7f7bd406e350 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc15ea10 tx=0x7f7be4007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:07:54.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.760+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bec054fc0 con 0x7f7bfc06d000 2024-01-15T18:07:55.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:07:54.998+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47165", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bfc15ae10 con 0x7f7bd406be90 2024-01-15T18:08:01.588 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:01.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:03.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.505+0000 7f7bf8c00640 1 -- 172.21.15.2:0/3148489781 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7bfc15ae10 con 0x7f7bd406be90 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bd406be90 msgr2=0x7f7bd406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bd406be90 0x7f7bd406e350 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc15ea10 tx=0x7f7be4007930 comp rx=0 tx=0).stop 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 msgr2=0x7f7bfc158ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc158ee0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00d7d0 tx=0x7f7bec00dc90 comp rx=0 tx=0).stop 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 shutdown_connections 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7bd406be90 0x7f7bd406e350 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7bfc144c00 0x7f7bfc13eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7bfc0cdd70 0x7f7bfc13e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 --2- 172.21.15.2:0/3148489781 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7bfc06d000 0x7f7bfc158ee0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 >> 172.21.15.2:0/3148489781 conn(0x7f7bfc0b45f0 msgr2=0x7f7bfc143b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:03.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 shutdown_connections 2024-01-15T18:08:03.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.509+0000 7f7c03c00640 1 -- 172.21.15.2:0/3148489781 wait complete. 2024-01-15T18:08:03.536 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47166 2024-01-15T18:08:03.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.752+0000 7faa5ba00640 1 Processor -- start 2024-01-15T18:08:03.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.752+0000 7faa5ba00640 1 -- start start 2024-01-15T18:08:03.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.753+0000 7faa5ba00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faa540b4c70 0x7faa540d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:03.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.753+0000 7faa5ba00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540d1900 con 0x7faa540b8c30 2024-01-15T18:08:03.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.753+0000 7faa5ba00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540d1be0 con 0x7faa5415d730 2024-01-15T18:08:03.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.753+0000 7faa5ba00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa540d30b0 con 0x7faa540b4c70 2024-01-15T18:08:03.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faa540b4c70 0x7faa540d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:03.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faa540b4c70 0x7faa540d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46690/0 (socket says 172.21.15.2:46690) 2024-01-15T18:08:03.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59000640 1 -- 172.21.15.2:0/2847901713 learned_addr learned my addr 172.21.15.2:0/2847901713 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:03.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59a00640 1 --1- 172.21.15.2:0/2847901713 >> v1:172.21.15.2:6789/0 conn(0x7faa540b8c30 0x7faa540d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:34126/0 (socket says 172.21.15.2:34126) 2024-01-15T18:08:03.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59000640 1 -- 172.21.15.2:0/2847901713 >> v1:172.21.15.137:6789/0 conn(0x7faa5415d730 legacy=0x7faa540b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:03.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59000640 1 -- 172.21.15.2:0/2847901713 >> v1:172.21.15.2:6789/0 conn(0x7faa540b8c30 legacy=0x7faa540d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:03.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.754+0000 7faa59000640 1 -- 172.21.15.2:0/2847901713 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540d1ec0 con 0x7faa540b4c70 2024-01-15T18:08:03.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.755+0000 7faa59000640 1 --2- 172.21.15.2:0/2847901713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa540b4c70 0x7faa540d1270 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009870 tx=0x7faa3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe04854fbd9d083 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:03.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.756+0000 7faa53400640 1 -- 172.21.15.2:0/2847901713 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c030860 con 0x7faa540b4c70 2024-01-15T18:08:03.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.756+0000 7faa53400640 1 -- 172.21.15.2:0/2847901713 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3c0309c0 con 0x7faa540b4c70 2024-01-15T18:08:03.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.756+0000 7faa53400640 1 -- 172.21.15.2:0/2847901713 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa3c030d50 con 0x7faa540b4c70 2024-01-15T18:08:03.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.756+0000 7faa5ba00640 1 -- 172.21.15.2:0/2847901713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa540b4c70 msgr2=0x7faa540d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:03.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.756+0000 7faa5ba00640 1 --2- 172.21.15.2:0/2847901713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa540b4c70 0x7faa540d1270 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7faa3c009870 tx=0x7faa3c02b9a0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.757+0000 7faa5ba00640 1 -- 172.21.15.2:0/2847901713 shutdown_connections 2024-01-15T18:08:03.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.757+0000 7faa5ba00640 1 --2- 172.21.15.2:0/2847901713 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa540b4c70 0x7faa540d1270 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.757+0000 7faa5ba00640 1 -- 172.21.15.2:0/2847901713 >> 172.21.15.2:0/2847901713 conn(0x7faa54072210 msgr2=0x7faa540cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:03.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.757+0000 7faa5ba00640 1 -- 172.21.15.2:0/2847901713 shutdown_connections 2024-01-15T18:08:03.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.757+0000 7faa5ba00640 1 -- 172.21.15.2:0/2847901713 wait complete. 2024-01-15T18:08:03.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.758+0000 7faa5ba00640 1 Processor -- start 2024-01-15T18:08:03.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.759+0000 7faa5ba00640 1 -- start start 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa5ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa540b4c70 0x7faa540c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa5ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa540b8c30 0x7faa5415db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa5ba00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 0x7faa54160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa5ba00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5415ee70 con 0x7faa5415d730 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa5ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5415e260 con 0x7faa540b4c70 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa5ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5415e540 con 0x7faa540b8c30 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa59a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 0x7faa54160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa59a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 0x7faa54160090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46700/0 (socket says 172.21.15.2:46700) 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa59a00640 1 -- 172.21.15.2:0/1987121310 learned_addr learned my addr 172.21.15.2:0/1987121310 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa59000640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa540b4c70 0x7faa540c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:03.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa53e00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa540b8c30 0x7faa5415db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.760+0000 7faa59a00640 1 -- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa540b8c30 msgr2=0x7faa5415db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa59a00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa540b8c30 0x7faa5415db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa59a00640 1 -- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa540b4c70 msgr2=0x7faa540c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa59a00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa540b4c70 0x7faa540c7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:03.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa59a00640 1 -- 172.21.15.2:0/1987121310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0094d0 con 0x7faa5415d730 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa59a00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 0x7faa54160090 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00c400 tx=0x7faa4c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c0074e0 con 0x7faa5415d730 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c007640 con 0x7faa5415d730 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c0049e0 con 0x7faa5415d730 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5415e820 con 0x7faa5415d730 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.761+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa541605d0 con 0x7faa5415d730 2024-01-15T18:08:03.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.763+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5415f020 con 0x7faa5415d730 2024-01-15T18:08:03.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.764+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7faa4c004b40 con 0x7faa5415d730 2024-01-15T18:08:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.765+0000 7faa50c00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa2c06be90 0x7faa2c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.765+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7faa4c088230 con 0x7faa5415d730 2024-01-15T18:08:03.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.765+0000 7faa59000640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa2c06be90 0x7faa2c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:03.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.766+0000 7faa59000640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa2c06be90 0x7faa2c06e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faa3c036000 tx=0x7faa3c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:03.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:03.770+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa5415f020 con 0x7faa5415d730 2024-01-15T18:08:04.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:04.010+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47166", "target": ["mon-mgr", ""]}) v1 -- 0x7faa540c6ae0 con 0x7faa2c06be90 2024-01-15T18:08:08.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.362+0000 7faa50c00640 1 -- 172.21.15.2:0/1987121310 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faa540c6ae0 con 0x7faa2c06be90 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa2c06be90 msgr2=0x7faa2c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa2c06be90 0x7faa2c06e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7faa3c036000 tx=0x7faa3c008030 comp rx=0 tx=0).stop 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 msgr2=0x7faa54160090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 0x7faa54160090 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7faa4c00c400 tx=0x7faa4c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 shutdown_connections 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faa2c06be90 0x7faa2c06e350 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faa5415d730 0x7faa54160090 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faa540b8c30 0x7faa5415db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 --2- 172.21.15.2:0/1987121310 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faa540b4c70 0x7faa540c7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 >> 172.21.15.2:0/1987121310 conn(0x7faa54072210 msgr2=0x7faa5415ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:08.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.366+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 shutdown_connections 2024-01-15T18:08:08.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.367+0000 7faa5ba00640 1 -- 172.21.15.2:0/1987121310 wait complete. 2024-01-15T18:08:08.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47167 2024-01-15T18:08:08.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.617+0000 7f84ab200640 1 Processor -- start 2024-01-15T18:08:08.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.618+0000 7f84ab200640 1 -- start start 2024-01-15T18:08:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.619+0000 7f84ab200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 0x7f84a40acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.619+0000 7f84ab200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a415dbc0 con 0x7f84a40b2430 2024-01-15T18:08:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.619+0000 7f84ab200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a415dea0 con 0x7f84a415ae80 2024-01-15T18:08:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.619+0000 7f84ab200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a415e840 con 0x7f84a406d000 2024-01-15T18:08:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.620+0000 7f84a9200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f84a40b2430 0x7f84a40b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:34134/0 (socket says 172.21.15.2:34134) 2024-01-15T18:08:08.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.620+0000 7f84a9200640 1 -- 172.21.15.2:0/2103839891 learned_addr learned my addr 172.21.15.2:0/2103839891 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:08:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.620+0000 7f849be00640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 0x7f84a40acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.620+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997213604 0 0) 0x7f84a415dbc0 con 0x7f84a40b2430 2024-01-15T18:08:08.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.620+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484003520 con 0x7f84a40b2430 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.620+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1599542526 0 0) 0x7f8484003520 con 0x7f84a40b2430 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 msgr2=0x7f84a40acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 0x7f84a40acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 >> v1:172.21.15.137:6789/0 conn(0x7f84a415ae80 legacy=0x7f84a406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a415e180 con 0x7f84a40b2430 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f849c0019f0 con 0x7f84a40b2430 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 >> v1:172.21.15.2:6789/0 conn(0x7f84a40b2430 legacy=0x7f84a40b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 0x7f84840039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8484004d00 0x7f8484005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a415e840 con 0x7f8484003f70 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848400bac0 con 0x7f8484004d00 2024-01-15T18:08:08.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f848400bc80 con 0x7f84a406d000 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f849be00640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a9200640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8484004d00 0x7f8484005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.621+0000 7f84a3e00640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 0x7f84840039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.622+0000 7f849be00640 1 -- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 msgr2=0x7f84840039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.622+0000 7f849be00640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a406d000 0x7f84840039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.622+0000 7f849be00640 1 -- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8484004d00 msgr2=0x7f8484005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.622+0000 7f849be00640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8484004d00 0x7f8484005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.622+0000 7f849be00640 1 -- 172.21.15.2:0/2103839891 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8484003520 con 0x7f8484003f70 2024-01-15T18:08:08.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.622+0000 7f84a9200640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8484004d00 0x7f8484005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.623+0000 7f849be00640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84840047c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f848c009ac0 tx=0x7f84840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf864bff15b88801 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.623+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c02c600 con 0x7f8484003f70 2024-01-15T18:08:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.623+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c02c760 con 0x7f8484003f70 2024-01-15T18:08:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.623+0000 7f84a3400640 1 -- 172.21.15.2:0/2103839891 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f848c02caf0 con 0x7f8484003f70 2024-01-15T18:08:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84ab200640 1 -- 172.21.15.2:0/2103839891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 msgr2=0x7f84840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84ab200640 1 --2- 172.21.15.2:0/2103839891 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84840047c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f848c009ac0 tx=0x7f84840057f0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84a9200640 1 -- 172.21.15.2:0/2103839891 reap_dead start 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84ab200640 1 -- 172.21.15.2:0/2103839891 shutdown_connections 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84ab200640 1 -- 172.21.15.2:0/2103839891 >> 172.21.15.2:0/2103839891 conn(0x7f84a40b45f0 msgr2=0x7f84a40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84ab200640 1 -- 172.21.15.2:0/2103839891 shutdown_connections 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.624+0000 7f84ab200640 1 -- 172.21.15.2:0/2103839891 wait complete. 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.626+0000 7f84ab200640 1 Processor -- start 2024-01-15T18:08:08.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.626+0000 7f84ab200640 1 -- start start 2024-01-15T18:08:08.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84ab200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84a41558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84ab200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a41f4200 0x7f84a40c49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84ab200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a41f4660 0x7f84a40c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84ab200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4158c30 con 0x7f8484003f70 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84ab200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4158ea0 con 0x7f84a41f4200 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84ab200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4159180 con 0x7f84a41f4660 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84a41558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84a41558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46724/0 (socket says 172.21.15.2:46724) 2024-01-15T18:08:08.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.627+0000 7f84a3e00640 1 -- 172.21.15.2:0/2378003303 learned_addr learned my addr 172.21.15.2:0/2378003303 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:08.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f849be00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a41f4200 0x7f84a40c49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a9200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a41f4660 0x7f84a40c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a3e00640 1 -- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a41f4660 msgr2=0x7f84a40c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a3e00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a41f4660 0x7f84a40c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a3e00640 1 -- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a41f4200 msgr2=0x7f84a40c49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a3e00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a41f4200 0x7f84a40c49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a3e00640 1 -- 172.21.15.2:0/2378003303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4159460 con 0x7f8484003f70 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a9200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a41f4660 0x7f84a40c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f849be00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a41f4200 0x7f84a40c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a3e00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84a41558e0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f849400e550 tx=0x7f849400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84940052e0 con 0x7f8484003f70 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8494005440 con 0x7f8484003f70 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8494003810 con 0x7f8484003f70 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.628+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4159740 con 0x7f8484003f70 2024-01-15T18:08:08.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.629+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4156000 con 0x7f8484003f70 2024-01-15T18:08:08.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.630+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a415d280 con 0x7f8484003f70 2024-01-15T18:08:08.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.631+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8494003970 con 0x7f8484003f70 2024-01-15T18:08:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.632+0000 7f84a0c00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f847c06bf10 0x7f847c06e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:08.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.632+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8494088590 con 0x7f8484003f70 2024-01-15T18:08:08.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.633+0000 7f849be00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f847c06bf10 0x7f847c06e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:08.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.634+0000 7f849be00640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f847c06bf10 0x7f847c06e3d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f848c036000 tx=0x7f848c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:08.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.636+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8494054bf0 con 0x7f8484003f70 2024-01-15T18:08:08.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:08.870+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47167", "target": ["mon-mgr", ""]}) v1 -- 0x7f84a4156900 con 0x7f847c06bf10 2024-01-15T18:08:13.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.717+0000 7f84a0c00640 1 -- 172.21.15.2:0/2378003303 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f84a4156900 con 0x7f847c06bf10 2024-01-15T18:08:13.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f847c06bf10 msgr2=0x7f847c06e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f847c06bf10 0x7f847c06e3d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f848c036000 tx=0x7f848c030000 comp rx=0 tx=0).stop 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 msgr2=0x7f84a41558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84a41558e0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f849400e550 tx=0x7f849400ea10 comp rx=0 tx=0).stop 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 shutdown_connections 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f847c06bf10 0x7f847c06e3d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84a41f4660 0x7f84a40c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84a41f4200 0x7f84a40c49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 --2- 172.21.15.2:0/2378003303 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8484003f70 0x7f84a41558e0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 >> 172.21.15.2:0/2378003303 conn(0x7f84a40b45f0 msgr2=0x7f84a40ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 shutdown_connections 2024-01-15T18:08:13.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.721+0000 7f84ab200640 1 -- 172.21.15.2:0/2378003303 wait complete. 2024-01-15T18:08:13.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47168 2024-01-15T18:08:13.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.969+0000 7f26d5a00640 1 Processor -- start 2024-01-15T18:08:13.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.969+0000 7f26d5a00640 1 -- start start 2024-01-15T18:08:13.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26d5a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f26d00b47d0 0x7f26d00ad280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26d5a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d00b4340 con 0x7f26d00b3190 2024-01-15T18:08:13.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26d5a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26d006f360 con 0x7f26d0157a50 2024-01-15T18:08:13.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26d5a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26d00b3730 con 0x7f26d00b47d0 2024-01-15T18:08:13.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26d5000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f26d00b3190 0x7f26d006a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51912/0 (socket says 172.21.15.2:51912) 2024-01-15T18:08:13.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26d5000640 1 -- 172.21.15.2:0/334658910 learned_addr learned my addr 172.21.15.2:0/334658910 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:08:13.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.971+0000 7f26cf400640 1 --2- 172.21.15.2:0/334658910 >> v2:172.21.15.137:3300/0 conn(0x7f26d00b47d0 0x7f26d00ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1106910298 0 0) 0x7f26d00b4340 con 0x7f26d00b3190 2024-01-15T18:08:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b0003520 con 0x7f26d00b3190 2024-01-15T18:08:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4078106541 0 0) 0x7f26b0003520 con 0x7f26d00b3190 2024-01-15T18:08:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 >> v2:172.21.15.137:3300/0 conn(0x7f26d00b47d0 msgr2=0x7f26d00ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 --2- 172.21.15.2:0/334658910 >> v2:172.21.15.137:3300/0 conn(0x7f26d00b47d0 0x7f26d00ad280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 >> v1:172.21.15.137:6789/0 conn(0x7f26d0157a50 legacy=0x7f26d015e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d00b3a10 con 0x7f26d00b3190 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f26c80019f0 con 0x7f26d00b3190 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 >> v1:172.21.15.2:6789/0 conn(0x7f26d00b3190 legacy=0x7f26d006a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b00047c0 0x7f26b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0005180 0x7f26b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d00b3730 con 0x7f26b0003970 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26b000c020 con 0x7f26b0005180 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.972+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26b000c1e0 con 0x7f26b00047c0 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cfe00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cf400640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b00047c0 0x7f26b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26d5000640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0005180 0x7f26b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cfe00640 1 -- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b00047c0 msgr2=0x7f26b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cfe00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b00047c0 0x7f26b0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cfe00640 1 -- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0005180 msgr2=0x7f26b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cfe00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0005180 0x7f26b000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cfe00640 1 -- 172.21.15.2:0/334658910 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b0003520 con 0x7f26b0003970 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26d5000640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26b0005180 0x7f26b000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26d5000640 1 -- 172.21.15.2:0/334658910 reap_dead start 2024-01-15T18:08:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.973+0000 7f26cf400640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26b00047c0 0x7f26b0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.974+0000 7f26cfe00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26b0003d90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f26c0001750 tx=0x7f26b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fda2fa17b732429 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c00052d0 con 0x7f26b0003970 2024-01-15T18:08:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c002e050 con 0x7f26b0003970 2024-01-15T18:08:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26cea00640 1 -- 172.21.15.2:0/334658910 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0003830 con 0x7f26b0003970 2024-01-15T18:08:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 -- 172.21.15.2:0/334658910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 msgr2=0x7f26b0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26b0003d90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f26c0001750 tx=0x7f26b00057f0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 -- 172.21.15.2:0/334658910 shutdown_connections 2024-01-15T18:08:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 --2- 172.21.15.2:0/334658910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26b0003d90 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 -- 172.21.15.2:0/334658910 >> 172.21.15.2:0/334658910 conn(0x7f26d00cf700 msgr2=0x7f26d00cfb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 -- 172.21.15.2:0/334658910 shutdown_connections 2024-01-15T18:08:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.975+0000 7f26d5a00640 1 -- 172.21.15.2:0/334658910 wait complete. 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.977+0000 7f26d5a00640 1 Processor -- start 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.977+0000 7f26d5a00640 1 -- start start 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.978+0000 7f26d5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26d0154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.978+0000 7f26d5a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26d0141ae0 0x7f26d015c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.978+0000 7f26d5a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26d015c940 0x7f26d01531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.978+0000 7f26d5a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01f3b10 con 0x7f26b0003970 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.978+0000 7f26d5a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d015cdf0 con 0x7f26d015c940 2024-01-15T18:08:13.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.978+0000 7f26d5a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26d01f2bf0 con 0x7f26d0141ae0 2024-01-15T18:08:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26d0154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26d0154e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54456/0 (socket says 172.21.15.2:54456) 2024-01-15T18:08:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 -- 172.21.15.2:0/4208929248 learned_addr learned my addr 172.21.15.2:0/4208929248 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 -- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26d0141ae0 msgr2=0x7f26d015c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26d0141ae0 0x7f26d015c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 -- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26d015c940 msgr2=0x7f26d01531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26d015c940 0x7f26d01531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 -- 172.21.15.2:0/4208929248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d01553d0 con 0x7f26b0003970 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.979+0000 7f26cfe00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26d0154e90 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f26c0005040 tx=0x7f26c0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.980+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c00076c0 con 0x7f26b0003970 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.980+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c0007a30 con 0x7f26b0003970 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.980+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c0006c20 con 0x7f26b0003970 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.980+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d01f2e90 con 0x7f26b0003970 2024-01-15T18:08:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.980+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d01f3450 con 0x7f26b0003970 2024-01-15T18:08:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.981+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d01598e0 con 0x7f26b0003970 2024-01-15T18:08:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.982+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f26c00035f0 con 0x7f26b0003970 2024-01-15T18:08:13.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.983+0000 7f26c7e00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26a806be90 0x7f26a806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:13.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.983+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f26c00aa420 con 0x7f26b0003970 2024-01-15T18:08:13.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.984+0000 7f26cf400640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26a806be90 0x7f26a806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:13.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.985+0000 7f26cf400640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26a806be90 0x7f26a806e350 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f26d014fbf0 tx=0x7f26b8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:13.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:13.988+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f26c00dfe90 con 0x7f26b0003970 2024-01-15T18:08:14.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:14.220+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47168", "target": ["mon-mgr", ""]}) v1 -- 0x7f26d00b3fd0 con 0x7f26a806be90 2024-01-15T18:08:18.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.647+0000 7f26c7e00640 1 -- 172.21.15.2:0/4208929248 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f26b8006bd0 con 0x7f26a806be90 2024-01-15T18:08:18.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.650+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26a806be90 msgr2=0x7f26a806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.650+0000 7f26d5a00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26a806be90 0x7f26a806e350 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f26d014fbf0 tx=0x7f26b8008040 comp rx=0 tx=0).stop 2024-01-15T18:08:18.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.650+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 msgr2=0x7f26d0154e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.650+0000 7f26d5a00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26d0154e90 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f26c0005040 tx=0x7f26c0001120 comp rx=0 tx=0).stop 2024-01-15T18:08:18.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.650+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 shutdown_connections 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f26a806be90 0x7f26a806e350 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f26d015c940 0x7f26d01531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f26d0141ae0 0x7f26d015c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 --2- 172.21.15.2:0/4208929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26b0003970 0x7f26d0154e90 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 >> 172.21.15.2:0/4208929248 conn(0x7f26d00cf700 msgr2=0x7f26d006afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 shutdown_connections 2024-01-15T18:08:18.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.651+0000 7f26d5a00640 1 -- 172.21.15.2:0/4208929248 wait complete. 2024-01-15T18:08:18.676 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone47169 2024-01-15T18:08:18.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.897+0000 7fa780a00640 1 Processor -- start 2024-01-15T18:08:18.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.897+0000 7fa780a00640 1 -- start start 2024-01-15T18:08:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.899+0000 7fa780a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa77c161890 0x7fa77c160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.899+0000 7fa780a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa77c1611a0 0x7fa77c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.899+0000 7fa780a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa77c1615f0 con 0x7fa77c15d270 2024-01-15T18:08:18.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.899+0000 7fa780a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c06c5e0 con 0x7fa77c1611a0 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.899+0000 7fa780a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa77c06c8c0 con 0x7fa77c161890 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.899+0000 7fa77b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa77c1611a0 0x7fa77c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa77c1611a0 0x7fa77c06cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54468/0 (socket says 172.21.15.2:54468) 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 -- 172.21.15.2:0/4266929248 learned_addr learned my addr 172.21.15.2:0/4266929248 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa773e00640 1 --2- 172.21.15.2:0/4266929248 >> v2:172.21.15.137:3300/0 conn(0x7fa77c161890 0x7fa77c160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 -- 172.21.15.2:0/4266929248 >> v2:172.21.15.137:3300/0 conn(0x7fa77c161890 msgr2=0x7fa77c160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 --2- 172.21.15.2:0/4266929248 >> v2:172.21.15.137:3300/0 conn(0x7fa77c161890 0x7fa77c160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 -- 172.21.15.2:0/4266929248 >> v1:172.21.15.137:6789/0 conn(0x7fa77c15d270 legacy=0x7fa77c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 -- 172.21.15.2:0/4266929248 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa77c071c00 con 0x7fa77c1611a0 2024-01-15T18:08:18.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa77b200640 1 --2- 172.21.15.2:0/4266929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c1611a0 0x7fa77c06cd50 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0ceba0 tx=0x7fa77402bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c0548d74de9a6e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:18.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.901+0000 7fa779e00640 1 -- 172.21.15.2:0/4266929248 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77402c960 con 0x7fa77c1611a0 2024-01-15T18:08:18.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa779e00640 1 -- 172.21.15.2:0/4266929248 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77402cac0 con 0x7fa77c1611a0 2024-01-15T18:08:18.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa779e00640 1 -- 172.21.15.2:0/4266929248 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77402ce50 con 0x7fa77c1611a0 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 -- 172.21.15.2:0/4266929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c1611a0 msgr2=0x7fa77c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 --2- 172.21.15.2:0/4266929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c1611a0 0x7fa77c06cd50 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fa77c0ceba0 tx=0x7fa77402bdc0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 -- 172.21.15.2:0/4266929248 shutdown_connections 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 --2- 172.21.15.2:0/4266929248 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c1611a0 0x7fa77c06cd50 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 --2- 172.21.15.2:0/4266929248 >> v2:172.21.15.137:3300/0 conn(0x7fa77c161890 0x7fa77c160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 -- 172.21.15.2:0/4266929248 >> 172.21.15.2:0/4266929248 conn(0x7fa77c0b0840 msgr2=0x7fa77c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.902+0000 7fa780a00640 1 -- 172.21.15.2:0/4266929248 shutdown_connections 2024-01-15T18:08:18.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.904+0000 7fa780a00640 1 -- 172.21.15.2:0/4266929248 wait complete. 2024-01-15T18:08:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.904+0000 7fa780a00640 1 Processor -- start 2024-01-15T18:08:18.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.904+0000 7fa780a00640 1 -- start start 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.905+0000 7fa780a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa77c15d270 0x7fa77c1f3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.905+0000 7fa780a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa77c1611a0 0x7fa77c0aec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.905+0000 7fa780a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 0x7fa77c1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.905+0000 7fa780a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c1560f0 con 0x7fa77c161890 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.905+0000 7fa780a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c156300 con 0x7fa77c15d270 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.905+0000 7fa780a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c1565e0 con 0x7fa77c1611a0 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 0x7fa77c1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 0x7fa77c1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54476/0 (socket says 172.21.15.2:54476) 2024-01-15T18:08:18.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 -- 172.21.15.2:0/4094805275 learned_addr learned my addr 172.21.15.2:0/4094805275 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa773e00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa77c1611a0 0x7fa77c0aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 -- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa77c1611a0 msgr2=0x7fa77c0aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77a800640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa77c15d270 0x7fa77c1f3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa77c1611a0 0x7fa77c0aec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 -- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa77c15d270 msgr2=0x7fa77c1f3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa77c15d270 0x7fa77c1f3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 -- 172.21.15.2:0/4094805275 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa774009b30 con 0x7fa77c161890 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77a800640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa77c15d270 0x7fa77c1f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.906+0000 7fa77b200640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 0x7fa77c1573a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa77402c300 tx=0x7fa774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:18.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.907+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa77402ce30 con 0x7fa77c161890 2024-01-15T18:08:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.907+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa774040470 con 0x7fa77c161890 2024-01-15T18:08:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.907+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa774040800 con 0x7fa77c161890 2024-01-15T18:08:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.907+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c1568c0 con 0x7fa77c161890 2024-01-15T18:08:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.907+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c156e80 con 0x7fa77c161890 2024-01-15T18:08:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.908+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c157070 con 0x7fa77c161890 2024-01-15T18:08:18.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.909+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa774040960 con 0x7fa77c161890 2024-01-15T18:08:18.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.911+0000 7fa772a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa75406be90 0x7fa75406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:18.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.911+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fa7740ab200 con 0x7fa77c161890 2024-01-15T18:08:18.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.912+0000 7fa77a800640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa75406be90 0x7fa75406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:18.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.913+0000 7fa77a800640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa75406be90 0x7fa75406e350 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa76c003eb0 tx=0x7fa76c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:18.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:18.915+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa774077ac0 con 0x7fa77c161890 2024-01-15T18:08:19.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:19.147+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone47169", "target": ["mon-mgr", ""]}) v1 -- 0x7fa77c0b2d30 con 0x7fa75406be90 2024-01-15T18:08:28.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.808+0000 7fa772a00640 1 -- 172.21.15.2:0/4094805275 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa76c005030 con 0x7fa75406be90 2024-01-15T18:08:28.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.811+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa75406be90 msgr2=0x7fa75406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.811+0000 7fa780a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa75406be90 0x7fa75406e350 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fa76c003eb0 tx=0x7fa76c009040 comp rx=0 tx=0).stop 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.811+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 msgr2=0x7fa77c1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.811+0000 7fa780a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 0x7fa77c1573a0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fa77402c300 tx=0x7fa774004a40 comp rx=0 tx=0).stop 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 shutdown_connections 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa75406be90 0x7fa75406e350 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa77c161890 0x7fa77c1573a0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa77c1611a0 0x7fa77c0aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 --2- 172.21.15.2:0/4094805275 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa77c15d270 0x7fa77c1f3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 >> 172.21.15.2:0/4094805275 conn(0x7fa77c0b0840 msgr2=0x7fa77c06d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 shutdown_connections 2024-01-15T18:08:28.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:28.812+0000 7fa780a00640 1 -- 172.21.15.2:0/4094805275 wait complete. 2024-01-15T18:08:28.837 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471610 2024-01-15T18:08:29.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.050+0000 7f5082a00640 1 Processor -- start 2024-01-15T18:08:29.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.050+0000 7f5082a00640 1 -- start start 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5082a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c0b0b80 0x7f507c0b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5082a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f507c070560 0x7f507c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5082a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c071250 con 0x7f507c150890 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5082a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0b1540 con 0x7f507c070560 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5082a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c06d000 con 0x7f507c0b0b80 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5080a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f507c070560 0x7f507c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5080a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f507c070560 0x7f507c0709d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36738/0 (socket says 172.21.15.2:36738) 2024-01-15T18:08:29.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.052+0000 7f5080a00640 1 -- 172.21.15.2:0/3386869687 learned_addr learned my addr 172.21.15.2:0/3386869687 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:29.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.053+0000 7f5073200640 1 --2- 172.21.15.2:0/3386869687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c0b0b80 0x7f507c0b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:29.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.053+0000 7f5080a00640 1 -- 172.21.15.2:0/3386869687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c0b0b80 msgr2=0x7f507c0b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:29.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.053+0000 7f5080a00640 1 --2- 172.21.15.2:0/3386869687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c0b0b80 0x7f507c0b1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.053+0000 7f5080a00640 1 -- 172.21.15.2:0/3386869687 >> v1:172.21.15.137:6789/0 conn(0x7f507c150890 legacy=0x7f507c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:29.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.053+0000 7f5080a00640 1 -- 172.21.15.2:0/3386869687 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c071530 con 0x7f507c070560 2024-01-15T18:08:29.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.054+0000 7f5080a00640 1 --2- 172.21.15.2:0/3386869687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c070560 0x7f507c0709d0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5074009870 tx=0x7f507402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd25928b66ed7ae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:29.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f507b400640 1 -- 172.21.15.2:0/3386869687 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074031860 con 0x7f507c070560 2024-01-15T18:08:29.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f507b400640 1 -- 172.21.15.2:0/3386869687 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50740319c0 con 0x7f507c070560 2024-01-15T18:08:29.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f507b400640 1 -- 172.21.15.2:0/3386869687 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5074031d50 con 0x7f507c070560 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 -- 172.21.15.2:0/3386869687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c070560 msgr2=0x7f507c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 --2- 172.21.15.2:0/3386869687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c070560 0x7f507c0709d0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5074009870 tx=0x7f507402b900 comp rx=0 tx=0).stop 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 -- 172.21.15.2:0/3386869687 shutdown_connections 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 --2- 172.21.15.2:0/3386869687 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c070560 0x7f507c0709d0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 --2- 172.21.15.2:0/3386869687 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c0b0b80 0x7f507c0b1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 -- 172.21.15.2:0/3386869687 >> 172.21.15.2:0/3386869687 conn(0x7f507c0b45f0 msgr2=0x7f507c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.055+0000 7f5082a00640 1 -- 172.21.15.2:0/3386869687 shutdown_connections 2024-01-15T18:08:29.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.056+0000 7f5082a00640 1 -- 172.21.15.2:0/3386869687 wait complete. 2024-01-15T18:08:29.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.058+0000 7f5082a00640 1 Processor -- start 2024-01-15T18:08:29.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.058+0000 7f5082a00640 1 -- start start 2024-01-15T18:08:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5082a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c070560 0x7f507c14e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5082a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 0x7f507c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5082a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507c150890 0x7f507c158de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5082a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c0a8ce0 con 0x7f507c0b0b80 2024-01-15T18:08:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5082a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c152a60 con 0x7f507c150890 2024-01-15T18:08:29.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5082a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c153470 con 0x7f507c070560 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5073200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 0x7f507c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5073200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 0x7f507c1588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36744/0 (socket says 172.21.15.2:36744) 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f5073200640 1 -- 172.21.15.2:0/119840439 learned_addr learned my addr 172.21.15.2:0/119840439 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.059+0000 7f507be00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c070560 0x7f507c14e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5080a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507c150890 0x7f507c158de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5073200640 1 -- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c070560 msgr2=0x7f507c14e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5073200640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c070560 0x7f507c14e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5073200640 1 -- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507c150890 msgr2=0x7f507c158de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5073200640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507c150890 0x7f507c158de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5073200640 1 -- 172.21.15.2:0/119840439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740094d0 con 0x7f507c0b0b80 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f507be00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c070560 0x7f507c14e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:29.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5073200640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 0x7f507c1588a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f506400d600 tx=0x7f506400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:29.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640125f0 con 0x7f507c0b0b80 2024-01-15T18:08:29.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5064012770 con 0x7f507c0b0b80 2024-01-15T18:08:29.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064012b20 con 0x7f507c0b0b80 2024-01-15T18:08:29.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c0a8460 con 0x7f507c0b0b80 2024-01-15T18:08:29.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.060+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c0a8a00 con 0x7f507c0b0b80 2024-01-15T18:08:29.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.062+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c0686f0 con 0x7f507c0b0b80 2024-01-15T18:08:29.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.063+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5064012c80 con 0x7f507c0b0b80 2024-01-15T18:08:29.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.063+0000 7f5078c00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406bee0 0x7f505406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:29.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.064+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f5064088d20 con 0x7f507c0b0b80 2024-01-15T18:08:29.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.064+0000 7f507be00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406bee0 0x7f505406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:29.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.065+0000 7f507be00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406bee0 0x7f505406e3a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f506c0012e0 tx=0x7f506c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.069+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5064055380 con 0x7f507c0b0b80 2024-01-15T18:08:29.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.303+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471610", "target": ["mon-mgr", ""]}) v1 -- 0x7f507c071810 con 0x7f505406bee0 2024-01-15T18:08:29.821 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:08:29.818+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:29.821 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:08:29.818+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:29.821 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:08:29.818+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:29.822 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:08:29.818+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:29.822 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:08:29.818+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:08:29.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.912+0000 7f5078c00640 1 -- 172.21.15.2:0/119840439 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f506c001a70 con 0x7f505406bee0 2024-01-15T18:08:29.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.915+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406bee0 msgr2=0x7f505406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:29.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.915+0000 7f5082a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406bee0 0x7f505406e3a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f506c0012e0 tx=0x7f506c009040 comp rx=0 tx=0).stop 2024-01-15T18:08:29.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.915+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 msgr2=0x7f507c1588a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:29.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.915+0000 7f5082a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 0x7f507c1588a0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f506400d600 tx=0x7f506400dac0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 shutdown_connections 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f505406bee0 0x7f505406e3a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507c150890 0x7f507c158de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507c0b0b80 0x7f507c1588a0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 --2- 172.21.15.2:0/119840439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507c070560 0x7f507c14e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 >> 172.21.15.2:0/119840439 conn(0x7f507c0b45f0 msgr2=0x7f507c0b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 shutdown_connections 2024-01-15T18:08:29.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:29.916+0000 7f5082a00640 1 -- 172.21.15.2:0/119840439 wait complete. 2024-01-15T18:08:29.941 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471611 2024-01-15T18:08:30.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.155+0000 7f9210400640 1 Processor -- start 2024-01-15T18:08:30.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.155+0000 7f9210400640 1 -- start start 2024-01-15T18:08:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.156+0000 7f9210400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9208144c00 0x7f92080cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f9210400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f92080cdd70 0x7f920806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f9210400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f920806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f9210400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92080ce230 con 0x7f92080cdd70 2024-01-15T18:08:30.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f9210400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92080ce510 con 0x7f9208144c00 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f9210400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92080ce7f0 con 0x7f920815c450 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f92080cdd70 0x7f920806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f92080cdd70 0x7f920806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36750/0 (socket says 172.21.15.2:36750) 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 -- 172.21.15.2:0/2939394276 learned_addr learned my addr 172.21.15.2:0/2939394276 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920e400640 1 --2- 172.21.15.2:0/2939394276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f920806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 -- 172.21.15.2:0/2939394276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 msgr2=0x7f920806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 --2- 172.21.15.2:0/2939394276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f920806ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 -- 172.21.15.2:0/2939394276 >> v2:172.21.15.137:3300/0 conn(0x7f9208144c00 msgr2=0x7f92080cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 --2- 172.21.15.2:0/2939394276 >> v2:172.21.15.137:3300/0 conn(0x7f9208144c00 0x7f92080cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.157+0000 7f920d000640 1 -- 172.21.15.2:0/2939394276 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92080705f0 con 0x7f92080cdd70 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.159+0000 7f920d000640 1 --2- 172.21.15.2:0/2939394276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f920806c8f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f92080b1c60 tx=0x7f91f002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd82c19c70e28071 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.159+0000 7f9207e00640 1 -- 172.21.15.2:0/2939394276 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0031860 con 0x7f92080cdd70 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.159+0000 7f9207e00640 1 -- 172.21.15.2:0/2939394276 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f00319c0 con 0x7f92080cdd70 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.159+0000 7f9207e00640 1 -- 172.21.15.2:0/2939394276 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0031d50 con 0x7f92080cdd70 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.159+0000 7f9210400640 1 -- 172.21.15.2:0/2939394276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 msgr2=0x7f920806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.159+0000 7f9210400640 1 --2- 172.21.15.2:0/2939394276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f920806c8f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f92080b1c60 tx=0x7f91f002b900 comp rx=0 tx=0).stop 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 -- 172.21.15.2:0/2939394276 shutdown_connections 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 --2- 172.21.15.2:0/2939394276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f920806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 --2- 172.21.15.2:0/2939394276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f920806c8f0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 --2- 172.21.15.2:0/2939394276 >> v2:172.21.15.137:3300/0 conn(0x7f9208144c00 0x7f92080cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 -- 172.21.15.2:0/2939394276 >> 172.21.15.2:0/2939394276 conn(0x7f92080b45f0 msgr2=0x7f92080b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 -- 172.21.15.2:0/2939394276 shutdown_connections 2024-01-15T18:08:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.160+0000 7f9210400640 1 -- 172.21.15.2:0/2939394276 wait complete. 2024-01-15T18:08:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.161+0000 7f9210400640 1 Processor -- start 2024-01-15T18:08:30.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.162+0000 7f9210400640 1 -- start start 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f9210400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f9208158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f9210400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208144c00 0x7f92081f1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f9210400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f92081f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f9210400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92081f4bc0 con 0x7f92080cdd70 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f9210400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92081f0630 con 0x7f9208144c00 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f9210400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92081f4600 con 0x7f920815c450 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f9208158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f9208158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36762/0 (socket says 172.21.15.2:36762) 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 -- 172.21.15.2:0/3668762052 learned_addr learned my addr 172.21.15.2:0/3668762052 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:30.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920e400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f92081f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920d000640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208144c00 0x7f92081f1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 -- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 msgr2=0x7f92081f3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f92081f3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 -- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208144c00 msgr2=0x7f92081f1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.163+0000 7f920da00640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208144c00 0x7f92081f1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f920da00640 1 -- 172.21.15.2:0/3668762052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f00094d0 con 0x7f92080cdd70 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f920da00640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f9208158250 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f91f800b4a0 tx=0x7f91f800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8010610 con 0x7f92080cdd70 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f8010770 con 0x7f92080cdd70 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f8010b20 con 0x7f92080cdd70 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92081f48e0 con 0x7f92080cdd70 2024-01-15T18:08:30.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.164+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92081f0b20 con 0x7f92080cdd70 2024-01-15T18:08:30.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.165+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92081f0d10 con 0x7f92080cdd70 2024-01-15T18:08:30.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.166+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f91f8010c80 con 0x7f92080cdd70 2024-01-15T18:08:30.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.167+0000 7f9205600640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91e006bee0 0x7f91e006e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:30.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.167+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f91f8088420 con 0x7f92080cdd70 2024-01-15T18:08:30.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.168+0000 7f920d000640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91e006bee0 0x7f91e006e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:30.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.169+0000 7f920d000640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91e006bee0 0x7f91e006e3a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f91f0036000 tx=0x7f91f003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:30.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.172+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91f8054a80 con 0x7f92080cdd70 2024-01-15T18:08:30.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:30.403+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471611", "target": ["mon-mgr", ""]}) v1 -- 0x7f92080686f0 con 0x7f91e006bee0 2024-01-15T18:08:31.727 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:31.735 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:08:31.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:31.771+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f91f8054e50 con 0x7f92080cdd70 2024-01-15T18:08:35.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.150+0000 7f9205600640 1 -- 172.21.15.2:0/3668762052 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f92080686f0 con 0x7f91e006bee0 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91e006bee0 msgr2=0x7f91e006e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91e006bee0 0x7f91e006e3a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f91f0036000 tx=0x7f91f003c040 comp rx=0 tx=0).stop 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 msgr2=0x7f9208158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f9208158250 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f91f800b4a0 tx=0x7f91f800b960 comp rx=0 tx=0).stop 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 shutdown_connections 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91e006bee0 0x7f91e006e3a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f920815c450 0x7f92081f3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9208144c00 0x7f92081f1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 --2- 172.21.15.2:0/3668762052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92080cdd70 0x7f9208158250 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 >> 172.21.15.2:0/3668762052 conn(0x7f92080b45f0 msgr2=0x7f9208070e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:35.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.154+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 shutdown_connections 2024-01-15T18:08:35.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.155+0000 7f9210400640 1 -- 172.21.15.2:0/3668762052 wait complete. 2024-01-15T18:08:35.183 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471612 2024-01-15T18:08:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.403+0000 7f4f33200640 1 Processor -- start 2024-01-15T18:08:35.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.403+0000 7f4f33200640 1 -- start start 2024-01-15T18:08:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.405+0000 7f4f33200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4f2c15f260 0x7f4f2c06c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.405+0000 7f4f33200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c071090 con 0x7f4f2c0720d0 2024-01-15T18:08:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.405+0000 7f4f33200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c144560 con 0x7f4f2c143d20 2024-01-15T18:08:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.405+0000 7f4f33200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c071430 con 0x7f4f2c15f260 2024-01-15T18:08:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.405+0000 7f4f31400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4f2c15f260 0x7f4f2c06c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.405+0000 7f4f31400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4f2c15f260 0x7f4f2c06c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37042/0 (socket says 172.21.15.2:37042) 2024-01-15T18:08:35.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.406+0000 7f4f31400640 1 -- 172.21.15.2:0/1267343821 learned_addr learned my addr 172.21.15.2:0/1267343821 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.406+0000 7f4f31400640 1 -- 172.21.15.2:0/1267343821 >> v1:172.21.15.137:6789/0 conn(0x7f4f2c143d20 legacy=0x7f4f2c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.406+0000 7f4f31400640 1 -- 172.21.15.2:0/1267343821 >> v1:172.21.15.2:6789/0 conn(0x7f4f2c0720d0 legacy=0x7f4f2c06c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.406+0000 7f4f31400640 1 -- 172.21.15.2:0/1267343821 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c144280 con 0x7f4f2c15f260 2024-01-15T18:08:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.406+0000 7f4f30a00640 1 -- 172.21.15.2:0/1267343821 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729194590 0 0) 0x7f4f2c071090 con 0x7f4f2c0720d0 2024-01-15T18:08:35.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.407+0000 7f4f31400640 1 --2- 172.21.15.2:0/1267343821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c15f260 0x7f4f2c06c150 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009870 tx=0x7f4f1402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6464beab92b28f4c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.407+0000 7f4f30a00640 1 -- 172.21.15.2:0/1267343821 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14030850 con 0x7f4f2c15f260 2024-01-15T18:08:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f30a00640 1 -- 172.21.15.2:0/1267343821 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f140309b0 con 0x7f4f2c15f260 2024-01-15T18:08:35.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f30a00640 1 -- 172.21.15.2:0/1267343821 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f14030d40 con 0x7f4f2c15f260 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 -- 172.21.15.2:0/1267343821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c15f260 msgr2=0x7f4f2c06c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 --2- 172.21.15.2:0/1267343821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c15f260 0x7f4f2c06c150 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009870 tx=0x7f4f1402b990 comp rx=0 tx=0).stop 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 -- 172.21.15.2:0/1267343821 shutdown_connections 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 --2- 172.21.15.2:0/1267343821 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c15f260 0x7f4f2c06c150 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 -- 172.21.15.2:0/1267343821 >> 172.21.15.2:0/1267343821 conn(0x7f4f2c0b0840 msgr2=0x7f4f2c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 -- 172.21.15.2:0/1267343821 shutdown_connections 2024-01-15T18:08:35.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.408+0000 7f4f33200640 1 -- 172.21.15.2:0/1267343821 wait complete. 2024-01-15T18:08:35.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.410+0000 7f4f33200640 1 Processor -- start 2024-01-15T18:08:35.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.410+0000 7f4f33200640 1 -- start start 2024-01-15T18:08:35.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.411+0000 7f4f33200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 0x7f4f2c13dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.411+0000 7f4f33200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f2c143d20 0x7f4f2c13fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.411+0000 7f4f33200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f2c15f260 0x7f4f2c13d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.411+0000 7f4f33200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c13eef0 con 0x7f4f2c0720d0 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.411+0000 7f4f33200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c13e370 con 0x7f4f2c15f260 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.411+0000 7f4f33200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c13e650 con 0x7f4f2c143d20 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 0x7f4f2c13dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 0x7f4f2c13dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37054/0 (socket says 172.21.15.2:37054) 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 -- 172.21.15.2:0/3623618538 learned_addr learned my addr 172.21.15.2:0/3623618538 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31400640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f2c143d20 0x7f4f2c13fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 -- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f2c143d20 msgr2=0x7f4f2c13fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f2c143d20 0x7f4f2c13fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 -- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f2c15f260 msgr2=0x7f4f2c13d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f32800640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f2c15f260 0x7f4f2c13d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f2c15f260 0x7f4f2c13d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 -- 172.21.15.2:0/3623618538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f140094d0 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.412+0000 7f4f31e00640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 0x7f4f2c13dd90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00c450 tx=0x7f4f1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.413+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c0052d0 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.413+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c005430 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.413+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c012870 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.413+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c13e930 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.413+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c1403e0 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.414+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c074770 con 0x7f4f2c0720d0 2024-01-15T18:08:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.415+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f4f1c0129d0 con 0x7f4f2c0720d0 2024-01-15T18:08:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.416+0000 7f4f2a000640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4f0406bfa0 0x7f4f0406e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.416+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4f1c088120 con 0x7f4f2c0720d0 2024-01-15T18:08:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.417+0000 7f4f31400640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4f0406bfa0 0x7f4f0406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.418+0000 7f4f31400640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4f0406bfa0 0x7f4f0406e460 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4f14036000 tx=0x7f4f1403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.421+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f1c0549e0 con 0x7f4f2c0720d0 2024-01-15T18:08:35.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:35.652+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471612", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f2c071850 con 0x7f4f0406bfa0 2024-01-15T18:08:40.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.195+0000 7f4f2a000640 1 -- 172.21.15.2:0/3623618538 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4f14030ba0 con 0x7f4f0406bfa0 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4f0406bfa0 msgr2=0x7f4f0406e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4f0406bfa0 0x7f4f0406e460 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f4f14036000 tx=0x7f4f1403b040 comp rx=0 tx=0).stop 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 msgr2=0x7f4f2c13dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 0x7f4f2c13dd90 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00c450 tx=0x7f4f1c00c910 comp rx=0 tx=0).stop 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 shutdown_connections 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4f0406bfa0 0x7f4f0406e460 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f2c15f260 0x7f4f2c13d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f2c143d20 0x7f4f2c13fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 --2- 172.21.15.2:0/3623618538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f2c0720d0 0x7f4f2c13dd90 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 >> 172.21.15.2:0/3623618538 conn(0x7f4f2c0b0840 msgr2=0x7f4f2c0afe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 shutdown_connections 2024-01-15T18:08:40.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.198+0000 7f4f33200640 1 -- 172.21.15.2:0/3623618538 wait complete. 2024-01-15T18:08:40.225 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471613 2024-01-15T18:08:40.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.432+0000 7fef5ec00640 1 Processor -- start 2024-01-15T18:08:40.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.432+0000 7fef5ec00640 1 -- start start 2024-01-15T18:08:40.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef580716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5ec00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fef580cc3b0 0x7fef580cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5ec00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef580b2fa0 con 0x7fef58143620 2024-01-15T18:08:40.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5ec00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef58071bf0 con 0x7fef580cc3b0 2024-01-15T18:08:40.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef580b3280 con 0x7fef58071230 2024-01-15T18:08:40.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5d800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fef58143620 0x7fef580b2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35684/0 (socket says 172.21.15.2:35684) 2024-01-15T18:08:40.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5d800640 1 -- 172.21.15.2:0/279141793 learned_addr learned my addr 172.21.15.2:0/279141793 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.434+0000 7fef5ce00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef580716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204635222 0 0) 0x7fef580b2fa0 con 0x7fef58143620 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef5e200640 1 --2- 172.21.15.2:0/279141793 >> v2:172.21.15.137:3300/0 conn(0x7fef580cc3b0 0x7fef580cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38003520 con 0x7fef58143620 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 423617077 0 0) 0x7fef38003520 con 0x7fef58143620 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 msgr2=0x7fef580716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef580716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 >> v2:172.21.15.137:3300/0 conn(0x7fef580cc3b0 msgr2=0x7fef580cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 --2- 172.21.15.2:0/279141793 >> v2:172.21.15.137:3300/0 conn(0x7fef580cc3b0 0x7fef580cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef580b3560 con 0x7fef58143620 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fef40001670 con 0x7fef58143620 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 >> v1:172.21.15.2:6789/0 conn(0x7fef58143620 legacy=0x7fef580b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef38003970 0x7fef38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58071bf0 con 0x7fef380047c0 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef580b3280 con 0x7fef38003970 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.435+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3800b980 con 0x7fef58071230 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5ce00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5d800640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef38003970 0x7fef38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5ce00640 1 -- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 msgr2=0x7fef38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5e200640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5ce00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5ce00640 1 -- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef38003970 msgr2=0x7fef38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:40.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5ce00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef38003970 0x7fef38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5ce00640 1 -- 172.21.15.2:0/279141793 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef38003520 con 0x7fef380047c0 2024-01-15T18:08:40.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.436+0000 7fef5e200640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58071230 0x7fef38005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.437+0000 7fef5ce00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef38004c40 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fef48009ac0 tx=0x7fef380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c7d5a1ae1715d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.438+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4802c6b0 con 0x7fef380047c0 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.438+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4802c810 con 0x7fef380047c0 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.438+0000 7fef57e00640 1 -- 172.21.15.2:0/279141793 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4802cba0 con 0x7fef380047c0 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.438+0000 7fef5ec00640 1 -- 172.21.15.2:0/279141793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 msgr2=0x7fef38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.438+0000 7fef5ec00640 1 --2- 172.21.15.2:0/279141793 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef38004c40 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fef48009ac0 tx=0x7fef380057f0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.438+0000 7fef5e200640 1 -- 172.21.15.2:0/279141793 reap_dead start 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.439+0000 7fef5ec00640 1 -- 172.21.15.2:0/279141793 shutdown_connections 2024-01-15T18:08:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.439+0000 7fef5ec00640 1 -- 172.21.15.2:0/279141793 >> 172.21.15.2:0/279141793 conn(0x7fef580b0840 msgr2=0x7fef580b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:40.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.439+0000 7fef5ec00640 1 -- 172.21.15.2:0/279141793 shutdown_connections 2024-01-15T18:08:40.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.439+0000 7fef5ec00640 1 -- 172.21.15.2:0/279141793 wait complete. 2024-01-15T18:08:40.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.440+0000 7fef5ec00640 1 Processor -- start 2024-01-15T18:08:40.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.441+0000 7fef5ec00640 1 -- start start 2024-01-15T18:08:40.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.441+0000 7fef5ec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef581f3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef58141c80 0x7fef5813a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58154720 0x7fef5813f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5ec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58157d80 con 0x7fef380047c0 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef58142be0 con 0x7fef58141c80 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef58142490 con 0x7fef58154720 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef581f3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef581f3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37070/0 (socket says 172.21.15.2:37070) 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 -- 172.21.15.2:0/1893674813 learned_addr learned my addr 172.21.15.2:0/1893674813 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5e200640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58154720 0x7fef5813f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 -- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58154720 msgr2=0x7fef5813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58154720 0x7fef5813f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 -- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef58141c80 msgr2=0x7fef5813a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef58141c80 0x7fef5813a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.442+0000 7fef5d800640 1 -- 172.21.15.2:0/1893674813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef581f40a0 con 0x7fef380047c0 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.443+0000 7fef5d800640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef581f3b60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fef4000e700 tx=0x7fef4000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.443+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4000ee10 con 0x7fef380047c0 2024-01-15T18:08:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.443+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef4000c8d0 con 0x7fef380047c0 2024-01-15T18:08:40.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.443+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef4000cc80 con 0x7fef380047c0 2024-01-15T18:08:40.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.443+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef581421b0 con 0x7fef380047c0 2024-01-15T18:08:40.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.443+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef58158290 con 0x7fef380047c0 2024-01-15T18:08:40.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.444+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef58074770 con 0x7fef380047c0 2024-01-15T18:08:40.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.445+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fef4001d460 con 0x7fef380047c0 2024-01-15T18:08:40.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.447+0000 7fef55600640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef3006bf20 0x7fef3006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:40.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.447+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fef400885c0 con 0x7fef380047c0 2024-01-15T18:08:40.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.448+0000 7fef5ce00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef3006bf20 0x7fef3006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:40.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.449+0000 7fef5ce00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef3006bf20 0x7fef3006e3e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fef48036000 tx=0x7fef4802e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:40.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.452+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef40054e80 con 0x7fef380047c0 2024-01-15T18:08:40.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:40.682+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471613", "target": ["mon-mgr", ""]}) v1 -- 0x7fef58143a00 con 0x7fef3006bf20 2024-01-15T18:08:50.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.345+0000 7fef55600640 1 -- 172.21.15.2:0/1893674813 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fef48004020 con 0x7fef3006bf20 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef3006bf20 msgr2=0x7fef3006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef3006bf20 0x7fef3006e3e0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fef48036000 tx=0x7fef4802e000 comp rx=0 tx=0).stop 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 msgr2=0x7fef581f3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef581f3b60 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fef4000e700 tx=0x7fef4000ebc0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 shutdown_connections 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef3006bf20 0x7fef3006e3e0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef58154720 0x7fef5813f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef58141c80 0x7fef5813a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 --2- 172.21.15.2:0/1893674813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef380047c0 0x7fef581f3b60 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.348+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 >> 172.21.15.2:0/1893674813 conn(0x7fef580b0840 msgr2=0x7fef580af200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:50.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.349+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 shutdown_connections 2024-01-15T18:08:50.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.349+0000 7fef5ec00640 1 -- 172.21.15.2:0/1893674813 wait complete. 2024-01-15T18:08:50.375 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471614 2024-01-15T18:08:50.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.594+0000 7faed8800640 1 Processor -- start 2024-01-15T18:08:50.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.594+0000 7faed8800640 1 -- start start 2024-01-15T18:08:50.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed8800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7faed00b32b0 0x7faed00d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed8800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faed015d9a0 0x7faed015de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed8800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed006f440 con 0x7faed0146ba0 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed8800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed0146f80 con 0x7faed015d9a0 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed8800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed015e390 con 0x7faed00b32b0 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed5e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7faed0146ba0 0x7faed00b2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39278/0 (socket says 172.21.15.2:39278) 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed5e00640 1 -- 172.21.15.2:0/2114495755 learned_addr learned my addr 172.21.15.2:0/2114495755 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:08:50.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.596+0000 7faed6800640 1 --2- 172.21.15.2:0/2114495755 >> v2:172.21.15.2:3300/0 conn(0x7faed015d9a0 0x7faed015de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.597+0000 7faed5400640 1 --2- 172.21.15.2:0/2114495755 >> v2:172.21.15.137:3300/0 conn(0x7faed00b32b0 0x7faed00d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.597+0000 7faed6800640 1 -- 172.21.15.2:0/2114495755 >> v2:172.21.15.137:3300/0 conn(0x7faed00b32b0 msgr2=0x7faed00d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:50.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.597+0000 7faed6800640 1 --2- 172.21.15.2:0/2114495755 >> v2:172.21.15.137:3300/0 conn(0x7faed00b32b0 0x7faed00d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.597+0000 7faed6800640 1 -- 172.21.15.2:0/2114495755 >> v1:172.21.15.2:6789/0 conn(0x7faed0146ba0 legacy=0x7faed00b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.597+0000 7faed6800640 1 -- 172.21.15.2:0/2114495755 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed015e5f0 con 0x7faed015d9a0 2024-01-15T18:08:50.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.598+0000 7faed6800640 1 --2- 172.21.15.2:0/2114495755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed015d9a0 0x7faed015de50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7faed0071f70 tx=0x7faec802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28fee547c9028a84 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.598+0000 7faed4a00640 1 -- 172.21.15.2:0/2114495755 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec802cbe0 con 0x7faed015d9a0 2024-01-15T18:08:50.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.598+0000 7faed4a00640 1 -- 172.21.15.2:0/2114495755 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec802cd40 con 0x7faed015d9a0 2024-01-15T18:08:50.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.598+0000 7faed4a00640 1 -- 172.21.15.2:0/2114495755 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec8034660 con 0x7faed015d9a0 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 -- 172.21.15.2:0/2114495755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed015d9a0 msgr2=0x7faed015de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 --2- 172.21.15.2:0/2114495755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed015d9a0 0x7faed015de50 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7faed0071f70 tx=0x7faec802be30 comp rx=0 tx=0).stop 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 -- 172.21.15.2:0/2114495755 shutdown_connections 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 --2- 172.21.15.2:0/2114495755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed015d9a0 0x7faed015de50 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 --2- 172.21.15.2:0/2114495755 >> v2:172.21.15.137:3300/0 conn(0x7faed00b32b0 0x7faed00d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 -- 172.21.15.2:0/2114495755 >> 172.21.15.2:0/2114495755 conn(0x7faed00b6430 msgr2=0x7faed00b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 -- 172.21.15.2:0/2114495755 shutdown_connections 2024-01-15T18:08:50.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.599+0000 7faed8800640 1 -- 172.21.15.2:0/2114495755 wait complete. 2024-01-15T18:08:50.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.601+0000 7faed8800640 1 Processor -- start 2024-01-15T18:08:50.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.601+0000 7faed8800640 1 -- start start 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed8800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faed00b32b0 0x7faed013f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed8800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 0x7faed00b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed8800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faed015d9a0 0x7faed00b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed8800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed013fb20 con 0x7faed0146ba0 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed8800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed0142810 con 0x7faed015d9a0 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed8800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed0142af0 con 0x7faed00b32b0 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 0x7faed00b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 0x7faed00b1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47366/0 (socket says 172.21.15.2:47366) 2024-01-15T18:08:50.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.602+0000 7faed5400640 1 -- 172.21.15.2:0/1235803491 learned_addr learned my addr 172.21.15.2:0/1235803491 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed6800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faed015d9a0 0x7faed00b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5e00640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faed00b32b0 0x7faed013f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5400640 1 -- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faed00b32b0 msgr2=0x7faed013f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5400640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faed00b32b0 0x7faed013f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5400640 1 -- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faed015d9a0 msgr2=0x7faed00b2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5400640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faed015d9a0 0x7faed00b2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5400640 1 -- 172.21.15.2:0/1235803491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec8009a70 con 0x7faed0146ba0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed6800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faed015d9a0 0x7faed00b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faed5400640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 0x7faed00b1e50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7faeb800d5f0 tx=0x7faeb800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80125f0 con 0x7faed0146ba0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.603+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeb8012750 con 0x7faed0146ba0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.604+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb8012b00 con 0x7faed0146ba0 2024-01-15T18:08:50.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.604+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed0142dd0 con 0x7faed0146ba0 2024-01-15T18:08:50.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.604+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed0143390 con 0x7faed0146ba0 2024-01-15T18:08:50.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.605+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed0143520 con 0x7faed0146ba0 2024-01-15T18:08:50.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.606+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7faeb8012c60 con 0x7faed0146ba0 2024-01-15T18:08:50.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.607+0000 7faece000640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faea806bf20 0x7faea806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:50.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.607+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7faeb8088ab0 con 0x7faed0146ba0 2024-01-15T18:08:50.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.607+0000 7faed5e00640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faea806bf20 0x7faea806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:50.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.608+0000 7faed5e00640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faea806bf20 0x7faea806e3e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7faec0005880 tx=0x7faec0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:50.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.612+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faeb8055370 con 0x7faed0146ba0 2024-01-15T18:08:50.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:50.841+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471614", "target": ["mon-mgr", ""]}) v1 -- 0x7faed00d1260 con 0x7faea806bf20 2024-01-15T18:08:53.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.348+0000 7faece000640 1 -- 172.21.15.2:0/1235803491 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faec0001340 con 0x7faea806bf20 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faea806bf20 msgr2=0x7faea806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faea806bf20 0x7faea806e3e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7faec0005880 tx=0x7faec0009040 comp rx=0 tx=0).stop 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 msgr2=0x7faed00b1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 0x7faed00b1e50 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7faeb800d5f0 tx=0x7faeb800dab0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 shutdown_connections 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faea806bf20 0x7faea806e3e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faed015d9a0 0x7faed00b2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faed0146ba0 0x7faed00b1e50 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 --2- 172.21.15.2:0/1235803491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faed00b32b0 0x7faed013f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.352+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 >> 172.21.15.2:0/1235803491 conn(0x7faed00b6430 msgr2=0x7faed015b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:53.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.353+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 shutdown_connections 2024-01-15T18:08:53.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.353+0000 7faed8800640 1 -- 172.21.15.2:0/1235803491 wait complete. 2024-01-15T18:08:53.384 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471615 2024-01-15T18:08:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.605+0000 7f2eb7c00640 1 Processor -- start 2024-01-15T18:08:53.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.605+0000 7f2eb7c00640 1 -- start start 2024-01-15T18:08:53.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb7c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2eb01439f0 0x7f2eb00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb7c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2eb00b3420 0x7f2eb00cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb7c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb015dd70 con 0x7f2eb00ccc70 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb7c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb00b38b0 con 0x7f2eb01439f0 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb7c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb00711b0 con 0x7f2eb00b3420 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb5200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2eb01439f0 0x7f2eb00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb5200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2eb01439f0 0x7f2eb00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34874/0 (socket says 172.21.15.2:34874) 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.607+0000 7f2eb5200640 1 -- 172.21.15.2:0/918238898 learned_addr learned my addr 172.21.15.2:0/918238898 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.608+0000 7f2eb5200640 1 -- 172.21.15.2:0/918238898 >> v2:172.21.15.137:3300/0 conn(0x7f2eb00b3420 msgr2=0x7f2eb00cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.608+0000 7f2eb5200640 1 --2- 172.21.15.2:0/918238898 >> v2:172.21.15.137:3300/0 conn(0x7f2eb00b3420 0x7f2eb00cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.608+0000 7f2eb5200640 1 -- 172.21.15.2:0/918238898 >> v1:172.21.15.137:6789/0 conn(0x7f2eb00ccc70 legacy=0x7f2eb006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:53.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.608+0000 7f2eb5200640 1 -- 172.21.15.2:0/918238898 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0143dd0 con 0x7f2eb01439f0 2024-01-15T18:08:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.609+0000 7f2eb5200640 1 --2- 172.21.15.2:0/918238898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb01439f0 0x7f2eb00702c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0072670 tx=0x7f2ea002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=371d8c498d54e594 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.609+0000 7f2eafe00640 1 -- 172.21.15.2:0/918238898 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea002f8e0 con 0x7f2eb01439f0 2024-01-15T18:08:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.609+0000 7f2eafe00640 1 -- 172.21.15.2:0/918238898 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea002fa40 con 0x7f2eb01439f0 2024-01-15T18:08:53.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.609+0000 7f2eafe00640 1 -- 172.21.15.2:0/918238898 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea002fdd0 con 0x7f2eb01439f0 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 -- 172.21.15.2:0/918238898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb01439f0 msgr2=0x7f2eb00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/918238898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb01439f0 0x7f2eb00702c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0072670 tx=0x7f2ea002db20 comp rx=0 tx=0).stop 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 -- 172.21.15.2:0/918238898 shutdown_connections 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/918238898 >> v2:172.21.15.137:3300/0 conn(0x7f2eb00b3420 0x7f2eb00cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/918238898 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb01439f0 0x7f2eb00702c0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 -- 172.21.15.2:0/918238898 >> 172.21.15.2:0/918238898 conn(0x7f2eb00b45f0 msgr2=0x7f2eb00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 -- 172.21.15.2:0/918238898 shutdown_connections 2024-01-15T18:08:53.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.610+0000 7f2eb7c00640 1 -- 172.21.15.2:0/918238898 wait complete. 2024-01-15T18:08:53.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.612+0000 7f2eb7c00640 1 Processor -- start 2024-01-15T18:08:53.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.612+0000 7f2eb7c00640 1 -- start start 2024-01-15T18:08:53.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.613+0000 7f2eb7c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 0x7f2eb013e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:53.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.613+0000 7f2eb7c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eb00ccc70 0x7f2eb013b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:53.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.613+0000 7f2eb7c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eb01439f0 0x7f2eb015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:53.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.613+0000 7f2eb7c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb013c0c0 con 0x7f2eb00b3420 2024-01-15T18:08:53.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.613+0000 7f2eb7c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb013be60 con 0x7f2eb01439f0 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.613+0000 7f2eb7c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb015d800 con 0x7f2eb00ccc70 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 0x7f2eb013e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 0x7f2eb013e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34880/0 (socket says 172.21.15.2:34880) 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 -- 172.21.15.2:0/239261950 learned_addr learned my addr 172.21.15.2:0/239261950 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2ea7e00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eb00ccc70 0x7f2eb013b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 -- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eb00ccc70 msgr2=0x7f2eb013b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eb01439f0 0x7f2eb015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eb00ccc70 0x7f2eb013b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 -- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eb01439f0 msgr2=0x7f2eb015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eb01439f0 0x7f2eb015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5200640 1 -- 172.21.15.2:0/239261950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea00094d0 con 0x7f2eb00b3420 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2ea7e00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eb00ccc70 0x7f2eb013b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.614+0000 7f2eb5c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eb01439f0 0x7f2eb015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:53.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.615+0000 7f2eb5200640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 0x7f2eb013e830 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00090b0 tx=0x7f2ea00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.615+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea002f890 con 0x7f2eb00b3420 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.615+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0042470 con 0x7f2eb00b3420 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.615+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0042820 con 0x7f2eb00b3420 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.615+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb013fa50 con 0x7f2eb00b3420 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.615+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb0140010 con 0x7f2eb00b3420 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.616+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb00686f0 con 0x7f2eb00b3420 2024-01-15T18:08:53.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.617+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2ea0042980 con 0x7f2eb00b3420 2024-01-15T18:08:53.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.619+0000 7f2ead600640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e8806bf20 0x7f2e8806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.619+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f2ea0033070 con 0x7f2eb00b3420 2024-01-15T18:08:53.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.619+0000 7f2ea7e00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e8806bf20 0x7f2e8806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:53.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.621+0000 7f2ea7e00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e8806bf20 0x7f2e8806e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2e98006fd0 tx=0x7f2e98008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:53.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.624+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ea0077bc0 con 0x7f2eb00b3420 2024-01-15T18:08:53.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:53.860+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471615", "target": ["mon-mgr", ""]}) v1 -- 0x7f2eb0070ed0 con 0x7f2e8806bf20 2024-01-15T18:08:56.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.537+0000 7f2ead600640 1 -- 172.21.15.2:0/239261950 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2e9800ae20 con 0x7f2e8806bf20 2024-01-15T18:08:56.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e8806bf20 msgr2=0x7f2e8806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e8806bf20 0x7f2e8806e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f2e98006fd0 tx=0x7f2e98008040 comp rx=0 tx=0).stop 2024-01-15T18:08:56.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 msgr2=0x7f2eb013e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 0x7f2eb013e830 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00090b0 tx=0x7f2ea00046c0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 shutdown_connections 2024-01-15T18:08:56.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e8806bf20 0x7f2e8806e3e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2eb01439f0 0x7f2eb015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2eb00ccc70 0x7f2eb013b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 --2- 172.21.15.2:0/239261950 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2eb00b3420 0x7f2eb013e830 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.541+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 >> 172.21.15.2:0/239261950 conn(0x7f2eb00b45f0 msgr2=0x7f2eb00af7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:56.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.542+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 shutdown_connections 2024-01-15T18:08:56.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.542+0000 7f2eb7c00640 1 -- 172.21.15.2:0/239261950 wait complete. 2024-01-15T18:08:56.568 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471616 2024-01-15T18:08:56.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.782+0000 7f45eca00640 1 Processor -- start 2024-01-15T18:08:56.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.783+0000 7f45eca00640 1 -- start start 2024-01-15T18:08:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.784+0000 7f45eca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f45e8143620 0x7f45e80b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.784+0000 7f45eca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f45e80b2a90 0x7f45e8071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.784+0000 7f45eca00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45e80b2fa0 con 0x7f45e8071770 2024-01-15T18:08:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.784+0000 7f45eca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e8071c00 con 0x7f45e80b2a90 2024-01-15T18:08:56.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.784+0000 7f45eca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e80b3280 con 0x7f45e8143620 2024-01-15T18:08:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45dfe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f45e80b2a90 0x7f45e8071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45e7200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f45e8071770 0x7f45e8070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39358/0 (socket says 172.21.15.2:39358) 2024-01-15T18:08:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45e7200640 1 -- 172.21.15.2:0/2572572256 learned_addr learned my addr 172.21.15.2:0/2572572256 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:08:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45e6800640 1 --2- 172.21.15.2:0/2572572256 >> v2:172.21.15.137:3300/0 conn(0x7f45e8143620 0x7f45e80b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45dfe00640 1 -- 172.21.15.2:0/2572572256 >> v2:172.21.15.137:3300/0 conn(0x7f45e8143620 msgr2=0x7f45e80b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:08:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45dfe00640 1 --2- 172.21.15.2:0/2572572256 >> v2:172.21.15.137:3300/0 conn(0x7f45e8143620 0x7f45e80b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45dfe00640 1 -- 172.21.15.2:0/2572572256 >> v1:172.21.15.2:6789/0 conn(0x7f45e8071770 legacy=0x7f45e8070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45dfe00640 1 -- 172.21.15.2:0/2572572256 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e80b3560 con 0x7f45e80b2a90 2024-01-15T18:08:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.785+0000 7f45e5e00640 1 -- 172.21.15.2:0/2572572256 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553836938 0 0) 0x7f45e80b2fa0 con 0x7f45e8071770 2024-01-15T18:08:56.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.786+0000 7f45dfe00640 1 --2- 172.21.15.2:0/2572572256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e8071230 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f45e806bb70 tx=0x7f45d002e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e29f79451d447ac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45e5e00640 1 -- 172.21.15.2:0/2572572256 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d002ee50 con 0x7f45e80b2a90 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45e5e00640 1 -- 172.21.15.2:0/2572572256 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d0036440 con 0x7f45e80b2a90 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45e5e00640 1 -- 172.21.15.2:0/2572572256 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d00367d0 con 0x7f45e80b2a90 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45eca00640 1 -- 172.21.15.2:0/2572572256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 msgr2=0x7f45e8071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45eca00640 1 --2- 172.21.15.2:0/2572572256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e8071230 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f45e806bb70 tx=0x7f45d002e0a0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45eca00640 1 -- 172.21.15.2:0/2572572256 shutdown_connections 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45eca00640 1 --2- 172.21.15.2:0/2572572256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e8071230 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45eca00640 1 --2- 172.21.15.2:0/2572572256 >> v2:172.21.15.137:3300/0 conn(0x7f45e8143620 0x7f45e80b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.787+0000 7f45eca00640 1 -- 172.21.15.2:0/2572572256 >> 172.21.15.2:0/2572572256 conn(0x7f45e80b0840 msgr2=0x7f45e80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:08:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.788+0000 7f45eca00640 1 -- 172.21.15.2:0/2572572256 shutdown_connections 2024-01-15T18:08:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.788+0000 7f45eca00640 1 -- 172.21.15.2:0/2572572256 wait complete. 2024-01-15T18:08:56.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.789+0000 7f45eca00640 1 Processor -- start 2024-01-15T18:08:56.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.789+0000 7f45eca00640 1 -- start start 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.790+0000 7f45eca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45e8071770 0x7f45e813a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.790+0000 7f45eca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e80ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.790+0000 7f45eca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45e8143620 0x7f45e8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.790+0000 7f45eca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e813b370 con 0x7f45e80b2a90 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.790+0000 7f45eca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e80adb10 con 0x7f45e8143620 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.790+0000 7f45eca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e813a6f0 con 0x7f45e8071770 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e80ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e80ad5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:34890/0 (socket says 172.21.15.2:34890) 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 -- 172.21.15.2:0/2213634031 learned_addr learned my addr 172.21.15.2:0/2213634031 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45e6800640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45e8071770 0x7f45e813a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45e7200640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45e8143620 0x7f45e8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 -- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45e8071770 msgr2=0x7f45e813a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45e8071770 0x7f45e813a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 -- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45e8143620 msgr2=0x7f45e8156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:08:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45e8143620 0x7f45e8156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:08:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 -- 172.21.15.2:0/2213634031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d000b6a0 con 0x7f45e80b2a90 2024-01-15T18:08:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45e7200640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45e8143620 0x7f45e8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:08:56.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.791+0000 7f45dfe00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e80ad5d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.792+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d003e7d0 con 0x7f45e80b2a90 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.792+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45d003e930 con 0x7f45e80b2a90 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.792+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d003eca0 con 0x7f45e80b2a90 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.792+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e813a9d0 con 0x7f45e80b2a90 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.792+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e813af90 con 0x7f45e80b2a90 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.793+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e813b4b0 con 0x7f45e80b2a90 2024-01-15T18:08:56.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.794+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f45d003d070 con 0x7f45e80b2a90 2024-01-15T18:08:56.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.795+0000 7f45dea00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f45c006bf50 0x7f45c006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:08:56.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.795+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f45d00aa960 con 0x7f45e80b2a90 2024-01-15T18:08:56.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.796+0000 7f45e6800640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f45c006bf50 0x7f45c006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:08:56.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.797+0000 7f45e6800640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f45c006bf50 0x7f45c006e410 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f45d8006fd0 tx=0x7f45d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:08:56.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:56.800+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45e813b4b0 con 0x7f45e80b2a90 2024-01-15T18:08:57.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:08:57.039+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471616", "target": ["mon-mgr", ""]}) v1 -- 0x7f45e8143a00 con 0x7f45c006bf50 2024-01-15T18:09:01.867 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:01.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:03.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.630+0000 7f45dea00640 1 -- 172.21.15.2:0/2213634031 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f45d800f860 con 0x7f45c006bf50 2024-01-15T18:09:03.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f45c006bf50 msgr2=0x7f45c006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:03.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f45c006bf50 0x7f45c006e410 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f45d8006fd0 tx=0x7f45d8008040 comp rx=0 tx=0).stop 2024-01-15T18:09:03.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 msgr2=0x7f45e80ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:03.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e80ad5d0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f45d0000c00 tx=0x7f45d00046c0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 shutdown_connections 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f45c006bf50 0x7f45c006e410 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45e8143620 0x7f45e8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45e80b2a90 0x7f45e80ad5d0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 --2- 172.21.15.2:0/2213634031 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45e8071770 0x7f45e813a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.634+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 >> 172.21.15.2:0/2213634031 conn(0x7f45e80b0840 msgr2=0x7f45e80b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.635+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 shutdown_connections 2024-01-15T18:09:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.635+0000 7f45eca00640 1 -- 172.21.15.2:0/2213634031 wait complete. 2024-01-15T18:09:03.662 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471617 2024-01-15T18:09:03.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.882+0000 7ff891600640 1 Processor -- start 2024-01-15T18:09:03.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.883+0000 7ff891600640 1 -- start start 2024-01-15T18:09:03.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.884+0000 7ff891600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff88c15ae80 0x7ff88c0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.884+0000 7ff891600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.884+0000 7ff891600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff88c0b2430 0x7ff88c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.884+0000 7ff891600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c15e840 con 0x7ff88c15ae80 2024-01-15T18:09:03.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.884+0000 7ff891600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88c06d290 con 0x7ff88c0b2430 2024-01-15T18:09:03.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.884+0000 7ff891600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c15dbc0 con 0x7ff88c06c8f0 2024-01-15T18:09:03.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff88c15ae80 0x7ff88c0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff88c15ae80 0x7ff88c0acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38320/0 (socket says 172.21.15.2:38320) 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 -- 172.21.15.2:0/230877017 learned_addr learned my addr 172.21.15.2:0/230877017 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 -- 172.21.15.2:0/230877017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 msgr2=0x7ff88c06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88b400640 1 --2- 172.21.15.2:0/230877017 >> v2:172.21.15.137:3300/0 conn(0x7ff88c0b2430 0x7ff88c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88a000640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 -- 172.21.15.2:0/230877017 >> v2:172.21.15.137:3300/0 conn(0x7ff88c0b2430 msgr2=0x7ff88c0b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 --2- 172.21.15.2:0/230877017 >> v2:172.21.15.137:3300/0 conn(0x7ff88c0b2430 0x7ff88c0b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88aa00640 1 -- 172.21.15.2:0/230877017 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88c15dea0 con 0x7ff88c15ae80 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88b400640 1 --2- 172.21.15.2:0/230877017 >> v2:172.21.15.137:3300/0 conn(0x7ff88c0b2430 0x7ff88c0b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:03.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.885+0000 7ff88a000640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.887+0000 7ff88aa00640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c0acd00 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0cea40 tx=0x7ff87c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109b9420e9c6073a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.887+0000 7ff889600640 1 -- 172.21.15.2:0/230877017 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c031640 con 0x7ff88c15ae80 2024-01-15T18:09:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.887+0000 7ff889600640 1 -- 172.21.15.2:0/230877017 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c0317a0 con 0x7ff88c15ae80 2024-01-15T18:09:03.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.887+0000 7ff889600640 1 -- 172.21.15.2:0/230877017 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c031b30 con 0x7ff88c15ae80 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 -- 172.21.15.2:0/230877017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 msgr2=0x7ff88c0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c0acd00 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7ff88c0cea40 tx=0x7ff87c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 -- 172.21.15.2:0/230877017 shutdown_connections 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 --2- 172.21.15.2:0/230877017 >> v2:172.21.15.137:3300/0 conn(0x7ff88c0b2430 0x7ff88c0b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 --2- 172.21.15.2:0/230877017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c0acd00 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 -- 172.21.15.2:0/230877017 >> 172.21.15.2:0/230877017 conn(0x7ff88c0b45f0 msgr2=0x7ff88c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 -- 172.21.15.2:0/230877017 shutdown_connections 2024-01-15T18:09:03.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.888+0000 7ff891600640 1 -- 172.21.15.2:0/230877017 wait complete. 2024-01-15T18:09:03.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.890+0000 7ff891600640 1 Processor -- start 2024-01-15T18:09:03.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.890+0000 7ff891600640 1 -- start start 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff891600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c1f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff891600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff88c0b2430 0x7ff88c15fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff891600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c1601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff891600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0aeff0 con 0x7ff88c15ae80 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff891600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0af260 con 0x7ff88c0b2430 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff891600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff88c0af540 con 0x7ff88c06c8f0 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff88b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c1601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff88b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c1601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38330/0 (socket says 172.21.15.2:38330) 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff88b400640 1 -- 172.21.15.2:0/2136411183 learned_addr learned my addr 172.21.15.2:0/2136411183 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.891+0000 7ff88aa00640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c1f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88a000640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff88c0b2430 0x7ff88c15fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88b400640 1 -- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 msgr2=0x7ff88c1f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88b400640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c1f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88b400640 1 -- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff88c0b2430 msgr2=0x7ff88c15fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88b400640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff88c0b2430 0x7ff88c15fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:03.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88b400640 1 -- 172.21.15.2:0/2136411183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0094d0 con 0x7ff88c15ae80 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff88b400640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c1601d0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff88400c7f0 tx=0x7ff88400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884007820 con 0x7ff88c15ae80 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff884007980 con 0x7ff88c15ae80 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884007d70 con 0x7ff88c15ae80 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.892+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c0af820 con 0x7ff88c15ae80 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.893+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88c14c750 con 0x7ff88c15ae80 2024-01-15T18:09:03.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.894+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88c0afd40 con 0x7ff88c15ae80 2024-01-15T18:09:03.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.895+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff8840113f0 con 0x7ff88c15ae80 2024-01-15T18:09:03.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.896+0000 7ff882a00640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff86406bf20 0x7ff86406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:03.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.896+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7ff884088c20 con 0x7ff88c15ae80 2024-01-15T18:09:03.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.896+0000 7ff88aa00640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff86406bf20 0x7ff86406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:03.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.897+0000 7ff88aa00640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff86406bf20 0x7ff86406e3e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff87c036000 tx=0x7ff87c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:03.900+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff88c0afd40 con 0x7ff88c15ae80 2024-01-15T18:09:04.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:04.134+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471617", "target": ["mon-mgr", ""]}) v1 -- 0x7ff88c0b2e00 con 0x7ff86406bf20 2024-01-15T18:09:13.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.789+0000 7ff882a00640 1 -- 172.21.15.2:0/2136411183 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff87c031a90 con 0x7ff86406bf20 2024-01-15T18:09:13.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.792+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff86406bf20 msgr2=0x7ff86406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.792+0000 7ff891600640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff86406bf20 0x7ff86406e3e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff87c036000 tx=0x7ff87c03c040 comp rx=0 tx=0).stop 2024-01-15T18:09:13.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 msgr2=0x7ff88c1601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:13.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c1601d0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7ff88400c7f0 tx=0x7ff88400ccb0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 shutdown_connections 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff86406bf20 0x7ff86406e3e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff88c15ae80 0x7ff88c1601d0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff88c0b2430 0x7ff88c15fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 --2- 172.21.15.2:0/2136411183 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff88c06c8f0 0x7ff88c1f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 >> 172.21.15.2:0/2136411183 conn(0x7ff88c0b45f0 msgr2=0x7ff88c071ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 shutdown_connections 2024-01-15T18:09:13.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:13.793+0000 7ff891600640 1 -- 172.21.15.2:0/2136411183 wait complete. 2024-01-15T18:09:13.819 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471618 2024-01-15T18:09:14.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.041+0000 7f0d10200640 1 Processor -- start 2024-01-15T18:09:14.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.042+0000 7f0d10200640 1 -- start start 2024-01-15T18:09:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.043+0000 7f0d10200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0d081439f0 0x7f0d080702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.043+0000 7f0d10200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0d0806c8f0 0x7f0d0806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.043+0000 7f0d10200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d0815dd70 con 0x7f0d080b3420 2024-01-15T18:09:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.043+0000 7f0d10200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080b38b0 con 0x7f0d0806c8f0 2024-01-15T18:09:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.043+0000 7f0d10200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d0806d230 con 0x7f0d081439f0 2024-01-15T18:09:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0ce00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0d080b3420 0x7f0d080cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:59426/0 (socket says 172.21.15.2:59426) 2024-01-15T18:09:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0ce00640 1 -- 172.21.15.2:0/2286997784 learned_addr learned my addr 172.21.15.2:0/2286997784 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:14.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0e200640 1 --2- 172.21.15.2:0/2286997784 >> v2:172.21.15.2:3300/0 conn(0x7f0d0806c8f0 0x7f0d0806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0d800640 1 --2- 172.21.15.2:0/2286997784 >> v2:172.21.15.137:3300/0 conn(0x7f0d081439f0 0x7f0d080702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0e200640 1 -- 172.21.15.2:0/2286997784 >> v2:172.21.15.137:3300/0 conn(0x7f0d081439f0 msgr2=0x7f0d080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0e200640 1 --2- 172.21.15.2:0/2286997784 >> v2:172.21.15.137:3300/0 conn(0x7f0d081439f0 0x7f0d080702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.044+0000 7f0d0e200640 1 -- 172.21.15.2:0/2286997784 >> v1:172.21.15.2:6789/0 conn(0x7f0d080b3420 legacy=0x7f0d080cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.045+0000 7f0d0e200640 1 -- 172.21.15.2:0/2286997784 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080711b0 con 0x7f0d0806c8f0 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.045+0000 7f0d07e00640 1 -- 172.21.15.2:0/2286997784 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103766299 0 0) 0x7f0d0815dd70 con 0x7f0d080b3420 2024-01-15T18:09:14.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.046+0000 7f0d0e200640 1 --2- 172.21.15.2:0/2286997784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d0806c8f0 0x7f0d0806ccf0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0d080b28d0 tx=0x7f0d0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321d9dc7a44d68df server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.046+0000 7f0d07e00640 1 -- 172.21.15.2:0/2286997784 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0002cc80 con 0x7f0d0806c8f0 2024-01-15T18:09:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.046+0000 7f0d07e00640 1 -- 172.21.15.2:0/2286997784 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0002cde0 con 0x7f0d0806c8f0 2024-01-15T18:09:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.046+0000 7f0d07e00640 1 -- 172.21.15.2:0/2286997784 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d00034660 con 0x7f0d0806c8f0 2024-01-15T18:09:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 -- 172.21.15.2:0/2286997784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d0806c8f0 msgr2=0x7f0d0806ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 --2- 172.21.15.2:0/2286997784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d0806c8f0 0x7f0d0806ccf0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f0d080b28d0 tx=0x7f0d0002bed0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 -- 172.21.15.2:0/2286997784 shutdown_connections 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 --2- 172.21.15.2:0/2286997784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d0806c8f0 0x7f0d0806ccf0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 --2- 172.21.15.2:0/2286997784 >> v2:172.21.15.137:3300/0 conn(0x7f0d081439f0 0x7f0d080702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 -- 172.21.15.2:0/2286997784 >> 172.21.15.2:0/2286997784 conn(0x7f0d080b45f0 msgr2=0x7f0d080b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 -- 172.21.15.2:0/2286997784 shutdown_connections 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.047+0000 7f0d10200640 1 -- 172.21.15.2:0/2286997784 wait complete. 2024-01-15T18:09:14.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.049+0000 7f0d10200640 1 Processor -- start 2024-01-15T18:09:14.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.049+0000 7f0d10200640 1 -- start start 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d10200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d0806c8f0 0x7f0d0815b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d10200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 0x7f0d0815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d10200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d081439f0 0x7f0d080adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d10200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815ad70 con 0x7f0d080b3420 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d10200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815bec0 con 0x7f0d0806c8f0 2024-01-15T18:09:14.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d10200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0815c190 con 0x7f0d081439f0 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d0ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 0x7f0d0815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d0ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 0x7f0d0815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47168/0 (socket says 172.21.15.2:47168) 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.050+0000 7f0d0ce00640 1 -- 172.21.15.2:0/3994690023 learned_addr learned my addr 172.21.15.2:0/3994690023 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0d800640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d0806c8f0 0x7f0d0815b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0ce00640 1 -- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d081439f0 msgr2=0x7f0d080adb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0ce00640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d081439f0 0x7f0d080adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0ce00640 1 -- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d0806c8f0 msgr2=0x7f0d0815b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0ce00640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d0806c8f0 0x7f0d0815b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0ce00640 1 -- 172.21.15.2:0/3994690023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d00009a70 con 0x7f0d080b3420 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d0ce00640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 0x7f0d0815b980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000c4d0 tx=0x7f0cf000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf00074e0 con 0x7f0d080b3420 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf0007660 con 0x7f0d080b3420 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0004a20 con 0x7f0d080b3420 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.051+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0815a720 con 0x7f0d080b3420 2024-01-15T18:09:14.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.052+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0815c470 con 0x7f0d080b3420 2024-01-15T18:09:14.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.053+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d0815aee0 con 0x7f0d080b3420 2024-01-15T18:09:14.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.054+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0cf0004b80 con 0x7f0d080b3420 2024-01-15T18:09:14.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.054+0000 7f0d05600640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ce006bf20 0x7f0ce006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:14.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.055+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0cf0087a50 con 0x7f0d080b3420 2024-01-15T18:09:14.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.055+0000 7f0d0d800640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ce006bf20 0x7f0ce006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:14.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.056+0000 7f0d0d800640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ce006bf20 0x7f0ce006e3e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:14.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.059+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cf0054090 con 0x7f0d080b3420 2024-01-15T18:09:14.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:14.289+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471618", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d08070bf0 con 0x7f0ce006bf20 2024-01-15T18:09:17.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.973+0000 7f0d05600640 1 -- 172.21.15.2:0/3994690023 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0cf8009540 con 0x7f0ce006bf20 2024-01-15T18:09:17.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.978+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ce006bf20 msgr2=0x7f0ce006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:17.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.978+0000 7f0d10200640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ce006bf20 0x7f0ce006e3e0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8008040 comp rx=0 tx=0).stop 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.978+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 msgr2=0x7f0d0815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.978+0000 7f0d10200640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 0x7f0d0815b980 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f0cf000c4d0 tx=0x7f0cf000c990 comp rx=0 tx=0).stop 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.979+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 shutdown_connections 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.979+0000 7f0d10200640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ce006bf20 0x7f0ce006e3e0 secure :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0cf8006fd0 tx=0x7f0cf8008040 comp rx=0 tx=0).stop 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.979+0000 7f0d10200640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0d081439f0 0x7f0d080adb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.979+0000 7f0d10200640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0d080b3420 0x7f0d0815b980 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.979+0000 7f0d10200640 1 --2- 172.21.15.2:0/3994690023 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0d0806c8f0 0x7f0d0815b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.979+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 >> 172.21.15.2:0/3994690023 conn(0x7f0d080b45f0 msgr2=0x7f0d08071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.980+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 shutdown_connections 2024-01-15T18:09:17.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:17.980+0000 7f0d10200640 1 -- 172.21.15.2:0/3994690023 wait complete. 2024-01-15T18:09:18.008 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471619 2024-01-15T18:09:18.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.286+0000 7f895a800640 1 Processor -- start 2024-01-15T18:09:18.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.287+0000 7f895a800640 1 -- start start 2024-01-15T18:09:18.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.289+0000 7f895a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f895414b3b0 0x7f89540709d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.289+0000 7f8953400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f895414afd0 0x7f895406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:59438/0 (socket says 172.21.15.2:59438) 2024-01-15T18:09:18.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.290+0000 7f8953400640 1 -- 172.21.15.2:0/3298114235 learned_addr learned my addr 172.21.15.2:0/3298114235 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:18.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.290+0000 7f895a800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89540b25f0 con 0x7f895414afd0 2024-01-15T18:09:18.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.290+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89540b28d0 con 0x7f895406d000 2024-01-15T18:09:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.290+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8954070ff0 con 0x7f895414b3b0 2024-01-15T18:09:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.290+0000 7f8953e00640 1 --2- 172.21.15.2:0/3298114235 >> v2:172.21.15.2:3300/0 conn(0x7f895414b3b0 0x7f89540709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.290+0000 7f8952000640 1 -- 172.21.15.2:0/3298114235 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541801232 0 0) 0x7f89540b25f0 con 0x7f895414afd0 2024-01-15T18:09:18.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.291+0000 7f8952000640 1 -- 172.21.15.2:0/3298114235 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8934003520 con 0x7f895414afd0 2024-01-15T18:09:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.291+0000 7f8953e00640 1 -- 172.21.15.2:0/3298114235 >> v1:172.21.15.137:6789/0 conn(0x7f895406d000 legacy=0x7f89540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.291+0000 7f8953e00640 1 -- 172.21.15.2:0/3298114235 >> v1:172.21.15.2:6789/0 conn(0x7f895414afd0 legacy=0x7f895406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.291+0000 7f8952000640 1 -- 172.21.15.2:0/3298114235 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 681477541 0 0) 0x7f8934003520 con 0x7f895414afd0 2024-01-15T18:09:18.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.291+0000 7f8953e00640 1 -- 172.21.15.2:0/3298114235 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89540b2bb0 con 0x7f895414b3b0 2024-01-15T18:09:18.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.292+0000 7f8953e00640 1 --2- 172.21.15.2:0/3298114235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89540709d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f894c0097a0 tx=0x7f894c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c25bfa113f9781f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:18.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.293+0000 7f8952000640 1 -- 172.21.15.2:0/3298114235 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89440019f0 con 0x7f895414b3b0 2024-01-15T18:09:18.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.293+0000 7f8952000640 1 -- 172.21.15.2:0/3298114235 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f894c02e050 con 0x7f895414b3b0 2024-01-15T18:09:18.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.293+0000 7f8952000640 1 -- 172.21.15.2:0/3298114235 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894c030730 con 0x7f895414b3b0 2024-01-15T18:09:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.294+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 msgr2=0x7f89540709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.294+0000 7f895a800640 1 --2- 172.21.15.2:0/3298114235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89540709d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f894c0097a0 tx=0x7f894c02b790 comp rx=0 tx=0).stop 2024-01-15T18:09:18.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.294+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 shutdown_connections 2024-01-15T18:09:18.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.294+0000 7f895a800640 1 --2- 172.21.15.2:0/3298114235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89540709d0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.294+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 >> 172.21.15.2:0/3298114235 conn(0x7f89540b45f0 msgr2=0x7f89540b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:18.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.295+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 shutdown_connections 2024-01-15T18:09:18.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.295+0000 7f895a800640 1 -- 172.21.15.2:0/3298114235 wait complete. 2024-01-15T18:09:18.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.298+0000 7f895a800640 1 Processor -- start 2024-01-15T18:09:18.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.298+0000 7f895a800640 1 -- start start 2024-01-15T18:09:18.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.300+0000 7f895a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f895406d000 0x7f89540cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.300+0000 7f895a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f895414afd0 0x7f89541fb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.300+0000 7f895a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89541e94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.300+0000 7f895a800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89541de7c0 con 0x7f895414b3b0 2024-01-15T18:09:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.300+0000 7f895a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89540ce8a0 con 0x7f895406d000 2024-01-15T18:09:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.300+0000 7f895a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89541fb850 con 0x7f895414afd0 2024-01-15T18:09:18.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8953e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89541e94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8953e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89541e94e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47186/0 (socket says 172.21.15.2:47186) 2024-01-15T18:09:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8953e00640 1 -- 172.21.15.2:0/3844554634 learned_addr learned my addr 172.21.15.2:0/3844554634 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8952a00640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f895414afd0 0x7f89541fb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8953400640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f895406d000 0x7f89540cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8952a00640 1 -- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f895406d000 msgr2=0x7f89540cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8952a00640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f895406d000 0x7f89540cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8952a00640 1 -- 172.21.15.2:0/3844554634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 msgr2=0x7f89541e94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.301+0000 7f8952a00640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89541e94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.302+0000 7f8952a00640 1 -- 172.21.15.2:0/3844554634 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894c009400 con 0x7f895414afd0 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.302+0000 7f8952a00640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f895414afd0 0x7f89541fb310 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f893c00c390 tx=0x7f893c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.302+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c0052d0 con 0x7f895414afd0 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.303+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893c005430 con 0x7f895414afd0 2024-01-15T18:09:18.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.303+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893c003830 con 0x7f895414afd0 2024-01-15T18:09:18.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.303+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89540ce060 con 0x7f895414afd0 2024-01-15T18:09:18.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.303+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89540ce5c0 con 0x7f895414afd0 2024-01-15T18:09:18.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.305+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8954149d90 con 0x7f895414afd0 2024-01-15T18:09:18.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.307+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f893c003990 con 0x7f895414afd0 2024-01-15T18:09:18.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.308+0000 7f894b400640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f892c06bf20 0x7f892c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:18.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.309+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f893c087d10 con 0x7f895414afd0 2024-01-15T18:09:18.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.323+0000 7f8953400640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f892c06bf20 0x7f892c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:18.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.325+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f893c054400 con 0x7f895414afd0 2024-01-15T18:09:18.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.326+0000 7f8953400640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f892c06bf20 0x7f892c06e3e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8944001560 tx=0x7f8944009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:18.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.685+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471619", "target": ["mon-mgr", ""]}) v1 -- 0x7f89540686f0 con 0x7f892c06bf20 2024-01-15T18:09:18.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.865+0000 7f894b400640 1 -- 172.21.15.2:0/3844554634 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f89540686f0 con 0x7f892c06bf20 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f892c06bf20 msgr2=0x7f892c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f892c06bf20 0x7f892c06e3e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8944001560 tx=0x7f8944009040 comp rx=0 tx=0).stop 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f895414afd0 msgr2=0x7f89541fb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f895414afd0 0x7f89541fb310 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f893c00c390 tx=0x7f893c00c850 comp rx=0 tx=0).stop 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 shutdown_connections 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f892c06bf20 0x7f892c06e3e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f895414b3b0 0x7f89541e94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f895414afd0 0x7f89541fb310 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 --2- 172.21.15.2:0/3844554634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f895406d000 0x7f89540cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 >> 172.21.15.2:0/3844554634 conn(0x7f89540b45f0 msgr2=0x7f8954145040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:18.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 shutdown_connections 2024-01-15T18:09:18.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:18.869+0000 7f895a800640 1 -- 172.21.15.2:0/3844554634 wait complete. 2024-01-15T18:09:18.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone471620 2024-01-15T18:09:19.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.101+0000 7f75c3e00640 1 Processor -- start 2024-01-15T18:09:19.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.101+0000 7f75c3e00640 1 -- start start 2024-01-15T18:09:19.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.102+0000 7f75c3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f75c40d1870 0x7f75c40d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.102+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40b2210 0x7f75c415d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.102+0000 7f75c3e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c40d1250 con 0x7f75c40d1c50 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.102+0000 7f75c3e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c40ce6d0 con 0x7f75c40d1870 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.102+0000 7f75c3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40ce8e0 con 0x7f75c40b2210 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f75c40d1870 0x7f75c40d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f75c40d1870 0x7f75c40d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47202/0 (socket says 172.21.15.2:47202) 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 -- 172.21.15.2:0/3083519404 learned_addr learned my addr 172.21.15.2:0/3083519404 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2000640 1 --2- 172.21.15.2:0/3083519404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40b2210 0x7f75c415d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 -- 172.21.15.2:0/3083519404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40b2210 msgr2=0x7f75c415d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 --2- 172.21.15.2:0/3083519404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40b2210 0x7f75c415d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 -- 172.21.15.2:0/3083519404 >> v1:172.21.15.137:6789/0 conn(0x7f75c40d1c50 legacy=0x7f75c40cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.103+0000 7f75c2a00640 1 -- 172.21.15.2:0/3083519404 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c40cebc0 con 0x7f75c40d1870 2024-01-15T18:09:19.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.105+0000 7f75c2a00640 1 --2- 172.21.15.2:0/3083519404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c40d3bd0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009870 tx=0x7f75b402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f10b06b05530f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.105+0000 7f75c1600640 1 -- 172.21.15.2:0/3083519404 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4030840 con 0x7f75c40d1870 2024-01-15T18:09:19.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.105+0000 7f75c1600640 1 -- 172.21.15.2:0/3083519404 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b40309a0 con 0x7f75c40d1870 2024-01-15T18:09:19.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.105+0000 7f75c1600640 1 -- 172.21.15.2:0/3083519404 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b4030d30 con 0x7f75c40d1870 2024-01-15T18:09:19.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 -- 172.21.15.2:0/3083519404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 msgr2=0x7f75c40d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 --2- 172.21.15.2:0/3083519404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c40d3bd0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009870 tx=0x7f75b402b980 comp rx=0 tx=0).stop 2024-01-15T18:09:19.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 -- 172.21.15.2:0/3083519404 shutdown_connections 2024-01-15T18:09:19.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 --2- 172.21.15.2:0/3083519404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40b2210 0x7f75c415d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 --2- 172.21.15.2:0/3083519404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c40d3bd0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 -- 172.21.15.2:0/3083519404 >> 172.21.15.2:0/3083519404 conn(0x7f75c40d30b0 msgr2=0x7f75c40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:19.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 -- 172.21.15.2:0/3083519404 shutdown_connections 2024-01-15T18:09:19.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.106+0000 7f75c3e00640 1 -- 172.21.15.2:0/3083519404 wait complete. 2024-01-15T18:09:19.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.108+0000 7f75c3e00640 1 Processor -- start 2024-01-15T18:09:19.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.108+0000 7f75c3e00640 1 -- start start 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75c40b2210 0x7f75c40cae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c41f77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40d1c50 0x7f75c4148ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40b8640 con 0x7f75c40d1870 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c40b68b0 con 0x7f75c40b2210 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c41f7d10 con 0x7f75c40d1c50 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c41f77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c41f77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:47206/0 (socket says 172.21.15.2:47206) 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.109+0000 7f75c2000640 1 -- 172.21.15.2:0/2668991862 learned_addr learned my addr 172.21.15.2:0/2668991862 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:19.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c3400640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40d1c50 0x7f75c4148ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2a00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75c40b2210 0x7f75c40cae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2000640 1 -- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40d1c50 msgr2=0x7f75c4148ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2000640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40d1c50 0x7f75c4148ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2000640 1 -- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75c40b2210 msgr2=0x7f75c40cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2000640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75c40b2210 0x7f75c40cae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2000640 1 -- 172.21.15.2:0/2668991862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b40094d0 con 0x7f75c40d1870 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2a00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75c40b2210 0x7f75c40cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.110+0000 7f75c2000640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c41f77d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00d800 tx=0x7f75ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.111+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00b6c0 con 0x7f75c40d1870 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.111+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac00b820 con 0x7f75c40d1870 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.111+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac00bbb0 con 0x7f75c40d1870 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.111+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c40b8890 con 0x7f75c40d1870 2024-01-15T18:09:19.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.111+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c40b6e00 con 0x7f75c40d1870 2024-01-15T18:09:19.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.112+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4074770 con 0x7f75c40d1870 2024-01-15T18:09:19.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.113+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f75ac00bd10 con 0x7f75c40d1870 2024-01-15T18:09:19.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.114+0000 7f75baa00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f759c06bf20 0x7f759c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:19.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.114+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f75ac088ba0 con 0x7f75c40d1870 2024-01-15T18:09:19.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.114+0000 7f75c2a00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f759c06bf20 0x7f759c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:19.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.116+0000 7f75c2a00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f759c06bf20 0x7f759c06e3e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f75b4036000 tx=0x7f75b403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.118+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75ac055460 con 0x7f75c40d1870 2024-01-15T18:09:19.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:19.348+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone471620", "target": ["mon-mgr", ""]}) v1 -- 0x7f75c40d0f20 con 0x7f759c06bf20 2024-01-15T18:09:20.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:20.668+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f75ac055830 con 0x7f75c40d1870 2024-01-15T18:09:28.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.451+0000 7f75baa00640 1 -- 172.21.15.2:0/2668991862 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f75b4030be0 con 0x7f759c06bf20 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f759c06bf20 msgr2=0x7f759c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f759c06bf20 0x7f759c06e3e0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f75b4036000 tx=0x7f75b403b040 comp rx=0 tx=0).stop 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 msgr2=0x7f75c41f77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c41f77d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00d800 tx=0x7f75ac00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 shutdown_connections 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f759c06bf20 0x7f759c06e3e0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f75c40d1c50 0x7f75c4148ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f75c40d1870 0x7f75c41f77d0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 --2- 172.21.15.2:0/2668991862 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f75c40b2210 0x7f75c40cae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.454+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 >> 172.21.15.2:0/2668991862 conn(0x7f75c40d30b0 msgr2=0x7f75c40cdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:28.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.455+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 shutdown_connections 2024-01-15T18:09:28.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.455+0000 7f75c3e00640 1 -- 172.21.15.2:0/2668991862 wait complete. 2024-01-15T18:09:28.482 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:28.482 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47161 2024-01-15T18:09:28.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.707+0000 7f3d69600640 1 Processor -- start 2024-01-15T18:09:28.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.707+0000 7f3d69600640 1 -- start start 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.708+0000 7f3d69600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d640702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.708+0000 7f3d69600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d640b3420 0x7f3d640cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.708+0000 7f3d69600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d6415dd70 con 0x7f3d640ccc70 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.708+0000 7f3d69600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d640b3880 con 0x7f3d640b3420 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.708+0000 7f3d69600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640711b0 con 0x7f3d641439f0 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d640b3420 0x7f3d640cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d640b3420 0x7f3d640cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60284/0 (socket says 172.21.15.2:60284) 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 -- 172.21.15.2:0/1082578171 learned_addr learned my addr 172.21.15.2:0/1082578171 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d62a00640 1 --2- 172.21.15.2:0/1082578171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d640702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 -- 172.21.15.2:0/1082578171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 msgr2=0x7f3d640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 --2- 172.21.15.2:0/1082578171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d640702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 -- 172.21.15.2:0/1082578171 >> v1:172.21.15.137:6789/0 conn(0x7f3d640ccc70 legacy=0x7f3d6406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:28.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.709+0000 7f3d5be00640 1 -- 172.21.15.2:0/1082578171 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d64143dd0 con 0x7f3d640b3420 2024-01-15T18:09:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.711+0000 7f3d5be00640 1 --2- 172.21.15.2:0/1082578171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d640cc730 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009870 tx=0x7f3d4c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e307dc22a62e9e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.711+0000 7f3d62000640 1 -- 172.21.15.2:0/1082578171 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c031640 con 0x7f3d640b3420 2024-01-15T18:09:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.711+0000 7f3d62000640 1 -- 172.21.15.2:0/1082578171 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c0317a0 con 0x7f3d640b3420 2024-01-15T18:09:28.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.711+0000 7f3d62000640 1 -- 172.21.15.2:0/1082578171 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c031b30 con 0x7f3d640b3420 2024-01-15T18:09:28.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 -- 172.21.15.2:0/1082578171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 msgr2=0x7f3d640cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:28.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 --2- 172.21.15.2:0/1082578171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d640cc730 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c009870 tx=0x7f3d4c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 -- 172.21.15.2:0/1082578171 shutdown_connections 2024-01-15T18:09:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 --2- 172.21.15.2:0/1082578171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d640cc730 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 --2- 172.21.15.2:0/1082578171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d640702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 -- 172.21.15.2:0/1082578171 >> 172.21.15.2:0/1082578171 conn(0x7f3d640b45f0 msgr2=0x7f3d640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 -- 172.21.15.2:0/1082578171 shutdown_connections 2024-01-15T18:09:28.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.712+0000 7f3d69600640 1 -- 172.21.15.2:0/1082578171 wait complete. 2024-01-15T18:09:28.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.714+0000 7f3d69600640 1 Processor -- start 2024-01-15T18:09:28.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.715+0000 7f3d69600640 1 -- start start 2024-01-15T18:09:28.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d69600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d64166160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d69600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d640ccc70 0x7f3d640adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d69600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d64167900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d69600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640ae460 con 0x7f3d640b3420 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d69600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640a8570 con 0x7f3d640ccc70 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d69600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d64165fa0 con 0x7f3d641439f0 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d62a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d64166160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d62a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d64166160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60296/0 (socket says 172.21.15.2:60296) 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.716+0000 7f3d62a00640 1 -- 172.21.15.2:0/356767573 learned_addr learned my addr 172.21.15.2:0/356767573 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d62a00640 1 -- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 msgr2=0x7f3d64167900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d5be00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d640ccc70 0x7f3d640adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d63400640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d64167900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d62a00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d64167900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d62a00640 1 -- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d640ccc70 msgr2=0x7f3d640adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:28.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d62a00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d640ccc70 0x7f3d640adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d62a00640 1 -- 172.21.15.2:0/356767573 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c0094d0 con 0x7f3d640b3420 2024-01-15T18:09:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d5be00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d640ccc70 0x7f3d640adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d63400640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d64167900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:28.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.717+0000 7f3d62a00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d64166160 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400d830 tx=0x7f3d5400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.718+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5400b660 con 0x7f3d640b3420 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.718+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5400b7c0 con 0x7f3d640b3420 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.718+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5400bb50 con 0x7f3d640b3420 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.718+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d64166860 con 0x7f3d640b3420 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.718+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d64166e20 con 0x7f3d640b3420 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.719+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d640686f0 con 0x7f3d640b3420 2024-01-15T18:09:28.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.720+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3d5401e460 con 0x7f3d640b3420 2024-01-15T18:09:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.721+0000 7f3d5aa00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3c06be90 0x7f3d3c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:28.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.721+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f3d54088ed0 con 0x7f3d640b3420 2024-01-15T18:09:28.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.722+0000 7f3d5be00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3c06be90 0x7f3d3c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:28.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.723+0000 7f3d5be00640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3c06be90 0x7f3d3c06e350 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c036000 tx=0x7f3d4c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:28.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.726+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d54055590 con 0x7f3d640b3420 2024-01-15T18:09:28.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:28.957+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47161", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d64070ed0 con 0x7f3d3c06be90 2024-01-15T18:09:28.973 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:28.969+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:28.973 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:28.969+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:28.973 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:28.969+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:28.973 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:28.969+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:28.973 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:28.969+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:29.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.001+0000 7f3d5aa00640 1 -- 172.21.15.2:0/356767573 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f3d4c031a10 con 0x7f3d3c06be90 2024-01-15T18:09:29.005 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:29.005 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:29.006 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:09:29.006 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:29.006 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:29.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3c06be90 msgr2=0x7f3d3c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3c06be90 0x7f3d3c06e350 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c036000 tx=0x7f3d4c03c040 comp rx=0 tx=0).stop 2024-01-15T18:09:29.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 msgr2=0x7f3d64166160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d64166160 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f3d5400d830 tx=0x7f3d5400dcf0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 shutdown_connections 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3c06be90 0x7f3d3c06e350 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d641439f0 0x7f3d64167900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d640ccc70 0x7f3d640adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 --2- 172.21.15.2:0/356767573 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d640b3420 0x7f3d64166160 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 >> 172.21.15.2:0/356767573 conn(0x7f3d640b45f0 msgr2=0x7f3d640afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 shutdown_connections 2024-01-15T18:09:29.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.004+0000 7f3d69600640 1 -- 172.21.15.2:0/356767573 wait complete. 2024-01-15T18:09:29.028 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:29.029 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47162 2024-01-15T18:09:29.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.248+0000 7f92dee00640 1 Processor -- start 2024-01-15T18:09:29.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.248+0000 7f92dee00640 1 -- start start 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.250+0000 7f92dee00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f92d806d000 0x7f92d8165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.250+0000 7f92dee00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d80ce230 con 0x7f92d8144c00 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.250+0000 7f92dee00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92d80ce510 con 0x7f92d815c450 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.250+0000 7f92dee00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92d80ce7f0 con 0x7f92d806d000 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.250+0000 7f92d7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f92d8144c00 0x7f92d80cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60322/0 (socket says 172.21.15.2:60322) 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.250+0000 7f92d7e00640 1 -- 172.21.15.2:0/2388329301 learned_addr learned my addr 172.21.15.2:0/2388329301 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:29.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.251+0000 7f92dce00640 1 --2- 172.21.15.2:0/2388329301 >> v2:172.21.15.2:3300/0 conn(0x7f92d806d000 0x7f92d8165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.251+0000 7f92d7400640 1 -- 172.21.15.2:0/2388329301 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183132926 0 0) 0x7f92d80ce230 con 0x7f92d8144c00 2024-01-15T18:09:29.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.251+0000 7f92d7400640 1 -- 172.21.15.2:0/2388329301 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b8003520 con 0x7f92d8144c00 2024-01-15T18:09:29.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.251+0000 7f92dce00640 1 -- 172.21.15.2:0/2388329301 >> v1:172.21.15.137:6789/0 conn(0x7f92d815c450 legacy=0x7f92d806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.251+0000 7f92dce00640 1 -- 172.21.15.2:0/2388329301 >> v1:172.21.15.2:6789/0 conn(0x7f92d8144c00 legacy=0x7f92d80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.251+0000 7f92dce00640 1 -- 172.21.15.2:0/2388329301 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d80705f0 con 0x7f92d806d000 2024-01-15T18:09:29.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.253+0000 7f92dce00640 1 --2- 172.21.15.2:0/2388329301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8165d60 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f92d00097a0 tx=0x7f92d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2fa6493dddb938f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:29.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.253+0000 7f92d7400640 1 -- 172.21.15.2:0/2388329301 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0030610 con 0x7f92d806d000 2024-01-15T18:09:29.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.253+0000 7f92d7400640 1 -- 172.21.15.2:0/2388329301 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d0030770 con 0x7f92d806d000 2024-01-15T18:09:29.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.253+0000 7f92d7400640 1 -- 172.21.15.2:0/2388329301 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d0030b00 con 0x7f92d806d000 2024-01-15T18:09:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 -- 172.21.15.2:0/2388329301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 msgr2=0x7f92d8165d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 --2- 172.21.15.2:0/2388329301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8165d60 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f92d00097a0 tx=0x7f92d002b960 comp rx=0 tx=0).stop 2024-01-15T18:09:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 -- 172.21.15.2:0/2388329301 shutdown_connections 2024-01-15T18:09:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 --2- 172.21.15.2:0/2388329301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8165d60 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 -- 172.21.15.2:0/2388329301 >> 172.21.15.2:0/2388329301 conn(0x7f92d80b45f0 msgr2=0x7f92d80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:29.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 -- 172.21.15.2:0/2388329301 shutdown_connections 2024-01-15T18:09:29.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.254+0000 7f92dee00640 1 -- 172.21.15.2:0/2388329301 wait complete. 2024-01-15T18:09:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.256+0000 7f92dee00640 1 Processor -- start 2024-01-15T18:09:29.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.256+0000 7f92dee00640 1 -- start start 2024-01-15T18:09:29.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.257+0000 7f92dee00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.257+0000 7f92dee00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d8144c00 0x7f92d80b1540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.257+0000 7f92dee00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d815c450 0x7f92d8138610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.257+0000 7f92dee00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d80b1a80 con 0x7f92d806d000 2024-01-15T18:09:29.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.257+0000 7f92dee00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8140f60 con 0x7f92d8144c00 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.257+0000 7f92dee00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d8141200 con 0x7f92d815c450 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8142740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60302/0 (socket says 172.21.15.2:60302) 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 -- 172.21.15.2:0/1028070969 learned_addr learned my addr 172.21.15.2:0/1028070969 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92dce00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d815c450 0x7f92d8138610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 -- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d815c450 msgr2=0x7f92d8138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92cf200640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d8144c00 0x7f92d80b1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d815c450 0x7f92d8138610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 -- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d8144c00 msgr2=0x7f92d80b1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d8144c00 0x7f92d80b1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92d7e00640 1 -- 172.21.15.2:0/1028070969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d0009400 con 0x7f92d806d000 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.258+0000 7f92cf200640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d8144c00 0x7f92d80b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.259+0000 7f92d7e00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8142740 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f92c800cbc0 tx=0x7f92c800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:29.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.259+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c800ad40 con 0x7f92d806d000 2024-01-15T18:09:29.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.259+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c8012400 con 0x7f92d806d000 2024-01-15T18:09:29.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.259+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d81414e0 con 0x7f92d806d000 2024-01-15T18:09:29.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.259+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c800aea0 con 0x7f92d806d000 2024-01-15T18:09:29.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.259+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d8141aa0 con 0x7f92d806d000 2024-01-15T18:09:29.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.260+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d8141cc0 con 0x7f92d806d000 2024-01-15T18:09:29.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.262+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f92c8007070 con 0x7f92d806d000 2024-01-15T18:09:29.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.262+0000 7f92d4c00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92b006be90 0x7f92b006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.263+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f92c8088710 con 0x7f92d806d000 2024-01-15T18:09:29.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.263+0000 7f92cf200640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92b006be90 0x7f92b006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.264+0000 7f92cf200640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92b006be90 0x7f92b006e350 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f92d8165b90 tx=0x7f92c0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:29.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.268+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92c8054e20 con 0x7f92d806d000 2024-01-15T18:09:29.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.499+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47162", "target": ["mon-mgr", ""]}) v1 -- 0x7f92d80ae990 con 0x7f92b006be90 2024-01-15T18:09:29.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.522+0000 7f92d4c00640 1 -- 172.21.15.2:0/1028070969 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f92d80ae990 con 0x7f92b006be90 2024-01-15T18:09:29.526 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:29.526 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:29.526 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:09:29.527 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:29.527 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:29.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92b006be90 msgr2=0x7f92b006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92b006be90 0x7f92b006e350 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f92d8165b90 tx=0x7f92c0009040 comp rx=0 tx=0).stop 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 msgr2=0x7f92d8142740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8142740 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f92c800cbc0 tx=0x7f92c800a430 comp rx=0 tx=0).stop 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 shutdown_connections 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92b006be90 0x7f92b006e350 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d815c450 0x7f92d8138610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d8144c00 0x7f92d80b1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 --2- 172.21.15.2:0/1028070969 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d806d000 0x7f92d8142740 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.524+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 >> 172.21.15.2:0/1028070969 conn(0x7f92d80b45f0 msgr2=0x7f92d8154500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.525+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 shutdown_connections 2024-01-15T18:09:29.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.525+0000 7f92dee00640 1 -- 172.21.15.2:0/1028070969 wait complete. 2024-01-15T18:09:29.549 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:29.549 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47163 2024-01-15T18:09:29.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.777+0000 7f96d6400640 1 Processor -- start 2024-01-15T18:09:29.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.777+0000 7f96d6400640 1 -- start start 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d6400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f96d00b1500 0x7f96d00b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00d1930 0x7f96d00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d6400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f96d00cdd00 0x7f96d006b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d6400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d00cfa50 con 0x7f96d00b1500 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d6400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96d00b18e0 con 0x7f96d00cdd00 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00d02b0 con 0x7f96d00d1930 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f96d00b1500 0x7f96d00b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f96d00b1500 0x7f96d00b1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60314/0 (socket says 172.21.15.2:60314) 2024-01-15T18:09:29.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.778+0000 7f96d5000640 1 -- 172.21.15.2:0/600508783 learned_addr learned my addr 172.21.15.2:0/600508783 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:29.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.779+0000 7f96d5a00640 1 --2- 172.21.15.2:0/600508783 >> v2:172.21.15.137:3300/0 conn(0x7f96d00cdd00 0x7f96d006b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.779+0000 7f96d5000640 1 -- 172.21.15.2:0/600508783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00d1930 msgr2=0x7f96d00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:29.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.779+0000 7f96d5000640 1 --2- 172.21.15.2:0/600508783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00d1930 0x7f96d00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.779+0000 7f96d5000640 1 -- 172.21.15.2:0/600508783 >> v2:172.21.15.137:3300/0 conn(0x7f96d00cdd00 msgr2=0x7f96d006b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:29.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.779+0000 7f96d5000640 1 --2- 172.21.15.2:0/600508783 >> v2:172.21.15.137:3300/0 conn(0x7f96d00cdd00 0x7f96d006b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.779+0000 7f96d5000640 1 -- 172.21.15.2:0/600508783 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d00ce160 con 0x7f96d00b1500 2024-01-15T18:09:29.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.780+0000 7f96d5000640 1 --2- 172.21.15.2:0/600508783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00b1500 0x7f96d00b1f20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f96d0071d00 tx=0x7f96c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1513d4dc3da8b483 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:29.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.780+0000 7f96cf400640 1 -- 172.21.15.2:0/600508783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0031870 con 0x7f96d00b1500 2024-01-15T18:09:29.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96cf400640 1 -- 172.21.15.2:0/600508783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c00319d0 con 0x7f96d00b1500 2024-01-15T18:09:29.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96cf400640 1 -- 172.21.15.2:0/600508783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0031d60 con 0x7f96d00b1500 2024-01-15T18:09:29.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 -- 172.21.15.2:0/600508783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00b1500 msgr2=0x7f96d00b1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 --2- 172.21.15.2:0/600508783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00b1500 0x7f96d00b1f20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f96d0071d00 tx=0x7f96c002b910 comp rx=0 tx=0).stop 2024-01-15T18:09:29.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 -- 172.21.15.2:0/600508783 shutdown_connections 2024-01-15T18:09:29.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 --2- 172.21.15.2:0/600508783 >> v2:172.21.15.137:3300/0 conn(0x7f96d00cdd00 0x7f96d006b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 --2- 172.21.15.2:0/600508783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00d1930 0x7f96d00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 --2- 172.21.15.2:0/600508783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00b1500 0x7f96d00b1f20 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 -- 172.21.15.2:0/600508783 >> 172.21.15.2:0/600508783 conn(0x7f96d00d0e10 msgr2=0x7f96d00d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:29.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 -- 172.21.15.2:0/600508783 shutdown_connections 2024-01-15T18:09:29.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.781+0000 7f96d6400640 1 -- 172.21.15.2:0/600508783 wait complete. 2024-01-15T18:09:29.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.783+0000 7f96d6400640 1 Processor -- start 2024-01-15T18:09:29.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.783+0000 7f96d6400640 1 -- start start 2024-01-15T18:09:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.784+0000 7f96d6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00b1500 0x7f96d015c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.784+0000 7f96d6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96d00cdd00 0x7f96d01fa8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.784+0000 7f96d6400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 0x7f96d01fae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.784+0000 7f96d6400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00c4780 con 0x7f96d00d1930 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.784+0000 7f96d6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d00c4fe0 con 0x7f96d00cdd00 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.784+0000 7f96d6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96d015c9c0 con 0x7f96d00b1500 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 0x7f96d01fae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 0x7f96d01fae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60318/0 (socket says 172.21.15.2:60318) 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 -- 172.21.15.2:0/1226295858 learned_addr learned my addr 172.21.15.2:0/1226295858 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5000640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00b1500 0x7f96d015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96cfe00640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96d00cdd00 0x7f96d01fa8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 -- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00b1500 msgr2=0x7f96d015c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00b1500 0x7f96d015c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 -- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96d00cdd00 msgr2=0x7f96d01fa8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96d00cdd00 0x7f96d01fa8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96d5a00640 1 -- 172.21.15.2:0/1226295858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c00094d0 con 0x7f96d00d1930 2024-01-15T18:09:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.785+0000 7f96cfe00640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96d00cdd00 0x7f96d01fa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.786+0000 7f96d5a00640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 0x7f96d01fae10 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f96c8007e40 tx=0x7f96c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.786+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c8014620 con 0x7f96d00d1930 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.786+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c800ee90 con 0x7f96d00d1930 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.786+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d009ebe0 con 0x7f96d00d1930 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.786+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c80149e0 con 0x7f96d00d1930 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.786+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d00c4990 con 0x7f96d00d1930 2024-01-15T18:09:29.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.787+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d00c5150 con 0x7f96d00d1930 2024-01-15T18:09:29.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.788+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f96c8014b40 con 0x7f96d00d1930 2024-01-15T18:09:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.790+0000 7f96ccc00640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96a806bee0 0x7f96a806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.790+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f96c8088f90 con 0x7f96d00d1930 2024-01-15T18:09:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.790+0000 7f96d5000640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96a806bee0 0x7f96a806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:29.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.792+0000 7f96d5000640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96a806bee0 0x7f96a806e3a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f96c0036000 tx=0x7f96c003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:29.794+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96c8055850 con 0x7f96d00d1930 2024-01-15T18:09:30.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.024+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47163", "target": ["mon-mgr", ""]}) v1 -- 0x7f96d00cff40 con 0x7f96a806bee0 2024-01-15T18:09:30.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.050+0000 7f96ccc00640 1 -- 172.21.15.2:0/1226295858 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f96c0031bc0 con 0x7f96a806bee0 2024-01-15T18:09:30.054 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:30.054 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:30.054 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:09:30.054 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:30.054 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:30.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96a806bee0 msgr2=0x7f96a806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96a806bee0 0x7f96a806e3a0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f96c0036000 tx=0x7f96c003c040 comp rx=0 tx=0).stop 2024-01-15T18:09:30.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 msgr2=0x7f96d01fae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 0x7f96d01fae10 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f96c8007e40 tx=0x7f96c800e4b0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 shutdown_connections 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f96a806bee0 0x7f96a806e3a0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f96d00d1930 0x7f96d01fae10 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f96d00cdd00 0x7f96d01fa8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 --2- 172.21.15.2:0/1226295858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f96d00b1500 0x7f96d015c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 >> 172.21.15.2:0/1226295858 conn(0x7f96d00d0e10 msgr2=0x7f96d00adbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 shutdown_connections 2024-01-15T18:09:30.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.052+0000 7f96d6400640 1 -- 172.21.15.2:0/1226295858 wait complete. 2024-01-15T18:09:30.074 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:30.074 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47164 2024-01-15T18:09:30.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.291+0000 7f4d25c00640 1 Processor -- start 2024-01-15T18:09:30.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.292+0000 7f4d25c00640 1 -- start start 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d25c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d200b4cf0 0x7f4d200b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d25c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d200d12b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d25c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2015d8b0 con 0x7f4d200d17f0 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d25c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d200b72f0 con 0x7f4d200b4cf0 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d25c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200d2980 con 0x7f4d200d2e10 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d1ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d200b4cf0 0x7f4d200b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d1ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d200b4cf0 0x7f4d200b50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60326/0 (socket says 172.21.15.2:60326) 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.293+0000 7f4d1ea00640 1 -- 172.21.15.2:0/635566541 learned_addr learned my addr 172.21.15.2:0/635566541 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:30.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.294+0000 7f4d1e000640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d200d12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.294+0000 7f4d1ea00640 1 -- 172.21.15.2:0/635566541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 msgr2=0x7f4d200d12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.294+0000 7f4d1ea00640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d200d12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.294+0000 7f4d1ea00640 1 -- 172.21.15.2:0/635566541 >> v1:172.21.15.137:6789/0 conn(0x7f4d200d17f0 legacy=0x7f4d2006b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.294+0000 7f4d1ea00640 1 -- 172.21.15.2:0/635566541 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d200b5610 con 0x7f4d200b4cf0 2024-01-15T18:09:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.294+0000 7f4d1e000640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d200d12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:30.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.295+0000 7f4d1ea00640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d200b50d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009870 tx=0x7f4d1002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ae510f15ded8c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.295+0000 7f4d1d600640 1 -- 172.21.15.2:0/635566541 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10030840 con 0x7f4d200b4cf0 2024-01-15T18:09:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.295+0000 7f4d1d600640 1 -- 172.21.15.2:0/635566541 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d100309a0 con 0x7f4d200b4cf0 2024-01-15T18:09:30.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.295+0000 7f4d1d600640 1 -- 172.21.15.2:0/635566541 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d10030d30 con 0x7f4d200b4cf0 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 -- 172.21.15.2:0/635566541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 msgr2=0x7f4d200b50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d200b50d0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009870 tx=0x7f4d1002b980 comp rx=0 tx=0).stop 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 -- 172.21.15.2:0/635566541 shutdown_connections 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d200d12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 --2- 172.21.15.2:0/635566541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d200b50d0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 -- 172.21.15.2:0/635566541 >> 172.21.15.2:0/635566541 conn(0x7f4d20072210 msgr2=0x7f4d200cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 -- 172.21.15.2:0/635566541 shutdown_connections 2024-01-15T18:09:30.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.296+0000 7f4d25c00640 1 -- 172.21.15.2:0/635566541 wait complete. 2024-01-15T18:09:30.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.298+0000 7f4d25c00640 1 Processor -- start 2024-01-15T18:09:30.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.298+0000 7f4d25c00640 1 -- start start 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d25c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d20154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d25c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d200d17f0 0x7f4d20155970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d25c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d20155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d25c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2015e9c0 con 0x7f4d200b4cf0 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d25c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2015eca0 con 0x7f4d200d17f0 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d25c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2015ef80 con 0x7f4d200d2e10 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d20154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d20154940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60328/0 (socket says 172.21.15.2:60328) 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 -- 172.21.15.2:0/3553243552 learned_addr learned my addr 172.21.15.2:0/3553243552 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:30.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1e000640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d200d17f0 0x7f4d20155970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 -- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 msgr2=0x7f4d20155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d20155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 -- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d200d17f0 msgr2=0x7f4d20155970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d200d17f0 0x7f4d20155970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1ea00640 1 -- 172.21.15.2:0/3553243552 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d100094d0 con 0x7f4d200b4cf0 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.299+0000 7f4d1e000640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d200d17f0 0x7f4d20155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:30.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.300+0000 7f4d1ea00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d20154940 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009870 tx=0x7f4d10004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.300+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100307f0 con 0x7f4d200b4cf0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.300+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d10043400 con 0x7f4d200b4cf0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.300+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d100437b0 con 0x7f4d200b4cf0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.300+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2015f260 con 0x7f4d200b4cf0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.300+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d20156580 con 0x7f4d200b4cf0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.301+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2015f780 con 0x7f4d200b4cf0 2024-01-15T18:09:30.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.302+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4d10043910 con 0x7f4d200b4cf0 2024-01-15T18:09:30.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.303+0000 7f4d16a00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cf806be90 0x7f4cf806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.303+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4d1003e070 con 0x7f4d200b4cf0 2024-01-15T18:09:30.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.304+0000 7f4d1e000640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cf806be90 0x7f4cf806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.305+0000 7f4d1e000640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cf806be90 0x7f4cf806e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4d08006fd0 tx=0x7f4d08008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:30.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.308+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d2015f780 con 0x7f4d200b4cf0 2024-01-15T18:09:30.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.538+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d200d2330 con 0x7f4cf806be90 2024-01-15T18:09:30.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.564+0000 7f4d16a00640 1 -- 172.21.15.2:0/3553243552 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f4d0800c040 con 0x7f4cf806be90 2024-01-15T18:09:30.569 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:09:30.569 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:30.569 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:30.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cf806be90 msgr2=0x7f4cf806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cf806be90 0x7f4cf806e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4d08006fd0 tx=0x7f4d08008040 comp rx=0 tx=0).stop 2024-01-15T18:09:30.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 msgr2=0x7f4d20154940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d20154940 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f4d10009870 tx=0x7f4d10004730 comp rx=0 tx=0).stop 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 shutdown_connections 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cf806be90 0x7f4cf806e350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d200d2e10 0x7f4d20155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d200d17f0 0x7f4d20155970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 --2- 172.21.15.2:0/3553243552 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d200b4cf0 0x7f4d20154940 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 >> 172.21.15.2:0/3553243552 conn(0x7f4d20072210 msgr2=0x7f4d200cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 shutdown_connections 2024-01-15T18:09:30.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.567+0000 7f4d25c00640 1 -- 172.21.15.2:0/3553243552 wait complete. 2024-01-15T18:09:30.591 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:30.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47165 2024-01-15T18:09:30.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.806+0000 7f6c00e00640 1 Processor -- start 2024-01-15T18:09:30.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.806+0000 7f6c00e00640 1 -- start start 2024-01-15T18:09:30.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.807+0000 7f6c00e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6bfc15c450 0x7f6bfc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.807+0000 7f6c00e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6bfc06ce30 0x7f6bfc070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6c00e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bfc06d290 con 0x7f6bfc144c00 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6c00e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bfc0ce230 con 0x7f6bfc06ce30 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6c00e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bfc0ce510 con 0x7f6bfc15c450 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6bfc06ce30 0x7f6bfc070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6bfc06ce30 0x7f6bfc070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60332/0 (socket says 172.21.15.2:60332) 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 -- 172.21.15.2:0/1181177141 learned_addr learned my addr 172.21.15.2:0/1181177141 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 -- 172.21.15.2:0/1181177141 >> v2:172.21.15.137:3300/0 conn(0x7f6bfc15c450 msgr2=0x7f6bfc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 --2- 172.21.15.2:0/1181177141 >> v2:172.21.15.137:3300/0 conn(0x7f6bfc15c450 0x7f6bfc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 -- 172.21.15.2:0/1181177141 >> v1:172.21.15.137:6789/0 conn(0x7f6bfc144c00 legacy=0x7f6bfc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:30.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.808+0000 7f6bfa800640 1 -- 172.21.15.2:0/1181177141 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc0ce7f0 con 0x7f6bfc06ce30 2024-01-15T18:09:30.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.809+0000 7f6bfa800640 1 --2- 172.21.15.2:0/1181177141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc06ce30 0x7f6bfc070b10 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc06f9f0 tx=0x7f6bf402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc5d99d6f8edbf8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:30.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.810+0000 7f6bf8a00640 1 -- 172.21.15.2:0/1181177141 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf402ca80 con 0x7f6bfc06ce30 2024-01-15T18:09:30.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.810+0000 7f6bf8a00640 1 -- 172.21.15.2:0/1181177141 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf402cbe0 con 0x7f6bfc06ce30 2024-01-15T18:09:30.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.810+0000 7f6bf8a00640 1 -- 172.21.15.2:0/1181177141 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40343f0 con 0x7f6bfc06ce30 2024-01-15T18:09:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 -- 172.21.15.2:0/1181177141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc06ce30 msgr2=0x7f6bfc070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 --2- 172.21.15.2:0/1181177141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc06ce30 0x7f6bfc070b10 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc06f9f0 tx=0x7f6bf402bee0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 -- 172.21.15.2:0/1181177141 shutdown_connections 2024-01-15T18:09:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 --2- 172.21.15.2:0/1181177141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc06ce30 0x7f6bfc070b10 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 --2- 172.21.15.2:0/1181177141 >> v2:172.21.15.137:3300/0 conn(0x7f6bfc15c450 0x7f6bfc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 -- 172.21.15.2:0/1181177141 >> 172.21.15.2:0/1181177141 conn(0x7f6bfc0b45f0 msgr2=0x7f6bfc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:30.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 -- 172.21.15.2:0/1181177141 shutdown_connections 2024-01-15T18:09:30.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.811+0000 7f6c00e00640 1 -- 172.21.15.2:0/1181177141 wait complete. 2024-01-15T18:09:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.813+0000 7f6c00e00640 1 Processor -- start 2024-01-15T18:09:30.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.813+0000 7f6c00e00640 1 -- start start 2024-01-15T18:09:30.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.814+0000 7f6c00e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6bfc06ce30 0x7f6bfc142b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.814+0000 7f6c00e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 0x7f6bfc13a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.814+0000 7f6c00e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6bfc15c450 0x7f6bfc0b0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.814+0000 7f6c00e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc13b940 con 0x7f6bfc144c00 2024-01-15T18:09:30.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.814+0000 7f6c00e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc13acc0 con 0x7f6bfc15c450 2024-01-15T18:09:30.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.814+0000 7f6c00e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bfc13afa0 con 0x7f6bfc06ce30 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.815+0000 7f6bf9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 0x7f6bfc13a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.815+0000 7f6bf9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 0x7f6bfc13a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60342/0 (socket says 172.21.15.2:60342) 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.815+0000 7f6bf9400640 1 -- 172.21.15.2:0/798528927 learned_addr learned my addr 172.21.15.2:0/798528927 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.815+0000 7f6bf9e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6bfc06ce30 0x7f6bfc142b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.815+0000 7f6bfa800640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6bfc15c450 0x7f6bfc0b0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf9400640 1 -- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6bfc06ce30 msgr2=0x7f6bfc142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf9400640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6bfc06ce30 0x7f6bfc142b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf9400640 1 -- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6bfc15c450 msgr2=0x7f6bfc0b0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf9400640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6bfc15c450 0x7f6bfc0b0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf9400640 1 -- 172.21.15.2:0/798528927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf4009b30 con 0x7f6bfc144c00 2024-01-15T18:09:30.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf9400640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 0x7f6bfc13a3e0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b410 tx=0x7f6be400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:30.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4011830 con 0x7f6bfc144c00 2024-01-15T18:09:30.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be40119b0 con 0x7f6bfc144c00 2024-01-15T18:09:30.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bfc13b280 con 0x7f6bfc144c00 2024-01-15T18:09:30.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be4011d60 con 0x7f6bfc144c00 2024-01-15T18:09:30.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.816+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bfc13a920 con 0x7f6bfc144c00 2024-01-15T18:09:30.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.817+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bfc13ba80 con 0x7f6bfc144c00 2024-01-15T18:09:30.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.819+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6be401a400 con 0x7f6bfc144c00 2024-01-15T18:09:30.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.820+0000 7f6bf2000640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bd406bf20 0x7f6bd406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:30.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.820+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6be4088860 con 0x7f6bfc144c00 2024-01-15T18:09:30.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.820+0000 7f6bf9e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bd406bf20 0x7f6bd406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:30.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.822+0000 7f6bf9e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bd406bf20 0x7f6bd406e3e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc15a990 tx=0x7f6bec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:30.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:30.824+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6bfc13ba80 con 0x7f6bfc144c00 2024-01-15T18:09:31.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.056+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47165", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bfc0cead0 con 0x7f6bd406bf20 2024-01-15T18:09:31.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.081+0000 7f6bf2000640 1 -- 172.21.15.2:0/798528927 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f6bec001440 con 0x7f6bd406bf20 2024-01-15T18:09:31.085 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:31.086 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:31.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.083+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bd406bf20 msgr2=0x7f6bd406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bd406bf20 0x7f6bd406e3e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc15a990 tx=0x7f6bec009040 comp rx=0 tx=0).stop 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 msgr2=0x7f6bfc13a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 0x7f6bfc13a3e0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f6be400b410 tx=0x7f6be400b8d0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 shutdown_connections 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6bd406bf20 0x7f6bd406e3e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6bfc15c450 0x7f6bfc0b0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6bfc144c00 0x7f6bfc13a3e0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 --2- 172.21.15.2:0/798528927 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6bfc06ce30 0x7f6bfc142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 >> 172.21.15.2:0/798528927 conn(0x7f6bfc0b45f0 msgr2=0x7f6bfc143cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 shutdown_connections 2024-01-15T18:09:31.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.084+0000 7f6c00e00640 1 -- 172.21.15.2:0/798528927 wait complete. 2024-01-15T18:09:31.108 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:31.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47166 2024-01-15T18:09:31.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.330+0000 7fec93400640 1 Processor -- start 2024-01-15T18:09:31.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.330+0000 7fec93400640 1 -- start start 2024-01-15T18:09:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.332+0000 7fec93400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 0x7fec8c06c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.332+0000 7fec93400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 0x7fec8c06c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.332+0000 7fec93400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c071090 con 0x7fec8c143d20 2024-01-15T18:09:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.332+0000 7fec93400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8c071430 con 0x7fec8c0720d0 2024-01-15T18:09:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.332+0000 7fec93400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c144100 con 0x7fec8c15f260 2024-01-15T18:09:31.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.333+0000 7fec91600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 0x7fec8c06c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.333+0000 7fec92a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 0x7fec8c06c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.333+0000 7fec92a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 0x7fec8c06c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:41530/0 (socket says 172.21.15.2:41530) 2024-01-15T18:09:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.333+0000 7fec91600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 0x7fec8c06c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:54142/0 (socket says 172.21.15.2:54142) 2024-01-15T18:09:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.333+0000 7fec92000640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fec8c143d20 0x7fec8c15eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:57322/0 (socket says 172.21.15.2:57322) 2024-01-15T18:09:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.333+0000 7fec91600640 1 -- 172.21.15.2:0/3986872485 learned_addr learned my addr 172.21.15.2:0/3986872485 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:31.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.334+0000 7fec91600640 1 -- 172.21.15.2:0/3986872485 >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 msgr2=0x7fec8c06c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:31.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.334+0000 7fec91600640 1 --2- 172.21.15.2:0/3986872485 >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 0x7fec8c06c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.334+0000 7fec91600640 1 -- 172.21.15.2:0/3986872485 >> v1:172.21.15.137:6789/0 conn(0x7fec8c143d20 legacy=0x7fec8c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.334+0000 7fec91600640 1 -- 172.21.15.2:0/3986872485 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c144560 con 0x7fec8c15f260 2024-01-15T18:09:31.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.334+0000 7fec92a00640 1 --2- 172.21.15.2:0/3986872485 >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 0x7fec8c06c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:31.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.336+0000 7fec91600640 1 --2- 172.21.15.2:0/3986872485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 0x7fec8c06c150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec74009870 tx=0x7fec7402b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27f29e85b340cf5d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.336+0000 7fec90c00640 1 -- 172.21.15.2:0/3986872485 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740317c0 con 0x7fec8c15f260 2024-01-15T18:09:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.336+0000 7fec90c00640 1 -- 172.21.15.2:0/3986872485 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec74031920 con 0x7fec8c15f260 2024-01-15T18:09:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.336+0000 7fec90c00640 1 -- 172.21.15.2:0/3986872485 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74031cb0 con 0x7fec8c15f260 2024-01-15T18:09:31.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 -- 172.21.15.2:0/3986872485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 msgr2=0x7fec8c06c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 --2- 172.21.15.2:0/3986872485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 0x7fec8c06c150 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fec74009870 tx=0x7fec7402b860 comp rx=0 tx=0).stop 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 -- 172.21.15.2:0/3986872485 shutdown_connections 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 --2- 172.21.15.2:0/3986872485 >> v2:172.21.15.137:3300/0 conn(0x7fec8c0720d0 0x7fec8c06c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 --2- 172.21.15.2:0/3986872485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c15f260 0x7fec8c06c150 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 -- 172.21.15.2:0/3986872485 >> 172.21.15.2:0/3986872485 conn(0x7fec8c0b0840 msgr2=0x7fec8c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 -- 172.21.15.2:0/3986872485 shutdown_connections 2024-01-15T18:09:31.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.337+0000 7fec93400640 1 -- 172.21.15.2:0/3986872485 wait complete. 2024-01-15T18:09:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.339+0000 7fec93400640 1 Processor -- start 2024-01-15T18:09:31.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.339+0000 7fec93400640 1 -- start start 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec93400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c0720d0 0x7fec8c15d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec93400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec8c143d20 0x7fec8c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec93400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 0x7fec8c15bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec93400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c13d140 con 0x7fec8c15f260 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec93400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0cc6a0 con 0x7fec8c143d20 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec93400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c15b5a0 con 0x7fec8c0720d0 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.340+0000 7fec92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 0x7fec8c15bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 0x7fec8c15bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60354/0 (socket says 172.21.15.2:60354) 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 -- 172.21.15.2:0/525828440 learned_addr learned my addr 172.21.15.2:0/525828440 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:31.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec91600640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec8c143d20 0x7fec8c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92000640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c0720d0 0x7fec8c15d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 -- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c0720d0 msgr2=0x7fec8c15d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c0720d0 0x7fec8c15d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 -- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec8c143d20 msgr2=0x7fec8c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec8c143d20 0x7fec8c0cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 -- 172.21.15.2:0/525828440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84009400 con 0x7fec8c15f260 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.341+0000 7fec92a00640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 0x7fec8c15bd10 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e880 tx=0x7fec8400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.342+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec84009660 con 0x7fec8c15f260 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.342+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec84004020 con 0x7fec8c15f260 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.342+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec840043b0 con 0x7fec8c15f260 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.342+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec740094d0 con 0x7fec8c15f260 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.342+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c15b880 con 0x7fec8c15f260 2024-01-15T18:09:31.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.343+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c0709a0 con 0x7fec8c15f260 2024-01-15T18:09:31.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.344+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fec8400c940 con 0x7fec8c15f260 2024-01-15T18:09:31.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.345+0000 7fec8a000640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6406bf50 0x7fec6406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.345+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fec84088630 con 0x7fec8c15f260 2024-01-15T18:09:31.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.346+0000 7fec92000640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6406bf50 0x7fec6406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.347+0000 7fec92000640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6406bf50 0x7fec6406e410 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0012e0 tx=0x7fec7c005b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:31.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.350+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec84054cd0 con 0x7fec8c15f260 2024-01-15T18:09:31.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.580+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47166", "target": ["mon-mgr", ""]}) v1 -- 0x7fec8c159430 con 0x7fec6406bf50 2024-01-15T18:09:31.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.599+0000 7fec8a000640 1 -- 172.21.15.2:0/525828440 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fec8c159430 con 0x7fec6406bf50 2024-01-15T18:09:31.603 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:31.604 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6406bf50 msgr2=0x7fec6406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6406bf50 0x7fec6406e410 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0012e0 tx=0x7fec7c005b70 comp rx=0 tx=0).stop 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 msgr2=0x7fec8c15bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 0x7fec8c15bd10 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e880 tx=0x7fec8400ed40 comp rx=0 tx=0).stop 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 shutdown_connections 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec6406bf50 0x7fec6406e410 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec8c15f260 0x7fec8c15bd10 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec8c143d20 0x7fec8c0cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 --2- 172.21.15.2:0/525828440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec8c0720d0 0x7fec8c15d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 >> 172.21.15.2:0/525828440 conn(0x7fec8c0b0840 msgr2=0x7fec8c06c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 shutdown_connections 2024-01-15T18:09:31.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.602+0000 7fec93400640 1 -- 172.21.15.2:0/525828440 wait complete. 2024-01-15T18:09:31.627 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:31.627 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47167 2024-01-15T18:09:31.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.851+0000 7f844a400640 1 Processor -- start 2024-01-15T18:09:31.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.852+0000 7f844a400640 1 -- start start 2024-01-15T18:09:31.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.853+0000 7f844a400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f84441439f0 0x7f84440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.853+0000 7f844a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.853+0000 7f844a400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f844415dd70 con 0x7f84440ccc70 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.853+0000 7f844a400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84440b3880 con 0x7f84441439f0 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.853+0000 7f844a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84440711b0 con 0x7f84440b3420 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.854+0000 7f8443400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f84441439f0 0x7f84440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.854+0000 7f8442a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.854+0000 7f8443e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f84440ccc70 0x7f844406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:57324/0 (socket says 172.21.15.2:57324) 2024-01-15T18:09:31.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.854+0000 7f8443e00640 1 -- 172.21.15.2:0/1497131502 learned_addr learned my addr 172.21.15.2:0/1497131502 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:31.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.855+0000 7f8443400640 1 -- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 msgr2=0x7f84440cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:31.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.855+0000 7f8443400640 1 --2- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.855+0000 7f8443400640 1 -- 172.21.15.2:0/1497131502 >> v1:172.21.15.137:6789/0 conn(0x7f84440ccc70 legacy=0x7f844406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.855+0000 7f8443400640 1 -- 172.21.15.2:0/1497131502 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444143dd0 con 0x7f84441439f0 2024-01-15T18:09:31.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.857+0000 7f8443400640 1 --2- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84441439f0 0x7f84440702c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8434009d90 tx=0x7f843402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6efead5b6855a90c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.857+0000 7f8442000640 1 -- 172.21.15.2:0/1497131502 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843402cc50 con 0x7f84441439f0 2024-01-15T18:09:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.857+0000 7f8442000640 1 -- 172.21.15.2:0/1497131502 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843402cdb0 con 0x7f84441439f0 2024-01-15T18:09:31.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.857+0000 7f8442000640 1 -- 172.21.15.2:0/1497131502 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8434034660 con 0x7f84441439f0 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 -- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84441439f0 msgr2=0x7f84440702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 --2- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84441439f0 0x7f84440702c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8434009d90 tx=0x7f843402bea0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 -- 172.21.15.2:0/1497131502 shutdown_connections 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 --2- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 --2- 172.21.15.2:0/1497131502 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84441439f0 0x7f84440702c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 -- 172.21.15.2:0/1497131502 >> 172.21.15.2:0/1497131502 conn(0x7f84440b45f0 msgr2=0x7f84440b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 -- 172.21.15.2:0/1497131502 shutdown_connections 2024-01-15T18:09:31.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.858+0000 7f844a400640 1 -- 172.21.15.2:0/1497131502 wait complete. 2024-01-15T18:09:31.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.860+0000 7f844a400640 1 Processor -- start 2024-01-15T18:09:31.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.860+0000 7f844a400640 1 -- start start 2024-01-15T18:09:31.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.861+0000 7f844a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.861+0000 7f844a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84440ccc70 0x7f84441f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.861+0000 7f844a400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 0x7f84440aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.861+0000 7f844a400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84440c6320 con 0x7f84441439f0 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.861+0000 7f844a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84441f3910 con 0x7f84440ccc70 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.861+0000 7f844a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84440af600 con 0x7f84440b3420 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 0x7f84440aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 0x7f84440aeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60360/0 (socket says 172.21.15.2:60360) 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 -- 172.21.15.2:0/2440821623 learned_addr learned my addr 172.21.15.2:0/2440821623 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8442a00640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84440ccc70 0x7f84441f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 -- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 msgr2=0x7f84440c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 -- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84440ccc70 msgr2=0x7f84441f43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84440ccc70 0x7f84441f43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8443e00640 1 -- 172.21.15.2:0/2440821623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8434009a70 con 0x7f84441439f0 2024-01-15T18:09:31.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.862+0000 7f8442a00640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84440ccc70 0x7f84441f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:31.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.863+0000 7f8443e00640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 0x7f84440aeb60 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f843c00c3c0 tx=0x7f843c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:31.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.863+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c005ce0 con 0x7f84441439f0 2024-01-15T18:09:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.863+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843c005e40 con 0x7f84441439f0 2024-01-15T18:09:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.863+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843c0072f0 con 0x7f84441439f0 2024-01-15T18:09:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.863+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84440af8e0 con 0x7f84441439f0 2024-01-15T18:09:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.863+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84441f3ea0 con 0x7f84441439f0 2024-01-15T18:09:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.864+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84440686f0 con 0x7f84441439f0 2024-01-15T18:09:31.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.865+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f843c007510 con 0x7f84441439f0 2024-01-15T18:09:31.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.866+0000 7f843b400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f841c06bf20 0x7f841c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:31.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.866+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f843c087db0 con 0x7f84441439f0 2024-01-15T18:09:31.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.867+0000 7f8443400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f841c06bf20 0x7f841c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:31.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.868+0000 7f8443400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f841c06bf20 0x7f841c06e3e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8434036000 tx=0x7f8434030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:31.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:31.871+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f843c0543f0 con 0x7f84441439f0 2024-01-15T18:09:32.003 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:32.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:09:32.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.108+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47167", "target": ["mon-mgr", ""]}) v1 -- 0x7f8444070ed0 con 0x7f841c06bf20 2024-01-15T18:09:32.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.126+0000 7f843b400640 1 -- 172.21.15.2:0/2440821623 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f84340344c0 con 0x7f841c06bf20 2024-01-15T18:09:32.130 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:32.130 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:32.130 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:32.130 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:32.130 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:32.131 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:32.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f841c06bf20 msgr2=0x7f841c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f841c06bf20 0x7f841c06e3e0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8434036000 tx=0x7f8434030000 comp rx=0 tx=0).stop 2024-01-15T18:09:32.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 msgr2=0x7f84440aeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 0x7f84440aeb60 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f843c00c3c0 tx=0x7f843c00c880 comp rx=0 tx=0).stop 2024-01-15T18:09:32.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 shutdown_connections 2024-01-15T18:09:32.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f841c06bf20 0x7f841c06e3e0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f84441439f0 0x7f84440aeb60 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84440ccc70 0x7f84441f43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 --2- 172.21.15.2:0/2440821623 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84440b3420 0x7f84440c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.128+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 >> 172.21.15.2:0/2440821623 conn(0x7f84440b45f0 msgr2=0x7f844415d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.129+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 shutdown_connections 2024-01-15T18:09:32.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.129+0000 7f844a400640 1 -- 172.21.15.2:0/2440821623 wait complete. 2024-01-15T18:09:32.152 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:32.152 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47168 2024-01-15T18:09:32.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.417+0000 7f6a6d600640 1 Processor -- start 2024-01-15T18:09:32.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.417+0000 7f6a6d600640 1 -- start start 2024-01-15T18:09:32.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a6d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6a681439f0 0x7f6a680702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a6d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a680cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a6d600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6a680ccc70 0x7f6a6806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a6d600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6815dd70 con 0x7f6a681439f0 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a6d600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a680b3880 con 0x7f6a680ccc70 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a6d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a680711b0 con 0x7f6a680b3420 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a66a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6a681439f0 0x7f6a680702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a66a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6a681439f0 0x7f6a680702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60366/0 (socket says 172.21.15.2:60366) 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a66a00640 1 -- 172.21.15.2:0/2828268674 learned_addr learned my addr 172.21.15.2:0/2828268674 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:32.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.419+0000 7f6a66000640 1 --2- 172.21.15.2:0/2828268674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a680cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.420+0000 7f6a67400640 1 --2- 172.21.15.2:0/2828268674 >> v2:172.21.15.137:3300/0 conn(0x7f6a680ccc70 0x7f6a6806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.420+0000 7f6a66a00640 1 -- 172.21.15.2:0/2828268674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 msgr2=0x7f6a680cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.420+0000 7f6a66a00640 1 --2- 172.21.15.2:0/2828268674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a680cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.420+0000 7f6a66a00640 1 -- 172.21.15.2:0/2828268674 >> v2:172.21.15.137:3300/0 conn(0x7f6a680ccc70 msgr2=0x7f6a6806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.420+0000 7f6a66a00640 1 --2- 172.21.15.2:0/2828268674 >> v2:172.21.15.137:3300/0 conn(0x7f6a680ccc70 0x7f6a6806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.420+0000 7f6a66a00640 1 -- 172.21.15.2:0/2828268674 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a68143dd0 con 0x7f6a681439f0 2024-01-15T18:09:32.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.421+0000 7f6a66a00640 1 --2- 172.21.15.2:0/2828268674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a680702c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6a680b3110 tx=0x7f6a5802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b94b145afb3852f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.421+0000 7f6a65600640 1 -- 172.21.15.2:0/2828268674 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58031860 con 0x7f6a681439f0 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.421+0000 7f6a65600640 1 -- 172.21.15.2:0/2828268674 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a580319c0 con 0x7f6a681439f0 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.421+0000 7f6a65600640 1 -- 172.21.15.2:0/2828268674 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58031d50 con 0x7f6a681439f0 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 -- 172.21.15.2:0/2828268674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 msgr2=0x7f6a680702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 --2- 172.21.15.2:0/2828268674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a680702c0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f6a680b3110 tx=0x7f6a5802b900 comp rx=0 tx=0).stop 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 -- 172.21.15.2:0/2828268674 shutdown_connections 2024-01-15T18:09:32.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 --2- 172.21.15.2:0/2828268674 >> v2:172.21.15.137:3300/0 conn(0x7f6a680ccc70 0x7f6a6806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 --2- 172.21.15.2:0/2828268674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a680cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 --2- 172.21.15.2:0/2828268674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a680702c0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 -- 172.21.15.2:0/2828268674 >> 172.21.15.2:0/2828268674 conn(0x7f6a680b45f0 msgr2=0x7f6a680b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:32.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 -- 172.21.15.2:0/2828268674 shutdown_connections 2024-01-15T18:09:32.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.422+0000 7f6a6d600640 1 -- 172.21.15.2:0/2828268674 wait complete. 2024-01-15T18:09:32.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.424+0000 7f6a6d600640 1 Processor -- start 2024-01-15T18:09:32.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.424+0000 7f6a6d600640 1 -- start start 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a6d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a68159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a6d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a680ccc70 0x7f6a680c3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a6d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a6813ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a6d600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6813da30 con 0x7f6a681439f0 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a6d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6813dca0 con 0x7f6a680ccc70 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a6d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6813df80 con 0x7f6a680b3420 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a6813ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a6813ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60374/0 (socket says 172.21.15.2:60374) 2024-01-15T18:09:32.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 -- 172.21.15.2:0/3017372218 learned_addr learned my addr 172.21.15.2:0/3017372218 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a66000640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a680ccc70 0x7f6a680c3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a66a00640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a68159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 -- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 msgr2=0x7f6a68159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a68159bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 -- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a680ccc70 msgr2=0x7f6a680c3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a680ccc70 0x7f6a680c3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a67400640 1 -- 172.21.15.2:0/3017372218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580094d0 con 0x7f6a681439f0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.425+0000 7f6a66000640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a680ccc70 0x7f6a680c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.426+0000 7f6a67400640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a6813ea60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000cd70 tx=0x7f6a6000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.426+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60003b90 con 0x7f6a681439f0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.426+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a60003cf0 con 0x7f6a681439f0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.426+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a60004240 con 0x7f6a681439f0 2024-01-15T18:09:32.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.426+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6813e260 con 0x7f6a681439f0 2024-01-15T18:09:32.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.426+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6815a100 con 0x7f6a681439f0 2024-01-15T18:09:32.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.428+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6813e780 con 0x7f6a681439f0 2024-01-15T18:09:32.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.428+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6a600043a0 con 0x7f6a681439f0 2024-01-15T18:09:32.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.429+0000 7f6a5ea00640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a4006bf50 0x7f6a4006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.429+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6a60012070 con 0x7f6a681439f0 2024-01-15T18:09:32.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.430+0000 7f6a66a00640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a4006bf50 0x7f6a4006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.431+0000 7f6a66a00640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a4006bf50 0x7f6a4006e410 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6a58036000 tx=0x7f6a5803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:32.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.433+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a6813e780 con 0x7f6a681439f0 2024-01-15T18:09:32.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.667+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47168", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a6813de10 con 0x7f6a4006bf50 2024-01-15T18:09:32.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.693+0000 7f6a5ea00640 1 -- 172.21.15.2:0/3017372218 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f6a6813de10 con 0x7f6a4006bf50 2024-01-15T18:09:32.697 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:32.698 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:32.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a4006bf50 msgr2=0x7f6a4006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a4006bf50 0x7f6a4006e410 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f6a58036000 tx=0x7f6a5803c040 comp rx=0 tx=0).stop 2024-01-15T18:09:32.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 msgr2=0x7f6a6813ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a6813ea60 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f6a6000cd70 tx=0x7f6a6000a660 comp rx=0 tx=0).stop 2024-01-15T18:09:32.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 shutdown_connections 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a4006bf50 0x7f6a4006e410 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a681439f0 0x7f6a6813ea60 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a680ccc70 0x7f6a680c3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 --2- 172.21.15.2:0/3017372218 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a680b3420 0x7f6a68159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.695+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 >> 172.21.15.2:0/3017372218 conn(0x7f6a680b45f0 msgr2=0x7f6a680ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.696+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 shutdown_connections 2024-01-15T18:09:32.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.696+0000 7f6a6d600640 1 -- 172.21.15.2:0/3017372218 wait complete. 2024-01-15T18:09:32.719 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:32.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone47169 2024-01-15T18:09:32.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.941+0000 7f4766600640 1 Processor -- start 2024-01-15T18:09:32.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.941+0000 7f4766600640 1 -- start start 2024-01-15T18:09:32.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.942+0000 7f4766600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47601439f0 0x7f47600702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.942+0000 7f4766600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f47600b3420 0x7f47600cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4766600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476015dd70 con 0x7f47600ccc70 2024-01-15T18:09:32.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4766600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47600b3880 con 0x7f47600b3420 2024-01-15T18:09:32.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4766600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47600711b0 con 0x7f47601439f0 2024-01-15T18:09:32.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4757e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f47600b3420 0x7f47600cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f475fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f47600ccc70 0x7f476006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52554/0 (socket says 172.21.15.2:52554) 2024-01-15T18:09:32.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f475fe00640 1 -- 172.21.15.2:0/804171814 learned_addr learned my addr 172.21.15.2:0/804171814 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:32.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f475f400640 1 --2- 172.21.15.2:0/804171814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47601439f0 0x7f47600702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4757e00640 1 -- 172.21.15.2:0/804171814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47601439f0 msgr2=0x7f47600702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4757e00640 1 --2- 172.21.15.2:0/804171814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47601439f0 0x7f47600702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4757e00640 1 -- 172.21.15.2:0/804171814 >> v1:172.21.15.2:6789/0 conn(0x7f47600ccc70 legacy=0x7f476006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f4757e00640 1 -- 172.21.15.2:0/804171814 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4760143dd0 con 0x7f47600b3420 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.943+0000 7f475ea00640 1 -- 172.21.15.2:0/804171814 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831571584 0 0) 0x7f476015dd70 con 0x7f47600ccc70 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.945+0000 7f4757e00640 1 --2- 172.21.15.2:0/804171814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47600b3420 0x7f47600cc730 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f47480097a0 tx=0x7f474802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=127edafe42bbddb1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.945+0000 7f475ea00640 1 -- 172.21.15.2:0/804171814 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474802d040 con 0x7f47600b3420 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.945+0000 7f475ea00640 1 -- 172.21.15.2:0/804171814 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4748031810 con 0x7f47600b3420 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.945+0000 7f475ea00640 1 -- 172.21.15.2:0/804171814 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4748031ba0 con 0x7f47600b3420 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 -- 172.21.15.2:0/804171814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47600b3420 msgr2=0x7f47600cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 --2- 172.21.15.2:0/804171814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47600b3420 0x7f47600cc730 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f47480097a0 tx=0x7f474802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 -- 172.21.15.2:0/804171814 shutdown_connections 2024-01-15T18:09:32.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 --2- 172.21.15.2:0/804171814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47600b3420 0x7f47600cc730 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 --2- 172.21.15.2:0/804171814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47601439f0 0x7f47600702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 -- 172.21.15.2:0/804171814 >> 172.21.15.2:0/804171814 conn(0x7f47600b45f0 msgr2=0x7f47600b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:32.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 -- 172.21.15.2:0/804171814 shutdown_connections 2024-01-15T18:09:32.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.946+0000 7f4766600640 1 -- 172.21.15.2:0/804171814 wait complete. 2024-01-15T18:09:32.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.948+0000 7f4766600640 1 Processor -- start 2024-01-15T18:09:32.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.948+0000 7f4766600640 1 -- start start 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f4766600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47600b3420 0x7f47600aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f4766600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47600ccc70 0x7f47600af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f4766600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 0x7f476012f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f4766600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476015c1b0 con 0x7f47601439f0 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f4766600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f476015c490 con 0x7f47600b3420 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f4766600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f476015c770 con 0x7f47600ccc70 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f475fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 0x7f476012f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f475fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 0x7f476012f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53838/0 (socket says 172.21.15.2:53838) 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.949+0000 7f475fe00640 1 -- 172.21.15.2:0/1706745156 learned_addr learned my addr 172.21.15.2:0/1706745156 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f4757e00640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47600ccc70 0x7f47600af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475f400640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47600b3420 0x7f47600aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475fe00640 1 -- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47600ccc70 msgr2=0x7f47600af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475fe00640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47600ccc70 0x7f47600af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475fe00640 1 -- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47600b3420 msgr2=0x7f47600aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475fe00640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47600b3420 0x7f47600aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:32.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475fe00640 1 -- 172.21.15.2:0/1706745156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748009400 con 0x7f47601439f0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f475fe00640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 0x7f476012f6b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f475800c7e0 tx=0x7f475800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475800a640 con 0x7f47601439f0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475800a7c0 con 0x7f47601439f0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.950+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475800ab70 con 0x7f47601439f0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.951+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476015ca50 con 0x7f47601439f0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.951+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4760130450 con 0x7f47601439f0 2024-01-15T18:09:32.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.952+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47600686f0 con 0x7f47601439f0 2024-01-15T18:09:32.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.953+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f475800acd0 con 0x7f47601439f0 2024-01-15T18:09:32.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.954+0000 7f4757400640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f473806bf20 0x7f473806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:32.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.954+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4758087eb0 con 0x7f47601439f0 2024-01-15T18:09:32.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.955+0000 7f475f400640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f473806bf20 0x7f473806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:32.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.956+0000 7f475f400640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f473806bf20 0x7f473806e3e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4750006fd0 tx=0x7f4750008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:32.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:32.959+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47580544c0 con 0x7f47601439f0 2024-01-15T18:09:33.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.188+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone47169", "target": ["mon-mgr", ""]}) v1 -- 0x7f476015c600 con 0x7f473806bf20 2024-01-15T18:09:33.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.209+0000 7f4757400640 1 -- 172.21.15.2:0/1706745156 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f476015c600 con 0x7f473806bf20 2024-01-15T18:09:33.213 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:33.214 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f473806bf20 msgr2=0x7f473806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f473806bf20 0x7f473806e3e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4750006fd0 tx=0x7f4750008040 comp rx=0 tx=0).stop 2024-01-15T18:09:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 msgr2=0x7f476012f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 0x7f476012f6b0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f475800c7e0 tx=0x7f475800cca0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 shutdown_connections 2024-01-15T18:09:33.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f473806bf20 0x7f473806e3e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f47601439f0 0x7f476012f6b0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.211+0000 7f4766600640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47600ccc70 0x7f47600af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.212+0000 7f4766600640 1 --2- 172.21.15.2:0/1706745156 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47600b3420 0x7f47600aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.212+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 >> 172.21.15.2:0/1706745156 conn(0x7f47600b45f0 msgr2=0x7f4760159a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:33.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.212+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 shutdown_connections 2024-01-15T18:09:33.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.212+0000 7f4766600640 1 -- 172.21.15.2:0/1706745156 wait complete. 2024-01-15T18:09:33.237 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:33.237 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471610 2024-01-15T18:09:33.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.462+0000 7fb223600640 1 Processor -- start 2024-01-15T18:09:33.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.463+0000 7fb223600640 1 -- start start 2024-01-15T18:09:33.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.464+0000 7fb223600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c15b900 0x7fb21c15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.464+0000 7fb223600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb21c071410 0x7fb21c071880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.464+0000 7fb223600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c15c440 con 0x7fb21c155960 2024-01-15T18:09:33.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.464+0000 7fb223600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c071dc0 con 0x7fb21c071410 2024-01-15T18:09:33.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.464+0000 7fb223600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c15c720 con 0x7fb21c15b900 2024-01-15T18:09:33.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.465+0000 7fb21be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c15b900 0x7fb21c15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.465+0000 7fb221600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb21c071410 0x7fb21c071880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.465+0000 7fb220c00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fb21c155960 0x7fb21c0cf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:48798/0 (socket says 172.21.15.2:48798) 2024-01-15T18:09:33.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.465+0000 7fb220c00640 1 -- 172.21.15.2:0/1547244536 learned_addr learned my addr 172.21.15.2:0/1547244536 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:33.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.466+0000 7fb221600640 1 -- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c15b900 msgr2=0x7fb21c15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:33.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.466+0000 7fb221600640 1 --2- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c15b900 0x7fb21c15bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.466+0000 7fb221600640 1 -- 172.21.15.2:0/1547244536 >> v1:172.21.15.137:6789/0 conn(0x7fb21c155960 legacy=0x7fb21c0cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.466+0000 7fb221600640 1 -- 172.21.15.2:0/1547244536 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c0ced10 con 0x7fb21c071410 2024-01-15T18:09:33.470 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.466+0000 7fb21be00640 1 --2- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c15b900 0x7fb21c15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:33.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.467+0000 7fb221600640 1 --2- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c071880 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb214009d90 tx=0x7fb21402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a841a303396661a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.467+0000 7fb21b400640 1 -- 172.21.15.2:0/1547244536 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21402cc60 con 0x7fb21c071410 2024-01-15T18:09:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb21b400640 1 -- 172.21.15.2:0/1547244536 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21402cdc0 con 0x7fb21c071410 2024-01-15T18:09:33.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb21b400640 1 -- 172.21.15.2:0/1547244536 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb214034660 con 0x7fb21c071410 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 -- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 msgr2=0x7fb21c071880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 --2- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c071880 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb214009d90 tx=0x7fb21402beb0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 -- 172.21.15.2:0/1547244536 shutdown_connections 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 --2- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c071880 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 --2- 172.21.15.2:0/1547244536 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c15b900 0x7fb21c15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 -- 172.21.15.2:0/1547244536 >> 172.21.15.2:0/1547244536 conn(0x7fb21c0b1c70 msgr2=0x7fb21c0b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 -- 172.21.15.2:0/1547244536 shutdown_connections 2024-01-15T18:09:33.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.468+0000 7fb223600640 1 -- 172.21.15.2:0/1547244536 wait complete. 2024-01-15T18:09:33.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.470+0000 7fb223600640 1 Processor -- start 2024-01-15T18:09:33.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.470+0000 7fb223600640 1 -- start start 2024-01-15T18:09:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb223600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c15aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb223600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c155960 0x7fb21c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb223600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 0x7fb21c141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb223600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c156ac0 con 0x7fb21c15b900 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb223600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c159e10 con 0x7fb21c071410 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb223600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c156d60 con 0x7fb21c155960 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.471+0000 7fb221600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 0x7fb21c141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 0x7fb21c141830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53848/0 (socket says 172.21.15.2:53848) 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 -- 172.21.15.2:0/3923723519 learned_addr learned my addr 172.21.15.2:0/3923723519 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb21be00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c155960 0x7fb21c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb220c00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c15aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 -- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c155960 msgr2=0x7fb21c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c155960 0x7fb21c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 -- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 msgr2=0x7fb21c15aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c15aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb221600640 1 -- 172.21.15.2:0/3923723519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb204009400 con 0x7fb21c15b900 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb220c00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:33.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.472+0000 7fb21be00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c155960 0x7fb21c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.473+0000 7fb221600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 0x7fb21c141830 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb214009eb0 tx=0x7fb214004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.473+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21403e610 con 0x7fb21c15b900 2024-01-15T18:09:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.473+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb21403e770 con 0x7fb21c15b900 2024-01-15T18:09:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.473+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb21403eb60 con 0x7fb21c15b900 2024-01-15T18:09:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.473+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb214009a70 con 0x7fb21c15b900 2024-01-15T18:09:33.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.473+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c15a0f0 con 0x7fb21c15b900 2024-01-15T18:09:33.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.474+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c157280 con 0x7fb21c15b900 2024-01-15T18:09:33.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.475+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb21403b050 con 0x7fb21c15b900 2024-01-15T18:09:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.476+0000 7fb218c00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1f406bf20 0x7fb1f406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.476+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fb2140aab40 con 0x7fb21c15b900 2024-01-15T18:09:33.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.477+0000 7fb220c00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1f406bf20 0x7fb1f406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.478+0000 7fb220c00640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1f406bf20 0x7fb1f406e3e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb20c000f60 tx=0x7fb20c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:33.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.481+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2140770f0 con 0x7fb21c15b900 2024-01-15T18:09:33.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.715+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471610", "target": ["mon-mgr", ""]}) v1 -- 0x7fb21c0cc980 con 0x7fb1f406bf20 2024-01-15T18:09:33.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.736+0000 7fb218c00640 1 -- 172.21.15.2:0/3923723519 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb20c001cd0 con 0x7fb1f406bf20 2024-01-15T18:09:33.740 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:33.740 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:33.741 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1f406bf20 msgr2=0x7fb1f406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1f406bf20 0x7fb1f406e3e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb20c000f60 tx=0x7fb20c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 msgr2=0x7fb21c141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 0x7fb21c141830 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fb214009eb0 tx=0x7fb214004a40 comp rx=0 tx=0).stop 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 shutdown_connections 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1f406bf20 0x7fb1f406e3e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb21c15b900 0x7fb21c141830 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb21c155960 0x7fb21c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 --2- 172.21.15.2:0/3923723519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb21c071410 0x7fb21c15aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 >> 172.21.15.2:0/3923723519 conn(0x7fb21c0b1c70 msgr2=0x7fb21c0cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:33.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 shutdown_connections 2024-01-15T18:09:33.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.739+0000 7fb223600640 1 -- 172.21.15.2:0/3923723519 wait complete. 2024-01-15T18:09:33.763 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:33.764 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471611 2024-01-15T18:09:33.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.982+0000 7f7512c00640 1 Processor -- start 2024-01-15T18:09:33.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.982+0000 7f7512c00640 1 -- start start 2024-01-15T18:09:33.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.983+0000 7f7512c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f750c06d000 0x7f750c165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.983+0000 7f7512c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c0ce230 con 0x7f750c15c450 2024-01-15T18:09:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.983+0000 7f7512c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c0ce510 con 0x7f750c144c00 2024-01-15T18:09:33.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.983+0000 7f7512c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0ce7f0 con 0x7f750c06d000 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.984+0000 7f750b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f750c15c450 0x7f750c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52556/0 (socket says 172.21.15.2:52556) 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.984+0000 7f750b400640 1 -- 172.21.15.2:0/1903413107 learned_addr learned my addr 172.21.15.2:0/1903413107 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.984+0000 7f7510c00640 1 --2- 172.21.15.2:0/1903413107 >> v2:172.21.15.137:3300/0 conn(0x7f750c06d000 0x7f750c165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.984+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258911201 0 0) 0x7f750c0ce230 con 0x7f750c15c450 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.984+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74ec003520 con 0x7f750c15c450 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 831024192 0 0) 0x7f74ec003520 con 0x7f750c15c450 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 >> v2:172.21.15.137:3300/0 conn(0x7f750c06d000 msgr2=0x7f750c165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:33.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 --2- 172.21.15.2:0/1903413107 >> v2:172.21.15.137:3300/0 conn(0x7f750c06d000 0x7f750c165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 >> v1:172.21.15.137:6789/0 conn(0x7f750c144c00 legacy=0x7f750c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0705f0 con 0x7f750c15c450 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f74f4001670 con 0x7f750c15c450 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 >> v1:172.21.15.2:6789/0 conn(0x7f750c15c450 legacy=0x7f750c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74ec0039c0 0x7f74ec003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74ec0047c0 0x7f74ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec005180 0x7f74ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0ce7f0 con 0x7f74ec0039c0 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec00c080 con 0x7f74ec005180 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec00c240 con 0x7f74ec0047c0 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750be00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74ec0039c0 0x7f74ec003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f7510c00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec005180 0x7f74ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.985+0000 7f750b400640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74ec0047c0 0x7f74ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.986+0000 7f750be00640 1 -- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74ec0047c0 msgr2=0x7f74ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:33.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.986+0000 7f750be00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74ec0047c0 0x7f74ec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.986+0000 7f750be00640 1 -- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec005180 msgr2=0x7f74ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:33.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.986+0000 7f750be00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec005180 0x7f74ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.986+0000 7f750be00640 1 -- 172.21.15.2:0/1903413107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec003520 con 0x7f74ec0039c0 2024-01-15T18:09:33.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.986+0000 7f7510c00640 1 -- 172.21.15.2:0/1903413107 reap_dead start 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.987+0000 7f750be00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74ec0039c0 0x7f74ec003de0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f74fc001750 tx=0x7f74ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3498ca8e63d231f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.987+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc0052d0 con 0x7f74ec0039c0 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.987+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74fc02e050 con 0x7f74ec0039c0 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.987+0000 7f750aa00640 1 -- 172.21.15.2:0/1903413107 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74fc003830 con 0x7f74ec0039c0 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 -- 172.21.15.2:0/1903413107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74ec0039c0 msgr2=0x7f74ec003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74ec0039c0 0x7f74ec003de0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f74fc001750 tx=0x7f74ec0057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 -- 172.21.15.2:0/1903413107 shutdown_connections 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 --2- 172.21.15.2:0/1903413107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74ec0039c0 0x7f74ec003de0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 -- 172.21.15.2:0/1903413107 >> 172.21.15.2:0/1903413107 conn(0x7f750c0b45f0 msgr2=0x7f750c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:33.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 -- 172.21.15.2:0/1903413107 shutdown_connections 2024-01-15T18:09:33.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.988+0000 7f7512c00640 1 -- 172.21.15.2:0/1903413107 wait complete. 2024-01-15T18:09:33.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.990+0000 7f7512c00640 1 Processor -- start 2024-01-15T18:09:33.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.990+0000 7f7512c00640 1 -- start start 2024-01-15T18:09:33.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7512c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec0039c0 0x7f750c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7512c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f750c15aba0 0x7f750c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7512c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 0x7f750c15da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7512c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c14e8b0 con 0x7f750c1434e0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7512c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c1514c0 con 0x7f74ec0039c0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7512c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c14dc30 con 0x7f750c15aba0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.991+0000 7f7510c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 0x7f750c15da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 0x7f750c15da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53858/0 (socket says 172.21.15.2:53858) 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 -- 172.21.15.2:0/3116755392 learned_addr learned my addr 172.21.15.2:0/3116755392 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f750be00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec0039c0 0x7f750c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f750b400640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f750c15aba0 0x7f750c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 -- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f750c15aba0 msgr2=0x7f750c150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f750c15aba0 0x7f750c150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 -- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec0039c0 msgr2=0x7f750c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec0039c0 0x7f750c0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 -- 172.21.15.2:0/3116755392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c14df10 con 0x7f750c1434e0 2024-01-15T18:09:33.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.992+0000 7f7510c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 0x7f750c15da10 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f750c0cca20 tx=0x7f750400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.993+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750400e040 con 0x7f750c1434e0 2024-01-15T18:09:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.993+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750400f040 con 0x7f750c1434e0 2024-01-15T18:09:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.993+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c14e1f0 con 0x7f750c1434e0 2024-01-15T18:09:33.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.993+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504015430 con 0x7f750c1434e0 2024-01-15T18:09:33.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.993+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c143960 con 0x7f750c1434e0 2024-01-15T18:09:33.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.994+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c0686f0 con 0x7f750c1434e0 2024-01-15T18:09:33.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.995+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f7504015590 con 0x7f750c1434e0 2024-01-15T18:09:34.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.996+0000 7f7503e00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f74e406bf50 0x7f74e406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.996+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f7504089180 con 0x7f750c1434e0 2024-01-15T18:09:34.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.997+0000 7f750be00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f74e406bf50 0x7f74e406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:34.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:33.998+0000 7f750be00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f74e406bf50 0x7f74e406e410 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f74fc033000 tx=0x7f74fc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:34.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.000+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75040557c0 con 0x7f750c1434e0 2024-01-15T18:09:34.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.231+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471611", "target": ["mon-mgr", ""]}) v1 -- 0x7f750c0ace10 con 0x7f74e406bf50 2024-01-15T18:09:34.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.254+0000 7f7503e00640 1 -- 172.21.15.2:0/3116755392 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f750c0ace10 con 0x7f74e406bf50 2024-01-15T18:09:34.258 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:34.258 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:34.258 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:34.259 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:34.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.256+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f74e406bf50 msgr2=0x7f74e406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.256+0000 7f7512c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f74e406bf50 0x7f74e406e410 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f74fc033000 tx=0x7f74fc009030 comp rx=0 tx=0).stop 2024-01-15T18:09:34.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.256+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 msgr2=0x7f750c15da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.256+0000 7f7512c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 0x7f750c15da10 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f750c0cca20 tx=0x7f750400f580 comp rx=0 tx=0).stop 2024-01-15T18:09:34.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.256+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 shutdown_connections 2024-01-15T18:09:34.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f74e406bf50 0x7f74e406e410 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f750c1434e0 0x7f750c15da10 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f750c15aba0 0x7f750c150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 --2- 172.21.15.2:0/3116755392 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74ec0039c0 0x7f750c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 >> 172.21.15.2:0/3116755392 conn(0x7f750c0b45f0 msgr2=0x7f750c15b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:34.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 shutdown_connections 2024-01-15T18:09:34.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.257+0000 7f7512c00640 1 -- 172.21.15.2:0/3116755392 wait complete. 2024-01-15T18:09:34.281 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:34.289 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471612 2024-01-15T18:09:34.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.511+0000 7fed71000640 1 Processor -- start 2024-01-15T18:09:34.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.512+0000 7fed71000640 1 -- start start 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.513+0000 7fed71000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fed6c143d20 0x7fed6c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.513+0000 7fed71000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fed6c1447d0 0x7fed6c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.513+0000 7fed71000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6c071090 con 0x7fed6c0720d0 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.513+0000 7fed71000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c071430 con 0x7fed6c1447d0 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.513+0000 7fed71000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6c144100 con 0x7fed6c143d20 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fed6c1447d0 0x7fed6c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fed6c1447d0 0x7fed6c15f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53874/0 (socket says 172.21.15.2:53874) 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6be00640 1 -- 172.21.15.2:0/3908753753 learned_addr learned my addr 172.21.15.2:0/3908753753 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6b400640 1 --2- 172.21.15.2:0/3908753753 >> v2:172.21.15.137:3300/0 conn(0x7fed6c143d20 0x7fed6c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6be00640 1 -- 172.21.15.2:0/3908753753 >> v2:172.21.15.137:3300/0 conn(0x7fed6c143d20 msgr2=0x7fed6c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6be00640 1 --2- 172.21.15.2:0/3908753753 >> v2:172.21.15.137:3300/0 conn(0x7fed6c143d20 0x7fed6c15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.514+0000 7fed6be00640 1 -- 172.21.15.2:0/3908753753 >> v1:172.21.15.137:6789/0 conn(0x7fed6c0720d0 legacy=0x7fed6c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.515+0000 7fed6be00640 1 -- 172.21.15.2:0/3908753753 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c06d160 con 0x7fed6c1447d0 2024-01-15T18:09:34.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.516+0000 7fed6be00640 1 --2- 172.21.15.2:0/3908753753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15f090 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fed6c06e600 tx=0x7fed6402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74872df75b9aafd1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:34.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.516+0000 7fed6a000640 1 -- 172.21.15.2:0/3908753753 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6402c9f0 con 0x7fed6c1447d0 2024-01-15T18:09:34.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.516+0000 7fed6a000640 1 -- 172.21.15.2:0/3908753753 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6402cb50 con 0x7fed6c1447d0 2024-01-15T18:09:34.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.516+0000 7fed6a000640 1 -- 172.21.15.2:0/3908753753 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed640343f0 con 0x7fed6c1447d0 2024-01-15T18:09:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 -- 172.21.15.2:0/3908753753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 msgr2=0x7fed6c15f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 --2- 172.21.15.2:0/3908753753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15f090 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fed6c06e600 tx=0x7fed6402be50 comp rx=0 tx=0).stop 2024-01-15T18:09:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 -- 172.21.15.2:0/3908753753 shutdown_connections 2024-01-15T18:09:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 --2- 172.21.15.2:0/3908753753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15f090 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 --2- 172.21.15.2:0/3908753753 >> v2:172.21.15.137:3300/0 conn(0x7fed6c143d20 0x7fed6c15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 -- 172.21.15.2:0/3908753753 >> 172.21.15.2:0/3908753753 conn(0x7fed6c0b0840 msgr2=0x7fed6c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 -- 172.21.15.2:0/3908753753 shutdown_connections 2024-01-15T18:09:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.517+0000 7fed71000640 1 -- 172.21.15.2:0/3908753753 wait complete. 2024-01-15T18:09:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.519+0000 7fed71000640 1 Processor -- start 2024-01-15T18:09:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.519+0000 7fed71000640 1 -- start start 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed71000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed6c0720d0 0x7fed6c1523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed71000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed6c143d20 0x7fed6c1f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed71000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed71000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c13e8d0 con 0x7fed6c1447d0 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed71000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c13df30 con 0x7fed6c143d20 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed71000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6c13e210 con 0x7fed6c0720d0 2024-01-15T18:09:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53880/0 (socket says 172.21.15.2:53880) 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed6be00640 1 -- 172.21.15.2:0/1823957645 learned_addr learned my addr 172.21.15.2:0/1823957645 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed6aa00640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed6c143d20 0x7fed6c1f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed6be00640 1 -- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed6c0720d0 msgr2=0x7fed6c1523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.520+0000 7fed6be00640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed6c0720d0 0x7fed6c1523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed6be00640 1 -- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed6c143d20 msgr2=0x7fed6c1f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed6be00640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed6c143d20 0x7fed6c1f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed6be00640 1 -- 172.21.15.2:0/1823957645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64009b30 con 0x7fed6c1447d0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed6be00640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15d090 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fed64036040 tx=0x7fed64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64009510 con 0x7fed6c1447d0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6403e450 con 0x7fed6c1447d0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6403e800 con 0x7fed6c1447d0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6c13e4f0 con 0x7fed6c1447d0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.521+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c1f3380 con 0x7fed6c1447d0 2024-01-15T18:09:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.522+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6c13ea10 con 0x7fed6c1447d0 2024-01-15T18:09:34.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.523+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fed6403e960 con 0x7fed6c1447d0 2024-01-15T18:09:34.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.524+0000 7fed63400640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed4406bf20 0x7fed4406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:34.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.524+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fed640aba50 con 0x7fed6c1447d0 2024-01-15T18:09:34.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.525+0000 7fed6b400640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed4406bf20 0x7fed4406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:34.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.526+0000 7fed6b400640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed4406bf20 0x7fed4406e3e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0098f0 tx=0x7fed5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:34.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.529+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed6c13ea10 con 0x7fed6c1447d0 2024-01-15T18:09:34.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.765+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471612", "target": ["mon-mgr", ""]}) v1 -- 0x7fed6c13e0a0 con 0x7fed4406bf20 2024-01-15T18:09:34.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.789+0000 7fed63400640 1 -- 172.21.15.2:0/1823957645 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fed6c13e0a0 con 0x7fed4406bf20 2024-01-15T18:09:34.793 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:34.793 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:34.793 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:34.793 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:34.794 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed4406bf20 msgr2=0x7fed4406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed4406bf20 0x7fed4406e3e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fed5c0098f0 tx=0x7fed5c008040 comp rx=0 tx=0).stop 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 msgr2=0x7fed6c15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15d090 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fed64036040 tx=0x7fed64004a40 comp rx=0 tx=0).stop 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 shutdown_connections 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed4406bf20 0x7fed4406e3e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed6c1447d0 0x7fed6c15d090 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed6c143d20 0x7fed6c1f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 --2- 172.21.15.2:0/1823957645 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed6c0720d0 0x7fed6c1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 >> 172.21.15.2:0/1823957645 conn(0x7fed6c0b0840 msgr2=0x7fed6c141310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 shutdown_connections 2024-01-15T18:09:34.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:34.791+0000 7fed71000640 1 -- 172.21.15.2:0/1823957645 wait complete. 2024-01-15T18:09:34.814 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:34.814 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471613 2024-01-15T18:09:35.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.033+0000 7f048b800640 1 Processor -- start 2024-01-15T18:09:35.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.033+0000 7f048b800640 1 -- start start 2024-01-15T18:09:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f048b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 0x7f0484165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f048b800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840ce230 con 0x7f0484144c00 2024-01-15T18:09:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f048b800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840ce510 con 0x7f048415c450 2024-01-15T18:09:35.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f048b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04840ce7f0 con 0x7f048406d000 2024-01-15T18:09:35.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f0488e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0484144c00 0x7f04840cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52572/0 (socket says 172.21.15.2:52572) 2024-01-15T18:09:35.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f0488e00640 1 -- 172.21.15.2:0/1784678966 learned_addr learned my addr 172.21.15.2:0/1784678966 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.035+0000 7f0489800640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 0x7f0484165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117734737 0 0) 0x7f04840ce230 con 0x7f0484144c00 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0464003520 con 0x7f0484144c00 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3302842667 0 0) 0x7f0464003520 con 0x7f0484144c00 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 msgr2=0x7f0484165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 0x7f0484165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 >> v1:172.21.15.137:6789/0 conn(0x7f048415c450 legacy=0x7f048406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840705f0 con 0x7f0484144c00 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f0474001670 con 0x7f0484144c00 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 >> v1:172.21.15.2:6789/0 conn(0x7f0484144c00 legacy=0x7f04840cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 0x7f04640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0464004d00 0x7f0464005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04840ce7f0 con 0x7f0464004d00 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046400ba60 con 0x7f04640039a0 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.036+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046400bc20 con 0x7f048406d000 2024-01-15T18:09:35.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0489800640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0464004d00 0x7f0464005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f047be00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 0x7f04640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0488e00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0489800640 1 -- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 msgr2=0x7f04640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0489800640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048406d000 0x7f04640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0489800640 1 -- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04640039a0 msgr2=0x7f0464003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0489800640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0489800640 1 -- 172.21.15.2:0/1784678966 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464003520 con 0x7f0464004d00 2024-01-15T18:09:35.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.037+0000 7f0488e00640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:35.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.038+0000 7f0489800640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0464004d00 0x7f0464005180 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f047c009f90 tx=0x7f04640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f27768eb314da8a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.038+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c02cbd0 con 0x7f0464004d00 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.038+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f047c00f040 con 0x7f0464004d00 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.038+0000 7f0483e00640 1 -- 172.21.15.2:0/1784678966 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047c034440 con 0x7f0464004d00 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f048b800640 1 -- 172.21.15.2:0/1784678966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0464004d00 msgr2=0x7f0464005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f048b800640 1 --2- 172.21.15.2:0/1784678966 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0464004d00 0x7f0464005180 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f047c009f90 tx=0x7f04640057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f0489800640 1 -- 172.21.15.2:0/1784678966 reap_dead start 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f048b800640 1 -- 172.21.15.2:0/1784678966 shutdown_connections 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f048b800640 1 -- 172.21.15.2:0/1784678966 >> 172.21.15.2:0/1784678966 conn(0x7f04840b45f0 msgr2=0x7f04840b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f048b800640 1 -- 172.21.15.2:0/1784678966 shutdown_connections 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.039+0000 7f048b800640 1 -- 172.21.15.2:0/1784678966 wait complete. 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.041+0000 7f048b800640 1 Processor -- start 2024-01-15T18:09:35.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.041+0000 7f048b800640 1 -- start start 2024-01-15T18:09:35.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f048b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0464004d00 0x7f0484159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f048b800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 0x7f048413e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f048b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048413a120 0x7f04840b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f048b800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048415a050 con 0x7f04841f3370 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f048b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04841432e0 con 0x7f0464004d00 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f048b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048413f2a0 con 0x7f048413a120 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f047be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 0x7f048413e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f047be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 0x7f048413e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53894/0 (socket says 172.21.15.2:53894) 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.042+0000 7f047be00640 1 -- 172.21.15.2:0/1686043036 learned_addr learned my addr 172.21.15.2:0/1686043036 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f0489800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048413a120 0x7f04840b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f0488e00640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0464004d00 0x7f0484159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f047be00640 1 -- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048413a120 msgr2=0x7f04840b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f047be00640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048413a120 0x7f04840b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f047be00640 1 -- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0464004d00 msgr2=0x7f0484159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f047be00640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0464004d00 0x7f0484159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f047be00640 1 -- 172.21.15.2:0/1686043036 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048413ea00 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f047be00640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 0x7f048413e100 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f046c00e550 tx=0x7f046c00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c0052e0 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c005440 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.043+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c003810 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.044+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048413ece0 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.044+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048413e640 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.045+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04841430b0 con 0x7f04841f3370 2024-01-15T18:09:35.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.046+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f046c003970 con 0x7f04841f3370 2024-01-15T18:09:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.047+0000 7f0481600640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f045c06bf50 0x7f045c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.047+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f046c0882d0 con 0x7f04841f3370 2024-01-15T18:09:35.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.048+0000 7f0488e00640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f045c06bf50 0x7f045c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.049+0000 7f0488e00640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f045c06bf50 0x7f045c06e410 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0474001200 tx=0x7f0474009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:35.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.052+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04841430b0 con 0x7f04841f3370 2024-01-15T18:09:35.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.288+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471613", "target": ["mon-mgr", ""]}) v1 -- 0x7f04840cead0 con 0x7f045c06bf50 2024-01-15T18:09:35.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.312+0000 7f0481600640 1 -- 172.21.15.2:0/1686043036 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f047400d050 con 0x7f045c06bf50 2024-01-15T18:09:35.317 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:35.317 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:35.317 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:35.317 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:35.317 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:35.318 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:35.318 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:35.318 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:35.318 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:35.318 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:35.318 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:35.319 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:35.319 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:35.319 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:35.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f045c06bf50 msgr2=0x7f045c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f045c06bf50 0x7f045c06e410 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f0474001200 tx=0x7f0474009040 comp rx=0 tx=0).stop 2024-01-15T18:09:35.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 msgr2=0x7f048413e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 0x7f048413e100 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f046c00e550 tx=0x7f046c00ea10 comp rx=0 tx=0).stop 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 shutdown_connections 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f045c06bf50 0x7f045c06e410 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f048413a120 0x7f04840b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.314+0000 7f048b800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04841f3370 0x7f048413e100 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.315+0000 7f048b800640 1 --2- 172.21.15.2:0/1686043036 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0464004d00 0x7f0484159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.315+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 >> 172.21.15.2:0/1686043036 conn(0x7f04840b45f0 msgr2=0x7f048406c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.315+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 shutdown_connections 2024-01-15T18:09:35.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.315+0000 7f048b800640 1 -- 172.21.15.2:0/1686043036 wait complete. 2024-01-15T18:09:35.338 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:35.338 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471614 2024-01-15T18:09:35.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.560+0000 7fdf93200640 1 Processor -- start 2024-01-15T18:09:35.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.560+0000 7fdf93200640 1 -- start start 2024-01-15T18:09:35.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.562+0000 7fdf93200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf8c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.562+0000 7fdf93200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8c0ce230 con 0x7fdf8c06ce30 2024-01-15T18:09:35.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.562+0000 7fdf93200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8c0ce510 con 0x7fdf8c144c00 2024-01-15T18:09:35.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.562+0000 7fdf93200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c06d2c0 con 0x7fdf8c15c450 2024-01-15T18:09:35.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf91200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdf8c06ce30 0x7fdf8c070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52588/0 (socket says 172.21.15.2:52588) 2024-01-15T18:09:35.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf91200640 1 -- 172.21.15.2:0/2349839845 learned_addr learned my addr 172.21.15.2:0/2349839845 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:35.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf83200640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf8c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440153934 0 0) 0x7fdf8c0ce230 con 0x7fdf8c06ce30 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c003520 con 0x7fdf8c06ce30 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2063823561 0 0) 0x7fdf6c003520 con 0x7fdf8c06ce30 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 msgr2=0x7fdf8c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.563+0000 7fdf8b400640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf8c06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 >> v1:172.21.15.137:6789/0 conn(0x7fdf8c144c00 legacy=0x7fdf8c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c0ce7f0 con 0x7fdf8c06ce30 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fdf84001670 con 0x7fdf8c06ce30 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 >> v1:172.21.15.2:6789/0 conn(0x7fdf8c06ce30 legacy=0x7fdf8c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf6c0039a0 0x7fdf6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c06d2c0 con 0x7fdf6c0039a0 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00ba30 con 0x7fdf6c0047c0 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00bbf0 con 0x7fdf8c15c450 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8be00640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf6c0039a0 0x7fdf6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf83200640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf91200640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.564+0000 7fdf8be00640 1 -- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 msgr2=0x7fdf6c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.565+0000 7fdf8be00640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf6c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.565+0000 7fdf8be00640 1 -- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0047c0 msgr2=0x7fdf6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:35.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.565+0000 7fdf8be00640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.565+0000 7fdf8be00640 1 -- 172.21.15.2:0/2349839845 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c003520 con 0x7fdf6c0039a0 2024-01-15T18:09:35.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.565+0000 7fdf91200640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf8c15c450 0x7fdf6c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:35.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf8be00640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf6c0039a0 0x7fdf6c003dc0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c001e20 tx=0x7fdf6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d62e6733abcd47 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:35.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c003b70 con 0x7fdf6c0039a0 2024-01-15T18:09:35.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c003cd0 con 0x7fdf6c0039a0 2024-01-15T18:09:35.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf8b400640 1 -- 172.21.15.2:0/2349839845 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c005310 con 0x7fdf6c0039a0 2024-01-15T18:09:35.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf93200640 1 -- 172.21.15.2:0/2349839845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf6c0039a0 msgr2=0x7fdf6c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf93200640 1 --2- 172.21.15.2:0/2349839845 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf6c0039a0 0x7fdf6c003dc0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c001e20 tx=0x7fdf6c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.566+0000 7fdf91200640 1 -- 172.21.15.2:0/2349839845 reap_dead start 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.567+0000 7fdf93200640 1 -- 172.21.15.2:0/2349839845 shutdown_connections 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.567+0000 7fdf93200640 1 -- 172.21.15.2:0/2349839845 >> 172.21.15.2:0/2349839845 conn(0x7fdf8c0b45f0 msgr2=0x7fdf8c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.567+0000 7fdf93200640 1 -- 172.21.15.2:0/2349839845 shutdown_connections 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.567+0000 7fdf93200640 1 -- 172.21.15.2:0/2349839845 wait complete. 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.568+0000 7fdf93200640 1 Processor -- start 2024-01-15T18:09:35.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.569+0000 7fdf93200640 1 -- start start 2024-01-15T18:09:35.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf93200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0039a0 0x7fdf8c1e4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf93200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8c0b3630 0x7fdf8c15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf93200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 0x7fdf8c1e5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf93200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1e4a20 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf93200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c1e6490 con 0x7fdf8c0b3630 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf93200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8c0b3ab0 con 0x7fdf6c0039a0 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 0x7fdf8c1e5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 0x7fdf8c1e5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53912/0 (socket says 172.21.15.2:53912) 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 -- 172.21.15.2:0/4029309389 learned_addr learned my addr 172.21.15.2:0/4029309389 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf8be00640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0039a0 0x7fdf8c1e4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf83200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8c0b3630 0x7fdf8c15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 -- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0039a0 msgr2=0x7fdf8c1e4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0039a0 0x7fdf8c1e4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 -- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8c0b3630 msgr2=0x7fdf8c15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8c0b3630 0x7fdf8c15aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf91200640 1 -- 172.21.15.2:0/4029309389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c1e3da0 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.570+0000 7fdf83200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8c0b3630 0x7fdf8c15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.571+0000 7fdf91200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 0x7fdf8c1e5f50 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c06f9f0 tx=0x7fdf8400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.571+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf84007230 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.571+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf84007390 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.571+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf840047e0 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.571+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf8c1e4080 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.571+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8c1e4640 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.572+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8c1e5380 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.573+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fdf84004940 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.574+0000 7fdf88c00640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf6406bf20 0x7fdf6406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:35.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.574+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fdf84088540 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.575+0000 7fdf8be00640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf6406bf20 0x7fdf6406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:35.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.576+0000 7fdf8be00640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf6406bf20 0x7fdf6406e3e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c034000 tx=0x7fdf7c009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:35.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.578+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf84054be0 con 0x7fdf8c1e5ad0 2024-01-15T18:09:35.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.827+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471614", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf8c0cead0 con 0x7fdf6406bf20 2024-01-15T18:09:35.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.852+0000 7fdf88c00640 1 -- 172.21.15.2:0/4029309389 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fdf7c03f040 con 0x7fdf6406bf20 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:35.857 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:35.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf6406bf20 msgr2=0x7fdf6406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf6406bf20 0x7fdf6406e3e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c034000 tx=0x7fdf7c009030 comp rx=0 tx=0).stop 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 msgr2=0x7fdf8c1e5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 0x7fdf8c1e5f50 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c06f9f0 tx=0x7fdf8400dc00 comp rx=0 tx=0).stop 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 shutdown_connections 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdf6406bf20 0x7fdf6406e3e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdf8c1e5ad0 0x7fdf8c1e5f50 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdf8c0b3630 0x7fdf8c15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 --2- 172.21.15.2:0/4029309389 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdf6c0039a0 0x7fdf8c1e4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 >> 172.21.15.2:0/4029309389 conn(0x7fdf8c0b45f0 msgr2=0x7fdf8c0aca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 shutdown_connections 2024-01-15T18:09:35.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:35.855+0000 7fdf93200640 1 -- 172.21.15.2:0/4029309389 wait complete. 2024-01-15T18:09:35.879 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:35.880 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471615 2024-01-15T18:09:36.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.099+0000 7f8881800640 1 Processor -- start 2024-01-15T18:09:36.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.100+0000 7f8881800640 1 -- start start 2024-01-15T18:09:36.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.101+0000 7f8881800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c070820 0x7f887c161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.101+0000 7f8881800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f887c072100 0x7f887c06cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.101+0000 7f8881800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f887c1620d0 con 0x7f887c14b160 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.101+0000 7f8881800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887c0b35b0 con 0x7f887c072100 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.101+0000 7f8881800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c0b1510 con 0x7f887c070820 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f887c072100 0x7f887c06cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f887c072100 0x7f887c06cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53914/0 (socket says 172.21.15.2:53914) 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 -- 172.21.15.2:0/742887944 learned_addr learned my addr 172.21.15.2:0/742887944 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8873200640 1 --2- 172.21.15.2:0/742887944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c070820 0x7f887c161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 -- 172.21.15.2:0/742887944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c070820 msgr2=0x7f887c161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 --2- 172.21.15.2:0/742887944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c070820 0x7f887c161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 -- 172.21.15.2:0/742887944 >> v1:172.21.15.137:6789/0 conn(0x7f887c14b160 legacy=0x7f887c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:36.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.102+0000 7f8880e00640 1 -- 172.21.15.2:0/742887944 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0b32d0 con 0x7f887c072100 2024-01-15T18:09:36.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.103+0000 7f8880e00640 1 --2- 172.21.15.2:0/742887944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c072100 0x7f887c06cdd0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f8874009e50 tx=0x7f887402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638e1b907f3097a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:36.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.104+0000 7f887b400640 1 -- 172.21.15.2:0/742887944 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887402ca70 con 0x7f887c072100 2024-01-15T18:09:36.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.104+0000 7f887b400640 1 -- 172.21.15.2:0/742887944 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887402cbd0 con 0x7f887c072100 2024-01-15T18:09:36.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.104+0000 7f887b400640 1 -- 172.21.15.2:0/742887944 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740343f0 con 0x7f887c072100 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 -- 172.21.15.2:0/742887944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c072100 msgr2=0x7f887c06cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 --2- 172.21.15.2:0/742887944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c072100 0x7f887c06cdd0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f8874009e50 tx=0x7f887402bed0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 -- 172.21.15.2:0/742887944 shutdown_connections 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 --2- 172.21.15.2:0/742887944 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c072100 0x7f887c06cdd0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 --2- 172.21.15.2:0/742887944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c070820 0x7f887c161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 -- 172.21.15.2:0/742887944 >> 172.21.15.2:0/742887944 conn(0x7f887c0b0840 msgr2=0x7f887c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:36.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 -- 172.21.15.2:0/742887944 shutdown_connections 2024-01-15T18:09:36.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.105+0000 7f8881800640 1 -- 172.21.15.2:0/742887944 wait complete. 2024-01-15T18:09:36.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.107+0000 7f8881800640 1 Processor -- start 2024-01-15T18:09:36.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.107+0000 7f8881800640 1 -- start start 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8881800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 0x7f887c13fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8881800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c072100 0x7f887c13e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8881800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887c14b160 0x7f887c145640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8881800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c140f10 con 0x7f887c070820 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8881800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887c140140 con 0x7f887c14b160 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8881800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887c140390 con 0x7f887c072100 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 0x7f887c13fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 0x7f887c13fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53926/0 (socket says 172.21.15.2:53926) 2024-01-15T18:09:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 -- 172.21.15.2:0/3275600797 learned_addr learned my addr 172.21.15.2:0/3275600797 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8873200640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c072100 0x7f887c13e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 -- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c072100 msgr2=0x7f887c13e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8880e00640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887c14b160 0x7f887c145640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c072100 0x7f887c13e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 -- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887c14b160 msgr2=0x7f887c145640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887c14b160 0x7f887c145640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f887be00640 1 -- 172.21.15.2:0/3275600797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874009b30 con 0x7f887c070820 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.109+0000 7f8873200640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c072100 0x7f887c13e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.110+0000 7f887be00640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 0x7f887c13fc00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f886c00c460 tx=0x7f886c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.110+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c012830 con 0x7f887c070820 2024-01-15T18:09:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.110+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886c0129d0 con 0x7f887c070820 2024-01-15T18:09:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.110+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887c140670 con 0x7f887c070820 2024-01-15T18:09:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.110+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886c012d80 con 0x7f887c070820 2024-01-15T18:09:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.110+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887c140c30 con 0x7f887c070820 2024-01-15T18:09:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.111+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887c074770 con 0x7f887c070820 2024-01-15T18:09:36.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.113+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f886c01b400 con 0x7f887c070820 2024-01-15T18:09:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.113+0000 7f8878c00640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f885406bf20 0x7f885406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.114+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f886c0887c0 con 0x7f887c070820 2024-01-15T18:09:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.114+0000 7f8873200640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f885406bf20 0x7f885406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.116+0000 7f8873200640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f885406bf20 0x7f885406e3e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f887c0cf220 tx=0x7f8864008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.118+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f886c054e60 con 0x7f887c070820 2024-01-15T18:09:36.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.350+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471615", "target": ["mon-mgr", ""]}) v1 -- 0x7f887c06e180 con 0x7f885406bf20 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.374+0000 7f8878c00640 1 -- 172.21.15.2:0/3275600797 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f887c06e180 con 0x7f885406bf20 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:36.379 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:36.380 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:36.380 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:36.380 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:36.380 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:36.380 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:36.380 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f885406bf20 msgr2=0x7f885406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f885406bf20 0x7f885406e3e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f887c0cf220 tx=0x7f8864008040 comp rx=0 tx=0).stop 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 msgr2=0x7f887c13fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 0x7f887c13fc00 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f886c00c460 tx=0x7f886c00c920 comp rx=0 tx=0).stop 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 shutdown_connections 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f885406bf20 0x7f885406e3e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f887c14b160 0x7f887c145640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f887c072100 0x7f887c13e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 --2- 172.21.15.2:0/3275600797 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f887c070820 0x7f887c13fc00 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 >> 172.21.15.2:0/3275600797 conn(0x7f887c0b0840 msgr2=0x7f887c0af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 shutdown_connections 2024-01-15T18:09:36.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.377+0000 7f8881800640 1 -- 172.21.15.2:0/3275600797 wait complete. 2024-01-15T18:09:36.402 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:36.403 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471616 2024-01-15T18:09:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.623+0000 7f2fd8a00640 1 Processor -- start 2024-01-15T18:09:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.623+0000 7f2fd8a00640 1 -- start start 2024-01-15T18:09:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.625+0000 7f2fd8a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2fd40ce010 0x7f2fd406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.625+0000 7f2fd8a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2fd406a9f0 0x7f2fd4160da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.625+0000 7f2fd8a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40d0ba0 con 0x7f2fd4160800 2024-01-15T18:09:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.625+0000 7f2fd8a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd4160be0 con 0x7f2fd406a9f0 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.625+0000 7f2fd8a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd406ae60 con 0x7f2fd40ce010 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.625+0000 7f2fd3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2fd406a9f0 0x7f2fd4160da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2fd406a9f0 0x7f2fd4160da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53936/0 (socket says 172.21.15.2:53936) 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd3200640 1 -- 172.21.15.2:0/1266298342 learned_addr learned my addr 172.21.15.2:0/1266298342 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd2800640 1 --1- 172.21.15.2:0/1266298342 >> v1:172.21.15.2:6789/0 conn(0x7f2fd4160800 0x7f2fd40cd900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52602/0 (socket says 172.21.15.2:52602) 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd1e00640 1 --2- 172.21.15.2:0/1266298342 >> v2:172.21.15.137:3300/0 conn(0x7f2fd40ce010 0x7f2fd406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd3200640 1 -- 172.21.15.2:0/1266298342 >> v2:172.21.15.137:3300/0 conn(0x7f2fd40ce010 msgr2=0x7f2fd406a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd3200640 1 --2- 172.21.15.2:0/1266298342 >> v2:172.21.15.137:3300/0 conn(0x7f2fd40ce010 0x7f2fd406a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd3200640 1 -- 172.21.15.2:0/1266298342 >> v1:172.21.15.2:6789/0 conn(0x7f2fd4160800 legacy=0x7f2fd40cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd3200640 1 -- 172.21.15.2:0/1266298342 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd406d250 con 0x7f2fd406a9f0 2024-01-15T18:09:36.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.626+0000 7f2fd1e00640 1 --2- 172.21.15.2:0/1266298342 >> v2:172.21.15.137:3300/0 conn(0x7f2fd40ce010 0x7f2fd406a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:36.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.627+0000 7f2fd3200640 1 --2- 172.21.15.2:0/1266298342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd406a9f0 0x7f2fd4160da0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40c6d80 tx=0x7f2fcc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed858cd1ffb3532 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:36.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd1400640 1 -- 172.21.15.2:0/1266298342 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc02c9f0 con 0x7f2fd406a9f0 2024-01-15T18:09:36.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd1400640 1 -- 172.21.15.2:0/1266298342 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc02cb50 con 0x7f2fd406a9f0 2024-01-15T18:09:36.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd1400640 1 -- 172.21.15.2:0/1266298342 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0343f0 con 0x7f2fd406a9f0 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 -- 172.21.15.2:0/1266298342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd406a9f0 msgr2=0x7f2fd4160da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/1266298342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd406a9f0 0x7f2fd4160da0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2fd40c6d80 tx=0x7f2fcc02be50 comp rx=0 tx=0).stop 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 -- 172.21.15.2:0/1266298342 shutdown_connections 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/1266298342 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd406a9f0 0x7f2fd4160da0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/1266298342 >> v2:172.21.15.137:3300/0 conn(0x7f2fd40ce010 0x7f2fd406a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 -- 172.21.15.2:0/1266298342 >> 172.21.15.2:0/1266298342 conn(0x7f2fd40cfd30 msgr2=0x7f2fd40d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 -- 172.21.15.2:0/1266298342 shutdown_connections 2024-01-15T18:09:36.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.628+0000 7f2fd8a00640 1 -- 172.21.15.2:0/1266298342 wait complete. 2024-01-15T18:09:36.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.630+0000 7f2fd8a00640 1 Processor -- start 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.630+0000 7f2fd8a00640 1 -- start start 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.631+0000 7f2fd8a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd406a9f0 0x7f2fd415c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.631+0000 7f2fd8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd4160800 0x7f2fd4149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.631+0000 7f2fd8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 0x7f2fd415ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.631+0000 7f2fd8a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40acde0 con 0x7f2fd40c6e00 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.631+0000 7f2fd8a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40ad080 con 0x7f2fd406a9f0 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.631+0000 7f2fd8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd40ad360 con 0x7f2fd4160800 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 0x7f2fd415ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 0x7f2fd415ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53948/0 (socket says 172.21.15.2:53948) 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 -- 172.21.15.2:0/2508936176 learned_addr learned my addr 172.21.15.2:0/2508936176 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd2800640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd406a9f0 0x7f2fd415c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 -- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd4160800 msgr2=0x7f2fd4149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd4160800 0x7f2fd4149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 -- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd406a9f0 msgr2=0x7f2fd415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd406a9f0 0x7f2fd415c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.632+0000 7f2fd3200640 1 -- 172.21.15.2:0/2508936176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc009b30 con 0x7f2fd40c6e00 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.633+0000 7f2fd3200640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 0x7f2fd415ee40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009e50 tx=0x7f2fcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.633+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc009510 con 0x7f2fd40c6e00 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.633+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc03e450 con 0x7f2fd40c6e00 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.633+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc03e800 con 0x7f2fd40c6e00 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.633+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd40ad640 con 0x7f2fd40c6e00 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.633+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd40c72b0 con 0x7f2fd40c6e00 2024-01-15T18:09:36.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.634+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd40c7440 con 0x7f2fd40c6e00 2024-01-15T18:09:36.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.635+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2fcc03e960 con 0x7f2fd40c6e00 2024-01-15T18:09:36.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.636+0000 7f2fcaa00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fac06bf50 0x7f2fac06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:36.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.636+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f2fcc0ab670 con 0x7f2fd40c6e00 2024-01-15T18:09:36.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.637+0000 7f2fd2800640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fac06bf50 0x7f2fac06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:36.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.638+0000 7f2fd2800640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fac06bf50 0x7f2fac06e410 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4000f60 tx=0x7f2fc4007c80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:36.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.640+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2fd40c7440 con 0x7f2fd40c6e00 2024-01-15T18:09:36.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.872+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471616", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fd40ad1f0 con 0x7f2fac06bf50 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.894+0000 7f2fcaa00640 1 -- 172.21.15.2:0/2508936176 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f2fd40ad1f0 con 0x7f2fac06bf50 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:36.899 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:36.900 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:36.900 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:36.900 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:36.900 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:36.900 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:36.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fac06bf50 msgr2=0x7f2fac06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fac06bf50 0x7f2fac06e410 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4000f60 tx=0x7f2fc4007c80 comp rx=0 tx=0).stop 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 msgr2=0x7f2fd415ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 0x7f2fd415ee40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc009e50 tx=0x7f2fcc004a40 comp rx=0 tx=0).stop 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 shutdown_connections 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fac06bf50 0x7f2fac06e410 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2fd40c6e00 0x7f2fd415ee40 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2fd4160800 0x7f2fd4149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 --2- 172.21.15.2:0/2508936176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2fd406a9f0 0x7f2fd415c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 >> 172.21.15.2:0/2508936176 conn(0x7f2fd40cfd30 msgr2=0x7f2fd406f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 shutdown_connections 2024-01-15T18:09:36.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:36.897+0000 7f2fd8a00640 1 -- 172.21.15.2:0/2508936176 wait complete. 2024-01-15T18:09:36.921 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:36.922 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471617 2024-01-15T18:09:37.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.137+0000 7f472b200640 1 Processor -- start 2024-01-15T18:09:37.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.137+0000 7f472b200640 1 -- start start 2024-01-15T18:09:37.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.138+0000 7f472b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4724071230 0x7f47240716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.138+0000 7f472b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 0x7f47240cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.138+0000 7f472b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47240b2fa0 con 0x7f4724143620 2024-01-15T18:09:37.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.139+0000 7f472b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4724071bf0 con 0x7f4724071230 2024-01-15T18:09:37.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.139+0000 7f472b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47240b3280 con 0x7f47240cc3b0 2024-01-15T18:09:37.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.139+0000 7f472a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 0x7f47240cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.139+0000 7f472a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 0x7f47240cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:36022/0 (socket says 172.21.15.2:36022) 2024-01-15T18:09:37.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.139+0000 7f472a800640 1 -- 172.21.15.2:0/4271366976 learned_addr learned my addr 172.21.15.2:0/4271366976 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.139+0000 7f4729400640 1 --2- 172.21.15.2:0/4271366976 >> v2:172.21.15.137:3300/0 conn(0x7f4724071230 0x7f47240716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.140+0000 7f4728a00640 1 -- 172.21.15.2:0/4271366976 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645937540 0 0) 0x7f47240b2fa0 con 0x7f4724143620 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.140+0000 7f4728a00640 1 -- 172.21.15.2:0/4271366976 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4704003520 con 0x7f4724143620 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.140+0000 7f472a800640 1 -- 172.21.15.2:0/4271366976 >> v2:172.21.15.137:3300/0 conn(0x7f4724071230 msgr2=0x7f47240716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.140+0000 7f472a800640 1 --2- 172.21.15.2:0/4271366976 >> v2:172.21.15.137:3300/0 conn(0x7f4724071230 0x7f47240716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.140+0000 7f472a800640 1 -- 172.21.15.2:0/4271366976 >> v1:172.21.15.137:6789/0 conn(0x7f4724143620 legacy=0x7f47240b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.140+0000 7f472a800640 1 -- 172.21.15.2:0/4271366976 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47240b3560 con 0x7f47240cc3b0 2024-01-15T18:09:37.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.142+0000 7f472a800640 1 --2- 172.21.15.2:0/4271366976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 0x7f47240cc840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f471c009e50 tx=0x7f471c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce74e78180459432 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:37.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.142+0000 7f4728a00640 1 -- 172.21.15.2:0/4271366976 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c02cb00 con 0x7f47240cc3b0 2024-01-15T18:09:37.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.142+0000 7f4728a00640 1 -- 172.21.15.2:0/4271366976 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c02cc60 con 0x7f47240cc3b0 2024-01-15T18:09:37.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.142+0000 7f4728a00640 1 -- 172.21.15.2:0/4271366976 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c0343f0 con 0x7f47240cc3b0 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 -- 172.21.15.2:0/4271366976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 msgr2=0x7f47240cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 --2- 172.21.15.2:0/4271366976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 0x7f47240cc840 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f471c009e50 tx=0x7f471c02bf60 comp rx=0 tx=0).stop 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 -- 172.21.15.2:0/4271366976 shutdown_connections 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 --2- 172.21.15.2:0/4271366976 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f47240cc3b0 0x7f47240cc840 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 --2- 172.21.15.2:0/4271366976 >> v2:172.21.15.137:3300/0 conn(0x7f4724071230 0x7f47240716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 -- 172.21.15.2:0/4271366976 >> 172.21.15.2:0/4271366976 conn(0x7f47240b0840 msgr2=0x7f47240b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 -- 172.21.15.2:0/4271366976 shutdown_connections 2024-01-15T18:09:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.143+0000 7f472b200640 1 -- 172.21.15.2:0/4271366976 wait complete. 2024-01-15T18:09:37.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.146+0000 7f472b200640 1 Processor -- start 2024-01-15T18:09:37.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.146+0000 7f472b200640 1 -- start start 2024-01-15T18:09:37.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.147+0000 7f472b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 0x7f4724166490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.147+0000 7f472b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240cc3b0 0x7f4724152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.147+0000 7f472b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4724143620 0x7f472413aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.147+0000 7f472b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472413af90 con 0x7f4724071230 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.147+0000 7f472b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47241677a0 con 0x7f47240cc3b0 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.147+0000 7f472b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4724166b90 con 0x7f4724143620 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 0x7f4724166490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 0x7f4724166490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53962/0 (socket says 172.21.15.2:53962) 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 -- 172.21.15.2:0/2295364060 learned_addr learned my addr 172.21.15.2:0/2295364060 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729400640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240cc3b0 0x7f4724152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f472a800640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4724143620 0x7f472413aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 -- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4724143620 msgr2=0x7f472413aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4724143620 0x7f472413aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 -- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240cc3b0 msgr2=0x7f4724152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240cc3b0 0x7f4724152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f4729e00640 1 -- 172.21.15.2:0/2295364060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c009b30 con 0x7f4724071230 2024-01-15T18:09:37.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.148+0000 7f472a800640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4724143620 0x7f472413aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.149+0000 7f4729e00640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 0x7f4724166490 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f471400c430 tx=0x7f471400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.149+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47140052d0 con 0x7f4724071230 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.149+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4724166e70 con 0x7f4724071230 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.149+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4714005450 con 0x7f4724071230 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.149+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4714003830 con 0x7f4724071230 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.149+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4724167430 con 0x7f4724071230 2024-01-15T18:09:37.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.150+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4724074770 con 0x7f4724071230 2024-01-15T18:09:37.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.151+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f4714003990 con 0x7f4724071230 2024-01-15T18:09:37.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.152+0000 7f4722000640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46fc06bf20 0x7f46fc06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.152+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4714087fa0 con 0x7f4724071230 2024-01-15T18:09:37.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.153+0000 7f4729400640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46fc06bf20 0x7f46fc06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.155+0000 7f4729400640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46fc06bf20 0x7f46fc06e3e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f47240cac60 tx=0x7f470c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:37.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.160+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47140544d0 con 0x7f4724071230 2024-01-15T18:09:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.391+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471617", "target": ["mon-mgr", ""]}) v1 -- 0x7f4724143a00 con 0x7f46fc06bf20 2024-01-15T18:09:37.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.414+0000 7f4722000640 1 -- 172.21.15.2:0/2295364060 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f470c009800 con 0x7f46fc06bf20 2024-01-15T18:09:37.419 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:37.419 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:37.420 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:37.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46fc06bf20 msgr2=0x7f46fc06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46fc06bf20 0x7f46fc06e3e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f47240cac60 tx=0x7f470c008040 comp rx=0 tx=0).stop 2024-01-15T18:09:37.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 msgr2=0x7f4724166490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 0x7f4724166490 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f471400c430 tx=0x7f471400c8f0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 shutdown_connections 2024-01-15T18:09:37.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f46fc06bf20 0x7f46fc06e3e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4724143620 0x7f472413aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f47240cc3b0 0x7f4724152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.417+0000 7f472b200640 1 --2- 172.21.15.2:0/2295364060 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4724071230 0x7f4724166490 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.418+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 >> 172.21.15.2:0/2295364060 conn(0x7f47240b0840 msgr2=0x7f472406c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:37.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.418+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 shutdown_connections 2024-01-15T18:09:37.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.418+0000 7f472b200640 1 -- 172.21.15.2:0/2295364060 wait complete. 2024-01-15T18:09:37.440 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:37.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471618 2024-01-15T18:09:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.662+0000 7f49bf200640 1 Processor -- start 2024-01-15T18:09:37.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.663+0000 7f49bf200640 1 -- start start 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.664+0000 7f49bf200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f49c00d1870 0x7f49c00d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.664+0000 7f49bf200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f49c00d1c50 0x7f49c00b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.664+0000 7f49bf200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c00d1250 con 0x7f49c00b2210 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.664+0000 7f49bf200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c00ce6d0 con 0x7f49c00d1c50 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.664+0000 7f49bf200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c00ce8e0 con 0x7f49c00d1870 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f49c00d1c50 0x7f49c00b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f49c00d1c50 0x7f49c00b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53972/0 (socket says 172.21.15.2:53972) 2024-01-15T18:09:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 -- 172.21.15.2:0/133058461 learned_addr learned my addr 172.21.15.2:0/133058461 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:37.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49bde00640 1 --2- 172.21.15.2:0/133058461 >> v2:172.21.15.137:3300/0 conn(0x7f49c00d1870 0x7f49c00d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 -- 172.21.15.2:0/133058461 >> v2:172.21.15.137:3300/0 conn(0x7f49c00d1870 msgr2=0x7f49c00d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 --2- 172.21.15.2:0/133058461 >> v2:172.21.15.137:3300/0 conn(0x7f49c00d1870 0x7f49c00d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 -- 172.21.15.2:0/133058461 >> v1:172.21.15.137:6789/0 conn(0x7f49c00b2210 legacy=0x7f49c015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49be800640 1 -- 172.21.15.2:0/133058461 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c00cebc0 con 0x7f49c00d1c50 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.665+0000 7f49bde00640 1 --2- 172.21.15.2:0/133058461 >> v2:172.21.15.137:3300/0 conn(0x7f49c00d1870 0x7f49c00d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.667+0000 7f49be800640 1 --2- 172.21.15.2:0/133058461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1c50 0x7f49c00b26f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f49c0067890 tx=0x7f49b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b84c4938531262 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:37.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.667+0000 7f49bca00640 1 -- 172.21.15.2:0/133058461 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8030860 con 0x7f49c00d1c50 2024-01-15T18:09:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.667+0000 7f49bca00640 1 -- 172.21.15.2:0/133058461 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b80309c0 con 0x7f49c00d1c50 2024-01-15T18:09:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.667+0000 7f49bca00640 1 -- 172.21.15.2:0/133058461 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8030d50 con 0x7f49c00d1c50 2024-01-15T18:09:37.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 -- 172.21.15.2:0/133058461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1c50 msgr2=0x7f49c00b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 --2- 172.21.15.2:0/133058461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1c50 0x7f49c00b26f0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f49c0067890 tx=0x7f49b802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 -- 172.21.15.2:0/133058461 shutdown_connections 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 --2- 172.21.15.2:0/133058461 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1c50 0x7f49c00b26f0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 --2- 172.21.15.2:0/133058461 >> v2:172.21.15.137:3300/0 conn(0x7f49c00d1870 0x7f49c00d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 -- 172.21.15.2:0/133058461 >> 172.21.15.2:0/133058461 conn(0x7f49c00d30b0 msgr2=0x7f49c00d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 -- 172.21.15.2:0/133058461 shutdown_connections 2024-01-15T18:09:37.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.668+0000 7f49bf200640 1 -- 172.21.15.2:0/133058461 wait complete. 2024-01-15T18:09:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.670+0000 7f49bf200640 1 Processor -- start 2024-01-15T18:09:37.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.670+0000 7f49bf200640 1 -- start start 2024-01-15T18:09:37.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bf200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49c00b2210 0x7f49c00c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 0x7f49c00cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49c00d1c50 0x7f49c0156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bf200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00cb890 con 0x7f49c00d1870 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bf200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00cbb70 con 0x7f49c00b2210 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c00cbe50 con 0x7f49c00d1c50 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bd400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 0x7f49c00cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bd400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 0x7f49c00cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53986/0 (socket says 172.21.15.2:53986) 2024-01-15T18:09:37.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.671+0000 7f49bd400640 1 -- 172.21.15.2:0/1061355933 learned_addr learned my addr 172.21.15.2:0/1061355933 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bde00640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49c00b2210 0x7f49c00c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49be800640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49c00d1c50 0x7f49c0156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bd400640 1 -- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49c00d1c50 msgr2=0x7f49c0156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bd400640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49c00d1c50 0x7f49c0156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bd400640 1 -- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49c00b2210 msgr2=0x7f49c00c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bd400640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49c00b2210 0x7f49c00c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bd400640 1 -- 172.21.15.2:0/1061355933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b80094d0 con 0x7f49c00d1870 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bde00640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49c00b2210 0x7f49c00c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49be800640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49c00d1c50 0x7f49c0156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49bd400640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 0x7f49c00cc930 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c400 tx=0x7f49a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a80052d0 con 0x7f49c00d1870 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8005430 con 0x7f49c00d1870 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.672+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8003830 con 0x7f49c00d1870 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.673+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c00cc130 con 0x7f49c00d1870 2024-01-15T18:09:37.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.673+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c00cce70 con 0x7f49c00d1870 2024-01-15T18:09:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.674+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c00cc650 con 0x7f49c00d1870 2024-01-15T18:09:37.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.675+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f49a8003990 con 0x7f49c00d1870 2024-01-15T18:09:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.676+0000 7f49b6000640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f499806bf50 0x7f499806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.676+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f49a8087e10 con 0x7f49c00d1870 2024-01-15T18:09:37.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.676+0000 7f49bde00640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f499806bf50 0x7f499806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:37.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.678+0000 7f49bde00640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f499806bf50 0x7f499806e410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f49b000d430 tx=0x7f49b0007530 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:37.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.680+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49c00cc650 con 0x7f49c00d1870 2024-01-15T18:09:37.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.914+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471618", "target": ["mon-mgr", ""]}) v1 -- 0x7f49c00cbce0 con 0x7f499806bf50 2024-01-15T18:09:37.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.936+0000 7f49b6000640 1 -- 172.21.15.2:0/1061355933 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f49c00cbce0 con 0x7f499806bf50 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:37.941 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:37.942 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.939+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f499806bf50 msgr2=0x7f499806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.939+0000 7f49bf200640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f499806bf50 0x7f499806e410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f49b000d430 tx=0x7f49b0007530 comp rx=0 tx=0).stop 2024-01-15T18:09:37.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.939+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 msgr2=0x7f49c00cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:37.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.939+0000 7f49bf200640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 0x7f49c00cc930 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c400 tx=0x7f49a800c8c0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.939+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 shutdown_connections 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.939+0000 7f49bf200640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f499806bf50 0x7f499806e410 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.940+0000 7f49bf200640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49c00d1c50 0x7f49c0156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.940+0000 7f49bf200640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49c00d1870 0x7f49c00cc930 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.940+0000 7f49bf200640 1 --2- 172.21.15.2:0/1061355933 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49c00b2210 0x7f49c00c9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.940+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 >> 172.21.15.2:0/1061355933 conn(0x7f49c00d30b0 msgr2=0x7f49c00afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.940+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 shutdown_connections 2024-01-15T18:09:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:37.940+0000 7f49bf200640 1 -- 172.21.15.2:0/1061355933 wait complete. 2024-01-15T18:09:37.963 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:37.963 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471619 2024-01-15T18:09:38.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.188+0000 7f09d4e00640 1 Processor -- start 2024-01-15T18:09:38.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.188+0000 7f09d4e00640 1 -- start start 2024-01-15T18:09:38.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.189+0000 7f09d4e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d015d530 0x7f09d0099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.189+0000 7f09d4e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f09d00cc110 0x7f09d00cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.189+0000 7f09d4e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09d006b5c0 con 0x7f09d00d19a0 2024-01-15T18:09:38.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.189+0000 7f09d4e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09d006ece0 con 0x7f09d00cc110 2024-01-15T18:09:38.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.189+0000 7f09d4e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00cbc80 con 0x7f09d015d530 2024-01-15T18:09:38.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.190+0000 7f09cf400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d015d530 0x7f09d0099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.190+0000 7f09cea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f09d00d19a0 0x7f09d0099f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52618/0 (socket says 172.21.15.2:52618) 2024-01-15T18:09:38.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.190+0000 7f09cea00640 1 -- 172.21.15.2:0/2031374095 learned_addr learned my addr 172.21.15.2:0/2031374095 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:38.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.190+0000 7f09cfe00640 1 --2- 172.21.15.2:0/2031374095 >> v2:172.21.15.137:3300/0 conn(0x7f09d00cc110 0x7f09d00cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.191+0000 7f09ce000640 1 -- 172.21.15.2:0/2031374095 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727403640 0 0) 0x7f09d006b5c0 con 0x7f09d00d19a0 2024-01-15T18:09:38.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.191+0000 7f09ce000640 1 -- 172.21.15.2:0/2031374095 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b0003520 con 0x7f09d00d19a0 2024-01-15T18:09:38.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.191+0000 7f09cf400640 1 -- 172.21.15.2:0/2031374095 >> v2:172.21.15.137:3300/0 conn(0x7f09d00cc110 msgr2=0x7f09d00cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:38.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.191+0000 7f09cf400640 1 --2- 172.21.15.2:0/2031374095 >> v2:172.21.15.137:3300/0 conn(0x7f09d00cc110 0x7f09d00cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.191+0000 7f09cf400640 1 -- 172.21.15.2:0/2031374095 >> v1:172.21.15.2:6789/0 conn(0x7f09d00d19a0 legacy=0x7f09d0099f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.191+0000 7f09cf400640 1 -- 172.21.15.2:0/2031374095 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d006ea40 con 0x7f09d015d530 2024-01-15T18:09:38.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.192+0000 7f09cf400640 1 --2- 172.21.15.2:0/2031374095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d015d530 0x7f09d0099a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f09c00097a0 tx=0x7f09c002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e6f569e45838cc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:38.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.193+0000 7f09ce000640 1 -- 172.21.15.2:0/2031374095 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c002d040 con 0x7f09d015d530 2024-01-15T18:09:38.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.193+0000 7f09ce000640 1 -- 172.21.15.2:0/2031374095 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c0031810 con 0x7f09d015d530 2024-01-15T18:09:38.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.193+0000 7f09ce000640 1 -- 172.21.15.2:0/2031374095 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0031ba0 con 0x7f09d015d530 2024-01-15T18:09:38.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.193+0000 7f09d4e00640 1 -- 172.21.15.2:0/2031374095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d015d530 msgr2=0x7f09d0099a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.193+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2031374095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d015d530 0x7f09d0099a00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f09c00097a0 tx=0x7f09c002b8b0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.194+0000 7f09d4e00640 1 -- 172.21.15.2:0/2031374095 shutdown_connections 2024-01-15T18:09:38.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.194+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2031374095 >> v2:172.21.15.137:3300/0 conn(0x7f09d00cc110 0x7f09d00cc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.194+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2031374095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d015d530 0x7f09d0099a00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.194+0000 7f09d4e00640 1 -- 172.21.15.2:0/2031374095 >> 172.21.15.2:0/2031374095 conn(0x7f09d00d0e40 msgr2=0x7f09d00d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:38.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.194+0000 7f09d4e00640 1 -- 172.21.15.2:0/2031374095 shutdown_connections 2024-01-15T18:09:38.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.194+0000 7f09d4e00640 1 -- 172.21.15.2:0/2031374095 wait complete. 2024-01-15T18:09:38.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.196+0000 7f09d4e00640 1 Processor -- start 2024-01-15T18:09:38.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.196+0000 7f09d4e00640 1 -- start start 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09d4e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d00cc110 0x7f09d00b6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09d4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 0x7f09d0156a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09d4e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d015d530 0x7f09d00aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09d4e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0157250 con 0x7f09d00d19a0 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09d4e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09d0157af0 con 0x7f09d015d530 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09d4e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09d00b67b0 con 0x7f09d00cc110 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09cea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 0x7f09d0156a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09cea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 0x7f09d0156a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53990/0 (socket says 172.21.15.2:53990) 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09cea00640 1 -- 172.21.15.2:0/2185036919 learned_addr learned my addr 172.21.15.2:0/2185036919 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09cf400640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d00cc110 0x7f09d00b6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.198+0000 7f09cfe00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d015d530 0x7f09d00aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09cea00640 1 -- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d00cc110 msgr2=0x7f09d00b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09cea00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d00cc110 0x7f09d00b6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09cea00640 1 -- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d015d530 msgr2=0x7f09d00aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09cea00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d015d530 0x7f09d00aeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09cea00640 1 -- 172.21.15.2:0/2185036919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009400 con 0x7f09d00d19a0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09cea00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 0x7f09d0156a50 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f09b800cbf0 tx=0x7f09b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b800ad40 con 0x7f09d00d19a0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b8012400 con 0x7f09d00d19a0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b800aea0 con 0x7f09d00d19a0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.199+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09d00cb3d0 con 0x7f09d00d19a0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.200+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09d01574a0 con 0x7f09d00d19a0 2024-01-15T18:09:38.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.201+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09d0157ca0 con 0x7f09d00d19a0 2024-01-15T18:09:38.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.202+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f09b8007070 con 0x7f09d00d19a0 2024-01-15T18:09:38.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.203+0000 7f09c7400640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f09a806bf20 0x7f09a806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.203+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f09b8088870 con 0x7f09d00d19a0 2024-01-15T18:09:38.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.204+0000 7f09cf400640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f09a806bf20 0x7f09a806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.205+0000 7f09cf400640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f09a806bf20 0x7f09a806e3e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036000 tx=0x7f09c003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:38.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.208+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09d0157ca0 con 0x7f09d00d19a0 2024-01-15T18:09:38.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.449+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471619", "target": ["mon-mgr", ""]}) v1 -- 0x7f09d006e6c0 con 0x7f09a806bf20 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.470+0000 7f09c7400640 1 -- 172.21.15.2:0/2185036919 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f09c0031ad0 con 0x7f09a806bf20 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:38.474 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:38.475 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:38.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.472+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f09a806bf20 msgr2=0x7f09a806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.472+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f09a806bf20 0x7f09a806e3e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f09c0036000 tx=0x7f09c003c040 comp rx=0 tx=0).stop 2024-01-15T18:09:38.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.472+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 msgr2=0x7f09d0156a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.472+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 0x7f09d0156a50 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f09b800cbf0 tx=0x7f09b800a430 comp rx=0 tx=0).stop 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 shutdown_connections 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f09a806bf20 0x7f09a806e3e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f09d015d530 0x7f09d00aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f09d00d19a0 0x7f09d0156a50 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 --2- 172.21.15.2:0/2185036919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f09d00cc110 0x7f09d00b6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 >> 172.21.15.2:0/2185036919 conn(0x7f09d00d0e40 msgr2=0x7f09d006eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 shutdown_connections 2024-01-15T18:09:38.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.473+0000 7f09d4e00640 1 -- 172.21.15.2:0/2185036919 wait complete. 2024-01-15T18:09:38.497 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-15T18:09:38.497 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone471620 2024-01-15T18:09:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.724+0000 7fd2a9600640 1 Processor -- start 2024-01-15T18:09:38.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.724+0000 7fd2a9600640 1 -- start start 2024-01-15T18:09:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd2a41439f0 0x7fd2a40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a9600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd2a40b3420 0x7fd2a40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a40ccc70 0x7fd2a406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a9600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a415dd70 con 0x7fd2a40b3420 2024-01-15T18:09:38.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40b38b0 con 0x7fd2a41439f0 2024-01-15T18:09:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40711b0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a2000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd2a40b3420 0x7fd2a40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a2000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd2a40b3420 0x7fd2a40cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53994/0 (socket says 172.21.15.2:53994) 2024-01-15T18:09:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a2000640 1 -- 172.21.15.2:0/891137928 learned_addr learned my addr 172.21.15.2:0/891137928 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a2a00640 1 --2- 172.21.15.2:0/891137928 >> v2:172.21.15.137:3300/0 conn(0x7fd2a41439f0 0x7fd2a40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a3400640 1 --2- 172.21.15.2:0/891137928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a40ccc70 0x7fd2a406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.726+0000 7fd2a2000640 1 -- 172.21.15.2:0/891137928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a40ccc70 msgr2=0x7fd2a406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.727+0000 7fd2a2000640 1 --2- 172.21.15.2:0/891137928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a40ccc70 0x7fd2a406c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.727+0000 7fd2a2000640 1 -- 172.21.15.2:0/891137928 >> v2:172.21.15.137:3300/0 conn(0x7fd2a41439f0 msgr2=0x7fd2a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.727+0000 7fd2a2000640 1 --2- 172.21.15.2:0/891137928 >> v2:172.21.15.137:3300/0 conn(0x7fd2a41439f0 0x7fd2a40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.727+0000 7fd2a2000640 1 -- 172.21.15.2:0/891137928 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a4143dd0 con 0x7fd2a40b3420 2024-01-15T18:09:38.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.727+0000 7fd2a2a00640 1 --2- 172.21.15.2:0/891137928 >> v2:172.21.15.137:3300/0 conn(0x7fd2a41439f0 0x7fd2a40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.728+0000 7fd2a2000640 1 --2- 172.21.15.2:0/891137928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40b3420 0x7fd2a40cc730 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40b28d0 tx=0x7fd28c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7763d7c79a4938e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.728+0000 7fd2a1600640 1 -- 172.21.15.2:0/891137928 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c031860 con 0x7fd2a40b3420 2024-01-15T18:09:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.728+0000 7fd2a1600640 1 -- 172.21.15.2:0/891137928 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c0319c0 con 0x7fd2a40b3420 2024-01-15T18:09:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.728+0000 7fd2a1600640 1 -- 172.21.15.2:0/891137928 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c031d50 con 0x7fd2a40b3420 2024-01-15T18:09:38.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 -- 172.21.15.2:0/891137928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40b3420 msgr2=0x7fd2a40cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 --2- 172.21.15.2:0/891137928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40b3420 0x7fd2a40cc730 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40b28d0 tx=0x7fd28c02b900 comp rx=0 tx=0).stop 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 -- 172.21.15.2:0/891137928 shutdown_connections 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 --2- 172.21.15.2:0/891137928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a40ccc70 0x7fd2a406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 --2- 172.21.15.2:0/891137928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40b3420 0x7fd2a40cc730 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 --2- 172.21.15.2:0/891137928 >> v2:172.21.15.137:3300/0 conn(0x7fd2a41439f0 0x7fd2a40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 -- 172.21.15.2:0/891137928 >> 172.21.15.2:0/891137928 conn(0x7fd2a40b45f0 msgr2=0x7fd2a40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 -- 172.21.15.2:0/891137928 shutdown_connections 2024-01-15T18:09:38.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.729+0000 7fd2a9600640 1 -- 172.21.15.2:0/891137928 wait complete. 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.731+0000 7fd2a9600640 1 Processor -- start 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.731+0000 7fd2a9600640 1 -- start start 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2a40b3420 0x7fd2a4159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 0x7fd2a40ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a41439f0 0x7fd2a40ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a9600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a415cbc0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a415c730 con 0x7fd2a40b3420 2024-01-15T18:09:38.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a415bb20 con 0x7fd2a41439f0 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 0x7fd2a40ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 0x7fd2a40ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54010/0 (socket says 172.21.15.2:54010) 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.732+0000 7fd2a2000640 1 -- 172.21.15.2:0/3125582674 learned_addr learned my addr 172.21.15.2:0/3125582674 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a3400640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a41439f0 0x7fd2a40ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2000640 1 -- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a41439f0 msgr2=0x7fd2a40ab000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2a00640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2a40b3420 0x7fd2a4159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2000640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a41439f0 0x7fd2a40ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2000640 1 -- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2a40b3420 msgr2=0x7fd2a4159c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2000640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2a40b3420 0x7fd2a4159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:38.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2000640 1 -- 172.21.15.2:0/3125582674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c0094d0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd2a2000640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 0x7fd2a40ade00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd28c02b940 tx=0x7fd28c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:38.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.733+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c031850 con 0x7fd2a40ccc70 2024-01-15T18:09:38.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.734+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c0443f0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.734+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0447a0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.734+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a415be00 con 0x7fd2a40ccc70 2024-01-15T18:09:38.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.734+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a415c3c0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.735+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a415c8c0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.740 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.736+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fd28c04d450 con 0x7fd2a40ccc70 2024-01-15T18:09:38.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.737+0000 7fd29aa00640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd27c06bf20 0x7fd27c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:38.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.737+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fd28c0aafa0 con 0x7fd2a40ccc70 2024-01-15T18:09:38.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.738+0000 7fd2a2a00640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd27c06bf20 0x7fd27c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:38.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.739+0000 7fd2a2a00640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd27c06bf20 0x7fd27c06e3e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40cb390 tx=0x7fd294008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:38.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.741+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd28c0b3020 con 0x7fd2a40ccc70 2024-01-15T18:09:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.969+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone471620", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2a4070ed0 con 0x7fd27c06bf20 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.992+0000 7fd29aa00640 1 -- 172.21.15.2:0/3125582674 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fd29400c040 con 0x7fd27c06bf20 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:09:38.997 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:38.998 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:09:38.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd27c06bf20 msgr2=0x7fd27c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:38.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd27c06bf20 0x7fd27c06e3e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40cb390 tx=0x7fd294008040 comp rx=0 tx=0).stop 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 msgr2=0x7fd2a40ade00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 0x7fd2a40ade00 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7fd28c02b940 tx=0x7fd28c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 shutdown_connections 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd27c06bf20 0x7fd27c06e3e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd2a41439f0 0x7fd2a40ab000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd2a40ccc70 0x7fd2a40ade00 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 --2- 172.21.15.2:0/3125582674 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd2a40b3420 0x7fd2a4159c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 >> 172.21.15.2:0/3125582674 conn(0x7fd2a40b45f0 msgr2=0x7fd2a415a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 shutdown_connections 2024-01-15T18:09:39.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:38.995+0000 7fd2a9600640 1 -- 172.21.15.2:0/3125582674 wait complete. 2024-01-15T18:09:39.018 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:09:39.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.238+0000 7faf4da00640 1 Processor -- start 2024-01-15T18:09:39.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.239+0000 7faf4da00640 1 -- start start 2024-01-15T18:09:39.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.240+0000 7faf4da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d2de0 0x7faf480d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.240+0000 7faf4da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faf480d0ed0 0x7faf480d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.240+0000 7faf4da00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf480b30e0 con 0x7faf4815d200 2024-01-15T18:09:39.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.240+0000 7faf4da00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf480d3780 con 0x7faf480d0ed0 2024-01-15T18:09:39.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.240+0000 7faf4da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf480d1ca0 con 0x7faf480d2de0 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faf480d0ed0 0x7faf480d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7faf480d0ed0 0x7faf480d1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54018/0 (socket says 172.21.15.2:54018) 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 -- 172.21.15.2:0/1557840803 learned_addr learned my addr 172.21.15.2:0/1557840803 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf47400640 1 --2- 172.21.15.2:0/1557840803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d2de0 0x7faf480d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 -- 172.21.15.2:0/1557840803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d2de0 msgr2=0x7faf480d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 --2- 172.21.15.2:0/1557840803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d2de0 0x7faf480d3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 -- 172.21.15.2:0/1557840803 >> v1:172.21.15.2:6789/0 conn(0x7faf4815d200 legacy=0x7faf480cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf4d000640 1 -- 172.21.15.2:0/1557840803 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf480d1f80 con 0x7faf480d0ed0 2024-01-15T18:09:39.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.241+0000 7faf46a00640 1 -- 172.21.15.2:0/1557840803 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1093207026 0 0) 0x7faf480b30e0 con 0x7faf4815d200 2024-01-15T18:09:39.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.242+0000 7faf4d000640 1 --2- 172.21.15.2:0/1557840803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d0ed0 0x7faf480d1360 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7faf40009870 tx=0x7faf4002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bc7b9a1bbee3d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:39.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf46a00640 1 -- 172.21.15.2:0/1557840803 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf40030840 con 0x7faf480d0ed0 2024-01-15T18:09:39.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf46a00640 1 -- 172.21.15.2:0/1557840803 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf400309a0 con 0x7faf480d0ed0 2024-01-15T18:09:39.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf46a00640 1 -- 172.21.15.2:0/1557840803 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf40030d30 con 0x7faf480d0ed0 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 -- 172.21.15.2:0/1557840803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d0ed0 msgr2=0x7faf480d1360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 --2- 172.21.15.2:0/1557840803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d0ed0 0x7faf480d1360 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7faf40009870 tx=0x7faf4002b980 comp rx=0 tx=0).stop 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 -- 172.21.15.2:0/1557840803 shutdown_connections 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 --2- 172.21.15.2:0/1557840803 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d0ed0 0x7faf480d1360 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 --2- 172.21.15.2:0/1557840803 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d2de0 0x7faf480d3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 -- 172.21.15.2:0/1557840803 >> 172.21.15.2:0/1557840803 conn(0x7faf4808caf0 msgr2=0x7faf4809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.243+0000 7faf4da00640 1 -- 172.21.15.2:0/1557840803 shutdown_connections 2024-01-15T18:09:39.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.244+0000 7faf4da00640 1 -- 172.21.15.2:0/1557840803 wait complete. 2024-01-15T18:09:39.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.245+0000 7faf4da00640 1 Processor -- start 2024-01-15T18:09:39.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.245+0000 7faf4da00640 1 -- start start 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.246+0000 7faf4da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d0ed0 0x7faf480b7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.246+0000 7faf4da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 0x7faf48158590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.246+0000 7faf4da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf4815d200 0x7faf48159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.246+0000 7faf4da00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b7fc0 con 0x7faf480d2de0 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.246+0000 7faf4da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b82a0 con 0x7faf4815d200 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.246+0000 7faf4da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf480b8580 con 0x7faf480d0ed0 2024-01-15T18:09:39.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 0x7faf48158590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 0x7faf48158590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54030/0 (socket says 172.21.15.2:54030) 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 -- 172.21.15.2:0/1171032404 learned_addr learned my addr 172.21.15.2:0/1171032404 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf4d000640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf4815d200 0x7faf48159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47e00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d0ed0 0x7faf480b7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 -- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d0ed0 msgr2=0x7faf480b7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d0ed0 0x7faf480b7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 -- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf4815d200 msgr2=0x7faf48159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf4815d200 0x7faf48159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 -- 172.21.15.2:0/1171032404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf400094d0 con 0x7faf480d2de0 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47e00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d0ed0 0x7faf480b7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf4d000640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf4815d200 0x7faf48159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.247+0000 7faf47400640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 0x7faf48158590 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7faf3000d800 tx=0x7faf3000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:39.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.248+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3000b6c0 con 0x7faf480d2de0 2024-01-15T18:09:39.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.248+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3000b820 con 0x7faf480d2de0 2024-01-15T18:09:39.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.248+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3000bbb0 con 0x7faf480d2de0 2024-01-15T18:09:39.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.248+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf480b8860 con 0x7faf480d2de0 2024-01-15T18:09:39.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.248+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf480b7cb0 con 0x7faf480d2de0 2024-01-15T18:09:39.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.249+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf480b2840 con 0x7faf480d2de0 2024-01-15T18:09:39.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.250+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7faf3000bd10 con 0x7faf480d2de0 2024-01-15T18:09:39.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.251+0000 7faf3fe00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf2006bf20 0x7faf2006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.251+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7faf30088bd0 con 0x7faf480d2de0 2024-01-15T18:09:39.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.252+0000 7faf47e00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf2006bf20 0x7faf2006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.253+0000 7faf47e00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf2006bf20 0x7faf2006e3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7faf380012e0 tx=0x7faf38009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:39.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.255+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf30055400 con 0x7faf480d2de0 2024-01-15T18:09:39.468 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.464+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7faf480b8410 con 0x7faf2006bf20 2024-01-15T18:09:39.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.468+0000 7faf3fe00640 1 -- 172.21.15.2:0/1171032404 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7faf480b8410 con 0x7faf2006bf20 2024-01-15T18:09:39.472 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:09:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:09:39.472 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:39.472 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:09:39.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf2006bf20 msgr2=0x7faf2006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf2006bf20 0x7faf2006e3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7faf380012e0 tx=0x7faf38009040 comp rx=0 tx=0).stop 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 msgr2=0x7faf48158590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 0x7faf48158590 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7faf3000d800 tx=0x7faf3000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 shutdown_connections 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7faf2006bf20 0x7faf2006e3e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7faf4815d200 0x7faf48159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7faf480d2de0 0x7faf48158590 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 --2- 172.21.15.2:0/1171032404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7faf480d0ed0 0x7faf480b7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 >> 172.21.15.2:0/1171032404 conn(0x7faf4808caf0 msgr2=0x7faf480cdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 shutdown_connections 2024-01-15T18:09:39.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.470+0000 7faf4da00640 1 -- 172.21.15.2:0/1171032404 wait complete. 2024-01-15T18:09:39.493 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:09:39.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.709+0000 7feca7800640 1 Processor -- start 2024-01-15T18:09:39.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.709+0000 7feca7800640 1 -- start start 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7feca7800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca014afd0 0x7feca00b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7feca7800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feca014b3b0 0x7feca006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7feca7800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca006d270 con 0x7feca006ce30 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7feca7800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca0070ff0 con 0x7feca014b3b0 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7feca7800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca00b25f0 con 0x7feca014afd0 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7fec9fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feca014b3b0 0x7feca006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7fec9fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7feca014b3b0 0x7feca006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54040/0 (socket says 172.21.15.2:54040) 2024-01-15T18:09:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.711+0000 7fec9fe00640 1 -- 172.21.15.2:0/2364929433 learned_addr learned my addr 172.21.15.2:0/2364929433 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.712+0000 7feca4e00640 1 --2- 172.21.15.2:0/2364929433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca014afd0 0x7feca00b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.712+0000 7fec9fe00640 1 -- 172.21.15.2:0/2364929433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca014afd0 msgr2=0x7feca00b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.712+0000 7fec9fe00640 1 --2- 172.21.15.2:0/2364929433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca014afd0 0x7feca00b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.712+0000 7fec9fe00640 1 -- 172.21.15.2:0/2364929433 >> v1:172.21.15.137:6789/0 conn(0x7feca006ce30 legacy=0x7feca00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:39.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.712+0000 7fec9fe00640 1 -- 172.21.15.2:0/2364929433 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca00b28d0 con 0x7feca014b3b0 2024-01-15T18:09:39.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.713+0000 7fec9fe00640 1 --2- 172.21.15.2:0/2364929433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca006c8f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fec88009870 tx=0x7fec8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=696c2f98d5a27ff6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:39.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.713+0000 7fec9f400640 1 -- 172.21.15.2:0/2364929433 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec880317c0 con 0x7feca014b3b0 2024-01-15T18:09:39.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7fec9f400640 1 -- 172.21.15.2:0/2364929433 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec88031920 con 0x7feca014b3b0 2024-01-15T18:09:39.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7fec9f400640 1 -- 172.21.15.2:0/2364929433 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88031cb0 con 0x7feca014b3b0 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7feca7800640 1 -- 172.21.15.2:0/2364929433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 msgr2=0x7feca006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7feca7800640 1 --2- 172.21.15.2:0/2364929433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca006c8f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fec88009870 tx=0x7fec8802b860 comp rx=0 tx=0).stop 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7feca7800640 1 -- 172.21.15.2:0/2364929433 shutdown_connections 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7feca7800640 1 --2- 172.21.15.2:0/2364929433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca006c8f0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7feca7800640 1 --2- 172.21.15.2:0/2364929433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca014afd0 0x7feca00b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.714+0000 7feca7800640 1 -- 172.21.15.2:0/2364929433 >> 172.21.15.2:0/2364929433 conn(0x7feca00b45f0 msgr2=0x7feca00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.715+0000 7feca7800640 1 -- 172.21.15.2:0/2364929433 shutdown_connections 2024-01-15T18:09:39.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.715+0000 7feca7800640 1 -- 172.21.15.2:0/2364929433 wait complete. 2024-01-15T18:09:39.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.716+0000 7feca7800640 1 Processor -- start 2024-01-15T18:09:39.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.716+0000 7feca7800640 1 -- start start 2024-01-15T18:09:39.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.717+0000 7feca7800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca006ce30 0x7feca0144980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.717+0000 7feca7800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca014afd0 0x7feca00a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.717+0000 7feca7800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca00cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.717+0000 7feca7800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0144ec0 con 0x7feca014b3b0 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.717+0000 7feca7800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca00aa760 con 0x7feca014afd0 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.717+0000 7feca7800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca00a8c90 con 0x7feca006ce30 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca00cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca00cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54046/0 (socket says 172.21.15.2:54046) 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 -- 172.21.15.2:0/69442118 learned_addr learned my addr 172.21.15.2:0/69442118 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca4e00640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca006ce30 0x7feca0144980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7fec9fe00640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca014afd0 0x7feca00a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 -- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca006ce30 msgr2=0x7feca0144980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca006ce30 0x7feca0144980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 -- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca014afd0 msgr2=0x7feca00a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:39.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca014afd0 0x7feca00a8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 -- 172.21.15.2:0/69442118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec880094d0 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca4e00640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca006ce30 0x7feca0144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.718+0000 7feca5800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca00cd880 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fec9800b970 tx=0x7fec9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.719+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec9800cb40 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.719+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec9800cca0 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.719+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98014680 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.719+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca00a8f70 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.719+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca00aa9b0 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.720+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca00686f0 con 0x7feca014b3b0 2024-01-15T18:09:39.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.721+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fec9801d3b0 con 0x7feca014b3b0 2024-01-15T18:09:39.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.722+0000 7fec9cc00640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec7806bf20 0x7fec7806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:39.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.722+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fec98088690 con 0x7feca014b3b0 2024-01-15T18:09:39.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.723+0000 7feca4e00640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec7806bf20 0x7fec7806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:39.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.724+0000 7feca4e00640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec7806bf20 0x7fec7806e3e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fec90006fd0 tx=0x7fec90008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:39.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.726+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec98054cd0 con 0x7feca014b3b0 2024-01-15T18:09:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.947+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feca00b3ba0 con 0x7fec7806bf20 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.993+0000 7fec9cc00640 1 -- 172.21.15.2:0/69442118 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1029 (secure 0 0 0) 0x7feca00b3ba0 con 0x7fec7806bf20 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47162" 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:39.998 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471615" 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47161" 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471618" 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47169" 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:39.999 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47164" 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471610" 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471616" 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471614" 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471620" 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.000 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471613" 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471611" 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47166" 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471617" 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471612" 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.001 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47168" 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47165" 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47163" 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone47167" 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone471619" 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:40.002 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:09:40.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec7806bf20 msgr2=0x7fec7806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec7806bf20 0x7fec7806e3e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fec90006fd0 tx=0x7fec90008040 comp rx=0 tx=0).stop 2024-01-15T18:09:40.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 msgr2=0x7feca00cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca00cd880 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7fec9800b970 tx=0x7fec9800be30 comp rx=0 tx=0).stop 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 shutdown_connections 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fec7806bf20 0x7fec7806e3e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7feca014b3b0 0x7feca00cd880 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7feca014afd0 0x7feca00a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 --2- 172.21.15.2:0/69442118 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7feca006ce30 0x7feca0144980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 >> 172.21.15.2:0/69442118 conn(0x7feca00b45f0 msgr2=0x7feca01458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.996+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 shutdown_connections 2024-01-15T18:09:40.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:39.997+0000 7feca7800640 1 -- 172.21.15.2:0/69442118 wait complete. 2024-01-15T18:09:40.019 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47162 2024-01-15T18:09:40.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.241+0000 7fde69000640 1 Processor -- start 2024-01-15T18:09:40.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.241+0000 7fde69000640 1 -- start start 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.243+0000 7fde69000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fde640b32b0 0x7fde640d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.243+0000 7fde69000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fde6415d9a0 0x7fde6415de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.243+0000 7fde69000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6406f440 con 0x7fde64146ba0 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.243+0000 7fde69000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde64146f80 con 0x7fde6415d9a0 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.243+0000 7fde69000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6415e390 con 0x7fde640b32b0 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde62800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fde6415d9a0 0x7fde6415de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde61e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fde64146ba0 0x7fde640b2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52624/0 (socket says 172.21.15.2:52624) 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde61e00640 1 -- 172.21.15.2:0/3965745218 learned_addr learned my addr 172.21.15.2:0/3965745218 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde61400640 1 --2- 172.21.15.2:0/3965745218 >> v2:172.21.15.137:3300/0 conn(0x7fde640b32b0 0x7fde640d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde62800640 1 -- 172.21.15.2:0/3965745218 >> v2:172.21.15.137:3300/0 conn(0x7fde640b32b0 msgr2=0x7fde640d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde62800640 1 --2- 172.21.15.2:0/3965745218 >> v2:172.21.15.137:3300/0 conn(0x7fde640b32b0 0x7fde640d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde62800640 1 -- 172.21.15.2:0/3965745218 >> v1:172.21.15.2:6789/0 conn(0x7fde64146ba0 legacy=0x7fde640b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde62800640 1 -- 172.21.15.2:0/3965745218 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6415e5f0 con 0x7fde6415d9a0 2024-01-15T18:09:40.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.244+0000 7fde60a00640 1 -- 172.21.15.2:0/3965745218 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250270253 0 0) 0x7fde6406f440 con 0x7fde64146ba0 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.245+0000 7fde62800640 1 --2- 172.21.15.2:0/3965745218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde6415d9a0 0x7fde6415de50 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fde64071f70 tx=0x7fde5c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6c2aaae51ba05a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde60a00640 1 -- 172.21.15.2:0/3965745218 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c02e070 con 0x7fde6415d9a0 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde60a00640 1 -- 172.21.15.2:0/3965745218 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c0315a0 con 0x7fde6415d9a0 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde60a00640 1 -- 172.21.15.2:0/3965745218 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c031930 con 0x7fde6415d9a0 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 -- 172.21.15.2:0/3965745218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde6415d9a0 msgr2=0x7fde6415de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 --2- 172.21.15.2:0/3965745218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde6415d9a0 0x7fde6415de50 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7fde64071f70 tx=0x7fde5c02b840 comp rx=0 tx=0).stop 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 -- 172.21.15.2:0/3965745218 shutdown_connections 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 --2- 172.21.15.2:0/3965745218 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde6415d9a0 0x7fde6415de50 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 --2- 172.21.15.2:0/3965745218 >> v2:172.21.15.137:3300/0 conn(0x7fde640b32b0 0x7fde640d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 -- 172.21.15.2:0/3965745218 >> 172.21.15.2:0/3965745218 conn(0x7fde640b6430 msgr2=0x7fde640b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:40.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 -- 172.21.15.2:0/3965745218 shutdown_connections 2024-01-15T18:09:40.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.246+0000 7fde69000640 1 -- 172.21.15.2:0/3965745218 wait complete. 2024-01-15T18:09:40.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.248+0000 7fde69000640 1 Processor -- start 2024-01-15T18:09:40.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.248+0000 7fde69000640 1 -- start start 2024-01-15T18:09:40.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde69000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 0x7fde640affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde69000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde64146ba0 0x7fde64134bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde69000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde6415d9a0 0x7fde64135130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde69000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640cfb60 con 0x7fde640b32b0 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde69000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640cf8a0 con 0x7fde6415d9a0 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde69000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde64132cf0 con 0x7fde64146ba0 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.249+0000 7fde61e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 0x7fde640affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 0x7fde640affa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54056/0 (socket says 172.21.15.2:54056) 2024-01-15T18:09:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 -- 172.21.15.2:0/216073641 learned_addr learned my addr 172.21.15.2:0/216073641 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:40.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde62800640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde6415d9a0 0x7fde64135130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61400640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde64146ba0 0x7fde64134bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 -- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde64146ba0 msgr2=0x7fde64134bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde64146ba0 0x7fde64134bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 -- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde6415d9a0 msgr2=0x7fde64135130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde6415d9a0 0x7fde64135130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 -- 172.21.15.2:0/216073641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c009400 con 0x7fde640b32b0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde61e00640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 0x7fde640affa0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fde5400c4d0 tx=0x7fde5400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.250+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde540074e0 con 0x7fde640b32b0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.251+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde54007660 con 0x7fde640b32b0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.251+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54004a20 con 0x7fde640b32b0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.251+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde64132fd0 con 0x7fde640b32b0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.251+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde64135670 con 0x7fde640b32b0 2024-01-15T18:09:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.252+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde641334f0 con 0x7fde640b32b0 2024-01-15T18:09:40.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.253+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fde54004b80 con 0x7fde640b32b0 2024-01-15T18:09:40.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.254+0000 7fde5a000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde3c06bf50 0x7fde3c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.254+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fde54087a40 con 0x7fde640b32b0 2024-01-15T18:09:40.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.254+0000 7fde61400640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde3c06bf50 0x7fde3c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.256+0000 7fde61400640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde3c06bf50 0x7fde3c06e410 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fde640cceb0 tx=0x7fde4c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:40.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.261+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde641334f0 con 0x7fde640b32b0 2024-01-15T18:09:40.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.481+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47162", "target": ["mon-mgr", ""]}) v1 -- 0x7fde640d1260 con 0x7fde3c06bf50 2024-01-15T18:09:40.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.533+0000 7fde5a000640 1 -- 172.21.15.2:0/216073641 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fde4c007e10 con 0x7fde3c06bf50 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde3c06bf50 msgr2=0x7fde3c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde3c06bf50 0x7fde3c06e410 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fde640cceb0 tx=0x7fde4c00b040 comp rx=0 tx=0).stop 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 msgr2=0x7fde640affa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 0x7fde640affa0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7fde5400c4d0 tx=0x7fde5400c990 comp rx=0 tx=0).stop 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 shutdown_connections 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde3c06bf50 0x7fde3c06e410 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde6415d9a0 0x7fde64135130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde64146ba0 0x7fde64134bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 --2- 172.21.15.2:0/216073641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde640b32b0 0x7fde640affa0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 >> 172.21.15.2:0/216073641 conn(0x7fde640b6430 msgr2=0x7fde640b2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 shutdown_connections 2024-01-15T18:09:40.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.535+0000 7fde69000640 1 -- 172.21.15.2:0/216073641 wait complete. 2024-01-15T18:09:40.545 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.541+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.545 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.541+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.545 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.541+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.545 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.541+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.546 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.541+0000 7fe20fc00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.557 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471615 2024-01-15T18:09:40.568 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.564+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.568 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.564+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.568 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.564+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.568 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.564+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.568 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:09:40.564+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:09:40.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.747+0000 7f4d05a00640 1 Processor -- start 2024-01-15T18:09:40.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.747+0000 7f4d05a00640 1 -- start start 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.748+0000 7f4d05a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4d00144c00 0x7f4d000cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.748+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d000cdd70 0x7f4d0006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.748+0000 7f4d05a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d0015c450 0x7f4d0006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.748+0000 7f4d05a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d000ce230 con 0x7f4d0015c450 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.748+0000 7f4d05a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d000ce510 con 0x7f4d00144c00 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.748+0000 7f4d05a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d000ce7f0 con 0x7f4d000cdd70 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d0015c450 0x7f4d0006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d0015c450 0x7f4d0006ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54058/0 (socket says 172.21.15.2:54058) 2024-01-15T18:09:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 -- 172.21.15.2:0/1107547439 learned_addr learned my addr 172.21.15.2:0/1107547439 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cf7e00640 1 --2- 172.21.15.2:0/1107547439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d000cdd70 0x7f4d0006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cfea00640 1 --2- 172.21.15.2:0/1107547439 >> v2:172.21.15.137:3300/0 conn(0x7f4d00144c00 0x7f4d000cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 -- 172.21.15.2:0/1107547439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d000cdd70 msgr2=0x7f4d0006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 --2- 172.21.15.2:0/1107547439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d000cdd70 0x7f4d0006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 -- 172.21.15.2:0/1107547439 >> v2:172.21.15.137:3300/0 conn(0x7f4d00144c00 msgr2=0x7f4d000cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 --2- 172.21.15.2:0/1107547439 >> v2:172.21.15.137:3300/0 conn(0x7f4d00144c00 0x7f4d000cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cff400640 1 -- 172.21.15.2:0/1107547439 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d000705f0 con 0x7f4d0015c450 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.749+0000 7f4cfea00640 1 --2- 172.21.15.2:0/1107547439 >> v2:172.21.15.137:3300/0 conn(0x7f4d00144c00 0x7f4d000cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:40.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.750+0000 7f4cff400640 1 --2- 172.21.15.2:0/1107547439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d0015c450 0x7f4d0006ce30 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f4d000b1c60 tx=0x7f4cf802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5f38e519a6af34 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4cfe000640 1 -- 172.21.15.2:0/1107547439 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf802ca80 con 0x7f4d0015c450 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4cfe000640 1 -- 172.21.15.2:0/1107547439 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf802cbe0 con 0x7f4d0015c450 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4cfe000640 1 -- 172.21.15.2:0/1107547439 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf80343f0 con 0x7f4d0015c450 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 -- 172.21.15.2:0/1107547439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d0015c450 msgr2=0x7f4d0006ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 --2- 172.21.15.2:0/1107547439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d0015c450 0x7f4d0006ce30 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f4d000b1c60 tx=0x7f4cf802bee0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 -- 172.21.15.2:0/1107547439 shutdown_connections 2024-01-15T18:09:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 --2- 172.21.15.2:0/1107547439 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d0015c450 0x7f4d0006ce30 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 --2- 172.21.15.2:0/1107547439 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d000cdd70 0x7f4d0006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 --2- 172.21.15.2:0/1107547439 >> v2:172.21.15.137:3300/0 conn(0x7f4d00144c00 0x7f4d000cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.751+0000 7f4d05a00640 1 -- 172.21.15.2:0/1107547439 >> 172.21.15.2:0/1107547439 conn(0x7f4d000b45f0 msgr2=0x7f4d000b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.752+0000 7f4d05a00640 1 -- 172.21.15.2:0/1107547439 shutdown_connections 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.752+0000 7f4d05a00640 1 -- 172.21.15.2:0/1107547439 wait complete. 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.753+0000 7f4d05a00640 1 Processor -- start 2024-01-15T18:09:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.753+0000 7f4d05a00640 1 -- start start 2024-01-15T18:09:40.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.754+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 0x7f4d000ada10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.754+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d00144c00 0x7f4d00143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.754+0000 7f4d05a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d0015c450 0x7f4d00144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.754+0000 7f4d05a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0013c330 con 0x7f4d000cdd70 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.754+0000 7f4d05a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0013fc80 con 0x7f4d0015c450 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.754+0000 7f4d05a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0014d420 con 0x7f4d00144c00 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 0x7f4d000ada10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 0x7f4d000ada10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54074/0 (socket says 172.21.15.2:54074) 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 -- 172.21.15.2:0/3830446057 learned_addr learned my addr 172.21.15.2:0/3830446057 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cf7e00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d00144c00 0x7f4d00143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cff400640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d0015c450 0x7f4d00144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 -- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d00144c00 msgr2=0x7f4d00143ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d00144c00 0x7f4d00143ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 -- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d0015c450 msgr2=0x7f4d00144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d0015c450 0x7f4d00144000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:40.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.755+0000 7f4cfea00640 1 -- 172.21.15.2:0/3830446057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf8009b30 con 0x7f4d000cdd70 2024-01-15T18:09:40.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.756+0000 7f4cfea00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 0x7f4d000ada10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000c7f0 tx=0x7f4cf000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:40.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.756+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0007a30 con 0x7f4d000cdd70 2024-01-15T18:09:40.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.756+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d0014d670 con 0x7f4d000cdd70 2024-01-15T18:09:40.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.756+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf0007b90 con 0x7f4d000cdd70 2024-01-15T18:09:40.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.756+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf000ce40 con 0x7f4d000cdd70 2024-01-15T18:09:40.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.756+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d0014dc30 con 0x7f4d000cdd70 2024-01-15T18:09:40.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.757+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d000686f0 con 0x7f4d000cdd70 2024-01-15T18:09:40.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.758+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f4cf00113f0 con 0x7f4d000cdd70 2024-01-15T18:09:40.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.759+0000 7f4cf6a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cd806bf20 0x7f4cd806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:40.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.759+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4cf0088a60 con 0x7f4d000cdd70 2024-01-15T18:09:40.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.760+0000 7f4cf7e00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cd806bf20 0x7f4cd806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:40.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.761+0000 7f4cf7e00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cd806bf20 0x7f4cd806e3e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4d00072660 tx=0x7f4ce8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:40.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.764+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cf0055080 con 0x7f4d000cdd70 2024-01-15T18:09:40.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:40.983+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471615", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d000723b0 con 0x7f4cd806bf20 2024-01-15T18:09:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.024+0000 7f4cf6a00640 1 -- 172.21.15.2:0/3830446057 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4d000723b0 con 0x7f4cd806bf20 2024-01-15T18:09:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cd806bf20 msgr2=0x7f4cd806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cd806bf20 0x7f4cd806e3e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f4d00072660 tx=0x7f4ce8008040 comp rx=0 tx=0).stop 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 msgr2=0x7f4d000ada10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 0x7f4d000ada10 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f4cf000c7f0 tx=0x7f4cf000cbf0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 shutdown_connections 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cd806bf20 0x7f4cd806e3e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d0015c450 0x7f4d00144000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d00144c00 0x7f4d00143ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 --2- 172.21.15.2:0/3830446057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d000cdd70 0x7f4d000ada10 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.027+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 >> 172.21.15.2:0/3830446057 conn(0x7f4d000b45f0 msgr2=0x7f4d00070b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.028+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 shutdown_connections 2024-01-15T18:09:41.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.028+0000 7f4d05a00640 1 -- 172.21.15.2:0/3830446057 wait complete. 2024-01-15T18:09:41.051 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47161 2024-01-15T18:09:41.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.267+0000 7fe400400640 1 Processor -- start 2024-01-15T18:09:41.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.267+0000 7fe400400640 1 -- start start 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe400400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe3f81439f0 0x7fe3f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe400400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f815dd70 con 0x7fe3f80b3420 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe400400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f80b38b0 con 0x7fe3f806c8f0 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe400400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f80711b0 con 0x7fe3f81439f0 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe3fda00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe3f81439f0 0x7fe3f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe3fda00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe3f81439f0 0x7fe3f80702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54076/0 (socket says 172.21.15.2:54076) 2024-01-15T18:09:41.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe3fda00640 1 -- 172.21.15.2:0/3378543987 learned_addr learned my addr 172.21.15.2:0/3378543987 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:41.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.269+0000 7fe3fda00640 1 -- 172.21.15.2:0/3378543987 >> v1:172.21.15.137:6789/0 conn(0x7fe3f806c8f0 legacy=0x7fe3f806ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:41.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.270+0000 7fe3fda00640 1 -- 172.21.15.2:0/3378543987 >> v1:172.21.15.2:6789/0 conn(0x7fe3f80b3420 legacy=0x7fe3f80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.270+0000 7fe3fda00640 1 -- 172.21.15.2:0/3378543987 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8143dd0 con 0x7fe3f81439f0 2024-01-15T18:09:41.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.270+0000 7fe3f7e00640 1 -- 172.21.15.2:0/3378543987 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164663457 0 0) 0x7fe3f815dd70 con 0x7fe3f80b3420 2024-01-15T18:09:41.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.271+0000 7fe3fda00640 1 --2- 172.21.15.2:0/3378543987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f81439f0 0x7fe3f80702c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009870 tx=0x7fe3e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aabd786c4f6cf72 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:41.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.271+0000 7fe3f7e00640 1 -- 172.21.15.2:0/3378543987 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8030860 con 0x7fe3f81439f0 2024-01-15T18:09:41.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.271+0000 7fe3f7e00640 1 -- 172.21.15.2:0/3378543987 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e80309c0 con 0x7fe3f81439f0 2024-01-15T18:09:41.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.271+0000 7fe3f7e00640 1 -- 172.21.15.2:0/3378543987 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8030d50 con 0x7fe3f81439f0 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 -- 172.21.15.2:0/3378543987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f81439f0 msgr2=0x7fe3f80702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 --2- 172.21.15.2:0/3378543987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f81439f0 0x7fe3f80702c0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009870 tx=0x7fe3e802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 -- 172.21.15.2:0/3378543987 shutdown_connections 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 --2- 172.21.15.2:0/3378543987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f81439f0 0x7fe3f80702c0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 -- 172.21.15.2:0/3378543987 >> 172.21.15.2:0/3378543987 conn(0x7fe3f80b45f0 msgr2=0x7fe3f80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 -- 172.21.15.2:0/3378543987 shutdown_connections 2024-01-15T18:09:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.272+0000 7fe400400640 1 -- 172.21.15.2:0/3378543987 wait complete. 2024-01-15T18:09:41.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.273+0000 7fe400400640 1 Processor -- start 2024-01-15T18:09:41.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.274+0000 7fe400400640 1 -- start start 2024-01-15T18:09:41.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe400400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3f806c8f0 0x7fe3f80b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe400400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 0x7fe3f80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe400400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3f81439f0 0x7fe3f813ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe400400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f815bef0 con 0x7fe3f80b3420 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe400400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f815c1d0 con 0x7fe3f806c8f0 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe400400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f815c4b0 con 0x7fe3f81439f0 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 0x7fe3f80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 0x7fe3f80b1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54092/0 (socket says 172.21.15.2:54092) 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 -- 172.21.15.2:0/1771257107 learned_addr learned my addr 172.21.15.2:0/1771257107 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fe400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3f81439f0 0x7fe3f813ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fda00640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3f806c8f0 0x7fe3f80b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 -- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3f81439f0 msgr2=0x7fe3f813ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3f81439f0 0x7fe3f813ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 -- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3f806c8f0 msgr2=0x7fe3f80b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3f806c8f0 0x7fe3f80b0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fd000640 1 -- 172.21.15.2:0/1771257107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e80094d0 con 0x7fe3f80b3420 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.275+0000 7fe3fda00640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3f806c8f0 0x7fe3f80b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.276+0000 7fe3fd000640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 0x7fe3f80b1520 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c4b0 tx=0x7fe3e000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.276+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e0005480 con 0x7fe3f80b3420 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.276+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e00035f0 con 0x7fe3f80b3420 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.276+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e00039e0 con 0x7fe3f80b3420 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.276+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3f815c790 con 0x7fe3f80b3420 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.276+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f815ba50 con 0x7fe3f80b3420 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.277+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f815ccb0 con 0x7fe3f80b3420 2024-01-15T18:09:41.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.278+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe3e0003b40 con 0x7fe3f80b3420 2024-01-15T18:09:41.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.279+0000 7fe3f5600640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe3d006bf20 0x7fe3d006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.279+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fe3e0087e50 con 0x7fe3f80b3420 2024-01-15T18:09:41.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.280+0000 7fe3fda00640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe3d006bf20 0x7fe3d006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.281+0000 7fe3fda00640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe3d006bf20 0x7fe3d006e3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8036000 tx=0x7fe3e803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:41.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.284+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3f815ccb0 con 0x7fe3f80b3420 2024-01-15T18:09:41.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.502+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47161", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3f815bcd0 con 0x7fe3d006bf20 2024-01-15T18:09:41.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.553+0000 7fe3f5600640 1 -- 172.21.15.2:0/1771257107 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe3e8030d20 con 0x7fe3d006bf20 2024-01-15T18:09:41.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe3d006bf20 msgr2=0x7fe3d006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe3d006bf20 0x7fe3d006e3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8036000 tx=0x7fe3e803b040 comp rx=0 tx=0).stop 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 msgr2=0x7fe3f80b1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 0x7fe3f80b1520 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fe3e000c4b0 tx=0x7fe3e000c970 comp rx=0 tx=0).stop 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 shutdown_connections 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe3d006bf20 0x7fe3d006e3e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe3f81439f0 0x7fe3f813ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe3f80b3420 0x7fe3f80b1520 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 --2- 172.21.15.2:0/1771257107 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe3f806c8f0 0x7fe3f80b0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.557+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 >> 172.21.15.2:0/1771257107 conn(0x7fe3f80b45f0 msgr2=0x7fe3f81429e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.558+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 shutdown_connections 2024-01-15T18:09:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.558+0000 7fe400400640 1 -- 172.21.15.2:0/1771257107 wait complete. 2024-01-15T18:09:41.582 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471618 2024-01-15T18:09:41.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.800+0000 7f0773e00640 1 Processor -- start 2024-01-15T18:09:41.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.801+0000 7f0773e00640 1 -- start start 2024-01-15T18:09:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.802+0000 7f0773e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 0x7f077406c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.802+0000 7f0773e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0774071090 con 0x7f077415f260 2024-01-15T18:09:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f0773e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0774144560 con 0x7f0774143d20 2024-01-15T18:09:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f0773e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0774071430 con 0x7f07740720d0 2024-01-15T18:09:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f076be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f077415f260 0x7f077406c150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52650/0 (socket says 172.21.15.2:52650) 2024-01-15T18:09:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f076be00640 1 -- 172.21.15.2:0/1422728516 learned_addr learned my addr 172.21.15.2:0/1422728516 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f0773400640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 0x7f077406c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3299932709 0 0) 0x7f0774071090 con 0x7f077415f260 2024-01-15T18:09:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.803+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0754003520 con 0x7f077415f260 2024-01-15T18:09:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829415343 0 0) 0x7f0754003520 con 0x7f077415f260 2024-01-15T18:09:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 msgr2=0x7f077406c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 0x7f077406c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 >> v1:172.21.15.137:6789/0 conn(0x7f0774143d20 legacy=0x7f077415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774144280 con 0x7f077415f260 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f075c001670 con 0x7f077415f260 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 >> v1:172.21.15.2:6789/0 conn(0x7f077415f260 legacy=0x7f077406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 0x7f07540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0754003f20 0x7f07540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0754005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0774071430 con 0x7f0754004d00 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075400bbc0 con 0x7f0754003f20 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075400bd80 con 0x7f07740720d0 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.804+0000 7f0773400640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0754005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f0772a00640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 0x7f07540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f076be00640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0754003f20 0x7f07540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f0773400640 1 -- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 msgr2=0x7f07540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f0773400640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740720d0 0x7f07540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f0773400640 1 -- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0754003f20 msgr2=0x7f07540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f0773400640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0754003f20 0x7f07540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.805+0000 7f0773400640 1 -- 172.21.15.2:0/1422728516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754003520 con 0x7f0754004d00 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.806+0000 7f0773400640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0754005180 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f076c009fc0 tx=0x7f07540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d4dd4592c32fab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.806+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c02cc20 con 0x7f0754004d00 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.806+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c00f040 con 0x7f0754004d00 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.806+0000 7f0772000640 1 -- 172.21.15.2:0/1422728516 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c034440 con 0x7f0754004d00 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773e00640 1 -- 172.21.15.2:0/1422728516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 msgr2=0x7f0754005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773e00640 1 --2- 172.21.15.2:0/1422728516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0754005180 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f076c009fc0 tx=0x7f07540057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773400640 1 -- 172.21.15.2:0/1422728516 reap_dead start 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773e00640 1 -- 172.21.15.2:0/1422728516 shutdown_connections 2024-01-15T18:09:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773e00640 1 -- 172.21.15.2:0/1422728516 >> 172.21.15.2:0/1422728516 conn(0x7f07740b0840 msgr2=0x7f07740b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:41.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773e00640 1 -- 172.21.15.2:0/1422728516 shutdown_connections 2024-01-15T18:09:41.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.807+0000 7f0773e00640 1 -- 172.21.15.2:0/1422728516 wait complete. 2024-01-15T18:09:41.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.809+0000 7f0773e00640 1 Processor -- start 2024-01-15T18:09:41.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.809+0000 7f0773e00640 1 -- start start 2024-01-15T18:09:41.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0773e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0774141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0773e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740b0050 0x7f07740b2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0773e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07741f2a00 0x7f077406ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0773e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07740abec0 con 0x7f0754004d00 2024-01-15T18:09:41.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0773e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07740ab560 con 0x7f07741f2a00 2024-01-15T18:09:41.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0773e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740ab800 con 0x7f07740b0050 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0772a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0774141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0772a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0774141940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54112/0 (socket says 172.21.15.2:54112) 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.810+0000 7f0772a00640 1 -- 172.21.15.2:0/990903772 learned_addr learned my addr 172.21.15.2:0/990903772 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f076be00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740b0050 0x7f07740b2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0773400640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07741f2a00 0x7f077406ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0772a00640 1 -- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740b0050 msgr2=0x7f07740b2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0772a00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740b0050 0x7f07740b2610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0772a00640 1 -- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07741f2a00 msgr2=0x7f077406ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0772a00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07741f2a00 0x7f077406ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0772a00640 1 -- 172.21.15.2:0/990903772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740abae0 con 0x7f0754004d00 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0772a00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0774141940 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f076400e550 tx=0x7f076400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640052e0 con 0x7f0754004d00 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764005440 con 0x7f0754004d00 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764003810 con 0x7f0754004d00 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.811+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07740b2ea0 con 0x7f0754004d00 2024-01-15T18:09:41.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.812+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07741f2e80 con 0x7f0754004d00 2024-01-15T18:09:41.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.813+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07740ac130 con 0x7f0754004d00 2024-01-15T18:09:41.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.814+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0764003970 con 0x7f0754004d00 2024-01-15T18:09:41.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.815+0000 7f076b400640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f074c06bf50 0x7f074c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:41.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.815+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f07640882f0 con 0x7f0754004d00 2024-01-15T18:09:41.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.816+0000 7f076be00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f074c06bf50 0x7f074c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:41.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.817+0000 7f076be00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f074c06bf50 0x7f074c06e410 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f075c0010e0 tx=0x7f075c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:41.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:41.820+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0764054bb0 con 0x7f0754004d00 2024-01-15T18:09:42.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.050+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471618", "target": ["mon-mgr", ""]}) v1 -- 0x7f0774071850 con 0x7f074c06bf50 2024-01-15T18:09:42.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.109+0000 7f076b400640 1 -- 172.21.15.2:0/990903772 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f075c0109e0 con 0x7f074c06bf50 2024-01-15T18:09:42.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.112+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f074c06bf50 msgr2=0x7f074c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.112+0000 7f0773e00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f074c06bf50 0x7f074c06e410 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f075c0010e0 tx=0x7f075c009040 comp rx=0 tx=0).stop 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.112+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 msgr2=0x7f0774141940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.112+0000 7f0773e00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0774141940 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f076400e550 tx=0x7f076400ea10 comp rx=0 tx=0).stop 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 shutdown_connections 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f074c06bf50 0x7f074c06e410 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07741f2a00 0x7f077406ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07740b0050 0x7f07740b2610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 --2- 172.21.15.2:0/990903772 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0754004d00 0x7f0774141940 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 >> 172.21.15.2:0/990903772 conn(0x7f07740b0840 msgr2=0x7f07740cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 shutdown_connections 2024-01-15T18:09:42.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.113+0000 7f0773e00640 1 -- 172.21.15.2:0/990903772 wait complete. 2024-01-15T18:09:42.137 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47169 2024-01-15T18:09:42.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.345+0000 7f936a600640 1 Processor -- start 2024-01-15T18:09:42.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.345+0000 7f936a600640 1 -- start start 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f936a600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f93640b0b80 0x7f93640b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f936a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f93640709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f936a600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9364071250 con 0x7f9364150890 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f936a600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93640b1570 con 0x7f93640b0b80 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f936a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936406d000 con 0x7f9364070560 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9363400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9364150890 0x7f936406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52658/0 (socket says 172.21.15.2:52658) 2024-01-15T18:09:42.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9362a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f93640b0b80 0x7f93640b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9363400640 1 -- 172.21.15.2:0/3762946695 learned_addr learned my addr 172.21.15.2:0/3762946695 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:42.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9363e00640 1 --2- 172.21.15.2:0/3762946695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f93640709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9362a00640 1 -- 172.21.15.2:0/3762946695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 msgr2=0x7f93640709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:42.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9362a00640 1 --2- 172.21.15.2:0/3762946695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f93640709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.347+0000 7f9362a00640 1 -- 172.21.15.2:0/3762946695 >> v1:172.21.15.2:6789/0 conn(0x7f9364150890 legacy=0x7f936406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.348+0000 7f9362a00640 1 -- 172.21.15.2:0/3762946695 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9364071530 con 0x7f93640b0b80 2024-01-15T18:09:42.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.348+0000 7f9362000640 1 -- 172.21.15.2:0/3762946695 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527864478 0 0) 0x7f9364071250 con 0x7f9364150890 2024-01-15T18:09:42.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.349+0000 7f9362a00640 1 --2- 172.21.15.2:0/3762946695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640b1030 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f934c0097d0 tx=0x7f934c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d8eee266f34f16 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:42.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.349+0000 7f9362000640 1 -- 172.21.15.2:0/3762946695 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c031770 con 0x7f93640b0b80 2024-01-15T18:09:42.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.349+0000 7f9362000640 1 -- 172.21.15.2:0/3762946695 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c0318d0 con 0x7f93640b0b80 2024-01-15T18:09:42.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.349+0000 7f9362000640 1 -- 172.21.15.2:0/3762946695 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c031c60 con 0x7f93640b0b80 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 -- 172.21.15.2:0/3762946695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 msgr2=0x7f93640b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 --2- 172.21.15.2:0/3762946695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640b1030 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f934c0097d0 tx=0x7f934c02b810 comp rx=0 tx=0).stop 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 -- 172.21.15.2:0/3762946695 shutdown_connections 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 --2- 172.21.15.2:0/3762946695 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f93640709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 --2- 172.21.15.2:0/3762946695 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640b1030 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 -- 172.21.15.2:0/3762946695 >> 172.21.15.2:0/3762946695 conn(0x7f93640b45f0 msgr2=0x7f93640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 -- 172.21.15.2:0/3762946695 shutdown_connections 2024-01-15T18:09:42.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.350+0000 7f936a600640 1 -- 172.21.15.2:0/3762946695 wait complete. 2024-01-15T18:09:42.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.352+0000 7f936a600640 1 Processor -- start 2024-01-15T18:09:42.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.352+0000 7f936a600640 1 -- start start 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f936a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f936414bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f936a600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640ce330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f936a600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9364150890 0x7f93641f6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f936a600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93640cdf10 con 0x7f93640b0b80 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f936a600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936414ed80 con 0x7f9364150890 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f936a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936414e7c0 con 0x7f9364070560 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f9362a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f9362a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54122/0 (socket says 172.21.15.2:54122) 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f9362a00640 1 -- 172.21.15.2:0/1216660911 learned_addr learned my addr 172.21.15.2:0/1216660911 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.353+0000 7f9363e00640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9364150890 0x7f93641f6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9363400640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f936414bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9362a00640 1 -- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 msgr2=0x7f936414bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9362a00640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f936414bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9362a00640 1 -- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9364150890 msgr2=0x7f93641f6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9362a00640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9364150890 0x7f93641f6930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9362a00640 1 -- 172.21.15.2:0/1216660911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f934c009400 con 0x7f93640b0b80 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f9362a00640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640ce330 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f934c034040 tx=0x7f934c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.354+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c004230 con 0x7f93640b0b80 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.355+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f934c031f00 con 0x7f93640b0b80 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.355+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936414eaa0 con 0x7f93640b0b80 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.355+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f934c003610 con 0x7f93640b0b80 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.355+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93640ce870 con 0x7f93640b0b80 2024-01-15T18:09:42.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.356+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93640ce050 con 0x7f93640b0b80 2024-01-15T18:09:42.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.357+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f934c003770 con 0x7f93640b0b80 2024-01-15T18:09:42.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.358+0000 7f935b400640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f933c06bf50 0x7f933c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.358+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f934c0aa8c0 con 0x7f93640b0b80 2024-01-15T18:09:42.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.358+0000 7f9363400640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f933c06bf50 0x7f933c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.360+0000 7f9363400640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f933c06bf50 0x7f933c06e410 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9354000f60 tx=0x7f9354009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:42.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.362+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93640ce050 con 0x7f93640b0b80 2024-01-15T18:09:42.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.584+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47169", "target": ["mon-mgr", ""]}) v1 -- 0x7f9364150160 con 0x7f933c06bf50 2024-01-15T18:09:42.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.629+0000 7f935b400640 1 -- 172.21.15.2:0/1216660911 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9364150160 con 0x7f933c06bf50 2024-01-15T18:09:42.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f933c06bf50 msgr2=0x7f933c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f933c06bf50 0x7f933c06e410 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9354000f60 tx=0x7f9354009040 comp rx=0 tx=0).stop 2024-01-15T18:09:42.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 msgr2=0x7f93640ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640ce330 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f934c034040 tx=0x7f934c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 shutdown_connections 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f933c06bf50 0x7f933c06e410 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9364150890 0x7f93641f6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f93640b0b80 0x7f93640ce330 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 --2- 172.21.15.2:0/1216660911 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9364070560 0x7f936414bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.632+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 >> 172.21.15.2:0/1216660911 conn(0x7f93640b45f0 msgr2=0x7f93640af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.633+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 shutdown_connections 2024-01-15T18:09:42.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.633+0000 7f936a600640 1 -- 172.21.15.2:0/1216660911 wait complete. 2024-01-15T18:09:42.659 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:09:42.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.868+0000 7f3003c00640 1 Processor -- start 2024-01-15T18:09:42.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.868+0000 7f3003c00640 1 -- start start 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f3003c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f3003c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ffc15b840 0x7f2ffc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f3003c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ffc0b4760 con 0x7f2ffc06e830 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f3003c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ffc144f70 con 0x7f2ffc15b840 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f3003c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc070690 con 0x7f2ffc144b90 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f2ffbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ffc15b840 0x7f2ffc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f2ffbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ffc15b840 0x7f2ffc06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54134/0 (socket says 172.21.15.2:54134) 2024-01-15T18:09:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.870+0000 7f2ffbe00640 1 -- 172.21.15.2:0/376568754 learned_addr learned my addr 172.21.15.2:0/376568754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.871+0000 7f3001200640 1 --2- 172.21.15.2:0/376568754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.871+0000 7f2ffbe00640 1 -- 172.21.15.2:0/376568754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 msgr2=0x7f2ffc0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.871+0000 7f2ffbe00640 1 --2- 172.21.15.2:0/376568754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc0b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.871+0000 7f2ffbe00640 1 -- 172.21.15.2:0/376568754 >> v1:172.21.15.137:6789/0 conn(0x7f2ffc06e830 legacy=0x7f2ffc06ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:42.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.871+0000 7f2ffbe00640 1 -- 172.21.15.2:0/376568754 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc070970 con 0x7f2ffc15b840 2024-01-15T18:09:42.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.872+0000 7f2ffbe00640 1 --2- 172.21.15.2:0/376568754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc06fd20 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009e50 tx=0x7f2fec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311a3f7a20127cee server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:42.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.872+0000 7f2ffb400640 1 -- 172.21.15.2:0/376568754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec02ca70 con 0x7f2ffc15b840 2024-01-15T18:09:42.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.872+0000 7f2ffb400640 1 -- 172.21.15.2:0/376568754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec02cbd0 con 0x7f2ffc15b840 2024-01-15T18:09:42.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.872+0000 7f2ffb400640 1 -- 172.21.15.2:0/376568754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec0343f0 con 0x7f2ffc15b840 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 -- 172.21.15.2:0/376568754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 msgr2=0x7f2ffc06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 --2- 172.21.15.2:0/376568754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc06fd20 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2fec009e50 tx=0x7f2fec02bed0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 -- 172.21.15.2:0/376568754 shutdown_connections 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 --2- 172.21.15.2:0/376568754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc06fd20 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 --2- 172.21.15.2:0/376568754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 -- 172.21.15.2:0/376568754 >> 172.21.15.2:0/376568754 conn(0x7f2ffc0b2c10 msgr2=0x7f2ffc0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 -- 172.21.15.2:0/376568754 shutdown_connections 2024-01-15T18:09:42.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.873+0000 7f3003c00640 1 -- 172.21.15.2:0/376568754 wait complete. 2024-01-15T18:09:42.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.875+0000 7f3003c00640 1 Processor -- start 2024-01-15T18:09:42.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.875+0000 7f3003c00640 1 -- start start 2024-01-15T18:09:42.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3003c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc06e830 0x7f2ffc1f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3003c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3003c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3003c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc154d60 con 0x7f2ffc15b840 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3003c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc1f3be0 con 0x7f2ffc06e830 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3003c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc1f3110 con 0x7f2ffc144b90 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.876+0000 7f3001c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54150/0 (socket says 172.21.15.2:54150) 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 -- 172.21.15.2:0/4067987537 learned_addr learned my addr 172.21.15.2:0/4067987537 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001200640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc06e830 0x7f2ffc1f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f2ffbe00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 -- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 msgr2=0x7f2ffc06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 -- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc06e830 msgr2=0x7f2ffc1f2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc06e830 0x7f2ffc1f2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 -- 172.21.15.2:0/4067987537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec009b30 con 0x7f2ffc15b840 2024-01-15T18:09:42.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.877+0000 7f3001c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc155180 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400c410 tx=0x7f2ff400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:42.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.878+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4011790 con 0x7f2ffc15b840 2024-01-15T18:09:42.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.878+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff40118f0 con 0x7f2ffc15b840 2024-01-15T18:09:42.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.878+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4011ce0 con 0x7f2ffc15b840 2024-01-15T18:09:42.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.878+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ffc1549c0 con 0x7f2ffc15b840 2024-01-15T18:09:42.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.878+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ffc1f3620 con 0x7f2ffc15b840 2024-01-15T18:09:42.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.879+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ffc154ea0 con 0x7f2ffc15b840 2024-01-15T18:09:42.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.880+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f2ff401c460 con 0x7f2ffc15b840 2024-01-15T18:09:42.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.881+0000 7f2ff8c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fd406c040 0x7f2fd406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:42.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.881+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f2ff4088590 con 0x7f2ffc15b840 2024-01-15T18:09:42.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.882+0000 7f3001200640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fd406c040 0x7f2fd406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:42.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.883+0000 7f3001200640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fd406c040 0x7f2fd406e500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc06f2d0 tx=0x7f2fe4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:42.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:42.886+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ffc154ea0 con 0x7f2ffc15b840 2024-01-15T18:09:43.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.166+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ffc0cfd10 con 0x7f2fd406c040 2024-01-15T18:09:43.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.194+0000 7f2ff8c00640 1 -- 172.21.15.2:0/4067987537 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f2fe400c050 con 0x7f2fd406c040 2024-01-15T18:09:43.201 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:09:43.201 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:09:43.201 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:09:43.201 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:09:43.201 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:09:43.201 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:09:43.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.198+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fd406c040 msgr2=0x7f2fd406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.198+0000 7f3003c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fd406c040 0x7f2fd406e500 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc06f2d0 tx=0x7f2fe4008040 comp rx=0 tx=0).stop 2024-01-15T18:09:43.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.198+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 msgr2=0x7f2ffc155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.198+0000 7f3003c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc155180 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400c410 tx=0x7f2ff400c8d0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.198+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 shutdown_connections 2024-01-15T18:09:43.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fd406c040 0x7f2fd406e500 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ffc15b840 0x7f2ffc155180 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ffc144b90 0x7f2ffc06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 --2- 172.21.15.2:0/4067987537 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ffc06e830 0x7f2ffc1f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 >> 172.21.15.2:0/4067987537 conn(0x7f2ffc0b2c10 msgr2=0x7f2ffc0b0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:43.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 shutdown_connections 2024-01-15T18:09:43.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.199+0000 7f3003c00640 1 -- 172.21.15.2:0/4067987537 wait complete. 2024-01-15T18:09:43.235 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:09:43.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.521+0000 7fca84200640 1 Processor -- start 2024-01-15T18:09:43.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.521+0000 7fca84200640 1 -- start start 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.522+0000 7fca84200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.522+0000 7fca84200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0b25f0 con 0x7fca7c14afd0 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.522+0000 7fca84200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c0b28d0 con 0x7fca7c14b3b0 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.522+0000 7fca84200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c070ff0 con 0x7fca7c06d000 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.522+0000 7fca81800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fca7c14afd0 0x7fca7c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42682/0 (socket says 172.21.15.2:42682) 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.522+0000 7fca81800640 1 -- 172.21.15.2:0/3806450970 learned_addr learned my addr 172.21.15.2:0/3806450970 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:43.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca80e00640 1 --2- 172.21.15.2:0/3806450970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:43.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca7be00640 1 -- 172.21.15.2:0/3806450970 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027261299 0 0) 0x7fca7c0b25f0 con 0x7fca7c14afd0 2024-01-15T18:09:43.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca7be00640 1 -- 172.21.15.2:0/3806450970 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca5c003520 con 0x7fca7c14afd0 2024-01-15T18:09:43.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca7be00640 1 -- 172.21.15.2:0/3806450970 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547514999 0 0) 0x7fca7c0b28d0 con 0x7fca7c14b3b0 2024-01-15T18:09:43.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca80e00640 1 -- 172.21.15.2:0/3806450970 >> v1:172.21.15.137:6789/0 conn(0x7fca7c14b3b0 legacy=0x7fca7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca80e00640 1 -- 172.21.15.2:0/3806450970 >> v1:172.21.15.2:6789/0 conn(0x7fca7c14afd0 legacy=0x7fca7c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.523+0000 7fca80e00640 1 -- 172.21.15.2:0/3806450970 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c0b2bb0 con 0x7fca7c06d000 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.524+0000 7fca80e00640 1 --2- 172.21.15.2:0/3806450970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c0b33b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fca640097d0 tx=0x7fca6402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d0aa9efda0d5c01 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca7be00640 1 -- 172.21.15.2:0/3806450970 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64031800 con 0x7fca7c06d000 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca7be00640 1 -- 172.21.15.2:0/3806450970 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca64031960 con 0x7fca7c06d000 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca7be00640 1 -- 172.21.15.2:0/3806450970 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64031cf0 con 0x7fca7c06d000 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 -- 172.21.15.2:0/3806450970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 msgr2=0x7fca7c0b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 --2- 172.21.15.2:0/3806450970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c0b33b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fca640097d0 tx=0x7fca6402b8a0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 -- 172.21.15.2:0/3806450970 shutdown_connections 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 --2- 172.21.15.2:0/3806450970 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c0b33b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 -- 172.21.15.2:0/3806450970 >> 172.21.15.2:0/3806450970 conn(0x7fca7c0b45f0 msgr2=0x7fca7c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 -- 172.21.15.2:0/3806450970 shutdown_connections 2024-01-15T18:09:43.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.525+0000 7fca84200640 1 -- 172.21.15.2:0/3806450970 wait complete. 2024-01-15T18:09:43.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.527+0000 7fca84200640 1 Processor -- start 2024-01-15T18:09:43.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.527+0000 7fca84200640 1 -- start start 2024-01-15T18:09:43.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca84200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c1602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca84200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 0x7fca7c143bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca84200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca7c14b3b0 0x7fca7c1440f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca84200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cd090 con 0x7fca7c14afd0 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca84200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cc410 con 0x7fca7c14b3b0 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca84200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cc6f0 con 0x7fca7c06d000 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca80e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 0x7fca7c143bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca80e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 0x7fca7c143bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33140/0 (socket says 172.21.15.2:33140) 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.528+0000 7fca80e00640 1 -- 172.21.15.2:0/156368648 learned_addr learned my addr 172.21.15.2:0/156368648 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca82200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca7c14b3b0 0x7fca7c1440f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca81800640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c1602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca80e00640 1 -- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 msgr2=0x7fca7c1602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca80e00640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c1602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca80e00640 1 -- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca7c14b3b0 msgr2=0x7fca7c1440f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca80e00640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca7c14b3b0 0x7fca7c1440f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca80e00640 1 -- 172.21.15.2:0/156368648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca64009400 con 0x7fca7c14afd0 2024-01-15T18:09:43.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.529+0000 7fca80e00640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 0x7fca7c143bb0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fca64034040 tx=0x7fca640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:43.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.530+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64004230 con 0x7fca7c14afd0 2024-01-15T18:09:43.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.530+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca64004390 con 0x7fca7c14afd0 2024-01-15T18:09:43.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.530+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6403aad0 con 0x7fca7c14afd0 2024-01-15T18:09:43.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.530+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c0cc9d0 con 0x7fca7c14afd0 2024-01-15T18:09:43.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.530+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c0cc0c0 con 0x7fca7c14afd0 2024-01-15T18:09:43.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.531+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c0686f0 con 0x7fca7c14afd0 2024-01-15T18:09:43.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.532+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fca64042070 con 0x7fca7c14afd0 2024-01-15T18:09:43.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.533+0000 7fca79600640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca5406c090 0x7fca5406e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:43.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.533+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fca6402e070 con 0x7fca7c14afd0 2024-01-15T18:09:43.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.533+0000 7fca81800640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca5406c090 0x7fca5406e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:43.538 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.534+0000 7fca81800640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca5406c090 0x7fca5406e550 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0011e0 tx=0x7fca6c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:43.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.537+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca64076bb0 con 0x7fca7c14afd0 2024-01-15T18:09:43.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.763+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fca7c0cfc90 con 0x7fca5406c090 2024-01-15T18:09:43.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.828+0000 7fca79600640 1 -- 172.21.15.2:0/156368648 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fca7c0cfc90 con 0x7fca5406c090 2024-01-15T18:09:43.833 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:09:43.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca5406c090 msgr2=0x7fca5406e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca5406c090 0x7fca5406e550 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fca6c0011e0 tx=0x7fca6c007d30 comp rx=0 tx=0).stop 2024-01-15T18:09:43.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 msgr2=0x7fca7c143bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:43.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 0x7fca7c143bb0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fca64034040 tx=0x7fca640046c0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 shutdown_connections 2024-01-15T18:09:43.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fca5406c090 0x7fca5406e550 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca7c14b3b0 0x7fca7c1440f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca7c14afd0 0x7fca7c143bb0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 --2- 172.21.15.2:0/156368648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca7c06d000 0x7fca7c1602b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.831+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 >> 172.21.15.2:0/156368648 conn(0x7fca7c0b45f0 msgr2=0x7fca7c06c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.832+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 shutdown_connections 2024-01-15T18:09:43.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:43.832+0000 7fca84200640 1 -- 172.21.15.2:0/156368648 wait complete. 2024-01-15T18:09:43.854 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:09:44.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.069+0000 7f4b96200640 1 Processor -- start 2024-01-15T18:09:44.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.070+0000 7f4b96200640 1 -- start start 2024-01-15T18:09:44.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.071+0000 7f4b96200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b900cd9f0 0x7f4b900cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.072+0000 7f4b96200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b90070da0 con 0x7f4b9006e830 2024-01-15T18:09:44.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.072+0000 7f4b96200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9006cd70 con 0x7f4b90146b80 2024-01-15T18:09:44.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.072+0000 7f4b96200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9006c240 con 0x7f4b900cd9f0 2024-01-15T18:09:44.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.072+0000 7f4b8fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4b9006e830 0x7f4b9006fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42688/0 (socket says 172.21.15.2:42688) 2024-01-15T18:09:44.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.072+0000 7f4b8fe00640 1 -- 172.21.15.2:0/1799272498 learned_addr learned my addr 172.21.15.2:0/1799272498 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:44.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.073+0000 7f4b8ea00640 1 --2- 172.21.15.2:0/1799272498 >> v2:172.21.15.137:3300/0 conn(0x7f4b900cd9f0 0x7f4b900cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.073+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794584875 0 0) 0x7f4b90070da0 con 0x7f4b9006e830 2024-01-15T18:09:44.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.073+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b70003520 con 0x7f4b9006e830 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.073+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3021700885 0 0) 0x7f4b70003520 con 0x7f4b9006e830 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 >> v2:172.21.15.137:3300/0 conn(0x7f4b900cd9f0 msgr2=0x7f4b900cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 --2- 172.21.15.2:0/1799272498 >> v2:172.21.15.137:3300/0 conn(0x7f4b900cd9f0 0x7f4b900cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 >> v1:172.21.15.137:6789/0 conn(0x7f4b90146b80 legacy=0x7f4b9006c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b90070ac0 con 0x7f4b9006e830 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4b88001a10 con 0x7f4b9006e830 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 >> v1:172.21.15.2:6789/0 conn(0x7f4b9006e830 legacy=0x7f4b9006fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b70003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b700047c0 0x7f4b70004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700052a0 0x7f4b7000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9006c240 con 0x7f4b700039c0 2024-01-15T18:09:44.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c1a0 con 0x7f4b700047c0 2024-01-15T18:09:44.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7000c360 con 0x7f4b700052a0 2024-01-15T18:09:44.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.074+0000 7f4b8f400640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b70003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8fe00640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700052a0 0x7f4b7000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8f400640 1 -- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700052a0 msgr2=0x7f4b7000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8f400640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700052a0 0x7f4b7000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8f400640 1 -- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b700047c0 msgr2=0x7f4b70004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8f400640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b700047c0 0x7f4b70004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8f400640 1 -- 172.21.15.2:0/1799272498 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70003520 con 0x7f4b700039c0 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8fe00640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b700052a0 0x7f4b7000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.075+0000 7f4b8fe00640 1 -- 172.21.15.2:0/1799272498 reap_dead start 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.076+0000 7f4b8f400640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b70003de0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001750 tx=0x7f4b700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1083e8ee4690bb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.077+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b800052d0 con 0x7f4b700039c0 2024-01-15T18:09:44.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.077+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b8002e050 con 0x7f4b700039c0 2024-01-15T18:09:44.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.077+0000 7f4b8e000640 1 -- 172.21.15.2:0/1799272498 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80003830 con 0x7f4b700039c0 2024-01-15T18:09:44.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.077+0000 7f4b96200640 1 -- 172.21.15.2:0/1799272498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 msgr2=0x7f4b70003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.077+0000 7f4b96200640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b70003de0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f4b80001750 tx=0x7f4b700057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.078+0000 7f4b96200640 1 -- 172.21.15.2:0/1799272498 shutdown_connections 2024-01-15T18:09:44.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.078+0000 7f4b96200640 1 --2- 172.21.15.2:0/1799272498 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b70003de0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.078+0000 7f4b96200640 1 -- 172.21.15.2:0/1799272498 >> 172.21.15.2:0/1799272498 conn(0x7f4b900b2c10 msgr2=0x7f4b900b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:44.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.078+0000 7f4b96200640 1 -- 172.21.15.2:0/1799272498 shutdown_connections 2024-01-15T18:09:44.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.078+0000 7f4b96200640 1 -- 172.21.15.2:0/1799272498 wait complete. 2024-01-15T18:09:44.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.080+0000 7f4b96200640 1 Processor -- start 2024-01-15T18:09:44.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.081+0000 7f4b96200640 1 -- start start 2024-01-15T18:09:44.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b96200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b900ccc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b96200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b9015f820 0x7f4b901f79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b96200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9015c290 0x7f4b9006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b96200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9013fd60 con 0x7f4b700039c0 2024-01-15T18:09:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b96200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9015fce0 con 0x7f4b9015f820 2024-01-15T18:09:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b96200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b90140010 con 0x7f4b9015c290 2024-01-15T18:09:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b900ccc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b8f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b900ccc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33150/0 (socket says 172.21.15.2:33150) 2024-01-15T18:09:44.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.085+0000 7f4b8f400640 1 -- 172.21.15.2:0/2794168961 learned_addr learned my addr 172.21.15.2:0/2794168961 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.086+0000 7f4b8fe00640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9015c290 0x7f4b9006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.086+0000 7f4b8f400640 1 -- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9015c290 msgr2=0x7f4b9006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.086+0000 7f4b8f400640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9015c290 0x7f4b9006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.086+0000 7f4b8f400640 1 -- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b9015f820 msgr2=0x7f4b901f79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.086+0000 7f4b8f400640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b9015f820 0x7f4b901f79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.086+0000 7f4b8f400640 1 -- 172.21.15.2:0/2794168961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b901402f0 con 0x7f4b700039c0 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.087+0000 7f4b8fe00640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9015c290 0x7f4b9006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:44.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.087+0000 7f4b8f400640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b900ccc20 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4b80000c00 tx=0x7f4b80001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:44.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.088+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b800076c0 con 0x7f4b700039c0 2024-01-15T18:09:44.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.088+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b80007a30 con 0x7f4b700039c0 2024-01-15T18:09:44.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.088+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80006be0 con 0x7f4b700039c0 2024-01-15T18:09:44.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.088+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b901405d0 con 0x7f4b700039c0 2024-01-15T18:09:44.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.088+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b901f7f10 con 0x7f4b700039c0 2024-01-15T18:09:44.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.089+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b900686f0 con 0x7f4b700039c0 2024-01-15T18:09:44.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.093+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f4b80006d40 con 0x7f4b700039c0 2024-01-15T18:09:44.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.095+0000 7f4b87400640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6806c090 0x7f4b6806e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.095+0000 7f4b8ea00640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6806c090 0x7f4b6806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.096+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4b800aa5d0 con 0x7f4b700039c0 2024-01-15T18:09:44.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.097+0000 7f4b8ea00640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6806c090 0x7f4b6806e550 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4b78006e80 tx=0x7f4b78006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:44.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.103+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b80076b50 con 0x7f4b700039c0 2024-01-15T18:09:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.516+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b9013fea0 con 0x7f4b6806c090 2024-01-15T18:09:44.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.533+0000 7f4b87400640 1 -- 172.21.15.2:0/2794168961 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4b9013fea0 con 0x7f4b6806c090 2024-01-15T18:09:44.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6806c090 msgr2=0x7f4b6806e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6806c090 0x7f4b6806e550 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4b78006e80 tx=0x7f4b78006dd0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 msgr2=0x7f4b900ccc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b900ccc20 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f4b80000c00 tx=0x7f4b80001120 comp rx=0 tx=0).stop 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 shutdown_connections 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6806c090 0x7f4b6806e550 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b9015c290 0x7f4b9006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b9015f820 0x7f4b901f79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 --2- 172.21.15.2:0/2794168961 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b700039c0 0x7f4b900ccc20 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 >> 172.21.15.2:0/2794168961 conn(0x7f4b900b2c10 msgr2=0x7f4b900b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 shutdown_connections 2024-01-15T18:09:44.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.536+0000 7f4b96200640 1 -- 172.21.15.2:0/2794168961 wait complete. 2024-01-15T18:09:44.560 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47164 2024-01-15T18:09:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.736+0000 7f75a2e00640 1 Processor -- start 2024-01-15T18:09:44.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.736+0000 7f75a2e00640 1 -- start start 2024-01-15T18:09:44.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f75a2e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f759c06e830 0x7f759c06ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f75a2e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f759c06c3f0 0x7f759c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f75a2e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759c06f150 con 0x7f759c070260 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f75a2e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c070690 con 0x7f759c06c3f0 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f75a2e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f759c070890 con 0x7f759c06e830 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f7593e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f759c06c3f0 0x7f759c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f7593e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f759c06c3f0 0x7f759c06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33162/0 (socket says 172.21.15.2:33162) 2024-01-15T18:09:44.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.738+0000 7f7593e00640 1 -- 172.21.15.2:0/1031562483 learned_addr learned my addr 172.21.15.2:0/1031562483 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.739+0000 7f759be00640 1 --2- 172.21.15.2:0/1031562483 >> v2:172.21.15.137:3300/0 conn(0x7f759c06e830 0x7f759c06ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.739+0000 7f7593e00640 1 -- 172.21.15.2:0/1031562483 >> v2:172.21.15.137:3300/0 conn(0x7f759c06e830 msgr2=0x7f759c06ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.739+0000 7f7593e00640 1 --2- 172.21.15.2:0/1031562483 >> v2:172.21.15.137:3300/0 conn(0x7f759c06e830 0x7f759c06ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.739+0000 7f7593e00640 1 -- 172.21.15.2:0/1031562483 >> v1:172.21.15.2:6789/0 conn(0x7f759c070260 legacy=0x7f759c0cda10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.739+0000 7f7593e00640 1 -- 172.21.15.2:0/1031562483 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c070b70 con 0x7f759c06c3f0 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.740+0000 7f7593e00640 1 --2- 172.21.15.2:0/1031562483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06c3f0 0x7f759c06fd20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f759c0b37b0 tx=0x7f758402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ea03f21dad598e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:44.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.740+0000 7f759b400640 1 -- 172.21.15.2:0/1031562483 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584031870 con 0x7f759c06c3f0 2024-01-15T18:09:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.740+0000 7f759b400640 1 -- 172.21.15.2:0/1031562483 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75840319d0 con 0x7f759c06c3f0 2024-01-15T18:09:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.740+0000 7f759b400640 1 -- 172.21.15.2:0/1031562483 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584031d60 con 0x7f759c06c3f0 2024-01-15T18:09:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 -- 172.21.15.2:0/1031562483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06c3f0 msgr2=0x7f759c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 --2- 172.21.15.2:0/1031562483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06c3f0 0x7f759c06fd20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f759c0b37b0 tx=0x7f758402b910 comp rx=0 tx=0).stop 2024-01-15T18:09:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 -- 172.21.15.2:0/1031562483 shutdown_connections 2024-01-15T18:09:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 --2- 172.21.15.2:0/1031562483 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06c3f0 0x7f759c06fd20 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 --2- 172.21.15.2:0/1031562483 >> v2:172.21.15.137:3300/0 conn(0x7f759c06e830 0x7f759c06ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 -- 172.21.15.2:0/1031562483 >> 172.21.15.2:0/1031562483 conn(0x7f759c0b2c10 msgr2=0x7f759c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 -- 172.21.15.2:0/1031562483 shutdown_connections 2024-01-15T18:09:44.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.741+0000 7f75a2e00640 1 -- 172.21.15.2:0/1031562483 wait complete. 2024-01-15T18:09:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.743+0000 7f75a2e00640 1 Processor -- start 2024-01-15T18:09:44.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.743+0000 7f75a2e00640 1 -- start start 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f75a2e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759c06c3f0 0x7f759c1fb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f75a2e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 0x7f759c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f75a2e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f759c070260 0x7f759c159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f75a2e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c147850 con 0x7f759c06e830 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f75a2e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759c1facd0 con 0x7f759c070260 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f75a2e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759c15b170 con 0x7f759c06c3f0 2024-01-15T18:09:44.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f7593e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 0x7f759c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f7593e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 0x7f759c15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33170/0 (socket says 172.21.15.2:33170) 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f7593e00640 1 -- 172.21.15.2:0/473333965 learned_addr learned my addr 172.21.15.2:0/473333965 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.744+0000 7f759be00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759c06c3f0 0x7f759c1fb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7593e00640 1 -- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759c06c3f0 msgr2=0x7f759c1fb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7593e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759c06c3f0 0x7f759c1fb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7593e00640 1 -- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f759c070260 msgr2=0x7f759c159210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f75a0e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f759c070260 0x7f759c159210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7593e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f759c070260 0x7f759c159210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7593e00640 1 -- 172.21.15.2:0/473333965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75840094d0 con 0x7f759c06e830 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7593e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 0x7f759c15f550 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f7584000c00 tx=0x7f75840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584031860 con 0x7f759c06e830 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75840443f0 con 0x7f759c06e830 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75840447a0 con 0x7f759c06e830 2024-01-15T18:09:44.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f759c1faf40 con 0x7f759c06e830 2024-01-15T18:09:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.745+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c15ae20 con 0x7f759c06e830 2024-01-15T18:09:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.747+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759c1fb460 con 0x7f759c06e830 2024-01-15T18:09:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.747+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f7584044900 con 0x7f759c06e830 2024-01-15T18:09:44.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.748+0000 7f7598c00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757406c040 0x7f757406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:44.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.748+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f758402d040 con 0x7f759c06e830 2024-01-15T18:09:44.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.749+0000 7f759be00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757406c040 0x7f757406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:44.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.750+0000 7f759be00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757406c040 0x7f757406e500 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f758c00a970 tx=0x7f758c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:44.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.753+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f759c1fb460 con 0x7f759c06e830 2024-01-15T18:09:44.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:44.969+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47164", "target": ["mon-mgr", ""]}) v1 -- 0x7f759c0b41a0 con 0x7f757406c040 2024-01-15T18:09:45.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.011+0000 7f7598c00640 1 -- 172.21.15.2:0/473333965 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f758c00c050 con 0x7f757406c040 2024-01-15T18:09:45.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.014+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757406c040 msgr2=0x7f757406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.014+0000 7f75a2e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757406c040 0x7f757406e500 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f758c00a970 tx=0x7f758c008040 comp rx=0 tx=0).stop 2024-01-15T18:09:45.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.014+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 msgr2=0x7f759c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.014+0000 7f75a2e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 0x7f759c15f550 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f7584000c00 tx=0x7f75840046c0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 shutdown_connections 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f757406c040 0x7f757406e500 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f759c070260 0x7f759c159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f759c06e830 0x7f759c15f550 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 --2- 172.21.15.2:0/473333965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f759c06c3f0 0x7f759c1fb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 >> 172.21.15.2:0/473333965 conn(0x7f759c0b2c10 msgr2=0x7f759c06c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 shutdown_connections 2024-01-15T18:09:45.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.015+0000 7f75a2e00640 1 -- 172.21.15.2:0/473333965 wait complete. 2024-01-15T18:09:45.044 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471610 2024-01-15T18:09:45.235 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.231+0000 7f0ac6400640 1 Processor -- start 2024-01-15T18:09:45.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.231+0000 7f0ac6400640 1 -- start start 2024-01-15T18:09:45.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0ac6400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ac0144c00 0x7f0ac00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0ac6400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00ce230 con 0x7f0ac006d000 2024-01-15T18:09:45.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0ac6400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac00ce510 con 0x7f0ac00cdd70 2024-01-15T18:09:45.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0ac6400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac00ce7f0 con 0x7f0ac0144c00 2024-01-15T18:09:45.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0abf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ac0144c00 0x7f0ac00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0abf400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ac0144c00 0x7f0ac00cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33180/0 (socket says 172.21.15.2:33180) 2024-01-15T18:09:45.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.233+0000 7f0abf400640 1 -- 172.21.15.2:0/1696104505 learned_addr learned my addr 172.21.15.2:0/1696104505 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:45.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.234+0000 7f0abf400640 1 -- 172.21.15.2:0/1696104505 >> v1:172.21.15.137:6789/0 conn(0x7f0ac00cdd70 legacy=0x7f0ac006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:45.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.234+0000 7f0abf400640 1 -- 172.21.15.2:0/1696104505 >> v1:172.21.15.2:6789/0 conn(0x7f0ac006d000 legacy=0x7f0ac0070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.234+0000 7f0abf400640 1 -- 172.21.15.2:0/1696104505 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac00705f0 con 0x7f0ac0144c00 2024-01-15T18:09:45.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.235+0000 7f0abf400640 1 --2- 172.21.15.2:0/1696104505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac00cd830 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009870 tx=0x7f0ab002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0afe31d14123fcf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:45.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0abe000640 1 -- 172.21.15.2:0/1696104505 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0030860 con 0x7f0ac0144c00 2024-01-15T18:09:45.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0abe000640 1 -- 172.21.15.2:0/1696104505 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab00309c0 con 0x7f0ac0144c00 2024-01-15T18:09:45.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0abe000640 1 -- 172.21.15.2:0/1696104505 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0030d50 con 0x7f0ac0144c00 2024-01-15T18:09:45.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 -- 172.21.15.2:0/1696104505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 msgr2=0x7f0ac00cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 --2- 172.21.15.2:0/1696104505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac00cd830 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009870 tx=0x7f0ab002b9a0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 -- 172.21.15.2:0/1696104505 shutdown_connections 2024-01-15T18:09:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 --2- 172.21.15.2:0/1696104505 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac00cd830 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 -- 172.21.15.2:0/1696104505 >> 172.21.15.2:0/1696104505 conn(0x7f0ac00b45f0 msgr2=0x7f0ac00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 -- 172.21.15.2:0/1696104505 shutdown_connections 2024-01-15T18:09:45.241 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.236+0000 7f0ac6400640 1 -- 172.21.15.2:0/1696104505 wait complete. 2024-01-15T18:09:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.238+0000 7f0ac6400640 1 Processor -- start 2024-01-15T18:09:45.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.238+0000 7f0ac6400640 1 -- start start 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.239+0000 7f0ac6400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac006d000 0x7f0ac01f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.239+0000 7f0ac6400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00cdd70 0x7f0ac00b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.239+0000 7f0ac6400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac013abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.239+0000 7f0ac6400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01f6170 con 0x7f0ac0144c00 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.239+0000 7f0ac6400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00b09a0 con 0x7f0ac006d000 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.239+0000 7f0ac6400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00b0c80 con 0x7f0ac00cdd70 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac013abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac013abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33192/0 (socket says 172.21.15.2:33192) 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 -- 172.21.15.2:0/3672532864 learned_addr learned my addr 172.21.15.2:0/3672532864 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abf400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac006d000 0x7f0ac01f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abea00640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00cdd70 0x7f0ac00b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 -- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00cdd70 msgr2=0x7f0ac00b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00cdd70 0x7f0ac00b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 -- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac006d000 msgr2=0x7f0ac01f5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac006d000 0x7f0ac01f5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 -- 172.21.15.2:0/3672532864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00094d0 con 0x7f0ac0144c00 2024-01-15T18:09:45.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.240+0000 7f0abfe00640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac013abf0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c730 tx=0x7f0ab800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:45.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.241+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8003b70 con 0x7f0ac0144c00 2024-01-15T18:09:45.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.241+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab8003cd0 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.241+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800aa90 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.241+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac00b0f60 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.241+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac00b1520 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.242+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac00b1740 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.243+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f0ab800abf0 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.243+0000 7f0ab7400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a9806c040 0x7f0a9806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.244+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0ab8088810 con 0x7f0ac0144c00 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.244+0000 7f0abf400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a9806c040 0x7f0a9806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.245+0000 7f0abf400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a9806c040 0x7f0a9806e500 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036000 tx=0x7f0ab003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:45.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.248+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ab8054d60 con 0x7f0ac0144c00 2024-01-15T18:09:45.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.461+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471610", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ac00686f0 con 0x7f0a9806c040 2024-01-15T18:09:45.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.502+0000 7f0ab7400640 1 -- 172.21.15.2:0/3672532864 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0ac00686f0 con 0x7f0a9806c040 2024-01-15T18:09:45.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a9806c040 msgr2=0x7f0a9806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a9806c040 0x7f0a9806e500 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0036000 tx=0x7f0ab003b040 comp rx=0 tx=0).stop 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 msgr2=0x7f0ac013abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac013abf0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800c730 tx=0x7f0ab800cbf0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 shutdown_connections 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0a9806c040 0x7f0a9806e500 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ac0144c00 0x7f0ac013abf0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ac00cdd70 0x7f0ac00b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 --2- 172.21.15.2:0/3672532864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ac006d000 0x7f0ac01f5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.504+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 >> 172.21.15.2:0/3672532864 conn(0x7f0ac00b45f0 msgr2=0x7f0ac015c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:45.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.505+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 shutdown_connections 2024-01-15T18:09:45.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.505+0000 7f0ac6400640 1 -- 172.21.15.2:0/3672532864 wait complete. 2024-01-15T18:09:45.529 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471616 2024-01-15T18:09:45.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.737+0000 7f6277200640 1 Processor -- start 2024-01-15T18:09:45.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.738+0000 7f6277200640 1 -- start start 2024-01-15T18:09:45.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.739+0000 7f6277200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f627814b160 0x7f6278161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.739+0000 7f6277200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6278070820 0x7f627806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.739+0000 7f6277200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62781620d0 con 0x7f6278072100 2024-01-15T18:09:45.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.739+0000 7f6277200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62780b35b0 con 0x7f6278070820 2024-01-15T18:09:45.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.739+0000 7f6277200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62780b1510 con 0x7f627814b160 2024-01-15T18:09:45.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.740+0000 7f6276800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6278070820 0x7f627806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.740+0000 7f6275e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f627814b160 0x7f6278161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.740+0000 7f626fe00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f6278072100 0x7f6278070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:37592/0 (socket says 172.21.15.2:37592) 2024-01-15T18:09:45.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.740+0000 7f626fe00640 1 -- 172.21.15.2:0/2851699534 learned_addr learned my addr 172.21.15.2:0/2851699534 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:45.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6275400640 1 -- 172.21.15.2:0/2851699534 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867060001 0 0) 0x7f62781620d0 con 0x7f6278072100 2024-01-15T18:09:45.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6275400640 1 -- 172.21.15.2:0/2851699534 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6258003520 con 0x7f6278072100 2024-01-15T18:09:45.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6276800640 1 -- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f627814b160 msgr2=0x7f6278161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:45.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6276800640 1 --2- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f627814b160 0x7f6278161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6276800640 1 -- 172.21.15.2:0/2851699534 >> v1:172.21.15.137:6789/0 conn(0x7f6278072100 legacy=0x7f6278070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6276800640 1 -- 172.21.15.2:0/2851699534 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62780b32d0 con 0x7f6278070820 2024-01-15T18:09:45.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.741+0000 7f6275e00640 1 --2- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f627814b160 0x7f6278161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:45.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.743+0000 7f6276800640 1 --2- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6278070820 0x7f627806cdd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6270009d90 tx=0x7f627002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e56e396bb946f6c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:45.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.743+0000 7f6275400640 1 -- 172.21.15.2:0/2851699534 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627002cc60 con 0x7f6278070820 2024-01-15T18:09:45.747 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.743+0000 7f6275400640 1 -- 172.21.15.2:0/2851699534 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627002cdc0 con 0x7f6278070820 2024-01-15T18:09:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.743+0000 7f6275400640 1 -- 172.21.15.2:0/2851699534 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6270034640 con 0x7f6278070820 2024-01-15T18:09:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 -- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6278070820 msgr2=0x7f627806cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 --2- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6278070820 0x7f627806cdd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6270009d90 tx=0x7f627002beb0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 -- 172.21.15.2:0/2851699534 shutdown_connections 2024-01-15T18:09:45.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 --2- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6278070820 0x7f627806cdd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 --2- 172.21.15.2:0/2851699534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f627814b160 0x7f6278161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 -- 172.21.15.2:0/2851699534 >> 172.21.15.2:0/2851699534 conn(0x7f62780b0840 msgr2=0x7f62780b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:45.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 -- 172.21.15.2:0/2851699534 shutdown_connections 2024-01-15T18:09:45.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.744+0000 7f6277200640 1 -- 172.21.15.2:0/2851699534 wait complete. 2024-01-15T18:09:45.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.746+0000 7f6277200640 1 Processor -- start 2024-01-15T18:09:45.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.746+0000 7f6277200640 1 -- start start 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6277200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278070820 0x7f6278161040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6277200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 0x7f62781310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6277200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f627814b160 0x7f62781f6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6277200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278131b40 con 0x7f6278072100 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6277200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278131e20 con 0x7f627814b160 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6277200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6278132100 con 0x7f6278070820 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 0x7f62781310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 0x7f62781310c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33204/0 (socket says 172.21.15.2:33204) 2024-01-15T18:09:45.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 -- 172.21.15.2:0/1316453876 learned_addr learned my addr 172.21.15.2:0/1316453876 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f6276800640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f627814b160 0x7f62781f6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 -- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278070820 msgr2=0x7f6278161040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278070820 0x7f6278161040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 -- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f627814b160 msgr2=0x7f62781f6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f627814b160 0x7f62781f6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.747+0000 7f626fe00640 1 -- 172.21.15.2:0/1316453876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6268009400 con 0x7f6278072100 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.748+0000 7f626fe00640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 0x7f62781310c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f626000c430 tx=0x7f626000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.748+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6260005ce0 con 0x7f6278072100 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.748+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6260005e40 con 0x7f6278072100 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.748+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62600072f0 con 0x7f6278072100 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.748+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6270009a70 con 0x7f6278072100 2024-01-15T18:09:45.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.748+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62781326c0 con 0x7f6278072100 2024-01-15T18:09:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.749+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62781328e0 con 0x7f6278072100 2024-01-15T18:09:45.754 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.750+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f6260007450 con 0x7f6278072100 2024-01-15T18:09:45.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.751+0000 7f626e000640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c070 0x7f625006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:45.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.751+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6260087f70 con 0x7f6278072100 2024-01-15T18:09:45.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.752+0000 7f6275e00640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c070 0x7f625006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:45.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.753+0000 7f6275e00640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c070 0x7f625006e530 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6268006e80 tx=0x7f6268006dd0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:45.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.755+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6260054580 con 0x7f6278072100 2024-01-15T18:09:45.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:45.969+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471616", "target": ["mon-mgr", ""]}) v1 -- 0x7f6278131f90 con 0x7f625006c070 2024-01-15T18:09:46.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.011+0000 7f626e000640 1 -- 172.21.15.2:0/1316453876 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6278131f90 con 0x7f625006c070 2024-01-15T18:09:46.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c070 msgr2=0x7f625006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c070 0x7f625006e530 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f6268006e80 tx=0x7f6268006dd0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 msgr2=0x7f62781310c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 0x7f62781310c0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f626000c430 tx=0x7f626000c8f0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 shutdown_connections 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c070 0x7f625006e530 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f627814b160 0x7f62781f6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278072100 0x7f62781310c0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 --2- 172.21.15.2:0/1316453876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278070820 0x7f6278161040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 >> 172.21.15.2:0/1316453876 conn(0x7f62780b0840 msgr2=0x7f62781486d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 shutdown_connections 2024-01-15T18:09:46.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.014+0000 7f6277200640 1 -- 172.21.15.2:0/1316453876 wait complete. 2024-01-15T18:09:46.038 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471614 2024-01-15T18:09:46.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.247+0000 7f8c03c00640 1 Processor -- start 2024-01-15T18:09:46.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.247+0000 7f8c03c00640 1 -- start start 2024-01-15T18:09:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c03c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 0x7f8bfc0b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c03c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8bfc06c8f0 0x7f8bfc06cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c03c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc15dd70 con 0x7f8bfc1439f0 2024-01-15T18:09:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c03c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bfc070720 con 0x7f8bfc06c8f0 2024-01-15T18:09:46.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c03c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc06d2a0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c01c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8bfc06c8f0 0x7f8bfc06cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 0x7f8bfc0b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 0x7f8bfc0b3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:42018/0 (socket says 172.21.15.2:42018) 2024-01-15T18:09:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8bfbe00640 1 -- 172.21.15.2:0/3968926987 learned_addr learned my addr 172.21.15.2:0/3968926987 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.248+0000 7f8c01200640 1 --1- 172.21.15.2:0/3968926987 >> v1:172.21.15.137:6789/0 conn(0x7f8bfc1439f0 0x7f8bfc0cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:37598/0 (socket says 172.21.15.2:37598) 2024-01-15T18:09:46.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.249+0000 7f8c01c00640 1 -- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 msgr2=0x7f8bfc0b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.249+0000 7f8c01c00640 1 --2- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 0x7f8bfc0b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.249+0000 7f8c01c00640 1 -- 172.21.15.2:0/3968926987 >> v1:172.21.15.137:6789/0 conn(0x7f8bfc1439f0 legacy=0x7f8bfc0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.249+0000 7f8c01c00640 1 -- 172.21.15.2:0/3968926987 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0711b0 con 0x7f8bfc06c8f0 2024-01-15T18:09:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.249+0000 7f8bfb400640 1 -- 172.21.15.2:0/3968926987 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637913365 0 0) 0x7f8bfc15dd70 con 0x7f8bfc1439f0 2024-01-15T18:09:46.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.249+0000 7f8bfbe00640 1 --2- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 0x7f8bfc0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:46.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.251+0000 7f8c01c00640 1 --2- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc06c8f0 0x7f8bfc06cd60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009dc0 tx=0x7f8bf402be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e6a63007b6ba98f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:46.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.251+0000 7f8bfb400640 1 -- 172.21.15.2:0/3968926987 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf402cc30 con 0x7f8bfc06c8f0 2024-01-15T18:09:46.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8bfb400640 1 -- 172.21.15.2:0/3968926987 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf402cd90 con 0x7f8bfc06c8f0 2024-01-15T18:09:46.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8bfb400640 1 -- 172.21.15.2:0/3968926987 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4034660 con 0x7f8bfc06c8f0 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 -- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc06c8f0 msgr2=0x7f8bfc06cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 --2- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc06c8f0 0x7f8bfc06cd60 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8bf4009dc0 tx=0x7f8bf402be80 comp rx=0 tx=0).stop 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 -- 172.21.15.2:0/3968926987 shutdown_connections 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 --2- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc06c8f0 0x7f8bfc06cd60 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 --2- 172.21.15.2:0/3968926987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc0702c0 0x7f8bfc0b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 -- 172.21.15.2:0/3968926987 >> 172.21.15.2:0/3968926987 conn(0x7f8bfc0b45f0 msgr2=0x7f8bfc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 -- 172.21.15.2:0/3968926987 shutdown_connections 2024-01-15T18:09:46.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.252+0000 7f8c03c00640 1 -- 172.21.15.2:0/3968926987 wait complete. 2024-01-15T18:09:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.255+0000 7f8c03c00640 1 Processor -- start 2024-01-15T18:09:46.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.255+0000 7f8c03c00640 1 -- start start 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c03c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc06c8f0 0x7f8bfc159be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c03c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 0x7f8bfc13ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c03c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc1439f0 0x7f8bfc13f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c03c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc139ab0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c03c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc139d90 con 0x7f8bfc1439f0 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c03c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bfc13a070 con 0x7f8bfc06c8f0 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 0x7f8bfc13ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 0x7f8bfc13ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33220/0 (socket says 172.21.15.2:33220) 2024-01-15T18:09:46.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 -- 172.21.15.2:0/1483937618 learned_addr learned my addr 172.21.15.2:0/1483937618 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c01c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc1439f0 0x7f8bfc13f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c01200640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc06c8f0 0x7f8bfc159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 -- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc06c8f0 msgr2=0x7f8bfc159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc06c8f0 0x7f8bfc159be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 -- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc1439f0 msgr2=0x7f8bfc13f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc1439f0 0x7f8bfc13f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 -- 172.21.15.2:0/1483937618 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be40094d0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8c01200640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc06c8f0 0x7f8bfc159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.256+0000 7f8bfbe00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 0x7f8bfc13ebe0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f8be400e9d0 tx=0x7f8be400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.257+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be400ca60 con 0x7f8bfc0702c0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.257+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be400cbc0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.257+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be4013470 con 0x7f8bfc0702c0 2024-01-15T18:09:46.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.257+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf4009a70 con 0x7f8bfc0702c0 2024-01-15T18:09:46.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.257+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc13a630 con 0x7f8bfc0702c0 2024-01-15T18:09:46.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.258+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc0686f0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.259+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f8be40135d0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.260+0000 7f8bf8c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bd406c040 0x7f8bd406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.260+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8be4088cc0 con 0x7f8bfc0702c0 2024-01-15T18:09:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.261+0000 7f8c01200640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bd406c040 0x7f8bd406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.263+0000 7f8c01200640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bd406c040 0x7f8bd406e500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8bec000f60 tx=0x7f8bec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:46.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.265+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8be4055240 con 0x7f8bfc0702c0 2024-01-15T18:09:46.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.490+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471614", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bfc070bf0 con 0x7f8bd406c040 2024-01-15T18:09:46.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.526+0000 7f8bf8c00640 1 -- 172.21.15.2:0/1483937618 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8bec0012f0 con 0x7f8bd406c040 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bd406c040 msgr2=0x7f8bd406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bd406c040 0x7f8bd406e500 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f8bec000f60 tx=0x7f8bec009040 comp rx=0 tx=0).stop 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 msgr2=0x7f8bfc13ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 0x7f8bfc13ebe0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f8be400e9d0 tx=0x7f8be400ee90 comp rx=0 tx=0).stop 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 shutdown_connections 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8bd406c040 0x7f8bd406e500 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8bfc1439f0 0x7f8bfc13f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8bfc0702c0 0x7f8bfc13ebe0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 --2- 172.21.15.2:0/1483937618 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8bfc06c8f0 0x7f8bfc159be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.528+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 >> 172.21.15.2:0/1483937618 conn(0x7f8bfc0b45f0 msgr2=0x7f8bfc13c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:46.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.529+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 shutdown_connections 2024-01-15T18:09:46.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.529+0000 7f8c03c00640 1 -- 172.21.15.2:0/1483937618 wait complete. 2024-01-15T18:09:46.553 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471620 2024-01-15T18:09:46.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.761+0000 7f0b87200640 1 Processor -- start 2024-01-15T18:09:46.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.761+0000 7f0b87200640 1 -- start start 2024-01-15T18:09:46.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.762+0000 7f0b87200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0b88143d20 0x7f0b8815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.762+0000 7f0b87200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b880720d0 0x7f0b8815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.762+0000 7f0b87200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b88071090 con 0x7f0b881447d0 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.762+0000 7f0b87200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b88071430 con 0x7f0b88143d20 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.762+0000 7f0b87200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b88144100 con 0x7f0b880720d0 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0b88143d20 0x7f0b8815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0b88143d20 0x7f0b8815eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33228/0 (socket says 172.21.15.2:33228) 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 -- 172.21.15.2:0/3625666378 learned_addr learned my addr 172.21.15.2:0/3625666378 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b86800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0b881447d0 0x7f0b880cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42722/0 (socket says 172.21.15.2:42722) 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b7d200640 1 --2- 172.21.15.2:0/3625666378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b880720d0 0x7f0b8815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 -- 172.21.15.2:0/3625666378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b880720d0 msgr2=0x7f0b8815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 --2- 172.21.15.2:0/3625666378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b880720d0 0x7f0b8815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 -- 172.21.15.2:0/3625666378 >> v1:172.21.15.2:6789/0 conn(0x7f0b881447d0 legacy=0x7f0b880cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.763+0000 7f0b85e00640 1 -- 172.21.15.2:0/3625666378 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8806d160 con 0x7f0b88143d20 2024-01-15T18:09:46.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.764+0000 7f0b85e00640 1 --2- 172.21.15.2:0/3625666378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b88143d20 0x7f0b8815eb50 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009e50 tx=0x7f0b7802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fe7ca035569085 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:46.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.765+0000 7f0b85400640 1 -- 172.21.15.2:0/3625666378 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7802ca80 con 0x7f0b88143d20 2024-01-15T18:09:46.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.765+0000 7f0b85400640 1 -- 172.21.15.2:0/3625666378 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7802cbe0 con 0x7f0b88143d20 2024-01-15T18:09:46.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.765+0000 7f0b85400640 1 -- 172.21.15.2:0/3625666378 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b780343f0 con 0x7f0b88143d20 2024-01-15T18:09:46.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.765+0000 7f0b87200640 1 -- 172.21.15.2:0/3625666378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b88143d20 msgr2=0x7f0b8815eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.765+0000 7f0b87200640 1 --2- 172.21.15.2:0/3625666378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b88143d20 0x7f0b8815eb50 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0b78009e50 tx=0x7f0b7802bee0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.766+0000 7f0b87200640 1 -- 172.21.15.2:0/3625666378 shutdown_connections 2024-01-15T18:09:46.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.766+0000 7f0b87200640 1 --2- 172.21.15.2:0/3625666378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b880720d0 0x7f0b8815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.766+0000 7f0b87200640 1 --2- 172.21.15.2:0/3625666378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b88143d20 0x7f0b8815eb50 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.766+0000 7f0b87200640 1 -- 172.21.15.2:0/3625666378 >> 172.21.15.2:0/3625666378 conn(0x7f0b880b0840 msgr2=0x7f0b880b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:46.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.766+0000 7f0b87200640 1 -- 172.21.15.2:0/3625666378 shutdown_connections 2024-01-15T18:09:46.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.766+0000 7f0b87200640 1 -- 172.21.15.2:0/3625666378 wait complete. 2024-01-15T18:09:46.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 Processor -- start 2024-01-15T18:09:46.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 -- start start 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 0x7f0b8815a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b88143d20 0x7f0b8815d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b881447d0 0x7f0b8813e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880acf50 con 0x7f0b880720d0 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880b2a50 con 0x7f0b881447d0 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.768+0000 7f0b87200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b880b2d30 con 0x7f0b88143d20 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 0x7f0b8815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 0x7f0b8815a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33238/0 (socket says 172.21.15.2:33238) 2024-01-15T18:09:46.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 -- 172.21.15.2:0/2500392337 learned_addr learned my addr 172.21.15.2:0/2500392337 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b7d200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b88143d20 0x7f0b8815d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b86800640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b881447d0 0x7f0b8813e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 -- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b88143d20 msgr2=0x7f0b8815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b88143d20 0x7f0b8815d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 -- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b881447d0 msgr2=0x7f0b8813e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b881447d0 0x7f0b8813e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.769+0000 7f0b85e00640 1 -- 172.21.15.2:0/2500392337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009b30 con 0x7f0b880720d0 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.770+0000 7f0b85e00640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 0x7f0b8815a690 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0b78000f80 tx=0x7f0b78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.770+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7803e450 con 0x7f0b880720d0 2024-01-15T18:09:46.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.770+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7803e5b0 con 0x7f0b880720d0 2024-01-15T18:09:46.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.770+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7803e960 con 0x7f0b880720d0 2024-01-15T18:09:46.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.770+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b880ac900 con 0x7f0b880720d0 2024-01-15T18:09:46.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.770+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8815abd0 con 0x7f0b880720d0 2024-01-15T18:09:46.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.771+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b88074770 con 0x7f0b880720d0 2024-01-15T18:09:46.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.772+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f0b7802cd50 con 0x7f0b880720d0 2024-01-15T18:09:46.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.773+0000 7f0b7ea00640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b6006c070 0x7f0b6006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:46.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.773+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0b780ab330 con 0x7f0b880720d0 2024-01-15T18:09:46.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.774+0000 7f0b7d200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b6006c070 0x7f0b6006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:46.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.775+0000 7f0b7d200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b6006c070 0x7f0b6006e530 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0b70006fd0 tx=0x7f0b70008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:46.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:46.777+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b780779a0 con 0x7f0b880720d0 2024-01-15T18:09:47.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.001+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471620", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b8815f720 con 0x7f0b6006c070 2024-01-15T18:09:47.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.049+0000 7f0b7ea00640 1 -- 172.21.15.2:0/2500392337 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0b7000f830 con 0x7f0b6006c070 2024-01-15T18:09:47.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.051+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b6006c070 msgr2=0x7f0b6006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b6006c070 0x7f0b6006e530 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0b70006fd0 tx=0x7f0b70008040 comp rx=0 tx=0).stop 2024-01-15T18:09:47.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 msgr2=0x7f0b8815a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 0x7f0b8815a690 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0b78000f80 tx=0x7f0b78004a40 comp rx=0 tx=0).stop 2024-01-15T18:09:47.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 shutdown_connections 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b6006c070 0x7f0b6006e530 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b881447d0 0x7f0b8813e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b88143d20 0x7f0b8815d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 --2- 172.21.15.2:0/2500392337 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b880720d0 0x7f0b8815a690 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 >> 172.21.15.2:0/2500392337 conn(0x7f0b880b0840 msgr2=0x7f0b880aff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 shutdown_connections 2024-01-15T18:09:47.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.052+0000 7f0b87200640 1 -- 172.21.15.2:0/2500392337 wait complete. 2024-01-15T18:09:47.080 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471613 2024-01-15T18:09:47.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.377+0000 7ffb6ae00640 1 Processor -- start 2024-01-15T18:09:47.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.378+0000 7ffb6ae00640 1 -- start start 2024-01-15T18:09:47.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb6ae00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffb640b2b40 0x7ffb640acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb6ae00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6415dbc0 con 0x7ffb6415ae80 2024-01-15T18:09:47.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb6ae00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb6415dea0 con 0x7ffb6406d000 2024-01-15T18:09:47.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb6ae00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb6415e840 con 0x7ffb640b2b40 2024-01-15T18:09:47.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb68e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffb640b2b40 0x7ffb640acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb63e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7ffb6415ae80 0x7ffb6406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42732/0 (socket says 172.21.15.2:42732) 2024-01-15T18:09:47.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.381+0000 7ffb63e00640 1 -- 172.21.15.2:0/3844538027 learned_addr learned my addr 172.21.15.2:0/3844538027 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:47.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.380+0000 7ffb63400640 1 --1- 172.21.15.2:0/3844538027 >> v1:172.21.15.137:6789/0 conn(0x7ffb6406d000 0x7ffb640b2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:37600/0 (socket says 172.21.15.2:37600) 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268425597 0 0) 0x7ffb6415dea0 con 0x7ffb6406d000 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb44003520 con 0x7ffb6406d000 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 <== mon.1 v1:172.21.15.137:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701601680 0 0) 0x7ffb44003520 con 0x7ffb6406d000 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 >> v2:172.21.15.2:3300/0 conn(0x7ffb640b2b40 msgr2=0x7ffb640acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 --2- 172.21.15.2:0/3844538027 >> v2:172.21.15.2:3300/0 conn(0x7ffb640b2b40 0x7ffb640acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 >> v1:172.21.15.2:6789/0 conn(0x7ffb6415ae80 legacy=0x7ffb6406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.382+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 --> v1:172.21.15.137:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6415e180 con 0x7ffb6406d000 2024-01-15T18:09:47.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 <== mon.1 v1:172.21.15.137:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7ffb4c001670 con 0x7ffb6406d000 2024-01-15T18:09:47.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 >> v1:172.21.15.137:6789/0 conn(0x7ffb6406d000 legacy=0x7ffb640b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb44003970 0x7ffb44003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb44004370 0x7ffb44004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44005040 0x7ffb4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6415e840 con 0x7ffb44003970 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4400bfa0 con 0x7ffb44004370 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4400c160 con 0x7ffb44005040 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb68e00640 1 --2- 172.21.15.2:0/3844538027 >> v2:172.21.15.2:3300/0 conn(0x7ffb640b2b40 0x7ffb640acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.383+0000 7ffb63400640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb44004370 0x7ffb44004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.384+0000 7ffb63400640 1 -- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44005040 msgr2=0x7ffb4400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.384+0000 7ffb63400640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44005040 0x7ffb4400b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.384+0000 7ffb63400640 1 -- 172.21.15.2:0/3844538027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb44003970 msgr2=0x7ffb44003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.384+0000 7ffb63400640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb44003970 0x7ffb44003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.384+0000 7ffb63400640 1 -- 172.21.15.2:0/3844538027 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6415e180 con 0x7ffb44004370 2024-01-15T18:09:47.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.384+0000 7ffb68e00640 1 -- 172.21.15.2:0/3844538027 reap_dead start 2024-01-15T18:09:47.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.385+0000 7ffb63400640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb44004370 0x7ffb44004840 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c160 tx=0x7ffb440057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72982162f9110d68 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.386+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c005a10 con 0x7ffb44004370 2024-01-15T18:09:47.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.387+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c005b70 con 0x7ffb44004370 2024-01-15T18:09:47.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.387+0000 7ffb62a00640 1 -- 172.21.15.2:0/3844538027 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c007070 con 0x7ffb44004370 2024-01-15T18:09:47.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.387+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb44004370 msgr2=0x7ffb44004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.387+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb44004370 0x7ffb44004840 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c160 tx=0x7ffb440057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.388+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3844538027 shutdown_connections 2024-01-15T18:09:47.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.388+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3844538027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb44004370 0x7ffb44004840 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.388+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3844538027 >> 172.21.15.2:0/3844538027 conn(0x7ffb640b45f0 msgr2=0x7ffb640b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:47.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.388+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3844538027 shutdown_connections 2024-01-15T18:09:47.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.388+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3844538027 wait complete. 2024-01-15T18:09:47.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.391+0000 7ffb6ae00640 1 Processor -- start 2024-01-15T18:09:47.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.391+0000 7ffb6ae00640 1 -- start start 2024-01-15T18:09:47.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.392+0000 7ffb6ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44004370 0x7ffb640b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.392+0000 7ffb6ae00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 0x7ffb64140d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.392+0000 7ffb6ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb641f1ee0 0x7ffb641f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.392+0000 7ffb6ae00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640b1000 con 0x7ffb64140390 2024-01-15T18:09:47.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.392+0000 7ffb6ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640b12e0 con 0x7ffb641f1ee0 2024-01-15T18:09:47.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.392+0000 7ffb6ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb640b15c0 con 0x7ffb44004370 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 0x7ffb64140d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 0x7ffb64140d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33268/0 (socket says 172.21.15.2:33268) 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 -- 172.21.15.2:0/3301863006 learned_addr learned my addr 172.21.15.2:0/3301863006 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb68e00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb641f1ee0 0x7ffb641f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63e00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44004370 0x7ffb640b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 -- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44004370 msgr2=0x7ffb640b0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44004370 0x7ffb640b0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 -- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb641f1ee0 msgr2=0x7ffb641f2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb641f1ee0 0x7ffb641f2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb63400640 1 -- 172.21.15.2:0/3301863006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb640b18a0 con 0x7ffb64140390 2024-01-15T18:09:47.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.393+0000 7ffb68e00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb641f1ee0 0x7ffb641f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:09:47.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.394+0000 7ffb63400640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 0x7ffb64140d90 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c280 tx=0x7ffb4c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.394+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c0049e0 con 0x7ffb64140390 2024-01-15T18:09:47.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.394+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb4c03a070 con 0x7ffb64140390 2024-01-15T18:09:47.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.394+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c004d60 con 0x7ffb64140390 2024-01-15T18:09:47.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.394+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6413e500 con 0x7ffb64140390 2024-01-15T18:09:47.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.394+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb641407f0 con 0x7ffb64140390 2024-01-15T18:09:47.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.395+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6413e690 con 0x7ffb64140390 2024-01-15T18:09:47.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.398+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7ffb4c032070 con 0x7ffb64140390 2024-01-15T18:09:47.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.398+0000 7ffb5be00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb3c06c040 0x7ffb3c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:47.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.399+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7ffb4c0aa610 con 0x7ffb64140390 2024-01-15T18:09:47.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.399+0000 7ffb63e00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb3c06c040 0x7ffb3c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:47.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.400+0000 7ffb63e00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb3c06c040 0x7ffb3c06e500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007fa0 tx=0x7ffb54007f30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:47.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.407+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb4c076c20 con 0x7ffb64140390 2024-01-15T18:09:47.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.798+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471613", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb640acd00 con 0x7ffb3c06c040 2024-01-15T18:09:47.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.826+0000 7ffb5be00640 1 -- 172.21.15.2:0/3301863006 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ffb54004c70 con 0x7ffb3c06c040 2024-01-15T18:09:47.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb3c06c040 msgr2=0x7ffb3c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb3c06c040 0x7ffb3c06e500 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ffb54007fa0 tx=0x7ffb54007f30 comp rx=0 tx=0).stop 2024-01-15T18:09:47.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 msgr2=0x7ffb64140d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:47.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 0x7ffb64140d90 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c00c280 tx=0x7ffb4c001930 comp rx=0 tx=0).stop 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 shutdown_connections 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb3c06c040 0x7ffb3c06e500 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb641f1ee0 0x7ffb641f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb64140390 0x7ffb64140d90 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 --2- 172.21.15.2:0/3301863006 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb44004370 0x7ffb640b0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 >> 172.21.15.2:0/3301863006 conn(0x7ffb640b45f0 msgr2=0x7ffb6406c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 shutdown_connections 2024-01-15T18:09:47.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:47.829+0000 7ffb6ae00640 1 -- 172.21.15.2:0/3301863006 wait complete. 2024-01-15T18:09:47.849 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471611 2024-01-15T18:09:48.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.024+0000 7f8b8e800640 1 Processor -- start 2024-01-15T18:09:48.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.024+0000 7f8b8e800640 1 -- start start 2024-01-15T18:09:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.025+0000 7f8b8e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8815ae80 0x7f8b880acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.025+0000 7f8b8e800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8b8806c8f0 0x7f8b8806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.025+0000 7f8b8e800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8806d290 con 0x7f8b880b2430 2024-01-15T18:09:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.025+0000 7f8b8e800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8815e840 con 0x7f8b8806c8f0 2024-01-15T18:09:48.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.025+0000 7f8b8e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8815dbc0 con 0x7f8b8815ae80 2024-01-15T18:09:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.026+0000 7f8b87400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8815ae80 0x7f8b880acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.026+0000 7f8b87e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8b880b2430 0x7f8b880b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:37608/0 (socket says 172.21.15.2:37608) 2024-01-15T18:09:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.026+0000 7f8b87e00640 1 -- 172.21.15.2:0/1914749424 learned_addr learned my addr 172.21.15.2:0/1914749424 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.026+0000 7f8b86a00640 1 --2- 172.21.15.2:0/1914749424 >> v2:172.21.15.137:3300/0 conn(0x7f8b8806c8f0 0x7f8b8806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.027+0000 7f8b86000640 1 -- 172.21.15.2:0/1914749424 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448015691 0 0) 0x7f8b8806d290 con 0x7f8b880b2430 2024-01-15T18:09:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.027+0000 7f8b86000640 1 -- 172.21.15.2:0/1914749424 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b68003520 con 0x7f8b880b2430 2024-01-15T18:09:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.027+0000 7f8b87400640 1 -- 172.21.15.2:0/1914749424 >> v2:172.21.15.137:3300/0 conn(0x7f8b8806c8f0 msgr2=0x7f8b8806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.027+0000 7f8b87400640 1 --2- 172.21.15.2:0/1914749424 >> v2:172.21.15.137:3300/0 conn(0x7f8b8806c8f0 0x7f8b8806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.027+0000 7f8b87400640 1 -- 172.21.15.2:0/1914749424 >> v1:172.21.15.137:6789/0 conn(0x7f8b880b2430 legacy=0x7f8b880b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.027+0000 7f8b87400640 1 -- 172.21.15.2:0/1914749424 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8815dea0 con 0x7f8b8815ae80 2024-01-15T18:09:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.029+0000 7f8b87400640 1 --2- 172.21.15.2:0/1914749424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8815ae80 0x7f8b880acd00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8b780097a0 tx=0x7f8b7802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64ca7f7c41c6fd79 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.029+0000 7f8b86000640 1 -- 172.21.15.2:0/1914749424 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78031830 con 0x7f8b8815ae80 2024-01-15T18:09:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.029+0000 7f8b86000640 1 -- 172.21.15.2:0/1914749424 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b78031990 con 0x7f8b8815ae80 2024-01-15T18:09:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.029+0000 7f8b86000640 1 -- 172.21.15.2:0/1914749424 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b78031d20 con 0x7f8b8815ae80 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 -- 172.21.15.2:0/1914749424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8815ae80 msgr2=0x7f8b880acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 --2- 172.21.15.2:0/1914749424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8815ae80 0x7f8b880acd00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f8b780097a0 tx=0x7f8b7802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 -- 172.21.15.2:0/1914749424 shutdown_connections 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 --2- 172.21.15.2:0/1914749424 >> v2:172.21.15.137:3300/0 conn(0x7f8b8806c8f0 0x7f8b8806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 --2- 172.21.15.2:0/1914749424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8815ae80 0x7f8b880acd00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 -- 172.21.15.2:0/1914749424 >> 172.21.15.2:0/1914749424 conn(0x7f8b880b45f0 msgr2=0x7f8b880b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 -- 172.21.15.2:0/1914749424 shutdown_connections 2024-01-15T18:09:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.030+0000 7f8b8e800640 1 -- 172.21.15.2:0/1914749424 wait complete. 2024-01-15T18:09:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.031+0000 7f8b8e800640 1 Processor -- start 2024-01-15T18:09:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 -- start start 2024-01-15T18:09:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8806c8f0 0x7f8b880b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 0x7f8b880afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815ae80 0x7f8b88158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880afff0 con 0x7f8b880b2430 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880b0220 con 0x7f8b8815ae80 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.032+0000 7f8b8e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b880b0500 con 0x7f8b8806c8f0 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 0x7f8b880afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 0x7f8b880afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33276/0 (socket says 172.21.15.2:33276) 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 -- 172.21.15.2:0/381088844 learned_addr learned my addr 172.21.15.2:0/381088844 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b87400640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8806c8f0 0x7f8b880b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b87e00640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815ae80 0x7f8b88158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 -- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8806c8f0 msgr2=0x7f8b880b1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8806c8f0 0x7f8b880b1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 -- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815ae80 msgr2=0x7f8b88158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815ae80 0x7f8b88158b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b86a00640 1 -- 172.21.15.2:0/381088844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b78009400 con 0x7f8b880b2430 2024-01-15T18:09:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.033+0000 7f8b87400640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8806c8f0 0x7f8b880b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:09:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.034+0000 7f8b86a00640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 0x7f8b880afab0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000efc0 tx=0x7f8b7000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.034+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70014480 con 0x7f8b880b2430 2024-01-15T18:09:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.034+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7000f040 con 0x7f8b880b2430 2024-01-15T18:09:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.034+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b70014800 con 0x7f8b880b2430 2024-01-15T18:09:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.034+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b880b07e0 con 0x7f8b880b2430 2024-01-15T18:09:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.034+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b880b0ac0 con 0x7f8b880b2430 2024-01-15T18:09:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.035+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b880686f0 con 0x7f8b880b2430 2024-01-15T18:09:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.036+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f8b70014960 con 0x7f8b880b2430 2024-01-15T18:09:48.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.037+0000 7f8b7f400640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8b6006c040 0x7f8b6006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.037+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8b70089800 con 0x7f8b880b2430 2024-01-15T18:09:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.038+0000 7f8b87400640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8b6006c040 0x7f8b6006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.039+0000 7f8b87400640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8b6006c040 0x7f8b6006e500 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8b78036000 tx=0x7f8b7803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.042+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b70055e10 con 0x7f8b880b2430 2024-01-15T18:09:48.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.331+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471611", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b880b0670 con 0x7f8b6006c040 2024-01-15T18:09:48.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.372+0000 7f8b7f400640 1 -- 172.21.15.2:0/381088844 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8b880b0670 con 0x7f8b6006c040 2024-01-15T18:09:48.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.375+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8b6006c040 msgr2=0x7f8b6006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.375+0000 7f8b8e800640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8b6006c040 0x7f8b6006e500 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f8b78036000 tx=0x7f8b7803c040 comp rx=0 tx=0).stop 2024-01-15T18:09:48.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 msgr2=0x7f8b880afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 0x7f8b880afab0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000efc0 tx=0x7f8b7000c490 comp rx=0 tx=0).stop 2024-01-15T18:09:48.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 shutdown_connections 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8b6006c040 0x7f8b6006e500 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8b8815ae80 0x7f8b88158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8b880b2430 0x7f8b880afab0 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 --2- 172.21.15.2:0/381088844 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8b8806c8f0 0x7f8b880b1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 >> 172.21.15.2:0/381088844 conn(0x7f8b880b45f0 msgr2=0x7f8b880ad7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 shutdown_connections 2024-01-15T18:09:48.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.376+0000 7f8b8e800640 1 -- 172.21.15.2:0/381088844 wait complete. 2024-01-15T18:09:48.399 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47166 2024-01-15T18:09:48.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.597+0000 7f86a3400640 1 Processor -- start 2024-01-15T18:09:48.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.598+0000 7f86a3400640 1 -- start start 2024-01-15T18:09:48.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a3400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f869c0d1870 0x7f869c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f869c0b2210 0x7f869c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1c50 0x7f869c15d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a3400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c0ce6d0 con 0x7f869c0b2210 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a3400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c15db60 con 0x7f869c0d1870 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0d1250 con 0x7f869c0d1c50 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.599+0000 7f86a1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f869c0b2210 0x7f869c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f869c0b2210 0x7f869c15d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33288/0 (socket says 172.21.15.2:33288) 2024-01-15T18:09:48.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 -- 172.21.15.2:0/2700193928 learned_addr learned my addr 172.21.15.2:0/2700193928 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a2a00640 1 --2- 172.21.15.2:0/2700193928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1c50 0x7f869c15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a2000640 1 --2- 172.21.15.2:0/2700193928 >> v2:172.21.15.137:3300/0 conn(0x7f869c0d1870 0x7f869c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 -- 172.21.15.2:0/2700193928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1c50 msgr2=0x7f869c15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 --2- 172.21.15.2:0/2700193928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1c50 0x7f869c15d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 -- 172.21.15.2:0/2700193928 >> v2:172.21.15.137:3300/0 conn(0x7f869c0d1870 msgr2=0x7f869c0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 --2- 172.21.15.2:0/2700193928 >> v2:172.21.15.137:3300/0 conn(0x7f869c0d1870 0x7f869c0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.600+0000 7f86a1600640 1 -- 172.21.15.2:0/2700193928 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0ce8e0 con 0x7f869c0b2210 2024-01-15T18:09:48.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.601+0000 7f86a1600640 1 --2- 172.21.15.2:0/2700193928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0b2210 0x7f869c15d0e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f869c067890 tx=0x7f868402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c785f004b68d0bf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.602+0000 7f86a0c00640 1 -- 172.21.15.2:0/2700193928 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684031870 con 0x7f869c0b2210 2024-01-15T18:09:48.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.602+0000 7f86a0c00640 1 -- 172.21.15.2:0/2700193928 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86840319d0 con 0x7f869c0b2210 2024-01-15T18:09:48.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.602+0000 7f86a0c00640 1 -- 172.21.15.2:0/2700193928 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684031d60 con 0x7f869c0b2210 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.602+0000 7f86a3400640 1 -- 172.21.15.2:0/2700193928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0b2210 msgr2=0x7f869c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.602+0000 7f86a3400640 1 --2- 172.21.15.2:0/2700193928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0b2210 0x7f869c15d0e0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f869c067890 tx=0x7f868402b910 comp rx=0 tx=0).stop 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 -- 172.21.15.2:0/2700193928 shutdown_connections 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 --2- 172.21.15.2:0/2700193928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1c50 0x7f869c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 --2- 172.21.15.2:0/2700193928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0b2210 0x7f869c15d0e0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 --2- 172.21.15.2:0/2700193928 >> v2:172.21.15.137:3300/0 conn(0x7f869c0d1870 0x7f869c0d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 -- 172.21.15.2:0/2700193928 >> 172.21.15.2:0/2700193928 conn(0x7f869c0d30b0 msgr2=0x7f869c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 -- 172.21.15.2:0/2700193928 shutdown_connections 2024-01-15T18:09:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.603+0000 7f86a3400640 1 -- 172.21.15.2:0/2700193928 wait complete. 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.605+0000 7f86a3400640 1 Processor -- start 2024-01-15T18:09:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.605+0000 7f86a3400640 1 -- start start 2024-01-15T18:09:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a3400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f869c0b2210 0x7f869c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a3400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1870 0x7f869c1f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 0x7f869c1603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a3400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1f3fe0 con 0x7f869c0d1c50 2024-01-15T18:09:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a3400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0cd030 con 0x7f869c0b2210 2024-01-15T18:09:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a3400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c1f3510 con 0x7f869c0d1870 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.606+0000 7f86a2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 0x7f869c1603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 0x7f869c1603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33290/0 (socket says 172.21.15.2:33290) 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 -- 172.21.15.2:0/3578298524 learned_addr learned my addr 172.21.15.2:0/3578298524 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a1600640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1870 0x7f869c1f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 -- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1870 msgr2=0x7f869c1f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2000640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f869c0b2210 0x7f869c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1870 0x7f869c1f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 -- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f869c0b2210 msgr2=0x7f869c0ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f869c0b2210 0x7f869c0ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 -- 172.21.15.2:0/3578298524 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86840094d0 con 0x7f869c0d1c50 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f86a2a00640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 0x7f869c1603d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f868c007d80 tx=0x7f868c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.607+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c014420 con 0x7f869c0d1c50 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.608+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868c014580 con 0x7f869c0d1c50 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.608+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c014930 con 0x7f869c0d1c50 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.608+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c160910 con 0x7f869c0d1c50 2024-01-15T18:09:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.608+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c1f3a20 con 0x7f869c0d1c50 2024-01-15T18:09:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.609+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c074770 con 0x7f869c0d1c50 2024-01-15T18:09:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.610+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f868c014a90 con 0x7f869c0d1c50 2024-01-15T18:09:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.611+0000 7f869a000640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f867406c070 0x7f867406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.611+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f868c0891c0 con 0x7f869c0d1c50 2024-01-15T18:09:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.611+0000 7f86a2000640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f867406c070 0x7f867406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:48.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.612+0000 7f86a2000640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f867406c070 0x7f867406e530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8694006fd0 tx=0x7f8694008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.615+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f868c0557d0 con 0x7f869c0d1c50 2024-01-15T18:09:48.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:48.836+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47166", "target": ["mon-mgr", ""]}) v1 -- 0x7f869c0d0f20 con 0x7f867406c070 2024-01-15T18:09:53.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.284+0000 7f869a000640 1 -- 172.21.15.2:0/3578298524 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f869400e4c0 con 0x7f867406c070 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f867406c070 msgr2=0x7f867406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f867406c070 0x7f867406e530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f8694006fd0 tx=0x7f8694008040 comp rx=0 tx=0).stop 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 msgr2=0x7f869c1603d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 0x7f869c1603d0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f868c007d80 tx=0x7f868c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 shutdown_connections 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f867406c070 0x7f867406e530 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f869c0d1c50 0x7f869c1603d0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f869c0d1870 0x7f869c1f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 --2- 172.21.15.2:0/3578298524 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f869c0b2210 0x7f869c0ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.287+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 >> 172.21.15.2:0/3578298524 conn(0x7f869c0d30b0 msgr2=0x7f869c0c9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.288+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 shutdown_connections 2024-01-15T18:09:53.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.288+0000 7f86a3400640 1 -- 172.21.15.2:0/3578298524 wait complete. 2024-01-15T18:09:53.319 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471617 2024-01-15T18:09:53.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.526+0000 7f0e6cc00640 1 Processor -- start 2024-01-15T18:09:53.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.527+0000 7f0e6cc00640 1 -- start start 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.528+0000 7f0e6cc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0e680d1870 0x7f0e680d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.528+0000 7f0e6cc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0e680d1c50 0x7f0e680b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.528+0000 7f0e6cc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e680d1250 con 0x7f0e680b2210 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.528+0000 7f0e6cc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e680ce6d0 con 0x7f0e680d1870 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.528+0000 7f0e6cc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e680ce8e0 con 0x7f0e680d1c50 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e66a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0e680b2210 0x7f0e6815d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56360/0 (socket says 172.21.15.2:56360) 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e66a00640 1 -- 172.21.15.2:0/2316986015 learned_addr learned my addr 172.21.15.2:0/2316986015 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:53.532 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e67400640 1 --2- 172.21.15.2:0/2316986015 >> v2:172.21.15.2:3300/0 conn(0x7f0e680d1870 0x7f0e680d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e66000640 1 -- 172.21.15.2:0/2316986015 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469182055 0 0) 0x7f0e680d1250 con 0x7f0e680b2210 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e67e00640 1 --2- 172.21.15.2:0/2316986015 >> v2:172.21.15.137:3300/0 conn(0x7f0e680d1c50 0x7f0e680b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e66000640 1 -- 172.21.15.2:0/2316986015 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e48003520 con 0x7f0e680b2210 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e67400640 1 -- 172.21.15.2:0/2316986015 >> v2:172.21.15.137:3300/0 conn(0x7f0e680d1c50 msgr2=0x7f0e680b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e67400640 1 --2- 172.21.15.2:0/2316986015 >> v2:172.21.15.137:3300/0 conn(0x7f0e680d1c50 0x7f0e680b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e67400640 1 -- 172.21.15.2:0/2316986015 >> v1:172.21.15.2:6789/0 conn(0x7f0e680b2210 legacy=0x7f0e6815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.529+0000 7f0e67400640 1 -- 172.21.15.2:0/2316986015 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e680cebc0 con 0x7f0e680d1870 2024-01-15T18:09:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.531+0000 7f0e67400640 1 --2- 172.21.15.2:0/2316986015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1870 0x7f0e680d3bd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f0e6808d7e0 tx=0x7f0e5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d7ae6ef89d9f5b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.531+0000 7f0e66000640 1 -- 172.21.15.2:0/2316986015 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50031830 con 0x7f0e680d1870 2024-01-15T18:09:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.531+0000 7f0e66000640 1 -- 172.21.15.2:0/2316986015 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e50031990 con 0x7f0e680d1870 2024-01-15T18:09:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.531+0000 7f0e66000640 1 -- 172.21.15.2:0/2316986015 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e50031d00 con 0x7f0e680d1870 2024-01-15T18:09:53.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.531+0000 7f0e6cc00640 1 -- 172.21.15.2:0/2316986015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1870 msgr2=0x7f0e680d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.531+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/2316986015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1870 0x7f0e680d3bd0 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f0e6808d7e0 tx=0x7f0e5002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.532+0000 7f0e6cc00640 1 -- 172.21.15.2:0/2316986015 shutdown_connections 2024-01-15T18:09:53.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.532+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/2316986015 >> v2:172.21.15.137:3300/0 conn(0x7f0e680d1c50 0x7f0e680b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.532+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/2316986015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1870 0x7f0e680d3bd0 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.532+0000 7f0e6cc00640 1 -- 172.21.15.2:0/2316986015 >> 172.21.15.2:0/2316986015 conn(0x7f0e680d30b0 msgr2=0x7f0e680d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:53.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.532+0000 7f0e6cc00640 1 -- 172.21.15.2:0/2316986015 shutdown_connections 2024-01-15T18:09:53.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.532+0000 7f0e6cc00640 1 -- 172.21.15.2:0/2316986015 wait complete. 2024-01-15T18:09:53.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.534+0000 7f0e6cc00640 1 Processor -- start 2024-01-15T18:09:53.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.534+0000 7f0e6cc00640 1 -- start start 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e6cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e680b2210 0x7f0e680a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e6cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e680d1870 0x7f0e681f6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e6cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 0x7f0e6814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e6cc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e680a0ee0 con 0x7f0e680d1c50 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e6cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e680c9710 con 0x7f0e680d1870 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e6cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6814b010 con 0x7f0e680b2210 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e67e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 0x7f0e6814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:53.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e67e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 0x7f0e6814b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40700/0 (socket says 172.21.15.2:40700) 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e67e00640 1 -- 172.21.15.2:0/4027475770 learned_addr learned my addr 172.21.15.2:0/4027475770 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.535+0000 7f0e66a00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e680d1870 0x7f0e681f6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67e00640 1 -- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e680b2210 msgr2=0x7f0e680a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67400640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e680b2210 0x7f0e680a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67e00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e680b2210 0x7f0e680a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67e00640 1 -- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e680d1870 msgr2=0x7f0e681f6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67e00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e680d1870 0x7f0e681f6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67e00640 1 -- 172.21.15.2:0/4027475770 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e50009400 con 0x7f0e680d1c50 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e66a00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e680d1870 0x7f0e681f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:53.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e67e00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 0x7f0e6814b780 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000d5d0 tx=0x7f0e6000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:53.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e600125f0 con 0x7f0e680d1c50 2024-01-15T18:09:53.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e60012770 con 0x7f0e680d1c50 2024-01-15T18:09:53.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e681f6bc0 con 0x7f0e680d1c50 2024-01-15T18:09:53.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e60012b20 con 0x7f0e680d1c50 2024-01-15T18:09:53.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.536+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e681f7140 con 0x7f0e680d1c50 2024-01-15T18:09:53.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.537+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e68074770 con 0x7f0e680d1c50 2024-01-15T18:09:53.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.539+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f0e60012c80 con 0x7f0e680d1c50 2024-01-15T18:09:53.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.540+0000 7f0e5f400640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0e4006c070 0x7f0e4006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:53.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.540+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f0e60088d80 con 0x7f0e680d1c50 2024-01-15T18:09:53.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.541+0000 7f0e67400640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0e4006c070 0x7f0e4006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:53.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.542+0000 7f0e67400640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0e4006c070 0x7f0e4006e530 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0e50036000 tx=0x7f0e50008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:53.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.545+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e60055370 con 0x7f0e680d1c50 2024-01-15T18:09:53.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.768+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471617", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e6815cc00 con 0x7f0e4006c070 2024-01-15T18:09:53.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.808+0000 7f0e5f400640 1 -- 172.21.15.2:0/4027475770 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0e6815cc00 con 0x7f0e4006c070 2024-01-15T18:09:53.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.811+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0e4006c070 msgr2=0x7f0e4006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.811+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0e4006c070 0x7f0e4006e530 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f0e50036000 tx=0x7f0e50008030 comp rx=0 tx=0).stop 2024-01-15T18:09:53.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.811+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 msgr2=0x7f0e6814b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:53.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.811+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 0x7f0e6814b780 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f0e6000d5d0 tx=0x7f0e6000da90 comp rx=0 tx=0).stop 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 shutdown_connections 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0e4006c070 0x7f0e4006e530 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0e680d1c50 0x7f0e6814b780 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0e680d1870 0x7f0e681f6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 --2- 172.21.15.2:0/4027475770 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0e680b2210 0x7f0e680a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 >> 172.21.15.2:0/4027475770 conn(0x7f0e680d30b0 msgr2=0x7f0e6815c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 shutdown_connections 2024-01-15T18:09:53.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:53.812+0000 7f0e6cc00640 1 -- 172.21.15.2:0/4027475770 wait complete. 2024-01-15T18:09:53.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471612 2024-01-15T18:09:54.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.041+0000 7f3537000640 1 Processor -- start 2024-01-15T18:09:54.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.042+0000 7f3537000640 1 -- start start 2024-01-15T18:09:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.043+0000 7f3537000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 0x7f3530165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.043+0000 7f3537000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35300ce230 con 0x7f3530144c00 2024-01-15T18:09:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.043+0000 7f3537000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35300ce510 con 0x7f353015c450 2024-01-15T18:09:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.043+0000 7f3537000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35300ce7f0 con 0x7f353006d000 2024-01-15T18:09:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.044+0000 7f352fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3530144c00 0x7f35300cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56362/0 (socket says 172.21.15.2:56362) 2024-01-15T18:09:54.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.044+0000 7f352fe00640 1 -- 172.21.15.2:0/3530105404 learned_addr learned my addr 172.21.15.2:0/3530105404 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.044+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274584857 0 0) 0x7f35300ce230 con 0x7f3530144c00 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.044+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3510003520 con 0x7f3530144c00 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.044+0000 7f3535000640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 0x7f3530165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.044+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2887497032 0 0) 0x7f3510003520 con 0x7f3530144c00 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 msgr2=0x7f3530165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 0x7f3530165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 >> v1:172.21.15.137:6789/0 conn(0x7f353015c450 legacy=0x7f353006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35300705f0 con 0x7f3530144c00 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3520001670 con 0x7f3530144c00 2024-01-15T18:09:54.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 >> v1:172.21.15.2:6789/0 conn(0x7f3530144c00 legacy=0x7f35300cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35100039a0 0x7f3510003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 0x7f35100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35300ce7f0 con 0x7f35100039a0 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351000bb90 con 0x7f3510004d00 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351000bd50 con 0x7f353006d000 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352fe00640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35100039a0 0x7f3510003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f3527e00640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 0x7f35100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f3535000640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352fe00640 1 -- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 msgr2=0x7f35100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.045+0000 7f352fe00640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f353006d000 0x7f35100047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.046+0000 7f352fe00640 1 -- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3510004d00 msgr2=0x7f3510005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.046+0000 7f352fe00640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3510004d00 0x7f3510005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.046+0000 7f352fe00640 1 -- 172.21.15.2:0/3530105404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510003520 con 0x7f35100039a0 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f352fe00640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35100039a0 0x7f3510003dc0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f352000c0e0 tx=0x7f35100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a68bf6e143cd3810 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3520005ad0 con 0x7f35100039a0 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3520005c30 con 0x7f35100039a0 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f352f400640 1 -- 172.21.15.2:0/3530105404 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f352002c810 con 0x7f35100039a0 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f3537000640 1 -- 172.21.15.2:0/3530105404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35100039a0 msgr2=0x7f3510003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f3537000640 1 --2- 172.21.15.2:0/3530105404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f35100039a0 0x7f3510003dc0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f352000c0e0 tx=0x7f35100057f0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.047+0000 7f3535000640 1 -- 172.21.15.2:0/3530105404 reap_dead start 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.048+0000 7f3537000640 1 -- 172.21.15.2:0/3530105404 shutdown_connections 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.048+0000 7f3537000640 1 -- 172.21.15.2:0/3530105404 >> 172.21.15.2:0/3530105404 conn(0x7f35300b45f0 msgr2=0x7f35300b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.048+0000 7f3537000640 1 -- 172.21.15.2:0/3530105404 shutdown_connections 2024-01-15T18:09:54.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.048+0000 7f3537000640 1 -- 172.21.15.2:0/3530105404 wait complete. 2024-01-15T18:09:54.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.050+0000 7f3537000640 1 Processor -- start 2024-01-15T18:09:54.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.050+0000 7f3537000640 1 -- start start 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3537000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35100039a0 0x7f353013b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3537000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35300afbb0 0x7f35301f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3537000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 0x7f35300b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3537000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35300b0010 con 0x7f3530154800 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3537000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35300b1300 con 0x7f35300afbb0 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3537000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35300b0240 con 0x7f35100039a0 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 0x7f35300b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 0x7f35300b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40718/0 (socket says 172.21.15.2:40718) 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 -- 172.21.15.2:0/3469892734 learned_addr learned my addr 172.21.15.2:0/3469892734 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3527e00640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35300afbb0 0x7f35301f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 -- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35100039a0 msgr2=0x7f353013b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35100039a0 0x7f353013b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 -- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35300afbb0 msgr2=0x7f35301f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35300afbb0 0x7f35301f5880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.051+0000 7f3535000640 1 -- 172.21.15.2:0/3469892734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35300b0520 con 0x7f3530154800 2024-01-15T18:09:54.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.052+0000 7f3535000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 0x7f35300b2af0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f353006e7d0 tx=0x7f3528009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.052+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528007cf0 con 0x7f3530154800 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.052+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3528007e50 con 0x7f3530154800 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.052+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3528014640 con 0x7f3530154800 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.052+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35300b0800 con 0x7f3530154800 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.052+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3530139180 con 0x7f3530154800 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.053+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35301393a0 con 0x7f3530154800 2024-01-15T18:09:54.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.055+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f352801d3b0 con 0x7f3530154800 2024-01-15T18:09:54.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.056+0000 7f352cc00640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350806c070 0x7f350806e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.056+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3528088600 con 0x7f3530154800 2024-01-15T18:09:54.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.056+0000 7f352fe00640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350806c070 0x7f350806e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.058+0000 7f352fe00640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350806c070 0x7f350806e530 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3520034000 tx=0x7f3520007000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:54.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.060+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3528054c10 con 0x7f3530154800 2024-01-15T18:09:54.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.279+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471612", "target": ["mon-mgr", ""]}) v1 -- 0x7f35300cead0 con 0x7f350806c070 2024-01-15T18:09:54.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.321+0000 7f352cc00640 1 -- 172.21.15.2:0/3469892734 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f35200013b0 con 0x7f350806c070 2024-01-15T18:09:54.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.323+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350806c070 msgr2=0x7f350806e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.323+0000 7f3537000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350806c070 0x7f350806e530 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f3520034000 tx=0x7f3520007000 comp rx=0 tx=0).stop 2024-01-15T18:09:54.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.323+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 msgr2=0x7f35300b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.323+0000 7f3537000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 0x7f35300b2af0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f353006e7d0 tx=0x7f3528009d30 comp rx=0 tx=0).stop 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 shutdown_connections 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f350806c070 0x7f350806e530 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3530154800 0x7f35300b2af0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f35300afbb0 0x7f35301f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 --2- 172.21.15.2:0/3469892734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f35100039a0 0x7f353013b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 >> 172.21.15.2:0/3469892734 conn(0x7f35300b45f0 msgr2=0x7f35300b3a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 shutdown_connections 2024-01-15T18:09:54.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.324+0000 7f3537000640 1 -- 172.21.15.2:0/3469892734 wait complete. 2024-01-15T18:09:54.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47168 2024-01-15T18:09:54.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.553+0000 7fa39e200640 1 Processor -- start 2024-01-15T18:09:54.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.553+0000 7fa39e200640 1 -- start start 2024-01-15T18:09:54.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39e200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3980d1870 0x7fa3980d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39e200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3980d1250 con 0x7fa3980d1c50 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39e200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3980ce8e0 con 0x7fa3980b2210 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39e200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3980ce6d0 con 0x7fa3980d1870 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39ce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3980d1870 0x7fa3980d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39ce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3980d1870 0x7fa3980d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40724/0 (socket says 172.21.15.2:40724) 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.554+0000 7fa39ce00640 1 -- 172.21.15.2:0/3574511127 learned_addr learned my addr 172.21.15.2:0/3574511127 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:54.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.555+0000 7fa39ce00640 1 -- 172.21.15.2:0/3574511127 >> v1:172.21.15.137:6789/0 conn(0x7fa3980b2210 legacy=0x7fa39815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.555+0000 7fa39ce00640 1 -- 172.21.15.2:0/3574511127 >> v1:172.21.15.2:6789/0 conn(0x7fa3980d1c50 legacy=0x7fa3980b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.555+0000 7fa39ce00640 1 -- 172.21.15.2:0/3574511127 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3980cebc0 con 0x7fa3980d1870 2024-01-15T18:09:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.556+0000 7fa39ce00640 1 --2- 172.21.15.2:0/3574511127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa3980d3bd0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa388009870 tx=0x7fa38802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95c7d7178ed79879 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.556+0000 7fa397e00640 1 -- 172.21.15.2:0/3574511127 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388030860 con 0x7fa3980d1870 2024-01-15T18:09:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.556+0000 7fa397e00640 1 -- 172.21.15.2:0/3574511127 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3880309c0 con 0x7fa3980d1870 2024-01-15T18:09:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.556+0000 7fa397e00640 1 -- 172.21.15.2:0/3574511127 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa388030d50 con 0x7fa3980d1870 2024-01-15T18:09:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 -- 172.21.15.2:0/3574511127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 msgr2=0x7fa3980d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 --2- 172.21.15.2:0/3574511127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa3980d3bd0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fa388009870 tx=0x7fa38802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 -- 172.21.15.2:0/3574511127 shutdown_connections 2024-01-15T18:09:54.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 --2- 172.21.15.2:0/3574511127 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa3980d3bd0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 -- 172.21.15.2:0/3574511127 >> 172.21.15.2:0/3574511127 conn(0x7fa3980d30b0 msgr2=0x7fa3980d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:54.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 -- 172.21.15.2:0/3574511127 shutdown_connections 2024-01-15T18:09:54.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.557+0000 7fa39e200640 1 -- 172.21.15.2:0/3574511127 wait complete. 2024-01-15T18:09:54.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.559+0000 7fa39e200640 1 Processor -- start 2024-01-15T18:09:54.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.559+0000 7fa39e200640 1 -- start start 2024-01-15T18:09:54.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa39e200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3980b2210 0x7fa398154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa39e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa398157b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa39e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3980d1c50 0x7fa3980c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa39e200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3981580a0 con 0x7fa3980d1870 2024-01-15T18:09:54.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa39e200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3980c8f00 con 0x7fa3980b2210 2024-01-15T18:09:54.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa39e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa398158290 con 0x7fa3980d1c50 2024-01-15T18:09:54.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa38fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa398157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa38fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa398157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40730/0 (socket says 172.21.15.2:40730) 2024-01-15T18:09:54.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.560+0000 7fa38fe00640 1 -- 172.21.15.2:0/3210290046 learned_addr learned my addr 172.21.15.2:0/3210290046 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa38fe00640 1 -- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3980d1c50 msgr2=0x7fa3980c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa39ce00640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3980b2210 0x7fa398154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa38fe00640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3980d1c50 0x7fa3980c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa38fe00640 1 -- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3980b2210 msgr2=0x7fa398154430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa38fe00640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3980b2210 0x7fa398154430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa38fe00640 1 -- 172.21.15.2:0/3210290046 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3880094d0 con 0x7fa3980d1870 2024-01-15T18:09:54.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa38fe00640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa398157b60 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fa38000c400 tx=0x7fa38000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3800035f0 con 0x7fa3980d1870 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa380003750 con 0x7fa3980d1870 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa380003b00 con 0x7fa3980d1870 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa398158570 con 0x7fa3980d1870 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.561+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa398158b30 con 0x7fa3980d1870 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.562+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa398159050 con 0x7fa3980d1870 2024-01-15T18:09:54.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.563+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fa380003c60 con 0x7fa3980d1870 2024-01-15T18:09:54.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.564+0000 7fa395600640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa37006c040 0x7fa37006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:54.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.564+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fa380087ab0 con 0x7fa3980d1870 2024-01-15T18:09:54.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.565+0000 7fa39ce00640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa37006c040 0x7fa37006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:54.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.566+0000 7fa39ce00640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa37006c040 0x7fa37006e500 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa388036000 tx=0x7fa38803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:54.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.568+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa380054170 con 0x7fa3980d1870 2024-01-15T18:09:54.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.790+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47168", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3980d0ef0 con 0x7fa37006c040 2024-01-15T18:09:54.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.835+0000 7fa395600640 1 -- 172.21.15.2:0/3210290046 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa3980d0ef0 con 0x7fa37006c040 2024-01-15T18:09:54.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa37006c040 msgr2=0x7fa37006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa37006c040 0x7fa37006e500 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa388036000 tx=0x7fa38803b040 comp rx=0 tx=0).stop 2024-01-15T18:09:54.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 msgr2=0x7fa398157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:54.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa398157b60 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fa38000c400 tx=0x7fa38000c8c0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 shutdown_connections 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa37006c040 0x7fa37006e500 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3980d1c50 0x7fa3980c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3980d1870 0x7fa398157b60 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 --2- 172.21.15.2:0/3210290046 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3980b2210 0x7fa398154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 >> 172.21.15.2:0/3210290046 conn(0x7fa3980d30b0 msgr2=0x7fa3980afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 shutdown_connections 2024-01-15T18:09:54.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:54.838+0000 7fa39e200640 1 -- 172.21.15.2:0/3210290046 wait complete. 2024-01-15T18:09:54.861 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47165 2024-01-15T18:09:55.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.067+0000 7fe97ca00640 1 Processor -- start 2024-01-15T18:09:55.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.067+0000 7fe97ca00640 1 -- start start 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe97ca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe97815f260 0x7fe97806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe97ca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe9780720d0 0x7fe97806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe97ca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe978071090 con 0x7fe978143d20 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe97ca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe978071430 con 0x7fe97815f260 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe97ca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe978144100 con 0x7fe9780720d0 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe975e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe97815f260 0x7fe97806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe975e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe97815f260 0x7fe97806c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40744/0 (socket says 172.21.15.2:40744) 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.069+0000 7fe975e00640 1 -- 172.21.15.2:0/4261703468 learned_addr learned my addr 172.21.15.2:0/4261703468 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.070+0000 7fe977200640 1 --2- 172.21.15.2:0/4261703468 >> v2:172.21.15.137:3300/0 conn(0x7fe9780720d0 0x7fe97806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.070+0000 7fe975e00640 1 -- 172.21.15.2:0/4261703468 >> v2:172.21.15.137:3300/0 conn(0x7fe9780720d0 msgr2=0x7fe97806c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:55.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.070+0000 7fe975e00640 1 --2- 172.21.15.2:0/4261703468 >> v2:172.21.15.137:3300/0 conn(0x7fe9780720d0 0x7fe97806c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.070+0000 7fe975e00640 1 -- 172.21.15.2:0/4261703468 >> v1:172.21.15.137:6789/0 conn(0x7fe978143d20 legacy=0x7fe97815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:55.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.070+0000 7fe975e00640 1 -- 172.21.15.2:0/4261703468 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe978144560 con 0x7fe97815f260 2024-01-15T18:09:55.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.072+0000 7fe975e00640 1 --2- 172.21.15.2:0/4261703468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe97815f260 0x7fe97806c150 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe9780cfc70 tx=0x7fe96002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb1bbedbc8d9c8f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.072+0000 7fe975400640 1 -- 172.21.15.2:0/4261703468 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9600317e0 con 0x7fe97815f260 2024-01-15T18:09:55.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.072+0000 7fe975400640 1 -- 172.21.15.2:0/4261703468 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe960031940 con 0x7fe97815f260 2024-01-15T18:09:55.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.072+0000 7fe975400640 1 -- 172.21.15.2:0/4261703468 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960031cd0 con 0x7fe97815f260 2024-01-15T18:09:55.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 -- 172.21.15.2:0/4261703468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe97815f260 msgr2=0x7fe97806c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 --2- 172.21.15.2:0/4261703468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe97815f260 0x7fe97806c150 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe9780cfc70 tx=0x7fe96002b880 comp rx=0 tx=0).stop 2024-01-15T18:09:55.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 -- 172.21.15.2:0/4261703468 shutdown_connections 2024-01-15T18:09:55.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 --2- 172.21.15.2:0/4261703468 >> v2:172.21.15.137:3300/0 conn(0x7fe9780720d0 0x7fe97806c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 --2- 172.21.15.2:0/4261703468 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe97815f260 0x7fe97806c150 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 -- 172.21.15.2:0/4261703468 >> 172.21.15.2:0/4261703468 conn(0x7fe9780b0840 msgr2=0x7fe9780b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:55.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 -- 172.21.15.2:0/4261703468 shutdown_connections 2024-01-15T18:09:55.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.073+0000 7fe97ca00640 1 -- 172.21.15.2:0/4261703468 wait complete. 2024-01-15T18:09:55.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.075+0000 7fe97ca00640 1 Processor -- start 2024-01-15T18:09:55.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.075+0000 7fe97ca00640 1 -- start start 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe97ca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9780720d0 0x7fe978156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe97ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 0x7fe9780af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe97ca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe97815f260 0x7fe97813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe97ca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe978154800 con 0x7fe978143d20 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe97ca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe978154990 con 0x7fe97815f260 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe97ca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9780ad820 con 0x7fe9780720d0 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.076+0000 7fe975e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 0x7fe9780af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 0x7fe9780af470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40746/0 (socket says 172.21.15.2:40746) 2024-01-15T18:09:55.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 -- 172.21.15.2:0/2180574633 learned_addr learned my addr 172.21.15.2:0/2180574633 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe976800640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9780720d0 0x7fe978156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe977200640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe97815f260 0x7fe97813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 -- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9780720d0 msgr2=0x7fe978156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9780720d0 0x7fe978156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 -- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe97815f260 msgr2=0x7fe97813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe97815f260 0x7fe97813fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe975e00640 1 -- 172.21.15.2:0/2180574633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9600094d0 con 0x7fe978143d20 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.077+0000 7fe976800640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9780720d0 0x7fe978156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.078+0000 7fe975e00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 0x7fe9780af470 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe9600090b0 tx=0x7fe9600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.078+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960032880 con 0x7fe978143d20 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.078+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9600329e0 con 0x7fe978143d20 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.078+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9780adb00 con 0x7fe978143d20 2024-01-15T18:09:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.078+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960032d90 con 0x7fe978143d20 2024-01-15T18:09:55.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.078+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9780ae0c0 con 0x7fe978143d20 2024-01-15T18:09:55.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.079+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe978074770 con 0x7fe978143d20 2024-01-15T18:09:55.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.081+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fe96004e450 con 0x7fe978143d20 2024-01-15T18:09:55.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.081+0000 7fe96ea00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe95006c070 0x7fe95006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.082+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fe9600aa930 con 0x7fe978143d20 2024-01-15T18:09:55.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.082+0000 7fe976800640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe95006c070 0x7fe95006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.083+0000 7fe976800640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe95006c070 0x7fe95006e530 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe978142da0 tx=0x7fe968009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.086+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe960077060 con 0x7fe978143d20 2024-01-15T18:09:55.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.306+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47165", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9780ad990 con 0x7fe95006c070 2024-01-15T18:09:55.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.346+0000 7fe96ea00640 1 -- 172.21.15.2:0/2180574633 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe9780ad990 con 0x7fe95006c070 2024-01-15T18:09:55.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe95006c070 msgr2=0x7fe95006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe95006c070 0x7fe95006e530 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fe978142da0 tx=0x7fe968009040 comp rx=0 tx=0).stop 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 msgr2=0x7fe9780af470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 0x7fe9780af470 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe9600090b0 tx=0x7fe9600046c0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 shutdown_connections 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe95006c070 0x7fe95006e530 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe97815f260 0x7fe97813fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe978143d20 0x7fe9780af470 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 --2- 172.21.15.2:0/2180574633 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe9780720d0 0x7fe978156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 >> 172.21.15.2:0/2180574633 conn(0x7fe9780b0840 msgr2=0x7fe978143010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 shutdown_connections 2024-01-15T18:09:55.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.350+0000 7fe97ca00640 1 -- 172.21.15.2:0/2180574633 wait complete. 2024-01-15T18:09:55.375 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47163 2024-01-15T18:09:55.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.596+0000 7f643b200640 1 Processor -- start 2024-01-15T18:09:55.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.596+0000 7f643b200640 1 -- start start 2024-01-15T18:09:55.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.597+0000 7f643b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f643c143620 0x7f643c0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.597+0000 7f643b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f643c0b2a90 0x7f643c071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f643b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f643c0b2fa0 con 0x7f643c071770 2024-01-15T18:09:55.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f643b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c071c00 con 0x7f643c0b2a90 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f643b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f643c0b3280 con 0x7f643c143620 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f6439400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f643c0b2a90 0x7f643c071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f6439400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f643c0b2a90 0x7f643c071230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40750/0 (socket says 172.21.15.2:40750) 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f6439400640 1 -- 172.21.15.2:0/3517217051 learned_addr learned my addr 172.21.15.2:0/3517217051 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f643a800640 1 --1- 172.21.15.2:0/3517217051 >> v1:172.21.15.2:6789/0 conn(0x7f643c071770 0x7f643c070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56372/0 (socket says 172.21.15.2:56372) 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.598+0000 7f6439e00640 1 --2- 172.21.15.2:0/3517217051 >> v2:172.21.15.137:3300/0 conn(0x7f643c143620 0x7f643c0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.599+0000 7f6439400640 1 -- 172.21.15.2:0/3517217051 >> v2:172.21.15.137:3300/0 conn(0x7f643c143620 msgr2=0x7f643c0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.599+0000 7f6439400640 1 --2- 172.21.15.2:0/3517217051 >> v2:172.21.15.137:3300/0 conn(0x7f643c143620 0x7f643c0b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.599+0000 7f6439400640 1 -- 172.21.15.2:0/3517217051 >> v1:172.21.15.2:6789/0 conn(0x7f643c071770 legacy=0x7f643c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.599+0000 7f6439400640 1 -- 172.21.15.2:0/3517217051 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c0b3560 con 0x7f643c0b2a90 2024-01-15T18:09:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.600+0000 7f6439400640 1 --2- 172.21.15.2:0/3517217051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c0b2a90 0x7f643c071230 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f643c06bb70 tx=0x7f642402e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423ef781dedc1a7b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.600+0000 7f6438a00640 1 -- 172.21.15.2:0/3517217051 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424036440 con 0x7f643c0b2a90 2024-01-15T18:09:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.600+0000 7f6438a00640 1 -- 172.21.15.2:0/3517217051 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64240365a0 con 0x7f643c0b2a90 2024-01-15T18:09:55.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.600+0000 7f6438a00640 1 -- 172.21.15.2:0/3517217051 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424036930 con 0x7f643c0b2a90 2024-01-15T18:09:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 -- 172.21.15.2:0/3517217051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c0b2a90 msgr2=0x7f643c071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 --2- 172.21.15.2:0/3517217051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c0b2a90 0x7f643c071230 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f643c06bb70 tx=0x7f642402e180 comp rx=0 tx=0).stop 2024-01-15T18:09:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 -- 172.21.15.2:0/3517217051 shutdown_connections 2024-01-15T18:09:55.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 --2- 172.21.15.2:0/3517217051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c0b2a90 0x7f643c071230 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 --2- 172.21.15.2:0/3517217051 >> v2:172.21.15.137:3300/0 conn(0x7f643c143620 0x7f643c0b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 -- 172.21.15.2:0/3517217051 >> 172.21.15.2:0/3517217051 conn(0x7f643c0b0840 msgr2=0x7f643c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:09:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 -- 172.21.15.2:0/3517217051 shutdown_connections 2024-01-15T18:09:55.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.601+0000 7f643b200640 1 -- 172.21.15.2:0/3517217051 wait complete. 2024-01-15T18:09:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.603+0000 7f643b200640 1 Processor -- start 2024-01-15T18:09:55.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.603+0000 7f643b200640 1 -- start start 2024-01-15T18:09:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f643c071770 0x7f643c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f643c0b2a90 0x7f643c136500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 0x7f643c0ac8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.607 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c136e20 con 0x7f643c143620 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643c1f6e20 con 0x7f643c071770 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643c1f7100 con 0x7f643c0b2a90 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 0x7f643c0ac8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 0x7f643c0ac8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40758/0 (socket says 172.21.15.2:40758) 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f643a800640 1 -- 172.21.15.2:0/3209771642 learned_addr learned my addr 172.21.15.2:0/3209771642 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f6439e00640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f643c071770 0x7f643c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.604+0000 7f6439400640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f643c0b2a90 0x7f643c136500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643a800640 1 -- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f643c0b2a90 msgr2=0x7f643c136500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643a800640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f643c0b2a90 0x7f643c136500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643a800640 1 -- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f643c071770 msgr2=0x7f643c1f65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:09:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643a800640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f643c071770 0x7f643c1f65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:09:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643a800640 1 -- 172.21.15.2:0/3209771642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642400b770 con 0x7f643c143620 2024-01-15T18:09:55.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f6439e00640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f643c071770 0x7f643c1f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643a800640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 0x7f643c0ac8a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f643400c780 tx=0x7f643400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643400ce90 con 0x7f643c143620 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643400a640 con 0x7f643c143620 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643400a9f0 con 0x7f643c143620 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f643c1f73e0 con 0x7f643c143620 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.605+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643c1f6b10 con 0x7f643c143620 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.606+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643c137300 con 0x7f643c143620 2024-01-15T18:09:55.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.608+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f643400ab50 con 0x7f643c143620 2024-01-15T18:09:55.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.608+0000 7f6432000640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f641406c040 0x7f641406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:09:55.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.609+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f6434087fe0 con 0x7f643c143620 2024-01-15T18:09:55.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.609+0000 7f6439e00640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f641406c040 0x7f641406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:09:55.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.610+0000 7f6439e00640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f641406c040 0x7f641406e500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f642c006fd0 tx=0x7f642c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:09:55.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.613+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f643c137300 con 0x7f643c143620 2024-01-15T18:09:55.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:09:55.840+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47163", "target": ["mon-mgr", ""]}) v1 -- 0x7f643c143a00 con 0x7f641406c040 2024-01-15T18:10:02.141 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:02.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:03.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.280+0000 7f6432000640 1 -- 172.21.15.2:0/3209771642 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f642c009530 con 0x7f641406c040 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f641406c040 msgr2=0x7f641406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f641406c040 0x7f641406e500 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f642c006fd0 tx=0x7f642c008040 comp rx=0 tx=0).stop 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 msgr2=0x7f643c0ac8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 0x7f643c0ac8a0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f643400c780 tx=0x7f643400cc40 comp rx=0 tx=0).stop 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 shutdown_connections 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f641406c040 0x7f641406e500 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f643c143620 0x7f643c0ac8a0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f643c0b2a90 0x7f643c136500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 --2- 172.21.15.2:0/3209771642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f643c071770 0x7f643c1f65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.284+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 >> 172.21.15.2:0/3209771642 conn(0x7f643c0b0840 msgr2=0x7f643c0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.285+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 shutdown_connections 2024-01-15T18:10:03.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.285+0000 7f643b200640 1 -- 172.21.15.2:0/3209771642 wait complete. 2024-01-15T18:10:03.311 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone47167 2024-01-15T18:10:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.534+0000 7fc944200640 1 Processor -- start 2024-01-15T18:10:03.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.534+0000 7fc944200640 1 -- start start 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.535+0000 7fc944200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc93c144c00 0x7fc93c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.535+0000 7fc944200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc93c0cdd70 0x7fc93c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc944200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93c0ce230 con 0x7fc93c15c450 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc944200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c0ce510 con 0x7fc93c0cdd70 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc944200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c0ce7f0 con 0x7fc93c144c00 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc93c0cdd70 0x7fc93c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc93c0cdd70 0x7fc93c06c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41556/0 (socket says 172.21.15.2:41556) 2024-01-15T18:10:03.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 -- 172.21.15.2:0/30281198 learned_addr learned my addr 172.21.15.2:0/30281198 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc941800640 1 --2- 172.21.15.2:0/30281198 >> v2:172.21.15.137:3300/0 conn(0x7fc93c144c00 0x7fc93c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 -- 172.21.15.2:0/30281198 >> v2:172.21.15.137:3300/0 conn(0x7fc93c144c00 msgr2=0x7fc93c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 --2- 172.21.15.2:0/30281198 >> v2:172.21.15.137:3300/0 conn(0x7fc93c144c00 0x7fc93c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 -- 172.21.15.2:0/30281198 >> v1:172.21.15.137:6789/0 conn(0x7fc93c15c450 legacy=0x7fc93c0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc940e00640 1 -- 172.21.15.2:0/30281198 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c0705f0 con 0x7fc93c0cdd70 2024-01-15T18:10:03.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.536+0000 7fc941800640 1 --2- 172.21.15.2:0/30281198 >> v2:172.21.15.137:3300/0 conn(0x7fc93c144c00 0x7fc93c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:03.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.537+0000 7fc940e00640 1 --2- 172.21.15.2:0/30281198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c0cdd70 0x7fc93c06c8f0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0b1c60 tx=0x7fc92402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d46cd03f161114f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:03.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.538+0000 7fc93be00640 1 -- 172.21.15.2:0/30281198 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924031870 con 0x7fc93c0cdd70 2024-01-15T18:10:03.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.538+0000 7fc93be00640 1 -- 172.21.15.2:0/30281198 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9240319d0 con 0x7fc93c0cdd70 2024-01-15T18:10:03.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.538+0000 7fc93be00640 1 -- 172.21.15.2:0/30281198 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924031d60 con 0x7fc93c0cdd70 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.538+0000 7fc944200640 1 -- 172.21.15.2:0/30281198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c0cdd70 msgr2=0x7fc93c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.538+0000 7fc944200640 1 --2- 172.21.15.2:0/30281198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c0cdd70 0x7fc93c06c8f0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0b1c60 tx=0x7fc92402b910 comp rx=0 tx=0).stop 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.539+0000 7fc944200640 1 -- 172.21.15.2:0/30281198 shutdown_connections 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.539+0000 7fc944200640 1 --2- 172.21.15.2:0/30281198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c0cdd70 0x7fc93c06c8f0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.539+0000 7fc944200640 1 --2- 172.21.15.2:0/30281198 >> v2:172.21.15.137:3300/0 conn(0x7fc93c144c00 0x7fc93c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.539+0000 7fc944200640 1 -- 172.21.15.2:0/30281198 >> 172.21.15.2:0/30281198 conn(0x7fc93c0b45f0 msgr2=0x7fc93c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.539+0000 7fc944200640 1 -- 172.21.15.2:0/30281198 shutdown_connections 2024-01-15T18:10:03.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.539+0000 7fc944200640 1 -- 172.21.15.2:0/30281198 wait complete. 2024-01-15T18:10:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.541+0000 7fc944200640 1 Processor -- start 2024-01-15T18:10:03.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.541+0000 7fc944200640 1 -- start start 2024-01-15T18:10:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc944200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc93c0cdd70 0x7fc93c0ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc944200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc93c144c00 0x7fc93c0b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc944200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 0x7fc93c0b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc944200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c155550 con 0x7fc93c15c450 2024-01-15T18:10:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc944200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0b3360 con 0x7fc93c0cdd70 2024-01-15T18:10:03.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc944200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0adad0 con 0x7fc93c144c00 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 0x7fc93c0b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 0x7fc93c0b0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41570/0 (socket says 172.21.15.2:41570) 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 -- 172.21.15.2:0/1846902411 learned_addr learned my addr 172.21.15.2:0/1846902411 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc941800640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc93c0cdd70 0x7fc93c0ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc940e00640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc93c144c00 0x7fc93c0b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 -- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc93c144c00 msgr2=0x7fc93c0b2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc93c144c00 0x7fc93c0b2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 -- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc93c0cdd70 msgr2=0x7fc93c0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.542+0000 7fc942200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc93c0cdd70 0x7fc93c0ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc942200640 1 -- 172.21.15.2:0/1846902411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240094d0 con 0x7fc93c15c450 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc942200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 0x7fc93c0b0470 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc93400ba10 tx=0x7fc93400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93400cb40 con 0x7fc93c15c450 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93400cca0 con 0x7fc93c15c450 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934014680 con 0x7fc93c15c450 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c0acec0 con 0x7fc93c15c450 2024-01-15T18:10:03.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.543+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c0ad480 con 0x7fc93c15c450 2024-01-15T18:10:03.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.544+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c0adc40 con 0x7fc93c15c450 2024-01-15T18:10:03.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.546+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fc9340147e0 con 0x7fc93c15c450 2024-01-15T18:10:03.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.546+0000 7fc939600640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc91406c040 0x7fc91406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:03.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.547+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7fc9340888c0 con 0x7fc93c15c450 2024-01-15T18:10:03.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.547+0000 7fc941800640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc91406c040 0x7fc91406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:03.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.548+0000 7fc941800640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc91406c040 0x7fc91406e500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0b41d0 tx=0x7fc92c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:03.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.552+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc934054d90 con 0x7fc93c15c450 2024-01-15T18:10:03.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.777+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone47167", "target": ["mon-mgr", ""]}) v1 -- 0x7fc93c06ce30 con 0x7fc91406c040 2024-01-15T18:10:03.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.820+0000 7fc939600640 1 -- 172.21.15.2:0/1846902411 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc92c00c040 con 0x7fc91406c040 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc91406c040 msgr2=0x7fc91406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc91406c040 0x7fc91406e500 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc93c0b41d0 tx=0x7fc92c008040 comp rx=0 tx=0).stop 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 msgr2=0x7fc93c0b0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 0x7fc93c0b0470 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7fc93400ba10 tx=0x7fc93400bed0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 shutdown_connections 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc91406c040 0x7fc91406e500 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc93c15c450 0x7fc93c0b0470 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc93c144c00 0x7fc93c0b2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 --2- 172.21.15.2:0/1846902411 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc93c0cdd70 0x7fc93c0ac870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:03.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.823+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 >> 172.21.15.2:0/1846902411 conn(0x7fc93c0b45f0 msgr2=0x7fc93c0cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:03.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.824+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 shutdown_connections 2024-01-15T18:10:03.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:03.824+0000 7fc944200640 1 -- 172.21.15.2:0/1846902411 wait complete. 2024-01-15T18:10:03.851 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone471619 2024-01-15T18:10:04.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.059+0000 7f1f82600640 1 Processor -- start 2024-01-15T18:10:04.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.060+0000 7f1f82600640 1 -- start start 2024-01-15T18:10:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.061+0000 7f1f82600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 0x7f1f7c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.061+0000 7f1f82600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c0d1250 con 0x7f1f7c0d1c50 2024-01-15T18:10:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.061+0000 7f1f82600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7c0ce8e0 con 0x7f1f7c0b2210 2024-01-15T18:10:04.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.061+0000 7f1f82600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0ce6d0 con 0x7f1f7c0d1870 2024-01-15T18:10:04.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.061+0000 7f1f81c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1f7c0d1c50 0x7f1f7c0b26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45036/0 (socket says 172.21.15.2:45036) 2024-01-15T18:10:04.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.061+0000 7f1f81c00640 1 -- 172.21.15.2:0/1370605745 learned_addr learned my addr 172.21.15.2:0/1370605745 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f81200640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 0x7f1f7c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010906532 0 0) 0x7f1f7c0d1250 con 0x7f1f7c0d1c50 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f5c003520 con 0x7f1f7c0d1c50 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310718156 0 0) 0x7f1f5c003520 con 0x7f1f7c0d1c50 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 msgr2=0x7f1f7c0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 0x7f1f7c0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 >> v1:172.21.15.137:6789/0 conn(0x7f1f7c0b2210 legacy=0x7f1f7c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c0cebc0 con 0x7f1f7c0d1c50 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1f74001670 con 0x7f1f7c0d1c50 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 >> v1:172.21.15.2:6789/0 conn(0x7f1f7c0d1c50 legacy=0x7f1f7c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f5c0039a0 0x7f1f5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f5c0047c0 0x7f1f5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 0x7f1f5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0ce6d0 con 0x7f1f5c0047c0 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c00ba30 con 0x7f1f5c0039a0 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.062+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5c00bbf0 con 0x7f1f7c0d1870 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f7be00640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f5c0047c0 0x7f1f5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f81200640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f5c0039a0 0x7f1f5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f81c00640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 0x7f1f5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f7be00640 1 -- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 msgr2=0x7f1f5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f7be00640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f7c0d1870 0x7f1f5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f7be00640 1 -- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f5c0039a0 msgr2=0x7f1f5c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:04.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f7be00640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f5c0039a0 0x7f1f5c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f7be00640 1 -- 172.21.15.2:0/1370605745 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f5c003520 con 0x7f1f5c0047c0 2024-01-15T18:10:04.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.063+0000 7f1f81200640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f5c0039a0 0x7f1f5c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.064+0000 7f1f7be00640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f5c0047c0 0x7f1f5c004c40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b2650 tx=0x7f1f5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657be65ebcb4a328 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f640052d0 con 0x7f1f5c0047c0 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6402e050 con 0x7f1f5c0047c0 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f7b400640 1 -- 172.21.15.2:0/1370605745 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f640035f0 con 0x7f1f5c0047c0 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f82600640 1 -- 172.21.15.2:0/1370605745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f5c0047c0 msgr2=0x7f1f5c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f82600640 1 --2- 172.21.15.2:0/1370605745 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f5c0047c0 0x7f1f5c004c40 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f1f7c0b2650 tx=0x7f1f5c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f81c00640 1 -- 172.21.15.2:0/1370605745 reap_dead start 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f82600640 1 -- 172.21.15.2:0/1370605745 shutdown_connections 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f82600640 1 -- 172.21.15.2:0/1370605745 >> 172.21.15.2:0/1370605745 conn(0x7f1f7c0d30b0 msgr2=0x7f1f7c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f82600640 1 -- 172.21.15.2:0/1370605745 shutdown_connections 2024-01-15T18:10:04.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.065+0000 7f1f82600640 1 -- 172.21.15.2:0/1370605745 wait complete. 2024-01-15T18:10:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.067+0000 7f1f82600640 1 Processor -- start 2024-01-15T18:10:04.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.067+0000 7f1f82600640 1 -- start start 2024-01-15T18:10:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.068+0000 7f1f82600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f5c0047c0 0x7f1f7c1518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.068+0000 7f1f82600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 0x7f1f7c0b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.068+0000 7f1f82600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7c0b6f80 0x7f1f7c0b7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.068+0000 7f1f82600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0b16d0 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.068+0000 7f1f82600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0b6dd0 con 0x7f1f7c0b6f80 2024-01-15T18:10:04.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.068+0000 7f1f82600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7c0b5d70 con 0x7f1f5c0047c0 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 0x7f1f7c0b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 0x7f1f7c0b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41596/0 (socket says 172.21.15.2:41596) 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 -- 172.21.15.2:0/614765705 learned_addr learned my addr 172.21.15.2:0/614765705 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 -- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f5c0047c0 msgr2=0x7f1f7c1518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f5c0047c0 0x7f1f7c1518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 -- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7c0b6f80 msgr2=0x7f1f7c0b7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7c0b6f80 0x7f1f7c0b7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 -- 172.21.15.2:0/614765705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7c0b6010 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.069+0000 7f1f7be00640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 0x7f1f7c0b1190 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f1f64008fd0 tx=0x7f1f64001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.070+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f640076c0 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.070+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f64007a30 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.070+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f64006be0 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.070+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7c0b62f0 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.070+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7c0b68b0 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.071+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f7c0b6a70 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.072+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f1f64006d40 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.073+0000 7f1f78c00640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f5406c040 0x7f1f5406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.073+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f1f640aa6c0 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.074+0000 7f1f81200640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f5406c040 0x7f1f5406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.075+0000 7f1f81200640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f5406c040 0x7f1f5406e500 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c007440 tx=0x7f1f6c009750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:04.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.077+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f64076b80 con 0x7f1f7c0b0d10 2024-01-15T18:10:04.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.301+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone471619", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f7c0b16d0 con 0x7f1f5406c040 2024-01-15T18:10:04.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.350+0000 7f1f78c00640 1 -- 172.21.15.2:0/614765705 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1f7c0b16d0 con 0x7f1f5406c040 2024-01-15T18:10:04.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f5406c040 msgr2=0x7f1f5406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f5406c040 0x7f1f5406e500 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c007440 tx=0x7f1f6c009750 comp rx=0 tx=0).stop 2024-01-15T18:10:04.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 msgr2=0x7f1f7c0b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 0x7f1f7c0b1190 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f1f64008fd0 tx=0x7f1f64001120 comp rx=0 tx=0).stop 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 shutdown_connections 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1f5406c040 0x7f1f5406e500 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1f7c0b6f80 0x7f1f7c0b7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1f7c0b0d10 0x7f1f7c0b1190 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 --2- 172.21.15.2:0/614765705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1f5c0047c0 0x7f1f7c1518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 >> 172.21.15.2:0/614765705 conn(0x7f1f7c0d30b0 msgr2=0x7f1f7c0d2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.353+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 shutdown_connections 2024-01-15T18:10:04.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.354+0000 7f1f82600640 1 -- 172.21.15.2:0/614765705 wait complete. 2024-01-15T18:10:04.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:04.380 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:10:04.448 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-01-15T18:10:04.449 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:10:04.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.655+0000 7f3f74200640 1 Processor -- start 2024-01-15T18:10:04.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.655+0000 7f3f74200640 1 -- start start 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.657+0000 7f3f74200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3f6c146ba0 0x7f3f6c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.657+0000 7f3f74200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c15e0b0 0x7f3f6c0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.657+0000 7f3f74200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f6c06f440 con 0x7f3f6c0b2ba0 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.657+0000 7f3f74200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6c146f80 con 0x7f3f6c146ba0 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.657+0000 7f3f74200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0b3520 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.657+0000 7f3f71800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3f6c146ba0 0x7f3f6c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.658+0000 7f3f71800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3f6c146ba0 0x7f3f6c0d1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41602/0 (socket says 172.21.15.2:41602) 2024-01-15T18:10:04.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.658+0000 7f3f71800640 1 -- 172.21.15.2:0/2456456654 learned_addr learned my addr 172.21.15.2:0/2456456654 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:04.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.658+0000 7f3f71800640 1 -- 172.21.15.2:0/2456456654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c15e0b0 msgr2=0x7f3f6c0b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:04.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.658+0000 7f3f71800640 1 --2- 172.21.15.2:0/2456456654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c15e0b0 0x7f3f6c0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.658+0000 7f3f71800640 1 -- 172.21.15.2:0/2456456654 >> v1:172.21.15.137:6789/0 conn(0x7f3f6c0b2ba0 legacy=0x7f3f6c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:04.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.658+0000 7f3f71800640 1 -- 172.21.15.2:0/2456456654 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c15e5f0 con 0x7f3f6c146ba0 2024-01-15T18:10:04.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.659+0000 7f3f71800640 1 --2- 172.21.15.2:0/2456456654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c146ba0 0x7f3f6c0d1a60 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009870 tx=0x7f3f5402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ff5885da1ce579 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:04.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.659+0000 7f3f6be00640 1 -- 172.21.15.2:0/2456456654 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f540317c0 con 0x7f3f6c146ba0 2024-01-15T18:10:04.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.659+0000 7f3f6be00640 1 -- 172.21.15.2:0/2456456654 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54031920 con 0x7f3f6c146ba0 2024-01-15T18:10:04.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.659+0000 7f3f6be00640 1 -- 172.21.15.2:0/2456456654 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54031cb0 con 0x7f3f6c146ba0 2024-01-15T18:10:04.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 -- 172.21.15.2:0/2456456654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c146ba0 msgr2=0x7f3f6c0d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 --2- 172.21.15.2:0/2456456654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c146ba0 0x7f3f6c0d1a60 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009870 tx=0x7f3f5402b860 comp rx=0 tx=0).stop 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 -- 172.21.15.2:0/2456456654 shutdown_connections 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 --2- 172.21.15.2:0/2456456654 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c15e0b0 0x7f3f6c0b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 --2- 172.21.15.2:0/2456456654 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c146ba0 0x7f3f6c0d1a60 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 -- 172.21.15.2:0/2456456654 >> 172.21.15.2:0/2456456654 conn(0x7f3f6c0b6430 msgr2=0x7f3f6c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 -- 172.21.15.2:0/2456456654 shutdown_connections 2024-01-15T18:10:04.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.660+0000 7f3f74200640 1 -- 172.21.15.2:0/2456456654 wait complete. 2024-01-15T18:10:04.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.662+0000 7f3f74200640 1 Processor -- start 2024-01-15T18:10:04.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.662+0000 7f3f74200640 1 -- start start 2024-01-15T18:10:04.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f74200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0b2ba0 0x7f3f6c0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f74200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c146ba0 0x7f3f6c13cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f74200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 0x7f3f6c0cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f74200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cf750 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f74200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cf960 con 0x7f3f6c0b2ba0 2024-01-15T18:10:04.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f74200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0cfc40 con 0x7f3f6c146ba0 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.663+0000 7f3f72200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 0x7f3f6c0cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 0x7f3f6c0cf210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41612/0 (socket says 172.21.15.2:41612) 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 -- 172.21.15.2:0/1666488359 learned_addr learned my addr 172.21.15.2:0/1666488359 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f71800640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0b2ba0 0x7f3f6c0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f70e00640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c146ba0 0x7f3f6c13cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 -- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c146ba0 msgr2=0x7f3f6c13cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c146ba0 0x7f3f6c13cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 -- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0b2ba0 msgr2=0x7f3f6c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0b2ba0 0x7f3f6c0ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 -- 172.21.15.2:0/1666488359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f540094d0 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f70e00640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c146ba0 0x7f3f6c13cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:04.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.664+0000 7f3f72200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 0x7f3f6c0cf210 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400b410 tx=0x7f3f6400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:04.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.665+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64011830 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.665+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f64011990 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.665+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f64011d40 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.665+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c0cff20 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.665+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c0d04e0 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.666+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c0686f0 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.667+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f3f6401a400 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.668+0000 7f3f69600640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3f4406c040 0x7f3f4406e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:04.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.668+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5879+0+0 (secure 0 0 0) 0x7f3f640889f0 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.669+0000 7f3f71800640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3f4406c040 0x7f3f4406e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:04.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.670+0000 7f3f71800640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3f4406c040 0x7f3f4406e500 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3f54036000 tx=0x7f3f5403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:04.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.672+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3f64055000 con 0x7f3f6c15e0b0 2024-01-15T18:10:04.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:04.894+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f6c0d1260 con 0x7f3f4406c040 2024-01-15T18:10:05.027 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T18:10:05.021+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:10:05.126 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:10:06.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:06.796+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f3f64088590 con 0x7f3f6c15e0b0 2024-01-15T18:10:08.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.073+0000 7f3f69600640 1 -- 172.21.15.2:0/1666488359 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f3f54031b90 con 0x7f3f4406c040 2024-01-15T18:10:08.077 INFO:teuthology.orchestra.run.smithi002.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.078+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3f4406c040 msgr2=0x7f3f4406e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.078+0000 7f3f74200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3f4406c040 0x7f3f4406e500 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3f54036000 tx=0x7f3f5403d040 comp rx=0 tx=0).stop 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.078+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 msgr2=0x7f3f6c0cf210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.078+0000 7f3f74200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 0x7f3f6c0cf210 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400b410 tx=0x7f3f6400b810 comp rx=0 tx=0).stop 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.079+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 shutdown_connections 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.079+0000 7f3f74200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3f4406c040 0x7f3f4406e500 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.079+0000 7f3f74200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3f6c15e0b0 0x7f3f6c0cf210 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.079+0000 7f3f74200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3f6c146ba0 0x7f3f6c13cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.079+0000 7f3f74200640 1 --2- 172.21.15.2:0/1666488359 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3f6c0b2ba0 0x7f3f6c0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.079+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 >> 172.21.15.2:0/1666488359 conn(0x7f3f6c0b6430 msgr2=0x7f3f6c0b46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:08.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.080+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 shutdown_connections 2024-01-15T18:10:08.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.081+0000 7f3f74200640 1 -- 172.21.15.2:0/1666488359 wait complete. 2024-01-15T18:10:08.115 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:10:08.115 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:10:08.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:10:08.316 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:10:08.317 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:10:08.317 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:08.390 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:08.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.646+0000 7fbaa8a00640 1 Processor -- start 2024-01-15T18:10:08.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.646+0000 7fbaa8a00640 1 -- start start 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa8a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fbaa406e1e0 0x7fbaa40b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa8a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40ccbe0 con 0x7fbaa4143810 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa8a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbaa40b3520 con 0x7fbaa406e1e0 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa4143bf0 con 0x7fbaa40b2080 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa2800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fbaa4143810 0x7fbaa406dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45046/0 (socket says 172.21.15.2:45046) 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa2800640 1 -- 172.21.15.2:0/2408487026 learned_addr learned my addr 172.21.15.2:0/2408487026 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:08.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fba9be00640 1 --2- 172.21.15.2:0/2408487026 >> v2:172.21.15.2:3300/0 conn(0x7fbaa406e1e0 0x7fbaa40b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa3200640 1 --2- 172.21.15.2:0/2408487026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fba9be00640 1 -- 172.21.15.2:0/2408487026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 msgr2=0x7fbaa40cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:08.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fba9be00640 1 --2- 172.21.15.2:0/2408487026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fba9be00640 1 -- 172.21.15.2:0/2408487026 >> v1:172.21.15.2:6789/0 conn(0x7fbaa4143810 legacy=0x7fbaa406dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.649+0000 7fba9be00640 1 -- 172.21.15.2:0/2408487026 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa40b2ac0 con 0x7fbaa406e1e0 2024-01-15T18:10:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.648+0000 7fbaa1e00640 1 -- 172.21.15.2:0/2408487026 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718385320 0 0) 0x7fbaa40ccbe0 con 0x7fbaa4143810 2024-01-15T18:10:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.650+0000 7fba9be00640 1 --2- 172.21.15.2:0/2408487026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa40b1b40 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0097a0 tx=0x7fba8c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec4c77bb6f68f78 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.650+0000 7fbaa1e00640 1 -- 172.21.15.2:0/2408487026 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c031780 con 0x7fbaa406e1e0 2024-01-15T18:10:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.650+0000 7fbaa1e00640 1 -- 172.21.15.2:0/2408487026 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8c0318e0 con 0x7fbaa406e1e0 2024-01-15T18:10:08.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.650+0000 7fbaa1e00640 1 -- 172.21.15.2:0/2408487026 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8c031c70 con 0x7fbaa406e1e0 2024-01-15T18:10:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.650+0000 7fbaa8a00640 1 -- 172.21.15.2:0/2408487026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 msgr2=0x7fbaa40b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.650+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/2408487026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa40b1b40 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7fba8c0097a0 tx=0x7fba8c02b820 comp rx=0 tx=0).stop 2024-01-15T18:10:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.651+0000 7fbaa8a00640 1 -- 172.21.15.2:0/2408487026 shutdown_connections 2024-01-15T18:10:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.651+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/2408487026 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.651+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/2408487026 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa40b1b40 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.651+0000 7fbaa8a00640 1 -- 172.21.15.2:0/2408487026 >> 172.21.15.2:0/2408487026 conn(0x7fbaa40b0840 msgr2=0x7fbaa40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.651+0000 7fbaa8a00640 1 -- 172.21.15.2:0/2408487026 shutdown_connections 2024-01-15T18:10:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.651+0000 7fbaa8a00640 1 -- 172.21.15.2:0/2408487026 wait complete. 2024-01-15T18:10:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.652+0000 7fbaa8a00640 1 Processor -- start 2024-01-15T18:10:08.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 -- start start 2024-01-15T18:10:08.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa415c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa4143810 0x7fbaa40ad6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa41f2690 con 0x7fbaa406e1e0 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40a9450 con 0x7fbaa4143810 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.653+0000 7fbaa8a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa40a8e90 con 0x7fbaa40b2080 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa415c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa415c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41648/0 (socket says 172.21.15.2:41648) 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 -- 172.21.15.2:0/560205655 learned_addr learned my addr 172.21.15.2:0/560205655 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fba9be00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa3200640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa4143810 0x7fbaa40ad6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 -- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 msgr2=0x7fbaa40a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 -- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa4143810 msgr2=0x7fbaa40ad6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa4143810 0x7fbaa40ad6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fbaa2800640 1 -- 172.21.15.2:0/560205655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c009400 con 0x7fbaa406e1e0 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.654+0000 7fba9be00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.655+0000 7fbaa2800640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa415c230 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fba9400c430 tx=0x7fba9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.655+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba940074e0 con 0x7fbaa406e1e0 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.655+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba94007660 con 0x7fbaa406e1e0 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.655+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba94004a20 con 0x7fbaa406e1e0 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.655+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa40a9170 con 0x7fbaa406e1e0 2024-01-15T18:10:08.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.655+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa40c82d0 con 0x7fbaa406e1e0 2024-01-15T18:10:08.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.657+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa4074770 con 0x7fbaa406e1e0 2024-01-15T18:10:08.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.657+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fba94004b80 con 0x7fbaa406e1e0 2024-01-15T18:10:08.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.658+0000 7fba9aa00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba7c06bfb0 0x7fba7c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:08.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.658+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fba94088480 con 0x7fbaa406e1e0 2024-01-15T18:10:08.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.659+0000 7fba9be00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba7c06bfb0 0x7fba7c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:08.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.660+0000 7fba9be00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba7c06bfb0 0x7fba7c06e470 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fba8c036000 tx=0x7fba8c03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:08.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.663+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba94054d40 con 0x7fbaa406e1e0 2024-01-15T18:10:08.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.860+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa40b3a80 con 0x7fbaa406e1e0 2024-01-15T18:10:08.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.863+0000 7fba9aa00640 1 -- 172.21.15.2:0/560205655 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13056 (secure 0 0 0) 0x7fba94058360 con 0x7fbaa406e1e0 2024-01-15T18:10:08.866 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:08.866 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":80,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:08.043678+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/4035045426":"2024-01-16T18:10:05.021990+0000","172.21.15.2:6834/4035045426":"2024-01-16T18:10:05.021990+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:08.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba7c06bfb0 msgr2=0x7fba7c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba7c06bfb0 0x7fba7c06e470 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fba8c036000 tx=0x7fba8c03d040 comp rx=0 tx=0).stop 2024-01-15T18:10:08.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 msgr2=0x7fbaa415c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:08.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa415c230 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fba9400c430 tx=0x7fba9400c8f0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 shutdown_connections 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fba7c06bfb0 0x7fba7c06e470 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fbaa4143810 0x7fbaa40ad6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fbaa40b2080 0x7fbaa40a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.865+0000 7fbaa8a00640 1 --2- 172.21.15.2:0/560205655 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fbaa406e1e0 0x7fbaa415c230 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.866+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 >> 172.21.15.2:0/560205655 conn(0x7fbaa40b0840 msgr2=0x7fbaa40aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.866+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 shutdown_connections 2024-01-15T18:10:08.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:08.866+0000 7fbaa8a00640 1 -- 172.21.15.2:0/560205655 wait complete. 2024-01-15T18:10:08.888 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:10:08.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:10:09.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.095+0000 7fd32f200640 1 Processor -- start 2024-01-15T18:10:09.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.095+0000 7fd32f200640 1 -- start start 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd330143620 0x7fd3300b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd3300b2a90 0x7fd330071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd330071770 0x7fd330070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32f200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd330071c20 con 0x7fd3300b2a90 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3300b2fa0 con 0x7fd330071770 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300b3280 con 0x7fd330143620 2024-01-15T18:10:09.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd327e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd3300b2a90 0x7fd330071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd327e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd3300b2a90 0x7fd330071230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41660/0 (socket says 172.21.15.2:41660) 2024-01-15T18:10:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd327e00640 1 -- 172.21.15.2:0/2748393442 learned_addr learned my addr 172.21.15.2:0/2748393442 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32e800640 1 --2- 172.21.15.2:0/2748393442 >> v2:172.21.15.137:3300/0 conn(0x7fd330071770 0x7fd330070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.097+0000 7fd32de00640 1 --2- 172.21.15.2:0/2748393442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd330143620 0x7fd3300b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.098+0000 7fd327e00640 1 -- 172.21.15.2:0/2748393442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd330143620 msgr2=0x7fd3300b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.098+0000 7fd327e00640 1 --2- 172.21.15.2:0/2748393442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd330143620 0x7fd3300b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.098+0000 7fd327e00640 1 -- 172.21.15.2:0/2748393442 >> v2:172.21.15.137:3300/0 conn(0x7fd330071770 msgr2=0x7fd330070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.098+0000 7fd327e00640 1 --2- 172.21.15.2:0/2748393442 >> v2:172.21.15.137:3300/0 conn(0x7fd330071770 0x7fd330070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.098+0000 7fd327e00640 1 -- 172.21.15.2:0/2748393442 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300b3560 con 0x7fd3300b2a90 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.098+0000 7fd32e800640 1 --2- 172.21.15.2:0/2748393442 >> v2:172.21.15.137:3300/0 conn(0x7fd330071770 0x7fd330070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:09.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.099+0000 7fd327e00640 1 --2- 172.21.15.2:0/2748393442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3300b2a90 0x7fd330071230 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd33015d990 tx=0x7fd31802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c274c594f6dfb406 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.099+0000 7fd32d400640 1 -- 172.21.15.2:0/2748393442 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318031870 con 0x7fd3300b2a90 2024-01-15T18:10:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.099+0000 7fd32d400640 1 -- 172.21.15.2:0/2748393442 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3180319d0 con 0x7fd3300b2a90 2024-01-15T18:10:09.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.099+0000 7fd32d400640 1 -- 172.21.15.2:0/2748393442 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318031d60 con 0x7fd3300b2a90 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 -- 172.21.15.2:0/2748393442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3300b2a90 msgr2=0x7fd330071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 --2- 172.21.15.2:0/2748393442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3300b2a90 0x7fd330071230 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fd33015d990 tx=0x7fd31802b910 comp rx=0 tx=0).stop 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 -- 172.21.15.2:0/2748393442 shutdown_connections 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 --2- 172.21.15.2:0/2748393442 >> v2:172.21.15.137:3300/0 conn(0x7fd330071770 0x7fd330070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 --2- 172.21.15.2:0/2748393442 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3300b2a90 0x7fd330071230 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 --2- 172.21.15.2:0/2748393442 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd330143620 0x7fd3300b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 -- 172.21.15.2:0/2748393442 >> 172.21.15.2:0/2748393442 conn(0x7fd3300b0840 msgr2=0x7fd3300b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 -- 172.21.15.2:0/2748393442 shutdown_connections 2024-01-15T18:10:09.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.100+0000 7fd32f200640 1 -- 172.21.15.2:0/2748393442 wait complete. 2024-01-15T18:10:09.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.102+0000 7fd32f200640 1 Processor -- start 2024-01-15T18:10:09.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.102+0000 7fd32f200640 1 -- start start 2024-01-15T18:10:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.103+0000 7fd32f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd330071770 0x7fd330138d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.103+0000 7fd32f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3300b2a90 0x7fd3301f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.103+0000 7fd32f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 0x7fd33014be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.103+0000 7fd32f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33013f820 con 0x7fd330143620 2024-01-15T18:10:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.103+0000 7fd32f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330139280 con 0x7fd330071770 2024-01-15T18:10:09.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.103+0000 7fd32f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33013f460 con 0x7fd3300b2a90 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 0x7fd33014be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 0x7fd33014be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41670/0 (socket says 172.21.15.2:41670) 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 -- 172.21.15.2:0/1474660437 learned_addr learned my addr 172.21.15.2:0/1474660437 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd327e00640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3300b2a90 0x7fd3301f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32de00640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd330071770 0x7fd330138d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 -- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3300b2a90 msgr2=0x7fd3301f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3300b2a90 0x7fd3301f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 -- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd330071770 msgr2=0x7fd330138d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd330071770 0x7fd330138d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.104+0000 7fd32e800640 1 -- 172.21.15.2:0/1474660437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3180094d0 con 0x7fd330143620 2024-01-15T18:10:09.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.105+0000 7fd32e800640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 0x7fd33014be10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd32800cef0 tx=0x7fd32800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:09.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.105+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32800d840 con 0x7fd330143620 2024-01-15T18:10:09.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.105+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32800d9c0 con 0x7fd330143620 2024-01-15T18:10:09.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.105+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33013eaf0 con 0x7fd330143620 2024-01-15T18:10:09.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.105+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32800dd70 con 0x7fd330143620 2024-01-15T18:10:09.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.105+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3301f2b00 con 0x7fd330143620 2024-01-15T18:10:09.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.106+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd33013f960 con 0x7fd330143620 2024-01-15T18:10:09.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.107+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fd32801e3b0 con 0x7fd330143620 2024-01-15T18:10:09.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.108+0000 7fd326000640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806bfb0 0x7fd30806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.108+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fd328089090 con 0x7fd330143620 2024-01-15T18:10:09.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.109+0000 7fd32de00640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806bfb0 0x7fd30806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.110+0000 7fd32de00640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806bfb0 0x7fd30806e470 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd320006fd0 tx=0x7fd320008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:09.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.113+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd33013f960 con 0x7fd330143620 2024-01-15T18:10:09.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.305+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd33013f960 con 0x7fd330143620 2024-01-15T18:10:09.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.305+0000 7fd326000640 1 -- 172.21.15.2:0/1474660437 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd33013f960 con 0x7fd330143620 2024-01-15T18:10:09.309 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:10:09.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806bfb0 msgr2=0x7fd30806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806bfb0 0x7fd30806e470 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd320006fd0 tx=0x7fd320008040 comp rx=0 tx=0).stop 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 msgr2=0x7fd33014be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 0x7fd33014be10 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fd32800cef0 tx=0x7fd32800a710 comp rx=0 tx=0).stop 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 shutdown_connections 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806bfb0 0x7fd30806e470 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd330143620 0x7fd33014be10 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3300b2a90 0x7fd3301f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 --2- 172.21.15.2:0/1474660437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd330071770 0x7fd330138d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 >> 172.21.15.2:0/1474660437 conn(0x7fd3300b0840 msgr2=0x7fd3300ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 shutdown_connections 2024-01-15T18:10:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.308+0000 7fd32f200640 1 -- 172.21.15.2:0/1474660437 wait complete. 2024-01-15T18:10:09.331 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:10:09.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.555+0000 7f62da200640 1 Processor -- start 2024-01-15T18:10:09.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.556+0000 7f62da200640 1 -- start start 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62da200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f62d4160e50 0x7f62d40cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62da200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f62d40ce3b0 0x7f62d415d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62da200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d40cb3a0 con 0x7f62d406d350 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62da200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d40cb680 con 0x7f62d4160e50 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62da200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62d40cb960 con 0x7f62d40ce3b0 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62d8e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f62d406d350 0x7f62d406d730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45062/0 (socket says 172.21.15.2:45062) 2024-01-15T18:10:09.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62d8e00640 1 -- 172.21.15.2:0/3649258301 learned_addr learned my addr 172.21.15.2:0/3649258301 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.557+0000 7f62cbc00640 1 --2- 172.21.15.2:0/3649258301 >> v2:172.21.15.2:3300/0 conn(0x7f62d4160e50 0x7f62d40cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62d9800640 1 --2- 172.21.15.2:0/3649258301 >> v2:172.21.15.137:3300/0 conn(0x7f62d40ce3b0 0x7f62d415d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62d3e00640 1 -- 172.21.15.2:0/3649258301 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241915396 0 0) 0x7f62d40cb3a0 con 0x7f62d406d350 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62d3e00640 1 -- 172.21.15.2:0/3649258301 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b4003520 con 0x7f62d406d350 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62cbc00640 1 -- 172.21.15.2:0/3649258301 >> v2:172.21.15.137:3300/0 conn(0x7f62d40ce3b0 msgr2=0x7f62d415d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62cbc00640 1 --2- 172.21.15.2:0/3649258301 >> v2:172.21.15.137:3300/0 conn(0x7f62d40ce3b0 0x7f62d415d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62cbc00640 1 -- 172.21.15.2:0/3649258301 >> v1:172.21.15.2:6789/0 conn(0x7f62d406d350 legacy=0x7f62d406d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.558+0000 7f62cbc00640 1 -- 172.21.15.2:0/3649258301 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d40cbc40 con 0x7f62d4160e50 2024-01-15T18:10:09.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.559+0000 7f62cbc00640 1 --2- 172.21.15.2:0/3649258301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40cde70 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f62d4070060 tx=0x7f62bc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffaca415bfe288c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:09.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.559+0000 7f62d3e00640 1 -- 172.21.15.2:0/3649258301 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc031830 con 0x7f62d4160e50 2024-01-15T18:10:09.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.559+0000 7f62d3e00640 1 -- 172.21.15.2:0/3649258301 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc031990 con 0x7f62d4160e50 2024-01-15T18:10:09.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.559+0000 7f62d3e00640 1 -- 172.21.15.2:0/3649258301 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc031d00 con 0x7f62d4160e50 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 -- 172.21.15.2:0/3649258301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 msgr2=0x7f62d40cde70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 --2- 172.21.15.2:0/3649258301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40cde70 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f62d4070060 tx=0x7f62bc02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 -- 172.21.15.2:0/3649258301 shutdown_connections 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 --2- 172.21.15.2:0/3649258301 >> v2:172.21.15.137:3300/0 conn(0x7f62d40ce3b0 0x7f62d415d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 --2- 172.21.15.2:0/3649258301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40cde70 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 -- 172.21.15.2:0/3649258301 >> 172.21.15.2:0/3649258301 conn(0x7f62d40b0840 msgr2=0x7f62d40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 -- 172.21.15.2:0/3649258301 shutdown_connections 2024-01-15T18:10:09.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.560+0000 7f62da200640 1 -- 172.21.15.2:0/3649258301 wait complete. 2024-01-15T18:10:09.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.562+0000 7f62da200640 1 Processor -- start 2024-01-15T18:10:09.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.562+0000 7f62da200640 1 -- start start 2024-01-15T18:10:09.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.563+0000 7f62da200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d406d350 0x7f62d413cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.563+0000 7f62da200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62d40ce3b0 0x7f62d40b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.563+0000 7f62da200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.563+0000 7f62da200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d41530a0 con 0x7f62d4160e50 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.563+0000 7f62da200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d413f770 con 0x7f62d406d350 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.563+0000 7f62da200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d413fa50 con 0x7f62d40ce3b0 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41702/0 (socket says 172.21.15.2:41702) 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 -- 172.21.15.2:0/547956167 learned_addr learned my addr 172.21.15.2:0/547956167 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d8e00640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d406d350 0x7f62d413cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62cbc00640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62d40ce3b0 0x7f62d40b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 -- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62d40ce3b0 msgr2=0x7f62d40b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62d40ce3b0 0x7f62d40b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 -- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d406d350 msgr2=0x7f62d413cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d406d350 0x7f62d413cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 -- 172.21.15.2:0/547956167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc009400 con 0x7f62d4160e50 2024-01-15T18:10:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62cbc00640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62d40ce3b0 0x7f62d40b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:09.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.564+0000 7f62d9800640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40b2720 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f62cc007e40 tx=0x7f62cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:09.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.565+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc014620 con 0x7f62d4160e50 2024-01-15T18:10:09.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.565+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62cc00ee90 con 0x7f62d4160e50 2024-01-15T18:10:09.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.565+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc0149e0 con 0x7f62d4160e50 2024-01-15T18:10:09.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.565+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d413fd30 con 0x7f62d4160e50 2024-01-15T18:10:09.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.565+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d4140530 con 0x7f62d4160e50 2024-01-15T18:10:09.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.566+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d4067720 con 0x7f62d4160e50 2024-01-15T18:10:09.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.567+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f62cc014b40 con 0x7f62d4160e50 2024-01-15T18:10:09.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.568+0000 7f62d1600640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62ac06bfb0 0x7f62ac06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:09.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.568+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f62cc089030 con 0x7f62d4160e50 2024-01-15T18:10:09.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.569+0000 7f62d8e00640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62ac06bfb0 0x7f62ac06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:09.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.570+0000 7f62d8e00640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62ac06bfb0 0x7f62ac06e470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f62c4005880 tx=0x7f62c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:09.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.572+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62cc0558f0 con 0x7f62d4160e50 2024-01-15T18:10:09.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.780+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f62d406c760 con 0x7f62d4160e50 2024-01-15T18:10:09.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.781+0000 7f62d1600640 1 -- 172.21.15.2:0/547956167 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) v1 ==== 80+0+120 (secure 0 0 0) 0x7f62cc058f10 con 0x7f62d4160e50 2024-01-15T18:10:09.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6835/4035045426 2024-01-16T18:10:05.021990+0000 2024-01-15T18:10:09.785 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/4035045426 2024-01-16T18:10:05.021990+0000 2024-01-15T18:10:09.785 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:10:09.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62ac06bfb0 msgr2=0x7f62ac06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62ac06bfb0 0x7f62ac06e470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f62c4005880 tx=0x7f62c4009040 comp rx=0 tx=0).stop 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 msgr2=0x7f62d40b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40b2720 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f62cc007e40 tx=0x7f62cc00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 shutdown_connections 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f62ac06bfb0 0x7f62ac06e470 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f62d4160e50 0x7f62d40b2720 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f62d40ce3b0 0x7f62d40b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 --2- 172.21.15.2:0/547956167 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62d406d350 0x7f62d413cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 >> 172.21.15.2:0/547956167 conn(0x7f62d40b0840 msgr2=0x7f62d40af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 shutdown_connections 2024-01-15T18:10:09.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:09.784+0000 7f62da200640 1 -- 172.21.15.2:0/547956167 wait complete. 2024-01-15T18:10:09.807 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:10:09.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:09.808 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:09.905 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[61789]: fuse finished with error 0 and tester_r 0 2024-01-15T18:10:09.911 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:10:09.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:09.911 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:10:10.135 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:10:10.135 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:10.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:10.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.416+0000 7f844e200640 1 Processor -- start 2024-01-15T18:10:10.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.417+0000 7f844e200640 1 -- start start 2024-01-15T18:10:10.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.418+0000 7f844e200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8448146dd0 0x7f844806c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.418+0000 7f844e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f844806c9b0 0x7f84480b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.418+0000 7f844e200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84480ac870 con 0x7f84481469f0 2024-01-15T18:10:10.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.418+0000 7f844e200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f844806ce20 con 0x7f8448146dd0 2024-01-15T18:10:10.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.418+0000 7f844e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844815d970 con 0x7f844806c9b0 2024-01-15T18:10:10.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8448146dd0 0x7f844806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8448146dd0 0x7f844806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41706/0 (socket says 172.21.15.2:41706) 2024-01-15T18:10:10.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 -- 172.21.15.2:0/3693848229 learned_addr learned my addr 172.21.15.2:0/3693848229 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:10.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8447e00640 1 --2- 172.21.15.2:0/3693848229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f844806c9b0 0x7f84480b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 -- 172.21.15.2:0/3693848229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f844806c9b0 msgr2=0x7f84480b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:10.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 --2- 172.21.15.2:0/3693848229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f844806c9b0 0x7f84480b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 -- 172.21.15.2:0/3693848229 >> v1:172.21.15.137:6789/0 conn(0x7f84481469f0 legacy=0x7f8448071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.419+0000 7f8446a00640 1 -- 172.21.15.2:0/3693848229 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844815dc30 con 0x7f8448146dd0 2024-01-15T18:10:10.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.420+0000 7f8446a00640 1 --2- 172.21.15.2:0/3693848229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8448146dd0 0x7f844806c470 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f8430009e50 tx=0x7f843002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e18faa35bb941c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:10.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f8446000640 1 -- 172.21.15.2:0/3693848229 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843002cb00 con 0x7f8448146dd0 2024-01-15T18:10:10.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f8446000640 1 -- 172.21.15.2:0/3693848229 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843002cc60 con 0x7f8448146dd0 2024-01-15T18:10:10.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f8446000640 1 -- 172.21.15.2:0/3693848229 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84300343f0 con 0x7f8448146dd0 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 -- 172.21.15.2:0/3693848229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8448146dd0 msgr2=0x7f844806c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 --2- 172.21.15.2:0/3693848229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8448146dd0 0x7f844806c470 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f8430009e50 tx=0x7f843002bf60 comp rx=0 tx=0).stop 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 -- 172.21.15.2:0/3693848229 shutdown_connections 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 --2- 172.21.15.2:0/3693848229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f844806c9b0 0x7f84480b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 --2- 172.21.15.2:0/3693848229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8448146dd0 0x7f844806c470 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 -- 172.21.15.2:0/3693848229 >> 172.21.15.2:0/3693848229 conn(0x7f844806da20 msgr2=0x7f844806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 -- 172.21.15.2:0/3693848229 shutdown_connections 2024-01-15T18:10:10.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.421+0000 7f844e200640 1 -- 172.21.15.2:0/3693848229 wait complete. 2024-01-15T18:10:10.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.423+0000 7f844e200640 1 Processor -- start 2024-01-15T18:10:10.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.423+0000 7f844e200640 1 -- start start 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.424+0000 7f844e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 0x7f844815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.424+0000 7f844e200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84481469f0 0x7f844814ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.424+0000 7f844e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84481402d0 0x7f844814ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.424+0000 7f844e200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f844813f550 con 0x7f844806c9b0 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.424+0000 7f844e200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8448138620 con 0x7f84481469f0 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.424+0000 7f844e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f844813f7f0 con 0x7f84481402d0 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 0x7f844815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 0x7f844815c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41710/0 (socket says 172.21.15.2:41710) 2024-01-15T18:10:10.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 -- 172.21.15.2:0/4131459235 learned_addr learned my addr 172.21.15.2:0/4131459235 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:10.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8446a00640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84481469f0 0x7f844814ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447e00640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84481402d0 0x7f844814ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 -- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84481402d0 msgr2=0x7f844814ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84481402d0 0x7f844814ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 -- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84481469f0 msgr2=0x7f844814ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84481469f0 0x7f844814ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 -- 172.21.15.2:0/4131459235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8430009b30 con 0x7f844806c9b0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.425+0000 7f8447400640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 0x7f844815c8a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f843800c450 tx=0x7f843800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.426+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84380052d0 con 0x7f844806c9b0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.426+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8438005430 con 0x7f844806c9b0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.426+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f844813fad0 con 0x7f844806c9b0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.426+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438012870 con 0x7f844806c9b0 2024-01-15T18:10:10.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.426+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844815cde0 con 0x7f844806c9b0 2024-01-15T18:10:10.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.427+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84480686f0 con 0x7f844806c9b0 2024-01-15T18:10:10.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.428+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f84380129d0 con 0x7f844806c9b0 2024-01-15T18:10:10.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.429+0000 7f843f400640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f842006bfb0 0x7f842006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.429+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f8438088150 con 0x7f844806c9b0 2024-01-15T18:10:10.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.430+0000 7f8446a00640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f842006bfb0 0x7f842006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.431+0000 7f8446a00640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f842006bfb0 0x7f842006e470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8430035000 tx=0x7f8430030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:10.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.433+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8438054a10 con 0x7f844806c9b0 2024-01-15T18:10:10.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.630+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f844815d390 con 0x7f844806c9b0 2024-01-15T18:10:10.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.632+0000 7f843f400640 1 -- 172.21.15.2:0/4131459235 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13056 (secure 0 0 0) 0x7f8438058030 con 0x7f844806c9b0 2024-01-15T18:10:10.636 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:10.637 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":80,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:08.043678+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/4035045426":"2024-01-16T18:10:05.021990+0000","172.21.15.2:6834/4035045426":"2024-01-16T18:10:05.021990+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:10.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f842006bfb0 msgr2=0x7f842006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f842006bfb0 0x7f842006e470 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f8430035000 tx=0x7f8430030000 comp rx=0 tx=0).stop 2024-01-15T18:10:10.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 msgr2=0x7f844815c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 0x7f844815c8a0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f843800c450 tx=0x7f843800c910 comp rx=0 tx=0).stop 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 shutdown_connections 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f842006bfb0 0x7f842006e470 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f84481402d0 0x7f844814ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f84481469f0 0x7f844814ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 --2- 172.21.15.2:0/4131459235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f844806c9b0 0x7f844815c8a0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 >> 172.21.15.2:0/4131459235 conn(0x7f844806da20 msgr2=0x7f84480ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 shutdown_connections 2024-01-15T18:10:10.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.635+0000 7f844e200640 1 -- 172.21.15.2:0/4131459235 wait complete. 2024-01-15T18:10:10.660 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:10:10.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:10:10.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.880+0000 7fde55200640 1 Processor -- start 2024-01-15T18:10:10.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.881+0000 7fde55200640 1 -- start start 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.882+0000 7fde55200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d3090 0x7fde500d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.882+0000 7fde55200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fde500d1680 0x7fde500d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.882+0000 7fde55200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde500d3a30 con 0x7fde5015d340 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.882+0000 7fde55200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde500ce430 con 0x7fde500d1680 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.882+0000 7fde55200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde500b3de0 con 0x7fde500d3090 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.883+0000 7fde4fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fde500d1680 0x7fde500d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.883+0000 7fde4ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d3090 0x7fde500d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.883+0000 7fde4fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fde500d1680 0x7fde500d1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:45960/0 (socket says 172.21.15.2:45960) 2024-01-15T18:10:10.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.883+0000 7fde4fe00640 1 -- 172.21.15.2:0/2253502828 learned_addr learned my addr 172.21.15.2:0/2253502828 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:10.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.883+0000 7fde4f400640 1 --1- 172.21.15.2:0/2253502828 >> v1:172.21.15.137:6789/0 conn(0x7fde5015d340 0x7fde500cdd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:43314/0 (socket says 172.21.15.2:43314) 2024-01-15T18:10:10.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.884+0000 7fde4fe00640 1 -- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d3090 msgr2=0x7fde500d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:10.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.884+0000 7fde4fe00640 1 --2- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d3090 0x7fde500d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.884+0000 7fde4fe00640 1 -- 172.21.15.2:0/2253502828 >> v1:172.21.15.137:6789/0 conn(0x7fde5015d340 legacy=0x7fde500cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.884+0000 7fde4fe00640 1 -- 172.21.15.2:0/2253502828 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde500b40c0 con 0x7fde500d1680 2024-01-15T18:10:10.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.884+0000 7fde4ea00640 1 --2- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d3090 0x7fde500d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:10.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.886+0000 7fde4fe00640 1 --2- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde500d1680 0x7fde500d1b10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fde48009e50 tx=0x7fde4802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9802d31965862b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:10.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.886+0000 7fde4e000640 1 -- 172.21.15.2:0/2253502828 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde4802caf0 con 0x7fde500d1680 2024-01-15T18:10:10.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.887+0000 7fde4e000640 1 -- 172.21.15.2:0/2253502828 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde4802cc50 con 0x7fde500d1680 2024-01-15T18:10:10.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.887+0000 7fde4e000640 1 -- 172.21.15.2:0/2253502828 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde480343f0 con 0x7fde500d1680 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.887+0000 7fde55200640 1 -- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde500d1680 msgr2=0x7fde500d1b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.887+0000 7fde55200640 1 --2- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde500d1680 0x7fde500d1b10 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fde48009e50 tx=0x7fde4802bf50 comp rx=0 tx=0).stop 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.888+0000 7fde55200640 1 -- 172.21.15.2:0/2253502828 shutdown_connections 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.888+0000 7fde55200640 1 --2- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde500d1680 0x7fde500d1b10 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.888+0000 7fde55200640 1 --2- 172.21.15.2:0/2253502828 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d3090 0x7fde500d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.888+0000 7fde55200640 1 -- 172.21.15.2:0/2253502828 >> 172.21.15.2:0/2253502828 conn(0x7fde5008caf0 msgr2=0x7fde5009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.888+0000 7fde55200640 1 -- 172.21.15.2:0/2253502828 shutdown_connections 2024-01-15T18:10:10.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.888+0000 7fde55200640 1 -- 172.21.15.2:0/2253502828 wait complete. 2024-01-15T18:10:10.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.890+0000 7fde55200640 1 Processor -- start 2024-01-15T18:10:10.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.890+0000 7fde55200640 1 -- start start 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde55200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d1680 0x7fde501f8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde55200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 0x7fde501f9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde55200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5015d340 0x7fde501fa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde55200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde501fa4c0 con 0x7fde500d3090 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde55200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde50158d00 con 0x7fde5015d340 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde55200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde501fb620 con 0x7fde500d1680 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde4ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 0x7fde501f9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde4ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 0x7fde501f9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41712/0 (socket says 172.21.15.2:41712) 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.892+0000 7fde4ea00640 1 -- 172.21.15.2:0/229136508 learned_addr learned my addr 172.21.15.2:0/229136508 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4f400640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d1680 0x7fde501f8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4fe00640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5015d340 0x7fde501fa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4ea00640 1 -- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d1680 msgr2=0x7fde501f8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4ea00640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d1680 0x7fde501f8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4ea00640 1 -- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5015d340 msgr2=0x7fde501fa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:10.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4ea00640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5015d340 0x7fde501fa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:10.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4ea00640 1 -- 172.21.15.2:0/229136508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde38009400 con 0x7fde500d3090 2024-01-15T18:10:10.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde4ea00640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 0x7fde501f9f80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e990 tx=0x7fde3800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:10.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3800c980 con 0x7fde500d3090 2024-01-15T18:10:10.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.893+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde3800cae0 con 0x7fde500d3090 2024-01-15T18:10:10.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.894+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde48009b30 con 0x7fde500d3090 2024-01-15T18:10:10.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.894+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde3800ce70 con 0x7fde500d3090 2024-01-15T18:10:10.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.894+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde501586b0 con 0x7fde500d3090 2024-01-15T18:10:10.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.895+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde50067970 con 0x7fde500d3090 2024-01-15T18:10:10.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.896+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fde38013470 con 0x7fde500d3090 2024-01-15T18:10:10.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.897+0000 7fde47400640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde2806bfe0 0x7fde2806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:10.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.897+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fde38088c10 con 0x7fde500d3090 2024-01-15T18:10:10.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.898+0000 7fde4f400640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde2806bfe0 0x7fde2806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:10.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.899+0000 7fde4f400640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde2806bfe0 0x7fde2806e4a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fde40000f60 tx=0x7fde40007c80 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:10.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:10.901+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde38055450 con 0x7fde500d3090 2024-01-15T18:10:11.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.098+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fde50074770 con 0x7fde500d3090 2024-01-15T18:10:11.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.098+0000 7fde47400640 1 -- 172.21.15.2:0/229136508 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7fde38058a70 con 0x7fde500d3090 2024-01-15T18:10:11.103 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:10:11.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde2806bfe0 msgr2=0x7fde2806e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde2806bfe0 0x7fde2806e4a0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fde40000f60 tx=0x7fde40007c80 comp rx=0 tx=0).stop 2024-01-15T18:10:11.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 msgr2=0x7fde501f9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 0x7fde501f9f80 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7fde3800e990 tx=0x7fde3800ee50 comp rx=0 tx=0).stop 2024-01-15T18:10:11.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 shutdown_connections 2024-01-15T18:10:11.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fde2806bfe0 0x7fde2806e4a0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fde5015d340 0x7fde501fa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fde500d3090 0x7fde501f9f80 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 --2- 172.21.15.2:0/229136508 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fde500d1680 0x7fde501f8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 >> 172.21.15.2:0/229136508 conn(0x7fde5008caf0 msgr2=0x7fde500b4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 shutdown_connections 2024-01-15T18:10:11.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.101+0000 7fde55200640 1 -- 172.21.15.2:0/229136508 wait complete. 2024-01-15T18:10:11.126 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:10:11.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.341+0000 7f181da00640 1 Processor -- start 2024-01-15T18:10:11.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.341+0000 7f181da00640 1 -- start start 2024-01-15T18:10:11.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.343+0000 7f181da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f181815d260 0x7f18180d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.343+0000 7f181da00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f18180d11f0 0x7f18180d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.343+0000 7f181da00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f18180d1be0 0x7f1818160e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.343+0000 7f181da00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18180d2010 con 0x7f18180d1be0 2024-01-15T18:10:11.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.343+0000 7f181da00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18180cdd90 con 0x7f18180d11f0 2024-01-15T18:10:11.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.343+0000 7f181da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18180ce070 con 0x7f181815d260 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f18180d1be0 0x7f1818160e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f18180d1be0 0x7f1818160e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41722/0 (socket says 172.21.15.2:41722) 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 -- 172.21.15.2:0/631459372 learned_addr learned my addr 172.21.15.2:0/631459372 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f180fe00640 1 --2- 172.21.15.2:0/631459372 >> v2:172.21.15.137:3300/0 conn(0x7f18180d11f0 0x7f18180d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f1817e00640 1 --2- 172.21.15.2:0/631459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f181815d260 0x7f18180d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 -- 172.21.15.2:0/631459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f181815d260 msgr2=0x7f18180d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 --2- 172.21.15.2:0/631459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f181815d260 0x7f18180d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 -- 172.21.15.2:0/631459372 >> v2:172.21.15.137:3300/0 conn(0x7f18180d11f0 msgr2=0x7f18180d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 --2- 172.21.15.2:0/631459372 >> v2:172.21.15.137:3300/0 conn(0x7f18180d11f0 0x7f18180d16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.344+0000 7f181d000640 1 -- 172.21.15.2:0/631459372 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18180ce350 con 0x7f18180d1be0 2024-01-15T18:10:11.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.346+0000 7f181d000640 1 --2- 172.21.15.2:0/631459372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f1818160e50 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f181808cd30 tx=0x7f181002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec9dbb301c6570b8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:11.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.346+0000 7f1817400640 1 -- 172.21.15.2:0/631459372 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181002ca70 con 0x7f18180d1be0 2024-01-15T18:10:11.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.346+0000 7f1817400640 1 -- 172.21.15.2:0/631459372 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181002cbd0 con 0x7f18180d1be0 2024-01-15T18:10:11.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.346+0000 7f1817400640 1 -- 172.21.15.2:0/631459372 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100343f0 con 0x7f18180d1be0 2024-01-15T18:10:11.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.346+0000 7f181da00640 1 -- 172.21.15.2:0/631459372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 msgr2=0x7f1818160e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.346+0000 7f181da00640 1 --2- 172.21.15.2:0/631459372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f1818160e50 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f181808cd30 tx=0x7f181002bed0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 -- 172.21.15.2:0/631459372 shutdown_connections 2024-01-15T18:10:11.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 --2- 172.21.15.2:0/631459372 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f1818160e50 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 --2- 172.21.15.2:0/631459372 >> v2:172.21.15.137:3300/0 conn(0x7f18180d11f0 0x7f18180d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 --2- 172.21.15.2:0/631459372 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f181815d260 0x7f18180d0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 -- 172.21.15.2:0/631459372 >> 172.21.15.2:0/631459372 conn(0x7f181809c3b0 msgr2=0x7f18180d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:11.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 -- 172.21.15.2:0/631459372 shutdown_connections 2024-01-15T18:10:11.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.347+0000 7f181da00640 1 -- 172.21.15.2:0/631459372 wait complete. 2024-01-15T18:10:11.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.348+0000 7f181da00640 1 Processor -- start 2024-01-15T18:10:11.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.349+0000 7f181da00640 1 -- start start 2024-01-15T18:10:11.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181da00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f18180d11f0 0x7f18181f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f18180b6d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181da00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f181815d260 0x7f181815d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181da00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181815ede0 con 0x7f18180d1be0 2024-01-15T18:10:11.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181da00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18181f3d20 con 0x7f181815d260 2024-01-15T18:10:11.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181da00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18180b86c0 con 0x7f18180d11f0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f18180b6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f18180b6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41738/0 (socket says 172.21.15.2:41738) 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 -- 172.21.15.2:0/3601502150 learned_addr learned my addr 172.21.15.2:0/3601502150 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f1817e00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f18180d11f0 0x7f18181f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f181d000640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f181815d260 0x7f181815d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 -- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f18180d11f0 msgr2=0x7f18181f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f18180d11f0 0x7f18181f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 -- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f181815d260 msgr2=0x7f181815d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f181815d260 0x7f181815d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.350+0000 7f180fe00640 1 -- 172.21.15.2:0/3601502150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1810009b30 con 0x7f18180d1be0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f181d000640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f181815d260 0x7f181815d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f180fe00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f18180b6d20 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f180000e840 tx=0x7f180000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000c730 con 0x7f18180d1be0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180000c890 con 0x7f18180d1be0 2024-01-15T18:10:11.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180000cc40 con 0x7f18180d1be0 2024-01-15T18:10:11.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18180cefb0 con 0x7f18180d1be0 2024-01-15T18:10:11.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.351+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18180b7be0 con 0x7f18180d1be0 2024-01-15T18:10:11.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.352+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18180b5b40 con 0x7f18180d1be0 2024-01-15T18:10:11.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.353+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1800013470 con 0x7f18180d1be0 2024-01-15T18:10:11.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.354+0000 7f1814c00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17f006bfb0 0x7f17f006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:11.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.354+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f1800088620 con 0x7f18180d1be0 2024-01-15T18:10:11.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.355+0000 7f1817e00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17f006bfb0 0x7f17f006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:11.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.356+0000 7f1817e00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17f006bfb0 0x7f17f006e470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1808009ad0 tx=0x7f1808008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:11.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.359+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1800054ee0 con 0x7f18180d1be0 2024-01-15T18:10:11.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.564+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1818074770 con 0x7f18180d1be0 2024-01-15T18:10:11.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.565+0000 7f1814c00640 1 -- 172.21.15.2:0/3601502150 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v80) v1 ==== 80+0+120 (secure 0 0 0) 0x7f1800021080 con 0x7f18180d1be0 2024-01-15T18:10:11.569 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6835/4035045426 2024-01-16T18:10:05.021990+0000 2024-01-15T18:10:11.569 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/4035045426 2024-01-16T18:10:05.021990+0000 2024-01-15T18:10:11.569 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:10:11.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17f006bfb0 msgr2=0x7f17f006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17f006bfb0 0x7f17f006e470 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f1808009ad0 tx=0x7f1808008040 comp rx=0 tx=0).stop 2024-01-15T18:10:11.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 msgr2=0x7f18180b6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:11.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f18180b6d20 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f180000e840 tx=0x7f180000ed00 comp rx=0 tx=0).stop 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 shutdown_connections 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17f006bfb0 0x7f17f006e470 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f181815d260 0x7f181815d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18180d1be0 0x7f18180b6d20 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 --2- 172.21.15.2:0/3601502150 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f18180d11f0 0x7f18181f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.567+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 >> 172.21.15.2:0/3601502150 conn(0x7f181809c3b0 msgr2=0x7f18180b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.568+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 shutdown_connections 2024-01-15T18:10:11.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:11.568+0000 7f181da00640 1 -- 172.21.15.2:0/3601502150 wait complete. 2024-01-15T18:10:11.592 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:10:11.592 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:11.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:11.687 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45006]: fuse finished with error 0 and tester_r 0 2024-01-15T18:10:11.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:10:11.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:11.691 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:10:11.870 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:10:11.870 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:11.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:10:11.930 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:10:11.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:11.930 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:10:11.984 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:10:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:10:11.985 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:10:11.986 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:12.042 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:10:12.042 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:10:12.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:12.042 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:10:12.096 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:10:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:10:12.098 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:10:12.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.323+0000 7fa61d000640 1 Processor -- start 2024-01-15T18:10:12.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.323+0000 7fa61d000640 1 -- start start 2024-01-15T18:10:12.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa61d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa6181469f0 0x7fa618071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa61d000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180ac870 con 0x7fa61806cb80 2024-01-15T18:10:12.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa61d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61815d970 con 0x7fa618146dd0 2024-01-15T18:10:12.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa61d000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61815dc30 con 0x7fa6181469f0 2024-01-15T18:10:12.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa616800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa61806cb80 0x7fa6180b34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45068/0 (socket says 172.21.15.2:45068) 2024-01-15T18:10:12.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa616800640 1 -- 172.21.15.2:0/2024105408 learned_addr learned my addr 172.21.15.2:0/2024105408 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:12.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.325+0000 7fa615e00640 1 --2- 172.21.15.2:0/2024105408 >> v2:172.21.15.2:3300/0 conn(0x7fa6181469f0 0x7fa618071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.326+0000 7fa615e00640 1 -- 172.21.15.2:0/2024105408 >> v1:172.21.15.137:6789/0 conn(0x7fa618146dd0 legacy=0x7fa61806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:12.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.326+0000 7fa615e00640 1 -- 172.21.15.2:0/2024105408 >> v1:172.21.15.2:6789/0 conn(0x7fa61806cb80 legacy=0x7fa6180b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.326+0000 7fa615400640 1 -- 172.21.15.2:0/2024105408 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040688191 0 0) 0x7fa6180ac870 con 0x7fa61806cb80 2024-01-15T18:10:12.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.326+0000 7fa615e00640 1 -- 172.21.15.2:0/2024105408 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61815df10 con 0x7fa6181469f0 2024-01-15T18:10:12.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.327+0000 7fa615e00640 1 --2- 172.21.15.2:0/2024105408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6181469f0 0x7fa618071df0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097a0 tx=0x7fa60802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d630525a241f4835 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa615400640 1 -- 172.21.15.2:0/2024105408 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608031830 con 0x7fa6181469f0 2024-01-15T18:10:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa615400640 1 -- 172.21.15.2:0/2024105408 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa608031990 con 0x7fa6181469f0 2024-01-15T18:10:12.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa615400640 1 -- 172.21.15.2:0/2024105408 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608031d20 con 0x7fa6181469f0 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa61d000640 1 -- 172.21.15.2:0/2024105408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6181469f0 msgr2=0x7fa618071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa61d000640 1 --2- 172.21.15.2:0/2024105408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6181469f0 0x7fa618071df0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097a0 tx=0x7fa60802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa61d000640 1 -- 172.21.15.2:0/2024105408 shutdown_connections 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa61d000640 1 --2- 172.21.15.2:0/2024105408 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa6181469f0 0x7fa618071df0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.328+0000 7fa61d000640 1 -- 172.21.15.2:0/2024105408 >> 172.21.15.2:0/2024105408 conn(0x7fa61806da20 msgr2=0x7fa61806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.329+0000 7fa61d000640 1 -- 172.21.15.2:0/2024105408 shutdown_connections 2024-01-15T18:10:12.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.329+0000 7fa61d000640 1 -- 172.21.15.2:0/2024105408 wait complete. 2024-01-15T18:10:12.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.330+0000 7fa61d000640 1 Processor -- start 2024-01-15T18:10:12.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.330+0000 7fa61d000640 1 -- start start 2024-01-15T18:10:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.331+0000 7fa61d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 0x7fa61815cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.331+0000 7fa61d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6181469f0 0x7fa6180c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.331+0000 7fa61d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa618145490 0x7fa618145940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.331+0000 7fa61d000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c4df0 con 0x7fa61806cb80 2024-01-15T18:10:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.331+0000 7fa61d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618145e80 con 0x7fa618145490 2024-01-15T18:10:12.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.331+0000 7fa61d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61815d140 con 0x7fa6181469f0 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 0x7fa61815cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 0x7fa61815cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41752/0 (socket says 172.21.15.2:41752) 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 -- 172.21.15.2:0/1458811267 learned_addr learned my addr 172.21.15.2:0/1458811267 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa60fe00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6181469f0 0x7fa6180c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa616800640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa618145490 0x7fa618145940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 -- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6181469f0 msgr2=0x7fa6180c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6181469f0 0x7fa6180c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 -- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa618145490 msgr2=0x7fa618145940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa618145490 0x7fa618145940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 -- 172.21.15.2:0/1458811267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa608009400 con 0x7fa61806cb80 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa60fe00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6181469f0 0x7fa6180c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.332+0000 7fa615e00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 0x7fa61815cc00 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fa6080098c0 tx=0x7fa6080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.333+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60803bbf0 con 0x7fa61806cb80 2024-01-15T18:10:12.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.333+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60803bd50 con 0x7fa61806cb80 2024-01-15T18:10:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.333+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa608045630 con 0x7fa61806cb80 2024-01-15T18:10:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.333+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6180c4170 con 0x7fa61806cb80 2024-01-15T18:10:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.333+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6180c4730 con 0x7fa61806cb80 2024-01-15T18:10:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.334+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6180686f0 con 0x7fa61806cb80 2024-01-15T18:10:12.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.335+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fa608045790 con 0x7fa61806cb80 2024-01-15T18:10:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.336+0000 7fa60e000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa5f006bfb0 0x7fa5f006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.336+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fa6080aa910 con 0x7fa61806cb80 2024-01-15T18:10:12.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.337+0000 7fa60fe00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa5f006bfb0 0x7fa5f006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.338+0000 7fa60fe00640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa5f006bfb0 0x7fa5f006e470 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa618144270 tx=0x7fa600008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:12.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.341+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa6080772f0 con 0x7fa61806cb80 2024-01-15T18:10:12.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.574+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61815fde0 con 0x7fa61806cb80 2024-01-15T18:10:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.575+0000 7fa60e000640 1 -- 172.21.15.2:0/1458811267 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fa60807a910 con 0x7fa61806cb80 2024-01-15T18:10:12.581 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:12.581 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5573,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2894571444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2894571444},{"type":"v1","addr":"172.21.15.137:6836","nonce":2894571444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5980,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/2585344206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2585344206},{"type":"v1","addr":"172.21.15.137:6837","nonce":2585344206}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2342931858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2342931858},{"type":"v1","addr":"172.21.15.2:6833","nonce":2342931858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7923,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3426460349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3426460349},{"type":"v1","addr":"172.21.15.2:6835","nonce":3426460349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 49 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa5f006bfb0 msgr2=0x7fa5f006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa5f006bfb0 0x7fa5f006e470 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa618144270 tx=0x7fa600008040 comp rx=0 tx=0).stop 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 msgr2=0x7fa61815cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 0x7fa61815cc00 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fa6080098c0 tx=0x7fa6080046c0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 shutdown_connections 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa5f006bfb0 0x7fa5f006e470 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa618145490 0x7fa618145940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa6181469f0 0x7fa6180c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 --2- 172.21.15.2:0/1458811267 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa61806cb80 0x7fa61815cc00 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 >> 172.21.15.2:0/1458811267 conn(0x7fa61806da20 msgr2=0x7fa61815ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.577+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 shutdown_connections 2024-01-15T18:10:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.578+0000 7fa61d000640 1 -- 172.21.15.2:0/1458811267 wait complete. 2024-01-15T18:10:12.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:10:12.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.824+0000 7f746f200640 1 Processor -- start 2024-01-15T18:10:12.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.825+0000 7f746f200640 1 -- start start 2024-01-15T18:10:12.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.826+0000 7f746f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700d1870 0x7f74700d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.826+0000 7f746f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f74700b2210 0x7f747015d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.826+0000 7f746f200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700d1250 con 0x7f74700d1c50 2024-01-15T18:10:12.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74700ce6d0 con 0x7f74700b2210 2024-01-15T18:10:12.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700ce8e0 con 0x7f74700d1870 2024-01-15T18:10:12.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746e800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f74700d1c50 0x7f74700cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:45072/0 (socket says 172.21.15.2:45072) 2024-01-15T18:10:12.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746e800640 1 -- 172.21.15.2:0/2956670088 learned_addr learned my addr 172.21.15.2:0/2956670088 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:12.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746d400640 1 --2- 172.21.15.2:0/2956670088 >> v2:172.21.15.137:3300/0 conn(0x7f74700b2210 0x7f747015d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746de00640 1 --2- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700d1870 0x7f74700d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.827+0000 7f746ca00640 1 -- 172.21.15.2:0/2956670088 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3796277637 0 0) 0x7f74700d1250 con 0x7f74700d1c50 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.828+0000 7f746ca00640 1 -- 172.21.15.2:0/2956670088 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7450003520 con 0x7f74700d1c50 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.828+0000 7f746d400640 1 -- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700d1870 msgr2=0x7f74700d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.828+0000 7f746d400640 1 --2- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700d1870 0x7f74700d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.828+0000 7f746ca00640 1 -- 172.21.15.2:0/2956670088 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1455017734 0 0) 0x7f7450003520 con 0x7f74700d1c50 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.828+0000 7f746d400640 1 -- 172.21.15.2:0/2956670088 >> v1:172.21.15.2:6789/0 conn(0x7f74700d1c50 legacy=0x7f74700cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.828+0000 7f746d400640 1 -- 172.21.15.2:0/2956670088 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700cebc0 con 0x7f74700b2210 2024-01-15T18:10:12.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.830+0000 7f746d400640 1 --2- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700b2210 0x7f747015d0e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7460009d90 tx=0x7f746002bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17361c01e5f9780a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:12.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.830+0000 7f746ca00640 1 -- 172.21.15.2:0/2956670088 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7468001670 con 0x7f74700b2210 2024-01-15T18:10:12.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.830+0000 7f746ca00640 1 -- 172.21.15.2:0/2956670088 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746002c8a0 con 0x7f74700b2210 2024-01-15T18:10:12.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.830+0000 7f746ca00640 1 -- 172.21.15.2:0/2956670088 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746002cc30 con 0x7f74700b2210 2024-01-15T18:10:12.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 -- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700b2210 msgr2=0x7f747015d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 --2- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700b2210 0x7f747015d0e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7460009d90 tx=0x7f746002bd70 comp rx=0 tx=0).stop 2024-01-15T18:10:12.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 -- 172.21.15.2:0/2956670088 shutdown_connections 2024-01-15T18:10:12.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 --2- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700b2210 0x7f747015d0e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 --2- 172.21.15.2:0/2956670088 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700d1870 0x7f74700d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 -- 172.21.15.2:0/2956670088 >> 172.21.15.2:0/2956670088 conn(0x7f74700d30b0 msgr2=0x7f74700d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:12.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 -- 172.21.15.2:0/2956670088 shutdown_connections 2024-01-15T18:10:12.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.831+0000 7f746f200640 1 -- 172.21.15.2:0/2956670088 wait complete. 2024-01-15T18:10:12.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.833+0000 7f746f200640 1 Processor -- start 2024-01-15T18:10:12.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.833+0000 7f746f200640 1 -- start start 2024-01-15T18:10:12.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700b2210 0x7f74700b88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 0x7f74701f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700d1c50 0x7f747009e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700b5e70 con 0x7f74700d1870 2024-01-15T18:10:12.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74700b6150 con 0x7f74700d1c50 2024-01-15T18:10:12.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700b6430 con 0x7f74700b2210 2024-01-15T18:10:12.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.834+0000 7f746d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 0x7f74701f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 0x7f74701f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41758/0 (socket says 172.21.15.2:41758) 2024-01-15T18:10:12.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 -- 172.21.15.2:0/4281190334 learned_addr learned my addr 172.21.15.2:0/4281190334 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:12.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746de00640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700b2210 0x7f74700b88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746e800640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700d1c50 0x7f747009e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 -- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700b2210 msgr2=0x7f74700b88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700b2210 0x7f74700b88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 -- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700d1c50 msgr2=0x7f747009e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700d1c50 0x7f747009e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 -- 172.21.15.2:0/4281190334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7460009a70 con 0x7f74700d1870 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.835+0000 7f746d400640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 0x7f74701f2c00 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f7460037040 tx=0x7f7460004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.836+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460009450 con 0x7f74700d1870 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.836+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7460040450 con 0x7f74700d1870 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.836+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74700b6710 con 0x7f74700d1870 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.836+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7460040800 con 0x7f74700d1870 2024-01-15T18:10:12.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.836+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74700b8fc0 con 0x7f74700d1870 2024-01-15T18:10:12.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.837+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74700c9fd0 con 0x7f74700d1870 2024-01-15T18:10:12.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.838+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7460040960 con 0x7f74700d1870 2024-01-15T18:10:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.839+0000 7f7466000640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f744806bfb0 0x7f744806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:12.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.839+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f74600ab340 con 0x7f74700d1870 2024-01-15T18:10:12.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.840+0000 7f746de00640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f744806bfb0 0x7f744806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:12.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.841+0000 7f746de00640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f744806bfb0 0x7f744806e470 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7458009ad0 tx=0x7f7458008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:12.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:12.844+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7460077a90 con 0x7f74700d1870 2024-01-15T18:10:13.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.036+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f74700b5fb0 con 0x7f74700d1870 2024-01-15T18:10:13.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.043+0000 7f7466000640 1 -- 172.21.15.2:0/4281190334 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v28) v1 ==== 120+0+0 (secure 0 0 0) 0x7f74700b5fb0 con 0x7f74700d1870 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f744806bfb0 msgr2=0x7f744806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f744806bfb0 0x7f744806e470 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7458009ad0 tx=0x7f7458008040 comp rx=0 tx=0).stop 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 msgr2=0x7f74701f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 0x7f74701f2c00 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f7460037040 tx=0x7f7460004a40 comp rx=0 tx=0).stop 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 shutdown_connections 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f744806bfb0 0x7f744806e470 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f74700d1c50 0x7f747009e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f74700d1870 0x7f74701f2c00 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 --2- 172.21.15.2:0/4281190334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f74700b2210 0x7f74700b88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 >> 172.21.15.2:0/4281190334 conn(0x7f74700d30b0 msgr2=0x7f74700b47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 shutdown_connections 2024-01-15T18:10:13.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.046+0000 7f746f200640 1 -- 172.21.15.2:0/4281190334 wait complete. 2024-01-15T18:10:13.070 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.291+0000 7fca11400640 1 Processor -- start 2024-01-15T18:10:13.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.291+0000 7fca11400640 1 -- start start 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.292+0000 7fca11400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca0c143620 0x7fca0c0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.292+0000 7fca11400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b2fa0 con 0x7fca0c0b2a90 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca11400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca0c0b3280 con 0x7fca0c071940 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca11400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca0c0b3560 con 0x7fca0c143620 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca0be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca0c143620 0x7fca0c0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca03e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fca0c0b2a90 0x7fca0c071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54428/0 (socket says 172.21.15.2:54428) 2024-01-15T18:10:13.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca03e00640 1 -- 172.21.15.2:0/2185469643 learned_addr learned my addr 172.21.15.2:0/2185469643 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca0be00640 1 -- 172.21.15.2:0/2185469643 >> v1:172.21.15.137:6789/0 conn(0x7fca0c071940 legacy=0x7fca0c0cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca0be00640 1 -- 172.21.15.2:0/2185469643 >> v1:172.21.15.2:6789/0 conn(0x7fca0c0b2a90 legacy=0x7fca0c071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca0b400640 1 -- 172.21.15.2:0/2185469643 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913377417 0 0) 0x7fca0c0b2fa0 con 0x7fca0c0b2a90 2024-01-15T18:10:13.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.293+0000 7fca0be00640 1 -- 172.21.15.2:0/2185469643 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0c0b3840 con 0x7fca0c143620 2024-01-15T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.295+0000 7fca0be00640 1 --2- 172.21.15.2:0/2185469643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c143620 0x7fca0c0b2550 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0097a0 tx=0x7fc9fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8269801d2f1030b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.295+0000 7fca0b400640 1 -- 172.21.15.2:0/2185469643 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc031830 con 0x7fca0c143620 2024-01-15T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.295+0000 7fca0b400640 1 -- 172.21.15.2:0/2185469643 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc031990 con 0x7fca0c143620 2024-01-15T18:10:13.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.295+0000 7fca0b400640 1 -- 172.21.15.2:0/2185469643 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc031d20 con 0x7fca0c143620 2024-01-15T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.295+0000 7fca11400640 1 -- 172.21.15.2:0/2185469643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c143620 msgr2=0x7fca0c0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.295+0000 7fca11400640 1 --2- 172.21.15.2:0/2185469643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c143620 0x7fca0c0b2550 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0097a0 tx=0x7fc9fc02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.296+0000 7fca11400640 1 -- 172.21.15.2:0/2185469643 shutdown_connections 2024-01-15T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.296+0000 7fca11400640 1 --2- 172.21.15.2:0/2185469643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c143620 0x7fca0c0b2550 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.296+0000 7fca11400640 1 -- 172.21.15.2:0/2185469643 >> 172.21.15.2:0/2185469643 conn(0x7fca0c0b0840 msgr2=0x7fca0c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:13.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.296+0000 7fca11400640 1 -- 172.21.15.2:0/2185469643 shutdown_connections 2024-01-15T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.296+0000 7fca11400640 1 -- 172.21.15.2:0/2185469643 wait complete. 2024-01-15T18:10:13.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.297+0000 7fca11400640 1 Processor -- start 2024-01-15T18:10:13.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 -- start start 2024-01-15T18:10:13.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 0x7fca0c1e68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca0c0b2a90 0x7fca0c06d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0c143620 0x7fca0c1e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1e78f0 con 0x7fca0c071940 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1407b0 con 0x7fca0c0b2a90 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.298+0000 7fca11400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c1e6e00 con 0x7fca0c143620 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 0x7fca0c1e68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 0x7fca0c1e68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43620/0 (socket says 172.21.15.2:43620) 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 -- 172.21.15.2:0/885621001 learned_addr learned my addr 172.21.15.2:0/885621001 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca03e00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca0c0b2a90 0x7fca0c06d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca10a00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0c143620 0x7fca0c1e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 -- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0c143620 msgr2=0x7fca0c1e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0c143620 0x7fca0c1e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 -- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca0c0b2a90 msgr2=0x7fca0c06d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca0c0b2a90 0x7fca0c06d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca0be00640 1 -- 172.21.15.2:0/885621001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc009400 con 0x7fca0c071940 2024-01-15T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.299+0000 7fca10a00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0c143620 0x7fca0c1e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:13.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.300+0000 7fca0be00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 0x7fca0c1e68c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0097a0 tx=0x7fc9fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.300+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc03aa60 con 0x7fca0c071940 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.300+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9fc03abc0 con 0x7fca0c071940 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.300+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc0443f0 con 0x7fca0c071940 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.300+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c1e7050 con 0x7fca0c071940 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.300+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c1e7610 con 0x7fca0c071940 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.301+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c074770 con 0x7fca0c071940 2024-01-15T18:10:13.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.302+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc9fc04d450 con 0x7fca0c071940 2024-01-15T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.303+0000 7fca08c00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e406bfb0 0x7fc9e406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.303+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7fc9fc0aa840 con 0x7fca0c071940 2024-01-15T18:10:13.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.304+0000 7fca03e00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e406bfb0 0x7fc9e406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.305+0000 7fca03e00640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e406bfb0 0x7fc9e406e470 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fca0c1515a0 tx=0x7fc9f4007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.307+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc9fc077100 con 0x7fca0c071940 2024-01-15T18:10:13.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.502+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fca0c0cd6c0 con 0x7fca0c071940 2024-01-15T18:10:13.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.509+0000 7fca08c00640 1 -- 172.21.15.2:0/885621001 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc9fc07a720 con 0x7fca0c071940 2024-01-15T18:10:13.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e406bfb0 msgr2=0x7fc9e406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e406bfb0 0x7fc9e406e470 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fca0c1515a0 tx=0x7fc9f4007b50 comp rx=0 tx=0).stop 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 msgr2=0x7fca0c1e68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 0x7fca0c1e68c0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0097a0 tx=0x7fc9fc0046c0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 shutdown_connections 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e406bfb0 0x7fc9e406e470 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca0c143620 0x7fca0c1e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca0c0b2a90 0x7fca0c06d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 --2- 172.21.15.2:0/885621001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca0c071940 0x7fca0c1e68c0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 >> 172.21.15.2:0/885621001 conn(0x7fca0c0b0840 msgr2=0x7fca0c1e85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 shutdown_connections 2024-01-15T18:10:13.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.512+0000 7fca11400640 1 -- 172.21.15.2:0/885621001 wait complete. 2024-01-15T18:10:13.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-15T18:10:13.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.756+0000 7f2308000640 1 Processor -- start 2024-01-15T18:10:13.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.756+0000 7f2308000640 1 -- start start 2024-01-15T18:10:13.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2308000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f230015c0e0 0x7f23000cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2308000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 0x7f230006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2308000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230015cb00 con 0x7f230015c4c0 2024-01-15T18:10:13.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2308000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23000ccec0 con 0x7f230015c0e0 2024-01-15T18:10:13.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2308000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23000cd1a0 con 0x7f230015b7d0 2024-01-15T18:10:13.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2306000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f230015c4c0 0x7f230015ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54434/0 (socket says 172.21.15.2:54434) 2024-01-15T18:10:13.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.758+0000 7f2306000640 1 -- 172.21.15.2:0/2918006748 learned_addr learned my addr 172.21.15.2:0/2918006748 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f2305600640 1 --2- 172.21.15.2:0/2918006748 >> v2:172.21.15.137:3300/0 conn(0x7f230015c0e0 0x7f23000cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113596534 0 0) 0x7f230015cb00 con 0x7f230015c4c0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e0003520 con 0x7f230015c4c0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f2304c00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 0x7f230006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3626652033 0 0) 0x7f22e0003520 con 0x7f230015c4c0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 msgr2=0x7f230006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 0x7f230006c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 >> v2:172.21.15.137:3300/0 conn(0x7f230015c0e0 msgr2=0x7f23000cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 --2- 172.21.15.2:0/2918006748 >> v2:172.21.15.137:3300/0 conn(0x7f230015c0e0 0x7f23000cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000cd480 con 0x7f230015c4c0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f22f80019f0 con 0x7f230015c4c0 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 >> v1:172.21.15.2:6789/0 conn(0x7f230015c4c0 legacy=0x7f230015ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22e0003970 0x7f22e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f22e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 0x7f22e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000ccec0 con 0x7f22e00047c0 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000cd1a0 con 0x7f22e0003970 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.759+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e000b9d0 con 0x7f230015b7d0 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2304c00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f22e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2305600640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22e0003970 0x7f22e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2304c00640 1 -- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 msgr2=0x7f22e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2304c00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f230015b7d0 0x7f22e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2304c00640 1 -- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22e0003970 msgr2=0x7f22e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2304c00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f22e0003970 0x7f22e0003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.760+0000 7f2304c00640 1 -- 172.21.15.2:0/2918006748 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e0003520 con 0x7f22e00047c0 2024-01-15T18:10:13.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.761+0000 7f2304c00640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f22e0004c40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22e800ba00 tx=0x7f22e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0fe69a7d9d1d0b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.761+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8032640 con 0x7f22e00047c0 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.761+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e80327a0 con 0x7f22e00047c0 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.761+0000 7f22ffe00640 1 -- 172.21.15.2:0/2918006748 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e8032b30 con 0x7f22e00047c0 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2308000640 1 -- 172.21.15.2:0/2918006748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 msgr2=0x7f22e0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2308000640 1 --2- 172.21.15.2:0/2918006748 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f22e0004c40 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22e800ba00 tx=0x7f22e00057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2306000640 1 -- 172.21.15.2:0/2918006748 reap_dead start 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2308000640 1 -- 172.21.15.2:0/2918006748 shutdown_connections 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2308000640 1 -- 172.21.15.2:0/2918006748 >> 172.21.15.2:0/2918006748 conn(0x7f230006da20 msgr2=0x7f230006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2308000640 1 -- 172.21.15.2:0/2918006748 shutdown_connections 2024-01-15T18:10:13.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.762+0000 7f2308000640 1 -- 172.21.15.2:0/2918006748 wait complete. 2024-01-15T18:10:13.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.764+0000 7f2308000640 1 Processor -- start 2024-01-15T18:10:13.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.764+0000 7f2308000640 1 -- start start 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2308000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f2300146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2308000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2300071400 0x7f2300071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2308000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23001541f0 0x7f230015a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2308000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300147380 con 0x7f22e00047c0 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2308000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300153150 con 0x7f2300071400 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2308000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300153430 con 0x7f23001541f0 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2305600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f2300146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2305600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f2300146e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43634/0 (socket says 172.21.15.2:43634) 2024-01-15T18:10:13.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.765+0000 7f2305600640 1 -- 172.21.15.2:0/1456343062 learned_addr learned my addr 172.21.15.2:0/1456343062 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2306000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23001541f0 0x7f230015a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2305600640 1 -- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23001541f0 msgr2=0x7f230015a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2305600640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23001541f0 0x7f230015a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2305600640 1 -- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2300071400 msgr2=0x7f2300071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2305600640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2300071400 0x7f2300071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2305600640 1 -- 172.21.15.2:0/1456343062 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300153710 con 0x7f22e00047c0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f2305600640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f2300146e40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f22f00108e0 tx=0x7f22f0010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f000eb60 con 0x7f22e00047c0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f000ecc0 con 0x7f22e00047c0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.766+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f001c400 con 0x7f22e00047c0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.767+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23001539f0 con 0x7f22e00047c0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.767+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2300072180 con 0x7f22e00047c0 2024-01-15T18:10:13.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.768+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23000686f0 con 0x7f22e00047c0 2024-01-15T18:10:13.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.769+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f22f001c560 con 0x7f22e00047c0 2024-01-15T18:10:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.770+0000 7f22fd600640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f22d806bfe0 0x7f22d806e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.770+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f22f0088c90 con 0x7f22e00047c0 2024-01-15T18:10:13.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.771+0000 7f2304c00640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f22d806bfe0 0x7f22d806e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:13.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.772+0000 7f2304c00640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f22d806bfe0 0x7f22d806e4a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f230015bc10 tx=0x7f22e8032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:13.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:13.774+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22f0055600 con 0x7f22e00047c0 2024-01-15T18:10:14.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.043+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7f230006b260 con 0x7f22e00047c0 2024-01-15T18:10:14.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.077+0000 7f22fd600640 1 -- 172.21.15.2:0/1456343062 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v582) v1 ==== 166+0+0 (secure 0 0 0) 0x7f22f0058c20 con 0x7f22e00047c0 2024-01-15T18:10:14.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f22d806bfe0 msgr2=0x7f22d806e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f22d806bfe0 0x7f22d806e4a0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f230015bc10 tx=0x7f22e8032000 comp rx=0 tx=0).stop 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 msgr2=0x7f2300146e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f2300146e40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f22f00108e0 tx=0x7f22f0010da0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 shutdown_connections 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f22d806bfe0 0x7f22d806e4a0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f23001541f0 0x7f230015a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2300071400 0x7f2300071c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 --2- 172.21.15.2:0/1456343062 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f22e00047c0 0x7f2300146e40 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 >> 172.21.15.2:0/1456343062 conn(0x7f230006da20 msgr2=0x7f230006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 shutdown_connections 2024-01-15T18:10:14.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.082+0000 7f2308000640 1 -- 172.21.15.2:0/1456343062 wait complete. 2024-01-15T18:10:14.105 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-15T18:10:14.106 INFO:tasks.cephfs_test_runner:Starting test: test_stats_are_printed (tasks.cephfs.test_volumes.TestCloneProgress) Test that the command "ceph fs clone status" prints progress stats 2024-01-15T18:10:14.106 INFO:tasks.cephfs_test_runner:test_stats_are_printed (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:10:14.106 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:14.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.326+0000 7ff824a00640 1 Processor -- start 2024-01-15T18:10:14.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.326+0000 7ff824a00640 1 -- start start 2024-01-15T18:10:14.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.327+0000 7ff824a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.327+0000 7ff824a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8200cdbb0 con 0x7ff82006f720 2024-01-15T18:10:14.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.327+0000 7ff824a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff820071ff0 con 0x7ff8200cf070 2024-01-15T18:10:14.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.327+0000 7ff824a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff82015d800 con 0x7ff82015d420 2024-01-15T18:10:14.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81ea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7ff82006f720 0x7ff8200711d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54438/0 (socket says 172.21.15.2:54438) 2024-01-15T18:10:14.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81ea00640 1 -- 172.21.15.2:0/294953499 learned_addr learned my addr 172.21.15.2:0/294953499 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:14.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81f400640 1 --2- 172.21.15.2:0/294953499 >> v2:172.21.15.2:3300/0 conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81f400640 1 -- 172.21.15.2:0/294953499 >> v1:172.21.15.137:6789/0 conn(0x7ff8200cf070 legacy=0x7ff8200718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:14.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81f400640 1 -- 172.21.15.2:0/294953499 >> v1:172.21.15.2:6789/0 conn(0x7ff82006f720 legacy=0x7ff8200711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81f400640 1 -- 172.21.15.2:0/294953499 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff820161420 con 0x7ff82015d420 2024-01-15T18:10:14.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.328+0000 7ff81e000640 1 -- 172.21.15.2:0/294953499 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3292958019 0 0) 0x7ff8200cdbb0 con 0x7ff82006f720 2024-01-15T18:10:14.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.329+0000 7ff81f400640 1 --2- 172.21.15.2:0/294953499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff8100097a0 tx=0x7ff81002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8004531338be0c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.330+0000 7ff81e000640 1 -- 172.21.15.2:0/294953499 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810030610 con 0x7ff82015d420 2024-01-15T18:10:14.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.330+0000 7ff81e000640 1 -- 172.21.15.2:0/294953499 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff810030770 con 0x7ff82015d420 2024-01-15T18:10:14.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.330+0000 7ff81e000640 1 -- 172.21.15.2:0/294953499 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810030b00 con 0x7ff82015d420 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.330+0000 7ff824a00640 1 -- 172.21.15.2:0/294953499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 msgr2=0x7ff82006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.330+0000 7ff824a00640 1 --2- 172.21.15.2:0/294953499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7ff8100097a0 tx=0x7ff81002b960 comp rx=0 tx=0).stop 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.331+0000 7ff824a00640 1 -- 172.21.15.2:0/294953499 shutdown_connections 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.331+0000 7ff824a00640 1 --2- 172.21.15.2:0/294953499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.331+0000 7ff824a00640 1 -- 172.21.15.2:0/294953499 >> 172.21.15.2:0/294953499 conn(0x7ff8200cfd20 msgr2=0x7ff8200d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.331+0000 7ff824a00640 1 -- 172.21.15.2:0/294953499 shutdown_connections 2024-01-15T18:10:14.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.331+0000 7ff824a00640 1 -- 172.21.15.2:0/294953499 wait complete. 2024-01-15T18:10:14.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.332+0000 7ff824a00640 1 Processor -- start 2024-01-15T18:10:14.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.333+0000 7ff824a00640 1 -- start start 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.333+0000 7ff824a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff82006f720 0x7ff820158660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.333+0000 7ff824a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8200cf070 0x7ff8200b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.333+0000 7ff824a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.333+0000 7ff824a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8200b3720 con 0x7ff82015d420 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff824a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff820158ba0 con 0x7ff8200cf070 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff824a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8200b2b10 con 0x7ff82006f720 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43644/0 (socket says 172.21.15.2:43644) 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 -- 172.21.15.2:0/2385458080 learned_addr learned my addr 172.21.15.2:0/2385458080 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81f400640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff82006f720 0x7ff820158660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81ea00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8200cf070 0x7ff8200b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 -- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff82006f720 msgr2=0x7ff820158660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff82006f720 0x7ff820158660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 -- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8200cf070 msgr2=0x7ff8200b52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8200cf070 0x7ff8200b52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81fe00640 1 -- 172.21.15.2:0/2385458080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff810009400 con 0x7ff82015d420 2024-01-15T18:10:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81f400640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff82006f720 0x7ff820158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.334+0000 7ff81ea00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8200cf070 0x7ff8200b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:14.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.335+0000 7ff81fe00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff81800c3e0 tx=0x7ff81800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.335+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818011790 con 0x7ff82015d420 2024-01-15T18:10:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.335+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8180118f0 con 0x7ff82015d420 2024-01-15T18:10:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.335+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff818011ce0 con 0x7ff82015d420 2024-01-15T18:10:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.335+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8200b2df0 con 0x7ff82015d420 2024-01-15T18:10:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.335+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8200b33b0 con 0x7ff82015d420 2024-01-15T18:10:14.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.337+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8200b38d0 con 0x7ff82015d420 2024-01-15T18:10:14.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.337+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ff81801c460 con 0x7ff82015d420 2024-01-15T18:10:14.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.338+0000 7ff817400640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7f806bfb0 0x7ff7f806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.338+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7ff8180884d0 con 0x7ff82015d420 2024-01-15T18:10:14.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.339+0000 7ff81f400640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7f806bfb0 0x7ff7f806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.340+0000 7ff81f400640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7f806bfb0 0x7ff7f806e470 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff810036000 tx=0x7ff81003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.343+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8200b38d0 con 0x7ff82015d420 2024-01-15T18:10:14.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.562+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff820074770 con 0x7ff82015d420 2024-01-15T18:10:14.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.563+0000 7ff817400640 1 -- 172.21.15.2:0/2385458080 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+13056 (secure 0 0 0) 0x7ff818054d90 con 0x7ff82015d420 2024-01-15T18:10:14.568 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:14.568 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":80,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:08.043678+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/4035045426":"2024-01-16T18:10:05.021990+0000","172.21.15.2:6834/4035045426":"2024-01-16T18:10:05.021990+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:14.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7f806bfb0 msgr2=0x7ff7f806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7f806bfb0 0x7ff7f806e470 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff810036000 tx=0x7ff81003b040 comp rx=0 tx=0).stop 2024-01-15T18:10:14.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 msgr2=0x7ff82006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7ff81800c3e0 tx=0x7ff81800c8a0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 shutdown_connections 2024-01-15T18:10:14.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff7f806bfb0 0x7ff7f806e470 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff82015d420 0x7ff82006a4b0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff8200cf070 0x7ff8200b52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 --2- 172.21.15.2:0/2385458080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff82006f720 0x7ff820158660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 >> 172.21.15.2:0/2385458080 conn(0x7ff8200cfd20 msgr2=0x7ff8200ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 shutdown_connections 2024-01-15T18:10:14.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.566+0000 7ff824a00640 1 -- 172.21.15.2:0/2385458080 wait complete. 2024-01-15T18:10:14.588 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:10:14.588 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:10:14.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.804+0000 7f8296200640 1 Processor -- start 2024-01-15T18:10:14.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.805+0000 7f8296200640 1 -- start start 2024-01-15T18:10:14.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.806+0000 7f8296200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8290144c00 0x7f82900cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f8296200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f8296200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82900ce230 con 0x7f829015c450 2024-01-15T18:10:14.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f8296200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82900ce510 con 0x7f8290144c00 2024-01-15T18:10:14.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f8296200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82900ce7f0 con 0x7f82900cdd70 2024-01-15T18:10:14.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f828f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8290144c00 0x7f82900cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f828fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f829015c450 0x7f82900b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54444/0 (socket says 172.21.15.2:54444) 2024-01-15T18:10:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.807+0000 7f828fe00640 1 -- 172.21.15.2:0/1017160305 learned_addr learned my addr 172.21.15.2:0/1017160305 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f8287e00640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f828f400640 1 -- 172.21.15.2:0/1017160305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 msgr2=0x7f829006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f828f400640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829006c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f828ea00640 1 -- 172.21.15.2:0/1017160305 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963855085 0 0) 0x7f82900ce230 con 0x7f829015c450 2024-01-15T18:10:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f828f400640 1 -- 172.21.15.2:0/1017160305 >> v1:172.21.15.2:6789/0 conn(0x7f829015c450 legacy=0x7f82900b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f828f400640 1 -- 172.21.15.2:0/1017160305 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82900705f0 con 0x7f8290144c00 2024-01-15T18:10:14.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.808+0000 7f8287e00640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.809+0000 7f828f400640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8290144c00 0x7f82900cd830 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f82800097a0 tx=0x7f828002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd4d8b6766e73a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.809+0000 7f828ea00640 1 -- 172.21.15.2:0/1017160305 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828002d040 con 0x7f8290144c00 2024-01-15T18:10:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.809+0000 7f828ea00640 1 -- 172.21.15.2:0/1017160305 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8280031810 con 0x7f8290144c00 2024-01-15T18:10:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.809+0000 7f828ea00640 1 -- 172.21.15.2:0/1017160305 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8280031ba0 con 0x7f8290144c00 2024-01-15T18:10:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 -- 172.21.15.2:0/1017160305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8290144c00 msgr2=0x7f82900cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8290144c00 0x7f82900cd830 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f82800097a0 tx=0x7f828002b8b0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 -- 172.21.15.2:0/1017160305 shutdown_connections 2024-01-15T18:10:14.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829006c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 --2- 172.21.15.2:0/1017160305 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8290144c00 0x7f82900cd830 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 -- 172.21.15.2:0/1017160305 >> 172.21.15.2:0/1017160305 conn(0x7f82900b45f0 msgr2=0x7f82900b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:14.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 -- 172.21.15.2:0/1017160305 shutdown_connections 2024-01-15T18:10:14.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.810+0000 7f8296200640 1 -- 172.21.15.2:0/1017160305 wait complete. 2024-01-15T18:10:14.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.812+0000 7f8296200640 1 Processor -- start 2024-01-15T18:10:14.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.812+0000 7f8296200640 1 -- start start 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f8296200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829013d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f8296200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8290144c00 0x7f8290140680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f8296200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 0x7f829013b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f8296200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829013e5a0 con 0x7f829015c450 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f8296200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829013d650 con 0x7f8290144c00 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f8296200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f829013d920 con 0x7f82900cdd70 2024-01-15T18:10:14.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.813+0000 7f828fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 0x7f829013b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 0x7f829013b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43666/0 (socket says 172.21.15.2:43666) 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 -- 172.21.15.2:0/3113617943 learned_addr learned my addr 172.21.15.2:0/3113617943 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828f400640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829013d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f8287e00640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8290144c00 0x7f8290140680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 -- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 msgr2=0x7f829013d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829013d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 -- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8290144c00 msgr2=0x7f8290140680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8290144c00 0x7f8290140680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:14.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 -- 172.21.15.2:0/3113617943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280009400 con 0x7f829015c450 2024-01-15T18:10:14.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.814+0000 7f828fe00640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 0x7f829013b820 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f828800c700 tx=0x7f828800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.815+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828800ce10 con 0x7f829015c450 2024-01-15T18:10:14.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.815+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f828800a640 con 0x7f829015c450 2024-01-15T18:10:14.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.815+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f828800a9f0 con 0x7f829015c450 2024-01-15T18:10:14.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.815+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829013dc00 con 0x7f829015c450 2024-01-15T18:10:14.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.815+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829013e1c0 con 0x7f829015c450 2024-01-15T18:10:14.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.816+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829013e6e0 con 0x7f829015c450 2024-01-15T18:10:14.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.817+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f828800ab50 con 0x7f829015c450 2024-01-15T18:10:14.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.818+0000 7f8287400640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f826806bfb0 0x7f826806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:14.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.818+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f8288087c40 con 0x7f829015c450 2024-01-15T18:10:14.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.819+0000 7f828f400640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f826806bfb0 0x7f826806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:14.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.820+0000 7f828f400640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f826806bfb0 0x7f826806e470 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8280036000 tx=0x7f828003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:14.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:14.822+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8288054540 con 0x7f829015c450 2024-01-15T18:10:15.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.014+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f829013e6e0 con 0x7f829015c450 2024-01-15T18:10:15.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.015+0000 7f8287400640 1 -- 172.21.15.2:0/3113617943 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v80) v1 ==== 93+0+10 (secure 0 0 0) 0x7f829013e6e0 con 0x7f829015c450 2024-01-15T18:10:15.020 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f826806bfb0 msgr2=0x7f826806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f826806bfb0 0x7f826806e470 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f8280036000 tx=0x7f828003c040 comp rx=0 tx=0).stop 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 msgr2=0x7f829013b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 0x7f829013b820 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f828800c700 tx=0x7f828800cbc0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 shutdown_connections 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f826806bfb0 0x7f826806e470 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f829015c450 0x7f829013b820 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8290144c00 0x7f8290140680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 --2- 172.21.15.2:0/3113617943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f82900cdd70 0x7f829013d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 >> 172.21.15.2:0/3113617943 conn(0x7f82900b45f0 msgr2=0x7f829015e190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 shutdown_connections 2024-01-15T18:10:15.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.018+0000 7f8296200640 1 -- 172.21.15.2:0/3113617943 wait complete. 2024-01-15T18:10:15.043 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed' 2024-01-15T18:10:15.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.247+0000 7f7368e00640 1 Processor -- start 2024-01-15T18:10:15.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.248+0000 7f7368e00640 1 -- start start 2024-01-15T18:10:15.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.249+0000 7f7368e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f736415c0e0 0x7f73640cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:15.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.249+0000 7f7368e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f736415b7d0 0x7f736406c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:15.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7368e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736415cb00 con 0x7f736415c4c0 2024-01-15T18:10:15.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7368e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640ccec0 con 0x7f736415b7d0 2024-01-15T18:10:15.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7368e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640cd1a0 con 0x7f736415c0e0 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7362800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f736415c4c0 0x7f736415ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54458/0 (socket says 172.21.15.2:54458) 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7362800640 1 -- 172.21.15.2:0/3732579245 learned_addr learned my addr 172.21.15.2:0/3732579245 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7361400640 1 --2- 172.21.15.2:0/3732579245 >> v2:172.21.15.2:3300/0 conn(0x7f736415b7d0 0x7f736406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.250+0000 7f7361e00640 1 --2- 172.21.15.2:0/3732579245 >> v2:172.21.15.137:3300/0 conn(0x7f736415c0e0 0x7f73640cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.251+0000 7f7360a00640 1 -- 172.21.15.2:0/3732579245 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659249453 0 0) 0x7f736415cb00 con 0x7f736415c4c0 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.251+0000 7f7360a00640 1 -- 172.21.15.2:0/3732579245 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7344003520 con 0x7f736415c4c0 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.251+0000 7f7361400640 1 -- 172.21.15.2:0/3732579245 >> v2:172.21.15.137:3300/0 conn(0x7f736415c0e0 msgr2=0x7f73640cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:15.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.251+0000 7f7361400640 1 --2- 172.21.15.2:0/3732579245 >> v2:172.21.15.137:3300/0 conn(0x7f736415c0e0 0x7f73640cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.251+0000 7f7361400640 1 -- 172.21.15.2:0/3732579245 >> v1:172.21.15.2:6789/0 conn(0x7f736415c4c0 legacy=0x7f736415ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:15.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.251+0000 7f7361400640 1 -- 172.21.15.2:0/3732579245 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73640cd480 con 0x7f736415b7d0 2024-01-15T18:10:15.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.252+0000 7f7361400640 1 --2- 172.21.15.2:0/3732579245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736406c470 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f73640725f0 tx=0x7f734c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a01245e99fd48a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:15.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.252+0000 7f7360a00640 1 -- 172.21.15.2:0/3732579245 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c031830 con 0x7f736415b7d0 2024-01-15T18:10:15.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.252+0000 7f7360a00640 1 -- 172.21.15.2:0/3732579245 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c031990 con 0x7f736415b7d0 2024-01-15T18:10:15.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.252+0000 7f7360a00640 1 -- 172.21.15.2:0/3732579245 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c031d20 con 0x7f736415b7d0 2024-01-15T18:10:15.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 -- 172.21.15.2:0/3732579245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 msgr2=0x7f736406c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:15.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 --2- 172.21.15.2:0/3732579245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736406c470 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f73640725f0 tx=0x7f734c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 -- 172.21.15.2:0/3732579245 shutdown_connections 2024-01-15T18:10:15.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 --2- 172.21.15.2:0/3732579245 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736406c470 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 --2- 172.21.15.2:0/3732579245 >> v2:172.21.15.137:3300/0 conn(0x7f736415c0e0 0x7f73640cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 -- 172.21.15.2:0/3732579245 >> 172.21.15.2:0/3732579245 conn(0x7f736406da20 msgr2=0x7f736406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:15.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 -- 172.21.15.2:0/3732579245 shutdown_connections 2024-01-15T18:10:15.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.253+0000 7f7368e00640 1 -- 172.21.15.2:0/3732579245 wait complete. 2024-01-15T18:10:15.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.255+0000 7f7368e00640 1 Processor -- start 2024-01-15T18:10:15.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.255+0000 7f7368e00640 1 -- start start 2024-01-15T18:10:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7368e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736414d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:15.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7368e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f736415c0e0 0x7f736414f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7368e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f736415c4c0 0x7f7364153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7368e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736414ae10 con 0x7f736415b7d0 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7368e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736414b0f0 con 0x7f736415c4c0 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7368e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736414b3d0 con 0x7f736415c0e0 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7361e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736414d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7361e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736414d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43690/0 (socket says 172.21.15.2:43690) 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.256+0000 7f7361e00640 1 -- 172.21.15.2:0/110122287 learned_addr learned my addr 172.21.15.2:0/110122287 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361400640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f736415c0e0 0x7f736414f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:15.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361e00640 1 -- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f736415c0e0 msgr2=0x7f736414f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f736415c0e0 0x7f736414f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361e00640 1 -- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f736415c4c0 msgr2=0x7f7364153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f736415c4c0 0x7f7364153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361e00640 1 -- 172.21.15.2:0/110122287 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c009400 con 0x7f736415b7d0 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7361e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736414d9f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f735400e870 tx=0x7f735400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73540094f0 con 0x7f736415b7d0 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735400c940 con 0x7f736415b7d0 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735400ccf0 con 0x7f736415b7d0 2024-01-15T18:10:15.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.257+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736414b6b0 con 0x7f736415b7d0 2024-01-15T18:10:15.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.258+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73641503b0 con 0x7f736415b7d0 2024-01-15T18:10:15.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.259+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73640686f0 con 0x7f736415b7d0 2024-01-15T18:10:15.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.260+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7354013470 con 0x7f736415b7d0 2024-01-15T18:10:15.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.261+0000 7f735a000640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f733c06bfb0 0x7f733c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:15.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.261+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f7354088730 con 0x7f736415b7d0 2024-01-15T18:10:15.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.262+0000 7f7361400640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f733c06bfb0 0x7f733c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:15.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.263+0000 7f7361400640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f733c06bfb0 0x7f733c06e470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f734c036000 tx=0x7f734c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:15.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.265+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73540bde90 con 0x7f736415b7d0 2024-01-15T18:10:15.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:15.527+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]} v 0) v1 -- 0x7f736406b260 con 0x7f736415b7d0 2024-01-15T18:10:16.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.093+0000 7f735a000640 1 -- 172.21.15.2:0/110122287 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]}]=0 v584) v1 ==== 148+0+0 (secure 0 0 0) 0x7f73540170d0 con 0x7f736415b7d0 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.096+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f733c06bfb0 msgr2=0x7f733c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.096+0000 7f7368e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f733c06bfb0 0x7f733c06e470 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f734c036000 tx=0x7f734c03c040 comp rx=0 tx=0).stop 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.096+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 msgr2=0x7f736414d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.096+0000 7f7368e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736414d9f0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f735400e870 tx=0x7f735400ed30 comp rx=0 tx=0).stop 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 shutdown_connections 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f733c06bfb0 0x7f733c06e470 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f736415c4c0 0x7f7364153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f736415c0e0 0x7f736414f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 --2- 172.21.15.2:0/110122287 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f736415b7d0 0x7f736414d9f0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 >> 172.21.15.2:0/110122287 conn(0x7f736406da20 msgr2=0x7f7364071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:16.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 shutdown_connections 2024-01-15T18:10:16.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.097+0000 7f7368e00640 1 -- 172.21.15.2:0/110122287 wait complete. 2024-01-15T18:10:16.122 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:10:16.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.334+0000 7f526fc00640 1 Processor -- start 2024-01-15T18:10:16.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.335+0000 7f526fc00640 1 -- start start 2024-01-15T18:10:16.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.336+0000 7f526fc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5268144c00 0x7f52680cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.336+0000 7f526fc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f526806d000 0x7f5268070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.336+0000 7f526fc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52680ce230 con 0x7f52680cdd70 2024-01-15T18:10:16.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.336+0000 7f526fc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52680ce510 con 0x7f5268144c00 2024-01-15T18:10:16.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.336+0000 7f526fc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52680ce7f0 con 0x7f526806d000 2024-01-15T18:10:16.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.337+0000 7f526dc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f526806d000 0x7f5268070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.337+0000 7f526dc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f526806d000 0x7f5268070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:48624/0 (socket says 172.21.15.2:48624) 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.337+0000 7f526dc00640 1 -- 172.21.15.2:0/2407287635 learned_addr learned my addr 172.21.15.2:0/2407287635 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.337+0000 7f526d200640 1 --2- 172.21.15.2:0/2407287635 >> v2:172.21.15.2:3300/0 conn(0x7f5268144c00 0x7f52680cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.338+0000 7f526dc00640 1 -- 172.21.15.2:0/2407287635 >> v2:172.21.15.2:3300/0 conn(0x7f5268144c00 msgr2=0x7f52680cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.338+0000 7f526dc00640 1 --2- 172.21.15.2:0/2407287635 >> v2:172.21.15.2:3300/0 conn(0x7f5268144c00 0x7f52680cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.338+0000 7f526dc00640 1 -- 172.21.15.2:0/2407287635 >> v1:172.21.15.137:6789/0 conn(0x7f52680cdd70 legacy=0x7f526806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.338+0000 7f526dc00640 1 -- 172.21.15.2:0/2407287635 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52680705f0 con 0x7f526806d000 2024-01-15T18:10:16.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.338+0000 7f5267e00640 1 -- 172.21.15.2:0/2407287635 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971116844 0 0) 0x7f52680ce230 con 0x7f52680cdd70 2024-01-15T18:10:16.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.340+0000 7f526dc00640 1 --2- 172.21.15.2:0/2407287635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f526806d000 0x7f5268070b10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f526815ec80 tx=0x7f526002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55372d125e04dcdb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:16.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.340+0000 7f5267e00640 1 -- 172.21.15.2:0/2407287635 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f526002cb20 con 0x7f526806d000 2024-01-15T18:10:16.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.340+0000 7f5267e00640 1 -- 172.21.15.2:0/2407287635 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f526002cc80 con 0x7f526806d000 2024-01-15T18:10:16.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.340+0000 7f5267e00640 1 -- 172.21.15.2:0/2407287635 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52600343f0 con 0x7f526806d000 2024-01-15T18:10:16.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 -- 172.21.15.2:0/2407287635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f526806d000 msgr2=0x7f5268070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 --2- 172.21.15.2:0/2407287635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f526806d000 0x7f5268070b10 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f526815ec80 tx=0x7f526002bf80 comp rx=0 tx=0).stop 2024-01-15T18:10:16.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 -- 172.21.15.2:0/2407287635 shutdown_connections 2024-01-15T18:10:16.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 --2- 172.21.15.2:0/2407287635 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f526806d000 0x7f5268070b10 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 --2- 172.21.15.2:0/2407287635 >> v2:172.21.15.2:3300/0 conn(0x7f5268144c00 0x7f52680cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 -- 172.21.15.2:0/2407287635 >> 172.21.15.2:0/2407287635 conn(0x7f52680b45f0 msgr2=0x7f52680b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:16.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.341+0000 7f526fc00640 1 -- 172.21.15.2:0/2407287635 shutdown_connections 2024-01-15T18:10:16.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.342+0000 7f526fc00640 1 -- 172.21.15.2:0/2407287635 wait complete. 2024-01-15T18:10:16.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.344+0000 7f526fc00640 1 Processor -- start 2024-01-15T18:10:16.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.344+0000 7f526fc00640 1 -- start start 2024-01-15T18:10:16.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526fc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 0x7f52680afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526fc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680cdd70 0x7f52681411d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526fc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5268144c00 0x7f526812b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526fc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52680b12c0 con 0x7f526806d000 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526fc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5268141f80 con 0x7f5268144c00 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526fc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5268140030 con 0x7f52680cdd70 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 0x7f52680afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 0x7f52680afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43702/0 (socket says 172.21.15.2:43702) 2024-01-15T18:10:16.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.345+0000 7f526d200640 1 -- 172.21.15.2:0/4217477080 learned_addr learned my addr 172.21.15.2:0/4217477080 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:16.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526dc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5268144c00 0x7f526812b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f525fe00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680cdd70 0x7f52681411d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526d200640 1 -- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680cdd70 msgr2=0x7f52681411d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526d200640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680cdd70 0x7f52681411d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526d200640 1 -- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5268144c00 msgr2=0x7f526812b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526d200640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5268144c00 0x7f526812b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526d200640 1 -- 172.21.15.2:0/4217477080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5260009b30 con 0x7f526806d000 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526dc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5268144c00 0x7f526812b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f526d200640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 0x7f52680afff0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f525800e870 tx=0x7f525800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.346+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52580094f0 con 0x7f526806d000 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.347+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525800c940 con 0x7f526806d000 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.347+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f525800ccf0 con 0x7f526806d000 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.347+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526812bc20 con 0x7f526806d000 2024-01-15T18:10:16.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.347+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52680b0920 con 0x7f526806d000 2024-01-15T18:10:16.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.348+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f526806ce20 con 0x7f526806d000 2024-01-15T18:10:16.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.349+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f5258013470 con 0x7f526806d000 2024-01-15T18:10:16.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.350+0000 7f5265600640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bfb0 0x7f524006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:16.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.350+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f5258088730 con 0x7f526806d000 2024-01-15T18:10:16.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.351+0000 7f525fe00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bfb0 0x7f524006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:16.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.352+0000 7f525fe00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bfb0 0x7f524006e470 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5268155240 tx=0x7f5250009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:16.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.355+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52580bde90 con 0x7f526806d000 2024-01-15T18:10:16.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.555+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f52680686f0 con 0x7f526806d000 2024-01-15T18:10:16.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.562+0000 7f5265600640 1 -- 172.21.15.2:0/4217477080 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f52580210c0 con 0x7f526806d000 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bfb0 msgr2=0x7f524006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bfb0 0x7f524006e470 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f5268155240 tx=0x7f5250009040 comp rx=0 tx=0).stop 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 msgr2=0x7f52680afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 0x7f52680afff0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f525800e870 tx=0x7f525800ed30 comp rx=0 tx=0).stop 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 shutdown_connections 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f524006bfb0 0x7f524006e470 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5268144c00 0x7f526812b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52680cdd70 0x7f52681411d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 --2- 172.21.15.2:0/4217477080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f526806d000 0x7f52680afff0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 >> 172.21.15.2:0/4217477080 conn(0x7f52680b45f0 msgr2=0x7f52680716d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 shutdown_connections 2024-01-15T18:10:16.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:16.565+0000 7f526fc00640 1 -- 172.21.15.2:0/4217477080 wait complete. 2024-01-15T18:10:16.590 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:10:16.590 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:10:16.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:10:16.829 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:10:16.829 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:10:16.829 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:16.889 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:10:16.889 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:10:16.889 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:16.945 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:10:16.945 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:10:16.946 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:17.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:10:17.006 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:10:17.006 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:17.062 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:10:17.062 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:10:17.063 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:10:17.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.295+0000 7f5a6f200640 1 Processor -- start 2024-01-15T18:10:17.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.295+0000 7f5a6f200640 1 -- start start 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a70141880 0x7f5a70070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a7015f0e0 0x7f5a700cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6f200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7006dae0 con 0x7f5a7015ec60 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6f200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a70071f20 con 0x7f5a70141880 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7006c6b0 con 0x7f5a7015f0e0 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a70141880 0x7f5a70070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5a70141880 0x7f5a70070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43712/0 (socket says 172.21.15.2:43712) 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6de00640 1 -- 172.21.15.2:0/2433817993 learned_addr learned my addr 172.21.15.2:0/2433817993 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a65200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5a7015ec60 0x7f5a700cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54472/0 (socket says 172.21.15.2:54472) 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6de00640 1 -- 172.21.15.2:0/2433817993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a7015f0e0 msgr2=0x7f5a700cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.297+0000 7f5a6e800640 1 --2- 172.21.15.2:0/2433817993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a7015f0e0 0x7f5a700cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.298+0000 7f5a6de00640 1 --2- 172.21.15.2:0/2433817993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a7015f0e0 0x7f5a700cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.298+0000 7f5a6de00640 1 -- 172.21.15.2:0/2433817993 >> v1:172.21.15.2:6789/0 conn(0x7f5a7015ec60 legacy=0x7f5a700cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:17.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.298+0000 7f5a6de00640 1 -- 172.21.15.2:0/2433817993 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7015f570 con 0x7f5a70141880 2024-01-15T18:10:17.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.298+0000 7f5a6d400640 1 -- 172.21.15.2:0/2433817993 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690818991 0 0) 0x7f5a7006dae0 con 0x7f5a7015ec60 2024-01-15T18:10:17.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.299+0000 7f5a6de00640 1 --2- 172.21.15.2:0/2433817993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a70141880 0x7f5a70070fc0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f5a600098a0 tx=0x7f5a6002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e96f48316c466bb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:17.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.299+0000 7f5a6d400640 1 -- 172.21.15.2:0/2433817993 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60031640 con 0x7f5a70141880 2024-01-15T18:10:17.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.299+0000 7f5a6d400640 1 -- 172.21.15.2:0/2433817993 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a600317a0 con 0x7f5a70141880 2024-01-15T18:10:17.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.299+0000 7f5a6d400640 1 -- 172.21.15.2:0/2433817993 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a60031b30 con 0x7f5a70141880 2024-01-15T18:10:17.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 -- 172.21.15.2:0/2433817993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a70141880 msgr2=0x7f5a70070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:17.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 --2- 172.21.15.2:0/2433817993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a70141880 0x7f5a70070fc0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f5a600098a0 tx=0x7f5a6002b8f0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 -- 172.21.15.2:0/2433817993 shutdown_connections 2024-01-15T18:10:17.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 --2- 172.21.15.2:0/2433817993 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a7015f0e0 0x7f5a700cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 --2- 172.21.15.2:0/2433817993 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a70141880 0x7f5a70070fc0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 -- 172.21.15.2:0/2433817993 >> 172.21.15.2:0/2433817993 conn(0x7f5a700b0840 msgr2=0x7f5a700b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:17.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 -- 172.21.15.2:0/2433817993 shutdown_connections 2024-01-15T18:10:17.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.300+0000 7f5a6f200640 1 -- 172.21.15.2:0/2433817993 wait complete. 2024-01-15T18:10:17.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.302+0000 7f5a6f200640 1 Processor -- start 2024-01-15T18:10:17.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.302+0000 7f5a6f200640 1 -- start start 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a70141880 0x7f5a70140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a7015ec60 0x7f5a70155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 0x7f5a70143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70153d50 con 0x7f5a7015f0e0 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70153fa0 con 0x7f5a7015ec60 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a70154280 con 0x7f5a70141880 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 0x7f5a70143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 0x7f5a70143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43724/0 (socket says 172.21.15.2:43724) 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.303+0000 7f5a6e800640 1 -- 172.21.15.2:0/973187155 learned_addr learned my addr 172.21.15.2:0/973187155 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6e800640 1 -- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a70141880 msgr2=0x7f5a70140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6e800640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a70141880 0x7f5a70140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6e800640 1 -- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a7015ec60 msgr2=0x7f5a70155e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:17.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6e800640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a7015ec60 0x7f5a70155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6e800640 1 -- 172.21.15.2:0/973187155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a600094d0 con 0x7f5a7015f0e0 2024-01-15T18:10:17.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6e800640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 0x7f5a70143730 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800cd70 tx=0x7f5a6800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:17.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68003b90 con 0x7f5a7015f0e0 2024-01-15T18:10:17.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a68003cf0 con 0x7f5a7015f0e0 2024-01-15T18:10:17.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a68004240 con 0x7f5a7015f0e0 2024-01-15T18:10:17.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.304+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a70154560 con 0x7f5a7015f0e0 2024-01-15T18:10:17.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.305+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a70156360 con 0x7f5a7015f0e0 2024-01-15T18:10:17.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.306+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a70154a80 con 0x7f5a7015f0e0 2024-01-15T18:10:17.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.306+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f5a680043a0 con 0x7f5a7015f0e0 2024-01-15T18:10:17.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.308+0000 7f5a66a00640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a4806bfb0 0x7f5a4806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:17.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.308+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f5a68012070 con 0x7f5a7015f0e0 2024-01-15T18:10:17.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.308+0000 7f5a6de00640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a4806bfb0 0x7f5a4806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:17.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.310+0000 7f5a6de00640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a4806bfb0 0x7f5a4806e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5a60036000 tx=0x7f5a60030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:17.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.312+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5a70154a80 con 0x7f5a7015f0e0 2024-01-15T18:10:17.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.543+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5a70154a80 con 0x7f5a7015f0e0 2024-01-15T18:10:17.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.545+0000 7f5a66a00640 1 -- 172.21.15.2:0/973187155 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+3699 (secure 0 0 0) 0x7f5a70154a80 con 0x7f5a7015f0e0 2024-01-15T18:10:17.549 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:17.549 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":49,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5573,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2894571444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2894571444},{"type":"v1","addr":"172.21.15.137:6836","nonce":2894571444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":5980,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/2585344206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2585344206},{"type":"v1","addr":"172.21.15.137:6837","nonce":2585344206}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":41},{"gid":6984,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2342931858","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2342931858},{"type":"v1","addr":"172.21.15.2:6833","nonce":2342931858}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7923,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3426460349","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3426460349},{"type":"v1","addr":"172.21.15.2:6835","nonce":3426460349}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":49}],"filesystems":[]} 2024-01-15T18:10:17.549 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 49 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.547+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a4806bfb0 msgr2=0x7f5a4806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.547+0000 7f5a6f200640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a4806bfb0 0x7f5a4806e470 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f5a60036000 tx=0x7f5a60030000 comp rx=0 tx=0).stop 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.547+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 msgr2=0x7f5a70143730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.547+0000 7f5a6f200640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 0x7f5a70143730 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f5a6800cd70 tx=0x7f5a6800a660 comp rx=0 tx=0).stop 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 shutdown_connections 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5a4806bfb0 0x7f5a4806e470 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a7015f0e0 0x7f5a70143730 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a7015ec60 0x7f5a70155e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 --2- 172.21.15.2:0/973187155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a70141880 0x7f5a70140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:17.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 >> 172.21.15.2:0/973187155 conn(0x7f5a700b0840 msgr2=0x7f5a70153250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:17.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 shutdown_connections 2024-01-15T18:10:17.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:17.548+0000 7f5a6f200640 1 -- 172.21.15.2:0/973187155 wait complete. 2024-01-15T18:10:17.573 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:10:17.573 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:10:17.573 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:10:17.573 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:10:17.573 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:10:17.573 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:10:17.574 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:10:17.574 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:10:17.574 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:10:17.574 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:10:17.574 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:10:17.574 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:10:17.574 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:10:17.574 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:10:17.575 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:10:17.575 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:10:17.591 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:10:17.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:10:17.597 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:10:17.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:10:17.598 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:10:17.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:10:17.639 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:10:17.640 DEBUG:teuthology.parallel:result is None 2024-01-15T18:10:17.645 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:10:17.645 DEBUG:teuthology.parallel:result is None 2024-01-15T18:10:17.651 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:10:17.651 DEBUG:teuthology.parallel:result is None 2024-01-15T18:10:17.805 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T18:10:17.806 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:10:17.807 INFO:tasks.ceph.mds.b.smithi137.stderr:2024-01-15T18:10:17.804+0000 7f21eaa00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-01-15T18:10:17.829 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:10:23.592 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:10:23.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:10:23.601 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:10:23.601 DEBUG:teuthology.parallel:result is None 2024-01-15T18:10:23.601 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:10:23.763 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:10:23.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.802+0000 7f30d9e00640 1 Processor -- start 2024-01-15T18:10:23.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.803+0000 7f30d9e00640 1 -- start start 2024-01-15T18:10:23.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.804+0000 7f30d9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 0x7f30d406c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.804+0000 7f30d9e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d41619b0 con 0x7f30d406cdf0 2024-01-15T18:10:23.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.804+0000 7f30d9e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d4161c90 con 0x7f30d4143850 2024-01-15T18:10:23.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.804+0000 7f30d9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d406d260 con 0x7f30d415e430 2024-01-15T18:10:23.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.804+0000 7f30d3400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f30d406cdf0 0x7f30d4071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47480/0 (socket says 172.21.15.2:47480) 2024-01-15T18:10:23.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.804+0000 7f30d3400640 1 -- 172.21.15.2:0/4023765358 learned_addr learned my addr 172.21.15.2:0/4023765358 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30c9e00640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 0x7f30d406c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173589428 0 0) 0x7f30d41619b0 con 0x7f30d406cdf0 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b4003520 con 0x7f30d406cdf0 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3017021870 0 0) 0x7f30b4003520 con 0x7f30d406cdf0 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 msgr2=0x7f30d406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 0x7f30d406c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 >> v1:172.21.15.137:6789/0 conn(0x7f30d4143850 legacy=0x7f30d4160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d415dc30 con 0x7f30d406cdf0 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f30cc0019f0 con 0x7f30d406cdf0 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 >> v1:172.21.15.2:6789/0 conn(0x7f30d406cdf0 legacy=0x7f30d4071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b40039a0 0x7f30b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.805+0000 7f30d2000640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 0x7f30b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d2000640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30b4004d00 0x7f30b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d406d260 con 0x7f30b4004d00 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b400bb10 con 0x7f30b40039a0 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b400bcd0 con 0x7f30d415e430 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d3400640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30b4004d00 0x7f30b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30c9e00640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 0x7f30b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d2a00640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b40039a0 0x7f30b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d3400640 1 -- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 msgr2=0x7f30b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d3400640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d415e430 0x7f30b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d3400640 1 -- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b40039a0 msgr2=0x7f30b4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d3400640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b40039a0 0x7f30b4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:23.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.806+0000 7f30d3400640 1 -- 172.21.15.2:0/4023765358 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4003520 con 0x7f30b4004d00 2024-01-15T18:10:23.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.807+0000 7f30d3400640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30b4004d00 0x7f30b4005180 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00a760 tx=0x7f30b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41538ded23c8971d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:23.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc034630 con 0x7f30b4004d00 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc02f070 con 0x7f30b4004d00 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d2000640 1 -- 172.21.15.2:0/4023765358 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc034a10 con 0x7f30b4004d00 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d9e00640 1 -- 172.21.15.2:0/4023765358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30b4004d00 msgr2=0x7f30b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d9e00640 1 --2- 172.21.15.2:0/4023765358 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30b4004d00 0x7f30b4005180 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00a760 tx=0x7f30b40057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d3400640 1 -- 172.21.15.2:0/4023765358 reap_dead start 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d9e00640 1 -- 172.21.15.2:0/4023765358 shutdown_connections 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.808+0000 7f30d9e00640 1 -- 172.21.15.2:0/4023765358 >> 172.21.15.2:0/4023765358 conn(0x7f30d40b46d0 msgr2=0x7f30d40b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.809+0000 7f30d9e00640 1 -- 172.21.15.2:0/4023765358 shutdown_connections 2024-01-15T18:10:23.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.809+0000 7f30d9e00640 1 -- 172.21.15.2:0/4023765358 wait complete. 2024-01-15T18:10:23.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.810+0000 7f30d9e00640 1 Processor -- start 2024-01-15T18:10:23.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.811+0000 7f30d9e00640 1 -- start start 2024-01-15T18:10:23.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.811+0000 7f30d9e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b4004d00 0x7f30d41308b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d41602f0 0x7f30d4160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 0x7f30d41f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d9e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4160c90 con 0x7f30d41f2360 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d9e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d40b2ad0 con 0x7f30b4004d00 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d4130df0 con 0x7f30d41602f0 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 0x7f30d41f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 0x7f30d41f27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36020/0 (socket says 172.21.15.2:36020) 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 -- 172.21.15.2:0/981710725 learned_addr learned my addr 172.21.15.2:0/981710725 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d2a00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b4004d00 0x7f30d41308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30c9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d41602f0 0x7f30d4160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 -- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d41602f0 msgr2=0x7f30d4160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d41602f0 0x7f30d4160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 -- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b4004d00 msgr2=0x7f30d41308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b4004d00 0x7f30d41308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.812+0000 7f30d3400640 1 -- 172.21.15.2:0/981710725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d41f3010 con 0x7f30d41f2360 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.813+0000 7f30d3400640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 0x7f30d41f27e0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00a760 tx=0x7f30cc001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:23.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.813+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc034630 con 0x7f30d41f2360 2024-01-15T18:10:23.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.813+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc034790 con 0x7f30d41f2360 2024-01-15T18:10:23.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.813+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc034b00 con 0x7f30d41f2360 2024-01-15T18:10:23.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.813+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d41f32f0 con 0x7f30d41f2360 2024-01-15T18:10:23.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.813+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d41f38b0 con 0x7f30d41f2360 2024-01-15T18:10:23.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.814+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d40686f0 con 0x7f30d41f2360 2024-01-15T18:10:23.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.815+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f30cc031030 con 0x7f30d41f2360 2024-01-15T18:10:23.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.816+0000 7f30cbe00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30ac06bfb0 0x7f30ac06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:23.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.816+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5027+0+0 (secure 0 0 0) 0x7f30cc0aac30 con 0x7f30d41f2360 2024-01-15T18:10:23.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.817+0000 7f30d2a00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30ac06bfb0 0x7f30ac06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:23.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.818+0000 7f30d2a00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30ac06bfb0 0x7f30ac06e470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f30c4007fd0 tx=0x7f30c4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:23.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:23.821+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30cc0772f0 con 0x7f30d41f2360 2024-01-15T18:10:24.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.025+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f30d40686f0 con 0x7f30d41f2360 2024-01-15T18:10:24.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.793+0000 7f30cbe00640 1 -- 172.21.15.2:0/981710725 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v85) v1 ==== 97+0+0 (secure 0 0 0) 0x7f30cc07a910 con 0x7f30d41f2360 2024-01-15T18:10:24.797 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30ac06bfb0 msgr2=0x7f30ac06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30ac06bfb0 0x7f30ac06e470 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f30c4007fd0 tx=0x7f30c4009040 comp rx=0 tx=0).stop 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 msgr2=0x7f30d41f27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 0x7f30d41f27e0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00a760 tx=0x7f30cc001cb0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 shutdown_connections 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30ac06bfb0 0x7f30ac06e470 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30d41f2360 0x7f30d41f27e0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30d41602f0 0x7f30d4160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 --2- 172.21.15.2:0/981710725 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30b4004d00 0x7f30d41308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:24.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.797+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 >> 172.21.15.2:0/981710725 conn(0x7f30d40b46d0 msgr2=0x7f30d4142a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:24.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.798+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 shutdown_connections 2024-01-15T18:10:24.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:24.798+0000 7f30d9e00640 1 -- 172.21.15.2:0/981710725 wait complete. 2024-01-15T18:10:24.823 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:10:25.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.031+0000 7f130c800640 1 Processor -- start 2024-01-15T18:10:25.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.032+0000 7f130c800640 1 -- start start 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.033+0000 7f130c800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f130415d360 0x7f13040b8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.033+0000 7f130c800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f13040cdeb0 0x7f13040ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.033+0000 7f130c800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13040d1bd0 con 0x7f1304160f10 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.033+0000 7f130c800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040ce800 con 0x7f130415d360 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.033+0000 7f130c800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13040cf8e0 con 0x7f13040cdeb0 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f130415d360 0x7f13040b8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f130415d360 0x7f13040b8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36028/0 (socket says 172.21.15.2:36028) 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 -- 172.21.15.2:0/3921535585 learned_addr learned my addr 172.21.15.2:0/3921535585 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f130a800640 1 --2- 172.21.15.2:0/3921535585 >> v2:172.21.15.137:3300/0 conn(0x7f13040cdeb0 0x7f13040ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 -- 172.21.15.2:0/3921535585 >> v2:172.21.15.137:3300/0 conn(0x7f13040cdeb0 msgr2=0x7f13040ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 --2- 172.21.15.2:0/3921535585 >> v2:172.21.15.137:3300/0 conn(0x7f13040cdeb0 0x7f13040ce2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 -- 172.21.15.2:0/3921535585 >> v1:172.21.15.137:6789/0 conn(0x7f1304160f10 legacy=0x7f13040d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.034+0000 7f1309e00640 1 -- 172.21.15.2:0/3921535585 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1304161680 con 0x7f130415d360 2024-01-15T18:10:25.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.035+0000 7f1309e00640 1 --2- 172.21.15.2:0/3921535585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b8d20 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f13040d30c0 tx=0x7f12f402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c658875ba2de4ee4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.036+0000 7f1308a00640 1 -- 172.21.15.2:0/3921535585 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f402ca80 con 0x7f130415d360 2024-01-15T18:10:25.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.036+0000 7f1308a00640 1 -- 172.21.15.2:0/3921535585 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f402cbe0 con 0x7f130415d360 2024-01-15T18:10:25.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.036+0000 7f1308a00640 1 -- 172.21.15.2:0/3921535585 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f40343f0 con 0x7f130415d360 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.036+0000 7f130c800640 1 -- 172.21.15.2:0/3921535585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 msgr2=0x7f13040b8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.036+0000 7f130c800640 1 --2- 172.21.15.2:0/3921535585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b8d20 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f13040d30c0 tx=0x7f12f402bee0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.037+0000 7f130c800640 1 -- 172.21.15.2:0/3921535585 shutdown_connections 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.037+0000 7f130c800640 1 --2- 172.21.15.2:0/3921535585 >> v2:172.21.15.137:3300/0 conn(0x7f13040cdeb0 0x7f13040ce2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.037+0000 7f130c800640 1 --2- 172.21.15.2:0/3921535585 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b8d20 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.037+0000 7f130c800640 1 -- 172.21.15.2:0/3921535585 >> 172.21.15.2:0/3921535585 conn(0x7f1304072000 msgr2=0x7f1304072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.037+0000 7f130c800640 1 -- 172.21.15.2:0/3921535585 shutdown_connections 2024-01-15T18:10:25.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.037+0000 7f130c800640 1 -- 172.21.15.2:0/3921535585 wait complete. 2024-01-15T18:10:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.039+0000 7f130c800640 1 Processor -- start 2024-01-15T18:10:25.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.039+0000 7f130c800640 1 -- start start 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130c800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cdeb0 0x7f13041513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130c800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130c800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1304160f10 0x7f13040b7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130c800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040cab50 con 0x7f130415d360 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130c800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13040c8550 con 0x7f1304160f10 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130c800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040c8830 con 0x7f13040cdeb0 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36042/0 (socket says 172.21.15.2:36042) 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 -- 172.21.15.2:0/4110231858 learned_addr learned my addr 172.21.15.2:0/4110231858 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309e00640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cdeb0 0x7f13041513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f130a800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1304160f10 0x7f13040b7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 -- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cdeb0 msgr2=0x7f13041513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cdeb0 0x7f13041513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 -- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1304160f10 msgr2=0x7f13040b7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.041+0000 7f1309400640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1304160f10 0x7f13040b7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f1309400640 1 -- 172.21.15.2:0/4110231858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f4009b30 con 0x7f130415d360 2024-01-15T18:10:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f1309400640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b6ce0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c410 tx=0x7f12ec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec0035f0 con 0x7f130415d360 2024-01-15T18:10:25.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec003750 con 0x7f130415d360 2024-01-15T18:10:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ec003b40 con 0x7f130415d360 2024-01-15T18:10:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13040c8b10 con 0x7f130415d360 2024-01-15T18:10:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.042+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13040c90d0 con 0x7f130415d360 2024-01-15T18:10:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.044+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1304073c10 con 0x7f130415d360 2024-01-15T18:10:25.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.045+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f12ec003ca0 con 0x7f130415d360 2024-01-15T18:10:25.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.045+0000 7f1302000640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f12dc06bfb0 0x7f12dc06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.046+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f12ec0878e0 con 0x7f130415d360 2024-01-15T18:10:25.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.046+0000 7f1309e00640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f12dc06bfb0 0x7f12dc06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.047+0000 7f1309e00640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f12dc06bfb0 0x7f12dc06e470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f12f4035000 tx=0x7f12f4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.050+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12ec054110 con 0x7f130415d360 2024-01-15T18:10:25.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.313+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f1304160a60 con 0x7f130415d360 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.315+0000 7f1302000640 1 -- 172.21.15.2:0/4110231858 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f12ec017090 con 0x7f130415d360 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T18:10:25.319 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.320 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.321 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T18:10:25.322 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.323 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.324 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.325 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.326 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.326 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T18:10:25.327 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.328 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow r", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.329 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T18:10:25.331 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.332 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T18:10:25.333 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:10:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f12dc06bfb0 msgr2=0x7f12dc06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f12dc06bfb0 0x7f12dc06e470 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f12f4035000 tx=0x7f12f4030000 comp rx=0 tx=0).stop 2024-01-15T18:10:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 msgr2=0x7f13040b6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b6ce0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f12ec00c410 tx=0x7f12ec00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 shutdown_connections 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f12dc06bfb0 0x7f12dc06e470 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1304160f10 0x7f13040b7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f130415d360 0x7f13040b6ce0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 --2- 172.21.15.2:0/4110231858 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f13040cdeb0 0x7f13041513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 >> 172.21.15.2:0/4110231858 conn(0x7f1304072000 msgr2=0x7f13040b32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 shutdown_connections 2024-01-15T18:10:25.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.318+0000 7f130c800640 1 -- 172.21.15.2:0/4110231858 wait complete. 2024-01-15T18:10:25.341 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:25.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.541+0000 7fe5b7200640 1 Processor -- start 2024-01-15T18:10:25.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.541+0000 7fe5b7200640 1 -- start start 2024-01-15T18:10:25.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.542+0000 7fe5b7200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5b015ec60 0x7fe5b015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.542+0000 7fe5b7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b006cad0 0x7fe5b006cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.542+0000 7fe5b7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5b00716d0 con 0x7fe5b0141880 2024-01-15T18:10:25.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.542+0000 7fe5b7200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5b0071f20 con 0x7fe5b015ec60 2024-01-15T18:10:25.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.542+0000 7fe5b7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b006c6b0 con 0x7fe5b006cad0 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5b015ec60 0x7fe5b015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5b015ec60 0x7fe5b015f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36052/0 (socket says 172.21.15.2:36052) 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 -- 172.21.15.2:0/2502467789 learned_addr learned my addr 172.21.15.2:0/2502467789 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b6800640 1 --2- 172.21.15.2:0/2502467789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b006cad0 0x7fe5b006cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 -- 172.21.15.2:0/2502467789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b006cad0 msgr2=0x7fe5b006cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 --2- 172.21.15.2:0/2502467789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b006cad0 0x7fe5b006cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 -- 172.21.15.2:0/2502467789 >> v1:172.21.15.137:6789/0 conn(0x7fe5b0141880 legacy=0x7fe5b0070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.543+0000 7fe5b5400640 1 -- 172.21.15.2:0/2502467789 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5b0071960 con 0x7fe5b015ec60 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.544+0000 7fe5b5400640 1 --2- 172.21.15.2:0/2502467789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b015ec60 0x7fe5b015f0e0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fe5980098a0 tx=0x7fe59802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61193a20fa0b1abc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b4a00640 1 -- 172.21.15.2:0/2502467789 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598031640 con 0x7fe5b015ec60 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b4a00640 1 -- 172.21.15.2:0/2502467789 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5980317a0 con 0x7fe5b015ec60 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b4a00640 1 -- 172.21.15.2:0/2502467789 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598031b30 con 0x7fe5b015ec60 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 -- 172.21.15.2:0/2502467789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b015ec60 msgr2=0x7fe5b015f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 --2- 172.21.15.2:0/2502467789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b015ec60 0x7fe5b015f0e0 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7fe5980098a0 tx=0x7fe59802b8f0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 -- 172.21.15.2:0/2502467789 shutdown_connections 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 --2- 172.21.15.2:0/2502467789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b006cad0 0x7fe5b006cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 --2- 172.21.15.2:0/2502467789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b015ec60 0x7fe5b015f0e0 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 -- 172.21.15.2:0/2502467789 >> 172.21.15.2:0/2502467789 conn(0x7fe5b00b0840 msgr2=0x7fe5b00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 -- 172.21.15.2:0/2502467789 shutdown_connections 2024-01-15T18:10:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.545+0000 7fe5b7200640 1 -- 172.21.15.2:0/2502467789 wait complete. 2024-01-15T18:10:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.547+0000 7fe5b7200640 1 Processor -- start 2024-01-15T18:10:25.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.547+0000 7fe5b7200640 1 -- start start 2024-01-15T18:10:25.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.548+0000 7fe5b7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 0x7fe5b0138760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.548+0000 7fe5b7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b0141880 0x7fe5b015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.548+0000 7fe5b7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b015ec60 0x7fe5b0157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.548+0000 7fe5b7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b0158af0 con 0x7fe5b006cad0 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.548+0000 7fe5b7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b015a540 con 0x7fe5b015ec60 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.548+0000 7fe5b7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b00ad7f0 con 0x7fe5b0141880 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 0x7fe5b0138760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 0x7fe5b0138760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36060/0 (socket says 172.21.15.2:36060) 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 -- 172.21.15.2:0/1504956909 learned_addr learned my addr 172.21.15.2:0/1504956909 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5400640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b0141880 0x7fe5b015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b6800640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b015ec60 0x7fe5b0157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 -- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b0141880 msgr2=0x7fe5b015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b0141880 0x7fe5b015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 -- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b015ec60 msgr2=0x7fe5b0157ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b015ec60 0x7fe5b0157ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.549+0000 7fe5b5e00640 1 -- 172.21.15.2:0/1504956909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5980094d0 con 0x7fe5b006cad0 2024-01-15T18:10:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.550+0000 7fe5b5e00640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 0x7fe5b0138760 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000c870 tx=0x7fe5a000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.550+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0003b70 con 0x7fe5b006cad0 2024-01-15T18:10:25.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.550+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0003cf0 con 0x7fe5b006cad0 2024-01-15T18:10:25.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.550+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a000aa90 con 0x7fe5b006cad0 2024-01-15T18:10:25.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.550+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b00adad0 con 0x7fe5b006cad0 2024-01-15T18:10:25.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.550+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b00ae090 con 0x7fe5b006cad0 2024-01-15T18:10:25.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.551+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b0074770 con 0x7fe5b006cad0 2024-01-15T18:10:25.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.552+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fe5a000abf0 con 0x7fe5b006cad0 2024-01-15T18:10:25.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.553+0000 7fe5ae000640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe58806bfb0 0x7fe58806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.553+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fe5a00884e0 con 0x7fe5b006cad0 2024-01-15T18:10:25.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.554+0000 7fe5b5400640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe58806bfb0 0x7fe58806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.555+0000 7fe5b5400640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe58806bfb0 0x7fe58806e470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe598036000 tx=0x7fe59803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.557+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5a008e030 con 0x7fe5b006cad0 2024-01-15T18:10:25.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.747+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5b00b22d0 con 0x7fe5b006cad0 2024-01-15T18:10:25.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.749+0000 7fe5ae000640 1 -- 172.21.15.2:0/1504956909 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+12929 (secure 0 0 0) 0x7fe5a0003e60 con 0x7fe5b006cad0 2024-01-15T18:10:25.754 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:25.754 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":85,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:24.778839+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.751+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe58806bfb0 msgr2=0x7fe58806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.751+0000 7fe5b7200640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe58806bfb0 0x7fe58806e470 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe598036000 tx=0x7fe59803c040 comp rx=0 tx=0).stop 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 msgr2=0x7fe5b0138760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 0x7fe5b0138760 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000c870 tx=0x7fe5a000cd30 comp rx=0 tx=0).stop 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 shutdown_connections 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe58806bfb0 0x7fe58806e470 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5b015ec60 0x7fe5b0157ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5b0141880 0x7fe5b015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 --2- 172.21.15.2:0/1504956909 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5b006cad0 0x7fe5b0138760 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 >> 172.21.15.2:0/1504956909 conn(0x7fe5b00b0840 msgr2=0x7fe5b0153310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:25.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 shutdown_connections 2024-01-15T18:10:25.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.752+0000 7fe5b7200640 1 -- 172.21.15.2:0/1504956909 wait complete. 2024-01-15T18:10:25.775 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:10:25.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:10:25.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.984+0000 7f087e800640 1 Processor -- start 2024-01-15T18:10:25.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.985+0000 7f087e800640 1 -- start start 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.986+0000 7f087e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f087815f260 0x7f087806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.986+0000 7f087e800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f08780720d0 0x7f087806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.986+0000 7f087e800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0878071090 con 0x7f0878143d20 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.986+0000 7f087e800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0878071430 con 0x7f087815f260 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.986+0000 7f087e800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0878144100 con 0x7f08780720d0 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.986+0000 7f087ca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f087815f260 0x7f087806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087d400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0878143d20 0x7f087815eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47490/0 (socket says 172.21.15.2:47490) 2024-01-15T18:10:25.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087d400640 1 -- 172.21.15.2:0/137350615 learned_addr learned my addr 172.21.15.2:0/137350615 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:25.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087de00640 1 --2- 172.21.15.2:0/137350615 >> v2:172.21.15.137:3300/0 conn(0x7f08780720d0 0x7f087806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087ca00640 1 -- 172.21.15.2:0/137350615 >> v2:172.21.15.137:3300/0 conn(0x7f08780720d0 msgr2=0x7f087806c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:25.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087ca00640 1 --2- 172.21.15.2:0/137350615 >> v2:172.21.15.137:3300/0 conn(0x7f08780720d0 0x7f087806c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087ca00640 1 -- 172.21.15.2:0/137350615 >> v1:172.21.15.2:6789/0 conn(0x7f0878143d20 legacy=0x7f087815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.987+0000 7f087ca00640 1 -- 172.21.15.2:0/137350615 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0878144560 con 0x7f087815f260 2024-01-15T18:10:25.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.988+0000 7f087ca00640 1 --2- 172.21.15.2:0/137350615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087806c150 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f08780cfc70 tx=0x7f086002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e33be6595bf19 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.988+0000 7f0877e00640 1 -- 172.21.15.2:0/137350615 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086002e070 con 0x7f087815f260 2024-01-15T18:10:25.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f0877e00640 1 -- 172.21.15.2:0/137350615 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08600315a0 con 0x7f087815f260 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f0877e00640 1 -- 172.21.15.2:0/137350615 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860031930 con 0x7f087815f260 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 -- 172.21.15.2:0/137350615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 msgr2=0x7f087806c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 --2- 172.21.15.2:0/137350615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087806c150 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f08780cfc70 tx=0x7f086002b840 comp rx=0 tx=0).stop 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 -- 172.21.15.2:0/137350615 shutdown_connections 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 --2- 172.21.15.2:0/137350615 >> v2:172.21.15.137:3300/0 conn(0x7f08780720d0 0x7f087806c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 --2- 172.21.15.2:0/137350615 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087806c150 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 -- 172.21.15.2:0/137350615 >> 172.21.15.2:0/137350615 conn(0x7f08780b0840 msgr2=0x7f08780b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 -- 172.21.15.2:0/137350615 shutdown_connections 2024-01-15T18:10:25.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.989+0000 7f087e800640 1 -- 172.21.15.2:0/137350615 wait complete. 2024-01-15T18:10:25.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.991+0000 7f087e800640 1 Processor -- start 2024-01-15T18:10:25.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.991+0000 7f087e800640 1 -- start start 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08780720d0 0x7f08781563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0878143d20 0x7f087813f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087813d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087813c7d0 con 0x7f087815f260 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0878159bd0 con 0x7f08780720d0 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f087813e970 con 0x7f0878143d20 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087813d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087813d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36076/0 (socket says 172.21.15.2:36076) 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.992+0000 7f087de00640 1 -- 172.21.15.2:0/358906869 learned_addr learned my addr 172.21.15.2:0/358906869 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087d400640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08780720d0 0x7f08781563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087de00640 1 -- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0878143d20 msgr2=0x7f087813f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087de00640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0878143d20 0x7f087813f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087de00640 1 -- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08780720d0 msgr2=0x7f08781563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:25.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087de00640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08780720d0 0x7f08781563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087de00640 1 -- 172.21.15.2:0/358906869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860009400 con 0x7f087815f260 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087d400640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08780720d0 0x7f08781563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087de00640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087813d240 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0870007e40 tx=0x7f087000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0870014620 con 0x7f087815f260 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f087000ee90 con 0x7f087815f260 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08700149e0 con 0x7f087815f260 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.993+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087813ec50 con 0x7f087815f260 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.994+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087813cd20 con 0x7f087815f260 2024-01-15T18:10:25.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.995+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0878070b10 con 0x7f087815f260 2024-01-15T18:10:26.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.996+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f0870014b40 con 0x7f087815f260 2024-01-15T18:10:26.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.996+0000 7f0875600640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f085006bfb0 0x7f085006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.997+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f0870089010 con 0x7f087815f260 2024-01-15T18:10:26.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.997+0000 7f087d400640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f085006bfb0 0x7f085006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:25.998+0000 7f087d400640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f085006bfb0 0x7f085006e470 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0868007fd0 tx=0x7f0868009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.001+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08700558d0 con 0x7f087815f260 2024-01-15T18:10:26.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.190+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f087813cf10 con 0x7f087815f260 2024-01-15T18:10:26.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.191+0000 7f0875600640 1 -- 172.21.15.2:0/358906869 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0870058ef0 con 0x7f087815f260 2024-01-15T18:10:26.195 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:10:26.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f085006bfb0 msgr2=0x7f085006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f085006bfb0 0x7f085006e470 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f0868007fd0 tx=0x7f0868009040 comp rx=0 tx=0).stop 2024-01-15T18:10:26.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 msgr2=0x7f087813d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087813d240 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f0870007e40 tx=0x7f087000e4b0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 shutdown_connections 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f085006bfb0 0x7f085006e470 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f087815f260 0x7f087813d240 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0878143d20 0x7f087813f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 --2- 172.21.15.2:0/358906869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f08780720d0 0x7f08781563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 >> 172.21.15.2:0/358906869 conn(0x7f08780b0840 msgr2=0x7f087806c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 shutdown_connections 2024-01-15T18:10:26.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.194+0000 7f087e800640 1 -- 172.21.15.2:0/358906869 wait complete. 2024-01-15T18:10:26.217 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:10:26.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.430+0000 7f9084600640 1 Processor -- start 2024-01-15T18:10:26.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.430+0000 7f9084600640 1 -- start start 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.432+0000 7f9084600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c1469f0 0x7f907c071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.432+0000 7f9084600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f907c146dd0 0x7f907c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.432+0000 7f9084600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907c06cdf0 con 0x7f907c06c9b0 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.432+0000 7f9084600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907c0ac870 con 0x7f907c146dd0 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.432+0000 7f9084600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c15d970 con 0x7f907c1469f0 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.432+0000 7f9081200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f907c146dd0 0x7f907c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f907c146dd0 0x7f907c06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36080/0 (socket says 172.21.15.2:36080) 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081200640 1 -- 172.21.15.2:0/443637999 learned_addr learned my addr 172.21.15.2:0/443637999 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:26.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081c00640 1 --2- 172.21.15.2:0/443637999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c1469f0 0x7f907c071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081200640 1 -- 172.21.15.2:0/443637999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c1469f0 msgr2=0x7f907c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:26.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081200640 1 --2- 172.21.15.2:0/443637999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c1469f0 0x7f907c071df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081200640 1 -- 172.21.15.2:0/443637999 >> v1:172.21.15.137:6789/0 conn(0x7f907c06c9b0 legacy=0x7f907c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:26.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.433+0000 7f9081200640 1 -- 172.21.15.2:0/443637999 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907c15dc30 con 0x7f907c146dd0 2024-01-15T18:10:26.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.434+0000 7f9081200640 1 --2- 172.21.15.2:0/443637999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c146dd0 0x7f907c06c470 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f9064009870 tx=0x7f906402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d655fc9df5234c04 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.434+0000 7f907be00640 1 -- 172.21.15.2:0/443637999 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064031640 con 0x7f907c146dd0 2024-01-15T18:10:26.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.434+0000 7f907be00640 1 -- 172.21.15.2:0/443637999 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90640317a0 con 0x7f907c146dd0 2024-01-15T18:10:26.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.434+0000 7f907be00640 1 -- 172.21.15.2:0/443637999 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064031b30 con 0x7f907c146dd0 2024-01-15T18:10:26.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 -- 172.21.15.2:0/443637999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c146dd0 msgr2=0x7f907c06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 --2- 172.21.15.2:0/443637999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c146dd0 0x7f907c06c470 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f9064009870 tx=0x7f906402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 -- 172.21.15.2:0/443637999 shutdown_connections 2024-01-15T18:10:26.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 --2- 172.21.15.2:0/443637999 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c146dd0 0x7f907c06c470 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 --2- 172.21.15.2:0/443637999 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c1469f0 0x7f907c071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 -- 172.21.15.2:0/443637999 >> 172.21.15.2:0/443637999 conn(0x7f907c06da20 msgr2=0x7f907c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:26.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 -- 172.21.15.2:0/443637999 shutdown_connections 2024-01-15T18:10:26.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.435+0000 7f9084600640 1 -- 172.21.15.2:0/443637999 wait complete. 2024-01-15T18:10:26.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.437+0000 7f9084600640 1 Processor -- start 2024-01-15T18:10:26.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.437+0000 7f9084600640 1 -- start start 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9084600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c06c9b0 0x7f907c13b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9084600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907c1469f0 0x7f907c145350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9084600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 0x7f907c14ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9084600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c1402a0 con 0x7f907c13edd0 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9084600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907c138c50 con 0x7f907c1469f0 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9084600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f907c13f620 con 0x7f907c06c9b0 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.438+0000 7f9082600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 0x7f907c14ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 0x7f907c14ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36090/0 (socket says 172.21.15.2:36090) 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 -- 172.21.15.2:0/894962751 learned_addr learned my addr 172.21.15.2:0/894962751 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9081200640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907c1469f0 0x7f907c145350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9081c00640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c06c9b0 0x7f907c13b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 -- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c06c9b0 msgr2=0x7f907c13b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c06c9b0 0x7f907c13b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 -- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907c1469f0 msgr2=0x7f907c145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907c1469f0 0x7f907c145350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 -- 172.21.15.2:0/894962751 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90640094d0 con 0x7f907c13edd0 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9081200640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907c1469f0 0x7f907c145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.439+0000 7f9082600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 0x7f907c14ec50 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f907400c780 tx=0x7f907400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.440+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9074004b60 con 0x7f907c13edd0 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.440+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f907400ce90 con 0x7f907c13edd0 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.440+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f907400a850 con 0x7f907c13edd0 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.440+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907c13f900 con 0x7f907c13edd0 2024-01-15T18:10:26.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.440+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907c13fec0 con 0x7f907c13edd0 2024-01-15T18:10:26.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.441+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907c1403e0 con 0x7f907c13edd0 2024-01-15T18:10:26.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.442+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f907400a9b0 con 0x7f907c13edd0 2024-01-15T18:10:26.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.443+0000 7f9079600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905406bfe0 0x7f905406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.444+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4941+0+0 (secure 0 0 0) 0x7f90740886d0 con 0x7f907c13edd0 2024-01-15T18:10:26.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.444+0000 7f9081c00640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905406bfe0 0x7f905406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.445+0000 7f9081c00640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905406bfe0 0x7f905406e4a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f906c006fd0 tx=0x7f906c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.448+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90740bde90 con 0x7f907c13edd0 2024-01-15T18:10:26.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.644+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f907c0686f0 con 0x7f907c13edd0 2024-01-15T18:10:26.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.645+0000 7f9079600640 1 -- 172.21.15.2:0/894962751 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v85) v1 ==== 84+0+65 (secure 0 0 0) 0x7f907c1403e0 con 0x7f907c13edd0 2024-01-15T18:10:26.649 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:26.650 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:10:26.650 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:26.650 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T18:10:26.650 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T18:10:26.650 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:26.650 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905406bfe0 msgr2=0x7f905406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905406bfe0 0x7f905406e4a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f906c006fd0 tx=0x7f906c008040 comp rx=0 tx=0).stop 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 msgr2=0x7f907c14ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 0x7f907c14ec50 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f907400c780 tx=0x7f907400cc40 comp rx=0 tx=0).stop 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 shutdown_connections 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f905406bfe0 0x7f905406e4a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f907c13edd0 0x7f907c14ec50 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f907c1469f0 0x7f907c145350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 --2- 172.21.15.2:0/894962751 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f907c06c9b0 0x7f907c13b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 >> 172.21.15.2:0/894962751 conn(0x7f907c06da20 msgr2=0x7f907c161480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 shutdown_connections 2024-01-15T18:10:26.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.648+0000 7f9084600640 1 -- 172.21.15.2:0/894962751 wait complete. 2024-01-15T18:10:26.670 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:10:26.671 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:10:26.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.878+0000 7fcef3e00640 1 Processor -- start 2024-01-15T18:10:26.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.878+0000 7fcef3e00640 1 -- start start 2024-01-15T18:10:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fcef406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef3e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef4071090 con 0x7fcef40720d0 2024-01-15T18:10:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef3e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef4144560 con 0x7fcef4143d20 2024-01-15T18:10:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4071430 con 0x7fcef415f260 2024-01-15T18:10:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef3400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcef40720d0 0x7fcef406c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47498/0 (socket says 172.21.15.2:47498) 2024-01-15T18:10:26.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef3400640 1 -- 172.21.15.2:0/1970580639 learned_addr learned my addr 172.21.15.2:0/1970580639 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:26.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.880+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161073389 0 0) 0x7fcef4071090 con 0x7fcef40720d0 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced4003520 con 0x7fcef40720d0 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef2000640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fcef406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3148448231 0 0) 0x7fced4003520 con 0x7fcef40720d0 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 msgr2=0x7fcef406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fcef406c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 >> v1:172.21.15.137:6789/0 conn(0x7fcef4143d20 legacy=0x7fcef415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef4144280 con 0x7fcef40720d0 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fceec0019f0 con 0x7fcef40720d0 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef2000640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fcef406c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 >> v1:172.21.15.2:6789/0 conn(0x7fcef40720d0 legacy=0x7fcef406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced40039a0 0x7fced4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fced40047c0 0x7fced4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fced400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef4071430 con 0x7fced40039a0 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4005580 con 0x7fced40047c0 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced400bec0 con 0x7fcef415f260 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.881+0000 7fcef2a00640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced40039a0 0x7fced4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef3400640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fced400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2000640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fced40047c0 0x7fced4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2a00640 1 -- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 msgr2=0x7fced400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2a00640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fced400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2a00640 1 -- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fced40047c0 msgr2=0x7fced4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2a00640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fced40047c0 0x7fced4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2a00640 1 -- 172.21.15.2:0/1970580639 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced4003520 con 0x7fced40039a0 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef3400640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcef415f260 0x7fced400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.882+0000 7fcef2000640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fced40047c0 0x7fced4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.883+0000 7fcef2a00640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced40039a0 0x7fced4003dc0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcee4001750 tx=0x7fced40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbbbfd47f8646 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.883+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee40052d0 con 0x7fced40039a0 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.883+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee402e050 con 0x7fced40039a0 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.883+0000 7fcef1600640 1 -- 172.21.15.2:0/1970580639 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee40035f0 con 0x7fced40039a0 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3e00640 1 -- 172.21.15.2:0/1970580639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced40039a0 msgr2=0x7fced4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3e00640 1 --2- 172.21.15.2:0/1970580639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fced40039a0 0x7fced4003dc0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fcee4001750 tx=0x7fced40057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3400640 1 -- 172.21.15.2:0/1970580639 reap_dead start 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3e00640 1 -- 172.21.15.2:0/1970580639 shutdown_connections 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3e00640 1 -- 172.21.15.2:0/1970580639 >> 172.21.15.2:0/1970580639 conn(0x7fcef40b0840 msgr2=0x7fcef40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3e00640 1 -- 172.21.15.2:0/1970580639 shutdown_connections 2024-01-15T18:10:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.884+0000 7fcef3e00640 1 -- 172.21.15.2:0/1970580639 wait complete. 2024-01-15T18:10:26.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.886+0000 7fcef3e00640 1 Processor -- start 2024-01-15T18:10:26.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.886+0000 7fcef3e00640 1 -- start start 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fced40039a0 0x7fcef4131160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 0x7fcef40ad700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcef40b2920 0x7fcef40b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef40c3ab0 con 0x7fcef40ad280 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef40addd0 con 0x7fcef40b2920 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef40ae0b0 con 0x7fced40039a0 2024-01-15T18:10:26.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.887+0000 7fcef2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 0x7fcef40ad700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 0x7fcef40ad700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36118/0 (socket says 172.21.15.2:36118) 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 -- 172.21.15.2:0/282778116 learned_addr learned my addr 172.21.15.2:0/282778116 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2a00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fced40039a0 0x7fcef4131160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 -- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fced40039a0 msgr2=0x7fcef4131160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef3400640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcef40b2920 0x7fcef40b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fced40039a0 0x7fcef4131160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 -- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcef40b2920 msgr2=0x7fcef40b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcef40b2920 0x7fcef40b2da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 -- 172.21.15.2:0/282778116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef40c3d70 con 0x7fcef40ad280 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2a00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fced40039a0 0x7fcef4131160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.888+0000 7fcef2000640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 0x7fcef40ad700 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00f550 tx=0x7fcedc00fa10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.889+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc014890 con 0x7fcef40ad280 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.889+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcedc00c030 con 0x7fcef40ad280 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.889+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcedc014bf0 con 0x7fcef40ad280 2024-01-15T18:10:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.889+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef40c4050 con 0x7fcef40ad280 2024-01-15T18:10:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.889+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef41316a0 con 0x7fcef40ad280 2024-01-15T18:10:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.890+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef4074770 con 0x7fcef40ad280 2024-01-15T18:10:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.891+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fcedc014d50 con 0x7fcef40ad280 2024-01-15T18:10:26.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.892+0000 7fceeb400640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcecc06bfb0 0x7fcecc06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:26.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.892+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 4941+0+0 (secure 0 0 0) 0x7fcedc088870 con 0x7fcef40ad280 2024-01-15T18:10:26.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.892+0000 7fcef2a00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcecc06bfb0 0x7fcecc06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:26.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.894+0000 7fcef2a00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcecc06bfb0 0x7fcecc06e470 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcee4033000 tx=0x7fcee4009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:26.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:26.896+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcedc08e030 con 0x7fcef40ad280 2024-01-15T18:10:27.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.089+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcef40707f0 con 0x7fcef40ad280 2024-01-15T18:10:27.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.821+0000 7fceeb400640 1 -- 172.21.15.2:0/282778116 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v86) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcedc058bc0 con 0x7fcef40ad280 2024-01-15T18:10:27.825 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T18:10:27.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.824+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcecc06bfb0 msgr2=0x7fcecc06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.824+0000 7fcef3e00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcecc06bfb0 0x7fcecc06e470 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fcee4033000 tx=0x7fcee4009030 comp rx=0 tx=0).stop 2024-01-15T18:10:27.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 msgr2=0x7fcef40ad700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:27.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 0x7fcef40ad700 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fcedc00f550 tx=0x7fcedc00fa10 comp rx=0 tx=0).stop 2024-01-15T18:10:27.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 shutdown_connections 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcecc06bfb0 0x7fcecc06e470 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcef40b2920 0x7fcef40b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcef40ad280 0x7fcef40ad700 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 --2- 172.21.15.2:0/282778116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fced40039a0 0x7fcef4131160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 >> 172.21.15.2:0/282778116 conn(0x7fcef40b0840 msgr2=0x7fcef40ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 shutdown_connections 2024-01-15T18:10:27.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:27.825+0000 7fcef3e00640 1 -- 172.21.15.2:0/282778116 wait complete. 2024-01-15T18:10:27.856 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:10:28.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.063+0000 7fcdf1a00640 1 Processor -- start 2024-01-15T18:10:28.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.064+0000 7fcdf1a00640 1 -- start start 2024-01-15T18:10:28.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.065+0000 7fcdf1a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcdec15ae80 0x7fcdec0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:28.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.065+0000 7fcdf1a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec06c8f0 0x7fcdec06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:28.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.065+0000 7fcdf1a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec06d290 con 0x7fcdec0b2430 2024-01-15T18:10:28.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.065+0000 7fcdf1a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdec15e840 con 0x7fcdec15ae80 2024-01-15T18:10:28.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.065+0000 7fcdf1a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec15dbc0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.066+0000 7fcdeaa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcdec15ae80 0x7fcdec0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.066+0000 7fcdea000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec06c8f0 0x7fcdec06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.066+0000 7fcdeaa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcdec15ae80 0x7fcdec0acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:37612/0 (socket says 172.21.15.2:37612) 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.066+0000 7fcdeaa00640 1 -- 172.21.15.2:0/1421447553 learned_addr learned my addr 172.21.15.2:0/1421447553 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.066+0000 7fcdeb400640 1 --1- 172.21.15.2:0/1421447553 >> v1:172.21.15.137:6789/0 conn(0x7fcdec0b2430 0x7fcdec0b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:58296/0 (socket says 172.21.15.2:58296) 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.067+0000 7fcdea000640 1 -- 172.21.15.2:0/1421447553 >> v2:172.21.15.137:3300/0 conn(0x7fcdec15ae80 msgr2=0x7fcdec0acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.067+0000 7fcdea000640 1 --2- 172.21.15.2:0/1421447553 >> v2:172.21.15.137:3300/0 conn(0x7fcdec15ae80 0x7fcdec0acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.067+0000 7fcdea000640 1 -- 172.21.15.2:0/1421447553 >> v1:172.21.15.137:6789/0 conn(0x7fcdec0b2430 legacy=0x7fcdec0b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:28.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.067+0000 7fcdea000640 1 -- 172.21.15.2:0/1421447553 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec15dea0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.069+0000 7fcdea000640 1 --2- 172.21.15.2:0/1421447553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec06c8f0 0x7fcdec06cd50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40097a0 tx=0x7fcdd402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34b0e6ce413f1c8a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:28.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.069+0000 7fcde9600640 1 -- 172.21.15.2:0/1421447553 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd402d040 con 0x7fcdec06c8f0 2024-01-15T18:10:28.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.069+0000 7fcde9600640 1 -- 172.21.15.2:0/1421447553 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd4031810 con 0x7fcdec06c8f0 2024-01-15T18:10:28.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.069+0000 7fcde9600640 1 -- 172.21.15.2:0/1421447553 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd4031ba0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 -- 172.21.15.2:0/1421447553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec06c8f0 msgr2=0x7fcdec06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:28.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/1421447553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec06c8f0 0x7fcdec06cd50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40097a0 tx=0x7fcdd402b8b0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 -- 172.21.15.2:0/1421447553 shutdown_connections 2024-01-15T18:10:28.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/1421447553 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec06c8f0 0x7fcdec06cd50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/1421447553 >> v2:172.21.15.137:3300/0 conn(0x7fcdec15ae80 0x7fcdec0acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 -- 172.21.15.2:0/1421447553 >> 172.21.15.2:0/1421447553 conn(0x7fcdec0b45f0 msgr2=0x7fcdec0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:28.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 -- 172.21.15.2:0/1421447553 shutdown_connections 2024-01-15T18:10:28.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.070+0000 7fcdf1a00640 1 -- 172.21.15.2:0/1421447553 wait complete. 2024-01-15T18:10:28.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.072+0000 7fcdf1a00640 1 Processor -- start 2024-01-15T18:10:28.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.073+0000 7fcdf1a00640 1 -- start start 2024-01-15T18:10:28.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.073+0000 7fcdf1a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 0x7fcdec0af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:28.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdf1a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdec0b2430 0x7fcdec15a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:28.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdf1a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec15ae80 0x7fcdec15c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:28.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdf1a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec15a780 con 0x7fcdec06c8f0 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdf1a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec14f5c0 con 0x7fcdec0b2430 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdf1a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdec0aec00 con 0x7fcdec15ae80 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdeaa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 0x7fcdec0af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdeaa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 0x7fcdec0af6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36124/0 (socket says 172.21.15.2:36124) 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdeaa00640 1 -- 172.21.15.2:0/2342054 learned_addr learned my addr 172.21.15.2:0/2342054 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdea000640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdec0b2430 0x7fcdec15a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:28.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.074+0000 7fcdeb400640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec15ae80 0x7fcdec15c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:28.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeaa00640 1 -- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec15ae80 msgr2=0x7fcdec15c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:28.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeaa00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec15ae80 0x7fcdec15c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeaa00640 1 -- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdec0b2430 msgr2=0x7fcdec15a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:28.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeaa00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdec0b2430 0x7fcdec15a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeaa00640 1 -- 172.21.15.2:0/2342054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd4009400 con 0x7fcdec06c8f0 2024-01-15T18:10:28.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeb400640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec15ae80 0x7fcdec15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdea000640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdec0b2430 0x7fcdec15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdeaa00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 0x7fcdec0af6e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00e840 tx=0x7fcddc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00c8e0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc00ca40 con 0x7fcdec06c8f0 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00cdf0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.075+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec0aeee0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.076+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec14b9d0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.077+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec14bbf0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.078+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fcddc013470 con 0x7fcdec06c8f0 2024-01-15T18:10:28.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.079+0000 7fcde2a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdc406bfb0 0x7fcdc406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:28.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.079+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5311+0+0 (secure 0 0 0) 0x7fcddc088d50 con 0x7fcdec06c8f0 2024-01-15T18:10:28.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.080+0000 7fcdea000640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdc406bfb0 0x7fcdc406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:28.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.081+0000 7fcdea000640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdc406bfb0 0x7fcdc406e470 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4036000 tx=0x7fcdd403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:28.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.083+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcddc055650 con 0x7fcdec06c8f0 2024-01-15T18:10:28.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.278+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcdec14bbf0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.827+0000 7fcde2a00640 1 -- 172.21.15.2:0/2342054 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v87) v1 ==== 170+0+0 (secure 0 0 0) 0x7fcdec14bbf0 con 0x7fcdec06c8f0 2024-01-15T18:10:28.832 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T18:10:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdc406bfb0 msgr2=0x7fcdc406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdc406bfb0 0x7fcdc406e470 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4036000 tx=0x7fcdd403c040 comp rx=0 tx=0).stop 2024-01-15T18:10:28.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 msgr2=0x7fcdec0af6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 0x7fcdec0af6e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00e840 tx=0x7fcddc00ed00 comp rx=0 tx=0).stop 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 shutdown_connections 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdc406bfb0 0x7fcdc406e470 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcdec15ae80 0x7fcdec15c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcdec0b2430 0x7fcdec15a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.833+0000 7fcdf1a00640 1 --2- 172.21.15.2:0/2342054 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcdec06c8f0 0x7fcdec0af6e0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.834+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 >> 172.21.15.2:0/2342054 conn(0x7fcdec0b45f0 msgr2=0x7fcdec0ce320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.835+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 shutdown_connections 2024-01-15T18:10:28.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:28.835+0000 7fcdf1a00640 1 -- 172.21.15.2:0/2342054 wait complete. 2024-01-15T18:10:28.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:10:29.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.062+0000 7f4ceca00640 1 Processor -- start 2024-01-15T18:10:29.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.062+0000 7f4ceca00640 1 -- start start 2024-01-15T18:10:29.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4ceca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4ce8143d20 0x7f4ce815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:29.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4ceca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4ce81447d0 0x7f4ce815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:29.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4ceca00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ce8071090 con 0x7f4ce80720d0 2024-01-15T18:10:29.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4ceca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce8071430 con 0x7f4ce81447d0 2024-01-15T18:10:29.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4ceca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ce8144100 con 0x7f4ce8143d20 2024-01-15T18:10:29.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4ce7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4ce81447d0 0x7f4ce815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4cde800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4ce80720d0 0x7f4ce80cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47514/0 (socket says 172.21.15.2:47514) 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.064+0000 7f4cde800640 1 -- 172.21.15.2:0/3376966168 learned_addr learned my addr 172.21.15.2:0/3376966168 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.065+0000 7f4ce7400640 1 --2- 172.21.15.2:0/3376966168 >> v2:172.21.15.137:3300/0 conn(0x7f4ce8143d20 0x7f4ce815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.065+0000 7f4ce7e00640 1 -- 172.21.15.2:0/3376966168 >> v2:172.21.15.137:3300/0 conn(0x7f4ce8143d20 msgr2=0x7f4ce815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.065+0000 7f4ce7e00640 1 --2- 172.21.15.2:0/3376966168 >> v2:172.21.15.137:3300/0 conn(0x7f4ce8143d20 0x7f4ce815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.065+0000 7f4ce7e00640 1 -- 172.21.15.2:0/3376966168 >> v1:172.21.15.2:6789/0 conn(0x7f4ce80720d0 legacy=0x7f4ce80cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.065+0000 7f4ce7e00640 1 -- 172.21.15.2:0/3376966168 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce806d160 con 0x7f4ce81447d0 2024-01-15T18:10:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.065+0000 7f4ce6a00640 1 -- 172.21.15.2:0/3376966168 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543958469 0 0) 0x7f4ce8071090 con 0x7f4ce80720d0 2024-01-15T18:10:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.066+0000 7f4ce7e00640 1 --2- 172.21.15.2:0/3376966168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce815f090 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f4ce806e600 tx=0x7f4ce002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4843a5a1818567e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.066+0000 7f4ce6a00640 1 -- 172.21.15.2:0/3376966168 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce002cbf0 con 0x7f4ce81447d0 2024-01-15T18:10:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.066+0000 7f4ce6a00640 1 -- 172.21.15.2:0/3376966168 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce002cd50 con 0x7f4ce81447d0 2024-01-15T18:10:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.066+0000 7f4ce6a00640 1 -- 172.21.15.2:0/3376966168 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce0034660 con 0x7f4ce81447d0 2024-01-15T18:10:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 -- 172.21.15.2:0/3376966168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 msgr2=0x7f4ce815f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 --2- 172.21.15.2:0/3376966168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce815f090 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f4ce806e600 tx=0x7f4ce002be40 comp rx=0 tx=0).stop 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 -- 172.21.15.2:0/3376966168 shutdown_connections 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 --2- 172.21.15.2:0/3376966168 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce815f090 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 --2- 172.21.15.2:0/3376966168 >> v2:172.21.15.137:3300/0 conn(0x7f4ce8143d20 0x7f4ce815eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 -- 172.21.15.2:0/3376966168 >> 172.21.15.2:0/3376966168 conn(0x7f4ce80b0840 msgr2=0x7f4ce80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 -- 172.21.15.2:0/3376966168 shutdown_connections 2024-01-15T18:10:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.067+0000 7f4ceca00640 1 -- 172.21.15.2:0/3376966168 wait complete. 2024-01-15T18:10:29.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.069+0000 7f4ceca00640 1 Processor -- start 2024-01-15T18:10:29.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.069+0000 7f4ceca00640 1 -- start start 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ceca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ce80720d0 0x7f4ce815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ceca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ce8143d20 0x7f4ce815c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ceca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce81f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ceca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce815c580 con 0x7f4ce81447d0 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ceca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce815d780 con 0x7f4ce8143d20 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ceca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce815da60 con 0x7f4ce80720d0 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ce7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce81f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ce7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce81f21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36154/0 (socket says 172.21.15.2:36154) 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.070+0000 7f4ce7e00640 1 -- 172.21.15.2:0/2890733842 learned_addr learned my addr 172.21.15.2:0/2890733842 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7400640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ce80720d0 0x7f4ce815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4cde800640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ce8143d20 0x7f4ce815c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:29.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7e00640 1 -- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ce80720d0 msgr2=0x7f4ce815d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7e00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ce80720d0 0x7f4ce815d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7e00640 1 -- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ce8143d20 msgr2=0x7f4ce815c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7e00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ce8143d20 0x7f4ce815c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7e00640 1 -- 172.21.15.2:0/2890733842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0009a70 con 0x7f4ce81447d0 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7400640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ce80720d0 0x7f4ce815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4cde800640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ce8143d20 0x7f4ce815c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.071+0000 7f4ce7e00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce81f21f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f4ce002c450 tx=0x7f4ce0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.072+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce003e610 con 0x7f4ce81447d0 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.072+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce003e770 con 0x7f4ce81447d0 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.072+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce003eb20 con 0x7f4ce81447d0 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.072+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce81596b0 con 0x7f4ce81447d0 2024-01-15T18:10:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.072+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce8159c70 con 0x7f4ce81447d0 2024-01-15T18:10:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.073+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce8159e90 con 0x7f4ce81447d0 2024-01-15T18:10:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.074+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f4ce003b050 con 0x7f4ce81447d0 2024-01-15T18:10:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.075+0000 7f4cdfe00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cc006bfb0 0x7f4cc006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.075+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5693+0+0 (secure 0 0 0) 0x7f4ce00ab1a0 con 0x7f4ce81447d0 2024-01-15T18:10:29.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.076+0000 7f4ce7400640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cc006bfb0 0x7f4cc006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:29.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.077+0000 7f4ce7400640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cc006bfb0 0x7f4cc006e470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8006fd0 tx=0x7f4cd8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:29.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.079+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ce0077b80 con 0x7f4ce81447d0 2024-01-15T18:10:29.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.318+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4ce8074770 con 0x7f4ce81447d0 2024-01-15T18:10:29.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.850+0000 7f4cdfe00640 1 -- 172.21.15.2:0/2890733842 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4ce00390c0 con 0x7f4ce81447d0 2024-01-15T18:10:29.855 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 12 and data pool 13 2024-01-15T18:10:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cc006bfb0 msgr2=0x7f4cc006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cc006bfb0 0x7f4cc006e470 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f4cd8006fd0 tx=0x7f4cd8008040 comp rx=0 tx=0).stop 2024-01-15T18:10:29.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 msgr2=0x7f4ce81f21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce81f21f0 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f4ce002c450 tx=0x7f4ce0004a40 comp rx=0 tx=0).stop 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 shutdown_connections 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4cc006bfb0 0x7f4cc006e470 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4ce81447d0 0x7f4ce81f21f0 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4ce8143d20 0x7f4ce815c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 --2- 172.21.15.2:0/2890733842 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4ce80720d0 0x7f4ce815d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.854+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 >> 172.21.15.2:0/2890733842 conn(0x7f4ce80b0840 msgr2=0x7f4ce81413a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.855+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 shutdown_connections 2024-01-15T18:10:29.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:29.855+0000 7f4ceca00640 1 -- 172.21.15.2:0/2890733842 wait complete. 2024-01-15T18:10:29.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.077+0000 7f6d34a00640 1 Processor -- start 2024-01-15T18:10:30.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.078+0000 7f6d34a00640 1 -- start start 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d34a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6d30141880 0x7f6d30070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d34a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6d3015ec60 0x7f6d3015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d34a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3006dae0 con 0x7f6d300cd490 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d34a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d30071f20 con 0x7f6d30141880 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d34a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3006c6b0 con 0x7f6d3015ec60 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6d30141880 0x7f6d30070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6d30141880 0x7f6d30070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36156/0 (socket says 172.21.15.2:36156) 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6d300cd490 0x7f6d300cd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47524/0 (socket says 172.21.15.2:47524) 2024-01-15T18:10:30.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 -- 172.21.15.2:0/535372863 learned_addr learned my addr 172.21.15.2:0/535372863 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2ea00640 1 --2- 172.21.15.2:0/535372863 >> v2:172.21.15.137:3300/0 conn(0x7f6d3015ec60 0x7f6d3015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 -- 172.21.15.2:0/535372863 >> v2:172.21.15.137:3300/0 conn(0x7f6d3015ec60 msgr2=0x7f6d3015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 --2- 172.21.15.2:0/535372863 >> v2:172.21.15.137:3300/0 conn(0x7f6d3015ec60 0x7f6d3015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 -- 172.21.15.2:0/535372863 >> v1:172.21.15.2:6789/0 conn(0x7f6d300cd490 legacy=0x7f6d300cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2f400640 1 -- 172.21.15.2:0/535372863 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d30071960 con 0x7f6d30141880 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.080+0000 7f6d2e000640 1 -- 172.21.15.2:0/535372863 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906457274 0 0) 0x7f6d3006dae0 con 0x7f6d300cd490 2024-01-15T18:10:30.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.082+0000 7f6d2f400640 1 --2- 172.21.15.2:0/535372863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d30141880 0x7f6d30070fc0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f6d300722d0 tx=0x7f6d2002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b288884b793f35e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.082+0000 7f6d2e000640 1 -- 172.21.15.2:0/535372863 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20031870 con 0x7f6d30141880 2024-01-15T18:10:30.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.082+0000 7f6d2e000640 1 -- 172.21.15.2:0/535372863 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d200319d0 con 0x7f6d30141880 2024-01-15T18:10:30.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.082+0000 7f6d2e000640 1 -- 172.21.15.2:0/535372863 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20031d60 con 0x7f6d30141880 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.082+0000 7f6d34a00640 1 -- 172.21.15.2:0/535372863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d30141880 msgr2=0x7f6d30070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.082+0000 7f6d34a00640 1 --2- 172.21.15.2:0/535372863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d30141880 0x7f6d30070fc0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f6d300722d0 tx=0x7f6d2002b910 comp rx=0 tx=0).stop 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.083+0000 7f6d34a00640 1 -- 172.21.15.2:0/535372863 shutdown_connections 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.083+0000 7f6d34a00640 1 --2- 172.21.15.2:0/535372863 >> v2:172.21.15.137:3300/0 conn(0x7f6d3015ec60 0x7f6d3015f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.083+0000 7f6d34a00640 1 --2- 172.21.15.2:0/535372863 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d30141880 0x7f6d30070fc0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.083+0000 7f6d34a00640 1 -- 172.21.15.2:0/535372863 >> 172.21.15.2:0/535372863 conn(0x7f6d300b0840 msgr2=0x7f6d300b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.083+0000 7f6d34a00640 1 -- 172.21.15.2:0/535372863 shutdown_connections 2024-01-15T18:10:30.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.083+0000 7f6d34a00640 1 -- 172.21.15.2:0/535372863 wait complete. 2024-01-15T18:10:30.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.084+0000 7f6d34a00640 1 Processor -- start 2024-01-15T18:10:30.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.085+0000 7f6d34a00640 1 -- start start 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.085+0000 7f6d34a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d300cd490 0x7f6d301f7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d34a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d30141880 0x7f6d301e48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d34a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 0x7f6d301f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d34a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d301e50c0 con 0x7f6d3015ec60 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d34a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d301e4e20 con 0x7f6d300cd490 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d34a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d30130f10 con 0x7f6d30141880 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 0x7f6d301f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 0x7f6d301f6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36170/0 (socket says 172.21.15.2:36170) 2024-01-15T18:10:30.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 -- 172.21.15.2:0/4001977037 learned_addr learned my addr 172.21.15.2:0/4001977037 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2ea00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d30141880 0x7f6d301e48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2f400640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d300cd490 0x7f6d301f7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 -- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d30141880 msgr2=0x7f6d301e48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d30141880 0x7f6d301e48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 -- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d300cd490 msgr2=0x7f6d301f7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d300cd490 0x7f6d301f7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2fe00640 1 -- 172.21.15.2:0/4001977037 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d200094d0 con 0x7f6d3015ec60 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.086+0000 7f6d2ea00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d30141880 0x7f6d301e48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.087+0000 7f6d2fe00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 0x7f6d301f6c60 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c7e0 tx=0x7f6d2800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.087+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800a640 con 0x7f6d3015ec60 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.087+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2800a7c0 con 0x7f6d3015ec60 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.087+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d301311f0 con 0x7f6d3015ec60 2024-01-15T18:10:30.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.087+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800ab70 con 0x7f6d3015ec60 2024-01-15T18:10:30.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.087+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d301317b0 con 0x7f6d3015ec60 2024-01-15T18:10:30.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.088+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d301319d0 con 0x7f6d3015ec60 2024-01-15T18:10:30.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.089+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f6d2800acd0 con 0x7f6d3015ec60 2024-01-15T18:10:30.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.092+0000 7f6d27400640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6d0806c000 0x7f6d0806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.092+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f6d28087fe0 con 0x7f6d3015ec60 2024-01-15T18:10:30.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.092+0000 7f6d2f400640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6d0806c000 0x7f6d0806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.092+0000 7f6d2f400640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6d0806c000 0x7f6d0806e4c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6d20036000 tx=0x7f6d2003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.096+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d280548a0 con 0x7f6d3015ec60 2024-01-15T18:10:30.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.284+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d30074770 con 0x7f6d3015ec60 2024-01-15T18:10:30.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.287+0000 7f6d27400640 1 -- 172.21.15.2:0/4001977037 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16644 (secure 0 0 0) 0x7f6d28018030 con 0x7f6d3015ec60 2024-01-15T18:10:30.291 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:30.292 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:29.320528+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:10:27.147601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-15T18:10:28.332573+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.289+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6d0806c000 msgr2=0x7f6d0806e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.289+0000 7f6d34a00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6d0806c000 0x7f6d0806e4c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f6d20036000 tx=0x7f6d2003c040 comp rx=0 tx=0).stop 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.289+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 msgr2=0x7f6d301f6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.289+0000 7f6d34a00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 0x7f6d301f6c60 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800c7e0 tx=0x7f6d2800cca0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 shutdown_connections 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6d0806c000 0x7f6d0806e4c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6d3015ec60 0x7f6d301f6c60 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6d30141880 0x7f6d301e48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 --2- 172.21.15.2:0/4001977037 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6d300cd490 0x7f6d301f7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 >> 172.21.15.2:0/4001977037 conn(0x7f6d300b0840 msgr2=0x7f6d300b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 shutdown_connections 2024-01-15T18:10:30.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.290+0000 7f6d34a00640 1 -- 172.21.15.2:0/4001977037 wait complete. 2024-01-15T18:10:30.311 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:30.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.515+0000 7fc666600640 1 Processor -- start 2024-01-15T18:10:30.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.515+0000 7fc666600640 1 -- start start 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.516+0000 7fc666600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc660141880 0x7fc660070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.516+0000 7fc666600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc66015ec60 0x7fc66015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.516+0000 7fc666600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc66006dae0 con 0x7fc6600cd490 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.516+0000 7fc666600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc660071f20 con 0x7fc660141880 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.516+0000 7fc666600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc66006c6b0 con 0x7fc66015ec60 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc660141880 0x7fc660070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc660141880 0x7fc660070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36180/0 (socket says 172.21.15.2:36180) 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 -- 172.21.15.2:0/2266265725 learned_addr learned my addr 172.21.15.2:0/2266265725 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc657c00640 1 --2- 172.21.15.2:0/2266265725 >> v2:172.21.15.137:3300/0 conn(0x7fc66015ec60 0x7fc66015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 -- 172.21.15.2:0/2266265725 >> v2:172.21.15.137:3300/0 conn(0x7fc66015ec60 msgr2=0x7fc66015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 --2- 172.21.15.2:0/2266265725 >> v2:172.21.15.137:3300/0 conn(0x7fc66015ec60 0x7fc66015f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 -- 172.21.15.2:0/2266265725 >> v1:172.21.15.137:6789/0 conn(0x7fc6600cd490 legacy=0x7fc6600cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.517+0000 7fc665200640 1 -- 172.21.15.2:0/2266265725 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc660071960 con 0x7fc660141880 2024-01-15T18:10:30.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc665200640 1 --2- 172.21.15.2:0/2266265725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc660141880 0x7fc660070fc0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc6600722d0 tx=0x7fc65002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0596b9e9bede8fc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc65fe00640 1 -- 172.21.15.2:0/2266265725 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6500317e0 con 0x7fc660141880 2024-01-15T18:10:30.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc65fe00640 1 -- 172.21.15.2:0/2266265725 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc650031940 con 0x7fc660141880 2024-01-15T18:10:30.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc65fe00640 1 -- 172.21.15.2:0/2266265725 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc650031cd0 con 0x7fc660141880 2024-01-15T18:10:30.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 -- 172.21.15.2:0/2266265725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc660141880 msgr2=0x7fc660070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 --2- 172.21.15.2:0/2266265725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc660141880 0x7fc660070fc0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fc6600722d0 tx=0x7fc65002b880 comp rx=0 tx=0).stop 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 -- 172.21.15.2:0/2266265725 shutdown_connections 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 --2- 172.21.15.2:0/2266265725 >> v2:172.21.15.137:3300/0 conn(0x7fc66015ec60 0x7fc66015f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 --2- 172.21.15.2:0/2266265725 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc660141880 0x7fc660070fc0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 -- 172.21.15.2:0/2266265725 >> 172.21.15.2:0/2266265725 conn(0x7fc6600b0840 msgr2=0x7fc6600b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 -- 172.21.15.2:0/2266265725 shutdown_connections 2024-01-15T18:10:30.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.519+0000 7fc666600640 1 -- 172.21.15.2:0/2266265725 wait complete. 2024-01-15T18:10:30.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.521+0000 7fc666600640 1 Processor -- start 2024-01-15T18:10:30.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.521+0000 7fc666600640 1 -- start start 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.522+0000 7fc666600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6600cd490 0x7fc660158e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.522+0000 7fc666600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc660141880 0x7fc6600c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.522+0000 7fc666600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 0x7fc6600c4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.522+0000 7fc666600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015c480 con 0x7fc66015ec60 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.522+0000 7fc666600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015b610 con 0x7fc660141880 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.522+0000 7fc666600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc66015c8a0 con 0x7fc6600cd490 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 0x7fc6600c4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 0x7fc6600c4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36194/0 (socket says 172.21.15.2:36194) 2024-01-15T18:10:30.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 -- 172.21.15.2:0/3910065519 learned_addr learned my addr 172.21.15.2:0/3910065519 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc657c00640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc660141880 0x7fc6600c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665200640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6600cd490 0x7fc660158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 -- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6600cd490 msgr2=0x7fc660158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6600cd490 0x7fc660158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 -- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc660141880 msgr2=0x7fc6600c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc660141880 0x7fc6600c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 -- 172.21.15.2:0/3910065519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500094d0 con 0x7fc66015ec60 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc657c00640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc660141880 0x7fc6600c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665200640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6600cd490 0x7fc660158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.523+0000 7fc665c00640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 0x7fc6600c4770 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc65800c780 tx=0x7fc65800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.524+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658003d80 con 0x7fc66015ec60 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.524+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800ce30 con 0x7fc66015ec60 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.524+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc660159340 con 0x7fc66015ec60 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.524+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658005290 con 0x7fc66015ec60 2024-01-15T18:10:30.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.524+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc66015bae0 con 0x7fc66015ec60 2024-01-15T18:10:30.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.525+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66015c5c0 con 0x7fc66015ec60 2024-01-15T18:10:30.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.526+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc6580053f0 con 0x7fc66015ec60 2024-01-15T18:10:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.527+0000 7fc65d600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc63806bfb0 0x7fc63806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.527+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fc658087e30 con 0x7fc66015ec60 2024-01-15T18:10:30.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.529+0000 7fc665200640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc63806bfb0 0x7fc63806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.530+0000 7fc665200640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc63806bfb0 0x7fc63806e470 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc650036000 tx=0x7fc65003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.531+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6580546f0 con 0x7fc66015ec60 2024-01-15T18:10:30.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.726+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc660074770 con 0x7fc66015ec60 2024-01-15T18:10:30.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.728+0000 7fc65d600640 1 -- 172.21.15.2:0/3910065519 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+16644 (secure 0 0 0) 0x7fc658017030 con 0x7fc66015ec60 2024-01-15T18:10:30.732 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:30.732 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":88,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:29.320528+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:10:27.147601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-15T18:10:28.332573+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:30.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc63806bfb0 msgr2=0x7fc63806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc63806bfb0 0x7fc63806e470 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fc650036000 tx=0x7fc65003d040 comp rx=0 tx=0).stop 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 msgr2=0x7fc6600c4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 0x7fc6600c4770 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fc65800c780 tx=0x7fc65800cc40 comp rx=0 tx=0).stop 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 shutdown_connections 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc63806bfb0 0x7fc63806e470 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc66015ec60 0x7fc6600c4770 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc660141880 0x7fc6600c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 --2- 172.21.15.2:0/3910065519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc6600cd490 0x7fc660158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 >> 172.21.15.2:0/3910065519 conn(0x7fc6600b0840 msgr2=0x7fc66015d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 shutdown_connections 2024-01-15T18:10:30.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.731+0000 7fc666600640 1 -- 172.21.15.2:0/3910065519 wait complete. 2024-01-15T18:10:30.754 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:10:30.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.980+0000 7f2b1bc00640 1 Processor -- start 2024-01-15T18:10:30.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.980+0000 7f2b1bc00640 1 -- start start 2024-01-15T18:10:30.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b1bc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2b1415d4d0 0x7f2b140b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b1bc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b1406f060 con 0x7f2b140af6a0 2024-01-15T18:10:30.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b1bc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b140aef50 con 0x7f2b1406a4b0 2024-01-15T18:10:30.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b1bc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1415d8b0 con 0x7f2b1415d4d0 2024-01-15T18:10:30.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b19200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2b1415d4d0 0x7f2b140b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b19200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2b1415d4d0 0x7f2b140b2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36202/0 (socket says 172.21.15.2:36202) 2024-01-15T18:10:30.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.982+0000 7f2b19200640 1 -- 172.21.15.2:0/1600115270 learned_addr learned my addr 172.21.15.2:0/1600115270 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:30.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.983+0000 7f2b19200640 1 -- 172.21.15.2:0/1600115270 >> v1:172.21.15.137:6789/0 conn(0x7f2b1406a4b0 legacy=0x7f2b140713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:30.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.983+0000 7f2b19200640 1 -- 172.21.15.2:0/1600115270 >> v1:172.21.15.2:6789/0 conn(0x7f2b140af6a0 legacy=0x7f2b1406f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.983+0000 7f2b19200640 1 -- 172.21.15.2:0/1600115270 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b140af230 con 0x7f2b1415d4d0 2024-01-15T18:10:30.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.983+0000 7f2b13400640 1 -- 172.21.15.2:0/1600115270 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060012139 0 0) 0x7f2b1406f060 con 0x7f2b140af6a0 2024-01-15T18:10:30.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.984+0000 7f2b19200640 1 --2- 172.21.15.2:0/1600115270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b1415d4d0 0x7f2b140b2860 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009870 tx=0x7f2b0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e11c74cea271f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b13400640 1 -- 172.21.15.2:0/1600115270 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b040317e0 con 0x7f2b1415d4d0 2024-01-15T18:10:30.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b13400640 1 -- 172.21.15.2:0/1600115270 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b04031940 con 0x7f2b1415d4d0 2024-01-15T18:10:30.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b13400640 1 -- 172.21.15.2:0/1600115270 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b04031cd0 con 0x7f2b1415d4d0 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 -- 172.21.15.2:0/1600115270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b1415d4d0 msgr2=0x7f2b140b2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/1600115270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b1415d4d0 0x7f2b140b2860 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f2b04009870 tx=0x7f2b0402b880 comp rx=0 tx=0).stop 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 -- 172.21.15.2:0/1600115270 shutdown_connections 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/1600115270 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b1415d4d0 0x7f2b140b2860 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 -- 172.21.15.2:0/1600115270 >> 172.21.15.2:0/1600115270 conn(0x7f2b140b1d70 msgr2=0x7f2b140b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 -- 172.21.15.2:0/1600115270 shutdown_connections 2024-01-15T18:10:30.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.985+0000 7f2b1bc00640 1 -- 172.21.15.2:0/1600115270 wait complete. 2024-01-15T18:10:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.987+0000 7f2b1bc00640 1 Processor -- start 2024-01-15T18:10:30.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.987+0000 7f2b1bc00640 1 -- start start 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.988+0000 7f2b1bc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b1406a4b0 0x7f2b14147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.988+0000 7f2b1bc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 0x7f2b140cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.988+0000 7f2b1bc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b1415d4d0 0x7f2b14152480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.988+0000 7f2b1bc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b141598b0 con 0x7f2b140af6a0 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.988+0000 7f2b1bc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14152e60 con 0x7f2b1415d4d0 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.988+0000 7f2b1bc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b14153140 con 0x7f2b1406a4b0 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 0x7f2b140cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 0x7f2b140cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36210/0 (socket says 172.21.15.2:36210) 2024-01-15T18:10:30.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 -- 172.21.15.2:0/675755975 learned_addr learned my addr 172.21.15.2:0/675755975 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b19200640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b1406a4b0 0x7f2b14147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b19c00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b1415d4d0 0x7f2b14152480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 -- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b1406a4b0 msgr2=0x7f2b14147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b1406a4b0 0x7f2b14147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 -- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b1415d4d0 msgr2=0x7f2b14152480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b1415d4d0 0x7f2b14152480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b13e00640 1 -- 172.21.15.2:0/675755975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b040094d0 con 0x7f2b140af6a0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.989+0000 7f2b19200640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b1406a4b0 0x7f2b14147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.990+0000 7f2b13e00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 0x7f2b140cd080 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00c890 tx=0x7f2afc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.990+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc003da0 con 0x7f2b140af6a0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.990+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2afc005070 con 0x7f2b140af6a0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.990+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2afc005420 con 0x7f2b140af6a0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.990+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b14153420 con 0x7f2b140af6a0 2024-01-15T18:10:30.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.990+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b14147f70 con 0x7f2b140af6a0 2024-01-15T18:10:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.991+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b141599f0 con 0x7f2b140af6a0 2024-01-15T18:10:30.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.992+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f2afc005580 con 0x7f2b140af6a0 2024-01-15T18:10:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.993+0000 7f2b10c00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2aec06bfe0 0x7f2aec06e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:30.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.993+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f2afc087f00 con 0x7f2b140af6a0 2024-01-15T18:10:30.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.994+0000 7f2b19200640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2aec06bfe0 0x7f2aec06e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:30.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:30.995+0000 7f2b19200640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2aec06bfe0 0x7f2aec06e4a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2b04036000 tx=0x7f2b0403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:31.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.000+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b141599f0 con 0x7f2b140af6a0 2024-01-15T18:10:31.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.237+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2b141599f0 con 0x7f2b140af6a0 2024-01-15T18:10:31.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.911+0000 7f2b10c00640 1 -- 172.21.15.2:0/675755975 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2b141599f0 con 0x7f2b140af6a0 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2aec06bfe0 msgr2=0x7f2aec06e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2aec06bfe0 0x7f2aec06e4a0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2b04036000 tx=0x7f2b0403d040 comp rx=0 tx=0).stop 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 msgr2=0x7f2b140cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 0x7f2b140cd080 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f2afc00c890 tx=0x7f2afc00cd50 comp rx=0 tx=0).stop 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 shutdown_connections 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2aec06bfe0 0x7f2aec06e4a0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b1415d4d0 0x7f2b14152480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:31.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b140af6a0 0x7f2b140cd080 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 --2- 172.21.15.2:0/675755975 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b1406a4b0 0x7f2b14147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 >> 172.21.15.2:0/675755975 conn(0x7f2b140b1d70 msgr2=0x7f2b140713d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.915+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 shutdown_connections 2024-01-15T18:10:31.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:31.916+0000 7f2b1bc00640 1 -- 172.21.15.2:0/675755975 wait complete. 2024-01-15T18:10:31.939 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:10:31.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:10:32.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.160+0000 7fda8e800640 1 Processor -- start 2024-01-15T18:10:32.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.161+0000 7fda8e800640 1 -- start start 2024-01-15T18:10:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.162+0000 7fda8e800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda88143d20 0x7fda8815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.162+0000 7fda8e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda880720d0 0x7fda8815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8e800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda88071090 con 0x7fda881447d0 2024-01-15T18:10:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8e800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda88071430 con 0x7fda88143d20 2024-01-15T18:10:32.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda88144100 con 0x7fda880720d0 2024-01-15T18:10:32.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8d400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda88143d20 0x7fda8815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8d400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fda88143d20 0x7fda8815eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:37678/0 (socket says 172.21.15.2:37678) 2024-01-15T18:10:32.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8d400640 1 -- 172.21.15.2:0/1953493515 learned_addr learned my addr 172.21.15.2:0/1953493515 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:32.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.163+0000 7fda8ca00640 1 --2- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda880720d0 0x7fda8815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.164+0000 7fda8d400640 1 -- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda880720d0 msgr2=0x7fda8815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.164+0000 7fda8d400640 1 --2- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda880720d0 0x7fda8815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.164+0000 7fda8d400640 1 -- 172.21.15.2:0/1953493515 >> v1:172.21.15.137:6789/0 conn(0x7fda881447d0 legacy=0x7fda880cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.164+0000 7fda8d400640 1 -- 172.21.15.2:0/1953493515 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8806d160 con 0x7fda88143d20 2024-01-15T18:10:32.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.166+0000 7fda8d400640 1 --2- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda88143d20 0x7fda8815eb50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fda78009870 tx=0x7fda7802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f74e27e9fb081773 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.166+0000 7fda87e00640 1 -- 172.21.15.2:0/1953493515 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78031640 con 0x7fda88143d20 2024-01-15T18:10:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.166+0000 7fda87e00640 1 -- 172.21.15.2:0/1953493515 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda780317a0 con 0x7fda88143d20 2024-01-15T18:10:32.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.166+0000 7fda87e00640 1 -- 172.21.15.2:0/1953493515 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda78031b30 con 0x7fda88143d20 2024-01-15T18:10:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 -- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda88143d20 msgr2=0x7fda8815eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 --2- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda88143d20 0x7fda8815eb50 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fda78009870 tx=0x7fda7802b8f0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 -- 172.21.15.2:0/1953493515 shutdown_connections 2024-01-15T18:10:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 --2- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda880720d0 0x7fda8815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 --2- 172.21.15.2:0/1953493515 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda88143d20 0x7fda8815eb50 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 -- 172.21.15.2:0/1953493515 >> 172.21.15.2:0/1953493515 conn(0x7fda880b0840 msgr2=0x7fda880b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:32.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 -- 172.21.15.2:0/1953493515 shutdown_connections 2024-01-15T18:10:32.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.167+0000 7fda8e800640 1 -- 172.21.15.2:0/1953493515 wait complete. 2024-01-15T18:10:32.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.169+0000 7fda8e800640 1 Processor -- start 2024-01-15T18:10:32.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.169+0000 7fda8e800640 1 -- start start 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda880720d0 0x7fda8813ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 0x7fda8813b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda881447d0 0x7fda881377f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8813bd20 con 0x7fda88143d20 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8813d830 con 0x7fda880720d0 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8813db10 con 0x7fda881447d0 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 0x7fda8813b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8ca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 0x7fda8813b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36216/0 (socket says 172.21.15.2:36216) 2024-01-15T18:10:32.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8ca00640 1 -- 172.21.15.2:0/63257073 learned_addr learned my addr 172.21.15.2:0/63257073 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8d400640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda880720d0 0x7fda8813ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.170+0000 7fda8de00640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda881447d0 0x7fda881377f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8ca00640 1 -- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda881447d0 msgr2=0x7fda881377f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8ca00640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda881447d0 0x7fda881377f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8ca00640 1 -- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda880720d0 msgr2=0x7fda8813ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8ca00640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda880720d0 0x7fda8813ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8ca00640 1 -- 172.21.15.2:0/63257073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda780094d0 con 0x7fda88143d20 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8ca00640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 0x7fda8813b7e0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e870 tx=0x7fda7000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:32.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda700094f0 con 0x7fda88143d20 2024-01-15T18:10:32.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7000c940 con 0x7fda88143d20 2024-01-15T18:10:32.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8813ddf0 con 0x7fda88143d20 2024-01-15T18:10:32.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.171+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7000ccf0 con 0x7fda88143d20 2024-01-15T18:10:32.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.172+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8813e3b0 con 0x7fda88143d20 2024-01-15T18:10:32.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.173+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda88074770 con 0x7fda88143d20 2024-01-15T18:10:32.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.174+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fda70013470 con 0x7fda88143d20 2024-01-15T18:10:32.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.175+0000 7fda85600640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6006bfe0 0x7fda6006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:32.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.175+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fda700886c0 con 0x7fda88143d20 2024-01-15T18:10:32.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.175+0000 7fda8d400640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6006bfe0 0x7fda6006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:32.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.177+0000 7fda8d400640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6006bfe0 0x7fda6006e4a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fda78036000 tx=0x7fda7803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:32.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.179+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fda70054f80 con 0x7fda88143d20 2024-01-15T18:10:32.307 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:32.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:10:32.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.415+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fda881424a0 con 0x7fda88143d20 2024-01-15T18:10:32.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.914+0000 7fda85600640 1 -- 172.21.15.2:0/63257073 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v56) v1 ==== 122+0+0 (secure 0 0 0) 0x7fda700170c0 con 0x7fda88143d20 2024-01-15T18:10:32.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6006bfe0 msgr2=0x7fda6006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6006bfe0 0x7fda6006e4a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fda78036000 tx=0x7fda7803c040 comp rx=0 tx=0).stop 2024-01-15T18:10:32.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 msgr2=0x7fda8813b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 0x7fda8813b7e0 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7fda7000e870 tx=0x7fda7000ed30 comp rx=0 tx=0).stop 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 shutdown_connections 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fda6006bfe0 0x7fda6006e4a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fda881447d0 0x7fda881377f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fda88143d20 0x7fda8813b7e0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 --2- 172.21.15.2:0/63257073 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fda880720d0 0x7fda8813ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 >> 172.21.15.2:0/63257073 conn(0x7fda880b0840 msgr2=0x7fda88142fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 shutdown_connections 2024-01-15T18:10:32.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:32.918+0000 7fda8e800640 1 -- 172.21.15.2:0/63257073 wait complete. 2024-01-15T18:10:32.941 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:10:32.942 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:10:33.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.141+0000 7fd83f400640 1 Processor -- start 2024-01-15T18:10:33.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.141+0000 7fd83f400640 1 -- start start 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.142+0000 7fd83f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd83815e340 0x7fd8380cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.142+0000 7fd83f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd8380cdfa0 0x7fd838070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83f400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8380aeab0 con 0x7fd83813ce30 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83f400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380ce6d0 con 0x7fd8380cdfa0 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83f400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8380ce9b0 con 0x7fd83815e340 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd8380cdfa0 0x7fd838070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd8380cdfa0 0x7fd838070ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55654/0 (socket says 172.21.15.2:55654) 2024-01-15T18:10:33.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 -- 172.21.15.2:0/173986110 learned_addr learned my addr 172.21.15.2:0/173986110 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83ca00640 1 --1- 172.21.15.2:0/173986110 >> v1:172.21.15.2:6789/0 conn(0x7fd83813ce30 0x7fd83815dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44290/0 (socket says 172.21.15.2:44290) 2024-01-15T18:10:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd837e00640 1 --2- 172.21.15.2:0/173986110 >> v2:172.21.15.137:3300/0 conn(0x7fd83815e340 0x7fd8380cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 -- 172.21.15.2:0/173986110 >> v2:172.21.15.137:3300/0 conn(0x7fd83815e340 msgr2=0x7fd8380cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 --2- 172.21.15.2:0/173986110 >> v2:172.21.15.137:3300/0 conn(0x7fd83815e340 0x7fd8380cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 -- 172.21.15.2:0/173986110 >> v1:172.21.15.2:6789/0 conn(0x7fd83813ce30 legacy=0x7fd83815dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd83d400640 1 -- 172.21.15.2:0/173986110 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838070380 con 0x7fd8380cdfa0 2024-01-15T18:10:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd837400640 1 -- 172.21.15.2:0/173986110 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092049741 0 0) 0x7fd8380aeab0 con 0x7fd83813ce30 2024-01-15T18:10:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.143+0000 7fd837e00640 1 --2- 172.21.15.2:0/173986110 >> v2:172.21.15.137:3300/0 conn(0x7fd83815e340 0x7fd8380cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:33.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.145+0000 7fd83d400640 1 --2- 172.21.15.2:0/173986110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8380cdfa0 0x7fd838070ad0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd8380721d0 tx=0x7fd83002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b91f5e1c9052e44 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.145+0000 7fd837400640 1 -- 172.21.15.2:0/173986110 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83002cb20 con 0x7fd8380cdfa0 2024-01-15T18:10:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.145+0000 7fd837400640 1 -- 172.21.15.2:0/173986110 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83002cc80 con 0x7fd8380cdfa0 2024-01-15T18:10:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.145+0000 7fd837400640 1 -- 172.21.15.2:0/173986110 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8300343f0 con 0x7fd8380cdfa0 2024-01-15T18:10:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.145+0000 7fd83f400640 1 -- 172.21.15.2:0/173986110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8380cdfa0 msgr2=0x7fd838070ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.145+0000 7fd83f400640 1 --2- 172.21.15.2:0/173986110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8380cdfa0 0x7fd838070ad0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7fd8380721d0 tx=0x7fd83002bf80 comp rx=0 tx=0).stop 2024-01-15T18:10:33.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.146+0000 7fd83f400640 1 -- 172.21.15.2:0/173986110 shutdown_connections 2024-01-15T18:10:33.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.146+0000 7fd83f400640 1 --2- 172.21.15.2:0/173986110 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd8380cdfa0 0x7fd838070ad0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.146+0000 7fd83f400640 1 --2- 172.21.15.2:0/173986110 >> v2:172.21.15.137:3300/0 conn(0x7fd83815e340 0x7fd8380cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.146+0000 7fd83f400640 1 -- 172.21.15.2:0/173986110 >> 172.21.15.2:0/173986110 conn(0x7fd83806da20 msgr2=0x7fd83806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:33.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.146+0000 7fd83f400640 1 -- 172.21.15.2:0/173986110 shutdown_connections 2024-01-15T18:10:33.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.146+0000 7fd83f400640 1 -- 172.21.15.2:0/173986110 wait complete. 2024-01-15T18:10:33.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.148+0000 7fd83f400640 1 Processor -- start 2024-01-15T18:10:33.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.148+0000 7fd83f400640 1 -- start start 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83f400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8380cdfa0 0x7fd838136ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 0x7fd8381522b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd83815e340 0x7fd83813f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83f400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838153940 con 0x7fd83813ce30 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83f400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd838152d30 con 0x7fd8380cdfa0 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83f400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd838153010 con 0x7fd83815e340 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd837e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 0x7fd8381522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd837e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 0x7fd8381522b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55662/0 (socket says 172.21.15.2:55662) 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd837e00640 1 -- 172.21.15.2:0/2513368823 learned_addr learned my addr 172.21.15.2:0/2513368823 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83ca00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8380cdfa0 0x7fd838136ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.149+0000 7fd83d400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd83815e340 0x7fd83813f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd837e00640 1 -- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd83815e340 msgr2=0x7fd83813f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd837e00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd83815e340 0x7fd83813f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd837e00640 1 -- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8380cdfa0 msgr2=0x7fd838136ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd837e00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8380cdfa0 0x7fd838136ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd837e00640 1 -- 172.21.15.2:0/2513368823 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830009b30 con 0x7fd83813ce30 2024-01-15T18:10:33.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd837e00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 0x7fd8381522b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fd82000c7f0 tx=0x7fd82000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:33.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820007820 con 0x7fd83813ce30 2024-01-15T18:10:33.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd820007980 con 0x7fd83813ce30 2024-01-15T18:10:33.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd820007cf0 con 0x7fd83813ce30 2024-01-15T18:10:33.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8381532f0 con 0x7fd83813ce30 2024-01-15T18:10:33.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.150+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8381371e0 con 0x7fd83813ce30 2024-01-15T18:10:33.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.151+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8380686f0 con 0x7fd83813ce30 2024-01-15T18:10:33.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.152+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fd8200113f0 con 0x7fd83813ce30 2024-01-15T18:10:33.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.153+0000 7fd834c00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd81006bfe0 0x7fd81006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.153+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fd820088930 con 0x7fd83813ce30 2024-01-15T18:10:33.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.154+0000 7fd83ca00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd81006bfe0 0x7fd81006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.155+0000 7fd83ca00640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd81006bfe0 0x7fd81006e4a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd828000f60 tx=0x7fd828007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:33.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.158+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd8200551f0 con 0x7fd83813ce30 2024-01-15T18:10:33.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.393+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8380724d0 con 0x7fd83813ce30 2024-01-15T18:10:33.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.394+0000 7fd834c00640 1 -- 172.21.15.2:0/2513368823 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fd820058810 con 0x7fd83813ce30 2024-01-15T18:10:33.398 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:33.398 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2471268746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2471268746},{"type":"v1","addr":"172.21.15.137:6836","nonce":2471268746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1012168762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":1012168762},{"type":"v1","addr":"172.21.15.137:6837","nonce":1012168762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6559,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2801413765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2801413765},{"type":"v1","addr":"172.21.15.2:6833","nonce":2801413765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:10:29.320912+0000","modified":"2024-01-15T18:10:32.417194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8055},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8055":{"gid":8055,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6835/3200314764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3200314764},{"type":"v1","addr":"172.21.15.2:6835","nonce":3200314764}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-15T18:10:33.398 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 56 2024-01-15T18:10:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd81006bfe0 msgr2=0x7fd81006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd81006bfe0 0x7fd81006e4a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fd828000f60 tx=0x7fd828007b50 comp rx=0 tx=0).stop 2024-01-15T18:10:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 msgr2=0x7fd8381522b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 0x7fd8381522b0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7fd82000c7f0 tx=0x7fd82000ccb0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 shutdown_connections 2024-01-15T18:10:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd81006bfe0 0x7fd81006e4a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd83815e340 0x7fd83813f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd83813ce30 0x7fd8381522b0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 --2- 172.21.15.2:0/2513368823 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd8380cdfa0 0x7fd838136ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 >> 172.21.15.2:0/2513368823 conn(0x7fd83806da20 msgr2=0x7fd83806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 shutdown_connections 2024-01-15T18:10:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.397+0000 7fd83f400640 1 -- 172.21.15.2:0/2513368823 wait complete. 2024-01-15T18:10:33.418 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:10:33.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.611+0000 7f2abea00640 1 Processor -- start 2024-01-15T18:10:33.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.611+0000 7f2abea00640 1 -- start start 2024-01-15T18:10:33.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ab80d37f0 0x7f2ab80c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abea00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80c6e40 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abea00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab80d2090 con 0x7f2ab80cfad0 2024-01-15T18:10:33.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abea00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab80d17f0 con 0x7f2ab80d37f0 2024-01-15T18:10:33.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abcc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ab80d37f0 0x7f2ab80c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abe000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2ab80d3ca0 0x7f2ab80c7810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44302/0 (socket says 172.21.15.2:44302) 2024-01-15T18:10:33.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.613+0000 7f2abe000640 1 -- 172.21.15.2:0/3816009822 learned_addr learned my addr 172.21.15.2:0/3816009822 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:33.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.614+0000 7f2abcc00640 1 -- 172.21.15.2:0/3816009822 >> v1:172.21.15.137:6789/0 conn(0x7f2ab80cfad0 legacy=0x7f2ab80cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:33.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.614+0000 7f2abcc00640 1 -- 172.21.15.2:0/3816009822 >> v1:172.21.15.2:6789/0 conn(0x7f2ab80d3ca0 legacy=0x7f2ab80c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.614+0000 7f2ab7e00640 1 -- 172.21.15.2:0/3816009822 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390135716 0 0) 0x7f2ab80c6e40 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.614+0000 7f2abcc00640 1 -- 172.21.15.2:0/3816009822 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab80d1ad0 con 0x7f2ab80d37f0 2024-01-15T18:10:33.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.615+0000 7f2abcc00640 1 --2- 172.21.15.2:0/3816009822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d37f0 0x7f2ab80c72d0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00097a0 tx=0x7f2aa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e72a3bc96ee8386 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:33.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.615+0000 7f2ab7e00640 1 -- 172.21.15.2:0/3816009822 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0031830 con 0x7f2ab80d37f0 2024-01-15T18:10:33.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.615+0000 7f2ab7e00640 1 -- 172.21.15.2:0/3816009822 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa0031990 con 0x7f2ab80d37f0 2024-01-15T18:10:33.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.615+0000 7f2ab7e00640 1 -- 172.21.15.2:0/3816009822 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa0031d20 con 0x7f2ab80d37f0 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 -- 172.21.15.2:0/3816009822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d37f0 msgr2=0x7f2ab80c72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 --2- 172.21.15.2:0/3816009822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d37f0 0x7f2ab80c72d0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f2aa00097a0 tx=0x7f2aa002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 -- 172.21.15.2:0/3816009822 shutdown_connections 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 --2- 172.21.15.2:0/3816009822 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d37f0 0x7f2ab80c72d0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 -- 172.21.15.2:0/3816009822 >> 172.21.15.2:0/3816009822 conn(0x7f2ab809c3b0 msgr2=0x7f2ab80d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 -- 172.21.15.2:0/3816009822 shutdown_connections 2024-01-15T18:10:33.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.616+0000 7f2abea00640 1 -- 172.21.15.2:0/3816009822 wait complete. 2024-01-15T18:10:33.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.618+0000 7f2abea00640 1 Processor -- start 2024-01-15T18:10:33.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.618+0000 7f2abea00640 1 -- start start 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ab80cfad0 0x7f2ab80ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ab80d37f0 0x7f2ab8150e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 0x7f2ab815f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abea00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80ca6f0 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab8151370 con 0x7f2ab80d37f0 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab80ca990 con 0x7f2ab80cfad0 2024-01-15T18:10:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abe000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 0x7f2ab815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abe000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 0x7f2ab815f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55690/0 (socket says 172.21.15.2:55690) 2024-01-15T18:10:33.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abe000640 1 -- 172.21.15.2:0/14772149 learned_addr learned my addr 172.21.15.2:0/14772149 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abd600640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ab80cfad0 0x7f2ab80ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.619+0000 7f2abcc00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ab80d37f0 0x7f2ab8150e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abe000640 1 -- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ab80cfad0 msgr2=0x7f2ab80ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abe000640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ab80cfad0 0x7f2ab80ca1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abe000640 1 -- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ab80d37f0 msgr2=0x7f2ab8150e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abe000640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ab80d37f0 0x7f2ab8150e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abe000640 1 -- 172.21.15.2:0/14772149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa0009400 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abd600640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ab80cfad0 0x7f2ab80ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:33.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abcc00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ab80d37f0 0x7f2ab8150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abe000640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 0x7f2ab815f070 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000c7e0 tx=0x7f2ab000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000a640 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab000a7c0 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab80cac70 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000ab70 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.620+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab80cb230 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.621+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab8074770 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.622+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f2ab000acd0 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.624+0000 7f2ab5600640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2a9006bfe0 0x7f2a9006e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:33.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.624+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f2ab0087a30 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.624+0000 7f2abd600640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2a9006bfe0 0x7f2a9006e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:33.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.626+0000 7f2abd600640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2a9006bfe0 0x7f2a9006e4a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800a830 tx=0x7f2aa8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:33.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.628+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ab0054350 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.821+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ab8067650 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.823+0000 7f2ab5600640 1 -- 172.21.15.2:0/14772149 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f2ab0018090 con 0x7f2ab80d3ca0 2024-01-15T18:10:33.826 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:33.826 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":89,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:10:30.874409+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:10:27.147601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"88","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-15T18:10:28.332573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.825+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2a9006bfe0 msgr2=0x7f2a9006e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.825+0000 7f2abea00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2a9006bfe0 0x7f2a9006e4a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f2aa800a830 tx=0x7f2aa8008040 comp rx=0 tx=0).stop 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.825+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 msgr2=0x7f2ab815f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.825+0000 7f2abea00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 0x7f2ab815f070 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000c7e0 tx=0x7f2ab000cca0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 shutdown_connections 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2a9006bfe0 0x7f2a9006e4a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ab80d3ca0 0x7f2ab815f070 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ab80d37f0 0x7f2ab8150e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 --2- 172.21.15.2:0/14772149 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ab80cfad0 0x7f2ab80ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:33.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 >> 172.21.15.2:0/14772149 conn(0x7f2ab809c3b0 msgr2=0x7f2ab80b82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:33.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 shutdown_connections 2024-01-15T18:10:33.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:33.826+0000 7f2abea00640 1 -- 172.21.15.2:0/14772149 wait complete. 2024-01-15T18:10:33.846 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:10:33.846 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:10:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.035+0000 7f1242800640 1 Processor -- start 2024-01-15T18:10:34.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.035+0000 7f1242800640 1 -- start start 2024-01-15T18:10:34.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.037+0000 7f1242800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f123c15d340 0x7f123c0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.037+0000 7f1242800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.037+0000 7f1242800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f123c0d3a30 con 0x7f123c0d1680 2024-01-15T18:10:34.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.037+0000 7f1242800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f123c0b3de0 con 0x7f123c15d340 2024-01-15T18:10:34.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.037+0000 7f1242800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c0b40c0 con 0x7f123c0d3090 2024-01-15T18:10:34.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f123c15d340 0x7f123c0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1240a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f123c0d1680 0x7f123c0d1b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:57076/0 (socket says 172.21.15.2:57076) 2024-01-15T18:10:34.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241e00640 1 -- 172.21.15.2:0/3995179420 learned_addr learned my addr 172.21.15.2:0/3995179420 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241400640 1 -- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 msgr2=0x7f123c0d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241400640 1 --2- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0d34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241400640 1 -- 172.21.15.2:0/3995179420 >> v1:172.21.15.137:6789/0 conn(0x7f123c0d1680 legacy=0x7f123c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.038+0000 7f1241400640 1 -- 172.21.15.2:0/3995179420 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f123c0d0e80 con 0x7f123c15d340 2024-01-15T18:10:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.040+0000 7f1241400640 1 --2- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c15d340 0x7f123c0cdd20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f122c0097a0 tx=0x7f122c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29d356babc792f9c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.041+0000 7f123be00640 1 -- 172.21.15.2:0/3995179420 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c031830 con 0x7f123c15d340 2024-01-15T18:10:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.041+0000 7f123be00640 1 -- 172.21.15.2:0/3995179420 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f122c031990 con 0x7f123c15d340 2024-01-15T18:10:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.041+0000 7f123be00640 1 -- 172.21.15.2:0/3995179420 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f122c031d20 con 0x7f123c15d340 2024-01-15T18:10:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.041+0000 7f1242800640 1 -- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c15d340 msgr2=0x7f123c0cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.041+0000 7f1242800640 1 --2- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c15d340 0x7f123c0cdd20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f122c0097a0 tx=0x7f122c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.042+0000 7f1242800640 1 -- 172.21.15.2:0/3995179420 shutdown_connections 2024-01-15T18:10:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.042+0000 7f1242800640 1 --2- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.042+0000 7f1242800640 1 --2- 172.21.15.2:0/3995179420 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c15d340 0x7f123c0cdd20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.042+0000 7f1242800640 1 -- 172.21.15.2:0/3995179420 >> 172.21.15.2:0/3995179420 conn(0x7f123c08caf0 msgr2=0x7f123c09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.042+0000 7f1242800640 1 -- 172.21.15.2:0/3995179420 shutdown_connections 2024-01-15T18:10:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.042+0000 7f1242800640 1 -- 172.21.15.2:0/3995179420 wait complete. 2024-01-15T18:10:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.044+0000 7f1242800640 1 Processor -- start 2024-01-15T18:10:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.044+0000 7f1242800640 1 -- start start 2024-01-15T18:10:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.045+0000 7f1242800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c0d1680 0x7f123c1587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.045+0000 7f1242800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.045+0000 7f1242800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 0x7f123c0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.045+0000 7f1242800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c1f4230 con 0x7f123c15d340 2024-01-15T18:10:34.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1242800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f123c0c7960 con 0x7f123c0d1680 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1242800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f123c1f3ef0 con 0x7f123c0d3090 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1241e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 0x7f123c0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1241e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 0x7f123c0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55694/0 (socket says 172.21.15.2:55694) 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1241e00640 1 -- 172.21.15.2:0/3932522304 learned_addr learned my addr 172.21.15.2:0/3932522304 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1241400640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c0d1680 0x7f123c1587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.046+0000 7f1240a00640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1241e00640 1 -- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 msgr2=0x7f123c0ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1241e00640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1241e00640 1 -- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c0d1680 msgr2=0x7f123c1587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1241e00640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c0d1680 0x7f123c1587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1241e00640 1 -- 172.21.15.2:0/3932522304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f122c009400 con 0x7f123c15d340 2024-01-15T18:10:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1241e00640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 0x7f123c0ceaa0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f123400ca80 tx=0x7f123400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:34.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123400ac30 con 0x7f123c15d340 2024-01-15T18:10:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123400ad90 con 0x7f123c15d340 2024-01-15T18:10:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1234012600 con 0x7f123c15d340 2024-01-15T18:10:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.047+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f123c0c7c00 con 0x7f123c15d340 2024-01-15T18:10:34.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.048+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f123c0c81c0 con 0x7f123c15d340 2024-01-15T18:10:34.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.049+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f123c0c83e0 con 0x7f123c15d340 2024-01-15T18:10:34.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.050+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1234007070 con 0x7f123c15d340 2024-01-15T18:10:34.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.051+0000 7f1239600640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f121406bfb0 0x7f121406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.051+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1234088390 con 0x7f123c15d340 2024-01-15T18:10:34.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.051+0000 7f1241400640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f121406bfb0 0x7f121406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.052+0000 7f1241400640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f121406bfb0 0x7f121406e470 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f122c036000 tx=0x7f122c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:34.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.055+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1234054d00 con 0x7f123c15d340 2024-01-15T18:10:34.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.246+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f123c0ce260 con 0x7f121406bfb0 2024-01-15T18:10:34.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.274+0000 7f1239600640 1 -- 172.21.15.2:0/3932522304 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282200 (secure 0 0 0) 0x7f122c03a420 con 0x7f121406bfb0 2024-01-15T18:10:34.281 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:34.283 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f121406bfb0 msgr2=0x7f121406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f121406bfb0 0x7f121406e470 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f122c036000 tx=0x7f122c03c040 comp rx=0 tx=0).stop 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 msgr2=0x7f123c0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 0x7f123c0ceaa0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f123400ca80 tx=0x7f123400cf40 comp rx=0 tx=0).stop 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 shutdown_connections 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f121406bfb0 0x7f121406e470 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f123c15d340 0x7f123c0ceaa0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f123c0d3090 0x7f123c0ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 --2- 172.21.15.2:0/3932522304 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f123c0d1680 0x7f123c1587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.280+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 >> 172.21.15.2:0/3932522304 conn(0x7f123c08caf0 msgr2=0x7f123c06b520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:34.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.281+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 shutdown_connections 2024-01-15T18:10:34.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.281+0000 7f1242800640 1 -- 172.21.15.2:0/3932522304 wait complete. 2024-01-15T18:10:34.304 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-01-15T18:10:33.713319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":522368,"kb_used_data":9480,"kb_used_omap":12,"kb_used_meta":512755,"kb_avail":754452352,"statfs":{"total":773094113280,"available":772559208448,"internally_reserved":0,"allocated":9707520,"data_stored":4849976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":525061681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.884985"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.853366+0000","last_change":"2024-01-15T18:10:28.853366+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.853366+0000","last_undegraded":"2024-01-15T18:10:28.853366+0000","last_fullsized":"2024-01-15T18:10:28.853366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845232+0000","last_change":"2024-01-15T18:10:28.845232+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845232+0000","last_undegraded":"2024-01-15T18:10:28.845232+0000","last_fullsized":"2024-01-15T18:10:28.845232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.889646+0000","last_change":"2024-01-15T18:10:29.888180+0000","last_active":"2024-01-15T18:10:29.889646+0000","last_peered":"2024-01-15T18:10:29.889646+0000","last_clean":"2024-01-15T18:10:29.889646+0000","last_became_active":"2024-01-15T18:10:29.886925+0000","last_became_peered":"2024-01-15T18:10:29.886925+0000","last_unstale":"2024-01-15T18:10:29.889646+0000","last_undegraded":"2024-01-15T18:10:29.889646+0000","last_fullsized":"2024-01-15T18:10:29.889646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:18.035419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841871+0000","last_change":"2024-01-15T18:10:28.841871+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841871+0000","last_undegraded":"2024-01-15T18:10:28.841871+0000","last_fullsized":"2024-01-15T18:10:28.841871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854388+0000","last_change":"2024-01-15T18:10:28.854388+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854388+0000","last_undegraded":"2024-01-15T18:10:28.854388+0000","last_fullsized":"2024-01-15T18:10:28.854388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843476+0000","last_change":"2024-01-15T18:10:28.843476+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843476+0000","last_undegraded":"2024-01-15T18:10:28.843476+0000","last_fullsized":"2024-01-15T18:10:28.843476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.844115+0000","last_change":"2024-01-15T18:10:28.844115+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.844115+0000","last_undegraded":"2024-01-15T18:10:28.844115+0000","last_fullsized":"2024-01-15T18:10:28.844115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838777+0000","last_change":"2024-01-15T18:10:28.838777+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838777+0000","last_undegraded":"2024-01-15T18:10:28.838777+0000","last_fullsized":"2024-01-15T18:10:28.838777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841145+0000","last_change":"2024-01-15T18:10:28.841145+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841145+0000","last_undegraded":"2024-01-15T18:10:28.841145+0000","last_fullsized":"2024-01-15T18:10:28.841145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846140+0000","last_change":"2024-01-15T18:10:28.846140+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846140+0000","last_undegraded":"2024-01-15T18:10:28.846140+0000","last_fullsized":"2024-01-15T18:10:28.846140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840656+0000","last_change":"2024-01-15T18:10:28.840656+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840656+0000","last_undegraded":"2024-01-15T18:10:28.840656+0000","last_fullsized":"2024-01-15T18:10:28.840656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.848480+0000","last_change":"2024-01-15T18:10:28.848480+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.848480+0000","last_undegraded":"2024-01-15T18:10:28.848480+0000","last_fullsized":"2024-01-15T18:10:28.848480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.888056+0000","last_change":"2024-01-15T18:10:29.884896+0000","last_active":"2024-01-15T18:10:29.888056+0000","last_peered":"2024-01-15T18:10:29.888056+0000","last_clean":"2024-01-15T18:10:29.888056+0000","last_became_active":"2024-01-15T18:10:29.883901+0000","last_became_peered":"2024-01-15T18:10:29.883901+0000","last_unstale":"2024-01-15T18:10:29.888056+0000","last_undegraded":"2024-01-15T18:10:29.888056+0000","last_fullsized":"2024-01-15T18:10:29.888056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:57.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840579+0000","last_change":"2024-01-15T18:10:28.840579+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840579+0000","last_undegraded":"2024-01-15T18:10:28.840579+0000","last_fullsized":"2024-01-15T18:10:28.840579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.842561+0000","last_change":"2024-01-15T18:10:28.842561+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.842561+0000","last_undegraded":"2024-01-15T18:10:28.842561+0000","last_fullsized":"2024-01-15T18:10:28.842561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.847589+0000","last_change":"2024-01-15T18:10:28.847589+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.847589+0000","last_undegraded":"2024-01-15T18:10:28.847589+0000","last_fullsized":"2024-01-15T18:10:28.847589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.852113+0000","last_change":"2024-01-15T18:10:28.852113+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.852113+0000","last_undegraded":"2024-01-15T18:10:28.852113+0000","last_fullsized":"2024-01-15T18:10:28.852113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843978+0000","last_change":"2024-01-15T18:10:28.843978+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843978+0000","last_undegraded":"2024-01-15T18:10:28.843978+0000","last_fullsized":"2024-01-15T18:10:28.843978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840150+0000","last_change":"2024-01-15T18:10:28.840150+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840150+0000","last_undegraded":"2024-01-15T18:10:28.840150+0000","last_fullsized":"2024-01-15T18:10:28.840150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840702+0000","last_change":"2024-01-15T18:10:28.840702+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840702+0000","last_undegraded":"2024-01-15T18:10:28.840702+0000","last_fullsized":"2024-01-15T18:10:28.840702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843143+0000","last_change":"2024-01-15T18:10:28.843143+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843143+0000","last_undegraded":"2024-01-15T18:10:28.843143+0000","last_fullsized":"2024-01-15T18:10:28.843143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845765+0000","last_change":"2024-01-15T18:10:28.845765+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845765+0000","last_undegraded":"2024-01-15T18:10:28.845765+0000","last_fullsized":"2024-01-15T18:10:28.845765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854373+0000","last_change":"2024-01-15T18:10:28.854373+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854373+0000","last_undegraded":"2024-01-15T18:10:28.854373+0000","last_fullsized":"2024-01-15T18:10:28.854373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838275+0000","last_change":"2024-01-15T18:10:28.838275+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838275+0000","last_undegraded":"2024-01-15T18:10:28.838275+0000","last_fullsized":"2024-01-15T18:10:28.838275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838843+0000","last_change":"2024-01-15T18:10:28.838843+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838843+0000","last_undegraded":"2024-01-15T18:10:28.838843+0000","last_fullsized":"2024-01-15T18:10:28.838843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886570+0000","last_change":"2024-01-15T18:10:29.879335+0000","last_active":"2024-01-15T18:10:29.886570+0000","last_peered":"2024-01-15T18:10:29.886570+0000","last_clean":"2024-01-15T18:10:29.886570+0000","last_became_active":"2024-01-15T18:10:29.878999+0000","last_became_peered":"2024-01-15T18:10:29.878999+0000","last_unstale":"2024-01-15T18:10:29.886570+0000","last_undegraded":"2024-01-15T18:10:29.886570+0000","last_fullsized":"2024-01-15T18:10:29.886570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.857241+0000","last_change":"2024-01-15T18:10:28.857241+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.857241+0000","last_undegraded":"2024-01-15T18:10:28.857241+0000","last_fullsized":"2024-01-15T18:10:28.857241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.847343+0000","last_change":"2024-01-15T18:10:28.847343+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.847343+0000","last_undegraded":"2024-01-15T18:10:28.847343+0000","last_fullsized":"2024-01-15T18:10:28.847343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840119+0000","last_change":"2024-01-15T18:10:28.840119+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840119+0000","last_undegraded":"2024-01-15T18:10:28.840119+0000","last_fullsized":"2024-01-15T18:10:28.840119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845208+0000","last_change":"2024-01-15T18:10:28.845208+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845208+0000","last_undegraded":"2024-01-15T18:10:28.845208+0000","last_fullsized":"2024-01-15T18:10:28.845208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846185+0000","last_change":"2024-01-15T18:10:28.846185+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846185+0000","last_undegraded":"2024-01-15T18:10:28.846185+0000","last_fullsized":"2024-01-15T18:10:28.846185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846205+0000","last_change":"2024-01-15T18:10:28.846205+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846205+0000","last_undegraded":"2024-01-15T18:10:28.846205+0000","last_fullsized":"2024-01-15T18:10:28.846205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856360+0000","last_change":"2024-01-15T18:10:28.856360+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856360+0000","last_undegraded":"2024-01-15T18:10:28.856360+0000","last_fullsized":"2024-01-15T18:10:28.856360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864067+0000","last_change":"2024-01-15T18:10:28.860747+0000","last_active":"2024-01-15T18:10:28.864067+0000","last_peered":"2024-01-15T18:10:28.864067+0000","last_clean":"2024-01-15T18:10:28.864067+0000","last_became_active":"2024-01-15T18:10:28.860414+0000","last_became_peered":"2024-01-15T18:10:28.860414+0000","last_unstale":"2024-01-15T18:10:28.864067+0000","last_undegraded":"2024-01-15T18:10:28.864067+0000","last_fullsized":"2024-01-15T18:10:28.864067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:51.403865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.852905+0000","last_change":"2024-01-15T18:10:28.852905+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.852905+0000","last_undegraded":"2024-01-15T18:10:28.852905+0000","last_fullsized":"2024-01-15T18:10:28.852905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865034+0000","last_change":"2024-01-15T18:10:28.861707+0000","last_active":"2024-01-15T18:10:28.865034+0000","last_peered":"2024-01-15T18:10:28.865034+0000","last_clean":"2024-01-15T18:10:28.865034+0000","last_became_active":"2024-01-15T18:10:28.860381+0000","last_became_peered":"2024-01-15T18:10:28.860381+0000","last_unstale":"2024-01-15T18:10:28.865034+0000","last_undegraded":"2024-01-15T18:10:28.865034+0000","last_fullsized":"2024-01-15T18:10:28.865034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:47.922260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846213+0000","last_change":"2024-01-15T18:10:28.846213+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846213+0000","last_undegraded":"2024-01-15T18:10:28.846213+0000","last_fullsized":"2024-01-15T18:10:28.846213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857292+0000","last_change":"2024-01-15T18:10:28.851518+0000","last_active":"2024-01-15T18:10:28.857292+0000","last_peered":"2024-01-15T18:10:28.857292+0000","last_clean":"2024-01-15T18:10:28.857292+0000","last_became_active":"2024-01-15T18:10:28.851216+0000","last_became_peered":"2024-01-15T18:10:28.851216+0000","last_unstale":"2024-01-15T18:10:28.857292+0000","last_undegraded":"2024-01-15T18:10:28.857292+0000","last_fullsized":"2024-01-15T18:10:28.857292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:12.969455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.892135+0000","last_change":"2024-01-15T18:10:29.890269+0000","last_active":"2024-01-15T18:10:29.892135+0000","last_peered":"2024-01-15T18:10:29.892135+0000","last_clean":"2024-01-15T18:10:29.892135+0000","last_became_active":"2024-01-15T18:10:29.889995+0000","last_became_peered":"2024-01-15T18:10:29.889995+0000","last_unstale":"2024-01-15T18:10:29.892135+0000","last_undegraded":"2024-01-15T18:10:29.892135+0000","last_fullsized":"2024-01-15T18:10:29.892135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:26.719685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.854427+0000","last_change":"2024-01-15T18:10:28.847695+0000","last_active":"2024-01-15T18:10:28.854427+0000","last_peered":"2024-01-15T18:10:28.854427+0000","last_clean":"2024-01-15T18:10:28.854427+0000","last_became_active":"2024-01-15T18:10:28.847366+0000","last_became_peered":"2024-01-15T18:10:28.847366+0000","last_unstale":"2024-01-15T18:10:28.854427+0000","last_undegraded":"2024-01-15T18:10:28.854427+0000","last_fullsized":"2024-01-15T18:10:28.854427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:05.863393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843192+0000","last_change":"2024-01-15T18:10:28.843192+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843192+0000","last_undegraded":"2024-01-15T18:10:28.843192+0000","last_fullsized":"2024-01-15T18:10:28.843192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.863872+0000","last_change":"2024-01-15T18:10:28.860229+0000","last_active":"2024-01-15T18:10:28.863872+0000","last_peered":"2024-01-15T18:10:28.863872+0000","last_clean":"2024-01-15T18:10:28.863872+0000","last_became_active":"2024-01-15T18:10:28.859548+0000","last_became_peered":"2024-01-15T18:10:28.859548+0000","last_unstale":"2024-01-15T18:10:28.863872+0000","last_undegraded":"2024-01-15T18:10:28.863872+0000","last_fullsized":"2024-01-15T18:10:28.863872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:37.763893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.851504+0000","last_change":"2024-01-15T18:10:28.851504+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.851504+0000","last_undegraded":"2024-01-15T18:10:28.851504+0000","last_fullsized":"2024-01-15T18:10:28.851504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864540+0000","last_change":"2024-01-15T18:10:28.860903+0000","last_active":"2024-01-15T18:10:28.864540+0000","last_peered":"2024-01-15T18:10:28.864540+0000","last_clean":"2024-01-15T18:10:28.864540+0000","last_became_active":"2024-01-15T18:10:28.860595+0000","last_became_peered":"2024-01-15T18:10:28.860595+0000","last_unstale":"2024-01-15T18:10:28.864540+0000","last_undegraded":"2024-01-15T18:10:28.864540+0000","last_fullsized":"2024-01-15T18:10:28.864540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:15:54.741452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874796+0000","last_change":"2024-01-15T18:10:29.871081+0000","last_active":"2024-01-15T18:10:29.874796+0000","last_peered":"2024-01-15T18:10:29.874796+0000","last_clean":"2024-01-15T18:10:29.874796+0000","last_became_active":"2024-01-15T18:10:29.870386+0000","last_became_peered":"2024-01-15T18:10:29.870386+0000","last_unstale":"2024-01-15T18:10:29.874796+0000","last_undegraded":"2024-01-15T18:10:29.874796+0000","last_fullsized":"2024-01-15T18:10:29.874796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:25.828362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857132+0000","last_change":"2024-01-15T18:10:28.856007+0000","last_active":"2024-01-15T18:10:28.857132+0000","last_peered":"2024-01-15T18:10:28.857132+0000","last_clean":"2024-01-15T18:10:28.857132+0000","last_became_active":"2024-01-15T18:10:28.855729+0000","last_became_peered":"2024-01-15T18:10:28.855729+0000","last_unstale":"2024-01-15T18:10:28.857132+0000","last_undegraded":"2024-01-15T18:10:28.857132+0000","last_fullsized":"2024-01-15T18:10:28.857132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:02.736339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.842052+0000","last_change":"2024-01-15T18:10:28.842052+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.842052+0000","last_undegraded":"2024-01-15T18:10:28.842052+0000","last_fullsized":"2024-01-15T18:10:28.842052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.855932+0000","last_change":"2024-01-15T18:10:28.854028+0000","last_active":"2024-01-15T18:10:28.855932+0000","last_peered":"2024-01-15T18:10:28.855932+0000","last_clean":"2024-01-15T18:10:28.855932+0000","last_became_active":"2024-01-15T18:10:28.853569+0000","last_became_peered":"2024-01-15T18:10:28.853569+0000","last_unstale":"2024-01-15T18:10:28.855932+0000","last_undegraded":"2024-01-15T18:10:28.855932+0000","last_fullsized":"2024-01-15T18:10:28.855932+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:42.155021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841682+0000","last_change":"2024-01-15T18:10:28.841682+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841682+0000","last_undegraded":"2024-01-15T18:10:28.841682+0000","last_fullsized":"2024-01-15T18:10:28.841682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865184+0000","last_change":"2024-01-15T18:10:28.862577+0000","last_active":"2024-01-15T18:10:28.865184+0000","last_peered":"2024-01-15T18:10:28.865184+0000","last_clean":"2024-01-15T18:10:28.865184+0000","last_became_active":"2024-01-15T18:10:28.862234+0000","last_became_peered":"2024-01-15T18:10:28.862234+0000","last_unstale":"2024-01-15T18:10:28.865184+0000","last_undegraded":"2024-01-15T18:10:28.865184+0000","last_fullsized":"2024-01-15T18:10:28.865184+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:13:15.657582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840553+0000","last_change":"2024-01-15T18:10:28.840553+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840553+0000","last_undegraded":"2024-01-15T18:10:28.840553+0000","last_fullsized":"2024-01-15T18:10:28.840553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861386+0000","last_change":"2024-01-15T18:10:28.859089+0000","last_active":"2024-01-15T18:10:28.861386+0000","last_peered":"2024-01-15T18:10:28.861386+0000","last_clean":"2024-01-15T18:10:28.861386+0000","last_became_active":"2024-01-15T18:10:28.858772+0000","last_became_peered":"2024-01-15T18:10:28.858772+0000","last_unstale":"2024-01-15T18:10:28.861386+0000","last_undegraded":"2024-01-15T18:10:28.861386+0000","last_fullsized":"2024-01-15T18:10:28.861386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:36.585906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840915+0000","last_change":"2024-01-15T18:10:28.840915+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840915+0000","last_undegraded":"2024-01-15T18:10:28.840915+0000","last_fullsized":"2024-01-15T18:10:28.840915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865505+0000","last_change":"2024-01-15T18:10:28.864142+0000","last_active":"2024-01-15T18:10:28.865505+0000","last_peered":"2024-01-15T18:10:28.865505+0000","last_clean":"2024-01-15T18:10:28.865505+0000","last_became_active":"2024-01-15T18:10:28.863733+0000","last_became_peered":"2024-01-15T18:10:28.863733+0000","last_unstale":"2024-01-15T18:10:28.865505+0000","last_undegraded":"2024-01-15T18:10:28.865505+0000","last_fullsized":"2024-01-15T18:10:28.865505+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:26.704112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856623+0000","last_change":"2024-01-15T18:10:28.856623+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856623+0000","last_undegraded":"2024-01-15T18:10:28.856623+0000","last_fullsized":"2024-01-15T18:10:28.856623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861826+0000","last_change":"2024-01-15T18:10:28.852757+0000","last_active":"2024-01-15T18:10:28.861826+0000","last_peered":"2024-01-15T18:10:28.861826+0000","last_clean":"2024-01-15T18:10:28.861826+0000","last_became_active":"2024-01-15T18:10:28.852416+0000","last_became_peered":"2024-01-15T18:10:28.852416+0000","last_unstale":"2024-01-15T18:10:28.861826+0000","last_undegraded":"2024-01-15T18:10:28.861826+0000","last_fullsized":"2024-01-15T18:10:28.861826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:37.726569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839982+0000","last_change":"2024-01-15T18:10:28.839982+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839982+0000","last_undegraded":"2024-01-15T18:10:28.839982+0000","last_fullsized":"2024-01-15T18:10:28.839982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847523+0000","last_change":"2024-01-15T18:10:28.846674+0000","last_active":"2024-01-15T18:10:28.847523+0000","last_peered":"2024-01-15T18:10:28.847523+0000","last_clean":"2024-01-15T18:10:28.847523+0000","last_became_active":"2024-01-15T18:10:28.845875+0000","last_became_peered":"2024-01-15T18:10:28.845875+0000","last_unstale":"2024-01-15T18:10:28.847523+0000","last_undegraded":"2024-01-15T18:10:28.847523+0000","last_fullsized":"2024-01-15T18:10:28.847523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:40.717853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.884387+0000","last_change":"2024-01-15T18:10:29.876230+0000","last_active":"2024-01-15T18:10:29.884387+0000","last_peered":"2024-01-15T18:10:29.884387+0000","last_clean":"2024-01-15T18:10:29.884387+0000","last_became_active":"2024-01-15T18:10:29.874419+0000","last_became_peered":"2024-01-15T18:10:29.874419+0000","last_unstale":"2024-01-15T18:10:29.884387+0000","last_undegraded":"2024-01-15T18:10:29.884387+0000","last_fullsized":"2024-01-15T18:10:29.884387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:57.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859460+0000","last_change":"2024-01-15T18:10:28.855952+0000","last_active":"2024-01-15T18:10:28.859460+0000","last_peered":"2024-01-15T18:10:28.859460+0000","last_clean":"2024-01-15T18:10:28.859460+0000","last_became_active":"2024-01-15T18:10:28.855647+0000","last_became_peered":"2024-01-15T18:10:28.855647+0000","last_unstale":"2024-01-15T18:10:28.859460+0000","last_undegraded":"2024-01-15T18:10:28.859460+0000","last_fullsized":"2024-01-15T18:10:28.859460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:17.019853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838341+0000","last_change":"2024-01-15T18:10:28.838341+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838341+0000","last_undegraded":"2024-01-15T18:10:28.838341+0000","last_fullsized":"2024-01-15T18:10:28.838341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861907+0000","last_change":"2024-01-15T18:10:28.853310+0000","last_active":"2024-01-15T18:10:28.861907+0000","last_peered":"2024-01-15T18:10:28.861907+0000","last_clean":"2024-01-15T18:10:28.861907+0000","last_became_active":"2024-01-15T18:10:28.853030+0000","last_became_peered":"2024-01-15T18:10:28.853030+0000","last_unstale":"2024-01-15T18:10:28.861907+0000","last_undegraded":"2024-01-15T18:10:28.861907+0000","last_fullsized":"2024-01-15T18:10:28.861907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:41.304535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839292+0000","last_change":"2024-01-15T18:10:28.839292+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839292+0000","last_undegraded":"2024-01-15T18:10:28.839292+0000","last_fullsized":"2024-01-15T18:10:28.839292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.858606+0000","last_change":"2024-01-15T18:10:28.856512+0000","last_active":"2024-01-15T18:10:28.858606+0000","last_peered":"2024-01-15T18:10:28.858606+0000","last_clean":"2024-01-15T18:10:28.858606+0000","last_became_active":"2024-01-15T18:10:28.855528+0000","last_became_peered":"2024-01-15T18:10:28.855528+0000","last_unstale":"2024-01-15T18:10:28.858606+0000","last_undegraded":"2024-01-15T18:10:28.858606+0000","last_fullsized":"2024-01-15T18:10:28.858606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:51.724767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886591+0000","last_change":"2024-01-15T18:10:29.881440+0000","last_active":"2024-01-15T18:10:29.886591+0000","last_peered":"2024-01-15T18:10:29.886591+0000","last_clean":"2024-01-15T18:10:29.886591+0000","last_became_active":"2024-01-15T18:10:29.881163+0000","last_became_peered":"2024-01-15T18:10:29.881163+0000","last_unstale":"2024-01-15T18:10:29.886591+0000","last_undegraded":"2024-01-15T18:10:29.886591+0000","last_fullsized":"2024-01-15T18:10:29.886591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:10.131197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865460+0000","last_change":"2024-01-15T18:10:28.864218+0000","last_active":"2024-01-15T18:10:28.865460+0000","last_peered":"2024-01-15T18:10:28.865460+0000","last_clean":"2024-01-15T18:10:28.865460+0000","last_became_active":"2024-01-15T18:10:28.863828+0000","last_became_peered":"2024-01-15T18:10:28.863828+0000","last_unstale":"2024-01-15T18:10:28.865460+0000","last_undegraded":"2024-01-15T18:10:28.865460+0000","last_fullsized":"2024-01-15T18:10:28.865460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:20.078184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.851348+0000","last_change":"2024-01-15T18:10:28.851348+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.851348+0000","last_undegraded":"2024-01-15T18:10:28.851348+0000","last_fullsized":"2024-01-15T18:10:28.851348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.860686+0000","last_change":"2024-01-15T18:10:28.859456+0000","last_active":"2024-01-15T18:10:28.860686+0000","last_peered":"2024-01-15T18:10:28.860686+0000","last_clean":"2024-01-15T18:10:28.860686+0000","last_became_active":"2024-01-15T18:10:28.858710+0000","last_became_peered":"2024-01-15T18:10:28.858710+0000","last_unstale":"2024-01-15T18:10:28.860686+0000","last_undegraded":"2024-01-15T18:10:28.860686+0000","last_fullsized":"2024-01-15T18:10:28.860686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:16.178426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856094+0000","last_change":"2024-01-15T18:10:28.856094+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856094+0000","last_undegraded":"2024-01-15T18:10:28.856094+0000","last_fullsized":"2024-01-15T18:10:28.856094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859561+0000","last_change":"2024-01-15T18:10:28.856970+0000","last_active":"2024-01-15T18:10:28.859561+0000","last_peered":"2024-01-15T18:10:28.859561+0000","last_clean":"2024-01-15T18:10:28.859561+0000","last_became_active":"2024-01-15T18:10:28.856677+0000","last_became_peered":"2024-01-15T18:10:28.856677+0000","last_unstale":"2024-01-15T18:10:28.859561+0000","last_undegraded":"2024-01-15T18:10:28.859561+0000","last_fullsized":"2024-01-15T18:10:28.859561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:41:56.867321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837448+0000","last_change":"2024-01-15T18:10:28.837448+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837448+0000","last_undegraded":"2024-01-15T18:10:28.837448+0000","last_fullsized":"2024-01-15T18:10:28.837448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857674+0000","last_change":"2024-01-15T18:10:28.853663+0000","last_active":"2024-01-15T18:10:28.857674+0000","last_peered":"2024-01-15T18:10:28.857674+0000","last_clean":"2024-01-15T18:10:28.857674+0000","last_became_active":"2024-01-15T18:10:28.849935+0000","last_became_peered":"2024-01-15T18:10:28.849935+0000","last_unstale":"2024-01-15T18:10:28.857674+0000","last_undegraded":"2024-01-15T18:10:28.857674+0000","last_fullsized":"2024-01-15T18:10:28.857674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:09.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839003+0000","last_change":"2024-01-15T18:10:28.839003+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839003+0000","last_undegraded":"2024-01-15T18:10:28.839003+0000","last_fullsized":"2024-01-15T18:10:28.839003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.852250+0000","last_change":"2024-01-15T18:10:28.848970+0000","last_active":"2024-01-15T18:10:28.852250+0000","last_peered":"2024-01-15T18:10:28.852250+0000","last_clean":"2024-01-15T18:10:28.852250+0000","last_became_active":"2024-01-15T18:10:28.848694+0000","last_became_peered":"2024-01-15T18:10:28.848694+0000","last_unstale":"2024-01-15T18:10:28.852250+0000","last_undegraded":"2024-01-15T18:10:28.852250+0000","last_fullsized":"2024-01-15T18:10:28.852250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:10.841154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837767+0000","last_change":"2024-01-15T18:10:28.837767+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837767+0000","last_undegraded":"2024-01-15T18:10:28.837767+0000","last_fullsized":"2024-01-15T18:10:28.837767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864081+0000","last_change":"2024-01-15T18:10:28.863335+0000","last_active":"2024-01-15T18:10:28.864081+0000","last_peered":"2024-01-15T18:10:28.864081+0000","last_clean":"2024-01-15T18:10:28.864081+0000","last_became_active":"2024-01-15T18:10:28.862934+0000","last_became_peered":"2024-01-15T18:10:28.862934+0000","last_unstale":"2024-01-15T18:10:28.864081+0000","last_undegraded":"2024-01-15T18:10:28.864081+0000","last_fullsized":"2024-01-15T18:10:28.864081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:22.153917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840868+0000","last_change":"2024-01-15T18:10:28.840868+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840868+0000","last_undegraded":"2024-01-15T18:10:28.840868+0000","last_fullsized":"2024-01-15T18:10:28.840868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857659+0000","last_change":"2024-01-15T18:10:28.853080+0000","last_active":"2024-01-15T18:10:28.857659+0000","last_peered":"2024-01-15T18:10:28.857659+0000","last_clean":"2024-01-15T18:10:28.857659+0000","last_became_active":"2024-01-15T18:10:28.848348+0000","last_became_peered":"2024-01-15T18:10:28.848348+0000","last_unstale":"2024-01-15T18:10:28.857659+0000","last_undegraded":"2024-01-15T18:10:28.857659+0000","last_fullsized":"2024-01-15T18:10:28.857659+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:16.924785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854397+0000","last_change":"2024-01-15T18:10:28.854397+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854397+0000","last_undegraded":"2024-01-15T18:10:28.854397+0000","last_fullsized":"2024-01-15T18:10:28.854397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.856697+0000","last_change":"2024-01-15T18:10:28.853859+0000","last_active":"2024-01-15T18:10:28.856697+0000","last_peered":"2024-01-15T18:10:28.856697+0000","last_clean":"2024-01-15T18:10:28.856697+0000","last_became_active":"2024-01-15T18:10:28.853567+0000","last_became_peered":"2024-01-15T18:10:28.853567+0000","last_unstale":"2024-01-15T18:10:28.856697+0000","last_undegraded":"2024-01-15T18:10:28.856697+0000","last_fullsized":"2024-01-15T18:10:28.856697+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:42.094019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.891545+0000","last_change":"2024-01-15T18:10:29.887191+0000","last_active":"2024-01-15T18:10:29.891545+0000","last_peered":"2024-01-15T18:10:29.891545+0000","last_clean":"2024-01-15T18:10:29.891545+0000","last_became_active":"2024-01-15T18:10:29.886890+0000","last_became_peered":"2024-01-15T18:10:29.886890+0000","last_unstale":"2024-01-15T18:10:29.891545+0000","last_undegraded":"2024-01-15T18:10:29.891545+0000","last_fullsized":"2024-01-15T18:10:29.891545+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847836+0000","last_change":"2024-01-15T18:10:28.847096+0000","last_active":"2024-01-15T18:10:28.847836+0000","last_peered":"2024-01-15T18:10:28.847836+0000","last_clean":"2024-01-15T18:10:28.847836+0000","last_became_active":"2024-01-15T18:10:28.845910+0000","last_became_peered":"2024-01-15T18:10:28.845910+0000","last_unstale":"2024-01-15T18:10:28.847836+0000","last_undegraded":"2024-01-15T18:10:28.847836+0000","last_fullsized":"2024-01-15T18:10:28.847836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:29.009395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.844184+0000","last_change":"2024-01-15T18:10:28.844184+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.844184+0000","last_undegraded":"2024-01-15T18:10:28.844184+0000","last_fullsized":"2024-01-15T18:10:28.844184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859604+0000","last_change":"2024-01-15T18:10:28.853688+0000","last_active":"2024-01-15T18:10:28.859604+0000","last_peered":"2024-01-15T18:10:28.859604+0000","last_clean":"2024-01-15T18:10:28.859604+0000","last_became_active":"2024-01-15T18:10:28.853347+0000","last_became_peered":"2024-01-15T18:10:28.853347+0000","last_unstale":"2024-01-15T18:10:28.859604+0000","last_undegraded":"2024-01-15T18:10:28.859604+0000","last_fullsized":"2024-01-15T18:10:28.859604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:19:32.207939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.855546+0000","last_change":"2024-01-15T18:10:28.855546+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.855546+0000","last_undegraded":"2024-01-15T18:10:28.855546+0000","last_fullsized":"2024-01-15T18:10:28.855546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.863889+0000","last_change":"2024-01-15T18:10:28.861226+0000","last_active":"2024-01-15T18:10:28.863889+0000","last_peered":"2024-01-15T18:10:28.863889+0000","last_clean":"2024-01-15T18:10:28.863889+0000","last_became_active":"2024-01-15T18:10:28.860947+0000","last_became_peered":"2024-01-15T18:10:28.860947+0000","last_unstale":"2024-01-15T18:10:28.863889+0000","last_undegraded":"2024-01-15T18:10:28.863889+0000","last_fullsized":"2024-01-15T18:10:28.863889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:07:04.692565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841823+0000","last_change":"2024-01-15T18:10:28.841823+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841823+0000","last_undegraded":"2024-01-15T18:10:28.841823+0000","last_fullsized":"2024-01-15T18:10:28.841823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.854829+0000","last_change":"2024-01-15T18:10:28.853003+0000","last_active":"2024-01-15T18:10:28.854829+0000","last_peered":"2024-01-15T18:10:28.854829+0000","last_clean":"2024-01-15T18:10:28.854829+0000","last_became_active":"2024-01-15T18:10:28.852522+0000","last_became_peered":"2024-01-15T18:10:28.852522+0000","last_unstale":"2024-01-15T18:10:28.854829+0000","last_undegraded":"2024-01-15T18:10:28.854829+0000","last_fullsized":"2024-01-15T18:10:28.854829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:19.865670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839292+0000","last_change":"2024-01-15T18:10:28.839292+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839292+0000","last_undegraded":"2024-01-15T18:10:28.839292+0000","last_fullsized":"2024-01-15T18:10:28.839292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865251+0000","last_change":"2024-01-15T18:10:28.861120+0000","last_active":"2024-01-15T18:10:28.865251+0000","last_peered":"2024-01-15T18:10:28.865251+0000","last_clean":"2024-01-15T18:10:28.865251+0000","last_became_active":"2024-01-15T18:10:28.860131+0000","last_became_peered":"2024-01-15T18:10:28.860131+0000","last_unstale":"2024-01-15T18:10:28.865251+0000","last_undegraded":"2024-01-15T18:10:28.865251+0000","last_fullsized":"2024-01-15T18:10:28.865251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:31.398737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886778+0000","last_change":"2024-01-15T18:10:29.883153+0000","last_active":"2024-01-15T18:10:29.886778+0000","last_peered":"2024-01-15T18:10:29.886778+0000","last_clean":"2024-01-15T18:10:29.886778+0000","last_became_active":"2024-01-15T18:10:29.882185+0000","last_became_peered":"2024-01-15T18:10:29.882185+0000","last_unstale":"2024-01-15T18:10:29.886778+0000","last_undegraded":"2024-01-15T18:10:29.886778+0000","last_fullsized":"2024-01-15T18:10:29.886778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865274+0000","last_change":"2024-01-15T18:10:28.863474+0000","last_active":"2024-01-15T18:10:28.865274+0000","last_peered":"2024-01-15T18:10:28.865274+0000","last_clean":"2024-01-15T18:10:28.865274+0000","last_became_active":"2024-01-15T18:10:28.863019+0000","last_became_peered":"2024-01-15T18:10:28.863019+0000","last_unstale":"2024-01-15T18:10:28.865274+0000","last_undegraded":"2024-01-15T18:10:28.865274+0000","last_fullsized":"2024-01-15T18:10:28.865274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:14.613744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.853917+0000","last_change":"2024-01-15T18:10:28.853917+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.853917+0000","last_undegraded":"2024-01-15T18:10:28.853917+0000","last_fullsized":"2024-01-15T18:10:28.853917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865274+0000","last_change":"2024-01-15T18:10:28.863368+0000","last_active":"2024-01-15T18:10:28.865274+0000","last_peered":"2024-01-15T18:10:28.865274+0000","last_clean":"2024-01-15T18:10:28.865274+0000","last_became_active":"2024-01-15T18:10:28.861032+0000","last_became_peered":"2024-01-15T18:10:28.861032+0000","last_unstale":"2024-01-15T18:10:28.865274+0000","last_undegraded":"2024-01-15T18:10:28.865274+0000","last_fullsized":"2024-01-15T18:10:28.865274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:54:11.743800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837201+0000","last_change":"2024-01-15T18:10:28.837201+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837201+0000","last_undegraded":"2024-01-15T18:10:28.837201+0000","last_fullsized":"2024-01-15T18:10:28.837201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.870806+0000","last_change":"2024-01-15T18:10:28.859062+0000","last_active":"2024-01-15T18:10:29.870806+0000","last_peered":"2024-01-15T18:10:29.870806+0000","last_clean":"2024-01-15T18:10:29.870806+0000","last_became_active":"2024-01-15T18:10:28.858021+0000","last_became_peered":"2024-01-15T18:10:28.858021+0000","last_unstale":"2024-01-15T18:10:29.870806+0000","last_undegraded":"2024-01-15T18:10:29.870806+0000","last_fullsized":"2024-01-15T18:10:29.870806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":227,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847139+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:10:28.847139+0000","last_peered":"2024-01-15T18:10:28.847139+0000","last_clean":"2024-01-15T18:10:28.847139+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:10:28.847139+0000","last_undegraded":"2024-01-15T18:10:28.847139+0000","last_fullsized":"2024-01-15T18:10:28.847139+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77156,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294684,"statfs":{"total":96636764160,"available":96557756416,"internally_reserved":0,"allocated":1658880,"data_stored":1049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.3799999999999999},"min":{"1min":0.81899999999999995,"5min":0.626,"15min":0.53600000000000003},"max":{"1min":1.677,"5min":4.0019999999999998,"15min":14.114000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.7030000000000001,"15min":1.5680000000000001},"min":{"1min":0.91200000000000003,"5min":0.69999999999999996,"15min":0.51700000000000002},"max":{"1min":1.6970000000000001,"5min":27.32,"15min":27.32},"last":1.6859999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.8300000000000001,"15min":1.4670000000000001},"min":{"1min":0.97299999999999998,"5min":0.53300000000000003,"15min":0.40300000000000002},"max":{"1min":1.617,"5min":29.167999999999999,"15min":29.167999999999999},"last":1.103},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.835,"5min":4.2699999999999996,"15min":12.565},"last":1.04}]},{"osd":2,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3360000000000001,"15min":1.464},"min":{"1min":0.61399999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.55,"5min":4.1920000000000002,"15min":13.801},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.7,"15min":1.542},"min":{"1min":0.91500000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.788,"5min":27.358000000000001,"15min":27.358000000000001},"last":1.1970000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.853,"15min":1.609},"min":{"1min":1.097,"5min":0.61299999999999999,"15min":0.47099999999999997},"max":{"1min":1.619,"5min":29.204999999999998,"15min":29.204999999999998},"last":1.377},{"interface":"front","average":{"1min":1.425,"5min":1.506,"15min":1.5600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":1.8300000000000001,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.516}]},{"osd":4,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.7509999999999999,"15min":1.621},"min":{"1min":0.91200000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.724,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.569},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.887,"15min":1.653},"min":{"1min":1.151,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":1.782,"5min":29.25,"15min":29.25},"last":1.2689999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.78,"15min":1.679},"min":{"1min":0.91500000000000004,"5min":0.83199999999999996,"15min":0.39000000000000001},"max":{"1min":1.944,"5min":11.834,"15min":12.518000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.9159999999999999,"15min":1.722},"min":{"1min":1.03,"5min":0.60499999999999998,"15min":0.41099999999999998},"max":{"1min":1.883,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.421}]},{"osd":6,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0579999999999998,"15min":2.367},"min":{"1min":1.0169999999999999,"5min":0.86499999999999999,"15min":0.45600000000000002},"max":{"1min":2.0390000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.772},{"interface":"front","average":{"1min":1.591,"5min":2.101,"15min":1.8080000000000001},"min":{"1min":1.1899999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":1.923,"5min":29.106999999999999,"15min":29.106999999999999},"last":1.4710000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58592,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313248,"statfs":{"total":96636764160,"available":96576765952,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.276,"15min":1.1950000000000001},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":5.0330000000000004,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.405,"5min":1.3959999999999999,"15min":1.339},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.1659999999999999,"5min":9.548,"15min":21.565000000000001},"last":1.2110000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4319999999999999,"15min":1.3440000000000001},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"max":{"1min":5.1150000000000002,"5min":8.1189999999999998,"15min":8.1189999999999998},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.462,"15min":1.4370000000000001},"min":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.73,"5min":9.6389999999999993,"15min":13.984999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3819999999999999,"15min":1.4630000000000001},"min":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":2.919,"5min":7.0030000000000001,"15min":34.670000000000002},"last":1.151},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.411},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.1609999999999996,"5min":6.585,"15min":6.585},"last":1.0529999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6479999999999999,"15min":1.5529999999999999},"min":{"1min":1.0069999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.5680000000000001,"5min":16.855,"15min":16.855},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6180000000000001,"15min":1.679},"min":{"1min":0.96599999999999997,"5min":0.628,"15min":0.48499999999999999},"max":{"1min":2.9630000000000001,"5min":8.0139999999999993,"15min":34.725000000000001},"last":1.4950000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.512,"15min":1.4390000000000001},"min":{"1min":0.95399999999999996,"5min":0.56299999999999994,"15min":0.38400000000000001},"max":{"1min":5.2030000000000003,"5min":6.5220000000000002,"15min":8.5489999999999995},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.784,"5min":1.6379999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.63600000000000001,"15min":0.501},"max":{"1min":7.3230000000000004,"5min":10.118,"15min":10.118},"last":1.359}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.0720000000000001,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":2.8420000000000001,"5min":7.9500000000000002,"15min":34.798000000000002},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.7410000000000001,"15min":1.609},"min":{"1min":1.228,"5min":0.66000000000000003,"15min":0.47899999999999998},"max":{"1min":8.0129999999999999,"5min":10.077,"15min":10.077},"last":1.2689999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":1.748,"15min":1.669},"min":{"1min":1.228,"5min":0.78900000000000003,"15min":0.64000000000000001},"max":{"1min":5.5060000000000002,"5min":10.302,"15min":10.302},"last":1.544},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.7629999999999999,"15min":1.8500000000000001},"min":{"1min":1.2110000000000001,"5min":0.84599999999999997,"15min":0.52600000000000002},"max":{"1min":2.782,"5min":8.0519999999999996,"15min":34.762999999999998},"last":1.401}]}]},{"osd":5,"up_from":14,"seq":60129542305,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54432,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317408,"statfs":{"total":96636764160,"available":96581025792,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.5780000000000001,"15min":1.3660000000000001},"min":{"1min":0.71499999999999997,"5min":0.67900000000000005,"15min":0.52700000000000002},"max":{"1min":1.5980000000000001,"5min":26.077999999999999,"15min":26.077999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.23,"5min":1.431,"15min":1.4470000000000001},"min":{"1min":0.76400000000000001,"5min":0.73999999999999999,"15min":0.52400000000000002},"max":{"1min":1.7270000000000001,"5min":7.7220000000000004,"15min":14.711},"last":1.5669999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.653,"15min":1.387},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6679999999999999,"5min":26,"15min":26},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.409,"15min":1.3400000000000001},"min":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":1.6919999999999999,"5min":7.5309999999999997,"15min":12.619},"last":1.5269999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.8340000000000001,"15min":1.5409999999999999},"min":{"1min":0.874,"5min":0.73299999999999998,"15min":0.55700000000000005},"max":{"1min":4.8719999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.605,"5min":1.825,"15min":1.6020000000000001},"min":{"1min":0.90200000000000002,"5min":0.63900000000000001,"15min":0.59099999999999997},"max":{"1min":4.968,"5min":17.126999999999999,"15min":17.126999999999999},"last":1.3959999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4670000000000001,"15min":1.448},"min":{"1min":0.85999999999999999,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":1.722,"5min":7.6150000000000002,"15min":12.722},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.8740000000000001,"15min":1.637},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.63400000000000001},"max":{"1min":1.776,"5min":26.134,"15min":26.134},"last":1.167}]},{"osd":4,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.754,"5min":0.68999999999999995,"15min":0.55900000000000005},"max":{"1min":1.72,"5min":7.8730000000000002,"15min":25.776},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.9470000000000001,"15min":1.734},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.47799999999999998},"max":{"1min":1.8799999999999999,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.6259999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.466,"15min":1.5489999999999999},"min":{"1min":0.84099999999999997,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":1.9390000000000001,"5min":7.9240000000000004,"15min":12.769},"last":1.337},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.798,"15min":1.8029999999999999},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.435},"max":{"1min":1.865,"5min":26.186,"15min":36.087000000000003},"last":1.2869999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.9319999999999999,"15min":1.8959999999999999},"min":{"1min":1.0680000000000001,"5min":0.79100000000000004,"15min":0.68000000000000005},"max":{"1min":1.98,"5min":9.6859999999999999,"15min":13.706},"last":1.679},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.784},"min":{"1min":0.80000000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9159999999999999,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.6040000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58588,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313252,"statfs":{"total":96636764160,"available":96576770048,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1870000000000001,"15min":1.3580000000000001},"min":{"1min":0.55700000000000005,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.9419999999999999,"5min":4.8170000000000002,"15min":21.643000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.3049999999999999,"15min":1.2949999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.861,"5min":8.9109999999999996,"15min":11.196},"last":1.238}]},{"osd":1,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5960000000000001,"15min":1.4059999999999999},"min":{"1min":0.59299999999999997,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":1.587,"5min":28.663,"15min":28.663},"last":1.452},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.53,"15min":1.4970000000000001},"min":{"1min":0.72599999999999998,"5min":0.46200000000000002,"15min":0.35399999999999998},"max":{"1min":1.837,"5min":13.614000000000001,"15min":21.440000000000001},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.6220000000000001,"15min":1.446},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.8049999999999999,"5min":16.023,"15min":16.023},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.446,"15min":1.5109999999999999},"min":{"1min":0.94299999999999995,"5min":0.52500000000000002,"15min":0.38100000000000001},"max":{"1min":1.7010000000000001,"5min":7.9169999999999998,"15min":22.350999999999999},"last":1.2010000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.375,"15min":1.512},"min":{"1min":0.874,"5min":0.56699999999999995,"15min":0.48799999999999999},"max":{"1min":1.5329999999999999,"5min":5.0860000000000003,"15min":15.093},"last":1.355},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.6919999999999999,"15min":1.651},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.6100000000000001,"5min":18.027000000000001,"15min":19.486000000000001},"last":1.256}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3859999999999999,"15min":1.766},"min":{"1min":0.67800000000000005,"5min":0.41899999999999998,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":5.29,"15min":54.938000000000002},"last":1.387},{"interface":"front","average":{"1min":1.294,"5min":1.5620000000000001,"15min":1.5309999999999999},"min":{"1min":0.63,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":1.9610000000000001,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.1559999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.6890000000000001,"15min":1.615},"min":{"1min":0.90800000000000003,"5min":0.495,"15min":0.38700000000000001},"max":{"1min":1.9279999999999999,"5min":16.103999999999999,"15min":17.733000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.7050000000000001,"15min":1.853},"min":{"1min":0.94699999999999995,"5min":0.499,"15min":0.499},"max":{"1min":2.0430000000000001,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.0600000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.98,"15min":1.825},"min":{"1min":0.79100000000000004,"5min":0.46999999999999997,"15min":0.45100000000000001},"max":{"1min":2.0600000000000001,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.3},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8939999999999999,"15min":1.847},"min":{"1min":0.65200000000000002,"5min":0.62,"15min":0.51000000000000001},"max":{"1min":2.0310000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.552}]}]},{"osd":3,"up_from":14,"seq":60129542305,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68252,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303588,"statfs":{"total":96636764160,"available":96566874112,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.371,"15min":1.0429999999999999},"min":{"1min":0.372,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.2050000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":1.399},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.151,"15min":1.028},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.665,"5min":5.0190000000000001,"15min":5.0529999999999999},"last":0.84899999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1060000000000001,"15min":1.0920000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.34399999999999997},"max":{"1min":1.365,"5min":5.226,"15min":8.1359999999999992},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.179},"min":{"1min":0.56899999999999995,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.8140000000000001,"5min":2.1339999999999999,"15min":7.2169999999999996},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.4510000000000001,"15min":1.256},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.859,"5min":14.025,"15min":14.025},"last":1.127},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3460000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.57399999999999995,"15min":0.46000000000000002},"max":{"1min":1.9710000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.242}]},{"osd":4,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.569,"15min":1.423},"min":{"1min":0.94099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2930000000000001,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.7430000000000001,"15min":1.5369999999999999},"min":{"1min":1.1100000000000001,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.9119999999999999,"5min":15.769,"15min":15.769},"last":1.302}]},{"osd":5,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.659,"15min":1.53},"min":{"1min":1.079,"5min":0.69299999999999995,"15min":0.65000000000000002},"max":{"1min":1.984,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.556},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.2490000000000001,"15min":1.8919999999999999},"min":{"1min":0.97599999999999998,"5min":0.67900000000000005,"15min":0.58299999999999996},"max":{"1min":2.0800000000000001,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.7270000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.86899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.6339999999999999,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.514},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.8180000000000001,"15min":1.708},"min":{"1min":1.246,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.048,"5min":6.8639999999999999,"15min":16.195},"last":1.381}]},{"osd":7,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.2949999999999999,"15min":1.859},"min":{"1min":1.034,"5min":0.76000000000000001,"15min":0.63900000000000001},"max":{"1min":2.2629999999999999,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1030000000000002,"15min":1.722},"min":{"1min":1.0049999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3679999999999999,"5min":17.622,"15min":17.622},"last":1.788}]}]},{"osd":2,"up_from":14,"seq":60129542304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68256,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303584,"statfs":{"total":96636764160,"available":96566870016,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98899999999999999,"15min":1.032},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.30099999999999999},"max":{"1min":1.3029999999999999,"5min":3.9199999999999999,"15min":6.101},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.087,"5min":1.1919999999999999,"15min":1.1419999999999999},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28299999999999997},"max":{"1min":1.6839999999999999,"5min":10.319000000000001,"15min":10.319000000000001},"last":1.6040000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0509999999999999,"15min":1.105},"min":{"1min":0.79600000000000004,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":1.5229999999999999,"5min":4.4939999999999998,"15min":10.807},"last":1.246},{"interface":"front","average":{"1min":1.131,"5min":1.2889999999999999,"15min":1.228},"min":{"1min":0.61599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5960000000000001,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":0.94899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.6180000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":2.8010000000000002,"5min":25.891999999999999,"15min":25.891999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.099,"5min":1.373,"15min":1.256},"min":{"1min":0.78700000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.7270000000000001,"5min":14.946,"15min":14.946},"last":1.073}]},{"osd":4,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.546,"15min":1.552},"min":{"1min":0.874,"5min":0.66100000000000003,"15min":0.64900000000000002},"max":{"1min":1.9339999999999999,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.6870000000000001,"15min":1.5509999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":2.4249999999999998,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.742,"15min":1.669},"min":{"1min":1.02,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8519999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.6359999999999999,"15min":1.722},"min":{"1min":1,"5min":0.71699999999999997,"15min":0.55200000000000005},"max":{"1min":2.02,"5min":8.5250000000000004,"15min":37.020000000000003},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.645,"15min":1.7150000000000001},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.889,"5min":10.185,"15min":23.762},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.9139999999999999,"15min":1.867},"min":{"1min":1.052,"5min":0.85899999999999999,"15min":0.73499999999999999},"max":{"1min":2.8860000000000001,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.5680000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":2.1349999999999998,"15min":1.931},"min":{"1min":1.0169999999999999,"5min":0.85999999999999999,"15min":0.68999999999999995},"max":{"1min":1.9470000000000001,"5min":19.986000000000001,"15min":52.767000000000003},"last":1.786},{"interface":"front","average":{"1min":1.605,"5min":1.988,"15min":1.804},"min":{"1min":0.97899999999999998,"5min":0.88100000000000001,"15min":0.83499999999999996},"max":{"1min":2.9169999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":1.7370000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542304,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68256,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303584,"statfs":{"total":96636764160,"available":96566870016,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.167,"15min":1.0189999999999999},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3160000000000001,"5min":19.218,"15min":19.218},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.288,"15min":1.1879999999999999},"min":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.573,"5min":19.823,"15min":19.823},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0129999999999999,"15min":1.0940000000000001},"min":{"1min":0.47599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6759999999999999,"5min":2.8980000000000001,"15min":7.6680000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.407,"15min":1.2470000000000001},"min":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.32500000000000001},"max":{"1min":1.52,"5min":19.318000000000001,"15min":19.318000000000001},"last":0.82299999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5109999999999999,"15min":1.365},"min":{"1min":0.56699999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6100000000000001,"5min":19.387,"15min":19.387},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.4379999999999999,"15min":1.351},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.8919999999999999,"5min":19.733000000000001,"15min":19.733000000000001},"last":1.28}]},{"osd":4,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.7010000000000001,"15min":1.542},"min":{"1min":0.79600000000000004,"5min":0.70599999999999996,"15min":0.497},"max":{"1min":1.7549999999999999,"5min":19.456,"15min":19.456},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.619,"15min":1.665},"min":{"1min":0.76600000000000001,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.859,"5min":19.925999999999998,"15min":22.204000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6279999999999999,"15min":1.6830000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":1.8169999999999999,"5min":20.012,"15min":20.012},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.464,"15min":1.6240000000000001},"min":{"1min":0.71599999999999997,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":1.774,"5min":2.6190000000000002,"15min":19.998999999999999},"last":1.714}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.7669999999999999,"15min":1.6910000000000001},"min":{"1min":0.95799999999999996,"5min":0.70799999999999996,"15min":0.54100000000000004},"max":{"1min":1.8120000000000001,"5min":19.526,"15min":19.526},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.8520000000000001,"15min":1.8200000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":1.9550000000000001,"5min":20.064,"15min":27.579999999999998},"last":1.395}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.8460000000000001,"15min":1.8120000000000001},"min":{"1min":0.81699999999999995,"5min":0.63,"15min":0.505},"max":{"1min":1.6930000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.486},{"interface":"front","average":{"1min":1.4730000000000001,"5min":2.0139999999999998,"15min":1.8240000000000001},"min":{"1min":0.94099999999999995,"5min":0.76100000000000001,"15min":0.54500000000000004},"max":{"1min":1.8009999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.6160000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68836,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303004,"statfs":{"total":96636764160,"available":96566276096,"internally_reserved":0,"allocated":1658880,"data_stored":1049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.1020000000000001,"15min":1.1180000000000001},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5620000000000001,"5min":20.437999999999999,"15min":20.437999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1020000000000001,"15min":1.2030000000000001},"min":{"1min":0.61899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4370000000000001,"5min":3.492,"15min":35.137999999999998},"last":0.72699999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.375,"15min":1.278},"min":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.101,"5min":20.556000000000001,"15min":20.556000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.256,"15min":1.3640000000000001},"min":{"1min":0.44400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7350000000000001,"5min":4.7430000000000003,"15min":35.183},"last":0.96099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.143,"15min":1.1379999999999999},"min":{"1min":0.52800000000000002,"5min":0.46000000000000002,"15min":0.41699999999999998},"max":{"1min":1.3149999999999999,"5min":5.8499999999999996,"15min":8.9949999999999992},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5209999999999999,"15min":1.569},"min":{"1min":0.66700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.4119999999999999,"5min":20.670000000000002,"15min":25.372},"last":0.85099999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.514,"15min":1.6759999999999999},"min":{"1min":0.98099999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7310000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.282},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.383,"15min":1.452},"min":{"1min":0.81200000000000006,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6739999999999999,"5min":2.5049999999999999,"15min":15.968999999999999},"last":1.0449999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.7050000000000001,"15min":1.6080000000000001},"min":{"1min":0.78600000000000003,"5min":0.69799999999999995,"15min":0.63800000000000001},"max":{"1min":1.8,"5min":12.173999999999999,"15min":12.173999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.964,"15min":1.786},"min":{"1min":1.121,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":20.789999999999999,"15min":20.789999999999999},"last":1.1619999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.109,"15min":1.972},"min":{"1min":1.0109999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.74,"5min":20.893000000000001,"15min":36.390999999999998},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.726,"15min":1.9099999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.8919999999999999,"5min":8.3829999999999991,"15min":36.207000000000001},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.986,"15min":1.9339999999999999},"min":{"1min":0.94899999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.0049999999999999,"5min":24.556999999999999,"15min":35.636000000000003},"last":1.206},{"interface":"front","average":{"1min":1.429,"5min":1.768,"15min":1.649},"min":{"1min":0.91900000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.8300000000000001,"5min":11.470000000000001,"15min":11.470000000000001},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:10:34.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:10:34.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.568+0000 7f4309400640 1 Processor -- start 2024-01-15T18:10:34.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.568+0000 7f4309400640 1 -- start start 2024-01-15T18:10:34.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.569+0000 7f4309400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43040d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.569+0000 7f4309400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43040cc3f0 0x7f43040b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.569+0000 7f4309400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f430415b7b0 con 0x7f430415b3d0 2024-01-15T18:10:34.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.569+0000 7f4309400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43040cef10 con 0x7f43040cc3f0 2024-01-15T18:10:34.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.569+0000 7f4309400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43040cc9b0 con 0x7f43040cfb10 2024-01-15T18:10:34.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.570+0000 7f4308a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43040cc3f0 0x7f43040b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.570+0000 7f4303400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43040d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.570+0000 7f4308a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f43040cc3f0 0x7f43040b0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:47994/0 (socket says 172.21.15.2:47994) 2024-01-15T18:10:34.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.570+0000 7f4308a00640 1 -- 172.21.15.2:0/4029634530 learned_addr learned my addr 172.21.15.2:0/4029634530 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:34.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.571+0000 7f4308a00640 1 -- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 msgr2=0x7f43040d1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:34.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.571+0000 7f4308a00640 1 --2- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43040d1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.571+0000 7f4308a00640 1 -- 172.21.15.2:0/4029634530 >> v1:172.21.15.137:6789/0 conn(0x7f430415b3d0 legacy=0x7f43040cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.571+0000 7f4308a00640 1 -- 172.21.15.2:0/4029634530 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43040ccc90 con 0x7f43040cc3f0 2024-01-15T18:10:34.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.571+0000 7f4303400640 1 --2- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43040d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:10:34.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.572+0000 7f4308a00640 1 --2- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43040cc3f0 0x7f43040b0f10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009e50 tx=0x7f42fc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ece8836a7679416e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:34.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.573+0000 7f4302a00640 1 -- 172.21.15.2:0/4029634530 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc02cb00 con 0x7f43040cc3f0 2024-01-15T18:10:34.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.573+0000 7f4302a00640 1 -- 172.21.15.2:0/4029634530 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42fc02cc60 con 0x7f43040cc3f0 2024-01-15T18:10:34.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.573+0000 7f4302a00640 1 -- 172.21.15.2:0/4029634530 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42fc0343f0 con 0x7f43040cc3f0 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 -- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43040cc3f0 msgr2=0x7f43040b0f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 --2- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43040cc3f0 0x7f43040b0f10 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f42fc009e50 tx=0x7f42fc02bf60 comp rx=0 tx=0).stop 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 -- 172.21.15.2:0/4029634530 shutdown_connections 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 --2- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f43040cc3f0 0x7f43040b0f10 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 --2- 172.21.15.2:0/4029634530 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43040d1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 -- 172.21.15.2:0/4029634530 >> 172.21.15.2:0/4029634530 conn(0x7f43040d0e20 msgr2=0x7f43040d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:34.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 -- 172.21.15.2:0/4029634530 shutdown_connections 2024-01-15T18:10:34.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.574+0000 7f4309400640 1 -- 172.21.15.2:0/4029634530 wait complete. 2024-01-15T18:10:34.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.576+0000 7f4309400640 1 Processor -- start 2024-01-15T18:10:34.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.576+0000 7f4309400640 1 -- start start 2024-01-15T18:10:34.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4309400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 0x7f43040c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4309400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43041e6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4309400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f430415b3d0 0x7f43041e6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4309400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43041e4ae0 con 0x7f43040cc3f0 2024-01-15T18:10:34.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4309400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43040c49f0 con 0x7f430415b3d0 2024-01-15T18:10:34.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4309400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43040c4cd0 con 0x7f43040cfb10 2024-01-15T18:10:34.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4303e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 0x7f43040c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4303e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 0x7f43040c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55706/0 (socket says 172.21.15.2:55706) 2024-01-15T18:10:34.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4303e00640 1 -- 172.21.15.2:0/599569128 learned_addr learned my addr 172.21.15.2:0/599569128 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4308a00640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f430415b3d0 0x7f43041e6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4303400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43041e6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4303e00640 1 -- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 msgr2=0x7f43041e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.577+0000 7f4303e00640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43041e6640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f4303e00640 1 -- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f430415b3d0 msgr2=0x7f43041e6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f4303e00640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f430415b3d0 0x7f43041e6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f4303e00640 1 -- 172.21.15.2:0/599569128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec009400 con 0x7f43040cc3f0 2024-01-15T18:10:34.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f4303e00640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 0x7f43040c70d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c400 tx=0x7f42f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:34.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f40052d0 con 0x7f43040cc3f0 2024-01-15T18:10:34.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc009b30 con 0x7f43040cc3f0 2024-01-15T18:10:34.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42f4001140 con 0x7f43040cc3f0 2024-01-15T18:10:34.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4304159940 con 0x7f43040cc3f0 2024-01-15T18:10:34.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.578+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42f40035f0 con 0x7f43040cc3f0 2024-01-15T18:10:34.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.579+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43040ce720 con 0x7f43040cc3f0 2024-01-15T18:10:34.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.581+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f42f4003750 con 0x7f43040cc3f0 2024-01-15T18:10:34.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.582+0000 7f42fbe00640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42dc06bfb0 0x7f42dc06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:34.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.582+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f42f40878b0 con 0x7f43040cc3f0 2024-01-15T18:10:34.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.583+0000 7f4303400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42dc06bfb0 0x7f42dc06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:34.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.584+0000 7f4303400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42dc06bfb0 0x7f42dc06e470 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f43040b6460 tx=0x7f42ec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:34.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.586+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42f4057d10 con 0x7f43040cc3f0 2024-01-15T18:10:34.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.780+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f430406f9f0 con 0x7f42dc06bfb0 2024-01-15T18:10:34.784 INFO:tasks.ceph.mds.c.smithi002.stderr:2024-01-15T18:10:34.781+0000 7ff72c200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:10:34.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.802+0000 7f42fbe00640 1 -- 172.21.15.2:0/599569128 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282200 (secure 0 0 0) 0x7f430406f9f0 con 0x7f42dc06bfb0 2024-01-15T18:10:34.809 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:34.810 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:10:34.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.807+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42dc06bfb0 msgr2=0x7f42dc06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.807+0000 7f4309400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42dc06bfb0 0x7f42dc06e470 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f43040b6460 tx=0x7f42ec008040 comp rx=0 tx=0).stop 2024-01-15T18:10:34.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 msgr2=0x7f43040c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:34.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 0x7f43040c70d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f42f400c400 tx=0x7f42f400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 shutdown_connections 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f42dc06bfb0 0x7f42dc06e470 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f430415b3d0 0x7f43041e6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f43040cfb10 0x7f43041e6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 --2- 172.21.15.2:0/599569128 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f43040cc3f0 0x7f43040c70d0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 >> 172.21.15.2:0/599569128 conn(0x7f43040d0e20 msgr2=0x7f43040b17c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 shutdown_connections 2024-01-15T18:10:34.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:34.808+0000 7f4309400640 1 -- 172.21.15.2:0/599569128 wait complete. 2024-01-15T18:10:34.831 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-01-15T18:10:33.713319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":522368,"kb_used_data":9480,"kb_used_omap":12,"kb_used_meta":512755,"kb_avail":754452352,"statfs":{"total":773094113280,"available":772559208448,"internally_reserved":0,"allocated":9707520,"data_stored":4849976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":525061681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.884985"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.853366+0000","last_change":"2024-01-15T18:10:28.853366+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.853366+0000","last_undegraded":"2024-01-15T18:10:28.853366+0000","last_fullsized":"2024-01-15T18:10:28.853366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845232+0000","last_change":"2024-01-15T18:10:28.845232+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845232+0000","last_undegraded":"2024-01-15T18:10:28.845232+0000","last_fullsized":"2024-01-15T18:10:28.845232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.889646+0000","last_change":"2024-01-15T18:10:29.888180+0000","last_active":"2024-01-15T18:10:29.889646+0000","last_peered":"2024-01-15T18:10:29.889646+0000","last_clean":"2024-01-15T18:10:29.889646+0000","last_became_active":"2024-01-15T18:10:29.886925+0000","last_became_peered":"2024-01-15T18:10:29.886925+0000","last_unstale":"2024-01-15T18:10:29.889646+0000","last_undegraded":"2024-01-15T18:10:29.889646+0000","last_fullsized":"2024-01-15T18:10:29.889646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:18.035419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841871+0000","last_change":"2024-01-15T18:10:28.841871+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841871+0000","last_undegraded":"2024-01-15T18:10:28.841871+0000","last_fullsized":"2024-01-15T18:10:28.841871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854388+0000","last_change":"2024-01-15T18:10:28.854388+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854388+0000","last_undegraded":"2024-01-15T18:10:28.854388+0000","last_fullsized":"2024-01-15T18:10:28.854388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843476+0000","last_change":"2024-01-15T18:10:28.843476+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843476+0000","last_undegraded":"2024-01-15T18:10:28.843476+0000","last_fullsized":"2024-01-15T18:10:28.843476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.844115+0000","last_change":"2024-01-15T18:10:28.844115+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.844115+0000","last_undegraded":"2024-01-15T18:10:28.844115+0000","last_fullsized":"2024-01-15T18:10:28.844115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838777+0000","last_change":"2024-01-15T18:10:28.838777+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838777+0000","last_undegraded":"2024-01-15T18:10:28.838777+0000","last_fullsized":"2024-01-15T18:10:28.838777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841145+0000","last_change":"2024-01-15T18:10:28.841145+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841145+0000","last_undegraded":"2024-01-15T18:10:28.841145+0000","last_fullsized":"2024-01-15T18:10:28.841145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846140+0000","last_change":"2024-01-15T18:10:28.846140+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846140+0000","last_undegraded":"2024-01-15T18:10:28.846140+0000","last_fullsized":"2024-01-15T18:10:28.846140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840656+0000","last_change":"2024-01-15T18:10:28.840656+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840656+0000","last_undegraded":"2024-01-15T18:10:28.840656+0000","last_fullsized":"2024-01-15T18:10:28.840656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.848480+0000","last_change":"2024-01-15T18:10:28.848480+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.848480+0000","last_undegraded":"2024-01-15T18:10:28.848480+0000","last_fullsized":"2024-01-15T18:10:28.848480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.888056+0000","last_change":"2024-01-15T18:10:29.884896+0000","last_active":"2024-01-15T18:10:29.888056+0000","last_peered":"2024-01-15T18:10:29.888056+0000","last_clean":"2024-01-15T18:10:29.888056+0000","last_became_active":"2024-01-15T18:10:29.883901+0000","last_became_peered":"2024-01-15T18:10:29.883901+0000","last_unstale":"2024-01-15T18:10:29.888056+0000","last_undegraded":"2024-01-15T18:10:29.888056+0000","last_fullsized":"2024-01-15T18:10:29.888056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:57.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840579+0000","last_change":"2024-01-15T18:10:28.840579+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840579+0000","last_undegraded":"2024-01-15T18:10:28.840579+0000","last_fullsized":"2024-01-15T18:10:28.840579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.842561+0000","last_change":"2024-01-15T18:10:28.842561+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.842561+0000","last_undegraded":"2024-01-15T18:10:28.842561+0000","last_fullsized":"2024-01-15T18:10:28.842561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.847589+0000","last_change":"2024-01-15T18:10:28.847589+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.847589+0000","last_undegraded":"2024-01-15T18:10:28.847589+0000","last_fullsized":"2024-01-15T18:10:28.847589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.852113+0000","last_change":"2024-01-15T18:10:28.852113+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.852113+0000","last_undegraded":"2024-01-15T18:10:28.852113+0000","last_fullsized":"2024-01-15T18:10:28.852113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843978+0000","last_change":"2024-01-15T18:10:28.843978+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843978+0000","last_undegraded":"2024-01-15T18:10:28.843978+0000","last_fullsized":"2024-01-15T18:10:28.843978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840150+0000","last_change":"2024-01-15T18:10:28.840150+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840150+0000","last_undegraded":"2024-01-15T18:10:28.840150+0000","last_fullsized":"2024-01-15T18:10:28.840150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840702+0000","last_change":"2024-01-15T18:10:28.840702+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840702+0000","last_undegraded":"2024-01-15T18:10:28.840702+0000","last_fullsized":"2024-01-15T18:10:28.840702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843143+0000","last_change":"2024-01-15T18:10:28.843143+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843143+0000","last_undegraded":"2024-01-15T18:10:28.843143+0000","last_fullsized":"2024-01-15T18:10:28.843143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845765+0000","last_change":"2024-01-15T18:10:28.845765+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845765+0000","last_undegraded":"2024-01-15T18:10:28.845765+0000","last_fullsized":"2024-01-15T18:10:28.845765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854373+0000","last_change":"2024-01-15T18:10:28.854373+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854373+0000","last_undegraded":"2024-01-15T18:10:28.854373+0000","last_fullsized":"2024-01-15T18:10:28.854373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838275+0000","last_change":"2024-01-15T18:10:28.838275+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838275+0000","last_undegraded":"2024-01-15T18:10:28.838275+0000","last_fullsized":"2024-01-15T18:10:28.838275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838843+0000","last_change":"2024-01-15T18:10:28.838843+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838843+0000","last_undegraded":"2024-01-15T18:10:28.838843+0000","last_fullsized":"2024-01-15T18:10:28.838843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886570+0000","last_change":"2024-01-15T18:10:29.879335+0000","last_active":"2024-01-15T18:10:29.886570+0000","last_peered":"2024-01-15T18:10:29.886570+0000","last_clean":"2024-01-15T18:10:29.886570+0000","last_became_active":"2024-01-15T18:10:29.878999+0000","last_became_peered":"2024-01-15T18:10:29.878999+0000","last_unstale":"2024-01-15T18:10:29.886570+0000","last_undegraded":"2024-01-15T18:10:29.886570+0000","last_fullsized":"2024-01-15T18:10:29.886570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.857241+0000","last_change":"2024-01-15T18:10:28.857241+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.857241+0000","last_undegraded":"2024-01-15T18:10:28.857241+0000","last_fullsized":"2024-01-15T18:10:28.857241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.847343+0000","last_change":"2024-01-15T18:10:28.847343+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.847343+0000","last_undegraded":"2024-01-15T18:10:28.847343+0000","last_fullsized":"2024-01-15T18:10:28.847343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840119+0000","last_change":"2024-01-15T18:10:28.840119+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840119+0000","last_undegraded":"2024-01-15T18:10:28.840119+0000","last_fullsized":"2024-01-15T18:10:28.840119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845208+0000","last_change":"2024-01-15T18:10:28.845208+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845208+0000","last_undegraded":"2024-01-15T18:10:28.845208+0000","last_fullsized":"2024-01-15T18:10:28.845208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846185+0000","last_change":"2024-01-15T18:10:28.846185+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846185+0000","last_undegraded":"2024-01-15T18:10:28.846185+0000","last_fullsized":"2024-01-15T18:10:28.846185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846205+0000","last_change":"2024-01-15T18:10:28.846205+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846205+0000","last_undegraded":"2024-01-15T18:10:28.846205+0000","last_fullsized":"2024-01-15T18:10:28.846205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856360+0000","last_change":"2024-01-15T18:10:28.856360+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856360+0000","last_undegraded":"2024-01-15T18:10:28.856360+0000","last_fullsized":"2024-01-15T18:10:28.856360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864067+0000","last_change":"2024-01-15T18:10:28.860747+0000","last_active":"2024-01-15T18:10:28.864067+0000","last_peered":"2024-01-15T18:10:28.864067+0000","last_clean":"2024-01-15T18:10:28.864067+0000","last_became_active":"2024-01-15T18:10:28.860414+0000","last_became_peered":"2024-01-15T18:10:28.860414+0000","last_unstale":"2024-01-15T18:10:28.864067+0000","last_undegraded":"2024-01-15T18:10:28.864067+0000","last_fullsized":"2024-01-15T18:10:28.864067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:51.403865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.852905+0000","last_change":"2024-01-15T18:10:28.852905+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.852905+0000","last_undegraded":"2024-01-15T18:10:28.852905+0000","last_fullsized":"2024-01-15T18:10:28.852905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865034+0000","last_change":"2024-01-15T18:10:28.861707+0000","last_active":"2024-01-15T18:10:28.865034+0000","last_peered":"2024-01-15T18:10:28.865034+0000","last_clean":"2024-01-15T18:10:28.865034+0000","last_became_active":"2024-01-15T18:10:28.860381+0000","last_became_peered":"2024-01-15T18:10:28.860381+0000","last_unstale":"2024-01-15T18:10:28.865034+0000","last_undegraded":"2024-01-15T18:10:28.865034+0000","last_fullsized":"2024-01-15T18:10:28.865034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:47.922260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846213+0000","last_change":"2024-01-15T18:10:28.846213+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846213+0000","last_undegraded":"2024-01-15T18:10:28.846213+0000","last_fullsized":"2024-01-15T18:10:28.846213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857292+0000","last_change":"2024-01-15T18:10:28.851518+0000","last_active":"2024-01-15T18:10:28.857292+0000","last_peered":"2024-01-15T18:10:28.857292+0000","last_clean":"2024-01-15T18:10:28.857292+0000","last_became_active":"2024-01-15T18:10:28.851216+0000","last_became_peered":"2024-01-15T18:10:28.851216+0000","last_unstale":"2024-01-15T18:10:28.857292+0000","last_undegraded":"2024-01-15T18:10:28.857292+0000","last_fullsized":"2024-01-15T18:10:28.857292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:12.969455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.892135+0000","last_change":"2024-01-15T18:10:29.890269+0000","last_active":"2024-01-15T18:10:29.892135+0000","last_peered":"2024-01-15T18:10:29.892135+0000","last_clean":"2024-01-15T18:10:29.892135+0000","last_became_active":"2024-01-15T18:10:29.889995+0000","last_became_peered":"2024-01-15T18:10:29.889995+0000","last_unstale":"2024-01-15T18:10:29.892135+0000","last_undegraded":"2024-01-15T18:10:29.892135+0000","last_fullsized":"2024-01-15T18:10:29.892135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:26.719685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.854427+0000","last_change":"2024-01-15T18:10:28.847695+0000","last_active":"2024-01-15T18:10:28.854427+0000","last_peered":"2024-01-15T18:10:28.854427+0000","last_clean":"2024-01-15T18:10:28.854427+0000","last_became_active":"2024-01-15T18:10:28.847366+0000","last_became_peered":"2024-01-15T18:10:28.847366+0000","last_unstale":"2024-01-15T18:10:28.854427+0000","last_undegraded":"2024-01-15T18:10:28.854427+0000","last_fullsized":"2024-01-15T18:10:28.854427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:05.863393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843192+0000","last_change":"2024-01-15T18:10:28.843192+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843192+0000","last_undegraded":"2024-01-15T18:10:28.843192+0000","last_fullsized":"2024-01-15T18:10:28.843192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.863872+0000","last_change":"2024-01-15T18:10:28.860229+0000","last_active":"2024-01-15T18:10:28.863872+0000","last_peered":"2024-01-15T18:10:28.863872+0000","last_clean":"2024-01-15T18:10:28.863872+0000","last_became_active":"2024-01-15T18:10:28.859548+0000","last_became_peered":"2024-01-15T18:10:28.859548+0000","last_unstale":"2024-01-15T18:10:28.863872+0000","last_undegraded":"2024-01-15T18:10:28.863872+0000","last_fullsized":"2024-01-15T18:10:28.863872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:37.763893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.851504+0000","last_change":"2024-01-15T18:10:28.851504+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.851504+0000","last_undegraded":"2024-01-15T18:10:28.851504+0000","last_fullsized":"2024-01-15T18:10:28.851504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864540+0000","last_change":"2024-01-15T18:10:28.860903+0000","last_active":"2024-01-15T18:10:28.864540+0000","last_peered":"2024-01-15T18:10:28.864540+0000","last_clean":"2024-01-15T18:10:28.864540+0000","last_became_active":"2024-01-15T18:10:28.860595+0000","last_became_peered":"2024-01-15T18:10:28.860595+0000","last_unstale":"2024-01-15T18:10:28.864540+0000","last_undegraded":"2024-01-15T18:10:28.864540+0000","last_fullsized":"2024-01-15T18:10:28.864540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:15:54.741452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874796+0000","last_change":"2024-01-15T18:10:29.871081+0000","last_active":"2024-01-15T18:10:29.874796+0000","last_peered":"2024-01-15T18:10:29.874796+0000","last_clean":"2024-01-15T18:10:29.874796+0000","last_became_active":"2024-01-15T18:10:29.870386+0000","last_became_peered":"2024-01-15T18:10:29.870386+0000","last_unstale":"2024-01-15T18:10:29.874796+0000","last_undegraded":"2024-01-15T18:10:29.874796+0000","last_fullsized":"2024-01-15T18:10:29.874796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:25.828362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857132+0000","last_change":"2024-01-15T18:10:28.856007+0000","last_active":"2024-01-15T18:10:28.857132+0000","last_peered":"2024-01-15T18:10:28.857132+0000","last_clean":"2024-01-15T18:10:28.857132+0000","last_became_active":"2024-01-15T18:10:28.855729+0000","last_became_peered":"2024-01-15T18:10:28.855729+0000","last_unstale":"2024-01-15T18:10:28.857132+0000","last_undegraded":"2024-01-15T18:10:28.857132+0000","last_fullsized":"2024-01-15T18:10:28.857132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:02.736339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.842052+0000","last_change":"2024-01-15T18:10:28.842052+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.842052+0000","last_undegraded":"2024-01-15T18:10:28.842052+0000","last_fullsized":"2024-01-15T18:10:28.842052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.855932+0000","last_change":"2024-01-15T18:10:28.854028+0000","last_active":"2024-01-15T18:10:28.855932+0000","last_peered":"2024-01-15T18:10:28.855932+0000","last_clean":"2024-01-15T18:10:28.855932+0000","last_became_active":"2024-01-15T18:10:28.853569+0000","last_became_peered":"2024-01-15T18:10:28.853569+0000","last_unstale":"2024-01-15T18:10:28.855932+0000","last_undegraded":"2024-01-15T18:10:28.855932+0000","last_fullsized":"2024-01-15T18:10:28.855932+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:42.155021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841682+0000","last_change":"2024-01-15T18:10:28.841682+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841682+0000","last_undegraded":"2024-01-15T18:10:28.841682+0000","last_fullsized":"2024-01-15T18:10:28.841682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865184+0000","last_change":"2024-01-15T18:10:28.862577+0000","last_active":"2024-01-15T18:10:28.865184+0000","last_peered":"2024-01-15T18:10:28.865184+0000","last_clean":"2024-01-15T18:10:28.865184+0000","last_became_active":"2024-01-15T18:10:28.862234+0000","last_became_peered":"2024-01-15T18:10:28.862234+0000","last_unstale":"2024-01-15T18:10:28.865184+0000","last_undegraded":"2024-01-15T18:10:28.865184+0000","last_fullsized":"2024-01-15T18:10:28.865184+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:13:15.657582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840553+0000","last_change":"2024-01-15T18:10:28.840553+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840553+0000","last_undegraded":"2024-01-15T18:10:28.840553+0000","last_fullsized":"2024-01-15T18:10:28.840553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861386+0000","last_change":"2024-01-15T18:10:28.859089+0000","last_active":"2024-01-15T18:10:28.861386+0000","last_peered":"2024-01-15T18:10:28.861386+0000","last_clean":"2024-01-15T18:10:28.861386+0000","last_became_active":"2024-01-15T18:10:28.858772+0000","last_became_peered":"2024-01-15T18:10:28.858772+0000","last_unstale":"2024-01-15T18:10:28.861386+0000","last_undegraded":"2024-01-15T18:10:28.861386+0000","last_fullsized":"2024-01-15T18:10:28.861386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:36.585906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840915+0000","last_change":"2024-01-15T18:10:28.840915+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840915+0000","last_undegraded":"2024-01-15T18:10:28.840915+0000","last_fullsized":"2024-01-15T18:10:28.840915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865505+0000","last_change":"2024-01-15T18:10:28.864142+0000","last_active":"2024-01-15T18:10:28.865505+0000","last_peered":"2024-01-15T18:10:28.865505+0000","last_clean":"2024-01-15T18:10:28.865505+0000","last_became_active":"2024-01-15T18:10:28.863733+0000","last_became_peered":"2024-01-15T18:10:28.863733+0000","last_unstale":"2024-01-15T18:10:28.865505+0000","last_undegraded":"2024-01-15T18:10:28.865505+0000","last_fullsized":"2024-01-15T18:10:28.865505+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:26.704112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856623+0000","last_change":"2024-01-15T18:10:28.856623+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856623+0000","last_undegraded":"2024-01-15T18:10:28.856623+0000","last_fullsized":"2024-01-15T18:10:28.856623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861826+0000","last_change":"2024-01-15T18:10:28.852757+0000","last_active":"2024-01-15T18:10:28.861826+0000","last_peered":"2024-01-15T18:10:28.861826+0000","last_clean":"2024-01-15T18:10:28.861826+0000","last_became_active":"2024-01-15T18:10:28.852416+0000","last_became_peered":"2024-01-15T18:10:28.852416+0000","last_unstale":"2024-01-15T18:10:28.861826+0000","last_undegraded":"2024-01-15T18:10:28.861826+0000","last_fullsized":"2024-01-15T18:10:28.861826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:37.726569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839982+0000","last_change":"2024-01-15T18:10:28.839982+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839982+0000","last_undegraded":"2024-01-15T18:10:28.839982+0000","last_fullsized":"2024-01-15T18:10:28.839982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847523+0000","last_change":"2024-01-15T18:10:28.846674+0000","last_active":"2024-01-15T18:10:28.847523+0000","last_peered":"2024-01-15T18:10:28.847523+0000","last_clean":"2024-01-15T18:10:28.847523+0000","last_became_active":"2024-01-15T18:10:28.845875+0000","last_became_peered":"2024-01-15T18:10:28.845875+0000","last_unstale":"2024-01-15T18:10:28.847523+0000","last_undegraded":"2024-01-15T18:10:28.847523+0000","last_fullsized":"2024-01-15T18:10:28.847523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:40.717853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.884387+0000","last_change":"2024-01-15T18:10:29.876230+0000","last_active":"2024-01-15T18:10:29.884387+0000","last_peered":"2024-01-15T18:10:29.884387+0000","last_clean":"2024-01-15T18:10:29.884387+0000","last_became_active":"2024-01-15T18:10:29.874419+0000","last_became_peered":"2024-01-15T18:10:29.874419+0000","last_unstale":"2024-01-15T18:10:29.884387+0000","last_undegraded":"2024-01-15T18:10:29.884387+0000","last_fullsized":"2024-01-15T18:10:29.884387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:57.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859460+0000","last_change":"2024-01-15T18:10:28.855952+0000","last_active":"2024-01-15T18:10:28.859460+0000","last_peered":"2024-01-15T18:10:28.859460+0000","last_clean":"2024-01-15T18:10:28.859460+0000","last_became_active":"2024-01-15T18:10:28.855647+0000","last_became_peered":"2024-01-15T18:10:28.855647+0000","last_unstale":"2024-01-15T18:10:28.859460+0000","last_undegraded":"2024-01-15T18:10:28.859460+0000","last_fullsized":"2024-01-15T18:10:28.859460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:17.019853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838341+0000","last_change":"2024-01-15T18:10:28.838341+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838341+0000","last_undegraded":"2024-01-15T18:10:28.838341+0000","last_fullsized":"2024-01-15T18:10:28.838341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861907+0000","last_change":"2024-01-15T18:10:28.853310+0000","last_active":"2024-01-15T18:10:28.861907+0000","last_peered":"2024-01-15T18:10:28.861907+0000","last_clean":"2024-01-15T18:10:28.861907+0000","last_became_active":"2024-01-15T18:10:28.853030+0000","last_became_peered":"2024-01-15T18:10:28.853030+0000","last_unstale":"2024-01-15T18:10:28.861907+0000","last_undegraded":"2024-01-15T18:10:28.861907+0000","last_fullsized":"2024-01-15T18:10:28.861907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:41.304535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839292+0000","last_change":"2024-01-15T18:10:28.839292+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839292+0000","last_undegraded":"2024-01-15T18:10:28.839292+0000","last_fullsized":"2024-01-15T18:10:28.839292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.858606+0000","last_change":"2024-01-15T18:10:28.856512+0000","last_active":"2024-01-15T18:10:28.858606+0000","last_peered":"2024-01-15T18:10:28.858606+0000","last_clean":"2024-01-15T18:10:28.858606+0000","last_became_active":"2024-01-15T18:10:28.855528+0000","last_became_peered":"2024-01-15T18:10:28.855528+0000","last_unstale":"2024-01-15T18:10:28.858606+0000","last_undegraded":"2024-01-15T18:10:28.858606+0000","last_fullsized":"2024-01-15T18:10:28.858606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:51.724767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886591+0000","last_change":"2024-01-15T18:10:29.881440+0000","last_active":"2024-01-15T18:10:29.886591+0000","last_peered":"2024-01-15T18:10:29.886591+0000","last_clean":"2024-01-15T18:10:29.886591+0000","last_became_active":"2024-01-15T18:10:29.881163+0000","last_became_peered":"2024-01-15T18:10:29.881163+0000","last_unstale":"2024-01-15T18:10:29.886591+0000","last_undegraded":"2024-01-15T18:10:29.886591+0000","last_fullsized":"2024-01-15T18:10:29.886591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:10.131197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865460+0000","last_change":"2024-01-15T18:10:28.864218+0000","last_active":"2024-01-15T18:10:28.865460+0000","last_peered":"2024-01-15T18:10:28.865460+0000","last_clean":"2024-01-15T18:10:28.865460+0000","last_became_active":"2024-01-15T18:10:28.863828+0000","last_became_peered":"2024-01-15T18:10:28.863828+0000","last_unstale":"2024-01-15T18:10:28.865460+0000","last_undegraded":"2024-01-15T18:10:28.865460+0000","last_fullsized":"2024-01-15T18:10:28.865460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:20.078184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.851348+0000","last_change":"2024-01-15T18:10:28.851348+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.851348+0000","last_undegraded":"2024-01-15T18:10:28.851348+0000","last_fullsized":"2024-01-15T18:10:28.851348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.860686+0000","last_change":"2024-01-15T18:10:28.859456+0000","last_active":"2024-01-15T18:10:28.860686+0000","last_peered":"2024-01-15T18:10:28.860686+0000","last_clean":"2024-01-15T18:10:28.860686+0000","last_became_active":"2024-01-15T18:10:28.858710+0000","last_became_peered":"2024-01-15T18:10:28.858710+0000","last_unstale":"2024-01-15T18:10:28.860686+0000","last_undegraded":"2024-01-15T18:10:28.860686+0000","last_fullsized":"2024-01-15T18:10:28.860686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:16.178426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856094+0000","last_change":"2024-01-15T18:10:28.856094+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856094+0000","last_undegraded":"2024-01-15T18:10:28.856094+0000","last_fullsized":"2024-01-15T18:10:28.856094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859561+0000","last_change":"2024-01-15T18:10:28.856970+0000","last_active":"2024-01-15T18:10:28.859561+0000","last_peered":"2024-01-15T18:10:28.859561+0000","last_clean":"2024-01-15T18:10:28.859561+0000","last_became_active":"2024-01-15T18:10:28.856677+0000","last_became_peered":"2024-01-15T18:10:28.856677+0000","last_unstale":"2024-01-15T18:10:28.859561+0000","last_undegraded":"2024-01-15T18:10:28.859561+0000","last_fullsized":"2024-01-15T18:10:28.859561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:41:56.867321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837448+0000","last_change":"2024-01-15T18:10:28.837448+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837448+0000","last_undegraded":"2024-01-15T18:10:28.837448+0000","last_fullsized":"2024-01-15T18:10:28.837448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857674+0000","last_change":"2024-01-15T18:10:28.853663+0000","last_active":"2024-01-15T18:10:28.857674+0000","last_peered":"2024-01-15T18:10:28.857674+0000","last_clean":"2024-01-15T18:10:28.857674+0000","last_became_active":"2024-01-15T18:10:28.849935+0000","last_became_peered":"2024-01-15T18:10:28.849935+0000","last_unstale":"2024-01-15T18:10:28.857674+0000","last_undegraded":"2024-01-15T18:10:28.857674+0000","last_fullsized":"2024-01-15T18:10:28.857674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:09.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839003+0000","last_change":"2024-01-15T18:10:28.839003+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839003+0000","last_undegraded":"2024-01-15T18:10:28.839003+0000","last_fullsized":"2024-01-15T18:10:28.839003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.852250+0000","last_change":"2024-01-15T18:10:28.848970+0000","last_active":"2024-01-15T18:10:28.852250+0000","last_peered":"2024-01-15T18:10:28.852250+0000","last_clean":"2024-01-15T18:10:28.852250+0000","last_became_active":"2024-01-15T18:10:28.848694+0000","last_became_peered":"2024-01-15T18:10:28.848694+0000","last_unstale":"2024-01-15T18:10:28.852250+0000","last_undegraded":"2024-01-15T18:10:28.852250+0000","last_fullsized":"2024-01-15T18:10:28.852250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:10.841154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837767+0000","last_change":"2024-01-15T18:10:28.837767+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837767+0000","last_undegraded":"2024-01-15T18:10:28.837767+0000","last_fullsized":"2024-01-15T18:10:28.837767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864081+0000","last_change":"2024-01-15T18:10:28.863335+0000","last_active":"2024-01-15T18:10:28.864081+0000","last_peered":"2024-01-15T18:10:28.864081+0000","last_clean":"2024-01-15T18:10:28.864081+0000","last_became_active":"2024-01-15T18:10:28.862934+0000","last_became_peered":"2024-01-15T18:10:28.862934+0000","last_unstale":"2024-01-15T18:10:28.864081+0000","last_undegraded":"2024-01-15T18:10:28.864081+0000","last_fullsized":"2024-01-15T18:10:28.864081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:22.153917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840868+0000","last_change":"2024-01-15T18:10:28.840868+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840868+0000","last_undegraded":"2024-01-15T18:10:28.840868+0000","last_fullsized":"2024-01-15T18:10:28.840868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857659+0000","last_change":"2024-01-15T18:10:28.853080+0000","last_active":"2024-01-15T18:10:28.857659+0000","last_peered":"2024-01-15T18:10:28.857659+0000","last_clean":"2024-01-15T18:10:28.857659+0000","last_became_active":"2024-01-15T18:10:28.848348+0000","last_became_peered":"2024-01-15T18:10:28.848348+0000","last_unstale":"2024-01-15T18:10:28.857659+0000","last_undegraded":"2024-01-15T18:10:28.857659+0000","last_fullsized":"2024-01-15T18:10:28.857659+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:16.924785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854397+0000","last_change":"2024-01-15T18:10:28.854397+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854397+0000","last_undegraded":"2024-01-15T18:10:28.854397+0000","last_fullsized":"2024-01-15T18:10:28.854397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.856697+0000","last_change":"2024-01-15T18:10:28.853859+0000","last_active":"2024-01-15T18:10:28.856697+0000","last_peered":"2024-01-15T18:10:28.856697+0000","last_clean":"2024-01-15T18:10:28.856697+0000","last_became_active":"2024-01-15T18:10:28.853567+0000","last_became_peered":"2024-01-15T18:10:28.853567+0000","last_unstale":"2024-01-15T18:10:28.856697+0000","last_undegraded":"2024-01-15T18:10:28.856697+0000","last_fullsized":"2024-01-15T18:10:28.856697+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:42.094019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.891545+0000","last_change":"2024-01-15T18:10:29.887191+0000","last_active":"2024-01-15T18:10:29.891545+0000","last_peered":"2024-01-15T18:10:29.891545+0000","last_clean":"2024-01-15T18:10:29.891545+0000","last_became_active":"2024-01-15T18:10:29.886890+0000","last_became_peered":"2024-01-15T18:10:29.886890+0000","last_unstale":"2024-01-15T18:10:29.891545+0000","last_undegraded":"2024-01-15T18:10:29.891545+0000","last_fullsized":"2024-01-15T18:10:29.891545+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847836+0000","last_change":"2024-01-15T18:10:28.847096+0000","last_active":"2024-01-15T18:10:28.847836+0000","last_peered":"2024-01-15T18:10:28.847836+0000","last_clean":"2024-01-15T18:10:28.847836+0000","last_became_active":"2024-01-15T18:10:28.845910+0000","last_became_peered":"2024-01-15T18:10:28.845910+0000","last_unstale":"2024-01-15T18:10:28.847836+0000","last_undegraded":"2024-01-15T18:10:28.847836+0000","last_fullsized":"2024-01-15T18:10:28.847836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:29.009395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.844184+0000","last_change":"2024-01-15T18:10:28.844184+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.844184+0000","last_undegraded":"2024-01-15T18:10:28.844184+0000","last_fullsized":"2024-01-15T18:10:28.844184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859604+0000","last_change":"2024-01-15T18:10:28.853688+0000","last_active":"2024-01-15T18:10:28.859604+0000","last_peered":"2024-01-15T18:10:28.859604+0000","last_clean":"2024-01-15T18:10:28.859604+0000","last_became_active":"2024-01-15T18:10:28.853347+0000","last_became_peered":"2024-01-15T18:10:28.853347+0000","last_unstale":"2024-01-15T18:10:28.859604+0000","last_undegraded":"2024-01-15T18:10:28.859604+0000","last_fullsized":"2024-01-15T18:10:28.859604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:19:32.207939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.855546+0000","last_change":"2024-01-15T18:10:28.855546+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.855546+0000","last_undegraded":"2024-01-15T18:10:28.855546+0000","last_fullsized":"2024-01-15T18:10:28.855546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.863889+0000","last_change":"2024-01-15T18:10:28.861226+0000","last_active":"2024-01-15T18:10:28.863889+0000","last_peered":"2024-01-15T18:10:28.863889+0000","last_clean":"2024-01-15T18:10:28.863889+0000","last_became_active":"2024-01-15T18:10:28.860947+0000","last_became_peered":"2024-01-15T18:10:28.860947+0000","last_unstale":"2024-01-15T18:10:28.863889+0000","last_undegraded":"2024-01-15T18:10:28.863889+0000","last_fullsized":"2024-01-15T18:10:28.863889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:07:04.692565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841823+0000","last_change":"2024-01-15T18:10:28.841823+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841823+0000","last_undegraded":"2024-01-15T18:10:28.841823+0000","last_fullsized":"2024-01-15T18:10:28.841823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.854829+0000","last_change":"2024-01-15T18:10:28.853003+0000","last_active":"2024-01-15T18:10:28.854829+0000","last_peered":"2024-01-15T18:10:28.854829+0000","last_clean":"2024-01-15T18:10:28.854829+0000","last_became_active":"2024-01-15T18:10:28.852522+0000","last_became_peered":"2024-01-15T18:10:28.852522+0000","last_unstale":"2024-01-15T18:10:28.854829+0000","last_undegraded":"2024-01-15T18:10:28.854829+0000","last_fullsized":"2024-01-15T18:10:28.854829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:19.865670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839292+0000","last_change":"2024-01-15T18:10:28.839292+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839292+0000","last_undegraded":"2024-01-15T18:10:28.839292+0000","last_fullsized":"2024-01-15T18:10:28.839292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865251+0000","last_change":"2024-01-15T18:10:28.861120+0000","last_active":"2024-01-15T18:10:28.865251+0000","last_peered":"2024-01-15T18:10:28.865251+0000","last_clean":"2024-01-15T18:10:28.865251+0000","last_became_active":"2024-01-15T18:10:28.860131+0000","last_became_peered":"2024-01-15T18:10:28.860131+0000","last_unstale":"2024-01-15T18:10:28.865251+0000","last_undegraded":"2024-01-15T18:10:28.865251+0000","last_fullsized":"2024-01-15T18:10:28.865251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:31.398737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886778+0000","last_change":"2024-01-15T18:10:29.883153+0000","last_active":"2024-01-15T18:10:29.886778+0000","last_peered":"2024-01-15T18:10:29.886778+0000","last_clean":"2024-01-15T18:10:29.886778+0000","last_became_active":"2024-01-15T18:10:29.882185+0000","last_became_peered":"2024-01-15T18:10:29.882185+0000","last_unstale":"2024-01-15T18:10:29.886778+0000","last_undegraded":"2024-01-15T18:10:29.886778+0000","last_fullsized":"2024-01-15T18:10:29.886778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865274+0000","last_change":"2024-01-15T18:10:28.863474+0000","last_active":"2024-01-15T18:10:28.865274+0000","last_peered":"2024-01-15T18:10:28.865274+0000","last_clean":"2024-01-15T18:10:28.865274+0000","last_became_active":"2024-01-15T18:10:28.863019+0000","last_became_peered":"2024-01-15T18:10:28.863019+0000","last_unstale":"2024-01-15T18:10:28.865274+0000","last_undegraded":"2024-01-15T18:10:28.865274+0000","last_fullsized":"2024-01-15T18:10:28.865274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:14.613744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.853917+0000","last_change":"2024-01-15T18:10:28.853917+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.853917+0000","last_undegraded":"2024-01-15T18:10:28.853917+0000","last_fullsized":"2024-01-15T18:10:28.853917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865274+0000","last_change":"2024-01-15T18:10:28.863368+0000","last_active":"2024-01-15T18:10:28.865274+0000","last_peered":"2024-01-15T18:10:28.865274+0000","last_clean":"2024-01-15T18:10:28.865274+0000","last_became_active":"2024-01-15T18:10:28.861032+0000","last_became_peered":"2024-01-15T18:10:28.861032+0000","last_unstale":"2024-01-15T18:10:28.865274+0000","last_undegraded":"2024-01-15T18:10:28.865274+0000","last_fullsized":"2024-01-15T18:10:28.865274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:54:11.743800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837201+0000","last_change":"2024-01-15T18:10:28.837201+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837201+0000","last_undegraded":"2024-01-15T18:10:28.837201+0000","last_fullsized":"2024-01-15T18:10:28.837201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.870806+0000","last_change":"2024-01-15T18:10:28.859062+0000","last_active":"2024-01-15T18:10:29.870806+0000","last_peered":"2024-01-15T18:10:29.870806+0000","last_clean":"2024-01-15T18:10:29.870806+0000","last_became_active":"2024-01-15T18:10:28.858021+0000","last_became_peered":"2024-01-15T18:10:28.858021+0000","last_unstale":"2024-01-15T18:10:29.870806+0000","last_undegraded":"2024-01-15T18:10:29.870806+0000","last_fullsized":"2024-01-15T18:10:29.870806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":227,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847139+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:10:28.847139+0000","last_peered":"2024-01-15T18:10:28.847139+0000","last_clean":"2024-01-15T18:10:28.847139+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:10:28.847139+0000","last_undegraded":"2024-01-15T18:10:28.847139+0000","last_fullsized":"2024-01-15T18:10:28.847139+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77156,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294684,"statfs":{"total":96636764160,"available":96557756416,"internally_reserved":0,"allocated":1658880,"data_stored":1049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.3799999999999999},"min":{"1min":0.81899999999999995,"5min":0.626,"15min":0.53600000000000003},"max":{"1min":1.677,"5min":4.0019999999999998,"15min":14.114000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.7030000000000001,"15min":1.5680000000000001},"min":{"1min":0.91200000000000003,"5min":0.69999999999999996,"15min":0.51700000000000002},"max":{"1min":1.6970000000000001,"5min":27.32,"15min":27.32},"last":1.6859999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.8300000000000001,"15min":1.4670000000000001},"min":{"1min":0.97299999999999998,"5min":0.53300000000000003,"15min":0.40300000000000002},"max":{"1min":1.617,"5min":29.167999999999999,"15min":29.167999999999999},"last":1.103},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.835,"5min":4.2699999999999996,"15min":12.565},"last":1.04}]},{"osd":2,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3360000000000001,"15min":1.464},"min":{"1min":0.61399999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.55,"5min":4.1920000000000002,"15min":13.801},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.7,"15min":1.542},"min":{"1min":0.91500000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.788,"5min":27.358000000000001,"15min":27.358000000000001},"last":1.1970000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.853,"15min":1.609},"min":{"1min":1.097,"5min":0.61299999999999999,"15min":0.47099999999999997},"max":{"1min":1.619,"5min":29.204999999999998,"15min":29.204999999999998},"last":1.377},{"interface":"front","average":{"1min":1.425,"5min":1.506,"15min":1.5600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":1.8300000000000001,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.516}]},{"osd":4,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.7509999999999999,"15min":1.621},"min":{"1min":0.91200000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.724,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.569},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.887,"15min":1.653},"min":{"1min":1.151,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":1.782,"5min":29.25,"15min":29.25},"last":1.2689999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.78,"15min":1.679},"min":{"1min":0.91500000000000004,"5min":0.83199999999999996,"15min":0.39000000000000001},"max":{"1min":1.944,"5min":11.834,"15min":12.518000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.9159999999999999,"15min":1.722},"min":{"1min":1.03,"5min":0.60499999999999998,"15min":0.41099999999999998},"max":{"1min":1.883,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.421}]},{"osd":6,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0579999999999998,"15min":2.367},"min":{"1min":1.0169999999999999,"5min":0.86499999999999999,"15min":0.45600000000000002},"max":{"1min":2.0390000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.772},{"interface":"front","average":{"1min":1.591,"5min":2.101,"15min":1.8080000000000001},"min":{"1min":1.1899999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":1.923,"5min":29.106999999999999,"15min":29.106999999999999},"last":1.4710000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58592,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313248,"statfs":{"total":96636764160,"available":96576765952,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.276,"15min":1.1950000000000001},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":5.0330000000000004,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.405,"5min":1.3959999999999999,"15min":1.339},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.1659999999999999,"5min":9.548,"15min":21.565000000000001},"last":1.2110000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4319999999999999,"15min":1.3440000000000001},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"max":{"1min":5.1150000000000002,"5min":8.1189999999999998,"15min":8.1189999999999998},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.462,"15min":1.4370000000000001},"min":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.73,"5min":9.6389999999999993,"15min":13.984999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3819999999999999,"15min":1.4630000000000001},"min":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":2.919,"5min":7.0030000000000001,"15min":34.670000000000002},"last":1.151},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.411},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.1609999999999996,"5min":6.585,"15min":6.585},"last":1.0529999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6479999999999999,"15min":1.5529999999999999},"min":{"1min":1.0069999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.5680000000000001,"5min":16.855,"15min":16.855},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6180000000000001,"15min":1.679},"min":{"1min":0.96599999999999997,"5min":0.628,"15min":0.48499999999999999},"max":{"1min":2.9630000000000001,"5min":8.0139999999999993,"15min":34.725000000000001},"last":1.4950000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.512,"15min":1.4390000000000001},"min":{"1min":0.95399999999999996,"5min":0.56299999999999994,"15min":0.38400000000000001},"max":{"1min":5.2030000000000003,"5min":6.5220000000000002,"15min":8.5489999999999995},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.784,"5min":1.6379999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.63600000000000001,"15min":0.501},"max":{"1min":7.3230000000000004,"5min":10.118,"15min":10.118},"last":1.359}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.0720000000000001,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":2.8420000000000001,"5min":7.9500000000000002,"15min":34.798000000000002},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.7410000000000001,"15min":1.609},"min":{"1min":1.228,"5min":0.66000000000000003,"15min":0.47899999999999998},"max":{"1min":8.0129999999999999,"5min":10.077,"15min":10.077},"last":1.2689999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":1.748,"15min":1.669},"min":{"1min":1.228,"5min":0.78900000000000003,"15min":0.64000000000000001},"max":{"1min":5.5060000000000002,"5min":10.302,"15min":10.302},"last":1.544},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.7629999999999999,"15min":1.8500000000000001},"min":{"1min":1.2110000000000001,"5min":0.84599999999999997,"15min":0.52600000000000002},"max":{"1min":2.782,"5min":8.0519999999999996,"15min":34.762999999999998},"last":1.401}]}]},{"osd":5,"up_from":14,"seq":60129542305,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54432,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317408,"statfs":{"total":96636764160,"available":96581025792,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.5780000000000001,"15min":1.3660000000000001},"min":{"1min":0.71499999999999997,"5min":0.67900000000000005,"15min":0.52700000000000002},"max":{"1min":1.5980000000000001,"5min":26.077999999999999,"15min":26.077999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.23,"5min":1.431,"15min":1.4470000000000001},"min":{"1min":0.76400000000000001,"5min":0.73999999999999999,"15min":0.52400000000000002},"max":{"1min":1.7270000000000001,"5min":7.7220000000000004,"15min":14.711},"last":1.5669999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.653,"15min":1.387},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6679999999999999,"5min":26,"15min":26},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.409,"15min":1.3400000000000001},"min":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":1.6919999999999999,"5min":7.5309999999999997,"15min":12.619},"last":1.5269999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.8340000000000001,"15min":1.5409999999999999},"min":{"1min":0.874,"5min":0.73299999999999998,"15min":0.55700000000000005},"max":{"1min":4.8719999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.605,"5min":1.825,"15min":1.6020000000000001},"min":{"1min":0.90200000000000002,"5min":0.63900000000000001,"15min":0.59099999999999997},"max":{"1min":4.968,"5min":17.126999999999999,"15min":17.126999999999999},"last":1.3959999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4670000000000001,"15min":1.448},"min":{"1min":0.85999999999999999,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":1.722,"5min":7.6150000000000002,"15min":12.722},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.8740000000000001,"15min":1.637},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.63400000000000001},"max":{"1min":1.776,"5min":26.134,"15min":26.134},"last":1.167}]},{"osd":4,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.754,"5min":0.68999999999999995,"15min":0.55900000000000005},"max":{"1min":1.72,"5min":7.8730000000000002,"15min":25.776},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.9470000000000001,"15min":1.734},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.47799999999999998},"max":{"1min":1.8799999999999999,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.6259999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.466,"15min":1.5489999999999999},"min":{"1min":0.84099999999999997,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":1.9390000000000001,"5min":7.9240000000000004,"15min":12.769},"last":1.337},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.798,"15min":1.8029999999999999},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.435},"max":{"1min":1.865,"5min":26.186,"15min":36.087000000000003},"last":1.2869999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.9319999999999999,"15min":1.8959999999999999},"min":{"1min":1.0680000000000001,"5min":0.79100000000000004,"15min":0.68000000000000005},"max":{"1min":1.98,"5min":9.6859999999999999,"15min":13.706},"last":1.679},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.784},"min":{"1min":0.80000000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9159999999999999,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.6040000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58588,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313252,"statfs":{"total":96636764160,"available":96576770048,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1870000000000001,"15min":1.3580000000000001},"min":{"1min":0.55700000000000005,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.9419999999999999,"5min":4.8170000000000002,"15min":21.643000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.3049999999999999,"15min":1.2949999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.861,"5min":8.9109999999999996,"15min":11.196},"last":1.238}]},{"osd":1,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5960000000000001,"15min":1.4059999999999999},"min":{"1min":0.59299999999999997,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":1.587,"5min":28.663,"15min":28.663},"last":1.452},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.53,"15min":1.4970000000000001},"min":{"1min":0.72599999999999998,"5min":0.46200000000000002,"15min":0.35399999999999998},"max":{"1min":1.837,"5min":13.614000000000001,"15min":21.440000000000001},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.6220000000000001,"15min":1.446},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.8049999999999999,"5min":16.023,"15min":16.023},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.446,"15min":1.5109999999999999},"min":{"1min":0.94299999999999995,"5min":0.52500000000000002,"15min":0.38100000000000001},"max":{"1min":1.7010000000000001,"5min":7.9169999999999998,"15min":22.350999999999999},"last":1.2010000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.375,"15min":1.512},"min":{"1min":0.874,"5min":0.56699999999999995,"15min":0.48799999999999999},"max":{"1min":1.5329999999999999,"5min":5.0860000000000003,"15min":15.093},"last":1.355},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.6919999999999999,"15min":1.651},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.6100000000000001,"5min":18.027000000000001,"15min":19.486000000000001},"last":1.256}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3859999999999999,"15min":1.766},"min":{"1min":0.67800000000000005,"5min":0.41899999999999998,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":5.29,"15min":54.938000000000002},"last":1.387},{"interface":"front","average":{"1min":1.294,"5min":1.5620000000000001,"15min":1.5309999999999999},"min":{"1min":0.63,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":1.9610000000000001,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.1559999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.6890000000000001,"15min":1.615},"min":{"1min":0.90800000000000003,"5min":0.495,"15min":0.38700000000000001},"max":{"1min":1.9279999999999999,"5min":16.103999999999999,"15min":17.733000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.7050000000000001,"15min":1.853},"min":{"1min":0.94699999999999995,"5min":0.499,"15min":0.499},"max":{"1min":2.0430000000000001,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.0600000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.98,"15min":1.825},"min":{"1min":0.79100000000000004,"5min":0.46999999999999997,"15min":0.45100000000000001},"max":{"1min":2.0600000000000001,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.3},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8939999999999999,"15min":1.847},"min":{"1min":0.65200000000000002,"5min":0.62,"15min":0.51000000000000001},"max":{"1min":2.0310000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.552}]}]},{"osd":3,"up_from":14,"seq":60129542305,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68252,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303588,"statfs":{"total":96636764160,"available":96566874112,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.371,"15min":1.0429999999999999},"min":{"1min":0.372,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.2050000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":1.399},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.151,"15min":1.028},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.665,"5min":5.0190000000000001,"15min":5.0529999999999999},"last":0.84899999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1060000000000001,"15min":1.0920000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.34399999999999997},"max":{"1min":1.365,"5min":5.226,"15min":8.1359999999999992},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.179},"min":{"1min":0.56899999999999995,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.8140000000000001,"5min":2.1339999999999999,"15min":7.2169999999999996},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.4510000000000001,"15min":1.256},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.859,"5min":14.025,"15min":14.025},"last":1.127},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3460000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.57399999999999995,"15min":0.46000000000000002},"max":{"1min":1.9710000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.242}]},{"osd":4,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.569,"15min":1.423},"min":{"1min":0.94099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2930000000000001,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.7430000000000001,"15min":1.5369999999999999},"min":{"1min":1.1100000000000001,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.9119999999999999,"5min":15.769,"15min":15.769},"last":1.302}]},{"osd":5,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.659,"15min":1.53},"min":{"1min":1.079,"5min":0.69299999999999995,"15min":0.65000000000000002},"max":{"1min":1.984,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.556},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.2490000000000001,"15min":1.8919999999999999},"min":{"1min":0.97599999999999998,"5min":0.67900000000000005,"15min":0.58299999999999996},"max":{"1min":2.0800000000000001,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.7270000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.86899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.6339999999999999,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.514},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.8180000000000001,"15min":1.708},"min":{"1min":1.246,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.048,"5min":6.8639999999999999,"15min":16.195},"last":1.381}]},{"osd":7,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.2949999999999999,"15min":1.859},"min":{"1min":1.034,"5min":0.76000000000000001,"15min":0.63900000000000001},"max":{"1min":2.2629999999999999,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1030000000000002,"15min":1.722},"min":{"1min":1.0049999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3679999999999999,"5min":17.622,"15min":17.622},"last":1.788}]}]},{"osd":2,"up_from":14,"seq":60129542304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68256,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303584,"statfs":{"total":96636764160,"available":96566870016,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98899999999999999,"15min":1.032},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.30099999999999999},"max":{"1min":1.3029999999999999,"5min":3.9199999999999999,"15min":6.101},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.087,"5min":1.1919999999999999,"15min":1.1419999999999999},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28299999999999997},"max":{"1min":1.6839999999999999,"5min":10.319000000000001,"15min":10.319000000000001},"last":1.6040000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0509999999999999,"15min":1.105},"min":{"1min":0.79600000000000004,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":1.5229999999999999,"5min":4.4939999999999998,"15min":10.807},"last":1.246},{"interface":"front","average":{"1min":1.131,"5min":1.2889999999999999,"15min":1.228},"min":{"1min":0.61599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5960000000000001,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":0.94899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.6180000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":2.8010000000000002,"5min":25.891999999999999,"15min":25.891999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.099,"5min":1.373,"15min":1.256},"min":{"1min":0.78700000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.7270000000000001,"5min":14.946,"15min":14.946},"last":1.073}]},{"osd":4,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.546,"15min":1.552},"min":{"1min":0.874,"5min":0.66100000000000003,"15min":0.64900000000000002},"max":{"1min":1.9339999999999999,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.6870000000000001,"15min":1.5509999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":2.4249999999999998,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.742,"15min":1.669},"min":{"1min":1.02,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8519999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.6359999999999999,"15min":1.722},"min":{"1min":1,"5min":0.71699999999999997,"15min":0.55200000000000005},"max":{"1min":2.02,"5min":8.5250000000000004,"15min":37.020000000000003},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.645,"15min":1.7150000000000001},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.889,"5min":10.185,"15min":23.762},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.9139999999999999,"15min":1.867},"min":{"1min":1.052,"5min":0.85899999999999999,"15min":0.73499999999999999},"max":{"1min":2.8860000000000001,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.5680000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":2.1349999999999998,"15min":1.931},"min":{"1min":1.0169999999999999,"5min":0.85999999999999999,"15min":0.68999999999999995},"max":{"1min":1.9470000000000001,"5min":19.986000000000001,"15min":52.767000000000003},"last":1.786},{"interface":"front","average":{"1min":1.605,"5min":1.988,"15min":1.804},"min":{"1min":0.97899999999999998,"5min":0.88100000000000001,"15min":0.83499999999999996},"max":{"1min":2.9169999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":1.7370000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542304,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68256,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303584,"statfs":{"total":96636764160,"available":96566870016,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.167,"15min":1.0189999999999999},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3160000000000001,"5min":19.218,"15min":19.218},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.288,"15min":1.1879999999999999},"min":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.573,"5min":19.823,"15min":19.823},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0129999999999999,"15min":1.0940000000000001},"min":{"1min":0.47599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6759999999999999,"5min":2.8980000000000001,"15min":7.6680000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.407,"15min":1.2470000000000001},"min":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.32500000000000001},"max":{"1min":1.52,"5min":19.318000000000001,"15min":19.318000000000001},"last":0.82299999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5109999999999999,"15min":1.365},"min":{"1min":0.56699999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6100000000000001,"5min":19.387,"15min":19.387},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.4379999999999999,"15min":1.351},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.8919999999999999,"5min":19.733000000000001,"15min":19.733000000000001},"last":1.28}]},{"osd":4,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.7010000000000001,"15min":1.542},"min":{"1min":0.79600000000000004,"5min":0.70599999999999996,"15min":0.497},"max":{"1min":1.7549999999999999,"5min":19.456,"15min":19.456},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.619,"15min":1.665},"min":{"1min":0.76600000000000001,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.859,"5min":19.925999999999998,"15min":22.204000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6279999999999999,"15min":1.6830000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":1.8169999999999999,"5min":20.012,"15min":20.012},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.464,"15min":1.6240000000000001},"min":{"1min":0.71599999999999997,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":1.774,"5min":2.6190000000000002,"15min":19.998999999999999},"last":1.714}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.7669999999999999,"15min":1.6910000000000001},"min":{"1min":0.95799999999999996,"5min":0.70799999999999996,"15min":0.54100000000000004},"max":{"1min":1.8120000000000001,"5min":19.526,"15min":19.526},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.8520000000000001,"15min":1.8200000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":1.9550000000000001,"5min":20.064,"15min":27.579999999999998},"last":1.395}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.8460000000000001,"15min":1.8120000000000001},"min":{"1min":0.81699999999999995,"5min":0.63,"15min":0.505},"max":{"1min":1.6930000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.486},{"interface":"front","average":{"1min":1.4730000000000001,"5min":2.0139999999999998,"15min":1.8240000000000001},"min":{"1min":0.94099999999999995,"5min":0.76100000000000001,"15min":0.54500000000000004},"max":{"1min":1.8009999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.6160000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68836,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303004,"statfs":{"total":96636764160,"available":96566276096,"internally_reserved":0,"allocated":1658880,"data_stored":1049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.1020000000000001,"15min":1.1180000000000001},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5620000000000001,"5min":20.437999999999999,"15min":20.437999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1020000000000001,"15min":1.2030000000000001},"min":{"1min":0.61899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4370000000000001,"5min":3.492,"15min":35.137999999999998},"last":0.72699999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.375,"15min":1.278},"min":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.101,"5min":20.556000000000001,"15min":20.556000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.256,"15min":1.3640000000000001},"min":{"1min":0.44400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7350000000000001,"5min":4.7430000000000003,"15min":35.183},"last":0.96099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.143,"15min":1.1379999999999999},"min":{"1min":0.52800000000000002,"5min":0.46000000000000002,"15min":0.41699999999999998},"max":{"1min":1.3149999999999999,"5min":5.8499999999999996,"15min":8.9949999999999992},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5209999999999999,"15min":1.569},"min":{"1min":0.66700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.4119999999999999,"5min":20.670000000000002,"15min":25.372},"last":0.85099999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.514,"15min":1.6759999999999999},"min":{"1min":0.98099999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7310000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.282},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.383,"15min":1.452},"min":{"1min":0.81200000000000006,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6739999999999999,"5min":2.5049999999999999,"15min":15.968999999999999},"last":1.0449999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.7050000000000001,"15min":1.6080000000000001},"min":{"1min":0.78600000000000003,"5min":0.69799999999999995,"15min":0.63800000000000001},"max":{"1min":1.8,"5min":12.173999999999999,"15min":12.173999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.964,"15min":1.786},"min":{"1min":1.121,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":20.789999999999999,"15min":20.789999999999999},"last":1.1619999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.109,"15min":1.972},"min":{"1min":1.0109999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.74,"5min":20.893000000000001,"15min":36.390999999999998},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.726,"15min":1.9099999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.8919999999999999,"5min":8.3829999999999991,"15min":36.207000000000001},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.986,"15min":1.9339999999999999},"min":{"1min":0.94899999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.0049999999999999,"5min":24.556999999999999,"15min":35.636000000000003},"last":1.206},{"interface":"front","average":{"1min":1.429,"5min":1.768,"15min":1.649},"min":{"1min":0.91900000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.8300000000000001,"5min":11.470000000000001,"15min":11.470000000000001},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:10:34.835 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:10:35.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.092+0000 7f8113e00640 1 Processor -- start 2024-01-15T18:10:35.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.092+0000 7f8113e00640 1 -- start start 2024-01-15T18:10:35.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8113e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f810c15b7d0 0x7f810c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8113e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f810c15c4c0 0x7f810c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8113e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810c15cb00 con 0x7f810c15c0e0 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8113e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c0cccd0 con 0x7f810c15c4c0 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8113e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c0ccec0 con 0x7f810c15b7d0 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8111e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f810c15c4c0 0x7f810c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8111e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f810c15c4c0 0x7f810c06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55716/0 (socket says 172.21.15.2:55716) 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.094+0000 7f8111e00640 1 -- 172.21.15.2:0/26860896 learned_addr learned my addr 172.21.15.2:0/26860896 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.095+0000 7f8110a00640 1 --2- 172.21.15.2:0/26860896 >> v2:172.21.15.137:3300/0 conn(0x7f810c15b7d0 0x7f810c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.095+0000 7f8111e00640 1 -- 172.21.15.2:0/26860896 >> v2:172.21.15.137:3300/0 conn(0x7f810c15b7d0 msgr2=0x7f810c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:35.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.095+0000 7f8111e00640 1 --2- 172.21.15.2:0/26860896 >> v2:172.21.15.137:3300/0 conn(0x7f810c15b7d0 0x7f810c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.095+0000 7f8111e00640 1 -- 172.21.15.2:0/26860896 >> v1:172.21.15.137:6789/0 conn(0x7f810c15c0e0 legacy=0x7f810c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:35.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.095+0000 7f8111e00640 1 -- 172.21.15.2:0/26860896 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c0cd1a0 con 0x7f810c15c4c0 2024-01-15T18:10:35.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.097+0000 7f8111e00640 1 --2- 172.21.15.2:0/26860896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c4c0 0x7f810c06c9b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f810c072830 tx=0x7f810402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9746f7ef0b655ecd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.097+0000 7f810be00640 1 -- 172.21.15.2:0/26860896 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810402ca80 con 0x7f810c15c4c0 2024-01-15T18:10:35.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.097+0000 7f810be00640 1 -- 172.21.15.2:0/26860896 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810402cbe0 con 0x7f810c15c4c0 2024-01-15T18:10:35.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.097+0000 7f810be00640 1 -- 172.21.15.2:0/26860896 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040343f0 con 0x7f810c15c4c0 2024-01-15T18:10:35.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.097+0000 7f8113e00640 1 -- 172.21.15.2:0/26860896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c4c0 msgr2=0x7f810c06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.097+0000 7f8113e00640 1 --2- 172.21.15.2:0/26860896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c4c0 0x7f810c06c9b0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f810c072830 tx=0x7f810402bee0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.098+0000 7f8113e00640 1 -- 172.21.15.2:0/26860896 shutdown_connections 2024-01-15T18:10:35.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.098+0000 7f8113e00640 1 --2- 172.21.15.2:0/26860896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c4c0 0x7f810c06c9b0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.098+0000 7f8113e00640 1 --2- 172.21.15.2:0/26860896 >> v2:172.21.15.137:3300/0 conn(0x7f810c15b7d0 0x7f810c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.098+0000 7f8113e00640 1 -- 172.21.15.2:0/26860896 >> 172.21.15.2:0/26860896 conn(0x7f810c06da20 msgr2=0x7f810c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:35.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.098+0000 7f8113e00640 1 -- 172.21.15.2:0/26860896 shutdown_connections 2024-01-15T18:10:35.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.098+0000 7f8113e00640 1 -- 172.21.15.2:0/26860896 wait complete. 2024-01-15T18:10:35.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.100+0000 7f8113e00640 1 Processor -- start 2024-01-15T18:10:35.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.100+0000 7f8113e00640 1 -- start start 2024-01-15T18:10:35.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.101+0000 7f8113e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f810c15b7d0 0x7f810c1e8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.101+0000 7f8113e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 0x7f810c14cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.101+0000 7f8113e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f810c15c4c0 0x7f810c14d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.101+0000 7f8113e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c1e9520 con 0x7f810c15c0e0 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.101+0000 7f8113e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c0b3870 con 0x7f810c15c4c0 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.101+0000 7f8113e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c153010 con 0x7f810c15b7d0 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 0x7f810c14cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 0x7f810c14cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55724/0 (socket says 172.21.15.2:55724) 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 -- 172.21.15.2:0/999259869 learned_addr learned my addr 172.21.15.2:0/999259869 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8111e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f810c15c4c0 0x7f810c14d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8111400640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f810c15b7d0 0x7f810c1e8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 -- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f810c15b7d0 msgr2=0x7f810c1e8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f810c15b7d0 0x7f810c1e8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 -- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f810c15c4c0 msgr2=0x7f810c14d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f810c15c4c0 0x7f810c14d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.102+0000 7f8110a00640 1 -- 172.21.15.2:0/999259869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8104009b30 con 0x7f810c15c0e0 2024-01-15T18:10:35.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.103+0000 7f8110a00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 0x7f810c14cdc0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f80f400d800 tx=0x7f80f400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.103+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f400b6c0 con 0x7f810c15c0e0 2024-01-15T18:10:35.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.103+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f400b820 con 0x7f810c15c0e0 2024-01-15T18:10:35.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.103+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f400bbb0 con 0x7f810c15c0e0 2024-01-15T18:10:35.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.103+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c1532f0 con 0x7f810c15c0e0 2024-01-15T18:10:35.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.103+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c0b2960 con 0x7f810c15c0e0 2024-01-15T18:10:35.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.105+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c0686f0 con 0x7f810c15c0e0 2024-01-15T18:10:35.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.106+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f80f400bd10 con 0x7f810c15c0e0 2024-01-15T18:10:35.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.107+0000 7f8109600640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80e406bfe0 0x7f80e406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.107+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f80f4088db0 con 0x7f810c15c0e0 2024-01-15T18:10:35.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.107+0000 7f8111400640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80e406bfe0 0x7f80e406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.109+0000 7f8111400640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80e406bfe0 0x7f80e406e4a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f80fc004ea0 tx=0x7f80fc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.111+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80f40553e0 con 0x7f810c15c0e0 2024-01-15T18:10:35.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.304+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f810c0b3b50 con 0x7f80e406bfe0 2024-01-15T18:10:35.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.327+0000 7f8109600640 1 -- 172.21.15.2:0/999259869 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+282200 (secure 0 0 0) 0x7f810c0b3b50 con 0x7f80e406bfe0 2024-01-15T18:10:35.336 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:35.337 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80e406bfe0 msgr2=0x7f80e406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80e406bfe0 0x7f80e406e4a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f80fc004ea0 tx=0x7f80fc009040 comp rx=0 tx=0).stop 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 msgr2=0x7f810c14cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 0x7f810c14cdc0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f80f400d800 tx=0x7f80f400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 shutdown_connections 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f80e406bfe0 0x7f80e406e4a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f810c15c4c0 0x7f810c14d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f810c15c0e0 0x7f810c14cdc0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 --2- 172.21.15.2:0/999259869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f810c15b7d0 0x7f810c1e8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.333+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 >> 172.21.15.2:0/999259869 conn(0x7f810c06da20 msgr2=0x7f810c15acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:35.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.334+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 shutdown_connections 2024-01-15T18:10:35.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.334+0000 7f8113e00640 1 -- 172.21.15.2:0/999259869 wait complete. 2024-01-15T18:10:35.358 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":477,"stamp":"2024-01-15T18:10:33.713319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":522368,"kb_used_data":9480,"kb_used_omap":12,"kb_used_meta":512755,"kb_avail":754452352,"statfs":{"total":773094113280,"available":772559208448,"internally_reserved":0,"allocated":9707520,"data_stored":4849976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":525061681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.884985"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.853366+0000","last_change":"2024-01-15T18:10:28.853366+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.853366+0000","last_undegraded":"2024-01-15T18:10:28.853366+0000","last_fullsized":"2024-01-15T18:10:28.853366+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845232+0000","last_change":"2024-01-15T18:10:28.845232+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845232+0000","last_undegraded":"2024-01-15T18:10:28.845232+0000","last_fullsized":"2024-01-15T18:10:28.845232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.889646+0000","last_change":"2024-01-15T18:10:29.888180+0000","last_active":"2024-01-15T18:10:29.889646+0000","last_peered":"2024-01-15T18:10:29.889646+0000","last_clean":"2024-01-15T18:10:29.889646+0000","last_became_active":"2024-01-15T18:10:29.886925+0000","last_became_peered":"2024-01-15T18:10:29.886925+0000","last_unstale":"2024-01-15T18:10:29.889646+0000","last_undegraded":"2024-01-15T18:10:29.889646+0000","last_fullsized":"2024-01-15T18:10:29.889646+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:18.035419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841871+0000","last_change":"2024-01-15T18:10:28.841871+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841871+0000","last_undegraded":"2024-01-15T18:10:28.841871+0000","last_fullsized":"2024-01-15T18:10:28.841871+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854388+0000","last_change":"2024-01-15T18:10:28.854388+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854388+0000","last_undegraded":"2024-01-15T18:10:28.854388+0000","last_fullsized":"2024-01-15T18:10:28.854388+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843476+0000","last_change":"2024-01-15T18:10:28.843476+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843476+0000","last_undegraded":"2024-01-15T18:10:28.843476+0000","last_fullsized":"2024-01-15T18:10:28.843476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.844115+0000","last_change":"2024-01-15T18:10:28.844115+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.844115+0000","last_undegraded":"2024-01-15T18:10:28.844115+0000","last_fullsized":"2024-01-15T18:10:28.844115+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838777+0000","last_change":"2024-01-15T18:10:28.838777+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838777+0000","last_undegraded":"2024-01-15T18:10:28.838777+0000","last_fullsized":"2024-01-15T18:10:28.838777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841145+0000","last_change":"2024-01-15T18:10:28.841145+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841145+0000","last_undegraded":"2024-01-15T18:10:28.841145+0000","last_fullsized":"2024-01-15T18:10:28.841145+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846140+0000","last_change":"2024-01-15T18:10:28.846140+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846140+0000","last_undegraded":"2024-01-15T18:10:28.846140+0000","last_fullsized":"2024-01-15T18:10:28.846140+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840656+0000","last_change":"2024-01-15T18:10:28.840656+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840656+0000","last_undegraded":"2024-01-15T18:10:28.840656+0000","last_fullsized":"2024-01-15T18:10:28.840656+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.848480+0000","last_change":"2024-01-15T18:10:28.848480+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.848480+0000","last_undegraded":"2024-01-15T18:10:28.848480+0000","last_fullsized":"2024-01-15T18:10:28.848480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.888056+0000","last_change":"2024-01-15T18:10:29.884896+0000","last_active":"2024-01-15T18:10:29.888056+0000","last_peered":"2024-01-15T18:10:29.888056+0000","last_clean":"2024-01-15T18:10:29.888056+0000","last_became_active":"2024-01-15T18:10:29.883901+0000","last_became_peered":"2024-01-15T18:10:29.883901+0000","last_unstale":"2024-01-15T18:10:29.888056+0000","last_undegraded":"2024-01-15T18:10:29.888056+0000","last_fullsized":"2024-01-15T18:10:29.888056+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:57.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840579+0000","last_change":"2024-01-15T18:10:28.840579+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840579+0000","last_undegraded":"2024-01-15T18:10:28.840579+0000","last_fullsized":"2024-01-15T18:10:28.840579+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.842561+0000","last_change":"2024-01-15T18:10:28.842561+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.842561+0000","last_undegraded":"2024-01-15T18:10:28.842561+0000","last_fullsized":"2024-01-15T18:10:28.842561+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.847589+0000","last_change":"2024-01-15T18:10:28.847589+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.847589+0000","last_undegraded":"2024-01-15T18:10:28.847589+0000","last_fullsized":"2024-01-15T18:10:28.847589+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.852113+0000","last_change":"2024-01-15T18:10:28.852113+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.852113+0000","last_undegraded":"2024-01-15T18:10:28.852113+0000","last_fullsized":"2024-01-15T18:10:28.852113+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843978+0000","last_change":"2024-01-15T18:10:28.843978+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843978+0000","last_undegraded":"2024-01-15T18:10:28.843978+0000","last_fullsized":"2024-01-15T18:10:28.843978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840150+0000","last_change":"2024-01-15T18:10:28.840150+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840150+0000","last_undegraded":"2024-01-15T18:10:28.840150+0000","last_fullsized":"2024-01-15T18:10:28.840150+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840702+0000","last_change":"2024-01-15T18:10:28.840702+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840702+0000","last_undegraded":"2024-01-15T18:10:28.840702+0000","last_fullsized":"2024-01-15T18:10:28.840702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843143+0000","last_change":"2024-01-15T18:10:28.843143+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843143+0000","last_undegraded":"2024-01-15T18:10:28.843143+0000","last_fullsized":"2024-01-15T18:10:28.843143+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845765+0000","last_change":"2024-01-15T18:10:28.845765+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845765+0000","last_undegraded":"2024-01-15T18:10:28.845765+0000","last_fullsized":"2024-01-15T18:10:28.845765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854373+0000","last_change":"2024-01-15T18:10:28.854373+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854373+0000","last_undegraded":"2024-01-15T18:10:28.854373+0000","last_fullsized":"2024-01-15T18:10:28.854373+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838275+0000","last_change":"2024-01-15T18:10:28.838275+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838275+0000","last_undegraded":"2024-01-15T18:10:28.838275+0000","last_fullsized":"2024-01-15T18:10:28.838275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838843+0000","last_change":"2024-01-15T18:10:28.838843+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838843+0000","last_undegraded":"2024-01-15T18:10:28.838843+0000","last_fullsized":"2024-01-15T18:10:28.838843+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886570+0000","last_change":"2024-01-15T18:10:29.879335+0000","last_active":"2024-01-15T18:10:29.886570+0000","last_peered":"2024-01-15T18:10:29.886570+0000","last_clean":"2024-01-15T18:10:29.886570+0000","last_became_active":"2024-01-15T18:10:29.878999+0000","last_became_peered":"2024-01-15T18:10:29.878999+0000","last_unstale":"2024-01-15T18:10:29.886570+0000","last_undegraded":"2024-01-15T18:10:29.886570+0000","last_fullsized":"2024-01-15T18:10:29.886570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.857241+0000","last_change":"2024-01-15T18:10:28.857241+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.857241+0000","last_undegraded":"2024-01-15T18:10:28.857241+0000","last_fullsized":"2024-01-15T18:10:28.857241+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.847343+0000","last_change":"2024-01-15T18:10:28.847343+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.847343+0000","last_undegraded":"2024-01-15T18:10:28.847343+0000","last_fullsized":"2024-01-15T18:10:28.847343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840119+0000","last_change":"2024-01-15T18:10:28.840119+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840119+0000","last_undegraded":"2024-01-15T18:10:28.840119+0000","last_fullsized":"2024-01-15T18:10:28.840119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.845208+0000","last_change":"2024-01-15T18:10:28.845208+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.845208+0000","last_undegraded":"2024-01-15T18:10:28.845208+0000","last_fullsized":"2024-01-15T18:10:28.845208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846185+0000","last_change":"2024-01-15T18:10:28.846185+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846185+0000","last_undegraded":"2024-01-15T18:10:28.846185+0000","last_fullsized":"2024-01-15T18:10:28.846185+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846205+0000","last_change":"2024-01-15T18:10:28.846205+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846205+0000","last_undegraded":"2024-01-15T18:10:28.846205+0000","last_fullsized":"2024-01-15T18:10:28.846205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856360+0000","last_change":"2024-01-15T18:10:28.856360+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856360+0000","last_undegraded":"2024-01-15T18:10:28.856360+0000","last_fullsized":"2024-01-15T18:10:28.856360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864067+0000","last_change":"2024-01-15T18:10:28.860747+0000","last_active":"2024-01-15T18:10:28.864067+0000","last_peered":"2024-01-15T18:10:28.864067+0000","last_clean":"2024-01-15T18:10:28.864067+0000","last_became_active":"2024-01-15T18:10:28.860414+0000","last_became_peered":"2024-01-15T18:10:28.860414+0000","last_unstale":"2024-01-15T18:10:28.864067+0000","last_undegraded":"2024-01-15T18:10:28.864067+0000","last_fullsized":"2024-01-15T18:10:28.864067+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:51.403865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.852905+0000","last_change":"2024-01-15T18:10:28.852905+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.852905+0000","last_undegraded":"2024-01-15T18:10:28.852905+0000","last_fullsized":"2024-01-15T18:10:28.852905+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865034+0000","last_change":"2024-01-15T18:10:28.861707+0000","last_active":"2024-01-15T18:10:28.865034+0000","last_peered":"2024-01-15T18:10:28.865034+0000","last_clean":"2024-01-15T18:10:28.865034+0000","last_became_active":"2024-01-15T18:10:28.860381+0000","last_became_peered":"2024-01-15T18:10:28.860381+0000","last_unstale":"2024-01-15T18:10:28.865034+0000","last_undegraded":"2024-01-15T18:10:28.865034+0000","last_fullsized":"2024-01-15T18:10:28.865034+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:47.922260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.846213+0000","last_change":"2024-01-15T18:10:28.846213+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.846213+0000","last_undegraded":"2024-01-15T18:10:28.846213+0000","last_fullsized":"2024-01-15T18:10:28.846213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857292+0000","last_change":"2024-01-15T18:10:28.851518+0000","last_active":"2024-01-15T18:10:28.857292+0000","last_peered":"2024-01-15T18:10:28.857292+0000","last_clean":"2024-01-15T18:10:28.857292+0000","last_became_active":"2024-01-15T18:10:28.851216+0000","last_became_peered":"2024-01-15T18:10:28.851216+0000","last_unstale":"2024-01-15T18:10:28.857292+0000","last_undegraded":"2024-01-15T18:10:28.857292+0000","last_fullsized":"2024-01-15T18:10:28.857292+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:58:12.969455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.892135+0000","last_change":"2024-01-15T18:10:29.890269+0000","last_active":"2024-01-15T18:10:29.892135+0000","last_peered":"2024-01-15T18:10:29.892135+0000","last_clean":"2024-01-15T18:10:29.892135+0000","last_became_active":"2024-01-15T18:10:29.889995+0000","last_became_peered":"2024-01-15T18:10:29.889995+0000","last_unstale":"2024-01-15T18:10:29.892135+0000","last_undegraded":"2024-01-15T18:10:29.892135+0000","last_fullsized":"2024-01-15T18:10:29.892135+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:26.719685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.854427+0000","last_change":"2024-01-15T18:10:28.847695+0000","last_active":"2024-01-15T18:10:28.854427+0000","last_peered":"2024-01-15T18:10:28.854427+0000","last_clean":"2024-01-15T18:10:28.854427+0000","last_became_active":"2024-01-15T18:10:28.847366+0000","last_became_peered":"2024-01-15T18:10:28.847366+0000","last_unstale":"2024-01-15T18:10:28.854427+0000","last_undegraded":"2024-01-15T18:10:28.854427+0000","last_fullsized":"2024-01-15T18:10:28.854427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:05.863393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.843192+0000","last_change":"2024-01-15T18:10:28.843192+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.843192+0000","last_undegraded":"2024-01-15T18:10:28.843192+0000","last_fullsized":"2024-01-15T18:10:28.843192+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.863872+0000","last_change":"2024-01-15T18:10:28.860229+0000","last_active":"2024-01-15T18:10:28.863872+0000","last_peered":"2024-01-15T18:10:28.863872+0000","last_clean":"2024-01-15T18:10:28.863872+0000","last_became_active":"2024-01-15T18:10:28.859548+0000","last_became_peered":"2024-01-15T18:10:28.859548+0000","last_unstale":"2024-01-15T18:10:28.863872+0000","last_undegraded":"2024-01-15T18:10:28.863872+0000","last_fullsized":"2024-01-15T18:10:28.863872+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:27:37.763893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.851504+0000","last_change":"2024-01-15T18:10:28.851504+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.851504+0000","last_undegraded":"2024-01-15T18:10:28.851504+0000","last_fullsized":"2024-01-15T18:10:28.851504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864540+0000","last_change":"2024-01-15T18:10:28.860903+0000","last_active":"2024-01-15T18:10:28.864540+0000","last_peered":"2024-01-15T18:10:28.864540+0000","last_clean":"2024-01-15T18:10:28.864540+0000","last_became_active":"2024-01-15T18:10:28.860595+0000","last_became_peered":"2024-01-15T18:10:28.860595+0000","last_unstale":"2024-01-15T18:10:28.864540+0000","last_undegraded":"2024-01-15T18:10:28.864540+0000","last_fullsized":"2024-01-15T18:10:28.864540+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:15:54.741452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874796+0000","last_change":"2024-01-15T18:10:29.871081+0000","last_active":"2024-01-15T18:10:29.874796+0000","last_peered":"2024-01-15T18:10:29.874796+0000","last_clean":"2024-01-15T18:10:29.874796+0000","last_became_active":"2024-01-15T18:10:29.870386+0000","last_became_peered":"2024-01-15T18:10:29.870386+0000","last_unstale":"2024-01-15T18:10:29.874796+0000","last_undegraded":"2024-01-15T18:10:29.874796+0000","last_fullsized":"2024-01-15T18:10:29.874796+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:25.828362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857132+0000","last_change":"2024-01-15T18:10:28.856007+0000","last_active":"2024-01-15T18:10:28.857132+0000","last_peered":"2024-01-15T18:10:28.857132+0000","last_clean":"2024-01-15T18:10:28.857132+0000","last_became_active":"2024-01-15T18:10:28.855729+0000","last_became_peered":"2024-01-15T18:10:28.855729+0000","last_unstale":"2024-01-15T18:10:28.857132+0000","last_undegraded":"2024-01-15T18:10:28.857132+0000","last_fullsized":"2024-01-15T18:10:28.857132+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:02.736339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.842052+0000","last_change":"2024-01-15T18:10:28.842052+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.842052+0000","last_undegraded":"2024-01-15T18:10:28.842052+0000","last_fullsized":"2024-01-15T18:10:28.842052+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.855932+0000","last_change":"2024-01-15T18:10:28.854028+0000","last_active":"2024-01-15T18:10:28.855932+0000","last_peered":"2024-01-15T18:10:28.855932+0000","last_clean":"2024-01-15T18:10:28.855932+0000","last_became_active":"2024-01-15T18:10:28.853569+0000","last_became_peered":"2024-01-15T18:10:28.853569+0000","last_unstale":"2024-01-15T18:10:28.855932+0000","last_undegraded":"2024-01-15T18:10:28.855932+0000","last_fullsized":"2024-01-15T18:10:28.855932+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:42.155021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841682+0000","last_change":"2024-01-15T18:10:28.841682+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841682+0000","last_undegraded":"2024-01-15T18:10:28.841682+0000","last_fullsized":"2024-01-15T18:10:28.841682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865184+0000","last_change":"2024-01-15T18:10:28.862577+0000","last_active":"2024-01-15T18:10:28.865184+0000","last_peered":"2024-01-15T18:10:28.865184+0000","last_clean":"2024-01-15T18:10:28.865184+0000","last_became_active":"2024-01-15T18:10:28.862234+0000","last_became_peered":"2024-01-15T18:10:28.862234+0000","last_unstale":"2024-01-15T18:10:28.865184+0000","last_undegraded":"2024-01-15T18:10:28.865184+0000","last_fullsized":"2024-01-15T18:10:28.865184+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:13:15.657582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840553+0000","last_change":"2024-01-15T18:10:28.840553+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840553+0000","last_undegraded":"2024-01-15T18:10:28.840553+0000","last_fullsized":"2024-01-15T18:10:28.840553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861386+0000","last_change":"2024-01-15T18:10:28.859089+0000","last_active":"2024-01-15T18:10:28.861386+0000","last_peered":"2024-01-15T18:10:28.861386+0000","last_clean":"2024-01-15T18:10:28.861386+0000","last_became_active":"2024-01-15T18:10:28.858772+0000","last_became_peered":"2024-01-15T18:10:28.858772+0000","last_unstale":"2024-01-15T18:10:28.861386+0000","last_undegraded":"2024-01-15T18:10:28.861386+0000","last_fullsized":"2024-01-15T18:10:28.861386+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:36.585906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840915+0000","last_change":"2024-01-15T18:10:28.840915+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840915+0000","last_undegraded":"2024-01-15T18:10:28.840915+0000","last_fullsized":"2024-01-15T18:10:28.840915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865505+0000","last_change":"2024-01-15T18:10:28.864142+0000","last_active":"2024-01-15T18:10:28.865505+0000","last_peered":"2024-01-15T18:10:28.865505+0000","last_clean":"2024-01-15T18:10:28.865505+0000","last_became_active":"2024-01-15T18:10:28.863733+0000","last_became_peered":"2024-01-15T18:10:28.863733+0000","last_unstale":"2024-01-15T18:10:28.865505+0000","last_undegraded":"2024-01-15T18:10:28.865505+0000","last_fullsized":"2024-01-15T18:10:28.865505+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:26.704112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856623+0000","last_change":"2024-01-15T18:10:28.856623+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856623+0000","last_undegraded":"2024-01-15T18:10:28.856623+0000","last_fullsized":"2024-01-15T18:10:28.856623+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861826+0000","last_change":"2024-01-15T18:10:28.852757+0000","last_active":"2024-01-15T18:10:28.861826+0000","last_peered":"2024-01-15T18:10:28.861826+0000","last_clean":"2024-01-15T18:10:28.861826+0000","last_became_active":"2024-01-15T18:10:28.852416+0000","last_became_peered":"2024-01-15T18:10:28.852416+0000","last_unstale":"2024-01-15T18:10:28.861826+0000","last_undegraded":"2024-01-15T18:10:28.861826+0000","last_fullsized":"2024-01-15T18:10:28.861826+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:41:37.726569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839982+0000","last_change":"2024-01-15T18:10:28.839982+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839982+0000","last_undegraded":"2024-01-15T18:10:28.839982+0000","last_fullsized":"2024-01-15T18:10:28.839982+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847523+0000","last_change":"2024-01-15T18:10:28.846674+0000","last_active":"2024-01-15T18:10:28.847523+0000","last_peered":"2024-01-15T18:10:28.847523+0000","last_clean":"2024-01-15T18:10:28.847523+0000","last_became_active":"2024-01-15T18:10:28.845875+0000","last_became_peered":"2024-01-15T18:10:28.845875+0000","last_unstale":"2024-01-15T18:10:28.847523+0000","last_undegraded":"2024-01-15T18:10:28.847523+0000","last_fullsized":"2024-01-15T18:10:28.847523+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:40.717853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.884387+0000","last_change":"2024-01-15T18:10:29.876230+0000","last_active":"2024-01-15T18:10:29.884387+0000","last_peered":"2024-01-15T18:10:29.884387+0000","last_clean":"2024-01-15T18:10:29.884387+0000","last_became_active":"2024-01-15T18:10:29.874419+0000","last_became_peered":"2024-01-15T18:10:29.874419+0000","last_unstale":"2024-01-15T18:10:29.884387+0000","last_undegraded":"2024-01-15T18:10:29.884387+0000","last_fullsized":"2024-01-15T18:10:29.884387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:57.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859460+0000","last_change":"2024-01-15T18:10:28.855952+0000","last_active":"2024-01-15T18:10:28.859460+0000","last_peered":"2024-01-15T18:10:28.859460+0000","last_clean":"2024-01-15T18:10:28.859460+0000","last_became_active":"2024-01-15T18:10:28.855647+0000","last_became_peered":"2024-01-15T18:10:28.855647+0000","last_unstale":"2024-01-15T18:10:28.859460+0000","last_undegraded":"2024-01-15T18:10:28.859460+0000","last_fullsized":"2024-01-15T18:10:28.859460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:17.019853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.838341+0000","last_change":"2024-01-15T18:10:28.838341+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.838341+0000","last_undegraded":"2024-01-15T18:10:28.838341+0000","last_fullsized":"2024-01-15T18:10:28.838341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.861907+0000","last_change":"2024-01-15T18:10:28.853310+0000","last_active":"2024-01-15T18:10:28.861907+0000","last_peered":"2024-01-15T18:10:28.861907+0000","last_clean":"2024-01-15T18:10:28.861907+0000","last_became_active":"2024-01-15T18:10:28.853030+0000","last_became_peered":"2024-01-15T18:10:28.853030+0000","last_unstale":"2024-01-15T18:10:28.861907+0000","last_undegraded":"2024-01-15T18:10:28.861907+0000","last_fullsized":"2024-01-15T18:10:28.861907+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:08:41.304535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839292+0000","last_change":"2024-01-15T18:10:28.839292+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839292+0000","last_undegraded":"2024-01-15T18:10:28.839292+0000","last_fullsized":"2024-01-15T18:10:28.839292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.858606+0000","last_change":"2024-01-15T18:10:28.856512+0000","last_active":"2024-01-15T18:10:28.858606+0000","last_peered":"2024-01-15T18:10:28.858606+0000","last_clean":"2024-01-15T18:10:28.858606+0000","last_became_active":"2024-01-15T18:10:28.855528+0000","last_became_peered":"2024-01-15T18:10:28.855528+0000","last_unstale":"2024-01-15T18:10:28.858606+0000","last_undegraded":"2024-01-15T18:10:28.858606+0000","last_fullsized":"2024-01-15T18:10:28.858606+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:51.724767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886591+0000","last_change":"2024-01-15T18:10:29.881440+0000","last_active":"2024-01-15T18:10:29.886591+0000","last_peered":"2024-01-15T18:10:29.886591+0000","last_clean":"2024-01-15T18:10:29.886591+0000","last_became_active":"2024-01-15T18:10:29.881163+0000","last_became_peered":"2024-01-15T18:10:29.881163+0000","last_unstale":"2024-01-15T18:10:29.886591+0000","last_undegraded":"2024-01-15T18:10:29.886591+0000","last_fullsized":"2024-01-15T18:10:29.886591+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:10.131197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865460+0000","last_change":"2024-01-15T18:10:28.864218+0000","last_active":"2024-01-15T18:10:28.865460+0000","last_peered":"2024-01-15T18:10:28.865460+0000","last_clean":"2024-01-15T18:10:28.865460+0000","last_became_active":"2024-01-15T18:10:28.863828+0000","last_became_peered":"2024-01-15T18:10:28.863828+0000","last_unstale":"2024-01-15T18:10:28.865460+0000","last_undegraded":"2024-01-15T18:10:28.865460+0000","last_fullsized":"2024-01-15T18:10:28.865460+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:20.078184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.851348+0000","last_change":"2024-01-15T18:10:28.851348+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.851348+0000","last_undegraded":"2024-01-15T18:10:28.851348+0000","last_fullsized":"2024-01-15T18:10:28.851348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.860686+0000","last_change":"2024-01-15T18:10:28.859456+0000","last_active":"2024-01-15T18:10:28.860686+0000","last_peered":"2024-01-15T18:10:28.860686+0000","last_clean":"2024-01-15T18:10:28.860686+0000","last_became_active":"2024-01-15T18:10:28.858710+0000","last_became_peered":"2024-01-15T18:10:28.858710+0000","last_unstale":"2024-01-15T18:10:28.860686+0000","last_undegraded":"2024-01-15T18:10:28.860686+0000","last_fullsized":"2024-01-15T18:10:28.860686+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:16.178426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.856094+0000","last_change":"2024-01-15T18:10:28.856094+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.856094+0000","last_undegraded":"2024-01-15T18:10:28.856094+0000","last_fullsized":"2024-01-15T18:10:28.856094+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859561+0000","last_change":"2024-01-15T18:10:28.856970+0000","last_active":"2024-01-15T18:10:28.859561+0000","last_peered":"2024-01-15T18:10:28.859561+0000","last_clean":"2024-01-15T18:10:28.859561+0000","last_became_active":"2024-01-15T18:10:28.856677+0000","last_became_peered":"2024-01-15T18:10:28.856677+0000","last_unstale":"2024-01-15T18:10:28.859561+0000","last_undegraded":"2024-01-15T18:10:28.859561+0000","last_fullsized":"2024-01-15T18:10:28.859561+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:41:56.867321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837448+0000","last_change":"2024-01-15T18:10:28.837448+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837448+0000","last_undegraded":"2024-01-15T18:10:28.837448+0000","last_fullsized":"2024-01-15T18:10:28.837448+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857674+0000","last_change":"2024-01-15T18:10:28.853663+0000","last_active":"2024-01-15T18:10:28.857674+0000","last_peered":"2024-01-15T18:10:28.857674+0000","last_clean":"2024-01-15T18:10:28.857674+0000","last_became_active":"2024-01-15T18:10:28.849935+0000","last_became_peered":"2024-01-15T18:10:28.849935+0000","last_unstale":"2024-01-15T18:10:28.857674+0000","last_undegraded":"2024-01-15T18:10:28.857674+0000","last_fullsized":"2024-01-15T18:10:28.857674+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:09.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839003+0000","last_change":"2024-01-15T18:10:28.839003+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839003+0000","last_undegraded":"2024-01-15T18:10:28.839003+0000","last_fullsized":"2024-01-15T18:10:28.839003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.852250+0000","last_change":"2024-01-15T18:10:28.848970+0000","last_active":"2024-01-15T18:10:28.852250+0000","last_peered":"2024-01-15T18:10:28.852250+0000","last_clean":"2024-01-15T18:10:28.852250+0000","last_became_active":"2024-01-15T18:10:28.848694+0000","last_became_peered":"2024-01-15T18:10:28.848694+0000","last_unstale":"2024-01-15T18:10:28.852250+0000","last_undegraded":"2024-01-15T18:10:28.852250+0000","last_fullsized":"2024-01-15T18:10:28.852250+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:10.841154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837767+0000","last_change":"2024-01-15T18:10:28.837767+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837767+0000","last_undegraded":"2024-01-15T18:10:28.837767+0000","last_fullsized":"2024-01-15T18:10:28.837767+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.864081+0000","last_change":"2024-01-15T18:10:28.863335+0000","last_active":"2024-01-15T18:10:28.864081+0000","last_peered":"2024-01-15T18:10:28.864081+0000","last_clean":"2024-01-15T18:10:28.864081+0000","last_became_active":"2024-01-15T18:10:28.862934+0000","last_became_peered":"2024-01-15T18:10:28.862934+0000","last_unstale":"2024-01-15T18:10:28.864081+0000","last_undegraded":"2024-01-15T18:10:28.864081+0000","last_fullsized":"2024-01-15T18:10:28.864081+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:57:22.153917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.840868+0000","last_change":"2024-01-15T18:10:28.840868+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.840868+0000","last_undegraded":"2024-01-15T18:10:28.840868+0000","last_fullsized":"2024-01-15T18:10:28.840868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.857659+0000","last_change":"2024-01-15T18:10:28.853080+0000","last_active":"2024-01-15T18:10:28.857659+0000","last_peered":"2024-01-15T18:10:28.857659+0000","last_clean":"2024-01-15T18:10:28.857659+0000","last_became_active":"2024-01-15T18:10:28.848348+0000","last_became_peered":"2024-01-15T18:10:28.848348+0000","last_unstale":"2024-01-15T18:10:28.857659+0000","last_undegraded":"2024-01-15T18:10:28.857659+0000","last_fullsized":"2024-01-15T18:10:28.857659+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:16.924785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.854397+0000","last_change":"2024-01-15T18:10:28.854397+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.854397+0000","last_undegraded":"2024-01-15T18:10:28.854397+0000","last_fullsized":"2024-01-15T18:10:28.854397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.856697+0000","last_change":"2024-01-15T18:10:28.853859+0000","last_active":"2024-01-15T18:10:28.856697+0000","last_peered":"2024-01-15T18:10:28.856697+0000","last_clean":"2024-01-15T18:10:28.856697+0000","last_became_active":"2024-01-15T18:10:28.853567+0000","last_became_peered":"2024-01-15T18:10:28.853567+0000","last_unstale":"2024-01-15T18:10:28.856697+0000","last_undegraded":"2024-01-15T18:10:28.856697+0000","last_fullsized":"2024-01-15T18:10:28.856697+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:42.094019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.891545+0000","last_change":"2024-01-15T18:10:29.887191+0000","last_active":"2024-01-15T18:10:29.891545+0000","last_peered":"2024-01-15T18:10:29.891545+0000","last_clean":"2024-01-15T18:10:29.891545+0000","last_became_active":"2024-01-15T18:10:29.886890+0000","last_became_peered":"2024-01-15T18:10:29.886890+0000","last_unstale":"2024-01-15T18:10:29.891545+0000","last_undegraded":"2024-01-15T18:10:29.891545+0000","last_fullsized":"2024-01-15T18:10:29.891545+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847836+0000","last_change":"2024-01-15T18:10:28.847096+0000","last_active":"2024-01-15T18:10:28.847836+0000","last_peered":"2024-01-15T18:10:28.847836+0000","last_clean":"2024-01-15T18:10:28.847836+0000","last_became_active":"2024-01-15T18:10:28.845910+0000","last_became_peered":"2024-01-15T18:10:28.845910+0000","last_unstale":"2024-01-15T18:10:28.847836+0000","last_undegraded":"2024-01-15T18:10:28.847836+0000","last_fullsized":"2024-01-15T18:10:28.847836+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:50:29.009395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.844184+0000","last_change":"2024-01-15T18:10:28.844184+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.844184+0000","last_undegraded":"2024-01-15T18:10:28.844184+0000","last_fullsized":"2024-01-15T18:10:28.844184+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.859604+0000","last_change":"2024-01-15T18:10:28.853688+0000","last_active":"2024-01-15T18:10:28.859604+0000","last_peered":"2024-01-15T18:10:28.859604+0000","last_clean":"2024-01-15T18:10:28.859604+0000","last_became_active":"2024-01-15T18:10:28.853347+0000","last_became_peered":"2024-01-15T18:10:28.853347+0000","last_unstale":"2024-01-15T18:10:28.859604+0000","last_undegraded":"2024-01-15T18:10:28.859604+0000","last_fullsized":"2024-01-15T18:10:28.859604+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:19:32.207939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.855546+0000","last_change":"2024-01-15T18:10:28.855546+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.855546+0000","last_undegraded":"2024-01-15T18:10:28.855546+0000","last_fullsized":"2024-01-15T18:10:28.855546+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.863889+0000","last_change":"2024-01-15T18:10:28.861226+0000","last_active":"2024-01-15T18:10:28.863889+0000","last_peered":"2024-01-15T18:10:28.863889+0000","last_clean":"2024-01-15T18:10:28.863889+0000","last_became_active":"2024-01-15T18:10:28.860947+0000","last_became_peered":"2024-01-15T18:10:28.860947+0000","last_unstale":"2024-01-15T18:10:28.863889+0000","last_undegraded":"2024-01-15T18:10:28.863889+0000","last_fullsized":"2024-01-15T18:10:28.863889+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:07:04.692565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.841823+0000","last_change":"2024-01-15T18:10:28.841823+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.841823+0000","last_undegraded":"2024-01-15T18:10:28.841823+0000","last_fullsized":"2024-01-15T18:10:28.841823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.854829+0000","last_change":"2024-01-15T18:10:28.853003+0000","last_active":"2024-01-15T18:10:28.854829+0000","last_peered":"2024-01-15T18:10:28.854829+0000","last_clean":"2024-01-15T18:10:28.854829+0000","last_became_active":"2024-01-15T18:10:28.852522+0000","last_became_peered":"2024-01-15T18:10:28.852522+0000","last_unstale":"2024-01-15T18:10:28.854829+0000","last_undegraded":"2024-01-15T18:10:28.854829+0000","last_fullsized":"2024-01-15T18:10:28.854829+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:19.865670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.839292+0000","last_change":"2024-01-15T18:10:28.839292+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.839292+0000","last_undegraded":"2024-01-15T18:10:28.839292+0000","last_fullsized":"2024-01-15T18:10:28.839292+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865251+0000","last_change":"2024-01-15T18:10:28.861120+0000","last_active":"2024-01-15T18:10:28.865251+0000","last_peered":"2024-01-15T18:10:28.865251+0000","last_clean":"2024-01-15T18:10:28.865251+0000","last_became_active":"2024-01-15T18:10:28.860131+0000","last_became_peered":"2024-01-15T18:10:28.860131+0000","last_unstale":"2024-01-15T18:10:28.865251+0000","last_undegraded":"2024-01-15T18:10:28.865251+0000","last_fullsized":"2024-01-15T18:10:28.865251+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:31.398737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.886778+0000","last_change":"2024-01-15T18:10:29.883153+0000","last_active":"2024-01-15T18:10:29.886778+0000","last_peered":"2024-01-15T18:10:29.886778+0000","last_clean":"2024-01-15T18:10:29.886778+0000","last_became_active":"2024-01-15T18:10:29.882185+0000","last_became_peered":"2024-01-15T18:10:29.882185+0000","last_unstale":"2024-01-15T18:10:29.886778+0000","last_undegraded":"2024-01-15T18:10:29.886778+0000","last_fullsized":"2024-01-15T18:10:29.886778+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865274+0000","last_change":"2024-01-15T18:10:28.863474+0000","last_active":"2024-01-15T18:10:28.865274+0000","last_peered":"2024-01-15T18:10:28.865274+0000","last_clean":"2024-01-15T18:10:28.865274+0000","last_became_active":"2024-01-15T18:10:28.863019+0000","last_became_peered":"2024-01-15T18:10:28.863019+0000","last_unstale":"2024-01-15T18:10:28.865274+0000","last_undegraded":"2024-01-15T18:10:28.865274+0000","last_fullsized":"2024-01-15T18:10:28.865274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:14.613744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.853917+0000","last_change":"2024-01-15T18:10:28.853917+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.853917+0000","last_undegraded":"2024-01-15T18:10:28.853917+0000","last_fullsized":"2024-01-15T18:10:28.853917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.865274+0000","last_change":"2024-01-15T18:10:28.863368+0000","last_active":"2024-01-15T18:10:28.865274+0000","last_peered":"2024-01-15T18:10:28.865274+0000","last_clean":"2024-01-15T18:10:28.865274+0000","last_became_active":"2024-01-15T18:10:28.861032+0000","last_became_peered":"2024-01-15T18:10:28.861032+0000","last_unstale":"2024-01-15T18:10:28.865274+0000","last_undegraded":"2024-01-15T18:10:28.865274+0000","last_fullsized":"2024-01-15T18:10:28.865274+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:54:11.743800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":1,"reported_epoch":87,"state":"creating+peering","last_fresh":"2024-01-15T18:10:28.837201+0000","last_change":"2024-01-15T18:10:28.837201+0000","last_active":"2024-01-15T18:10:28.822137+0000","last_peered":"2024-01-15T18:10:28.822137+0000","last_clean":"2024-01-15T18:10:28.822137+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:10:28.837201+0000","last_undegraded":"2024-01-15T18:10:28.837201+0000","last_fullsized":"2024-01-15T18:10:28.837201+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.870806+0000","last_change":"2024-01-15T18:10:28.859062+0000","last_active":"2024-01-15T18:10:29.870806+0000","last_peered":"2024-01-15T18:10:29.870806+0000","last_clean":"2024-01-15T18:10:29.870806+0000","last_became_active":"2024-01-15T18:10:28.858021+0000","last_became_peered":"2024-01-15T18:10:28.858021+0000","last_unstale":"2024-01-15T18:10:29.870806+0000","last_undegraded":"2024-01-15T18:10:29.870806+0000","last_fullsized":"2024-01-15T18:10:29.870806+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":227,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-01-15T18:10:28.847139+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:10:28.847139+0000","last_peered":"2024-01-15T18:10:28.847139+0000","last_clean":"2024-01-15T18:10:28.847139+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:10:28.847139+0000","last_undegraded":"2024-01-15T18:10:28.847139+0000","last_fullsized":"2024-01-15T18:10:28.847139+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77156,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294684,"statfs":{"total":96636764160,"available":96557756416,"internally_reserved":0,"allocated":1658880,"data_stored":1049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.3799999999999999},"min":{"1min":0.81899999999999995,"5min":0.626,"15min":0.53600000000000003},"max":{"1min":1.677,"5min":4.0019999999999998,"15min":14.114000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.7030000000000001,"15min":1.5680000000000001},"min":{"1min":0.91200000000000003,"5min":0.69999999999999996,"15min":0.51700000000000002},"max":{"1min":1.6970000000000001,"5min":27.32,"15min":27.32},"last":1.6859999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.8300000000000001,"15min":1.4670000000000001},"min":{"1min":0.97299999999999998,"5min":0.53300000000000003,"15min":0.40300000000000002},"max":{"1min":1.617,"5min":29.167999999999999,"15min":29.167999999999999},"last":1.103},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.835,"5min":4.2699999999999996,"15min":12.565},"last":1.04}]},{"osd":2,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3360000000000001,"15min":1.464},"min":{"1min":0.61399999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.55,"5min":4.1920000000000002,"15min":13.801},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.7,"15min":1.542},"min":{"1min":0.91500000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.788,"5min":27.358000000000001,"15min":27.358000000000001},"last":1.1970000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.853,"15min":1.609},"min":{"1min":1.097,"5min":0.61299999999999999,"15min":0.47099999999999997},"max":{"1min":1.619,"5min":29.204999999999998,"15min":29.204999999999998},"last":1.377},{"interface":"front","average":{"1min":1.425,"5min":1.506,"15min":1.5600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":1.8300000000000001,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.516}]},{"osd":4,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.7509999999999999,"15min":1.621},"min":{"1min":0.91200000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.724,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.569},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.887,"15min":1.653},"min":{"1min":1.151,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":1.782,"5min":29.25,"15min":29.25},"last":1.2689999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.78,"15min":1.679},"min":{"1min":0.91500000000000004,"5min":0.83199999999999996,"15min":0.39000000000000001},"max":{"1min":1.944,"5min":11.834,"15min":12.518000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.9159999999999999,"15min":1.722},"min":{"1min":1.03,"5min":0.60499999999999998,"15min":0.41099999999999998},"max":{"1min":1.883,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.421}]},{"osd":6,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0579999999999998,"15min":2.367},"min":{"1min":1.0169999999999999,"5min":0.86499999999999999,"15min":0.45600000000000002},"max":{"1min":2.0390000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.772},{"interface":"front","average":{"1min":1.591,"5min":2.101,"15min":1.8080000000000001},"min":{"1min":1.1899999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":1.923,"5min":29.106999999999999,"15min":29.106999999999999},"last":1.4710000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542303,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58592,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313248,"statfs":{"total":96636764160,"available":96576765952,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.276,"15min":1.1950000000000001},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":5.0330000000000004,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.405,"5min":1.3959999999999999,"15min":1.339},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.1659999999999999,"5min":9.548,"15min":21.565000000000001},"last":1.2110000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4319999999999999,"15min":1.3440000000000001},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"max":{"1min":5.1150000000000002,"5min":8.1189999999999998,"15min":8.1189999999999998},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.462,"15min":1.4370000000000001},"min":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.73,"5min":9.6389999999999993,"15min":13.984999999999999},"last":0.98899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3819999999999999,"15min":1.4630000000000001},"min":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":2.919,"5min":7.0030000000000001,"15min":34.670000000000002},"last":1.151},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.411},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.1609999999999996,"5min":6.585,"15min":6.585},"last":1.0529999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6479999999999999,"15min":1.5529999999999999},"min":{"1min":1.0069999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.5680000000000001,"5min":16.855,"15min":16.855},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6180000000000001,"15min":1.679},"min":{"1min":0.96599999999999997,"5min":0.628,"15min":0.48499999999999999},"max":{"1min":2.9630000000000001,"5min":8.0139999999999993,"15min":34.725000000000001},"last":1.4950000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.512,"15min":1.4390000000000001},"min":{"1min":0.95399999999999996,"5min":0.56299999999999994,"15min":0.38400000000000001},"max":{"1min":5.2030000000000003,"5min":6.5220000000000002,"15min":8.5489999999999995},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.784,"5min":1.6379999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.63600000000000001,"15min":0.501},"max":{"1min":7.3230000000000004,"5min":10.118,"15min":10.118},"last":1.359}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.0720000000000001,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":2.8420000000000001,"5min":7.9500000000000002,"15min":34.798000000000002},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.7410000000000001,"15min":1.609},"min":{"1min":1.228,"5min":0.66000000000000003,"15min":0.47899999999999998},"max":{"1min":8.0129999999999999,"5min":10.077,"15min":10.077},"last":1.2689999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":1.748,"15min":1.669},"min":{"1min":1.228,"5min":0.78900000000000003,"15min":0.64000000000000001},"max":{"1min":5.5060000000000002,"5min":10.302,"15min":10.302},"last":1.544},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.7629999999999999,"15min":1.8500000000000001},"min":{"1min":1.2110000000000001,"5min":0.84599999999999997,"15min":0.52600000000000002},"max":{"1min":2.782,"5min":8.0519999999999996,"15min":34.762999999999998},"last":1.401}]}]},{"osd":5,"up_from":14,"seq":60129542305,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54432,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317408,"statfs":{"total":96636764160,"available":96581025792,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.5780000000000001,"15min":1.3660000000000001},"min":{"1min":0.71499999999999997,"5min":0.67900000000000005,"15min":0.52700000000000002},"max":{"1min":1.5980000000000001,"5min":26.077999999999999,"15min":26.077999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.23,"5min":1.431,"15min":1.4470000000000001},"min":{"1min":0.76400000000000001,"5min":0.73999999999999999,"15min":0.52400000000000002},"max":{"1min":1.7270000000000001,"5min":7.7220000000000004,"15min":14.711},"last":1.5669999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.653,"15min":1.387},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6679999999999999,"5min":26,"15min":26},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.409,"15min":1.3400000000000001},"min":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":1.6919999999999999,"5min":7.5309999999999997,"15min":12.619},"last":1.5269999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.8340000000000001,"15min":1.5409999999999999},"min":{"1min":0.874,"5min":0.73299999999999998,"15min":0.55700000000000005},"max":{"1min":4.8719999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.605,"5min":1.825,"15min":1.6020000000000001},"min":{"1min":0.90200000000000002,"5min":0.63900000000000001,"15min":0.59099999999999997},"max":{"1min":4.968,"5min":17.126999999999999,"15min":17.126999999999999},"last":1.3959999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4670000000000001,"15min":1.448},"min":{"1min":0.85999999999999999,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":1.722,"5min":7.6150000000000002,"15min":12.722},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.8740000000000001,"15min":1.637},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.63400000000000001},"max":{"1min":1.776,"5min":26.134,"15min":26.134},"last":1.167}]},{"osd":4,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.754,"5min":0.68999999999999995,"15min":0.55900000000000005},"max":{"1min":1.72,"5min":7.8730000000000002,"15min":25.776},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.9470000000000001,"15min":1.734},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.47799999999999998},"max":{"1min":1.8799999999999999,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.6259999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.466,"15min":1.5489999999999999},"min":{"1min":0.84099999999999997,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":1.9390000000000001,"5min":7.9240000000000004,"15min":12.769},"last":1.337},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.798,"15min":1.8029999999999999},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.435},"max":{"1min":1.865,"5min":26.186,"15min":36.087000000000003},"last":1.2869999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.9319999999999999,"15min":1.8959999999999999},"min":{"1min":1.0680000000000001,"5min":0.79100000000000004,"15min":0.68000000000000005},"max":{"1min":1.98,"5min":9.6859999999999999,"15min":13.706},"last":1.679},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.784},"min":{"1min":0.80000000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9159999999999999,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.6040000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542304,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58588,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313252,"statfs":{"total":96636764160,"available":96576770048,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1870000000000001,"15min":1.3580000000000001},"min":{"1min":0.55700000000000005,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.9419999999999999,"5min":4.8170000000000002,"15min":21.643000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.3049999999999999,"15min":1.2949999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.861,"5min":8.9109999999999996,"15min":11.196},"last":1.238}]},{"osd":1,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5960000000000001,"15min":1.4059999999999999},"min":{"1min":0.59299999999999997,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":1.587,"5min":28.663,"15min":28.663},"last":1.452},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.53,"15min":1.4970000000000001},"min":{"1min":0.72599999999999998,"5min":0.46200000000000002,"15min":0.35399999999999998},"max":{"1min":1.837,"5min":13.614000000000001,"15min":21.440000000000001},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.6220000000000001,"15min":1.446},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.8049999999999999,"5min":16.023,"15min":16.023},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.446,"15min":1.5109999999999999},"min":{"1min":0.94299999999999995,"5min":0.52500000000000002,"15min":0.38100000000000001},"max":{"1min":1.7010000000000001,"5min":7.9169999999999998,"15min":22.350999999999999},"last":1.2010000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.375,"15min":1.512},"min":{"1min":0.874,"5min":0.56699999999999995,"15min":0.48799999999999999},"max":{"1min":1.5329999999999999,"5min":5.0860000000000003,"15min":15.093},"last":1.355},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.6919999999999999,"15min":1.651},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.6100000000000001,"5min":18.027000000000001,"15min":19.486000000000001},"last":1.256}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3859999999999999,"15min":1.766},"min":{"1min":0.67800000000000005,"5min":0.41899999999999998,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":5.29,"15min":54.938000000000002},"last":1.387},{"interface":"front","average":{"1min":1.294,"5min":1.5620000000000001,"15min":1.5309999999999999},"min":{"1min":0.63,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":1.9610000000000001,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.1559999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.6890000000000001,"15min":1.615},"min":{"1min":0.90800000000000003,"5min":0.495,"15min":0.38700000000000001},"max":{"1min":1.9279999999999999,"5min":16.103999999999999,"15min":17.733000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.7050000000000001,"15min":1.853},"min":{"1min":0.94699999999999995,"5min":0.499,"15min":0.499},"max":{"1min":2.0430000000000001,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.0600000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.98,"15min":1.825},"min":{"1min":0.79100000000000004,"5min":0.46999999999999997,"15min":0.45100000000000001},"max":{"1min":2.0600000000000001,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.3},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8939999999999999,"15min":1.847},"min":{"1min":0.65200000000000002,"5min":0.62,"15min":0.51000000000000001},"max":{"1min":2.0310000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.552}]}]},{"osd":3,"up_from":14,"seq":60129542305,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68252,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303588,"statfs":{"total":96636764160,"available":96566874112,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.371,"15min":1.0429999999999999},"min":{"1min":0.372,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.2050000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":1.399},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.151,"15min":1.028},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.665,"5min":5.0190000000000001,"15min":5.0529999999999999},"last":0.84899999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1060000000000001,"15min":1.0920000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.34399999999999997},"max":{"1min":1.365,"5min":5.226,"15min":8.1359999999999992},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.179},"min":{"1min":0.56899999999999995,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.8140000000000001,"5min":2.1339999999999999,"15min":7.2169999999999996},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.4510000000000001,"15min":1.256},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.859,"5min":14.025,"15min":14.025},"last":1.127},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3460000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.57399999999999995,"15min":0.46000000000000002},"max":{"1min":1.9710000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.242}]},{"osd":4,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.569,"15min":1.423},"min":{"1min":0.94099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2930000000000001,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.7430000000000001,"15min":1.5369999999999999},"min":{"1min":1.1100000000000001,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.9119999999999999,"5min":15.769,"15min":15.769},"last":1.302}]},{"osd":5,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.659,"15min":1.53},"min":{"1min":1.079,"5min":0.69299999999999995,"15min":0.65000000000000002},"max":{"1min":1.984,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.556},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.2490000000000001,"15min":1.8919999999999999},"min":{"1min":0.97599999999999998,"5min":0.67900000000000005,"15min":0.58299999999999996},"max":{"1min":2.0800000000000001,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.7270000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.86899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.6339999999999999,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.514},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.8180000000000001,"15min":1.708},"min":{"1min":1.246,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.048,"5min":6.8639999999999999,"15min":16.195},"last":1.381}]},{"osd":7,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.2949999999999999,"15min":1.859},"min":{"1min":1.034,"5min":0.76000000000000001,"15min":0.63900000000000001},"max":{"1min":2.2629999999999999,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.8320000000000001},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1030000000000002,"15min":1.722},"min":{"1min":1.0049999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3679999999999999,"5min":17.622,"15min":17.622},"last":1.788}]}]},{"osd":2,"up_from":14,"seq":60129542304,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68256,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303584,"statfs":{"total":96636764160,"available":96566870016,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98899999999999999,"15min":1.032},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.30099999999999999},"max":{"1min":1.3029999999999999,"5min":3.9199999999999999,"15min":6.101},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.087,"5min":1.1919999999999999,"15min":1.1419999999999999},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28299999999999997},"max":{"1min":1.6839999999999999,"5min":10.319000000000001,"15min":10.319000000000001},"last":1.6040000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0509999999999999,"15min":1.105},"min":{"1min":0.79600000000000004,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":1.5229999999999999,"5min":4.4939999999999998,"15min":10.807},"last":1.246},{"interface":"front","average":{"1min":1.131,"5min":1.2889999999999999,"15min":1.228},"min":{"1min":0.61599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5960000000000001,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":0.94899999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.6180000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":2.8010000000000002,"5min":25.891999999999999,"15min":25.891999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.099,"5min":1.373,"15min":1.256},"min":{"1min":0.78700000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.7270000000000001,"5min":14.946,"15min":14.946},"last":1.073}]},{"osd":4,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.546,"15min":1.552},"min":{"1min":0.874,"5min":0.66100000000000003,"15min":0.64900000000000002},"max":{"1min":1.9339999999999999,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.6870000000000001,"15min":1.5509999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":2.4249999999999998,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.742,"15min":1.669},"min":{"1min":1.02,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8519999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.427,"5min":1.6359999999999999,"15min":1.722},"min":{"1min":1,"5min":0.71699999999999997,"15min":0.55200000000000005},"max":{"1min":2.02,"5min":8.5250000000000004,"15min":37.020000000000003},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.645,"15min":1.7150000000000001},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.889,"5min":10.185,"15min":23.762},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.9139999999999999,"15min":1.867},"min":{"1min":1.052,"5min":0.85899999999999999,"15min":0.73499999999999999},"max":{"1min":2.8860000000000001,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.5680000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":2.1349999999999998,"15min":1.931},"min":{"1min":1.0169999999999999,"5min":0.85999999999999999,"15min":0.68999999999999995},"max":{"1min":1.9470000000000001,"5min":19.986000000000001,"15min":52.767000000000003},"last":1.786},{"interface":"front","average":{"1min":1.605,"5min":1.988,"15min":1.804},"min":{"1min":0.97899999999999998,"5min":0.88100000000000001,"15min":0.83499999999999996},"max":{"1min":2.9169999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":1.7370000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542304,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68256,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303584,"statfs":{"total":96636764160,"available":96566870016,"internally_reserved":0,"allocated":1064960,"data_stored":458655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.167,"15min":1.0189999999999999},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3160000000000001,"5min":19.218,"15min":19.218},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.288,"15min":1.1879999999999999},"min":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.573,"5min":19.823,"15min":19.823},"last":0.93300000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0129999999999999,"15min":1.0940000000000001},"min":{"1min":0.47599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6759999999999999,"5min":2.8980000000000001,"15min":7.6680000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.105,"5min":1.407,"15min":1.2470000000000001},"min":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.32500000000000001},"max":{"1min":1.52,"5min":19.318000000000001,"15min":19.318000000000001},"last":0.82299999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5109999999999999,"15min":1.365},"min":{"1min":0.56699999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6100000000000001,"5min":19.387,"15min":19.387},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.4379999999999999,"15min":1.351},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.8919999999999999,"5min":19.733000000000001,"15min":19.733000000000001},"last":1.28}]},{"osd":4,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.7010000000000001,"15min":1.542},"min":{"1min":0.79600000000000004,"5min":0.70599999999999996,"15min":0.497},"max":{"1min":1.7549999999999999,"5min":19.456,"15min":19.456},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.619,"15min":1.665},"min":{"1min":0.76600000000000001,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.859,"5min":19.925999999999998,"15min":22.204000000000001},"last":1.1559999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6279999999999999,"15min":1.6830000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":1.8169999999999999,"5min":20.012,"15min":20.012},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.464,"15min":1.6240000000000001},"min":{"1min":0.71599999999999997,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":1.774,"5min":2.6190000000000002,"15min":19.998999999999999},"last":1.714}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.7669999999999999,"15min":1.6910000000000001},"min":{"1min":0.95799999999999996,"5min":0.70799999999999996,"15min":0.54100000000000004},"max":{"1min":1.8120000000000001,"5min":19.526,"15min":19.526},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.48,"5min":1.8520000000000001,"15min":1.8200000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":1.9550000000000001,"5min":20.064,"15min":27.579999999999998},"last":1.395}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.8460000000000001,"15min":1.8120000000000001},"min":{"1min":0.81699999999999995,"5min":0.63,"15min":0.505},"max":{"1min":1.6930000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.486},{"interface":"front","average":{"1min":1.4730000000000001,"5min":2.0139999999999998,"15min":1.8240000000000001},"min":{"1min":0.94099999999999995,"5min":0.76100000000000001,"15min":0.54500000000000004},"max":{"1min":1.8009999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.6160000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542303,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68836,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303004,"statfs":{"total":96636764160,"available":96566276096,"internally_reserved":0,"allocated":1658880,"data_stored":1049023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.1020000000000001,"15min":1.1180000000000001},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5620000000000001,"5min":20.437999999999999,"15min":20.437999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1020000000000001,"15min":1.2030000000000001},"min":{"1min":0.61899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4370000000000001,"5min":3.492,"15min":35.137999999999998},"last":0.72699999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.375,"15min":1.278},"min":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.101,"5min":20.556000000000001,"15min":20.556000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.256,"15min":1.3640000000000001},"min":{"1min":0.44400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7350000000000001,"5min":4.7430000000000003,"15min":35.183},"last":0.96099999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.143,"15min":1.1379999999999999},"min":{"1min":0.52800000000000002,"5min":0.46000000000000002,"15min":0.41699999999999998},"max":{"1min":1.3149999999999999,"5min":5.8499999999999996,"15min":8.9949999999999992},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5209999999999999,"15min":1.569},"min":{"1min":0.66700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.4119999999999999,"5min":20.670000000000002,"15min":25.372},"last":0.85099999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.514,"15min":1.6759999999999999},"min":{"1min":0.98099999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7310000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.282},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.383,"15min":1.452},"min":{"1min":0.81200000000000006,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6739999999999999,"5min":2.5049999999999999,"15min":15.968999999999999},"last":1.0449999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.7050000000000001,"15min":1.6080000000000001},"min":{"1min":0.78600000000000003,"5min":0.69799999999999995,"15min":0.63800000000000001},"max":{"1min":1.8,"5min":12.173999999999999,"15min":12.173999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.964,"15min":1.786},"min":{"1min":1.121,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":20.789999999999999,"15min":20.789999999999999},"last":1.1619999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.109,"15min":1.972},"min":{"1min":1.0109999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.74,"5min":20.893000000000001,"15min":36.390999999999998},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.726,"15min":1.9099999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.8919999999999999,"5min":8.3829999999999991,"15min":36.207000000000001},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.986,"15min":1.9339999999999999},"min":{"1min":0.94899999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.0049999999999999,"5min":24.556999999999999,"15min":35.636000000000003},"last":1.206},{"interface":"front","average":{"1min":1.429,"5min":1.768,"15min":1.649},"min":{"1min":0.91900000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.8300000000000001,"5min":11.470000000000001,"15min":11.470000000000001},"last":0.83599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3f is not active+clean 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.853366+0000', 'last_change': '2024-01-15T18:10:28.853366+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.853366+0000', 'last_undegraded': '2024-01-15T18:10:28.853366+0000', 'last_fullsized': '2024-01-15T18:10:28.853366+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3e is not active+clean 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.845232+0000', 'last_change': '2024-01-15T18:10:28.845232+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.845232+0000', 'last_undegraded': '2024-01-15T18:10:28.845232+0000', 'last_fullsized': '2024-01-15T18:10:28.845232+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3c is not active+clean 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.841871+0000', 'last_change': '2024-01-15T18:10:28.841871+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.841871+0000', 'last_undegraded': '2024-01-15T18:10:28.841871+0000', 'last_fullsized': '2024-01-15T18:10:28.841871+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3b is not active+clean 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.854388+0000', 'last_change': '2024-01-15T18:10:28.854388+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.854388+0000', 'last_undegraded': '2024-01-15T18:10:28.854388+0000', 'last_fullsized': '2024-01-15T18:10:28.854388+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3a is not active+clean 2024-01-15T18:10:35.363 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.843476+0000', 'last_change': '2024-01-15T18:10:28.843476+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.843476+0000', 'last_undegraded': '2024-01-15T18:10:28.843476+0000', 'last_fullsized': '2024-01-15T18:10:28.843476+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.39 is not active+clean 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.844115+0000', 'last_change': '2024-01-15T18:10:28.844115+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.844115+0000', 'last_undegraded': '2024-01-15T18:10:28.844115+0000', 'last_fullsized': '2024-01-15T18:10:28.844115+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.38 is not active+clean 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.838777+0000', 'last_change': '2024-01-15T18:10:28.838777+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.838777+0000', 'last_undegraded': '2024-01-15T18:10:28.838777+0000', 'last_fullsized': '2024-01-15T18:10:28.838777+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.37 is not active+clean 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.841145+0000', 'last_change': '2024-01-15T18:10:28.841145+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.841145+0000', 'last_undegraded': '2024-01-15T18:10:28.841145+0000', 'last_fullsized': '2024-01-15T18:10:28.841145+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.36 is not active+clean 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.846140+0000', 'last_change': '2024-01-15T18:10:28.846140+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.846140+0000', 'last_undegraded': '2024-01-15T18:10:28.846140+0000', 'last_fullsized': '2024-01-15T18:10:28.846140+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.35 is not active+clean 2024-01-15T18:10:35.364 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840656+0000', 'last_change': '2024-01-15T18:10:28.840656+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840656+0000', 'last_undegraded': '2024-01-15T18:10:28.840656+0000', 'last_fullsized': '2024-01-15T18:10:28.840656+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.34 is not active+clean 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.848480+0000', 'last_change': '2024-01-15T18:10:28.848480+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.848480+0000', 'last_undegraded': '2024-01-15T18:10:28.848480+0000', 'last_fullsized': '2024-01-15T18:10:28.848480+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.32 is not active+clean 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840579+0000', 'last_change': '2024-01-15T18:10:28.840579+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840579+0000', 'last_undegraded': '2024-01-15T18:10:28.840579+0000', 'last_fullsized': '2024-01-15T18:10:28.840579+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.31 is not active+clean 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.842561+0000', 'last_change': '2024-01-15T18:10:28.842561+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.842561+0000', 'last_undegraded': '2024-01-15T18:10:28.842561+0000', 'last_fullsized': '2024-01-15T18:10:28.842561+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.30 is not active+clean 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.847589+0000', 'last_change': '2024-01-15T18:10:28.847589+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.847589+0000', 'last_undegraded': '2024-01-15T18:10:28.847589+0000', 'last_fullsized': '2024-01-15T18:10:28.847589+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2f is not active+clean 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.852113+0000', 'last_change': '2024-01-15T18:10:28.852113+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.852113+0000', 'last_undegraded': '2024-01-15T18:10:28.852113+0000', 'last_fullsized': '2024-01-15T18:10:28.852113+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2e is not active+clean 2024-01-15T18:10:35.365 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.843978+0000', 'last_change': '2024-01-15T18:10:28.843978+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.843978+0000', 'last_undegraded': '2024-01-15T18:10:28.843978+0000', 'last_fullsized': '2024-01-15T18:10:28.843978+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 6], 'acting': [2, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2d is not active+clean 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840150+0000', 'last_change': '2024-01-15T18:10:28.840150+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840150+0000', 'last_undegraded': '2024-01-15T18:10:28.840150+0000', 'last_fullsized': '2024-01-15T18:10:28.840150+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2c is not active+clean 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840702+0000', 'last_change': '2024-01-15T18:10:28.840702+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840702+0000', 'last_undegraded': '2024-01-15T18:10:28.840702+0000', 'last_fullsized': '2024-01-15T18:10:28.840702+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2b is not active+clean 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.843143+0000', 'last_change': '2024-01-15T18:10:28.843143+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.843143+0000', 'last_undegraded': '2024-01-15T18:10:28.843143+0000', 'last_fullsized': '2024-01-15T18:10:28.843143+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2a is not active+clean 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.845765+0000', 'last_change': '2024-01-15T18:10:28.845765+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.845765+0000', 'last_undegraded': '2024-01-15T18:10:28.845765+0000', 'last_fullsized': '2024-01-15T18:10:28.845765+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [1, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.29 is not active+clean 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.854373+0000', 'last_change': '2024-01-15T18:10:28.854373+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.854373+0000', 'last_undegraded': '2024-01-15T18:10:28.854373+0000', 'last_fullsized': '2024-01-15T18:10:28.854373+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.28 is not active+clean 2024-01-15T18:10:35.366 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.838275+0000', 'last_change': '2024-01-15T18:10:28.838275+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.838275+0000', 'last_undegraded': '2024-01-15T18:10:28.838275+0000', 'last_fullsized': '2024-01-15T18:10:28.838275+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.27 is not active+clean 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.838843+0000', 'last_change': '2024-01-15T18:10:28.838843+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.838843+0000', 'last_undegraded': '2024-01-15T18:10:28.838843+0000', 'last_fullsized': '2024-01-15T18:10:28.838843+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.25 is not active+clean 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.857241+0000', 'last_change': '2024-01-15T18:10:28.857241+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.857241+0000', 'last_undegraded': '2024-01-15T18:10:28.857241+0000', 'last_fullsized': '2024-01-15T18:10:28.857241+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.24 is not active+clean 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.847343+0000', 'last_change': '2024-01-15T18:10:28.847343+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.847343+0000', 'last_undegraded': '2024-01-15T18:10:28.847343+0000', 'last_fullsized': '2024-01-15T18:10:28.847343+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.23 is not active+clean 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840119+0000', 'last_change': '2024-01-15T18:10:28.840119+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840119+0000', 'last_undegraded': '2024-01-15T18:10:28.840119+0000', 'last_fullsized': '2024-01-15T18:10:28.840119+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.22 is not active+clean 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.845208+0000', 'last_change': '2024-01-15T18:10:28.845208+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.845208+0000', 'last_undegraded': '2024-01-15T18:10:28.845208+0000', 'last_fullsized': '2024-01-15T18:10:28.845208+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.367 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.21 is not active+clean 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.846185+0000', 'last_change': '2024-01-15T18:10:28.846185+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.846185+0000', 'last_undegraded': '2024-01-15T18:10:28.846185+0000', 'last_fullsized': '2024-01-15T18:10:28.846185+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.20 is not active+clean 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.846205+0000', 'last_change': '2024-01-15T18:10:28.846205+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.846205+0000', 'last_undegraded': '2024-01-15T18:10:28.846205+0000', 'last_fullsized': '2024-01-15T18:10:28.846205+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1e is not active+clean 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.856360+0000', 'last_change': '2024-01-15T18:10:28.856360+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.856360+0000', 'last_undegraded': '2024-01-15T18:10:28.856360+0000', 'last_fullsized': '2024-01-15T18:10:28.856360+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1f is not active+clean 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.852905+0000', 'last_change': '2024-01-15T18:10:28.852905+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.852905+0000', 'last_undegraded': '2024-01-15T18:10:28.852905+0000', 'last_fullsized': '2024-01-15T18:10:28.852905+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1c is not active+clean 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.846213+0000', 'last_change': '2024-01-15T18:10:28.846213+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.846213+0000', 'last_undegraded': '2024-01-15T18:10:28.846213+0000', 'last_fullsized': '2024-01-15T18:10:28.846213+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1a is not active+clean 2024-01-15T18:10:35.368 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.843192+0000', 'last_change': '2024-01-15T18:10:28.843192+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.843192+0000', 'last_undegraded': '2024-01-15T18:10:28.843192+0000', 'last_fullsized': '2024-01-15T18:10:28.843192+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1b is not active+clean 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.851504+0000', 'last_change': '2024-01-15T18:10:28.851504+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.851504+0000', 'last_undegraded': '2024-01-15T18:10:28.851504+0000', 'last_fullsized': '2024-01-15T18:10:28.851504+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.19 is not active+clean 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.842052+0000', 'last_change': '2024-01-15T18:10:28.842052+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.842052+0000', 'last_undegraded': '2024-01-15T18:10:28.842052+0000', 'last_fullsized': '2024-01-15T18:10:28.842052+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.16 is not active+clean 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.841682+0000', 'last_change': '2024-01-15T18:10:28.841682+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.841682+0000', 'last_undegraded': '2024-01-15T18:10:28.841682+0000', 'last_fullsized': '2024-01-15T18:10:28.841682+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.17 is not active+clean 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840553+0000', 'last_change': '2024-01-15T18:10:28.840553+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840553+0000', 'last_undegraded': '2024-01-15T18:10:28.840553+0000', 'last_fullsized': '2024-01-15T18:10:28.840553+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.14 is not active+clean 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840915+0000', 'last_change': '2024-01-15T18:10:28.840915+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840915+0000', 'last_undegraded': '2024-01-15T18:10:28.840915+0000', 'last_fullsized': '2024-01-15T18:10:28.840915+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.15 is not active+clean 2024-01-15T18:10:35.369 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.856623+0000', 'last_change': '2024-01-15T18:10:28.856623+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.856623+0000', 'last_undegraded': '2024-01-15T18:10:28.856623+0000', 'last_fullsized': '2024-01-15T18:10:28.856623+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.12 is not active+clean 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.839982+0000', 'last_change': '2024-01-15T18:10:28.839982+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.839982+0000', 'last_undegraded': '2024-01-15T18:10:28.839982+0000', 'last_fullsized': '2024-01-15T18:10:28.839982+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.10 is not active+clean 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.838341+0000', 'last_change': '2024-01-15T18:10:28.838341+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.838341+0000', 'last_undegraded': '2024-01-15T18:10:28.838341+0000', 'last_fullsized': '2024-01-15T18:10:28.838341+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.11 is not active+clean 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.839292+0000', 'last_change': '2024-01-15T18:10:28.839292+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.839292+0000', 'last_undegraded': '2024-01-15T18:10:28.839292+0000', 'last_fullsized': '2024-01-15T18:10:28.839292+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.f is not active+clean 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.851348+0000', 'last_change': '2024-01-15T18:10:28.851348+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.851348+0000', 'last_undegraded': '2024-01-15T18:10:28.851348+0000', 'last_fullsized': '2024-01-15T18:10:28.851348+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.d is not active+clean 2024-01-15T18:10:35.370 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.856094+0000', 'last_change': '2024-01-15T18:10:28.856094+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.856094+0000', 'last_undegraded': '2024-01-15T18:10:28.856094+0000', 'last_fullsized': '2024-01-15T18:10:28.856094+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.a is not active+clean 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.837448+0000', 'last_change': '2024-01-15T18:10:28.837448+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.837448+0000', 'last_undegraded': '2024-01-15T18:10:28.837448+0000', 'last_fullsized': '2024-01-15T18:10:28.837448+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.b is not active+clean 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.839003+0000', 'last_change': '2024-01-15T18:10:28.839003+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.839003+0000', 'last_undegraded': '2024-01-15T18:10:28.839003+0000', 'last_fullsized': '2024-01-15T18:10:28.839003+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.8 is not active+clean 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.837767+0000', 'last_change': '2024-01-15T18:10:28.837767+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.837767+0000', 'last_undegraded': '2024-01-15T18:10:28.837767+0000', 'last_fullsized': '2024-01-15T18:10:28.837767+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.9 is not active+clean 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.840868+0000', 'last_change': '2024-01-15T18:10:28.840868+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.840868+0000', 'last_undegraded': '2024-01-15T18:10:28.840868+0000', 'last_fullsized': '2024-01-15T18:10:28.840868+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.6 is not active+clean 2024-01-15T18:10:35.371 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.854397+0000', 'last_change': '2024-01-15T18:10:28.854397+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.854397+0000', 'last_undegraded': '2024-01-15T18:10:28.854397+0000', 'last_fullsized': '2024-01-15T18:10:28.854397+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.4 is not active+clean 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.844184+0000', 'last_change': '2024-01-15T18:10:28.844184+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.844184+0000', 'last_undegraded': '2024-01-15T18:10:28.844184+0000', 'last_fullsized': '2024-01-15T18:10:28.844184+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.5 is not active+clean 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.855546+0000', 'last_change': '2024-01-15T18:10:28.855546+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.855546+0000', 'last_undegraded': '2024-01-15T18:10:28.855546+0000', 'last_fullsized': '2024-01-15T18:10:28.855546+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.2 is not active+clean 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.841823+0000', 'last_change': '2024-01-15T18:10:28.841823+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.841823+0000', 'last_undegraded': '2024-01-15T18:10:28.841823+0000', 'last_fullsized': '2024-01-15T18:10:28.841823+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.3 is not active+clean 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.839292+0000', 'last_change': '2024-01-15T18:10:28.839292+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.839292+0000', 'last_undegraded': '2024-01-15T18:10:28.839292+0000', 'last_fullsized': '2024-01-15T18:10:28.839292+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 4], 'acting': [0, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.1 is not active+clean 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.853917+0000', 'last_change': '2024-01-15T18:10:28.853917+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.853917+0000', 'last_undegraded': '2024-01-15T18:10:28.853917+0000', 'last_fullsized': '2024-01-15T18:10:28.853917+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-15T18:10:35.372 INFO:tasks.cephfs.filesystem.ceph_manager:PG 13.c is not active+clean 2024-01-15T18:10:35.373 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '13.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 87, 'state': 'creating+peering', 'last_fresh': '2024-01-15T18:10:28.837201+0000', 'last_change': '2024-01-15T18:10:28.837201+0000', 'last_active': '2024-01-15T18:10:28.822137+0000', 'last_peered': '2024-01-15T18:10:28.822137+0000', 'last_clean': '2024-01-15T18:10:28.822137+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-15T18:10:28.837201+0000', 'last_undegraded': '2024-01-15T18:10:28.837201+0000', 'last_fullsized': '2024-01-15T18:10:28.837201+0000', 'mapping_epoch': 87, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 87, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'last_clean_scrub_stamp': '2024-01-15T18:10:28.822137+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-01-15T18:10:35.374 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:10:35.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.621+0000 7f7d97e00640 1 Processor -- start 2024-01-15T18:10:35.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.621+0000 7f7d97e00640 1 -- start start 2024-01-15T18:10:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.623+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d98070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.623+0000 7f7d97e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9806dae0 con 0x7f7d9815ec60 2024-01-15T18:10:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.623+0000 7f7d97e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9815f570 con 0x7f7d9815f0e0 2024-01-15T18:10:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.623+0000 7f7d97e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98071f20 con 0x7f7d98141880 2024-01-15T18:10:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.623+0000 7f7d8fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7d9815ec60 0x7f7d980cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44316/0 (socket says 172.21.15.2:44316) 2024-01-15T18:10:35.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.623+0000 7f7d8fe00640 1 -- 172.21.15.2:0/806713113 learned_addr learned my addr 172.21.15.2:0/806713113 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:35.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96a00640 1 --2- 172.21.15.2:0/806713113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d98070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635924776 0 0) 0x7f7d9806dae0 con 0x7f7d9815ec60 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d78003520 con 0x7f7d9815ec60 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018326301 0 0) 0x7f7d9815f570 con 0x7f7d9815f0e0 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d9806dae0 con 0x7f7d9815f0e0 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96a00640 1 -- 172.21.15.2:0/806713113 >> v1:172.21.15.137:6789/0 conn(0x7f7d9815f0e0 legacy=0x7f7d980cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 324592239 0 0) 0x7f7d78003520 con 0x7f7d9815ec60 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96a00640 1 -- 172.21.15.2:0/806713113 >> v1:172.21.15.2:6789/0 conn(0x7f7d9815ec60 legacy=0x7f7d980cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.624+0000 7f7d96a00640 1 -- 172.21.15.2:0/806713113 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98071960 con 0x7f7d98141880 2024-01-15T18:10:35.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.627+0000 7f7d96a00640 1 --2- 172.21.15.2:0/806713113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d98070fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7d880097d0 tx=0x7f7d8802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=643ff549b505401 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.627+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d80001670 con 0x7f7d98141880 2024-01-15T18:10:35.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.627+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8802e050 con 0x7f7d98141880 2024-01-15T18:10:35.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.627+0000 7f7d96000640 1 -- 172.21.15.2:0/806713113 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88030730 con 0x7f7d98141880 2024-01-15T18:10:35.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 -- 172.21.15.2:0/806713113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 msgr2=0x7f7d98070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 --2- 172.21.15.2:0/806713113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d98070fc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7d880097d0 tx=0x7f7d8802b770 comp rx=0 tx=0).stop 2024-01-15T18:10:35.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 -- 172.21.15.2:0/806713113 shutdown_connections 2024-01-15T18:10:35.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 --2- 172.21.15.2:0/806713113 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d98070fc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 -- 172.21.15.2:0/806713113 >> 172.21.15.2:0/806713113 conn(0x7f7d980b0840 msgr2=0x7f7d980b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:35.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 -- 172.21.15.2:0/806713113 shutdown_connections 2024-01-15T18:10:35.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.628+0000 7f7d97e00640 1 -- 172.21.15.2:0/806713113 wait complete. 2024-01-15T18:10:35.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.630+0000 7f7d97e00640 1 Processor -- start 2024-01-15T18:10:35.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.630+0000 7f7d97e00640 1 -- start start 2024-01-15T18:10:35.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d980ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d9815ec60 0x7f7d9815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 0x7f7d981433c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d9806c6b0 con 0x7f7d9815f0e0 2024-01-15T18:10:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d981597e0 con 0x7f7d9815ec60 2024-01-15T18:10:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98157b60 con 0x7f7d98141880 2024-01-15T18:10:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 0x7f7d981433c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 0x7f7d981433c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55740/0 (socket says 172.21.15.2:55740) 2024-01-15T18:10:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.631+0000 7f7d97400640 1 -- 172.21.15.2:0/4105382987 learned_addr learned my addr 172.21.15.2:0/4105382987 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d9815ec60 0x7f7d9815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d96a00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d980ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d97400640 1 -- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 msgr2=0x7f7d980ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d97400640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d980ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d97400640 1 -- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d9815ec60 msgr2=0x7f7d9815a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d97400640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d9815ec60 0x7f7d9815a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d97400640 1 -- 172.21.15.2:0/4105382987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d88009400 con 0x7f7d9815f0e0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d9815ec60 0x7f7d9815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.632+0000 7f7d97400640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 0x7f7d981433c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000c790 tx=0x7f7d9000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.633+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000cea0 con 0x7f7d9815f0e0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.633+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90003b70 con 0x7f7d9815f0e0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.633+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9000a850 con 0x7f7d9815f0e0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.633+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d9815a730 con 0x7f7d9815f0e0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.633+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d98157880 con 0x7f7d9815f0e0 2024-01-15T18:10:35.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.634+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d98066e50 con 0x7f7d9815f0e0 2024-01-15T18:10:35.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.635+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7d9000a9b0 con 0x7f7d9815f0e0 2024-01-15T18:10:35.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.636+0000 7f7d8f400640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d7006c030 0x7f7d7006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:35.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.636+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f7d90088480 con 0x7f7d9815f0e0 2024-01-15T18:10:35.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.637+0000 7f7d96a00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d7006c030 0x7f7d7006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:35.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.638+0000 7f7d96a00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d7006c030 0x7f7d7006e4f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7d88034000 tx=0x7f7d88008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:35.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.642+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d90054d40 con 0x7f7d9815f0e0 2024-01-15T18:10:35.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.908+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7d98074770 con 0x7f7d9815f0e0 2024-01-15T18:10:35.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.909+0000 7f7d8f400640 1 -- 172.21.15.2:0/4105382987 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1034 (secure 0 0 0) 0x7f7d90058360 con 0x7f7d9815f0e0 2024-01-15T18:10:35.912 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:35.912 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":803,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":89,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341443,"num_in_osds":8,"osd_in_since":1705341435,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":55},{"state_name":"active+clean","count":42}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":534904832,"bytes_avail":772559208448,"bytes_total":773094113280,"inactive_pgs_ratio":0.56701028347015381},"fsmap":{"epoch":56,"id":6,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":6,"rank":0,"name":"c","status":"up:active","gid":8055}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-01-15T18:06:49.623999+0000","services":{}},"progress_events":{}} 2024-01-15T18:10:35.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d7006c030 msgr2=0x7f7d7006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d7006c030 0x7f7d7006e4f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7d88034000 tx=0x7f7d88008030 comp rx=0 tx=0).stop 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 msgr2=0x7f7d981433c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 0x7f7d981433c0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f7d9000c790 tx=0x7f7d9000cc50 comp rx=0 tx=0).stop 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 shutdown_connections 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d7006c030 0x7f7d7006e4f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9815f0e0 0x7f7d981433c0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d9815ec60 0x7f7d9815a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 --2- 172.21.15.2:0/4105382987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d98141880 0x7f7d980ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 >> 172.21.15.2:0/4105382987 conn(0x7f7d980b0840 msgr2=0x7f7d9815d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 shutdown_connections 2024-01-15T18:10:35.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:35.912+0000 7f7d97e00640 1 -- 172.21.15.2:0/4105382987 wait complete. 2024-01-15T18:10:35.933 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:10:35.933 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:10:36.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.147+0000 7f6057200640 1 Processor -- start 2024-01-15T18:10:36.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.148+0000 7f6057200640 1 -- start start 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.149+0000 7f6057200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f60500d37f0 0x7f60500c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.149+0000 7f6057200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60500c6e40 con 0x7f60500cfad0 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.149+0000 7f6057200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60500d2090 con 0x7f60500d3ca0 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.149+0000 7f6057200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60500d17f0 con 0x7f60500d37f0 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.150+0000 7f6055e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f60500cfad0 0x7f60500cfeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44330/0 (socket says 172.21.15.2:44330) 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.150+0000 7f6055e00640 1 -- 172.21.15.2:0/724242485 learned_addr learned my addr 172.21.15.2:0/724242485 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.150+0000 7f6055400640 1 --2- 172.21.15.2:0/724242485 >> v2:172.21.15.137:3300/0 conn(0x7f60500d37f0 0x7f60500c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:36.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.150+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257742065 0 0) 0x7f60500c6e40 con 0x7f60500cfad0 2024-01-15T18:10:36.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.150+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6030003520 con 0x7f60500cfad0 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3472801779 0 0) 0x7f6030003520 con 0x7f60500cfad0 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 >> v2:172.21.15.137:3300/0 conn(0x7f60500d37f0 msgr2=0x7f60500c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 --2- 172.21.15.2:0/724242485 >> v2:172.21.15.137:3300/0 conn(0x7f60500d37f0 0x7f60500c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 >> v1:172.21.15.137:6789/0 conn(0x7f60500d3ca0 legacy=0x7f60500c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60500d1ad0 con 0x7f60500cfad0 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6040001670 con 0x7f60500cfad0 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 >> v1:172.21.15.2:6789/0 conn(0x7f60500cfad0 legacy=0x7f60500cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60300039c0 0x7f6030003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60300047c0 0x7f6030004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6030005180 0x7f603000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60500d17f0 con 0x7f60300039c0 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c080 con 0x7f60300047c0 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603000c240 con 0x7f6030005180 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.151+0000 7f6055e00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60300039c0 0x7f6030003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.152+0000 7f6055e00640 1 -- 172.21.15.2:0/724242485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6030005180 msgr2=0x7f603000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.152+0000 7f6055e00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6030005180 0x7f603000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.152+0000 7f6055e00640 1 -- 172.21.15.2:0/724242485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60300047c0 msgr2=0x7f6030004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.152+0000 7f6055e00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f60300047c0 0x7f6030004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.152+0000 7f6055e00640 1 -- 172.21.15.2:0/724242485 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6030003520 con 0x7f60300039c0 2024-01-15T18:10:36.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.152+0000 7f6056800640 1 -- 172.21.15.2:0/724242485 reap_dead start 2024-01-15T18:10:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.153+0000 7f6055e00640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60300039c0 0x7f6030003de0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f604000c0e0 tx=0x7f60300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5665b4ed04fcf2ba server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.153+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040005ad0 con 0x7f60300039c0 2024-01-15T18:10:36.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.153+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040005c50 con 0x7f60300039c0 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.153+0000 7f6054a00640 1 -- 172.21.15.2:0/724242485 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604002c810 con 0x7f60300039c0 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 -- 172.21.15.2:0/724242485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60300039c0 msgr2=0x7f6030003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60300039c0 0x7f6030003de0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f604000c0e0 tx=0x7f60300057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 -- 172.21.15.2:0/724242485 shutdown_connections 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 --2- 172.21.15.2:0/724242485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60300039c0 0x7f6030003de0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 -- 172.21.15.2:0/724242485 >> 172.21.15.2:0/724242485 conn(0x7f605009c3b0 msgr2=0x7f60500d30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 -- 172.21.15.2:0/724242485 shutdown_connections 2024-01-15T18:10:36.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.154+0000 7f6057200640 1 -- 172.21.15.2:0/724242485 wait complete. 2024-01-15T18:10:36.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.156+0000 7f6057200640 1 Processor -- start 2024-01-15T18:10:36.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.156+0000 7f6057200640 1 -- start start 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6057200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f60300039c0 0x7f60500b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6057200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605006c5b0 0x7f60500cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6057200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 0x7f605015b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6057200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605015dcc0 con 0x7f60500ccd70 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6057200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605015dfa0 con 0x7f605006c5b0 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6057200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605015e280 con 0x7f60300039c0 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.157+0000 7f6056800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 0x7f605015b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 0x7f605015b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55758/0 (socket says 172.21.15.2:55758) 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 -- 172.21.15.2:0/1390027538 learned_addr learned my addr 172.21.15.2:0/1390027538 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6055e00640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f60300039c0 0x7f60500b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6055400640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605006c5b0 0x7f60500cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 -- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f60300039c0 msgr2=0x7f60500b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f60300039c0 0x7f60500b0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 -- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605006c5b0 msgr2=0x7f60500cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605006c5b0 0x7f60500cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 -- 172.21.15.2:0/1390027538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605015e560 con 0x7f60500ccd70 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.158+0000 7f6056800640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 0x7f605015b260 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f60500d4080 tx=0x7f604800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.159+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6048005d40 con 0x7f60500ccd70 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.159+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604800ced0 con 0x7f60500ccd70 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.159+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604800a850 con 0x7f60500ccd70 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.159+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60500b10f0 con 0x7f60500ccd70 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.159+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605006ca30 con 0x7f60500ccd70 2024-01-15T18:10:36.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.160+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6050074770 con 0x7f60500ccd70 2024-01-15T18:10:36.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.161+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f604800a9b0 con 0x7f60500ccd70 2024-01-15T18:10:36.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.162+0000 7f604e000640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f602806bfb0 0x7f602806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:36.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.162+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f60480873e0 con 0x7f60500ccd70 2024-01-15T18:10:36.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.163+0000 7f6055e00640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f602806bfb0 0x7f602806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:36.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.164+0000 7f6055e00640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f602806bfb0 0x7f602806e470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6040034000 tx=0x7f6040009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:36.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.166+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6048053d00 con 0x7f60500ccd70 2024-01-15T18:10:36.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.357+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f60500ceeb0 con 0x7f602806bfb0 2024-01-15T18:10:36.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.380+0000 7f604e000640 1 -- 172.21.15.2:0/1390027538 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286830 (secure 0 0 0) 0x7f604003d040 con 0x7f602806bfb0 2024-01-15T18:10:36.387 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:36.388 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f602806bfb0 msgr2=0x7f602806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f602806bfb0 0x7f602806e470 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f6040034000 tx=0x7f6040009030 comp rx=0 tx=0).stop 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 msgr2=0x7f605015b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 0x7f605015b260 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f60500d4080 tx=0x7f604800cb90 comp rx=0 tx=0).stop 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 shutdown_connections 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f602806bfb0 0x7f602806e470 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f60500ccd70 0x7f605015b260 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f605006c5b0 0x7f60500cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 --2- 172.21.15.2:0/1390027538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f60300039c0 0x7f60500b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:36.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.385+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 >> 172.21.15.2:0/1390027538 conn(0x7f605009c3b0 msgr2=0x7f60500b7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:36.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.386+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 shutdown_connections 2024-01-15T18:10:36.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:36.386+0000 7f6057200640 1 -- 172.21.15.2:0/1390027538 wait complete. 2024-01-15T18:10:36.408 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-01-15T18:10:35.713992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":522620,"kb_used_data":9736,"kb_used_omap":12,"kb_used_meta":512755,"kb_avail":754452100,"statfs":{"total":773094113280,"available":772558950400,"internally_reserved":0,"allocated":9969664,"data_stored":4960740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":525061681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.881406"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086275+0000","last_change":"2024-01-15T18:10:29.889349+0000","last_active":"2024-01-15T18:10:31.086275+0000","last_peered":"2024-01-15T18:10:31.086275+0000","last_clean":"2024-01-15T18:10:31.086275+0000","last_became_active":"2024-01-15T18:10:29.889079+0000","last_became_peered":"2024-01-15T18:10:29.889079+0000","last_unstale":"2024-01-15T18:10:31.086275+0000","last_undegraded":"2024-01-15T18:10:31.086275+0000","last_fullsized":"2024-01-15T18:10:31.086275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:32.626031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087942+0000","last_change":"2024-01-15T18:10:29.887330+0000","last_active":"2024-01-15T18:10:31.087942+0000","last_peered":"2024-01-15T18:10:31.087942+0000","last_clean":"2024-01-15T18:10:31.087942+0000","last_became_active":"2024-01-15T18:10:29.887024+0000","last_became_peered":"2024-01-15T18:10:29.887024+0000","last_unstale":"2024-01-15T18:10:31.087942+0000","last_undegraded":"2024-01-15T18:10:31.087942+0000","last_fullsized":"2024-01-15T18:10:31.087942+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:52.419431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085792+0000","last_change":"2024-01-15T18:10:29.888180+0000","last_active":"2024-01-15T18:10:31.085792+0000","last_peered":"2024-01-15T18:10:31.085792+0000","last_clean":"2024-01-15T18:10:31.085792+0000","last_became_active":"2024-01-15T18:10:29.886925+0000","last_became_peered":"2024-01-15T18:10:29.886925+0000","last_unstale":"2024-01-15T18:10:31.085792+0000","last_undegraded":"2024-01-15T18:10:31.085792+0000","last_fullsized":"2024-01-15T18:10:31.085792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:10.571774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884428+0000","last_change":"2024-01-15T18:10:29.883489+0000","last_active":"2024-01-15T18:10:30.884428+0000","last_peered":"2024-01-15T18:10:30.884428+0000","last_clean":"2024-01-15T18:10:30.884428+0000","last_became_active":"2024-01-15T18:10:29.883195+0000","last_became_peered":"2024-01-15T18:10:29.883195+0000","last_unstale":"2024-01-15T18:10:30.884428+0000","last_undegraded":"2024-01-15T18:10:30.884428+0000","last_fullsized":"2024-01-15T18:10:30.884428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:24:48.158391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884370+0000","last_change":"2024-01-15T18:10:29.892558+0000","last_active":"2024-01-15T18:10:30.884370+0000","last_peered":"2024-01-15T18:10:30.884370+0000","last_clean":"2024-01-15T18:10:30.884370+0000","last_became_active":"2024-01-15T18:10:29.891669+0000","last_became_peered":"2024-01-15T18:10:29.891669+0000","last_unstale":"2024-01-15T18:10:30.884370+0000","last_undegraded":"2024-01-15T18:10:30.884370+0000","last_fullsized":"2024-01-15T18:10:30.884370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:59:55.883984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886845+0000","last_change":"2024-01-15T18:10:29.882639+0000","last_active":"2024-01-15T18:10:30.886845+0000","last_peered":"2024-01-15T18:10:30.886845+0000","last_clean":"2024-01-15T18:10:30.886845+0000","last_became_active":"2024-01-15T18:10:29.882359+0000","last_became_peered":"2024-01-15T18:10:29.882359+0000","last_unstale":"2024-01-15T18:10:30.886845+0000","last_undegraded":"2024-01-15T18:10:30.886845+0000","last_fullsized":"2024-01-15T18:10:30.886845+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:25.828362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883604+0000","last_change":"2024-01-15T18:10:29.887844+0000","last_active":"2024-01-15T18:10:30.883604+0000","last_peered":"2024-01-15T18:10:30.883604+0000","last_clean":"2024-01-15T18:10:30.883604+0000","last_became_active":"2024-01-15T18:10:29.887501+0000","last_became_peered":"2024-01-15T18:10:29.887501+0000","last_unstale":"2024-01-15T18:10:30.883604+0000","last_undegraded":"2024-01-15T18:10:30.883604+0000","last_fullsized":"2024-01-15T18:10:30.883604+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:03:05.714901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886761+0000","last_change":"2024-01-15T18:10:29.876459+0000","last_active":"2024-01-15T18:10:30.886761+0000","last_peered":"2024-01-15T18:10:30.886761+0000","last_clean":"2024-01-15T18:10:30.886761+0000","last_became_active":"2024-01-15T18:10:29.876059+0000","last_became_peered":"2024-01-15T18:10:29.876059+0000","last_unstale":"2024-01-15T18:10:30.886761+0000","last_undegraded":"2024-01-15T18:10:30.886761+0000","last_fullsized":"2024-01-15T18:10:30.886761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:48.870265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887915+0000","last_change":"2024-01-15T18:10:29.885819+0000","last_active":"2024-01-15T18:10:30.887915+0000","last_peered":"2024-01-15T18:10:30.887915+0000","last_clean":"2024-01-15T18:10:30.887915+0000","last_became_active":"2024-01-15T18:10:29.884423+0000","last_became_peered":"2024-01-15T18:10:29.884423+0000","last_unstale":"2024-01-15T18:10:30.887915+0000","last_undegraded":"2024-01-15T18:10:30.887915+0000","last_fullsized":"2024-01-15T18:10:30.887915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085443+0000","last_change":"2024-01-15T18:10:29.886368+0000","last_active":"2024-01-15T18:10:31.085443+0000","last_peered":"2024-01-15T18:10:31.085443+0000","last_clean":"2024-01-15T18:10:31.085443+0000","last_became_active":"2024-01-15T18:10:29.885765+0000","last_became_peered":"2024-01-15T18:10:29.885765+0000","last_unstale":"2024-01-15T18:10:31.085443+0000","last_undegraded":"2024-01-15T18:10:31.085443+0000","last_fullsized":"2024-01-15T18:10:31.085443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:09.028738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087754+0000","last_change":"2024-01-15T18:10:29.879828+0000","last_active":"2024-01-15T18:10:31.087754+0000","last_peered":"2024-01-15T18:10:31.087754+0000","last_clean":"2024-01-15T18:10:31.087754+0000","last_became_active":"2024-01-15T18:10:29.879547+0000","last_became_peered":"2024-01-15T18:10:29.879547+0000","last_unstale":"2024-01-15T18:10:31.087754+0000","last_undegraded":"2024-01-15T18:10:31.087754+0000","last_fullsized":"2024-01-15T18:10:31.087754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:21.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084274+0000","last_change":"2024-01-15T18:10:29.887061+0000","last_active":"2024-01-15T18:10:31.084274+0000","last_peered":"2024-01-15T18:10:31.084274+0000","last_clean":"2024-01-15T18:10:31.084274+0000","last_became_active":"2024-01-15T18:10:29.885624+0000","last_became_peered":"2024-01-15T18:10:29.885624+0000","last_unstale":"2024-01-15T18:10:31.084274+0000","last_undegraded":"2024-01-15T18:10:31.084274+0000","last_fullsized":"2024-01-15T18:10:31.084274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084754+0000","last_change":"2024-01-15T18:10:29.884896+0000","last_active":"2024-01-15T18:10:31.084754+0000","last_peered":"2024-01-15T18:10:31.084754+0000","last_clean":"2024-01-15T18:10:31.084754+0000","last_became_active":"2024-01-15T18:10:29.883901+0000","last_became_peered":"2024-01-15T18:10:29.883901+0000","last_unstale":"2024-01-15T18:10:31.084754+0000","last_undegraded":"2024-01-15T18:10:31.084754+0000","last_fullsized":"2024-01-15T18:10:31.084754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:32.626031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883780+0000","last_change":"2024-01-15T18:10:29.884516+0000","last_active":"2024-01-15T18:10:30.883780+0000","last_peered":"2024-01-15T18:10:30.883780+0000","last_clean":"2024-01-15T18:10:30.883780+0000","last_became_active":"2024-01-15T18:10:29.883939+0000","last_became_peered":"2024-01-15T18:10:29.883939+0000","last_unstale":"2024-01-15T18:10:30.883780+0000","last_undegraded":"2024-01-15T18:10:30.883780+0000","last_fullsized":"2024-01-15T18:10:30.883780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:30:18.904062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.083431+0000","last_change":"2024-01-15T18:10:29.871576+0000","last_active":"2024-01-15T18:10:31.083431+0000","last_peered":"2024-01-15T18:10:31.083431+0000","last_clean":"2024-01-15T18:10:31.083431+0000","last_became_active":"2024-01-15T18:10:29.870742+0000","last_became_peered":"2024-01-15T18:10:29.870742+0000","last_unstale":"2024-01-15T18:10:31.083431+0000","last_undegraded":"2024-01-15T18:10:31.083431+0000","last_fullsized":"2024-01-15T18:10:31.083431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:10.571774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883712+0000","last_change":"2024-01-15T18:10:29.890607+0000","last_active":"2024-01-15T18:10:30.883712+0000","last_peered":"2024-01-15T18:10:30.883712+0000","last_clean":"2024-01-15T18:10:30.883712+0000","last_became_active":"2024-01-15T18:10:29.889442+0000","last_became_peered":"2024-01-15T18:10:29.889442+0000","last_unstale":"2024-01-15T18:10:30.883712+0000","last_undegraded":"2024-01-15T18:10:30.883712+0000","last_fullsized":"2024-01-15T18:10:30.883712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:59.655054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885709+0000","last_change":"2024-01-15T18:10:29.893877+0000","last_active":"2024-01-15T18:10:30.885709+0000","last_peered":"2024-01-15T18:10:30.885709+0000","last_clean":"2024-01-15T18:10:30.885709+0000","last_became_active":"2024-01-15T18:10:29.893574+0000","last_became_peered":"2024-01-15T18:10:29.893574+0000","last_unstale":"2024-01-15T18:10:30.885709+0000","last_undegraded":"2024-01-15T18:10:30.885709+0000","last_fullsized":"2024-01-15T18:10:30.885709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:49.239687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884812+0000","last_change":"2024-01-15T18:10:29.888339+0000","last_active":"2024-01-15T18:10:30.884812+0000","last_peered":"2024-01-15T18:10:30.884812+0000","last_clean":"2024-01-15T18:10:30.884812+0000","last_became_active":"2024-01-15T18:10:29.888053+0000","last_became_peered":"2024-01-15T18:10:29.888053+0000","last_unstale":"2024-01-15T18:10:30.884812+0000","last_undegraded":"2024-01-15T18:10:30.884812+0000","last_fullsized":"2024-01-15T18:10:30.884812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:21:03.349644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085910+0000","last_change":"2024-01-15T18:10:29.890238+0000","last_active":"2024-01-15T18:10:31.085910+0000","last_peered":"2024-01-15T18:10:31.085910+0000","last_clean":"2024-01-15T18:10:31.085910+0000","last_became_active":"2024-01-15T18:10:29.888327+0000","last_became_peered":"2024-01-15T18:10:29.888327+0000","last_unstale":"2024-01-15T18:10:31.085910+0000","last_undegraded":"2024-01-15T18:10:31.085910+0000","last_fullsized":"2024-01-15T18:10:31.085910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:32.626031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884433+0000","last_change":"2024-01-15T18:10:29.880093+0000","last_active":"2024-01-15T18:10:30.884433+0000","last_peered":"2024-01-15T18:10:30.884433+0000","last_clean":"2024-01-15T18:10:30.884433+0000","last_became_active":"2024-01-15T18:10:29.879766+0000","last_became_peered":"2024-01-15T18:10:29.879766+0000","last_unstale":"2024-01-15T18:10:30.884433+0000","last_undegraded":"2024-01-15T18:10:30.884433+0000","last_fullsized":"2024-01-15T18:10:30.884433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:46:25.456136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884248+0000","last_change":"2024-01-15T18:10:29.878993+0000","last_active":"2024-01-15T18:10:30.884248+0000","last_peered":"2024-01-15T18:10:30.884248+0000","last_clean":"2024-01-15T18:10:30.884248+0000","last_became_active":"2024-01-15T18:10:29.878223+0000","last_became_peered":"2024-01-15T18:10:29.878223+0000","last_unstale":"2024-01-15T18:10:30.884248+0000","last_undegraded":"2024-01-15T18:10:30.884248+0000","last_fullsized":"2024-01-15T18:10:30.884248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:34.700358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086577+0000","last_change":"2024-01-15T18:10:29.878232+0000","last_active":"2024-01-15T18:10:31.086577+0000","last_peered":"2024-01-15T18:10:31.086577+0000","last_clean":"2024-01-15T18:10:31.086577+0000","last_became_active":"2024-01-15T18:10:29.877216+0000","last_became_peered":"2024-01-15T18:10:29.877216+0000","last_unstale":"2024-01-15T18:10:31.086577+0000","last_undegraded":"2024-01-15T18:10:31.086577+0000","last_fullsized":"2024-01-15T18:10:31.086577+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:19:33.223505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084033+0000","last_change":"2024-01-15T18:10:29.894531+0000","last_active":"2024-01-15T18:10:31.084033+0000","last_peered":"2024-01-15T18:10:31.084033+0000","last_clean":"2024-01-15T18:10:31.084033+0000","last_became_active":"2024-01-15T18:10:29.894255+0000","last_became_peered":"2024-01-15T18:10:29.894255+0000","last_unstale":"2024-01-15T18:10:31.084033+0000","last_undegraded":"2024-01-15T18:10:31.084033+0000","last_fullsized":"2024-01-15T18:10:31.084033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883469+0000","last_change":"2024-01-15T18:10:29.875269+0000","last_active":"2024-01-15T18:10:30.883469+0000","last_peered":"2024-01-15T18:10:30.883469+0000","last_clean":"2024-01-15T18:10:30.883469+0000","last_became_active":"2024-01-15T18:10:29.874953+0000","last_became_peered":"2024-01-15T18:10:29.874953+0000","last_unstale":"2024-01-15T18:10:30.883469+0000","last_undegraded":"2024-01-15T18:10:30.883469+0000","last_fullsized":"2024-01-15T18:10:30.883469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:45.345681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087649+0000","last_change":"2024-01-15T18:10:29.887757+0000","last_active":"2024-01-15T18:10:31.087649+0000","last_peered":"2024-01-15T18:10:31.087649+0000","last_clean":"2024-01-15T18:10:31.087649+0000","last_became_active":"2024-01-15T18:10:29.887342+0000","last_became_peered":"2024-01-15T18:10:29.887342+0000","last_unstale":"2024-01-15T18:10:31.087649+0000","last_undegraded":"2024-01-15T18:10:31.087649+0000","last_fullsized":"2024-01-15T18:10:31.087649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:07:05.708131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085556+0000","last_change":"2024-01-15T18:10:29.879335+0000","last_active":"2024-01-15T18:10:31.085556+0000","last_peered":"2024-01-15T18:10:31.085556+0000","last_clean":"2024-01-15T18:10:31.085556+0000","last_became_active":"2024-01-15T18:10:29.878999+0000","last_became_peered":"2024-01-15T18:10:29.878999+0000","last_unstale":"2024-01-15T18:10:31.085556+0000","last_undegraded":"2024-01-15T18:10:31.085556+0000","last_fullsized":"2024-01-15T18:10:31.085556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:12:20.074528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884895+0000","last_change":"2024-01-15T18:10:29.885016+0000","last_active":"2024-01-15T18:10:30.884895+0000","last_peered":"2024-01-15T18:10:30.884895+0000","last_clean":"2024-01-15T18:10:30.884895+0000","last_became_active":"2024-01-15T18:10:29.884752+0000","last_became_peered":"2024-01-15T18:10:29.884752+0000","last_unstale":"2024-01-15T18:10:30.884895+0000","last_undegraded":"2024-01-15T18:10:30.884895+0000","last_fullsized":"2024-01-15T18:10:30.884895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:27:52.380249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084315+0000","last_change":"2024-01-15T18:10:29.893008+0000","last_active":"2024-01-15T18:10:31.084315+0000","last_peered":"2024-01-15T18:10:31.084315+0000","last_clean":"2024-01-15T18:10:31.084315+0000","last_became_active":"2024-01-15T18:10:29.892710+0000","last_became_peered":"2024-01-15T18:10:29.892710+0000","last_unstale":"2024-01-15T18:10:31.084315+0000","last_undegraded":"2024-01-15T18:10:31.084315+0000","last_fullsized":"2024-01-15T18:10:31.084315+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:10.131197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086963+0000","last_change":"2024-01-15T18:10:29.881928+0000","last_active":"2024-01-15T18:10:31.086963+0000","last_peered":"2024-01-15T18:10:31.086963+0000","last_clean":"2024-01-15T18:10:31.086963+0000","last_became_active":"2024-01-15T18:10:29.879237+0000","last_became_peered":"2024-01-15T18:10:29.879237+0000","last_unstale":"2024-01-15T18:10:31.086963+0000","last_undegraded":"2024-01-15T18:10:31.086963+0000","last_fullsized":"2024-01-15T18:10:31.086963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:37.601472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086452+0000","last_change":"2024-01-15T18:10:29.878883+0000","last_active":"2024-01-15T18:10:31.086452+0000","last_peered":"2024-01-15T18:10:31.086452+0000","last_clean":"2024-01-15T18:10:31.086452+0000","last_became_active":"2024-01-15T18:10:29.878568+0000","last_became_peered":"2024-01-15T18:10:29.878568+0000","last_unstale":"2024-01-15T18:10:31.086452+0000","last_undegraded":"2024-01-15T18:10:31.086452+0000","last_fullsized":"2024-01-15T18:10:31.086452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:48.891903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884305+0000","last_change":"2024-01-15T18:10:29.890635+0000","last_active":"2024-01-15T18:10:30.884305+0000","last_peered":"2024-01-15T18:10:30.884305+0000","last_clean":"2024-01-15T18:10:30.884305+0000","last_became_active":"2024-01-15T18:10:29.888122+0000","last_became_peered":"2024-01-15T18:10:29.888122+0000","last_unstale":"2024-01-15T18:10:30.884305+0000","last_undegraded":"2024-01-15T18:10:30.884305+0000","last_fullsized":"2024-01-15T18:10:30.884305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:10.984586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084512+0000","last_change":"2024-01-15T18:10:29.891784+0000","last_active":"2024-01-15T18:10:31.084512+0000","last_peered":"2024-01-15T18:10:31.084512+0000","last_clean":"2024-01-15T18:10:31.084512+0000","last_became_active":"2024-01-15T18:10:29.891459+0000","last_became_peered":"2024-01-15T18:10:29.891459+0000","last_unstale":"2024-01-15T18:10:31.084512+0000","last_undegraded":"2024-01-15T18:10:31.084512+0000","last_fullsized":"2024-01-15T18:10:31.084512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:18.035419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885422+0000","last_change":"2024-01-15T18:10:29.888437+0000","last_active":"2024-01-15T18:10:30.885422+0000","last_peered":"2024-01-15T18:10:30.885422+0000","last_clean":"2024-01-15T18:10:30.885422+0000","last_became_active":"2024-01-15T18:10:29.888167+0000","last_became_peered":"2024-01-15T18:10:29.888167+0000","last_unstale":"2024-01-15T18:10:30.885422+0000","last_undegraded":"2024-01-15T18:10:30.885422+0000","last_fullsized":"2024-01-15T18:10:30.885422+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:54:18.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.904381+0000","last_change":"2024-01-15T18:10:28.860747+0000","last_active":"2024-01-15T18:10:29.904381+0000","last_peered":"2024-01-15T18:10:29.904381+0000","last_clean":"2024-01-15T18:10:29.904381+0000","last_became_active":"2024-01-15T18:10:28.860414+0000","last_became_peered":"2024-01-15T18:10:28.860414+0000","last_unstale":"2024-01-15T18:10:29.904381+0000","last_undegraded":"2024-01-15T18:10:29.904381+0000","last_fullsized":"2024-01-15T18:10:29.904381+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:31.063023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085936+0000","last_change":"2024-01-15T18:10:29.896067+0000","last_active":"2024-01-15T18:10:31.085936+0000","last_peered":"2024-01-15T18:10:31.085936+0000","last_clean":"2024-01-15T18:10:31.085936+0000","last_became_active":"2024-01-15T18:10:29.895466+0000","last_became_peered":"2024-01-15T18:10:29.895466+0000","last_unstale":"2024-01-15T18:10:31.085936+0000","last_undegraded":"2024-01-15T18:10:31.085936+0000","last_fullsized":"2024-01-15T18:10:31.085936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:18.991298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.899240+0000","last_change":"2024-01-15T18:10:28.861707+0000","last_active":"2024-01-15T18:10:29.899240+0000","last_peered":"2024-01-15T18:10:29.899240+0000","last_clean":"2024-01-15T18:10:29.899240+0000","last_became_active":"2024-01-15T18:10:28.860381+0000","last_became_peered":"2024-01-15T18:10:28.860381+0000","last_unstale":"2024-01-15T18:10:29.899240+0000","last_undegraded":"2024-01-15T18:10:29.899240+0000","last_fullsized":"2024-01-15T18:10:29.899240+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:28.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085899+0000","last_change":"2024-01-15T18:10:29.893303+0000","last_active":"2024-01-15T18:10:31.085899+0000","last_peered":"2024-01-15T18:10:31.085899+0000","last_clean":"2024-01-15T18:10:31.085899+0000","last_became_active":"2024-01-15T18:10:29.891192+0000","last_became_peered":"2024-01-15T18:10:29.891192+0000","last_unstale":"2024-01-15T18:10:31.085899+0000","last_undegraded":"2024-01-15T18:10:31.085899+0000","last_fullsized":"2024-01-15T18:10:31.085899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:54.921393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.903961+0000","last_change":"2024-01-15T18:10:28.851518+0000","last_active":"2024-01-15T18:10:29.903961+0000","last_peered":"2024-01-15T18:10:29.903961+0000","last_clean":"2024-01-15T18:10:29.903961+0000","last_became_active":"2024-01-15T18:10:28.851216+0000","last_became_peered":"2024-01-15T18:10:28.851216+0000","last_unstale":"2024-01-15T18:10:29.903961+0000","last_undegraded":"2024-01-15T18:10:29.903961+0000","last_fullsized":"2024-01-15T18:10:29.903961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:25.704119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086539+0000","last_change":"2024-01-15T18:10:29.890269+0000","last_active":"2024-01-15T18:10:31.086539+0000","last_peered":"2024-01-15T18:10:31.086539+0000","last_clean":"2024-01-15T18:10:31.086539+0000","last_became_active":"2024-01-15T18:10:29.889995+0000","last_became_peered":"2024-01-15T18:10:29.889995+0000","last_unstale":"2024-01-15T18:10:31.086539+0000","last_undegraded":"2024-01-15T18:10:31.086539+0000","last_fullsized":"2024-01-15T18:10:31.086539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:49.720321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.903898+0000","last_change":"2024-01-15T18:10:28.847695+0000","last_active":"2024-01-15T18:10:29.903898+0000","last_peered":"2024-01-15T18:10:29.903898+0000","last_clean":"2024-01-15T18:10:29.903898+0000","last_became_active":"2024-01-15T18:10:28.847366+0000","last_became_peered":"2024-01-15T18:10:28.847366+0000","last_unstale":"2024-01-15T18:10:29.903898+0000","last_undegraded":"2024-01-15T18:10:29.903898+0000","last_fullsized":"2024-01-15T18:10:29.903898+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:09.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086618+0000","last_change":"2024-01-15T18:10:29.875494+0000","last_active":"2024-01-15T18:10:31.086618+0000","last_peered":"2024-01-15T18:10:31.086618+0000","last_clean":"2024-01-15T18:10:31.086618+0000","last_became_active":"2024-01-15T18:10:29.875157+0000","last_became_peered":"2024-01-15T18:10:29.875157+0000","last_unstale":"2024-01-15T18:10:31.086618+0000","last_undegraded":"2024-01-15T18:10:31.086618+0000","last_fullsized":"2024-01-15T18:10:31.086618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:15.629310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.901308+0000","last_change":"2024-01-15T18:10:28.860229+0000","last_active":"2024-01-15T18:10:29.901308+0000","last_peered":"2024-01-15T18:10:29.901308+0000","last_clean":"2024-01-15T18:10:29.901308+0000","last_became_active":"2024-01-15T18:10:28.859548+0000","last_became_peered":"2024-01-15T18:10:28.859548+0000","last_unstale":"2024-01-15T18:10:29.901308+0000","last_undegraded":"2024-01-15T18:10:29.901308+0000","last_fullsized":"2024-01-15T18:10:29.901308+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:10.860846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884423+0000","last_change":"2024-01-15T18:10:29.893025+0000","last_active":"2024-01-15T18:10:30.884423+0000","last_peered":"2024-01-15T18:10:30.884423+0000","last_clean":"2024-01-15T18:10:30.884423+0000","last_became_active":"2024-01-15T18:10:29.892694+0000","last_became_peered":"2024-01-15T18:10:29.892694+0000","last_unstale":"2024-01-15T18:10:30.884423+0000","last_undegraded":"2024-01-15T18:10:30.884423+0000","last_fullsized":"2024-01-15T18:10:30.884423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:37.045766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.898844+0000","last_change":"2024-01-15T18:10:28.860903+0000","last_active":"2024-01-15T18:10:29.898844+0000","last_peered":"2024-01-15T18:10:29.898844+0000","last_clean":"2024-01-15T18:10:29.898844+0000","last_became_active":"2024-01-15T18:10:28.860595+0000","last_became_peered":"2024-01-15T18:10:28.860595+0000","last_unstale":"2024-01-15T18:10:29.898844+0000","last_undegraded":"2024-01-15T18:10:29.898844+0000","last_fullsized":"2024-01-15T18:10:29.898844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:51.755398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084229+0000","last_change":"2024-01-15T18:10:29.871081+0000","last_active":"2024-01-15T18:10:31.084229+0000","last_peered":"2024-01-15T18:10:31.084229+0000","last_clean":"2024-01-15T18:10:31.084229+0000","last_became_active":"2024-01-15T18:10:29.870386+0000","last_became_peered":"2024-01-15T18:10:29.870386+0000","last_unstale":"2024-01-15T18:10:31.084229+0000","last_undegraded":"2024-01-15T18:10:31.084229+0000","last_fullsized":"2024-01-15T18:10:31.084229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:54.921393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.868000+0000","last_change":"2024-01-15T18:10:28.856007+0000","last_active":"2024-01-15T18:10:29.868000+0000","last_peered":"2024-01-15T18:10:29.868000+0000","last_clean":"2024-01-15T18:10:29.868000+0000","last_became_active":"2024-01-15T18:10:28.855729+0000","last_became_peered":"2024-01-15T18:10:28.855729+0000","last_unstale":"2024-01-15T18:10:29.868000+0000","last_undegraded":"2024-01-15T18:10:29.868000+0000","last_fullsized":"2024-01-15T18:10:29.868000+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:24.812796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.083675+0000","last_change":"2024-01-15T18:10:29.871794+0000","last_active":"2024-01-15T18:10:31.083675+0000","last_peered":"2024-01-15T18:10:31.083675+0000","last_clean":"2024-01-15T18:10:31.083675+0000","last_became_active":"2024-01-15T18:10:29.871250+0000","last_became_peered":"2024-01-15T18:10:29.871250+0000","last_unstale":"2024-01-15T18:10:31.083675+0000","last_undegraded":"2024-01-15T18:10:31.083675+0000","last_fullsized":"2024-01-15T18:10:31.083675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:07:43.534696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.869439+0000","last_change":"2024-01-15T18:10:28.854028+0000","last_active":"2024-01-15T18:10:29.869439+0000","last_peered":"2024-01-15T18:10:29.869439+0000","last_clean":"2024-01-15T18:10:29.869439+0000","last_became_active":"2024-01-15T18:10:28.853569+0000","last_became_peered":"2024-01-15T18:10:28.853569+0000","last_unstale":"2024-01-15T18:10:29.869439+0000","last_undegraded":"2024-01-15T18:10:29.869439+0000","last_fullsized":"2024-01-15T18:10:29.869439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:36.806891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085218+0000","last_change":"2024-01-15T18:10:29.878271+0000","last_active":"2024-01-15T18:10:31.085218+0000","last_peered":"2024-01-15T18:10:31.085218+0000","last_clean":"2024-01-15T18:10:31.085218+0000","last_became_active":"2024-01-15T18:10:29.877770+0000","last_became_peered":"2024-01-15T18:10:29.877770+0000","last_unstale":"2024-01-15T18:10:31.085218+0000","last_undegraded":"2024-01-15T18:10:31.085218+0000","last_fullsized":"2024-01-15T18:10:31.085218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.445206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.878955+0000","last_change":"2024-01-15T18:10:28.862577+0000","last_active":"2024-01-15T18:10:29.878955+0000","last_peered":"2024-01-15T18:10:29.878955+0000","last_clean":"2024-01-15T18:10:29.878955+0000","last_became_active":"2024-01-15T18:10:28.862234+0000","last_became_peered":"2024-01-15T18:10:28.862234+0000","last_unstale":"2024-01-15T18:10:29.878955+0000","last_undegraded":"2024-01-15T18:10:29.878955+0000","last_fullsized":"2024-01-15T18:10:29.878955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:47.854699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887978+0000","last_change":"2024-01-15T18:10:29.888839+0000","last_active":"2024-01-15T18:10:30.887978+0000","last_peered":"2024-01-15T18:10:30.887978+0000","last_clean":"2024-01-15T18:10:30.887978+0000","last_became_active":"2024-01-15T18:10:29.888544+0000","last_became_peered":"2024-01-15T18:10:29.888544+0000","last_unstale":"2024-01-15T18:10:30.887978+0000","last_undegraded":"2024-01-15T18:10:30.887978+0000","last_fullsized":"2024-01-15T18:10:30.887978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.875214+0000","last_change":"2024-01-15T18:10:28.859089+0000","last_active":"2024-01-15T18:10:29.875214+0000","last_peered":"2024-01-15T18:10:29.875214+0000","last_clean":"2024-01-15T18:10:29.875214+0000","last_became_active":"2024-01-15T18:10:28.858772+0000","last_became_peered":"2024-01-15T18:10:28.858772+0000","last_unstale":"2024-01-15T18:10:29.875214+0000","last_undegraded":"2024-01-15T18:10:29.875214+0000","last_fullsized":"2024-01-15T18:10:29.875214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:16.178426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084403+0000","last_change":"2024-01-15T18:10:29.896917+0000","last_active":"2024-01-15T18:10:31.084403+0000","last_peered":"2024-01-15T18:10:31.084403+0000","last_clean":"2024-01-15T18:10:31.084403+0000","last_became_active":"2024-01-15T18:10:29.894525+0000","last_became_peered":"2024-01-15T18:10:29.894525+0000","last_unstale":"2024-01-15T18:10:31.084403+0000","last_undegraded":"2024-01-15T18:10:31.084403+0000","last_fullsized":"2024-01-15T18:10:31.084403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:09.028738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902394+0000","last_change":"2024-01-15T18:10:28.864142+0000","last_active":"2024-01-15T18:10:29.902394+0000","last_peered":"2024-01-15T18:10:29.902394+0000","last_clean":"2024-01-15T18:10:29.902394+0000","last_became_active":"2024-01-15T18:10:28.863733+0000","last_became_peered":"2024-01-15T18:10:28.863733+0000","last_unstale":"2024-01-15T18:10:29.902394+0000","last_undegraded":"2024-01-15T18:10:29.902394+0000","last_fullsized":"2024-01-15T18:10:29.902394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:19.865670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885210+0000","last_change":"2024-01-15T18:10:29.889650+0000","last_active":"2024-01-15T18:10:30.885210+0000","last_peered":"2024-01-15T18:10:30.885210+0000","last_clean":"2024-01-15T18:10:30.885210+0000","last_became_active":"2024-01-15T18:10:29.889309+0000","last_became_peered":"2024-01-15T18:10:29.889309+0000","last_unstale":"2024-01-15T18:10:30.885210+0000","last_undegraded":"2024-01-15T18:10:30.885210+0000","last_fullsized":"2024-01-15T18:10:30.885210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:09.942053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.905427+0000","last_change":"2024-01-15T18:10:28.852757+0000","last_active":"2024-01-15T18:10:29.905427+0000","last_peered":"2024-01-15T18:10:29.905427+0000","last_clean":"2024-01-15T18:10:29.905427+0000","last_became_active":"2024-01-15T18:10:28.852416+0000","last_became_peered":"2024-01-15T18:10:28.852416+0000","last_unstale":"2024-01-15T18:10:29.905427+0000","last_undegraded":"2024-01-15T18:10:29.905427+0000","last_fullsized":"2024-01-15T18:10:29.905427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:05.863393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085400+0000","last_change":"2024-01-15T18:10:29.892056+0000","last_active":"2024-01-15T18:10:31.085400+0000","last_peered":"2024-01-15T18:10:31.085400+0000","last_clean":"2024-01-15T18:10:31.085400+0000","last_became_active":"2024-01-15T18:10:29.889387+0000","last_became_peered":"2024-01-15T18:10:29.889387+0000","last_unstale":"2024-01-15T18:10:31.085400+0000","last_undegraded":"2024-01-15T18:10:31.085400+0000","last_fullsized":"2024-01-15T18:10:31.085400+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:18.991298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.904711+0000","last_change":"2024-01-15T18:10:28.846674+0000","last_active":"2024-01-15T18:10:29.904711+0000","last_peered":"2024-01-15T18:10:29.904711+0000","last_clean":"2024-01-15T18:10:29.904711+0000","last_became_active":"2024-01-15T18:10:28.845875+0000","last_became_peered":"2024-01-15T18:10:28.845875+0000","last_unstale":"2024-01-15T18:10:29.904711+0000","last_undegraded":"2024-01-15T18:10:29.904711+0000","last_fullsized":"2024-01-15T18:10:29.904711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:23.936503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085124+0000","last_change":"2024-01-15T18:10:29.876230+0000","last_active":"2024-01-15T18:10:31.085124+0000","last_peered":"2024-01-15T18:10:31.085124+0000","last_clean":"2024-01-15T18:10:31.085124+0000","last_became_active":"2024-01-15T18:10:29.874419+0000","last_became_peered":"2024-01-15T18:10:29.874419+0000","last_unstale":"2024-01-15T18:10:31.085124+0000","last_undegraded":"2024-01-15T18:10:31.085124+0000","last_fullsized":"2024-01-15T18:10:31.085124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:42.772403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.856671+0000","last_change":"2024-01-15T18:10:28.855952+0000","last_active":"2024-01-15T18:10:29.856671+0000","last_peered":"2024-01-15T18:10:29.856671+0000","last_clean":"2024-01-15T18:10:29.856671+0000","last_became_active":"2024-01-15T18:10:28.855647+0000","last_became_peered":"2024-01-15T18:10:28.855647+0000","last_unstale":"2024-01-15T18:10:29.856671+0000","last_undegraded":"2024-01-15T18:10:29.856671+0000","last_fullsized":"2024-01-15T18:10:29.856671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.083977+0000","last_change":"2024-01-15T18:10:29.876118+0000","last_active":"2024-01-15T18:10:31.083977+0000","last_peered":"2024-01-15T18:10:31.083977+0000","last_clean":"2024-01-15T18:10:31.083977+0000","last_became_active":"2024-01-15T18:10:29.874531+0000","last_became_peered":"2024-01-15T18:10:29.874531+0000","last_unstale":"2024-01-15T18:10:31.083977+0000","last_undegraded":"2024-01-15T18:10:31.083977+0000","last_fullsized":"2024-01-15T18:10:31.083977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:44.687374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.865704+0000","last_change":"2024-01-15T18:10:28.853310+0000","last_active":"2024-01-15T18:10:29.865704+0000","last_peered":"2024-01-15T18:10:29.865704+0000","last_clean":"2024-01-15T18:10:29.865704+0000","last_became_active":"2024-01-15T18:10:28.853030+0000","last_became_peered":"2024-01-15T18:10:28.853030+0000","last_unstale":"2024-01-15T18:10:29.865704+0000","last_undegraded":"2024-01-15T18:10:29.865704+0000","last_fullsized":"2024-01-15T18:10:29.865704+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:41:56.867321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085205+0000","last_change":"2024-01-15T18:10:29.887053+0000","last_active":"2024-01-15T18:10:31.085205+0000","last_peered":"2024-01-15T18:10:31.085205+0000","last_clean":"2024-01-15T18:10:31.085205+0000","last_became_active":"2024-01-15T18:10:29.886310+0000","last_became_peered":"2024-01-15T18:10:29.886310+0000","last_unstale":"2024-01-15T18:10:31.085205+0000","last_undegraded":"2024-01-15T18:10:31.085205+0000","last_fullsized":"2024-01-15T18:10:31.085205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:26.317819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.904150+0000","last_change":"2024-01-15T18:10:28.856512+0000","last_active":"2024-01-15T18:10:29.904150+0000","last_peered":"2024-01-15T18:10:29.904150+0000","last_clean":"2024-01-15T18:10:29.904150+0000","last_became_active":"2024-01-15T18:10:28.855528+0000","last_became_peered":"2024-01-15T18:10:28.855528+0000","last_unstale":"2024-01-15T18:10:29.904150+0000","last_undegraded":"2024-01-15T18:10:29.904150+0000","last_fullsized":"2024-01-15T18:10:29.904150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:56.753249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085914+0000","last_change":"2024-01-15T18:10:29.881440+0000","last_active":"2024-01-15T18:10:31.085914+0000","last_peered":"2024-01-15T18:10:31.085914+0000","last_clean":"2024-01-15T18:10:31.085914+0000","last_became_active":"2024-01-15T18:10:29.881163+0000","last_became_peered":"2024-01-15T18:10:29.881163+0000","last_unstale":"2024-01-15T18:10:31.085914+0000","last_undegraded":"2024-01-15T18:10:31.085914+0000","last_fullsized":"2024-01-15T18:10:31.085914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:07:43.534696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.899591+0000","last_change":"2024-01-15T18:10:28.864218+0000","last_active":"2024-01-15T18:10:29.899591+0000","last_peered":"2024-01-15T18:10:29.899591+0000","last_clean":"2024-01-15T18:10:29.899591+0000","last_became_active":"2024-01-15T18:10:28.863828+0000","last_became_peered":"2024-01-15T18:10:28.863828+0000","last_unstale":"2024-01-15T18:10:29.899591+0000","last_undegraded":"2024-01-15T18:10:29.899591+0000","last_fullsized":"2024-01-15T18:10:29.899591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:24.812796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086310+0000","last_change":"2024-01-15T18:10:29.894064+0000","last_active":"2024-01-15T18:10:31.086310+0000","last_peered":"2024-01-15T18:10:31.086310+0000","last_clean":"2024-01-15T18:10:31.086310+0000","last_became_active":"2024-01-15T18:10:29.893707+0000","last_became_peered":"2024-01-15T18:10:29.893707+0000","last_unstale":"2024-01-15T18:10:31.086310+0000","last_undegraded":"2024-01-15T18:10:31.086310+0000","last_fullsized":"2024-01-15T18:10:31.086310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:42.772403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902966+0000","last_change":"2024-01-15T18:10:28.859456+0000","last_active":"2024-01-15T18:10:29.902966+0000","last_peered":"2024-01-15T18:10:29.902966+0000","last_clean":"2024-01-15T18:10:29.902966+0000","last_became_active":"2024-01-15T18:10:28.858710+0000","last_became_peered":"2024-01-15T18:10:28.858710+0000","last_unstale":"2024-01-15T18:10:29.902966+0000","last_undegraded":"2024-01-15T18:10:29.902966+0000","last_fullsized":"2024-01-15T18:10:29.902966+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:22.308478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886254+0000","last_change":"2024-01-15T18:10:29.893444+0000","last_active":"2024-01-15T18:10:30.886254+0000","last_peered":"2024-01-15T18:10:30.886254+0000","last_clean":"2024-01-15T18:10:30.886254+0000","last_became_active":"2024-01-15T18:10:29.893114+0000","last_became_peered":"2024-01-15T18:10:29.893114+0000","last_unstale":"2024-01-15T18:10:30.886254+0000","last_undegraded":"2024-01-15T18:10:30.886254+0000","last_fullsized":"2024-01-15T18:10:30.886254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:21.928711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.877354+0000","last_change":"2024-01-15T18:10:28.856970+0000","last_active":"2024-01-15T18:10:29.877354+0000","last_peered":"2024-01-15T18:10:29.877354+0000","last_clean":"2024-01-15T18:10:29.877354+0000","last_became_active":"2024-01-15T18:10:28.856677+0000","last_became_peered":"2024-01-15T18:10:28.856677+0000","last_unstale":"2024-01-15T18:10:29.877354+0000","last_undegraded":"2024-01-15T18:10:29.877354+0000","last_fullsized":"2024-01-15T18:10:29.877354+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:17.019853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884031+0000","last_change":"2024-01-15T18:10:29.884301+0000","last_active":"2024-01-15T18:10:30.884031+0000","last_peered":"2024-01-15T18:10:30.884031+0000","last_clean":"2024-01-15T18:10:30.884031+0000","last_became_active":"2024-01-15T18:10:29.883550+0000","last_became_peered":"2024-01-15T18:10:29.883550+0000","last_unstale":"2024-01-15T18:10:30.884031+0000","last_undegraded":"2024-01-15T18:10:30.884031+0000","last_fullsized":"2024-01-15T18:10:30.884031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:37.827724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902506+0000","last_change":"2024-01-15T18:10:28.853663+0000","last_active":"2024-01-15T18:10:29.902506+0000","last_peered":"2024-01-15T18:10:29.902506+0000","last_clean":"2024-01-15T18:10:29.902506+0000","last_became_active":"2024-01-15T18:10:28.849935+0000","last_became_peered":"2024-01-15T18:10:28.849935+0000","last_unstale":"2024-01-15T18:10:29.902506+0000","last_undegraded":"2024-01-15T18:10:29.902506+0000","last_fullsized":"2024-01-15T18:10:29.902506+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:17.975732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886998+0000","last_change":"2024-01-15T18:10:29.878910+0000","last_active":"2024-01-15T18:10:30.886998+0000","last_peered":"2024-01-15T18:10:30.886998+0000","last_clean":"2024-01-15T18:10:30.886998+0000","last_became_active":"2024-01-15T18:10:29.878619+0000","last_became_peered":"2024-01-15T18:10:29.878619+0000","last_unstale":"2024-01-15T18:10:30.886998+0000","last_undegraded":"2024-01-15T18:10:30.886998+0000","last_fullsized":"2024-01-15T18:10:30.886998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:57.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.865578+0000","last_change":"2024-01-15T18:10:28.848970+0000","last_active":"2024-01-15T18:10:29.865578+0000","last_peered":"2024-01-15T18:10:29.865578+0000","last_clean":"2024-01-15T18:10:29.865578+0000","last_became_active":"2024-01-15T18:10:28.848694+0000","last_became_peered":"2024-01-15T18:10:28.848694+0000","last_unstale":"2024-01-15T18:10:29.865578+0000","last_undegraded":"2024-01-15T18:10:29.865578+0000","last_fullsized":"2024-01-15T18:10:29.865578+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:44:45.666850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883564+0000","last_change":"2024-01-15T18:10:29.886092+0000","last_active":"2024-01-15T18:10:30.883564+0000","last_peered":"2024-01-15T18:10:30.883564+0000","last_clean":"2024-01-15T18:10:30.883564+0000","last_became_active":"2024-01-15T18:10:29.880525+0000","last_became_peered":"2024-01-15T18:10:29.880525+0000","last_unstale":"2024-01-15T18:10:30.883564+0000","last_undegraded":"2024-01-15T18:10:30.883564+0000","last_fullsized":"2024-01-15T18:10:30.883564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:00.656213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.864004+0000","last_change":"2024-01-15T18:10:28.863335+0000","last_active":"2024-01-15T18:10:29.864004+0000","last_peered":"2024-01-15T18:10:29.864004+0000","last_clean":"2024-01-15T18:10:29.864004+0000","last_became_active":"2024-01-15T18:10:28.862934+0000","last_became_peered":"2024-01-15T18:10:28.862934+0000","last_unstale":"2024-01-15T18:10:29.864004+0000","last_undegraded":"2024-01-15T18:10:29.864004+0000","last_fullsized":"2024-01-15T18:10:29.864004+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:04.581565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887218+0000","last_change":"2024-01-15T18:10:29.883072+0000","last_active":"2024-01-15T18:10:30.887218+0000","last_peered":"2024-01-15T18:10:30.887218+0000","last_clean":"2024-01-15T18:10:30.887218+0000","last_became_active":"2024-01-15T18:10:29.882241+0000","last_became_peered":"2024-01-15T18:10:29.882241+0000","last_unstale":"2024-01-15T18:10:30.887218+0000","last_undegraded":"2024-01-15T18:10:30.887218+0000","last_fullsized":"2024-01-15T18:10:30.887218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:57.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.856692+0000","last_change":"2024-01-15T18:10:28.853080+0000","last_active":"2024-01-15T18:10:29.856692+0000","last_peered":"2024-01-15T18:10:29.856692+0000","last_clean":"2024-01-15T18:10:29.856692+0000","last_became_active":"2024-01-15T18:10:28.848348+0000","last_became_peered":"2024-01-15T18:10:28.848348+0000","last_unstale":"2024-01-15T18:10:29.856692+0000","last_undegraded":"2024-01-15T18:10:29.856692+0000","last_fullsized":"2024-01-15T18:10:29.856692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:53.905827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885405+0000","last_change":"2024-01-15T18:10:29.889034+0000","last_active":"2024-01-15T18:10:30.885405+0000","last_peered":"2024-01-15T18:10:30.885405+0000","last_clean":"2024-01-15T18:10:30.885405+0000","last_became_active":"2024-01-15T18:10:29.888741+0000","last_became_peered":"2024-01-15T18:10:29.888741+0000","last_unstale":"2024-01-15T18:10:30.885405+0000","last_undegraded":"2024-01-15T18:10:30.885405+0000","last_fullsized":"2024-01-15T18:10:30.885405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:05.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.901824+0000","last_change":"2024-01-15T18:10:28.853859+0000","last_active":"2024-01-15T18:10:29.901824+0000","last_peered":"2024-01-15T18:10:29.901824+0000","last_clean":"2024-01-15T18:10:29.901824+0000","last_became_active":"2024-01-15T18:10:28.853567+0000","last_became_peered":"2024-01-15T18:10:28.853567+0000","last_unstale":"2024-01-15T18:10:29.901824+0000","last_undegraded":"2024-01-15T18:10:29.901824+0000","last_fullsized":"2024-01-15T18:10:29.901824+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:56:53.496817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086285+0000","last_change":"2024-01-15T18:10:29.887191+0000","last_active":"2024-01-15T18:10:31.086285+0000","last_peered":"2024-01-15T18:10:31.086285+0000","last_clean":"2024-01-15T18:10:31.086285+0000","last_became_active":"2024-01-15T18:10:29.886890+0000","last_became_peered":"2024-01-15T18:10:29.886890+0000","last_unstale":"2024-01-15T18:10:31.086285+0000","last_undegraded":"2024-01-15T18:10:31.086285+0000","last_fullsized":"2024-01-15T18:10:31.086285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.445206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.088005+0000","last_change":"2024-01-15T18:10:28.847096+0000","last_active":"2024-01-15T18:10:31.088005+0000","last_peered":"2024-01-15T18:10:31.088005+0000","last_clean":"2024-01-15T18:10:31.088005+0000","last_became_active":"2024-01-15T18:10:28.845910+0000","last_became_peered":"2024-01-15T18:10:28.845910+0000","last_unstale":"2024-01-15T18:10:31.088005+0000","last_undegraded":"2024-01-15T18:10:31.088005+0000","last_fullsized":"2024-01-15T18:10:31.088005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:46.755994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085281+0000","last_change":"2024-01-15T18:10:29.896978+0000","last_active":"2024-01-15T18:10:31.085281+0000","last_peered":"2024-01-15T18:10:31.085281+0000","last_clean":"2024-01-15T18:10:31.085281+0000","last_became_active":"2024-01-15T18:10:29.895549+0000","last_became_peered":"2024-01-15T18:10:29.895549+0000","last_unstale":"2024-01-15T18:10:31.085281+0000","last_undegraded":"2024-01-15T18:10:31.085281+0000","last_fullsized":"2024-01-15T18:10:31.085281+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:26.719685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874581+0000","last_change":"2024-01-15T18:10:28.853688+0000","last_active":"2024-01-15T18:10:29.874581+0000","last_peered":"2024-01-15T18:10:29.874581+0000","last_clean":"2024-01-15T18:10:29.874581+0000","last_became_active":"2024-01-15T18:10:28.853347+0000","last_became_peered":"2024-01-15T18:10:28.853347+0000","last_unstale":"2024-01-15T18:10:29.874581+0000","last_undegraded":"2024-01-15T18:10:29.874581+0000","last_fullsized":"2024-01-15T18:10:29.874581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:02.736339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885571+0000","last_change":"2024-01-15T18:10:29.889239+0000","last_active":"2024-01-15T18:10:30.885571+0000","last_peered":"2024-01-15T18:10:30.885571+0000","last_clean":"2024-01-15T18:10:30.885571+0000","last_became_active":"2024-01-15T18:10:29.887314+0000","last_became_peered":"2024-01-15T18:10:29.887314+0000","last_unstale":"2024-01-15T18:10:30.885571+0000","last_undegraded":"2024-01-15T18:10:30.885571+0000","last_fullsized":"2024-01-15T18:10:30.885571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:41.860420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.880493+0000","last_change":"2024-01-15T18:10:28.861226+0000","last_active":"2024-01-15T18:10:29.880493+0000","last_peered":"2024-01-15T18:10:29.880493+0000","last_clean":"2024-01-15T18:10:29.880493+0000","last_became_active":"2024-01-15T18:10:28.860947+0000","last_became_peered":"2024-01-15T18:10:28.860947+0000","last_unstale":"2024-01-15T18:10:29.880493+0000","last_undegraded":"2024-01-15T18:10:29.880493+0000","last_fullsized":"2024-01-15T18:10:29.880493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:56.753249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887268+0000","last_change":"2024-01-15T18:10:29.871769+0000","last_active":"2024-01-15T18:10:30.887268+0000","last_peered":"2024-01-15T18:10:30.887268+0000","last_clean":"2024-01-15T18:10:30.887268+0000","last_became_active":"2024-01-15T18:10:29.871132+0000","last_became_peered":"2024-01-15T18:10:29.871132+0000","last_unstale":"2024-01-15T18:10:30.887268+0000","last_undegraded":"2024-01-15T18:10:30.887268+0000","last_fullsized":"2024-01-15T18:10:30.887268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.871998+0000","last_change":"2024-01-15T18:10:28.853003+0000","last_active":"2024-01-15T18:10:29.871998+0000","last_peered":"2024-01-15T18:10:29.871998+0000","last_clean":"2024-01-15T18:10:29.871998+0000","last_became_active":"2024-01-15T18:10:28.852522+0000","last_became_peered":"2024-01-15T18:10:28.852522+0000","last_unstale":"2024-01-15T18:10:29.871998+0000","last_undegraded":"2024-01-15T18:10:29.871998+0000","last_fullsized":"2024-01-15T18:10:29.871998+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:56.854998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085137+0000","last_change":"2024-01-15T18:10:29.889868+0000","last_active":"2024-01-15T18:10:31.085137+0000","last_peered":"2024-01-15T18:10:31.085137+0000","last_clean":"2024-01-15T18:10:31.085137+0000","last_became_active":"2024-01-15T18:10:29.887146+0000","last_became_peered":"2024-01-15T18:10:29.887146+0000","last_unstale":"2024-01-15T18:10:31.085137+0000","last_undegraded":"2024-01-15T18:10:31.085137+0000","last_fullsized":"2024-01-15T18:10:31.085137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:54.921393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.879486+0000","last_change":"2024-01-15T18:10:28.861120+0000","last_active":"2024-01-15T18:10:29.879486+0000","last_peered":"2024-01-15T18:10:29.879486+0000","last_clean":"2024-01-15T18:10:29.879486+0000","last_became_active":"2024-01-15T18:10:28.860131+0000","last_became_peered":"2024-01-15T18:10:28.860131+0000","last_unstale":"2024-01-15T18:10:29.879486+0000","last_undegraded":"2024-01-15T18:10:29.879486+0000","last_fullsized":"2024-01-15T18:10:29.879486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:53.466810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084619+0000","last_change":"2024-01-15T18:10:29.883153+0000","last_active":"2024-01-15T18:10:31.084619+0000","last_peered":"2024-01-15T18:10:31.084619+0000","last_clean":"2024-01-15T18:10:31.084619+0000","last_became_active":"2024-01-15T18:10:29.882185+0000","last_became_peered":"2024-01-15T18:10:29.882185+0000","last_unstale":"2024-01-15T18:10:31.084619+0000","last_undegraded":"2024-01-15T18:10:31.084619+0000","last_fullsized":"2024-01-15T18:10:31.084619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:18.991298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874272+0000","last_change":"2024-01-15T18:10:28.863474+0000","last_active":"2024-01-15T18:10:29.874272+0000","last_peered":"2024-01-15T18:10:29.874272+0000","last_clean":"2024-01-15T18:10:29.874272+0000","last_became_active":"2024-01-15T18:10:28.863019+0000","last_became_peered":"2024-01-15T18:10:28.863019+0000","last_unstale":"2024-01-15T18:10:29.874272+0000","last_undegraded":"2024-01-15T18:10:29.874272+0000","last_fullsized":"2024-01-15T18:10:29.874272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:04.910114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084125+0000","last_change":"2024-01-15T18:10:29.896146+0000","last_active":"2024-01-15T18:10:31.084125+0000","last_peered":"2024-01-15T18:10:31.084125+0000","last_clean":"2024-01-15T18:10:31.084125+0000","last_became_active":"2024-01-15T18:10:29.895838+0000","last_became_peered":"2024-01-15T18:10:29.895838+0000","last_unstale":"2024-01-15T18:10:31.084125+0000","last_undegraded":"2024-01-15T18:10:31.084125+0000","last_fullsized":"2024-01-15T18:10:31.084125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.877158+0000","last_change":"2024-01-15T18:10:28.863368+0000","last_active":"2024-01-15T18:10:29.877158+0000","last_peered":"2024-01-15T18:10:29.877158+0000","last_clean":"2024-01-15T18:10:29.877158+0000","last_became_active":"2024-01-15T18:10:28.861032+0000","last_became_peered":"2024-01-15T18:10:28.861032+0000","last_unstale":"2024-01-15T18:10:29.877158+0000","last_undegraded":"2024-01-15T18:10:29.877158+0000","last_fullsized":"2024-01-15T18:10:29.877158+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:10:55.062365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087326+0000","last_change":"2024-01-15T18:10:29.878751+0000","last_active":"2024-01-15T18:10:31.087326+0000","last_peered":"2024-01-15T18:10:31.087326+0000","last_clean":"2024-01-15T18:10:31.087326+0000","last_became_active":"2024-01-15T18:10:29.876467+0000","last_became_peered":"2024-01-15T18:10:29.876467+0000","last_unstale":"2024-01-15T18:10:31.087326+0000","last_undegraded":"2024-01-15T18:10:31.087326+0000","last_fullsized":"2024-01-15T18:10:31.087326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:27.719678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902018+0000","last_change":"2024-01-15T18:10:28.859062+0000","last_active":"2024-01-15T18:10:29.902018+0000","last_peered":"2024-01-15T18:10:29.902018+0000","last_clean":"2024-01-15T18:10:29.902018+0000","last_became_active":"2024-01-15T18:10:28.858021+0000","last_became_peered":"2024-01-15T18:10:28.858021+0000","last_unstale":"2024-01-15T18:10:29.902018+0000","last_undegraded":"2024-01-15T18:10:29.902018+0000","last_fullsized":"2024-01-15T18:10:29.902018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":228,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.866943+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:10:29.866943+0000","last_peered":"2024-01-15T18:10:29.866943+0000","last_clean":"2024-01-15T18:10:29.866943+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:10:29.866943+0000","last_undegraded":"2024-01-15T18:10:29.866943+0000","last_fullsized":"2024-01-15T18:10:29.866943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542304,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77180,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294660,"statfs":{"total":96636764160,"available":96557731840,"internally_reserved":0,"allocated":1687552,"data_stored":1062336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.3799999999999999},"min":{"1min":0.81899999999999995,"5min":0.626,"15min":0.53600000000000003},"max":{"1min":1.677,"5min":4.0019999999999998,"15min":14.114000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.7030000000000001,"15min":1.5680000000000001},"min":{"1min":0.91200000000000003,"5min":0.69999999999999996,"15min":0.51700000000000002},"max":{"1min":1.6970000000000001,"5min":27.32,"15min":27.32},"last":0.997}]},{"osd":1,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.8300000000000001,"15min":1.4670000000000001},"min":{"1min":0.97299999999999998,"5min":0.53300000000000003,"15min":0.40300000000000002},"max":{"1min":1.617,"5min":29.167999999999999,"15min":29.167999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.835,"5min":4.2699999999999996,"15min":12.565},"last":0.80900000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3360000000000001,"15min":1.464},"min":{"1min":0.61399999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.55,"5min":4.1920000000000002,"15min":13.801},"last":1.073},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.7,"15min":1.542},"min":{"1min":0.91500000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.788,"5min":27.358000000000001,"15min":27.358000000000001},"last":1.2350000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.853,"15min":1.609},"min":{"1min":1.097,"5min":0.61299999999999999,"15min":0.47099999999999997},"max":{"1min":1.619,"5min":29.204999999999998,"15min":29.204999999999998},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.425,"5min":1.506,"15min":1.5600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":1.8300000000000001,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.1919999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.7509999999999999,"15min":1.621},"min":{"1min":0.91200000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.724,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.355},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.887,"15min":1.653},"min":{"1min":1.151,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":1.782,"5min":29.25,"15min":29.25},"last":1.044}]},{"osd":5,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.78,"15min":1.679},"min":{"1min":0.91500000000000004,"5min":0.83199999999999996,"15min":0.39000000000000001},"max":{"1min":1.944,"5min":11.834,"15min":12.518000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.9159999999999999,"15min":1.722},"min":{"1min":1.03,"5min":0.60499999999999998,"15min":0.41099999999999998},"max":{"1min":1.883,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.1479999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0579999999999998,"15min":2.367},"min":{"1min":1.0169999999999999,"5min":0.86499999999999999,"15min":0.45600000000000002},"max":{"1min":2.0390000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.591,"5min":2.101,"15min":1.8080000000000001},"min":{"1min":1.1899999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":1.923,"5min":29.106999999999999,"15min":29.106999999999999},"last":1.262}]}]},{"osd":6,"up_from":14,"seq":60129542304,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58624,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313216,"statfs":{"total":96636764160,"available":96576733184,"internally_reserved":0,"allocated":1101824,"data_stored":472592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.276,"15min":1.1950000000000001},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":5.0330000000000004,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.3959999999999999,"15min":1.339},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.1659999999999999,"5min":9.548,"15min":21.565000000000001},"last":1.621}]},{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4319999999999999,"15min":1.3440000000000001},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"max":{"1min":5.1150000000000002,"5min":8.1189999999999998,"15min":8.1189999999999998},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.462,"15min":1.4370000000000001},"min":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.73,"5min":9.6389999999999993,"15min":13.984999999999999},"last":1.415}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3819999999999999,"15min":1.4630000000000001},"min":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":2.919,"5min":7.0030000000000001,"15min":34.670000000000002},"last":0.877},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.411},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.1609999999999996,"5min":6.585,"15min":6.585},"last":1.008}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6479999999999999,"15min":1.5529999999999999},"min":{"1min":1.0069999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.5680000000000001,"5min":16.855,"15min":16.855},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6180000000000001,"15min":1.679},"min":{"1min":0.96599999999999997,"5min":0.628,"15min":0.48499999999999999},"max":{"1min":2.9630000000000001,"5min":8.0139999999999993,"15min":34.725000000000001},"last":1.4690000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.512,"15min":1.4390000000000001},"min":{"1min":0.95399999999999996,"5min":0.56299999999999994,"15min":0.38400000000000001},"max":{"1min":5.2030000000000003,"5min":6.5220000000000002,"15min":8.5489999999999995},"last":1.157},{"interface":"front","average":{"1min":1.784,"5min":1.6379999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.63600000000000001,"15min":0.501},"max":{"1min":7.3230000000000004,"5min":10.118,"15min":10.118},"last":1.2090000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.0720000000000001,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":2.8420000000000001,"5min":7.9500000000000002,"15min":34.798000000000002},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.7410000000000001,"15min":1.609},"min":{"1min":1.228,"5min":0.66000000000000003,"15min":0.47899999999999998},"max":{"1min":8.0129999999999999,"5min":10.077,"15min":10.077},"last":1.5029999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":1.748,"15min":1.669},"min":{"1min":1.228,"5min":0.78900000000000003,"15min":0.64000000000000001},"max":{"1min":5.5060000000000002,"5min":10.302,"15min":10.302},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.7629999999999999,"15min":1.8500000000000001},"min":{"1min":1.2110000000000001,"5min":0.84599999999999997,"15min":0.52600000000000002},"max":{"1min":2.782,"5min":8.0519999999999996,"15min":34.762999999999998},"last":1.3049999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542306,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":1105920,"data_stored":473532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.5780000000000001,"15min":1.3660000000000001},"min":{"1min":0.71499999999999997,"5min":0.67900000000000005,"15min":0.52700000000000002},"max":{"1min":1.5980000000000001,"5min":26.077999999999999,"15min":26.077999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.431,"15min":1.4470000000000001},"min":{"1min":0.76400000000000001,"5min":0.73999999999999999,"15min":0.52400000000000002},"max":{"1min":1.7270000000000001,"5min":7.7220000000000004,"15min":14.711},"last":1.0209999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.653,"15min":1.387},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6679999999999999,"5min":26,"15min":26},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.409,"15min":1.3400000000000001},"min":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":1.6919999999999999,"5min":7.5309999999999997,"15min":12.619},"last":1.0940000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.8340000000000001,"15min":1.5409999999999999},"min":{"1min":0.874,"5min":0.73299999999999998,"15min":0.55700000000000005},"max":{"1min":4.8719999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.605,"5min":1.825,"15min":1.6020000000000001},"min":{"1min":0.90200000000000002,"5min":0.63900000000000001,"15min":0.59099999999999997},"max":{"1min":4.968,"5min":17.126999999999999,"15min":17.126999999999999},"last":1.3779999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4670000000000001,"15min":1.448},"min":{"1min":0.85999999999999999,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":1.722,"5min":7.6150000000000002,"15min":12.722},"last":1.504},{"interface":"front","average":{"1min":1.387,"5min":1.8740000000000001,"15min":1.637},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.63400000000000001},"max":{"1min":1.776,"5min":26.134,"15min":26.134},"last":1.4199999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.754,"5min":0.68999999999999995,"15min":0.55900000000000005},"max":{"1min":1.72,"5min":7.8730000000000002,"15min":25.776},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.409,"5min":1.9470000000000001,"15min":1.734},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.47799999999999998},"max":{"1min":1.8799999999999999,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.452}]},{"osd":6,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.466,"15min":1.5489999999999999},"min":{"1min":0.84099999999999997,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":1.9390000000000001,"5min":7.9240000000000004,"15min":12.769},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.798,"15min":1.8029999999999999},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.435},"max":{"1min":1.865,"5min":26.186,"15min":36.087000000000003},"last":1.484}]},{"osd":7,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.9319999999999999,"15min":1.8959999999999999},"min":{"1min":1.0680000000000001,"5min":0.79100000000000004,"15min":0.68000000000000005},"max":{"1min":1.98,"5min":9.6859999999999999,"15min":13.706},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.784},"min":{"1min":0.80000000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9159999999999999,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.7829999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542305,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58620,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313220,"statfs":{"total":96636764160,"available":96576737280,"internally_reserved":0,"allocated":1089536,"data_stored":471898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1870000000000001,"15min":1.3580000000000001},"min":{"1min":0.55700000000000005,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.9419999999999999,"5min":4.8170000000000002,"15min":21.643000000000001},"last":0.996},{"interface":"front","average":{"1min":1.113,"5min":1.3049999999999999,"15min":1.2949999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.861,"5min":8.9109999999999996,"15min":11.196},"last":1.4019999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5960000000000001,"15min":1.4059999999999999},"min":{"1min":0.59299999999999997,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":1.587,"5min":28.663,"15min":28.663},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.53,"15min":1.4970000000000001},"min":{"1min":0.72599999999999998,"5min":0.46200000000000002,"15min":0.35399999999999998},"max":{"1min":1.837,"5min":13.614000000000001,"15min":21.440000000000001},"last":1.3660000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.6220000000000001,"15min":1.446},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.8049999999999999,"5min":16.023,"15min":16.023},"last":1.038},{"interface":"front","average":{"1min":1.28,"5min":1.446,"15min":1.5109999999999999},"min":{"1min":0.94299999999999995,"5min":0.52500000000000002,"15min":0.38100000000000001},"max":{"1min":1.7010000000000001,"5min":7.9169999999999998,"15min":22.350999999999999},"last":1.1399999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.375,"15min":1.512},"min":{"1min":0.874,"5min":0.56699999999999995,"15min":0.48799999999999999},"max":{"1min":1.5329999999999999,"5min":5.0860000000000003,"15min":15.093},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.6919999999999999,"15min":1.651},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.6100000000000001,"5min":18.027000000000001,"15min":19.486000000000001},"last":0.84799999999999998}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3859999999999999,"15min":1.766},"min":{"1min":0.67800000000000005,"5min":0.41899999999999998,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":5.29,"15min":54.938000000000002},"last":1.2969999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.5620000000000001,"15min":1.5309999999999999},"min":{"1min":0.63,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":1.9610000000000001,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.0920000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.6890000000000001,"15min":1.615},"min":{"1min":0.90800000000000003,"5min":0.495,"15min":0.38700000000000001},"max":{"1min":1.9279999999999999,"5min":16.103999999999999,"15min":17.733000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.494,"5min":1.7050000000000001,"15min":1.853},"min":{"1min":0.94699999999999995,"5min":0.499,"15min":0.499},"max":{"1min":2.0430000000000001,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.4259999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.98,"15min":1.825},"min":{"1min":0.79100000000000004,"5min":0.46999999999999997,"15min":0.45100000000000001},"max":{"1min":2.0600000000000001,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8939999999999999,"15min":1.847},"min":{"1min":0.65200000000000002,"5min":0.62,"15min":0.51000000000000001},"max":{"1min":2.0310000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.256}]}]},{"osd":3,"up_from":14,"seq":60129542306,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68296,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303544,"statfs":{"total":96636764160,"available":96566829056,"internally_reserved":0,"allocated":1101824,"data_stored":473510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.371,"15min":1.0429999999999999},"min":{"1min":0.372,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.2050000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.151,"15min":1.028},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.665,"5min":5.0190000000000001,"15min":5.0529999999999999},"last":0.65200000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1060000000000001,"15min":1.0920000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.34399999999999997},"max":{"1min":1.365,"5min":5.226,"15min":8.1359999999999992},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.179},"min":{"1min":0.56899999999999995,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.8140000000000001,"5min":2.1339999999999999,"15min":7.2169999999999996},"last":1.2350000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.4510000000000001,"15min":1.256},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.859,"5min":14.025,"15min":14.025},"last":1.294},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3460000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.57399999999999995,"15min":0.46000000000000002},"max":{"1min":1.9710000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.4530000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.569,"15min":1.423},"min":{"1min":0.94099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2930000000000001,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.7430000000000001,"15min":1.5369999999999999},"min":{"1min":1.1100000000000001,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.9119999999999999,"5min":15.769,"15min":15.769},"last":1.1499999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.659,"15min":1.53},"min":{"1min":1.079,"5min":0.69299999999999995,"15min":0.65000000000000002},"max":{"1min":1.984,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.2490000000000001,"15min":1.8919999999999999},"min":{"1min":0.97599999999999998,"5min":0.67900000000000005,"15min":0.58299999999999996},"max":{"1min":2.0800000000000001,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.8160000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.86899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.6339999999999999,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.8180000000000001,"15min":1.708},"min":{"1min":1.246,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.048,"5min":6.8639999999999999,"15min":16.195},"last":1.512}]},{"osd":7,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.2949999999999999,"15min":1.859},"min":{"1min":1.034,"5min":0.76000000000000001,"15min":0.63900000000000001},"max":{"1min":2.2629999999999999,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.746},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1030000000000002,"15min":1.722},"min":{"1min":1.0049999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3679999999999999,"5min":17.622,"15min":17.622},"last":1.4850000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542305,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68280,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303560,"statfs":{"total":96636764160,"available":96566845440,"internally_reserved":0,"allocated":1093632,"data_stored":472480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98899999999999999,"15min":1.032},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.30099999999999999},"max":{"1min":1.3029999999999999,"5min":3.9199999999999999,"15min":6.101},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.087,"5min":1.1919999999999999,"15min":1.1419999999999999},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28299999999999997},"max":{"1min":1.6839999999999999,"5min":10.319000000000001,"15min":10.319000000000001},"last":1.494}]},{"osd":1,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0509999999999999,"15min":1.105},"min":{"1min":0.79600000000000004,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":1.5229999999999999,"5min":4.4939999999999998,"15min":10.807},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.2889999999999999,"15min":1.228},"min":{"1min":0.61599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5960000000000001,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":1.99}]},{"osd":3,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.6180000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":2.8010000000000002,"5min":25.891999999999999,"15min":25.891999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.373,"15min":1.256},"min":{"1min":0.78700000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.7270000000000001,"5min":14.946,"15min":14.946},"last":1.016}]},{"osd":4,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.546,"15min":1.552},"min":{"1min":0.874,"5min":0.66100000000000003,"15min":0.64900000000000002},"max":{"1min":1.9339999999999999,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.054},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.6870000000000001,"15min":1.5509999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":2.4249999999999998,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.2569999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.742,"15min":1.669},"min":{"1min":1.02,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8519999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":1.9590000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.6359999999999999,"15min":1.722},"min":{"1min":1,"5min":0.71699999999999997,"15min":0.55200000000000005},"max":{"1min":2.02,"5min":8.5250000000000004,"15min":37.020000000000003},"last":1.4119999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.645,"15min":1.7150000000000001},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.889,"5min":10.185,"15min":23.762},"last":1.768},{"interface":"front","average":{"1min":1.595,"5min":1.9139999999999999,"15min":1.867},"min":{"1min":1.052,"5min":0.85899999999999999,"15min":0.73499999999999999},"max":{"1min":2.8860000000000001,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.833}]},{"osd":7,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":2.1349999999999998,"15min":1.931},"min":{"1min":1.0169999999999999,"5min":0.85999999999999999,"15min":0.68999999999999995},"max":{"1min":1.9470000000000001,"5min":19.986000000000001,"15min":52.767000000000003},"last":1.887},{"interface":"front","average":{"1min":1.605,"5min":1.988,"15min":1.804},"min":{"1min":0.97899999999999998,"5min":0.88100000000000001,"15min":0.83499999999999996},"max":{"1min":2.9169999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":2.0499999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542305,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68288,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303552,"statfs":{"total":96636764160,"available":96566837248,"internally_reserved":0,"allocated":1101824,"data_stored":472092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.167,"15min":1.0189999999999999},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3160000000000001,"5min":19.218,"15min":19.218},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.288,"15min":1.1879999999999999},"min":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.573,"5min":19.823,"15min":19.823},"last":0.77000000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0129999999999999,"15min":1.0940000000000001},"min":{"1min":0.47599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6759999999999999,"5min":2.8980000000000001,"15min":7.6680000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.105,"5min":1.407,"15min":1.2470000000000001},"min":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.32500000000000001},"max":{"1min":1.52,"5min":19.318000000000001,"15min":19.318000000000001},"last":1.4059999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5109999999999999,"15min":1.365},"min":{"1min":0.56699999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6100000000000001,"5min":19.387,"15min":19.387},"last":0.876},{"interface":"front","average":{"1min":1.218,"5min":1.4379999999999999,"15min":1.351},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.8919999999999999,"5min":19.733000000000001,"15min":19.733000000000001},"last":1.1539999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.7010000000000001,"15min":1.542},"min":{"1min":0.79600000000000004,"5min":0.70599999999999996,"15min":0.497},"max":{"1min":1.7549999999999999,"5min":19.456,"15min":19.456},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.619,"15min":1.665},"min":{"1min":0.76600000000000001,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.859,"5min":19.925999999999998,"15min":22.204000000000001},"last":1.3320000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6279999999999999,"15min":1.6830000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":1.8169999999999999,"5min":20.012,"15min":20.012},"last":1.47},{"interface":"front","average":{"1min":1.367,"5min":1.464,"15min":1.6240000000000001},"min":{"1min":0.71599999999999997,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":1.774,"5min":2.6190000000000002,"15min":19.998999999999999},"last":1.377}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.7669999999999999,"15min":1.6910000000000001},"min":{"1min":0.95799999999999996,"5min":0.70799999999999996,"15min":0.54100000000000004},"max":{"1min":1.8120000000000001,"5min":19.526,"15min":19.526},"last":1.508},{"interface":"front","average":{"1min":1.48,"5min":1.8520000000000001,"15min":1.8200000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":1.9550000000000001,"5min":20.064,"15min":27.579999999999998},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.8460000000000001,"15min":1.8120000000000001},"min":{"1min":0.81699999999999995,"5min":0.63,"15min":0.505},"max":{"1min":1.6930000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":2.0139999999999998,"15min":1.8240000000000001},"min":{"1min":0.94099999999999995,"5min":0.76100000000000001,"15min":0.54500000000000004},"max":{"1min":1.8009999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.2450000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542304,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68864,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302976,"statfs":{"total":96636764160,"available":96566247424,"internally_reserved":0,"allocated":1687552,"data_stored":1062300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.1020000000000001,"15min":1.1180000000000001},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5620000000000001,"5min":20.437999999999999,"15min":20.437999999999999},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1020000000000001,"15min":1.2030000000000001},"min":{"1min":0.61899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4370000000000001,"5min":3.492,"15min":35.137999999999998},"last":1.3220000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.375,"15min":1.278},"min":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.101,"5min":20.556000000000001,"15min":20.556000000000001},"last":1.29},{"interface":"front","average":{"1min":1.079,"5min":1.256,"15min":1.3640000000000001},"min":{"1min":0.44400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7350000000000001,"5min":4.7430000000000003,"15min":35.183},"last":1.0820000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.143,"15min":1.1379999999999999},"min":{"1min":0.52800000000000002,"5min":0.46000000000000002,"15min":0.41699999999999998},"max":{"1min":1.3149999999999999,"5min":5.8499999999999996,"15min":8.9949999999999992},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5209999999999999,"15min":1.569},"min":{"1min":0.66700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.4119999999999999,"5min":20.670000000000002,"15min":25.372},"last":1.363}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.514,"15min":1.6759999999999999},"min":{"1min":0.98099999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7310000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.232},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.383,"15min":1.452},"min":{"1min":0.81200000000000006,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6739999999999999,"5min":2.5049999999999999,"15min":15.968999999999999},"last":1.7809999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.7050000000000001,"15min":1.6080000000000001},"min":{"1min":0.78600000000000003,"5min":0.69799999999999995,"15min":0.63800000000000001},"max":{"1min":1.8,"5min":12.173999999999999,"15min":12.173999999999999},"last":2.2010000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.964,"15min":1.786},"min":{"1min":1.121,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":20.789999999999999,"15min":20.789999999999999},"last":2.1629999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.109,"15min":1.972},"min":{"1min":1.0109999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.74,"5min":20.893000000000001,"15min":36.390999999999998},"last":1.8779999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.726,"15min":1.9099999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.8919999999999999,"5min":8.3829999999999991,"15min":36.207000000000001},"last":2.2400000000000002}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.986,"15min":1.9339999999999999},"min":{"1min":0.94899999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.0049999999999999,"5min":24.556999999999999,"15min":35.636000000000003},"last":2.0550000000000002},{"interface":"front","average":{"1min":1.429,"5min":1.768,"15min":1.649},"min":{"1min":0.91900000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.8300000000000001,"5min":11.470000000000001,"15min":11.470000000000001},"last":1.722}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:10:39.414 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:10:39.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.641+0000 7f1ef2400640 1 Processor -- start 2024-01-15T18:10:39.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.641+0000 7f1ef2400640 1 -- start start 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.642+0000 7f1ef2400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1eec15d670 0x7f1eec0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.642+0000 7f1ef2400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1eec0ce2b0 0x7f1eec0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.642+0000 7f1ef2400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec0ce930 con 0x7f1eec0d2f10 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.642+0000 7f1ef2400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec0ce740 con 0x7f1eec15d670 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.642+0000 7f1ef2400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec0cec10 con 0x7f1eec0ce2b0 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1eeb400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1eec0d2f10 0x7f1eec0d32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44338/0 (socket says 172.21.15.2:44338) 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1eeb400640 1 -- 172.21.15.2:0/928557071 learned_addr learned my addr 172.21.15.2:0/928557071 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:39.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1ee3e00640 1 --2- 172.21.15.2:0/928557071 >> v2:172.21.15.2:3300/0 conn(0x7f1eec15d670 0x7f1eec0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1eeaa00640 1 -- 172.21.15.2:0/928557071 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783437960 0 0) 0x7f1eec0ce930 con 0x7f1eec0d2f10 2024-01-15T18:10:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1eeaa00640 1 -- 172.21.15.2:0/928557071 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ecc003520 con 0x7f1eec0d2f10 2024-01-15T18:10:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1eebe00640 1 --2- 172.21.15.2:0/928557071 >> v2:172.21.15.137:3300/0 conn(0x7f1eec0ce2b0 0x7f1eec0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.643+0000 7f1ee3e00640 1 -- 172.21.15.2:0/928557071 >> v2:172.21.15.137:3300/0 conn(0x7f1eec0ce2b0 msgr2=0x7f1eec0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:39.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.644+0000 7f1ee3e00640 1 --2- 172.21.15.2:0/928557071 >> v2:172.21.15.137:3300/0 conn(0x7f1eec0ce2b0 0x7f1eec0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.644+0000 7f1ee3e00640 1 -- 172.21.15.2:0/928557071 >> v1:172.21.15.2:6789/0 conn(0x7f1eec0d2f10 legacy=0x7f1eec0d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:39.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.644+0000 7f1ee3e00640 1 -- 172.21.15.2:0/928557071 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec0ceef0 con 0x7f1eec15d670 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.645+0000 7f1ee3e00640 1 --2- 172.21.15.2:0/928557071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec15d670 0x7f1eec0cdd70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0cd840 tx=0x7f1ed402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893ff66967978f53 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.645+0000 7f1eeaa00640 1 -- 172.21.15.2:0/928557071 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4031830 con 0x7f1eec15d670 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.645+0000 7f1eeaa00640 1 -- 172.21.15.2:0/928557071 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed4031990 con 0x7f1eec15d670 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.645+0000 7f1eeaa00640 1 -- 172.21.15.2:0/928557071 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4031d20 con 0x7f1eec15d670 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 -- 172.21.15.2:0/928557071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec15d670 msgr2=0x7f1eec0cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 --2- 172.21.15.2:0/928557071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec15d670 0x7f1eec0cdd70 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0cd840 tx=0x7f1ed402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 -- 172.21.15.2:0/928557071 shutdown_connections 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 --2- 172.21.15.2:0/928557071 >> v2:172.21.15.137:3300/0 conn(0x7f1eec0ce2b0 0x7f1eec0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 --2- 172.21.15.2:0/928557071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec15d670 0x7f1eec0cdd70 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 -- 172.21.15.2:0/928557071 >> 172.21.15.2:0/928557071 conn(0x7f1eec08cc30 msgr2=0x7f1eec09c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 -- 172.21.15.2:0/928557071 shutdown_connections 2024-01-15T18:10:39.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.646+0000 7f1ef2400640 1 -- 172.21.15.2:0/928557071 wait complete. 2024-01-15T18:10:39.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.648+0000 7f1ef2400640 1 Processor -- start 2024-01-15T18:10:39.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.648+0000 7f1ef2400640 1 -- start start 2024-01-15T18:10:39.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ef2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eec0ce2b0 0x7f1eec1f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:39.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ef2400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 0x7f1eec15bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:39.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ef2400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eec15a5c0 0x7f1eec0cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:39.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ef2400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec0ca960 con 0x7f1eec0d2f10 2024-01-15T18:10:39.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ef2400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec15b6d0 con 0x7f1eec15a5c0 2024-01-15T18:10:39.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ef2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eec1f3cd0 con 0x7f1eec0ce2b0 2024-01-15T18:10:39.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ee3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 0x7f1eec15bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:39.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ee3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 0x7f1eec15bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55790/0 (socket says 172.21.15.2:55790) 2024-01-15T18:10:39.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.649+0000 7f1ee3e00640 1 -- 172.21.15.2:0/2263074159 learned_addr learned my addr 172.21.15.2:0/2263074159 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:39.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1eebe00640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eec15a5c0 0x7f1eec0cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:39.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1eeb400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eec0ce2b0 0x7f1eec1f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:39.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3e00640 1 -- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eec0ce2b0 msgr2=0x7f1eec1f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:39.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3e00640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eec0ce2b0 0x7f1eec1f3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3e00640 1 -- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eec15a5c0 msgr2=0x7f1eec0cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3e00640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eec15a5c0 0x7f1eec0cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3e00640 1 -- 172.21.15.2:0/2263074159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed4009400 con 0x7f1eec0d2f10 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1eebe00640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eec15a5c0 0x7f1eec0cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3e00640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 0x7f1eec15bb60 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4000c00 tx=0x7f1ed40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed403a830 con 0x7f1eec0d2f10 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed403a990 con 0x7f1eec0d2f10 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec15c0a0 con 0x7f1eec0d2f10 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.650+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed403ad40 con 0x7f1eec0d2f10 2024-01-15T18:10:39.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.651+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec15ada0 con 0x7f1eec0d2f10 2024-01-15T18:10:39.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.652+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec0686f0 con 0x7f1eec0d2f10 2024-01-15T18:10:39.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.653+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1ed404d450 con 0x7f1eec0d2f10 2024-01-15T18:10:39.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.653+0000 7f1ee3400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ec406bfe0 0x7f1ec406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:39.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.654+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1ed40aa730 con 0x7f1eec0d2f10 2024-01-15T18:10:39.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.654+0000 7f1eeb400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ec406bfe0 0x7f1ec406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:39.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.656+0000 7f1eeb400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ec406bfe0 0x7f1ec406e4a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1edc005880 tx=0x7f1edc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:39.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.658+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ed4076f30 con 0x7f1eec0d2f10 2024-01-15T18:10:39.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.850+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1eec0d1070 con 0x7f1ec406bfe0 2024-01-15T18:10:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.873+0000 7f1ee3400640 1 -- 172.21.15.2:0/2263074159 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286629 (secure 0 0 0) 0x7f1edc001c60 con 0x7f1ec406bfe0 2024-01-15T18:10:39.880 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:10:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ec406bfe0 msgr2=0x7f1ec406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ec406bfe0 0x7f1ec406e4a0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1edc005880 tx=0x7f1edc009040 comp rx=0 tx=0).stop 2024-01-15T18:10:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 msgr2=0x7f1eec15bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 0x7f1eec15bb60 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4000c00 tx=0x7f1ed40046c0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 shutdown_connections 2024-01-15T18:10:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ec406bfe0 0x7f1ec406e4a0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eec15a5c0 0x7f1eec0cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eec0d2f10 0x7f1eec15bb60 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 --2- 172.21.15.2:0/2263074159 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eec0ce2b0 0x7f1eec1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 >> 172.21.15.2:0/2263074159 conn(0x7f1eec08cc30 msgr2=0x7f1eec09c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 shutdown_connections 2024-01-15T18:10:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:39.878+0000 7f1ef2400640 1 -- 172.21.15.2:0/2263074159 wait complete. 2024-01-15T18:10:39.900 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-01-15T18:10:39.715665+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":522620,"kb_used_data":9736,"kb_used_omap":12,"kb_used_meta":512755,"kb_avail":754452100,"statfs":{"total":773094113280,"available":772558950400,"internally_reserved":0,"allocated":9969664,"data_stored":4960740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":525061681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.858117"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086275+0000","last_change":"2024-01-15T18:10:29.889349+0000","last_active":"2024-01-15T18:10:31.086275+0000","last_peered":"2024-01-15T18:10:31.086275+0000","last_clean":"2024-01-15T18:10:31.086275+0000","last_became_active":"2024-01-15T18:10:29.889079+0000","last_became_peered":"2024-01-15T18:10:29.889079+0000","last_unstale":"2024-01-15T18:10:31.086275+0000","last_undegraded":"2024-01-15T18:10:31.086275+0000","last_fullsized":"2024-01-15T18:10:31.086275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:32.626031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087942+0000","last_change":"2024-01-15T18:10:29.887330+0000","last_active":"2024-01-15T18:10:31.087942+0000","last_peered":"2024-01-15T18:10:31.087942+0000","last_clean":"2024-01-15T18:10:31.087942+0000","last_became_active":"2024-01-15T18:10:29.887024+0000","last_became_peered":"2024-01-15T18:10:29.887024+0000","last_unstale":"2024-01-15T18:10:31.087942+0000","last_undegraded":"2024-01-15T18:10:31.087942+0000","last_fullsized":"2024-01-15T18:10:31.087942+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:52.419431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085792+0000","last_change":"2024-01-15T18:10:29.888180+0000","last_active":"2024-01-15T18:10:31.085792+0000","last_peered":"2024-01-15T18:10:31.085792+0000","last_clean":"2024-01-15T18:10:31.085792+0000","last_became_active":"2024-01-15T18:10:29.886925+0000","last_became_peered":"2024-01-15T18:10:29.886925+0000","last_unstale":"2024-01-15T18:10:31.085792+0000","last_undegraded":"2024-01-15T18:10:31.085792+0000","last_fullsized":"2024-01-15T18:10:31.085792+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:10.571774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884428+0000","last_change":"2024-01-15T18:10:29.883489+0000","last_active":"2024-01-15T18:10:30.884428+0000","last_peered":"2024-01-15T18:10:30.884428+0000","last_clean":"2024-01-15T18:10:30.884428+0000","last_became_active":"2024-01-15T18:10:29.883195+0000","last_became_peered":"2024-01-15T18:10:29.883195+0000","last_unstale":"2024-01-15T18:10:30.884428+0000","last_undegraded":"2024-01-15T18:10:30.884428+0000","last_fullsized":"2024-01-15T18:10:30.884428+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:24:48.158391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884370+0000","last_change":"2024-01-15T18:10:29.892558+0000","last_active":"2024-01-15T18:10:30.884370+0000","last_peered":"2024-01-15T18:10:30.884370+0000","last_clean":"2024-01-15T18:10:30.884370+0000","last_became_active":"2024-01-15T18:10:29.891669+0000","last_became_peered":"2024-01-15T18:10:29.891669+0000","last_unstale":"2024-01-15T18:10:30.884370+0000","last_undegraded":"2024-01-15T18:10:30.884370+0000","last_fullsized":"2024-01-15T18:10:30.884370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:59:55.883984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886845+0000","last_change":"2024-01-15T18:10:29.882639+0000","last_active":"2024-01-15T18:10:30.886845+0000","last_peered":"2024-01-15T18:10:30.886845+0000","last_clean":"2024-01-15T18:10:30.886845+0000","last_became_active":"2024-01-15T18:10:29.882359+0000","last_became_peered":"2024-01-15T18:10:29.882359+0000","last_unstale":"2024-01-15T18:10:30.886845+0000","last_undegraded":"2024-01-15T18:10:30.886845+0000","last_fullsized":"2024-01-15T18:10:30.886845+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:25.828362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883604+0000","last_change":"2024-01-15T18:10:29.887844+0000","last_active":"2024-01-15T18:10:30.883604+0000","last_peered":"2024-01-15T18:10:30.883604+0000","last_clean":"2024-01-15T18:10:30.883604+0000","last_became_active":"2024-01-15T18:10:29.887501+0000","last_became_peered":"2024-01-15T18:10:29.887501+0000","last_unstale":"2024-01-15T18:10:30.883604+0000","last_undegraded":"2024-01-15T18:10:30.883604+0000","last_fullsized":"2024-01-15T18:10:30.883604+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:03:05.714901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886761+0000","last_change":"2024-01-15T18:10:29.876459+0000","last_active":"2024-01-15T18:10:30.886761+0000","last_peered":"2024-01-15T18:10:30.886761+0000","last_clean":"2024-01-15T18:10:30.886761+0000","last_became_active":"2024-01-15T18:10:29.876059+0000","last_became_peered":"2024-01-15T18:10:29.876059+0000","last_unstale":"2024-01-15T18:10:30.886761+0000","last_undegraded":"2024-01-15T18:10:30.886761+0000","last_fullsized":"2024-01-15T18:10:30.886761+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:48.870265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887915+0000","last_change":"2024-01-15T18:10:29.885819+0000","last_active":"2024-01-15T18:10:30.887915+0000","last_peered":"2024-01-15T18:10:30.887915+0000","last_clean":"2024-01-15T18:10:30.887915+0000","last_became_active":"2024-01-15T18:10:29.884423+0000","last_became_peered":"2024-01-15T18:10:29.884423+0000","last_unstale":"2024-01-15T18:10:30.887915+0000","last_undegraded":"2024-01-15T18:10:30.887915+0000","last_fullsized":"2024-01-15T18:10:30.887915+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085443+0000","last_change":"2024-01-15T18:10:29.886368+0000","last_active":"2024-01-15T18:10:31.085443+0000","last_peered":"2024-01-15T18:10:31.085443+0000","last_clean":"2024-01-15T18:10:31.085443+0000","last_became_active":"2024-01-15T18:10:29.885765+0000","last_became_peered":"2024-01-15T18:10:29.885765+0000","last_unstale":"2024-01-15T18:10:31.085443+0000","last_undegraded":"2024-01-15T18:10:31.085443+0000","last_fullsized":"2024-01-15T18:10:31.085443+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:09.028738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087754+0000","last_change":"2024-01-15T18:10:29.879828+0000","last_active":"2024-01-15T18:10:31.087754+0000","last_peered":"2024-01-15T18:10:31.087754+0000","last_clean":"2024-01-15T18:10:31.087754+0000","last_became_active":"2024-01-15T18:10:29.879547+0000","last_became_peered":"2024-01-15T18:10:29.879547+0000","last_unstale":"2024-01-15T18:10:31.087754+0000","last_undegraded":"2024-01-15T18:10:31.087754+0000","last_fullsized":"2024-01-15T18:10:31.087754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:49:21.093750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084274+0000","last_change":"2024-01-15T18:10:29.887061+0000","last_active":"2024-01-15T18:10:31.084274+0000","last_peered":"2024-01-15T18:10:31.084274+0000","last_clean":"2024-01-15T18:10:31.084274+0000","last_became_active":"2024-01-15T18:10:29.885624+0000","last_became_peered":"2024-01-15T18:10:29.885624+0000","last_unstale":"2024-01-15T18:10:31.084274+0000","last_undegraded":"2024-01-15T18:10:31.084274+0000","last_fullsized":"2024-01-15T18:10:31.084274+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084754+0000","last_change":"2024-01-15T18:10:29.884896+0000","last_active":"2024-01-15T18:10:31.084754+0000","last_peered":"2024-01-15T18:10:31.084754+0000","last_clean":"2024-01-15T18:10:31.084754+0000","last_became_active":"2024-01-15T18:10:29.883901+0000","last_became_peered":"2024-01-15T18:10:29.883901+0000","last_unstale":"2024-01-15T18:10:31.084754+0000","last_undegraded":"2024-01-15T18:10:31.084754+0000","last_fullsized":"2024-01-15T18:10:31.084754+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:32.626031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883780+0000","last_change":"2024-01-15T18:10:29.884516+0000","last_active":"2024-01-15T18:10:30.883780+0000","last_peered":"2024-01-15T18:10:30.883780+0000","last_clean":"2024-01-15T18:10:30.883780+0000","last_became_active":"2024-01-15T18:10:29.883939+0000","last_became_peered":"2024-01-15T18:10:29.883939+0000","last_unstale":"2024-01-15T18:10:30.883780+0000","last_undegraded":"2024-01-15T18:10:30.883780+0000","last_fullsized":"2024-01-15T18:10:30.883780+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:30:18.904062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.083431+0000","last_change":"2024-01-15T18:10:29.871576+0000","last_active":"2024-01-15T18:10:31.083431+0000","last_peered":"2024-01-15T18:10:31.083431+0000","last_clean":"2024-01-15T18:10:31.083431+0000","last_became_active":"2024-01-15T18:10:29.870742+0000","last_became_peered":"2024-01-15T18:10:29.870742+0000","last_unstale":"2024-01-15T18:10:31.083431+0000","last_undegraded":"2024-01-15T18:10:31.083431+0000","last_fullsized":"2024-01-15T18:10:31.083431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:42:10.571774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883712+0000","last_change":"2024-01-15T18:10:29.890607+0000","last_active":"2024-01-15T18:10:30.883712+0000","last_peered":"2024-01-15T18:10:30.883712+0000","last_clean":"2024-01-15T18:10:30.883712+0000","last_became_active":"2024-01-15T18:10:29.889442+0000","last_became_peered":"2024-01-15T18:10:29.889442+0000","last_unstale":"2024-01-15T18:10:30.883712+0000","last_undegraded":"2024-01-15T18:10:30.883712+0000","last_fullsized":"2024-01-15T18:10:30.883712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:58:59.655054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885709+0000","last_change":"2024-01-15T18:10:29.893877+0000","last_active":"2024-01-15T18:10:30.885709+0000","last_peered":"2024-01-15T18:10:30.885709+0000","last_clean":"2024-01-15T18:10:30.885709+0000","last_became_active":"2024-01-15T18:10:29.893574+0000","last_became_peered":"2024-01-15T18:10:29.893574+0000","last_unstale":"2024-01-15T18:10:30.885709+0000","last_undegraded":"2024-01-15T18:10:30.885709+0000","last_fullsized":"2024-01-15T18:10:30.885709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:53:49.239687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884812+0000","last_change":"2024-01-15T18:10:29.888339+0000","last_active":"2024-01-15T18:10:30.884812+0000","last_peered":"2024-01-15T18:10:30.884812+0000","last_clean":"2024-01-15T18:10:30.884812+0000","last_became_active":"2024-01-15T18:10:29.888053+0000","last_became_peered":"2024-01-15T18:10:29.888053+0000","last_unstale":"2024-01-15T18:10:30.884812+0000","last_undegraded":"2024-01-15T18:10:30.884812+0000","last_fullsized":"2024-01-15T18:10:30.884812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:21:03.349644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085910+0000","last_change":"2024-01-15T18:10:29.890238+0000","last_active":"2024-01-15T18:10:31.085910+0000","last_peered":"2024-01-15T18:10:31.085910+0000","last_clean":"2024-01-15T18:10:31.085910+0000","last_became_active":"2024-01-15T18:10:29.888327+0000","last_became_peered":"2024-01-15T18:10:29.888327+0000","last_unstale":"2024-01-15T18:10:31.085910+0000","last_undegraded":"2024-01-15T18:10:31.085910+0000","last_fullsized":"2024-01-15T18:10:31.085910+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:36:32.626031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884433+0000","last_change":"2024-01-15T18:10:29.880093+0000","last_active":"2024-01-15T18:10:30.884433+0000","last_peered":"2024-01-15T18:10:30.884433+0000","last_clean":"2024-01-15T18:10:30.884433+0000","last_became_active":"2024-01-15T18:10:29.879766+0000","last_became_peered":"2024-01-15T18:10:29.879766+0000","last_unstale":"2024-01-15T18:10:30.884433+0000","last_undegraded":"2024-01-15T18:10:30.884433+0000","last_fullsized":"2024-01-15T18:10:30.884433+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:46:25.456136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884248+0000","last_change":"2024-01-15T18:10:29.878993+0000","last_active":"2024-01-15T18:10:30.884248+0000","last_peered":"2024-01-15T18:10:30.884248+0000","last_clean":"2024-01-15T18:10:30.884248+0000","last_became_active":"2024-01-15T18:10:29.878223+0000","last_became_peered":"2024-01-15T18:10:29.878223+0000","last_unstale":"2024-01-15T18:10:30.884248+0000","last_undegraded":"2024-01-15T18:10:30.884248+0000","last_fullsized":"2024-01-15T18:10:30.884248+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:34.700358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086577+0000","last_change":"2024-01-15T18:10:29.878232+0000","last_active":"2024-01-15T18:10:31.086577+0000","last_peered":"2024-01-15T18:10:31.086577+0000","last_clean":"2024-01-15T18:10:31.086577+0000","last_became_active":"2024-01-15T18:10:29.877216+0000","last_became_peered":"2024-01-15T18:10:29.877216+0000","last_unstale":"2024-01-15T18:10:31.086577+0000","last_undegraded":"2024-01-15T18:10:31.086577+0000","last_fullsized":"2024-01-15T18:10:31.086577+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:19:33.223505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084033+0000","last_change":"2024-01-15T18:10:29.894531+0000","last_active":"2024-01-15T18:10:31.084033+0000","last_peered":"2024-01-15T18:10:31.084033+0000","last_clean":"2024-01-15T18:10:31.084033+0000","last_became_active":"2024-01-15T18:10:29.894255+0000","last_became_peered":"2024-01-15T18:10:29.894255+0000","last_unstale":"2024-01-15T18:10:31.084033+0000","last_undegraded":"2024-01-15T18:10:31.084033+0000","last_fullsized":"2024-01-15T18:10:31.084033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883469+0000","last_change":"2024-01-15T18:10:29.875269+0000","last_active":"2024-01-15T18:10:30.883469+0000","last_peered":"2024-01-15T18:10:30.883469+0000","last_clean":"2024-01-15T18:10:30.883469+0000","last_became_active":"2024-01-15T18:10:29.874953+0000","last_became_peered":"2024-01-15T18:10:29.874953+0000","last_unstale":"2024-01-15T18:10:30.883469+0000","last_undegraded":"2024-01-15T18:10:30.883469+0000","last_fullsized":"2024-01-15T18:10:30.883469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:40:45.345681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087649+0000","last_change":"2024-01-15T18:10:29.887757+0000","last_active":"2024-01-15T18:10:31.087649+0000","last_peered":"2024-01-15T18:10:31.087649+0000","last_clean":"2024-01-15T18:10:31.087649+0000","last_became_active":"2024-01-15T18:10:29.887342+0000","last_became_peered":"2024-01-15T18:10:29.887342+0000","last_unstale":"2024-01-15T18:10:31.087649+0000","last_undegraded":"2024-01-15T18:10:31.087649+0000","last_fullsized":"2024-01-15T18:10:31.087649+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:07:05.708131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085556+0000","last_change":"2024-01-15T18:10:29.879335+0000","last_active":"2024-01-15T18:10:31.085556+0000","last_peered":"2024-01-15T18:10:31.085556+0000","last_clean":"2024-01-15T18:10:31.085556+0000","last_became_active":"2024-01-15T18:10:29.878999+0000","last_became_peered":"2024-01-15T18:10:29.878999+0000","last_unstale":"2024-01-15T18:10:31.085556+0000","last_undegraded":"2024-01-15T18:10:31.085556+0000","last_fullsized":"2024-01-15T18:10:31.085556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:12:20.074528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884895+0000","last_change":"2024-01-15T18:10:29.885016+0000","last_active":"2024-01-15T18:10:30.884895+0000","last_peered":"2024-01-15T18:10:30.884895+0000","last_clean":"2024-01-15T18:10:30.884895+0000","last_became_active":"2024-01-15T18:10:29.884752+0000","last_became_peered":"2024-01-15T18:10:29.884752+0000","last_unstale":"2024-01-15T18:10:30.884895+0000","last_undegraded":"2024-01-15T18:10:30.884895+0000","last_fullsized":"2024-01-15T18:10:30.884895+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:27:52.380249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084315+0000","last_change":"2024-01-15T18:10:29.893008+0000","last_active":"2024-01-15T18:10:31.084315+0000","last_peered":"2024-01-15T18:10:31.084315+0000","last_clean":"2024-01-15T18:10:31.084315+0000","last_became_active":"2024-01-15T18:10:29.892710+0000","last_became_peered":"2024-01-15T18:10:29.892710+0000","last_unstale":"2024-01-15T18:10:31.084315+0000","last_undegraded":"2024-01-15T18:10:31.084315+0000","last_fullsized":"2024-01-15T18:10:31.084315+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:10.131197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086963+0000","last_change":"2024-01-15T18:10:29.881928+0000","last_active":"2024-01-15T18:10:31.086963+0000","last_peered":"2024-01-15T18:10:31.086963+0000","last_clean":"2024-01-15T18:10:31.086963+0000","last_became_active":"2024-01-15T18:10:29.879237+0000","last_became_peered":"2024-01-15T18:10:29.879237+0000","last_unstale":"2024-01-15T18:10:31.086963+0000","last_undegraded":"2024-01-15T18:10:31.086963+0000","last_fullsized":"2024-01-15T18:10:31.086963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:37.601472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086452+0000","last_change":"2024-01-15T18:10:29.878883+0000","last_active":"2024-01-15T18:10:31.086452+0000","last_peered":"2024-01-15T18:10:31.086452+0000","last_clean":"2024-01-15T18:10:31.086452+0000","last_became_active":"2024-01-15T18:10:29.878568+0000","last_became_peered":"2024-01-15T18:10:29.878568+0000","last_unstale":"2024-01-15T18:10:31.086452+0000","last_undegraded":"2024-01-15T18:10:31.086452+0000","last_fullsized":"2024-01-15T18:10:31.086452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:48.891903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884305+0000","last_change":"2024-01-15T18:10:29.890635+0000","last_active":"2024-01-15T18:10:30.884305+0000","last_peered":"2024-01-15T18:10:30.884305+0000","last_clean":"2024-01-15T18:10:30.884305+0000","last_became_active":"2024-01-15T18:10:29.888122+0000","last_became_peered":"2024-01-15T18:10:29.888122+0000","last_unstale":"2024-01-15T18:10:30.884305+0000","last_undegraded":"2024-01-15T18:10:30.884305+0000","last_fullsized":"2024-01-15T18:10:30.884305+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:32:10.984586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084512+0000","last_change":"2024-01-15T18:10:29.891784+0000","last_active":"2024-01-15T18:10:31.084512+0000","last_peered":"2024-01-15T18:10:31.084512+0000","last_clean":"2024-01-15T18:10:31.084512+0000","last_became_active":"2024-01-15T18:10:29.891459+0000","last_became_peered":"2024-01-15T18:10:29.891459+0000","last_unstale":"2024-01-15T18:10:31.084512+0000","last_undegraded":"2024-01-15T18:10:31.084512+0000","last_fullsized":"2024-01-15T18:10:31.084512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:18.035419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885422+0000","last_change":"2024-01-15T18:10:29.888437+0000","last_active":"2024-01-15T18:10:30.885422+0000","last_peered":"2024-01-15T18:10:30.885422+0000","last_clean":"2024-01-15T18:10:30.885422+0000","last_became_active":"2024-01-15T18:10:29.888167+0000","last_became_peered":"2024-01-15T18:10:29.888167+0000","last_unstale":"2024-01-15T18:10:30.885422+0000","last_undegraded":"2024-01-15T18:10:30.885422+0000","last_fullsized":"2024-01-15T18:10:30.885422+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:54:18.336566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"88'4","reported_seq":16,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.904381+0000","last_change":"2024-01-15T18:10:28.860747+0000","last_active":"2024-01-15T18:10:29.904381+0000","last_peered":"2024-01-15T18:10:29.904381+0000","last_clean":"2024-01-15T18:10:29.904381+0000","last_became_active":"2024-01-15T18:10:28.860414+0000","last_became_peered":"2024-01-15T18:10:28.860414+0000","last_unstale":"2024-01-15T18:10:29.904381+0000","last_undegraded":"2024-01-15T18:10:29.904381+0000","last_fullsized":"2024-01-15T18:10:29.904381+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:31.063023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085936+0000","last_change":"2024-01-15T18:10:29.896067+0000","last_active":"2024-01-15T18:10:31.085936+0000","last_peered":"2024-01-15T18:10:31.085936+0000","last_clean":"2024-01-15T18:10:31.085936+0000","last_became_active":"2024-01-15T18:10:29.895466+0000","last_became_peered":"2024-01-15T18:10:29.895466+0000","last_unstale":"2024-01-15T18:10:31.085936+0000","last_undegraded":"2024-01-15T18:10:31.085936+0000","last_fullsized":"2024-01-15T18:10:31.085936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:18.991298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.899240+0000","last_change":"2024-01-15T18:10:28.861707+0000","last_active":"2024-01-15T18:10:29.899240+0000","last_peered":"2024-01-15T18:10:29.899240+0000","last_clean":"2024-01-15T18:10:29.899240+0000","last_became_active":"2024-01-15T18:10:28.860381+0000","last_became_peered":"2024-01-15T18:10:28.860381+0000","last_unstale":"2024-01-15T18:10:29.899240+0000","last_undegraded":"2024-01-15T18:10:29.899240+0000","last_fullsized":"2024-01-15T18:10:29.899240+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:49:28.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085899+0000","last_change":"2024-01-15T18:10:29.893303+0000","last_active":"2024-01-15T18:10:31.085899+0000","last_peered":"2024-01-15T18:10:31.085899+0000","last_clean":"2024-01-15T18:10:31.085899+0000","last_became_active":"2024-01-15T18:10:29.891192+0000","last_became_peered":"2024-01-15T18:10:29.891192+0000","last_unstale":"2024-01-15T18:10:31.085899+0000","last_undegraded":"2024-01-15T18:10:31.085899+0000","last_fullsized":"2024-01-15T18:10:31.085899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:54.921393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.903961+0000","last_change":"2024-01-15T18:10:28.851518+0000","last_active":"2024-01-15T18:10:29.903961+0000","last_peered":"2024-01-15T18:10:29.903961+0000","last_clean":"2024-01-15T18:10:29.903961+0000","last_became_active":"2024-01-15T18:10:28.851216+0000","last_became_peered":"2024-01-15T18:10:28.851216+0000","last_unstale":"2024-01-15T18:10:29.903961+0000","last_undegraded":"2024-01-15T18:10:29.903961+0000","last_fullsized":"2024-01-15T18:10:29.903961+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:25.704119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086539+0000","last_change":"2024-01-15T18:10:29.890269+0000","last_active":"2024-01-15T18:10:31.086539+0000","last_peered":"2024-01-15T18:10:31.086539+0000","last_clean":"2024-01-15T18:10:31.086539+0000","last_became_active":"2024-01-15T18:10:29.889995+0000","last_became_peered":"2024-01-15T18:10:29.889995+0000","last_unstale":"2024-01-15T18:10:31.086539+0000","last_undegraded":"2024-01-15T18:10:31.086539+0000","last_fullsized":"2024-01-15T18:10:31.086539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:27:49.720321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.903898+0000","last_change":"2024-01-15T18:10:28.847695+0000","last_active":"2024-01-15T18:10:29.903898+0000","last_peered":"2024-01-15T18:10:29.903898+0000","last_clean":"2024-01-15T18:10:29.903898+0000","last_became_active":"2024-01-15T18:10:28.847366+0000","last_became_peered":"2024-01-15T18:10:28.847366+0000","last_unstale":"2024-01-15T18:10:29.903898+0000","last_undegraded":"2024-01-15T18:10:29.903898+0000","last_fullsized":"2024-01-15T18:10:29.903898+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:09.115631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086618+0000","last_change":"2024-01-15T18:10:29.875494+0000","last_active":"2024-01-15T18:10:31.086618+0000","last_peered":"2024-01-15T18:10:31.086618+0000","last_clean":"2024-01-15T18:10:31.086618+0000","last_became_active":"2024-01-15T18:10:29.875157+0000","last_became_peered":"2024-01-15T18:10:29.875157+0000","last_unstale":"2024-01-15T18:10:31.086618+0000","last_undegraded":"2024-01-15T18:10:31.086618+0000","last_fullsized":"2024-01-15T18:10:31.086618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:11:15.629310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.901308+0000","last_change":"2024-01-15T18:10:28.860229+0000","last_active":"2024-01-15T18:10:29.901308+0000","last_peered":"2024-01-15T18:10:29.901308+0000","last_clean":"2024-01-15T18:10:29.901308+0000","last_became_active":"2024-01-15T18:10:28.859548+0000","last_became_peered":"2024-01-15T18:10:28.859548+0000","last_unstale":"2024-01-15T18:10:29.901308+0000","last_undegraded":"2024-01-15T18:10:29.901308+0000","last_fullsized":"2024-01-15T18:10:29.901308+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:24:10.860846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884423+0000","last_change":"2024-01-15T18:10:29.893025+0000","last_active":"2024-01-15T18:10:30.884423+0000","last_peered":"2024-01-15T18:10:30.884423+0000","last_clean":"2024-01-15T18:10:30.884423+0000","last_became_active":"2024-01-15T18:10:29.892694+0000","last_became_peered":"2024-01-15T18:10:29.892694+0000","last_unstale":"2024-01-15T18:10:30.884423+0000","last_undegraded":"2024-01-15T18:10:30.884423+0000","last_fullsized":"2024-01-15T18:10:30.884423+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:37.045766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.898844+0000","last_change":"2024-01-15T18:10:28.860903+0000","last_active":"2024-01-15T18:10:29.898844+0000","last_peered":"2024-01-15T18:10:29.898844+0000","last_clean":"2024-01-15T18:10:29.898844+0000","last_became_active":"2024-01-15T18:10:28.860595+0000","last_became_peered":"2024-01-15T18:10:28.860595+0000","last_unstale":"2024-01-15T18:10:29.898844+0000","last_undegraded":"2024-01-15T18:10:29.898844+0000","last_fullsized":"2024-01-15T18:10:29.898844+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:51.755398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084229+0000","last_change":"2024-01-15T18:10:29.871081+0000","last_active":"2024-01-15T18:10:31.084229+0000","last_peered":"2024-01-15T18:10:31.084229+0000","last_clean":"2024-01-15T18:10:31.084229+0000","last_became_active":"2024-01-15T18:10:29.870386+0000","last_became_peered":"2024-01-15T18:10:29.870386+0000","last_unstale":"2024-01-15T18:10:31.084229+0000","last_undegraded":"2024-01-15T18:10:31.084229+0000","last_fullsized":"2024-01-15T18:10:31.084229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:54.921393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.868000+0000","last_change":"2024-01-15T18:10:28.856007+0000","last_active":"2024-01-15T18:10:29.868000+0000","last_peered":"2024-01-15T18:10:29.868000+0000","last_clean":"2024-01-15T18:10:29.868000+0000","last_became_active":"2024-01-15T18:10:28.855729+0000","last_became_peered":"2024-01-15T18:10:28.855729+0000","last_unstale":"2024-01-15T18:10:29.868000+0000","last_undegraded":"2024-01-15T18:10:29.868000+0000","last_fullsized":"2024-01-15T18:10:29.868000+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:24.812796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.083675+0000","last_change":"2024-01-15T18:10:29.871794+0000","last_active":"2024-01-15T18:10:31.083675+0000","last_peered":"2024-01-15T18:10:31.083675+0000","last_clean":"2024-01-15T18:10:31.083675+0000","last_became_active":"2024-01-15T18:10:29.871250+0000","last_became_peered":"2024-01-15T18:10:29.871250+0000","last_unstale":"2024-01-15T18:10:31.083675+0000","last_undegraded":"2024-01-15T18:10:31.083675+0000","last_fullsized":"2024-01-15T18:10:31.083675+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:07:43.534696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.869439+0000","last_change":"2024-01-15T18:10:28.854028+0000","last_active":"2024-01-15T18:10:29.869439+0000","last_peered":"2024-01-15T18:10:29.869439+0000","last_clean":"2024-01-15T18:10:29.869439+0000","last_became_active":"2024-01-15T18:10:28.853569+0000","last_became_peered":"2024-01-15T18:10:28.853569+0000","last_unstale":"2024-01-15T18:10:29.869439+0000","last_undegraded":"2024-01-15T18:10:29.869439+0000","last_fullsized":"2024-01-15T18:10:29.869439+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:36.806891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085218+0000","last_change":"2024-01-15T18:10:29.878271+0000","last_active":"2024-01-15T18:10:31.085218+0000","last_peered":"2024-01-15T18:10:31.085218+0000","last_clean":"2024-01-15T18:10:31.085218+0000","last_became_active":"2024-01-15T18:10:29.877770+0000","last_became_peered":"2024-01-15T18:10:29.877770+0000","last_unstale":"2024-01-15T18:10:31.085218+0000","last_undegraded":"2024-01-15T18:10:31.085218+0000","last_fullsized":"2024-01-15T18:10:31.085218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.445206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.878955+0000","last_change":"2024-01-15T18:10:28.862577+0000","last_active":"2024-01-15T18:10:29.878955+0000","last_peered":"2024-01-15T18:10:29.878955+0000","last_clean":"2024-01-15T18:10:29.878955+0000","last_became_active":"2024-01-15T18:10:28.862234+0000","last_became_peered":"2024-01-15T18:10:28.862234+0000","last_unstale":"2024-01-15T18:10:29.878955+0000","last_undegraded":"2024-01-15T18:10:29.878955+0000","last_fullsized":"2024-01-15T18:10:29.878955+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:47.854699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887978+0000","last_change":"2024-01-15T18:10:29.888839+0000","last_active":"2024-01-15T18:10:30.887978+0000","last_peered":"2024-01-15T18:10:30.887978+0000","last_clean":"2024-01-15T18:10:30.887978+0000","last_became_active":"2024-01-15T18:10:29.888544+0000","last_became_peered":"2024-01-15T18:10:29.888544+0000","last_unstale":"2024-01-15T18:10:30.887978+0000","last_undegraded":"2024-01-15T18:10:30.887978+0000","last_fullsized":"2024-01-15T18:10:30.887978+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:41:17.940351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.875214+0000","last_change":"2024-01-15T18:10:28.859089+0000","last_active":"2024-01-15T18:10:29.875214+0000","last_peered":"2024-01-15T18:10:29.875214+0000","last_clean":"2024-01-15T18:10:29.875214+0000","last_became_active":"2024-01-15T18:10:28.858772+0000","last_became_peered":"2024-01-15T18:10:28.858772+0000","last_unstale":"2024-01-15T18:10:29.875214+0000","last_undegraded":"2024-01-15T18:10:29.875214+0000","last_fullsized":"2024-01-15T18:10:29.875214+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:27:16.178426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084403+0000","last_change":"2024-01-15T18:10:29.896917+0000","last_active":"2024-01-15T18:10:31.084403+0000","last_peered":"2024-01-15T18:10:31.084403+0000","last_clean":"2024-01-15T18:10:31.084403+0000","last_became_active":"2024-01-15T18:10:29.894525+0000","last_became_peered":"2024-01-15T18:10:29.894525+0000","last_unstale":"2024-01-15T18:10:31.084403+0000","last_undegraded":"2024-01-15T18:10:31.084403+0000","last_fullsized":"2024-01-15T18:10:31.084403+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:09.028738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"88'3","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902394+0000","last_change":"2024-01-15T18:10:28.864142+0000","last_active":"2024-01-15T18:10:29.902394+0000","last_peered":"2024-01-15T18:10:29.902394+0000","last_clean":"2024-01-15T18:10:29.902394+0000","last_became_active":"2024-01-15T18:10:28.863733+0000","last_became_peered":"2024-01-15T18:10:28.863733+0000","last_unstale":"2024-01-15T18:10:29.902394+0000","last_undegraded":"2024-01-15T18:10:29.902394+0000","last_fullsized":"2024-01-15T18:10:29.902394+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:19.865670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885210+0000","last_change":"2024-01-15T18:10:29.889650+0000","last_active":"2024-01-15T18:10:30.885210+0000","last_peered":"2024-01-15T18:10:30.885210+0000","last_clean":"2024-01-15T18:10:30.885210+0000","last_became_active":"2024-01-15T18:10:29.889309+0000","last_became_peered":"2024-01-15T18:10:29.889309+0000","last_unstale":"2024-01-15T18:10:30.885210+0000","last_undegraded":"2024-01-15T18:10:30.885210+0000","last_fullsized":"2024-01-15T18:10:30.885210+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:10:09.942053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"88'6","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.905427+0000","last_change":"2024-01-15T18:10:28.852757+0000","last_active":"2024-01-15T18:10:29.905427+0000","last_peered":"2024-01-15T18:10:29.905427+0000","last_clean":"2024-01-15T18:10:29.905427+0000","last_became_active":"2024-01-15T18:10:28.852416+0000","last_became_peered":"2024-01-15T18:10:28.852416+0000","last_unstale":"2024-01-15T18:10:29.905427+0000","last_undegraded":"2024-01-15T18:10:29.905427+0000","last_fullsized":"2024-01-15T18:10:29.905427+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:15:05.863393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085400+0000","last_change":"2024-01-15T18:10:29.892056+0000","last_active":"2024-01-15T18:10:31.085400+0000","last_peered":"2024-01-15T18:10:31.085400+0000","last_clean":"2024-01-15T18:10:31.085400+0000","last_became_active":"2024-01-15T18:10:29.889387+0000","last_became_peered":"2024-01-15T18:10:29.889387+0000","last_unstale":"2024-01-15T18:10:31.085400+0000","last_undegraded":"2024-01-15T18:10:31.085400+0000","last_fullsized":"2024-01-15T18:10:31.085400+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:18.991298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.904711+0000","last_change":"2024-01-15T18:10:28.846674+0000","last_active":"2024-01-15T18:10:29.904711+0000","last_peered":"2024-01-15T18:10:29.904711+0000","last_clean":"2024-01-15T18:10:29.904711+0000","last_became_active":"2024-01-15T18:10:28.845875+0000","last_became_peered":"2024-01-15T18:10:28.845875+0000","last_unstale":"2024-01-15T18:10:29.904711+0000","last_undegraded":"2024-01-15T18:10:29.904711+0000","last_fullsized":"2024-01-15T18:10:29.904711+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:52:23.936503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085124+0000","last_change":"2024-01-15T18:10:29.876230+0000","last_active":"2024-01-15T18:10:31.085124+0000","last_peered":"2024-01-15T18:10:31.085124+0000","last_clean":"2024-01-15T18:10:31.085124+0000","last_became_active":"2024-01-15T18:10:29.874419+0000","last_became_peered":"2024-01-15T18:10:29.874419+0000","last_unstale":"2024-01-15T18:10:31.085124+0000","last_undegraded":"2024-01-15T18:10:31.085124+0000","last_fullsized":"2024-01-15T18:10:31.085124+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:42.772403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.856671+0000","last_change":"2024-01-15T18:10:28.855952+0000","last_active":"2024-01-15T18:10:29.856671+0000","last_peered":"2024-01-15T18:10:29.856671+0000","last_clean":"2024-01-15T18:10:29.856671+0000","last_became_active":"2024-01-15T18:10:28.855647+0000","last_became_peered":"2024-01-15T18:10:28.855647+0000","last_unstale":"2024-01-15T18:10:29.856671+0000","last_undegraded":"2024-01-15T18:10:29.856671+0000","last_fullsized":"2024-01-15T18:10:29.856671+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.083977+0000","last_change":"2024-01-15T18:10:29.876118+0000","last_active":"2024-01-15T18:10:31.083977+0000","last_peered":"2024-01-15T18:10:31.083977+0000","last_clean":"2024-01-15T18:10:31.083977+0000","last_became_active":"2024-01-15T18:10:29.874531+0000","last_became_peered":"2024-01-15T18:10:29.874531+0000","last_unstale":"2024-01-15T18:10:31.083977+0000","last_undegraded":"2024-01-15T18:10:31.083977+0000","last_fullsized":"2024-01-15T18:10:31.083977+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:07:44.687374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.865704+0000","last_change":"2024-01-15T18:10:28.853310+0000","last_active":"2024-01-15T18:10:29.865704+0000","last_peered":"2024-01-15T18:10:29.865704+0000","last_clean":"2024-01-15T18:10:29.865704+0000","last_became_active":"2024-01-15T18:10:28.853030+0000","last_became_peered":"2024-01-15T18:10:28.853030+0000","last_unstale":"2024-01-15T18:10:29.865704+0000","last_undegraded":"2024-01-15T18:10:29.865704+0000","last_fullsized":"2024-01-15T18:10:29.865704+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:41:56.867321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085205+0000","last_change":"2024-01-15T18:10:29.887053+0000","last_active":"2024-01-15T18:10:31.085205+0000","last_peered":"2024-01-15T18:10:31.085205+0000","last_clean":"2024-01-15T18:10:31.085205+0000","last_became_active":"2024-01-15T18:10:29.886310+0000","last_became_peered":"2024-01-15T18:10:29.886310+0000","last_unstale":"2024-01-15T18:10:31.085205+0000","last_undegraded":"2024-01-15T18:10:31.085205+0000","last_fullsized":"2024-01-15T18:10:31.085205+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:01:26.317819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"88'8","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.904150+0000","last_change":"2024-01-15T18:10:28.856512+0000","last_active":"2024-01-15T18:10:29.904150+0000","last_peered":"2024-01-15T18:10:29.904150+0000","last_clean":"2024-01-15T18:10:29.904150+0000","last_became_active":"2024-01-15T18:10:28.855528+0000","last_became_peered":"2024-01-15T18:10:28.855528+0000","last_unstale":"2024-01-15T18:10:29.904150+0000","last_undegraded":"2024-01-15T18:10:29.904150+0000","last_fullsized":"2024-01-15T18:10:29.904150+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:56.753249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085914+0000","last_change":"2024-01-15T18:10:29.881440+0000","last_active":"2024-01-15T18:10:31.085914+0000","last_peered":"2024-01-15T18:10:31.085914+0000","last_clean":"2024-01-15T18:10:31.085914+0000","last_became_active":"2024-01-15T18:10:29.881163+0000","last_became_peered":"2024-01-15T18:10:29.881163+0000","last_unstale":"2024-01-15T18:10:31.085914+0000","last_undegraded":"2024-01-15T18:10:31.085914+0000","last_fullsized":"2024-01-15T18:10:31.085914+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:07:43.534696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.899591+0000","last_change":"2024-01-15T18:10:28.864218+0000","last_active":"2024-01-15T18:10:29.899591+0000","last_peered":"2024-01-15T18:10:29.899591+0000","last_clean":"2024-01-15T18:10:29.899591+0000","last_became_active":"2024-01-15T18:10:28.863828+0000","last_became_peered":"2024-01-15T18:10:28.863828+0000","last_unstale":"2024-01-15T18:10:29.899591+0000","last_undegraded":"2024-01-15T18:10:29.899591+0000","last_fullsized":"2024-01-15T18:10:29.899591+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:30:24.812796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086310+0000","last_change":"2024-01-15T18:10:29.894064+0000","last_active":"2024-01-15T18:10:31.086310+0000","last_peered":"2024-01-15T18:10:31.086310+0000","last_clean":"2024-01-15T18:10:31.086310+0000","last_became_active":"2024-01-15T18:10:29.893707+0000","last_became_peered":"2024-01-15T18:10:29.893707+0000","last_unstale":"2024-01-15T18:10:31.086310+0000","last_undegraded":"2024-01-15T18:10:31.086310+0000","last_fullsized":"2024-01-15T18:10:31.086310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:42.772403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902966+0000","last_change":"2024-01-15T18:10:28.859456+0000","last_active":"2024-01-15T18:10:29.902966+0000","last_peered":"2024-01-15T18:10:29.902966+0000","last_clean":"2024-01-15T18:10:29.902966+0000","last_became_active":"2024-01-15T18:10:28.858710+0000","last_became_peered":"2024-01-15T18:10:28.858710+0000","last_unstale":"2024-01-15T18:10:29.902966+0000","last_undegraded":"2024-01-15T18:10:29.902966+0000","last_fullsized":"2024-01-15T18:10:29.902966+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:22.308478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886254+0000","last_change":"2024-01-15T18:10:29.893444+0000","last_active":"2024-01-15T18:10:30.886254+0000","last_peered":"2024-01-15T18:10:30.886254+0000","last_clean":"2024-01-15T18:10:30.886254+0000","last_became_active":"2024-01-15T18:10:29.893114+0000","last_became_peered":"2024-01-15T18:10:29.893114+0000","last_unstale":"2024-01-15T18:10:30.886254+0000","last_undegraded":"2024-01-15T18:10:30.886254+0000","last_fullsized":"2024-01-15T18:10:30.886254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:21.928711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.877354+0000","last_change":"2024-01-15T18:10:28.856970+0000","last_active":"2024-01-15T18:10:29.877354+0000","last_peered":"2024-01-15T18:10:29.877354+0000","last_clean":"2024-01-15T18:10:29.877354+0000","last_became_active":"2024-01-15T18:10:28.856677+0000","last_became_peered":"2024-01-15T18:10:28.856677+0000","last_unstale":"2024-01-15T18:10:29.877354+0000","last_undegraded":"2024-01-15T18:10:29.877354+0000","last_fullsized":"2024-01-15T18:10:29.877354+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:46:17.019853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.884031+0000","last_change":"2024-01-15T18:10:29.884301+0000","last_active":"2024-01-15T18:10:30.884031+0000","last_peered":"2024-01-15T18:10:30.884031+0000","last_clean":"2024-01-15T18:10:30.884031+0000","last_became_active":"2024-01-15T18:10:29.883550+0000","last_became_peered":"2024-01-15T18:10:29.883550+0000","last_unstale":"2024-01-15T18:10:30.884031+0000","last_undegraded":"2024-01-15T18:10:30.884031+0000","last_fullsized":"2024-01-15T18:10:30.884031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:15:37.827724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902506+0000","last_change":"2024-01-15T18:10:28.853663+0000","last_active":"2024-01-15T18:10:29.902506+0000","last_peered":"2024-01-15T18:10:29.902506+0000","last_clean":"2024-01-15T18:10:29.902506+0000","last_became_active":"2024-01-15T18:10:28.849935+0000","last_became_peered":"2024-01-15T18:10:28.849935+0000","last_unstale":"2024-01-15T18:10:29.902506+0000","last_undegraded":"2024-01-15T18:10:29.902506+0000","last_fullsized":"2024-01-15T18:10:29.902506+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:17.975732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.886998+0000","last_change":"2024-01-15T18:10:29.878910+0000","last_active":"2024-01-15T18:10:30.886998+0000","last_peered":"2024-01-15T18:10:30.886998+0000","last_clean":"2024-01-15T18:10:30.886998+0000","last_became_active":"2024-01-15T18:10:29.878619+0000","last_became_peered":"2024-01-15T18:10:29.878619+0000","last_unstale":"2024-01-15T18:10:30.886998+0000","last_undegraded":"2024-01-15T18:10:30.886998+0000","last_fullsized":"2024-01-15T18:10:30.886998+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:57.768815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.865578+0000","last_change":"2024-01-15T18:10:28.848970+0000","last_active":"2024-01-15T18:10:29.865578+0000","last_peered":"2024-01-15T18:10:29.865578+0000","last_clean":"2024-01-15T18:10:29.865578+0000","last_became_active":"2024-01-15T18:10:28.848694+0000","last_became_peered":"2024-01-15T18:10:28.848694+0000","last_unstale":"2024-01-15T18:10:29.865578+0000","last_undegraded":"2024-01-15T18:10:29.865578+0000","last_fullsized":"2024-01-15T18:10:29.865578+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:44:45.666850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.883564+0000","last_change":"2024-01-15T18:10:29.886092+0000","last_active":"2024-01-15T18:10:30.883564+0000","last_peered":"2024-01-15T18:10:30.883564+0000","last_clean":"2024-01-15T18:10:30.883564+0000","last_became_active":"2024-01-15T18:10:29.880525+0000","last_became_peered":"2024-01-15T18:10:29.880525+0000","last_unstale":"2024-01-15T18:10:30.883564+0000","last_undegraded":"2024-01-15T18:10:30.883564+0000","last_fullsized":"2024-01-15T18:10:30.883564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:05:00.656213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.864004+0000","last_change":"2024-01-15T18:10:28.863335+0000","last_active":"2024-01-15T18:10:29.864004+0000","last_peered":"2024-01-15T18:10:29.864004+0000","last_clean":"2024-01-15T18:10:29.864004+0000","last_became_active":"2024-01-15T18:10:28.862934+0000","last_became_peered":"2024-01-15T18:10:28.862934+0000","last_unstale":"2024-01-15T18:10:29.864004+0000","last_undegraded":"2024-01-15T18:10:29.864004+0000","last_fullsized":"2024-01-15T18:10:29.864004+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:53:04.581565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887218+0000","last_change":"2024-01-15T18:10:29.883072+0000","last_active":"2024-01-15T18:10:30.887218+0000","last_peered":"2024-01-15T18:10:30.887218+0000","last_clean":"2024-01-15T18:10:30.887218+0000","last_became_active":"2024-01-15T18:10:29.882241+0000","last_became_peered":"2024-01-15T18:10:29.882241+0000","last_unstale":"2024-01-15T18:10:30.887218+0000","last_undegraded":"2024-01-15T18:10:30.887218+0000","last_fullsized":"2024-01-15T18:10:30.887218+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:57.870564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.856692+0000","last_change":"2024-01-15T18:10:28.853080+0000","last_active":"2024-01-15T18:10:29.856692+0000","last_peered":"2024-01-15T18:10:29.856692+0000","last_clean":"2024-01-15T18:10:29.856692+0000","last_became_active":"2024-01-15T18:10:28.848348+0000","last_became_peered":"2024-01-15T18:10:28.848348+0000","last_unstale":"2024-01-15T18:10:29.856692+0000","last_undegraded":"2024-01-15T18:10:29.856692+0000","last_fullsized":"2024-01-15T18:10:29.856692+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:53.905827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885405+0000","last_change":"2024-01-15T18:10:29.889034+0000","last_active":"2024-01-15T18:10:30.885405+0000","last_peered":"2024-01-15T18:10:30.885405+0000","last_clean":"2024-01-15T18:10:30.885405+0000","last_became_active":"2024-01-15T18:10:29.888741+0000","last_became_peered":"2024-01-15T18:10:29.888741+0000","last_unstale":"2024-01-15T18:10:30.885405+0000","last_undegraded":"2024-01-15T18:10:30.885405+0000","last_fullsized":"2024-01-15T18:10:30.885405+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:10:05.302458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"88'2","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.901824+0000","last_change":"2024-01-15T18:10:28.853859+0000","last_active":"2024-01-15T18:10:29.901824+0000","last_peered":"2024-01-15T18:10:29.901824+0000","last_clean":"2024-01-15T18:10:29.901824+0000","last_became_active":"2024-01-15T18:10:28.853567+0000","last_became_peered":"2024-01-15T18:10:28.853567+0000","last_unstale":"2024-01-15T18:10:29.901824+0000","last_undegraded":"2024-01-15T18:10:29.901824+0000","last_fullsized":"2024-01-15T18:10:29.901824+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:56:53.496817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.086285+0000","last_change":"2024-01-15T18:10:29.887191+0000","last_active":"2024-01-15T18:10:31.086285+0000","last_peered":"2024-01-15T18:10:31.086285+0000","last_clean":"2024-01-15T18:10:31.086285+0000","last_became_active":"2024-01-15T18:10:29.886890+0000","last_became_peered":"2024-01-15T18:10:29.886890+0000","last_unstale":"2024-01-15T18:10:31.086285+0000","last_undegraded":"2024-01-15T18:10:31.086285+0000","last_fullsized":"2024-01-15T18:10:31.086285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:35.445206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"89'1","reported_seq":13,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.088005+0000","last_change":"2024-01-15T18:10:28.847096+0000","last_active":"2024-01-15T18:10:31.088005+0000","last_peered":"2024-01-15T18:10:31.088005+0000","last_clean":"2024-01-15T18:10:31.088005+0000","last_became_active":"2024-01-15T18:10:28.845910+0000","last_became_peered":"2024-01-15T18:10:28.845910+0000","last_unstale":"2024-01-15T18:10:31.088005+0000","last_undegraded":"2024-01-15T18:10:31.088005+0000","last_fullsized":"2024-01-15T18:10:31.088005+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:25:46.755994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085281+0000","last_change":"2024-01-15T18:10:29.896978+0000","last_active":"2024-01-15T18:10:31.085281+0000","last_peered":"2024-01-15T18:10:31.085281+0000","last_clean":"2024-01-15T18:10:31.085281+0000","last_became_active":"2024-01-15T18:10:29.895549+0000","last_became_peered":"2024-01-15T18:10:29.895549+0000","last_unstale":"2024-01-15T18:10:31.085281+0000","last_undegraded":"2024-01-15T18:10:31.085281+0000","last_fullsized":"2024-01-15T18:10:31.085281+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:33:26.719685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874581+0000","last_change":"2024-01-15T18:10:28.853688+0000","last_active":"2024-01-15T18:10:29.874581+0000","last_peered":"2024-01-15T18:10:29.874581+0000","last_clean":"2024-01-15T18:10:29.874581+0000","last_became_active":"2024-01-15T18:10:28.853347+0000","last_became_peered":"2024-01-15T18:10:28.853347+0000","last_unstale":"2024-01-15T18:10:29.874581+0000","last_undegraded":"2024-01-15T18:10:29.874581+0000","last_fullsized":"2024-01-15T18:10:29.874581+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:29:02.736339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.885571+0000","last_change":"2024-01-15T18:10:29.889239+0000","last_active":"2024-01-15T18:10:30.885571+0000","last_peered":"2024-01-15T18:10:30.885571+0000","last_clean":"2024-01-15T18:10:30.885571+0000","last_became_active":"2024-01-15T18:10:29.887314+0000","last_became_peered":"2024-01-15T18:10:29.887314+0000","last_unstale":"2024-01-15T18:10:30.885571+0000","last_undegraded":"2024-01-15T18:10:30.885571+0000","last_fullsized":"2024-01-15T18:10:30.885571+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:41.860420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.880493+0000","last_change":"2024-01-15T18:10:28.861226+0000","last_active":"2024-01-15T18:10:29.880493+0000","last_peered":"2024-01-15T18:10:29.880493+0000","last_clean":"2024-01-15T18:10:29.880493+0000","last_became_active":"2024-01-15T18:10:28.860947+0000","last_became_peered":"2024-01-15T18:10:28.860947+0000","last_unstale":"2024-01-15T18:10:29.880493+0000","last_undegraded":"2024-01-15T18:10:29.880493+0000","last_fullsized":"2024-01-15T18:10:29.880493+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:56.753249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:30.887268+0000","last_change":"2024-01-15T18:10:29.871769+0000","last_active":"2024-01-15T18:10:30.887268+0000","last_peered":"2024-01-15T18:10:30.887268+0000","last_clean":"2024-01-15T18:10:30.887268+0000","last_became_active":"2024-01-15T18:10:29.871132+0000","last_became_peered":"2024-01-15T18:10:29.871132+0000","last_unstale":"2024-01-15T18:10:30.887268+0000","last_undegraded":"2024-01-15T18:10:30.887268+0000","last_fullsized":"2024-01-15T18:10:30.887268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:21:23.324044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.871998+0000","last_change":"2024-01-15T18:10:28.853003+0000","last_active":"2024-01-15T18:10:29.871998+0000","last_peered":"2024-01-15T18:10:29.871998+0000","last_clean":"2024-01-15T18:10:29.871998+0000","last_became_active":"2024-01-15T18:10:28.852522+0000","last_became_peered":"2024-01-15T18:10:28.852522+0000","last_unstale":"2024-01-15T18:10:29.871998+0000","last_undegraded":"2024-01-15T18:10:29.871998+0000","last_fullsized":"2024-01-15T18:10:29.871998+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:38:56.854998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.085137+0000","last_change":"2024-01-15T18:10:29.889868+0000","last_active":"2024-01-15T18:10:31.085137+0000","last_peered":"2024-01-15T18:10:31.085137+0000","last_clean":"2024-01-15T18:10:31.085137+0000","last_became_active":"2024-01-15T18:10:29.887146+0000","last_became_peered":"2024-01-15T18:10:29.887146+0000","last_unstale":"2024-01-15T18:10:31.085137+0000","last_undegraded":"2024-01-15T18:10:31.085137+0000","last_fullsized":"2024-01-15T18:10:31.085137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:42:54.921393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.879486+0000","last_change":"2024-01-15T18:10:28.861120+0000","last_active":"2024-01-15T18:10:29.879486+0000","last_peered":"2024-01-15T18:10:29.879486+0000","last_clean":"2024-01-15T18:10:29.879486+0000","last_became_active":"2024-01-15T18:10:28.860131+0000","last_became_peered":"2024-01-15T18:10:28.860131+0000","last_unstale":"2024-01-15T18:10:29.879486+0000","last_undegraded":"2024-01-15T18:10:29.879486+0000","last_fullsized":"2024-01-15T18:10:29.879486+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:53.466810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084619+0000","last_change":"2024-01-15T18:10:29.883153+0000","last_active":"2024-01-15T18:10:31.084619+0000","last_peered":"2024-01-15T18:10:31.084619+0000","last_clean":"2024-01-15T18:10:31.084619+0000","last_became_active":"2024-01-15T18:10:29.882185+0000","last_became_peered":"2024-01-15T18:10:29.882185+0000","last_unstale":"2024-01-15T18:10:31.084619+0000","last_undegraded":"2024-01-15T18:10:31.084619+0000","last_fullsized":"2024-01-15T18:10:31.084619+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:12:18.991298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.874272+0000","last_change":"2024-01-15T18:10:28.863474+0000","last_active":"2024-01-15T18:10:29.874272+0000","last_peered":"2024-01-15T18:10:29.874272+0000","last_clean":"2024-01-15T18:10:29.874272+0000","last_became_active":"2024-01-15T18:10:28.863019+0000","last_became_peered":"2024-01-15T18:10:28.863019+0000","last_unstale":"2024-01-15T18:10:29.874272+0000","last_undegraded":"2024-01-15T18:10:29.874272+0000","last_fullsized":"2024-01-15T18:10:29.874272+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:24:04.910114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.084125+0000","last_change":"2024-01-15T18:10:29.896146+0000","last_active":"2024-01-15T18:10:31.084125+0000","last_peered":"2024-01-15T18:10:31.084125+0000","last_clean":"2024-01-15T18:10:31.084125+0000","last_became_active":"2024-01-15T18:10:29.895838+0000","last_became_peered":"2024-01-15T18:10:29.895838+0000","last_unstale":"2024-01-15T18:10:31.084125+0000","last_undegraded":"2024-01-15T18:10:31.084125+0000","last_fullsized":"2024-01-15T18:10:31.084125+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:37:13.634051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.877158+0000","last_change":"2024-01-15T18:10:28.863368+0000","last_active":"2024-01-15T18:10:29.877158+0000","last_peered":"2024-01-15T18:10:29.877158+0000","last_clean":"2024-01-15T18:10:29.877158+0000","last_became_active":"2024-01-15T18:10:28.861032+0000","last_became_peered":"2024-01-15T18:10:28.861032+0000","last_unstale":"2024-01-15T18:10:29.877158+0000","last_undegraded":"2024-01-15T18:10:29.877158+0000","last_fullsized":"2024-01-15T18:10:29.877158+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:10:55.062365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-01-15T18:10:31.087326+0000","last_change":"2024-01-15T18:10:29.878751+0000","last_active":"2024-01-15T18:10:31.087326+0000","last_peered":"2024-01-15T18:10:31.087326+0000","last_clean":"2024-01-15T18:10:31.087326+0000","last_became_active":"2024-01-15T18:10:29.876467+0000","last_became_peered":"2024-01-15T18:10:29.876467+0000","last_unstale":"2024-01-15T18:10:31.087326+0000","last_undegraded":"2024-01-15T18:10:31.087326+0000","last_fullsized":"2024-01-15T18:10:31.087326+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:28.822137+0000","last_clean_scrub_stamp":"2024-01-15T18:10:28.822137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:14:27.719678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"88'1","reported_seq":13,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.902018+0000","last_change":"2024-01-15T18:10:28.859062+0000","last_active":"2024-01-15T18:10:29.902018+0000","last_peered":"2024-01-15T18:10:29.902018+0000","last_clean":"2024-01-15T18:10:29.902018+0000","last_became_active":"2024-01-15T18:10:28.858021+0000","last_became_peered":"2024-01-15T18:10:28.858021+0000","last_unstale":"2024-01-15T18:10:29.902018+0000","last_undegraded":"2024-01-15T18:10:29.902018+0000","last_fullsized":"2024-01-15T18:10:29.902018+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:10:27.806571+0000","last_clean_scrub_stamp":"2024-01-15T18:10:27.806571+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:26:08.013171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":228,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-01-15T18:10:29.866943+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:10:29.866943+0000","last_peered":"2024-01-15T18:10:29.866943+0000","last_clean":"2024-01-15T18:10:29.866943+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:10:29.866943+0000","last_undegraded":"2024-01-15T18:10:29.866943+0000","last_fullsized":"2024-01-15T18:10:29.866943+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542304,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77180,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294660,"statfs":{"total":96636764160,"available":96557731840,"internally_reserved":0,"allocated":1687552,"data_stored":1062336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.3799999999999999},"min":{"1min":0.81899999999999995,"5min":0.626,"15min":0.53600000000000003},"max":{"1min":1.677,"5min":4.0019999999999998,"15min":14.114000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.7030000000000001,"15min":1.5680000000000001},"min":{"1min":0.91200000000000003,"5min":0.69999999999999996,"15min":0.51700000000000002},"max":{"1min":1.6970000000000001,"5min":27.32,"15min":27.32},"last":0.997}]},{"osd":1,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.8300000000000001,"15min":1.4670000000000001},"min":{"1min":0.97299999999999998,"5min":0.53300000000000003,"15min":0.40300000000000002},"max":{"1min":1.617,"5min":29.167999999999999,"15min":29.167999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.3560000000000001,"15min":1.375},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.835,"5min":4.2699999999999996,"15min":12.565},"last":0.80900000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.3360000000000001,"15min":1.464},"min":{"1min":0.61399999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.55,"5min":4.1920000000000002,"15min":13.801},"last":1.073},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.7,"15min":1.542},"min":{"1min":0.91500000000000004,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.788,"5min":27.358000000000001,"15min":27.358000000000001},"last":1.2350000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.853,"15min":1.609},"min":{"1min":1.097,"5min":0.61299999999999999,"15min":0.47099999999999997},"max":{"1min":1.619,"5min":29.204999999999998,"15min":29.204999999999998},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.425,"5min":1.506,"15min":1.5600000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":1.8300000000000001,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.1919999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.7509999999999999,"15min":1.621},"min":{"1min":0.91200000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.724,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.355},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.887,"15min":1.653},"min":{"1min":1.151,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":1.782,"5min":29.25,"15min":29.25},"last":1.044}]},{"osd":5,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.78,"15min":1.679},"min":{"1min":0.91500000000000004,"5min":0.83199999999999996,"15min":0.39000000000000001},"max":{"1min":1.944,"5min":11.834,"15min":12.518000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.9159999999999999,"15min":1.722},"min":{"1min":1.03,"5min":0.60499999999999998,"15min":0.41099999999999998},"max":{"1min":1.883,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.1479999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0579999999999998,"15min":2.367},"min":{"1min":1.0169999999999999,"5min":0.86499999999999999,"15min":0.45600000000000002},"max":{"1min":2.0390000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.591,"5min":2.101,"15min":1.8080000000000001},"min":{"1min":1.1899999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":1.923,"5min":29.106999999999999,"15min":29.106999999999999},"last":1.262}]}]},{"osd":6,"up_from":14,"seq":60129542305,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58624,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313216,"statfs":{"total":96636764160,"available":96576733184,"internally_reserved":0,"allocated":1101824,"data_stored":472592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.276,"15min":1.1950000000000001},"min":{"1min":0.64700000000000002,"5min":0.438,"15min":0.438},"max":{"1min":5.0330000000000004,"5min":5.1189999999999998,"15min":5.1189999999999998},"last":1.161},{"interface":"front","average":{"1min":1.405,"5min":1.3959999999999999,"15min":1.339},"min":{"1min":0.70999999999999996,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":3.1659999999999999,"5min":9.548,"15min":21.565000000000001},"last":0.93600000000000005}]},{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4319999999999999,"15min":1.3440000000000001},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.52100000000000002},"max":{"1min":5.1150000000000002,"5min":8.1189999999999998,"15min":8.1189999999999998},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.462,"15min":1.4370000000000001},"min":{"1min":0.624,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.73,"5min":9.6389999999999993,"15min":13.984999999999999},"last":1.4450000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3819999999999999,"15min":1.4630000000000001},"min":{"1min":0.92700000000000005,"5min":0.79800000000000004,"15min":0.54300000000000004},"max":{"1min":2.919,"5min":7.0030000000000001,"15min":34.670000000000002},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.516,"15min":1.411},"min":{"1min":0.90200000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":5.1609999999999996,"5min":6.585,"15min":6.585},"last":1.486}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.6479999999999999,"15min":1.5529999999999999},"min":{"1min":1.0069999999999999,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":3.5680000000000001,"5min":16.855,"15min":16.855},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.6180000000000001,"15min":1.679},"min":{"1min":0.96599999999999997,"5min":0.628,"15min":0.48499999999999999},"max":{"1min":2.9630000000000001,"5min":8.0139999999999993,"15min":34.725000000000001},"last":1.6479999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.512,"15min":1.4390000000000001},"min":{"1min":0.95399999999999996,"5min":0.56299999999999994,"15min":0.38400000000000001},"max":{"1min":5.2030000000000003,"5min":6.5220000000000002,"15min":8.5489999999999995},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.784,"5min":1.6379999999999999,"15min":1.5309999999999999},"min":{"1min":0.95999999999999996,"5min":0.63600000000000001,"15min":0.501},"max":{"1min":7.3230000000000004,"5min":10.118,"15min":10.118},"last":1.121}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.6839999999999999,"15min":1.657},"min":{"1min":1.0720000000000001,"5min":0.71199999999999997,"15min":0.57699999999999996},"max":{"1min":2.8420000000000001,"5min":7.9500000000000002,"15min":34.798000000000002},"last":1.264},{"interface":"front","average":{"1min":1.9259999999999999,"5min":1.7410000000000001,"15min":1.609},"min":{"1min":1.228,"5min":0.66000000000000003,"15min":0.47899999999999998},"max":{"1min":8.0129999999999999,"5min":10.077,"15min":10.077},"last":1.613}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.851,"5min":1.748,"15min":1.669},"min":{"1min":1.228,"5min":0.78900000000000003,"15min":0.64000000000000001},"max":{"1min":5.5060000000000002,"5min":10.302,"15min":10.302},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.7629999999999999,"15min":1.8500000000000001},"min":{"1min":1.2110000000000001,"5min":0.84599999999999997,"15min":0.52600000000000002},"max":{"1min":2.782,"5min":8.0519999999999996,"15min":34.762999999999998},"last":1.3939999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542307,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54468,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317372,"statfs":{"total":96636764160,"available":96580988928,"internally_reserved":0,"allocated":1105920,"data_stored":473532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.5780000000000001,"15min":1.3660000000000001},"min":{"1min":0.71499999999999997,"5min":0.67900000000000005,"15min":0.52700000000000002},"max":{"1min":1.5980000000000001,"5min":26.077999999999999,"15min":26.077999999999999},"last":1.262},{"interface":"front","average":{"1min":1.23,"5min":1.431,"15min":1.4470000000000001},"min":{"1min":0.76400000000000001,"5min":0.73999999999999999,"15min":0.52400000000000002},"max":{"1min":1.7270000000000001,"5min":7.7220000000000004,"15min":14.711},"last":1.323}]},{"osd":1,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.653,"15min":1.387},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.60099999999999998},"max":{"1min":1.6679999999999999,"5min":26,"15min":26},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.409,"15min":1.3400000000000001},"min":{"1min":0.64600000000000002,"5min":0.61099999999999999,"15min":0.54200000000000004},"max":{"1min":1.6919999999999999,"5min":7.5309999999999997,"15min":12.619},"last":1.204}]},{"osd":2,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.571,"5min":1.8340000000000001,"15min":1.5409999999999999},"min":{"1min":0.874,"5min":0.73299999999999998,"15min":0.55700000000000005},"max":{"1min":4.8719999999999999,"5min":26.234999999999999,"15min":26.234999999999999},"last":1.474},{"interface":"front","average":{"1min":1.605,"5min":1.825,"15min":1.6020000000000001},"min":{"1min":0.90200000000000002,"5min":0.63900000000000001,"15min":0.59099999999999997},"max":{"1min":4.968,"5min":17.126999999999999,"15min":17.126999999999999},"last":1.3899999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4670000000000001,"15min":1.448},"min":{"1min":0.85999999999999999,"5min":0.68100000000000005,"15min":0.622},"max":{"1min":1.722,"5min":7.6150000000000002,"15min":12.722},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.387,"5min":1.8740000000000001,"15min":1.637},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.63400000000000001},"max":{"1min":1.776,"5min":26.134,"15min":26.134},"last":0.98499999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.5029999999999999,"15min":1.508},"min":{"1min":0.754,"5min":0.68999999999999995,"15min":0.55900000000000005},"max":{"1min":1.72,"5min":7.8730000000000002,"15min":25.776},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.9470000000000001,"15min":1.734},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.47799999999999998},"max":{"1min":1.8799999999999999,"5min":16.414000000000001,"15min":16.414000000000001},"last":1.1519999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.466,"15min":1.5489999999999999},"min":{"1min":0.84099999999999997,"5min":0.46300000000000002,"15min":0.46200000000000002},"max":{"1min":1.9390000000000001,"5min":7.9240000000000004,"15min":12.769},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.798,"15min":1.8029999999999999},"min":{"1min":0.752,"5min":0.49199999999999999,"15min":0.435},"max":{"1min":1.865,"5min":26.186,"15min":36.087000000000003},"last":1.5589999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:47 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.9319999999999999,"15min":1.8959999999999999},"min":{"1min":1.0680000000000001,"5min":0.79100000000000004,"15min":0.68000000000000005},"max":{"1min":1.98,"5min":9.6859999999999999,"15min":13.706},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.6519999999999999,"15min":1.784},"min":{"1min":0.80000000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.9159999999999999,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.5089999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542306,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58620,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313220,"statfs":{"total":96636764160,"available":96576737280,"internally_reserved":0,"allocated":1089536,"data_stored":471898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1870000000000001,"15min":1.3580000000000001},"min":{"1min":0.55700000000000005,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.9419999999999999,"5min":4.8170000000000002,"15min":21.643000000000001},"last":1.405},{"interface":"front","average":{"1min":1.113,"5min":1.3049999999999999,"15min":1.2949999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.861,"5min":8.9109999999999996,"15min":11.196},"last":1.1200000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5960000000000001,"15min":1.4059999999999999},"min":{"1min":0.59299999999999997,"5min":0.41499999999999998,"15min":0.32200000000000001},"max":{"1min":1.587,"5min":28.663,"15min":28.663},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.53,"15min":1.4970000000000001},"min":{"1min":0.72599999999999998,"5min":0.46200000000000002,"15min":0.35399999999999998},"max":{"1min":1.837,"5min":13.614000000000001,"15min":21.440000000000001},"last":1.6439999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.6220000000000001,"15min":1.446},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.8049999999999999,"5min":16.023,"15min":16.023},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.446,"15min":1.5109999999999999},"min":{"1min":0.94299999999999995,"5min":0.52500000000000002,"15min":0.38100000000000001},"max":{"1min":1.7010000000000001,"5min":7.9169999999999998,"15min":22.350999999999999},"last":1.6919999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.375,"15min":1.512},"min":{"1min":0.874,"5min":0.56699999999999995,"15min":0.48799999999999999},"max":{"1min":1.5329999999999999,"5min":5.0860000000000003,"15min":15.093},"last":1.014},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.6919999999999999,"15min":1.651},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.6100000000000001,"5min":18.027000000000001,"15min":19.486000000000001},"last":1.1779999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3859999999999999,"15min":1.766},"min":{"1min":0.67800000000000005,"5min":0.41899999999999998,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":5.29,"15min":54.938000000000002},"last":1.583},{"interface":"front","average":{"1min":1.294,"5min":1.5620000000000001,"15min":1.5309999999999999},"min":{"1min":0.63,"5min":0.42399999999999999,"15min":0.34000000000000002},"max":{"1min":1.9610000000000001,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.22}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.6890000000000001,"15min":1.615},"min":{"1min":0.90800000000000003,"5min":0.495,"15min":0.38700000000000001},"max":{"1min":1.9279999999999999,"5min":16.103999999999999,"15min":17.733000000000001},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.7050000000000001,"15min":1.853},"min":{"1min":0.94699999999999995,"5min":0.499,"15min":0.499},"max":{"1min":2.0430000000000001,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.74}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.98,"15min":1.825},"min":{"1min":0.79100000000000004,"5min":0.46999999999999997,"15min":0.45100000000000001},"max":{"1min":2.0600000000000001,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.47},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.8939999999999999,"15min":1.847},"min":{"1min":0.65200000000000002,"5min":0.62,"15min":0.51000000000000001},"max":{"1min":2.0310000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.5269999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542307,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68296,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303544,"statfs":{"total":96636764160,"available":96566829056,"internally_reserved":0,"allocated":1101824,"data_stored":473510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":68811208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.371,"15min":1.0429999999999999},"min":{"1min":0.372,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.2050000000000001,"5min":36.408000000000001,"15min":36.408000000000001},"last":1.198},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.151,"15min":1.028},"min":{"1min":0.42499999999999999,"5min":0.377,"15min":0.377},"max":{"1min":1.665,"5min":5.0190000000000001,"15min":5.0529999999999999},"last":1.3859999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":1.1060000000000001,"15min":1.0920000000000001},"min":{"1min":0.55300000000000005,"5min":0.436,"15min":0.34399999999999997},"max":{"1min":1.365,"5min":5.226,"15min":8.1359999999999992},"last":1.264},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.161,"15min":1.179},"min":{"1min":0.56899999999999995,"5min":0.51600000000000001,"15min":0.40000000000000002},"max":{"1min":1.8140000000000001,"5min":2.1339999999999999,"15min":7.2169999999999996},"last":1.4419999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.4510000000000001,"15min":1.256},"min":{"1min":0.65900000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.859,"5min":14.025,"15min":14.025},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3460000000000001,"15min":1.2789999999999999},"min":{"1min":0.73399999999999999,"5min":0.57399999999999995,"15min":0.46000000000000002},"max":{"1min":1.9710000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.3240000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.569,"15min":1.423},"min":{"1min":0.94099999999999995,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2930000000000001,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.046},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.7430000000000001,"15min":1.5369999999999999},"min":{"1min":1.1100000000000001,"5min":0.748,"15min":0.69499999999999995},"max":{"1min":1.9119999999999999,"5min":15.769,"15min":15.769},"last":1.474}]},{"osd":5,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.659,"15min":1.53},"min":{"1min":1.079,"5min":0.69299999999999995,"15min":0.65000000000000002},"max":{"1min":1.984,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.669},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.2490000000000001,"15min":1.8919999999999999},"min":{"1min":0.97599999999999998,"5min":0.67900000000000005,"15min":0.58299999999999996},"max":{"1min":2.0800000000000001,"5min":38.832999999999998,"15min":40.384999999999998},"last":1.5249999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":2.1379999999999999,"15min":1.6299999999999999},"min":{"1min":0.86899999999999999,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.6339999999999999,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.8180000000000001,"15min":1.708},"min":{"1min":1.246,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.048,"5min":6.8639999999999999,"15min":16.195},"last":1.804}]},{"osd":7,"last update":"Mon Jan 15 18:09:45 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":2.2949999999999999,"15min":1.859},"min":{"1min":1.034,"5min":0.76000000000000001,"15min":0.63900000000000001},"max":{"1min":2.2629999999999999,"5min":24.146999999999998,"15min":24.146999999999998},"last":1.589},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1030000000000002,"15min":1.722},"min":{"1min":1.0049999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":2.3679999999999999,"5min":17.622,"15min":17.622},"last":1.623}]}]},{"osd":2,"up_from":14,"seq":60129542306,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68280,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303560,"statfs":{"total":96636764160,"available":96566845440,"internally_reserved":0,"allocated":1093632,"data_stored":472480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.98899999999999999,"15min":1.032},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.30099999999999999},"max":{"1min":1.3029999999999999,"5min":3.9199999999999999,"15min":6.101},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.087,"5min":1.1919999999999999,"15min":1.1419999999999999},"min":{"1min":0.41199999999999998,"5min":0.33700000000000002,"15min":0.28299999999999997},"max":{"1min":1.6839999999999999,"5min":10.319000000000001,"15min":10.319000000000001},"last":0.95099999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0509999999999999,"15min":1.105},"min":{"1min":0.79600000000000004,"5min":0.38600000000000001,"15min":0.379},"max":{"1min":1.5229999999999999,"5min":4.4939999999999998,"15min":10.807},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.131,"5min":1.2889999999999999,"15min":1.228},"min":{"1min":0.61599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.5960000000000001,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":1.171}]},{"osd":3,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.6180000000000001,"15min":1.454},"min":{"1min":0.75700000000000001,"5min":0.44400000000000001,"15min":0.41399999999999998},"max":{"1min":2.8010000000000002,"5min":25.891999999999999,"15min":25.891999999999999},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.099,"5min":1.373,"15min":1.256},"min":{"1min":0.78700000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.7270000000000001,"5min":14.946,"15min":14.946},"last":1.0369999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.546,"15min":1.552},"min":{"1min":0.874,"5min":0.66100000000000003,"15min":0.64900000000000002},"max":{"1min":1.9339999999999999,"5min":9.0679999999999996,"15min":9.0679999999999996},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.6870000000000001,"15min":1.5509999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":2.4249999999999998,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.702}]},{"osd":5,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.742,"15min":1.669},"min":{"1min":1.02,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":2.8519999999999999,"5min":11.071999999999999,"15min":11.071999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.427,"5min":1.6359999999999999,"15min":1.722},"min":{"1min":1,"5min":0.71699999999999997,"15min":0.55200000000000005},"max":{"1min":2.02,"5min":8.5250000000000004,"15min":37.020000000000003},"last":1.6240000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.645,"15min":1.7150000000000001},"min":{"1min":0.85199999999999998,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.889,"5min":10.185,"15min":23.762},"last":1.7989999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.9139999999999999,"15min":1.867},"min":{"1min":1.052,"5min":0.85899999999999999,"15min":0.73499999999999999},"max":{"1min":2.8860000000000001,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.3300000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:09:43 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":2.1349999999999998,"15min":1.931},"min":{"1min":1.0169999999999999,"5min":0.85999999999999999,"15min":0.68999999999999995},"max":{"1min":1.9470000000000001,"5min":19.986000000000001,"15min":52.767000000000003},"last":1.869},{"interface":"front","average":{"1min":1.605,"5min":1.988,"15min":1.804},"min":{"1min":0.97899999999999998,"5min":0.88100000000000001,"15min":0.83499999999999996},"max":{"1min":2.9169999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":1.5389999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68288,"kb_used_data":1076,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303552,"statfs":{"total":96636764160,"available":96566837248,"internally_reserved":0,"allocated":1101824,"data_stored":472092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.167,"15min":1.0189999999999999},"min":{"1min":0.41199999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.3160000000000001,"5min":19.218,"15min":19.218},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.288,"15min":1.1879999999999999},"min":{"1min":0.53900000000000003,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.573,"5min":19.823,"15min":19.823},"last":1.2410000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0129999999999999,"15min":1.0940000000000001},"min":{"1min":0.47599999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.6759999999999999,"5min":2.8980000000000001,"15min":7.6680000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.407,"15min":1.2470000000000001},"min":{"1min":0.46000000000000002,"5min":0.44800000000000001,"15min":0.32500000000000001},"max":{"1min":1.52,"5min":19.318000000000001,"15min":19.318000000000001},"last":0.93799999999999994}]},{"osd":3,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5109999999999999,"15min":1.365},"min":{"1min":0.56699999999999995,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.6100000000000001,"5min":19.387,"15min":19.387},"last":1.222},{"interface":"front","average":{"1min":1.218,"5min":1.4379999999999999,"15min":1.351},"min":{"1min":0.436,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.8919999999999999,"5min":19.733000000000001,"15min":19.733000000000001},"last":0.85299999999999998}]},{"osd":4,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.7010000000000001,"15min":1.542},"min":{"1min":0.79600000000000004,"5min":0.70599999999999996,"15min":0.497},"max":{"1min":1.7549999999999999,"5min":19.456,"15min":19.456},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.619,"15min":1.665},"min":{"1min":0.76600000000000001,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.859,"5min":19.925999999999998,"15min":22.204000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.6279999999999999,"15min":1.6830000000000001},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":1.8169999999999999,"5min":20.012,"15min":20.012},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.464,"15min":1.6240000000000001},"min":{"1min":0.71599999999999997,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":1.774,"5min":2.6190000000000002,"15min":19.998999999999999},"last":1.1120000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.7669999999999999,"15min":1.6910000000000001},"min":{"1min":0.95799999999999996,"5min":0.70799999999999996,"15min":0.54100000000000004},"max":{"1min":1.8120000000000001,"5min":19.526,"15min":19.526},"last":1.286},{"interface":"front","average":{"1min":1.48,"5min":1.8520000000000001,"15min":1.8200000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":1.9550000000000001,"5min":20.064,"15min":27.579999999999998},"last":1.363}]},{"osd":7,"last update":"Mon Jan 15 18:09:49 2024","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.8460000000000001,"15min":1.8120000000000001},"min":{"1min":0.81699999999999995,"5min":0.63,"15min":0.505},"max":{"1min":1.6930000000000001,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.393},{"interface":"front","average":{"1min":1.4730000000000001,"5min":2.0139999999999998,"15min":1.8240000000000001},"min":{"1min":0.94099999999999995,"5min":0.76100000000000001,"15min":0.54500000000000004},"max":{"1min":1.8009999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.4570000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542305,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68864,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302976,"statfs":{"total":96636764160,"available":96566247424,"internally_reserved":0,"allocated":1687552,"data_stored":1062300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":1.1020000000000001,"15min":1.1180000000000001},"min":{"1min":0.38100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.5620000000000001,"5min":20.437999999999999,"15min":20.437999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.1020000000000001,"15min":1.2030000000000001},"min":{"1min":0.61899999999999999,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":1.4370000000000001,"5min":3.492,"15min":35.137999999999998},"last":1.3}]},{"osd":2,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.375,"15min":1.278},"min":{"1min":0.60499999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.101,"5min":20.556000000000001,"15min":20.556000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.256,"15min":1.3640000000000001},"min":{"1min":0.44400000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7350000000000001,"5min":4.7430000000000003,"15min":35.183},"last":1.012}]},{"osd":3,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.143,"15min":1.1379999999999999},"min":{"1min":0.52800000000000002,"5min":0.46000000000000002,"15min":0.41699999999999998},"max":{"1min":1.3149999999999999,"5min":5.8499999999999996,"15min":8.9949999999999992},"last":1.135},{"interface":"front","average":{"1min":1.1879999999999999,"5min":1.5209999999999999,"15min":1.569},"min":{"1min":0.66700000000000004,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.4119999999999999,"5min":20.670000000000002,"15min":25.372},"last":1.427}]},{"osd":4,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.514,"15min":1.6759999999999999},"min":{"1min":0.98099999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7310000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.512},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.383,"15min":1.452},"min":{"1min":0.81200000000000006,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.6739999999999999,"5min":2.5049999999999999,"15min":15.968999999999999},"last":1.3660000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.7050000000000001,"15min":1.6080000000000001},"min":{"1min":0.78600000000000003,"5min":0.69799999999999995,"15min":0.63800000000000001},"max":{"1min":1.8,"5min":12.173999999999999,"15min":12.173999999999999},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.964,"15min":1.786},"min":{"1min":1.121,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":20.789999999999999,"15min":20.789999999999999},"last":1.542}]},{"osd":6,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":2.109,"15min":1.972},"min":{"1min":1.0109999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.74,"5min":20.893000000000001,"15min":36.390999999999998},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.726,"15min":1.9099999999999999},"min":{"1min":0.70299999999999996,"5min":0.62,"15min":0.62},"max":{"1min":1.8919999999999999,"5min":8.3829999999999991,"15min":36.207000000000001},"last":1.6559999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:09:48 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.986,"15min":1.9339999999999999},"min":{"1min":0.94899999999999995,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.0049999999999999,"5min":24.556999999999999,"15min":35.636000000000003},"last":1.599},{"interface":"front","average":{"1min":1.429,"5min":1.768,"15min":1.649},"min":{"1min":0.91900000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.8300000000000001,"5min":11.470000000000001,"15min":11.470000000000001},"last":1.6930000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:10:39.905 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:10:39.905 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:10:40.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.171+0000 7fc457e00640 1 Processor -- start 2024-01-15T18:10:40.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.171+0000 7fc457e00640 1 -- start start 2024-01-15T18:10:40.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.173+0000 7fc457e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc4580cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.173+0000 7fc457e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4580d3bd0 con 0x7fc4580cfa50 2024-01-15T18:10:40.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.173+0000 7fc457e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4580d3eb0 con 0x7fc4580b37e0 2024-01-15T18:10:40.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.173+0000 7fc457e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4580d18a0 con 0x7fc4580b3cb0 2024-01-15T18:10:40.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.173+0000 7fc456a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc4580cfa50 0x7fc4580b30d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44350/0 (socket says 172.21.15.2:44350) 2024-01-15T18:10:40.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.173+0000 7fc456a00640 1 -- 172.21.15.2:0/1840332813 learned_addr learned my addr 172.21.15.2:0/1840332813 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc44fe00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc4580cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965205768 0 0) 0x7fc4580d3bd0 con 0x7fc4580cfa50 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc438003520 con 0x7fc4580cfa50 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 887481831 0 0) 0x7fc438003520 con 0x7fc4580cfa50 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 msgr2=0x7fc4580cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc4580cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 >> v1:172.21.15.137:6789/0 conn(0x7fc4580b37e0 legacy=0x7fc4580ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.174+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4580d1b80 con 0x7fc4580cfa50 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc448001670 con 0x7fc4580cfa50 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 >> v1:172.21.15.2:6789/0 conn(0x7fc4580cfa50 legacy=0x7fc4580b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4380039a0 0x7fc438003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4380047c0 0x7fc438004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc438005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4580d18a0 con 0x7fc4380039a0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43800ba30 con 0x7fc4380047c0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43800bbf0 con 0x7fc4580b3cb0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456a00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4380039a0 0x7fc438003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc457400640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc438005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc44fe00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4380047c0 0x7fc438004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456a00640 1 -- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 msgr2=0x7fc438005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456a00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc438005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456a00640 1 -- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4380047c0 msgr2=0x7fc438004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456a00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4380047c0 0x7fc438004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc456a00640 1 -- 172.21.15.2:0/1840332813 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc438003520 con 0x7fc4380039a0 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc457400640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4580b3cb0 0x7fc438005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.175+0000 7fc44fe00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4380047c0 0x7fc438004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:40.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc456a00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4380039a0 0x7fc438003dc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fc44800c210 tx=0x7fc4380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1889ba689233b687 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc448005ad0 con 0x7fc4380039a0 2024-01-15T18:10:40.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc448005c30 con 0x7fc4380039a0 2024-01-15T18:10:40.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc456000640 1 -- 172.21.15.2:0/1840332813 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44802cab0 con 0x7fc4380039a0 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc457e00640 1 -- 172.21.15.2:0/1840332813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4380039a0 msgr2=0x7fc438003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc457e00640 1 --2- 172.21.15.2:0/1840332813 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4380039a0 0x7fc438003dc0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fc44800c210 tx=0x7fc4380057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.177+0000 7fc457400640 1 -- 172.21.15.2:0/1840332813 reap_dead start 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.178+0000 7fc457e00640 1 -- 172.21.15.2:0/1840332813 shutdown_connections 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.178+0000 7fc457e00640 1 -- 172.21.15.2:0/1840332813 >> 172.21.15.2:0/1840332813 conn(0x7fc4580d30b0 msgr2=0x7fc4580d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.178+0000 7fc457e00640 1 -- 172.21.15.2:0/1840332813 shutdown_connections 2024-01-15T18:10:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.178+0000 7fc457e00640 1 -- 172.21.15.2:0/1840332813 wait complete. 2024-01-15T18:10:40.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.180+0000 7fc457e00640 1 Processor -- start 2024-01-15T18:10:40.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.180+0000 7fc457e00640 1 -- start start 2024-01-15T18:10:40.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380039a0 0x7fc45815c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4580cc370 0x7fc4580cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 0x7fc4580ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4580ccd30 con 0x7fc4580ca180 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4580cab40 con 0x7fc4580cc370 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45815f160 con 0x7fc4380039a0 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 0x7fc4580ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 0x7fc4580ca600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55808/0 (socket says 172.21.15.2:55808) 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc457400640 1 -- 172.21.15.2:0/1805038641 learned_addr learned my addr 172.21.15.2:0/1805038641 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc456a00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380039a0 0x7fc45815c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.181+0000 7fc44fe00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4580cc370 0x7fc4580cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457400640 1 -- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380039a0 msgr2=0x7fc45815c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457400640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380039a0 0x7fc45815c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457400640 1 -- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4580cc370 msgr2=0x7fc4580cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457400640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4580cc370 0x7fc4580cc7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457400640 1 -- 172.21.15.2:0/1805038641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4580cad60 con 0x7fc4580ca180 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc44fe00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4580cc370 0x7fc4580cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457400640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 0x7fc4580ca600 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc458072bc0 tx=0x7fc45000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450003b70 con 0x7fc4580ca180 2024-01-15T18:10:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450003cf0 con 0x7fc4580ca180 2024-01-15T18:10:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500052b0 con 0x7fc4580ca180 2024-01-15T18:10:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4580cb040 con 0x7fc4580ca180 2024-01-15T18:10:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.182+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4580cb600 con 0x7fc4580ca180 2024-01-15T18:10:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.183+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458074770 con 0x7fc4580ca180 2024-01-15T18:10:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.184+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fc450005410 con 0x7fc4580ca180 2024-01-15T18:10:40.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.185+0000 7fc44ea00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc43006bfb0 0x7fc43006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.185+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fc450087f70 con 0x7fc4580ca180 2024-01-15T18:10:40.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.186+0000 7fc456a00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc43006bfb0 0x7fc43006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.187+0000 7fc456a00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc43006bfb0 0x7fc43006e470 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc448034000 tx=0x7fc448009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.192 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.189+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc450054830 con 0x7fc4580ca180 2024-01-15T18:10:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.454+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc4580d35c0 con 0x7fc4580ca180 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.459+0000 7fc44ea00640 1 -- 172.21.15.2:0/1805038641 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fc450018030 con 0x7fc4580ca180 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:10:40.463 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T18:10:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc43006bfb0 msgr2=0x7fc43006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc43006bfb0 0x7fc43006e470 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc448034000 tx=0x7fc448009030 comp rx=0 tx=0).stop 2024-01-15T18:10:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 msgr2=0x7fc4580ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 0x7fc4580ca600 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc458072bc0 tx=0x7fc45000dc00 comp rx=0 tx=0).stop 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 shutdown_connections 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc43006bfb0 0x7fc43006e470 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc4580ca180 0x7fc4580ca600 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc4580cc370 0x7fc4580cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 --2- 172.21.15.2:0/1805038641 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc4380039a0 0x7fc45815c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 >> 172.21.15.2:0/1805038641 conn(0x7fc4580d30b0 msgr2=0x7fc4580afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 shutdown_connections 2024-01-15T18:10:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.463+0000 7fc457e00640 1 -- 172.21.15.2:0/1805038641 wait complete. 2024-01-15T18:10:40.485 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:10:40.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.705+0000 7f642b200640 1 Processor -- start 2024-01-15T18:10:40.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.705+0000 7f642b200640 1 -- start start 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f642b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f642c15ec60 0x7f642c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f642b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c06cad0 0x7f642c06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f642b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0716d0 con 0x7f642c141880 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f642b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c071f20 con 0x7f642c15ec60 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f642b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c06c6b0 con 0x7f642c06cad0 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f6429e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f642c141880 0x7f642c070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44360/0 (socket says 172.21.15.2:44360) 2024-01-15T18:10:40.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f6429e00640 1 -- 172.21.15.2:0/383412194 learned_addr learned my addr 172.21.15.2:0/383412194 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:40.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.707+0000 7f6423e00640 1 --2- 172.21.15.2:0/383412194 >> v2:172.21.15.2:3300/0 conn(0x7f642c15ec60 0x7f642c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f642a800640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c06cad0 0x7f642c06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f6423e00640 1 -- 172.21.15.2:0/383412194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c06cad0 msgr2=0x7f642c06cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:40.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f6423e00640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c06cad0 0x7f642c06cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f6429400640 1 -- 172.21.15.2:0/383412194 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669696353 0 0) 0x7f642c0716d0 con 0x7f642c141880 2024-01-15T18:10:40.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f6423e00640 1 -- 172.21.15.2:0/383412194 >> v1:172.21.15.2:6789/0 conn(0x7f642c141880 legacy=0x7f642c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f6423e00640 1 -- 172.21.15.2:0/383412194 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c071960 con 0x7f642c15ec60 2024-01-15T18:10:40.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.708+0000 7f642a800640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c06cad0 0x7f642c06cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:40.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.709+0000 7f6423e00640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c15ec60 0x7f642c15f0e0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f64140097d0 tx=0x7f641402b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe320ca8972ee8b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.709+0000 7f6429400640 1 -- 172.21.15.2:0/383412194 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414031770 con 0x7f642c15ec60 2024-01-15T18:10:40.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.709+0000 7f6429400640 1 -- 172.21.15.2:0/383412194 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64140318d0 con 0x7f642c15ec60 2024-01-15T18:10:40.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.709+0000 7f6429400640 1 -- 172.21.15.2:0/383412194 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414031c60 con 0x7f642c15ec60 2024-01-15T18:10:40.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 -- 172.21.15.2:0/383412194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c15ec60 msgr2=0x7f642c15f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c15ec60 0x7f642c15f0e0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f64140097d0 tx=0x7f641402b810 comp rx=0 tx=0).stop 2024-01-15T18:10:40.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 -- 172.21.15.2:0/383412194 shutdown_connections 2024-01-15T18:10:40.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c06cad0 0x7f642c06cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 --2- 172.21.15.2:0/383412194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c15ec60 0x7f642c15f0e0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 -- 172.21.15.2:0/383412194 >> 172.21.15.2:0/383412194 conn(0x7f642c0b0840 msgr2=0x7f642c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:40.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 -- 172.21.15.2:0/383412194 shutdown_connections 2024-01-15T18:10:40.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.710+0000 7f642b200640 1 -- 172.21.15.2:0/383412194 wait complete. 2024-01-15T18:10:40.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.712+0000 7f642b200640 1 Processor -- start 2024-01-15T18:10:40.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.712+0000 7f642b200640 1 -- start start 2024-01-15T18:10:40.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f642b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c06cad0 0x7f642c13ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f642b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 0x7f642c0b3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f642b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c15ec60 0x7f642c1f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f642b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c13f3b0 con 0x7f642c141880 2024-01-15T18:10:40.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f642b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0b24f0 con 0x7f642c06cad0 2024-01-15T18:10:40.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f642b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0b27d0 con 0x7f642c15ec60 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.713+0000 7f6423e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 0x7f642c0b3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 0x7f642c0b3590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55822/0 (socket says 172.21.15.2:55822) 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 -- 172.21.15.2:0/4015072754 learned_addr learned my addr 172.21.15.2:0/4015072754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6429e00640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c06cad0 0x7f642c13ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f642a800640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c15ec60 0x7f642c1f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 -- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c15ec60 msgr2=0x7f642c1f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c15ec60 0x7f642c1f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 -- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c06cad0 msgr2=0x7f642c13ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c06cad0 0x7f642c13ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 -- 172.21.15.2:0/4015072754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6414009400 con 0x7f642c141880 2024-01-15T18:10:40.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.714+0000 7f6423e00640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 0x7f642c0b3590 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f641402b850 tx=0x7f64140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.715+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414032810 con 0x7f642c141880 2024-01-15T18:10:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.715+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6414032970 con 0x7f642c141880 2024-01-15T18:10:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.715+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c0b2ab0 con 0x7f642c141880 2024-01-15T18:10:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.715+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6414032d20 con 0x7f642c141880 2024-01-15T18:10:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.715+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c0b3070 con 0x7f642c141880 2024-01-15T18:10:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.716+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c074770 con 0x7f642c141880 2024-01-15T18:10:40.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.717+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f641404d3f0 con 0x7f642c141880 2024-01-15T18:10:40.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.718+0000 7f6422000640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406bfb0 0x7f640406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:40.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.718+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f64140aa510 con 0x7f642c141880 2024-01-15T18:10:40.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.719+0000 7f6429e00640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406bfb0 0x7f640406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:40.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.720+0000 7f6429e00640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406bfb0 0x7f640406e470 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f641c000f60 tx=0x7f641c007b50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:40.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.722+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6414076b10 con 0x7f642c141880 2024-01-15T18:10:40.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.954+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f642c0b3260 con 0x7f642c141880 2024-01-15T18:10:40.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.956+0000 7f6422000640 1 -- 172.21.15.2:0/4015072754 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f641407a130 con 0x7f642c141880 2024-01-15T18:10:40.959 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:10:40.959 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2471268746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2471268746},{"type":"v1","addr":"172.21.15.137:6836","nonce":2471268746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1012168762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":1012168762},{"type":"v1","addr":"172.21.15.137:6837","nonce":1012168762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6559,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2801413765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2801413765},{"type":"v1","addr":"172.21.15.2:6833","nonce":2801413765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:10:29.320912+0000","modified":"2024-01-15T18:10:32.417194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8055},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8055":{"gid":8055,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.2:6835/3200314764","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":3200314764},{"type":"v1","addr":"172.21.15.2:6835","nonce":3200314764}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-15T18:10:40.959 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 56 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.958+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406bfb0 msgr2=0x7f640406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.958+0000 7f642b200640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406bfb0 0x7f640406e470 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f641c000f60 tx=0x7f641c007b50 comp rx=0 tx=0).stop 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.958+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 msgr2=0x7f642c0b3590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.958+0000 7f642b200640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 0x7f642c0b3590 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f641402b850 tx=0x7f64140046c0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 shutdown_connections 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f640406bfb0 0x7f640406e470 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f642c15ec60 0x7f642c1f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f642c141880 0x7f642c0b3590 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 --2- 172.21.15.2:0/4015072754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f642c06cad0 0x7f642c13ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:40.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 >> 172.21.15.2:0/4015072754 conn(0x7f642c0b0840 msgr2=0x7f642c15f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:40.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 shutdown_connections 2024-01-15T18:10:40.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:40.959+0000 7f642b200640 1 -- 172.21.15.2:0/4015072754 wait complete. 2024-01-15T18:10:41.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:10:29.320912+0000', 'modified': '2024-01-15T18:10:32.417194+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8055}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8055': {'gid': 8055, 'name': 'c', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.2:6835/3200314764', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.2:6834', 'nonce': 3200314764}, {'type': 'v1', 'addr': '172.21.15.2:6835', 'nonce': 3200314764}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:10:41.385 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:10:41.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-15T18:10:41.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.615+0000 7f20bf200640 1 Processor -- start 2024-01-15T18:10:41.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.615+0000 7f20bf200640 1 -- start start 2024-01-15T18:10:41.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.617+0000 7f20bf200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f20b8143850 0x7f20b815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.617+0000 7f20bf200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f20b8160f60 0x7f20b806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.617+0000 7f20bf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8161410 0x7f20b806cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.617+0000 7f20bf200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b815e970 con 0x7f20b8143850 2024-01-15T18:10:41.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.617+0000 7f20bf200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20b81619b0 con 0x7f20b8160f60 2024-01-15T18:10:41.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.617+0000 7f20bf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b8161c90 con 0x7f20b8161410 2024-01-15T18:10:41.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f20b8143850 0x7f20b815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f20b8143850 0x7f20b815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55838/0 (socket says 172.21.15.2:55838) 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 -- 172.21.15.2:0/2333949945 learned_addr learned my addr 172.21.15.2:0/2333949945 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20af200640 1 --2- 172.21.15.2:0/2333949945 >> v2:172.21.15.137:3300/0 conn(0x7f20b8160f60 0x7f20b806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20bd200640 1 --2- 172.21.15.2:0/2333949945 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8161410 0x7f20b806cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 -- 172.21.15.2:0/2333949945 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8161410 msgr2=0x7f20b806cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 --2- 172.21.15.2:0/2333949945 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8161410 0x7f20b806cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 -- 172.21.15.2:0/2333949945 >> v2:172.21.15.137:3300/0 conn(0x7f20b8160f60 msgr2=0x7f20b806c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 --2- 172.21.15.2:0/2333949945 >> v2:172.21.15.137:3300/0 conn(0x7f20b8160f60 0x7f20b806c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.618+0000 7f20b7e00640 1 -- 172.21.15.2:0/2333949945 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b815dc30 con 0x7f20b8143850 2024-01-15T18:10:41.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.620+0000 7f20b7e00640 1 --2- 172.21.15.2:0/2333949945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815e430 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f20b80cec40 tx=0x7f20a802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f42bdfb30c2147 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.620+0000 7f20b7400640 1 -- 172.21.15.2:0/2333949945 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a802c950 con 0x7f20b8143850 2024-01-15T18:10:41.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.620+0000 7f20b7400640 1 -- 172.21.15.2:0/2333949945 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a802cab0 con 0x7f20b8143850 2024-01-15T18:10:41.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.620+0000 7f20b7400640 1 -- 172.21.15.2:0/2333949945 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a802ce40 con 0x7f20b8143850 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.620+0000 7f20bf200640 1 -- 172.21.15.2:0/2333949945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 msgr2=0x7f20b815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.620+0000 7f20bf200640 1 --2- 172.21.15.2:0/2333949945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815e430 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f20b80cec40 tx=0x7f20a802bdb0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 -- 172.21.15.2:0/2333949945 shutdown_connections 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 --2- 172.21.15.2:0/2333949945 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8161410 0x7f20b806cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 --2- 172.21.15.2:0/2333949945 >> v2:172.21.15.137:3300/0 conn(0x7f20b8160f60 0x7f20b806c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 --2- 172.21.15.2:0/2333949945 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815e430 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 -- 172.21.15.2:0/2333949945 >> 172.21.15.2:0/2333949945 conn(0x7f20b80b46d0 msgr2=0x7f20b80b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 -- 172.21.15.2:0/2333949945 shutdown_connections 2024-01-15T18:10:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.621+0000 7f20bf200640 1 -- 172.21.15.2:0/2333949945 wait complete. 2024-01-15T18:10:41.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.623+0000 7f20bf200640 1 Processor -- start 2024-01-15T18:10:41.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.623+0000 7f20bf200640 1 -- start start 2024-01-15T18:10:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20bf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20bf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8160f60 0x7f20b80cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20bf200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b8161410 0x7f20b80cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20bf200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b80ce240 con 0x7f20b8143850 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20bf200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20b80ce520 con 0x7f20b8161410 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20bf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20b80ce800 con 0x7f20b8160f60 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55840/0 (socket says 172.21.15.2:55840) 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 -- 172.21.15.2:0/3436683741 learned_addr learned my addr 172.21.15.2:0/3436683741 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 -- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8160f60 msgr2=0x7f20b80cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8160f60 0x7f20b80cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 -- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b8161410 msgr2=0x7f20b80cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b8161410 0x7f20b80cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.624+0000 7f20b7e00640 1 -- 172.21.15.2:0/3436683741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8009b30 con 0x7f20b8143850 2024-01-15T18:10:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.625+0000 7f20b7e00640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815c730 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f20a8000f80 tx=0x7f20a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.625+0000 7f20b4c00640 1 -- 172.21.15.2:0/3436683741 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a8009510 con 0x7f20b8143850 2024-01-15T18:10:41.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.625+0000 7f20b4c00640 1 -- 172.21.15.2:0/3436683741 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a803e450 con 0x7f20b8143850 2024-01-15T18:10:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.625+0000 7f20b4c00640 1 -- 172.21.15.2:0/3436683741 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a803e800 con 0x7f20b8143850 2024-01-15T18:10:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.625+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20b80ceae0 con 0x7f20b8143850 2024-01-15T18:10:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.625+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b80cd4f0 con 0x7f20b8143850 2024-01-15T18:10:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.628+0000 7f20b4c00640 1 -- 172.21.15.2:0/3436683741 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f20a803e960 con 0x7f20b8143850 2024-01-15T18:10:41.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.628+0000 7f20b4c00640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f209006bfb0 0x7f209006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.629+0000 7f20b4c00640 1 -- 172.21.15.2:0/3436683741 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f20a80ab700 con 0x7f20b8143850 2024-01-15T18:10:41.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.629+0000 7f20af200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f209006bfb0 0x7f209006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.629+0000 7f20c1f6f740 1 Processor -- start 2024-01-15T18:10:41.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.629+0000 7f20c1f6f740 1 -- start start 2024-01-15T18:10:41.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x56539150b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x565391505ba0 0x56539150a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x56539154e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565391469880 con 0x56539150a5e0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x565391478ae0 con 0x56539150aea0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653914761d0 con 0x565391505ba0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.630+0000 7f20af200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f209006bfb0 0x7f209006e470 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f20a0006fd0 tx=0x7f20a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x56539154e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x56539154e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55848/0 (socket says 172.21.15.2:55848) 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 -- 172.21.15.2:0/2466708971 learned_addr learned my addr 172.21.15.2:0/2466708971 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20af200640 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x565391505ba0 0x56539150a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20b7e00640 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x56539150b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 -- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x565391505ba0 msgr2=0x56539150a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x565391505ba0 0x56539150a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 -- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 msgr2=0x56539150b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x56539150b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 -- 172.21.15.2:0/2466708971 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565391439860 con 0x56539150a5e0 2024-01-15T18:10:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.631+0000 7f20bd200640 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x56539154e560 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f20b00097c0 tx=0x7f20b0009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf3e4a10c81dc42 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20aca00640 1 -- 172.21.15.2:0/2466708971 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b0018070 con 0x56539150a5e0 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20aca00640 1 -- 172.21.15.2:0/2466708971 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b000d950 con 0x56539150a5e0 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20aca00640 1 -- 172.21.15.2:0/2466708971 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b000dae0 con 0x56539150a5e0 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2466708971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 msgr2=0x56539154e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x56539154e560 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f20b00097c0 tx=0x7f20b0009c80 comp rx=0 tx=0).stop 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2466708971 shutdown_connections 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x56539154e560 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x565391505ba0 0x56539150a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2466708971 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x56539150b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.632+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2466708971 >> 172.21.15.2:0/2466708971 conn(0x5653914ccc70 msgr2=0x565391550990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.633+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2466708971 shutdown_connections 2024-01-15T18:10:41.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.633+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2466708971 wait complete. 2024-01-15T18:10:41.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.633+0000 7f20c1f6f740 1 -- start start 2024-01-15T18:10:41.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.634+0000 7f20c1f6f740 1 Processor -- start 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 0x5653915588f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150a5e0 0x565391547d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x565391548260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x565391471ac0 con 0x565391505ba0 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x56539140e2f0 con 0x56539150aea0 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x56539142bc70 con 0x56539150a5e0 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20c1f6f740 10 client.0 _ll_register_callbacks cb 0x5653914303b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:10:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 0x5653915588f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 0x5653915588f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55850/0 (socket says 172.21.15.2:55850) 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 -- 172.21.15.2:0/462480716 learned_addr learned my addr 172.21.15.2:0/462480716 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20af200640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150a5e0 0x565391547d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20bd200640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x565391548260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 -- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150a5e0 msgr2=0x565391547d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150a5e0 0x565391547d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 -- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 msgr2=0x565391548260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x565391548260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.635+0000 7f20b7e00640 1 -- 172.21.15.2:0/462480716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20b000e040 con 0x565391505ba0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20b7e00640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 0x5653915588f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f20a802c3c0 tx=0x7f20a807c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac9a0ee0ba27eca1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20a5600640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20a5600640 1 -- 172.21.15.2:0/462480716 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a807fb40 con 0x565391505ba0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20a5600640 1 -- 172.21.15.2:0/462480716 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f20a8083cf0 con 0x565391505ba0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56539146aaa0 con 0x565391505ba0 2024-01-15T18:10:41.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20a5600640 1 -- 172.21.15.2:0/462480716 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20a80ab080 con 0x565391505ba0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20c1f6f740 10 client.8211 fetch_fsmap learned FSMap version 56 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x565391470140 con 0x565391505ba0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.636+0000 7f20a5600640 1 -- 172.21.15.2:0/462480716 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a8097070 con 0x565391505ba0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20a5600640 1 -- 172.21.15.2:0/462480716 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f20a807cd20 con 0x565391505ba0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20c1f6f740 10 client.8211 fetch_fsmap finished waiting for FSMap version 56 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20c1f6f740 10 client.8211 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20c1f6f740 20 client.8211 populate_metadata read hostname 'smithi002' 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x5653915308d0 0x56539156dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20c1f6f740 4 client.8211 mds_command: new command op to 8055 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 --> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56539146a7c0 con 0x5653915308d0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.637+0000 7f20af200640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x5653915308d0 0x56539156dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.638+0000 7f20af200640 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x5653915308d0 0x56539156dd40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=573ea264c5360049 server_cookie=ce57c0978cee9670 in_seq=0 out_seq=0 2024-01-15T18:10:41.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.638+0000 7f20a5600640 10 client.8211 ms_handle_connect on v2:172.21.15.2:6834/3200314764 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.638+0000 7f20a5600640 1 -- 172.21.15.2:0/462480716 <== mds.0 v2:172.21.15.2:6834/3200314764 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56539146a7c0 con 0x5653915308d0 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.638+0000 7f20a5600640 10 client.8211 handle_command_reply: tid=0 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.638+0000 7f20c1f6f740 1 client.8211 shutdown 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 msgr2=0x5653915588f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 0x5653915588f0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f20a802c3c0 tx=0x7f20a807c0f0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 shutdown_connections 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x5653915308d0 0x56539156dd40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x56539150aea0 0x565391548260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150a5e0 0x565391547d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/462480716 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x565391505ba0 0x5653915588f0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20a5600640 0 client.8211 ms_handle_reset on v2:172.21.15.2:6834/3200314764 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 >> 172.21.15.2:0/462480716 conn(0x5653914ccc70 msgr2=0x56539150b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 shutdown_connections 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 1 -- 172.21.15.2:0/462480716 wait complete. 2024-01-15T18:10:41.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.639+0000 7f20c1f6f740 20 client.8211 trim_cache size 0 max 16384 2024-01-15T18:10:41.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.656+0000 7f20c1f6f740 1 Processor -- start 2024-01-15T18:10:41.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 -- start start 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x56539150b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915724a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653911d4f20 con 0x56539150a5e0 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x56539154dfd0 con 0x565391505ba0 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.657+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x56539154ad40 con 0x56539150aea0 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915724a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55864/0 (socket says 172.21.15.2:55864) 2024-01-15T18:10:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 -- 172.21.15.2:0/2432953820 learned_addr learned my addr 172.21.15.2:0/2432953820 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20b7e00640 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x56539150b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20bd200640 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 -- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 msgr2=0x565391547d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 -- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 msgr2=0x56539150b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x56539150b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.658+0000 7f20af200640 1 -- 172.21.15.2:0/2432953820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56539150c1b0 con 0x56539150a5e0 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20af200640 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915724a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f20b80c8890 tx=0x7f20a0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e87b224237ffa3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20ad400640 1 -- 172.21.15.2:0/2432953820 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a000dd40 con 0x56539150a5e0 2024-01-15T18:10:41.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20ad400640 1 -- 172.21.15.2:0/2432953820 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a0029070 con 0x56539150a5e0 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20ad400640 1 -- 172.21.15.2:0/2432953820 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0015b20 con 0x56539150a5e0 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2432953820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 msgr2=0x5653915724a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915724a0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f20b80c8890 tx=0x7f20a0010000 comp rx=0 tx=0).stop 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2432953820 shutdown_connections 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915724a0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/2432953820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x56539150b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.659+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2432953820 >> 172.21.15.2:0/2432953820 conn(0x5653914ccc70 msgr2=0x565391509d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.660+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2432953820 shutdown_connections 2024-01-15T18:10:41.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.660+0000 7f20c1f6f740 1 -- 172.21.15.2:0/2432953820 wait complete. 2024-01-15T18:10:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.660+0000 7f20c1f6f740 1 -- start start 2024-01-15T18:10:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.660+0000 7f20c1f6f740 1 Processor -- start 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x565391535540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915a4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653915bc600 con 0x56539150a5e0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653915bc080 con 0x565391505ba0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653915bb490 con 0x56539150aea0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.661+0000 7f20c1f6f740 10 client.0 _ll_register_callbacks cb 0x7f20a802c960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915a4470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55866/0 (socket says 172.21.15.2:55866) 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 -- 172.21.15.2:0/455217545 learned_addr learned my addr 172.21.15.2:0/455217545 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20b7e00640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x565391535540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20bd200640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 -- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 msgr2=0x565391547d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 -- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 msgr2=0x565391535540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x565391535540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 -- 172.21.15.2:0/455217545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a000e040 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.662+0000 7f20af200640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915a4470 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f20a0006e80 tx=0x7f20a0006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f4d489eb8b7adf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20a5600640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20a5600640 1 -- 172.21.15.2:0/455217545 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0005b90 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20a5600640 1 -- 172.21.15.2:0/455217545 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f20a001c040 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20b000d950 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20a5600640 1 -- 172.21.15.2:0/455217545 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20a0005cf0 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20c1f6f740 10 client.8217 fetch_fsmap learned FSMap version 56 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x56539140db80 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20a5600640 1 -- 172.21.15.2:0/455217545 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b000d950 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.663+0000 7f20a5600640 1 -- 172.21.15.2:0/455217545 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 56) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f20a0015920 con 0x56539150a5e0 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20c1f6f740 10 client.8217 fetch_fsmap finished waiting for FSMap version 56 2024-01-15T18:10:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20c1f6f740 10 client.8217 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20c1f6f740 20 client.8217 populate_metadata read hostname 'smithi002' 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x565391546b70 0x5653915334e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20c1f6f740 4 client.8217 mds_command: new command op to 8055 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 --> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5653911726c0 con 0x565391546b70 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20b7e00640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x565391546b70 0x5653915334e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20b7e00640 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x565391546b70 0x5653915334e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae9249dad2172a90 server_cookie=1994d3378d1ad06f in_seq=0 out_seq=0 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.664+0000 7f20a5600640 10 client.8217 ms_handle_connect on v2:172.21.15.2:6834/3200314764 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20a5600640 1 -- 172.21.15.2:0/455217545 <== mds.0 v2:172.21.15.2:6834/3200314764 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5653911726c0 con 0x565391546b70 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20a5600640 10 client.8217 handle_command_reply: tid=0 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 client.8217 shutdown 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 msgr2=0x5653915a4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915a4470 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f20a0006e80 tx=0x7f20a0006e10 comp rx=0 tx=0).stop 2024-01-15T18:10:41.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 shutdown_connections 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:6834/3200314764,v1:172.21.15.2:6835/3200314764] conn(0x565391546b70 0x5653915334e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x56539150aea0 0x565391547d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x56539150a5e0 0x5653915a4470 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 --2- 172.21.15.2:0/455217545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x565391505ba0 0x565391535540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20a5600640 0 client.8217 ms_handle_reset on v2:172.21.15.2:6834/3200314764 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 >> 172.21.15.2:0/455217545 conn(0x5653914ccc70 msgr2=0x56539154d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 shutdown_connections 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 1 -- 172.21.15.2:0/455217545 wait complete. 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.665+0000 7f20c1f6f740 20 client.8217 trim_cache size 0 max 16384 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:10:41.673 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 8055, 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 11.811662414000001, 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 56, 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 89, 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 89, 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 17.899949372999998 2024-01-15T18:10:41.674 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f209006bfb0 msgr2=0x7f209006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f209006bfb0 0x7f209006e470 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f20a0006fd0 tx=0x7f20a0008040 comp rx=0 tx=0).stop 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 msgr2=0x7f20b815c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815c730 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f20a8000f80 tx=0x7f20a8004a40 comp rx=0 tx=0).stop 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 shutdown_connections 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f209006bfb0 0x7f209006e470 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f20b8161410 0x7f20b80cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f20b8160f60 0x7f20b80cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 --2- 172.21.15.2:0/3436683741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f20b8143850 0x7f20b815c730 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 >> 172.21.15.2:0/3436683741 conn(0x7f20b80b46d0 msgr2=0x7f20b815fb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 shutdown_connections 2024-01-15T18:10:41.676 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.666+0000 7f20bf200640 1 -- 172.21.15.2:0/3436683741 wait complete. 2024-01-15T18:10:41.686 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:10:41.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.893+0000 7f0cebe00640 1 Processor -- start 2024-01-15T18:10:41.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.893+0000 7f0cebe00640 1 -- start start 2024-01-15T18:10:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0cebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec15d340 0x7f0cec0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0cebe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0cec0d3090 0x7f0cec0d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0cebe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0cec0d1680 0x7f0cec0d1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0cebe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cec0d3a30 con 0x7f0cec0d3090 2024-01-15T18:10:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0cebe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cec0b3de0 con 0x7f0cec0d1680 2024-01-15T18:10:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0cebe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0b40c0 con 0x7f0cec15d340 2024-01-15T18:10:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0ce1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0cec0d3090 0x7f0cec0d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0ce1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0cec0d3090 0x7f0cec0d34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55874/0 (socket says 172.21.15.2:55874) 2024-01-15T18:10:41.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.895+0000 7f0ce1e00640 1 -- 172.21.15.2:0/3963253784 learned_addr learned my addr 172.21.15.2:0/3963253784 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ceb400640 1 --2- 172.21.15.2:0/3963253784 >> v2:172.21.15.137:3300/0 conn(0x7f0cec0d1680 0x7f0cec0d1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ceaa00640 1 --2- 172.21.15.2:0/3963253784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec15d340 0x7f0cec0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ce1e00640 1 -- 172.21.15.2:0/3963253784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec15d340 msgr2=0x7f0cec0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ce1e00640 1 --2- 172.21.15.2:0/3963253784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec15d340 0x7f0cec0cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ce1e00640 1 -- 172.21.15.2:0/3963253784 >> v2:172.21.15.137:3300/0 conn(0x7f0cec0d1680 msgr2=0x7f0cec0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ce1e00640 1 --2- 172.21.15.2:0/3963253784 >> v2:172.21.15.137:3300/0 conn(0x7f0cec0d1680 0x7f0cec0d1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.896+0000 7f0ce1e00640 1 -- 172.21.15.2:0/3963253784 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec0d0e80 con 0x7f0cec0d3090 2024-01-15T18:10:41.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.897+0000 7f0ce1e00640 1 --2- 172.21.15.2:0/3963253784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec0d3090 0x7f0cec0d34f0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0ca1b0 tx=0x7f0cd402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc323f6c1cfd678 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.897+0000 7f0cea000640 1 -- 172.21.15.2:0/3963253784 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4031870 con 0x7f0cec0d3090 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.897+0000 7f0cea000640 1 -- 172.21.15.2:0/3963253784 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd40319d0 con 0x7f0cec0d3090 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.897+0000 7f0cea000640 1 -- 172.21.15.2:0/3963253784 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4031d60 con 0x7f0cec0d3090 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 -- 172.21.15.2:0/3963253784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec0d3090 msgr2=0x7f0cec0d34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 --2- 172.21.15.2:0/3963253784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec0d3090 0x7f0cec0d34f0 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f0cec0ca1b0 tx=0x7f0cd402b910 comp rx=0 tx=0).stop 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 -- 172.21.15.2:0/3963253784 shutdown_connections 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 --2- 172.21.15.2:0/3963253784 >> v2:172.21.15.137:3300/0 conn(0x7f0cec0d1680 0x7f0cec0d1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 --2- 172.21.15.2:0/3963253784 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec0d3090 0x7f0cec0d34f0 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 --2- 172.21.15.2:0/3963253784 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec15d340 0x7f0cec0cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 -- 172.21.15.2:0/3963253784 >> 172.21.15.2:0/3963253784 conn(0x7f0cec08caf0 msgr2=0x7f0cec09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 -- 172.21.15.2:0/3963253784 shutdown_connections 2024-01-15T18:10:41.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.898+0000 7f0cebe00640 1 -- 172.21.15.2:0/3963253784 wait complete. 2024-01-15T18:10:41.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.900+0000 7f0cebe00640 1 Processor -- start 2024-01-15T18:10:41.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.900+0000 7f0cebe00640 1 -- start start 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0cebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0d1680 0x7f0cec0cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0cebe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec0d3090 0x7f0cec0cbe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0cebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 0x7f0cec0cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0cebe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec15e480 con 0x7f0cec15d340 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0cebe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0cc880 con 0x7f0cec0d3090 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0cebe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0b88f0 con 0x7f0cec0d1680 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0ceb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 0x7f0cec0cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0ceb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 0x7f0cec0cc340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55890/0 (socket says 172.21.15.2:55890) 2024-01-15T18:10:41.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.901+0000 7f0ceb400640 1 -- 172.21.15.2:0/286020477 learned_addr learned my addr 172.21.15.2:0/286020477 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ce1e00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec0d3090 0x7f0cec0cbe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceaa00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0d1680 0x7f0cec0cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceb400640 1 -- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0d1680 msgr2=0x7f0cec0cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceb400640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0d1680 0x7f0cec0cb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceb400640 1 -- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec0d3090 msgr2=0x7f0cec0cbe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceb400640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec0d3090 0x7f0cec0cbe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:41.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceb400640 1 -- 172.21.15.2:0/286020477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40094d0 con 0x7f0cec15d340 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.902+0000 7f0ceb400640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 0x7f0cec0cc340 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4007d20 tx=0x7f0ce400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.903+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4014420 con 0x7f0cec15d340 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.903+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce40145a0 con 0x7f0cec15d340 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.903+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4014950 con 0x7f0cec15d340 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.903+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec15e6d0 con 0x7f0cec15d340 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.903+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec15ec90 con 0x7f0cec15d340 2024-01-15T18:10:41.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.904+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec0739e0 con 0x7f0cec15d340 2024-01-15T18:10:41.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.905+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f0ce4014ab0 con 0x7f0cec15d340 2024-01-15T18:10:41.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.906+0000 7f0ce3400640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406bfb0 0x7f0cc406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:41.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.906+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f0ce4088ef0 con 0x7f0cec15d340 2024-01-15T18:10:41.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.906+0000 7f0ceaa00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406bfb0 0x7f0cc406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:41.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.908+0000 7f0ceaa00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406bfb0 0x7f0cc406e470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc006fd0 tx=0x7f0cdc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:41.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:41.910+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ce40557b0 con 0x7f0cec15d340 2024-01-15T18:10:42.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.144+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0cec074770 con 0x7f0cec15d340 2024-01-15T18:10:42.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.145+0000 7f0ce3400640 1 -- 172.21.15.2:0/286020477 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0ce4058dd0 con 0x7f0cec15d340 2024-01-15T18:10:42.148 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:10:42.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406bfb0 msgr2=0x7f0cc406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406bfb0 0x7f0cc406e470 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc006fd0 tx=0x7f0cdc008040 comp rx=0 tx=0).stop 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 msgr2=0x7f0cec0cc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 0x7f0cec0cc340 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4007d20 tx=0x7f0ce400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 shutdown_connections 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406bfb0 0x7f0cc406e470 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec15d340 0x7f0cec0cc340 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec0d3090 0x7f0cec0cbe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 --2- 172.21.15.2:0/286020477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0d1680 0x7f0cec0cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 >> 172.21.15.2:0/286020477 conn(0x7f0cec08caf0 msgr2=0x7f0cec0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 shutdown_connections 2024-01-15T18:10:42.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:42.148+0000 7f0cebe00640 1 -- 172.21.15.2:0/286020477 wait complete. 2024-01-15T18:10:42.168 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:10:42.169 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:10:42.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:42.169 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:10:42.225 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41840sec preferred_lft 41840sec 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.226 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T18:10:42.227 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:42.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:42.228 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:10:42.325 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T18:10:42.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:42.330 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T18:10:42.385 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T18:10:42.385 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:10:42.386 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T18:10:42.386 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T18:10:42.386 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:10:42.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:42.386 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T18:10:42.386 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T18:10:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:10:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:10:42.386 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:10:42.386 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T18:10:42.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:42.579 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:10:42.633 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:10:42.633 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:10:42.634 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:10:42.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:42.634 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T18:10:42.734 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:10:42.734 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:10:42.734 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:42.792 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:10:42.792 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:10:42.792 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:42.848 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:10:42.952 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:10:43.013 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:10:43.013 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:10:43.013 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:10:43.014 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:10:43.015 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:10:43.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:43.017 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:10:43.074 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:10:43.075 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:10:43.082 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:10:43.161 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:10:43.202 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:10:43.203 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:10:43.204 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:10:43.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:43.205 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:10:43.275 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T18:10:43.272+0000 7f526044b480 -1 init, newargv = 0x565469573b20 newargc=15 2024-01-15T18:10:43.276 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[69158]: starting ceph client 2024-01-15T18:10:43.284 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[69158]: starting fuse 2024-01-15T18:10:43.290 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T18:10:43.290 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T18:10:43.290 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:10:43.290 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:10:43.291 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T18:10:43.292 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T18:10:43.495 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.69158.asok 2024-01-15T18:10:43.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:10:43.513 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.69158.asok 2024-01-15T18:10:43.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:43.513 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.69158.asok status 2024-01-15T18:10:43.761 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "69158", 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 8229, 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T18:10:43.762 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 8229 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2801444166 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 2801444166 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:10:43.763 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.8229 192.168.0.1:0/2801444166", 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/2801444166", 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 56, 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 89, 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T18:10:43.764 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:10:43.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:43.777 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T18:10:43.777 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:43.832 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T18:10:43.832 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:43.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:43.833 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:10:43.939 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:10:44.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.149+0000 7f7af7200640 1 Processor -- start 2024-01-15T18:10:44.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.149+0000 7f7af7200640 1 -- start start 2024-01-15T18:10:44.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.151+0000 7f7af7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7af8141880 0x7f7af8070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.151+0000 7f7af7200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af806dae0 con 0x7f7af815f0e0 2024-01-15T18:10:44.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.151+0000 7f7af7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af8071960 con 0x7f7af815ec60 2024-01-15T18:10:44.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.151+0000 7f7af7200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af8071f20 con 0x7f7af8141880 2024-01-15T18:10:44.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.151+0000 7f7af6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7af815f0e0 0x7f7af80cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41526/0 (socket says 172.21.15.2:41526) 2024-01-15T18:10:44.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.151+0000 7f7af6800640 1 -- 172.21.15.2:0/2382264017 learned_addr learned my addr 172.21.15.2:0/2382264017 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:44.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af5e00640 1 --2- 172.21.15.2:0/2382264017 >> v2:172.21.15.137:3300/0 conn(0x7f7af8141880 0x7f7af8070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910930711 0 0) 0x7f7af806dae0 con 0x7f7af815f0e0 2024-01-15T18:10:44.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad8003520 con 0x7f7af815f0e0 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 812664433 0 0) 0x7f7ad8003520 con 0x7f7af815f0e0 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 >> v2:172.21.15.137:3300/0 conn(0x7f7af8141880 msgr2=0x7f7af8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 --2- 172.21.15.2:0/2382264017 >> v2:172.21.15.137:3300/0 conn(0x7f7af8141880 0x7f7af8070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 >> v1:172.21.15.137:6789/0 conn(0x7f7af815ec60 legacy=0x7f7af80cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8071c40 con 0x7f7af815f0e0 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7af00019f0 con 0x7f7af815f0e0 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.152+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 >> v1:172.21.15.2:6789/0 conn(0x7f7af815f0e0 legacy=0x7f7af80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af4a00640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80039c0 0x7f7ad8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af4a00640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ad80047c0 0x7f7ad8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af4a00640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ad80051d0 0x7f7ad800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8071f20 con 0x7f7ad80047c0 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad800c120 con 0x7f7ad80039c0 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad800c2e0 con 0x7f7ad80051d0 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5400640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ad80047c0 0x7f7ad8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5e00640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80039c0 0x7f7ad8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5400640 1 -- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ad80051d0 msgr2=0x7f7ad800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5400640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7ad80051d0 0x7f7ad800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5400640 1 -- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80039c0 msgr2=0x7f7ad8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5400640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80039c0 0x7f7ad8003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af5400640 1 -- 172.21.15.2:0/2382264017 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad8003520 con 0x7f7ad80047c0 2024-01-15T18:10:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.153+0000 7f7af6800640 1 -- 172.21.15.2:0/2382264017 reap_dead start 2024-01-15T18:10:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.154+0000 7f7af5400640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ad80047c0 0x7f7ad8004c40 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0001720 tx=0x7f7ad80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68087b7169a07644 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00052d0 con 0x7f7ad80047c0 2024-01-15T18:10:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae002e050 con 0x7f7ad80047c0 2024-01-15T18:10:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af4a00640 1 -- 172.21.15.2:0/2382264017 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae0003830 con 0x7f7ad80047c0 2024-01-15T18:10:44.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 -- 172.21.15.2:0/2382264017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ad80047c0 msgr2=0x7f7ad8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ad80047c0 0x7f7ad8004c40 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0001720 tx=0x7f7ad80057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 -- 172.21.15.2:0/2382264017 shutdown_connections 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 --2- 172.21.15.2:0/2382264017 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7ad80047c0 0x7f7ad8004c40 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 -- 172.21.15.2:0/2382264017 >> 172.21.15.2:0/2382264017 conn(0x7f7af80b0840 msgr2=0x7f7af80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 -- 172.21.15.2:0/2382264017 shutdown_connections 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.155+0000 7f7af7200640 1 -- 172.21.15.2:0/2382264017 wait complete. 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.157+0000 7f7af7200640 1 Processor -- start 2024-01-15T18:10:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.157+0000 7f7af7200640 1 -- start start 2024-01-15T18:10:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80047c0 0x7f7af8158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7af8157370 0x7f7af80b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 0x7f7af813fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af806c6b0 con 0x7f7af80b2a90 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80b2f10 con 0x7f7ad80047c0 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8157ba0 con 0x7f7af8157370 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.158+0000 7f7af6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 0x7f7af813fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 0x7f7af813fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52770/0 (socket says 172.21.15.2:52770) 2024-01-15T18:10:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 -- 172.21.15.2:0/3977361236 learned_addr learned my addr 172.21.15.2:0/3977361236 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af5400640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7af8157370 0x7f7af80b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af5e00640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80047c0 0x7f7af8158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 -- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7af8157370 msgr2=0x7f7af80b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7af8157370 0x7f7af80b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 -- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80047c0 msgr2=0x7f7af8158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80047c0 0x7f7af8158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 -- 172.21.15.2:0/3977361236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8157e80 con 0x7f7af80b2a90 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.159+0000 7f7af6800640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 0x7f7af813fdb0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f7af815f4c0 tx=0x7f7af000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.160+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007c20 con 0x7f7af80b2a90 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.160+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af000a610 con 0x7f7af80b2a90 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.160+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af8158160 con 0x7f7af80b2a90 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.160+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af000aa00 con 0x7f7af80b2a90 2024-01-15T18:10:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.160+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af8158720 con 0x7f7af80b2a90 2024-01-15T18:10:44.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.161+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af8074770 con 0x7f7af80b2a90 2024-01-15T18:10:44.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.162+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f7af000ab60 con 0x7f7af80b2a90 2024-01-15T18:10:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.163+0000 7f7aee000640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ad006bfb0 0x7f7ad006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.163+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f7af00882e0 con 0x7f7af80b2a90 2024-01-15T18:10:44.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.164+0000 7f7af5e00640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ad006bfb0 0x7f7ad006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.165+0000 7f7af5e00640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ad006bfb0 0x7f7ad006e470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ae3d0 tx=0x7f7ae8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:44.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.168+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7af0054ba0 con 0x7f7af80b2a90 2024-01-15T18:10:44.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.398+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7af806e1c0 con 0x7f7af80b2a90 2024-01-15T18:10:44.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.399+0000 7f7aee000640 1 -- 172.21.15.2:0/3977361236 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7af00581c0 con 0x7f7af80b2a90 2024-01-15T18:10:44.402 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:10:44.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.401+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ad006bfb0 msgr2=0x7f7ad006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.401+0000 7f7af7200640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ad006bfb0 0x7f7ad006e470 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ae3d0 tx=0x7f7ae8008040 comp rx=0 tx=0).stop 2024-01-15T18:10:44.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 msgr2=0x7f7af813fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:44.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 0x7f7af813fdb0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f7af815f4c0 tx=0x7f7af000ccd0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 shutdown_connections 2024-01-15T18:10:44.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7ad006bfb0 0x7f7ad006e470 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7af80b2a90 0x7f7af813fdb0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7af8157370 0x7f7af80b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 --2- 172.21.15.2:0/3977361236 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7ad80047c0 0x7f7af8158c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:44.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 >> 172.21.15.2:0/3977361236 conn(0x7f7af80b0840 msgr2=0x7f7af806c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:44.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 shutdown_connections 2024-01-15T18:10:44.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:44.402+0000 7f7af7200640 1 -- 172.21.15.2:0/3977361236 wait complete. 2024-01-15T18:10:44.424 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:10:44.424 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:10:44.424 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T18:10:44.424 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:10:44.425 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:44.425 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:10:44.425 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:10:44.425 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:10:44.425 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:10:44.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:44.425 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T18:10:44.482 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41826sec preferred_lft 41826sec 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T18:10:44.483 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T18:10:44.484 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:10:44.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:44.485 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:10:44.574 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T18:10:44.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:44.578 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T18:10:44.628 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T18:10:44.629 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:10:44.629 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:10:44.629 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:10:44.629 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:10:44.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:44.629 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T18:10:44.629 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T18:10:44.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:10:44.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:10:44.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:10:44.629 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T18:10:44.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:44.805 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:10:44.858 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:10:44.858 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:10:44.858 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:10:44.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:44.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T18:10:44.950 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:10:44.950 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:10:44.950 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:44.997 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:10:44.998 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:10:44.998 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:45.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:10:45.189 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:10:45.243 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:10:45.243 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:10:45.243 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:10:45.243 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:10:45.243 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:10:45.243 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:10:45.244 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:10:45.245 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:10:45.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:45.246 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:10:45.297 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:10:45.298 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:10:45.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:10:45.380 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:10:45.423 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:10:45.423 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:10:45.423 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:10:45.423 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:10:45.423 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:10:45.424 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:10:45.425 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:10:45.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:45.426 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:10:45.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T18:10:45.488+0000 7f832de68480 -1 init, newargv = 0x557329e02bf0 newargc=15 2024-01-15T18:10:45.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45936]: starting ceph client 2024-01-15T18:10:45.503 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45936]: starting fuse 2024-01-15T18:10:45.508 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T18:10:45.508 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T18:10:45.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:10:45.508 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T18:10:45.508 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T18:10:45.508 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T18:10:45.508 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T18:10:45.508 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T18:10:45.509 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T18:10:45.510 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T18:10:45.510 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:10:45.510 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:10:45.510 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T18:10:45.510 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T18:10:45.698 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.45936.asok 2024-01-15T18:10:45.702 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T18:10:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:10:45.715 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45936.asok 2024-01-15T18:10:45.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:45.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45936.asok status 2024-01-15T18:10:45.954 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T18:10:45.954 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "45936", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 6116, 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 6116 2024-01-15T18:10:45.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2346452416 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:10:45.956 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2346452416 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.6116 192.168.0.1:0/2346452416", 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/2346452416", 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 56, 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 89, 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T18:10:45.959 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T18:10:45.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:45.969 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T18:10:45.970 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:46.022 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T18:10:46.022 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:46.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:46.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:10:46.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:10:46.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.346+0000 7f1af5600640 1 Processor -- start 2024-01-15T18:10:46.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.346+0000 7f1af5600640 1 -- start start 2024-01-15T18:10:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1af5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 0x7f1af00d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1af5600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af006f440 con 0x7f1af015d9a0 2024-01-15T18:10:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1af5600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af015e5f0 con 0x7f1af0146ba0 2024-01-15T18:10:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1af5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0146f80 con 0x7f1af00b32b0 2024-01-15T18:10:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1aef400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1af015d9a0 0x7f1af006b9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41528/0 (socket says 172.21.15.2:41528) 2024-01-15T18:10:46.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1aef400640 1 -- 172.21.15.2:0/3506935032 learned_addr learned my addr 172.21.15.2:0/3506935032 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:46.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.348+0000 7f1aee000640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 0x7f1af00d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961677613 0 0) 0x7f1af006f440 con 0x7f1af015d9a0 2024-01-15T18:10:46.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad0003520 con 0x7f1af015d9a0 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4255051145 0 0) 0x7f1ad0003520 con 0x7f1af015d9a0 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 msgr2=0x7f1af00d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 0x7f1af00d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 >> v1:172.21.15.137:6789/0 conn(0x7f1af0146ba0 legacy=0x7f1af00b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af015e8d0 con 0x7f1af015d9a0 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1ae8001670 con 0x7f1af015d9a0 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 >> v1:172.21.15.2:6789/0 conn(0x7f1af015d9a0 legacy=0x7f1af006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ad00039a0 0x7f1ad0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1ad0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 0x7f1ad000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af0146f80 con 0x7f1ad00047c0 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad00054e0 con 0x7f1ad00039a0 2024-01-15T18:10:46.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad000bec0 con 0x7f1af00b32b0 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.349+0000 7f1aee000640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1ad0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aef400640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 0x7f1ad000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aeea00640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ad00039a0 0x7f1ad0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aee000640 1 -- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 msgr2=0x7f1ad000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aee000640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af00b32b0 0x7f1ad000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aee000640 1 -- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ad00039a0 msgr2=0x7f1ad0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aee000640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1ad00039a0 0x7f1ad0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.350+0000 7f1aee000640 1 -- 172.21.15.2:0/3506935032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0003520 con 0x7f1ad00047c0 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.351+0000 7f1aee000640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1ad0004c40 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800ba30 tx=0x7f1ad00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e00b228c774035 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.352+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8032640 con 0x7f1ad00047c0 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.352+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad80327a0 con 0x7f1ad00047c0 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.352+0000 7f1aed600640 1 -- 172.21.15.2:0/3506935032 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8032b30 con 0x7f1ad00047c0 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.352+0000 7f1af5600640 1 -- 172.21.15.2:0/3506935032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 msgr2=0x7f1ad0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.352+0000 7f1af5600640 1 --2- 172.21.15.2:0/3506935032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1ad0004c40 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800ba30 tx=0x7f1ad00057f0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.352+0000 7f1aef400640 1 -- 172.21.15.2:0/3506935032 reap_dead start 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.353+0000 7f1af5600640 1 -- 172.21.15.2:0/3506935032 shutdown_connections 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.353+0000 7f1af5600640 1 -- 172.21.15.2:0/3506935032 >> 172.21.15.2:0/3506935032 conn(0x7f1af00b6430 msgr2=0x7f1af00b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:46.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.353+0000 7f1af5600640 1 -- 172.21.15.2:0/3506935032 shutdown_connections 2024-01-15T18:10:46.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.353+0000 7f1af5600640 1 -- 172.21.15.2:0/3506935032 wait complete. 2024-01-15T18:10:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.355+0000 7f1af5600640 1 Processor -- start 2024-01-15T18:10:46.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.355+0000 7f1af5600640 1 -- start start 2024-01-15T18:10:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1af5600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1af015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1af5600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af0140c10 0x7f1af0141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1af5600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af00cbc80 0x7f1af00cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1af5600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af01415d0 con 0x7f1ad00047c0 2024-01-15T18:10:46.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1af5600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af015d040 con 0x7f1af00cbc80 2024-01-15T18:10:46.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1af5600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af01f2fc0 con 0x7f1af0140c10 2024-01-15T18:10:46.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1aeea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1af015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1aeea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1af015cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52788/0 (socket says 172.21.15.2:52788) 2024-01-15T18:10:46.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.356+0000 7f1aeea00640 1 -- 172.21.15.2:0/1035456292 learned_addr learned my addr 172.21.15.2:0/1035456292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aef400640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af00cbc80 0x7f1af00cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aee000640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af0140c10 0x7f1af0141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aeea00640 1 -- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af0140c10 msgr2=0x7f1af0141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aeea00640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af0140c10 0x7f1af0141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aeea00640 1 -- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af00cbc80 msgr2=0x7f1af00cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aeea00640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af00cbc80 0x7f1af00cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aeea00640 1 -- 172.21.15.2:0/1035456292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af01f32a0 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1aeea00640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1af015cb00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000e8c0 tx=0x7f1ae000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0003b80 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae0003ce0 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae00052f0 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.357+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af01f3580 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.358+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af00cc640 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.359+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af00686f0 con 0x7f1ad00047c0 2024-01-15T18:10:46.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.360+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1ae0005450 con 0x7f1ad00047c0 2024-01-15T18:10:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.360+0000 7f1ae6a00640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ac806bfb0 0x7f1ac806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:46.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.361+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7f1ae0088a10 con 0x7f1ad00047c0 2024-01-15T18:10:46.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.361+0000 7f1aee000640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ac806bfb0 0x7f1ac806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:46.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.362+0000 7f1aee000640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ac806bfb0 0x7f1ac806e470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800ba00 tx=0x7f1ad803d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:46.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.365+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ae0055040 con 0x7f1ad00047c0 2024-01-15T18:10:46.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.599+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1af00cc7d0 con 0x7f1ad00047c0 2024-01-15T18:10:46.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.832+0000 7f1ae6a00640 1 -- 172.21.15.2:0/1035456292 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v57) v1 ==== 136+0+0 (secure 0 0 0) 0x7f1ae001a310 con 0x7f1ad00047c0 2024-01-15T18:10:46.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.835+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ac806bfb0 msgr2=0x7f1ac806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.835+0000 7f1af5600640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ac806bfb0 0x7f1ac806e470 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1ad800ba00 tx=0x7f1ad803d040 comp rx=0 tx=0).stop 2024-01-15T18:10:46.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 msgr2=0x7f1af015cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:46.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1af015cb00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f1ae000e8c0 tx=0x7f1ae000ed80 comp rx=0 tx=0).stop 2024-01-15T18:10:46.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 shutdown_connections 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ac806bfb0 0x7f1ac806e470 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1af00cbc80 0x7f1af00cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1af0140c10 0x7f1af0141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 --2- 172.21.15.2:0/1035456292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1ad00047c0 0x7f1af015cb00 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 >> 172.21.15.2:0/1035456292 conn(0x7f1af00b6430 msgr2=0x7f1af015eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 shutdown_connections 2024-01-15T18:10:46.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:46.836+0000 7f1af5600640 1 -- 172.21.15.2:0/1035456292 wait complete. 2024-01-15T18:10:46.861 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:10:47.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.083+0000 7fe5a5000640 1 Processor -- start 2024-01-15T18:10:47.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.084+0000 7fe5a5000640 1 -- start start 2024-01-15T18:10:47.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.085+0000 7fe5a5000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe5a00b30a0 0x7fe5a006c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.085+0000 7fe5a5000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5a006cdf0 0x7fe5a015b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.085+0000 7fe5a5000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a006d220 con 0x7fe5a0144b80 2024-01-15T18:10:47.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.085+0000 7fe5a5000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a00b4220 con 0x7fe5a006cdf0 2024-01-15T18:10:47.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.085+0000 7fe5a5000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a00cc1f0 con 0x7fe5a00b30a0 2024-01-15T18:10:47.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.086+0000 7fe59e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5a006cdf0 0x7fe5a015b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.086+0000 7fe59e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5a006cdf0 0x7fe5a015b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52800/0 (socket says 172.21.15.2:52800) 2024-01-15T18:10:47.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.086+0000 7fe59e800640 1 -- 172.21.15.2:0/1657788607 learned_addr learned my addr 172.21.15.2:0/1657788607 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:47.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.086+0000 7fe597e00640 1 --2- 172.21.15.2:0/1657788607 >> v2:172.21.15.137:3300/0 conn(0x7fe5a00b30a0 0x7fe5a006c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.087+0000 7fe59e800640 1 -- 172.21.15.2:0/1657788607 >> v2:172.21.15.137:3300/0 conn(0x7fe5a00b30a0 msgr2=0x7fe5a006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:47.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.087+0000 7fe59e800640 1 --2- 172.21.15.2:0/1657788607 >> v2:172.21.15.137:3300/0 conn(0x7fe5a00b30a0 0x7fe5a006c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.087+0000 7fe59e800640 1 -- 172.21.15.2:0/1657788607 >> v1:172.21.15.137:6789/0 conn(0x7fe5a0144b80 legacy=0x7fe5a00cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.087+0000 7fe59e800640 1 -- 172.21.15.2:0/1657788607 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a00b35a0 con 0x7fe5a006cdf0 2024-01-15T18:10:47.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.088+0000 7fe59e800640 1 --2- 172.21.15.2:0/1657788607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a006cdf0 0x7fe5a015b9e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00b2940 tx=0x7fe59802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0b28996d0067b07 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:47.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe59d400640 1 -- 172.21.15.2:0/1657788607 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59802ca70 con 0x7fe5a006cdf0 2024-01-15T18:10:47.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe59d400640 1 -- 172.21.15.2:0/1657788607 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59802cbd0 con 0x7fe5a006cdf0 2024-01-15T18:10:47.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe59d400640 1 -- 172.21.15.2:0/1657788607 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5980343f0 con 0x7fe5a006cdf0 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 -- 172.21.15.2:0/1657788607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a006cdf0 msgr2=0x7fe5a015b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 --2- 172.21.15.2:0/1657788607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a006cdf0 0x7fe5a015b9e0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00b2940 tx=0x7fe59802bed0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 -- 172.21.15.2:0/1657788607 shutdown_connections 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 --2- 172.21.15.2:0/1657788607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a006cdf0 0x7fe5a015b9e0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 --2- 172.21.15.2:0/1657788607 >> v2:172.21.15.137:3300/0 conn(0x7fe5a00b30a0 0x7fe5a006c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 -- 172.21.15.2:0/1657788607 >> 172.21.15.2:0/1657788607 conn(0x7fe5a00b46d0 msgr2=0x7fe5a00b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.089+0000 7fe5a5000640 1 -- 172.21.15.2:0/1657788607 shutdown_connections 2024-01-15T18:10:47.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.090+0000 7fe5a5000640 1 -- 172.21.15.2:0/1657788607 wait complete. 2024-01-15T18:10:47.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.091+0000 7fe5a5000640 1 Processor -- start 2024-01-15T18:10:47.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.091+0000 7fe5a5000640 1 -- start start 2024-01-15T18:10:47.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.092+0000 7fe5a5000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5a006cdf0 0x7fe5a00aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.092+0000 7fe5a5000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5a00b30a0 0x7fe5a015aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.092+0000 7fe5a5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 0x7fe5a0141b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.092+0000 7fe5a5000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00c4f80 con 0x7fe5a0144b80 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.092+0000 7fe5a5000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00af670 con 0x7fe5a00b30a0 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.092+0000 7fe5a5000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00af950 con 0x7fe5a006cdf0 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 0x7fe5a0141b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 0x7fe5a0141b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52804/0 (socket says 172.21.15.2:52804) 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 -- 172.21.15.2:0/406755917 learned_addr learned my addr 172.21.15.2:0/406755917 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe597e00640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5a00b30a0 0x7fe5a015aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59de00640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5a006cdf0 0x7fe5a00aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 -- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5a006cdf0 msgr2=0x7fe5a00aed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5a006cdf0 0x7fe5a00aed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 -- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5a00b30a0 msgr2=0x7fe5a015aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5a00b30a0 0x7fe5a015aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.093+0000 7fe59e800640 1 -- 172.21.15.2:0/406755917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe598009b30 con 0x7fe5a0144b80 2024-01-15T18:10:47.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.094+0000 7fe59e800640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 0x7fe5a0141b90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe598009e50 tx=0x7fe598004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:47.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.094+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe598009510 con 0x7fe5a0144b80 2024-01-15T18:10:47.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.094+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe59803e450 con 0x7fe5a0144b80 2024-01-15T18:10:47.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.094+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59803e800 con 0x7fe5a0144b80 2024-01-15T18:10:47.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.094+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a00afc30 con 0x7fe5a0144b80 2024-01-15T18:10:47.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.094+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a00af2a0 con 0x7fe5a0144b80 2024-01-15T18:10:47.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.095+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a00c50f0 con 0x7fe5a0144b80 2024-01-15T18:10:47.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.096+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fe59803e960 con 0x7fe5a0144b80 2024-01-15T18:10:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.097+0000 7fe596000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe57806bfb0 0x7fe57806e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.097+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fe5980ab620 con 0x7fe5a0144b80 2024-01-15T18:10:47.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.098+0000 7fe59de00640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe57806bfb0 0x7fe57806e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.099+0000 7fe59de00640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe57806bfb0 0x7fe57806e470 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe590007fd0 tx=0x7fe590009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:47.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.102+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe598077c50 con 0x7fe5a0144b80 2024-01-15T18:10:47.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.319+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5a00b3e40 con 0x7fe57806bfb0 2024-01-15T18:10:47.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.323+0000 7fe596000640 1 -- 172.21.15.2:0/406755917 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe5900050d0 con 0x7fe57806bfb0 2024-01-15T18:10:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:10:47.326 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:10:47.326 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:10:47.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:10:47.326 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:10:47.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe57806bfb0 msgr2=0x7fe57806e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe57806bfb0 0x7fe57806e470 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fe590007fd0 tx=0x7fe590009040 comp rx=0 tx=0).stop 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 msgr2=0x7fe5a0141b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 0x7fe5a0141b90 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7fe598009e50 tx=0x7fe598004a40 comp rx=0 tx=0).stop 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 shutdown_connections 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe57806bfb0 0x7fe57806e470 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5a0144b80 0x7fe5a0141b90 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5a00b30a0 0x7fe5a015aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 --2- 172.21.15.2:0/406755917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5a006cdf0 0x7fe5a00aed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 >> 172.21.15.2:0/406755917 conn(0x7fe5a00b46d0 msgr2=0x7fe5a00b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 shutdown_connections 2024-01-15T18:10:47.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.325+0000 7fe5a5000640 1 -- 172.21.15.2:0/406755917 wait complete. 2024-01-15T18:10:47.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:10:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.572+0000 7fa124a00640 1 Processor -- start 2024-01-15T18:10:47.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.572+0000 7fa124a00640 1 -- start start 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.573+0000 7fa124a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa11c0b3840 0x7fa11c15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.573+0000 7fa124a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c0cf530 con 0x7fa11c15d910 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa124a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa11c0cf810 con 0x7fa11c15eda0 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa124a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11c0cfaf0 con 0x7fa11c0b3840 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa122000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa11c15d910 0x7fa11c0d05d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41536/0 (socket says 172.21.15.2:41536) 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa122000640 1 -- 172.21.15.2:0/1300951557 learned_addr learned my addr 172.21.15.2:0/1300951557 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:10:47.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa122a00640 1 --2- 172.21.15.2:0/1300951557 >> v2:172.21.15.2:3300/0 conn(0x7fa11c0b3840 0x7fa11c15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa120c00640 1 -- 172.21.15.2:0/1300951557 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21126904 0 0) 0x7fa11c0cf530 con 0x7fa11c15d910 2024-01-15T18:10:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.574+0000 7fa120c00640 1 -- 172.21.15.2:0/1300951557 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc003520 con 0x7fa11c15d910 2024-01-15T18:10:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.575+0000 7fa122a00640 1 -- 172.21.15.2:0/1300951557 >> v1:172.21.15.137:6789/0 conn(0x7fa11c15eda0 legacy=0x7fa11c099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.575+0000 7fa122a00640 1 -- 172.21.15.2:0/1300951557 >> v1:172.21.15.2:6789/0 conn(0x7fa11c15d910 legacy=0x7fa11c0d05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.575+0000 7fa122a00640 1 -- 172.21.15.2:0/1300951557 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c0cfdd0 con 0x7fa11c0b3840 2024-01-15T18:10:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.576+0000 7fa122a00640 1 --2- 172.21.15.2:0/1300951557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c15dcf0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fa114009d90 tx=0x7fa11402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530a78a15cc2def6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.576+0000 7fa120c00640 1 -- 172.21.15.2:0/1300951557 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11402cc60 con 0x7fa11c0b3840 2024-01-15T18:10:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.576+0000 7fa120c00640 1 -- 172.21.15.2:0/1300951557 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11402cdc0 con 0x7fa11c0b3840 2024-01-15T18:10:47.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.576+0000 7fa120c00640 1 -- 172.21.15.2:0/1300951557 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114034660 con 0x7fa11c0b3840 2024-01-15T18:10:47.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 -- 172.21.15.2:0/1300951557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 msgr2=0x7fa11c15dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 --2- 172.21.15.2:0/1300951557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c15dcf0 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fa114009d90 tx=0x7fa11402beb0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 -- 172.21.15.2:0/1300951557 shutdown_connections 2024-01-15T18:10:47.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 --2- 172.21.15.2:0/1300951557 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c15dcf0 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 -- 172.21.15.2:0/1300951557 >> 172.21.15.2:0/1300951557 conn(0x7fa11c0b2b10 msgr2=0x7fa11c0b2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:47.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 -- 172.21.15.2:0/1300951557 shutdown_connections 2024-01-15T18:10:47.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.577+0000 7fa124a00640 1 -- 172.21.15.2:0/1300951557 wait complete. 2024-01-15T18:10:47.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.579+0000 7fa124a00640 1 Processor -- start 2024-01-15T18:10:47.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.580+0000 7fa124a00640 1 -- start start 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa124a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c158950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa124a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa11c15d910 0x7fa11c0b5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa124a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa11c15eda0 0x7fa11c0b5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa124a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0add50 con 0x7fa11c0b3840 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa124a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0ae030 con 0x7fa11c15d910 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa124a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa11c0ae4c0 con 0x7fa11c15eda0 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa122000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa122000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52830/0 (socket says 172.21.15.2:52830) 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.581+0000 7fa122000640 1 -- 172.21.15.2:0/3074441724 learned_addr learned my addr 172.21.15.2:0/3074441724 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa122000640 1 -- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa11c15eda0 msgr2=0x7fa11c0b5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa122000640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa11c15eda0 0x7fa11c0b5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa122000640 1 -- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa11c15d910 msgr2=0x7fa11c0b5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa122000640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa11c15d910 0x7fa11c0b5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa122000640 1 -- 172.21.15.2:0/3074441724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa114009a70 con 0x7fa11c0b3840 2024-01-15T18:10:47.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa122000640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c158950 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c500 tx=0x7fa10c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c005ce0 con 0x7fa11c0b3840 2024-01-15T18:10:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10c005e40 con 0x7fa11c0b3840 2024-01-15T18:10:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10c007330 con 0x7fa11c0b3840 2024-01-15T18:10:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.582+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11c0b1690 con 0x7fa11c0b3840 2024-01-15T18:10:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.583+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11c0c8750 con 0x7fa11c0b3840 2024-01-15T18:10:47.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.584+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11c074850 con 0x7fa11c0b3840 2024-01-15T18:10:47.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.585+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fa10c007550 con 0x7fa11c0b3840 2024-01-15T18:10:47.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.586+0000 7fa11a000640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa0f406bfb0 0x7fa0f406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:47.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.586+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fa10c087cb0 con 0x7fa11c0b3840 2024-01-15T18:10:47.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.587+0000 7fa121600640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa0f406bfb0 0x7fa0f406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:47.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.588+0000 7fa121600640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa0f406bfb0 0x7fa0f406e470 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa104000f60 tx=0x7fa1040079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:47.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.590+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa10c054630 con 0x7fa11c0b3840 2024-01-15T18:10:47.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.786+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa11c06d020 con 0x7fa11c0b3840 2024-01-15T18:10:47.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.786+0000 7fa11a000640 1 -- 172.21.15.2:0/3074441724 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa10c017070 con 0x7fa11c0b3840 2024-01-15T18:10:47.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa0f406bfb0 msgr2=0x7fa0f406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa0f406bfb0 0x7fa0f406e470 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fa104000f60 tx=0x7fa1040079b0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 msgr2=0x7fa11c158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c158950 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7fa10c00c500 tx=0x7fa10c00c9c0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 shutdown_connections 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa0f406bfb0 0x7fa0f406e470 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa11c15eda0 0x7fa11c0b5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa11c15d910 0x7fa11c0b5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 --2- 172.21.15.2:0/3074441724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa11c0b3840 0x7fa11c158950 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 >> 172.21.15.2:0/3074441724 conn(0x7fa11c0b2b10 msgr2=0x7fa11c0cba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 shutdown_connections 2024-01-15T18:10:47.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:47.789+0000 7fa124a00640 1 -- 172.21.15.2:0/3074441724 wait complete. 2024-01-15T18:10:47.811 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:10:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.027+0000 7fd541600640 1 Processor -- start 2024-01-15T18:10:48.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.027+0000 7fd541600640 1 -- start start 2024-01-15T18:10:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd541600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd53c143620 0x7fd53c0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd541600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd53c0b2a90 0x7fd53c071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd541600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd541600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c071c20 con 0x7fd53c143620 2024-01-15T18:10:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd541600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c0b2fa0 con 0x7fd53c0b2a90 2024-01-15T18:10:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd541600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0b3280 con 0x7fd53c071770 2024-01-15T18:10:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd53be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd53c143620 0x7fd53c0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd53be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd53c143620 0x7fd53c0b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52842/0 (socket says 172.21.15.2:52842) 2024-01-15T18:10:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.029+0000 7fd53be00640 1 -- 172.21.15.2:0/1304255712 learned_addr learned my addr 172.21.15.2:0/1304255712 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:48.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd533200640 1 --2- 172.21.15.2:0/1304255712 >> v2:172.21.15.137:3300/0 conn(0x7fd53c0b2a90 0x7fd53c071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd540c00640 1 --2- 172.21.15.2:0/1304255712 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd53be00640 1 -- 172.21.15.2:0/1304255712 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 msgr2=0x7fd53c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd53be00640 1 --2- 172.21.15.2:0/1304255712 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd53be00640 1 -- 172.21.15.2:0/1304255712 >> v2:172.21.15.137:3300/0 conn(0x7fd53c0b2a90 msgr2=0x7fd53c071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd53be00640 1 --2- 172.21.15.2:0/1304255712 >> v2:172.21.15.137:3300/0 conn(0x7fd53c0b2a90 0x7fd53c071230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.030+0000 7fd53be00640 1 -- 172.21.15.2:0/1304255712 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0b3560 con 0x7fd53c143620 2024-01-15T18:10:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.031+0000 7fd53be00640 1 --2- 172.21.15.2:0/1304255712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c143620 0x7fd53c0b2550 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd53c06b380 tx=0x7fd52c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62473fe6de4d192e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.031+0000 7fd53b400640 1 -- 172.21.15.2:0/1304255712 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c031870 con 0x7fd53c143620 2024-01-15T18:10:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd53b400640 1 -- 172.21.15.2:0/1304255712 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c0319d0 con 0x7fd53c143620 2024-01-15T18:10:48.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd53b400640 1 -- 172.21.15.2:0/1304255712 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c031d60 con 0x7fd53c143620 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 -- 172.21.15.2:0/1304255712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c143620 msgr2=0x7fd53c0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 --2- 172.21.15.2:0/1304255712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c143620 0x7fd53c0b2550 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7fd53c06b380 tx=0x7fd52c02b910 comp rx=0 tx=0).stop 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 -- 172.21.15.2:0/1304255712 shutdown_connections 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 --2- 172.21.15.2:0/1304255712 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c070110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 --2- 172.21.15.2:0/1304255712 >> v2:172.21.15.137:3300/0 conn(0x7fd53c0b2a90 0x7fd53c071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 --2- 172.21.15.2:0/1304255712 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c143620 0x7fd53c0b2550 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 -- 172.21.15.2:0/1304255712 >> 172.21.15.2:0/1304255712 conn(0x7fd53c0b0840 msgr2=0x7fd53c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.032+0000 7fd541600640 1 -- 172.21.15.2:0/1304255712 shutdown_connections 2024-01-15T18:10:48.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.033+0000 7fd541600640 1 -- 172.21.15.2:0/1304255712 wait complete. 2024-01-15T18:10:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.034+0000 7fd541600640 1 Processor -- start 2024-01-15T18:10:48.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.034+0000 7fd541600640 1 -- start start 2024-01-15T18:10:48.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.035+0000 7fd541600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.035+0000 7fd541600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 0x7fd53c149d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.035+0000 7fd541600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd53c143620 0x7fd53c14a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.035+0000 7fd541600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c14a800 con 0x7fd53c0b2a90 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.035+0000 7fd541600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c14aab0 con 0x7fd53c143620 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.035+0000 7fd541600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c14ad90 con 0x7fd53c071770 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 0x7fd53c149d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 0x7fd53c149d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52844/0 (socket says 172.21.15.2:52844) 2024-01-15T18:10:48.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 -- 172.21.15.2:0/2627782334 learned_addr learned my addr 172.21.15.2:0/2627782334 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:48.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd53be00640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd540c00640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd53c143620 0x7fd53c14a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 -- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 msgr2=0x7fd53c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 -- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd53c143620 msgr2=0x7fd53c14a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd53c143620 0x7fd53c14a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.036+0000 7fd533200640 1 -- 172.21.15.2:0/2627782334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c0094d0 con 0x7fd53c0b2a90 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.037+0000 7fd533200640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 0x7fd53c149d80 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd52400d7d0 tx=0x7fd52400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.037+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52400b6c0 con 0x7fd53c0b2a90 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.037+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52400b840 con 0x7fd53c0b2a90 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.037+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52400bbf0 con 0x7fd53c0b2a90 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.037+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c14b070 con 0x7fd53c0b2a90 2024-01-15T18:10:48.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.037+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c14b630 con 0x7fd53c0b2a90 2024-01-15T18:10:48.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.038+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c14b7f0 con 0x7fd53c0b2a90 2024-01-15T18:10:48.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.039+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fd52400bd50 con 0x7fd53c0b2a90 2024-01-15T18:10:48.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.040+0000 7fd538c00640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd51406bfb0 0x7fd51406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.040+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fd524088d80 con 0x7fd53c0b2a90 2024-01-15T18:10:48.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.041+0000 7fd53be00640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd51406bfb0 0x7fd51406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.042+0000 7fd53be00640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd51406bfb0 0x7fd51406e470 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd52c036000 tx=0x7fd52c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.045+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5240553b0 con 0x7fd53c0b2a90 2024-01-15T18:10:48.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.270+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fd53c143a00 con 0x7fd51406bfb0 2024-01-15T18:10:48.285 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:10:48.281+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:48.285 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:10:48.281+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:48.285 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:10:48.281+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:48.285 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:10:48.281+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:48.285 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:10:48.281+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:10:48.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.354+0000 7fd538c00640 1 -- 172.21.15.2:0/2627782334 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd52c031bc0 con 0x7fd51406bfb0 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.357+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd51406bfb0 msgr2=0x7fd51406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.357+0000 7fd541600640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd51406bfb0 0x7fd51406e470 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd52c036000 tx=0x7fd52c03c040 comp rx=0 tx=0).stop 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.357+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 msgr2=0x7fd53c149d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.357+0000 7fd541600640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 0x7fd53c149d80 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7fd52400d7d0 tx=0x7fd52400dc90 comp rx=0 tx=0).stop 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 shutdown_connections 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd51406bfb0 0x7fd51406e470 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd53c143620 0x7fd53c14a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd53c0b2a90 0x7fd53c149d80 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 --2- 172.21.15.2:0/2627782334 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd53c071770 0x7fd53c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 >> 172.21.15.2:0/2627782334 conn(0x7fd53c0b0840 msgr2=0x7fd53c0b1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:48.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 shutdown_connections 2024-01-15T18:10:48.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.358+0000 7fd541600640 1 -- 172.21.15.2:0/2627782334 wait complete. 2024-01-15T18:10:48.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:10:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.602+0000 7ffb73e00640 1 Processor -- start 2024-01-15T18:10:48.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.603+0000 7ffb73e00640 1 -- start start 2024-01-15T18:10:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.604+0000 7ffb73e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffb74143d20 0x7ffb7415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.604+0000 7ffb73e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ffb740720d0 0x7ffb7415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.604+0000 7ffb73e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb74071090 con 0x7ffb741447d0 2024-01-15T18:10:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.604+0000 7ffb73e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb74071430 con 0x7ffb74143d20 2024-01-15T18:10:48.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.604+0000 7ffb73e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb74144100 con 0x7ffb740720d0 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffb74143d20 0x7ffb7415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ffb74143d20 0x7ffb7415eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52856/0 (socket says 172.21.15.2:52856) 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 -- 172.21.15.2:0/4196718916 learned_addr learned my addr 172.21.15.2:0/4196718916 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 -- 172.21.15.2:0/4196718916 >> v2:172.21.15.137:3300/0 conn(0x7ffb740720d0 msgr2=0x7ffb7415f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb6be00640 1 --2- 172.21.15.2:0/4196718916 >> v2:172.21.15.137:3300/0 conn(0x7ffb740720d0 0x7ffb7415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 --2- 172.21.15.2:0/4196718916 >> v2:172.21.15.137:3300/0 conn(0x7ffb740720d0 0x7ffb7415f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 -- 172.21.15.2:0/4196718916 >> v1:172.21.15.137:6789/0 conn(0x7ffb741447d0 legacy=0x7ffb740cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb72a00640 1 -- 172.21.15.2:0/4196718916 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7406d160 con 0x7ffb74143d20 2024-01-15T18:10:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.605+0000 7ffb6be00640 1 --2- 172.21.15.2:0/4196718916 >> v2:172.21.15.137:3300/0 conn(0x7ffb740720d0 0x7ffb7415f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:10:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.606+0000 7ffb72a00640 1 --2- 172.21.15.2:0/4196718916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb7415eb50 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ffb7404f900 tx=0x7ffb6402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=441672b32b31972 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.607+0000 7ffb72000640 1 -- 172.21.15.2:0/4196718916 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640317e0 con 0x7ffb74143d20 2024-01-15T18:10:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.607+0000 7ffb72000640 1 -- 172.21.15.2:0/4196718916 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb64031940 con 0x7ffb74143d20 2024-01-15T18:10:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.607+0000 7ffb72000640 1 -- 172.21.15.2:0/4196718916 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64031cd0 con 0x7ffb74143d20 2024-01-15T18:10:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.607+0000 7ffb73e00640 1 -- 172.21.15.2:0/4196718916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 msgr2=0x7ffb7415eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.607+0000 7ffb73e00640 1 --2- 172.21.15.2:0/4196718916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb7415eb50 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7ffb7404f900 tx=0x7ffb6402b880 comp rx=0 tx=0).stop 2024-01-15T18:10:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.608+0000 7ffb73e00640 1 -- 172.21.15.2:0/4196718916 shutdown_connections 2024-01-15T18:10:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.608+0000 7ffb73e00640 1 --2- 172.21.15.2:0/4196718916 >> v2:172.21.15.137:3300/0 conn(0x7ffb740720d0 0x7ffb7415f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.608+0000 7ffb73e00640 1 --2- 172.21.15.2:0/4196718916 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb7415eb50 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.608+0000 7ffb73e00640 1 -- 172.21.15.2:0/4196718916 >> 172.21.15.2:0/4196718916 conn(0x7ffb740b0840 msgr2=0x7ffb740b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.608+0000 7ffb73e00640 1 -- 172.21.15.2:0/4196718916 shutdown_connections 2024-01-15T18:10:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.608+0000 7ffb73e00640 1 -- 172.21.15.2:0/4196718916 wait complete. 2024-01-15T18:10:48.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.609+0000 7ffb73e00640 1 Processor -- start 2024-01-15T18:10:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.610+0000 7ffb73e00640 1 -- start start 2024-01-15T18:10:48.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb740720d0 0x7ffb740b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb74159820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb741447d0 0x7ffb7415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740af480 con 0x7ffb74143d20 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740b38f0 con 0x7ffb741447d0 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb740ae3e0 con 0x7ffb740720d0 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb74159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb74159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52864/0 (socket says 172.21.15.2:52864) 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 -- 172.21.15.2:0/2999891489 learned_addr learned my addr 172.21.15.2:0/2999891489 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb72a00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb740720d0 0x7ffb740b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb73400640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb741447d0 0x7ffb7415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 -- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb740720d0 msgr2=0x7ffb740b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb740720d0 0x7ffb740b2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 -- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb741447d0 msgr2=0x7ffb7415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb741447d0 0x7ffb7415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.611+0000 7ffb6be00640 1 -- 172.21.15.2:0/2999891489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb640094d0 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.612+0000 7ffb6be00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb74159820 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00c820 tx=0x7ffb5c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.612+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c007a30 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.612+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5c007b90 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.612+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb5c00ce10 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.612+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb740ae6c0 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.612+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb740aec80 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.613+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb74070b10 con 0x7ffb74143d20 2024-01-15T18:10:48.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.614+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7ffb5c0113f0 con 0x7ffb74143d20 2024-01-15T18:10:48.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.615+0000 7ffb6aa00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb4c06bfb0 0x7ffb4c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:10:48.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.615+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5793+0+0 (secure 0 0 0) 0x7ffb5c088870 con 0x7ffb74143d20 2024-01-15T18:10:48.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.616+0000 7ffb72a00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb4c06bfb0 0x7ffb4c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:10:48.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.617+0000 7ffb72a00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb4c06bfb0 0x7ffb4c06e470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffb64036000 tx=0x7ffb6403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:10:48.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.620+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb5c054ff0 con 0x7ffb74143d20 2024-01-15T18:10:48.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.845+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb7415f720 con 0x7ffb4c06bfb0 2024-01-15T18:10:48.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.857+0000 7ffb6aa00640 1 -- 172.21.15.2:0/2999891489 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7ffb64031cd0 con 0x7ffb4c06bfb0 2024-01-15T18:10:48.863 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb4c06bfb0 msgr2=0x7ffb4c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb4c06bfb0 0x7ffb4c06e470 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7ffb64036000 tx=0x7ffb6403d040 comp rx=0 tx=0).stop 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 msgr2=0x7ffb74159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb74159820 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7ffb5c00c820 tx=0x7ffb5c00cce0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 shutdown_connections 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ffb4c06bfb0 0x7ffb4c06e470 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ffb741447d0 0x7ffb7415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ffb74143d20 0x7ffb74159820 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.859+0000 7ffb73e00640 1 --2- 172.21.15.2:0/2999891489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ffb740720d0 0x7ffb740b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:10:48.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.860+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 >> 172.21.15.2:0/2999891489 conn(0x7ffb740b0840 msgr2=0x7ffb7415a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:10:48.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.860+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 shutdown_connections 2024-01-15T18:10:48.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:10:48.860+0000 7ffb73e00640 1 -- 172.21.15.2:0/2999891489 wait complete. 2024-01-15T18:10:48.883 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:10:48.884 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:10:48.884 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:10:48.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:10:48.939 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:11:02.447 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:02.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:05.268 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:11:05.268 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:11:05.268 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.2567 s, 66.0 MB/s 2024-01-15T18:11:05.271 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:11:05.271 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:05.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:05.326 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:11:19.032 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:11:19.032 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:11:19.032 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.6487 s, 78.7 MB/s 2024-01-15T18:11:19.034 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:11:19.034 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:11:19.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:19.090 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:11:32.603 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:32.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:11:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:11:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:11:32.799 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.6485 s, 78.7 MB/s 2024-01-15T18:11:32.801 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-15T18:11:33.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.038+0000 7f8cbfe00640 1 Processor -- start 2024-01-15T18:11:33.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.039+0000 7f8cbfe00640 1 -- start start 2024-01-15T18:11:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.040+0000 7f8cbfe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8cb80b4c70 0x7f8cb80d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.040+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb815d730 0x7f8cb80b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.040+0000 7f8cbfe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cb80b5590 con 0x7f8cb80b8c30 2024-01-15T18:11:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.040+0000 7f8cbfe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cb80d30b0 con 0x7f8cb80b4c70 2024-01-15T18:11:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.040+0000 7f8cbfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80d1900 con 0x7f8cb815d730 2024-01-15T18:11:33.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbde00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f8cb80b8c30 0x7f8cb80d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51208/0 (socket says 172.21.15.2:51208) 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbd400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8cb80b4c70 0x7f8cb80d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbde00640 1 -- 172.21.15.2:0/31851170 learned_addr learned my addr 172.21.15.2:0/31851170 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbca00640 1 --2- 172.21.15.2:0/31851170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb815d730 0x7f8cb80b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbd400640 1 -- 172.21.15.2:0/31851170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb815d730 msgr2=0x7f8cb80b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbd400640 1 --2- 172.21.15.2:0/31851170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb815d730 0x7f8cb80b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbd400640 1 -- 172.21.15.2:0/31851170 >> v1:172.21.15.2:6789/0 conn(0x7f8cb80b8c30 legacy=0x7f8cb80d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.041+0000 7f8cbd400640 1 -- 172.21.15.2:0/31851170 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb80d1be0 con 0x7f8cb80b4c70 2024-01-15T18:11:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.042+0000 7f8cbd400640 1 --2- 172.21.15.2:0/31851170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80d1270 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80097a0 tx=0x7f8ca802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418e0cc12397bca1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.043+0000 7f8cb7e00640 1 -- 172.21.15.2:0/31851170 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca802d040 con 0x7f8cb80b4c70 2024-01-15T18:11:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.043+0000 7f8cb7e00640 1 -- 172.21.15.2:0/31851170 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ca8031810 con 0x7f8cb80b4c70 2024-01-15T18:11:33.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.043+0000 7f8cb7e00640 1 -- 172.21.15.2:0/31851170 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8031ba0 con 0x7f8cb80b4c70 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.043+0000 7f8cbfe00640 1 -- 172.21.15.2:0/31851170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb80b4c70 msgr2=0x7f8cb80d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.043+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/31851170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80d1270 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80097a0 tx=0x7f8ca802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.044+0000 7f8cbfe00640 1 -- 172.21.15.2:0/31851170 shutdown_connections 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.044+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/31851170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb815d730 0x7f8cb80b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.044+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/31851170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80d1270 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.044+0000 7f8cbfe00640 1 -- 172.21.15.2:0/31851170 >> 172.21.15.2:0/31851170 conn(0x7f8cb8072210 msgr2=0x7f8cb80cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.044+0000 7f8cbfe00640 1 -- 172.21.15.2:0/31851170 shutdown_connections 2024-01-15T18:11:33.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.044+0000 7f8cbfe00640 1 -- 172.21.15.2:0/31851170 wait complete. 2024-01-15T18:11:33.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.046+0000 7f8cbfe00640 1 Processor -- start 2024-01-15T18:11:33.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.046+0000 7f8cbfe00640 1 -- start start 2024-01-15T18:11:33.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb80b8c30 0x7f8cb8160320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 0x7f8cb80c7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbfe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c8fc0 con 0x7f8cb815d730 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbfe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c8a00 con 0x7f8cb80b4c70 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbfe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80c8ce0 con 0x7f8cb80b8c30 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 0x7f8cb80c7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 0x7f8cb80c7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33070/0 (socket says 172.21.15.2:33070) 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbde00640 1 -- 172.21.15.2:0/679631611 learned_addr learned my addr 172.21.15.2:0/679631611 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.047+0000 7f8cbd400640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbde00640 1 -- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb80b8c30 msgr2=0x7f8cb8160320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbca00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb80b8c30 0x7f8cb8160320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbde00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb80b8c30 0x7f8cb8160320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbde00640 1 -- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cb80b4c70 msgr2=0x7f8cb80b7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbde00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80b7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbde00640 1 -- 172.21.15.2:0/679631611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8009400 con 0x7f8cb815d730 2024-01-15T18:11:33.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbde00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 0x7f8cb80c7f40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000cbc0 tx=0x7f8cb000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:33.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000ad40 con 0x7f8cb815d730 2024-01-15T18:11:33.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb0012400 con 0x7f8cb815d730 2024-01-15T18:11:33.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb815fb20 con 0x7f8cb815d730 2024-01-15T18:11:33.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.048+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb000aea0 con 0x7f8cb815d730 2024-01-15T18:11:33.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.049+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb80c61c0 con 0x7f8cb815d730 2024-01-15T18:11:33.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.050+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb815f980 con 0x7f8cb815d730 2024-01-15T18:11:33.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.051+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 31) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f8cb0007070 con 0x7f8cb815d730 2024-01-15T18:11:33.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.052+0000 7f8cb5600640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8c9006c040 0x7f8c9006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.052+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8cb00888e0 con 0x7f8cb815d730 2024-01-15T18:11:33.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.052+0000 7f8cbd400640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8c9006c040 0x7f8c9006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.053+0000 7f8cbd400640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8c9006c040 0x7f8c9006e500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8036000 tx=0x7f8ca803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:33.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.058+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cb815f980 con 0x7f8cb815d730 2024-01-15T18:11:33.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.255+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f8cb8160040 con 0x7f8cb815d730 2024-01-15T18:11:33.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.263+0000 7f8cb5600640 1 -- 172.21.15.2:0/679631611 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v31)=0 v31) v1 ==== 135+0+0 (secure 0 0 0) 0x7f8cb8160040 con 0x7f8cb815d730 2024-01-15T18:11:33.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8c9006c040 msgr2=0x7f8c9006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8c9006c040 0x7f8c9006e500 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8036000 tx=0x7f8ca803c040 comp rx=0 tx=0).stop 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 msgr2=0x7f8cb80c7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 0x7f8cb80c7f40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000cbc0 tx=0x7f8cb000a430 comp rx=0 tx=0).stop 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 shutdown_connections 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8c9006c040 0x7f8c9006e500 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8cb815d730 0x7f8cb80c7f40 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8cb80b8c30 0x7f8cb8160320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 --2- 172.21.15.2:0/679631611 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8cb80b4c70 0x7f8cb80b7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 >> 172.21.15.2:0/679631611 conn(0x7f8cb8072210 msgr2=0x7f8cb80b64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 shutdown_connections 2024-01-15T18:11:33.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.266+0000 7f8cbfe00640 1 -- 172.21.15.2:0/679631611 wait complete. 2024-01-15T18:11:33.288 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:11:33.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.503+0000 7f8edd800640 1 Processor -- start 2024-01-15T18:11:33.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.503+0000 7f8edd800640 1 -- start start 2024-01-15T18:11:33.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8edd800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8ed8144c00 0x7f8ed80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8edd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8edd800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8ed815c450 0x7f8ed806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8edd800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80ce230 con 0x7f8ed815c450 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8edd800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed80ce510 con 0x7f8ed8144c00 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8edd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80ce7f0 con 0x7f8ed80cdd70 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8ed7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8ed815c450 0x7f8ed806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8ed7400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8ed815c450 0x7f8ed806ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33080/0 (socket says 172.21.15.2:33080) 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8ecfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8ed7400640 1 -- 172.21.15.2:0/3071567561 learned_addr learned my addr 172.21.15.2:0/3071567561 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:33.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.505+0000 7f8ed6a00640 1 --2- 172.21.15.2:0/3071567561 >> v2:172.21.15.137:3300/0 conn(0x7f8ed8144c00 0x7f8ed80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.506+0000 7f8ed7400640 1 -- 172.21.15.2:0/3071567561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 msgr2=0x7f8ed806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:33.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.506+0000 7f8ed7400640 1 --2- 172.21.15.2:0/3071567561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.506+0000 7f8ed7400640 1 -- 172.21.15.2:0/3071567561 >> v2:172.21.15.137:3300/0 conn(0x7f8ed8144c00 msgr2=0x7f8ed80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:33.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.506+0000 7f8ed7400640 1 --2- 172.21.15.2:0/3071567561 >> v2:172.21.15.137:3300/0 conn(0x7f8ed8144c00 0x7f8ed80cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.506+0000 7f8ed7400640 1 -- 172.21.15.2:0/3071567561 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed80705f0 con 0x7f8ed815c450 2024-01-15T18:11:33.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.507+0000 7f8ed7400640 1 --2- 172.21.15.2:0/3071567561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed815c450 0x7f8ed806ce30 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80b1c60 tx=0x7f8ed002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef64e8c7564d27dd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:33.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.507+0000 7f8ed6000640 1 -- 172.21.15.2:0/3071567561 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed002ca70 con 0x7f8ed815c450 2024-01-15T18:11:33.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8ed6000640 1 -- 172.21.15.2:0/3071567561 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed002cbd0 con 0x7f8ed815c450 2024-01-15T18:11:33.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8ed6000640 1 -- 172.21.15.2:0/3071567561 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed00343f0 con 0x7f8ed815c450 2024-01-15T18:11:33.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 -- 172.21.15.2:0/3071567561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed815c450 msgr2=0x7f8ed806ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 --2- 172.21.15.2:0/3071567561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed815c450 0x7f8ed806ce30 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80b1c60 tx=0x7f8ed002bed0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 -- 172.21.15.2:0/3071567561 shutdown_connections 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 --2- 172.21.15.2:0/3071567561 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed815c450 0x7f8ed806ce30 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 --2- 172.21.15.2:0/3071567561 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 --2- 172.21.15.2:0/3071567561 >> v2:172.21.15.137:3300/0 conn(0x7f8ed8144c00 0x7f8ed80cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.508+0000 7f8edd800640 1 -- 172.21.15.2:0/3071567561 >> 172.21.15.2:0/3071567561 conn(0x7f8ed80b45f0 msgr2=0x7f8ed80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.509+0000 7f8edd800640 1 -- 172.21.15.2:0/3071567561 shutdown_connections 2024-01-15T18:11:33.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.509+0000 7f8edd800640 1 -- 172.21.15.2:0/3071567561 wait complete. 2024-01-15T18:11:33.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.510+0000 7f8edd800640 1 Processor -- start 2024-01-15T18:11:33.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.510+0000 7f8edd800640 1 -- start start 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.511+0000 7f8edd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed813b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.511+0000 7f8edd800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 0x7f8ed8141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.511+0000 7f8edd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ed815c450 0x7f8ed815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.511+0000 7f8edd800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed813bcc0 con 0x7f8ed8144c00 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.511+0000 7f8edd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed81544a0 con 0x7f8ed815c450 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.511+0000 7f8edd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8141b60 con 0x7f8ed80cdd70 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 0x7f8ed8141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 0x7f8ed8141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33090/0 (socket says 172.21.15.2:33090) 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 -- 172.21.15.2:0/1086526903 learned_addr learned my addr 172.21.15.2:0/1086526903 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ed6a00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed813b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ed7400640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ed815c450 0x7f8ed815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 -- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 msgr2=0x7f8ed813b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed813b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 -- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ed815c450 msgr2=0x7f8ed815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ed815c450 0x7f8ed815cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.512+0000 7f8ecfe00640 1 -- 172.21.15.2:0/1086526903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009b30 con 0x7f8ed8144c00 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.513+0000 7f8ecfe00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 0x7f8ed8141620 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e8c0 tx=0x7f8ec000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.513+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000c940 con 0x7f8ed8144c00 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.513+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec000caa0 con 0x7f8ed8144c00 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.513+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec000ce30 con 0x7f8ed8144c00 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.513+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed813b9e0 con 0x7f8ed8144c00 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.513+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8154130 con 0x7f8ed8144c00 2024-01-15T18:11:33.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.514+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed8154630 con 0x7f8ed8144c00 2024-01-15T18:11:33.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.515+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 31) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f8ec0013470 con 0x7f8ed8144c00 2024-01-15T18:11:33.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.516+0000 7f8ecea00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8eb006c040 0x7f8eb006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:33.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.516+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8ec0088b10 con 0x7f8ed8144c00 2024-01-15T18:11:33.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.517+0000 7f8ed6a00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8eb006c040 0x7f8eb006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:33.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.518+0000 7f8ed6a00640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8eb006c040 0x7f8eb006e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009ad0 tx=0x7f8ec8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:33.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.520+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ec0055120 con 0x7f8ed8144c00 2024-01-15T18:11:33.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.746+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ed80686f0 con 0x7f8eb006c040 2024-01-15T18:11:33.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.786+0000 7f8ecea00640 1 -- 172.21.15.2:0/1086526903 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8ed80686f0 con 0x7f8eb006c040 2024-01-15T18:11:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8eb006c040 msgr2=0x7f8eb006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8eb006c040 0x7f8eb006e500 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009ad0 tx=0x7f8ec8008040 comp rx=0 tx=0).stop 2024-01-15T18:11:33.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 msgr2=0x7f8ed8141620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 0x7f8ed8141620 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e8c0 tx=0x7f8ec000ed80 comp rx=0 tx=0).stop 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 shutdown_connections 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8eb006c040 0x7f8eb006e500 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8ed815c450 0x7f8ed815cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8ed8144c00 0x7f8ed8141620 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 --2- 172.21.15.2:0/1086526903 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8ed80cdd70 0x7f8ed813b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 >> 172.21.15.2:0/1086526903 conn(0x7f8ed80b45f0 msgr2=0x7f8ed8143200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 shutdown_connections 2024-01-15T18:11:33.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:33.789+0000 7f8edd800640 1 -- 172.21.15.2:0/1086526903 wait complete. 2024-01-15T18:11:33.811 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:11:34.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.030+0000 7f5c7e400640 1 Processor -- start 2024-01-15T18:11:34.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.030+0000 7f5c7e400640 1 -- start start 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c7e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5c781439f0 0x7f5c780702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c7e400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5c780b3420 0x7f5c780cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c7e400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7815dd70 con 0x7f5c780ccc70 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c7e400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c780b38b0 con 0x7f5c780b3420 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c7e400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c780711b0 con 0x7f5c781439f0 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c6fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5c780b3420 0x7f5c780cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:34.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c6fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5c780b3420 0x7f5c780cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33100/0 (socket says 172.21.15.2:33100) 2024-01-15T18:11:34.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.032+0000 7f5c6fe00640 1 -- 172.21.15.2:0/2266633011 learned_addr learned my addr 172.21.15.2:0/2266633011 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:34.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.033+0000 7f5c6fe00640 1 -- 172.21.15.2:0/2266633011 >> v2:172.21.15.137:3300/0 conn(0x7f5c781439f0 msgr2=0x7f5c780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:34.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.033+0000 7f5c6fe00640 1 --2- 172.21.15.2:0/2266633011 >> v2:172.21.15.137:3300/0 conn(0x7f5c781439f0 0x7f5c780702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.033+0000 7f5c6fe00640 1 -- 172.21.15.2:0/2266633011 >> v1:172.21.15.137:6789/0 conn(0x7f5c780ccc70 legacy=0x7f5c7806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:34.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.033+0000 7f5c6fe00640 1 -- 172.21.15.2:0/2266633011 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c78143dd0 con 0x7f5c780b3420 2024-01-15T18:11:34.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.034+0000 7f5c6fe00640 1 --2- 172.21.15.2:0/2266633011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c780cc730 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f5c780b28d0 tx=0x7f5c6002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5b945f5d65ee94c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:34.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.034+0000 7f5c76a00640 1 -- 172.21.15.2:0/2266633011 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60031870 con 0x7f5c780b3420 2024-01-15T18:11:34.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.034+0000 7f5c76a00640 1 -- 172.21.15.2:0/2266633011 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c600319d0 con 0x7f5c780b3420 2024-01-15T18:11:34.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.034+0000 7f5c76a00640 1 -- 172.21.15.2:0/2266633011 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60031d60 con 0x7f5c780b3420 2024-01-15T18:11:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 -- 172.21.15.2:0/2266633011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 msgr2=0x7f5c780cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2266633011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c780cc730 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f5c780b28d0 tx=0x7f5c6002b910 comp rx=0 tx=0).stop 2024-01-15T18:11:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 -- 172.21.15.2:0/2266633011 shutdown_connections 2024-01-15T18:11:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2266633011 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c780cc730 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2266633011 >> v2:172.21.15.137:3300/0 conn(0x7f5c781439f0 0x7f5c780702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 -- 172.21.15.2:0/2266633011 >> 172.21.15.2:0/2266633011 conn(0x7f5c780b45f0 msgr2=0x7f5c780b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:34.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 -- 172.21.15.2:0/2266633011 shutdown_connections 2024-01-15T18:11:34.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.035+0000 7f5c7e400640 1 -- 172.21.15.2:0/2266633011 wait complete. 2024-01-15T18:11:34.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.037+0000 7f5c7e400640 1 Processor -- start 2024-01-15T18:11:34.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.037+0000 7f5c7e400640 1 -- start start 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.038+0000 7f5c7e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c78140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.038+0000 7f5c7e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c780ccc70 0x7f5c781400e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.038+0000 7f5c7e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c781439f0 0x7f5c78140620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.038+0000 7f5c7e400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780a8700 con 0x7f5c780b3420 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.038+0000 7f5c7e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7815a0e0 con 0x7f5c781439f0 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.038+0000 7f5c7e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c780adb20 con 0x7f5c780ccc70 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c78140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c78140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33110/0 (socket says 172.21.15.2:33110) 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 -- 172.21.15.2:0/2512608697 learned_addr learned my addr 172.21.15.2:0/2512608697 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77e00640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c781439f0 0x7f5c78140620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:34.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c6fe00640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c780ccc70 0x7f5c781400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 -- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c780ccc70 msgr2=0x7f5c781400e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c780ccc70 0x7f5c781400e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 -- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c781439f0 msgr2=0x7f5c78140620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c781439f0 0x7f5c78140620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 -- 172.21.15.2:0/2512608697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c600094d0 con 0x7f5c780b3420 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77e00640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c781439f0 0x7f5c78140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c6fe00640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c780ccc70 0x7f5c781400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.039+0000 7f5c77400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c78140e50 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800b410 tx=0x7f5c6800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.040+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68011830 con 0x7f5c780b3420 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.040+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c680119b0 con 0x7f5c780b3420 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.040+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c68011d60 con 0x7f5c780b3420 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.040+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c780ade00 con 0x7f5c780b3420 2024-01-15T18:11:34.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.040+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c78141390 con 0x7f5c780b3420 2024-01-15T18:11:34.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.041+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c780ae320 con 0x7f5c780b3420 2024-01-15T18:11:34.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.042+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 31) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f5c6801a400 con 0x7f5c780b3420 2024-01-15T18:11:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.043+0000 7f5c6f400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c5006c040 0x7f5c5006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:34.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.044+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f5c68088a50 con 0x7f5c780b3420 2024-01-15T18:11:34.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.044+0000 7f5c6fe00640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c5006c040 0x7f5c5006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:34.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.045+0000 7f5c6fe00640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c5006c040 0x7f5c5006e500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5c60036000 tx=0x7f5c6003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:34.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.048+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c780ae320 con 0x7f5c780b3420 2024-01-15T18:11:34.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.273+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c78070ed0 con 0x7f5c5006c040 2024-01-15T18:11:34.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.286+0000 7f5c6f400640 1 -- 172.21.15.2:0/2512608697 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f5c60031d10 con 0x7f5c5006c040 2024-01-15T18:11:34.289 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:11:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c5006c040 msgr2=0x7f5c5006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c5006c040 0x7f5c5006e500 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f5c60036000 tx=0x7f5c6003c040 comp rx=0 tx=0).stop 2024-01-15T18:11:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 msgr2=0x7f5c78140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c78140e50 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f5c6800b410 tx=0x7f5c6800b8d0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 shutdown_connections 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5c5006c040 0x7f5c5006e500 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5c781439f0 0x7f5c78140620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c780ccc70 0x7f5c781400e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 --2- 172.21.15.2:0/2512608697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c780b3420 0x7f5c78140e50 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.288+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 >> 172.21.15.2:0/2512608697 conn(0x7f5c780b45f0 msgr2=0x7f5c780b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.289+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 shutdown_connections 2024-01-15T18:11:34.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:34.289+0000 7f5c7e400640 1 -- 172.21.15.2:0/2512608697 wait complete. 2024-01-15T18:11:34.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:34.309 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e) 2024-01-15T18:11:34.369 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:11:34.369 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:11:34.369 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:11:34.370 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:11:36.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:36.372 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e) 2024-01-15T18:11:36.441 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:11:36.441 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:11:36.441 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:11:36.443 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:11:38.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:38.444 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e) 2024-01-15T18:11:38.512 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:11:38.512 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:11:38.512 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:11:38.513 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:11:40.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:11:40.514 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e) 2024-01-15T18:11:40.581 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/70890500-8d7f-47a7-90f0-ff003a69915e 2024-01-15T18:11:40.581 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:11:40.581 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:11:40.583 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:11:40.583 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T18:11:40.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.813+0000 7fa87fe00640 1 Processor -- start 2024-01-15T18:11:40.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.813+0000 7fa87fe00640 1 -- start start 2024-01-15T18:11:40.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.815+0000 7fa87fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa88015d270 0x7fa8800b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:40.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.815+0000 7fa87fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa880161890 0x7fa880160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:40.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.815+0000 7fa87fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa88006c5e0 con 0x7fa8800b2b60 2024-01-15T18:11:40.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.815+0000 7fa87fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88006c8c0 con 0x7fa88015d270 2024-01-15T18:11:40.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.815+0000 7fa87fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa880071c00 con 0x7fa880161890 2024-01-15T18:11:40.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.816+0000 7fa877e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa880161890 0x7fa880160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:40.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.816+0000 7fa87ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa88015d270 0x7fa8800b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:40.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.816+0000 7fa87ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa88015d270 0x7fa8800b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:50348/0 (socket says 172.21.15.2:50348) 2024-01-15T18:11:40.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.816+0000 7fa87ea00640 1 -- 172.21.15.2:0/3770238956 learned_addr learned my addr 172.21.15.2:0/3770238956 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:40.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.817+0000 7fa87ea00640 1 -- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa880161890 msgr2=0x7fa880160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:40.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.817+0000 7fa87ea00640 1 --2- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa880161890 0x7fa880160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:40.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.817+0000 7fa87ea00640 1 -- 172.21.15.2:0/3770238956 >> v1:172.21.15.137:6789/0 conn(0x7fa8800b2b60 legacy=0x7fa8800702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:40.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.817+0000 7fa87ea00640 1 -- 172.21.15.2:0/3770238956 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa880071ee0 con 0x7fa88015d270 2024-01-15T18:11:40.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.818+0000 7fa87ea00640 1 --2- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa88015d270 0x7fa8800b2620 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa870009870 tx=0x7fa87002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bb4048d805f910 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:40.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87e000640 1 -- 172.21.15.2:0/3770238956 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8700317c0 con 0x7fa88015d270 2024-01-15T18:11:40.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87e000640 1 -- 172.21.15.2:0/3770238956 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa870031920 con 0x7fa88015d270 2024-01-15T18:11:40.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87e000640 1 -- 172.21.15.2:0/3770238956 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870031cb0 con 0x7fa88015d270 2024-01-15T18:11:40.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87fe00640 1 -- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa88015d270 msgr2=0x7fa8800b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87fe00640 1 --2- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa88015d270 0x7fa8800b2620 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa870009870 tx=0x7fa87002b860 comp rx=0 tx=0).stop 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87fe00640 1 -- 172.21.15.2:0/3770238956 shutdown_connections 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87fe00640 1 --2- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa880161890 0x7fa880160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87fe00640 1 --2- 172.21.15.2:0/3770238956 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa88015d270 0x7fa8800b2620 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.819+0000 7fa87fe00640 1 -- 172.21.15.2:0/3770238956 >> 172.21.15.2:0/3770238956 conn(0x7fa8800b0840 msgr2=0x7fa8800b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.820+0000 7fa87fe00640 1 -- 172.21.15.2:0/3770238956 shutdown_connections 2024-01-15T18:11:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.820+0000 7fa87fe00640 1 -- 172.21.15.2:0/3770238956 wait complete. 2024-01-15T18:11:40.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.821+0000 7fa87fe00640 1 Processor -- start 2024-01-15T18:11:40.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 -- start start 2024-01-15T18:11:40.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8800b2b60 0x7fa880147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:40.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 0x7fa880153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa880161890 0x7fa88015b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8801534b0 con 0x7fa88015d270 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88015af20 con 0x7fa880161890 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.822+0000 7fa87fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8801531d0 con 0x7fa8800b2b60 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa877e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 0x7fa880153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa877e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 0x7fa880153810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33120/0 (socket says 172.21.15.2:33120) 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa877e00640 1 -- 172.21.15.2:0/167985665 learned_addr learned my addr 172.21.15.2:0/167985665 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa87ea00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8800b2b60 0x7fa880147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa87f400640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa880161890 0x7fa88015b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa877e00640 1 -- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8800b2b60 msgr2=0x7fa880147810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa877e00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8800b2b60 0x7fa880147810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.823+0000 7fa877e00640 1 -- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa880161890 msgr2=0x7fa88015b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa877e00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa880161890 0x7fa88015b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa877e00640 1 -- 172.21.15.2:0/167985665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8700094d0 con 0x7fa88015d270 2024-01-15T18:11:40.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa877e00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 0x7fa880153810 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa86800e8a0 tx=0x7fa86800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:40.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680094f0 con 0x7fa88015d270 2024-01-15T18:11:40.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86800c940 con 0x7fa88015d270 2024-01-15T18:11:40.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86800ccf0 con 0x7fa88015d270 2024-01-15T18:11:40.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8800ad3b0 con 0x7fa88015d270 2024-01-15T18:11:40.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.824+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa880152e80 con 0x7fa88015d270 2024-01-15T18:11:40.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.825+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8800ad210 con 0x7fa88015d270 2024-01-15T18:11:40.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.826+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 31) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fa868013470 con 0x7fa88015d270 2024-01-15T18:11:40.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.827+0000 7fa876a00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85806c040 0x7fa85806e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:40.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.827+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fa868088970 con 0x7fa88015d270 2024-01-15T18:11:40.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.828+0000 7fa87ea00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85806c040 0x7fa85806e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:40.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.829+0000 7fa87ea00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85806c040 0x7fa85806e500 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa870036000 tx=0x7fa87003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:40.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:40.832+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8800ad210 con 0x7fa88015d270 2024-01-15T18:11:41.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.073+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8801611a0 con 0x7fa85806c040 2024-01-15T18:11:41.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.222+0000 7fa876a00640 1 -- 172.21.15.2:0/167985665 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa870031b90 con 0x7fa85806c040 2024-01-15T18:11:41.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85806c040 msgr2=0x7fa85806e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85806c040 0x7fa85806e500 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa870036000 tx=0x7fa87003d040 comp rx=0 tx=0).stop 2024-01-15T18:11:41.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 msgr2=0x7fa880153810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 0x7fa880153810 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa86800e8a0 tx=0x7fa86800ed60 comp rx=0 tx=0).stop 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 shutdown_connections 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85806c040 0x7fa85806e500 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa880161890 0x7fa88015b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa88015d270 0x7fa880153810 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 --2- 172.21.15.2:0/167985665 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8800b2b60 0x7fa880147810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 >> 172.21.15.2:0/167985665 conn(0x7fa8800b0840 msgr2=0x7fa88015ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.226+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 shutdown_connections 2024-01-15T18:11:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.227+0000 7fa87fe00640 1 -- 172.21.15.2:0/167985665 wait complete. 2024-01-15T18:11:41.236 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.232+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.236 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.232+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.236 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.232+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.236 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.232+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.236 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.232+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.254 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-15T18:11:41.254 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:41.269 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.265+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.269 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.265+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.269 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.265+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.269 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.265+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.269 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:11:41.265+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:11:41.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.471+0000 7f2678c00640 1 Processor -- start 2024-01-15T18:11:41.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.472+0000 7f2678c00640 1 -- start start 2024-01-15T18:11:41.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.473+0000 7f2678c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2674144c00 0x7f26740cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:41.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.473+0000 7f2678c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26740ce230 con 0x7f26740cdd70 2024-01-15T18:11:41.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.473+0000 7f2678c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26740ce510 con 0x7f267406d000 2024-01-15T18:11:41.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.473+0000 7f2678c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26740ce7f0 con 0x7f2674144c00 2024-01-15T18:11:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2671e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2674144c00 0x7f26740cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2669200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f26740cdd70 0x7f267406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51224/0 (socket says 172.21.15.2:51224) 2024-01-15T18:11:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2669200640 1 -- 172.21.15.2:0/2209062767 learned_addr learned my addr 172.21.15.2:0/2209062767 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2671e00640 1 -- 172.21.15.2:0/2209062767 >> v1:172.21.15.137:6789/0 conn(0x7f267406d000 legacy=0x7f2674070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2671e00640 1 -- 172.21.15.2:0/2209062767 >> v1:172.21.15.2:6789/0 conn(0x7f26740cdd70 legacy=0x7f267406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2671400640 1 -- 172.21.15.2:0/2209062767 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304358019 0 0) 0x7f26740ce230 con 0x7f26740cdd70 2024-01-15T18:11:41.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.474+0000 7f2671e00640 1 -- 172.21.15.2:0/2209062767 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26740705f0 con 0x7f2674144c00 2024-01-15T18:11:41.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.476+0000 7f2671e00640 1 --2- 172.21.15.2:0/2209062767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2674144c00 0x7f26740cd830 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2664009d90 tx=0x7f266402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997121cabf010e9e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:41.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.476+0000 7f2671400640 1 -- 172.21.15.2:0/2209062767 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266402cd00 con 0x7f2674144c00 2024-01-15T18:11:41.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.476+0000 7f2671400640 1 -- 172.21.15.2:0/2209062767 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266402ce60 con 0x7f2674144c00 2024-01-15T18:11:41.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.476+0000 7f2671400640 1 -- 172.21.15.2:0/2209062767 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664034660 con 0x7f2674144c00 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 -- 172.21.15.2:0/2209062767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2674144c00 msgr2=0x7f26740cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 --2- 172.21.15.2:0/2209062767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2674144c00 0x7f26740cd830 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f2664009d90 tx=0x7f266402bf50 comp rx=0 tx=0).stop 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 -- 172.21.15.2:0/2209062767 shutdown_connections 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 --2- 172.21.15.2:0/2209062767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2674144c00 0x7f26740cd830 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 -- 172.21.15.2:0/2209062767 >> 172.21.15.2:0/2209062767 conn(0x7f26740b45f0 msgr2=0x7f26740b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 -- 172.21.15.2:0/2209062767 shutdown_connections 2024-01-15T18:11:41.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.477+0000 7f2678c00640 1 -- 172.21.15.2:0/2209062767 wait complete. 2024-01-15T18:11:41.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.479+0000 7f2678c00640 1 Processor -- start 2024-01-15T18:11:41.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.479+0000 7f2678c00640 1 -- start start 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2678c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267406d000 0x7f26741f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2678c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 0x7f26740ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2678c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2674144c00 0x7f26741f4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2678c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26741ee560 con 0x7f26740cdd70 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2678c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2674132210 con 0x7f2674144c00 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2678c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26741f4bd0 con 0x7f267406d000 2024-01-15T18:11:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2669200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 0x7f26740ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:41.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2669200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 0x7f26740ac3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:33144/0 (socket says 172.21.15.2:33144) 2024-01-15T18:11:41.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.480+0000 7f2669200640 1 -- 172.21.15.2:0/589386942 learned_addr learned my addr 172.21.15.2:0/589386942 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2671e00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267406d000 0x7f26741f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2669200640 1 -- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267406d000 msgr2=0x7f26741f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2672800640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2674144c00 0x7f26741f4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2669200640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267406d000 0x7f26741f15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2669200640 1 -- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2674144c00 msgr2=0x7f26741f4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2669200640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2674144c00 0x7f26741f4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2669200640 1 -- 172.21.15.2:0/589386942 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2664009a70 con 0x7f26740cdd70 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2672800640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2674144c00 0x7f26741f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2669200640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 0x7f26740ac3a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f265c00c4d0 tx=0x7f265c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c0074e0 con 0x7f26740cdd70 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265c007660 con 0x7f26740cdd70 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265c004a20 con 0x7f26740cdd70 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.481+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26741f4eb0 con 0x7f26740cdd70 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.482+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26741f10a0 con 0x7f26740cdd70 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.483+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26740686f0 con 0x7f26740cdd70 2024-01-15T18:11:41.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.484+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 31) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f265c004b80 con 0x7f26740cdd70 2024-01-15T18:11:41.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.485+0000 7f266aa00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f264c06c040 0x7f264c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:41.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.485+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f265c087ae0 con 0x7f26740cdd70 2024-01-15T18:11:41.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.485+0000 7f2671e00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f264c06c040 0x7f264c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:41.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.486+0000 7f2671e00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f264c06c040 0x7f264c06e500 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2664036000 tx=0x7f2664030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:41.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.489+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f265c0540f0 con 0x7f26740cdd70 2024-01-15T18:11:41.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.720+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2674074810 con 0x7f264c06c040 2024-01-15T18:11:41.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.737+0000 7f266aa00640 1 -- 172.21.15.2:0/589386942 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f2674074810 con 0x7f264c06c040 2024-01-15T18:11:41.741 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:41.741 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:41.741 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:41.742 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:41.742 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:41.742 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:41.742 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:41.742 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:41.743 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:41.743 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f264c06c040 msgr2=0x7f264c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f264c06c040 0x7f264c06e500 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f2664036000 tx=0x7f2664030000 comp rx=0 tx=0).stop 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 msgr2=0x7f26740ac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 0x7f26740ac3a0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f265c00c4d0 tx=0x7f265c00c990 comp rx=0 tx=0).stop 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 shutdown_connections 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f264c06c040 0x7f264c06e500 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2674144c00 0x7f26741f4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f26740cdd70 0x7f26740ac3a0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 --2- 172.21.15.2:0/589386942 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f267406d000 0x7f26741f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.739+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 >> 172.21.15.2:0/589386942 conn(0x7f26740b45f0 msgr2=0x7f2674142b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.740+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 shutdown_connections 2024-01-15T18:11:41.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:41.740+0000 7f2678c00640 1 -- 172.21.15.2:0/589386942 wait complete. 2024-01-15T18:11:41.764 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:42.765 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.997+0000 7f9cfe400640 1 Processor -- start 2024-01-15T18:11:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.997+0000 7f9cfe400640 1 -- start start 2024-01-15T18:11:43.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.998+0000 7f9cfe400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b6b0 0x7f9cf8070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.998+0000 7f9cfe400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9cf80cf310 0x7f9cf80cf7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.998+0000 7f9cfe400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf80ccd00 con 0x7f9cf815b2d0 2024-01-15T18:11:43.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.998+0000 7f9cfe400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf80cfce0 con 0x7f9cf80cf310 2024-01-15T18:11:43.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.998+0000 7f9cfe400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf80adc70 con 0x7f9cf815b6b0 2024-01-15T18:11:43.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.999+0000 7f9cfd000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9cf815b2d0 0x7f9cf80cb8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48666/0 (socket says 172.21.15.2:48666) 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.999+0000 7f9cfd000640 1 -- 172.21.15.2:0/226808670 learned_addr learned my addr 172.21.15.2:0/226808670 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.999+0000 7f9cf7e00640 1 --2- 172.21.15.2:0/226808670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b6b0 0x7f9cf8070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.999+0000 7f9cfda00640 1 --2- 172.21.15.2:0/226808670 >> v2:172.21.15.137:3300/0 conn(0x7f9cf80cf310 0x7f9cf80cf7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.999+0000 7f9cf7400640 1 -- 172.21.15.2:0/226808670 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1449751228 0 0) 0x7f9cf80ccd00 con 0x7f9cf815b2d0 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:42.999+0000 7f9cf7400640 1 -- 172.21.15.2:0/226808670 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd8003520 con 0x7f9cf815b2d0 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.000+0000 7f9cf7e00640 1 -- 172.21.15.2:0/226808670 >> v2:172.21.15.137:3300/0 conn(0x7f9cf80cf310 msgr2=0x7f9cf80cf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.000+0000 7f9cf7e00640 1 --2- 172.21.15.2:0/226808670 >> v2:172.21.15.137:3300/0 conn(0x7f9cf80cf310 0x7f9cf80cf7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.000+0000 7f9cf7400640 1 -- 172.21.15.2:0/226808670 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2017956367 0 0) 0x7f9cd8003520 con 0x7f9cf815b2d0 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.000+0000 7f9cf7e00640 1 -- 172.21.15.2:0/226808670 >> v1:172.21.15.2:6789/0 conn(0x7f9cf815b2d0 legacy=0x7f9cf80cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.000+0000 7f9cf7e00640 1 -- 172.21.15.2:0/226808670 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf80cc0f0 con 0x7f9cf815b6b0 2024-01-15T18:11:43.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.000+0000 7f9cfda00640 1 --2- 172.21.15.2:0/226808670 >> v2:172.21.15.137:3300/0 conn(0x7f9cf80cf310 0x7f9cf80cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.001+0000 7f9cf7e00640 1 --2- 172.21.15.2:0/226808670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b6b0 0x7f9cf8070000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00097a0 tx=0x7f9ce002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2c8a24cb90ec3c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.001+0000 7f9cf7400640 1 -- 172.21.15.2:0/226808670 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8001670 con 0x7f9cf815b6b0 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cf7400640 1 -- 172.21.15.2:0/226808670 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce002e050 con 0x7f9cf815b6b0 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cf7400640 1 -- 172.21.15.2:0/226808670 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce0030710 con 0x7f9cf815b6b0 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cfe400640 1 -- 172.21.15.2:0/226808670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b6b0 msgr2=0x7f9cf8070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cfe400640 1 --2- 172.21.15.2:0/226808670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b6b0 0x7f9cf8070000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f9ce00097a0 tx=0x7f9ce002b790 comp rx=0 tx=0).stop 2024-01-15T18:11:43.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cfe400640 1 -- 172.21.15.2:0/226808670 shutdown_connections 2024-01-15T18:11:43.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cfe400640 1 --2- 172.21.15.2:0/226808670 >> v2:172.21.15.137:3300/0 conn(0x7f9cf80cf310 0x7f9cf80cf7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cfe400640 1 --2- 172.21.15.2:0/226808670 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b6b0 0x7f9cf8070000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.002+0000 7f9cfe400640 1 -- 172.21.15.2:0/226808670 >> 172.21.15.2:0/226808670 conn(0x7f9cf80d0ec0 msgr2=0x7f9cf80d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:43.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.003+0000 7f9cfe400640 1 -- 172.21.15.2:0/226808670 shutdown_connections 2024-01-15T18:11:43.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.003+0000 7f9cfe400640 1 -- 172.21.15.2:0/226808670 wait complete. 2024-01-15T18:11:43.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.004+0000 7f9cfe400640 1 Processor -- start 2024-01-15T18:11:43.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.004+0000 7f9cfe400640 1 -- start start 2024-01-15T18:11:43.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.005+0000 7f9cfe400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 0x7f9cf80c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.005+0000 7f9cfe400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b2d0 0x7f9cf81f5780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.005+0000 7f9cfe400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cf815b6b0 0x7f9cf81f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.005+0000 7f9cfe400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf80c8540 con 0x7f9cf80cf310 2024-01-15T18:11:43.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.005+0000 7f9cfe400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf80c6980 con 0x7f9cf815b6b0 2024-01-15T18:11:43.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.005+0000 7f9cfe400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf80cb150 con 0x7f9cf815b2d0 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 0x7f9cf80c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 0x7f9cf80c8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50556/0 (socket says 172.21.15.2:50556) 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 -- 172.21.15.2:0/2524674996 learned_addr learned my addr 172.21.15.2:0/2524674996 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cf7e00640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b2d0 0x7f9cf81f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfda00640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cf815b6b0 0x7f9cf81f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 -- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b2d0 msgr2=0x7f9cf81f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b2d0 0x7f9cf81f5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 -- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cf815b6b0 msgr2=0x7f9cf81f60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cf815b6b0 0x7f9cf81f60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.006+0000 7f9cfd000640 1 -- 172.21.15.2:0/2524674996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0009a70 con 0x7f9cf80cf310 2024-01-15T18:11:43.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.007+0000 7f9cfd000640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 0x7f9cf80c8000 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c750 tx=0x7f9ce800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:43.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.007+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8005ad0 con 0x7f9cf80cf310 2024-01-15T18:11:43.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.007+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8005c30 con 0x7f9cf80cf310 2024-01-15T18:11:43.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.007+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8007070 con 0x7f9cf80cf310 2024-01-15T18:11:43.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.007+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce0009400 con 0x7f9cf80cf310 2024-01-15T18:11:43.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.007+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf80c6330 con 0x7f9cf80cf310 2024-01-15T18:11:43.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.008+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf80cd170 con 0x7f9cf80cf310 2024-01-15T18:11:43.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.009+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f9ce80071d0 con 0x7f9cf80cf310 2024-01-15T18:11:43.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.011+0000 7f9cf4c00640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd006c160 0x7f9cd006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:43.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.011+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f9ce8088090 con 0x7f9cf80cf310 2024-01-15T18:11:43.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.011+0000 7f9cf7e00640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd006c160 0x7f9cd006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:43.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.013+0000 7f9cf7e00640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd006c160 0x7f9cd006e620 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0034000 tx=0x7f9ce002f000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:43.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.015+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9cf80cd170 con 0x7f9cf80cf310 2024-01-15T18:11:43.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.246+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cf80cc990 con 0x7f9cd006c160 2024-01-15T18:11:43.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.260+0000 7f9cf4c00640 1 -- 172.21.15.2:0/2524674996 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f9ce00043e0 con 0x7f9cd006c160 2024-01-15T18:11:43.264 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:43.264 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:43.265 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:43.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd006c160 msgr2=0x7f9cd006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd006c160 0x7f9cd006e620 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f9ce0034000 tx=0x7f9ce002f000 comp rx=0 tx=0).stop 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 msgr2=0x7f9cf80c8000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 0x7f9cf80c8000 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c750 tx=0x7f9ce800cc10 comp rx=0 tx=0).stop 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 shutdown_connections 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9cd006c160 0x7f9cd006e620 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9cf815b6b0 0x7f9cf81f60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9cf815b2d0 0x7f9cf81f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 --2- 172.21.15.2:0/2524674996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9cf80cf310 0x7f9cf80c8000 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 >> 172.21.15.2:0/2524674996 conn(0x7f9cf80d0ec0 msgr2=0x7f9cf80b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 shutdown_connections 2024-01-15T18:11:43.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:43.263+0000 7f9cfe400640 1 -- 172.21.15.2:0/2524674996 wait complete. 2024-01-15T18:11:43.286 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:44.287 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:44.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.501+0000 7fcfd5200640 1 Processor -- start 2024-01-15T18:11:44.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.502+0000 7fcfd5200640 1 -- start start 2024-01-15T18:11:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.503+0000 7fcfd5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcfd015e0b0 0x7fcfd00d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.503+0000 7fcfd5200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd006f440 con 0x7fcfd00b32b0 2024-01-15T18:11:44.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.503+0000 7fcfd5200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd015e5f0 con 0x7fcfd0146ba0 2024-01-15T18:11:44.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.503+0000 7fcfd5200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd0146f80 con 0x7fcfd015e0b0 2024-01-15T18:11:44.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.504+0000 7fcfcd400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcfd00b32b0 0x7fcfd015d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48680/0 (socket says 172.21.15.2:48680) 2024-01-15T18:11:44.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.504+0000 7fcfcd400640 1 -- 172.21.15.2:0/2117481642 learned_addr learned my addr 172.21.15.2:0/2117481642 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.504+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295094597 0 0) 0x7fcfd006f440 con 0x7fcfd00b32b0 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.504+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfb0003520 con 0x7fcfd00b32b0 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.504+0000 7fcfce800640 1 --2- 172.21.15.2:0/2117481642 >> v2:172.21.15.137:3300/0 conn(0x7fcfd015e0b0 0x7fcfd00d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.504+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2905913580 0 0) 0x7fcfb0003520 con 0x7fcfd00b32b0 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 >> v2:172.21.15.137:3300/0 conn(0x7fcfd015e0b0 msgr2=0x7fcfd00d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 --2- 172.21.15.2:0/2117481642 >> v2:172.21.15.137:3300/0 conn(0x7fcfd015e0b0 0x7fcfd00d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 >> v1:172.21.15.137:6789/0 conn(0x7fcfd0146ba0 legacy=0x7fcfd00b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd015e8d0 con 0x7fcfd00b32b0 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fcfb8001670 con 0x7fcfd00b32b0 2024-01-15T18:11:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 >> v1:172.21.15.2:6789/0 conn(0x7fcfd00b32b0 legacy=0x7fcfd015d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00039c0 0x7fcfb0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb00047c0 0x7fcfb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb0005250 0x7fcfb000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd0146f80 con 0x7fcfb00047c0 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb000c1a0 con 0x7fcfb00039c0 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb000c360 con 0x7fcfb0005250 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb00047c0 0x7fcfb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcde00640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00039c0 0x7fcfb0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcd400640 1 -- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb0005250 msgr2=0x7fcfb000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfb0005250 0x7fcfb000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcd400640 1 -- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00039c0 msgr2=0x7fcfb0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00039c0 0x7fcfb0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.505+0000 7fcfcd400640 1 -- 172.21.15.2:0/2117481642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb0003520 con 0x7fcfb00047c0 2024-01-15T18:11:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.506+0000 7fcfce800640 1 -- 172.21.15.2:0/2117481642 reap_dead start 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.507+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb00047c0 0x7fcfb0004c40 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c190 tx=0x7fcfb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1085b0d6b023341a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.507+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb8005ad0 con 0x7fcfb00047c0 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.507+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb8005c30 con 0x7fcfb00047c0 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.507+0000 7fcfcca00640 1 -- 172.21.15.2:0/2117481642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb802c810 con 0x7fcfb00047c0 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.507+0000 7fcfd5200640 1 -- 172.21.15.2:0/2117481642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb00047c0 msgr2=0x7fcfb0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.507+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb00047c0 0x7fcfb0004c40 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcfb800c190 tx=0x7fcfb00057f0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.508+0000 7fcfd5200640 1 -- 172.21.15.2:0/2117481642 shutdown_connections 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.508+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2117481642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfb00047c0 0x7fcfb0004c40 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.508+0000 7fcfd5200640 1 -- 172.21.15.2:0/2117481642 >> 172.21.15.2:0/2117481642 conn(0x7fcfd00b6430 msgr2=0x7fcfd00b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.508+0000 7fcfd5200640 1 -- 172.21.15.2:0/2117481642 shutdown_connections 2024-01-15T18:11:44.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.508+0000 7fcfd5200640 1 -- 172.21.15.2:0/2117481642 wait complete. 2024-01-15T18:11:44.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.509+0000 7fcfd5200640 1 Processor -- start 2024-01-15T18:11:44.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.510+0000 7fcfd5200640 1 -- start start 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.510+0000 7fcfd5200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00047c0 0x7fcfd00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfd5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 0x7fcfd00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfd5200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd00cfad0 0x7fcfd0131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfd5200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00cff50 con 0x7fcfd01f26c0 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfd5200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00ce120 con 0x7fcfb00047c0 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfd5200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd00cd510 con 0x7fcfd00cfad0 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 0x7fcfd00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 0x7fcfd00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50588/0 (socket says 172.21.15.2:50588) 2024-01-15T18:11:44.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 -- 172.21.15.2:0/2316409097 learned_addr learned my addr 172.21.15.2:0/2316409097 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:44.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfce800640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd00cfad0 0x7fcfd0131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcde00640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00047c0 0x7fcfd00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 -- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd00cfad0 msgr2=0x7fcfd0131790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd00cfad0 0x7fcfd0131790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 -- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00047c0 msgr2=0x7fcfd00cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00047c0 0x7fcfd00cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfcd400640 1 -- 172.21.15.2:0/2316409097 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00cd7f0 con 0x7fcfd01f26c0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.511+0000 7fcfce800640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd00cfad0 0x7fcfd0131790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.512+0000 7fcfcd400640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 0x7fcfd00cf590 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fcfb802c3b0 tx=0x7fcfb802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.512+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb803c040 con 0x7fcfd01f26c0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.512+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfb802cd00 con 0x7fcfd01f26c0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.512+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfb80072b0 con 0x7fcfd01f26c0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.512+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd00cdad0 con 0x7fcfd01f26c0 2024-01-15T18:11:44.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.512+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd01f2b40 con 0x7fcfd01f26c0 2024-01-15T18:11:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.514+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd00d1780 con 0x7fcfd01f26c0 2024-01-15T18:11:44.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.515+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fcfb8032070 con 0x7fcfd01f26c0 2024-01-15T18:11:44.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.515+0000 7fcfc6000640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcfa806c190 0x7fcfa806e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:44.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.515+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fcfb80a9b30 con 0x7fcfd01f26c0 2024-01-15T18:11:44.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.516+0000 7fcfcde00640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcfa806c190 0x7fcfa806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:44.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.517+0000 7fcfcde00640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcfa806c190 0x7fcfa806e650 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0007fd0 tx=0x7fcfc0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:44.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.520+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcfb8076500 con 0x7fcfd01f26c0 2024-01-15T18:11:44.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.753+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfd006f320 con 0x7fcfa806c190 2024-01-15T18:11:44.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.771+0000 7fcfc6000640 1 -- 172.21.15.2:0/2316409097 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fcfd006f320 con 0x7fcfa806c190 2024-01-15T18:11:44.775 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:44.776 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:44.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcfa806c190 msgr2=0x7fcfa806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:44.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcfa806c190 0x7fcfa806e650 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fcfc0007fd0 tx=0x7fcfc0009040 comp rx=0 tx=0).stop 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 msgr2=0x7fcfd00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 0x7fcfd00cf590 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fcfb802c3b0 tx=0x7fcfb802e070 comp rx=0 tx=0).stop 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 shutdown_connections 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcfa806c190 0x7fcfa806e650 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcfd00cfad0 0x7fcfd0131790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcfd01f26c0 0x7fcfd00cf590 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 --2- 172.21.15.2:0/2316409097 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcfb00047c0 0x7fcfd00cb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 >> 172.21.15.2:0/2316409097 conn(0x7fcfd00b6430 msgr2=0x7fcfd00669d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 shutdown_connections 2024-01-15T18:11:44.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:44.774+0000 7fcfd5200640 1 -- 172.21.15.2:0/2316409097 wait complete. 2024-01-15T18:11:44.798 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:45.799 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:46.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.026+0000 7f4d68c00640 1 Processor -- start 2024-01-15T18:11:46.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.026+0000 7f4d68c00640 1 -- start start 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d68c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 0x7f4d6406c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d68c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64071090 con 0x7f4d640720d0 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d68c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d64144560 con 0x7f4d64143d20 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d68c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d64071430 con 0x7f4d6415f260 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d63e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4d640720d0 0x7f4d6406c690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48692/0 (socket says 172.21.15.2:48692) 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d63e00640 1 -- 172.21.15.2:0/392717027 learned_addr learned my addr 172.21.15.2:0/392717027 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.028+0000 7f4d62a00640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 0x7f4d6406c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:46.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109996372 0 0) 0x7f4d64071090 con 0x7f4d640720d0 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d44003520 con 0x7f4d640720d0 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3522776897 0 0) 0x7f4d44003520 con 0x7f4d640720d0 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 msgr2=0x7f4d6406c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 0x7f4d6406c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 >> v1:172.21.15.137:6789/0 conn(0x7f4d64143d20 legacy=0x7f4d6415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d64144280 con 0x7f4d640720d0 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4d5c0019f0 con 0x7f4d640720d0 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 >> v1:172.21.15.2:6789/0 conn(0x7f4d640720d0 legacy=0x7f4d6406c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d44003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 0x7f4d440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d44004d00 0x7f4d44005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d64071430 con 0x7f4d440039a0 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4400bb40 con 0x7f4d44004d00 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.029+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4400bd00 con 0x7f4d6415f260 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63400640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d44003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d62a00640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 0x7f4d440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63e00640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d44004d00 0x7f4d44005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63400640 1 -- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 msgr2=0x7f4d440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63400640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415f260 0x7f4d440047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63400640 1 -- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d44004d00 msgr2=0x7f4d44005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:46.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63400640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d44004d00 0x7f4d44005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.030+0000 7f4d63400640 1 -- 172.21.15.2:0/392717027 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d44003520 con 0x7f4d440039a0 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.031+0000 7f4d63400640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d44003dc0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f4d54001750 tx=0x7f4d440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6032c92dcf9fc2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.031+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540052d0 con 0x7f4d440039a0 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.031+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5402e050 con 0x7f4d440039a0 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d62000640 1 -- 172.21.15.2:0/392717027 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540035f0 con 0x7f4d440039a0 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d68c00640 1 -- 172.21.15.2:0/392717027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 msgr2=0x7f4d44003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d68c00640 1 --2- 172.21.15.2:0/392717027 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d44003dc0 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f4d54001750 tx=0x7f4d440057f0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d63e00640 1 -- 172.21.15.2:0/392717027 reap_dead start 2024-01-15T18:11:46.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d68c00640 1 -- 172.21.15.2:0/392717027 shutdown_connections 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d68c00640 1 -- 172.21.15.2:0/392717027 >> 172.21.15.2:0/392717027 conn(0x7f4d640b0840 msgr2=0x7f4d640b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.032+0000 7f4d68c00640 1 -- 172.21.15.2:0/392717027 shutdown_connections 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.033+0000 7f4d68c00640 1 -- 172.21.15.2:0/392717027 wait complete. 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.034+0000 7f4d68c00640 1 Processor -- start 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.035+0000 7f4d68c00640 1 -- start start 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d68c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d640c6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d68c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415cae0 0x7f4d6415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d68c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d6415a4e0 0x7f4d6415a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d68c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6415aed0 con 0x7f4d440039a0 2024-01-15T18:11:46.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d68c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6415d5a0 con 0x7f4d6415a4e0 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d68c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6415d880 con 0x7f4d6415cae0 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d640c6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d640c6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50620/0 (socket says 172.21.15.2:50620) 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 -- 172.21.15.2:0/4165654320 learned_addr learned my addr 172.21.15.2:0/4165654320 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 -- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415cae0 msgr2=0x7f4d6415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415cae0 0x7f4d6415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 -- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d6415a4e0 msgr2=0x7f4d6415a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d6415a4e0 0x7f4d6415a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.036+0000 7f4d63400640 1 -- 172.21.15.2:0/4165654320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6415db60 con 0x7f4d440039a0 2024-01-15T18:11:46.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.037+0000 7f4d63400640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d640c6cc0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4d54008fd0 tx=0x7f4d54001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:46.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.037+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540076c0 con 0x7f4d440039a0 2024-01-15T18:11:46.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.037+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d54007a30 con 0x7f4d440039a0 2024-01-15T18:11:46.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.037+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54006be0 con 0x7f4d440039a0 2024-01-15T18:11:46.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.037+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6415de40 con 0x7f4d440039a0 2024-01-15T18:11:46.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.037+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d64135ea0 con 0x7f4d440039a0 2024-01-15T18:11:46.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.038+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6415d9c0 con 0x7f4d440039a0 2024-01-15T18:11:46.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.039+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4d54006d40 con 0x7f4d440039a0 2024-01-15T18:11:46.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.040+0000 7f4d5b400640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3c06c190 0x7f4d3c06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:46.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.040+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4d540aa740 con 0x7f4d440039a0 2024-01-15T18:11:46.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.041+0000 7f4d62a00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3c06c190 0x7f4d3c06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:46.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.042+0000 7f4d62a00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3c06c190 0x7f4d3c06e650 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4d640c6a20 tx=0x7f4d4c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:46.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.044+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d6415d9c0 con 0x7f4d440039a0 2024-01-15T18:11:46.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.274+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d64136030 con 0x7f4d3c06c190 2024-01-15T18:11:46.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.292+0000 7f4d5b400640 1 -- 172.21.15.2:0/4165654320 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f4d64136030 con 0x7f4d3c06c190 2024-01-15T18:11:46.296 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:46.296 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:46.296 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:46.297 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:46.297 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:46.297 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:46.297 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:46.297 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:46.298 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:46.298 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:46.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3c06c190 msgr2=0x7f4d3c06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3c06c190 0x7f4d3c06e650 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4d640c6a20 tx=0x7f4d4c008040 comp rx=0 tx=0).stop 2024-01-15T18:11:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 msgr2=0x7f4d640c6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d640c6cc0 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4d54008fd0 tx=0x7f4d54001120 comp rx=0 tx=0).stop 2024-01-15T18:11:46.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 shutdown_connections 2024-01-15T18:11:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3c06c190 0x7f4d3c06e650 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d6415a4e0 0x7f4d6415a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d6415cae0 0x7f4d6415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 --2- 172.21.15.2:0/4165654320 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d440039a0 0x7f4d640c6cc0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 >> 172.21.15.2:0/4165654320 conn(0x7f4d640b0840 msgr2=0x7f4d640ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.294+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 shutdown_connections 2024-01-15T18:11:46.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:46.295+0000 7f4d68c00640 1 -- 172.21.15.2:0/4165654320 wait complete. 2024-01-15T18:11:46.319 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:47.319 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:47.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.549+0000 7fe1e1200640 1 Processor -- start 2024-01-15T18:11:47.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.550+0000 7fe1e1200640 1 -- start start 2024-01-15T18:11:47.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1e1200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe1dc146ba0 0x7fe1dc0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1e1200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe1dc0b2ba0 0x7fe1dc0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1e1200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc06f440 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1e1200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc146f80 con 0x7fe1dc0b2ba0 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1e1200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b3520 con 0x7fe1dc146ba0 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1da800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe1dc15d9a0 0x7fe1dc15de30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48698/0 (socket says 172.21.15.2:48698) 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.551+0000 7fe1da800640 1 -- 172.21.15.2:0/2735350039 learned_addr learned my addr 172.21.15.2:0/2735350039 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d9400640 1 --2- 172.21.15.2:0/2735350039 >> v2:172.21.15.2:3300/0 conn(0x7fe1dc0b2ba0 0x7fe1dc0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d9e00640 1 --2- 172.21.15.2:0/2735350039 >> v2:172.21.15.137:3300/0 conn(0x7fe1dc146ba0 0x7fe1dc0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d8a00640 1 -- 172.21.15.2:0/2735350039 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984140157 0 0) 0x7fe1dc06f440 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d8a00640 1 -- 172.21.15.2:0/2735350039 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc003520 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d9400640 1 -- 172.21.15.2:0/2735350039 >> v2:172.21.15.137:3300/0 conn(0x7fe1dc146ba0 msgr2=0x7fe1dc0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d9400640 1 --2- 172.21.15.2:0/2735350039 >> v2:172.21.15.137:3300/0 conn(0x7fe1dc146ba0 0x7fe1dc0d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d9400640 1 -- 172.21.15.2:0/2735350039 >> v1:172.21.15.2:6789/0 conn(0x7fe1dc15d9a0 legacy=0x7fe1dc15de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.552+0000 7fe1d9400640 1 -- 172.21.15.2:0/2735350039 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc15e5f0 con 0x7fe1dc0b2ba0 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.553+0000 7fe1d9400640 1 --2- 172.21.15.2:0/2735350039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc0b2ba0 0x7fe1dc0b2fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc06e430 tx=0x7fe1c402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b600da598bfdb240 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1d8a00640 1 -- 172.21.15.2:0/2735350039 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c402e070 con 0x7fe1dc0b2ba0 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1d8a00640 1 -- 172.21.15.2:0/2735350039 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c40315a0 con 0x7fe1dc0b2ba0 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1d8a00640 1 -- 172.21.15.2:0/2735350039 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c4031930 con 0x7fe1dc0b2ba0 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 -- 172.21.15.2:0/2735350039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc0b2ba0 msgr2=0x7fe1dc0b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 --2- 172.21.15.2:0/2735350039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc0b2ba0 0x7fe1dc0b2fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc06e430 tx=0x7fe1c402b840 comp rx=0 tx=0).stop 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 -- 172.21.15.2:0/2735350039 shutdown_connections 2024-01-15T18:11:47.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 --2- 172.21.15.2:0/2735350039 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc0b2ba0 0x7fe1dc0b2fe0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 --2- 172.21.15.2:0/2735350039 >> v2:172.21.15.137:3300/0 conn(0x7fe1dc146ba0 0x7fe1dc0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 -- 172.21.15.2:0/2735350039 >> 172.21.15.2:0/2735350039 conn(0x7fe1dc0b6430 msgr2=0x7fe1dc0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 -- 172.21.15.2:0/2735350039 shutdown_connections 2024-01-15T18:11:47.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.554+0000 7fe1e1200640 1 -- 172.21.15.2:0/2735350039 wait complete. 2024-01-15T18:11:47.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.556+0000 7fe1e1200640 1 Processor -- start 2024-01-15T18:11:47.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.556+0000 7fe1e1200640 1 -- start start 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.557+0000 7fe1e1200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1dc0b2ba0 0x7fe1dc143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.557+0000 7fe1e1200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1dc146ba0 0x7fe1dc0ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.557+0000 7fe1e1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 0x7fe1dc1444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.557+0000 7fe1e1200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0cee40 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.557+0000 7fe1e1200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc1436c0 con 0x7fe1dc146ba0 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.557+0000 7fe1e1200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc160370 con 0x7fe1dc0b2ba0 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 0x7fe1dc1444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 0x7fe1dc1444c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50642/0 (socket says 172.21.15.2:50642) 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 -- 172.21.15.2:0/3807187278 learned_addr learned my addr 172.21.15.2:0/3807187278 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1d9e00640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1dc0b2ba0 0x7fe1dc143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1d9400640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1dc146ba0 0x7fe1dc0ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 -- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1dc0b2ba0 msgr2=0x7fe1dc143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1dc0b2ba0 0x7fe1dc143180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 -- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1dc146ba0 msgr2=0x7fe1dc0ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1dc146ba0 0x7fe1dc0ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1da800640 1 -- 172.21.15.2:0/3807187278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c4009400 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.558+0000 7fe1d9400640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1dc146ba0 0x7fe1dc0ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.559+0000 7fe1da800640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 0x7fe1dc1444c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400cb40 tx=0x7fe1d400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.559+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d400aca0 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.559+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400ae00 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.559+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1d4012620 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.559+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc160650 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.559+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc0cea60 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.560+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc0cef80 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.561+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fe1d4007070 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.563+0000 7fe1d2000640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe1b406c160 0x7fe1b406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:47.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.563+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fe1d40889e0 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.564+0000 7fe1d9e00640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe1b406c160 0x7fe1b406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:47.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.565+0000 7fe1d9e00640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe1b406c160 0x7fe1b406e620 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009ad0 tx=0x7fe1cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:47.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.567+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1dc0cef80 con 0x7fe1dc15d9a0 2024-01-15T18:11:47.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.796+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1dc0d1260 con 0x7fe1b406c160 2024-01-15T18:11:47.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.811+0000 7fe1d2000640 1 -- 172.21.15.2:0/3807187278 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fe1cc009800 con 0x7fe1b406c160 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:47.815 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe1b406c160 msgr2=0x7fe1b406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe1b406c160 0x7fe1b406e620 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc009ad0 tx=0x7fe1cc008040 comp rx=0 tx=0).stop 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 msgr2=0x7fe1dc1444c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 0x7fe1dc1444c0 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7fe1d400cb40 tx=0x7fe1d400a060 comp rx=0 tx=0).stop 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 shutdown_connections 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe1b406c160 0x7fe1b406e620 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe1dc15d9a0 0x7fe1dc1444c0 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe1dc146ba0 0x7fe1dc0ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 --2- 172.21.15.2:0/3807187278 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe1dc0b2ba0 0x7fe1dc143180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:47.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 >> 172.21.15.2:0/3807187278 conn(0x7fe1dc0b6430 msgr2=0x7fe1dc06c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 shutdown_connections 2024-01-15T18:11:47.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:47.813+0000 7fe1e1200640 1 -- 172.21.15.2:0/3807187278 wait complete. 2024-01-15T18:11:47.836 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:48.838 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:49.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.053+0000 7f07b4600640 1 Processor -- start 2024-01-15T18:11:49.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.054+0000 7f07b4600640 1 -- start start 2024-01-15T18:11:49.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.055+0000 7f07b4600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.055+0000 7f07b4600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0ce230 con 0x7f07ac15c450 2024-01-15T18:11:49.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.055+0000 7f07b4600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ac0ce510 con 0x7f07ac144c00 2024-01-15T18:11:49.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.055+0000 7f07b4600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac0ce7f0 con 0x7f07ac06d000 2024-01-15T18:11:49.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.056+0000 7f07b1200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f07ac15c450 0x7f07ac06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48702/0 (socket says 172.21.15.2:48702) 2024-01-15T18:11:49.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.056+0000 7f07b1200640 1 -- 172.21.15.2:0/461413257 learned_addr learned my addr 172.21.15.2:0/461413257 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.056+0000 7f07b2600640 1 --2- 172.21.15.2:0/461413257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.056+0000 7f07abe00640 1 -- 172.21.15.2:0/461413257 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403518264 0 0) 0x7f07ac0ce230 con 0x7f07ac15c450 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.056+0000 7f07abe00640 1 -- 172.21.15.2:0/461413257 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f078c003520 con 0x7f07ac15c450 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.057+0000 7f07b2600640 1 -- 172.21.15.2:0/461413257 >> v1:172.21.15.137:6789/0 conn(0x7f07ac144c00 legacy=0x7f07ac0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.057+0000 7f07b2600640 1 -- 172.21.15.2:0/461413257 >> v1:172.21.15.2:6789/0 conn(0x7f07ac15c450 legacy=0x7f07ac06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.057+0000 7f07abe00640 1 -- 172.21.15.2:0/461413257 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703959445 0 0) 0x7f07ac0ce510 con 0x7f07ac144c00 2024-01-15T18:11:49.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.057+0000 7f07b2600640 1 -- 172.21.15.2:0/461413257 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ac0705f0 con 0x7f07ac06d000 2024-01-15T18:11:49.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.058+0000 7f07b2600640 1 --2- 172.21.15.2:0/461413257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac165d60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009dc0 tx=0x7f07a402bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e681a1def763db2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07abe00640 1 -- 172.21.15.2:0/461413257 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794001670 con 0x7f07ac06d000 2024-01-15T18:11:49.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07abe00640 1 -- 172.21.15.2:0/461413257 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a402c840 con 0x7f07ac06d000 2024-01-15T18:11:49.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07abe00640 1 -- 172.21.15.2:0/461413257 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a402cbb0 con 0x7f07ac06d000 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07b4600640 1 -- 172.21.15.2:0/461413257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 msgr2=0x7f07ac165d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07b4600640 1 --2- 172.21.15.2:0/461413257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac165d60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009dc0 tx=0x7f07a402bd10 comp rx=0 tx=0).stop 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07b4600640 1 -- 172.21.15.2:0/461413257 shutdown_connections 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07b4600640 1 --2- 172.21.15.2:0/461413257 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac165d60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.059+0000 7f07b4600640 1 -- 172.21.15.2:0/461413257 >> 172.21.15.2:0/461413257 conn(0x7f07ac0b45f0 msgr2=0x7f07ac0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.060+0000 7f07b4600640 1 -- 172.21.15.2:0/461413257 shutdown_connections 2024-01-15T18:11:49.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.060+0000 7f07b4600640 1 -- 172.21.15.2:0/461413257 wait complete. 2024-01-15T18:11:49.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.061+0000 7f07b4600640 1 Processor -- start 2024-01-15T18:11:49.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.061+0000 7f07b4600640 1 -- start start 2024-01-15T18:11:49.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.062+0000 7f07b4600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac0b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.062+0000 7f07b4600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07ac144c00 0x7f07ac0b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.062+0000 7f07b4600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 0x7f07ac0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.062+0000 7f07b4600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac1539a0 con 0x7f07ac15c450 2024-01-15T18:11:49.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.062+0000 7f07b4600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac152db0 con 0x7f07ac144c00 2024-01-15T18:11:49.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.062+0000 7f07b4600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ac1f2c90 con 0x7f07ac06d000 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 0x7f07ac0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 0x7f07ac0b1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50654/0 (socket says 172.21.15.2:50654) 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 -- 172.21.15.2:0/2597025047 learned_addr learned my addr 172.21.15.2:0/2597025047 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b1c00640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac0b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b1200640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07ac144c00 0x7f07ac0b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 -- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 msgr2=0x7f07ac0b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac0b0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 -- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07ac144c00 msgr2=0x7f07ac0b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07ac144c00 0x7f07ac0b0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b2600640 1 -- 172.21.15.2:0/2597025047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4009a70 con 0x7f07ac15c450 2024-01-15T18:11:49.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.063+0000 7f07b1200640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07ac144c00 0x7f07ac0b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.064+0000 7f07b2600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 0x7f07ac0b1460 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009eb0 tx=0x7f07a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.064+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a4009450 con 0x7f07ac15c450 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.064+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a402ce30 con 0x7f07ac15c450 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.064+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a403f420 con 0x7f07ac15c450 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.064+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07ac153000 con 0x7f07ac15c450 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.064+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ac1535c0 con 0x7f07ac15c450 2024-01-15T18:11:49.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.065+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ac153ae0 con 0x7f07ac15c450 2024-01-15T18:11:49.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.066+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f07a403f580 con 0x7f07ac15c450 2024-01-15T18:11:49.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.067+0000 7f07a9600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f078406c160 0x7f078406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:49.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.067+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f07a40ab450 con 0x7f07ac15c450 2024-01-15T18:11:49.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.068+0000 7f07b1c00640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f078406c160 0x7f078406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:49.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.069+0000 7f07b1c00640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f078406c160 0x7f078406e620 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f079c000f60 tx=0x7f079c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:49.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.072+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07ac153ae0 con 0x7f07ac15c450 2024-01-15T18:11:49.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.305+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f07ac0cead0 con 0x7f078406c160 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.323+0000 7f07a9600640 1 -- 172.21.15.2:0/2597025047 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f079c007c40 con 0x7f078406c160 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:49.328 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:49.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f078406c160 msgr2=0x7f078406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f078406c160 0x7f078406e620 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f079c000f60 tx=0x7f079c009040 comp rx=0 tx=0).stop 2024-01-15T18:11:49.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 msgr2=0x7f07ac0b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 0x7f07ac0b1460 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009eb0 tx=0x7f07a4004a40 comp rx=0 tx=0).stop 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 shutdown_connections 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f078406c160 0x7f078406e620 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f07ac15c450 0x7f07ac0b1460 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f07ac144c00 0x7f07ac0b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 --2- 172.21.15.2:0/2597025047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f07ac06d000 0x7f07ac0b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 >> 172.21.15.2:0/2597025047 conn(0x7f07ac0b45f0 msgr2=0x7f07ac143d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 shutdown_connections 2024-01-15T18:11:49.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:49.326+0000 7f07b4600640 1 -- 172.21.15.2:0/2597025047 wait complete. 2024-01-15T18:11:49.350 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:50.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:50.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.582+0000 7fdef9400640 1 Processor -- start 2024-01-15T18:11:50.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.583+0000 7fdef9400640 1 -- start start 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdef9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef4143d20 0x7fdef415eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdef9400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdef40720d0 0x7fdef415f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdef9400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdef41447d0 0x7fdef40cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdef9400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4071430 con 0x7fdef40720d0 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdef9400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdef4144100 con 0x7fdef41447d0 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdef9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef4071090 con 0x7fdef4143d20 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdeebe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdef40720d0 0x7fdef415f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdeebe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdef40720d0 0x7fdef415f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50668/0 (socket says 172.21.15.2:50668) 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.584+0000 7fdeebe00640 1 -- 172.21.15.2:0/1501045286 learned_addr learned my addr 172.21.15.2:0/1501045286 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:50.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdef3e00640 1 --2- 172.21.15.2:0/1501045286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef4143d20 0x7fdef415eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdef8a00640 1 --2- 172.21.15.2:0/1501045286 >> v2:172.21.15.137:3300/0 conn(0x7fdef41447d0 0x7fdef40cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdeebe00640 1 -- 172.21.15.2:0/1501045286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef4143d20 msgr2=0x7fdef415eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdeebe00640 1 --2- 172.21.15.2:0/1501045286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef4143d20 0x7fdef415eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdeebe00640 1 -- 172.21.15.2:0/1501045286 >> v2:172.21.15.137:3300/0 conn(0x7fdef41447d0 msgr2=0x7fdef40cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdeebe00640 1 --2- 172.21.15.2:0/1501045286 >> v2:172.21.15.137:3300/0 conn(0x7fdef41447d0 0x7fdef40cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdeebe00640 1 -- 172.21.15.2:0/1501045286 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef4144560 con 0x7fdef40720d0 2024-01-15T18:11:50.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.585+0000 7fdef8a00640 1 --2- 172.21.15.2:0/1501045286 >> v2:172.21.15.137:3300/0 conn(0x7fdef41447d0 0x7fdef40cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.586+0000 7fdeebe00640 1 --2- 172.21.15.2:0/1501045286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef40720d0 0x7fdef415f090 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fdef406e0a0 tx=0x7fdedc02b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1240d83375612dc3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.586+0000 7fdef3400640 1 -- 172.21.15.2:0/1501045286 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc0317d0 con 0x7fdef40720d0 2024-01-15T18:11:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.586+0000 7fdef3400640 1 -- 172.21.15.2:0/1501045286 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdedc031930 con 0x7fdef40720d0 2024-01-15T18:11:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.586+0000 7fdef3400640 1 -- 172.21.15.2:0/1501045286 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdedc031cc0 con 0x7fdef40720d0 2024-01-15T18:11:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 -- 172.21.15.2:0/1501045286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef40720d0 msgr2=0x7fdef415f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 --2- 172.21.15.2:0/1501045286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef40720d0 0x7fdef415f090 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fdef406e0a0 tx=0x7fdedc02b870 comp rx=0 tx=0).stop 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 -- 172.21.15.2:0/1501045286 shutdown_connections 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 --2- 172.21.15.2:0/1501045286 >> v2:172.21.15.137:3300/0 conn(0x7fdef41447d0 0x7fdef40cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 --2- 172.21.15.2:0/1501045286 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef40720d0 0x7fdef415f090 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 --2- 172.21.15.2:0/1501045286 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef4143d20 0x7fdef415eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 -- 172.21.15.2:0/1501045286 >> 172.21.15.2:0/1501045286 conn(0x7fdef40b0840 msgr2=0x7fdef40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 -- 172.21.15.2:0/1501045286 shutdown_connections 2024-01-15T18:11:50.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.587+0000 7fdef9400640 1 -- 172.21.15.2:0/1501045286 wait complete. 2024-01-15T18:11:50.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.589+0000 7fdef9400640 1 Processor -- start 2024-01-15T18:11:50.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.589+0000 7fdef9400640 1 -- start start 2024-01-15T18:11:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef9400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef40720d0 0x7fdef40c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef9400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdef4143d20 0x7fdef41f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 0x7fdef41f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef9400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef41f3b80 con 0x7fdef41447d0 2024-01-15T18:11:50.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef9400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef415d2e0 con 0x7fdef4143d20 2024-01-15T18:11:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef9400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdef415d5c0 con 0x7fdef40720d0 2024-01-15T18:11:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 0x7fdef41f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef8a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 0x7fdef41f3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50676/0 (socket says 172.21.15.2:50676) 2024-01-15T18:11:50.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.590+0000 7fdef8a00640 1 -- 172.21.15.2:0/3852667548 learned_addr learned my addr 172.21.15.2:0/3852667548 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:50.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef3e00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef40720d0 0x7fdef40c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdeebe00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdef4143d20 0x7fdef41f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef8a00640 1 -- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef40720d0 msgr2=0x7fdef40c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef8a00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef40720d0 0x7fdef40c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef8a00640 1 -- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdef4143d20 msgr2=0x7fdef41f3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef8a00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdef4143d20 0x7fdef41f3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef8a00640 1 -- 172.21.15.2:0/3852667548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc0094d0 con 0x7fdef41447d0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdeebe00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdef4143d20 0x7fdef41f3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef8a00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 0x7fdef41f3640 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00cd50 tx=0x7fdeec00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.591+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00f680 con 0x7fdef41447d0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.592+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeec00f800 con 0x7fdef41447d0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.592+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec00fbb0 con 0x7fdef41447d0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.592+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef415d8a0 con 0x7fdef41447d0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.592+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef415de60 con 0x7fdef41447d0 2024-01-15T18:11:50.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.593+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef4074770 con 0x7fdef41447d0 2024-01-15T18:11:50.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.594+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fdeec00b040 con 0x7fdef41447d0 2024-01-15T18:11:50.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.595+0000 7fdef0c00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdecc06c160 0x7fdecc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:50.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.595+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fdeec0890b0 con 0x7fdef41447d0 2024-01-15T18:11:50.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.595+0000 7fdef3e00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdecc06c160 0x7fdecc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:50.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.597+0000 7fdef3e00640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdecc06c160 0x7fdecc06e620 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fdee4006fd0 tx=0x7fdee4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:50.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.599+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdeec055970 con 0x7fdef41447d0 2024-01-15T18:11:50.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.837+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdef415fa00 con 0x7fdecc06c160 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.854+0000 7fdef0c00640 1 -- 172.21.15.2:0/3852667548 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fdee400b640 con 0x7fdecc06c160 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:50.858 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:50.859 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdecc06c160 msgr2=0x7fdecc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdecc06c160 0x7fdecc06e620 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fdee4006fd0 tx=0x7fdee4008040 comp rx=0 tx=0).stop 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 msgr2=0x7fdef41f3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 0x7fdef41f3640 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00cd50 tx=0x7fdeec00a650 comp rx=0 tx=0).stop 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 shutdown_connections 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdecc06c160 0x7fdecc06e620 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdef41447d0 0x7fdef41f3640 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdef4143d20 0x7fdef41f3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 --2- 172.21.15.2:0/3852667548 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdef40720d0 0x7fdef40c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.856+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 >> 172.21.15.2:0/3852667548 conn(0x7fdef40b0840 msgr2=0x7fdef40ae3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.857+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 shutdown_connections 2024-01-15T18:11:50.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:50.857+0000 7fdef9400640 1 -- 172.21.15.2:0/3852667548 wait complete. 2024-01-15T18:11:50.879 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-15T18:11:51.879 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:52.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.109+0000 7f66dd000640 1 Processor -- start 2024-01-15T18:11:52.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.109+0000 7f66dd000640 1 -- start start 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66dd000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f66d815d270 0x7f66d80b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66dd000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d806c5e0 con 0x7f66d8161890 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66dd000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d806c8c0 con 0x7f66d8161370 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66dd000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d8071c00 con 0x7f66d815d270 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66d6a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f66d8161890 0x7f66d8160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48714/0 (socket says 172.21.15.2:48714) 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66d6a00640 1 -- 172.21.15.2:0/2587741542 learned_addr learned my addr 172.21.15.2:0/2587741542 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:52.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.111+0000 7f66d7400640 1 --2- 172.21.15.2:0/2587741542 >> v2:172.21.15.2:3300/0 conn(0x7f66d815d270 0x7f66d80b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.112+0000 7f66d6000640 1 -- 172.21.15.2:0/2587741542 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941753105 0 0) 0x7f66d806c5e0 con 0x7f66d8161890 2024-01-15T18:11:52.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.112+0000 7f66d6000640 1 -- 172.21.15.2:0/2587741542 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66b8003520 con 0x7f66d8161890 2024-01-15T18:11:52.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.112+0000 7f66d7400640 1 -- 172.21.15.2:0/2587741542 >> v1:172.21.15.137:6789/0 conn(0x7f66d8161370 legacy=0x7f66d806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.112+0000 7f66d7400640 1 -- 172.21.15.2:0/2587741542 >> v1:172.21.15.2:6789/0 conn(0x7f66d8161890 legacy=0x7f66d8160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.112+0000 7f66d7400640 1 -- 172.21.15.2:0/2587741542 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d8071ee0 con 0x7f66d815d270 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.113+0000 7f66d7400640 1 --2- 172.21.15.2:0/2587741542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d80b2620 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f66c00097a0 tx=0x7f66c002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e661fb1ebbd8fe0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.113+0000 7f66d6000640 1 -- 172.21.15.2:0/2587741542 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c002e070 con 0x7f66d815d270 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.113+0000 7f66d6000640 1 -- 172.21.15.2:0/2587741542 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c00315a0 con 0x7f66d815d270 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.113+0000 7f66d6000640 1 -- 172.21.15.2:0/2587741542 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0031930 con 0x7f66d815d270 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 -- 172.21.15.2:0/2587741542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 msgr2=0x7f66d80b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 --2- 172.21.15.2:0/2587741542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d80b2620 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f66c00097a0 tx=0x7f66c002b840 comp rx=0 tx=0).stop 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 -- 172.21.15.2:0/2587741542 shutdown_connections 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 --2- 172.21.15.2:0/2587741542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d80b2620 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 -- 172.21.15.2:0/2587741542 >> 172.21.15.2:0/2587741542 conn(0x7f66d80b0840 msgr2=0x7f66d80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 -- 172.21.15.2:0/2587741542 shutdown_connections 2024-01-15T18:11:52.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.114+0000 7f66dd000640 1 -- 172.21.15.2:0/2587741542 wait complete. 2024-01-15T18:11:52.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.116+0000 7f66dd000640 1 Processor -- start 2024-01-15T18:11:52.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.116+0000 7f66dd000640 1 -- start start 2024-01-15T18:11:52.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66dd000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d8148160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66dd000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66d8161370 0x7f66d8156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66dd000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66d8161890 0x7f66d81f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66dd000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d81575f0 con 0x7f66d815d270 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66dd000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d81569e0 con 0x7f66d8161890 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66dd000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d8156cc0 con 0x7f66d8161370 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66d7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d8148160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66d7400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d8148160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50694/0 (socket says 172.21.15.2:50694) 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66d7400640 1 -- 172.21.15.2:0/2863811491 learned_addr learned my addr 172.21.15.2:0/2863811491 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.117+0000 7f66d6a00640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66d8161370 0x7f66d8156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7e00640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66d8161890 0x7f66d81f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7400640 1 -- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66d8161370 msgr2=0x7f66d8156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7400640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66d8161370 0x7f66d8156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7400640 1 -- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66d8161890 msgr2=0x7f66d81f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7400640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66d8161890 0x7f66d81f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7400640 1 -- 172.21.15.2:0/2863811491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0009400 con 0x7f66d815d270 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7e00640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66d8161890 0x7f66d81f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66d7400640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d8148160 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f66c0034040 tx=0x7f66c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0032840 con 0x7f66d815d270 2024-01-15T18:11:52.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c00329a0 con 0x7f66d815d270 2024-01-15T18:11:52.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c0032d50 con 0x7f66d815d270 2024-01-15T18:11:52.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.118+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d8156fa0 con 0x7f66d815d270 2024-01-15T18:11:52.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.119+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d8156580 con 0x7f66d815d270 2024-01-15T18:11:52.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.120+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d8074770 con 0x7f66d815d270 2024-01-15T18:11:52.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.121+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f66c004e430 con 0x7f66d815d270 2024-01-15T18:11:52.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.121+0000 7f66cf400640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f66b006c160 0x7f66b006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:52.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.122+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f66c00aaad0 con 0x7f66d815d270 2024-01-15T18:11:52.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.122+0000 7f66d6a00640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f66b006c160 0x7f66b006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:52.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.123+0000 7f66d6a00640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f66b006c160 0x7f66b006e620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f66d80b0570 tx=0x7f66c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:52.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.126+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66c0076f70 con 0x7f66d815d270 2024-01-15T18:11:52.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:52.361+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f66d806d460 con 0x7f66b006c160 2024-01-15T18:11:53.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.800+0000 7f66cf400640 1 -- 172.21.15.2:0/2863811491 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f66c80012d0 con 0x7f66b006c160 2024-01-15T18:11:53.803 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:53.804 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.802+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f66b006c160 msgr2=0x7f66b006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.802+0000 7f66dd000640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f66b006c160 0x7f66b006e620 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f66d80b0570 tx=0x7f66c8009040 comp rx=0 tx=0).stop 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.802+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 msgr2=0x7f66d8148160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.802+0000 7f66dd000640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d8148160 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7f66c0034040 tx=0x7f66c00046c0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 shutdown_connections 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f66b006c160 0x7f66b006e620 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f66d8161890 0x7f66d81f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f66d8161370 0x7f66d8156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 --2- 172.21.15.2:0/2863811491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f66d815d270 0x7f66d8148160 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:53.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 >> 172.21.15.2:0/2863811491 conn(0x7f66d80b0840 msgr2=0x7f66d80b1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:53.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 shutdown_connections 2024-01-15T18:11:53.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:53.803+0000 7f66dd000640 1 -- 172.21.15.2:0/2863811491 wait complete. 2024-01-15T18:11:55.829 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:56.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.081+0000 7f0ee7c00640 1 Processor -- start 2024-01-15T18:11:56.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.082+0000 7f0ee7c00640 1 -- start start 2024-01-15T18:11:56.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee7c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0ee0146ba0 0x7f0ee00d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee7c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ee015e0b0 0x7f0ee00b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee7c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee006f440 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee7c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee0146f80 con 0x7f0ee015e0b0 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee7c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ee00b3520 con 0x7f0ee0146ba0 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee5c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ee015e0b0 0x7f0ee00b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee5c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ee015e0b0 0x7f0ee00b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49036/0 (socket says 172.21.15.2:49036) 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee5c00640 1 -- 172.21.15.2:0/3625070546 learned_addr learned my addr 172.21.15.2:0/3625070546 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.083+0000 7f0ee5200640 1 --2- 172.21.15.2:0/3625070546 >> v2:172.21.15.137:3300/0 conn(0x7f0ee0146ba0 0x7f0ee00d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.084+0000 7f0ee5c00640 1 -- 172.21.15.2:0/3625070546 >> v2:172.21.15.137:3300/0 conn(0x7f0ee0146ba0 msgr2=0x7f0ee00d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.084+0000 7f0ee5c00640 1 --2- 172.21.15.2:0/3625070546 >> v2:172.21.15.137:3300/0 conn(0x7f0ee0146ba0 0x7f0ee00d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.084+0000 7f0ee5c00640 1 -- 172.21.15.2:0/3625070546 >> v1:172.21.15.137:6789/0 conn(0x7f0ee00b2ba0 legacy=0x7f0ee015d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:56.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.084+0000 7f0ee5c00640 1 -- 172.21.15.2:0/3625070546 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee015e5f0 con 0x7f0ee015e0b0 2024-01-15T18:11:56.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.084+0000 7f0ee5200640 1 --2- 172.21.15.2:0/3625070546 >> v2:172.21.15.137:3300/0 conn(0x7f0ee0146ba0 0x7f0ee00d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:56.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.085+0000 7f0ee5c00640 1 --2- 172.21.15.2:0/3625070546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee015e0b0 0x7f0ee00b2fe0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0ee006e430 tx=0x7f0ed802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6034f15ae6d0be96 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.085+0000 7f0edfe00640 1 -- 172.21.15.2:0/3625070546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed802c9f0 con 0x7f0ee015e0b0 2024-01-15T18:11:56.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.085+0000 7f0edfe00640 1 -- 172.21.15.2:0/3625070546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed802cb50 con 0x7f0ee015e0b0 2024-01-15T18:11:56.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.085+0000 7f0edfe00640 1 -- 172.21.15.2:0/3625070546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed80343f0 con 0x7f0ee015e0b0 2024-01-15T18:11:56.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 -- 172.21.15.2:0/3625070546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee015e0b0 msgr2=0x7f0ee00b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/3625070546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee015e0b0 0x7f0ee00b2fe0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f0ee006e430 tx=0x7f0ed802be50 comp rx=0 tx=0).stop 2024-01-15T18:11:56.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 -- 172.21.15.2:0/3625070546 shutdown_connections 2024-01-15T18:11:56.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/3625070546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee015e0b0 0x7f0ee00b2fe0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/3625070546 >> v2:172.21.15.137:3300/0 conn(0x7f0ee0146ba0 0x7f0ee00d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 -- 172.21.15.2:0/3625070546 >> 172.21.15.2:0/3625070546 conn(0x7f0ee00b6430 msgr2=0x7f0ee00b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 -- 172.21.15.2:0/3625070546 shutdown_connections 2024-01-15T18:11:56.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.086+0000 7f0ee7c00640 1 -- 172.21.15.2:0/3625070546 wait complete. 2024-01-15T18:11:56.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.089+0000 7f0ee7c00640 1 Processor -- start 2024-01-15T18:11:56.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.089+0000 7f0ee7c00640 1 -- start start 2024-01-15T18:11:56.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee7c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 0x7f0ee0136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee7c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0146ba0 0x7f0ee015bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee7c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee015e0b0 0x7f0ee015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee7c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00cf2c0 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee7c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0160790 con 0x7f0ee015e0b0 2024-01-15T18:11:56.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee7c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00ce640 con 0x7f0ee0146ba0 2024-01-15T18:11:56.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 0x7f0ee0136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 0x7f0ee0136aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49038/0 (socket says 172.21.15.2:49038) 2024-01-15T18:11:56.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee5200640 1 -- 172.21.15.2:0/1242096986 learned_addr learned my addr 172.21.15.2:0/1242096986 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:56.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ee5c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee015e0b0 0x7f0ee015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.091+0000 7f0ed7e00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0146ba0 0x7f0ee015bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ee5200640 1 -- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0146ba0 msgr2=0x7f0ee015bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ee5200640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0146ba0 0x7f0ee015bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ee5200640 1 -- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee015e0b0 msgr2=0x7f0ee015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ee5200640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee015e0b0 0x7f0ee015c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ee5200640 1 -- 172.21.15.2:0/1242096986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed8009b30 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ed7e00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0146ba0 0x7f0ee015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.092+0000 7f0ee5200640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 0x7f0ee0136aa0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000d800 tx=0x7f0ed000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.093+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0004030 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.093+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed0004190 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.093+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0004520 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.093+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee00ce920 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.093+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee00ceee0 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.095+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee00686f0 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.097+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f0ed000b6c0 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.099+0000 7f0edd600640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:56.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.099+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0ed0088750 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.109+0000 7f0ed7e00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.111+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ed0055070 con 0x7f0ee00b2ba0 2024-01-15T18:11:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.112+0000 7f0ed7e00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:56.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.470+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ee00d1260 con 0x7f0eb806c160 2024-01-15T18:11:56.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.484+0000 7f0edd600640 1 -- 172.21.15.2:0/1242096986 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f0ec8009e40 con 0x7f0eb806c160 2024-01-15T18:11:56.488 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:56.488 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:56.488 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:11:56.488 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 0.0, 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "0 B/3 GB", 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "0/3", 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:56.489 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:56.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.488+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 msgr2=0x7f0eb806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.488+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8008040 comp rx=0 tx=0).stop 2024-01-15T18:11:56.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.488+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 msgr2=0x7f0ee0136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:56.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.488+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 0x7f0ee0136aa0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0ed000d800 tx=0x7f0ed000dcc0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.489+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 shutdown_connections 2024-01-15T18:11:56.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.489+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.489+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee015e0b0 0x7f0ee015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.489+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0146ba0 0x7f0ee015bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.489+0000 7f0ee7c00640 1 --2- 172.21.15.2:0/1242096986 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee00b2ba0 0x7f0ee0136aa0 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:56.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.489+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 >> 172.21.15.2:0/1242096986 conn(0x7f0ee00b6430 msgr2=0x7f0ee015d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:56.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.490+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 shutdown_connections 2024-01-15T18:11:56.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:56.490+0000 7f0ee7c00640 1 -- 172.21.15.2:0/1242096986 wait complete. 2024-01-15T18:11:56.532 INFO:tasks.cephfs.test_volumes:percetange completed = 0.0 2024-01-15T18:11:58.533 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:11:58.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.766+0000 7f34cb200640 1 Processor -- start 2024-01-15T18:11:58.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.766+0000 7f34cb200640 1 -- start start 2024-01-15T18:11:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.767+0000 7f34cb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f34cc15d340 0x7f34cc0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.767+0000 7f34cb200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0b3de0 con 0x7f34cc0d3090 2024-01-15T18:11:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.767+0000 7f34cb200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34cc0b40c0 con 0x7f34cc0d1d90 2024-01-15T18:11:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.767+0000 7f34cb200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34cc0d0e80 con 0x7f34cc15d340 2024-01-15T18:11:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c9400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f34cc0d3090 0x7f34cc0d1680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41940/0 (socket says 172.21.15.2:41940) 2024-01-15T18:11:58.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c9400640 1 -- 172.21.15.2:0/2771143297 learned_addr learned my addr 172.21.15.2:0/2771143297 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:11:58.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510258208 0 0) 0x7f34cc0b3de0 con 0x7f34cc0d3090 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac003520 con 0x7f34cc0d3090 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2771143297 >> v2:172.21.15.137:3300/0 conn(0x7f34cc15d340 0x7f34cc0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3084791946 0 0) 0x7f34ac003520 con 0x7f34cc0d3090 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 >> v2:172.21.15.137:3300/0 conn(0x7f34cc15d340 msgr2=0x7f34cc0cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.768+0000 7f34c8a00640 1 --2- 172.21.15.2:0/2771143297 >> v2:172.21.15.137:3300/0 conn(0x7f34cc15d340 0x7f34cc0cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 >> v1:172.21.15.137:6789/0 conn(0x7f34cc0d1d90 legacy=0x7f34cc0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc0d1160 con 0x7f34cc0d3090 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f34b40019f0 con 0x7f34cc0d3090 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 >> v1:172.21.15.2:6789/0 conn(0x7f34cc0d3090 legacy=0x7f34cc0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:58.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34ac003970 0x7f34ac003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34ac0047c0 0x7f34ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac005180 0x7f34ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc0d0e80 con 0x7f34ac003970 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac00c0b0 con 0x7f34ac005180 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34ac00c270 con 0x7f34ac0047c0 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34ac003970 0x7f34ac003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34c9400640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34ac0047c0 0x7f34ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.769+0000 7f34ca800640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac005180 0x7f34ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.770+0000 7f34c9e00640 1 -- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34ac0047c0 msgr2=0x7f34ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.770+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34ac0047c0 0x7f34ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.770+0000 7f34c9e00640 1 -- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac005180 msgr2=0x7f34ac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.770+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac005180 0x7f34ac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.770+0000 7f34c9e00640 1 -- 172.21.15.2:0/2771143297 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac003520 con 0x7f34ac003970 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.770+0000 7f34ca800640 1 -- 172.21.15.2:0/2771143297 reap_dead start 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.771+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34ac003970 0x7f34ac003d70 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00ba00 tx=0x7f34ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408896cd0ca7c70d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.771+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc032620 con 0x7f34ac003970 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.771+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc032780 con 0x7f34ac003970 2024-01-15T18:11:58.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.771+0000 7f34c8a00640 1 -- 172.21.15.2:0/2771143297 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc032b10 con 0x7f34ac003970 2024-01-15T18:11:58.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.772+0000 7f34cb200640 1 -- 172.21.15.2:0/2771143297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34ac003970 msgr2=0x7f34ac003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:58.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.772+0000 7f34cb200640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34ac003970 0x7f34ac003d70 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00ba00 tx=0x7f34ac0057f0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.772+0000 7f34cb200640 1 -- 172.21.15.2:0/2771143297 shutdown_connections 2024-01-15T18:11:58.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.772+0000 7f34cb200640 1 --2- 172.21.15.2:0/2771143297 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34ac003970 0x7f34ac003d70 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.772+0000 7f34cb200640 1 -- 172.21.15.2:0/2771143297 >> 172.21.15.2:0/2771143297 conn(0x7f34cc08caf0 msgr2=0x7f34cc09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.772+0000 7f34cb200640 1 -- 172.21.15.2:0/2771143297 shutdown_connections 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.773+0000 7f34cb200640 1 -- 172.21.15.2:0/2771143297 wait complete. 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.775+0000 7f34cb200640 1 Processor -- start 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.775+0000 7f34cb200640 1 -- start start 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34cb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac003970 0x7f34cc06c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34cb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34cc151840 0x7f34cc0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34cb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 0x7f34cc0ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34cb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc0cabf0 con 0x7f34cc0ca200 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34cb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc0cd1b0 con 0x7f34ac003970 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34cb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34cc151cc0 con 0x7f34cc151840 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34ca800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 0x7f34cc0ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34ca800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 0x7f34cc0ca6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49060/0 (socket says 172.21.15.2:49060) 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34ca800640 1 -- 172.21.15.2:0/2721035762 learned_addr learned my addr 172.21.15.2:0/2721035762 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34c9400640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34cc151840 0x7f34cc0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.776+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac003970 0x7f34cc06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34ca800640 1 -- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34cc151840 msgr2=0x7f34cc0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34ca800640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34cc151840 0x7f34cc0ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34ca800640 1 -- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac003970 msgr2=0x7f34cc06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:58.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34ca800640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac003970 0x7f34cc06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34ca800640 1 -- 172.21.15.2:0/2721035762 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34cc06c7d0 con 0x7f34cc0ca200 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34c9400640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34cc151840 0x7f34cc0ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34ca800640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 0x7f34cc0ca6b0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0b91c0 tx=0x7f34c400e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40054b0 con 0x7f34cc0ca200 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c40069e0 con 0x7f34cc0ca200 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c4006d90 con 0x7f34cc0ca200 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.777+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34cc06cab0 con 0x7f34cc0ca200 2024-01-15T18:11:58.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.778+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34cc0c7e40 con 0x7f34cc0ca200 2024-01-15T18:11:58.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.779+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34cc067970 con 0x7f34cc0ca200 2024-01-15T18:11:58.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.780+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f34c4007030 con 0x7f34cc0ca200 2024-01-15T18:11:58.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.781+0000 7f34c2000640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f34a406c190 0x7f34a406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:11:58.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.781+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f34c4087be0 con 0x7f34cc0ca200 2024-01-15T18:11:58.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.782+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f34a406c190 0x7f34a406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:11:58.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.783+0000 7f34c9e00640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f34a406c190 0x7f34a406e650 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0057a0 tx=0x7f34bc03d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:11:58.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:58.786+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34c4054080 con 0x7f34cc0ca200 2024-01-15T18:11:59.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.016+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f34cc074770 con 0x7f34a406c190 2024-01-15T18:11:59.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.029+0000 7f34c2000640 1 -- 172.21.15.2:0/2721035762 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f34cc074770 con 0x7f34a406c190 2024-01-15T18:11:59.032 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:11:59.033 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:11:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:11:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:11:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:59.034 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:11:59.034 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:11:59.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f34a406c190 msgr2=0x7f34a406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f34a406c190 0x7f34a406e650 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f34ac0057a0 tx=0x7f34bc03d040 comp rx=0 tx=0).stop 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 msgr2=0x7f34cc0ca6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 0x7f34cc0ca6b0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f34cc0b91c0 tx=0x7f34c400e8a0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 shutdown_connections 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f34a406c190 0x7f34a406e650 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f34cc0ca200 0x7f34cc0ca6b0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f34cc151840 0x7f34cc0ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 --2- 172.21.15.2:0/2721035762 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f34ac003970 0x7f34cc06c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:11:59.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 >> 172.21.15.2:0/2721035762 conn(0x7f34cc08caf0 msgr2=0x7f34cc06b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:11:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 shutdown_connections 2024-01-15T18:11:59.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:11:59.032+0000 7f34cb200640 1 -- 172.21.15.2:0/2721035762 wait complete. 2024-01-15T18:11:59.056 INFO:tasks.cephfs.test_volumes:percetange completed = 33.33333333333333 2024-01-15T18:12:01.058 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:12:01.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.359+0000 7fed3b800640 1 Processor -- start 2024-01-15T18:12:01.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.360+0000 7fed3b800640 1 -- start start 2024-01-15T18:12:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.364+0000 7fed3b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.364+0000 7fed3b800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fed340b8c30 0x7fed340d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.364+0000 7fed3b800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed340d1900 con 0x7fed340b4c70 2024-01-15T18:12:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.364+0000 7fed3b800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed340d1be0 con 0x7fed340b8c30 2024-01-15T18:12:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.364+0000 7fed3b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340d1ec0 con 0x7fed3415d730 2024-01-15T18:12:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.365+0000 7fed38e00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7fed340b4c70 0x7fed340b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:35602/0 (socket says 172.21.15.2:35602) 2024-01-15T18:12:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.365+0000 7fed38e00640 1 -- 172.21.15.2:0/2632351143 learned_addr learned my addr 172.21.15.2:0/2632351143 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.365+0000 7fed39800640 1 --2- 172.21.15.2:0/2632351143 >> v2:172.21.15.137:3300/0 conn(0x7fed340b8c30 0x7fed340d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.365+0000 7fed33e00640 1 --2- 172.21.15.2:0/2632351143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.366+0000 7fed33e00640 1 -- 172.21.15.2:0/2632351143 >> v2:172.21.15.137:3300/0 conn(0x7fed340b8c30 msgr2=0x7fed340d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.366+0000 7fed33e00640 1 --2- 172.21.15.2:0/2632351143 >> v2:172.21.15.137:3300/0 conn(0x7fed340b8c30 0x7fed340d29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.366+0000 7fed33e00640 1 -- 172.21.15.2:0/2632351143 >> v1:172.21.15.137:6789/0 conn(0x7fed340b4c70 legacy=0x7fed340b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.366+0000 7fed33e00640 1 -- 172.21.15.2:0/2632351143 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed340d21a0 con 0x7fed3415d730 2024-01-15T18:12:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.367+0000 7fed33e00640 1 --2- 172.21.15.2:0/2632351143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340d1270 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0097a0 tx=0x7fed1c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48055c968fef177a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.368+0000 7fed33400640 1 -- 172.21.15.2:0/2632351143 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c02e070 con 0x7fed3415d730 2024-01-15T18:12:01.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.369+0000 7fed33400640 1 -- 172.21.15.2:0/2632351143 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c0315a0 con 0x7fed3415d730 2024-01-15T18:12:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.369+0000 7fed33400640 1 -- 172.21.15.2:0/2632351143 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c031930 con 0x7fed3415d730 2024-01-15T18:12:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 -- 172.21.15.2:0/2632351143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 msgr2=0x7fed340d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 --2- 172.21.15.2:0/2632351143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340d1270 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fed1c0097a0 tx=0x7fed1c02b840 comp rx=0 tx=0).stop 2024-01-15T18:12:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 -- 172.21.15.2:0/2632351143 shutdown_connections 2024-01-15T18:12:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 --2- 172.21.15.2:0/2632351143 >> v2:172.21.15.137:3300/0 conn(0x7fed340b8c30 0x7fed340d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 --2- 172.21.15.2:0/2632351143 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340d1270 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 -- 172.21.15.2:0/2632351143 >> 172.21.15.2:0/2632351143 conn(0x7fed34072210 msgr2=0x7fed340cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 -- 172.21.15.2:0/2632351143 shutdown_connections 2024-01-15T18:12:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.370+0000 7fed3b800640 1 -- 172.21.15.2:0/2632351143 wait complete. 2024-01-15T18:12:01.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.373+0000 7fed3b800640 1 Processor -- start 2024-01-15T18:12:01.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.373+0000 7fed3b800640 1 -- start start 2024-01-15T18:12:01.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.375+0000 7fed3b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed340b4c70 0x7fed34153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.375+0000 7fed3b800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 0x7fed3415db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.375+0000 7fed3b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed3b800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340c8210 con 0x7fed340b8c30 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed3b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340cbc40 con 0x7fed340b4c70 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed3b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340cbf20 con 0x7fed3415d730 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed33e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 0x7fed3415db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed33e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 0x7fed3415db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49076/0 (socket says 172.21.15.2:49076) 2024-01-15T18:12:01.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed33e00640 1 -- 172.21.15.2:0/532021318 learned_addr learned my addr 172.21.15.2:0/532021318 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:01.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed39800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.376+0000 7fed38e00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed340b4c70 0x7fed34153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.377+0000 7fed33e00640 1 -- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 msgr2=0x7fed340c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.377+0000 7fed33e00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.377+0000 7fed33e00640 1 -- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed340b4c70 msgr2=0x7fed34153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.377+0000 7fed33e00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed340b4c70 0x7fed34153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.377+0000 7fed33e00640 1 -- 172.21.15.2:0/532021318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1c009400 con 0x7fed340b8c30 2024-01-15T18:12:01.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.378+0000 7fed33e00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 0x7fed3415db30 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fed1c034040 tx=0x7fed1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.379+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c032900 con 0x7fed340b8c30 2024-01-15T18:12:01.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.379+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c032a60 con 0x7fed340b8c30 2024-01-15T18:12:01.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.379+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c032e10 con 0x7fed340b8c30 2024-01-15T18:12:01.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.379+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed340d3060 con 0x7fed340b8c30 2024-01-15T18:12:01.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.379+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed3415e070 con 0x7fed340b8c30 2024-01-15T18:12:01.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.383+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fed1c03b3f0 con 0x7fed340b8c30 2024-01-15T18:12:01.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.384+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed340686f0 con 0x7fed340b8c30 2024-01-15T18:12:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.384+0000 7fed30c00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed0c06c160 0x7fed0c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:01.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.385+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fed1c0aa980 con 0x7fed340b8c30 2024-01-15T18:12:01.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.391+0000 7fed38e00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed0c06c160 0x7fed0c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.395+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed1c076fe0 con 0x7fed340b8c30 2024-01-15T18:12:01.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.396+0000 7fed38e00640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed0c06c160 0x7fed0c06e620 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fed24007c90 tx=0x7fed24007be0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:01.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.775+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fed34073540 con 0x7fed0c06c160 2024-01-15T18:12:01.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.796+0000 7fed30c00640 1 -- 172.21.15.2:0/532021318 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fed34073540 con 0x7fed0c06c160 2024-01-15T18:12:01.799 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:01.799 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 33.33333333333333, 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "1 GB/3 GB", 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "1/3", 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:01.800 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:01.801 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:01.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed0c06c160 msgr2=0x7fed0c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed0c06c160 0x7fed0c06e620 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fed24007c90 tx=0x7fed24007be0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 msgr2=0x7fed3415db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:01.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 0x7fed3415db30 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7fed1c034040 tx=0x7fed1c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 shutdown_connections 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed0c06c160 0x7fed0c06e620 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed3415d730 0x7fed340c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed340b8c30 0x7fed3415db30 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 --2- 172.21.15.2:0/532021318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed340b4c70 0x7fed34153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 >> 172.21.15.2:0/532021318 conn(0x7fed34072210 msgr2=0x7fed340cda40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 shutdown_connections 2024-01-15T18:12:01.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:01.800+0000 7fed3b800640 1 -- 172.21.15.2:0/532021318 wait complete. 2024-01-15T18:12:01.833 INFO:tasks.cephfs.test_volumes:percetange completed = 33.33333333333333 2024-01-15T18:12:02.757 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:02.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:03.834 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:12:04.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.096+0000 7f0b9fa00640 1 Processor -- start 2024-01-15T18:12:04.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.096+0000 7f0b9fa00640 1 -- start start 2024-01-15T18:12:04.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.097+0000 7f0b9fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b980b4540 0x7f0b980b4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.097+0000 7f0b9fa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0b98072050 0x7f0b9815b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.097+0000 7f0b9fa00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980b4e60 con 0x7f0b9815b320 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.097+0000 7f0b9fa00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9806f430 con 0x7f0b98072050 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.097+0000 7f0b9fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b9806f150 con 0x7f0b980b4540 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.098+0000 7f0b97e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0b9815b320 0x7f0b9806e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48728/0 (socket says 172.21.15.2:48728) 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.098+0000 7f0b97e00640 1 -- 172.21.15.2:0/396873423 learned_addr learned my addr 172.21.15.2:0/396873423 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.098+0000 7f0b9d000640 1 --2- 172.21.15.2:0/396873423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b980b4540 0x7f0b980b4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.098+0000 7f0b9da00640 1 --2- 172.21.15.2:0/396873423 >> v2:172.21.15.137:3300/0 conn(0x7f0b98072050 0x7f0b9815b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b97400640 1 -- 172.21.15.2:0/396873423 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1183306660 0 0) 0x7f0b980b4e60 con 0x7f0b9815b320 2024-01-15T18:12:04.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b97400640 1 -- 172.21.15.2:0/396873423 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b78003520 con 0x7f0b9815b320 2024-01-15T18:12:04.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b9d000640 1 -- 172.21.15.2:0/396873423 >> v2:172.21.15.137:3300/0 conn(0x7f0b98072050 msgr2=0x7f0b9815b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:04.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b9d000640 1 --2- 172.21.15.2:0/396873423 >> v2:172.21.15.137:3300/0 conn(0x7f0b98072050 0x7f0b9815b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b9d000640 1 -- 172.21.15.2:0/396873423 >> v1:172.21.15.2:6789/0 conn(0x7f0b9815b320 legacy=0x7f0b9806e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b9d000640 1 -- 172.21.15.2:0/396873423 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98071570 con 0x7f0b980b4540 2024-01-15T18:12:04.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.099+0000 7f0b9da00640 1 --2- 172.21.15.2:0/396873423 >> v2:172.21.15.137:3300/0 conn(0x7f0b98072050 0x7f0b9815b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:04.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.101+0000 7f0b9d000640 1 --2- 172.21.15.2:0/396873423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b980b4540 0x7f0b980b4920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0b880097a0 tx=0x7f0b8802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9afcfcce42b680a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.102+0000 7f0b97400640 1 -- 172.21.15.2:0/396873423 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88031780 con 0x7f0b980b4540 2024-01-15T18:12:04.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.102+0000 7f0b97400640 1 -- 172.21.15.2:0/396873423 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b880318e0 con 0x7f0b980b4540 2024-01-15T18:12:04.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.102+0000 7f0b97400640 1 -- 172.21.15.2:0/396873423 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88031c70 con 0x7f0b980b4540 2024-01-15T18:12:04.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.104+0000 7f0b9fa00640 1 -- 172.21.15.2:0/396873423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b980b4540 msgr2=0x7f0b980b4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.104+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/396873423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b980b4540 0x7f0b980b4920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0b880097a0 tx=0x7f0b8802b820 comp rx=0 tx=0).stop 2024-01-15T18:12:04.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.109+0000 7f0b9fa00640 1 -- 172.21.15.2:0/396873423 shutdown_connections 2024-01-15T18:12:04.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.109+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/396873423 >> v2:172.21.15.137:3300/0 conn(0x7f0b98072050 0x7f0b9815b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.109+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/396873423 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b980b4540 0x7f0b980b4920 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.109+0000 7f0b9fa00640 1 -- 172.21.15.2:0/396873423 >> 172.21.15.2:0/396873423 conn(0x7f0b980b1d70 msgr2=0x7f0b980b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.111+0000 7f0b9fa00640 1 -- 172.21.15.2:0/396873423 shutdown_connections 2024-01-15T18:12:04.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.111+0000 7f0b9fa00640 1 -- 172.21.15.2:0/396873423 wait complete. 2024-01-15T18:12:04.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.114+0000 7f0b9fa00640 1 Processor -- start 2024-01-15T18:12:04.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.115+0000 7f0b9fa00640 1 -- start start 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.116+0000 7f0b9fa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b98072050 0x7f0b981efa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.116+0000 7f0b9fa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b980b4540 0x7f0b980cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.116+0000 7f0b9fa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b9815b320 0x7f0b9815bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.116+0000 7f0b9fa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b981e2580 con 0x7f0b980b4540 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.116+0000 7f0b9fa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b980cf060 con 0x7f0b9815b320 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.116+0000 7f0b9fa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b980cf340 con 0x7f0b98072050 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.117+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b980b4540 0x7f0b980cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.117+0000 7f0b97e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b980b4540 0x7f0b980cb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36264/0 (socket says 172.21.15.2:36264) 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.117+0000 7f0b97e00640 1 -- 172.21.15.2:0/215838197 learned_addr learned my addr 172.21.15.2:0/215838197 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.117+0000 7f0b9d000640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b98072050 0x7f0b981efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.117+0000 7f0b9da00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b9815b320 0x7f0b9815bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9d000640 1 -- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b9815b320 msgr2=0x7f0b9815bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9d000640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b9815b320 0x7f0b9815bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9d000640 1 -- 172.21.15.2:0/215838197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b980b4540 msgr2=0x7f0b980cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9d000640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b980b4540 0x7f0b980cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9d000640 1 -- 172.21.15.2:0/215838197 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b88009400 con 0x7f0b98072050 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9da00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b9815b320 0x7f0b9815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b9d000640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b98072050 0x7f0b981efa00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0b88033000 tx=0x7f0b880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.118+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88032820 con 0x7f0b98072050 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.119+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b88032980 con 0x7f0b98072050 2024-01-15T18:12:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.119+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b980cf620 con 0x7f0b98072050 2024-01-15T18:12:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.119+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88032d30 con 0x7f0b98072050 2024-01-15T18:12:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.119+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b980cbcb0 con 0x7f0b98072050 2024-01-15T18:12:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.120+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b980686f0 con 0x7f0b98072050 2024-01-15T18:12:04.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.124+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f0b88067460 con 0x7f0b98072050 2024-01-15T18:12:04.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.125+0000 7f0b94c00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b7006c190 0x7f0b7006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:04.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.125+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0b880aa9b0 con 0x7f0b98072050 2024-01-15T18:12:04.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.132+0000 7f0b97e00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b7006c190 0x7f0b7006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:04.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.133+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0b88077440 con 0x7f0b98072050 2024-01-15T18:12:04.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.140+0000 7f0b97e00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b7006c190 0x7f0b7006e650 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0b9815cac0 tx=0x7f0b80009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:04.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.532+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b98071b30 con 0x7f0b7006c190 2024-01-15T18:12:04.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.560+0000 7f0b94c00640 1 -- 172.21.15.2:0/215838197 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f0b80001550 con 0x7f0b7006c190 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "in-progress", 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_report": { 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "percentage completed": 66.66666666666666, 2024-01-15T18:12:04.564 INFO:teuthology.orchestra.run.smithi002.stdout: "amount copied": "2 GB/3 GB", 2024-01-15T18:12:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: "regfiles copied": "2/3", 2024-01-15T18:12:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: "subdirs copied": "0/1", 2024-01-15T18:12:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: "symlinks copied": "0/0" 2024-01-15T18:12:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:04.565 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:04.565 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b7006c190 msgr2=0x7f0b7006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b7006c190 0x7f0b7006e650 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f0b9815cac0 tx=0x7f0b80009040 comp rx=0 tx=0).stop 2024-01-15T18:12:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b98072050 msgr2=0x7f0b981efa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:04.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b98072050 0x7f0b981efa00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0b88033000 tx=0x7f0b880046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 shutdown_connections 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0b7006c190 0x7f0b7006e650 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0b9815b320 0x7f0b9815bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0b980b4540 0x7f0b980cb770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 --2- 172.21.15.2:0/215838197 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0b98072050 0x7f0b981efa00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 >> 172.21.15.2:0/215838197 conn(0x7f0b980b1d70 msgr2=0x7f0b9815b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 shutdown_connections 2024-01-15T18:12:04.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:04.565+0000 7f0b9fa00640 1 -- 172.21.15.2:0/215838197 wait complete. 2024-01-15T18:12:04.598 INFO:tasks.cephfs.test_volumes:percetange completed = 66.66666666666666 2024-01-15T18:12:06.599 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:12:06.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.838+0000 7efd17200640 1 Processor -- start 2024-01-15T18:12:06.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.838+0000 7efd17200640 1 -- start start 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.839+0000 7efd17200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd18143620 0x7efd180b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd17200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efd18071940 0x7efd180cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd17200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd180b2fa0 con 0x7efd180b2a90 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd17200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd180b3280 con 0x7efd18071940 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd17200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180b3560 con 0x7efd18143620 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd0fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7efd180b2a90 0x7efd18071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48734/0 (socket says 172.21.15.2:48734) 2024-01-15T18:12:06.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd0fe00640 1 -- 172.21.15.2:0/451568825 learned_addr learned my addr 172.21.15.2:0/451568825 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:06.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.840+0000 7efd15e00640 1 --2- 172.21.15.2:0/451568825 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd18143620 0x7efd180b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd16800640 1 --2- 172.21.15.2:0/451568825 >> v2:172.21.15.137:3300/0 conn(0x7efd18071940 0x7efd180cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd15400640 1 -- 172.21.15.2:0/451568825 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993001134 0 0) 0x7efd180b2fa0 con 0x7efd180b2a90 2024-01-15T18:12:06.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd15400640 1 -- 172.21.15.2:0/451568825 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf8003520 con 0x7efd180b2a90 2024-01-15T18:12:06.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd15e00640 1 -- 172.21.15.2:0/451568825 >> v2:172.21.15.137:3300/0 conn(0x7efd18071940 msgr2=0x7efd180cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:06.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd15e00640 1 --2- 172.21.15.2:0/451568825 >> v2:172.21.15.137:3300/0 conn(0x7efd18071940 0x7efd180cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd15e00640 1 -- 172.21.15.2:0/451568825 >> v1:172.21.15.2:6789/0 conn(0x7efd180b2a90 legacy=0x7efd18071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.841+0000 7efd15e00640 1 -- 172.21.15.2:0/451568825 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd180b3840 con 0x7efd18143620 2024-01-15T18:12:06.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.843+0000 7efd15e00640 1 --2- 172.21.15.2:0/451568825 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd18143620 0x7efd180b2550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efd080097a0 tx=0x7efd0802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd5f1d6ee9ab885 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.843+0000 7efd15400640 1 -- 172.21.15.2:0/451568825 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0802d040 con 0x7efd18143620 2024-01-15T18:12:06.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.843+0000 7efd15400640 1 -- 172.21.15.2:0/451568825 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd08031810 con 0x7efd18143620 2024-01-15T18:12:06.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.843+0000 7efd15400640 1 -- 172.21.15.2:0/451568825 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08031ba0 con 0x7efd18143620 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 -- 172.21.15.2:0/451568825 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd18143620 msgr2=0x7efd180b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 --2- 172.21.15.2:0/451568825 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd18143620 0x7efd180b2550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efd080097a0 tx=0x7efd0802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 -- 172.21.15.2:0/451568825 shutdown_connections 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 --2- 172.21.15.2:0/451568825 >> v2:172.21.15.137:3300/0 conn(0x7efd18071940 0x7efd180cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 --2- 172.21.15.2:0/451568825 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd18143620 0x7efd180b2550 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 -- 172.21.15.2:0/451568825 >> 172.21.15.2:0/451568825 conn(0x7efd180b0840 msgr2=0x7efd180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 -- 172.21.15.2:0/451568825 shutdown_connections 2024-01-15T18:12:06.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.844+0000 7efd17200640 1 -- 172.21.15.2:0/451568825 wait complete. 2024-01-15T18:12:06.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.846+0000 7efd17200640 1 Processor -- start 2024-01-15T18:12:06.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.846+0000 7efd17200640 1 -- start start 2024-01-15T18:12:06.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd17200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 0x7efd181f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd17200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd180b2a90 0x7efd180ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd17200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd18143620 0x7efd180ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd17200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd181f6100 con 0x7efd18071940 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd17200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180a9200 con 0x7efd18143620 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd17200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180a94e0 con 0x7efd180b2a90 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd15e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 0x7efd181f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd15e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 0x7efd181f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36268/0 (socket says 172.21.15.2:36268) 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd15e00640 1 -- 172.21.15.2:0/3713709978 learned_addr learned my addr 172.21.15.2:0/3713709978 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd0fe00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd180b2a90 0x7efd180ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.847+0000 7efd16800640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd18143620 0x7efd180ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd15e00640 1 -- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd180b2a90 msgr2=0x7efd180ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd15e00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd180b2a90 0x7efd180ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd15e00640 1 -- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd18143620 msgr2=0x7efd180ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd15e00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd18143620 0x7efd180ad550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd15e00640 1 -- 172.21.15.2:0/3713709978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd08009400 con 0x7efd18071940 2024-01-15T18:12:06.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd0fe00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd180b2a90 0x7efd180ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:06.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd15e00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 0x7efd181f5bc0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7efd08000c00 tx=0x7efd080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08006e10 con 0x7efd18071940 2024-01-15T18:12:06.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0803ad60 con 0x7efd18071940 2024-01-15T18:12:06.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd08044630 con 0x7efd18071940 2024-01-15T18:12:06.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.848+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd180a97c0 con 0x7efd18071940 2024-01-15T18:12:06.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.849+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd180a9d80 con 0x7efd18071940 2024-01-15T18:12:06.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.850+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd180a9f70 con 0x7efd18071940 2024-01-15T18:12:06.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.851+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7efd0804d450 con 0x7efd18071940 2024-01-15T18:12:06.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.852+0000 7efd0e000640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efcf006c190 0x7efcf006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:06.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.852+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7efd080aab00 con 0x7efd18071940 2024-01-15T18:12:06.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.853+0000 7efd0fe00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efcf006c190 0x7efcf006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:06.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.854+0000 7efd0fe00640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efcf006c190 0x7efcf006e650 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efd1813aed0 tx=0x7efd00009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:06.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:06.857+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efd080770e0 con 0x7efd18071940 2024-01-15T18:12:07.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:07.092+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7efd180ce070 con 0x7efcf006c190 2024-01-15T18:12:08.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.791+0000 7efd0e000640 1 -- 172.21.15.2:0/3713709978 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7efd180ce070 con 0x7efcf006c190 2024-01-15T18:12:08.794 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:08.795 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:12:08.795 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:12:08.795 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:08.795 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.794+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efcf006c190 msgr2=0x7efcf006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.794+0000 7efd17200640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efcf006c190 0x7efcf006e650 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7efd1813aed0 tx=0x7efd00009040 comp rx=0 tx=0).stop 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.794+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 msgr2=0x7efd181f5bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.794+0000 7efd17200640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 0x7efd181f5bc0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7efd08000c00 tx=0x7efd080046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.794+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 shutdown_connections 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efcf006c190 0x7efcf006e650 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efd18143620 0x7efd180ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:08.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efd180b2a90 0x7efd180ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:08.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 --2- 172.21.15.2:0/3713709978 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efd18071940 0x7efd181f5bc0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:08.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 >> 172.21.15.2:0/3713709978 conn(0x7efd180b0840 msgr2=0x7efd180cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:08.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 shutdown_connections 2024-01-15T18:12:08.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:08.795+0000 7efd17200640 1 -- 172.21.15.2:0/3713709978 wait complete. 2024-01-15T18:12:08.821 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-15T18:12:08.821 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:12:09.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.056+0000 7f0594400640 1 Processor -- start 2024-01-15T18:12:09.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.056+0000 7f0594400640 1 -- start start 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0594400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0adb10 0x7f058c0adef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0594400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f058c1448e0 0x7f058c070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0594400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f058c0721f0 0x7f058c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0594400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058c06cd00 con 0x7f058c1448e0 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0594400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058c0ae430 con 0x7f058c0721f0 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0594400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c144150 con 0x7f058c0adb10 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0591000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f058c1448e0 0x7f058c070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0591000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f058c1448e0 0x7f058c070230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36274/0 (socket says 172.21.15.2:36274) 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.058+0000 7f0591000640 1 -- 172.21.15.2:0/2485202361 learned_addr learned my addr 172.21.15.2:0/2485202361 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:09.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0591a00640 1 --2- 172.21.15.2:0/2485202361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0adb10 0x7f058c0adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0592400640 1 --2- 172.21.15.2:0/2485202361 >> v2:172.21.15.137:3300/0 conn(0x7f058c0721f0 0x7f058c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0591000640 1 -- 172.21.15.2:0/2485202361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0adb10 msgr2=0x7f058c0adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0591000640 1 --2- 172.21.15.2:0/2485202361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0adb10 0x7f058c0adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0591000640 1 -- 172.21.15.2:0/2485202361 >> v2:172.21.15.137:3300/0 conn(0x7f058c0721f0 msgr2=0x7f058c06d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0591000640 1 --2- 172.21.15.2:0/2485202361 >> v2:172.21.15.137:3300/0 conn(0x7f058c0721f0 0x7f058c06d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0591000640 1 -- 172.21.15.2:0/2485202361 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0abed0 con 0x7f058c1448e0 2024-01-15T18:12:09.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.059+0000 7f0592400640 1 --2- 172.21.15.2:0/2485202361 >> v2:172.21.15.137:3300/0 conn(0x7f058c0721f0 0x7f058c06d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:09.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.060+0000 7f0591000640 1 --2- 172.21.15.2:0/2485202361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c070230 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f058c067db0 tx=0x7f057402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efac7a060a501fb7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f058be00640 1 -- 172.21.15.2:0/2485202361 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574031870 con 0x7f058c1448e0 2024-01-15T18:12:09.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f058be00640 1 -- 172.21.15.2:0/2485202361 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05740319d0 con 0x7f058c1448e0 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f058be00640 1 -- 172.21.15.2:0/2485202361 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574031d60 con 0x7f058c1448e0 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 -- 172.21.15.2:0/2485202361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 msgr2=0x7f058c070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 --2- 172.21.15.2:0/2485202361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c070230 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f058c067db0 tx=0x7f057402b910 comp rx=0 tx=0).stop 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 -- 172.21.15.2:0/2485202361 shutdown_connections 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 --2- 172.21.15.2:0/2485202361 >> v2:172.21.15.137:3300/0 conn(0x7f058c0721f0 0x7f058c06d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 --2- 172.21.15.2:0/2485202361 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c070230 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 --2- 172.21.15.2:0/2485202361 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0adb10 0x7f058c0adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 -- 172.21.15.2:0/2485202361 >> 172.21.15.2:0/2485202361 conn(0x7f058c0ced60 msgr2=0x7f058c0cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 -- 172.21.15.2:0/2485202361 shutdown_connections 2024-01-15T18:12:09.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.061+0000 7f0594400640 1 -- 172.21.15.2:0/2485202361 wait complete. 2024-01-15T18:12:09.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.063+0000 7f0594400640 1 Processor -- start 2024-01-15T18:12:09.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.063+0000 7f0594400640 1 -- start start 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0594400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0721f0 0x7f058c15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0594400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0adb10 0x7f058c15b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0594400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c09bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0594400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c1404f0 con 0x7f058c1448e0 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0594400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c1407d0 con 0x7f058c0adb10 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0594400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c140ab0 con 0x7f058c0721f0 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.064+0000 7f0592400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c09bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c09bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36276/0 (socket says 172.21.15.2:36276) 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 -- 172.21.15.2:0/619896750 learned_addr learned my addr 172.21.15.2:0/619896750 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:09.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0591000640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0adb10 0x7f058c15b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 -- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0721f0 msgr2=0x7f058c15aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0591a00640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0721f0 0x7f058c15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0721f0 0x7f058c15aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 -- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0adb10 msgr2=0x7f058c15b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0adb10 0x7f058c15b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 -- 172.21.15.2:0/619896750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740094d0 con 0x7f058c1448e0 2024-01-15T18:12:09.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.065+0000 7f0592400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c09bbd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f058400cd10 tx=0x7f058400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.066+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058400f680 con 0x7f058c1448e0 2024-01-15T18:12:09.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.066+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058400f800 con 0x7f058c1448e0 2024-01-15T18:12:09.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.066+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058400fbb0 con 0x7f058c1448e0 2024-01-15T18:12:09.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.066+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c140d90 con 0x7f058c1448e0 2024-01-15T18:12:09.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.066+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c140130 con 0x7f058c1448e0 2024-01-15T18:12:09.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.067+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c1412b0 con 0x7f058c1448e0 2024-01-15T18:12:09.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.068+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f058400b040 con 0x7f058c1448e0 2024-01-15T18:12:09.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.069+0000 7f0589600640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f056406c160 0x7f056406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.069+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0584089070 con 0x7f058c1448e0 2024-01-15T18:12:09.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.070+0000 7f0591a00640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f056406c160 0x7f056406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.071+0000 7f0591a00640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f056406c160 0x7f056406e620 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f057c006fd0 tx=0x7f057c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.074+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f058c1412b0 con 0x7f058c1448e0 2024-01-15T18:12:09.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.309+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f058c1412b0 con 0x7f056406c160 2024-01-15T18:12:09.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.322+0000 7f0589600640 1 -- 172.21.15.2:0/619896750 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f058c1412b0 con 0x7f056406c160 2024-01-15T18:12:09.326 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:09.326 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:12:09.326 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "complete" 2024-01-15T18:12:09.326 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:09.326 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f056406c160 msgr2=0x7f056406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f056406c160 0x7f056406e620 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f057c006fd0 tx=0x7f057c008040 comp rx=0 tx=0).stop 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 msgr2=0x7f058c09bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c09bbd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f058400cd10 tx=0x7f058400a650 comp rx=0 tx=0).stop 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 shutdown_connections 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f056406c160 0x7f056406e620 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058c1448e0 0x7f058c09bbd0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f058c0adb10 0x7f058c15b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 --2- 172.21.15.2:0/619896750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058c0721f0 0x7f058c15aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.325+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 >> 172.21.15.2:0/619896750 conn(0x7f058c0ced60 msgr2=0x7f058c0af8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:09.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.326+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 shutdown_connections 2024-01-15T18:12:09.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.326+0000 7f0594400640 1 -- 172.21.15.2:0/619896750 wait complete. 2024-01-15T18:12:09.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:12:09.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.571+0000 7f6334200640 1 Processor -- start 2024-01-15T18:12:09.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.572+0000 7f6334200640 1 -- start start 2024-01-15T18:12:09.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.573+0000 7f6334200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f632c144b80 0x7f632c0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.573+0000 7f6334200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f632c06cfc0 0x7f632c15b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.573+0000 7f6334200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632c0b35a0 con 0x7f632c0b30a0 2024-01-15T18:12:09.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.573+0000 7f6334200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c0b4220 con 0x7f632c06cfc0 2024-01-15T18:12:09.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.573+0000 7f6334200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c144f60 con 0x7f632c144b80 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f632c06cfc0 0x7f632c15b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f632c06cfc0 0x7f632c15b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36288/0 (socket says 172.21.15.2:36288) 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 -- 172.21.15.2:0/969057606 learned_addr learned my addr 172.21.15.2:0/969057606 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6331800640 1 --2- 172.21.15.2:0/969057606 >> v2:172.21.15.137:3300/0 conn(0x7f632c144b80 0x7f632c0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 -- 172.21.15.2:0/969057606 >> v2:172.21.15.137:3300/0 conn(0x7f632c144b80 msgr2=0x7f632c0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 --2- 172.21.15.2:0/969057606 >> v2:172.21.15.137:3300/0 conn(0x7f632c144b80 0x7f632c0cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 -- 172.21.15.2:0/969057606 >> v1:172.21.15.137:6789/0 conn(0x7f632c0b30a0 legacy=0x7f632c06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:09.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6332200640 1 -- 172.21.15.2:0/969057606 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0b3880 con 0x7f632c06cfc0 2024-01-15T18:12:09.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.574+0000 7f6331800640 1 --2- 172.21.15.2:0/969057606 >> v2:172.21.15.137:3300/0 conn(0x7f632c144b80 0x7f632c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:09.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.575+0000 7f6332200640 1 --2- 172.21.15.2:0/969057606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15b9e0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f632c0aebd0 tx=0x7f632402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=833288ce489f47aa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f632be00640 1 -- 172.21.15.2:0/969057606 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f632402c9f0 con 0x7f632c06cfc0 2024-01-15T18:12:09.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f632be00640 1 -- 172.21.15.2:0/969057606 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632402cb50 con 0x7f632c06cfc0 2024-01-15T18:12:09.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f632be00640 1 -- 172.21.15.2:0/969057606 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63240343f0 con 0x7f632c06cfc0 2024-01-15T18:12:09.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f6334200640 1 -- 172.21.15.2:0/969057606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 msgr2=0x7f632c15b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f6334200640 1 --2- 172.21.15.2:0/969057606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15b9e0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f632c0aebd0 tx=0x7f632402be50 comp rx=0 tx=0).stop 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f6334200640 1 -- 172.21.15.2:0/969057606 shutdown_connections 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f6334200640 1 --2- 172.21.15.2:0/969057606 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15b9e0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f6334200640 1 --2- 172.21.15.2:0/969057606 >> v2:172.21.15.137:3300/0 conn(0x7f632c144b80 0x7f632c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.576+0000 7f6334200640 1 -- 172.21.15.2:0/969057606 >> 172.21.15.2:0/969057606 conn(0x7f632c0b46d0 msgr2=0x7f632c0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.577+0000 7f6334200640 1 -- 172.21.15.2:0/969057606 shutdown_connections 2024-01-15T18:12:09.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.577+0000 7f6334200640 1 -- 172.21.15.2:0/969057606 wait complete. 2024-01-15T18:12:09.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.578+0000 7f6334200640 1 Processor -- start 2024-01-15T18:12:09.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.578+0000 7f6334200640 1 -- start start 2024-01-15T18:12:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.579+0000 7f6334200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.579+0000 7f6334200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c0b30a0 0x7f632c071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.579+0000 7f6334200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632c144b80 0x7f632c13db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.579+0000 7f6334200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1f2df0 con 0x7f632c06cfc0 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.579+0000 7f6334200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0ad760 con 0x7f632c0b30a0 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.579+0000 7f6334200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0ada40 con 0x7f632c144b80 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36300/0 (socket says 172.21.15.2:36300) 2024-01-15T18:12:09.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 -- 172.21.15.2:0/2830255269 learned_addr learned my addr 172.21.15.2:0/2830255269 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:09.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6332200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632c144b80 0x7f632c13db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 -- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632c144b80 msgr2=0x7f632c13db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6330e00640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c0b30a0 0x7f632c071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632c144b80 0x7f632c13db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 -- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c0b30a0 msgr2=0x7f632c071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c0b30a0 0x7f632c071cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6331800640 1 -- 172.21.15.2:0/2830255269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009b30 con 0x7f632c06cfc0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.580+0000 7f6330e00640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c0b30a0 0x7f632c071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.581+0000 7f6331800640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15aba0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c7f0 tx=0x7f631c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.581+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c007820 con 0x7f632c06cfc0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.581+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631c007980 con 0x7f632c06cfc0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.581+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c0add20 con 0x7f632c06cfc0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.581+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631c007cf0 con 0x7f632c06cfc0 2024-01-15T18:12:09.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.581+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c15b0e0 con 0x7f632c06cfc0 2024-01-15T18:12:09.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.582+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c0ae240 con 0x7f632c06cfc0 2024-01-15T18:12:09.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.583+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f631c0113f0 con 0x7f632c06cfc0 2024-01-15T18:12:09.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.584+0000 7f6329600640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406c160 0x7f630406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:09.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.584+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f631c088ca0 con 0x7f632c06cfc0 2024-01-15T18:12:09.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.585+0000 7f6330e00640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406c160 0x7f630406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:09.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.586+0000 7f6330e00640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406c160 0x7f630406e620 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f632c1410d0 tx=0x7f6314009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:09.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.588+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f631c055280 con 0x7f632c06cfc0 2024-01-15T18:12:09.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.813+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f632c0b3e40 con 0x7f630406c160 2024-01-15T18:12:09.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.817+0000 7f6329600640 1 -- 172.21.15.2:0/2830255269 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6314001cc0 con 0x7f630406c160 2024-01-15T18:12:09.820 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:12:09.820 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:09.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:12:09.820 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:09.820 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.819+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406c160 msgr2=0x7f630406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.819+0000 7f6334200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406c160 0x7f630406e620 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f632c1410d0 tx=0x7f6314009040 comp rx=0 tx=0).stop 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.819+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 msgr2=0x7f632c15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.819+0000 7f6334200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15aba0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f631c00c7f0 tx=0x7f631c00ccb0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 shutdown_connections 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f630406c160 0x7f630406e620 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f632c144b80 0x7f632c13db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f632c0b30a0 0x7f632c071cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 --2- 172.21.15.2:0/2830255269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f632c06cfc0 0x7f632c15aba0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:09.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 >> 172.21.15.2:0/2830255269 conn(0x7f632c0b46d0 msgr2=0x7f632c0cc9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:09.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 shutdown_connections 2024-01-15T18:12:09.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:09.820+0000 7f6334200640 1 -- 172.21.15.2:0/2830255269 wait complete. 2024-01-15T18:12:09.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:12:10.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.051+0000 7f5d84a00640 1 Processor -- start 2024-01-15T18:12:10.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.051+0000 7f5d84a00640 1 -- start start 2024-01-15T18:12:10.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.053+0000 7f5d84a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5d8015c050 0x7f5d8006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.053+0000 7f5d84a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8006dea0 con 0x7f5d800ced00 2024-01-15T18:12:10.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.053+0000 7f5d84a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8006e160 con 0x7f5d80158cb0 2024-01-15T18:12:10.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.053+0000 7f5d84a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d800668c0 con 0x7f5d8015c050 2024-01-15T18:12:10.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.053+0000 7f5d7de00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5d800ced00 0x7f5d8006e5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48736/0 (socket says 172.21.15.2:48736) 2024-01-15T18:12:10.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.053+0000 7f5d7de00640 1 -- 172.21.15.2:0/280724080 learned_addr learned my addr 172.21.15.2:0/280724080 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:10.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911725581 0 0) 0x7f5d8006dea0 con 0x7f5d800ced00 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60003520 con 0x7f5d800ced00 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7f200640 1 --2- 172.21.15.2:0/280724080 >> v2:172.21.15.137:3300/0 conn(0x7f5d8015c050 0x7f5d8006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2045787152 0 0) 0x7f5d60003520 con 0x7f5d800ced00 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 >> v2:172.21.15.137:3300/0 conn(0x7f5d8015c050 msgr2=0x7f5d8006a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 --2- 172.21.15.2:0/280724080 >> v2:172.21.15.137:3300/0 conn(0x7f5d8015c050 0x7f5d8006a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 >> v1:172.21.15.137:6789/0 conn(0x7f5d80158cb0 legacy=0x7f5d800ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8015c5a0 con 0x7f5d800ced00 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5d700019f0 con 0x7f5d800ced00 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 >> v1:172.21.15.2:6789/0 conn(0x7f5d800ced00 legacy=0x7f5d8006e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d600039c0 0x7f5d60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d600047c0 0x7f5d60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600051d0 0x7f5d6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800668c0 con 0x7f5d600047c0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6000c100 con 0x7f5d600039c0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.054+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6000c2c0 con 0x7f5d600051d0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7de00640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d600047c0 0x7f5d60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7e800640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d600039c0 0x7f5d60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7f200640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600051d0 0x7f5d6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7de00640 1 -- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600051d0 msgr2=0x7f5d6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7de00640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600051d0 0x7f5d6000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7de00640 1 -- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d600039c0 msgr2=0x7f5d60003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7de00640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d600039c0 0x7f5d60003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7de00640 1 -- 172.21.15.2:0/280724080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60003520 con 0x7f5d600047c0 2024-01-15T18:12:10.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.055+0000 7f5d7f200640 1 -- 172.21.15.2:0/280724080 reap_dead start 2024-01-15T18:12:10.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.056+0000 7f5d7de00640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d600047c0 0x7f5d60004c40 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000a720 tx=0x7f5d600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c7c703a69a0f58 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d70034630 con 0x7f5d600047c0 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7002f070 con 0x7f5d600047c0 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d7d400640 1 -- 172.21.15.2:0/280724080 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d700349f0 con 0x7f5d600047c0 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d84a00640 1 -- 172.21.15.2:0/280724080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d600047c0 msgr2=0x7f5d60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d84a00640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d600047c0 0x7f5d60004c40 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f5d7000a720 tx=0x7f5d600057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d84a00640 1 -- 172.21.15.2:0/280724080 shutdown_connections 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d84a00640 1 --2- 172.21.15.2:0/280724080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d600047c0 0x7f5d60004c40 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.057+0000 7f5d84a00640 1 -- 172.21.15.2:0/280724080 >> 172.21.15.2:0/280724080 conn(0x7f5d800cbf70 msgr2=0x7f5d800cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.058+0000 7f5d84a00640 1 -- 172.21.15.2:0/280724080 shutdown_connections 2024-01-15T18:12:10.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.058+0000 7f5d84a00640 1 -- 172.21.15.2:0/280724080 wait complete. 2024-01-15T18:12:10.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.059+0000 7f5d84a00640 1 Processor -- start 2024-01-15T18:12:10.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.060+0000 7f5d84a00640 1 -- start start 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d84a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600047c0 0x7f5d800cad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d84a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d80158070 0x7f5d800ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d84a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 0x7f5d800aee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d84a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800cb290 con 0x7f5d801584f0 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d84a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800ace50 con 0x7f5d80158070 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d84a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d800ad130 con 0x7f5d600047c0 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 0x7f5d800aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 0x7f5d800aee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36318/0 (socket says 172.21.15.2:36318) 2024-01-15T18:12:10.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 -- 172.21.15.2:0/248799691 learned_addr learned my addr 172.21.15.2:0/248799691 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:10.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7de00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d80158070 0x7f5d800ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 -- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600047c0 msgr2=0x7f5d800cad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7e800640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600047c0 0x7f5d800cad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600047c0 0x7f5d800cad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 -- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d80158070 msgr2=0x7f5d800ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d80158070 0x7f5d800ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.061+0000 7f5d7f200640 1 -- 172.21.15.2:0/248799691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d800ad410 con 0x7f5d801584f0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.062+0000 7f5d7f200640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 0x7f5d800aee20 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f5d800cf4a0 tx=0x7f5d7800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.062+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d78018070 con 0x7f5d801584f0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.062+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78004020 con 0x7f5d801584f0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.062+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7800c070 con 0x7f5d801584f0 2024-01-15T18:12:10.066 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.062+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d800ad6f0 con 0x7f5d801584f0 2024-01-15T18:12:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.062+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d801f3880 con 0x7f5d801584f0 2024-01-15T18:12:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.064+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d800adc10 con 0x7f5d801584f0 2024-01-15T18:12:10.067 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.064+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f5d78003610 con 0x7f5d801584f0 2024-01-15T18:12:10.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.065+0000 7f5d76a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d5806c160 0x7f5d5806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.065+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f5d78087c00 con 0x7f5d801584f0 2024-01-15T18:12:10.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.066+0000 7f5d7e800640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d5806c160 0x7f5d5806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.067+0000 7f5d7e800640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d5806c160 0x7f5d5806e620 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5d68007fd0 tx=0x7f5d68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.070+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d800adc10 con 0x7f5d801584f0 2024-01-15T18:12:10.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.296+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5d800665e0 con 0x7f5d5806c160 2024-01-15T18:12:10.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.307+0000 7f5d76a00640 1 -- 172.21.15.2:0/248799691 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f5d68016440 con 0x7f5d5806c160 2024-01-15T18:12:10.310 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:10.310 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:12:10.310 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:10.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:12:10.311 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:10.311 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:10.311 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1clone1" 2024-01-15T18:12:10.311 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:10.311 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:12:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d5806c160 msgr2=0x7f5d5806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d5806c160 0x7f5d5806e620 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5d68007fd0 tx=0x7f5d68009040 comp rx=0 tx=0).stop 2024-01-15T18:12:10.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 msgr2=0x7f5d800aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 0x7f5d800aee20 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f5d800cf4a0 tx=0x7f5d7800e830 comp rx=0 tx=0).stop 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 shutdown_connections 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5d5806c160 0x7f5d5806e620 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5d801584f0 0x7f5d800aee20 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5d80158070 0x7f5d800ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 --2- 172.21.15.2:0/248799691 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5d600047c0 0x7f5d800cad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 >> 172.21.15.2:0/248799691 conn(0x7f5d800cbf70 msgr2=0x7f5d8015b600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 shutdown_connections 2024-01-15T18:12:10.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.310+0000 7f5d84a00640 1 -- 172.21.15.2:0/248799691 wait complete. 2024-01-15T18:12:10.332 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:12:10.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.550+0000 7f45ad200640 1 Processor -- start 2024-01-15T18:12:10.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.550+0000 7f45ad200640 1 -- start start 2024-01-15T18:12:10.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.552+0000 7f45ad200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 0x7f45a80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.552+0000 7f45ad200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a80ce230 con 0x7f45a806d000 2024-01-15T18:12:10.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.552+0000 7f45ad200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45a80ce510 con 0x7f45a80cdd70 2024-01-15T18:12:10.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.552+0000 7f45ad200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80ce7f0 con 0x7f45a8144c00 2024-01-15T18:12:10.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.552+0000 7f45a6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f45a806d000 0x7f45a8070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:48744/0 (socket says 172.21.15.2:48744) 2024-01-15T18:12:10.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.552+0000 7f45a6800640 1 -- 172.21.15.2:0/54733841 learned_addr learned my addr 172.21.15.2:0/54733841 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:10.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a5e00640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 0x7f45a80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875310765 0 0) 0x7f45a80ce230 con 0x7f45a806d000 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4588003520 con 0x7f45a806d000 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831321093 0 0) 0x7f4588003520 con 0x7f45a806d000 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 msgr2=0x7f45a80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 0x7f45a80cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 >> v1:172.21.15.137:6789/0 conn(0x7f45a80cdd70 legacy=0x7f45a806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a80705f0 con 0x7f45a806d000 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f45a00019f0 con 0x7f45a806d000 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.553+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 >> v1:172.21.15.2:6789/0 conn(0x7f45a806d000 legacy=0x7f45a8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a4a00640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 0x7f45880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a4a00640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4588003f70 0x7f45880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a4a00640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588004d00 0x7f4588005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80ce7f0 con 0x7f4588003f70 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458800bac0 con 0x7f4588004d00 2024-01-15T18:12:10.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458800bc80 con 0x7f45a8144c00 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5400640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4588003f70 0x7f45880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5e00640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 0x7f45880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a6800640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588004d00 0x7f4588005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5400640 1 -- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 msgr2=0x7f45880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5400640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a8144c00 0x7f45880039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5400640 1 -- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588004d00 msgr2=0x7f4588005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5400640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588004d00 0x7f4588005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a5400640 1 -- 172.21.15.2:0/54733841 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588003520 con 0x7f4588003f70 2024-01-15T18:12:10.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.554+0000 7f45a6800640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588004d00 0x7f4588005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:10.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45a5400640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4588003f70 0x7f45880047c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f4590001750 tx=0x7f45880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5105c3875eba3560 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45900052d0 con 0x7f4588003f70 2024-01-15T18:12:10.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f459002e050 con 0x7f4588003f70 2024-01-15T18:12:10.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45a4a00640 1 -- 172.21.15.2:0/54733841 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590003830 con 0x7f4588003f70 2024-01-15T18:12:10.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45ad200640 1 -- 172.21.15.2:0/54733841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4588003f70 msgr2=0x7f45880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45ad200640 1 --2- 172.21.15.2:0/54733841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4588003f70 0x7f45880047c0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f4590001750 tx=0x7f45880057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45a6800640 1 -- 172.21.15.2:0/54733841 reap_dead start 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45ad200640 1 -- 172.21.15.2:0/54733841 shutdown_connections 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.556+0000 7f45ad200640 1 -- 172.21.15.2:0/54733841 >> 172.21.15.2:0/54733841 conn(0x7f45a80b45f0 msgr2=0x7f45a80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.557+0000 7f45ad200640 1 -- 172.21.15.2:0/54733841 shutdown_connections 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.557+0000 7f45ad200640 1 -- 172.21.15.2:0/54733841 wait complete. 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.558+0000 7f45ad200640 1 Processor -- start 2024-01-15T18:12:10.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.559+0000 7f45ad200640 1 -- start start 2024-01-15T18:12:10.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45ad200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588003f70 0x7f45a80b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45ad200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 0x7f45a8158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45ad200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a81f3f10 0x7f45a815d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45ad200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80ae060 con 0x7f45a81f6420 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45ad200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a80aca80 con 0x7f4588003f70 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45ad200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45a81591d0 con 0x7f45a81f3f10 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45a5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 0x7f45a8158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45a5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 0x7f45a8158c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36340/0 (socket says 172.21.15.2:36340) 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45a5400640 1 -- 172.21.15.2:0/1591086818 learned_addr learned my addr 172.21.15.2:0/1591086818 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.560+0000 7f45a5e00640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588003f70 0x7f45a80b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5400640 1 -- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a81f3f10 msgr2=0x7f45a815d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5400640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a81f3f10 0x7f45a815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5400640 1 -- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588003f70 msgr2=0x7f45a80b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5400640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588003f70 0x7f45a80b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5400640 1 -- 172.21.15.2:0/1591086818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a81f68a0 con 0x7f45a81f6420 2024-01-15T18:12:10.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5e00640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588003f70 0x7f45a80b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:10.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45a5400640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 0x7f45a8158c90 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f4590008fd0 tx=0x7f4590001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590007a30 con 0x7f45a81f6420 2024-01-15T18:12:10.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45900069e0 con 0x7f45a81f6420 2024-01-15T18:12:10.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4590006d90 con 0x7f45a81f6420 2024-01-15T18:12:10.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a81f6b20 con 0x7f45a81f6420 2024-01-15T18:12:10.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.561+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a81f70e0 con 0x7f45a81f6420 2024-01-15T18:12:10.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.563+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a81f72d0 con 0x7f45a81f6420 2024-01-15T18:12:10.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.564+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f45900035f0 con 0x7f45a81f6420 2024-01-15T18:12:10.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.565+0000 7f459e000640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458006c190 0x7f458006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:10.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.565+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f45900aa9a0 con 0x7f45a81f6420 2024-01-15T18:12:10.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.566+0000 7f45a5e00640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458006c190 0x7f458006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:10.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.567+0000 7f45a5e00640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458006c190 0x7f458006e650 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4598009a00 tx=0x7f4598008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:10.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.570+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4590077010 con 0x7f45a81f6420 2024-01-15T18:12:10.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.802+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f45a80ae060 con 0x7f458006c190 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.829+0000 7f459e000640 1 -- 172.21.15.2:0/1591086818 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f45a80ae060 con 0x7f458006c190 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:10.832 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:12:10.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458006c190 msgr2=0x7f458006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458006c190 0x7f458006e650 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f4598009a00 tx=0x7f4598008040 comp rx=0 tx=0).stop 2024-01-15T18:12:10.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 msgr2=0x7f45a8158c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 0x7f45a8158c90 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f4590008fd0 tx=0x7f4590001120 comp rx=0 tx=0).stop 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 shutdown_connections 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458006c190 0x7f458006e650 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45a81f3f10 0x7f45a815d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45a81f6420 0x7f45a8158c90 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 --2- 172.21.15.2:0/1591086818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4588003f70 0x7f45a80b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 >> 172.21.15.2:0/1591086818 conn(0x7f45a80b45f0 msgr2=0x7f45a8141be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 shutdown_connections 2024-01-15T18:12:10.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:10.832+0000 7f45ad200640 1 -- 172.21.15.2:0/1591086818 wait complete. 2024-01-15T18:12:10.856 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:12:11.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.077+0000 7f6878a00640 1 Processor -- start 2024-01-15T18:12:11.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.077+0000 7f6878a00640 1 -- start start 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6878a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f687006c920 0x7f68700cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6878a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f68700cd2c0 0x7f687006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6878a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68700cd750 con 0x7f687006d240 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6878a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687006eb20 con 0x7f687006c920 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6878a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700cd9f0 con 0x7f68700cd2c0 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6876000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f687006c920 0x7f68700cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6876000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f687006c920 0x7f68700cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36352/0 (socket says 172.21.15.2:36352) 2024-01-15T18:12:11.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.079+0000 7f6876000640 1 -- 172.21.15.2:0/3386597578 learned_addr learned my addr 172.21.15.2:0/3386597578 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:11.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.080+0000 7f6876000640 1 -- 172.21.15.2:0/3386597578 >> v2:172.21.15.137:3300/0 conn(0x7f68700cd2c0 msgr2=0x7f687006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.080+0000 7f6876000640 1 --2- 172.21.15.2:0/3386597578 >> v2:172.21.15.137:3300/0 conn(0x7f68700cd2c0 0x7f687006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.080+0000 7f6876000640 1 -- 172.21.15.2:0/3386597578 >> v1:172.21.15.137:6789/0 conn(0x7f687006d240 legacy=0x7f68700ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.080+0000 7f6876000640 1 -- 172.21.15.2:0/3386597578 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687006aec0 con 0x7f687006c920 2024-01-15T18:12:11.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.081+0000 7f6876000640 1 --2- 172.21.15.2:0/3386597578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f687006c920 0x7f68700cf050 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f68700b59c0 tx=0x7f686002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227a7c972529ef50 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.081+0000 7f6874c00640 1 -- 172.21.15.2:0/3386597578 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860031800 con 0x7f687006c920 2024-01-15T18:12:11.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.081+0000 7f6874c00640 1 -- 172.21.15.2:0/3386597578 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6860031960 con 0x7f687006c920 2024-01-15T18:12:11.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.081+0000 7f6874c00640 1 -- 172.21.15.2:0/3386597578 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860031cf0 con 0x7f687006c920 2024-01-15T18:12:11.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.082+0000 7f6878a00640 1 -- 172.21.15.2:0/3386597578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f687006c920 msgr2=0x7f68700cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.082+0000 7f6878a00640 1 --2- 172.21.15.2:0/3386597578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f687006c920 0x7f68700cf050 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f68700b59c0 tx=0x7f686002b8a0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.082+0000 7f6878a00640 1 -- 172.21.15.2:0/3386597578 shutdown_connections 2024-01-15T18:12:11.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.082+0000 7f6878a00640 1 --2- 172.21.15.2:0/3386597578 >> v2:172.21.15.137:3300/0 conn(0x7f68700cd2c0 0x7f687006cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.082+0000 7f6878a00640 1 --2- 172.21.15.2:0/3386597578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f687006c920 0x7f68700cf050 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.082+0000 7f6878a00640 1 -- 172.21.15.2:0/3386597578 >> 172.21.15.2:0/3386597578 conn(0x7f68700cc170 msgr2=0x7f68700cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.083+0000 7f6878a00640 1 -- 172.21.15.2:0/3386597578 shutdown_connections 2024-01-15T18:12:11.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.083+0000 7f6878a00640 1 -- 172.21.15.2:0/3386597578 wait complete. 2024-01-15T18:12:11.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.084+0000 7f6878a00640 1 Processor -- start 2024-01-15T18:12:11.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.084+0000 7f6878a00640 1 -- start start 2024-01-15T18:12:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.085+0000 7f6878a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f687006c920 0x7f68701e26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.085+0000 7f6878a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f687006d240 0x7f687015c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.085+0000 7f6878a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 0x7f687015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.085+0000 7f6878a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687015d030 con 0x7f68700cd2c0 2024-01-15T18:12:11.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.085+0000 7f6878a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687015b020 con 0x7f687006c920 2024-01-15T18:12:11.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.085+0000 7f6878a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687015b300 con 0x7f687006d240 2024-01-15T18:12:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 0x7f687015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 0x7f687015caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36356/0 (socket says 172.21.15.2:36356) 2024-01-15T18:12:11.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 -- 172.21.15.2:0/1627686578 learned_addr learned my addr 172.21.15.2:0/1627686578 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6875600640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f687006d240 0x7f687015c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876000640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f687006c920 0x7f68701e26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 -- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f687006d240 msgr2=0x7f687015c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f687006d240 0x7f687015c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 -- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f687006c920 msgr2=0x7f68701e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f687006c920 0x7f68701e26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6876a00640 1 -- 172.21.15.2:0/1627686578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68600094d0 con 0x7f68700cd2c0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.086+0000 7f6875600640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f687006d240 0x7f687015c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.087+0000 7f6876a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 0x7f687015caf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f686800c780 tx=0x7f686800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.087+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868003d80 con 0x7f68700cd2c0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.087+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686800ce30 con 0x7f68700cd2c0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.087+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f687015b5e0 con 0x7f68700cd2c0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.087+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868005290 con 0x7f68700cd2c0 2024-01-15T18:12:11.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.087+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f687015bba0 con 0x7f68700cd2c0 2024-01-15T18:12:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.088+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687015bdc0 con 0x7f68700cd2c0 2024-01-15T18:12:11.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.090+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f68680053f0 con 0x7f68700cd2c0 2024-01-15T18:12:11.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.091+0000 7f686e000640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f684806c160 0x7f684806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.091+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f68680885f0 con 0x7f68700cd2c0 2024-01-15T18:12:11.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.091+0000 7f6876000640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f684806c160 0x7f684806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.093+0000 7f6876000640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f684806c160 0x7f684806e620 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6860036000 tx=0x7f686003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.094+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6868054bd0 con 0x7f68700cd2c0 2024-01-15T18:12:11.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.322+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f687006b760 con 0x7f684806c160 2024-01-15T18:12:11.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.389+0000 7f686e000640 1 -- 172.21.15.2:0/1627686578 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6860031c50 con 0x7f684806c160 2024-01-15T18:12:11.392 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:11.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f684806c160 msgr2=0x7f684806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f684806c160 0x7f684806e620 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6860036000 tx=0x7f686003c040 comp rx=0 tx=0).stop 2024-01-15T18:12:11.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 msgr2=0x7f687015caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 0x7f687015caf0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f686800c780 tx=0x7f686800cc40 comp rx=0 tx=0).stop 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 shutdown_connections 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f684806c160 0x7f684806e620 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.392+0000 7f6878a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68700cd2c0 0x7f687015caf0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.393+0000 7f6878a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f687006d240 0x7f687015c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.393+0000 7f6878a00640 1 --2- 172.21.15.2:0/1627686578 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f687006c920 0x7f68701e26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.393+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 >> 172.21.15.2:0/1627686578 conn(0x7f68700cc170 msgr2=0x7f68700af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.393+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 shutdown_connections 2024-01-15T18:12:11.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.393+0000 7f6878a00640 1 -- 172.21.15.2:0/1627686578 wait complete. 2024-01-15T18:12:11.418 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:12:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.639+0000 7fd862000640 1 Processor -- start 2024-01-15T18:12:11.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.639+0000 7fd862000640 1 -- start start 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.641+0000 7fd862000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.641+0000 7fd862000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd85c161890 0x7fd85c160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.641+0000 7fd862000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd85c06c5e0 con 0x7fd85c0b2b60 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.641+0000 7fd862000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd85c06c8c0 con 0x7fd85c161890 2024-01-15T18:12:11.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.641+0000 7fd862000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c071c00 con 0x7fd85c15d270 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.642+0000 7fd85be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd85c161890 0x7fd85c160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.642+0000 7fd85be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd85c161890 0x7fd85c160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:46434/0 (socket says 172.21.15.2:46434) 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.642+0000 7fd85be00640 1 -- 172.21.15.2:0/2928319231 learned_addr learned my addr 172.21.15.2:0/2928319231 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:11.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.642+0000 7fd860c00640 1 --2- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.643+0000 7fd85be00640 1 -- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 msgr2=0x7fd85c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.643+0000 7fd85be00640 1 --2- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c0b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.643+0000 7fd85be00640 1 -- 172.21.15.2:0/2928319231 >> v1:172.21.15.137:6789/0 conn(0x7fd85c0b2b60 legacy=0x7fd85c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.643+0000 7fd85be00640 1 -- 172.21.15.2:0/2928319231 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85c071ee0 con 0x7fd85c161890 2024-01-15T18:12:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.644+0000 7fd85be00640 1 --2- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c161890 0x7fd85c160c60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd844009870 tx=0x7fd84402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d99b7a6476963e40 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd85b400640 1 -- 172.21.15.2:0/2928319231 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8440317c0 con 0x7fd85c161890 2024-01-15T18:12:11.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd85b400640 1 -- 172.21.15.2:0/2928319231 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd844031920 con 0x7fd85c161890 2024-01-15T18:12:11.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd85b400640 1 -- 172.21.15.2:0/2928319231 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd844031cb0 con 0x7fd85c161890 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd862000640 1 -- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c161890 msgr2=0x7fd85c160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd862000640 1 --2- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c161890 0x7fd85c160c60 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd844009870 tx=0x7fd84402b860 comp rx=0 tx=0).stop 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd862000640 1 -- 172.21.15.2:0/2928319231 shutdown_connections 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd862000640 1 --2- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c161890 0x7fd85c160c60 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd862000640 1 --2- 172.21.15.2:0/2928319231 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c0b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.645+0000 7fd862000640 1 -- 172.21.15.2:0/2928319231 >> 172.21.15.2:0/2928319231 conn(0x7fd85c0b0840 msgr2=0x7fd85c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.646+0000 7fd862000640 1 -- 172.21.15.2:0/2928319231 shutdown_connections 2024-01-15T18:12:11.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.646+0000 7fd862000640 1 -- 172.21.15.2:0/2928319231 wait complete. 2024-01-15T18:12:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.647+0000 7fd862000640 1 Processor -- start 2024-01-15T18:12:11.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.648+0000 7fd862000640 1 -- start start 2024-01-15T18:12:11.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.648+0000 7fd862000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c0b2b60 0x7fd85c1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.648+0000 7fd862000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd862000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 0x7fd85c15b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd862000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c1f27a0 con 0x7fd85c161890 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd862000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c154360 con 0x7fd85c0b2b60 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd862000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85c154640 con 0x7fd85c15d270 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd861600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 0x7fd85c15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd861600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 0x7fd85c15b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36364/0 (socket says 172.21.15.2:36364) 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd861600640 1 -- 172.21.15.2:0/1641094940 learned_addr learned my addr 172.21.15.2:0/1641094940 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd860c00640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c0b2b60 0x7fd85c1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.649+0000 7fd85be00640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd861600640 1 -- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 msgr2=0x7fd85c153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd861600640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd861600640 1 -- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c0b2b60 msgr2=0x7fd85c1f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd861600640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c0b2b60 0x7fd85c1f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd861600640 1 -- 172.21.15.2:0/1641094940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8440094d0 con 0x7fd85c161890 2024-01-15T18:12:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd861600640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 0x7fd85c15b490 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c730 tx=0x7fd85400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd854003b70 con 0x7fd85c161890 2024-01-15T18:12:11.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd854003cd0 con 0x7fd85c161890 2024-01-15T18:12:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.650+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85400aa90 con 0x7fd85c161890 2024-01-15T18:12:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.651+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd85c154920 con 0x7fd85c161890 2024-01-15T18:12:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.651+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd85c0ae010 con 0x7fd85c161890 2024-01-15T18:12:11.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.652+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd85c0adb90 con 0x7fd85c161890 2024-01-15T18:12:11.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.653+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fd85400abf0 con 0x7fd85c161890 2024-01-15T18:12:11.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.654+0000 7fd858c00640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd83406c160 0x7fd83406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:11.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.654+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fd854088380 con 0x7fd85c161890 2024-01-15T18:12:11.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.655+0000 7fd860c00640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd83406c160 0x7fd83406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:11.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.656+0000 7fd860c00640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd83406c160 0x7fd83406e620 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009b40 tx=0x7fd84c009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:11.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.658+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd85c0adb90 con 0x7fd85c161890 2024-01-15T18:12:11.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.883+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd85c1547b0 con 0x7fd83406c160 2024-01-15T18:12:11.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.927+0000 7fd858c00640 1 -- 172.21.15.2:0/1641094940 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd85c1547b0 con 0x7fd83406c160 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd83406c160 msgr2=0x7fd83406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd83406c160 0x7fd83406e620 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fd84c009b40 tx=0x7fd84c009190 comp rx=0 tx=0).stop 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 msgr2=0x7fd85c15b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 0x7fd85c15b490 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fd85400c730 tx=0x7fd85400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 shutdown_connections 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd83406c160 0x7fd83406e620 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd85c161890 0x7fd85c15b490 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd85c15d270 0x7fd85c153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 --2- 172.21.15.2:0/1641094940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd85c0b2b60 0x7fd85c1f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:11.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 >> 172.21.15.2:0/1641094940 conn(0x7fd85c0b0840 msgr2=0x7fd85c1578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:11.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 shutdown_connections 2024-01-15T18:12:11.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:11.930+0000 7fd862000640 1 -- 172.21.15.2:0/1641094940 wait complete. 2024-01-15T18:12:11.938 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.935+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.938 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.935+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.938 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.935+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.938 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.935+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.938 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.935+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.955 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T18:12:11.966 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.964+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.966 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.964+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.967 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.964+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.967 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.964+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:11.967 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:11.964+0000 7fe20e800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:12.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.154+0000 7ff410400640 1 Processor -- start 2024-01-15T18:12:12.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.155+0000 7ff410400640 1 -- start start 2024-01-15T18:12:12.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.156+0000 7ff410400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff40815d730 0x7ff4080d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:12.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.156+0000 7ff410400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff4080d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:12.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.156+0000 7ff410400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4080d1900 con 0x7ff4080b4c70 2024-01-15T18:12:12.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.156+0000 7ff410400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4080d1be0 con 0x7ff40815d730 2024-01-15T18:12:12.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.156+0000 7ff410400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080d1ec0 con 0x7ff4080b8c30 2024-01-15T18:12:12.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.157+0000 7ff40d000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff40815d730 0x7ff4080d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:12.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.157+0000 7ff40da00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7ff4080b4c70 0x7ff4080b5050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:59492/0 (socket says 172.21.15.2:59492) 2024-01-15T18:12:12.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.157+0000 7ff40da00640 1 -- 172.21.15.2:0/2139795944 learned_addr learned my addr 172.21.15.2:0/2139795944 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:12.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.157+0000 7ff40e400640 1 --2- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff4080d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:12.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.158+0000 7ff40d000640 1 -- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 msgr2=0x7ff4080d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:12.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.158+0000 7ff40d000640 1 --2- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff4080d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.158+0000 7ff40d000640 1 -- 172.21.15.2:0/2139795944 >> v1:172.21.15.137:6789/0 conn(0x7ff4080b4c70 legacy=0x7ff4080b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:12.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.158+0000 7ff40d000640 1 -- 172.21.15.2:0/2139795944 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4080d21a0 con 0x7ff40815d730 2024-01-15T18:12:12.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.159+0000 7ff40d000640 1 --2- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff40815d730 0x7ff4080d1270 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00097a0 tx=0x7ff3f002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7df526addbaa01be server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:12.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.159+0000 7ff407e00640 1 -- 172.21.15.2:0/2139795944 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0031830 con 0x7ff40815d730 2024-01-15T18:12:12.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.160+0000 7ff407e00640 1 -- 172.21.15.2:0/2139795944 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f0031990 con 0x7ff40815d730 2024-01-15T18:12:12.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.160+0000 7ff407e00640 1 -- 172.21.15.2:0/2139795944 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0031d20 con 0x7ff40815d730 2024-01-15T18:12:12.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.160+0000 7ff410400640 1 -- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff40815d730 msgr2=0x7ff4080d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:12.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.160+0000 7ff410400640 1 --2- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff40815d730 0x7ff4080d1270 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff3f00097a0 tx=0x7ff3f002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.161+0000 7ff410400640 1 -- 172.21.15.2:0/2139795944 shutdown_connections 2024-01-15T18:12:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.161+0000 7ff410400640 1 --2- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff4080d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.161+0000 7ff410400640 1 --2- 172.21.15.2:0/2139795944 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff40815d730 0x7ff4080d1270 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.161+0000 7ff410400640 1 -- 172.21.15.2:0/2139795944 >> 172.21.15.2:0/2139795944 conn(0x7ff408072210 msgr2=0x7ff4080cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.161+0000 7ff410400640 1 -- 172.21.15.2:0/2139795944 shutdown_connections 2024-01-15T18:12:12.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.161+0000 7ff410400640 1 -- 172.21.15.2:0/2139795944 wait complete. 2024-01-15T18:12:12.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.163+0000 7ff410400640 1 Processor -- start 2024-01-15T18:12:12.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.164+0000 7ff410400640 1 -- start start 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff410400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4080b4c70 0x7ff40814b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff410400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff40815af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff410400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 0x7ff40814a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff410400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4080cbb60 con 0x7ff40815d730 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff410400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40815fa00 con 0x7ff4080b4c70 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff410400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40814df20 con 0x7ff4080b8c30 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff40e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 0x7ff40814a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.165+0000 7ff40e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 0x7ff40814a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36368/0 (socket says 172.21.15.2:36368) 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 -- 172.21.15.2:0/3072897308 learned_addr learned my addr 172.21.15.2:0/3072897308 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40da00640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4080b4c70 0x7ff40814b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:12.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40d000640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff40815af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 -- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 msgr2=0x7ff40815af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff40815af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 -- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4080b4c70 msgr2=0x7ff40814b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4080b4c70 0x7ff40814b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 -- 172.21.15.2:0/3072897308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0009400 con 0x7ff40815d730 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.166+0000 7ff40e400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 0x7ff40814a1f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff400009dd0 tx=0x7ff40000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.167+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400014480 con 0x7ff40815d730 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.167+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40000f040 con 0x7ff40815d730 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.167+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4080d3060 con 0x7ff40815d730 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.167+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400014840 con 0x7ff40815d730 2024-01-15T18:12:12.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.167+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40815f0d0 con 0x7ff40815d730 2024-01-15T18:12:12.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.168+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4080686f0 con 0x7ff40815d730 2024-01-15T18:12:12.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.169+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7ff4000149a0 con 0x7ff40815d730 2024-01-15T18:12:12.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.170+0000 7ff405600640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3e006c160 0x7ff3e006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:12.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.170+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7ff4000894f0 con 0x7ff40815d730 2024-01-15T18:12:12.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.171+0000 7ff40da00640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3e006c160 0x7ff3e006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:12.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.172+0000 7ff40da00640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3e006c160 0x7ff3e006e620 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8005880 tx=0x7ff3f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:12.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.174+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff4000bee90 con 0x7ff40815d730 2024-01-15T18:12:12.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.397+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff4080c66f0 con 0x7ff3e006c160 2024-01-15T18:12:12.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.449+0000 7ff405600640 1 -- 172.21.15.2:0/3072897308 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff4080c66f0 con 0x7ff3e006c160 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3e006c160 msgr2=0x7ff3e006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3e006c160 0x7ff3e006e620 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7ff3f8005880 tx=0x7ff3f8009040 comp rx=0 tx=0).stop 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 msgr2=0x7ff40814a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 0x7ff40814a1f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff400009dd0 tx=0x7ff40000c490 comp rx=0 tx=0).stop 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 shutdown_connections 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3e006c160 0x7ff3e006e620 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff40815d730 0x7ff40814a1f0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4080b8c30 0x7ff40815af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 --2- 172.21.15.2:0/3072897308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4080b4c70 0x7ff40814b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:12.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 >> 172.21.15.2:0/3072897308 conn(0x7ff408072210 msgr2=0x7ff4080b6f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:12.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 shutdown_connections 2024-01-15T18:12:12.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:12.452+0000 7ff410400640 1 -- 172.21.15.2:0/3072897308 wait complete. 2024-01-15T18:12:12.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:12.475 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:12:13.448 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-01-15T18:12:13.450 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:12:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.679+0000 7f0edf200640 1 Processor -- start 2024-01-15T18:12:13.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.679+0000 7f0edf200640 1 -- start start 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.681+0000 7f0edf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0143d20 0x7f0ee015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.681+0000 7f0edf200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee0071090 con 0x7f0ee00720d0 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.681+0000 7f0edf200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ee006d160 con 0x7f0ee01447d0 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.681+0000 7f0edf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0071430 con 0x7f0ee0143d20 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.681+0000 7f0edd400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0ee00720d0 0x7f0ee00cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35288/0 (socket says 172.21.15.2:35288) 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.681+0000 7f0edd400640 1 -- 172.21.15.2:0/2643808349 learned_addr learned my addr 172.21.15.2:0/2643808349 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:13.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edde00640 1 --2- 172.21.15.2:0/2643808349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0143d20 0x7f0ee015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741099212 0 0) 0x7f0ee0071090 con 0x7f0ee00720d0 2024-01-15T18:12:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec0003520 con 0x7f0ee00720d0 2024-01-15T18:12:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043043714 0 0) 0x7f0ee006d160 con 0x7f0ee01447d0 2024-01-15T18:12:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ee0071090 con 0x7f0ee01447d0 2024-01-15T18:12:13.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1890308994 0 0) 0x7f0ec0003520 con 0x7f0ee00720d0 2024-01-15T18:12:13.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edde00640 1 -- 172.21.15.2:0/2643808349 >> v1:172.21.15.137:6789/0 conn(0x7f0ee01447d0 legacy=0x7f0ee00cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edde00640 1 -- 172.21.15.2:0/2643808349 >> v1:172.21.15.2:6789/0 conn(0x7f0ee00720d0 legacy=0x7f0ee00cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.682+0000 7f0edde00640 1 -- 172.21.15.2:0/2643808349 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee0144560 con 0x7f0ee0143d20 2024-01-15T18:12:13.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.684+0000 7f0edde00640 1 --2- 172.21.15.2:0/2643808349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0143d20 0x7f0ee015eb50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00097d0 tx=0x7f0ed002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8af279883e74636 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:13.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.684+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80019f0 con 0x7f0ee0143d20 2024-01-15T18:12:13.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.684+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ed002e050 con 0x7f0ee0143d20 2024-01-15T18:12:13.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edca00640 1 -- 172.21.15.2:0/2643808349 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ed0030710 con 0x7f0ee0143d20 2024-01-15T18:12:13.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edf200640 1 -- 172.21.15.2:0/2643808349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0143d20 msgr2=0x7f0ee015eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edf200640 1 --2- 172.21.15.2:0/2643808349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0143d20 0x7f0ee015eb50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0ed00097d0 tx=0x7f0ed002b770 comp rx=0 tx=0).stop 2024-01-15T18:12:13.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edf200640 1 -- 172.21.15.2:0/2643808349 shutdown_connections 2024-01-15T18:12:13.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edf200640 1 --2- 172.21.15.2:0/2643808349 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee0143d20 0x7f0ee015eb50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edf200640 1 -- 172.21.15.2:0/2643808349 >> 172.21.15.2:0/2643808349 conn(0x7f0ee00b0840 msgr2=0x7f0ee00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.685+0000 7f0edf200640 1 -- 172.21.15.2:0/2643808349 shutdown_connections 2024-01-15T18:12:13.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.686+0000 7f0edf200640 1 -- 172.21.15.2:0/2643808349 wait complete. 2024-01-15T18:12:13.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.688+0000 7f0edf200640 1 Processor -- start 2024-01-15T18:12:13.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.688+0000 7f0edf200640 1 -- start start 2024-01-15T18:12:13.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.689+0000 7f0edf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee00720d0 0x7f0ee013e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.689+0000 7f0edf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 0x7f0ee013f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.689+0000 7f0edf200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee01447d0 0x7f0ee01f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.689+0000 7f0edf200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee00c4920 con 0x7f0ee0143d20 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0edf200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0141e70 con 0x7f0ee01447d0 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0edf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ee0140bb0 con 0x7f0ee00720d0 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0edd400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 0x7f0ee013f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0edd400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 0x7f0ee013f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50080/0 (socket says 172.21.15.2:50080) 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0edd400640 1 -- 172.21.15.2:0/217428185 learned_addr learned my addr 172.21.15.2:0/217428185 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0edde00640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee00720d0 0x7f0ee013e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.690+0000 7f0ede800640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee01447d0 0x7f0ee01f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edd400640 1 -- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee00720d0 msgr2=0x7f0ee013e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edd400640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee00720d0 0x7f0ee013e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edd400640 1 -- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee01447d0 msgr2=0x7f0ee01f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edd400640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee01447d0 0x7f0ee01f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edd400640 1 -- 172.21.15.2:0/217428185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed0009400 con 0x7f0ee0143d20 2024-01-15T18:12:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0ede800640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee01447d0 0x7f0ee01f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edd400640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 0x7f0ee013f0b0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800cc50 tx=0x7f0ec800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec800ce60 con 0x7f0ee0143d20 2024-01-15T18:12:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec8012400 con 0x7f0ee0143d20 2024-01-15T18:12:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8012770 con 0x7f0ee0143d20 2024-01-15T18:12:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.691+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ee013f5f0 con 0x7f0ee0143d20 2024-01-15T18:12:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.692+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ee01408d0 con 0x7f0ee0143d20 2024-01-15T18:12:13.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.693+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ee0142000 con 0x7f0ee0143d20 2024-01-15T18:12:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.695+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 32) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f0ec8007070 con 0x7f0ee0143d20 2024-01-15T18:12:13.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.696+0000 7f0ed6000640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:13.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.696+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0ec8088940 con 0x7f0ee0143d20 2024-01-15T18:12:13.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.697+0000 7f0edde00640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:13.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.698+0000 7f0edde00640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0034000 tx=0x7f0ed0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:13.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.701+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ec8055030 con 0x7f0ee0143d20 2024-01-15T18:12:13.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:13.939+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f0ee015fa00 con 0x7f0eb806c160 2024-01-15T18:12:13.982 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T18:12:13.977+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:12:14.086 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:12:14.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:14.727+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 33) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f0ec8055400 con 0x7f0ee0143d20 2024-01-15T18:12:16.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:16.750+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f0ec8058f50 con 0x7f0ee0143d20 2024-01-15T18:12:17.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.027+0000 7f0ed6000640 1 -- 172.21.15.2:0/217428185 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f0ed0041070 con 0x7f0eb806c160 2024-01-15T18:12:17.031 INFO:teuthology.orchestra.run.smithi002.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:12:17.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 msgr2=0x7f0eb806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0034000 tx=0x7f0ed0008030 comp rx=0 tx=0).stop 2024-01-15T18:12:17.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 msgr2=0x7f0ee013f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 0x7f0ee013f0b0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800cc50 tx=0x7f0ec800a430 comp rx=0 tx=0).stop 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 shutdown_connections 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0eb806c160 0x7f0eb806e620 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ee01447d0 0x7f0ee01f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ee0143d20 0x7f0ee013f0b0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 --2- 172.21.15.2:0/217428185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ee00720d0 0x7f0ee013e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.033+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 >> 172.21.15.2:0/217428185 conn(0x7f0ee00b0840 msgr2=0x7f0ee00ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.034+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 shutdown_connections 2024-01-15T18:12:17.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.034+0000 7f0edf200640 1 -- 172.21.15.2:0/217428185 wait complete. 2024-01-15T18:12:17.068 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:12:17.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:12:17.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:12:17.292 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:12:17.293 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:12:17.293 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:17.350 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:17.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.653+0000 7fa3d6600640 1 Processor -- start 2024-01-15T18:12:17.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.654+0000 7fa3d6600640 1 -- start start 2024-01-15T18:12:17.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3d6600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa3d015c0e0 0x7fa3d00cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3d6600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3d015b7d0 0x7fa3d006c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3d6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015c4c0 0x7fa3d006c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3d6600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d015cb00 con 0x7fa3d015b7d0 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3d6600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3d00ccec0 con 0x7fa3d015c0e0 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3d6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d00cd1a0 con 0x7fa3d015c4c0 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3c7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3d015b7d0 0x7fa3d006c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3c7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa3d015b7d0 0x7fa3d006c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50096/0 (socket says 172.21.15.2:50096) 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3c7e00640 1 -- 172.21.15.2:0/3535583171 learned_addr learned my addr 172.21.15.2:0/3535583171 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3cfe00640 1 --2- 172.21.15.2:0/3535583171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015c4c0 0x7fa3d006c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.656+0000 7fa3cf400640 1 --2- 172.21.15.2:0/3535583171 >> v2:172.21.15.137:3300/0 conn(0x7fa3d015c0e0 0x7fa3d00cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.657+0000 7fa3c7e00640 1 -- 172.21.15.2:0/3535583171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015c4c0 msgr2=0x7fa3d006c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.657+0000 7fa3c7e00640 1 --2- 172.21.15.2:0/3535583171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015c4c0 0x7fa3d006c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.657+0000 7fa3c7e00640 1 -- 172.21.15.2:0/3535583171 >> v2:172.21.15.137:3300/0 conn(0x7fa3d015c0e0 msgr2=0x7fa3d00cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:17.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.657+0000 7fa3c7e00640 1 --2- 172.21.15.2:0/3535583171 >> v2:172.21.15.137:3300/0 conn(0x7fa3d015c0e0 0x7fa3d00cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.657+0000 7fa3c7e00640 1 -- 172.21.15.2:0/3535583171 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d00cd480 con 0x7fa3d015b7d0 2024-01-15T18:12:17.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.658+0000 7fa3c7e00640 1 --2- 172.21.15.2:0/3535583171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015b7d0 0x7fa3d006c470 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00725f0 tx=0x7fa3b802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d201d1a5a2f65e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.658+0000 7fa3cea00640 1 -- 172.21.15.2:0/3535583171 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8031860 con 0x7fa3d015b7d0 2024-01-15T18:12:17.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.658+0000 7fa3cea00640 1 -- 172.21.15.2:0/3535583171 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b80319c0 con 0x7fa3d015b7d0 2024-01-15T18:12:17.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.658+0000 7fa3cea00640 1 -- 172.21.15.2:0/3535583171 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b8031d50 con 0x7fa3d015b7d0 2024-01-15T18:12:17.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 -- 172.21.15.2:0/3535583171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015b7d0 msgr2=0x7fa3d006c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3535583171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015b7d0 0x7fa3d006c470 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fa3d00725f0 tx=0x7fa3b802b900 comp rx=0 tx=0).stop 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 -- 172.21.15.2:0/3535583171 shutdown_connections 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3535583171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015c4c0 0x7fa3d006c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3535583171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015b7d0 0x7fa3d006c470 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3535583171 >> v2:172.21.15.137:3300/0 conn(0x7fa3d015c0e0 0x7fa3d00cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 -- 172.21.15.2:0/3535583171 >> 172.21.15.2:0/3535583171 conn(0x7fa3d006da20 msgr2=0x7fa3d006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 -- 172.21.15.2:0/3535583171 shutdown_connections 2024-01-15T18:12:17.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.659+0000 7fa3d6600640 1 -- 172.21.15.2:0/3535583171 wait complete. 2024-01-15T18:12:17.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.661+0000 7fa3d6600640 1 Processor -- start 2024-01-15T18:12:17.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.661+0000 7fa3d6600640 1 -- start start 2024-01-15T18:12:17.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3d6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015b7d0 0x7fa3d0147450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3d6600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3d015c0e0 0x7fa3d0146480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3d6600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 0x7fa3d00b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3d6600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d0071850 con 0x7fa3d015c4c0 2024-01-15T18:12:17.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3d6600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d01469c0 con 0x7fa3d015c0e0 2024-01-15T18:12:17.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3d6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d00b2a30 con 0x7fa3d015b7d0 2024-01-15T18:12:17.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.662+0000 7fa3cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 0x7fa3d00b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 0x7fa3d00b3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50108/0 (socket says 172.21.15.2:50108) 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 -- 172.21.15.2:0/3777710890 learned_addr learned my addr 172.21.15.2:0/3777710890 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cf400640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015b7d0 0x7fa3d0147450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 -- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015b7d0 msgr2=0x7fa3d0147450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015b7d0 0x7fa3d0147450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3c7e00640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3d015c0e0 0x7fa3d0146480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 -- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3d015c0e0 msgr2=0x7fa3d0146480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3d015c0e0 0x7fa3d0146480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3cfe00640 1 -- 172.21.15.2:0/3777710890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b80094d0 con 0x7fa3d015c4c0 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.663+0000 7fa3c7e00640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3d015c0e0 0x7fa3d0146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:17.667 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.664+0000 7fa3cfe00640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 0x7fa3d00b3bb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8007e90 tx=0x7fa3c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.664+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8014420 con 0x7fa3d015c4c0 2024-01-15T18:12:17.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.664+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c8014580 con 0x7fa3d015c4c0 2024-01-15T18:12:17.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.664+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8014930 con 0x7fa3d015c4c0 2024-01-15T18:12:17.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.664+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d00b2ca0 con 0x7fa3d015c4c0 2024-01-15T18:12:17.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.664+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d0071470 con 0x7fa3d015c4c0 2024-01-15T18:12:17.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.666+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d0071990 con 0x7fa3d015c4c0 2024-01-15T18:12:17.669 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.667+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fa3c8014a90 con 0x7fa3d015c4c0 2024-01-15T18:12:17.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.667+0000 7fa3c7400640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa3a806c2c0 0x7fa3a806e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:17.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.667+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fa3c8089250 con 0x7fa3d015c4c0 2024-01-15T18:12:17.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.668+0000 7fa3cf400640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa3a806c2c0 0x7fa3a806e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:17.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.669+0000 7fa3cf400640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa3a806c2c0 0x7fa3a806e780 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0006fd0 tx=0x7fa3c0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:17.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.673+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3d0071990 con 0x7fa3d015c4c0 2024-01-15T18:12:17.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.862+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3d0071990 con 0x7fa3d015c4c0 2024-01-15T18:12:17.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.864+0000 7fa3c7400640 1 -- 172.21.15.2:0/3777710890 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13101 (secure 0 0 0) 0x7fa3d0071990 con 0x7fa3d015c4c0 2024-01-15T18:12:17.867 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:17.867 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":94,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:16.992434+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/3200314764":"2024-01-16T18:12:13.977758+0000","172.21.15.2:6834/3200314764":"2024-01-16T18:12:13.977758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:17.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.866+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa3a806c2c0 msgr2=0x7fa3a806e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.866+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa3a806c2c0 0x7fa3a806e780 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fa3c0006fd0 tx=0x7fa3c0008040 comp rx=0 tx=0).stop 2024-01-15T18:12:17.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.866+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 msgr2=0x7fa3d00b3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.866+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 0x7fa3d00b3bb0 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8007e90 tx=0x7fa3c800e4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 shutdown_connections 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa3a806c2c0 0x7fa3a806e780 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa3d015c4c0 0x7fa3d00b3bb0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa3d015c0e0 0x7fa3d0146480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 --2- 172.21.15.2:0/3777710890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa3d015b7d0 0x7fa3d0147450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 >> 172.21.15.2:0/3777710890 conn(0x7fa3d006da20 msgr2=0x7fa3d006c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 shutdown_connections 2024-01-15T18:12:17.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:17.867+0000 7fa3d6600640 1 -- 172.21.15.2:0/3777710890 wait complete. 2024-01-15T18:12:17.886 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:12:17.886 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:12:18.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.105+0000 7f1760200640 1 Processor -- start 2024-01-15T18:12:18.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.105+0000 7f1760200640 1 -- start start 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.107+0000 7f1760200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1758146b80 0x7f175806c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.107+0000 7f1760200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f175806cba0 0x7f17580cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.107+0000 7f1760200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17580cdf30 0x7f175806e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.107+0000 7f1760200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175806c240 con 0x7f17580cdf30 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.107+0000 7f1760200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1758146f60 con 0x7f1758146b80 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.107+0000 7f1760200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1758070da0 con 0x7f175806cba0 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17580cdf30 0x7f175806e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17580cdf30 0x7f175806e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50122/0 (socket says 172.21.15.2:50122) 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 -- 172.21.15.2:0/1895763018 learned_addr learned my addr 172.21.15.2:0/1895763018 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175d800640 1 --2- 172.21.15.2:0/1895763018 >> v2:172.21.15.137:3300/0 conn(0x7f1758146b80 0x7f175806c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175ce00640 1 --2- 172.21.15.2:0/1895763018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f175806cba0 0x7f17580cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 -- 172.21.15.2:0/1895763018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f175806cba0 msgr2=0x7f17580cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 --2- 172.21.15.2:0/1895763018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f175806cba0 0x7f17580cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 -- 172.21.15.2:0/1895763018 >> v2:172.21.15.137:3300/0 conn(0x7f1758146b80 msgr2=0x7f175806c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 --2- 172.21.15.2:0/1895763018 >> v2:172.21.15.137:3300/0 conn(0x7f1758146b80 0x7f175806c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.108+0000 7f175e200640 1 -- 172.21.15.2:0/1895763018 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1758070ac0 con 0x7f17580cdf30 2024-01-15T18:12:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.110+0000 7f175e200640 1 --2- 172.21.15.2:0/1895763018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17580cdf30 0x7f175806e830 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f17580b37b0 tx=0x7f175002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47a29be36ac7ca57 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.110+0000 7f1757e00640 1 -- 172.21.15.2:0/1895763018 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750031750 con 0x7f17580cdf30 2024-01-15T18:12:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.110+0000 7f1757e00640 1 -- 172.21.15.2:0/1895763018 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17500318b0 con 0x7f17580cdf30 2024-01-15T18:12:18.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.110+0000 7f1757e00640 1 -- 172.21.15.2:0/1895763018 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750031c40 con 0x7f17580cdf30 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 -- 172.21.15.2:0/1895763018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17580cdf30 msgr2=0x7f175806e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 --2- 172.21.15.2:0/1895763018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17580cdf30 0x7f175806e830 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f17580b37b0 tx=0x7f175002b7f0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 -- 172.21.15.2:0/1895763018 shutdown_connections 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 --2- 172.21.15.2:0/1895763018 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17580cdf30 0x7f175806e830 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 --2- 172.21.15.2:0/1895763018 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f175806cba0 0x7f17580cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 --2- 172.21.15.2:0/1895763018 >> v2:172.21.15.137:3300/0 conn(0x7f1758146b80 0x7f175806c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 -- 172.21.15.2:0/1895763018 >> 172.21.15.2:0/1895763018 conn(0x7f17580b2c10 msgr2=0x7f17580b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 -- 172.21.15.2:0/1895763018 shutdown_connections 2024-01-15T18:12:18.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.111+0000 7f1760200640 1 -- 172.21.15.2:0/1895763018 wait complete. 2024-01-15T18:12:18.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.113+0000 7f1760200640 1 Processor -- start 2024-01-15T18:12:18.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.113+0000 7f1760200640 1 -- start start 2024-01-15T18:12:18.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f1760200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17580cdf30 0x7f17581f9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f1760200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1758146b80 0x7f175813e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f1760200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 0x7f17580cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f1760200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175813d870 con 0x7f175815cba0 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f1760200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f175813f6d0 con 0x7f1758146b80 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f1760200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175813eac0 con 0x7f17580cdf30 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f175e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 0x7f17580cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f175e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 0x7f17580cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50138/0 (socket says 172.21.15.2:50138) 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f175e200640 1 -- 172.21.15.2:0/3242932102 learned_addr learned my addr 172.21.15.2:0/3242932102 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.114+0000 7f175d800640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17580cdf30 0x7f17581f9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175ce00640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1758146b80 0x7f175813e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175e200640 1 -- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17580cdf30 msgr2=0x7f17581f9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175e200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17580cdf30 0x7f17581f9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175e200640 1 -- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1758146b80 msgr2=0x7f175813e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175e200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1758146b80 0x7f175813e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175e200640 1 -- 172.21.15.2:0/3242932102 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17500094d0 con 0x7f175815cba0 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175ce00640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1758146b80 0x7f175813e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:18.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f175e200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 0x7f17580cfae0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f17500090b0 tx=0x7f17500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1750033c80 con 0x7f175815cba0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1750033e00 con 0x7f175815cba0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175003c640 con 0x7f175815cba0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.115+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f175813eda0 con 0x7f175815cba0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.116+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f175813f360 con 0x7f175815cba0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.117+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1758140130 con 0x7f175815cba0 2024-01-15T18:12:18.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.118+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f175003a040 con 0x7f175815cba0 2024-01-15T18:12:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.118+0000 7f1755600640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f173006c1f0 0x7f173006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.119+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f17500aa280 con 0x7f175815cba0 2024-01-15T18:12:18.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.119+0000 7f175d800640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f173006c1f0 0x7f173006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.120+0000 7f175d800640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f173006c1f0 0x7f173006e6b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f17480099c0 tx=0x7f1748003990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.123+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1758140130 con 0x7f175815cba0 2024-01-15T18:12:18.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.321+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1758140130 con 0x7f175815cba0 2024-01-15T18:12:18.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.322+0000 7f1755600640 1 -- 172.21.15.2:0/3242932102 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1758140130 con 0x7f175815cba0 2024-01-15T18:12:18.325 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f173006c1f0 msgr2=0x7f173006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f173006c1f0 0x7f173006e6b0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f17480099c0 tx=0x7f1748003990 comp rx=0 tx=0).stop 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 msgr2=0x7f17580cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 0x7f17580cfae0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f17500090b0 tx=0x7f17500046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 shutdown_connections 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f173006c1f0 0x7f173006e6b0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f175815cba0 0x7f17580cfae0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1758146b80 0x7f175813e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 --2- 172.21.15.2:0/3242932102 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17580cdf30 0x7f17581f9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 >> 172.21.15.2:0/3242932102 conn(0x7f17580b2c10 msgr2=0x7f17580d0d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 shutdown_connections 2024-01-15T18:12:18.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.325+0000 7f1760200640 1 -- 172.21.15.2:0/3242932102 wait complete. 2024-01-15T18:12:18.345 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:12:18.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.540+0000 7effe5800640 1 Processor -- start 2024-01-15T18:12:18.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.540+0000 7effe5800640 1 -- start start 2024-01-15T18:12:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.542+0000 7effe5800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7effe006cfc0 0x7effe00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.542+0000 7effe5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe0070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.542+0000 7effe5800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe0143dc0 con 0x7effe01439e0 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.542+0000 7effe5800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effe00cc360 con 0x7effe006cfc0 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.542+0000 7effe5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe015b620 con 0x7effe00cbed0 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7effe006cfc0 0x7effe00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7effe006cfc0 0x7effe00cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50146/0 (socket says 172.21.15.2:50146) 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 -- 172.21.15.2:0/701162640 learned_addr learned my addr 172.21.15.2:0/701162640 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:18.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effdf400640 1 --2- 172.21.15.2:0/701162640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe0070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 -- 172.21.15.2:0/701162640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 msgr2=0x7effe0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 --2- 172.21.15.2:0/701162640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe0070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 -- 172.21.15.2:0/701162640 >> v1:172.21.15.137:6789/0 conn(0x7effe01439e0 legacy=0x7effe006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.543+0000 7effde000640 1 -- 172.21.15.2:0/701162640 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe00cd9f0 con 0x7effe006cfc0 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.544+0000 7effde000640 1 --2- 172.21.15.2:0/701162640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe00cb990 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7effe00cc4a0 tx=0x7effc802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7a63365c2903e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.544+0000 7effdd600640 1 -- 172.21.15.2:0/701162640 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8030840 con 0x7effe006cfc0 2024-01-15T18:12:18.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effdd600640 1 -- 172.21.15.2:0/701162640 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc80309a0 con 0x7effe006cfc0 2024-01-15T18:12:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effdd600640 1 -- 172.21.15.2:0/701162640 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc8030d30 con 0x7effe006cfc0 2024-01-15T18:12:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 -- 172.21.15.2:0/701162640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 msgr2=0x7effe00cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 --2- 172.21.15.2:0/701162640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe00cb990 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7effe00cc4a0 tx=0x7effc802b980 comp rx=0 tx=0).stop 2024-01-15T18:12:18.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 -- 172.21.15.2:0/701162640 shutdown_connections 2024-01-15T18:12:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 --2- 172.21.15.2:0/701162640 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe0070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 --2- 172.21.15.2:0/701162640 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe00cb990 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 -- 172.21.15.2:0/701162640 >> 172.21.15.2:0/701162640 conn(0x7effe00b46d0 msgr2=0x7effe00b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 -- 172.21.15.2:0/701162640 shutdown_connections 2024-01-15T18:12:18.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.545+0000 7effe5800640 1 -- 172.21.15.2:0/701162640 wait complete. 2024-01-15T18:12:18.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.547+0000 7effe5800640 1 Processor -- start 2024-01-15T18:12:18.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.547+0000 7effe5800640 1 -- start start 2024-01-15T18:12:18.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.548+0000 7effe5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe0159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.548+0000 7effe5800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe013f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.548+0000 7effe5800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effe01439e0 0x7effe01e41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.548+0000 7effe5800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe015a210 con 0x7effe006cfc0 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.548+0000 7effe5800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01400d0 con 0x7effe01439e0 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.548+0000 7effe5800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe01403b0 con 0x7effe00cbed0 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe0159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe0159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50148/0 (socket says 172.21.15.2:50148) 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 -- 172.21.15.2:0/3667627616 learned_addr learned my addr 172.21.15.2:0/3667627616 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdf400640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effe01439e0 0x7effe01e41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effde000640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe013f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 -- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 msgr2=0x7effe013f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe013f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 -- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effe01439e0 msgr2=0x7effe01e41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effe01439e0 0x7effe01e41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 -- 172.21.15.2:0/3667627616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc80094d0 con 0x7effe006cfc0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.549+0000 7effdea00640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe0159cd0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7effd000c400 tx=0x7effd000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.550+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00052d0 con 0x7effe006cfc0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.550+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd0005430 con 0x7effe006cfc0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.550+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0003830 con 0x7effe006cfc0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.550+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0140690 con 0x7effe006cfc0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.550+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe0140c50 con 0x7effe006cfc0 2024-01-15T18:12:18.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.551+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe00686f0 con 0x7effe006cfc0 2024-01-15T18:12:18.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.552+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7effd0003990 con 0x7effe006cfc0 2024-01-15T18:12:18.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.553+0000 7effd6a00640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effb806c1f0 0x7effb806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:18.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.553+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7effd0087c60 con 0x7effe006cfc0 2024-01-15T18:12:18.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.553+0000 7effde000640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effb806c1f0 0x7effb806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:18.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.554+0000 7effde000640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effb806c1f0 0x7effb806e6b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7effc8036000 tx=0x7effc8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:18.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.557+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effd0054520 con 0x7effe006cfc0 2024-01-15T18:12:18.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.766+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7effe00b2db0 con 0x7effe006cfc0 2024-01-15T18:12:18.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.767+0000 7effd6a00640 1 -- 172.21.15.2:0/3667627616 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v94) v1 ==== 80+0+120 (secure 0 0 0) 0x7effd0017080 con 0x7effe006cfc0 2024-01-15T18:12:18.771 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6835/3200314764 2024-01-16T18:12:13.977758+0000 2024-01-15T18:12:18.771 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/3200314764 2024-01-16T18:12:13.977758+0000 2024-01-15T18:12:18.771 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effb806c1f0 msgr2=0x7effb806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effb806c1f0 0x7effb806e6b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7effc8036000 tx=0x7effc8008030 comp rx=0 tx=0).stop 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 msgr2=0x7effe0159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe0159cd0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7effd000c400 tx=0x7effd000c8c0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 shutdown_connections 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7effb806c1f0 0x7effb806e6b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7effe01439e0 0x7effe01e41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7effe00cbed0 0x7effe013f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 --2- 172.21.15.2:0/3667627616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7effe006cfc0 0x7effe0159cd0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.770+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 >> 172.21.15.2:0/3667627616 conn(0x7effe00b46d0 msgr2=0x7effe006c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.771+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 shutdown_connections 2024-01-15T18:12:18.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:18.771+0000 7effe5800640 1 -- 172.21.15.2:0/3667627616 wait complete. 2024-01-15T18:12:18.793 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:12:18.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:18.793 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:18.884 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[69158]: fuse finished with error 0 and tester_r 0 2024-01-15T18:12:18.889 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:12:18.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:18.889 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:12:19.101 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:12:19.101 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:19.158 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.384+0000 7f64bf200640 1 Processor -- start 2024-01-15T18:12:19.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.384+0000 7f64bf200640 1 -- start start 2024-01-15T18:12:19.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.386+0000 7f64bf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 0x7f64c015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.386+0000 7f64bf200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f64c006cad0 0x7f64c006cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.386+0000 7f64bf200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64c00716d0 con 0x7f64c0141880 2024-01-15T18:12:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.386+0000 7f64bf200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64c0071f20 con 0x7f64c006cad0 2024-01-15T18:12:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.386+0000 7f64bf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c006c6b0 con 0x7f64c015ec60 2024-01-15T18:12:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.387+0000 7f64bde00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f64c0141880 0x7f64c0070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35300/0 (socket says 172.21.15.2:35300) 2024-01-15T18:12:19.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.387+0000 7f64bde00640 1 -- 172.21.15.2:0/101586616 learned_addr learned my addr 172.21.15.2:0/101586616 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:19.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.387+0000 7f64be800640 1 --2- 172.21.15.2:0/101586616 >> v2:172.21.15.137:3300/0 conn(0x7f64c006cad0 0x7f64c006cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.387+0000 7f64bd400640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 0x7f64c015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.387+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247171639 0 0) 0x7f64c00716d0 con 0x7f64c0141880 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.387+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a0003520 con 0x7f64c0141880 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1198087444 0 0) 0x7f64a0003520 con 0x7f64c0141880 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 msgr2=0x7f64c015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 0x7f64c015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 >> v2:172.21.15.137:3300/0 conn(0x7f64c006cad0 msgr2=0x7f64c006cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 --2- 172.21.15.2:0/101586616 >> v2:172.21.15.137:3300/0 conn(0x7f64c006cad0 0x7f64c006cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c0071960 con 0x7f64c0141880 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f64b0001670 con 0x7f64c0141880 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 >> v1:172.21.15.2:6789/0 conn(0x7f64c0141880 legacy=0x7f64c0070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 0x7f64a0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64a0003f40 0x7f64a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c0071f20 con 0x7f64a0004d00 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c006c6b0 con 0x7f64a0003f40 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64a000ba30 con 0x7f64c015ec60 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.388+0000 7f64be800640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64bde00640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 0x7f64a0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64be800640 1 -- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 msgr2=0x7f64a0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64bd400640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64a0003f40 0x7f64a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64be800640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015ec60 0x7f64a0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64be800640 1 -- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64a0003f40 msgr2=0x7f64a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64be800640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64a0003f40 0x7f64a00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.389+0000 7f64be800640 1 -- 172.21.15.2:0/101586616 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a0003520 con 0x7f64a0004d00 2024-01-15T18:12:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.390+0000 7f64be800640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64a0005180 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009fc0 tx=0x7f64a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321d960c20563a2c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.390+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b802cc60 con 0x7f64a0004d00 2024-01-15T18:12:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.390+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b800f040 con 0x7f64a0004d00 2024-01-15T18:12:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.390+0000 7f64bca00640 1 -- 172.21.15.2:0/101586616 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b8034440 con 0x7f64a0004d00 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64bf200640 1 -- 172.21.15.2:0/101586616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 msgr2=0x7f64a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64bf200640 1 --2- 172.21.15.2:0/101586616 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64a0005180 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009fc0 tx=0x7f64a00057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64be800640 1 -- 172.21.15.2:0/101586616 reap_dead start 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64bf200640 1 -- 172.21.15.2:0/101586616 shutdown_connections 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64bf200640 1 -- 172.21.15.2:0/101586616 >> 172.21.15.2:0/101586616 conn(0x7f64c00b0840 msgr2=0x7f64c00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64bf200640 1 -- 172.21.15.2:0/101586616 shutdown_connections 2024-01-15T18:12:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.391+0000 7f64bf200640 1 -- 172.21.15.2:0/101586616 wait complete. 2024-01-15T18:12:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.393+0000 7f64bf200640 1 Processor -- start 2024-01-15T18:12:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.393+0000 7f64bf200640 1 -- start start 2024-01-15T18:12:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.394+0000 7f64bf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64c01f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.394+0000 7f64bf200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64c00b35a0 0x7f64c01364b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.394+0000 7f64bf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015b580 0x7f64c015ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.394+0000 7f64bf200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c015bf40 con 0x7f64a0004d00 2024-01-15T18:12:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.394+0000 7f64bf200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01369f0 con 0x7f64c00b35a0 2024-01-15T18:12:19.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.394+0000 7f64bf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64c01f3580 con 0x7f64c015b580 2024-01-15T18:12:19.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64c01f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64c01f2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50170/0 (socket says 172.21.15.2:50170) 2024-01-15T18:12:19.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 -- 172.21.15.2:0/2533496378 learned_addr learned my addr 172.21.15.2:0/2533496378 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 -- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015b580 msgr2=0x7f64c015ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015b580 0x7f64c015ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 -- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64c00b35a0 msgr2=0x7f64c01364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64c00b35a0 0x7f64c01364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 -- 172.21.15.2:0/2533496378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c01f3860 con 0x7f64a0004d00 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.395+0000 7f64bde00640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64c01f2f30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f64b0001050 tx=0x7f64b000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.396+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b000c850 con 0x7f64a0004d00 2024-01-15T18:12:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.396+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b000c9b0 con 0x7f64a0004d00 2024-01-15T18:12:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.396+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c01f3b40 con 0x7f64a0004d00 2024-01-15T18:12:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.396+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64b000cd60 con 0x7f64a0004d00 2024-01-15T18:12:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.396+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c00b3a20 con 0x7f64a0004d00 2024-01-15T18:12:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.397+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c0074770 con 0x7f64a0004d00 2024-01-15T18:12:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.398+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f64b001d460 con 0x7f64a0004d00 2024-01-15T18:12:19.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.399+0000 7f64b6000640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f649806c220 0x7f649806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.399+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f64b0088990 con 0x7f64a0004d00 2024-01-15T18:12:19.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.399+0000 7f64bd400640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f649806c220 0x7f649806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.400+0000 7f64bd400640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f649806c220 0x7f649806e6e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f64c00cc360 tx=0x7f64a8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:19.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.403+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f64b0054f60 con 0x7f64a0004d00 2024-01-15T18:12:19.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.600+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64c01f43a0 con 0x7f64a0004d00 2024-01-15T18:12:19.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.602+0000 7f64b6000640 1 -- 172.21.15.2:0/2533496378 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13101 (secure 0 0 0) 0x7f64b0058580 con 0x7f64a0004d00 2024-01-15T18:12:19.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:19.605 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":94,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:16.992434+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/3200314764":"2024-01-16T18:12:13.977758+0000","172.21.15.2:6834/3200314764":"2024-01-16T18:12:13.977758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.604+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f649806c220 msgr2=0x7f649806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.604+0000 7f64bf200640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f649806c220 0x7f649806e6e0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f64c00cc360 tx=0x7f64a8008040 comp rx=0 tx=0).stop 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.604+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 msgr2=0x7f64c01f2f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.604+0000 7f64bf200640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64c01f2f30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f64b0001050 tx=0x7f64b000ec60 comp rx=0 tx=0).stop 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 shutdown_connections 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f649806c220 0x7f649806e6e0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64c015b580 0x7f64c015ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64c00b35a0 0x7f64c01364b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 --2- 172.21.15.2:0/2533496378 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f64a0004d00 0x7f64c01f2f30 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 >> 172.21.15.2:0/2533496378 conn(0x7f64c00b0840 msgr2=0x7f64c0140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:19.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 shutdown_connections 2024-01-15T18:12:19.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.605+0000 7f64bf200640 1 -- 172.21.15.2:0/2533496378 wait complete. 2024-01-15T18:12:19.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:12:19.628 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:12:19.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.852+0000 7f9469000640 1 Processor -- start 2024-01-15T18:12:19.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.852+0000 7f9469000640 1 -- start start 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.854+0000 7f9469000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9464144b00 0x7f946415b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.854+0000 7f9469000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f946406abc0 0x7f94640ae460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.854+0000 7f9469000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946415bc50 con 0x7f946406d1f0 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.854+0000 7f9469000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9464144ee0 con 0x7f946406abc0 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.854+0000 7f9469000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94640adfd0 con 0x7f9464144b00 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.854+0000 7f9463e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f946406abc0 0x7f94640ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9462a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f946406d1f0 0x7f946406a4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35314/0 (socket says 172.21.15.2:35314) 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9462a00640 1 -- 172.21.15.2:0/2284986628 learned_addr learned my addr 172.21.15.2:0/2284986628 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9463400640 1 --2- 172.21.15.2:0/2284986628 >> v2:172.21.15.137:3300/0 conn(0x7f9464144b00 0x7f946415b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9463e00640 1 -- 172.21.15.2:0/2284986628 >> v2:172.21.15.137:3300/0 conn(0x7f9464144b00 msgr2=0x7f946415b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9463e00640 1 --2- 172.21.15.2:0/2284986628 >> v2:172.21.15.137:3300/0 conn(0x7f9464144b00 0x7f946415b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9463e00640 1 -- 172.21.15.2:0/2284986628 >> v1:172.21.15.2:6789/0 conn(0x7f946406d1f0 legacy=0x7f946406a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.855+0000 7f9463e00640 1 -- 172.21.15.2:0/2284986628 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94640ad3c0 con 0x7f946406abc0 2024-01-15T18:12:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.856+0000 7f9463e00640 1 --2- 172.21.15.2:0/2284986628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f946406abc0 0x7f94640ae460 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f946406b590 tx=0x7f945c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc28483580d6095 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.856+0000 7f9462000640 1 -- 172.21.15.2:0/2284986628 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c02cbe0 con 0x7f946406abc0 2024-01-15T18:12:19.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9462000640 1 -- 172.21.15.2:0/2284986628 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c02cd40 con 0x7f946406abc0 2024-01-15T18:12:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9462000640 1 -- 172.21.15.2:0/2284986628 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c034660 con 0x7f946406abc0 2024-01-15T18:12:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 -- 172.21.15.2:0/2284986628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f946406abc0 msgr2=0x7f94640ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 --2- 172.21.15.2:0/2284986628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f946406abc0 0x7f94640ae460 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f946406b590 tx=0x7f945c02be30 comp rx=0 tx=0).stop 2024-01-15T18:12:19.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 -- 172.21.15.2:0/2284986628 shutdown_connections 2024-01-15T18:12:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 --2- 172.21.15.2:0/2284986628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f946406abc0 0x7f94640ae460 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 --2- 172.21.15.2:0/2284986628 >> v2:172.21.15.137:3300/0 conn(0x7f9464144b00 0x7f946415b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 -- 172.21.15.2:0/2284986628 >> 172.21.15.2:0/2284986628 conn(0x7f946406b5c0 msgr2=0x7f946406b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 -- 172.21.15.2:0/2284986628 shutdown_connections 2024-01-15T18:12:19.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.857+0000 7f9469000640 1 -- 172.21.15.2:0/2284986628 wait complete. 2024-01-15T18:12:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.859+0000 7f9469000640 1 Processor -- start 2024-01-15T18:12:19.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.859+0000 7f9469000640 1 -- start start 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.860+0000 7f9469000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f946406abc0 0x7f94641412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.860+0000 7f9469000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f946406d1f0 0x7f9464149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.860+0000 7f9469000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 0x7f94641efe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.860+0000 7f9469000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9464141830 con 0x7f9464144b00 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.860+0000 7f9469000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9464142a10 con 0x7f946406d1f0 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.860+0000 7f9469000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9464142cf0 con 0x7f946406abc0 2024-01-15T18:12:19.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 0x7f94641efe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 0x7f94641efe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50180/0 (socket says 172.21.15.2:50180) 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 -- 172.21.15.2:0/994002291 learned_addr learned my addr 172.21.15.2:0/994002291 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9462a00640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f946406d1f0 0x7f9464149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463400640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f946406abc0 0x7f94641412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 -- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f946406abc0 msgr2=0x7f94641412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f946406abc0 0x7f94641412f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 -- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f946406d1f0 msgr2=0x7f9464149340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f946406d1f0 0x7f9464149340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 -- 172.21.15.2:0/994002291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c009a70 con 0x7f9464144b00 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9462a00640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f946406d1f0 0x7f9464149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:19.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.861+0000 7f9463e00640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 0x7f94641efe00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f945c009e80 tx=0x7f945c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.862+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c040650 con 0x7f9464144b00 2024-01-15T18:12:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.862+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c0407d0 con 0x7f9464144b00 2024-01-15T18:12:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.862+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9464142fd0 con 0x7f9464144b00 2024-01-15T18:12:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.862+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c040b80 con 0x7f9464144b00 2024-01-15T18:12:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.862+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9464143360 con 0x7f9464144b00 2024-01-15T18:12:19.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.863+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9464143890 con 0x7f9464144b00 2024-01-15T18:12:19.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.864+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f945c040ce0 con 0x7f9464144b00 2024-01-15T18:12:19.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.865+0000 7f945b400640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943c06c1f0 0x7f943c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:19.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.865+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f945c0ab690 con 0x7f9464144b00 2024-01-15T18:12:19.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.866+0000 7f9463400640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943c06c1f0 0x7f943c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:19.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.867+0000 7f9463400640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943c06c1f0 0x7f943c06e6b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9454006fd0 tx=0x7f9454008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:19.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:19.870+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f945c077e10 con 0x7f9464144b00 2024-01-15T18:12:20.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.061+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f94640afe30 con 0x7f9464144b00 2024-01-15T18:12:20.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.062+0000 7f945b400640 1 -- 172.21.15.2:0/994002291 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f945c07b430 con 0x7f9464144b00 2024-01-15T18:12:20.065 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:12:20.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943c06c1f0 msgr2=0x7f943c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943c06c1f0 0x7f943c06e6b0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9454006fd0 tx=0x7f9454008040 comp rx=0 tx=0).stop 2024-01-15T18:12:20.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 msgr2=0x7f94641efe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 0x7f94641efe00 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f945c009e80 tx=0x7f945c004a40 comp rx=0 tx=0).stop 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 shutdown_connections 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943c06c1f0 0x7f943c06e6b0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9464144b00 0x7f94641efe00 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f946406d1f0 0x7f9464149340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.065+0000 7f9469000640 1 --2- 172.21.15.2:0/994002291 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f946406abc0 0x7f94641412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.066+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 >> 172.21.15.2:0/994002291 conn(0x7f946406b5c0 msgr2=0x7f946415a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.066+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 shutdown_connections 2024-01-15T18:12:20.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.066+0000 7f9469000640 1 -- 172.21.15.2:0/994002291 wait complete. 2024-01-15T18:12:20.087 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:12:20.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.308+0000 7f9457200640 1 Processor -- start 2024-01-15T18:12:20.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.309+0000 7f9457200640 1 -- start start 2024-01-15T18:12:20.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.310+0000 7f9457200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 0x7f94580cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.310+0000 7f9457200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f94580d0ed0 0x7f94580d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.310+0000 7f9457200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580b30e0 con 0x7f94580d2de0 2024-01-15T18:12:20.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.310+0000 7f9457200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94580ce180 con 0x7f94580d0ed0 2024-01-15T18:12:20.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.310+0000 7f9457200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94580d1ca0 con 0x7f945815d200 2024-01-15T18:12:20.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.311+0000 7f9455400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f94580d2de0 0x7f94580d3240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35320/0 (socket says 172.21.15.2:35320) 2024-01-15T18:12:20.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.311+0000 7f9455400640 1 -- 172.21.15.2:0/912083491 learned_addr learned my addr 172.21.15.2:0/912083491 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:20.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.311+0000 7f9456800640 1 --2- 172.21.15.2:0/912083491 >> v2:172.21.15.137:3300/0 conn(0x7f94580d0ed0 0x7f94580d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.311+0000 7f9455e00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 0x7f94580cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.311+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058513821 0 0) 0x7f94580b30e0 con 0x7f94580d2de0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.311+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9438003520 con 0x7f94580d2de0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1461673304 0 0) 0x7f9438003520 con 0x7f94580d2de0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 msgr2=0x7f94580cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 0x7f94580cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 >> v2:172.21.15.137:3300/0 conn(0x7f94580d0ed0 msgr2=0x7f94580d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 --2- 172.21.15.2:0/912083491 >> v2:172.21.15.137:3300/0 conn(0x7f94580d0ed0 0x7f94580d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580d1f80 con 0x7f94580d2de0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9440001670 con 0x7f94580d2de0 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 >> v1:172.21.15.2:6789/0 conn(0x7f94580d2de0 legacy=0x7f94580d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f9438003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94380047c0 0x7f9438004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 0x7f9438005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580ce180 con 0x7f9438003970 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580d1ca0 con 0x7f94380047c0 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943800b980 con 0x7f945815d200 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9455e00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f9438003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9455400640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94380047c0 0x7f9438004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9455e00640 1 -- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 msgr2=0x7f9438005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.312+0000 7f9455e00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815d200 0x7f9438005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.313+0000 7f9455e00640 1 -- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94380047c0 msgr2=0x7f9438004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.313+0000 7f9455e00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f94380047c0 0x7f9438004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.313+0000 7f9455e00640 1 -- 172.21.15.2:0/912083491 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9438003520 con 0x7f9438003970 2024-01-15T18:12:20.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.314+0000 7f9455e00640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f9438003d90 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f9448009ac0 tx=0x7f94380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da2414a9852adeb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:20.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.314+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944802c710 con 0x7f9438003970 2024-01-15T18:12:20.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.314+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f944802c870 con 0x7f9438003970 2024-01-15T18:12:20.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.314+0000 7f9454a00640 1 -- 172.21.15.2:0/912083491 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f944802cc00 con 0x7f9438003970 2024-01-15T18:12:20.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9457200640 1 -- 172.21.15.2:0/912083491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 msgr2=0x7f9438003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9457200640 1 --2- 172.21.15.2:0/912083491 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f9438003d90 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f9448009ac0 tx=0x7f94380057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9456800640 1 -- 172.21.15.2:0/912083491 reap_dead start 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9457200640 1 -- 172.21.15.2:0/912083491 shutdown_connections 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9457200640 1 -- 172.21.15.2:0/912083491 >> 172.21.15.2:0/912083491 conn(0x7f945808caf0 msgr2=0x7f945809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9457200640 1 -- 172.21.15.2:0/912083491 shutdown_connections 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.315+0000 7f9457200640 1 -- 172.21.15.2:0/912083491 wait complete. 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.317+0000 7f9457200640 1 Processor -- start 2024-01-15T18:12:20.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.317+0000 7f9457200640 1 -- start start 2024-01-15T18:12:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9457200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f94580b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9457200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f945815b490 0x7f94580ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9457200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815fae0 0x7f945815ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9457200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580cd0d0 con 0x7f9438003970 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9457200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94580cacf0 con 0x7f945815b490 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9457200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945815b910 con 0x7f945815fae0 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f94580b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f94580b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50192/0 (socket says 172.21.15.2:50192) 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 -- 172.21.15.2:0/1713280820 learned_addr learned my addr 172.21.15.2:0/1713280820 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 -- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815fae0 msgr2=0x7f945815ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815fae0 0x7f945815ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 -- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f945815b490 msgr2=0x7f94580ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f945815b490 0x7f94580ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.318+0000 7f9455e00640 1 -- 172.21.15.2:0/1713280820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580b1c60 con 0x7f9438003970 2024-01-15T18:12:20.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.319+0000 7f9455e00640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f94580b15d0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f9448000c00 tx=0x7f9448004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:20.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.319+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448003810 con 0x7f9438003970 2024-01-15T18:12:20.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.319+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9448003990 con 0x7f9438003970 2024-01-15T18:12:20.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.319+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9448003d40 con 0x7f9438003970 2024-01-15T18:12:20.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.319+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94580b1f40 con 0x7f9438003970 2024-01-15T18:12:20.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.319+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94580b2500 con 0x7f9438003970 2024-01-15T18:12:20.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.321+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9458074770 con 0x7f9438003970 2024-01-15T18:12:20.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.322+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f9448040450 con 0x7f9438003970 2024-01-15T18:12:20.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.323+0000 7f944e000640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943006c1f0 0x7f943006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:20.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.323+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f94480aa610 con 0x7f9438003970 2024-01-15T18:12:20.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.323+0000 7f9455400640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943006c1f0 0x7f943006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:20.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.324+0000 7f9455400640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943006c1f0 0x7f943006e6b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f94400017a0 tx=0x7f9440009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:20.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.327+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9448076e30 con 0x7f9438003970 2024-01-15T18:12:20.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.535+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f94580678e0 con 0x7f9438003970 2024-01-15T18:12:20.539 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.536+0000 7f944e000640 1 -- 172.21.15.2:0/1713280820 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v94) v1 ==== 80+0+120 (secure 0 0 0) 0x7f944807a450 con 0x7f9438003970 2024-01-15T18:12:20.539 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6835/3200314764 2024-01-16T18:12:13.977758+0000 2024-01-15T18:12:20.539 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.2:6834/3200314764 2024-01-16T18:12:13.977758+0000 2024-01-15T18:12:20.540 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943006c1f0 msgr2=0x7f943006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943006c1f0 0x7f943006e6b0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f94400017a0 tx=0x7f9440009040 comp rx=0 tx=0).stop 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 msgr2=0x7f94580b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f94580b15d0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f9448000c00 tx=0x7f9448004730 comp rx=0 tx=0).stop 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 shutdown_connections 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f943006c1f0 0x7f943006e6b0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f945815fae0 0x7f945815ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f945815b490 0x7f94580ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 --2- 172.21.15.2:0/1713280820 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9438003970 0x7f94580b15d0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 >> 172.21.15.2:0/1713280820 conn(0x7f945808caf0 msgr2=0x7f94580b4b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.539+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 shutdown_connections 2024-01-15T18:12:20.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:20.540+0000 7f9457200640 1 -- 172.21.15.2:0/1713280820 wait complete. 2024-01-15T18:12:20.562 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:12:20.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:20.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:20.658 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[45936]: fuse finished with error 0 and tester_r 0 2024-01-15T18:12:20.662 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:12:20.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:20.662 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:12:20.837 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:12:20.838 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:20.893 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:12:20.893 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:12:20.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:20.893 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:12:20.950 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:12:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:12:20.951 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:12:20.951 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:21.006 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:12:21.007 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:12:21.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:21.007 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:12:21.059 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:12:21.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:12:21.060 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:12:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.283+0000 7fc238a00640 1 Processor -- start 2024-01-15T18:12:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.283+0000 7fc238a00640 1 -- start start 2024-01-15T18:12:21.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.285+0000 7fc238a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc23406d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.285+0000 7fc238a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2340700d0 0x7fc234070540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.285+0000 7fc238a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2340b1c70 con 0x7fc23406da60 2024-01-15T18:12:21.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.285+0000 7fc238a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2340b28f0 con 0x7fc2340700d0 2024-01-15T18:12:21.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.285+0000 7fc238a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc234070a80 con 0x7fc23414af80 2024-01-15T18:12:21.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.286+0000 7fc233400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc23406d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.286+0000 7fc233e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2340700d0 0x7fc234070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.286+0000 7fc233e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2340700d0 0x7fc234070540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:48024/0 (socket says 172.21.15.2:48024) 2024-01-15T18:12:21.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.286+0000 7fc233e00640 1 -- 172.21.15.2:0/2740249405 learned_addr learned my addr 172.21.15.2:0/2740249405 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:21.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.287+0000 7fc233e00640 1 -- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 msgr2=0x7fc23406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:21.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.287+0000 7fc233e00640 1 --2- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc23406d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.287+0000 7fc233e00640 1 -- 172.21.15.2:0/2740249405 >> v1:172.21.15.137:6789/0 conn(0x7fc23406da60 legacy=0x7fc234161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.287+0000 7fc233e00640 1 -- 172.21.15.2:0/2740249405 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2340b1f50 con 0x7fc2340700d0 2024-01-15T18:12:21.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.288+0000 7fc233e00640 1 --2- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc234070540 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009e80 tx=0x7fc22c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5d458e998e3e749 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.289+0000 7fc232a00640 1 -- 172.21.15.2:0/2740249405 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c02ca50 con 0x7fc2340700d0 2024-01-15T18:12:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.289+0000 7fc232a00640 1 -- 172.21.15.2:0/2740249405 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c02cbb0 con 0x7fc2340700d0 2024-01-15T18:12:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.289+0000 7fc232a00640 1 -- 172.21.15.2:0/2740249405 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c0343f0 con 0x7fc2340700d0 2024-01-15T18:12:21.292 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.289+0000 7fc238a00640 1 -- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 msgr2=0x7fc234070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.289+0000 7fc238a00640 1 --2- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc234070540 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc22c009e80 tx=0x7fc22c02beb0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.290+0000 7fc238a00640 1 -- 172.21.15.2:0/2740249405 shutdown_connections 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.290+0000 7fc238a00640 1 --2- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc234070540 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.290+0000 7fc238a00640 1 --2- 172.21.15.2:0/2740249405 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc23406d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.290+0000 7fc238a00640 1 -- 172.21.15.2:0/2740249405 >> 172.21.15.2:0/2740249405 conn(0x7fc2340b0840 msgr2=0x7fc2340b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.290+0000 7fc238a00640 1 -- 172.21.15.2:0/2740249405 shutdown_connections 2024-01-15T18:12:21.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.290+0000 7fc238a00640 1 -- 172.21.15.2:0/2740249405 wait complete. 2024-01-15T18:12:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.291+0000 7fc238a00640 1 Processor -- start 2024-01-15T18:12:21.294 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 -- start start 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 0x7fc2340aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc2340ad360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc234148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340af390 con 0x7fc23406da60 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340addb0 con 0x7fc2340700d0 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.292+0000 7fc238a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2340ae090 con 0x7fc23414af80 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 0x7fc2340aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 0x7fc2340aee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50208/0 (socket says 172.21.15.2:50208) 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 -- 172.21.15.2:0/991602376 learned_addr learned my addr 172.21.15.2:0/991602376 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc22be00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc2340ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.296 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233e00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc234148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 -- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 msgr2=0x7fc234148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc234148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 -- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 msgr2=0x7fc2340ad360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.293+0000 7fc233400640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc2340ad360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc233400640 1 -- 172.21.15.2:0/991602376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c009b30 con 0x7fc23406da60 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc233400640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 0x7fc2340aee50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e9b0 tx=0x7fc22400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224009670 con 0x7fc23406da60 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22400c9a0 con 0x7fc23406da60 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2340ae370 con 0x7fc23406da60 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22400cd50 con 0x7fc23406da60 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.294+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2340ae930 con 0x7fc23406da60 2024-01-15T18:12:21.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.296+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc234074770 con 0x7fc23406da60 2024-01-15T18:12:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.297+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fc224013470 con 0x7fc23406da60 2024-01-15T18:12:21.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.298+0000 7fc22b400640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc20c06c1f0 0x7fc20c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.298+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fc224088990 con 0x7fc23406da60 2024-01-15T18:12:21.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.298+0000 7fc22be00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc20c06c1f0 0x7fc20c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.300+0000 7fc22be00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc20c06c1f0 0x7fc20c06e6b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc23406ea50 tx=0x7fc21c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:21.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.302+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc224055250 con 0x7fc23406da60 2024-01-15T18:12:21.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.533+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc234163940 con 0x7fc23406da60 2024-01-15T18:12:21.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.534+0000 7fc22b400640 1 -- 172.21.15.2:0/991602376 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fc224021e90 con 0x7fc23406da60 2024-01-15T18:12:21.537 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:21.537 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2471268746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2471268746},{"type":"v1","addr":"172.21.15.137:6836","nonce":2471268746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1012168762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":1012168762},{"type":"v1","addr":"172.21.15.137:6837","nonce":1012168762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6559,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2801413765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2801413765},{"type":"v1","addr":"172.21.15.2:6833","nonce":2801413765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2721206574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2721206574},{"type":"v1","addr":"172.21.15.2:6835","nonce":2721206574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[]} 2024-01-15T18:12:21.538 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 59 2024-01-15T18:12:21.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc20c06c1f0 msgr2=0x7fc20c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc20c06c1f0 0x7fc20c06e6b0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc23406ea50 tx=0x7fc21c009040 comp rx=0 tx=0).stop 2024-01-15T18:12:21.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 msgr2=0x7fc2340aee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 0x7fc2340aee50 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7fc22400e9b0 tx=0x7fc22400ee70 comp rx=0 tx=0).stop 2024-01-15T18:12:21.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 shutdown_connections 2024-01-15T18:12:21.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc20c06c1f0 0x7fc20c06e6b0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc23414af80 0x7fc234148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2340700d0 0x7fc2340ad360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 --2- 172.21.15.2:0/991602376 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc23406da60 0x7fc2340aee50 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 >> 172.21.15.2:0/991602376 conn(0x7fc2340b0840 msgr2=0x7fc234162890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:21.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 shutdown_connections 2024-01-15T18:12:21.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.537+0000 7fc238a00640 1 -- 172.21.15.2:0/991602376 wait complete. 2024-01-15T18:12:21.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:12:21.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.785+0000 7fdbf7e00640 1 Processor -- start 2024-01-15T18:12:21.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.785+0000 7fdbf7e00640 1 -- start start 2024-01-15T18:12:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.787+0000 7fdbf7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf815f260 0x7fdbf806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.787+0000 7fdbf7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdbf80720d0 0x7fdbf806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.787+0000 7fdbf7e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbf8071090 con 0x7fdbf8143d20 2024-01-15T18:12:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.787+0000 7fdbf7e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbf8071430 con 0x7fdbf80720d0 2024-01-15T18:12:21.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.787+0000 7fdbf7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8144100 con 0x7fdbf815f260 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.787+0000 7fdbf6a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdbf8143d20 0x7fdbf815eb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35324/0 (socket says 172.21.15.2:35324) 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf6a00640 1 -- 172.21.15.2:0/2850901943 learned_addr learned my addr 172.21.15.2:0/2850901943 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf7400640 1 --2- 172.21.15.2:0/2850901943 >> v2:172.21.15.2:3300/0 conn(0x7fdbf80720d0 0x7fdbf806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbefe00640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf815f260 0x7fdbf806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf7400640 1 -- 172.21.15.2:0/2850901943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf815f260 msgr2=0x7fdbf806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf7400640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf815f260 0x7fdbf806c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf7400640 1 -- 172.21.15.2:0/2850901943 >> v1:172.21.15.2:6789/0 conn(0x7fdbf8143d20 legacy=0x7fdbf815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf6000640 1 -- 172.21.15.2:0/2850901943 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986455907 0 0) 0x7fdbf8071090 con 0x7fdbf8143d20 2024-01-15T18:12:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbf7400640 1 -- 172.21.15.2:0/2850901943 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf8144560 con 0x7fdbf80720d0 2024-01-15T18:12:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.788+0000 7fdbefe00640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf815f260 0x7fdbf806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:21.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.790+0000 7fdbf7400640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf80720d0 0x7fdbf806c690 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009d90 tx=0x7fdbf002bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80ba951c303ebdae server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:21.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.790+0000 7fdbf6000640 1 -- 172.21.15.2:0/2850901943 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf002cb30 con 0x7fdbf80720d0 2024-01-15T18:12:21.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.790+0000 7fdbf6000640 1 -- 172.21.15.2:0/2850901943 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf002cc90 con 0x7fdbf80720d0 2024-01-15T18:12:21.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.790+0000 7fdbf6000640 1 -- 172.21.15.2:0/2850901943 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0034440 con 0x7fdbf80720d0 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2850901943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf80720d0 msgr2=0x7fdbf806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf80720d0 0x7fdbf806c690 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7fdbf0009d90 tx=0x7fdbf002bd80 comp rx=0 tx=0).stop 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2850901943 shutdown_connections 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf80720d0 0x7fdbf806c690 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2850901943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf815f260 0x7fdbf806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2850901943 >> 172.21.15.2:0/2850901943 conn(0x7fdbf80b0840 msgr2=0x7fdbf80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2850901943 shutdown_connections 2024-01-15T18:12:21.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.791+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2850901943 wait complete. 2024-01-15T18:12:21.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.793+0000 7fdbf7e00640 1 Processor -- start 2024-01-15T18:12:21.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.793+0000 7fdbf7e00640 1 -- start start 2024-01-15T18:12:21.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbf7e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf80720d0 0x7fdbf813f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbf7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 0x7fdbf813d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbf7e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbf815f260 0x7fdbf815d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbf7e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf8140a40 con 0x7fdbf8143d20 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbf7e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf813f730 con 0x7fdbf815f260 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbf7e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf813e7a0 con 0x7fdbf80720d0 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbefe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 0x7fdbf813d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbefe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 0x7fdbf813d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50228/0 (socket says 172.21.15.2:50228) 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.794+0000 7fdbefe00640 1 -- 172.21.15.2:0/2293294705 learned_addr learned my addr 172.21.15.2:0/2293294705 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbf7400640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbf815f260 0x7fdbf815d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbf6a00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf80720d0 0x7fdbf813f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbefe00640 1 -- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf80720d0 msgr2=0x7fdbf813f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbefe00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf80720d0 0x7fdbf813f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbefe00640 1 -- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbf815f260 msgr2=0x7fdbf815d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbefe00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbf815f260 0x7fdbf815d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbefe00640 1 -- 172.21.15.2:0/2293294705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0009a70 con 0x7fdbf8143d20 2024-01-15T18:12:21.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbf6a00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf80720d0 0x7fdbf813f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbf7400640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbf815f260 0x7fdbf815d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbefe00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 0x7fdbf813d490 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000e7e0 tx=0x7fdbe000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0004030 con 0x7fdbf8143d20 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbe0004190 con 0x7fdbf8143d20 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbf8140c50 con 0x7fdbf8143d20 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.795+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbe0004520 con 0x7fdbf8143d20 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.796+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf813e9f0 con 0x7fdbf8143d20 2024-01-15T18:12:21.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.797+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbf8074770 con 0x7fdbf8143d20 2024-01-15T18:12:21.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.798+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fdbe000c670 con 0x7fdbf8143d20 2024-01-15T18:12:21.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.798+0000 7fdbeea00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdbd006c1f0 0x7fdbd006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:21.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.799+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fdbe00885e0 con 0x7fdbf8143d20 2024-01-15T18:12:21.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.799+0000 7fdbf6a00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdbd006c1f0 0x7fdbd006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.801+0000 7fdbf6a00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdbd006c1f0 0x7fdbd006e6b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8000f60 tx=0x7fdbe8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:21.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.803+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbe0054bb0 con 0x7fdbf8143d20 2024-01-15T18:12:22.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:21.999+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fdbf80ad3b0 con 0x7fdbf8143d20 2024-01-15T18:12:22.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.007+0000 7fdbeea00640 1 -- 172.21.15.2:0/2293294705 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v32) v1 ==== 120+0+0 (secure 0 0 0) 0x7fdbe00581d0 con 0x7fdbf8143d20 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.009+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdbd006c1f0 msgr2=0x7fdbd006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.009+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdbd006c1f0 0x7fdbd006e6b0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fdbe8000f60 tx=0x7fdbe8009040 comp rx=0 tx=0).stop 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.009+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 msgr2=0x7fdbf813d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.009+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 0x7fdbf813d490 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fdbe000e7e0 tx=0x7fdbe000eca0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 shutdown_connections 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdbd006c1f0 0x7fdbd006e6b0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbf815f260 0x7fdbf815d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbf8143d20 0x7fdbf813d490 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 --2- 172.21.15.2:0/2293294705 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbf80720d0 0x7fdbf813f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 >> 172.21.15.2:0/2293294705 conn(0x7fdbf80b0840 msgr2=0x7fdbf80ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:22.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 shutdown_connections 2024-01-15T18:12:22.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.010+0000 7fdbf7e00640 1 -- 172.21.15.2:0/2293294705 wait complete. 2024-01-15T18:12:22.034 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:12:22.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.253+0000 7fdbbb200640 1 Processor -- start 2024-01-15T18:12:22.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.253+0000 7fdbbb200640 1 -- start start 2024-01-15T18:12:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.254+0000 7fdbbb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 0x7fdbbc06ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.254+0000 7fdbbb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdbbc15ea40 0x7fdbbc099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.254+0000 7fdbbb200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc09a8b0 con 0x7fdbbc06f880 2024-01-15T18:12:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.254+0000 7fdbbb200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc09a5d0 con 0x7fdbbc15ea40 2024-01-15T18:12:22.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.254+0000 7fdbbb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0d19d0 con 0x7fdbbc15e660 2024-01-15T18:12:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.255+0000 7fdbba800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdbbc06f880 0x7fdbbc06e4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:35336/0 (socket says 172.21.15.2:35336) 2024-01-15T18:12:22.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.255+0000 7fdbba800640 1 -- 172.21.15.2:0/2885171093 learned_addr learned my addr 172.21.15.2:0/2885171093 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.255+0000 7fdbb9400640 1 --2- 172.21.15.2:0/2885171093 >> v2:172.21.15.137:3300/0 conn(0x7fdbbc15ea40 0x7fdbbc099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.255+0000 7fdbb9e00640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 0x7fdbbc06ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.255+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756695660 0 0) 0x7fdbbc09a8b0 con 0x7fdbbc06f880 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.255+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb9c003520 con 0x7fdbbc06f880 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 987430287 0 0) 0x7fdb9c003520 con 0x7fdbbc06f880 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 msgr2=0x7fdbbc06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 0x7fdbbc06ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 >> v2:172.21.15.137:3300/0 conn(0x7fdbbc15ea40 msgr2=0x7fdbbc099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 --2- 172.21.15.2:0/2885171093 >> v2:172.21.15.137:3300/0 conn(0x7fdbbc15ea40 0x7fdbbc099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0d1cb0 con 0x7fdbbc06f880 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fdbb40019f0 con 0x7fdbbc06f880 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 >> v1:172.21.15.2:6789/0 conn(0x7fdbbc06f880 legacy=0x7fdbbc06e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 0x7fdb9c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb9c003ef0 0x7fdb9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb9c004d00 0x7fdb9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc09a5d0 con 0x7fdb9c004d00 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0d19d0 con 0x7fdb9c003ef0 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb9c00b980 con 0x7fdbbc15e660 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbba800640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb9c004d00 0x7fdb9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.256+0000 7fdbb9e00640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 0x7fdb9c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.257+0000 7fdbb9400640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb9c003ef0 0x7fdb9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.257+0000 7fdbba800640 1 -- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 msgr2=0x7fdb9c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.257+0000 7fdbba800640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdbbc15e660 0x7fdb9c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.257+0000 7fdbba800640 1 -- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb9c003ef0 msgr2=0x7fdb9c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.257+0000 7fdbba800640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdb9c003ef0 0x7fdb9c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.257+0000 7fdbba800640 1 -- 172.21.15.2:0/2885171093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb9c003520 con 0x7fdb9c004d00 2024-01-15T18:12:22.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.258+0000 7fdbba800640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb9c004d00 0x7fdb9c005180 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400a860 tx=0x7fdb9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6123a9ff544e4c98 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.258+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4034630 con 0x7fdb9c004d00 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.258+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb402f070 con 0x7fdb9c004d00 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.258+0000 7fdbb8a00640 1 -- 172.21.15.2:0/2885171093 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4034a30 con 0x7fdb9c004d00 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbbb200640 1 -- 172.21.15.2:0/2885171093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb9c004d00 msgr2=0x7fdb9c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbbb200640 1 --2- 172.21.15.2:0/2885171093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdb9c004d00 0x7fdb9c005180 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400a860 tx=0x7fdb9c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbba800640 1 -- 172.21.15.2:0/2885171093 reap_dead start 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbbb200640 1 -- 172.21.15.2:0/2885171093 shutdown_connections 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbbb200640 1 -- 172.21.15.2:0/2885171093 >> 172.21.15.2:0/2885171093 conn(0x7fdbbc0d0e40 msgr2=0x7fdbbc0d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbbb200640 1 -- 172.21.15.2:0/2885171093 shutdown_connections 2024-01-15T18:12:22.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.259+0000 7fdbbb200640 1 -- 172.21.15.2:0/2885171093 wait complete. 2024-01-15T18:12:22.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.261+0000 7fdbbb200640 1 Processor -- start 2024-01-15T18:12:22.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.261+0000 7fdbbb200640 1 -- start start 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbbb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdb9c004d00 0x7fdbbc1f0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbbb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 0x7fdbbc1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbbb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbbc0b5cc0 0x7fdbbc0b6170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbbb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1f0770 con 0x7fdbbc0c7490 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbbb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0b66b0 con 0x7fdbbc0b5cc0 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbbb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0b4c60 con 0x7fdb9c004d00 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbb9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 0x7fdbbc1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbb9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 0x7fdbbc1f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50244/0 (socket says 172.21.15.2:50244) 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.262+0000 7fdbb9400640 1 -- 172.21.15.2:0/529993195 learned_addr learned my addr 172.21.15.2:0/529993195 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9e00640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdb9c004d00 0x7fdbbc1f0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbba800640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbbc0b5cc0 0x7fdbbc0b6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9400640 1 -- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdb9c004d00 msgr2=0x7fdbbc1f0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9400640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdb9c004d00 0x7fdbbc1f0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9400640 1 -- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbbc0b5cc0 msgr2=0x7fdbbc0b6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9400640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbbc0b5cc0 0x7fdbbc0b6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9400640 1 -- 172.21.15.2:0/529993195 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc1f3c90 con 0x7fdbbc0c7490 2024-01-15T18:12:22.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb9400640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 0x7fdbbc1f3750 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdba4011600 tx=0x7fdba4011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba400f8c0 con 0x7fdbbc0c7490 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba400fa20 con 0x7fdbbc0c7490 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc0b50f0 con 0x7fdbbc0c7490 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.263+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba400fd90 con 0x7fdbbc0c7490 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.264+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc0b4330 con 0x7fdbbc0c7490 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.265+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc074770 con 0x7fdbbc0c7490 2024-01-15T18:12:22.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.266+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fdba4015460 con 0x7fdbbc0c7490 2024-01-15T18:12:22.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.267+0000 7fdbb2000640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdb9406c220 0x7fdb9406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.267+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fdba4089240 con 0x7fdbbc0c7490 2024-01-15T18:12:22.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.267+0000 7fdbb9e00640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdb9406c220 0x7fdb9406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.268+0000 7fdbb9e00640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdb9406c220 0x7fdb9406e6e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009a10 tx=0x7fdbac008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:22.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.271+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdba4055810 con 0x7fdbbc0c7490 2024-01-15T18:12:22.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.468+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fdbbc0b4da0 con 0x7fdbbc0c7490 2024-01-15T18:12:22.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.473+0000 7fdbb2000640 1 -- 172.21.15.2:0/529993195 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7fdba4058e30 con 0x7fdbbc0c7490 2024-01-15T18:12:22.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdb9406c220 msgr2=0x7fdb9406e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdb9406c220 0x7fdb9406e6e0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009a10 tx=0x7fdbac008040 comp rx=0 tx=0).stop 2024-01-15T18:12:22.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 msgr2=0x7fdbbc1f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 0x7fdbbc1f3750 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fdba4011600 tx=0x7fdba4011ac0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 shutdown_connections 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdb9406c220 0x7fdb9406e6e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdbbc0b5cc0 0x7fdbbc0b6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdbbc0c7490 0x7fdbbc1f3750 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 --2- 172.21.15.2:0/529993195 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdb9c004d00 0x7fdbbc1f0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 >> 172.21.15.2:0/529993195 conn(0x7fdbbc0d0e40 msgr2=0x7fdbbc15ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 shutdown_connections 2024-01-15T18:12:22.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.476+0000 7fdbbb200640 1 -- 172.21.15.2:0/529993195 wait complete. 2024-01-15T18:12:22.501 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed' 2024-01-15T18:12:22.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.718+0000 7f0589800640 1 Processor -- start 2024-01-15T18:12:22.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.718+0000 7f0589800640 1 -- start start 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0589800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058415cf90 0x7f058415d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0589800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f05840b2d90 0x7f05840b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0589800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f05840cfa70 0x7f05840cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0589800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840d13a0 con 0x7f05840b2d90 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0589800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840d05e0 con 0x7f05840cfa70 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0589800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05840d08c0 con 0x7f058415cf90 2024-01-15T18:12:22.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0583400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f05840b2d90 0x7f05840b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0583400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f05840b2d90 0x7f05840b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50260/0 (socket says 172.21.15.2:50260) 2024-01-15T18:12:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.720+0000 7f0583400640 1 -- 172.21.15.2:0/4191209964 learned_addr learned my addr 172.21.15.2:0/4191209964 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:22.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583400640 1 -- 172.21.15.2:0/4191209964 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058415cf90 msgr2=0x7f058415d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583e00640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058415cf90 0x7f058415d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0588e00640 1 --2- 172.21.15.2:0/4191209964 >> v2:172.21.15.137:3300/0 conn(0x7f05840cfa70 0x7f05840cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583400640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058415cf90 0x7f058415d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583400640 1 -- 172.21.15.2:0/4191209964 >> v2:172.21.15.137:3300/0 conn(0x7f05840cfa70 msgr2=0x7f05840cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583400640 1 --2- 172.21.15.2:0/4191209964 >> v2:172.21.15.137:3300/0 conn(0x7f05840cfa70 0x7f05840cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583400640 1 -- 172.21.15.2:0/4191209964 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840d0ce0 con 0x7f05840b2d90 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0583e00640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058415cf90 0x7f058415d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.721+0000 7f0588e00640 1 --2- 172.21.15.2:0/4191209964 >> v2:172.21.15.137:3300/0 conn(0x7f05840cfa70 0x7f05840cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:22.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.722+0000 7f0583400640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05840b2d90 0x7f05840b31d0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f05840cd6c0 tx=0x7f056c02b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b708a4d254e111d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.722+0000 7f0582a00640 1 -- 172.21.15.2:0/4191209964 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0317f0 con 0x7f05840b2d90 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.722+0000 7f0582a00640 1 -- 172.21.15.2:0/4191209964 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c031950 con 0x7f05840b2d90 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.722+0000 7f0582a00640 1 -- 172.21.15.2:0/4191209964 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c031ce0 con 0x7f05840b2d90 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 -- 172.21.15.2:0/4191209964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05840b2d90 msgr2=0x7f05840b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05840b2d90 0x7f05840b31d0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f05840cd6c0 tx=0x7f056c02b890 comp rx=0 tx=0).stop 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 -- 172.21.15.2:0/4191209964 shutdown_connections 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 --2- 172.21.15.2:0/4191209964 >> v2:172.21.15.137:3300/0 conn(0x7f05840cfa70 0x7f05840cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f05840b2d90 0x7f05840b31d0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 --2- 172.21.15.2:0/4191209964 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f058415cf90 0x7f058415d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 -- 172.21.15.2:0/4191209964 >> 172.21.15.2:0/4191209964 conn(0x7f058408cad0 msgr2=0x7f058409c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:22.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 -- 172.21.15.2:0/4191209964 shutdown_connections 2024-01-15T18:12:22.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.723+0000 7f0589800640 1 -- 172.21.15.2:0/4191209964 wait complete. 2024-01-15T18:12:22.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.726+0000 7f0589800640 1 Processor -- start 2024-01-15T18:12:22.730 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.726+0000 7f0589800640 1 -- start start 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0589800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05840b2d90 0x7f0584157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0589800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05840cfa70 0x7f05841579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0589800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 0x7f05840b5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0589800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058415a0c0 con 0x7f058415cf90 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0589800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05841588b0 con 0x7f05840cfa70 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0589800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05840b6270 con 0x7f05840b2d90 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0588e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 0x7f05840b5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0588e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 0x7f05840b5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50270/0 (socket says 172.21.15.2:50270) 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.727+0000 7f0588e00640 1 -- 172.21.15.2:0/4179038309 learned_addr learned my addr 172.21.15.2:0/4179038309 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:22.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0583400640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05840cfa70 0x7f05841579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0583e00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05840b2d90 0x7f0584157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0588e00640 1 -- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05840b2d90 msgr2=0x7f0584157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0588e00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05840b2d90 0x7f0584157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0588e00640 1 -- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05840cfa70 msgr2=0x7f05841579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0588e00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05840cfa70 0x7f05841579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0588e00640 1 -- 172.21.15.2:0/4179038309 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c0094d0 con 0x7f058415cf90 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0583400640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05840cfa70 0x7f05841579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:22.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.728+0000 7f0588e00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 0x7f05840b5d30 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f057c00cef0 tx=0x7f057c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.729+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c00d840 con 0x7f058415cf90 2024-01-15T18:12:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.729+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c00d9a0 con 0x7f058415cf90 2024-01-15T18:12:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.729+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057c00dd50 con 0x7f058415cf90 2024-01-15T18:12:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.729+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05840b6640 con 0x7f058415cf90 2024-01-15T18:12:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.729+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05840b6c00 con 0x7f058415cf90 2024-01-15T18:12:22.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.730+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058415a200 con 0x7f058415cf90 2024-01-15T18:12:22.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.731+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f057c01e3b0 con 0x7f058415cf90 2024-01-15T18:12:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.732+0000 7f057be00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f055c06c1f0 0x7f055c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:22.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.732+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f057c0890d0 con 0x7f058415cf90 2024-01-15T18:12:22.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.733+0000 7f0583e00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f055c06c1f0 0x7f055c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:22.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.734+0000 7f0583e00640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f055c06c1f0 0x7f055c06e6b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f05840b64a0 tx=0x7f0574008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:22.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:22.737+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f058415a200 con 0x7f058415cf90 2024-01-15T18:12:23.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.002+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]} v 0) v1 -- 0x7f058415a200 con 0x7f058415cf90 2024-01-15T18:12:23.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.042+0000 7f057be00640 1 -- 172.21.15.2:0/4179038309 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_stats_are_printed"]}]=0 v675) v1 ==== 145+0+0 (secure 0 0 0) 0x7f058415a200 con 0x7f058415cf90 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f055c06c1f0 msgr2=0x7f055c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f055c06c1f0 0x7f055c06e6b0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f05840b64a0 tx=0x7f0574008040 comp rx=0 tx=0).stop 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 msgr2=0x7f05840b5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 0x7f05840b5d30 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f057c00cef0 tx=0x7f057c00a710 comp rx=0 tx=0).stop 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 shutdown_connections 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f055c06c1f0 0x7f055c06e6b0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f058415cf90 0x7f05840b5d30 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f05840cfa70 0x7f05841579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 --2- 172.21.15.2:0/4179038309 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f05840b2d90 0x7f0584157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 >> 172.21.15.2:0/4179038309 conn(0x7f058408cad0 msgr2=0x7f05840d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.045+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 shutdown_connections 2024-01-15T18:12:23.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.046+0000 7f0589800640 1 -- 172.21.15.2:0/4179038309 wait complete. 2024-01-15T18:12:23.067 INFO:tasks.cephfs_test_runner:Test that the command "ceph fs clone status" prints progress stats ... ok 2024-01-15T18:12:23.068 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:12:23.068 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:12:23.068 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:23.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.273+0000 7fa57f000640 1 Processor -- start 2024-01-15T18:12:23.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.273+0000 7fa57f000640 1 -- start start 2024-01-15T18:12:23.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.274+0000 7fa57f000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa578143810 0x7fa57806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.274+0000 7fa57f000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa5780b2250 0x7fa5780cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.274+0000 7fa57f000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa57806e470 con 0x7fa57806e010 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.274+0000 7fa57f000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780b3520 con 0x7fa5780b2250 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.274+0000 7fa57f000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5780ccbe0 con 0x7fa578143810 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa5780b2250 0x7fa5780cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa5780b2250 0x7fa5780cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53858/0 (socket says 172.21.15.2:53858) 2024-01-15T18:12:23.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 -- 172.21.15.2:0/3709326517 learned_addr learned my addr 172.21.15.2:0/3709326517 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:23.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57dc00640 1 --2- 172.21.15.2:0/3709326517 >> v2:172.21.15.137:3300/0 conn(0x7fa578143810 0x7fa57806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 -- 172.21.15.2:0/3709326517 >> v2:172.21.15.137:3300/0 conn(0x7fa578143810 msgr2=0x7fa57806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:23.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 --2- 172.21.15.2:0/3709326517 >> v2:172.21.15.137:3300/0 conn(0x7fa578143810 0x7fa57806dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 -- 172.21.15.2:0/3709326517 >> v1:172.21.15.2:6789/0 conn(0x7fa57806e010 legacy=0x7fa5780b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa577e00640 1 -- 172.21.15.2:0/3709326517 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 875215022 0 0) 0x7fa560001110 con 0x7fa57806e010 2024-01-15T18:12:23.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.275+0000 7fa57e600640 1 -- 172.21.15.2:0/3709326517 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578143bf0 con 0x7fa5780b2250 2024-01-15T18:12:23.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.276+0000 7fa57e600640 1 --2- 172.21.15.2:0/3709326517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5780b2250 0x7fa5780cd070 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fa578068540 tx=0x7fa57002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27831116fda2617a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:23.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa577e00640 1 -- 172.21.15.2:0/3709326517 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57002ca80 con 0x7fa5780b2250 2024-01-15T18:12:23.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa577e00640 1 -- 172.21.15.2:0/3709326517 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57002cbe0 con 0x7fa5780b2250 2024-01-15T18:12:23.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa577e00640 1 -- 172.21.15.2:0/3709326517 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5700343f0 con 0x7fa5780b2250 2024-01-15T18:12:23.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa57f000640 1 -- 172.21.15.2:0/3709326517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5780b2250 msgr2=0x7fa5780cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa57f000640 1 --2- 172.21.15.2:0/3709326517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5780b2250 0x7fa5780cd070 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fa578068540 tx=0x7fa57002bee0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa57f000640 1 -- 172.21.15.2:0/3709326517 shutdown_connections 2024-01-15T18:12:23.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa57f000640 1 --2- 172.21.15.2:0/3709326517 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa5780b2250 0x7fa5780cd070 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa57f000640 1 --2- 172.21.15.2:0/3709326517 >> v2:172.21.15.137:3300/0 conn(0x7fa578143810 0x7fa57806dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.277+0000 7fa57f000640 1 -- 172.21.15.2:0/3709326517 >> 172.21.15.2:0/3709326517 conn(0x7fa5780b0840 msgr2=0x7fa5780b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:23.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.278+0000 7fa57f000640 1 -- 172.21.15.2:0/3709326517 shutdown_connections 2024-01-15T18:12:23.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.278+0000 7fa57f000640 1 -- 172.21.15.2:0/3709326517 wait complete. 2024-01-15T18:12:23.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.279+0000 7fa57f000640 1 Processor -- start 2024-01-15T18:12:23.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 -- start start 2024-01-15T18:12:23.284 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 0x7fa5781402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5780b2250 0x7fa5780ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa578143810 0x7fa5780c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781e2760 con 0x7fa57806e010 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa578140820 con 0x7fa578143810 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.280+0000 7fa57f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781f6be0 con 0x7fa5780b2250 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 0x7fa5781402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 0x7fa5781402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53864/0 (socket says 172.21.15.2:53864) 2024-01-15T18:12:23.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 -- 172.21.15.2:0/2427268959 learned_addr learned my addr 172.21.15.2:0/2427268959 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:23.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57d200640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5780b2250 0x7fa5780ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57e600640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa578143810 0x7fa5780c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 -- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5780b2250 msgr2=0x7fa5780ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5780b2250 0x7fa5780ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 -- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa578143810 msgr2=0x7fa5780c8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa578143810 0x7fa5780c8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57dc00640 1 -- 172.21.15.2:0/2427268959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa570009b30 con 0x7fa57806e010 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.281+0000 7fa57e600640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa578143810 0x7fa5780c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.282+0000 7fa57dc00640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 0x7fa5781402e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa56800d800 tx=0x7fa56800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.282+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56800b6c0 con 0x7fa57806e010 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.282+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa56800b820 con 0x7fa57806e010 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.282+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56800bbb0 con 0x7fa57806e010 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.282+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5781f6ec0 con 0x7fa57806e010 2024-01-15T18:12:23.287 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.282+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5781f7480 con 0x7fa57806e010 2024-01-15T18:12:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.283+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa578074770 con 0x7fa57806e010 2024-01-15T18:12:23.288 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.284+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fa56800bd10 con 0x7fa57806e010 2024-01-15T18:12:23.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.285+0000 7fa575600640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa55006c1f0 0x7fa55006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.285+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fa568088f70 con 0x7fa57806e010 2024-01-15T18:12:23.289 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.286+0000 7fa57d200640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa55006c1f0 0x7fa55006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.287+0000 7fa57d200640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa55006c1f0 0x7fa55006e6b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa5781e8980 tx=0x7fa560009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:23.293 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.289+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa568055540 con 0x7fa57806e010 2024-01-15T18:12:23.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.485+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5781e2520 con 0x7fa57806e010 2024-01-15T18:12:23.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.487+0000 7fa575600640 1 -- 172.21.15.2:0/2427268959 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+13101 (secure 0 0 0) 0x7fa568058b60 con 0x7fa57806e010 2024-01-15T18:12:23.520 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:23.520 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":94,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:16.992434+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:6835/3200314764":"2024-01-16T18:12:13.977758+0000","172.21.15.2:6834/3200314764":"2024-01-16T18:12:13.977758+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:23.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.489+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa55006c1f0 msgr2=0x7fa55006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.489+0000 7fa57f000640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa55006c1f0 0x7fa55006e6b0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa5781e8980 tx=0x7fa560009040 comp rx=0 tx=0).stop 2024-01-15T18:12:23.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.489+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 msgr2=0x7fa5781402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.489+0000 7fa57f000640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 0x7fa5781402e0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fa56800d800 tx=0x7fa56800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 shutdown_connections 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa55006c1f0 0x7fa55006e6b0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa578143810 0x7fa5780c8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa5780b2250 0x7fa5780ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 --2- 172.21.15.2:0/2427268959 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa57806e010 0x7fa5781402e0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 >> 172.21.15.2:0/2427268959 conn(0x7fa5780b0840 msgr2=0x7fa578142580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 shutdown_connections 2024-01-15T18:12:23.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.490+0000 7fa57f000640 1 -- 172.21.15.2:0/2427268959 wait complete. 2024-01-15T18:12:23.523 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:12:23.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:12:23.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.745+0000 7f88d0c00640 1 Processor -- start 2024-01-15T18:12:23.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.746+0000 7f88d0c00640 1 -- start start 2024-01-15T18:12:23.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.747+0000 7f88d0c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88cc06e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.747+0000 7f88d0c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc06f150 con 0x7f88cc072050 2024-01-15T18:12:23.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.747+0000 7f88d0c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc071570 con 0x7f88cc0b4540 2024-01-15T18:12:23.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.747+0000 7f88d0c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc06f430 con 0x7f88cc15b320 2024-01-15T18:12:23.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.748+0000 7f88ca800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f88cc072050 0x7f88cc15b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44804/0 (socket says 172.21.15.2:44804) 2024-01-15T18:12:23.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.748+0000 7f88ca800640 1 -- 172.21.15.2:0/3068661596 learned_addr learned my addr 172.21.15.2:0/3068661596 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.748+0000 7f88c9400640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88cc06e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.748+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605157287 0 0) 0x7f88cc06f150 con 0x7f88cc072050 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.748+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88ac005730 con 0x7f88cc072050 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3343369957 0 0) 0x7f88ac005730 con 0x7f88cc072050 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 msgr2=0x7f88cc06e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88cc06e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 >> v1:172.21.15.137:6789/0 conn(0x7f88cc0b4540 legacy=0x7f88cc0b4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc071850 con 0x7f88cc072050 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f88c4001670 con 0x7f88cc072050 2024-01-15T18:12:23.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 >> v1:172.21.15.2:6789/0 conn(0x7f88cc072050 legacy=0x7f88cc15b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac005bb0 0x7f88ac005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88ac0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac006f10 0x7f88ac007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc06f430 con 0x7f88ac006f10 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac00dcc0 con 0x7f88ac005bb0 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac00de80 con 0x7f88cc15b320 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88ca800640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac006f10 0x7f88ac007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.749+0000 7f88c9e00640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac005bb0 0x7f88ac005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88c9400640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88ac0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88ca800640 1 -- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 msgr2=0x7f88ac0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88ca800640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88ac0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88ca800640 1 -- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac005bb0 msgr2=0x7f88ac005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:23.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88ca800640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac005bb0 0x7f88ac005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88ca800640 1 -- 172.21.15.2:0/3068661596 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac005730 con 0x7f88ac006f10 2024-01-15T18:12:23.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.750+0000 7f88c9400640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88cc15b320 0x7f88ac0069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:23.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.751+0000 7f88ca800640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac006f10 0x7f88ac007390 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f88c400c240 tx=0x7f88ac007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f355b0d7304ee20 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.751+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4005ad0 con 0x7f88ac006f10 2024-01-15T18:12:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.751+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c4005c50 con 0x7f88ac006f10 2024-01-15T18:12:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.751+0000 7f88c8a00640 1 -- 172.21.15.2:0/3068661596 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c402c8b0 con 0x7f88ac006f10 2024-01-15T18:12:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88d0c00640 1 -- 172.21.15.2:0/3068661596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac006f10 msgr2=0x7f88ac007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3068661596 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac006f10 0x7f88ac007390 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f88c400c240 tx=0x7f88ac007a00 comp rx=0 tx=0).stop 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88ca800640 1 -- 172.21.15.2:0/3068661596 reap_dead start 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88d0c00640 1 -- 172.21.15.2:0/3068661596 shutdown_connections 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88d0c00640 1 -- 172.21.15.2:0/3068661596 >> 172.21.15.2:0/3068661596 conn(0x7f88cc0b1d70 msgr2=0x7f88cc0b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88d0c00640 1 -- 172.21.15.2:0/3068661596 shutdown_connections 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.752+0000 7f88d0c00640 1 -- 172.21.15.2:0/3068661596 wait complete. 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.754+0000 7f88d0c00640 1 Processor -- start 2024-01-15T18:12:23.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.754+0000 7f88d0c00640 1 -- start start 2024-01-15T18:12:23.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88d0c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac006f10 0x7f88cc144940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88d0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 0x7f88cc154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88d0c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88cc154c50 0x7f88cc155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88d0c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc0ad810 con 0x7f88cc152fe0 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88d0c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc155740 con 0x7f88cc154c50 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88d0c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc144e80 con 0x7f88ac006f10 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88c9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 0x7f88cc154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88c9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 0x7f88cc154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53880/0 (socket says 172.21.15.2:53880) 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.755+0000 7f88c9400640 1 -- 172.21.15.2:0/3524979185 learned_addr learned my addr 172.21.15.2:0/3524979185 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9e00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac006f10 0x7f88cc144940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88ca800640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88cc154c50 0x7f88cc155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9400640 1 -- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac006f10 msgr2=0x7f88cc144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9400640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac006f10 0x7f88cc144940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9400640 1 -- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88cc154c50 msgr2=0x7f88cc155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9400640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88cc154c50 0x7f88cc155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9400640 1 -- 172.21.15.2:0/3524979185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc155180 con 0x7f88cc152fe0 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c9400640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 0x7f88cc154710 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f88b4007f30 tx=0x7f88b4011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.756+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b400e030 con 0x7f88cc152fe0 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.757+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b4016470 con 0x7f88cc152fe0 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.757+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4016820 con 0x7f88cc152fe0 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.757+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc155460 con 0x7f88cc152fe0 2024-01-15T18:12:23.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.757+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88cc1563e0 con 0x7f88cc152fe0 2024-01-15T18:12:23.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.758+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88cc0ad980 con 0x7f88cc152fe0 2024-01-15T18:12:23.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.759+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f88b4016980 con 0x7f88cc152fe0 2024-01-15T18:12:23.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.760+0000 7f88c2000640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88a406c220 0x7f88a406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:23.764 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.760+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f88b400f070 con 0x7f88cc152fe0 2024-01-15T18:12:23.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.761+0000 7f88c9e00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88a406c220 0x7f88a406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:23.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.762+0000 7f88c9e00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88a406c220 0x7f88a406e6e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f88bc005590 tx=0x7f88bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:23.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.765+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88b4055f70 con 0x7f88cc152fe0 2024-01-15T18:12:23.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.960+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88cc0686f0 con 0x7f88cc152fe0 2024-01-15T18:12:23.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.961+0000 7f88c2000640 1 -- 172.21.15.2:0/3524979185 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88b4059590 con 0x7f88cc152fe0 2024-01-15T18:12:23.965 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:12:23.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88a406c220 msgr2=0x7f88a406e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88a406c220 0x7f88a406e6e0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f88bc005590 tx=0x7f88bc009040 comp rx=0 tx=0).stop 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 msgr2=0x7f88cc154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 0x7f88cc154710 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f88b4007f30 tx=0x7f88b4011660 comp rx=0 tx=0).stop 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 shutdown_connections 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f88a406c220 0x7f88a406e6e0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88cc154c50 0x7f88cc155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88cc152fe0 0x7f88cc154710 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 --2- 172.21.15.2:0/3524979185 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac006f10 0x7f88cc144940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 >> 172.21.15.2:0/3524979185 conn(0x7f88cc0b1d70 msgr2=0x7f88cc15bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 shutdown_connections 2024-01-15T18:12:23.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:23.964+0000 7f88d0c00640 1 -- 172.21.15.2:0/3524979185 wait complete. 2024-01-15T18:12:23.986 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-15T18:12:24.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.201+0000 7f2972800640 1 Processor -- start 2024-01-15T18:12:24.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.201+0000 7f2972800640 1 -- start start 2024-01-15T18:12:24.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.202+0000 7f2972800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f296c0b8d20 0x7f296c160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.202+0000 7f2972800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c161450 con 0x7f296c15d360 2024-01-15T18:12:24.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.202+0000 7f2972800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296c0d1bd0 con 0x7f296c0cdeb0 2024-01-15T18:12:24.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.202+0000 7f2972800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296c0b9190 con 0x7f296c0b8d20 2024-01-15T18:12:24.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.202+0000 7f296b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f296c15d360 0x7f296c0d14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44814/0 (socket says 172.21.15.2:44814) 2024-01-15T18:12:24.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.202+0000 7f296b400640 1 -- 172.21.15.2:0/3987300826 learned_addr learned my addr 172.21.15.2:0/3987300826 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856980720 0 0) 0x7f296c161450 con 0x7f296c15d360 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c003520 con 0x7f296c15d360 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296aa00640 1 --2- 172.21.15.2:0/3987300826 >> v2:172.21.15.137:3300/0 conn(0x7f296c0b8d20 0x7f296c160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378128376 0 0) 0x7f294c003520 con 0x7f296c15d360 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 >> v2:172.21.15.137:3300/0 conn(0x7f296c0b8d20 msgr2=0x7f296c160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 --2- 172.21.15.2:0/3987300826 >> v2:172.21.15.137:3300/0 conn(0x7f296c0b8d20 0x7f296c160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 >> v1:172.21.15.137:6789/0 conn(0x7f296c0cdeb0 legacy=0x7f296c0ce2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:24.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c0cf8e0 con 0x7f296c15d360 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2954001670 con 0x7f296c15d360 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 >> v1:172.21.15.2:6789/0 conn(0x7f296c15d360 legacy=0x7f296c0d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f294c003970 0x7f294c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f294c0047c0 0x7f294c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c005180 0x7f294c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0b9190 con 0x7f294c003970 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c00c0d0 con 0x7f294c005180 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.203+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c00c290 con 0x7f294c0047c0 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296b400640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f294c003970 0x7f294c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296aa00640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f294c0047c0 0x7f294c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296b400640 1 -- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f294c0047c0 msgr2=0x7f294c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296be00640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c005180 0x7f294c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296b400640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f294c0047c0 0x7f294c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296b400640 1 -- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c005180 msgr2=0x7f294c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:24.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296b400640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c005180 0x7f294c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296b400640 1 -- 172.21.15.2:0/3987300826 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c003520 con 0x7f294c003970 2024-01-15T18:12:24.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296aa00640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f294c0047c0 0x7f294c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:24.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296be00640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c005180 0x7f294c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:24.210 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.204+0000 7f296be00640 1 -- 172.21.15.2:0/3987300826 reap_dead start 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.205+0000 7f296b400640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f294c003970 0x7f294c003d90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f295400c190 tx=0x7f294c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28705ffa3199590b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.205+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954005ad0 con 0x7f294c003970 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2954005c50 con 0x7f294c003970 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f296a000640 1 -- 172.21.15.2:0/3987300826 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295402c8a0 con 0x7f294c003970 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 -- 172.21.15.2:0/3987300826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f294c003970 msgr2=0x7f294c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f294c003970 0x7f294c003d90 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f295400c190 tx=0x7f294c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 -- 172.21.15.2:0/3987300826 shutdown_connections 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 --2- 172.21.15.2:0/3987300826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f294c003970 0x7f294c003d90 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 -- 172.21.15.2:0/3987300826 >> 172.21.15.2:0/3987300826 conn(0x7f296c072000 msgr2=0x7f296c072410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 -- 172.21.15.2:0/3987300826 shutdown_connections 2024-01-15T18:12:24.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.206+0000 7f2972800640 1 -- 172.21.15.2:0/3987300826 wait complete. 2024-01-15T18:12:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.208+0000 7f2972800640 1 Processor -- start 2024-01-15T18:12:24.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.208+0000 7f2972800640 1 -- start start 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f2972800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c003970 0x7f296c152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f2972800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 0x7f296c15f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f2972800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f296c0b37e0 0x7f296c0b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f2972800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c153390 con 0x7f296c0cb3b0 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f2972800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0b41b0 con 0x7f294c003970 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f2972800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296c0ca350 con 0x7f296c0b37e0 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f296aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 0x7f296c15f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f296aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 0x7f296c15f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53896/0 (socket says 172.21.15.2:53896) 2024-01-15T18:12:24.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.209+0000 7f296aa00640 1 -- 172.21.15.2:0/4260977546 learned_addr learned my addr 172.21.15.2:0/4260977546 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:24.214 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296b400640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c003970 0x7f296c152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296aa00640 1 -- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f296c0b37e0 msgr2=0x7f296c0b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296aa00640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f296c0b37e0 0x7f296c0b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296aa00640 1 -- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c003970 msgr2=0x7f296c152e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296aa00640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c003970 0x7f296c152e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296aa00640 1 -- 172.21.15.2:0/4260977546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f296c0ca5f0 con 0x7f296c0cb3b0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296b400640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c003970 0x7f296c152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f296aa00640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 0x7f296c15f1a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f295c00e540 tx=0x7f295c00ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c013610 con 0x7f296c0cb3b0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c013770 con 0x7f296c0cb3b0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c013b20 con 0x7f296c0cb3b0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c0ca8d0 con 0x7f296c0cb3b0 2024-01-15T18:12:24.215 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.210+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c0cae90 con 0x7f296c0cb3b0 2024-01-15T18:12:24.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.211+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c073c10 con 0x7f296c0cb3b0 2024-01-15T18:12:24.216 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.212+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f295c013c80 con 0x7f296c0cb3b0 2024-01-15T18:12:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.213+0000 7f2963400640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f294406c220 0x7f294406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:24.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.213+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f295c0881a0 con 0x7f296c0cb3b0 2024-01-15T18:12:24.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.214+0000 7f296b400640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f294406c220 0x7f294406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:24.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.215+0000 7f296b400640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f294406c220 0x7f294406e6e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2954034000 tx=0x7f2954009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:24.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.218+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f295c054a60 con 0x7f296c0cb3b0 2024-01-15T18:12:24.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:24.477+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f296c0686f0 con 0x7f296c0cb3b0 2024-01-15T18:12:25.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.055+0000 7f2963400640 1 -- 172.21.15.2:0/4260977546 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v677) v1 ==== 156+0+0 (secure 0 0 0) 0x7f295c058080 con 0x7f296c0cb3b0 2024-01-15T18:12:25.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f294406c220 msgr2=0x7f294406e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f294406c220 0x7f294406e6e0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2954034000 tx=0x7f2954009030 comp rx=0 tx=0).stop 2024-01-15T18:12:25.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 msgr2=0x7f296c15f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 0x7f296c15f1a0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f295c00e540 tx=0x7f295c00ea00 comp rx=0 tx=0).stop 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 shutdown_connections 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f294406c220 0x7f294406e6e0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f296c0b37e0 0x7f296c0b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f296c0cb3b0 0x7f296c15f1a0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 --2- 172.21.15.2:0/4260977546 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f294c003970 0x7f296c152e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 >> 172.21.15.2:0/4260977546 conn(0x7f296c072000 msgr2=0x7f296c0723e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.059+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 shutdown_connections 2024-01-15T18:12:25.064 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.060+0000 7f2972800640 1 -- 172.21.15.2:0/4260977546 wait complete. 2024-01-15T18:12:25.085 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:12:25.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.306+0000 7f1eb9e00640 1 Processor -- start 2024-01-15T18:12:25.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.307+0000 7f1eb9e00640 1 -- start start 2024-01-15T18:12:25.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.308+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:25.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.308+0000 7f1eb9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1eb40b3420 0x7f1eb40cc730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:25.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.308+0000 7f1eb9e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb415dd70 con 0x7f1eb40ccc70 2024-01-15T18:12:25.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.308+0000 7f1eb9e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40b3880 con 0x7f1eb40b3420 2024-01-15T18:12:25.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.308+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40711b0 con 0x7f1eb41439f0 2024-01-15T18:12:25.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1eb40b3420 0x7f1eb40cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:25.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1eb40b3420 0x7f1eb40cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53908/0 (socket says 172.21.15.2:53908) 2024-01-15T18:12:25.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 -- 172.21.15.2:0/1689160638 learned_addr learned my addr 172.21.15.2:0/1689160638 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:25.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1eb2a00640 1 --2- 172.21.15.2:0/1689160638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:25.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 -- 172.21.15.2:0/1689160638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 msgr2=0x7f1eb40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:25.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 --2- 172.21.15.2:0/1689160638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb40702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 -- 172.21.15.2:0/1689160638 >> v1:172.21.15.137:6789/0 conn(0x7f1eb40ccc70 legacy=0x7f1eb406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.309+0000 7f1ea9e00640 1 -- 172.21.15.2:0/1689160638 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4143dd0 con 0x7f1eb40b3420 2024-01-15T18:12:25.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.311+0000 7f1ea9e00640 1 --2- 172.21.15.2:0/1689160638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb40cc730 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00bb10 tx=0x7f1e9c02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4ee8e93ffee4d46 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.311+0000 7f1eb2000640 1 -- 172.21.15.2:0/1689160638 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c036440 con 0x7f1eb40b3420 2024-01-15T18:12:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.311+0000 7f1eb2000640 1 -- 172.21.15.2:0/1689160638 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e9c0365a0 con 0x7f1eb40b3420 2024-01-15T18:12:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.311+0000 7f1eb2000640 1 -- 172.21.15.2:0/1689160638 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c036930 con 0x7f1eb40b3420 2024-01-15T18:12:25.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 -- 172.21.15.2:0/1689160638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 msgr2=0x7f1eb40cc730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/1689160638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb40cc730 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00bb10 tx=0x7f1e9c02e150 comp rx=0 tx=0).stop 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 -- 172.21.15.2:0/1689160638 shutdown_connections 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/1689160638 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb40cc730 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/1689160638 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb40702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 -- 172.21.15.2:0/1689160638 >> 172.21.15.2:0/1689160638 conn(0x7f1eb40b45f0 msgr2=0x7f1eb40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 -- 172.21.15.2:0/1689160638 shutdown_connections 2024-01-15T18:12:25.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.312+0000 7f1eb9e00640 1 -- 172.21.15.2:0/1689160638 wait complete. 2024-01-15T18:12:25.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.314+0000 7f1eb9e00640 1 Processor -- start 2024-01-15T18:12:25.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.314+0000 7f1eb9e00640 1 -- start start 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb4141010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eb40ccc70 0x7f1eb40b0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb9e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb4159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40b0b70 con 0x7f1eb40b3420 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41422d0 con 0x7f1eb40ccc70 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb9e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb41416c0 con 0x7f1eb41439f0 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb4141010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb4141010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53910/0 (socket says 172.21.15.2:53910) 2024-01-15T18:12:25.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb2a00640 1 -- 172.21.15.2:0/3989449306 learned_addr learned my addr 172.21.15.2:0/3989449306 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1ea9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eb40ccc70 0x7f1eb40b0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb2a00640 1 -- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 msgr2=0x7f1eb4159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.315+0000 7f1eb3400640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb4159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb2a00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb4159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb2a00640 1 -- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eb40ccc70 msgr2=0x7f1eb40b0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb2a00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eb40ccc70 0x7f1eb40b0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb2a00640 1 -- 172.21.15.2:0/3989449306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c00b770 con 0x7f1eb40b3420 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1ea9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eb40ccc70 0x7f1eb40b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb2a00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb4141010 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400d800 tx=0x7f1ea400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea400b660 con 0x7f1eb40b3420 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea400b7c0 con 0x7f1eb40b3420 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea400bb50 con 0x7f1eb40b3420 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb41419a0 con 0x7f1eb40b3420 2024-01-15T18:12:25.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.316+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb4141f60 con 0x7f1eb40b3420 2024-01-15T18:12:25.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.318+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb4142120 con 0x7f1eb40b3420 2024-01-15T18:12:25.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.319+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f1ea400bcb0 con 0x7f1eb40b3420 2024-01-15T18:12:25.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.319+0000 7f1eabe00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e8c06c1f0 0x7f1e8c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:25.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.319+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7f1ea4088f10 con 0x7f1eb40b3420 2024-01-15T18:12:25.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.320+0000 7f1ea9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e8c06c1f0 0x7f1e8c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:25.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.321+0000 7f1ea9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e8c06c1f0 0x7f1e8c06e6b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c038000 tx=0x7f1e9c032000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:25.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.324+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ea40554e0 con 0x7f1eb40b3420 2024-01-15T18:12:25.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.520+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1eb40686f0 con 0x7f1eb40b3420 2024-01-15T18:12:25.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.527+0000 7f1eabe00640 1 -- 172.21.15.2:0/3989449306 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1ea4058b00 con 0x7f1eb40b3420 2024-01-15T18:12:25.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e8c06c1f0 msgr2=0x7f1e8c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e8c06c1f0 0x7f1e8c06e6b0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c038000 tx=0x7f1e9c032000 comp rx=0 tx=0).stop 2024-01-15T18:12:25.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 msgr2=0x7f1eb4141010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb4141010 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400d800 tx=0x7f1ea400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 shutdown_connections 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e8c06c1f0 0x7f1e8c06e6b0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1eb41439f0 0x7f1eb4159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1eb40ccc70 0x7f1eb40b0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 --2- 172.21.15.2:0/3989449306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1eb40b3420 0x7f1eb4141010 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 >> 172.21.15.2:0/3989449306 conn(0x7f1eb40b45f0 msgr2=0x7f1eb40cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 shutdown_connections 2024-01-15T18:12:25.535 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:25.530+0000 7f1eb9e00640 1 -- 172.21.15.2:0/3989449306 wait complete. 2024-01-15T18:12:25.555 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:12:25.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:12:25.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:12:25.790 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:12:25.790 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:12:25.790 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:25.850 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:12:25.851 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:12:25.851 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:25.905 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:12:25.905 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:12:25.905 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:25.964 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:12:25.965 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:12:25.965 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:26.021 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:12:26.021 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:12:26.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:12:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.253+0000 7fd0a0a00640 1 Processor -- start 2024-01-15T18:12:26.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.254+0000 7fd0a0a00640 1 -- start start 2024-01-15T18:12:26.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.255+0000 7fd0a0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 0x7fd09c06d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.255+0000 7fd0a0a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c0716d0 con 0x7fd09c15ec60 2024-01-15T18:12:26.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.255+0000 7fd0a0a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c15f570 con 0x7fd09c141880 2024-01-15T18:12:26.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.255+0000 7fd0a0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c071f20 con 0x7fd09c15f0e0 2024-01-15T18:12:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.256+0000 7fd09aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd09c15ec60 0x7fd09c06cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44822/0 (socket says 172.21.15.2:44822) 2024-01-15T18:12:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.256+0000 7fd09aa00640 1 -- 172.21.15.2:0/1824454489 learned_addr learned my addr 172.21.15.2:0/1824454489 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.256+0000 7fd09be00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 0x7fd09c06d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.256+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632781362 0 0) 0x7fd09c0716d0 con 0x7fd09c15ec60 2024-01-15T18:12:26.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.256+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd07c003520 con 0x7fd09c15ec60 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110182157 0 0) 0x7fd07c003520 con 0x7fd09c15ec60 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 msgr2=0x7fd09c06d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 0x7fd09c06d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 >> v1:172.21.15.137:6789/0 conn(0x7fd09c141880 legacy=0x7fd09c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c071960 con 0x7fd09c15ec60 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd084001670 con 0x7fd09c15ec60 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 >> v1:172.21.15.2:6789/0 conn(0x7fd09c15ec60 legacy=0x7fd09c06cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd07c0039a0 0x7fd07c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd07c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 0x7fd07c005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.262 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c071f20 con 0x7fd07c004350 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c00ba30 con 0x7fd07c0039a0 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c00bbf0 con 0x7fd09c15f0e0 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.257+0000 7fd09aa00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd07c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09b400640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd07c0039a0 0x7fd07c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09be00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 0x7fd07c005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09aa00640 1 -- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 msgr2=0x7fd07c005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09aa00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15f0e0 0x7fd07c005120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09aa00640 1 -- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd07c0039a0 msgr2=0x7fd07c003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09aa00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd07c0039a0 0x7fd07c003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.258+0000 7fd09aa00640 1 -- 172.21.15.2:0/1824454489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c003520 con 0x7fd07c004350 2024-01-15T18:12:26.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.259+0000 7fd09aa00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd07c004820 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd08400c210 tx=0x7fd07c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22ea3de6f563c0e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:26.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.259+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd084005ad0 con 0x7fd07c004350 2024-01-15T18:12:26.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.259+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd084005c50 con 0x7fd07c004350 2024-01-15T18:12:26.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.259+0000 7fd09a000640 1 -- 172.21.15.2:0/1824454489 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08402c8d0 con 0x7fd07c004350 2024-01-15T18:12:26.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd0a0a00640 1 -- 172.21.15.2:0/1824454489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 msgr2=0x7fd07c004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/1824454489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd07c004820 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fd08400c210 tx=0x7fd07c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd09be00640 1 -- 172.21.15.2:0/1824454489 reap_dead start 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd0a0a00640 1 -- 172.21.15.2:0/1824454489 shutdown_connections 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd0a0a00640 1 -- 172.21.15.2:0/1824454489 >> 172.21.15.2:0/1824454489 conn(0x7fd09c0b0840 msgr2=0x7fd09c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd0a0a00640 1 -- 172.21.15.2:0/1824454489 shutdown_connections 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.260+0000 7fd0a0a00640 1 -- 172.21.15.2:0/1824454489 wait complete. 2024-01-15T18:12:26.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.262+0000 7fd0a0a00640 1 Processor -- start 2024-01-15T18:12:26.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.262+0000 7fd0a0a00640 1 -- start start 2024-01-15T18:12:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd0a0a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd09c1f48c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd0a0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15c540 0x7fd09c1f09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd0a0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd09c13e470 0x7fd09c1400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd0a0a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c06c6b0 con 0x7fd07c004350 2024-01-15T18:12:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd0a0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1f1310 con 0x7fd09c13e470 2024-01-15T18:12:26.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd0a0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1f15f0 con 0x7fd09c15c540 2024-01-15T18:12:26.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd09b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd09c1f48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd09b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd09c1f48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53930/0 (socket says 172.21.15.2:53930) 2024-01-15T18:12:26.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.263+0000 7fd09b400640 1 -- 172.21.15.2:0/3606400634 learned_addr learned my addr 172.21.15.2:0/3606400634 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09be00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd09c13e470 0x7fd09c1400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09b400640 1 -- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15c540 msgr2=0x7fd09c1f09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09aa00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15c540 0x7fd09c1f09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09b400640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15c540 0x7fd09c1f09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09b400640 1 -- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd09c13e470 msgr2=0x7fd09c1400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09b400640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd09c13e470 0x7fd09c1400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09b400640 1 -- 172.21.15.2:0/3606400634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c1f40c0 con 0x7fd07c004350 2024-01-15T18:12:26.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09aa00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15c540 0x7fd09c1f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09be00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd09c13e470 0x7fd09c1400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd09b400640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd09c1f48c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00e5a0 tx=0x7fd08c00ea60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c0052e0 con 0x7fd07c004350 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c005440 con 0x7fd07c004350 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c003810 con 0x7fd07c004350 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.264+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c1f43a0 con 0x7fd07c004350 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.265+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c1f4e00 con 0x7fd07c004350 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.266+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c074770 con 0x7fd07c004350 2024-01-15T18:12:26.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.267+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fd08c003970 con 0x7fd07c004350 2024-01-15T18:12:26.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.268+0000 7fd093400640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd07406c1f0 0x7fd07406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.268+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fd08c088730 con 0x7fd07c004350 2024-01-15T18:12:26.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.268+0000 7fd09aa00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd07406c1f0 0x7fd07406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.270+0000 7fd09aa00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd07406c1f0 0x7fd07406e6b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd084034000 tx=0x7fd084009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:26.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.272+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd08c054d60 con 0x7fd07c004350 2024-01-15T18:12:26.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.504+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd09c06e1c0 con 0x7fd07c004350 2024-01-15T18:12:26.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.505+0000 7fd093400640 1 -- 172.21.15.2:0/3606400634 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fd08c058380 con 0x7fd07c004350 2024-01-15T18:12:26.509 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:26.509 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/2471268746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":2471268746},{"type":"v1","addr":"172.21.15.137:6836","nonce":2471268746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6550,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/1012168762","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":1012168762},{"type":"v1","addr":"172.21.15.137:6837","nonce":1012168762}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6559,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6833/2801413765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2801413765},{"type":"v1","addr":"172.21.15.2:6833","nonce":2801413765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8457,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/2721206574","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6834","nonce":2721206574},{"type":"v1","addr":"172.21.15.2:6835","nonce":2721206574}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[]} 2024-01-15T18:12:26.510 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 59 2024-01-15T18:12:26.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd07406c1f0 msgr2=0x7fd07406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd07406c1f0 0x7fd07406e6b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd084034000 tx=0x7fd084009030 comp rx=0 tx=0).stop 2024-01-15T18:12:26.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 msgr2=0x7fd09c1f48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd09c1f48c0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7fd08c00e5a0 tx=0x7fd08c00ea60 comp rx=0 tx=0).stop 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 shutdown_connections 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd07406c1f0 0x7fd07406e6b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd09c13e470 0x7fd09c1400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd09c15c540 0x7fd09c1f09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 --2- 172.21.15.2:0/3606400634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd07c004350 0x7fd09c1f48c0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 >> 172.21.15.2:0/3606400634 conn(0x7fd09c0b0840 msgr2=0x7fd09c0cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 shutdown_connections 2024-01-15T18:12:26.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.508+0000 7fd0a0a00640 1 -- 172.21.15.2:0/3606400634 wait complete. 2024-01-15T18:12:26.531 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:12:26.532 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:12:26.532 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:12:26.532 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:12:26.532 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:12:26.532 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:12:26.532 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:12:26.532 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:12:26.533 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:12:26.533 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:12:26.533 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:12:26.533 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:12:26.533 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:12:26.533 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:12:26.533 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:12:26.533 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:12:26.550 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:12:26.550 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:12:26.551 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:12:26.551 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:12:26.568 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:12:26.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:12:26.571 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:12:26.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:12:26.599 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:12:26.599 DEBUG:teuthology.parallel:result is None 2024-01-15T18:12:26.605 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:12:26.605 DEBUG:teuthology.parallel:result is None 2024-01-15T18:12:26.618 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:12:26.618 DEBUG:teuthology.parallel:result is None 2024-01-15T18:12:26.625 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:12:26.625 DEBUG:teuthology.parallel:result is None 2024-01-15T18:12:26.625 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:12:26.756 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:12:26.756 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:12:26.788 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:12:26.788 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T18:12:26.790 INFO:tasks.ceph.mds.d.smithi137.stderr:2024-01-15T18:12:26.786+0000 7f291d800640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-01-15T18:12:26.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.813+0000 7fb17fe00640 1 Processor -- start 2024-01-15T18:12:26.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.813+0000 7fb17fe00640 1 -- start start 2024-01-15T18:12:26.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.814+0000 7fb17fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb1800d1dc0 0x7fb1800cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17fe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800cf160 con 0x7fb1801609f0 2024-01-15T18:12:26.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1800ceba0 con 0x7fb1800ce390 2024-01-15T18:12:26.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1800c71c0 con 0x7fb1800d1dc0 2024-01-15T18:12:26.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17ea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb1801609f0 0x7fb1800d16b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44840/0 (socket says 172.21.15.2:44840) 2024-01-15T18:12:26.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17ea00640 1 -- 172.21.15.2:0/1874600335 learned_addr learned my addr 172.21.15.2:0/1874600335 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456375617 0 0) 0x7fb1800cf160 con 0x7fb1801609f0 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb160003520 con 0x7fb1801609f0 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.815+0000 7fb177e00640 1 --2- 172.21.15.2:0/1874600335 >> v2:172.21.15.137:3300/0 conn(0x7fb1800d1dc0 0x7fb1800cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892466540 0 0) 0x7fb160003520 con 0x7fb1801609f0 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 >> v2:172.21.15.137:3300/0 conn(0x7fb1800d1dc0 msgr2=0x7fb1800cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 --2- 172.21.15.2:0/1874600335 >> v2:172.21.15.137:3300/0 conn(0x7fb1800d1dc0 0x7fb1800cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 >> v1:172.21.15.137:6789/0 conn(0x7fb1800ce390 legacy=0x7fb1800c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1800cee80 con 0x7fb1801609f0 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb170001670 con 0x7fb1801609f0 2024-01-15T18:12:26.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 >> v1:172.21.15.2:6789/0 conn(0x7fb1801609f0 legacy=0x7fb1800d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb160003970 0x7fb160003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1600047c0 0x7fb160004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb160005380 0x7fb16000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1800c71c0 con 0x7fb1600047c0 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16000c210 con 0x7fb160003970 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16000c3d0 con 0x7fb160005380 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb177e00640 1 --2- 172.21.15.2:0/1874600335 >> v2:172.21.15.137:3300/0 conn(0x7fb1800d1dc0 0x7fb1800cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17ea00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb160003970 0x7fb160003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb177e00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1600047c0 0x7fb160004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.816+0000 7fb17f400640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb160005380 0x7fb16000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17ea00640 1 -- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb160005380 msgr2=0x7fb16000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17ea00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb160005380 0x7fb16000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17ea00640 1 -- 172.21.15.2:0/1874600335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1600047c0 msgr2=0x7fb160004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17ea00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb1600047c0 0x7fb160004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17ea00640 1 -- 172.21.15.2:0/1874600335 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160003520 con 0x7fb160003970 2024-01-15T18:12:26.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17f400640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb160005380 0x7fb16000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:26.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.817+0000 7fb17f400640 1 -- 172.21.15.2:0/1874600335 reap_dead start 2024-01-15T18:12:26.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.818+0000 7fb17ea00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb160003970 0x7fb160003d90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb17000c240 tx=0x7fb1600057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2bfb6a705f9274 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.818+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170005ad0 con 0x7fb160003970 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.818+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170005c30 con 0x7fb160003970 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.818+0000 7fb17e000640 1 -- 172.21.15.2:0/1874600335 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17002caf0 con 0x7fb160003970 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 -- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb160003970 msgr2=0x7fb160003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb160003970 0x7fb160003d90 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb17000c240 tx=0x7fb1600057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 -- 172.21.15.2:0/1874600335 shutdown_connections 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 --2- 172.21.15.2:0/1874600335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb160003970 0x7fb160003d90 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 -- 172.21.15.2:0/1874600335 >> 172.21.15.2:0/1874600335 conn(0x7fb18009c3d0 msgr2=0x7fb1800d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 -- 172.21.15.2:0/1874600335 shutdown_connections 2024-01-15T18:12:26.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.819+0000 7fb17fe00640 1 -- 172.21.15.2:0/1874600335 wait complete. 2024-01-15T18:12:26.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.821+0000 7fb17fe00640 1 Processor -- start 2024-01-15T18:12:26.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.821+0000 7fb17fe00640 1 -- start start 2024-01-15T18:12:26.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 0x7fb180157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1800cb5b0 0x7fb18015a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18015ac40 0x7fb1801550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1801f6520 con 0x7fb160003970 2024-01-15T18:12:26.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1801f67c0 con 0x7fb18015ac40 2024-01-15T18:12:26.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1801f6aa0 con 0x7fb1800cb5b0 2024-01-15T18:12:26.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 0x7fb180157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 0x7fb180157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53980/0 (socket says 172.21.15.2:53980) 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17ea00640 1 -- 172.21.15.2:0/3823621256 learned_addr learned my addr 172.21.15.2:0/3823621256 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb17f400640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18015ac40 0x7fb1801550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.822+0000 7fb177e00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1800cb5b0 0x7fb18015a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17ea00640 1 -- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1800cb5b0 msgr2=0x7fb18015a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17ea00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1800cb5b0 0x7fb18015a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17ea00640 1 -- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18015ac40 msgr2=0x7fb1801550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17ea00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18015ac40 0x7fb1801550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17ea00640 1 -- 172.21.15.2:0/3823621256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1801f6d80 con 0x7fb160003970 2024-01-15T18:12:26.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17ea00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 0x7fb180157930 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fb17002c450 tx=0x7fb17002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:26.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17003c040 con 0x7fb160003970 2024-01-15T18:12:26.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb170000fe0 con 0x7fb160003970 2024-01-15T18:12:26.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb170007070 con 0x7fb160003970 2024-01-15T18:12:26.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.823+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1801f7060 con 0x7fb160003970 2024-01-15T18:12:26.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.824+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1800cb9f0 con 0x7fb160003970 2024-01-15T18:12:26.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.825+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1801f6be0 con 0x7fb160003970 2024-01-15T18:12:26.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.826+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fb170032070 con 0x7fb160003970 2024-01-15T18:12:26.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.826+0000 7fb176a00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb15806c1f0 0x7fb15806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:26.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.827+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5055+0+0 (secure 0 0 0) 0x7fb1700a95e0 con 0x7fb160003970 2024-01-15T18:12:26.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.827+0000 7fb177e00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb15806c1f0 0x7fb15806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:26.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.827+0000 7fb177e00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb15806c1f0 0x7fb15806e6b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb1801f73f0 tx=0x7fb168008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:26.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:26.830+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1801f6be0 con 0x7fb160003970 2024-01-15T18:12:27.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:27.039+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb1801f6be0 con 0x7fb160003970 2024-01-15T18:12:28.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.092+0000 7fb176a00640 1 -- 172.21.15.2:0/3823621256 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v99) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb1801f6be0 con 0x7fb160003970 2024-01-15T18:12:28.097 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb15806c1f0 msgr2=0x7fb15806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb15806c1f0 0x7fb15806e6b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb1801f73f0 tx=0x7fb168008040 comp rx=0 tx=0).stop 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 msgr2=0x7fb180157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 0x7fb180157930 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fb17002c450 tx=0x7fb17002e070 comp rx=0 tx=0).stop 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 shutdown_connections 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb15806c1f0 0x7fb15806e6b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb18015ac40 0x7fb1801550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb1800cb5b0 0x7fb18015a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 --2- 172.21.15.2:0/3823621256 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb160003970 0x7fb180157930 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.096+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 >> 172.21.15.2:0/3823621256 conn(0x7fb18009c3d0 msgr2=0x7fb1800c83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.097+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 shutdown_connections 2024-01-15T18:12:28.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.097+0000 7fb17fe00640 1 -- 172.21.15.2:0/3823621256 wait complete. 2024-01-15T18:12:28.121 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:12:28.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.335+0000 7fb677e00640 1 Processor -- start 2024-01-15T18:12:28.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.335+0000 7fb677e00640 1 -- start start 2024-01-15T18:12:28.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.337+0000 7fb677e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb67806da40 0x7fb678071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.337+0000 7fb677e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67806c170 con 0x7fb67815eb30 2024-01-15T18:12:28.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.337+0000 7fb677e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6780cd1b0 con 0x7fb67815efc0 2024-01-15T18:12:28.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.337+0000 7fb677e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb678071560 con 0x7fb67806da40 2024-01-15T18:12:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.337+0000 7fb66de00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb67815eb30 0x7fb67815b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44850/0 (socket says 172.21.15.2:44850) 2024-01-15T18:12:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.337+0000 7fb66de00640 1 -- 172.21.15.2:0/2986311280 learned_addr learned my addr 172.21.15.2:0/2986311280 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676a00640 1 --2- 172.21.15.2:0/2986311280 >> v2:172.21.15.137:3300/0 conn(0x7fb67806da40 0x7fb678071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592617631 0 0) 0x7fb67806c170 con 0x7fb67815eb30 2024-01-15T18:12:28.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb658003520 con 0x7fb67815eb30 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1388965696 0 0) 0x7fb658003520 con 0x7fb67815eb30 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 >> v2:172.21.15.137:3300/0 conn(0x7fb67806da40 msgr2=0x7fb678071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 --2- 172.21.15.2:0/2986311280 >> v2:172.21.15.137:3300/0 conn(0x7fb67806da40 0x7fb678071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 >> v1:172.21.15.137:6789/0 conn(0x7fb67815efc0 legacy=0x7fb67815bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6780cd3e0 con 0x7fb67815eb30 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb660001670 con 0x7fb67815eb30 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 >> v1:172.21.15.2:6789/0 conn(0x7fb67815eb30 legacy=0x7fb67815b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.338+0000 7fb676000640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6580039c0 0x7fb658003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb676000640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6580047c0 0x7fb658004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb676000640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb658005180 0x7fb65800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678071560 con 0x7fb6580039c0 2024-01-15T18:12:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65800c080 con 0x7fb658005180 2024-01-15T18:12:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb65800c240 con 0x7fb6580047c0 2024-01-15T18:12:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb676a00640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6580039c0 0x7fb658003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb677400640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb658005180 0x7fb65800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.339+0000 7fb66de00640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6580047c0 0x7fb658004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.340+0000 7fb677400640 1 -- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6580047c0 msgr2=0x7fb658004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.340+0000 7fb677400640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6580047c0 0x7fb658004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.340+0000 7fb677400640 1 -- 172.21.15.2:0/2986311280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6580039c0 msgr2=0x7fb658003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.340+0000 7fb677400640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6580039c0 0x7fb658003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.340+0000 7fb677400640 1 -- 172.21.15.2:0/2986311280 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb658003520 con 0x7fb658005180 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.340+0000 7fb676a00640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6580039c0 0x7fb658003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.341+0000 7fb677400640 1 -- 172.21.15.2:0/2986311280 reap_dead start 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.341+0000 7fb677400640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb658005180 0x7fb65800b980 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb670001e50 tx=0x7fb6580057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=739fc842afaba991 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:28.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670003b70 con 0x7fb658005180 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb670003cf0 con 0x7fb658005180 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb676000640 1 -- 172.21.15.2:0/2986311280 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670005310 con 0x7fb658005180 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb677e00640 1 -- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb658005180 msgr2=0x7fb65800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb677e00640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb658005180 0x7fb65800b980 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb670001e50 tx=0x7fb6580057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb677e00640 1 -- 172.21.15.2:0/2986311280 shutdown_connections 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb677e00640 1 --2- 172.21.15.2:0/2986311280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb658005180 0x7fb65800b980 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.342+0000 7fb677e00640 1 -- 172.21.15.2:0/2986311280 >> 172.21.15.2:0/2986311280 conn(0x7fb6780b0840 msgr2=0x7fb6780b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:28.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.343+0000 7fb677e00640 1 -- 172.21.15.2:0/2986311280 shutdown_connections 2024-01-15T18:12:28.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.343+0000 7fb677e00640 1 -- 172.21.15.2:0/2986311280 wait complete. 2024-01-15T18:12:28.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.344+0000 7fb677e00640 1 Processor -- start 2024-01-15T18:12:28.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.345+0000 7fb677e00640 1 -- start start 2024-01-15T18:12:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 0x7fb67814d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb678156480 0x7fb678156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6780b1d20 0x7fb6780b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678156e40 con 0x7fb658005180 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6780b26f0 con 0x7fb6780b1d20 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6780b2920 con 0x7fb678156480 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 0x7fb67814d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 0x7fb67814d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54000/0 (socket says 172.21.15.2:54000) 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 -- 172.21.15.2:0/3457072506 learned_addr learned my addr 172.21.15.2:0/3457072506 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677400640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6780b1d20 0x7fb6780b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 -- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb678156480 msgr2=0x7fb678156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb678156480 0x7fb678156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 -- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6780b1d20 msgr2=0x7fb6780b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6780b1d20 0x7fb6780b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb676a00640 1 -- 172.21.15.2:0/3457072506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6780b2c00 con 0x7fb658005180 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.346+0000 7fb677400640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6780b1d20 0x7fb6780b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:28.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.347+0000 7fb676a00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 0x7fb67814d8a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb668011a70 tx=0x7fb668011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.347+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66800fbb0 con 0x7fb658005180 2024-01-15T18:12:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.347+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66800fd10 con 0x7fb658005180 2024-01-15T18:12:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.347+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668014640 con 0x7fb658005180 2024-01-15T18:12:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.347+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6780b2ee0 con 0x7fb658005180 2024-01-15T18:12:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.347+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6780b34a0 con 0x7fb658005180 2024-01-15T18:12:28.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.349+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb678074770 con 0x7fb658005180 2024-01-15T18:12:28.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.350+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fb6680147a0 con 0x7fb658005180 2024-01-15T18:12:28.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.351+0000 7fb66f400640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65006c1f0 0x7fb65006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.351+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4969+0+0 (secure 0 0 0) 0x7fb6680890b0 con 0x7fb658005180 2024-01-15T18:12:28.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.351+0000 7fb66de00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65006c1f0 0x7fb65006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.353+0000 7fb66de00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65006c1f0 0x7fb65006e6b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb660001200 tx=0x7fb660001170 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:28.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.356+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb668055a20 con 0x7fb658005180 2024-01-15T18:12:28.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.617+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb67806ded0 con 0x7fb658005180 2024-01-15T18:12:28.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.618+0000 7fb66f400640 1 -- 172.21.15.2:0/3457072506 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb668022020 con 0x7fb658005180 2024-01-15T18:12:28.622 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:28.622 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:28.622 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T18:12:28.622 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.623 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:12:28.624 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.625 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.626 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.627 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.628 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.629 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.630 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow r", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.631 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:12:28.632 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.633 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.634 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:28.635 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T18:12:28.635 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65006c1f0 msgr2=0x7fb65006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65006c1f0 0x7fb65006e6b0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fb660001200 tx=0x7fb660001170 comp rx=0 tx=0).stop 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 msgr2=0x7fb67814d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 0x7fb67814d8a0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fb668011a70 tx=0x7fb668011f30 comp rx=0 tx=0).stop 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 shutdown_connections 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb65006c1f0 0x7fb65006e6b0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6780b1d20 0x7fb6780b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb678156480 0x7fb678156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 --2- 172.21.15.2:0/3457072506 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb658005180 0x7fb67814d8a0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 >> 172.21.15.2:0/3457072506 conn(0x7fb6780b0840 msgr2=0x7fb678159aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 shutdown_connections 2024-01-15T18:12:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.621+0000 7fb677e00640 1 -- 172.21.15.2:0/3457072506 wait complete. 2024-01-15T18:12:28.646 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:28.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.856+0000 7f2e35e00640 1 Processor -- start 2024-01-15T18:12:28.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.856+0000 7f2e35e00640 1 -- start start 2024-01-15T18:12:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e35e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e3015d340 0x7f2e300cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e35e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e35e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e300d3a30 con 0x7f2e300d1680 2024-01-15T18:12:28.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e35e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e300b3de0 con 0x7f2e3015d340 2024-01-15T18:12:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e35e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300b40c0 con 0x7f2e300d3090 2024-01-15T18:12:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e34a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e3015d340 0x7f2e300cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.858+0000 7f2e34a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e3015d340 0x7f2e300cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54004/0 (socket says 172.21.15.2:54004) 2024-01-15T18:12:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.859+0000 7f2e34a00640 1 -- 172.21.15.2:0/307216292 learned_addr learned my addr 172.21.15.2:0/307216292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:28.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.859+0000 7f2e34a00640 1 -- 172.21.15.2:0/307216292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 msgr2=0x7f2e300d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.859+0000 7f2e27e00640 1 --2- 172.21.15.2:0/307216292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.859+0000 7f2e34a00640 1 --2- 172.21.15.2:0/307216292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300d34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.859+0000 7f2e34a00640 1 -- 172.21.15.2:0/307216292 >> v1:172.21.15.137:6789/0 conn(0x7f2e300d1680 legacy=0x7f2e300d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:28.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.859+0000 7f2e34a00640 1 -- 172.21.15.2:0/307216292 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e300d0e80 con 0x7f2e3015d340 2024-01-15T18:12:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.860+0000 7f2e34a00640 1 --2- 172.21.15.2:0/307216292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e300cdd20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009870 tx=0x7f2e2002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d45a02b97173066 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.860+0000 7f2e2fe00640 1 -- 172.21.15.2:0/307216292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20031640 con 0x7f2e3015d340 2024-01-15T18:12:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e2fe00640 1 -- 172.21.15.2:0/307216292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e200317a0 con 0x7f2e3015d340 2024-01-15T18:12:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e2fe00640 1 -- 172.21.15.2:0/307216292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e20031b30 con 0x7f2e3015d340 2024-01-15T18:12:28.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 -- 172.21.15.2:0/307216292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 msgr2=0x7f2e300cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 --2- 172.21.15.2:0/307216292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e300cdd20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f2e20009870 tx=0x7f2e2002b8f0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 -- 172.21.15.2:0/307216292 shutdown_connections 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 --2- 172.21.15.2:0/307216292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 --2- 172.21.15.2:0/307216292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e300cdd20 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 -- 172.21.15.2:0/307216292 >> 172.21.15.2:0/307216292 conn(0x7f2e3008caf0 msgr2=0x7f2e3009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 -- 172.21.15.2:0/307216292 shutdown_connections 2024-01-15T18:12:28.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.861+0000 7f2e35e00640 1 -- 172.21.15.2:0/307216292 wait complete. 2024-01-15T18:12:28.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.863+0000 7f2e35e00640 1 Processor -- start 2024-01-15T18:12:28.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.863+0000 7f2e35e00640 1 -- start start 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.864+0000 7f2e35e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e300d1680 0x7f2e301483f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.864+0000 7f2e35e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.864+0000 7f2e35e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e301561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.864+0000 7f2e35e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300b8470 con 0x7f2e3015d340 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.864+0000 7f2e35e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300b6640 con 0x7f2e300d1680 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.864+0000 7f2e35e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e300b6e20 con 0x7f2e300d3090 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e301561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e301561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54012/0 (socket says 172.21.15.2:54012) 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 -- 172.21.15.2:0/1169558437 learned_addr learned my addr 172.21.15.2:0/1169558437 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e27e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e34a00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e300d1680 0x7f2e301483f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 -- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 msgr2=0x7f2e300afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300afdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 -- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e300d1680 msgr2=0x7f2e301483f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e300d1680 0x7f2e301483f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:28.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 -- 172.21.15.2:0/1169558437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e200094d0 con 0x7f2e3015d340 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.865+0000 7f2e35400640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e301561f0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800b970 tx=0x7f2e2800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.866+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800cb40 con 0x7f2e3015d340 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.866+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2800cca0 con 0x7f2e3015d340 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.866+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28014680 con 0x7f2e3015d340 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.866+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e300b8710 con 0x7f2e3015d340 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.866+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e300b6900 con 0x7f2e3015d340 2024-01-15T18:12:28.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.867+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e300b6af0 con 0x7f2e3015d340 2024-01-15T18:12:28.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.868+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f2e2801d3b0 con 0x7f2e3015d340 2024-01-15T18:12:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.869+0000 7f2e2d600640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e0806c1f0 0x7f2e0806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.869+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f2e28088620 con 0x7f2e3015d340 2024-01-15T18:12:28.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.870+0000 7f2e34a00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e0806c1f0 0x7f2e0806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:28.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.871+0000 7f2e34a00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e0806c1f0 0x7f2e0806e6b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2e20036000 tx=0x7f2e2003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:28.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:28.873+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e28054de0 con 0x7f2e3015d340 2024-01-15T18:12:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.068+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e30074770 con 0x7f2e3015d340 2024-01-15T18:12:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.070+0000 7f2e2d600640 1 -- 172.21.15.2:0/1169558437 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+12974 (secure 0 0 0) 0x7f2e28088e60 con 0x7f2e3015d340 2024-01-15T18:12:29.074 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:29.074 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":99,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:28.090612+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.072+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e0806c1f0 msgr2=0x7f2e0806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.072+0000 7f2e35e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e0806c1f0 0x7f2e0806e6b0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f2e20036000 tx=0x7f2e2003c040 comp rx=0 tx=0).stop 2024-01-15T18:12:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.072+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 msgr2=0x7f2e301561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.072+0000 7f2e35e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e301561f0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800b970 tx=0x7f2e2800be30 comp rx=0 tx=0).stop 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 shutdown_connections 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e0806c1f0 0x7f2e0806e6b0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e3015d340 0x7f2e301561f0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e300d3090 0x7f2e300afdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 --2- 172.21.15.2:0/1169558437 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e300d1680 0x7f2e301483f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 >> 172.21.15.2:0/1169558437 conn(0x7f2e3008caf0 msgr2=0x7f2e3006b470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 shutdown_connections 2024-01-15T18:12:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.073+0000 7f2e35e00640 1 -- 172.21.15.2:0/1169558437 wait complete. 2024-01-15T18:12:29.096 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:12:29.096 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:12:29.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.319+0000 7fd6d4800640 1 Processor -- start 2024-01-15T18:12:29.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.320+0000 7fd6d4800640 1 -- start start 2024-01-15T18:12:29.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.321+0000 7fd6d4800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6cc165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.321+0000 7fd6d4800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0ce230 con 0x7fd6cc144c00 2024-01-15T18:12:29.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.321+0000 7fd6d4800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6cc0ce510 con 0x7fd6cc15c450 2024-01-15T18:12:29.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.321+0000 7fd6d4800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0ce7f0 con 0x7fd6cc06d000 2024-01-15T18:12:29.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.322+0000 7fd6d1e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd6cc144c00 0x7fd6cc0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44852/0 (socket says 172.21.15.2:44852) 2024-01-15T18:12:29.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.322+0000 7fd6d1e00640 1 -- 172.21.15.2:0/4202531157 learned_addr learned my addr 172.21.15.2:0/4202531157 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:29.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.322+0000 7fd6d2800640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6cc165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.322+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1704086135 0 0) 0x7fd6cc0ce230 con 0x7fd6cc144c00 2024-01-15T18:12:29.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.322+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6ac003520 con 0x7fd6cc144c00 2024-01-15T18:12:29.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2294837775 0 0) 0x7fd6ac003520 con 0x7fd6cc144c00 2024-01-15T18:12:29.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 msgr2=0x7fd6cc165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6cc165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 >> v1:172.21.15.137:6789/0 conn(0x7fd6cc15c450 legacy=0x7fd6cc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc0705f0 con 0x7fd6cc144c00 2024-01-15T18:12:29.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd6bc001670 con 0x7fd6cc144c00 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 >> v1:172.21.15.2:6789/0 conn(0x7fd6cc144c00 legacy=0x7fd6cc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6ac0039a0 0x7fd6ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac004d00 0x7fd6ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc0ce7f0 con 0x7fd6ac0039a0 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac00ba60 con 0x7fd6ac004d00 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac00bc20 con 0x7fd6cc06d000 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6ac0039a0 0x7fd6ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.323+0000 7fd6d1400640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d1e00640 1 -- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 msgr2=0x7fd6ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d2800640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac004d00 0x7fd6ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d1e00640 1 -- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac004d00 msgr2=0x7fd6ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac004d00 0x7fd6ac005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d1e00640 1 -- 172.21.15.2:0/4202531157 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac003520 con 0x7fd6ac0039a0 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.324+0000 7fd6d1400640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc06d000 0x7fd6ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.325+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6ac0039a0 0x7fd6ac003dc0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00c210 tx=0x7fd6ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33f73e5c0a61d332 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.325+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc005ad0 con 0x7fd6ac0039a0 2024-01-15T18:12:29.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.325+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6bc005c50 con 0x7fd6ac0039a0 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.325+0000 7fd6d0a00640 1 -- 172.21.15.2:0/4202531157 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc02c890 con 0x7fd6ac0039a0 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d4800640 1 -- 172.21.15.2:0/4202531157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6ac0039a0 msgr2=0x7fd6ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d4800640 1 --2- 172.21.15.2:0/4202531157 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6ac0039a0 0x7fd6ac003dc0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc00c210 tx=0x7fd6ac0057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d2800640 1 -- 172.21.15.2:0/4202531157 reap_dead start 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d4800640 1 -- 172.21.15.2:0/4202531157 shutdown_connections 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d4800640 1 -- 172.21.15.2:0/4202531157 >> 172.21.15.2:0/4202531157 conn(0x7fd6cc0b45f0 msgr2=0x7fd6cc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d4800640 1 -- 172.21.15.2:0/4202531157 shutdown_connections 2024-01-15T18:12:29.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.326+0000 7fd6d4800640 1 -- 172.21.15.2:0/4202531157 wait complete. 2024-01-15T18:12:29.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.328+0000 7fd6d4800640 1 Processor -- start 2024-01-15T18:12:29.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.328+0000 7fd6d4800640 1 -- start start 2024-01-15T18:12:29.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d4800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac0039a0 0x7fd6cc13c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d4800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc13cc40 0x7fd6cc1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d4800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 0x7fd6cc1f4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d4800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc13d0c0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d4800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1f18b0 con 0x7fd6ac0039a0 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d4800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6cc1f1ae0 con 0x7fd6cc13cc40 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 0x7fd6cc1f4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 0x7fd6cc1f4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54030/0 (socket says 172.21.15.2:54030) 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.329+0000 7fd6d2800640 1 -- 172.21.15.2:0/1401952071 learned_addr learned my addr 172.21.15.2:0/1401952071 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d1400640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc13cc40 0x7fd6cc1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac0039a0 0x7fd6cc13c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d2800640 1 -- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc13cc40 msgr2=0x7fd6cc1f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d2800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc13cc40 0x7fd6cc1f3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d2800640 1 -- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac0039a0 msgr2=0x7fd6cc13c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d2800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac0039a0 0x7fd6cc13c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d2800640 1 -- 172.21.15.2:0/1401952071 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6cc13d2e0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac0039a0 0x7fd6cc13c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d2800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 0x7fd6cc1f4980 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0cca20 tx=0x7fd6c400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4017400 con 0x7fd6cc1f4500 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c4011040 con 0x7fd6cc1f4500 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c40177c0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.330+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6cc13d5c0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.331+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6cc1f1d50 con 0x7fd6cc1f4500 2024-01-15T18:12:29.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.332+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6cc1f1f70 con 0x7fd6cc1f4500 2024-01-15T18:12:29.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.333+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fd6c4021440 con 0x7fd6cc1f4500 2024-01-15T18:12:29.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.334+0000 7fd6ca000640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6a406c220 0x7fd6a406e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.334+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4969+0+0 (secure 0 0 0) 0x7fd6c40887e0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.334+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6a406c220 0x7fd6a406e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.335+0000 7fd6d1e00640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6a406c220 0x7fd6a406e6e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc034000 tx=0x7fd6bc009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:29.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.338+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6c40550a0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.536 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.532+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6cc0686f0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.537 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.533+0000 7fd6ca000640 1 -- 172.21.15.2:0/1401952071 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6c40586c0 con 0x7fd6cc1f4500 2024-01-15T18:12:29.538 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6a406c220 msgr2=0x7fd6a406e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6a406c220 0x7fd6a406e6e0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc034000 tx=0x7fd6bc009030 comp rx=0 tx=0).stop 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 msgr2=0x7fd6cc1f4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 0x7fd6cc1f4980 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc0cca20 tx=0x7fd6c400f4a0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 shutdown_connections 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6a406c220 0x7fd6a406e6e0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6cc1f4500 0x7fd6cc1f4980 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6cc13cc40 0x7fd6cc1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 --2- 172.21.15.2:0/1401952071 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6ac0039a0 0x7fd6cc13c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 >> 172.21.15.2:0/1401952071 conn(0x7fd6cc0b45f0 msgr2=0x7fd6cc143e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 shutdown_connections 2024-01-15T18:12:29.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.536+0000 7fd6d4800640 1 -- 172.21.15.2:0/1401952071 wait complete. 2024-01-15T18:12:29.561 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:12:29.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.784+0000 7fc2f6000640 1 Processor -- start 2024-01-15T18:12:29.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.785+0000 7fc2f6000640 1 -- start start 2024-01-15T18:12:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2f6000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2f01461c0 0x7fc2f0070820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2f6000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 0x7fc2f00b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2f6000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f00b38c0 con 0x7fc2f00b3470 2024-01-15T18:12:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2f6000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f0161040 con 0x7fc2f01461c0 2024-01-15T18:12:29.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2f6000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0161320 con 0x7fc2f0146cb0 2024-01-15T18:12:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2efe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc2f00b3470 0x7fc2f006e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44868/0 (socket says 172.21.15.2:44868) 2024-01-15T18:12:29.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.786+0000 7fc2efe00640 1 -- 172.21.15.2:0/576489241 learned_addr learned my addr 172.21.15.2:0/576489241 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:29.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ef400640 1 --2- 172.21.15.2:0/576489241 >> v2:172.21.15.137:3300/0 conn(0x7fc2f01461c0 0x7fc2f0070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2eea00640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 0x7fc2f00b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943687167 0 0) 0x7fc2e8001110 con 0x7fc2f00b3470 2024-01-15T18:12:29.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2d0003520 con 0x7fc2f00b3470 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 581004947 0 0) 0x7fc2d0003520 con 0x7fc2f00b3470 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 msgr2=0x7fc2f00b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 0x7fc2f00b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 >> v2:172.21.15.137:3300/0 conn(0x7fc2f01461c0 msgr2=0x7fc2f0070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 --2- 172.21.15.2:0/576489241 >> v2:172.21.15.137:3300/0 conn(0x7fc2f01461c0 0x7fc2f0070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f0161600 con 0x7fc2f00b3470 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc2e8001800 con 0x7fc2f00b3470 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.787+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 >> v1:172.21.15.2:6789/0 conn(0x7fc2f00b3470 legacy=0x7fc2f006e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ee000640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2d0003970 0x7fc2d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ee000640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 0x7fc2d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ee000640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0161040 con 0x7fc2d0004d00 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f0161320 con 0x7fc2d0003970 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d000b9b0 con 0x7fc2f0146cb0 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2efe00640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2ef400640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2d0003970 0x7fc2d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2eea00640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 0x7fc2d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2efe00640 1 -- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 msgr2=0x7fc2d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2efe00640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2f0146cb0 0x7fc2d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2efe00640 1 -- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2d0003970 msgr2=0x7fc2d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2efe00640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2d0003970 0x7fc2d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.788+0000 7fc2efe00640 1 -- 172.21.15.2:0/576489241 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d0003520 con 0x7fc2d0004d00 2024-01-15T18:12:29.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2efe00640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c110 tx=0x7fc2d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abdaa0c50d7a056a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e8005ad0 con 0x7fc2d0004d00 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e8005c50 con 0x7fc2d0004d00 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2ee000640 1 -- 172.21.15.2:0/576489241 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e802c810 con 0x7fc2d0004d00 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2f6000640 1 -- 172.21.15.2:0/576489241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2d0004d00 msgr2=0x7fc2d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2f6000640 1 --2- 172.21.15.2:0/576489241 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2d0004d00 0x7fc2d0005180 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fc2e800c110 tx=0x7fc2d00057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.790+0000 7fc2efe00640 1 -- 172.21.15.2:0/576489241 reap_dead start 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.791+0000 7fc2f6000640 1 -- 172.21.15.2:0/576489241 shutdown_connections 2024-01-15T18:12:29.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.791+0000 7fc2f6000640 1 -- 172.21.15.2:0/576489241 >> 172.21.15.2:0/576489241 conn(0x7fc2f00663d0 msgr2=0x7fc2f00667b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.791+0000 7fc2f6000640 1 -- 172.21.15.2:0/576489241 shutdown_connections 2024-01-15T18:12:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.791+0000 7fc2f6000640 1 -- 172.21.15.2:0/576489241 wait complete. 2024-01-15T18:12:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.793+0000 7fc2f6000640 1 Processor -- start 2024-01-15T18:12:29.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.793+0000 7fc2f6000640 1 -- start start 2024-01-15T18:12:29.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.794+0000 7fc2f6000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2d0004d00 0x7fc2f00d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.794+0000 7fc2f6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 0x7fc2f01f3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2f6000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2f013ddb0 0x7fc2f013e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2f6000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f013e770 con 0x7fc2f0144990 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2f6000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f00d0780 con 0x7fc2f013ddb0 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2f6000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f01f3fd0 con 0x7fc2d0004d00 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 0x7fc2f01f3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 0x7fc2f01f3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54046/0 (socket says 172.21.15.2:54046) 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 -- 172.21.15.2:0/4149303144 learned_addr learned my addr 172.21.15.2:0/4149303144 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2efe00640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2f013ddb0 0x7fc2f013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2ef400640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2d0004d00 0x7fc2f00d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 -- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2d0004d00 msgr2=0x7fc2f00d0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2d0004d00 0x7fc2f00d0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 -- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2f013ddb0 msgr2=0x7fc2f013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:29.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2f013ddb0 0x7fc2f013e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:29.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.795+0000 7fc2eea00640 1 -- 172.21.15.2:0/4149303144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f013c370 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.796+0000 7fc2eea00640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 0x7fc2f01f3a90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8011590 tx=0x7fc2d8011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.796+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8004220 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.796+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8004380 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.796+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8015870 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.796+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f013c630 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.796+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f013cbf0 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.797+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f00686f0 con 0x7fc2f0144990 2024-01-15T18:12:29.802 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.799+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fc2d80159d0 con 0x7fc2f0144990 2024-01-15T18:12:29.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.799+0000 7fc2e7400640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc2c806c220 0x7fc2c806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:29.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.800+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4969+0+0 (secure 0 0 0) 0x7fc2d8088b70 con 0x7fc2f0144990 2024-01-15T18:12:29.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.800+0000 7fc2ef400640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc2c806c220 0x7fc2c806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:29.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.801+0000 7fc2ef400640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc2c806c220 0x7fc2c806e6e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009fd0 tx=0x7fc2e000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:29.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:29.804+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2d8055140 con 0x7fc2f0144990 2024-01-15T18:12:30.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.005+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc2f0145090 con 0x7fc2f0144990 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.006+0000 7fc2e7400640 1 -- 172.21.15.2:0/4149303144 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v99) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc2d8058760 con 0x7fc2f0144990 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T18:12:30.010 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:30.011 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:12:30.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc2c806c220 msgr2=0x7fc2c806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:30.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc2c806c220 0x7fc2c806e6e0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fc2e0009fd0 tx=0x7fc2e000b040 comp rx=0 tx=0).stop 2024-01-15T18:12:30.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 msgr2=0x7fc2f01f3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 0x7fc2f01f3a90 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fc2d8011590 tx=0x7fc2d8011a50 comp rx=0 tx=0).stop 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 shutdown_connections 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc2c806c220 0x7fc2c806e6e0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2f013ddb0 0x7fc2f013e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2f0144990 0x7fc2f01f3a90 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 --2- 172.21.15.2:0/4149303144 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2d0004d00 0x7fc2f00d0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.009+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 >> 172.21.15.2:0/4149303144 conn(0x7fc2f00663d0 msgr2=0x7fc2f006ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.010+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 shutdown_connections 2024-01-15T18:12:30.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.010+0000 7fc2f6000640 1 -- 172.21.15.2:0/4149303144 wait complete. 2024-01-15T18:12:30.034 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:12:30.034 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:12:30.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.262+0000 7f0791c00640 1 Processor -- start 2024-01-15T18:12:30.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.262+0000 7f0791c00640 1 -- start start 2024-01-15T18:12:30.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.264+0000 7f0791c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f078c144c00 0x7f078c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.264+0000 7f0791c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f078c06d000 0x7f078c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.264+0000 7f0791c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078c0ce230 con 0x7f078c0cdd70 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.264+0000 7f0791c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c0ce510 con 0x7f078c144c00 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.264+0000 7f0791c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c0ce7f0 con 0x7f078c06d000 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.264+0000 7f078aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f078c144c00 0x7f078c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f078c144c00 0x7f078c0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54056/0 (socket says 172.21.15.2:54056) 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078aa00640 1 -- 172.21.15.2:0/3143341389 learned_addr learned my addr 172.21.15.2:0/3143341389 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:30.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078b400640 1 --2- 172.21.15.2:0/3143341389 >> v2:172.21.15.137:3300/0 conn(0x7f078c06d000 0x7f078c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:30.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078aa00640 1 -- 172.21.15.2:0/3143341389 >> v2:172.21.15.137:3300/0 conn(0x7f078c06d000 msgr2=0x7f078c070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:30.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078aa00640 1 --2- 172.21.15.2:0/3143341389 >> v2:172.21.15.137:3300/0 conn(0x7f078c06d000 0x7f078c070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078aa00640 1 -- 172.21.15.2:0/3143341389 >> v1:172.21.15.137:6789/0 conn(0x7f078c0cdd70 legacy=0x7f078c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:30.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.265+0000 7f078aa00640 1 -- 172.21.15.2:0/3143341389 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c0705f0 con 0x7f078c144c00 2024-01-15T18:12:30.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.267+0000 7f078aa00640 1 --2- 172.21.15.2:0/3143341389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c144c00 0x7f078c0cd830 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f078c15ec80 tx=0x7f077c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f27846b7a4242c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:30.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.267+0000 7f078a000640 1 -- 172.21.15.2:0/3143341389 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c02ca80 con 0x7f078c144c00 2024-01-15T18:12:30.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.267+0000 7f078a000640 1 -- 172.21.15.2:0/3143341389 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c02cbe0 con 0x7f078c144c00 2024-01-15T18:12:30.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.267+0000 7f078a000640 1 -- 172.21.15.2:0/3143341389 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c0343f0 con 0x7f078c144c00 2024-01-15T18:12:30.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 -- 172.21.15.2:0/3143341389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c144c00 msgr2=0x7f078c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:30.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 --2- 172.21.15.2:0/3143341389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c144c00 0x7f078c0cd830 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f078c15ec80 tx=0x7f077c02bee0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 -- 172.21.15.2:0/3143341389 shutdown_connections 2024-01-15T18:12:30.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 --2- 172.21.15.2:0/3143341389 >> v2:172.21.15.137:3300/0 conn(0x7f078c06d000 0x7f078c070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 --2- 172.21.15.2:0/3143341389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c144c00 0x7f078c0cd830 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 -- 172.21.15.2:0/3143341389 >> 172.21.15.2:0/3143341389 conn(0x7f078c0b45f0 msgr2=0x7f078c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:30.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.268+0000 7f0791c00640 1 -- 172.21.15.2:0/3143341389 shutdown_connections 2024-01-15T18:12:30.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.269+0000 7f0791c00640 1 -- 172.21.15.2:0/3143341389 wait complete. 2024-01-15T18:12:30.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.271+0000 7f0791c00640 1 Processor -- start 2024-01-15T18:12:30.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.271+0000 7f0791c00640 1 -- start start 2024-01-15T18:12:30.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.272+0000 7f0791c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 0x7f078c13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:30.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.272+0000 7f0791c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f078c0cdd70 0x7f078c13ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.272+0000 7f0791c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f078c144c00 0x7f078c13f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.272+0000 7f0791c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c13b510 con 0x7f078c06d000 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f0791c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c13cf40 con 0x7f078c144c00 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f0791c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c13d220 con 0x7f078c0cdd70 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 0x7f078c13afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 0x7f078c13afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54070/0 (socket says 172.21.15.2:54070) 2024-01-15T18:12:30.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 -- 172.21.15.2:0/2859353222 learned_addr learned my addr 172.21.15.2:0/2859353222 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:30.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f0781e00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f078c0cdd70 0x7f078c13ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:30.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 -- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f078c0cdd70 msgr2=0x7f078c13ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:30.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078b400640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f078c144c00 0x7f078c13f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:30.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f078c0cdd70 0x7f078c13ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 -- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f078c144c00 msgr2=0x7f078c13f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:30.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f078c144c00 0x7f078c13f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f078aa00640 1 -- 172.21.15.2:0/2859353222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077c009b30 con 0x7f078c06d000 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.273+0000 7f0781e00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f078c0cdd70 0x7f078c13ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.274+0000 7f078aa00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 0x7f078c13afd0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f077c000f80 tx=0x7f077c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.274+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c03e450 con 0x7f078c06d000 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.274+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c03e5f0 con 0x7f078c06d000 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.274+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078c13d500 con 0x7f078c06d000 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.274+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c03e9a0 con 0x7f078c06d000 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.274+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078c13dac0 con 0x7f078c06d000 2024-01-15T18:12:30.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.275+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c0686f0 con 0x7f078c06d000 2024-01-15T18:12:30.280 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.276+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f077c02cd50 con 0x7f078c06d000 2024-01-15T18:12:30.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.277+0000 7f0783400640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f076406c1f0 0x7f076406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:30.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.277+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 4969+0+0 (secure 0 0 0) 0x7f077c0ab1e0 con 0x7f078c06d000 2024-01-15T18:12:30.281 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.278+0000 7f0781e00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f076406c1f0 0x7f076406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:30.283 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.279+0000 7f0781e00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f076406c1f0 0x7f076406e6b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f078c154310 tx=0x7f0774009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:30.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.282+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f077c077aa0 con 0x7f078c06d000 2024-01-15T18:12:30.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:30.475+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f078c142790 con 0x7f078c06d000 2024-01-15T18:12:31.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.124+0000 7f0783400640 1 -- 172.21.15.2:0/2859353222 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v100) v1 ==== 138+0+0 (secure 0 0 0) 0x7f077c07b0c0 con 0x7f078c06d000 2024-01-15T18:12:31.129 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f076406c1f0 msgr2=0x7f076406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f076406c1f0 0x7f076406e6b0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f078c154310 tx=0x7f0774009040 comp rx=0 tx=0).stop 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 msgr2=0x7f078c13afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 0x7f078c13afd0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f077c000f80 tx=0x7f077c004a40 comp rx=0 tx=0).stop 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 shutdown_connections 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f076406c1f0 0x7f076406e6b0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f078c144c00 0x7f078c13f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f078c0cdd70 0x7f078c13ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 --2- 172.21.15.2:0/2859353222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f078c06d000 0x7f078c13afd0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.128+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 >> 172.21.15.2:0/2859353222 conn(0x7f078c0b45f0 msgr2=0x7f078c15b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:31.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.129+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 shutdown_connections 2024-01-15T18:12:31.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.129+0000 7f0791c00640 1 -- 172.21.15.2:0/2859353222 wait complete. 2024-01-15T18:12:31.160 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:12:31.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.377+0000 7f1bfd800640 1 Processor -- start 2024-01-15T18:12:31.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.378+0000 7f1bfd800640 1 -- start start 2024-01-15T18:12:31.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bfd800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bf815c450 0x7f1bf806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:31.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bfd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf8070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:31.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bfd800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf80ce230 con 0x7f1bf8144c00 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bfd800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf806d2c0 con 0x7f1bf815c450 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bfd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf80ce510 con 0x7f1bf806ce30 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bf6a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1bf8144c00 0x7f1bf80cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44874/0 (socket says 172.21.15.2:44874) 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bf6a00640 1 -- 172.21.15.2:0/2036835194 learned_addr learned my addr 172.21.15.2:0/2036835194 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.379+0000 7f1bf6000640 1 --2- 172.21.15.2:0/2036835194 >> v2:172.21.15.2:3300/0 conn(0x7f1bf815c450 0x7f1bf806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf7400640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf8070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf5600640 1 -- 172.21.15.2:0/2036835194 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534380965 0 0) 0x7f1bf80ce230 con 0x7f1bf8144c00 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf5600640 1 -- 172.21.15.2:0/2036835194 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd8003520 con 0x7f1bf8144c00 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf6000640 1 -- 172.21.15.2:0/2036835194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 msgr2=0x7f1bf8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:31.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf6000640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf8070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf6000640 1 -- 172.21.15.2:0/2036835194 >> v1:172.21.15.2:6789/0 conn(0x7f1bf8144c00 legacy=0x7f1bf80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:31.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf6000640 1 -- 172.21.15.2:0/2036835194 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf80ce7f0 con 0x7f1bf815c450 2024-01-15T18:12:31.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.380+0000 7f1bf7400640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf8070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:31.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.381+0000 7f1bf6000640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf815c450 0x7f1bf806c8f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f1be00097a0 tx=0x7f1be002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31292bdec75a8130 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:31.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bf5600640 1 -- 172.21.15.2:0/2036835194 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be002d040 con 0x7f1bf815c450 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bf5600640 1 -- 172.21.15.2:0/2036835194 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be0031810 con 0x7f1bf815c450 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bf5600640 1 -- 172.21.15.2:0/2036835194 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be0031b80 con 0x7f1bf815c450 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 -- 172.21.15.2:0/2036835194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf815c450 msgr2=0x7f1bf806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf815c450 0x7f1bf806c8f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f1be00097a0 tx=0x7f1be002b8b0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 -- 172.21.15.2:0/2036835194 shutdown_connections 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf8070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2036835194 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf815c450 0x7f1bf806c8f0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 -- 172.21.15.2:0/2036835194 >> 172.21.15.2:0/2036835194 conn(0x7f1bf80b45f0 msgr2=0x7f1bf80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:31.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 -- 172.21.15.2:0/2036835194 shutdown_connections 2024-01-15T18:12:31.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.382+0000 7f1bfd800640 1 -- 172.21.15.2:0/2036835194 wait complete. 2024-01-15T18:12:31.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.384+0000 7f1bfd800640 1 Processor -- start 2024-01-15T18:12:31.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.384+0000 7f1bfd800640 1 -- start start 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.385+0000 7f1bfd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf814fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.385+0000 7f1bfd800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 0x7f1bf8071660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.385+0000 7f1bfd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf815c450 0x7f1bf8142a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.385+0000 7f1bfd800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf814f420 con 0x7f1bf8144c00 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.385+0000 7f1bfd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf81503b0 con 0x7f1bf815c450 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.385+0000 7f1bfd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf81f3540 con 0x7f1bf806ce30 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 0x7f1bf8071660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 0x7f1bf8071660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54098/0 (socket says 172.21.15.2:54098) 2024-01-15T18:12:31.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 -- 172.21.15.2:0/2804023697 learned_addr learned my addr 172.21.15.2:0/2804023697 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf7400640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf815c450 0x7f1bf8142a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6a00640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf814fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 -- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 msgr2=0x7f1bf814fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf814fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 -- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf815c450 msgr2=0x7f1bf8142a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf815c450 0x7f1bf8142a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.386+0000 7f1bf6000640 1 -- 172.21.15.2:0/2804023697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be0009400 con 0x7f1bf8144c00 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.387+0000 7f1bf6000640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 0x7f1bf8071660 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f1be0000c00 tx=0x7f1be00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.387+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be0006e10 con 0x7f1bf8144c00 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.387+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be003ad60 con 0x7f1bf8144c00 2024-01-15T18:12:31.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.387+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1be0044620 con 0x7f1bf8144c00 2024-01-15T18:12:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.387+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf81f3820 con 0x7f1bf8144c00 2024-01-15T18:12:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.387+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf814f950 con 0x7f1bf8144c00 2024-01-15T18:12:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.389+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf80cfe20 con 0x7f1bf8144c00 2024-01-15T18:12:31.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.390+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f1be004d440 con 0x7f1bf8144c00 2024-01-15T18:12:31.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.392+0000 7f1beea00640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bd006c1f0 0x7f1bd006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:31.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.392+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1be00aada0 con 0x7f1bf8144c00 2024-01-15T18:12:31.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.393+0000 7f1bf6a00640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bd006c1f0 0x7f1bd006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:31.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.394+0000 7f1bf6a00640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bd006c1f0 0x7f1bd006e6b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007fd0 tx=0x7f1be8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:31.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.397+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1be00777c0 con 0x7f1bf8144c00 2024-01-15T18:12:31.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:31.590+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1bf80686f0 con 0x7f1bf8144c00 2024-01-15T18:12:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.139+0000 7f1beea00640 1 -- 172.21.15.2:0/2804023697 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v101) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1be00350d0 con 0x7f1bf8144c00 2024-01-15T18:12:32.143 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bd006c1f0 msgr2=0x7f1bd006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bd006c1f0 0x7f1bd006e6b0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1be8007fd0 tx=0x7f1be8009040 comp rx=0 tx=0).stop 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 msgr2=0x7f1bf8071660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 0x7f1bf8071660 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f1be0000c00 tx=0x7f1be00046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 shutdown_connections 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1bd006c1f0 0x7f1bd006e6b0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bf815c450 0x7f1bf8142a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bf8144c00 0x7f1bf8071660 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 --2- 172.21.15.2:0/2804023697 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bf806ce30 0x7f1bf814fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 >> 172.21.15.2:0/2804023697 conn(0x7f1bf80b45f0 msgr2=0x7f1bf80cf610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:32.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.142+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 shutdown_connections 2024-01-15T18:12:32.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.143+0000 7f1bfd800640 1 -- 172.21.15.2:0/2804023697 wait complete. 2024-01-15T18:12:32.181 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:12:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.362+0000 7f982e000640 1 Processor -- start 2024-01-15T18:12:32.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.362+0000 7f982e000640 1 -- start start 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.363+0000 7f982e000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f98281439f0 0x7f98280702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f982e000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982815dd70 con 0x7f982806c8f0 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f982e000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98280b38b0 con 0x7f98280b3420 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f982e000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98280711b0 con 0x7f98281439f0 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f9827400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f98281439f0 0x7f98280702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f9827400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f98281439f0 0x7f98280702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54114/0 (socket says 172.21.15.2:54114) 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f9827400640 1 -- 172.21.15.2:0/2624601879 learned_addr learned my addr 172.21.15.2:0/2624601879 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f9827400640 1 -- 172.21.15.2:0/2624601879 >> v1:172.21.15.137:6789/0 conn(0x7f98280b3420 legacy=0x7f98280cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:32.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.364+0000 7f9827400640 1 -- 172.21.15.2:0/2624601879 >> v1:172.21.15.2:6789/0 conn(0x7f982806c8f0 legacy=0x7f982806ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:32.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.365+0000 7f9827400640 1 -- 172.21.15.2:0/2624601879 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9828143dd0 con 0x7f98281439f0 2024-01-15T18:12:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.366+0000 7f9827400640 1 --2- 172.21.15.2:0/2624601879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f98280702c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f9818009870 tx=0x7f981802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6d2ac7e07615f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.366+0000 7f9826000640 1 -- 172.21.15.2:0/2624601879 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818030860 con 0x7f98281439f0 2024-01-15T18:12:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.366+0000 7f9826000640 1 -- 172.21.15.2:0/2624601879 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98180309c0 con 0x7f98281439f0 2024-01-15T18:12:32.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.366+0000 7f9826000640 1 -- 172.21.15.2:0/2624601879 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818030d50 con 0x7f98281439f0 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 -- 172.21.15.2:0/2624601879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 msgr2=0x7f98280702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 --2- 172.21.15.2:0/2624601879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f98280702c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f9818009870 tx=0x7f981802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 -- 172.21.15.2:0/2624601879 shutdown_connections 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 --2- 172.21.15.2:0/2624601879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f98280702c0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 -- 172.21.15.2:0/2624601879 >> 172.21.15.2:0/2624601879 conn(0x7f98280b45f0 msgr2=0x7f98280b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 -- 172.21.15.2:0/2624601879 shutdown_connections 2024-01-15T18:12:32.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.367+0000 7f982e000640 1 -- 172.21.15.2:0/2624601879 wait complete. 2024-01-15T18:12:32.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.368+0000 7f982e000640 1 Processor -- start 2024-01-15T18:12:32.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.369+0000 7f982e000640 1 -- start start 2024-01-15T18:12:32.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.369+0000 7f982e000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f982806c8f0 0x7f9828169ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:32.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f982e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98280b3420 0x7f98280acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:32.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f982e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f9828137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f982e000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828159bd0 con 0x7f98281439f0 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f982e000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828169810 con 0x7f982806c8f0 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f982e000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9828169200 con 0x7f98280b3420 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f9828137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f9828137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54124/0 (socket says 172.21.15.2:54124) 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 -- 172.21.15.2:0/2466747136 learned_addr learned my addr 172.21.15.2:0/2466747136 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827400640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f982806c8f0 0x7f9828169ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9826a00640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98280b3420 0x7f98280acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 -- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98280b3420 msgr2=0x7f98280acb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:32.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98280b3420 0x7f98280acb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 -- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f982806c8f0 msgr2=0x7f9828169ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f982806c8f0 0x7f9828169ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827e00640 1 -- 172.21.15.2:0/2466747136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98180094d0 con 0x7f98281439f0 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.370+0000 7f9827400640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f982806c8f0 0x7f9828169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.371+0000 7f9827e00640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f9828137ce0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f982000c7a0 tx=0x7f982000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.371+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9820003b70 con 0x7f98281439f0 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.371+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9820003cf0 con 0x7f98281439f0 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.371+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000aa90 con 0x7f98281439f0 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.371+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98280cb5a0 con 0x7f98281439f0 2024-01-15T18:12:32.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.371+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98281694a0 con 0x7f98281439f0 2024-01-15T18:12:32.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.372+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98281699c0 con 0x7f98281439f0 2024-01-15T18:12:32.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.374+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f982000abf0 con 0x7f98281439f0 2024-01-15T18:12:32.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.375+0000 7f981f400640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f980006c240 0x7f980006e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:32.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.375+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f98200890e0 con 0x7f98281439f0 2024-01-15T18:12:32.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.375+0000 7f9827400640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f980006c240 0x7f980006e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:32.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.377+0000 7f9827400640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f980006c240 0x7f980006e700 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9818036000 tx=0x7f981803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:32.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.378+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98200556b0 con 0x7f98281439f0 2024-01-15T18:12:32.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:32.609+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9828169630 con 0x7f98281439f0 2024-01-15T18:12:32.895 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:32.903 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:12:33.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.154+0000 7f981f400640 1 -- 172.21.15.2:0/2466747136 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v61) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9820003e60 con 0x7f98281439f0 2024-01-15T18:12:33.158 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 14 and data pool 15 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.157+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f980006c240 msgr2=0x7f980006e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.157+0000 7f982e000640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f980006c240 0x7f980006e700 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9818036000 tx=0x7f981803b040 comp rx=0 tx=0).stop 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.157+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 msgr2=0x7f9828137ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.157+0000 7f982e000640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f9828137ce0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f982000c7a0 tx=0x7f982000cc60 comp rx=0 tx=0).stop 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 shutdown_connections 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f980006c240 0x7f980006e700 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f98281439f0 0x7f9828137ce0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f98280b3420 0x7f98280acb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 --2- 172.21.15.2:0/2466747136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f982806c8f0 0x7f9828169ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 >> 172.21.15.2:0/2466747136 conn(0x7f98280b45f0 msgr2=0x7f9828071450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:33.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 shutdown_connections 2024-01-15T18:12:33.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.158+0000 7f982e000640 1 -- 172.21.15.2:0/2466747136 wait complete. 2024-01-15T18:12:33.191 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:33.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.392+0000 7f0cdb400640 1 Processor -- start 2024-01-15T18:12:33.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.392+0000 7f0cdb400640 1 -- start start 2024-01-15T18:12:33.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cdb400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd40ad960 0x7f0cd415db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cdb400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0cd415e070 0x7f0cd4166d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cdb400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd40b2e70 con 0x7f0cd415d750 2024-01-15T18:12:33.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cdb400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd406c430 con 0x7f0cd415e070 2024-01-15T18:12:33.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cdb400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40b30c0 con 0x7f0cd40ad960 2024-01-15T18:12:33.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cdaa00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0cd415e070 0x7f0cd4166d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.394+0000 7f0cd9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd40ad960 0x7f0cd415db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.395+0000 7f0cd9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd40ad960 0x7f0cd415db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:56470/0 (socket says 172.21.15.2:56470) 2024-01-15T18:12:33.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.395+0000 7f0cd9600640 1 -- 172.21.15.2:0/150346186 learned_addr learned my addr 172.21.15.2:0/150346186 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.395+0000 7f0cdaa00640 1 -- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd40ad960 msgr2=0x7f0cd415db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.395+0000 7f0cdaa00640 1 --2- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd40ad960 0x7f0cd415db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.395+0000 7f0cdaa00640 1 -- 172.21.15.2:0/150346186 >> v1:172.21.15.137:6789/0 conn(0x7f0cd415d750 legacy=0x7f0cd406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.395+0000 7f0cdaa00640 1 -- 172.21.15.2:0/150346186 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40b33a0 con 0x7f0cd415e070 2024-01-15T18:12:33.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.397+0000 7f0cdaa00640 1 --2- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd4166d50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009d90 tx=0x7f0ccc02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87b2e6c2c5ce9275 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.397+0000 7f0cd8c00640 1 -- 172.21.15.2:0/150346186 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc02cc50 con 0x7f0cd415e070 2024-01-15T18:12:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.397+0000 7f0cd8c00640 1 -- 172.21.15.2:0/150346186 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc02cdb0 con 0x7f0cd415e070 2024-01-15T18:12:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.397+0000 7f0cd8c00640 1 -- 172.21.15.2:0/150346186 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc034660 con 0x7f0cd415e070 2024-01-15T18:12:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 -- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 msgr2=0x7f0cd4166d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 --2- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd4166d50 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009d90 tx=0x7f0ccc02bea0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 -- 172.21.15.2:0/150346186 shutdown_connections 2024-01-15T18:12:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 --2- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd4166d50 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 --2- 172.21.15.2:0/150346186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd40ad960 0x7f0cd415db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 -- 172.21.15.2:0/150346186 >> 172.21.15.2:0/150346186 conn(0x7f0cd40b0840 msgr2=0x7f0cd40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 -- 172.21.15.2:0/150346186 shutdown_connections 2024-01-15T18:12:33.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.398+0000 7f0cdb400640 1 -- 172.21.15.2:0/150346186 wait complete. 2024-01-15T18:12:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.400+0000 7f0cdb400640 1 Processor -- start 2024-01-15T18:12:33.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.400+0000 7f0cdb400640 1 -- start start 2024-01-15T18:12:33.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cdb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 0x7f0cd4147490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cdb400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd415d750 0x7f0cd40cbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cdb400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd40af590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cdb400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40caed0 con 0x7f0cd40ad960 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cdb400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40cb1b0 con 0x7f0cd415e070 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cdb400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40cb490 con 0x7f0cd415d750 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cda000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 0x7f0cd4147490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cda000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 0x7f0cd4147490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52064/0 (socket says 172.21.15.2:52064) 2024-01-15T18:12:33.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.401+0000 7f0cda000640 1 -- 172.21.15.2:0/1197692592 learned_addr learned my addr 172.21.15.2:0/1197692592 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cdaa00640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd40af590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cd9600640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd415d750 0x7f0cd40cbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cda000640 1 -- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd415d750 msgr2=0x7f0cd40cbc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cda000640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd415d750 0x7f0cd40cbc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cda000640 1 -- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 msgr2=0x7f0cd40af590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cda000640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd40af590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cda000640 1 -- 172.21.15.2:0/1197692592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ccc009a70 con 0x7f0cd40ad960 2024-01-15T18:12:33.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cdaa00640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd40af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.402+0000 7f0cda000640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 0x7f0cd4147490 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c450 tx=0x7f0cc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.403+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc40052d0 con 0x7f0cd40ad960 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.403+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc4005430 con 0x7f0cd40ad960 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.403+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd40cb770 con 0x7f0cd40ad960 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.403+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc4012870 con 0x7f0cd40ad960 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.403+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd40cc1d0 con 0x7f0cd40ad960 2024-01-15T18:12:33.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.404+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd40cb9b0 con 0x7f0cd40ad960 2024-01-15T18:12:33.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.405+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f0cc40129d0 con 0x7f0cd40ad960 2024-01-15T18:12:33.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.406+0000 7f0cd2000640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cac06c1f0 0x7f0cac06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.406+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f0cc4088ed0 con 0x7f0cd40ad960 2024-01-15T18:12:33.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.407+0000 7f0cd9600640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cac06c1f0 0x7f0cac06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.408+0000 7f0cd9600640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cac06c1f0 0x7f0cac06e6b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0cd415d6a0 tx=0x7f0cbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:33.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.410+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cc40554a0 con 0x7f0cd40ad960 2024-01-15T18:12:33.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.611+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cd40cb9b0 con 0x7f0cd40ad960 2024-01-15T18:12:33.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.614+0000 7f0cd2000640 1 -- 172.21.15.2:0/1197692592 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+16648 (secure 0 0 0) 0x7f0cd40cb9b0 con 0x7f0cd40ad960 2024-01-15T18:12:33.619 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:33.619 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":102,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:32.612185+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:12:30.532826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-15T18:12:31.644235+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cac06c1f0 msgr2=0x7f0cac06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cac06c1f0 0x7f0cac06e6b0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f0cd415d6a0 tx=0x7f0cbc008040 comp rx=0 tx=0).stop 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 msgr2=0x7f0cd4147490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 0x7f0cd4147490 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f0cc400c450 tx=0x7f0cc400c910 comp rx=0 tx=0).stop 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 shutdown_connections 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cac06c1f0 0x7f0cac06e6b0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd415e070 0x7f0cd40af590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd415d750 0x7f0cd40cbc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 --2- 172.21.15.2:0/1197692592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd40ad960 0x7f0cd4147490 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 >> 172.21.15.2:0/1197692592 conn(0x7f0cd40b0840 msgr2=0x7f0cd40ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:33.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 shutdown_connections 2024-01-15T18:12:33.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.617+0000 7f0cdb400640 1 -- 172.21.15.2:0/1197692592 wait complete. 2024-01-15T18:12:33.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:33.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.862+0000 7f00df200640 1 Processor -- start 2024-01-15T18:12:33.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.862+0000 7f00df200640 1 -- start start 2024-01-15T18:12:33.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.864+0000 7f00df200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f00e015ec60 0x7f00e015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.864+0000 7f00df200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e00716d0 con 0x7f00e006cad0 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.864+0000 7f00df200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e0071960 con 0x7f00e0141880 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.864+0000 7f00df200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e0071f20 con 0x7f00e015ec60 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.864+0000 7f00de800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f00e006cad0 0x7f00e006cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60528/0 (socket says 172.21.15.2:60528) 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.864+0000 7f00de800640 1 -- 172.21.15.2:0/3609036009 learned_addr learned my addr 172.21.15.2:0/3609036009 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00d7e00640 1 --2- 172.21.15.2:0/3609036009 >> v2:172.21.15.137:3300/0 conn(0x7f00e015ec60 0x7f00e015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268173972 0 0) 0x7f00e00716d0 con 0x7f00e006cad0 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c0003520 con 0x7f00e006cad0 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2749679852 0 0) 0x7f00c0003520 con 0x7f00e006cad0 2024-01-15T18:12:33.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 >> v2:172.21.15.137:3300/0 conn(0x7f00e015ec60 msgr2=0x7f00e015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 --2- 172.21.15.2:0/3609036009 >> v2:172.21.15.137:3300/0 conn(0x7f00e015ec60 0x7f00e015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 >> v1:172.21.15.137:6789/0 conn(0x7f00e0141880 legacy=0x7f00e0070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0071c40 con 0x7f00e006cad0 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f00d80019f0 con 0x7f00e006cad0 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.865+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 >> v1:172.21.15.2:6789/0 conn(0x7f00e006cad0 legacy=0x7f00e006cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dd400640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00c0003970 0x7f00c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dd400640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00c00047c0 0x7f00c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dd400640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e0071f20 con 0x7f00c0005180 2024-01-15T18:12:33.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c000c120 con 0x7f00c0003970 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c000c2e0 con 0x7f00c00047c0 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00de800640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00dde00640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00c0003970 0x7f00c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00d7e00640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00c00047c0 0x7f00c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00de800640 1 -- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00c00047c0 msgr2=0x7f00c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.866+0000 7f00de800640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00c00047c0 0x7f00c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.867+0000 7f00de800640 1 -- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00c0003970 msgr2=0x7f00c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.867+0000 7f00de800640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00c0003970 0x7f00c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.867+0000 7f00de800640 1 -- 172.21.15.2:0/3609036009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c0003520 con 0x7f00c0005180 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.868+0000 7f00de800640 1 -- 172.21.15.2:0/3609036009 reap_dead start 2024-01-15T18:12:33.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.868+0000 7f00de800640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00c000b980 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f00d800a830 tx=0x7f00c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d77b10e14339b334 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:33.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.868+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d80072d0 con 0x7f00c0005180 2024-01-15T18:12:33.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.868+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d802f070 con 0x7f00c0005180 2024-01-15T18:12:33.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.868+0000 7f00dd400640 1 -- 172.21.15.2:0/3609036009 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d80076b0 con 0x7f00c0005180 2024-01-15T18:12:33.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 -- 172.21.15.2:0/3609036009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 msgr2=0x7f00c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00c000b980 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7f00d800a830 tx=0x7f00c00057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 -- 172.21.15.2:0/3609036009 shutdown_connections 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 --2- 172.21.15.2:0/3609036009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00c000b980 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 -- 172.21.15.2:0/3609036009 >> 172.21.15.2:0/3609036009 conn(0x7f00e00b0840 msgr2=0x7f00e00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 -- 172.21.15.2:0/3609036009 shutdown_connections 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.869+0000 7f00df200640 1 -- 172.21.15.2:0/3609036009 wait complete. 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.871+0000 7f00df200640 1 Processor -- start 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.871+0000 7f00df200640 1 -- start start 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00df200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00e00cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00df200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00e0133ce0 0x7f00e01321f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00df200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00e0132730 0x7f00e0134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00df200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e006c6b0 con 0x7f00c0005180 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00df200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e0133850 con 0x7f00e0132730 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00df200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e00cb9f0 con 0x7f00e0133ce0 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00e00cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00e00cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52088/0 (socket says 172.21.15.2:52088) 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 -- 172.21.15.2:0/176659080 learned_addr learned my addr 172.21.15.2:0/176659080 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00de800640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00e0132730 0x7f00e0134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00d7e00640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00e0133ce0 0x7f00e01321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 -- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00e0133ce0 msgr2=0x7f00e01321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00e0133ce0 0x7f00e01321f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 -- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00e0132730 msgr2=0x7f00e0134250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00e0132730 0x7f00e0134250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.872+0000 7f00dde00640 1 -- 172.21.15.2:0/176659080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0132c40 con 0x7f00c0005180 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.873+0000 7f00dde00640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00e00cb4b0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f00d000e530 tx=0x7f00d000e9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.873+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d00052e0 con 0x7f00c0005180 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.873+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d0005440 con 0x7f00c0005180 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.873+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d0003810 con 0x7f00c0005180 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.873+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e0132f20 con 0x7f00c0005180 2024-01-15T18:12:33.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.873+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e01334e0 con 0x7f00c0005180 2024-01-15T18:12:33.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.874+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e0074770 con 0x7f00c0005180 2024-01-15T18:12:33.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.876+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f00d0003970 con 0x7f00c0005180 2024-01-15T18:12:33.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.876+0000 7f00d6000640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00b806c1f0 0x7f00b806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:33.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.877+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f00d0088700 con 0x7f00c0005180 2024-01-15T18:12:33.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.877+0000 7f00d7e00640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00b806c1f0 0x7f00b806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:33.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.878+0000 7f00d7e00640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00b806c1f0 0x7f00b806e6b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f00e006dcf0 tx=0x7f00c8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:33.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:33.881+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00d0054cd0 con 0x7f00c0005180 2024-01-15T18:12:34.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.080+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00e006e1c0 con 0x7f00c0005180 2024-01-15T18:12:34.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.083+0000 7f00d6000640 1 -- 172.21.15.2:0/176659080 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+16635 (secure 0 0 0) 0x7f00d00582f0 con 0x7f00c0005180 2024-01-15T18:12:34.088 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:34.088 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":103,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:33.684063+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:12:30.532826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-15T18:12:31.644235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00b806c1f0 msgr2=0x7f00b806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00b806c1f0 0x7f00b806e6b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f00e006dcf0 tx=0x7f00c8008040 comp rx=0 tx=0).stop 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 msgr2=0x7f00e00cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00e00cb4b0 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f00d000e530 tx=0x7f00d000e9f0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 shutdown_connections 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00b806c1f0 0x7f00b806e6b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f00e0132730 0x7f00e0134250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f00e0133ce0 0x7f00e01321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 --2- 172.21.15.2:0/176659080 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f00c0005180 0x7f00e00cb4b0 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.086+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 >> 172.21.15.2:0/176659080 conn(0x7f00e00b0840 msgr2=0x7f00e00ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:34.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.087+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 shutdown_connections 2024-01-15T18:12:34.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.087+0000 7f00df200640 1 -- 172.21.15.2:0/176659080 wait complete. 2024-01-15T18:12:34.110 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:12:34.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.337+0000 7fec1f200640 1 Processor -- start 2024-01-15T18:12:34.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.337+0000 7fec1f200640 1 -- start start 2024-01-15T18:12:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.338+0000 7fec1f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 0x7fec20070110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fec20161210 0x7fec20163600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1f200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec20070650 con 0x7fec20070b20 2024-01-15T18:12:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2006e030 con 0x7fec20161210 2024-01-15T18:12:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec20070880 con 0x7fec2006dbb0 2024-01-15T18:12:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1de00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fec20070b20 0x7fec20070f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60538/0 (socket says 172.21.15.2:60538) 2024-01-15T18:12:34.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1de00640 1 -- 172.21.15.2:0/2672059835 learned_addr learned my addr 172.21.15.2:0/2672059835 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:34.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec17e00640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 0x7fec20070110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1e800640 1 --2- 172.21.15.2:0/2672059835 >> v2:172.21.15.137:3300/0 conn(0x7fec20161210 0x7fec20163600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.339+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328670325 0 0) 0x7fec20070650 con 0x7fec20070b20 2024-01-15T18:12:34.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec00003520 con 0x7fec20070b20 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19553886 0 0) 0x7fec00003520 con 0x7fec20070b20 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 msgr2=0x7fec20070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 0x7fec20070110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 >> v2:172.21.15.137:3300/0 conn(0x7fec20161210 msgr2=0x7fec20163600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 --2- 172.21.15.2:0/2672059835 >> v2:172.21.15.137:3300/0 conn(0x7fec20161210 0x7fec20163600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec200b2b50 con 0x7fec20070b20 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fec10001670 con 0x7fec20070b20 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1e800640 1 --2- 172.21.15.2:0/2672059835 >> v2:172.21.15.137:3300/0 conn(0x7fec20161210 0x7fec20163600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:34.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 >> v1:172.21.15.2:6789/0 conn(0x7fec20070b20 legacy=0x7fec20070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec00003970 0x7fec00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 0x7fec000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec00004d00 0x7fec00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2006e030 con 0x7fec00004d00 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec20070880 con 0x7fec00003970 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.340+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0000b980 con 0x7fec2006dbb0 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1e800640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec00004d00 0x7fec00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec17e00640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 0x7fec000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1de00640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec00003970 0x7fec00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1e800640 1 -- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 msgr2=0x7fec000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:34.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1e800640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec2006dbb0 0x7fec000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1e800640 1 -- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec00003970 msgr2=0x7fec00003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:34.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1e800640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec00003970 0x7fec00003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.341+0000 7fec1e800640 1 -- 172.21.15.2:0/2672059835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00003520 con 0x7fec00004d00 2024-01-15T18:12:34.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.342+0000 7fec1e800640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec00004d00 0x7fec00005180 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fec180073e0 tx=0x7fec000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96e288ed89c6644 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:34.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1802cd70 con 0x7fec00004d00 2024-01-15T18:12:34.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec18034440 con 0x7fec00004d00 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1d400640 1 -- 172.21.15.2:0/2672059835 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec180347b0 con 0x7fec00004d00 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1f200640 1 -- 172.21.15.2:0/2672059835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec00004d00 msgr2=0x7fec00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1f200640 1 --2- 172.21.15.2:0/2672059835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec00004d00 0x7fec00005180 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fec180073e0 tx=0x7fec000057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1e800640 1 -- 172.21.15.2:0/2672059835 reap_dead start 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1f200640 1 -- 172.21.15.2:0/2672059835 shutdown_connections 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1f200640 1 -- 172.21.15.2:0/2672059835 >> 172.21.15.2:0/2672059835 conn(0x7fec200b0840 msgr2=0x7fec200b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.343+0000 7fec1f200640 1 -- 172.21.15.2:0/2672059835 shutdown_connections 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.344+0000 7fec1f200640 1 -- 172.21.15.2:0/2672059835 wait complete. 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.345+0000 7fec1f200640 1 Processor -- start 2024-01-15T18:12:34.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.345+0000 7fec1f200640 1 -- start start 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.346+0000 7fec1f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec00004d00 0x7fec200cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.346+0000 7fec1f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec201476e0 0x7fec200cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.346+0000 7fec1f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 0x7fec2015ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.346+0000 7fec1f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec200cd220 con 0x7fec200cbf20 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.346+0000 7fec1f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec200cc3a0 con 0x7fec201476e0 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.346+0000 7fec1f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec201615f0 con 0x7fec00004d00 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 0x7fec2015ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 0x7fec2015ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52108/0 (socket says 172.21.15.2:52108) 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 -- 172.21.15.2:0/2610271734 learned_addr learned my addr 172.21.15.2:0/2610271734 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1de00640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec00004d00 0x7fec200cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec17e00640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec201476e0 0x7fec200cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 -- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec00004d00 msgr2=0x7fec200cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec00004d00 0x7fec200cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 -- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec201476e0 msgr2=0x7fec200cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec201476e0 0x7fec200cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.347+0000 7fec1e800640 1 -- 172.21.15.2:0/2610271734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec20147070 con 0x7fec200cbf20 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.348+0000 7fec1e800640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 0x7fec2015ba80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fec18002230 tx=0x7fec18004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.348+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1803e440 con 0x7fec200cbf20 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.348+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1803e5c0 con 0x7fec200cbf20 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.348+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1803e970 con 0x7fec200cbf20 2024-01-15T18:12:34.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.348+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec200cc5a0 con 0x7fec200cbf20 2024-01-15T18:12:34.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.348+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec200ccb60 con 0x7fec200cbf20 2024-01-15T18:12:34.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.349+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2006e430 con 0x7fec200cbf20 2024-01-15T18:12:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.350+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fec1803ead0 con 0x7fec200cbf20 2024-01-15T18:12:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.351+0000 7fec16000640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7febf806c1f0 0x7febf806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.352+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fec180aad50 con 0x7fec200cbf20 2024-01-15T18:12:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.352+0000 7fec1de00640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7febf806c1f0 0x7febf806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.353+0000 7fec1de00640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7febf806c1f0 0x7febf806e6b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fec10000c00 tx=0x7fec10009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:34.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.356+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec18077320 con 0x7fec200cbf20 2024-01-15T18:12:34.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:34.591+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fec200cd360 con 0x7fec200cbf20 2024-01-15T18:12:35.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.181+0000 7fec16000640 1 -- 172.21.15.2:0/2610271734 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v64) v1 ==== 118+0+0 (secure 0 0 0) 0x7fec200cd360 con 0x7fec200cbf20 2024-01-15T18:12:35.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7febf806c1f0 msgr2=0x7febf806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:35.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7febf806c1f0 0x7febf806e6b0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fec10000c00 tx=0x7fec10009040 comp rx=0 tx=0).stop 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 msgr2=0x7fec2015ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 0x7fec2015ba80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fec18002230 tx=0x7fec18004ab0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 shutdown_connections 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7febf806c1f0 0x7febf806e6b0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fec200cbf20 0x7fec2015ba80 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fec201476e0 0x7fec200cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 --2- 172.21.15.2:0/2610271734 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fec00004d00 0x7fec200cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 >> 172.21.15.2:0/2610271734 conn(0x7fec200b0840 msgr2=0x7fec200b2d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.185+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 shutdown_connections 2024-01-15T18:12:35.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.186+0000 7fec1f200640 1 -- 172.21.15.2:0/2610271734 wait complete. 2024-01-15T18:12:35.210 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:12:35.211 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:12:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.436+0000 7f89a9a00640 1 Processor -- start 2024-01-15T18:12:35.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.436+0000 7f89a9a00640 1 -- start start 2024-01-15T18:12:35.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.437+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a415d120 0x7f89a40b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.437+0000 7f89a9a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f89a40cf920 0x7f89a40cfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.437+0000 7f89a9a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f89a40cddd0 0x7f89a40ce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.437+0000 7f89a9a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a40cf100 con 0x7f89a40cf920 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.437+0000 7f89a9a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a40d0280 con 0x7f89a40cddd0 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.437+0000 7f89a9a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40ceb40 con 0x7f89a415d120 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f89a40cf920 0x7f89a40cfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f89a40cf920 0x7f89a40cfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52120/0 (socket says 172.21.15.2:52120) 2024-01-15T18:12:35.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 -- 172.21.15.2:0/182375951 learned_addr learned my addr 172.21.15.2:0/182375951 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3e00640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a415d120 0x7f89a40b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 -- 172.21.15.2:0/182375951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a415d120 msgr2=0x7f89a40b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a415d120 0x7f89a40b2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 -- 172.21.15.2:0/182375951 >> v2:172.21.15.137:3300/0 conn(0x7f89a40cddd0 msgr2=0x7f89a40ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a9000640 1 --2- 172.21.15.2:0/182375951 >> v2:172.21.15.137:3300/0 conn(0x7f89a40cddd0 0x7f89a40ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 --2- 172.21.15.2:0/182375951 >> v2:172.21.15.137:3300/0 conn(0x7f89a40cddd0 0x7f89a40ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3400640 1 -- 172.21.15.2:0/182375951 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a40cee20 con 0x7f89a40cf920 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a3e00640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a415d120 0x7f89a40b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.438+0000 7f89a9000640 1 --2- 172.21.15.2:0/182375951 >> v2:172.21.15.137:3300/0 conn(0x7f89a40cddd0 0x7f89a40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.439+0000 7f89a3400640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cf920 0x7f89a40cfd40 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f89a40d4130 tx=0x7f898c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4373c429bd0dce4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:35.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.440+0000 7f89a2a00640 1 -- 172.21.15.2:0/182375951 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c031880 con 0x7f89a40cf920 2024-01-15T18:12:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.440+0000 7f89a2a00640 1 -- 172.21.15.2:0/182375951 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f898c0319e0 con 0x7f89a40cf920 2024-01-15T18:12:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.440+0000 7f89a2a00640 1 -- 172.21.15.2:0/182375951 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898c031d70 con 0x7f89a40cf920 2024-01-15T18:12:35.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 -- 172.21.15.2:0/182375951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cf920 msgr2=0x7f89a40cfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cf920 0x7f89a40cfd40 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f89a40d4130 tx=0x7f898c02b920 comp rx=0 tx=0).stop 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 -- 172.21.15.2:0/182375951 shutdown_connections 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 --2- 172.21.15.2:0/182375951 >> v2:172.21.15.137:3300/0 conn(0x7f89a40cddd0 0x7f89a40ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cf920 0x7f89a40cfd40 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 --2- 172.21.15.2:0/182375951 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a415d120 0x7f89a40b2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 -- 172.21.15.2:0/182375951 >> 172.21.15.2:0/182375951 conn(0x7f89a409c3e0 msgr2=0x7f89a40d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 -- 172.21.15.2:0/182375951 shutdown_connections 2024-01-15T18:12:35.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.441+0000 7f89a9a00640 1 -- 172.21.15.2:0/182375951 wait complete. 2024-01-15T18:12:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.443+0000 7f89a9a00640 1 Processor -- start 2024-01-15T18:12:35.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.443+0000 7f89a9a00640 1 -- start start 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 0x7f89a41601c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a40cf920 0x7f89a4158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a415d120 0x7f89a41f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40cc240 con 0x7f89a40cddd0 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40cb630 con 0x7f89a415d120 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40cb910 con 0x7f89a40cf920 2024-01-15T18:12:35.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 0x7f89a41601c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 0x7f89a41601c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52128/0 (socket says 172.21.15.2:52128) 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3e00640 1 -- 172.21.15.2:0/1369033887 learned_addr learned my addr 172.21.15.2:0/1369033887 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3400640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a40cf920 0x7f89a4158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3e00640 1 -- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a40cf920 msgr2=0x7f89a4158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3e00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a40cf920 0x7f89a4158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a9000640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a415d120 0x7f89a41f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.444+0000 7f89a3e00640 1 -- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a415d120 msgr2=0x7f89a41f3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f89a3e00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a415d120 0x7f89a41f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f89a3e00640 1 -- 172.21.15.2:0/1369033887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c0094d0 con 0x7f89a40cddd0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f89a3e00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 0x7f89a41601c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f899400c7f0 tx=0x7f899400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994007a30 con 0x7f89a40cddd0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8994007b90 con 0x7f89a40cddd0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899400ce40 con 0x7f89a40cddd0 2024-01-15T18:12:35.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a40cbbf0 con 0x7f89a40cddd0 2024-01-15T18:12:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.445+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a4160700 con 0x7f89a40cddd0 2024-01-15T18:12:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.447+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a4074770 con 0x7f89a40cddd0 2024-01-15T18:12:35.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.448+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f89940113f0 con 0x7f89a40cddd0 2024-01-15T18:12:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.449+0000 7f899be00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f897c06c220 0x7f897c06e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.449+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f89940887b0 con 0x7f89a40cddd0 2024-01-15T18:12:35.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.450+0000 7f89a3400640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f897c06c220 0x7f897c06e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:35.454 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.451+0000 7f89a3400640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f897c06c220 0x7f897c06e6e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f898c036000 tx=0x7f898c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:35.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.454+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8994055070 con 0x7f89a40cddd0 2024-01-15T18:12:35.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:35.683+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f89a408ce80 con 0x7f89a40cddd0 2024-01-15T18:12:36.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.192+0000 7f899be00640 1 -- 172.21.15.2:0/1369033887 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v65) v1 ==== 122+0+0 (secure 0 0 0) 0x7f899408a030 con 0x7f89a40cddd0 2024-01-15T18:12:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f897c06c220 msgr2=0x7f897c06e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f897c06c220 0x7f897c06e6e0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f898c036000 tx=0x7f898c03c040 comp rx=0 tx=0).stop 2024-01-15T18:12:36.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 msgr2=0x7f89a41601c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 0x7f89a41601c0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f899400c7f0 tx=0x7f899400cbf0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 shutdown_connections 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f897c06c220 0x7f897c06e6e0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f89a415d120 0x7f89a41f3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f89a40cf920 0x7f89a4158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 --2- 172.21.15.2:0/1369033887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f89a40cddd0 0x7f89a41601c0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 >> 172.21.15.2:0/1369033887 conn(0x7f89a409c3e0 msgr2=0x7f89a40d2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 shutdown_connections 2024-01-15T18:12:36.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.196+0000 7f89a9a00640 1 -- 172.21.15.2:0/1369033887 wait complete. 2024-01-15T18:12:36.221 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:12:36.221 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:12:36.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.441+0000 7f3454800640 1 Processor -- start 2024-01-15T18:12:36.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.441+0000 7f3454800640 1 -- start start 2024-01-15T18:12:36.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3454800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f344c15c0e0 0x7f344c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3454800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3454800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344c15cb00 con 0x7f344c15c4c0 2024-01-15T18:12:36.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3454800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344c0ccec0 con 0x7f344c15c0e0 2024-01-15T18:12:36.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3454800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0cd1a0 con 0x7f344c15b7d0 2024-01-15T18:12:36.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3451e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f344c15c0e0 0x7f344c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3451e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f344c15c0e0 0x7f344c0cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52134/0 (socket says 172.21.15.2:52134) 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.443+0000 7f3451e00640 1 -- 172.21.15.2:0/3278824750 learned_addr learned my addr 172.21.15.2:0/3278824750 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.444+0000 7f3451400640 1 --2- 172.21.15.2:0/3278824750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.444+0000 7f3451e00640 1 -- 172.21.15.2:0/3278824750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 msgr2=0x7f344c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.444+0000 7f3451e00640 1 --2- 172.21.15.2:0/3278824750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.444+0000 7f3451e00640 1 -- 172.21.15.2:0/3278824750 >> v1:172.21.15.137:6789/0 conn(0x7f344c15c4c0 legacy=0x7f344c15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:36.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.444+0000 7f3451e00640 1 -- 172.21.15.2:0/3278824750 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344c0cd480 con 0x7f344c15c0e0 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.445+0000 7f3451e00640 1 --2- 172.21.15.2:0/3278824750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c0cc5c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f343c009870 tx=0x7f343c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=550c3cfcd8c075f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.445+0000 7f3450a00640 1 -- 172.21.15.2:0/3278824750 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c031640 con 0x7f344c15c0e0 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3450a00640 1 -- 172.21.15.2:0/3278824750 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343c0317a0 con 0x7f344c15c0e0 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3450a00640 1 -- 172.21.15.2:0/3278824750 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343c031b30 con 0x7f344c15c0e0 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3454800640 1 -- 172.21.15.2:0/3278824750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 msgr2=0x7f344c0cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3454800640 1 --2- 172.21.15.2:0/3278824750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c0cc5c0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f343c009870 tx=0x7f343c02b8f0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3454800640 1 -- 172.21.15.2:0/3278824750 shutdown_connections 2024-01-15T18:12:36.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3454800640 1 --2- 172.21.15.2:0/3278824750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3454800640 1 --2- 172.21.15.2:0/3278824750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c0cc5c0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.446+0000 7f3454800640 1 -- 172.21.15.2:0/3278824750 >> 172.21.15.2:0/3278824750 conn(0x7f344c06da20 msgr2=0x7f344c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:36.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.447+0000 7f3454800640 1 -- 172.21.15.2:0/3278824750 shutdown_connections 2024-01-15T18:12:36.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.447+0000 7f3454800640 1 -- 172.21.15.2:0/3278824750 wait complete. 2024-01-15T18:12:36.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.449+0000 7f3454800640 1 Processor -- start 2024-01-15T18:12:36.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.449+0000 7f3454800640 1 -- start start 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.450+0000 7f3454800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c1f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3454800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c13f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3454800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f344c15c4c0 0x7f344c15a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3454800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c0706a0 con 0x7f344c15c0e0 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3454800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c1f69a0 con 0x7f344c15c4c0 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3454800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344c13fe50 con 0x7f344c15b7d0 2024-01-15T18:12:36.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c13f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c13f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52136/0 (socket says 172.21.15.2:52136) 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 -- 172.21.15.2:0/811712974 learned_addr learned my addr 172.21.15.2:0/811712974 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451e00640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c1f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3452800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f344c15c4c0 0x7f344c15a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 -- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 msgr2=0x7f344c1f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c1f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 -- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f344c15c4c0 msgr2=0x7f344c15a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f344c15c4c0 0x7f344c15a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.451+0000 7f3451400640 1 -- 172.21.15.2:0/811712974 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343c0094d0 con 0x7f344c15c0e0 2024-01-15T18:12:36.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.452+0000 7f3452800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f344c15c4c0 0x7f344c15a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:36.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.452+0000 7f3451400640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c13f910 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f343400d5c0 tx=0x7f343400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:36.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.452+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34340125f0 con 0x7f344c15c0e0 2024-01-15T18:12:36.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.452+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3434012750 con 0x7f344c15c0e0 2024-01-15T18:12:36.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.452+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3434012b00 con 0x7f344c15c0e0 2024-01-15T18:12:36.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.452+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c1f6c20 con 0x7f344c15c0e0 2024-01-15T18:12:36.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.453+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c1f71e0 con 0x7f344c15c0e0 2024-01-15T18:12:36.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.454+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c0686f0 con 0x7f344c15c0e0 2024-01-15T18:12:36.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.455+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f3434012c60 con 0x7f344c15c0e0 2024-01-15T18:12:36.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.455+0000 7f344a000640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f342406c1f0 0x7f342406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.456+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f3434088e30 con 0x7f344c15c0e0 2024-01-15T18:12:36.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.456+0000 7f3451e00640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f342406c1f0 0x7f342406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.457+0000 7f3451e00640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f342406c1f0 0x7f342406e6b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f343c036000 tx=0x7f343c03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:36.464 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.460+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3434055400 con 0x7f344c15c0e0 2024-01-15T18:12:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.691+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f344c06b260 con 0x7f344c15c0e0 2024-01-15T18:12:36.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.692+0000 7f344a000640 1 -- 172.21.15.2:0/811712974 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f3434058a20 con 0x7f344c15c0e0 2024-01-15T18:12:36.696 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:36.696 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":65,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3406786991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3406786991},{"type":"v1","addr":"172.21.15.2:6835","nonce":3406786991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/739459094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":739459094},{"type":"v1","addr":"172.21.15.137:6836","nonce":739459094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8565,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/2674783053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2674783053},{"type":"v1","addr":"172.21.15.2:6834","nonce":2674783053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:12:32.613003+0000","modified":"2024-01-15T18:12:35.685814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8580":{"gid":8580,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6837/320323477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":320323477},{"type":"v1","addr":"172.21.15.137:6837","nonce":320323477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-15T18:12:36.697 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 65 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f342406c1f0 msgr2=0x7f342406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f342406c1f0 0x7f342406e6b0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f343c036000 tx=0x7f343c03c040 comp rx=0 tx=0).stop 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 msgr2=0x7f344c13f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c13f910 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f343400d5c0 tx=0x7f343400da80 comp rx=0 tx=0).stop 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 shutdown_connections 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f342406c1f0 0x7f342406e6b0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f344c15c4c0 0x7f344c15a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.700 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f344c15c0e0 0x7f344c13f910 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 --2- 172.21.15.2:0/811712974 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f344c15b7d0 0x7f344c1f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 >> 172.21.15.2:0/811712974 conn(0x7f344c06da20 msgr2=0x7f344c1508e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:36.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.695+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 shutdown_connections 2024-01-15T18:12:36.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.696+0000 7f3454800640 1 -- 172.21.15.2:0/811712974 wait complete. 2024-01-15T18:12:36.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:12:36.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.929+0000 7f140d400640 1 Processor -- start 2024-01-15T18:12:36.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.930+0000 7f140d400640 1 -- start start 2024-01-15T18:12:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f140d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1408141880 0x7f1408070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f140d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f140815f0e0 0x7f14080cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f140d400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140806dae0 con 0x7f140815ec60 2024-01-15T18:12:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f140d400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1408071f20 con 0x7f140815f0e0 2024-01-15T18:12:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f140d400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f140806c6b0 con 0x7f1408141880 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f13ffe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f140815ec60 0x7f14080cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60552/0 (socket says 172.21.15.2:60552) 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f13ffe00640 1 -- 172.21.15.2:0/2001349984 learned_addr learned my addr 172.21.15.2:0/2001349984 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.931+0000 7f140ca00640 1 --2- 172.21.15.2:0/2001349984 >> v2:172.21.15.2:3300/0 conn(0x7f140815f0e0 0x7f14080cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f1407e00640 1 --2- 172.21.15.2:0/2001349984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1408141880 0x7f1408070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f1407400640 1 -- 172.21.15.2:0/2001349984 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807379632 0 0) 0x7f140806dae0 con 0x7f140815ec60 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f1407400640 1 -- 172.21.15.2:0/2001349984 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e8003520 con 0x7f140815ec60 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f140ca00640 1 -- 172.21.15.2:0/2001349984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1408141880 msgr2=0x7f1408070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f140ca00640 1 --2- 172.21.15.2:0/2001349984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1408141880 0x7f1408070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f140ca00640 1 -- 172.21.15.2:0/2001349984 >> v1:172.21.15.2:6789/0 conn(0x7f140815ec60 legacy=0x7f14080cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.932+0000 7f140ca00640 1 -- 172.21.15.2:0/2001349984 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140815f570 con 0x7f140815f0e0 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.933+0000 7f140ca00640 1 --2- 172.21.15.2:0/2001349984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815f0e0 0x7f14080cdba0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f1400009d90 tx=0x7f140002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79344fcdcdf7401e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.933+0000 7f1407400640 1 -- 172.21.15.2:0/2001349984 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140002cbd0 con 0x7f140815f0e0 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.933+0000 7f1407400640 1 -- 172.21.15.2:0/2001349984 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140002cd30 con 0x7f140815f0e0 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.933+0000 7f1407400640 1 -- 172.21.15.2:0/2001349984 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140002cea0 con 0x7f140815f0e0 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 -- 172.21.15.2:0/2001349984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815f0e0 msgr2=0x7f14080cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 --2- 172.21.15.2:0/2001349984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815f0e0 0x7f14080cdba0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f1400009d90 tx=0x7f140002be20 comp rx=0 tx=0).stop 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 -- 172.21.15.2:0/2001349984 shutdown_connections 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 --2- 172.21.15.2:0/2001349984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815f0e0 0x7f14080cdba0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 --2- 172.21.15.2:0/2001349984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1408141880 0x7f1408070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 -- 172.21.15.2:0/2001349984 >> 172.21.15.2:0/2001349984 conn(0x7f14080b0840 msgr2=0x7f14080b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 -- 172.21.15.2:0/2001349984 shutdown_connections 2024-01-15T18:12:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.934+0000 7f140d400640 1 -- 172.21.15.2:0/2001349984 wait complete. 2024-01-15T18:12:36.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.936+0000 7f140d400640 1 Processor -- start 2024-01-15T18:12:36.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.936+0000 7f140d400640 1 -- start start 2024-01-15T18:12:36.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f140d400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1408141880 0x7f14080b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f140d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 0x7f140813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f140d400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f140815f0e0 0x7f140813fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f140d400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080acd60 con 0x7f140815ec60 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f140d400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14080ac9c0 con 0x7f1408141880 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f140d400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14080cc270 con 0x7f140815f0e0 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 0x7f140813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 0x7f140813dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52152/0 (socket says 172.21.15.2:52152) 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 -- 172.21.15.2:0/372987965 learned_addr learned my addr 172.21.15.2:0/372987965 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 -- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f140815f0e0 msgr2=0x7f140813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f140815f0e0 0x7f140813fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 -- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1408141880 msgr2=0x7f14080b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1408141880 0x7f14080b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.937+0000 7f13ffe00640 1 -- 172.21.15.2:0/372987965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1400009a70 con 0x7f140815ec60 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.938+0000 7f13ffe00640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 0x7f140813dd00 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c410 tx=0x7f13f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.938+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0005ce0 con 0x7f140815ec60 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.938+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0005e40 con 0x7f140815ec60 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.938+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f00072f0 con 0x7f140815ec60 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.938+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14080cc550 con 0x7f140815ec60 2024-01-15T18:12:36.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.938+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1408154980 con 0x7f140815ec60 2024-01-15T18:12:36.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.939+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1408074770 con 0x7f140815ec60 2024-01-15T18:12:36.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.940+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f13f0007450 con 0x7f140815ec60 2024-01-15T18:12:36.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.941+0000 7f1404c00640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13e006c1f0 0x7f13e006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:36.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.941+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f13f0087d20 con 0x7f140815ec60 2024-01-15T18:12:36.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.942+0000 7f1407e00640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13e006c1f0 0x7f13e006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:36.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.943+0000 7f1407e00640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13e006c1f0 0x7f13e006e6b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f13f8006fd0 tx=0x7f13f8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:36.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:36.946+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13f0058130 con 0x7f140815ec60 2024-01-15T18:12:37.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.140+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f140806e1c0 con 0x7f140815ec60 2024-01-15T18:12:37.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.141+0000 7f1404c00640 1 -- 172.21.15.2:0/372987965 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+16635 (secure 0 0 0) 0x7f13f0017090 con 0x7f140815ec60 2024-01-15T18:12:37.146 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:37.146 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":103,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:12:33.684063+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:12:30.532826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-15T18:12:31.644235+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:12:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13e006c1f0 msgr2=0x7f13e006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13e006c1f0 0x7f13e006e6b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f13f8006fd0 tx=0x7f13f8008040 comp rx=0 tx=0).stop 2024-01-15T18:12:37.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 msgr2=0x7f140813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 0x7f140813dd00 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f13f000c410 tx=0x7f13f000c8d0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 shutdown_connections 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f13e006c1f0 0x7f13e006e6b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f140815f0e0 0x7f140813fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f140815ec60 0x7f140813dd00 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 --2- 172.21.15.2:0/372987965 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1408141880 0x7f14080b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 >> 172.21.15.2:0/372987965 conn(0x7f14080b0840 msgr2=0x7f14080af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.144+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 shutdown_connections 2024-01-15T18:12:37.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.145+0000 7f140d400640 1 -- 172.21.15.2:0/372987965 wait complete. 2024-01-15T18:12:37.166 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:12:37.166 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.377+0000 7f4b87e00640 1 Processor -- start 2024-01-15T18:12:37.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.377+0000 7f4b87e00640 1 -- start start 2024-01-15T18:12:37.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.378+0000 7f4b87e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4b8815ec60 0x7f4b8815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b87e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4b8806cad0 0x7f4b8806cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b87e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b880716d0 con 0x7f4b88141880 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b87e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b88071f20 con 0x7f4b8815ec60 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b87e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8806c6b0 con 0x7f4b8806cad0 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b86a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4b88141880 0x7f4b88070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60554/0 (socket says 172.21.15.2:60554) 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b86a00640 1 -- 172.21.15.2:0/4260509816 learned_addr learned my addr 172.21.15.2:0/4260509816 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:37.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b7de00640 1 --2- 172.21.15.2:0/4260509816 >> v2:172.21.15.2:3300/0 conn(0x7f4b8815ec60 0x7f4b8815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b87400640 1 --2- 172.21.15.2:0/4260509816 >> v2:172.21.15.137:3300/0 conn(0x7f4b8806cad0 0x7f4b8806cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b86000640 1 -- 172.21.15.2:0/4260509816 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 358812300 0 0) 0x7f4b880716d0 con 0x7f4b88141880 2024-01-15T18:12:37.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b7de00640 1 -- 172.21.15.2:0/4260509816 >> v2:172.21.15.137:3300/0 conn(0x7f4b8806cad0 msgr2=0x7f4b8806cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:37.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b7de00640 1 --2- 172.21.15.2:0/4260509816 >> v2:172.21.15.137:3300/0 conn(0x7f4b8806cad0 0x7f4b8806cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b7de00640 1 -- 172.21.15.2:0/4260509816 >> v1:172.21.15.2:6789/0 conn(0x7f4b88141880 legacy=0x7f4b88070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.379+0000 7f4b7de00640 1 -- 172.21.15.2:0/4260509816 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88071960 con 0x7f4b8815ec60 2024-01-15T18:12:37.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.381+0000 7f4b7de00640 1 --2- 172.21.15.2:0/4260509816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b8815ec60 0x7f4b8815f0e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f4b88070060 tx=0x7f4b7002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5f473174fa4e28 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:37.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.381+0000 7f4b86000640 1 -- 172.21.15.2:0/4260509816 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7002e070 con 0x7f4b8815ec60 2024-01-15T18:12:37.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.381+0000 7f4b86000640 1 -- 172.21.15.2:0/4260509816 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b700315a0 con 0x7f4b8815ec60 2024-01-15T18:12:37.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.381+0000 7f4b86000640 1 -- 172.21.15.2:0/4260509816 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70031930 con 0x7f4b8815ec60 2024-01-15T18:12:37.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.381+0000 7f4b87e00640 1 -- 172.21.15.2:0/4260509816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b8815ec60 msgr2=0x7f4b8815f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.381+0000 7f4b87e00640 1 --2- 172.21.15.2:0/4260509816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b8815ec60 0x7f4b8815f0e0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f4b88070060 tx=0x7f4b7002b840 comp rx=0 tx=0).stop 2024-01-15T18:12:37.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.382+0000 7f4b87e00640 1 -- 172.21.15.2:0/4260509816 shutdown_connections 2024-01-15T18:12:37.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.382+0000 7f4b87e00640 1 --2- 172.21.15.2:0/4260509816 >> v2:172.21.15.137:3300/0 conn(0x7f4b8806cad0 0x7f4b8806cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.382+0000 7f4b87e00640 1 --2- 172.21.15.2:0/4260509816 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b8815ec60 0x7f4b8815f0e0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.382+0000 7f4b87e00640 1 -- 172.21.15.2:0/4260509816 >> 172.21.15.2:0/4260509816 conn(0x7f4b880b0840 msgr2=0x7f4b880b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:37.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.382+0000 7f4b87e00640 1 -- 172.21.15.2:0/4260509816 shutdown_connections 2024-01-15T18:12:37.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.382+0000 7f4b87e00640 1 -- 172.21.15.2:0/4260509816 wait complete. 2024-01-15T18:12:37.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.383+0000 7f4b87e00640 1 Processor -- start 2024-01-15T18:12:37.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.384+0000 7f4b87e00640 1 -- start start 2024-01-15T18:12:37.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.384+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b8806cad0 0x7f4b880c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.384+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 0x7f4b8813e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b87e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b8815ec60 0x7f4b88143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b87e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8815a4a0 con 0x7f4b88141880 2024-01-15T18:12:37.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b87e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88159e00 con 0x7f4b8815ec60 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b87e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b88159780 con 0x7f4b8806cad0 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 0x7f4b8813e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 0x7f4b8813e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52174/0 (socket says 172.21.15.2:52174) 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 -- 172.21.15.2:0/1950728492 learned_addr learned my addr 172.21.15.2:0/1950728492 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b86a00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b8806cad0 0x7f4b880c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b87400640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b8815ec60 0x7f4b88143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 -- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b8806cad0 msgr2=0x7f4b880c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b8806cad0 0x7f4b880c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 -- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b8815ec60 msgr2=0x7f4b88143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b8815ec60 0x7f4b88143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.385+0000 7f4b7de00640 1 -- 172.21.15.2:0/1950728492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70009400 con 0x7f4b88141880 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.386+0000 7f4b7de00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 0x7f4b8813e580 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f4b70034040 tx=0x7f4b700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.386+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70004230 con 0x7f4b88141880 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.386+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b70031f00 con 0x7f4b88141880 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.386+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70032640 con 0x7f4b88141880 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.386+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8815a0c0 con 0x7f4b88141880 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.386+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b881441d0 con 0x7f4b88141880 2024-01-15T18:12:37.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.387+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b88074770 con 0x7f4b88141880 2024-01-15T18:12:37.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.388+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f4b700327a0 con 0x7f4b88141880 2024-01-15T18:12:37.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.389+0000 7f4b7f400640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6006c1f0 0x7f4b6006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.389+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f4b700aaa90 con 0x7f4b88141880 2024-01-15T18:12:37.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.390+0000 7f4b86a00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6006c1f0 0x7f4b6006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.391+0000 7f4b86a00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6006c1f0 0x7f4b6006e6b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4b780012e0 tx=0x7f4b78007ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:37.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.394+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b70077060 con 0x7f4b88141880 2024-01-15T18:12:37.551 INFO:tasks.ceph.mds.d.smithi137.stderr:2024-01-15T18:12:37.547+0000 7f2916000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:12:37.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.581+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b88159c80 con 0x7f4b6006c1f0 2024-01-15T18:12:37.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.608+0000 7f4b7f400640 1 -- 172.21.15.2:0/1950728492 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287276 (secure 0 0 0) 0x7f4b780079d0 con 0x7f4b6006c1f0 2024-01-15T18:12:37.616 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:37.618 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.613+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6006c1f0 msgr2=0x7f4b6006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.613+0000 7f4b87e00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6006c1f0 0x7f4b6006e6b0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f4b780012e0 tx=0x7f4b78007ed0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.613+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 msgr2=0x7f4b8813e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.613+0000 7f4b87e00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 0x7f4b8813e580 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f4b70034040 tx=0x7f4b700046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 shutdown_connections 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4b6006c1f0 0x7f4b6006e6b0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4b8815ec60 0x7f4b88143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4b88141880 0x7f4b8813e580 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 --2- 172.21.15.2:0/1950728492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4b8806cad0 0x7f4b880c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 >> 172.21.15.2:0/1950728492 conn(0x7f4b880b0840 msgr2=0x7f4b8815f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 shutdown_connections 2024-01-15T18:12:37.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.614+0000 7f4b87e00640 1 -- 172.21.15.2:0/1950728492 wait complete. 2024-01-15T18:12:37.637 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2024-01-15T18:12:35.763469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":535016,"kb_used_data":11120,"kb_used_omap":12,"kb_used_meta":523763,"kb_avail":754439704,"statfs":{"total":773094113280,"available":772546256896,"internally_reserved":0,"allocated":11386880,"data_stored":5608972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":536333873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.631433"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016456+0000","last_change":"2024-01-15T18:12:32.723167+0000","last_active":"2024-01-15T18:12:34.016456+0000","last_peered":"2024-01-15T18:12:34.016456+0000","last_clean":"2024-01-15T18:12:34.016456+0000","last_became_active":"2024-01-15T18:12:32.722883+0000","last_became_peered":"2024-01-15T18:12:32.722883+0000","last_unstale":"2024-01-15T18:12:34.016456+0000","last_undegraded":"2024-01-15T18:12:34.016456+0000","last_fullsized":"2024-01-15T18:12:34.016456+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:38.937762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690196+0000","last_change":"2024-01-15T18:12:32.722996+0000","last_active":"2024-01-15T18:12:33.690196+0000","last_peered":"2024-01-15T18:12:33.690196+0000","last_clean":"2024-01-15T18:12:33.690196+0000","last_became_active":"2024-01-15T18:12:32.722736+0000","last_became_peered":"2024-01-15T18:12:32.722736+0000","last_unstale":"2024-01-15T18:12:33.690196+0000","last_undegraded":"2024-01-15T18:12:33.690196+0000","last_fullsized":"2024-01-15T18:12:33.690196+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:09:31.258927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016493+0000","last_change":"2024-01-15T18:12:32.720166+0000","last_active":"2024-01-15T18:12:34.016493+0000","last_peered":"2024-01-15T18:12:34.016493+0000","last_clean":"2024-01-15T18:12:34.016493+0000","last_became_active":"2024-01-15T18:12:32.719812+0000","last_became_peered":"2024-01-15T18:12:32.719812+0000","last_unstale":"2024-01-15T18:12:34.016493+0000","last_undegraded":"2024-01-15T18:12:34.016493+0000","last_fullsized":"2024-01-15T18:12:34.016493+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:59.951955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241476+0000","last_change":"2024-01-15T18:12:32.717492+0000","last_active":"2024-01-15T18:12:34.241476+0000","last_peered":"2024-01-15T18:12:34.241476+0000","last_clean":"2024-01-15T18:12:34.241476+0000","last_became_active":"2024-01-15T18:12:32.717224+0000","last_became_peered":"2024-01-15T18:12:32.717224+0000","last_unstale":"2024-01-15T18:12:34.241476+0000","last_undegraded":"2024-01-15T18:12:34.241476+0000","last_fullsized":"2024-01-15T18:12:34.241476+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:06.892505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243002+0000","last_change":"2024-01-15T18:12:32.720577+0000","last_active":"2024-01-15T18:12:34.243002+0000","last_peered":"2024-01-15T18:12:34.243002+0000","last_clean":"2024-01-15T18:12:34.243002+0000","last_became_active":"2024-01-15T18:12:32.720255+0000","last_became_peered":"2024-01-15T18:12:32.720255+0000","last_unstale":"2024-01-15T18:12:34.243002+0000","last_undegraded":"2024-01-15T18:12:34.243002+0000","last_fullsized":"2024-01-15T18:12:34.243002+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:46.700338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242103+0000","last_change":"2024-01-15T18:12:32.717103+0000","last_active":"2024-01-15T18:12:34.242103+0000","last_peered":"2024-01-15T18:12:34.242103+0000","last_clean":"2024-01-15T18:12:34.242103+0000","last_became_active":"2024-01-15T18:12:32.716833+0000","last_became_peered":"2024-01-15T18:12:32.716833+0000","last_unstale":"2024-01-15T18:12:34.242103+0000","last_undegraded":"2024-01-15T18:12:34.242103+0000","last_fullsized":"2024-01-15T18:12:34.242103+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:45.679486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241916+0000","last_change":"2024-01-15T18:12:32.716841+0000","last_active":"2024-01-15T18:12:34.241916+0000","last_peered":"2024-01-15T18:12:34.241916+0000","last_clean":"2024-01-15T18:12:34.241916+0000","last_became_active":"2024-01-15T18:12:32.714917+0000","last_became_peered":"2024-01-15T18:12:32.714917+0000","last_unstale":"2024-01-15T18:12:34.241916+0000","last_undegraded":"2024-01-15T18:12:34.241916+0000","last_fullsized":"2024-01-15T18:12:34.241916+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:06.892505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690307+0000","last_change":"2024-01-15T18:12:32.716307+0000","last_active":"2024-01-15T18:12:33.690307+0000","last_peered":"2024-01-15T18:12:33.690307+0000","last_clean":"2024-01-15T18:12:33.690307+0000","last_became_active":"2024-01-15T18:12:32.715117+0000","last_became_peered":"2024-01-15T18:12:32.715117+0000","last_unstale":"2024-01-15T18:12:33.690307+0000","last_undegraded":"2024-01-15T18:12:33.690307+0000","last_fullsized":"2024-01-15T18:12:33.690307+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:59:58.622884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.699292+0000","last_change":"2024-01-15T18:12:32.724556+0000","last_active":"2024-01-15T18:12:33.699292+0000","last_peered":"2024-01-15T18:12:33.699292+0000","last_clean":"2024-01-15T18:12:33.699292+0000","last_became_active":"2024-01-15T18:12:32.724248+0000","last_became_peered":"2024-01-15T18:12:32.724248+0000","last_unstale":"2024-01-15T18:12:33.699292+0000","last_undegraded":"2024-01-15T18:12:33.699292+0000","last_fullsized":"2024-01-15T18:12:33.699292+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:51:53.729555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243938+0000","last_change":"2024-01-15T18:12:32.715758+0000","last_active":"2024-01-15T18:12:34.243938+0000","last_peered":"2024-01-15T18:12:34.243938+0000","last_clean":"2024-01-15T18:12:34.243938+0000","last_became_active":"2024-01-15T18:12:32.715029+0000","last_became_peered":"2024-01-15T18:12:32.715029+0000","last_unstale":"2024-01-15T18:12:34.243938+0000","last_undegraded":"2024-01-15T18:12:34.243938+0000","last_fullsized":"2024-01-15T18:12:34.243938+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:30.651263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242400+0000","last_change":"2024-01-15T18:12:32.722712+0000","last_active":"2024-01-15T18:12:34.242400+0000","last_peered":"2024-01-15T18:12:34.242400+0000","last_clean":"2024-01-15T18:12:34.242400+0000","last_became_active":"2024-01-15T18:12:32.721915+0000","last_became_peered":"2024-01-15T18:12:32.721915+0000","last_unstale":"2024-01-15T18:12:34.242400+0000","last_undegraded":"2024-01-15T18:12:34.242400+0000","last_fullsized":"2024-01-15T18:12:34.242400+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:16.896538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698549+0000","last_change":"2024-01-15T18:12:32.719251+0000","last_active":"2024-01-15T18:12:33.698549+0000","last_peered":"2024-01-15T18:12:33.698549+0000","last_clean":"2024-01-15T18:12:33.698549+0000","last_became_active":"2024-01-15T18:12:32.718985+0000","last_became_peered":"2024-01-15T18:12:32.718985+0000","last_unstale":"2024-01-15T18:12:33.698549+0000","last_undegraded":"2024-01-15T18:12:33.698549+0000","last_fullsized":"2024-01-15T18:12:33.698549+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:26:29.237185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241086+0000","last_change":"2024-01-15T18:12:32.723963+0000","last_active":"2024-01-15T18:12:34.241086+0000","last_peered":"2024-01-15T18:12:34.241086+0000","last_clean":"2024-01-15T18:12:34.241086+0000","last_became_active":"2024-01-15T18:12:32.722823+0000","last_became_peered":"2024-01-15T18:12:32.722823+0000","last_unstale":"2024-01-15T18:12:34.241086+0000","last_undegraded":"2024-01-15T18:12:34.241086+0000","last_fullsized":"2024-01-15T18:12:34.241086+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:13.244147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242030+0000","last_change":"2024-01-15T18:12:32.713875+0000","last_active":"2024-01-15T18:12:34.242030+0000","last_peered":"2024-01-15T18:12:34.242030+0000","last_clean":"2024-01-15T18:12:34.242030+0000","last_became_active":"2024-01-15T18:12:32.713550+0000","last_became_peered":"2024-01-15T18:12:32.713550+0000","last_unstale":"2024-01-15T18:12:34.242030+0000","last_undegraded":"2024-01-15T18:12:34.242030+0000","last_fullsized":"2024-01-15T18:12:34.242030+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:05.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240774+0000","last_change":"2024-01-15T18:12:32.710423+0000","last_active":"2024-01-15T18:12:34.240774+0000","last_peered":"2024-01-15T18:12:34.240774+0000","last_clean":"2024-01-15T18:12:34.240774+0000","last_became_active":"2024-01-15T18:12:32.710126+0000","last_became_peered":"2024-01-15T18:12:32.710126+0000","last_unstale":"2024-01-15T18:12:34.240774+0000","last_undegraded":"2024-01-15T18:12:34.240774+0000","last_fullsized":"2024-01-15T18:12:34.240774+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:41.748254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241912+0000","last_change":"2024-01-15T18:12:32.711048+0000","last_active":"2024-01-15T18:12:34.241912+0000","last_peered":"2024-01-15T18:12:34.241912+0000","last_clean":"2024-01-15T18:12:34.241912+0000","last_became_active":"2024-01-15T18:12:32.710511+0000","last_became_peered":"2024-01-15T18:12:32.710511+0000","last_unstale":"2024-01-15T18:12:34.241912+0000","last_undegraded":"2024-01-15T18:12:34.241912+0000","last_fullsized":"2024-01-15T18:12:34.241912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:04.235845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.691001+0000","last_change":"2024-01-15T18:12:32.723850+0000","last_active":"2024-01-15T18:12:33.691001+0000","last_peered":"2024-01-15T18:12:33.691001+0000","last_clean":"2024-01-15T18:12:33.691001+0000","last_became_active":"2024-01-15T18:12:32.723585+0000","last_became_peered":"2024-01-15T18:12:32.723585+0000","last_unstale":"2024-01-15T18:12:33.691001+0000","last_undegraded":"2024-01-15T18:12:33.691001+0000","last_fullsized":"2024-01-15T18:12:33.691001+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:59:25.099068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241466+0000","last_change":"2024-01-15T18:12:32.716397+0000","last_active":"2024-01-15T18:12:34.241466+0000","last_peered":"2024-01-15T18:12:34.241466+0000","last_clean":"2024-01-15T18:12:34.241466+0000","last_became_active":"2024-01-15T18:12:32.715478+0000","last_became_peered":"2024-01-15T18:12:32.715478+0000","last_unstale":"2024-01-15T18:12:34.241466+0000","last_undegraded":"2024-01-15T18:12:34.241466+0000","last_fullsized":"2024-01-15T18:12:34.241466+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:57:44.713007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243054+0000","last_change":"2024-01-15T18:12:32.715851+0000","last_active":"2024-01-15T18:12:34.243054+0000","last_peered":"2024-01-15T18:12:34.243054+0000","last_clean":"2024-01-15T18:12:34.243054+0000","last_became_active":"2024-01-15T18:12:32.715564+0000","last_became_peered":"2024-01-15T18:12:32.715564+0000","last_unstale":"2024-01-15T18:12:34.243054+0000","last_undegraded":"2024-01-15T18:12:34.243054+0000","last_fullsized":"2024-01-15T18:12:34.243054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:53.944268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698612+0000","last_change":"2024-01-15T18:12:32.724417+0000","last_active":"2024-01-15T18:12:33.698612+0000","last_peered":"2024-01-15T18:12:33.698612+0000","last_clean":"2024-01-15T18:12:33.698612+0000","last_became_active":"2024-01-15T18:12:32.723890+0000","last_became_peered":"2024-01-15T18:12:32.723890+0000","last_unstale":"2024-01-15T18:12:33.698612+0000","last_undegraded":"2024-01-15T18:12:33.698612+0000","last_fullsized":"2024-01-15T18:12:33.698612+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:46.400625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242523+0000","last_change":"2024-01-15T18:12:32.719872+0000","last_active":"2024-01-15T18:12:34.242523+0000","last_peered":"2024-01-15T18:12:34.242523+0000","last_clean":"2024-01-15T18:12:34.242523+0000","last_became_active":"2024-01-15T18:12:32.719480+0000","last_became_peered":"2024-01-15T18:12:32.719480+0000","last_unstale":"2024-01-15T18:12:34.242523+0000","last_undegraded":"2024-01-15T18:12:34.242523+0000","last_fullsized":"2024-01-15T18:12:34.242523+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:48.256212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689847+0000","last_change":"2024-01-15T18:12:32.718364+0000","last_active":"2024-01-15T18:12:33.689847+0000","last_peered":"2024-01-15T18:12:33.689847+0000","last_clean":"2024-01-15T18:12:33.689847+0000","last_became_active":"2024-01-15T18:12:32.718033+0000","last_became_peered":"2024-01-15T18:12:32.718033+0000","last_unstale":"2024-01-15T18:12:33.689847+0000","last_undegraded":"2024-01-15T18:12:33.689847+0000","last_fullsized":"2024-01-15T18:12:33.689847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:30:49.698394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689948+0000","last_change":"2024-01-15T18:12:32.720947+0000","last_active":"2024-01-15T18:12:33.689948+0000","last_peered":"2024-01-15T18:12:33.689948+0000","last_clean":"2024-01-15T18:12:33.689948+0000","last_became_active":"2024-01-15T18:12:32.720493+0000","last_became_peered":"2024-01-15T18:12:32.720493+0000","last_unstale":"2024-01-15T18:12:33.689948+0000","last_undegraded":"2024-01-15T18:12:33.689948+0000","last_fullsized":"2024-01-15T18:12:33.689948+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:30.651263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243319+0000","last_change":"2024-01-15T18:12:32.710019+0000","last_active":"2024-01-15T18:12:34.243319+0000","last_peered":"2024-01-15T18:12:34.243319+0000","last_clean":"2024-01-15T18:12:34.243319+0000","last_became_active":"2024-01-15T18:12:32.708465+0000","last_became_peered":"2024-01-15T18:12:32.708465+0000","last_unstale":"2024-01-15T18:12:34.243319+0000","last_undegraded":"2024-01-15T18:12:34.243319+0000","last_fullsized":"2024-01-15T18:12:34.243319+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:30.651263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.244356+0000","last_change":"2024-01-15T18:12:32.719091+0000","last_active":"2024-01-15T18:12:34.244356+0000","last_peered":"2024-01-15T18:12:34.244356+0000","last_clean":"2024-01-15T18:12:34.244356+0000","last_became_active":"2024-01-15T18:12:32.718798+0000","last_became_peered":"2024-01-15T18:12:32.718798+0000","last_unstale":"2024-01-15T18:12:34.244356+0000","last_undegraded":"2024-01-15T18:12:34.244356+0000","last_fullsized":"2024-01-15T18:12:34.244356+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:49:16.486465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690406+0000","last_change":"2024-01-15T18:12:32.721976+0000","last_active":"2024-01-15T18:12:33.690406+0000","last_peered":"2024-01-15T18:12:33.690406+0000","last_clean":"2024-01-15T18:12:33.690406+0000","last_became_active":"2024-01-15T18:12:32.721098+0000","last_became_peered":"2024-01-15T18:12:32.721098+0000","last_unstale":"2024-01-15T18:12:33.690406+0000","last_undegraded":"2024-01-15T18:12:33.690406+0000","last_fullsized":"2024-01-15T18:12:33.690406+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:10.680287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242763+0000","last_change":"2024-01-15T18:12:32.715205+0000","last_active":"2024-01-15T18:12:34.242763+0000","last_peered":"2024-01-15T18:12:34.242763+0000","last_clean":"2024-01-15T18:12:34.242763+0000","last_became_active":"2024-01-15T18:12:32.714830+0000","last_became_peered":"2024-01-15T18:12:32.714830+0000","last_unstale":"2024-01-15T18:12:34.242763+0000","last_undegraded":"2024-01-15T18:12:34.242763+0000","last_fullsized":"2024-01-15T18:12:34.242763+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:46.700338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243669+0000","last_change":"2024-01-15T18:12:32.719418+0000","last_active":"2024-01-15T18:12:34.243669+0000","last_peered":"2024-01-15T18:12:34.243669+0000","last_clean":"2024-01-15T18:12:34.243669+0000","last_became_active":"2024-01-15T18:12:32.719097+0000","last_became_peered":"2024-01-15T18:12:32.719097+0000","last_unstale":"2024-01-15T18:12:34.243669+0000","last_undegraded":"2024-01-15T18:12:34.243669+0000","last_fullsized":"2024-01-15T18:12:34.243669+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:33.019506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690284+0000","last_change":"2024-01-15T18:12:32.721827+0000","last_active":"2024-01-15T18:12:33.690284+0000","last_peered":"2024-01-15T18:12:33.690284+0000","last_clean":"2024-01-15T18:12:33.690284+0000","last_became_active":"2024-01-15T18:12:32.721517+0000","last_became_peered":"2024-01-15T18:12:32.721517+0000","last_unstale":"2024-01-15T18:12:33.690284+0000","last_undegraded":"2024-01-15T18:12:33.690284+0000","last_fullsized":"2024-01-15T18:12:33.690284+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:49:16.486465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242001+0000","last_change":"2024-01-15T18:12:32.716496+0000","last_active":"2024-01-15T18:12:34.242001+0000","last_peered":"2024-01-15T18:12:34.242001+0000","last_clean":"2024-01-15T18:12:34.242001+0000","last_became_active":"2024-01-15T18:12:32.716233+0000","last_became_peered":"2024-01-15T18:12:32.716233+0000","last_unstale":"2024-01-15T18:12:34.242001+0000","last_undegraded":"2024-01-15T18:12:34.242001+0000","last_fullsized":"2024-01-15T18:12:34.242001+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:52.894173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240705+0000","last_change":"2024-01-15T18:12:32.712505+0000","last_active":"2024-01-15T18:12:34.240705+0000","last_peered":"2024-01-15T18:12:34.240705+0000","last_clean":"2024-01-15T18:12:34.240705+0000","last_became_active":"2024-01-15T18:12:32.712167+0000","last_became_peered":"2024-01-15T18:12:32.712167+0000","last_unstale":"2024-01-15T18:12:34.240705+0000","last_undegraded":"2024-01-15T18:12:34.240705+0000","last_fullsized":"2024-01-15T18:12:34.240705+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:38.795536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015766+0000","last_change":"2024-01-15T18:12:32.709726+0000","last_active":"2024-01-15T18:12:34.015766+0000","last_peered":"2024-01-15T18:12:34.015766+0000","last_clean":"2024-01-15T18:12:34.015766+0000","last_became_active":"2024-01-15T18:12:32.709448+0000","last_became_peered":"2024-01-15T18:12:32.709448+0000","last_unstale":"2024-01-15T18:12:34.015766+0000","last_undegraded":"2024-01-15T18:12:34.015766+0000","last_fullsized":"2024-01-15T18:12:34.015766+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:44:24.987516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243490+0000","last_change":"2024-01-15T18:12:32.714781+0000","last_active":"2024-01-15T18:12:34.243490+0000","last_peered":"2024-01-15T18:12:34.243490+0000","last_clean":"2024-01-15T18:12:34.243490+0000","last_became_active":"2024-01-15T18:12:32.714499+0000","last_became_peered":"2024-01-15T18:12:32.714499+0000","last_unstale":"2024-01-15T18:12:34.243490+0000","last_undegraded":"2024-01-15T18:12:34.243490+0000","last_fullsized":"2024-01-15T18:12:34.243490+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:30.418661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"102'4","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690985+0000","last_change":"2024-01-15T18:12:32.174025+0000","last_active":"2024-01-15T18:12:33.690985+0000","last_peered":"2024-01-15T18:12:33.690985+0000","last_clean":"2024-01-15T18:12:33.690985+0000","last_became_active":"2024-01-15T18:12:32.172427+0000","last_became_peered":"2024-01-15T18:12:32.172427+0000","last_unstale":"2024-01-15T18:12:33.690985+0000","last_undegraded":"2024-01-15T18:12:33.690985+0000","last_fullsized":"2024-01-15T18:12:33.690985+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:32.003405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242172+0000","last_change":"2024-01-15T18:12:32.711420+0000","last_active":"2024-01-15T18:12:34.242172+0000","last_peered":"2024-01-15T18:12:34.242172+0000","last_clean":"2024-01-15T18:12:34.242172+0000","last_became_active":"2024-01-15T18:12:32.711010+0000","last_became_peered":"2024-01-15T18:12:32.711010+0000","last_unstale":"2024-01-15T18:12:34.242172+0000","last_undegraded":"2024-01-15T18:12:34.242172+0000","last_fullsized":"2024-01-15T18:12:34.242172+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:04.235845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242660+0000","last_change":"2024-01-15T18:12:32.175254+0000","last_active":"2024-01-15T18:12:34.242660+0000","last_peered":"2024-01-15T18:12:34.242660+0000","last_clean":"2024-01-15T18:12:34.242660+0000","last_became_active":"2024-01-15T18:12:32.174944+0000","last_became_peered":"2024-01-15T18:12:32.174944+0000","last_unstale":"2024-01-15T18:12:34.242660+0000","last_undegraded":"2024-01-15T18:12:34.242660+0000","last_fullsized":"2024-01-15T18:12:34.242660+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:37.779435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016329+0000","last_change":"2024-01-15T18:12:32.722958+0000","last_active":"2024-01-15T18:12:34.016329+0000","last_peered":"2024-01-15T18:12:34.016329+0000","last_clean":"2024-01-15T18:12:34.016329+0000","last_became_active":"2024-01-15T18:12:32.721379+0000","last_became_peered":"2024-01-15T18:12:32.721379+0000","last_unstale":"2024-01-15T18:12:34.016329+0000","last_undegraded":"2024-01-15T18:12:34.016329+0000","last_fullsized":"2024-01-15T18:12:34.016329+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:32.443345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690213+0000","last_change":"2024-01-15T18:12:32.173987+0000","last_active":"2024-01-15T18:12:33.690213+0000","last_peered":"2024-01-15T18:12:33.690213+0000","last_clean":"2024-01-15T18:12:33.690213+0000","last_became_active":"2024-01-15T18:12:32.173678+0000","last_became_peered":"2024-01-15T18:12:32.173678+0000","last_unstale":"2024-01-15T18:12:33.690213+0000","last_undegraded":"2024-01-15T18:12:33.690213+0000","last_fullsized":"2024-01-15T18:12:33.690213+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:44.663385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698591+0000","last_change":"2024-01-15T18:12:32.717351+0000","last_active":"2024-01-15T18:12:33.698591+0000","last_peered":"2024-01-15T18:12:33.698591+0000","last_clean":"2024-01-15T18:12:33.698591+0000","last_became_active":"2024-01-15T18:12:32.717050+0000","last_became_peered":"2024-01-15T18:12:32.717050+0000","last_unstale":"2024-01-15T18:12:33.698591+0000","last_undegraded":"2024-01-15T18:12:33.698591+0000","last_fullsized":"2024-01-15T18:12:33.698591+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:45.360374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690165+0000","last_change":"2024-01-15T18:12:32.169329+0000","last_active":"2024-01-15T18:12:33.690165+0000","last_peered":"2024-01-15T18:12:33.690165+0000","last_clean":"2024-01-15T18:12:33.690165+0000","last_became_active":"2024-01-15T18:12:32.168881+0000","last_became_peered":"2024-01-15T18:12:32.168881+0000","last_unstale":"2024-01-15T18:12:33.690165+0000","last_undegraded":"2024-01-15T18:12:33.690165+0000","last_fullsized":"2024-01-15T18:12:33.690165+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:54.945510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.696308+0000","last_change":"2024-01-15T18:12:32.717729+0000","last_active":"2024-01-15T18:12:33.696308+0000","last_peered":"2024-01-15T18:12:33.696308+0000","last_clean":"2024-01-15T18:12:33.696308+0000","last_became_active":"2024-01-15T18:12:32.717377+0000","last_became_peered":"2024-01-15T18:12:32.717377+0000","last_unstale":"2024-01-15T18:12:33.696308+0000","last_undegraded":"2024-01-15T18:12:33.696308+0000","last_fullsized":"2024-01-15T18:12:33.696308+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:51.651136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016181+0000","last_change":"2024-01-15T18:12:32.171924+0000","last_active":"2024-01-15T18:12:34.016181+0000","last_peered":"2024-01-15T18:12:34.016181+0000","last_clean":"2024-01-15T18:12:34.016181+0000","last_became_active":"2024-01-15T18:12:32.170824+0000","last_became_peered":"2024-01-15T18:12:32.170824+0000","last_unstale":"2024-01-15T18:12:34.016181+0000","last_undegraded":"2024-01-15T18:12:34.016181+0000","last_fullsized":"2024-01-15T18:12:34.016181+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:58:47.416771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242875+0000","last_change":"2024-01-15T18:12:32.712405+0000","last_active":"2024-01-15T18:12:34.242875+0000","last_peered":"2024-01-15T18:12:34.242875+0000","last_clean":"2024-01-15T18:12:34.242875+0000","last_became_active":"2024-01-15T18:12:32.711993+0000","last_became_peered":"2024-01-15T18:12:32.711993+0000","last_unstale":"2024-01-15T18:12:34.242875+0000","last_undegraded":"2024-01-15T18:12:34.242875+0000","last_fullsized":"2024-01-15T18:12:34.242875+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:33.019506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689923+0000","last_change":"2024-01-15T18:12:32.169450+0000","last_active":"2024-01-15T18:12:33.689923+0000","last_peered":"2024-01-15T18:12:33.689923+0000","last_clean":"2024-01-15T18:12:33.689923+0000","last_became_active":"2024-01-15T18:12:32.169195+0000","last_became_peered":"2024-01-15T18:12:32.169195+0000","last_unstale":"2024-01-15T18:12:33.689923+0000","last_undegraded":"2024-01-15T18:12:33.689923+0000","last_fullsized":"2024-01-15T18:12:33.689923+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:12.228046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240671+0000","last_change":"2024-01-15T18:12:32.715702+0000","last_active":"2024-01-15T18:12:34.240671+0000","last_peered":"2024-01-15T18:12:34.240671+0000","last_clean":"2024-01-15T18:12:34.240671+0000","last_became_active":"2024-01-15T18:12:32.711262+0000","last_became_peered":"2024-01-15T18:12:32.711262+0000","last_unstale":"2024-01-15T18:12:34.240671+0000","last_undegraded":"2024-01-15T18:12:34.240671+0000","last_fullsized":"2024-01-15T18:12:34.240671+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:55.634124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243119+0000","last_change":"2024-01-15T18:12:32.177242+0000","last_active":"2024-01-15T18:12:34.243119+0000","last_peered":"2024-01-15T18:12:34.243119+0000","last_clean":"2024-01-15T18:12:34.243119+0000","last_became_active":"2024-01-15T18:12:32.176980+0000","last_became_peered":"2024-01-15T18:12:32.176980+0000","last_unstale":"2024-01-15T18:12:34.243119+0000","last_undegraded":"2024-01-15T18:12:34.243119+0000","last_fullsized":"2024-01-15T18:12:34.243119+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:40:58.665583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.697912+0000","last_change":"2024-01-15T18:12:32.719081+0000","last_active":"2024-01-15T18:12:33.697912+0000","last_peered":"2024-01-15T18:12:33.697912+0000","last_clean":"2024-01-15T18:12:33.697912+0000","last_became_active":"2024-01-15T18:12:32.718763+0000","last_became_peered":"2024-01-15T18:12:32.718763+0000","last_unstale":"2024-01-15T18:12:33.697912+0000","last_undegraded":"2024-01-15T18:12:33.697912+0000","last_fullsized":"2024-01-15T18:12:33.697912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:45:04.246875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015826+0000","last_change":"2024-01-15T18:12:32.175060+0000","last_active":"2024-01-15T18:12:34.015826+0000","last_peered":"2024-01-15T18:12:34.015826+0000","last_clean":"2024-01-15T18:12:34.015826+0000","last_became_active":"2024-01-15T18:12:32.174748+0000","last_became_peered":"2024-01-15T18:12:32.174748+0000","last_unstale":"2024-01-15T18:12:34.015826+0000","last_undegraded":"2024-01-15T18:12:34.015826+0000","last_fullsized":"2024-01-15T18:12:34.015826+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:06.107711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690626+0000","last_change":"2024-01-15T18:12:32.726315+0000","last_active":"2024-01-15T18:12:33.690626+0000","last_peered":"2024-01-15T18:12:33.690626+0000","last_clean":"2024-01-15T18:12:33.690626+0000","last_became_active":"2024-01-15T18:12:32.725362+0000","last_became_peered":"2024-01-15T18:12:32.725362+0000","last_unstale":"2024-01-15T18:12:33.690626+0000","last_undegraded":"2024-01-15T18:12:33.690626+0000","last_fullsized":"2024-01-15T18:12:33.690626+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:36.011229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016746+0000","last_change":"2024-01-15T18:12:32.167152+0000","last_active":"2024-01-15T18:12:34.016746+0000","last_peered":"2024-01-15T18:12:34.016746+0000","last_clean":"2024-01-15T18:12:34.016746+0000","last_became_active":"2024-01-15T18:12:32.166871+0000","last_became_peered":"2024-01-15T18:12:32.166871+0000","last_unstale":"2024-01-15T18:12:34.016746+0000","last_undegraded":"2024-01-15T18:12:34.016746+0000","last_fullsized":"2024-01-15T18:12:34.016746+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:17.514303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.699299+0000","last_change":"2024-01-15T18:12:32.721080+0000","last_active":"2024-01-15T18:12:33.699299+0000","last_peered":"2024-01-15T18:12:33.699299+0000","last_clean":"2024-01-15T18:12:33.699299+0000","last_became_active":"2024-01-15T18:12:32.720789+0000","last_became_peered":"2024-01-15T18:12:32.720789+0000","last_unstale":"2024-01-15T18:12:33.699299+0000","last_undegraded":"2024-01-15T18:12:33.699299+0000","last_fullsized":"2024-01-15T18:12:33.699299+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:56.725359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241498+0000","last_change":"2024-01-15T18:12:32.172750+0000","last_active":"2024-01-15T18:12:34.241498+0000","last_peered":"2024-01-15T18:12:34.241498+0000","last_clean":"2024-01-15T18:12:34.241498+0000","last_became_active":"2024-01-15T18:12:32.172475+0000","last_became_peered":"2024-01-15T18:12:32.172475+0000","last_unstale":"2024-01-15T18:12:34.241498+0000","last_undegraded":"2024-01-15T18:12:34.241498+0000","last_fullsized":"2024-01-15T18:12:34.241498+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:09.953872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243887+0000","last_change":"2024-01-15T18:12:32.726087+0000","last_active":"2024-01-15T18:12:34.243887+0000","last_peered":"2024-01-15T18:12:34.243887+0000","last_clean":"2024-01-15T18:12:34.243887+0000","last_became_active":"2024-01-15T18:12:32.725614+0000","last_became_peered":"2024-01-15T18:12:32.725614+0000","last_unstale":"2024-01-15T18:12:34.243887+0000","last_undegraded":"2024-01-15T18:12:34.243887+0000","last_fullsized":"2024-01-15T18:12:34.243887+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:30:49.698394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"102'3","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243302+0000","last_change":"2024-01-15T18:12:32.171231+0000","last_active":"2024-01-15T18:12:34.243302+0000","last_peered":"2024-01-15T18:12:34.243302+0000","last_clean":"2024-01-15T18:12:34.243302+0000","last_became_active":"2024-01-15T18:12:32.170238+0000","last_became_peered":"2024-01-15T18:12:32.170238+0000","last_unstale":"2024-01-15T18:12:34.243302+0000","last_undegraded":"2024-01-15T18:12:34.243302+0000","last_fullsized":"2024-01-15T18:12:34.243302+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:57:43.696906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243081+0000","last_change":"2024-01-15T18:12:32.711064+0000","last_active":"2024-01-15T18:12:34.243081+0000","last_peered":"2024-01-15T18:12:34.243081+0000","last_clean":"2024-01-15T18:12:34.243081+0000","last_became_active":"2024-01-15T18:12:32.710623+0000","last_became_peered":"2024-01-15T18:12:32.710623+0000","last_unstale":"2024-01-15T18:12:34.243081+0000","last_undegraded":"2024-01-15T18:12:34.243081+0000","last_fullsized":"2024-01-15T18:12:34.243081+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:30:49.698394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"102'6","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698821+0000","last_change":"2024-01-15T18:12:32.169172+0000","last_active":"2024-01-15T18:12:33.698821+0000","last_peered":"2024-01-15T18:12:33.698821+0000","last_clean":"2024-01-15T18:12:33.698821+0000","last_became_active":"2024-01-15T18:12:32.168902+0000","last_became_peered":"2024-01-15T18:12:32.168902+0000","last_unstale":"2024-01-15T18:12:33.698821+0000","last_undegraded":"2024-01-15T18:12:33.698821+0000","last_fullsized":"2024-01-15T18:12:33.698821+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:46:24.548644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241082+0000","last_change":"2024-01-15T18:12:32.702338+0000","last_active":"2024-01-15T18:12:34.241082+0000","last_peered":"2024-01-15T18:12:34.241082+0000","last_clean":"2024-01-15T18:12:34.241082+0000","last_became_active":"2024-01-15T18:12:32.700501+0000","last_became_peered":"2024-01-15T18:12:32.700501+0000","last_unstale":"2024-01-15T18:12:34.241082+0000","last_undegraded":"2024-01-15T18:12:34.241082+0000","last_fullsized":"2024-01-15T18:12:34.241082+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:34:17.158053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242794+0000","last_change":"2024-01-15T18:12:32.155905+0000","last_active":"2024-01-15T18:12:34.242794+0000","last_peered":"2024-01-15T18:12:34.242794+0000","last_clean":"2024-01-15T18:12:34.242794+0000","last_became_active":"2024-01-15T18:12:32.155514+0000","last_became_peered":"2024-01-15T18:12:32.155514+0000","last_unstale":"2024-01-15T18:12:34.242794+0000","last_undegraded":"2024-01-15T18:12:34.242794+0000","last_fullsized":"2024-01-15T18:12:34.242794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:13:08.319290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015895+0000","last_change":"2024-01-15T18:12:32.718072+0000","last_active":"2024-01-15T18:12:34.015895+0000","last_peered":"2024-01-15T18:12:34.015895+0000","last_clean":"2024-01-15T18:12:34.015895+0000","last_became_active":"2024-01-15T18:12:32.717809+0000","last_became_peered":"2024-01-15T18:12:32.717809+0000","last_unstale":"2024-01-15T18:12:34.015895+0000","last_undegraded":"2024-01-15T18:12:34.015895+0000","last_fullsized":"2024-01-15T18:12:34.015895+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:47:08.699545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242476+0000","last_change":"2024-01-15T18:12:32.167849+0000","last_active":"2024-01-15T18:12:34.242476+0000","last_peered":"2024-01-15T18:12:34.242476+0000","last_clean":"2024-01-15T18:12:34.242476+0000","last_became_active":"2024-01-15T18:12:32.167507+0000","last_became_peered":"2024-01-15T18:12:32.167507+0000","last_unstale":"2024-01-15T18:12:34.242476+0000","last_undegraded":"2024-01-15T18:12:34.242476+0000","last_fullsized":"2024-01-15T18:12:34.242476+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:26:44.737791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.697675+0000","last_change":"2024-01-15T18:12:32.720100+0000","last_active":"2024-01-15T18:12:33.697675+0000","last_peered":"2024-01-15T18:12:33.697675+0000","last_clean":"2024-01-15T18:12:33.697675+0000","last_became_active":"2024-01-15T18:12:32.719713+0000","last_became_peered":"2024-01-15T18:12:32.719713+0000","last_unstale":"2024-01-15T18:12:33.697675+0000","last_undegraded":"2024-01-15T18:12:33.697675+0000","last_fullsized":"2024-01-15T18:12:33.697675+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:14.698019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698130+0000","last_change":"2024-01-15T18:12:32.166734+0000","last_active":"2024-01-15T18:12:33.698130+0000","last_peered":"2024-01-15T18:12:33.698130+0000","last_clean":"2024-01-15T18:12:33.698130+0000","last_became_active":"2024-01-15T18:12:32.165620+0000","last_became_peered":"2024-01-15T18:12:32.165620+0000","last_unstale":"2024-01-15T18:12:33.698130+0000","last_undegraded":"2024-01-15T18:12:33.698130+0000","last_fullsized":"2024-01-15T18:12:33.698130+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:45.537131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241696+0000","last_change":"2024-01-15T18:12:32.723410+0000","last_active":"2024-01-15T18:12:34.241696+0000","last_peered":"2024-01-15T18:12:34.241696+0000","last_clean":"2024-01-15T18:12:34.241696+0000","last_became_active":"2024-01-15T18:12:32.723110+0000","last_became_peered":"2024-01-15T18:12:32.723110+0000","last_unstale":"2024-01-15T18:12:34.241696+0000","last_undegraded":"2024-01-15T18:12:34.241696+0000","last_fullsized":"2024-01-15T18:12:34.241696+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:28.945346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"102'8","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015921+0000","last_change":"2024-01-15T18:12:32.169474+0000","last_active":"2024-01-15T18:12:34.015921+0000","last_peered":"2024-01-15T18:12:34.015921+0000","last_clean":"2024-01-15T18:12:34.015921+0000","last_became_active":"2024-01-15T18:12:32.169195+0000","last_became_peered":"2024-01-15T18:12:32.169195+0000","last_unstale":"2024-01-15T18:12:34.015921+0000","last_undegraded":"2024-01-15T18:12:34.015921+0000","last_fullsized":"2024-01-15T18:12:34.015921+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:19.616756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.244288+0000","last_change":"2024-01-15T18:12:32.722879+0000","last_active":"2024-01-15T18:12:34.244288+0000","last_peered":"2024-01-15T18:12:34.244288+0000","last_clean":"2024-01-15T18:12:34.244288+0000","last_became_active":"2024-01-15T18:12:32.722355+0000","last_became_peered":"2024-01-15T18:12:32.722355+0000","last_unstale":"2024-01-15T18:12:34.244288+0000","last_undegraded":"2024-01-15T18:12:34.244288+0000","last_fullsized":"2024-01-15T18:12:34.244288+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:09:31.258927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698709+0000","last_change":"2024-01-15T18:12:32.168558+0000","last_active":"2024-01-15T18:12:33.698709+0000","last_peered":"2024-01-15T18:12:33.698709+0000","last_clean":"2024-01-15T18:12:33.698709+0000","last_became_active":"2024-01-15T18:12:32.168212+0000","last_became_peered":"2024-01-15T18:12:32.168212+0000","last_unstale":"2024-01-15T18:12:33.698709+0000","last_undegraded":"2024-01-15T18:12:33.698709+0000","last_fullsized":"2024-01-15T18:12:33.698709+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:16.367691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.696327+0000","last_change":"2024-01-15T18:12:32.712788+0000","last_active":"2024-01-15T18:12:33.696327+0000","last_peered":"2024-01-15T18:12:33.696327+0000","last_clean":"2024-01-15T18:12:33.696327+0000","last_became_active":"2024-01-15T18:12:32.712490+0000","last_became_peered":"2024-01-15T18:12:32.712490+0000","last_unstale":"2024-01-15T18:12:33.696327+0000","last_undegraded":"2024-01-15T18:12:33.696327+0000","last_fullsized":"2024-01-15T18:12:33.696327+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:59.486284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243703+0000","last_change":"2024-01-15T18:12:32.169139+0000","last_active":"2024-01-15T18:12:34.243703+0000","last_peered":"2024-01-15T18:12:34.243703+0000","last_clean":"2024-01-15T18:12:34.243703+0000","last_became_active":"2024-01-15T18:12:32.168723+0000","last_became_peered":"2024-01-15T18:12:32.168723+0000","last_unstale":"2024-01-15T18:12:34.243703+0000","last_undegraded":"2024-01-15T18:12:34.243703+0000","last_fullsized":"2024-01-15T18:12:34.243703+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:12.228046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243075+0000","last_change":"2024-01-15T18:12:32.722973+0000","last_active":"2024-01-15T18:12:34.243075+0000","last_peered":"2024-01-15T18:12:34.243075+0000","last_clean":"2024-01-15T18:12:34.243075+0000","last_became_active":"2024-01-15T18:12:32.721390+0000","last_became_peered":"2024-01-15T18:12:32.721390+0000","last_unstale":"2024-01-15T18:12:34.243075+0000","last_undegraded":"2024-01-15T18:12:34.243075+0000","last_fullsized":"2024-01-15T18:12:34.243075+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:32.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690305+0000","last_change":"2024-01-15T18:12:32.169155+0000","last_active":"2024-01-15T18:12:33.690305+0000","last_peered":"2024-01-15T18:12:33.690305+0000","last_clean":"2024-01-15T18:12:33.690305+0000","last_became_active":"2024-01-15T18:12:32.168513+0000","last_became_peered":"2024-01-15T18:12:32.168513+0000","last_unstale":"2024-01-15T18:12:33.690305+0000","last_undegraded":"2024-01-15T18:12:33.690305+0000","last_fullsized":"2024-01-15T18:12:33.690305+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:03.219744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242261+0000","last_change":"2024-01-15T18:12:32.718945+0000","last_active":"2024-01-15T18:12:34.242261+0000","last_peered":"2024-01-15T18:12:34.242261+0000","last_clean":"2024-01-15T18:12:34.242261+0000","last_became_active":"2024-01-15T18:12:32.718082+0000","last_became_peered":"2024-01-15T18:12:32.718082+0000","last_unstale":"2024-01-15T18:12:34.242261+0000","last_undegraded":"2024-01-15T18:12:34.242261+0000","last_fullsized":"2024-01-15T18:12:34.242261+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:08.935819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690420+0000","last_change":"2024-01-15T18:12:32.176147+0000","last_active":"2024-01-15T18:12:33.690420+0000","last_peered":"2024-01-15T18:12:33.690420+0000","last_clean":"2024-01-15T18:12:33.690420+0000","last_became_active":"2024-01-15T18:12:32.175874+0000","last_became_peered":"2024-01-15T18:12:32.175874+0000","last_unstale":"2024-01-15T18:12:33.690420+0000","last_undegraded":"2024-01-15T18:12:33.690420+0000","last_fullsized":"2024-01-15T18:12:33.690420+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:57:43.696906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242653+0000","last_change":"2024-01-15T18:12:32.717201+0000","last_active":"2024-01-15T18:12:34.242653+0000","last_peered":"2024-01-15T18:12:34.242653+0000","last_clean":"2024-01-15T18:12:34.242653+0000","last_became_active":"2024-01-15T18:12:32.716936+0000","last_became_peered":"2024-01-15T18:12:32.716936+0000","last_unstale":"2024-01-15T18:12:34.242653+0000","last_undegraded":"2024-01-15T18:12:34.242653+0000","last_fullsized":"2024-01-15T18:12:34.242653+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:09.350690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.696364+0000","last_change":"2024-01-15T18:12:32.167360+0000","last_active":"2024-01-15T18:12:33.696364+0000","last_peered":"2024-01-15T18:12:33.696364+0000","last_clean":"2024-01-15T18:12:33.696364+0000","last_became_active":"2024-01-15T18:12:32.166211+0000","last_became_peered":"2024-01-15T18:12:32.166211+0000","last_unstale":"2024-01-15T18:12:33.696364+0000","last_undegraded":"2024-01-15T18:12:33.696364+0000","last_fullsized":"2024-01-15T18:12:33.696364+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:10.531632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241750+0000","last_change":"2024-01-15T18:12:32.724280+0000","last_active":"2024-01-15T18:12:34.241750+0000","last_peered":"2024-01-15T18:12:34.241750+0000","last_clean":"2024-01-15T18:12:34.241750+0000","last_became_active":"2024-01-15T18:12:32.721970+0000","last_became_peered":"2024-01-15T18:12:32.721970+0000","last_unstale":"2024-01-15T18:12:34.241750+0000","last_undegraded":"2024-01-15T18:12:34.241750+0000","last_fullsized":"2024-01-15T18:12:34.241750+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:04.065239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.695973+0000","last_change":"2024-01-15T18:12:32.168557+0000","last_active":"2024-01-15T18:12:33.695973+0000","last_peered":"2024-01-15T18:12:33.695973+0000","last_clean":"2024-01-15T18:12:33.695973+0000","last_became_active":"2024-01-15T18:12:32.167826+0000","last_became_peered":"2024-01-15T18:12:32.167826+0000","last_unstale":"2024-01-15T18:12:33.695973+0000","last_undegraded":"2024-01-15T18:12:33.695973+0000","last_fullsized":"2024-01-15T18:12:33.695973+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:30.036119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242301+0000","last_change":"2024-01-15T18:12:32.711184+0000","last_active":"2024-01-15T18:12:34.242301+0000","last_peered":"2024-01-15T18:12:34.242301+0000","last_clean":"2024-01-15T18:12:34.242301+0000","last_became_active":"2024-01-15T18:12:32.709348+0000","last_became_peered":"2024-01-15T18:12:32.709348+0000","last_unstale":"2024-01-15T18:12:34.242301+0000","last_undegraded":"2024-01-15T18:12:34.242301+0000","last_fullsized":"2024-01-15T18:12:34.242301+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:45.679486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689918+0000","last_change":"2024-01-15T18:12:32.170820+0000","last_active":"2024-01-15T18:12:33.689918+0000","last_peered":"2024-01-15T18:12:33.689918+0000","last_clean":"2024-01-15T18:12:33.689918+0000","last_became_active":"2024-01-15T18:12:32.170532+0000","last_became_peered":"2024-01-15T18:12:32.170532+0000","last_unstale":"2024-01-15T18:12:33.689918+0000","last_undegraded":"2024-01-15T18:12:33.689918+0000","last_fullsized":"2024-01-15T18:12:33.689918+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:40:58.665583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690635+0000","last_change":"2024-01-15T18:12:32.722139+0000","last_active":"2024-01-15T18:12:33.690635+0000","last_peered":"2024-01-15T18:12:33.690635+0000","last_clean":"2024-01-15T18:12:33.690635+0000","last_became_active":"2024-01-15T18:12:32.721670+0000","last_became_peered":"2024-01-15T18:12:32.721670+0000","last_unstale":"2024-01-15T18:12:33.690635+0000","last_undegraded":"2024-01-15T18:12:33.690635+0000","last_fullsized":"2024-01-15T18:12:33.690635+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:25.982795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.697354+0000","last_change":"2024-01-15T18:12:32.159866+0000","last_active":"2024-01-15T18:12:33.697354+0000","last_peered":"2024-01-15T18:12:33.697354+0000","last_clean":"2024-01-15T18:12:33.697354+0000","last_became_active":"2024-01-15T18:12:32.159482+0000","last_became_peered":"2024-01-15T18:12:32.159482+0000","last_unstale":"2024-01-15T18:12:33.697354+0000","last_undegraded":"2024-01-15T18:12:33.697354+0000","last_fullsized":"2024-01-15T18:12:33.697354+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:06:59.081912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016672+0000","last_change":"2024-01-15T18:12:32.722572+0000","last_active":"2024-01-15T18:12:34.016672+0000","last_peered":"2024-01-15T18:12:34.016672+0000","last_clean":"2024-01-15T18:12:34.016672+0000","last_became_active":"2024-01-15T18:12:32.714975+0000","last_became_peered":"2024-01-15T18:12:32.714975+0000","last_unstale":"2024-01-15T18:12:34.016672+0000","last_undegraded":"2024-01-15T18:12:34.016672+0000","last_fullsized":"2024-01-15T18:12:34.016672+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:16:53.650301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690770+0000","last_change":"2024-01-15T18:12:32.175271+0000","last_active":"2024-01-15T18:12:33.690770+0000","last_peered":"2024-01-15T18:12:33.690770+0000","last_clean":"2024-01-15T18:12:33.690770+0000","last_became_active":"2024-01-15T18:12:32.174807+0000","last_became_peered":"2024-01-15T18:12:32.174807+0000","last_unstale":"2024-01-15T18:12:33.690770+0000","last_undegraded":"2024-01-15T18:12:33.690770+0000","last_fullsized":"2024-01-15T18:12:33.690770+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:45.684237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016615+0000","last_change":"2024-01-15T18:12:32.717689+0000","last_active":"2024-01-15T18:12:34.016615+0000","last_peered":"2024-01-15T18:12:34.016615+0000","last_clean":"2024-01-15T18:12:34.016615+0000","last_became_active":"2024-01-15T18:12:32.714629+0000","last_became_peered":"2024-01-15T18:12:32.714629+0000","last_unstale":"2024-01-15T18:12:34.016615+0000","last_undegraded":"2024-01-15T18:12:34.016615+0000","last_fullsized":"2024-01-15T18:12:34.016615+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:12:14.192962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690349+0000","last_change":"2024-01-15T18:12:32.171453+0000","last_active":"2024-01-15T18:12:33.690349+0000","last_peered":"2024-01-15T18:12:33.690349+0000","last_clean":"2024-01-15T18:12:33.690349+0000","last_became_active":"2024-01-15T18:12:32.171105+0000","last_became_peered":"2024-01-15T18:12:32.171105+0000","last_unstale":"2024-01-15T18:12:33.690349+0000","last_undegraded":"2024-01-15T18:12:33.690349+0000","last_fullsized":"2024-01-15T18:12:33.690349+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:15.880437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243754+0000","last_change":"2024-01-15T18:12:32.720622+0000","last_active":"2024-01-15T18:12:34.243754+0000","last_peered":"2024-01-15T18:12:34.243754+0000","last_clean":"2024-01-15T18:12:34.243754+0000","last_became_active":"2024-01-15T18:12:32.720292+0000","last_became_peered":"2024-01-15T18:12:32.720292+0000","last_unstale":"2024-01-15T18:12:34.243754+0000","last_undegraded":"2024-01-15T18:12:34.243754+0000","last_fullsized":"2024-01-15T18:12:34.243754+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:32.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243901+0000","last_change":"2024-01-15T18:12:32.170644+0000","last_active":"2024-01-15T18:12:34.243901+0000","last_peered":"2024-01-15T18:12:34.243901+0000","last_clean":"2024-01-15T18:12:34.243901+0000","last_became_active":"2024-01-15T18:12:32.170352+0000","last_became_peered":"2024-01-15T18:12:32.170352+0000","last_unstale":"2024-01-15T18:12:34.243901+0000","last_undegraded":"2024-01-15T18:12:34.243901+0000","last_fullsized":"2024-01-15T18:12:34.243901+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:05.876403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242267+0000","last_change":"2024-01-15T18:12:32.724247+0000","last_active":"2024-01-15T18:12:34.242267+0000","last_peered":"2024-01-15T18:12:34.242267+0000","last_clean":"2024-01-15T18:12:34.242267+0000","last_became_active":"2024-01-15T18:12:32.723775+0000","last_became_peered":"2024-01-15T18:12:32.723775+0000","last_unstale":"2024-01-15T18:12:34.242267+0000","last_undegraded":"2024-01-15T18:12:34.242267+0000","last_fullsized":"2024-01-15T18:12:34.242267+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:09:23.111371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690495+0000","last_change":"2024-01-15T18:12:32.171656+0000","last_active":"2024-01-15T18:12:33.690495+0000","last_peered":"2024-01-15T18:12:33.690495+0000","last_clean":"2024-01-15T18:12:33.690495+0000","last_became_active":"2024-01-15T18:12:32.170575+0000","last_became_peered":"2024-01-15T18:12:32.170575+0000","last_unstale":"2024-01-15T18:12:33.690495+0000","last_undegraded":"2024-01-15T18:12:33.690495+0000","last_fullsized":"2024-01-15T18:12:33.690495+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:40.732153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698060+0000","last_change":"2024-01-15T18:12:32.716821+0000","last_active":"2024-01-15T18:12:33.698060+0000","last_peered":"2024-01-15T18:12:33.698060+0000","last_clean":"2024-01-15T18:12:33.698060+0000","last_became_active":"2024-01-15T18:12:32.716516+0000","last_became_peered":"2024-01-15T18:12:32.716516+0000","last_unstale":"2024-01-15T18:12:33.698060+0000","last_undegraded":"2024-01-15T18:12:33.698060+0000","last_fullsized":"2024-01-15T18:12:33.698060+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:42.724333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242470+0000","last_change":"2024-01-15T18:12:32.174584+0000","last_active":"2024-01-15T18:12:34.242470+0000","last_peered":"2024-01-15T18:12:34.242470+0000","last_clean":"2024-01-15T18:12:34.242470+0000","last_became_active":"2024-01-15T18:12:32.174154+0000","last_became_peered":"2024-01-15T18:12:32.174154+0000","last_unstale":"2024-01-15T18:12:34.242470+0000","last_undegraded":"2024-01-15T18:12:34.242470+0000","last_fullsized":"2024-01-15T18:12:34.242470+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:54.618023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241792+0000","last_change":"2024-01-15T18:12:32.722270+0000","last_active":"2024-01-15T18:12:34.241792+0000","last_peered":"2024-01-15T18:12:34.241792+0000","last_clean":"2024-01-15T18:12:34.241792+0000","last_became_active":"2024-01-15T18:12:32.721573+0000","last_became_peered":"2024-01-15T18:12:32.721573+0000","last_unstale":"2024-01-15T18:12:34.241792+0000","last_undegraded":"2024-01-15T18:12:34.241792+0000","last_fullsized":"2024-01-15T18:12:34.241792+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:19:43.275580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240819+0000","last_change":"2024-01-15T18:12:32.170398+0000","last_active":"2024-01-15T18:12:34.240819+0000","last_peered":"2024-01-15T18:12:34.240819+0000","last_clean":"2024-01-15T18:12:34.240819+0000","last_became_active":"2024-01-15T18:12:32.168116+0000","last_became_peered":"2024-01-15T18:12:32.168116+0000","last_unstale":"2024-01-15T18:12:34.240819+0000","last_undegraded":"2024-01-15T18:12:34.240819+0000","last_fullsized":"2024-01-15T18:12:34.240819+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:58.348630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242709+0000","last_change":"2024-01-15T18:12:32.712912+0000","last_active":"2024-01-15T18:12:34.242709+0000","last_peered":"2024-01-15T18:12:34.242709+0000","last_clean":"2024-01-15T18:12:34.242709+0000","last_became_active":"2024-01-15T18:12:32.712628+0000","last_became_peered":"2024-01-15T18:12:32.712628+0000","last_unstale":"2024-01-15T18:12:34.242709+0000","last_undegraded":"2024-01-15T18:12:34.242709+0000","last_fullsized":"2024-01-15T18:12:34.242709+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:16.896538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241999+0000","last_change":"2024-01-15T18:12:32.166071+0000","last_active":"2024-01-15T18:12:34.241999+0000","last_peered":"2024-01-15T18:12:34.241999+0000","last_clean":"2024-01-15T18:12:34.241999+0000","last_became_active":"2024-01-15T18:12:32.165790+0000","last_became_peered":"2024-01-15T18:12:32.165790+0000","last_unstale":"2024-01-15T18:12:34.241999+0000","last_undegraded":"2024-01-15T18:12:34.241999+0000","last_fullsized":"2024-01-15T18:12:34.241999+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:01.361758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243727+0000","last_change":"2024-01-15T18:12:32.722187+0000","last_active":"2024-01-15T18:12:34.243727+0000","last_peered":"2024-01-15T18:12:34.243727+0000","last_clean":"2024-01-15T18:12:34.243727+0000","last_became_active":"2024-01-15T18:12:32.721917+0000","last_became_peered":"2024-01-15T18:12:32.721917+0000","last_unstale":"2024-01-15T18:12:34.243727+0000","last_undegraded":"2024-01-15T18:12:34.243727+0000","last_fullsized":"2024-01-15T18:12:34.243727+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:36.412516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242951+0000","last_change":"2024-01-15T18:12:32.174284+0000","last_active":"2024-01-15T18:12:34.242951+0000","last_peered":"2024-01-15T18:12:34.242951+0000","last_clean":"2024-01-15T18:12:34.242951+0000","last_became_active":"2024-01-15T18:12:32.174011+0000","last_became_peered":"2024-01-15T18:12:32.174011+0000","last_unstale":"2024-01-15T18:12:34.242951+0000","last_undegraded":"2024-01-15T18:12:34.242951+0000","last_fullsized":"2024-01-15T18:12:34.242951+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:47.806633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":240,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-15T18:12:32.693861+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:12:32.693861+0000","last_peered":"2024-01-15T18:12:32.693861+0000","last_clean":"2024-01-15T18:12:32.693861+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:12:32.693861+0000","last_undegraded":"2024-01-15T18:12:32.693861+0000","last_fullsized":"2024-01-15T18:12:32.693861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542328,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64880,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306960,"statfs":{"total":96636764160,"available":96570327040,"internally_reserved":0,"allocated":1871872,"data_stored":1145010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.329,"15min":1.4039999999999999},"min":{"1min":0.747,"5min":0.63800000000000001,"15min":0.53600000000000003},"max":{"1min":1.9359999999999999,"5min":4.0019999999999998,"15min":14.114000000000001},"last":1.482},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.7090000000000001,"15min":1.5700000000000001},"min":{"1min":0.94199999999999995,"5min":0.872,"15min":0.51700000000000002},"max":{"1min":2.0059999999999998,"5min":27.32,"15min":27.32},"last":1.304}]},{"osd":1,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.3740000000000001,"15min":1.5069999999999999},"min":{"1min":0.93999999999999995,"5min":0.72699999999999998,"15min":0.40300000000000002},"max":{"1min":5.9690000000000003,"5min":8.3030000000000008,"15min":29.167999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3109999999999999,"15min":1.3979999999999999},"min":{"1min":0.88200000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7969999999999999,"5min":4.2699999999999996,"15min":12.565},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.3049999999999999,"15min":1.4870000000000001},"min":{"1min":0.97099999999999997,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.6439999999999999,"5min":4.1920000000000002,"15min":13.801},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.651,"15min":1.5289999999999999},"min":{"1min":0.82099999999999995,"5min":0.76400000000000001,"15min":0.56599999999999995},"max":{"1min":3.5920000000000001,"5min":27.358000000000001,"15min":27.358000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.544,"15min":1.6319999999999999},"min":{"1min":1.0700000000000001,"5min":0.72199999999999998,"15min":0.47099999999999997},"max":{"1min":6.133,"5min":10.476000000000001,"15min":29.204999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4730000000000001,"15min":1.5740000000000001},"min":{"1min":1.006,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.101,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.03}]},{"osd":4,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.6910000000000001,"15min":1.581},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":1.5329999999999999,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.135},{"interface":"front","average":{"1min":1.907,"5min":1.5649999999999999,"15min":1.6830000000000001},"min":{"1min":0.88500000000000001,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":6.4710000000000001,"5min":8.4199999999999999,"15min":29.25},"last":1.3680000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.5229999999999999,"15min":1.6539999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.885,"5min":6.4960000000000004,"15min":12.518000000000001},"last":1.611},{"interface":"front","average":{"1min":1.405,"5min":1.847,"15min":1.6930000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":1.728,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.421}]},{"osd":6,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.976,"15min":2.3460000000000001},"min":{"1min":1.2030000000000001,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":1.8160000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.776,"5min":1.726,"15min":1.835},"min":{"1min":1.1719999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":6.1870000000000003,"5min":8.1980000000000004,"15min":29.106999999999999},"last":1.536}]}]},{"osd":6,"up_from":14,"seq":60129542328,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64292,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307548,"statfs":{"total":96636764160,"available":96570929152,"internally_reserved":0,"allocated":1265664,"data_stored":548770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.294,"15min":1.2310000000000001},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.438},"max":{"1min":8.8350000000000009,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.292,"15min":1.3540000000000001},"min":{"1min":0.71799999999999997,"5min":0.61099999999999999,"15min":0.59599999999999997},"max":{"1min":1.851,"5min":3.1659999999999999,"15min":21.565000000000001},"last":0.75}]},{"osd":1,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.4410000000000001,"15min":1.3740000000000001},"min":{"1min":0.88300000000000001,"5min":0.71099999999999997,"15min":0.52100000000000002},"max":{"1min":8.8870000000000005,"5min":8.8870000000000005,"15min":8.8870000000000005},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.3300000000000001,"15min":1.4199999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4970000000000001,"5min":2.73,"15min":13.984999999999999},"last":1.1259999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.335,"15min":1.4530000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.5249999999999999,"5min":4.351,"15min":34.670000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.4890000000000001,"15min":1.4370000000000001},"min":{"1min":0.92400000000000004,"5min":0.90200000000000002,"15min":0.49399999999999999},"max":{"1min":8.9359999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":0.93200000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4870000000000001,"15min":1.5680000000000001},"min":{"1min":0.96899999999999997,"5min":0.81499999999999995,"15min":0.68500000000000005},"max":{"1min":1.732,"5min":3.5680000000000001,"15min":16.855},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.5489999999999999,"15min":1.6779999999999999},"min":{"1min":1.137,"5min":0.89100000000000001,"15min":0.48499999999999999},"max":{"1min":1.897,"5min":5.1379999999999999,"15min":34.725000000000001},"last":1.2889999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.456,"15min":1.462},"min":{"1min":0.92000000000000004,"5min":0.80600000000000005,"15min":0.38400000000000001},"max":{"1min":8.7539999999999996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.581},{"interface":"front","average":{"1min":1.333,"5min":1.5049999999999999,"15min":1.5109999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.501},"max":{"1min":1.976,"5min":7.3230000000000004,"15min":10.118},"last":1.359}]},{"osd":5,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":2.117,"5min":1.653,"15min":1.7050000000000001},"min":{"1min":0.92000000000000004,"5min":0.82699999999999996,"15min":0.57699999999999996},"max":{"1min":16.309999999999999,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.427},{"interface":"front","average":{"1min":1.857,"5min":1.698,"15min":1.631},"min":{"1min":1.141,"5min":0.80500000000000005,"15min":0.47899999999999998},"max":{"1min":8.9900000000000002,"5min":8.9900000000000002,"15min":10.077},"last":1.502}]},{"osd":7,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.623,"15min":1.645},"min":{"1min":1.111,"5min":1.111,"15min":0.64000000000000001},"max":{"1min":2.0379999999999998,"5min":5.5060000000000002,"15min":10.302},"last":1.462},{"interface":"front","average":{"1min":1.512,"5min":1.635,"15min":1.8260000000000001},"min":{"1min":1.048,"5min":0.94499999999999995,"15min":0.52600000000000002},"max":{"1min":2.008,"5min":5.298,"15min":34.762999999999998},"last":1.5269999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542330,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58792,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313048,"statfs":{"total":96636764160,"available":96576561152,"internally_reserved":0,"allocated":1273856,"data_stored":554600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.292,"15min":1.381},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":1.6419999999999999,"5min":8.0950000000000006,"15min":26.077999999999999},"last":1.23},{"interface":"front","average":{"1min":1.7569999999999999,"5min":1.4590000000000001,"15min":1.49},"min":{"1min":0.84799999999999998,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":9.8699999999999992,"5min":9.8699999999999992,"15min":14.711},"last":1.091}]},{"osd":1,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.393,"15min":1.413},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.8279999999999998,"5min":8.702,"15min":26},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3740000000000001,"15min":1.3700000000000001},"min":{"1min":0.751,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":2.8660000000000001,"5min":7.5309999999999997,"15min":12.619},"last":1.0129999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4850000000000001,"15min":1.544},"min":{"1min":0.95599999999999996,"5min":0.64300000000000002,"15min":0.55700000000000005},"max":{"1min":1.651,"5min":8.7739999999999991,"15min":26.234999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.744,"5min":1.6599999999999999,"15min":1.6299999999999999},"min":{"1min":1.05,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":10.09,"5min":10.09,"15min":17.126999999999999},"last":1.2769999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.321,"15min":1.446},"min":{"1min":0.95799999999999996,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.841,"5min":7.6150000000000002,"15min":12.722},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.5029999999999999,"15min":1.643},"min":{"1min":1.0509999999999999,"5min":0.70399999999999996,"15min":0.63400000000000001},"max":{"1min":1.8380000000000001,"5min":8.9730000000000008,"15min":26.134},"last":0.84599999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.409,"15min":1.5369999999999999},"min":{"1min":0.995,"5min":0.442,"15min":0.442},"max":{"1min":9.9979999999999993,"5min":9.9979999999999993,"15min":25.776},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.7549999999999999,"15min":1.766},"min":{"1min":1.1080000000000001,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":12.103,"5min":15.52,"15min":16.414000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.4159999999999999,"15min":1.5389999999999999},"min":{"1min":0.871,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":7.9240000000000004,"15min":12.769},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5389999999999999,"15min":1.802},"min":{"1min":0.99299999999999999,"5min":0.72899999999999998,"15min":0.435},"max":{"1min":2.0259999999999998,"5min":10.571999999999999,"15min":36.087000000000003},"last":1.3240000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":1.8480000000000001,"15min":1.9239999999999999},"min":{"1min":1.2649999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":12.326000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.494},{"interface":"front","average":{"1min":1.427,"5min":1.601,"15min":1.7709999999999999},"min":{"1min":1.04,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":1.875,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.391}]}]},{"osd":4,"up_from":14,"seq":60129542329,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64316,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307524,"statfs":{"total":96636764160,"available":96570904576,"internally_reserved":0,"allocated":1290240,"data_stored":556266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.238,"15min":1.379},"min":{"1min":0.57699999999999996,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":3.4100000000000001,"5min":4.8170000000000002,"15min":21.643000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.272,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":0.87,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.7130000000000001,"5min":4.5590000000000002,"15min":11.196},"last":1.004}]},{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.6020000000000001,"15min":1.4079999999999999},"min":{"1min":0.84399999999999997,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.919,"5min":28.663,"15min":28.663},"last":1.516},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.3919999999999999,"15min":1.52},"min":{"1min":0.89300000000000002,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":3.5299999999999998,"5min":6.4870000000000001,"15min":21.440000000000001},"last":1.145}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.5289999999999999,"15min":1.46},"min":{"1min":0.80100000000000005,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.9570000000000001,"5min":16.023,"15min":16.023},"last":1.3069999999999999},{"interface":"front","average":{"1min":2.371,"5min":1.581,"15min":1.589},"min":{"1min":0.751,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":18.858000000000001,"5min":18.858000000000001,"15min":22.350999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.369,"15min":1.506},"min":{"1min":0.59099999999999997,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":2.1230000000000002,"5min":5.0860000000000003,"15min":15.093},"last":1.486},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.7030000000000001,"15min":1.667},"min":{"1min":0.89200000000000002,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":2.073,"5min":18.027000000000001,"15min":19.486000000000001},"last":0.96999999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.4330000000000001,"15min":1.778},"min":{"1min":0.5,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":3.6429999999999998,"5min":5.0270000000000001,"15min":54.938000000000002},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.5449999999999999,"15min":1.524},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":1.992,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.2010000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.7470000000000001,"15min":1.631},"min":{"1min":0.72599999999999998,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.347,"5min":16.103999999999999,"15min":17.733000000000001},"last":1.242},{"interface":"front","average":{"1min":1.681,"5min":1.7729999999999999,"15min":1.871},"min":{"1min":0.621,"5min":0.621,"15min":0.499},"max":{"1min":3.7280000000000002,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.4079999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.863,"15min":1.8240000000000001},"min":{"1min":0.82199999999999995,"5min":0.75800000000000001,"15min":0.45100000000000001},"max":{"1min":2.468,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.9550000000000001,"15min":1.8680000000000001},"min":{"1min":1.369,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.2410000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.4510000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542330,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72632,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299208,"statfs":{"total":96636764160,"available":96562388992,"internally_reserved":0,"allocated":1286144,"data_stored":556164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.94699999999999995,"15min":1.0700000000000001},"min":{"1min":0.66400000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.347,"5min":2.0350000000000001,"15min":36.408000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.0660000000000001,"15min":1.052},"min":{"1min":0.72299999999999998,"5min":0.42399999999999999,"15min":0.377},"max":{"1min":1.6839999999999999,"5min":3.1440000000000001,"15min":5.0529999999999999},"last":0.83099999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.079,"15min":1.1080000000000001},"min":{"1min":0.80800000000000005,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.9590000000000001,"5min":4.1109999999999998,"15min":8.1359999999999992},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.202,"15min":1.2010000000000001},"min":{"1min":0.79200000000000004,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":2.5790000000000002,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.486,"15min":1.2929999999999999},"min":{"1min":1.0529999999999999,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":2.0859999999999999,"5min":14.025,"15min":14.025},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.3720000000000001,"15min":1.3049999999999999},"min":{"1min":0.72099999999999997,"5min":0.58099999999999996,"15min":0.46000000000000002},"max":{"1min":2.7130000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.022}]},{"osd":4,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.4810000000000001,"15min":1.4390000000000001},"min":{"1min":1.0649999999999999,"5min":0.67400000000000004,"15min":0.59999999999999998},"max":{"1min":1.9379999999999999,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.744,"5min":1.718,"15min":1.575},"min":{"1min":0.97799999999999998,"5min":0.81599999999999995,"15min":0.69499999999999995},"max":{"1min":2.6230000000000002,"5min":15.769,"15min":15.769},"last":1.079}]},{"osd":5,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":1.645,"15min":1.5780000000000001},"min":{"1min":1.22,"5min":0.77100000000000002,"15min":0.65000000000000002},"max":{"1min":2.7440000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.609},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.839,"15min":1.921},"min":{"1min":1.2589999999999999,"5min":0.97599999999999998,"15min":0.58299999999999996},"max":{"1min":2.6659999999999999,"5min":21.713000000000001,"15min":40.384999999999998},"last":1.1870000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":2.1190000000000002,"15min":1.673},"min":{"1min":1.2170000000000001,"5min":0.69599999999999995,"15min":0.51400000000000001},"max":{"1min":3.6800000000000002,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.744,"15min":1.7370000000000001},"min":{"1min":1.3620000000000001,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.863,"5min":6.8639999999999999,"15min":16.195},"last":1.2769999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9159999999999999,"15min":1.9039999999999999},"min":{"1min":1.1850000000000001,"5min":0.88300000000000001,"15min":0.63900000000000001},"max":{"1min":2.798,"5min":10.035,"15min":24.146999999999998},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.659,"5min":1.9990000000000001,"15min":1.7470000000000001},"min":{"1min":1.196,"5min":0.82699999999999996,"15min":0.56100000000000005},"max":{"1min":2.411,"5min":17.622,"15min":17.622},"last":1.5640000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542329,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68448,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303392,"statfs":{"total":96636764160,"available":96566673408,"internally_reserved":0,"allocated":1261568,"data_stored":548152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.95099999999999996,"15min":1.002},"min":{"1min":0.60299999999999998,"5min":0.44400000000000001,"15min":0.30099999999999999},"max":{"1min":1.7350000000000001,"5min":3.5670000000000002,"15min":6.101},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1259999999999999},"min":{"1min":0.65600000000000003,"5min":0.41199999999999998,"15min":0.28299999999999997},"max":{"1min":1.946,"5min":10.319000000000001,"15min":10.319000000000001},"last":0.95499999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0549999999999999,"15min":1.1100000000000001},"min":{"1min":0.72099999999999997,"5min":0.48099999999999998,"15min":0.379},"max":{"1min":1.984,"5min":4.4939999999999998,"15min":10.807},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2549999999999999,"15min":1.2210000000000001},"min":{"1min":0.82899999999999996,"5min":0.58799999999999997,"15min":0.35299999999999998},"max":{"1min":2.1190000000000002,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":1.097}]},{"osd":3,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.3220000000000001,"15min":1.4590000000000001},"min":{"1min":0.70599999999999996,"5min":0.61499999999999999,"15min":0.41399999999999998},"max":{"1min":1.7110000000000001,"5min":4.5839999999999996,"15min":25.891999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.216,"15min":1.266},"min":{"1min":0.89400000000000002,"5min":0.47199999999999998,"15min":0.376},"max":{"1min":2.1640000000000001,"5min":2.6899999999999999,"15min":14.946},"last":1.044}]},{"osd":4,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4259999999999999,"15min":1.5229999999999999},"min":{"1min":1.0609999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.8700000000000001,"5min":4.8159999999999998,"15min":9.0679999999999996},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.716,"15min":1.587},"min":{"1min":1.0609999999999999,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":8.4269999999999996,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.248}]},{"osd":5,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.5640000000000001,"15min":1.671},"min":{"1min":1.0740000000000001,"5min":0.71799999999999997,"15min":0.63500000000000001},"max":{"1min":1.905,"5min":7.4939999999999998,"15min":11.071999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.5660000000000001,"15min":1.7150000000000001},"min":{"1min":1.1200000000000001,"5min":0.83799999999999997,"15min":0.55200000000000005},"max":{"1min":1.9810000000000001,"5min":5.774,"15min":37.020000000000003},"last":1.4670000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":2.0179999999999998,"5min":1.7410000000000001,"15min":1.728},"min":{"1min":1.1859999999999999,"5min":0.85199999999999998,"15min":0.64800000000000002},"max":{"1min":8.5519999999999996,"5min":10.185,"15min":23.762},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.653,"5min":1.913,"15min":1.889},"min":{"1min":1.3,"5min":1.0129999999999999,"15min":0.73499999999999999},"max":{"1min":2.1709999999999998,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.286}]},{"osd":7,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":1.8120000000000001,"15min":1.9570000000000001},"min":{"1min":1.4199999999999999,"5min":1.0169999999999999,"15min":0.68999999999999995},"max":{"1min":8.6620000000000008,"5min":10.24,"15min":52.767000000000003},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.696,"5min":1.873,"15min":1.821},"min":{"1min":1.24,"5min":0.97899999999999998,"15min":0.83499999999999996},"max":{"1min":2.3039999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":1.153}]}]},{"osd":1,"up_from":14,"seq":60129542329,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72620,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299220,"statfs":{"total":96636764160,"available":96562401280,"internally_reserved":0,"allocated":1269760,"data_stored":554530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.2270000000000001,"15min":1.0549999999999999},"min":{"1min":0.63800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.2640000000000002,"5min":19.218,"15min":19.218},"last":1.151},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.0900000000000001,"15min":1.1919999999999999},"min":{"1min":0.45200000000000001,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":2.371,"5min":4.6349999999999998,"15min":19.823},"last":0.94099999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.0209999999999999,"15min":1.1140000000000001},"min":{"1min":0.746,"5min":0.35499999999999998,"15min":0.31900000000000001},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":7.6680000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.27},"min":{"1min":1.1379999999999999,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":1.829,"5min":19.318000000000001,"15min":19.318000000000001},"last":1.2749999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.5209999999999999,"15min":1.3859999999999999},"min":{"1min":1.081,"5min":0.51000000000000001,"15min":0.40899999999999997},"max":{"1min":2.4289999999999998,"5min":19.387,"15min":19.387},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1839999999999999,"15min":1.343},"min":{"1min":0.58499999999999996,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.5589999999999999,"5min":3.552,"15min":19.733000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1829999999999998,"5min":1.802,"15min":1.5920000000000001},"min":{"1min":1.2949999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":8.0559999999999992,"5min":19.456,"15min":19.456},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.46,"15min":1.667},"min":{"1min":1.169,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.8480000000000001,"5min":3.3180000000000001,"15min":22.204000000000001},"last":1.591}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4530000000000001,"15min":1.6779999999999999},"min":{"1min":0.91300000000000003,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":2.145,"5min":2.351,"15min":20.012},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.4510000000000001,"15min":1.6379999999999999},"min":{"1min":1.0980000000000001,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":19.998999999999999},"last":1.548}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.776,"15min":1.7130000000000001},"min":{"1min":1.3839999999999999,"5min":0.82099999999999995,"15min":0.54100000000000004},"max":{"1min":2.4870000000000001,"5min":19.526,"15min":19.526},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.597,"15min":1.8180000000000001},"min":{"1min":1.1040000000000001,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":2.2679999999999998,"5min":6.468,"15min":27.579999999999998},"last":1.2210000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.746,"15min":1.8120000000000001},"min":{"1min":1.155,"5min":0.63,"15min":0.505},"max":{"1min":2.3130000000000002,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.738},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.9119999999999999,"15min":1.8340000000000001},"min":{"1min":1.3919999999999999,"5min":0.92400000000000004,"15min":0.54500000000000004},"max":{"1min":2.4319999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.6839999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542328,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69036,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302804,"statfs":{"total":96636764160,"available":96566071296,"internally_reserved":0,"allocated":1867776,"data_stored":1145480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.0369999999999999,"15min":1.1839999999999999},"min":{"1min":0.56200000000000006,"5min":0.34200000000000003,"15min":0.30599999999999999},"max":{"1min":14.542,"5min":14.542,"15min":20.437999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.109,"5min":1.147,"15min":1.2110000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.32400000000000001},"max":{"1min":1.9430000000000001,"5min":3.492,"15min":35.137999999999998},"last":0.92000000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9450000000000001,"5min":1.284,"15min":1.3480000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.42799999999999999},"max":{"1min":14.651,"5min":14.651,"15min":20.556000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.264,"15min":1.3740000000000001},"min":{"1min":0.70699999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.2930000000000001,"5min":4.7430000000000003,"15min":35.183},"last":1.2190000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.1060000000000001,"15min":1.155},"min":{"1min":0.627,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":1.792,"5min":2.6789999999999998,"15min":8.9949999999999992},"last":1.161},{"interface":"front","average":{"1min":1.974,"5min":1.462,"15min":1.6200000000000001},"min":{"1min":0.64800000000000002,"5min":0.47199999999999998,"15min":0.45100000000000001},"max":{"1min":14.598000000000001,"5min":14.598000000000001,"15min":25.372},"last":0.76000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4870000000000001,"15min":1.6699999999999999},"min":{"1min":0.86199999999999999,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":2.0680000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.399},{"interface":"front","average":{"1min":1.385,"5min":1.369,"15min":1.452},"min":{"1min":0.96499999999999997,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":2.1179999999999999,"5min":2.407,"15min":15.968999999999999},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.645,"15min":1.621},"min":{"1min":1.099,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":2.133,"5min":7.9160000000000004,"15min":12.173999999999999},"last":1.661},{"interface":"front","average":{"1min":2.2320000000000002,"5min":1.78,"15min":1.8440000000000001},"min":{"1min":0.88300000000000001,"5min":0.85299999999999998,"15min":0.76000000000000001},"max":{"1min":14.859,"5min":14.859,"15min":20.789999999999999},"last":1.2569999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":2.3210000000000002,"5min":1.9419999999999999,"15min":2.032},"min":{"1min":1.141,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":15.022,"5min":16.023,"15min":36.390999999999998},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.669,"15min":1.9139999999999999},"min":{"1min":0.99399999999999999,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":2.2759999999999998,"5min":5.3239999999999998,"15min":36.207000000000001},"last":1.7210000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.7130000000000001,"15min":1.9350000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":2.1819999999999999,"5min":9.8580000000000005,"15min":35.636000000000003},"last":1.845},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.7769999999999999,"15min":1.665},"min":{"1min":1.129,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0529999999999999,"5min":11.470000000000001,"15min":11.470000000000001},"last":1.552}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:37.642 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:12:37.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.898+0000 7f7749c00640 1 Processor -- start 2024-01-15T18:12:37.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.898+0000 7f7749c00640 1 -- start start 2024-01-15T18:12:37.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.899+0000 7f7749c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f77440ce5c0 0x7f774415d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.899+0000 7f7749c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f774406d1c0 0x7f774406d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.899+0000 7f7749c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774406cd30 con 0x7f7744159690 2024-01-15T18:12:37.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.899+0000 7f7749c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77440b3e20 con 0x7f774406d1c0 2024-01-15T18:12:37.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.899+0000 7f7749c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7744070d60 con 0x7f77440ce5c0 2024-01-15T18:12:37.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7743e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7744159690 0x7f774415d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60562/0 (socket says 172.21.15.2:60562) 2024-01-15T18:12:37.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7743e00640 1 -- 172.21.15.2:0/767113649 learned_addr learned my addr 172.21.15.2:0/767113649 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:37.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7749200640 1 --2- 172.21.15.2:0/767113649 >> v2:172.21.15.2:3300/0 conn(0x7f774406d1c0 0x7f774406d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7743400640 1 --2- 172.21.15.2:0/767113649 >> v2:172.21.15.137:3300/0 conn(0x7f77440ce5c0 0x7f774415d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7742a00640 1 -- 172.21.15.2:0/767113649 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991745794 0 0) 0x7f774406cd30 con 0x7f7744159690 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7742a00640 1 -- 172.21.15.2:0/767113649 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7724003520 con 0x7f7744159690 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7749200640 1 -- 172.21.15.2:0/767113649 >> v2:172.21.15.137:3300/0 conn(0x7f77440ce5c0 msgr2=0x7f774415d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7749200640 1 --2- 172.21.15.2:0/767113649 >> v2:172.21.15.137:3300/0 conn(0x7f77440ce5c0 0x7f774415d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7749200640 1 -- 172.21.15.2:0/767113649 >> v1:172.21.15.2:6789/0 conn(0x7f7744159690 legacy=0x7f774415d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.900+0000 7f7749200640 1 -- 172.21.15.2:0/767113649 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744071040 con 0x7f774406d1c0 2024-01-15T18:12:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.901+0000 7f7749200640 1 --2- 172.21.15.2:0/767113649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f774406d1c0 0x7f774406d640 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f77440b3970 tx=0x7f773c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e36ab6755bf71f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7742a00640 1 -- 172.21.15.2:0/767113649 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c02ee90 con 0x7f774406d1c0 2024-01-15T18:12:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7742a00640 1 -- 172.21.15.2:0/767113649 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c036440 con 0x7f774406d1c0 2024-01-15T18:12:37.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7742a00640 1 -- 172.21.15.2:0/767113649 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c0367d0 con 0x7f774406d1c0 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 -- 172.21.15.2:0/767113649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f774406d1c0 msgr2=0x7f774406d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 --2- 172.21.15.2:0/767113649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f774406d1c0 0x7f774406d640 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f77440b3970 tx=0x7f773c02e0e0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 -- 172.21.15.2:0/767113649 shutdown_connections 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 --2- 172.21.15.2:0/767113649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f774406d1c0 0x7f774406d640 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 --2- 172.21.15.2:0/767113649 >> v2:172.21.15.137:3300/0 conn(0x7f77440ce5c0 0x7f774415d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 -- 172.21.15.2:0/767113649 >> 172.21.15.2:0/767113649 conn(0x7f77440b0840 msgr2=0x7f77440b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 -- 172.21.15.2:0/767113649 shutdown_connections 2024-01-15T18:12:37.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.902+0000 7f7749c00640 1 -- 172.21.15.2:0/767113649 wait complete. 2024-01-15T18:12:37.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.904+0000 7f7749c00640 1 Processor -- start 2024-01-15T18:12:37.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.904+0000 7f7749c00640 1 -- start start 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.905+0000 7f7749c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f774406d1c0 0x7f774415a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.905+0000 7f7749c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77440ce5c0 0x7f7744148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.905+0000 7f7749c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 0x7f77440b2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.905+0000 7f7749c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7744151ce0 con 0x7f7744159690 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.905+0000 7f7749c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77441f1fd0 con 0x7f77440ce5c0 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.905+0000 7f7749c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77441f22b0 con 0x7f774406d1c0 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 0x7f77440b2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 0x7f77440b2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52182/0 (socket says 172.21.15.2:52182) 2024-01-15T18:12:37.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 -- 172.21.15.2:0/289536677 learned_addr learned my addr 172.21.15.2:0/289536677 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7743e00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f774406d1c0 0x7f774415a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 -- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f774406d1c0 msgr2=0x7f774415a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f774406d1c0 0x7f774415a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7743400640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77440ce5c0 0x7f7744148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 -- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77440ce5c0 msgr2=0x7f7744148050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77440ce5c0 0x7f7744148050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 -- 172.21.15.2:0/289536677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c009a70 con 0x7f7744159690 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7743400640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77440ce5c0 0x7f7744148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:37.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.906+0000 7f7749200640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 0x7f77440b2cc0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f773c009e80 tx=0x7f773c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.907+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c042860 con 0x7f7744159690 2024-01-15T18:12:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.907+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c0429e0 con 0x7f7744159690 2024-01-15T18:12:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.907+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c042d90 con 0x7f7744159690 2024-01-15T18:12:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.907+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77441f2590 con 0x7f7744159690 2024-01-15T18:12:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.907+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7744148590 con 0x7f7744159690 2024-01-15T18:12:37.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.908+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77441f2ab0 con 0x7f7744159690 2024-01-15T18:12:37.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.909+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f773c040400 con 0x7f7744159690 2024-01-15T18:12:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.910+0000 7f773be00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771c06c1f0 0x7f771c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.910+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f773c0ad950 con 0x7f7744159690 2024-01-15T18:12:37.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.911+0000 7f7743e00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771c06c1f0 0x7f771c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:37.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.912+0000 7f7743e00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771c06c1f0 0x7f771c06e6b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f77340011e0 tx=0x7f7734009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:37.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:37.915+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77441f2ab0 con 0x7f7744159690 2024-01-15T18:12:38.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.108+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77440cbab0 con 0x7f771c06c1f0 2024-01-15T18:12:38.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.131+0000 7f773be00640 1 -- 172.21.15.2:0/289536677 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287276 (secure 0 0 0) 0x7f7734005890 con 0x7f771c06c1f0 2024-01-15T18:12:38.139 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:38.141 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:12:38.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771c06c1f0 msgr2=0x7f771c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771c06c1f0 0x7f771c06e6b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f77340011e0 tx=0x7f7734009040 comp rx=0 tx=0).stop 2024-01-15T18:12:38.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 msgr2=0x7f77440b2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 0x7f77440b2cc0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f773c009e80 tx=0x7f773c004a40 comp rx=0 tx=0).stop 2024-01-15T18:12:38.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 shutdown_connections 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f771c06c1f0 0x7f771c06e6b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7744159690 0x7f77440b2cc0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f77440ce5c0 0x7f7744148050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 --2- 172.21.15.2:0/289536677 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f774406d1c0 0x7f774415a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 >> 172.21.15.2:0/289536677 conn(0x7f77440b0840 msgr2=0x7f77440afb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 shutdown_connections 2024-01-15T18:12:38.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.139+0000 7f7749c00640 1 -- 172.21.15.2:0/289536677 wait complete. 2024-01-15T18:12:38.162 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":553,"stamp":"2024-01-15T18:12:37.764427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":164,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":535016,"kb_used_data":11120,"kb_used_omap":12,"kb_used_meta":523763,"kb_avail":754439704,"statfs":{"total":773094113280,"available":772546256896,"internally_reserved":0,"allocated":11386880,"data_stored":5608972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":536333873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002224"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016456+0000","last_change":"2024-01-15T18:12:32.723167+0000","last_active":"2024-01-15T18:12:34.016456+0000","last_peered":"2024-01-15T18:12:34.016456+0000","last_clean":"2024-01-15T18:12:34.016456+0000","last_became_active":"2024-01-15T18:12:32.722883+0000","last_became_peered":"2024-01-15T18:12:32.722883+0000","last_unstale":"2024-01-15T18:12:34.016456+0000","last_undegraded":"2024-01-15T18:12:34.016456+0000","last_fullsized":"2024-01-15T18:12:34.016456+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:19:38.937762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690196+0000","last_change":"2024-01-15T18:12:32.722996+0000","last_active":"2024-01-15T18:12:33.690196+0000","last_peered":"2024-01-15T18:12:33.690196+0000","last_clean":"2024-01-15T18:12:33.690196+0000","last_became_active":"2024-01-15T18:12:32.722736+0000","last_became_peered":"2024-01-15T18:12:32.722736+0000","last_unstale":"2024-01-15T18:12:33.690196+0000","last_undegraded":"2024-01-15T18:12:33.690196+0000","last_fullsized":"2024-01-15T18:12:33.690196+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:09:31.258927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016493+0000","last_change":"2024-01-15T18:12:32.720166+0000","last_active":"2024-01-15T18:12:34.016493+0000","last_peered":"2024-01-15T18:12:34.016493+0000","last_clean":"2024-01-15T18:12:34.016493+0000","last_became_active":"2024-01-15T18:12:32.719812+0000","last_became_peered":"2024-01-15T18:12:32.719812+0000","last_unstale":"2024-01-15T18:12:34.016493+0000","last_undegraded":"2024-01-15T18:12:34.016493+0000","last_fullsized":"2024-01-15T18:12:34.016493+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:48:59.951955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241476+0000","last_change":"2024-01-15T18:12:32.717492+0000","last_active":"2024-01-15T18:12:34.241476+0000","last_peered":"2024-01-15T18:12:34.241476+0000","last_clean":"2024-01-15T18:12:34.241476+0000","last_became_active":"2024-01-15T18:12:32.717224+0000","last_became_peered":"2024-01-15T18:12:32.717224+0000","last_unstale":"2024-01-15T18:12:34.241476+0000","last_undegraded":"2024-01-15T18:12:34.241476+0000","last_fullsized":"2024-01-15T18:12:34.241476+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:06.892505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243002+0000","last_change":"2024-01-15T18:12:32.720577+0000","last_active":"2024-01-15T18:12:34.243002+0000","last_peered":"2024-01-15T18:12:34.243002+0000","last_clean":"2024-01-15T18:12:34.243002+0000","last_became_active":"2024-01-15T18:12:32.720255+0000","last_became_peered":"2024-01-15T18:12:32.720255+0000","last_unstale":"2024-01-15T18:12:34.243002+0000","last_undegraded":"2024-01-15T18:12:34.243002+0000","last_fullsized":"2024-01-15T18:12:34.243002+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:46.700338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242103+0000","last_change":"2024-01-15T18:12:32.717103+0000","last_active":"2024-01-15T18:12:34.242103+0000","last_peered":"2024-01-15T18:12:34.242103+0000","last_clean":"2024-01-15T18:12:34.242103+0000","last_became_active":"2024-01-15T18:12:32.716833+0000","last_became_peered":"2024-01-15T18:12:32.716833+0000","last_unstale":"2024-01-15T18:12:34.242103+0000","last_undegraded":"2024-01-15T18:12:34.242103+0000","last_fullsized":"2024-01-15T18:12:34.242103+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:45.679486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241916+0000","last_change":"2024-01-15T18:12:32.716841+0000","last_active":"2024-01-15T18:12:34.241916+0000","last_peered":"2024-01-15T18:12:34.241916+0000","last_clean":"2024-01-15T18:12:34.241916+0000","last_became_active":"2024-01-15T18:12:32.714917+0000","last_became_peered":"2024-01-15T18:12:32.714917+0000","last_unstale":"2024-01-15T18:12:34.241916+0000","last_undegraded":"2024-01-15T18:12:34.241916+0000","last_fullsized":"2024-01-15T18:12:34.241916+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:06.892505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690307+0000","last_change":"2024-01-15T18:12:32.716307+0000","last_active":"2024-01-15T18:12:33.690307+0000","last_peered":"2024-01-15T18:12:33.690307+0000","last_clean":"2024-01-15T18:12:33.690307+0000","last_became_active":"2024-01-15T18:12:32.715117+0000","last_became_peered":"2024-01-15T18:12:32.715117+0000","last_unstale":"2024-01-15T18:12:33.690307+0000","last_undegraded":"2024-01-15T18:12:33.690307+0000","last_fullsized":"2024-01-15T18:12:33.690307+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:59:58.622884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.699292+0000","last_change":"2024-01-15T18:12:32.724556+0000","last_active":"2024-01-15T18:12:33.699292+0000","last_peered":"2024-01-15T18:12:33.699292+0000","last_clean":"2024-01-15T18:12:33.699292+0000","last_became_active":"2024-01-15T18:12:32.724248+0000","last_became_peered":"2024-01-15T18:12:32.724248+0000","last_unstale":"2024-01-15T18:12:33.699292+0000","last_undegraded":"2024-01-15T18:12:33.699292+0000","last_fullsized":"2024-01-15T18:12:33.699292+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:51:53.729555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243938+0000","last_change":"2024-01-15T18:12:32.715758+0000","last_active":"2024-01-15T18:12:34.243938+0000","last_peered":"2024-01-15T18:12:34.243938+0000","last_clean":"2024-01-15T18:12:34.243938+0000","last_became_active":"2024-01-15T18:12:32.715029+0000","last_became_peered":"2024-01-15T18:12:32.715029+0000","last_unstale":"2024-01-15T18:12:34.243938+0000","last_undegraded":"2024-01-15T18:12:34.243938+0000","last_fullsized":"2024-01-15T18:12:34.243938+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:30.651263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242400+0000","last_change":"2024-01-15T18:12:32.722712+0000","last_active":"2024-01-15T18:12:34.242400+0000","last_peered":"2024-01-15T18:12:34.242400+0000","last_clean":"2024-01-15T18:12:34.242400+0000","last_became_active":"2024-01-15T18:12:32.721915+0000","last_became_peered":"2024-01-15T18:12:32.721915+0000","last_unstale":"2024-01-15T18:12:34.242400+0000","last_undegraded":"2024-01-15T18:12:34.242400+0000","last_fullsized":"2024-01-15T18:12:34.242400+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:16.896538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698549+0000","last_change":"2024-01-15T18:12:32.719251+0000","last_active":"2024-01-15T18:12:33.698549+0000","last_peered":"2024-01-15T18:12:33.698549+0000","last_clean":"2024-01-15T18:12:33.698549+0000","last_became_active":"2024-01-15T18:12:32.718985+0000","last_became_peered":"2024-01-15T18:12:32.718985+0000","last_unstale":"2024-01-15T18:12:33.698549+0000","last_undegraded":"2024-01-15T18:12:33.698549+0000","last_fullsized":"2024-01-15T18:12:33.698549+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:26:29.237185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241086+0000","last_change":"2024-01-15T18:12:32.723963+0000","last_active":"2024-01-15T18:12:34.241086+0000","last_peered":"2024-01-15T18:12:34.241086+0000","last_clean":"2024-01-15T18:12:34.241086+0000","last_became_active":"2024-01-15T18:12:32.722823+0000","last_became_peered":"2024-01-15T18:12:32.722823+0000","last_unstale":"2024-01-15T18:12:34.241086+0000","last_undegraded":"2024-01-15T18:12:34.241086+0000","last_fullsized":"2024-01-15T18:12:34.241086+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:13.244147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242030+0000","last_change":"2024-01-15T18:12:32.713875+0000","last_active":"2024-01-15T18:12:34.242030+0000","last_peered":"2024-01-15T18:12:34.242030+0000","last_clean":"2024-01-15T18:12:34.242030+0000","last_became_active":"2024-01-15T18:12:32.713550+0000","last_became_peered":"2024-01-15T18:12:32.713550+0000","last_unstale":"2024-01-15T18:12:34.242030+0000","last_undegraded":"2024-01-15T18:12:34.242030+0000","last_fullsized":"2024-01-15T18:12:34.242030+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:26:05.201937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240774+0000","last_change":"2024-01-15T18:12:32.710423+0000","last_active":"2024-01-15T18:12:34.240774+0000","last_peered":"2024-01-15T18:12:34.240774+0000","last_clean":"2024-01-15T18:12:34.240774+0000","last_became_active":"2024-01-15T18:12:32.710126+0000","last_became_peered":"2024-01-15T18:12:32.710126+0000","last_unstale":"2024-01-15T18:12:34.240774+0000","last_undegraded":"2024-01-15T18:12:34.240774+0000","last_fullsized":"2024-01-15T18:12:34.240774+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:41.748254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241912+0000","last_change":"2024-01-15T18:12:32.711048+0000","last_active":"2024-01-15T18:12:34.241912+0000","last_peered":"2024-01-15T18:12:34.241912+0000","last_clean":"2024-01-15T18:12:34.241912+0000","last_became_active":"2024-01-15T18:12:32.710511+0000","last_became_peered":"2024-01-15T18:12:32.710511+0000","last_unstale":"2024-01-15T18:12:34.241912+0000","last_undegraded":"2024-01-15T18:12:34.241912+0000","last_fullsized":"2024-01-15T18:12:34.241912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:04.235845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.691001+0000","last_change":"2024-01-15T18:12:32.723850+0000","last_active":"2024-01-15T18:12:33.691001+0000","last_peered":"2024-01-15T18:12:33.691001+0000","last_clean":"2024-01-15T18:12:33.691001+0000","last_became_active":"2024-01-15T18:12:32.723585+0000","last_became_peered":"2024-01-15T18:12:32.723585+0000","last_unstale":"2024-01-15T18:12:33.691001+0000","last_undegraded":"2024-01-15T18:12:33.691001+0000","last_fullsized":"2024-01-15T18:12:33.691001+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:59:25.099068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241466+0000","last_change":"2024-01-15T18:12:32.716397+0000","last_active":"2024-01-15T18:12:34.241466+0000","last_peered":"2024-01-15T18:12:34.241466+0000","last_clean":"2024-01-15T18:12:34.241466+0000","last_became_active":"2024-01-15T18:12:32.715478+0000","last_became_peered":"2024-01-15T18:12:32.715478+0000","last_unstale":"2024-01-15T18:12:34.241466+0000","last_undegraded":"2024-01-15T18:12:34.241466+0000","last_fullsized":"2024-01-15T18:12:34.241466+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:57:44.713007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243054+0000","last_change":"2024-01-15T18:12:32.715851+0000","last_active":"2024-01-15T18:12:34.243054+0000","last_peered":"2024-01-15T18:12:34.243054+0000","last_clean":"2024-01-15T18:12:34.243054+0000","last_became_active":"2024-01-15T18:12:32.715564+0000","last_became_peered":"2024-01-15T18:12:32.715564+0000","last_unstale":"2024-01-15T18:12:34.243054+0000","last_undegraded":"2024-01-15T18:12:34.243054+0000","last_fullsized":"2024-01-15T18:12:34.243054+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:53.944268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698612+0000","last_change":"2024-01-15T18:12:32.724417+0000","last_active":"2024-01-15T18:12:33.698612+0000","last_peered":"2024-01-15T18:12:33.698612+0000","last_clean":"2024-01-15T18:12:33.698612+0000","last_became_active":"2024-01-15T18:12:32.723890+0000","last_became_peered":"2024-01-15T18:12:32.723890+0000","last_unstale":"2024-01-15T18:12:33.698612+0000","last_undegraded":"2024-01-15T18:12:33.698612+0000","last_fullsized":"2024-01-15T18:12:33.698612+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:53:46.400625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242523+0000","last_change":"2024-01-15T18:12:32.719872+0000","last_active":"2024-01-15T18:12:34.242523+0000","last_peered":"2024-01-15T18:12:34.242523+0000","last_clean":"2024-01-15T18:12:34.242523+0000","last_became_active":"2024-01-15T18:12:32.719480+0000","last_became_peered":"2024-01-15T18:12:32.719480+0000","last_unstale":"2024-01-15T18:12:34.242523+0000","last_undegraded":"2024-01-15T18:12:34.242523+0000","last_fullsized":"2024-01-15T18:12:34.242523+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:48.256212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689847+0000","last_change":"2024-01-15T18:12:32.718364+0000","last_active":"2024-01-15T18:12:33.689847+0000","last_peered":"2024-01-15T18:12:33.689847+0000","last_clean":"2024-01-15T18:12:33.689847+0000","last_became_active":"2024-01-15T18:12:32.718033+0000","last_became_peered":"2024-01-15T18:12:32.718033+0000","last_unstale":"2024-01-15T18:12:33.689847+0000","last_undegraded":"2024-01-15T18:12:33.689847+0000","last_fullsized":"2024-01-15T18:12:33.689847+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:30:49.698394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689948+0000","last_change":"2024-01-15T18:12:32.720947+0000","last_active":"2024-01-15T18:12:33.689948+0000","last_peered":"2024-01-15T18:12:33.689948+0000","last_clean":"2024-01-15T18:12:33.689948+0000","last_became_active":"2024-01-15T18:12:32.720493+0000","last_became_peered":"2024-01-15T18:12:32.720493+0000","last_unstale":"2024-01-15T18:12:33.689948+0000","last_undegraded":"2024-01-15T18:12:33.689948+0000","last_fullsized":"2024-01-15T18:12:33.689948+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:30.651263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243319+0000","last_change":"2024-01-15T18:12:32.710019+0000","last_active":"2024-01-15T18:12:34.243319+0000","last_peered":"2024-01-15T18:12:34.243319+0000","last_clean":"2024-01-15T18:12:34.243319+0000","last_became_active":"2024-01-15T18:12:32.708465+0000","last_became_peered":"2024-01-15T18:12:32.708465+0000","last_unstale":"2024-01-15T18:12:34.243319+0000","last_undegraded":"2024-01-15T18:12:34.243319+0000","last_fullsized":"2024-01-15T18:12:34.243319+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:05:30.651263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.244356+0000","last_change":"2024-01-15T18:12:32.719091+0000","last_active":"2024-01-15T18:12:34.244356+0000","last_peered":"2024-01-15T18:12:34.244356+0000","last_clean":"2024-01-15T18:12:34.244356+0000","last_became_active":"2024-01-15T18:12:32.718798+0000","last_became_peered":"2024-01-15T18:12:32.718798+0000","last_unstale":"2024-01-15T18:12:34.244356+0000","last_undegraded":"2024-01-15T18:12:34.244356+0000","last_fullsized":"2024-01-15T18:12:34.244356+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:49:16.486465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690406+0000","last_change":"2024-01-15T18:12:32.721976+0000","last_active":"2024-01-15T18:12:33.690406+0000","last_peered":"2024-01-15T18:12:33.690406+0000","last_clean":"2024-01-15T18:12:33.690406+0000","last_became_active":"2024-01-15T18:12:32.721098+0000","last_became_peered":"2024-01-15T18:12:32.721098+0000","last_unstale":"2024-01-15T18:12:33.690406+0000","last_undegraded":"2024-01-15T18:12:33.690406+0000","last_fullsized":"2024-01-15T18:12:33.690406+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:47:10.680287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242763+0000","last_change":"2024-01-15T18:12:32.715205+0000","last_active":"2024-01-15T18:12:34.242763+0000","last_peered":"2024-01-15T18:12:34.242763+0000","last_clean":"2024-01-15T18:12:34.242763+0000","last_became_active":"2024-01-15T18:12:32.714830+0000","last_became_peered":"2024-01-15T18:12:32.714830+0000","last_unstale":"2024-01-15T18:12:34.242763+0000","last_undegraded":"2024-01-15T18:12:34.242763+0000","last_fullsized":"2024-01-15T18:12:34.242763+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:46.700338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243669+0000","last_change":"2024-01-15T18:12:32.719418+0000","last_active":"2024-01-15T18:12:34.243669+0000","last_peered":"2024-01-15T18:12:34.243669+0000","last_clean":"2024-01-15T18:12:34.243669+0000","last_became_active":"2024-01-15T18:12:32.719097+0000","last_became_peered":"2024-01-15T18:12:32.719097+0000","last_unstale":"2024-01-15T18:12:34.243669+0000","last_undegraded":"2024-01-15T18:12:34.243669+0000","last_fullsized":"2024-01-15T18:12:34.243669+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:33.019506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690284+0000","last_change":"2024-01-15T18:12:32.721827+0000","last_active":"2024-01-15T18:12:33.690284+0000","last_peered":"2024-01-15T18:12:33.690284+0000","last_clean":"2024-01-15T18:12:33.690284+0000","last_became_active":"2024-01-15T18:12:32.721517+0000","last_became_peered":"2024-01-15T18:12:32.721517+0000","last_unstale":"2024-01-15T18:12:33.690284+0000","last_undegraded":"2024-01-15T18:12:33.690284+0000","last_fullsized":"2024-01-15T18:12:33.690284+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:49:16.486465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242001+0000","last_change":"2024-01-15T18:12:32.716496+0000","last_active":"2024-01-15T18:12:34.242001+0000","last_peered":"2024-01-15T18:12:34.242001+0000","last_clean":"2024-01-15T18:12:34.242001+0000","last_became_active":"2024-01-15T18:12:32.716233+0000","last_became_peered":"2024-01-15T18:12:32.716233+0000","last_unstale":"2024-01-15T18:12:34.242001+0000","last_undegraded":"2024-01-15T18:12:34.242001+0000","last_fullsized":"2024-01-15T18:12:34.242001+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:35:52.894173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240705+0000","last_change":"2024-01-15T18:12:32.712505+0000","last_active":"2024-01-15T18:12:34.240705+0000","last_peered":"2024-01-15T18:12:34.240705+0000","last_clean":"2024-01-15T18:12:34.240705+0000","last_became_active":"2024-01-15T18:12:32.712167+0000","last_became_peered":"2024-01-15T18:12:32.712167+0000","last_unstale":"2024-01-15T18:12:34.240705+0000","last_undegraded":"2024-01-15T18:12:34.240705+0000","last_fullsized":"2024-01-15T18:12:34.240705+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:38.795536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015766+0000","last_change":"2024-01-15T18:12:32.709726+0000","last_active":"2024-01-15T18:12:34.015766+0000","last_peered":"2024-01-15T18:12:34.015766+0000","last_clean":"2024-01-15T18:12:34.015766+0000","last_became_active":"2024-01-15T18:12:32.709448+0000","last_became_peered":"2024-01-15T18:12:32.709448+0000","last_unstale":"2024-01-15T18:12:34.015766+0000","last_undegraded":"2024-01-15T18:12:34.015766+0000","last_fullsized":"2024-01-15T18:12:34.015766+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:44:24.987516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243490+0000","last_change":"2024-01-15T18:12:32.714781+0000","last_active":"2024-01-15T18:12:34.243490+0000","last_peered":"2024-01-15T18:12:34.243490+0000","last_clean":"2024-01-15T18:12:34.243490+0000","last_became_active":"2024-01-15T18:12:32.714499+0000","last_became_peered":"2024-01-15T18:12:32.714499+0000","last_unstale":"2024-01-15T18:12:34.243490+0000","last_undegraded":"2024-01-15T18:12:34.243490+0000","last_fullsized":"2024-01-15T18:12:34.243490+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:01:30.418661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"102'4","reported_seq":16,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690985+0000","last_change":"2024-01-15T18:12:32.174025+0000","last_active":"2024-01-15T18:12:33.690985+0000","last_peered":"2024-01-15T18:12:33.690985+0000","last_clean":"2024-01-15T18:12:33.690985+0000","last_became_active":"2024-01-15T18:12:32.172427+0000","last_became_peered":"2024-01-15T18:12:32.172427+0000","last_unstale":"2024-01-15T18:12:33.690985+0000","last_undegraded":"2024-01-15T18:12:33.690985+0000","last_fullsized":"2024-01-15T18:12:33.690985+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:32.003405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242172+0000","last_change":"2024-01-15T18:12:32.711420+0000","last_active":"2024-01-15T18:12:34.242172+0000","last_peered":"2024-01-15T18:12:34.242172+0000","last_clean":"2024-01-15T18:12:34.242172+0000","last_became_active":"2024-01-15T18:12:32.711010+0000","last_became_peered":"2024-01-15T18:12:32.711010+0000","last_unstale":"2024-01-15T18:12:34.242172+0000","last_undegraded":"2024-01-15T18:12:34.242172+0000","last_fullsized":"2024-01-15T18:12:34.242172+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:04.235845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242660+0000","last_change":"2024-01-15T18:12:32.175254+0000","last_active":"2024-01-15T18:12:34.242660+0000","last_peered":"2024-01-15T18:12:34.242660+0000","last_clean":"2024-01-15T18:12:34.242660+0000","last_became_active":"2024-01-15T18:12:32.174944+0000","last_became_peered":"2024-01-15T18:12:32.174944+0000","last_unstale":"2024-01-15T18:12:34.242660+0000","last_undegraded":"2024-01-15T18:12:34.242660+0000","last_fullsized":"2024-01-15T18:12:34.242660+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:39:37.779435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016329+0000","last_change":"2024-01-15T18:12:32.722958+0000","last_active":"2024-01-15T18:12:34.016329+0000","last_peered":"2024-01-15T18:12:34.016329+0000","last_clean":"2024-01-15T18:12:34.016329+0000","last_became_active":"2024-01-15T18:12:32.721379+0000","last_became_peered":"2024-01-15T18:12:32.721379+0000","last_unstale":"2024-01-15T18:12:34.016329+0000","last_undegraded":"2024-01-15T18:12:34.016329+0000","last_fullsized":"2024-01-15T18:12:34.016329+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:40:32.443345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690213+0000","last_change":"2024-01-15T18:12:32.173987+0000","last_active":"2024-01-15T18:12:33.690213+0000","last_peered":"2024-01-15T18:12:33.690213+0000","last_clean":"2024-01-15T18:12:33.690213+0000","last_became_active":"2024-01-15T18:12:32.173678+0000","last_became_peered":"2024-01-15T18:12:32.173678+0000","last_unstale":"2024-01-15T18:12:33.690213+0000","last_undegraded":"2024-01-15T18:12:33.690213+0000","last_fullsized":"2024-01-15T18:12:33.690213+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:44.663385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698591+0000","last_change":"2024-01-15T18:12:32.717351+0000","last_active":"2024-01-15T18:12:33.698591+0000","last_peered":"2024-01-15T18:12:33.698591+0000","last_clean":"2024-01-15T18:12:33.698591+0000","last_became_active":"2024-01-15T18:12:32.717050+0000","last_became_peered":"2024-01-15T18:12:32.717050+0000","last_unstale":"2024-01-15T18:12:33.698591+0000","last_undegraded":"2024-01-15T18:12:33.698591+0000","last_fullsized":"2024-01-15T18:12:33.698591+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:45.360374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690165+0000","last_change":"2024-01-15T18:12:32.169329+0000","last_active":"2024-01-15T18:12:33.690165+0000","last_peered":"2024-01-15T18:12:33.690165+0000","last_clean":"2024-01-15T18:12:33.690165+0000","last_became_active":"2024-01-15T18:12:32.168881+0000","last_became_peered":"2024-01-15T18:12:32.168881+0000","last_unstale":"2024-01-15T18:12:33.690165+0000","last_undegraded":"2024-01-15T18:12:33.690165+0000","last_fullsized":"2024-01-15T18:12:33.690165+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:50:54.945510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.696308+0000","last_change":"2024-01-15T18:12:32.717729+0000","last_active":"2024-01-15T18:12:33.696308+0000","last_peered":"2024-01-15T18:12:33.696308+0000","last_clean":"2024-01-15T18:12:33.696308+0000","last_became_active":"2024-01-15T18:12:32.717377+0000","last_became_peered":"2024-01-15T18:12:32.717377+0000","last_unstale":"2024-01-15T18:12:33.696308+0000","last_undegraded":"2024-01-15T18:12:33.696308+0000","last_fullsized":"2024-01-15T18:12:33.696308+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:27:51.651136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016181+0000","last_change":"2024-01-15T18:12:32.171924+0000","last_active":"2024-01-15T18:12:34.016181+0000","last_peered":"2024-01-15T18:12:34.016181+0000","last_clean":"2024-01-15T18:12:34.016181+0000","last_became_active":"2024-01-15T18:12:32.170824+0000","last_became_peered":"2024-01-15T18:12:32.170824+0000","last_unstale":"2024-01-15T18:12:34.016181+0000","last_undegraded":"2024-01-15T18:12:34.016181+0000","last_fullsized":"2024-01-15T18:12:34.016181+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:58:47.416771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242875+0000","last_change":"2024-01-15T18:12:32.712405+0000","last_active":"2024-01-15T18:12:34.242875+0000","last_peered":"2024-01-15T18:12:34.242875+0000","last_clean":"2024-01-15T18:12:34.242875+0000","last_became_active":"2024-01-15T18:12:32.711993+0000","last_became_peered":"2024-01-15T18:12:32.711993+0000","last_unstale":"2024-01-15T18:12:34.242875+0000","last_undegraded":"2024-01-15T18:12:34.242875+0000","last_fullsized":"2024-01-15T18:12:34.242875+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:21:33.019506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689923+0000","last_change":"2024-01-15T18:12:32.169450+0000","last_active":"2024-01-15T18:12:33.689923+0000","last_peered":"2024-01-15T18:12:33.689923+0000","last_clean":"2024-01-15T18:12:33.689923+0000","last_became_active":"2024-01-15T18:12:32.169195+0000","last_became_peered":"2024-01-15T18:12:32.169195+0000","last_unstale":"2024-01-15T18:12:33.689923+0000","last_undegraded":"2024-01-15T18:12:33.689923+0000","last_fullsized":"2024-01-15T18:12:33.689923+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:12.228046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240671+0000","last_change":"2024-01-15T18:12:32.715702+0000","last_active":"2024-01-15T18:12:34.240671+0000","last_peered":"2024-01-15T18:12:34.240671+0000","last_clean":"2024-01-15T18:12:34.240671+0000","last_became_active":"2024-01-15T18:12:32.711262+0000","last_became_peered":"2024-01-15T18:12:32.711262+0000","last_unstale":"2024-01-15T18:12:34.240671+0000","last_undegraded":"2024-01-15T18:12:34.240671+0000","last_fullsized":"2024-01-15T18:12:34.240671+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:55.634124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243119+0000","last_change":"2024-01-15T18:12:32.177242+0000","last_active":"2024-01-15T18:12:34.243119+0000","last_peered":"2024-01-15T18:12:34.243119+0000","last_clean":"2024-01-15T18:12:34.243119+0000","last_became_active":"2024-01-15T18:12:32.176980+0000","last_became_peered":"2024-01-15T18:12:32.176980+0000","last_unstale":"2024-01-15T18:12:34.243119+0000","last_undegraded":"2024-01-15T18:12:34.243119+0000","last_fullsized":"2024-01-15T18:12:34.243119+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:40:58.665583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.697912+0000","last_change":"2024-01-15T18:12:32.719081+0000","last_active":"2024-01-15T18:12:33.697912+0000","last_peered":"2024-01-15T18:12:33.697912+0000","last_clean":"2024-01-15T18:12:33.697912+0000","last_became_active":"2024-01-15T18:12:32.718763+0000","last_became_peered":"2024-01-15T18:12:32.718763+0000","last_unstale":"2024-01-15T18:12:33.697912+0000","last_undegraded":"2024-01-15T18:12:33.697912+0000","last_fullsized":"2024-01-15T18:12:33.697912+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:45:04.246875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015826+0000","last_change":"2024-01-15T18:12:32.175060+0000","last_active":"2024-01-15T18:12:34.015826+0000","last_peered":"2024-01-15T18:12:34.015826+0000","last_clean":"2024-01-15T18:12:34.015826+0000","last_became_active":"2024-01-15T18:12:32.174748+0000","last_became_peered":"2024-01-15T18:12:32.174748+0000","last_unstale":"2024-01-15T18:12:34.015826+0000","last_undegraded":"2024-01-15T18:12:34.015826+0000","last_fullsized":"2024-01-15T18:12:34.015826+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:47:06.107711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690626+0000","last_change":"2024-01-15T18:12:32.726315+0000","last_active":"2024-01-15T18:12:33.690626+0000","last_peered":"2024-01-15T18:12:33.690626+0000","last_clean":"2024-01-15T18:12:33.690626+0000","last_became_active":"2024-01-15T18:12:32.725362+0000","last_became_peered":"2024-01-15T18:12:32.725362+0000","last_unstale":"2024-01-15T18:12:33.690626+0000","last_undegraded":"2024-01-15T18:12:33.690626+0000","last_fullsized":"2024-01-15T18:12:33.690626+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:04:36.011229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016746+0000","last_change":"2024-01-15T18:12:32.167152+0000","last_active":"2024-01-15T18:12:34.016746+0000","last_peered":"2024-01-15T18:12:34.016746+0000","last_clean":"2024-01-15T18:12:34.016746+0000","last_became_active":"2024-01-15T18:12:32.166871+0000","last_became_peered":"2024-01-15T18:12:32.166871+0000","last_unstale":"2024-01-15T18:12:34.016746+0000","last_undegraded":"2024-01-15T18:12:34.016746+0000","last_fullsized":"2024-01-15T18:12:34.016746+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:45:17.514303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.699299+0000","last_change":"2024-01-15T18:12:32.721080+0000","last_active":"2024-01-15T18:12:33.699299+0000","last_peered":"2024-01-15T18:12:33.699299+0000","last_clean":"2024-01-15T18:12:33.699299+0000","last_became_active":"2024-01-15T18:12:32.720789+0000","last_became_peered":"2024-01-15T18:12:32.720789+0000","last_unstale":"2024-01-15T18:12:33.699299+0000","last_undegraded":"2024-01-15T18:12:33.699299+0000","last_fullsized":"2024-01-15T18:12:33.699299+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:36:56.725359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241498+0000","last_change":"2024-01-15T18:12:32.172750+0000","last_active":"2024-01-15T18:12:34.241498+0000","last_peered":"2024-01-15T18:12:34.241498+0000","last_clean":"2024-01-15T18:12:34.241498+0000","last_became_active":"2024-01-15T18:12:32.172475+0000","last_became_peered":"2024-01-15T18:12:32.172475+0000","last_unstale":"2024-01-15T18:12:34.241498+0000","last_undegraded":"2024-01-15T18:12:34.241498+0000","last_fullsized":"2024-01-15T18:12:34.241498+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:03:09.953872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243887+0000","last_change":"2024-01-15T18:12:32.726087+0000","last_active":"2024-01-15T18:12:34.243887+0000","last_peered":"2024-01-15T18:12:34.243887+0000","last_clean":"2024-01-15T18:12:34.243887+0000","last_became_active":"2024-01-15T18:12:32.725614+0000","last_became_peered":"2024-01-15T18:12:32.725614+0000","last_unstale":"2024-01-15T18:12:34.243887+0000","last_undegraded":"2024-01-15T18:12:34.243887+0000","last_fullsized":"2024-01-15T18:12:34.243887+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:30:49.698394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"102'3","reported_seq":15,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243302+0000","last_change":"2024-01-15T18:12:32.171231+0000","last_active":"2024-01-15T18:12:34.243302+0000","last_peered":"2024-01-15T18:12:34.243302+0000","last_clean":"2024-01-15T18:12:34.243302+0000","last_became_active":"2024-01-15T18:12:32.170238+0000","last_became_peered":"2024-01-15T18:12:32.170238+0000","last_unstale":"2024-01-15T18:12:34.243302+0000","last_undegraded":"2024-01-15T18:12:34.243302+0000","last_fullsized":"2024-01-15T18:12:34.243302+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:57:43.696906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243081+0000","last_change":"2024-01-15T18:12:32.711064+0000","last_active":"2024-01-15T18:12:34.243081+0000","last_peered":"2024-01-15T18:12:34.243081+0000","last_clean":"2024-01-15T18:12:34.243081+0000","last_became_active":"2024-01-15T18:12:32.710623+0000","last_became_peered":"2024-01-15T18:12:32.710623+0000","last_unstale":"2024-01-15T18:12:34.243081+0000","last_undegraded":"2024-01-15T18:12:34.243081+0000","last_fullsized":"2024-01-15T18:12:34.243081+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:30:49.698394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"102'6","reported_seq":18,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698821+0000","last_change":"2024-01-15T18:12:32.169172+0000","last_active":"2024-01-15T18:12:33.698821+0000","last_peered":"2024-01-15T18:12:33.698821+0000","last_clean":"2024-01-15T18:12:33.698821+0000","last_became_active":"2024-01-15T18:12:32.168902+0000","last_became_peered":"2024-01-15T18:12:32.168902+0000","last_unstale":"2024-01-15T18:12:33.698821+0000","last_undegraded":"2024-01-15T18:12:33.698821+0000","last_fullsized":"2024-01-15T18:12:33.698821+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:46:24.548644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241082+0000","last_change":"2024-01-15T18:12:32.702338+0000","last_active":"2024-01-15T18:12:34.241082+0000","last_peered":"2024-01-15T18:12:34.241082+0000","last_clean":"2024-01-15T18:12:34.241082+0000","last_became_active":"2024-01-15T18:12:32.700501+0000","last_became_peered":"2024-01-15T18:12:32.700501+0000","last_unstale":"2024-01-15T18:12:34.241082+0000","last_undegraded":"2024-01-15T18:12:34.241082+0000","last_fullsized":"2024-01-15T18:12:34.241082+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:34:17.158053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242794+0000","last_change":"2024-01-15T18:12:32.155905+0000","last_active":"2024-01-15T18:12:34.242794+0000","last_peered":"2024-01-15T18:12:34.242794+0000","last_clean":"2024-01-15T18:12:34.242794+0000","last_became_active":"2024-01-15T18:12:32.155514+0000","last_became_peered":"2024-01-15T18:12:32.155514+0000","last_unstale":"2024-01-15T18:12:34.242794+0000","last_undegraded":"2024-01-15T18:12:34.242794+0000","last_fullsized":"2024-01-15T18:12:34.242794+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:13:08.319290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015895+0000","last_change":"2024-01-15T18:12:32.718072+0000","last_active":"2024-01-15T18:12:34.015895+0000","last_peered":"2024-01-15T18:12:34.015895+0000","last_clean":"2024-01-15T18:12:34.015895+0000","last_became_active":"2024-01-15T18:12:32.717809+0000","last_became_peered":"2024-01-15T18:12:32.717809+0000","last_unstale":"2024-01-15T18:12:34.015895+0000","last_undegraded":"2024-01-15T18:12:34.015895+0000","last_fullsized":"2024-01-15T18:12:34.015895+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:47:08.699545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242476+0000","last_change":"2024-01-15T18:12:32.167849+0000","last_active":"2024-01-15T18:12:34.242476+0000","last_peered":"2024-01-15T18:12:34.242476+0000","last_clean":"2024-01-15T18:12:34.242476+0000","last_became_active":"2024-01-15T18:12:32.167507+0000","last_became_peered":"2024-01-15T18:12:32.167507+0000","last_unstale":"2024-01-15T18:12:34.242476+0000","last_undegraded":"2024-01-15T18:12:34.242476+0000","last_fullsized":"2024-01-15T18:12:34.242476+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:26:44.737791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.697675+0000","last_change":"2024-01-15T18:12:32.720100+0000","last_active":"2024-01-15T18:12:33.697675+0000","last_peered":"2024-01-15T18:12:33.697675+0000","last_clean":"2024-01-15T18:12:33.697675+0000","last_became_active":"2024-01-15T18:12:32.719713+0000","last_became_peered":"2024-01-15T18:12:32.719713+0000","last_unstale":"2024-01-15T18:12:33.697675+0000","last_undegraded":"2024-01-15T18:12:33.697675+0000","last_fullsized":"2024-01-15T18:12:33.697675+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:01:14.698019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698130+0000","last_change":"2024-01-15T18:12:32.166734+0000","last_active":"2024-01-15T18:12:33.698130+0000","last_peered":"2024-01-15T18:12:33.698130+0000","last_clean":"2024-01-15T18:12:33.698130+0000","last_became_active":"2024-01-15T18:12:32.165620+0000","last_became_peered":"2024-01-15T18:12:32.165620+0000","last_unstale":"2024-01-15T18:12:33.698130+0000","last_undegraded":"2024-01-15T18:12:33.698130+0000","last_fullsized":"2024-01-15T18:12:33.698130+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:23:45.537131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241696+0000","last_change":"2024-01-15T18:12:32.723410+0000","last_active":"2024-01-15T18:12:34.241696+0000","last_peered":"2024-01-15T18:12:34.241696+0000","last_clean":"2024-01-15T18:12:34.241696+0000","last_became_active":"2024-01-15T18:12:32.723110+0000","last_became_peered":"2024-01-15T18:12:32.723110+0000","last_unstale":"2024-01-15T18:12:34.241696+0000","last_undegraded":"2024-01-15T18:12:34.241696+0000","last_fullsized":"2024-01-15T18:12:34.241696+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:30:28.945346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"102'8","reported_seq":20,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.015921+0000","last_change":"2024-01-15T18:12:32.169474+0000","last_active":"2024-01-15T18:12:34.015921+0000","last_peered":"2024-01-15T18:12:34.015921+0000","last_clean":"2024-01-15T18:12:34.015921+0000","last_became_active":"2024-01-15T18:12:32.169195+0000","last_became_peered":"2024-01-15T18:12:32.169195+0000","last_unstale":"2024-01-15T18:12:34.015921+0000","last_undegraded":"2024-01-15T18:12:34.015921+0000","last_fullsized":"2024-01-15T18:12:34.015921+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:22:19.616756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.244288+0000","last_change":"2024-01-15T18:12:32.722879+0000","last_active":"2024-01-15T18:12:34.244288+0000","last_peered":"2024-01-15T18:12:34.244288+0000","last_clean":"2024-01-15T18:12:34.244288+0000","last_became_active":"2024-01-15T18:12:32.722355+0000","last_became_peered":"2024-01-15T18:12:32.722355+0000","last_unstale":"2024-01-15T18:12:34.244288+0000","last_undegraded":"2024-01-15T18:12:34.244288+0000","last_fullsized":"2024-01-15T18:12:34.244288+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:09:31.258927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698709+0000","last_change":"2024-01-15T18:12:32.168558+0000","last_active":"2024-01-15T18:12:33.698709+0000","last_peered":"2024-01-15T18:12:33.698709+0000","last_clean":"2024-01-15T18:12:33.698709+0000","last_became_active":"2024-01-15T18:12:32.168212+0000","last_became_peered":"2024-01-15T18:12:32.168212+0000","last_unstale":"2024-01-15T18:12:33.698709+0000","last_undegraded":"2024-01-15T18:12:33.698709+0000","last_fullsized":"2024-01-15T18:12:33.698709+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:16.367691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.696327+0000","last_change":"2024-01-15T18:12:32.712788+0000","last_active":"2024-01-15T18:12:33.696327+0000","last_peered":"2024-01-15T18:12:33.696327+0000","last_clean":"2024-01-15T18:12:33.696327+0000","last_became_active":"2024-01-15T18:12:32.712490+0000","last_became_peered":"2024-01-15T18:12:32.712490+0000","last_unstale":"2024-01-15T18:12:33.696327+0000","last_undegraded":"2024-01-15T18:12:33.696327+0000","last_fullsized":"2024-01-15T18:12:33.696327+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:59.486284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243703+0000","last_change":"2024-01-15T18:12:32.169139+0000","last_active":"2024-01-15T18:12:34.243703+0000","last_peered":"2024-01-15T18:12:34.243703+0000","last_clean":"2024-01-15T18:12:34.243703+0000","last_became_active":"2024-01-15T18:12:32.168723+0000","last_became_peered":"2024-01-15T18:12:32.168723+0000","last_unstale":"2024-01-15T18:12:34.243703+0000","last_undegraded":"2024-01-15T18:12:34.243703+0000","last_fullsized":"2024-01-15T18:12:34.243703+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:39:12.228046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243075+0000","last_change":"2024-01-15T18:12:32.722973+0000","last_active":"2024-01-15T18:12:34.243075+0000","last_peered":"2024-01-15T18:12:34.243075+0000","last_clean":"2024-01-15T18:12:34.243075+0000","last_became_active":"2024-01-15T18:12:32.721390+0000","last_became_peered":"2024-01-15T18:12:32.721390+0000","last_unstale":"2024-01-15T18:12:34.243075+0000","last_undegraded":"2024-01-15T18:12:34.243075+0000","last_fullsized":"2024-01-15T18:12:34.243075+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:32.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690305+0000","last_change":"2024-01-15T18:12:32.169155+0000","last_active":"2024-01-15T18:12:33.690305+0000","last_peered":"2024-01-15T18:12:33.690305+0000","last_clean":"2024-01-15T18:12:33.690305+0000","last_became_active":"2024-01-15T18:12:32.168513+0000","last_became_peered":"2024-01-15T18:12:32.168513+0000","last_unstale":"2024-01-15T18:12:33.690305+0000","last_undegraded":"2024-01-15T18:12:33.690305+0000","last_fullsized":"2024-01-15T18:12:33.690305+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:33:03.219744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242261+0000","last_change":"2024-01-15T18:12:32.718945+0000","last_active":"2024-01-15T18:12:34.242261+0000","last_peered":"2024-01-15T18:12:34.242261+0000","last_clean":"2024-01-15T18:12:34.242261+0000","last_became_active":"2024-01-15T18:12:32.718082+0000","last_became_peered":"2024-01-15T18:12:32.718082+0000","last_unstale":"2024-01-15T18:12:34.242261+0000","last_undegraded":"2024-01-15T18:12:34.242261+0000","last_fullsized":"2024-01-15T18:12:34.242261+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:08.935819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690420+0000","last_change":"2024-01-15T18:12:32.176147+0000","last_active":"2024-01-15T18:12:33.690420+0000","last_peered":"2024-01-15T18:12:33.690420+0000","last_clean":"2024-01-15T18:12:33.690420+0000","last_became_active":"2024-01-15T18:12:32.175874+0000","last_became_peered":"2024-01-15T18:12:32.175874+0000","last_unstale":"2024-01-15T18:12:33.690420+0000","last_undegraded":"2024-01-15T18:12:33.690420+0000","last_fullsized":"2024-01-15T18:12:33.690420+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:57:43.696906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242653+0000","last_change":"2024-01-15T18:12:32.717201+0000","last_active":"2024-01-15T18:12:34.242653+0000","last_peered":"2024-01-15T18:12:34.242653+0000","last_clean":"2024-01-15T18:12:34.242653+0000","last_became_active":"2024-01-15T18:12:32.716936+0000","last_became_peered":"2024-01-15T18:12:32.716936+0000","last_unstale":"2024-01-15T18:12:34.242653+0000","last_undegraded":"2024-01-15T18:12:34.242653+0000","last_fullsized":"2024-01-15T18:12:34.242653+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:36:09.350690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.696364+0000","last_change":"2024-01-15T18:12:32.167360+0000","last_active":"2024-01-15T18:12:33.696364+0000","last_peered":"2024-01-15T18:12:33.696364+0000","last_clean":"2024-01-15T18:12:33.696364+0000","last_became_active":"2024-01-15T18:12:32.166211+0000","last_became_peered":"2024-01-15T18:12:32.166211+0000","last_unstale":"2024-01-15T18:12:33.696364+0000","last_undegraded":"2024-01-15T18:12:33.696364+0000","last_fullsized":"2024-01-15T18:12:33.696364+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:50:10.531632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241750+0000","last_change":"2024-01-15T18:12:32.724280+0000","last_active":"2024-01-15T18:12:34.241750+0000","last_peered":"2024-01-15T18:12:34.241750+0000","last_clean":"2024-01-15T18:12:34.241750+0000","last_became_active":"2024-01-15T18:12:32.721970+0000","last_became_peered":"2024-01-15T18:12:32.721970+0000","last_unstale":"2024-01-15T18:12:34.241750+0000","last_undegraded":"2024-01-15T18:12:34.241750+0000","last_fullsized":"2024-01-15T18:12:34.241750+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:47:04.065239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.695973+0000","last_change":"2024-01-15T18:12:32.168557+0000","last_active":"2024-01-15T18:12:33.695973+0000","last_peered":"2024-01-15T18:12:33.695973+0000","last_clean":"2024-01-15T18:12:33.695973+0000","last_became_active":"2024-01-15T18:12:32.167826+0000","last_became_peered":"2024-01-15T18:12:32.167826+0000","last_unstale":"2024-01-15T18:12:33.695973+0000","last_undegraded":"2024-01-15T18:12:33.695973+0000","last_fullsized":"2024-01-15T18:12:33.695973+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:26:30.036119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242301+0000","last_change":"2024-01-15T18:12:32.711184+0000","last_active":"2024-01-15T18:12:34.242301+0000","last_peered":"2024-01-15T18:12:34.242301+0000","last_clean":"2024-01-15T18:12:34.242301+0000","last_became_active":"2024-01-15T18:12:32.709348+0000","last_became_peered":"2024-01-15T18:12:32.709348+0000","last_unstale":"2024-01-15T18:12:34.242301+0000","last_undegraded":"2024-01-15T18:12:34.242301+0000","last_fullsized":"2024-01-15T18:12:34.242301+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:45.679486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.689918+0000","last_change":"2024-01-15T18:12:32.170820+0000","last_active":"2024-01-15T18:12:33.689918+0000","last_peered":"2024-01-15T18:12:33.689918+0000","last_clean":"2024-01-15T18:12:33.689918+0000","last_became_active":"2024-01-15T18:12:32.170532+0000","last_became_peered":"2024-01-15T18:12:32.170532+0000","last_unstale":"2024-01-15T18:12:33.689918+0000","last_undegraded":"2024-01-15T18:12:33.689918+0000","last_fullsized":"2024-01-15T18:12:33.689918+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:40:58.665583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690635+0000","last_change":"2024-01-15T18:12:32.722139+0000","last_active":"2024-01-15T18:12:33.690635+0000","last_peered":"2024-01-15T18:12:33.690635+0000","last_clean":"2024-01-15T18:12:33.690635+0000","last_became_active":"2024-01-15T18:12:32.721670+0000","last_became_peered":"2024-01-15T18:12:32.721670+0000","last_unstale":"2024-01-15T18:12:33.690635+0000","last_undegraded":"2024-01-15T18:12:33.690635+0000","last_fullsized":"2024-01-15T18:12:33.690635+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:25.982795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"102'2","reported_seq":14,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.697354+0000","last_change":"2024-01-15T18:12:32.159866+0000","last_active":"2024-01-15T18:12:33.697354+0000","last_peered":"2024-01-15T18:12:33.697354+0000","last_clean":"2024-01-15T18:12:33.697354+0000","last_became_active":"2024-01-15T18:12:32.159482+0000","last_became_peered":"2024-01-15T18:12:32.159482+0000","last_unstale":"2024-01-15T18:12:33.697354+0000","last_undegraded":"2024-01-15T18:12:33.697354+0000","last_fullsized":"2024-01-15T18:12:33.697354+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:06:59.081912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016672+0000","last_change":"2024-01-15T18:12:32.722572+0000","last_active":"2024-01-15T18:12:34.016672+0000","last_peered":"2024-01-15T18:12:34.016672+0000","last_clean":"2024-01-15T18:12:34.016672+0000","last_became_active":"2024-01-15T18:12:32.714975+0000","last_became_peered":"2024-01-15T18:12:32.714975+0000","last_unstale":"2024-01-15T18:12:34.016672+0000","last_undegraded":"2024-01-15T18:12:34.016672+0000","last_fullsized":"2024-01-15T18:12:34.016672+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:16:53.650301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690770+0000","last_change":"2024-01-15T18:12:32.175271+0000","last_active":"2024-01-15T18:12:33.690770+0000","last_peered":"2024-01-15T18:12:33.690770+0000","last_clean":"2024-01-15T18:12:33.690770+0000","last_became_active":"2024-01-15T18:12:32.174807+0000","last_became_peered":"2024-01-15T18:12:32.174807+0000","last_unstale":"2024-01-15T18:12:33.690770+0000","last_undegraded":"2024-01-15T18:12:33.690770+0000","last_fullsized":"2024-01-15T18:12:33.690770+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:39:45.684237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.016615+0000","last_change":"2024-01-15T18:12:32.717689+0000","last_active":"2024-01-15T18:12:34.016615+0000","last_peered":"2024-01-15T18:12:34.016615+0000","last_clean":"2024-01-15T18:12:34.016615+0000","last_became_active":"2024-01-15T18:12:32.714629+0000","last_became_peered":"2024-01-15T18:12:32.714629+0000","last_unstale":"2024-01-15T18:12:34.016615+0000","last_undegraded":"2024-01-15T18:12:34.016615+0000","last_fullsized":"2024-01-15T18:12:34.016615+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:12:14.192962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690349+0000","last_change":"2024-01-15T18:12:32.171453+0000","last_active":"2024-01-15T18:12:33.690349+0000","last_peered":"2024-01-15T18:12:33.690349+0000","last_clean":"2024-01-15T18:12:33.690349+0000","last_became_active":"2024-01-15T18:12:32.171105+0000","last_became_peered":"2024-01-15T18:12:32.171105+0000","last_unstale":"2024-01-15T18:12:33.690349+0000","last_undegraded":"2024-01-15T18:12:33.690349+0000","last_fullsized":"2024-01-15T18:12:33.690349+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:15.880437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243754+0000","last_change":"2024-01-15T18:12:32.720622+0000","last_active":"2024-01-15T18:12:34.243754+0000","last_peered":"2024-01-15T18:12:34.243754+0000","last_clean":"2024-01-15T18:12:34.243754+0000","last_became_active":"2024-01-15T18:12:32.720292+0000","last_became_peered":"2024-01-15T18:12:32.720292+0000","last_unstale":"2024-01-15T18:12:34.243754+0000","last_undegraded":"2024-01-15T18:12:34.243754+0000","last_fullsized":"2024-01-15T18:12:34.243754+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:32.333079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243901+0000","last_change":"2024-01-15T18:12:32.170644+0000","last_active":"2024-01-15T18:12:34.243901+0000","last_peered":"2024-01-15T18:12:34.243901+0000","last_clean":"2024-01-15T18:12:34.243901+0000","last_became_active":"2024-01-15T18:12:32.170352+0000","last_became_peered":"2024-01-15T18:12:32.170352+0000","last_unstale":"2024-01-15T18:12:34.243901+0000","last_undegraded":"2024-01-15T18:12:34.243901+0000","last_fullsized":"2024-01-15T18:12:34.243901+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:34:05.876403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242267+0000","last_change":"2024-01-15T18:12:32.724247+0000","last_active":"2024-01-15T18:12:34.242267+0000","last_peered":"2024-01-15T18:12:34.242267+0000","last_clean":"2024-01-15T18:12:34.242267+0000","last_became_active":"2024-01-15T18:12:32.723775+0000","last_became_peered":"2024-01-15T18:12:32.723775+0000","last_unstale":"2024-01-15T18:12:34.242267+0000","last_undegraded":"2024-01-15T18:12:34.242267+0000","last_fullsized":"2024-01-15T18:12:34.242267+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:09:23.111371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.690495+0000","last_change":"2024-01-15T18:12:32.171656+0000","last_active":"2024-01-15T18:12:33.690495+0000","last_peered":"2024-01-15T18:12:33.690495+0000","last_clean":"2024-01-15T18:12:33.690495+0000","last_became_active":"2024-01-15T18:12:32.170575+0000","last_became_peered":"2024-01-15T18:12:32.170575+0000","last_unstale":"2024-01-15T18:12:33.690495+0000","last_undegraded":"2024-01-15T18:12:33.690495+0000","last_fullsized":"2024-01-15T18:12:33.690495+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:40:40.732153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:33.698060+0000","last_change":"2024-01-15T18:12:32.716821+0000","last_active":"2024-01-15T18:12:33.698060+0000","last_peered":"2024-01-15T18:12:33.698060+0000","last_clean":"2024-01-15T18:12:33.698060+0000","last_became_active":"2024-01-15T18:12:32.716516+0000","last_became_peered":"2024-01-15T18:12:32.716516+0000","last_unstale":"2024-01-15T18:12:33.698060+0000","last_undegraded":"2024-01-15T18:12:33.698060+0000","last_fullsized":"2024-01-15T18:12:33.698060+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:22:42.724333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242470+0000","last_change":"2024-01-15T18:12:32.174584+0000","last_active":"2024-01-15T18:12:34.242470+0000","last_peered":"2024-01-15T18:12:34.242470+0000","last_clean":"2024-01-15T18:12:34.242470+0000","last_became_active":"2024-01-15T18:12:32.174154+0000","last_became_peered":"2024-01-15T18:12:32.174154+0000","last_unstale":"2024-01-15T18:12:34.242470+0000","last_undegraded":"2024-01-15T18:12:34.242470+0000","last_fullsized":"2024-01-15T18:12:34.242470+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:54.618023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241792+0000","last_change":"2024-01-15T18:12:32.722270+0000","last_active":"2024-01-15T18:12:34.241792+0000","last_peered":"2024-01-15T18:12:34.241792+0000","last_clean":"2024-01-15T18:12:34.241792+0000","last_became_active":"2024-01-15T18:12:32.721573+0000","last_became_peered":"2024-01-15T18:12:32.721573+0000","last_unstale":"2024-01-15T18:12:34.241792+0000","last_undegraded":"2024-01-15T18:12:34.241792+0000","last_fullsized":"2024-01-15T18:12:34.241792+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:19:43.275580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.240819+0000","last_change":"2024-01-15T18:12:32.170398+0000","last_active":"2024-01-15T18:12:34.240819+0000","last_peered":"2024-01-15T18:12:34.240819+0000","last_clean":"2024-01-15T18:12:34.240819+0000","last_became_active":"2024-01-15T18:12:32.168116+0000","last_became_peered":"2024-01-15T18:12:32.168116+0000","last_unstale":"2024-01-15T18:12:34.240819+0000","last_undegraded":"2024-01-15T18:12:34.240819+0000","last_fullsized":"2024-01-15T18:12:34.240819+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:12:58.348630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242709+0000","last_change":"2024-01-15T18:12:32.712912+0000","last_active":"2024-01-15T18:12:34.242709+0000","last_peered":"2024-01-15T18:12:34.242709+0000","last_clean":"2024-01-15T18:12:34.242709+0000","last_became_active":"2024-01-15T18:12:32.712628+0000","last_became_peered":"2024-01-15T18:12:32.712628+0000","last_unstale":"2024-01-15T18:12:34.242709+0000","last_undegraded":"2024-01-15T18:12:34.242709+0000","last_fullsized":"2024-01-15T18:12:34.242709+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:45:16.896538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.241999+0000","last_change":"2024-01-15T18:12:32.166071+0000","last_active":"2024-01-15T18:12:34.241999+0000","last_peered":"2024-01-15T18:12:34.241999+0000","last_clean":"2024-01-15T18:12:34.241999+0000","last_became_active":"2024-01-15T18:12:32.165790+0000","last_became_peered":"2024-01-15T18:12:32.165790+0000","last_unstale":"2024-01-15T18:12:34.241999+0000","last_undegraded":"2024-01-15T18:12:34.241999+0000","last_fullsized":"2024-01-15T18:12:34.241999+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:07:01.361758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.243727+0000","last_change":"2024-01-15T18:12:32.722187+0000","last_active":"2024-01-15T18:12:34.243727+0000","last_peered":"2024-01-15T18:12:34.243727+0000","last_clean":"2024-01-15T18:12:34.243727+0000","last_became_active":"2024-01-15T18:12:32.721917+0000","last_became_peered":"2024-01-15T18:12:32.721917+0000","last_unstale":"2024-01-15T18:12:34.243727+0000","last_undegraded":"2024-01-15T18:12:34.243727+0000","last_fullsized":"2024-01-15T18:12:34.243727+0000","mapping_epoch":101,"log_start":"0'0","ondisk_log_start":"0'0","created":101,"last_epoch_clean":102,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:32.126372+0000","last_clean_scrub_stamp":"2024-01-15T18:12:32.126372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:25:36.412516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"102'1","reported_seq":13,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-01-15T18:12:34.242951+0000","last_change":"2024-01-15T18:12:32.174284+0000","last_active":"2024-01-15T18:12:34.242951+0000","last_peered":"2024-01-15T18:12:34.242951+0000","last_clean":"2024-01-15T18:12:34.242951+0000","last_became_active":"2024-01-15T18:12:32.174011+0000","last_became_peered":"2024-01-15T18:12:32.174011+0000","last_unstale":"2024-01-15T18:12:34.242951+0000","last_undegraded":"2024-01-15T18:12:34.242951+0000","last_fullsized":"2024-01-15T18:12:34.242951+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":100,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:12:31.110271+0000","last_clean_scrub_stamp":"2024-01-15T18:12:31.110271+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:22:47.806633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":240,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-01-15T18:12:32.693861+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:12:32.693861+0000","last_peered":"2024-01-15T18:12:32.693861+0000","last_clean":"2024-01-15T18:12:32.693861+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:12:32.693861+0000","last_undegraded":"2024-01-15T18:12:32.693861+0000","last_fullsized":"2024-01-15T18:12:32.693861+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542328,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64880,"kb_used_data":1828,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306960,"statfs":{"total":96636764160,"available":96570327040,"internally_reserved":0,"allocated":1871872,"data_stored":1145010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.329,"15min":1.4039999999999999},"min":{"1min":0.747,"5min":0.63800000000000001,"15min":0.53600000000000003},"max":{"1min":1.9359999999999999,"5min":4.0019999999999998,"15min":14.114000000000001},"last":1.482},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.7090000000000001,"15min":1.5700000000000001},"min":{"1min":0.94199999999999995,"5min":0.872,"15min":0.51700000000000002},"max":{"1min":2.0059999999999998,"5min":27.32,"15min":27.32},"last":1.304}]},{"osd":1,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.3740000000000001,"15min":1.5069999999999999},"min":{"1min":0.93999999999999995,"5min":0.72699999999999998,"15min":0.40300000000000002},"max":{"1min":5.9690000000000003,"5min":8.3030000000000008,"15min":29.167999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3109999999999999,"15min":1.3979999999999999},"min":{"1min":0.88200000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7969999999999999,"5min":4.2699999999999996,"15min":12.565},"last":0.91700000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.3049999999999999,"15min":1.4870000000000001},"min":{"1min":0.97099999999999997,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.6439999999999999,"5min":4.1920000000000002,"15min":13.801},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.651,"15min":1.5289999999999999},"min":{"1min":0.82099999999999995,"5min":0.76400000000000001,"15min":0.56599999999999995},"max":{"1min":3.5920000000000001,"5min":27.358000000000001,"15min":27.358000000000001},"last":0.98699999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6870000000000001,"5min":1.544,"15min":1.6319999999999999},"min":{"1min":1.0700000000000001,"5min":0.72199999999999998,"15min":0.47099999999999997},"max":{"1min":6.133,"5min":10.476000000000001,"15min":29.204999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.4730000000000001,"15min":1.5740000000000001},"min":{"1min":1.006,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.101,"5min":4.3879999999999999,"15min":16.088999999999999},"last":1.03}]},{"osd":4,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.6910000000000001,"15min":1.581},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":1.5329999999999999,"5min":27.219000000000001,"15min":27.219000000000001},"last":1.135},{"interface":"front","average":{"1min":1.907,"5min":1.5649999999999999,"15min":1.6830000000000001},"min":{"1min":0.88500000000000001,"5min":0.63400000000000001,"15min":0.438},"max":{"1min":6.4710000000000001,"5min":8.4199999999999999,"15min":29.25},"last":1.3680000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.5229999999999999,"15min":1.6539999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.885,"5min":6.4960000000000004,"15min":12.518000000000001},"last":1.611},{"interface":"front","average":{"1min":1.405,"5min":1.847,"15min":1.6930000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":1.728,"5min":27.283000000000001,"15min":27.283000000000001},"last":1.421}]},{"osd":6,"last update":"Mon Jan 15 18:11:58 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.976,"15min":2.3460000000000001},"min":{"1min":1.2030000000000001,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":1.8160000000000001,"5min":27.393999999999998,"15min":53.664999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.776,"5min":1.726,"15min":1.835},"min":{"1min":1.1719999999999999,"5min":0.57299999999999995,"15min":0.44},"max":{"1min":6.1870000000000003,"5min":8.1980000000000004,"15min":29.106999999999999},"last":1.536}]}]},{"osd":6,"up_from":14,"seq":60129542328,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64292,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307548,"statfs":{"total":96636764160,"available":96570929152,"internally_reserved":0,"allocated":1265664,"data_stored":548770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.294,"15min":1.2310000000000001},"min":{"1min":0.59499999999999997,"5min":0.58699999999999997,"15min":0.438},"max":{"1min":8.8350000000000009,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.292,"15min":1.3540000000000001},"min":{"1min":0.71799999999999997,"5min":0.61099999999999999,"15min":0.59599999999999997},"max":{"1min":1.851,"5min":3.1659999999999999,"15min":21.565000000000001},"last":0.75}]},{"osd":1,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.4410000000000001,"15min":1.3740000000000001},"min":{"1min":0.88300000000000001,"5min":0.71099999999999997,"15min":0.52100000000000002},"max":{"1min":8.8870000000000005,"5min":8.8870000000000005,"15min":8.8870000000000005},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.196,"5min":1.3300000000000001,"15min":1.4199999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.4970000000000001,"5min":2.73,"15min":13.984999999999999},"last":1.1259999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.335,"15min":1.4530000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.5249999999999999,"5min":4.351,"15min":34.670000000000002},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.4890000000000001,"15min":1.4370000000000001},"min":{"1min":0.92400000000000004,"5min":0.90200000000000002,"15min":0.49399999999999999},"max":{"1min":8.9359999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":0.93200000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4870000000000001,"15min":1.5680000000000001},"min":{"1min":0.96899999999999997,"5min":0.81499999999999995,"15min":0.68500000000000005},"max":{"1min":1.732,"5min":3.5680000000000001,"15min":16.855},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.5489999999999999,"15min":1.6779999999999999},"min":{"1min":1.137,"5min":0.89100000000000001,"15min":0.48499999999999999},"max":{"1min":1.897,"5min":5.1379999999999999,"15min":34.725000000000001},"last":1.2889999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.456,"15min":1.462},"min":{"1min":0.92000000000000004,"5min":0.80600000000000005,"15min":0.38400000000000001},"max":{"1min":8.7539999999999996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.581},{"interface":"front","average":{"1min":1.333,"5min":1.5049999999999999,"15min":1.5109999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.501},"max":{"1min":1.976,"5min":7.3230000000000004,"15min":10.118},"last":1.359}]},{"osd":5,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":2.117,"5min":1.653,"15min":1.7050000000000001},"min":{"1min":0.92000000000000004,"5min":0.82699999999999996,"15min":0.57699999999999996},"max":{"1min":16.309999999999999,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.427},{"interface":"front","average":{"1min":1.857,"5min":1.698,"15min":1.631},"min":{"1min":1.141,"5min":0.80500000000000005,"15min":0.47899999999999998},"max":{"1min":8.9900000000000002,"5min":8.9900000000000002,"15min":10.077},"last":1.502}]},{"osd":7,"last update":"Mon Jan 15 18:11:54 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.623,"15min":1.645},"min":{"1min":1.111,"5min":1.111,"15min":0.64000000000000001},"max":{"1min":2.0379999999999998,"5min":5.5060000000000002,"15min":10.302},"last":1.462},{"interface":"front","average":{"1min":1.512,"5min":1.635,"15min":1.8260000000000001},"min":{"1min":1.048,"5min":0.94499999999999995,"15min":0.52600000000000002},"max":{"1min":2.008,"5min":5.298,"15min":34.762999999999998},"last":1.5269999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542330,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58792,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313048,"statfs":{"total":96636764160,"available":96576561152,"internally_reserved":0,"allocated":1273856,"data_stored":554600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.292,"15min":1.381},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":1.6419999999999999,"5min":8.0950000000000006,"15min":26.077999999999999},"last":1.23},{"interface":"front","average":{"1min":1.7569999999999999,"5min":1.4590000000000001,"15min":1.49},"min":{"1min":0.84799999999999998,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":9.8699999999999992,"5min":9.8699999999999992,"15min":14.711},"last":1.091}]},{"osd":1,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.393,"15min":1.413},"min":{"1min":0.79400000000000004,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.8279999999999998,"5min":8.702,"15min":26},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3740000000000001,"15min":1.3700000000000001},"min":{"1min":0.751,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":2.8660000000000001,"5min":7.5309999999999997,"15min":12.619},"last":1.0129999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4850000000000001,"15min":1.544},"min":{"1min":0.95599999999999996,"5min":0.64300000000000002,"15min":0.55700000000000005},"max":{"1min":1.651,"5min":8.7739999999999991,"15min":26.234999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.744,"5min":1.6599999999999999,"15min":1.6299999999999999},"min":{"1min":1.05,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":10.09,"5min":10.09,"15min":17.126999999999999},"last":1.2769999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.321,"15min":1.446},"min":{"1min":0.95799999999999996,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.841,"5min":7.6150000000000002,"15min":12.722},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.5029999999999999,"15min":1.643},"min":{"1min":1.0509999999999999,"5min":0.70399999999999996,"15min":0.63400000000000001},"max":{"1min":1.8380000000000001,"5min":8.9730000000000008,"15min":26.134},"last":0.84599999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.409,"15min":1.5369999999999999},"min":{"1min":0.995,"5min":0.442,"15min":0.442},"max":{"1min":9.9979999999999993,"5min":9.9979999999999993,"15min":25.776},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.9239999999999999,"5min":1.7549999999999999,"15min":1.766},"min":{"1min":1.1080000000000001,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":12.103,"5min":15.52,"15min":16.414000000000001},"last":1.3600000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.4159999999999999,"15min":1.5389999999999999},"min":{"1min":0.871,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.7609999999999999,"5min":7.9240000000000004,"15min":12.769},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5389999999999999,"15min":1.802},"min":{"1min":0.99299999999999999,"5min":0.72899999999999998,"15min":0.435},"max":{"1min":2.0259999999999998,"5min":10.571999999999999,"15min":36.087000000000003},"last":1.3240000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:11:49 2024","interfaces":[{"interface":"back","average":{"1min":2.0649999999999999,"5min":1.8480000000000001,"15min":1.9239999999999999},"min":{"1min":1.2649999999999999,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":12.326000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.494},{"interface":"front","average":{"1min":1.427,"5min":1.601,"15min":1.7709999999999999},"min":{"1min":1.04,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":1.875,"5min":12.175000000000001,"15min":21.728000000000002},"last":1.391}]}]},{"osd":4,"up_from":14,"seq":60129542329,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64316,"kb_used_data":1260,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307524,"statfs":{"total":96636764160,"available":96570904576,"internally_reserved":0,"allocated":1290240,"data_stored":556266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.238,"15min":1.379},"min":{"1min":0.57699999999999996,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":3.4100000000000001,"5min":4.8170000000000002,"15min":21.643000000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.272,"5min":1.288,"15min":1.3089999999999999},"min":{"1min":0.87,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.7130000000000001,"5min":4.5590000000000002,"15min":11.196},"last":1.004}]},{"osd":1,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":1.6020000000000001,"15min":1.4079999999999999},"min":{"1min":0.84399999999999997,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.919,"5min":28.663,"15min":28.663},"last":1.516},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.3919999999999999,"15min":1.52},"min":{"1min":0.89300000000000002,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":3.5299999999999998,"5min":6.4870000000000001,"15min":21.440000000000001},"last":1.145}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.5289999999999999,"15min":1.46},"min":{"1min":0.80100000000000005,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":1.9570000000000001,"5min":16.023,"15min":16.023},"last":1.3069999999999999},{"interface":"front","average":{"1min":2.371,"5min":1.581,"15min":1.589},"min":{"1min":0.751,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":18.858000000000001,"5min":18.858000000000001,"15min":22.350999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.369,"15min":1.506},"min":{"1min":0.59099999999999997,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":2.1230000000000002,"5min":5.0860000000000003,"15min":15.093},"last":1.486},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.7030000000000001,"15min":1.667},"min":{"1min":0.89200000000000002,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":2.073,"5min":18.027000000000001,"15min":19.486000000000001},"last":0.96999999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.4330000000000001,"15min":1.778},"min":{"1min":0.5,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":3.6429999999999998,"5min":5.0270000000000001,"15min":54.938000000000002},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.5449999999999999,"15min":1.524},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":1.992,"5min":8.5660000000000007,"15min":18.289999999999999},"last":1.2010000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.7470000000000001,"15min":1.631},"min":{"1min":0.72599999999999998,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.347,"5min":16.103999999999999,"15min":17.733000000000001},"last":1.242},{"interface":"front","average":{"1min":1.681,"5min":1.7729999999999999,"15min":1.871},"min":{"1min":0.621,"5min":0.621,"15min":0.499},"max":{"1min":3.7280000000000002,"5min":12.936999999999999,"15min":28.797000000000001},"last":1.4079999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.583,"5min":1.863,"15min":1.8240000000000001},"min":{"1min":0.82199999999999995,"5min":0.75800000000000001,"15min":0.45100000000000001},"max":{"1min":2.468,"5min":23.390000000000001,"15min":23.390000000000001},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.7430000000000001,"5min":1.9550000000000001,"15min":1.8680000000000001},"min":{"1min":1.369,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.2410000000000001,"5min":16.763000000000002,"15min":22.751000000000001},"last":1.4510000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542330,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72632,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299208,"statfs":{"total":96636764160,"available":96562388992,"internally_reserved":0,"allocated":1286144,"data_stored":556164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.94699999999999995,"15min":1.0700000000000001},"min":{"1min":0.66400000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.347,"5min":2.0350000000000001,"15min":36.408000000000001},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.155,"5min":1.0660000000000001,"15min":1.052},"min":{"1min":0.72299999999999998,"5min":0.42399999999999999,"15min":0.377},"max":{"1min":1.6839999999999999,"5min":3.1440000000000001,"15min":5.0529999999999999},"last":0.83099999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.079,"15min":1.1080000000000001},"min":{"1min":0.80800000000000005,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.9590000000000001,"5min":4.1109999999999998,"15min":8.1359999999999992},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.202,"15min":1.2010000000000001},"min":{"1min":0.79200000000000004,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":2.5790000000000002,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":0.65500000000000003}]},{"osd":2,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.486,"15min":1.2929999999999999},"min":{"1min":1.0529999999999999,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":2.0859999999999999,"5min":14.025,"15min":14.025},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.3720000000000001,"15min":1.3049999999999999},"min":{"1min":0.72099999999999997,"5min":0.58099999999999996,"15min":0.46000000000000002},"max":{"1min":2.7130000000000001,"5min":3.6240000000000001,"15min":5.0910000000000002},"last":1.022}]},{"osd":4,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.4810000000000001,"15min":1.4390000000000001},"min":{"1min":1.0649999999999999,"5min":0.67400000000000004,"15min":0.59999999999999998},"max":{"1min":1.9379999999999999,"5min":9.0069999999999997,"15min":17.082000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.744,"5min":1.718,"15min":1.575},"min":{"1min":0.97799999999999998,"5min":0.81599999999999995,"15min":0.69499999999999995},"max":{"1min":2.6230000000000002,"5min":15.769,"15min":15.769},"last":1.079}]},{"osd":5,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":1.645,"15min":1.5780000000000001},"min":{"1min":1.22,"5min":0.77100000000000002,"15min":0.65000000000000002},"max":{"1min":2.7440000000000002,"5min":5.5220000000000002,"15min":5.5220000000000002},"last":1.609},{"interface":"front","average":{"1min":1.6919999999999999,"5min":1.839,"15min":1.921},"min":{"1min":1.2589999999999999,"5min":0.97599999999999998,"15min":0.58299999999999996},"max":{"1min":2.6659999999999999,"5min":21.713000000000001,"15min":40.384999999999998},"last":1.1870000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":2.1190000000000002,"15min":1.673},"min":{"1min":1.2170000000000001,"5min":0.69599999999999995,"15min":0.51400000000000001},"max":{"1min":3.6800000000000002,"5min":32.259999999999998,"15min":32.259999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.845,"5min":1.744,"15min":1.7370000000000001},"min":{"1min":1.3620000000000001,"5min":0.94799999999999995,"15min":0.747},"max":{"1min":2.863,"5min":6.8639999999999999,"15min":16.195},"last":1.2769999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.74,"5min":1.9159999999999999,"15min":1.9039999999999999},"min":{"1min":1.1850000000000001,"5min":0.88300000000000001,"15min":0.63900000000000001},"max":{"1min":2.798,"5min":10.035,"15min":24.146999999999998},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.659,"5min":1.9990000000000001,"15min":1.7470000000000001},"min":{"1min":1.196,"5min":0.82699999999999996,"15min":0.56100000000000005},"max":{"1min":2.411,"5min":17.622,"15min":17.622},"last":1.5640000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542329,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68448,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303392,"statfs":{"total":96636764160,"available":96566673408,"internally_reserved":0,"allocated":1261568,"data_stored":548152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.95099999999999996,"15min":1.002},"min":{"1min":0.60299999999999998,"5min":0.44400000000000001,"15min":0.30099999999999999},"max":{"1min":1.7350000000000001,"5min":3.5670000000000002,"15min":6.101},"last":0.85699999999999998},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1259999999999999},"min":{"1min":0.65600000000000003,"5min":0.41199999999999998,"15min":0.28299999999999997},"max":{"1min":1.946,"5min":10.319000000000001,"15min":10.319000000000001},"last":0.95499999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.0549999999999999,"15min":1.1100000000000001},"min":{"1min":0.72099999999999997,"5min":0.48099999999999998,"15min":0.379},"max":{"1min":1.984,"5min":4.4939999999999998,"15min":10.807},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2549999999999999,"15min":1.2210000000000001},"min":{"1min":0.82899999999999996,"5min":0.58799999999999997,"15min":0.35299999999999998},"max":{"1min":2.1190000000000002,"5min":7.5119999999999996,"15min":7.5119999999999996},"last":1.097}]},{"osd":3,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.3220000000000001,"15min":1.4590000000000001},"min":{"1min":0.70599999999999996,"5min":0.61499999999999999,"15min":0.41399999999999998},"max":{"1min":1.7110000000000001,"5min":4.5839999999999996,"15min":25.891999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.216,"15min":1.266},"min":{"1min":0.89400000000000002,"5min":0.47199999999999998,"15min":0.376},"max":{"1min":2.1640000000000001,"5min":2.6899999999999999,"15min":14.946},"last":1.044}]},{"osd":4,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4259999999999999,"15min":1.5229999999999999},"min":{"1min":1.0609999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.8700000000000001,"5min":4.8159999999999998,"15min":9.0679999999999996},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.716,"15min":1.587},"min":{"1min":1.0609999999999999,"5min":0.72399999999999998,"15min":0.54500000000000004},"max":{"1min":8.4269999999999996,"5min":10.279999999999999,"15min":10.279999999999999},"last":1.248}]},{"osd":5,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4810000000000001,"5min":1.5640000000000001,"15min":1.671},"min":{"1min":1.0740000000000001,"5min":0.71799999999999997,"15min":0.63500000000000001},"max":{"1min":1.905,"5min":7.4939999999999998,"15min":11.071999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.532,"5min":1.5660000000000001,"15min":1.7150000000000001},"min":{"1min":1.1200000000000001,"5min":0.83799999999999997,"15min":0.55200000000000005},"max":{"1min":1.9810000000000001,"5min":5.774,"15min":37.020000000000003},"last":1.4670000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":2.0179999999999998,"5min":1.7410000000000001,"15min":1.728},"min":{"1min":1.1859999999999999,"5min":0.85199999999999998,"15min":0.64800000000000002},"max":{"1min":8.5519999999999996,"5min":10.185,"15min":23.762},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.653,"5min":1.913,"15min":1.889},"min":{"1min":1.3,"5min":1.0129999999999999,"15min":0.73499999999999999},"max":{"1min":2.1709999999999998,"5min":22.382000000000001,"15min":39.926000000000002},"last":1.286}]},{"osd":7,"last update":"Mon Jan 15 18:11:45 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":1.8120000000000001,"15min":1.9570000000000001},"min":{"1min":1.4199999999999999,"5min":1.0169999999999999,"15min":0.68999999999999995},"max":{"1min":8.6620000000000008,"5min":10.24,"15min":52.767000000000003},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.696,"5min":1.873,"15min":1.821},"min":{"1min":1.24,"5min":0.97899999999999998,"15min":0.83499999999999996},"max":{"1min":2.3039999999999998,"5min":11.893000000000001,"15min":13.055999999999999},"last":1.153}]}]},{"osd":1,"up_from":14,"seq":60129542329,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72620,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299220,"statfs":{"total":96636764160,"available":96562401280,"internally_reserved":0,"allocated":1269760,"data_stored":554530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.2270000000000001,"15min":1.0549999999999999},"min":{"1min":0.63800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":6.2640000000000002,"5min":19.218,"15min":19.218},"last":1.151},{"interface":"front","average":{"1min":1.2270000000000001,"5min":1.0900000000000001,"15min":1.1919999999999999},"min":{"1min":0.45200000000000001,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":2.371,"5min":4.6349999999999998,"15min":19.823},"last":0.94099999999999995}]},{"osd":2,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.0209999999999999,"15min":1.1140000000000001},"min":{"1min":0.746,"5min":0.35499999999999998,"15min":0.31900000000000001},"max":{"1min":3.4359999999999999,"5min":3.4359999999999999,"15min":7.6680000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.27},"min":{"1min":1.1379999999999999,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":1.829,"5min":19.318000000000001,"15min":19.318000000000001},"last":1.2749999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.5209999999999999,"15min":1.3859999999999999},"min":{"1min":1.081,"5min":0.51000000000000001,"15min":0.40899999999999997},"max":{"1min":2.4289999999999998,"5min":19.387,"15min":19.387},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.1839999999999999,"15min":1.343},"min":{"1min":0.58499999999999996,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":1.5589999999999999,"5min":3.552,"15min":19.733000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1829999999999998,"5min":1.802,"15min":1.5920000000000001},"min":{"1min":1.2949999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":8.0559999999999992,"5min":19.456,"15min":19.456},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.46,"15min":1.667},"min":{"1min":1.169,"5min":0.56499999999999995,"15min":0.44800000000000001},"max":{"1min":1.8480000000000001,"5min":3.3180000000000001,"15min":22.204000000000001},"last":1.591}]},{"osd":5,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.4530000000000001,"15min":1.6779999999999999},"min":{"1min":0.91300000000000003,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":2.145,"5min":2.351,"15min":20.012},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.625,"5min":1.4510000000000001,"15min":1.6379999999999999},"min":{"1min":1.0980000000000001,"5min":0.69299999999999995,"15min":0.499},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":19.998999999999999},"last":1.548}]},{"osd":6,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.8560000000000001,"5min":1.776,"15min":1.7130000000000001},"min":{"1min":1.3839999999999999,"5min":0.82099999999999995,"15min":0.54100000000000004},"max":{"1min":2.4870000000000001,"5min":19.526,"15min":19.526},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.597,"15min":1.8180000000000001},"min":{"1min":1.1040000000000001,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":2.2679999999999998,"5min":6.468,"15min":27.579999999999998},"last":1.2210000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:11:56 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.746,"15min":1.8120000000000001},"min":{"1min":1.155,"5min":0.63,"15min":0.505},"max":{"1min":2.3130000000000002,"5min":20.132000000000001,"15min":20.132000000000001},"last":1.738},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.9119999999999999,"15min":1.8340000000000001},"min":{"1min":1.3919999999999999,"5min":0.92400000000000004,"15min":0.54500000000000004},"max":{"1min":2.4319999999999999,"5min":27.582000000000001,"15min":27.582000000000001},"last":1.6839999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542328,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69036,"kb_used_data":1824,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302804,"statfs":{"total":96636764160,"available":96566071296,"internally_reserved":0,"allocated":1867776,"data_stored":1145480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.0369999999999999,"15min":1.1839999999999999},"min":{"1min":0.56200000000000006,"5min":0.34200000000000003,"15min":0.30599999999999999},"max":{"1min":14.542,"5min":14.542,"15min":20.437999999999999},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.109,"5min":1.147,"15min":1.2110000000000001},"min":{"1min":0.437,"5min":0.39900000000000002,"15min":0.32400000000000001},"max":{"1min":1.9430000000000001,"5min":3.492,"15min":35.137999999999998},"last":0.92000000000000004}]},{"osd":2,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.9450000000000001,"5min":1.284,"15min":1.3480000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.42799999999999999},"max":{"1min":14.651,"5min":14.651,"15min":20.556000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.264,"15min":1.3740000000000001},"min":{"1min":0.70699999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.2930000000000001,"5min":4.7430000000000003,"15min":35.183},"last":1.2190000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.1060000000000001,"15min":1.155},"min":{"1min":0.627,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":1.792,"5min":2.6789999999999998,"15min":8.9949999999999992},"last":1.161},{"interface":"front","average":{"1min":1.974,"5min":1.462,"15min":1.6200000000000001},"min":{"1min":0.64800000000000002,"5min":0.47199999999999998,"15min":0.45100000000000001},"max":{"1min":14.598000000000001,"5min":14.598000000000001,"15min":25.372},"last":0.76000000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4870000000000001,"15min":1.6699999999999999},"min":{"1min":0.86199999999999999,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":2.0680000000000001,"5min":4.431,"15min":35.223999999999997},"last":1.399},{"interface":"front","average":{"1min":1.385,"5min":1.369,"15min":1.452},"min":{"1min":0.96499999999999997,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":2.1179999999999999,"5min":2.407,"15min":15.968999999999999},"last":1.4510000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5980000000000001,"5min":1.645,"15min":1.621},"min":{"1min":1.099,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":2.133,"5min":7.9160000000000004,"15min":12.173999999999999},"last":1.661},{"interface":"front","average":{"1min":2.2320000000000002,"5min":1.78,"15min":1.8440000000000001},"min":{"1min":0.88300000000000001,"5min":0.85299999999999998,"15min":0.76000000000000001},"max":{"1min":14.859,"5min":14.859,"15min":20.789999999999999},"last":1.2569999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":2.3210000000000002,"5min":1.9419999999999999,"15min":2.032},"min":{"1min":1.141,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":15.022,"5min":16.023,"15min":36.390999999999998},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.615,"5min":1.669,"15min":1.9139999999999999},"min":{"1min":0.99399999999999999,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":2.2759999999999998,"5min":5.3239999999999998,"15min":36.207000000000001},"last":1.7210000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:11:50 2024","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.7130000000000001,"15min":1.9350000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":2.1819999999999999,"5min":9.8580000000000005,"15min":35.636000000000003},"last":1.845},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.7769999999999999,"15min":1.665},"min":{"1min":1.129,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0529999999999999,"5min":11.470000000000001,"15min":11.470000000000001},"last":1.552}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:12:38.200 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:12:38.200 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:12:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.429+0000 7f92f4a00640 1 Processor -- start 2024-01-15T18:12:38.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.429+0000 7f92f4a00640 1 -- start start 2024-01-15T18:12:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.430+0000 7f92f4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 0x7f92f00cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.430+0000 7f92f4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f92f0071eb0 0x7f92f015be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.430+0000 7f92f4a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92f0071460 con 0x7f92f006a4b0 2024-01-15T18:12:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.430+0000 7f92f4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92f00cd490 con 0x7f92f0071eb0 2024-01-15T18:12:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.430+0000 7f92f4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0071a20 con 0x7f92f015ba60 2024-01-15T18:12:38.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.431+0000 7f92ef200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f92f006a4b0 0x7f92f006e8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60570/0 (socket says 172.21.15.2:60570) 2024-01-15T18:12:38.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.431+0000 7f92ef200640 1 -- 172.21.15.2:0/173474221 learned_addr learned my addr 172.21.15.2:0/173474221 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:38.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.431+0000 7f92ede00640 1 --2- 172.21.15.2:0/173474221 >> v2:172.21.15.137:3300/0 conn(0x7f92f0071eb0 0x7f92f015be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.431+0000 7f92ee800640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 0x7f92f00cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.431+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792292638 0 0) 0x7f92f0071460 con 0x7f92f006a4b0 2024-01-15T18:12:38.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.431+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92d0003520 con 0x7f92f006a4b0 2024-01-15T18:12:38.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 907363219 0 0) 0x7f92d0003520 con 0x7f92f006a4b0 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 msgr2=0x7f92f00cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 0x7f92f00cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 >> v2:172.21.15.137:3300/0 conn(0x7f92f0071eb0 msgr2=0x7f92f015be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 --2- 172.21.15.2:0/173474221 >> v2:172.21.15.137:3300/0 conn(0x7f92f0071eb0 0x7f92f015be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f0072330 con 0x7f92f006a4b0 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f92e80019f0 con 0x7f92f006a4b0 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 >> v1:172.21.15.2:6789/0 conn(0x7f92f006a4b0 legacy=0x7f92f006e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d0003970 0x7f92d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 0x7f92d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0004d00 0x7f92d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f00cd490 con 0x7f92d0004d00 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0071a20 con 0x7f92d0003970 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92d000b980 con 0x7f92f015ba60 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ef200640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0004d00 0x7f92d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ee800640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d0003970 0x7f92d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.432+0000 7f92ede00640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 0x7f92d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.433+0000 7f92ef200640 1 -- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 msgr2=0x7f92d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.433+0000 7f92ef200640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92f015ba60 0x7f92d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.433+0000 7f92ef200640 1 -- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d0003970 msgr2=0x7f92d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.433+0000 7f92ef200640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92d0003970 0x7f92d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.433+0000 7f92ef200640 1 -- 172.21.15.2:0/173474221 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92d0003520 con 0x7f92d0004d00 2024-01-15T18:12:38.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92ef200640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0004d00 0x7f92d0005180 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f92e800a830 tx=0x7f92d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49750a2920093ccc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:38.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8034630 con 0x7f92d0004d00 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e802f070 con 0x7f92d0004d00 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92ed400640 1 -- 172.21.15.2:0/173474221 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e80349f0 con 0x7f92d0004d00 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92f4a00640 1 -- 172.21.15.2:0/173474221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0004d00 msgr2=0x7f92d0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92f4a00640 1 --2- 172.21.15.2:0/173474221 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92d0004d00 0x7f92d0005180 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f92e800a830 tx=0x7f92d00057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.435+0000 7f92ef200640 1 -- 172.21.15.2:0/173474221 reap_dead start 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.436+0000 7f92f4a00640 1 -- 172.21.15.2:0/173474221 shutdown_connections 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.436+0000 7f92f4a00640 1 -- 172.21.15.2:0/173474221 >> 172.21.15.2:0/173474221 conn(0x7f92f00cff90 msgr2=0x7f92f00d03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.436+0000 7f92f4a00640 1 -- 172.21.15.2:0/173474221 shutdown_connections 2024-01-15T18:12:38.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.436+0000 7f92f4a00640 1 -- 172.21.15.2:0/173474221 wait complete. 2024-01-15T18:12:38.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.438+0000 7f92f4a00640 1 Processor -- start 2024-01-15T18:12:38.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.438+0000 7f92f4a00640 1 -- start start 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92f4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d0004d00 0x7f92f0141730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92f4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 0x7f92f015d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92f4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92f0142290 0x7f92f0142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92f4a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f015dcc0 con 0x7f92f015d300 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92f4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92f0143000 con 0x7f92f0142290 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92f4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92f01432e0 con 0x7f92d0004d00 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92ede00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 0x7f92f015d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92ede00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 0x7f92f015d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52200/0 (socket says 172.21.15.2:52200) 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.439+0000 7f92ede00640 1 -- 172.21.15.2:0/1778907196 learned_addr learned my addr 172.21.15.2:0/1778907196 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92ede00640 1 -- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d0004d00 msgr2=0x7f92f0141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92ede00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d0004d00 0x7f92f0141730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92ede00640 1 -- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92f0142290 msgr2=0x7f92f0142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92ede00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92f0142290 0x7f92f0142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92ede00640 1 -- 172.21.15.2:0/1778907196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f01435c0 con 0x7f92f015d300 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92ede00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 0x7f92f015d780 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f92d8011a70 tx=0x7f92d8011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d8009680 con 0x7f92f015d300 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d800fad0 con 0x7f92f015d300 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92f01438a0 con 0x7f92f015d300 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d800fe60 con 0x7f92f015d300 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.440+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92f0141c70 con 0x7f92f015d300 2024-01-15T18:12:38.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.441+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92f0074770 con 0x7f92f015d300 2024-01-15T18:12:38.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.442+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f92d8014440 con 0x7f92f015d300 2024-01-15T18:12:38.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.443+0000 7f92e6a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92c806c220 0x7f92c806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.443+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f92d8088f80 con 0x7f92f015d300 2024-01-15T18:12:38.447 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.444+0000 7f92ee800640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92c806c220 0x7f92c806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.445+0000 7f92ee800640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92c806c220 0x7f92c806e6e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009a40 tx=0x7f92e0009320 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:38.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.447+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92d8055840 con 0x7f92f015d300 2024-01-15T18:12:38.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.713+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f92f00cc3a0 con 0x7f92f015d300 2024-01-15T18:12:38.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.718+0000 7f92e6a00640 1 -- 172.21.15.2:0/1778907196 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+147 (secure 0 0 0) 0x7f92d8058e60 con 0x7f92f015d300 2024-01-15T18:12:38.722 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T18:12:38.722 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T18:12:38.722 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T18:12:38.722 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T18:12:38.722 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:12:38.722 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T18:12:38.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92c806c220 msgr2=0x7f92c806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92c806c220 0x7f92c806e6e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f92e0009a40 tx=0x7f92e0009320 comp rx=0 tx=0).stop 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 msgr2=0x7f92f015d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 0x7f92f015d780 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f92d8011a70 tx=0x7f92d8011f30 comp rx=0 tx=0).stop 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 shutdown_connections 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f92c806c220 0x7f92c806e6e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f92f0142290 0x7f92f0142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f92f015d300 0x7f92f015d780 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.720+0000 7f92f4a00640 1 --2- 172.21.15.2:0/1778907196 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f92d0004d00 0x7f92f0141730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.721+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 >> 172.21.15.2:0/1778907196 conn(0x7f92f00cff90 msgr2=0x7f92f00cf480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.721+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 shutdown_connections 2024-01-15T18:12:38.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.721+0000 7f92f4a00640 1 -- 172.21.15.2:0/1778907196 wait complete. 2024-01-15T18:12:38.746 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:12:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.963+0000 7f9142800640 1 Processor -- start 2024-01-15T18:12:38.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.963+0000 7f9142800640 1 -- start start 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9142800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f913c141880 0x7f913c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9142800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 0x7f913c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9142800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c06dae0 con 0x7f913c15ec60 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9142800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c071f20 con 0x7f913c141880 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9142800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c06c6b0 con 0x7f913c15f0e0 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9140a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f913c15ec60 0x7f913c0cd490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60582/0 (socket says 172.21.15.2:60582) 2024-01-15T18:12:38.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.965+0000 7f9140a00640 1 -- 172.21.15.2:0/1520275850 learned_addr learned my addr 172.21.15.2:0/1520275850 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:38.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f9141400640 1 --2- 172.21.15.2:0/1520275850 >> v2:172.21.15.137:3300/0 conn(0x7f913c141880 0x7f913c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f9141e00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 0x7f913c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 467461868 0 0) 0x7f913c06dae0 con 0x7f913c15ec60 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f911c003520 con 0x7f913c15ec60 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2358381567 0 0) 0x7f911c003520 con 0x7f913c15ec60 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 msgr2=0x7f913c0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 0x7f913c0cdba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 >> v2:172.21.15.137:3300/0 conn(0x7f913c141880 msgr2=0x7f913c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 --2- 172.21.15.2:0/1520275850 >> v2:172.21.15.137:3300/0 conn(0x7f913c141880 0x7f913c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c15f570 con 0x7f913c15ec60 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9124001670 con 0x7f913c15ec60 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 >> v1:172.21.15.2:6789/0 conn(0x7f913c15ec60 legacy=0x7f913c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c0039a0 0x7f911c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 0x7f911c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.966+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c071f20 con 0x7f911c0047c0 2024-01-15T18:12:38.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c06c6b0 con 0x7f911c0039a0 2024-01-15T18:12:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911c00ba00 con 0x7f913c15f0e0 2024-01-15T18:12:38.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9140a00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9141400640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c0039a0 0x7f911c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9141e00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 0x7f911c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9140a00640 1 -- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 msgr2=0x7f911c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.973 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9140a00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f913c15f0e0 0x7f911c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9140a00640 1 -- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c0039a0 msgr2=0x7f911c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9140a00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f911c0039a0 0x7f911c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.967+0000 7f9140a00640 1 -- 172.21.15.2:0/1520275850 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c003520 con 0x7f911c0047c0 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.968+0000 7f9140a00640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f912400c0e0 tx=0x7f911c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4d512968e40efb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.969+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124005ad0 con 0x7f911c0047c0 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.969+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124005c50 con 0x7f911c0047c0 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.969+0000 7f913be00640 1 -- 172.21.15.2:0/1520275850 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912402c810 con 0x7f911c0047c0 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.969+0000 7f9142800640 1 -- 172.21.15.2:0/1520275850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 msgr2=0x7f911c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.969+0000 7f9142800640 1 --2- 172.21.15.2:0/1520275850 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f911c0047c0 0x7f911c004c40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f912400c0e0 tx=0x7f911c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.969+0000 7f9141e00640 1 -- 172.21.15.2:0/1520275850 reap_dead start 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.970+0000 7f9142800640 1 -- 172.21.15.2:0/1520275850 shutdown_connections 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.970+0000 7f9142800640 1 -- 172.21.15.2:0/1520275850 >> 172.21.15.2:0/1520275850 conn(0x7f913c0b0840 msgr2=0x7f913c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.970+0000 7f9142800640 1 -- 172.21.15.2:0/1520275850 shutdown_connections 2024-01-15T18:12:38.974 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.970+0000 7f9142800640 1 -- 172.21.15.2:0/1520275850 wait complete. 2024-01-15T18:12:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.971+0000 7f9142800640 1 Processor -- start 2024-01-15T18:12:38.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.972+0000 7f9142800640 1 -- start start 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.972+0000 7f9142800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f911c0047c0 0x7f913c1f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.972+0000 7f9142800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 0x7f913c0b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9142800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c1f5960 0x7f913c1f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9142800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c157cf0 con 0x7f913c0b39e0 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9142800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c1f6320 con 0x7f913c1f5960 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9142800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0ae1d0 con 0x7f911c0047c0 2024-01-15T18:12:38.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 0x7f913c0b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 0x7f913c0b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52222/0 (socket says 172.21.15.2:52222) 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 -- 172.21.15.2:0/1902892778 learned_addr learned my addr 172.21.15.2:0/1902892778 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9141400640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f911c0047c0 0x7f913c1f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 -- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f911c0047c0 msgr2=0x7f913c1f2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f911c0047c0 0x7f913c1f2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 -- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c1f5960 msgr2=0x7f913c1f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c1f5960 0x7f913c1f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.973+0000 7f9140a00640 1 -- 172.21.15.2:0/1902892778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c157350 con 0x7f913c0b39e0 2024-01-15T18:12:38.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.974+0000 7f9140a00640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 0x7f913c0b2680 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f912402c480 tx=0x7f9124001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.974+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912403c040 con 0x7f913c0b39e0 2024-01-15T18:12:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.974+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912402cc20 con 0x7f913c0b39e0 2024-01-15T18:12:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.974+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9124007070 con 0x7f913c0b39e0 2024-01-15T18:12:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.974+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c157630 con 0x7f913c0b39e0 2024-01-15T18:12:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.974+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c1f3450 con 0x7f913c0b39e0 2024-01-15T18:12:38.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.975+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c157e30 con 0x7f913c0b39e0 2024-01-15T18:12:38.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.976+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f9124032070 con 0x7f913c0b39e0 2024-01-15T18:12:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.977+0000 7f9139600640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406c1f0 0x7f911406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.977+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f91240a9a80 con 0x7f913c0b39e0 2024-01-15T18:12:38.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.978+0000 7f9141400640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406c1f0 0x7f911406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:38.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.979+0000 7f9141400640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406c1f0 0x7f911406e6b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f912c009950 tx=0x7f912c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:38.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:38.982+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f913c157e30 con 0x7f913c0b39e0 2024-01-15T18:12:39.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.213+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f913c157e30 con 0x7f913c0b39e0 2024-01-15T18:12:39.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.214+0000 7f9139600640 1 -- 172.21.15.2:0/1902892778 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+4909 (secure 0 0 0) 0x7f913c157e30 con 0x7f913c0b39e0 2024-01-15T18:12:39.218 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:12:39.218 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":65,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3406786991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3406786991},{"type":"v1","addr":"172.21.15.2:6835","nonce":3406786991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/739459094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":739459094},{"type":"v1","addr":"172.21.15.137:6836","nonce":739459094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8565,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/2674783053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2674783053},{"type":"v1","addr":"172.21.15.2:6834","nonce":2674783053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:12:32.613003+0000","modified":"2024-01-15T18:12:35.685814+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8580":{"gid":8580,"name":"d","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6837/320323477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":320323477},{"type":"v1","addr":"172.21.15.137:6837","nonce":320323477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-15T18:12:39.218 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 65 2024-01-15T18:12:39.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406c1f0 msgr2=0x7f911406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406c1f0 0x7f911406e6b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f912c009950 tx=0x7f912c008040 comp rx=0 tx=0).stop 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 msgr2=0x7f913c0b2680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 0x7f913c0b2680 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f912402c480 tx=0x7f9124001930 comp rx=0 tx=0).stop 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 shutdown_connections 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f911406c1f0 0x7f911406e6b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f913c1f5960 0x7f913c1f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f913c0b39e0 0x7f913c0b2680 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 --2- 172.21.15.2:0/1902892778 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f911c0047c0 0x7f913c1f2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 >> 172.21.15.2:0/1902892778 conn(0x7f913c0b0840 msgr2=0x7f913c0afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 shutdown_connections 2024-01-15T18:12:39.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.217+0000 7f9142800640 1 -- 172.21.15.2:0/1902892778 wait complete. 2024-01-15T18:12:39.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 65, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:12:32.613003+0000', 'modified': '2024-01-15T18:12:35.685814+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8580}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8580': {'gid': 8580, 'name': 'd', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.137:6837/320323477', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.137:6835', 'nonce': 320323477}, {'type': 'v1', 'addr': '172.21.15.137:6837', 'nonce': 320323477}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:12:39.642 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:12:39.642 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-15T18:12:39.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.868+0000 7f645e400640 1 Processor -- start 2024-01-15T18:12:39.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.869+0000 7f645e400640 1 -- start start 2024-01-15T18:12:39.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f645e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 0x7f64580cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f645e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f64580d0ed0 0x7f64580d1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f645e400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64580b30e0 con 0x7f64580d2de0 2024-01-15T18:12:39.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f645e400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64580ce180 con 0x7f64580d0ed0 2024-01-15T18:12:39.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f645e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64580d1ca0 con 0x7f645815d200 2024-01-15T18:12:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f6457e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f64580d2de0 0x7f64580d3240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60596/0 (socket says 172.21.15.2:60596) 2024-01-15T18:12:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.870+0000 7f6457e00640 1 -- 172.21.15.2:0/2304513521 learned_addr learned my addr 172.21.15.2:0/2304513521 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f645d000640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 0x7f64580cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f645da00640 1 --2- 172.21.15.2:0/2304513521 >> v2:172.21.15.137:3300/0 conn(0x7f64580d0ed0 0x7f64580d1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301175178 0 0) 0x7f64580b30e0 con 0x7f64580d2de0 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438003520 con 0x7f64580d2de0 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4155954001 0 0) 0x7f6438003520 con 0x7f64580d2de0 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 msgr2=0x7f64580cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 0x7f64580cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 >> v2:172.21.15.137:3300/0 conn(0x7f64580d0ed0 msgr2=0x7f64580d1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 --2- 172.21.15.2:0/2304513521 >> v2:172.21.15.137:3300/0 conn(0x7f64580d0ed0 0x7f64580d1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64580d1f80 con 0x7f64580d2de0 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6440001670 con 0x7f64580d2de0 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.871+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 >> v1:172.21.15.2:6789/0 conn(0x7f64580d2de0 legacy=0x7f64580d3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457400640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f6438003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457400640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64380047c0 0x7f6438004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457400640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 0x7f6438005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64580ce180 con 0x7f6438003970 2024-01-15T18:12:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64580d1ca0 con 0x7f64380047c0 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f643800b980 con 0x7f645815d200 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645d000640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f6438003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645da00640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 0x7f6438005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f6457e00640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64380047c0 0x7f6438004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645d000640 1 -- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 msgr2=0x7f6438005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645d000640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f645815d200 0x7f6438005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645d000640 1 -- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64380047c0 msgr2=0x7f6438004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645d000640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64380047c0 0x7f6438004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.872+0000 7f645d000640 1 -- 172.21.15.2:0/2304513521 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6438003520 con 0x7f6438003970 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f645d000640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f6438003d90 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f6448009ac0 tx=0x7f64380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d1ee7923b9e76b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644802c6b0 con 0x7f6438003970 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644802c810 con 0x7f6438003970 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f6457400640 1 -- 172.21.15.2:0/2304513521 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644802cba0 con 0x7f6438003970 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f645e400640 1 -- 172.21.15.2:0/2304513521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 msgr2=0x7f6438003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f645e400640 1 --2- 172.21.15.2:0/2304513521 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f6438003d90 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f6448009ac0 tx=0x7f64380057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.874+0000 7f645da00640 1 -- 172.21.15.2:0/2304513521 reap_dead start 2024-01-15T18:12:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.875+0000 7f645e400640 1 -- 172.21.15.2:0/2304513521 shutdown_connections 2024-01-15T18:12:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.875+0000 7f645e400640 1 -- 172.21.15.2:0/2304513521 >> 172.21.15.2:0/2304513521 conn(0x7f645808caf0 msgr2=0x7f645809c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.875+0000 7f645e400640 1 -- 172.21.15.2:0/2304513521 shutdown_connections 2024-01-15T18:12:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.875+0000 7f645e400640 1 -- 172.21.15.2:0/2304513521 wait complete. 2024-01-15T18:12:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.877+0000 7f645e400640 1 Processor -- start 2024-01-15T18:12:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.877+0000 7f645e400640 1 -- start start 2024-01-15T18:12:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645e400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f645815e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645e400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64580ccd10 0x7f64580cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645e400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64580b1260 0x7f6458151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645e400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64580cb9d0 con 0x7f6438003970 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645e400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64580cbc70 con 0x7f64580b1260 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645e400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64580cbf50 con 0x7f64580ccd10 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f645815e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f645815e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52240/0 (socket says 172.21.15.2:52240) 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645d000640 1 -- 172.21.15.2:0/1510582301 learned_addr learned my addr 172.21.15.2:0/1510582301 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f6457e00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64580ccd10 0x7f64580cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.878+0000 7f645da00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64580b1260 0x7f6458151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645d000640 1 -- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64580ccd10 msgr2=0x7f64580cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645d000640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64580ccd10 0x7f64580cb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645d000640 1 -- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64580b1260 msgr2=0x7f6458151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645d000640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64580b1260 0x7f6458151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645d000640 1 -- 172.21.15.2:0/1510582301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64580cc230 con 0x7f6438003970 2024-01-15T18:12:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f6457e00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64580ccd10 0x7f64580cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645da00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64580b1260 0x7f6458151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645d000640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f645815e9d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6448000c00 tx=0x7f6448004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f6454c00640 1 -- 172.21.15.2:0/1510582301 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448003810 con 0x7f6438003970 2024-01-15T18:12:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f6454c00640 1 -- 172.21.15.2:0/1510582301 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6448003970 con 0x7f6438003970 2024-01-15T18:12:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64580cc510 con 0x7f6438003970 2024-01-15T18:12:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f6454c00640 1 -- 172.21.15.2:0/1510582301 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448003d20 con 0x7f6438003970 2024-01-15T18:12:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.879+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64580b16c0 con 0x7f6438003970 2024-01-15T18:12:39.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.882+0000 7f6454c00640 1 -- 172.21.15.2:0/1510582301 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f6448034440 con 0x7f6438003970 2024-01-15T18:12:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.882+0000 7f6454c00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f643006c220 0x7f643006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.883+0000 7f6454c00640 1 -- 172.21.15.2:0/1510582301 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f64480aa9e0 con 0x7f6438003970 2024-01-15T18:12:39.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.883+0000 7f6457e00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f643006c220 0x7f643006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.883+0000 7f6462766740 1 Processor -- start 2024-01-15T18:12:39.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.883+0000 7f6462766740 1 -- start start 2024-01-15T18:12:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6d070 0x558e03b6d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6462766740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 0x558e03b6c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bb0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6462766740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03acb880 con 0x558e03b67d70 2024-01-15T18:12:39.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03adaae0 con 0x558e03b6c7b0 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e03ad81d0 con 0x558e03b6d070 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.884+0000 7f6457e00640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f643006c220 0x7f643006e6e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6440000fb0 tx=0x7f6440001bb0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 0x558e03b6c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 0x558e03b6c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52246/0 (socket says 172.21.15.2:52246) 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 -- 172.21.15.2:0/962085836 learned_addr learned my addr 172.21.15.2:0/962085836 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f645d000640 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6d070 0x558e03b6d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f645da00640 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bb0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 -- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6d070 msgr2=0x558e03b6d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6d070 0x558e03b6d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 -- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 msgr2=0x558e03bb0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bb0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 -- 172.21.15.2:0/962085836 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e03a9b860 con 0x558e03b67d70 2024-01-15T18:12:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.885+0000 7f6457e00640 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 0x558e03b6c270 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f644000f910 tx=0x7f6440012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d15f8d686cc779 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f644d600640 1 -- 172.21.15.2:0/962085836 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64400162d0 con 0x558e03b67d70 2024-01-15T18:12:39.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f644d600640 1 -- 172.21.15.2:0/962085836 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644000fe90 con 0x558e03b67d70 2024-01-15T18:12:39.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f644d600640 1 -- 172.21.15.2:0/962085836 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440016bd0 con 0x558e03b67d70 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 -- 172.21.15.2:0/962085836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 msgr2=0x558e03b6c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 0x558e03b6c270 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f644000f910 tx=0x7f6440012000 comp rx=0 tx=0).stop 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 -- 172.21.15.2:0/962085836 shutdown_connections 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bb0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b67d70 0x558e03b6c270 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 --2- 172.21.15.2:0/962085836 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6d070 0x558e03b6d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 -- 172.21.15.2:0/962085836 >> 172.21.15.2:0/962085836 conn(0x558e03b2ec70 msgr2=0x558e03bb2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.886+0000 7f6462766740 1 -- 172.21.15.2:0/962085836 shutdown_connections 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.887+0000 7f6462766740 1 -- 172.21.15.2:0/962085836 wait complete. 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.887+0000 7f6462766740 1 -- start start 2024-01-15T18:12:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.887+0000 7f6462766740 1 Processor -- start 2024-01-15T18:12:39.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.888+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03bba400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.888+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bba940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.888+0000 7f6462766740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03ba9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.888+0000 7f6462766740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03ad3ac0 con 0x558e03b6d070 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.888+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03a702f0 con 0x558e03b6c7b0 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.888+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e03a8dc70 con 0x558e03b67d70 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f6462766740 10 client.0 _ll_register_callbacks cb 0x558e03a923b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03ba9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03ba9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52248/0 (socket says 172.21.15.2:52248) 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 -- 172.21.15.2:0/3390548661 learned_addr learned my addr 172.21.15.2:0/3390548661 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645d000640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03bba400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f6457e00640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bba940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 -- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 msgr2=0x558e03bba400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03bba400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 -- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 msgr2=0x558e03bba940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bba940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 -- 172.21.15.2:0/3390548661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64580d1f80 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645d000640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03bba400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f6457e00640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bba940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f645da00640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03ba9e30 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f645806afb0 tx=0x7f6450012610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f585f6a485f7617 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.889+0000 7f6446000640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6446000640 1 -- 172.21.15.2:0/3390548661 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000e040 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6446000640 1 -- 172.21.15.2:0/3390548661 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f6450017440 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e03accaa0 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6446000640 1 -- 172.21.15.2:0/3390548661 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6450012ea0 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 10 client.8715 fetch_fsmap learned FSMap version 65 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) v3 -- 0x558e03ad2140 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6446000640 1 -- 172.21.15.2:0/3390548661 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558e03accaa0 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6446000640 1 -- 172.21.15.2:0/3390548661 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 65) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6450003990 con 0x558e03b6d070 2024-01-15T18:12:39.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 10 client.8715 fetch_fsmap finished waiting for FSMap version 65 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 10 client.8715 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 20 client.8715 populate_metadata read hostname 'smithi002' 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bcde70 0x558e03bd0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 4 client.8715 mds_command: new command op to 8580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.890+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 --> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e03acc7c0 con 0x558e03bcde70 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.891+0000 7f645d000640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bcde70 0x558e03bd0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.892+0000 7f645d000640 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bcde70 0x558e03bd0260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6aba10a790503ca3 server_cookie=cc26bfd3d760512e in_seq=0 out_seq=0 2024-01-15T18:12:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.892+0000 7f6446000640 10 client.8715 ms_handle_connect on v2:172.21.15.137:6835/320323477 2024-01-15T18:12:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6446000640 1 -- 172.21.15.2:0/3390548661 <== mds.0 v2:172.21.15.137:6835/320323477 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558e03acc7c0 con 0x558e03bcde70 2024-01-15T18:12:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6446000640 10 client.8715 handle_command_reply: tid=0 2024-01-15T18:12:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 client.8715 shutdown 2024-01-15T18:12:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 msgr2=0x558e03ba9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03ba9e30 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f645806afb0 tx=0x7f6450012610 comp rx=0 tx=0).stop 2024-01-15T18:12:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 shutdown_connections 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bcde70 0x558e03bd0260 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03ba9e30 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6c7b0 0x558e03bba940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 --2- 172.21.15.2:0/3390548661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03bba400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6446000640 0 client.8715 ms_handle_reset on v2:172.21.15.137:6835/320323477 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 >> 172.21.15.2:0/3390548661 conn(0x558e03b2ec70 msgr2=0x558e03b66dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 shutdown_connections 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.894+0000 7f6462766740 1 -- 172.21.15.2:0/3390548661 wait complete. 2024-01-15T18:12:39.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.895+0000 7f6462766740 20 client.8715 trim_cache size 0 max 16384 2024-01-15T18:12:39.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.912+0000 7f6462766740 1 Processor -- start 2024-01-15T18:12:39.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.912+0000 7f6462766740 1 -- start start 2024-01-15T18:12:39.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03ba9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6462766740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 0x558e03b6d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6d070 0x558e03b94850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6462766740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03836f20 con 0x558e03b6c7b0 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03ba9c50 con 0x558e03b6d070 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e03b6e260 con 0x558e03b67d70 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6457e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 0x558e03b6d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6457e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 0x558e03b6d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52250/0 (socket says 172.21.15.2:52250) 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.913+0000 7f6457e00640 1 -- 172.21.15.2:0/838425294 learned_addr learned my addr 172.21.15.2:0/838425294 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f6457e00640 1 -- 172.21.15.2:0/838425294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 msgr2=0x558e03ba9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f6457e00640 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03ba9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f6457e00640 1 -- 172.21.15.2:0/838425294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6d070 msgr2=0x558e03b94850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f6457e00640 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6d070 0x558e03b94850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f6457e00640 1 -- 172.21.15.2:0/838425294 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e03bacf10 con 0x558e03b6c7b0 2024-01-15T18:12:39.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f6457e00640 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 0x558e03b6d450 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f6440016820 tx=0x7f6440013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e756e256556d81 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f644e000640 1 -- 172.21.15.2:0/838425294 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64400162d0 con 0x558e03b6c7b0 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f644e000640 1 -- 172.21.15.2:0/838425294 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64400169d0 con 0x558e03b6c7b0 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.914+0000 7f644e000640 1 -- 172.21.15.2:0/838425294 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440016d80 con 0x558e03b6c7b0 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 -- 172.21.15.2:0/838425294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 msgr2=0x558e03b6d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 0x558e03b6d450 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f6440016820 tx=0x7f6440013000 comp rx=0 tx=0).stop 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 -- 172.21.15.2:0/838425294 shutdown_connections 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b6d070 0x558e03b94850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6c7b0 0x558e03b6d450 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 --2- 172.21.15.2:0/838425294 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b67d70 0x558e03ba9710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 -- 172.21.15.2:0/838425294 >> 172.21.15.2:0/838425294 conn(0x558e03b2ec70 msgr2=0x558e03bb7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 -- 172.21.15.2:0/838425294 shutdown_connections 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.915+0000 7f6462766740 1 -- 172.21.15.2:0/838425294 wait complete. 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.916+0000 7f6462766740 1 -- start start 2024-01-15T18:12:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.916+0000 7f6462766740 1 Processor -- start 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b67d70 0x558e03b93cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6c7b0 0x558e03c207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03bd4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03c20390 con 0x558e03b6d070 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e03b94210 con 0x558e03b67d70 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e03c20080 con 0x558e03b6c7b0 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f6462766740 10 client.0 _ll_register_callbacks cb 0x7f6450009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f645da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03bd4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f645da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03bd4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52258/0 (socket says 172.21.15.2:52258) 2024-01-15T18:12:39.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.917+0000 7f645da00640 1 -- 172.21.15.2:0/2453541932 learned_addr learned my addr 172.21.15.2:0/2453541932 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645da00640 1 -- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6c7b0 msgr2=0x558e03c207b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645d000640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b67d70 0x558e03b93cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f6457e00640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6c7b0 0x558e03c207b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645da00640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6c7b0 0x558e03c207b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645da00640 1 -- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b67d70 msgr2=0x558e03b93cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645da00640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b67d70 0x558e03b93cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645da00640 1 -- 172.21.15.2:0/2453541932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e03a9b860 con 0x558e03b6d070 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f645da00640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03bd4280 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f64500146e0 tx=0x7f6450004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5707e5d032e37fa4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f6446000640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f6446000640 1 -- 172.21.15.2:0/2453541932 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645000e070 con 0x558e03b6d070 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f6446000640 1 -- 172.21.15.2:0/2453541932 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f6450004210 con 0x558e03b6d070 2024-01-15T18:12:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.918+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f644000fe90 con 0x558e03b6d070 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6446000640 1 -- 172.21.15.2:0/2453541932 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=65) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6450004430 con 0x558e03b6d070 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 10 client.8721 fetch_fsmap learned FSMap version 65 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=65,monmap=2+}) v3 -- 0x558e03a6fb80 con 0x558e03b6d070 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6446000640 1 -- 172.21.15.2:0/2453541932 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644000fe90 con 0x558e03b6d070 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6446000640 1 -- 172.21.15.2:0/2453541932 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 65) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6450004690 con 0x558e03b6d070 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 10 client.8721 fetch_fsmap finished waiting for FSMap version 65 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 10 client.8721 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 20 client.8721 populate_metadata read hostname 'smithi002' 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bd47c0 0x558e03bd4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 4 client.8721 mds_command: new command op to 8580 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:12:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.919+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 --> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558e037d46c0 con 0x558e03bd47c0 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.920+0000 7f645d000640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bd47c0 0x558e03bd4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.920+0000 7f645d000640 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bd47c0 0x558e03bd4ba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d693b6909d2223dc server_cookie=341f6de8265435d3 in_seq=0 out_seq=0 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.920+0000 7f6446000640 10 client.8721 ms_handle_connect on v2:172.21.15.137:6835/320323477 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6446000640 1 -- 172.21.15.2:0/2453541932 <== mds.0 v2:172.21.15.137:6835/320323477 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x558e037d46c0 con 0x558e03bd47c0 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6446000640 10 client.8721 handle_command_reply: tid=0 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 client.8721 shutdown 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 msgr2=0x558e03bd4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03bd4280 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f64500146e0 tx=0x7f6450004a40 comp rx=0 tx=0).stop 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 shutdown_connections 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:6835/320323477,v1:172.21.15.137:6837/320323477] conn(0x558e03bd47c0 0x558e03bd4ba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558e03b6d070 0x558e03bd4280 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558e03b6c7b0 0x558e03c207b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 --2- 172.21.15.2:0/2453541932 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558e03b67d70 0x558e03b93cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 >> 172.21.15.2:0/2453541932 conn(0x558e03b2ec70 msgr2=0x558e03bb7500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6446000640 0 client.8721 ms_handle_reset on v2:172.21.15.137:6835/320323477 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.921+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 shutdown_connections 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f6462766740 1 -- 172.21.15.2:0/2453541932 wait complete. 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f6462766740 20 client.8721 trim_cache size 0 max 16384 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T18:12:39.929 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 8580, 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 6.7653030279999999, 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 65, 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 103, 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 103, 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 13.132418218 2024-01-15T18:12:39.930 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f643006c220 msgr2=0x7f643006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f643006c220 0x7f643006e6e0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f6440000fb0 tx=0x7f6440001bb0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 msgr2=0x7f645815e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f645815e9d0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f6448000c00 tx=0x7f6448004730 comp rx=0 tx=0).stop 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 shutdown_connections 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f643006c220 0x7f643006e6e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f64580b1260 0x7f6458151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f64580ccd10 0x7f64580cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 --2- 172.21.15.2:0/1510582301 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6438003970 0x7f645815e9d0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.922+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 >> 172.21.15.2:0/1510582301 conn(0x7f645808caf0 msgr2=0x7f64580b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.923+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 shutdown_connections 2024-01-15T18:12:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:39.923+0000 7f645e400640 1 -- 172.21.15.2:0/1510582301 wait complete. 2024-01-15T18:12:39.943 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:12:40.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.148+0000 7f7e12a00640 1 Processor -- start 2024-01-15T18:12:40.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.148+0000 7f7e12a00640 1 -- start start 2024-01-15T18:12:40.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.149+0000 7f7e12a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.149+0000 7f7e12a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7e0c15ec60 0x7f7e0c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.149+0000 7f7e12a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7e0c0cd490 0x7f7e0c0cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.149+0000 7f7e12a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0c071f20 con 0x7f7e0c0cd490 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.149+0000 7f7e12a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0c06c6b0 con 0x7f7e0c15ec60 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.149+0000 7f7e12a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c15f620 con 0x7f7e0c141880 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7e0c0cd490 0x7f7e0c0cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7e0c0cd490 0x7f7e0c0cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52268/0 (socket says 172.21.15.2:52268) 2024-01-15T18:12:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 -- 172.21.15.2:0/2120247364 learned_addr learned my addr 172.21.15.2:0/2120247364 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e10c00640 1 --2- 172.21.15.2:0/2120247364 >> v2:172.21.15.137:3300/0 conn(0x7f7e0c15ec60 0x7f7e0c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e11600640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 -- 172.21.15.2:0/2120247364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 msgr2=0x7f7e0c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 -- 172.21.15.2:0/2120247364 >> v2:172.21.15.137:3300/0 conn(0x7f7e0c15ec60 msgr2=0x7f7e0c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 --2- 172.21.15.2:0/2120247364 >> v2:172.21.15.137:3300/0 conn(0x7f7e0c15ec60 0x7f7e0c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e12000640 1 -- 172.21.15.2:0/2120247364 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c071960 con 0x7f7e0c0cd490 2024-01-15T18:12:40.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.150+0000 7f7e11600640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.152+0000 7f7e12000640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c0cd920 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c06e5e0 tx=0x7f7e0402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2160ed624152ac server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.152+0000 7f7e0be00640 1 -- 172.21.15.2:0/2120247364 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0402c9f0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.152+0000 7f7e0be00640 1 -- 172.21.15.2:0/2120247364 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0402cb50 con 0x7f7e0c0cd490 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.152+0000 7f7e0be00640 1 -- 172.21.15.2:0/2120247364 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e040343f0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 -- 172.21.15.2:0/2120247364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 msgr2=0x7f7e0c0cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c0cd920 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c06e5e0 tx=0x7f7e0402be50 comp rx=0 tx=0).stop 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 -- 172.21.15.2:0/2120247364 shutdown_connections 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c0cd920 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2120247364 >> v2:172.21.15.137:3300/0 conn(0x7f7e0c15ec60 0x7f7e0c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2120247364 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 -- 172.21.15.2:0/2120247364 >> 172.21.15.2:0/2120247364 conn(0x7f7e0c0b0840 msgr2=0x7f7e0c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 -- 172.21.15.2:0/2120247364 shutdown_connections 2024-01-15T18:12:40.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.153+0000 7f7e12a00640 1 -- 172.21.15.2:0/2120247364 wait complete. 2024-01-15T18:12:40.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.155+0000 7f7e12a00640 1 Processor -- start 2024-01-15T18:12:40.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.155+0000 7f7e12a00640 1 -- start start 2024-01-15T18:12:40.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e12a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c13e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e12a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c15d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e12a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e0c15ec60 0x7f7e0c15d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e12a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c143d00 con 0x7f7e0c0cd490 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e12a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c1430f0 con 0x7f7e0c15ec60 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e12a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c1433d0 con 0x7f7e0c141880 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e11600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c13e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e11600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c13e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52272/0 (socket says 172.21.15.2:52272) 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.156+0000 7f7e11600640 1 -- 172.21.15.2:0/2259875804 learned_addr learned my addr 172.21.15.2:0/2259875804 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e10c00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c15d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e12000640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e0c15ec60 0x7f7e0c15d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e11600640 1 -- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 msgr2=0x7f7e0c15d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e11600640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c15d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e11600640 1 -- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e0c15ec60 msgr2=0x7f7e0c15d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e11600640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e0c15ec60 0x7f7e0c15d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e11600640 1 -- 172.21.15.2:0/2259875804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04009b30 con 0x7f7e0c0cd490 2024-01-15T18:12:40.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e11600640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c13e610 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c7c0 tx=0x7f7dfc00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc007a30 con 0x7f7e0c0cd490 2024-01-15T18:12:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dfc007b90 con 0x7f7e0c0cd490 2024-01-15T18:12:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dfc00ce70 con 0x7f7e0c0cd490 2024-01-15T18:12:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.157+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0c1436b0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.158+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c13eb50 con 0x7f7e0c0cd490 2024-01-15T18:12:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.159+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0c074770 con 0x7f7e0c0cd490 2024-01-15T18:12:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.160+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f7dfc0113f0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.161+0000 7f7e09600640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7de406c1f0 0x7f7de406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.161+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f7dfc0887e0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.165 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.161+0000 7f7e10c00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7de406c1f0 0x7f7de406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:40.166 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.163+0000 7f7e10c00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7de406c1f0 0x7f7de406e6b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7df4006fd0 tx=0x7f7df4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:40.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.166+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7dfc0550a0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.400+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7e0c0ce690 con 0x7f7e0c0cd490 2024-01-15T18:12:40.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.401+0000 7f7e09600640 1 -- 172.21.15.2:0/2259875804 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v65) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7dfc0586c0 con 0x7f7e0c0cd490 2024-01-15T18:12:40.405 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:12:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7de406c1f0 msgr2=0x7f7de406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7de406c1f0 0x7f7de406e6b0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f7df4006fd0 tx=0x7f7df4008040 comp rx=0 tx=0).stop 2024-01-15T18:12:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 msgr2=0x7f7e0c13e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c13e610 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc00c7c0 tx=0x7f7dfc00cc80 comp rx=0 tx=0).stop 2024-01-15T18:12:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 shutdown_connections 2024-01-15T18:12:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7de406c1f0 0x7f7de406e6b0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7e0c15ec60 0x7f7e0c15d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7e0c141880 0x7f7e0c15d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 --2- 172.21.15.2:0/2259875804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7e0c0cd490 0x7f7e0c13e610 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 >> 172.21.15.2:0/2259875804 conn(0x7f7e0c0b0840 msgr2=0x7f7e0c0ceb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 shutdown_connections 2024-01-15T18:12:40.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:40.404+0000 7f7e12a00640 1 -- 172.21.15.2:0/2259875804 wait complete. 2024-01-15T18:12:40.427 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:12:40.427 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:12:40.428 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:12:40.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.428 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:12:40.481 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41722sec preferred_lft 41722sec 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:12:40.482 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T18:12:40.483 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:40.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.484 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:12:40.576 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T18:12:40.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.581 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T18:12:40.635 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T18:12:40.636 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:12:40.636 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T18:12:40.636 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T18:12:40.636 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:12:40.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.637 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T18:12:40.637 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T18:12:40.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:12:40.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:12:40.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:12:40.637 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T18:12:40.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.834 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:12:40.890 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:12:40.890 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:12:40.891 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:12:40.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:40.891 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T18:12:40.991 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:12:40.991 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:40.991 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:41.047 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:12:41.047 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:41.047 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:41.103 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:12:41.205 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:41.266 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:12:41.267 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:12:41.268 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:41.269 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:41.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:41.271 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:12:41.325 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:12:41.325 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:12:41.331 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:12:41.410 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:41.454 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:12:41.454 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:12:41.454 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:12:41.454 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:12:41.455 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:12:41.456 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:41.457 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:12:41.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:41.458 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:12:41.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T18:12:41.517+0000 7f6a20183480 -1 init, newargv = 0x55bdcaf170e0 newargc=15 2024-01-15T18:12:41.521 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[72848]: starting ceph client 2024-01-15T18:12:41.532 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[72848]: starting fuse 2024-01-15T18:12:41.544 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T18:12:41.545 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T18:12:41.545 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:12:41.545 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T18:12:41.545 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T18:12:41.545 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T18:12:41.545 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T18:12:41.545 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:12:41.546 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T18:12:41.547 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T18:12:41.746 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.72848.asok 2024-01-15T18:12:41.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:12:41.764 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72848.asok 2024-01-15T18:12:41.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:41.764 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72848.asok status 2024-01-15T18:12:42.014 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:12:42.014 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "72848", 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T18:12:42.015 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 8733, 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 8733 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:42.016 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1576460582 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 1576460582 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.8733 192.168.0.1:0/1576460582", 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/1576460582", 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T18:12:42.017 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 65, 2024-01-15T18:12:42.018 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 103, 2024-01-15T18:12:42.018 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:12:42.018 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T18:12:42.018 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T18:12:42.018 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:12:42.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:42.031 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T18:12:42.032 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:42.090 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T18:12:42.091 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:42.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:42.091 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:12:42.190 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:12:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.405+0000 7fb345e00640 1 Processor -- start 2024-01-15T18:12:42.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.405+0000 7fb345e00640 1 -- start start 2024-01-15T18:12:42.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb345e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb34015ec60 0x7fb34015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb345e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 0x7fb34006cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb345e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3400716d0 con 0x7fb340141880 2024-01-15T18:12:42.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb345e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb340071f20 con 0x7fb34015ec60 2024-01-15T18:12:42.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb345e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34006c6b0 con 0x7fb34006cad0 2024-01-15T18:12:42.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb344a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb340141880 0x7fb340070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60608/0 (socket says 172.21.15.2:60608) 2024-01-15T18:12:42.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb344a00640 1 -- 172.21.15.2:0/2577598093 learned_addr learned my addr 172.21.15.2:0/2577598093 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:42.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb345400640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 0x7fb34006cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:42.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33fe00640 1 --2- 172.21.15.2:0/2577598093 >> v2:172.21.15.137:3300/0 conn(0x7fb34015ec60 0x7fb34015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687406830 0 0) 0x7fb3400716d0 con 0x7fb340141880 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb320003520 con 0x7fb340141880 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 305683161 0 0) 0x7fb320003520 con 0x7fb340141880 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 msgr2=0x7fb34006cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 0x7fb34006cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 >> v2:172.21.15.137:3300/0 conn(0x7fb34015ec60 msgr2=0x7fb34015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 --2- 172.21.15.2:0/2577598093 >> v2:172.21.15.137:3300/0 conn(0x7fb34015ec60 0x7fb34015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb340071960 con 0x7fb340141880 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb330001670 con 0x7fb340141880 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 >> v1:172.21.15.2:6789/0 conn(0x7fb340141880 legacy=0x7fb340070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:42.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb320003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 0x7fb3200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb320004d00 0x7fb320005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb340071f20 con 0x7fb3200039a0 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34006c6b0 con 0x7fb320004d00 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.407+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32000b9b0 con 0x7fb34006cad0 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb344a00640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb320003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb33fe00640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 0x7fb3200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb344a00640 1 -- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 msgr2=0x7fb3200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb344a00640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb34006cad0 0x7fb3200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb344a00640 1 -- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb320004d00 msgr2=0x7fb320005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:42.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb344a00640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb320004d00 0x7fb320005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.408+0000 7fb344a00640 1 -- 172.21.15.2:0/2577598093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320003520 con 0x7fb3200039a0 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.410+0000 7fb344a00640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb320003dc0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c240 tx=0x7fb3200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9958b808bb204e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.411+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330005ad0 con 0x7fb3200039a0 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.411+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb330005c30 con 0x7fb3200039a0 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.411+0000 7fb33f400640 1 -- 172.21.15.2:0/2577598093 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33002caf0 con 0x7fb3200039a0 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.411+0000 7fb345e00640 1 -- 172.21.15.2:0/2577598093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 msgr2=0x7fb320003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.411+0000 7fb345e00640 1 --2- 172.21.15.2:0/2577598093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb320003dc0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c240 tx=0x7fb3200057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.411+0000 7fb345400640 1 -- 172.21.15.2:0/2577598093 reap_dead start 2024-01-15T18:12:42.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.412+0000 7fb345e00640 1 -- 172.21.15.2:0/2577598093 shutdown_connections 2024-01-15T18:12:42.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.412+0000 7fb345e00640 1 -- 172.21.15.2:0/2577598093 >> 172.21.15.2:0/2577598093 conn(0x7fb3400b0840 msgr2=0x7fb3400b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:42.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.412+0000 7fb345e00640 1 -- 172.21.15.2:0/2577598093 shutdown_connections 2024-01-15T18:12:42.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.412+0000 7fb345e00640 1 -- 172.21.15.2:0/2577598093 wait complete. 2024-01-15T18:12:42.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.414+0000 7fb345e00640 1 Processor -- start 2024-01-15T18:12:42.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.414+0000 7fb345e00640 1 -- start start 2024-01-15T18:12:42.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb34013ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3400c80b0 0x7fb3400c8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3400b1e70 0x7fb3400b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb340152850 con 0x7fb3200039a0 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3400c8a70 con 0x7fb3400b1e70 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3400b2ba0 con 0x7fb3400c80b0 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb34013ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb34013ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52292/0 (socket says 172.21.15.2:52292) 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 -- 172.21.15.2:0/2708687628 learned_addr learned my addr 172.21.15.2:0/2708687628 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 -- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3400c80b0 msgr2=0x7fb3400c8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3400c80b0 0x7fb3400c8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 -- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3400b1e70 msgr2=0x7fb3400b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:42.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3400b1e70 0x7fb3400b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 -- 172.21.15.2:0/2708687628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3400b2e80 con 0x7fb3200039a0 2024-01-15T18:12:42.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb344a00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb34013ffd0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c240 tx=0x7fb330001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330044070 con 0x7fb3200039a0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33002cac0 con 0x7fb3200039a0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33002cc70 con 0x7fb3200039a0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3400b3160 con 0x7fb3200039a0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.415+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb340140510 con 0x7fb3200039a0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.416+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb340074770 con 0x7fb3200039a0 2024-01-15T18:12:42.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.417+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fb330032070 con 0x7fb3200039a0 2024-01-15T18:12:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.418+0000 7fb33cc00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb31806c220 0x7fb31806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.418+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fb3300a9720 con 0x7fb3200039a0 2024-01-15T18:12:42.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.419+0000 7fb33fe00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb31806c220 0x7fb31806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:42.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.420+0000 7fb33fe00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb31806c220 0x7fb31806e6e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb3400b3e20 tx=0x7fb328008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:42.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.422+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3300760a0 con 0x7fb3200039a0 2024-01-15T18:12:42.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.655+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb34006e1c0 con 0x7fb3200039a0 2024-01-15T18:12:42.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.655+0000 7fb33cc00640 1 -- 172.21.15.2:0/2708687628 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v65) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb3300796c0 con 0x7fb3200039a0 2024-01-15T18:12:42.659 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:12:42.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb31806c220 msgr2=0x7fb31806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:42.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb31806c220 0x7fb31806e6e0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb3400b3e20 tx=0x7fb328008040 comp rx=0 tx=0).stop 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 msgr2=0x7fb34013ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb34013ffd0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c240 tx=0x7fb330001a00 comp rx=0 tx=0).stop 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 shutdown_connections 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb31806c220 0x7fb31806e6e0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3400b1e70 0x7fb3400b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb3400c80b0 0x7fb3400c8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 --2- 172.21.15.2:0/2708687628 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3200039a0 0x7fb34013ffd0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 >> 172.21.15.2:0/2708687628 conn(0x7fb3400b0840 msgr2=0x7fb3400af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 shutdown_connections 2024-01-15T18:12:42.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:42.658+0000 7fb345e00640 1 -- 172.21.15.2:0/2708687628 wait complete. 2024-01-15T18:12:42.681 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:12:42.681 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:12:42.681 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T18:12:42.681 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:12:42.681 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:42.681 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:12:42.682 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:12:42.682 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:12:42.682 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:12:42.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:42.682 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T18:12:42.739 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:12:42.739 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:12:42.739 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:12:42.739 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:12:42.740 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41708sec preferred_lft 41708sec 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:42.741 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:12:42.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:42.742 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:12:42.828 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T18:12:42.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:42.832 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T18:12:42.884 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T18:12:42.884 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:12:42.884 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:12:42.884 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:12:42.884 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:12:42.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:42.885 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T18:12:42.885 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T18:12:42.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:12:42.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:12:42.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:12:42.885 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T18:12:43.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:43.063 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:12:43.113 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:12:43.113 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:12:43.114 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:12:43.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:43.114 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T18:12:43.210 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:12:43.210 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:43.210 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:43.261 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:12:43.262 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:12:43.262 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:43.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:12:43.402 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:12:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:12:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:12:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:12:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:12:43.457 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:12:43.458 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:12:43.459 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:43.460 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:43.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:43.462 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:12:43.515 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:12:43.515 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:12:43.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:12:43.602 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:12:43.648 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:12:43.649 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:12:43.650 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:12:43.651 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:43.651 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:12:43.651 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:43.651 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:12:43.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:43.652 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:12:43.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T18:12:43.706+0000 7f84f2438480 -1 init, newargv = 0x55b13212b090 newargc=15 2024-01-15T18:12:43.710 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[46689]: starting ceph client 2024-01-15T18:12:43.719 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[46689]: starting fuse 2024-01-15T18:12:43.737 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T18:12:43.737 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:12:43.738 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T18:12:43.739 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T18:12:43.740 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:12:43.740 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:12:43.740 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T18:12:43.740 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T18:12:43.934 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.46689.asok 2024-01-15T18:12:43.939 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T18:12:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:12:43.952 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.46689.asok 2024-01-15T18:12:43.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:43.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.46689.asok status 2024-01-15T18:12:44.188 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T18:12:44.188 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "46689", 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T18:12:44.189 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 6398, 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 6398 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:12:44.190 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3291941900 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3291941900 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.6398 192.168.0.1:0/3291941900", 2024-01-15T18:12:44.191 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/3291941900", 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 65, 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 103, 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T18:12:44.192 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T18:12:44.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:44.204 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T18:12:44.204 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:44.261 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T18:12:44.261 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:44.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:44.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:12:44.350 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:12:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.584+0000 7f2c5e000640 1 Processor -- start 2024-01-15T18:12:44.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.584+0000 7f2c5e000640 1 -- start start 2024-01-15T18:12:44.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.586+0000 7f2c5e000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2c58144c00 0x7f2c580cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.586+0000 7f2c5e000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580ce230 con 0x7f2c5806d000 2024-01-15T18:12:44.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.586+0000 7f2c5e000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c580ce510 con 0x7f2c580cdd70 2024-01-15T18:12:44.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.586+0000 7f2c5e000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c580ce7f0 con 0x7f2c58144c00 2024-01-15T18:12:44.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.586+0000 7f2c57e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2c5806d000 0x7f2c58070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52470/0 (socket says 172.21.15.2:52470) 2024-01-15T18:12:44.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.586+0000 7f2c57e00640 1 -- 172.21.15.2:0/3523002869 learned_addr learned my addr 172.21.15.2:0/3523002869 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c57400640 1 --2- 172.21.15.2:0/3523002869 >> v2:172.21.15.137:3300/0 conn(0x7f2c58144c00 0x7f2c580cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890561528 0 0) 0x7f2c580ce230 con 0x7f2c5806d000 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c38003520 con 0x7f2c5806d000 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714566375 0 0) 0x7f2c38003520 con 0x7f2c5806d000 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 >> v2:172.21.15.137:3300/0 conn(0x7f2c58144c00 msgr2=0x7f2c580cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c56000640 1 --2- 172.21.15.2:0/3523002869 >> v2:172.21.15.137:3300/0 conn(0x7f2c58144c00 0x7f2c580cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.587+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 >> v1:172.21.15.137:6789/0 conn(0x7f2c580cdd70 legacy=0x7f2c5806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580705f0 con 0x7f2c5806d000 2024-01-15T18:12:44.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2c500019f0 con 0x7f2c5806d000 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 >> v1:172.21.15.2:6789/0 conn(0x7f2c5806d000 legacy=0x7f2c58070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c380047c0 0x7f2c38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c38005180 0x7f2c3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580ce7f0 con 0x7f2c38003970 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3800c020 con 0x7f2c380047c0 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3800c1e0 con 0x7f2c38005180 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57400640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57400640 1 -- 172.21.15.2:0/3523002869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c38005180 msgr2=0x7f2c3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57400640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c38005180 0x7f2c3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57400640 1 -- 172.21.15.2:0/3523002869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c380047c0 msgr2=0x7f2c38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:44.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57400640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c380047c0 0x7f2c38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57400640 1 -- 172.21.15.2:0/3523002869 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c38003520 con 0x7f2c38003970 2024-01-15T18:12:44.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.588+0000 7f2c57e00640 1 -- 172.21.15.2:0/3523002869 reap_dead start 2024-01-15T18:12:44.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.590+0000 7f2c57400640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c38003d90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009ac0 tx=0x7f2c380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5f9e9a8dce4b75 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:44.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.590+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4802c6d0 con 0x7f2c38003970 2024-01-15T18:12:44.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.590+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4802c830 con 0x7f2c38003970 2024-01-15T18:12:44.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.590+0000 7f2c56000640 1 -- 172.21.15.2:0/3523002869 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c4802cbc0 con 0x7f2c38003970 2024-01-15T18:12:44.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.590+0000 7f2c5e000640 1 -- 172.21.15.2:0/3523002869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 msgr2=0x7f2c38003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.590+0000 7f2c5e000640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c38003d90 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009ac0 tx=0x7f2c380057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.591+0000 7f2c5e000640 1 -- 172.21.15.2:0/3523002869 shutdown_connections 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.591+0000 7f2c5e000640 1 --2- 172.21.15.2:0/3523002869 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c38003d90 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.591+0000 7f2c5e000640 1 -- 172.21.15.2:0/3523002869 >> 172.21.15.2:0/3523002869 conn(0x7f2c580b45f0 msgr2=0x7f2c580b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.591+0000 7f2c5e000640 1 -- 172.21.15.2:0/3523002869 shutdown_connections 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.591+0000 7f2c5e000640 1 -- 172.21.15.2:0/3523002869 wait complete. 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.593+0000 7f2c5e000640 1 Processor -- start 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.593+0000 7f2c5e000640 1 -- start start 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c5e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c580b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c5e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c580b2f00 0x7f2c580b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c5e000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c58158890 0x7f2c58158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c5e000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580b3de0 con 0x7f2c38003970 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c5e000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580ac910 con 0x7f2c58158890 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c5e000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580af710 con 0x7f2c580b2f00 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c57400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c580b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c57400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c580b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45400/0 (socket says 172.21.15.2:45400) 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.594+0000 7f2c57400640 1 -- 172.21.15.2:0/1889632847 learned_addr learned my addr 172.21.15.2:0/1889632847 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c57400640 1 -- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c580b2f00 msgr2=0x7f2c580b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c57400640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c580b2f00 0x7f2c580b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c57400640 1 -- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c58158890 msgr2=0x7f2c58158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:44.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c57400640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c58158890 0x7f2c58158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c57400640 1 -- 172.21.15.2:0/1889632847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580af9f0 con 0x7f2c38003970 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c57400640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c580b04f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f2c4802c270 tx=0x7f2c48004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48003810 con 0x7f2c38003970 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c48003970 con 0x7f2c38003970 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c48003ce0 con 0x7f2c38003970 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c580afcd0 con 0x7f2c38003970 2024-01-15T18:12:44.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.595+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c58159250 con 0x7f2c38003970 2024-01-15T18:12:44.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.596+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c580aca80 con 0x7f2c38003970 2024-01-15T18:12:44.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.597+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f2c48040450 con 0x7f2c38003970 2024-01-15T18:12:44.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.598+0000 7f2c4f400640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c3006c220 0x7f2c3006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:44.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.598+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f2c480aa570 con 0x7f2c38003970 2024-01-15T18:12:44.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.599+0000 7f2c56a00640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c3006c220 0x7f2c3006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:44.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.600+0000 7f2c56a00640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c3006c220 0x7f2c3006e6e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2c580b2a20 tx=0x7f2c400079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:44.606 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.603+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c48076e30 con 0x7f2c38003970 2024-01-15T18:12:44.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:44.836+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2c580686f0 con 0x7f2c38003970 2024-01-15T18:12:45.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.729+0000 7f2c4f400640 1 -- 172.21.15.2:0/1889632847 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v66) v1 ==== 136+0+0 (secure 0 0 0) 0x7f2c480044e0 con 0x7f2c38003970 2024-01-15T18:12:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c3006c220 msgr2=0x7f2c3006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c3006c220 0x7f2c3006e6e0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f2c580b2a20 tx=0x7f2c400079d0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 msgr2=0x7f2c580b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c580b04f0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f2c4802c270 tx=0x7f2c48004730 comp rx=0 tx=0).stop 2024-01-15T18:12:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 shutdown_connections 2024-01-15T18:12:45.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2c3006c220 0x7f2c3006e6e0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2c58158890 0x7f2c58158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2c580b2f00 0x7f2c580b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 --2- 172.21.15.2:0/1889632847 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2c38003970 0x7f2c580b04f0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.732+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 >> 172.21.15.2:0/1889632847 conn(0x7f2c580b45f0 msgr2=0x7f2c580b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.733+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 shutdown_connections 2024-01-15T18:12:45.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.733+0000 7f2c5e000640 1 -- 172.21.15.2:0/1889632847 wait complete. 2024-01-15T18:12:45.759 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:12:45.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.980+0000 7fa877e00640 1 Processor -- start 2024-01-15T18:12:45.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.980+0000 7fa877e00640 1 -- start start 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa877e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa8781439a0 0x7fa8780cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa877e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa87806d240 0x7fa87806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa877e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8780cbcd0 con 0x7fa87815e8f0 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa877e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87806dc30 con 0x7fa8781439a0 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa877e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa878143d80 con 0x7fa87806d240 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa86de00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa87815e8f0 0x7fa8780b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52480/0 (socket says 172.21.15.2:52480) 2024-01-15T18:12:45.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa86de00640 1 -- 172.21.15.2:0/269537141 learned_addr learned my addr 172.21.15.2:0/269537141 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.982+0000 7fa876a00640 1 --2- 172.21.15.2:0/269537141 >> v2:172.21.15.2:3300/0 conn(0x7fa8781439a0 0x7fa8780cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa877400640 1 --2- 172.21.15.2:0/269537141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa87806d240 0x7fa87806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa876000640 1 -- 172.21.15.2:0/269537141 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453762205 0 0) 0x7fa8780cbcd0 con 0x7fa87815e8f0 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa876000640 1 -- 172.21.15.2:0/269537141 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa858005730 con 0x7fa87815e8f0 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa876a00640 1 -- 172.21.15.2:0/269537141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa87806d240 msgr2=0x7fa87806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa876a00640 1 --2- 172.21.15.2:0/269537141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa87806d240 0x7fa87806d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa876a00640 1 -- 172.21.15.2:0/269537141 >> v1:172.21.15.2:6789/0 conn(0x7fa87815e8f0 legacy=0x7fa8780b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.983+0000 7fa876a00640 1 -- 172.21.15.2:0/269537141 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8780b2250 con 0x7fa8781439a0 2024-01-15T18:12:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.984+0000 7fa876a00640 1 --2- 172.21.15.2:0/269537141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8781439a0 0x7fa8780cc160 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fa8680097a0 tx=0x7fa86802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8fb95f8b69d2e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.984+0000 7fa876000640 1 -- 172.21.15.2:0/269537141 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86802d040 con 0x7fa8781439a0 2024-01-15T18:12:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.984+0000 7fa876000640 1 -- 172.21.15.2:0/269537141 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868031810 con 0x7fa8781439a0 2024-01-15T18:12:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.984+0000 7fa876000640 1 -- 172.21.15.2:0/269537141 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868031ba0 con 0x7fa8781439a0 2024-01-15T18:12:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 -- 172.21.15.2:0/269537141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8781439a0 msgr2=0x7fa8780cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 --2- 172.21.15.2:0/269537141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8781439a0 0x7fa8780cc160 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7fa8680097a0 tx=0x7fa86802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 -- 172.21.15.2:0/269537141 shutdown_connections 2024-01-15T18:12:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 --2- 172.21.15.2:0/269537141 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa87806d240 0x7fa87806d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 --2- 172.21.15.2:0/269537141 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa8781439a0 0x7fa8780cc160 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 -- 172.21.15.2:0/269537141 >> 172.21.15.2:0/269537141 conn(0x7fa8780b0840 msgr2=0x7fa8780b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 -- 172.21.15.2:0/269537141 shutdown_connections 2024-01-15T18:12:45.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.985+0000 7fa877e00640 1 -- 172.21.15.2:0/269537141 wait complete. 2024-01-15T18:12:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.987+0000 7fa877e00640 1 Processor -- start 2024-01-15T18:12:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.987+0000 7fa877e00640 1 -- start start 2024-01-15T18:12:45.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa877e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 0x7fa8780ae020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa877e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8781439a0 0x7fa87814a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa877e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa87814b030 0x7fa87814a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa877e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8780ae560 con 0x7fa87806d240 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa877e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878148350 con 0x7fa87814b030 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa877e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87813e700 con 0x7fa8781439a0 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 0x7fa8780ae020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 0x7fa8780ae020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45414/0 (socket says 172.21.15.2:45414) 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 -- 172.21.15.2:0/3678166783 learned_addr learned my addr 172.21.15.2:0/3678166783 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 -- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8781439a0 msgr2=0x7fa87814a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8781439a0 0x7fa87814a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 -- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa87814b030 msgr2=0x7fa87814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa87814b030 0x7fa87814a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.988+0000 7fa876a00640 1 -- 172.21.15.2:0/3678166783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868009400 con 0x7fa87806d240 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.989+0000 7fa876a00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 0x7fa8780ae020 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fa8680098c0 tx=0x7fa8680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.989+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86803ad60 con 0x7fa87806d240 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.989+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa868044400 con 0x7fa87806d240 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.989+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8680445b0 con 0x7fa87806d240 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.989+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87813e9e0 con 0x7fa87806d240 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.989+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa878148880 con 0x7fa87806d240 2024-01-15T18:12:45.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.990+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87806bfa0 con 0x7fa87806d240 2024-01-15T18:12:45.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.992+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fa86804d440 con 0x7fa87806d240 2024-01-15T18:12:45.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.993+0000 7fa86f400640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85006c270 0x7fa85006e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:45.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.993+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fa8680aa910 con 0x7fa87806d240 2024-01-15T18:12:45.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.993+0000 7fa86de00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85006c270 0x7fa85006e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:45.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.995+0000 7fa86de00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85006c270 0x7fa85006e730 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa87813f1e0 tx=0x7fa8600015f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:45.997+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8680772f0 con 0x7fa87806d240 2024-01-15T18:12:46.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.218+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa87813ebf0 con 0x7fa85006c270 2024-01-15T18:12:46.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.222+0000 7fa86f400640 1 -- 172.21.15.2:0/3678166783 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa87813ebf0 con 0x7fa85006c270 2024-01-15T18:12:46.225 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:12:46.225 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:12:46.225 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:12:46.225 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:12:46.225 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:12:46.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85006c270 msgr2=0x7fa85006e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85006c270 0x7fa85006e730 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fa87813f1e0 tx=0x7fa8600015f0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 msgr2=0x7fa8780ae020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 0x7fa8780ae020 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fa8680098c0 tx=0x7fa8680046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 shutdown_connections 2024-01-15T18:12:46.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa85006c270 0x7fa85006e730 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa87814b030 0x7fa87814a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa8781439a0 0x7fa87814a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 --2- 172.21.15.2:0/3678166783 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa87806d240 0x7fa8780ae020 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 >> 172.21.15.2:0/3678166783 conn(0x7fa8780b0840 msgr2=0x7fa8780ce2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:46.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 shutdown_connections 2024-01-15T18:12:46.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.224+0000 7fa877e00640 1 -- 172.21.15.2:0/3678166783 wait complete. 2024-01-15T18:12:46.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:12:46.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.468+0000 7f1865e00640 1 Processor -- start 2024-01-15T18:12:46.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.469+0000 7f1865e00640 1 -- start start 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.470+0000 7f1865e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1860163be0 0x7f1860165fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.470+0000 7f1865e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f186006c500 con 0x7f18600ae260 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.470+0000 7f1865e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18600723f0 con 0x7f186006da80 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.470+0000 7f1865e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f186015c4b0 con 0x7f1860163be0 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.470+0000 7f1865400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1860163be0 0x7f1860165fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.471+0000 7f1864a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f18600ae260 0x7f1860071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52488/0 (socket says 172.21.15.2:52488) 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.471+0000 7f1864a00640 1 -- 172.21.15.2:0/943910755 learned_addr learned my addr 172.21.15.2:0/943910755 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:46.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.471+0000 7f1865400640 1 -- 172.21.15.2:0/943910755 >> v1:172.21.15.137:6789/0 conn(0x7f186006da80 legacy=0x7f186006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:46.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.471+0000 7f1865400640 1 -- 172.21.15.2:0/943910755 >> v1:172.21.15.2:6789/0 conn(0x7f18600ae260 legacy=0x7f1860071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.471+0000 7f1865400640 1 -- 172.21.15.2:0/943910755 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18600714d0 con 0x7f1860163be0 2024-01-15T18:12:46.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.473+0000 7f1865400640 1 --2- 172.21.15.2:0/943910755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1860163be0 0x7f1860165fd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f1858009d90 tx=0x7f185802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4466d9e7e73dc02d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.473+0000 7f185fe00640 1 -- 172.21.15.2:0/943910755 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185802cbd0 con 0x7f1860163be0 2024-01-15T18:12:46.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.473+0000 7f185fe00640 1 -- 172.21.15.2:0/943910755 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f185802cd30 con 0x7f1860163be0 2024-01-15T18:12:46.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.473+0000 7f185fe00640 1 -- 172.21.15.2:0/943910755 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f185802cea0 con 0x7f1860163be0 2024-01-15T18:12:46.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.473+0000 7f1865e00640 1 -- 172.21.15.2:0/943910755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1860163be0 msgr2=0x7f1860165fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.473+0000 7f1865e00640 1 --2- 172.21.15.2:0/943910755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1860163be0 0x7f1860165fd0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f1858009d90 tx=0x7f185802be20 comp rx=0 tx=0).stop 2024-01-15T18:12:46.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.474+0000 7f1865e00640 1 -- 172.21.15.2:0/943910755 shutdown_connections 2024-01-15T18:12:46.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.474+0000 7f1865e00640 1 --2- 172.21.15.2:0/943910755 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1860163be0 0x7f1860165fd0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.474+0000 7f1865e00640 1 -- 172.21.15.2:0/943910755 >> 172.21.15.2:0/943910755 conn(0x7f18600cebc0 msgr2=0x7f18600cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:46.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.474+0000 7f1865e00640 1 -- 172.21.15.2:0/943910755 shutdown_connections 2024-01-15T18:12:46.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.474+0000 7f1865e00640 1 -- 172.21.15.2:0/943910755 wait complete. 2024-01-15T18:12:46.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.475+0000 7f1865e00640 1 Processor -- start 2024-01-15T18:12:46.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.476+0000 7f1865e00640 1 -- start start 2024-01-15T18:12:46.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1865e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f186006da80 0x7f186015d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1865e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 0x7f18600aec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1865e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1860163be0 0x7f186015e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1865e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860154400 con 0x7f18600ae260 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1865e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18601537c0 con 0x7f1860163be0 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1865e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1860153a60 con 0x7f186006da80 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 0x7f18600aec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 0x7f18600aec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45432/0 (socket says 172.21.15.2:45432) 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 -- 172.21.15.2:0/1382810739 learned_addr learned my addr 172.21.15.2:0/1382810739 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1864a00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f186006da80 0x7f186015d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 -- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f186006da80 msgr2=0x7f186015d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f186006da80 0x7f186015d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 -- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1860163be0 msgr2=0x7f186015e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1860163be0 0x7f186015e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.477+0000 7f1857e00640 1 -- 172.21.15.2:0/1382810739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858009a70 con 0x7f18600ae260 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.478+0000 7f1857e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 0x7f18600aec10 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f184800b3f0 tx=0x7f184800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.478+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848011830 con 0x7f18600ae260 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.478+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1848011990 con 0x7f18600ae260 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.478+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1848011d40 con 0x7f18600ae260 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.478+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1860153d40 con 0x7f18600ae260 2024-01-15T18:12:46.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.478+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18600af150 con 0x7f18600ae260 2024-01-15T18:12:46.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.479+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1860153ed0 con 0x7f18600ae260 2024-01-15T18:12:46.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.480+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f184801a400 con 0x7f18600ae260 2024-01-15T18:12:46.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.481+0000 7f185d600640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f183806c1f0 0x7f183806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.481+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f18480885e0 con 0x7f18600ae260 2024-01-15T18:12:46.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.482+0000 7f1864a00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f183806c1f0 0x7f183806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.483+0000 7f1864a00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f183806c1f0 0x7f183806e6b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1850005880 tx=0x7f1850009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.486+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1848054de0 con 0x7f18600ae260 2024-01-15T18:12:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.679+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f1860074770 con 0x7f18600ae260 2024-01-15T18:12:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.679+0000 7f185d600640 1 -- 172.21.15.2:0/1382810739 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f18480150c0 con 0x7f18600ae260 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f183806c1f0 msgr2=0x7f183806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f183806c1f0 0x7f183806e6b0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f1850005880 tx=0x7f1850009040 comp rx=0 tx=0).stop 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 msgr2=0x7f18600aec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 0x7f18600aec10 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f184800b3f0 tx=0x7f184800b8b0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 shutdown_connections 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f183806c1f0 0x7f183806e6b0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1860163be0 0x7f186015e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f18600ae260 0x7f18600aec10 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 --2- 172.21.15.2:0/1382810739 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f186006da80 0x7f186015d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 >> 172.21.15.2:0/1382810739 conn(0x7f18600cebc0 msgr2=0x7f18600abd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:46.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 shutdown_connections 2024-01-15T18:12:46.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.682+0000 7f1865e00640 1 -- 172.21.15.2:0/1382810739 wait complete. 2024-01-15T18:12:46.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:12:46.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.927+0000 7fb788a00640 1 Processor -- start 2024-01-15T18:12:46.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.928+0000 7fb788a00640 1 -- start start 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.929+0000 7fb788a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb784143620 0x7fb7840b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.929+0000 7fb788a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb784071940 0x7fb7840cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.929+0000 7fb788a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7840b2fa0 con 0x7fb7840b2a90 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.929+0000 7fb788a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7840b3280 con 0x7fb784143620 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.929+0000 7fb788a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7840b3560 con 0x7fb784071940 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb784143620 0x7fb7840b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb784143620 0x7fb7840b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45448/0 (socket says 172.21.15.2:45448) 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783400640 1 -- 172.21.15.2:0/3655043696 learned_addr learned my addr 172.21.15.2:0/3655043696 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783e00640 1 --2- 172.21.15.2:0/3655043696 >> v2:172.21.15.137:3300/0 conn(0x7fb784071940 0x7fb7840cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783400640 1 -- 172.21.15.2:0/3655043696 >> v2:172.21.15.137:3300/0 conn(0x7fb784071940 msgr2=0x7fb7840cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783400640 1 --2- 172.21.15.2:0/3655043696 >> v2:172.21.15.137:3300/0 conn(0x7fb784071940 0x7fb7840cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.930+0000 7fb783400640 1 -- 172.21.15.2:0/3655043696 >> v1:172.21.15.137:6789/0 conn(0x7fb7840b2a90 legacy=0x7fb784071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.931+0000 7fb783400640 1 -- 172.21.15.2:0/3655043696 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7840b3840 con 0x7fb784143620 2024-01-15T18:12:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.932+0000 7fb783400640 1 --2- 172.21.15.2:0/3655043696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840b2550 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fb78406fc80 tx=0x7fb77402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5920cdbe0cc514f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.933+0000 7fb782000640 1 -- 172.21.15.2:0/3655043696 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774030860 con 0x7fb784143620 2024-01-15T18:12:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.933+0000 7fb782000640 1 -- 172.21.15.2:0/3655043696 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7740309c0 con 0x7fb784143620 2024-01-15T18:12:46.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.933+0000 7fb782000640 1 -- 172.21.15.2:0/3655043696 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774030d50 con 0x7fb784143620 2024-01-15T18:12:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 -- 172.21.15.2:0/3655043696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 msgr2=0x7fb7840b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 --2- 172.21.15.2:0/3655043696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840b2550 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fb78406fc80 tx=0x7fb77402b9a0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 -- 172.21.15.2:0/3655043696 shutdown_connections 2024-01-15T18:12:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 --2- 172.21.15.2:0/3655043696 >> v2:172.21.15.137:3300/0 conn(0x7fb784071940 0x7fb7840cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 --2- 172.21.15.2:0/3655043696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840b2550 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 -- 172.21.15.2:0/3655043696 >> 172.21.15.2:0/3655043696 conn(0x7fb7840b0840 msgr2=0x7fb7840b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 -- 172.21.15.2:0/3655043696 shutdown_connections 2024-01-15T18:12:46.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.934+0000 7fb788a00640 1 -- 172.21.15.2:0/3655043696 wait complete. 2024-01-15T18:12:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.936+0000 7fb788a00640 1 Processor -- start 2024-01-15T18:12:46.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.936+0000 7fb788a00640 1 -- start start 2024-01-15T18:12:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb788a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb784071940 0x7fb78413ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb788a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7840b2a90 0x7fb7840ad7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb788a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb788a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78413c630 con 0x7fb784143620 2024-01-15T18:12:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb788a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78413b590 con 0x7fb7840b2a90 2024-01-15T18:12:46.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb788a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78413b870 con 0x7fb784071940 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840adce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45458/0 (socket says 172.21.15.2:45458) 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 -- 172.21.15.2:0/1043121318 learned_addr learned my addr 172.21.15.2:0/1043121318 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb782a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7840b2a90 0x7fb7840ad7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783400640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb784071940 0x7fb78413ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 -- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb784071940 msgr2=0x7fb78413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb784071940 0x7fb78413ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 -- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7840b2a90 msgr2=0x7fb7840ad7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:46.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7840b2a90 0x7fb7840ad7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.937+0000 7fb783e00640 1 -- 172.21.15.2:0/1043121318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7740094d0 con 0x7fb784143620 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.938+0000 7fb783e00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840adce0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fb77c007e30 tx=0x7fb77c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.938+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c014620 con 0x7fb784143620 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.938+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77c00ee90 con 0x7fb784143620 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.938+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78413bb50 con 0x7fb784143620 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.938+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0149c0 con 0x7fb784143620 2024-01-15T18:12:46.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.939+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78413c110 con 0x7fb784143620 2024-01-15T18:12:46.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.940+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb784074770 con 0x7fb784143620 2024-01-15T18:12:46.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.941+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fb77c014b20 con 0x7fb784143620 2024-01-15T18:12:46.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.942+0000 7fb77b400640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb75c06c220 0x7fb75c06e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:46.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.942+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fb77c089350 con 0x7fb784143620 2024-01-15T18:12:46.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.942+0000 7fb783400640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb75c06c220 0x7fb75c06e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:46.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.944+0000 7fb783400640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb75c06c220 0x7fb75c06e6e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb774036000 tx=0x7fb77403b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:46.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:46.947+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb77c055920 con 0x7fb784143620 2024-01-15T18:12:47.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.141+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fb7840cab20 con 0x7fb784143620 2024-01-15T18:12:47.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.148+0000 7fb77b400640 1 -- 172.21.15.2:0/1043121318 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v35)=0 v35) v1 ==== 131+0+0 (secure 0 0 0) 0x7fb77c058f40 con 0x7fb784143620 2024-01-15T18:12:47.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.151+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb75c06c220 msgr2=0x7fb75c06e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.151+0000 7fb788a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb75c06c220 0x7fb75c06e6e0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fb774036000 tx=0x7fb77403b040 comp rx=0 tx=0).stop 2024-01-15T18:12:47.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.151+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 msgr2=0x7fb7840adce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.151+0000 7fb788a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840adce0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fb77c007e30 tx=0x7fb77c00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 shutdown_connections 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb75c06c220 0x7fb75c06e6e0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb784143620 0x7fb7840adce0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb7840b2a90 0x7fb7840ad7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 --2- 172.21.15.2:0/1043121318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb784071940 0x7fb78413ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 >> 172.21.15.2:0/1043121318 conn(0x7fb7840b0840 msgr2=0x7fb78414efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 shutdown_connections 2024-01-15T18:12:47.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.152+0000 7fb788a00640 1 -- 172.21.15.2:0/1043121318 wait complete. 2024-01-15T18:12:47.175 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-15T18:12:47.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.397+0000 7ff022a00640 1 Processor -- start 2024-01-15T18:12:47.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.397+0000 7ff022a00640 1 -- start start 2024-01-15T18:12:47.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff022a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff01c143d20 0x7ff01c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff022a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff01c0720d0 0x7ff01c15f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff022a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 0x7ff01c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff022a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c071430 con 0x7ff01c0720d0 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff022a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c144100 con 0x7ff01c143d20 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff022a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c071090 con 0x7ff01c1447d0 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff020c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff01c0720d0 0x7ff01c15f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff020c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff01c0720d0 0x7ff01c15f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45460/0 (socket says 172.21.15.2:45460) 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.399+0000 7ff020c00640 1 -- 172.21.15.2:0/3420316597 learned_addr learned my addr 172.21.15.2:0/3420316597 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.400+0000 7ff020c00640 1 -- 172.21.15.2:0/3420316597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 msgr2=0x7ff01c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.400+0000 7ff020c00640 1 --2- 172.21.15.2:0/3420316597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 0x7ff01c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.400+0000 7ff020c00640 1 -- 172.21.15.2:0/3420316597 >> v2:172.21.15.137:3300/0 conn(0x7ff01c143d20 msgr2=0x7ff01c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:47.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.400+0000 7ff021600640 1 --2- 172.21.15.2:0/3420316597 >> v2:172.21.15.137:3300/0 conn(0x7ff01c143d20 0x7ff01c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.400+0000 7ff020c00640 1 --2- 172.21.15.2:0/3420316597 >> v2:172.21.15.137:3300/0 conn(0x7ff01c143d20 0x7ff01c15eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.400+0000 7ff020c00640 1 -- 172.21.15.2:0/3420316597 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c144560 con 0x7ff01c0720d0 2024-01-15T18:12:47.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.401+0000 7ff020c00640 1 --2- 172.21.15.2:0/3420316597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c0720d0 0x7ff01c15f090 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7ff01c06e600 tx=0x7ff00402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d83edc4eaceb8a24 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:47.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.401+0000 7ff01be00640 1 -- 172.21.15.2:0/3420316597 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0040317e0 con 0x7ff01c0720d0 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.401+0000 7ff01be00640 1 -- 172.21.15.2:0/3420316597 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff004031940 con 0x7ff01c0720d0 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.401+0000 7ff01be00640 1 -- 172.21.15.2:0/3420316597 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004031cd0 con 0x7ff01c0720d0 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 -- 172.21.15.2:0/3420316597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c0720d0 msgr2=0x7ff01c15f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 --2- 172.21.15.2:0/3420316597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c0720d0 0x7ff01c15f090 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7ff01c06e600 tx=0x7ff00402b880 comp rx=0 tx=0).stop 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 -- 172.21.15.2:0/3420316597 shutdown_connections 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 --2- 172.21.15.2:0/3420316597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 0x7ff01c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 --2- 172.21.15.2:0/3420316597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c0720d0 0x7ff01c15f090 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 --2- 172.21.15.2:0/3420316597 >> v2:172.21.15.137:3300/0 conn(0x7ff01c143d20 0x7ff01c15eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 -- 172.21.15.2:0/3420316597 >> 172.21.15.2:0/3420316597 conn(0x7ff01c0b0840 msgr2=0x7ff01c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:47.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 -- 172.21.15.2:0/3420316597 shutdown_connections 2024-01-15T18:12:47.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.402+0000 7ff022a00640 1 -- 172.21.15.2:0/3420316597 wait complete. 2024-01-15T18:12:47.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.404+0000 7ff022a00640 1 Processor -- start 2024-01-15T18:12:47.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.404+0000 7ff022a00640 1 -- start start 2024-01-15T18:12:47.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff022a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff01c0720d0 0x7ff01c1421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff022a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 0x7ff01c142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff022a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 0x7ff01c140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff022a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c140da0 con 0x7ff01c143d20 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff022a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c141080 con 0x7ff01c0720d0 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff022a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c15fc20 con 0x7ff01c1447d0 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.405+0000 7ff020c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 0x7ff01c142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 0x7ff01c142710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45468/0 (socket says 172.21.15.2:45468) 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 -- 172.21.15.2:0/3713667223 learned_addr learned my addr 172.21.15.2:0/3713667223 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 -- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 msgr2=0x7ff01c140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 0x7ff01c140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 -- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff01c0720d0 msgr2=0x7ff01c1421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff01c0720d0 0x7ff01c1421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 -- 172.21.15.2:0/3713667223 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0040094d0 con 0x7ff01c143d20 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.406+0000 7ff020c00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 0x7ff01c142710 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff0040090b0 tx=0x7ff0040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.407+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff004032980 con 0x7ff01c143d20 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.407+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff004032b00 con 0x7ff01c143d20 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.407+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0040453f0 con 0x7ff01c143d20 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.407+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01c15ff00 con 0x7ff01c143d20 2024-01-15T18:12:47.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.407+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c13d970 con 0x7ff01c143d20 2024-01-15T18:12:47.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.408+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c074770 con 0x7ff01c143d20 2024-01-15T18:12:47.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.409+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7ff00404d3f0 con 0x7ff01c143d20 2024-01-15T18:12:47.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.410+0000 7ff019600640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feff406c1f0 0x7feff406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.410+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7ff0040aa500 con 0x7ff01c143d20 2024-01-15T18:12:47.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.411+0000 7ff021600640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feff406c1f0 0x7feff406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.412+0000 7ff021600640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feff406c1f0 0x7feff406e6b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00a7f0 tx=0x7ff00c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:47.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.414+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff004076ed0 con 0x7ff01c143d20 2024-01-15T18:12:47.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.607+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7ff01c15fa00 con 0x7ff01c143d20 2024-01-15T18:12:47.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.613+0000 7ff019600640 1 -- 172.21.15.2:0/3713667223 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v36)=0 v36) v1 ==== 137+0+0 (secure 0 0 0) 0x7ff00407a4f0 con 0x7ff01c143d20 2024-01-15T18:12:47.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feff406c1f0 msgr2=0x7feff406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feff406c1f0 0x7feff406e6b0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00a7f0 tx=0x7ff00c008040 comp rx=0 tx=0).stop 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 msgr2=0x7ff01c142710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 0x7ff01c142710 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7ff0040090b0 tx=0x7ff0040046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 shutdown_connections 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7feff406c1f0 0x7feff406e6b0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff01c1447d0 0x7ff01c140060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff01c143d20 0x7ff01c142710 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 --2- 172.21.15.2:0/3713667223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff01c0720d0 0x7ff01c1421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 >> 172.21.15.2:0/3713667223 conn(0x7ff01c0b0840 msgr2=0x7ff01c06ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 shutdown_connections 2024-01-15T18:12:47.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.616+0000 7ff022a00640 1 -- 172.21.15.2:0/3713667223 wait complete. 2024-01-15T18:12:47.637 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:12:47.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.851+0000 7ff735e00640 1 Processor -- start 2024-01-15T18:12:47.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.851+0000 7ff735e00640 1 -- start start 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.853+0000 7ff735e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff73006d000 0x7ff730165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.853+0000 7ff735e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7300ce230 con 0x7ff73015c450 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.853+0000 7ff735e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7300ce510 con 0x7ff730144c00 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.853+0000 7ff735e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7300ce7f0 con 0x7ff73006d000 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72e000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7ff73015c450 0x7ff73006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52502/0 (socket says 172.21.15.2:52502) 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72e000640 1 -- 172.21.15.2:0/243228326 learned_addr learned my addr 172.21.15.2:0/243228326 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:47.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72f400640 1 --2- 172.21.15.2:0/243228326 >> v2:172.21.15.2:3300/0 conn(0x7ff73006d000 0x7ff730165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72d600640 1 -- 172.21.15.2:0/243228326 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519138113 0 0) 0x7ff7300ce230 con 0x7ff73015c450 2024-01-15T18:12:47.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72d600640 1 -- 172.21.15.2:0/243228326 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff710003520 con 0x7ff73015c450 2024-01-15T18:12:47.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72f400640 1 -- 172.21.15.2:0/243228326 >> v1:172.21.15.137:6789/0 conn(0x7ff730144c00 legacy=0x7ff7300cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72f400640 1 -- 172.21.15.2:0/243228326 >> v1:172.21.15.2:6789/0 conn(0x7ff73015c450 legacy=0x7ff73006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.854+0000 7ff72f400640 1 -- 172.21.15.2:0/243228326 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300705f0 con 0x7ff73006d000 2024-01-15T18:12:47.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.855+0000 7ff72f400640 1 --2- 172.21.15.2:0/243228326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730165d60 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff728009d90 tx=0x7ff72802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed92c174c56e4f00 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:47.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff72d600640 1 -- 172.21.15.2:0/243228326 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72802ccf0 con 0x7ff73006d000 2024-01-15T18:12:47.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff72d600640 1 -- 172.21.15.2:0/243228326 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72802ce50 con 0x7ff73006d000 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff72d600640 1 -- 172.21.15.2:0/243228326 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728034640 con 0x7ff73006d000 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff735e00640 1 -- 172.21.15.2:0/243228326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 msgr2=0x7ff730165d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff735e00640 1 --2- 172.21.15.2:0/243228326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730165d60 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7ff728009d90 tx=0x7ff72802bf40 comp rx=0 tx=0).stop 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff735e00640 1 -- 172.21.15.2:0/243228326 shutdown_connections 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff735e00640 1 --2- 172.21.15.2:0/243228326 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730165d60 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff735e00640 1 -- 172.21.15.2:0/243228326 >> 172.21.15.2:0/243228326 conn(0x7ff7300b45f0 msgr2=0x7ff7300b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.856+0000 7ff735e00640 1 -- 172.21.15.2:0/243228326 shutdown_connections 2024-01-15T18:12:47.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.857+0000 7ff735e00640 1 -- 172.21.15.2:0/243228326 wait complete. 2024-01-15T18:12:47.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.858+0000 7ff735e00640 1 Processor -- start 2024-01-15T18:12:47.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.858+0000 7ff735e00640 1 -- start start 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff735e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff735e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff730144c00 0x7ff7301f4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff735e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff73015c450 0x7ff7301f4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff735e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7301577e0 con 0x7ff73006d000 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff735e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73013dcd0 con 0x7ff73015c450 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff735e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff730153890 con 0x7ff730144c00 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff72ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff72ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45476/0 (socket says 172.21.15.2:45476) 2024-01-15T18:12:47.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.859+0000 7ff72ea00640 1 -- 172.21.15.2:0/1742295123 learned_addr learned my addr 172.21.15.2:0/1742295123 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72f400640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff73015c450 0x7ff7301f4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72e000640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff730144c00 0x7ff7301f4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72ea00640 1 -- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff730144c00 msgr2=0x7ff7301f4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72ea00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff730144c00 0x7ff7301f4030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72ea00640 1 -- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff73015c450 msgr2=0x7ff7301f4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72ea00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff73015c450 0x7ff7301f4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72ea00640 1 -- 172.21.15.2:0/1742295123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff728009a70 con 0x7ff73006d000 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72f400640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff73015c450 0x7ff7301f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff72ea00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730153350 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff71800c400 tx=0x7ff71800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7180052d0 con 0x7ff73006d000 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff718005450 con 0x7ff73006d000 2024-01-15T18:12:47.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.860+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff730157220 con 0x7ff73006d000 2024-01-15T18:12:47.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.861+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff718007260 con 0x7ff73006d000 2024-01-15T18:12:47.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.861+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73013d8f0 con 0x7ff73006d000 2024-01-15T18:12:47.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.861+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73013d750 con 0x7ff73006d000 2024-01-15T18:12:47.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.863+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7ff7180073c0 con 0x7ff73006d000 2024-01-15T18:12:47.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.864+0000 7ff727400640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff70806c1f0 0x7ff70806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:47.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.864+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7ff718088170 con 0x7ff73006d000 2024-01-15T18:12:47.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.864+0000 7ff72e000640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff70806c1f0 0x7ff70806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:47.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.866+0000 7ff72e000640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff70806c1f0 0x7ff70806e6b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff7300b1bd0 tx=0x7ff720009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:47.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:47.867+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff718054740 con 0x7ff73006d000 2024-01-15T18:12:48.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.079+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7300cead0 con 0x7ff70806c1f0 2024-01-15T18:12:48.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:48.090+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:48.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:48.090+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:48.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:48.090+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:48.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:48.090+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:48.093 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:12:48.090+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:12:48.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.150+0000 7ff727400640 1 -- 172.21.15.2:0/1742295123 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff7200012d0 con 0x7ff70806c1f0 2024-01-15T18:12:48.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff70806c1f0 msgr2=0x7ff70806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff70806c1f0 0x7ff70806e6b0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff7300b1bd0 tx=0x7ff720009040 comp rx=0 tx=0).stop 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 msgr2=0x7ff730153350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730153350 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7ff71800c400 tx=0x7ff71800c8c0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 shutdown_connections 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff70806c1f0 0x7ff70806e6b0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff73015c450 0x7ff7301f4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff730144c00 0x7ff7301f4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 --2- 172.21.15.2:0/1742295123 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff73006d000 0x7ff730153350 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.152+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 >> 172.21.15.2:0/1742295123 conn(0x7ff7300b45f0 msgr2=0x7ff730070830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.153+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 shutdown_connections 2024-01-15T18:12:48.156 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.153+0000 7ff735e00640 1 -- 172.21.15.2:0/1742295123 wait complete. 2024-01-15T18:12:48.176 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:12:48.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.383+0000 7f27ece00640 1 Processor -- start 2024-01-15T18:12:48.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.383+0000 7f27ece00640 1 -- start start 2024-01-15T18:12:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27ece00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f27e8143d20 0x7f27e815eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27ece00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27e80720d0 0x7f27e815f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27ece00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e8071090 con 0x7f27e81447d0 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27ece00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e8071430 con 0x7f27e8143d20 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27ece00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e8144100 con 0x7f27e80720d0 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27e7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f27e81447d0 0x7f27e80cd330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52510/0 (socket says 172.21.15.2:52510) 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.385+0000 7f27e7e00640 1 -- 172.21.15.2:0/1105939343 learned_addr learned my addr 172.21.15.2:0/1105939343 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27dfe00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27e80720d0 0x7f27e815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e7400640 1 --2- 172.21.15.2:0/1105939343 >> v2:172.21.15.137:3300/0 conn(0x7f27e8143d20 0x7f27e815eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222795435 0 0) 0x7f27e8071090 con 0x7f27e81447d0 2024-01-15T18:12:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c8003520 con 0x7f27e81447d0 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1958143780 0 0) 0x7f27c8003520 con 0x7f27e81447d0 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27e80720d0 msgr2=0x7f27e815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27e80720d0 0x7f27e815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 >> v2:172.21.15.137:3300/0 conn(0x7f27e8143d20 msgr2=0x7f27e815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 --2- 172.21.15.2:0/1105939343 >> v2:172.21.15.137:3300/0 conn(0x7f27e8143d20 0x7f27e815eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e806d160 con 0x7f27e81447d0 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f27e00019f0 con 0x7f27e81447d0 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.386+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 >> v1:172.21.15.2:6789/0 conn(0x7f27e81447d0 legacy=0x7f27e80cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e6a00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27c8003970 0x7f27c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e6a00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c80047c0 0x7f27c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e6a00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27c8005180 0x7f27c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e8071430 con 0x7f27c8005180 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e8144100 con 0x7f27c8003970 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c800bfa0 con 0x7f27c80047c0 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27dfe00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27e80720d0 0x7f27e815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7e00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27c8005180 0x7f27c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27dfe00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c80047c0 0x7f27c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7400640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27c8003970 0x7f27c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7e00640 1 -- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c80047c0 msgr2=0x7f27c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7e00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c80047c0 0x7f27c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7e00640 1 -- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27c8003970 msgr2=0x7f27c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7e00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27c8003970 0x7f27c8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.387+0000 7f27e7e00640 1 -- 172.21.15.2:0/1105939343 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c8003520 con 0x7f27c8005180 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.388+0000 7f27dfe00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c80047c0 0x7f27c8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.389+0000 7f27e7e00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27c8005180 0x7f27c800b980 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f27e000a860 tx=0x7f27c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=504d5f144e065265 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.389+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0034630 con 0x7f27c8005180 2024-01-15T18:12:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.389+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e002f070 con 0x7f27c8005180 2024-01-15T18:12:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.389+0000 7f27e6a00640 1 -- 172.21.15.2:0/1105939343 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00349f0 con 0x7f27c8005180 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.389+0000 7f27ece00640 1 -- 172.21.15.2:0/1105939343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27c8005180 msgr2=0x7f27c800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.389+0000 7f27ece00640 1 --2- 172.21.15.2:0/1105939343 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27c8005180 0x7f27c800b980 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f27e000a860 tx=0x7f27c80057f0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.390+0000 7f27e7e00640 1 -- 172.21.15.2:0/1105939343 reap_dead start 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.390+0000 7f27ece00640 1 -- 172.21.15.2:0/1105939343 shutdown_connections 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.390+0000 7f27ece00640 1 -- 172.21.15.2:0/1105939343 >> 172.21.15.2:0/1105939343 conn(0x7f27e80b0840 msgr2=0x7f27e80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.390+0000 7f27ece00640 1 -- 172.21.15.2:0/1105939343 shutdown_connections 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.390+0000 7f27ece00640 1 -- 172.21.15.2:0/1105939343 wait complete. 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.391+0000 7f27ece00640 1 Processor -- start 2024-01-15T18:12:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.392+0000 7f27ece00640 1 -- start start 2024-01-15T18:12:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.392+0000 7f27ece00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c8005180 0x7f27e813b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27ece00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27e81f6da0 0x7f27e815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27ece00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 0x7f27e81f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27ece00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e815d330 con 0x7f27e81f6190 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27ece00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e81f6b50 con 0x7f27e81f6da0 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27ece00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e815ad30 con 0x7f27c8005180 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 0x7f27e81f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 0x7f27e81f6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45490/0 (socket says 172.21.15.2:45490) 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 -- 172.21.15.2:0/36945002 learned_addr learned my addr 172.21.15.2:0/36945002 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27dfe00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27e81f6da0 0x7f27e815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7400640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c8005180 0x7f27e813b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 -- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c8005180 msgr2=0x7f27e813b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c8005180 0x7f27e813b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 -- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27e81f6da0 msgr2=0x7f27e815b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27e81f6da0 0x7f27e815b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.393+0000 7f27e7e00640 1 -- 172.21.15.2:0/36945002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e815a490 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.394+0000 7f27e7e00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 0x7f27e81f6610 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f27e002cba0 tx=0x7f27e0001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.394+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0001fd0 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.394+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0034640 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.394+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00349b0 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.394+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e815a770 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.394+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e815bae0 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.395+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e815d470 con 0x7f27e81f6190 2024-01-15T18:12:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.396+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f27e0031030 con 0x7f27e81f6190 2024-01-15T18:12:48.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.398+0000 7f27df400640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f27c006c1f0 0x7f27c006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.398+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f27e00aaa60 con 0x7f27e81f6190 2024-01-15T18:12:48.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.399+0000 7f27e7400640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f27c006c1f0 0x7f27c006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.400+0000 7f27e7400640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f27c006c1f0 0x7f27c006e6b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009a90 tx=0x7f27d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:48.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.404+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27e815d470 con 0x7f27e81f6190 2024-01-15T18:12:48.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.634+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f27e815f720 con 0x7f27c006c1f0 2024-01-15T18:12:48.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.646+0000 7f27df400640 1 -- 172.21.15.2:0/36945002 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f27d80129b0 con 0x7f27c006c1f0 2024-01-15T18:12:48.649 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392 2024-01-15T18:12:48.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f27c006c1f0 msgr2=0x7f27c006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f27c006c1f0 0x7f27c006e6b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f27d8009a90 tx=0x7f27d8008040 comp rx=0 tx=0).stop 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 msgr2=0x7f27e81f6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 0x7f27e81f6610 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f27e002cba0 tx=0x7f27e0001d80 comp rx=0 tx=0).stop 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 shutdown_connections 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f27c006c1f0 0x7f27c006e6b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f27e81f6190 0x7f27e81f6610 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f27e81f6da0 0x7f27e815b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 --2- 172.21.15.2:0/36945002 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f27c8005180 0x7f27e813b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.648+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 >> 172.21.15.2:0/36945002 conn(0x7f27e80b0840 msgr2=0x7f27e8142fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.649+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 shutdown_connections 2024-01-15T18:12:48.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.649+0000 7f27ece00640 1 -- 172.21.15.2:0/36945002 wait complete. 2024-01-15T18:12:48.671 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:12:48.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.892+0000 7f51dce00640 1 Processor -- start 2024-01-15T18:12:48.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.892+0000 7f51dce00640 1 -- start start 2024-01-15T18:12:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.894+0000 7f51dce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d815f260 0x7f51d806c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.894+0000 7f51dce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51d80720d0 0x7f51d806c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.894+0000 7f51dce00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51d8071090 con 0x7f51d8143d20 2024-01-15T18:12:48.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51dce00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51d8071430 con 0x7f51d80720d0 2024-01-15T18:12:48.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51dce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d8144100 con 0x7f51d815f260 2024-01-15T18:12:48.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51d80720d0 0x7f51d806c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f51d80720d0 0x7f51d806c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45492/0 (socket says 172.21.15.2:45492) 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 -- 172.21.15.2:0/379899421 learned_addr learned my addr 172.21.15.2:0/379899421 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51ce800640 1 --2- 172.21.15.2:0/379899421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d815f260 0x7f51d806c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 -- 172.21.15.2:0/379899421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d815f260 msgr2=0x7f51d806c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 --2- 172.21.15.2:0/379899421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d815f260 0x7f51d806c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 -- 172.21.15.2:0/379899421 >> v1:172.21.15.137:6789/0 conn(0x7f51d8143d20 legacy=0x7f51d815eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.895+0000 7f51d7e00640 1 -- 172.21.15.2:0/379899421 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d8144560 con 0x7f51d80720d0 2024-01-15T18:12:48.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.897+0000 7f51d7e00640 1 --2- 172.21.15.2:0/379899421 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d80720d0 0x7f51d806c690 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009870 tx=0x7f51d002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1b866c03be5ff3a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.897+0000 7f51d6a00640 1 -- 172.21.15.2:0/379899421 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0031740 con 0x7f51d80720d0 2024-01-15T18:12:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.897+0000 7f51d6a00640 1 -- 172.21.15.2:0/379899421 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d00318a0 con 0x7f51d80720d0 2024-01-15T18:12:48.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.897+0000 7f51d6a00640 1 -- 172.21.15.2:0/379899421 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0031c30 con 0x7f51d80720d0 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 -- 172.21.15.2:0/379899421 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d80720d0 msgr2=0x7f51d806c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 --2- 172.21.15.2:0/379899421 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d80720d0 0x7f51d806c690 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009870 tx=0x7f51d002b7e0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 -- 172.21.15.2:0/379899421 shutdown_connections 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 --2- 172.21.15.2:0/379899421 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d80720d0 0x7f51d806c690 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 --2- 172.21.15.2:0/379899421 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d815f260 0x7f51d806c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 -- 172.21.15.2:0/379899421 >> 172.21.15.2:0/379899421 conn(0x7f51d80b0840 msgr2=0x7f51d80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 -- 172.21.15.2:0/379899421 shutdown_connections 2024-01-15T18:12:48.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.898+0000 7f51dce00640 1 -- 172.21.15.2:0/379899421 wait complete. 2024-01-15T18:12:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.900+0000 7f51dce00640 1 Processor -- start 2024-01-15T18:12:48.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.900+0000 7f51dce00640 1 -- start start 2024-01-15T18:12:48.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51dce00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d813d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51dce00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d8143d20 0x7f51d813fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51dce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 0x7f51d8140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51dce00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d815bc60 con 0x7f51d815f260 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51dce00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51d8140c40 con 0x7f51d8143d20 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51dce00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d8140f20 con 0x7f51d80720d0 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 0x7f51d8140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 0x7f51d8140370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45502/0 (socket says 172.21.15.2:45502) 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.901+0000 7f51d7e00640 1 -- 172.21.15.2:0/2302814880 learned_addr learned my addr 172.21.15.2:0/2302814880 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7400640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d813d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7e00640 1 -- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 msgr2=0x7f51d813d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51ce800640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d8143d20 0x7f51d813fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d813d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7e00640 1 -- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d8143d20 msgr2=0x7f51d813fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d8143d20 0x7f51d813fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7e00640 1 -- 172.21.15.2:0/2302814880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d00094d0 con 0x7f51d815f260 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7400640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d813d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.902+0000 7f51d7e00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 0x7f51d8140370 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009870 tx=0x7f51d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.903+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d0033c70 con 0x7f51d815f260 2024-01-15T18:12:48.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.903+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d0033df0 con 0x7f51d815f260 2024-01-15T18:12:48.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.903+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d003c640 con 0x7f51d815f260 2024-01-15T18:12:48.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.903+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51d815b5a0 con 0x7f51d815f260 2024-01-15T18:12:48.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.903+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51d813db00 con 0x7f51d815f260 2024-01-15T18:12:48.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.904+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51d815bda0 con 0x7f51d815f260 2024-01-15T18:12:48.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.905+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f51d003a040 con 0x7f51d815f260 2024-01-15T18:12:48.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.906+0000 7f51cfe00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006c1f0 0x7f51b006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:12:48.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.906+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f51d00aa070 con 0x7f51d815f260 2024-01-15T18:12:48.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.907+0000 7f51d7400640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006c1f0 0x7f51b006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:12:48.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.908+0000 7f51d7400640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006c1f0 0x7f51b006e6b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f51d8159b60 tx=0x7f51c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:12:48.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:48.911+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51d815bda0 con 0x7f51d815f260 2024-01-15T18:12:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.132+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f51d815fa00 con 0x7f51b006c1f0 2024-01-15T18:12:49.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.143+0000 7f51cfe00640 1 -- 172.21.15.2:0/2302814880 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f51c8005050 con 0x7f51b006c1f0 2024-01-15T18:12:49.146 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006c1f0 msgr2=0x7f51b006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006c1f0 0x7f51b006e6b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f51d8159b60 tx=0x7f51c8009040 comp rx=0 tx=0).stop 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 msgr2=0x7f51d8140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 0x7f51d8140370 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f51d0009870 tx=0x7f51d00046c0 comp rx=0 tx=0).stop 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 shutdown_connections 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f51b006c1f0 0x7f51b006e6b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:49.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f51d815f260 0x7f51d8140370 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:49.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f51d8143d20 0x7f51d813fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:49.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 --2- 172.21.15.2:0/2302814880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f51d80720d0 0x7f51d813d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:12:49.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 >> 172.21.15.2:0/2302814880 conn(0x7f51d80b0840 msgr2=0x7f51d806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:12:49.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 shutdown_connections 2024-01-15T18:12:49.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:12:49.146+0000 7f51dce00640 1 -- 172.21.15.2:0/2302814880 wait complete. 2024-01-15T18:12:49.167 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392 2024-01-15T18:12:49.168 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:12:49.168 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:12:49.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:12:49.220 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:13:03.034 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:03.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:06.333 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:13:06.333 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:13:06.333 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 17.0217 s, 63.1 MB/s 2024-01-15T18:13:06.335 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:13:06.335 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:13:06.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:13:06.392 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:13:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:13:20.831 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:13:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.3712 s, 74.7 MB/s 2024-01-15T18:13:20.833 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:13:20.833 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:13:20.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:13:20.887 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:13:33.173 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:33.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:13:35.342 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:13:35.342 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:13:35.342 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.3885 s, 74.6 MB/s 2024-01-15T18:13:35.344 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:13:35.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.577+0000 7fdc60400640 1 Processor -- start 2024-01-15T18:13:35.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.577+0000 7fdc60400640 1 -- start start 2024-01-15T18:13:35.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.579+0000 7fdc60400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc5806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:35.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.579+0000 7fdc60400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdc5806ce30 0x7fdc58070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:35.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.579+0000 7fdc60400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc580ce230 con 0x7fdc58144c00 2024-01-15T18:13:35.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.579+0000 7fdc60400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5806d2c0 con 0x7fdc5806ce30 2024-01-15T18:13:35.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.579+0000 7fdc60400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc580ce510 con 0x7fdc5815c450 2024-01-15T18:13:35.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.580+0000 7fdc5d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc5806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:35.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.580+0000 7fdc5e400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdc5806ce30 0x7fdc58070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:35.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.580+0000 7fdc5d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc5806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:44798/0 (socket says 172.21.15.2:44798) 2024-01-15T18:13:35.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.580+0000 7fdc5d000640 1 -- 172.21.15.2:0/3388562848 learned_addr learned my addr 172.21.15.2:0/3388562848 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:35.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.581+0000 7fdc5d000640 1 -- 172.21.15.2:0/3388562848 >> v2:172.21.15.137:3300/0 conn(0x7fdc5806ce30 msgr2=0x7fdc58070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:35.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.581+0000 7fdc5d000640 1 --2- 172.21.15.2:0/3388562848 >> v2:172.21.15.137:3300/0 conn(0x7fdc5806ce30 0x7fdc58070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.581+0000 7fdc5d000640 1 -- 172.21.15.2:0/3388562848 >> v1:172.21.15.137:6789/0 conn(0x7fdc58144c00 legacy=0x7fdc580cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:35.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.581+0000 7fdc5d000640 1 -- 172.21.15.2:0/3388562848 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc580ce7f0 con 0x7fdc5815c450 2024-01-15T18:13:35.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.582+0000 7fdc5d000640 1 --2- 172.21.15.2:0/3388562848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc5806c8f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009e50 tx=0x7fdc4002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d348bd174be5752 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:35.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.583+0000 7fdc57e00640 1 -- 172.21.15.2:0/3388562848 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4002cb00 con 0x7fdc5815c450 2024-01-15T18:13:35.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.583+0000 7fdc57e00640 1 -- 172.21.15.2:0/3388562848 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4002cc60 con 0x7fdc5815c450 2024-01-15T18:13:35.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.583+0000 7fdc57e00640 1 -- 172.21.15.2:0/3388562848 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc400343f0 con 0x7fdc5815c450 2024-01-15T18:13:35.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.583+0000 7fdc60400640 1 -- 172.21.15.2:0/3388562848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 msgr2=0x7fdc5806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.583+0000 7fdc60400640 1 --2- 172.21.15.2:0/3388562848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc5806c8f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009e50 tx=0x7fdc4002bf60 comp rx=0 tx=0).stop 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.584+0000 7fdc60400640 1 -- 172.21.15.2:0/3388562848 shutdown_connections 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.584+0000 7fdc60400640 1 --2- 172.21.15.2:0/3388562848 >> v2:172.21.15.137:3300/0 conn(0x7fdc5806ce30 0x7fdc58070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.584+0000 7fdc60400640 1 --2- 172.21.15.2:0/3388562848 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc5806c8f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.584+0000 7fdc60400640 1 -- 172.21.15.2:0/3388562848 >> 172.21.15.2:0/3388562848 conn(0x7fdc580b45f0 msgr2=0x7fdc580b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.584+0000 7fdc60400640 1 -- 172.21.15.2:0/3388562848 shutdown_connections 2024-01-15T18:13:35.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.584+0000 7fdc60400640 1 -- 172.21.15.2:0/3388562848 wait complete. 2024-01-15T18:13:35.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.585+0000 7fdc60400640 1 Processor -- start 2024-01-15T18:13:35.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.586+0000 7fdc60400640 1 -- start start 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.586+0000 7fdc60400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 0x7fdc580b3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.586+0000 7fdc60400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc58144c00 0x7fdc581388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.586+0000 7fdc60400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc581f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc60400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc580b0da0 con 0x7fdc5806ce30 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc60400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc58138df0 con 0x7fdc58144c00 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc60400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5813be30 con 0x7fdc5815c450 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc5da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 0x7fdc580b3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:35.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc5da00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 0x7fdc580b3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44372/0 (socket says 172.21.15.2:44372) 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc5da00640 1 -- 172.21.15.2:0/1065832269 learned_addr learned my addr 172.21.15.2:0/1065832269 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc5e400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc581f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.587+0000 7fdc5d000640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc58144c00 0x7fdc581388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5da00640 1 -- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 msgr2=0x7fdc581f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5da00640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc581f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5da00640 1 -- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc58144c00 msgr2=0x7fdc581388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5da00640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc58144c00 0x7fdc581388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5da00640 1 -- 172.21.15.2:0/1065832269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40009b30 con 0x7fdc5806ce30 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5e400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc581f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc5da00640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 0x7fdc580b3ba0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fdc4800c480 tx=0x7fdc4800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc480052d0 con 0x7fdc5806ce30 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc48005430 con 0x7fdc5806ce30 2024-01-15T18:13:35.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc580b0ff0 con 0x7fdc5806ce30 2024-01-15T18:13:35.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc48012870 con 0x7fdc5806ce30 2024-01-15T18:13:35.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.588+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc580b15b0 con 0x7fdc5806ce30 2024-01-15T18:13:35.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.589+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc580686f0 con 0x7fdc5806ce30 2024-01-15T18:13:35.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.591+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 35) v1 ==== 90967+0+0 (secure 0 0 0) 0x7fdc480129d0 con 0x7fdc5806ce30 2024-01-15T18:13:35.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.592+0000 7fdc55600640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc3006c280 0x7fdc3006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:35.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.592+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fdc480882d0 con 0x7fdc5806ce30 2024-01-15T18:13:35.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.592+0000 7fdc5d000640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc3006c280 0x7fdc3006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:35.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.593+0000 7fdc5d000640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc3006c280 0x7fdc3006e740 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdc40035000 tx=0x7fdc40030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:35.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.597+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc48054b90 con 0x7fdc5806ce30 2024-01-15T18:13:35.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.829+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc580adb30 con 0x7fdc3006c280 2024-01-15T18:13:35.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.862+0000 7fdc55600640 1 -- 172.21.15.2:0/1065832269 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdc580adb30 con 0x7fdc3006c280 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc3006c280 msgr2=0x7fdc3006e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc3006c280 0x7fdc3006e740 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fdc40035000 tx=0x7fdc40030000 comp rx=0 tx=0).stop 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 msgr2=0x7fdc580b3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 0x7fdc580b3ba0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7fdc4800c480 tx=0x7fdc4800c940 comp rx=0 tx=0).stop 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 shutdown_connections 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc3006c280 0x7fdc3006e740 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc5815c450 0x7fdc581f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc58144c00 0x7fdc581388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 --2- 172.21.15.2:0/1065832269 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc5806ce30 0x7fdc580b3ba0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:35.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 >> 172.21.15.2:0/1065832269 conn(0x7fdc580b45f0 msgr2=0x7fdc580714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:35.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 shutdown_connections 2024-01-15T18:13:35.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:35.865+0000 7fdc60400640 1 -- 172.21.15.2:0/1065832269 wait complete. 2024-01-15T18:13:35.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:13:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.109+0000 7f7a1fe00640 1 Processor -- start 2024-01-15T18:13:36.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.109+0000 7f7a1fe00640 1 -- start start 2024-01-15T18:13:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a1815c450 0x7f7a1806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7a1806ce30 0x7f7a18070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a180ce230 con 0x7f7a18144c00 2024-01-15T18:13:36.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1fe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1806d2c0 con 0x7f7a1806ce30 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a180ce510 con 0x7f7a1815c450 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7a1806ce30 0x7f7a18070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7a1806ce30 0x7f7a18070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44374/0 (socket says 172.21.15.2:44374) 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.111+0000 7f7a1de00640 1 -- 172.21.15.2:0/2877307165 learned_addr learned my addr 172.21.15.2:0/2877307165 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.112+0000 7f7a1ca00640 1 --2- 172.21.15.2:0/2877307165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a1815c450 0x7f7a1806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.112+0000 7f7a1de00640 1 -- 172.21.15.2:0/2877307165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a1815c450 msgr2=0x7f7a1806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.112+0000 7f7a1de00640 1 --2- 172.21.15.2:0/2877307165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a1815c450 0x7f7a1806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.112+0000 7f7a1de00640 1 -- 172.21.15.2:0/2877307165 >> v1:172.21.15.137:6789/0 conn(0x7f7a18144c00 legacy=0x7f7a180cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:36.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.112+0000 7f7a1de00640 1 -- 172.21.15.2:0/2877307165 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a180ce7f0 con 0x7f7a1806ce30 2024-01-15T18:13:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.113+0000 7f7a1de00640 1 --2- 172.21.15.2:0/2877307165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a18070b10 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009e50 tx=0x7f7a1002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f750610c9ee942ee server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.114+0000 7f7a17e00640 1 -- 172.21.15.2:0/2877307165 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a1002ca70 con 0x7f7a1806ce30 2024-01-15T18:13:36.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.114+0000 7f7a17e00640 1 -- 172.21.15.2:0/2877307165 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1002cbd0 con 0x7f7a1806ce30 2024-01-15T18:13:36.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.114+0000 7f7a17e00640 1 -- 172.21.15.2:0/2877307165 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100343f0 con 0x7f7a1806ce30 2024-01-15T18:13:36.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.114+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2877307165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 msgr2=0x7f7a18070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.114+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2877307165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a18070b10 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7a10009e50 tx=0x7f7a1002bed0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.115+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2877307165 shutdown_connections 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.115+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2877307165 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a18070b10 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.115+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2877307165 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a1815c450 0x7f7a1806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.115+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2877307165 >> 172.21.15.2:0/2877307165 conn(0x7f7a180b45f0 msgr2=0x7f7a180b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.115+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2877307165 shutdown_connections 2024-01-15T18:13:36.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.115+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2877307165 wait complete. 2024-01-15T18:13:36.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.116+0000 7f7a1fe00640 1 Processor -- start 2024-01-15T18:13:36.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.117+0000 7f7a1fe00640 1 -- start start 2024-01-15T18:13:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a180aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a18144c00 0x7f7a18138840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a1815c450 0x7f7a1813c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a180b1ff0 con 0x7f7a1806ce30 2024-01-15T18:13:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a180af450 con 0x7f7a1815c450 2024-01-15T18:13:36.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a180af730 con 0x7f7a18144c00 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a180aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1d400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a180aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44376/0 (socket says 172.21.15.2:44376) 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1d400640 1 -- 172.21.15.2:0/2903873798 learned_addr learned my addr 172.21.15.2:0/2903873798 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1de00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a1815c450 0x7f7a1813c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.118+0000 7f7a1ca00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a18144c00 0x7f7a18138840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1d400640 1 -- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a18144c00 msgr2=0x7f7a18138840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1d400640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a18144c00 0x7f7a18138840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1d400640 1 -- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a1815c450 msgr2=0x7f7a1813c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1d400640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a1815c450 0x7f7a1813c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1d400640 1 -- 172.21.15.2:0/2903873798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10009b30 con 0x7f7a1806ce30 2024-01-15T18:13:36.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1ca00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a18144c00 0x7f7a18138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:36.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1d400640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a180aeb20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800c430 tx=0x7f7a0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:36.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08011790 con 0x7f7a1806ce30 2024-01-15T18:13:36.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a080118f0 con 0x7f7a1806ce30 2024-01-15T18:13:36.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a08011ce0 con 0x7f7a1806ce30 2024-01-15T18:13:36.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a180b2240 con 0x7f7a1806ce30 2024-01-15T18:13:36.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.119+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a180af060 con 0x7f7a1806ce30 2024-01-15T18:13:36.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.121+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a180af1f0 con 0x7f7a1806ce30 2024-01-15T18:13:36.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.122+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 35) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f7a0801c460 con 0x7f7a1806ce30 2024-01-15T18:13:36.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.123+0000 7f7a15600640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79f006c280 0x7f79f006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:36.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.123+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f7a080886d0 con 0x7f7a1806ce30 2024-01-15T18:13:36.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.123+0000 7f7a1ca00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79f006c280 0x7f79f006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:36.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.124+0000 7f7a1ca00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79f006c280 0x7f79f006e740 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006fd0 tx=0x7f7a00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:36.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.127+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a08054f90 con 0x7f7a1806ce30 2024-01-15T18:13:36.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.359+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a180ae990 con 0x7f79f006c280 2024-01-15T18:13:36.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.373+0000 7f7a15600640 1 -- 172.21.15.2:0/2903873798 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f7a180ae990 con 0x7f79f006c280 2024-01-15T18:13:36.376 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79f006c280 msgr2=0x7f79f006e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79f006c280 0x7f79f006e740 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006fd0 tx=0x7f7a00008040 comp rx=0 tx=0).stop 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 msgr2=0x7f7a180aeb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a180aeb20 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f7a0800c430 tx=0x7f7a0800c8f0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 shutdown_connections 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79f006c280 0x7f79f006e740 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a1815c450 0x7f7a1813c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a18144c00 0x7f7a18138840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 --2- 172.21.15.2:0/2903873798 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a1806ce30 0x7f7a180aeb20 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 >> 172.21.15.2:0/2903873798 conn(0x7f7a180b45f0 msgr2=0x7f7a180cf5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:36.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 shutdown_connections 2024-01-15T18:13:36.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:36.376+0000 7f7a1fe00640 1 -- 172.21.15.2:0/2903873798 wait complete. 2024-01-15T18:13:36.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:13:36.399 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392) 2024-01-15T18:13:36.464 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392 2024-01-15T18:13:36.464 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3138256896" 2024-01-15T18:13:36.464 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:13:36.465 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-15T18:13:38.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:13:38.466 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392) 2024-01-15T18:13:38.536 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/b9845208-d3c2-4b30-957e-a9dfbe467392 2024-01-15T18:13:38.536 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:13:38.537 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:13:38.538 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:13:38.538 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-15T18:13:38.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.766+0000 7f4441e00640 1 Processor -- start 2024-01-15T18:13:38.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.766+0000 7f4441e00640 1 -- start start 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.767+0000 7f4441e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f443c161890 0x7f443c160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f443c1611a0 0x7f443c06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c1615f0 con 0x7f443c15d270 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c06c5e0 con 0x7f443c1611a0 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c06c8c0 con 0x7f443c161890 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f443c1611a0 0x7f443c06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4440a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f443c15d270 0x7f443c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:50964/0 (socket says 172.21.15.2:50964) 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4440a00640 1 -- 172.21.15.2:0/4201702973 learned_addr learned my addr 172.21.15.2:0/4201702973 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:13:38.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4433e00640 1 --2- 172.21.15.2:0/4201702973 >> v2:172.21.15.137:3300/0 conn(0x7f443c161890 0x7f443c160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:38.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441400640 1 -- 172.21.15.2:0/4201702973 >> v2:172.21.15.137:3300/0 conn(0x7f443c161890 msgr2=0x7f443c160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:38.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441400640 1 --2- 172.21.15.2:0/4201702973 >> v2:172.21.15.137:3300/0 conn(0x7f443c161890 0x7f443c160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:38.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f4441400640 1 -- 172.21.15.2:0/4201702973 >> v1:172.21.15.2:6789/0 conn(0x7f443c15d270 legacy=0x7f443c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:38.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.769+0000 7f4441400640 1 -- 172.21.15.2:0/4201702973 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c071c00 con 0x7f443c1611a0 2024-01-15T18:13:38.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.768+0000 7f443be00640 1 -- 172.21.15.2:0/4201702973 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104846455 0 0) 0x7f442c001110 con 0x7f443c15d270 2024-01-15T18:13:38.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.770+0000 7f4441400640 1 --2- 172.21.15.2:0/4201702973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c1611a0 0x7f443c06cd50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f443c0ceba0 tx=0x7f443402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6cf3cd76425c71 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:38.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.770+0000 7f443be00640 1 -- 172.21.15.2:0/4201702973 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443402cb60 con 0x7f443c1611a0 2024-01-15T18:13:38.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.770+0000 7f443be00640 1 -- 172.21.15.2:0/4201702973 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443402ccc0 con 0x7f443c1611a0 2024-01-15T18:13:38.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.770+0000 7f443be00640 1 -- 172.21.15.2:0/4201702973 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434034660 con 0x7f443c1611a0 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.770+0000 7f4441e00640 1 -- 172.21.15.2:0/4201702973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c1611a0 msgr2=0x7f443c06cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 --2- 172.21.15.2:0/4201702973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c1611a0 0x7f443c06cd50 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f443c0ceba0 tx=0x7f443402bdb0 comp rx=0 tx=0).stop 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 -- 172.21.15.2:0/4201702973 shutdown_connections 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 --2- 172.21.15.2:0/4201702973 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c1611a0 0x7f443c06cd50 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 --2- 172.21.15.2:0/4201702973 >> v2:172.21.15.137:3300/0 conn(0x7f443c161890 0x7f443c160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 -- 172.21.15.2:0/4201702973 >> 172.21.15.2:0/4201702973 conn(0x7f443c0b0840 msgr2=0x7f443c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 -- 172.21.15.2:0/4201702973 shutdown_connections 2024-01-15T18:13:38.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.771+0000 7f4441e00640 1 -- 172.21.15.2:0/4201702973 wait complete. 2024-01-15T18:13:38.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.773+0000 7f4441e00640 1 Processor -- start 2024-01-15T18:13:38.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.773+0000 7f4441e00640 1 -- start start 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443c15d270 0x7f443c1f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f443c1611a0 0x7f443c0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 0x7f443c0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1f35e0 con 0x7f443c161890 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1f38c0 con 0x7f443c1611a0 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c157ad0 con 0x7f443c15d270 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 0x7f443c0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 0x7f443c0ae3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44392/0 (socket says 172.21.15.2:44392) 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4441400640 1 -- 172.21.15.2:0/3466532814 learned_addr learned my addr 172.21.15.2:0/3466532814 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.774+0000 7f4440a00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443c15d270 0x7f443c1f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4433e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f443c1611a0 0x7f443c0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4441400640 1 -- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443c15d270 msgr2=0x7f443c1f2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4441400640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443c15d270 0x7f443c1f2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:38.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4441400640 1 -- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f443c1611a0 msgr2=0x7f443c0ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4441400640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f443c1611a0 0x7f443c0ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4441400640 1 -- 172.21.15.2:0/3466532814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434009a70 con 0x7f443c161890 2024-01-15T18:13:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4433e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f443c1611a0 0x7f443c0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:38.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.775+0000 7f4441400640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 0x7f443c0ae3a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e80 tx=0x7f4434004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:38.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.777+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434040650 con 0x7f443c161890 2024-01-15T18:13:38.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.777+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44340407d0 con 0x7f443c161890 2024-01-15T18:13:38.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.777+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4434040b80 con 0x7f443c161890 2024-01-15T18:13:38.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.777+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c157db0 con 0x7f443c161890 2024-01-15T18:13:38.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.777+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c1f3080 con 0x7f443c161890 2024-01-15T18:13:38.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.779+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c157930 con 0x7f443c161890 2024-01-15T18:13:38.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.779+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 35) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f4434040ce0 con 0x7f443c161890 2024-01-15T18:13:38.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.780+0000 7f4439600640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f441406c2b0 0x7f441406e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:38.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.780+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f44340ab8c0 con 0x7f443c161890 2024-01-15T18:13:38.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.781+0000 7f4440a00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f441406c2b0 0x7f441406e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:38.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.782+0000 7f4440a00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f441406c2b0 0x7f441406e770 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f442c000f60 tx=0x7f442c0079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:38.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:38.785+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f443c157930 con 0x7f443c161890 2024-01-15T18:13:39.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.012+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f443c0c41a0 con 0x7f441406c2b0 2024-01-15T18:13:39.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.172+0000 7f4439600640 1 -- 172.21.15.2:0/3466532814 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f443c0c41a0 con 0x7f441406c2b0 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f441406c2b0 msgr2=0x7f441406e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f441406c2b0 0x7f441406e770 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f442c000f60 tx=0x7f442c0079b0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 msgr2=0x7f443c0ae3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 0x7f443c0ae3a0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e80 tx=0x7f4434004a40 comp rx=0 tx=0).stop 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 shutdown_connections 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f441406c2b0 0x7f441406e770 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f443c161890 0x7f443c0ae3a0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f443c1611a0 0x7f443c0ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 --2- 172.21.15.2:0/3466532814 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f443c15d270 0x7f443c1f2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 >> 172.21.15.2:0/3466532814 conn(0x7f443c0b0840 msgr2=0x7f443c06d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:39.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 shutdown_connections 2024-01-15T18:13:39.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.175+0000 7f4441e00640 1 -- 172.21.15.2:0/3466532814 wait complete. 2024-01-15T18:13:39.182 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.179+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.183 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.179+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.183 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.179+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.183 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.179+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.183 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.179+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.201 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:13:39.202 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:13:39.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.202+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.202+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.202+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.202+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.205 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:39.202+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:39.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.429+0000 7f9b1f200640 1 Processor -- start 2024-01-15T18:13:39.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.429+0000 7f9b1f200640 1 -- start start 2024-01-15T18:13:39.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1f200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9b18161890 0x7f9b18160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1f200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9b181611a0 0x7f9b1806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1f200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b181615f0 con 0x7f9b1815d270 2024-01-15T18:13:39.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1f200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1806c5e0 con 0x7f9b18161890 2024-01-15T18:13:39.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1f200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1806c8c0 con 0x7f9b181611a0 2024-01-15T18:13:39.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9b18161890 0x7f9b18160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1d400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9b18161890 0x7f9b18160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44406/0 (socket says 172.21.15.2:44406) 2024-01-15T18:13:39.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.431+0000 7f9b1d400640 1 -- 172.21.15.2:0/3513130367 learned_addr learned my addr 172.21.15.2:0/3513130367 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:39.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.432+0000 7f9b1d400640 1 -- 172.21.15.2:0/3513130367 >> v2:172.21.15.137:3300/0 conn(0x7f9b181611a0 msgr2=0x7f9b1806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:39.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.432+0000 7f9b1d400640 1 --2- 172.21.15.2:0/3513130367 >> v2:172.21.15.137:3300/0 conn(0x7f9b181611a0 0x7f9b1806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.432+0000 7f9b1d400640 1 -- 172.21.15.2:0/3513130367 >> v1:172.21.15.137:6789/0 conn(0x7f9b1815d270 legacy=0x7f9b180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:39.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.432+0000 7f9b1d400640 1 -- 172.21.15.2:0/3513130367 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b18071c00 con 0x7f9b18161890 2024-01-15T18:13:39.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.433+0000 7f9b1d400640 1 --2- 172.21.15.2:0/3513130367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b18160c60 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f9b1806fed0 tx=0x7f9b0002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb269c60bb43c8e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:39.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.433+0000 7f9b1ca00640 1 -- 172.21.15.2:0/3513130367 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00031750 con 0x7f9b18161890 2024-01-15T18:13:39.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1ca00640 1 -- 172.21.15.2:0/3513130367 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b000318b0 con 0x7f9b18161890 2024-01-15T18:13:39.436 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1ca00640 1 -- 172.21.15.2:0/3513130367 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b00031c40 con 0x7f9b18161890 2024-01-15T18:13:39.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1f200640 1 -- 172.21.15.2:0/3513130367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 msgr2=0x7f9b18160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3513130367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b18160c60 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f9b1806fed0 tx=0x7f9b0002b7f0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1f200640 1 -- 172.21.15.2:0/3513130367 shutdown_connections 2024-01-15T18:13:39.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3513130367 >> v2:172.21.15.137:3300/0 conn(0x7f9b181611a0 0x7f9b1806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3513130367 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b18160c60 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.434+0000 7f9b1f200640 1 -- 172.21.15.2:0/3513130367 >> 172.21.15.2:0/3513130367 conn(0x7f9b180b0840 msgr2=0x7f9b180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:39.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.435+0000 7f9b1f200640 1 -- 172.21.15.2:0/3513130367 shutdown_connections 2024-01-15T18:13:39.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.435+0000 7f9b1f200640 1 -- 172.21.15.2:0/3513130367 wait complete. 2024-01-15T18:13:39.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.437+0000 7f9b1f200640 1 Processor -- start 2024-01-15T18:13:39.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.437+0000 7f9b1f200640 1 -- start start 2024-01-15T18:13:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1f200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b1815d270 0x7f9b1815bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1f200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181611a0 0x7f9b181413a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b180ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1f200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180ac870 con 0x7f9b18161890 2024-01-15T18:13:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1f200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180af4f0 con 0x7f9b181611a0 2024-01-15T18:13:39.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1f200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b180acda0 con 0x7f9b1815d270 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b180ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b180ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44414/0 (socket says 172.21.15.2:44414) 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1e800640 1 -- 172.21.15.2:0/3968053804 learned_addr learned my addr 172.21.15.2:0/3968053804 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1d400640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181611a0 0x7f9b181413a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1e800640 1 -- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b1815d270 msgr2=0x7f9b1815bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.438+0000 7f9b1e800640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b1815d270 0x7f9b1815bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1e800640 1 -- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181611a0 msgr2=0x7f9b181413a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1e800640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181611a0 0x7f9b181413a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1e800640 1 -- 172.21.15.2:0/3968053804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b000094d0 con 0x7f9b18161890 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1d400640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181611a0 0x7f9b181413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1e800640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b180ccae0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000b970 tx=0x7f9b1000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1000cb40 con 0x7f9b18161890 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b1000ccc0 con 0x7f9b18161890 2024-01-15T18:13:39.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b180acac0 con 0x7f9b18161890 2024-01-15T18:13:39.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b10014680 con 0x7f9b18161890 2024-01-15T18:13:39.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.439+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b180afa70 con 0x7f9b18161890 2024-01-15T18:13:39.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.440+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b18074770 con 0x7f9b18161890 2024-01-15T18:13:39.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.441+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 35) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f9b1001d3b0 con 0x7f9b18161890 2024-01-15T18:13:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.442+0000 7f9b16000640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006c280 0x7f9af006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.443+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f9b100889b0 con 0x7f9b18161890 2024-01-15T18:13:39.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.443+0000 7f9b1de00640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006c280 0x7f9af006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.444+0000 7f9b1de00640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006c280 0x7f9af006e740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9b08003cc0 tx=0x7f9b080079b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:39.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.447+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b10054f60 con 0x7f9b18161890 2024-01-15T18:13:39.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.678+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9b180b2d30 con 0x7f9af006c280 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.696+0000 7f9b16000640 1 -- 172.21.15.2:0/3968053804 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f9b08003950 con 0x7f9af006c280 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:39.699 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:13:39.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.698+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006c280 msgr2=0x7f9af006e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.698+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006c280 0x7f9af006e740 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9b08003cc0 tx=0x7f9b080079b0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.698+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 msgr2=0x7f9b180ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.698+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b180ccae0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f9b1000b970 tx=0x7f9b1000be30 comp rx=0 tx=0).stop 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 shutdown_connections 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9af006c280 0x7f9af006e740 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9b18161890 0x7f9b180ccae0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b181611a0 0x7f9b181413a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 --2- 172.21.15.2:0/3968053804 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b1815d270 0x7f9b1815bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 >> 172.21.15.2:0/3968053804 conn(0x7f9b180b0840 msgr2=0x7f9b18157930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 shutdown_connections 2024-01-15T18:13:39.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.699+0000 7f9b1f200640 1 -- 172.21.15.2:0/3968053804 wait complete. 2024-01-15T18:13:39.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-01-15T18:13:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.947+0000 7f70df000640 1 Processor -- start 2024-01-15T18:13:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.948+0000 7f70df000640 1 -- start start 2024-01-15T18:13:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.949+0000 7f70df000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f70d8144c00 0x7f70d80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.949+0000 7f70df000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d80cdd70 0x7f70d806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.949+0000 7f70df000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d80ce230 con 0x7f70d815c450 2024-01-15T18:13:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.949+0000 7f70df000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d80ce510 con 0x7f70d8144c00 2024-01-15T18:13:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.949+0000 7f70df000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d80ce7f0 con 0x7f70d80cdd70 2024-01-15T18:13:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.950+0000 7f70d7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d80cdd70 0x7f70d806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.950+0000 7f70d7400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d80cdd70 0x7f70d806c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:44854/0 (socket says 172.21.15.2:44854) 2024-01-15T18:13:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.950+0000 7f70d7e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f70d8144c00 0x7f70d80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.950+0000 7f70d7400640 1 -- 172.21.15.2:0/2061387864 learned_addr learned my addr 172.21.15.2:0/2061387864 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.951+0000 7f70d7e00640 1 -- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d80cdd70 msgr2=0x7f70d806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.951+0000 7f70d7e00640 1 --2- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d80cdd70 0x7f70d806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.951+0000 7f70d7e00640 1 -- 172.21.15.2:0/2061387864 >> v1:172.21.15.137:6789/0 conn(0x7f70d815c450 legacy=0x7f70d80b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.951+0000 7f70d7e00640 1 -- 172.21.15.2:0/2061387864 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d80705f0 con 0x7f70d8144c00 2024-01-15T18:13:39.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.953+0000 7f70d7e00640 1 --2- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d8144c00 0x7f70d80cd830 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f70c80097a0 tx=0x7f70c802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d022150f9aa46af6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.953+0000 7f70d6a00640 1 -- 172.21.15.2:0/2061387864 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c802d040 con 0x7f70d8144c00 2024-01-15T18:13:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.953+0000 7f70d6a00640 1 -- 172.21.15.2:0/2061387864 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c8031810 con 0x7f70d8144c00 2024-01-15T18:13:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.953+0000 7f70d6a00640 1 -- 172.21.15.2:0/2061387864 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8031ba0 con 0x7f70d8144c00 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 -- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d8144c00 msgr2=0x7f70d80cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 --2- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d8144c00 0x7f70d80cd830 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f70c80097a0 tx=0x7f70c802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 -- 172.21.15.2:0/2061387864 shutdown_connections 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 --2- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d80cdd70 0x7f70d806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 --2- 172.21.15.2:0/2061387864 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d8144c00 0x7f70d80cd830 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 -- 172.21.15.2:0/2061387864 >> 172.21.15.2:0/2061387864 conn(0x7f70d80b45f0 msgr2=0x7f70d80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 -- 172.21.15.2:0/2061387864 shutdown_connections 2024-01-15T18:13:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.954+0000 7f70df000640 1 -- 172.21.15.2:0/2061387864 wait complete. 2024-01-15T18:13:39.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.956+0000 7f70df000640 1 Processor -- start 2024-01-15T18:13:39.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.956+0000 7f70df000640 1 -- start start 2024-01-15T18:13:39.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70df000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 0x7f70d8158c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70df000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d8144c00 0x7f70d813b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70df000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d815c450 0x7f70d8157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70df000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81587b0 con 0x7f70d80cdd70 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70df000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8157ba0 con 0x7f70d815c450 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70df000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8157e80 con 0x7f70d8144c00 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 0x7f70d8158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70d7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 0x7f70d8158c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44430/0 (socket says 172.21.15.2:44430) 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70d7e00640 1 -- 172.21.15.2:0/3512082155 learned_addr learned my addr 172.21.15.2:0/3512082155 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70d7400640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d8144c00 0x7f70d813b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.957+0000 7f70d7e00640 1 -- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d8144c00 msgr2=0x7f70d813b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70d7e00640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d8144c00 0x7f70d813b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70d7e00640 1 -- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d815c450 msgr2=0x7f70d8157220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70d7e00640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d815c450 0x7f70d8157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70d7e00640 1 -- 172.21.15.2:0/3512082155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8009400 con 0x7f70d80cdd70 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70d7400640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d8144c00 0x7f70d813b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70d7e00640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 0x7f70d8158c40 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f70c80098c0 tx=0x7f70c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:39.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8006e10 con 0x7f70d80cdd70 2024-01-15T18:13:39.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c803ad60 con 0x7f70d80cdd70 2024-01-15T18:13:39.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8044620 con 0x7f70d80cdd70 2024-01-15T18:13:39.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d8158160 con 0x7f70d80cdd70 2024-01-15T18:13:39.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.958+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d8159180 con 0x7f70d80cdd70 2024-01-15T18:13:39.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.959+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d80686f0 con 0x7f70d80cdd70 2024-01-15T18:13:39.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.960+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 35) v1 ==== 90967+0+0 (secure 0 0 0) 0x7f70c804d440 con 0x7f70d80cdd70 2024-01-15T18:13:39.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.962+0000 7f70cfe00640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f70b006c280 0x7f70b006e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:39.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.962+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f70c80aae20 con 0x7f70d80cdd70 2024-01-15T18:13:39.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.963+0000 7f70d7400640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f70b006c280 0x7f70b006e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:39.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.964+0000 7f70d7400640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f70b006c280 0x7f70b006e740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f70d8141640 tx=0x7f70c000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:39.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:39.967+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70c8077340 con 0x7f70d80cdd70 2024-01-15T18:13:40.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:40.198+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f70d80cf270 con 0x7f70b006c280 2024-01-15T18:13:40.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:40.807+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f70c8077710 con 0x7f70d80cdd70 2024-01-15T18:13:51.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.974+0000 7f70cfe00640 1 -- 172.21.15.2:0/3512082155 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f70d80cf270 con 0x7f70b006c280 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.977+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f70b006c280 msgr2=0x7f70b006e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.977+0000 7f70df000640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f70b006c280 0x7f70b006e740 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f70d8141640 tx=0x7f70c000b040 comp rx=0 tx=0).stop 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 msgr2=0x7f70d8158c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 0x7f70d8158c40 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f70c80098c0 tx=0x7f70c80046c0 comp rx=0 tx=0).stop 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 shutdown_connections 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f70b006c280 0x7f70b006e740 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f70d815c450 0x7f70d8157220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f70d8144c00 0x7f70d813b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 --2- 172.21.15.2:0/3512082155 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f70d80cdd70 0x7f70d8158c40 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 >> 172.21.15.2:0/3512082155 conn(0x7f70d80b45f0 msgr2=0x7f70d8070a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 shutdown_connections 2024-01-15T18:13:51.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:51.978+0000 7f70df000640 1 -- 172.21.15.2:0/3512082155 wait complete. 2024-01-15T18:13:52.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-15T18:13:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.228+0000 7f421d800640 1 Processor -- start 2024-01-15T18:13:52.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.228+0000 7f421d800640 1 -- start start 2024-01-15T18:13:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.230+0000 7f421d800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f421806dd60 0x7f4218167dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.230+0000 7f421d800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421806c5e0 con 0x7f421815d270 2024-01-15T18:13:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.230+0000 7f421d800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f421806c8c0 con 0x7f4218161890 2024-01-15T18:13:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.230+0000 7f421d800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4218071c00 con 0x7f421806dd60 2024-01-15T18:13:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.230+0000 7f4217e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f421815d270 0x7f42180b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:54460/0 (socket says 172.21.15.2:54460) 2024-01-15T18:13:52.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.230+0000 7f4217e00640 1 -- 172.21.15.2:0/4152048492 learned_addr learned my addr 172.21.15.2:0/4152048492 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:13:52.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f421ce00640 1 --2- 172.21.15.2:0/4152048492 >> v2:172.21.15.137:3300/0 conn(0x7f421806dd60 0x7f4218167dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254248419 0 0) 0x7f421806c5e0 con 0x7f421815d270 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f8003520 con 0x7f421815d270 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903517019 0 0) 0x7f41f8003520 con 0x7f421815d270 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 >> v2:172.21.15.137:3300/0 conn(0x7f421806dd60 msgr2=0x7f4218167dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 --2- 172.21.15.2:0/4152048492 >> v2:172.21.15.137:3300/0 conn(0x7f421806dd60 0x7f4218167dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 >> v1:172.21.15.137:6789/0 conn(0x7f4218161890 legacy=0x7f4218160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218071ee0 con 0x7f421815d270 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4208001670 con 0x7f421815d270 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.231+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 >> v1:172.21.15.2:6789/0 conn(0x7f421815d270 legacy=0x7f42180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4216a00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41f80039c0 0x7f41f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4216a00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80047c0 0x7f41f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4216a00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41f80052a0 0x7f41f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4218071c00 con 0x7f41f80039c0 2024-01-15T18:13:52.237 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f800c1a0 con 0x7f41f80047c0 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f800c360 con 0x7f41f80052a0 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4217e00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41f80039c0 0x7f41f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4217e00640 1 -- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41f80052a0 msgr2=0x7f41f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f421ce00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41f80052a0 0x7f41f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4217e00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f41f80052a0 0x7f41f800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4217e00640 1 -- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80047c0 msgr2=0x7f41f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4217e00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80047c0 0x7f41f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f4217e00640 1 -- 172.21.15.2:0/4152048492 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8003520 con 0x7f41f80039c0 2024-01-15T18:13:52.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.232+0000 7f421ce00640 1 -- 172.21.15.2:0/4152048492 reap_dead start 2024-01-15T18:13:52.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f4217e00640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41f80039c0 0x7f41f8003de0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f420800c0e0 tx=0x7f41f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee2fa9433d8eb5c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:52.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208005ad0 con 0x7f41f80039c0 2024-01-15T18:13:52.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4208005c50 con 0x7f41f80039c0 2024-01-15T18:13:52.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f4216a00640 1 -- 172.21.15.2:0/4152048492 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208007070 con 0x7f41f80039c0 2024-01-15T18:13:52.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f421d800640 1 -- 172.21.15.2:0/4152048492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41f80039c0 msgr2=0x7f41f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:52.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f421d800640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41f80039c0 0x7f41f8003de0 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f420800c0e0 tx=0x7f41f80057f0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f421d800640 1 -- 172.21.15.2:0/4152048492 shutdown_connections 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.234+0000 7f421d800640 1 --2- 172.21.15.2:0/4152048492 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f41f80039c0 0x7f41f8003de0 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.235+0000 7f421d800640 1 -- 172.21.15.2:0/4152048492 >> 172.21.15.2:0/4152048492 conn(0x7f42180b0840 msgr2=0x7f42180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.235+0000 7f421d800640 1 -- 172.21.15.2:0/4152048492 shutdown_connections 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.235+0000 7f421d800640 1 -- 172.21.15.2:0/4152048492 wait complete. 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.236+0000 7f421d800640 1 Processor -- start 2024-01-15T18:13:52.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.237+0000 7f421d800640 1 -- start start 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.237+0000 7f421d800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80039c0 0x7f4218150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421d800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 0x7f4218146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421d800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42180ade10 0x7f42180ae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421d800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180ae2c0 con 0x7f42180ac870 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421d800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42180acd70 con 0x7f41f80039c0 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421d800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42180ad050 con 0x7f42180ade10 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 0x7f4218146750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 0x7f4218146750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:60552/0 (socket says 172.21.15.2:60552) 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 -- 172.21.15.2:0/628321709 learned_addr learned my addr 172.21.15.2:0/628321709 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217e00640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80039c0 0x7f4218150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421ce00640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42180ade10 0x7f42180ae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 -- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42180ade10 msgr2=0x7f42180ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:52.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42180ade10 0x7f42180ae9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 -- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80039c0 msgr2=0x7f4218150d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:52.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80039c0 0x7f4218150d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:52.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f4217400640 1 -- 172.21.15.2:0/628321709 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42180ad330 con 0x7f42180ac870 2024-01-15T18:13:52.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.238+0000 7f421ce00640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42180ade10 0x7f42180ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.239+0000 7f4217400640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 0x7f4218146750 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f420000c430 tx=0x7f420000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.239+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42000052d0 con 0x7f42180ac870 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.239+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4200005450 con 0x7f42180ac870 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.239+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42180ad610 con 0x7f42180ac870 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.239+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4200003830 con 0x7f42180ac870 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.239+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42180cd830 con 0x7f42180ac870 2024-01-15T18:13:52.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.240+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42180cd9f0 con 0x7f42180ac870 2024-01-15T18:13:52.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.241+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f4200003990 con 0x7f42180ac870 2024-01-15T18:13:52.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.242+0000 7f420fe00640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41f006c3d0 0x7f41f006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:52.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.242+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f4200088080 con 0x7f42180ac870 2024-01-15T18:13:52.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.243+0000 7f4217e00640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41f006c3d0 0x7f41f006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:52.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.244+0000 7f4217e00640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41f006c3d0 0x7f41f006e890 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4208034000 tx=0x7f4208009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:52.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.247+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4200054600 con 0x7f42180ac870 2024-01-15T18:13:52.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:52.476+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4218074770 con 0x7f41f006c3d0 2024-01-15T18:13:53.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.002+0000 7f420fe00640 1 -- 172.21.15.2:0/628321709 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f4218074770 con 0x7f41f006c3d0 2024-01-15T18:13:53.005 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "canceled", 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "failure": { 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "errno": "4", 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: "error_msg": "user interrupted clone operation" 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:53.006 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:13:53.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41f006c3d0 msgr2=0x7f41f006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41f006c3d0 0x7f41f006e890 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f4208034000 tx=0x7f4208009030 comp rx=0 tx=0).stop 2024-01-15T18:13:53.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 msgr2=0x7f4218146750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 0x7f4218146750 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f420000c430 tx=0x7f420000c8f0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 shutdown_connections 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f41f006c3d0 0x7f41f006e890 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f42180ade10 0x7f42180ae9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f42180ac870 0x7f4218146750 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 --2- 172.21.15.2:0/628321709 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f41f80039c0 0x7f4218150d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 >> 172.21.15.2:0/628321709 conn(0x7f42180b0840 msgr2=0x7f421815dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 shutdown_connections 2024-01-15T18:13:53.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.005+0000 7f421d800640 1 -- 172.21.15.2:0/628321709 wait complete. 2024-01-15T18:13:53.030 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-15T18:13:53.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.245+0000 7f507a800640 1 Processor -- start 2024-01-15T18:13:53.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.245+0000 7f507a800640 1 -- start start 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f507a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f507415c0e0 0x7f50740cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f507a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415b7d0 0x7f507406c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f507a800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507415cb00 con 0x7f507415c4c0 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f507a800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50740ccec0 con 0x7f507415c0e0 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f507a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50740cd1a0 con 0x7f507415b7d0 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f5073400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f507415c0e0 0x7f50740cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f5073400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f507415c0e0 0x7f50740cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49884/0 (socket says 172.21.15.2:49884) 2024-01-15T18:13:53.251 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.247+0000 7f5073400640 1 -- 172.21.15.2:0/4158741723 learned_addr learned my addr 172.21.15.2:0/4158741723 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f5073400640 1 -- 172.21.15.2:0/4158741723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415b7d0 msgr2=0x7f507406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f506be00640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415b7d0 0x7f507406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f5073400640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415b7d0 0x7f507406c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f5073400640 1 -- 172.21.15.2:0/4158741723 >> v1:172.21.15.2:6789/0 conn(0x7f507415c4c0 legacy=0x7f507415ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f5073400640 1 -- 172.21.15.2:0/4158741723 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50740cd480 con 0x7f507415c0e0 2024-01-15T18:13:53.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f5072a00640 1 -- 172.21.15.2:0/4158741723 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273749838 0 0) 0x7f507415cb00 con 0x7f507415c4c0 2024-01-15T18:13:53.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.248+0000 7f506be00640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415b7d0 0x7f507406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.249+0000 7f5073400640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415c0e0 0x7f50740cc5c0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f5064009870 tx=0x7f506402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=727d2a3724cd6c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.249+0000 7f5072a00640 1 -- 172.21.15.2:0/4158741723 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064030840 con 0x7f507415c0e0 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.249+0000 7f5072a00640 1 -- 172.21.15.2:0/4158741723 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50640309a0 con 0x7f507415c0e0 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.249+0000 7f5072a00640 1 -- 172.21.15.2:0/4158741723 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064030d30 con 0x7f507415c0e0 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 -- 172.21.15.2:0/4158741723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415c0e0 msgr2=0x7f50740cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415c0e0 0x7f50740cc5c0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f5064009870 tx=0x7f506402b980 comp rx=0 tx=0).stop 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 -- 172.21.15.2:0/4158741723 shutdown_connections 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415b7d0 0x7f507406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 --2- 172.21.15.2:0/4158741723 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415c0e0 0x7f50740cc5c0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 -- 172.21.15.2:0/4158741723 >> 172.21.15.2:0/4158741723 conn(0x7f507406da20 msgr2=0x7f507406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:53.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 -- 172.21.15.2:0/4158741723 shutdown_connections 2024-01-15T18:13:53.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.250+0000 7f507a800640 1 -- 172.21.15.2:0/4158741723 wait complete. 2024-01-15T18:13:53.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.252+0000 7f507a800640 1 Processor -- start 2024-01-15T18:13:53.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.252+0000 7f507a800640 1 -- start start 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f507a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 0x7f5074155140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f507a800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507415c0e0 0x7f507415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f507a800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415c4c0 0x7f507413e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f507a800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507415fb80 con 0x7f507415b7d0 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f507a800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5074147910 con 0x7f507415c0e0 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f507a800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50741f64f0 con 0x7f507415c4c0 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f5073400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 0x7f5074155140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f5073400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 0x7f5074155140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49890/0 (socket says 172.21.15.2:49890) 2024-01-15T18:13:53.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.253+0000 7f5073400640 1 -- 172.21.15.2:0/3244678873 learned_addr learned my addr 172.21.15.2:0/3244678873 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073e00640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415c4c0 0x7f507413e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073400640 1 -- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415c4c0 msgr2=0x7f507413e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f506be00640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507415c0e0 0x7f507415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073400640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415c4c0 0x7f507413e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073400640 1 -- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507415c0e0 msgr2=0x7f507415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073400640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507415c0e0 0x7f507415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073400640 1 -- 172.21.15.2:0/3244678873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640094d0 con 0x7f507415b7d0 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073e00640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415c4c0 0x7f507413e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f5073400640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 0x7f5074155140 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5064000c00 tx=0x7f5064004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640307f0 con 0x7f507415b7d0 2024-01-15T18:13:53.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5064043400 con 0x7f507415b7d0 2024-01-15T18:13:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640437b0 con 0x7f507415b7d0 2024-01-15T18:13:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.254+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50740704b0 con 0x7f507415b7d0 2024-01-15T18:13:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.255+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5074070a70 con 0x7f507415b7d0 2024-01-15T18:13:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.256+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50740686f0 con 0x7f507415b7d0 2024-01-15T18:13:53.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.257+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f5064043910 con 0x7f507415b7d0 2024-01-15T18:13:53.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.257+0000 7f506b400640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06c3a0 0x7f504c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.258+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f506403e070 con 0x7f507415b7d0 2024-01-15T18:13:53.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.258+0000 7f506be00640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06c3a0 0x7f504c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.259+0000 7f506be00640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06c3a0 0x7f504c06e860 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5074151cb0 tx=0x7f505c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:53.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.262+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50640783d0 con 0x7f507415b7d0 2024-01-15T18:13:53.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.521+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f507406b260 con 0x7f507415b7d0 2024-01-15T18:13:53.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.522+0000 7f506b400640 1 -- 172.21.15.2:0/3244678873 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1856 (secure 0 0 0) 0x7f5064034750 con 0x7f507415b7d0 2024-01-15T18:13:53.526 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:13:53.526 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:13:53.526 INFO:teuthology.orchestra.run.smithi002.stdout: "fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:13:53.526 INFO:teuthology.orchestra.run.smithi002.stdout: "health": { 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "HEALTH_OK", 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "checks": {}, 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "mutes": [] 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "election_epoch": 4, 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum": [ 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: 0, 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: 1, 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: 2 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_names": [ 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "a", 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "b", 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: "c" 2024-01-15T18:13:53.527 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "quorum_age": 1001, 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "monmap": { 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "min_mon_release_name": "reef", 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "num_mons": 3 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap": { 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 103, 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "num_osds": 8, 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "num_up_osds": 8, 2024-01-15T18:13:53.528 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_up_since": 1705341443, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "num_in_osds": 8, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_in_since": 1705341435, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "num_remapped_pgs": 0 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "pgmap": { 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "pgs_by_state": [ 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "state_name": "active+clean", 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "count": 97 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pgs": 97, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "num_pools": 3, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "num_objects": 797, 2024-01-15T18:13:53.529 INFO:teuthology.orchestra.run.smithi002.stdout: "data_bytes": 3222115648, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_used": 6995156992, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_avail": 766098956288, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "bytes_total": 773094113280, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "read_bytes_sec": 170, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "write_bytes_sec": 701640, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "read_op_per_sec": 0, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "write_op_per_sec": 1 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "fsmap": { 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 66, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 7, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "up": 1, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "in": 1, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "max": 1, 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "by_rank": [ 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:13:53.530 INFO:teuthology.orchestra.run.smithi002.stdout: "filesystem_id": 7, 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "rank": 0, 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "d", 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "status": "up:active", 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "gid": 8580 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "up:standby": 3 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "mgrmap": { 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "available": true, 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "num_standbys": 1, 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "modules": [ 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "iostat", 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "nfs", 2024-01-15T18:13:53.531 INFO:teuthology.orchestra.run.smithi002.stdout: "restful" 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: "servicemap": { 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 19, 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: "modified": "2024-01-15T18:13:37.789675+0000", 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: "services": {} 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout: "progress_events": {} 2024-01-15T18:13:53.532 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06c3a0 msgr2=0x7f504c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06c3a0 0x7f504c06e860 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f5074151cb0 tx=0x7f505c008040 comp rx=0 tx=0).stop 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 msgr2=0x7f5074155140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 0x7f5074155140 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5064000c00 tx=0x7f5064004730 comp rx=0 tx=0).stop 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 shutdown_connections 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f504c06c3a0 0x7f504c06e860 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f507415c4c0 0x7f507413e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.533 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f507415c0e0 0x7f507415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 --2- 172.21.15.2:0/3244678873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f507415b7d0 0x7f5074155140 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 >> 172.21.15.2:0/3244678873 conn(0x7f507406da20 msgr2=0x7f507406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 shutdown_connections 2024-01-15T18:13:53.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.525+0000 7f507a800640 1 -- 172.21.15.2:0/3244678873 wait complete. 2024-01-15T18:13:53.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:13:53.545 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/b9845208-d3c2-4b30-957e-a9dfbe467392) 2024-01-15T18:13:53.601 INFO:teuthology.orchestra.run.smithi002.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/b9845208-d3c2-4b30-957e-a9dfbe467392': No such file or directory 2024-01-15T18:13:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-15T18:13:53.603 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:13:53.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.818+0000 7fd75b000640 1 Processor -- start 2024-01-15T18:13:53.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.819+0000 7fd75b000640 1 -- start start 2024-01-15T18:13:53.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.820+0000 7fd75b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 0x7fd7540cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.820+0000 7fd75b000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540b35a0 con 0x7fd7540b30a0 2024-01-15T18:13:53.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.820+0000 7fd75b000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7540b3880 con 0x7fd75406cfc0 2024-01-15T18:13:53.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.820+0000 7fd75b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540b4220 con 0x7fd754144b80 2024-01-15T18:13:53.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.821+0000 7fd74be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd7540b30a0 0x7fd75406c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57480/0 (socket says 172.21.15.2:57480) 2024-01-15T18:13:53.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.821+0000 7fd74be00640 1 -- 172.21.15.2:0/439228896 learned_addr learned my addr 172.21.15.2:0/439228896 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:13:53.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.821+0000 7fd753e00640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 0x7fd7540cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.821+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145770095 0 0) 0x7fd7540b35a0 con 0x7fd7540b30a0 2024-01-15T18:13:53.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.821+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd734003520 con 0x7fd7540b30a0 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 231539513 0 0) 0x7fd734003520 con 0x7fd7540b30a0 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 msgr2=0x7fd7540cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 0x7fd7540cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 >> v1:172.21.15.137:6789/0 conn(0x7fd75406cfc0 legacy=0x7fd75415b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540b3b60 con 0x7fd7540b30a0 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd73c001670 con 0x7fd7540b30a0 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 >> v1:172.21.15.2:6789/0 conn(0x7fd7540b30a0 legacy=0x7fd75406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 0x7fd7340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540b4220 con 0x7fd734004d00 2024-01-15T18:13:53.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73400bb70 con 0x7fd734003f70 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73400bd30 con 0x7fd754144b80 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd759000640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd734005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.822+0000 7fd74be00640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.823+0000 7fd753e00640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 0x7fd7340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.823+0000 7fd759000640 1 -- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 msgr2=0x7fd7340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.823+0000 7fd759000640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd754144b80 0x7fd7340039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.823+0000 7fd759000640 1 -- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd734003f70 msgr2=0x7fd7340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.823+0000 7fd759000640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd734003f70 0x7fd7340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.823+0000 7fd759000640 1 -- 172.21.15.2:0/439228896 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd734003520 con 0x7fd734004d00 2024-01-15T18:13:53.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd759000640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd734005180 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fd7540701f0 tx=0x7fd7340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a47dd2fa6f57e1d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:53.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c003b70 con 0x7fd734004d00 2024-01-15T18:13:53.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c003cd0 con 0x7fd734004d00 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd753400640 1 -- 172.21.15.2:0/439228896 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c005310 con 0x7fd734004d00 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd75b000640 1 -- 172.21.15.2:0/439228896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 msgr2=0x7fd734005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd75b000640 1 --2- 172.21.15.2:0/439228896 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd734005180 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7fd7540701f0 tx=0x7fd7340057f0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.824+0000 7fd759000640 1 -- 172.21.15.2:0/439228896 reap_dead start 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.825+0000 7fd75b000640 1 -- 172.21.15.2:0/439228896 shutdown_connections 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.825+0000 7fd75b000640 1 -- 172.21.15.2:0/439228896 >> 172.21.15.2:0/439228896 conn(0x7fd7540b46d0 msgr2=0x7fd7540b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.825+0000 7fd75b000640 1 -- 172.21.15.2:0/439228896 shutdown_connections 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.825+0000 7fd75b000640 1 -- 172.21.15.2:0/439228896 wait complete. 2024-01-15T18:13:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.826+0000 7fd75b000640 1 Processor -- start 2024-01-15T18:13:53.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.827+0000 7fd75b000640 1 -- start start 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.827+0000 7fd75b000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd754140390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd75b000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7540aebf0 0x7fd7540c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd75b000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7540af070 0x7fd7540c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd75b000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540c3f40 con 0x7fd734004d00 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd75b000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540c4ac0 con 0x7fd7540aebf0 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd75b000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7541408d0 con 0x7fd7540af070 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd754140390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd754140390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49896/0 (socket says 172.21.15.2:49896) 2024-01-15T18:13:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 -- 172.21.15.2:0/1506292560 learned_addr learned my addr 172.21.15.2:0/1506292560 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 -- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7540af070 msgr2=0x7fd7540c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7540af070 0x7fd7540c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd74be00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7540aebf0 0x7fd7540c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 -- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7540aebf0 msgr2=0x7fd7540c4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7540aebf0 0x7fd7540c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.828+0000 7fd753e00640 1 -- 172.21.15.2:0/1506292560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540c4190 con 0x7fd734004d00 2024-01-15T18:13:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.829+0000 7fd753e00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd754140390 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd7440107e0 tx=0x7fd744010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.829+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744009610 con 0x7fd734004d00 2024-01-15T18:13:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.829+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74400eb00 con 0x7fd734004d00 2024-01-15T18:13:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.829+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74400ee90 con 0x7fd734004d00 2024-01-15T18:13:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.829+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7540c4470 con 0x7fd734004d00 2024-01-15T18:13:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.829+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7540c5c70 con 0x7fd734004d00 2024-01-15T18:13:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.830+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7540686f0 con 0x7fd734004d00 2024-01-15T18:13:53.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.831+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fd74401c400 con 0x7fd734004d00 2024-01-15T18:13:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.832+0000 7fd750c00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd72c06c3a0 0x7fd72c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.832+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7fd744088f50 con 0x7fd734004d00 2024-01-15T18:13:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.833+0000 7fd74be00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd72c06c3a0 0x7fd72c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:53.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.834+0000 7fd74be00640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd72c06c3a0 0x7fd72c06e860 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd73c001bb0 tx=0x7fd73c000f70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:53.837+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7440554d0 con 0x7fd734004d00 2024-01-15T18:13:54.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.058+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7540c4c00 con 0x7fd72c06c3a0 2024-01-15T18:13:54.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.062+0000 7fd750c00640 1 -- 172.21.15.2:0/1506292560 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd73c005e60 con 0x7fd72c06c3a0 2024-01-15T18:13:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:13:54.065 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:13:54.065 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:13:54.065 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:54.065 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:13:54.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd72c06c3a0 msgr2=0x7fd72c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd72c06c3a0 0x7fd72c06e860 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd73c001bb0 tx=0x7fd73c000f70 comp rx=0 tx=0).stop 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 msgr2=0x7fd754140390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd754140390 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7fd7440107e0 tx=0x7fd744010ca0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 shutdown_connections 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd72c06c3a0 0x7fd72c06e860 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7540af070 0x7fd7540c5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7540aebf0 0x7fd7540c4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 --2- 172.21.15.2:0/1506292560 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd734004d00 0x7fd754140390 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.064+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 >> 172.21.15.2:0/1506292560 conn(0x7fd7540b46d0 msgr2=0x7fd7540cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.065+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 shutdown_connections 2024-01-15T18:13:54.069 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.065+0000 7fd75b000640 1 -- 172.21.15.2:0/1506292560 wait complete. 2024-01-15T18:13:54.086 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:13:54.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.298+0000 7f5fb0400640 1 Processor -- start 2024-01-15T18:13:54.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.298+0000 7f5fb0400640 1 -- start start 2024-01-15T18:13:54.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fb0400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5fa815d4a0 0x7f5fa815d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fb0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806e890 0x7f5fa806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fb0400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fa81611c0 con 0x7f5fa80b3a50 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fb0400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fa815e920 con 0x7f5fa815d4a0 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fb0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa815ddc0 con 0x7f5fa806e890 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fada00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5fa815d4a0 0x7f5fa815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fada00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5fa815d4a0 0x7f5fa815d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49902/0 (socket says 172.21.15.2:49902) 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fad000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5fa80b3a50 0x7f5fa8161650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57494/0 (socket says 172.21.15.2:57494) 2024-01-15T18:13:54.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fada00640 1 -- 172.21.15.2:0/64556122 learned_addr learned my addr 172.21.15.2:0/64556122 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:54.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.300+0000 7f5fae400640 1 --2- 172.21.15.2:0/64556122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806e890 0x7f5fa806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.301+0000 7f5fada00640 1 -- 172.21.15.2:0/64556122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806e890 msgr2=0x7f5fa806ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:54.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.301+0000 7f5fada00640 1 --2- 172.21.15.2:0/64556122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806e890 0x7f5fa806ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.301+0000 7f5fada00640 1 -- 172.21.15.2:0/64556122 >> v1:172.21.15.2:6789/0 conn(0x7f5fa80b3a50 legacy=0x7f5fa8161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.301+0000 7f5fa7e00640 1 -- 172.21.15.2:0/64556122 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951111800 0 0) 0x7f5fa81611c0 con 0x7f5fa80b3a50 2024-01-15T18:13:54.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.301+0000 7f5fada00640 1 -- 172.21.15.2:0/64556122 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa815e080 con 0x7f5fa815d4a0 2024-01-15T18:13:54.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.302+0000 7f5fada00640 1 --2- 172.21.15.2:0/64556122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa815d4a0 0x7f5fa815d880 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009870 tx=0x7f5f9802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f92bda53776e461 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:54.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.302+0000 7f5fa7e00640 1 -- 172.21.15.2:0/64556122 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98031640 con 0x7f5fa815d4a0 2024-01-15T18:13:54.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.302+0000 7f5fa7e00640 1 -- 172.21.15.2:0/64556122 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f980317a0 con 0x7f5fa815d4a0 2024-01-15T18:13:54.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.302+0000 7f5fa7e00640 1 -- 172.21.15.2:0/64556122 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98031b30 con 0x7f5fa815d4a0 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 -- 172.21.15.2:0/64556122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa815d4a0 msgr2=0x7f5fa815d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 --2- 172.21.15.2:0/64556122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa815d4a0 0x7f5fa815d880 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f5f98009870 tx=0x7f5f9802b8f0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 -- 172.21.15.2:0/64556122 shutdown_connections 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 --2- 172.21.15.2:0/64556122 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa806e890 0x7f5fa806ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 --2- 172.21.15.2:0/64556122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa815d4a0 0x7f5fa815d880 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 -- 172.21.15.2:0/64556122 >> 172.21.15.2:0/64556122 conn(0x7f5fa80b1c70 msgr2=0x7f5fa80b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 -- 172.21.15.2:0/64556122 shutdown_connections 2024-01-15T18:13:54.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.303+0000 7f5fb0400640 1 -- 172.21.15.2:0/64556122 wait complete. 2024-01-15T18:13:54.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.305+0000 7f5fb0400640 1 Processor -- start 2024-01-15T18:13:54.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.305+0000 7f5fb0400640 1 -- start start 2024-01-15T18:13:54.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fb0400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa806e890 0x7f5fa8158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fb0400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 0x7f5fa80cea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fb0400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815d4a0 0x7f5fa80cef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fb0400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8159330 con 0x7f5fa80b3a50 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fb0400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ae6d0 con 0x7f5fa806e890 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fb0400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa80ae9b0 con 0x7f5fa815d4a0 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 0x7f5fa80cea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 0x7f5fa80cea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49904/0 (socket says 172.21.15.2:49904) 2024-01-15T18:13:54.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 -- 172.21.15.2:0/3929551354 learned_addr learned my addr 172.21.15.2:0/3929551354 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:54.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fae400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815d4a0 0x7f5fa80cef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fada00640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa806e890 0x7f5fa8158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 -- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815d4a0 msgr2=0x7f5fa80cef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815d4a0 0x7f5fa80cef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 -- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa806e890 msgr2=0x7f5fa8158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.306+0000 7f5fad000640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa806e890 0x7f5fa8158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fad000640 1 -- 172.21.15.2:0/3929551354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f980094d0 con 0x7f5fa80b3a50 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fad000640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 0x7f5fa80cea30 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c4d0 tx=0x7f5f9000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f900074e0 con 0x7f5fa80b3a50 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f90007660 con 0x7f5fa80b3a50 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90004a20 con 0x7f5fa80b3a50 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa80aec90 con 0x7f5fa80b3a50 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.307+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa80af250 con 0x7f5fa80b3a50 2024-01-15T18:13:54.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.308+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa80686f0 con 0x7f5fa80b3a50 2024-01-15T18:13:54.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.309+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f5f90004b80 con 0x7f5fa80b3a50 2024-01-15T18:13:54.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.310+0000 7f5fa5600640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f8006c3d0 0x7f5f8006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.310+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f5f90087920 con 0x7f5fa80b3a50 2024-01-15T18:13:54.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.311+0000 7f5fada00640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f8006c3d0 0x7f5f8006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.312+0000 7f5fada00640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f8006c3d0 0x7f5f8006e890 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5f98036000 tx=0x7f5f9803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:54.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.314+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f900541e0 con 0x7f5fa80b3a50 2024-01-15T18:13:54.544 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.540+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5fa815e640 con 0x7f5f8006c3d0 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.552+0000 7f5fa5600640 1 -- 172.21.15.2:0/3929551354 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f5f98031990 con 0x7f5f8006c3d0 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1clone1" 2024-01-15T18:13:54.556 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:54.557 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:13:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f8006c3d0 msgr2=0x7f5f8006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f8006c3d0 0x7f5f8006e890 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f5f98036000 tx=0x7f5f9803c040 comp rx=0 tx=0).stop 2024-01-15T18:13:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 msgr2=0x7f5fa80cea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 0x7f5fa80cea30 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000c4d0 tx=0x7f5f9000c990 comp rx=0 tx=0).stop 2024-01-15T18:13:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 shutdown_connections 2024-01-15T18:13:54.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5f8006c3d0 0x7f5f8006e890 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5fa815d4a0 0x7f5fa80cef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5fa80b3a50 0x7f5fa80cea30 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 --2- 172.21.15.2:0/3929551354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5fa806e890 0x7f5fa8158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 >> 172.21.15.2:0/3929551354 conn(0x7f5fa80b1c70 msgr2=0x7f5fa806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 shutdown_connections 2024-01-15T18:13:54.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.555+0000 7f5fb0400640 1 -- 172.21.15.2:0/3929551354 wait complete. 2024-01-15T18:13:54.578 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:13:54.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.799+0000 7f0117e00640 1 Processor -- start 2024-01-15T18:13:54.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.799+0000 7f0117e00640 1 -- start start 2024-01-15T18:13:54.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f0117e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0118161890 0x7f0118160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f0117e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01181611a0 0x7f011806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f0117e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f011806c5e0 con 0x7f011815d270 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f0117e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0118161630 con 0x7f0118161890 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f0117e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011806c8c0 con 0x7f01181611a0 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f010fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0118161890 0x7f0118160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f010fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0118161890 0x7f0118160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49910/0 (socket says 172.21.15.2:49910) 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f010fe00640 1 -- 172.21.15.2:0/2760981292 learned_addr learned my addr 172.21.15.2:0/2760981292 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.801+0000 7f0117400640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01181611a0 0x7f011806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.802+0000 7f010fe00640 1 -- 172.21.15.2:0/2760981292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01181611a0 msgr2=0x7f011806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.802+0000 7f010fe00640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01181611a0 0x7f011806cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.802+0000 7f010fe00640 1 -- 172.21.15.2:0/2760981292 >> v1:172.21.15.137:6789/0 conn(0x7f011815d270 legacy=0x7f01180b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.802+0000 7f010fe00640 1 -- 172.21.15.2:0/2760981292 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118071c00 con 0x7f0118161890 2024-01-15T18:13:54.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.802+0000 7f0117400640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01181611a0 0x7f011806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:54.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.803+0000 7f010fe00640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0118161890 0x7f0118160c60 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0100009870 tx=0x7f010002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42921fe404902eca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:54.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.803+0000 7f0116000640 1 -- 172.21.15.2:0/2760981292 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100031740 con 0x7f0118161890 2024-01-15T18:13:54.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.803+0000 7f0116000640 1 -- 172.21.15.2:0/2760981292 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01000318a0 con 0x7f0118161890 2024-01-15T18:13:54.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.803+0000 7f0116000640 1 -- 172.21.15.2:0/2760981292 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100031c30 con 0x7f0118161890 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 -- 172.21.15.2:0/2760981292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0118161890 msgr2=0x7f0118160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0118161890 0x7f0118160c60 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f0100009870 tx=0x7f010002b7e0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 -- 172.21.15.2:0/2760981292 shutdown_connections 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01181611a0 0x7f011806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 --2- 172.21.15.2:0/2760981292 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0118161890 0x7f0118160c60 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 -- 172.21.15.2:0/2760981292 >> 172.21.15.2:0/2760981292 conn(0x7f01180b0840 msgr2=0x7f01180b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 -- 172.21.15.2:0/2760981292 shutdown_connections 2024-01-15T18:13:54.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.804+0000 7f0117e00640 1 -- 172.21.15.2:0/2760981292 wait complete. 2024-01-15T18:13:54.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.806+0000 7f0117e00640 1 Processor -- start 2024-01-15T18:13:54.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.806+0000 7f0117e00640 1 -- start start 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f0117e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f011815d270 0x7f011814b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f0117e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 0x7f011814a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f0117e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0118161890 0x7f01180c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f0117e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f011814aea0 con 0x7f01181611a0 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f0117e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01180c7d10 con 0x7f011815d270 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f0117e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f011814d450 con 0x7f0118161890 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f010fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 0x7f011814a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f010fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 0x7f011814a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49914/0 (socket says 172.21.15.2:49914) 2024-01-15T18:13:54.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.807+0000 7f010fe00640 1 -- 172.21.15.2:0/295589547 learned_addr learned my addr 172.21.15.2:0/295589547 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f0117400640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0118161890 0x7f01180c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f0116a00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f011815d270 0x7f011814b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f010fe00640 1 -- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0118161890 msgr2=0x7f01180c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f010fe00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0118161890 0x7f01180c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f010fe00640 1 -- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f011815d270 msgr2=0x7f011814b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f010fe00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f011815d270 0x7f011814b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f010fe00640 1 -- 172.21.15.2:0/295589547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01000094d0 con 0x7f01181611a0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f0116a00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f011815d270 0x7f011814b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.808+0000 7f010fe00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 0x7f011814a960 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f01000090b0 tx=0x7f01000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.809+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100033c70 con 0x7f01181611a0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.809+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100033e10 con 0x7f01181611a0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.809+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010003c640 con 0x7f01181611a0 2024-01-15T18:13:54.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.809+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f011814d730 con 0x7f01181611a0 2024-01-15T18:13:54.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.809+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f011814dcf0 con 0x7f01181611a0 2024-01-15T18:13:54.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.810+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0118074770 con 0x7f01181611a0 2024-01-15T18:13:54.814 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.811+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f010003a040 con 0x7f01181611a0 2024-01-15T18:13:54.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.812+0000 7f010f400640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00f006c3a0 0x7f00f006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:54.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.812+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f01000aa790 con 0x7f01181611a0 2024-01-15T18:13:54.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.813+0000 7f0116a00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00f006c3a0 0x7f00f006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:54.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.814+0000 7f0116a00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00f006c3a0 0x7f00f006e860 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0108003eb0 tx=0x7f0108009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:54.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:54.817+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0100076e30 con 0x7f01181611a0 2024-01-15T18:13:55.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.042+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01180b2d30 con 0x7f00f006c3a0 2024-01-15T18:13:55.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.075+0000 7f010f400640 1 -- 172.21.15.2:0/295589547 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f0108005030 con 0x7f00f006c3a0 2024-01-15T18:13:55.078 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:13:55.079 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:13:55.079 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:13:55.079 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:13:55.079 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:13:55.079 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:13:55.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00f006c3a0 msgr2=0x7f00f006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00f006c3a0 0x7f00f006e860 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f0108003eb0 tx=0x7f0108009040 comp rx=0 tx=0).stop 2024-01-15T18:13:55.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 msgr2=0x7f011814a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 0x7f011814a960 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f01000090b0 tx=0x7f01000046c0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 shutdown_connections 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f00f006c3a0 0x7f00f006e860 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0118161890 0x7f01180c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01181611a0 0x7f011814a960 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 --2- 172.21.15.2:0/295589547 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f011815d270 0x7f011814b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.077+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 >> 172.21.15.2:0/295589547 conn(0x7f01180b0840 msgr2=0x7f0118160020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.078+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 shutdown_connections 2024-01-15T18:13:55.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.078+0000 7f0117e00640 1 -- 172.21.15.2:0/295589547 wait complete. 2024-01-15T18:13:55.101 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:13:55.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.323+0000 7f499d200640 1 Processor -- start 2024-01-15T18:13:55.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.324+0000 7f499d200640 1 -- start start 2024-01-15T18:13:55.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f499d200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f49980d1d40 0x7f49980cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f499d200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f49980b3fa0 0x7f49980d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f499d200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49980d0af0 con 0x7f49981609d0 2024-01-15T18:13:55.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f499d200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49980ce7b0 con 0x7f49980b3fa0 2024-01-15T18:13:55.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f499d200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49980cdbe0 con 0x7f49980d1d40 2024-01-15T18:13:55.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f4997400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f49981609d0 0x7f49980cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57500/0 (socket says 172.21.15.2:57500) 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f4997400640 1 -- 172.21.15.2:0/2689118184 learned_addr learned my addr 172.21.15.2:0/2689118184 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f4997e00640 1 --2- 172.21.15.2:0/2689118184 >> v2:172.21.15.2:3300/0 conn(0x7f49980b3fa0 0x7f49980d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.326+0000 7f4996a00640 1 --2- 172.21.15.2:0/2689118184 >> v2:172.21.15.137:3300/0 conn(0x7f49980d1d40 0x7f49980cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.327+0000 7f4996000640 1 -- 172.21.15.2:0/2689118184 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987650034 0 0) 0x7f49980d0af0 con 0x7f49981609d0 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.327+0000 7f4996000640 1 -- 172.21.15.2:0/2689118184 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4978003520 con 0x7f49981609d0 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.327+0000 7f4997e00640 1 -- 172.21.15.2:0/2689118184 >> v2:172.21.15.137:3300/0 conn(0x7f49980d1d40 msgr2=0x7f49980cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.327+0000 7f4997e00640 1 --2- 172.21.15.2:0/2689118184 >> v2:172.21.15.137:3300/0 conn(0x7f49980d1d40 0x7f49980cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.327+0000 7f4997e00640 1 -- 172.21.15.2:0/2689118184 >> v1:172.21.15.2:6789/0 conn(0x7f49981609d0 legacy=0x7f49980cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.327+0000 7f4997e00640 1 -- 172.21.15.2:0/2689118184 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49980cde80 con 0x7f49980b3fa0 2024-01-15T18:13:55.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.328+0000 7f4997e00640 1 --2- 172.21.15.2:0/2689118184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49980b3fa0 0x7f49980d01d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f49980b9180 tx=0x7f499002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f3ea78377f8eea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:55.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.328+0000 7f4996000640 1 -- 172.21.15.2:0/2689118184 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499002cc80 con 0x7f49980b3fa0 2024-01-15T18:13:55.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.328+0000 7f4996000640 1 -- 172.21.15.2:0/2689118184 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499002cde0 con 0x7f49980b3fa0 2024-01-15T18:13:55.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.328+0000 7f4996000640 1 -- 172.21.15.2:0/2689118184 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990034660 con 0x7f49980b3fa0 2024-01-15T18:13:55.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 -- 172.21.15.2:0/2689118184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49980b3fa0 msgr2=0x7f49980d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 --2- 172.21.15.2:0/2689118184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49980b3fa0 0x7f49980d01d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f49980b9180 tx=0x7f499002bed0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 -- 172.21.15.2:0/2689118184 shutdown_connections 2024-01-15T18:13:55.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 --2- 172.21.15.2:0/2689118184 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49980b3fa0 0x7f49980d01d0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 --2- 172.21.15.2:0/2689118184 >> v2:172.21.15.137:3300/0 conn(0x7f49980d1d40 0x7f49980cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 -- 172.21.15.2:0/2689118184 >> 172.21.15.2:0/2689118184 conn(0x7f499809c3e0 msgr2=0x7f49980d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:55.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 -- 172.21.15.2:0/2689118184 shutdown_connections 2024-01-15T18:13:55.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.329+0000 7f499d200640 1 -- 172.21.15.2:0/2689118184 wait complete. 2024-01-15T18:13:55.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.331+0000 7f499d200640 1 Processor -- start 2024-01-15T18:13:55.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.331+0000 7f499d200640 1 -- start start 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.332+0000 7f499d200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49980b3fa0 0x7f49980b8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.332+0000 7f499d200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49980d1d40 0x7f49980b8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.332+0000 7f499d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 0x7f49980d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.332+0000 7f499d200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49981f3140 con 0x7f49981609d0 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.332+0000 7f499d200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49981f3420 con 0x7f49980d1d40 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.332+0000 7f499d200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49981f3700 con 0x7f49980b3fa0 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 0x7f49980d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 0x7f49980d30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49932/0 (socket says 172.21.15.2:49932) 2024-01-15T18:13:55.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 -- 172.21.15.2:0/613728115 learned_addr learned my addr 172.21.15.2:0/613728115 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997400640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49980b3fa0 0x7f49980b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4996a00640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49980d1d40 0x7f49980b8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 -- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49980b3fa0 msgr2=0x7f49980b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49980b3fa0 0x7f49980b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 -- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49980d1d40 msgr2=0x7f49980b8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49980d1d40 0x7f49980b8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4997e00640 1 -- 172.21.15.2:0/613728115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990009a70 con 0x7f49981609d0 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.333+0000 7f4996a00640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49980d1d40 0x7f49980b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:55.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.334+0000 7f4997e00640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 0x7f49980d30d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f4990009e80 tx=0x7f4990004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:55.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.334+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990040650 con 0x7f49981609d0 2024-01-15T18:13:55.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.334+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49900407b0 con 0x7f49981609d0 2024-01-15T18:13:55.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.334+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49981f39e0 con 0x7f49981609d0 2024-01-15T18:13:55.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.334+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4990040b60 con 0x7f49981609d0 2024-01-15T18:13:55.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.334+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49981f2da0 con 0x7f49981609d0 2024-01-15T18:13:55.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.335+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49980b4780 con 0x7f49981609d0 2024-01-15T18:13:55.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.337+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f4990040cc0 con 0x7f49981609d0 2024-01-15T18:13:55.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.338+0000 7f498f400640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f497006c3a0 0x7f497006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.338+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f49900ab450 con 0x7f49981609d0 2024-01-15T18:13:55.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.339+0000 7f4997400640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f497006c3a0 0x7f497006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.340+0000 7f4997400640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f497006c3a0 0x7f497006e860 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f49880011e0 tx=0x7f4988009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:55.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.343+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4990077d10 con 0x7f49981609d0 2024-01-15T18:13:55.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.576+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49981f2f30 con 0x7f497006c3a0 2024-01-15T18:13:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.644+0000 7f498f400640 1 -- 172.21.15.2:0/613728115 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f49981f2f30 con 0x7f497006c3a0 2024-01-15T18:13:55.648 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:13:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f497006c3a0 msgr2=0x7f497006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f497006c3a0 0x7f497006e860 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f49880011e0 tx=0x7f4988009040 comp rx=0 tx=0).stop 2024-01-15T18:13:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 msgr2=0x7f49980d30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 0x7f49980d30d0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f4990009e80 tx=0x7f4990004a40 comp rx=0 tx=0).stop 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 shutdown_connections 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f497006c3a0 0x7f497006e860 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f49981609d0 0x7f49980d30d0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f49980d1d40 0x7f49980b8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 --2- 172.21.15.2:0/613728115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f49980b3fa0 0x7f49980b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 >> 172.21.15.2:0/613728115 conn(0x7f499809c3e0 msgr2=0x7f49980b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:55.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 shutdown_connections 2024-01-15T18:13:55.652 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.647+0000 7f499d200640 1 -- 172.21.15.2:0/613728115 wait complete. 2024-01-15T18:13:55.670 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-15T18:13:55.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.893+0000 7f73f6e00640 1 Processor -- start 2024-01-15T18:13:55.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.894+0000 7f73f6e00640 1 -- start start 2024-01-15T18:13:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.895+0000 7f73f6e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f73f0146b80 0x7f73f006c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.895+0000 7f73f6e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f0070da0 con 0x7f73f006cba0 2024-01-15T18:13:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.895+0000 7f73f6e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f0070ac0 con 0x7f73f00ce100 2024-01-15T18:13:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.895+0000 7f73f6e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f006c240 con 0x7f73f0146b80 2024-01-15T18:13:55.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73efe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f73f0146b80 0x7f73f006c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73ef400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f73f006cba0 0x7f73f00cd9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57508/0 (socket says 172.21.15.2:57508) 2024-01-15T18:13:55.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73efe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f73f0146b80 0x7f73f006c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49946/0 (socket says 172.21.15.2:49946) 2024-01-15T18:13:55.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73ef400640 1 -- 172.21.15.2:0/336338954 learned_addr learned my addr 172.21.15.2:0/336338954 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:13:55.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73efe00640 1 -- 172.21.15.2:0/336338954 >> v1:172.21.15.137:6789/0 conn(0x7f73f00ce100 legacy=0x7f73f006e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73efe00640 1 -- 172.21.15.2:0/336338954 >> v1:172.21.15.2:6789/0 conn(0x7f73f006cba0 legacy=0x7f73f00cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73efe00640 1 -- 172.21.15.2:0/336338954 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f006bb80 con 0x7f73f0146b80 2024-01-15T18:13:55.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.896+0000 7f73eea00640 1 -- 172.21.15.2:0/336338954 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395429707 0 0) 0x7f73f0070da0 con 0x7f73f006cba0 2024-01-15T18:13:55.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.898+0000 7f73efe00640 1 --2- 172.21.15.2:0/336338954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f0146b80 0x7f73f006c660 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009d90 tx=0x7f73e002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b6ef45a351e560c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.898+0000 7f73eea00640 1 -- 172.21.15.2:0/336338954 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e002cbd0 con 0x7f73f0146b80 2024-01-15T18:13:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.898+0000 7f73eea00640 1 -- 172.21.15.2:0/336338954 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e002cd30 con 0x7f73f0146b80 2024-01-15T18:13:55.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.898+0000 7f73eea00640 1 -- 172.21.15.2:0/336338954 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e002cea0 con 0x7f73f0146b80 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 -- 172.21.15.2:0/336338954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f0146b80 msgr2=0x7f73f006c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 --2- 172.21.15.2:0/336338954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f0146b80 0x7f73f006c660 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f73e0009d90 tx=0x7f73e002be20 comp rx=0 tx=0).stop 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 -- 172.21.15.2:0/336338954 shutdown_connections 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 --2- 172.21.15.2:0/336338954 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f0146b80 0x7f73f006c660 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 -- 172.21.15.2:0/336338954 >> 172.21.15.2:0/336338954 conn(0x7f73f00b2c10 msgr2=0x7f73f00b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 -- 172.21.15.2:0/336338954 shutdown_connections 2024-01-15T18:13:55.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.899+0000 7f73f6e00640 1 -- 172.21.15.2:0/336338954 wait complete. 2024-01-15T18:13:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.901+0000 7f73f6e00640 1 Processor -- start 2024-01-15T18:13:55.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.901+0000 7f73f6e00640 1 -- start start 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f6e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73f0146b80 0x7f73f00cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f6e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73f00a8460 0x7f73f015cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f6e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 0x7f73f00aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f6e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f00aa6c0 con 0x7f73f00a9cf0 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f6e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f015d300 con 0x7f73f00a8460 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f6e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f00a89d0 con 0x7f73f0146b80 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 0x7f73f00aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 0x7f73f00aa180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49948/0 (socket says 172.21.15.2:49948) 2024-01-15T18:13:55.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.902+0000 7f73f4e00640 1 -- 172.21.15.2:0/269371658 learned_addr learned my addr 172.21.15.2:0/269371658 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73ef400640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73f00a8460 0x7f73f015cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73efe00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73f0146b80 0x7f73f00cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f4e00640 1 -- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73f0146b80 msgr2=0x7f73f00cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f4e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73f0146b80 0x7f73f00cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f4e00640 1 -- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73f00a8460 msgr2=0x7f73f015cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f4e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73f00a8460 0x7f73f015cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f4e00640 1 -- 172.21.15.2:0/269371658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e0009a70 con 0x7f73f00a9cf0 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73ef400640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73f00a8460 0x7f73f015cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73efe00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73f0146b80 0x7f73f00cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f4e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 0x7f73f00aa180 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f73e800c3a0 tx=0x7f73e800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:55.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e80052d0 con 0x7f73f00a9cf0 2024-01-15T18:13:55.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73e8005430 con 0x7f73f00a9cf0 2024-01-15T18:13:55.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73e8003830 con 0x7f73f00a9cf0 2024-01-15T18:13:55.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.903+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f00a8cb0 con 0x7f73f00a9cf0 2024-01-15T18:13:55.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.904+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f00a9270 con 0x7f73f00a9cf0 2024-01-15T18:13:55.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.905+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f00ca320 con 0x7f73f00a9cf0 2024-01-15T18:13:55.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.906+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f73e8003990 con 0x7f73f00a9cf0 2024-01-15T18:13:55.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.908+0000 7f73e7e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73c806c3a0 0x7f73c806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:55.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.908+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f73e8087fa0 con 0x7f73f00a9cf0 2024-01-15T18:13:55.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.908+0000 7f73efe00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73c806c3a0 0x7f73c806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:55.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.910+0000 7f73efe00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73c806c3a0 0x7f73c806e860 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f73e0036000 tx=0x7f73e0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:55.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:55.912+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73f00ca320 con 0x7f73f00a9cf0 2024-01-15T18:13:56.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.134+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f73f006fd90 con 0x7f73c806c3a0 2024-01-15T18:13:56.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.179+0000 7f73e7e00640 1 -- 172.21.15.2:0/269371658 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f73e00440b0 con 0x7f73c806c3a0 2024-01-15T18:13:56.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.181+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73c806c3a0 msgr2=0x7f73c806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.181+0000 7f73f6e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73c806c3a0 0x7f73c806e860 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f73e0036000 tx=0x7f73e0030000 comp rx=0 tx=0).stop 2024-01-15T18:13:56.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.181+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 msgr2=0x7f73f00aa180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.181+0000 7f73f6e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 0x7f73f00aa180 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f73e800c3a0 tx=0x7f73e800c860 comp rx=0 tx=0).stop 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 shutdown_connections 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73c806c3a0 0x7f73c806e860 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73f00a9cf0 0x7f73f00aa180 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73f00a8460 0x7f73f015cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 --2- 172.21.15.2:0/269371658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73f0146b80 0x7f73f00cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 >> 172.21.15.2:0/269371658 conn(0x7f73f00b2c10 msgr2=0x7f73f00d01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 shutdown_connections 2024-01-15T18:13:56.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.182+0000 7f73f6e00640 1 -- 172.21.15.2:0/269371658 wait complete. 2024-01-15T18:13:56.192 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.189+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.192 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.189+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.192 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.189+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.192 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.189+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.192 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.189+0000 7fe20de00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.206 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-15T18:13:56.213 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.211+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.213 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.211+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.213 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.211+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.213 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.211+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.213 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:13:56.211+0000 7fe20f200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:13:56.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.421+0000 7efc6ba00640 1 Processor -- start 2024-01-15T18:13:56.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.422+0000 7efc6ba00640 1 -- start start 2024-01-15T18:13:56.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc6ba00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efc64144b80 0x7efc6415ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc6ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640b2ba0 0x7efc640b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc6ba00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc640cfec0 con 0x7efc6415b860 2024-01-15T18:13:56.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc6ba00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc640b3510 con 0x7efc64144b80 2024-01-15T18:13:56.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc6ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc640b4620 con 0x7efc640b2ba0 2024-01-15T18:13:56.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc69000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efc64144b80 0x7efc6415ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc69a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640b2ba0 0x7efc640b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.424+0000 7efc69000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efc64144b80 0x7efc6415ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:42552/0 (socket says 172.21.15.2:42552) 2024-01-15T18:13:56.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.425+0000 7efc69000640 1 -- 172.21.15.2:0/1142977092 learned_addr learned my addr 172.21.15.2:0/1142977092 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:56.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.425+0000 7efc69000640 1 -- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640b2ba0 msgr2=0x7efc640b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:56.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.425+0000 7efc69000640 1 --2- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640b2ba0 0x7efc640b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.425+0000 7efc69000640 1 -- 172.21.15.2:0/1142977092 >> v1:172.21.15.137:6789/0 conn(0x7efc6415b860 legacy=0x7efc640b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.425+0000 7efc69000640 1 -- 172.21.15.2:0/1142977092 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6415fc30 con 0x7efc64144b80 2024-01-15T18:13:56.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.427+0000 7efc69000640 1 --2- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc64144b80 0x7efc6415ed00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efc54009870 tx=0x7efc5402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cacc4e39ab1969a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:56.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.427+0000 7efc63400640 1 -- 172.21.15.2:0/1142977092 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54030840 con 0x7efc64144b80 2024-01-15T18:13:56.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.427+0000 7efc63400640 1 -- 172.21.15.2:0/1142977092 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc540309a0 con 0x7efc64144b80 2024-01-15T18:13:56.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.427+0000 7efc63400640 1 -- 172.21.15.2:0/1142977092 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54030d30 con 0x7efc64144b80 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 -- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc64144b80 msgr2=0x7efc6415ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 --2- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc64144b80 0x7efc6415ed00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7efc54009870 tx=0x7efc5402b980 comp rx=0 tx=0).stop 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 -- 172.21.15.2:0/1142977092 shutdown_connections 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 --2- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640b2ba0 0x7efc640b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 --2- 172.21.15.2:0/1142977092 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc64144b80 0x7efc6415ed00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 -- 172.21.15.2:0/1142977092 >> 172.21.15.2:0/1142977092 conn(0x7efc640b6430 msgr2=0x7efc640b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 -- 172.21.15.2:0/1142977092 shutdown_connections 2024-01-15T18:13:56.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.428+0000 7efc6ba00640 1 -- 172.21.15.2:0/1142977092 wait complete. 2024-01-15T18:13:56.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.430+0000 7efc6ba00640 1 Processor -- start 2024-01-15T18:13:56.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.430+0000 7efc6ba00640 1 -- start start 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc6ba00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 0x7efc641f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc6ba00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64144b80 0x7efc640d1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc6ba00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc6415b860 0x7efc641f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc6ba00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc641f6e30 con 0x7efc640b2ba0 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc6ba00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc641f3d40 con 0x7efc6415b860 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc6ba00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc641f4020 con 0x7efc64144b80 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc69000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 0x7efc641f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc69000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 0x7efc641f72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49956/0 (socket says 172.21.15.2:49956) 2024-01-15T18:13:56.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.431+0000 7efc69000640 1 -- 172.21.15.2:0/2853082788 learned_addr learned my addr 172.21.15.2:0/2853082788 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69a00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc6415b860 0x7efc641f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69000640 1 -- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64144b80 msgr2=0x7efc640d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc63e00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64144b80 0x7efc640d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69000640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64144b80 0x7efc640d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69000640 1 -- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc6415b860 msgr2=0x7efc641f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69000640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc6415b860 0x7efc641f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69000640 1 -- 172.21.15.2:0/2853082788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc540094d0 con 0x7efc640b2ba0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69a00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc6415b860 0x7efc641f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc63e00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64144b80 0x7efc640d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.432+0000 7efc69000640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 0x7efc641f72c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7efc54000c00 tx=0x7efc54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.433+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54039c90 con 0x7efc640b2ba0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.433+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc54039e10 con 0x7efc640b2ba0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.433+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc54043640 con 0x7efc640b2ba0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.433+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc641f67e0 con 0x7efc640b2ba0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.433+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc64143c50 con 0x7efc640b2ba0 2024-01-15T18:13:56.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.434+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc640686f0 con 0x7efc640b2ba0 2024-01-15T18:13:56.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.435+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7efc540437a0 con 0x7efc640b2ba0 2024-01-15T18:13:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.437+0000 7efc60c00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc3c06c3a0 0x7efc3c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.437+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7efc5403e070 con 0x7efc640b2ba0 2024-01-15T18:13:56.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.437+0000 7efc63e00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc3c06c3a0 0x7efc3c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.439+0000 7efc63e00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc3c06c3a0 0x7efc3c06e860 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efc640afef0 tx=0x7efc4c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:56.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.441+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc54078010 con 0x7efc640b2ba0 2024-01-15T18:13:56.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.663+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7efc6415f950 con 0x7efc3c06c3a0 2024-01-15T18:13:56.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.702+0000 7efc60c00640 1 -- 172.21.15.2:0/2853082788 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efc4c001340 con 0x7efc3c06c3a0 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc3c06c3a0 msgr2=0x7efc3c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc3c06c3a0 0x7efc3c06e860 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efc640afef0 tx=0x7efc4c009040 comp rx=0 tx=0).stop 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 msgr2=0x7efc641f72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 0x7efc641f72c0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7efc54000c00 tx=0x7efc54004730 comp rx=0 tx=0).stop 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 shutdown_connections 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc3c06c3a0 0x7efc3c06e860 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc6415b860 0x7efc641f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64144b80 0x7efc640d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 --2- 172.21.15.2:0/2853082788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640b2ba0 0x7efc641f72c0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 >> 172.21.15.2:0/2853082788 conn(0x7efc640b6430 msgr2=0x7efc6415bc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 shutdown_connections 2024-01-15T18:13:56.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.705+0000 7efc6ba00640 1 -- 172.21.15.2:0/2853082788 wait complete. 2024-01-15T18:13:56.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:13:56.726 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-15T18:13:56.784 INFO:teuthology.orchestra.run.smithi002.stdout:2 2024-01-15T18:13:56.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-15T18:13:56.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.974+0000 7f1464a00640 1 Processor -- start 2024-01-15T18:13:56.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.975+0000 7f1464a00640 1 -- start start 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.976+0000 7f1464a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f14600d1870 0x7f14600d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.976+0000 7f1464a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 0x7f14600b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.976+0000 7f1464a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14600d1250 con 0x7f14600b2210 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.976+0000 7f1464a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14600ce6d0 con 0x7f14600d1870 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.976+0000 7f1464a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600ce8e0 con 0x7f14600d1c50 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145de00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f14600b2210 0x7f146015d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57518/0 (socket says 172.21.15.2:57518) 2024-01-15T18:13:56.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145de00640 1 -- 172.21.15.2:0/3869011701 learned_addr learned my addr 172.21.15.2:0/3869011701 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:13:56.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145e800640 1 --2- 172.21.15.2:0/3869011701 >> v2:172.21.15.137:3300/0 conn(0x7f14600d1870 0x7f14600d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145f200640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 0x7f14600b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387270777 0 0) 0x7f14600d1250 con 0x7f14600b2210 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1440003520 con 0x7f14600b2210 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991849181 0 0) 0x7f1440003520 con 0x7f14600b2210 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 msgr2=0x7f14600b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 0x7f14600b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 >> v2:172.21.15.137:3300/0 conn(0x7f14600d1870 msgr2=0x7f14600d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 --2- 172.21.15.2:0/3869011701 >> v2:172.21.15.137:3300/0 conn(0x7f14600d1870 0x7f14600d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14600cebc0 con 0x7f14600b2210 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.977+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1448001670 con 0x7f14600b2210 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 >> v1:172.21.15.2:6789/0 conn(0x7f14600b2210 legacy=0x7f146015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 0x7f1440003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1440003f40 0x7f14400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1440004d00 0x7f1440005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.983 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600ce6d0 con 0x7f1440004d00 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14600ce8e0 con 0x7f1440003f40 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144000b980 con 0x7f14600d1c50 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145f200640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1440004d00 0x7f1440005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145de00640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1440003f40 0x7f14400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145f200640 1 -- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 msgr2=0x7f1440003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145f200640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f14600d1c50 0x7f1440003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145f200640 1 -- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1440003f40 msgr2=0x7f14400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145f200640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1440003f40 0x7f14400047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.978+0000 7f145f200640 1 -- 172.21.15.2:0/3869011701 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1440003520 con 0x7f1440004d00 2024-01-15T18:13:56.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.979+0000 7f145f200640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1440004d00 0x7f1440005180 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f1458009ac0 tx=0x7f14400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=478a792d0a0a5dfd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:56.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145802c6b0 con 0x7f1440004d00 2024-01-15T18:13:56.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145802c810 con 0x7f1440004d00 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f145d400640 1 -- 172.21.15.2:0/3869011701 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145802cc00 con 0x7f1440004d00 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f1464a00640 1 -- 172.21.15.2:0/3869011701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1440004d00 msgr2=0x7f1440005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f1464a00640 1 --2- 172.21.15.2:0/3869011701 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1440004d00 0x7f1440005180 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f1458009ac0 tx=0x7f14400057f0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f145f200640 1 -- 172.21.15.2:0/3869011701 reap_dead start 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.980+0000 7f1464a00640 1 -- 172.21.15.2:0/3869011701 shutdown_connections 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.981+0000 7f1464a00640 1 -- 172.21.15.2:0/3869011701 >> 172.21.15.2:0/3869011701 conn(0x7f14600d30b0 msgr2=0x7f14600d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.981+0000 7f1464a00640 1 -- 172.21.15.2:0/3869011701 shutdown_connections 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.981+0000 7f1464a00640 1 -- 172.21.15.2:0/3869011701 wait complete. 2024-01-15T18:13:56.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.982+0000 7f1464a00640 1 Processor -- start 2024-01-15T18:13:56.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.983+0000 7f1464a00640 1 -- start start 2024-01-15T18:13:56.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f1464a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f146006bea0 0x7f14600b7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f1464a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 0x7f1460160210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f1464a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f146006c280 0x7f1460160750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f1464a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146015dc70 con 0x7f14600b5cd0 2024-01-15T18:13:56.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f1464a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146015e4d0 con 0x7f146006c280 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f1464a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14600b8830 con 0x7f146006bea0 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 0x7f1460160210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 0x7f1460160210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49970/0 (socket says 172.21.15.2:49970) 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 -- 172.21.15.2:0/3481546244 learned_addr learned my addr 172.21.15.2:0/3481546244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 -- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f146006bea0 msgr2=0x7f14600b7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f146006bea0 0x7f14600b7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 -- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f146006c280 msgr2=0x7f1460160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f146006c280 0x7f1460160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.984+0000 7f145de00640 1 -- 172.21.15.2:0/3481546244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146015de80 con 0x7f14600b5cd0 2024-01-15T18:13:56.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.985+0000 7f145de00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 0x7f1460160210 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f144800db10 tx=0x7f1448001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.985+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448004cb0 con 0x7f14600b5cd0 2024-01-15T18:13:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.985+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448004e10 con 0x7f14600b5cd0 2024-01-15T18:13:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.985+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146015e160 con 0x7f14600b5cd0 2024-01-15T18:13:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.985+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448007310 con 0x7f14600b5cd0 2024-01-15T18:13:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.985+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14600b85e0 con 0x7f14600b5cd0 2024-01-15T18:13:56.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.986+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1460074770 con 0x7f14600b5cd0 2024-01-15T18:13:56.991 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.987+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 36) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f1448007470 con 0x7f14600b5cd0 2024-01-15T18:13:56.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.989+0000 7f1456a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f143806c3d0 0x7f143806e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:13:56.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.989+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5821+0+0 (secure 0 0 0) 0x7f1448087f50 con 0x7f14600b5cd0 2024-01-15T18:13:56.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.990+0000 7f145e800640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f143806c3d0 0x7f143806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:13:56.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.991+0000 7f145e800640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f143806c3d0 0x7f143806e890 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1450004b30 tx=0x7f145000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:13:56.997 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:56.993+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1448054810 con 0x7f14600b5cd0 2024-01-15T18:13:57.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:57.204+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f14600d0f20 con 0x7f143806c3d0 2024-01-15T18:13:57.262 INFO:tasks.ceph.mon.a.smithi002.stderr:2024-01-15T18:13:57.258+0000 7f3173c00640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-15T18:13:57.357 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:13:58.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:13:58.802+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f1448054be0 con 0x7f14600b5cd0 2024-01-15T18:14:00.295 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.291+0000 7f1456a00640 1 -- 172.21.15.2:0/3481546244 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f145000f040 con 0x7f143806c3d0 2024-01-15T18:14:00.295 INFO:teuthology.orchestra.run.smithi002.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-15T18:14:00.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.295+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f143806c3d0 msgr2=0x7f143806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.295+0000 7f1464a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f143806c3d0 0x7f143806e890 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1450004b30 tx=0x7f145000b040 comp rx=0 tx=0).stop 2024-01-15T18:14:00.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.295+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 msgr2=0x7f1460160210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.295+0000 7f1464a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 0x7f1460160210 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f144800db10 tx=0x7f1448001d40 comp rx=0 tx=0).stop 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.296+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 shutdown_connections 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.296+0000 7f1464a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f143806c3d0 0x7f143806e890 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.296+0000 7f1464a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f146006c280 0x7f1460160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.296+0000 7f1464a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f14600b5cd0 0x7f1460160210 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.296+0000 7f1464a00640 1 --2- 172.21.15.2:0/3481546244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f146006bea0 0x7f14600b7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.296+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 >> 172.21.15.2:0/3481546244 conn(0x7f14600d30b0 msgr2=0x7f14600afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:00.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.297+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 shutdown_connections 2024-01-15T18:14:00.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.297+0000 7f1464a00640 1 -- 172.21.15.2:0/3481546244 wait complete. 2024-01-15T18:14:00.331 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:14:00.331 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:14:00.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:14:00.600 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:14:00.600 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:14:00.600 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:00.642 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.896+0000 7fa761600640 1 Processor -- start 2024-01-15T18:14:00.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.897+0000 7fa761600640 1 -- start start 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa761600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa75c1469f0 0x7fa75c071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa761600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa75c146dd0 0x7fa75c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa761600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c0ac870 con 0x7fa75c06c9b0 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa761600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c06ce20 con 0x7fa75c1469f0 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa761600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c15d970 con 0x7fa75c146dd0 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa75aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa75c1469f0 0x7fa75c071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa75b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa75c06c9b0 0x7fa75c070380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57522/0 (socket says 172.21.15.2:57522) 2024-01-15T18:14:00.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.898+0000 7fa75b400640 1 -- 172.21.15.2:0/2321346312 learned_addr learned my addr 172.21.15.2:0/2321346312 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.899+0000 7fa75a000640 1 --2- 172.21.15.2:0/2321346312 >> v2:172.21.15.137:3300/0 conn(0x7fa75c146dd0 0x7fa75c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.899+0000 7fa75aa00640 1 -- 172.21.15.2:0/2321346312 >> v2:172.21.15.137:3300/0 conn(0x7fa75c146dd0 msgr2=0x7fa75c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.899+0000 7fa75aa00640 1 --2- 172.21.15.2:0/2321346312 >> v2:172.21.15.137:3300/0 conn(0x7fa75c146dd0 0x7fa75c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.899+0000 7fa75aa00640 1 -- 172.21.15.2:0/2321346312 >> v1:172.21.15.2:6789/0 conn(0x7fa75c06c9b0 legacy=0x7fa75c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.899+0000 7fa75aa00640 1 -- 172.21.15.2:0/2321346312 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c15dc30 con 0x7fa75c1469f0 2024-01-15T18:14:00.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.899+0000 7fa75a000640 1 --2- 172.21.15.2:0/2321346312 >> v2:172.21.15.137:3300/0 conn(0x7fa75c146dd0 0x7fa75c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.900+0000 7fa75aa00640 1 --2- 172.21.15.2:0/2321346312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c1469f0 0x7fa75c071df0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fa75c06d600 tx=0x7fa74c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b688702614d6a568 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.901+0000 7fa759600640 1 -- 172.21.15.2:0/2321346312 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c031830 con 0x7fa75c1469f0 2024-01-15T18:14:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.901+0000 7fa759600640 1 -- 172.21.15.2:0/2321346312 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c031990 con 0x7fa75c1469f0 2024-01-15T18:14:00.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.901+0000 7fa759600640 1 -- 172.21.15.2:0/2321346312 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c031d20 con 0x7fa75c1469f0 2024-01-15T18:14:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.901+0000 7fa761600640 1 -- 172.21.15.2:0/2321346312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c1469f0 msgr2=0x7fa75c071df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.901+0000 7fa761600640 1 --2- 172.21.15.2:0/2321346312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c1469f0 0x7fa75c071df0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fa75c06d600 tx=0x7fa74c02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.902+0000 7fa761600640 1 -- 172.21.15.2:0/2321346312 shutdown_connections 2024-01-15T18:14:00.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.902+0000 7fa761600640 1 --2- 172.21.15.2:0/2321346312 >> v2:172.21.15.137:3300/0 conn(0x7fa75c146dd0 0x7fa75c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.902+0000 7fa761600640 1 --2- 172.21.15.2:0/2321346312 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c1469f0 0x7fa75c071df0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.902+0000 7fa761600640 1 -- 172.21.15.2:0/2321346312 >> 172.21.15.2:0/2321346312 conn(0x7fa75c06da20 msgr2=0x7fa75c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:00.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.902+0000 7fa761600640 1 -- 172.21.15.2:0/2321346312 shutdown_connections 2024-01-15T18:14:00.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.902+0000 7fa761600640 1 -- 172.21.15.2:0/2321346312 wait complete. 2024-01-15T18:14:00.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.904+0000 7fa761600640 1 Processor -- start 2024-01-15T18:14:00.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.904+0000 7fa761600640 1 -- start start 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa761600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 0x7fa75c14f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa761600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa75c1469f0 0x7fa75c138240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa761600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa75c14ca20 0x7fa75c138780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa761600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c14faa0 con 0x7fa75c06c9b0 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa761600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c13b1e0 con 0x7fa75c14ca20 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa761600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c13b4c0 con 0x7fa75c1469f0 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa75aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 0x7fa75c14f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa75aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 0x7fa75c14f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49992/0 (socket says 172.21.15.2:49992) 2024-01-15T18:14:00.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.905+0000 7fa75aa00640 1 -- 172.21.15.2:0/3111615568 learned_addr learned my addr 172.21.15.2:0/3111615568 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:00.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75b400640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa75c14ca20 0x7fa75c138780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75a000640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa75c1469f0 0x7fa75c138240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75aa00640 1 -- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa75c1469f0 msgr2=0x7fa75c138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75aa00640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa75c1469f0 0x7fa75c138240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75aa00640 1 -- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa75c14ca20 msgr2=0x7fa75c138780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75aa00640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa75c14ca20 0x7fa75c138780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75aa00640 1 -- 172.21.15.2:0/3111615568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c009400 con 0x7fa75c06c9b0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75b400640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa75c14ca20 0x7fa75c138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa75aa00640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 0x7fa75c14f560 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009890 tx=0x7fa74c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c03ba70 con 0x7fa75c06c9b0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c03bbf0 con 0x7fa75c06c9b0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c045400 con 0x7fa75c06c9b0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.906+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c070380 con 0x7fa75c06c9b0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.907+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c070940 con 0x7fa75c06c9b0 2024-01-15T18:14:00.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.908+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c06b230 con 0x7fa75c06c9b0 2024-01-15T18:14:00.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.909+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa74c04e440 con 0x7fa75c06c9b0 2024-01-15T18:14:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.909+0000 7fa752a00640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa73406c430 0x7fa73406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.909+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fa74c0aae40 con 0x7fa75c06c9b0 2024-01-15T18:14:00.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.910+0000 7fa75a000640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa73406c430 0x7fa73406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:00.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.911+0000 7fa75a000640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa73406c430 0x7fa73406e8f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa75c141f30 tx=0x7fa744008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:00.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:00.914+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa74c0773b0 con 0x7fa75c06c9b0 2024-01-15T18:14:01.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.108+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa75c0686f0 con 0x7fa75c06c9b0 2024-01-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.110+0000 7fa752a00640 1 -- 172.21.15.2:0/3111615568 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+13157 (secure 0 0 0) 0x7fa74c035090 con 0x7fa75c06c9b0 2024-01-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:01.114 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":106,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:00.272802+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6837/320323477":"2024-01-16T18:13:57.259474+0000","172.21.15.137:6835/320323477":"2024-01-16T18:13:57.259474+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa73406c430 msgr2=0x7fa73406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa73406c430 0x7fa73406e8f0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa75c141f30 tx=0x7fa744008040 comp rx=0 tx=0).stop 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 msgr2=0x7fa75c14f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 0x7fa75c14f560 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009890 tx=0x7fa74c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 shutdown_connections 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa73406c430 0x7fa73406e8f0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa75c14ca20 0x7fa75c138780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa75c1469f0 0x7fa75c138240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 --2- 172.21.15.2:0/3111615568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa75c06c9b0 0x7fa75c14f560 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 >> 172.21.15.2:0/3111615568 conn(0x7fa75c06da20 msgr2=0x7fa75c0ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 shutdown_connections 2024-01-15T18:14:01.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.113+0000 7fa761600640 1 -- 172.21.15.2:0/3111615568 wait complete. 2024-01-15T18:14:01.137 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:14:01.137 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:14:01.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.359+0000 7fed98e00640 1 Processor -- start 2024-01-15T18:14:01.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.359+0000 7fed98e00640 1 -- start start 2024-01-15T18:14:01.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed98e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed940ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed98e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940ca4a0 con 0x7fed9406abc0 2024-01-15T18:14:01.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed98e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed940716a0 con 0x7fed94071210 2024-01-15T18:14:01.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed98e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed94072220 con 0x7fed94164cd0 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed93e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fed9406abc0 0x7fed940ca910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57526/0 (socket says 172.21.15.2:57526) 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed93e00640 1 -- 172.21.15.2:0/3236592151 learned_addr learned my addr 172.21.15.2:0/3236592151 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.361+0000 7fed93400640 1 --2- 172.21.15.2:0/3236592151 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed940ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.362+0000 7fed92a00640 1 -- 172.21.15.2:0/3236592151 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325310939 0 0) 0x7fed940ca4a0 con 0x7fed9406abc0 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.362+0000 7fed92a00640 1 -- 172.21.15.2:0/3236592151 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed74003520 con 0x7fed9406abc0 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.362+0000 7fed93400640 1 -- 172.21.15.2:0/3236592151 >> v1:172.21.15.137:6789/0 conn(0x7fed94071210 legacy=0x7fed9406a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.362+0000 7fed93400640 1 -- 172.21.15.2:0/3236592151 >> v1:172.21.15.2:6789/0 conn(0x7fed9406abc0 legacy=0x7fed940ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.362+0000 7fed93400640 1 -- 172.21.15.2:0/3236592151 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed94071980 con 0x7fed94164cd0 2024-01-15T18:14:01.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.364+0000 7fed93400640 1 --2- 172.21.15.2:0/3236592151 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed940ce310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed840097d0 tx=0x7fed8402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=157e8e4279617e6c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:01.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.364+0000 7fed92a00640 1 -- 172.21.15.2:0/3236592151 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84031890 con 0x7fed94164cd0 2024-01-15T18:14:01.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.364+0000 7fed92a00640 1 -- 172.21.15.2:0/3236592151 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed840319f0 con 0x7fed94164cd0 2024-01-15T18:14:01.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.364+0000 7fed92a00640 1 -- 172.21.15.2:0/3236592151 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed84031d60 con 0x7fed94164cd0 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 -- 172.21.15.2:0/3236592151 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 msgr2=0x7fed940ce310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 --2- 172.21.15.2:0/3236592151 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed940ce310 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fed840097d0 tx=0x7fed8402b930 comp rx=0 tx=0).stop 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 -- 172.21.15.2:0/3236592151 shutdown_connections 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 --2- 172.21.15.2:0/3236592151 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed940ce310 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 -- 172.21.15.2:0/3236592151 >> 172.21.15.2:0/3236592151 conn(0x7fed940cfd30 msgr2=0x7fed940d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 -- 172.21.15.2:0/3236592151 shutdown_connections 2024-01-15T18:14:01.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.365+0000 7fed98e00640 1 -- 172.21.15.2:0/3236592151 wait complete. 2024-01-15T18:14:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.367+0000 7fed98e00640 1 Processor -- start 2024-01-15T18:14:01.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.367+0000 7fed98e00640 1 -- start start 2024-01-15T18:14:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed98e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed9406abc0 0x7fed941f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed98e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed94162040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed98e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 0x7fed940b2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed98e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed940c8700 con 0x7fed940aef20 2024-01-15T18:14:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed98e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed941f7b10 con 0x7fed9406abc0 2024-01-15T18:14:01.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed98e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed940b33a0 con 0x7fed94164cd0 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed93e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 0x7fed940b2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed93e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 0x7fed940b2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50008/0 (socket says 172.21.15.2:50008) 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.368+0000 7fed93e00640 1 -- 172.21.15.2:0/2153940634 learned_addr learned my addr 172.21.15.2:0/2153940634 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed93e00640 1 -- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 msgr2=0x7fed94162040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed93e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed94162040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed93e00640 1 -- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed9406abc0 msgr2=0x7fed941f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed93e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed9406abc0 0x7fed941f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed93e00640 1 -- 172.21.15.2:0/2153940634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed84009400 con 0x7fed940aef20 2024-01-15T18:14:01.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed93e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 0x7fed940b2a50 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00c400 tx=0x7fed8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c005ce0 con 0x7fed940aef20 2024-01-15T18:14:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8c005e40 con 0x7fed940aef20 2024-01-15T18:14:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed940b3680 con 0x7fed940aef20 2024-01-15T18:14:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c0072d0 con 0x7fed940aef20 2024-01-15T18:14:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.369+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed940b3c40 con 0x7fed940aef20 2024-01-15T18:14:01.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.370+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed94074770 con 0x7fed940aef20 2024-01-15T18:14:01.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.371+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fed8c0074f0 con 0x7fed940aef20 2024-01-15T18:14:01.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.372+0000 7fed8be00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed6c06c480 0x7fed6c06e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.372+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fed8c087ff0 con 0x7fed940aef20 2024-01-15T18:14:01.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.373+0000 7fed93400640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed6c06c480 0x7fed6c06e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.374+0000 7fed93400640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed6c06c480 0x7fed6c06e940 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fed84036000 tx=0x7fed84030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:01.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.377+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed8c054610 con 0x7fed940aef20 2024-01-15T18:14:01.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.574+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed940d0b60 con 0x7fed940aef20 2024-01-15T18:14:01.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.575+0000 7fed8be00640 1 -- 172.21.15.2:0/2153940634 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed8c017020 con 0x7fed940aef20 2024-01-15T18:14:01.578 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:14:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.577+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed6c06c480 msgr2=0x7fed6c06e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed6c06c480 0x7fed6c06e940 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fed84036000 tx=0x7fed84030000 comp rx=0 tx=0).stop 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 msgr2=0x7fed940b2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 0x7fed940b2a50 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00c400 tx=0x7fed8c00c8c0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 shutdown_connections 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fed6c06c480 0x7fed6c06e940 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fed940aef20 0x7fed940b2a50 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fed94164cd0 0x7fed94162040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 --2- 172.21.15.2:0/2153940634 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fed9406abc0 0x7fed941f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 >> 172.21.15.2:0/2153940634 conn(0x7fed940cfd30 msgr2=0x7fed9406f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 shutdown_connections 2024-01-15T18:14:01.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.578+0000 7fed98e00640 1 -- 172.21.15.2:0/2153940634 wait complete. 2024-01-15T18:14:01.601 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:14:01.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.812+0000 7fdc4d000640 1 Processor -- start 2024-01-15T18:14:01.816 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.812+0000 7fdc4d000640 1 -- start start 2024-01-15T18:14:01.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc4d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc4806c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc4d000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdc480cc0a0 0x7fdc48070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc4d000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4806d250 con 0x7fdc4806cdf0 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc4d000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc48143dc0 con 0x7fdc480cc0a0 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc4d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4815b620 con 0x7fdc481439e0 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc46800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdc480cc0a0 0x7fdc48070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc46800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fdc480cc0a0 0x7fdc48070380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50014/0 (socket says 172.21.15.2:50014) 2024-01-15T18:14:01.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.814+0000 7fdc46800640 1 -- 172.21.15.2:0/3499720648 learned_addr learned my addr 172.21.15.2:0/3499720648 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:01.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.815+0000 7fdc45e00640 1 --2- 172.21.15.2:0/3499720648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc4806c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.815+0000 7fdc46800640 1 -- 172.21.15.2:0/3499720648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 msgr2=0x7fdc4806c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:01.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.815+0000 7fdc46800640 1 --2- 172.21.15.2:0/3499720648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc4806c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.815+0000 7fdc46800640 1 -- 172.21.15.2:0/3499720648 >> v1:172.21.15.137:6789/0 conn(0x7fdc4806cdf0 legacy=0x7fdc480cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.815+0000 7fdc46800640 1 -- 172.21.15.2:0/3499720648 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc480cd9f0 con 0x7fdc480cc0a0 2024-01-15T18:14:01.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.816+0000 7fdc46800640 1 --2- 172.21.15.2:0/3499720648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc480cc0a0 0x7fdc48070380 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009e50 tx=0x7fdc4002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0c808ff5683048 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:01.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.816+0000 7fdc44a00640 1 -- 172.21.15.2:0/3499720648 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4002cb00 con 0x7fdc480cc0a0 2024-01-15T18:14:01.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.816+0000 7fdc44a00640 1 -- 172.21.15.2:0/3499720648 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4002cc60 con 0x7fdc480cc0a0 2024-01-15T18:14:01.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.816+0000 7fdc44a00640 1 -- 172.21.15.2:0/3499720648 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc400343f0 con 0x7fdc480cc0a0 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 -- 172.21.15.2:0/3499720648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc480cc0a0 msgr2=0x7fdc48070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3499720648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc480cc0a0 0x7fdc48070380 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fdc40009e50 tx=0x7fdc4002bf60 comp rx=0 tx=0).stop 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 -- 172.21.15.2:0/3499720648 shutdown_connections 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3499720648 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc480cc0a0 0x7fdc48070380 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3499720648 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc4806c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 -- 172.21.15.2:0/3499720648 >> 172.21.15.2:0/3499720648 conn(0x7fdc480b46d0 msgr2=0x7fdc480b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 -- 172.21.15.2:0/3499720648 shutdown_connections 2024-01-15T18:14:01.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.817+0000 7fdc4d000640 1 -- 172.21.15.2:0/3499720648 wait complete. 2024-01-15T18:14:01.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.819+0000 7fdc4d000640 1 Processor -- start 2024-01-15T18:14:01.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.819+0000 7fdc4d000640 1 -- start start 2024-01-15T18:14:01.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc4d000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 0x7fdc4813b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc4d000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc480cc0a0 0x7fdc4814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc4d000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc481e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc4d000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4814b850 con 0x7fdc4806cdf0 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc4d000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4813e800 con 0x7fdc480cc0a0 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc4d000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4813bf40 con 0x7fdc481439e0 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc45e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 0x7fdc4813b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc45e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 0x7fdc4813b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50016/0 (socket says 172.21.15.2:50016) 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.820+0000 7fdc45e00640 1 -- 172.21.15.2:0/3249125422 learned_addr learned my addr 172.21.15.2:0/3249125422 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc45e00640 1 -- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 msgr2=0x7fdc481e6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc46800640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc481e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc45e00640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc481e6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc45e00640 1 -- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc480cc0a0 msgr2=0x7fdc4814b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc45e00640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc480cc0a0 0x7fdc4814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc45e00640 1 -- 172.21.15.2:0/3249125422 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc40009b30 con 0x7fdc4806cdf0 2024-01-15T18:14:01.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc45e00640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 0x7fdc4813b930 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c7f0 tx=0x7fdc3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:01.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38004220 con 0x7fdc4806cdf0 2024-01-15T18:14:01.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc38004380 con 0x7fdc4806cdf0 2024-01-15T18:14:01.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc38007840 con 0x7fdc4806cdf0 2024-01-15T18:14:01.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.821+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4813e1b0 con 0x7fdc4806cdf0 2024-01-15T18:14:01.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.822+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4813dea0 con 0x7fdc4806cdf0 2024-01-15T18:14:01.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.823+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc480686f0 con 0x7fdc4806cdf0 2024-01-15T18:14:01.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.824+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fdc38007bb0 con 0x7fdc4806cdf0 2024-01-15T18:14:01.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.825+0000 7fdc3e000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc2006c430 0x7fdc2006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:01.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.825+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fdc380889f0 con 0x7fdc4806cdf0 2024-01-15T18:14:01.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.826+0000 7fdc45400640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc2006c430 0x7fdc2006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:01.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.827+0000 7fdc45400640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc2006c430 0x7fdc2006e8f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdc30000f60 tx=0x7fdc300079d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:01.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:01.830+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdc38054f60 con 0x7fdc4806cdf0 2024-01-15T18:14:02.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.041+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdc480cb340 con 0x7fdc4806cdf0 2024-01-15T18:14:02.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.042+0000 7fdc3e000640 1 -- 172.21.15.2:0/3249125422 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v106) v1 ==== 80+0+122 (secure 0 0 0) 0x7fdc3801e070 con 0x7fdc4806cdf0 2024-01-15T18:14:02.046 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.137:6837/320323477 2024-01-16T18:13:57.259474+0000 2024-01-15T18:14:02.046 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.137:6835/320323477 2024-01-16T18:13:57.259474+0000 2024-01-15T18:14:02.046 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:14:02.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.044+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc2006c430 msgr2=0x7fdc2006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.044+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc2006c430 0x7fdc2006e8f0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fdc30000f60 tx=0x7fdc300079d0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.044+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 msgr2=0x7fdc4813b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.044+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 0x7fdc4813b930 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800c7f0 tx=0x7fdc3800ccb0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 shutdown_connections 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdc2006c430 0x7fdc2006e8f0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdc481439e0 0x7fdc481e6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdc480cc0a0 0x7fdc4814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 --2- 172.21.15.2:0/3249125422 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdc4806cdf0 0x7fdc4813b930 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 >> 172.21.15.2:0/3249125422 conn(0x7fdc480b46d0 msgr2=0x7fdc480b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:02.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 shutdown_connections 2024-01-15T18:14:02.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.045+0000 7fdc4d000640 1 -- 172.21.15.2:0/3249125422 wait complete. 2024-01-15T18:14:02.069 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:14:02.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:02.069 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:02.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:14:02.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:02.172 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:14:02.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[72848]: fuse finished with error 0 and tester_r 0 2024-01-15T18:14:02.391 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:14:02.391 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:02.448 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:02.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.671+0000 7fce59600640 1 Processor -- start 2024-01-15T18:14:02.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.671+0000 7fce59600640 1 -- start start 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.672+0000 7fce59600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fce5415b3d0 0x7fce540d1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce59600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cfb10 0x7fce540b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce59600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5415b7b0 con 0x7fce540cc3f0 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce59600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce540cef10 con 0x7fce5415b3d0 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce59600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce540cc9b0 con 0x7fce540cfb10 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fce5415b3d0 0x7fce540d1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fce5415b3d0 0x7fce540d1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50032/0 (socket says 172.21.15.2:50032) 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 -- 172.21.15.2:0/2466734782 learned_addr learned my addr 172.21.15.2:0/2466734782 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 -- 172.21.15.2:0/2466734782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cfb10 msgr2=0x7fce540b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 --2- 172.21.15.2:0/2466734782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cfb10 0x7fce540b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 -- 172.21.15.2:0/2466734782 >> v1:172.21.15.137:6789/0 conn(0x7fce540cc3f0 legacy=0x7fce540cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:02.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.673+0000 7fce53e00640 1 -- 172.21.15.2:0/2466734782 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce540ccc90 con 0x7fce5415b3d0 2024-01-15T18:14:02.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.674+0000 7fce53e00640 1 --2- 172.21.15.2:0/2466734782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce5415b3d0 0x7fce540d1940 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fce44009e50 tx=0x7fce4402be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ea60f1fd5c1f92 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:02.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.675+0000 7fce52a00640 1 -- 172.21.15.2:0/2466734782 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4402ca10 con 0x7fce5415b3d0 2024-01-15T18:14:02.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.675+0000 7fce52a00640 1 -- 172.21.15.2:0/2466734782 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4402cb70 con 0x7fce5415b3d0 2024-01-15T18:14:02.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.675+0000 7fce52a00640 1 -- 172.21.15.2:0/2466734782 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440343f0 con 0x7fce5415b3d0 2024-01-15T18:14:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.675+0000 7fce59600640 1 -- 172.21.15.2:0/2466734782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce5415b3d0 msgr2=0x7fce540d1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.675+0000 7fce59600640 1 --2- 172.21.15.2:0/2466734782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce5415b3d0 0x7fce540d1940 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fce44009e50 tx=0x7fce4402be70 comp rx=0 tx=0).stop 2024-01-15T18:14:02.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.675+0000 7fce59600640 1 -- 172.21.15.2:0/2466734782 shutdown_connections 2024-01-15T18:14:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.676+0000 7fce59600640 1 --2- 172.21.15.2:0/2466734782 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cfb10 0x7fce540b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.676+0000 7fce59600640 1 --2- 172.21.15.2:0/2466734782 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce5415b3d0 0x7fce540d1940 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.676+0000 7fce59600640 1 -- 172.21.15.2:0/2466734782 >> 172.21.15.2:0/2466734782 conn(0x7fce540d0e20 msgr2=0x7fce540d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.676+0000 7fce59600640 1 -- 172.21.15.2:0/2466734782 shutdown_connections 2024-01-15T18:14:02.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.676+0000 7fce59600640 1 -- 172.21.15.2:0/2466734782 wait complete. 2024-01-15T18:14:02.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.677+0000 7fce59600640 1 Processor -- start 2024-01-15T18:14:02.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.677+0000 7fce59600640 1 -- start start 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.678+0000 7fce59600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cc3f0 0x7fce54156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.678+0000 7fce59600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 0x7fce5406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.678+0000 7fce59600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce5415b3d0 0x7fce5406bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.678+0000 7fce59600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540c5160 con 0x7fce540cfb10 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.678+0000 7fce59600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce540c8120 con 0x7fce5415b3d0 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.678+0000 7fce59600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce540c4950 con 0x7fce540cc3f0 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 0x7fce5406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 0x7fce5406b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50044/0 (socket says 172.21.15.2:50044) 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 -- 172.21.15.2:0/979229104 learned_addr learned my addr 172.21.15.2:0/979229104 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce58c00640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce5415b3d0 0x7fce5406bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53e00640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cc3f0 0x7fce54156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 -- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cc3f0 msgr2=0x7fce54156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cc3f0 0x7fce54156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 -- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce5415b3d0 msgr2=0x7fce5406bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce5415b3d0 0x7fce5406bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce53400640 1 -- 172.21.15.2:0/979229104 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce44009b30 con 0x7fce540cfb10 2024-01-15T18:14:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.679+0000 7fce58c00640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce5415b3d0 0x7fce5406bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:02.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.680+0000 7fce53400640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 0x7fce5406b5c0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c410 tx=0x7fce3c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.680+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0035f0 con 0x7fce540cfb10 2024-01-15T18:14:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.680+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3c003750 con 0x7fce540cfb10 2024-01-15T18:14:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.680+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c003b40 con 0x7fce540cfb10 2024-01-15T18:14:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.680+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce540c4ba0 con 0x7fce540cfb10 2024-01-15T18:14:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.680+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce540c8330 con 0x7fce540cfb10 2024-01-15T18:14:02.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.681+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce540ce720 con 0x7fce540cfb10 2024-01-15T18:14:02.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.682+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fce3c003ca0 con 0x7fce540cfb10 2024-01-15T18:14:02.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.683+0000 7fce4be00640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce2c06c4b0 0x7fce2c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:02.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.683+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fce3c088280 con 0x7fce540cfb10 2024-01-15T18:14:02.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.684+0000 7fce53e00640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce2c06c4b0 0x7fce2c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:02.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.685+0000 7fce53e00640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce2c06c4b0 0x7fce2c06e970 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fce44035000 tx=0x7fce44030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:02.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.687+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce3c054760 con 0x7fce540cfb10 2024-01-15T18:14:02.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.886+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce54074770 con 0x7fce540cfb10 2024-01-15T18:14:02.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.888+0000 7fce4be00640 1 -- 172.21.15.2:0/979229104 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+13157 (secure 0 0 0) 0x7fce3c017030 con 0x7fce540cfb10 2024-01-15T18:14:02.892 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:02.892 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":106,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:00.272802+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6837/320323477":"2024-01-16T18:13:57.259474+0000","172.21.15.137:6835/320323477":"2024-01-16T18:13:57.259474+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:02.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.890+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce2c06c4b0 msgr2=0x7fce2c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.890+0000 7fce59600640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce2c06c4b0 0x7fce2c06e970 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fce44035000 tx=0x7fce44030000 comp rx=0 tx=0).stop 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 msgr2=0x7fce5406b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 0x7fce5406b5c0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00c410 tx=0x7fce3c00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 shutdown_connections 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fce2c06c4b0 0x7fce2c06e970 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fce5415b3d0 0x7fce5406bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fce540cfb10 0x7fce5406b5c0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 --2- 172.21.15.2:0/979229104 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fce540cc3f0 0x7fce54156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 >> 172.21.15.2:0/979229104 conn(0x7fce540d0e20 msgr2=0x7fce5406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 shutdown_connections 2024-01-15T18:14:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:02.891+0000 7fce59600640 1 -- 172.21.15.2:0/979229104 wait complete. 2024-01-15T18:14:02.914 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:14:02.914 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:14:03.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.133+0000 7efc89200640 1 Processor -- start 2024-01-15T18:14:03.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.134+0000 7efc89200640 1 -- start start 2024-01-15T18:14:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.135+0000 7efc89200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7efc8415d270 0x7efc840b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.136+0000 7efc89200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8406c5e0 con 0x7efc84161370 2024-01-15T18:14:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.136+0000 7efc89200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8406c8c0 con 0x7efc84161890 2024-01-15T18:14:03.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.136+0000 7efc89200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc84071c00 con 0x7efc8415d270 2024-01-15T18:14:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.136+0000 7efc83e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7efc84161370 0x7efc8406cd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42656/0 (socket says 172.21.15.2:42656) 2024-01-15T18:14:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.136+0000 7efc83e00640 1 -- 172.21.15.2:0/125241917 learned_addr learned my addr 172.21.15.2:0/125241917 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.136+0000 7efc83400640 1 --2- 172.21.15.2:0/125241917 >> v2:172.21.15.137:3300/0 conn(0x7efc8415d270 0x7efc840b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856657855 0 0) 0x7efc8406c5e0 con 0x7efc84161370 2024-01-15T18:14:03.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc64003520 con 0x7efc84161370 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3585643113 0 0) 0x7efc64003520 con 0x7efc84161370 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 >> v2:172.21.15.137:3300/0 conn(0x7efc8415d270 msgr2=0x7efc840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 --2- 172.21.15.2:0/125241917 >> v2:172.21.15.137:3300/0 conn(0x7efc8415d270 0x7efc840b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 >> v1:172.21.15.137:6789/0 conn(0x7efc84161890 legacy=0x7efc84160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc84071ee0 con 0x7efc84161370 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7efc7c0019f0 con 0x7efc84161370 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 >> v1:172.21.15.2:6789/0 conn(0x7efc84161370 legacy=0x7efc8406cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64003970 0x7efc64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc640047c0 0x7efc64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640052a0 0x7efc6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc84071c00 con 0x7efc640052a0 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6400c170 con 0x7efc640047c0 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.137+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6400c330 con 0x7efc64003970 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83e00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640052a0 0x7efc6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83400640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64003970 0x7efc64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc7be00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc640047c0 0x7efc64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83e00640 1 -- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64003970 msgr2=0x7efc64003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83e00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc64003970 0x7efc64003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83e00640 1 -- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc640047c0 msgr2=0x7efc64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83e00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc640047c0 0x7efc64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.138+0000 7efc83e00640 1 -- 172.21.15.2:0/125241917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc64003520 con 0x7efc640052a0 2024-01-15T18:14:03.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.139+0000 7efc83e00640 1 -- 172.21.15.2:0/125241917 reap_dead start 2024-01-15T18:14:03.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.139+0000 7efc83e00640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640052a0 0x7efc6400b980 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7efc840b4010 tx=0x7efc640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937dd14a0479c039 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.140+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c0072d0 con 0x7efc640052a0 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.140+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7c02f070 con 0x7efc640052a0 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.140+0000 7efc82a00640 1 -- 172.21.15.2:0/125241917 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7c007670 con 0x7efc640052a0 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.140+0000 7efc89200640 1 -- 172.21.15.2:0/125241917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640052a0 msgr2=0x7efc6400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.140+0000 7efc89200640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640052a0 0x7efc6400b980 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7efc840b4010 tx=0x7efc640057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.140+0000 7efc89200640 1 -- 172.21.15.2:0/125241917 shutdown_connections 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.141+0000 7efc89200640 1 --2- 172.21.15.2:0/125241917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc640052a0 0x7efc6400b980 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.141+0000 7efc89200640 1 -- 172.21.15.2:0/125241917 >> 172.21.15.2:0/125241917 conn(0x7efc840b0840 msgr2=0x7efc840b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.141+0000 7efc89200640 1 -- 172.21.15.2:0/125241917 shutdown_connections 2024-01-15T18:14:03.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.141+0000 7efc89200640 1 -- 172.21.15.2:0/125241917 wait complete. 2024-01-15T18:14:03.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.142+0000 7efc89200640 1 Processor -- start 2024-01-15T18:14:03.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.143+0000 7efc89200640 1 -- start start 2024-01-15T18:14:03.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc89200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640052a0 0x7efc8414c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc89200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 0x7efc8415db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc89200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc8415b180 0x7efc8415b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc89200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8415e0b0 con 0x7efc8415d6f0 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc89200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8415bed0 con 0x7efc8415b180 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc89200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc840ac9a0 con 0x7efc640052a0 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc7be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 0x7efc8415db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc7be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 0x7efc8415db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44986/0 (socket says 172.21.15.2:44986) 2024-01-15T18:14:03.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc7be00640 1 -- 172.21.15.2:0/481550500 learned_addr learned my addr 172.21.15.2:0/481550500 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:03.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc83400640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640052a0 0x7efc8414c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc83e00640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc8415b180 0x7efc8415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.144+0000 7efc7be00640 1 -- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640052a0 msgr2=0x7efc8414c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7be00640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640052a0 0x7efc8414c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7be00640 1 -- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc8415b180 msgr2=0x7efc8415b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7be00640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc8415b180 0x7efc8415b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7be00640 1 -- 172.21.15.2:0/481550500 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc840acc80 con 0x7efc8415d6f0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc83e00640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc8415b180 0x7efc8415b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7be00640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 0x7efc8415db70 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00e480 tx=0x7efc6c00e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0052e0 con 0x7efc8415d6f0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c005440 con 0x7efc8415d6f0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c003830 con 0x7efc8415d6f0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc840acf60 con 0x7efc8415d6f0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.145+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc8414cb80 con 0x7efc8415d6f0 2024-01-15T18:14:03.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.146+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc84074770 con 0x7efc8415d6f0 2024-01-15T18:14:03.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.147+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7efc6c003990 con 0x7efc8415d6f0 2024-01-15T18:14:03.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.149+0000 7efc7b400640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc5c06c430 0x7efc5c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.149+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7efc6c088900 con 0x7efc8415d6f0 2024-01-15T18:14:03.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.150+0000 7efc83400640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc5c06c430 0x7efc5c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.151+0000 7efc83400640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc5c06c430 0x7efc5c06e8f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7efc74009a80 tx=0x7efc74008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.154+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc6c054e70 con 0x7efc8415d6f0 2024-01-15T18:14:03.346 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:03.354 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:03.355 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.347+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efc84158170 con 0x7efc8415d6f0 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.348+0000 7efc7b400640 1 -- 172.21.15.2:0/481550500 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7efc6c059e60 con 0x7efc8415d6f0 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc5c06c430 msgr2=0x7efc5c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc5c06c430 0x7efc5c06e8f0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7efc74009a80 tx=0x7efc74008040 comp rx=0 tx=0).stop 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 msgr2=0x7efc8415db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 0x7efc8415db70 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00e480 tx=0x7efc6c00e940 comp rx=0 tx=0).stop 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 shutdown_connections 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efc5c06c430 0x7efc5c06e8f0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efc8415b180 0x7efc8415b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efc8415d6f0 0x7efc8415db70 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 --2- 172.21.15.2:0/481550500 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efc640052a0 0x7efc8414c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 >> 172.21.15.2:0/481550500 conn(0x7efc840b0840 msgr2=0x7efc840b2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:03.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 shutdown_connections 2024-01-15T18:14:03.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.351+0000 7efc89200640 1 -- 172.21.15.2:0/481550500 wait complete. 2024-01-15T18:14:03.374 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:14:03.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.631+0000 7fd04b800640 1 Processor -- start 2024-01-15T18:14:03.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.631+0000 7fd04b800640 1 -- start start 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.633+0000 7fd04b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd0440cb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.633+0000 7fd04b800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd0440cbed0 0x7fd044070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.633+0000 7fd04b800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd044143dc0 con 0x7fd0441439e0 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.633+0000 7fd04b800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd04415b620 con 0x7fd0440cbed0 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.633+0000 7fd04b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0440cd9f0 con 0x7fd04406cfc0 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.633+0000 7fd048e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd0441439e0 0x7fd04406c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42672/0 (socket says 172.21.15.2:42672) 2024-01-15T18:14:03.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd048e00640 1 -- 172.21.15.2:0/2310562116 learned_addr learned my addr 172.21.15.2:0/2310562116 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd049800640 1 --2- 172.21.15.2:0/2310562116 >> v2:172.21.15.137:3300/0 conn(0x7fd0440cbed0 0x7fd044070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd0440cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765430718 0 0) 0x7fd044143dc0 con 0x7fd0441439e0 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024003520 con 0x7fd0441439e0 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1615138593 0 0) 0x7fd024003520 con 0x7fd0441439e0 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 msgr2=0x7fd0440cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd0440cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 >> v2:172.21.15.137:3300/0 conn(0x7fd0440cbed0 msgr2=0x7fd044070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 --2- 172.21.15.2:0/2310562116 >> v2:172.21.15.137:3300/0 conn(0x7fd0440cbed0 0x7fd044070380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0440cc570 con 0x7fd0441439e0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.634+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd0340019f0 con 0x7fd0441439e0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 >> v1:172.21.15.2:6789/0 conn(0x7fd0441439e0 legacy=0x7fd04406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd024003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd024003ef0 0x7fd0240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd024004d00 0x7fd024005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04415b620 con 0x7fd024003ef0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0440cd9f0 con 0x7fd024004d00 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02400b980 con 0x7fd04406cfc0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd024003ef0 0x7fd0240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd049800640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd024004d00 0x7fd024005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd048e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd024003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043e00640 1 -- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 msgr2=0x7fd024003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd024003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043e00640 1 -- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd024004d00 msgr2=0x7fd024005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:03.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd024004d00 0x7fd024005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd043e00640 1 -- 172.21.15.2:0/2310562116 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024003520 con 0x7fd024003ef0 2024-01-15T18:14:03.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd049800640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd024004d00 0x7fd024005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:03.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.635+0000 7fd048e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd04406cfc0 0x7fd024003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd043e00640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd024003ef0 0x7fd0240047c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009ac0 tx=0x7fd0240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90f6120dd29c826 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c02c6b0 con 0x7fd024003ef0 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c02c810 con 0x7fd024003ef0 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd043400640 1 -- 172.21.15.2:0/2310562116 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c02cba0 con 0x7fd024003ef0 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd04b800640 1 -- 172.21.15.2:0/2310562116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd024003ef0 msgr2=0x7fd0240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd04b800640 1 --2- 172.21.15.2:0/2310562116 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd024003ef0 0x7fd0240047c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009ac0 tx=0x7fd0240057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.637+0000 7fd049800640 1 -- 172.21.15.2:0/2310562116 reap_dead start 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.638+0000 7fd04b800640 1 -- 172.21.15.2:0/2310562116 shutdown_connections 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.638+0000 7fd04b800640 1 -- 172.21.15.2:0/2310562116 >> 172.21.15.2:0/2310562116 conn(0x7fd0440b46d0 msgr2=0x7fd0440b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.638+0000 7fd04b800640 1 -- 172.21.15.2:0/2310562116 shutdown_connections 2024-01-15T18:14:03.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.638+0000 7fd04b800640 1 -- 172.21.15.2:0/2310562116 wait complete. 2024-01-15T18:14:03.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.640+0000 7fd04b800640 1 Processor -- start 2024-01-15T18:14:03.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.640+0000 7fd04b800640 1 -- start start 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd04b800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd024003ef0 0x7fd044138870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd04b800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 0x7fd044157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd04b800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd044158490 0x7fd044158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd04b800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd044157ba0 con 0x7fd0441571e0 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd04b800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd044159180 con 0x7fd044158490 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd04b800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd044159460 con 0x7fd024003ef0 2024-01-15T18:14:03.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd043e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 0x7fd044157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd043e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 0x7fd044157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45012/0 (socket says 172.21.15.2:45012) 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.641+0000 7fd043e00640 1 -- 172.21.15.2:0/3817663083 learned_addr learned my addr 172.21.15.2:0/3817663083 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd048e00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd024003ef0 0x7fd044138870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd049800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd044158490 0x7fd044158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd043e00640 1 -- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd024003ef0 msgr2=0x7fd044138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd043e00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd024003ef0 0x7fd044138870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd043e00640 1 -- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd044158490 msgr2=0x7fd044158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd043e00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd044158490 0x7fd044158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd043e00640 1 -- 172.21.15.2:0/3817663083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd044159740 con 0x7fd0441571e0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd043e00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 0x7fd044157660 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd02c02c2b0 tx=0x7fd02c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c003810 con 0x7fd0441571e0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c003970 con 0x7fd0441571e0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c003ce0 con 0x7fd0441571e0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.642+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd044159a20 con 0x7fd0441571e0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.643+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd044158e80 con 0x7fd0441571e0 2024-01-15T18:14:03.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.644+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04415b3b0 con 0x7fd0441571e0 2024-01-15T18:14:03.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.645+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd02c047450 con 0x7fd0441571e0 2024-01-15T18:14:03.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.646+0000 7fd040c00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd01c06c430 0x7fd01c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:03.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.646+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fd02c0aac80 con 0x7fd0441571e0 2024-01-15T18:14:03.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.647+0000 7fd048e00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd01c06c430 0x7fd01c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:03.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.648+0000 7fd048e00640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd01c06c430 0x7fd01c06e8f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd034002230 tx=0x7fd034009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:03.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.650+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd02c077300 con 0x7fd0441571e0 2024-01-15T18:14:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.866+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd044157ba0 con 0x7fd0441571e0 2024-01-15T18:14:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.867+0000 7fd040c00640 1 -- 172.21.15.2:0/3817663083 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v106) v1 ==== 80+0+122 (secure 0 0 0) 0x7fd02c04c090 con 0x7fd0441571e0 2024-01-15T18:14:03.870 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.137:6837/320323477 2024-01-16T18:13:57.259474+0000 2024-01-15T18:14:03.871 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.15.137:6835/320323477 2024-01-16T18:13:57.259474+0000 2024-01-15T18:14:03.871 INFO:teuthology.orchestra.run.smithi002.stderr:listed 2 entries 2024-01-15T18:14:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.869+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd01c06c430 msgr2=0x7fd01c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.869+0000 7fd04b800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd01c06c430 0x7fd01c06e8f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd034002230 tx=0x7fd034009040 comp rx=0 tx=0).stop 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.869+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 msgr2=0x7fd044157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.869+0000 7fd04b800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 0x7fd044157660 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fd02c02c2b0 tx=0x7fd02c004730 comp rx=0 tx=0).stop 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.869+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 shutdown_connections 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd01c06c430 0x7fd01c06e8f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd044158490 0x7fd044158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0441571e0 0x7fd044157660 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 --2- 172.21.15.2:0/3817663083 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd024003ef0 0x7fd044138870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 >> 172.21.15.2:0/3817663083 conn(0x7fd0440b46d0 msgr2=0x7fd0440b2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 shutdown_connections 2024-01-15T18:14:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:03.870+0000 7fd04b800640 1 -- 172.21.15.2:0/3817663083 wait complete. 2024-01-15T18:14:03.893 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:14:03.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:03.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:03.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[46689]: fuse finished with error 0 and tester_r 0 2024-01-15T18:14:03.985 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:14:03.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:03.985 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:14:04.159 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:14:04.160 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:04.218 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:14:04.218 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:14:04.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:04.218 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-15T18:14:04.269 INFO:teuthology.orchestra.run.smithi002.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-15T18:14:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:14:04.270 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:14:04.270 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:04.327 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:14:04.327 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:14:04.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:04.327 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-15T18:14:04.380 INFO:teuthology.orchestra.run.smithi137.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-15T18:14:04.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:14:04.381 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:14:04.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.608+0000 7fcc63e00640 1 Processor -- start 2024-01-15T18:14:04.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.608+0000 7fcc63e00640 1 -- start start 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc63e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcc64161c90 0x7fcc641620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc63e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcc640700d0 0x7fcc640704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc63e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc64070a20 con 0x7fcc6414af80 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc63e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640b28f0 con 0x7fcc640700d0 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc63e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc640b1c70 con 0x7fcc64161c90 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc62a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcc6414af80 0x7fcc6406d520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42678/0 (socket says 172.21.15.2:42678) 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc62a00640 1 -- 172.21.15.2:0/3517393746 learned_addr learned my addr 172.21.15.2:0/3517393746 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:04.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.610+0000 7fcc63400640 1 --2- 172.21.15.2:0/3517393746 >> v2:172.21.15.2:3300/0 conn(0x7fcc640700d0 0x7fcc640704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:04.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.611+0000 7fcc63400640 1 -- 172.21.15.2:0/3517393746 >> v2:172.21.15.137:3300/0 conn(0x7fcc64161c90 msgr2=0x7fcc641620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:04.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.611+0000 7fcc63400640 1 --2- 172.21.15.2:0/3517393746 >> v2:172.21.15.137:3300/0 conn(0x7fcc64161c90 0x7fcc641620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.611+0000 7fcc63400640 1 -- 172.21.15.2:0/3517393746 >> v1:172.21.15.2:6789/0 conn(0x7fcc6414af80 legacy=0x7fcc6406d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:04.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.611+0000 7fcc63400640 1 -- 172.21.15.2:0/3517393746 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc640b1f50 con 0x7fcc640700d0 2024-01-15T18:14:04.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.611+0000 7fcc61600640 1 -- 172.21.15.2:0/3517393746 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827391992 0 0) 0x7fcc64070a20 con 0x7fcc6414af80 2024-01-15T18:14:04.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.612+0000 7fcc63400640 1 --2- 172.21.15.2:0/3517393746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc640704e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fcc640b1800 tx=0x7fcc5c02b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afec46ddc292d7d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:04.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.612+0000 7fcc61600640 1 -- 172.21.15.2:0/3517393746 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c031520 con 0x7fcc640700d0 2024-01-15T18:14:04.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.612+0000 7fcc61600640 1 -- 172.21.15.2:0/3517393746 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc5c031680 con 0x7fcc640700d0 2024-01-15T18:14:04.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc61600640 1 -- 172.21.15.2:0/3517393746 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c031a10 con 0x7fcc640700d0 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 -- 172.21.15.2:0/3517393746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 msgr2=0x7fcc640704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 --2- 172.21.15.2:0/3517393746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc640704e0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7fcc640b1800 tx=0x7fcc5c02b7c0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 -- 172.21.15.2:0/3517393746 shutdown_connections 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 --2- 172.21.15.2:0/3517393746 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc640704e0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 --2- 172.21.15.2:0/3517393746 >> v2:172.21.15.137:3300/0 conn(0x7fcc64161c90 0x7fcc641620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 -- 172.21.15.2:0/3517393746 >> 172.21.15.2:0/3517393746 conn(0x7fcc640b0840 msgr2=0x7fcc640b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 -- 172.21.15.2:0/3517393746 shutdown_connections 2024-01-15T18:14:04.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.613+0000 7fcc63e00640 1 -- 172.21.15.2:0/3517393746 wait complete. 2024-01-15T18:14:04.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.615+0000 7fcc63e00640 1 Processor -- start 2024-01-15T18:14:04.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.615+0000 7fcc63e00640 1 -- start start 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc641607b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc6414af80 0x7fcc640af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc63e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc64161c90 0x7fcc641fa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc63e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64142bc0 con 0x7fcc640700d0 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc63e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64142320 con 0x7fcc64161c90 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc63e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc64142600 con 0x7fcc6414af80 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.616+0000 7fcc62a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc641607b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc641607b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45024/0 (socket says 172.21.15.2:45024) 2024-01-15T18:14:04.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 -- 172.21.15.2:0/476557175 learned_addr learned my addr 172.21.15.2:0/476557175 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc63400640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc64161c90 0x7fcc641fa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62000640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc6414af80 0x7fcc640af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 -- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc6414af80 msgr2=0x7fcc640af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc6414af80 0x7fcc640af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 -- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc64161c90 msgr2=0x7fcc641fa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc64161c90 0x7fcc641fa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 -- 172.21.15.2:0/476557175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c009400 con 0x7fcc640700d0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.617+0000 7fcc62a00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc641607b0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00c430 tx=0x7fcc4c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.618+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c0074e0 con 0x7fcc640700d0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.618+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc4c007660 con 0x7fcc640700d0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.618+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c004a20 con 0x7fcc640700d0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.618+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc641428e0 con 0x7fcc640700d0 2024-01-15T18:14:04.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.618+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc64160cf0 con 0x7fcc640700d0 2024-01-15T18:14:04.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.619+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc641fa540 con 0x7fcc640700d0 2024-01-15T18:14:04.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.620+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fcc4c004b80 con 0x7fcc640700d0 2024-01-15T18:14:04.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.621+0000 7fcc5aa00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcc3c06c480 0x7fcc3c06e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:04.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.621+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fcc4c088970 con 0x7fcc640700d0 2024-01-15T18:14:04.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.622+0000 7fcc62000640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcc3c06c480 0x7fcc3c06e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:04.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.623+0000 7fcc62000640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcc3c06c480 0x7fcc3c06e940 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcc641fa130 tx=0x7fcc540094f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:04.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.626+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc641fa540 con 0x7fcc640700d0 2024-01-15T18:14:04.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.860+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc64142d00 con 0x7fcc640700d0 2024-01-15T18:14:04.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.861+0000 7fcc5aa00640 1 -- 172.21.15.2:0/476557175 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fcc4c054ee0 con 0x7fcc640700d0 2024-01-15T18:14:04.865 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:04.865 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3406786991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3406786991},{"type":"v1","addr":"172.21.15.2:6835","nonce":3406786991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/739459094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":739459094},{"type":"v1","addr":"172.21.15.137:6836","nonce":739459094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7132,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/2680407877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2680407877},{"type":"v1","addr":"172.21.15.137:6837","nonce":2680407877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":8565,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/2674783053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2674783053},{"type":"v1","addr":"172.21.15.2:6834","nonce":2674783053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2024-01-15T18:14:04.866 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 68 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcc3c06c480 msgr2=0x7fcc3c06e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcc3c06c480 0x7fcc3c06e940 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fcc641fa130 tx=0x7fcc540094f0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 msgr2=0x7fcc641607b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc641607b0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00c430 tx=0x7fcc4c00c8f0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 shutdown_connections 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcc3c06c480 0x7fcc3c06e940 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcc64161c90 0x7fcc641fa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcc6414af80 0x7fcc640af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 --2- 172.21.15.2:0/476557175 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcc640700d0 0x7fcc641607b0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:04.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 >> 172.21.15.2:0/476557175 conn(0x7fcc640b0840 msgr2=0x7fcc64164950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:04.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 shutdown_connections 2024-01-15T18:14:04.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:04.864+0000 7fcc63e00640 1 -- 172.21.15.2:0/476557175 wait complete. 2024-01-15T18:14:04.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-15T18:14:05.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.105+0000 7f52ebe00640 1 Processor -- start 2024-01-15T18:14:05.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.105+0000 7f52ebe00640 1 -- start start 2024-01-15T18:14:05.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.106+0000 7f52ebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 0x7f52ec070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52ebe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0ccd00 con 0x7f52ec15b6b0 2024-01-15T18:14:05.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52ebe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52ec0adc70 con 0x7f52ec0cf310 2024-01-15T18:14:05.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52ebe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0cc0f0 con 0x7f52ec15b2d0 2024-01-15T18:14:05.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52ea000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f52ec15b6b0 0x7f52ec0cb8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42680/0 (socket says 172.21.15.2:42680) 2024-01-15T18:14:05.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52ea000640 1 -- 172.21.15.2:0/602344091 learned_addr learned my addr 172.21.15.2:0/602344091 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:05.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52eaa00640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 0x7f52ec070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.107+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422563619 0 0) 0x7f52ec0ccd00 con 0x7f52ec15b6b0 2024-01-15T18:14:05.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52cc003520 con 0x7f52ec15b6b0 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190933188 0 0) 0x7f52cc003520 con 0x7f52ec15b6b0 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 msgr2=0x7f52ec070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 0x7f52ec070000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 >> v1:172.21.15.137:6789/0 conn(0x7f52ec0cf310 legacy=0x7f52ec0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0cc3d0 con 0x7f52ec15b6b0 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f52d4001670 con 0x7f52ec15b6b0 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 >> v1:172.21.15.2:6789/0 conn(0x7f52ec15b6b0 legacy=0x7f52ec0cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0039a0 0x7f52cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52cc0047c0 0x7f52cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 0x7f52cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec0cc0f0 con 0x7f52cc0047c0 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc00ba30 con 0x7f52cc0039a0 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc00bbf0 con 0x7f52ec15b2d0 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.108+0000 7f52ea000640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52cc0047c0 0x7f52cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.109+0000 7f52eaa00640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0039a0 0x7f52cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.109+0000 7f52ea000640 1 -- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 msgr2=0x7f52cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.109+0000 7f52ea000640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec15b2d0 0x7f52cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.109+0000 7f52ea000640 1 -- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0039a0 msgr2=0x7f52cc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.109+0000 7f52ea000640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0039a0 0x7f52cc003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.109+0000 7f52ea000640 1 -- 172.21.15.2:0/602344091 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc003520 con 0x7f52cc0047c0 2024-01-15T18:14:05.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.110+0000 7f52ea000640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52cc0047c0 0x7f52cc004c40 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f52d400c210 tx=0x7f52cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd633c47c6c5acf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:05.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.110+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d4005ad0 con 0x7f52cc0047c0 2024-01-15T18:14:05.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.110+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d4005c30 con 0x7f52cc0047c0 2024-01-15T18:14:05.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.110+0000 7f52e9600640 1 -- 172.21.15.2:0/602344091 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d402c870 con 0x7f52cc0047c0 2024-01-15T18:14:05.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52ebe00640 1 -- 172.21.15.2:0/602344091 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52cc0047c0 msgr2=0x7f52cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52ebe00640 1 --2- 172.21.15.2:0/602344091 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52cc0047c0 0x7f52cc004c40 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f52d400c210 tx=0x7f52cc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52eb400640 1 -- 172.21.15.2:0/602344091 reap_dead start 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52ebe00640 1 -- 172.21.15.2:0/602344091 shutdown_connections 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52ebe00640 1 -- 172.21.15.2:0/602344091 >> 172.21.15.2:0/602344091 conn(0x7f52ec0d0ec0 msgr2=0x7f52ec0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52ebe00640 1 -- 172.21.15.2:0/602344091 shutdown_connections 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.111+0000 7f52ebe00640 1 -- 172.21.15.2:0/602344091 wait complete. 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.113+0000 7f52ebe00640 1 Processor -- start 2024-01-15T18:14:05.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.113+0000 7f52ebe00640 1 -- start start 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52ebe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0047c0 0x7f52ec0caba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52ebe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec06b5c0 0x7f52ec06cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52ebe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 0x7f52ec151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52ebe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec06d2f0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52ebe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec06bd10 con 0x7f52cc0047c0 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52ebe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52ec06bff0 con 0x7f52ec06b5c0 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52eb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 0x7f52ec151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52eb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 0x7f52ec151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45036/0 (socket says 172.21.15.2:45036) 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.114+0000 7f52eb400640 1 -- 172.21.15.2:0/3046629643 learned_addr learned my addr 172.21.15.2:0/3046629643 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eaa00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0047c0 0x7f52ec0caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52ea000640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec06b5c0 0x7f52ec06cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eb400640 1 -- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec06b5c0 msgr2=0x7f52ec06cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eb400640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec06b5c0 0x7f52ec06cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eb400640 1 -- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0047c0 msgr2=0x7f52ec0caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eb400640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0047c0 0x7f52ec0caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eb400640 1 -- 172.21.15.2:0/3046629643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec06c2d0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.115+0000 7f52eb400640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 0x7f52ec151380 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f52ec071c30 tx=0x7f52e400cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:05.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.116+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e4005d40 con 0x7f52ec0c88f0 2024-01-15T18:14:05.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.116+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52e400ced0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.116+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52e400a850 con 0x7f52ec0c88f0 2024-01-15T18:14:05.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.116+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52ec06c5b0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.116+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52ec06ba20 con 0x7f52ec0c88f0 2024-01-15T18:14:05.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.117+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52ec06cad0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.118+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f52e400a9b0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.119+0000 7f52e2a00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f52c406c430 0x7f52c406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.119+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f52e4087cb0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.120+0000 7f52eaa00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f52c406c430 0x7f52c406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.121+0000 7f52eaa00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f52c406c430 0x7f52c406e8f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009a30 tx=0x7f52dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:05.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.123+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f52ec06cad0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.318+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f52ec06cad0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.324+0000 7f52e2a00640 1 -- 172.21.15.2:0/3046629643 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7f52ec06cad0 con 0x7f52ec0c88f0 2024-01-15T18:14:05.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.326+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f52c406c430 msgr2=0x7f52c406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f52c406c430 0x7f52c406e8f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f52dc009a30 tx=0x7f52dc008040 comp rx=0 tx=0).stop 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 msgr2=0x7f52ec151380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 0x7f52ec151380 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f52ec071c30 tx=0x7f52e400cb90 comp rx=0 tx=0).stop 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 shutdown_connections 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f52c406c430 0x7f52c406e8f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f52ec0c88f0 0x7f52ec151380 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f52ec06b5c0 0x7f52ec06cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 --2- 172.21.15.2:0/3046629643 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f52cc0047c0 0x7f52ec0caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 >> 172.21.15.2:0/3046629643 conn(0x7f52ec0d0ec0 msgr2=0x7f52ec0d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 shutdown_connections 2024-01-15T18:14:05.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.327+0000 7f52ebe00640 1 -- 172.21.15.2:0/3046629643 wait complete. 2024-01-15T18:14:05.349 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-15T18:14:05.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.569+0000 7f969be00640 1 Processor -- start 2024-01-15T18:14:05.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.569+0000 7f969be00640 1 -- start start 2024-01-15T18:14:05.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.571+0000 7f969be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 0x7f969c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.571+0000 7f969be00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0d1250 con 0x7f969c0b2210 2024-01-15T18:14:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.571+0000 7f969be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969c0ce8e0 con 0x7f969c0d1c50 2024-01-15T18:14:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.571+0000 7f969be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0ce6d0 con 0x7f969c0d1870 2024-01-15T18:14:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.571+0000 7f9693e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f969c0b2210 0x7f969c15d0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42696/0 (socket says 172.21.15.2:42696) 2024-01-15T18:14:05.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.571+0000 7f9693e00640 1 -- 172.21.15.2:0/1535276356 learned_addr learned my addr 172.21.15.2:0/1535276356 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969aa00640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 0x7f969c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44086896 0 0) 0x7f969c0d1250 con 0x7f969c0b2210 2024-01-15T18:14:05.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f967c003520 con 0x7f969c0b2210 2024-01-15T18:14:05.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4200222090 0 0) 0x7f967c003520 con 0x7f969c0b2210 2024-01-15T18:14:05.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 msgr2=0x7f969c0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:05.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 0x7f969c0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 >> v1:172.21.15.137:6789/0 conn(0x7f969c0d1c50 legacy=0x7f969c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c0cebc0 con 0x7f969c0b2210 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9684001670 con 0x7f969c0b2210 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 >> v1:172.21.15.2:6789/0 conn(0x7f969c0b2210 legacy=0x7f969c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.572+0000 7f969a000640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 0x7f967c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969a000640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f967c003f20 0x7f967c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969a000640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f967c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c0ce6d0 con 0x7f967c004d00 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c00ba60 con 0x7f967c003f20 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f967c00bc20 con 0x7f969c0d1870 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969b400640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f967c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f9693e00640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f967c003f20 0x7f967c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969aa00640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 0x7f967c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969b400640 1 -- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 msgr2=0x7f967c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969b400640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0d1870 0x7f967c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969b400640 1 -- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f967c003f20 msgr2=0x7f967c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:05.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969b400640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f967c003f20 0x7f967c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.573+0000 7f969b400640 1 -- 172.21.15.2:0/1535276356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c003520 con 0x7f967c004d00 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969b400640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f967c005180 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f9694001e20 tx=0x7f967c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cc5b3f93c6982a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694003b70 con 0x7f967c004d00 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9694003cd0 con 0x7f967c004d00 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969a000640 1 -- 172.21.15.2:0/1535276356 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9694005310 con 0x7f967c004d00 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969be00640 1 -- 172.21.15.2:0/1535276356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 msgr2=0x7f967c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969be00640 1 --2- 172.21.15.2:0/1535276356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f967c005180 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f9694001e20 tx=0x7f967c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969b400640 1 -- 172.21.15.2:0/1535276356 reap_dead start 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969be00640 1 -- 172.21.15.2:0/1535276356 shutdown_connections 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.575+0000 7f969be00640 1 -- 172.21.15.2:0/1535276356 >> 172.21.15.2:0/1535276356 conn(0x7f969c0d30b0 msgr2=0x7f969c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.576+0000 7f969be00640 1 -- 172.21.15.2:0/1535276356 shutdown_connections 2024-01-15T18:14:05.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.576+0000 7f969be00640 1 -- 172.21.15.2:0/1535276356 wait complete. 2024-01-15T18:14:05.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.577+0000 7f969be00640 1 Processor -- start 2024-01-15T18:14:05.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.578+0000 7f969be00640 1 -- start start 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f969c0c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f969c15e630 0x7f969c15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0c7030 0x7f969c0c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15eaf0 con 0x7f967c004d00 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15edd0 con 0x7f969c15e630 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969c15f0b0 con 0x7f969c0c7030 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f969c0c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f969c0c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45052/0 (socket says 172.21.15.2:45052) 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 -- 172.21.15.2:0/691786069 learned_addr learned my addr 172.21.15.2:0/691786069 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f9693e00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f969c15e630 0x7f969c15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 -- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0c7030 msgr2=0x7f969c0c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0c7030 0x7f969c0c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 -- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f969c15e630 msgr2=0x7f969c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f969c15e630 0x7f969c15fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.579+0000 7f969aa00640 1 -- 172.21.15.2:0/691786069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969c15f390 con 0x7f967c004d00 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.580+0000 7f969aa00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f969c0c8e30 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f968c011600 tx=0x7f968c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.580+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c015870 con 0x7f967c004d00 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.580+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968c0159d0 con 0x7f967c004d00 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.580+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969c0c9370 con 0x7f967c004d00 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.580+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968c015d40 con 0x7f967c004d00 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.580+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969c1601d0 con 0x7f967c004d00 2024-01-15T18:14:05.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.581+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c066d20 con 0x7f967c004d00 2024-01-15T18:14:05.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.583+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f968c00f490 con 0x7f967c004d00 2024-01-15T18:14:05.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.584+0000 7f9692a00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f967406c430 0x7f967406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:05.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.584+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f968c089150 con 0x7f967c004d00 2024-01-15T18:14:05.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.584+0000 7f9693e00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f967406c430 0x7f967406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:05.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.586+0000 7f9693e00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f967406c430 0x7f967406e8f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9684001050 tx=0x7f9684001d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:05.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.588+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f968c0556c0 con 0x7f967c004d00 2024-01-15T18:14:05.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.785+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f969c1601d0 con 0x7f967c004d00 2024-01-15T18:14:05.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.790+0000 7f9692a00640 1 -- 172.21.15.2:0/691786069 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v38) v1 ==== 120+0+0 (secure 0 0 0) 0x7f969c1601d0 con 0x7f967c004d00 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f967406c430 msgr2=0x7f967406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f967406c430 0x7f967406e8f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9684001050 tx=0x7f9684001d30 comp rx=0 tx=0).stop 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 msgr2=0x7f969c0c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f969c0c8e30 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f968c011600 tx=0x7f968c011ac0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 shutdown_connections 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f967406c430 0x7f967406e8f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f969c0c7030 0x7f969c0c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f969c15e630 0x7f969c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 --2- 172.21.15.2:0/691786069 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f967c004d00 0x7f969c0c8e30 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:05.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 >> 172.21.15.2:0/691786069 conn(0x7f969c0d30b0 msgr2=0x7f969c06c7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:05.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 shutdown_connections 2024-01-15T18:14:05.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:05.793+0000 7f969be00640 1 -- 172.21.15.2:0/691786069 wait complete. 2024-01-15T18:14:05.817 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-15T18:14:06.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.038+0000 7f5bb9600640 1 Processor -- start 2024-01-15T18:14:06.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.039+0000 7f5bb9600640 1 -- start start 2024-01-15T18:14:06.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.040+0000 7f5bb9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfa50 0x7f5bb40b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.040+0000 7f5bb9600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5bb40b3cb0 0x7f5bb40b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.040+0000 7f5bb9600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb40d3bd0 con 0x7f5bb40cfe30 2024-01-15T18:14:06.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.040+0000 7f5bb9600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb40b3b50 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.040+0000 7f5bb9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40d3eb0 con 0x7f5bb40cfa50 2024-01-15T18:14:06.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.041+0000 7f5bb8c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5bb40cfe30 0x7f5bb41612f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42698/0 (socket says 172.21.15.2:42698) 2024-01-15T18:14:06.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.041+0000 7f5bb8c00640 1 -- 172.21.15.2:0/3225555928 learned_addr learned my addr 172.21.15.2:0/3225555928 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:06.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.041+0000 7f5bb3e00640 1 --2- 172.21.15.2:0/3225555928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfa50 0x7f5bb40b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.041+0000 7f5bb3400640 1 --2- 172.21.15.2:0/3225555928 >> v2:172.21.15.137:3300/0 conn(0x7f5bb40b3cb0 0x7f5bb40b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.041+0000 7f5bb2a00640 1 -- 172.21.15.2:0/3225555928 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855044417 0 0) 0x7f5bb40d3bd0 con 0x7f5bb40cfe30 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.041+0000 7f5bb2a00640 1 -- 172.21.15.2:0/3225555928 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b94003520 con 0x7f5bb40cfe30 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.042+0000 7f5bb2a00640 1 -- 172.21.15.2:0/3225555928 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 852806303 0 0) 0x7f5b94003520 con 0x7f5bb40cfe30 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.042+0000 7f5bb3e00640 1 -- 172.21.15.2:0/3225555928 >> v2:172.21.15.137:3300/0 conn(0x7f5bb40b3cb0 msgr2=0x7f5bb40b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.042+0000 7f5bb3e00640 1 --2- 172.21.15.2:0/3225555928 >> v2:172.21.15.137:3300/0 conn(0x7f5bb40b3cb0 0x7f5bb40b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.042+0000 7f5bb3e00640 1 -- 172.21.15.2:0/3225555928 >> v1:172.21.15.2:6789/0 conn(0x7f5bb40cfe30 legacy=0x7f5bb41612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.042+0000 7f5bb3e00640 1 -- 172.21.15.2:0/3225555928 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb40d18a0 con 0x7f5bb40cfa50 2024-01-15T18:14:06.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.043+0000 7f5bb3e00640 1 --2- 172.21.15.2:0/3225555928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfa50 0x7f5bb40b30d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40097a0 tx=0x7f5ba402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb872058daa0b8de server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:06.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.044+0000 7f5bb2a00640 1 -- 172.21.15.2:0/3225555928 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bac0019f0 con 0x7f5bb40cfa50 2024-01-15T18:14:06.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.044+0000 7f5bb2a00640 1 -- 172.21.15.2:0/3225555928 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba402e050 con 0x7f5bb40cfa50 2024-01-15T18:14:06.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.044+0000 7f5bb2a00640 1 -- 172.21.15.2:0/3225555928 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4030730 con 0x7f5bb40cfa50 2024-01-15T18:14:06.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.044+0000 7f5bb9600640 1 -- 172.21.15.2:0/3225555928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfa50 msgr2=0x7f5bb40b30d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 --2- 172.21.15.2:0/3225555928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfa50 0x7f5bb40b30d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5ba40097a0 tx=0x7f5ba402b790 comp rx=0 tx=0).stop 2024-01-15T18:14:06.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 -- 172.21.15.2:0/3225555928 shutdown_connections 2024-01-15T18:14:06.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 --2- 172.21.15.2:0/3225555928 >> v2:172.21.15.137:3300/0 conn(0x7f5bb40b3cb0 0x7f5bb40b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 --2- 172.21.15.2:0/3225555928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfa50 0x7f5bb40b30d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 -- 172.21.15.2:0/3225555928 >> 172.21.15.2:0/3225555928 conn(0x7f5bb40d30b0 msgr2=0x7f5bb40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:06.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 -- 172.21.15.2:0/3225555928 shutdown_connections 2024-01-15T18:14:06.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.045+0000 7f5bb9600640 1 -- 172.21.15.2:0/3225555928 wait complete. 2024-01-15T18:14:06.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.047+0000 7f5bb9600640 1 Processor -- start 2024-01-15T18:14:06.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.047+0000 7f5bb9600640 1 -- start start 2024-01-15T18:14:06.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 0x7f5bb409f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb9600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bb40cfa50 0x7f5bb41f2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb9600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfe30 0x7f5bb41f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb9600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40a0360 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb9600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40a0640 con 0x7f5bb40cfa50 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb9600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40a0920 con 0x7f5bb40cfe30 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 0x7f5bb409f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 0x7f5bb409f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45060/0 (socket says 172.21.15.2:45060) 2024-01-15T18:14:06.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3e00640 1 -- 172.21.15.2:0/1183979545 learned_addr learned my addr 172.21.15.2:0/1183979545 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3400640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bb40cfa50 0x7f5bb41f2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb8c00640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfe30 0x7f5bb41f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3e00640 1 -- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfe30 msgr2=0x7f5bb41f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3e00640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfe30 0x7f5bb41f3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.049+0000 7f5bb3e00640 1 -- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bb40cfa50 msgr2=0x7f5bb41f2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5bb3e00640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bb40cfa50 0x7f5bb41f2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5bb3e00640 1 -- 172.21.15.2:0/1183979545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba4009400 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5bb3400640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bb40cfa50 0x7f5bb41f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5bb3e00640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 0x7f5bb409f8f0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4035040 tx=0x7f5ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba403b810 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba403b990 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb40a0c00 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.050+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba403bd40 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.051+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb4154d70 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.052+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb40a0e10 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.056 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.053+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f5ba4039040 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.054+0000 7f5babe00640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b8c06c430 0x7f5b8c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.054+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f5ba40a9e60 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.055+0000 7f5bb3400640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b8c06c430 0x7f5b8c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.056+0000 7f5bb3400640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b8c06c430 0x7f5b8c06e8f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5bb409f6e0 tx=0x7f5b9c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:06.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.058+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ba40763d0 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.260+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5bb4074770 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.265+0000 7f5babe00640 1 -- 172.21.15.2:0/1183979545 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5ba40370a0 con 0x7f5bb40b3cb0 2024-01-15T18:14:06.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b8c06c430 msgr2=0x7f5b8c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b8c06c430 0x7f5b8c06e8f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5bb409f6e0 tx=0x7f5b9c008040 comp rx=0 tx=0).stop 2024-01-15T18:14:06.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 msgr2=0x7f5bb409f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 0x7f5bb409f8f0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4035040 tx=0x7f5ba40046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 shutdown_connections 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5b8c06c430 0x7f5b8c06e8f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bb40cfe30 0x7f5bb41f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bb40cfa50 0x7f5bb41f2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 --2- 172.21.15.2:0/1183979545 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bb40b3cb0 0x7f5bb409f8f0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.268+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 >> 172.21.15.2:0/1183979545 conn(0x7f5bb40d30b0 msgr2=0x7f5bb40afcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.269+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 shutdown_connections 2024-01-15T18:14:06.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.269+0000 7f5bb9600640 1 -- 172.21.15.2:0/1183979545 wait complete. 2024-01-15T18:14:06.291 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-15T18:14:06.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.514+0000 7f4502e00640 1 Processor -- start 2024-01-15T18:14:06.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.514+0000 7f4502e00640 1 -- start start 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.515+0000 7f4502e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f44fc15d670 0x7f44fc0cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.515+0000 7f4502e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44fc0ce2b0 0x7f44fc0b2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.515+0000 7f4502e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc0ce930 con 0x7f44fc0d2f10 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.515+0000 7f4502e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc0ce740 con 0x7f44fc15d670 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.515+0000 7f4502e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc0cec10 con 0x7f44fc0ce2b0 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.515+0000 7f44fb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f44fc15d670 0x7f44fc0cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f44fc0d2f10 0x7f44fc0d32f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42712/0 (socket says 172.21.15.2:42712) 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f44fc15d670 0x7f44fc0cdd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45068/0 (socket says 172.21.15.2:45068) 2024-01-15T18:14:06.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fbe00640 1 -- 172.21.15.2:0/3275787158 learned_addr learned my addr 172.21.15.2:0/3275787158 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f4500e00640 1 --2- 172.21.15.2:0/3275787158 >> v2:172.21.15.137:3300/0 conn(0x7f44fc0ce2b0 0x7f44fc0b2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fb400640 1 -- 172.21.15.2:0/3275787158 >> v2:172.21.15.137:3300/0 conn(0x7f44fc0ce2b0 msgr2=0x7f44fc0b2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fb400640 1 --2- 172.21.15.2:0/3275787158 >> v2:172.21.15.137:3300/0 conn(0x7f44fc0ce2b0 0x7f44fc0b2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fb400640 1 -- 172.21.15.2:0/3275787158 >> v1:172.21.15.2:6789/0 conn(0x7f44fc0d2f10 legacy=0x7f44fc0d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.520 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.516+0000 7f44fb400640 1 -- 172.21.15.2:0/3275787158 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0ceef0 con 0x7f44fc15d670 2024-01-15T18:14:06.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.517+0000 7f44fb400640 1 --2- 172.21.15.2:0/3275787158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc15d670 0x7f44fc0cdd70 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0cd840 tx=0x7f44e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=224bc2a1ae7f8da9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:06.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.517+0000 7f44faa00640 1 -- 172.21.15.2:0/3275787158 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4031830 con 0x7f44fc15d670 2024-01-15T18:14:06.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f44faa00640 1 -- 172.21.15.2:0/3275787158 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4031990 con 0x7f44fc15d670 2024-01-15T18:14:06.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f44faa00640 1 -- 172.21.15.2:0/3275787158 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4031d20 con 0x7f44fc15d670 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 -- 172.21.15.2:0/3275787158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc15d670 msgr2=0x7f44fc0cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 --2- 172.21.15.2:0/3275787158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc15d670 0x7f44fc0cdd70 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f44fc0cd840 tx=0x7f44e402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 -- 172.21.15.2:0/3275787158 shutdown_connections 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 --2- 172.21.15.2:0/3275787158 >> v2:172.21.15.137:3300/0 conn(0x7f44fc0ce2b0 0x7f44fc0b2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 --2- 172.21.15.2:0/3275787158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc15d670 0x7f44fc0cdd70 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 -- 172.21.15.2:0/3275787158 >> 172.21.15.2:0/3275787158 conn(0x7f44fc08cc30 msgr2=0x7f44fc09c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 -- 172.21.15.2:0/3275787158 shutdown_connections 2024-01-15T18:14:06.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.518+0000 7f4502e00640 1 -- 172.21.15.2:0/3275787158 wait complete. 2024-01-15T18:14:06.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.520+0000 7f4502e00640 1 Processor -- start 2024-01-15T18:14:06.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.520+0000 7f4502e00640 1 -- start start 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4502e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc0ce2b0 0x7f44fc154560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4502e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0d2f10 0x7f44fc0cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4502e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 0x7f44fc0b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4502e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc15fb00 con 0x7f44fc0c8f70 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4502e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0cba70 con 0x7f44fc0d2f10 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4502e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc15eef0 con 0x7f44fc0ce2b0 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4500e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 0x7f44fc0b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4500e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 0x7f44fc0b5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45076/0 (socket says 172.21.15.2:45076) 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.521+0000 7f4500e00640 1 -- 172.21.15.2:0/4119087617 learned_addr learned my addr 172.21.15.2:0/4119087617 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f44fb400640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0d2f10 0x7f44fc0cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f44fbe00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc0ce2b0 0x7f44fc154560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f4500e00640 1 -- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc0ce2b0 msgr2=0x7f44fc154560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f4500e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc0ce2b0 0x7f44fc154560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f4500e00640 1 -- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0d2f10 msgr2=0x7f44fc0cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f4500e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0d2f10 0x7f44fc0cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f4500e00640 1 -- 172.21.15.2:0/4119087617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e4009400 con 0x7f44fc0c8f70 2024-01-15T18:14:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f44fbe00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc0ce2b0 0x7f44fc154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.522+0000 7f4500e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 0x7f44fc0b5e80 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f44f4007e40 tx=0x7f44f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:06.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.523+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f4014620 con 0x7f44fc0c8f70 2024-01-15T18:14:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.523+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f400ee90 con 0x7f44fc0c8f70 2024-01-15T18:14:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.523+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc15f1d0 con 0x7f44fc0c8f70 2024-01-15T18:14:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.523+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f40149e0 con 0x7f44fc0c8f70 2024-01-15T18:14:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.523+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc15f790 con 0x7f44fc0c8f70 2024-01-15T18:14:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.524+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc15fc70 con 0x7f44fc0c8f70 2024-01-15T18:14:06.528 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.525+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f44f4014b40 con 0x7f44fc0c8f70 2024-01-15T18:14:06.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.526+0000 7f44f3e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406c430 0x7f44d406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:06.529 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.526+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f44f4089480 con 0x7f44fc0c8f70 2024-01-15T18:14:06.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.527+0000 7f44fbe00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406c430 0x7f44d406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:06.531 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.528+0000 7f44fbe00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406c430 0x7f44d406e8f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000f60 tx=0x7f44ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:06.534 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.530+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44f4055a50 con 0x7f44fc0c8f70 2024-01-15T18:14:06.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:06.799+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f44fc0686f0 con 0x7f44fc0c8f70 2024-01-15T18:14:07.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.752+0000 7f44f3e00640 1 -- 172.21.15.2:0/4119087617 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v746) v1 ==== 153+0+0 (secure 0 0 0) 0x7f44f4059070 con 0x7f44fc0c8f70 2024-01-15T18:14:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406c430 msgr2=0x7f44d406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406c430 0x7f44d406e8f0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f44ec000f60 tx=0x7f44ec009040 comp rx=0 tx=0).stop 2024-01-15T18:14:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 msgr2=0x7f44fc0b5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 0x7f44fc0b5e80 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f44f4007e40 tx=0x7f44f400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:14:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 shutdown_connections 2024-01-15T18:14:07.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f44d406c430 0x7f44d406e8f0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f44fc0c8f70 0x7f44fc0b5e80 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44fc0d2f10 0x7f44fc0cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 --2- 172.21.15.2:0/4119087617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44fc0ce2b0 0x7f44fc154560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 >> 172.21.15.2:0/4119087617 conn(0x7f44fc08cc30 msgr2=0x7f44fc06bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 shutdown_connections 2024-01-15T18:14:07.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:07.756+0000 7f4502e00640 1 -- 172.21.15.2:0/4119087617 wait complete. 2024-01-15T18:14:07.783 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-15T18:14:07.783 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:14:07.783 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:14:07.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:08.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.005+0000 7f8e10a00640 1 Processor -- start 2024-01-15T18:14:08.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.005+0000 7f8e10a00640 1 -- start start 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e10a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8e0c141880 0x7f8e0c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e10a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e10a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e0c0cd490 0x7f8e0c0cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e10a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0c071f20 con 0x7f8e0c0cd490 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e10a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0c06c6b0 con 0x7f8e0c141880 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e10a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c15f620 con 0x7f8e0c15ec60 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e0be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e0c0cd490 0x7f8e0c0cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e0be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e0c0cd490 0x7f8e0c0cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45082/0 (socket says 172.21.15.2:45082) 2024-01-15T18:14:08.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.007+0000 7f8e0be00640 1 -- 172.21.15.2:0/4137841499 learned_addr learned my addr 172.21.15.2:0/4137841499 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e03e00640 1 --2- 172.21.15.2:0/4137841499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e0b400640 1 --2- 172.21.15.2:0/4137841499 >> v2:172.21.15.137:3300/0 conn(0x7f8e0c141880 0x7f8e0c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e0be00640 1 -- 172.21.15.2:0/4137841499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 msgr2=0x7f8e0c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e0be00640 1 --2- 172.21.15.2:0/4137841499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c15f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e0be00640 1 -- 172.21.15.2:0/4137841499 >> v2:172.21.15.137:3300/0 conn(0x7f8e0c141880 msgr2=0x7f8e0c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e0be00640 1 --2- 172.21.15.2:0/4137841499 >> v2:172.21.15.137:3300/0 conn(0x7f8e0c141880 0x7f8e0c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.008+0000 7f8e0be00640 1 -- 172.21.15.2:0/4137841499 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0c071960 con 0x7f8e0c0cd490 2024-01-15T18:14:08.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.009+0000 7f8e0be00640 1 --2- 172.21.15.2:0/4137841499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c0cd920 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c068540 tx=0x7f8e0402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=598dfb928ac8124e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.009+0000 7f8e0aa00640 1 -- 172.21.15.2:0/4137841499 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0402c9e0 con 0x7f8e0c0cd490 2024-01-15T18:14:08.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.009+0000 7f8e0aa00640 1 -- 172.21.15.2:0/4137841499 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0402cb40 con 0x7f8e0c0cd490 2024-01-15T18:14:08.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.009+0000 7f8e0aa00640 1 -- 172.21.15.2:0/4137841499 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e040343f0 con 0x7f8e0c0cd490 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 -- 172.21.15.2:0/4137841499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c0cd490 msgr2=0x7f8e0c0cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 --2- 172.21.15.2:0/4137841499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c0cd920 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c068540 tx=0x7f8e0402be40 comp rx=0 tx=0).stop 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 -- 172.21.15.2:0/4137841499 shutdown_connections 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 --2- 172.21.15.2:0/4137841499 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c0cd920 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 --2- 172.21.15.2:0/4137841499 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 --2- 172.21.15.2:0/4137841499 >> v2:172.21.15.137:3300/0 conn(0x7f8e0c141880 0x7f8e0c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 -- 172.21.15.2:0/4137841499 >> 172.21.15.2:0/4137841499 conn(0x7f8e0c0b0840 msgr2=0x7f8e0c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 -- 172.21.15.2:0/4137841499 shutdown_connections 2024-01-15T18:14:08.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.010+0000 7f8e10a00640 1 -- 172.21.15.2:0/4137841499 wait complete. 2024-01-15T18:14:08.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.012+0000 7f8e10a00640 1 Processor -- start 2024-01-15T18:14:08.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.012+0000 7f8e10a00640 1 -- start start 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e10a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e10a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 0x7f8e0c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e10a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e10a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c137020 con 0x7f8e0c141880 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e10a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0b2cd0 con 0x7f8e0c0cd490 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e10a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0c0b2fb0 con 0x7f8e0c15ec60 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e03e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 0x7f8e0c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e03e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 0x7f8e0c0b2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45090/0 (socket says 172.21.15.2:45090) 2024-01-15T18:14:08.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.013+0000 7f8e03e00640 1 -- 172.21.15.2:0/3671755577 learned_addr learned my addr 172.21.15.2:0/3671755577 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e0be00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e0b400640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03e00640 1 -- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 msgr2=0x7f8e0c142df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03e00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03e00640 1 -- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0c0cd490 msgr2=0x7f8e0c136ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03e00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c136ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03e00640 1 -- 172.21.15.2:0/3671755577 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e04009b30 con 0x7f8e0c141880 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03e00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 0x7f8e0c0b2210 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f8df400d800 tx=0x7f8df400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df400b6c0 con 0x7f8e0c141880 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df400b820 con 0x7f8e0c141880 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.014+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df400bbb0 con 0x7f8e0c141880 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.015+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e0c0b3290 con 0x7f8e0c141880 2024-01-15T18:14:08.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.015+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0c0b3850 con 0x7f8e0c141880 2024-01-15T18:14:08.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.016+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e0c074770 con 0x7f8e0c141880 2024-01-15T18:14:08.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.017+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8df400bd10 con 0x7f8e0c141880 2024-01-15T18:14:08.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.018+0000 7f8e03400640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8de406c430 0x7f8de406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.018+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f8df4089060 con 0x7f8e0c141880 2024-01-15T18:14:08.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.018+0000 7f8e0b400640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8de406c430 0x7f8de406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.019+0000 7f8e0b400640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8de406c430 0x7f8de406e8f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc006fd0 tx=0x7f8dfc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.022+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8df40555d0 con 0x7f8e0c141880 2024-01-15T18:14:08.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.219+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e0c15e570 con 0x7f8e0c141880 2024-01-15T18:14:08.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.222+0000 7f8e03400640 1 -- 172.21.15.2:0/3671755577 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+13157 (secure 0 0 0) 0x7f8df4058bf0 con 0x7f8e0c141880 2024-01-15T18:14:08.226 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:08.226 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":106,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:00.272802+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6837/320323477":"2024-01-16T18:13:57.259474+0000","172.21.15.137:6835/320323477":"2024-01-16T18:13:57.259474+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:08.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.224+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8de406c430 msgr2=0x7f8de406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.224+0000 7f8e10a00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8de406c430 0x7f8de406e8f0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc006fd0 tx=0x7f8dfc008040 comp rx=0 tx=0).stop 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.224+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 msgr2=0x7f8e0c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.224+0000 7f8e10a00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 0x7f8e0c0b2210 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f8df400d800 tx=0x7f8df400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 shutdown_connections 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8de406c430 0x7f8de406e8f0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e0c15ec60 0x7f8e0c142df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e0c141880 0x7f8e0c0b2210 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 --2- 172.21.15.2:0/3671755577 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e0c0cd490 0x7f8e0c136ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 >> 172.21.15.2:0/3671755577 conn(0x7f8e0c0b0840 msgr2=0x7f8e0c0cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 shutdown_connections 2024-01-15T18:14:08.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.225+0000 7f8e10a00640 1 -- 172.21.15.2:0/3671755577 wait complete. 2024-01-15T18:14:08.247 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:14:08.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:14:08.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.466+0000 7f87adc00640 1 Processor -- start 2024-01-15T18:14:08.469 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.466+0000 7f87adc00640 1 -- start start 2024-01-15T18:14:08.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.467+0000 7f87adc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f87a80d1870 0x7f87a80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.467+0000 7f87adc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2210 0x7f87a815d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.471 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.467+0000 7f87adc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f87a80d1c50 0x7f87a815d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.467+0000 7f87adc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a80ce6d0 con 0x7f87a80d1870 2024-01-15T18:14:08.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.467+0000 7f87adc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87a815db60 con 0x7f87a80d1c50 2024-01-15T18:14:08.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.467+0000 7f87adc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a80d1250 con 0x7f87a80b2210 2024-01-15T18:14:08.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f87a80d1870 0x7f87a80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f87a80d1870 0x7f87a80d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45098/0 (socket says 172.21.15.2:45098) 2024-01-15T18:14:08.472 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 -- 172.21.15.2:0/3318266479 learned_addr learned my addr 172.21.15.2:0/3318266479 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:08.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87ad200640 1 --2- 172.21.15.2:0/3318266479 >> v2:172.21.15.137:3300/0 conn(0x7f87a80d1c50 0x7f87a815d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7400640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2210 0x7f87a815d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 -- 172.21.15.2:0/3318266479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2210 msgr2=0x7f87a815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2210 0x7f87a815d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 -- 172.21.15.2:0/3318266479 >> v2:172.21.15.137:3300/0 conn(0x7f87a80d1c50 msgr2=0x7f87a815d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 --2- 172.21.15.2:0/3318266479 >> v2:172.21.15.137:3300/0 conn(0x7f87a80d1c50 0x7f87a815d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7e00640 1 -- 172.21.15.2:0/3318266479 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a80ce8e0 con 0x7f87a80d1870 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.468+0000 7f87a7400640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2210 0x7f87a815d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.469+0000 7f87a7e00640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80d1870 0x7f87a80d3bd0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f87a8161120 tx=0x7f879802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b82843fbd685ef7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.470+0000 7f87a6a00640 1 -- 172.21.15.2:0/3318266479 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798031860 con 0x7f87a80d1870 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.470+0000 7f87a6a00640 1 -- 172.21.15.2:0/3318266479 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87980319c0 con 0x7f87a80d1870 2024-01-15T18:14:08.474 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.470+0000 7f87a6a00640 1 -- 172.21.15.2:0/3318266479 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798031d50 con 0x7f87a80d1870 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.470+0000 7f87adc00640 1 -- 172.21.15.2:0/3318266479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80d1870 msgr2=0x7f87a80d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.470+0000 7f87adc00640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80d1870 0x7f87a80d3bd0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f87a8161120 tx=0x7f879802b900 comp rx=0 tx=0).stop 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 -- 172.21.15.2:0/3318266479 shutdown_connections 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 --2- 172.21.15.2:0/3318266479 >> v2:172.21.15.137:3300/0 conn(0x7f87a80d1c50 0x7f87a815d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80b2210 0x7f87a815d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 --2- 172.21.15.2:0/3318266479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80d1870 0x7f87a80d3bd0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 -- 172.21.15.2:0/3318266479 >> 172.21.15.2:0/3318266479 conn(0x7f87a80d30b0 msgr2=0x7f87a80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 -- 172.21.15.2:0/3318266479 shutdown_connections 2024-01-15T18:14:08.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.471+0000 7f87adc00640 1 -- 172.21.15.2:0/3318266479 wait complete. 2024-01-15T18:14:08.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.473+0000 7f87adc00640 1 Processor -- start 2024-01-15T18:14:08.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.473+0000 7f87adc00640 1 -- start start 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87adc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 0x7f87a81f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87adc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80d1870 0x7f87a80b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87adc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a80d1c50 0x7f87a80a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87adc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a809ce40 con 0x7f87a80b2210 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87adc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a809d090 con 0x7f87a80d1c50 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87adc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a809d370 con 0x7f87a80d1870 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 0x7f87a81f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87a7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 0x7f87a81f6b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45114/0 (socket says 172.21.15.2:45114) 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87a7e00640 1 -- 172.21.15.2:0/4044070090 learned_addr learned my addr 172.21.15.2:0/4044070090 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87ad200640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a80d1c50 0x7f87a80a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87a7e00640 1 -- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80d1870 msgr2=0x7f87a80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.474+0000 7f87a7e00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80d1870 0x7f87a80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f87a7e00640 1 -- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a80d1c50 msgr2=0x7f87a80a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f87a7e00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a80d1c50 0x7f87a80a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f87a7e00640 1 -- 172.21.15.2:0/4044070090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87980094d0 con 0x7f87a80b2210 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f87a7e00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 0x7f87a81f6b80 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f879802b940 tx=0x7f87980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879803ac60 con 0x7f87a80b2210 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879803adc0 con 0x7f87a80b2210 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798044610 con 0x7f87a80b2210 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a809d650 con 0x7f87a80b2210 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.475+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a815ae60 con 0x7f87a80b2210 2024-01-15T18:14:08.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.476+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a809db70 con 0x7f87a80b2210 2024-01-15T18:14:08.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.478+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f879804d450 con 0x7f87a80b2210 2024-01-15T18:14:08.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.478+0000 7f879fe00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f878006c480 0x7f878006e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.479+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f87980ab130 con 0x7f87a80b2210 2024-01-15T18:14:08.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.479+0000 7f87a7400640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f878006c480 0x7f878006e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.480+0000 7f87a7400640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f878006c480 0x7f878006e940 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8790007130 tx=0x7f87900073c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.483+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87a809db70 con 0x7f87a80b2210 2024-01-15T18:14:08.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.675+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87a8074770 con 0x7f87a80b2210 2024-01-15T18:14:08.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.676+0000 7f879fe00640 1 -- 172.21.15.2:0/4044070090 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8798035030 con 0x7f87a80b2210 2024-01-15T18:14:08.680 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f878006c480 msgr2=0x7f878006e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f878006c480 0x7f878006e940 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f8790007130 tx=0x7f87900073c0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 msgr2=0x7f87a81f6b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 0x7f87a81f6b80 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f879802b940 tx=0x7f87980046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 shutdown_connections 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f878006c480 0x7f878006e940 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f87a80d1c50 0x7f87a80a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f87a80d1870 0x7f87a80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 --2- 172.21.15.2:0/4044070090 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f87a80b2210 0x7f87a81f6b80 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 >> 172.21.15.2:0/4044070090 conn(0x7f87a80d30b0 msgr2=0x7f87a80afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:08.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 shutdown_connections 2024-01-15T18:14:08.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.679+0000 7f87adc00640 1 -- 172.21.15.2:0/4044070090 wait complete. 2024-01-15T18:14:08.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-15T18:14:08.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.920+0000 7f9411000640 1 Processor -- start 2024-01-15T18:14:08.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.921+0000 7f9411000640 1 -- start start 2024-01-15T18:14:08.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.922+0000 7f9411000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f940c141880 0x7f940c070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.922+0000 7f9411000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f940c15f0e0 0x7f940c0cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.922+0000 7f9411000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940c06dae0 con 0x7f940c15ec60 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.922+0000 7f9411000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c071f20 con 0x7f940c15f0e0 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.922+0000 7f9411000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940c06c6b0 con 0x7f940c141880 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f940c15f0e0 0x7f940c0cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f940c15f0e0 0x7f940c0cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45128/0 (socket says 172.21.15.2:45128) 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 -- 172.21.15.2:0/1828658676 learned_addr learned my addr 172.21.15.2:0/1828658676 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:08.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940b400640 1 --2- 172.21.15.2:0/1828658676 >> v2:172.21.15.137:3300/0 conn(0x7f940c141880 0x7f940c070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 -- 172.21.15.2:0/1828658676 >> v2:172.21.15.137:3300/0 conn(0x7f940c141880 msgr2=0x7f940c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:08.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 --2- 172.21.15.2:0/1828658676 >> v2:172.21.15.137:3300/0 conn(0x7f940c141880 0x7f940c070fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 -- 172.21.15.2:0/1828658676 >> v1:172.21.15.137:6789/0 conn(0x7f940c15ec60 legacy=0x7f940c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:08.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.923+0000 7f940be00640 1 -- 172.21.15.2:0/1828658676 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940c071960 con 0x7f940c15f0e0 2024-01-15T18:14:08.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.925+0000 7f940be00640 1 --2- 172.21.15.2:0/1828658676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c15f0e0 0x7f940c0cdba0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f940c068540 tx=0x7f940402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef2420869d2d7f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.925+0000 7f940aa00640 1 -- 172.21.15.2:0/1828658676 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940402c9f0 con 0x7f940c15f0e0 2024-01-15T18:14:08.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.925+0000 7f940aa00640 1 -- 172.21.15.2:0/1828658676 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940402cb50 con 0x7f940c15f0e0 2024-01-15T18:14:08.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.925+0000 7f940aa00640 1 -- 172.21.15.2:0/1828658676 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94040343f0 con 0x7f940c15f0e0 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 -- 172.21.15.2:0/1828658676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c15f0e0 msgr2=0x7f940c0cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 --2- 172.21.15.2:0/1828658676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c15f0e0 0x7f940c0cdba0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f940c068540 tx=0x7f940402be50 comp rx=0 tx=0).stop 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 -- 172.21.15.2:0/1828658676 shutdown_connections 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 --2- 172.21.15.2:0/1828658676 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c15f0e0 0x7f940c0cdba0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 --2- 172.21.15.2:0/1828658676 >> v2:172.21.15.137:3300/0 conn(0x7f940c141880 0x7f940c070fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 -- 172.21.15.2:0/1828658676 >> 172.21.15.2:0/1828658676 conn(0x7f940c0b0840 msgr2=0x7f940c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 -- 172.21.15.2:0/1828658676 shutdown_connections 2024-01-15T18:14:08.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.926+0000 7f9411000640 1 -- 172.21.15.2:0/1828658676 wait complete. 2024-01-15T18:14:08.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.928+0000 7f9411000640 1 Processor -- start 2024-01-15T18:14:08.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.928+0000 7f9411000640 1 -- start start 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f9411000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 0x7f940c15a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f9411000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f940c15ec60 0x7f940c13b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f9411000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f940c15f0e0 0x7f940c1401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f9411000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c15a750 con 0x7f940c141880 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f9411000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940c13bdf0 con 0x7f940c15ec60 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f9411000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940c1580d0 con 0x7f940c15f0e0 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f940b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 0x7f940c15a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f940b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 0x7f940c15a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45140/0 (socket says 172.21.15.2:45140) 2024-01-15T18:14:08.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.929+0000 7f940b400640 1 -- 172.21.15.2:0/1186510987 learned_addr learned my addr 172.21.15.2:0/1186510987 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f9403e00640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f940c15ec60 0x7f940c13b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940be00640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f940c15f0e0 0x7f940c1401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940b400640 1 -- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f940c15f0e0 msgr2=0x7f940c1401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940b400640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f940c15f0e0 0x7f940c1401b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940b400640 1 -- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f940c15ec60 msgr2=0x7f940c13b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940b400640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f940c15ec60 0x7f940c13b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940b400640 1 -- 172.21.15.2:0/1186510987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404009b30 con 0x7f940c141880 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f940b400640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 0x7f940c15a210 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00d620 tx=0x7f93fc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.930+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00b620 con 0x7f940c141880 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.931+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc00b780 con 0x7f940c141880 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.931+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc00bb30 con 0x7f940c141880 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.931+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940c0adb10 con 0x7f940c141880 2024-01-15T18:14:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.931+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f940c0ae0d0 con 0x7f940c141880 2024-01-15T18:14:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.932+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f940c074770 con 0x7f940c141880 2024-01-15T18:14:08.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.933+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f93fc00bc90 con 0x7f940c141880 2024-01-15T18:14:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.934+0000 7f9403400640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f93e406c430 0x7f93e406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.934+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f93fc088fa0 con 0x7f940c141880 2024-01-15T18:14:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.935+0000 7f9403e00640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f93e406c430 0x7f93e406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.936+0000 7f9403e00640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f93e406c430 0x7f93e406e8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006fd0 tx=0x7f93f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:08.938+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93fc055510 con 0x7f940c141880 2024-01-15T18:14:09.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.202+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f940c13c270 con 0x7f940c141880 2024-01-15T18:14:09.768 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.764+0000 7f9403400640 1 -- 172.21.15.2:0/1186510987 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v748) v1 ==== 158+0+0 (secure 0 0 0) 0x7f93fc058b30 con 0x7f940c141880 2024-01-15T18:14:09.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f93e406c430 msgr2=0x7f93e406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:09.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f93e406c430 0x7f93e406e8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006fd0 tx=0x7f93f4008040 comp rx=0 tx=0).stop 2024-01-15T18:14:09.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 msgr2=0x7f940c15a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:09.772 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 0x7f940c15a210 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f93fc00d620 tx=0x7f93fc00dae0 comp rx=0 tx=0).stop 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 shutdown_connections 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f93e406c430 0x7f93e406e8f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f940c15f0e0 0x7f940c1401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f940c15ec60 0x7f940c13b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 --2- 172.21.15.2:0/1186510987 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f940c141880 0x7f940c15a210 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 >> 172.21.15.2:0/1186510987 conn(0x7f940c0b0840 msgr2=0x7f940c070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 shutdown_connections 2024-01-15T18:14:09.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:09.769+0000 7f9411000640 1 -- 172.21.15.2:0/1186510987 wait complete. 2024-01-15T18:14:09.793 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:14:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.006+0000 7fe60dc00640 1 Processor -- start 2024-01-15T18:14:10.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.006+0000 7fe60dc00640 1 -- start start 2024-01-15T18:14:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.007+0000 7fe60dc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe6080cd3d0 0x7fe6080cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.007+0000 7fe60dc00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe60806b5a0 0x7fe60806ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.007+0000 7fe60dc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080ce9a0 con 0x7fe6080b47b0 2024-01-15T18:14:10.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.007+0000 7fe60dc00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6080ce7c0 con 0x7fe6080cd3d0 2024-01-15T18:14:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.007+0000 7fe60dc00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6080aee40 con 0x7fe60806b5a0 2024-01-15T18:14:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe6080cd3d0 0x7fe6080cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe6080cd3d0 0x7fe6080cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45156/0 (socket says 172.21.15.2:45156) 2024-01-15T18:14:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 -- 172.21.15.2:0/4204709636 learned_addr learned my addr 172.21.15.2:0/4204709636 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:10.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607400640 1 --1- 172.21.15.2:0/4204709636 >> v1:172.21.15.2:6789/0 conn(0x7fe6080b47b0 0x7fe60806ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42714/0 (socket says 172.21.15.2:42714) 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe606a00640 1 -- 172.21.15.2:0/4204709636 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470147397 0 0) 0x7fe6080ce9a0 con 0x7fe6080b47b0 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe60d200640 1 --2- 172.21.15.2:0/4204709636 >> v2:172.21.15.137:3300/0 conn(0x7fe60806b5a0 0x7fe60806ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe606a00640 1 -- 172.21.15.2:0/4204709636 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e8003520 con 0x7fe6080b47b0 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 -- 172.21.15.2:0/4204709636 >> v2:172.21.15.137:3300/0 conn(0x7fe60806b5a0 msgr2=0x7fe60806ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 --2- 172.21.15.2:0/4204709636 >> v2:172.21.15.137:3300/0 conn(0x7fe60806b5a0 0x7fe60806ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 -- 172.21.15.2:0/4204709636 >> v1:172.21.15.2:6789/0 conn(0x7fe6080b47b0 legacy=0x7fe60806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe607e00640 1 -- 172.21.15.2:0/4204709636 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080b25d0 con 0x7fe6080cd3d0 2024-01-15T18:14:10.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.008+0000 7fe60d200640 1 --2- 172.21.15.2:0/4204709636 >> v2:172.21.15.137:3300/0 conn(0x7fe60806b5a0 0x7fe60806ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:10.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe607e00640 1 --2- 172.21.15.2:0/4204709636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6080cd3d0 0x7fe6080cee30 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fe60806f450 tx=0x7fe5f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf424f8cb7e2f722 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:10.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe606a00640 1 -- 172.21.15.2:0/4204709636 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8031870 con 0x7fe6080cd3d0 2024-01-15T18:14:10.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe606a00640 1 -- 172.21.15.2:0/4204709636 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f80319d0 con 0x7fe6080cd3d0 2024-01-15T18:14:10.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe606a00640 1 -- 172.21.15.2:0/4204709636 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8031d40 con 0x7fe6080cd3d0 2024-01-15T18:14:10.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 -- 172.21.15.2:0/4204709636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6080cd3d0 msgr2=0x7fe6080cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 --2- 172.21.15.2:0/4204709636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6080cd3d0 0x7fe6080cee30 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fe60806f450 tx=0x7fe5f802b910 comp rx=0 tx=0).stop 2024-01-15T18:14:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 -- 172.21.15.2:0/4204709636 shutdown_connections 2024-01-15T18:14:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 --2- 172.21.15.2:0/4204709636 >> v2:172.21.15.137:3300/0 conn(0x7fe60806b5a0 0x7fe60806ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 --2- 172.21.15.2:0/4204709636 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6080cd3d0 0x7fe6080cee30 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 -- 172.21.15.2:0/4204709636 >> 172.21.15.2:0/4204709636 conn(0x7fe6080cbf70 msgr2=0x7fe6080cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 -- 172.21.15.2:0/4204709636 shutdown_connections 2024-01-15T18:14:10.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.011+0000 7fe60dc00640 1 -- 172.21.15.2:0/4204709636 wait complete. 2024-01-15T18:14:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 Processor -- start 2024-01-15T18:14:10.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 -- start start 2024-01-15T18:14:10.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 0x7fe608155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6080cd3d0 0x7fe608155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe608147900 0x7fe608158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608159840 con 0x7fe60806b5a0 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe60806d0f0 con 0x7fe608147900 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe60dc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60806d3d0 con 0x7fe6080cd3d0 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 0x7fe608155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 0x7fe608155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45166/0 (socket says 172.21.15.2:45166) 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 -- 172.21.15.2:0/1291670356 learned_addr learned my addr 172.21.15.2:0/1291670356 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607400640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6080cd3d0 0x7fe608155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 -- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6080cd3d0 msgr2=0x7fe608155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6080cd3d0 0x7fe608155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 -- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe608147900 msgr2=0x7fe608158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe608147900 0x7fe608158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.014+0000 7fe607e00640 1 -- 172.21.15.2:0/1291670356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f80094d0 con 0x7fe60806b5a0 2024-01-15T18:14:10.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.015+0000 7fe607e00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 0x7fe608155570 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fe5f802b950 tx=0x7fe5f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:10.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.015+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f803ac60 con 0x7fe60806b5a0 2024-01-15T18:14:10.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.015+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f803adc0 con 0x7fe60806b5a0 2024-01-15T18:14:10.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.015+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80445f0 con 0x7fe60806b5a0 2024-01-15T18:14:10.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.015+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60806d6b0 con 0x7fe60806b5a0 2024-01-15T18:14:10.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.015+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60814b380 con 0x7fe60806b5a0 2024-01-15T18:14:10.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.017+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60814b540 con 0x7fe60806b5a0 2024-01-15T18:14:10.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.017+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fe5f804d450 con 0x7fe60806b5a0 2024-01-15T18:14:10.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.020+0000 7fe5ffe00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5e006c430 0x7fe5e006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.020+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7fe5f802d040 con 0x7fe60806b5a0 2024-01-15T18:14:10.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.020+0000 7fe607400640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5e006c430 0x7fe5e006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.021+0000 7fe607400640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5e006c430 0x7fe5e006e8f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6081543a0 tx=0x7fe5f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:10.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.025+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5f8077590 con 0x7fe60806b5a0 2024-01-15T18:14:10.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.214+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe608074770 con 0x7fe60806b5a0 2024-01-15T18:14:10.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.222+0000 7fe5ffe00640 1 -- 172.21.15.2:0/1291670356 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe5f8035030 con 0x7fe60806b5a0 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5e006c430 msgr2=0x7fe5e006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5e006c430 0x7fe5e006e8f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe6081543a0 tx=0x7fe5f0009040 comp rx=0 tx=0).stop 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 msgr2=0x7fe608155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 0x7fe608155570 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fe5f802b950 tx=0x7fe5f80046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 shutdown_connections 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5e006c430 0x7fe5e006e8f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe608147900 0x7fe608158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6080cd3d0 0x7fe608155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 --2- 172.21.15.2:0/1291670356 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe60806b5a0 0x7fe608155570 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 >> 172.21.15.2:0/1291670356 conn(0x7fe6080cbf70 msgr2=0x7fe608070f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 shutdown_connections 2024-01-15T18:14:10.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.225+0000 7fe60dc00640 1 -- 172.21.15.2:0/1291670356 wait complete. 2024-01-15T18:14:10.245 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-15T18:14:10.246 DEBUG:teuthology.orchestra.run.smithi002:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:14:10.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-15T18:14:10.470 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-15T18:14:10.470 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:14:10.470 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:10.529 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:14:10.529 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:14:10.529 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:10.586 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:14:10.586 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:14:10.586 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:10.647 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:14:10.647 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:14:10.647 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:10.700 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:14:10.700 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-15T18:14:10.701 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:14:10.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.926+0000 7f6edf200640 1 Processor -- start 2024-01-15T18:14:10.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.927+0000 7f6edf200640 1 -- start start 2024-01-15T18:14:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.928+0000 7f6edf200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f6ee015d340 0x7f6ee00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.928+0000 7f6edf200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ee00d3090 0x7f6ee00d3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.928+0000 7f6edf200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee00d3a60 con 0x7f6ee00d1680 2024-01-15T18:14:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.928+0000 7f6edf200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00b3de0 con 0x7f6ee00d3090 2024-01-15T18:14:10.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.928+0000 7f6edf200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee00b40c0 con 0x7f6ee015d340 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ee00d3090 0x7f6ee00d3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6ee00d3090 0x7f6ee00d3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45172/0 (socket says 172.21.15.2:45172) 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 -- 172.21.15.2:0/1764132041 learned_addr learned my addr 172.21.15.2:0/1764132041 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6edde00640 1 --2- 172.21.15.2:0/1764132041 >> v2:172.21.15.137:3300/0 conn(0x7f6ee015d340 0x7f6ee00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 -- 172.21.15.2:0/1764132041 >> v2:172.21.15.137:3300/0 conn(0x7f6ee015d340 msgr2=0x7f6ee00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 --2- 172.21.15.2:0/1764132041 >> v2:172.21.15.137:3300/0 conn(0x7f6ee015d340 0x7f6ee00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 -- 172.21.15.2:0/1764132041 >> v1:172.21.15.137:6789/0 conn(0x7f6ee00d1680 legacy=0x7f6ee00d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:10.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.929+0000 7f6ed7e00640 1 -- 172.21.15.2:0/1764132041 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee00d0e80 con 0x7f6ee00d3090 2024-01-15T18:14:10.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.930+0000 7f6ed7e00640 1 --2- 172.21.15.2:0/1764132041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d3090 0x7f6ee00d3520 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f6ee008cf70 tx=0x7f6ec802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38e1162b9f943d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:10.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edd400640 1 -- 172.21.15.2:0/1764132041 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8031870 con 0x7f6ee00d3090 2024-01-15T18:14:10.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edd400640 1 -- 172.21.15.2:0/1764132041 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec80319d0 con 0x7f6ee00d3090 2024-01-15T18:14:10.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edd400640 1 -- 172.21.15.2:0/1764132041 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec8031d60 con 0x7f6ee00d3090 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edf200640 1 -- 172.21.15.2:0/1764132041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d3090 msgr2=0x7f6ee00d3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edf200640 1 --2- 172.21.15.2:0/1764132041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d3090 0x7f6ee00d3520 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f6ee008cf70 tx=0x7f6ec802b910 comp rx=0 tx=0).stop 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edf200640 1 -- 172.21.15.2:0/1764132041 shutdown_connections 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edf200640 1 --2- 172.21.15.2:0/1764132041 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d3090 0x7f6ee00d3520 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.931+0000 7f6edf200640 1 --2- 172.21.15.2:0/1764132041 >> v2:172.21.15.137:3300/0 conn(0x7f6ee015d340 0x7f6ee00cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.932+0000 7f6edf200640 1 -- 172.21.15.2:0/1764132041 >> 172.21.15.2:0/1764132041 conn(0x7f6ee008caf0 msgr2=0x7f6ee009c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:10.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.932+0000 7f6edf200640 1 -- 172.21.15.2:0/1764132041 shutdown_connections 2024-01-15T18:14:10.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.932+0000 7f6edf200640 1 -- 172.21.15.2:0/1764132041 wait complete. 2024-01-15T18:14:10.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.933+0000 7f6edf200640 1 Processor -- start 2024-01-15T18:14:10.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.933+0000 7f6edf200640 1 -- start start 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.934+0000 7f6edf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 0x7f6ee00ca960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.934+0000 7f6edf200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ee00d3090 0x7f6ee01f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.934+0000 7f6edf200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ee015d340 0x7f6ee01f6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.934+0000 7f6edf200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee01f73a0 con 0x7f6ee00d1680 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edf200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00caff0 con 0x7f6ee015d340 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edf200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee00cb2d0 con 0x7f6ee00d3090 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 0x7f6ee00ca960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 0x7f6ee00ca960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45184/0 (socket says 172.21.15.2:45184) 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 -- 172.21.15.2:0/3604193424 learned_addr learned my addr 172.21.15.2:0/3604193424 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6ed7e00640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ee00d3090 0x7f6ee01f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6ede800640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ee015d340 0x7f6ee01f6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 -- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ee00d3090 msgr2=0x7f6ee01f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ee00d3090 0x7f6ee01f2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 -- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ee015d340 msgr2=0x7f6ee01f6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:10.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ee015d340 0x7f6ee01f6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:10.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.935+0000 7f6edde00640 1 -- 172.21.15.2:0/3604193424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80094d0 con 0x7f6ee00d1680 2024-01-15T18:14:10.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.936+0000 7f6edde00640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 0x7f6ee00ca960 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d5f0 tx=0x7f6ed000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.936+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00125f0 con 0x7f6ee00d1680 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.936+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed0012750 con 0x7f6ee00d1680 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.936+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee00cb5b0 con 0x7f6ee00d1680 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.936+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0012b00 con 0x7f6ee00d1680 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.936+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0156a60 con 0x7f6ee00d1680 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.937+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee0156bf0 con 0x7f6ee00d1680 2024-01-15T18:14:10.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.938+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6ed0012c60 con 0x7f6ee00d1680 2024-01-15T18:14:10.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.939+0000 7f6ed6000640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6eb806c430 0x7f6eb806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:10.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.939+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f6ed0088fc0 con 0x7f6ee00d1680 2024-01-15T18:14:10.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.940+0000 7f6ed7e00640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6eb806c430 0x7f6eb806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:10.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.941+0000 7f6ed7e00640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6eb806c430 0x7f6eb806e8f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8036000 tx=0x7f6ec803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:10.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:10.943+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ed0055530 con 0x7f6ee00d1680 2024-01-15T18:14:11.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.179+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee0074770 con 0x7f6ee00d1680 2024-01-15T18:14:11.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.180+0000 7f6ed6000640 1 -- 172.21.15.2:0/3604193424 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3696 (secure 0 0 0) 0x7f6ed0058b50 con 0x7f6ee00d1680 2024-01-15T18:14:11.182 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6311,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3406786991","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3406786991},{"type":"v1","addr":"172.21.15.2:6835","nonce":3406786991}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6320,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/739459094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":739459094},{"type":"v1","addr":"172.21.15.137:6836","nonce":739459094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7132,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6837/2680407877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2680407877},{"type":"v1","addr":"172.21.15.137:6837","nonce":2680407877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":8565,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/2674783053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":2674783053},{"type":"v1","addr":"172.21.15.2:6834","nonce":2674783053}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[]} 2024-01-15T18:14:11.183 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 68 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6eb806c430 msgr2=0x7f6eb806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6eb806c430 0x7f6eb806e8f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8036000 tx=0x7f6ec803c040 comp rx=0 tx=0).stop 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 msgr2=0x7f6ee00ca960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 0x7f6ee00ca960 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000d5f0 tx=0x7f6ed000dab0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 shutdown_connections 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6eb806c430 0x7f6eb806e8f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6ee015d340 0x7f6ee01f6e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6ee00d3090 0x7f6ee01f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 --2- 172.21.15.2:0/3604193424 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6ee00d1680 0x7f6ee00ca960 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 >> 172.21.15.2:0/3604193424 conn(0x7f6ee008caf0 msgr2=0x7f6ee00d0220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:11.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 shutdown_connections 2024-01-15T18:14:11.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.183+0000 7f6edf200640 1 -- 172.21.15.2:0/3604193424 wait complete. 2024-01-15T18:14:11.206 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-15T18:14:11.206 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-15T18:14:11.206 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:14:11.206 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:14:11.207 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-15T18:14:11.207 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-15T18:14:11.207 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:14:11.207 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:14:11.207 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-15T18:14:11.207 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-15T18:14:11.207 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:14:11.208 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:14:11.208 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-15T18:14:11.208 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-15T18:14:11.208 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:14:11.208 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:14:11.228 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:14:11.228 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-15T18:14:11.229 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:14:11.229 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-15T18:14:11.237 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:14:11.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-15T18:14:11.238 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:14:11.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-15T18:14:11.276 INFO:tasks.ceph.mds.c:Started 2024-01-15T18:14:11.277 DEBUG:teuthology.parallel:result is None 2024-01-15T18:14:11.283 INFO:tasks.ceph.mds.a:Started 2024-01-15T18:14:11.283 DEBUG:teuthology.parallel:result is None 2024-01-15T18:14:11.285 INFO:tasks.ceph.mds.d:Started 2024-01-15T18:14:11.285 DEBUG:teuthology.parallel:result is None 2024-01-15T18:14:11.290 INFO:tasks.ceph.mds.b:Started 2024-01-15T18:14:11.290 DEBUG:teuthology.parallel:result is None 2024-01-15T18:14:11.291 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-15T18:14:11.437 INFO:tasks.ceph.mds.c.smithi002.stdout:starting mds.c at 2024-01-15T18:14:11.437 INFO:tasks.ceph.mds.a.smithi002.stdout:starting mds.a at 2024-01-15T18:14:11.450 INFO:tasks.ceph.mds.b.smithi137.stdout:starting mds.b at 2024-01-15T18:14:11.450 INFO:tasks.ceph.mds.d.smithi137.stdout:starting mds.d at 2024-01-15T18:14:11.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.479+0000 7f5599c00640 1 Processor -- start 2024-01-15T18:14:11.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.479+0000 7f5599c00640 1 -- start start 2024-01-15T18:14:11.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.480+0000 7f5599c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f55940cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:11.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5599c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f559406cfc0 0x7f559415b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:11.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5599c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55940b35a0 con 0x7f55940b30a0 2024-01-15T18:14:11.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5599c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55940b4220 con 0x7f559406cfc0 2024-01-15T18:14:11.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5599c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5594144f60 con 0x7f5594144b80 2024-01-15T18:14:11.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5589e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f55940b30a0 0x7f559406c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42730/0 (socket says 172.21.15.2:42730) 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5589e00640 1 -- 172.21.15.2:0/27275676 learned_addr learned my addr 172.21.15.2:0/27275676 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5592a00640 1 --2- 172.21.15.2:0/27275676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f55940cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5593400640 1 --2- 172.21.15.2:0/27275676 >> v2:172.21.15.137:3300/0 conn(0x7f559406cfc0 0x7f559415b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.481+0000 7f5592000640 1 -- 172.21.15.2:0/27275676 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3080027077 0 0) 0x7f55940b35a0 con 0x7f55940b30a0 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.482+0000 7f5592000640 1 -- 172.21.15.2:0/27275676 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5574003520 con 0x7f55940b30a0 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.482+0000 7f5592a00640 1 -- 172.21.15.2:0/27275676 >> v2:172.21.15.137:3300/0 conn(0x7f559406cfc0 msgr2=0x7f559415b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.482+0000 7f5592a00640 1 --2- 172.21.15.2:0/27275676 >> v2:172.21.15.137:3300/0 conn(0x7f559406cfc0 0x7f559415b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.482+0000 7f5592a00640 1 -- 172.21.15.2:0/27275676 >> v1:172.21.15.2:6789/0 conn(0x7f55940b30a0 legacy=0x7f559406c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.482+0000 7f5592a00640 1 -- 172.21.15.2:0/27275676 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55940b3880 con 0x7f5594144b80 2024-01-15T18:14:11.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.482+0000 7f5593400640 1 --2- 172.21.15.2:0/27275676 >> v2:172.21.15.137:3300/0 conn(0x7f559406cfc0 0x7f559415b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:14:11.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.483+0000 7f5592a00640 1 --2- 172.21.15.2:0/27275676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f55940cbae0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f55941450a0 tx=0x7f558402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d66f2280dacc6f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:11.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.483+0000 7f5592000640 1 -- 172.21.15.2:0/27275676 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584031770 con 0x7f5594144b80 2024-01-15T18:14:11.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.483+0000 7f5592000640 1 -- 172.21.15.2:0/27275676 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55840318d0 con 0x7f5594144b80 2024-01-15T18:14:11.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.483+0000 7f5592000640 1 -- 172.21.15.2:0/27275676 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5584031c60 con 0x7f5594144b80 2024-01-15T18:14:11.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 -- 172.21.15.2:0/27275676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 msgr2=0x7f55940cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 --2- 172.21.15.2:0/27275676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f55940cbae0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f55941450a0 tx=0x7f558402b810 comp rx=0 tx=0).stop 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 -- 172.21.15.2:0/27275676 shutdown_connections 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 --2- 172.21.15.2:0/27275676 >> v2:172.21.15.137:3300/0 conn(0x7f559406cfc0 0x7f559415b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 --2- 172.21.15.2:0/27275676 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f55940cbae0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 -- 172.21.15.2:0/27275676 >> 172.21.15.2:0/27275676 conn(0x7f55940b46d0 msgr2=0x7f55940b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 -- 172.21.15.2:0/27275676 shutdown_connections 2024-01-15T18:14:11.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.484+0000 7f5599c00640 1 -- 172.21.15.2:0/27275676 wait complete. 2024-01-15T18:14:11.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.485+0000 7f5599c00640 1 Processor -- start 2024-01-15T18:14:11.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.486+0000 7f5599c00640 1 -- start start 2024-01-15T18:14:11.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5599c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f559406cfc0 0x7f55940aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5599c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 0x7f55940b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5599c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f559413d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5599c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55940af1d0 con 0x7f55940b30a0 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5599c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55940af4b0 con 0x7f559406cfc0 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5599c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55940af790 con 0x7f5594144b80 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 0x7f55940b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5592a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f559406cfc0 0x7f55940aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 0x7f55940b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45232/0 (socket says 172.21.15.2:45232) 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 -- 172.21.15.2:0/850433833 learned_addr learned my addr 172.21.15.2:0/850433833 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:11.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5593400640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f559413d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:11.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 -- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 msgr2=0x7f559413d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f559413d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 -- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f559406cfc0 msgr2=0x7f55940aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f559406cfc0 0x7f55940aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.487+0000 7f5589e00640 1 -- 172.21.15.2:0/850433833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c009a70 con 0x7f55940b30a0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.488+0000 7f5589e00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 0x7f55940b0270 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c430 tx=0x7f557c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.488+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c005ce0 con 0x7f55940b30a0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.488+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c005e40 con 0x7f55940b30a0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.488+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c0072f0 con 0x7f55940b30a0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.488+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5584009400 con 0x7f55940b30a0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.488+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55940afd50 con 0x7f55940b30a0 2024-01-15T18:14:11.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.489+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55940686f0 con 0x7f55940b30a0 2024-01-15T18:14:11.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.490+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f557c007450 con 0x7f55940b30a0 2024-01-15T18:14:11.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.491+0000 7f558b400640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f556c06c430 0x7f556c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:11.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.491+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5083+0+0 (secure 0 0 0) 0x7f557c088010 con 0x7f55940b30a0 2024-01-15T18:14:11.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.492+0000 7f5592a00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f556c06c430 0x7f556c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:11.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.493+0000 7f5592a00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f556c06c430 0x7f556c06e8f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5584036000 tx=0x7f558403d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:11.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.495+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f557c054630 con 0x7f55940b30a0 2024-01-15T18:14:11.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.702+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5594141af0 con 0x7f55940b30a0 2024-01-15T18:14:11.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.803+0000 7f558b400640 1 -- 172.21.15.2:0/850433833 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v108) v1 ==== 97+0+0 (secure 0 0 0) 0x7f557c017070 con 0x7f55940b30a0 2024-01-15T18:14:11.806 INFO:teuthology.orchestra.run.smithi002.stderr: removed all blocklist entries 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f556c06c430 msgr2=0x7f556c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f556c06c430 0x7f556c06e8f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f5584036000 tx=0x7f558403d040 comp rx=0 tx=0).stop 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 msgr2=0x7f55940b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 0x7f55940b0270 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c430 tx=0x7f557c00c830 comp rx=0 tx=0).stop 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 shutdown_connections 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f556c06c430 0x7f556c06e8f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5594144b80 0x7f559413d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f55940b30a0 0x7f55940b0270 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 --2- 172.21.15.2:0/850433833 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f559406cfc0 0x7f55940aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 >> 172.21.15.2:0/850433833 conn(0x7f55940b46d0 msgr2=0x7f5594070380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 shutdown_connections 2024-01-15T18:14:11.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:11.806+0000 7f5599c00640 1 -- 172.21.15.2:0/850433833 wait complete. 2024-01-15T18:14:11.831 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-15T18:14:12.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.042+0000 7f68d2a00640 1 Processor -- start 2024-01-15T18:14:12.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.042+0000 7f68d2a00640 1 -- start start 2024-01-15T18:14:12.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.044+0000 7f68d2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68cc071da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.044+0000 7f68d2a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc0722e0 con 0x7f68cc15ad70 2024-01-15T18:14:12.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.044+0000 7f68d2a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68cc15e780 con 0x7f68cc0ac870 2024-01-15T18:14:12.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.044+0000 7f68d2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc070110 con 0x7f68cc071940 2024-01-15T18:14:12.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.045+0000 7f68d1600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f68cc15ad70 0x7f68cc070e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:42746/0 (socket says 172.21.15.2:42746) 2024-01-15T18:14:12.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.045+0000 7f68d1600640 1 -- 172.21.15.2:0/679655477 learned_addr learned my addr 172.21.15.2:0/679655477 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:12.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.045+0000 7f68d0c00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68cc071da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.045+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964341443 0 0) 0x7f68cc0722e0 con 0x7f68cc15ad70 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.045+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ac003520 con 0x7f68cc15ad70 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.045+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2696297943 0 0) 0x7f68ac003520 con 0x7f68cc15ad70 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 msgr2=0x7f68cc071da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68cc071da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 >> v1:172.21.15.137:6789/0 conn(0x7f68cc0ac870 legacy=0x7f68cc0acce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc071510 con 0x7f68cc15ad70 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f68bc001670 con 0x7f68cc15ad70 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 >> v1:172.21.15.2:6789/0 conn(0x7f68cc15ad70 legacy=0x7f68cc070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68ac0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68ac003f70 0x7f68ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ac004d00 0x7f68ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc070110 con 0x7f68ac003f70 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac00bb50 con 0x7f68ac004d00 2024-01-15T18:14:12.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac00bd10 con 0x7f68cc071940 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68d0c00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68ac003f70 0x7f68ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68d1600640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68ac0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68d2000640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ac004d00 0x7f68ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68d0c00640 1 -- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 msgr2=0x7f68ac0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.046+0000 7f68d0c00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68ac0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.047+0000 7f68d0c00640 1 -- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ac004d00 msgr2=0x7f68ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.047+0000 7f68d0c00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68ac004d00 0x7f68ac005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.047+0000 7f68d0c00640 1 -- 172.21.15.2:0/679655477 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac003520 con 0x7f68ac003f70 2024-01-15T18:14:12.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.047+0000 7f68d1600640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68cc071940 0x7f68ac0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:12.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.048+0000 7f68d0c00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68ac003f70 0x7f68ac0047c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009ac0 tx=0x7f68ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dafc0276421ecade server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:12.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.048+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b402c600 con 0x7f68ac003f70 2024-01-15T18:14:12.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.048+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b402c760 con 0x7f68ac003f70 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.048+0000 7f68cbe00640 1 -- 172.21.15.2:0/679655477 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b402caf0 con 0x7f68ac003f70 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2a00640 1 -- 172.21.15.2:0/679655477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68ac003f70 msgr2=0x7f68ac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2a00640 1 --2- 172.21.15.2:0/679655477 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68ac003f70 0x7f68ac0047c0 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009ac0 tx=0x7f68ac0057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2000640 1 -- 172.21.15.2:0/679655477 reap_dead start 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2a00640 1 -- 172.21.15.2:0/679655477 shutdown_connections 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2a00640 1 -- 172.21.15.2:0/679655477 >> 172.21.15.2:0/679655477 conn(0x7f68cc0b0840 msgr2=0x7f68cc0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2a00640 1 -- 172.21.15.2:0/679655477 shutdown_connections 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.049+0000 7f68d2a00640 1 -- 172.21.15.2:0/679655477 wait complete. 2024-01-15T18:14:12.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.051+0000 7f68d2a00640 1 Processor -- start 2024-01-15T18:14:12.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.051+0000 7f68d2a00640 1 -- start start 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68ac003f70 0x7f68cc14c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 0x7f68cc14b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc14aa00 0x7f68cc1505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d2a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc156890 con 0x7f68cc156d20 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc155c80 con 0x7f68cc14aa00 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68cc155f60 con 0x7f68ac003f70 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 0x7f68cc14b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 0x7f68cc14b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45246/0 (socket says 172.21.15.2:45246) 2024-01-15T18:14:12.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.052+0000 7f68d0c00640 1 -- 172.21.15.2:0/1889261839 learned_addr learned my addr 172.21.15.2:0/1889261839 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d2000640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc14aa00 0x7f68cc1505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d1600640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68ac003f70 0x7f68cc14c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d0c00640 1 -- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68ac003f70 msgr2=0x7f68cc14c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d0c00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68ac003f70 0x7f68cc14c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d0c00640 1 -- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc14aa00 msgr2=0x7f68cc1505d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d0c00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc14aa00 0x7f68cc1505d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d0c00640 1 -- 172.21.15.2:0/1889261839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68cc156240 con 0x7f68cc156d20 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68d0c00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 0x7f68cc14b2b0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f68b402c1a0 tx=0x7f68b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4003810 con 0x7f68cc156d20 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.053+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b4003970 con 0x7f68cc156d20 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.054+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4003ce0 con 0x7f68cc156d20 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.054+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68cc156520 con 0x7f68cc156d20 2024-01-15T18:14:12.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.054+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68cc14b070 con 0x7f68cc156d20 2024-01-15T18:14:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.055+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68cc06d6a0 con 0x7f68cc156d20 2024-01-15T18:14:12.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.056+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f68b4034440 con 0x7f68cc156d20 2024-01-15T18:14:12.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.057+0000 7f68c9600640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f68a406c430 0x7f68a406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.057+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4997+0+0 (secure 0 0 0) 0x7f68b40aa970 con 0x7f68cc156d20 2024-01-15T18:14:12.060 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.057+0000 7f68d1600640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f68a406c430 0x7f68a406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.059+0000 7f68d1600640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f68a406c430 0x7f68a406e8f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0017a0 tx=0x7f68bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:12.063 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.061+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68b4077060 con 0x7f68cc156d20 2024-01-15T18:14:12.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.325+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f68cc074770 con 0x7f68cc156d20 2024-01-15T18:14:12.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.326+0000 7f68c9600640 1 -- 172.21.15.2:0/1889261839 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f68b404c090 con 0x7f68cc156d20 2024-01-15T18:14:12.329 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:12.329 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:14:12.329 INFO:teuthology.orchestra.run.smithi002.stdout: "auth_dump": [ 2024-01-15T18:14:12.329 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.329 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.a", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlS9OHJRAAFv4D6ChzF3ic/79RUYrTmA==", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.b", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlBFbSDRAAD9e78mi8PP1CTa/41zfTMA==", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.330 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.c", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlVLqbNxAAPPNw2nKCdzVA5rBnZSHKuQ==", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mds.d", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlGzcqHxAAk2LMoNVOwAW0dvHnyMBDFQ==", 2024-01-15T18:14:12.331 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.0", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDUcaVlxqrsEBAAghxt0fhEesLiSi2NnnQe2A==", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.332 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.1", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDXcaVljVwQKhAAu5DoadTuCwWMD3moa0NXKA==", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.2", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDacaVloDEJNBAAhZYFgtEhjQwMv3/ZaUnaig==", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.333 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.3", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDecaVlN8snEhAAqCYMjIBG5hnB7L0G1wW5aw==", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.4", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDlcaVlc3U5HBAA3JfhcLxNKihsnTPzde+65Q==", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.334 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.5", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDocaVlO8zkNBAA1b0up/JZGgCtdDKj1MXWkw==", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.6", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDscaVlmF6MERAAKJnH/Oa07LogdJKHJbKptQ==", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.335 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "osd.7", 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDvcaVl4gDAGxAAG5YJabx0iT0MrkMjN5C0wQ==", 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow profile osd", 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile osd", 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.0", 2024-01-15T18:14:12.336 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w==", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow r", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.1", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlOxGcCBAAzrViK8jKfCd6u2k9wiZTYw==", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:14:12.337 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.2", 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDRcaVlUvDaOBAAecCS3jtAdmNyXZ66zmyTrw==", 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.3", 2024-01-15T18:14:12.338 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDScaVlNev3ExAAL74iEeecJBof4jweH9vFAw==", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow r", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow rw", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow rwx" 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "client.admin", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDPcaVlAJFoFxAAouMBoHx1AHeKoLeZIjDVCw==", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": "allow *", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow *", 2024-01-15T18:14:12.339 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.x", 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlwFxWGhAAr2kqdiEWUW9fzHLbODfjbw==", 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "entity": "mgr.y", 2024-01-15T18:14:12.340 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "AQDQcaVlm9t9DhAADUp34rqGYAD9vBPyJDlxqw==", 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: "caps": { 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": "allow *", 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": "allow profile mgr", 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": "allow *" 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-01-15T18:14:12.341 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f68a406c430 msgr2=0x7f68a406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f68a406c430 0x7f68a406e8f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f68bc0017a0 tx=0x7f68bc009040 comp rx=0 tx=0).stop 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 msgr2=0x7f68cc14b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 0x7f68cc14b2b0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f68b402c1a0 tx=0x7f68b4004730 comp rx=0 tx=0).stop 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 shutdown_connections 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f68a406c430 0x7f68a406e8f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f68cc14aa00 0x7f68cc1505d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68cc156d20 0x7f68cc14b2b0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 --2- 172.21.15.2:0/1889261839 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f68ac003f70 0x7f68cc14c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 >> 172.21.15.2:0/1889261839 conn(0x7f68cc0b0840 msgr2=0x7f68cc0b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 shutdown_connections 2024-01-15T18:14:12.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.329+0000 7f68d2a00640 1 -- 172.21.15.2:0/1889261839 wait complete. 2024-01-15T18:14:12.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:12.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.568+0000 7ff3d6600640 1 Processor -- start 2024-01-15T18:14:12.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.568+0000 7ff3d6600640 1 -- start start 2024-01-15T18:14:12.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.570+0000 7ff3d6600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff3d0143810 0x7ff3d006dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.570+0000 7ff3d6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 0x7ff3d00cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.570+0000 7ff3d6600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d006e470 con 0x7ff3d006e010 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.570+0000 7ff3d6600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d00b3520 con 0x7ff3d0143810 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.570+0000 7ff3d6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00ccbe0 con 0x7ff3d00b2250 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 0x7ff3d00cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff3d0143810 0x7ff3d006dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 0x7ff3d00cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:55228/0 (socket says 172.21.15.2:55228) 2024-01-15T18:14:12.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 -- 172.21.15.2:0/3532731283 learned_addr learned my addr 172.21.15.2:0/3532731283 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:12.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 -- 172.21.15.2:0/3532731283 >> v2:172.21.15.137:3300/0 conn(0x7ff3d0143810 msgr2=0x7ff3d006dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:12.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 --2- 172.21.15.2:0/3532731283 >> v2:172.21.15.137:3300/0 conn(0x7ff3d0143810 0x7ff3d006dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 -- 172.21.15.2:0/3532731283 >> v1:172.21.15.137:6789/0 conn(0x7ff3d006e010 legacy=0x7ff3d00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.571+0000 7ff3d5c00640 1 -- 172.21.15.2:0/3532731283 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0143bf0 con 0x7ff3d00b2250 2024-01-15T18:14:12.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.573+0000 7ff3d5c00640 1 --2- 172.21.15.2:0/3532731283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 0x7ff3d00cd070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009e50 tx=0x7ff3c802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0e6262d4813c8d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:12.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3cf400640 1 -- 172.21.15.2:0/3532731283 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c802ca70 con 0x7ff3d00b2250 2024-01-15T18:14:12.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3cf400640 1 -- 172.21.15.2:0/3532731283 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c802cbd0 con 0x7ff3d00b2250 2024-01-15T18:14:12.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3cf400640 1 -- 172.21.15.2:0/3532731283 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80343f0 con 0x7ff3d00b2250 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3d6600640 1 -- 172.21.15.2:0/3532731283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 msgr2=0x7ff3d00cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3532731283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 0x7ff3d00cd070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009e50 tx=0x7ff3c802bed0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3d6600640 1 -- 172.21.15.2:0/3532731283 shutdown_connections 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3532731283 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d00b2250 0x7ff3d00cd070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3532731283 >> v2:172.21.15.137:3300/0 conn(0x7ff3d0143810 0x7ff3d006dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.574+0000 7ff3d6600640 1 -- 172.21.15.2:0/3532731283 >> 172.21.15.2:0/3532731283 conn(0x7ff3d00b0840 msgr2=0x7ff3d00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.575+0000 7ff3d6600640 1 -- 172.21.15.2:0/3532731283 shutdown_connections 2024-01-15T18:14:12.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.575+0000 7ff3d6600640 1 -- 172.21.15.2:0/3532731283 wait complete. 2024-01-15T18:14:12.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.576+0000 7ff3d6600640 1 Processor -- start 2024-01-15T18:14:12.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.577+0000 7ff3d6600640 1 -- start start 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.577+0000 7ff3d6600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 0x7ff3d00cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d6600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3d00b2250 0x7ff3d0142400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d6600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d0143810 0x7ff3d0142940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d6600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00c8b80 con 0x7ff3d006e010 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d6600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00c8e60 con 0x7ff3d00b2250 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d6600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d00c9140 con 0x7ff3d0143810 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 0x7ff3d00cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3cfe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3d00b2250 0x7ff3d0142400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d5200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 0x7ff3d00cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:45252/0 (socket says 172.21.15.2:45252) 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d5200640 1 -- 172.21.15.2:0/3079679479 learned_addr learned my addr 172.21.15.2:0/3079679479 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:12.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.578+0000 7ff3d5c00640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d0143810 0x7ff3d0142940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d5200640 1 -- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d0143810 msgr2=0x7ff3d0142940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d5200640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d0143810 0x7ff3d0142940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d5200640 1 -- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3d00b2250 msgr2=0x7ff3d0142400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d5200640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3d00b2250 0x7ff3d0142400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d5200640 1 -- 172.21.15.2:0/3079679479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c8009b30 con 0x7ff3d006e010 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d5200640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 0x7ff3d00cd2e0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9b0 tx=0x7ff3c000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0009670 con 0x7ff3d006e010 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c000c9a0 con 0x7ff3d006e010 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000cd50 con 0x7ff3d006e010 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.579+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d00c9420 con 0x7ff3d006e010 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.580+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d01d70c0 con 0x7ff3d006e010 2024-01-15T18:14:12.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.581+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d0074770 con 0x7ff3d006e010 2024-01-15T18:14:12.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.582+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7ff3c0013470 con 0x7ff3d006e010 2024-01-15T18:14:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.583+0000 7ff3ccc00640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3a806c430 0x7ff3a806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:12.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.583+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4997+0+0 (secure 0 0 0) 0x7ff3c0088c70 con 0x7ff3d006e010 2024-01-15T18:14:12.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.583+0000 7ff3cfe00640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3a806c430 0x7ff3a806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:12.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.584+0000 7ff3cfe00640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3a806c430 0x7ff3a806e8f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff3d01f5e10 tx=0x7ff3b8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:12.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.587+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3c0055290 con 0x7ff3d006e010 2024-01-15T18:14:12.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.783+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3d0070170 con 0x7ff3d006e010 2024-01-15T18:14:12.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.785+0000 7ff3ccc00640 1 -- 172.21.15.2:0/3079679479 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+13028 (secure 0 0 0) 0x7ff3c0017090 con 0x7ff3d006e010 2024-01-15T18:14:12.788 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:12.788 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":110,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:11.822897+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":101,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3a806c430 msgr2=0x7ff3a806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3a806c430 0x7ff3a806e8f0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7ff3d01f5e10 tx=0x7ff3b8009040 comp rx=0 tx=0).stop 2024-01-15T18:14:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 msgr2=0x7ff3d00cd2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 0x7ff3d00cd2e0 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9b0 tx=0x7ff3c000ee70 comp rx=0 tx=0).stop 2024-01-15T18:14:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 shutdown_connections 2024-01-15T18:14:12.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff3a806c430 0x7ff3a806e8f0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff3d0143810 0x7ff3d0142940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff3d00b2250 0x7ff3d0142400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 --2- 172.21.15.2:0/3079679479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff3d006e010 0x7ff3d00cd2e0 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:12.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 >> 172.21.15.2:0/3079679479 conn(0x7ff3d00b0840 msgr2=0x7ff3d00cde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:12.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 shutdown_connections 2024-01-15T18:14:12.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:12.788+0000 7ff3d6600640 1 -- 172.21.15.2:0/3079679479 wait complete. 2024-01-15T18:14:12.811 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:14:12.811 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:14:13.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.020+0000 7fd9a3e00640 1 Processor -- start 2024-01-15T18:14:13.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.021+0000 7fd9a3e00640 1 -- start start 2024-01-15T18:14:13.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.022+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c144c00 0x7fd99c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.022+0000 7fd9a3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd99c06d000 0x7fd99c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.022+0000 7fd9a3e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0ce230 con 0x7fd99c0cdd70 2024-01-15T18:14:13.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.022+0000 7fd9a3e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd99c0ce510 con 0x7fd99c06d000 2024-01-15T18:14:13.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.022+0000 7fd9a3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0ce7f0 con 0x7fd99c144c00 2024-01-15T18:14:13.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd99c06d000 0x7fd99c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd99c06d000 0x7fd99c070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37616/0 (socket says 172.21.15.2:37616) 2024-01-15T18:14:13.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 -- 172.21.15.2:0/3959846703 learned_addr learned my addr 172.21.15.2:0/3959846703 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:13.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 -- 172.21.15.2:0/3959846703 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c144c00 msgr2=0x7fd99c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:13.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1400640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c144c00 0x7fd99c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c144c00 0x7fd99c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 -- 172.21.15.2:0/3959846703 >> v1:172.21.15.137:6789/0 conn(0x7fd99c0cdd70 legacy=0x7fd99c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:13.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1e00640 1 -- 172.21.15.2:0/3959846703 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c0705f0 con 0x7fd99c06d000 2024-01-15T18:14:13.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.023+0000 7fd9a1400640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c144c00 0x7fd99c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:13.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.025+0000 7fd9a1e00640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c06d000 0x7fd99c070b10 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fd994009e50 tx=0x7fd99402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea14be1c477b81f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.025+0000 7fd99be00640 1 -- 172.21.15.2:0/3959846703 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99402ca70 con 0x7fd99c06d000 2024-01-15T18:14:13.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.025+0000 7fd99be00640 1 -- 172.21.15.2:0/3959846703 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99402cbd0 con 0x7fd99c06d000 2024-01-15T18:14:13.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.025+0000 7fd99be00640 1 -- 172.21.15.2:0/3959846703 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9940343f0 con 0x7fd99c06d000 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 -- 172.21.15.2:0/3959846703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c06d000 msgr2=0x7fd99c070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c06d000 0x7fd99c070b10 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fd994009e50 tx=0x7fd99402bed0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 -- 172.21.15.2:0/3959846703 shutdown_connections 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c06d000 0x7fd99c070b10 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/3959846703 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c144c00 0x7fd99c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 -- 172.21.15.2:0/3959846703 >> 172.21.15.2:0/3959846703 conn(0x7fd99c0b45f0 msgr2=0x7fd99c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 -- 172.21.15.2:0/3959846703 shutdown_connections 2024-01-15T18:14:13.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.026+0000 7fd9a3e00640 1 -- 172.21.15.2:0/3959846703 wait complete. 2024-01-15T18:14:13.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.028+0000 7fd9a3e00640 1 Processor -- start 2024-01-15T18:14:13.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.028+0000 7fd9a3e00640 1 -- start start 2024-01-15T18:14:13.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c06d000 0x7fd99c0a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd99c0cdd70 0x7fd99c137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 0x7fd99c0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0af6e0 con 0x7fd99c144c00 2024-01-15T18:14:13.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0af9c0 con 0x7fd99c0cdd70 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c0afca0 con 0x7fd99c06d000 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 0x7fd99c0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 0x7fd99c0b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37618/0 (socket says 172.21.15.2:37618) 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1e00640 1 -- 172.21.15.2:0/2726658661 learned_addr learned my addr 172.21.15.2:0/2726658661 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a0a00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd99c0cdd70 0x7fd99c137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1400640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c06d000 0x7fd99c0a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1e00640 1 -- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c06d000 msgr2=0x7fd99c0a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c06d000 0x7fd99c0a8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.030+0000 7fd9a1e00640 1 -- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd99c0cdd70 msgr2=0x7fd99c137bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd9a1e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd99c0cdd70 0x7fd99c137bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd9a1e00640 1 -- 172.21.15.2:0/2726658661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994009b30 con 0x7fd99c144c00 2024-01-15T18:14:13.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd9a1e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 0x7fd99c0b0a60 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fd994036040 tx=0x7fd994004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99403e450 con 0x7fd99c144c00 2024-01-15T18:14:13.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd99403e5d0 con 0x7fd99c144c00 2024-01-15T18:14:13.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c0aff80 con 0x7fd99c144c00 2024-01-15T18:14:13.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd99403e980 con 0x7fd99c144c00 2024-01-15T18:14:13.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.031+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c0b0540 con 0x7fd99c144c00 2024-01-15T18:14:13.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.033+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99c0b0730 con 0x7fd99c144c00 2024-01-15T18:14:13.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.034+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd99403eae0 con 0x7fd99c144c00 2024-01-15T18:14:13.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.035+0000 7fd999600640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd97406c430 0x7fd97406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.035+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4997+0+0 (secure 0 0 0) 0x7fd9940ab9b0 con 0x7fd99c144c00 2024-01-15T18:14:13.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.036+0000 7fd9a1400640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd97406c430 0x7fd97406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.037+0000 7fd9a1400640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd97406c430 0x7fd97406e8f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd99c1542a0 tx=0x7fd98c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.040+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd994077f20 con 0x7fd99c144c00 2024-01-15T18:14:13.239 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.236+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd99c0686f0 con 0x7fd99c144c00 2024-01-15T18:14:13.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.237+0000 7fd999600640 1 -- 172.21.15.2:0/2726658661 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v110) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9940430a0 con 0x7fd99c144c00 2024-01-15T18:14:13.240 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.240+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd97406c430 msgr2=0x7fd97406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.240+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd97406c430 0x7fd97406e8f0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fd99c1542a0 tx=0x7fd98c008040 comp rx=0 tx=0).stop 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.240+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 msgr2=0x7fd99c0b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.240+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 0x7fd99c0b0a60 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fd994036040 tx=0x7fd994004a40 comp rx=0 tx=0).stop 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 shutdown_connections 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd97406c430 0x7fd97406e8f0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd99c144c00 0x7fd99c0b0a60 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd99c0cdd70 0x7fd99c137bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 --2- 172.21.15.2:0/2726658661 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd99c06d000 0x7fd99c0a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 >> 172.21.15.2:0/2726658661 conn(0x7fd99c0b45f0 msgr2=0x7fd99c13fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:13.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 shutdown_connections 2024-01-15T18:14:13.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.241+0000 7fd9a3e00640 1 -- 172.21.15.2:0/2726658661 wait complete. 2024-01-15T18:14:13.263 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-15T18:14:13.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.479+0000 7f9930c00640 1 Processor -- start 2024-01-15T18:14:13.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.479+0000 7f9930c00640 1 -- start start 2024-01-15T18:14:13.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.480+0000 7f9930c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f992c146dd0 0x7f992c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.480+0000 7f9930c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f992c06c9b0 0x7f992c0b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.480+0000 7f9930c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c0ac870 con 0x7f992c1469f0 2024-01-15T18:14:13.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f9930c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c06ce20 con 0x7f992c06c9b0 2024-01-15T18:14:13.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f9930c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c15d970 con 0x7f992c146dd0 2024-01-15T18:14:13.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f992c06c9b0 0x7f992c0b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f992c06c9b0 0x7f992c0b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37622/0 (socket says 172.21.15.2:37622) 2024-01-15T18:14:13.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 -- 172.21.15.2:0/336670764 learned_addr learned my addr 172.21.15.2:0/336670764 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f9923e00640 1 --2- 172.21.15.2:0/336670764 >> v2:172.21.15.137:3300/0 conn(0x7f992c146dd0 0x7f992c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 -- 172.21.15.2:0/336670764 >> v2:172.21.15.137:3300/0 conn(0x7f992c146dd0 msgr2=0x7f992c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 --2- 172.21.15.2:0/336670764 >> v2:172.21.15.137:3300/0 conn(0x7f992c146dd0 0x7f992c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 -- 172.21.15.2:0/336670764 >> v1:172.21.15.2:6789/0 conn(0x7f992c1469f0 legacy=0x7f992c071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.481+0000 7f992a800640 1 -- 172.21.15.2:0/336670764 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c15dc30 con 0x7f992c06c9b0 2024-01-15T18:14:13.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.482+0000 7f992a800640 1 --2- 172.21.15.2:0/336670764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c0b34a0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f992c06d600 tx=0x7f992402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38e88a957260976 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.483+0000 7f9929400640 1 -- 172.21.15.2:0/336670764 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992402cb10 con 0x7f992c06c9b0 2024-01-15T18:14:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.483+0000 7f9929400640 1 -- 172.21.15.2:0/336670764 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992402cc70 con 0x7f992c06c9b0 2024-01-15T18:14:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.483+0000 7f9929400640 1 -- 172.21.15.2:0/336670764 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99240343f0 con 0x7f992c06c9b0 2024-01-15T18:14:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.483+0000 7f9930c00640 1 -- 172.21.15.2:0/336670764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 msgr2=0x7f992c0b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.483+0000 7f9930c00640 1 --2- 172.21.15.2:0/336670764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c0b34a0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f992c06d600 tx=0x7f992402bf70 comp rx=0 tx=0).stop 2024-01-15T18:14:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.484+0000 7f9930c00640 1 -- 172.21.15.2:0/336670764 shutdown_connections 2024-01-15T18:14:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.484+0000 7f9930c00640 1 --2- 172.21.15.2:0/336670764 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c0b34a0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.484+0000 7f9930c00640 1 --2- 172.21.15.2:0/336670764 >> v2:172.21.15.137:3300/0 conn(0x7f992c146dd0 0x7f992c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.484+0000 7f9930c00640 1 -- 172.21.15.2:0/336670764 >> 172.21.15.2:0/336670764 conn(0x7f992c06da20 msgr2=0x7f992c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.484+0000 7f9930c00640 1 -- 172.21.15.2:0/336670764 shutdown_connections 2024-01-15T18:14:13.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.484+0000 7f9930c00640 1 -- 172.21.15.2:0/336670764 wait complete. 2024-01-15T18:14:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.485+0000 7f9930c00640 1 Processor -- start 2024-01-15T18:14:13.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.485+0000 7f9930c00640 1 -- start start 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.486+0000 7f9930c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.486+0000 7f9930c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f992c1469f0 0x7f992c1f6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.486+0000 7f9930c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f992c14ef90 0x7f992c161590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.486+0000 7f9930c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c1f73e0 con 0x7f992c06c9b0 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.486+0000 7f9930c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c145090 con 0x7f992c14ef90 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.486+0000 7f9930c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992c145370 con 0x7f992c1469f0 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c152770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37628/0 (socket says 172.21.15.2:37628) 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 -- 172.21.15.2:0/1180718233 learned_addr learned my addr 172.21.15.2:0/1180718233 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f992a800640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f992c14ef90 0x7f992c161590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9923e00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f992c1469f0 0x7f992c1f6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 -- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f992c1469f0 msgr2=0x7f992c1f6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f992c1469f0 0x7f992c1f6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 -- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f992c14ef90 msgr2=0x7f992c161590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f992c14ef90 0x7f992c161590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f9929e00640 1 -- 172.21.15.2:0/1180718233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9924009b30 con 0x7f992c06c9b0 2024-01-15T18:14:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.487+0000 7f992a800640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f992c14ef90 0x7f992c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:13.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.488+0000 7f9929e00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c152770 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c450 tx=0x7f991c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.488+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c0052d0 con 0x7f992c06c9b0 2024-01-15T18:14:13.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.488+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c005430 con 0x7f992c06c9b0 2024-01-15T18:14:13.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.488+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c012870 con 0x7f992c06c9b0 2024-01-15T18:14:13.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.488+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c145650 con 0x7f992c06c9b0 2024-01-15T18:14:13.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.488+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c14ea70 con 0x7f992c06c9b0 2024-01-15T18:14:13.492 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.489+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c0686f0 con 0x7f992c06c9b0 2024-01-15T18:14:13.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.490+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f991c0129d0 con 0x7f992c06c9b0 2024-01-15T18:14:13.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.491+0000 7f9922000640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f990406c430 0x7f990406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.491+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4997+0+0 (secure 0 0 0) 0x7f991c088620 con 0x7f992c06c9b0 2024-01-15T18:14:13.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.492+0000 7f9923e00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f990406c430 0x7f990406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.493+0000 7f9923e00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f990406c430 0x7f990406e8f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f992c15d2f0 tx=0x7f9914008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.496+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f991c054b90 con 0x7f992c06c9b0 2024-01-15T18:14:13.694 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.691+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f992c0ace10 con 0x7f992c06c9b0 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.692+0000 7f9922000640 1 -- 172.21.15.2:0/1180718233 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v110) v1 ==== 84+0+65 (secure 0 0 0) 0x7f991c0581b0 con 0x7f992c06c9b0 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout: "poolnum": 1, 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout: "poolname": ".mgr" 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:13.695 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f990406c430 msgr2=0x7f990406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f990406c430 0x7f990406e8f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f992c15d2f0 tx=0x7f9914008040 comp rx=0 tx=0).stop 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 msgr2=0x7f992c152770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c152770 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c450 tx=0x7f991c00c910 comp rx=0 tx=0).stop 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 shutdown_connections 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f990406c430 0x7f990406e8f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f992c14ef90 0x7f992c161590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f992c1469f0 0x7f992c1f6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 --2- 172.21.15.2:0/1180718233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f992c06c9b0 0x7f992c152770 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 >> 172.21.15.2:0/1180718233 conn(0x7f992c06da20 msgr2=0x7f992c0ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 shutdown_connections 2024-01-15T18:14:13.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.695+0000 7f9930c00640 1 -- 172.21.15.2:0/1180718233 wait complete. 2024-01-15T18:14:13.716 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-15T18:14:13.716 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-15T18:14:13.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.939+0000 7f3302e00640 1 Processor -- start 2024-01-15T18:14:13.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.939+0000 7f3302e00640 1 -- start start 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.940+0000 7f3302e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f32fc143620 0x7f32fc0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.940+0000 7f3302e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.940+0000 7f3302e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32fc0b2fa0 con 0x7f32fc071770 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.940+0000 7f3302e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32fc071c20 con 0x7f32fc143620 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.940+0000 7f3302e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0b3280 con 0x7f32fc0b2a90 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3301a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f32fc143620 0x7f32fc0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3302400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f32fc071770 0x7f32fc070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46510/0 (socket says 172.21.15.2:46510) 2024-01-15T18:14:13.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3302400640 1 -- 172.21.15.2:0/2909496590 learned_addr learned my addr 172.21.15.2:0/2909496590 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:13.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3301000640 1 --2- 172.21.15.2:0/2909496590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3301a00640 1 -- 172.21.15.2:0/2909496590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 msgr2=0x7f32fc071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:13.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3301a00640 1 --2- 172.21.15.2:0/2909496590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc071230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3301a00640 1 -- 172.21.15.2:0/2909496590 >> v1:172.21.15.2:6789/0 conn(0x7f32fc071770 legacy=0x7f32fc070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.941+0000 7f3301a00640 1 -- 172.21.15.2:0/2909496590 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32fc0b3560 con 0x7f32fc143620 2024-01-15T18:14:13.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.942+0000 7f3301a00640 1 --2- 172.21.15.2:0/2909496590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc0b2550 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0097a0 tx=0x7f32ec02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35b40c56a883cb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f32fbe00640 1 -- 172.21.15.2:0/2909496590 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec02d040 con 0x7f32fc143620 2024-01-15T18:14:13.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f32fbe00640 1 -- 172.21.15.2:0/2909496590 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ec031810 con 0x7f32fc143620 2024-01-15T18:14:13.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f32fbe00640 1 -- 172.21.15.2:0/2909496590 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ec031ba0 con 0x7f32fc143620 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 -- 172.21.15.2:0/2909496590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 msgr2=0x7f32fc0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 --2- 172.21.15.2:0/2909496590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc0b2550 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0097a0 tx=0x7f32ec02b8b0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 -- 172.21.15.2:0/2909496590 shutdown_connections 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 --2- 172.21.15.2:0/2909496590 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 --2- 172.21.15.2:0/2909496590 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc0b2550 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 -- 172.21.15.2:0/2909496590 >> 172.21.15.2:0/2909496590 conn(0x7f32fc0b0840 msgr2=0x7f32fc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 -- 172.21.15.2:0/2909496590 shutdown_connections 2024-01-15T18:14:13.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.943+0000 7f3302e00640 1 -- 172.21.15.2:0/2909496590 wait complete. 2024-01-15T18:14:13.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.945+0000 7f3302e00640 1 Processor -- start 2024-01-15T18:14:13.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.945+0000 7f3302e00640 1 -- start start 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32fc071770 0x7f32fc0cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc13a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc155230 con 0x7f32fc143620 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc0cda00 con 0x7f32fc071770 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32fc155480 con 0x7f32fc0b2a90 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.946+0000 7f3302400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37644/0 (socket says 172.21.15.2:37644) 2024-01-15T18:14:13.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 -- 172.21.15.2:0/1672178005 learned_addr learned my addr 172.21.15.2:0/1672178005 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:13.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3301a00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32fc071770 0x7f32fc0cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3301000640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc13a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 -- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 msgr2=0x7f32fc13a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc13a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 -- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32fc071770 msgr2=0x7f32fc0cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32fc071770 0x7f32fc0cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 -- 172.21.15.2:0/1672178005 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec009400 con 0x7f32fc143620 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3301a00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32fc071770 0x7f32fc0cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f3302400640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc152680 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f32f400cbc0 tx=0x7f32f400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.947+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f400ad40 con 0x7f32fc143620 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.948+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f4012400 con 0x7f32fc143620 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.948+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f400aea0 con 0x7f32fc143620 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.948+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32fc155760 con 0x7f32fc143620 2024-01-15T18:14:13.951 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.948+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32fc0cd6f0 con 0x7f32fc143620 2024-01-15T18:14:13.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.949+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32fc155c80 con 0x7f32fc143620 2024-01-15T18:14:13.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.950+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f32f4007070 con 0x7f32fc143620 2024-01-15T18:14:13.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.951+0000 7f32f9600640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32d406c430 0x7f32d406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:13.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.951+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 4997+0+0 (secure 0 0 0) 0x7f32f4088950 con 0x7f32fc143620 2024-01-15T18:14:13.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.951+0000 7f3301a00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32d406c430 0x7f32d406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:13.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.953+0000 7f3301a00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32d406c430 0x7f32d406e8f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f32ec036000 tx=0x7f32ec03c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:13.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:13.955+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32fc155c80 con 0x7f32fc143620 2024-01-15T18:14:14.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.149+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f32fc155c80 con 0x7f32fc143620 2024-01-15T18:14:14.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.824+0000 7f32f9600640 1 -- 172.21.15.2:0/1672178005 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v111) v1 ==== 138+0+0 (secure 0 0 0) 0x7f32fc155c80 con 0x7f32fc143620 2024-01-15T18:14:14.827 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_metadata' created 2024-01-15T18:14:14.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32d406c430 msgr2=0x7f32d406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:14.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32d406c430 0x7f32d406e8f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f32ec036000 tx=0x7f32ec03c040 comp rx=0 tx=0).stop 2024-01-15T18:14:14.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 msgr2=0x7f32fc152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc152680 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f32f400cbc0 tx=0x7f32f400a430 comp rx=0 tx=0).stop 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 shutdown_connections 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32d406c430 0x7f32d406e8f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32fc143620 0x7f32fc152680 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32fc0b2a90 0x7f32fc13a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 --2- 172.21.15.2:0/1672178005 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32fc071770 0x7f32fc0cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.827+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 >> 172.21.15.2:0/1672178005 conn(0x7f32fc0b0840 msgr2=0x7f32fc141620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.828+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 shutdown_connections 2024-01-15T18:14:14.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:14.828+0000 7f3302e00640 1 -- 172.21.15.2:0/1672178005 wait complete. 2024-01-15T18:14:14.860 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-15T18:14:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.079+0000 7f45ab200640 1 Processor -- start 2024-01-15T18:14:15.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.080+0000 7f45ab200640 1 -- start start 2024-01-15T18:14:15.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45ab200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f45ac0720d0 0x7f45ac06c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45ab200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac071090 con 0x7f45ac15f260 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45ab200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45ac144560 con 0x7f45ac143d20 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45ab200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45ac071430 con 0x7f45ac0720d0 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45aa800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f45ac0720d0 0x7f45ac06c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45aa800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f45ac0720d0 0x7f45ac06c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37660/0 (socket says 172.21.15.2:37660) 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.082+0000 7f45aa800640 1 -- 172.21.15.2:0/488139643 learned_addr learned my addr 172.21.15.2:0/488139643 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:15.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.083+0000 7f45aa800640 1 -- 172.21.15.2:0/488139643 >> v1:172.21.15.137:6789/0 conn(0x7f45ac143d20 legacy=0x7f45ac15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:15.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.083+0000 7f45aa800640 1 -- 172.21.15.2:0/488139643 >> v1:172.21.15.2:6789/0 conn(0x7f45ac15f260 legacy=0x7f45ac06c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.083+0000 7f45aa800640 1 -- 172.21.15.2:0/488139643 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ac144280 con 0x7f45ac0720d0 2024-01-15T18:14:15.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.084+0000 7f45aa800640 1 --2- 172.21.15.2:0/488139643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac0720d0 0x7f45ac06c860 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009e50 tx=0x7f45a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a386d3a54597a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:15.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.084+0000 7f45a8a00640 1 -- 172.21.15.2:0/488139643 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a402ca80 con 0x7f45ac0720d0 2024-01-15T18:14:15.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.084+0000 7f45a8a00640 1 -- 172.21.15.2:0/488139643 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a402cbe0 con 0x7f45ac0720d0 2024-01-15T18:14:15.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.084+0000 7f45a8a00640 1 -- 172.21.15.2:0/488139643 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a40343f0 con 0x7f45ac0720d0 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 -- 172.21.15.2:0/488139643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac0720d0 msgr2=0x7f45ac06c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 --2- 172.21.15.2:0/488139643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac0720d0 0x7f45ac06c860 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009e50 tx=0x7f45a402bee0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 -- 172.21.15.2:0/488139643 shutdown_connections 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 --2- 172.21.15.2:0/488139643 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac0720d0 0x7f45ac06c860 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 -- 172.21.15.2:0/488139643 >> 172.21.15.2:0/488139643 conn(0x7f45ac0b0840 msgr2=0x7f45ac0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 -- 172.21.15.2:0/488139643 shutdown_connections 2024-01-15T18:14:15.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.085+0000 7f45ab200640 1 -- 172.21.15.2:0/488139643 wait complete. 2024-01-15T18:14:15.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.087+0000 7f45ab200640 1 Processor -- start 2024-01-15T18:14:15.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.087+0000 7f45ab200640 1 -- start start 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45ab200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45ac0720d0 0x7f45ac15d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45ab200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45ac143d20 0x7f45ac1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45ab200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 0x7f45ac13d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45ab200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac15c3e0 con 0x7f45ac15f260 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45ab200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac15da40 con 0x7f45ac143d20 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45ab200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45ac15b860 con 0x7f45ac0720d0 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45aa800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 0x7f45ac13d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45aa800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 0x7f45ac13d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37662/0 (socket says 172.21.15.2:37662) 2024-01-15T18:14:15.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.088+0000 7f45aa800640 1 -- 172.21.15.2:0/912112750 learned_addr learned my addr 172.21.15.2:0/912112750 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45a9400640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45ac143d20 0x7f45ac1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45a9e00640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45ac0720d0 0x7f45ac15d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45aa800640 1 -- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45ac0720d0 msgr2=0x7f45ac15d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45aa800640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45ac0720d0 0x7f45ac15d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45aa800640 1 -- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45ac143d20 msgr2=0x7f45ac1401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45aa800640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45ac143d20 0x7f45ac1401d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45aa800640 1 -- 172.21.15.2:0/912112750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a4009b30 con 0x7f45ac15f260 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.089+0000 7f45aa800640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 0x7f45ac13d1d0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f45a4036040 tx=0x7f45a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.090+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a403e450 con 0x7f45ac15f260 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.090+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a403e5d0 con 0x7f45ac15f260 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.090+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a403e980 con 0x7f45ac15f260 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.090+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45ac15bb40 con 0x7f45ac15f260 2024-01-15T18:14:15.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.090+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ac15c100 con 0x7f45ac15f260 2024-01-15T18:14:15.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.091+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45ac0a84f0 con 0x7f45ac15f260 2024-01-15T18:14:15.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.092+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f45a402cd50 con 0x7f45ac15f260 2024-01-15T18:14:15.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.094+0000 7f45a2000640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458406c460 0x7f458406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:15.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.094+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5367+0+0 (secure 0 0 0) 0x7f45a40abb30 con 0x7f45ac15f260 2024-01-15T18:14:15.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.095+0000 7f45a9e00640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458406c460 0x7f458406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:15.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.096+0000 7f45a9e00640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458406c460 0x7f458406e920 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f459c004ea0 tx=0x7f459c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:15.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.098+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45a40781c0 con 0x7f45ac15f260 2024-01-15T18:14:15.297 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.294+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f45ac0a84f0 con 0x7f45ac15f260 2024-01-15T18:14:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.842+0000 7f45a2000640 1 -- 172.21.15.2:0/912112750 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v112) v1 ==== 170+0+0 (secure 0 0 0) 0x7f45ac0a84f0 con 0x7f45ac15f260 2024-01-15T18:14:15.845 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'cephfs_data' created 2024-01-15T18:14:15.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.846+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458406c460 msgr2=0x7f458406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.846+0000 7f45ab200640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458406c460 0x7f458406e920 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f459c004ea0 tx=0x7f459c009040 comp rx=0 tx=0).stop 2024-01-15T18:14:15.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.846+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 msgr2=0x7f45ac13d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:15.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.846+0000 7f45ab200640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 0x7f45ac13d1d0 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f45a4036040 tx=0x7f45a4004a40 comp rx=0 tx=0).stop 2024-01-15T18:14:15.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.847+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 shutdown_connections 2024-01-15T18:14:15.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.847+0000 7f45ab200640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f458406c460 0x7f458406e920 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.847+0000 7f45ab200640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f45ac15f260 0x7f45ac13d1d0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.847+0000 7f45ab200640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f45ac143d20 0x7f45ac1401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.847+0000 7f45ab200640 1 --2- 172.21.15.2:0/912112750 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f45ac0720d0 0x7f45ac15d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:15.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.847+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 >> 172.21.15.2:0/912112750 conn(0x7f45ac0b0840 msgr2=0x7f45ac0ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:15.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.848+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 shutdown_connections 2024-01-15T18:14:15.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:15.848+0000 7f45ab200640 1 -- 172.21.15.2:0/912112750 wait complete. 2024-01-15T18:14:15.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-15T18:14:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.072+0000 7f5af3000640 1 Processor -- start 2024-01-15T18:14:16.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.072+0000 7f5af3000640 1 -- start start 2024-01-15T18:14:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af3000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5aec143620 0x7f5aec0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec0b2a90 0x7f5aec071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af3000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aec0b2fa0 con 0x7f5aec071770 2024-01-15T18:14:16.076 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af3000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aec071c20 con 0x7f5aec143620 2024-01-15T18:14:16.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec0b3280 con 0x7f5aec0b2a90 2024-01-15T18:14:16.077 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af1c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5aec143620 0x7f5aec0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af1c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5aec143620 0x7f5aec0b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37668/0 (socket says 172.21.15.2:37668) 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af1c00640 1 -- 172.21.15.2:0/3503375443 learned_addr learned my addr 172.21.15.2:0/3503375443 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af2600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5aec071770 0x7f5aec070110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46528/0 (socket says 172.21.15.2:46528) 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.074+0000 7f5af1200640 1 --2- 172.21.15.2:0/3503375443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec0b2a90 0x7f5aec071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.075+0000 7f5af1c00640 1 -- 172.21.15.2:0/3503375443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec0b2a90 msgr2=0x7f5aec071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.075+0000 7f5af1c00640 1 --2- 172.21.15.2:0/3503375443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec0b2a90 0x7f5aec071230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.075+0000 7f5af1c00640 1 -- 172.21.15.2:0/3503375443 >> v1:172.21.15.2:6789/0 conn(0x7f5aec071770 legacy=0x7f5aec070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.075+0000 7f5af1c00640 1 -- 172.21.15.2:0/3503375443 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec0b3560 con 0x7f5aec143620 2024-01-15T18:14:16.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.075+0000 7f5aebe00640 1 -- 172.21.15.2:0/3503375443 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083085391 0 0) 0x7f5aec0b2fa0 con 0x7f5aec071770 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.076+0000 7f5af1c00640 1 --2- 172.21.15.2:0/3503375443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec143620 0x7f5aec0b2550 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009870 tx=0x7f5adc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6486525bcb27727 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.076+0000 7f5aebe00640 1 -- 172.21.15.2:0/3503375443 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc030840 con 0x7f5aec143620 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.076+0000 7f5aebe00640 1 -- 172.21.15.2:0/3503375443 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5adc0309a0 con 0x7f5aec143620 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.076+0000 7f5aebe00640 1 -- 172.21.15.2:0/3503375443 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5adc030d30 con 0x7f5aec143620 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 -- 172.21.15.2:0/3503375443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec143620 msgr2=0x7f5aec0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 --2- 172.21.15.2:0/3503375443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec143620 0x7f5aec0b2550 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f5adc009870 tx=0x7f5adc02b980 comp rx=0 tx=0).stop 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 -- 172.21.15.2:0/3503375443 shutdown_connections 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 --2- 172.21.15.2:0/3503375443 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec0b2a90 0x7f5aec071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 --2- 172.21.15.2:0/3503375443 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec143620 0x7f5aec0b2550 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 -- 172.21.15.2:0/3503375443 >> 172.21.15.2:0/3503375443 conn(0x7f5aec0b0840 msgr2=0x7f5aec0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 -- 172.21.15.2:0/3503375443 shutdown_connections 2024-01-15T18:14:16.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.077+0000 7f5af3000640 1 -- 172.21.15.2:0/3503375443 wait complete. 2024-01-15T18:14:16.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.079+0000 7f5af3000640 1 Processor -- start 2024-01-15T18:14:16.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.079+0000 7f5af3000640 1 -- start start 2024-01-15T18:14:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af3000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec071770 0x7f5aec151370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af3000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 0x7f5aec152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af3000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5aec143620 0x7f5aec0cf180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:16.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af3000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec15d220 con 0x7f5aec0b2a90 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af3000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec152610 con 0x7f5aec143620 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af3000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aec152fe0 con 0x7f5aec071770 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 0x7f5aec152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 0x7f5aec152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37676/0 (socket says 172.21.15.2:37676) 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.080+0000 7f5af1200640 1 -- 172.21.15.2:0/619542835 learned_addr learned my addr 172.21.15.2:0/619542835 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1c00640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec071770 0x7f5aec151370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af2600640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5aec143620 0x7f5aec0cf180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1200640 1 -- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec071770 msgr2=0x7f5aec151370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1200640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec071770 0x7f5aec151370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1200640 1 -- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5aec143620 msgr2=0x7f5aec0cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1200640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5aec143620 0x7f5aec0cf180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1200640 1 -- 172.21.15.2:0/619542835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0094d0 con 0x7f5aec0b2a90 2024-01-15T18:14:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5af1200640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 0x7f5aec152aa0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400d7d0 tx=0x7f5ad400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.081+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400b6c0 con 0x7f5aec0b2a90 2024-01-15T18:14:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.082+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad400b840 con 0x7f5aec0b2a90 2024-01-15T18:14:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.082+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400bbf0 con 0x7f5aec0b2a90 2024-01-15T18:14:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.082+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec0cf6c0 con 0x7f5aec0b2a90 2024-01-15T18:14:16.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.082+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec151ce0 con 0x7f5aec0b2a90 2024-01-15T18:14:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.083+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aec152780 con 0x7f5aec0b2a90 2024-01-15T18:14:16.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.084+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f5ad400bd50 con 0x7f5aec0b2a90 2024-01-15T18:14:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.085+0000 7f5ae9600640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ac406c460 0x7f5ac406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.086+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5749+0+0 (secure 0 0 0) 0x7f5ad4089020 con 0x7f5aec0b2a90 2024-01-15T18:14:16.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.086+0000 7f5af1c00640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ac406c460 0x7f5ac406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:16.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.087+0000 7f5af1c00640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ac406c460 0x7f5ac406e920 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5adc036000 tx=0x7f5adc03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:16.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.091+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ad4055590 con 0x7f5aec0b2a90 2024-01-15T18:14:16.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.320+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5aec074770 con 0x7f5aec0b2a90 2024-01-15T18:14:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.887+0000 7f5ae9600640 1 -- 172.21.15.2:0/619542835 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5ad4058bb0 con 0x7f5aec0b2a90 2024-01-15T18:14:16.891 INFO:teuthology.orchestra.run.smithi002.stderr:new fs with metadata pool 16 and data pool 17 2024-01-15T18:14:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.893+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ac406c460 msgr2=0x7f5ac406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.893+0000 7f5af3000640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ac406c460 0x7f5ac406e920 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5adc036000 tx=0x7f5adc03b040 comp rx=0 tx=0).stop 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.893+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 msgr2=0x7f5aec152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.893+0000 7f5af3000640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 0x7f5aec152aa0 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f5ad400d7d0 tx=0x7f5ad400dc90 comp rx=0 tx=0).stop 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 shutdown_connections 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5ac406c460 0x7f5ac406e920 secure :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f5adc036000 tx=0x7f5adc03b040 comp rx=0 tx=0).stop 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5aec143620 0x7f5aec0cf180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5aec0b2a90 0x7f5aec152aa0 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 --2- 172.21.15.2:0/619542835 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5aec071770 0x7f5aec151370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 >> 172.21.15.2:0/619542835 conn(0x7f5aec0b0840 msgr2=0x7f5aec06de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.894+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 shutdown_connections 2024-01-15T18:14:16.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:16.895+0000 7f5af3000640 1 -- 172.21.15.2:0/619542835 wait complete. 2024-01-15T18:14:16.923 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:17.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.103+0000 7fe327200640 1 Processor -- start 2024-01-15T18:14:17.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.104+0000 7fe327200640 1 -- start start 2024-01-15T18:14:17.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.105+0000 7fe327200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 0x7fe32015e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.105+0000 7fe327200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32013d9f0 0x7fe32006e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.105+0000 7fe327200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3200ceee0 con 0x7fe32013d560 2024-01-15T18:14:17.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.105+0000 7fe327200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32015ac20 con 0x7fe32015e0a0 2024-01-15T18:14:17.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.105+0000 7fe327200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32015e9c0 con 0x7fe32013d9f0 2024-01-15T18:14:17.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.106+0000 7fe31fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 0x7fe32015e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.106+0000 7fe325200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32013d9f0 0x7fe32006e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.106+0000 7fe31fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 0x7fe32015e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:49102/0 (socket says 172.21.15.2:49102) 2024-01-15T18:14:17.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.106+0000 7fe31fe00640 1 -- 172.21.15.2:0/1724560482 learned_addr learned my addr 172.21.15.2:0/1724560482 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:17.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.107+0000 7fe325200640 1 -- 172.21.15.2:0/1724560482 >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 msgr2=0x7fe32015e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:17.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.107+0000 7fe325200640 1 --2- 172.21.15.2:0/1724560482 >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 0x7fe32015e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.107+0000 7fe325200640 1 -- 172.21.15.2:0/1724560482 >> v1:172.21.15.137:6789/0 conn(0x7fe32013d560 legacy=0x7fe32015a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.107+0000 7fe325200640 1 -- 172.21.15.2:0/1724560482 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32015ae80 con 0x7fe32013d9f0 2024-01-15T18:14:17.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.107+0000 7fe31fe00640 1 --2- 172.21.15.2:0/1724560482 >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 0x7fe32015e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:14:17.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.108+0000 7fe325200640 1 --2- 172.21.15.2:0/1724560482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32013d9f0 0x7fe32006e8c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe318009d90 tx=0x7fe31802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6390df77ea70bdc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:17.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.109+0000 7fe31ea00640 1 -- 172.21.15.2:0/1724560482 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31802cc50 con 0x7fe32013d9f0 2024-01-15T18:14:17.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.109+0000 7fe31ea00640 1 -- 172.21.15.2:0/1724560482 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31802cdb0 con 0x7fe32013d9f0 2024-01-15T18:14:17.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.109+0000 7fe31ea00640 1 -- 172.21.15.2:0/1724560482 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318034660 con 0x7fe32013d9f0 2024-01-15T18:14:17.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 -- 172.21.15.2:0/1724560482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32013d9f0 msgr2=0x7fe32006e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 --2- 172.21.15.2:0/1724560482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32013d9f0 0x7fe32006e8c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe318009d90 tx=0x7fe31802bea0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 -- 172.21.15.2:0/1724560482 shutdown_connections 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 --2- 172.21.15.2:0/1724560482 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32013d9f0 0x7fe32006e8c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 --2- 172.21.15.2:0/1724560482 >> v2:172.21.15.137:3300/0 conn(0x7fe32015e0a0 0x7fe32015e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 -- 172.21.15.2:0/1724560482 >> 172.21.15.2:0/1724560482 conn(0x7fe32006d410 msgr2=0x7fe32006d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 -- 172.21.15.2:0/1724560482 shutdown_connections 2024-01-15T18:14:17.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.110+0000 7fe327200640 1 -- 172.21.15.2:0/1724560482 wait complete. 2024-01-15T18:14:17.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.112+0000 7fe327200640 1 Processor -- start 2024-01-15T18:14:17.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.112+0000 7fe327200640 1 -- start start 2024-01-15T18:14:17.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe327200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 0x7fe32012c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe327200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32013d9f0 0x7fe320152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe327200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32015e0a0 0x7fe320150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe327200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200b4470 con 0x7fe32013d560 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe327200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32012c660 con 0x7fe32013d9f0 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe327200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32012c940 con 0x7fe32015e0a0 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe31fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 0x7fe32012c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe31fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 0x7fe32012c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37680/0 (socket says 172.21.15.2:37680) 2024-01-15T18:14:17.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe31fe00640 1 -- 172.21.15.2:0/3044415876 learned_addr learned my addr 172.21.15.2:0/3044415876 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:17.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.113+0000 7fe325200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32015e0a0 0x7fe320150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31f400640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32013d9f0 0x7fe320152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31fe00640 1 -- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32015e0a0 msgr2=0x7fe320150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31fe00640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32015e0a0 0x7fe320150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31fe00640 1 -- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32013d9f0 msgr2=0x7fe320152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31fe00640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32013d9f0 0x7fe320152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31fe00640 1 -- 172.21.15.2:0/3044415876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100094d0 con 0x7fe32013d560 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31f400640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32013d9f0 0x7fe320152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe31fe00640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 0x7fe32012c010 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fe31000e9b0 tx=0x7fe31000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31000c9a0 con 0x7fe32013d560 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31000cb00 con 0x7fe32013d560 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31000ce70 con 0x7fe32013d560 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.114+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe318009a70 con 0x7fe32013d560 2024-01-15T18:14:17.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.115+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3201530f0 con 0x7fe32013d560 2024-01-15T18:14:17.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.116+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200686f0 con 0x7fe32013d560 2024-01-15T18:14:17.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.117+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fe310013470 con 0x7fe32013d560 2024-01-15T18:14:17.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.117+0000 7fe317e00640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f806c430 0x7fe2f806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.118+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fe310089030 con 0x7fe32013d560 2024-01-15T18:14:17.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.118+0000 7fe31f400640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f806c430 0x7fe2f806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.119+0000 7fe31f400640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f806c430 0x7fe2f806e8f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe320152410 tx=0x7fe308009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:17.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.122+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe310055600 con 0x7fe32013d560 2024-01-15T18:14:17.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.319+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3200b45b0 con 0x7fe32013d560 2024-01-15T18:14:17.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.321+0000 7fe317e00640 1 -- 172.21.15.2:0/3044415876 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16701 (secure 0 0 0) 0x7fe3200b45b0 con 0x7fe32013d560 2024-01-15T18:14:17.324 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:17.324 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":113,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:16.322304+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:17.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f806c430 msgr2=0x7fe2f806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f806c430 0x7fe2f806e8f0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fe320152410 tx=0x7fe308009040 comp rx=0 tx=0).stop 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 msgr2=0x7fe32012c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 0x7fe32012c010 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7fe31000e9b0 tx=0x7fe31000ee70 comp rx=0 tx=0).stop 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 shutdown_connections 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe2f806c430 0x7fe2f806e8f0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe32015e0a0 0x7fe320150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe32013d9f0 0x7fe320152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 --2- 172.21.15.2:0/3044415876 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe32013d560 0x7fe32012c010 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.323+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 >> 172.21.15.2:0/3044415876 conn(0x7fe32006d410 msgr2=0x7fe3201598f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.324+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 shutdown_connections 2024-01-15T18:14:17.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.324+0000 7fe327200640 1 -- 172.21.15.2:0/3044415876 wait complete. 2024-01-15T18:14:17.344 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:17.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.565+0000 7efeed600640 1 Processor -- start 2024-01-15T18:14:17.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.565+0000 7efeed600640 1 -- start start 2024-01-15T18:14:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.566+0000 7efeed600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee8143810 0x7efee806dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.566+0000 7efeed600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7efee80b2250 0x7efee80cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.566+0000 7efeed600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efee806e470 con 0x7efee806e010 2024-01-15T18:14:17.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.566+0000 7efeed600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efee80b3520 con 0x7efee80b2250 2024-01-15T18:14:17.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.566+0000 7efeed600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee80ccbe0 con 0x7efee8143810 2024-01-15T18:14:17.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efedf200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7efee806e010 0x7efee80b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46534/0 (socket says 172.21.15.2:46534) 2024-01-15T18:14:17.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efedf200640 1 -- 172.21.15.2:0/1861937947 learned_addr learned my addr 172.21.15.2:0/1861937947 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:17.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efeecc00640 1 --2- 172.21.15.2:0/1861937947 >> v2:172.21.15.2:3300/0 conn(0x7efee80b2250 0x7efee80cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efee7e00640 1 --2- 172.21.15.2:0/1861937947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee8143810 0x7efee806dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efee7400640 1 -- 172.21.15.2:0/1861937947 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 373687899 0 0) 0x7efed0001110 con 0x7efee806e010 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efee7400640 1 -- 172.21.15.2:0/1861937947 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec8003520 con 0x7efee806e010 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.567+0000 7efeecc00640 1 -- 172.21.15.2:0/1861937947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee8143810 msgr2=0x7efee806dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.568+0000 7efeecc00640 1 --2- 172.21.15.2:0/1861937947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee8143810 0x7efee806dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.568+0000 7efeecc00640 1 -- 172.21.15.2:0/1861937947 >> v1:172.21.15.2:6789/0 conn(0x7efee806e010 legacy=0x7efee80b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.568+0000 7efeecc00640 1 -- 172.21.15.2:0/1861937947 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8143bf0 con 0x7efee80b2250 2024-01-15T18:14:17.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.569+0000 7efeecc00640 1 --2- 172.21.15.2:0/1861937947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80cd070 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7efee0009dc0 tx=0x7efee002be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd13d601ee42456d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.569+0000 7efee7400640 1 -- 172.21.15.2:0/1861937947 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee002cbb0 con 0x7efee80b2250 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.569+0000 7efee7400640 1 -- 172.21.15.2:0/1861937947 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee002cd10 con 0x7efee80b2250 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.569+0000 7efee7400640 1 -- 172.21.15.2:0/1861937947 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0034660 con 0x7efee80b2250 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 -- 172.21.15.2:0/1861937947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 msgr2=0x7efee80cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 --2- 172.21.15.2:0/1861937947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80cd070 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7efee0009dc0 tx=0x7efee002be00 comp rx=0 tx=0).stop 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 -- 172.21.15.2:0/1861937947 shutdown_connections 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 --2- 172.21.15.2:0/1861937947 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80cd070 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 --2- 172.21.15.2:0/1861937947 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee8143810 0x7efee806dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 -- 172.21.15.2:0/1861937947 >> 172.21.15.2:0/1861937947 conn(0x7efee80b0840 msgr2=0x7efee80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 -- 172.21.15.2:0/1861937947 shutdown_connections 2024-01-15T18:14:17.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.570+0000 7efeed600640 1 -- 172.21.15.2:0/1861937947 wait complete. 2024-01-15T18:14:17.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.571+0000 7efeed600640 1 Processor -- start 2024-01-15T18:14:17.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.572+0000 7efeed600640 1 -- start start 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.572+0000 7efeed600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee806e010 0x7efee815f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.572+0000 7efeed600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80ac500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efeed600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efee8143810 0x7efee80aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efeed600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee815e9d0 con 0x7efee80b2250 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efeed600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee815f270 con 0x7efee8143810 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efeed600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efee81612e0 con 0x7efee806e010 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80ac500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80ac500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37702/0 (socket says 172.21.15.2:37702) 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 -- 172.21.15.2:0/1657039607 learned_addr learned my addr 172.21.15.2:0/1657039607 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efee7e00640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee806e010 0x7efee815f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efeecc00640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efee8143810 0x7efee80aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 -- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee806e010 msgr2=0x7efee815f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee806e010 0x7efee815f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 -- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efee8143810 msgr2=0x7efee80aca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efee8143810 0x7efee80aca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efedf200640 1 -- 172.21.15.2:0/1657039607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0009a70 con 0x7efee80b2250 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.573+0000 7efeecc00640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efee8143810 0x7efee80aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.574+0000 7efedf200640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80ac500 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7efed000c450 tx=0x7efed000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.574+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0005ce0 con 0x7efee80b2250 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.574+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efed0005e40 con 0x7efee80b2250 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.574+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed0007290 con 0x7efee80b2250 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.574+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee815ec20 con 0x7efee80b2250 2024-01-15T18:14:17.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.574+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee815fc40 con 0x7efee80b2250 2024-01-15T18:14:17.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.575+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee815f400 con 0x7efee80b2250 2024-01-15T18:14:17.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.576+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7efed00073f0 con 0x7efee80b2250 2024-01-15T18:14:17.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.577+0000 7efee4c00640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efec006c430 0x7efec006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:17.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.577+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5849+0+0 (secure 0 0 0) 0x7efed0087dd0 con 0x7efee80b2250 2024-01-15T18:14:17.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.578+0000 7efee7e00640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efec006c430 0x7efec006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:17.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.579+0000 7efee7e00640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efec006c430 0x7efec006e8f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7efed8009ad0 tx=0x7efed8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:17.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.581+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efed00543a0 con 0x7efee80b2250 2024-01-15T18:14:17.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.784+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efee806e470 con 0x7efee80b2250 2024-01-15T18:14:17.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.787+0000 7efee4c00640 1 -- 172.21.15.2:0/1657039607 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16701 (secure 0 0 0) 0x7efee806e470 con 0x7efee80b2250 2024-01-15T18:14:17.790 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:17.790 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":113,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:16.322304+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:17.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.789+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efec006c430 msgr2=0x7efec006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.789+0000 7efeed600640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efec006c430 0x7efec006e8f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7efed8009ad0 tx=0x7efed8008040 comp rx=0 tx=0).stop 2024-01-15T18:14:17.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.789+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 msgr2=0x7efee80ac500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:17.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.789+0000 7efeed600640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80ac500 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7efed000c450 tx=0x7efed000c910 comp rx=0 tx=0).stop 2024-01-15T18:14:17.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 shutdown_connections 2024-01-15T18:14:17.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7efec006c430 0x7efec006e8f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7efee8143810 0x7efee80aca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7efee80b2250 0x7efee80ac500 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 --2- 172.21.15.2:0/1657039607 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7efee806e010 0x7efee815f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:17.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 >> 172.21.15.2:0/1657039607 conn(0x7efee80b0840 msgr2=0x7efee8071750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:17.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 shutdown_connections 2024-01-15T18:14:17.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:17.790+0000 7efeed600640 1 -- 172.21.15.2:0/1657039607 wait complete. 2024-01-15T18:14:17.812 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-15T18:14:17.828 INFO:tasks.ceph.mds.b.smithi137.stderr:2024-01-15T18:14:17.824+0000 7f6f02e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-15T18:14:18.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.026+0000 7f8e24400640 1 Processor -- start 2024-01-15T18:14:18.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.027+0000 7f8e24400640 1 -- start start 2024-01-15T18:14:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.028+0000 7f8e24400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c15b840 0x7f8e1c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.028+0000 7f8e24400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8e1c06e830 0x7f8e1c06ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.028+0000 7f8e24400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e1c06f1a0 con 0x7f8e1c144b90 2024-01-15T18:14:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.028+0000 7f8e24400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e1c144f70 con 0x7f8e1c06e830 2024-01-15T18:14:18.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.028+0000 7f8e24400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c070690 con 0x7f8e1c15b840 2024-01-15T18:14:18.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e21a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f8e1c144b90 0x7f8e1c0b4220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46542/0 (socket says 172.21.15.2:46542) 2024-01-15T18:14:18.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e21a00640 1 -- 172.21.15.2:0/648700977 learned_addr learned my addr 172.21.15.2:0/648700977 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:18.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e22400640 1 --2- 172.21.15.2:0/648700977 >> v2:172.21.15.2:3300/0 conn(0x7f8e1c06e830 0x7f8e1c06ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:18.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e21000640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c15b840 0x7f8e1c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e22400640 1 -- 172.21.15.2:0/648700977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c15b840 msgr2=0x7f8e1c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e22400640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c15b840 0x7f8e1c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e22400640 1 -- 172.21.15.2:0/648700977 >> v1:172.21.15.2:6789/0 conn(0x7f8e1c144b90 legacy=0x7f8e1c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.029+0000 7f8e22400640 1 -- 172.21.15.2:0/648700977 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c070970 con 0x7f8e1c06e830 2024-01-15T18:14:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.030+0000 7f8e21000640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c15b840 0x7f8e1c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:18.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.031+0000 7f8e22400640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c06ec60 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f8e140097a0 tx=0x7f8e1402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a0319a6fb3d0c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.031+0000 7f8e1be00640 1 -- 172.21.15.2:0/648700977 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1402d040 con 0x7f8e1c06e830 2024-01-15T18:14:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.031+0000 7f8e1be00640 1 -- 172.21.15.2:0/648700977 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e14031810 con 0x7f8e1c06e830 2024-01-15T18:14:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.031+0000 7f8e1be00640 1 -- 172.21.15.2:0/648700977 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e14031ba0 con 0x7f8e1c06e830 2024-01-15T18:14:18.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 -- 172.21.15.2:0/648700977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 msgr2=0x7f8e1c06ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c06ec60 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f8e140097a0 tx=0x7f8e1402b8b0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 -- 172.21.15.2:0/648700977 shutdown_connections 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c06ec60 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 --2- 172.21.15.2:0/648700977 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c15b840 0x7f8e1c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 -- 172.21.15.2:0/648700977 >> 172.21.15.2:0/648700977 conn(0x7f8e1c0b2c10 msgr2=0x7f8e1c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 -- 172.21.15.2:0/648700977 shutdown_connections 2024-01-15T18:14:18.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.032+0000 7f8e24400640 1 -- 172.21.15.2:0/648700977 wait complete. 2024-01-15T18:14:18.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.034+0000 7f8e24400640 1 Processor -- start 2024-01-15T18:14:18.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.034+0000 7f8e24400640 1 -- start start 2024-01-15T18:14:18.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e24400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c0b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:18.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e24400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c144b90 0x7f8e1c06b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e24400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e1c15b840 0x7f8e1c1f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e24400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c155a30 con 0x7f8e1c06e830 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e24400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c1f6640 con 0x7f8e1c15b840 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e24400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1c06bc50 con 0x7f8e1c144b90 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c0b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c0b4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37722/0 (socket says 172.21.15.2:37722) 2024-01-15T18:14:18.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 -- 172.21.15.2:0/907653930 learned_addr learned my addr 172.21.15.2:0/907653930 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:18.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21000640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c144b90 0x7f8e1c06b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:18.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e22400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e1c15b840 0x7f8e1c1f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:18.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 -- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c144b90 msgr2=0x7f8e1c06b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:18.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c144b90 0x7f8e1c06b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 -- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e1c15b840 msgr2=0x7f8e1c1f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e1c15b840 0x7f8e1c1f6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.035+0000 7f8e21a00640 1 -- 172.21.15.2:0/907653930 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e14009400 con 0x7f8e1c06e830 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.036+0000 7f8e21a00640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c0b4f80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00cb90 tx=0x7f8e0c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.036+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00ad40 con 0x7f8e1c06e830 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.036+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c012400 con 0x7f8e1c06e830 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.036+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c00aea0 con 0x7f8e1c06e830 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.036+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e1c06bf30 con 0x7f8e1c06e830 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.036+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e1c06c4f0 con 0x7f8e1c06e830 2024-01-15T18:14:18.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.038+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1c0686f0 con 0x7f8e1c06e830 2024-01-15T18:14:18.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.039+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8e0c007070 con 0x7f8e1c06e830 2024-01-15T18:14:18.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.039+0000 7f8e19600640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8df406c460 0x7f8df406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:18.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.040+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f8e0c088a80 con 0x7f8e1c06e830 2024-01-15T18:14:18.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.040+0000 7f8e21000640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8df406c460 0x7f8df406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:18.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.041+0000 7f8e21000640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8df406c460 0x7f8df406e920 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c15acb0 tx=0x7f8e04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:18.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.044+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e0c055050 con 0x7f8e1c06e830 2024-01-15T18:14:18.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.279+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8e1c152240 con 0x7f8e1c06e830 2024-01-15T18:14:18.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.860+0000 7f8e19600640 1 -- 172.21.15.2:0/907653930 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8e0c017020 con 0x7f8e1c06e830 2024-01-15T18:14:18.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8df406c460 msgr2=0x7f8df406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:18.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8df406c460 0x7f8df406e920 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8e1c15acb0 tx=0x7f8e04008040 comp rx=0 tx=0).stop 2024-01-15T18:14:18.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 msgr2=0x7f8e1c0b4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:18.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c0b4f80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00cb90 tx=0x7f8e0c00a430 comp rx=0 tx=0).stop 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 shutdown_connections 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8df406c460 0x7f8df406e920 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8e1c15b840 0x7f8e1c1f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8e1c144b90 0x7f8e1c06b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 --2- 172.21.15.2:0/907653930 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8e1c06e830 0x7f8e1c0b4f80 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.864+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 >> 172.21.15.2:0/907653930 conn(0x7f8e1c0b2c10 msgr2=0x7f8e1c152790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.865+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 shutdown_connections 2024-01-15T18:14:18.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:18.865+0000 7f8e24400640 1 -- 172.21.15.2:0/907653930 wait complete. 2024-01-15T18:14:18.887 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-15T18:14:18.888 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-15T18:14:19.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.118+0000 7fc78a200640 1 Processor -- start 2024-01-15T18:14:19.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.118+0000 7fc78a200640 1 -- start start 2024-01-15T18:14:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.119+0000 7fc78a200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc7840b4c70 0x7fc7840d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.119+0000 7fc78a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78415d730 0x7fc7840b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.119+0000 7fc78a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7840b5590 con 0x7fc7840b8c30 2024-01-15T18:14:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.119+0000 7fc78a200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7840d30b0 con 0x7fc7840b4c70 2024-01-15T18:14:19.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.119+0000 7fc78a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7840d1900 con 0x7fc78415d730 2024-01-15T18:14:19.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.120+0000 7fc783400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc7840b4c70 0x7fc7840d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:19.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.120+0000 7fc783400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc7840b4c70 0x7fc7840d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37734/0 (socket says 172.21.15.2:37734) 2024-01-15T18:14:19.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.120+0000 7fc783400640 1 -- 172.21.15.2:0/3595148198 learned_addr learned my addr 172.21.15.2:0/3595148198 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:19.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.120+0000 7fc782a00640 1 --2- 172.21.15.2:0/3595148198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78415d730 0x7fc7840b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:19.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.121+0000 7fc783400640 1 -- 172.21.15.2:0/3595148198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78415d730 msgr2=0x7fc7840b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:19.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.121+0000 7fc783400640 1 --2- 172.21.15.2:0/3595148198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78415d730 0x7fc7840b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.121+0000 7fc783400640 1 -- 172.21.15.2:0/3595148198 >> v1:172.21.15.137:6789/0 conn(0x7fc7840b8c30 legacy=0x7fc7840d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:19.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.121+0000 7fc783400640 1 -- 172.21.15.2:0/3595148198 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7840d1be0 con 0x7fc7840b4c70 2024-01-15T18:14:19.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.123+0000 7fc783400640 1 --2- 172.21.15.2:0/3595148198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7840b4c70 0x7fc7840d1270 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc774009870 tx=0x7fc77402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f65206cf30c86c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:19.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.123+0000 7fc782000640 1 -- 172.21.15.2:0/3595148198 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774030840 con 0x7fc7840b4c70 2024-01-15T18:14:19.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.123+0000 7fc782000640 1 -- 172.21.15.2:0/3595148198 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7740309a0 con 0x7fc7840b4c70 2024-01-15T18:14:19.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.123+0000 7fc782000640 1 -- 172.21.15.2:0/3595148198 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774030d30 con 0x7fc7840b4c70 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 -- 172.21.15.2:0/3595148198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7840b4c70 msgr2=0x7fc7840d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 --2- 172.21.15.2:0/3595148198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7840b4c70 0x7fc7840d1270 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fc774009870 tx=0x7fc77402b980 comp rx=0 tx=0).stop 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 -- 172.21.15.2:0/3595148198 shutdown_connections 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 --2- 172.21.15.2:0/3595148198 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc78415d730 0x7fc7840b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 --2- 172.21.15.2:0/3595148198 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc7840b4c70 0x7fc7840d1270 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 -- 172.21.15.2:0/3595148198 >> 172.21.15.2:0/3595148198 conn(0x7fc784072210 msgr2=0x7fc7840cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 -- 172.21.15.2:0/3595148198 shutdown_connections 2024-01-15T18:14:19.127 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.124+0000 7fc78a200640 1 -- 172.21.15.2:0/3595148198 wait complete. 2024-01-15T18:14:19.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.126+0000 7fc78a200640 1 Processor -- start 2024-01-15T18:14:19.129 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.126+0000 7fc78a200640 1 -- start start 2024-01-15T18:14:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc78a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7840b4c70 0x7fc78414da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:19.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc78a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7840b8c30 0x7fc7840b7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc78a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 0x7fc7840b81d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc78a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7840c8f50 con 0x7fc78415d730 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc78a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7840c7f40 con 0x7fc7840b8c30 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc78a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7840c8190 con 0x7fc7840b4c70 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc783e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 0x7fc7840b81d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.127+0000 7fc783e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 0x7fc7840b81d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37744/0 (socket says 172.21.15.2:37744) 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 -- 172.21.15.2:0/3052474592 learned_addr learned my addr 172.21.15.2:0/3052474592 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc782a00640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7840b8c30 0x7fc7840b7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783400640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7840b4c70 0x7fc78414da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 -- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7840b4c70 msgr2=0x7fc78414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7840b4c70 0x7fc78414da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 -- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7840b8c30 msgr2=0x7fc7840b7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7840b8c30 0x7fc7840b7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 -- 172.21.15.2:0/3052474592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7740094d0 con 0x7fc78415d730 2024-01-15T18:14:19.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.128+0000 7fc783e00640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 0x7fc7840b81d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c7a0 tx=0x7fc77c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:19.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.129+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c003b70 con 0x7fc78415d730 2024-01-15T18:14:19.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.129+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c003cf0 con 0x7fc78415d730 2024-01-15T18:14:19.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.129+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c00aa90 con 0x7fc78415d730 2024-01-15T18:14:19.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.129+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7840c8470 con 0x7fc78415d730 2024-01-15T18:14:19.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.129+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7840c8a30 con 0x7fc78415d730 2024-01-15T18:14:19.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.130+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7840686f0 con 0x7fc78415d730 2024-01-15T18:14:19.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.131+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc77c00abf0 con 0x7fc78415d730 2024-01-15T18:14:19.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.132+0000 7fc77b400640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc75c06c430 0x7fc75c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:19.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.132+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fc77c0888f0 con 0x7fc78415d730 2024-01-15T18:14:19.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.133+0000 7fc783400640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc75c06c430 0x7fc75c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:19.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.134+0000 7fc783400640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc75c06c430 0x7fc75c06e8f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc774036000 tx=0x7fc774008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:19.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.136+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc77c054e60 con 0x7fc78415d730 2024-01-15T18:14:19.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.368+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc78415cbe0 con 0x7fc78415d730 2024-01-15T18:14:19.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.881+0000 7fc77b400640 1 -- 172.21.15.2:0/3052474592 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v74) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc77c003e60 con 0x7fc78415d730 2024-01-15T18:14:19.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc75c06c430 msgr2=0x7fc75c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:19.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc75c06c430 0x7fc75c06e8f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc774036000 tx=0x7fc774008030 comp rx=0 tx=0).stop 2024-01-15T18:14:19.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 msgr2=0x7fc7840b81d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 0x7fc7840b81d0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c7a0 tx=0x7fc77c00cc60 comp rx=0 tx=0).stop 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 shutdown_connections 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc75c06c430 0x7fc75c06e8f0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc78415d730 0x7fc7840b81d0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc7840b8c30 0x7fc7840b7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 --2- 172.21.15.2:0/3052474592 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc7840b4c70 0x7fc78414da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 >> 172.21.15.2:0/3052474592 conn(0x7fc784072210 msgr2=0x7fc7840b64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.885+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 shutdown_connections 2024-01-15T18:14:19.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:19.886+0000 7fc78a200640 1 -- 172.21.15.2:0/3052474592 wait complete. 2024-01-15T18:14:19.911 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-15T18:14:19.911 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:14:20.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.131+0000 7f76ff400640 1 Processor -- start 2024-01-15T18:14:20.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.131+0000 7f76ff400640 1 -- start start 2024-01-15T18:14:20.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76ff400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f76f806e220 0x7f76f80b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76ff400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f76f80b2fe0 0x7f76f815a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76ff400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f80b37f0 con 0x7f76f8157ae0 2024-01-15T18:14:20.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76ff400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f815e8b0 con 0x7f76f806e220 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76ff400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f815b520 con 0x7f76f80b2fe0 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76fd600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f76f806e220 0x7f76f80b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76fd600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f76f806e220 0x7f76f80b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37746/0 (socket says 172.21.15.2:37746) 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.133+0000 7f76fd600640 1 -- 172.21.15.2:0/1107802453 learned_addr learned my addr 172.21.15.2:0/1107802453 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.134+0000 7f76fea00640 1 --2- 172.21.15.2:0/1107802453 >> v2:172.21.15.137:3300/0 conn(0x7f76f80b2fe0 0x7f76f815a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.134+0000 7f76fd600640 1 -- 172.21.15.2:0/1107802453 >> v2:172.21.15.137:3300/0 conn(0x7f76f80b2fe0 msgr2=0x7f76f815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.134+0000 7f76fd600640 1 --2- 172.21.15.2:0/1107802453 >> v2:172.21.15.137:3300/0 conn(0x7f76f80b2fe0 0x7f76f815a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.134+0000 7f76fd600640 1 -- 172.21.15.2:0/1107802453 >> v1:172.21.15.137:6789/0 conn(0x7f76f8157ae0 legacy=0x7f76f806db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:20.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.134+0000 7f76fd600640 1 -- 172.21.15.2:0/1107802453 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f815daf0 con 0x7f76f806e220 2024-01-15T18:14:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.135+0000 7f76fd600640 1 --2- 172.21.15.2:0/1107802453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f806e220 0x7f76f80b2aa0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f76f806e600 tx=0x7f76e002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a76bffeeadebf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.135+0000 7f76fcc00640 1 -- 172.21.15.2:0/1107802453 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0031750 con 0x7f76f806e220 2024-01-15T18:14:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.135+0000 7f76fcc00640 1 -- 172.21.15.2:0/1107802453 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e00318b0 con 0x7f76f806e220 2024-01-15T18:14:20.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.135+0000 7f76fcc00640 1 -- 172.21.15.2:0/1107802453 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e0031c40 con 0x7f76f806e220 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 -- 172.21.15.2:0/1107802453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f806e220 msgr2=0x7f76f80b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 --2- 172.21.15.2:0/1107802453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f806e220 0x7f76f80b2aa0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f76f806e600 tx=0x7f76e002b7f0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 -- 172.21.15.2:0/1107802453 shutdown_connections 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 --2- 172.21.15.2:0/1107802453 >> v2:172.21.15.137:3300/0 conn(0x7f76f80b2fe0 0x7f76f815a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 --2- 172.21.15.2:0/1107802453 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f806e220 0x7f76f80b2aa0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 -- 172.21.15.2:0/1107802453 >> 172.21.15.2:0/1107802453 conn(0x7f76f80b0840 msgr2=0x7f76f80b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 -- 172.21.15.2:0/1107802453 shutdown_connections 2024-01-15T18:14:20.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.136+0000 7f76ff400640 1 -- 172.21.15.2:0/1107802453 wait complete. 2024-01-15T18:14:20.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.138+0000 7f76ff400640 1 Processor -- start 2024-01-15T18:14:20.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.138+0000 7f76ff400640 1 -- start start 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76ff400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76f806e220 0x7f76f80cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76ff400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76f80b2fe0 0x7f76f80ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76ff400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 0x7f76f80ce1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76ff400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8143140 con 0x7f76f8157ae0 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76ff400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80cf540 con 0x7f76f80b2fe0 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76ff400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80ce720 con 0x7f76f806e220 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76fea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 0x7f76f80ce1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76fea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 0x7f76f80ce1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37760/0 (socket says 172.21.15.2:37760) 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.139+0000 7f76fea00640 1 -- 172.21.15.2:0/2444428880 learned_addr learned my addr 172.21.15.2:0/2444428880 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fd600640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76f80b2fe0 0x7f76f80ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fe000640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76f806e220 0x7f76f80cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fea00640 1 -- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76f806e220 msgr2=0x7f76f80cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fea00640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76f806e220 0x7f76f80cdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fea00640 1 -- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76f80b2fe0 msgr2=0x7f76f80ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fea00640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76f80b2fe0 0x7f76f80ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fea00640 1 -- 172.21.15.2:0/2444428880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e00094d0 con 0x7f76f8157ae0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fd600640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76f80b2fe0 0x7f76f80ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76fea00640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 0x7f76f80ce1e0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f76f000cd70 tx=0x7f76f000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0003b90 con 0x7f76f8157ae0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f0003cf0 con 0x7f76f8157ae0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0004240 con 0x7f76f8157ae0 2024-01-15T18:14:20.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.140+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f80ce9c0 con 0x7f76f8157ae0 2024-01-15T18:14:20.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.141+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f80cef80 con 0x7f76f8157ae0 2024-01-15T18:14:20.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.142+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f8074770 con 0x7f76f8157ae0 2024-01-15T18:14:20.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.143+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f76f00043a0 con 0x7f76f8157ae0 2024-01-15T18:14:20.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.143+0000 7f76f6000640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76d006c430 0x7f76d006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.144+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f76f0012070 con 0x7f76f8157ae0 2024-01-15T18:14:20.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.144+0000 7f76fe000640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76d006c430 0x7f76d006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.145+0000 7f76fe000640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76d006c430 0x7f76d006e8f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f76f81f1f40 tx=0x7f76e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:20.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.148+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76f0055a50 con 0x7f76f8157ae0 2024-01-15T18:14:20.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.379+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76f8073b50 con 0x7f76f8157ae0 2024-01-15T18:14:20.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.380+0000 7f76f6000640 1 -- 172.21.15.2:0/2444428880 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f76f8073b50 con 0x7f76f8157ae0 2024-01-15T18:14:20.383 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:20.383 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/3995733153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3995733153},{"type":"v1","addr":"172.21.15.2:6834","nonce":3995733153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3120477923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3120477923},{"type":"v1","addr":"172.21.15.2:6835","nonce":3120477923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1748176015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1748176015},{"type":"v1","addr":"172.21.15.137:6836","nonce":1748176015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:14:16.323114+0000","modified":"2024-01-15T18:14:19.369618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9051},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9051":{"gid":9051,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6837/2854972827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2854972827},{"type":"v1","addr":"172.21.15.137:6837","nonce":2854972827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-15T18:14:20.383 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 74 2024-01-15T18:14:20.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76d006c430 msgr2=0x7f76d006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76d006c430 0x7f76d006e8f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f76f81f1f40 tx=0x7f76e8008040 comp rx=0 tx=0).stop 2024-01-15T18:14:20.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 msgr2=0x7f76f80ce1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 0x7f76f80ce1e0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f76f000cd70 tx=0x7f76f000a660 comp rx=0 tx=0).stop 2024-01-15T18:14:20.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 shutdown_connections 2024-01-15T18:14:20.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f76d006c430 0x7f76d006e8f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76f8157ae0 0x7f76f80ce1e0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76f80b2fe0 0x7f76f80ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 --2- 172.21.15.2:0/2444428880 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76f806e220 0x7f76f80cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 >> 172.21.15.2:0/2444428880 conn(0x7f76f80b0840 msgr2=0x7f76f80ae7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:20.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 shutdown_connections 2024-01-15T18:14:20.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.383+0000 7f76ff400640 1 -- 172.21.15.2:0/2444428880 wait complete. 2024-01-15T18:14:20.405 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:14:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.617+0000 7ff132400640 1 Processor -- start 2024-01-15T18:14:20.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.617+0000 7ff132400640 1 -- start start 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.618+0000 7ff132400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff12c15d340 0x7ff12c0cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.618+0000 7ff132400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c0b3de0 con 0x7ff12c0d3090 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.618+0000 7ff132400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c0b40c0 con 0x7ff12c0d1d90 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.618+0000 7ff132400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff12c0d0e80 con 0x7ff12c15d340 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.618+0000 7ff131000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff12c15d340 0x7ff12c0cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.618+0000 7ff131000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff12c15d340 0x7ff12c0cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37766/0 (socket says 172.21.15.2:37766) 2024-01-15T18:14:20.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.619+0000 7ff131000640 1 -- 172.21.15.2:0/4067611873 learned_addr learned my addr 172.21.15.2:0/4067611873 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:20.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.619+0000 7ff131000640 1 -- 172.21.15.2:0/4067611873 >> v1:172.21.15.137:6789/0 conn(0x7ff12c0d1d90 legacy=0x7ff12c0b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:20.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.619+0000 7ff131000640 1 -- 172.21.15.2:0/4067611873 >> v1:172.21.15.2:6789/0 conn(0x7ff12c0d3090 legacy=0x7ff12c0d1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.619+0000 7ff131000640 1 -- 172.21.15.2:0/4067611873 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0d1160 con 0x7ff12c15d340 2024-01-15T18:14:20.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.620+0000 7ff131000640 1 --2- 172.21.15.2:0/4067611873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c0cdd20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009870 tx=0x7ff11c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1878b58eda83b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:20.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.620+0000 7ff12be00640 1 -- 172.21.15.2:0/4067611873 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c030860 con 0x7ff12c15d340 2024-01-15T18:14:20.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff12be00640 1 -- 172.21.15.2:0/4067611873 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11c0309c0 con 0x7ff12c15d340 2024-01-15T18:14:20.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff12be00640 1 -- 172.21.15.2:0/4067611873 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11c030d50 con 0x7ff12c15d340 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 -- 172.21.15.2:0/4067611873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 msgr2=0x7ff12c0cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 --2- 172.21.15.2:0/4067611873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c0cdd20 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7ff11c009870 tx=0x7ff11c02b9a0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 -- 172.21.15.2:0/4067611873 shutdown_connections 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 --2- 172.21.15.2:0/4067611873 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c0cdd20 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 -- 172.21.15.2:0/4067611873 >> 172.21.15.2:0/4067611873 conn(0x7ff12c08caf0 msgr2=0x7ff12c09c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 -- 172.21.15.2:0/4067611873 shutdown_connections 2024-01-15T18:14:20.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.621+0000 7ff132400640 1 -- 172.21.15.2:0/4067611873 wait complete. 2024-01-15T18:14:20.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.623+0000 7ff132400640 1 Processor -- start 2024-01-15T18:14:20.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.623+0000 7ff132400640 1 -- start start 2024-01-15T18:14:20.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.624+0000 7ff132400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff12c0d1d90 0x7ff12c158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.624+0000 7ff132400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff12c0d3090 0x7ff12c1f3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.624+0000 7ff132400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c1f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.624+0000 7ff132400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c1f3420 con 0x7ff12c15d340 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.624+0000 7ff132400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c1f3d80 con 0x7ff12c0d3090 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.624+0000 7ff132400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12c1f6e70 con 0x7ff12c0d1d90 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c1f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c1f63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37770/0 (socket says 172.21.15.2:37770) 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 -- 172.21.15.2:0/609956373 learned_addr learned my addr 172.21.15.2:0/609956373 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131000640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff12c0d1d90 0x7ff12c158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff123e00640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff12c0d3090 0x7ff12c1f3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 -- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff12c0d1d90 msgr2=0x7ff12c158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff12c0d1d90 0x7ff12c158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 -- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff12c0d3090 msgr2=0x7ff12c1f3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff12c0d3090 0x7ff12c1f3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff131a00640 1 -- 172.21.15.2:0/609956373 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c0094d0 con 0x7ff12c15d340 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.625+0000 7ff123e00640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff12c0d3090 0x7ff12c1f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:20.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.626+0000 7ff131a00640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c1f63b0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7ff12400c780 tx=0x7ff12400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.626+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff124004b60 con 0x7ff12c15d340 2024-01-15T18:14:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.626+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12400ce90 con 0x7ff12c15d340 2024-01-15T18:14:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.626+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12400a850 con 0x7ff12c15d340 2024-01-15T18:14:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.626+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12c1f7150 con 0x7ff12c15d340 2024-01-15T18:14:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.626+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12c1f2d40 con 0x7ff12c15d340 2024-01-15T18:14:20.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.627+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12c1f3560 con 0x7ff12c15d340 2024-01-15T18:14:20.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.628+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7ff12400a9b0 con 0x7ff12c15d340 2024-01-15T18:14:20.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.629+0000 7ff129600640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff10406c480 0x7ff10406e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:20.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.629+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7ff124088c60 con 0x7ff12c15d340 2024-01-15T18:14:20.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.630+0000 7ff131000640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff10406c480 0x7ff10406e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:20.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.631+0000 7ff131000640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff10406c480 0x7ff10406e940 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff11c036000 tx=0x7ff11c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:20.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.633+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1240551d0 con 0x7ff12c15d340 2024-01-15T18:14:20.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.827+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff12c1f7340 con 0x7ff12c15d340 2024-01-15T18:14:20.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.829+0000 7ff129600640 1 -- 172.21.15.2:0/609956373 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v114) v1 ==== 74+0+16688 (secure 0 0 0) 0x7ff12c1f7340 con 0x7ff12c15d340 2024-01-15T18:14:20.833 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:20.833 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":114,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:14:17.847631+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff10406c480 msgr2=0x7ff10406e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff10406c480 0x7ff10406e940 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff11c036000 tx=0x7ff11c03b040 comp rx=0 tx=0).stop 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 msgr2=0x7ff12c1f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c1f63b0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7ff12400c780 tx=0x7ff12400cc40 comp rx=0 tx=0).stop 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 shutdown_connections 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff10406c480 0x7ff10406e940 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff12c15d340 0x7ff12c1f63b0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff12c0d3090 0x7ff12c1f3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 --2- 172.21.15.2:0/609956373 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff12c0d1d90 0x7ff12c158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:20.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 >> 172.21.15.2:0/609956373 conn(0x7ff12c08caf0 msgr2=0x7ff12c06b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:20.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 shutdown_connections 2024-01-15T18:14:20.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:20.832+0000 7ff132400640 1 -- 172.21.15.2:0/609956373 wait complete. 2024-01-15T18:14:20.853 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-15T18:14:20.853 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:14:21.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.045+0000 7f795ec00640 1 Processor -- start 2024-01-15T18:14:21.047 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.045+0000 7f795ec00640 1 -- start start 2024-01-15T18:14:21.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.046+0000 7f795ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 0x7f79580b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.046+0000 7f795ec00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795809a7e0 con 0x7f795815b500 2024-01-15T18:14:21.049 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f795ec00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79580cf9d0 con 0x7f7958099a00 2024-01-15T18:14:21.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f795ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79580cfcb0 con 0x7f79580cf4a0 2024-01-15T18:14:21.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f795ce00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f795815b500 0x7f79580c54f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46550/0 (socket says 172.21.15.2:46550) 2024-01-15T18:14:21.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f795ce00640 1 -- 172.21.15.2:0/1703761827 learned_addr learned my addr 172.21.15.2:0/1703761827 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:21.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f795e200640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 0x7f79580b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587609664 0 0) 0x7f795809a7e0 con 0x7f795815b500 2024-01-15T18:14:21.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.047+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7938003520 con 0x7f795815b500 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2364950985 0 0) 0x7f7938003520 con 0x7f795815b500 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 msgr2=0x7f79580b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 0x7f79580b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 >> v1:172.21.15.137:6789/0 conn(0x7f7958099a00 legacy=0x7f7958099de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79580c4a10 con 0x7f795815b500 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f7940001670 con 0x7f795815b500 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 >> v1:172.21.15.2:6789/0 conn(0x7f795815b500 legacy=0x7f79580c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79380039a0 0x7f7938003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79380047c0 0x7f7938004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 0x7f7938005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580cfcb0 con 0x7f79380039a0 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793800ba90 con 0x7f79380047c0 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f793800bc50 con 0x7f79580cf4a0 2024-01-15T18:14:21.052 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.048+0000 7f795d800640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79380039a0 0x7f7938003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.049+0000 7f795ce00640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79380047c0 0x7f7938004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.049+0000 7f795d800640 1 -- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 msgr2=0x7f7938005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.049+0000 7f795d800640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79580cf4a0 0x7f7938005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.049+0000 7f795d800640 1 -- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79380047c0 msgr2=0x7f7938004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.049+0000 7f795d800640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f79380047c0 0x7f7938004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.053 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.049+0000 7f795d800640 1 -- 172.21.15.2:0/1703761827 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7938003520 con 0x7f79380039a0 2024-01-15T18:14:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.050+0000 7f795d800640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79380039a0 0x7f7938003dc0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f7948001720 tx=0x7f79380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb33a21008ddd5d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.050+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79480052d0 con 0x7f79380039a0 2024-01-15T18:14:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.050+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794802e050 con 0x7f79380039a0 2024-01-15T18:14:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.050+0000 7f7957e00640 1 -- 172.21.15.2:0/1703761827 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7948003830 con 0x7f79380039a0 2024-01-15T18:14:21.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795ec00640 1 -- 172.21.15.2:0/1703761827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79380039a0 msgr2=0x7f7938003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795ec00640 1 --2- 172.21.15.2:0/1703761827 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79380039a0 0x7f7938003dc0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f7948001720 tx=0x7f79380057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795e200640 1 -- 172.21.15.2:0/1703761827 reap_dead start 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795ec00640 1 -- 172.21.15.2:0/1703761827 shutdown_connections 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795ec00640 1 -- 172.21.15.2:0/1703761827 >> 172.21.15.2:0/1703761827 conn(0x7f79580b29b0 msgr2=0x7f79580b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795ec00640 1 -- 172.21.15.2:0/1703761827 shutdown_connections 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.051+0000 7f795ec00640 1 -- 172.21.15.2:0/1703761827 wait complete. 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.052+0000 7f795ec00640 1 Processor -- start 2024-01-15T18:14:21.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.053+0000 7f795ec00640 1 -- start start 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.053+0000 7f795ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79380039a0 0x7f7958159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 0x7f79580caad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f795814b140 0x7f795814b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79580cb010 con 0x7f79580af430 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795814a110 con 0x7f795814b140 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79580af8b0 con 0x7f79380039a0 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 0x7f79580caad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 0x7f79580caad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37788/0 (socket says 172.21.15.2:37788) 2024-01-15T18:14:21.057 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 -- 172.21.15.2:0/2076869007 learned_addr learned my addr 172.21.15.2:0/2076869007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795e200640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f795814b140 0x7f795814b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795d800640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79380039a0 0x7f7958159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 -- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79380039a0 msgr2=0x7f7958159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79380039a0 0x7f7958159070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 -- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f795814b140 msgr2=0x7f795814b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f795814b140 0x7f795814b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.054+0000 7f795ce00640 1 -- 172.21.15.2:0/2076869007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795814a380 con 0x7f79580af430 2024-01-15T18:14:21.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.055+0000 7f795ce00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 0x7f79580caad0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f794000d6c0 tx=0x7f794000db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.055+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79400072d0 con 0x7f79580af430 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.055+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7940007430 con 0x7f79580af430 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.055+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79400047e0 con 0x7f79580af430 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.055+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795814a660 con 0x7f79580af430 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.055+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795814ac20 con 0x7f79580af430 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.056+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7958074770 con 0x7f79580af430 2024-01-15T18:14:21.059 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.057+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f7940004940 con 0x7f79580af430 2024-01-15T18:14:21.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.058+0000 7f7955600640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f793006c460 0x7f793006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.058+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f79400883d0 con 0x7f79580af430 2024-01-15T18:14:21.061 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.059+0000 7f795d800640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f793006c460 0x7f793006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.062 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.060+0000 7f795d800640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f793006c460 0x7f793006e920 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7948033000 tx=0x7f7948009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:21.065 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.062+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7940054940 con 0x7f79580af430 2024-01-15T18:14:21.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.250+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f79580c4fd0 con 0x7f793006c460 2024-01-15T18:14:21.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.275+0000 7f7955600640 1 -- 172.21.15.2:0/2076869007 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285331 (secure 0 0 0) 0x7f7948005340 con 0x7f793006c460 2024-01-15T18:14:21.282 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:21.284 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:14:21.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f793006c460 msgr2=0x7f793006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f793006c460 0x7f793006e920 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7948033000 tx=0x7f7948009030 comp rx=0 tx=0).stop 2024-01-15T18:14:21.285 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 msgr2=0x7f79580caad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 0x7f79580caad0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f794000d6c0 tx=0x7f794000db80 comp rx=0 tx=0).stop 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 shutdown_connections 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f793006c460 0x7f793006e920 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f795814b140 0x7f795814b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f79580af430 0x7f79580caad0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 --2- 172.21.15.2:0/2076869007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f79380039a0 0x7f7958159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.281+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 >> 172.21.15.2:0/2076869007 conn(0x7f79580b29b0 msgr2=0x7f795815d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.282+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 shutdown_connections 2024-01-15T18:14:21.286 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.282+0000 7f795ec00640 1 -- 172.21.15.2:0/2076869007 wait complete. 2024-01-15T18:14:21.304 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2024-01-15T18:14:19.824285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":544432,"kb_used_data":12220,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754430288,"statfs":{"total":773094113280,"available":772536614912,"internally_reserved":0,"allocated":12513280,"data_stored":6122906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":544853553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.993161"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679832+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:14:18.679832+0000","last_peered":"2024-01-15T18:14:18.679832+0000","last_clean":"2024-01-15T18:14:18.679832+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:14:18.679832+0000","last_undegraded":"2024-01-15T18:14:18.679832+0000","last_fullsized":"2024-01-15T18:14:18.679832+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869474+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:14:17.869474+0000","last_peered":"2024-01-15T18:14:17.869474+0000","last_clean":"2024-01-15T18:14:17.869474+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:14:17.869474+0000","last_undegraded":"2024-01-15T18:14:17.869474+0000","last_fullsized":"2024-01-15T18:14:17.869474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677711+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:14:18.677711+0000","last_peered":"2024-01-15T18:14:18.677711+0000","last_clean":"2024-01-15T18:14:18.677711+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:14:18.677711+0000","last_undegraded":"2024-01-15T18:14:18.677711+0000","last_fullsized":"2024-01-15T18:14:18.677711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861012+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:14:17.861012+0000","last_peered":"2024-01-15T18:14:17.861012+0000","last_clean":"2024-01-15T18:14:17.861012+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:14:17.861012+0000","last_undegraded":"2024-01-15T18:14:17.861012+0000","last_fullsized":"2024-01-15T18:14:17.861012+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680661+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:14:18.680661+0000","last_peered":"2024-01-15T18:14:18.680661+0000","last_clean":"2024-01-15T18:14:18.680661+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:14:18.680661+0000","last_undegraded":"2024-01-15T18:14:18.680661+0000","last_fullsized":"2024-01-15T18:14:18.680661+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867714+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:14:17.867714+0000","last_peered":"2024-01-15T18:14:17.867714+0000","last_clean":"2024-01-15T18:14:17.867714+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:14:17.867714+0000","last_undegraded":"2024-01-15T18:14:17.867714+0000","last_fullsized":"2024-01-15T18:14:17.867714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680196+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:14:18.680196+0000","last_peered":"2024-01-15T18:14:18.680196+0000","last_clean":"2024-01-15T18:14:18.680196+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:14:18.680196+0000","last_undegraded":"2024-01-15T18:14:18.680196+0000","last_fullsized":"2024-01-15T18:14:18.680196+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678552+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:14:18.678552+0000","last_peered":"2024-01-15T18:14:18.678552+0000","last_clean":"2024-01-15T18:14:18.678552+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:14:18.678552+0000","last_undegraded":"2024-01-15T18:14:18.678552+0000","last_fullsized":"2024-01-15T18:14:18.678552+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869827+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:14:17.869827+0000","last_peered":"2024-01-15T18:14:17.869827+0000","last_clean":"2024-01-15T18:14:17.869827+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:14:17.869827+0000","last_undegraded":"2024-01-15T18:14:17.869827+0000","last_fullsized":"2024-01-15T18:14:17.869827+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677750+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:14:18.677750+0000","last_peered":"2024-01-15T18:14:18.677750+0000","last_clean":"2024-01-15T18:14:18.677750+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:14:18.677750+0000","last_undegraded":"2024-01-15T18:14:18.677750+0000","last_fullsized":"2024-01-15T18:14:18.677750+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679775+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:14:18.679775+0000","last_peered":"2024-01-15T18:14:18.679775+0000","last_clean":"2024-01-15T18:14:18.679775+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:14:18.679775+0000","last_undegraded":"2024-01-15T18:14:18.679775+0000","last_fullsized":"2024-01-15T18:14:18.679775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676165+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:14:18.676165+0000","last_peered":"2024-01-15T18:14:18.676165+0000","last_clean":"2024-01-15T18:14:18.676165+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:14:18.676165+0000","last_undegraded":"2024-01-15T18:14:18.676165+0000","last_fullsized":"2024-01-15T18:14:18.676165+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870017+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:14:17.870017+0000","last_peered":"2024-01-15T18:14:17.870017+0000","last_clean":"2024-01-15T18:14:17.870017+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:14:17.870017+0000","last_undegraded":"2024-01-15T18:14:17.870017+0000","last_fullsized":"2024-01-15T18:14:17.870017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678195+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:14:18.678195+0000","last_peered":"2024-01-15T18:14:18.678195+0000","last_clean":"2024-01-15T18:14:18.678195+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:14:18.678195+0000","last_undegraded":"2024-01-15T18:14:18.678195+0000","last_fullsized":"2024-01-15T18:14:18.678195+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869183+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:14:17.869183+0000","last_peered":"2024-01-15T18:14:17.869183+0000","last_clean":"2024-01-15T18:14:17.869183+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:14:17.869183+0000","last_undegraded":"2024-01-15T18:14:17.869183+0000","last_fullsized":"2024-01-15T18:14:17.869183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679907+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:14:18.679907+0000","last_peered":"2024-01-15T18:14:18.679907+0000","last_clean":"2024-01-15T18:14:18.679907+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:14:18.679907+0000","last_undegraded":"2024-01-15T18:14:18.679907+0000","last_fullsized":"2024-01-15T18:14:18.679907+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867987+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:14:17.867987+0000","last_peered":"2024-01-15T18:14:17.867987+0000","last_clean":"2024-01-15T18:14:17.867987+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:14:17.867987+0000","last_undegraded":"2024-01-15T18:14:17.867987+0000","last_fullsized":"2024-01-15T18:14:17.867987+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678230+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:14:18.678230+0000","last_peered":"2024-01-15T18:14:18.678230+0000","last_clean":"2024-01-15T18:14:18.678230+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:14:18.678230+0000","last_undegraded":"2024-01-15T18:14:18.678230+0000","last_fullsized":"2024-01-15T18:14:18.678230+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870023+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:14:17.870023+0000","last_peered":"2024-01-15T18:14:17.870023+0000","last_clean":"2024-01-15T18:14:17.870023+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:14:17.870023+0000","last_undegraded":"2024-01-15T18:14:17.870023+0000","last_fullsized":"2024-01-15T18:14:17.870023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678920+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:14:18.678920+0000","last_peered":"2024-01-15T18:14:18.678920+0000","last_clean":"2024-01-15T18:14:18.678920+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:14:18.678920+0000","last_undegraded":"2024-01-15T18:14:18.678920+0000","last_fullsized":"2024-01-15T18:14:18.678920+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680581+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:14:18.680581+0000","last_peered":"2024-01-15T18:14:18.680581+0000","last_clean":"2024-01-15T18:14:18.680581+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:14:18.680581+0000","last_undegraded":"2024-01-15T18:14:18.680581+0000","last_fullsized":"2024-01-15T18:14:18.680581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681772+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:14:18.681772+0000","last_peered":"2024-01-15T18:14:18.681772+0000","last_clean":"2024-01-15T18:14:18.681772+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:14:18.681772+0000","last_undegraded":"2024-01-15T18:14:18.681772+0000","last_fullsized":"2024-01-15T18:14:18.681772+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868586+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:14:17.868586+0000","last_peered":"2024-01-15T18:14:17.868586+0000","last_clean":"2024-01-15T18:14:17.868586+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:14:17.868586+0000","last_undegraded":"2024-01-15T18:14:17.868586+0000","last_fullsized":"2024-01-15T18:14:17.868586+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678719+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:14:18.678719+0000","last_peered":"2024-01-15T18:14:18.678719+0000","last_clean":"2024-01-15T18:14:18.678719+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:14:18.678719+0000","last_undegraded":"2024-01-15T18:14:18.678719+0000","last_fullsized":"2024-01-15T18:14:18.678719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678401+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:14:18.678401+0000","last_peered":"2024-01-15T18:14:18.678401+0000","last_clean":"2024-01-15T18:14:18.678401+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:14:18.678401+0000","last_undegraded":"2024-01-15T18:14:18.678401+0000","last_fullsized":"2024-01-15T18:14:18.678401+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680132+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:14:18.680132+0000","last_peered":"2024-01-15T18:14:18.680132+0000","last_clean":"2024-01-15T18:14:18.680132+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:14:18.680132+0000","last_undegraded":"2024-01-15T18:14:18.680132+0000","last_fullsized":"2024-01-15T18:14:18.680132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869567+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:14:17.869567+0000","last_peered":"2024-01-15T18:14:17.869567+0000","last_clean":"2024-01-15T18:14:17.869567+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:14:17.869567+0000","last_undegraded":"2024-01-15T18:14:17.869567+0000","last_fullsized":"2024-01-15T18:14:17.869567+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679121+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:14:18.679121+0000","last_peered":"2024-01-15T18:14:18.679121+0000","last_clean":"2024-01-15T18:14:18.679121+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:14:18.679121+0000","last_undegraded":"2024-01-15T18:14:18.679121+0000","last_fullsized":"2024-01-15T18:14:18.679121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-15T18:14:14.832896+0000","last_change":"2024-01-15T18:14:14.832896+0000","last_active":"2024-01-15T18:14:14.811812+0000","last_peered":"2024-01-15T18:14:14.811812+0000","last_clean":"2024-01-15T18:14:14.811812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:14.832896+0000","last_undegraded":"2024-01-15T18:14:14.832896+0000","last_fullsized":"2024-01-15T18:14:14.832896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681711+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:14:18.681711+0000","last_peered":"2024-01-15T18:14:18.681711+0000","last_clean":"2024-01-15T18:14:18.681711+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:14:18.681711+0000","last_undegraded":"2024-01-15T18:14:18.681711+0000","last_fullsized":"2024-01-15T18:14:18.681711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861547+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:14:17.861547+0000","last_peered":"2024-01-15T18:14:17.861547+0000","last_clean":"2024-01-15T18:14:17.861547+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:14:17.861547+0000","last_undegraded":"2024-01-15T18:14:17.861547+0000","last_fullsized":"2024-01-15T18:14:17.861547+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861087+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:14:17.861087+0000","last_peered":"2024-01-15T18:14:17.861087+0000","last_clean":"2024-01-15T18:14:17.861087+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:14:17.861087+0000","last_undegraded":"2024-01-15T18:14:17.861087+0000","last_fullsized":"2024-01-15T18:14:17.861087+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678804+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:14:18.678804+0000","last_peered":"2024-01-15T18:14:18.678804+0000","last_clean":"2024-01-15T18:14:18.678804+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:14:18.678804+0000","last_undegraded":"2024-01-15T18:14:18.678804+0000","last_fullsized":"2024-01-15T18:14:18.678804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867738+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:14:17.867738+0000","last_peered":"2024-01-15T18:14:17.867738+0000","last_clean":"2024-01-15T18:14:17.867738+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:14:17.867738+0000","last_undegraded":"2024-01-15T18:14:17.867738+0000","last_fullsized":"2024-01-15T18:14:17.867738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868928+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:14:17.868928+0000","last_peered":"2024-01-15T18:14:17.868928+0000","last_clean":"2024-01-15T18:14:17.868928+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:14:17.868928+0000","last_undegraded":"2024-01-15T18:14:17.868928+0000","last_fullsized":"2024-01-15T18:14:17.868928+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680146+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:14:18.680146+0000","last_peered":"2024-01-15T18:14:18.680146+0000","last_clean":"2024-01-15T18:14:18.680146+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:14:18.680146+0000","last_undegraded":"2024-01-15T18:14:18.680146+0000","last_fullsized":"2024-01-15T18:14:18.680146+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676494+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:14:18.676494+0000","last_peered":"2024-01-15T18:14:18.676494+0000","last_clean":"2024-01-15T18:14:18.676494+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:14:18.676494+0000","last_undegraded":"2024-01-15T18:14:18.676494+0000","last_fullsized":"2024-01-15T18:14:18.676494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867867+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:14:17.867867+0000","last_peered":"2024-01-15T18:14:17.867867+0000","last_clean":"2024-01-15T18:14:17.867867+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:14:17.867867+0000","last_undegraded":"2024-01-15T18:14:17.867867+0000","last_fullsized":"2024-01-15T18:14:17.867867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869074+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:14:17.869074+0000","last_peered":"2024-01-15T18:14:17.869074+0000","last_clean":"2024-01-15T18:14:17.869074+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:14:17.869074+0000","last_undegraded":"2024-01-15T18:14:17.869074+0000","last_fullsized":"2024-01-15T18:14:17.869074+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677064+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:14:18.677064+0000","last_peered":"2024-01-15T18:14:18.677064+0000","last_clean":"2024-01-15T18:14:18.677064+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:14:18.677064+0000","last_undegraded":"2024-01-15T18:14:18.677064+0000","last_fullsized":"2024-01-15T18:14:18.677064+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868864+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:14:17.868864+0000","last_peered":"2024-01-15T18:14:17.868864+0000","last_clean":"2024-01-15T18:14:17.868864+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:14:17.868864+0000","last_undegraded":"2024-01-15T18:14:17.868864+0000","last_fullsized":"2024-01-15T18:14:17.868864+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861295+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:14:17.861295+0000","last_peered":"2024-01-15T18:14:17.861295+0000","last_clean":"2024-01-15T18:14:17.861295+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:14:17.861295+0000","last_undegraded":"2024-01-15T18:14:17.861295+0000","last_fullsized":"2024-01-15T18:14:17.861295+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679149+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:14:18.679149+0000","last_peered":"2024-01-15T18:14:18.679149+0000","last_clean":"2024-01-15T18:14:18.679149+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:14:18.679149+0000","last_undegraded":"2024-01-15T18:14:18.679149+0000","last_fullsized":"2024-01-15T18:14:18.679149+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682044+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:14:18.682044+0000","last_peered":"2024-01-15T18:14:18.682044+0000","last_clean":"2024-01-15T18:14:18.682044+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:14:18.682044+0000","last_undegraded":"2024-01-15T18:14:18.682044+0000","last_fullsized":"2024-01-15T18:14:18.682044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682040+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:14:18.682040+0000","last_peered":"2024-01-15T18:14:18.682040+0000","last_clean":"2024-01-15T18:14:18.682040+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:14:18.682040+0000","last_undegraded":"2024-01-15T18:14:18.682040+0000","last_fullsized":"2024-01-15T18:14:18.682040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678030+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:14:18.678030+0000","last_peered":"2024-01-15T18:14:18.678030+0000","last_clean":"2024-01-15T18:14:18.678030+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:14:18.678030+0000","last_undegraded":"2024-01-15T18:14:18.678030+0000","last_fullsized":"2024-01-15T18:14:18.678030+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868462+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:14:17.868462+0000","last_peered":"2024-01-15T18:14:17.868462+0000","last_clean":"2024-01-15T18:14:17.868462+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:14:17.868462+0000","last_undegraded":"2024-01-15T18:14:17.868462+0000","last_fullsized":"2024-01-15T18:14:17.868462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678050+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:14:18.678050+0000","last_peered":"2024-01-15T18:14:18.678050+0000","last_clean":"2024-01-15T18:14:18.678050+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:14:18.678050+0000","last_undegraded":"2024-01-15T18:14:18.678050+0000","last_fullsized":"2024-01-15T18:14:18.678050+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679975+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:14:18.679975+0000","last_peered":"2024-01-15T18:14:18.679975+0000","last_clean":"2024-01-15T18:14:18.679975+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:14:18.679975+0000","last_undegraded":"2024-01-15T18:14:18.679975+0000","last_fullsized":"2024-01-15T18:14:18.679975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681067+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:14:18.681067+0000","last_peered":"2024-01-15T18:14:18.681067+0000","last_clean":"2024-01-15T18:14:18.681067+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:14:18.681067+0000","last_undegraded":"2024-01-15T18:14:18.681067+0000","last_fullsized":"2024-01-15T18:14:18.681067+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681154+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:14:18.681154+0000","last_peered":"2024-01-15T18:14:18.681154+0000","last_clean":"2024-01-15T18:14:18.681154+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:14:18.681154+0000","last_undegraded":"2024-01-15T18:14:18.681154+0000","last_fullsized":"2024-01-15T18:14:18.681154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678245+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:14:18.678245+0000","last_peered":"2024-01-15T18:14:18.678245+0000","last_clean":"2024-01-15T18:14:18.678245+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:14:18.678245+0000","last_undegraded":"2024-01-15T18:14:18.678245+0000","last_fullsized":"2024-01-15T18:14:18.678245+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678042+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:14:18.678042+0000","last_peered":"2024-01-15T18:14:18.678042+0000","last_clean":"2024-01-15T18:14:18.678042+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:14:18.678042+0000","last_undegraded":"2024-01-15T18:14:18.678042+0000","last_fullsized":"2024-01-15T18:14:18.678042+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860447+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:14:17.860447+0000","last_peered":"2024-01-15T18:14:17.860447+0000","last_clean":"2024-01-15T18:14:17.860447+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:14:17.860447+0000","last_undegraded":"2024-01-15T18:14:17.860447+0000","last_fullsized":"2024-01-15T18:14:17.860447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861344+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:14:17.861344+0000","last_peered":"2024-01-15T18:14:17.861344+0000","last_clean":"2024-01-15T18:14:17.861344+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:14:17.861344+0000","last_undegraded":"2024-01-15T18:14:17.861344+0000","last_fullsized":"2024-01-15T18:14:17.861344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680983+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:14:18.680983+0000","last_peered":"2024-01-15T18:14:18.680983+0000","last_clean":"2024-01-15T18:14:18.680983+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:14:18.680983+0000","last_undegraded":"2024-01-15T18:14:18.680983+0000","last_fullsized":"2024-01-15T18:14:18.680983+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681629+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:14:18.681629+0000","last_peered":"2024-01-15T18:14:18.681629+0000","last_clean":"2024-01-15T18:14:18.681629+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:14:18.681629+0000","last_undegraded":"2024-01-15T18:14:18.681629+0000","last_fullsized":"2024-01-15T18:14:18.681629+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861323+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:14:17.861323+0000","last_peered":"2024-01-15T18:14:17.861323+0000","last_clean":"2024-01-15T18:14:17.861323+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:14:17.861323+0000","last_undegraded":"2024-01-15T18:14:17.861323+0000","last_fullsized":"2024-01-15T18:14:17.861323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678475+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:14:18.678475+0000","last_peered":"2024-01-15T18:14:18.678475+0000","last_clean":"2024-01-15T18:14:18.678475+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:14:18.678475+0000","last_undegraded":"2024-01-15T18:14:18.678475+0000","last_fullsized":"2024-01-15T18:14:18.678475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867557+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:14:17.867557+0000","last_peered":"2024-01-15T18:14:17.867557+0000","last_clean":"2024-01-15T18:14:17.867557+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:14:17.867557+0000","last_undegraded":"2024-01-15T18:14:17.867557+0000","last_fullsized":"2024-01-15T18:14:17.867557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681172+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:14:18.681172+0000","last_peered":"2024-01-15T18:14:18.681172+0000","last_clean":"2024-01-15T18:14:18.681172+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:14:18.681172+0000","last_undegraded":"2024-01-15T18:14:18.681172+0000","last_fullsized":"2024-01-15T18:14:18.681172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676821+0000","last_change":"2024-01-15T18:14:16.886669+0000","last_active":"2024-01-15T18:14:18.676821+0000","last_peered":"2024-01-15T18:14:18.676821+0000","last_clean":"2024-01-15T18:14:18.676821+0000","last_became_active":"2024-01-15T18:14:16.886347+0000","last_became_peered":"2024-01-15T18:14:16.886347+0000","last_unstale":"2024-01-15T18:14:18.676821+0000","last_undegraded":"2024-01-15T18:14:18.676821+0000","last_fullsized":"2024-01-15T18:14:18.676821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:20:59.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860455+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:14:17.860455+0000","last_peered":"2024-01-15T18:14:17.860455+0000","last_clean":"2024-01-15T18:14:17.860455+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:14:17.860455+0000","last_undegraded":"2024-01-15T18:14:17.860455+0000","last_fullsized":"2024-01-15T18:14:17.860455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679825+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:14:18.679825+0000","last_peered":"2024-01-15T18:14:18.679825+0000","last_clean":"2024-01-15T18:14:18.679825+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:14:18.679825+0000","last_undegraded":"2024-01-15T18:14:18.679825+0000","last_fullsized":"2024-01-15T18:14:18.679825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679233+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:14:18.679233+0000","last_peered":"2024-01-15T18:14:18.679233+0000","last_clean":"2024-01-15T18:14:18.679233+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:14:18.679233+0000","last_undegraded":"2024-01-15T18:14:18.679233+0000","last_fullsized":"2024-01-15T18:14:18.679233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677309+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:14:18.677309+0000","last_peered":"2024-01-15T18:14:18.677309+0000","last_clean":"2024-01-15T18:14:18.677309+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:14:18.677309+0000","last_undegraded":"2024-01-15T18:14:18.677309+0000","last_fullsized":"2024-01-15T18:14:18.677309+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679019+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:14:18.679019+0000","last_peered":"2024-01-15T18:14:18.679019+0000","last_clean":"2024-01-15T18:14:18.679019+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:14:18.679019+0000","last_undegraded":"2024-01-15T18:14:18.679019+0000","last_fullsized":"2024-01-15T18:14:18.679019+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680985+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:14:18.680985+0000","last_peered":"2024-01-15T18:14:18.680985+0000","last_clean":"2024-01-15T18:14:18.680985+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:14:18.680985+0000","last_undegraded":"2024-01-15T18:14:18.680985+0000","last_fullsized":"2024-01-15T18:14:18.680985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680560+0000","last_change":"2024-01-15T18:14:16.894298+0000","last_active":"2024-01-15T18:14:18.680560+0000","last_peered":"2024-01-15T18:14:18.680560+0000","last_clean":"2024-01-15T18:14:18.680560+0000","last_became_active":"2024-01-15T18:14:16.889887+0000","last_became_peered":"2024-01-15T18:14:16.889887+0000","last_unstale":"2024-01-15T18:14:18.680560+0000","last_undegraded":"2024-01-15T18:14:18.680560+0000","last_fullsized":"2024-01-15T18:14:18.680560+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:00.776476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678648+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:14:18.678648+0000","last_peered":"2024-01-15T18:14:18.678648+0000","last_clean":"2024-01-15T18:14:18.678648+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:14:18.678648+0000","last_undegraded":"2024-01-15T18:14:18.678648+0000","last_fullsized":"2024-01-15T18:14:18.678648+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678758+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:14:18.678758+0000","last_peered":"2024-01-15T18:14:18.678758+0000","last_clean":"2024-01-15T18:14:18.678758+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:14:18.678758+0000","last_undegraded":"2024-01-15T18:14:18.678758+0000","last_fullsized":"2024-01-15T18:14:18.678758+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678265+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:14:18.678265+0000","last_peered":"2024-01-15T18:14:18.678265+0000","last_clean":"2024-01-15T18:14:18.678265+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:14:18.678265+0000","last_undegraded":"2024-01-15T18:14:18.678265+0000","last_fullsized":"2024-01-15T18:14:18.678265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678549+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:14:18.678549+0000","last_peered":"2024-01-15T18:14:18.678549+0000","last_clean":"2024-01-15T18:14:18.678549+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:14:18.678549+0000","last_undegraded":"2024-01-15T18:14:18.678549+0000","last_fullsized":"2024-01-15T18:14:18.678549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682177+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:14:18.682177+0000","last_peered":"2024-01-15T18:14:18.682177+0000","last_clean":"2024-01-15T18:14:18.682177+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:14:18.682177+0000","last_undegraded":"2024-01-15T18:14:18.682177+0000","last_fullsized":"2024-01-15T18:14:18.682177+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868169+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:14:17.868169+0000","last_peered":"2024-01-15T18:14:17.868169+0000","last_clean":"2024-01-15T18:14:17.868169+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:14:17.868169+0000","last_undegraded":"2024-01-15T18:14:17.868169+0000","last_fullsized":"2024-01-15T18:14:17.868169+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867991+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:14:17.867991+0000","last_peered":"2024-01-15T18:14:17.867991+0000","last_clean":"2024-01-15T18:14:17.867991+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:14:17.867991+0000","last_undegraded":"2024-01-15T18:14:17.867991+0000","last_fullsized":"2024-01-15T18:14:17.867991+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869259+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:14:17.869259+0000","last_peered":"2024-01-15T18:14:17.869259+0000","last_clean":"2024-01-15T18:14:17.869259+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:14:17.869259+0000","last_undegraded":"2024-01-15T18:14:17.869259+0000","last_fullsized":"2024-01-15T18:14:17.869259+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868344+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:14:17.868344+0000","last_peered":"2024-01-15T18:14:17.868344+0000","last_clean":"2024-01-15T18:14:17.868344+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:14:17.868344+0000","last_undegraded":"2024-01-15T18:14:17.868344+0000","last_fullsized":"2024-01-15T18:14:17.868344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860501+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:14:17.860501+0000","last_peered":"2024-01-15T18:14:17.860501+0000","last_clean":"2024-01-15T18:14:17.860501+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:14:17.860501+0000","last_undegraded":"2024-01-15T18:14:17.860501+0000","last_fullsized":"2024-01-15T18:14:17.860501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678418+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:14:18.678418+0000","last_peered":"2024-01-15T18:14:18.678418+0000","last_clean":"2024-01-15T18:14:18.678418+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:14:18.678418+0000","last_undegraded":"2024-01-15T18:14:18.678418+0000","last_fullsized":"2024-01-15T18:14:18.678418+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678982+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:14:18.678982+0000","last_peered":"2024-01-15T18:14:18.678982+0000","last_clean":"2024-01-15T18:14:18.678982+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:14:18.678982+0000","last_undegraded":"2024-01-15T18:14:18.678982+0000","last_fullsized":"2024-01-15T18:14:18.678982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678496+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:14:18.678496+0000","last_peered":"2024-01-15T18:14:18.678496+0000","last_clean":"2024-01-15T18:14:18.678496+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:14:18.678496+0000","last_undegraded":"2024-01-15T18:14:18.678496+0000","last_fullsized":"2024-01-15T18:14:18.678496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867951+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:14:17.867951+0000","last_peered":"2024-01-15T18:14:17.867951+0000","last_clean":"2024-01-15T18:14:17.867951+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:14:17.867951+0000","last_undegraded":"2024-01-15T18:14:17.867951+0000","last_fullsized":"2024-01-15T18:14:17.867951+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":246,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:14:12.276924+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:14:12.276924+0000","last_peered":"2024-01-15T18:14:12.276924+0000","last_clean":"2024-01-15T18:14:12.276924+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:14:12.276924+0000","last_undegraded":"2024-01-15T18:14:12.276924+0000","last_fullsized":"2024-01-15T18:14:12.276924+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542348,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64976,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306864,"statfs":{"total":96636764160,"available":96570228736,"internally_reserved":0,"allocated":1970176,"data_stored":1190315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.268,"15min":1.4239999999999999},"min":{"1min":0.746,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.722,"5min":1.9359999999999999,"15min":14.114000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.5680000000000001},"min":{"1min":1.0700000000000001,"5min":0.78800000000000003,"15min":0.51700000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":27.32},"last":1.0069999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2809999999999999,"15min":1.52},"min":{"1min":0.85799999999999998,"5min":0.68799999999999994,"15min":0.40300000000000002},"max":{"1min":1.5580000000000001,"5min":5.9690000000000003,"15min":29.167999999999999},"last":1.532},{"interface":"front","average":{"1min":1.325,"5min":1.288,"15min":1.4239999999999999},"min":{"1min":0.89200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7749999999999999,"5min":2.1110000000000002,"15min":12.565},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3109999999999999,"15min":1.526},"min":{"1min":0.86199999999999999,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.8360000000000001,"5min":2.7360000000000002,"15min":13.801},"last":1.335},{"interface":"front","average":{"1min":1.387,"5min":1.361,"15min":1.52},"min":{"1min":0.96399999999999997,"5min":0.70099999999999996,"15min":0.56599999999999995},"max":{"1min":2.2160000000000002,"5min":3.5920000000000001,"15min":27.358000000000001},"last":0.89900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.629},"min":{"1min":0.99399999999999999,"5min":0.83099999999999996,"15min":0.47099999999999997},"max":{"1min":1.8109999999999999,"5min":6.133,"15min":29.204999999999998},"last":1.377},{"interface":"front","average":{"1min":1.413,"5min":1.4410000000000001,"15min":1.5900000000000001},"min":{"1min":0.91400000000000003,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.173,"5min":2.617,"15min":16.088999999999999},"last":1.0940000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3360000000000001,"15min":1.544},"min":{"1min":0.91300000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":2.0329999999999999,"5min":2.2599999999999998,"15min":27.219000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.5069999999999999,"15min":1.6859999999999999},"min":{"1min":0.94799999999999995,"5min":0.88100000000000001,"15min":0.438},"max":{"1min":1.6850000000000001,"5min":6.4710000000000001,"15min":29.25},"last":1.6080000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4299999999999999,"15min":1.643},"min":{"1min":0.90800000000000003,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.77,"5min":2.23,"15min":12.518000000000001},"last":1.423},{"interface":"front","average":{"1min":1.524,"5min":1.51,"15min":1.669},"min":{"1min":1.0580000000000001,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":2.3889999999999998,"5min":2.4489999999999998,"15min":27.283000000000001},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.615,"15min":2.3260000000000001},"min":{"1min":1.153,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":53.664999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.609,"15min":1.8460000000000001},"min":{"1min":1.032,"5min":1.032,"15min":0.44},"max":{"1min":1.7549999999999999,"5min":6.1870000000000003,"15min":29.106999999999999},"last":1.671}]}]},{"osd":6,"up_from":14,"seq":60129542349,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64440,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307400,"statfs":{"total":96636764160,"available":96570777600,"internally_reserved":0,"allocated":1417216,"data_stored":619787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.242,"15min":1.24},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.438},"max":{"1min":1.708,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.25,"15min":1.3620000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59599999999999997},"max":{"1min":1.982,"5min":3.1659999999999999,"15min":21.565000000000001},"last":1.4810000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.494,"15min":1.4159999999999999},"min":{"1min":0.86599999999999999,"5min":0.64000000000000001,"15min":0.52100000000000002},"max":{"1min":13.269,"5min":13.269,"15min":13.269},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.681,"5min":1.3360000000000001,"15min":1.429},"min":{"1min":0.93300000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":6.335,"5min":6.335,"15min":13.984999999999999},"last":1.595}]},{"osd":2,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.25,"15min":1.45},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.7110000000000001,"5min":2.919,"15min":34.670000000000002},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.429,"15min":1.446},"min":{"1min":0.90700000000000003,"5min":0.71599999999999997,"15min":0.49399999999999999},"max":{"1min":1.9159999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.482,"15min":1.5820000000000001},"min":{"1min":1.1990000000000001,"5min":0.83599999999999997,"15min":0.68500000000000005},"max":{"1min":2.6640000000000001,"5min":3.5680000000000001,"15min":16.855},"last":1.329},{"interface":"front","average":{"1min":1.448,"5min":1.4239999999999999,"15min":1.6870000000000001},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.48499999999999999},"max":{"1min":1.883,"5min":2.9630000000000001,"15min":34.725000000000001},"last":1.4179999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.458,"15min":1.4650000000000001},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.38400000000000001},"max":{"1min":1.996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.05},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.504,"15min":1.51},"min":{"1min":0.88200000000000001,"5min":0.54900000000000004,"15min":0.501},"max":{"1min":2.8340000000000001,"5min":7.3230000000000004,"15min":10.118},"last":1.724}]},{"osd":5,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.5309999999999999,"15min":1.708},"min":{"1min":0.98899999999999999,"5min":0.67000000000000004,"15min":0.57699999999999996},"max":{"1min":2.0070000000000001,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.55},{"interface":"front","average":{"1min":1.55,"5min":1.6599999999999999,"15min":1.635},"min":{"1min":1.0840000000000001,"5min":0.69399999999999995,"15min":0.47899999999999998},"max":{"1min":2.0880000000000001,"5min":8.9900000000000002,"15min":10.077},"last":1.3720000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.6439999999999999,"15min":1.6379999999999999},"min":{"1min":1.208,"5min":0.72299999999999998,"15min":0.64000000000000001},"max":{"1min":2.7549999999999999,"5min":5.5060000000000002,"15min":10.302},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.556,"15min":1.8129999999999999},"min":{"1min":1.1339999999999999,"5min":0.58399999999999996,"15min":0.52600000000000002},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":34.762999999999998},"last":1.677}]}]},{"osd":5,"up_from":14,"seq":60129542351,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58944,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312896,"statfs":{"total":96636764160,"available":96576405504,"internally_reserved":0,"allocated":1421312,"data_stored":620321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.2450000000000001,"15min":1.403},"min":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":26.077999999999999},"last":1.294},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.4399999999999999,"15min":1.5189999999999999},"min":{"1min":0.875,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":5.3949999999999996,"5min":9.8699999999999992,"15min":14.711},"last":1.474}]},{"osd":1,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.282,"15min":1.4339999999999999},"min":{"1min":0.97699999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9730000000000001,"5min":2.8279999999999998,"15min":26},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.282,"15min":1.3959999999999999},"min":{"1min":0.85599999999999998,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":1.931,"5min":2.8660000000000001,"15min":12.619},"last":1.3400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.3680000000000001,"15min":1.5429999999999999},"min":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":2.8029999999999999,"5min":4.8719999999999999,"15min":26.234999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4650000000000001,"15min":1.6220000000000001},"min":{"1min":0.96199999999999997,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":1.9910000000000001,"5min":10.09,"15min":17.126999999999999},"last":1.218}]},{"osd":3,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2709999999999999,"15min":1.454},"min":{"1min":0.97999999999999998,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.802,"5min":1.841,"15min":12.722},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4279999999999999,"15min":1.6499999999999999},"min":{"1min":1.143,"5min":0.67900000000000005,"15min":0.63400000000000001},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":26.134},"last":1.5669999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3740000000000001,"15min":1.544},"min":{"1min":1.111,"5min":0.442,"15min":0.442},"max":{"1min":2.2869999999999999,"5min":9.9979999999999993,"15min":25.776},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.5509999999999999,"15min":1.768},"min":{"1min":1.214,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":2.1030000000000002,"5min":12.103,"15min":16.414000000000001},"last":1.395}]},{"osd":6,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.337,"15min":1.534},"min":{"1min":1.071,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.994,"5min":1.994,"15min":12.769},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.673,"5min":1.4630000000000001,"15min":1.798},"min":{"1min":1.1779999999999999,"5min":0.55500000000000005,"15min":0.435},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":36.087000000000003},"last":1.5129999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.661,"15min":1.921},"min":{"1min":1.319,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.1930000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.482,"15min":1.766},"min":{"1min":1.1719999999999999,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":3.1139999999999999,"5min":3.1139999999999999,"15min":21.728000000000002},"last":1.2609999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542350,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64432,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307408,"statfs":{"total":96636764160,"available":96570785792,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1379999999999999,"15min":1.3779999999999999},"min":{"1min":0.63500000000000001,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":1.637,"5min":3.4100000000000001,"15min":21.643000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.5349999999999999,"5min":2.1240000000000001,"15min":11.196},"last":1.4319999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.204,"15min":1.425},"min":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.8740000000000001,"5min":2.7589999999999999,"15min":28.663},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.54},"min":{"1min":0.82899999999999996,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":1.8680000000000001,"5min":3.5299999999999998,"15min":21.440000000000001},"last":1.2190000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3,"15min":1.4950000000000001},"min":{"1min":0.78300000000000003,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":2.032,"5min":2.7050000000000001,"15min":16.023},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5609999999999999,"15min":1.607},"min":{"1min":0.93600000000000005,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":1.748,"5min":18.858000000000001,"15min":22.350999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2969999999999999,"15min":1.5149999999999999},"min":{"1min":1.1180000000000001,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":1.9890000000000001,"5min":2.1890000000000001,"15min":15.093},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4239999999999999,"15min":1.6879999999999999},"min":{"1min":1.0309999999999999,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.9910000000000001,"5min":2.585,"15min":19.486000000000001},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.391,"15min":1.776},"min":{"1min":0.70999999999999996,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":1.996,"5min":3.6429999999999998,"15min":54.938000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.361,"15min":1.5249999999999999},"min":{"1min":0.99199999999999999,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":2.2349999999999999,"5min":2.6030000000000002,"15min":18.289999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5069999999999999,"15min":1.649},"min":{"1min":0.94599999999999995,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.375,"5min":2.9820000000000002,"15min":17.733000000000001},"last":1.389},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5680000000000001,"15min":1.877},"min":{"1min":1.0029999999999999,"5min":0.621,"15min":0.499},"max":{"1min":2.3079999999999998,"5min":3.7280000000000002,"15min":28.797000000000001},"last":1.355}]},{"osd":7,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.8260000000000001},"min":{"1min":0.93799999999999994,"5min":0.55800000000000005,"15min":0.45100000000000001},"max":{"1min":1.9710000000000001,"5min":2.5419999999999998,"15min":23.390000000000001},"last":1.496},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.633,"15min":1.8879999999999999},"min":{"1min":1.163,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.3599999999999999,"5min":2.851,"15min":22.751000000000001},"last":1.2569999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542351,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72768,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299072,"statfs":{"total":96636764160,"available":96562249728,"internally_reserved":0,"allocated":1429504,"data_stored":621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98299999999999998,"15min":1.1060000000000001},"min":{"1min":0.442,"5min":0.372,"15min":0.29799999999999999},"max":{"1min":2.0099999999999998,"5min":3.6960000000000002,"15min":36.408000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.024,"15min":1.0649999999999999},"min":{"1min":0.69499999999999995,"5min":0.42499999999999999,"15min":0.377},"max":{"1min":1.732,"5min":1.732,"15min":5.0529999999999999},"last":0.85099999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.109},"min":{"1min":0.53700000000000003,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.72,"5min":1.9590000000000001,"15min":8.1359999999999992},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2350000000000001,"15min":1.214},"min":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":1.8600000000000001,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":1.044}]},{"osd":2,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.306,"15min":1.329},"min":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":14.025},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.347,"15min":1.3180000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.46000000000000002},"max":{"1min":1.827,"5min":2.7130000000000001,"15min":5.0910000000000002},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.399,"15min":1.46},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.59999999999999998},"max":{"1min":1.8879999999999999,"5min":6.7720000000000002,"15min":17.082000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":1.105,"5min":0.92900000000000005,"15min":0.69499999999999995},"max":{"1min":9.8040000000000003,"5min":9.8040000000000003,"15min":15.769},"last":1.22}]},{"osd":5,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.6080000000000001},"min":{"1min":1.046,"5min":0.85599999999999998,"15min":0.65000000000000002},"max":{"1min":2.1389999999999998,"5min":2.7440000000000002,"15min":5.5220000000000002},"last":1.28},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.581,"15min":1.9430000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58299999999999996},"max":{"1min":1.9550000000000001,"5min":4.3390000000000004,"15min":40.384999999999998},"last":1.504}]},{"osd":6,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.571,"15min":1.7130000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.51400000000000001},"max":{"1min":2.7629999999999999,"5min":3.6800000000000002,"15min":32.259999999999998},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.679,"5min":1.6799999999999999,"15min":1.7609999999999999},"min":{"1min":1.1579999999999999,"5min":1.153,"15min":0.747},"max":{"1min":2.2519999999999998,"5min":3.9670000000000001,"15min":16.195},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.7490000000000001,"15min":1.913},"min":{"1min":1.026,"5min":0.95599999999999996,"15min":0.63900000000000001},"max":{"1min":2.0720000000000001,"5min":10.035,"15min":24.146999999999998},"last":1.659},{"interface":"front","average":{"1min":1.492,"5min":1.6100000000000001,"15min":1.774},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.56100000000000005},"max":{"1min":2.0270000000000001,"5min":3.4700000000000002,"15min":17.622},"last":1.7729999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542350,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0309999999999999,"15min":1.0129999999999999},"min":{"1min":0.60599999999999998,"5min":0.373,"15min":0.30099999999999999},"max":{"1min":1.641,"5min":10.256,"15min":10.256},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.0960000000000001,"15min":1.1020000000000001},"min":{"1min":0.77000000000000002,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.835,"5min":2.4020000000000001,"15min":10.319000000000001},"last":1.4239999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0309999999999999,"15min":1.119},"min":{"1min":0.68799999999999994,"5min":0.443,"15min":0.38600000000000001},"max":{"1min":1.6040000000000001,"5min":3.1360000000000001,"15min":10.807},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":0.95199999999999996,"5min":0.47099999999999997,"15min":0.35299999999999998},"max":{"1min":2.1429999999999998,"5min":4.7060000000000004,"15min":7.5119999999999996},"last":1.6619999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2589999999999999,"15min":1.4690000000000001},"min":{"1min":0.82099999999999995,"5min":0.55800000000000005,"15min":0.41399999999999998},"max":{"1min":1.6879999999999999,"5min":4.3970000000000002,"15min":25.891999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.49399999999999999,"15min":0.376},"max":{"1min":1.927,"5min":6.7229999999999999,"15min":14.946},"last":1.337}]},{"osd":4,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4119999999999999,"15min":1.516},"min":{"1min":0.98599999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.837,"5min":4.4649999999999999,"15min":9.0679999999999996},"last":1.544},{"interface":"front","average":{"1min":1.542,"5min":1.5389999999999999,"15min":1.591},"min":{"1min":1.1220000000000001,"5min":0.65300000000000002,"15min":0.54500000000000004},"max":{"1min":2.3540000000000001,"5min":8.4269999999999996,"15min":10.279999999999999},"last":2.145}]},{"osd":5,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.468,"15min":1.6679999999999999},"min":{"1min":1.016,"5min":0.67100000000000004,"15min":0.63500000000000001},"max":{"1min":1.7809999999999999,"5min":4.7759999999999998,"15min":11.071999999999999},"last":1.792},{"interface":"front","average":{"1min":1.486,"5min":1.534,"15min":1.7230000000000001},"min":{"1min":1.012,"5min":0.89700000000000002,"15min":0.55200000000000005},"max":{"1min":2.0609999999999999,"5min":4.5579999999999998,"15min":37.020000000000003},"last":1.7070000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6539999999999999,"15min":1.7170000000000001},"min":{"1min":1.3480000000000001,"5min":0.79100000000000004,"15min":0.64800000000000002},"max":{"1min":2.5550000000000002,"5min":8.5519999999999996,"15min":23.762},"last":1.859},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.625,"15min":1.911},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.73499999999999999},"max":{"1min":2.004,"5min":4.819,"15min":39.926000000000002},"last":1.603}]},{"osd":7,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.712,"5min":1.7410000000000001,"15min":1.958},"min":{"1min":1.1699999999999999,"5min":0.84299999999999997,"15min":0.68999999999999995},"max":{"1min":2.633,"5min":8.6620000000000008,"15min":52.767000000000003},"last":1.968},{"interface":"front","average":{"1min":1.623,"5min":1.6679999999999999,"15min":1.833},"min":{"1min":1.0880000000000001,"5min":0.877,"15min":0.83499999999999996},"max":{"1min":1.9219999999999999,"5min":4.8620000000000001,"15min":13.055999999999999},"last":2.0950000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542350,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1421312,"data_stored":619841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.1399999999999999,"15min":1.111},"min":{"1min":0.70499999999999996,"5min":0.36499999999999999,"15min":0.36199999999999999},"max":{"1min":11.964,"5min":11.964,"15min":19.218},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.901,"5min":1.2509999999999999,"15min":1.248},"min":{"1min":0.64700000000000002,"5min":0.42199999999999999,"15min":0.38700000000000001},"max":{"1min":15.176,"5min":15.176,"15min":19.823},"last":1.149}]},{"osd":2,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.226,"15min":1.2},"min":{"1min":0.68200000000000005,"5min":0.376,"15min":0.31900000000000001},"max":{"1min":19.02,"5min":19.02,"15min":19.02},"last":1.0369999999999999},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.4930000000000001,"15min":1.385},"min":{"1min":0.58899999999999997,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":29.251000000000001,"5min":29.251000000000001,"15min":29.251000000000001},"last":1.3}]},{"osd":3,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":1.5960000000000001,"15min":1.4870000000000001},"min":{"1min":0.65900000000000003,"5min":0.51100000000000001,"15min":0.40899999999999997},"max":{"1min":27.021000000000001,"5min":27.021000000000001,"15min":27.021000000000001},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.7610000000000001,"5min":1.49,"15min":1.4430000000000001},"min":{"1min":0.75700000000000001,"5min":0.43099999999999999,"15min":0.34799999999999998},"max":{"1min":31.218,"5min":31.218,"15min":31.218},"last":1.631}]},{"osd":4,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8170000000000002,"5min":1.8500000000000001,"15min":1.6910000000000001},"min":{"1min":1.0429999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":27.059999999999999,"5min":27.059999999999999,"15min":27.059999999999999},"last":1.486},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.772,"15min":1.764},"min":{"1min":1.0669999999999999,"5min":0.71599999999999997,"15min":0.44800000000000001},"max":{"1min":31.433,"5min":31.433,"15min":31.433},"last":1.2609999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.629,"15min":1.726},"min":{"1min":1.1870000000000001,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":20.012},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.53,"15min":1.6719999999999999},"min":{"1min":0.89900000000000002,"5min":0.71599999999999997,"15min":0.499},"max":{"1min":3.9910000000000001,"5min":3.9910000000000001,"15min":19.998999999999999},"last":1.4419999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":1.893,"15min":1.825},"min":{"1min":1.1899999999999999,"5min":0.85899999999999999,"15min":0.54100000000000004},"max":{"1min":29.530000000000001,"5min":29.530000000000001,"15min":29.530000000000001},"last":1.419},{"interface":"front","average":{"1min":3.3340000000000001,"5min":1.9019999999999999,"15min":1.931},"min":{"1min":1.165,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":31.186,"5min":31.186,"15min":31.186},"last":1.52}]},{"osd":7,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.8109999999999999,"15min":1.9239999999999999},"min":{"1min":1.2809999999999999,"5min":0.81699999999999995,"15min":0.505},"max":{"1min":26.393999999999998,"5min":26.393999999999998,"15min":26.393999999999998},"last":1.214},{"interface":"front","average":{"1min":2.25,"5min":1.728,"15min":1.887},"min":{"1min":1.145,"5min":0.92800000000000005,"15min":0.54500000000000004},"max":{"1min":9.7799999999999994,"5min":9.7799999999999994,"15min":27.582000000000001},"last":1.5900000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542349,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73360,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298480,"statfs":{"total":96636764160,"available":96561643520,"internally_reserved":0,"allocated":2027520,"data_stored":1211809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.177,"15min":1.2410000000000001},"min":{"1min":0.60799999999999998,"5min":0.38100000000000001,"15min":0.30599999999999999},"max":{"1min":12.353,"5min":14.542,"15min":20.437999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.198,"5min":1.0800000000000001,"15min":1.2190000000000001},"min":{"1min":0.628,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":1.704,"5min":1.9430000000000001,"15min":35.137999999999998},"last":1.2010000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.282,"15min":1.373},"min":{"1min":0.75700000000000001,"5min":0.44500000000000001,"15min":0.42799999999999999},"max":{"1min":2.6120000000000001,"5min":14.651,"15min":20.556000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.1990000000000001,"15min":1.387},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":35.183},"last":1.117}]},{"osd":3,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.087,"15min":1.163},"min":{"1min":0.85699999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":8.9949999999999992},"last":0.751},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5189999999999999,"15min":1.651},"min":{"1min":0.82899999999999996,"5min":0.623,"15min":0.45100000000000001},"max":{"1min":2.7269999999999999,"5min":14.598000000000001,"15min":25.372},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.4039999999999999,"15min":1.6850000000000001},"min":{"1min":1.2110000000000001,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":35.223999999999997},"last":1.474},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3160000000000001,"15min":1.4530000000000001},"min":{"1min":1.0009999999999999,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":1.851,"5min":2.1179999999999999,"15min":15.968999999999999},"last":1.597}]},{"osd":5,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.516,"15min":1.645},"min":{"1min":1.0529999999999999,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":12.173999999999999},"last":1.329},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6990000000000001,"15min":1.865},"min":{"1min":1.236,"5min":0.88300000000000001,"15min":0.76000000000000001},"max":{"1min":2.9049999999999998,"5min":14.859,"15min":20.789999999999999},"last":1.6890000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.7130000000000001,"15min":2.0539999999999998},"min":{"1min":1.1950000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7989999999999999,"5min":15.022,"15min":36.390999999999998},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.583,"15min":1.9430000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":6.8710000000000004,"5min":6.8710000000000004,"15min":36.207000000000001},"last":1.282}]},{"osd":7,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.645,"15min":1.96},"min":{"1min":0.91200000000000003,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":6.7969999999999997,"5min":6.7969999999999997,"15min":35.636000000000003},"last":1.637},{"interface":"front","average":{"1min":1.583,"5min":1.528,"15min":1.675},"min":{"1min":1.101,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0840000000000001,"5min":2.25,"15min":11.470000000000001},"last":1.3799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:14:21.309 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:14:21.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.568+0000 7f76b1200640 1 Processor -- start 2024-01-15T18:14:21.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.568+0000 7f76b1200640 1 -- start start 2024-01-15T18:14:21.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.570+0000 7f76b1200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 0x7f76ac0cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.570+0000 7f76b1200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ac0aeab0 con 0x7f76ac0cdfa0 2024-01-15T18:14:21.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.570+0000 7f76b1200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ac070380 con 0x7f76ac13ce30 2024-01-15T18:14:21.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.570+0000 7f76b1200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac0ce410 con 0x7f76ac15e340 2024-01-15T18:14:21.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.570+0000 7f76aa800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f76ac0cdfa0 0x7f76ac070ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46554/0 (socket says 172.21.15.2:46554) 2024-01-15T18:14:21.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.570+0000 7f76aa800640 1 -- 172.21.15.2:0/4133881915 learned_addr learned my addr 172.21.15.2:0/4133881915 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a1200640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 0x7f76ac0cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030181714 0 0) 0x7f76ac0aeab0 con 0x7f76ac0cdfa0 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f768c003520 con 0x7f76ac0cdfa0 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740862405 0 0) 0x7f768c003520 con 0x7f76ac0cdfa0 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 msgr2=0x7f76ac0cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 0x7f76ac0cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 >> v1:172.21.15.137:6789/0 conn(0x7f76ac13ce30 legacy=0x7f76ac15dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac0ce6d0 con 0x7f76ac0cdfa0 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f76a4001670 con 0x7f76ac0cdfa0 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 >> v1:172.21.15.2:6789/0 conn(0x7f76ac0cdfa0 legacy=0x7f76ac070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.571+0000 7f76a9400640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 0x7f768c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a9400640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f768c003f20 0x7f768c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a9400640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f768c004d00 0x7f768c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac0ce410 con 0x7f768c003f20 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c00dc70 con 0x7f768c004d00 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f768c00de30 con 0x7f76ac15e340 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a1200640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f768c003f20 0x7f768c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a9e00640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 0x7f768c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a1200640 1 -- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 msgr2=0x7f768c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a1200640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f76ac15e340 0x7f768c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a1200640 1 -- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f768c004d00 msgr2=0x7f768c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a1200640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f768c004d00 0x7f768c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.572+0000 7f76a1200640 1 -- 172.21.15.2:0/4133881915 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c003520 con 0x7f768c003f20 2024-01-15T18:14:21.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.573+0000 7f76a1200640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f768c003f20 0x7f768c0047c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f7694009ac0 tx=0x7f768c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44e0fac709989af server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:21.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769402c6b0 con 0x7f768c003f20 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769402c810 con 0x7f768c003f20 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76a9400640 1 -- 172.21.15.2:0/4133881915 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769402cba0 con 0x7f768c003f20 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76b1200640 1 -- 172.21.15.2:0/4133881915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f768c003f20 msgr2=0x7f768c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76b1200640 1 --2- 172.21.15.2:0/4133881915 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f768c003f20 0x7f768c0047c0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f7694009ac0 tx=0x7f768c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76aa800640 1 -- 172.21.15.2:0/4133881915 reap_dead start 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76b1200640 1 -- 172.21.15.2:0/4133881915 shutdown_connections 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76b1200640 1 -- 172.21.15.2:0/4133881915 >> 172.21.15.2:0/4133881915 conn(0x7f76ac06da20 msgr2=0x7f76ac06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.574+0000 7f76b1200640 1 -- 172.21.15.2:0/4133881915 shutdown_connections 2024-01-15T18:14:21.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.575+0000 7f76b1200640 1 -- 172.21.15.2:0/4133881915 wait complete. 2024-01-15T18:14:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.576+0000 7f76b1200640 1 Processor -- start 2024-01-15T18:14:21.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.576+0000 7f76b1200640 1 -- start start 2024-01-15T18:14:21.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.577+0000 7f76b1200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f768c003f20 0x7f76ac13d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.577+0000 7f76b1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 0x7f76ac0b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.577+0000 7f76b1200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac13ee10 0x7f76ac13f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.577+0000 7f76b1200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac13f7d0 con 0x7f76ac130a50 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.577+0000 7f76b1200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac13e980 con 0x7f76ac13ee10 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.577+0000 7f76b1200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac13daa0 con 0x7f768c003f20 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 0x7f76ac0b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 0x7f76ac0b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37802/0 (socket says 172.21.15.2:37802) 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 -- 172.21.15.2:0/2073859595 learned_addr learned my addr 172.21.15.2:0/2073859595 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a9e00640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f768c003f20 0x7f76ac13d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76aa800640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac13ee10 0x7f76ac13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 -- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f768c003f20 msgr2=0x7f76ac13d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f768c003f20 0x7f76ac13d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.581 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 -- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac13ee10 msgr2=0x7f76ac13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac13ee10 0x7f76ac13f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76a1200640 1 -- 172.21.15.2:0/2073859595 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac13dd70 con 0x7f76ac130a50 2024-01-15T18:14:21.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.578+0000 7f76aa800640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac13ee10 0x7f76ac13f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.579+0000 7f76a1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 0x7f76ac0b0820 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f769402c250 tx=0x7f7694004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.579+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694003810 con 0x7f76ac130a50 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.579+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7694003970 con 0x7f76ac130a50 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.579+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac13e050 con 0x7f76ac130a50 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.579+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7694003ce0 con 0x7f76ac130a50 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.579+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac13e610 con 0x7f76ac130a50 2024-01-15T18:14:21.583 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.580+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac13eaf0 con 0x7f76ac130a50 2024-01-15T18:14:21.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.582+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f7694047450 con 0x7f76ac130a50 2024-01-15T18:14:21.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.582+0000 7f76a2a00640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f768406c480 0x7f768406e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:21.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.583+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f76940aada0 con 0x7f76ac130a50 2024-01-15T18:14:21.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.583+0000 7f76a9e00640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f768406c480 0x7f768406e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:21.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.584+0000 7f76a9e00640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f768406c480 0x7f768406e940 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f769c007fd0 tx=0x7f769c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:21.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.587+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76940773c0 con 0x7f76ac130a50 2024-01-15T18:14:21.778 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.776+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76ac0705b0 con 0x7f768406c480 2024-01-15T18:14:21.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.801+0000 7f76a2a00640 1 -- 172.21.15.2:0/2073859595 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285331 (secure 0 0 0) 0x7f769c005780 con 0x7f768406c480 2024-01-15T18:14:21.808 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:21.809 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f768406c480 msgr2=0x7f768406e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f768406c480 0x7f768406e940 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f769c007fd0 tx=0x7f769c009040 comp rx=0 tx=0).stop 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 msgr2=0x7f76ac0b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 0x7f76ac0b0820 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f769402c250 tx=0x7f7694004730 comp rx=0 tx=0).stop 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 shutdown_connections 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f768406c480 0x7f768406e940 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f76ac13ee10 0x7f76ac13f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f76ac130a50 0x7f76ac0b0820 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 --2- 172.21.15.2:0/2073859595 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f768c003f20 0x7f76ac13d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 >> 172.21.15.2:0/2073859595 conn(0x7f76ac06da20 msgr2=0x7f76ac15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:21.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 shutdown_connections 2024-01-15T18:14:21.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:21.808+0000 7f76b1200640 1 -- 172.21.15.2:0/2073859595 wait complete. 2024-01-15T18:14:21.832 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2024-01-15T18:14:19.824285+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":154,"ondisk_log_size":154,"up":172,"acting":172,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":544432,"kb_used_data":12220,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754430288,"statfs":{"total":773094113280,"available":772536614912,"internally_reserved":0,"allocated":12513280,"data_stored":6122906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":544853553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.993161"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679832+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:14:18.679832+0000","last_peered":"2024-01-15T18:14:18.679832+0000","last_clean":"2024-01-15T18:14:18.679832+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:14:18.679832+0000","last_undegraded":"2024-01-15T18:14:18.679832+0000","last_fullsized":"2024-01-15T18:14:18.679832+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869474+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:14:17.869474+0000","last_peered":"2024-01-15T18:14:17.869474+0000","last_clean":"2024-01-15T18:14:17.869474+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:14:17.869474+0000","last_undegraded":"2024-01-15T18:14:17.869474+0000","last_fullsized":"2024-01-15T18:14:17.869474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677711+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:14:18.677711+0000","last_peered":"2024-01-15T18:14:18.677711+0000","last_clean":"2024-01-15T18:14:18.677711+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:14:18.677711+0000","last_undegraded":"2024-01-15T18:14:18.677711+0000","last_fullsized":"2024-01-15T18:14:18.677711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861012+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:14:17.861012+0000","last_peered":"2024-01-15T18:14:17.861012+0000","last_clean":"2024-01-15T18:14:17.861012+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:14:17.861012+0000","last_undegraded":"2024-01-15T18:14:17.861012+0000","last_fullsized":"2024-01-15T18:14:17.861012+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680661+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:14:18.680661+0000","last_peered":"2024-01-15T18:14:18.680661+0000","last_clean":"2024-01-15T18:14:18.680661+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:14:18.680661+0000","last_undegraded":"2024-01-15T18:14:18.680661+0000","last_fullsized":"2024-01-15T18:14:18.680661+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867714+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:14:17.867714+0000","last_peered":"2024-01-15T18:14:17.867714+0000","last_clean":"2024-01-15T18:14:17.867714+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:14:17.867714+0000","last_undegraded":"2024-01-15T18:14:17.867714+0000","last_fullsized":"2024-01-15T18:14:17.867714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680196+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:14:18.680196+0000","last_peered":"2024-01-15T18:14:18.680196+0000","last_clean":"2024-01-15T18:14:18.680196+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:14:18.680196+0000","last_undegraded":"2024-01-15T18:14:18.680196+0000","last_fullsized":"2024-01-15T18:14:18.680196+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678552+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:14:18.678552+0000","last_peered":"2024-01-15T18:14:18.678552+0000","last_clean":"2024-01-15T18:14:18.678552+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:14:18.678552+0000","last_undegraded":"2024-01-15T18:14:18.678552+0000","last_fullsized":"2024-01-15T18:14:18.678552+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869827+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:14:17.869827+0000","last_peered":"2024-01-15T18:14:17.869827+0000","last_clean":"2024-01-15T18:14:17.869827+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:14:17.869827+0000","last_undegraded":"2024-01-15T18:14:17.869827+0000","last_fullsized":"2024-01-15T18:14:17.869827+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677750+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:14:18.677750+0000","last_peered":"2024-01-15T18:14:18.677750+0000","last_clean":"2024-01-15T18:14:18.677750+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:14:18.677750+0000","last_undegraded":"2024-01-15T18:14:18.677750+0000","last_fullsized":"2024-01-15T18:14:18.677750+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679775+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:14:18.679775+0000","last_peered":"2024-01-15T18:14:18.679775+0000","last_clean":"2024-01-15T18:14:18.679775+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:14:18.679775+0000","last_undegraded":"2024-01-15T18:14:18.679775+0000","last_fullsized":"2024-01-15T18:14:18.679775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676165+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:14:18.676165+0000","last_peered":"2024-01-15T18:14:18.676165+0000","last_clean":"2024-01-15T18:14:18.676165+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:14:18.676165+0000","last_undegraded":"2024-01-15T18:14:18.676165+0000","last_fullsized":"2024-01-15T18:14:18.676165+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870017+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:14:17.870017+0000","last_peered":"2024-01-15T18:14:17.870017+0000","last_clean":"2024-01-15T18:14:17.870017+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:14:17.870017+0000","last_undegraded":"2024-01-15T18:14:17.870017+0000","last_fullsized":"2024-01-15T18:14:17.870017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678195+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:14:18.678195+0000","last_peered":"2024-01-15T18:14:18.678195+0000","last_clean":"2024-01-15T18:14:18.678195+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:14:18.678195+0000","last_undegraded":"2024-01-15T18:14:18.678195+0000","last_fullsized":"2024-01-15T18:14:18.678195+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869183+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:14:17.869183+0000","last_peered":"2024-01-15T18:14:17.869183+0000","last_clean":"2024-01-15T18:14:17.869183+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:14:17.869183+0000","last_undegraded":"2024-01-15T18:14:17.869183+0000","last_fullsized":"2024-01-15T18:14:17.869183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679907+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:14:18.679907+0000","last_peered":"2024-01-15T18:14:18.679907+0000","last_clean":"2024-01-15T18:14:18.679907+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:14:18.679907+0000","last_undegraded":"2024-01-15T18:14:18.679907+0000","last_fullsized":"2024-01-15T18:14:18.679907+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867987+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:14:17.867987+0000","last_peered":"2024-01-15T18:14:17.867987+0000","last_clean":"2024-01-15T18:14:17.867987+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:14:17.867987+0000","last_undegraded":"2024-01-15T18:14:17.867987+0000","last_fullsized":"2024-01-15T18:14:17.867987+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678230+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:14:18.678230+0000","last_peered":"2024-01-15T18:14:18.678230+0000","last_clean":"2024-01-15T18:14:18.678230+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:14:18.678230+0000","last_undegraded":"2024-01-15T18:14:18.678230+0000","last_fullsized":"2024-01-15T18:14:18.678230+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870023+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:14:17.870023+0000","last_peered":"2024-01-15T18:14:17.870023+0000","last_clean":"2024-01-15T18:14:17.870023+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:14:17.870023+0000","last_undegraded":"2024-01-15T18:14:17.870023+0000","last_fullsized":"2024-01-15T18:14:17.870023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678920+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:14:18.678920+0000","last_peered":"2024-01-15T18:14:18.678920+0000","last_clean":"2024-01-15T18:14:18.678920+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:14:18.678920+0000","last_undegraded":"2024-01-15T18:14:18.678920+0000","last_fullsized":"2024-01-15T18:14:18.678920+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680581+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:14:18.680581+0000","last_peered":"2024-01-15T18:14:18.680581+0000","last_clean":"2024-01-15T18:14:18.680581+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:14:18.680581+0000","last_undegraded":"2024-01-15T18:14:18.680581+0000","last_fullsized":"2024-01-15T18:14:18.680581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681772+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:14:18.681772+0000","last_peered":"2024-01-15T18:14:18.681772+0000","last_clean":"2024-01-15T18:14:18.681772+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:14:18.681772+0000","last_undegraded":"2024-01-15T18:14:18.681772+0000","last_fullsized":"2024-01-15T18:14:18.681772+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868586+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:14:17.868586+0000","last_peered":"2024-01-15T18:14:17.868586+0000","last_clean":"2024-01-15T18:14:17.868586+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:14:17.868586+0000","last_undegraded":"2024-01-15T18:14:17.868586+0000","last_fullsized":"2024-01-15T18:14:17.868586+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678719+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:14:18.678719+0000","last_peered":"2024-01-15T18:14:18.678719+0000","last_clean":"2024-01-15T18:14:18.678719+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:14:18.678719+0000","last_undegraded":"2024-01-15T18:14:18.678719+0000","last_fullsized":"2024-01-15T18:14:18.678719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678401+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:14:18.678401+0000","last_peered":"2024-01-15T18:14:18.678401+0000","last_clean":"2024-01-15T18:14:18.678401+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:14:18.678401+0000","last_undegraded":"2024-01-15T18:14:18.678401+0000","last_fullsized":"2024-01-15T18:14:18.678401+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680132+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:14:18.680132+0000","last_peered":"2024-01-15T18:14:18.680132+0000","last_clean":"2024-01-15T18:14:18.680132+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:14:18.680132+0000","last_undegraded":"2024-01-15T18:14:18.680132+0000","last_fullsized":"2024-01-15T18:14:18.680132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869567+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:14:17.869567+0000","last_peered":"2024-01-15T18:14:17.869567+0000","last_clean":"2024-01-15T18:14:17.869567+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:14:17.869567+0000","last_undegraded":"2024-01-15T18:14:17.869567+0000","last_fullsized":"2024-01-15T18:14:17.869567+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679121+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:14:18.679121+0000","last_peered":"2024-01-15T18:14:18.679121+0000","last_clean":"2024-01-15T18:14:18.679121+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:14:18.679121+0000","last_undegraded":"2024-01-15T18:14:18.679121+0000","last_fullsized":"2024-01-15T18:14:18.679121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":111,"state":"creating+peering","last_fresh":"2024-01-15T18:14:14.832896+0000","last_change":"2024-01-15T18:14:14.832896+0000","last_active":"2024-01-15T18:14:14.811812+0000","last_peered":"2024-01-15T18:14:14.811812+0000","last_clean":"2024-01-15T18:14:14.811812+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:14.832896+0000","last_undegraded":"2024-01-15T18:14:14.832896+0000","last_fullsized":"2024-01-15T18:14:14.832896+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681711+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:14:18.681711+0000","last_peered":"2024-01-15T18:14:18.681711+0000","last_clean":"2024-01-15T18:14:18.681711+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:14:18.681711+0000","last_undegraded":"2024-01-15T18:14:18.681711+0000","last_fullsized":"2024-01-15T18:14:18.681711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861547+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:14:17.861547+0000","last_peered":"2024-01-15T18:14:17.861547+0000","last_clean":"2024-01-15T18:14:17.861547+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:14:17.861547+0000","last_undegraded":"2024-01-15T18:14:17.861547+0000","last_fullsized":"2024-01-15T18:14:17.861547+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861087+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:14:17.861087+0000","last_peered":"2024-01-15T18:14:17.861087+0000","last_clean":"2024-01-15T18:14:17.861087+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:14:17.861087+0000","last_undegraded":"2024-01-15T18:14:17.861087+0000","last_fullsized":"2024-01-15T18:14:17.861087+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678804+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:14:18.678804+0000","last_peered":"2024-01-15T18:14:18.678804+0000","last_clean":"2024-01-15T18:14:18.678804+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:14:18.678804+0000","last_undegraded":"2024-01-15T18:14:18.678804+0000","last_fullsized":"2024-01-15T18:14:18.678804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867738+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:14:17.867738+0000","last_peered":"2024-01-15T18:14:17.867738+0000","last_clean":"2024-01-15T18:14:17.867738+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:14:17.867738+0000","last_undegraded":"2024-01-15T18:14:17.867738+0000","last_fullsized":"2024-01-15T18:14:17.867738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868928+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:14:17.868928+0000","last_peered":"2024-01-15T18:14:17.868928+0000","last_clean":"2024-01-15T18:14:17.868928+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:14:17.868928+0000","last_undegraded":"2024-01-15T18:14:17.868928+0000","last_fullsized":"2024-01-15T18:14:17.868928+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680146+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:14:18.680146+0000","last_peered":"2024-01-15T18:14:18.680146+0000","last_clean":"2024-01-15T18:14:18.680146+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:14:18.680146+0000","last_undegraded":"2024-01-15T18:14:18.680146+0000","last_fullsized":"2024-01-15T18:14:18.680146+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676494+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:14:18.676494+0000","last_peered":"2024-01-15T18:14:18.676494+0000","last_clean":"2024-01-15T18:14:18.676494+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:14:18.676494+0000","last_undegraded":"2024-01-15T18:14:18.676494+0000","last_fullsized":"2024-01-15T18:14:18.676494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867867+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:14:17.867867+0000","last_peered":"2024-01-15T18:14:17.867867+0000","last_clean":"2024-01-15T18:14:17.867867+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:14:17.867867+0000","last_undegraded":"2024-01-15T18:14:17.867867+0000","last_fullsized":"2024-01-15T18:14:17.867867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869074+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:14:17.869074+0000","last_peered":"2024-01-15T18:14:17.869074+0000","last_clean":"2024-01-15T18:14:17.869074+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:14:17.869074+0000","last_undegraded":"2024-01-15T18:14:17.869074+0000","last_fullsized":"2024-01-15T18:14:17.869074+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677064+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:14:18.677064+0000","last_peered":"2024-01-15T18:14:18.677064+0000","last_clean":"2024-01-15T18:14:18.677064+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:14:18.677064+0000","last_undegraded":"2024-01-15T18:14:18.677064+0000","last_fullsized":"2024-01-15T18:14:18.677064+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868864+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:14:17.868864+0000","last_peered":"2024-01-15T18:14:17.868864+0000","last_clean":"2024-01-15T18:14:17.868864+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:14:17.868864+0000","last_undegraded":"2024-01-15T18:14:17.868864+0000","last_fullsized":"2024-01-15T18:14:17.868864+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861295+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:14:17.861295+0000","last_peered":"2024-01-15T18:14:17.861295+0000","last_clean":"2024-01-15T18:14:17.861295+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:14:17.861295+0000","last_undegraded":"2024-01-15T18:14:17.861295+0000","last_fullsized":"2024-01-15T18:14:17.861295+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679149+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:14:18.679149+0000","last_peered":"2024-01-15T18:14:18.679149+0000","last_clean":"2024-01-15T18:14:18.679149+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:14:18.679149+0000","last_undegraded":"2024-01-15T18:14:18.679149+0000","last_fullsized":"2024-01-15T18:14:18.679149+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682044+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:14:18.682044+0000","last_peered":"2024-01-15T18:14:18.682044+0000","last_clean":"2024-01-15T18:14:18.682044+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:14:18.682044+0000","last_undegraded":"2024-01-15T18:14:18.682044+0000","last_fullsized":"2024-01-15T18:14:18.682044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682040+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:14:18.682040+0000","last_peered":"2024-01-15T18:14:18.682040+0000","last_clean":"2024-01-15T18:14:18.682040+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:14:18.682040+0000","last_undegraded":"2024-01-15T18:14:18.682040+0000","last_fullsized":"2024-01-15T18:14:18.682040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678030+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:14:18.678030+0000","last_peered":"2024-01-15T18:14:18.678030+0000","last_clean":"2024-01-15T18:14:18.678030+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:14:18.678030+0000","last_undegraded":"2024-01-15T18:14:18.678030+0000","last_fullsized":"2024-01-15T18:14:18.678030+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868462+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:14:17.868462+0000","last_peered":"2024-01-15T18:14:17.868462+0000","last_clean":"2024-01-15T18:14:17.868462+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:14:17.868462+0000","last_undegraded":"2024-01-15T18:14:17.868462+0000","last_fullsized":"2024-01-15T18:14:17.868462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678050+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:14:18.678050+0000","last_peered":"2024-01-15T18:14:18.678050+0000","last_clean":"2024-01-15T18:14:18.678050+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:14:18.678050+0000","last_undegraded":"2024-01-15T18:14:18.678050+0000","last_fullsized":"2024-01-15T18:14:18.678050+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679975+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:14:18.679975+0000","last_peered":"2024-01-15T18:14:18.679975+0000","last_clean":"2024-01-15T18:14:18.679975+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:14:18.679975+0000","last_undegraded":"2024-01-15T18:14:18.679975+0000","last_fullsized":"2024-01-15T18:14:18.679975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681067+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:14:18.681067+0000","last_peered":"2024-01-15T18:14:18.681067+0000","last_clean":"2024-01-15T18:14:18.681067+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:14:18.681067+0000","last_undegraded":"2024-01-15T18:14:18.681067+0000","last_fullsized":"2024-01-15T18:14:18.681067+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681154+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:14:18.681154+0000","last_peered":"2024-01-15T18:14:18.681154+0000","last_clean":"2024-01-15T18:14:18.681154+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:14:18.681154+0000","last_undegraded":"2024-01-15T18:14:18.681154+0000","last_fullsized":"2024-01-15T18:14:18.681154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678245+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:14:18.678245+0000","last_peered":"2024-01-15T18:14:18.678245+0000","last_clean":"2024-01-15T18:14:18.678245+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:14:18.678245+0000","last_undegraded":"2024-01-15T18:14:18.678245+0000","last_fullsized":"2024-01-15T18:14:18.678245+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678042+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:14:18.678042+0000","last_peered":"2024-01-15T18:14:18.678042+0000","last_clean":"2024-01-15T18:14:18.678042+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:14:18.678042+0000","last_undegraded":"2024-01-15T18:14:18.678042+0000","last_fullsized":"2024-01-15T18:14:18.678042+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860447+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:14:17.860447+0000","last_peered":"2024-01-15T18:14:17.860447+0000","last_clean":"2024-01-15T18:14:17.860447+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:14:17.860447+0000","last_undegraded":"2024-01-15T18:14:17.860447+0000","last_fullsized":"2024-01-15T18:14:17.860447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861344+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:14:17.861344+0000","last_peered":"2024-01-15T18:14:17.861344+0000","last_clean":"2024-01-15T18:14:17.861344+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:14:17.861344+0000","last_undegraded":"2024-01-15T18:14:17.861344+0000","last_fullsized":"2024-01-15T18:14:17.861344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680983+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:14:18.680983+0000","last_peered":"2024-01-15T18:14:18.680983+0000","last_clean":"2024-01-15T18:14:18.680983+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:14:18.680983+0000","last_undegraded":"2024-01-15T18:14:18.680983+0000","last_fullsized":"2024-01-15T18:14:18.680983+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681629+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:14:18.681629+0000","last_peered":"2024-01-15T18:14:18.681629+0000","last_clean":"2024-01-15T18:14:18.681629+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:14:18.681629+0000","last_undegraded":"2024-01-15T18:14:18.681629+0000","last_fullsized":"2024-01-15T18:14:18.681629+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861323+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:14:17.861323+0000","last_peered":"2024-01-15T18:14:17.861323+0000","last_clean":"2024-01-15T18:14:17.861323+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:14:17.861323+0000","last_undegraded":"2024-01-15T18:14:17.861323+0000","last_fullsized":"2024-01-15T18:14:17.861323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678475+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:14:18.678475+0000","last_peered":"2024-01-15T18:14:18.678475+0000","last_clean":"2024-01-15T18:14:18.678475+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:14:18.678475+0000","last_undegraded":"2024-01-15T18:14:18.678475+0000","last_fullsized":"2024-01-15T18:14:18.678475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867557+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:14:17.867557+0000","last_peered":"2024-01-15T18:14:17.867557+0000","last_clean":"2024-01-15T18:14:17.867557+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:14:17.867557+0000","last_undegraded":"2024-01-15T18:14:17.867557+0000","last_fullsized":"2024-01-15T18:14:17.867557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681172+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:14:18.681172+0000","last_peered":"2024-01-15T18:14:18.681172+0000","last_clean":"2024-01-15T18:14:18.681172+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:14:18.681172+0000","last_undegraded":"2024-01-15T18:14:18.681172+0000","last_fullsized":"2024-01-15T18:14:18.681172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676821+0000","last_change":"2024-01-15T18:14:16.886669+0000","last_active":"2024-01-15T18:14:18.676821+0000","last_peered":"2024-01-15T18:14:18.676821+0000","last_clean":"2024-01-15T18:14:18.676821+0000","last_became_active":"2024-01-15T18:14:16.886347+0000","last_became_peered":"2024-01-15T18:14:16.886347+0000","last_unstale":"2024-01-15T18:14:18.676821+0000","last_undegraded":"2024-01-15T18:14:18.676821+0000","last_fullsized":"2024-01-15T18:14:18.676821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:20:59.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860455+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:14:17.860455+0000","last_peered":"2024-01-15T18:14:17.860455+0000","last_clean":"2024-01-15T18:14:17.860455+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:14:17.860455+0000","last_undegraded":"2024-01-15T18:14:17.860455+0000","last_fullsized":"2024-01-15T18:14:17.860455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679825+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:14:18.679825+0000","last_peered":"2024-01-15T18:14:18.679825+0000","last_clean":"2024-01-15T18:14:18.679825+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:14:18.679825+0000","last_undegraded":"2024-01-15T18:14:18.679825+0000","last_fullsized":"2024-01-15T18:14:18.679825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679233+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:14:18.679233+0000","last_peered":"2024-01-15T18:14:18.679233+0000","last_clean":"2024-01-15T18:14:18.679233+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:14:18.679233+0000","last_undegraded":"2024-01-15T18:14:18.679233+0000","last_fullsized":"2024-01-15T18:14:18.679233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677309+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:14:18.677309+0000","last_peered":"2024-01-15T18:14:18.677309+0000","last_clean":"2024-01-15T18:14:18.677309+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:14:18.677309+0000","last_undegraded":"2024-01-15T18:14:18.677309+0000","last_fullsized":"2024-01-15T18:14:18.677309+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679019+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:14:18.679019+0000","last_peered":"2024-01-15T18:14:18.679019+0000","last_clean":"2024-01-15T18:14:18.679019+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:14:18.679019+0000","last_undegraded":"2024-01-15T18:14:18.679019+0000","last_fullsized":"2024-01-15T18:14:18.679019+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680985+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:14:18.680985+0000","last_peered":"2024-01-15T18:14:18.680985+0000","last_clean":"2024-01-15T18:14:18.680985+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:14:18.680985+0000","last_undegraded":"2024-01-15T18:14:18.680985+0000","last_fullsized":"2024-01-15T18:14:18.680985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680560+0000","last_change":"2024-01-15T18:14:16.894298+0000","last_active":"2024-01-15T18:14:18.680560+0000","last_peered":"2024-01-15T18:14:18.680560+0000","last_clean":"2024-01-15T18:14:18.680560+0000","last_became_active":"2024-01-15T18:14:16.889887+0000","last_became_peered":"2024-01-15T18:14:16.889887+0000","last_unstale":"2024-01-15T18:14:18.680560+0000","last_undegraded":"2024-01-15T18:14:18.680560+0000","last_fullsized":"2024-01-15T18:14:18.680560+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:00.776476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678648+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:14:18.678648+0000","last_peered":"2024-01-15T18:14:18.678648+0000","last_clean":"2024-01-15T18:14:18.678648+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:14:18.678648+0000","last_undegraded":"2024-01-15T18:14:18.678648+0000","last_fullsized":"2024-01-15T18:14:18.678648+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678758+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:14:18.678758+0000","last_peered":"2024-01-15T18:14:18.678758+0000","last_clean":"2024-01-15T18:14:18.678758+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:14:18.678758+0000","last_undegraded":"2024-01-15T18:14:18.678758+0000","last_fullsized":"2024-01-15T18:14:18.678758+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678265+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:14:18.678265+0000","last_peered":"2024-01-15T18:14:18.678265+0000","last_clean":"2024-01-15T18:14:18.678265+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:14:18.678265+0000","last_undegraded":"2024-01-15T18:14:18.678265+0000","last_fullsized":"2024-01-15T18:14:18.678265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678549+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:14:18.678549+0000","last_peered":"2024-01-15T18:14:18.678549+0000","last_clean":"2024-01-15T18:14:18.678549+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:14:18.678549+0000","last_undegraded":"2024-01-15T18:14:18.678549+0000","last_fullsized":"2024-01-15T18:14:18.678549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682177+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:14:18.682177+0000","last_peered":"2024-01-15T18:14:18.682177+0000","last_clean":"2024-01-15T18:14:18.682177+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:14:18.682177+0000","last_undegraded":"2024-01-15T18:14:18.682177+0000","last_fullsized":"2024-01-15T18:14:18.682177+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868169+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:14:17.868169+0000","last_peered":"2024-01-15T18:14:17.868169+0000","last_clean":"2024-01-15T18:14:17.868169+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:14:17.868169+0000","last_undegraded":"2024-01-15T18:14:17.868169+0000","last_fullsized":"2024-01-15T18:14:17.868169+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867991+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:14:17.867991+0000","last_peered":"2024-01-15T18:14:17.867991+0000","last_clean":"2024-01-15T18:14:17.867991+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:14:17.867991+0000","last_undegraded":"2024-01-15T18:14:17.867991+0000","last_fullsized":"2024-01-15T18:14:17.867991+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869259+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:14:17.869259+0000","last_peered":"2024-01-15T18:14:17.869259+0000","last_clean":"2024-01-15T18:14:17.869259+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:14:17.869259+0000","last_undegraded":"2024-01-15T18:14:17.869259+0000","last_fullsized":"2024-01-15T18:14:17.869259+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868344+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:14:17.868344+0000","last_peered":"2024-01-15T18:14:17.868344+0000","last_clean":"2024-01-15T18:14:17.868344+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:14:17.868344+0000","last_undegraded":"2024-01-15T18:14:17.868344+0000","last_fullsized":"2024-01-15T18:14:17.868344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-01-15T18:14:15.825359+0000","last_change":"2024-01-15T18:14:15.825359+0000","last_active":"2024-01-15T18:14:15.825359+0000","last_peered":"2024-01-15T18:14:15.825359+0000","last_clean":"2024-01-15T18:14:15.825359+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-15T18:14:15.825359+0000","last_undegraded":"2024-01-15T18:14:15.825359+0000","last_fullsized":"2024-01-15T18:14:15.825359+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860501+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:14:17.860501+0000","last_peered":"2024-01-15T18:14:17.860501+0000","last_clean":"2024-01-15T18:14:17.860501+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:14:17.860501+0000","last_undegraded":"2024-01-15T18:14:17.860501+0000","last_fullsized":"2024-01-15T18:14:17.860501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678418+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:14:18.678418+0000","last_peered":"2024-01-15T18:14:18.678418+0000","last_clean":"2024-01-15T18:14:18.678418+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:14:18.678418+0000","last_undegraded":"2024-01-15T18:14:18.678418+0000","last_fullsized":"2024-01-15T18:14:18.678418+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678982+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:14:18.678982+0000","last_peered":"2024-01-15T18:14:18.678982+0000","last_clean":"2024-01-15T18:14:18.678982+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:14:18.678982+0000","last_undegraded":"2024-01-15T18:14:18.678982+0000","last_fullsized":"2024-01-15T18:14:18.678982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678496+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:14:18.678496+0000","last_peered":"2024-01-15T18:14:18.678496+0000","last_clean":"2024-01-15T18:14:18.678496+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:14:18.678496+0000","last_undegraded":"2024-01-15T18:14:18.678496+0000","last_fullsized":"2024-01-15T18:14:18.678496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867951+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:14:17.867951+0000","last_peered":"2024-01-15T18:14:17.867951+0000","last_clean":"2024-01-15T18:14:17.867951+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:14:17.867951+0000","last_undegraded":"2024-01-15T18:14:17.867951+0000","last_fullsized":"2024-01-15T18:14:17.867951+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":246,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-15T18:14:12.276924+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:14:12.276924+0000","last_peered":"2024-01-15T18:14:12.276924+0000","last_clean":"2024-01-15T18:14:12.276924+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:14:12.276924+0000","last_undegraded":"2024-01-15T18:14:12.276924+0000","last_fullsized":"2024-01-15T18:14:12.276924+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":106,"acting":106,"num_store_stats":7},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":4204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":37,"ondisk_log_size":37,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542348,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64976,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306864,"statfs":{"total":96636764160,"available":96570228736,"internally_reserved":0,"allocated":1970176,"data_stored":1190315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.268,"15min":1.4239999999999999},"min":{"1min":0.746,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.722,"5min":1.9359999999999999,"15min":14.114000000000001},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.5680000000000001},"min":{"1min":1.0700000000000001,"5min":0.78800000000000003,"15min":0.51700000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":27.32},"last":1.0069999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2809999999999999,"15min":1.52},"min":{"1min":0.85799999999999998,"5min":0.68799999999999994,"15min":0.40300000000000002},"max":{"1min":1.5580000000000001,"5min":5.9690000000000003,"15min":29.167999999999999},"last":1.532},{"interface":"front","average":{"1min":1.325,"5min":1.288,"15min":1.4239999999999999},"min":{"1min":0.89200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7749999999999999,"5min":2.1110000000000002,"15min":12.565},"last":0.72399999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3109999999999999,"15min":1.526},"min":{"1min":0.86199999999999999,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.8360000000000001,"5min":2.7360000000000002,"15min":13.801},"last":1.335},{"interface":"front","average":{"1min":1.387,"5min":1.361,"15min":1.52},"min":{"1min":0.96399999999999997,"5min":0.70099999999999996,"15min":0.56599999999999995},"max":{"1min":2.2160000000000002,"5min":3.5920000000000001,"15min":27.358000000000001},"last":0.89900000000000002}]},{"osd":3,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.629},"min":{"1min":0.99399999999999999,"5min":0.83099999999999996,"15min":0.47099999999999997},"max":{"1min":1.8109999999999999,"5min":6.133,"15min":29.204999999999998},"last":1.377},{"interface":"front","average":{"1min":1.413,"5min":1.4410000000000001,"15min":1.5900000000000001},"min":{"1min":0.91400000000000003,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.173,"5min":2.617,"15min":16.088999999999999},"last":1.0940000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3360000000000001,"15min":1.544},"min":{"1min":0.91300000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":2.0329999999999999,"5min":2.2599999999999998,"15min":27.219000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.5069999999999999,"15min":1.6859999999999999},"min":{"1min":0.94799999999999995,"5min":0.88100000000000001,"15min":0.438},"max":{"1min":1.6850000000000001,"5min":6.4710000000000001,"15min":29.25},"last":1.6080000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4299999999999999,"15min":1.643},"min":{"1min":0.90800000000000003,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.77,"5min":2.23,"15min":12.518000000000001},"last":1.423},{"interface":"front","average":{"1min":1.524,"5min":1.51,"15min":1.669},"min":{"1min":1.0580000000000001,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":2.3889999999999998,"5min":2.4489999999999998,"15min":27.283000000000001},"last":1.2969999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.615,"15min":2.3260000000000001},"min":{"1min":1.153,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":53.664999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.609,"15min":1.8460000000000001},"min":{"1min":1.032,"5min":1.032,"15min":0.44},"max":{"1min":1.7549999999999999,"5min":6.1870000000000003,"15min":29.106999999999999},"last":1.671}]}]},{"osd":6,"up_from":14,"seq":60129542349,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64440,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307400,"statfs":{"total":96636764160,"available":96570777600,"internally_reserved":0,"allocated":1417216,"data_stored":619787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.242,"15min":1.24},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.438},"max":{"1min":1.708,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.25,"15min":1.3620000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59599999999999997},"max":{"1min":1.982,"5min":3.1659999999999999,"15min":21.565000000000001},"last":1.4810000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.494,"15min":1.4159999999999999},"min":{"1min":0.86599999999999999,"5min":0.64000000000000001,"15min":0.52100000000000002},"max":{"1min":13.269,"5min":13.269,"15min":13.269},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.681,"5min":1.3360000000000001,"15min":1.429},"min":{"1min":0.93300000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":6.335,"5min":6.335,"15min":13.984999999999999},"last":1.595}]},{"osd":2,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.25,"15min":1.45},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.7110000000000001,"5min":2.919,"15min":34.670000000000002},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.429,"15min":1.446},"min":{"1min":0.90700000000000003,"5min":0.71599999999999997,"15min":0.49399999999999999},"max":{"1min":1.9159999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.482,"15min":1.5820000000000001},"min":{"1min":1.1990000000000001,"5min":0.83599999999999997,"15min":0.68500000000000005},"max":{"1min":2.6640000000000001,"5min":3.5680000000000001,"15min":16.855},"last":1.329},{"interface":"front","average":{"1min":1.448,"5min":1.4239999999999999,"15min":1.6870000000000001},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.48499999999999999},"max":{"1min":1.883,"5min":2.9630000000000001,"15min":34.725000000000001},"last":1.4179999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.458,"15min":1.4650000000000001},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.38400000000000001},"max":{"1min":1.996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.05},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.504,"15min":1.51},"min":{"1min":0.88200000000000001,"5min":0.54900000000000004,"15min":0.501},"max":{"1min":2.8340000000000001,"5min":7.3230000000000004,"15min":10.118},"last":1.724}]},{"osd":5,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.5309999999999999,"15min":1.708},"min":{"1min":0.98899999999999999,"5min":0.67000000000000004,"15min":0.57699999999999996},"max":{"1min":2.0070000000000001,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.55},{"interface":"front","average":{"1min":1.55,"5min":1.6599999999999999,"15min":1.635},"min":{"1min":1.0840000000000001,"5min":0.69399999999999995,"15min":0.47899999999999998},"max":{"1min":2.0880000000000001,"5min":8.9900000000000002,"15min":10.077},"last":1.3720000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.6439999999999999,"15min":1.6379999999999999},"min":{"1min":1.208,"5min":0.72299999999999998,"15min":0.64000000000000001},"max":{"1min":2.7549999999999999,"5min":5.5060000000000002,"15min":10.302},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.556,"15min":1.8129999999999999},"min":{"1min":1.1339999999999999,"5min":0.58399999999999996,"15min":0.52600000000000002},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":34.762999999999998},"last":1.677}]}]},{"osd":5,"up_from":14,"seq":60129542351,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58944,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312896,"statfs":{"total":96636764160,"available":96576405504,"internally_reserved":0,"allocated":1421312,"data_stored":620321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.2450000000000001,"15min":1.403},"min":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":26.077999999999999},"last":1.294},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.4399999999999999,"15min":1.5189999999999999},"min":{"1min":0.875,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":5.3949999999999996,"5min":9.8699999999999992,"15min":14.711},"last":1.474}]},{"osd":1,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.282,"15min":1.4339999999999999},"min":{"1min":0.97699999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9730000000000001,"5min":2.8279999999999998,"15min":26},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.282,"15min":1.3959999999999999},"min":{"1min":0.85599999999999998,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":1.931,"5min":2.8660000000000001,"15min":12.619},"last":1.3400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.3680000000000001,"15min":1.5429999999999999},"min":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":2.8029999999999999,"5min":4.8719999999999999,"15min":26.234999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4650000000000001,"15min":1.6220000000000001},"min":{"1min":0.96199999999999997,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":1.9910000000000001,"5min":10.09,"15min":17.126999999999999},"last":1.218}]},{"osd":3,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2709999999999999,"15min":1.454},"min":{"1min":0.97999999999999998,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.802,"5min":1.841,"15min":12.722},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4279999999999999,"15min":1.6499999999999999},"min":{"1min":1.143,"5min":0.67900000000000005,"15min":0.63400000000000001},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":26.134},"last":1.5669999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3740000000000001,"15min":1.544},"min":{"1min":1.111,"5min":0.442,"15min":0.442},"max":{"1min":2.2869999999999999,"5min":9.9979999999999993,"15min":25.776},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.5509999999999999,"15min":1.768},"min":{"1min":1.214,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":2.1030000000000002,"5min":12.103,"15min":16.414000000000001},"last":1.395}]},{"osd":6,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.337,"15min":1.534},"min":{"1min":1.071,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.994,"5min":1.994,"15min":12.769},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.673,"5min":1.4630000000000001,"15min":1.798},"min":{"1min":1.1779999999999999,"5min":0.55500000000000005,"15min":0.435},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":36.087000000000003},"last":1.5129999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.661,"15min":1.921},"min":{"1min":1.319,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.1930000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.482,"15min":1.766},"min":{"1min":1.1719999999999999,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":3.1139999999999999,"5min":3.1139999999999999,"15min":21.728000000000002},"last":1.2609999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542350,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64432,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307408,"statfs":{"total":96636764160,"available":96570785792,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1379999999999999,"15min":1.3779999999999999},"min":{"1min":0.63500000000000001,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":1.637,"5min":3.4100000000000001,"15min":21.643000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.5349999999999999,"5min":2.1240000000000001,"15min":11.196},"last":1.4319999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.204,"15min":1.425},"min":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.8740000000000001,"5min":2.7589999999999999,"15min":28.663},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.54},"min":{"1min":0.82899999999999996,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":1.8680000000000001,"5min":3.5299999999999998,"15min":21.440000000000001},"last":1.2190000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3,"15min":1.4950000000000001},"min":{"1min":0.78300000000000003,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":2.032,"5min":2.7050000000000001,"15min":16.023},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5609999999999999,"15min":1.607},"min":{"1min":0.93600000000000005,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":1.748,"5min":18.858000000000001,"15min":22.350999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2969999999999999,"15min":1.5149999999999999},"min":{"1min":1.1180000000000001,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":1.9890000000000001,"5min":2.1890000000000001,"15min":15.093},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4239999999999999,"15min":1.6879999999999999},"min":{"1min":1.0309999999999999,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.9910000000000001,"5min":2.585,"15min":19.486000000000001},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.391,"15min":1.776},"min":{"1min":0.70999999999999996,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":1.996,"5min":3.6429999999999998,"15min":54.938000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.361,"15min":1.5249999999999999},"min":{"1min":0.99199999999999999,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":2.2349999999999999,"5min":2.6030000000000002,"15min":18.289999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5069999999999999,"15min":1.649},"min":{"1min":0.94599999999999995,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.375,"5min":2.9820000000000002,"15min":17.733000000000001},"last":1.389},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5680000000000001,"15min":1.877},"min":{"1min":1.0029999999999999,"5min":0.621,"15min":0.499},"max":{"1min":2.3079999999999998,"5min":3.7280000000000002,"15min":28.797000000000001},"last":1.355}]},{"osd":7,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.8260000000000001},"min":{"1min":0.93799999999999994,"5min":0.55800000000000005,"15min":0.45100000000000001},"max":{"1min":1.9710000000000001,"5min":2.5419999999999998,"15min":23.390000000000001},"last":1.496},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.633,"15min":1.8879999999999999},"min":{"1min":1.163,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.3599999999999999,"5min":2.851,"15min":22.751000000000001},"last":1.2569999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542351,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72768,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299072,"statfs":{"total":96636764160,"available":96562249728,"internally_reserved":0,"allocated":1429504,"data_stored":621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98299999999999998,"15min":1.1060000000000001},"min":{"1min":0.442,"5min":0.372,"15min":0.29799999999999999},"max":{"1min":2.0099999999999998,"5min":3.6960000000000002,"15min":36.408000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.024,"15min":1.0649999999999999},"min":{"1min":0.69499999999999995,"5min":0.42499999999999999,"15min":0.377},"max":{"1min":1.732,"5min":1.732,"15min":5.0529999999999999},"last":0.85099999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.109},"min":{"1min":0.53700000000000003,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.72,"5min":1.9590000000000001,"15min":8.1359999999999992},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2350000000000001,"15min":1.214},"min":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":1.8600000000000001,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":1.044}]},{"osd":2,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.306,"15min":1.329},"min":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":14.025},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.347,"15min":1.3180000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.46000000000000002},"max":{"1min":1.827,"5min":2.7130000000000001,"15min":5.0910000000000002},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.399,"15min":1.46},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.59999999999999998},"max":{"1min":1.8879999999999999,"5min":6.7720000000000002,"15min":17.082000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":1.105,"5min":0.92900000000000005,"15min":0.69499999999999995},"max":{"1min":9.8040000000000003,"5min":9.8040000000000003,"15min":15.769},"last":1.22}]},{"osd":5,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.6080000000000001},"min":{"1min":1.046,"5min":0.85599999999999998,"15min":0.65000000000000002},"max":{"1min":2.1389999999999998,"5min":2.7440000000000002,"15min":5.5220000000000002},"last":1.28},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.581,"15min":1.9430000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58299999999999996},"max":{"1min":1.9550000000000001,"5min":4.3390000000000004,"15min":40.384999999999998},"last":1.504}]},{"osd":6,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.571,"15min":1.7130000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.51400000000000001},"max":{"1min":2.7629999999999999,"5min":3.6800000000000002,"15min":32.259999999999998},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.679,"5min":1.6799999999999999,"15min":1.7609999999999999},"min":{"1min":1.1579999999999999,"5min":1.153,"15min":0.747},"max":{"1min":2.2519999999999998,"5min":3.9670000000000001,"15min":16.195},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.7490000000000001,"15min":1.913},"min":{"1min":1.026,"5min":0.95599999999999996,"15min":0.63900000000000001},"max":{"1min":2.0720000000000001,"5min":10.035,"15min":24.146999999999998},"last":1.659},{"interface":"front","average":{"1min":1.492,"5min":1.6100000000000001,"15min":1.774},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.56100000000000005},"max":{"1min":2.0270000000000001,"5min":3.4700000000000002,"15min":17.622},"last":1.7729999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542350,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0309999999999999,"15min":1.0129999999999999},"min":{"1min":0.60599999999999998,"5min":0.373,"15min":0.30099999999999999},"max":{"1min":1.641,"5min":10.256,"15min":10.256},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.0960000000000001,"15min":1.1020000000000001},"min":{"1min":0.77000000000000002,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.835,"5min":2.4020000000000001,"15min":10.319000000000001},"last":1.4239999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0309999999999999,"15min":1.119},"min":{"1min":0.68799999999999994,"5min":0.443,"15min":0.38600000000000001},"max":{"1min":1.6040000000000001,"5min":3.1360000000000001,"15min":10.807},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":0.95199999999999996,"5min":0.47099999999999997,"15min":0.35299999999999998},"max":{"1min":2.1429999999999998,"5min":4.7060000000000004,"15min":7.5119999999999996},"last":1.6619999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2589999999999999,"15min":1.4690000000000001},"min":{"1min":0.82099999999999995,"5min":0.55800000000000005,"15min":0.41399999999999998},"max":{"1min":1.6879999999999999,"5min":4.3970000000000002,"15min":25.891999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.49399999999999999,"15min":0.376},"max":{"1min":1.927,"5min":6.7229999999999999,"15min":14.946},"last":1.337}]},{"osd":4,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4119999999999999,"15min":1.516},"min":{"1min":0.98599999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.837,"5min":4.4649999999999999,"15min":9.0679999999999996},"last":1.544},{"interface":"front","average":{"1min":1.542,"5min":1.5389999999999999,"15min":1.591},"min":{"1min":1.1220000000000001,"5min":0.65300000000000002,"15min":0.54500000000000004},"max":{"1min":2.3540000000000001,"5min":8.4269999999999996,"15min":10.279999999999999},"last":2.145}]},{"osd":5,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.468,"15min":1.6679999999999999},"min":{"1min":1.016,"5min":0.67100000000000004,"15min":0.63500000000000001},"max":{"1min":1.7809999999999999,"5min":4.7759999999999998,"15min":11.071999999999999},"last":1.792},{"interface":"front","average":{"1min":1.486,"5min":1.534,"15min":1.7230000000000001},"min":{"1min":1.012,"5min":0.89700000000000002,"15min":0.55200000000000005},"max":{"1min":2.0609999999999999,"5min":4.5579999999999998,"15min":37.020000000000003},"last":1.7070000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6539999999999999,"15min":1.7170000000000001},"min":{"1min":1.3480000000000001,"5min":0.79100000000000004,"15min":0.64800000000000002},"max":{"1min":2.5550000000000002,"5min":8.5519999999999996,"15min":23.762},"last":1.859},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.625,"15min":1.911},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.73499999999999999},"max":{"1min":2.004,"5min":4.819,"15min":39.926000000000002},"last":1.603}]},{"osd":7,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.712,"5min":1.7410000000000001,"15min":1.958},"min":{"1min":1.1699999999999999,"5min":0.84299999999999997,"15min":0.68999999999999995},"max":{"1min":2.633,"5min":8.6620000000000008,"15min":52.767000000000003},"last":1.968},{"interface":"front","average":{"1min":1.623,"5min":1.6679999999999999,"15min":1.833},"min":{"1min":1.0880000000000001,"5min":0.877,"15min":0.83499999999999996},"max":{"1min":1.9219999999999999,"5min":4.8620000000000001,"15min":13.055999999999999},"last":2.0950000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542350,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1421312,"data_stored":619841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.1399999999999999,"15min":1.111},"min":{"1min":0.70499999999999996,"5min":0.36499999999999999,"15min":0.36199999999999999},"max":{"1min":11.964,"5min":11.964,"15min":19.218},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.901,"5min":1.2509999999999999,"15min":1.248},"min":{"1min":0.64700000000000002,"5min":0.42199999999999999,"15min":0.38700000000000001},"max":{"1min":15.176,"5min":15.176,"15min":19.823},"last":1.149}]},{"osd":2,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.226,"15min":1.2},"min":{"1min":0.68200000000000005,"5min":0.376,"15min":0.31900000000000001},"max":{"1min":19.02,"5min":19.02,"15min":19.02},"last":1.0369999999999999},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.4930000000000001,"15min":1.385},"min":{"1min":0.58899999999999997,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":29.251000000000001,"5min":29.251000000000001,"15min":29.251000000000001},"last":1.3}]},{"osd":3,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":1.5960000000000001,"15min":1.4870000000000001},"min":{"1min":0.65900000000000003,"5min":0.51100000000000001,"15min":0.40899999999999997},"max":{"1min":27.021000000000001,"5min":27.021000000000001,"15min":27.021000000000001},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.7610000000000001,"5min":1.49,"15min":1.4430000000000001},"min":{"1min":0.75700000000000001,"5min":0.43099999999999999,"15min":0.34799999999999998},"max":{"1min":31.218,"5min":31.218,"15min":31.218},"last":1.631}]},{"osd":4,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8170000000000002,"5min":1.8500000000000001,"15min":1.6910000000000001},"min":{"1min":1.0429999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":27.059999999999999,"5min":27.059999999999999,"15min":27.059999999999999},"last":1.486},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.772,"15min":1.764},"min":{"1min":1.0669999999999999,"5min":0.71599999999999997,"15min":0.44800000000000001},"max":{"1min":31.433,"5min":31.433,"15min":31.433},"last":1.2609999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.629,"15min":1.726},"min":{"1min":1.1870000000000001,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":20.012},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.53,"15min":1.6719999999999999},"min":{"1min":0.89900000000000002,"5min":0.71599999999999997,"15min":0.499},"max":{"1min":3.9910000000000001,"5min":3.9910000000000001,"15min":19.998999999999999},"last":1.4419999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":1.893,"15min":1.825},"min":{"1min":1.1899999999999999,"5min":0.85899999999999999,"15min":0.54100000000000004},"max":{"1min":29.530000000000001,"5min":29.530000000000001,"15min":29.530000000000001},"last":1.419},{"interface":"front","average":{"1min":3.3340000000000001,"5min":1.9019999999999999,"15min":1.931},"min":{"1min":1.165,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":31.186,"5min":31.186,"15min":31.186},"last":1.52}]},{"osd":7,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.8109999999999999,"15min":1.9239999999999999},"min":{"1min":1.2809999999999999,"5min":0.81699999999999995,"15min":0.505},"max":{"1min":26.393999999999998,"5min":26.393999999999998,"15min":26.393999999999998},"last":1.214},{"interface":"front","average":{"1min":2.25,"5min":1.728,"15min":1.887},"min":{"1min":1.145,"5min":0.92800000000000005,"15min":0.54500000000000004},"max":{"1min":9.7799999999999994,"5min":9.7799999999999994,"15min":27.582000000000001},"last":1.5900000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542349,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73360,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298480,"statfs":{"total":96636764160,"available":96561643520,"internally_reserved":0,"allocated":2027520,"data_stored":1211809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.177,"15min":1.2410000000000001},"min":{"1min":0.60799999999999998,"5min":0.38100000000000001,"15min":0.30599999999999999},"max":{"1min":12.353,"5min":14.542,"15min":20.437999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.198,"5min":1.0800000000000001,"15min":1.2190000000000001},"min":{"1min":0.628,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":1.704,"5min":1.9430000000000001,"15min":35.137999999999998},"last":1.2010000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.282,"15min":1.373},"min":{"1min":0.75700000000000001,"5min":0.44500000000000001,"15min":0.42799999999999999},"max":{"1min":2.6120000000000001,"5min":14.651,"15min":20.556000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.1990000000000001,"15min":1.387},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":35.183},"last":1.117}]},{"osd":3,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.087,"15min":1.163},"min":{"1min":0.85699999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":8.9949999999999992},"last":0.751},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5189999999999999,"15min":1.651},"min":{"1min":0.82899999999999996,"5min":0.623,"15min":0.45100000000000001},"max":{"1min":2.7269999999999999,"5min":14.598000000000001,"15min":25.372},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.4039999999999999,"15min":1.6850000000000001},"min":{"1min":1.2110000000000001,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":35.223999999999997},"last":1.474},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3160000000000001,"15min":1.4530000000000001},"min":{"1min":1.0009999999999999,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":1.851,"5min":2.1179999999999999,"15min":15.968999999999999},"last":1.597}]},{"osd":5,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.516,"15min":1.645},"min":{"1min":1.0529999999999999,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":12.173999999999999},"last":1.329},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6990000000000001,"15min":1.865},"min":{"1min":1.236,"5min":0.88300000000000001,"15min":0.76000000000000001},"max":{"1min":2.9049999999999998,"5min":14.859,"15min":20.789999999999999},"last":1.6890000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.7130000000000001,"15min":2.0539999999999998},"min":{"1min":1.1950000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7989999999999999,"5min":15.022,"15min":36.390999999999998},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.583,"15min":1.9430000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":6.8710000000000004,"5min":6.8710000000000004,"15min":36.207000000000001},"last":1.282}]},{"osd":7,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.645,"15min":1.96},"min":{"1min":0.91200000000000003,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":6.7969999999999997,"5min":6.7969999999999997,"15min":35.636000000000003},"last":1.637},{"interface":"front","average":{"1min":1.583,"5min":1.528,"15min":1.675},"min":{"1min":1.101,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0840000000000001,"5min":2.25,"15min":11.470000000000001},"last":1.3799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:14:21.836 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:14:22.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.097+0000 7fb6a0a00640 1 Processor -- start 2024-01-15T18:14:22.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.097+0000 7fb6a0a00640 1 -- start start 2024-01-15T18:14:22.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb6a0a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb69c15ec60 0x7fb69c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb6a0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c06cad0 0x7fb69c06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb6a0a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0716d0 con 0x7fb69c141880 2024-01-15T18:14:22.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb6a0a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb69c071f20 con 0x7fb69c15ec60 2024-01-15T18:14:22.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb6a0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c06c6b0 con 0x7fb69c06cad0 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb692800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb69c15ec60 0x7fb69c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.099+0000 7fb69b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb69c141880 0x7fb69c070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46566/0 (socket says 172.21.15.2:46566) 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb69b400640 1 -- 172.21.15.2:0/3355264284 learned_addr learned my addr 172.21.15.2:0/3355264284 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb69be00640 1 --2- 172.21.15.2:0/3355264284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c06cad0 0x7fb69c06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb692800640 1 -- 172.21.15.2:0/3355264284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c06cad0 msgr2=0x7fb69c06cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb692800640 1 --2- 172.21.15.2:0/3355264284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c06cad0 0x7fb69c06cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb692800640 1 -- 172.21.15.2:0/3355264284 >> v1:172.21.15.2:6789/0 conn(0x7fb69c141880 legacy=0x7fb69c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb692800640 1 -- 172.21.15.2:0/3355264284 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c071960 con 0x7fb69c15ec60 2024-01-15T18:14:22.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.100+0000 7fb69aa00640 1 -- 172.21.15.2:0/3355264284 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069745484 0 0) 0x7fb69c0716d0 con 0x7fb69c141880 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.101+0000 7fb692800640 1 --2- 172.21.15.2:0/3355264284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15f0e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb6840097d0 tx=0x7fb68402b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d905740a89bbcb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb69aa00640 1 -- 172.21.15.2:0/3355264284 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684031770 con 0x7fb69c15ec60 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb69aa00640 1 -- 172.21.15.2:0/3355264284 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6840318d0 con 0x7fb69c15ec60 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb69aa00640 1 -- 172.21.15.2:0/3355264284 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684031c60 con 0x7fb69c15ec60 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 -- 172.21.15.2:0/3355264284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 msgr2=0x7fb69c15f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/3355264284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15f0e0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fb6840097d0 tx=0x7fb68402b810 comp rx=0 tx=0).stop 2024-01-15T18:14:22.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 -- 172.21.15.2:0/3355264284 shutdown_connections 2024-01-15T18:14:22.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/3355264284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c06cad0 0x7fb69c06cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/3355264284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15f0e0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 -- 172.21.15.2:0/3355264284 >> 172.21.15.2:0/3355264284 conn(0x7fb69c0b0840 msgr2=0x7fb69c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:22.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 -- 172.21.15.2:0/3355264284 shutdown_connections 2024-01-15T18:14:22.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.102+0000 7fb6a0a00640 1 -- 172.21.15.2:0/3355264284 wait complete. 2024-01-15T18:14:22.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.104+0000 7fb6a0a00640 1 Processor -- start 2024-01-15T18:14:22.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.104+0000 7fb6a0a00640 1 -- start start 2024-01-15T18:14:22.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.105+0000 7fb6a0a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb69c06cad0 0x7fb69c136740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.105+0000 7fb6a0a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c141880 0x7fb69c159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.105+0000 7fb6a0a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.105+0000 7fb6a0a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c1421a0 con 0x7fb69c15ec60 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.105+0000 7fb6a0a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c142480 con 0x7fb69c06cad0 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.105+0000 7fb6a0a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c142760 con 0x7fb69c141880 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37822/0 (socket says 172.21.15.2:37822) 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 -- 172.21.15.2:0/106620261 learned_addr learned my addr 172.21.15.2:0/106620261 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb692800640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c141880 0x7fb69c159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69b400640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb69c06cad0 0x7fb69c136740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 -- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c141880 msgr2=0x7fb69c159380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c141880 0x7fb69c159380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 -- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb69c06cad0 msgr2=0x7fb69c136740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb69c06cad0 0x7fb69c136740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.106+0000 7fb69be00640 1 -- 172.21.15.2:0/106620261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb684009400 con 0x7fb69c15ec60 2024-01-15T18:14:22.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.107+0000 7fb69be00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15bca0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fb694007e40 tx=0x7fb69400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:22.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.107+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694014620 con 0x7fb69c15ec60 2024-01-15T18:14:22.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.107+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69400ee90 con 0x7fb69c15ec60 2024-01-15T18:14:22.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.107+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6940149e0 con 0x7fb69c15ec60 2024-01-15T18:14:22.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.107+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c142a40 con 0x7fb69c15ec60 2024-01-15T18:14:22.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.107+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c1f2ef0 con 0x7fb69c15ec60 2024-01-15T18:14:22.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.108+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c074770 con 0x7fb69c15ec60 2024-01-15T18:14:22.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.109+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fb694014b40 con 0x7fb69c15ec60 2024-01-15T18:14:22.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.110+0000 7fb693e00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb67406c430 0x7fb67406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.110+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fb6940894e0 con 0x7fb69c15ec60 2024-01-15T18:14:22.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.111+0000 7fb69b400640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb67406c430 0x7fb67406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.112+0000 7fb69b400640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb67406c430 0x7fb67406e8f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb68c000f60 tx=0x7fb68c007b70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:22.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.115+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb694055a50 con 0x7fb69c15ec60 2024-01-15T18:14:22.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.304+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb69c1425f0 con 0x7fb67406c430 2024-01-15T18:14:22.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.329+0000 7fb693e00640 1 -- 172.21.15.2:0/106620261 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286970 (secure 0 0 0) 0x7fb69c1425f0 con 0x7fb67406c430 2024-01-15T18:14:22.336 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:22.337 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb67406c430 msgr2=0x7fb67406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb67406c430 0x7fb67406e8f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fb68c000f60 tx=0x7fb68c007b70 comp rx=0 tx=0).stop 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 msgr2=0x7fb69c15bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15bca0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fb694007e40 tx=0x7fb69400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 shutdown_connections 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb67406c430 0x7fb67406e8f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb69c15ec60 0x7fb69c15bca0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb69c141880 0x7fb69c159380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 --2- 172.21.15.2:0/106620261 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb69c06cad0 0x7fb69c136740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 >> 172.21.15.2:0/106620261 conn(0x7fb69c0b0840 msgr2=0x7fb69c0ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:22.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 shutdown_connections 2024-01-15T18:14:22.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.336+0000 7fb6a0a00640 1 -- 172.21.15.2:0/106620261 wait complete. 2024-01-15T18:14:22.360 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-01-15T18:14:21.825120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":544480,"kb_used_data":12264,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754430240,"statfs":{"total":773094113280,"available":772536565760,"internally_reserved":0,"allocated":12558336,"data_stored":6143292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":544853553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002045"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679832+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:14:18.679832+0000","last_peered":"2024-01-15T18:14:18.679832+0000","last_clean":"2024-01-15T18:14:18.679832+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:14:18.679832+0000","last_undegraded":"2024-01-15T18:14:18.679832+0000","last_fullsized":"2024-01-15T18:14:18.679832+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869474+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:14:17.869474+0000","last_peered":"2024-01-15T18:14:17.869474+0000","last_clean":"2024-01-15T18:14:17.869474+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:14:17.869474+0000","last_undegraded":"2024-01-15T18:14:17.869474+0000","last_fullsized":"2024-01-15T18:14:17.869474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678459+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:14:18.678459+0000","last_peered":"2024-01-15T18:14:18.678459+0000","last_clean":"2024-01-15T18:14:18.678459+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:14:18.678459+0000","last_undegraded":"2024-01-15T18:14:18.678459+0000","last_fullsized":"2024-01-15T18:14:18.678459+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677711+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:14:18.677711+0000","last_peered":"2024-01-15T18:14:18.677711+0000","last_clean":"2024-01-15T18:14:18.677711+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:14:18.677711+0000","last_undegraded":"2024-01-15T18:14:18.677711+0000","last_fullsized":"2024-01-15T18:14:18.677711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861012+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:14:17.861012+0000","last_peered":"2024-01-15T18:14:17.861012+0000","last_clean":"2024-01-15T18:14:17.861012+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:14:17.861012+0000","last_undegraded":"2024-01-15T18:14:17.861012+0000","last_fullsized":"2024-01-15T18:14:17.861012+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680661+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:14:18.680661+0000","last_peered":"2024-01-15T18:14:18.680661+0000","last_clean":"2024-01-15T18:14:18.680661+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:14:18.680661+0000","last_undegraded":"2024-01-15T18:14:18.680661+0000","last_fullsized":"2024-01-15T18:14:18.680661+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867714+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:14:17.867714+0000","last_peered":"2024-01-15T18:14:17.867714+0000","last_clean":"2024-01-15T18:14:17.867714+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:14:17.867714+0000","last_undegraded":"2024-01-15T18:14:17.867714+0000","last_fullsized":"2024-01-15T18:14:17.867714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676681+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:14:18.676681+0000","last_peered":"2024-01-15T18:14:18.676681+0000","last_clean":"2024-01-15T18:14:18.676681+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:14:18.676681+0000","last_undegraded":"2024-01-15T18:14:18.676681+0000","last_fullsized":"2024-01-15T18:14:18.676681+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680196+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:14:18.680196+0000","last_peered":"2024-01-15T18:14:18.680196+0000","last_clean":"2024-01-15T18:14:18.680196+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:14:18.680196+0000","last_undegraded":"2024-01-15T18:14:18.680196+0000","last_fullsized":"2024-01-15T18:14:18.680196+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678552+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:14:18.678552+0000","last_peered":"2024-01-15T18:14:18.678552+0000","last_clean":"2024-01-15T18:14:18.678552+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:14:18.678552+0000","last_undegraded":"2024-01-15T18:14:18.678552+0000","last_fullsized":"2024-01-15T18:14:18.678552+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869827+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:14:17.869827+0000","last_peered":"2024-01-15T18:14:17.869827+0000","last_clean":"2024-01-15T18:14:17.869827+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:14:17.869827+0000","last_undegraded":"2024-01-15T18:14:17.869827+0000","last_fullsized":"2024-01-15T18:14:17.869827+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677750+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:14:18.677750+0000","last_peered":"2024-01-15T18:14:18.677750+0000","last_clean":"2024-01-15T18:14:18.677750+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:14:18.677750+0000","last_undegraded":"2024-01-15T18:14:18.677750+0000","last_fullsized":"2024-01-15T18:14:18.677750+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677619+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:14:18.677619+0000","last_peered":"2024-01-15T18:14:18.677619+0000","last_clean":"2024-01-15T18:14:18.677619+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:14:18.677619+0000","last_undegraded":"2024-01-15T18:14:18.677619+0000","last_fullsized":"2024-01-15T18:14:18.677619+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676680+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:14:18.676680+0000","last_peered":"2024-01-15T18:14:18.676680+0000","last_clean":"2024-01-15T18:14:18.676680+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:14:18.676680+0000","last_undegraded":"2024-01-15T18:14:18.676680+0000","last_fullsized":"2024-01-15T18:14:18.676680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679775+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:14:18.679775+0000","last_peered":"2024-01-15T18:14:18.679775+0000","last_clean":"2024-01-15T18:14:18.679775+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:14:18.679775+0000","last_undegraded":"2024-01-15T18:14:18.679775+0000","last_fullsized":"2024-01-15T18:14:18.679775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676165+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:14:18.676165+0000","last_peered":"2024-01-15T18:14:18.676165+0000","last_clean":"2024-01-15T18:14:18.676165+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:14:18.676165+0000","last_undegraded":"2024-01-15T18:14:18.676165+0000","last_fullsized":"2024-01-15T18:14:18.676165+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870017+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:14:17.870017+0000","last_peered":"2024-01-15T18:14:17.870017+0000","last_clean":"2024-01-15T18:14:17.870017+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:14:17.870017+0000","last_undegraded":"2024-01-15T18:14:17.870017+0000","last_fullsized":"2024-01-15T18:14:17.870017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678195+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:14:18.678195+0000","last_peered":"2024-01-15T18:14:18.678195+0000","last_clean":"2024-01-15T18:14:18.678195+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:14:18.678195+0000","last_undegraded":"2024-01-15T18:14:18.678195+0000","last_fullsized":"2024-01-15T18:14:18.678195+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678973+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:14:18.678973+0000","last_peered":"2024-01-15T18:14:18.678973+0000","last_clean":"2024-01-15T18:14:18.678973+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:14:18.678973+0000","last_undegraded":"2024-01-15T18:14:18.678973+0000","last_fullsized":"2024-01-15T18:14:18.678973+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869183+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:14:17.869183+0000","last_peered":"2024-01-15T18:14:17.869183+0000","last_clean":"2024-01-15T18:14:17.869183+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:14:17.869183+0000","last_undegraded":"2024-01-15T18:14:17.869183+0000","last_fullsized":"2024-01-15T18:14:17.869183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679907+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:14:18.679907+0000","last_peered":"2024-01-15T18:14:18.679907+0000","last_clean":"2024-01-15T18:14:18.679907+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:14:18.679907+0000","last_undegraded":"2024-01-15T18:14:18.679907+0000","last_fullsized":"2024-01-15T18:14:18.679907+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867987+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:14:17.867987+0000","last_peered":"2024-01-15T18:14:17.867987+0000","last_clean":"2024-01-15T18:14:17.867987+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:14:17.867987+0000","last_undegraded":"2024-01-15T18:14:17.867987+0000","last_fullsized":"2024-01-15T18:14:17.867987+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678230+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:14:18.678230+0000","last_peered":"2024-01-15T18:14:18.678230+0000","last_clean":"2024-01-15T18:14:18.678230+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:14:18.678230+0000","last_undegraded":"2024-01-15T18:14:18.678230+0000","last_fullsized":"2024-01-15T18:14:18.678230+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870023+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:14:17.870023+0000","last_peered":"2024-01-15T18:14:17.870023+0000","last_clean":"2024-01-15T18:14:17.870023+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:14:17.870023+0000","last_undegraded":"2024-01-15T18:14:17.870023+0000","last_fullsized":"2024-01-15T18:14:17.870023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678920+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:14:18.678920+0000","last_peered":"2024-01-15T18:14:18.678920+0000","last_clean":"2024-01-15T18:14:18.678920+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:14:18.678920+0000","last_undegraded":"2024-01-15T18:14:18.678920+0000","last_fullsized":"2024-01-15T18:14:18.678920+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680581+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:14:18.680581+0000","last_peered":"2024-01-15T18:14:18.680581+0000","last_clean":"2024-01-15T18:14:18.680581+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:14:18.680581+0000","last_undegraded":"2024-01-15T18:14:18.680581+0000","last_fullsized":"2024-01-15T18:14:18.680581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681772+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:14:18.681772+0000","last_peered":"2024-01-15T18:14:18.681772+0000","last_clean":"2024-01-15T18:14:18.681772+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:14:18.681772+0000","last_undegraded":"2024-01-15T18:14:18.681772+0000","last_fullsized":"2024-01-15T18:14:18.681772+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868586+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:14:17.868586+0000","last_peered":"2024-01-15T18:14:17.868586+0000","last_clean":"2024-01-15T18:14:17.868586+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:14:17.868586+0000","last_undegraded":"2024-01-15T18:14:17.868586+0000","last_fullsized":"2024-01-15T18:14:17.868586+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678719+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:14:18.678719+0000","last_peered":"2024-01-15T18:14:18.678719+0000","last_clean":"2024-01-15T18:14:18.678719+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:14:18.678719+0000","last_undegraded":"2024-01-15T18:14:18.678719+0000","last_fullsized":"2024-01-15T18:14:18.678719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678401+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:14:18.678401+0000","last_peered":"2024-01-15T18:14:18.678401+0000","last_clean":"2024-01-15T18:14:18.678401+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:14:18.678401+0000","last_undegraded":"2024-01-15T18:14:18.678401+0000","last_fullsized":"2024-01-15T18:14:18.678401+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680132+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:14:18.680132+0000","last_peered":"2024-01-15T18:14:18.680132+0000","last_clean":"2024-01-15T18:14:18.680132+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:14:18.680132+0000","last_undegraded":"2024-01-15T18:14:18.680132+0000","last_fullsized":"2024-01-15T18:14:18.680132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869567+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:14:17.869567+0000","last_peered":"2024-01-15T18:14:17.869567+0000","last_clean":"2024-01-15T18:14:17.869567+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:14:17.869567+0000","last_undegraded":"2024-01-15T18:14:17.869567+0000","last_fullsized":"2024-01-15T18:14:17.869567+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679121+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:14:18.679121+0000","last_peered":"2024-01-15T18:14:18.679121+0000","last_clean":"2024-01-15T18:14:18.679121+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:14:18.679121+0000","last_undegraded":"2024-01-15T18:14:18.679121+0000","last_fullsized":"2024-01-15T18:14:18.679121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-15T18:14:16.899848+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:14:16.899848+0000","last_peered":"2024-01-15T18:14:16.899848+0000","last_clean":"2024-01-15T18:14:16.899848+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:14:16.899848+0000","last_undegraded":"2024-01-15T18:14:16.899848+0000","last_fullsized":"2024-01-15T18:14:16.899848+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681711+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:14:18.681711+0000","last_peered":"2024-01-15T18:14:18.681711+0000","last_clean":"2024-01-15T18:14:18.681711+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:14:18.681711+0000","last_undegraded":"2024-01-15T18:14:18.681711+0000","last_fullsized":"2024-01-15T18:14:18.681711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861547+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:14:17.861547+0000","last_peered":"2024-01-15T18:14:17.861547+0000","last_clean":"2024-01-15T18:14:17.861547+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:14:17.861547+0000","last_undegraded":"2024-01-15T18:14:17.861547+0000","last_fullsized":"2024-01-15T18:14:17.861547+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678678+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:14:18.678678+0000","last_peered":"2024-01-15T18:14:18.678678+0000","last_clean":"2024-01-15T18:14:18.678678+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:14:18.678678+0000","last_undegraded":"2024-01-15T18:14:18.678678+0000","last_fullsized":"2024-01-15T18:14:18.678678+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861087+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:14:17.861087+0000","last_peered":"2024-01-15T18:14:17.861087+0000","last_clean":"2024-01-15T18:14:17.861087+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:14:17.861087+0000","last_undegraded":"2024-01-15T18:14:17.861087+0000","last_fullsized":"2024-01-15T18:14:17.861087+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678804+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:14:18.678804+0000","last_peered":"2024-01-15T18:14:18.678804+0000","last_clean":"2024-01-15T18:14:18.678804+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:14:18.678804+0000","last_undegraded":"2024-01-15T18:14:18.678804+0000","last_fullsized":"2024-01-15T18:14:18.678804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867738+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:14:17.867738+0000","last_peered":"2024-01-15T18:14:17.867738+0000","last_clean":"2024-01-15T18:14:17.867738+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:14:17.867738+0000","last_undegraded":"2024-01-15T18:14:17.867738+0000","last_fullsized":"2024-01-15T18:14:17.867738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868928+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:14:17.868928+0000","last_peered":"2024-01-15T18:14:17.868928+0000","last_clean":"2024-01-15T18:14:17.868928+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:14:17.868928+0000","last_undegraded":"2024-01-15T18:14:17.868928+0000","last_fullsized":"2024-01-15T18:14:17.868928+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680146+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:14:18.680146+0000","last_peered":"2024-01-15T18:14:18.680146+0000","last_clean":"2024-01-15T18:14:18.680146+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:14:18.680146+0000","last_undegraded":"2024-01-15T18:14:18.680146+0000","last_fullsized":"2024-01-15T18:14:18.680146+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676494+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:14:18.676494+0000","last_peered":"2024-01-15T18:14:18.676494+0000","last_clean":"2024-01-15T18:14:18.676494+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:14:18.676494+0000","last_undegraded":"2024-01-15T18:14:18.676494+0000","last_fullsized":"2024-01-15T18:14:18.676494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867867+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:14:17.867867+0000","last_peered":"2024-01-15T18:14:17.867867+0000","last_clean":"2024-01-15T18:14:17.867867+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:14:17.867867+0000","last_undegraded":"2024-01-15T18:14:17.867867+0000","last_fullsized":"2024-01-15T18:14:17.867867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869074+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:14:17.869074+0000","last_peered":"2024-01-15T18:14:17.869074+0000","last_clean":"2024-01-15T18:14:17.869074+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:14:17.869074+0000","last_undegraded":"2024-01-15T18:14:17.869074+0000","last_fullsized":"2024-01-15T18:14:17.869074+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677064+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:14:18.677064+0000","last_peered":"2024-01-15T18:14:18.677064+0000","last_clean":"2024-01-15T18:14:18.677064+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:14:18.677064+0000","last_undegraded":"2024-01-15T18:14:18.677064+0000","last_fullsized":"2024-01-15T18:14:18.677064+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868864+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:14:17.868864+0000","last_peered":"2024-01-15T18:14:17.868864+0000","last_clean":"2024-01-15T18:14:17.868864+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:14:17.868864+0000","last_undegraded":"2024-01-15T18:14:17.868864+0000","last_fullsized":"2024-01-15T18:14:17.868864+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861295+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:14:17.861295+0000","last_peered":"2024-01-15T18:14:17.861295+0000","last_clean":"2024-01-15T18:14:17.861295+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:14:17.861295+0000","last_undegraded":"2024-01-15T18:14:17.861295+0000","last_fullsized":"2024-01-15T18:14:17.861295+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679149+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:14:18.679149+0000","last_peered":"2024-01-15T18:14:18.679149+0000","last_clean":"2024-01-15T18:14:18.679149+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:14:18.679149+0000","last_undegraded":"2024-01-15T18:14:18.679149+0000","last_fullsized":"2024-01-15T18:14:18.679149+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682044+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:14:18.682044+0000","last_peered":"2024-01-15T18:14:18.682044+0000","last_clean":"2024-01-15T18:14:18.682044+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:14:18.682044+0000","last_undegraded":"2024-01-15T18:14:18.682044+0000","last_fullsized":"2024-01-15T18:14:18.682044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682040+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:14:18.682040+0000","last_peered":"2024-01-15T18:14:18.682040+0000","last_clean":"2024-01-15T18:14:18.682040+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:14:18.682040+0000","last_undegraded":"2024-01-15T18:14:18.682040+0000","last_fullsized":"2024-01-15T18:14:18.682040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678030+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:14:18.678030+0000","last_peered":"2024-01-15T18:14:18.678030+0000","last_clean":"2024-01-15T18:14:18.678030+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:14:18.678030+0000","last_undegraded":"2024-01-15T18:14:18.678030+0000","last_fullsized":"2024-01-15T18:14:18.678030+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868462+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:14:17.868462+0000","last_peered":"2024-01-15T18:14:17.868462+0000","last_clean":"2024-01-15T18:14:17.868462+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:14:17.868462+0000","last_undegraded":"2024-01-15T18:14:17.868462+0000","last_fullsized":"2024-01-15T18:14:17.868462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678050+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:14:18.678050+0000","last_peered":"2024-01-15T18:14:18.678050+0000","last_clean":"2024-01-15T18:14:18.678050+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:14:18.678050+0000","last_undegraded":"2024-01-15T18:14:18.678050+0000","last_fullsized":"2024-01-15T18:14:18.678050+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679975+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:14:18.679975+0000","last_peered":"2024-01-15T18:14:18.679975+0000","last_clean":"2024-01-15T18:14:18.679975+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:14:18.679975+0000","last_undegraded":"2024-01-15T18:14:18.679975+0000","last_fullsized":"2024-01-15T18:14:18.679975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681067+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:14:18.681067+0000","last_peered":"2024-01-15T18:14:18.681067+0000","last_clean":"2024-01-15T18:14:18.681067+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:14:18.681067+0000","last_undegraded":"2024-01-15T18:14:18.681067+0000","last_fullsized":"2024-01-15T18:14:18.681067+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681154+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:14:18.681154+0000","last_peered":"2024-01-15T18:14:18.681154+0000","last_clean":"2024-01-15T18:14:18.681154+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:14:18.681154+0000","last_undegraded":"2024-01-15T18:14:18.681154+0000","last_fullsized":"2024-01-15T18:14:18.681154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678245+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:14:18.678245+0000","last_peered":"2024-01-15T18:14:18.678245+0000","last_clean":"2024-01-15T18:14:18.678245+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:14:18.678245+0000","last_undegraded":"2024-01-15T18:14:18.678245+0000","last_fullsized":"2024-01-15T18:14:18.678245+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678042+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:14:18.678042+0000","last_peered":"2024-01-15T18:14:18.678042+0000","last_clean":"2024-01-15T18:14:18.678042+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:14:18.678042+0000","last_undegraded":"2024-01-15T18:14:18.678042+0000","last_fullsized":"2024-01-15T18:14:18.678042+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860447+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:14:17.860447+0000","last_peered":"2024-01-15T18:14:17.860447+0000","last_clean":"2024-01-15T18:14:17.860447+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:14:17.860447+0000","last_undegraded":"2024-01-15T18:14:17.860447+0000","last_fullsized":"2024-01-15T18:14:17.860447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861344+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:14:17.861344+0000","last_peered":"2024-01-15T18:14:17.861344+0000","last_clean":"2024-01-15T18:14:17.861344+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:14:17.861344+0000","last_undegraded":"2024-01-15T18:14:17.861344+0000","last_fullsized":"2024-01-15T18:14:17.861344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680983+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:14:18.680983+0000","last_peered":"2024-01-15T18:14:18.680983+0000","last_clean":"2024-01-15T18:14:18.680983+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:14:18.680983+0000","last_undegraded":"2024-01-15T18:14:18.680983+0000","last_fullsized":"2024-01-15T18:14:18.680983+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681629+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:14:18.681629+0000","last_peered":"2024-01-15T18:14:18.681629+0000","last_clean":"2024-01-15T18:14:18.681629+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:14:18.681629+0000","last_undegraded":"2024-01-15T18:14:18.681629+0000","last_fullsized":"2024-01-15T18:14:18.681629+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861323+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:14:17.861323+0000","last_peered":"2024-01-15T18:14:17.861323+0000","last_clean":"2024-01-15T18:14:17.861323+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:14:17.861323+0000","last_undegraded":"2024-01-15T18:14:17.861323+0000","last_fullsized":"2024-01-15T18:14:17.861323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677207+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:14:18.677207+0000","last_peered":"2024-01-15T18:14:18.677207+0000","last_clean":"2024-01-15T18:14:18.677207+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:14:18.677207+0000","last_undegraded":"2024-01-15T18:14:18.677207+0000","last_fullsized":"2024-01-15T18:14:18.677207+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678475+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:14:18.678475+0000","last_peered":"2024-01-15T18:14:18.678475+0000","last_clean":"2024-01-15T18:14:18.678475+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:14:18.678475+0000","last_undegraded":"2024-01-15T18:14:18.678475+0000","last_fullsized":"2024-01-15T18:14:18.678475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678856+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:14:18.678856+0000","last_peered":"2024-01-15T18:14:18.678856+0000","last_clean":"2024-01-15T18:14:18.678856+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:14:18.678856+0000","last_undegraded":"2024-01-15T18:14:18.678856+0000","last_fullsized":"2024-01-15T18:14:18.678856+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867557+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:14:17.867557+0000","last_peered":"2024-01-15T18:14:17.867557+0000","last_clean":"2024-01-15T18:14:17.867557+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:14:17.867557+0000","last_undegraded":"2024-01-15T18:14:17.867557+0000","last_fullsized":"2024-01-15T18:14:17.867557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677088+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:14:18.677088+0000","last_peered":"2024-01-15T18:14:18.677088+0000","last_clean":"2024-01-15T18:14:18.677088+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:14:18.677088+0000","last_undegraded":"2024-01-15T18:14:18.677088+0000","last_fullsized":"2024-01-15T18:14:18.677088+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681172+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:14:18.681172+0000","last_peered":"2024-01-15T18:14:18.681172+0000","last_clean":"2024-01-15T18:14:18.681172+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:14:18.681172+0000","last_undegraded":"2024-01-15T18:14:18.681172+0000","last_fullsized":"2024-01-15T18:14:18.681172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676821+0000","last_change":"2024-01-15T18:14:16.886669+0000","last_active":"2024-01-15T18:14:18.676821+0000","last_peered":"2024-01-15T18:14:18.676821+0000","last_clean":"2024-01-15T18:14:18.676821+0000","last_became_active":"2024-01-15T18:14:16.886347+0000","last_became_peered":"2024-01-15T18:14:16.886347+0000","last_unstale":"2024-01-15T18:14:18.676821+0000","last_undegraded":"2024-01-15T18:14:18.676821+0000","last_fullsized":"2024-01-15T18:14:18.676821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:20:59.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860455+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:14:17.860455+0000","last_peered":"2024-01-15T18:14:17.860455+0000","last_clean":"2024-01-15T18:14:17.860455+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:14:17.860455+0000","last_undegraded":"2024-01-15T18:14:17.860455+0000","last_fullsized":"2024-01-15T18:14:17.860455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676782+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:14:18.676782+0000","last_peered":"2024-01-15T18:14:18.676782+0000","last_clean":"2024-01-15T18:14:18.676782+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:14:18.676782+0000","last_undegraded":"2024-01-15T18:14:18.676782+0000","last_fullsized":"2024-01-15T18:14:18.676782+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679825+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:14:18.679825+0000","last_peered":"2024-01-15T18:14:18.679825+0000","last_clean":"2024-01-15T18:14:18.679825+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:14:18.679825+0000","last_undegraded":"2024-01-15T18:14:18.679825+0000","last_fullsized":"2024-01-15T18:14:18.679825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679233+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:14:18.679233+0000","last_peered":"2024-01-15T18:14:18.679233+0000","last_clean":"2024-01-15T18:14:18.679233+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:14:18.679233+0000","last_undegraded":"2024-01-15T18:14:18.679233+0000","last_fullsized":"2024-01-15T18:14:18.679233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677309+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:14:18.677309+0000","last_peered":"2024-01-15T18:14:18.677309+0000","last_clean":"2024-01-15T18:14:18.677309+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:14:18.677309+0000","last_undegraded":"2024-01-15T18:14:18.677309+0000","last_fullsized":"2024-01-15T18:14:18.677309+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679019+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:14:18.679019+0000","last_peered":"2024-01-15T18:14:18.679019+0000","last_clean":"2024-01-15T18:14:18.679019+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:14:18.679019+0000","last_undegraded":"2024-01-15T18:14:18.679019+0000","last_fullsized":"2024-01-15T18:14:18.679019+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680985+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:14:18.680985+0000","last_peered":"2024-01-15T18:14:18.680985+0000","last_clean":"2024-01-15T18:14:18.680985+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:14:18.680985+0000","last_undegraded":"2024-01-15T18:14:18.680985+0000","last_fullsized":"2024-01-15T18:14:18.680985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680560+0000","last_change":"2024-01-15T18:14:16.894298+0000","last_active":"2024-01-15T18:14:18.680560+0000","last_peered":"2024-01-15T18:14:18.680560+0000","last_clean":"2024-01-15T18:14:18.680560+0000","last_became_active":"2024-01-15T18:14:16.889887+0000","last_became_peered":"2024-01-15T18:14:16.889887+0000","last_unstale":"2024-01-15T18:14:18.680560+0000","last_undegraded":"2024-01-15T18:14:18.680560+0000","last_fullsized":"2024-01-15T18:14:18.680560+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:00.776476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678648+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:14:18.678648+0000","last_peered":"2024-01-15T18:14:18.678648+0000","last_clean":"2024-01-15T18:14:18.678648+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:14:18.678648+0000","last_undegraded":"2024-01-15T18:14:18.678648+0000","last_fullsized":"2024-01-15T18:14:18.678648+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678758+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:14:18.678758+0000","last_peered":"2024-01-15T18:14:18.678758+0000","last_clean":"2024-01-15T18:14:18.678758+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:14:18.678758+0000","last_undegraded":"2024-01-15T18:14:18.678758+0000","last_fullsized":"2024-01-15T18:14:18.678758+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678265+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:14:18.678265+0000","last_peered":"2024-01-15T18:14:18.678265+0000","last_clean":"2024-01-15T18:14:18.678265+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:14:18.678265+0000","last_undegraded":"2024-01-15T18:14:18.678265+0000","last_fullsized":"2024-01-15T18:14:18.678265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678549+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:14:18.678549+0000","last_peered":"2024-01-15T18:14:18.678549+0000","last_clean":"2024-01-15T18:14:18.678549+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:14:18.678549+0000","last_undegraded":"2024-01-15T18:14:18.678549+0000","last_fullsized":"2024-01-15T18:14:18.678549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682177+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:14:18.682177+0000","last_peered":"2024-01-15T18:14:18.682177+0000","last_clean":"2024-01-15T18:14:18.682177+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:14:18.682177+0000","last_undegraded":"2024-01-15T18:14:18.682177+0000","last_fullsized":"2024-01-15T18:14:18.682177+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868169+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:14:17.868169+0000","last_peered":"2024-01-15T18:14:17.868169+0000","last_clean":"2024-01-15T18:14:17.868169+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:14:17.868169+0000","last_undegraded":"2024-01-15T18:14:17.868169+0000","last_fullsized":"2024-01-15T18:14:17.868169+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867991+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:14:17.867991+0000","last_peered":"2024-01-15T18:14:17.867991+0000","last_clean":"2024-01-15T18:14:17.867991+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:14:17.867991+0000","last_undegraded":"2024-01-15T18:14:17.867991+0000","last_fullsized":"2024-01-15T18:14:17.867991+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869259+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:14:17.869259+0000","last_peered":"2024-01-15T18:14:17.869259+0000","last_clean":"2024-01-15T18:14:17.869259+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:14:17.869259+0000","last_undegraded":"2024-01-15T18:14:17.869259+0000","last_fullsized":"2024-01-15T18:14:17.869259+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868344+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:14:17.868344+0000","last_peered":"2024-01-15T18:14:17.868344+0000","last_clean":"2024-01-15T18:14:17.868344+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:14:17.868344+0000","last_undegraded":"2024-01-15T18:14:17.868344+0000","last_fullsized":"2024-01-15T18:14:17.868344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677066+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:14:18.677066+0000","last_peered":"2024-01-15T18:14:18.677066+0000","last_clean":"2024-01-15T18:14:18.677066+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:14:18.677066+0000","last_undegraded":"2024-01-15T18:14:18.677066+0000","last_fullsized":"2024-01-15T18:14:18.677066+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860501+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:14:17.860501+0000","last_peered":"2024-01-15T18:14:17.860501+0000","last_clean":"2024-01-15T18:14:17.860501+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:14:17.860501+0000","last_undegraded":"2024-01-15T18:14:17.860501+0000","last_fullsized":"2024-01-15T18:14:17.860501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678418+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:14:18.678418+0000","last_peered":"2024-01-15T18:14:18.678418+0000","last_clean":"2024-01-15T18:14:18.678418+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:14:18.678418+0000","last_undegraded":"2024-01-15T18:14:18.678418+0000","last_fullsized":"2024-01-15T18:14:18.678418+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678982+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:14:18.678982+0000","last_peered":"2024-01-15T18:14:18.678982+0000","last_clean":"2024-01-15T18:14:18.678982+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:14:18.678982+0000","last_undegraded":"2024-01-15T18:14:18.678982+0000","last_fullsized":"2024-01-15T18:14:18.678982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678496+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:14:18.678496+0000","last_peered":"2024-01-15T18:14:18.678496+0000","last_clean":"2024-01-15T18:14:18.678496+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:14:18.678496+0000","last_undegraded":"2024-01-15T18:14:18.678496+0000","last_fullsized":"2024-01-15T18:14:18.678496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867951+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:14:17.867951+0000","last_peered":"2024-01-15T18:14:17.867951+0000","last_clean":"2024-01-15T18:14:17.867951+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:14:17.867951+0000","last_undegraded":"2024-01-15T18:14:17.867951+0000","last_fullsized":"2024-01-15T18:14:17.867951+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":249,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-15T18:14:16.880953+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:14:16.880953+0000","last_peered":"2024-01-15T18:14:16.880953+0000","last_clean":"2024-01-15T18:14:16.880953+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:14:16.880953+0000","last_undegraded":"2024-01-15T18:14:16.880953+0000","last_fullsized":"2024-01-15T18:14:16.880953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542349,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65024,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":2015232,"data_stored":1210701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.268,"15min":1.4239999999999999},"min":{"1min":0.746,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.722,"5min":1.9359999999999999,"15min":14.114000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.5680000000000001},"min":{"1min":1.0700000000000001,"5min":0.78800000000000003,"15min":0.51700000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":27.32},"last":1.2110000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2809999999999999,"15min":1.52},"min":{"1min":0.85799999999999998,"5min":0.68799999999999994,"15min":0.40300000000000002},"max":{"1min":1.5580000000000001,"5min":5.9690000000000003,"15min":29.167999999999999},"last":1.3},{"interface":"front","average":{"1min":1.325,"5min":1.288,"15min":1.4239999999999999},"min":{"1min":0.89200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7749999999999999,"5min":2.1110000000000002,"15min":12.565},"last":1.137}]},{"osd":2,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3109999999999999,"15min":1.526},"min":{"1min":0.86199999999999999,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.8360000000000001,"5min":2.7360000000000002,"15min":13.801},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.361,"15min":1.52},"min":{"1min":0.96399999999999997,"5min":0.70099999999999996,"15min":0.56599999999999995},"max":{"1min":2.2160000000000002,"5min":3.5920000000000001,"15min":27.358000000000001},"last":1.466}]},{"osd":3,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.629},"min":{"1min":0.99399999999999999,"5min":0.83099999999999996,"15min":0.47099999999999997},"max":{"1min":1.8109999999999999,"5min":6.133,"15min":29.204999999999998},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.4410000000000001,"15min":1.5900000000000001},"min":{"1min":0.91400000000000003,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.173,"5min":2.617,"15min":16.088999999999999},"last":1.3640000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3360000000000001,"15min":1.544},"min":{"1min":0.91300000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":2.0329999999999999,"5min":2.2599999999999998,"15min":27.219000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.5069999999999999,"15min":1.6859999999999999},"min":{"1min":0.94799999999999995,"5min":0.88100000000000001,"15min":0.438},"max":{"1min":1.6850000000000001,"5min":6.4710000000000001,"15min":29.25},"last":1.6020000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4299999999999999,"15min":1.643},"min":{"1min":0.90800000000000003,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.77,"5min":2.23,"15min":12.518000000000001},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.51,"15min":1.669},"min":{"1min":1.0580000000000001,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":2.3889999999999998,"5min":2.4489999999999998,"15min":27.283000000000001},"last":1.6319999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.615,"15min":2.3260000000000001},"min":{"1min":1.153,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":53.664999999999999},"last":1.72},{"interface":"front","average":{"1min":1.462,"5min":1.609,"15min":1.8460000000000001},"min":{"1min":1.032,"5min":1.032,"15min":0.44},"max":{"1min":1.7549999999999999,"5min":6.1870000000000003,"15min":29.106999999999999},"last":1.4930000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542349,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64440,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307400,"statfs":{"total":96636764160,"available":96570777600,"internally_reserved":0,"allocated":1417216,"data_stored":619787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.242,"15min":1.24},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.438},"max":{"1min":1.708,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.25,"15min":1.3620000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59599999999999997},"max":{"1min":1.982,"5min":3.1659999999999999,"15min":21.565000000000001},"last":1.4810000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.494,"15min":1.4159999999999999},"min":{"1min":0.86599999999999999,"5min":0.64000000000000001,"15min":0.52100000000000002},"max":{"1min":13.269,"5min":13.269,"15min":13.269},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.681,"5min":1.3360000000000001,"15min":1.429},"min":{"1min":0.93300000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":6.335,"5min":6.335,"15min":13.984999999999999},"last":1.595}]},{"osd":2,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.25,"15min":1.45},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.7110000000000001,"5min":2.919,"15min":34.670000000000002},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.429,"15min":1.446},"min":{"1min":0.90700000000000003,"5min":0.71599999999999997,"15min":0.49399999999999999},"max":{"1min":1.9159999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.482,"15min":1.5820000000000001},"min":{"1min":1.1990000000000001,"5min":0.83599999999999997,"15min":0.68500000000000005},"max":{"1min":2.6640000000000001,"5min":3.5680000000000001,"15min":16.855},"last":1.329},{"interface":"front","average":{"1min":1.448,"5min":1.4239999999999999,"15min":1.6870000000000001},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.48499999999999999},"max":{"1min":1.883,"5min":2.9630000000000001,"15min":34.725000000000001},"last":1.4179999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.458,"15min":1.4650000000000001},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.38400000000000001},"max":{"1min":1.996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.05},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.504,"15min":1.51},"min":{"1min":0.88200000000000001,"5min":0.54900000000000004,"15min":0.501},"max":{"1min":2.8340000000000001,"5min":7.3230000000000004,"15min":10.118},"last":1.724}]},{"osd":5,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.5309999999999999,"15min":1.708},"min":{"1min":0.98899999999999999,"5min":0.67000000000000004,"15min":0.57699999999999996},"max":{"1min":2.0070000000000001,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.55},{"interface":"front","average":{"1min":1.55,"5min":1.6599999999999999,"15min":1.635},"min":{"1min":1.0840000000000001,"5min":0.69399999999999995,"15min":0.47899999999999998},"max":{"1min":2.0880000000000001,"5min":8.9900000000000002,"15min":10.077},"last":1.3720000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.6439999999999999,"15min":1.6379999999999999},"min":{"1min":1.208,"5min":0.72299999999999998,"15min":0.64000000000000001},"max":{"1min":2.7549999999999999,"5min":5.5060000000000002,"15min":10.302},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.556,"15min":1.8129999999999999},"min":{"1min":1.1339999999999999,"5min":0.58399999999999996,"15min":0.52600000000000002},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":34.762999999999998},"last":1.677}]}]},{"osd":5,"up_from":14,"seq":60129542351,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58944,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312896,"statfs":{"total":96636764160,"available":96576405504,"internally_reserved":0,"allocated":1421312,"data_stored":620321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.2450000000000001,"15min":1.403},"min":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":26.077999999999999},"last":1.294},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.4399999999999999,"15min":1.5189999999999999},"min":{"1min":0.875,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":5.3949999999999996,"5min":9.8699999999999992,"15min":14.711},"last":1.474}]},{"osd":1,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.282,"15min":1.4339999999999999},"min":{"1min":0.97699999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9730000000000001,"5min":2.8279999999999998,"15min":26},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.282,"15min":1.3959999999999999},"min":{"1min":0.85599999999999998,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":1.931,"5min":2.8660000000000001,"15min":12.619},"last":1.3400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.3680000000000001,"15min":1.5429999999999999},"min":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":2.8029999999999999,"5min":4.8719999999999999,"15min":26.234999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4650000000000001,"15min":1.6220000000000001},"min":{"1min":0.96199999999999997,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":1.9910000000000001,"5min":10.09,"15min":17.126999999999999},"last":1.218}]},{"osd":3,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2709999999999999,"15min":1.454},"min":{"1min":0.97999999999999998,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.802,"5min":1.841,"15min":12.722},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4279999999999999,"15min":1.6499999999999999},"min":{"1min":1.143,"5min":0.67900000000000005,"15min":0.63400000000000001},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":26.134},"last":1.5669999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3740000000000001,"15min":1.544},"min":{"1min":1.111,"5min":0.442,"15min":0.442},"max":{"1min":2.2869999999999999,"5min":9.9979999999999993,"15min":25.776},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.5509999999999999,"15min":1.768},"min":{"1min":1.214,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":2.1030000000000002,"5min":12.103,"15min":16.414000000000001},"last":1.395}]},{"osd":6,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.337,"15min":1.534},"min":{"1min":1.071,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.994,"5min":1.994,"15min":12.769},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.673,"5min":1.4630000000000001,"15min":1.798},"min":{"1min":1.1779999999999999,"5min":0.55500000000000005,"15min":0.435},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":36.087000000000003},"last":1.5129999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.661,"15min":1.921},"min":{"1min":1.319,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.1930000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.482,"15min":1.766},"min":{"1min":1.1719999999999999,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":3.1139999999999999,"5min":3.1139999999999999,"15min":21.728000000000002},"last":1.2609999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542350,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64432,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307408,"statfs":{"total":96636764160,"available":96570785792,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1379999999999999,"15min":1.3779999999999999},"min":{"1min":0.63500000000000001,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":1.637,"5min":3.4100000000000001,"15min":21.643000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.5349999999999999,"5min":2.1240000000000001,"15min":11.196},"last":1.4319999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.204,"15min":1.425},"min":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.8740000000000001,"5min":2.7589999999999999,"15min":28.663},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.54},"min":{"1min":0.82899999999999996,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":1.8680000000000001,"5min":3.5299999999999998,"15min":21.440000000000001},"last":1.2190000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3,"15min":1.4950000000000001},"min":{"1min":0.78300000000000003,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":2.032,"5min":2.7050000000000001,"15min":16.023},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5609999999999999,"15min":1.607},"min":{"1min":0.93600000000000005,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":1.748,"5min":18.858000000000001,"15min":22.350999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2969999999999999,"15min":1.5149999999999999},"min":{"1min":1.1180000000000001,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":1.9890000000000001,"5min":2.1890000000000001,"15min":15.093},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4239999999999999,"15min":1.6879999999999999},"min":{"1min":1.0309999999999999,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.9910000000000001,"5min":2.585,"15min":19.486000000000001},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.391,"15min":1.776},"min":{"1min":0.70999999999999996,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":1.996,"5min":3.6429999999999998,"15min":54.938000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.361,"15min":1.5249999999999999},"min":{"1min":0.99199999999999999,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":2.2349999999999999,"5min":2.6030000000000002,"15min":18.289999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5069999999999999,"15min":1.649},"min":{"1min":0.94599999999999995,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.375,"5min":2.9820000000000002,"15min":17.733000000000001},"last":1.389},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5680000000000001,"15min":1.877},"min":{"1min":1.0029999999999999,"5min":0.621,"15min":0.499},"max":{"1min":2.3079999999999998,"5min":3.7280000000000002,"15min":28.797000000000001},"last":1.355}]},{"osd":7,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.8260000000000001},"min":{"1min":0.93799999999999994,"5min":0.55800000000000005,"15min":0.45100000000000001},"max":{"1min":1.9710000000000001,"5min":2.5419999999999998,"15min":23.390000000000001},"last":1.496},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.633,"15min":1.8879999999999999},"min":{"1min":1.163,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.3599999999999999,"5min":2.851,"15min":22.751000000000001},"last":1.2569999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542351,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72768,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299072,"statfs":{"total":96636764160,"available":96562249728,"internally_reserved":0,"allocated":1429504,"data_stored":621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98299999999999998,"15min":1.1060000000000001},"min":{"1min":0.442,"5min":0.372,"15min":0.29799999999999999},"max":{"1min":2.0099999999999998,"5min":3.6960000000000002,"15min":36.408000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.024,"15min":1.0649999999999999},"min":{"1min":0.69499999999999995,"5min":0.42499999999999999,"15min":0.377},"max":{"1min":1.732,"5min":1.732,"15min":5.0529999999999999},"last":0.85099999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.109},"min":{"1min":0.53700000000000003,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.72,"5min":1.9590000000000001,"15min":8.1359999999999992},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2350000000000001,"15min":1.214},"min":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":1.8600000000000001,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":1.044}]},{"osd":2,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.306,"15min":1.329},"min":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":14.025},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.347,"15min":1.3180000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.46000000000000002},"max":{"1min":1.827,"5min":2.7130000000000001,"15min":5.0910000000000002},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.399,"15min":1.46},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.59999999999999998},"max":{"1min":1.8879999999999999,"5min":6.7720000000000002,"15min":17.082000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":1.105,"5min":0.92900000000000005,"15min":0.69499999999999995},"max":{"1min":9.8040000000000003,"5min":9.8040000000000003,"15min":15.769},"last":1.22}]},{"osd":5,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.6080000000000001},"min":{"1min":1.046,"5min":0.85599999999999998,"15min":0.65000000000000002},"max":{"1min":2.1389999999999998,"5min":2.7440000000000002,"15min":5.5220000000000002},"last":1.28},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.581,"15min":1.9430000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58299999999999996},"max":{"1min":1.9550000000000001,"5min":4.3390000000000004,"15min":40.384999999999998},"last":1.504}]},{"osd":6,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.571,"15min":1.7130000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.51400000000000001},"max":{"1min":2.7629999999999999,"5min":3.6800000000000002,"15min":32.259999999999998},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.679,"5min":1.6799999999999999,"15min":1.7609999999999999},"min":{"1min":1.1579999999999999,"5min":1.153,"15min":0.747},"max":{"1min":2.2519999999999998,"5min":3.9670000000000001,"15min":16.195},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.7490000000000001,"15min":1.913},"min":{"1min":1.026,"5min":0.95599999999999996,"15min":0.63900000000000001},"max":{"1min":2.0720000000000001,"5min":10.035,"15min":24.146999999999998},"last":1.659},{"interface":"front","average":{"1min":1.492,"5min":1.6100000000000001,"15min":1.774},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.56100000000000005},"max":{"1min":2.0270000000000001,"5min":3.4700000000000002,"15min":17.622},"last":1.7729999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542350,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0309999999999999,"15min":1.0129999999999999},"min":{"1min":0.60599999999999998,"5min":0.373,"15min":0.30099999999999999},"max":{"1min":1.641,"5min":10.256,"15min":10.256},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.0960000000000001,"15min":1.1020000000000001},"min":{"1min":0.77000000000000002,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.835,"5min":2.4020000000000001,"15min":10.319000000000001},"last":1.4239999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0309999999999999,"15min":1.119},"min":{"1min":0.68799999999999994,"5min":0.443,"15min":0.38600000000000001},"max":{"1min":1.6040000000000001,"5min":3.1360000000000001,"15min":10.807},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":0.95199999999999996,"5min":0.47099999999999997,"15min":0.35299999999999998},"max":{"1min":2.1429999999999998,"5min":4.7060000000000004,"15min":7.5119999999999996},"last":1.6619999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2589999999999999,"15min":1.4690000000000001},"min":{"1min":0.82099999999999995,"5min":0.55800000000000005,"15min":0.41399999999999998},"max":{"1min":1.6879999999999999,"5min":4.3970000000000002,"15min":25.891999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.49399999999999999,"15min":0.376},"max":{"1min":1.927,"5min":6.7229999999999999,"15min":14.946},"last":1.337}]},{"osd":4,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4119999999999999,"15min":1.516},"min":{"1min":0.98599999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.837,"5min":4.4649999999999999,"15min":9.0679999999999996},"last":1.544},{"interface":"front","average":{"1min":1.542,"5min":1.5389999999999999,"15min":1.591},"min":{"1min":1.1220000000000001,"5min":0.65300000000000002,"15min":0.54500000000000004},"max":{"1min":2.3540000000000001,"5min":8.4269999999999996,"15min":10.279999999999999},"last":2.145}]},{"osd":5,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.468,"15min":1.6679999999999999},"min":{"1min":1.016,"5min":0.67100000000000004,"15min":0.63500000000000001},"max":{"1min":1.7809999999999999,"5min":4.7759999999999998,"15min":11.071999999999999},"last":1.792},{"interface":"front","average":{"1min":1.486,"5min":1.534,"15min":1.7230000000000001},"min":{"1min":1.012,"5min":0.89700000000000002,"15min":0.55200000000000005},"max":{"1min":2.0609999999999999,"5min":4.5579999999999998,"15min":37.020000000000003},"last":1.7070000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6539999999999999,"15min":1.7170000000000001},"min":{"1min":1.3480000000000001,"5min":0.79100000000000004,"15min":0.64800000000000002},"max":{"1min":2.5550000000000002,"5min":8.5519999999999996,"15min":23.762},"last":1.859},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.625,"15min":1.911},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.73499999999999999},"max":{"1min":2.004,"5min":4.819,"15min":39.926000000000002},"last":1.603}]},{"osd":7,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.712,"5min":1.7410000000000001,"15min":1.958},"min":{"1min":1.1699999999999999,"5min":0.84299999999999997,"15min":0.68999999999999995},"max":{"1min":2.633,"5min":8.6620000000000008,"15min":52.767000000000003},"last":1.968},{"interface":"front","average":{"1min":1.623,"5min":1.6679999999999999,"15min":1.833},"min":{"1min":1.0880000000000001,"5min":0.877,"15min":0.83499999999999996},"max":{"1min":1.9219999999999999,"5min":4.8620000000000001,"15min":13.055999999999999},"last":2.0950000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542350,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1421312,"data_stored":619841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.1399999999999999,"15min":1.111},"min":{"1min":0.70499999999999996,"5min":0.36499999999999999,"15min":0.36199999999999999},"max":{"1min":11.964,"5min":11.964,"15min":19.218},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.901,"5min":1.2509999999999999,"15min":1.248},"min":{"1min":0.64700000000000002,"5min":0.42199999999999999,"15min":0.38700000000000001},"max":{"1min":15.176,"5min":15.176,"15min":19.823},"last":1.149}]},{"osd":2,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.226,"15min":1.2},"min":{"1min":0.68200000000000005,"5min":0.376,"15min":0.31900000000000001},"max":{"1min":19.02,"5min":19.02,"15min":19.02},"last":1.0369999999999999},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.4930000000000001,"15min":1.385},"min":{"1min":0.58899999999999997,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":29.251000000000001,"5min":29.251000000000001,"15min":29.251000000000001},"last":1.3}]},{"osd":3,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":1.5960000000000001,"15min":1.4870000000000001},"min":{"1min":0.65900000000000003,"5min":0.51100000000000001,"15min":0.40899999999999997},"max":{"1min":27.021000000000001,"5min":27.021000000000001,"15min":27.021000000000001},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.7610000000000001,"5min":1.49,"15min":1.4430000000000001},"min":{"1min":0.75700000000000001,"5min":0.43099999999999999,"15min":0.34799999999999998},"max":{"1min":31.218,"5min":31.218,"15min":31.218},"last":1.631}]},{"osd":4,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8170000000000002,"5min":1.8500000000000001,"15min":1.6910000000000001},"min":{"1min":1.0429999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":27.059999999999999,"5min":27.059999999999999,"15min":27.059999999999999},"last":1.486},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.772,"15min":1.764},"min":{"1min":1.0669999999999999,"5min":0.71599999999999997,"15min":0.44800000000000001},"max":{"1min":31.433,"5min":31.433,"15min":31.433},"last":1.2609999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.629,"15min":1.726},"min":{"1min":1.1870000000000001,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":20.012},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.53,"15min":1.6719999999999999},"min":{"1min":0.89900000000000002,"5min":0.71599999999999997,"15min":0.499},"max":{"1min":3.9910000000000001,"5min":3.9910000000000001,"15min":19.998999999999999},"last":1.4419999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":1.893,"15min":1.825},"min":{"1min":1.1899999999999999,"5min":0.85899999999999999,"15min":0.54100000000000004},"max":{"1min":29.530000000000001,"5min":29.530000000000001,"15min":29.530000000000001},"last":1.419},{"interface":"front","average":{"1min":3.3340000000000001,"5min":1.9019999999999999,"15min":1.931},"min":{"1min":1.165,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":31.186,"5min":31.186,"15min":31.186},"last":1.52}]},{"osd":7,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.8109999999999999,"15min":1.9239999999999999},"min":{"1min":1.2809999999999999,"5min":0.81699999999999995,"15min":0.505},"max":{"1min":26.393999999999998,"5min":26.393999999999998,"15min":26.393999999999998},"last":1.214},{"interface":"front","average":{"1min":2.25,"5min":1.728,"15min":1.887},"min":{"1min":1.145,"5min":0.92800000000000005,"15min":0.54500000000000004},"max":{"1min":9.7799999999999994,"5min":9.7799999999999994,"15min":27.582000000000001},"last":1.5900000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542349,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73360,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298480,"statfs":{"total":96636764160,"available":96561643520,"internally_reserved":0,"allocated":2027520,"data_stored":1211809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.177,"15min":1.2410000000000001},"min":{"1min":0.60799999999999998,"5min":0.38100000000000001,"15min":0.30599999999999999},"max":{"1min":12.353,"5min":14.542,"15min":20.437999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.198,"5min":1.0800000000000001,"15min":1.2190000000000001},"min":{"1min":0.628,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":1.704,"5min":1.9430000000000001,"15min":35.137999999999998},"last":1.2010000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.282,"15min":1.373},"min":{"1min":0.75700000000000001,"5min":0.44500000000000001,"15min":0.42799999999999999},"max":{"1min":2.6120000000000001,"5min":14.651,"15min":20.556000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.1990000000000001,"15min":1.387},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":35.183},"last":1.117}]},{"osd":3,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.087,"15min":1.163},"min":{"1min":0.85699999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":8.9949999999999992},"last":0.751},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5189999999999999,"15min":1.651},"min":{"1min":0.82899999999999996,"5min":0.623,"15min":0.45100000000000001},"max":{"1min":2.7269999999999999,"5min":14.598000000000001,"15min":25.372},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.4039999999999999,"15min":1.6850000000000001},"min":{"1min":1.2110000000000001,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":35.223999999999997},"last":1.474},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3160000000000001,"15min":1.4530000000000001},"min":{"1min":1.0009999999999999,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":1.851,"5min":2.1179999999999999,"15min":15.968999999999999},"last":1.597}]},{"osd":5,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.516,"15min":1.645},"min":{"1min":1.0529999999999999,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":12.173999999999999},"last":1.329},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6990000000000001,"15min":1.865},"min":{"1min":1.236,"5min":0.88300000000000001,"15min":0.76000000000000001},"max":{"1min":2.9049999999999998,"5min":14.859,"15min":20.789999999999999},"last":1.6890000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.7130000000000001,"15min":2.0539999999999998},"min":{"1min":1.1950000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7989999999999999,"5min":15.022,"15min":36.390999999999998},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.583,"15min":1.9430000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":6.8710000000000004,"5min":6.8710000000000004,"15min":36.207000000000001},"last":1.282}]},{"osd":7,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.645,"15min":1.96},"min":{"1min":0.91200000000000003,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":6.7969999999999997,"5min":6.7969999999999997,"15min":35.636000000000003},"last":1.637},{"interface":"front","average":{"1min":1.583,"5min":1.528,"15min":1.675},"min":{"1min":1.101,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0840000000000001,"5min":2.25,"15min":11.470000000000001},"last":1.3799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:14:22.365 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-15T18:14:22.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.626+0000 7fd7d4a00640 1 Processor -- start 2024-01-15T18:14:22.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.626+0000 7fd7d4a00640 1 -- start start 2024-01-15T18:14:22.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.628+0000 7fd7d4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd7cc15b7d0 0x7fd7cc06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.628+0000 7fd7d4a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc15cb00 con 0x7fd7cc15c0e0 2024-01-15T18:14:22.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.628+0000 7fd7d4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc0ccec0 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.628+0000 7fd7d4a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc0cccd0 con 0x7fd7cc15b7d0 2024-01-15T18:14:22.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.629+0000 7fd7d1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd7cc15b7d0 0x7fd7cc06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.629+0000 7fd7d1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd7cc15b7d0 0x7fd7cc06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37824/0 (socket says 172.21.15.2:37824) 2024-01-15T18:14:22.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.629+0000 7fd7d1600640 1 -- 172.21.15.2:0/694431327 learned_addr learned my addr 172.21.15.2:0/694431327 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:22.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.629+0000 7fd7d1600640 1 -- 172.21.15.2:0/694431327 >> v1:172.21.15.137:6789/0 conn(0x7fd7cc15c4c0 legacy=0x7fd7cc15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:22.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.629+0000 7fd7d1600640 1 -- 172.21.15.2:0/694431327 >> v1:172.21.15.2:6789/0 conn(0x7fd7cc15c0e0 legacy=0x7fd7cc0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.629+0000 7fd7d1600640 1 -- 172.21.15.2:0/694431327 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc0cd1a0 con 0x7fd7cc15b7d0 2024-01-15T18:14:22.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.630+0000 7fd7d1600640 1 --2- 172.21.15.2:0/694431327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15b7d0 0x7fd7cc06c470 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009870 tx=0x7fd7b402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c30a6616b6f460 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d0c00640 1 -- 172.21.15.2:0/694431327 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4030850 con 0x7fd7cc15b7d0 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d0c00640 1 -- 172.21.15.2:0/694431327 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b40309b0 con 0x7fd7cc15b7d0 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d0c00640 1 -- 172.21.15.2:0/694431327 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4030d40 con 0x7fd7cc15b7d0 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 -- 172.21.15.2:0/694431327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15b7d0 msgr2=0x7fd7cc06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/694431327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15b7d0 0x7fd7cc06c470 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4009870 tx=0x7fd7b402b990 comp rx=0 tx=0).stop 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 -- 172.21.15.2:0/694431327 shutdown_connections 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/694431327 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15b7d0 0x7fd7cc06c470 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 -- 172.21.15.2:0/694431327 >> 172.21.15.2:0/694431327 conn(0x7fd7cc06da20 msgr2=0x7fd7cc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 -- 172.21.15.2:0/694431327 shutdown_connections 2024-01-15T18:14:22.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.631+0000 7fd7d4a00640 1 -- 172.21.15.2:0/694431327 wait complete. 2024-01-15T18:14:22.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.633+0000 7fd7d4a00640 1 Processor -- start 2024-01-15T18:14:22.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.633+0000 7fd7d4a00640 1 -- start start 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7cc15b7d0 0x7fd7cc13e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7cc15c0e0 0x7fd7cc13e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 0x7fd7cc1e5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d4a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc153100 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc13ed50 con 0x7fd7cc15c0e0 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc14f260 con 0x7fd7cc15b7d0 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.634+0000 7fd7d2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 0x7fd7cc1e5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 0x7fd7cc1e5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37840/0 (socket says 172.21.15.2:37840) 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 -- 172.21.15.2:0/3870050874 learned_addr learned my addr 172.21.15.2:0/3870050874 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2000640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7cc15b7d0 0x7fd7cc13e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 -- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7cc15b7d0 msgr2=0x7fd7cc13e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7cc15b7d0 0x7fd7cc13e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d1600640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7cc15c0e0 0x7fd7cc13e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 -- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7cc15c0e0 msgr2=0x7fd7cc13e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7cc15c0e0 0x7fd7cc13e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 -- 172.21.15.2:0/3870050874 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b40094d0 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2000640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7cc15b7d0 0x7fd7cc13e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d1600640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7cc15c0e0 0x7fd7cc13e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.635+0000 7fd7d2a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 0x7fd7cc1e5b50 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400c780 tx=0x7fd7c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.636+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4004b60 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.636+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c400ce90 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.636+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc14f540 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.636+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c400a850 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.636+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc152d20 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.637+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc153240 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.638+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd7c400a9b0 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.639+0000 7fd7ca000640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd7a406c430 0x7fd7a406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:22.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.639+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd7c4088d40 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.640+0000 7fd7d2000640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd7a406c430 0x7fd7a406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:22.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.641+0000 7fd7d2000640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd7a406c430 0x7fd7a406e8f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc000f60 tx=0x7fd7bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:22.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.643+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd7cc153240 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.916+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd7cc153240 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.917+0000 7fd7ca000640 1 -- 172.21.15.2:0/3870050874 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1015 (secure 0 0 0) 0x7fd7cc153240 con 0x7fd7cc15c4c0 2024-01-15T18:14:22.920 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:22.920 INFO:teuthology.orchestra.run.smithi002.stdout:{"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1030,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":114,"num_osds":8,"num_up_osds":8,"osd_up_since":1705341443,"num_in_osds":8,"osd_in_since":1705341435,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":23,"data_bytes":592778,"bytes_used":557547520,"bytes_avail":772536565760,"bytes_total":773094113280,"write_bytes_sec":2388,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":74,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"b","status":"up:active","gid":9051}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":19,"modified":"2024-01-15T18:13:37.789675+0000","services":{}},"progress_events":{}} 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd7a406c430 msgr2=0x7fd7a406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd7a406c430 0x7fd7a406e8f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc000f60 tx=0x7fd7bc009040 comp rx=0 tx=0).stop 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 msgr2=0x7fd7cc1e5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 0x7fd7cc1e5b50 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400c780 tx=0x7fd7c400cc40 comp rx=0 tx=0).stop 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 shutdown_connections 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd7a406c430 0x7fd7a406e8f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd7cc15c4c0 0x7fd7cc1e5b50 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd7cc15c0e0 0x7fd7cc13e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 --2- 172.21.15.2:0/3870050874 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd7cc15b7d0 0x7fd7cc13e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:22.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.920+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 >> 172.21.15.2:0/3870050874 conn(0x7fd7cc06da20 msgr2=0x7fd7cc0aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:22.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.921+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 shutdown_connections 2024-01-15T18:14:22.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:22.921+0000 7fd7d4a00640 1 -- 172.21.15.2:0/3870050874 wait complete. 2024-01-15T18:14:22.944 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-15T18:14:22.944 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:14:23.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.166+0000 7fc01fe00640 1 Processor -- start 2024-01-15T18:14:23.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.166+0000 7fc01fe00640 1 -- start start 2024-01-15T18:14:23.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc020141880 0x7fc020070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:23.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc02015ec60 0x7fc02015f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:23.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01fe00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc02006dae0 con 0x7fc0200cd490 2024-01-15T18:14:23.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01fe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc020071f20 con 0x7fc020141880 2024-01-15T18:14:23.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc02006c6b0 con 0x7fc02015ec60 2024-01-15T18:14:23.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc020141880 0x7fc020070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01f400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc0200cd490 0x7fc0200cd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49434/0 (socket says 172.21.15.2:49434) 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.168+0000 7fc01f400640 1 -- 172.21.15.2:0/1621346460 learned_addr learned my addr 172.21.15.2:0/1621346460 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.169+0000 7fc01e000640 1 --2- 172.21.15.2:0/1621346460 >> v2:172.21.15.137:3300/0 conn(0x7fc02015ec60 0x7fc02015f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.169+0000 7fc01ea00640 1 -- 172.21.15.2:0/1621346460 >> v2:172.21.15.137:3300/0 conn(0x7fc02015ec60 msgr2=0x7fc02015f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.169+0000 7fc01ea00640 1 --2- 172.21.15.2:0/1621346460 >> v2:172.21.15.137:3300/0 conn(0x7fc02015ec60 0x7fc02015f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.169+0000 7fc01ea00640 1 -- 172.21.15.2:0/1621346460 >> v1:172.21.15.2:6789/0 conn(0x7fc0200cd490 legacy=0x7fc0200cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:23.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.169+0000 7fc01ea00640 1 -- 172.21.15.2:0/1621346460 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc020071960 con 0x7fc020141880 2024-01-15T18:14:23.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.170+0000 7fc01ea00640 1 --2- 172.21.15.2:0/1621346460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc020141880 0x7fc020070fc0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc0200722d0 tx=0x7fc01002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4d5372c0116ab5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:23.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01d600640 1 -- 172.21.15.2:0/1621346460 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01002e070 con 0x7fc020141880 2024-01-15T18:14:23.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01d600640 1 -- 172.21.15.2:0/1621346460 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0100315a0 con 0x7fc020141880 2024-01-15T18:14:23.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01d600640 1 -- 172.21.15.2:0/1621346460 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc010031930 con 0x7fc020141880 2024-01-15T18:14:23.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 -- 172.21.15.2:0/1621346460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc020141880 msgr2=0x7fc020070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:23.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 --2- 172.21.15.2:0/1621346460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc020141880 0x7fc020070fc0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fc0200722d0 tx=0x7fc01002b840 comp rx=0 tx=0).stop 2024-01-15T18:14:23.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 -- 172.21.15.2:0/1621346460 shutdown_connections 2024-01-15T18:14:23.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 --2- 172.21.15.2:0/1621346460 >> v2:172.21.15.137:3300/0 conn(0x7fc02015ec60 0x7fc02015f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 --2- 172.21.15.2:0/1621346460 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc020141880 0x7fc020070fc0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 -- 172.21.15.2:0/1621346460 >> 172.21.15.2:0/1621346460 conn(0x7fc0200b0840 msgr2=0x7fc0200b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:23.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 -- 172.21.15.2:0/1621346460 shutdown_connections 2024-01-15T18:14:23.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.171+0000 7fc01fe00640 1 -- 172.21.15.2:0/1621346460 wait complete. 2024-01-15T18:14:23.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.173+0000 7fc01fe00640 1 Processor -- start 2024-01-15T18:14:23.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.173+0000 7fc01fe00640 1 -- start start 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.174+0000 7fc01fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0200cd490 0x7fc020156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.174+0000 7fc01fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc020141880 0x7fc0201573b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.174+0000 7fc01fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 0x7fc0201581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.174+0000 7fc01fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc020157e10 con 0x7fc02015ec60 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.174+0000 7fc01fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0201578f0 con 0x7fc020141880 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.174+0000 7fc01fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200b1ad0 con 0x7fc0200cd490 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 0x7fc0201581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 0x7fc0201581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53000/0 (socket says 172.21.15.2:53000) 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 -- 172.21.15.2:0/67131610 learned_addr learned my addr 172.21.15.2:0/67131610 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01ea00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0200cd490 0x7fc020156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01e000640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc020141880 0x7fc0201573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:23.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 -- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0200cd490 msgr2=0x7fc020156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:23.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0200cd490 0x7fc020156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 -- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc020141880 msgr2=0x7fc0201573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:23.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc020141880 0x7fc0201573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 -- 172.21.15.2:0/67131610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc010009400 con 0x7fc02015ec60 2024-01-15T18:14:23.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.175+0000 7fc01f400640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 0x7fc0201581a0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fc01800cbc0 tx=0x7fc01800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.176+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01800ace0 con 0x7fc02015ec60 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.176+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01800ae40 con 0x7fc02015ec60 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.176+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc020157b30 con 0x7fc02015ec60 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.176+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc018012600 con 0x7fc02015ec60 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.176+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0200b2050 con 0x7fc02015ec60 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.177+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0200b2240 con 0x7fc02015ec60 2024-01-15T18:14:23.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.178+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc018007070 con 0x7fc02015ec60 2024-01-15T18:14:23.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.179+0000 7fc016a00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff806c430 0x7fbff806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:23.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.179+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fc018088850 con 0x7fc02015ec60 2024-01-15T18:14:23.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.180+0000 7fc01ea00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff806c430 0x7fbff806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:23.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.181+0000 7fc01ea00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff806c430 0x7fbff806e8f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc010036000 tx=0x7fc01003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:23.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.183+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc018054ed0 con 0x7fc02015ec60 2024-01-15T18:14:23.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.377+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc020141e80 con 0x7fbff806c430 2024-01-15T18:14:23.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.402+0000 7fc016a00640 1 -- 172.21.15.2:0/67131610 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286970 (secure 0 0 0) 0x7fc010031a10 con 0x7fbff806c430 2024-01-15T18:14:23.409 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:23.411 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:14:23.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.409+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff806c430 msgr2=0x7fbff806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.409+0000 7fc01fe00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff806c430 0x7fbff806e8f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc010036000 tx=0x7fc01003d040 comp rx=0 tx=0).stop 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.409+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 msgr2=0x7fc0201581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.409+0000 7fc01fe00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 0x7fc0201581a0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fc01800cbc0 tx=0x7fc01800a430 comp rx=0 tx=0).stop 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 shutdown_connections 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff806c430 0x7fbff806e8f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc02015ec60 0x7fc0201581a0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc020141880 0x7fc0201573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 --2- 172.21.15.2:0/67131610 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc0200cd490 0x7fc020156e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 >> 172.21.15.2:0/67131610 conn(0x7fc0200b0840 msgr2=0x7fc0200ce160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 shutdown_connections 2024-01-15T18:14:23.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:23.410+0000 7fc01fe00640 1 -- 172.21.15.2:0/67131610 wait complete. 2024-01-15T18:14:23.435 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-01-15T18:14:21.825120+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":544480,"kb_used_data":12264,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754430240,"statfs":{"total":773094113280,"available":772536565760,"internally_reserved":0,"allocated":12558336,"data_stored":6143292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":544853553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002045"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679832+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:14:18.679832+0000","last_peered":"2024-01-15T18:14:18.679832+0000","last_clean":"2024-01-15T18:14:18.679832+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:14:18.679832+0000","last_undegraded":"2024-01-15T18:14:18.679832+0000","last_fullsized":"2024-01-15T18:14:18.679832+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869474+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:14:17.869474+0000","last_peered":"2024-01-15T18:14:17.869474+0000","last_clean":"2024-01-15T18:14:17.869474+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:14:17.869474+0000","last_undegraded":"2024-01-15T18:14:17.869474+0000","last_fullsized":"2024-01-15T18:14:17.869474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678459+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:14:18.678459+0000","last_peered":"2024-01-15T18:14:18.678459+0000","last_clean":"2024-01-15T18:14:18.678459+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:14:18.678459+0000","last_undegraded":"2024-01-15T18:14:18.678459+0000","last_fullsized":"2024-01-15T18:14:18.678459+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677711+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:14:18.677711+0000","last_peered":"2024-01-15T18:14:18.677711+0000","last_clean":"2024-01-15T18:14:18.677711+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:14:18.677711+0000","last_undegraded":"2024-01-15T18:14:18.677711+0000","last_fullsized":"2024-01-15T18:14:18.677711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861012+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:14:17.861012+0000","last_peered":"2024-01-15T18:14:17.861012+0000","last_clean":"2024-01-15T18:14:17.861012+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:14:17.861012+0000","last_undegraded":"2024-01-15T18:14:17.861012+0000","last_fullsized":"2024-01-15T18:14:17.861012+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680661+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:14:18.680661+0000","last_peered":"2024-01-15T18:14:18.680661+0000","last_clean":"2024-01-15T18:14:18.680661+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:14:18.680661+0000","last_undegraded":"2024-01-15T18:14:18.680661+0000","last_fullsized":"2024-01-15T18:14:18.680661+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867714+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:14:17.867714+0000","last_peered":"2024-01-15T18:14:17.867714+0000","last_clean":"2024-01-15T18:14:17.867714+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:14:17.867714+0000","last_undegraded":"2024-01-15T18:14:17.867714+0000","last_fullsized":"2024-01-15T18:14:17.867714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676681+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:14:18.676681+0000","last_peered":"2024-01-15T18:14:18.676681+0000","last_clean":"2024-01-15T18:14:18.676681+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:14:18.676681+0000","last_undegraded":"2024-01-15T18:14:18.676681+0000","last_fullsized":"2024-01-15T18:14:18.676681+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680196+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:14:18.680196+0000","last_peered":"2024-01-15T18:14:18.680196+0000","last_clean":"2024-01-15T18:14:18.680196+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:14:18.680196+0000","last_undegraded":"2024-01-15T18:14:18.680196+0000","last_fullsized":"2024-01-15T18:14:18.680196+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678552+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:14:18.678552+0000","last_peered":"2024-01-15T18:14:18.678552+0000","last_clean":"2024-01-15T18:14:18.678552+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:14:18.678552+0000","last_undegraded":"2024-01-15T18:14:18.678552+0000","last_fullsized":"2024-01-15T18:14:18.678552+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869827+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:14:17.869827+0000","last_peered":"2024-01-15T18:14:17.869827+0000","last_clean":"2024-01-15T18:14:17.869827+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:14:17.869827+0000","last_undegraded":"2024-01-15T18:14:17.869827+0000","last_fullsized":"2024-01-15T18:14:17.869827+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677750+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:14:18.677750+0000","last_peered":"2024-01-15T18:14:18.677750+0000","last_clean":"2024-01-15T18:14:18.677750+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:14:18.677750+0000","last_undegraded":"2024-01-15T18:14:18.677750+0000","last_fullsized":"2024-01-15T18:14:18.677750+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677619+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:14:18.677619+0000","last_peered":"2024-01-15T18:14:18.677619+0000","last_clean":"2024-01-15T18:14:18.677619+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:14:18.677619+0000","last_undegraded":"2024-01-15T18:14:18.677619+0000","last_fullsized":"2024-01-15T18:14:18.677619+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676680+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:14:18.676680+0000","last_peered":"2024-01-15T18:14:18.676680+0000","last_clean":"2024-01-15T18:14:18.676680+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:14:18.676680+0000","last_undegraded":"2024-01-15T18:14:18.676680+0000","last_fullsized":"2024-01-15T18:14:18.676680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679775+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:14:18.679775+0000","last_peered":"2024-01-15T18:14:18.679775+0000","last_clean":"2024-01-15T18:14:18.679775+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:14:18.679775+0000","last_undegraded":"2024-01-15T18:14:18.679775+0000","last_fullsized":"2024-01-15T18:14:18.679775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676165+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:14:18.676165+0000","last_peered":"2024-01-15T18:14:18.676165+0000","last_clean":"2024-01-15T18:14:18.676165+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:14:18.676165+0000","last_undegraded":"2024-01-15T18:14:18.676165+0000","last_fullsized":"2024-01-15T18:14:18.676165+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870017+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:14:17.870017+0000","last_peered":"2024-01-15T18:14:17.870017+0000","last_clean":"2024-01-15T18:14:17.870017+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:14:17.870017+0000","last_undegraded":"2024-01-15T18:14:17.870017+0000","last_fullsized":"2024-01-15T18:14:17.870017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678195+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:14:18.678195+0000","last_peered":"2024-01-15T18:14:18.678195+0000","last_clean":"2024-01-15T18:14:18.678195+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:14:18.678195+0000","last_undegraded":"2024-01-15T18:14:18.678195+0000","last_fullsized":"2024-01-15T18:14:18.678195+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678973+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:14:18.678973+0000","last_peered":"2024-01-15T18:14:18.678973+0000","last_clean":"2024-01-15T18:14:18.678973+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:14:18.678973+0000","last_undegraded":"2024-01-15T18:14:18.678973+0000","last_fullsized":"2024-01-15T18:14:18.678973+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869183+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:14:17.869183+0000","last_peered":"2024-01-15T18:14:17.869183+0000","last_clean":"2024-01-15T18:14:17.869183+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:14:17.869183+0000","last_undegraded":"2024-01-15T18:14:17.869183+0000","last_fullsized":"2024-01-15T18:14:17.869183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679907+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:14:18.679907+0000","last_peered":"2024-01-15T18:14:18.679907+0000","last_clean":"2024-01-15T18:14:18.679907+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:14:18.679907+0000","last_undegraded":"2024-01-15T18:14:18.679907+0000","last_fullsized":"2024-01-15T18:14:18.679907+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867987+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:14:17.867987+0000","last_peered":"2024-01-15T18:14:17.867987+0000","last_clean":"2024-01-15T18:14:17.867987+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:14:17.867987+0000","last_undegraded":"2024-01-15T18:14:17.867987+0000","last_fullsized":"2024-01-15T18:14:17.867987+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678230+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:14:18.678230+0000","last_peered":"2024-01-15T18:14:18.678230+0000","last_clean":"2024-01-15T18:14:18.678230+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:14:18.678230+0000","last_undegraded":"2024-01-15T18:14:18.678230+0000","last_fullsized":"2024-01-15T18:14:18.678230+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870023+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:14:17.870023+0000","last_peered":"2024-01-15T18:14:17.870023+0000","last_clean":"2024-01-15T18:14:17.870023+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:14:17.870023+0000","last_undegraded":"2024-01-15T18:14:17.870023+0000","last_fullsized":"2024-01-15T18:14:17.870023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678920+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:14:18.678920+0000","last_peered":"2024-01-15T18:14:18.678920+0000","last_clean":"2024-01-15T18:14:18.678920+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:14:18.678920+0000","last_undegraded":"2024-01-15T18:14:18.678920+0000","last_fullsized":"2024-01-15T18:14:18.678920+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680581+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:14:18.680581+0000","last_peered":"2024-01-15T18:14:18.680581+0000","last_clean":"2024-01-15T18:14:18.680581+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:14:18.680581+0000","last_undegraded":"2024-01-15T18:14:18.680581+0000","last_fullsized":"2024-01-15T18:14:18.680581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681772+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:14:18.681772+0000","last_peered":"2024-01-15T18:14:18.681772+0000","last_clean":"2024-01-15T18:14:18.681772+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:14:18.681772+0000","last_undegraded":"2024-01-15T18:14:18.681772+0000","last_fullsized":"2024-01-15T18:14:18.681772+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868586+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:14:17.868586+0000","last_peered":"2024-01-15T18:14:17.868586+0000","last_clean":"2024-01-15T18:14:17.868586+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:14:17.868586+0000","last_undegraded":"2024-01-15T18:14:17.868586+0000","last_fullsized":"2024-01-15T18:14:17.868586+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678719+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:14:18.678719+0000","last_peered":"2024-01-15T18:14:18.678719+0000","last_clean":"2024-01-15T18:14:18.678719+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:14:18.678719+0000","last_undegraded":"2024-01-15T18:14:18.678719+0000","last_fullsized":"2024-01-15T18:14:18.678719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678401+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:14:18.678401+0000","last_peered":"2024-01-15T18:14:18.678401+0000","last_clean":"2024-01-15T18:14:18.678401+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:14:18.678401+0000","last_undegraded":"2024-01-15T18:14:18.678401+0000","last_fullsized":"2024-01-15T18:14:18.678401+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680132+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:14:18.680132+0000","last_peered":"2024-01-15T18:14:18.680132+0000","last_clean":"2024-01-15T18:14:18.680132+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:14:18.680132+0000","last_undegraded":"2024-01-15T18:14:18.680132+0000","last_fullsized":"2024-01-15T18:14:18.680132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869567+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:14:17.869567+0000","last_peered":"2024-01-15T18:14:17.869567+0000","last_clean":"2024-01-15T18:14:17.869567+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:14:17.869567+0000","last_undegraded":"2024-01-15T18:14:17.869567+0000","last_fullsized":"2024-01-15T18:14:17.869567+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679121+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:14:18.679121+0000","last_peered":"2024-01-15T18:14:18.679121+0000","last_clean":"2024-01-15T18:14:18.679121+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:14:18.679121+0000","last_undegraded":"2024-01-15T18:14:18.679121+0000","last_fullsized":"2024-01-15T18:14:18.679121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-15T18:14:16.899848+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:14:16.899848+0000","last_peered":"2024-01-15T18:14:16.899848+0000","last_clean":"2024-01-15T18:14:16.899848+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:14:16.899848+0000","last_undegraded":"2024-01-15T18:14:16.899848+0000","last_fullsized":"2024-01-15T18:14:16.899848+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681711+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:14:18.681711+0000","last_peered":"2024-01-15T18:14:18.681711+0000","last_clean":"2024-01-15T18:14:18.681711+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:14:18.681711+0000","last_undegraded":"2024-01-15T18:14:18.681711+0000","last_fullsized":"2024-01-15T18:14:18.681711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861547+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:14:17.861547+0000","last_peered":"2024-01-15T18:14:17.861547+0000","last_clean":"2024-01-15T18:14:17.861547+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:14:17.861547+0000","last_undegraded":"2024-01-15T18:14:17.861547+0000","last_fullsized":"2024-01-15T18:14:17.861547+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678678+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:14:18.678678+0000","last_peered":"2024-01-15T18:14:18.678678+0000","last_clean":"2024-01-15T18:14:18.678678+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:14:18.678678+0000","last_undegraded":"2024-01-15T18:14:18.678678+0000","last_fullsized":"2024-01-15T18:14:18.678678+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861087+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:14:17.861087+0000","last_peered":"2024-01-15T18:14:17.861087+0000","last_clean":"2024-01-15T18:14:17.861087+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:14:17.861087+0000","last_undegraded":"2024-01-15T18:14:17.861087+0000","last_fullsized":"2024-01-15T18:14:17.861087+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678804+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:14:18.678804+0000","last_peered":"2024-01-15T18:14:18.678804+0000","last_clean":"2024-01-15T18:14:18.678804+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:14:18.678804+0000","last_undegraded":"2024-01-15T18:14:18.678804+0000","last_fullsized":"2024-01-15T18:14:18.678804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867738+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:14:17.867738+0000","last_peered":"2024-01-15T18:14:17.867738+0000","last_clean":"2024-01-15T18:14:17.867738+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:14:17.867738+0000","last_undegraded":"2024-01-15T18:14:17.867738+0000","last_fullsized":"2024-01-15T18:14:17.867738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868928+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:14:17.868928+0000","last_peered":"2024-01-15T18:14:17.868928+0000","last_clean":"2024-01-15T18:14:17.868928+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:14:17.868928+0000","last_undegraded":"2024-01-15T18:14:17.868928+0000","last_fullsized":"2024-01-15T18:14:17.868928+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680146+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:14:18.680146+0000","last_peered":"2024-01-15T18:14:18.680146+0000","last_clean":"2024-01-15T18:14:18.680146+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:14:18.680146+0000","last_undegraded":"2024-01-15T18:14:18.680146+0000","last_fullsized":"2024-01-15T18:14:18.680146+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676494+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:14:18.676494+0000","last_peered":"2024-01-15T18:14:18.676494+0000","last_clean":"2024-01-15T18:14:18.676494+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:14:18.676494+0000","last_undegraded":"2024-01-15T18:14:18.676494+0000","last_fullsized":"2024-01-15T18:14:18.676494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867867+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:14:17.867867+0000","last_peered":"2024-01-15T18:14:17.867867+0000","last_clean":"2024-01-15T18:14:17.867867+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:14:17.867867+0000","last_undegraded":"2024-01-15T18:14:17.867867+0000","last_fullsized":"2024-01-15T18:14:17.867867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869074+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:14:17.869074+0000","last_peered":"2024-01-15T18:14:17.869074+0000","last_clean":"2024-01-15T18:14:17.869074+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:14:17.869074+0000","last_undegraded":"2024-01-15T18:14:17.869074+0000","last_fullsized":"2024-01-15T18:14:17.869074+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677064+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:14:18.677064+0000","last_peered":"2024-01-15T18:14:18.677064+0000","last_clean":"2024-01-15T18:14:18.677064+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:14:18.677064+0000","last_undegraded":"2024-01-15T18:14:18.677064+0000","last_fullsized":"2024-01-15T18:14:18.677064+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868864+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:14:17.868864+0000","last_peered":"2024-01-15T18:14:17.868864+0000","last_clean":"2024-01-15T18:14:17.868864+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:14:17.868864+0000","last_undegraded":"2024-01-15T18:14:17.868864+0000","last_fullsized":"2024-01-15T18:14:17.868864+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861295+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:14:17.861295+0000","last_peered":"2024-01-15T18:14:17.861295+0000","last_clean":"2024-01-15T18:14:17.861295+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:14:17.861295+0000","last_undegraded":"2024-01-15T18:14:17.861295+0000","last_fullsized":"2024-01-15T18:14:17.861295+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679149+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:14:18.679149+0000","last_peered":"2024-01-15T18:14:18.679149+0000","last_clean":"2024-01-15T18:14:18.679149+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:14:18.679149+0000","last_undegraded":"2024-01-15T18:14:18.679149+0000","last_fullsized":"2024-01-15T18:14:18.679149+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682044+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:14:18.682044+0000","last_peered":"2024-01-15T18:14:18.682044+0000","last_clean":"2024-01-15T18:14:18.682044+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:14:18.682044+0000","last_undegraded":"2024-01-15T18:14:18.682044+0000","last_fullsized":"2024-01-15T18:14:18.682044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682040+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:14:18.682040+0000","last_peered":"2024-01-15T18:14:18.682040+0000","last_clean":"2024-01-15T18:14:18.682040+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:14:18.682040+0000","last_undegraded":"2024-01-15T18:14:18.682040+0000","last_fullsized":"2024-01-15T18:14:18.682040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678030+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:14:18.678030+0000","last_peered":"2024-01-15T18:14:18.678030+0000","last_clean":"2024-01-15T18:14:18.678030+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:14:18.678030+0000","last_undegraded":"2024-01-15T18:14:18.678030+0000","last_fullsized":"2024-01-15T18:14:18.678030+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868462+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:14:17.868462+0000","last_peered":"2024-01-15T18:14:17.868462+0000","last_clean":"2024-01-15T18:14:17.868462+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:14:17.868462+0000","last_undegraded":"2024-01-15T18:14:17.868462+0000","last_fullsized":"2024-01-15T18:14:17.868462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678050+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:14:18.678050+0000","last_peered":"2024-01-15T18:14:18.678050+0000","last_clean":"2024-01-15T18:14:18.678050+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:14:18.678050+0000","last_undegraded":"2024-01-15T18:14:18.678050+0000","last_fullsized":"2024-01-15T18:14:18.678050+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679975+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:14:18.679975+0000","last_peered":"2024-01-15T18:14:18.679975+0000","last_clean":"2024-01-15T18:14:18.679975+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:14:18.679975+0000","last_undegraded":"2024-01-15T18:14:18.679975+0000","last_fullsized":"2024-01-15T18:14:18.679975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681067+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:14:18.681067+0000","last_peered":"2024-01-15T18:14:18.681067+0000","last_clean":"2024-01-15T18:14:18.681067+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:14:18.681067+0000","last_undegraded":"2024-01-15T18:14:18.681067+0000","last_fullsized":"2024-01-15T18:14:18.681067+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681154+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:14:18.681154+0000","last_peered":"2024-01-15T18:14:18.681154+0000","last_clean":"2024-01-15T18:14:18.681154+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:14:18.681154+0000","last_undegraded":"2024-01-15T18:14:18.681154+0000","last_fullsized":"2024-01-15T18:14:18.681154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678245+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:14:18.678245+0000","last_peered":"2024-01-15T18:14:18.678245+0000","last_clean":"2024-01-15T18:14:18.678245+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:14:18.678245+0000","last_undegraded":"2024-01-15T18:14:18.678245+0000","last_fullsized":"2024-01-15T18:14:18.678245+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678042+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:14:18.678042+0000","last_peered":"2024-01-15T18:14:18.678042+0000","last_clean":"2024-01-15T18:14:18.678042+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:14:18.678042+0000","last_undegraded":"2024-01-15T18:14:18.678042+0000","last_fullsized":"2024-01-15T18:14:18.678042+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860447+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:14:17.860447+0000","last_peered":"2024-01-15T18:14:17.860447+0000","last_clean":"2024-01-15T18:14:17.860447+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:14:17.860447+0000","last_undegraded":"2024-01-15T18:14:17.860447+0000","last_fullsized":"2024-01-15T18:14:17.860447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861344+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:14:17.861344+0000","last_peered":"2024-01-15T18:14:17.861344+0000","last_clean":"2024-01-15T18:14:17.861344+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:14:17.861344+0000","last_undegraded":"2024-01-15T18:14:17.861344+0000","last_fullsized":"2024-01-15T18:14:17.861344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680983+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:14:18.680983+0000","last_peered":"2024-01-15T18:14:18.680983+0000","last_clean":"2024-01-15T18:14:18.680983+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:14:18.680983+0000","last_undegraded":"2024-01-15T18:14:18.680983+0000","last_fullsized":"2024-01-15T18:14:18.680983+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681629+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:14:18.681629+0000","last_peered":"2024-01-15T18:14:18.681629+0000","last_clean":"2024-01-15T18:14:18.681629+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:14:18.681629+0000","last_undegraded":"2024-01-15T18:14:18.681629+0000","last_fullsized":"2024-01-15T18:14:18.681629+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861323+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:14:17.861323+0000","last_peered":"2024-01-15T18:14:17.861323+0000","last_clean":"2024-01-15T18:14:17.861323+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:14:17.861323+0000","last_undegraded":"2024-01-15T18:14:17.861323+0000","last_fullsized":"2024-01-15T18:14:17.861323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677207+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:14:18.677207+0000","last_peered":"2024-01-15T18:14:18.677207+0000","last_clean":"2024-01-15T18:14:18.677207+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:14:18.677207+0000","last_undegraded":"2024-01-15T18:14:18.677207+0000","last_fullsized":"2024-01-15T18:14:18.677207+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678475+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:14:18.678475+0000","last_peered":"2024-01-15T18:14:18.678475+0000","last_clean":"2024-01-15T18:14:18.678475+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:14:18.678475+0000","last_undegraded":"2024-01-15T18:14:18.678475+0000","last_fullsized":"2024-01-15T18:14:18.678475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678856+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:14:18.678856+0000","last_peered":"2024-01-15T18:14:18.678856+0000","last_clean":"2024-01-15T18:14:18.678856+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:14:18.678856+0000","last_undegraded":"2024-01-15T18:14:18.678856+0000","last_fullsized":"2024-01-15T18:14:18.678856+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867557+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:14:17.867557+0000","last_peered":"2024-01-15T18:14:17.867557+0000","last_clean":"2024-01-15T18:14:17.867557+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:14:17.867557+0000","last_undegraded":"2024-01-15T18:14:17.867557+0000","last_fullsized":"2024-01-15T18:14:17.867557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677088+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:14:18.677088+0000","last_peered":"2024-01-15T18:14:18.677088+0000","last_clean":"2024-01-15T18:14:18.677088+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:14:18.677088+0000","last_undegraded":"2024-01-15T18:14:18.677088+0000","last_fullsized":"2024-01-15T18:14:18.677088+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681172+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:14:18.681172+0000","last_peered":"2024-01-15T18:14:18.681172+0000","last_clean":"2024-01-15T18:14:18.681172+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:14:18.681172+0000","last_undegraded":"2024-01-15T18:14:18.681172+0000","last_fullsized":"2024-01-15T18:14:18.681172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676821+0000","last_change":"2024-01-15T18:14:16.886669+0000","last_active":"2024-01-15T18:14:18.676821+0000","last_peered":"2024-01-15T18:14:18.676821+0000","last_clean":"2024-01-15T18:14:18.676821+0000","last_became_active":"2024-01-15T18:14:16.886347+0000","last_became_peered":"2024-01-15T18:14:16.886347+0000","last_unstale":"2024-01-15T18:14:18.676821+0000","last_undegraded":"2024-01-15T18:14:18.676821+0000","last_fullsized":"2024-01-15T18:14:18.676821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:20:59.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860455+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:14:17.860455+0000","last_peered":"2024-01-15T18:14:17.860455+0000","last_clean":"2024-01-15T18:14:17.860455+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:14:17.860455+0000","last_undegraded":"2024-01-15T18:14:17.860455+0000","last_fullsized":"2024-01-15T18:14:17.860455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676782+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:14:18.676782+0000","last_peered":"2024-01-15T18:14:18.676782+0000","last_clean":"2024-01-15T18:14:18.676782+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:14:18.676782+0000","last_undegraded":"2024-01-15T18:14:18.676782+0000","last_fullsized":"2024-01-15T18:14:18.676782+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679825+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:14:18.679825+0000","last_peered":"2024-01-15T18:14:18.679825+0000","last_clean":"2024-01-15T18:14:18.679825+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:14:18.679825+0000","last_undegraded":"2024-01-15T18:14:18.679825+0000","last_fullsized":"2024-01-15T18:14:18.679825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679233+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:14:18.679233+0000","last_peered":"2024-01-15T18:14:18.679233+0000","last_clean":"2024-01-15T18:14:18.679233+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:14:18.679233+0000","last_undegraded":"2024-01-15T18:14:18.679233+0000","last_fullsized":"2024-01-15T18:14:18.679233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677309+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:14:18.677309+0000","last_peered":"2024-01-15T18:14:18.677309+0000","last_clean":"2024-01-15T18:14:18.677309+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:14:18.677309+0000","last_undegraded":"2024-01-15T18:14:18.677309+0000","last_fullsized":"2024-01-15T18:14:18.677309+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679019+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:14:18.679019+0000","last_peered":"2024-01-15T18:14:18.679019+0000","last_clean":"2024-01-15T18:14:18.679019+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:14:18.679019+0000","last_undegraded":"2024-01-15T18:14:18.679019+0000","last_fullsized":"2024-01-15T18:14:18.679019+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680985+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:14:18.680985+0000","last_peered":"2024-01-15T18:14:18.680985+0000","last_clean":"2024-01-15T18:14:18.680985+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:14:18.680985+0000","last_undegraded":"2024-01-15T18:14:18.680985+0000","last_fullsized":"2024-01-15T18:14:18.680985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680560+0000","last_change":"2024-01-15T18:14:16.894298+0000","last_active":"2024-01-15T18:14:18.680560+0000","last_peered":"2024-01-15T18:14:18.680560+0000","last_clean":"2024-01-15T18:14:18.680560+0000","last_became_active":"2024-01-15T18:14:16.889887+0000","last_became_peered":"2024-01-15T18:14:16.889887+0000","last_unstale":"2024-01-15T18:14:18.680560+0000","last_undegraded":"2024-01-15T18:14:18.680560+0000","last_fullsized":"2024-01-15T18:14:18.680560+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:00.776476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678648+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:14:18.678648+0000","last_peered":"2024-01-15T18:14:18.678648+0000","last_clean":"2024-01-15T18:14:18.678648+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:14:18.678648+0000","last_undegraded":"2024-01-15T18:14:18.678648+0000","last_fullsized":"2024-01-15T18:14:18.678648+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678758+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:14:18.678758+0000","last_peered":"2024-01-15T18:14:18.678758+0000","last_clean":"2024-01-15T18:14:18.678758+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:14:18.678758+0000","last_undegraded":"2024-01-15T18:14:18.678758+0000","last_fullsized":"2024-01-15T18:14:18.678758+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678265+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:14:18.678265+0000","last_peered":"2024-01-15T18:14:18.678265+0000","last_clean":"2024-01-15T18:14:18.678265+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:14:18.678265+0000","last_undegraded":"2024-01-15T18:14:18.678265+0000","last_fullsized":"2024-01-15T18:14:18.678265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678549+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:14:18.678549+0000","last_peered":"2024-01-15T18:14:18.678549+0000","last_clean":"2024-01-15T18:14:18.678549+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:14:18.678549+0000","last_undegraded":"2024-01-15T18:14:18.678549+0000","last_fullsized":"2024-01-15T18:14:18.678549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682177+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:14:18.682177+0000","last_peered":"2024-01-15T18:14:18.682177+0000","last_clean":"2024-01-15T18:14:18.682177+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:14:18.682177+0000","last_undegraded":"2024-01-15T18:14:18.682177+0000","last_fullsized":"2024-01-15T18:14:18.682177+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868169+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:14:17.868169+0000","last_peered":"2024-01-15T18:14:17.868169+0000","last_clean":"2024-01-15T18:14:17.868169+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:14:17.868169+0000","last_undegraded":"2024-01-15T18:14:17.868169+0000","last_fullsized":"2024-01-15T18:14:17.868169+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867991+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:14:17.867991+0000","last_peered":"2024-01-15T18:14:17.867991+0000","last_clean":"2024-01-15T18:14:17.867991+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:14:17.867991+0000","last_undegraded":"2024-01-15T18:14:17.867991+0000","last_fullsized":"2024-01-15T18:14:17.867991+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869259+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:14:17.869259+0000","last_peered":"2024-01-15T18:14:17.869259+0000","last_clean":"2024-01-15T18:14:17.869259+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:14:17.869259+0000","last_undegraded":"2024-01-15T18:14:17.869259+0000","last_fullsized":"2024-01-15T18:14:17.869259+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868344+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:14:17.868344+0000","last_peered":"2024-01-15T18:14:17.868344+0000","last_clean":"2024-01-15T18:14:17.868344+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:14:17.868344+0000","last_undegraded":"2024-01-15T18:14:17.868344+0000","last_fullsized":"2024-01-15T18:14:17.868344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677066+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:14:18.677066+0000","last_peered":"2024-01-15T18:14:18.677066+0000","last_clean":"2024-01-15T18:14:18.677066+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:14:18.677066+0000","last_undegraded":"2024-01-15T18:14:18.677066+0000","last_fullsized":"2024-01-15T18:14:18.677066+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860501+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:14:17.860501+0000","last_peered":"2024-01-15T18:14:17.860501+0000","last_clean":"2024-01-15T18:14:17.860501+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:14:17.860501+0000","last_undegraded":"2024-01-15T18:14:17.860501+0000","last_fullsized":"2024-01-15T18:14:17.860501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678418+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:14:18.678418+0000","last_peered":"2024-01-15T18:14:18.678418+0000","last_clean":"2024-01-15T18:14:18.678418+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:14:18.678418+0000","last_undegraded":"2024-01-15T18:14:18.678418+0000","last_fullsized":"2024-01-15T18:14:18.678418+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678982+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:14:18.678982+0000","last_peered":"2024-01-15T18:14:18.678982+0000","last_clean":"2024-01-15T18:14:18.678982+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:14:18.678982+0000","last_undegraded":"2024-01-15T18:14:18.678982+0000","last_fullsized":"2024-01-15T18:14:18.678982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678496+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:14:18.678496+0000","last_peered":"2024-01-15T18:14:18.678496+0000","last_clean":"2024-01-15T18:14:18.678496+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:14:18.678496+0000","last_undegraded":"2024-01-15T18:14:18.678496+0000","last_fullsized":"2024-01-15T18:14:18.678496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867951+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:14:17.867951+0000","last_peered":"2024-01-15T18:14:17.867951+0000","last_clean":"2024-01-15T18:14:17.867951+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:14:17.867951+0000","last_undegraded":"2024-01-15T18:14:17.867951+0000","last_fullsized":"2024-01-15T18:14:17.867951+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":249,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-15T18:14:16.880953+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:14:16.880953+0000","last_peered":"2024-01-15T18:14:16.880953+0000","last_clean":"2024-01-15T18:14:16.880953+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:14:16.880953+0000","last_undegraded":"2024-01-15T18:14:16.880953+0000","last_fullsized":"2024-01-15T18:14:16.880953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542349,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65024,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":2015232,"data_stored":1210701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.268,"15min":1.4239999999999999},"min":{"1min":0.746,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.722,"5min":1.9359999999999999,"15min":14.114000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.5680000000000001},"min":{"1min":1.0700000000000001,"5min":0.78800000000000003,"15min":0.51700000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":27.32},"last":1.2110000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2809999999999999,"15min":1.52},"min":{"1min":0.85799999999999998,"5min":0.68799999999999994,"15min":0.40300000000000002},"max":{"1min":1.5580000000000001,"5min":5.9690000000000003,"15min":29.167999999999999},"last":1.3},{"interface":"front","average":{"1min":1.325,"5min":1.288,"15min":1.4239999999999999},"min":{"1min":0.89200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7749999999999999,"5min":2.1110000000000002,"15min":12.565},"last":1.137}]},{"osd":2,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3109999999999999,"15min":1.526},"min":{"1min":0.86199999999999999,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.8360000000000001,"5min":2.7360000000000002,"15min":13.801},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.361,"15min":1.52},"min":{"1min":0.96399999999999997,"5min":0.70099999999999996,"15min":0.56599999999999995},"max":{"1min":2.2160000000000002,"5min":3.5920000000000001,"15min":27.358000000000001},"last":1.466}]},{"osd":3,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.629},"min":{"1min":0.99399999999999999,"5min":0.83099999999999996,"15min":0.47099999999999997},"max":{"1min":1.8109999999999999,"5min":6.133,"15min":29.204999999999998},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.4410000000000001,"15min":1.5900000000000001},"min":{"1min":0.91400000000000003,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.173,"5min":2.617,"15min":16.088999999999999},"last":1.3640000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3360000000000001,"15min":1.544},"min":{"1min":0.91300000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":2.0329999999999999,"5min":2.2599999999999998,"15min":27.219000000000001},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.5069999999999999,"15min":1.6859999999999999},"min":{"1min":0.94799999999999995,"5min":0.88100000000000001,"15min":0.438},"max":{"1min":1.6850000000000001,"5min":6.4710000000000001,"15min":29.25},"last":1.6020000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4299999999999999,"15min":1.643},"min":{"1min":0.90800000000000003,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.77,"5min":2.23,"15min":12.518000000000001},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.51,"15min":1.669},"min":{"1min":1.0580000000000001,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":2.3889999999999998,"5min":2.4489999999999998,"15min":27.283000000000001},"last":1.6319999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.615,"15min":2.3260000000000001},"min":{"1min":1.153,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":53.664999999999999},"last":1.72},{"interface":"front","average":{"1min":1.462,"5min":1.609,"15min":1.8460000000000001},"min":{"1min":1.032,"5min":1.032,"15min":0.44},"max":{"1min":1.7549999999999999,"5min":6.1870000000000003,"15min":29.106999999999999},"last":1.4930000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542349,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64440,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307400,"statfs":{"total":96636764160,"available":96570777600,"internally_reserved":0,"allocated":1417216,"data_stored":619787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.242,"15min":1.24},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.438},"max":{"1min":1.708,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.25,"15min":1.3620000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59599999999999997},"max":{"1min":1.982,"5min":3.1659999999999999,"15min":21.565000000000001},"last":1.4810000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.494,"15min":1.4159999999999999},"min":{"1min":0.86599999999999999,"5min":0.64000000000000001,"15min":0.52100000000000002},"max":{"1min":13.269,"5min":13.269,"15min":13.269},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.681,"5min":1.3360000000000001,"15min":1.429},"min":{"1min":0.93300000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":6.335,"5min":6.335,"15min":13.984999999999999},"last":1.595}]},{"osd":2,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.25,"15min":1.45},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.7110000000000001,"5min":2.919,"15min":34.670000000000002},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.429,"15min":1.446},"min":{"1min":0.90700000000000003,"5min":0.71599999999999997,"15min":0.49399999999999999},"max":{"1min":1.9159999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":1.2629999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.482,"15min":1.5820000000000001},"min":{"1min":1.1990000000000001,"5min":0.83599999999999997,"15min":0.68500000000000005},"max":{"1min":2.6640000000000001,"5min":3.5680000000000001,"15min":16.855},"last":1.329},{"interface":"front","average":{"1min":1.448,"5min":1.4239999999999999,"15min":1.6870000000000001},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.48499999999999999},"max":{"1min":1.883,"5min":2.9630000000000001,"15min":34.725000000000001},"last":1.4179999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.458,"15min":1.4650000000000001},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.38400000000000001},"max":{"1min":1.996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.05},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.504,"15min":1.51},"min":{"1min":0.88200000000000001,"5min":0.54900000000000004,"15min":0.501},"max":{"1min":2.8340000000000001,"5min":7.3230000000000004,"15min":10.118},"last":1.724}]},{"osd":5,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.5309999999999999,"15min":1.708},"min":{"1min":0.98899999999999999,"5min":0.67000000000000004,"15min":0.57699999999999996},"max":{"1min":2.0070000000000001,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.55},{"interface":"front","average":{"1min":1.55,"5min":1.6599999999999999,"15min":1.635},"min":{"1min":1.0840000000000001,"5min":0.69399999999999995,"15min":0.47899999999999998},"max":{"1min":2.0880000000000001,"5min":8.9900000000000002,"15min":10.077},"last":1.3720000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.6439999999999999,"15min":1.6379999999999999},"min":{"1min":1.208,"5min":0.72299999999999998,"15min":0.64000000000000001},"max":{"1min":2.7549999999999999,"5min":5.5060000000000002,"15min":10.302},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.556,"15min":1.8129999999999999},"min":{"1min":1.1339999999999999,"5min":0.58399999999999996,"15min":0.52600000000000002},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":34.762999999999998},"last":1.677}]}]},{"osd":5,"up_from":14,"seq":60129542351,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58944,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312896,"statfs":{"total":96636764160,"available":96576405504,"internally_reserved":0,"allocated":1421312,"data_stored":620321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.2450000000000001,"15min":1.403},"min":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":26.077999999999999},"last":1.294},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.4399999999999999,"15min":1.5189999999999999},"min":{"1min":0.875,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":5.3949999999999996,"5min":9.8699999999999992,"15min":14.711},"last":1.474}]},{"osd":1,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.282,"15min":1.4339999999999999},"min":{"1min":0.97699999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9730000000000001,"5min":2.8279999999999998,"15min":26},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.282,"15min":1.3959999999999999},"min":{"1min":0.85599999999999998,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":1.931,"5min":2.8660000000000001,"15min":12.619},"last":1.3400000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.3680000000000001,"15min":1.5429999999999999},"min":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":2.8029999999999999,"5min":4.8719999999999999,"15min":26.234999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.4650000000000001,"15min":1.6220000000000001},"min":{"1min":0.96199999999999997,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":1.9910000000000001,"5min":10.09,"15min":17.126999999999999},"last":1.218}]},{"osd":3,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2709999999999999,"15min":1.454},"min":{"1min":0.97999999999999998,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.802,"5min":1.841,"15min":12.722},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4279999999999999,"15min":1.6499999999999999},"min":{"1min":1.143,"5min":0.67900000000000005,"15min":0.63400000000000001},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":26.134},"last":1.5669999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3740000000000001,"15min":1.544},"min":{"1min":1.111,"5min":0.442,"15min":0.442},"max":{"1min":2.2869999999999999,"5min":9.9979999999999993,"15min":25.776},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.5509999999999999,"15min":1.768},"min":{"1min":1.214,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":2.1030000000000002,"5min":12.103,"15min":16.414000000000001},"last":1.395}]},{"osd":6,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.337,"15min":1.534},"min":{"1min":1.071,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.994,"5min":1.994,"15min":12.769},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.673,"5min":1.4630000000000001,"15min":1.798},"min":{"1min":1.1779999999999999,"5min":0.55500000000000005,"15min":0.435},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":36.087000000000003},"last":1.5129999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.661,"15min":1.921},"min":{"1min":1.319,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.1930000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.482,"15min":1.766},"min":{"1min":1.1719999999999999,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":3.1139999999999999,"5min":3.1139999999999999,"15min":21.728000000000002},"last":1.2609999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542350,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64432,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307408,"statfs":{"total":96636764160,"available":96570785792,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1379999999999999,"15min":1.3779999999999999},"min":{"1min":0.63500000000000001,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":1.637,"5min":3.4100000000000001,"15min":21.643000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.5349999999999999,"5min":2.1240000000000001,"15min":11.196},"last":1.4319999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.204,"15min":1.425},"min":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.8740000000000001,"5min":2.7589999999999999,"15min":28.663},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.54},"min":{"1min":0.82899999999999996,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":1.8680000000000001,"5min":3.5299999999999998,"15min":21.440000000000001},"last":1.2190000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3,"15min":1.4950000000000001},"min":{"1min":0.78300000000000003,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":2.032,"5min":2.7050000000000001,"15min":16.023},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5609999999999999,"15min":1.607},"min":{"1min":0.93600000000000005,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":1.748,"5min":18.858000000000001,"15min":22.350999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2969999999999999,"15min":1.5149999999999999},"min":{"1min":1.1180000000000001,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":1.9890000000000001,"5min":2.1890000000000001,"15min":15.093},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4239999999999999,"15min":1.6879999999999999},"min":{"1min":1.0309999999999999,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.9910000000000001,"5min":2.585,"15min":19.486000000000001},"last":1.1200000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.391,"15min":1.776},"min":{"1min":0.70999999999999996,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":1.996,"5min":3.6429999999999998,"15min":54.938000000000002},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.361,"15min":1.5249999999999999},"min":{"1min":0.99199999999999999,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":2.2349999999999999,"5min":2.6030000000000002,"15min":18.289999999999999},"last":0.92200000000000004}]},{"osd":6,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5069999999999999,"15min":1.649},"min":{"1min":0.94599999999999995,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.375,"5min":2.9820000000000002,"15min":17.733000000000001},"last":1.389},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5680000000000001,"15min":1.877},"min":{"1min":1.0029999999999999,"5min":0.621,"15min":0.499},"max":{"1min":2.3079999999999998,"5min":3.7280000000000002,"15min":28.797000000000001},"last":1.355}]},{"osd":7,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.8260000000000001},"min":{"1min":0.93799999999999994,"5min":0.55800000000000005,"15min":0.45100000000000001},"max":{"1min":1.9710000000000001,"5min":2.5419999999999998,"15min":23.390000000000001},"last":1.496},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.633,"15min":1.8879999999999999},"min":{"1min":1.163,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.3599999999999999,"5min":2.851,"15min":22.751000000000001},"last":1.2569999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542351,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72768,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299072,"statfs":{"total":96636764160,"available":96562249728,"internally_reserved":0,"allocated":1429504,"data_stored":621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98299999999999998,"15min":1.1060000000000001},"min":{"1min":0.442,"5min":0.372,"15min":0.29799999999999999},"max":{"1min":2.0099999999999998,"5min":3.6960000000000002,"15min":36.408000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.024,"15min":1.0649999999999999},"min":{"1min":0.69499999999999995,"5min":0.42499999999999999,"15min":0.377},"max":{"1min":1.732,"5min":1.732,"15min":5.0529999999999999},"last":0.85099999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.109},"min":{"1min":0.53700000000000003,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.72,"5min":1.9590000000000001,"15min":8.1359999999999992},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2350000000000001,"15min":1.214},"min":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":1.8600000000000001,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":1.044}]},{"osd":2,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.306,"15min":1.329},"min":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":14.025},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.347,"15min":1.3180000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.46000000000000002},"max":{"1min":1.827,"5min":2.7130000000000001,"15min":5.0910000000000002},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.399,"15min":1.46},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.59999999999999998},"max":{"1min":1.8879999999999999,"5min":6.7720000000000002,"15min":17.082000000000001},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":1.105,"5min":0.92900000000000005,"15min":0.69499999999999995},"max":{"1min":9.8040000000000003,"5min":9.8040000000000003,"15min":15.769},"last":1.22}]},{"osd":5,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.6080000000000001},"min":{"1min":1.046,"5min":0.85599999999999998,"15min":0.65000000000000002},"max":{"1min":2.1389999999999998,"5min":2.7440000000000002,"15min":5.5220000000000002},"last":1.28},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.581,"15min":1.9430000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58299999999999996},"max":{"1min":1.9550000000000001,"5min":4.3390000000000004,"15min":40.384999999999998},"last":1.504}]},{"osd":6,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.571,"15min":1.7130000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.51400000000000001},"max":{"1min":2.7629999999999999,"5min":3.6800000000000002,"15min":32.259999999999998},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.679,"5min":1.6799999999999999,"15min":1.7609999999999999},"min":{"1min":1.1579999999999999,"5min":1.153,"15min":0.747},"max":{"1min":2.2519999999999998,"5min":3.9670000000000001,"15min":16.195},"last":1.5629999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.7490000000000001,"15min":1.913},"min":{"1min":1.026,"5min":0.95599999999999996,"15min":0.63900000000000001},"max":{"1min":2.0720000000000001,"5min":10.035,"15min":24.146999999999998},"last":1.659},{"interface":"front","average":{"1min":1.492,"5min":1.6100000000000001,"15min":1.774},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.56100000000000005},"max":{"1min":2.0270000000000001,"5min":3.4700000000000002,"15min":17.622},"last":1.7729999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542350,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0309999999999999,"15min":1.0129999999999999},"min":{"1min":0.60599999999999998,"5min":0.373,"15min":0.30099999999999999},"max":{"1min":1.641,"5min":10.256,"15min":10.256},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.079,"5min":1.0960000000000001,"15min":1.1020000000000001},"min":{"1min":0.77000000000000002,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.835,"5min":2.4020000000000001,"15min":10.319000000000001},"last":1.4239999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0309999999999999,"15min":1.119},"min":{"1min":0.68799999999999994,"5min":0.443,"15min":0.38600000000000001},"max":{"1min":1.6040000000000001,"5min":3.1360000000000001,"15min":10.807},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":0.95199999999999996,"5min":0.47099999999999997,"15min":0.35299999999999998},"max":{"1min":2.1429999999999998,"5min":4.7060000000000004,"15min":7.5119999999999996},"last":1.6619999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2589999999999999,"15min":1.4690000000000001},"min":{"1min":0.82099999999999995,"5min":0.55800000000000005,"15min":0.41399999999999998},"max":{"1min":1.6879999999999999,"5min":4.3970000000000002,"15min":25.891999999999999},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.49399999999999999,"15min":0.376},"max":{"1min":1.927,"5min":6.7229999999999999,"15min":14.946},"last":1.337}]},{"osd":4,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4119999999999999,"15min":1.516},"min":{"1min":0.98599999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.837,"5min":4.4649999999999999,"15min":9.0679999999999996},"last":1.544},{"interface":"front","average":{"1min":1.542,"5min":1.5389999999999999,"15min":1.591},"min":{"1min":1.1220000000000001,"5min":0.65300000000000002,"15min":0.54500000000000004},"max":{"1min":2.3540000000000001,"5min":8.4269999999999996,"15min":10.279999999999999},"last":2.145}]},{"osd":5,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.468,"15min":1.6679999999999999},"min":{"1min":1.016,"5min":0.67100000000000004,"15min":0.63500000000000001},"max":{"1min":1.7809999999999999,"5min":4.7759999999999998,"15min":11.071999999999999},"last":1.792},{"interface":"front","average":{"1min":1.486,"5min":1.534,"15min":1.7230000000000001},"min":{"1min":1.012,"5min":0.89700000000000002,"15min":0.55200000000000005},"max":{"1min":2.0609999999999999,"5min":4.5579999999999998,"15min":37.020000000000003},"last":1.7070000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6539999999999999,"15min":1.7170000000000001},"min":{"1min":1.3480000000000001,"5min":0.79100000000000004,"15min":0.64800000000000002},"max":{"1min":2.5550000000000002,"5min":8.5519999999999996,"15min":23.762},"last":1.859},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.625,"15min":1.911},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.73499999999999999},"max":{"1min":2.004,"5min":4.819,"15min":39.926000000000002},"last":1.603}]},{"osd":7,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.712,"5min":1.7410000000000001,"15min":1.958},"min":{"1min":1.1699999999999999,"5min":0.84299999999999997,"15min":0.68999999999999995},"max":{"1min":2.633,"5min":8.6620000000000008,"15min":52.767000000000003},"last":1.968},{"interface":"front","average":{"1min":1.623,"5min":1.6679999999999999,"15min":1.833},"min":{"1min":1.0880000000000001,"5min":0.877,"15min":0.83499999999999996},"max":{"1min":1.9219999999999999,"5min":4.8620000000000001,"15min":13.055999999999999},"last":2.0950000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542350,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1421312,"data_stored":619841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.1399999999999999,"15min":1.111},"min":{"1min":0.70499999999999996,"5min":0.36499999999999999,"15min":0.36199999999999999},"max":{"1min":11.964,"5min":11.964,"15min":19.218},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.901,"5min":1.2509999999999999,"15min":1.248},"min":{"1min":0.64700000000000002,"5min":0.42199999999999999,"15min":0.38700000000000001},"max":{"1min":15.176,"5min":15.176,"15min":19.823},"last":1.149}]},{"osd":2,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.226,"15min":1.2},"min":{"1min":0.68200000000000005,"5min":0.376,"15min":0.31900000000000001},"max":{"1min":19.02,"5min":19.02,"15min":19.02},"last":1.0369999999999999},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.4930000000000001,"15min":1.385},"min":{"1min":0.58899999999999997,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":29.251000000000001,"5min":29.251000000000001,"15min":29.251000000000001},"last":1.3}]},{"osd":3,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":1.5960000000000001,"15min":1.4870000000000001},"min":{"1min":0.65900000000000003,"5min":0.51100000000000001,"15min":0.40899999999999997},"max":{"1min":27.021000000000001,"5min":27.021000000000001,"15min":27.021000000000001},"last":1.5580000000000001},{"interface":"front","average":{"1min":2.7610000000000001,"5min":1.49,"15min":1.4430000000000001},"min":{"1min":0.75700000000000001,"5min":0.43099999999999999,"15min":0.34799999999999998},"max":{"1min":31.218,"5min":31.218,"15min":31.218},"last":1.631}]},{"osd":4,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8170000000000002,"5min":1.8500000000000001,"15min":1.6910000000000001},"min":{"1min":1.0429999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":27.059999999999999,"5min":27.059999999999999,"15min":27.059999999999999},"last":1.486},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.772,"15min":1.764},"min":{"1min":1.0669999999999999,"5min":0.71599999999999997,"15min":0.44800000000000001},"max":{"1min":31.433,"5min":31.433,"15min":31.433},"last":1.2609999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.629,"15min":1.726},"min":{"1min":1.1870000000000001,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":20.012},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.722,"5min":1.53,"15min":1.6719999999999999},"min":{"1min":0.89900000000000002,"5min":0.71599999999999997,"15min":0.499},"max":{"1min":3.9910000000000001,"5min":3.9910000000000001,"15min":19.998999999999999},"last":1.4419999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":1.893,"15min":1.825},"min":{"1min":1.1899999999999999,"5min":0.85899999999999999,"15min":0.54100000000000004},"max":{"1min":29.530000000000001,"5min":29.530000000000001,"15min":29.530000000000001},"last":1.419},{"interface":"front","average":{"1min":3.3340000000000001,"5min":1.9019999999999999,"15min":1.931},"min":{"1min":1.165,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":31.186,"5min":31.186,"15min":31.186},"last":1.52}]},{"osd":7,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.8109999999999999,"15min":1.9239999999999999},"min":{"1min":1.2809999999999999,"5min":0.81699999999999995,"15min":0.505},"max":{"1min":26.393999999999998,"5min":26.393999999999998,"15min":26.393999999999998},"last":1.214},{"interface":"front","average":{"1min":2.25,"5min":1.728,"15min":1.887},"min":{"1min":1.145,"5min":0.92800000000000005,"15min":0.54500000000000004},"max":{"1min":9.7799999999999994,"5min":9.7799999999999994,"15min":27.582000000000001},"last":1.5900000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542349,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73360,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298480,"statfs":{"total":96636764160,"available":96561643520,"internally_reserved":0,"allocated":2027520,"data_stored":1211809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.177,"15min":1.2410000000000001},"min":{"1min":0.60799999999999998,"5min":0.38100000000000001,"15min":0.30599999999999999},"max":{"1min":12.353,"5min":14.542,"15min":20.437999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.198,"5min":1.0800000000000001,"15min":1.2190000000000001},"min":{"1min":0.628,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":1.704,"5min":1.9430000000000001,"15min":35.137999999999998},"last":1.2010000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.282,"15min":1.373},"min":{"1min":0.75700000000000001,"5min":0.44500000000000001,"15min":0.42799999999999999},"max":{"1min":2.6120000000000001,"5min":14.651,"15min":20.556000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.1990000000000001,"15min":1.387},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":35.183},"last":1.117}]},{"osd":3,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.087,"15min":1.163},"min":{"1min":0.85699999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":8.9949999999999992},"last":0.751},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5189999999999999,"15min":1.651},"min":{"1min":0.82899999999999996,"5min":0.623,"15min":0.45100000000000001},"max":{"1min":2.7269999999999999,"5min":14.598000000000001,"15min":25.372},"last":1.0600000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.4039999999999999,"15min":1.6850000000000001},"min":{"1min":1.2110000000000001,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":35.223999999999997},"last":1.474},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3160000000000001,"15min":1.4530000000000001},"min":{"1min":1.0009999999999999,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":1.851,"5min":2.1179999999999999,"15min":15.968999999999999},"last":1.597}]},{"osd":5,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.516,"15min":1.645},"min":{"1min":1.0529999999999999,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":12.173999999999999},"last":1.329},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6990000000000001,"15min":1.865},"min":{"1min":1.236,"5min":0.88300000000000001,"15min":0.76000000000000001},"max":{"1min":2.9049999999999998,"5min":14.859,"15min":20.789999999999999},"last":1.6890000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.7130000000000001,"15min":2.0539999999999998},"min":{"1min":1.1950000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7989999999999999,"5min":15.022,"15min":36.390999999999998},"last":1.8200000000000001},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.583,"15min":1.9430000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":6.8710000000000004,"5min":6.8710000000000004,"15min":36.207000000000001},"last":1.282}]},{"osd":7,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.645,"15min":1.96},"min":{"1min":0.91200000000000003,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":6.7969999999999997,"5min":6.7969999999999997,"15min":35.636000000000003},"last":1.637},{"interface":"front","average":{"1min":1.583,"5min":1.528,"15min":1.675},"min":{"1min":1.101,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0840000000000001,"5min":2.25,"15min":11.470000000000001},"last":1.3799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:14:26.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:14:26.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.667+0000 7f32e5e00640 1 Processor -- start 2024-01-15T18:14:26.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.668+0000 7f32e5e00640 1 -- start start 2024-01-15T18:14:26.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.669+0000 7f32e5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 0x7f32e006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.669+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 0x7f32e0161330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.669+0000 7f32e5e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e015e600 con 0x7f32e0161870 2024-01-15T18:14:26.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.669+0000 7f32e5e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e006fbc0 con 0x7f32e015d140 2024-01-15T18:14:26.673 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.669+0000 7f32e5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e015dd60 con 0x7f32e015d520 2024-01-15T18:14:26.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.670+0000 7f32e4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 0x7f32e006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.674 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.670+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 0x7f32e0161330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.670+0000 7f32e4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 0x7f32e006f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:54714/0 (socket says 172.21.15.2:54714) 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.670+0000 7f32e4a00640 1 -- 172.21.15.2:0/3312995614 learned_addr learned my addr 172.21.15.2:0/3312995614 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.670+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 0x7f32e0161330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:59498/0 (socket says 172.21.15.2:59498) 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.671+0000 7f32dfe00640 1 -- 172.21.15.2:0/3312995614 >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 msgr2=0x7f32e006f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.671+0000 7f32dfe00640 1 --2- 172.21.15.2:0/3312995614 >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 0x7f32e006f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.671+0000 7f32dfe00640 1 -- 172.21.15.2:0/3312995614 >> v1:172.21.15.137:6789/0 conn(0x7f32e0161870 legacy=0x7f32e00cdaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.671+0000 7f32dfe00640 1 -- 172.21.15.2:0/3312995614 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e015e040 con 0x7f32e015d520 2024-01-15T18:14:26.675 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.671+0000 7f32e4a00640 1 --2- 172.21.15.2:0/3312995614 >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 0x7f32e006f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:14:26.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.672+0000 7f32dfe00640 1 --2- 172.21.15.2:0/3312995614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 0x7f32e0161330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f32c80097a0 tx=0x7f32c802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb68885589feecd2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.673+0000 7f32df400640 1 -- 172.21.15.2:0/3312995614 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8031830 con 0x7f32e015d520 2024-01-15T18:14:26.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.673+0000 7f32df400640 1 -- 172.21.15.2:0/3312995614 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c8031990 con 0x7f32e015d520 2024-01-15T18:14:26.677 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.673+0000 7f32df400640 1 -- 172.21.15.2:0/3312995614 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c8031d20 con 0x7f32e015d520 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 -- 172.21.15.2:0/3312995614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 msgr2=0x7f32e0161330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 --2- 172.21.15.2:0/3312995614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 0x7f32e0161330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f32c80097a0 tx=0x7f32c802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 -- 172.21.15.2:0/3312995614 shutdown_connections 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 --2- 172.21.15.2:0/3312995614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d520 0x7f32e0161330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 --2- 172.21.15.2:0/3312995614 >> v2:172.21.15.137:3300/0 conn(0x7f32e015d140 0x7f32e006f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 -- 172.21.15.2:0/3312995614 >> 172.21.15.2:0/3312995614 conn(0x7f32e00cfd00 msgr2=0x7f32e00d00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 -- 172.21.15.2:0/3312995614 shutdown_connections 2024-01-15T18:14:26.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.674+0000 7f32e5e00640 1 -- 172.21.15.2:0/3312995614 wait complete. 2024-01-15T18:14:26.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.676+0000 7f32e5e00640 1 Processor -- start 2024-01-15T18:14:26.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.676+0000 7f32e5e00640 1 -- start start 2024-01-15T18:14:26.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d140 0x7f32e0157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 0x7f32e00c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32e00c77b0 0x7f32e0150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e01584e0 con 0x7f32e015d520 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00af330 con 0x7f32e00c77b0 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e00af610 con 0x7f32e015d140 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 0x7f32e00c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32dfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 0x7f32e00c94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53008/0 (socket says 172.21.15.2:53008) 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32dfe00640 1 -- 172.21.15.2:0/2080855247 learned_addr learned my addr 172.21.15.2:0/2080855247 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e5400640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32e00c77b0 0x7f32e0150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.677+0000 7f32e4a00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d140 0x7f32e0157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dfe00640 1 -- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d140 msgr2=0x7f32e0157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dfe00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d140 0x7f32e0157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dfe00640 1 -- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32e00c77b0 msgr2=0x7f32e0150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dfe00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32e00c77b0 0x7f32e0150bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dfe00640 1 -- 172.21.15.2:0/2080855247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00094d0 con 0x7f32e015d520 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dfe00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 0x7f32e00c94c0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f32c802bee0 tx=0x7f32c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c803a6f0 con 0x7f32e015d520 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c803a870 con 0x7f32e015d520 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.678+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c8009400 con 0x7f32e015d520 2024-01-15T18:14:26.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.679+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c803ac20 con 0x7f32e015d520 2024-01-15T18:14:26.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.679+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e00afbd0 con 0x7f32e015d520 2024-01-15T18:14:26.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.680+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e00d0bd0 con 0x7f32e015d520 2024-01-15T18:14:26.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.681+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f32c804c3b0 con 0x7f32e015d520 2024-01-15T18:14:26.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.682+0000 7f32dcc00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32b806c430 0x7f32b806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:26.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.682+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f32c80aa6b0 con 0x7f32e015d520 2024-01-15T18:14:26.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.682+0000 7f32e4a00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32b806c430 0x7f32b806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:26.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.683+0000 7f32e4a00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32b806c430 0x7f32b806e8f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009cf0 tx=0x7f32d0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:26.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.686+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32c8076c80 con 0x7f32e015d520 2024-01-15T18:14:26.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.877+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f32e00ca1c0 con 0x7f32b806c430 2024-01-15T18:14:26.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.901+0000 7f32dcc00640 1 -- 172.21.15.2:0/2080855247 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286871 (secure 0 0 0) 0x7f32e00ca1c0 con 0x7f32b806c430 2024-01-15T18:14:26.910 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:26.912 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32b806c430 msgr2=0x7f32b806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32b806c430 0x7f32b806e8f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f32d0009cf0 tx=0x7f32d0008040 comp rx=0 tx=0).stop 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 msgr2=0x7f32e00c94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 0x7f32e00c94c0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f32c802bee0 tx=0x7f32c80046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 shutdown_connections 2024-01-15T18:14:26.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f32b806c430 0x7f32b806e8f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f32e00c77b0 0x7f32e0150bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f32e015d520 0x7f32e00c94c0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 --2- 172.21.15.2:0/2080855247 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f32e015d140 0x7f32e0157fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 >> 172.21.15.2:0/2080855247 conn(0x7f32e00cfd00 msgr2=0x7f32e006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 shutdown_connections 2024-01-15T18:14:26.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:26.907+0000 7f32e5e00640 1 -- 172.21.15.2:0/2080855247 wait complete. 2024-01-15T18:14:26.930 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-01-15T18:14:25.826398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":544480,"kb_used_data":12264,"kb_used_omap":12,"kb_used_meta":532083,"kb_avail":754430240,"statfs":{"total":773094113280,"available":772536565760,"internally_reserved":0,"allocated":12558336,"data_stored":6143292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":544853553},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.976183"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679832+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:14:18.679832+0000","last_peered":"2024-01-15T18:14:18.679832+0000","last_clean":"2024-01-15T18:14:18.679832+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:14:18.679832+0000","last_undegraded":"2024-01-15T18:14:18.679832+0000","last_fullsized":"2024-01-15T18:14:18.679832+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869474+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:14:17.869474+0000","last_peered":"2024-01-15T18:14:17.869474+0000","last_clean":"2024-01-15T18:14:17.869474+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:14:17.869474+0000","last_undegraded":"2024-01-15T18:14:17.869474+0000","last_fullsized":"2024-01-15T18:14:17.869474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678459+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:14:18.678459+0000","last_peered":"2024-01-15T18:14:18.678459+0000","last_clean":"2024-01-15T18:14:18.678459+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:14:18.678459+0000","last_undegraded":"2024-01-15T18:14:18.678459+0000","last_fullsized":"2024-01-15T18:14:18.678459+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677711+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:14:18.677711+0000","last_peered":"2024-01-15T18:14:18.677711+0000","last_clean":"2024-01-15T18:14:18.677711+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:14:18.677711+0000","last_undegraded":"2024-01-15T18:14:18.677711+0000","last_fullsized":"2024-01-15T18:14:18.677711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861012+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:14:17.861012+0000","last_peered":"2024-01-15T18:14:17.861012+0000","last_clean":"2024-01-15T18:14:17.861012+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:14:17.861012+0000","last_undegraded":"2024-01-15T18:14:17.861012+0000","last_fullsized":"2024-01-15T18:14:17.861012+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680661+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:14:18.680661+0000","last_peered":"2024-01-15T18:14:18.680661+0000","last_clean":"2024-01-15T18:14:18.680661+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:14:18.680661+0000","last_undegraded":"2024-01-15T18:14:18.680661+0000","last_fullsized":"2024-01-15T18:14:18.680661+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867714+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:14:17.867714+0000","last_peered":"2024-01-15T18:14:17.867714+0000","last_clean":"2024-01-15T18:14:17.867714+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:14:17.867714+0000","last_undegraded":"2024-01-15T18:14:17.867714+0000","last_fullsized":"2024-01-15T18:14:17.867714+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676681+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:14:18.676681+0000","last_peered":"2024-01-15T18:14:18.676681+0000","last_clean":"2024-01-15T18:14:18.676681+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:14:18.676681+0000","last_undegraded":"2024-01-15T18:14:18.676681+0000","last_fullsized":"2024-01-15T18:14:18.676681+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680196+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:14:18.680196+0000","last_peered":"2024-01-15T18:14:18.680196+0000","last_clean":"2024-01-15T18:14:18.680196+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:14:18.680196+0000","last_undegraded":"2024-01-15T18:14:18.680196+0000","last_fullsized":"2024-01-15T18:14:18.680196+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678552+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:14:18.678552+0000","last_peered":"2024-01-15T18:14:18.678552+0000","last_clean":"2024-01-15T18:14:18.678552+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:14:18.678552+0000","last_undegraded":"2024-01-15T18:14:18.678552+0000","last_fullsized":"2024-01-15T18:14:18.678552+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869827+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:14:17.869827+0000","last_peered":"2024-01-15T18:14:17.869827+0000","last_clean":"2024-01-15T18:14:17.869827+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:14:17.869827+0000","last_undegraded":"2024-01-15T18:14:17.869827+0000","last_fullsized":"2024-01-15T18:14:17.869827+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677750+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:14:18.677750+0000","last_peered":"2024-01-15T18:14:18.677750+0000","last_clean":"2024-01-15T18:14:18.677750+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:14:18.677750+0000","last_undegraded":"2024-01-15T18:14:18.677750+0000","last_fullsized":"2024-01-15T18:14:18.677750+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677619+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:14:18.677619+0000","last_peered":"2024-01-15T18:14:18.677619+0000","last_clean":"2024-01-15T18:14:18.677619+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:14:18.677619+0000","last_undegraded":"2024-01-15T18:14:18.677619+0000","last_fullsized":"2024-01-15T18:14:18.677619+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676680+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:14:18.676680+0000","last_peered":"2024-01-15T18:14:18.676680+0000","last_clean":"2024-01-15T18:14:18.676680+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:14:18.676680+0000","last_undegraded":"2024-01-15T18:14:18.676680+0000","last_fullsized":"2024-01-15T18:14:18.676680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679775+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:14:18.679775+0000","last_peered":"2024-01-15T18:14:18.679775+0000","last_clean":"2024-01-15T18:14:18.679775+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:14:18.679775+0000","last_undegraded":"2024-01-15T18:14:18.679775+0000","last_fullsized":"2024-01-15T18:14:18.679775+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676165+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:14:18.676165+0000","last_peered":"2024-01-15T18:14:18.676165+0000","last_clean":"2024-01-15T18:14:18.676165+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:14:18.676165+0000","last_undegraded":"2024-01-15T18:14:18.676165+0000","last_fullsized":"2024-01-15T18:14:18.676165+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870017+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:14:17.870017+0000","last_peered":"2024-01-15T18:14:17.870017+0000","last_clean":"2024-01-15T18:14:17.870017+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:14:17.870017+0000","last_undegraded":"2024-01-15T18:14:17.870017+0000","last_fullsized":"2024-01-15T18:14:17.870017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678195+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:14:18.678195+0000","last_peered":"2024-01-15T18:14:18.678195+0000","last_clean":"2024-01-15T18:14:18.678195+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:14:18.678195+0000","last_undegraded":"2024-01-15T18:14:18.678195+0000","last_fullsized":"2024-01-15T18:14:18.678195+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678973+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:14:18.678973+0000","last_peered":"2024-01-15T18:14:18.678973+0000","last_clean":"2024-01-15T18:14:18.678973+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:14:18.678973+0000","last_undegraded":"2024-01-15T18:14:18.678973+0000","last_fullsized":"2024-01-15T18:14:18.678973+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869183+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:14:17.869183+0000","last_peered":"2024-01-15T18:14:17.869183+0000","last_clean":"2024-01-15T18:14:17.869183+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:14:17.869183+0000","last_undegraded":"2024-01-15T18:14:17.869183+0000","last_fullsized":"2024-01-15T18:14:17.869183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679907+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:14:18.679907+0000","last_peered":"2024-01-15T18:14:18.679907+0000","last_clean":"2024-01-15T18:14:18.679907+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:14:18.679907+0000","last_undegraded":"2024-01-15T18:14:18.679907+0000","last_fullsized":"2024-01-15T18:14:18.679907+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867987+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:14:17.867987+0000","last_peered":"2024-01-15T18:14:17.867987+0000","last_clean":"2024-01-15T18:14:17.867987+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:14:17.867987+0000","last_undegraded":"2024-01-15T18:14:17.867987+0000","last_fullsized":"2024-01-15T18:14:17.867987+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678230+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:14:18.678230+0000","last_peered":"2024-01-15T18:14:18.678230+0000","last_clean":"2024-01-15T18:14:18.678230+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:14:18.678230+0000","last_undegraded":"2024-01-15T18:14:18.678230+0000","last_fullsized":"2024-01-15T18:14:18.678230+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.870023+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:14:17.870023+0000","last_peered":"2024-01-15T18:14:17.870023+0000","last_clean":"2024-01-15T18:14:17.870023+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:14:17.870023+0000","last_undegraded":"2024-01-15T18:14:17.870023+0000","last_fullsized":"2024-01-15T18:14:17.870023+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678920+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:14:18.678920+0000","last_peered":"2024-01-15T18:14:18.678920+0000","last_clean":"2024-01-15T18:14:18.678920+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:14:18.678920+0000","last_undegraded":"2024-01-15T18:14:18.678920+0000","last_fullsized":"2024-01-15T18:14:18.678920+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680581+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:14:18.680581+0000","last_peered":"2024-01-15T18:14:18.680581+0000","last_clean":"2024-01-15T18:14:18.680581+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:14:18.680581+0000","last_undegraded":"2024-01-15T18:14:18.680581+0000","last_fullsized":"2024-01-15T18:14:18.680581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681772+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:14:18.681772+0000","last_peered":"2024-01-15T18:14:18.681772+0000","last_clean":"2024-01-15T18:14:18.681772+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:14:18.681772+0000","last_undegraded":"2024-01-15T18:14:18.681772+0000","last_fullsized":"2024-01-15T18:14:18.681772+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868586+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:14:17.868586+0000","last_peered":"2024-01-15T18:14:17.868586+0000","last_clean":"2024-01-15T18:14:17.868586+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:14:17.868586+0000","last_undegraded":"2024-01-15T18:14:17.868586+0000","last_fullsized":"2024-01-15T18:14:17.868586+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869569+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:14:17.869569+0000","last_peered":"2024-01-15T18:14:17.869569+0000","last_clean":"2024-01-15T18:14:17.869569+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:14:17.869569+0000","last_undegraded":"2024-01-15T18:14:17.869569+0000","last_fullsized":"2024-01-15T18:14:17.869569+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678719+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:14:18.678719+0000","last_peered":"2024-01-15T18:14:18.678719+0000","last_clean":"2024-01-15T18:14:18.678719+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:14:18.678719+0000","last_undegraded":"2024-01-15T18:14:18.678719+0000","last_fullsized":"2024-01-15T18:14:18.678719+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678401+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:14:18.678401+0000","last_peered":"2024-01-15T18:14:18.678401+0000","last_clean":"2024-01-15T18:14:18.678401+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:14:18.678401+0000","last_undegraded":"2024-01-15T18:14:18.678401+0000","last_fullsized":"2024-01-15T18:14:18.678401+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680132+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:14:18.680132+0000","last_peered":"2024-01-15T18:14:18.680132+0000","last_clean":"2024-01-15T18:14:18.680132+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:14:18.680132+0000","last_undegraded":"2024-01-15T18:14:18.680132+0000","last_fullsized":"2024-01-15T18:14:18.680132+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":16,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869567+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:14:17.869567+0000","last_peered":"2024-01-15T18:14:17.869567+0000","last_clean":"2024-01-15T18:14:17.869567+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:14:17.869567+0000","last_undegraded":"2024-01-15T18:14:17.869567+0000","last_fullsized":"2024-01-15T18:14:17.869567+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679121+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:14:18.679121+0000","last_peered":"2024-01-15T18:14:18.679121+0000","last_clean":"2024-01-15T18:14:18.679121+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:14:18.679121+0000","last_undegraded":"2024-01-15T18:14:18.679121+0000","last_fullsized":"2024-01-15T18:14:18.679121+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":13,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-15T18:14:16.899848+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:14:16.899848+0000","last_peered":"2024-01-15T18:14:16.899848+0000","last_clean":"2024-01-15T18:14:16.899848+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:14:16.899848+0000","last_undegraded":"2024-01-15T18:14:16.899848+0000","last_fullsized":"2024-01-15T18:14:16.899848+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681711+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:14:18.681711+0000","last_peered":"2024-01-15T18:14:18.681711+0000","last_clean":"2024-01-15T18:14:18.681711+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:14:18.681711+0000","last_undegraded":"2024-01-15T18:14:18.681711+0000","last_fullsized":"2024-01-15T18:14:18.681711+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861547+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:14:17.861547+0000","last_peered":"2024-01-15T18:14:17.861547+0000","last_clean":"2024-01-15T18:14:17.861547+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:14:17.861547+0000","last_undegraded":"2024-01-15T18:14:17.861547+0000","last_fullsized":"2024-01-15T18:14:17.861547+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678678+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:14:18.678678+0000","last_peered":"2024-01-15T18:14:18.678678+0000","last_clean":"2024-01-15T18:14:18.678678+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:14:18.678678+0000","last_undegraded":"2024-01-15T18:14:18.678678+0000","last_fullsized":"2024-01-15T18:14:18.678678+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861087+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:14:17.861087+0000","last_peered":"2024-01-15T18:14:17.861087+0000","last_clean":"2024-01-15T18:14:17.861087+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:14:17.861087+0000","last_undegraded":"2024-01-15T18:14:17.861087+0000","last_fullsized":"2024-01-15T18:14:17.861087+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678804+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:14:18.678804+0000","last_peered":"2024-01-15T18:14:18.678804+0000","last_clean":"2024-01-15T18:14:18.678804+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:14:18.678804+0000","last_undegraded":"2024-01-15T18:14:18.678804+0000","last_fullsized":"2024-01-15T18:14:18.678804+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867738+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:14:17.867738+0000","last_peered":"2024-01-15T18:14:17.867738+0000","last_clean":"2024-01-15T18:14:17.867738+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:14:17.867738+0000","last_undegraded":"2024-01-15T18:14:17.867738+0000","last_fullsized":"2024-01-15T18:14:17.867738+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868928+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:14:17.868928+0000","last_peered":"2024-01-15T18:14:17.868928+0000","last_clean":"2024-01-15T18:14:17.868928+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:14:17.868928+0000","last_undegraded":"2024-01-15T18:14:17.868928+0000","last_fullsized":"2024-01-15T18:14:17.868928+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680146+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:14:18.680146+0000","last_peered":"2024-01-15T18:14:18.680146+0000","last_clean":"2024-01-15T18:14:18.680146+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:14:18.680146+0000","last_undegraded":"2024-01-15T18:14:18.680146+0000","last_fullsized":"2024-01-15T18:14:18.680146+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676494+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:14:18.676494+0000","last_peered":"2024-01-15T18:14:18.676494+0000","last_clean":"2024-01-15T18:14:18.676494+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:14:18.676494+0000","last_undegraded":"2024-01-15T18:14:18.676494+0000","last_fullsized":"2024-01-15T18:14:18.676494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867867+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:14:17.867867+0000","last_peered":"2024-01-15T18:14:17.867867+0000","last_clean":"2024-01-15T18:14:17.867867+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:14:17.867867+0000","last_undegraded":"2024-01-15T18:14:17.867867+0000","last_fullsized":"2024-01-15T18:14:17.867867+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869074+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:14:17.869074+0000","last_peered":"2024-01-15T18:14:17.869074+0000","last_clean":"2024-01-15T18:14:17.869074+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:14:17.869074+0000","last_undegraded":"2024-01-15T18:14:17.869074+0000","last_fullsized":"2024-01-15T18:14:17.869074+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677064+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:14:18.677064+0000","last_peered":"2024-01-15T18:14:18.677064+0000","last_clean":"2024-01-15T18:14:18.677064+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:14:18.677064+0000","last_undegraded":"2024-01-15T18:14:18.677064+0000","last_fullsized":"2024-01-15T18:14:18.677064+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868864+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:14:17.868864+0000","last_peered":"2024-01-15T18:14:17.868864+0000","last_clean":"2024-01-15T18:14:17.868864+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:14:17.868864+0000","last_undegraded":"2024-01-15T18:14:17.868864+0000","last_fullsized":"2024-01-15T18:14:17.868864+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861295+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:14:17.861295+0000","last_peered":"2024-01-15T18:14:17.861295+0000","last_clean":"2024-01-15T18:14:17.861295+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:14:17.861295+0000","last_undegraded":"2024-01-15T18:14:17.861295+0000","last_fullsized":"2024-01-15T18:14:17.861295+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679149+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:14:18.679149+0000","last_peered":"2024-01-15T18:14:18.679149+0000","last_clean":"2024-01-15T18:14:18.679149+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:14:18.679149+0000","last_undegraded":"2024-01-15T18:14:18.679149+0000","last_fullsized":"2024-01-15T18:14:18.679149+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682044+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:14:18.682044+0000","last_peered":"2024-01-15T18:14:18.682044+0000","last_clean":"2024-01-15T18:14:18.682044+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:14:18.682044+0000","last_undegraded":"2024-01-15T18:14:18.682044+0000","last_fullsized":"2024-01-15T18:14:18.682044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682040+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:14:18.682040+0000","last_peered":"2024-01-15T18:14:18.682040+0000","last_clean":"2024-01-15T18:14:18.682040+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:14:18.682040+0000","last_undegraded":"2024-01-15T18:14:18.682040+0000","last_fullsized":"2024-01-15T18:14:18.682040+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"113'3","reported_seq":15,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678030+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:14:18.678030+0000","last_peered":"2024-01-15T18:14:18.678030+0000","last_clean":"2024-01-15T18:14:18.678030+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:14:18.678030+0000","last_undegraded":"2024-01-15T18:14:18.678030+0000","last_fullsized":"2024-01-15T18:14:18.678030+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868462+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:14:17.868462+0000","last_peered":"2024-01-15T18:14:17.868462+0000","last_clean":"2024-01-15T18:14:17.868462+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:14:17.868462+0000","last_undegraded":"2024-01-15T18:14:17.868462+0000","last_fullsized":"2024-01-15T18:14:17.868462+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"113'6","reported_seq":18,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678050+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:14:18.678050+0000","last_peered":"2024-01-15T18:14:18.678050+0000","last_clean":"2024-01-15T18:14:18.678050+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:14:18.678050+0000","last_undegraded":"2024-01-15T18:14:18.678050+0000","last_fullsized":"2024-01-15T18:14:18.678050+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679975+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:14:18.679975+0000","last_peered":"2024-01-15T18:14:18.679975+0000","last_clean":"2024-01-15T18:14:18.679975+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:14:18.679975+0000","last_undegraded":"2024-01-15T18:14:18.679975+0000","last_fullsized":"2024-01-15T18:14:18.679975+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681067+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:14:18.681067+0000","last_peered":"2024-01-15T18:14:18.681067+0000","last_clean":"2024-01-15T18:14:18.681067+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:14:18.681067+0000","last_undegraded":"2024-01-15T18:14:18.681067+0000","last_fullsized":"2024-01-15T18:14:18.681067+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681154+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:14:18.681154+0000","last_peered":"2024-01-15T18:14:18.681154+0000","last_clean":"2024-01-15T18:14:18.681154+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:14:18.681154+0000","last_undegraded":"2024-01-15T18:14:18.681154+0000","last_fullsized":"2024-01-15T18:14:18.681154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678245+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:14:18.678245+0000","last_peered":"2024-01-15T18:14:18.678245+0000","last_clean":"2024-01-15T18:14:18.678245+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:14:18.678245+0000","last_undegraded":"2024-01-15T18:14:18.678245+0000","last_fullsized":"2024-01-15T18:14:18.678245+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678042+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:14:18.678042+0000","last_peered":"2024-01-15T18:14:18.678042+0000","last_clean":"2024-01-15T18:14:18.678042+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:14:18.678042+0000","last_undegraded":"2024-01-15T18:14:18.678042+0000","last_fullsized":"2024-01-15T18:14:18.678042+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":20,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860447+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:14:17.860447+0000","last_peered":"2024-01-15T18:14:17.860447+0000","last_clean":"2024-01-15T18:14:17.860447+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:14:17.860447+0000","last_undegraded":"2024-01-15T18:14:17.860447+0000","last_fullsized":"2024-01-15T18:14:17.860447+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861344+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:14:17.861344+0000","last_peered":"2024-01-15T18:14:17.861344+0000","last_clean":"2024-01-15T18:14:17.861344+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:14:17.861344+0000","last_undegraded":"2024-01-15T18:14:17.861344+0000","last_fullsized":"2024-01-15T18:14:17.861344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680983+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:14:18.680983+0000","last_peered":"2024-01-15T18:14:18.680983+0000","last_clean":"2024-01-15T18:14:18.680983+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:14:18.680983+0000","last_undegraded":"2024-01-15T18:14:18.680983+0000","last_fullsized":"2024-01-15T18:14:18.680983+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681629+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:14:18.681629+0000","last_peered":"2024-01-15T18:14:18.681629+0000","last_clean":"2024-01-15T18:14:18.681629+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:14:18.681629+0000","last_undegraded":"2024-01-15T18:14:18.681629+0000","last_fullsized":"2024-01-15T18:14:18.681629+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.861323+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:14:17.861323+0000","last_peered":"2024-01-15T18:14:17.861323+0000","last_clean":"2024-01-15T18:14:17.861323+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:14:17.861323+0000","last_undegraded":"2024-01-15T18:14:17.861323+0000","last_fullsized":"2024-01-15T18:14:17.861323+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677207+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:14:18.677207+0000","last_peered":"2024-01-15T18:14:18.677207+0000","last_clean":"2024-01-15T18:14:18.677207+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:14:18.677207+0000","last_undegraded":"2024-01-15T18:14:18.677207+0000","last_fullsized":"2024-01-15T18:14:18.677207+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678475+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:14:18.678475+0000","last_peered":"2024-01-15T18:14:18.678475+0000","last_clean":"2024-01-15T18:14:18.678475+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:14:18.678475+0000","last_undegraded":"2024-01-15T18:14:18.678475+0000","last_fullsized":"2024-01-15T18:14:18.678475+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678856+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:14:18.678856+0000","last_peered":"2024-01-15T18:14:18.678856+0000","last_clean":"2024-01-15T18:14:18.678856+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:14:18.678856+0000","last_undegraded":"2024-01-15T18:14:18.678856+0000","last_fullsized":"2024-01-15T18:14:18.678856+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867557+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:14:17.867557+0000","last_peered":"2024-01-15T18:14:17.867557+0000","last_clean":"2024-01-15T18:14:17.867557+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:14:17.867557+0000","last_undegraded":"2024-01-15T18:14:17.867557+0000","last_fullsized":"2024-01-15T18:14:17.867557+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677088+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:14:18.677088+0000","last_peered":"2024-01-15T18:14:18.677088+0000","last_clean":"2024-01-15T18:14:18.677088+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:14:18.677088+0000","last_undegraded":"2024-01-15T18:14:18.677088+0000","last_fullsized":"2024-01-15T18:14:18.677088+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.681172+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:14:18.681172+0000","last_peered":"2024-01-15T18:14:18.681172+0000","last_clean":"2024-01-15T18:14:18.681172+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:14:18.681172+0000","last_undegraded":"2024-01-15T18:14:18.681172+0000","last_fullsized":"2024-01-15T18:14:18.681172+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676821+0000","last_change":"2024-01-15T18:14:16.886669+0000","last_active":"2024-01-15T18:14:18.676821+0000","last_peered":"2024-01-15T18:14:18.676821+0000","last_clean":"2024-01-15T18:14:18.676821+0000","last_became_active":"2024-01-15T18:14:16.886347+0000","last_became_peered":"2024-01-15T18:14:16.886347+0000","last_unstale":"2024-01-15T18:14:18.676821+0000","last_undegraded":"2024-01-15T18:14:18.676821+0000","last_fullsized":"2024-01-15T18:14:18.676821+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:20:59.894992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860455+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:14:17.860455+0000","last_peered":"2024-01-15T18:14:17.860455+0000","last_clean":"2024-01-15T18:14:17.860455+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:14:17.860455+0000","last_undegraded":"2024-01-15T18:14:17.860455+0000","last_fullsized":"2024-01-15T18:14:17.860455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.676782+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:14:18.676782+0000","last_peered":"2024-01-15T18:14:18.676782+0000","last_clean":"2024-01-15T18:14:18.676782+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:14:18.676782+0000","last_undegraded":"2024-01-15T18:14:18.676782+0000","last_fullsized":"2024-01-15T18:14:18.676782+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679825+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:14:18.679825+0000","last_peered":"2024-01-15T18:14:18.679825+0000","last_clean":"2024-01-15T18:14:18.679825+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:14:18.679825+0000","last_undegraded":"2024-01-15T18:14:18.679825+0000","last_fullsized":"2024-01-15T18:14:18.679825+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679233+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:14:18.679233+0000","last_peered":"2024-01-15T18:14:18.679233+0000","last_clean":"2024-01-15T18:14:18.679233+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:14:18.679233+0000","last_undegraded":"2024-01-15T18:14:18.679233+0000","last_fullsized":"2024-01-15T18:14:18.679233+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677309+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:14:18.677309+0000","last_peered":"2024-01-15T18:14:18.677309+0000","last_clean":"2024-01-15T18:14:18.677309+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:14:18.677309+0000","last_undegraded":"2024-01-15T18:14:18.677309+0000","last_fullsized":"2024-01-15T18:14:18.677309+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.679019+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:14:18.679019+0000","last_peered":"2024-01-15T18:14:18.679019+0000","last_clean":"2024-01-15T18:14:18.679019+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:14:18.679019+0000","last_undegraded":"2024-01-15T18:14:18.679019+0000","last_fullsized":"2024-01-15T18:14:18.679019+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":14,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680985+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:14:18.680985+0000","last_peered":"2024-01-15T18:14:18.680985+0000","last_clean":"2024-01-15T18:14:18.680985+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:14:18.680985+0000","last_undegraded":"2024-01-15T18:14:18.680985+0000","last_fullsized":"2024-01-15T18:14:18.680985+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.680560+0000","last_change":"2024-01-15T18:14:16.894298+0000","last_active":"2024-01-15T18:14:18.680560+0000","last_peered":"2024-01-15T18:14:18.680560+0000","last_clean":"2024-01-15T18:14:18.680560+0000","last_became_active":"2024-01-15T18:14:16.889887+0000","last_became_peered":"2024-01-15T18:14:16.889887+0000","last_unstale":"2024-01-15T18:14:18.680560+0000","last_undegraded":"2024-01-15T18:14:18.680560+0000","last_fullsized":"2024-01-15T18:14:18.680560+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:51:00.776476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"114'1","reported_seq":13,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:19.766904+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:14:19.766904+0000","last_peered":"2024-01-15T18:14:19.766904+0000","last_clean":"2024-01-15T18:14:19.766904+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:14:19.766904+0000","last_undegraded":"2024-01-15T18:14:19.766904+0000","last_fullsized":"2024-01-15T18:14:19.766904+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678758+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:14:18.678758+0000","last_peered":"2024-01-15T18:14:18.678758+0000","last_clean":"2024-01-15T18:14:18.678758+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:14:18.678758+0000","last_undegraded":"2024-01-15T18:14:18.678758+0000","last_fullsized":"2024-01-15T18:14:18.678758+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678265+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:14:18.678265+0000","last_peered":"2024-01-15T18:14:18.678265+0000","last_clean":"2024-01-15T18:14:18.678265+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:14:18.678265+0000","last_undegraded":"2024-01-15T18:14:18.678265+0000","last_fullsized":"2024-01-15T18:14:18.678265+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678549+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:14:18.678549+0000","last_peered":"2024-01-15T18:14:18.678549+0000","last_clean":"2024-01-15T18:14:18.678549+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:14:18.678549+0000","last_undegraded":"2024-01-15T18:14:18.678549+0000","last_fullsized":"2024-01-15T18:14:18.678549+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.682177+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:14:18.682177+0000","last_peered":"2024-01-15T18:14:18.682177+0000","last_clean":"2024-01-15T18:14:18.682177+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:14:18.682177+0000","last_undegraded":"2024-01-15T18:14:18.682177+0000","last_fullsized":"2024-01-15T18:14:18.682177+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868169+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:14:17.868169+0000","last_peered":"2024-01-15T18:14:17.868169+0000","last_clean":"2024-01-15T18:14:17.868169+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:14:17.868169+0000","last_undegraded":"2024-01-15T18:14:17.868169+0000","last_fullsized":"2024-01-15T18:14:17.868169+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867991+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:14:17.867991+0000","last_peered":"2024-01-15T18:14:17.867991+0000","last_clean":"2024-01-15T18:14:17.867991+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:14:17.867991+0000","last_undegraded":"2024-01-15T18:14:17.867991+0000","last_fullsized":"2024-01-15T18:14:17.867991+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.869259+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:14:17.869259+0000","last_peered":"2024-01-15T18:14:17.869259+0000","last_clean":"2024-01-15T18:14:17.869259+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:14:17.869259+0000","last_undegraded":"2024-01-15T18:14:17.869259+0000","last_fullsized":"2024-01-15T18:14:17.869259+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.868344+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:14:17.868344+0000","last_peered":"2024-01-15T18:14:17.868344+0000","last_clean":"2024-01-15T18:14:17.868344+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:14:17.868344+0000","last_undegraded":"2024-01-15T18:14:17.868344+0000","last_fullsized":"2024-01-15T18:14:17.868344+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.677066+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:14:18.677066+0000","last_peered":"2024-01-15T18:14:18.677066+0000","last_clean":"2024-01-15T18:14:18.677066+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:14:18.677066+0000","last_undegraded":"2024-01-15T18:14:18.677066+0000","last_fullsized":"2024-01-15T18:14:18.677066+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.860501+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:14:17.860501+0000","last_peered":"2024-01-15T18:14:17.860501+0000","last_clean":"2024-01-15T18:14:17.860501+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:14:17.860501+0000","last_undegraded":"2024-01-15T18:14:17.860501+0000","last_fullsized":"2024-01-15T18:14:17.860501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678418+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:14:18.678418+0000","last_peered":"2024-01-15T18:14:18.678418+0000","last_clean":"2024-01-15T18:14:18.678418+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:14:18.678418+0000","last_undegraded":"2024-01-15T18:14:18.678418+0000","last_fullsized":"2024-01-15T18:14:18.678418+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678982+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:14:18.678982+0000","last_peered":"2024-01-15T18:14:18.678982+0000","last_clean":"2024-01-15T18:14:18.678982+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:14:18.678982+0000","last_undegraded":"2024-01-15T18:14:18.678982+0000","last_fullsized":"2024-01-15T18:14:18.678982+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:18.678496+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:14:18.678496+0000","last_peered":"2024-01-15T18:14:18.678496+0000","last_clean":"2024-01-15T18:14:18.678496+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:14:18.678496+0000","last_undegraded":"2024-01-15T18:14:18.678496+0000","last_fullsized":"2024-01-15T18:14:18.678496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":12,"reported_epoch":114,"state":"active+clean","last_fresh":"2024-01-15T18:14:17.867951+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:14:17.867951+0000","last_peered":"2024-01-15T18:14:17.867951+0000","last_clean":"2024-01-15T18:14:17.867951+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:14:17.867951+0000","last_undegraded":"2024-01-15T18:14:17.867951+0000","last_fullsized":"2024-01-15T18:14:17.867951+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":249,"reported_epoch":113,"state":"active+clean","last_fresh":"2024-01-15T18:14:16.880953+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:14:16.880953+0000","last_peered":"2024-01-15T18:14:16.880953+0000","last_clean":"2024-01-15T18:14:16.880953+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:14:16.880953+0000","last_undegraded":"2024-01-15T18:14:16.880953+0000","last_fullsized":"2024-01-15T18:14:16.880953+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542350,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65024,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306816,"statfs":{"total":96636764160,"available":96570179584,"internally_reserved":0,"allocated":2015232,"data_stored":1210701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64551370},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.268,"15min":1.4239999999999999},"min":{"1min":0.746,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.722,"5min":1.9359999999999999,"15min":14.114000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":1.494,"5min":1.421,"15min":1.5680000000000001},"min":{"1min":1.0700000000000001,"5min":0.78800000000000003,"15min":0.51700000000000002},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":27.32},"last":1.1579999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.2809999999999999,"15min":1.52},"min":{"1min":0.85799999999999998,"5min":0.68799999999999994,"15min":0.40300000000000002},"max":{"1min":1.5580000000000001,"5min":5.9690000000000003,"15min":29.167999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.325,"5min":1.288,"15min":1.4239999999999999},"min":{"1min":0.89200000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.7749999999999999,"5min":2.1110000000000002,"15min":12.565},"last":0.95699999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.3109999999999999,"15min":1.526},"min":{"1min":0.86199999999999999,"5min":0.61399999999999999,"15min":0.56000000000000005},"max":{"1min":1.8360000000000001,"5min":2.7360000000000002,"15min":13.801},"last":1.073},{"interface":"front","average":{"1min":1.387,"5min":1.361,"15min":1.52},"min":{"1min":0.96399999999999997,"5min":0.70099999999999996,"15min":0.56599999999999995},"max":{"1min":2.2160000000000002,"5min":3.5920000000000001,"15min":27.358000000000001},"last":0.72499999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.409,"15min":1.629},"min":{"1min":0.99399999999999999,"5min":0.83099999999999996,"15min":0.47099999999999997},"max":{"1min":1.8109999999999999,"5min":6.133,"15min":29.204999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.413,"5min":1.4410000000000001,"15min":1.5900000000000001},"min":{"1min":0.91400000000000003,"5min":0.57799999999999996,"15min":0.57299999999999995},"max":{"1min":2.173,"5min":2.617,"15min":16.088999999999999},"last":1.224}]},{"osd":4,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.3360000000000001,"15min":1.544},"min":{"1min":0.91300000000000003,"5min":0.78900000000000003,"15min":0.50700000000000001},"max":{"1min":2.0329999999999999,"5min":2.2599999999999998,"15min":27.219000000000001},"last":1.284},{"interface":"front","average":{"1min":1.367,"5min":1.5069999999999999,"15min":1.6859999999999999},"min":{"1min":0.94799999999999995,"5min":0.88100000000000001,"15min":0.438},"max":{"1min":1.6850000000000001,"5min":6.4710000000000001,"15min":29.25},"last":1.0129999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4299999999999999,"15min":1.643},"min":{"1min":0.90800000000000003,"5min":0.79100000000000004,"15min":0.39000000000000001},"max":{"1min":1.77,"5min":2.23,"15min":12.518000000000001},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.524,"5min":1.51,"15min":1.669},"min":{"1min":1.0580000000000001,"5min":0.90800000000000003,"15min":0.41099999999999998},"max":{"1min":2.3889999999999998,"5min":2.4489999999999998,"15min":27.283000000000001},"last":1.403}]},{"osd":6,"last update":"Mon Jan 15 18:14:06 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.615,"15min":2.3260000000000001},"min":{"1min":1.153,"5min":1.0169999999999999,"15min":0.45600000000000002},"max":{"1min":2.4870000000000001,"5min":2.4870000000000001,"15min":53.664999999999999},"last":1.46},{"interface":"front","average":{"1min":1.462,"5min":1.609,"15min":1.8460000000000001},"min":{"1min":1.032,"5min":1.032,"15min":0.44},"max":{"1min":1.7549999999999999,"5min":6.1870000000000003,"15min":29.106999999999999},"last":1.1140000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542350,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64440,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307400,"statfs":{"total":96636764160,"available":96570777600,"internally_reserved":0,"allocated":1417216,"data_stored":619787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.242,"15min":1.24},"min":{"1min":0.71699999999999997,"5min":0.56899999999999995,"15min":0.438},"max":{"1min":1.708,"5min":8.8350000000000009,"15min":8.8350000000000009},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.25,"15min":1.3620000000000001},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.59599999999999997},"max":{"1min":1.982,"5min":3.1659999999999999,"15min":21.565000000000001},"last":1.2509999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.494,"15min":1.4159999999999999},"min":{"1min":0.86599999999999999,"5min":0.64000000000000001,"15min":0.52100000000000002},"max":{"1min":13.269,"5min":13.269,"15min":13.269},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.681,"5min":1.3360000000000001,"15min":1.429},"min":{"1min":0.93300000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":6.335,"5min":6.335,"15min":13.984999999999999},"last":0.76700000000000002}]},{"osd":2,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.25,"15min":1.45},"min":{"1min":0.80800000000000005,"5min":0.69799999999999995,"15min":0.54300000000000004},"max":{"1min":1.7110000000000001,"5min":2.919,"15min":34.670000000000002},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.405,"5min":1.429,"15min":1.446},"min":{"1min":0.90700000000000003,"5min":0.71599999999999997,"15min":0.49399999999999999},"max":{"1min":1.9159999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"last":1.093}]},{"osd":3,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.482,"15min":1.5820000000000001},"min":{"1min":1.1990000000000001,"5min":0.83599999999999997,"15min":0.68500000000000005},"max":{"1min":2.6640000000000001,"5min":3.5680000000000001,"15min":16.855},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.4239999999999999,"15min":1.6870000000000001},"min":{"1min":0.92000000000000004,"5min":0.77400000000000002,"15min":0.48499999999999999},"max":{"1min":1.883,"5min":2.9630000000000001,"15min":34.725000000000001},"last":1.5760000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.458,"15min":1.4650000000000001},"min":{"1min":0.96299999999999997,"5min":0.621,"15min":0.38400000000000001},"max":{"1min":1.996,"5min":8.7539999999999996,"15min":8.7539999999999996},"last":1.21},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.504,"15min":1.51},"min":{"1min":0.88200000000000001,"5min":0.54900000000000004,"15min":0.501},"max":{"1min":2.8340000000000001,"5min":7.3230000000000004,"15min":10.118},"last":1.3240000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.5309999999999999,"15min":1.708},"min":{"1min":0.98899999999999999,"5min":0.67000000000000004,"15min":0.57699999999999996},"max":{"1min":2.0070000000000001,"5min":16.309999999999999,"15min":34.798000000000002},"last":1.476},{"interface":"front","average":{"1min":1.55,"5min":1.6599999999999999,"15min":1.635},"min":{"1min":1.0840000000000001,"5min":0.69399999999999995,"15min":0.47899999999999998},"max":{"1min":2.0880000000000001,"5min":8.9900000000000002,"15min":10.077},"last":1.298}]},{"osd":7,"last update":"Mon Jan 15 18:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.6439999999999999,"15min":1.6379999999999999},"min":{"1min":1.208,"5min":0.72299999999999998,"15min":0.64000000000000001},"max":{"1min":2.7549999999999999,"5min":5.5060000000000002,"15min":10.302},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.611,"5min":1.556,"15min":1.8129999999999999},"min":{"1min":1.1339999999999999,"5min":0.58399999999999996,"15min":0.52600000000000002},"max":{"1min":2.9540000000000002,"5min":2.9540000000000002,"15min":34.762999999999998},"last":1.6950000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542352,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58944,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312896,"statfs":{"total":96636764160,"available":96576405504,"internally_reserved":0,"allocated":1421312,"data_stored":620321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.2450000000000001,"15min":1.403},"min":{"1min":0.67800000000000005,"5min":0.63800000000000001,"15min":0.52700000000000002},"max":{"1min":5.3410000000000002,"5min":5.3410000000000002,"15min":26.077999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.4399999999999999,"15min":1.5189999999999999},"min":{"1min":0.875,"5min":0.56399999999999995,"15min":0.52400000000000002},"max":{"1min":5.3949999999999996,"5min":9.8699999999999992,"15min":14.711},"last":0.77900000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.282,"15min":1.4339999999999999},"min":{"1min":0.97699999999999998,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.9730000000000001,"5min":2.8279999999999998,"15min":26},"last":1.194},{"interface":"front","average":{"1min":1.429,"5min":1.282,"15min":1.3959999999999999},"min":{"1min":0.85599999999999998,"5min":0.58999999999999997,"15min":0.54200000000000004},"max":{"1min":1.931,"5min":2.8660000000000001,"15min":12.619},"last":1.335}]},{"osd":2,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.3680000000000001,"15min":1.5429999999999999},"min":{"1min":0.93500000000000005,"5min":0.56699999999999995,"15min":0.55700000000000005},"max":{"1min":2.8029999999999999,"5min":4.8719999999999999,"15min":26.234999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.4650000000000001,"15min":1.6220000000000001},"min":{"1min":0.96199999999999997,"5min":0.629,"15min":0.59099999999999997},"max":{"1min":1.9910000000000001,"5min":10.09,"15min":17.126999999999999},"last":1.095}]},{"osd":3,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2709999999999999,"15min":1.454},"min":{"1min":0.97999999999999998,"5min":0.67200000000000004,"15min":0.622},"max":{"1min":1.802,"5min":1.841,"15min":12.722},"last":1.139},{"interface":"front","average":{"1min":1.6419999999999999,"5min":1.4279999999999999,"15min":1.6499999999999999},"min":{"1min":1.143,"5min":0.67900000000000005,"15min":0.63400000000000001},"max":{"1min":2.9060000000000001,"5min":2.9060000000000001,"15min":26.134},"last":1.288}]},{"osd":4,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.3740000000000001,"15min":1.544},"min":{"1min":1.111,"5min":0.442,"15min":0.442},"max":{"1min":2.2869999999999999,"5min":9.9979999999999993,"15min":25.776},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.583,"5min":1.5509999999999999,"15min":1.768},"min":{"1min":1.214,"5min":0.498,"15min":0.47799999999999998},"max":{"1min":2.1030000000000002,"5min":12.103,"15min":16.414000000000001},"last":1.425}]},{"osd":6,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.337,"15min":1.534},"min":{"1min":1.071,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.994,"5min":1.994,"15min":12.769},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.673,"5min":1.4630000000000001,"15min":1.798},"min":{"1min":1.1779999999999999,"5min":0.55500000000000005,"15min":0.435},"max":{"1min":3.0139999999999998,"5min":3.0139999999999998,"15min":36.087000000000003},"last":1.054}]},{"osd":7,"last update":"Mon Jan 15 18:13:51 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.661,"15min":1.921},"min":{"1min":1.319,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.1930000000000001,"5min":12.326000000000001,"15min":13.706},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":1.482,"15min":1.766},"min":{"1min":1.1719999999999999,"5min":0.73599999999999999,"15min":0.60099999999999998},"max":{"1min":3.1139999999999999,"5min":3.1139999999999999,"15min":21.728000000000002},"last":1.3999999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542351,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64432,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307408,"statfs":{"total":96636764160,"available":96570785792,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1379999999999999,"15min":1.3779999999999999},"min":{"1min":0.63500000000000001,"5min":0.55700000000000005,"15min":0.31900000000000001},"max":{"1min":1.637,"5min":3.4100000000000001,"15min":21.643000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.181,"5min":1.1899999999999999,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.5349999999999999,"5min":2.1240000000000001,"15min":11.196},"last":1.419}]},{"osd":1,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.204,"15min":1.425},"min":{"1min":0.80300000000000005,"5min":0.59299999999999997,"15min":0.32200000000000001},"max":{"1min":1.8740000000000001,"5min":2.7589999999999999,"15min":28.663},"last":1.008},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.54},"min":{"1min":0.82899999999999996,"5min":0.72599999999999998,"15min":0.35399999999999998},"max":{"1min":1.8680000000000001,"5min":3.5299999999999998,"15min":21.440000000000001},"last":1.7969999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.3,"15min":1.4950000000000001},"min":{"1min":0.78300000000000003,"5min":0.48799999999999999,"15min":0.35599999999999998},"max":{"1min":2.032,"5min":2.7050000000000001,"15min":16.023},"last":1.389},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.5609999999999999,"15min":1.607},"min":{"1min":0.93600000000000005,"5min":0.751,"15min":0.38100000000000001},"max":{"1min":1.748,"5min":18.858000000000001,"15min":22.350999999999999},"last":1.544}]},{"osd":3,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.2969999999999999,"15min":1.5149999999999999},"min":{"1min":1.1180000000000001,"5min":0.55600000000000005,"15min":0.48799999999999999},"max":{"1min":1.9890000000000001,"5min":2.1890000000000001,"15min":15.093},"last":1.048},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4239999999999999,"15min":1.6879999999999999},"min":{"1min":1.0309999999999999,"5min":0.54600000000000004,"15min":0.42199999999999999},"max":{"1min":1.9910000000000001,"5min":2.585,"15min":19.486000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.391,"15min":1.776},"min":{"1min":0.70999999999999996,"5min":0.5,"15min":0.41499999999999998},"max":{"1min":1.996,"5min":3.6429999999999998,"15min":54.938000000000002},"last":1.74},{"interface":"front","average":{"1min":1.452,"5min":1.361,"15min":1.5249999999999999},"min":{"1min":0.99199999999999999,"5min":0.53400000000000003,"15min":0.34000000000000002},"max":{"1min":2.2349999999999999,"5min":2.6030000000000002,"15min":18.289999999999999},"last":1.3480000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.462,"5min":1.5069999999999999,"15min":1.649},"min":{"1min":0.94599999999999995,"5min":0.67700000000000005,"15min":0.38700000000000001},"max":{"1min":2.375,"5min":2.9820000000000002,"15min":17.733000000000001},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.5680000000000001,"15min":1.877},"min":{"1min":1.0029999999999999,"5min":0.621,"15min":0.499},"max":{"1min":2.3079999999999998,"5min":3.7280000000000002,"15min":28.797000000000001},"last":1.8540000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5169999999999999,"15min":1.8260000000000001},"min":{"1min":0.93799999999999994,"5min":0.55800000000000005,"15min":0.45100000000000001},"max":{"1min":1.9710000000000001,"5min":2.5419999999999998,"15min":23.390000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.633,"15min":1.8879999999999999},"min":{"1min":1.163,"5min":0.65200000000000002,"15min":0.51000000000000001},"max":{"1min":2.3599999999999999,"5min":2.851,"15min":22.751000000000001},"last":1.607}]}]},{"osd":3,"up_from":14,"seq":60129542352,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72768,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299072,"statfs":{"total":96636764160,"available":96562249728,"internally_reserved":0,"allocated":1429504,"data_stored":621399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":73071048},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":0.98299999999999998,"15min":1.1060000000000001},"min":{"1min":0.442,"5min":0.372,"15min":0.29799999999999999},"max":{"1min":2.0099999999999998,"5min":3.6960000000000002,"15min":36.408000000000001},"last":1.419},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.024,"15min":1.0649999999999999},"min":{"1min":0.69499999999999995,"5min":0.42499999999999999,"15min":0.377},"max":{"1min":1.732,"5min":1.732,"15min":5.0529999999999999},"last":1.208}]},{"osd":1,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.03,"15min":1.109},"min":{"1min":0.53700000000000003,"5min":0.41999999999999998,"15min":0.34399999999999997},"max":{"1min":1.72,"5min":1.9590000000000001,"15min":8.1359999999999992},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.2350000000000001,"15min":1.214},"min":{"1min":0.65600000000000003,"5min":0.56899999999999995,"15min":0.40000000000000002},"max":{"1min":1.8600000000000001,"5min":2.5790000000000002,"15min":7.2169999999999996},"last":1.133}]},{"osd":2,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.306,"15min":1.329},"min":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.44600000000000001},"max":{"1min":7.0999999999999996,"5min":7.0999999999999996,"15min":14.025},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.347,"15min":1.3180000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.46000000000000002},"max":{"1min":1.827,"5min":2.7130000000000001,"15min":5.0910000000000002},"last":1.2390000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.399,"15min":1.46},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.59999999999999998},"max":{"1min":1.8879999999999999,"5min":6.7720000000000002,"15min":17.082000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.8819999999999999,"5min":1.6080000000000001,"15min":1.6259999999999999},"min":{"1min":1.105,"5min":0.92900000000000005,"15min":0.69499999999999995},"max":{"1min":9.8040000000000003,"5min":9.8040000000000003,"15min":15.769},"last":1.181}]},{"osd":5,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.581,"15min":1.6080000000000001},"min":{"1min":1.046,"5min":0.85599999999999998,"15min":0.65000000000000002},"max":{"1min":2.1389999999999998,"5min":2.7440000000000002,"15min":5.5220000000000002},"last":1.341},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.581,"15min":1.9430000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.58299999999999996},"max":{"1min":1.9550000000000001,"5min":4.3390000000000004,"15min":40.384999999999998},"last":1.5780000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.571,"15min":1.7130000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.51400000000000001},"max":{"1min":2.7629999999999999,"5min":3.6800000000000002,"15min":32.259999999999998},"last":1.387},{"interface":"front","average":{"1min":1.679,"5min":1.6799999999999999,"15min":1.7609999999999999},"min":{"1min":1.1579999999999999,"5min":1.153,"15min":0.747},"max":{"1min":2.2519999999999998,"5min":3.9670000000000001,"15min":16.195},"last":1.4790000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:13:55 2024","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.7490000000000001,"15min":1.913},"min":{"1min":1.026,"5min":0.95599999999999996,"15min":0.63900000000000001},"max":{"1min":2.0720000000000001,"5min":10.035,"15min":24.146999999999998},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.492,"5min":1.6100000000000001,"15min":1.774},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.56100000000000005},"max":{"1min":2.0270000000000001,"5min":3.4700000000000002,"15min":17.622},"last":1.5469999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542351,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72752,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299088,"statfs":{"total":96636764160,"available":96562266112,"internally_reserved":0,"allocated":1413120,"data_stored":619717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":1.0309999999999999,"15min":1.0129999999999999},"min":{"1min":0.60599999999999998,"5min":0.373,"15min":0.30099999999999999},"max":{"1min":1.641,"5min":10.256,"15min":10.256},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.0960000000000001,"15min":1.1020000000000001},"min":{"1min":0.77000000000000002,"5min":0.35299999999999998,"15min":0.28299999999999997},"max":{"1min":1.835,"5min":2.4020000000000001,"15min":10.319000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0309999999999999,"15min":1.119},"min":{"1min":0.68799999999999994,"5min":0.443,"15min":0.38600000000000001},"max":{"1min":1.6040000000000001,"5min":3.1360000000000001,"15min":10.807},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.2290000000000001,"15min":1.2250000000000001},"min":{"1min":0.95199999999999996,"5min":0.47099999999999997,"15min":0.35299999999999998},"max":{"1min":2.1429999999999998,"5min":4.7060000000000004,"15min":7.5119999999999996},"last":1.214}]},{"osd":3,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.2589999999999999,"15min":1.4690000000000001},"min":{"1min":0.82099999999999995,"5min":0.55800000000000005,"15min":0.41399999999999998},"max":{"1min":1.6879999999999999,"5min":4.3970000000000002,"15min":25.891999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.266,"15min":1.302},"min":{"1min":0.67100000000000004,"5min":0.49399999999999999,"15min":0.376},"max":{"1min":1.927,"5min":6.7229999999999999,"15min":14.946},"last":1.1519999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4119999999999999,"15min":1.516},"min":{"1min":0.98599999999999999,"5min":0.73199999999999998,"15min":0.64900000000000002},"max":{"1min":1.837,"5min":4.4649999999999999,"15min":9.0679999999999996},"last":1.494},{"interface":"front","average":{"1min":1.542,"5min":1.5389999999999999,"15min":1.591},"min":{"1min":1.1220000000000001,"5min":0.65300000000000002,"15min":0.54500000000000004},"max":{"1min":2.3540000000000001,"5min":8.4269999999999996,"15min":10.279999999999999},"last":1.764}]},{"osd":5,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.468,"15min":1.6679999999999999},"min":{"1min":1.016,"5min":0.67100000000000004,"15min":0.63500000000000001},"max":{"1min":1.7809999999999999,"5min":4.7759999999999998,"15min":11.071999999999999},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.486,"5min":1.534,"15min":1.7230000000000001},"min":{"1min":1.012,"5min":0.89700000000000002,"15min":0.55200000000000005},"max":{"1min":2.0609999999999999,"5min":4.5579999999999998,"15min":37.020000000000003},"last":1.6850000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6559999999999999,"5min":1.6539999999999999,"15min":1.7170000000000001},"min":{"1min":1.3480000000000001,"5min":0.79100000000000004,"15min":0.64800000000000002},"max":{"1min":2.5550000000000002,"5min":8.5519999999999996,"15min":23.762},"last":1.333},{"interface":"front","average":{"1min":1.5740000000000001,"5min":1.625,"15min":1.911},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.73499999999999999},"max":{"1min":2.004,"5min":4.819,"15min":39.926000000000002},"last":1.534}]},{"osd":7,"last update":"Mon Jan 15 18:13:48 2024","interfaces":[{"interface":"back","average":{"1min":1.712,"5min":1.7410000000000001,"15min":1.958},"min":{"1min":1.1699999999999999,"5min":0.84299999999999997,"15min":0.68999999999999995},"max":{"1min":2.633,"5min":8.6620000000000008,"15min":52.767000000000003},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.623,"5min":1.6679999999999999,"15min":1.833},"min":{"1min":1.0880000000000001,"5min":0.877,"15min":0.83499999999999996},"max":{"1min":1.9219999999999999,"5min":4.8620000000000001,"15min":13.055999999999999},"last":1.3879999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542351,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72760,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299080,"statfs":{"total":96636764160,"available":96562257920,"internally_reserved":0,"allocated":1421312,"data_stored":619841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.1399999999999999,"15min":1.111},"min":{"1min":0.70499999999999996,"5min":0.36499999999999999,"15min":0.36199999999999999},"max":{"1min":11.964,"5min":11.964,"15min":19.218},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.901,"5min":1.2509999999999999,"15min":1.248},"min":{"1min":0.64700000000000002,"5min":0.42199999999999999,"15min":0.38700000000000001},"max":{"1min":15.176,"5min":15.176,"15min":19.823},"last":1.488}]},{"osd":2,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.226,"15min":1.2},"min":{"1min":0.68200000000000005,"5min":0.376,"15min":0.31900000000000001},"max":{"1min":19.02,"5min":19.02,"15min":19.02},"last":0.84099999999999997},{"interface":"front","average":{"1min":2.6749999999999998,"5min":1.4930000000000001,"15min":1.385},"min":{"1min":0.58899999999999997,"5min":0.41599999999999998,"15min":0.32500000000000001},"max":{"1min":29.251000000000001,"5min":29.251000000000001,"15min":29.251000000000001},"last":0.71899999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.6840000000000002,"5min":1.5960000000000001,"15min":1.4870000000000001},"min":{"1min":0.65900000000000003,"5min":0.51100000000000001,"15min":0.40899999999999997},"max":{"1min":27.021000000000001,"5min":27.021000000000001,"15min":27.021000000000001},"last":1.0940000000000001},{"interface":"front","average":{"1min":2.7610000000000001,"5min":1.49,"15min":1.4430000000000001},"min":{"1min":0.75700000000000001,"5min":0.43099999999999999,"15min":0.34799999999999998},"max":{"1min":31.218,"5min":31.218,"15min":31.218},"last":1.2829999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.8170000000000002,"5min":1.8500000000000001,"15min":1.6910000000000001},"min":{"1min":1.0429999999999999,"5min":0.77000000000000002,"15min":0.497},"max":{"1min":27.059999999999999,"5min":27.059999999999999,"15min":27.059999999999999},"last":1.2330000000000001},{"interface":"front","average":{"1min":3.0169999999999999,"5min":1.772,"15min":1.764},"min":{"1min":1.0669999999999999,"5min":0.71599999999999997,"15min":0.44800000000000001},"max":{"1min":31.433,"5min":31.433,"15min":31.433},"last":1.5509999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":2.2749999999999999,"5min":1.629,"15min":1.726},"min":{"1min":1.1870000000000001,"5min":0.68500000000000005,"15min":0.48299999999999998},"max":{"1min":11.994,"5min":11.994,"15min":20.012},"last":1.409},{"interface":"front","average":{"1min":1.722,"5min":1.53,"15min":1.6719999999999999},"min":{"1min":0.89900000000000002,"5min":0.71599999999999997,"15min":0.499},"max":{"1min":3.9910000000000001,"5min":3.9910000000000001,"15min":19.998999999999999},"last":0.99399999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.1400000000000001,"5min":1.893,"15min":1.825},"min":{"1min":1.1899999999999999,"5min":0.85899999999999999,"15min":0.54100000000000004},"max":{"1min":29.530000000000001,"5min":29.530000000000001,"15min":29.530000000000001},"last":1.3400000000000001},{"interface":"front","average":{"1min":3.3340000000000001,"5min":1.9019999999999999,"15min":1.931},"min":{"1min":1.165,"5min":0.61399999999999999,"15min":0.51600000000000001},"max":{"1min":31.186,"5min":31.186,"15min":31.186},"last":1.611}]},{"osd":7,"last update":"Mon Jan 15 18:14:01 2024","interfaces":[{"interface":"back","average":{"1min":3.0289999999999999,"5min":1.8109999999999999,"15min":1.9239999999999999},"min":{"1min":1.2809999999999999,"5min":0.81699999999999995,"15min":0.505},"max":{"1min":26.393999999999998,"5min":26.393999999999998,"15min":26.393999999999998},"last":1.468},{"interface":"front","average":{"1min":2.25,"5min":1.728,"15min":1.887},"min":{"1min":1.145,"5min":0.92800000000000005,"15min":0.54500000000000004},"max":{"1min":9.7799999999999994,"5min":9.7799999999999994,"15min":27.582000000000001},"last":1.3799999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542350,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73360,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298480,"statfs":{"total":96636764160,"available":96561643520,"internally_reserved":0,"allocated":2027520,"data_stored":1211809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.177,"15min":1.2410000000000001},"min":{"1min":0.60799999999999998,"5min":0.38100000000000001,"15min":0.30599999999999999},"max":{"1min":12.353,"5min":14.542,"15min":20.437999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.198,"5min":1.0800000000000001,"15min":1.2190000000000001},"min":{"1min":0.628,"5min":0.33800000000000002,"15min":0.32400000000000001},"max":{"1min":1.704,"5min":1.9430000000000001,"15min":35.137999999999998},"last":1.079}]},{"osd":2,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.282,"15min":1.373},"min":{"1min":0.75700000000000001,"5min":0.44500000000000001,"15min":0.42799999999999999},"max":{"1min":2.6120000000000001,"5min":14.651,"15min":20.556000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.1990000000000001,"15min":1.387},"min":{"1min":0.58099999999999996,"5min":0.44400000000000001,"15min":0.39000000000000001},"max":{"1min":2.9239999999999999,"5min":2.9239999999999999,"15min":35.183},"last":1.671}]},{"osd":3,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.087,"15min":1.163},"min":{"1min":0.85699999999999998,"5min":0.42999999999999999,"15min":0.41699999999999998},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":8.9949999999999992},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.5189999999999999,"15min":1.651},"min":{"1min":0.82899999999999996,"5min":0.623,"15min":0.45100000000000001},"max":{"1min":2.7269999999999999,"5min":14.598000000000001,"15min":25.372},"last":1.526}]},{"osd":4,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.4039999999999999,"15min":1.6850000000000001},"min":{"1min":1.2110000000000001,"5min":0.68100000000000005,"15min":0.67200000000000004},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":35.223999999999997},"last":1.95},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3160000000000001,"15min":1.4530000000000001},"min":{"1min":1.0009999999999999,"5min":0.81200000000000006,"15min":0.58799999999999997},"max":{"1min":1.851,"5min":2.1179999999999999,"15min":15.968999999999999},"last":1.3129999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.7070000000000001,"5min":1.516,"15min":1.645},"min":{"1min":1.0529999999999999,"5min":0.78600000000000003,"15min":0.63800000000000001},"max":{"1min":5.0800000000000001,"5min":5.0800000000000001,"15min":12.173999999999999},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.6719999999999999,"5min":1.6990000000000001,"15min":1.865},"min":{"1min":1.236,"5min":0.88300000000000001,"15min":0.76000000000000001},"max":{"1min":2.9049999999999998,"5min":14.859,"15min":20.789999999999999},"last":1.772}]},{"osd":6,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.7130000000000001,"15min":2.0539999999999998},"min":{"1min":1.1950000000000001,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.7989999999999999,"5min":15.022,"15min":36.390999999999998},"last":1.899},{"interface":"front","average":{"1min":1.8899999999999999,"5min":1.583,"15min":1.9430000000000001},"min":{"1min":0.95299999999999996,"5min":0.70299999999999996,"15min":0.62},"max":{"1min":6.8710000000000004,"5min":6.8710000000000004,"15min":36.207000000000001},"last":1.835}]},{"osd":7,"last update":"Mon Jan 15 18:14:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.645,"15min":1.96},"min":{"1min":0.91200000000000003,"5min":0.84899999999999998,"15min":0.56200000000000006},"max":{"1min":6.7969999999999997,"5min":6.7969999999999997,"15min":35.636000000000003},"last":1.399},{"interface":"front","average":{"1min":1.583,"5min":1.528,"15min":1.675},"min":{"1min":1.101,"5min":0.70499999999999996,"15min":0.64000000000000001},"max":{"1min":2.0840000000000001,"5min":2.25,"15min":11.470000000000001},"last":1.6120000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:14:26.938 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-15T18:14:26.938 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-15T18:14:27.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.202+0000 7fb2e1600640 1 Processor -- start 2024-01-15T18:14:27.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.202+0000 7fb2e1600640 1 -- start start 2024-01-15T18:14:27.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.203+0000 7fb2e1600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb2dc15c450 0x7fb2dc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.203+0000 7fb2e1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 0x7fb2dc070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.203+0000 7fb2e1600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0ce230 con 0x7fb2dc144c00 2024-01-15T18:14:27.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.204+0000 7fb2e1600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc06d2c0 con 0x7fb2dc15c450 2024-01-15T18:14:27.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.204+0000 7fb2e1600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0ce510 con 0x7fb2dc06ce30 2024-01-15T18:14:27.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.204+0000 7fb2db400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 0x7fb2dc070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.204+0000 7fb2d3e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb2dc15c450 0x7fb2dc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.204+0000 7fb2db400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 0x7fb2dc070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:59520/0 (socket says 172.21.15.2:59520) 2024-01-15T18:14:27.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.204+0000 7fb2db400640 1 -- 172.21.15.2:0/653463318 learned_addr learned my addr 172.21.15.2:0/653463318 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:27.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.205+0000 7fb2db400640 1 -- 172.21.15.2:0/653463318 >> v2:172.21.15.137:3300/0 conn(0x7fb2dc15c450 msgr2=0x7fb2dc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:27.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.205+0000 7fb2db400640 1 --2- 172.21.15.2:0/653463318 >> v2:172.21.15.137:3300/0 conn(0x7fb2dc15c450 0x7fb2dc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.205+0000 7fb2db400640 1 -- 172.21.15.2:0/653463318 >> v1:172.21.15.137:6789/0 conn(0x7fb2dc144c00 legacy=0x7fb2dc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.205+0000 7fb2db400640 1 -- 172.21.15.2:0/653463318 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc0ce7f0 con 0x7fb2dc06ce30 2024-01-15T18:14:27.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.206+0000 7fb2db400640 1 --2- 172.21.15.2:0/653463318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 0x7fb2dc070b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e50 tx=0x7fb2d402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0d49a8d28b848d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.211 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.207+0000 7fb2da000640 1 -- 172.21.15.2:0/653463318 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d402cb00 con 0x7fb2dc06ce30 2024-01-15T18:14:27.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.207+0000 7fb2da000640 1 -- 172.21.15.2:0/653463318 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d402cc60 con 0x7fb2dc06ce30 2024-01-15T18:14:27.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.207+0000 7fb2da000640 1 -- 172.21.15.2:0/653463318 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d40343f0 con 0x7fb2dc06ce30 2024-01-15T18:14:27.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.207+0000 7fb2e1600640 1 -- 172.21.15.2:0/653463318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 msgr2=0x7fb2dc070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.207+0000 7fb2e1600640 1 --2- 172.21.15.2:0/653463318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 0x7fb2dc070b10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e50 tx=0x7fb2d402bf60 comp rx=0 tx=0).stop 2024-01-15T18:14:27.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.208+0000 7fb2e1600640 1 -- 172.21.15.2:0/653463318 shutdown_connections 2024-01-15T18:14:27.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.208+0000 7fb2e1600640 1 --2- 172.21.15.2:0/653463318 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc06ce30 0x7fb2dc070b10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.208+0000 7fb2e1600640 1 --2- 172.21.15.2:0/653463318 >> v2:172.21.15.137:3300/0 conn(0x7fb2dc15c450 0x7fb2dc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.208+0000 7fb2e1600640 1 -- 172.21.15.2:0/653463318 >> 172.21.15.2:0/653463318 conn(0x7fb2dc0b45f0 msgr2=0x7fb2dc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:27.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.208+0000 7fb2e1600640 1 -- 172.21.15.2:0/653463318 shutdown_connections 2024-01-15T18:14:27.213 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.208+0000 7fb2e1600640 1 -- 172.21.15.2:0/653463318 wait complete. 2024-01-15T18:14:27.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.210+0000 7fb2e1600640 1 Processor -- start 2024-01-15T18:14:27.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.210+0000 7fb2e1600640 1 -- start start 2024-01-15T18:14:27.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2e1600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc06ce30 0x7fb2dc13df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2e1600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc144c00 0x7fb2dc13e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2e1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 0x7fb2dc13ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.217 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2e1600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0b3c50 con 0x7fb2dc15c450 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2e1600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc137700 con 0x7fb2dc06ce30 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2e1600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc137970 con 0x7fb2dc144c00 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2db400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 0x7fb2dc13ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2db400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 0x7fb2dc13ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53018/0 (socket says 172.21.15.2:53018) 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2db400640 1 -- 172.21.15.2:0/872723445 learned_addr learned my addr 172.21.15.2:0/872723445 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2daa00640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc06ce30 0x7fb2dc13df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.218 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.211+0000 7fb2d3e00640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc144c00 0x7fb2dc13e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2db400640 1 -- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc144c00 msgr2=0x7fb2dc13e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2db400640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc144c00 0x7fb2dc13e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2db400640 1 -- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc06ce30 msgr2=0x7fb2dc13df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2db400640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc06ce30 0x7fb2dc13df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2db400640 1 -- 172.21.15.2:0/872723445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d4009b30 con 0x7fb2dc15c450 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2daa00640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc06ce30 0x7fb2dc13df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2db400640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 0x7fb2dc13ec50 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e50 tx=0x7fb2d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4009510 con 0x7fb2dc15c450 2024-01-15T18:14:27.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d403e450 con 0x7fb2dc15c450 2024-01-15T18:14:27.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d403e7e0 con 0x7fb2dc15c450 2024-01-15T18:14:27.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.212+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc137c50 con 0x7fb2dc15c450 2024-01-15T18:14:27.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.213+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc138210 con 0x7fb2dc15c450 2024-01-15T18:14:27.220 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.214+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc0686f0 con 0x7fb2dc15c450 2024-01-15T18:14:27.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.215+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fb2d403e940 con 0x7fb2dc15c450 2024-01-15T18:14:27.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.216+0000 7fb2d2a00640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2b406c430 0x7fb2b406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.216+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fb2d40ab790 con 0x7fb2dc15c450 2024-01-15T18:14:27.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.216+0000 7fb2daa00640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2b406c430 0x7fb2b406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.217+0000 7fb2daa00640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2b406c430 0x7fb2b406e8f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0b1bd0 tx=0x7fb2cc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.220+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2d4077d00 con 0x7fb2dc15c450 2024-01-15T18:14:27.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.472+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb2dc13e4f0 con 0x7fb2dc15c450 2024-01-15T18:14:27.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.477+0000 7fb2d2a00640 1 -- 172.21.15.2:0/872723445 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb2d4043020 con 0x7fb2dc15c450 2024-01-15T18:14:27.480 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-01-15T18:14:27.480 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQDRcaVlB5dYLRAAysPf7YLiaT4WtmYstwzy+w== 2024-01-15T18:14:27.480 INFO:teuthology.orchestra.run.smithi002.stdout: caps mds = "allow" 2024-01-15T18:14:27.481 INFO:teuthology.orchestra.run.smithi002.stdout: caps mon = "allow r" 2024-01-15T18:14:27.481 INFO:teuthology.orchestra.run.smithi002.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-15T18:14:27.481 INFO:teuthology.orchestra.run.smithi002.stderr:updated caps for client.0 2024-01-15T18:14:27.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.479+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2b406c430 msgr2=0x7fb2b406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.479+0000 7fb2e1600640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2b406c430 0x7fb2b406e8f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0b1bd0 tx=0x7fb2cc009040 comp rx=0 tx=0).stop 2024-01-15T18:14:27.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.479+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 msgr2=0x7fb2dc13ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.479+0000 7fb2e1600640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 0x7fb2dc13ec50 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e50 tx=0x7fb2d4004a40 comp rx=0 tx=0).stop 2024-01-15T18:14:27.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 shutdown_connections 2024-01-15T18:14:27.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb2b406c430 0x7fb2b406e8f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2dc15c450 0x7fb2dc13ec50 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb2dc144c00 0x7fb2dc13e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 --2- 172.21.15.2:0/872723445 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2dc06ce30 0x7fb2dc13df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 >> 172.21.15.2:0/872723445 conn(0x7fb2dc0b45f0 msgr2=0x7fb2dc15db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 shutdown_connections 2024-01-15T18:14:27.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.480+0000 7fb2e1600640 1 -- 172.21.15.2:0/872723445 wait complete. 2024-01-15T18:14:27.504 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:14:27.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.712+0000 7f4dea400640 1 Processor -- start 2024-01-15T18:14:27.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.713+0000 7f4dea400640 1 -- start start 2024-01-15T18:14:27.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4dea400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 0x7f4de415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4dea400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4de406cad0 0x7f4de406cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4dea400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40716d0 con 0x7f4de4141880 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4dea400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de4071f20 con 0x7f4de406cad0 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4dea400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de406c6b0 con 0x7f4de415ec60 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4de9000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4de4141880 0x7f4de4070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49438/0 (socket says 172.21.15.2:49438) 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.715+0000 7f4de9000640 1 -- 172.21.15.2:0/982223436 learned_addr learned my addr 172.21.15.2:0/982223436 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:27.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3e00640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 0x7f4de415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de9a00640 1 --2- 172.21.15.2:0/982223436 >> v2:172.21.15.137:3300/0 conn(0x7f4de406cad0 0x7f4de406cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322023747 0 0) 0x7f4de40716d0 con 0x7f4de4141880 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dc4003520 con 0x7f4de4141880 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2628207826 0 0) 0x7f4dc4003520 con 0x7f4de4141880 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 msgr2=0x7f4de415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 0x7f4de415f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 >> v2:172.21.15.137:3300/0 conn(0x7f4de406cad0 msgr2=0x7f4de406cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 --2- 172.21.15.2:0/982223436 >> v2:172.21.15.137:3300/0 conn(0x7f4de406cad0 0x7f4de406cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4071960 con 0x7f4de4141880 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f4dd4001670 con 0x7f4de4141880 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 >> v1:172.21.15.2:6789/0 conn(0x7f4de4141880 legacy=0x7f4de4070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dc4003970 0x7f4dc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc40047c0 0x7f4dc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 0x7f4dc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.726 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4071f20 con 0x7f4dc40047c0 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de406c6b0 con 0x7f4dc4003970 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.716+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc400b9d0 con 0x7f4de415ec60 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de3e00640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc40047c0 0x7f4dc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de3e00640 1 -- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 msgr2=0x7f4dc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de3e00640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4de415ec60 0x7f4dc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de3e00640 1 -- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dc4003970 msgr2=0x7f4dc4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de9000640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dc4003970 0x7f4dc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.727 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de3e00640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dc4003970 0x7f4dc4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de3e00640 1 -- 172.21.15.2:0/982223436 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc4003520 con 0x7f4dc40047c0 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.717+0000 7f4de9000640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4dc4003970 0x7f4dc4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.718+0000 7f4de3e00640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc40047c0 0x7f4dc4004c40 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009af0 tx=0x7f4dc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab40210d517a8f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc02c710 con 0x7f4dc40047c0 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc02c870 con 0x7f4dc40047c0 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4de3400640 1 -- 172.21.15.2:0/982223436 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc02cc00 con 0x7f4dc40047c0 2024-01-15T18:14:27.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4dea400640 1 -- 172.21.15.2:0/982223436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc40047c0 msgr2=0x7f4dc4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4dea400640 1 --2- 172.21.15.2:0/982223436 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4dc40047c0 0x7f4dc4004c40 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009af0 tx=0x7f4dc40057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4de9a00640 1 -- 172.21.15.2:0/982223436 reap_dead start 2024-01-15T18:14:27.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4dea400640 1 -- 172.21.15.2:0/982223436 shutdown_connections 2024-01-15T18:14:27.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4dea400640 1 -- 172.21.15.2:0/982223436 >> 172.21.15.2:0/982223436 conn(0x7f4de40b0840 msgr2=0x7f4de40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:27.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4dea400640 1 -- 172.21.15.2:0/982223436 shutdown_connections 2024-01-15T18:14:27.729 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.719+0000 7f4dea400640 1 -- 172.21.15.2:0/982223436 wait complete. 2024-01-15T18:14:27.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.721+0000 7f4dea400640 1 Processor -- start 2024-01-15T18:14:27.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.721+0000 7f4dea400640 1 -- start start 2024-01-15T18:14:27.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.722+0000 7f4dea400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dc40047c0 0x7f4de40cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.734 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.722+0000 7f4dea400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 0x7f4de41e5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.722+0000 7f4dea400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de4142270 0x7f4de415b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.722+0000 7f4dea400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de4140ac0 con 0x7f4de40cd500 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.722+0000 7f4dea400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40cdb20 con 0x7f4de4142270 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.722+0000 7f4dea400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de40cde00 con 0x7f4dc40047c0 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 0x7f4de41e5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 0x7f4de41e5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53032/0 (socket says 172.21.15.2:53032) 2024-01-15T18:14:27.735 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 -- 172.21.15.2:0/2679668946 learned_addr learned my addr 172.21.15.2:0/2679668946 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de9a00640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de4142270 0x7f4de415b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de9000640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dc40047c0 0x7f4de40cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 -- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dc40047c0 msgr2=0x7f4de40cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dc40047c0 0x7f4de40cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 -- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de4142270 msgr2=0x7f4de415b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de4142270 0x7f4de415b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.736 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de3e00640 1 -- 172.21.15.2:0/2679668946 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40ce0e0 con 0x7f4de40cd500 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.723+0000 7f4de9a00640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de4142270 0x7f4de415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.724+0000 7f4de3e00640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 0x7f4de41e5480 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc037040 tx=0x7f4dcc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.724+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc003810 con 0x7f4de40cd500 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.724+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc02de90 con 0x7f4de40cd500 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.724+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc003b70 con 0x7f4de40cd500 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.724+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de40ce3c0 con 0x7f4de40cd500 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.724+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de41426d0 con 0x7f4de40cd500 2024-01-15T18:14:27.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.725+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de4141e40 con 0x7f4de40cd500 2024-01-15T18:14:27.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.726+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4dcc003cd0 con 0x7f4de40cd500 2024-01-15T18:14:27.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.727+0000 7f4de0c00640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dbc06c480 0x7f4dbc06e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:27.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.727+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f4dcc0aab00 con 0x7f4de40cd500 2024-01-15T18:14:27.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.728+0000 7f4de9000640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dbc06c480 0x7f4dbc06e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:27.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.729+0000 7f4de9000640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dbc06c480 0x7f4dbc06e940 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4001eb0 tx=0x7f4dd4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:27.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.731+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4dcc077190 con 0x7f4de40cd500 2024-01-15T18:14:27.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.962+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4de4142920 con 0x7f4de40cd500 2024-01-15T18:14:27.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.963+0000 7f4de0c00640 1 -- 172.21.15.2:0/2679668946 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f4dcc07a7b0 con 0x7f4de40cd500 2024-01-15T18:14:27.966 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:14:27.966 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/3995733153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3995733153},{"type":"v1","addr":"172.21.15.2:6834","nonce":3995733153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3120477923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3120477923},{"type":"v1","addr":"172.21.15.2:6835","nonce":3120477923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1748176015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1748176015},{"type":"v1","addr":"172.21.15.137:6836","nonce":1748176015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:14:16.323114+0000","modified":"2024-01-15T18:14:19.369618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9051},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9051":{"gid":9051,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6837/2854972827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2854972827},{"type":"v1","addr":"172.21.15.137:6837","nonce":2854972827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-15T18:14:27.966 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 74 2024-01-15T18:14:27.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dbc06c480 msgr2=0x7f4dbc06e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dbc06c480 0x7f4dbc06e940 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4001eb0 tx=0x7f4dd4009040 comp rx=0 tx=0).stop 2024-01-15T18:14:27.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 msgr2=0x7f4de41e5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:27.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 0x7f4de41e5480 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc037040 tx=0x7f4dcc004730 comp rx=0 tx=0).stop 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 shutdown_connections 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dbc06c480 0x7f4dbc06e940 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4de4142270 0x7f4de415b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4de40cd500 0x7f4de41e5480 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 --2- 172.21.15.2:0/2679668946 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4dc40047c0 0x7f4de40cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 >> 172.21.15.2:0/2679668946 conn(0x7f4de40b0840 msgr2=0x7f4de415df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:27.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.965+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 shutdown_connections 2024-01-15T18:14:27.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:27.966+0000 7f4dea400640 1 -- 172.21.15.2:0/2679668946 wait complete. 2024-01-15T18:14:28.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-15T18:14:16.323114+0000', 'modified': '2024-01-15T18:14:19.369618+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9051}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9051': {'gid': 9051, 'name': 'b', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.137:6837/2854972827', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.137:6835', 'nonce': 2854972827}, {'type': 'v1', 'addr': '172.21.15.137:6837', 'nonce': 2854972827}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-15T18:14:28.390 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-15T18:14:28.390 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-15T18:14:28.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.623+0000 7fd1b9000640 1 Processor -- start 2024-01-15T18:14:28.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.623+0000 7fd1b9000640 1 -- start start 2024-01-15T18:14:28.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.625+0000 7fd1b9000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd1b415ae70 0x7fd1b40d08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.625+0000 7fd1b9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b415d720 0x7fd1b406d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.625+0000 7fd1b9000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b415dce0 con 0x7fd1b406f1b0 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.625+0000 7fd1b9000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b415dfc0 con 0x7fd1b415ae70 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.625+0000 7fd1b9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b415e2a0 con 0x7fd1b415d720 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.625+0000 7fd1b3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd1b415ae70 0x7fd1b40d08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b3400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd1b415ae70 0x7fd1b40d08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53042/0 (socket says 172.21.15.2:53042) 2024-01-15T18:14:28.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b3400640 1 -- 172.21.15.2:0/3806983314 learned_addr learned my addr 172.21.15.2:0/3806983314 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/3806983314 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b415d720 0x7fd1b406d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b3400640 1 -- 172.21.15.2:0/3806983314 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b415d720 msgr2=0x7fd1b406d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b3400640 1 --2- 172.21.15.2:0/3806983314 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b415d720 0x7fd1b406d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b3400640 1 -- 172.21.15.2:0/3806983314 >> v1:172.21.15.137:6789/0 conn(0x7fd1b406f1b0 legacy=0x7fd1b40ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.626+0000 7fd1b3400640 1 -- 172.21.15.2:0/3806983314 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b415e580 con 0x7fd1b415ae70 2024-01-15T18:14:28.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.628+0000 7fd1b3400640 1 --2- 172.21.15.2:0/3806983314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415ae70 0x7fd1b40d08f0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009e50 tx=0x7fd1a402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fb135d8d7ec266 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.628+0000 7fd1b2000640 1 -- 172.21.15.2:0/3806983314 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a402c9e0 con 0x7fd1b415ae70 2024-01-15T18:14:28.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.628+0000 7fd1b2000640 1 -- 172.21.15.2:0/3806983314 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a402cb40 con 0x7fd1b415ae70 2024-01-15T18:14:28.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.628+0000 7fd1b2000640 1 -- 172.21.15.2:0/3806983314 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40343f0 con 0x7fd1b415ae70 2024-01-15T18:14:28.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 -- 172.21.15.2:0/3806983314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415ae70 msgr2=0x7fd1b40d08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 --2- 172.21.15.2:0/3806983314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415ae70 0x7fd1b40d08f0 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009e50 tx=0x7fd1a402be40 comp rx=0 tx=0).stop 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 -- 172.21.15.2:0/3806983314 shutdown_connections 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 --2- 172.21.15.2:0/3806983314 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b415d720 0x7fd1b406d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 --2- 172.21.15.2:0/3806983314 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415ae70 0x7fd1b40d08f0 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 -- 172.21.15.2:0/3806983314 >> 172.21.15.2:0/3806983314 conn(0x7fd1b40cfdd0 msgr2=0x7fd1b40d01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 -- 172.21.15.2:0/3806983314 shutdown_connections 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.629+0000 7fd1b9000640 1 -- 172.21.15.2:0/3806983314 wait complete. 2024-01-15T18:14:28.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.631+0000 7fd1b9000640 1 Processor -- start 2024-01-15T18:14:28.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.631+0000 7fd1b9000640 1 -- start start 2024-01-15T18:14:28.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b9000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b406f1b0 0x7fd1b40b4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b9000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1b415ae70 0x7fd1b41f6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b9000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 0x7fd1b415c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b9000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40b4940 con 0x7fd1b415d720 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b9000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40b4c20 con 0x7fd1b415ae70 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b9000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b40cd590 con 0x7fd1b406f1b0 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 0x7fd1b415c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 0x7fd1b415c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53056/0 (socket says 172.21.15.2:53056) 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b3e00640 1 -- 172.21.15.2:0/1119360335 learned_addr learned my addr 172.21.15.2:0/1119360335 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.632+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b406f1b0 0x7fd1b40b4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1b415ae70 0x7fd1b41f6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b3e00640 1 -- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b406f1b0 msgr2=0x7fd1b40b4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b3e00640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b406f1b0 0x7fd1b40b4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b3e00640 1 -- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1b415ae70 msgr2=0x7fd1b41f6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b3e00640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1b415ae70 0x7fd1b41f6b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b3e00640 1 -- 172.21.15.2:0/1119360335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a4009b30 con 0x7fd1b415d720 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b3e00640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 0x7fd1b415c8b0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00c410 tx=0x7fd1ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1ab400640 1 -- 172.21.15.2:0/1119360335 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac011790 con 0x7fd1b415d720 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1ab400640 1 -- 172.21.15.2:0/1119360335 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1ac0118f0 con 0x7fd1b415d720 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b40cd870 con 0x7fd1b415d720 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1ab400640 1 -- 172.21.15.2:0/1119360335 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1ac011ce0 con 0x7fd1b415d720 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.633+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b40cd1d0 con 0x7fd1b415d720 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.636+0000 7fd1ab400640 1 -- 172.21.15.2:0/1119360335 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd1ac01c460 con 0x7fd1b415d720 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.636+0000 7fd1ab400640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd18c06c460 0x7fd18c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.643 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.637+0000 7fd1ab400640 1 -- 172.21.15.2:0/1119360335 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd1ac088750 con 0x7fd1b415d720 2024-01-15T18:14:28.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.637+0000 7fd1bd1a0740 1 Processor -- start 2024-01-15T18:14:28.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.637+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd18c06c460 0x7fd18c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.637+0000 7fd1bd1a0740 1 -- start start 2024-01-15T18:14:28.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7c070 0x558c35d7c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35d7b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7b7b0 0x558c35dbf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35cda880 con 0x558c35d76d70 2024-01-15T18:14:28.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35ce9ae0 con 0x558c35d7c070 2024-01-15T18:14:28.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c35ce71d0 con 0x558c35d7b7b0 2024-01-15T18:14:28.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1b2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35d7b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1b2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35d7b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53058/0 (socket says 172.21.15.2:53058) 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1b2a00640 1 -- 172.21.15.2:0/1872402032 learned_addr learned my addr 172.21.15.2:0/1872402032 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.638+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd18c06c460 0x7fd18c06e920 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4035000 tx=0x7fd1a4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7c070 0x558c35d7c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b3e00640 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7b7b0 0x558c35dbf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b2a00640 1 -- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7b7b0 msgr2=0x558c35dbf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7b7b0 0x558c35dbf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b2a00640 1 -- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7c070 msgr2=0x558c35d7c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7c070 0x558c35d7c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b2a00640 1 -- 172.21.15.2:0/1872402032 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c35caa860 con 0x558c35d76d70 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35d7b270 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0098f0 tx=0x7fd19c009db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12ecdf28aed521f server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1a3e00640 1 -- 172.21.15.2:0/1872402032 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00c070 con 0x558c35d76d70 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1a3e00640 1 -- 172.21.15.2:0/1872402032 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00e950 con 0x558c35d76d70 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.639+0000 7fd1a3e00640 1 -- 172.21.15.2:0/1872402032 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c00ed00 con 0x558c35d76d70 2024-01-15T18:14:28.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1872402032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 msgr2=0x558c35d7b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35d7b270 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7fd19c0098f0 tx=0x7fd19c009db0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1872402032 shutdown_connections 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7b7b0 0x558c35dbf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35d7b270 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1872402032 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7c070 0x558c35d7c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1872402032 >> 172.21.15.2:0/1872402032 conn(0x558c35d3dc70 msgr2=0x558c35dc1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1872402032 shutdown_connections 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.640+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1872402032 wait complete. 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.641+0000 7fd1bd1a0740 1 -- start start 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.641+0000 7fd1bd1a0740 1 Processor -- start 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35dc8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35ce2ac0 con 0x558c35d76d70 2024-01-15T18:14:28.659 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35c7f2f0 con 0x558c35d7b7b0 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c35c9cc70 con 0x558c35d7c070 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.642+0000 7fd1bd1a0740 10 client.0 _ll_register_callbacks cb 0x558c35ca13b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53068/0 (socket says 172.21.15.2:53068) 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 -- 172.21.15.2:0/861525233 learned_addr learned my addr 172.21.15.2:0/861525233 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35dc8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3e00640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 -- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 msgr2=0x558c35dbff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 -- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 msgr2=0x558c35dc8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35dc8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.643+0000 7fd1b3400640 1 -- 172.21.15.2:0/861525233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c0094f0 con 0x558c35d76d70 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1b3400640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1f80 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009e50 tx=0x7fd1a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecde995a60f63bd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1a0c00640 1 -- 172.21.15.2:0/861525233 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4050070 con 0x558c35d76d70 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c35cdbaa0 con 0x558c35d76d70 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1a0c00640 1 -- 172.21.15.2:0/861525233 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd1a40347c0 con 0x558c35d76d70 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1a0c00640 1 -- 172.21.15.2:0/861525233 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1a4034980 con 0x558c35d76d70 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1bd1a0740 10 client.9213 fetch_fsmap learned FSMap version 74 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x558c35ce1140 con 0x558c35d76d70 2024-01-15T18:14:28.662 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.644+0000 7fd1a0c00640 1 -- 172.21.15.2:0/861525233 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558c35cdbaa0 con 0x558c35d76d70 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1a0c00640 1 -- 172.21.15.2:0/861525233 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 74) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd1a4053020 con 0x558c35d76d70 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1bd1a0740 10 client.9213 fetch_fsmap finished waiting for FSMap version 74 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1bd1a0740 10 client.9213 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1bd1a0740 20 client.9213 populate_metadata read hostname 'smithi002' 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35dbcc50 0x558c35dc7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1bd1a0740 4 client.9213 mds_command: new command op to 9051 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-15T18:14:28.663 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 --> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c35cdb7c0 con 0x558c35dbcc50 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.645+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35dbcc50 0x558c35dc7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.647+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35dbcc50 0x558c35dc7280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=380e18fd6addae93 server_cookie=26e83b641586021d in_seq=0 out_seq=0 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.647+0000 7fd1a0c00640 10 client.9213 ms_handle_connect on v2:172.21.15.137:6835/2854972827 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1a0c00640 1 -- 172.21.15.2:0/861525233 <== mds.0 v2:172.21.15.137:6835/2854972827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558c35cdb7c0 con 0x558c35dbcc50 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1a0c00640 10 client.9213 handle_command_reply: tid=0 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 client.9213 shutdown 2024-01-15T18:14:28.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 msgr2=0x558c35da1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1f80 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009e50 tx=0x7fd1a4004ab0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 shutdown_connections 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35dbcc50 0x558c35dc7280 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35dc8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1a0c00640 0 client.9213 ms_handle_reset on v2:172.21.15.137:6835/2854972827 2024-01-15T18:14:28.665 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/861525233 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1f80 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 >> 172.21.15.2:0/861525233 conn(0x558c35d3dc70 msgr2=0x558c35d75dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 shutdown_connections 2024-01-15T18:14:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.649+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/861525233 wait complete. 2024-01-15T18:14:28.666 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.650+0000 7fd1bd1a0740 20 client.9213 trim_cache size 0 max 16384 2024-01-15T18:14:28.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.667+0000 7fd1bd1a0740 1 Processor -- start 2024-01-15T18:14:28.670 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.667+0000 7fd1bd1a0740 1 -- start start 2024-01-15T18:14:28.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.668+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.668+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35d75fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.671 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.668+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.668+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35a45f20 con 0x558c35d76d70 2024-01-15T18:14:28.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.668+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35dd0e90 con 0x558c35d7b7b0 2024-01-15T18:14:28.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.668+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c35d7c620 con 0x558c35d7c070 2024-01-15T18:14:28.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53072/0 (socket says 172.21.15.2:53072) 2024-01-15T18:14:28.672 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 -- 172.21.15.2:0/1608262162 learned_addr learned my addr 172.21.15.2:0/1608262162 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 -- 172.21.15.2:0/1608262162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 msgr2=0x558c35dbf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 -- 172.21.15.2:0/1608262162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 msgr2=0x558c35d75fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35d75fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 -- 172.21.15.2:0/1608262162 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c35c7eb80 con 0x558c35d76d70 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.669+0000 7fd1b3400640 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1e20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd1a402c450 tx=0x7fd1a4049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa53074638bba822 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1a8c00640 1 -- 172.21.15.2:0/1608262162 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4042070 con 0x558c35d76d70 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1a8c00640 1 -- 172.21.15.2:0/1608262162 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a403e740 con 0x558c35d76d70 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1a8c00640 1 -- 172.21.15.2:0/1608262162 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4050030 con 0x558c35d76d70 2024-01-15T18:14:28.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1608262162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 msgr2=0x558c35da1e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1e20 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fd1a402c450 tx=0x7fd1a4049000 comp rx=0 tx=0).stop 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1608262162 shutdown_connections 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35dbf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35d75fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/1608262162 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35da1e20 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1608262162 >> 172.21.15.2:0/1608262162 conn(0x558c35d3dc70 msgr2=0x558c35d7a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.680 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1608262162 shutdown_connections 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.670+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/1608262162 wait complete. 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.671+0000 7fd1bd1a0740 1 -- start start 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.671+0000 7fd1bd1a0740 1 Processor -- start 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35e2be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35db74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35da0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.681 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35dcf8d0 con 0x558c35d76d70 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c35dbc430 con 0x558c35d7b7b0 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c35db7df0 con 0x558c35d7c070 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.672+0000 7fd1bd1a0740 10 client.0 _ll_register_callbacks cb 0x7fd1a4009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35e2be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35e2be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53074/0 (socket says 172.21.15.2:53074) 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 -- 172.21.15.2:0/2185554413 learned_addr learned my addr 172.21.15.2:0/2185554413 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 -- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 msgr2=0x558c35da0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35da0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 -- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 msgr2=0x558c35db74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35db74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 -- 172.21.15.2:0/2185554413 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558c35ce1140 con 0x558c35d76d70 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1b3400640 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35e2be30 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009680 tx=0x7fd1a4009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe756421a0302efa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.2:3300/0 2024-01-15T18:14:28.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.673+0000 7fd1a0c00640 1 -- 172.21.15.2:0/2185554413 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4042070 con 0x558c35d76d70 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1a0c00640 1 -- 172.21.15.2:0/2185554413 <== mon.0 v2:172.21.15.2:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd1a4051070 con 0x558c35d76d70 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd19c00e950 con 0x558c35d76d70 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1a0c00640 1 -- 172.21.15.2:0/2185554413 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1a4050040 con 0x558c35d76d70 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 10 client.9219 fetch_fsmap learned FSMap version 74 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x558c35ca4170 con 0x558c35d76d70 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1a0c00640 1 -- 172.21.15.2:0/2185554413 <== mon.0 v2:172.21.15.2:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c00e950 con 0x558c35d76d70 2024-01-15T18:14:28.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1a0c00640 1 -- 172.21.15.2:0/2185554413 <== mon.0 v2:172.21.15.2:3300/0 5 ==== fsmap(e 74) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd1a4063020 con 0x558c35d76d70 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 10 client.9219 fetch_fsmap finished waiting for FSMap version 74 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 10 client.9219 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 20 client.9219 populate_metadata read hostname 'smithi002' 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35da0ed0 0x558c35dcfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 4 client.9219 mds_command: new command op to 9051 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.674+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 --> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558c359e36c0 con 0x558c35da0ed0 2024-01-15T18:14:28.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.675+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35da0ed0 0x558c35dcfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.676+0000 7fd1b2a00640 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35da0ed0 0x558c35dcfa40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3506b051c029a765 server_cookie=adb1b67925c16955 in_seq=0 out_seq=0 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.676+0000 7fd1a0c00640 10 client.9219 ms_handle_connect on v2:172.21.15.137:6835/2854972827 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1a0c00640 1 -- 172.21.15.2:0/2185554413 <== mds.0 v2:172.21.15.137:6835/2854972827 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x558c359e36c0 con 0x558c35da0ed0 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1a0c00640 10 client.9219 handle_command_reply: tid=0 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 client.9219 shutdown 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 msgr2=0x558c35e2be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35e2be30 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009680 tx=0x7fd1a4009810 comp rx=0 tx=0).stop 2024-01-15T18:14:28.688 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 shutdown_connections 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:6835/2854972827,v1:172.21.15.137:6837/2854972827] conn(0x558c35da0ed0 0x558c35dcfa40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x558c35d7c070 0x558c35da0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x558c35d7b7b0 0x558c35db74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1a0c00640 0 client.9219 ms_handle_reset on v2:172.21.15.137:6835/2854972827 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 --2- 172.21.15.2:0/2185554413 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x558c35d76d70 0x558c35e2be30 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 >> 172.21.15.2:0/2185554413 conn(0x558c35d3dc70 msgr2=0x558c35da24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 shutdown_connections 2024-01-15T18:14:28.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.677+0000 7fd1bd1a0740 1 -- 172.21.15.2:0/2185554413 wait complete. 2024-01-15T18:14:28.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1bd1a0740 20 client.9219 trim_cache size 0 max 16384 2024-01-15T18:14:28.690 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:14:28.690 INFO:teuthology.orchestra.run.smithi002.stdout: "cluster_fsid": "c0e5773f-513d-405e-88b5-a3133821721f", 2024-01-15T18:14:28.690 INFO:teuthology.orchestra.run.smithi002.stdout: "whoami": 0, 2024-01-15T18:14:28.690 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 9051, 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "want_state": "up:active", 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "up:active", 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs", 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "rank_uptime": 11.788615232, 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "mdsmap_epoch": 74, 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch": 114, 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "osdmap_epoch_barrier": 114, 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout: "uptime": 17.224828519999999 2024-01-15T18:14:28.691 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd18c06c460 msgr2=0x7fd18c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd18c06c460 0x7fd18c06e920 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4035000 tx=0x7fd1a4030000 comp rx=0 tx=0).stop 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 msgr2=0x7fd1b415c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 0x7fd1b415c8b0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac00c410 tx=0x7fd1ac00c8d0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 shutdown_connections 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd18c06c460 0x7fd18c06e920 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd1b415d720 0x7fd1b415c8b0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd1b415ae70 0x7fd1b41f6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 --2- 172.21.15.2:0/1119360335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd1b406f1b0 0x7fd1b40b4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 >> 172.21.15.2:0/1119360335 conn(0x7fd1b40cfdd0 msgr2=0x7fd1b406d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.678+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 shutdown_connections 2024-01-15T18:14:28.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.679+0000 7fd1b9000640 1 -- 172.21.15.2:0/1119360335 wait complete. 2024-01-15T18:14:28.698 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:14:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.909+0000 7fc053e00640 1 Processor -- start 2024-01-15T18:14:28.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.909+0000 7fc053e00640 1 -- start start 2024-01-15T18:14:28.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.911+0000 7fc053e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc04c15b7d0 0x7fc04c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.911+0000 7fc053e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c4c0 0x7fc04c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.911+0000 7fc053e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c15cb00 con 0x7fc04c15c0e0 2024-01-15T18:14:28.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.911+0000 7fc053e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c0cccd0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.911+0000 7fc053e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0ccec0 con 0x7fc04c15c4c0 2024-01-15T18:14:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc050a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc04c15b7d0 0x7fc04c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc051e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c4c0 0x7fc04c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc050a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc04c15b7d0 0x7fc04c06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53084/0 (socket says 172.21.15.2:53084) 2024-01-15T18:14:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc050a00640 1 -- 172.21.15.2:0/3835881133 learned_addr learned my addr 172.21.15.2:0/3835881133 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc051400640 1 --1- 172.21.15.2:0/3835881133 >> v1:172.21.15.137:6789/0 conn(0x7fc04c15c0e0 0x7fc04c0cc5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:60504/0 (socket says 172.21.15.2:60504) 2024-01-15T18:14:28.915 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc050a00640 1 -- 172.21.15.2:0/3835881133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c4c0 msgr2=0x7fc04c06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:28.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.912+0000 7fc050a00640 1 --2- 172.21.15.2:0/3835881133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c4c0 0x7fc04c06c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.913+0000 7fc050a00640 1 -- 172.21.15.2:0/3835881133 >> v1:172.21.15.137:6789/0 conn(0x7fc04c15c0e0 legacy=0x7fc04c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.916 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.913+0000 7fc050a00640 1 -- 172.21.15.2:0/3835881133 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0cd1a0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.914+0000 7fc050a00640 1 --2- 172.21.15.2:0/3835881133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c06c470 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009e50 tx=0x7fc03c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fffbb471cddcdb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.915+0000 7fc04be00640 1 -- 172.21.15.2:0/3835881133 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c02cb00 con 0x7fc04c15b7d0 2024-01-15T18:14:28.917 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.915+0000 7fc04be00640 1 -- 172.21.15.2:0/3835881133 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c02cc60 con 0x7fc04c15b7d0 2024-01-15T18:14:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.915+0000 7fc04be00640 1 -- 172.21.15.2:0/3835881133 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0343f0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.915+0000 7fc053e00640 1 -- 172.21.15.2:0/3835881133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 msgr2=0x7fc04c06c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.915+0000 7fc053e00640 1 --2- 172.21.15.2:0/3835881133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c06c470 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fc03c009e50 tx=0x7fc03c02bf60 comp rx=0 tx=0).stop 2024-01-15T18:14:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.916+0000 7fc053e00640 1 -- 172.21.15.2:0/3835881133 shutdown_connections 2024-01-15T18:14:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.916+0000 7fc053e00640 1 --2- 172.21.15.2:0/3835881133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c4c0 0x7fc04c06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.916+0000 7fc053e00640 1 --2- 172.21.15.2:0/3835881133 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c06c470 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.916+0000 7fc053e00640 1 -- 172.21.15.2:0/3835881133 >> 172.21.15.2:0/3835881133 conn(0x7fc04c06da20 msgr2=0x7fc04c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.916+0000 7fc053e00640 1 -- 172.21.15.2:0/3835881133 shutdown_connections 2024-01-15T18:14:28.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.916+0000 7fc053e00640 1 -- 172.21.15.2:0/3835881133 wait complete. 2024-01-15T18:14:28.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.917+0000 7fc053e00640 1 Processor -- start 2024-01-15T18:14:28.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.918+0000 7fc053e00640 1 -- start start 2024-01-15T18:14:28.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc053e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c1f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc053e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c0e0 0x7fc04c14cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc053e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc04c15c4c0 0x7fc04c139510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc053e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c1f5090 con 0x7fc04c15b7d0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc053e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c147c60 con 0x7fc04c15c4c0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc053e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c147f40 con 0x7fc04c15c0e0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc051400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c1f0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc051400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c1f0ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53100/0 (socket says 172.21.15.2:53100) 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc051400640 1 -- 172.21.15.2:0/194681335 learned_addr learned my addr 172.21.15.2:0/194681335 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc050a00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c0e0 0x7fc04c14cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.919+0000 7fc051e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc04c15c4c0 0x7fc04c139510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051400640 1 -- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c0e0 msgr2=0x7fc04c14cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051400640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c0e0 0x7fc04c14cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051400640 1 -- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc04c15c4c0 msgr2=0x7fc04c139510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051400640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc04c15c4c0 0x7fc04c139510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051400640 1 -- 172.21.15.2:0/194681335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c009b30 con 0x7fc04c15b7d0 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc050a00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c0e0 0x7fc04c14cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:28.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc04c15c4c0 0x7fc04c139510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:14:28.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc051400640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c1f0ca0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c4d0 tx=0x7fc03400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034007480 con 0x7fc04c15b7d0 2024-01-15T18:14:28.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0340075e0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0340049e0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.920+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c148220 con 0x7fc04c15b7d0 2024-01-15T18:14:28.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.921+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c1f11e0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.922+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c0686f0 con 0x7fc04c15b7d0 2024-01-15T18:14:28.925 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.923+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc034004b40 con 0x7fc04c15b7d0 2024-01-15T18:14:28.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.924+0000 7fc049600640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc02406c430 0x7fc02406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:28.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.924+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fc034087b50 con 0x7fc04c15b7d0 2024-01-15T18:14:28.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.924+0000 7fc050a00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc02406c430 0x7fc02406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:28.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.926+0000 7fc050a00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc02406c430 0x7fc02406e8f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc03c035000 tx=0x7fc03c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:28.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:28.928+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc034054170 con 0x7fc04c15b7d0 2024-01-15T18:14:29.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.165+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc04c06b260 con 0x7fc04c15b7d0 2024-01-15T18:14:29.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.166+0000 7fc049600640 1 -- 172.21.15.2:0/194681335 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc034018020 con 0x7fc04c15b7d0 2024-01-15T18:14:29.168 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.168+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc02406c430 msgr2=0x7fc02406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.168+0000 7fc053e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc02406c430 0x7fc02406e8f0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fc03c035000 tx=0x7fc03c030000 comp rx=0 tx=0).stop 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.168+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 msgr2=0x7fc04c1f0ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.168+0000 7fc053e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c1f0ca0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fc03400c4d0 tx=0x7fc03400c990 comp rx=0 tx=0).stop 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 shutdown_connections 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc02406c430 0x7fc02406e8f0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc04c15c4c0 0x7fc04c139510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc04c15c0e0 0x7fc04c14cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 --2- 172.21.15.2:0/194681335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc04c15b7d0 0x7fc04c1f0ca0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 >> 172.21.15.2:0/194681335 conn(0x7fc04c06da20 msgr2=0x7fc04c0aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:29.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 shutdown_connections 2024-01-15T18:14:29.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:29.169+0000 7fc053e00640 1 -- 172.21.15.2:0/194681335 wait complete. 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi002.front.sepia.ceph.com 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:14:29.191 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:14:29.192 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:14:29.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:29.192 DEBUG:teuthology.orchestra.run.smithi002:> ip addr 2024-01-15T18:14:29.247 INFO:teuthology.orchestra.run.smithi002.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 ::1/128 scope host 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:02 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f0 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:6b:fe:03 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp7s0f1 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:20 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:29.248 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f0 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: inet 172.21.15.2/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft 41613sec preferred_lft 41613sec 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::ec4:7aff:febd:1920/64 scope link noprefixroute 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether 0c:c4:7a:bd:19:21 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: altname enp1s0f1 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::888:72ff:fe40:83b1/64 scope link 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:14:29.249 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether e2:02:73:ca:54:83 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:14:29.250 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e002:73ff:feca:5483/64 scope link 2024-01-15T18:14:29.250 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.250 INFO:teuthology.orchestra.run.smithi002.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:14:29.250 INFO:teuthology.orchestra.run.smithi002.stdout: link/ether ea:6b:e2:68:2a:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:14:29.250 INFO:teuthology.orchestra.run.smithi002.stdout: inet6 fe80::e86b:e2ff:fe68:2a61/64 scope link 2024-01-15T18:14:29.250 INFO:teuthology.orchestra.run.smithi002.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:29.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:29.251 DEBUG:teuthology.orchestra.run.smithi002:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:14:29.346 INFO:teuthology.orchestra.run.smithi002.stdout:1 2024-01-15T18:14:29.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:29.351 DEBUG:teuthology.orchestra.run.smithi002:> route 2024-01-15T18:14:29.407 INFO:teuthology.orchestra.run.smithi002.stdout:Kernel IP routing table 2024-01-15T18:14:29.407 INFO:teuthology.orchestra.run.smithi002.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:14:29.407 INFO:teuthology.orchestra.run.smithi002.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-15T18:14:29.407 INFO:teuthology.orchestra.run.smithi002.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-15T18:14:29.407 INFO:teuthology.orchestra.run.smithi002.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:14:29.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:29.408 DEBUG:teuthology.orchestra.run.smithi002:> (cd / && exec bash -c ' 2024-01-15T18:14:29.408 DEBUG:teuthology.orchestra.run.smithi002:> set -e 2024-01-15T18:14:29.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-15T18:14:29.408 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-15T18:14:29.409 DEBUG:teuthology.orchestra.run.smithi002:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-15T18:14:29.409 DEBUG:teuthology.orchestra.run.smithi002:> ') 2024-01-15T18:14:29.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:29.605 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:14:29.659 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:14:29.659 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:14:29.660 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:14:29.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:29.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link set brx.0 up 2024-01-15T18:14:29.759 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-15T18:14:29.759 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:14:29.759 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:29.816 INFO:teuthology.orchestra.run.smithi002.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-15T18:14:29.816 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:14:29.816 DEBUG:teuthology.orchestra.run.smithi002:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:29.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:14:29.971 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:14:30.027 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:14:30.027 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:14:30.027 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:14:30.027 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:14:30.027 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:14:30.027 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:14:30.028 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:14:30.029 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:14:30.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:30.030 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:14:30.083 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:14:30.083 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-15T18:14:30.090 DEBUG:teuthology.orchestra.run.smithi002:> sudo modprobe fuse 2024-01-15T18:14:30.166 DEBUG:teuthology.orchestra.run.smithi002:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/proc 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/sys 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/dev 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/security 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/shm 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/pts 2024-01-15T18:14:30.210 INFO:teuthology.orchestra.run.smithi002.stdout:/run 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/cgroup 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/pstore 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/ 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/selinux 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/hugepages 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/dev/mqueue 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/debug 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/tracing 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/fs/fuse/connections 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/sys/kernel/config 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:14:30.211 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/run/user/1000 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-0 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-1 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-2 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/var/lib/ceph/osd/ceph-3 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:14:30.212 INFO:teuthology.orchestra.run.smithi002.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:14:30.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:30.213 DEBUG:teuthology.orchestra.run.smithi002:> ls /sys/fs/fuse/connections 2024-01-15T18:14:30.272 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:2024-01-15T18:14:30.270+0000 7f85f2850480 -1 init, newargv = 0x55bdeced0990 newargc=15 2024-01-15T18:14:30.273 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[76355]: starting ceph client 2024-01-15T18:14:30.282 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[76355]: starting fuse 2024-01-15T18:14:30.299 INFO:teuthology.orchestra.run.smithi002.stdout:41 2024-01-15T18:14:30.299 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> import glob 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> import re 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> import os 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> import subprocess 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> def _find_admin_socket(client_name): 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> files = glob.glob(asok_path) 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> # Given a non-glob path, it better be there 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> if "*" not in asok_path: 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> assert(len(files) == 1) 2024-01-15T18:14:30.300 DEBUG:teuthology.orchestra.run.smithi002:> return files[0] 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> for f in files: 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> contents = proc_f.read() 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> if mountpoint in contents: 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> return f 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> print(_find_admin_socket("client.0")) 2024-01-15T18:14:30.301 DEBUG:teuthology.orchestra.run.smithi002:> ' 2024-01-15T18:14:30.507 INFO:teuthology.orchestra.run.smithi002.stdout:/var/run/ceph/ceph-client.0.76355.asok 2024-01-15T18:14:30.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:14:30.525 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.76355.asok 2024-01-15T18:14:30.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:30.526 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.76355.asok status 2024-01-15T18:14:30.776 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:14:30.776 INFO:teuthology.orchestra.run.smithi002.stdout: "metadata": { 2024-01-15T18:14:30.776 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:14:30.776 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "entity_id": "0", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "hostname": "smithi002", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "pid": "76355", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "root": "/" 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_count": 0, 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "dentry_pinned_count": 0, 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "id": 9231, 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "inst": { 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "name": { 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "client", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "num": 9231 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:14:30.777 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 4222672992 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": { 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "type": "v1", 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "nonce": 4222672992 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "inst_str": "client.9231 192.168.0.1:0/4222672992", 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "addr_str": "192.168.0.1:0/4222672992", 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "inode_count": 1, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "mds_epoch": 74, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch": 114, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "blocklisted": false, 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout: "fs_name": "cephfs" 2024-01-15T18:14:30.778 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:14:30.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:30.793 DEBUG:teuthology.orchestra.run.smithi002:> stat --file-system '--printf=%T 2024-01-15T18:14:30.793 DEBUG:teuthology.orchestra.run.smithi002:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:30.851 INFO:teuthology.orchestra.run.smithi002.stdout:fuseblk 2024-01-15T18:14:30.852 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:30.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:30.852 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-15T18:14:30.951 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-15T18:14:31.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.170+0000 7f9bd7200640 1 Processor -- start 2024-01-15T18:14:31.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.170+0000 7f9bd7200640 1 -- start start 2024-01-15T18:14:31.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bd7200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bd815f0e0 0x7f9bd806d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:31.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bd7200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd80716d0 con 0x7f9bd815ec60 2024-01-15T18:14:31.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bd7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd8071960 con 0x7f9bd8141880 2024-01-15T18:14:31.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bd7200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd8071f20 con 0x7f9bd815f0e0 2024-01-15T18:14:31.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bd6800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f9bd815f0e0 0x7f9bd806d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:31.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bcfe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9bd815ec60 0x7f9bd806cad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:49446/0 (socket says 172.21.15.2:49446) 2024-01-15T18:14:31.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.172+0000 7f9bcfe00640 1 -- 172.21.15.2:0/4292685389 learned_addr learned my addr 172.21.15.2:0/4292685389 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:31.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.173+0000 7f9bd6800640 1 -- 172.21.15.2:0/4292685389 >> v1:172.21.15.137:6789/0 conn(0x7f9bd8141880 legacy=0x7f9bd8070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.173+0000 7f9bd6800640 1 -- 172.21.15.2:0/4292685389 >> v1:172.21.15.2:6789/0 conn(0x7f9bd815ec60 legacy=0x7f9bd806cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.173+0000 7f9bd6800640 1 -- 172.21.15.2:0/4292685389 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd8071c40 con 0x7f9bd815f0e0 2024-01-15T18:14:31.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.174+0000 7f9bd6800640 1 --2- 172.21.15.2:0/4292685389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd806d1e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009d90 tx=0x7f9bd002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2356639b6c2b34a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:31.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.174+0000 7f9bd5400640 1 -- 172.21.15.2:0/4292685389 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd002cc60 con 0x7f9bd815f0e0 2024-01-15T18:14:31.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.174+0000 7f9bd5400640 1 -- 172.21.15.2:0/4292685389 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd002cdc0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.174+0000 7f9bd5400640 1 -- 172.21.15.2:0/4292685389 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0034660 con 0x7f9bd815f0e0 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 -- 172.21.15.2:0/4292685389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 msgr2=0x7f9bd806d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 --2- 172.21.15.2:0/4292685389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd806d1e0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009d90 tx=0x7f9bd002beb0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 -- 172.21.15.2:0/4292685389 shutdown_connections 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 --2- 172.21.15.2:0/4292685389 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd806d1e0 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 -- 172.21.15.2:0/4292685389 >> 172.21.15.2:0/4292685389 conn(0x7f9bd80b0840 msgr2=0x7f9bd80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 -- 172.21.15.2:0/4292685389 shutdown_connections 2024-01-15T18:14:31.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.175+0000 7f9bd7200640 1 -- 172.21.15.2:0/4292685389 wait complete. 2024-01-15T18:14:31.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.177+0000 7f9bd7200640 1 Processor -- start 2024-01-15T18:14:31.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.177+0000 7f9bd7200640 1 -- start start 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bd8141880 0x7f9bd813db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bd815ec60 0x7f9bd8143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd8143b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd806c6b0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8142000 con 0x7f9bd815ec60 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8144170 con 0x7f9bd8141880 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd8143b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:31.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd8143b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:53120/0 (socket says 172.21.15.2:53120) 2024-01-15T18:14:31.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.178+0000 7f9bd6800640 1 -- 172.21.15.2:0/3253384235 learned_addr learned my addr 172.21.15.2:0/3253384235 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bcfe00640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bd815ec60 0x7f9bd8143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd5e00640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bd8141880 0x7f9bd813db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd6800640 1 -- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bd8141880 msgr2=0x7f9bd813db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd6800640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bd8141880 0x7f9bd813db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd6800640 1 -- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bd815ec60 msgr2=0x7f9bd8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd6800640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bd815ec60 0x7f9bd8143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd6800640 1 -- 172.21.15.2:0/3253384235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0009a70 con 0x7f9bd815f0e0 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.179+0000 7f9bd6800640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd8143b40 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009d90 tx=0x7f9bd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.180+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0040650 con 0x7f9bd815f0e0 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.180+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd00407d0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.180+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd0040b80 con 0x7f9bd815f0e0 2024-01-15T18:14:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.180+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd81422a0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.180+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd8142860 con 0x7f9bd815f0e0 2024-01-15T18:14:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.181+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8074770 con 0x7f9bd815f0e0 2024-01-15T18:14:31.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.182+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f9bd0040ce0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.183+0000 7f9bce000640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9bb006c430 0x7f9bb006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:31.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.183+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f9bd00aba20 con 0x7f9bd815f0e0 2024-01-15T18:14:31.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.184+0000 7f9bd5e00640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9bb006c430 0x7f9bb006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:31.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.185+0000 7f9bd5e00640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9bb006c430 0x7f9bb006e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8000f60 tx=0x7f9bc8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:31.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.188+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bd0077f90 con 0x7f9bd815f0e0 2024-01-15T18:14:31.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.422+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9bd806e1c0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.423+0000 7f9bce000640 1 -- 172.21.15.2:0/3253384235 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9bd007b5b0 con 0x7f9bd815f0e0 2024-01-15T18:14:31.426 INFO:teuthology.orchestra.run.smithi002.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9bb006c430 msgr2=0x7f9bb006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9bb006c430 0x7f9bb006e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8000f60 tx=0x7f9bc8009040 comp rx=0 tx=0).stop 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 msgr2=0x7f9bd8143b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd8143b40 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009d90 tx=0x7f9bd0004a40 comp rx=0 tx=0).stop 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 shutdown_connections 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9bb006c430 0x7f9bb006e8f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9bd815f0e0 0x7f9bd8143b40 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9bd815ec60 0x7f9bd8143600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 --2- 172.21.15.2:0/3253384235 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9bd8141880 0x7f9bd813db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 >> 172.21.15.2:0/3253384235 conn(0x7f9bd80b0840 msgr2=0x7f9bd80aff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:31.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 shutdown_connections 2024-01-15T18:14:31.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:31.426+0000 7f9bd7200640 1 -- 172.21.15.2:0/3253384235 wait complete. 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi137.front.sepia.ceph.com 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-15T18:14:31.449 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:14:31.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:31.449 DEBUG:teuthology.orchestra.run.smithi137:> ip addr 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: inet 127.0.0.1/8 scope host lo 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 ::1/128 scope host 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7e brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f0 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:14:31.508 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:d9:36:7f brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: altname enp7s0f1 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:36 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 0c:c4:7a:88:81:37 brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: inet 172.21.15.137/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft 41599sec preferred_lft 41599sec 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::ec4:7aff:fe88:8137/64 scope link 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::cce:48ff:fe87:f2c0/64 scope link 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.509 INFO:teuthology.orchestra.run.smithi137.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether da:0c:5e:f2:81:3c brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::d80c:5eff:fef2:813c/64 scope link 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout: link/ether 46:1d:f8:7c:b8:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout: inet6 fe80::441d:f8ff:fe7c:b8ff/64 scope link 2024-01-15T18:14:31.510 INFO:teuthology.orchestra.run.smithi137.stdout: valid_lft forever preferred_lft forever 2024-01-15T18:14:31.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:31.511 DEBUG:teuthology.orchestra.run.smithi137:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-15T18:14:31.598 INFO:teuthology.orchestra.run.smithi137.stdout:1 2024-01-15T18:14:31.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:31.602 DEBUG:teuthology.orchestra.run.smithi137:> route 2024-01-15T18:14:31.652 INFO:teuthology.orchestra.run.smithi137.stdout:Kernel IP routing table 2024-01-15T18:14:31.652 INFO:teuthology.orchestra.run.smithi137.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-15T18:14:31.652 INFO:teuthology.orchestra.run.smithi137.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-15T18:14:31.652 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-15T18:14:31.653 INFO:teuthology.orchestra.run.smithi137.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-15T18:14:31.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:31.654 DEBUG:teuthology.orchestra.run.smithi137:> (cd / && exec bash -c ' 2024-01-15T18:14:31.654 DEBUG:teuthology.orchestra.run.smithi137:> set -e 2024-01-15T18:14:31.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-15T18:14:31.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-15T18:14:31.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-15T18:14:31.654 DEBUG:teuthology.orchestra.run.smithi137:> ') 2024-01-15T18:14:31.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:31.836 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:14:31.884 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:14:31.884 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:14:31.885 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:14:31.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:31.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link set brx.0 up 2024-01-15T18:14:31.972 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-15T18:14:31.973 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:14:31.973 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:32.024 INFO:teuthology.orchestra.run.smithi137.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-15T18:14:32.025 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-15T18:14:32.025 DEBUG:teuthology.orchestra.run.smithi137:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:32.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:14:32.163 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:14:32.217 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:14:32.217 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:14:32.218 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:14:32.219 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:14:32.220 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:14:32.220 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:14:32.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:32.221 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:14:32.268 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-15T18:14:32.268 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-15T18:14:32.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo modprobe fuse 2024-01-15T18:14:32.345 DEBUG:teuthology.orchestra.run.smithi137:> cat /proc/self/mounts | awk '{print $2}' 2024-01-15T18:14:32.385 INFO:teuthology.orchestra.run.smithi137.stdout:/proc 2024-01-15T18:14:32.385 INFO:teuthology.orchestra.run.smithi137.stdout:/sys 2024-01-15T18:14:32.385 INFO:teuthology.orchestra.run.smithi137.stdout:/dev 2024-01-15T18:14:32.385 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/security 2024-01-15T18:14:32.385 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/shm 2024-01-15T18:14:32.385 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/pts 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/run 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/cgroup 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/pstore 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/ 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/selinux 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/proc/sys/fs/binfmt_misc 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/mqueue 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/dev/hugepages 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/debug 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/tracing 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/fs/fuse/connections 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/sys/kernel/config 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-sysctl.service 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-15T18:14:32.386 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/nfs/rpc_pipefs 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/run/user/1000 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-4 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-5 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-6 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/var/lib/ceph/osd/ceph-7 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:14:32.387 INFO:teuthology.orchestra.run.smithi137.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:14:32.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:32.388 DEBUG:teuthology.orchestra.run.smithi137:> ls /sys/fs/fuse/connections 2024-01-15T18:14:32.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:2024-01-15T18:14:32.453+0000 7f562863e480 -1 init, newargv = 0x56096b0667f0 newargc=15 2024-01-15T18:14:32.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[47446]: starting ceph client 2024-01-15T18:14:32.466 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[47446]: starting fuse 2024-01-15T18:14:32.472 INFO:teuthology.orchestra.run.smithi137.stdout:40 2024-01-15T18:14:32.472 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo stdin-killer -- python3 -c ' 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> import glob 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> import re 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> import os 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> import subprocess 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> def _find_admin_socket(client_name): 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> files = glob.glob(asok_path) 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> # Given a non-glob path, it better be there 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> if "*" not in asok_path: 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> assert(len(files) == 1) 2024-01-15T18:14:32.473 DEBUG:teuthology.orchestra.run.smithi137:> return files[0] 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> for f in files: 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> contents = proc_f.read() 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> if mountpoint in contents: 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> return f 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> print(_find_admin_socket("client.1")) 2024-01-15T18:14:32.474 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-01-15T18:14:32.667 INFO:teuthology.orchestra.run.smithi137.stdout:/var/run/ceph/ceph-client.1.47446.asok 2024-01-15T18:14:32.671 INFO:teuthology.orchestra.run.smithi137.stderr:2024-01-15T18:14:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-15T18:14:32.685 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47446.asok 2024-01-15T18:14:32.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:32.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47446.asok status 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "metadata": { 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_sha1": "95c90f413c4bb65e3e56909855f2308fd9bda05b", 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 18.0.0-7629-g95c90f41 (95c90f413c4bb65e3e56909855f2308fd9bda05b) reef (dev)", 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "entity_id": "1", 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "hostname": "smithi137", 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "pid": "47446", 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "root": "/" 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_count": 0, 2024-01-15T18:14:32.926 INFO:teuthology.orchestra.run.smithi137.stdout: "dentry_pinned_count": 0, 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "id": 6638, 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "inst": { 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "name": { 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "client", 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "num": 6638 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1591427153 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": { 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-01-15T18:14:32.927 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "192.168.0.1:0", 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1591427153 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "inst_str": "client.6638 192.168.0.1:0/1591427153", 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "addr_str": "192.168.0.1:0/1591427153", 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "inode_count": 1, 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_epoch": 74, 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch": 114, 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_epoch_barrier": 0, 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklisted": false, 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "cephfs" 2024-01-15T18:14:32.928 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-01-15T18:14:32.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:32.941 DEBUG:teuthology.orchestra.run.smithi137:> stat --file-system '--printf=%T 2024-01-15T18:14:32.941 DEBUG:teuthology.orchestra.run.smithi137:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:32.993 INFO:teuthology.orchestra.run.smithi137.stdout:fuseblk 2024-01-15T18:14:32.993 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:32.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:32.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-15T18:14:33.083 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-15T18:14:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.315+0000 7fae91200640 1 Processor -- start 2024-01-15T18:14:33.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.315+0000 7fae91200640 1 -- start start 2024-01-15T18:14:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.316+0000 7fae91200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fae8c0d1870 0x7fae8c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.316+0000 7fae91200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fae8c0b2210 0x7fae8c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.316+0000 7fae91200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8c0d1250 con 0x7fae8c0d1c50 2024-01-15T18:14:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.316+0000 7fae91200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0ce6d0 con 0x7fae8c0b2210 2024-01-15T18:14:33.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.316+0000 7fae91200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8c0ce8e0 con 0x7fae8c0d1870 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.317+0000 7fae8be00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fae8c0d1c50 0x7fae8c0cf910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:43500/0 (socket says 172.21.15.2:43500) 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.317+0000 7fae8be00640 1 -- 172.21.15.2:0/3873672624 learned_addr learned my addr 172.21.15.2:0/3873672624 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.317+0000 7fae83e00640 1 --2- 172.21.15.2:0/3873672624 >> v2:172.21.15.2:3300/0 conn(0x7fae8c0b2210 0x7fae8c15d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.317+0000 7fae8b400640 1 --2- 172.21.15.2:0/3873672624 >> v2:172.21.15.137:3300/0 conn(0x7fae8c0d1870 0x7fae8c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.317+0000 7fae8aa00640 1 -- 172.21.15.2:0/3873672624 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145906948 0 0) 0x7fae8c0d1250 con 0x7fae8c0d1c50 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.317+0000 7fae8aa00640 1 -- 172.21.15.2:0/3873672624 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae6c003520 con 0x7fae8c0d1c50 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.318+0000 7fae83e00640 1 -- 172.21.15.2:0/3873672624 >> v2:172.21.15.137:3300/0 conn(0x7fae8c0d1870 msgr2=0x7fae8c0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:33.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.318+0000 7fae83e00640 1 --2- 172.21.15.2:0/3873672624 >> v2:172.21.15.137:3300/0 conn(0x7fae8c0d1870 0x7fae8c0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.318+0000 7fae83e00640 1 -- 172.21.15.2:0/3873672624 >> v1:172.21.15.2:6789/0 conn(0x7fae8c0d1c50 legacy=0x7fae8c0cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:33.321 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.318+0000 7fae83e00640 1 -- 172.21.15.2:0/3873672624 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0cebc0 con 0x7fae8c0b2210 2024-01-15T18:14:33.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.319+0000 7fae83e00640 1 --2- 172.21.15.2:0/3873672624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0b2210 0x7fae8c15d0e0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fae8c067890 tx=0x7fae7402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cea170225725421 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:33.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.319+0000 7fae8aa00640 1 -- 172.21.15.2:0/3873672624 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74031830 con 0x7fae8c0b2210 2024-01-15T18:14:33.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.319+0000 7fae8aa00640 1 -- 172.21.15.2:0/3873672624 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae74031990 con 0x7fae8c0b2210 2024-01-15T18:14:33.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.319+0000 7fae8aa00640 1 -- 172.21.15.2:0/3873672624 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae74031d20 con 0x7fae8c0b2210 2024-01-15T18:14:33.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.319+0000 7fae91200640 1 -- 172.21.15.2:0/3873672624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0b2210 msgr2=0x7fae8c15d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.319+0000 7fae91200640 1 --2- 172.21.15.2:0/3873672624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0b2210 0x7fae8c15d0e0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7fae8c067890 tx=0x7fae7402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.320+0000 7fae91200640 1 -- 172.21.15.2:0/3873672624 shutdown_connections 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.320+0000 7fae91200640 1 --2- 172.21.15.2:0/3873672624 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0b2210 0x7fae8c15d0e0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.320+0000 7fae91200640 1 --2- 172.21.15.2:0/3873672624 >> v2:172.21.15.137:3300/0 conn(0x7fae8c0d1870 0x7fae8c0d3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.320+0000 7fae91200640 1 -- 172.21.15.2:0/3873672624 >> 172.21.15.2:0/3873672624 conn(0x7fae8c0d30b0 msgr2=0x7fae8c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.320+0000 7fae91200640 1 -- 172.21.15.2:0/3873672624 shutdown_connections 2024-01-15T18:14:33.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.320+0000 7fae91200640 1 -- 172.21.15.2:0/3873672624 wait complete. 2024-01-15T18:14:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.322+0000 7fae91200640 1 Processor -- start 2024-01-15T18:14:33.324 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.322+0000 7fae91200640 1 -- start start 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae91200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae8c0b2210 0x7fae8c156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae91200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 0x7fae8c1e25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae91200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae8c0d1c50 0x7fae8c1e2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae91200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c1e7220 con 0x7fae8c0d1870 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae91200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c1e32f0 con 0x7fae8c0b2210 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae91200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8c1e35d0 con 0x7fae8c0d1c50 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 0x7fae8c1e25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae83e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 0x7fae8c1e25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37114/0 (socket says 172.21.15.2:37114) 2024-01-15T18:14:33.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.323+0000 7fae83e00640 1 -- 172.21.15.2:0/194899663 learned_addr learned my addr 172.21.15.2:0/194899663 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:33.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae8b400640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae8c0b2210 0x7fae8c156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:33.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae8be00640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae8c0d1c50 0x7fae8c1e2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83e00640 1 -- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae8c0d1c50 msgr2=0x7fae8c1e2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83e00640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae8c0d1c50 0x7fae8c1e2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83e00640 1 -- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae8c0b2210 msgr2=0x7fae8c156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83e00640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae8c0b2210 0x7fae8c156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83e00640 1 -- 172.21.15.2:0/194899663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae74009400 con 0x7fae8c0d1870 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae8b400640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae8c0b2210 0x7fae8c156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83e00640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 0x7fae8c1e25d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fae74009890 tx=0x7fae740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7403a6f0 con 0x7fae8c0d1870 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae7403a870 con 0x7fae8c0d1870 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae7403ac20 con 0x7fae8c0d1870 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.324+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae8c1e38b0 con 0x7fae8c0d1870 2024-01-15T18:14:33.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.325+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae8c157410 con 0x7fae8c0d1870 2024-01-15T18:14:33.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.326+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae8c074770 con 0x7fae8c0d1870 2024-01-15T18:14:33.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.327+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fae74043410 con 0x7fae8c0d1870 2024-01-15T18:14:33.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.328+0000 7fae83400640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae6406c430 0x7fae6406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:33.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.328+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fae740aa780 con 0x7fae8c0d1870 2024-01-15T18:14:33.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.328+0000 7fae8b400640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae6406c430 0x7fae6406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:33.332 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.330+0000 7fae8b400640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae6406c430 0x7fae6406e8f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fae7c006fd0 tx=0x7fae7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:33.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.333+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae74076e10 con 0x7fae8c0d1870 2024-01-15T18:14:33.483 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:33.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:14:33.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.568+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fae8c0d2e90 con 0x7fae8c0d1870 2024-01-15T18:14:33.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.753+0000 7fae83400640 1 -- 172.21.15.2:0/194899663 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 136+0+0 (secure 0 0 0) 0x7fae74035020 con 0x7fae8c0d1870 2024-01-15T18:14:33.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae6406c430 msgr2=0x7fae6406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:33.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae6406c430 0x7fae6406e8f0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fae7c006fd0 tx=0x7fae7c008040 comp rx=0 tx=0).stop 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 msgr2=0x7fae8c1e25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 0x7fae8c1e25d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7fae74009890 tx=0x7fae740046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 shutdown_connections 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fae6406c430 0x7fae6406e8f0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fae8c0d1c50 0x7fae8c1e2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fae8c0d1870 0x7fae8c1e25d0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 --2- 172.21.15.2:0/194899663 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fae8c0b2210 0x7fae8c156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 >> 172.21.15.2:0/194899663 conn(0x7fae8c0d30b0 msgr2=0x7fae8c0d3490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:33.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 shutdown_connections 2024-01-15T18:14:33.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:33.757+0000 7fae91200640 1 -- 172.21.15.2:0/194899663 wait complete. 2024-01-15T18:14:33.785 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:14:34.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.003+0000 7f04b2400640 1 Processor -- start 2024-01-15T18:14:34.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.003+0000 7f04b2400640 1 -- start start 2024-01-15T18:14:34.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.004+0000 7f04b2400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f04ac1439e0 0x7f04ac06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.004+0000 7f04b2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac06cdf0 0x7f04ac0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04b2400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ac06d250 con 0x7f04ac0cbed0 2024-01-15T18:14:34.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04b2400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ac143dc0 con 0x7f04ac1439e0 2024-01-15T18:14:34.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04b2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0cc300 con 0x7f04ac06cdf0 2024-01-15T18:14:34.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04ab400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f04ac1439e0 0x7f04ac06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04abe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f04ac0cbed0 0x7f04ac0b0c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:43512/0 (socket says 172.21.15.2:43512) 2024-01-15T18:14:34.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04abe00640 1 -- 172.21.15.2:0/432971368 learned_addr learned my addr 172.21.15.2:0/432971368 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04a2800640 1 --2- 172.21.15.2:0/432971368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac06cdf0 0x7f04ac0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04aaa00640 1 -- 172.21.15.2:0/432971368 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314444948 0 0) 0x7f04ac06d250 con 0x7f04ac0cbed0 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.005+0000 7f04aaa00640 1 -- 172.21.15.2:0/432971368 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f048c003520 con 0x7f04ac0cbed0 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.006+0000 7f04ab400640 1 -- 172.21.15.2:0/432971368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac06cdf0 msgr2=0x7f04ac0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.006+0000 7f04ab400640 1 --2- 172.21.15.2:0/432971368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac06cdf0 0x7f04ac0cb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.006+0000 7f04ab400640 1 -- 172.21.15.2:0/432971368 >> v1:172.21.15.2:6789/0 conn(0x7f04ac0cbed0 legacy=0x7f04ac0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.006+0000 7f04ab400640 1 -- 172.21.15.2:0/432971368 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac15b620 con 0x7f04ac1439e0 2024-01-15T18:14:34.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.007+0000 7f04ab400640 1 --2- 172.21.15.2:0/432971368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac1439e0 0x7f04ac06c8b0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f049c009d90 tx=0x7f049c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146425fc24634569 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.007+0000 7f04aaa00640 1 -- 172.21.15.2:0/432971368 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c02cc50 con 0x7f04ac1439e0 2024-01-15T18:14:34.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.007+0000 7f04aaa00640 1 -- 172.21.15.2:0/432971368 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049c02cdb0 con 0x7f04ac1439e0 2024-01-15T18:14:34.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.007+0000 7f04aaa00640 1 -- 172.21.15.2:0/432971368 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c034640 con 0x7f04ac1439e0 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 -- 172.21.15.2:0/432971368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac1439e0 msgr2=0x7f04ac06c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 --2- 172.21.15.2:0/432971368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac1439e0 0x7f04ac06c8b0 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f049c009d90 tx=0x7f049c02bea0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 -- 172.21.15.2:0/432971368 shutdown_connections 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 --2- 172.21.15.2:0/432971368 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac06cdf0 0x7f04ac0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 --2- 172.21.15.2:0/432971368 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac1439e0 0x7f04ac06c8b0 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 -- 172.21.15.2:0/432971368 >> 172.21.15.2:0/432971368 conn(0x7f04ac0b46d0 msgr2=0x7f04ac0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 -- 172.21.15.2:0/432971368 shutdown_connections 2024-01-15T18:14:34.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.008+0000 7f04b2400640 1 -- 172.21.15.2:0/432971368 wait complete. 2024-01-15T18:14:34.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.010+0000 7f04b2400640 1 Processor -- start 2024-01-15T18:14:34.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.010+0000 7f04b2400640 1 -- start start 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04b2400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac06cdf0 0x7f04ac1307d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04b2400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 0x7f04ac0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04b2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac1439e0 0x7f04ac0c4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04b2400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac0ab5d0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04b2400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac130e20 con 0x7f04ac06cdf0 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04b2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac131100 con 0x7f04ac1439e0 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04a2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 0x7f04ac0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04a2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 0x7f04ac0ab090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37122/0 (socket says 172.21.15.2:37122) 2024-01-15T18:14:34.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.011+0000 7f04a2800640 1 -- 172.21.15.2:0/1090177928 learned_addr learned my addr 172.21.15.2:0/1090177928 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:34.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a2800640 1 -- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac1439e0 msgr2=0x7f04ac0c4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a2800640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac1439e0 0x7f04ac0c4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a2800640 1 -- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac06cdf0 msgr2=0x7f04ac1307d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a2800640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac06cdf0 0x7f04ac1307d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a2800640 1 -- 172.21.15.2:0/1090177928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c009a70 con 0x7f04ac0cbed0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a2800640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 0x7f04ac0ab090 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f049400d800 tx=0x7f049400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049400b6c0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049400b820 con 0x7f04ac0cbed0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049400bbb0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac1313e0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.012+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac1319a0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.014+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04ac131b30 con 0x7f04ac0cbed0 2024-01-15T18:14:34.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.015+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f049400bd10 con 0x7f04ac0cbed0 2024-01-15T18:14:34.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.016+0000 7f04a3e00640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f048406c430 0x7f048406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.016+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f04940890f0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.016+0000 7f04ab400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f048406c430 0x7f048406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.017+0000 7f04ab400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f048406c430 0x7f048406e8f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f049c036000 tx=0x7f049c02e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.020+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f04940555d0 con 0x7f04ac0cbed0 2024-01-15T18:14:34.240 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.236+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f04ac0cc570 con 0x7f048406c430 2024-01-15T18:14:34.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.241+0000 7f04a3e00640 1 -- 172.21.15.2:0/1090177928 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f049c034440 con 0x7f048406c430 2024-01-15T18:14:34.244 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:14:34.244 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:14:34.244 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:14:34.244 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:14:34.244 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f048406c430 msgr2=0x7f048406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f048406c430 0x7f048406e8f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f049c036000 tx=0x7f049c02e000 comp rx=0 tx=0).stop 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 msgr2=0x7f04ac0ab090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 0x7f04ac0ab090 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f049400d800 tx=0x7f049400dcc0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 shutdown_connections 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f048406c430 0x7f048406e8f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f04ac1439e0 0x7f04ac0c4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f04ac0cbed0 0x7f04ac0ab090 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 --2- 172.21.15.2:0/1090177928 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f04ac06cdf0 0x7f04ac1307d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 >> 172.21.15.2:0/1090177928 conn(0x7f04ac0b46d0 msgr2=0x7f04ac0b1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 shutdown_connections 2024-01-15T18:14:34.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.243+0000 7f04b2400640 1 -- 172.21.15.2:0/1090177928 wait complete. 2024-01-15T18:14:34.266 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.489+0000 7f5069400640 1 Processor -- start 2024-01-15T18:14:34.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.490+0000 7f5069400640 1 -- start start 2024-01-15T18:14:34.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5069400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f50641609d0 0x7f50640cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5069400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f50640d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5069400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50640d0af0 con 0x7f50640d1d40 2024-01-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5069400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50640ce7b0 con 0x7f50641609d0 2024-01-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5069400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50640cdbe0 con 0x7f50640b3fa0 2024-01-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5063400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f50640d1d40 0x7f50640cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:43526/0 (socket says 172.21.15.2:43526) 2024-01-15T18:14:34.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.491+0000 7f5063400640 1 -- 172.21.15.2:0/2912096789 learned_addr learned my addr 172.21.15.2:0/2912096789 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5063e00640 1 --2- 172.21.15.2:0/2912096789 >> v2:172.21.15.137:3300/0 conn(0x7f50641609d0 0x7f50640cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5068a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f50640d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949212362 0 0) 0x7f50640d0af0 con 0x7f50640d1d40 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5044003520 con 0x7f50640d1d40 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3419588895 0 0) 0x7f5044003520 con 0x7f50640d1d40 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 msgr2=0x7f50640d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f50640d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 >> v2:172.21.15.137:3300/0 conn(0x7f50641609d0 msgr2=0x7f50640cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 --2- 172.21.15.2:0/2912096789 >> v2:172.21.15.137:3300/0 conn(0x7f50641609d0 0x7f50640cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640cde80 con 0x7f50640d1d40 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f504c0019f0 con 0x7f50640d1d40 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 >> v1:172.21.15.2:6789/0 conn(0x7f50640d1d40 legacy=0x7f50640cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5044003970 0x7f5044003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50440047c0 0x7f5044004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f5044005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50640ce7b0 con 0x7f5044003970 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50640cdbe0 con 0x7f50440047c0 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.492+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504400b980 con 0x7f50640b3fa0 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063e00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5044003970 0x7f5044003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063400640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50440047c0 0x7f5044004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5068a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f5044005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063e00640 1 -- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 msgr2=0x7f5044005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063e00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f5044005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063e00640 1 -- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50440047c0 msgr2=0x7f5044004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063e00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f50440047c0 0x7f5044004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5063e00640 1 -- 172.21.15.2:0/2912096789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5044003520 con 0x7f5044003970 2024-01-15T18:14:34.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.493+0000 7f5068a00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640b3fa0 0x7f5044005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.494+0000 7f5063e00640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5044003970 0x7f5044003d90 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f5054009ac0 tx=0x7f50440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a9f949fdb2f3336 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505402c6d0 con 0x7f5044003970 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f505402c830 con 0x7f5044003970 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5062a00640 1 -- 172.21.15.2:0/2912096789 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505402cbc0 con 0x7f5044003970 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5069400640 1 -- 172.21.15.2:0/2912096789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5044003970 msgr2=0x7f5044003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5069400640 1 --2- 172.21.15.2:0/2912096789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5044003970 0x7f5044003d90 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f5054009ac0 tx=0x7f50440057f0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5068a00640 1 -- 172.21.15.2:0/2912096789 reap_dead start 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5069400640 1 -- 172.21.15.2:0/2912096789 shutdown_connections 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5069400640 1 -- 172.21.15.2:0/2912096789 >> 172.21.15.2:0/2912096789 conn(0x7f506409c3e0 msgr2=0x7f50640d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5069400640 1 -- 172.21.15.2:0/2912096789 shutdown_connections 2024-01-15T18:14:34.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.495+0000 7f5069400640 1 -- 172.21.15.2:0/2912096789 wait complete. 2024-01-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.497+0000 7f5069400640 1 Processor -- start 2024-01-15T18:14:34.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.498+0000 7f5069400640 1 -- start start 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5069400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5044003970 0x7f50641f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5069400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 0x7f50641f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5069400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640cc260 0x7f50640b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5069400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641f3700 con 0x7f50640cbde0 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5069400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50641f39e0 con 0x7f5044003970 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5069400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50640ca320 con 0x7f50640cc260 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 0x7f50641f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 0x7f50641f2c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37150/0 (socket says 172.21.15.2:37150) 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 -- 172.21.15.2:0/2060324741 learned_addr learned my addr 172.21.15.2:0/2060324741 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:34.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063e00640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5044003970 0x7f50641f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5068a00640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640cc260 0x7f50640b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 -- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640cc260 msgr2=0x7f50640b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640cc260 0x7f50640b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 -- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5044003970 msgr2=0x7f50641f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5044003970 0x7f50641f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.499+0000 7f5063400640 1 -- 172.21.15.2:0/2060324741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640ca600 con 0x7f50640cbde0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.500+0000 7f5063400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 0x7f50641f2c40 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f504c0079e0 tx=0x7f504c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.500+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c00ee50 con 0x7f50640cbde0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.500+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504c013420 con 0x7f50640cbde0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.500+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504c0137d0 con 0x7f50640cbde0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.500+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50641f4340 con 0x7f50640cbde0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.500+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50641f3180 con 0x7f50640cbde0 2024-01-15T18:14:34.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.502+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5064074770 con 0x7f50640cbde0 2024-01-15T18:14:34.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.503+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f504c013930 con 0x7f50640cbde0 2024-01-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.504+0000 7f505be00640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f503c06c4b0 0x7f503c06e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.504+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f504c089290 con 0x7f50640cbde0 2024-01-15T18:14:34.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.504+0000 7f5063e00640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f503c06c4b0 0x7f503c06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.505+0000 7f5063e00640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f503c06c4b0 0x7f503c06e970 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5054036000 tx=0x7f5054030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.508+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f504c055840 con 0x7f50640cbde0 2024-01-15T18:14:34.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.701+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f506406ce40 con 0x7f50640cbde0 2024-01-15T18:14:34.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.702+0000 7f505be00640 1 -- 172.21.15.2:0/2060324741 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f504c058e60 con 0x7f50640cbde0 2024-01-15T18:14:34.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f503c06c4b0 msgr2=0x7f503c06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f503c06c4b0 0x7f503c06e970 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f5054036000 tx=0x7f5054030000 comp rx=0 tx=0).stop 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 msgr2=0x7f50641f2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 0x7f50641f2c40 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f504c0079e0 tx=0x7f504c00e530 comp rx=0 tx=0).stop 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 shutdown_connections 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f503c06c4b0 0x7f503c06e970 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f50640cc260 0x7f50640b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.704+0000 7f5069400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f50640cbde0 0x7f50641f2c40 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.705+0000 7f5069400640 1 --2- 172.21.15.2:0/2060324741 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5044003970 0x7f50641f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.705+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 >> 172.21.15.2:0/2060324741 conn(0x7f506409c3e0 msgr2=0x7f50640b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.705+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 shutdown_connections 2024-01-15T18:14:34.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.705+0000 7f5069400640 1 -- 172.21.15.2:0/2060324741 wait complete. 2024-01-15T18:14:34.727 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-15T18:14:34.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.949+0000 7fa0bb200640 1 Processor -- start 2024-01-15T18:14:34.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.950+0000 7fa0bb200640 1 -- start start 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.951+0000 7fa0bb200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa0bc0d1870 0x7fa0bc0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.951+0000 7fa0bb200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa0bc0d1c50 0x7fa0bc0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.951+0000 7fa0bb200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0bc0d1250 con 0x7fa0bc0b2210 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.951+0000 7fa0bb200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc0ce6d0 con 0x7fa0bc0d1870 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.951+0000 7fa0bb200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0bc0ce8e0 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.951+0000 7fa0b9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa0bc0d1870 0x7fa0bc0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0b9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa0bc0d1870 0x7fa0bc0d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37154/0 (socket says 172.21.15.2:37154) 2024-01-15T18:14:34.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0b9e00640 1 -- 172.21.15.2:0/2536306249 learned_addr learned my addr 172.21.15.2:0/2536306249 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:34.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0b9e00640 1 -- 172.21.15.2:0/2536306249 >> v2:172.21.15.137:3300/0 conn(0x7fa0bc0d1c50 msgr2=0x7fa0bc0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0ba800640 1 --2- 172.21.15.2:0/2536306249 >> v2:172.21.15.137:3300/0 conn(0x7fa0bc0d1c50 0x7fa0bc0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0b9e00640 1 --2- 172.21.15.2:0/2536306249 >> v2:172.21.15.137:3300/0 conn(0x7fa0bc0d1c50 0x7fa0bc0b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0b9e00640 1 -- 172.21.15.2:0/2536306249 >> v1:172.21.15.137:6789/0 conn(0x7fa0bc0b2210 legacy=0x7fa0bc15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:34.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0b9e00640 1 -- 172.21.15.2:0/2536306249 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc0cebc0 con 0x7fa0bc0d1870 2024-01-15T18:14:34.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.952+0000 7fa0ba800640 1 --2- 172.21.15.2:0/2536306249 >> v2:172.21.15.137:3300/0 conn(0x7fa0bc0d1c50 0x7fa0bc0b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:34.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.954+0000 7fa0b9e00640 1 --2- 172.21.15.2:0/2536306249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc0d3bd0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc08d7e0 tx=0x7fa0a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af317f50f89a5e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.954+0000 7fa0b8a00640 1 -- 172.21.15.2:0/2536306249 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4031870 con 0x7fa0bc0d1870 2024-01-15T18:14:34.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.954+0000 7fa0b8a00640 1 -- 172.21.15.2:0/2536306249 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a40319d0 con 0x7fa0bc0d1870 2024-01-15T18:14:34.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.954+0000 7fa0b8a00640 1 -- 172.21.15.2:0/2536306249 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4031d60 con 0x7fa0bc0d1870 2024-01-15T18:14:34.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 -- 172.21.15.2:0/2536306249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1870 msgr2=0x7fa0bc0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 --2- 172.21.15.2:0/2536306249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc0d3bd0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fa0bc08d7e0 tx=0x7fa0a402b910 comp rx=0 tx=0).stop 2024-01-15T18:14:34.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 -- 172.21.15.2:0/2536306249 shutdown_connections 2024-01-15T18:14:34.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 --2- 172.21.15.2:0/2536306249 >> v2:172.21.15.137:3300/0 conn(0x7fa0bc0d1c50 0x7fa0bc0b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 --2- 172.21.15.2:0/2536306249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc0d3bd0 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 -- 172.21.15.2:0/2536306249 >> 172.21.15.2:0/2536306249 conn(0x7fa0bc0d30b0 msgr2=0x7fa0bc0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:34.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 -- 172.21.15.2:0/2536306249 shutdown_connections 2024-01-15T18:14:34.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.955+0000 7fa0bb200640 1 -- 172.21.15.2:0/2536306249 wait complete. 2024-01-15T18:14:34.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.957+0000 7fa0bb200640 1 Processor -- start 2024-01-15T18:14:34.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.957+0000 7fa0bb200640 1 -- start start 2024-01-15T18:14:34.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.958+0000 7fa0bb200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0bc0b2210 0x7fa0bc15b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.958+0000 7fa0bb200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc1f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0bb200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 0x7fa0bc160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0bb200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc15dc70 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0bb200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc15dec0 con 0x7fa0bc0d1870 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0bb200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0bc15e1a0 con 0x7fa0bc0b2210 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0ba800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 0x7fa0bc160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0ba800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 0x7fa0bc160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37164/0 (socket says 172.21.15.2:37164) 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0ba800640 1 -- 172.21.15.2:0/4138822132 learned_addr learned my addr 172.21.15.2:0/4138822132 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0b9e00640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0bc0b2210 0x7fa0bc15b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0b9400640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc1f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0ba800640 1 -- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0bc0b2210 msgr2=0x7fa0bc15b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0ba800640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0bc0b2210 0x7fa0bc15b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.959+0000 7fa0ba800640 1 -- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0bc0d1870 msgr2=0x7fa0bc1f3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0ba800640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc1f3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0ba800640 1 -- 172.21.15.2:0/4138822132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a40094d0 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0ba800640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 0x7fa0bc160240 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c7c0 tx=0x7fa0b400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4004230 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b4004390 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b4007860 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0bc15e480 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.960+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0bc1f3540 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.962+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0bc15e9a0 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.963+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa0b40079c0 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.964+0000 7fa0b2000640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa09406c430 0x7fa09406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:34.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.964+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fa0b4088d20 con 0x7fa0bc0d1c50 2024-01-15T18:14:34.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.965+0000 7fa0b9e00640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa09406c430 0x7fa09406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:34.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.966+0000 7fa0b9e00640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa09406c430 0x7fa09406e8f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4036000 tx=0x7fa0a403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:34.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:34.969+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0bc15e9a0 con 0x7fa0bc0d1c50 2024-01-15T18:14:35.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.164+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7fa0bc15e9a0 con 0x7fa0bc0d1c50 2024-01-15T18:14:35.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.169+0000 7fa0b2000640 1 -- 172.21.15.2:0/4138822132 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v41)=0 v41) v1 ==== 131+0+0 (secure 0 0 0) 0x7fa0bc15e9a0 con 0x7fa0bc0d1c50 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa09406c430 msgr2=0x7fa09406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa09406c430 0x7fa09406e8f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4036000 tx=0x7fa0a403c040 comp rx=0 tx=0).stop 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 msgr2=0x7fa0bc160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 0x7fa0bc160240 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fa0b400c7c0 tx=0x7fa0b400cc80 comp rx=0 tx=0).stop 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 shutdown_connections 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa09406c430 0x7fa09406e8f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa0bc0d1c50 0x7fa0bc160240 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa0bc0d1870 0x7fa0bc1f3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 --2- 172.21.15.2:0/4138822132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa0bc0b2210 0x7fa0bc15b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 >> 172.21.15.2:0/4138822132 conn(0x7fa0bc0d30b0 msgr2=0x7fa0bc0b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:35.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.172+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 shutdown_connections 2024-01-15T18:14:35.177 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.173+0000 7fa0bb200640 1 -- 172.21.15.2:0/4138822132 wait complete. 2024-01-15T18:14:35.194 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-15T18:14:35.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.411+0000 7f10bd800640 1 Processor -- start 2024-01-15T18:14:35.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.411+0000 7f10bd800640 1 -- start start 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10bd800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f10b8071230 0x7f10b80716b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10bd800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f10b80cc3b0 0x7f10b80cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10bd800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b80b2fa0 con 0x7f10b8143620 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10bd800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b8071bf0 con 0x7f10b80cc3b0 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10bd800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b80b3280 con 0x7f10b8071230 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10bce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f10b80cc3b0 0x7f10b80cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10b7e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f10b8143620 0x7f10b80b2550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:43530/0 (socket says 172.21.15.2:43530) 2024-01-15T18:14:35.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.413+0000 7f10b7e00640 1 -- 172.21.15.2:0/1612369371 learned_addr learned my addr 172.21.15.2:0/1612369371 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:14:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.414+0000 7f10afe00640 1 --2- 172.21.15.2:0/1612369371 >> v2:172.21.15.137:3300/0 conn(0x7f10b8071230 0x7f10b80716b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.414+0000 7f10bce00640 1 -- 172.21.15.2:0/1612369371 >> v2:172.21.15.137:3300/0 conn(0x7f10b8071230 msgr2=0x7f10b80716b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.414+0000 7f10bce00640 1 --2- 172.21.15.2:0/1612369371 >> v2:172.21.15.137:3300/0 conn(0x7f10b8071230 0x7f10b80716b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.414+0000 7f10bce00640 1 -- 172.21.15.2:0/1612369371 >> v1:172.21.15.2:6789/0 conn(0x7f10b8143620 legacy=0x7f10b80b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.414+0000 7f10bce00640 1 -- 172.21.15.2:0/1612369371 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b80b3560 con 0x7f10b80cc3b0 2024-01-15T18:14:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.415+0000 7f10bce00640 1 --2- 172.21.15.2:0/1612369371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b80cc3b0 0x7f10b80cc820 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f10b806b380 tx=0x7f10b002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7434eca2ac654dea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.415+0000 7f10b7400640 1 -- 172.21.15.2:0/1612369371 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b002cc70 con 0x7f10b80cc3b0 2024-01-15T18:14:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.415+0000 7f10b7400640 1 -- 172.21.15.2:0/1612369371 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b002cdd0 con 0x7f10b80cc3b0 2024-01-15T18:14:35.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.415+0000 7f10b7400640 1 -- 172.21.15.2:0/1612369371 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b0034660 con 0x7f10b80cc3b0 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 -- 172.21.15.2:0/1612369371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b80cc3b0 msgr2=0x7f10b80cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 --2- 172.21.15.2:0/1612369371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b80cc3b0 0x7f10b80cc820 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f10b806b380 tx=0x7f10b002bec0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 -- 172.21.15.2:0/1612369371 shutdown_connections 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 --2- 172.21.15.2:0/1612369371 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b80cc3b0 0x7f10b80cc820 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 --2- 172.21.15.2:0/1612369371 >> v2:172.21.15.137:3300/0 conn(0x7f10b8071230 0x7f10b80716b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 -- 172.21.15.2:0/1612369371 >> 172.21.15.2:0/1612369371 conn(0x7f10b80b0840 msgr2=0x7f10b80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 -- 172.21.15.2:0/1612369371 shutdown_connections 2024-01-15T18:14:35.420 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.416+0000 7f10bd800640 1 -- 172.21.15.2:0/1612369371 wait complete. 2024-01-15T18:14:35.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.418+0000 7f10bd800640 1 Processor -- start 2024-01-15T18:14:35.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.418+0000 7f10bd800640 1 -- start start 2024-01-15T18:14:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10bd800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 0x7f10b814f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10bd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10b80cc3b0 0x7f10b81f14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10bd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10b8143620 0x7f10b81f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10bd800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b81f1f20 con 0x7f10b8071230 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10bd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b813bfc0 con 0x7f10b8143620 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10bd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b813c2a0 con 0x7f10b80cc3b0 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 0x7f10b814f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10b7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 0x7f10b814f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37176/0 (socket says 172.21.15.2:37176) 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.419+0000 7f10b7e00640 1 -- 172.21.15.2:0/3828149047 learned_addr learned my addr 172.21.15.2:0/3828149047 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10afe00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10b80cc3b0 0x7f10b81f14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10bce00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10b8143620 0x7f10b81f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b7e00640 1 -- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10b80cc3b0 msgr2=0x7f10b81f14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b7e00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10b80cc3b0 0x7f10b81f14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b7e00640 1 -- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10b8143620 msgr2=0x7f10b81f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b7e00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10b8143620 0x7f10b81f19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b7e00640 1 -- 172.21.15.2:0/3828149047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b0009a70 con 0x7f10b8071230 2024-01-15T18:14:35.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b7e00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 0x7f10b814f070 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c480 tx=0x7f10a800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a8005ce0 con 0x7f10b8071230 2024-01-15T18:14:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a8005e40 con 0x7f10b8071230 2024-01-15T18:14:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a80072f0 con 0x7f10b8071230 2024-01-15T18:14:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b813c580 con 0x7f10b8071230 2024-01-15T18:14:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.420+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b813cb40 con 0x7f10b8071230 2024-01-15T18:14:35.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.421+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b8074770 con 0x7f10b8071230 2024-01-15T18:14:35.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.422+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f10a8007510 con 0x7f10b8071230 2024-01-15T18:14:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.423+0000 7f10b4c00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f109006c430 0x7f109006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.423+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f10a8087f80 con 0x7f10b8071230 2024-01-15T18:14:35.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.424+0000 7f10afe00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f109006c430 0x7f109006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.425+0000 7f10afe00640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f109006c430 0x7f109006e8f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f10b8142ef0 tx=0x7f10a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:35.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.428+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10a80545a0 con 0x7f10b8071230 2024-01-15T18:14:35.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.622+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f10b8150cb0 con 0x7f10b8071230 2024-01-15T18:14:35.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.628+0000 7f10b4c00640 1 -- 172.21.15.2:0/3828149047 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v42)=0 v42) v1 ==== 137+0+0 (secure 0 0 0) 0x7f10a8017070 con 0x7f10b8071230 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f109006c430 msgr2=0x7f109006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f109006c430 0x7f109006e8f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f10b8142ef0 tx=0x7f10a0008040 comp rx=0 tx=0).stop 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 msgr2=0x7f10b814f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 0x7f10b814f070 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f10a800c480 tx=0x7f10a800c940 comp rx=0 tx=0).stop 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 shutdown_connections 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f109006c430 0x7f109006e8f0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f10b8143620 0x7f10b81f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f10b80cc3b0 0x7f10b81f14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 --2- 172.21.15.2:0/3828149047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f10b8071230 0x7f10b814f070 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 >> 172.21.15.2:0/3828149047 conn(0x7f10b80b0840 msgr2=0x7f10b80af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 shutdown_connections 2024-01-15T18:14:35.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.631+0000 7f10bd800640 1 -- 172.21.15.2:0/3828149047 wait complete. 2024-01-15T18:14:35.653 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-15T18:14:35.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.869+0000 7f4f24a00640 1 Processor -- start 2024-01-15T18:14:35.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.869+0000 7f4f24a00640 1 -- start start 2024-01-15T18:14:35.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f24a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4f20143d20 0x7f4f2015eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f24a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f200720d0 0x7f4f2015f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f24a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4f201447d0 0x7f4f200cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f24a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f20071430 con 0x7f4f20143d20 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f24a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f20144100 con 0x7f4f201447d0 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f24a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f20071090 con 0x7f4f200720d0 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f1f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4f20143d20 0x7f4f2015eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f1f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4f20143d20 0x7f4f2015eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37180/0 (socket says 172.21.15.2:37180) 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.870+0000 7f4f1f400640 1 -- 172.21.15.2:0/4287488696 learned_addr learned my addr 172.21.15.2:0/4287488696 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.871+0000 7f4f1f400640 1 -- 172.21.15.2:0/4287488696 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f200720d0 msgr2=0x7f4f2015f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.871+0000 7f4f1f400640 1 --2- 172.21.15.2:0/4287488696 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f200720d0 0x7f4f2015f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.871+0000 7f4f1f400640 1 -- 172.21.15.2:0/4287488696 >> v2:172.21.15.137:3300/0 conn(0x7f4f201447d0 msgr2=0x7f4f200cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.871+0000 7f4f1f400640 1 --2- 172.21.15.2:0/4287488696 >> v2:172.21.15.137:3300/0 conn(0x7f4f201447d0 0x7f4f200cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.871+0000 7f4f1f400640 1 -- 172.21.15.2:0/4287488696 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f20144560 con 0x7f4f20143d20 2024-01-15T18:14:35.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.872+0000 7f4f1f400640 1 --2- 172.21.15.2:0/4287488696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20143d20 0x7f4f2015eb50 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f4f2006e0a0 tx=0x7f4f1002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb045d19eb01aca server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:35.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.872+0000 7f4f1ea00640 1 -- 172.21.15.2:0/4287488696 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f100317e0 con 0x7f4f20143d20 2024-01-15T18:14:35.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.872+0000 7f4f1ea00640 1 -- 172.21.15.2:0/4287488696 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f10031940 con 0x7f4f20143d20 2024-01-15T18:14:35.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.872+0000 7f4f1ea00640 1 -- 172.21.15.2:0/4287488696 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10031cd0 con 0x7f4f20143d20 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 -- 172.21.15.2:0/4287488696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20143d20 msgr2=0x7f4f2015eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 --2- 172.21.15.2:0/4287488696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20143d20 0x7f4f2015eb50 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f4f2006e0a0 tx=0x7f4f1002b880 comp rx=0 tx=0).stop 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 -- 172.21.15.2:0/4287488696 shutdown_connections 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 --2- 172.21.15.2:0/4287488696 >> v2:172.21.15.137:3300/0 conn(0x7f4f201447d0 0x7f4f200cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 --2- 172.21.15.2:0/4287488696 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f200720d0 0x7f4f2015f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 --2- 172.21.15.2:0/4287488696 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f20143d20 0x7f4f2015eb50 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 -- 172.21.15.2:0/4287488696 >> 172.21.15.2:0/4287488696 conn(0x7f4f200b0840 msgr2=0x7f4f200b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 -- 172.21.15.2:0/4287488696 shutdown_connections 2024-01-15T18:14:35.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.873+0000 7f4f24a00640 1 -- 172.21.15.2:0/4287488696 wait complete. 2024-01-15T18:14:35.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.875+0000 7f4f24a00640 1 Processor -- start 2024-01-15T18:14:35.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.875+0000 7f4f24a00640 1 -- start start 2024-01-15T18:14:35.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f24a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f200720d0 0x7f4f200adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f24a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f20143d20 0x7f4f20152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f24a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 0x7f4f200b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f24a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f200b2b30 con 0x7f4f201447d0 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f24a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f20152ad0 con 0x7f4f200720d0 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f24a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f200b1c40 con 0x7f4f20143d20 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f1fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 0x7f4f200b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f1fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 0x7f4f200b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37186/0 (socket says 172.21.15.2:37186) 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.876+0000 7f4f1fe00640 1 -- 172.21.15.2:0/2173930480 learned_addr learned my addr 172.21.15.2:0/2173930480 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1f400640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f200720d0 0x7f4f200adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f17e00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f20143d20 0x7f4f20152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1fe00640 1 -- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f20143d20 msgr2=0x7f4f20152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1fe00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f20143d20 0x7f4f20152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1fe00640 1 -- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f200720d0 msgr2=0x7f4f200adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1fe00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f200720d0 0x7f4f200adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1fe00640 1 -- 172.21.15.2:0/2173930480 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f100094d0 con 0x7f4f201447d0 2024-01-15T18:14:35.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f1fe00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 0x7f4f200b2fc0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800b970 tx=0x7f4f1800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1800cb40 con 0x7f4f201447d0 2024-01-15T18:14:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1800ccc0 con 0x7f4f201447d0 2024-01-15T18:14:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f18014680 con 0x7f4f201447d0 2024-01-15T18:14:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.877+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f200b1f20 con 0x7f4f201447d0 2024-01-15T18:14:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.878+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f200b24e0 con 0x7f4f201447d0 2024-01-15T18:14:35.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.878+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f200b2ca0 con 0x7f4f201447d0 2024-01-15T18:14:35.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.880+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4f1801d3b0 con 0x7f4f201447d0 2024-01-15T18:14:35.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.880+0000 7f4f17400640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ef806c430 0x7f4ef806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:35.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.881+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f4f18088a40 con 0x7f4f201447d0 2024-01-15T18:14:35.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.881+0000 7f4f1f400640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ef806c430 0x7f4ef806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:35.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.882+0000 7f4f1f400640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ef806c430 0x7f4ef806e8f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4f10036000 tx=0x7f4f1003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:35.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:35.885+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f18054fb0 con 0x7f4f201447d0 2024-01-15T18:14:36.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.106+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f2015fa00 con 0x7f4ef806c430 2024-01-15T18:14:36.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:14:36.118+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:36.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:14:36.118+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:36.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:14:36.118+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:36.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:14:36.118+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:36.120 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:14:36.118+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:14:36.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.176+0000 7f4f17400640 1 -- 172.21.15.2:0/2173930480 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4f10031bb0 con 0x7f4ef806c430 2024-01-15T18:14:36.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ef806c430 msgr2=0x7f4ef806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ef806c430 0x7f4ef806e8f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4f10036000 tx=0x7f4f1003d040 comp rx=0 tx=0).stop 2024-01-15T18:14:36.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 msgr2=0x7f4f200b2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 0x7f4f200b2fc0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f4f1800b970 tx=0x7f4f1800be30 comp rx=0 tx=0).stop 2024-01-15T18:14:36.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 shutdown_connections 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ef806c430 0x7f4ef806e8f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f201447d0 0x7f4f200b2fc0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f20143d20 0x7f4f20152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 --2- 172.21.15.2:0/2173930480 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f200720d0 0x7f4f200adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 >> 172.21.15.2:0/2173930480 conn(0x7f4f200b0840 msgr2=0x7f4f2015e070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 shutdown_connections 2024-01-15T18:14:36.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.179+0000 7f4f24a00640 1 -- 172.21.15.2:0/2173930480 wait complete. 2024-01-15T18:14:36.202 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:14:36.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.423+0000 7f1bc5e00640 1 Processor -- start 2024-01-15T18:14:36.427 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.423+0000 7f1bc5e00640 1 -- start start 2024-01-15T18:14:36.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.424+0000 7f1bc5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bc00d1870 0x7f1bc00d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.424+0000 7f1bc5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00b2210 0x7f1bc015d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.424+0000 7f1bc5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1bc00d1c50 0x7f1bc015d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.424+0000 7f1bc5e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc00ce6d0 con 0x7f1bc00d1870 2024-01-15T18:14:36.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.424+0000 7f1bc5e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bc015db60 con 0x7f1bc00d1c50 2024-01-15T18:14:36.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.424+0000 7f1bc5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00d1250 con 0x7f1bc00b2210 2024-01-15T18:14:36.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bc00d1870 0x7f1bc00d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1bc00d1870 0x7f1bc00d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37200/0 (socket says 172.21.15.2:37200) 2024-01-15T18:14:36.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 -- 172.21.15.2:0/2250164992 learned_addr learned my addr 172.21.15.2:0/2250164992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bbfe00640 1 --2- 172.21.15.2:0/2250164992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00b2210 0x7f1bc015d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc5400640 1 --2- 172.21.15.2:0/2250164992 >> v2:172.21.15.137:3300/0 conn(0x7f1bc00d1c50 0x7f1bc015d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 -- 172.21.15.2:0/2250164992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00b2210 msgr2=0x7f1bc015d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 --2- 172.21.15.2:0/2250164992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00b2210 0x7f1bc015d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 -- 172.21.15.2:0/2250164992 >> v2:172.21.15.137:3300/0 conn(0x7f1bc00d1c50 msgr2=0x7f1bc015d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.425+0000 7f1bc4a00640 1 --2- 172.21.15.2:0/2250164992 >> v2:172.21.15.137:3300/0 conn(0x7f1bc00d1c50 0x7f1bc015d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.426+0000 7f1bc4a00640 1 -- 172.21.15.2:0/2250164992 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00ce8e0 con 0x7f1bc00d1870 2024-01-15T18:14:36.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.427+0000 7f1bc4a00640 1 --2- 172.21.15.2:0/2250164992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00d1870 0x7f1bc00d3bd0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0161120 tx=0x7f1bb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57afc02b48252f9e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:36.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.427+0000 7f1bbf400640 1 -- 172.21.15.2:0/2250164992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0031870 con 0x7f1bc00d1870 2024-01-15T18:14:36.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.427+0000 7f1bbf400640 1 -- 172.21.15.2:0/2250164992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb00319d0 con 0x7f1bc00d1870 2024-01-15T18:14:36.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.427+0000 7f1bbf400640 1 -- 172.21.15.2:0/2250164992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0031d60 con 0x7f1bc00d1870 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 -- 172.21.15.2:0/2250164992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00d1870 msgr2=0x7f1bc00d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/2250164992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00d1870 0x7f1bc00d3bd0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0161120 tx=0x7f1bb002b910 comp rx=0 tx=0).stop 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 -- 172.21.15.2:0/2250164992 shutdown_connections 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/2250164992 >> v2:172.21.15.137:3300/0 conn(0x7f1bc00d1c50 0x7f1bc015d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/2250164992 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00b2210 0x7f1bc015d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/2250164992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00d1870 0x7f1bc00d3bd0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 -- 172.21.15.2:0/2250164992 >> 172.21.15.2:0/2250164992 conn(0x7f1bc00d30b0 msgr2=0x7f1bc00d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 -- 172.21.15.2:0/2250164992 shutdown_connections 2024-01-15T18:14:36.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.428+0000 7f1bc5e00640 1 -- 172.21.15.2:0/2250164992 wait complete. 2024-01-15T18:14:36.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.430+0000 7f1bc5e00640 1 Processor -- start 2024-01-15T18:14:36.434 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.430+0000 7f1bc5e00640 1 -- start start 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 0x7f1bc015c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00d1870 0x7f1bc009cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc5e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bc00d1c50 0x7f1bc009d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc5e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00b7500 con 0x7f1bc00b2210 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc5e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00b77e0 con 0x7f1bc00d1c50 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc00b7ac0 con 0x7f1bc00d1870 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.431+0000 7f1bc4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 0x7f1bc015c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 0x7f1bc015c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37204/0 (socket says 172.21.15.2:37204) 2024-01-15T18:14:36.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 -- 172.21.15.2:0/940736431 learned_addr learned my addr 172.21.15.2:0/940736431 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc5400640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bc00d1c50 0x7f1bc009d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bbfe00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00d1870 0x7f1bc009cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 -- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00d1870 msgr2=0x7f1bc009cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00d1870 0x7f1bc009cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 -- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bc00d1c50 msgr2=0x7f1bc009d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bc00d1c50 0x7f1bc009d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 -- 172.21.15.2:0/940736431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb00094d0 con 0x7f1bc00b2210 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bbfe00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00d1870 0x7f1bc009cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.432+0000 7f1bc4a00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 0x7f1bc015c5c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1bb002b950 tx=0x7f1bb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.433+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb003ad30 con 0x7f1bc00b2210 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.433+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb00443f0 con 0x7f1bc00b2210 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.433+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb00447a0 con 0x7f1bc00b2210 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.433+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc00b7da0 con 0x7f1bc00b2210 2024-01-15T18:14:36.437 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.433+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc015cb00 con 0x7f1bc00b2210 2024-01-15T18:14:36.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.434+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc00b82c0 con 0x7f1bc00b2210 2024-01-15T18:14:36.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.435+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f1bb004d450 con 0x7f1bc00b2210 2024-01-15T18:14:36.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.436+0000 7f1bbcc00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b9806c430 0x7f1b9806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.436+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f1bb002d040 con 0x7f1bc00b2210 2024-01-15T18:14:36.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.437+0000 7f1bbfe00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b9806c430 0x7f1b9806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.438+0000 7f1bbfe00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b9806c430 0x7f1b9806e8f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00b1300 tx=0x7f1ba8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:36.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.440+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bc00b82c0 con 0x7f1bc00b2210 2024-01-15T18:14:36.668 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.664+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1bc00d0f20 con 0x7f1b9806c430 2024-01-15T18:14:36.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.676+0000 7f1bbcc00640 1 -- 172.21.15.2:0/940736431 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1ba8009db0 con 0x7f1b9806c430 2024-01-15T18:14:36.679 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b9806c430 msgr2=0x7f1b9806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b9806c430 0x7f1b9806e8f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f1bc00b1300 tx=0x7f1ba8008040 comp rx=0 tx=0).stop 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 msgr2=0x7f1bc015c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 0x7f1bc015c5c0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f1bb002b950 tx=0x7f1bb00046c0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 shutdown_connections 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1b9806c430 0x7f1b9806e8f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1bc00d1c50 0x7f1bc009d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1bc00d1870 0x7f1bc009cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 --2- 172.21.15.2:0/940736431 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1bc00b2210 0x7f1bc015c5c0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 >> 172.21.15.2:0/940736431 conn(0x7f1bc00d30b0 msgr2=0x7f1bc00afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:36.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.678+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 shutdown_connections 2024-01-15T18:14:36.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.679+0000 7f1bc5e00640 1 -- 172.21.15.2:0/940736431 wait complete. 2024-01-15T18:14:36.703 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:14:36.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.925+0000 7f2ee8e00640 1 Processor -- start 2024-01-15T18:14:36.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.925+0000 7f2ee8e00640 1 -- start start 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2ee8e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2ee4144c00 0x7f2ee40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2ee8e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ee40cdd70 0x7f2ee406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2ee8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee415c450 0x7f2ee406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2ee8e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee40ce230 con 0x7f2ee40cdd70 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2ee8e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee40ce510 con 0x7f2ee4144c00 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2ee8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee40ce7f0 con 0x7f2ee415c450 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.927+0000 7f2edbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ee40cdd70 0x7f2ee406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ee40cdd70 0x7f2ee406c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37208/0 (socket says 172.21.15.2:37208) 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 -- 172.21.15.2:0/3976003879 learned_addr learned my addr 172.21.15.2:0/3976003879 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2ee2800640 1 --2- 172.21.15.2:0/3976003879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee415c450 0x7f2ee406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2ee1e00640 1 --2- 172.21.15.2:0/3976003879 >> v2:172.21.15.137:3300/0 conn(0x7f2ee4144c00 0x7f2ee40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 -- 172.21.15.2:0/3976003879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee415c450 msgr2=0x7f2ee406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:36.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 --2- 172.21.15.2:0/3976003879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee415c450 0x7f2ee406ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 -- 172.21.15.2:0/3976003879 >> v2:172.21.15.137:3300/0 conn(0x7f2ee4144c00 msgr2=0x7f2ee40cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:14:36.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 --2- 172.21.15.2:0/3976003879 >> v2:172.21.15.137:3300/0 conn(0x7f2ee4144c00 0x7f2ee40cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.928+0000 7f2edbe00640 1 -- 172.21.15.2:0/3976003879 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40705f0 con 0x7f2ee40cdd70 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.929+0000 7f2edbe00640 1 --2- 172.21.15.2:0/3976003879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee40cdd70 0x7f2ee406c8f0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40b1c60 tx=0x7f2ecc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f182c2b6b973761e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee1400640 1 -- 172.21.15.2:0/3976003879 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc031870 con 0x7f2ee40cdd70 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee1400640 1 -- 172.21.15.2:0/3976003879 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc0319d0 con 0x7f2ee40cdd70 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee1400640 1 -- 172.21.15.2:0/3976003879 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc031d60 con 0x7f2ee40cdd70 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 -- 172.21.15.2:0/3976003879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee40cdd70 msgr2=0x7f2ee406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/3976003879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee40cdd70 0x7f2ee406c8f0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40b1c60 tx=0x7f2ecc02b910 comp rx=0 tx=0).stop 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 -- 172.21.15.2:0/3976003879 shutdown_connections 2024-01-15T18:14:36.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/3976003879 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee415c450 0x7f2ee406ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/3976003879 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee40cdd70 0x7f2ee406c8f0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/3976003879 >> v2:172.21.15.137:3300/0 conn(0x7f2ee4144c00 0x7f2ee40cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.930+0000 7f2ee8e00640 1 -- 172.21.15.2:0/3976003879 >> 172.21.15.2:0/3976003879 conn(0x7f2ee40b45f0 msgr2=0x7f2ee40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:36.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.931+0000 7f2ee8e00640 1 -- 172.21.15.2:0/3976003879 shutdown_connections 2024-01-15T18:14:36.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.931+0000 7f2ee8e00640 1 -- 172.21.15.2:0/3976003879 wait complete. 2024-01-15T18:14:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.932+0000 7f2ee8e00640 1 Processor -- start 2024-01-15T18:14:36.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.933+0000 7f2ee8e00640 1 -- start start 2024-01-15T18:14:36.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.933+0000 7f2ee8e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee40cdd70 0x7f2ee4142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.933+0000 7f2ee8e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee4144c00 0x7f2ee413f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee8e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 0x7f2ee40ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee8e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee4140180 con 0x7f2ee415c450 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee8e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee413c8a0 con 0x7f2ee40cdd70 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee8e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee413cb80 con 0x7f2ee4144c00 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 0x7f2ee40ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 0x7f2ee40ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:37210/0 (socket says 172.21.15.2:37210) 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 -- 172.21.15.2:0/4285635047 learned_addr learned my addr 172.21.15.2:0/4285635047 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee1e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee40cdd70 0x7f2ee4142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2edbe00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee4144c00 0x7f2ee413f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 -- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee4144c00 msgr2=0x7f2ee413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee4144c00 0x7f2ee413f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 -- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee40cdd70 msgr2=0x7f2ee4142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:36.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee40cdd70 0x7f2ee4142b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.934+0000 7f2ee2800640 1 -- 172.21.15.2:0/4285635047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc0094d0 con 0x7f2ee415c450 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.935+0000 7f2ee2800640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 0x7f2ee40ceee0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009fd0 tx=0x7f2edc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.935+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc014420 con 0x7f2ee415c450 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.935+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc014580 con 0x7f2ee415c450 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.935+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2edc014930 con 0x7f2ee415c450 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.935+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee413ce60 con 0x7f2ee415c450 2024-01-15T18:14:36.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.935+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee4140800 con 0x7f2ee415c450 2024-01-15T18:14:36.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.936+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee413c700 con 0x7f2ee415c450 2024-01-15T18:14:36.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.937+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 37) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2edc014a90 con 0x7f2ee415c450 2024-01-15T18:14:36.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.938+0000 7f2eda000640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2ebc06c4b0 0x7f2ebc06e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:14:36.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.938+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5849+0+0 (secure 0 0 0) 0x7f2edc0893b0 con 0x7f2ee415c450 2024-01-15T18:14:36.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.939+0000 7f2ee1e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2ebc06c4b0 0x7f2ebc06e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:14:36.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.940+0000 7f2ee1e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2ebc06c4b0 0x7f2ebc06e970 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4006fd0 tx=0x7f2ed4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:14:36.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:36.944+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ee413c700 con 0x7f2ee415c450 2024-01-15T18:14:37.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.166+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ee413c700 con 0x7f2ebc06c4b0 2024-01-15T18:14:37.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.177+0000 7f2eda000640 1 -- 172.21.15.2:0/4285635047 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f2ee413c700 con 0x7f2ebc06c4b0 2024-01-15T18:14:37.181 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156 2024-01-15T18:14:37.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2ebc06c4b0 msgr2=0x7f2ebc06e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2ebc06c4b0 0x7f2ebc06e970 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4006fd0 tx=0x7f2ed4008040 comp rx=0 tx=0).stop 2024-01-15T18:14:37.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 msgr2=0x7f2ee40ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:14:37.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 0x7f2ee40ceee0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f2edc009fd0 tx=0x7f2edc00e4b0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 shutdown_connections 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2ebc06c4b0 0x7f2ebc06e970 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ee415c450 0x7f2ee40ceee0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ee4144c00 0x7f2ee413f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 --2- 172.21.15.2:0/4285635047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ee40cdd70 0x7f2ee4142b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.179+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 >> 172.21.15.2:0/4285635047 conn(0x7f2ee40b45f0 msgr2=0x7f2ee4071a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.180+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 shutdown_connections 2024-01-15T18:14:37.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:14:37.180+0000 7f2ee8e00640 1 -- 172.21.15.2:0/4285635047 wait complete. 2024-01-15T18:14:37.203 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156 2024-01-15T18:14:37.203 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:14:37.203 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:37.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:37.256 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:14:53.311 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:14:53.312 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:14:53.312 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.9673 s, 67.2 MB/s 2024-01-15T18:14:53.313 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:14:53.314 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:14:53.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:14:53.369 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:15:03.623 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:03.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:07.401 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:15:07.401 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:15:07.401 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.9728 s, 76.8 MB/s 2024-01-15T18:15:07.403 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:15:07.403 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:15:07.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:07.462 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-15T18:15:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records in 2024-01-15T18:15:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:1024+0 records out 2024-01-15T18:15:20.863 INFO:teuthology.orchestra.run.smithi002.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.3373 s, 80.5 MB/s 2024-01-15T18:15:20.868 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-15T18:15:21.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.093+0000 7f9aece00640 1 Processor -- start 2024-01-15T18:15:21.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.093+0000 7f9aece00640 1 -- start start 2024-01-15T18:15:21.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.095+0000 7f9aece00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9ae815e0b0 0x7f9ae80d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.095+0000 7f9aece00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae806f440 con 0x7f9ae8146ba0 2024-01-15T18:15:21.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.095+0000 7f9aece00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae815e5f0 con 0x7f9ae80b32b0 2024-01-15T18:15:21.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.095+0000 7f9aece00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae8146f80 con 0x7f9ae815e0b0 2024-01-15T18:15:21.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.095+0000 7f9ae5e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9ae8146ba0 0x7f9ae80b2ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44756/0 (socket says 172.21.15.2:44756) 2024-01-15T18:15:21.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.095+0000 7f9ae5e00640 1 -- 172.21.15.2:0/563456035 learned_addr learned my addr 172.21.15.2:0/563456035 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370505763 0 0) 0x7f9ae806f440 con 0x7f9ae8146ba0 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ac8003520 con 0x7f9ae8146ba0 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae6800640 1 --2- 172.21.15.2:0/563456035 >> v2:172.21.15.137:3300/0 conn(0x7f9ae815e0b0 0x7f9ae80d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1153342728 0 0) 0x7f9ac8003520 con 0x7f9ae8146ba0 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 >> v2:172.21.15.137:3300/0 conn(0x7f9ae815e0b0 msgr2=0x7f9ae80d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 --2- 172.21.15.2:0/563456035 >> v2:172.21.15.137:3300/0 conn(0x7f9ae815e0b0 0x7f9ae80d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 >> v1:172.21.15.137:6789/0 conn(0x7f9ae80b32b0 legacy=0x7f9ae815d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae815e8d0 con 0x7f9ae8146ba0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9ad8001670 con 0x7f9ae8146ba0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae6800640 1 --2- 172.21.15.2:0/563456035 >> v2:172.21.15.137:3300/0 conn(0x7f9ae815e0b0 0x7f9ae80d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 >> v1:172.21.15.2:6789/0 conn(0x7f9ae8146ba0 legacy=0x7f9ae80b2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac80039c0 0x7f9ac8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80047c0 0x7f9ac8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.096+0000 7f9ae4a00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac80052a0 0x7f9ac800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae8146f80 con 0x7f9ac80039c0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac800c1c0 con 0x7f9ac80052a0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac800c380 con 0x7f9ac80047c0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac80039c0 0x7f9ac8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae6800640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac80052a0 0x7f9ac800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5400640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80047c0 0x7f9ac8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5e00640 1 -- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80047c0 msgr2=0x7f9ac8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:21.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80047c0 0x7f9ac8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5e00640 1 -- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac80052a0 msgr2=0x7f9ac800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:21.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac80052a0 0x7f9ac800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae5e00640 1 -- 172.21.15.2:0/563456035 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac8003520 con 0x7f9ac80039c0 2024-01-15T18:15:21.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae6800640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ac80052a0 0x7f9ac800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:21.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.097+0000 7f9ae6800640 1 -- 172.21.15.2:0/563456035 reap_dead start 2024-01-15T18:15:21.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac80039c0 0x7f9ac8003de0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c0e0 tx=0x7f9ac80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7c4208c7b671eb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad8005ad0 con 0x7f9ac80039c0 2024-01-15T18:15:21.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad8005c30 con 0x7f9ac80039c0 2024-01-15T18:15:21.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9ae4a00640 1 -- 172.21.15.2:0/563456035 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad802ca30 con 0x7f9ac80039c0 2024-01-15T18:15:21.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9aece00640 1 -- 172.21.15.2:0/563456035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac80039c0 msgr2=0x7f9ac8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9aece00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac80039c0 0x7f9ac8003de0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800c0e0 tx=0x7f9ac80057f0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9aece00640 1 -- 172.21.15.2:0/563456035 shutdown_connections 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9aece00640 1 --2- 172.21.15.2:0/563456035 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ac80039c0 0x7f9ac8003de0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.099+0000 7f9aece00640 1 -- 172.21.15.2:0/563456035 >> 172.21.15.2:0/563456035 conn(0x7f9ae80b6430 msgr2=0x7f9ae80b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.100+0000 7f9aece00640 1 -- 172.21.15.2:0/563456035 shutdown_connections 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.100+0000 7f9aece00640 1 -- 172.21.15.2:0/563456035 wait complete. 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.101+0000 7f9aece00640 1 Processor -- start 2024-01-15T18:15:21.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.102+0000 7f9aece00640 1 -- start start 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.102+0000 7f9aece00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80039c0 0x7f9ae815c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9aece00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 0x7f9ae80cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9aece00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae80cb820 0x7f9ae80cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9aece00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae81372a0 con 0x7f9ae8136de0 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9aece00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae80cd970 con 0x7f9ae80cb820 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9aece00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae80cd620 con 0x7f9ac80039c0 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 0x7f9ae80cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 0x7f9ae80cb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48582/0 (socket says 172.21.15.2:48582) 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 -- 172.21.15.2:0/623707440 learned_addr learned my addr 172.21.15.2:0/623707440 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80039c0 0x7f9ae815c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae6800640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae80cb820 0x7f9ae80cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 -- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80039c0 msgr2=0x7f9ae815c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80039c0 0x7f9ae815c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 -- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae80cb820 msgr2=0x7f9ae80cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae80cb820 0x7f9ae80cbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae5400640 1 -- 172.21.15.2:0/623707440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae80d0170 con 0x7f9ae8136de0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.103+0000 7f9ae6800640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae80cb820 0x7f9ae80cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.104+0000 7f9ae5400640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 0x7f9ae80cb2e0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000d4a0 tx=0x7f9ad000d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.104+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0005480 con 0x7f9ae8136de0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.104+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad0003610 con 0x7f9ae8136de0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.104+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae80d0450 con 0x7f9ae8136de0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.104+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad0003980 con 0x7f9ae8136de0 2024-01-15T18:15:21.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.104+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae81f65a0 con 0x7f9ae8136de0 2024-01-15T18:15:21.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.106+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae80686f0 con 0x7f9ae8136de0 2024-01-15T18:15:21.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.106+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 38) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f9ad0003ae0 con 0x7f9ae8136de0 2024-01-15T18:15:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.107+0000 7f9ade000640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ac006c4f0 0x7f9ac006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.107+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f9ad0088860 con 0x7f9ae8136de0 2024-01-15T18:15:21.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.108+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ac006c4f0 0x7f9ac006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.109+0000 7f9ae5e00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ac006c4f0 0x7f9ac006e9b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8034000 tx=0x7f9ad8009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.112+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ad0054e10 con 0x7f9ae8136de0 2024-01-15T18:15:21.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.341+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ae80cd7e0 con 0x7f9ac006c4f0 2024-01-15T18:15:21.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.378+0000 7f9ade000640 1 -- 172.21.15.2:0/623707440 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9ae80cd7e0 con 0x7f9ac006c4f0 2024-01-15T18:15:21.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.380+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ac006c4f0 msgr2=0x7f9ac006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.380+0000 7f9aece00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ac006c4f0 0x7f9ac006e9b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9ad8034000 tx=0x7f9ad8009030 comp rx=0 tx=0).stop 2024-01-15T18:15:21.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.380+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 msgr2=0x7f9ae80cb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.380+0000 7f9aece00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 0x7f9ae80cb2e0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f9ad000d4a0 tx=0x7f9ad000d960 comp rx=0 tx=0).stop 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 shutdown_connections 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ac006c4f0 0x7f9ac006e9b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae80cb820 0x7f9ae80cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae8136de0 0x7f9ae80cb2e0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 --2- 172.21.15.2:0/623707440 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ac80039c0 0x7f9ae815c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 >> 172.21.15.2:0/623707440 conn(0x7f9ae80b6430 msgr2=0x7f9ae80b6810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 shutdown_connections 2024-01-15T18:15:21.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.381+0000 7f9aece00640 1 -- 172.21.15.2:0/623707440 wait complete. 2024-01-15T18:15:21.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-15T18:15:21.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.624+0000 7fa7b1400640 1 Processor -- start 2024-01-15T18:15:21.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.624+0000 7fa7b1400640 1 -- start start 2024-01-15T18:15:21.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.626+0000 7fa7b1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa7ac15d270 0x7fa7ac0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.626+0000 7fa7b1400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fa7ac161370 0x7fa7ac06cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.626+0000 7fa7b1400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ac06c5e0 con 0x7fa7ac161890 2024-01-15T18:15:21.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.626+0000 7fa7b1400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ac06c8c0 con 0x7fa7ac15d270 2024-01-15T18:15:21.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.626+0000 7fa7b1400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ac071c00 con 0x7fa7ac161370 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.626+0000 7fa7abe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fa7ac15d270 0x7fa7ac0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7a3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa7ac161890 0x7fa7ac160c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44762/0 (socket says 172.21.15.2:44762) 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7a3e00640 1 -- 172.21.15.2:0/134112485 learned_addr learned my addr 172.21.15.2:0/134112485 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7b0a00640 1 --2- 172.21.15.2:0/134112485 >> v2:172.21.15.137:3300/0 conn(0x7fa7ac161370 0x7fa7ac06cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7abe00640 1 -- 172.21.15.2:0/134112485 >> v2:172.21.15.137:3300/0 conn(0x7fa7ac161370 msgr2=0x7fa7ac06cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7abe00640 1 --2- 172.21.15.2:0/134112485 >> v2:172.21.15.137:3300/0 conn(0x7fa7ac161370 0x7fa7ac06cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7abe00640 1 -- 172.21.15.2:0/134112485 >> v1:172.21.15.2:6789/0 conn(0x7fa7ac161890 legacy=0x7fa7ac160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.627+0000 7fa7abe00640 1 -- 172.21.15.2:0/134112485 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ac071ee0 con 0x7fa7ac15d270 2024-01-15T18:15:21.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.628+0000 7fa7abe00640 1 --2- 172.21.15.2:0/134112485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0b2620 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac06fed0 tx=0x7fa79c02b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ec9eabb59b4eab server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.628+0000 7fa7ab400640 1 -- 172.21.15.2:0/134112485 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c031710 con 0x7fa7ac15d270 2024-01-15T18:15:21.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7ab400640 1 -- 172.21.15.2:0/134112485 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c02f050 con 0x7fa7ac15d270 2024-01-15T18:15:21.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7ab400640 1 -- 172.21.15.2:0/134112485 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c031a90 con 0x7fa7ac15d270 2024-01-15T18:15:21.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7b1400640 1 -- 172.21.15.2:0/134112485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 msgr2=0x7fa7ac0b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7b1400640 1 --2- 172.21.15.2:0/134112485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0b2620 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac06fed0 tx=0x7fa79c02b7b0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7b1400640 1 -- 172.21.15.2:0/134112485 shutdown_connections 2024-01-15T18:15:21.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7b1400640 1 --2- 172.21.15.2:0/134112485 >> v2:172.21.15.137:3300/0 conn(0x7fa7ac161370 0x7fa7ac06cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.633 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7b1400640 1 --2- 172.21.15.2:0/134112485 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0b2620 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.629+0000 7fa7b1400640 1 -- 172.21.15.2:0/134112485 >> 172.21.15.2:0/134112485 conn(0x7fa7ac0b0840 msgr2=0x7fa7ac0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.630+0000 7fa7b1400640 1 -- 172.21.15.2:0/134112485 shutdown_connections 2024-01-15T18:15:21.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.630+0000 7fa7b1400640 1 -- 172.21.15.2:0/134112485 wait complete. 2024-01-15T18:15:21.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.631+0000 7fa7b1400640 1 Processor -- start 2024-01-15T18:15:21.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.631+0000 7fa7b1400640 1 -- start start 2024-01-15T18:15:21.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.632+0000 7fa7b1400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0ab070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.632+0000 7fa7b1400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ac161370 0x7fa7ac14a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.632+0000 7fa7b1400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ac161890 0x7fa7ac14a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.632+0000 7fa7b1400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac15a090 con 0x7fa7ac15d270 2024-01-15T18:15:21.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.632+0000 7fa7b1400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac14ac20 con 0x7fa7ac161370 2024-01-15T18:15:21.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.632+0000 7fa7b1400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac159480 con 0x7fa7ac161890 2024-01-15T18:15:21.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0ab070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0ab070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48612/0 (socket says 172.21.15.2:48612) 2024-01-15T18:15:21.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 -- 172.21.15.2:0/1419003400 learned_addr learned my addr 172.21.15.2:0/1419003400 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:21.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7a3e00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ac161370 0x7fa7ac14a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 -- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ac161890 msgr2=0x7fa7ac14a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7b0a00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ac161890 0x7fa7ac14a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ac161890 0x7fa7ac14a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 -- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ac161370 msgr2=0x7fa7ac14a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ac161370 0x7fa7ac14a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7abe00640 1 -- 172.21.15.2:0/1419003400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c009400 con 0x7fa7ac15d270 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.633+0000 7fa7a3e00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ac161370 0x7fa7ac14a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.634+0000 7fa7abe00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0ab070 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009890 tx=0x7fa79c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.634+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c033910 con 0x7fa7ac15d270 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.634+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa79c033a70 con 0x7fa7ac15d270 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.634+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa79c033e20 con 0x7fa7ac15d270 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.634+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ac159760 con 0x7fa7ac15d270 2024-01-15T18:15:21.639 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.634+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ac159d20 con 0x7fa7ac15d270 2024-01-15T18:15:21.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.635+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ac159eb0 con 0x7fa7ac15d270 2024-01-15T18:15:21.640 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.636+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 38) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa79c03a040 con 0x7fa7ac15d270 2024-01-15T18:15:21.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.637+0000 7fa7a8c00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa78406c510 0x7fa78406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:21.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.637+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fa79c0a9e40 con 0x7fa7ac15d270 2024-01-15T18:15:21.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.638+0000 7fa7a3e00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa78406c510 0x7fa78406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:21.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.639+0000 7fa7a3e00640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa78406c510 0x7fa78406e9d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac15c190 tx=0x7fa794009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:21.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.642+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa79c076390 con 0x7fa7ac15d270 2024-01-15T18:15:21.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.863+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7ac06d290 con 0x7fa78406c510 2024-01-15T18:15:21.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.875+0000 7fa7a8c00640 1 -- 172.21.15.2:0/1419003400 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fa794001c40 con 0x7fa78406c510 2024-01-15T18:15:21.879 INFO:teuthology.orchestra.run.smithi002.stdout:/volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156 2024-01-15T18:15:21.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.877+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa78406c510 msgr2=0x7fa78406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.877+0000 7fa7b1400640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa78406c510 0x7fa78406e9d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac15c190 tx=0x7fa794009040 comp rx=0 tx=0).stop 2024-01-15T18:15:21.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.877+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 msgr2=0x7fa7ac0ab070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.877+0000 7fa7b1400640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0ab070 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7fa79c009890 tx=0x7fa79c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 shutdown_connections 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa78406c510 0x7fa78406e9d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa7ac161890 0x7fa7ac14a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7ac161370 0x7fa7ac14a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 --2- 172.21.15.2:0/1419003400 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7ac15d270 0x7fa7ac0ab070 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 >> 172.21.15.2:0/1419003400 conn(0x7fa7ac0b0840 msgr2=0x7fa7ac0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 shutdown_connections 2024-01-15T18:15:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:21.878+0000 7fa7b1400640 1 -- 172.21.15.2:0/1419003400 wait complete. 2024-01-15T18:15:21.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:15:21.901 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156) 2024-01-15T18:15:21.963 INFO:teuthology.orchestra.run.smithi002.stdout:# file: volumes/_nogroup/sv1/51dcd4f4-168e-4798-93b6-e3bf3b0a5156 2024-01-15T18:15:21.963 INFO:teuthology.orchestra.run.smithi002.stdout:ceph.dir.rbytes="3221225472" 2024-01-15T18:15:21.963 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:15:21.964 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-15T18:15:21.965 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29951 2024-01-15T18:15:22.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.190+0000 7f7d96600640 1 Processor -- start 2024-01-15T18:15:22.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.191+0000 7f7d96600640 1 -- start start 2024-01-15T18:15:22.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.192+0000 7f7d96600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7d900b4c70 0x7f7d900d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.192+0000 7f7d96600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d9015d730 0x7f7d900b5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.192+0000 7f7d96600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7d900b8c30 0x7f7d900d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.192+0000 7f7d96600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d900b5590 con 0x7f7d900b4c70 2024-01-15T18:15:22.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.192+0000 7f7d96600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d900d1900 con 0x7f7d900b8c30 2024-01-15T18:15:22.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.192+0000 7f7d96600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900d1be0 con 0x7f7d9015d730 2024-01-15T18:15:22.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7d900b4c70 0x7f7d900d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7d900b4c70 0x7f7d900d1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48618/0 (socket says 172.21.15.2:48618) 2024-01-15T18:15:22.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 -- 172.21.15.2:0/4289697996 learned_addr learned my addr 172.21.15.2:0/4289697996 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:22.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8ea00640 1 --2- 172.21.15.2:0/4289697996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d9015d730 0x7f7d900b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/4289697996 >> v2:172.21.15.137:3300/0 conn(0x7f7d900b8c30 0x7f7d900d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 -- 172.21.15.2:0/4289697996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d9015d730 msgr2=0x7f7d900b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 --2- 172.21.15.2:0/4289697996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d9015d730 0x7f7d900b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 -- 172.21.15.2:0/4289697996 >> v2:172.21.15.137:3300/0 conn(0x7f7d900b8c30 msgr2=0x7f7d900d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 --2- 172.21.15.2:0/4289697996 >> v2:172.21.15.137:3300/0 conn(0x7f7d900b8c30 0x7f7d900d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8f400640 1 -- 172.21.15.2:0/4289697996 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d900d1ec0 con 0x7f7d900b4c70 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.193+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/4289697996 >> v2:172.21.15.137:3300/0 conn(0x7f7d900b8c30 0x7f7d900d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:22.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.194+0000 7f7d8f400640 1 --2- 172.21.15.2:0/4289697996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d900b4c70 0x7f7d900d1270 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f7d900b5bf0 tx=0x7f7d8002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292a2b95fa7f08e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.195+0000 7f7d8e000640 1 -- 172.21.15.2:0/4289697996 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8002ca80 con 0x7f7d900b4c70 2024-01-15T18:15:22.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.195+0000 7f7d8e000640 1 -- 172.21.15.2:0/4289697996 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8002cbe0 con 0x7f7d900b4c70 2024-01-15T18:15:22.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.195+0000 7f7d8e000640 1 -- 172.21.15.2:0/4289697996 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d800343f0 con 0x7f7d900b4c70 2024-01-15T18:15:22.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.195+0000 7f7d96600640 1 -- 172.21.15.2:0/4289697996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d900b4c70 msgr2=0x7f7d900d1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.195+0000 7f7d96600640 1 --2- 172.21.15.2:0/4289697996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d900b4c70 0x7f7d900d1270 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f7d900b5bf0 tx=0x7f7d8002bee0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 -- 172.21.15.2:0/4289697996 shutdown_connections 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 --2- 172.21.15.2:0/4289697996 >> v2:172.21.15.137:3300/0 conn(0x7f7d900b8c30 0x7f7d900d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 --2- 172.21.15.2:0/4289697996 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d9015d730 0x7f7d900b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 --2- 172.21.15.2:0/4289697996 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d900b4c70 0x7f7d900d1270 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 -- 172.21.15.2:0/4289697996 >> 172.21.15.2:0/4289697996 conn(0x7f7d90072210 msgr2=0x7f7d900cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 -- 172.21.15.2:0/4289697996 shutdown_connections 2024-01-15T18:15:22.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.196+0000 7f7d96600640 1 -- 172.21.15.2:0/4289697996 wait complete. 2024-01-15T18:15:22.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.198+0000 7f7d96600640 1 Processor -- start 2024-01-15T18:15:22.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.198+0000 7f7d96600640 1 -- start start 2024-01-15T18:15:22.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d96600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d900b4c70 0x7f7d9015e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d96600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d900b8c30 0x7f7d900cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d96600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 0x7f7d900cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d96600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900c8710 con 0x7f7d9015d730 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d96600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900c9250 con 0x7f7d900b8c30 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d96600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900c8920 con 0x7f7d900b4c70 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d8fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 0x7f7d900cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d8fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 0x7f7d900cc6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48628/0 (socket says 172.21.15.2:48628) 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.199+0000 7f7d8fe00640 1 -- 172.21.15.2:0/2309210238 learned_addr learned my addr 172.21.15.2:0/2309210238 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8f400640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d900b4c70 0x7f7d9015e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8ea00640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d900b8c30 0x7f7d900cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8fe00640 1 -- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d900b4c70 msgr2=0x7f7d9015e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d900b4c70 0x7f7d9015e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8fe00640 1 -- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d900b8c30 msgr2=0x7f7d900cac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d900b8c30 0x7f7d900cac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8fe00640 1 -- 172.21.15.2:0/2309210238 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d80009b30 con 0x7f7d9015d730 2024-01-15T18:15:22.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8ea00640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d900b8c30 0x7f7d900cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:22.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d8fe00640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 0x7f7d900cc6a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800c7f0 tx=0x7f7d8800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88007820 con 0x7f7d9015d730 2024-01-15T18:15:22.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d88007980 con 0x7f7d9015d730 2024-01-15T18:15:22.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d900c8c00 con 0x7f7d9015d730 2024-01-15T18:15:22.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.200+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d88007d30 con 0x7f7d9015d730 2024-01-15T18:15:22.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.201+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d900cb1d0 con 0x7f7d9015d730 2024-01-15T18:15:22.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.202+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d900c93c0 con 0x7f7d9015d730 2024-01-15T18:15:22.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.203+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 38) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f7d880113f0 con 0x7f7d9015d730 2024-01-15T18:15:22.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.204+0000 7f7d87400640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d6806c4c0 0x7f7d6806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.204+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f7d88088e80 con 0x7f7d9015d730 2024-01-15T18:15:22.208 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.205+0000 7f7d8f400640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d6806c4c0 0x7f7d6806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.206+0000 7f7d8f400640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d6806c4c0 0x7f7d6806e980 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7d80035000 tx=0x7f7d80030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.212 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.209+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d88055490 con 0x7f7d9015d730 2024-01-15T18:15:22.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.438+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29951", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d900b90c0 con 0x7f7d6806c4c0 2024-01-15T18:15:22.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.595+0000 7f7d87400640 1 -- 172.21.15.2:0/2309210238 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7d80044100 con 0x7f7d6806c4c0 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d6806c4c0 msgr2=0x7f7d6806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d6806c4c0 0x7f7d6806e980 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f7d80035000 tx=0x7f7d80030000 comp rx=0 tx=0).stop 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 msgr2=0x7f7d900cc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 0x7f7d900cc6a0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f7d8800c7f0 tx=0x7f7d8800ccb0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 shutdown_connections 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7d6806c4c0 0x7f7d6806e980 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7d9015d730 0x7f7d900cc6a0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7d900b8c30 0x7f7d900cac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 --2- 172.21.15.2:0/2309210238 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7d900b4c70 0x7f7d9015e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 >> 172.21.15.2:0/2309210238 conn(0x7f7d90072210 msgr2=0x7f7d900b70a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 shutdown_connections 2024-01-15T18:15:22.604 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.599+0000 7f7d96600640 1 -- 172.21.15.2:0/2309210238 wait complete. 2024-01-15T18:15:22.608 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.604+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.608 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.604+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.608 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.604+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.608 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.604+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.608 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.604+0000 7fe212e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.622 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29952 2024-01-15T18:15:22.632 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.628+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.632 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.628+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.632 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.628+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.632 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.628+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.632 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:22.628+0000 7fe213800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:22.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.841+0000 7f3c12e00640 1 Processor -- start 2024-01-15T18:15:22.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.841+0000 7f3c12e00640 1 -- start start 2024-01-15T18:15:22.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.843+0000 7f3c12e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 0x7f3c0c15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.843+0000 7f3c12e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c071090 con 0x7f3c0c1447d0 2024-01-15T18:15:22.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.843+0000 7f3c12e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c06d160 con 0x7f3c0c0720d0 2024-01-15T18:15:22.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.843+0000 7f3c12e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c071430 con 0x7f3c0c143d20 2024-01-15T18:15:22.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.843+0000 7f3c12400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3c0c1447d0 0x7f3c0c0cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:44772/0 (socket says 172.21.15.2:44772) 2024-01-15T18:15:22.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.843+0000 7f3c12400640 1 -- 172.21.15.2:0/4027249617 learned_addr learned my addr 172.21.15.2:0/4027249617 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c11a00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 0x7f3c0c15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857958833 0 0) 0x7f3c0c071090 con 0x7f3c0c1447d0 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bec003520 con 0x7f3c0c1447d0 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1451588435 0 0) 0x7f3bec003520 con 0x7f3c0c1447d0 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 msgr2=0x7f3c0c15eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 0x7f3c0c15eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 >> v1:172.21.15.137:6789/0 conn(0x7f3c0c0720d0 legacy=0x7f3c0c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c144560 con 0x7f3c0c1447d0 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f3c040019f0 con 0x7f3c0c1447d0 2024-01-15T18:15:22.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 >> v1:172.21.15.2:6789/0 conn(0x7f3c0c1447d0 legacy=0x7f3c0c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.844+0000 7f3c0be00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 0x7f3bec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c0be00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bec003f70 0x7f3bec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c0be00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c071430 con 0x7f3bec004d00 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec00bb70 con 0x7f3bec003f70 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec00bd30 con 0x7f3c0c143d20 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c12400640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c11000640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bec003f70 0x7f3bec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c11a00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 0x7f3bec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c12400640 1 -- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 msgr2=0x7f3bec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c12400640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c143d20 0x7f3bec0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c12400640 1 -- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bec003f70 msgr2=0x7f3bec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c12400640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bec003f70 0x7f3bec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c12400640 1 -- 172.21.15.2:0/4027249617 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec003520 con 0x7f3bec004d00 2024-01-15T18:15:22.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.845+0000 7f3c11000640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3bec003f70 0x7f3bec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:22.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.846+0000 7f3c12400640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400a760 tx=0x7f3bec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362d331a4a381008 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04034630 con 0x7f3bec004d00 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0402f070 con 0x7f3bec004d00 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c0be00640 1 -- 172.21.15.2:0/4027249617 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04034a50 con 0x7f3bec004d00 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c12e00640 1 -- 172.21.15.2:0/4027249617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 msgr2=0x7f3bec005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c12e00640 1 --2- 172.21.15.2:0/4027249617 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3bec005180 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f3c0400a760 tx=0x7f3bec0057f0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c12400640 1 -- 172.21.15.2:0/4027249617 reap_dead start 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c12e00640 1 -- 172.21.15.2:0/4027249617 shutdown_connections 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c12e00640 1 -- 172.21.15.2:0/4027249617 >> 172.21.15.2:0/4027249617 conn(0x7f3c0c0b0840 msgr2=0x7f3c0c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.847+0000 7f3c12e00640 1 -- 172.21.15.2:0/4027249617 shutdown_connections 2024-01-15T18:15:22.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.848+0000 7f3c12e00640 1 -- 172.21.15.2:0/4027249617 wait complete. 2024-01-15T18:15:22.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.849+0000 7f3c12e00640 1 Processor -- start 2024-01-15T18:15:22.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.849+0000 7f3c12e00640 1 -- start start 2024-01-15T18:15:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.850+0000 7f3c12e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3c0c1f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.850+0000 7f3c12e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c132980 0x7f3c0c1f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.850+0000 7f3c12e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c0c1f6280 0x7f3c0c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.850+0000 7f3c12e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c131fa0 con 0x7f3bec004d00 2024-01-15T18:15:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.850+0000 7f3c12e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1f6cc0 con 0x7f3c0c1f6280 2024-01-15T18:15:22.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.850+0000 7f3c12e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c0c1f2670 con 0x7f3c0c132980 2024-01-15T18:15:22.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3c0c1f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3c0c1f2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48640/0 (socket says 172.21.15.2:48640) 2024-01-15T18:15:22.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 -- 172.21.15.2:0/912894690 learned_addr learned my addr 172.21.15.2:0/912894690 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c12400640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c0c1f6280 0x7f3c0c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11000640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c132980 0x7f3c0c1f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 -- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c132980 msgr2=0x7f3c0c1f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c132980 0x7f3c0c1f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 -- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c0c1f6280 msgr2=0x7f3c0c0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c0c1f6280 0x7f3c0c0cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:22.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.851+0000 7f3c11a00640 1 -- 172.21.15.2:0/912894690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c1f6700 con 0x7f3bec004d00 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.852+0000 7f3c11a00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3c0c1f2130 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc010890 tx=0x7f3bfc010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.852+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc009610 con 0x7f3bec004d00 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.852+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc00eb60 con 0x7f3bec004d00 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.852+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc01c400 con 0x7f3bec004d00 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.852+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c1f69e0 con 0x7f3bec004d00 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.852+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c132460 con 0x7f3bec004d00 2024-01-15T18:15:22.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.853+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c074770 con 0x7f3bec004d00 2024-01-15T18:15:22.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.854+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 38) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f3bfc01c560 con 0x7f3bec004d00 2024-01-15T18:15:22.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.855+0000 7f3c09600640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3be406c4c0 0x7f3be406e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:22.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.855+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f3bfc0890a0 con 0x7f3bec004d00 2024-01-15T18:15:22.859 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.856+0000 7f3c11000640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3be406c4c0 0x7f3be406e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:22.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.857+0000 7f3c11000640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3be406c4c0 0x7f3be406e980 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000c00 tx=0x7f3bf4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:22.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:22.860+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bfc0556a0 con 0x7f3bec004d00 2024-01-15T18:15:23.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.090+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29952", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c0c15fa00 con 0x7f3be406c4c0 2024-01-15T18:15:23.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.245+0000 7f3c09600640 1 -- 172.21.15.2:0/912894690 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3bf400d040 con 0x7f3be406c4c0 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3be406c4c0 msgr2=0x7f3be406e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3be406c4c0 0x7f3be406e980 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4000c00 tx=0x7f3bf4009040 comp rx=0 tx=0).stop 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 msgr2=0x7f3c0c1f2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3c0c1f2130 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc010890 tx=0x7f3bfc010d50 comp rx=0 tx=0).stop 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 shutdown_connections 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3be406c4c0 0x7f3be406e980 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3c0c1f6280 0x7f3c0c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3c0c132980 0x7f3c0c1f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 --2- 172.21.15.2:0/912894690 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3bec004d00 0x7f3c0c1f2130 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 >> 172.21.15.2:0/912894690 conn(0x7f3c0c0b0840 msgr2=0x7f3c0c0ce8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:23.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 shutdown_connections 2024-01-15T18:15:23.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.248+0000 7f3c12e00640 1 -- 172.21.15.2:0/912894690 wait complete. 2024-01-15T18:15:23.274 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone29953 2024-01-15T18:15:23.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.494+0000 7f6a5cc00640 1 Processor -- start 2024-01-15T18:15:23.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.494+0000 7f6a5cc00640 1 -- start start 2024-01-15T18:15:23.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.496+0000 7f6a5cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 0x7f6a580acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.496+0000 7f6a5cc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5815dbc0 con 0x7f6a580b2b40 2024-01-15T18:15:23.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.496+0000 7f6a5cc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a5815dea0 con 0x7f6a5806c8f0 2024-01-15T18:15:23.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.496+0000 7f6a5cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5815e840 con 0x7f6a5815ae80 2024-01-15T18:15:23.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.496+0000 7f6a56800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f6a580b2b40 0x7f6a58161150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47296/0 (socket says 172.21.15.2:47296) 2024-01-15T18:15:23.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.496+0000 7f6a56800640 1 -- 172.21.15.2:0/196223887 learned_addr learned my addr 172.21.15.2:0/196223887 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a55e00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 0x7f6a580acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289639263 0 0) 0x7f6a5815dbc0 con 0x7f6a580b2b40 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a38003520 con 0x7f6a580b2b40 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285708084 0 0) 0x7f6a38003520 con 0x7f6a580b2b40 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 msgr2=0x7f6a580acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 0x7f6a580acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 >> v1:172.21.15.137:6789/0 conn(0x7f6a5806c8f0 legacy=0x7f6a580b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5815e180 con 0x7f6a580b2b40 2024-01-15T18:15:23.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f6a500019f0 con 0x7f6a580b2b40 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 >> v1:172.21.15.2:6789/0 conn(0x7f6a580b2b40 legacy=0x7f6a58161150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 0x7f6a380039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a38004d00 0x7f6a38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5815e840 con 0x7f6a38003f20 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3800ba60 con 0x7f6a38004d00 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.497+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3800bc20 con 0x7f6a5815ae80 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55400640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a56800640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a38004d00 0x7f6a38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55e00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 0x7f6a380039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55400640 1 -- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 msgr2=0x7f6a380039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55400640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5815ae80 0x7f6a380039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55400640 1 -- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a38004d00 msgr2=0x7f6a38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:23.503 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55400640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a38004d00 0x7f6a38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.498+0000 7f6a55400640 1 -- 172.21.15.2:0/196223887 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38003520 con 0x7f6a38003f20 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a55400640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a380047c0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f6a40001750 tx=0x7f6a380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67b3296127f84ba7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a400052d0 con 0x7f6a38003f20 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4002e050 con 0x7f6a38003f20 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a54a00640 1 -- 172.21.15.2:0/196223887 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40003830 con 0x7f6a38003f20 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a5cc00640 1 -- 172.21.15.2:0/196223887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 msgr2=0x7f6a380047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/196223887 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a380047c0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f6a40001750 tx=0x7f6a380057f0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a56800640 1 -- 172.21.15.2:0/196223887 reap_dead start 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a5cc00640 1 -- 172.21.15.2:0/196223887 shutdown_connections 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.500+0000 7f6a5cc00640 1 -- 172.21.15.2:0/196223887 >> 172.21.15.2:0/196223887 conn(0x7f6a580b45f0 msgr2=0x7f6a580b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.501+0000 7f6a5cc00640 1 -- 172.21.15.2:0/196223887 shutdown_connections 2024-01-15T18:15:23.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.501+0000 7f6a5cc00640 1 -- 172.21.15.2:0/196223887 wait complete. 2024-01-15T18:15:23.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.502+0000 7f6a5cc00640 1 Processor -- start 2024-01-15T18:15:23.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.502+0000 7f6a5cc00640 1 -- start start 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.503+0000 7f6a5cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a5815d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.503+0000 7f6a5cc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5814caf0 0x7f6a5814dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.503+0000 7f6a5cc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a580b1cc0 0x7f6a580b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.503+0000 7f6a5cc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580b2680 con 0x7f6a38003f20 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.503+0000 7f6a5cc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a580aeb70 con 0x7f6a580b1cc0 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.503+0000 7f6a5cc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a5814cf70 con 0x7f6a5814caf0 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a5815d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a5815d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54722/0 (socket says 172.21.15.2:54722) 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 -- 172.21.15.2:0/2717197276 learned_addr learned my addr 172.21.15.2:0/2717197276 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55400640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5814caf0 0x7f6a5814dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a56800640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a580b1cc0 0x7f6a580b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 -- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5814caf0 msgr2=0x7f6a5814dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5814caf0 0x7f6a5814dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 -- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a580b1cc0 msgr2=0x7f6a580b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a580b1cc0 0x7f6a580b2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a55e00640 1 -- 172.21.15.2:0/2717197276 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a5814e590 con 0x7f6a38003f20 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.504+0000 7f6a56800640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a580b1cc0 0x7f6a580b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:23.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.505+0000 7f6a55e00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a5815d260 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6a48011630 tx=0x7f6a48011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.505+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4800f920 con 0x7f6a38003f20 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.505+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4800fa80 con 0x7f6a38003f20 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.505+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a5814e870 con 0x7f6a38003f20 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.505+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4800fdf0 con 0x7f6a38003f20 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.505+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a5814ee30 con 0x7f6a38003f20 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.506+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a580aef10 con 0x7f6a38003f20 2024-01-15T18:15:23.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.507+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 38) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f6a48015460 con 0x7f6a38003f20 2024-01-15T18:15:23.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.509+0000 7f6a4e000640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a3006c4f0 0x7f6a3006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:23.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.509+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f6a48089100 con 0x7f6a38003f20 2024-01-15T18:15:23.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.509+0000 7f6a55400640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a3006c4f0 0x7f6a3006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:23.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.510+0000 7f6a55400640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a3006c4f0 0x7f6a3006e9b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6a40033000 tx=0x7f6a40009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:23.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.513+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a48055930 con 0x7f6a38003f20 2024-01-15T18:15:23.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:23.741+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone29953", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a580acd00 con 0x7f6a3006c4f0 2024-01-15T18:15:24.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:24.803+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 39) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f6a48055d00 con 0x7f6a38003f20 2024-01-15T18:15:26.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.630+0000 7f6a4e000640 1 -- 172.21.15.2:0/2717197276 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6a40005370 con 0x7f6a3006c4f0 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.633+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a3006c4f0 msgr2=0x7f6a3006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.633+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a3006c4f0 0x7f6a3006e9b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6a40033000 tx=0x7f6a40009030 comp rx=0 tx=0).stop 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.633+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 msgr2=0x7f6a5815d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.633+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a5815d260 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6a48011630 tx=0x7f6a48011af0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.633+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 shutdown_connections 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.633+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f6a3006c4f0 0x7f6a3006e9b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.634+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6a580b1cc0 0x7f6a580b2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.634+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6a5814caf0 0x7f6a5814dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.634+0000 7f6a5cc00640 1 --2- 172.21.15.2:0/2717197276 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6a38003f20 0x7f6a5815d260 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.634+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 >> 172.21.15.2:0/2717197276 conn(0x7f6a580b45f0 msgr2=0x7f6a58072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.634+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 shutdown_connections 2024-01-15T18:15:26.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.634+0000 7f6a5cc00640 1 -- 172.21.15.2:0/2717197276 wait complete. 2024-01-15T18:15:26.659 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-15T18:15:26.660 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone29953 2024-01-15T18:15:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.881+0000 7f02da800640 1 Processor -- start 2024-01-15T18:15:26.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.881+0000 7f02da800640 1 -- start start 2024-01-15T18:15:26.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.882+0000 7f02da800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d4099a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.882+0000 7f02da800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02d40d19a0 0x7f02d4099f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.882+0000 7f02da800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f02d409a480 0x7f02d40cef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.882+0000 7f02da800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d406ece0 con 0x7f02d409a480 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.882+0000 7f02da800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d406b5c0 con 0x7f02d40d19a0 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.882+0000 7f02da800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40cbc80 con 0x7f02d415d530 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f02d409a480 0x7f02d40cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f02d409a480 0x7f02d40cef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54732/0 (socket says 172.21.15.2:54732) 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d9e00640 1 -- 172.21.15.2:0/3423177433 learned_addr learned my addr 172.21.15.2:0/3423177433 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d9400640 1 --2- 172.21.15.2:0/3423177433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d4099a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d8a00640 1 --2- 172.21.15.2:0/3423177433 >> v2:172.21.15.137:3300/0 conn(0x7f02d40d19a0 0x7f02d4099f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d9e00640 1 -- 172.21.15.2:0/3423177433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 msgr2=0x7f02d4099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.883+0000 7f02d9e00640 1 --2- 172.21.15.2:0/3423177433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d4099a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.884+0000 7f02d9e00640 1 -- 172.21.15.2:0/3423177433 >> v2:172.21.15.137:3300/0 conn(0x7f02d40d19a0 msgr2=0x7f02d4099f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.884+0000 7f02d9e00640 1 --2- 172.21.15.2:0/3423177433 >> v2:172.21.15.137:3300/0 conn(0x7f02d40d19a0 0x7f02d4099f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.884+0000 7f02d9e00640 1 -- 172.21.15.2:0/3423177433 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d406ea40 con 0x7f02d409a480 2024-01-15T18:15:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.885+0000 7f02d9e00640 1 --2- 172.21.15.2:0/3423177433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d40cef60 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f02d4071650 tx=0x7f02cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3eecedab6e4a210 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.885+0000 7f02d3e00640 1 -- 172.21.15.2:0/3423177433 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc031870 con 0x7f02d409a480 2024-01-15T18:15:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.885+0000 7f02d3e00640 1 -- 172.21.15.2:0/3423177433 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc0319d0 con 0x7f02d409a480 2024-01-15T18:15:26.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.885+0000 7f02d3e00640 1 -- 172.21.15.2:0/3423177433 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc031d60 con 0x7f02d409a480 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 -- 172.21.15.2:0/3423177433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 msgr2=0x7f02d40cef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 --2- 172.21.15.2:0/3423177433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d40cef60 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f02d4071650 tx=0x7f02cc02b910 comp rx=0 tx=0).stop 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 -- 172.21.15.2:0/3423177433 shutdown_connections 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 --2- 172.21.15.2:0/3423177433 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d40cef60 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 --2- 172.21.15.2:0/3423177433 >> v2:172.21.15.137:3300/0 conn(0x7f02d40d19a0 0x7f02d4099f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 --2- 172.21.15.2:0/3423177433 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d4099a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 -- 172.21.15.2:0/3423177433 >> 172.21.15.2:0/3423177433 conn(0x7f02d40d0e40 msgr2=0x7f02d40d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 -- 172.21.15.2:0/3423177433 shutdown_connections 2024-01-15T18:15:26.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.886+0000 7f02da800640 1 -- 172.21.15.2:0/3423177433 wait complete. 2024-01-15T18:15:26.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.888+0000 7f02da800640 1 Processor -- start 2024-01-15T18:15:26.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.888+0000 7f02da800640 1 -- start start 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02da800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d4159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02da800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02d40d19a0 0x7f02d40b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02da800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d41587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02da800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d4159410 con 0x7f02d409a480 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02da800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d415d910 con 0x7f02d40d19a0 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02da800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d415db60 con 0x7f02d415d530 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02d9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d4159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02d9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d4159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54738/0 (socket says 172.21.15.2:54738) 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.889+0000 7f02d9400640 1 -- 172.21.15.2:0/2416923917 learned_addr learned my addr 172.21.15.2:0/2416923917 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d9400640 1 -- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 msgr2=0x7f02d41587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d9400640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d41587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d9400640 1 -- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02d40d19a0 msgr2=0x7f02d40b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d8a00640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02d40d19a0 0x7f02d40b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d9400640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02d40d19a0 0x7f02d40b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d9400640 1 -- 172.21.15.2:0/2416923917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02cc0094d0 con 0x7f02d409a480 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d9400640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d4159610 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f02c4007e40 tx=0x7f02c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4014620 con 0x7f02d409a480 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c400ee90 con 0x7f02d409a480 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d415de40 con 0x7f02d409a480 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40149c0 con 0x7f02d409a480 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.890+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d415e400 con 0x7f02d409a480 2024-01-15T18:15:26.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.892+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d4074770 con 0x7f02d409a480 2024-01-15T18:15:26.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.893+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 39) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f02c4014b20 con 0x7f02d409a480 2024-01-15T18:15:26.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.894+0000 7f02d1600640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02ac06c5e0 0x7f02ac06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:26.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.894+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f02c4089550 con 0x7f02d409a480 2024-01-15T18:15:26.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.895+0000 7f02d8a00640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02ac06c5e0 0x7f02ac06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:26.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.896+0000 7f02d8a00640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02ac06c5e0 0x7f02ac06eaa0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f02d4159b50 tx=0x7f02bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:26.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:26.899+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02c4055a70 con 0x7f02d409a480 2024-01-15T18:15:27.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:27.130+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone29953", "target": ["mon-mgr", ""]}) v1 -- 0x7f02d409a9e0 con 0x7f02ac06c5e0 2024-01-15T18:15:31.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.493+0000 7f02d1600640 1 -- 172.21.15.2:0/2416923917 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f02d409a9e0 con 0x7f02ac06c5e0 2024-01-15T18:15:31.496 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: "status": { 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: "state": "pending", 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: "source": { 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: "volume": "cephfs", 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: "subvolume": "sv1", 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: "snapshot": "ss1" 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:15:31.497 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02ac06c5e0 msgr2=0x7f02ac06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02ac06c5e0 0x7f02ac06eaa0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f02d4159b50 tx=0x7f02bc008040 comp rx=0 tx=0).stop 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 msgr2=0x7f02d4159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d4159610 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f02c4007e40 tx=0x7f02c400e4b0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 shutdown_connections 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f02ac06c5e0 0x7f02ac06eaa0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f02d415d530 0x7f02d41587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02d40d19a0 0x7f02d40b5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 --2- 172.21.15.2:0/2416923917 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f02d409a480 0x7f02d4159610 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 >> 172.21.15.2:0/2416923917 conn(0x7f02d40d0e40 msgr2=0x7f02d40b2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:31.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 shutdown_connections 2024-01-15T18:15:31.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.496+0000 7f02da800640 1 -- 172.21.15.2:0/2416923917 wait complete. 2024-01-15T18:15:31.519 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone29951 2024-01-15T18:15:31.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.742+0000 7fd6fac00640 1 Processor -- start 2024-01-15T18:15:31.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.743+0000 7fd6fac00640 1 -- start start 2024-01-15T18:15:31.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.744+0000 7fd6fac00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd6f406d000 0x7fd6f4165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:31.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.744+0000 7fd6fac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40ce230 con 0x7fd6f4144c00 2024-01-15T18:15:31.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.744+0000 7fd6fac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f40ce510 con 0x7fd6f415c450 2024-01-15T18:15:31.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.744+0000 7fd6fac00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f40ce7f0 con 0x7fd6f406d000 2024-01-15T18:15:31.748 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd6f4144c00 0x7fd6f40cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47300/0 (socket says 172.21.15.2:47300) 2024-01-15T18:15:31.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f3e00640 1 -- 172.21.15.2:0/4015197700 learned_addr learned my addr 172.21.15.2:0/4015197700 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:15:31.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f8c00640 1 --2- 172.21.15.2:0/4015197700 >> v2:172.21.15.2:3300/0 conn(0x7fd6f406d000 0x7fd6f4165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:31.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f2a00640 1 -- 172.21.15.2:0/4015197700 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 348880186 0 0) 0x7fd6f40ce230 con 0x7fd6f4144c00 2024-01-15T18:15:31.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f2a00640 1 -- 172.21.15.2:0/4015197700 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d4003520 con 0x7fd6f4144c00 2024-01-15T18:15:31.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f8c00640 1 -- 172.21.15.2:0/4015197700 >> v1:172.21.15.137:6789/0 conn(0x7fd6f415c450 legacy=0x7fd6f406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:31.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f8c00640 1 -- 172.21.15.2:0/4015197700 >> v1:172.21.15.2:6789/0 conn(0x7fd6f4144c00 legacy=0x7fd6f40cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.745+0000 7fd6f8c00640 1 -- 172.21.15.2:0/4015197700 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f40705f0 con 0x7fd6f406d000 2024-01-15T18:15:31.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6f8c00640 1 --2- 172.21.15.2:0/4015197700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f4165d60 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009d90 tx=0x7fd6ec02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd39cdfb07206ef7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:31.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6f2a00640 1 -- 172.21.15.2:0/4015197700 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec02cc60 con 0x7fd6f406d000 2024-01-15T18:15:31.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6f2a00640 1 -- 172.21.15.2:0/4015197700 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec02cdc0 con 0x7fd6f406d000 2024-01-15T18:15:31.750 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6f2a00640 1 -- 172.21.15.2:0/4015197700 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec034660 con 0x7fd6f406d000 2024-01-15T18:15:31.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6fac00640 1 -- 172.21.15.2:0/4015197700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 msgr2=0x7fd6f4165d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6fac00640 1 --2- 172.21.15.2:0/4015197700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f4165d60 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009d90 tx=0x7fd6ec02beb0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.747+0000 7fd6fac00640 1 -- 172.21.15.2:0/4015197700 shutdown_connections 2024-01-15T18:15:31.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.748+0000 7fd6fac00640 1 --2- 172.21.15.2:0/4015197700 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f4165d60 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.748+0000 7fd6fac00640 1 -- 172.21.15.2:0/4015197700 >> 172.21.15.2:0/4015197700 conn(0x7fd6f40b45f0 msgr2=0x7fd6f40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:31.751 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.748+0000 7fd6fac00640 1 -- 172.21.15.2:0/4015197700 shutdown_connections 2024-01-15T18:15:31.752 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.748+0000 7fd6fac00640 1 -- 172.21.15.2:0/4015197700 wait complete. 2024-01-15T18:15:31.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.749+0000 7fd6fac00640 1 Processor -- start 2024-01-15T18:15:31.753 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.750+0000 7fd6fac00640 1 -- start start 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.750+0000 7fd6fac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f40aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6fac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6f4144c00 0x7fd6f40af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6fac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6f415c450 0x7fd6f40b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6fac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40b34f0 con 0x7fd6f406d000 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6fac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40b0f60 con 0x7fd6f4144c00 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6fac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f40b2870 con 0x7fd6f415c450 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f40aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f40aeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54744/0 (socket says 172.21.15.2:54744) 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 -- 172.21.15.2:0/3116377888 learned_addr learned my addr 172.21.15.2:0/3116377888 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 -- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6f415c450 msgr2=0x7fd6f40b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6f415c450 0x7fd6f40b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.755 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 -- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6f4144c00 msgr2=0x7fd6f40af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:31.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6f4144c00 0x7fd6f40af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:31.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.751+0000 7fd6f3e00640 1 -- 172.21.15.2:0/3116377888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec009a70 con 0x7fd6f406d000 2024-01-15T18:15:31.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.752+0000 7fd6f3e00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f40aeb60 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c500 tx=0x7fd6e400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:31.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.752+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4005ce0 con 0x7fd6f406d000 2024-01-15T18:15:31.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.752+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4005e40 con 0x7fd6f406d000 2024-01-15T18:15:31.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.752+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4007330 con 0x7fd6f406d000 2024-01-15T18:15:31.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.752+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f40b2b50 con 0x7fd6f406d000 2024-01-15T18:15:31.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.752+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f40b3110 con 0x7fd6f406d000 2024-01-15T18:15:31.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.753+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f40b3630 con 0x7fd6f406d000 2024-01-15T18:15:31.757 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.754+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 39) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd6e4007550 con 0x7fd6f406d000 2024-01-15T18:15:31.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.755+0000 7fd6ebe00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6cc06c5e0 0x7fd6cc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:31.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.755+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fd6e4088240 con 0x7fd6f406d000 2024-01-15T18:15:31.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.756+0000 7fd6f3400640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6cc06c5e0 0x7fd6cc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:31.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.757+0000 7fd6f3400640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6cc06c5e0 0x7fd6cc06eaa0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc006fd0 tx=0x7fd6dc006bc0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:31.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.760+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6f40b3630 con 0x7fd6f406d000 2024-01-15T18:15:31.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:31.993+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone29951", "target": ["mon-mgr", ""]}) v1 -- 0x7fd6f40cead0 con 0x7fd6cc06c5e0 2024-01-15T18:15:33.759 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:33.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:15:36.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:36.497+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:36.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:36.497+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:36.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:36.497+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:36.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:36.497+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:36.500 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:15:36.497+0000 7fe211a00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:15:36.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.688+0000 7fd6ebe00640 1 -- 172.21.15.2:0/3116377888 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd6dc009280 con 0x7fd6cc06c5e0 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6cc06c5e0 msgr2=0x7fd6cc06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6cc06c5e0 0x7fd6cc06eaa0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc006fd0 tx=0x7fd6dc006bc0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 msgr2=0x7fd6f40aeb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f40aeb60 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400c500 tx=0x7fd6e400c9c0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 shutdown_connections 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6cc06c5e0 0x7fd6cc06eaa0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6f415c450 0x7fd6f40b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6f4144c00 0x7fd6f40af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 --2- 172.21.15.2:0/3116377888 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6f406d000 0x7fd6f40aeb60 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 >> 172.21.15.2:0/3116377888 conn(0x7fd6f40b45f0 msgr2=0x7fd6f40ad690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:36.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 shutdown_connections 2024-01-15T18:15:36.696 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.691+0000 7fd6fac00640 1 -- 172.21.15.2:0/3116377888 wait complete. 2024-01-15T18:15:36.717 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone29952 2024-01-15T18:15:36.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.949+0000 7f3d78000640 1 Processor -- start 2024-01-15T18:15:36.953 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.949+0000 7f3d78000640 1 -- start start 2024-01-15T18:15:36.954 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d78000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d70144c00 0x7f3d700cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d78000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d7006d000 0x7f3d70070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d78000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d700ce230 con 0x7f3d700cdd70 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d78000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d700ce510 con 0x7f3d7006d000 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d78000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700ce7f0 con 0x7f3d70144c00 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d74c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f3d700cdd70 0x7f3d7006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:60856/0 (socket says 172.21.15.2:60856) 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d74c00640 1 -- 172.21.15.2:0/4119702741 learned_addr learned my addr 172.21.15.2:0/4119702741 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:15:36.955 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.951+0000 7f3d76000640 1 --2- 172.21.15.2:0/4119702741 >> v2:172.21.15.2:3300/0 conn(0x7f3d7006d000 0x7f3d70070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:36.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.952+0000 7f3d75600640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d70144c00 0x7f3d700cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:36.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.952+0000 7f3d76000640 1 -- 172.21.15.2:0/4119702741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d70144c00 msgr2=0x7f3d700cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:15:36.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.952+0000 7f3d76000640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d70144c00 0x7f3d700cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.952+0000 7f3d76000640 1 -- 172.21.15.2:0/4119702741 >> v1:172.21.15.2:6789/0 conn(0x7f3d700cdd70 legacy=0x7f3d7006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:36.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.952+0000 7f3d76000640 1 -- 172.21.15.2:0/4119702741 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d700705f0 con 0x7f3d7006d000 2024-01-15T18:15:36.956 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.952+0000 7f3d75600640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d70144c00 0x7f3d700cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:15:36.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.953+0000 7f3d76000640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d7006d000 0x7f3d70070b10 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009d90 tx=0x7f3d6802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be2740fd0dc17a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:36.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.953+0000 7f3d6fe00640 1 -- 172.21.15.2:0/4119702741 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6802cc50 con 0x7f3d7006d000 2024-01-15T18:15:36.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.953+0000 7f3d6fe00640 1 -- 172.21.15.2:0/4119702741 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6802cdb0 con 0x7f3d7006d000 2024-01-15T18:15:36.957 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.953+0000 7f3d6fe00640 1 -- 172.21.15.2:0/4119702741 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68034660 con 0x7f3d7006d000 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 -- 172.21.15.2:0/4119702741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d7006d000 msgr2=0x7f3d70070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d7006d000 0x7f3d70070b10 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009d90 tx=0x7f3d6802bea0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 -- 172.21.15.2:0/4119702741 shutdown_connections 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d7006d000 0x7f3d70070b10 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 --2- 172.21.15.2:0/4119702741 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d70144c00 0x7f3d700cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 -- 172.21.15.2:0/4119702741 >> 172.21.15.2:0/4119702741 conn(0x7f3d700b45f0 msgr2=0x7f3d700b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 -- 172.21.15.2:0/4119702741 shutdown_connections 2024-01-15T18:15:36.958 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.954+0000 7f3d78000640 1 -- 172.21.15.2:0/4119702741 wait complete. 2024-01-15T18:15:36.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.956+0000 7f3d78000640 1 Processor -- start 2024-01-15T18:15:36.959 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.956+0000 7f3d78000640 1 -- start start 2024-01-15T18:15:36.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d78000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d7006d000 0x7f3d700b3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:36.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d78000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d700cdd70 0x7f3d700aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:36.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d78000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d70144c00 0x7f3d700b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:36.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d78000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700aff80 con 0x7f3d70144c00 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d78000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700af060 con 0x7f3d700cdd70 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d78000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700af300 con 0x7f3d7006d000 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d76000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d70144c00 0x7f3d700b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d76000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d70144c00 0x7f3d700b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:48294/0 (socket says 172.21.15.2:48294) 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d76000640 1 -- 172.21.15.2:0/3493927321 learned_addr learned my addr 172.21.15.2:0/3493927321 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d75600640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d7006d000 0x7f3d700b3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d74c00640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d700cdd70 0x7f3d700aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.957+0000 7f3d76000640 1 -- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d7006d000 msgr2=0x7f3d700b3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d76000640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d7006d000 0x7f3d700b3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d76000640 1 -- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d700cdd70 msgr2=0x7f3d700aeb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d76000640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d700cdd70 0x7f3d700aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d76000640 1 -- 172.21.15.2:0/3493927321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68009a70 con 0x7f3d70144c00 2024-01-15T18:15:36.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d76000640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d70144c00 0x7f3d700b03a0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009d90 tx=0x7f3d68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:36.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68040650 con 0x7f3d70144c00 2024-01-15T18:15:36.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d680407d0 con 0x7f3d70144c00 2024-01-15T18:15:36.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68040b80 con 0x7f3d70144c00 2024-01-15T18:15:36.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d78000640 1 -- 172.21.15.2:0/3493927321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d700af5e0 con 0x7f3d70144c00 2024-01-15T18:15:36.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.958+0000 7f3d78000640 1 -- 172.21.15.2:0/3493927321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d700afba0 con 0x7f3d70144c00 2024-01-15T18:15:36.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.959+0000 7f3d78000640 1 -- 172.21.15.2:0/3493927321 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d700686f0 con 0x7f3d70144c00 2024-01-15T18:15:36.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.960+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 39) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3d68040ce0 con 0x7f3d70144c00 2024-01-15T18:15:36.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.961+0000 7f3d6d600640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d4806c5e0 0x7f3d4806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:15:36.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.961+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f3d680aba00 con 0x7f3d70144c00 2024-01-15T18:15:36.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.962+0000 7f3d75600640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d4806c5e0 0x7f3d4806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:15:36.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.963+0000 7f3d75600640 1 --2- 172.21.15.2:0/3493927321 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d4806c5e0 0x7f3d4806eaa0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f3d6000a950 tx=0x7f3d60008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:15:36.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:36.966+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d68077f20 con 0x7f3d70144c00 2024-01-15T18:15:37.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:37.197+0000 7f3d78000640 1 -- 172.21.15.2:0/3493927321 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone29952", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d70074810 con 0x7f3d4806c5e0 2024-01-15T18:15:38.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:15:38.817+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 40) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3d6803cdc0 con 0x7f3d70144c00 2024-01-15T18:16:03.909 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:03.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:34.056 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:34.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:16:50.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:16:50.967+0000 7f3d6d600640 1 -- 172.21.15.2:0/3493927321 <== mon.0 v2:172.21.15.2:3300/0 8 ==== mgrmap(e 41) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f3d6803ca30 con 0x7f3d70144c00 2024-01-15T18:17:04.210 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:04.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:34.351 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:34.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:17:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-15T18:17:36.842 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-15T18:17:37.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.073+0000 7fef6be00640 1 Processor -- start 2024-01-15T18:17:37.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.073+0000 7fef6be00640 1 -- start start 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.074+0000 7fef6be00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fef6c15d3d0 0x7fef6c0b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d7b0 0x7fef6c0d1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fef6c099a00 0x7fef6c099eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6be00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6c09a3f0 con 0x7fef6c099a00 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6be00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6c0b27b0 con 0x7fef6c15d3d0 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c0b1fb0 con 0x7fef6c15d7b0 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fef6c099a00 0x7fef6c099eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fef6c099a00 0x7fef6c099eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51076/0 (socket says 172.21.15.2:51076) 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6b400640 1 -- 172.21.15.2:0/2991281753 learned_addr learned my addr 172.21.15.2:0/2991281753 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6a000640 1 --2- 172.21.15.2:0/2991281753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d7b0 0x7fef6c0d1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.075+0000 7fef6aa00640 1 --2- 172.21.15.2:0/2991281753 >> v2:172.21.15.137:3300/0 conn(0x7fef6c15d3d0 0x7fef6c0b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.076+0000 7fef6b400640 1 -- 172.21.15.2:0/2991281753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d7b0 msgr2=0x7fef6c0d1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.076+0000 7fef6b400640 1 --2- 172.21.15.2:0/2991281753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d7b0 0x7fef6c0d1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.076+0000 7fef6b400640 1 -- 172.21.15.2:0/2991281753 >> v2:172.21.15.137:3300/0 conn(0x7fef6c15d3d0 msgr2=0x7fef6c0b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.076+0000 7fef6b400640 1 --2- 172.21.15.2:0/2991281753 >> v2:172.21.15.137:3300/0 conn(0x7fef6c15d3d0 0x7fef6c0b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.076+0000 7fef6b400640 1 -- 172.21.15.2:0/2991281753 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6c0b1100 con 0x7fef6c099a00 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.077+0000 7fef6b400640 1 --2- 172.21.15.2:0/2991281753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c099a00 0x7fef6c099eb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fef6c06d7a0 tx=0x7fef6402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91988ca62613fadf server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.077+0000 7fef69600640 1 -- 172.21.15.2:0/2991281753 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6402cb00 con 0x7fef6c099a00 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.077+0000 7fef69600640 1 -- 172.21.15.2:0/2991281753 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6402cc60 con 0x7fef6c099a00 2024-01-15T18:17:37.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.077+0000 7fef69600640 1 -- 172.21.15.2:0/2991281753 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef640343f0 con 0x7fef6c099a00 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 -- 172.21.15.2:0/2991281753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c099a00 msgr2=0x7fef6c099eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 --2- 172.21.15.2:0/2991281753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c099a00 0x7fef6c099eb0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fef6c06d7a0 tx=0x7fef6402bf60 comp rx=0 tx=0).stop 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 -- 172.21.15.2:0/2991281753 shutdown_connections 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 --2- 172.21.15.2:0/2991281753 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c099a00 0x7fef6c099eb0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 --2- 172.21.15.2:0/2991281753 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d7b0 0x7fef6c0d1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 --2- 172.21.15.2:0/2991281753 >> v2:172.21.15.137:3300/0 conn(0x7fef6c15d3d0 0x7fef6c0b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 -- 172.21.15.2:0/2991281753 >> 172.21.15.2:0/2991281753 conn(0x7fef6c0d0ee0 msgr2=0x7fef6c0d12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 -- 172.21.15.2:0/2991281753 shutdown_connections 2024-01-15T18:17:37.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.078+0000 7fef6be00640 1 -- 172.21.15.2:0/2991281753 wait complete. 2024-01-15T18:17:37.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.080+0000 7fef6be00640 1 Processor -- start 2024-01-15T18:17:37.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.080+0000 7fef6be00640 1 -- start start 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef6c099a00 0x7fef6c0b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d3d0 0x7fef6c1f6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 0x7fef6c15e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c160990 con 0x7fef6c15d7b0 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c1f6e40 con 0x7fef6c099a00 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6c160e20 con 0x7fef6c15d3d0 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.081+0000 7fef6b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 0x7fef6c15e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 0x7fef6c15e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51090/0 (socket says 172.21.15.2:51090) 2024-01-15T18:17:37.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 -- 172.21.15.2:0/2162088940 learned_addr learned my addr 172.21.15.2:0/2162088940 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:37.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6aa00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef6c099a00 0x7fef6c0b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6a000640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d3d0 0x7fef6c1f6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 -- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d3d0 msgr2=0x7fef6c1f6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d3d0 0x7fef6c1f6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 -- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef6c099a00 msgr2=0x7fef6c0b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef6c099a00 0x7fef6c0b6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6b400640 1 -- 172.21.15.2:0/2162088940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef64009b30 con 0x7fef6c15d7b0 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6aa00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef6c099a00 0x7fef6c0b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.082+0000 7fef6a000640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d3d0 0x7fef6c1f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.083+0000 7fef6b400640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 0x7fef6c15e2c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fef64036040 tx=0x7fef64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.083+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef64009510 con 0x7fef6c15d7b0 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.083+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6403e450 con 0x7fef6c15d7b0 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.083+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6403e800 con 0x7fef6c15d7b0 2024-01-15T18:17:37.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.083+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef6c15fd80 con 0x7fef6c15d7b0 2024-01-15T18:17:37.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.083+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef6c160340 con 0x7fef6c15d7b0 2024-01-15T18:17:37.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.084+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6c160b40 con 0x7fef6c15d7b0 2024-01-15T18:17:37.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.086+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 41) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fef6403e960 con 0x7fef6c15d7b0 2024-01-15T18:17:37.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.086+0000 7fef63400640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4406c610 0x7fef4406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.087+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fef640ab8c0 con 0x7fef6c15d7b0 2024-01-15T18:17:37.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.087+0000 7fef6aa00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4406c610 0x7fef4406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.088+0000 7fef6aa00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4406c610 0x7fef4406ead0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fef6c06c580 tx=0x7fef5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:37.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.092+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef6c160b40 con 0x7fef6c15d7b0 2024-01-15T18:17:37.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.312+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fef6c06c7e0 con 0x7fef4406c610 2024-01-15T18:17:37.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.317+0000 7fef63400640 1 -- 172.21.15.2:0/2162088940 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fef6c06c7e0 con 0x7fef4406c610 2024-01-15T18:17:37.320 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:17:37.320 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:17:37.320 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "cephfs" 2024-01-15T18:17:37.320 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:17:37.320 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:17:37.322 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.318+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4406c610 msgr2=0x7fef4406ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.318+0000 7fef6be00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4406c610 0x7fef4406ead0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fef6c06c580 tx=0x7fef5c008040 comp rx=0 tx=0).stop 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.318+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 msgr2=0x7fef6c15e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.318+0000 7fef6be00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 0x7fef6c15e2c0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fef64036040 tx=0x7fef64004a40 comp rx=0 tx=0).stop 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 shutdown_connections 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4406c610 0x7fef4406ead0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef6c15d7b0 0x7fef6c15e2c0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef6c15d3d0 0x7fef6c1f6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 --2- 172.21.15.2:0/2162088940 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef6c099a00 0x7fef6c0b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 >> 172.21.15.2:0/2162088940 conn(0x7fef6c0d0ee0 msgr2=0x7fef6c15b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 shutdown_connections 2024-01-15T18:17:37.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.319+0000 7fef6be00640 1 -- 172.21.15.2:0/2162088940 wait complete. 2024-01-15T18:17:37.342 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-15T18:17:37.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.557+0000 7fcd86000640 1 Processor -- start 2024-01-15T18:17:37.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.557+0000 7fcd86000640 1 -- start start 2024-01-15T18:17:37.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.559+0000 7fcd86000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcd8006c920 0x7fcd800cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.559+0000 7fcd86000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd8006d410 0x7fcd80166f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.559+0000 7fcd86000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd800cd750 con 0x7fcd800cd2c0 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.559+0000 7fcd86000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8006eb20 con 0x7fcd8006c920 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.559+0000 7fcd86000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800cd9f0 con 0x7fcd8006d410 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcd8006c920 0x7fcd800cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcd8006c920 0x7fcd800cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51096/0 (socket says 172.21.15.2:51096) 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 -- 172.21.15.2:0/3823896306 learned_addr learned my addr 172.21.15.2:0/3823896306 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 -- 172.21.15.2:0/3823896306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd8006d410 msgr2=0x7fcd80166f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 --2- 172.21.15.2:0/3823896306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd8006d410 0x7fcd80166f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 -- 172.21.15.2:0/3823896306 >> v1:172.21.15.137:6789/0 conn(0x7fcd800cd2c0 legacy=0x7fcd8006cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:37.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.560+0000 7fcd7f400640 1 -- 172.21.15.2:0/3823896306 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8006aec0 con 0x7fcd8006c920 2024-01-15T18:17:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.561+0000 7fcd7f400640 1 --2- 172.21.15.2:0/3823896306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd800cf050 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009e50 tx=0x7fcd7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe54261a13d66a2d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.562+0000 7fcd7ea00640 1 -- 172.21.15.2:0/3823896306 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7002ca70 con 0x7fcd8006c920 2024-01-15T18:17:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.562+0000 7fcd7ea00640 1 -- 172.21.15.2:0/3823896306 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7002cbd0 con 0x7fcd8006c920 2024-01-15T18:17:37.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.562+0000 7fcd7ea00640 1 -- 172.21.15.2:0/3823896306 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd700343f0 con 0x7fcd8006c920 2024-01-15T18:17:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.562+0000 7fcd86000640 1 -- 172.21.15.2:0/3823896306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 msgr2=0x7fcd800cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.562+0000 7fcd86000640 1 --2- 172.21.15.2:0/3823896306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd800cf050 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009e50 tx=0x7fcd7002bed0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.563+0000 7fcd86000640 1 -- 172.21.15.2:0/3823896306 shutdown_connections 2024-01-15T18:17:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.563+0000 7fcd86000640 1 --2- 172.21.15.2:0/3823896306 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd8006d410 0x7fcd80166f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.563+0000 7fcd86000640 1 --2- 172.21.15.2:0/3823896306 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd800cf050 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.563+0000 7fcd86000640 1 -- 172.21.15.2:0/3823896306 >> 172.21.15.2:0/3823896306 conn(0x7fcd800cc170 msgr2=0x7fcd800cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:37.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.563+0000 7fcd86000640 1 -- 172.21.15.2:0/3823896306 shutdown_connections 2024-01-15T18:17:37.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.563+0000 7fcd86000640 1 -- 172.21.15.2:0/3823896306 wait complete. 2024-01-15T18:17:37.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.564+0000 7fcd86000640 1 Processor -- start 2024-01-15T18:17:37.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.565+0000 7fcd86000640 1 -- start start 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd86000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd80152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd86000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd8006d410 0x7fcd80140d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd86000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd800cd2c0 0x7fcd80141240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd86000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800b4420 con 0x7fcd8006c920 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd86000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800b4700 con 0x7fcd8006d410 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd86000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd800b49e0 con 0x7fcd800cd2c0 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd80152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd80152010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51098/0 (socket says 172.21.15.2:51098) 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 -- 172.21.15.2:0/2295040271 learned_addr learned my addr 172.21.15.2:0/2295040271 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:37.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 -- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd800cd2c0 msgr2=0x7fcd80141240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd800cd2c0 0x7fcd80141240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 -- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd8006d410 msgr2=0x7fcd80140d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd76800640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd8006d410 0x7fcd80140d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd8006d410 0x7fcd80140d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd7f400640 1 -- 172.21.15.2:0/2295040271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd70009b30 con 0x7fcd8006c920 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.566+0000 7fcd76800640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd8006d410 0x7fcd80140d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:37.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.567+0000 7fcd7f400640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd80152010 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009e50 tx=0x7fcd70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.567+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7003e450 con 0x7fcd8006c920 2024-01-15T18:17:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.567+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7003e5b0 con 0x7fcd8006c920 2024-01-15T18:17:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.567+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7003e960 con 0x7fcd8006c920 2024-01-15T18:17:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.567+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd800b4cc0 con 0x7fcd8006c920 2024-01-15T18:17:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.567+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd80152730 con 0x7fcd8006c920 2024-01-15T18:17:37.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.568+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd800b50a0 con 0x7fcd8006c920 2024-01-15T18:17:37.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.569+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 41) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fcd7003eac0 con 0x7fcd8006c920 2024-01-15T18:17:37.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.570+0000 7fcd77e00640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcd5806c5e0 0x7fcd5806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:37.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.571+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fcd700ab940 con 0x7fcd8006c920 2024-01-15T18:17:37.574 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.571+0000 7fcd76800640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcd5806c5e0 0x7fcd5806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:37.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.572+0000 7fcd76800640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcd5806c5e0 0x7fcd5806eaa0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcd800ad660 tx=0x7fcd68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:37.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.575+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd70077e60 con 0x7fcd8006c920 2024-01-15T18:17:37.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.793+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd8006b760 con 0x7fcd5806c5e0 2024-01-15T18:17:37.807 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:37.805+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:37.807 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:37.805+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:37.807 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:37.805+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:37.807 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:37.805+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:37.808 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:37.805+0000 7fe20c000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:17:37.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.826+0000 7fcd77e00640 1 -- 172.21.15.2:0/2295040271 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+185 (secure 0 0 0) 0x7fcd68005050 con 0x7fcd5806c5e0 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "sv1" 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone29952" 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone29953" 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "subvol_clone29951" 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:17:37.830 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcd5806c5e0 msgr2=0x7fcd5806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcd5806c5e0 0x7fcd5806eaa0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fcd800ad660 tx=0x7fcd68009040 comp rx=0 tx=0).stop 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 msgr2=0x7fcd80152010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd80152010 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7fcd70009e50 tx=0x7fcd70004a40 comp rx=0 tx=0).stop 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 shutdown_connections 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcd5806c5e0 0x7fcd5806eaa0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd800cd2c0 0x7fcd80141240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcd8006d410 0x7fcd80140d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 --2- 172.21.15.2:0/2295040271 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd8006c920 0x7fcd80152010 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.829+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 >> 172.21.15.2:0/2295040271 conn(0x7fcd800cc170 msgr2=0x7fcd8006d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.830+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 shutdown_connections 2024-01-15T18:17:37.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:37.830+0000 7fcd86000640 1 -- 172.21.15.2:0/2295040271 wait complete. 2024-01-15T18:17:37.855 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-15T18:17:38.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.076+0000 7f9a66800640 1 Processor -- start 2024-01-15T18:17:38.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.076+0000 7f9a66800640 1 -- start start 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.077+0000 7f9a66800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9a60143620 0x7f9a600b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a66800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 0x7f9a600cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a66800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a600b2fa0 con 0x7f9a600b2a90 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a66800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a600b3280 con 0x7f9a60143620 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a66800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b3560 con 0x7f9a60071940 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a64a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9a600b2a90 0x7f9a60071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47462/0 (socket says 172.21.15.2:47462) 2024-01-15T18:17:38.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a64a00640 1 -- 172.21.15.2:0/1553345365 learned_addr learned my addr 172.21.15.2:0/1553345365 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:17:38.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a65400640 1 --2- 172.21.15.2:0/1553345365 >> v2:172.21.15.137:3300/0 conn(0x7f9a60143620 0x7f9a600b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a65e00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 0x7f9a600cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.078+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610290208 0 0) 0x7f9a600b2fa0 con 0x7f9a600b2a90 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a40005730 con 0x7f9a600b2a90 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445329319 0 0) 0x7f9a40005730 con 0x7f9a600b2a90 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 msgr2=0x7f9a600cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 0x7f9a600cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 >> v2:172.21.15.137:3300/0 conn(0x7f9a60143620 msgr2=0x7f9a600b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 --2- 172.21.15.2:0/1553345365 >> v2:172.21.15.137:3300/0 conn(0x7f9a60143620 0x7f9a600b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a600b3840 con 0x7f9a600b2a90 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9a48001670 con 0x7f9a600b2a90 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 >> v1:172.21.15.2:6789/0 conn(0x7f9a600b2a90 legacy=0x7f9a60071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a40005b80 0x7f9a40005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 0x7f9a400069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40006f10 0x7f9a40007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.084 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b3280 con 0x7f9a40005b80 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a600b3560 con 0x7f9a40006f10 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a4000db90 con 0x7f9a60071940 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.079+0000 7f9a65400640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a40005b80 0x7f9a40005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a64a00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 0x7f9a400069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a65400640 1 -- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 msgr2=0x7f9a400069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a65400640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a60071940 0x7f9a400069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a65400640 1 -- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40006f10 msgr2=0x7f9a40007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a65e00640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40006f10 0x7f9a40007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a65400640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40006f10 0x7f9a40007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.080+0000 7f9a65400640 1 -- 172.21.15.2:0/1553345365 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a40005730 con 0x7f9a40005b80 2024-01-15T18:17:38.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.081+0000 7f9a65400640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a40005b80 0x7f9a40005fa0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009af0 tx=0x7f9a40007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70dcd770a2169408 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:38.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.081+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5002c710 con 0x7f9a40005b80 2024-01-15T18:17:38.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.081+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5002c870 con 0x7f9a40005b80 2024-01-15T18:17:38.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.081+0000 7f9a5fe00640 1 -- 172.21.15.2:0/1553345365 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5002cc00 con 0x7f9a40005b80 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a66800640 1 -- 172.21.15.2:0/1553345365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a40005b80 msgr2=0x7f9a40005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a66800640 1 --2- 172.21.15.2:0/1553345365 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a40005b80 0x7f9a40005fa0 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009af0 tx=0x7f9a40007a00 comp rx=0 tx=0).stop 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a65e00640 1 -- 172.21.15.2:0/1553345365 reap_dead start 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a66800640 1 -- 172.21.15.2:0/1553345365 shutdown_connections 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a66800640 1 -- 172.21.15.2:0/1553345365 >> 172.21.15.2:0/1553345365 conn(0x7f9a600b0840 msgr2=0x7f9a600b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a66800640 1 -- 172.21.15.2:0/1553345365 shutdown_connections 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.082+0000 7f9a66800640 1 -- 172.21.15.2:0/1553345365 wait complete. 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.084+0000 7f9a66800640 1 Processor -- start 2024-01-15T18:17:38.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.084+0000 7f9a66800640 1 -- start start 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a66800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40005b80 0x7f9a60139d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a66800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a600cee80 0x7f9a601e8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a66800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 0x7f9a601e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a66800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60138c70 con 0x7f9a600cf300 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a66800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60138f50 con 0x7f9a40005b80 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a66800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a60139230 con 0x7f9a600cee80 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a65e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 0x7f9a601e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a65e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 0x7f9a601e8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51116/0 (socket says 172.21.15.2:51116) 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.085+0000 7f9a65e00640 1 -- 172.21.15.2:0/1576619206 learned_addr learned my addr 172.21.15.2:0/1576619206 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:38.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a65e00640 1 -- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a600cee80 msgr2=0x7f9a601e8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a65e00640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a600cee80 0x7f9a601e8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a65e00640 1 -- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40005b80 msgr2=0x7f9a60139d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a65e00640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40005b80 0x7f9a60139d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a65e00640 1 -- 172.21.15.2:0/1576619206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a60139510 con 0x7f9a600cf300 2024-01-15T18:17:38.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a65e00640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 0x7f9a601e8ba0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f9a6006fca0 tx=0x7f9a5800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:38.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a58017400 con 0x7f9a600cf300 2024-01-15T18:17:38.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a58011040 con 0x7f9a600cf300 2024-01-15T18:17:38.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a580177e0 con 0x7f9a600cf300 2024-01-15T18:17:38.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.086+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a601397f0 con 0x7f9a600cf300 2024-01-15T18:17:38.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.087+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a601380f0 con 0x7f9a600cf300 2024-01-15T18:17:38.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.088+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a60074770 con 0x7f9a600cf300 2024-01-15T18:17:38.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.089+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 41) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f9a58017940 con 0x7f9a600cf300 2024-01-15T18:17:38.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.090+0000 7f9a5d600640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a3806c5e0 0x7f9a3806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.090+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f9a58089130 con 0x7f9a600cf300 2024-01-15T18:17:38.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.090+0000 7f9a65400640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a3806c5e0 0x7f9a3806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.091+0000 7f9a65400640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a3806c5e0 0x7f9a3806eaa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9a50036000 tx=0x7f9a5002e000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:38.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.094+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a58055650 con 0x7f9a600cf300 2024-01-15T18:17:38.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.325+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a6006c330 con 0x7f9a3806c5e0 2024-01-15T18:17:38.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.358+0000 7f9a5d600640 1 -- 172.21.15.2:0/1576619206 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f9a5002c9e0 con 0x7f9a3806c5e0 2024-01-15T18:17:38.361 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:38.361 INFO:teuthology.orchestra.run.smithi002.stdout:[ 2024-01-15T18:17:38.361 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-01-15T18:17:38.361 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "ss1" 2024-01-15T18:17:38.362 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-01-15T18:17:38.362 INFO:teuthology.orchestra.run.smithi002.stdout:] 2024-01-15T18:17:38.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a3806c5e0 msgr2=0x7f9a3806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a3806c5e0 0x7f9a3806eaa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9a50036000 tx=0x7f9a5002e000 comp rx=0 tx=0).stop 2024-01-15T18:17:38.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 msgr2=0x7f9a601e8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 0x7f9a601e8ba0 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f9a6006fca0 tx=0x7f9a5800f4a0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 shutdown_connections 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9a3806c5e0 0x7f9a3806eaa0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9a600cf300 0x7f9a601e8ba0 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.360+0000 7f9a66800640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9a600cee80 0x7f9a601e8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.361+0000 7f9a66800640 1 --2- 172.21.15.2:0/1576619206 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9a40005b80 0x7f9a60139d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.361+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 >> 172.21.15.2:0/1576619206 conn(0x7f9a600b0840 msgr2=0x7f9a601424e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.361+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 shutdown_connections 2024-01-15T18:17:38.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.361+0000 7f9a66800640 1 -- 172.21.15.2:0/1576619206 wait complete. 2024-01-15T18:17:38.383 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:38.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.610+0000 7f5a03e00640 1 Processor -- start 2024-01-15T18:17:38.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.610+0000 7f5a03e00640 1 -- start start 2024-01-15T18:17:38.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.612+0000 7f5a03e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5a040cd3d0 0x7f5a040cee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.612+0000 7f5a03e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406b5a0 0x7f5a0406ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.612+0000 7f5a03e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a040ce9a0 con 0x7f5a040b47b0 2024-01-15T18:17:38.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.612+0000 7f5a03e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a040ce7c0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.612+0000 7f5a03e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a040aee40 con 0x7f5a0406b5a0 2024-01-15T18:17:38.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.613+0000 7f5a02a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5a040cd3d0 0x7f5a040cee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.613+0000 7f5a02a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5a040cd3d0 0x7f5a040cee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:35922/0 (socket says 172.21.15.2:35922) 2024-01-15T18:17:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.613+0000 7f5a02a00640 1 -- 172.21.15.2:0/2188417130 learned_addr learned my addr 172.21.15.2:0/2188417130 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.613+0000 7f5a03400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406b5a0 0x7f5a0406ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.614+0000 7f5a02a00640 1 -- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406b5a0 msgr2=0x7f5a0406ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.614+0000 7f5a02a00640 1 --2- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406b5a0 0x7f5a0406ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.614+0000 7f5a02a00640 1 -- 172.21.15.2:0/2188417130 >> v1:172.21.15.137:6789/0 conn(0x7f5a040b47b0 legacy=0x7f5a0406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.614+0000 7f5a02a00640 1 -- 172.21.15.2:0/2188417130 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a040b25d0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.614+0000 7f5a03400640 1 --2- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406b5a0 0x7f5a0406ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:38.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.616+0000 7f5a02a00640 1 --2- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a040cd3d0 0x7f5a040cee30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009e50 tx=0x7f59f402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56dee59babee92cd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:38.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.616+0000 7f5a01600640 1 -- 172.21.15.2:0/2188417130 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f402ca80 con 0x7f5a040cd3d0 2024-01-15T18:17:38.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.616+0000 7f5a01600640 1 -- 172.21.15.2:0/2188417130 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f402cbe0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.616+0000 7f5a01600640 1 -- 172.21.15.2:0/2188417130 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f40343f0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.617+0000 7f5a03e00640 1 -- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a040cd3d0 msgr2=0x7f5a040cee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.617+0000 7f5a03e00640 1 --2- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a040cd3d0 0x7f5a040cee30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f59f4009e50 tx=0x7f59f402bee0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.617+0000 7f5a03e00640 1 -- 172.21.15.2:0/2188417130 shutdown_connections 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.617+0000 7f5a03e00640 1 --2- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406b5a0 0x7f5a0406ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.617+0000 7f5a03e00640 1 --2- 172.21.15.2:0/2188417130 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a040cd3d0 0x7f5a040cee30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.617+0000 7f5a03e00640 1 -- 172.21.15.2:0/2188417130 >> 172.21.15.2:0/2188417130 conn(0x7f5a040cbf70 msgr2=0x7f5a040cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.618+0000 7f5a03e00640 1 -- 172.21.15.2:0/2188417130 shutdown_connections 2024-01-15T18:17:38.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.618+0000 7f5a03e00640 1 -- 172.21.15.2:0/2188417130 wait complete. 2024-01-15T18:17:38.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.620+0000 7f5a03e00640 1 Processor -- start 2024-01-15T18:17:38.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.620+0000 7f5a03e00640 1 -- start start 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a0406b5a0 0x7f5a0414f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a040cd3d0 0x7f5a040afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406e9c0 0x7f5a040aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a041573e0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04157660 con 0x7f5a0406b5a0 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a04157940 con 0x7f5a0406e9c0 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a02000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a040cd3d0 0x7f5a040afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a02000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a040cd3d0 0x7f5a040afa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51122/0 (socket says 172.21.15.2:51122) 2024-01-15T18:17:38.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a02000640 1 -- 172.21.15.2:0/69375333 learned_addr learned my addr 172.21.15.2:0/69375333 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a02a00640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a0406b5a0 0x7f5a0414f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.622+0000 7f5a03400640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406e9c0 0x7f5a040aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02000640 1 -- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406e9c0 msgr2=0x7f5a040aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02000640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5a0406e9c0 0x7f5a040aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02000640 1 -- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a0406b5a0 msgr2=0x7f5a0414f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02000640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a0406b5a0 0x7f5a0414f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02000640 1 -- 172.21.15.2:0/69375333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc009400 con 0x7f5a040cd3d0 2024-01-15T18:17:38.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02a00640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5a0406b5a0 0x7f5a0414f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a02000640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5a040cd3d0 0x7f5a040afa20 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f59ec00c3c0 tx=0x7f59ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec0052d0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ec005450 con 0x7f5a040cd3d0 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f5a03e00640 1 -- 172.21.15.2:0/69375333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59f4009b30 con 0x7f5a040cd3d0 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.623+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59ec003830 con 0x7f5a040cd3d0 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.624+0000 7f5a03e00640 1 -- 172.21.15.2:0/69375333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a04157f00 con 0x7f5a040cd3d0 2024-01-15T18:17:38.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.625+0000 7f5a03e00640 1 -- 172.21.15.2:0/69375333 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a041580f0 con 0x7f5a040cd3d0 2024-01-15T18:17:38.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.626+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 41) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f59ec003990 con 0x7f5a040cd3d0 2024-01-15T18:17:38.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.627+0000 7f59faa00640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59dc06c610 0x7f59dc06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:38.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.627+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f59ec088070 con 0x7f5a040cd3d0 2024-01-15T18:17:38.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.627+0000 7f5a02a00640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59dc06c610 0x7f59dc06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:38.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.629+0000 7f5a02a00640 1 --2- 172.21.15.2:0/69375333 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f59dc06c610 0x7f59dc06ead0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f59f4035000 tx=0x7f59f4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:38.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.631+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59ec054590 con 0x7f5a040cd3d0 2024-01-15T18:17:38.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.854+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f59ec054960 con 0x7f5a040cd3d0 2024-01-15T18:17:38.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.863+0000 7f5a03e00640 1 -- 172.21.15.2:0/69375333 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a04166e20 con 0x7f59dc06c610 2024-01-15T18:17:38.879 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:38.877+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:38.877+0000 7f59faa00640 1 -- 172.21.15.2:0/69375333 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f59f40440f0 con 0x7f59dc06c610 2024-01-15T18:17:38.880 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:38.880 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:38.900 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:40.901 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:41.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.136+0000 7fc024600640 1 Processor -- start 2024-01-15T18:17:41.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.136+0000 7fc024600640 1 -- start start 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.138+0000 7fc024600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc01c146ba0 0x7fc01c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.138+0000 7fc024600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc01c15e0b0 0x7fc01c0b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.138+0000 7fc024600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc01c06f440 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.138+0000 7fc024600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01c146f80 con 0x7fc01c15e0b0 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.138+0000 7fc024600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc01c0b3520 con 0x7fc01c146ba0 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc022600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc01c15e0b0 0x7fc01c0b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc021200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc01c0b2ba0 0x7fc01c15d9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:47476/0 (socket says 172.21.15.2:47476) 2024-01-15T18:17:41.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc021200640 1 -- 172.21.15.2:0/3560842785 learned_addr learned my addr 172.21.15.2:0/3560842785 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:17:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc021c00640 1 --2- 172.21.15.2:0/3560842785 >> v2:172.21.15.137:3300/0 conn(0x7fc01c146ba0 0x7fc01c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc022600640 1 -- 172.21.15.2:0/3560842785 >> v2:172.21.15.137:3300/0 conn(0x7fc01c146ba0 msgr2=0x7fc01c0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc022600640 1 --2- 172.21.15.2:0/3560842785 >> v2:172.21.15.137:3300/0 conn(0x7fc01c146ba0 0x7fc01c0d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc022600640 1 -- 172.21.15.2:0/3560842785 >> v1:172.21.15.2:6789/0 conn(0x7fc01c0b2ba0 legacy=0x7fc01c15d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc022600640 1 -- 172.21.15.2:0/3560842785 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc01c15e5f0 con 0x7fc01c15e0b0 2024-01-15T18:17:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.139+0000 7fc021c00640 1 --2- 172.21.15.2:0/3560842785 >> v2:172.21.15.137:3300/0 conn(0x7fc01c146ba0 0x7fc01c0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:41.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.141+0000 7fc022600640 1 --2- 172.21.15.2:0/3560842785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c15e0b0 0x7fc01c0b2fe0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fc01c06e430 tx=0x7fc01402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e90ddcb9760f8fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:41.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.141+0000 7fc01be00640 1 -- 172.21.15.2:0/3560842785 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01402cbf0 con 0x7fc01c15e0b0 2024-01-15T18:17:41.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.141+0000 7fc01be00640 1 -- 172.21.15.2:0/3560842785 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01402cd50 con 0x7fc01c15e0b0 2024-01-15T18:17:41.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.141+0000 7fc01be00640 1 -- 172.21.15.2:0/3560842785 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014034660 con 0x7fc01c15e0b0 2024-01-15T18:17:41.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 -- 172.21.15.2:0/3560842785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c15e0b0 msgr2=0x7fc01c0b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:41.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 --2- 172.21.15.2:0/3560842785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c15e0b0 0x7fc01c0b2fe0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fc01c06e430 tx=0x7fc01402be40 comp rx=0 tx=0).stop 2024-01-15T18:17:41.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 -- 172.21.15.2:0/3560842785 shutdown_connections 2024-01-15T18:17:41.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 --2- 172.21.15.2:0/3560842785 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c15e0b0 0x7fc01c0b2fe0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:41.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 --2- 172.21.15.2:0/3560842785 >> v2:172.21.15.137:3300/0 conn(0x7fc01c146ba0 0x7fc01c0d1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:41.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 -- 172.21.15.2:0/3560842785 >> 172.21.15.2:0/3560842785 conn(0x7fc01c0b6430 msgr2=0x7fc01c0b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:41.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 -- 172.21.15.2:0/3560842785 shutdown_connections 2024-01-15T18:17:41.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.142+0000 7fc024600640 1 -- 172.21.15.2:0/3560842785 wait complete. 2024-01-15T18:17:41.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.144+0000 7fc024600640 1 Processor -- start 2024-01-15T18:17:41.147 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.144+0000 7fc024600640 1 -- start start 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc024600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c0b2ba0 0x7fc01c136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc024600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc01c146ba0 0x7fc01c0afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc024600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc01c15e0b0 0x7fc01c134ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc024600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c134710 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc024600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c136540 con 0x7fc01c15e0b0 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc024600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01c133b00 con 0x7fc01c146ba0 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc021c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c0b2ba0 0x7fc01c136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc021c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c0b2ba0 0x7fc01c136000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51128/0 (socket says 172.21.15.2:51128) 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.145+0000 7fc021c00640 1 -- 172.21.15.2:0/3771194479 learned_addr learned my addr 172.21.15.2:0/3771194479 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:41.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc022600640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc01c15e0b0 0x7fc01c134ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:41.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021200640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc01c146ba0 0x7fc01c0afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:41.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021c00640 1 -- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc01c146ba0 msgr2=0x7fc01c0afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021c00640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc01c146ba0 0x7fc01c0afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021c00640 1 -- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc01c15e0b0 msgr2=0x7fc01c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021c00640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc01c15e0b0 0x7fc01c134ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021c00640 1 -- 172.21.15.2:0/3771194479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014009a70 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc022600640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc01c15e0b0 0x7fc01c134ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.146+0000 7fc021c00640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc01c0b2ba0 0x7fc01c136000 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00c820 tx=0x7fc00c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.147+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c007a30 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.147+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c007b90 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.147+0000 7fc024600640 1 -- 172.21.15.2:0/3771194479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01c133de0 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.147+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c00ce70 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.151 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.147+0000 7fc024600640 1 -- 172.21.15.2:0/3771194479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc01c1343a0 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.148+0000 7fc024600640 1 -- 172.21.15.2:0/3771194479 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc01c0748b0 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.152 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.149+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fc00c0113f0 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.150+0000 7fc019600640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff406c6a0 0x7fbff406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:41.153 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.150+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fc00c088dc0 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.154 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.151+0000 7fc021200640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff406c6a0 0x7fbff406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:41.155 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.152+0000 7fc021200640 1 --2- 172.21.15.2:0/3771194479 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fbff406c6a0 0x7fbff406eb60 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fc01c06c6d0 tx=0x7fc004009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:41.159 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.154+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc00c0552d0 con 0x7fc01c0b2ba0 2024-01-15T18:17:41.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.388+0000 7fc024600640 1 -- 172.21.15.2:0/3771194479 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc01c0b2fe0 con 0x7fbff406c6a0 2024-01-15T18:17:41.405 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:41.403+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:41.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:41.403+0000 7fc019600640 1 -- 172.21.15.2:0/3771194479 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc01c0b2fe0 con 0x7fbff406c6a0 2024-01-15T18:17:41.406 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:41.406 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:43.422 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:43.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.702+0000 7f4e1a000640 1 Processor -- start 2024-01-15T18:17:43.705 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.702+0000 7f4e1a000640 1 -- start start 2024-01-15T18:17:43.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.703+0000 7f4e1a000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4e140702c0 0x7f4e140b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e1a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4e1406c8f0 0x7f4e1406ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e1a000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e1415dd70 con 0x7f4e141439f0 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e1a000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e1406d230 con 0x7f4e1406c8f0 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e1a000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e140711b0 con 0x7f4e140702c0 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e13e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4e1406c8f0 0x7f4e1406ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e13e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4e1406c8f0 0x7f4e1406ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57182/0 (socket says 172.21.15.2:57182) 2024-01-15T18:17:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e13e00640 1 -- 172.21.15.2:0/2407647686 learned_addr learned my addr 172.21.15.2:0/2407647686 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e13e00640 1 -- 172.21.15.2:0/2407647686 >> v2:172.21.15.137:3300/0 conn(0x7f4e140702c0 msgr2=0x7f4e140b3420 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-15T18:17:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e13e00640 1 --2- 172.21.15.2:0/2407647686 >> v2:172.21.15.137:3300/0 conn(0x7f4e140702c0 0x7f4e140b3420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.704+0000 7f4e13e00640 1 -- 172.21.15.2:0/2407647686 >> v1:172.21.15.137:6789/0 conn(0x7f4e141439f0 legacy=0x7f4e140cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.705+0000 7f4e13e00640 1 -- 172.21.15.2:0/2407647686 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e14143dd0 con 0x7f4e1406c8f0 2024-01-15T18:17:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.706+0000 7f4e13e00640 1 --2- 172.21.15.2:0/2407647686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e1406c8f0 0x7f4e1406ccf0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4e14072670 tx=0x7f4e0c02e110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f6401a1bfe9d5a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.706+0000 7f4e12000640 1 -- 172.21.15.2:0/2407647686 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c02eab0 con 0x7f4e1406c8f0 2024-01-15T18:17:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.707+0000 7f4e12000640 1 -- 172.21.15.2:0/2407647686 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0c02ec10 con 0x7f4e1406c8f0 2024-01-15T18:17:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.707+0000 7f4e12000640 1 -- 172.21.15.2:0/2407647686 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c009600 con 0x7f4e1406c8f0 2024-01-15T18:17:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.707+0000 7f4e1a000640 1 -- 172.21.15.2:0/2407647686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e1406c8f0 msgr2=0x7f4e1406ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.707+0000 7f4e1a000640 1 --2- 172.21.15.2:0/2407647686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e1406c8f0 0x7f4e1406ccf0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f4e14072670 tx=0x7f4e0c02e110 comp rx=0 tx=0).stop 2024-01-15T18:17:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.708+0000 7f4e1a000640 1 -- 172.21.15.2:0/2407647686 shutdown_connections 2024-01-15T18:17:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.708+0000 7f4e1a000640 1 --2- 172.21.15.2:0/2407647686 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e1406c8f0 0x7f4e1406ccf0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.708+0000 7f4e1a000640 1 --2- 172.21.15.2:0/2407647686 >> v2:172.21.15.137:3300/0 conn(0x7f4e140702c0 0x7f4e140b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.708+0000 7f4e1a000640 1 -- 172.21.15.2:0/2407647686 >> 172.21.15.2:0/2407647686 conn(0x7f4e140b45f0 msgr2=0x7f4e140b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.708+0000 7f4e1a000640 1 -- 172.21.15.2:0/2407647686 shutdown_connections 2024-01-15T18:17:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.708+0000 7f4e1a000640 1 -- 172.21.15.2:0/2407647686 wait complete. 2024-01-15T18:17:43.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.711+0000 7f4e1a000640 1 Processor -- start 2024-01-15T18:17:43.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.711+0000 7f4e1a000640 1 -- start start 2024-01-15T18:17:43.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.714+0000 7f4e1a000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e1406c8f0 0x7f4e14159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:43.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.714+0000 7f4e1a000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e140702c0 0x7f4e1413a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:43.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.714+0000 7f4e1a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e141439f0 0x7f4e1415cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.714+0000 7f4e1a000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1415b440 con 0x7f4e141439f0 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.714+0000 7f4e1a000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1415b720 con 0x7f4e1406c8f0 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.714+0000 7f4e1a000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e1415ba00 con 0x7f4e140702c0 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.715+0000 7f4e13e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e141439f0 0x7f4e1415cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.715+0000 7f4e12a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e140702c0 0x7f4e1413a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.715+0000 7f4e12a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e140702c0 0x7f4e1413a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:35110/0 (socket says 172.21.15.2:35110) 2024-01-15T18:17:43.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.715+0000 7f4e12a00640 1 -- 172.21.15.2:0/2101364851 learned_addr learned my addr 172.21.15.2:0/2101364851 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.715+0000 7f4e13400640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e1406c8f0 0x7f4e14159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.716+0000 7f4e12a00640 1 -- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e1406c8f0 msgr2=0x7f4e14159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.716+0000 7f4e12a00640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e1406c8f0 0x7f4e14159a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.716+0000 7f4e12a00640 1 -- 172.21.15.2:0/2101364851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e141439f0 msgr2=0x7f4e1415cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.716+0000 7f4e12a00640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e141439f0 0x7f4e1415cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:43.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.716+0000 7f4e12a00640 1 -- 172.21.15.2:0/2101364851 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c00bd40 con 0x7f4e140702c0 2024-01-15T18:17:43.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.716+0000 7f4e12a00640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e140702c0 0x7f4e1413a3c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400b450 tx=0x7f4e0400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:43.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.717+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04011890 con 0x7f4e140702c0 2024-01-15T18:17:43.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.717+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04011a10 con 0x7f4e140702c0 2024-01-15T18:17:43.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.717+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04011dc0 con 0x7f4e140702c0 2024-01-15T18:17:43.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.717+0000 7f4e1a000640 1 -- 172.21.15.2:0/2101364851 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e1415bce0 con 0x7f4e140702c0 2024-01-15T18:17:43.721 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.717+0000 7f4e1a000640 1 -- 172.21.15.2:0/2101364851 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e14159f70 con 0x7f4e140702c0 2024-01-15T18:17:43.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.719+0000 7f4e1a000640 1 -- 172.21.15.2:0/2101364851 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e1415c200 con 0x7f4e140702c0 2024-01-15T18:17:43.723 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.720+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mon.2 v2:172.21.15.137:3301/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f4e0401a400 con 0x7f4e140702c0 2024-01-15T18:17:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.721+0000 7f4e0b400640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dec06c6c0 0x7f4dec06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:43.724 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.721+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mon.2 v2:172.21.15.137:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f4e04088ee0 con 0x7f4e140702c0 2024-01-15T18:17:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.721+0000 7f4e13400640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dec06c6c0 0x7f4dec06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:43.725 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.722+0000 7f4e13400640 1 --2- 172.21.15.2:0/2101364851 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4dec06c6c0 0x7f4dec06eb80 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0017f0 tx=0x7f4dfc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:43.733 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.729+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mon.2 v2:172.21.15.137:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e1415c200 con 0x7f4e140702c0 2024-01-15T18:17:43.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:43.988+0000 7f4e1a000640 1 -- 172.21.15.2:0/2101364851 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e1413c0c0 con 0x7f4dec06c6c0 2024-01-15T18:17:44.007 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:44.005+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:44.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:44.004+0000 7f4e0b400640 1 -- 172.21.15.2:0/2101364851 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4e1413c0c0 con 0x7f4dec06c6c0 2024-01-15T18:17:44.008 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:44.008 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:46.023 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:46.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.259+0000 7f24b8400640 1 Processor -- start 2024-01-15T18:17:46.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.259+0000 7f24b8400640 1 -- start start 2024-01-15T18:17:46.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.261+0000 7f24b8400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24b00cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.261+0000 7f24b8400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b00ce230 con 0x7f24b00cdd70 2024-01-15T18:17:46.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.261+0000 7f24b8400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b00ce510 con 0x7f24b006d000 2024-01-15T18:17:46.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.261+0000 7f24b8400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00ce7f0 con 0x7f24b0144c00 2024-01-15T18:17:46.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.261+0000 7f24b5000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f24b00cdd70 0x7f24b006c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:51994/0 (socket says 172.21.15.2:51994) 2024-01-15T18:17:46.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.261+0000 7f24b5000640 1 -- 172.21.15.2:0/3839101597 learned_addr learned my addr 172.21.15.2:0/3839101597 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:17:46.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.262+0000 7f24b5a00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24b00cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.262+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114333297 0 0) 0x7f24b00ce230 con 0x7f24b00cdd70 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.262+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2490003520 con 0x7f24b00cdd70 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.262+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1253839773 0 0) 0x7f24b00ce510 con 0x7f24b006d000 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.262+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24b00ce230 con 0x7f24b006d000 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2930559142 0 0) 0x7f2490003520 con 0x7f24b00cdd70 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 msgr2=0x7f24b00cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24b00cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 >> v1:172.21.15.137:6789/0 conn(0x7f24b006d000 legacy=0x7f24b0070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b00705f0 con 0x7f24b00cdd70 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2498001670 con 0x7f24b00cdd70 2024-01-15T18:17:46.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 >> v1:172.21.15.2:6789/0 conn(0x7f24b00cdd70 legacy=0x7f24b006c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2490003b30 0x7f2490003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490004d00 0x7f2490005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00ce7f0 con 0x7f2490003b30 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249000ba30 con 0x7f2490004d00 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249000bba0 con 0x7f24b0144c00 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24b5a00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2490003b30 0x7f2490003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.263+0000 7f24b6400640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490004d00 0x7f2490005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5a00640 1 -- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 msgr2=0x7f24900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5000640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5a00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5a00640 1 -- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490004d00 msgr2=0x7f2490005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5a00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490004d00 0x7f2490005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5a00640 1 -- 172.21.15.2:0/3839101597 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2490003520 con 0x7f2490003b30 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.264+0000 7f24b5000640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b0144c00 0x7f24900047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:46.270 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.265+0000 7f24b5a00640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2490003b30 0x7f2490003f30 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ac0 tx=0x7f24900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8bc1bebc8acf2f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.265+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a002c6b0 con 0x7f2490003b30 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.265+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a002c810 con 0x7f2490003b30 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.265+0000 7f24afe00640 1 -- 172.21.15.2:0/3839101597 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a002cba0 con 0x7f2490003b30 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b8400640 1 -- 172.21.15.2:0/3839101597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2490003b30 msgr2=0x7f2490003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b8400640 1 --2- 172.21.15.2:0/3839101597 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2490003b30 0x7f2490003f30 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009ac0 tx=0x7f24900057f0 comp rx=0 tx=0).stop 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b6400640 1 -- 172.21.15.2:0/3839101597 reap_dead start 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b8400640 1 -- 172.21.15.2:0/3839101597 shutdown_connections 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b8400640 1 -- 172.21.15.2:0/3839101597 >> 172.21.15.2:0/3839101597 conn(0x7f24b00b45f0 msgr2=0x7f24b00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b8400640 1 -- 172.21.15.2:0/3839101597 shutdown_connections 2024-01-15T18:17:46.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.266+0000 7f24b8400640 1 -- 172.21.15.2:0/3839101597 wait complete. 2024-01-15T18:17:46.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.268+0000 7f24b8400640 1 Processor -- start 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.268+0000 7f24b8400640 1 -- start start 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b8400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490003b30 0x7f24b0140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b8400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24b0141440 0x7f24b00ac870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b8400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b00acdb0 0x7f24b00ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b8400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00b2550 con 0x7f24b0141440 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b8400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01418c0 con 0x7f2490003b30 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b8400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01f3a20 con 0x7f24b00acdb0 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24b0141440 0x7f24b00ac870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b5000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24b0141440 0x7f24b00ac870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57208/0 (socket says 172.21.15.2:57208) 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b5000640 1 -- 172.21.15.2:0/1255415476 learned_addr learned my addr 172.21.15.2:0/1255415476 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b5a00640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490003b30 0x7f24b0140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.269+0000 7f24b6400640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b00acdb0 0x7f24b00ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b5000640 1 -- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b00acdb0 msgr2=0x7f24b00ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b5000640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24b00acdb0 0x7f24b00ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b5000640 1 -- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490003b30 msgr2=0x7f24b0140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b5000640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2490003b30 0x7f24b0140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b5000640 1 -- 172.21.15.2:0/1255415476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b01f3d00 con 0x7f24b0141440 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b5000640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24b0141440 0x7f24b00ac870 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f249800d810 tx=0x7f249800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24980074e0 con 0x7f24b0141440 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2498007640 con 0x7f24b0141440 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498004a20 con 0x7f24b0141440 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b8400640 1 -- 172.21.15.2:0/1255415476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b00b2760 con 0x7f24b0141440 2024-01-15T18:17:46.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.270+0000 7f24b8400640 1 -- 172.21.15.2:0/1255415476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b01f40b0 con 0x7f24b0141440 2024-01-15T18:17:46.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.272+0000 7f24b8400640 1 -- 172.21.15.2:0/1255415476 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b01f3e90 con 0x7f24b0141440 2024-01-15T18:17:46.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.273+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f2498004b80 con 0x7f24b0141440 2024-01-15T18:17:46.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.274+0000 7f24ad600640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f248806c6a0 0x7f248806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:46.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.274+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f2498088570 con 0x7f24b0141440 2024-01-15T18:17:46.278 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.275+0000 7f24b5a00640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f248806c6a0 0x7f248806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:46.279 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.276+0000 7f24b5a00640 1 --2- 172.21.15.2:0/1255415476 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f248806c6a0 0x7f248806eb60 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f24a0036000 tx=0x7f24a0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:46.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.279+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24b01f3e90 con 0x7f24b0141440 2024-01-15T18:17:46.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.508+0000 7f24b8400640 1 -- 172.21.15.2:0/1255415476 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24b00ae990 con 0x7f248806c6a0 2024-01-15T18:17:46.525 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:46.523+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:46.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:46.522+0000 7f24ad600640 1 -- 172.21.15.2:0/1255415476 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f24b00ae990 con 0x7f248806c6a0 2024-01-15T18:17:46.526 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:46.526 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:48.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:48.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.780+0000 7ff484c00640 1 Processor -- start 2024-01-15T18:17:48.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.780+0000 7ff484c00640 1 -- start start 2024-01-15T18:17:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.782+0000 7ff484c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff4800ae260 0x7ff480071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.782+0000 7ff484c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff480070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.782+0000 7ff484c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48006c500 con 0x7ff48006da80 2024-01-15T18:17:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.782+0000 7ff484c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48015c4b0 con 0x7ff4800ae260 2024-01-15T18:17:48.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.782+0000 7ff484c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48015b8a0 con 0x7ff4800704c0 2024-01-15T18:17:48.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.783+0000 7ff47fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff480070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:48.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.783+0000 7ff47f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff4800ae260 0x7ff480071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:48.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.783+0000 7ff47fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff480070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:35148/0 (socket says 172.21.15.2:35148) 2024-01-15T18:17:48.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.783+0000 7ff47fe00640 1 -- 172.21.15.2:0/2797378785 learned_addr learned my addr 172.21.15.2:0/2797378785 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:48.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.784+0000 7ff47fe00640 1 -- 172.21.15.2:0/2797378785 >> v2:172.21.15.137:3300/0 conn(0x7ff4800ae260 msgr2=0x7ff480071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:48.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.784+0000 7ff47fe00640 1 --2- 172.21.15.2:0/2797378785 >> v2:172.21.15.137:3300/0 conn(0x7ff4800ae260 0x7ff480071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:48.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.784+0000 7ff47fe00640 1 -- 172.21.15.2:0/2797378785 >> v1:172.21.15.137:6789/0 conn(0x7ff48006da80 legacy=0x7ff48006df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:48.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.784+0000 7ff47fe00640 1 -- 172.21.15.2:0/2797378785 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48015bb80 con 0x7ff4800704c0 2024-01-15T18:17:48.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.784+0000 7ff47f400640 1 --2- 172.21.15.2:0/2797378785 >> v2:172.21.15.137:3300/0 conn(0x7ff4800ae260 0x7ff480071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:17:48.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff47fe00640 1 --2- 172.21.15.2:0/2797378785 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff480070950 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff478009e50 tx=0x7ff47802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b70cba586d2e8ea server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff47e000640 1 -- 172.21.15.2:0/2797378785 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47802ca70 con 0x7ff4800704c0 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff47e000640 1 -- 172.21.15.2:0/2797378785 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47802cbd0 con 0x7ff4800704c0 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff47e000640 1 -- 172.21.15.2:0/2797378785 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4780343f0 con 0x7ff4800704c0 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 -- 172.21.15.2:0/2797378785 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 msgr2=0x7ff480070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 --2- 172.21.15.2:0/2797378785 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff480070950 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff478009e50 tx=0x7ff47802bed0 comp rx=0 tx=0).stop 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 -- 172.21.15.2:0/2797378785 shutdown_connections 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 --2- 172.21.15.2:0/2797378785 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff480070950 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 --2- 172.21.15.2:0/2797378785 >> v2:172.21.15.137:3300/0 conn(0x7ff4800ae260 0x7ff480071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 -- 172.21.15.2:0/2797378785 >> 172.21.15.2:0/2797378785 conn(0x7ff4800cebc0 msgr2=0x7ff4800cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 -- 172.21.15.2:0/2797378785 shutdown_connections 2024-01-15T18:17:48.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.787+0000 7ff484c00640 1 -- 172.21.15.2:0/2797378785 wait complete. 2024-01-15T18:17:48.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.789+0000 7ff484c00640 1 Processor -- start 2024-01-15T18:17:48.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.789+0000 7ff484c00640 1 -- start start 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.790+0000 7ff484c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff48006da80 0x7ff48015d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.790+0000 7ff484c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff48014f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.790+0000 7ff484c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4800ae260 0x7ff48015ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.790+0000 7ff484c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4800b4700 con 0x7ff48006da80 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.790+0000 7ff484c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48015ddc0 con 0x7ff4800ae260 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.790+0000 7ff484c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48015e0a0 con 0x7ff4800704c0 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff48006da80 0x7ff48015d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff48006da80 0x7ff48015d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57224/0 (socket says 172.21.15.2:57224) 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 -- 172.21.15.2:0/1182167889 learned_addr learned my addr 172.21.15.2:0/1182167889 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47ea00640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff48014f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:48.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47fe00640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4800ae260 0x7ff48015ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:48.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 -- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 msgr2=0x7ff48014f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:48.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4800704c0 0x7ff48014f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:48.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 -- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4800ae260 msgr2=0x7ff48015ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:48.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4800ae260 0x7ff48015ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:48.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.791+0000 7ff47f400640 1 -- 172.21.15.2:0/1182167889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff470009400 con 0x7ff48006da80 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.792+0000 7ff47f400640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff48006da80 0x7ff48015d730 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7ff47000ebe0 tx=0x7ff47000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.792+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff470009760 con 0x7ff48006da80 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.792+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47000cc30 con 0x7ff48006da80 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.792+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47000eee0 con 0x7ff48006da80 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.792+0000 7ff484c00640 1 -- 172.21.15.2:0/1182167889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff478009b30 con 0x7ff48006da80 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.792+0000 7ff484c00640 1 -- 172.21.15.2:0/1182167889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48015e660 con 0x7ff48006da80 2024-01-15T18:17:48.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.793+0000 7ff484c00640 1 -- 172.21.15.2:0/1182167889 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff480074770 con 0x7ff48006da80 2024-01-15T18:17:48.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.794+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7ff470013470 con 0x7ff48006da80 2024-01-15T18:17:48.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.795+0000 7ff477400640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff45806c670 0x7ff45806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:48.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.795+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7ff470088e60 con 0x7ff48006da80 2024-01-15T18:17:48.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.796+0000 7ff47ea00640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff45806c670 0x7ff45806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:48.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.797+0000 7ff47ea00640 1 --2- 172.21.15.2:0/1182167889 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff45806c670 0x7ff45806eb30 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7ff4800b1b60 tx=0x7ff468009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:48.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:48.800+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff470055420 con 0x7ff48006da80 2024-01-15T18:17:49.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:49.033+0000 7ff484c00640 1 -- 172.21.15.2:0/1182167889 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4800abd30 con 0x7ff45806c670 2024-01-15T18:17:49.050 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:49.047+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:49.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:49.048+0000 7ff477400640 1 -- 172.21.15.2:0/1182167889 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff468001380 con 0x7ff45806c670 2024-01-15T18:17:49.051 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:49.051 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:51.065 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:51.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.304+0000 7f83e7200640 1 Processor -- start 2024-01-15T18:17:51.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.304+0000 7f83e7200640 1 -- start start 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e7200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f83e80ae260 0x7f83e8071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e7200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f83e806da80 0x7f83e806df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e7200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e806c500 con 0x7f83e815eab0 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e7200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e815c4b0 con 0x7f83e806da80 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e7200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e80714d0 con 0x7f83e80ae260 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f83e815eab0 0x7f83e815ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52002/0 (socket says 172.21.15.2:52002) 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e6800640 1 -- 172.21.15.2:0/1730582825 learned_addr learned my addr 172.21.15.2:0/1730582825 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e5400640 1 --2- 172.21.15.2:0/1730582825 >> v2:172.21.15.2:3300/0 conn(0x7f83e806da80 0x7f83e806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:51.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.306+0000 7f83e5e00640 1 --2- 172.21.15.2:0/1730582825 >> v2:172.21.15.137:3300/0 conn(0x7f83e80ae260 0x7f83e8071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.307+0000 7f83e5400640 1 -- 172.21.15.2:0/1730582825 >> v2:172.21.15.137:3300/0 conn(0x7f83e80ae260 msgr2=0x7f83e8071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:51.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.307+0000 7f83e5400640 1 --2- 172.21.15.2:0/1730582825 >> v2:172.21.15.137:3300/0 conn(0x7f83e80ae260 0x7f83e8071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.307+0000 7f83e5400640 1 -- 172.21.15.2:0/1730582825 >> v1:172.21.15.2:6789/0 conn(0x7f83e815eab0 legacy=0x7f83e815ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.307+0000 7f83e5400640 1 -- 172.21.15.2:0/1730582825 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e815b8a0 con 0x7f83e806da80 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.307+0000 7f83e4a00640 1 -- 172.21.15.2:0/1730582825 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454375057 0 0) 0x7f83e0001110 con 0x7f83e815eab0 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.308+0000 7f83e5400640 1 --2- 172.21.15.2:0/1730582825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e806df00 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f83e804f900 tx=0x7f83d002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57bc5e2240a03e11 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.308+0000 7f83e4a00640 1 -- 172.21.15.2:0/1730582825 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d002cbf0 con 0x7f83e806da80 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.308+0000 7f83e4a00640 1 -- 172.21.15.2:0/1730582825 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d002cd50 con 0x7f83e806da80 2024-01-15T18:17:51.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.308+0000 7f83e4a00640 1 -- 172.21.15.2:0/1730582825 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d0034660 con 0x7f83e806da80 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 -- 172.21.15.2:0/1730582825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 msgr2=0x7f83e806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 --2- 172.21.15.2:0/1730582825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e806df00 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f83e804f900 tx=0x7f83d002be40 comp rx=0 tx=0).stop 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 -- 172.21.15.2:0/1730582825 shutdown_connections 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 --2- 172.21.15.2:0/1730582825 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e806df00 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 --2- 172.21.15.2:0/1730582825 >> v2:172.21.15.137:3300/0 conn(0x7f83e80ae260 0x7f83e8071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 -- 172.21.15.2:0/1730582825 >> 172.21.15.2:0/1730582825 conn(0x7f83e80cebc0 msgr2=0x7f83e80cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 -- 172.21.15.2:0/1730582825 shutdown_connections 2024-01-15T18:17:51.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.309+0000 7f83e7200640 1 -- 172.21.15.2:0/1730582825 wait complete. 2024-01-15T18:17:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.311+0000 7f83e7200640 1 Processor -- start 2024-01-15T18:17:51.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.311+0000 7f83e7200640 1 -- start start 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e80b1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e80ae260 0x7f83e8156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e815eab0 0x7f83e80b35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8156d50 con 0x7f83e806da80 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81f4140 con 0x7f83e80ae260 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81f4420 con 0x7f83e815eab0 2024-01-15T18:17:51.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e80b1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e80b1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57232/0 (socket says 172.21.15.2:57232) 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.312+0000 7f83e5e00640 1 -- 172.21.15.2:0/3249706658 learned_addr learned my addr 172.21.15.2:0/3249706658 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5e00640 1 -- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e815eab0 msgr2=0x7f83e80b35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5e00640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f83e815eab0 0x7f83e80b35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5e00640 1 -- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e80ae260 msgr2=0x7f83e8156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5400640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e80ae260 0x7f83e8156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5e00640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f83e80ae260 0x7f83e8156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5e00640 1 -- 172.21.15.2:0/3249706658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0009a70 con 0x7f83e806da80 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e5e00640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f83e806da80 0x7f83e80b1960 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f83d800d800 tx=0x7f83d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d800b6c0 con 0x7f83e806da80 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d800b820 con 0x7f83e806da80 2024-01-15T18:17:51.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d800bbb0 con 0x7f83e806da80 2024-01-15T18:17:51.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.313+0000 7f83e7200640 1 -- 172.21.15.2:0/3249706658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e81f4700 con 0x7f83e806da80 2024-01-15T18:17:51.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.314+0000 7f83e7200640 1 -- 172.21.15.2:0/3249706658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e81f4cc0 con 0x7f83e806da80 2024-01-15T18:17:51.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.315+0000 7f83e7200640 1 -- 172.21.15.2:0/3249706658 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8074770 con 0x7f83e806da80 2024-01-15T18:17:51.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.315+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f83d800bd10 con 0x7f83e806da80 2024-01-15T18:17:51.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.316+0000 7f83de000640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83c006c6a0 0x7f83c006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:51.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.316+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f83d8089150 con 0x7f83e806da80 2024-01-15T18:17:51.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.317+0000 7f83e5400640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83c006c6a0 0x7f83c006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:51.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.318+0000 7f83e5400640 1 --2- 172.21.15.2:0/3249706658 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f83c006c6a0 0x7f83c006eb60 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f83d0036000 tx=0x7f83d0030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:51.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.321+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83d8055660 con 0x7f83e806da80 2024-01-15T18:17:51.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.560+0000 7f83e7200640 1 -- 172.21.15.2:0/3249706658 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83e815c140 con 0x7f83c006c6a0 2024-01-15T18:17:51.576 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:51.573+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:51.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:51.574+0000 7f83de000640 1 -- 172.21.15.2:0/3249706658 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83d0009450 con 0x7f83c006c6a0 2024-01-15T18:17:51.577 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:51.577 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:53.592 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:53.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.828+0000 7f856f000640 1 Processor -- start 2024-01-15T18:17:53.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.828+0000 7f856f000640 1 -- start start 2024-01-15T18:17:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856f000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f856813cf10 0x7f856806da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856f000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f85680719a0 0x7f856806dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856f000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856813d2f0 con 0x7f856813d610 2024-01-15T18:17:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856f000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8568071df0 con 0x7f856813cf10 2024-01-15T18:17:53.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856f000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856815a510 con 0x7f85680719a0 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856dc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f856813cf10 0x7f856806da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856dc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f856813cf10 0x7f856806da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36944/0 (socket says 172.21.15.2:36944) 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.830+0000 7f856dc00640 1 -- 172.21.15.2:0/989169015 learned_addr learned my addr 172.21.15.2:0/989169015 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.831+0000 7f856e600640 1 --2- 172.21.15.2:0/989169015 >> v2:172.21.15.137:3300/0 conn(0x7f85680719a0 0x7f856806dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.831+0000 7f856dc00640 1 -- 172.21.15.2:0/989169015 >> v2:172.21.15.137:3300/0 conn(0x7f85680719a0 msgr2=0x7f856806dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.831+0000 7f856dc00640 1 --2- 172.21.15.2:0/989169015 >> v2:172.21.15.137:3300/0 conn(0x7f85680719a0 0x7f856806dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:53.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.831+0000 7f856dc00640 1 -- 172.21.15.2:0/989169015 >> v1:172.21.15.137:6789/0 conn(0x7f856813d610 legacy=0x7f8568071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.831+0000 7f856dc00640 1 -- 172.21.15.2:0/989169015 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856815a7f0 con 0x7f856813cf10 2024-01-15T18:17:53.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.832+0000 7f856dc00640 1 --2- 172.21.15.2:0/989169015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856806da90 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f8568070020 tx=0x7f855802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95b8dd3ce0a88fc1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:53.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.832+0000 7f8567e00640 1 -- 172.21.15.2:0/989169015 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558031870 con 0x7f856813cf10 2024-01-15T18:17:53.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.832+0000 7f8567e00640 1 -- 172.21.15.2:0/989169015 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85580319d0 con 0x7f856813cf10 2024-01-15T18:17:53.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.832+0000 7f8567e00640 1 -- 172.21.15.2:0/989169015 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558031d60 con 0x7f856813cf10 2024-01-15T18:17:53.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 -- 172.21.15.2:0/989169015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 msgr2=0x7f856806da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:53.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 --2- 172.21.15.2:0/989169015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856806da90 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f8568070020 tx=0x7f855802b910 comp rx=0 tx=0).stop 2024-01-15T18:17:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 -- 172.21.15.2:0/989169015 shutdown_connections 2024-01-15T18:17:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 --2- 172.21.15.2:0/989169015 >> v2:172.21.15.137:3300/0 conn(0x7f85680719a0 0x7f856806dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 --2- 172.21.15.2:0/989169015 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856806da90 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 -- 172.21.15.2:0/989169015 >> 172.21.15.2:0/989169015 conn(0x7f85680b07d0 msgr2=0x7f85680b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 -- 172.21.15.2:0/989169015 shutdown_connections 2024-01-15T18:17:53.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.833+0000 7f856f000640 1 -- 172.21.15.2:0/989169015 wait complete. 2024-01-15T18:17:53.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.835+0000 7f856f000640 1 Processor -- start 2024-01-15T18:17:53.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.835+0000 7f856f000640 1 -- start start 2024-01-15T18:17:53.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.836+0000 7f856f000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85680719a0 0x7f85680b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:53.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.836+0000 7f856f000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856813b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.836+0000 7f856f000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856813d610 0x7f856813bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.836+0000 7f856f000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85680b2580 con 0x7f856813cf10 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.836+0000 7f856f000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85680b2860 con 0x7f85680719a0 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.836+0000 7f856f000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85680cc3c0 con 0x7f856813d610 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856813b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856813b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36954/0 (socket says 172.21.15.2:36954) 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 -- 172.21.15.2:0/3663487169 learned_addr learned my addr 172.21.15.2:0/3663487169 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856e600640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856813d610 0x7f856813bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856dc00640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85680719a0 0x7f85680b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 -- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856813d610 msgr2=0x7f856813bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f856813d610 0x7f856813bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 -- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85680719a0 msgr2=0x7f85680b2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85680719a0 0x7f85680b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 -- 172.21.15.2:0/3663487169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580094d0 con 0x7f856813cf10 2024-01-15T18:17:53.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.837+0000 7f856d200640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f856813cf10 0x7f856813b5f0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f855000c430 tx=0x7f855000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:53.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.838+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8550011790 con 0x7f856813cf10 2024-01-15T18:17:53.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.838+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85500118f0 con 0x7f856813cf10 2024-01-15T18:17:53.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.838+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8550011ce0 con 0x7f856813cf10 2024-01-15T18:17:53.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.838+0000 7f856f000640 1 -- 172.21.15.2:0/3663487169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85680cc6a0 con 0x7f856813cf10 2024-01-15T18:17:53.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.838+0000 7f856f000640 1 -- 172.21.15.2:0/3663487169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85680b38b0 con 0x7f856813cf10 2024-01-15T18:17:53.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.839+0000 7f856f000640 1 -- 172.21.15.2:0/3663487169 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8568067970 con 0x7f856813cf10 2024-01-15T18:17:53.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.840+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f855001c460 con 0x7f856813cf10 2024-01-15T18:17:53.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.841+0000 7f8565600640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f854006c670 0x7f854006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:53.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.841+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8550088870 con 0x7f856813cf10 2024-01-15T18:17:53.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.842+0000 7f856dc00640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f854006c670 0x7f854006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:53.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.843+0000 7f856dc00640 1 --2- 172.21.15.2:0/3663487169 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f854006c670 0x7f854006eb30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f8558036000 tx=0x7f855803c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:53.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:53.845+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85500550a0 con 0x7f856813cf10 2024-01-15T18:17:54.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:54.076+0000 7f856f000640 1 -- 172.21.15.2:0/3663487169 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85680b3a70 con 0x7f854006c670 2024-01-15T18:17:54.094 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:54.091+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:54.094 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:54.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:54.092+0000 7f8565600640 1 -- 172.21.15.2:0/3663487169 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8558031d50 con 0x7f854006c670 2024-01-15T18:17:54.094 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:54.112 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:56.113 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.347+0000 7f0cf4600640 1 Processor -- start 2024-01-15T18:17:56.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.348+0000 7f0cf4600640 1 -- start start 2024-01-15T18:17:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.349+0000 7f0cf4600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 0x7f0cec0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.349+0000 7f0cf4600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0cec06d000 0x7f0cec070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.349+0000 7f0cf4600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cec0ce230 con 0x7f0cec0cdd70 2024-01-15T18:17:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.349+0000 7f0cf4600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cec0ce510 con 0x7f0cec06d000 2024-01-15T18:17:56.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.349+0000 7f0cf4600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0ce7f0 con 0x7f0cec144c00 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.350+0000 7f0cf1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 0x7f0cec0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.350+0000 7f0cf2600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0cec06d000 0x7f0cec070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.350+0000 7f0cf1c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 0x7f0cec0cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:53372/0 (socket says 172.21.15.2:53372) 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.350+0000 7f0cf1c00640 1 -- 172.21.15.2:0/4256099377 learned_addr learned my addr 172.21.15.2:0/4256099377 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.351+0000 7f0cf1c00640 1 -- 172.21.15.2:0/4256099377 >> v2:172.21.15.137:3300/0 conn(0x7f0cec06d000 msgr2=0x7f0cec070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.351+0000 7f0cf1c00640 1 --2- 172.21.15.2:0/4256099377 >> v2:172.21.15.137:3300/0 conn(0x7f0cec06d000 0x7f0cec070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.351+0000 7f0cf1c00640 1 -- 172.21.15.2:0/4256099377 >> v1:172.21.15.137:6789/0 conn(0x7f0cec0cdd70 legacy=0x7f0cec06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.351+0000 7f0cf1c00640 1 -- 172.21.15.2:0/4256099377 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec0705f0 con 0x7f0cec144c00 2024-01-15T18:17:56.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.353+0000 7f0cf1c00640 1 --2- 172.21.15.2:0/4256099377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 0x7f0cec0cd830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009e50 tx=0x7f0cdc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77889e92f6a4d259 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:56.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.353+0000 7f0cebe00640 1 -- 172.21.15.2:0/4256099377 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc02cb00 con 0x7f0cec144c00 2024-01-15T18:17:56.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.353+0000 7f0cebe00640 1 -- 172.21.15.2:0/4256099377 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cdc02cc60 con 0x7f0cec144c00 2024-01-15T18:17:56.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.353+0000 7f0cebe00640 1 -- 172.21.15.2:0/4256099377 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cdc0343f0 con 0x7f0cec144c00 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 -- 172.21.15.2:0/4256099377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 msgr2=0x7f0cec0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 --2- 172.21.15.2:0/4256099377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 0x7f0cec0cd830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc009e50 tx=0x7f0cdc02bf60 comp rx=0 tx=0).stop 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 -- 172.21.15.2:0/4256099377 shutdown_connections 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 --2- 172.21.15.2:0/4256099377 >> v2:172.21.15.137:3300/0 conn(0x7f0cec06d000 0x7f0cec070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 --2- 172.21.15.2:0/4256099377 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec144c00 0x7f0cec0cd830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 -- 172.21.15.2:0/4256099377 >> 172.21.15.2:0/4256099377 conn(0x7f0cec0b45f0 msgr2=0x7f0cec0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 -- 172.21.15.2:0/4256099377 shutdown_connections 2024-01-15T18:17:56.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.354+0000 7f0cf4600640 1 -- 172.21.15.2:0/4256099377 wait complete. 2024-01-15T18:17:56.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.356+0000 7f0cf4600640 1 Processor -- start 2024-01-15T18:17:56.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.356+0000 7f0cf4600640 1 -- start start 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.357+0000 7f0cf4600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec06d000 0x7f0cec1405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.357+0000 7f0cf4600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0cdd70 0x7f0cec0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.357+0000 7f0cf4600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec144c00 0x7f0cec0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.357+0000 7f0cf4600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec1411c0 con 0x7f0cec144c00 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.357+0000 7f0cf4600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec13d0d0 con 0x7f0cec06d000 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.357+0000 7f0cf4600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec13c7a0 con 0x7f0cec0cdd70 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec144c00 0x7f0cec0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec144c00 0x7f0cec0b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36962/0 (socket says 172.21.15.2:36962) 2024-01-15T18:17:56.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 -- 172.21.15.2:0/3164726652 learned_addr learned my addr 172.21.15.2:0/3164726652 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:56.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf1c00640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec06d000 0x7f0cec1405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf1200640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0cdd70 0x7f0cec0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 -- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0cdd70 msgr2=0x7f0cec0b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cec0cdd70 0x7f0cec0b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 -- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec06d000 msgr2=0x7f0cec1405c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec06d000 0x7f0cec1405c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 -- 172.21.15.2:0/3164726652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc009b30 con 0x7f0cec144c00 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf1c00640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cec06d000 0x7f0cec1405c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.358+0000 7f0cf2600640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cec144c00 0x7f0cec0b0de0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400e930 tx=0x7f0ce400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.359+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4009690 con 0x7f0cec144c00 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.359+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce400c980 con 0x7f0cec144c00 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.359+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce400cd30 con 0x7f0cec144c00 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.359+0000 7f0cf4600640 1 -- 172.21.15.2:0/3164726652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec13ca80 con 0x7f0cec144c00 2024-01-15T18:17:56.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.359+0000 7f0cf4600640 1 -- 172.21.15.2:0/3164726652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec0b14f0 con 0x7f0cec144c00 2024-01-15T18:17:56.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.360+0000 7f0cf4600640 1 -- 172.21.15.2:0/3164726652 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec13d280 con 0x7f0cec144c00 2024-01-15T18:17:56.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.361+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f0ce4013470 con 0x7f0cec144c00 2024-01-15T18:17:56.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.362+0000 7f0ce9600640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406c670 0x7f0cc406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:56.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.362+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f0ce4088e00 con 0x7f0cec144c00 2024-01-15T18:17:56.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.363+0000 7f0cf1c00640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406c670 0x7f0cc406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:56.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.364+0000 7f0cf1c00640 1 --2- 172.21.15.2:0/3164726652 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cc406c670 0x7f0cc406eb30 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc035000 tx=0x7f0cdc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:56.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.367+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cec13d280 con 0x7f0cec144c00 2024-01-15T18:17:56.603 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.601+0000 7f0cf4600640 1 -- 172.21.15.2:0/3164726652 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cec13d240 con 0x7f0cc406c670 2024-01-15T18:17:56.616 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:56.613+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:56.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:56.614+0000 7f0ce9600640 1 -- 172.21.15.2:0/3164726652 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0cec13d240 con 0x7f0cc406c670 2024-01-15T18:17:56.617 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:56.617 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:17:58.631 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:17:58.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.868+0000 7f72d4000640 1 Processor -- start 2024-01-15T18:17:58.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.868+0000 7f72d4000640 1 -- start start 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.870+0000 7f72d4000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f72cc14afd0 0x7f72cc0b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.870+0000 7f72d4000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f72cc14b3b0 0x7f72cc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.870+0000 7f72d4000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72cc0b25f0 con 0x7f72cc06ce30 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.870+0000 7f72d4000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72cc070ff0 con 0x7f72cc14afd0 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.870+0000 7f72d4000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72cc06d2a0 con 0x7f72cc14b3b0 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f72cc14afd0 0x7f72cc0b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f72cc14afd0 0x7f72cc0b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36966/0 (socket says 172.21.15.2:36966) 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 -- 172.21.15.2:0/1775048059 learned_addr learned my addr 172.21.15.2:0/1775048059 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:58.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d0c00640 1 --2- 172.21.15.2:0/1775048059 >> v2:172.21.15.137:3300/0 conn(0x7f72cc14b3b0 0x7f72cc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 -- 172.21.15.2:0/1775048059 >> v2:172.21.15.137:3300/0 conn(0x7f72cc14b3b0 msgr2=0x7f72cc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:58.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 --2- 172.21.15.2:0/1775048059 >> v2:172.21.15.137:3300/0 conn(0x7f72cc14b3b0 0x7f72cc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 -- 172.21.15.2:0/1775048059 >> v1:172.21.15.137:6789/0 conn(0x7f72cc06ce30 legacy=0x7f72cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:17:58.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d1600640 1 -- 172.21.15.2:0/1775048059 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc0b28d0 con 0x7f72cc14afd0 2024-01-15T18:17:58.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.871+0000 7f72d0c00640 1 --2- 172.21.15.2:0/1775048059 >> v2:172.21.15.137:3300/0 conn(0x7f72cc14b3b0 0x7f72cc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:17:58.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.872+0000 7f72d1600640 1 --2- 172.21.15.2:0/1775048059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc14afd0 0x7f72cc0b33b0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0aeb00 tx=0x7f72bc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834ca33babb0f3ec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72cbe00640 1 -- 172.21.15.2:0/1775048059 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc0317e0 con 0x7f72cc14afd0 2024-01-15T18:17:58.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72cbe00640 1 -- 172.21.15.2:0/1775048059 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc031940 con 0x7f72cc14afd0 2024-01-15T18:17:58.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72cbe00640 1 -- 172.21.15.2:0/1775048059 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc031cd0 con 0x7f72cc14afd0 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72d4000640 1 -- 172.21.15.2:0/1775048059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc14afd0 msgr2=0x7f72cc0b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72d4000640 1 --2- 172.21.15.2:0/1775048059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc14afd0 0x7f72cc0b33b0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f72cc0aeb00 tx=0x7f72bc02b880 comp rx=0 tx=0).stop 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72d4000640 1 -- 172.21.15.2:0/1775048059 shutdown_connections 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72d4000640 1 --2- 172.21.15.2:0/1775048059 >> v2:172.21.15.137:3300/0 conn(0x7f72cc14b3b0 0x7f72cc06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72d4000640 1 --2- 172.21.15.2:0/1775048059 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc14afd0 0x7f72cc0b33b0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.873+0000 7f72d4000640 1 -- 172.21.15.2:0/1775048059 >> 172.21.15.2:0/1775048059 conn(0x7f72cc0b45f0 msgr2=0x7f72cc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.874+0000 7f72d4000640 1 -- 172.21.15.2:0/1775048059 shutdown_connections 2024-01-15T18:17:58.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.874+0000 7f72d4000640 1 -- 172.21.15.2:0/1775048059 wait complete. 2024-01-15T18:17:58.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.875+0000 7f72d4000640 1 Processor -- start 2024-01-15T18:17:58.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.875+0000 7f72d4000640 1 -- start start 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.876+0000 7f72d4000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc06ce30 0x7f72cc0aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.876+0000 7f72d4000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72cc14afd0 0x7f72cc0addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d4000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72cc14b3b0 0x7f72cc0ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d4000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc0aa6a0 con 0x7f72cc06ce30 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d4000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc0a9a20 con 0x7f72cc14b3b0 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d4000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc0a9d00 con 0x7f72cc14afd0 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc06ce30 0x7f72cc0aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc06ce30 0x7f72cc0aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36978/0 (socket says 172.21.15.2:36978) 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 -- 172.21.15.2:0/45425093 learned_addr learned my addr 172.21.15.2:0/45425093 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d0c00640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72cc14afd0 0x7f72cc0addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d2000640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72cc14b3b0 0x7f72cc0ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 -- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72cc14afd0 msgr2=0x7f72cc0addf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f72cc14afd0 0x7f72cc0addf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 -- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72cc14b3b0 msgr2=0x7f72cc0ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72cc14b3b0 0x7f72cc0ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d1600640 1 -- 172.21.15.2:0/45425093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc0094d0 con 0x7f72cc06ce30 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.877+0000 7f72d2000640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f72cc14b3b0 0x7f72cc0ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.878+0000 7f72d1600640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f72cc06ce30 0x7f72cc0aaac0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f72bc000c00 tx=0x7f72bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.878+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc032880 con 0x7f72cc06ce30 2024-01-15T18:17:58.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.878+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc032a00 con 0x7f72cc06ce30 2024-01-15T18:17:58.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.878+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc032db0 con 0x7f72cc06ce30 2024-01-15T18:17:58.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.878+0000 7f72d4000640 1 -- 172.21.15.2:0/45425093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72cc0a9fe0 con 0x7f72cc06ce30 2024-01-15T18:17:58.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.878+0000 7f72d4000640 1 -- 172.21.15.2:0/45425093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72cc0ab000 con 0x7f72cc06ce30 2024-01-15T18:17:58.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.879+0000 7f72d4000640 1 -- 172.21.15.2:0/45425093 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72cc0a9e40 con 0x7f72cc06ce30 2024-01-15T18:17:58.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.880+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f72bc04d3f0 con 0x7f72cc06ce30 2024-01-15T18:17:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.881+0000 7f72c9600640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f72a406c670 0x7f72a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:17:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.881+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f72bc0aa9a0 con 0x7f72cc06ce30 2024-01-15T18:17:58.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.882+0000 7f72d0c00640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f72a406c670 0x7f72a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:17:58.885 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.883+0000 7f72d0c00640 1 --2- 172.21.15.2:0/45425093 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f72a406c670 0x7f72a406eb30 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f72cc153e40 tx=0x7f72b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:17:58.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:58.886+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72cc0a9e40 con 0x7f72cc06ce30 2024-01-15T18:17:59.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:59.116+0000 7f72d4000640 1 -- 172.21.15.2:0/45425093 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72cc14a8a0 con 0x7f72a406c670 2024-01-15T18:17:59.132 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:17:59.129+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:17:59.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:17:59.130+0000 7f72c9600640 1 -- 172.21.15.2:0/45425093 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f72cc14a8a0 con 0x7f72a406c670 2024-01-15T18:17:59.133 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:17:59.133 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:17:59.147 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:01.149 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:01.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.384+0000 7f17ff200640 1 Processor -- start 2024-01-15T18:18:01.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.384+0000 7f17ff200640 1 -- start start 2024-01-15T18:18:01.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17ff200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f17f81439f0 0x7f17f80702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:01.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17ff200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17f806c8f0 0x7f17f806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:01.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17ff200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17f815dd70 con 0x7f17f80b3420 2024-01-15T18:18:01.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17ff200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f80b38b0 con 0x7f17f806c8f0 2024-01-15T18:18:01.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17ff200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17f806d230 con 0x7f17f81439f0 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17fd200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17f806c8f0 0x7f17f806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17fd200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f17f806c8f0 0x7f17f806ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36992/0 (socket says 172.21.15.2:36992) 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.386+0000 7f17fd200640 1 -- 172.21.15.2:0/3795863082 learned_addr learned my addr 172.21.15.2:0/3795863082 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.387+0000 7f17f7e00640 1 --2- 172.21.15.2:0/3795863082 >> v2:172.21.15.137:3300/0 conn(0x7f17f81439f0 0x7f17f80702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.387+0000 7f17fd200640 1 -- 172.21.15.2:0/3795863082 >> v2:172.21.15.137:3300/0 conn(0x7f17f81439f0 msgr2=0x7f17f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.387+0000 7f17fd200640 1 --2- 172.21.15.2:0/3795863082 >> v2:172.21.15.137:3300/0 conn(0x7f17f81439f0 0x7f17f80702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.387+0000 7f17fd200640 1 -- 172.21.15.2:0/3795863082 >> v1:172.21.15.137:6789/0 conn(0x7f17f80b3420 legacy=0x7f17f80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:01.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.387+0000 7f17fd200640 1 -- 172.21.15.2:0/3795863082 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f80711b0 con 0x7f17f806c8f0 2024-01-15T18:18:01.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.388+0000 7f17fd200640 1 --2- 172.21.15.2:0/3795863082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f806c8f0 0x7f17f806ccf0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f17f80b28d0 tx=0x7f17f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d5aabd8625aeb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:01.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17f6a00640 1 -- 172.21.15.2:0/3795863082 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f002ca80 con 0x7f17f806c8f0 2024-01-15T18:18:01.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17f6a00640 1 -- 172.21.15.2:0/3795863082 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f002cbe0 con 0x7f17f806c8f0 2024-01-15T18:18:01.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17f6a00640 1 -- 172.21.15.2:0/3795863082 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f00343f0 con 0x7f17f806c8f0 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 -- 172.21.15.2:0/3795863082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f806c8f0 msgr2=0x7f17f806ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 --2- 172.21.15.2:0/3795863082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f806c8f0 0x7f17f806ccf0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f17f80b28d0 tx=0x7f17f002bee0 comp rx=0 tx=0).stop 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 -- 172.21.15.2:0/3795863082 shutdown_connections 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 --2- 172.21.15.2:0/3795863082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f806c8f0 0x7f17f806ccf0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 --2- 172.21.15.2:0/3795863082 >> v2:172.21.15.137:3300/0 conn(0x7f17f81439f0 0x7f17f80702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 -- 172.21.15.2:0/3795863082 >> 172.21.15.2:0/3795863082 conn(0x7f17f80b45f0 msgr2=0x7f17f80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.389+0000 7f17ff200640 1 -- 172.21.15.2:0/3795863082 shutdown_connections 2024-01-15T18:18:01.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.390+0000 7f17ff200640 1 -- 172.21.15.2:0/3795863082 wait complete. 2024-01-15T18:18:01.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.391+0000 7f17ff200640 1 Processor -- start 2024-01-15T18:18:01.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.391+0000 7f17ff200640 1 -- start start 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.392+0000 7f17ff200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17f806c8f0 0x7f17f8150a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.392+0000 7f17ff200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17f80b3420 0x7f17f80cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.392+0000 7f17ff200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f81439f0 0x7f17f815c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.392+0000 7f17ff200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f813a190 con 0x7f17f81439f0 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.392+0000 7f17ff200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f8158b50 con 0x7f17f80b3420 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.392+0000 7f17ff200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17f80cd100 con 0x7f17f806c8f0 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f81439f0 0x7f17f815c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f81439f0 0x7f17f815c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:36994/0 (socket says 172.21.15.2:36994) 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 -- 172.21.15.2:0/1704252854 learned_addr learned my addr 172.21.15.2:0/1704252854 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17f7e00640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17f806c8f0 0x7f17f8150a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17f7400640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17f80b3420 0x7f17f80cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:01.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 -- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17f806c8f0 msgr2=0x7f17f8150a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:01.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f17f806c8f0 0x7f17f8150a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:01.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 -- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17f80b3420 msgr2=0x7f17f80cdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:01.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f17f80b3420 0x7f17f80cdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:01.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.393+0000 7f17fd200640 1 -- 172.21.15.2:0/1704252854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f0009b30 con 0x7f17f81439f0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.394+0000 7f17fd200640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f17f81439f0 0x7f17f815c6e0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f17f0036040 tx=0x7f17f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.394+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f0009510 con 0x7f17f81439f0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.394+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17f003e450 con 0x7f17f81439f0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.394+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f003e7e0 con 0x7f17f81439f0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.394+0000 7f17ff200640 1 -- 172.21.15.2:0/1704252854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f80cf330 con 0x7f17f81439f0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.394+0000 7f17ff200640 1 -- 172.21.15.2:0/1704252854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f80cd3c0 con 0x7f17f81439f0 2024-01-15T18:18:01.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.395+0000 7f17ff200640 1 -- 172.21.15.2:0/1704252854 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f80686f0 con 0x7f17f81439f0 2024-01-15T18:18:01.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.396+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f17f002cd50 con 0x7f17f81439f0 2024-01-15T18:18:01.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.397+0000 7f17efe00640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17d006c6a0 0x7f17d006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:01.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.397+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f17f00ab720 con 0x7f17f81439f0 2024-01-15T18:18:01.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.398+0000 7f17f7e00640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17d006c6a0 0x7f17d006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:01.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.399+0000 7f17f7e00640 1 --2- 172.21.15.2:0/1704252854 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f17d006c6a0 0x7f17d006eb60 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f17e8006fd0 tx=0x7f17e8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:01.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.402+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f17f0077d50 con 0x7f17f81439f0 2024-01-15T18:18:01.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.627+0000 7f17ff200640 1 -- 172.21.15.2:0/1704252854 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f17f8070bf0 con 0x7f17d006c6a0 2024-01-15T18:18:01.643 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:01.640+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:01.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:01.641+0000 7f17efe00640 1 -- 172.21.15.2:0/1704252854 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f17e800f860 con 0x7f17d006c6a0 2024-01-15T18:18:01.644 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:01.644 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:03.662 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:03.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.897+0000 7f1e85c00640 1 Processor -- start 2024-01-15T18:18:03.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.897+0000 7f1e85c00640 1 -- start start 2024-01-15T18:18:03.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.898+0000 7f1e85c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1e8015d730 0x7f1e800d1270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.898+0000 7f1e85c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e800d1900 con 0x7f1e800b8c30 2024-01-15T18:18:03.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.898+0000 7f1e85c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e800d1be0 con 0x7f1e800b4c70 2024-01-15T18:18:03.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.898+0000 7f1e85c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e800d30b0 con 0x7f1e8015d730 2024-01-15T18:18:03.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.899+0000 7f1e7f400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f1e800b8c30 0x7f1e800d29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40608/0 (socket says 172.21.15.2:40608) 2024-01-15T18:18:03.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.899+0000 7f1e7f400640 1 -- 172.21.15.2:0/2760081176 learned_addr learned my addr 172.21.15.2:0/2760081176 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:03.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.899+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2760081176 >> v2:172.21.15.137:3300/0 conn(0x7f1e8015d730 0x7f1e800d1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083732955 0 0) 0x7f1e800d1900 con 0x7f1e800b8c30 2024-01-15T18:18:03.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e60003520 con 0x7f1e800b8c30 2024-01-15T18:18:03.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102131794 0 0) 0x7f1e60003520 con 0x7f1e800b8c30 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 >> v2:172.21.15.137:3300/0 conn(0x7f1e8015d730 msgr2=0x7f1e800d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 --2- 172.21.15.2:0/2760081176 >> v2:172.21.15.137:3300/0 conn(0x7f1e8015d730 0x7f1e800d1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 >> v1:172.21.15.137:6789/0 conn(0x7f1e800b4c70 legacy=0x7f1e800b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e800d1ec0 con 0x7f1e800b8c30 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f1e780019f0 con 0x7f1e800b8c30 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 >> v1:172.21.15.2:6789/0 conn(0x7f1e800b8c30 legacy=0x7f1e800d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e60003970 0x7f1e60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e600047c0 0x7f1e60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e60005180 0x7f1e6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800d30b0 con 0x7f1e600047c0 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6000c020 con 0x7f1e60005180 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6000c1e0 con 0x7f1e60003970 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.900+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e600047c0 0x7f1e60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7f400640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e60005180 0x7f1e6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7e000640 1 -- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e60003970 msgr2=0x7f1e60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e60003970 0x7f1e60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7e000640 1 -- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e60005180 msgr2=0x7f1e6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e60005180 0x7f1e6000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7e000640 1 -- 172.21.15.2:0/2760081176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e60003520 con 0x7f1e600047c0 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7f400640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e60005180 0x7f1e6000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:03.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.901+0000 7f1e7f400640 1 -- 172.21.15.2:0/2760081176 reap_dead start 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.902+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e600047c0 0x7f1e60004c40 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009a90 tx=0x7f1e600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fa663532b256eb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.902+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6802c680 con 0x7f1e600047c0 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.902+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6802c7e0 con 0x7f1e600047c0 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.902+0000 7f1e7d600640 1 -- 172.21.15.2:0/2760081176 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6802cb70 con 0x7f1e600047c0 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 -- 172.21.15.2:0/2760081176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e600047c0 msgr2=0x7f1e60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e600047c0 0x7f1e60004c40 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7f1e68009a90 tx=0x7f1e600057f0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 -- 172.21.15.2:0/2760081176 shutdown_connections 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 --2- 172.21.15.2:0/2760081176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e600047c0 0x7f1e60004c40 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 -- 172.21.15.2:0/2760081176 >> 172.21.15.2:0/2760081176 conn(0x7f1e80072210 msgr2=0x7f1e800cfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 -- 172.21.15.2:0/2760081176 shutdown_connections 2024-01-15T18:18:03.906 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.903+0000 7f1e85c00640 1 -- 172.21.15.2:0/2760081176 wait complete. 2024-01-15T18:18:03.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.905+0000 7f1e85c00640 1 Processor -- start 2024-01-15T18:18:03.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.905+0000 7f1e85c00640 1 -- start start 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e85c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e600047c0 0x7f1e80152810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e85c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e800ca2c0 0x7f1e8015ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e85c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e8015f0f0 0x7f1e800c8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e85c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800c8720 con 0x7f1e8015f0f0 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e85c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800cad10 con 0x7f1e600047c0 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e85c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e800c80d0 con 0x7f1e800ca2c0 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e8015f0f0 0x7f1e800c8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e8015f0f0 0x7f1e800c8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46540/0 (socket says 172.21.15.2:46540) 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.906+0000 7f1e7f400640 1 -- 172.21.15.2:0/2118898404 learned_addr learned my addr 172.21.15.2:0/2118898404 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7ea00640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e600047c0 0x7f1e80152810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e800ca2c0 0x7f1e8015ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7f400640 1 -- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e800ca2c0 msgr2=0x7f1e8015ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7f400640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e800ca2c0 0x7f1e8015ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7f400640 1 -- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e600047c0 msgr2=0x7f1e80152810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7f400640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1e600047c0 0x7f1e80152810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7f400640 1 -- 172.21.15.2:0/2118898404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e800c83b0 con 0x7f1e8015f0f0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7e000640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1e800ca2c0 0x7f1e8015ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e7f400640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1e8015f0f0 0x7f1e800c8bb0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7f1e800b5bf0 tx=0x7f1e7800df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800b860 con 0x7f1e8015f0f0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7800b9e0 con 0x7f1e8015f0f0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e85c00640 1 -- 172.21.15.2:0/2118898404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e800caff0 con 0x7f1e8015f0f0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.907+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800bd90 con 0x7f1e8015f0f0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.908+0000 7f1e85c00640 1 -- 172.21.15.2:0/2118898404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e800ca740 con 0x7f1e8015f0f0 2024-01-15T18:18:03.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.909+0000 7f1e85c00640 1 -- 172.21.15.2:0/2118898404 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e800c6b90 con 0x7f1e8015f0f0 2024-01-15T18:18:03.912 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.910+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f1e78007110 con 0x7f1e8015f0f0 2024-01-15T18:18:03.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.911+0000 7f1e76a00640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e5806c670 0x7f1e5806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:03.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.911+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f1e78088a40 con 0x7f1e8015f0f0 2024-01-15T18:18:03.913 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.911+0000 7f1e7ea00640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e5806c670 0x7f1e5806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:03.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.912+0000 7f1e7ea00640 1 --2- 172.21.15.2:0/2118898404 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1e5806c670 0x7f1e5806eb30 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1e800c75d0 tx=0x7f1e70009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:03.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:03.916+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e78054f50 con 0x7f1e8015f0f0 2024-01-15T18:18:04.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:04.147+0000 7f1e85c00640 1 -- 172.21.15.2:0/2118898404 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1e800c7a10 con 0x7f1e5806c670 2024-01-15T18:18:04.163 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:04.159+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:04.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:04.161+0000 7f1e76a00640 1 -- 172.21.15.2:0/2118898404 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1e800c7a10 con 0x7f1e5806c670 2024-01-15T18:18:04.164 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:04.164 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:04.488 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:04.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:06.179 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:06.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.415+0000 7f684ec00640 1 Processor -- start 2024-01-15T18:18:06.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.415+0000 7f684ec00640 1 -- start start 2024-01-15T18:18:06.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848159e60 0x7f6848071af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:06.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ec00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f684806db30 0x7f684806dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:06.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ec00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6848071660 con 0x7f68480c92a0 2024-01-15T18:18:06.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ec00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684806e520 con 0x7f684806db30 2024-01-15T18:18:06.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68480ca810 con 0x7f6848159e60 2024-01-15T18:18:06.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ce00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f68480c92a0 0x7f68480ca100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40616/0 (socket says 172.21.15.2:40616) 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684ce00640 1 -- 172.21.15.2:0/3911896556 learned_addr learned my addr 172.21.15.2:0/3911896556 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.417+0000 7f684e200640 1 --2- 172.21.15.2:0/3911896556 >> v2:172.21.15.2:3300/0 conn(0x7f684806db30 0x7f684806dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f684d800640 1 --2- 172.21.15.2:0/3911896556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848159e60 0x7f6848071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f6847e00640 1 -- 172.21.15.2:0/3911896556 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727784224 0 0) 0x7f6848071660 con 0x7f68480c92a0 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f6847e00640 1 -- 172.21.15.2:0/3911896556 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6828005730 con 0x7f68480c92a0 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f684e200640 1 -- 172.21.15.2:0/3911896556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848159e60 msgr2=0x7f6848071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f684e200640 1 --2- 172.21.15.2:0/3911896556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848159e60 0x7f6848071af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f684e200640 1 -- 172.21.15.2:0/3911896556 >> v1:172.21.15.2:6789/0 conn(0x7f68480c92a0 legacy=0x7f68480ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:06.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.418+0000 7f684e200640 1 -- 172.21.15.2:0/3911896556 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684815a240 con 0x7f684806db30 2024-01-15T18:18:06.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.419+0000 7f684e200640 1 --2- 172.21.15.2:0/3911896556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f684806db30 0x7f684806dfe0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f6840009d90 tx=0x7f684002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=712d167e498cf867 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.419+0000 7f6847e00640 1 -- 172.21.15.2:0/3911896556 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684002cc50 con 0x7f684806db30 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f6847e00640 1 -- 172.21.15.2:0/3911896556 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f684002cdb0 con 0x7f684806db30 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f6847e00640 1 -- 172.21.15.2:0/3911896556 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840034640 con 0x7f684806db30 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 -- 172.21.15.2:0/3911896556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f684806db30 msgr2=0x7f684806dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 --2- 172.21.15.2:0/3911896556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f684806db30 0x7f684806dfe0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f6840009d90 tx=0x7f684002bea0 comp rx=0 tx=0).stop 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 -- 172.21.15.2:0/3911896556 shutdown_connections 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 --2- 172.21.15.2:0/3911896556 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f684806db30 0x7f684806dfe0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 --2- 172.21.15.2:0/3911896556 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6848159e60 0x7f6848071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 -- 172.21.15.2:0/3911896556 >> 172.21.15.2:0/3911896556 conn(0x7f68480cebc0 msgr2=0x7f68480cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:06.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 -- 172.21.15.2:0/3911896556 shutdown_connections 2024-01-15T18:18:06.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.420+0000 7f684ec00640 1 -- 172.21.15.2:0/3911896556 wait complete. 2024-01-15T18:18:06.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.422+0000 7f684ec00640 1 Processor -- start 2024-01-15T18:18:06.426 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.422+0000 7f684ec00640 1 -- start start 2024-01-15T18:18:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f684806db30 0x7f68480b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68480c92a0 0x7f6848151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:06.429 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848159e60 0x7f6848152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68480b2d30 con 0x7f68480c92a0 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68480b3010 con 0x7f6848159e60 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68480b32f0 con 0x7f684806db30 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.423+0000 7f684ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68480c92a0 0x7f6848151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68480c92a0 0x7f6848151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46558/0 (socket says 172.21.15.2:46558) 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 -- 172.21.15.2:0/1218910767 learned_addr learned my addr 172.21.15.2:0/1218910767 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684d800640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f684806db30 0x7f68480b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684e200640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848159e60 0x7f6848152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:06.430 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 -- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f684806db30 msgr2=0x7f68480b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f684806db30 0x7f68480b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 -- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848159e60 msgr2=0x7f6848152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f6848159e60 0x7f6848152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 -- 172.21.15.2:0/1218910767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6840009a70 con 0x7f68480c92a0 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.424+0000 7f684ce00640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f68480c92a0 0x7f6848151be0 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f683000c400 tx=0x7f683000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.425+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830005ce0 con 0x7f68480c92a0 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.425+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6830005e60 con 0x7f68480c92a0 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.425+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68300072f0 con 0x7f68480c92a0 2024-01-15T18:18:06.431 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.425+0000 7f684ec00640 1 -- 172.21.15.2:0/1218910767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68480b35d0 con 0x7f68480c92a0 2024-01-15T18:18:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.425+0000 7f684ec00640 1 -- 172.21.15.2:0/1218910767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6848153170 con 0x7f68480c92a0 2024-01-15T18:18:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.426+0000 7f684ec00640 1 -- 172.21.15.2:0/1218910767 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68480b3af0 con 0x7f68480c92a0 2024-01-15T18:18:06.432 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.427+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f6830007450 con 0x7f68480c92a0 2024-01-15T18:18:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.428+0000 7f6845600640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f682006c670 0x7f682006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.428+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f68300882a0 con 0x7f68480c92a0 2024-01-15T18:18:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.429+0000 7f684d800640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f682006c670 0x7f682006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:06.433 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.430+0000 7f684d800640 1 --2- 172.21.15.2:0/1218910767 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f682006c670 0x7f682006eb30 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f6838006fd0 tx=0x7f6838008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:06.435 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.432+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68480b3af0 con 0x7f68480c92a0 2024-01-15T18:18:06.664 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.661+0000 7f684ec00640 1 -- 172.21.15.2:0/1218910767 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f68480b3180 con 0x7f682006c670 2024-01-15T18:18:06.677 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:06.674+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:06.678 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:06.675+0000 7f6845600640 1 -- 172.21.15.2:0/1218910767 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f68480b3180 con 0x7f682006c670 2024-01-15T18:18:06.679 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:06.679 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:06.692 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:08.693 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.933+0000 7f017aa00640 1 Processor -- start 2024-01-15T18:18:08.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.933+0000 7f017aa00640 1 -- start start 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.934+0000 7f017aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740c9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.934+0000 7f017aa00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f01740c9960 0x7f01741631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.934+0000 7f017aa00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017406e3b0 con 0x7f017406d050 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.934+0000 7f017aa00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0174159aa0 con 0x7f01740c9960 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.934+0000 7f017aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01740c9df0 con 0x7f01740cda80 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f01740c9960 0x7f01741631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f01740c9960 0x7f01741631a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46564/0 (socket says 172.21.15.2:46564) 2024-01-15T18:18:08.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 -- 172.21.15.2:0/3210562341 learned_addr learned my addr 172.21.15.2:0/3210562341 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 -- 172.21.15.2:0/3210562341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 msgr2=0x7f01740c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f0178c00640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740c9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 -- 172.21.15.2:0/3210562341 >> v1:172.21.15.137:6789/0 conn(0x7f017406d050 legacy=0x7f0174142830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f017a000640 1 -- 172.21.15.2:0/3210562341 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017406d7a0 con 0x7f01740c9960 2024-01-15T18:18:08.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.935+0000 7f0178c00640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.936+0000 7f017a000640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f01741631a0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e50 tx=0x7f016c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a75220aee479b868 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.937+0000 7f0173e00640 1 -- 172.21.15.2:0/3210562341 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c02c9f0 con 0x7f01740c9960 2024-01-15T18:18:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.937+0000 7f0173e00640 1 -- 172.21.15.2:0/3210562341 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016c02cb50 con 0x7f01740c9960 2024-01-15T18:18:08.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.937+0000 7f0173e00640 1 -- 172.21.15.2:0/3210562341 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016c0343f0 con 0x7f01740c9960 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.937+0000 7f017aa00640 1 -- 172.21.15.2:0/3210562341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 msgr2=0x7f01741631a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.937+0000 7f017aa00640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f01741631a0 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f016c009e50 tx=0x7f016c02be50 comp rx=0 tx=0).stop 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.937+0000 7f017aa00640 1 -- 172.21.15.2:0/3210562341 shutdown_connections 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.938+0000 7f017aa00640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f01741631a0 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.938+0000 7f017aa00640 1 --2- 172.21.15.2:0/3210562341 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.938+0000 7f017aa00640 1 -- 172.21.15.2:0/3210562341 >> 172.21.15.2:0/3210562341 conn(0x7f01740cebc0 msgr2=0x7f01740cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:08.940 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.938+0000 7f017aa00640 1 -- 172.21.15.2:0/3210562341 shutdown_connections 2024-01-15T18:18:08.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.938+0000 7f017aa00640 1 -- 172.21.15.2:0/3210562341 wait complete. 2024-01-15T18:18:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.939+0000 7f017aa00640 1 Processor -- start 2024-01-15T18:18:08.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.939+0000 7f017aa00640 1 -- start start 2024-01-15T18:18:08.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.940+0000 7f017aa00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017406d050 0x7f0174158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:08.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.940+0000 7f017aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f017413d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:08.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.940+0000 7f017aa00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:08.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.940+0000 7f017aa00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01741390e0 con 0x7f01740c9960 2024-01-15T18:18:08.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.940+0000 7f017aa00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01741393c0 con 0x7f017406d050 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.940+0000 7f017aa00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01741396a0 con 0x7f01740cda80 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f017413d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f017413d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46572/0 (socket says 172.21.15.2:46572) 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 -- 172.21.15.2:0/3885459095 learned_addr learned my addr 172.21.15.2:0/3885459095 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0179600640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017406d050 0x7f0174158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f017a000640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 -- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 msgr2=0x7f01740ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f01740cda80 0x7f01740ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 -- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017406d050 msgr2=0x7f0174158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017406d050 0x7f0174158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 -- 172.21.15.2:0/3885459095 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c009b30 con 0x7f01740c9960 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0179600640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f017406d050 0x7f0174158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.941+0000 7f0178c00640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f01740c9960 0x7f017413d550 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f015c00c7f0 tx=0x7f015c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:08.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.942+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c007820 con 0x7f01740c9960 2024-01-15T18:18:08.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.942+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f015c007980 con 0x7f01740c9960 2024-01-15T18:18:08.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.942+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f015c007cf0 con 0x7f01740c9960 2024-01-15T18:18:08.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.942+0000 7f017aa00640 1 -- 172.21.15.2:0/3885459095 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0174139980 con 0x7f01740c9960 2024-01-15T18:18:08.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.942+0000 7f017aa00640 1 -- 172.21.15.2:0/3885459095 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0174099a00 con 0x7f01740c9960 2024-01-15T18:18:08.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.943+0000 7f017aa00640 1 -- 172.21.15.2:0/3885459095 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0174074770 con 0x7f01740c9960 2024-01-15T18:18:08.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.944+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f015c0113f0 con 0x7f01740c9960 2024-01-15T18:18:08.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.945+0000 7f0171600640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f014c06c670 0x7f014c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:08.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.945+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f015c088f30 con 0x7f01740c9960 2024-01-15T18:18:08.948 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.946+0000 7f0179600640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f014c06c670 0x7f014c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:08.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.947+0000 7f0179600640 1 --2- 172.21.15.2:0/3885459095 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f014c06c670 0x7f014c06eb30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f01740afb00 tx=0x7f0164009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:08.952 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:08.950+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f015c092020 con 0x7f01740c9960 2024-01-15T18:18:09.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:09.177+0000 7f017aa00640 1 -- 172.21.15.2:0/3885459095 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0174139530 con 0x7f014c06c670 2024-01-15T18:18:09.192 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:09.189+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:09.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:09.191+0000 7f0171600640 1 -- 172.21.15.2:0/3885459095 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0174139530 con 0x7f014c06c670 2024-01-15T18:18:09.193 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:09.193 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:11.209 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:11.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.446+0000 7f73d3e00640 1 Processor -- start 2024-01-15T18:18:11.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.447+0000 7f73d3e00640 1 -- start start 2024-01-15T18:18:11.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40b1500 0x7f73d40b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:11.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f73d40d1930 0x7f73d40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d40b18e0 con 0x7f73d40cdd00 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d40cfa50 con 0x7f73d40d1930 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d40d02b0 con 0x7f73d40b1500 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73c9e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f73d40d1930 0x7f73d40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f73d40cdd00 0x7f73d406b6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:40628/0 (socket says 172.21.15.2:40628) 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.448+0000 7f73d3400640 1 -- 172.21.15.2:0/4025782229 learned_addr learned my addr 172.21.15.2:0/4025782229 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73d2a00640 1 --2- 172.21.15.2:0/4025782229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40b1500 0x7f73d40b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73d2000640 1 -- 172.21.15.2:0/4025782229 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623816176 0 0) 0x7f73cc001490 con 0x7f73d40cdd00 2024-01-15T18:18:11.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73d2000640 1 -- 172.21.15.2:0/4025782229 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b4003520 con 0x7f73d40cdd00 2024-01-15T18:18:11.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73c9e00640 1 -- 172.21.15.2:0/4025782229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40b1500 msgr2=0x7f73d40b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:11.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73c9e00640 1 --2- 172.21.15.2:0/4025782229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40b1500 0x7f73d40b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:11.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73c9e00640 1 -- 172.21.15.2:0/4025782229 >> v1:172.21.15.2:6789/0 conn(0x7f73d40cdd00 legacy=0x7f73d406b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:11.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.449+0000 7f73c9e00640 1 -- 172.21.15.2:0/4025782229 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d40ce160 con 0x7f73d40d1930 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.450+0000 7f73c9e00640 1 --2- 172.21.15.2:0/4025782229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40d1930 0x7f73d40cd7c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0097a0 tx=0x7f73bc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2dc15bc52e6350 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.450+0000 7f73d2000640 1 -- 172.21.15.2:0/4025782229 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc02d040 con 0x7f73d40d1930 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.450+0000 7f73d2000640 1 -- 172.21.15.2:0/4025782229 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73bc031810 con 0x7f73d40d1930 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.450+0000 7f73d2000640 1 -- 172.21.15.2:0/4025782229 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73bc031b80 con 0x7f73d40d1930 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 -- 172.21.15.2:0/4025782229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40d1930 msgr2=0x7f73d40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 --2- 172.21.15.2:0/4025782229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40d1930 0x7f73d40cd7c0 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7f73bc0097a0 tx=0x7f73bc02b8b0 comp rx=0 tx=0).stop 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 -- 172.21.15.2:0/4025782229 shutdown_connections 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 --2- 172.21.15.2:0/4025782229 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40d1930 0x7f73d40cd7c0 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:11.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 --2- 172.21.15.2:0/4025782229 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40b1500 0x7f73d40b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:11.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 -- 172.21.15.2:0/4025782229 >> 172.21.15.2:0/4025782229 conn(0x7f73d40d0e10 msgr2=0x7f73d40d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:11.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 -- 172.21.15.2:0/4025782229 shutdown_connections 2024-01-15T18:18:11.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.451+0000 7f73d3e00640 1 -- 172.21.15.2:0/4025782229 wait complete. 2024-01-15T18:18:11.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.453+0000 7f73d3e00640 1 Processor -- start 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.453+0000 7f73d3e00640 1 -- start start 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40b1500 0x7f73d4160810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d3e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73d40cdd00 0x7f73d414aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d3e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40d1930 0x7f73d40ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d3e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d415f770 con 0x7f73d40b1500 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d3e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d415fa50 con 0x7f73d40cdd00 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d3e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d415fd30 con 0x7f73d40d1930 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40b1500 0x7f73d4160810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40b1500 0x7f73d4160810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46584/0 (socket says 172.21.15.2:46584) 2024-01-15T18:18:11.459 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d2a00640 1 -- 172.21.15.2:0/2498211048 learned_addr learned my addr 172.21.15.2:0/2498211048 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d2a00640 1 -- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40d1930 msgr2=0x7f73d40ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.454+0000 7f73d2a00640 1 --2- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f73d40d1930 0x7f73d40ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73d2a00640 1 -- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73d40cdd00 msgr2=0x7f73d414aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73d2a00640 1 --2- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f73d40cdd00 0x7f73d414aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73d2a00640 1 -- 172.21.15.2:0/2498211048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73bc009400 con 0x7f73d40b1500 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73d2a00640 1 --2- 172.21.15.2:0/2498211048 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f73d40b1500 0x7f73d4160810 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f73c400d5f0 tx=0x7f73c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c40125f0 con 0x7f73d40b1500 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c4012750 con 0x7f73d40b1500 2024-01-15T18:18:11.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4012ae0 con 0x7f73d40b1500 2024-01-15T18:18:11.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73d3e00640 1 -- 172.21.15.2:0/2498211048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d4160010 con 0x7f73d40b1500 2024-01-15T18:18:11.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.455+0000 7f73d3e00640 1 -- 172.21.15.2:0/2498211048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d4160d50 con 0x7f73d40b1500 2024-01-15T18:18:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.456+0000 7f73d3e00640 1 -- 172.21.15.2:0/2498211048 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d40cd210 con 0x7f73d40b1500 2024-01-15T18:18:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.457+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f73c4012c40 con 0x7f73d40b1500 2024-01-15T18:18:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.458+0000 7f73cb400640 1 --2- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73ac06c670 0x7f73ac06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.458+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f73c4089100 con 0x7f73d40b1500 2024-01-15T18:18:11.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.459+0000 7f73c9e00640 1 --2- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73ac06c670 0x7f73ac06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:11.463 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.460+0000 7f73c9e00640 1 --2- 172.21.15.2:0/2498211048 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f73ac06c670 0x7f73ac06eb30 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f73bc036000 tx=0x7f73bc030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:11.465 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.463+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73c4055610 con 0x7f73d40b1500 2024-01-15T18:18:11.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.704+0000 7f73d3e00640 1 -- 172.21.15.2:0/2498211048 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73d415fbc0 con 0x7f73ac06c670 2024-01-15T18:18:11.720 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:11.716+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:11.720 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:11.718+0000 7f73cb400640 1 -- 172.21.15.2:0/2498211048 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f73d415fbc0 con 0x7f73ac06c670 2024-01-15T18:18:11.721 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:11.721 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:13.736 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:13.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.971+0000 7f8f0c200640 1 Processor -- start 2024-01-15T18:18:13.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.971+0000 7f8f0c200640 1 -- start start 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.973+0000 7f8f0c200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f8f04144c00 0x7f8f040cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.973+0000 7f8f0c200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8f040cdd70 0x7f8f0406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f0c200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f040ce230 con 0x7f8f0415c450 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f0c200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f040ce510 con 0x7f8f040cdd70 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f0c200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f040ce7f0 con 0x7f8f04144c00 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f08e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8f040cdd70 0x7f8f0406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f0a200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f8f0415c450 0x7f8f040b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38462/0 (socket says 172.21.15.2:38462) 2024-01-15T18:18:13.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f0a200640 1 -- 172.21.15.2:0/1181755013 learned_addr learned my addr 172.21.15.2:0/1181755013 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f09800640 1 --2- 172.21.15.2:0/1181755013 >> v2:172.21.15.137:3300/0 conn(0x7f8f04144c00 0x7f8f040cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f08e00640 1 -- 172.21.15.2:0/1181755013 >> v2:172.21.15.137:3300/0 conn(0x7f8f04144c00 msgr2=0x7f8f040cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f08e00640 1 --2- 172.21.15.2:0/1181755013 >> v2:172.21.15.137:3300/0 conn(0x7f8f04144c00 0x7f8f040cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f08e00640 1 -- 172.21.15.2:0/1181755013 >> v1:172.21.15.2:6789/0 conn(0x7f8f0415c450 legacy=0x7f8f040b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.978 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.974+0000 7f8f08e00640 1 -- 172.21.15.2:0/1181755013 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f040705f0 con 0x7f8f040cdd70 2024-01-15T18:18:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.976+0000 7f8f08e00640 1 --2- 172.21.15.2:0/1181755013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f040cdd70 0x7f8f0406c8f0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f8f040b1c60 tx=0x7f8eec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddbc82ad0f600297 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.976+0000 7f8f03e00640 1 -- 172.21.15.2:0/1181755013 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec031830 con 0x7f8f040cdd70 2024-01-15T18:18:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.976+0000 7f8f03e00640 1 -- 172.21.15.2:0/1181755013 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec031990 con 0x7f8f040cdd70 2024-01-15T18:18:13.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.976+0000 7f8f03e00640 1 -- 172.21.15.2:0/1181755013 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec031d20 con 0x7f8f040cdd70 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 -- 172.21.15.2:0/1181755013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f040cdd70 msgr2=0x7f8f0406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 --2- 172.21.15.2:0/1181755013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f040cdd70 0x7f8f0406c8f0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f8f040b1c60 tx=0x7f8eec02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 -- 172.21.15.2:0/1181755013 shutdown_connections 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 --2- 172.21.15.2:0/1181755013 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f040cdd70 0x7f8f0406c8f0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 --2- 172.21.15.2:0/1181755013 >> v2:172.21.15.137:3300/0 conn(0x7f8f04144c00 0x7f8f040cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 -- 172.21.15.2:0/1181755013 >> 172.21.15.2:0/1181755013 conn(0x7f8f040b45f0 msgr2=0x7f8f040b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:13.980 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 -- 172.21.15.2:0/1181755013 shutdown_connections 2024-01-15T18:18:13.981 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.977+0000 7f8f0c200640 1 -- 172.21.15.2:0/1181755013 wait complete. 2024-01-15T18:18:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.979+0000 7f8f0c200640 1 Processor -- start 2024-01-15T18:18:13.982 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.979+0000 7f8f0c200640 1 -- start start 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f0c200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f040cdd70 0x7f8f041f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f0c200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f04144c00 0x7f8f04142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f0c200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f0415c450 0x7f8f04140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f0c200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04131f70 con 0x7f8f04144c00 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f0c200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04156ce0 con 0x7f8f040cdd70 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f0c200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f041560d0 con 0x7f8f0415c450 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f04144c00 0x7f8f04142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f04144c00 0x7f8f04142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57242/0 (socket says 172.21.15.2:57242) 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 -- 172.21.15.2:0/3431368262 learned_addr learned my addr 172.21.15.2:0/3431368262 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f09800640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f040cdd70 0x7f8f041f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 -- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f0415c450 msgr2=0x7f8f04140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8f0415c450 0x7f8f04140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 -- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f040cdd70 msgr2=0x7f8f041f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8f040cdd70 0x7f8f041f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.980+0000 7f8f08e00640 1 -- 172.21.15.2:0/3431368262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eec009400 con 0x7f8f04144c00 2024-01-15T18:18:13.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.981+0000 7f8f08e00640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8f04144c00 0x7f8f04142a40 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009890 tx=0x7f8eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.981+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec03ba70 con 0x7f8f04144c00 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.981+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eec03bbf0 con 0x7f8f04144c00 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.981+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eec045400 con 0x7f8f04144c00 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.981+0000 7f8f0c200640 1 -- 172.21.15.2:0/3431368262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f041563b0 con 0x7f8f04144c00 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.981+0000 7f8f0c200640 1 -- 172.21.15.2:0/3431368262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f04156970 con 0x7f8f04144c00 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.982+0000 7f8f0c200640 1 -- 172.21.15.2:0/3431368262 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f040686f0 con 0x7f8f04144c00 2024-01-15T18:18:13.986 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.983+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f8eec04e440 con 0x7f8f04144c00 2024-01-15T18:18:13.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.984+0000 7f8f01600640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8edc06c6c0 0x7f8edc06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:13.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.984+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8eec0aaf60 con 0x7f8f04144c00 2024-01-15T18:18:13.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.985+0000 7f8f09800640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8edc06c6c0 0x7f8edc06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:13.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.986+0000 7f8f09800640 1 --2- 172.21.15.2:0/3431368262 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f8edc06c6c0 0x7f8edc06eb80 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8ef4006fd0 tx=0x7f8ef4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:13.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:13.989+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8eec0773e0 con 0x7f8f04144c00 2024-01-15T18:18:14.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:14.226+0000 7f8f0c200640 1 -- 172.21.15.2:0/3431368262 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8f0415a9c0 con 0x7f8edc06c6c0 2024-01-15T18:18:14.242 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:14.240+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:14.240+0000 7f8f01600640 1 -- 172.21.15.2:0/3431368262 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8f0415a9c0 con 0x7f8edc06c6c0 2024-01-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:14.243 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:16.259 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:16.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.492+0000 7f2ff6200640 1 Processor -- start 2024-01-15T18:18:16.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.492+0000 7f2ff6200640 1 -- start start 2024-01-15T18:18:16.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff6200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2ff00c92a0 0x7f2ff00ca100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff6200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff0071660 con 0x7f2ff0159e60 2024-01-15T18:18:16.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff6200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff006e520 con 0x7f2ff00c92a0 2024-01-15T18:18:16.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015a240 con 0x7f2ff006db30 2024-01-15T18:18:16.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff4e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2ff0159e60 0x7f2ff0071af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38464/0 (socket says 172.21.15.2:38464) 2024-01-15T18:18:16.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2ff4e00640 1 -- 172.21.15.2:0/1963395554 learned_addr learned my addr 172.21.15.2:0/1963395554 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:16.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.494+0000 7f2fe7e00640 1 --2- 172.21.15.2:0/1963395554 >> v2:172.21.15.2:3300/0 conn(0x7f2ff00c92a0 0x7f2ff00ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.495+0000 7f2ff5800640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.495+0000 7f2fe7e00640 1 -- 172.21.15.2:0/1963395554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 msgr2=0x7f2ff006dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:16.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.495+0000 7f2fe7e00640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff006dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.495+0000 7f2fe7e00640 1 -- 172.21.15.2:0/1963395554 >> v1:172.21.15.2:6789/0 conn(0x7f2ff0159e60 legacy=0x7f2ff0071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.495+0000 7f2fe7e00640 1 -- 172.21.15.2:0/1963395554 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff0070a50 con 0x7f2ff00c92a0 2024-01-15T18:18:16.499 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.495+0000 7f2ff5800640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff006dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:16.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.496+0000 7f2fe7e00640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff00c92a0 0x7f2ff00ca100 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80097a0 tx=0x7f2fd802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17f9324472c1ef8b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:16.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.496+0000 7f2fefe00640 1 -- 172.21.15.2:0/1963395554 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd802d040 con 0x7f2ff00c92a0 2024-01-15T18:18:16.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.496+0000 7f2fefe00640 1 -- 172.21.15.2:0/1963395554 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8031810 con 0x7f2ff00c92a0 2024-01-15T18:18:16.500 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.496+0000 7f2fefe00640 1 -- 172.21.15.2:0/1963395554 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8031ba0 con 0x7f2ff00c92a0 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 -- 172.21.15.2:0/1963395554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff00c92a0 msgr2=0x7f2ff00ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff00c92a0 0x7f2ff00ca100 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f2fd80097a0 tx=0x7f2fd802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 -- 172.21.15.2:0/1963395554 shutdown_connections 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff006dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 --2- 172.21.15.2:0/1963395554 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff00c92a0 0x7f2ff00ca100 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 -- 172.21.15.2:0/1963395554 >> 172.21.15.2:0/1963395554 conn(0x7f2ff00cebc0 msgr2=0x7f2ff00cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 -- 172.21.15.2:0/1963395554 shutdown_connections 2024-01-15T18:18:16.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.497+0000 7f2ff6200640 1 -- 172.21.15.2:0/1963395554 wait complete. 2024-01-15T18:18:16.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.499+0000 7f2ff6200640 1 Processor -- start 2024-01-15T18:18:16.502 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.499+0000 7f2ff6200640 1 -- start start 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff6200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff015beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff6200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00c92a0 0x7f2ff0099dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff6200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff0159e60 0x7f2ff0143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff6200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015c790 con 0x7f2ff0159e60 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff6200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015ca70 con 0x7f2ff00c92a0 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff6200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff015cd50 con 0x7f2ff006db30 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.500+0000 7f2ff5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff0159e60 0x7f2ff0143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff0159e60 0x7f2ff0143210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57268/0 (socket says 172.21.15.2:57268) 2024-01-15T18:18:16.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 -- 172.21.15.2:0/1428624489 learned_addr learned my addr 172.21.15.2:0/1428624489 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:16.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2fe7e00640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00c92a0 0x7f2ff0099dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 -- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 msgr2=0x7f2ff015beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.505 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2ff006db30 0x7f2ff015beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 -- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00c92a0 msgr2=0x7f2ff0099dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2ff00c92a0 0x7f2ff0099dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 -- 172.21.15.2:0/1428624489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd8009400 con 0x7f2ff0159e60 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.501+0000 7f2ff5800640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2ff0159e60 0x7f2ff0143210 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cef0 tx=0x7f2fe800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.502+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800d840 con 0x7f2ff0159e60 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.502+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe800d9c0 con 0x7f2ff0159e60 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.502+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800dd70 con 0x7f2ff0159e60 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.502+0000 7f2ff6200640 1 -- 172.21.15.2:0/1428624489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff0156eb0 con 0x7f2ff0159e60 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.502+0000 7f2ff6200640 1 -- 172.21.15.2:0/1428624489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff015c3f0 con 0x7f2ff0159e60 2024-01-15T18:18:16.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.503+0000 7f2ff6200640 1 -- 172.21.15.2:0/1428624489 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff01573d0 con 0x7f2ff0159e60 2024-01-15T18:18:16.507 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.504+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f2fe801e3b0 con 0x7f2ff0159e60 2024-01-15T18:18:16.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.505+0000 7f2fed600640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fc806c670 0x7f2fc806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:16.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.505+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f2fe8089700 con 0x7f2ff0159e60 2024-01-15T18:18:16.509 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.506+0000 7f2ff4e00640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fc806c670 0x7f2fc806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:16.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.507+0000 7f2ff4e00640 1 --2- 172.21.15.2:0/1428624489 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2fc806c670 0x7f2fc806eb30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0001550 tx=0x7f2fe0007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:16.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.510+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ff01573d0 con 0x7f2ff0159e60 2024-01-15T18:18:16.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.740+0000 7f2ff6200640 1 -- 172.21.15.2:0/1428624489 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ff00712f0 con 0x7f2fc806c670 2024-01-15T18:18:16.757 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:16.755+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:16.754+0000 7f2fed600640 1 -- 172.21.15.2:0/1428624489 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2fe0001210 con 0x7f2fc806c670 2024-01-15T18:18:16.758 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:16.758 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:18.774 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:19.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.001+0000 7f2e68400640 1 Processor -- start 2024-01-15T18:18:19.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.001+0000 7f2e68400640 1 -- start start 2024-01-15T18:18:19.006 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e68400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e6015b860 0x7f2e600b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e68400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e600b2ba0 0x7f2e600b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e68400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e600cfec0 con 0x7f2e60144b80 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e68400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6015f410 con 0x7f2e6015b860 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e68400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e600b3510 con 0x7f2e600b2ba0 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e65000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e6015b860 0x7f2e600b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e65000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f2e6015b860 0x7f2e600b3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57272/0 (socket says 172.21.15.2:57272) 2024-01-15T18:18:19.007 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.003+0000 7f2e65000640 1 -- 172.21.15.2:0/2100321826 learned_addr learned my addr 172.21.15.2:0/2100321826 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:19.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.004+0000 7f2e66400640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e600b2ba0 0x7f2e600b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:19.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.004+0000 7f2e65000640 1 -- 172.21.15.2:0/2100321826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e600b2ba0 msgr2=0x7f2e600b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:19.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.004+0000 7f2e65000640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e600b2ba0 0x7f2e600b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:19.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.004+0000 7f2e65000640 1 -- 172.21.15.2:0/2100321826 >> v1:172.21.15.137:6789/0 conn(0x7f2e60144b80 legacy=0x7f2e6015ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:19.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.004+0000 7f2e65000640 1 -- 172.21.15.2:0/2100321826 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6015fc30 con 0x7f2e6015b860 2024-01-15T18:18:19.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.004+0000 7f2e66400640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e600b2ba0 0x7f2e600b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:19.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.005+0000 7f2e65000640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6015b860 0x7f2e600b3f10 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009870 tx=0x7f2e4802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857215bd741c95b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:19.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.005+0000 7f2e5fe00640 1 -- 172.21.15.2:0/2100321826 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48031640 con 0x7f2e6015b860 2024-01-15T18:18:19.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.005+0000 7f2e5fe00640 1 -- 172.21.15.2:0/2100321826 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e480317a0 con 0x7f2e6015b860 2024-01-15T18:18:19.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.005+0000 7f2e5fe00640 1 -- 172.21.15.2:0/2100321826 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e48031b30 con 0x7f2e6015b860 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 -- 172.21.15.2:0/2100321826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6015b860 msgr2=0x7f2e600b3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6015b860 0x7f2e600b3f10 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f2e48009870 tx=0x7f2e4802b8f0 comp rx=0 tx=0).stop 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 -- 172.21.15.2:0/2100321826 shutdown_connections 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e600b2ba0 0x7f2e600b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 --2- 172.21.15.2:0/2100321826 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e6015b860 0x7f2e600b3f10 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 -- 172.21.15.2:0/2100321826 >> 172.21.15.2:0/2100321826 conn(0x7f2e600b6430 msgr2=0x7f2e600b6840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 -- 172.21.15.2:0/2100321826 shutdown_connections 2024-01-15T18:18:19.010 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.006+0000 7f2e68400640 1 -- 172.21.15.2:0/2100321826 wait complete. 2024-01-15T18:18:19.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.008+0000 7f2e68400640 1 Processor -- start 2024-01-15T18:18:19.011 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.008+0000 7f2e68400640 1 -- start start 2024-01-15T18:18:19.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e68400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e600b2ba0 0x7f2e601f4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e68400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e60144b80 0x7f2e60154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e68400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6015b860 0x7f2e60156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e68400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e60154670 con 0x7f2e600b2ba0 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e68400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6013e480 con 0x7f2e6015b860 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e68400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e6013e760 con 0x7f2e60144b80 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.009+0000 7f2e65a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e600b2ba0 0x7f2e601f4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e600b2ba0 0x7f2e601f4f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57280/0 (socket says 172.21.15.2:57280) 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 -- 172.21.15.2:0/4270332838 learned_addr learned my addr 172.21.15.2:0/4270332838 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e66400640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6015b860 0x7f2e60156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 -- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e60144b80 msgr2=0x7f2e60154130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2e60144b80 0x7f2e60154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 -- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6015b860 msgr2=0x7f2e60156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:19.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2e6015b860 0x7f2e60156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 -- 172.21.15.2:0/4270332838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480094d0 con 0x7f2e600b2ba0 2024-01-15T18:18:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.010+0000 7f2e65a00640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2e600b2ba0 0x7f2e601f4f10 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000c430 tx=0x7f2e5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.011+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e50011790 con 0x7f2e600b2ba0 2024-01-15T18:18:19.015 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.011+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e500118f0 con 0x7f2e600b2ba0 2024-01-15T18:18:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.011+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e50011ce0 con 0x7f2e600b2ba0 2024-01-15T18:18:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.011+0000 7f2e68400640 1 -- 172.21.15.2:0/4270332838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e6013ea40 con 0x7f2e600b2ba0 2024-01-15T18:18:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.011+0000 7f2e68400640 1 -- 172.21.15.2:0/4270332838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e60142650 con 0x7f2e600b2ba0 2024-01-15T18:18:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.012+0000 7f2e68400640 1 -- 172.21.15.2:0/4270332838 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e600686f0 con 0x7f2e600b2ba0 2024-01-15T18:18:19.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.013+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f2e5001c460 con 0x7f2e600b2ba0 2024-01-15T18:18:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.014+0000 7f2e5d600640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e3806c670 0x7f2e3806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:19.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.014+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f2e500887d0 con 0x7f2e600b2ba0 2024-01-15T18:18:19.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.015+0000 7f2e65000640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e3806c670 0x7f2e3806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:19.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.016+0000 7f2e65000640 1 --2- 172.21.15.2:0/4270332838 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2e3806c670 0x7f2e3806eb30 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f2e48036000 tx=0x7f2e48030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:19.022 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.019+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e500550f0 con 0x7f2e600b2ba0 2024-01-15T18:18:19.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.244+0000 7f2e68400640 1 -- 172.21.15.2:0/4270332838 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e6015f950 con 0x7f2e3806c670 2024-01-15T18:18:19.260 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:19.258+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:19.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:19.258+0000 7f2e5d600640 1 -- 172.21.15.2:0/4270332838 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2e48031b20 con 0x7f2e3806c670 2024-01-15T18:18:19.261 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:19.261 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:19.274 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:21.276 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:21.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.509+0000 7fd917e00640 1 Processor -- start 2024-01-15T18:18:21.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.510+0000 7fd917e00640 1 -- start start 2024-01-15T18:18:21.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd917e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd9180ae260 0x7fd918071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd917e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd9180704c0 0x7fd918070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd917e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91806c500 con 0x7fd91806da80 2024-01-15T18:18:21.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd917e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91815c4b0 con 0x7fd9180704c0 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd917e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91815b8a0 con 0x7fd9180ae260 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd916000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd91806da80 0x7fd91806df00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:38470/0 (socket says 172.21.15.2:38470) 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd917400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd9180704c0 0x7fd918070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.511+0000 7fd916000640 1 -- 172.21.15.2:0/1622222181 learned_addr learned my addr 172.21.15.2:0/1622222181 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.512+0000 7fd916a00640 1 --2- 172.21.15.2:0/1622222181 >> v2:172.21.15.137:3300/0 conn(0x7fd9180ae260 0x7fd918071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.512+0000 7fd917400640 1 -- 172.21.15.2:0/1622222181 >> v2:172.21.15.137:3300/0 conn(0x7fd9180ae260 msgr2=0x7fd918071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.512+0000 7fd917400640 1 --2- 172.21.15.2:0/1622222181 >> v2:172.21.15.137:3300/0 conn(0x7fd9180ae260 0x7fd918071ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.512+0000 7fd917400640 1 -- 172.21.15.2:0/1622222181 >> v1:172.21.15.2:6789/0 conn(0x7fd91806da80 legacy=0x7fd91806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.512+0000 7fd917400640 1 -- 172.21.15.2:0/1622222181 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91815bb80 con 0x7fd9180704c0 2024-01-15T18:18:21.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.513+0000 7fd917400640 1 --2- 172.21.15.2:0/1622222181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9180704c0 0x7fd918070950 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fd91804f900 tx=0x7fd91002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2777590b7b114b9b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd915600640 1 -- 172.21.15.2:0/1622222181 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91002cbe0 con 0x7fd9180704c0 2024-01-15T18:18:21.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd915600640 1 -- 172.21.15.2:0/1622222181 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91002cd40 con 0x7fd9180704c0 2024-01-15T18:18:21.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd915600640 1 -- 172.21.15.2:0/1622222181 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910034660 con 0x7fd9180704c0 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 -- 172.21.15.2:0/1622222181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9180704c0 msgr2=0x7fd918070950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 --2- 172.21.15.2:0/1622222181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9180704c0 0x7fd918070950 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fd91804f900 tx=0x7fd91002be30 comp rx=0 tx=0).stop 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 -- 172.21.15.2:0/1622222181 shutdown_connections 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 --2- 172.21.15.2:0/1622222181 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd9180704c0 0x7fd918070950 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 --2- 172.21.15.2:0/1622222181 >> v2:172.21.15.137:3300/0 conn(0x7fd9180ae260 0x7fd918071ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 -- 172.21.15.2:0/1622222181 >> 172.21.15.2:0/1622222181 conn(0x7fd9180cebc0 msgr2=0x7fd9180cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 -- 172.21.15.2:0/1622222181 shutdown_connections 2024-01-15T18:18:21.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.514+0000 7fd917e00640 1 -- 172.21.15.2:0/1622222181 wait complete. 2024-01-15T18:18:21.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.516+0000 7fd917e00640 1 Processor -- start 2024-01-15T18:18:21.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.516+0000 7fd917e00640 1 -- start start 2024-01-15T18:18:21.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.517+0000 7fd917e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd91806da80 0x7fd9181f2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.517+0000 7fd917e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9180704c0 0x7fd91814d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.517+0000 7fd917e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9180ae260 0x7fd9180cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.517+0000 7fd917e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181f32f0 con 0x7fd91806da80 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.517+0000 7fd917e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181f3570 con 0x7fd9180ae260 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.517+0000 7fd917e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9181f3850 con 0x7fd9180704c0 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd91806da80 0x7fd9181f2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd91806da80 0x7fd9181f2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:57308/0 (socket says 172.21.15.2:57308) 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 -- 172.21.15.2:0/3129307280 learned_addr learned my addr 172.21.15.2:0/3129307280 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 -- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9180704c0 msgr2=0x7fd91814d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 --2- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd9180704c0 0x7fd91814d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 -- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9180ae260 msgr2=0x7fd9180cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 --2- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd9180ae260 0x7fd9180cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 -- 172.21.15.2:0/3129307280 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910009a70 con 0x7fd91806da80 2024-01-15T18:18:21.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.518+0000 7fd916a00640 1 --2- 172.21.15.2:0/3129307280 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd91806da80 0x7fd9181f2db0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fd90800d800 tx=0x7fd90800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.519+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90800b6c0 con 0x7fd91806da80 2024-01-15T18:18:21.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.519+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90800b820 con 0x7fd91806da80 2024-01-15T18:18:21.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.519+0000 7fd917e00640 1 -- 172.21.15.2:0/3129307280 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9181f3b30 con 0x7fd91806da80 2024-01-15T18:18:21.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.519+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90800bbb0 con 0x7fd91806da80 2024-01-15T18:18:21.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.519+0000 7fd917e00640 1 -- 172.21.15.2:0/3129307280 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91814dc00 con 0x7fd91806da80 2024-01-15T18:18:21.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.520+0000 7fd917e00640 1 -- 172.21.15.2:0/3129307280 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9180caa10 con 0x7fd91806da80 2024-01-15T18:18:21.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.521+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fd90800bd10 con 0x7fd91806da80 2024-01-15T18:18:21.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.522+0000 7fd90ea00640 1 --2- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8f006c670 0x7fd8f006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:21.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.522+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fd9080891c0 con 0x7fd91806da80 2024-01-15T18:18:21.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.523+0000 7fd916000640 1 --2- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8f006c670 0x7fd8f006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:21.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.524+0000 7fd916000640 1 --2- 172.21.15.2:0/3129307280 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd8f006c670 0x7fd8f006eb30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fd9180b4e10 tx=0x7fd900007a40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:21.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.526+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9180caa10 con 0x7fd91806da80 2024-01-15T18:18:21.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.755+0000 7fd917e00640 1 -- 172.21.15.2:0/3129307280 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9180abd30 con 0x7fd8f006c670 2024-01-15T18:18:21.775 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:21.773+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:21.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:21.773+0000 7fd90ea00640 1 -- 172.21.15.2:0/3129307280 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd900001210 con 0x7fd8f006c670 2024-01-15T18:18:21.776 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:21.776 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:23.791 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:24.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.023+0000 7fe6ef200640 1 Processor -- start 2024-01-15T18:18:24.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.023+0000 7fe6ef200640 1 -- start start 2024-01-15T18:18:24.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.024+0000 7fe6ef200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe6e815c450 0x7fe6e806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.025+0000 7fe6ef200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e806d290 con 0x7fe6e806ce30 2024-01-15T18:18:24.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.025+0000 7fe6ef200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80ce230 con 0x7fe6e8144c00 2024-01-15T18:18:24.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.025+0000 7fe6ef200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80ce510 con 0x7fe6e815c450 2024-01-15T18:18:24.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.025+0000 7fe6ed200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe6e806ce30 0x7fe6e8070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56064/0 (socket says 172.21.15.2:56064) 2024-01-15T18:18:24.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.025+0000 7fe6ed200640 1 -- 172.21.15.2:0/1526072821 learned_addr learned my addr 172.21.15.2:0/1526072821 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:24.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.025+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/1526072821 >> v2:172.21.15.137:3300/0 conn(0x7fe6e815c450 0x7fe6e806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.029 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.026+0000 7fe6e7400640 1 -- 172.21.15.2:0/1526072821 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958015859 0 0) 0x7fe6e806d290 con 0x7fe6e806ce30 2024-01-15T18:18:24.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.026+0000 7fe6e7400640 1 -- 172.21.15.2:0/1526072821 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6c8003520 con 0x7fe6e806ce30 2024-01-15T18:18:24.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.026+0000 7fe6dfe00640 1 -- 172.21.15.2:0/1526072821 >> v1:172.21.15.137:6789/0 conn(0x7fe6e8144c00 legacy=0x7fe6e80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.026+0000 7fe6dfe00640 1 -- 172.21.15.2:0/1526072821 >> v1:172.21.15.2:6789/0 conn(0x7fe6e806ce30 legacy=0x7fe6e8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.026+0000 7fe6dfe00640 1 -- 172.21.15.2:0/1526072821 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80ce7f0 con 0x7fe6e815c450 2024-01-15T18:18:24.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.028+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/1526072821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e815c450 0x7fe6e806c8f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00097a0 tx=0x7fe6d002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f1b810b50fdfa73 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.028+0000 7fe6e7400640 1 -- 172.21.15.2:0/1526072821 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d002f050 con 0x7fe6e815c450 2024-01-15T18:18:24.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.028+0000 7fe6e7400640 1 -- 172.21.15.2:0/1526072821 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0031700 con 0x7fe6e815c450 2024-01-15T18:18:24.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.028+0000 7fe6e7400640 1 -- 172.21.15.2:0/1526072821 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0031a90 con 0x7fe6e815c450 2024-01-15T18:18:24.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 -- 172.21.15.2:0/1526072821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e815c450 msgr2=0x7fe6e806c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.032 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 --2- 172.21.15.2:0/1526072821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e815c450 0x7fe6e806c8f0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00097a0 tx=0x7fe6d002b820 comp rx=0 tx=0).stop 2024-01-15T18:18:24.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 -- 172.21.15.2:0/1526072821 shutdown_connections 2024-01-15T18:18:24.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 --2- 172.21.15.2:0/1526072821 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e815c450 0x7fe6e806c8f0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 -- 172.21.15.2:0/1526072821 >> 172.21.15.2:0/1526072821 conn(0x7fe6e80b45f0 msgr2=0x7fe6e80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:24.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 -- 172.21.15.2:0/1526072821 shutdown_connections 2024-01-15T18:18:24.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.029+0000 7fe6ef200640 1 -- 172.21.15.2:0/1526072821 wait complete. 2024-01-15T18:18:24.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.031+0000 7fe6ef200640 1 Processor -- start 2024-01-15T18:18:24.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.031+0000 7fe6ef200640 1 -- start start 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ef200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e806ce30 0x7fe6e80b1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ef200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e8144c00 0x7fe6e813ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ef200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e815c450 0x7fe6e8150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ef200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e813ea90 con 0x7fe6e815c450 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ef200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e813de80 con 0x7fe6e8144c00 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ef200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e813e160 con 0x7fe6e806ce30 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ed200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e815c450 0x7fe6e8150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ed200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e815c450 0x7fe6e8150760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44008/0 (socket says 172.21.15.2:44008) 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6ed200640 1 -- 172.21.15.2:0/3568714968 learned_addr learned my addr 172.21.15.2:0/3568714968 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.032+0000 7fe6e7e00640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e806ce30 0x7fe6e80b1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6dfe00640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e8144c00 0x7fe6e813ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ed200640 1 -- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e806ce30 msgr2=0x7fe6e80b1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ed200640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe6e806ce30 0x7fe6e80b1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ed200640 1 -- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e8144c00 msgr2=0x7fe6e813ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ed200640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe6e8144c00 0x7fe6e813ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ed200640 1 -- 172.21.15.2:0/3568714968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d0009400 con 0x7fe6e815c450 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ed200640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe6e815c450 0x7fe6e8150760 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00020c0 tx=0x7fe6e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0012400 con 0x7fe6e815c450 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e0012560 con 0x7fe6e815c450 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0012930 con 0x7fe6e815c450 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.033+0000 7fe6ef200640 1 -- 172.21.15.2:0/3568714968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e813e440 con 0x7fe6e815c450 2024-01-15T18:18:24.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.034+0000 7fe6ef200640 1 -- 172.21.15.2:0/3568714968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e813f460 con 0x7fe6e815c450 2024-01-15T18:18:24.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.035+0000 7fe6ef200640 1 -- 172.21.15.2:0/3568714968 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e813ec00 con 0x7fe6e815c450 2024-01-15T18:18:24.039 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.036+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fe6e0012a90 con 0x7fe6e815c450 2024-01-15T18:18:24.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.037+0000 7fe6e4c00640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6c006c6a0 0x7fe6c006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:24.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.037+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fe6e0089080 con 0x7fe6e815c450 2024-01-15T18:18:24.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.038+0000 7fe6e7e00640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6c006c6a0 0x7fe6c006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:24.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.039+0000 7fe6e7e00640 1 --2- 172.21.15.2:0/3568714968 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe6c006c6a0 0x7fe6c006eb60 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8000f60 tx=0x7fe6d8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:24.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.042+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6e0055940 con 0x7fe6e815c450 2024-01-15T18:18:24.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.273+0000 7fe6ef200640 1 -- 172.21.15.2:0/3568714968 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6e80cead0 con 0x7fe6c006c6a0 2024-01-15T18:18:24.291 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:24.288+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:24.291 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:24.288+0000 7fe6e4c00640 1 -- 172.21.15.2:0/3568714968 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe6d80014a0 con 0x7fe6c006c6a0 2024-01-15T18:18:24.292 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:24.292 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:26.310 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:26.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.543+0000 7f7a0e000640 1 Processor -- start 2024-01-15T18:18:26.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.543+0000 7f7a0e000640 1 -- start start 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0e000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7a080ae260 0x7f7a08071ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0e000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0806c500 con 0x7f7a080704c0 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0e000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a0815b8a0 con 0x7f7a0806da80 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0e000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a0815c4b0 con 0x7f7a080ae260 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0cc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7a080ae260 0x7f7a08071ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0d600640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7a080704c0 0x7f7a08070950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56066/0 (socket says 172.21.15.2:56066) 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.545+0000 7f7a0d600640 1 -- 172.21.15.2:0/1512619016 learned_addr learned my addr 172.21.15.2:0/1512619016 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:26.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.546+0000 7f7a0cc00640 1 -- 172.21.15.2:0/1512619016 >> v1:172.21.15.137:6789/0 conn(0x7f7a0806da80 legacy=0x7f7a0806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:26.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.546+0000 7f7a0cc00640 1 -- 172.21.15.2:0/1512619016 >> v1:172.21.15.2:6789/0 conn(0x7f7a080704c0 legacy=0x7f7a08070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:26.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.546+0000 7f7a07400640 1 -- 172.21.15.2:0/1512619016 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158664351 0 0) 0x7f7a00001110 con 0x7f7a080704c0 2024-01-15T18:18:26.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.546+0000 7f7a0cc00640 1 -- 172.21.15.2:0/1512619016 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0815bb80 con 0x7f7a080ae260 2024-01-15T18:18:26.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.547+0000 7f7a0cc00640 1 --2- 172.21.15.2:0/1512619016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a08071ce0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009d90 tx=0x7f79f802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=535b0652ac8d6772 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:26.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.548+0000 7f7a07400640 1 -- 172.21.15.2:0/1512619016 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f802cbd0 con 0x7f7a080ae260 2024-01-15T18:18:26.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.548+0000 7f7a07400640 1 -- 172.21.15.2:0/1512619016 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f802cd30 con 0x7f7a080ae260 2024-01-15T18:18:26.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.548+0000 7f7a07400640 1 -- 172.21.15.2:0/1512619016 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f802cea0 con 0x7f7a080ae260 2024-01-15T18:18:26.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.548+0000 7f7a0e000640 1 -- 172.21.15.2:0/1512619016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 msgr2=0x7f7a08071ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:26.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.548+0000 7f7a0e000640 1 --2- 172.21.15.2:0/1512619016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a08071ce0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f79f8009d90 tx=0x7f79f802be20 comp rx=0 tx=0).stop 2024-01-15T18:18:26.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.549+0000 7f7a0e000640 1 -- 172.21.15.2:0/1512619016 shutdown_connections 2024-01-15T18:18:26.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.549+0000 7f7a0e000640 1 --2- 172.21.15.2:0/1512619016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a08071ce0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:26.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.549+0000 7f7a0e000640 1 -- 172.21.15.2:0/1512619016 >> 172.21.15.2:0/1512619016 conn(0x7f7a080cebc0 msgr2=0x7f7a080cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:26.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.549+0000 7f7a0e000640 1 -- 172.21.15.2:0/1512619016 shutdown_connections 2024-01-15T18:18:26.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.549+0000 7f7a0e000640 1 -- 172.21.15.2:0/1512619016 wait complete. 2024-01-15T18:18:26.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.550+0000 7f7a0e000640 1 Processor -- start 2024-01-15T18:18:26.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.551+0000 7f7a0e000640 1 -- start start 2024-01-15T18:18:26.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.551+0000 7f7a0e000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0806da80 0x7f7a080ca660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:26.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.551+0000 7f7a0e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080704c0 0x7f7a080cc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:26.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.551+0000 7f7a0e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a080cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:26.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0e000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080b2370 con 0x7f7a080ae260 2024-01-15T18:18:26.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0e000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080cdcb0 con 0x7f7a0806da80 2024-01-15T18:18:26.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0e000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a080caba0 con 0x7f7a080704c0 2024-01-15T18:18:26.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a080cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:26.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a080cccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44038/0 (socket says 172.21.15.2:44038) 2024-01-15T18:18:26.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 -- 172.21.15.2:0/1355825789 learned_addr learned my addr 172.21.15.2:0/1355825789 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:26.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0cc00640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0806da80 0x7f7a080ca660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:26.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a07e00640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080704c0 0x7f7a080cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 -- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080704c0 msgr2=0x7f7a080cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7a080704c0 0x7f7a080cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 -- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0806da80 msgr2=0x7f7a080ca660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7a0806da80 0x7f7a080ca660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.552+0000 7f7a0d600640 1 -- 172.21.15.2:0/1355825789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f8009a70 con 0x7f7a080ae260 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.553+0000 7f7a0d600640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7a080ae260 0x7f7a080cccd0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f7a0000c7b0 tx=0x7f7a0000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.553+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a000072d0 con 0x7f7a080ae260 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.553+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a00007450 con 0x7f7a080ae260 2024-01-15T18:18:26.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.553+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00001d10 con 0x7f7a080ae260 2024-01-15T18:18:26.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.553+0000 7f7a0e000640 1 -- 172.21.15.2:0/1355825789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a080ce140 con 0x7f7a080ae260 2024-01-15T18:18:26.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.553+0000 7f7a0e000640 1 -- 172.21.15.2:0/1355825789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a080b25c0 con 0x7f7a080ae260 2024-01-15T18:18:26.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.554+0000 7f7a0e000640 1 -- 172.21.15.2:0/1355825789 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a080cde40 con 0x7f7a080ae260 2024-01-15T18:18:26.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.555+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f7a000047e0 con 0x7f7a080ae260 2024-01-15T18:18:26.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.556+0000 7f7a04c00640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79e006c670 0x7f79e006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:26.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.556+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f7a00087bc0 con 0x7f7a080ae260 2024-01-15T18:18:26.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.557+0000 7f7a0cc00640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79e006c670 0x7f79e006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:26.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.558+0000 7f7a0cc00640 1 --2- 172.21.15.2:0/1355825789 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f79e006c670 0x7f79e006eb30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f79f8036000 tx=0x7f79f8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:26.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.560+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a000540d0 con 0x7f7a080ae260 2024-01-15T18:18:26.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.792+0000 7f7a0e000640 1 -- 172.21.15.2:0/1355825789 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a080cf670 con 0x7f79e006c670 2024-01-15T18:18:26.812 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:26.810+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:26.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:26.810+0000 7f7a04c00640 1 -- 172.21.15.2:0/1355825789 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7a080cf670 con 0x7f79e006c670 2024-01-15T18:18:26.813 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:26.813 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:28.825 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:29.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.064+0000 7fd0d2a00640 1 Processor -- start 2024-01-15T18:18:29.068 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.065+0000 7fd0d2a00640 1 -- start start 2024-01-15T18:18:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.066+0000 7fd0d2a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd0cc06d000 0x7fd0cc0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.066+0000 7fd0d2a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd0cc0b2430 0x7fd0cc0b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.066+0000 7fd0d2a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0cc15dbc0 con 0x7fd0cc15ae80 2024-01-15T18:18:29.070 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.066+0000 7fd0d2a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc15e840 con 0x7fd0cc06d000 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.066+0000 7fd0d2a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0cc15dea0 con 0x7fd0cc0b2430 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.067+0000 7fd0c3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd0cc06d000 0x7fd0cc0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.067+0000 7fd0c3e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd0cc06d000 0x7fd0cc0acd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44052/0 (socket says 172.21.15.2:44052) 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.067+0000 7fd0c3e00640 1 -- 172.21.15.2:0/1422977285 learned_addr learned my addr 172.21.15.2:0/1422977285 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.067+0000 7fd0d0a00640 1 --2- 172.21.15.2:0/1422977285 >> v2:172.21.15.137:3300/0 conn(0x7fd0cc0b2430 0x7fd0cc0b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.068+0000 7fd0c3e00640 1 -- 172.21.15.2:0/1422977285 >> v2:172.21.15.137:3300/0 conn(0x7fd0cc0b2430 msgr2=0x7fd0cc0b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.068+0000 7fd0c3e00640 1 --2- 172.21.15.2:0/1422977285 >> v2:172.21.15.137:3300/0 conn(0x7fd0cc0b2430 0x7fd0cc0b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.071 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.068+0000 7fd0c3e00640 1 -- 172.21.15.2:0/1422977285 >> v1:172.21.15.137:6789/0 conn(0x7fd0cc15ae80 legacy=0x7fd0cc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.068+0000 7fd0c3e00640 1 -- 172.21.15.2:0/1422977285 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc15e180 con 0x7fd0cc06d000 2024-01-15T18:18:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.069+0000 7fd0c3e00640 1 --2- 172.21.15.2:0/1422977285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0acd00 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0cf9a0 tx=0x7fd0b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537f5334ca638891 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.072 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.069+0000 7fd0cb400640 1 -- 172.21.15.2:0/1422977285 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4031870 con 0x7fd0cc06d000 2024-01-15T18:18:29.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.069+0000 7fd0cb400640 1 -- 172.21.15.2:0/1422977285 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b40319d0 con 0x7fd0cc06d000 2024-01-15T18:18:29.073 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.069+0000 7fd0cb400640 1 -- 172.21.15.2:0/1422977285 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b4031d60 con 0x7fd0cc06d000 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 -- 172.21.15.2:0/1422977285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 msgr2=0x7fd0cc0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 --2- 172.21.15.2:0/1422977285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0acd00 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7fd0cc0cf9a0 tx=0x7fd0b402b910 comp rx=0 tx=0).stop 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 -- 172.21.15.2:0/1422977285 shutdown_connections 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 --2- 172.21.15.2:0/1422977285 >> v2:172.21.15.137:3300/0 conn(0x7fd0cc0b2430 0x7fd0cc0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 --2- 172.21.15.2:0/1422977285 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0acd00 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 -- 172.21.15.2:0/1422977285 >> 172.21.15.2:0/1422977285 conn(0x7fd0cc0b45f0 msgr2=0x7fd0cc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 -- 172.21.15.2:0/1422977285 shutdown_connections 2024-01-15T18:18:29.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.070+0000 7fd0d2a00640 1 -- 172.21.15.2:0/1422977285 wait complete. 2024-01-15T18:18:29.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.072+0000 7fd0d2a00640 1 Processor -- start 2024-01-15T18:18:29.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.072+0000 7fd0d2a00640 1 -- start start 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0d2a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0d2a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0cc0b2430 0x7fd0cc15fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0d2a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc1601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0d2a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc1413a0 con 0x7fd0cc06d000 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0d2a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc144410 con 0x7fd0cc0b2430 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0d2a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc1f66e0 con 0x7fd0cc15ae80 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0cbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0cbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0b16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44056/0 (socket says 172.21.15.2:44056) 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.073+0000 7fd0cbe00640 1 -- 172.21.15.2:0/2904655043 learned_addr learned my addr 172.21.15.2:0/2904655043 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0c3e00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0cc0b2430 0x7fd0cc15fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.078 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0cbe00640 1 -- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0cc15ae80 msgr2=0x7fd0cc1601d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0cbe00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd0cc15ae80 0x7fd0cc1601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0cbe00640 1 -- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0cc0b2430 msgr2=0x7fd0cc15fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0cbe00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd0cc0b2430 0x7fd0cc15fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0cbe00640 1 -- 172.21.15.2:0/2904655043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b40094d0 con 0x7fd0cc06d000 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0cbe00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd0cc06d000 0x7fd0cc0b16c0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00b3f0 tx=0x7fd0bc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc011830 con 0x7fd0cc06d000 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc011990 con 0x7fd0cc06d000 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0d2a00640 1 -- 172.21.15.2:0/2904655043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc1f69c0 con 0x7fd0cc06d000 2024-01-15T18:18:29.079 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.074+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc011d40 con 0x7fd0cc06d000 2024-01-15T18:18:29.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.075+0000 7fd0d2a00640 1 -- 172.21.15.2:0/2904655043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc1f71c0 con 0x7fd0cc06d000 2024-01-15T18:18:29.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.075+0000 7fd0d2a00640 1 -- 172.21.15.2:0/2904655043 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc1f6ee0 con 0x7fd0cc06d000 2024-01-15T18:18:29.080 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.077+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fd0bc01a400 con 0x7fd0cc06d000 2024-01-15T18:18:29.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.078+0000 7fd0c8c00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd0a406c670 0x7fd0a406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:29.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.078+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fd0bc0886b0 con 0x7fd0cc06d000 2024-01-15T18:18:29.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.079+0000 7fd0c3e00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd0a406c670 0x7fd0a406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:29.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.080+0000 7fd0c3e00640 1 --2- 172.21.15.2:0/2904655043 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd0a406c670 0x7fd0a406eb30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fd0b4036000 tx=0x7fd0b403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:29.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.083+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0cc1f6ee0 con 0x7fd0cc06d000 2024-01-15T18:18:29.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.310+0000 7fd0d2a00640 1 -- 172.21.15.2:0/2904655043 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0cc1f6560 con 0x7fd0a406c670 2024-01-15T18:18:29.331 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:29.328+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:29.328+0000 7fd0c8c00640 1 -- 172.21.15.2:0/2904655043 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd0b4031d50 con 0x7fd0a406c670 2024-01-15T18:18:29.331 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:29.331 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:29.345 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:31.346 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:31.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.583+0000 7fd6dac00640 1 Processor -- start 2024-01-15T18:18:31.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.583+0000 7fd6dac00640 1 -- start start 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.585+0000 7fd6dac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd6d4144c00 0x7fd6d40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.585+0000 7fd6dac00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40ce230 con 0x7fd6d40cdd70 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.585+0000 7fd6dac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6d40ce510 con 0x7fd6d406d000 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.585+0000 7fd6dac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6d40ce7f0 con 0x7fd6d4144c00 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.586+0000 7fd6cb200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd6d40cdd70 0x7fd6d406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:56070/0 (socket says 172.21.15.2:56070) 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.586+0000 7fd6cb200640 1 -- 172.21.15.2:0/3431949223 learned_addr learned my addr 172.21.15.2:0/3431949223 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:31.590 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.586+0000 7fd6d3e00640 1 --2- 172.21.15.2:0/3431949223 >> v2:172.21.15.137:3300/0 conn(0x7fd6d4144c00 0x7fd6d40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.586+0000 7fd6d3400640 1 -- 172.21.15.2:0/3431949223 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921451796 0 0) 0x7fd6d40ce230 con 0x7fd6d40cdd70 2024-01-15T18:18:31.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.586+0000 7fd6d3400640 1 -- 172.21.15.2:0/3431949223 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b4003520 con 0x7fd6d40cdd70 2024-01-15T18:18:31.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.587+0000 7fd6d3e00640 1 -- 172.21.15.2:0/3431949223 >> v1:172.21.15.137:6789/0 conn(0x7fd6d406d000 legacy=0x7fd6d4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.587+0000 7fd6d3e00640 1 -- 172.21.15.2:0/3431949223 >> v1:172.21.15.2:6789/0 conn(0x7fd6d40cdd70 legacy=0x7fd6d406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.587+0000 7fd6d3400640 1 -- 172.21.15.2:0/3431949223 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 665444110 0 0) 0x7fd6b4003520 con 0x7fd6d40cdd70 2024-01-15T18:18:31.591 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.587+0000 7fd6d3e00640 1 -- 172.21.15.2:0/3431949223 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d40705f0 con 0x7fd6d4144c00 2024-01-15T18:18:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.588+0000 7fd6d3e00640 1 --2- 172.21.15.2:0/3431949223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d40cd830 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009d90 tx=0x7fd6c402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e9cca16d4f6a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6d3400640 1 -- 172.21.15.2:0/3431949223 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c402ccf0 con 0x7fd6d4144c00 2024-01-15T18:18:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6d3400640 1 -- 172.21.15.2:0/3431949223 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c402ce50 con 0x7fd6d4144c00 2024-01-15T18:18:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6d3400640 1 -- 172.21.15.2:0/3431949223 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c4034660 con 0x7fd6d4144c00 2024-01-15T18:18:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 -- 172.21.15.2:0/3431949223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 msgr2=0x7fd6d40cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 --2- 172.21.15.2:0/3431949223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d40cd830 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4009d90 tx=0x7fd6c402bf40 comp rx=0 tx=0).stop 2024-01-15T18:18:31.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 -- 172.21.15.2:0/3431949223 shutdown_connections 2024-01-15T18:18:31.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 --2- 172.21.15.2:0/3431949223 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d40cd830 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 -- 172.21.15.2:0/3431949223 >> 172.21.15.2:0/3431949223 conn(0x7fd6d40b45f0 msgr2=0x7fd6d40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:31.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 -- 172.21.15.2:0/3431949223 shutdown_connections 2024-01-15T18:18:31.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.589+0000 7fd6dac00640 1 -- 172.21.15.2:0/3431949223 wait complete. 2024-01-15T18:18:31.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.591+0000 7fd6dac00640 1 Processor -- start 2024-01-15T18:18:31.595 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.591+0000 7fd6dac00640 1 -- start start 2024-01-15T18:18:31.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.592+0000 7fd6dac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d406d000 0x7fd6d40b0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.592+0000 7fd6dac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6d40cdd70 0x7fd6d4138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.592+0000 7fd6dac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d413b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.592+0000 7fd6dac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d4137370 con 0x7fd6d406d000 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.592+0000 7fd6dac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d40b1520 con 0x7fd6d4144c00 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.592+0000 7fd6dac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6d41388c0 con 0x7fd6d40cdd70 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d406d000 0x7fd6d40b0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d406d000 0x7fd6d40b0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:44064/0 (socket says 172.21.15.2:44064) 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 -- 172.21.15.2:0/2341294908 learned_addr learned my addr 172.21.15.2:0/2341294908 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:31.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6cb200640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6d40cdd70 0x7fd6d4138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d8c00640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d413b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 -- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6d40cdd70 msgr2=0x7fd6d4138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd6d40cdd70 0x7fd6d4138380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 -- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 msgr2=0x7fd6d413b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d413b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d3e00640 1 -- 172.21.15.2:0/2341294908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c4009a70 con 0x7fd6d406d000 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.593+0000 7fd6d8c00640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd6d4144c00 0x7fd6d413b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.594+0000 7fd6d3e00640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd6d406d000 0x7fd6d40b0fe0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd6c4037040 tx=0x7fd6c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.594+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c403d600 con 0x7fd6d406d000 2024-01-15T18:18:31.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.594+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c403d7a0 con 0x7fd6d406d000 2024-01-15T18:18:31.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.594+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c403db50 con 0x7fd6d406d000 2024-01-15T18:18:31.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.594+0000 7fd6dac00640 1 -- 172.21.15.2:0/2341294908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6d41375c0 con 0x7fd6d406d000 2024-01-15T18:18:31.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.594+0000 7fd6dac00640 1 -- 172.21.15.2:0/2341294908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6d4137b80 con 0x7fd6d406d000 2024-01-15T18:18:31.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.595+0000 7fd6dac00640 1 -- 172.21.15.2:0/2341294908 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6d40686f0 con 0x7fd6d406d000 2024-01-15T18:18:31.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.596+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fd6c403b040 con 0x7fd6d406d000 2024-01-15T18:18:31.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.597+0000 7fd6d0c00640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6ac06c6c0 0x7fd6ac06eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:31.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.597+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fd6c40ab000 con 0x7fd6d406d000 2024-01-15T18:18:31.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.598+0000 7fd6cb200640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6ac06c6c0 0x7fd6ac06eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:31.602 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.599+0000 7fd6cb200640 1 --2- 172.21.15.2:0/2341294908 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd6ac06c6c0 0x7fd6ac06eb80 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc005880 tx=0x7fd6bc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:31.605 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.602+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6c40775a0 con 0x7fd6d406d000 2024-01-15T18:18:31.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.828+0000 7fd6dac00640 1 -- 172.21.15.2:0/2341294908 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd6d415ae40 con 0x7fd6ac06c6c0 2024-01-15T18:18:31.846 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:31.843+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:31.844+0000 7fd6d0c00640 1 -- 172.21.15.2:0/2341294908 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd6d415ae40 con 0x7fd6ac06c6c0 2024-01-15T18:18:31.846 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:31.847 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:31.861 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:33.863 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:34.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.093+0000 7f1c77200640 1 Processor -- start 2024-01-15T18:18:34.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.094+0000 7f1c77200640 1 -- start start 2024-01-15T18:18:34.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.095+0000 7f1c77200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1c7806da80 0x7f1c7806df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.095+0000 7f1c77200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1c780704c0 0x7f1c78070950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.095+0000 7f1c77200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7806c500 con 0x7f1c780ae260 2024-01-15T18:18:34.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.095+0000 7f1c77200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7815c4b0 con 0x7f1c7806da80 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.095+0000 7f1c77200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c780723f0 con 0x7f1c780704c0 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.095+0000 7f1c75400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1c7806da80 0x7f1c7806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c75400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1c7806da80 0x7f1c7806df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54000/0 (socket says 172.21.15.2:54000) 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c75400640 1 -- 172.21.15.2:0/3320731846 learned_addr learned my addr 172.21.15.2:0/3320731846 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c76800640 1 --2- 172.21.15.2:0/3320731846 >> v2:172.21.15.137:3300/0 conn(0x7f1c780704c0 0x7f1c78070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c75400640 1 -- 172.21.15.2:0/3320731846 >> v2:172.21.15.137:3300/0 conn(0x7f1c780704c0 msgr2=0x7f1c78070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c75400640 1 --2- 172.21.15.2:0/3320731846 >> v2:172.21.15.137:3300/0 conn(0x7f1c780704c0 0x7f1c78070950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c75400640 1 -- 172.21.15.2:0/3320731846 >> v1:172.21.15.137:6789/0 conn(0x7f1c780ae260 legacy=0x7f1c78071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:34.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.096+0000 7f1c75400640 1 -- 172.21.15.2:0/3320731846 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7815b8a0 con 0x7f1c7806da80 2024-01-15T18:18:34.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.097+0000 7f1c75400640 1 --2- 172.21.15.2:0/3320731846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c7806da80 0x7f1c7806df00 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f1c780ad710 tx=0x7f1c6002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e705d56743e723e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:34.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c74a00640 1 -- 172.21.15.2:0/3320731846 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6002c9f0 con 0x7f1c7806da80 2024-01-15T18:18:34.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c74a00640 1 -- 172.21.15.2:0/3320731846 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6002cb50 con 0x7f1c7806da80 2024-01-15T18:18:34.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c74a00640 1 -- 172.21.15.2:0/3320731846 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c600343f0 con 0x7f1c7806da80 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c77200640 1 -- 172.21.15.2:0/3320731846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c7806da80 msgr2=0x7f1c7806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c77200640 1 --2- 172.21.15.2:0/3320731846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c7806da80 0x7f1c7806df00 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f1c780ad710 tx=0x7f1c6002be50 comp rx=0 tx=0).stop 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c77200640 1 -- 172.21.15.2:0/3320731846 shutdown_connections 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c77200640 1 --2- 172.21.15.2:0/3320731846 >> v2:172.21.15.137:3300/0 conn(0x7f1c780704c0 0x7f1c78070950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c77200640 1 --2- 172.21.15.2:0/3320731846 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c7806da80 0x7f1c7806df00 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.098+0000 7f1c77200640 1 -- 172.21.15.2:0/3320731846 >> 172.21.15.2:0/3320731846 conn(0x7f1c780cebc0 msgr2=0x7f1c780cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.099+0000 7f1c77200640 1 -- 172.21.15.2:0/3320731846 shutdown_connections 2024-01-15T18:18:34.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.099+0000 7f1c77200640 1 -- 172.21.15.2:0/3320731846 wait complete. 2024-01-15T18:18:34.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.101+0000 7f1c77200640 1 Processor -- start 2024-01-15T18:18:34.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.101+0000 7f1c77200640 1 -- start start 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c77200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c7806da80 0x7f1c781f3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c77200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c780704c0 0x7f1c780b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c77200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c780ae260 0x7f1c7814c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c77200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c781f3a70 con 0x7f1c780704c0 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c77200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7815dba0 con 0x7f1c7806da80 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c77200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7815c5f0 con 0x7f1c780ae260 2024-01-15T18:18:34.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c75400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c780704c0 0x7f1c780b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c75400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c780704c0 0x7f1c780b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54002/0 (socket says 172.21.15.2:54002) 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.102+0000 7f1c75400640 1 -- 172.21.15.2:0/225825516 learned_addr learned my addr 172.21.15.2:0/225825516 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c75400640 1 -- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c780ae260 msgr2=0x7f1c7814c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c75400640 1 --2- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1c780ae260 0x7f1c7814c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c75400640 1 -- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c7806da80 msgr2=0x7f1c781f3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c75400640 1 --2- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1c7806da80 0x7f1c781f3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:34.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c75400640 1 -- 172.21.15.2:0/225825516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c60009b30 con 0x7f1c780704c0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c75400640 1 --2- 172.21.15.2:0/225825516 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1c780704c0 0x7f1c780b33d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f1c60036040 tx=0x7f1c60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.103+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6003e450 con 0x7f1c780704c0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.104+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6003e5f0 con 0x7f1c780704c0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.104+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6003e9a0 con 0x7f1c780704c0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.104+0000 7f1c77200640 1 -- 172.21.15.2:0/225825516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c780b3910 con 0x7f1c780704c0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.104+0000 7f1c77200640 1 -- 172.21.15.2:0/225825516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c780b2190 con 0x7f1c780704c0 2024-01-15T18:18:34.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.105+0000 7f1c77200640 1 -- 172.21.15.2:0/225825516 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c78074770 con 0x7f1c780704c0 2024-01-15T18:18:34.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.106+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f1c6003eb00 con 0x7f1c780704c0 2024-01-15T18:18:34.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.107+0000 7f1c6e000640 1 --2- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5006c6a0 0x7f1c5006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:34.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.107+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f1c600ab7a0 con 0x7f1c780704c0 2024-01-15T18:18:34.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.108+0000 7f1c75e00640 1 --2- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5006c6a0 0x7f1c5006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:34.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.109+0000 7f1c75e00640 1 --2- 172.21.15.2:0/225825516 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1c5006c6a0 0x7f1c5006eb60 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f1c7809cfe0 tx=0x7f1c68007990 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:34.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.113+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c60077cb0 con 0x7f1c780704c0 2024-01-15T18:18:34.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.341+0000 7f1c77200640 1 -- 172.21.15.2:0/225825516 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c7815c140 con 0x7f1c5006c6a0 2024-01-15T18:18:34.357 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:34.354+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:34.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:34.355+0000 7f1c6e000640 1 -- 172.21.15.2:0/225825516 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1c680011f0 con 0x7f1c5006c6a0 2024-01-15T18:18:34.358 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:34.358 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:34.626 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:34.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:18:36.375 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.612+0000 7fdd63e00640 1 Processor -- start 2024-01-15T18:18:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.613+0000 7fdd63e00640 1 -- start start 2024-01-15T18:18:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.614+0000 7fdd63e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fdd641609d0 0x7fdd640cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640b3fa0 0x7fdd640d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd63e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd640d0af0 con 0x7fdd640d1d40 2024-01-15T18:18:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd63e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd640ce7b0 con 0x7fdd641609d0 2024-01-15T18:18:36.617 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd63e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cdbe0 con 0x7fdd640b3fa0 2024-01-15T18:18:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd62000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fdd640d1d40 0x7fdd640cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:59650/0 (socket says 172.21.15.2:59650) 2024-01-15T18:18:36.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd62000640 1 -- 172.21.15.2:0/2894019426 learned_addr learned my addr 172.21.15.2:0/2894019426 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd62a00640 1 --2- 172.21.15.2:0/2894019426 >> v2:172.21.15.137:3300/0 conn(0x7fdd641609d0 0x7fdd640cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd63400640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640b3fa0 0x7fdd640d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.619 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260128175 0 0) 0x7fdd640d0af0 con 0x7fdd640d1d40 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.615+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd44003520 con 0x7fdd640d1d40 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2230858925 0 0) 0x7fdd44003520 con 0x7fdd640d1d40 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640b3fa0 msgr2=0x7fdd640d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640b3fa0 0x7fdd640d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 >> v2:172.21.15.137:3300/0 conn(0x7fdd641609d0 msgr2=0x7fdd640cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 --2- 172.21.15.2:0/2894019426 >> v2:172.21.15.137:3300/0 conn(0x7fdd641609d0 0x7fdd640cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd640cde80 con 0x7fdd640d1d40 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fdd4c0019f0 con 0x7fdd640d1d40 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 >> v1:172.21.15.2:6789/0 conn(0x7fdd640d1d40 legacy=0x7fdd640cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd44003970 0x7fdd44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd440047c0 0x7fdd44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd44005180 0x7fdd4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640ce7b0 con 0x7fdd44003970 2024-01-15T18:18:36.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640cdbe0 con 0x7fdd440047c0 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4400bef0 con 0x7fdd44005180 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd63400640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640b3fa0 0x7fdd640d01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.616+0000 7fdd62a00640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd44003970 0x7fdd44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62000640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd440047c0 0x7fdd44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62a00640 1 -- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd44005180 msgr2=0x7fdd4400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62a00640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd44005180 0x7fdd4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62a00640 1 -- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd440047c0 msgr2=0x7fdd44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62a00640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd440047c0 0x7fdd44004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62a00640 1 -- 172.21.15.2:0/2894019426 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd44003520 con 0x7fdd44003970 2024-01-15T18:18:36.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.617+0000 7fdd62000640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd440047c0 0x7fdd44004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:36.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.618+0000 7fdd62a00640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd44003970 0x7fdd44003d90 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009ac0 tx=0x7fdd440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d69daef0bad6fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.618+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5402c6d0 con 0x7fdd44003970 2024-01-15T18:18:36.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.618+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5402c830 con 0x7fdd44003970 2024-01-15T18:18:36.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.618+0000 7fdd61600640 1 -- 172.21.15.2:0/2894019426 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5402cbc0 con 0x7fdd44003970 2024-01-15T18:18:36.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63e00640 1 -- 172.21.15.2:0/2894019426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd44003970 msgr2=0x7fdd44003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63e00640 1 --2- 172.21.15.2:0/2894019426 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd44003970 0x7fdd44003d90 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009ac0 tx=0x7fdd440057f0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63400640 1 -- 172.21.15.2:0/2894019426 reap_dead start 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63e00640 1 -- 172.21.15.2:0/2894019426 shutdown_connections 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63e00640 1 -- 172.21.15.2:0/2894019426 >> 172.21.15.2:0/2894019426 conn(0x7fdd6409c3e0 msgr2=0x7fdd640d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63e00640 1 -- 172.21.15.2:0/2894019426 shutdown_connections 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.619+0000 7fdd63e00640 1 -- 172.21.15.2:0/2894019426 wait complete. 2024-01-15T18:18:36.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.621+0000 7fdd63e00640 1 Processor -- start 2024-01-15T18:18:36.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.621+0000 7fdd63e00640 1 -- start start 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.622+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd44003970 0x7fdd640ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.622+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640cabb0 0x7fdd6415a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.622+0000 7fdd63e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd6415e680 0x7fdd6415eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.622+0000 7fdd63e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6415f040 con 0x7fdd6415e680 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.622+0000 7fdd63e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd641f3d40 con 0x7fdd44003970 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.622+0000 7fdd63e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd641f38b0 con 0x7fdd640cabb0 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd6415e680 0x7fdd6415eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd6415e680 0x7fdd6415eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54024/0 (socket says 172.21.15.2:54024) 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 -- 172.21.15.2:0/1094693914 learned_addr learned my addr 172.21.15.2:0/1094693914 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd62a00640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd44003970 0x7fdd640ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd62000640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640cabb0 0x7fdd6415a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 -- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640cabb0 msgr2=0x7fdd6415a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fdd640cabb0 0x7fdd6415a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 -- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd44003970 msgr2=0x7fdd640ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fdd44003970 0x7fdd640ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.623+0000 7fdd63400640 1 -- 172.21.15.2:0/1094693914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd640ca6e0 con 0x7fdd6415e680 2024-01-15T18:18:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.624+0000 7fdd63400640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fdd6415e680 0x7fdd6415eb00 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c006fd0 tx=0x7fdd5c00e810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.624+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c00c070 con 0x7fdd6415e680 2024-01-15T18:18:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.624+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd5c004230 con 0x7fdd6415e680 2024-01-15T18:18:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.624+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5c003610 con 0x7fdd6415e680 2024-01-15T18:18:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.624+0000 7fdd63e00640 1 -- 172.21.15.2:0/1094693914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd641f2ce0 con 0x7fdd6415e680 2024-01-15T18:18:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.624+0000 7fdd63e00640 1 -- 172.21.15.2:0/1094693914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd641f2f80 con 0x7fdd6415e680 2024-01-15T18:18:36.628 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.625+0000 7fdd63e00640 1 -- 172.21.15.2:0/1094693914 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6406ccd0 con 0x7fdd6415e680 2024-01-15T18:18:36.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.626+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fdd5c003770 con 0x7fdd6415e680 2024-01-15T18:18:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.627+0000 7fdd5aa00640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd3c06c6a0 0x7fdd3c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.628+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fdd5c087a00 con 0x7fdd6415e680 2024-01-15T18:18:36.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.628+0000 7fdd62a00640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd3c06c6a0 0x7fdd3c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:36.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.629+0000 7fdd62a00640 1 --2- 172.21.15.2:0/1094693914 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fdd3c06c6a0 0x7fdd3c06eb60 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fdd54036000 tx=0x7fdd54030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:36.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.632+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd5c053f10 con 0x7fdd6415e680 2024-01-15T18:18:36.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.859+0000 7fdd63e00640 1 -- 172.21.15.2:0/1094693914 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd64160db0 con 0x7fdd3c06c6a0 2024-01-15T18:18:36.874 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:36.872+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:36.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:36.872+0000 7fdd5aa00640 1 -- 172.21.15.2:0/1094693914 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdd54004040 con 0x7fdd3c06c6a0 2024-01-15T18:18:36.875 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:36.875 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:38.892 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:39.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.128+0000 7fb704800640 1 Processor -- start 2024-01-15T18:18:39.131 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.128+0000 7fb704800640 1 -- start start 2024-01-15T18:18:39.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.130+0000 7fb704800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb6fc15c450 0x7fb6fc06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.130+0000 7fb704800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6fc06d290 con 0x7fb6fc06ce30 2024-01-15T18:18:39.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.130+0000 7fb704800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6fc0ce230 con 0x7fb6fc144c00 2024-01-15T18:18:39.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.130+0000 7fb704800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6fc0ce510 con 0x7fb6fc15c450 2024-01-15T18:18:39.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.130+0000 7fb702800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb6fc06ce30 0x7fb6fc070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:59658/0 (socket says 172.21.15.2:59658) 2024-01-15T18:18:39.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.130+0000 7fb702800640 1 -- 172.21.15.2:0/66534170 learned_addr learned my addr 172.21.15.2:0/66534170 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb701400640 1 --2- 172.21.15.2:0/66534170 >> v2:172.21.15.137:3300/0 conn(0x7fb6fc15c450 0x7fb6fc06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308537657 0 0) 0x7fb6fc06d290 con 0x7fb6fc06ce30 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6dc003520 con 0x7fb6fc06ce30 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 494198249 0 0) 0x7fb6dc003520 con 0x7fb6fc06ce30 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 >> v2:172.21.15.137:3300/0 conn(0x7fb6fc15c450 msgr2=0x7fb6fc06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 --2- 172.21.15.2:0/66534170 >> v2:172.21.15.137:3300/0 conn(0x7fb6fc15c450 0x7fb6fc06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 >> v1:172.21.15.137:6789/0 conn(0x7fb6fc144c00 legacy=0x7fb6fc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc0ce7f0 con 0x7fb6fc06ce30 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fb6f40019f0 con 0x7fb6fc06ce30 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 >> v1:172.21.15.2:6789/0 conn(0x7fb6fc06ce30 legacy=0x7fb6fc070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc003970 0x7fb6dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6dc0047c0 0x7fb6dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6dc005180 0x7fb6dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0ce510 con 0x7fb6dc0047c0 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.131+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc00c020 con 0x7fb6dc005180 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc00c1e0 con 0x7fb6dc003970 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb701400640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6dc0047c0 0x7fb6dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb701400640 1 -- 172.21.15.2:0/66534170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc003970 msgr2=0x7fb6dc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb701400640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc003970 0x7fb6dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb701400640 1 -- 172.21.15.2:0/66534170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6dc005180 msgr2=0x7fb6dc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb701400640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6dc005180 0x7fb6dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb701400640 1 -- 172.21.15.2:0/66534170 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc003520 con 0x7fb6dc0047c0 2024-01-15T18:18:39.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.132+0000 7fb702800640 1 -- 172.21.15.2:0/66534170 reap_dead start 2024-01-15T18:18:39.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.133+0000 7fb701400640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6dc0047c0 0x7fb6dc004c40 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009ac0 tx=0x7fb6dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83f5f5bd01db1289 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.133+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e402c710 con 0x7fb6dc0047c0 2024-01-15T18:18:39.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e402c870 con 0x7fb6dc0047c0 2024-01-15T18:18:39.137 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb700a00640 1 -- 172.21.15.2:0/66534170 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e402cc00 con 0x7fb6dc0047c0 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 -- 172.21.15.2:0/66534170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6dc0047c0 msgr2=0x7fb6dc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6dc0047c0 0x7fb6dc004c40 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fb6e4009ac0 tx=0x7fb6dc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 -- 172.21.15.2:0/66534170 shutdown_connections 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 --2- 172.21.15.2:0/66534170 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6dc0047c0 0x7fb6dc004c40 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 -- 172.21.15.2:0/66534170 >> 172.21.15.2:0/66534170 conn(0x7fb6fc0b45f0 msgr2=0x7fb6fc0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 -- 172.21.15.2:0/66534170 shutdown_connections 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.134+0000 7fb704800640 1 -- 172.21.15.2:0/66534170 wait complete. 2024-01-15T18:18:39.138 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.136+0000 7fb704800640 1 Processor -- start 2024-01-15T18:18:39.139 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.136+0000 7fb704800640 1 -- start start 2024-01-15T18:18:39.140 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.137+0000 7fb704800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc0047c0 0x7fb6fc13b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.137+0000 7fb704800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6fc13dec0 0x7fb6fc1f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.137+0000 7fb704800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6fc1f3440 0x7fb6fc1f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.137+0000 7fb704800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc0b1ff0 con 0x7fb6fc1f3440 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.137+0000 7fb704800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1f3e00 con 0x7fb6fc13dec0 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.137+0000 7fb704800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc13e410 con 0x7fb6dc0047c0 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6fc1f3440 0x7fb6fc1f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6fc1f3440 0x7fb6fc1f38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54046/0 (socket says 172.21.15.2:54046) 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 -- 172.21.15.2:0/602531176 learned_addr learned my addr 172.21.15.2:0/602531176 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb701e00640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc0047c0 0x7fb6fc13b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb701400640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6fc13dec0 0x7fb6fc1f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 -- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc0047c0 msgr2=0x7fb6fc13b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb6dc0047c0 0x7fb6fc13b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 -- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6fc13dec0 msgr2=0x7fb6fc1f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb6fc13dec0 0x7fb6fc1f2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:39.141 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.138+0000 7fb702800640 1 -- 172.21.15.2:0/602531176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc13e6f0 con 0x7fb6fc1f3440 2024-01-15T18:18:39.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.139+0000 7fb702800640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb6fc1f3440 0x7fb6fc1f38c0 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4001560 tx=0x7fb6f400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.139+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400cea0 con 0x7fb6fc1f3440 2024-01-15T18:18:39.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.139+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f400a610 con 0x7fb6fc1f3440 2024-01-15T18:18:39.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.139+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400a9a0 con 0x7fb6fc1f3440 2024-01-15T18:18:39.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.139+0000 7fb704800640 1 -- 172.21.15.2:0/602531176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6fc13e9d0 con 0x7fb6fc1f3440 2024-01-15T18:18:39.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.139+0000 7fb704800640 1 -- 172.21.15.2:0/602531176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc0b0cb0 con 0x7fb6fc1f3440 2024-01-15T18:18:39.143 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.140+0000 7fb704800640 1 -- 172.21.15.2:0/602531176 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc0b2440 con 0x7fb6fc1f3440 2024-01-15T18:18:39.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.142+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fb6f400ab00 con 0x7fb6fc1f3440 2024-01-15T18:18:39.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.143+0000 7fb6fa000640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6d406c6a0 0x7fb6d406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:39.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.143+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fb6f4088460 con 0x7fb6fc1f3440 2024-01-15T18:18:39.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.143+0000 7fb701e00640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6d406c6a0 0x7fb6d406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:39.146 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.144+0000 7fb701e00640 1 --2- 172.21.15.2:0/602531176 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb6d406c6a0 0x7fb6d406eb60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc13d1a0 tx=0x7fb6ec009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:39.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.147+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6fc0b2440 con 0x7fb6fc1f3440 2024-01-15T18:18:39.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.373+0000 7fb704800640 1 -- 172.21.15.2:0/602531176 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6fc0cead0 con 0x7fb6d406c6a0 2024-01-15T18:18:39.389 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:39.387+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:39.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:39.387+0000 7fb6fa000640 1 -- 172.21.15.2:0/602531176 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb6ec0012b0 con 0x7fb6d406c6a0 2024-01-15T18:18:39.390 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:39.390 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:39.403 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:41.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.641+0000 7fc21c400640 1 Processor -- start 2024-01-15T18:18:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.642+0000 7fc21c400640 1 -- start start 2024-01-15T18:18:41.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.643+0000 7fc21c400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fc2140702c0 0x7fc2140b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:41.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.643+0000 7fc21c400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc21406c8f0 0x7fc21406ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:41.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.643+0000 7fc21c400640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21415dd70 con 0x7fc2141439f0 2024-01-15T18:18:41.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.643+0000 7fc21c400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21406d230 con 0x7fc21406c8f0 2024-01-15T18:18:41.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.643+0000 7fc21c400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2140711b0 con 0x7fc2140702c0 2024-01-15T18:18:41.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc21406c8f0 0x7fc21406ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc21406c8f0 0x7fc21406ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54062/0 (socket says 172.21.15.2:54062) 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 -- 172.21.15.2:0/473183992 learned_addr learned my addr 172.21.15.2:0/473183992 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc219000640 1 --2- 172.21.15.2:0/473183992 >> v2:172.21.15.137:3300/0 conn(0x7fc2140702c0 0x7fc2140b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 -- 172.21.15.2:0/473183992 >> v2:172.21.15.137:3300/0 conn(0x7fc2140702c0 msgr2=0x7fc2140b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 --2- 172.21.15.2:0/473183992 >> v2:172.21.15.137:3300/0 conn(0x7fc2140702c0 0x7fc2140b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 -- 172.21.15.2:0/473183992 >> v1:172.21.15.137:6789/0 conn(0x7fc2141439f0 legacy=0x7fc2140cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.644+0000 7fc21a400640 1 -- 172.21.15.2:0/473183992 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc214143dd0 con 0x7fc21406c8f0 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.645+0000 7fc21a400640 1 --2- 172.21.15.2:0/473183992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc21406ccf0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fc214072670 tx=0x7fc20c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a45eb5b5fa844bd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc213e00640 1 -- 172.21.15.2:0/473183992 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c02ca80 con 0x7fc21406c8f0 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc213e00640 1 -- 172.21.15.2:0/473183992 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc20c02cbe0 con 0x7fc21406c8f0 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc213e00640 1 -- 172.21.15.2:0/473183992 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc20c0343f0 con 0x7fc21406c8f0 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc21c400640 1 -- 172.21.15.2:0/473183992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 msgr2=0x7fc21406ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc21c400640 1 --2- 172.21.15.2:0/473183992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc21406ccf0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fc214072670 tx=0x7fc20c02bee0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc21c400640 1 -- 172.21.15.2:0/473183992 shutdown_connections 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.646+0000 7fc21c400640 1 --2- 172.21.15.2:0/473183992 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc21406ccf0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.647+0000 7fc21c400640 1 --2- 172.21.15.2:0/473183992 >> v2:172.21.15.137:3300/0 conn(0x7fc2140702c0 0x7fc2140b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.647+0000 7fc21c400640 1 -- 172.21.15.2:0/473183992 >> 172.21.15.2:0/473183992 conn(0x7fc2140b45f0 msgr2=0x7fc2140b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.647+0000 7fc21c400640 1 -- 172.21.15.2:0/473183992 shutdown_connections 2024-01-15T18:18:41.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.647+0000 7fc21c400640 1 -- 172.21.15.2:0/473183992 wait complete. 2024-01-15T18:18:41.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.648+0000 7fc21c400640 1 Processor -- start 2024-01-15T18:18:41.651 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.648+0000 7fc21c400640 1 -- start start 2024-01-15T18:18:41.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.649+0000 7fc21c400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc2140ab220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.649+0000 7fc21c400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2140702c0 0x7fc2141597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.649+0000 7fc21c400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2141439f0 0x7fc214159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.649+0000 7fc21c400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a87e0 con 0x7fc21406c8f0 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.649+0000 7fc21c400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a8a30 con 0x7fc2140702c0 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.649+0000 7fc21c400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2140a8d10 con 0x7fc2141439f0 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc2140ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc2140ab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:54078/0 (socket says 172.21.15.2:54078) 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 -- 172.21.15.2:0/3003496818 learned_addr learned my addr 172.21.15.2:0/3003496818 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219000640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2140702c0 0x7fc2141597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 -- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2141439f0 msgr2=0x7fc214159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2141439f0 0x7fc214159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 -- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2140702c0 msgr2=0x7fc2141597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2140702c0 0x7fc2141597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.650+0000 7fc219a00640 1 -- 172.21.15.2:0/3003496818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20c009b30 con 0x7fc21406c8f0 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.651+0000 7fc219a00640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc21406c8f0 0x7fc2140ab220 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fc20400b3e0 tx=0x7fc20400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.651+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204011830 con 0x7fc21406c8f0 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.651+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2040119b0 con 0x7fc21406c8f0 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.651+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc204011d60 con 0x7fc21406c8f0 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.651+0000 7fc21c400640 1 -- 172.21.15.2:0/3003496818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2140a8ff0 con 0x7fc21406c8f0 2024-01-15T18:18:41.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.651+0000 7fc21c400640 1 -- 172.21.15.2:0/3003496818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2140ab760 con 0x7fc21406c8f0 2024-01-15T18:18:41.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.652+0000 7fc21c400640 1 -- 172.21.15.2:0/3003496818 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2140a9510 con 0x7fc21406c8f0 2024-01-15T18:18:41.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.653+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fc20401a400 con 0x7fc21406c8f0 2024-01-15T18:18:41.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.654+0000 7fc211600640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc1ec06c670 0x7fc1ec06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:41.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.654+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fc2040889e0 con 0x7fc21406c8f0 2024-01-15T18:18:41.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.655+0000 7fc219000640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc1ec06c670 0x7fc1ec06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.656+0000 7fc219000640 1 --2- 172.21.15.2:0/3003496818 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc1ec06c670 0x7fc1ec06eb30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fc214140d40 tx=0x7fc1fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.659+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2040552a0 con 0x7fc21406c8f0 2024-01-15T18:18:41.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.886+0000 7fc21c400640 1 -- 172.21.15.2:0/3003496818 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2140a8ba0 con 0x7fc1ec06c670 2024-01-15T18:18:41.902 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:41.899+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:41.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:41.901+0000 7fc211600640 1 -- 172.21.15.2:0/3003496818 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc2140a8ba0 con 0x7fc1ec06c670 2024-01-15T18:18:41.903 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:41.903 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:43.919 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:44.157 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.155+0000 7f1b14c00640 1 Processor -- start 2024-01-15T18:18:44.158 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.155+0000 7f1b14c00640 1 -- start start 2024-01-15T18:18:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b14c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1b101439f0 0x7f1b100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b14c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1b1006c8f0 0x7f1b1006ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b14c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b1015dd70 con 0x7f1b100b3420 2024-01-15T18:18:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b14c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b100b38b0 con 0x7f1b1006c8f0 2024-01-15T18:18:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b14c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b1006d230 con 0x7f1b101439f0 2024-01-15T18:18:44.160 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b0e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1b1006c8f0 0x7f1b1006ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b0e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f1b1006c8f0 0x7f1b1006ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42558/0 (socket says 172.21.15.2:42558) 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b0e800640 1 -- 172.21.15.2:0/1513940302 learned_addr learned my addr 172.21.15.2:0/1513940302 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.157+0000 7f1b0de00640 1 --2- 172.21.15.2:0/1513940302 >> v2:172.21.15.137:3300/0 conn(0x7f1b101439f0 0x7f1b100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.158+0000 7f1b0e800640 1 -- 172.21.15.2:0/1513940302 >> v2:172.21.15.137:3300/0 conn(0x7f1b101439f0 msgr2=0x7f1b100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.158+0000 7f1b0e800640 1 --2- 172.21.15.2:0/1513940302 >> v2:172.21.15.137:3300/0 conn(0x7f1b101439f0 0x7f1b100702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.158+0000 7f1b0e800640 1 -- 172.21.15.2:0/1513940302 >> v1:172.21.15.137:6789/0 conn(0x7f1b100b3420 legacy=0x7f1b100cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:44.161 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.158+0000 7f1b0e800640 1 -- 172.21.15.2:0/1513940302 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b100711b0 con 0x7f1b1006c8f0 2024-01-15T18:18:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.159+0000 7f1b0e800640 1 --2- 172.21.15.2:0/1513940302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1006c8f0 0x7f1b1006ccf0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b28d0 tx=0x7f1b0802e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e048cda8164c22c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.159+0000 7f1b0d400640 1 -- 172.21.15.2:0/1513940302 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b0802ec90 con 0x7f1b1006c8f0 2024-01-15T18:18:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.159+0000 7f1b0d400640 1 -- 172.21.15.2:0/1513940302 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b0802edf0 con 0x7f1b1006c8f0 2024-01-15T18:18:44.162 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.159+0000 7f1b0d400640 1 -- 172.21.15.2:0/1513940302 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b08036610 con 0x7f1b1006c8f0 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 -- 172.21.15.2:0/1513940302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1006c8f0 msgr2=0x7f1b1006ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 --2- 172.21.15.2:0/1513940302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1006c8f0 0x7f1b1006ccf0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f1b100b28d0 tx=0x7f1b0802e0f0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 -- 172.21.15.2:0/1513940302 shutdown_connections 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 --2- 172.21.15.2:0/1513940302 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b1006c8f0 0x7f1b1006ccf0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 --2- 172.21.15.2:0/1513940302 >> v2:172.21.15.137:3300/0 conn(0x7f1b101439f0 0x7f1b100702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 -- 172.21.15.2:0/1513940302 >> 172.21.15.2:0/1513940302 conn(0x7f1b100b45f0 msgr2=0x7f1b100b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 -- 172.21.15.2:0/1513940302 shutdown_connections 2024-01-15T18:18:44.163 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.160+0000 7f1b14c00640 1 -- 172.21.15.2:0/1513940302 wait complete. 2024-01-15T18:18:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.162+0000 7f1b14c00640 1 Processor -- start 2024-01-15T18:18:44.164 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.162+0000 7f1b14c00640 1 -- start start 2024-01-15T18:18:44.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b14c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1006c8f0 0x7f1b10159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b14c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b100b3420 0x7f1b100cd740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b14c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b101439f0 0x7f1b101f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b14c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b10139570 con 0x7f1b100b3420 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b14c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100cde40 con 0x7f1b1006c8f0 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b14c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b100ce120 con 0x7f1b101439f0 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.163+0000 7f1b07e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b100b3420 0x7f1b100cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b100b3420 0x7f1b100cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42574/0 (socket says 172.21.15.2:42574) 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 -- 172.21.15.2:0/307356406 learned_addr learned my addr 172.21.15.2:0/307356406 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b0de00640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1006c8f0 0x7f1b10159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b0e800640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b101439f0 0x7f1b101f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 -- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b101439f0 msgr2=0x7f1b101f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1b101439f0 0x7f1b101f02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.168 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 -- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1006c8f0 msgr2=0x7f1b10159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1b1006c8f0 0x7f1b10159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 -- 172.21.15.2:0/307356406 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b08009b30 con 0x7f1b100b3420 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.164+0000 7f1b07e00640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1b100b3420 0x7f1b100cd740 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f1af800b3e0 tx=0x7f1af800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.165+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8011830 con 0x7f1b100b3420 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.165+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af80119b0 con 0x7f1b100b3420 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.165+0000 7f1b14c00640 1 -- 172.21.15.2:0/307356406 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b100ce400 con 0x7f1b100b3420 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.165+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af8011d60 con 0x7f1b100b3420 2024-01-15T18:18:44.169 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.165+0000 7f1b14c00640 1 -- 172.21.15.2:0/307356406 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b100ce9c0 con 0x7f1b100b3420 2024-01-15T18:18:44.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.167+0000 7f1b14c00640 1 -- 172.21.15.2:0/307356406 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1015d810 con 0x7f1b100b3420 2024-01-15T18:18:44.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.167+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f1af801a400 con 0x7f1b100b3420 2024-01-15T18:18:44.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.168+0000 7f1b06000640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ae806c670 0x7f1ae806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:44.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.168+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f1af8088990 con 0x7f1b100b3420 2024-01-15T18:18:44.171 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.169+0000 7f1b0de00640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ae806c670 0x7f1ae806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:44.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.170+0000 7f1b0de00640 1 --2- 172.21.15.2:0/307356406 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1ae806c670 0x7f1ae806eb30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f1b00006fd0 tx=0x7f1b00008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.173+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1af80552b0 con 0x7f1b100b3420 2024-01-15T18:18:44.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.406+0000 7f1b14c00640 1 -- 172.21.15.2:0/307356406 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b10070bf0 con 0x7f1ae806c670 2024-01-15T18:18:44.422 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:44.418+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:44.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:44.420+0000 7f1b06000640 1 -- 172.21.15.2:0/307356406 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1b00009480 con 0x7f1ae806c670 2024-01-15T18:18:44.422 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:44.423 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:44.440 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:46.441 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:46.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.676+0000 7f3346200640 1 Processor -- start 2024-01-15T18:18:46.679 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.676+0000 7f3346200640 1 -- start start 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.678+0000 7f3346200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f334014afd0 0x7f33400b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.678+0000 7f3346200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f334014b3b0 0x7f334006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.678+0000 7f3346200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33400b25f0 con 0x7f334006ce30 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.678+0000 7f3346200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3340070ff0 con 0x7f334014b3b0 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.678+0000 7f3346200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334006d2a0 con 0x7f334014afd0 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f334014b3b0 0x7f334006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f334014b3b0 0x7f334006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42586/0 (socket says 172.21.15.2:42586) 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 -- 172.21.15.2:0/1718693006 learned_addr learned my addr 172.21.15.2:0/1718693006 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:46.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f333f400640 1 --2- 172.21.15.2:0/1718693006 >> v2:172.21.15.137:3300/0 conn(0x7f334014afd0 0x7f33400b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 -- 172.21.15.2:0/1718693006 >> v2:172.21.15.137:3300/0 conn(0x7f334014afd0 msgr2=0x7f33400b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:46.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 --2- 172.21.15.2:0/1718693006 >> v2:172.21.15.137:3300/0 conn(0x7f334014afd0 0x7f33400b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 -- 172.21.15.2:0/1718693006 >> v1:172.21.15.137:6789/0 conn(0x7f334006ce30 legacy=0x7f33400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:46.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.679+0000 7f3337e00640 1 -- 172.21.15.2:0/1718693006 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33400b28d0 con 0x7f334014b3b0 2024-01-15T18:18:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.680+0000 7f3337e00640 1 --2- 172.21.15.2:0/1718693006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f334006c8f0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f334006f9f0 tx=0x7f332802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=877258f72a9672e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f333ea00640 1 -- 172.21.15.2:0/1718693006 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33280317e0 con 0x7f334014b3b0 2024-01-15T18:18:46.684 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f333ea00640 1 -- 172.21.15.2:0/1718693006 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3328031940 con 0x7f334014b3b0 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f333ea00640 1 -- 172.21.15.2:0/1718693006 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328031cd0 con 0x7f334014b3b0 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 -- 172.21.15.2:0/1718693006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 msgr2=0x7f334006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 --2- 172.21.15.2:0/1718693006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f334006c8f0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f334006f9f0 tx=0x7f332802b880 comp rx=0 tx=0).stop 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 -- 172.21.15.2:0/1718693006 shutdown_connections 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 --2- 172.21.15.2:0/1718693006 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f334006c8f0 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 --2- 172.21.15.2:0/1718693006 >> v2:172.21.15.137:3300/0 conn(0x7f334014afd0 0x7f33400b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 -- 172.21.15.2:0/1718693006 >> 172.21.15.2:0/1718693006 conn(0x7f33400b45f0 msgr2=0x7f33400b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 -- 172.21.15.2:0/1718693006 shutdown_connections 2024-01-15T18:18:46.685 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.681+0000 7f3346200640 1 -- 172.21.15.2:0/1718693006 wait complete. 2024-01-15T18:18:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.683+0000 7f3346200640 1 Processor -- start 2024-01-15T18:18:46.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.683+0000 7f3346200640 1 -- start start 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f3346200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334006ce30 0x7f33400a8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f3346200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334014afd0 0x7f3340157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f3346200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f33400ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f3346200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33400a8d70 con 0x7f334014b3b0 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f3346200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33401577b0 con 0x7f334006ce30 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f3346200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33400b05c0 con 0x7f334014afd0 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.684+0000 7f333fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f33400ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.689 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f33400ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42590/0 (socket says 172.21.15.2:42590) 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 -- 172.21.15.2:0/2237183047 learned_addr learned my addr 172.21.15.2:0/2237183047 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333f400640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334006ce30 0x7f33400a8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f3337e00640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334014afd0 0x7f3340157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 -- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334014afd0 msgr2=0x7f3340157270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334014afd0 0x7f3340157270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 -- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334006ce30 msgr2=0x7f33400a8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334006ce30 0x7f33400a8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 -- 172.21.15.2:0/2237183047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33280094d0 con 0x7f334014b3b0 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f3337e00640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f334014afd0 0x7f3340157270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333f400640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f334006ce30 0x7f33400a8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:46.690 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.685+0000 7f333fe00640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f334014b3b0 0x7f33400ce260 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f333800c780 tx=0x7f333800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.686+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338003d80 con 0x7f334014b3b0 2024-01-15T18:18:46.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.686+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f333800ce30 con 0x7f334014b3b0 2024-01-15T18:18:46.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.686+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338005290 con 0x7f334014b3b0 2024-01-15T18:18:46.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.686+0000 7f3346200640 1 -- 172.21.15.2:0/2237183047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33400a8a90 con 0x7f334014b3b0 2024-01-15T18:18:46.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.686+0000 7f3346200640 1 -- 172.21.15.2:0/2237183047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33400b0ab0 con 0x7f334014b3b0 2024-01-15T18:18:46.691 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.687+0000 7f3346200640 1 -- 172.21.15.2:0/2237183047 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33400686f0 con 0x7f334014b3b0 2024-01-15T18:18:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.688+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 42) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f33380053f0 con 0x7f334014b3b0 2024-01-15T18:18:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.689+0000 7f3337400640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f331806c670 0x7f331806eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.689+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f33380880a0 con 0x7f334014b3b0 2024-01-15T18:18:46.692 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.689+0000 7f333f400640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f331806c670 0x7f331806eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:46.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.691+0000 7f333f400640 1 --2- 172.21.15.2:0/2237183047 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f331806c670 0x7f331806eb30 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f3330008890 tx=0x7f3330008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:46.695 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.693+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33380545b0 con 0x7f334014b3b0 2024-01-15T18:18:46.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.918+0000 7f3346200640 1 -- 172.21.15.2:0/2237183047 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33400b2e90 con 0x7f331806c670 2024-01-15T18:18:46.934 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:46.930+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:46.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:46.932+0000 7f3337400640 1 -- 172.21.15.2:0/2237183047 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f333000c040 con 0x7f331806c670 2024-01-15T18:18:46.935 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:46.935 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:48.950 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:49.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.185+0000 7f9d14a00640 1 Processor -- start 2024-01-15T18:18:49.188 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.186+0000 7f9d14a00640 1 -- start start 2024-01-15T18:18:49.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.187+0000 7f9d14a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d100b30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.187+0000 7f9d14a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9d100b3cb0 0x7f9d100b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.187+0000 7f9d14a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d100d3bd0 con 0x7f9d100cfe30 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.187+0000 7f9d14a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d100b3b50 con 0x7f9d100b3cb0 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.187+0000 7f9d14a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d100d3eb0 con 0x7f9d100cfa50 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.188+0000 7f9d0f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d100b30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.188+0000 7f9d0f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d100b30d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:45722/0 (socket says 172.21.15.2:45722) 2024-01-15T18:18:49.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.188+0000 7f9d0f400640 1 -- 172.21.15.2:0/1436579455 learned_addr learned my addr 172.21.15.2:0/1436579455 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.188+0000 7f9d0ea00640 1 --2- 172.21.15.2:0/1436579455 >> v2:172.21.15.137:3300/0 conn(0x7f9d100b3cb0 0x7f9d100b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.189+0000 7f9d0ea00640 1 -- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 msgr2=0x7f9d100b30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.189+0000 7f9d0ea00640 1 --2- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d100b30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.189+0000 7f9d0ea00640 1 -- 172.21.15.2:0/1436579455 >> v1:172.21.15.137:6789/0 conn(0x7f9d100cfe30 legacy=0x7f9d101612f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.189+0000 7f9d0e000640 1 -- 172.21.15.2:0/1436579455 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696639351 0 0) 0x7f9d100d3bd0 con 0x7f9d100cfe30 2024-01-15T18:18:49.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.189+0000 7f9d0ea00640 1 -- 172.21.15.2:0/1436579455 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d100d18a0 con 0x7f9d100b3cb0 2024-01-15T18:18:49.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.191+0000 7f9d0ea00640 1 --2- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 0x7f9d100b3610 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80097a0 tx=0x7f9cf802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=238cde6fdc8c027e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d0e000640 1 -- 172.21.15.2:0/1436579455 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf802d040 con 0x7f9d100b3cb0 2024-01-15T18:18:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d0e000640 1 -- 172.21.15.2:0/1436579455 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cf8031810 con 0x7f9d100b3cb0 2024-01-15T18:18:49.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d0e000640 1 -- 172.21.15.2:0/1436579455 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf8031ba0 con 0x7f9d100b3cb0 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 -- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 msgr2=0x7f9d100b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 --2- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 0x7f9d100b3610 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80097a0 tx=0x7f9cf802b8b0 comp rx=0 tx=0).stop 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 -- 172.21.15.2:0/1436579455 shutdown_connections 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 --2- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 0x7f9d100b3610 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 --2- 172.21.15.2:0/1436579455 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d100b30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 -- 172.21.15.2:0/1436579455 >> 172.21.15.2:0/1436579455 conn(0x7f9d100d30b0 msgr2=0x7f9d100d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.192+0000 7f9d14a00640 1 -- 172.21.15.2:0/1436579455 shutdown_connections 2024-01-15T18:18:49.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.193+0000 7f9d14a00640 1 -- 172.21.15.2:0/1436579455 wait complete. 2024-01-15T18:18:49.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.195+0000 7f9d14a00640 1 Processor -- start 2024-01-15T18:18:49.197 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.195+0000 7f9d14a00640 1 -- start start 2024-01-15T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.196+0000 7f9d14a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 0x7f9d1015ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.196+0000 7f9d14a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d1015fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.196+0000 7f9d14a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d100cfe30 0x7f9d100b4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.196+0000 7f9d14a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10161010 con 0x7f9d100cfe30 2024-01-15T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.196+0000 7f9d14a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10160490 con 0x7f9d100b3cb0 2024-01-15T18:18:49.199 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.196+0000 7f9d14a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d10160770 con 0x7f9d100cfa50 2024-01-15T18:18:49.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.197+0000 7f9d0fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d100cfe30 0x7f9d100b4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:49.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.197+0000 7f9d0fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d100cfe30 0x7f9d100b4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42600/0 (socket says 172.21.15.2:42600) 2024-01-15T18:18:49.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.197+0000 7f9d0fe00640 1 -- 172.21.15.2:0/1427653542 learned_addr learned my addr 172.21.15.2:0/1427653542 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:49.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.197+0000 7f9d0ea00640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d1015fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:49.200 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.197+0000 7f9d0f400640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 0x7f9d1015ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.197+0000 7f9d0fe00640 1 -- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 msgr2=0x7f9d1015fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d0fe00640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9d100cfa50 0x7f9d1015fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d0fe00640 1 -- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 msgr2=0x7f9d1015ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d0fe00640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9d100b3cb0 0x7f9d1015ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d0fe00640 1 -- 172.21.15.2:0/1427653542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf8009400 con 0x7f9d100cfe30 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d0fe00640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9d100cfe30 0x7f9d100b4b30 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800c480 tx=0x7f9d0800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d08005480 con 0x7f9d100cfe30 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d080035f0 con 0x7f9d100cfe30 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d080039a0 con 0x7f9d100cfe30 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d14a00640 1 -- 172.21.15.2:0/1427653542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d10160a50 con 0x7f9d100cfe30 2024-01-15T18:18:49.201 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.198+0000 7f9d14a00640 1 -- 172.21.15.2:0/1427653542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1015e300 con 0x7f9d100cfe30 2024-01-15T18:18:49.202 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.200+0000 7f9d14a00640 1 -- 172.21.15.2:0/1427653542 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1015e4f0 con 0x7f9d100cfe30 2024-01-15T18:18:49.203 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.201+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9d08003b00 con 0x7f9d100cfe30 2024-01-15T18:18:49.204 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.202+0000 7f9d07400640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ce806c550 0x7f9ce806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:49.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.202+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f9d080885b0 con 0x7f9d100cfe30 2024-01-15T18:18:49.205 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.203+0000 7f9d0f400640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ce806c550 0x7f9ce806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:49.206 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.204+0000 7f9d0f400640 1 --2- 172.21.15.2:0/1427653542 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9ce806c550 0x7f9ce806ea10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f9d000113e0 tx=0x7f9d0000b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:49.209 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.207+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d08054ba0 con 0x7f9d100cfe30 2024-01-15T18:18:49.439 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.436+0000 7f9d14a00640 1 -- 172.21.15.2:0/1427653542 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9d100d1e60 con 0x7f9ce806c550 2024-01-15T18:18:49.455 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:49.452+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:49.456 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:49.453+0000 7f9d07400640 1 -- 172.21.15.2:0/1427653542 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9d0000f040 con 0x7f9ce806c550 2024-01-15T18:18:49.456 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:49.456 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:51.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:51.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.702+0000 7fb558a00640 1 Processor -- start 2024-01-15T18:18:51.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.702+0000 7fb558a00640 1 -- start start 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.703+0000 7fb558a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb55015c450 0x7fb55006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.703+0000 7fb558a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb55006ce30 0x7fb550070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.703+0000 7fb558a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55006d290 con 0x7fb550144c00 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.703+0000 7fb558a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5500ce230 con 0x7fb55015c450 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.703+0000 7fb558a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5500ce510 con 0x7fb55006ce30 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb556000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb550144c00 0x7fb5500cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:41358/0 (socket says 172.21.15.2:41358) 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb556000640 1 -- 172.21.15.2:0/1735447051 learned_addr learned my addr 172.21.15.2:0/1735447051 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:18:51.706 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb555600640 1 --2- 172.21.15.2:0/1735447051 >> v2:172.21.15.2:3300/0 conn(0x7fb55015c450 0x7fb55006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:51.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb556a00640 1 --2- 172.21.15.2:0/1735447051 >> v2:172.21.15.137:3300/0 conn(0x7fb55006ce30 0x7fb550070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:51.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb554c00640 1 -- 172.21.15.2:0/1735447051 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642259749 0 0) 0x7fb55006d290 con 0x7fb550144c00 2024-01-15T18:18:51.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb555600640 1 -- 172.21.15.2:0/1735447051 >> v2:172.21.15.137:3300/0 conn(0x7fb55006ce30 msgr2=0x7fb550070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:51.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb555600640 1 --2- 172.21.15.2:0/1735447051 >> v2:172.21.15.137:3300/0 conn(0x7fb55006ce30 0x7fb550070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:51.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb555600640 1 -- 172.21.15.2:0/1735447051 >> v1:172.21.15.2:6789/0 conn(0x7fb550144c00 legacy=0x7fb5500cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:51.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.704+0000 7fb555600640 1 -- 172.21.15.2:0/1735447051 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5500ce7f0 con 0x7fb55015c450 2024-01-15T18:18:51.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.706+0000 7fb555600640 1 --2- 172.21.15.2:0/1735447051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55006c8f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb55015ec50 tx=0x7fb53802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992fa983ab87f4d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:51.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.706+0000 7fb554c00640 1 -- 172.21.15.2:0/1735447051 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538031830 con 0x7fb55015c450 2024-01-15T18:18:51.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.706+0000 7fb554c00640 1 -- 172.21.15.2:0/1735447051 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb538031990 con 0x7fb55015c450 2024-01-15T18:18:51.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.706+0000 7fb554c00640 1 -- 172.21.15.2:0/1735447051 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb538031d20 con 0x7fb55015c450 2024-01-15T18:18:51.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 -- 172.21.15.2:0/1735447051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 msgr2=0x7fb55006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:51.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 --2- 172.21.15.2:0/1735447051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55006c8f0 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb55015ec50 tx=0x7fb53802b8d0 comp rx=0 tx=0).stop 2024-01-15T18:18:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 -- 172.21.15.2:0/1735447051 shutdown_connections 2024-01-15T18:18:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 --2- 172.21.15.2:0/1735447051 >> v2:172.21.15.137:3300/0 conn(0x7fb55006ce30 0x7fb550070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 --2- 172.21.15.2:0/1735447051 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55006c8f0 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 -- 172.21.15.2:0/1735447051 >> 172.21.15.2:0/1735447051 conn(0x7fb5500b45f0 msgr2=0x7fb5500b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 -- 172.21.15.2:0/1735447051 shutdown_connections 2024-01-15T18:18:51.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.707+0000 7fb558a00640 1 -- 172.21.15.2:0/1735447051 wait complete. 2024-01-15T18:18:51.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.709+0000 7fb558a00640 1 Processor -- start 2024-01-15T18:18:51.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.709+0000 7fb558a00640 1 -- start start 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.710+0000 7fb558a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb55006ce30 0x7fb5501f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.710+0000 7fb558a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb550144c00 0x7fb5500b11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.710+0000 7fb558a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55013e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.710+0000 7fb558a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500af8f0 con 0x7fb55015c450 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.710+0000 7fb558a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500b16e0 con 0x7fb55006ce30 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.710+0000 7fb558a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5500aec70 con 0x7fb550144c00 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55013e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55013e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:42608/0 (socket says 172.21.15.2:42608) 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 -- 172.21.15.2:0/2864148082 learned_addr learned my addr 172.21.15.2:0/2864148082 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:51.713 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556000640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb55006ce30 0x7fb5501f28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb555600640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb550144c00 0x7fb5500b11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:51.714 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 -- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb550144c00 msgr2=0x7fb5500b11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb550144c00 0x7fb5500b11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 -- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb55006ce30 msgr2=0x7fb5501f28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb55006ce30 0x7fb5501f28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 -- 172.21.15.2:0/2864148082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538009400 con 0x7fb55015c450 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.711+0000 7fb556a00640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb55015c450 0x7fb55013e9e0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fb54000d800 tx=0x7fb54000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.712+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54000b6c0 con 0x7fb55015c450 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.712+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54000b840 con 0x7fb55015c450 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.712+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54000bbf0 con 0x7fb55015c450 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.712+0000 7fb558a00640 1 -- 172.21.15.2:0/2864148082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5500aef50 con 0x7fb55015c450 2024-01-15T18:18:51.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.712+0000 7fb558a00640 1 -- 172.21.15.2:0/2864148082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5500af510 con 0x7fb55015c450 2024-01-15T18:18:51.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.713+0000 7fb558a00640 1 -- 172.21.15.2:0/2864148082 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5500afa30 con 0x7fb55015c450 2024-01-15T18:18:51.716 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.714+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb54000bd50 con 0x7fb55015c450 2024-01-15T18:18:51.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.715+0000 7fb54e000640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb52806c550 0x7fb52806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:51.717 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.715+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fb540089120 con 0x7fb55015c450 2024-01-15T18:18:51.718 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.716+0000 7fb556000640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb52806c550 0x7fb52806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:51.719 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.717+0000 7fb556000640 1 --2- 172.21.15.2:0/2864148082 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb52806c550 0x7fb52806ea10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fb5480012e0 tx=0x7fb548007ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:51.722 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.720+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5500afa30 con 0x7fb55015c450 2024-01-15T18:18:51.949 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.947+0000 7fb558a00640 1 -- 172.21.15.2:0/2864148082 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5500cead0 con 0x7fb52806c550 2024-01-15T18:18:51.962 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:51.959+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:51.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:51.960+0000 7fb54e000640 1 -- 172.21.15.2:0/2864148082 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5480079d0 con 0x7fb52806c550 2024-01-15T18:18:51.963 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:51.963 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:53.977 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:54.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.217+0000 7f3d61a00640 1 Processor -- start 2024-01-15T18:18:54.219 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.217+0000 7f3d61a00640 1 -- start start 2024-01-15T18:18:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c070560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d61a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d5c070aa0 0x7f3d5c0b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d61a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5c071250 con 0x7f3d5c150890 2024-01-15T18:18:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d61a00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5c071530 con 0x7f3d5c070aa0 2024-01-15T18:18:54.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d61a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c0b2ae0 con 0x7f3d5c0b1290 2024-01-15T18:18:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d5b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d5c070aa0 0x7f3d5c0b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d5b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3d5c070aa0 0x7f3d5c0b32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49886/0 (socket says 172.21.15.2:49886) 2024-01-15T18:18:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d5b400640 1 -- 172.21.15.2:0/2166558910 learned_addr learned my addr 172.21.15.2:0/2166558910 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d5a000640 1 --2- 172.21.15.2:0/2166558910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c070560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:54.222 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d5b400640 1 -- 172.21.15.2:0/2166558910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 msgr2=0x7f3d5c070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.219+0000 7f3d5b400640 1 --2- 172.21.15.2:0/2166558910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c070560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.220+0000 7f3d5b400640 1 -- 172.21.15.2:0/2166558910 >> v1:172.21.15.137:6789/0 conn(0x7f3d5c150890 legacy=0x7f3d5c0b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.220+0000 7f3d5b400640 1 -- 172.21.15.2:0/2166558910 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c0b2dc0 con 0x7f3d5c070aa0 2024-01-15T18:18:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.221+0000 7f3d5b400640 1 --2- 172.21.15.2:0/2166558910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c0b32e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009870 tx=0x7f3d5402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ee2451f76c7dd68 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.221+0000 7f3d59600640 1 -- 172.21.15.2:0/2166558910 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d540317c0 con 0x7f3d5c070aa0 2024-01-15T18:18:54.223 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.221+0000 7f3d59600640 1 -- 172.21.15.2:0/2166558910 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d54031920 con 0x7f3d5c070aa0 2024-01-15T18:18:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.221+0000 7f3d59600640 1 -- 172.21.15.2:0/2166558910 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d54031cb0 con 0x7f3d5c070aa0 2024-01-15T18:18:54.224 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 -- 172.21.15.2:0/2166558910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 msgr2=0x7f3d5c0b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 --2- 172.21.15.2:0/2166558910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c0b32e0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f3d54009870 tx=0x7f3d5402b860 comp rx=0 tx=0).stop 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 -- 172.21.15.2:0/2166558910 shutdown_connections 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 --2- 172.21.15.2:0/2166558910 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c0b32e0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 --2- 172.21.15.2:0/2166558910 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 -- 172.21.15.2:0/2166558910 >> 172.21.15.2:0/2166558910 conn(0x7f3d5c0b45f0 msgr2=0x7f3d5c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 -- 172.21.15.2:0/2166558910 shutdown_connections 2024-01-15T18:18:54.225 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.222+0000 7f3d61a00640 1 -- 172.21.15.2:0/2166558910 wait complete. 2024-01-15T18:18:54.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.224+0000 7f3d61a00640 1 Processor -- start 2024-01-15T18:18:54.226 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.224+0000 7f3d61a00640 1 -- start start 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c1454c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c14c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d61a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d5c150890 0x7f3d5c0cd090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d61a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c0ce480 con 0x7f3d5c070aa0 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d61a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c0cd5d0 con 0x7f3d5c150890 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d61a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5c0cd870 con 0x7f3d5c0b1290 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d5aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c1454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:54.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d5aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c1454c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49902/0 (socket says 172.21.15.2:49902) 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d5aa00640 1 -- 172.21.15.2:0/2815310171 learned_addr learned my addr 172.21.15.2:0/2815310171 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d5a000640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c14c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d5b400640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d5c150890 0x7f3d5c0cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.225+0000 7f3d5aa00640 1 -- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 msgr2=0x7f3d5c14c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d5aa00640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3d5c0b1290 0x7f3d5c14c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d5aa00640 1 -- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d5c150890 msgr2=0x7f3d5c0cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d5aa00640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d5c150890 0x7f3d5c0cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d5aa00640 1 -- 172.21.15.2:0/2815310171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d540094d0 con 0x7f3d5c070aa0 2024-01-15T18:18:54.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d5b400640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3d5c150890 0x7f3d5c0cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:54.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d5aa00640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3d5c070aa0 0x7f3d5c1454c0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c00c7c0 tx=0x7f3d4c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:54.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c00aa50 con 0x7f3d5c070aa0 2024-01-15T18:18:54.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4c00abb0 con 0x7f3d5c070aa0 2024-01-15T18:18:54.230 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d61a00640 1 -- 172.21.15.2:0/2815310171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5c0cdb50 con 0x7f3d5c070aa0 2024-01-15T18:18:54.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.226+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c012420 con 0x7f3d5c070aa0 2024-01-15T18:18:54.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.227+0000 7f3d61a00640 1 -- 172.21.15.2:0/2815310171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5c0ce110 con 0x7f3d5c070aa0 2024-01-15T18:18:54.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.227+0000 7f3d61a00640 1 -- 172.21.15.2:0/2815310171 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5c0686f0 con 0x7f3d5c070aa0 2024-01-15T18:18:54.231 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.229+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3d4c012580 con 0x7f3d5c070aa0 2024-01-15T18:18:54.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.229+0000 7f3d52a00640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3406c550 0x7f3d3406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:54.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.230+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f3d4c088e20 con 0x7f3d5c070aa0 2024-01-15T18:18:54.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.230+0000 7f3d5a000640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3406c550 0x7f3d3406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:54.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.231+0000 7f3d5a000640 1 --2- 172.21.15.2:0/2815310171 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3d3406c550 0x7f3d3406ea10 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c1f6390 tx=0x7f3d44008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:54.236 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.234+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d4c055360 con 0x7f3d5c070aa0 2024-01-15T18:18:54.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.459+0000 7f3d61a00640 1 -- 172.21.15.2:0/2815310171 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d5c070f10 con 0x7f3d3406c550 2024-01-15T18:18:54.475 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:54.472+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:54.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:54.473+0000 7f3d52a00640 1 -- 172.21.15.2:0/2815310171 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3d4400b200 con 0x7f3d3406c550 2024-01-15T18:18:54.476 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:54.476 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:56.492 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:56.731 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.729+0000 7f5e6be00640 1 Processor -- start 2024-01-15T18:18:56.732 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.729+0000 7f5e6be00640 1 -- start start 2024-01-15T18:18:56.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c1609d0 0x7f5e6c0cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:56.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5e6c0b3fa0 0x7f5e6c0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:56.737 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6be00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c0d0af0 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6be00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c0ce7b0 con 0x7f5e6c0b3fa0 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0cdbe0 con 0x7f5e6c1609d0 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5e6c0b3fa0 0x7f5e6c0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6b400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f5e6c0b3fa0 0x7f5e6c0d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49904/0 (socket says 172.21.15.2:49904) 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.731+0000 7f5e6b400640 1 -- 172.21.15.2:0/2612297425 learned_addr learned my addr 172.21.15.2:0/2612297425 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.732+0000 7f5e6b400640 1 -- 172.21.15.2:0/2612297425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c1609d0 msgr2=0x7f5e6c0cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.732+0000 7f5e6aa00640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c1609d0 0x7f5e6c0cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.732+0000 7f5e6b400640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c1609d0 0x7f5e6c0cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.732+0000 7f5e6b400640 1 -- 172.21.15.2:0/2612297425 >> v1:172.21.15.137:6789/0 conn(0x7f5e6c0d1d40 legacy=0x7f5e6c0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.732+0000 7f5e6b400640 1 -- 172.21.15.2:0/2612297425 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0cde80 con 0x7f5e6c0b3fa0 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.732+0000 7f5e6aa00640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c1609d0 0x7f5e6c0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.733+0000 7f5e6b400640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0d01d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009e50 tx=0x7f5e6402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b78a9c6efc3af6b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:56.738 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.733+0000 7f5e69600640 1 -- 172.21.15.2:0/2612297425 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6402ca80 con 0x7f5e6c0b3fa0 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.733+0000 7f5e69600640 1 -- 172.21.15.2:0/2612297425 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6402cbe0 con 0x7f5e6c0b3fa0 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.733+0000 7f5e69600640 1 -- 172.21.15.2:0/2612297425 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e640343f0 con 0x7f5e6c0b3fa0 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 -- 172.21.15.2:0/2612297425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0b3fa0 msgr2=0x7f5e6c0d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0d01d0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009e50 tx=0x7f5e6402bee0 comp rx=0 tx=0).stop 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 -- 172.21.15.2:0/2612297425 shutdown_connections 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0d01d0 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 --2- 172.21.15.2:0/2612297425 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c1609d0 0x7f5e6c0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 -- 172.21.15.2:0/2612297425 >> 172.21.15.2:0/2612297425 conn(0x7f5e6c09c3e0 msgr2=0x7f5e6c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 -- 172.21.15.2:0/2612297425 shutdown_connections 2024-01-15T18:18:56.739 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.734+0000 7f5e6be00640 1 -- 172.21.15.2:0/2612297425 wait complete. 2024-01-15T18:18:56.741 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.736+0000 7f5e6be00640 1 Processor -- start 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.736+0000 7f5e6be00640 1 -- start start 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0d1d40 0x7f5e6c0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6be00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e6c1609d0 0x7f5e6c0ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6be00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0ccb00 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6be00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0ca6e0 con 0x7f5e6c1609d0 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6be00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0cbef0 con 0x7f5e6c0b3fa0 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0d1d40 0x7f5e6c0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6a000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0d1d40 0x7f5e6c0c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49908/0 (socket says 172.21.15.2:49908) 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.737+0000 7f5e6a000640 1 -- 172.21.15.2:0/1193429061 learned_addr learned my addr 172.21.15.2:0/1193429061 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:56.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6b400640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e6c1609d0 0x7f5e6c0ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6aa00640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6a000640 1 -- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c0b3fa0 msgr2=0x7f5e6c0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6a000640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6a000640 1 -- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e6c1609d0 msgr2=0x7f5e6c0ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6a000640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e6c1609d0 0x7f5e6c0ccf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6a000640 1 -- 172.21.15.2:0/1193429061 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e64009b30 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6b400640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5e6c1609d0 0x7f5e6c0ccf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6aa00640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5e6c0b3fa0 0x7f5e6c0ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.738+0000 7f5e6a000640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5e6c0d1d40 0x7f5e6c0c8fb0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400b3f0 tx=0x7f5e5400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.739+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54011830 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.739+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54011990 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.739+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54011d40 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.739+0000 7f5e6be00640 1 -- 172.21.15.2:0/1193429061 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c0cc1d0 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.743 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.739+0000 7f5e6be00640 1 -- 172.21.15.2:0/1193429061 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c0cc790 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.740+0000 7f5e6be00640 1 -- 172.21.15.2:0/1193429061 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c0b4780 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.741+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f5e5401a400 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.744 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.742+0000 7f5e62a00640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e4406c580 0x7f5e4406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:56.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.742+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f5e54088930 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.745 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.743+0000 7f5e6aa00640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e4406c580 0x7f5e4406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:56.746 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.744+0000 7f5e6aa00640 1 --2- 172.21.15.2:0/1193429061 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5e4406c580 0x7f5e4406ea40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c006fd0 tx=0x7f5e5c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:56.749 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.747+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e54054ed0 con 0x7f5e6c0d1d40 2024-01-15T18:18:56.985 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.983+0000 7f5e6be00640 1 -- 172.21.15.2:0/1193429061 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e6c160db0 con 0x7f5e4406c580 2024-01-15T18:18:56.998 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:56.995+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:56.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:56.996+0000 7f5e62a00640 1 -- 172.21.15.2:0/1193429061 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5e5c00c040 con 0x7f5e4406c580 2024-01-15T18:18:56.999 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:56.999 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:18:59.020 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:18:59.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.257+0000 7f4d5fe00640 1 Processor -- start 2024-01-15T18:18:59.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.257+0000 7f4d5fe00640 1 -- start start 2024-01-15T18:18:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d601609d0 0x7f4d600cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:59.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600d0af0 con 0x7f4d600cfac0 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5fe00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600ce7b0 con 0x7f4d601609d0 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600cdbe0 con 0x7f4d600d1d40 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d601609d0 0x7f4d600cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5ea00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4d601609d0 0x7f4d600cec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49922/0 (socket says 172.21.15.2:49922) 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d5ea00640 1 -- 172.21.15.2:0/2051781392 learned_addr learned my addr 172.21.15.2:0/2051781392 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.259+0000 7f4d57e00640 1 --2- 172.21.15.2:0/2051781392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.260+0000 7f4d5ea00640 1 -- 172.21.15.2:0/2051781392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 msgr2=0x7f4d600b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.260+0000 7f4d5ea00640 1 --2- 172.21.15.2:0/2051781392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.260+0000 7f4d5ea00640 1 -- 172.21.15.2:0/2051781392 >> v1:172.21.15.137:6789/0 conn(0x7f4d600cfac0 legacy=0x7f4d600cff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:18:59.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.260+0000 7f4d5ea00640 1 -- 172.21.15.2:0/2051781392 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600cde80 con 0x7f4d601609d0 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.261+0000 7f4d5ea00640 1 --2- 172.21.15.2:0/2051781392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d601609d0 0x7f4d600cec40 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009870 tx=0x7f4d5002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104d4594c0f1ae72 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.261+0000 7f4d5e000640 1 -- 172.21.15.2:0/2051781392 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50031640 con 0x7f4d601609d0 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.261+0000 7f4d5e000640 1 -- 172.21.15.2:0/2051781392 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d500317a0 con 0x7f4d601609d0 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.261+0000 7f4d5e000640 1 -- 172.21.15.2:0/2051781392 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50031b30 con 0x7f4d601609d0 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 -- 172.21.15.2:0/2051781392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d601609d0 msgr2=0x7f4d600cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 --2- 172.21.15.2:0/2051781392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d601609d0 0x7f4d600cec40 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009870 tx=0x7f4d5002b8f0 comp rx=0 tx=0).stop 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 -- 172.21.15.2:0/2051781392 shutdown_connections 2024-01-15T18:18:59.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 --2- 172.21.15.2:0/2051781392 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:59.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 --2- 172.21.15.2:0/2051781392 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d601609d0 0x7f4d600cec40 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:59.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 -- 172.21.15.2:0/2051781392 >> 172.21.15.2:0/2051781392 conn(0x7f4d6009c3e0 msgr2=0x7f4d600d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:18:59.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 -- 172.21.15.2:0/2051781392 shutdown_connections 2024-01-15T18:18:59.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.262+0000 7f4d5fe00640 1 -- 172.21.15.2:0/2051781392 wait complete. 2024-01-15T18:18:59.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.265+0000 7f4d5fe00640 1 Processor -- start 2024-01-15T18:18:59.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.265+0000 7f4d5fe00640 1 -- start start 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.266+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d600cfac0 0x7f4d6014f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.266+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600c6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.266+0000 7f4d5fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d601609d0 0x7f4d600c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.266+0000 7f4d5fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600cae00 con 0x7f4d600cfac0 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.266+0000 7f4d5fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014fce0 con 0x7f4d601609d0 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.266+0000 7f4d5fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600b7f60 con 0x7f4d600d1d40 2024-01-15T18:18:59.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d600cfac0 0x7f4d6014f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d600cfac0 0x7f4d6014f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49938/0 (socket says 172.21.15.2:49938) 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 -- 172.21.15.2:0/1139870831 learned_addr learned my addr 172.21.15.2:0/1139870831 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d57e00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600c6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 -- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 msgr2=0x7f4d600c6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5f400640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d601609d0 0x7f4d600c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600c6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 -- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d601609d0 msgr2=0x7f4d600c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d601609d0 0x7f4d600c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5ea00640 1 -- 172.21.15.2:0/1139870831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094d0 con 0x7f4d600cfac0 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d57e00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4d600d1d40 0x7f4d600c6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.267+0000 7f4d5f400640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4d601609d0 0x7f4d600c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:18:59.272 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.268+0000 7f4d5ea00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4d600cfac0 0x7f4d6014f7a0 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f4d5002b930 tx=0x7f4d500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.268+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5003ac60 con 0x7f4d600cfac0 2024-01-15T18:18:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.268+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5003adc0 con 0x7f4d600cfac0 2024-01-15T18:18:59.273 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.268+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50042630 con 0x7f4d600cfac0 2024-01-15T18:18:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.268+0000 7f4d5fe00640 1 -- 172.21.15.2:0/1139870831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600b8240 con 0x7f4d600cfac0 2024-01-15T18:18:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.268+0000 7f4d5fe00640 1 -- 172.21.15.2:0/1139870831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d600cb050 con 0x7f4d600cfac0 2024-01-15T18:18:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.270+0000 7f4d5fe00640 1 -- 172.21.15.2:0/1139870831 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d600b4780 con 0x7f4d600cfac0 2024-01-15T18:18:59.274 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.271+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f4d5002f050 con 0x7f4d600cfac0 2024-01-15T18:18:59.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.272+0000 7f4d56a00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3806c550 0x7f4d3806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:18:59.275 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.272+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f4d500aa0e0 con 0x7f4d600cfac0 2024-01-15T18:18:59.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.272+0000 7f4d57e00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3806c550 0x7f4d3806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:18:59.277 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.274+0000 7f4d57e00640 1 --2- 172.21.15.2:0/1139870831 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4d3806c550 0x7f4d3806ea10 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f4d48006fd0 tx=0x7f4d48008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:18:59.282 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.279+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4d500769a0 con 0x7f4d600cfac0 2024-01-15T18:18:59.508 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.504+0000 7f4d5fe00640 1 -- 172.21.15.2:0/1139870831 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4d600c69a0 con 0x7f4d3806c550 2024-01-15T18:18:59.524 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:18:59.521+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:18:59.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:18:59.521+0000 7f4d56a00640 1 -- 172.21.15.2:0/1139870831 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4d600c69a0 con 0x7f4d3806c550 2024-01-15T18:18:59.526 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:18:59.526 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:18:59.540 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:01.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:01.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.777+0000 7fb208400640 1 Processor -- start 2024-01-15T18:19:01.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.777+0000 7fb208400640 1 -- start start 2024-01-15T18:19:01.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.778+0000 7fb208400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb200144c00 0x7fb2000cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb208400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2000cdd70 0x7fb20006c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb208400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb20015c450 0x7fb20006ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb208400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2000ce230 con 0x7fb2000cdd70 2024-01-15T18:19:01.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb208400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2000ce510 con 0x7fb20015c450 2024-01-15T18:19:01.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb208400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2000ce7f0 con 0x7fb200144c00 2024-01-15T18:19:01.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb205000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2000cdd70 0x7fb20006c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:01.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb205000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb2000cdd70 0x7fb20006c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49942/0 (socket says 172.21.15.2:49942) 2024-01-15T18:19:01.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb205000640 1 -- 172.21.15.2:0/2574457132 learned_addr learned my addr 172.21.15.2:0/2574457132 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb205000640 1 -- 172.21.15.2:0/2574457132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb200144c00 msgr2=0x7fb2000cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.779+0000 7fb205000640 1 --2- 172.21.15.2:0/2574457132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb200144c00 0x7fb2000cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.780+0000 7fb205000640 1 -- 172.21.15.2:0/2574457132 >> v2:172.21.15.137:3300/0 conn(0x7fb20015c450 msgr2=0x7fb20006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.780+0000 7fb205000640 1 --2- 172.21.15.2:0/2574457132 >> v2:172.21.15.137:3300/0 conn(0x7fb20015c450 0x7fb20006ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.780+0000 7fb205000640 1 -- 172.21.15.2:0/2574457132 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2000705f0 con 0x7fb2000cdd70 2024-01-15T18:19:01.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.781+0000 7fb205000640 1 --2- 172.21.15.2:0/2574457132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2000cdd70 0x7fb20006c8f0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fb2000cf090 tx=0x7fb1e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1e473a2f338ebd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:01.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.781+0000 7fb1ffe00640 1 -- 172.21.15.2:0/2574457132 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8031870 con 0x7fb2000cdd70 2024-01-15T18:19:01.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.781+0000 7fb1ffe00640 1 -- 172.21.15.2:0/2574457132 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e80319d0 con 0x7fb2000cdd70 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.781+0000 7fb1ffe00640 1 -- 172.21.15.2:0/2574457132 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8031d60 con 0x7fb2000cdd70 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 -- 172.21.15.2:0/2574457132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2000cdd70 msgr2=0x7fb20006c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 --2- 172.21.15.2:0/2574457132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2000cdd70 0x7fb20006c8f0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7fb2000cf090 tx=0x7fb1e802b910 comp rx=0 tx=0).stop 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 -- 172.21.15.2:0/2574457132 shutdown_connections 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 --2- 172.21.15.2:0/2574457132 >> v2:172.21.15.137:3300/0 conn(0x7fb20015c450 0x7fb20006ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 --2- 172.21.15.2:0/2574457132 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb2000cdd70 0x7fb20006c8f0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 --2- 172.21.15.2:0/2574457132 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb200144c00 0x7fb2000cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 -- 172.21.15.2:0/2574457132 >> 172.21.15.2:0/2574457132 conn(0x7fb2000b45f0 msgr2=0x7fb2000b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 -- 172.21.15.2:0/2574457132 shutdown_connections 2024-01-15T18:19:01.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.782+0000 7fb208400640 1 -- 172.21.15.2:0/2574457132 wait complete. 2024-01-15T18:19:01.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.784+0000 7fb208400640 1 Processor -- start 2024-01-15T18:19:01.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.784+0000 7fb208400640 1 -- start start 2024-01-15T18:19:01.789 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb208400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2000cdd70 0x7fb2001403e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb208400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb200144c00 0x7fb200152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb208400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb20015c450 0x7fb20013e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb208400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20013cdc0 con 0x7fb200144c00 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb208400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200138ba0 con 0x7fb2000cdd70 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb208400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb200138e80 con 0x7fb20015c450 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb200144c00 0x7fb200152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb200144c00 0x7fb200152340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:49954/0 (socket says 172.21.15.2:49954) 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 -- 172.21.15.2:0/2509397478 learned_addr learned my addr 172.21.15.2:0/2509397478 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 -- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb20015c450 msgr2=0x7fb20013e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb20015c450 0x7fb20013e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 -- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2000cdd70 msgr2=0x7fb2001403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205a00640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2000cdd70 0x7fb2001403e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb2000cdd70 0x7fb2001403e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:01.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.785+0000 7fb205000640 1 -- 172.21.15.2:0/2509397478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e80094d0 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.786+0000 7fb205000640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb200144c00 0x7fb200152340 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8000c00 tx=0x7fb1e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.786+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e803ad10 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.786+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e803ae70 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.786+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e8044610 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.786+0000 7fb208400640 1 -- 172.21.15.2:0/2509397478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20013c700 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.786+0000 7fb208400640 1 -- 172.21.15.2:0/2509397478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb200152880 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.787+0000 7fb208400640 1 -- 172.21.15.2:0/2509397478 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20013cf00 con 0x7fb200144c00 2024-01-15T18:19:01.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.788+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb1e804d450 con 0x7fb200144c00 2024-01-15T18:19:01.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.789+0000 7fb1fd600640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1d806c550 0x7fb1d806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:01.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.789+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fb1e802d040 con 0x7fb200144c00 2024-01-15T18:19:01.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.790+0000 7fb205a00640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1d806c550 0x7fb1d806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:01.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.791+0000 7fb205a00640 1 --2- 172.21.15.2:0/2509397478 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb1d806c550 0x7fb1d806ea10 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fb1f000a810 tx=0x7fb1f0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:01.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:01.793+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb20013cf00 con 0x7fb200144c00 2024-01-15T18:19:02.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:02.025+0000 7fb208400640 1 -- 172.21.15.2:0/2509397478 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb200066fa0 con 0x7fb1d806c550 2024-01-15T18:19:02.044 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:02.041+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:02.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:02.041+0000 7fb1fd600640 1 -- 172.21.15.2:0/2509397478 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb200066fa0 con 0x7fb1d806c550 2024-01-15T18:19:02.045 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:02.045 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:02.058 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:04.059 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:04.298 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.295+0000 7f0ed5e00640 1 Processor -- start 2024-01-15T18:19:04.299 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.295+0000 7f0ed5e00640 1 -- start start 2024-01-15T18:19:04.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.297+0000 7f0ed5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed015ae80 0x7f0ed00acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:04.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.297+0000 7f0ed5e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0ed006c8f0 0x7f0ed006cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.297+0000 7f0ed5e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed006d290 con 0x7f0ed00b2430 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.297+0000 7f0ed5e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed015e840 con 0x7f0ed006c8f0 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.297+0000 7f0ed5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed015dbc0 con 0x7f0ed015ae80 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.298+0000 7f0ecea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed015ae80 0x7f0ed00acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.298+0000 7f0ecf400640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f0ed00b2430 0x7f0ed00b28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:50934/0 (socket says 172.21.15.2:50934) 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.298+0000 7f0ecf400640 1 -- 172.21.15.2:0/3177616613 learned_addr learned my addr 172.21.15.2:0/3177616613 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:04.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.298+0000 7f0ec7e00640 1 --2- 172.21.15.2:0/3177616613 >> v2:172.21.15.137:3300/0 conn(0x7f0ed006c8f0 0x7f0ed006cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:04.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.299+0000 7f0ecea00640 1 -- 172.21.15.2:0/3177616613 >> v2:172.21.15.137:3300/0 conn(0x7f0ed006c8f0 msgr2=0x7f0ed006cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:04.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.299+0000 7f0ecea00640 1 --2- 172.21.15.2:0/3177616613 >> v2:172.21.15.137:3300/0 conn(0x7f0ed006c8f0 0x7f0ed006cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:04.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.299+0000 7f0ece000640 1 -- 172.21.15.2:0/3177616613 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930684025 0 0) 0x7f0ed006d290 con 0x7f0ed00b2430 2024-01-15T18:19:04.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.299+0000 7f0ecea00640 1 -- 172.21.15.2:0/3177616613 >> v1:172.21.15.137:6789/0 conn(0x7f0ed00b2430 legacy=0x7f0ed00b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:04.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.299+0000 7f0ecea00640 1 -- 172.21.15.2:0/3177616613 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed015dea0 con 0x7f0ed015ae80 2024-01-15T18:19:04.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.300+0000 7f0ecea00640 1 --2- 172.21.15.2:0/3177616613 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed015ae80 0x7f0ed00acd00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00097a0 tx=0x7f0ec002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e572e2561d1eeb5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:04.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.301+0000 7f0ece000640 1 -- 172.21.15.2:0/3177616613 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0031830 con 0x7f0ed015ae80 2024-01-15T18:19:04.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.301+0000 7f0ece000640 1 -- 172.21.15.2:0/3177616613 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec0031990 con 0x7f0ed015ae80 2024-01-15T18:19:04.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.301+0000 7f0ece000640 1 -- 172.21.15.2:0/3177616613 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0031d20 con 0x7f0ed015ae80 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.301+0000 7f0ed5e00640 1 -- 172.21.15.2:0/3177616613 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed015ae80 msgr2=0x7f0ed00acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.301+0000 7f0ed5e00640 1 --2- 172.21.15.2:0/3177616613 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed015ae80 0x7f0ed00acd00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00097a0 tx=0x7f0ec002b8d0 comp rx=0 tx=0).stop 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.302+0000 7f0ed5e00640 1 -- 172.21.15.2:0/3177616613 shutdown_connections 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.302+0000 7f0ed5e00640 1 --2- 172.21.15.2:0/3177616613 >> v2:172.21.15.137:3300/0 conn(0x7f0ed006c8f0 0x7f0ed006cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.302+0000 7f0ed5e00640 1 --2- 172.21.15.2:0/3177616613 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed015ae80 0x7f0ed00acd00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.302+0000 7f0ed5e00640 1 -- 172.21.15.2:0/3177616613 >> 172.21.15.2:0/3177616613 conn(0x7f0ed00b45f0 msgr2=0x7f0ed00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.302+0000 7f0ed5e00640 1 -- 172.21.15.2:0/3177616613 shutdown_connections 2024-01-15T18:19:04.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.302+0000 7f0ed5e00640 1 -- 172.21.15.2:0/3177616613 wait complete. 2024-01-15T18:19:04.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.304+0000 7f0ed5e00640 1 Processor -- start 2024-01-15T18:19:04.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.304+0000 7f0ed5e00640 1 -- start start 2024-01-15T18:19:04.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ed5e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006c8f0 0x7f0ed0156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:04.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ed5e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed00b2430 0x7f0ed0158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:04.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ed5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed015ae80 0x7f0ed0158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ed5e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0159570 con 0x7f0ed015ae80 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ed5e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0159850 con 0x7f0ed006c8f0 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ed5e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0159b30 con 0x7f0ed00b2430 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ecf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed015ae80 0x7f0ed0158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ecf400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed015ae80 0x7f0ed0158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55894/0 (socket says 172.21.15.2:55894) 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ecf400640 1 -- 172.21.15.2:0/2934529244 learned_addr learned my addr 172.21.15.2:0/2934529244 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:04.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.305+0000 7f0ecea00640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006c8f0 0x7f0ed0156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:04.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ec7e00640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed00b2430 0x7f0ed0158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:04.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ecf400640 1 -- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed00b2430 msgr2=0x7f0ed0158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:04.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ecf400640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed00b2430 0x7f0ed0158a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:04.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ecf400640 1 -- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006c8f0 msgr2=0x7f0ed0156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:04.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ecf400640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ed006c8f0 0x7f0ed0156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:04.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ecf400640 1 -- 172.21.15.2:0/2934529244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec0009400 con 0x7f0ed015ae80 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ec7e00640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ed00b2430 0x7f0ed0158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ecf400640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ed015ae80 0x7f0ed0158f60 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800c7e0 tx=0x7f0ec800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec800a640 con 0x7f0ed015ae80 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec800a7c0 con 0x7f0ed015ae80 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec800ab70 con 0x7f0ed015ae80 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.306+0000 7f0ed5e00640 1 -- 172.21.15.2:0/2934529244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed0159e10 con 0x7f0ed015ae80 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.307+0000 7f0ed5e00640 1 -- 172.21.15.2:0/2934529244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed0157530 con 0x7f0ed015ae80 2024-01-15T18:19:04.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.308+0000 7f0ed5e00640 1 -- 172.21.15.2:0/2934529244 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed015a330 con 0x7f0ed015ae80 2024-01-15T18:19:04.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.309+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0ec800acd0 con 0x7f0ed015ae80 2024-01-15T18:19:04.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.309+0000 7f0ec7400640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806c580 0x7f0ea806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:04.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.310+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f0ec8088010 con 0x7f0ed015ae80 2024-01-15T18:19:04.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.310+0000 7f0ecea00640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806c580 0x7f0ea806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:04.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.311+0000 7f0ecea00640 1 --2- 172.21.15.2:0/2934529244 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0ea806c580 0x7f0ea806ea40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0036000 tx=0x7f0ec003c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:04.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.314+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ed015a330 con 0x7f0ed015ae80 2024-01-15T18:19:04.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.542+0000 7f0ed5e00640 1 -- 172.21.15.2:0/2934529244 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ed01599c0 con 0x7f0ea806c580 2024-01-15T18:19:04.559 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:04.555+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:04.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:04.556+0000 7f0ec7400640 1 -- 172.21.15.2:0/2934529244 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ed01599c0 con 0x7f0ea806c580 2024-01-15T18:19:04.559 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:04.560 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:04.764 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:04.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:06.575 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:06.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.809+0000 7f5ede200640 1 Processor -- start 2024-01-15T18:19:06.813 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.809+0000 7f5ede200640 1 -- start start 2024-01-15T18:19:06.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5ede200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5ed806df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5ede200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed806c500 con 0x7f5ed80ae260 2024-01-15T18:19:06.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5ede200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ed80723f0 con 0x7f5ed80704c0 2024-01-15T18:19:06.817 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5ede200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed815c4b0 con 0x7f5ed806da80 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5edce00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5ed80ae260 0x7f5ed8071ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57756/0 (socket says 172.21.15.2:57756) 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5edce00640 1 -- 172.21.15.2:0/3479521252 learned_addr learned my addr 172.21.15.2:0/3479521252 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.811+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851879063 0 0) 0x7f5ec8001490 con 0x7f5ed80ae260 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8003520 con 0x7f5ed80ae260 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5ed806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099908343 0 0) 0x7f5eb8003520 con 0x7f5ed80ae260 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 msgr2=0x7f5ed806df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5ed806df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 >> v1:172.21.15.137:6789/0 conn(0x7f5ed80704c0 legacy=0x7f5ed8070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed815b8a0 con 0x7f5ed80ae260 2024-01-15T18:19:06.818 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5ec8001b80 con 0x7f5ed80ae260 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 >> v1:172.21.15.2:6789/0 conn(0x7f5ed80ae260 legacy=0x7f5ed8071ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5eb80039a0 0x7f5eb8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5eb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5eb8005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed815c4b0 con 0x7f5eb80047c0 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb800bb40 con 0x7f5eb80039a0 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb800bd00 con 0x7f5ed806da80 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.812+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5eb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5edd800640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5eb8005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5edce00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5eb80039a0 0x7f5eb8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5ecfc00640 1 -- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 msgr2=0x7f5eb8005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5eb8005250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5ecfc00640 1 -- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5eb80039a0 msgr2=0x7f5eb8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5eb80039a0 0x7f5eb8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5ecfc00640 1 -- 172.21.15.2:0/3479521252 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8003520 con 0x7f5eb80047c0 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5edd800640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed806da80 0x7f5eb8005250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.813+0000 7f5edce00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5eb80039a0 0x7f5eb8003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:06.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.814+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5eb8004c40 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000ba30 tx=0x7f5eb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fb6d5b7ba4151b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:06.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.814+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0032640 con 0x7f5eb80047c0 2024-01-15T18:19:06.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.814+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec00327a0 con 0x7f5eb80047c0 2024-01-15T18:19:06.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.814+0000 7f5ed7e00640 1 -- 172.21.15.2:0/3479521252 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0032b30 con 0x7f5eb80047c0 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5ede200640 1 -- 172.21.15.2:0/3479521252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 msgr2=0x7f5eb8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5ede200640 1 --2- 172.21.15.2:0/3479521252 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5eb8004c40 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000ba30 tx=0x7f5eb80057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5edd800640 1 -- 172.21.15.2:0/3479521252 reap_dead start 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5ede200640 1 -- 172.21.15.2:0/3479521252 shutdown_connections 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5ede200640 1 -- 172.21.15.2:0/3479521252 >> 172.21.15.2:0/3479521252 conn(0x7f5ed80cebc0 msgr2=0x7f5ed80cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5ede200640 1 -- 172.21.15.2:0/3479521252 shutdown_connections 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.815+0000 7f5ede200640 1 -- 172.21.15.2:0/3479521252 wait complete. 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.817+0000 7f5ede200640 1 Processor -- start 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.817+0000 7f5ede200640 1 -- start start 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5ede200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5ed80ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5ede200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed809b910 0x7f5ed809bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5ede200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ed8156150 0x7f5ed81565d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5ede200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed80b3660 con 0x7f5eb80047c0 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5ede200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed809c2d0 con 0x7f5ed8156150 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5ede200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed8156b10 con 0x7f5ed809b910 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5edce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5ed80ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5edce00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5ed80ca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55906/0 (socket says 172.21.15.2:55906) 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.818+0000 7f5edce00640 1 -- 172.21.15.2:0/2508891136 learned_addr learned my addr 172.21.15.2:0/2508891136 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edd800640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ed8156150 0x7f5ed81565d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed809b910 0x7f5ed809bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edce00640 1 -- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed809b910 msgr2=0x7f5ed809bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:06.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edce00640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5ed809b910 0x7f5ed809bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edce00640 1 -- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ed8156150 msgr2=0x7f5ed81565d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edce00640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ed8156150 0x7f5ed81565d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edce00640 1 -- 172.21.15.2:0/2508891136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed80b3870 con 0x7f5eb80047c0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edd800640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5ed8156150 0x7f5ed81565d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5edce00640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5eb80047c0 0x7f5ed80ca070 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8001350 tx=0x7f5ec800b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec800be10 con 0x7f5eb80047c0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.819+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec80070d0 con 0x7f5eb80047c0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.820+0000 7f5ede200640 1 -- 172.21.15.2:0/2508891136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed80b3b50 con 0x7f5eb80047c0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.820+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8007440 con 0x7f5eb80047c0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.820+0000 7f5ede200640 1 -- 172.21.15.2:0/2508891136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed80b4110 con 0x7f5eb80047c0 2024-01-15T18:19:06.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.821+0000 7f5ede200640 1 -- 172.21.15.2:0/2508891136 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed8074770 con 0x7f5eb80047c0 2024-01-15T18:19:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.822+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f5ec80075a0 con 0x7f5eb80047c0 2024-01-15T18:19:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.823+0000 7f5ed5600640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5eb006c580 0x7f5eb006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.823+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f5ec8088ef0 con 0x7f5eb80047c0 2024-01-15T18:19:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.823+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5eb006c580 0x7f5eb006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:06.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.825+0000 7f5ecfc00640 1 --2- 172.21.15.2:0/2508891136 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5eb006c580 0x7f5eb006ea40 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000ba00 tx=0x7f5ec003d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:06.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:06.827+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ec8055430 con 0x7f5eb80047c0 2024-01-15T18:19:07.058 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:07.055+0000 7f5ede200640 1 -- 172.21.15.2:0/2508891136 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ed815c140 con 0x7f5eb006c580 2024-01-15T18:19:07.075 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:07.071+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:07.075 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:07.072+0000 7f5ed5600640 1 -- 172.21.15.2:0/2508891136 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5ec0032a90 con 0x7f5eb006c580 2024-01-15T18:19:07.075 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:07.076 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:09.093 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:09.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.320+0000 7fef74c00640 1 Processor -- start 2024-01-15T18:19:09.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.320+0000 7fef74c00640 1 -- start start 2024-01-15T18:19:09.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef74c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fef70144b90 0x7fef700b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef74c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef70070430 con 0x7fef7015b840 2024-01-15T18:19:09.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef74c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef700b4760 con 0x7fef7006e830 2024-01-15T18:19:09.325 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef74c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef70144f70 con 0x7fef70144b90 2024-01-15T18:19:09.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef6de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fef70144b90 0x7fef700b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef6de00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fef70144b90 0x7fef700b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55908/0 (socket says 172.21.15.2:55908) 2024-01-15T18:19:09.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.322+0000 7fef6de00640 1 -- 172.21.15.2:0/658556639 learned_addr learned my addr 172.21.15.2:0/658556639 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:09.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.323+0000 7fef6de00640 1 -- 172.21.15.2:0/658556639 >> v1:172.21.15.137:6789/0 conn(0x7fef7006e830 legacy=0x7fef700d0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:09.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.323+0000 7fef6de00640 1 -- 172.21.15.2:0/658556639 >> v1:172.21.15.2:6789/0 conn(0x7fef7015b840 legacy=0x7fef7006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.326 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.323+0000 7fef6de00640 1 -- 172.21.15.2:0/658556639 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef70070690 con 0x7fef70144b90 2024-01-15T18:19:09.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.324+0000 7fef6de00640 1 --2- 172.21.15.2:0/658556639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef70144b90 0x7fef700b4220 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fef60009870 tx=0x7fef6002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eafdc2ddc81906d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.324+0000 7fef6ca00640 1 -- 172.21.15.2:0/658556639 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef60030850 con 0x7fef70144b90 2024-01-15T18:19:09.327 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.324+0000 7fef6ca00640 1 -- 172.21.15.2:0/658556639 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef600309b0 con 0x7fef70144b90 2024-01-15T18:19:09.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.324+0000 7fef6ca00640 1 -- 172.21.15.2:0/658556639 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef60030d40 con 0x7fef70144b90 2024-01-15T18:19:09.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 -- 172.21.15.2:0/658556639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef70144b90 msgr2=0x7fef700b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.328 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 --2- 172.21.15.2:0/658556639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef70144b90 0x7fef700b4220 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fef60009870 tx=0x7fef6002b990 comp rx=0 tx=0).stop 2024-01-15T18:19:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 -- 172.21.15.2:0/658556639 shutdown_connections 2024-01-15T18:19:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 --2- 172.21.15.2:0/658556639 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef70144b90 0x7fef700b4220 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 -- 172.21.15.2:0/658556639 >> 172.21.15.2:0/658556639 conn(0x7fef700b2c10 msgr2=0x7fef700b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 -- 172.21.15.2:0/658556639 shutdown_connections 2024-01-15T18:19:09.329 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.325+0000 7fef74c00640 1 -- 172.21.15.2:0/658556639 wait complete. 2024-01-15T18:19:09.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.327+0000 7fef74c00640 1 Processor -- start 2024-01-15T18:19:09.330 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.327+0000 7fef74c00640 1 -- start start 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef74c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef7006e830 0x7fef7013e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef74c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef70144b90 0x7fef7015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef74c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef7015b840 0x7fef7006cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef74c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7013e940 con 0x7fef7015b840 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef74c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7015cfd0 con 0x7fef70144b90 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef74c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef7015ca10 con 0x7fef7006e830 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef6e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef7015b840 0x7fef7006cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef6e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef7015b840 0x7fef7006cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55910/0 (socket says 172.21.15.2:55910) 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.328+0000 7fef6e800640 1 -- 172.21.15.2:0/373530612 learned_addr learned my addr 172.21.15.2:0/373530612 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6d400640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef70144b90 0x7fef7015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6de00640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef7006e830 0x7fef7013e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.333 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6e800640 1 -- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef7006e830 msgr2=0x7fef7013e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6e800640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fef7006e830 0x7fef7013e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6e800640 1 -- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef70144b90 msgr2=0x7fef7015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6e800640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef70144b90 0x7fef7015d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6e800640 1 -- 172.21.15.2:0/373530612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef600094d0 con 0x7fef7015b840 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6d400640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fef70144b90 0x7fef7015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef6e800640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fef7015b840 0x7fef7006cb90 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7fef6800c780 tx=0x7fef6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68003d80 con 0x7fef7015b840 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6800ce90 con 0x7fef7015b840 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef680052b0 con 0x7fef7015b840 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.329+0000 7fef74c00640 1 -- 172.21.15.2:0/373530612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7015ccf0 con 0x7fef7015b840 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.330+0000 7fef74c00640 1 -- 172.21.15.2:0/373530612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef701f0160 con 0x7fef7015b840 2024-01-15T18:19:09.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.331+0000 7fef74c00640 1 -- 172.21.15.2:0/373530612 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef700686f0 con 0x7fef7015b840 2024-01-15T18:19:09.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.332+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fef68005410 con 0x7fef7015b840 2024-01-15T18:19:09.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.333+0000 7fef66000640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4806c550 0x7fef4806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:09.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.333+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fef68088060 con 0x7fef7015b840 2024-01-15T18:19:09.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.334+0000 7fef6de00640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4806c550 0x7fef4806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:09.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.335+0000 7fef6de00640 1 --2- 172.21.15.2:0/373530612 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fef4806c550 0x7fef4806ea10 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fef60036000 tx=0x7fef6003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:09.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.338+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef680545a0 con 0x7fef7015b840 2024-01-15T18:19:09.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.565+0000 7fef74c00640 1 -- 172.21.15.2:0/373530612 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef700cfd10 con 0x7fef4806c550 2024-01-15T18:19:09.581 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:09.579+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:09.579+0000 7fef66000640 1 -- 172.21.15.2:0/373530612 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fef60030d30 con 0x7fef4806c550 2024-01-15T18:19:09.582 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:09.582 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:11.601 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:11.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.831+0000 7f3866c00640 1 Processor -- start 2024-01-15T18:19:11.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.831+0000 7f3866c00640 1 -- start start 2024-01-15T18:19:11.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.832+0000 7f3866c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f38600d1d40 0x7f38600cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:11.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3866c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38600d0af0 con 0x7f38600b3fa0 2024-01-15T18:19:11.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3866c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38600ce7b0 con 0x7f38601609d0 2024-01-15T18:19:11.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3866c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38600d08e0 con 0x7f38600d1d40 2024-01-15T18:19:11.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3866200640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f38600b3fa0 0x7f38600d01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:57768/0 (socket says 172.21.15.2:57768) 2024-01-15T18:19:11.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3864e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f38600d1d40 0x7f38600cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:11.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3866200640 1 -- 172.21.15.2:0/888199595 learned_addr learned my addr 172.21.15.2:0/888199595 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:11.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f385fe00640 1 -- 172.21.15.2:0/888199595 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754707781 0 0) 0x7f38600d0af0 con 0x7f38600b3fa0 2024-01-15T18:19:11.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f385fe00640 1 -- 172.21.15.2:0/888199595 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3840003520 con 0x7f38600b3fa0 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3864e00640 1 -- 172.21.15.2:0/888199595 >> v1:172.21.15.137:6789/0 conn(0x7f38601609d0 legacy=0x7f38600cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3864e00640 1 -- 172.21.15.2:0/888199595 >> v1:172.21.15.2:6789/0 conn(0x7f38600b3fa0 legacy=0x7f38600d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.833+0000 7f3864e00640 1 -- 172.21.15.2:0/888199595 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38600cdbe0 con 0x7f38600d1d40 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.835+0000 7f3864e00640 1 --2- 172.21.15.2:0/888199595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600d1d40 0x7f38600cec40 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f38480097a0 tx=0x7f384802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8131528d95829577 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.835+0000 7f385fe00640 1 -- 172.21.15.2:0/888199595 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848030610 con 0x7f38600d1d40 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.835+0000 7f385fe00640 1 -- 172.21.15.2:0/888199595 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3848030770 con 0x7f38600d1d40 2024-01-15T18:19:11.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.835+0000 7f385fe00640 1 -- 172.21.15.2:0/888199595 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848030b00 con 0x7f38600d1d40 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 -- 172.21.15.2:0/888199595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600d1d40 msgr2=0x7f38600cec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 --2- 172.21.15.2:0/888199595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600d1d40 0x7f38600cec40 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f38480097a0 tx=0x7f384802b960 comp rx=0 tx=0).stop 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 -- 172.21.15.2:0/888199595 shutdown_connections 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 --2- 172.21.15.2:0/888199595 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600d1d40 0x7f38600cec40 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 -- 172.21.15.2:0/888199595 >> 172.21.15.2:0/888199595 conn(0x7f386009c3e0 msgr2=0x7f38600d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 -- 172.21.15.2:0/888199595 shutdown_connections 2024-01-15T18:19:11.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.836+0000 7f3866c00640 1 -- 172.21.15.2:0/888199595 wait complete. 2024-01-15T18:19:11.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.837+0000 7f3866c00640 1 Processor -- start 2024-01-15T18:19:11.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.838+0000 7f3866c00640 1 -- start start 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600b3fa0 0x7f38600b8810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38600d1d40 0x7f38600b7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38601609d0 0x7f386015d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600afe20 con 0x7f38600b3fa0 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38600b6e80 con 0x7f38600d1d40 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38600b8190 con 0x7f38601609d0 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600b3fa0 0x7f38600b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600b3fa0 0x7f38600b8810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55928/0 (socket says 172.21.15.2:55928) 2024-01-15T18:19:11.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 -- 172.21.15.2:0/2857449984 learned_addr learned my addr 172.21.15.2:0/2857449984 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3864e00640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38600d1d40 0x7f38600b7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3866200640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38601609d0 0x7f386015d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 -- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38601609d0 msgr2=0x7f386015d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f38601609d0 0x7f386015d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 -- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38600d1d40 msgr2=0x7f38600b7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38600d1d40 0x7f38600b7b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.839+0000 7f3865800640 1 -- 172.21.15.2:0/2857449984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848009400 con 0x7f38600b3fa0 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f3864e00640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f38600d1d40 0x7f38600b7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:11.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f3865800640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f38600b3fa0 0x7f38600b8810 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f385000c430 tx=0x7f385000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:11.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38500052d0 con 0x7f38600b3fa0 2024-01-15T18:19:11.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3850005430 con 0x7f38600b3fa0 2024-01-15T18:19:11.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850003830 con 0x7f38600b3fa0 2024-01-15T18:19:11.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f3866c00640 1 -- 172.21.15.2:0/2857449984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38600b8470 con 0x7f38600b3fa0 2024-01-15T18:19:11.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.840+0000 7f3866c00640 1 -- 172.21.15.2:0/2857449984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38600b0350 con 0x7f38600b3fa0 2024-01-15T18:19:11.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.841+0000 7f3866c00640 1 -- 172.21.15.2:0/2857449984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3860074770 con 0x7f38600b3fa0 2024-01-15T18:19:11.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.842+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3850003990 con 0x7f38600b3fa0 2024-01-15T18:19:11.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.844+0000 7f385d600640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383806c550 0x7f383806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:11.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.844+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f38500880f0 con 0x7f38600b3fa0 2024-01-15T18:19:11.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.845+0000 7f3864e00640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383806c550 0x7f383806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:11.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.846+0000 7f3864e00640 1 --2- 172.21.15.2:0/2857449984 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f383806c550 0x7f383806ea10 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f3848036000 tx=0x7f384803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:11.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:11.848+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3850054630 con 0x7f38600b3fa0 2024-01-15T18:19:12.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:12.081+0000 7f3866c00640 1 -- 172.21.15.2:0/2857449984 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f38600ce440 con 0x7f383806c550 2024-01-15T18:19:12.100 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:12.098+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:12.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:12.097+0000 7f385d600640 1 -- 172.21.15.2:0/2857449984 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3848030af0 con 0x7f383806c550 2024-01-15T18:19:12.101 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:12.101 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:12.116 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:14.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:14.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.351+0000 7f29cc800640 1 Processor -- start 2024-01-15T18:19:14.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.351+0000 7f29cc800640 1 -- start start 2024-01-15T18:19:14.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29cc800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f29c4144c00 0x7f29c40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:14.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29cc800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c4070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29cc800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c40ce230 con 0x7f29c40cdd70 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29cc800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c40ce510 con 0x7f29c4144c00 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29cc800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40ce7f0 con 0x7f29c406d000 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29ca800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c4070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29ca800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c4070b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:48284/0 (socket says 172.21.15.2:48284) 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.353+0000 7f29ca800640 1 -- 172.21.15.2:0/2645018563 learned_addr learned my addr 172.21.15.2:0/2645018563 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.354+0000 7f29c9e00640 1 --2- 172.21.15.2:0/2645018563 >> v2:172.21.15.137:3300/0 conn(0x7f29c4144c00 0x7f29c40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.354+0000 7f29ca800640 1 -- 172.21.15.2:0/2645018563 >> v2:172.21.15.137:3300/0 conn(0x7f29c4144c00 msgr2=0x7f29c40cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.354+0000 7f29ca800640 1 --2- 172.21.15.2:0/2645018563 >> v2:172.21.15.137:3300/0 conn(0x7f29c4144c00 0x7f29c40cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:14.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.354+0000 7f29ca800640 1 -- 172.21.15.2:0/2645018563 >> v1:172.21.15.137:6789/0 conn(0x7f29c40cdd70 legacy=0x7f29c406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:14.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.354+0000 7f29ca800640 1 -- 172.21.15.2:0/2645018563 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40705f0 con 0x7f29c406d000 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.355+0000 7f29ca800640 1 --2- 172.21.15.2:0/2645018563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c4070b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009e50 tx=0x7f29bc02bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be97f0d0d829601b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.356+0000 7f29c8a00640 1 -- 172.21.15.2:0/2645018563 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc02caf0 con 0x7f29c406d000 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.356+0000 7f29c8a00640 1 -- 172.21.15.2:0/2645018563 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc02cc50 con 0x7f29c406d000 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.356+0000 7f29c8a00640 1 -- 172.21.15.2:0/2645018563 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0343f0 con 0x7f29c406d000 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 -- 172.21.15.2:0/2645018563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 msgr2=0x7f29c4070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 --2- 172.21.15.2:0/2645018563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c4070b10 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f29bc009e50 tx=0x7f29bc02bf50 comp rx=0 tx=0).stop 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 -- 172.21.15.2:0/2645018563 shutdown_connections 2024-01-15T18:19:14.361 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 --2- 172.21.15.2:0/2645018563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c4070b10 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:14.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 --2- 172.21.15.2:0/2645018563 >> v2:172.21.15.137:3300/0 conn(0x7f29c4144c00 0x7f29c40cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:14.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 -- 172.21.15.2:0/2645018563 >> 172.21.15.2:0/2645018563 conn(0x7f29c40b45f0 msgr2=0x7f29c40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:14.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 -- 172.21.15.2:0/2645018563 shutdown_connections 2024-01-15T18:19:14.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.357+0000 7f29cc800640 1 -- 172.21.15.2:0/2645018563 wait complete. 2024-01-15T18:19:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.359+0000 7f29cc800640 1 Processor -- start 2024-01-15T18:19:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.359+0000 7f29cc800640 1 -- start start 2024-01-15T18:19:14.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29cc800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c40b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29cc800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c40cdd70 0x7f29c41f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29cc800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29c4144c00 0x7f29c41f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29cc800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40c3e60 con 0x7f29c40cdd70 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29cc800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40c4350 con 0x7f29c4144c00 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29cc800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40c4070 con 0x7f29c406d000 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29c9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c40cdd70 0x7f29c41f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29c9400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c40cdd70 0x7f29c41f2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38110/0 (socket says 172.21.15.2:38110) 2024-01-15T18:19:14.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29c9400640 1 -- 172.21.15.2:0/2567961493 learned_addr learned my addr 172.21.15.2:0/2567961493 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29c9e00640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c40b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.360+0000 7f29ca800640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29c4144c00 0x7f29c41f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c9400640 1 -- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 msgr2=0x7f29c40b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c9400640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f29c406d000 0x7f29c40b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c9400640 1 -- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29c4144c00 msgr2=0x7f29c41f2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c9400640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f29c4144c00 0x7f29c41f2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c9400640 1 -- 172.21.15.2:0/2567961493 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc009b30 con 0x7f29c40cdd70 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c9400640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f29c40cdd70 0x7f29c41f2920 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f29ac00c400 tx=0x7f29ac00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:14.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac0052d0 con 0x7f29c40cdd70 2024-01-15T18:19:14.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac005450 con 0x7f29c40cdd70 2024-01-15T18:19:14.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29cc800640 1 -- 172.21.15.2:0/2567961493 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c40afed0 con 0x7f29c40cdd70 2024-01-15T18:19:14.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ac007260 con 0x7f29c40cdd70 2024-01-15T18:19:14.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.361+0000 7f29cc800640 1 -- 172.21.15.2:0/2567961493 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c40b0490 con 0x7f29c40cdd70 2024-01-15T18:19:14.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.362+0000 7f29cc800640 1 -- 172.21.15.2:0/2567961493 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c40b06b0 con 0x7f29c40cdd70 2024-01-15T18:19:14.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.364+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f29ac0073c0 con 0x7f29c40cdd70 2024-01-15T18:19:14.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.365+0000 7f29c2000640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f299c06c550 0x7f299c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:14.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.365+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f29ac087f90 con 0x7f29c40cdd70 2024-01-15T18:19:14.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.366+0000 7f29c9e00640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f299c06c550 0x7f299c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:14.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.367+0000 7f29c9e00640 1 --2- 172.21.15.2:0/2567961493 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f299c06c550 0x7f299c06ea10 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006fd0 tx=0x7f29b4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:14.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.370+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29ac0544d0 con 0x7f29c40cdd70 2024-01-15T18:19:14.612 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.608+0000 7f29cc800640 1 -- 172.21.15.2:0/2567961493 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29c40686f0 con 0x7f299c06c550 2024-01-15T18:19:14.625 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:14.622+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:14.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:14.623+0000 7f29c2000640 1 -- 172.21.15.2:0/2567961493 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f29c40686f0 con 0x7f299c06c550 2024-01-15T18:19:14.626 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:14.626 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:16.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:16.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.881+0000 7f65b3200640 1 Processor -- start 2024-01-15T18:19:16.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.881+0000 7f65b3200640 1 -- start start 2024-01-15T18:19:16.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.882+0000 7f65b3200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f65b40cec40 0x7f65b40b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.882+0000 7f65b3200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40d0af0 con 0x7f65b41609d0 2024-01-15T18:19:16.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.882+0000 7f65b3200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b40ce7b0 con 0x7f65b40d1d40 2024-01-15T18:19:16.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.882+0000 7f65b3200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b40d08e0 con 0x7f65b40cec40 2024-01-15T18:19:16.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.882+0000 7f65b1e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f65b41609d0 0x7f65b40cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:37858/0 (socket says 172.21.15.2:37858) 2024-01-15T18:19:16.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.882+0000 7f65b1e00640 1 -- 172.21.15.2:0/1064488958 learned_addr learned my addr 172.21.15.2:0/1064488958 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870984291 0 0) 0x7f65b40d0af0 con 0x7f65b41609d0 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6594003520 con 0x7f65b41609d0 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b2800640 1 --2- 172.21.15.2:0/1064488958 >> v2:172.21.15.137:3300/0 conn(0x7f65b40cec40 0x7f65b40b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3517091081 0 0) 0x7f6594003520 con 0x7f65b41609d0 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 >> v2:172.21.15.137:3300/0 conn(0x7f65b40cec40 msgr2=0x7f65b40b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 --2- 172.21.15.2:0/1064488958 >> v2:172.21.15.137:3300/0 conn(0x7f65b40cec40 0x7f65b40b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 >> v1:172.21.15.137:6789/0 conn(0x7f65b40d1d40 legacy=0x7f65b40d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40cdbe0 con 0x7f65b41609d0 2024-01-15T18:19:16.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f65a4001670 con 0x7f65b41609d0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 >> v1:172.21.15.2:6789/0 conn(0x7f65b41609d0 legacy=0x7f65b40cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65940039c0 0x7f6594003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65940047c0 0x7f6594004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.883+0000 7f65b0a00640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f659400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40d08e0 con 0x7f65940051d0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659400c120 con 0x7f65940047c0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659400c2e0 con 0x7f65940039c0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b2800640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f659400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b1e00640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65940039c0 0x7f6594003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b1400640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65940047c0 0x7f6594004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b2800640 1 -- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65940039c0 msgr2=0x7f6594003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b2800640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65940039c0 0x7f6594003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b2800640 1 -- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65940047c0 msgr2=0x7f6594004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b2800640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65940047c0 0x7f6594004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b2800640 1 -- 172.21.15.2:0/1064488958 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6594003520 con 0x7f65940051d0 2024-01-15T18:19:16.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.884+0000 7f65b1400640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65940047c0 0x7f6594004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:16.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.885+0000 7f65b2800640 1 -- 172.21.15.2:0/1064488958 reap_dead start 2024-01-15T18:19:16.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.885+0000 7f65b2800640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f659400b980 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f65ac007360 tx=0x7f65940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c179a7de2e14771e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:16.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac02ccf0 con 0x7f65940051d0 2024-01-15T18:19:16.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac02ce90 con 0x7f65940051d0 2024-01-15T18:19:16.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b0a00640 1 -- 172.21.15.2:0/1064488958 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac003bd0 con 0x7f65940051d0 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 -- 172.21.15.2:0/1064488958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 msgr2=0x7f659400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f659400b980 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f65ac007360 tx=0x7f65940057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 -- 172.21.15.2:0/1064488958 shutdown_connections 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 --2- 172.21.15.2:0/1064488958 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f659400b980 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 -- 172.21.15.2:0/1064488958 >> 172.21.15.2:0/1064488958 conn(0x7f65b409c3e0 msgr2=0x7f65b40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 -- 172.21.15.2:0/1064488958 shutdown_connections 2024-01-15T18:19:16.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.886+0000 7f65b3200640 1 -- 172.21.15.2:0/1064488958 wait complete. 2024-01-15T18:19:16.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.888+0000 7f65b3200640 1 Processor -- start 2024-01-15T18:19:16.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.888+0000 7f65b3200640 1 -- start start 2024-01-15T18:19:16.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.889+0000 7f65b3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f65b415b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.889+0000 7f65b3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65b40caed0 0x7f65b40cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.889+0000 7f65b3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65b40c6c00 0x7f65b40c7080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.889+0000 7f65b3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40ca1a0 con 0x7f65940051d0 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.889+0000 7f65b3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40caa40 con 0x7f65b40c6c00 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.889+0000 7f65b3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b40c9a60 con 0x7f65b40caed0 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f65b415b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f65b415b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38142/0 (socket says 172.21.15.2:38142) 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 -- 172.21.15.2:0/2041563984 learned_addr learned my addr 172.21.15.2:0/2041563984 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1400640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65b40caed0 0x7f65b40cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 -- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65b40caed0 msgr2=0x7f65b40cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f65b40caed0 0x7f65b40cb350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 -- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65b40c6c00 msgr2=0x7f65b40c7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f65b40c6c00 0x7f65b40c7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 -- 172.21.15.2:0/2041563984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b40ca3f0 con 0x7f65940051d0 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.890+0000 7f65b1e00640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f65940051d0 0x7f65b415b780 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f65a400e700 tx=0x7f65a400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:16.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.891+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400c620 con 0x7f65940051d0 2024-01-15T18:19:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.891+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a400c780 con 0x7f65940051d0 2024-01-15T18:19:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.891+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a400cb30 con 0x7f65940051d0 2024-01-15T18:19:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.891+0000 7f65b3200640 1 -- 172.21.15.2:0/2041563984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b40ca6d0 con 0x7f65940051d0 2024-01-15T18:19:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.891+0000 7f65b3200640 1 -- 172.21.15.2:0/2041563984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b40cb9c0 con 0x7f65940051d0 2024-01-15T18:19:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.892+0000 7f65b3200640 1 -- 172.21.15.2:0/2041563984 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4074770 con 0x7f65940051d0 2024-01-15T18:19:16.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.893+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f65a400cc90 con 0x7f65940051d0 2024-01-15T18:19:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.894+0000 7f65aa000640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f658c06c550 0x7f658c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.894+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f65a4088a00 con 0x7f65940051d0 2024-01-15T18:19:16.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.894+0000 7f65b1400640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f658c06c550 0x7f658c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:16.899 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.896+0000 7f65b1400640 1 --2- 172.21.15.2:0/2041563984 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f658c06c550 0x7f658c06ea10 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f65b41f62d0 tx=0x7f659c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:16.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:16.898+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65a4054fa0 con 0x7f65940051d0 2024-01-15T18:19:17.128 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:17.125+0000 7f65b3200640 1 -- 172.21.15.2:0/2041563984 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f65b40ce440 con 0x7f658c06c550 2024-01-15T18:19:17.141 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:17.139+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:17.142 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:17.139+0000 7f65aa000640 1 -- 172.21.15.2:0/2041563984 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f659c00d040 con 0x7f658c06c550 2024-01-15T18:19:17.142 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:17.142 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:19.153 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.389+0000 7f7b8fe00640 1 Processor -- start 2024-01-15T18:19:19.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.390+0000 7f7b8fe00640 1 -- start start 2024-01-15T18:19:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.391+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 0x7f7b900d2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.391+0000 7f7b8fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 0x7f7b900d32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.391+0000 7f7b8fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b900d3b10 con 0x7f7b9015d2f0 2024-01-15T18:19:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.391+0000 7f7b8fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b900afd70 con 0x7f7b900d1df0 2024-01-15T18:19:19.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.391+0000 7f7b8fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b900b0050 con 0x7f7b9015d6d0 2024-01-15T18:19:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.392+0000 7f7b8e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 0x7f7b900d2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.392+0000 7f7b8f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 0x7f7b900d32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.392+0000 7f7b8e000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 0x7f7b900d2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:48312/0 (socket says 172.21.15.2:48312) 2024-01-15T18:19:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.392+0000 7f7b8f400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 0x7f7b900d32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3300/0 says I am v2:172.21.15.2:37612/0 (socket says 172.21.15.2:37612) 2024-01-15T18:19:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.392+0000 7f7b8e000640 1 -- 172.21.15.2:0/1336676035 learned_addr learned my addr 172.21.15.2:0/1336676035 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:19.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.392+0000 7f7b8ea00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f7b9015d2f0 0x7f7b900d16e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:42692/0 (socket says 172.21.15.2:42692) 2024-01-15T18:19:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.393+0000 7f7b8e000640 1 -- 172.21.15.2:0/1336676035 >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 msgr2=0x7f7b900d32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.393+0000 7f7b8e000640 1 --2- 172.21.15.2:0/1336676035 >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 0x7f7b900d32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.393+0000 7f7b8e000640 1 -- 172.21.15.2:0/1336676035 >> v1:172.21.15.137:6789/0 conn(0x7f7b9015d2f0 legacy=0x7f7b900d16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.393+0000 7f7b8e000640 1 -- 172.21.15.2:0/1336676035 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900b0330 con 0x7f7b9015d6d0 2024-01-15T18:19:19.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.393+0000 7f7b8f400640 1 --2- 172.21.15.2:0/1336676035 >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 0x7f7b900d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.395+0000 7f7b8e000640 1 --2- 172.21.15.2:0/1336676035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 0x7f7b900d2db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b7802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bafd866749129d6a server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.395+0000 7f7b8d600640 1 -- 172.21.15.2:0/1336676035 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78030840 con 0x7f7b9015d6d0 2024-01-15T18:19:19.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.395+0000 7f7b8d600640 1 -- 172.21.15.2:0/1336676035 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b780309a0 con 0x7f7b9015d6d0 2024-01-15T18:19:19.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.395+0000 7f7b8d600640 1 -- 172.21.15.2:0/1336676035 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b78030d30 con 0x7f7b9015d6d0 2024-01-15T18:19:19.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1336676035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 msgr2=0x7f7b900d2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 --2- 172.21.15.2:0/1336676035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 0x7f7b900d2db0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7b78009870 tx=0x7f7b7802b980 comp rx=0 tx=0).stop 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1336676035 shutdown_connections 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 --2- 172.21.15.2:0/1336676035 >> v2:172.21.15.137:3300/0 conn(0x7f7b900d1df0 0x7f7b900d32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 --2- 172.21.15.2:0/1336676035 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d6d0 0x7f7b900d2db0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1336676035 >> 172.21.15.2:0/1336676035 conn(0x7f7b9008cac0 msgr2=0x7f7b9009c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1336676035 shutdown_connections 2024-01-15T18:19:19.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.396+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1336676035 wait complete. 2024-01-15T18:19:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.398+0000 7f7b8fe00640 1 Processor -- start 2024-01-15T18:19:19.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.398+0000 7f7b8fe00640 1 -- start start 2024-01-15T18:19:19.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.399+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b900d1df0 0x7f7b90152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:19.402 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.399+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d2f0 0x7f7b900cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.399+0000 7f7b8fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b9015d6d0 0x7f7b900b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.399+0000 7f7b8fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015dab0 con 0x7f7b900d1df0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.399+0000 7f7b8fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015dd90 con 0x7f7b9015d6d0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.399+0000 7f7b8fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015e070 con 0x7f7b9015d2f0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b900d1df0 0x7f7b90152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b900d1df0 0x7f7b90152920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38152/0 (socket says 172.21.15.2:38152) 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 -- 172.21.15.2:0/1690946509 learned_addr learned my addr 172.21.15.2:0/1690946509 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8e000640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d2f0 0x7f7b900cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8f400640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b9015d6d0 0x7f7b900b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 -- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d2f0 msgr2=0x7f7b900cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7b9015d2f0 0x7f7b900cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 -- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b9015d6d0 msgr2=0x7f7b900b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7b9015d6d0 0x7f7b900b6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.400+0000 7f7b8ea00640 1 -- 172.21.15.2:0/1690946509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b88009a70 con 0x7f7b900d1df0 2024-01-15T18:19:19.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.401+0000 7f7b8ea00640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7b900d1df0 0x7f7b90152920 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f7b8000c4d0 tx=0x7f7b8000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:19.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.401+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b80007480 con 0x7f7b900d1df0 2024-01-15T18:19:19.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.401+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b800075e0 con 0x7f7b900d1df0 2024-01-15T18:19:19.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.401+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800049e0 con 0x7f7b900d1df0 2024-01-15T18:19:19.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.401+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1690946509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780094d0 con 0x7f7b900d1df0 2024-01-15T18:19:19.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.401+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1690946509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9015e630 con 0x7f7b900d1df0 2024-01-15T18:19:19.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.402+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1690946509 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90067970 con 0x7f7b900d1df0 2024-01-15T18:19:19.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.403+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f7b80004b40 con 0x7f7b900d1df0 2024-01-15T18:19:19.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.404+0000 7f7b86a00640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7b6806c550 0x7f7b6806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:19.406 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.404+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f7b80087c70 con 0x7f7b900d1df0 2024-01-15T18:19:19.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.405+0000 7f7b8e000640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7b6806c550 0x7f7b6806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:19.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.406+0000 7f7b8e000640 1 --2- 172.21.15.2:0/1690946509 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7b6806c550 0x7f7b6806ea10 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f7b78036000 tx=0x7f7b7803b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:19.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.409+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b80054260 con 0x7f7b900d1df0 2024-01-15T18:19:19.646 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.644+0000 7f7b8fe00640 1 -- 172.21.15.2:0/1690946509 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7b900d3830 con 0x7f7b6806c550 2024-01-15T18:19:19.659 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:19.657+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:19.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:19.658+0000 7f7b86a00640 1 -- 172.21.15.2:0/1690946509 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7b78030ca0 con 0x7f7b6806c550 2024-01-15T18:19:19.660 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:19.660 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:21.674 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:21.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.916+0000 7f57cc000640 1 Processor -- start 2024-01-15T18:19:21.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.916+0000 7f57cc000640 1 -- start start 2024-01-15T18:19:21.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57cc000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f57c415c450 0x7f57c406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:21.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57cc000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c406d290 con 0x7f57c406ce30 2024-01-15T18:19:21.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57cc000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c40ce230 con 0x7f57c4144c00 2024-01-15T18:19:21.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57cc000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c40ce510 con 0x7f57c415c450 2024-01-15T18:19:21.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57ca000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f57c406ce30 0x7f57c4070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:37862/0 (socket says 172.21.15.2:37862) 2024-01-15T18:19:21.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57ca000640 1 -- 172.21.15.2:0/1772771840 learned_addr learned my addr 172.21.15.2:0/1772771840 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:21.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.918+0000 7f57c8c00640 1 --2- 172.21.15.2:0/1772771840 >> v2:172.21.15.2:3300/0 conn(0x7f57c415c450 0x7f57c406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:21.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.919+0000 7f57c3e00640 1 -- 172.21.15.2:0/1772771840 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20849571 0 0) 0x7f57c406d290 con 0x7f57c406ce30 2024-01-15T18:19:21.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.919+0000 7f57c3e00640 1 -- 172.21.15.2:0/1772771840 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a4003520 con 0x7f57c406ce30 2024-01-15T18:19:21.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.919+0000 7f57c8c00640 1 -- 172.21.15.2:0/1772771840 >> v1:172.21.15.137:6789/0 conn(0x7f57c4144c00 legacy=0x7f57c40cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:21.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.919+0000 7f57c8c00640 1 -- 172.21.15.2:0/1772771840 >> v1:172.21.15.2:6789/0 conn(0x7f57c406ce30 legacy=0x7f57c4070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:21.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.919+0000 7f57c8c00640 1 -- 172.21.15.2:0/1772771840 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c40ce7f0 con 0x7f57c415c450 2024-01-15T18:19:21.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.920+0000 7f57c8c00640 1 --2- 172.21.15.2:0/1772771840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c415c450 0x7f57c406c8f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0097a0 tx=0x7f57ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cf0b5d5eadf2e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:21.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.920+0000 7f57c3e00640 1 -- 172.21.15.2:0/1772771840 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac030610 con 0x7f57c415c450 2024-01-15T18:19:21.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.920+0000 7f57c3e00640 1 -- 172.21.15.2:0/1772771840 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac030770 con 0x7f57c415c450 2024-01-15T18:19:21.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.920+0000 7f57c3e00640 1 -- 172.21.15.2:0/1772771840 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac030b00 con 0x7f57c415c450 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 -- 172.21.15.2:0/1772771840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c415c450 msgr2=0x7f57c406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 --2- 172.21.15.2:0/1772771840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c415c450 0x7f57c406c8f0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f57ac0097a0 tx=0x7f57ac02b960 comp rx=0 tx=0).stop 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 -- 172.21.15.2:0/1772771840 shutdown_connections 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 --2- 172.21.15.2:0/1772771840 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c415c450 0x7f57c406c8f0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 -- 172.21.15.2:0/1772771840 >> 172.21.15.2:0/1772771840 conn(0x7f57c40b45f0 msgr2=0x7f57c40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 -- 172.21.15.2:0/1772771840 shutdown_connections 2024-01-15T18:19:21.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.921+0000 7f57cc000640 1 -- 172.21.15.2:0/1772771840 wait complete. 2024-01-15T18:19:21.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.924+0000 7f57cc000640 1 Processor -- start 2024-01-15T18:19:21.926 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.924+0000 7f57cc000640 1 -- start start 2024-01-15T18:19:21.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57cc000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c406ce30 0x7f57c41389d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:21.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57cc000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c4144c00 0x7f57c413ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:21.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57cc000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415c450 0x7f57c4141440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57cc000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c413f5e0 con 0x7f57c4144c00 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57cc000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c413f020 con 0x7f57c406ce30 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57cc000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c413f300 con 0x7f57c415c450 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.925+0000 7f57c8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c4144c00 0x7f57c413ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c4144c00 0x7f57c413ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:38168/0 (socket says 172.21.15.2:38168) 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 -- 172.21.15.2:0/644560234 learned_addr learned my addr 172.21.15.2:0/644560234 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c9600640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c406ce30 0x7f57c41389d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57ca000640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415c450 0x7f57c4141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 -- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415c450 msgr2=0x7f57c4141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f57c415c450 0x7f57c4141440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 -- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c406ce30 msgr2=0x7f57c41389d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f57c406ce30 0x7f57c41389d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:21.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.926+0000 7f57c8c00640 1 -- 172.21.15.2:0/644560234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ac009400 con 0x7f57c4144c00 2024-01-15T18:19:21.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.927+0000 7f57c8c00640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f57c4144c00 0x7f57c413ba60 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f57ac02bf70 tx=0x7f57ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:21.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.927+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac03e070 con 0x7f57c4144c00 2024-01-15T18:19:21.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.927+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57ac0397a0 con 0x7f57c4144c00 2024-01-15T18:19:21.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.927+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57ac039b30 con 0x7f57c4144c00 2024-01-15T18:19:21.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.927+0000 7f57cc000640 1 -- 172.21.15.2:0/644560234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c4140c40 con 0x7f57c4144c00 2024-01-15T18:19:21.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.927+0000 7f57cc000640 1 -- 172.21.15.2:0/644560234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c413bfa0 con 0x7f57c4144c00 2024-01-15T18:19:21.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.928+0000 7f57cc000640 1 -- 172.21.15.2:0/644560234 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c4141160 con 0x7f57c4144c00 2024-01-15T18:19:21.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.929+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f57ac039c90 con 0x7f57c4144c00 2024-01-15T18:19:21.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.930+0000 7f57c1600640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06c550 0x7f579c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:21.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.930+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f57ac0aae00 con 0x7f57c4144c00 2024-01-15T18:19:21.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.931+0000 7f57c9600640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06c550 0x7f579c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:21.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.932+0000 7f57c9600640 1 --2- 172.21.15.2:0/644560234 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f579c06c550 0x7f579c06ea10 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f57b4000f60 tx=0x7f57b4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:21.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:21.935+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57c4141160 con 0x7f57c4144c00 2024-01-15T18:19:22.167 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:22.165+0000 7f57cc000640 1 -- 172.21.15.2:0/644560234 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57c413c130 con 0x7f579c06c550 2024-01-15T18:19:22.181 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:22.179+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:22.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:22.180+0000 7f57c1600640 1 -- 172.21.15.2:0/644560234 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f57b40014a0 con 0x7f579c06c550 2024-01-15T18:19:22.182 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:22.182 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:24.197 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:24.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.436+0000 7f4f0fe00640 1 Processor -- start 2024-01-15T18:19:24.438 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.436+0000 7f4f0fe00640 1 -- start start 2024-01-15T18:19:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.438+0000 7f4f0fe00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4f1006d250 0x7f4f1006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.438+0000 7f4f0fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f100ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.438+0000 7f4f0fe00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f100ce880 con 0x7f4f1006f2b0 2024-01-15T18:19:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.438+0000 7f4f0fe00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1006ae80 con 0x7f4f1006d250 2024-01-15T18:19:24.440 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.438+0000 7f4f0fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f100ca610 con 0x7f4f1006a9f0 2024-01-15T18:19:24.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f100ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:24.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f100ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:50440/0 (socket says 172.21.15.2:50440) 2024-01-15T18:19:24.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 -- 172.21.15.2:0/1662354578 learned_addr learned my addr 172.21.15.2:0/1662354578 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:24.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0ea00640 1 --1- 172.21.15.2:0/1662354578 >> v1:172.21.15.137:6789/0 conn(0x7f4f1006f2b0 0x7f4f10071c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:58260/0 (socket says 172.21.15.2:58260) 2024-01-15T18:19:24.441 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0e000640 1 --2- 172.21.15.2:0/1662354578 >> v2:172.21.15.137:3300/0 conn(0x7f4f1006d250 0x7f4f1006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:24.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0d600640 1 -- 172.21.15.2:0/1662354578 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1858174952 0 0) 0x7f4f100ce880 con 0x7f4f1006f2b0 2024-01-15T18:19:24.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 -- 172.21.15.2:0/1662354578 >> v2:172.21.15.137:3300/0 conn(0x7f4f1006d250 msgr2=0x7f4f1006a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:24.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 --2- 172.21.15.2:0/1662354578 >> v2:172.21.15.137:3300/0 conn(0x7f4f1006d250 0x7f4f1006a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:24.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 -- 172.21.15.2:0/1662354578 >> v1:172.21.15.137:6789/0 conn(0x7f4f1006f2b0 legacy=0x7f4f10071c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:24.442 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.439+0000 7f4f0f400640 1 -- 172.21.15.2:0/1662354578 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f100cdc70 con 0x7f4f1006a9f0 2024-01-15T18:19:24.443 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.441+0000 7f4f0f400640 1 --2- 172.21.15.2:0/1662354578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f100ced10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009870 tx=0x7f4f0802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11ba8dde1685f002 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:24.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.441+0000 7f4f0d600640 1 -- 172.21.15.2:0/1662354578 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08030840 con 0x7f4f1006a9f0 2024-01-15T18:19:24.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.441+0000 7f4f0d600640 1 -- 172.21.15.2:0/1662354578 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f080309a0 con 0x7f4f1006a9f0 2024-01-15T18:19:24.444 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.441+0000 7f4f0d600640 1 -- 172.21.15.2:0/1662354578 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08030d30 con 0x7f4f1006a9f0 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 -- 172.21.15.2:0/1662354578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 msgr2=0x7f4f100ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 --2- 172.21.15.2:0/1662354578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f100ced10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f4f08009870 tx=0x7f4f0802b980 comp rx=0 tx=0).stop 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 -- 172.21.15.2:0/1662354578 shutdown_connections 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 --2- 172.21.15.2:0/1662354578 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f100ced10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 --2- 172.21.15.2:0/1662354578 >> v2:172.21.15.137:3300/0 conn(0x7f4f1006d250 0x7f4f1006a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 -- 172.21.15.2:0/1662354578 >> 172.21.15.2:0/1662354578 conn(0x7f4f100cfd30 msgr2=0x7f4f100d0140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 -- 172.21.15.2:0/1662354578 shutdown_connections 2024-01-15T18:19:24.445 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.442+0000 7f4f0fe00640 1 -- 172.21.15.2:0/1662354578 wait complete. 2024-01-15T18:19:24.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.444+0000 7f4f0fe00640 1 Processor -- start 2024-01-15T18:19:24.446 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.444+0000 7f4f0fe00640 1 -- start start 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.445+0000 7f4f0fe00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f101f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.445+0000 7f4f0fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f1006d250 0x7f4f100b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.445+0000 7f4f0fe00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f1006f2b0 0x7f4f100b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.445+0000 7f4f0fe00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f101f68e0 con 0x7f4f1006d250 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.445+0000 7f4f0fe00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1015d620 con 0x7f4f1006f2b0 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.445+0000 7f4f0fe00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1015d900 con 0x7f4f1006a9f0 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f1006d250 0x7f4f100b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:24.448 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f1006d250 0x7f4f100b4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43450/0 (socket says 172.21.15.2:43450) 2024-01-15T18:19:24.449 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 -- 172.21.15.2:0/2101197995 learned_addr learned my addr 172.21.15.2:0/2101197995 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0ea00640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f101f63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0f400640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f1006f2b0 0x7f4f100b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 -- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 msgr2=0x7f4f101f63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4f1006a9f0 0x7f4f101f63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 -- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f1006f2b0 msgr2=0x7f4f100b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4f1006f2b0 0x7f4f100b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.446+0000 7f4f0e000640 1 -- 172.21.15.2:0/2101197995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080094d0 con 0x7f4f1006d250 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.447+0000 7f4f0e000640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4f1006d250 0x7f4f100b4bd0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f4ef800e870 tx=0x7f4ef800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.447+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef80094f0 con 0x7f4f1006d250 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.447+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef800c940 con 0x7f4f1006d250 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.447+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef800ccf0 con 0x7f4f1006d250 2024-01-15T18:19:24.450 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.447+0000 7f4f0fe00640 1 -- 172.21.15.2:0/2101197995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1015dbe0 con 0x7f4f1006d250 2024-01-15T18:19:24.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.447+0000 7f4f0fe00640 1 -- 172.21.15.2:0/2101197995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1015e1a0 con 0x7f4f1006d250 2024-01-15T18:19:24.451 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.448+0000 7f4f0fe00640 1 -- 172.21.15.2:0/2101197995 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1006b430 con 0x7f4f1006d250 2024-01-15T18:19:24.452 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.450+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f4ef8013470 con 0x7f4f1006d250 2024-01-15T18:19:24.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.450+0000 7f4f07400640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ee806c580 0x7f4ee806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:24.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.451+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f4ef8088950 con 0x7f4f1006d250 2024-01-15T18:19:24.453 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.451+0000 7f4f0ea00640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ee806c580 0x7f4ee806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:24.455 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.452+0000 7f4f0ea00640 1 --2- 172.21.15.2:0/2101197995 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4ee806c580 0x7f4ee806ea40 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4f000012e0 tx=0x7f4f00007ed0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:24.458 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.456+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ef8055180 con 0x7f4f1006d250 2024-01-15T18:19:24.693 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.690+0000 7f4f0fe00640 1 -- 172.21.15.2:0/2101197995 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f100ce510 con 0x7f4ee806c580 2024-01-15T18:19:24.706 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:24.704+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:24.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:24.704+0000 7f4f07400640 1 -- 172.21.15.2:0/2101197995 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4f000079d0 con 0x7f4ee806c580 2024-01-15T18:19:24.707 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:24.707 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:26.721 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:26.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.957+0000 7f1deda00640 1 Processor -- start 2024-01-15T18:19:26.960 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.957+0000 7f1deda00640 1 -- start start 2024-01-15T18:19:26.961 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.959+0000 7f1deda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 0x7f1de80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.959+0000 7f1deda00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1de806d000 0x7f1de8070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.959+0000 7f1deda00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de80ce230 con 0x7f1de80cdd70 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.959+0000 7f1deda00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de80ce510 con 0x7f1de806d000 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.959+0000 7f1deda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de80ce7f0 con 0x7f1de8144c00 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.960+0000 7f1de6a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 0x7f1de80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.960+0000 7f1de7400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1de806d000 0x7f1de8070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:26.962 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.960+0000 7f1de6a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 0x7f1de80cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:50464/0 (socket says 172.21.15.2:50464) 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.960+0000 7f1ddde00640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f1de80cdd70 0x7f1de806c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:58266/0 (socket says 172.21.15.2:58266) 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.960+0000 7f1de6a00640 1 -- 172.21.15.2:0/717216133 learned_addr learned my addr 172.21.15.2:0/717216133 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.961+0000 7f1de6a00640 1 -- 172.21.15.2:0/717216133 >> v2:172.21.15.137:3300/0 conn(0x7f1de806d000 msgr2=0x7f1de8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.961+0000 7f1de6a00640 1 --2- 172.21.15.2:0/717216133 >> v2:172.21.15.137:3300/0 conn(0x7f1de806d000 0x7f1de8070b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.961+0000 7f1de6a00640 1 -- 172.21.15.2:0/717216133 >> v1:172.21.15.137:6789/0 conn(0x7f1de80cdd70 legacy=0x7f1de806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.961+0000 7f1de6a00640 1 -- 172.21.15.2:0/717216133 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de80705f0 con 0x7f1de8144c00 2024-01-15T18:19:26.963 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.961+0000 7f1de7400640 1 --2- 172.21.15.2:0/717216133 >> v2:172.21.15.137:3300/0 conn(0x7f1de806d000 0x7f1de8070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:26.964 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.962+0000 7f1de6a00640 1 --2- 172.21.15.2:0/717216133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 0x7f1de80cd830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009e50 tx=0x7f1dd802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=401e5deb12d7af4c server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:26.965 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.963+0000 7f1de6000640 1 -- 172.21.15.2:0/717216133 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd802cb00 con 0x7f1de8144c00 2024-01-15T18:19:26.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.963+0000 7f1de6000640 1 -- 172.21.15.2:0/717216133 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd802cc60 con 0x7f1de8144c00 2024-01-15T18:19:26.966 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.963+0000 7f1de6000640 1 -- 172.21.15.2:0/717216133 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd80343f0 con 0x7f1de8144c00 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 -- 172.21.15.2:0/717216133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 msgr2=0x7f1de80cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 --2- 172.21.15.2:0/717216133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 0x7f1de80cd830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8009e50 tx=0x7f1dd802bf60 comp rx=0 tx=0).stop 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 -- 172.21.15.2:0/717216133 shutdown_connections 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 --2- 172.21.15.2:0/717216133 >> v2:172.21.15.137:3300/0 conn(0x7f1de806d000 0x7f1de8070b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 --2- 172.21.15.2:0/717216133 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de8144c00 0x7f1de80cd830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 -- 172.21.15.2:0/717216133 >> 172.21.15.2:0/717216133 conn(0x7f1de80b45f0 msgr2=0x7f1de80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 -- 172.21.15.2:0/717216133 shutdown_connections 2024-01-15T18:19:26.967 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.964+0000 7f1deda00640 1 -- 172.21.15.2:0/717216133 wait complete. 2024-01-15T18:19:26.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.966+0000 7f1deda00640 1 Processor -- start 2024-01-15T18:19:26.968 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.966+0000 7f1deda00640 1 -- start start 2024-01-15T18:19:26.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.967+0000 7f1deda00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de806d000 0x7f1de813ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:26.969 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.967+0000 7f1deda00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de80cdd70 0x7f1de8141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:26.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.967+0000 7f1deda00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de8144c00 0x7f1de80b10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:26.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.967+0000 7f1deda00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8141550 con 0x7f1de80cdd70 2024-01-15T18:19:26.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.967+0000 7f1deda00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8141eb0 con 0x7f1de8144c00 2024-01-15T18:19:26.970 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.967+0000 7f1deda00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8159aa0 con 0x7f1de806d000 2024-01-15T18:19:26.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de80cdd70 0x7f1de8141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:26.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de80cdd70 0x7f1de8141970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43464/0 (socket says 172.21.15.2:43464) 2024-01-15T18:19:26.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 -- 172.21.15.2:0/1557588943 learned_addr learned my addr 172.21.15.2:0/1557588943 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:26.971 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1de7400640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de8144c00 0x7f1de80b10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:26.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1de6a00640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de806d000 0x7f1de813ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:26.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 -- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de806d000 msgr2=0x7f1de813ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:26.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f1de806d000 0x7f1de813ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:26.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 -- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de8144c00 msgr2=0x7f1de80b10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:26.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1de8144c00 0x7f1de80b10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:26.972 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.968+0000 7f1ddde00640 1 -- 172.21.15.2:0/1557588943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de0009400 con 0x7f1de80cdd70 2024-01-15T18:19:26.975 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.969+0000 7f1ddde00640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f1de80cdd70 0x7f1de8141970 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f1dd000c400 tx=0x7f1dd000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:26.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.969+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00074e0 con 0x7f1de80cdd70 2024-01-15T18:19:26.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.969+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0007640 con 0x7f1de80cdd70 2024-01-15T18:19:26.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.969+0000 7f1deda00640 1 -- 172.21.15.2:0/1557588943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd8009b30 con 0x7f1de80cdd70 2024-01-15T18:19:26.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.969+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0004a00 con 0x7f1de80cdd70 2024-01-15T18:19:26.976 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.969+0000 7f1deda00640 1 -- 172.21.15.2:0/1557588943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de8140e90 con 0x7f1de80cdd70 2024-01-15T18:19:26.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.970+0000 7f1deda00640 1 -- 172.21.15.2:0/1557588943 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8141690 con 0x7f1de80cdd70 2024-01-15T18:19:26.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.971+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1dd0004b60 con 0x7f1de80cdd70 2024-01-15T18:19:26.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.972+0000 7f1ddf400640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dc006c550 0x7f1dc006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:26.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.972+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f1dd0088450 con 0x7f1de80cdd70 2024-01-15T18:19:26.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.973+0000 7f1de6a00640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dc006c550 0x7f1dc006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:26.977 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.974+0000 7f1de6a00640 1 --2- 172.21.15.2:0/1557588943 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f1dc006c550 0x7f1dc006ea10 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f1dd8035000 tx=0x7f1dd8030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:26.979 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:26.977+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1de8141690 con 0x7f1de80cdd70 2024-01-15T18:19:27.207 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:27.205+0000 7f1deda00640 1 -- 172.21.15.2:0/1557588943 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1de8141690 con 0x7f1dc006c550 2024-01-15T18:19:27.220 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:27.218+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:27.221 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:27.218+0000 7f1ddf400640 1 -- 172.21.15.2:0/1557588943 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1de8141690 con 0x7f1dc006c550 2024-01-15T18:19:27.221 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:27.221 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:29.244 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:29.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.480+0000 7f91e2c00640 1 Processor -- start 2024-01-15T18:19:29.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.481+0000 7f91e2c00640 1 -- start start 2024-01-15T18:19:29.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.482+0000 7f91e2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc1609d0 0x7f91dc0cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:29.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.482+0000 7f91e2c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f91dc0b3fa0 0x7f91dc0d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.482+0000 7f91e2c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91dc0d0af0 con 0x7f91dc0d1d40 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.482+0000 7f91e2c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91dc0ce7b0 con 0x7f91dc0b3fa0 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.482+0000 7f91e2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0cdbe0 con 0x7f91dc1609d0 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f91dc0b3fa0 0x7f91dc0d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f91dc0b3fa0 0x7f91dc0d01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43466/0 (socket says 172.21.15.2:43466) 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 -- 172.21.15.2:0/1615016892 learned_addr learned my addr 172.21.15.2:0/1615016892 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e1800640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc1609d0 0x7f91dc0cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 -- 172.21.15.2:0/1615016892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc1609d0 msgr2=0x7f91dc0cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc1609d0 0x7f91dc0cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 -- 172.21.15.2:0/1615016892 >> v1:172.21.15.137:6789/0 conn(0x7f91dc0d1d40 legacy=0x7f91dc0cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:29.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e2200640 1 -- 172.21.15.2:0/1615016892 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0cde80 con 0x7f91dc0b3fa0 2024-01-15T18:19:29.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.483+0000 7f91e1800640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc1609d0 0x7f91dc0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.484+0000 7f91e2200640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc0b3fa0 0x7f91dc0d01d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009e50 tx=0x7f91d402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd2ac3cd90b6c03 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.485+0000 7f91dbe00640 1 -- 172.21.15.2:0/1615016892 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d402ca70 con 0x7f91dc0b3fa0 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.485+0000 7f91dbe00640 1 -- 172.21.15.2:0/1615016892 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d402cbd0 con 0x7f91dc0b3fa0 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.485+0000 7f91dbe00640 1 -- 172.21.15.2:0/1615016892 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d40343f0 con 0x7f91dc0b3fa0 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.485+0000 7f91e2c00640 1 -- 172.21.15.2:0/1615016892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc0b3fa0 msgr2=0x7f91dc0d01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.485+0000 7f91e2c00640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc0b3fa0 0x7f91dc0d01d0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009e50 tx=0x7f91d402bed0 comp rx=0 tx=0).stop 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.486+0000 7f91e2c00640 1 -- 172.21.15.2:0/1615016892 shutdown_connections 2024-01-15T18:19:29.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.486+0000 7f91e2c00640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc0b3fa0 0x7f91dc0d01d0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.486+0000 7f91e2c00640 1 --2- 172.21.15.2:0/1615016892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc1609d0 0x7f91dc0cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.486+0000 7f91e2c00640 1 -- 172.21.15.2:0/1615016892 >> 172.21.15.2:0/1615016892 conn(0x7f91dc09c3e0 msgr2=0x7f91dc0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.486+0000 7f91e2c00640 1 -- 172.21.15.2:0/1615016892 shutdown_connections 2024-01-15T18:19:29.489 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.486+0000 7f91e2c00640 1 -- 172.21.15.2:0/1615016892 wait complete. 2024-01-15T18:19:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.487+0000 7f91e2c00640 1 Processor -- start 2024-01-15T18:19:29.490 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.488+0000 7f91e2c00640 1 -- start start 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91dc0b3fa0 0x7f91dc0ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc0d1d40 0x7f91dc0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc1609d0 0x7f91dc0ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0ccb00 con 0x7f91dc1609d0 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0ca6e0 con 0x7f91dc0b3fa0 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91dc0cbef0 con 0x7f91dc0d1d40 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc1609d0 0x7f91dc0ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc1609d0 0x7f91dc0ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43478/0 (socket says 172.21.15.2:43478) 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 -- 172.21.15.2:0/541283075 learned_addr learned my addr 172.21.15.2:0/541283075 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 -- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc0d1d40 msgr2=0x7f91dc0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f91dc0d1d40 0x7f91dc0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:29.493 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 -- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91dc0b3fa0 msgr2=0x7f91dc0ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e1800640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91dc0b3fa0 0x7f91dc0ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f91dc0b3fa0 0x7f91dc0ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.489+0000 7f91e2200640 1 -- 172.21.15.2:0/541283075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d4009b30 con 0x7f91dc1609d0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.490+0000 7f91e2200640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f91dc1609d0 0x7f91dc0ccf90 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f91d4009e50 tx=0x7f91d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.490+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4009510 con 0x7f91dc1609d0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.490+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d403e450 con 0x7f91dc1609d0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.490+0000 7f91e2c00640 1 -- 172.21.15.2:0/541283075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91dc0cc1d0 con 0x7f91dc1609d0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.490+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d403e800 con 0x7f91dc1609d0 2024-01-15T18:19:29.494 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.490+0000 7f91e2c00640 1 -- 172.21.15.2:0/541283075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91dc0cc790 con 0x7f91dc1609d0 2024-01-15T18:19:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.491+0000 7f91e2c00640 1 -- 172.21.15.2:0/541283075 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91dc0b4780 con 0x7f91dc1609d0 2024-01-15T18:19:29.495 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.493+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f91d403e960 con 0x7f91dc1609d0 2024-01-15T18:19:29.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.494+0000 7f91d9600640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91b406c580 0x7f91b406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:29.496 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.494+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f91d40ab600 con 0x7f91dc1609d0 2024-01-15T18:19:29.497 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.495+0000 7f91e1800640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91b406c580 0x7f91b406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:29.498 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.496+0000 7f91e1800640 1 --2- 172.21.15.2:0/541283075 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f91b406c580 0x7f91b406ea40 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f91cc008850 tx=0x7f91cc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:29.501 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.499+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f91d4077f70 con 0x7f91dc1609d0 2024-01-15T18:19:29.728 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.726+0000 7f91e2c00640 1 -- 172.21.15.2:0/541283075 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f91dc160db0 con 0x7f91b406c580 2024-01-15T18:19:29.741 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:29.739+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:29.742 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:29.740+0000 7f91d9600640 1 -- 172.21.15.2:0/541283075 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f91cc00c040 con 0x7f91b406c580 2024-01-15T18:19:29.742 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:29.742 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:31.753 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:31.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.987+0000 7f4e66600640 1 Processor -- start 2024-01-15T18:19:31.990 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.988+0000 7f4e66600640 1 -- start start 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.989+0000 7f4e66600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f4e601609d0 0x7f4e600cec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.989+0000 7f4e66600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4e600d1d40 0x7f4e600b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.989+0000 7f4e66600640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e600d0af0 con 0x7f4e600cfac0 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.989+0000 7f4e66600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e600ce7b0 con 0x7f4e600d1d40 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.989+0000 7f4e66600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e600cdbe0 con 0x7f4e601609d0 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e5fe00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4e600d1d40 0x7f4e600b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e65c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4e600cfac0 0x7f4e600cff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:53030/0 (socket says 172.21.15.2:53030) 2024-01-15T18:19:31.992 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e65c00640 1 -- 172.21.15.2:0/1385224852 learned_addr learned my addr 172.21.15.2:0/1385224852 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:31.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e65200640 1 --2- 172.21.15.2:0/1385224852 >> v2:172.21.15.137:3300/0 conn(0x7f4e601609d0 0x7f4e600cec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:31.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e5fe00640 1 -- 172.21.15.2:0/1385224852 >> v2:172.21.15.137:3300/0 conn(0x7f4e601609d0 msgr2=0x7f4e600cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:31.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e5fe00640 1 --2- 172.21.15.2:0/1385224852 >> v2:172.21.15.137:3300/0 conn(0x7f4e601609d0 0x7f4e600cec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:31.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e5fe00640 1 -- 172.21.15.2:0/1385224852 >> v1:172.21.15.2:6789/0 conn(0x7f4e600cfac0 legacy=0x7f4e600cff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:31.993 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e5fe00640 1 -- 172.21.15.2:0/1385224852 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600cde80 con 0x7f4e600d1d40 2024-01-15T18:19:31.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.990+0000 7f4e5f400640 1 -- 172.21.15.2:0/1385224852 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1240550344 0 0) 0x7f4e600d0af0 con 0x7f4e600cfac0 2024-01-15T18:19:31.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.991+0000 7f4e5fe00640 1 --2- 172.21.15.2:0/1385224852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e600b3fa0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f4e6008d270 tx=0x7f4e4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426610d06c1c96d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:31.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e5f400640 1 -- 172.21.15.2:0/1385224852 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4802cc80 con 0x7f4e600d1d40 2024-01-15T18:19:31.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e5f400640 1 -- 172.21.15.2:0/1385224852 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4802cde0 con 0x7f4e600d1d40 2024-01-15T18:19:31.994 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e5f400640 1 -- 172.21.15.2:0/1385224852 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48034660 con 0x7f4e600d1d40 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e66600640 1 -- 172.21.15.2:0/1385224852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 msgr2=0x7f4e600b3fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e66600640 1 --2- 172.21.15.2:0/1385224852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e600b3fa0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f4e6008d270 tx=0x7f4e4802bed0 comp rx=0 tx=0).stop 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e66600640 1 -- 172.21.15.2:0/1385224852 shutdown_connections 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e66600640 1 --2- 172.21.15.2:0/1385224852 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e600b3fa0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e66600640 1 --2- 172.21.15.2:0/1385224852 >> v2:172.21.15.137:3300/0 conn(0x7f4e601609d0 0x7f4e600cec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.992+0000 7f4e66600640 1 -- 172.21.15.2:0/1385224852 >> 172.21.15.2:0/1385224852 conn(0x7f4e6009c3e0 msgr2=0x7f4e600d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.993+0000 7f4e66600640 1 -- 172.21.15.2:0/1385224852 shutdown_connections 2024-01-15T18:19:31.995 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.993+0000 7f4e66600640 1 -- 172.21.15.2:0/1385224852 wait complete. 2024-01-15T18:19:31.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.994+0000 7f4e66600640 1 Processor -- start 2024-01-15T18:19:31.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.994+0000 7f4e66600640 1 -- start start 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.995+0000 7f4e66600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e600cfac0 0x7f4e600ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.995+0000 7f4e66600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e6015e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.995+0000 7f4e66600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e601609d0 0x7f4e601571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.995+0000 7f4e66600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600b5cf0 con 0x7f4e600d1d40 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.995+0000 7f4e66600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600cb1b0 con 0x7f4e601609d0 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.995+0000 7f4e66600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e600ca6e0 con 0x7f4e600cfac0 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e6015e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e6015e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:43490/0 (socket says 172.21.15.2:43490) 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 -- 172.21.15.2:0/63463955 learned_addr learned my addr 172.21.15.2:0/63463955 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e65c00640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e601609d0 0x7f4e601571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:31.999 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e65200640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e600cfac0 0x7f4e600ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 -- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e600cfac0 msgr2=0x7f4e600ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f4e600cfac0 0x7f4e600ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 -- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e601609d0 msgr2=0x7f4e601571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e601609d0 0x7f4e601571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e5fe00640 1 -- 172.21.15.2:0/63463955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e48009a70 con 0x7f4e600d1d40 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.996+0000 7f4e65c00640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f4e601609d0 0x7f4e601571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:32.000 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.997+0000 7f4e5fe00640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4e600d1d40 0x7f4e6015e8f0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f4e48037040 tx=0x7f4e48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:32.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.997+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48040650 con 0x7f4e600d1d40 2024-01-15T18:19:32.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.997+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e480407b0 con 0x7f4e600d1d40 2024-01-15T18:19:32.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.997+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e48040b60 con 0x7f4e600d1d40 2024-01-15T18:19:32.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.997+0000 7f4e66600640 1 -- 172.21.15.2:0/63463955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e600ca910 con 0x7f4e600d1d40 2024-01-15T18:19:32.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.997+0000 7f4e66600640 1 -- 172.21.15.2:0/63463955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e600caed0 con 0x7f4e600d1d40 2024-01-15T18:19:32.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.998+0000 7f4e66600640 1 -- 172.21.15.2:0/63463955 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e600b4780 con 0x7f4e600d1d40 2024-01-15T18:19:32.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:31.999+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f4e48040cc0 con 0x7f4e600d1d40 2024-01-15T18:19:32.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.001+0000 7f4e5cc00640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e3806c550 0x7f4e3806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:32.003 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.001+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f4e480aba30 con 0x7f4e600d1d40 2024-01-15T18:19:32.004 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.001+0000 7f4e65200640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e3806c550 0x7f4e3806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:32.005 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.003+0000 7f4e65200640 1 --2- 172.21.15.2:0/63463955 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f4e3806c550 0x7f4e3806ea10 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4e50006fd0 tx=0x7f4e50008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:32.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.006+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e48077f70 con 0x7f4e600d1d40 2024-01-15T18:19:32.238 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.236+0000 7f4e66600640 1 -- 172.21.15.2:0/63463955 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e60160db0 con 0x7f4e3806c550 2024-01-15T18:19:32.253 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:32.251+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:32.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:32.251+0000 7f4e5cc00640 1 -- 172.21.15.2:0/63463955 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4e5000f860 con 0x7f4e3806c550 2024-01-15T18:19:32.253 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:32.254 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:34.269 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:34.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.509+0000 7fe949c00640 1 Processor -- start 2024-01-15T18:19:34.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.509+0000 7fe949c00640 1 -- start start 2024-01-15T18:19:34.513 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe949c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe94414afd0 0x7fe9440b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe949c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe94414b3b0 0x7fe94406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe949c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9440b25f0 con 0x7fe94406ce30 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe949c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe944070ff0 con 0x7fe94414b3b0 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe949c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe94406d2a0 con 0x7fe94414afd0 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe943400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe94406ce30 0x7fe9440702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39642/0 (socket says 172.21.15.2:39642) 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.511+0000 7fe943400640 1 -- 172.21.15.2:0/998676052 learned_addr learned my addr 172.21.15.2:0/998676052 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:34.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.512+0000 7fe93be00640 1 --2- 172.21.15.2:0/998676052 >> v2:172.21.15.2:3300/0 conn(0x7fe94414b3b0 0x7fe94406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.512+0000 7fe942a00640 1 --2- 172.21.15.2:0/998676052 >> v2:172.21.15.137:3300/0 conn(0x7fe94414afd0 0x7fe9440b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.512+0000 7fe942000640 1 -- 172.21.15.2:0/998676052 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294514666 0 0) 0x7fe9440b25f0 con 0x7fe94406ce30 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.512+0000 7fe942000640 1 -- 172.21.15.2:0/998676052 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe924003520 con 0x7fe94406ce30 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.512+0000 7fe93be00640 1 -- 172.21.15.2:0/998676052 >> v2:172.21.15.137:3300/0 conn(0x7fe94414afd0 msgr2=0x7fe9440b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.513+0000 7fe93be00640 1 --2- 172.21.15.2:0/998676052 >> v2:172.21.15.137:3300/0 conn(0x7fe94414afd0 0x7fe9440b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.513+0000 7fe93be00640 1 -- 172.21.15.2:0/998676052 >> v1:172.21.15.2:6789/0 conn(0x7fe94406ce30 legacy=0x7fe9440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:34.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.513+0000 7fe93be00640 1 -- 172.21.15.2:0/998676052 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9440b28d0 con 0x7fe94414b3b0 2024-01-15T18:19:34.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.514+0000 7fe93be00640 1 --2- 172.21.15.2:0/998676052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe94406c8f0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fe94406f9f0 tx=0x7fe92c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50296cac5ec909b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:34.516 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.514+0000 7fe942000640 1 -- 172.21.15.2:0/998676052 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c02e070 con 0x7fe94414b3b0 2024-01-15T18:19:34.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.514+0000 7fe942000640 1 -- 172.21.15.2:0/998676052 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92c0315a0 con 0x7fe94414b3b0 2024-01-15T18:19:34.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.514+0000 7fe942000640 1 -- 172.21.15.2:0/998676052 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92c031930 con 0x7fe94414b3b0 2024-01-15T18:19:34.517 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 -- 172.21.15.2:0/998676052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 msgr2=0x7fe94406c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 --2- 172.21.15.2:0/998676052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe94406c8f0 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fe94406f9f0 tx=0x7fe92c02b840 comp rx=0 tx=0).stop 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 -- 172.21.15.2:0/998676052 shutdown_connections 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 --2- 172.21.15.2:0/998676052 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe94406c8f0 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 --2- 172.21.15.2:0/998676052 >> v2:172.21.15.137:3300/0 conn(0x7fe94414afd0 0x7fe9440b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 -- 172.21.15.2:0/998676052 >> 172.21.15.2:0/998676052 conn(0x7fe9440b45f0 msgr2=0x7fe9440b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 -- 172.21.15.2:0/998676052 shutdown_connections 2024-01-15T18:19:34.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.515+0000 7fe949c00640 1 -- 172.21.15.2:0/998676052 wait complete. 2024-01-15T18:19:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.517+0000 7fe949c00640 1 Processor -- start 2024-01-15T18:19:34.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.517+0000 7fe949c00640 1 -- start start 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe949c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe94406ce30 0x7fe9441f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe949c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe94414afd0 0x7fe9440ac280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe949c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe9441633b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe949c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9440aae60 con 0x7fe94414b3b0 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe949c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9440aa5c0 con 0x7fe94406ce30 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe949c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9440aa8a0 con 0x7fe94414afd0 2024-01-15T18:19:34.521 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe943400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe9441633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.518+0000 7fe943400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe9441633b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46190/0 (socket says 172.21.15.2:46190) 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 -- 172.21.15.2:0/610569196 learned_addr learned my addr 172.21.15.2:0/610569196 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe93be00640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe94414afd0 0x7fe9440ac280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe942a00640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe94406ce30 0x7fe9441f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 -- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe94414afd0 msgr2=0x7fe9440ac280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe94414afd0 0x7fe9440ac280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 -- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe94406ce30 msgr2=0x7fe9441f6800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe94406ce30 0x7fe9441f6800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:34.522 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 -- 172.21.15.2:0/610569196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c009400 con 0x7fe94414b3b0 2024-01-15T18:19:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.519+0000 7fe943400640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe94414b3b0 0x7fe9441633b0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fe93c00cb40 tx=0x7fe93c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.520+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c00aca0 con 0x7fe94414b3b0 2024-01-15T18:19:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.520+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c00ae00 con 0x7fe94414b3b0 2024-01-15T18:19:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.520+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c012620 con 0x7fe94414b3b0 2024-01-15T18:19:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.520+0000 7fe949c00640 1 -- 172.21.15.2:0/610569196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9440aab80 con 0x7fe94414b3b0 2024-01-15T18:19:34.523 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.520+0000 7fe949c00640 1 -- 172.21.15.2:0/610569196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9441f6d40 con 0x7fe94414b3b0 2024-01-15T18:19:34.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.521+0000 7fe949c00640 1 -- 172.21.15.2:0/610569196 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9440a8820 con 0x7fe94414b3b0 2024-01-15T18:19:34.524 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.522+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe93c007070 con 0x7fe94414b3b0 2024-01-15T18:19:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.523+0000 7fe93aa00640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe91c06c550 0x7fe91c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:34.525 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.523+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fe93c088be0 con 0x7fe94414b3b0 2024-01-15T18:19:34.526 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.524+0000 7fe942a00640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe91c06c550 0x7fe91c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:34.527 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.525+0000 7fe942a00640 1 --2- 172.21.15.2:0/610569196 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe91c06c550 0x7fe91c06ea10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe934009ad0 tx=0x7fe934008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:34.530 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.528+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe93c055120 con 0x7fe94414b3b0 2024-01-15T18:19:34.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.758+0000 7fe949c00640 1 -- 172.21.15.2:0/610569196 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe9440a8460 con 0x7fe91c06c550 2024-01-15T18:19:34.774 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:34.772+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:34.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:34.772+0000 7fe93aa00640 1 -- 172.21.15.2:0/610569196 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe934009800 con 0x7fe91c06c550 2024-01-15T18:19:34.775 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:34.775 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:34.920 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:34.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:19:36.790 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:37.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.028+0000 7fac99e00640 1 Processor -- start 2024-01-15T18:19:37.031 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.029+0000 7fac99e00640 1 -- start start 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.030+0000 7fac99e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fac9415e1d0 0x7fac940c9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.030+0000 7fac99e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940c9750 con 0x7fac940c7ef0 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.030+0000 7fac99e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940c99b0 con 0x7fac940709d0 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.030+0000 7fac99e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9415e680 con 0x7fac9415e1d0 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac98a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fac940c7ef0 0x7fac940c82d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39644/0 (socket says 172.21.15.2:39644) 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac98a00640 1 -- 172.21.15.2:0/2269259568 learned_addr learned my addr 172.21.15.2:0/2269259568 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:37.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac93e00640 1 --2- 172.21.15.2:0/2269259568 >> v2:172.21.15.2:3300/0 conn(0x7fac9415e1d0 0x7fac940c9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:37.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac93400640 1 -- 172.21.15.2:0/2269259568 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547527869 0 0) 0x7fac940c9750 con 0x7fac940c7ef0 2024-01-15T18:19:37.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac93400640 1 -- 172.21.15.2:0/2269259568 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac74003520 con 0x7fac940c7ef0 2024-01-15T18:19:37.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac93e00640 1 -- 172.21.15.2:0/2269259568 >> v1:172.21.15.137:6789/0 conn(0x7fac940709d0 legacy=0x7fac94070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:37.034 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac93e00640 1 -- 172.21.15.2:0/2269259568 >> v1:172.21.15.2:6789/0 conn(0x7fac940c7ef0 legacy=0x7fac940c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:37.035 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.031+0000 7fac93e00640 1 -- 172.21.15.2:0/2269259568 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac940c9c90 con 0x7fac9415e1d0 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.033+0000 7fac93e00640 1 --2- 172.21.15.2:0/2269259568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac9415e1d0 0x7fac940c9210 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0097a0 tx=0x7fac7c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89bd9298d2d2c269 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.033+0000 7fac93400640 1 -- 172.21.15.2:0/2269259568 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c030610 con 0x7fac9415e1d0 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.033+0000 7fac93400640 1 -- 172.21.15.2:0/2269259568 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c030770 con 0x7fac9415e1d0 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.033+0000 7fac93400640 1 -- 172.21.15.2:0/2269259568 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c030b00 con 0x7fac9415e1d0 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.033+0000 7fac99e00640 1 -- 172.21.15.2:0/2269259568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac9415e1d0 msgr2=0x7fac940c9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.033+0000 7fac99e00640 1 --2- 172.21.15.2:0/2269259568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac9415e1d0 0x7fac940c9210 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0097a0 tx=0x7fac7c02b960 comp rx=0 tx=0).stop 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.034+0000 7fac99e00640 1 -- 172.21.15.2:0/2269259568 shutdown_connections 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.034+0000 7fac99e00640 1 --2- 172.21.15.2:0/2269259568 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac9415e1d0 0x7fac940c9210 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.034+0000 7fac99e00640 1 -- 172.21.15.2:0/2269259568 >> 172.21.15.2:0/2269259568 conn(0x7fac940cebc0 msgr2=0x7fac940cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:37.036 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.034+0000 7fac99e00640 1 -- 172.21.15.2:0/2269259568 shutdown_connections 2024-01-15T18:19:37.037 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.034+0000 7fac99e00640 1 -- 172.21.15.2:0/2269259568 wait complete. 2024-01-15T18:19:37.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.036+0000 7fac99e00640 1 Processor -- start 2024-01-15T18:19:37.038 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.036+0000 7fac99e00640 1 -- start start 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac940709d0 0x7fac941563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac940c7ef0 0x7fac9409c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac9415e1d0 0x7fac94148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac941f3090 con 0x7fac940709d0 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac941499c0 con 0x7fac940c7ef0 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94149ca0 con 0x7fac9415e1d0 2024-01-15T18:19:37.040 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac98a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac940709d0 0x7fac941563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac98a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac940709d0 0x7fac941563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46204/0 (socket says 172.21.15.2:46204) 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac98a00640 1 -- 172.21.15.2:0/1050484535 learned_addr learned my addr 172.21.15.2:0/1050484535 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac98a00640 1 -- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac9415e1d0 msgr2=0x7fac94148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac93e00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac940c7ef0 0x7fac9409c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.037+0000 7fac99400640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac9415e1d0 0x7fac94148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac98a00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fac9415e1d0 0x7fac94148ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac98a00640 1 -- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac940c7ef0 msgr2=0x7fac9409c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac98a00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fac940c7ef0 0x7fac9409c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac98a00640 1 -- 172.21.15.2:0/1050484535 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c009400 con 0x7fac940709d0 2024-01-15T18:19:37.041 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac98a00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fac940709d0 0x7fac941563d0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7fac84001eb0 tx=0x7fac8400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:37.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84005ce0 con 0x7fac940709d0 2024-01-15T18:19:37.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84005e40 con 0x7fac940709d0 2024-01-15T18:19:37.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84007270 con 0x7fac940709d0 2024-01-15T18:19:37.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac99e00640 1 -- 172.21.15.2:0/1050484535 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9409bf30 con 0x7fac940709d0 2024-01-15T18:19:37.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.038+0000 7fac99e00640 1 -- 172.21.15.2:0/1050484535 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9409bc20 con 0x7fac940709d0 2024-01-15T18:19:37.042 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.039+0000 7fac99e00640 1 -- 172.21.15.2:0/1050484535 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9409c450 con 0x7fac940709d0 2024-01-15T18:19:37.043 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.040+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fac84007490 con 0x7fac940709d0 2024-01-15T18:19:37.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.042+0000 7fac90c00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac6c06c550 0x7fac6c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:37.044 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.042+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fac84088060 con 0x7fac940709d0 2024-01-15T18:19:37.045 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.042+0000 7fac93e00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac6c06c550 0x7fac6c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:37.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.044+0000 7fac93e00640 1 --2- 172.21.15.2:0/1050484535 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fac6c06c550 0x7fac6c06ea10 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fac7c036000 tx=0x7fac7c03b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:37.048 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.046+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac9409c450 con 0x7fac940709d0 2024-01-15T18:19:37.276 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.274+0000 7fac99e00640 1 -- 172.21.15.2:0/1050484535 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fac940ca250 con 0x7fac6c06c550 2024-01-15T18:19:37.290 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:37.287+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:37.290 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:37.289+0000 7fac90c00640 1 -- 172.21.15.2:0/1050484535 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fac7c0308e0 con 0x7fac6c06c550 2024-01-15T18:19:37.291 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:37.291 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:37.302 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:39.304 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:39.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.537+0000 7f5761c00640 1 Processor -- start 2024-01-15T18:19:39.540 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.538+0000 7f5761c00640 1 -- start start 2024-01-15T18:19:39.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.539+0000 7f5761c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 0x7f575c165d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.539+0000 7f5761c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0ce230 con 0x7f575c15c450 2024-01-15T18:19:39.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f5761c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c0ce510 con 0x7f575c144c00 2024-01-15T18:19:39.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f5761c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0ce7f0 con 0x7f575c06d000 2024-01-15T18:19:39.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f5753e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f575c15c450 0x7f575c06c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39658/0 (socket says 172.21.15.2:39658) 2024-01-15T18:19:39.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f5753e00640 1 -- 172.21.15.2:0/1558082614 learned_addr learned my addr 172.21.15.2:0/1558082614 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f575b400640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 0x7f575c165d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492861542 0 0) 0x7f575c0ce230 con 0x7f575c15c450 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.540+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f573c003520 con 0x7f575c15c450 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2693538591 0 0) 0x7f573c003520 con 0x7f575c15c450 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 msgr2=0x7f575c165d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 0x7f575c165d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 >> v1:172.21.15.137:6789/0 conn(0x7f575c144c00 legacy=0x7f575c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0705f0 con 0x7f575c15c450 2024-01-15T18:19:39.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5744001670 con 0x7f575c15c450 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 >> v1:172.21.15.2:6789/0 conn(0x7f575c15c450 legacy=0x7f575c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 0x7f573c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f573c003f90 0x7f573c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f573c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0ce7f0 con 0x7f573c004d00 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c00ba90 con 0x7f573c003f90 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573c00bc50 con 0x7f575c06d000 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.541+0000 7f575b400640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f573c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f575aa00640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 0x7f573c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f5753e00640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f573c003f90 0x7f573c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f575b400640 1 -- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 msgr2=0x7f573c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:39.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f575b400640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c06d000 0x7f573c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:39.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f575b400640 1 -- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f573c003f90 msgr2=0x7f573c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:39.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f575b400640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f573c003f90 0x7f573c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:39.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.542+0000 7f575b400640 1 -- 172.21.15.2:0/1558082614 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c003520 con 0x7f573c004d00 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.543+0000 7f575b400640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f573c005180 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f5754007330 tx=0x7f573c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24cae22e7463d5fb server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.543+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575402cca0 con 0x7f573c004d00 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.543+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575402ce00 con 0x7f573c004d00 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.543+0000 7f575a000640 1 -- 172.21.15.2:0/1558082614 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754034660 con 0x7f573c004d00 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f5761c00640 1 -- 172.21.15.2:0/1558082614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 msgr2=0x7f573c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f5761c00640 1 --2- 172.21.15.2:0/1558082614 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f573c005180 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f5754007330 tx=0x7f573c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f575b400640 1 -- 172.21.15.2:0/1558082614 reap_dead start 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f5761c00640 1 -- 172.21.15.2:0/1558082614 shutdown_connections 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f5761c00640 1 -- 172.21.15.2:0/1558082614 >> 172.21.15.2:0/1558082614 conn(0x7f575c0b45f0 msgr2=0x7f575c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:39.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f5761c00640 1 -- 172.21.15.2:0/1558082614 shutdown_connections 2024-01-15T18:19:39.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.544+0000 7f5761c00640 1 -- 172.21.15.2:0/1558082614 wait complete. 2024-01-15T18:19:39.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.546+0000 7f5761c00640 1 Processor -- start 2024-01-15T18:19:39.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.546+0000 7f5761c00640 1 -- start start 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.547+0000 7f5761c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f575c1f2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.547+0000 7f5761c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c136e80 0x7f575c1f3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.547+0000 7f5761c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f575c157ea0 0x7f575c158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.547+0000 7f5761c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b1050 con 0x7f573c004d00 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.547+0000 7f5761c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c1f2e60 con 0x7f575c157ea0 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.547+0000 7f5761c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c158860 con 0x7f575c136e80 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f575c1f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f575c1f2920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46210/0 (socket says 172.21.15.2:46210) 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 -- 172.21.15.2:0/52221780 learned_addr learned my addr 172.21.15.2:0/52221780 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 -- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c136e80 msgr2=0x7f575c1f3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 --2- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f575c136e80 0x7f575c1f3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 -- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f575c157ea0 msgr2=0x7f575c158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 --2- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f575c157ea0 0x7f575c158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:39.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 -- 172.21.15.2:0/52221780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c0b0440 con 0x7f573c004d00 2024-01-15T18:19:39.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.548+0000 7f575aa00640 1 --2- 172.21.15.2:0/52221780 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f573c004d00 0x7f575c1f2920 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f574c00e7d0 tx=0x7f574c00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:39.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.549+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c0050d0 con 0x7f573c004d00 2024-01-15T18:19:39.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.549+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c005230 con 0x7f573c004d00 2024-01-15T18:19:39.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.549+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c00c640 con 0x7f573c004d00 2024-01-15T18:19:39.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.549+0000 7f5761c00640 1 -- 172.21.15.2:0/52221780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0b0720 con 0x7f573c004d00 2024-01-15T18:19:39.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.549+0000 7f5761c00640 1 -- 172.21.15.2:0/52221780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c0b0ce0 con 0x7f573c004d00 2024-01-15T18:19:39.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.550+0000 7f5761c00640 1 -- 172.21.15.2:0/52221780 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c0b0e70 con 0x7f573c004d00 2024-01-15T18:19:39.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.551+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f574c00c7a0 con 0x7f573c004d00 2024-01-15T18:19:39.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.552+0000 7f5752a00640 1 --2- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f573406c580 0x7f573406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:39.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.553+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f574c0889f0 con 0x7f573c004d00 2024-01-15T18:19:39.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.553+0000 7f5753e00640 1 --2- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f573406c580 0x7f573406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:39.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.554+0000 7f5753e00640 1 --2- 172.21.15.2:0/52221780 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f573406c580 0x7f573406ea40 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f575c15b450 tx=0x7f57440014d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:39.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.557+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f574c054f30 con 0x7f573c004d00 2024-01-15T18:19:39.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.786+0000 7f5761c00640 1 -- 172.21.15.2:0/52221780 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f575c0cead0 con 0x7f573406c580 2024-01-15T18:19:39.801 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:19:39.798+0000 7fe20c000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-15T18:19:39.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:39.799+0000 7f5752a00640 1 -- 172.21.15.2:0/52221780 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f57440047e0 con 0x7f573406c580 2024-01-15T18:19:39.802 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:39.802 INFO:teuthology.orchestra.run.smithi002.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-15T18:19:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-15T18:19:39.823 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ERROR 2024-01-15T18:19:39.823 INFO:tasks.cephfs_test_runner:ERROR 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 8163, in test_when_3_clones_are_cancelled 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7984, in _cancel_clones_and_assert 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-15T18:19:39.824 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7965, in _cancel_clones 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone29952' 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:19:39.825 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:Ran 7 tests in 1286.456s 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-01-15T18:19:39.826 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 8163, in test_when_3_clones_are_cancelled 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7984, in _cancel_clones_and_assert 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7965, in _cancel_clones 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-15T18:19:39.827 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi002 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone29952' 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-15T18:19:39.828 INFO:tasks.cephfs_test_runner:ERROR: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-15T18:19:39.829 INFO:tasks.cephfs_test_runner: 2024-01-15T18:19:39.829 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:19:40.043 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=351bee4afd7240b382703f065db83af7 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_95c90f413c4bb65e3e56909855f2308fd9bda05b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-15T18:19:40.045 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-15T18:19:40.054 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-15T18:19:40.062 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-15T18:19:40.062 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:19:40.062 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:40.127 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:19:40.127 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:40.185 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:19:40.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.407+0000 7fcf70c00640 1 Processor -- start 2024-01-15T18:19:40.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.407+0000 7fcf70c00640 1 -- start start 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.409+0000 7fcf70c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fcf6c15c0e0 0x7fcf6c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.409+0000 7fcf70c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c15cb00 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.409+0000 7fcf70c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c0ccec0 con 0x7fcf6c06cb80 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.409+0000 7fcf70c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c0cd1a0 con 0x7fcf6c15c0e0 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf69400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcf6c15b7d0 0x7fcf6c06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39660/0 (socket says 172.21.15.2:39660) 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf69400640 1 -- 172.21.15.2:0/3424693057 learned_addr learned my addr 172.21.15.2:0/3424693057 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf69e00640 1 --2- 172.21.15.2:0/3424693057 >> v2:172.21.15.2:3300/0 conn(0x7fcf6c15c0e0 0x7fcf6c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf68a00640 1 -- 172.21.15.2:0/3424693057 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453638937 0 0) 0x7fcf6c15cb00 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf69e00640 1 -- 172.21.15.2:0/3424693057 >> v1:172.21.15.137:6789/0 conn(0x7fcf6c06cb80 legacy=0x7fcf6c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:40.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf69e00640 1 -- 172.21.15.2:0/3424693057 >> v1:172.21.15.2:6789/0 conn(0x7fcf6c15b7d0 legacy=0x7fcf6c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.410+0000 7fcf69e00640 1 -- 172.21.15.2:0/3424693057 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c0cd480 con 0x7fcf6c15c0e0 2024-01-15T18:19:40.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.411+0000 7fcf69e00640 1 --2- 172.21.15.2:0/3424693057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15c0e0 0x7fcf6c0cc5c0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fcf540097a0 tx=0x7fcf5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4845696b6903cc4e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:40.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.411+0000 7fcf68a00640 1 -- 172.21.15.2:0/3424693057 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf54031830 con 0x7fcf6c15c0e0 2024-01-15T18:19:40.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf68a00640 1 -- 172.21.15.2:0/3424693057 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf54031990 con 0x7fcf6c15c0e0 2024-01-15T18:19:40.414 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf68a00640 1 -- 172.21.15.2:0/3424693057 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf54031d20 con 0x7fcf6c15c0e0 2024-01-15T18:19:40.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf70c00640 1 -- 172.21.15.2:0/3424693057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15c0e0 msgr2=0x7fcf6c0cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3424693057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15c0e0 0x7fcf6c0cc5c0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7fcf540097a0 tx=0x7fcf5402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf70c00640 1 -- 172.21.15.2:0/3424693057 shutdown_connections 2024-01-15T18:19:40.415 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3424693057 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15c0e0 0x7fcf6c0cc5c0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf70c00640 1 -- 172.21.15.2:0/3424693057 >> 172.21.15.2:0/3424693057 conn(0x7fcf6c06da20 msgr2=0x7fcf6c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:40.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.412+0000 7fcf70c00640 1 -- 172.21.15.2:0/3424693057 shutdown_connections 2024-01-15T18:19:40.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.413+0000 7fcf70c00640 1 -- 172.21.15.2:0/3424693057 wait complete. 2024-01-15T18:19:40.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.415+0000 7fcf70c00640 1 Processor -- start 2024-01-15T18:19:40.417 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.415+0000 7fcf70c00640 1 -- start start 2024-01-15T18:19:40.418 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf70c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf6c06cb80 0x7fcf6c14db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf70c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 0x7fcf6c1f0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf70c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf6c15c0e0 0x7fcf6c159260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf70c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c14e0a0 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf70c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c1f1c40 con 0x7fcf6c06cb80 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf70c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c1f1680 con 0x7fcf6c15c0e0 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf69400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 0x7fcf6c1f0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf69400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 0x7fcf6c1f0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46240/0 (socket says 172.21.15.2:46240) 2024-01-15T18:19:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.416+0000 7fcf69400640 1 -- 172.21.15.2:0/3573183308 learned_addr learned my addr 172.21.15.2:0/3573183308 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:40.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69e00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf6c06cb80 0x7fcf6c14db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf6a800640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf6c15c0e0 0x7fcf6c159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69400640 1 -- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf6c15c0e0 msgr2=0x7fcf6c159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69400640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf6c15c0e0 0x7fcf6c159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.421 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69400640 1 -- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf6c06cb80 msgr2=0x7fcf6c14db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69400640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf6c06cb80 0x7fcf6c14db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69400640 1 -- 172.21.15.2:0/3573183308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf54009400 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.417+0000 7fcf69400640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 0x7fcf6c1f0cc0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c890 tx=0x7fcf5c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.418+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00a640 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.418+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf5c00a7c0 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.418+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c00ab70 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.418+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c1f1960 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.418+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c1560a0 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.419+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c1562c0 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.420+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fcf5c00acd0 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.423 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.421+0000 7fcf62000640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf4406c550 0x7fcf4406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.421+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fcf5c088120 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.424 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.422+0000 7fcf69e00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf4406c550 0x7fcf4406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.425 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.423+0000 7fcf69e00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf4406c550 0x7fcf4406ea10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fcf54036000 tx=0x7fcf5403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:40.428 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.426+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf5c054660 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.624+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf6c0686f0 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.628+0000 7fcf62000640 1 -- 172.21.15.2:0/3573183308 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16780 (secure 0 0 0) 0x7fcf5c018070 con 0x7fcf6c15b7d0 2024-01-15T18:19:40.631 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:40.631 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":116,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:15:19.802397+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.630+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf4406c550 msgr2=0x7fcf4406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.630+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf4406c550 0x7fcf4406ea10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fcf54036000 tx=0x7fcf5403c040 comp rx=0 tx=0).stop 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.630+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 msgr2=0x7fcf6c1f0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.630+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 0x7fcf6c1f0cc0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c00c890 tx=0x7fcf5c00cd50 comp rx=0 tx=0).stop 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 shutdown_connections 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcf4406c550 0x7fcf4406ea10 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcf6c15c0e0 0x7fcf6c159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcf6c15b7d0 0x7fcf6c1f0cc0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 --2- 172.21.15.2:0/3573183308 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcf6c06cb80 0x7fcf6c14db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 >> 172.21.15.2:0/3573183308 conn(0x7fcf6c06da20 msgr2=0x7fcf6c14fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:40.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 shutdown_connections 2024-01-15T18:19:40.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.631+0000 7fcf70c00640 1 -- 172.21.15.2:0/3573183308 wait complete. 2024-01-15T18:19:40.651 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:14:14.206729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:14:15.347816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:19:40.651 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:19:40.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.872+0000 7f9f86400640 1 Processor -- start 2024-01-15T18:19:40.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.873+0000 7f9f86400640 1 -- start start 2024-01-15T18:19:40.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f86400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9f8015d4d0 0x7f9f800b2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f86400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 0x7f9f8006f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f86400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8006f060 con 0x7f9f8006a4b0 2024-01-15T18:19:40.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f86400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8015d8b0 con 0x7f9f8015d4d0 2024-01-15T18:19:40.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f86400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800aef50 con 0x7f9f800af6a0 2024-01-15T18:19:40.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f7fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9f8006a4b0 0x7f9f800713d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39670/0 (socket says 172.21.15.2:39670) 2024-01-15T18:19:40.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.874+0000 7f9f7fe00640 1 -- 172.21.15.2:0/907843507 learned_addr learned my addr 172.21.15.2:0/907843507 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7f400640 1 --2- 172.21.15.2:0/907843507 >> v2:172.21.15.137:3300/0 conn(0x7f9f8015d4d0 0x7f9f800b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7ea00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 0x7f9f8006f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767129856 0 0) 0x7f9f8006f060 con 0x7f9f8006a4b0 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f60003520 con 0x7f9f8006a4b0 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 80605159 0 0) 0x7f9f60003520 con 0x7f9f8006a4b0 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 msgr2=0x7f9f8006f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:40.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 0x7f9f8006f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 >> v2:172.21.15.137:3300/0 conn(0x7f9f8015d4d0 msgr2=0x7f9f800b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 --2- 172.21.15.2:0/907843507 >> v2:172.21.15.137:3300/0 conn(0x7f9f8015d4d0 0x7f9f800b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f800af230 con 0x7f9f8006a4b0 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9f78001670 con 0x7f9f8006a4b0 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 >> v1:172.21.15.2:6789/0 conn(0x7f9f8006a4b0 legacy=0x7f9f800713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 0x7f9f60003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.875+0000 7f9f7e000640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f60003ef0 0x7f9f600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7e000640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8015d8b0 con 0x7f9f60004d00 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800aef50 con 0x7f9f60003ef0 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6000ba60 con 0x7f9f800af6a0 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7fe00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7ea00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f60003ef0 0x7f9f600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7f400640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 0x7f9f60003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7fe00640 1 -- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 msgr2=0x7f9f60003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7fe00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800af6a0 0x7f9f60003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7fe00640 1 -- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f60003ef0 msgr2=0x7f9f600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:40.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7fe00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f60003ef0 0x7f9f600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7fe00640 1 -- 172.21.15.2:0/907843507 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f60003520 con 0x7f9f60004d00 2024-01-15T18:19:40.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.876+0000 7f9f7ea00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f60003ef0 0x7f9f600047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f7fe00640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f60005180 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c240 tx=0x7f9f600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=422176273d2366b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7802cb00 con 0x7f9f60004d00 2024-01-15T18:19:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7802cc60 con 0x7f9f60004d00 2024-01-15T18:19:40.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f7e000640 1 -- 172.21.15.2:0/907843507 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f78005af0 con 0x7f9f60004d00 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f86400640 1 -- 172.21.15.2:0/907843507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 msgr2=0x7f9f60005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f86400640 1 --2- 172.21.15.2:0/907843507 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f60005180 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f9f7800c240 tx=0x7f9f600057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f7fe00640 1 -- 172.21.15.2:0/907843507 reap_dead start 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f86400640 1 -- 172.21.15.2:0/907843507 shutdown_connections 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.878+0000 7f9f86400640 1 -- 172.21.15.2:0/907843507 >> 172.21.15.2:0/907843507 conn(0x7f9f800b1d70 msgr2=0x7f9f800b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.879+0000 7f9f86400640 1 -- 172.21.15.2:0/907843507 shutdown_connections 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.879+0000 7f9f86400640 1 -- 172.21.15.2:0/907843507 wait complete. 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.880+0000 7f9f86400640 1 Processor -- start 2024-01-15T18:19:40.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 -- start start 2024-01-15T18:19:40.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f80154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800b0d30 0x7f9f800b11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f8015a440 0x7f9f8015a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f800b16f0 con 0x7f9f60004d00 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f80159350 con 0x7f9f8015a440 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.881+0000 7f9f86400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f801595f0 con 0x7f9f800b0d30 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f80154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f80154de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46264/0 (socket says 172.21.15.2:46264) 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 -- 172.21.15.2:0/114569724 learned_addr learned my addr 172.21.15.2:0/114569724 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7fe00640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f8015a440 0x7f9f8015a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 -- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800b0d30 msgr2=0x7f9f800b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800b0d30 0x7f9f800b11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 -- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f8015a440 msgr2=0x7f9f8015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f8015a440 0x7f9f8015a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7f400640 1 -- 172.21.15.2:0/114569724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f801598d0 con 0x7f9f60004d00 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.882+0000 7f9f7fe00640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f8015a440 0x7f9f8015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.883+0000 7f9f7f400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f80154de0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9f700116d0 tx=0x7f9f70011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.883+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000f9c0 con 0x7f9f60004d00 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.883+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7000fb20 con 0x7f9f60004d00 2024-01-15T18:19:40.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.883+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000fe90 con 0x7f9f60004d00 2024-01-15T18:19:40.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.883+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f80159bb0 con 0x7f9f60004d00 2024-01-15T18:19:40.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.883+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f800ce540 con 0x7f9f60004d00 2024-01-15T18:19:40.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.884+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f80067f30 con 0x7f9f60004d00 2024-01-15T18:19:40.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.885+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9f70015460 con 0x7f9f60004d00 2024-01-15T18:19:40.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.886+0000 7f9f77400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9f5806c550 0x7f9f5806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:40.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.886+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f9f700890f0 con 0x7f9f60004d00 2024-01-15T18:19:40.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.887+0000 7f9f7ea00640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9f5806c550 0x7f9f5806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:40.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.888+0000 7f9f7ea00640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9f5806c550 0x7f9f5806ea10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9f8015c870 tx=0x7f9f68009750 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:40.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:40.891+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f70055630 con 0x7f9f60004d00 2024-01-15T18:19:41.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.093+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9f800686f0 con 0x7f9f60004d00 2024-01-15T18:19:41.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.094+0000 7f9f77400640 1 -- 172.21.15.2:0/114569724 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9f70022020 con 0x7f9f60004d00 2024-01-15T18:19:41.096 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:19:41.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9f5806c550 msgr2=0x7f9f5806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9f5806c550 0x7f9f5806ea10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9f8015c870 tx=0x7f9f68009750 comp rx=0 tx=0).stop 2024-01-15T18:19:41.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 msgr2=0x7f9f80154de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f80154de0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9f700116d0 tx=0x7f9f70011b90 comp rx=0 tx=0).stop 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 shutdown_connections 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9f5806c550 0x7f9f5806ea10 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9f8015a440 0x7f9f8015a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9f800b0d30 0x7f9f800b11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 --2- 172.21.15.2:0/114569724 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9f60004d00 0x7f9f80154de0 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.096+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 >> 172.21.15.2:0/114569724 conn(0x7f9f800b1d70 msgr2=0x7f9f800ac910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.097+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 shutdown_connections 2024-01-15T18:19:41.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.097+0000 7f9f86400640 1 -- 172.21.15.2:0/114569724 wait complete. 2024-01-15T18:19:41.118 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:19:41.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.334+0000 7f0caa400640 1 Processor -- start 2024-01-15T18:19:41.336 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.334+0000 7f0caa400640 1 -- start start 2024-01-15T18:19:41.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0caa400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0ca4144c00 0x7f0ca40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0caa400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0ca40cdd70 0x7f0ca406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0caa400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca40ce230 con 0x7f0ca415c450 2024-01-15T18:19:41.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0caa400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40ce510 con 0x7f0ca4144c00 2024-01-15T18:19:41.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0caa400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40ce7f0 con 0x7f0ca40cdd70 2024-01-15T18:19:41.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0ca3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f0ca415c450 0x7f0ca40b0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:39684/0 (socket says 172.21.15.2:39684) 2024-01-15T18:19:41.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0ca3e00640 1 -- 172.21.15.2:0/3802647853 learned_addr learned my addr 172.21.15.2:0/3802647853 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0ca3400640 1 --2- 172.21.15.2:0/3802647853 >> v2:172.21.15.2:3300/0 conn(0x7f0ca4144c00 0x7f0ca40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.336+0000 7f0c9a800640 1 --2- 172.21.15.2:0/3802647853 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40cdd70 0x7f0ca406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0ca2a00640 1 -- 172.21.15.2:0/3802647853 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154710126 0 0) 0x7f0ca40ce230 con 0x7f0ca415c450 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0ca2a00640 1 -- 172.21.15.2:0/3802647853 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c84003520 con 0x7f0ca415c450 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0ca3400640 1 -- 172.21.15.2:0/3802647853 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40cdd70 msgr2=0x7f0ca406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0ca3400640 1 --2- 172.21.15.2:0/3802647853 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40cdd70 0x7f0ca406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0ca3400640 1 -- 172.21.15.2:0/3802647853 >> v1:172.21.15.2:6789/0 conn(0x7f0ca415c450 legacy=0x7f0ca40b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0ca3400640 1 -- 172.21.15.2:0/3802647853 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca40705f0 con 0x7f0ca4144c00 2024-01-15T18:19:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.337+0000 7f0c9a800640 1 --2- 172.21.15.2:0/3802647853 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40cdd70 0x7f0ca406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:41.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.338+0000 7f0ca3400640 1 --2- 172.21.15.2:0/3802647853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca4144c00 0x7f0ca40cd830 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f0ca406f9f0 tx=0x7f0c9402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a9f90f899638fc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:41.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0ca2a00640 1 -- 172.21.15.2:0/3802647853 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9402cc80 con 0x7f0ca4144c00 2024-01-15T18:19:41.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0ca2a00640 1 -- 172.21.15.2:0/3802647853 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9402cde0 con 0x7f0ca4144c00 2024-01-15T18:19:41.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0ca2a00640 1 -- 172.21.15.2:0/3802647853 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94034660 con 0x7f0ca4144c00 2024-01-15T18:19:41.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0caa400640 1 -- 172.21.15.2:0/3802647853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca4144c00 msgr2=0x7f0ca40cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0caa400640 1 --2- 172.21.15.2:0/3802647853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca4144c00 0x7f0ca40cd830 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f0ca406f9f0 tx=0x7f0c9402bed0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0caa400640 1 -- 172.21.15.2:0/3802647853 shutdown_connections 2024-01-15T18:19:41.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0caa400640 1 --2- 172.21.15.2:0/3802647853 >> v2:172.21.15.137:3300/0 conn(0x7f0ca40cdd70 0x7f0ca406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0caa400640 1 --2- 172.21.15.2:0/3802647853 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca4144c00 0x7f0ca40cd830 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.339+0000 7f0caa400640 1 -- 172.21.15.2:0/3802647853 >> 172.21.15.2:0/3802647853 conn(0x7f0ca40b45f0 msgr2=0x7f0ca40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:41.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.340+0000 7f0caa400640 1 -- 172.21.15.2:0/3802647853 shutdown_connections 2024-01-15T18:19:41.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.340+0000 7f0caa400640 1 -- 172.21.15.2:0/3802647853 wait complete. 2024-01-15T18:19:41.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.342+0000 7f0caa400640 1 Processor -- start 2024-01-15T18:19:41.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.342+0000 7f0caa400640 1 -- start start 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.343+0000 7f0caa400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca40cdd70 0x7f0ca415df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.343+0000 7f0caa400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca4144c00 0x7f0ca4138a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.343+0000 7f0caa400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 0x7f0ca4155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.343+0000 7f0caa400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca4137a00 con 0x7f0ca415c450 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.343+0000 7f0caa400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41370f0 con 0x7f0ca4144c00 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.343+0000 7f0caa400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca41393f0 con 0x7f0ca40cdd70 2024-01-15T18:19:41.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 0x7f0ca4155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 0x7f0ca4155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46276/0 (socket says 172.21.15.2:46276) 2024-01-15T18:19:41.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 -- 172.21.15.2:0/3921813065 learned_addr learned my addr 172.21.15.2:0/3921813065 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:41.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca40cdd70 0x7f0ca415df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0c9a800640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca4144c00 0x7f0ca4138a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 -- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca40cdd70 msgr2=0x7f0ca415df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca40cdd70 0x7f0ca415df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 -- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca4144c00 msgr2=0x7f0ca4138a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca4144c00 0x7f0ca4138a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.348 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 -- 172.21.15.2:0/3921813065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c94009a70 con 0x7f0ca415c450 2024-01-15T18:19:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.344+0000 7f0ca3e00640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 0x7f0ca4155990 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c001eb0 tx=0x7f0c9c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.345+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c005ce0 con 0x7f0ca415c450 2024-01-15T18:19:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.345+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c9c005e40 con 0x7f0ca415c450 2024-01-15T18:19:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.345+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c0072d0 con 0x7f0ca415c450 2024-01-15T18:19:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.345+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca4137340 con 0x7f0ca415c450 2024-01-15T18:19:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.345+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca415e4a0 con 0x7f0ca415c450 2024-01-15T18:19:41.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.346+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca415e630 con 0x7f0ca415c450 2024-01-15T18:19:41.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.347+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0c9c0074f0 con 0x7f0ca415c450 2024-01-15T18:19:41.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.348+0000 7f0c9be00640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c7c06c550 0x7f0c7c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.348+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f0c9c087f50 con 0x7f0ca415c450 2024-01-15T18:19:41.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.349+0000 7f0ca3400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c7c06c550 0x7f0c7c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.350+0000 7f0ca3400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c7c06c550 0x7f0c7c06ea10 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0c94036000 tx=0x7f0c94030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:41.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.353+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c9c0545a0 con 0x7f0ca415c450 2024-01-15T18:19:41.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.544+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0ca40686f0 con 0x7f0ca415c450 2024-01-15T18:19:41.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.545+0000 7f0c9be00640 1 -- 172.21.15.2:0/3921813065 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c9c0170a0 con 0x7f0ca415c450 2024-01-15T18:19:41.549 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T18:19:41.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c7c06c550 msgr2=0x7f0c7c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c7c06c550 0x7f0c7c06ea10 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f0c94036000 tx=0x7f0c94030000 comp rx=0 tx=0).stop 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 msgr2=0x7f0ca4155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 0x7f0ca4155990 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c001eb0 tx=0x7f0c9c00c820 comp rx=0 tx=0).stop 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 shutdown_connections 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0c7c06c550 0x7f0c7c06ea10 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0ca415c450 0x7f0ca4155990 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0ca4144c00 0x7f0ca4138a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 --2- 172.21.15.2:0/3921813065 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0ca40cdd70 0x7f0ca415df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.552 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 >> 172.21.15.2:0/3921813065 conn(0x7f0ca40b45f0 msgr2=0x7f0ca415aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:41.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 shutdown_connections 2024-01-15T18:19:41.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.548+0000 7f0caa400640 1 -- 172.21.15.2:0/3921813065 wait complete. 2024-01-15T18:19:41.572 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:19:41.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.793+0000 7fca18600640 1 Processor -- start 2024-01-15T18:19:41.797 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.794+0000 7fca18600640 1 -- start start 2024-01-15T18:19:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.795+0000 7fca18600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca10144b90 0x7fca100b4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.795+0000 7fca18600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fca1006e830 0x7fca1006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.795+0000 7fca18600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca10070430 con 0x7fca1015b840 2024-01-15T18:19:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.795+0000 7fca18600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca1006f1e0 con 0x7fca10144b90 2024-01-15T18:19:41.799 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.795+0000 7fca18600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100b4760 con 0x7fca1006e830 2024-01-15T18:19:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca10144b90 0x7fca100b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fca10144b90 0x7fca100b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46292/0 (socket says 172.21.15.2:46292) 2024-01-15T18:19:41.800 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 -- 172.21.15.2:0/2773862114 learned_addr learned my addr 172.21.15.2:0/2773862114 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 -- 172.21.15.2:0/2773862114 >> v2:172.21.15.137:3300/0 conn(0x7fca1006e830 msgr2=0x7fca1006eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca16600640 1 --2- 172.21.15.2:0/2773862114 >> v2:172.21.15.137:3300/0 conn(0x7fca1006e830 0x7fca1006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 --2- 172.21.15.2:0/2773862114 >> v2:172.21.15.137:3300/0 conn(0x7fca1006e830 0x7fca1006eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 -- 172.21.15.2:0/2773862114 >> v1:172.21.15.137:6789/0 conn(0x7fca1015b840 legacy=0x7fca1006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca15c00640 1 -- 172.21.15.2:0/2773862114 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca10144f70 con 0x7fca10144b90 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.796+0000 7fca16600640 1 --2- 172.21.15.2:0/2773862114 >> v2:172.21.15.137:3300/0 conn(0x7fca1006e830 0x7fca1006eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.798+0000 7fca15c00640 1 --2- 172.21.15.2:0/2773862114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca10144b90 0x7fca100b4220 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fca10155d00 tx=0x7fca0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957016228dbc724d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.798+0000 7fca0fe00640 1 -- 172.21.15.2:0/2773862114 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00031870 con 0x7fca10144b90 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.798+0000 7fca0fe00640 1 -- 172.21.15.2:0/2773862114 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca000319d0 con 0x7fca10144b90 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.798+0000 7fca0fe00640 1 -- 172.21.15.2:0/2773862114 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca00031d60 con 0x7fca10144b90 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 -- 172.21.15.2:0/2773862114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca10144b90 msgr2=0x7fca100b4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 --2- 172.21.15.2:0/2773862114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca10144b90 0x7fca100b4220 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fca10155d00 tx=0x7fca0002b910 comp rx=0 tx=0).stop 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 -- 172.21.15.2:0/2773862114 shutdown_connections 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 --2- 172.21.15.2:0/2773862114 >> v2:172.21.15.137:3300/0 conn(0x7fca1006e830 0x7fca1006eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 --2- 172.21.15.2:0/2773862114 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca10144b90 0x7fca100b4220 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 -- 172.21.15.2:0/2773862114 >> 172.21.15.2:0/2773862114 conn(0x7fca100b2c10 msgr2=0x7fca100b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:41.803 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 -- 172.21.15.2:0/2773862114 shutdown_connections 2024-01-15T18:19:41.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.799+0000 7fca18600640 1 -- 172.21.15.2:0/2773862114 wait complete. 2024-01-15T18:19:41.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.801+0000 7fca18600640 1 Processor -- start 2024-01-15T18:19:41.804 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.801+0000 7fca18600640 1 -- start start 2024-01-15T18:19:41.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca18600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 0x7fca1013db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.805 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca18600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca10144b90 0x7fca10140310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca18600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca1015b840 0x7fca10140850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca18600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1013fd20 con 0x7fca1006e830 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca18600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1013f110 con 0x7fca1015b840 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca18600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca1013f3f0 con 0x7fca10144b90 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca15c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 0x7fca1013db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca15c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 0x7fca1013db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46294/0 (socket says 172.21.15.2:46294) 2024-01-15T18:19:41.806 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.802+0000 7fca15c00640 1 -- 172.21.15.2:0/650183892 learned_addr learned my addr 172.21.15.2:0/650183892 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca16600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca1015b840 0x7fca10140850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15200640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca10144b90 0x7fca10140310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15c00640 1 -- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca10144b90 msgr2=0x7fca10140310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15c00640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca10144b90 0x7fca10140310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15c00640 1 -- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca1015b840 msgr2=0x7fca10140850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15c00640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca1015b840 0x7fca10140850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15c00640 1 -- 172.21.15.2:0/650183892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca000094d0 con 0x7fca1006e830 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca16600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca1015b840 0x7fca10140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca15c00640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 0x7fca1013db70 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fca00000c00 tx=0x7fca000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0003adf0 con 0x7fca1006e830 2024-01-15T18:19:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca000443f0 con 0x7fca1006e830 2024-01-15T18:19:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca000447a0 con 0x7fca1006e830 2024-01-15T18:19:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.803+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca1013f6d0 con 0x7fca1006e830 2024-01-15T18:19:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.804+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca1013e0b0 con 0x7fca1006e830 2024-01-15T18:19:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.805+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca100686f0 con 0x7fca1006e830 2024-01-15T18:19:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.806+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fca0004d450 con 0x7fca1006e830 2024-01-15T18:19:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.807+0000 7fca0d600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e806c550 0x7fc9e806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:41.810 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.807+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fca0002d040 con 0x7fca1006e830 2024-01-15T18:19:41.811 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.808+0000 7fca15200640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e806c550 0x7fc9e806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:41.812 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.809+0000 7fca15200640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e806c550 0x7fc9e806ea10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fca100b5d10 tx=0x7fc9f8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:41.815 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:41.811+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca000777d0 con 0x7fca1006e830 2024-01-15T18:19:42.008 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.005+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fca1006f580 con 0x7fca1006e830 2024-01-15T18:19:42.009 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.006+0000 7fca0d600640 1 -- 172.21.15.2:0/650183892 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7fca00035070 con 0x7fca1006e830 2024-01-15T18:19:42.009 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T18:19:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e806c550 msgr2=0x7fc9e806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e806c550 0x7fc9e806ea10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fca100b5d10 tx=0x7fc9f8009040 comp rx=0 tx=0).stop 2024-01-15T18:19:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 msgr2=0x7fca1013db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.012 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 0x7fca1013db70 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fca00000c00 tx=0x7fca000046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 shutdown_connections 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc9e806c550 0x7fc9e806ea10 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fca1015b840 0x7fca10140850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.008+0000 7fca18600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fca10144b90 0x7fca10140310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.009+0000 7fca18600640 1 --2- 172.21.15.2:0/650183892 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fca1006e830 0x7fca1013db70 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.009+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 >> 172.21.15.2:0/650183892 conn(0x7fca100b2c10 msgr2=0x7fca100d0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.009+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 shutdown_connections 2024-01-15T18:19:42.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.009+0000 7fca18600640 1 -- 172.21.15.2:0/650183892 wait complete. 2024-01-15T18:19:42.031 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:19:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.250+0000 7f85f4800640 1 Processor -- start 2024-01-15T18:19:42.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.250+0000 7f85f4800640 1 -- start start 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.252+0000 7f85f4800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f85ec144b80 0x7f85ec0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.252+0000 7f85f4800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f85ec0b30a0 0x7f85ec06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.252+0000 7f85f4800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec06d220 con 0x7f85ec06cdf0 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.252+0000 7f85f4800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec0b4220 con 0x7f85ec0b30a0 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.252+0000 7f85f4800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec144f60 con 0x7f85ec144b80 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f85ec0b30a0 0x7f85ec06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f85ec0b30a0 0x7f85ec06c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46306/0 (socket says 172.21.15.2:46306) 2024-01-15T18:19:42.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 -- 172.21.15.2:0/107742109 learned_addr learned my addr 172.21.15.2:0/107742109 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:42.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 -- 172.21.15.2:0/107742109 >> v2:172.21.15.137:3300/0 conn(0x7f85ec144b80 msgr2=0x7f85ec0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:42.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 --2- 172.21.15.2:0/107742109 >> v2:172.21.15.137:3300/0 conn(0x7f85ec144b80 0x7f85ec0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 -- 172.21.15.2:0/107742109 >> v1:172.21.15.137:6789/0 conn(0x7f85ec06cdf0 legacy=0x7f85ec0b0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.253+0000 7f85f1400640 1 -- 172.21.15.2:0/107742109 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec0b35a0 con 0x7f85ec0b30a0 2024-01-15T18:19:42.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.254+0000 7f85f1400640 1 --2- 172.21.15.2:0/107742109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec0b30a0 0x7f85ec06c8b0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0b2940 tx=0x7f85d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b53bbc5df4b48fa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.255+0000 7f85f0a00640 1 -- 172.21.15.2:0/107742109 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d40317c0 con 0x7f85ec0b30a0 2024-01-15T18:19:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.255+0000 7f85f0a00640 1 -- 172.21.15.2:0/107742109 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d4031920 con 0x7f85ec0b30a0 2024-01-15T18:19:42.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.255+0000 7f85f0a00640 1 -- 172.21.15.2:0/107742109 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d4031cb0 con 0x7f85ec0b30a0 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.255+0000 7f85f4800640 1 -- 172.21.15.2:0/107742109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec0b30a0 msgr2=0x7f85ec06c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.255+0000 7f85f4800640 1 --2- 172.21.15.2:0/107742109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec0b30a0 0x7f85ec06c8b0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f85ec0b2940 tx=0x7f85d402b860 comp rx=0 tx=0).stop 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.256+0000 7f85f4800640 1 -- 172.21.15.2:0/107742109 shutdown_connections 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.256+0000 7f85f4800640 1 --2- 172.21.15.2:0/107742109 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec0b30a0 0x7f85ec06c8b0 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.256+0000 7f85f4800640 1 --2- 172.21.15.2:0/107742109 >> v2:172.21.15.137:3300/0 conn(0x7f85ec144b80 0x7f85ec0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.256+0000 7f85f4800640 1 -- 172.21.15.2:0/107742109 >> 172.21.15.2:0/107742109 conn(0x7f85ec0b46d0 msgr2=0x7f85ec0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.256+0000 7f85f4800640 1 -- 172.21.15.2:0/107742109 shutdown_connections 2024-01-15T18:19:42.260 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.256+0000 7f85f4800640 1 -- 172.21.15.2:0/107742109 wait complete. 2024-01-15T18:19:42.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.257+0000 7f85f4800640 1 Processor -- start 2024-01-15T18:19:42.261 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.258+0000 7f85f4800640 1 -- start start 2024-01-15T18:19:42.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f4800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85ec06cdf0 0x7f85ec13c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f4800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85ec0b30a0 0x7f85ec15a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f4800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 0x7f85ec15d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f4800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec13ca60 con 0x7f85ec144b80 2024-01-15T18:19:42.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f4800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec13a000 con 0x7f85ec0b30a0 2024-01-15T18:19:42.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f4800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec13ed30 con 0x7f85ec06cdf0 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 0x7f85ec15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f2800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 0x7f85ec15d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46318/0 (socket says 172.21.15.2:46318) 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f2800640 1 -- 172.21.15.2:0/920978788 learned_addr learned my addr 172.21.15.2:0/920978788 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f1e00640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85ec06cdf0 0x7f85ec13c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f1400640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85ec0b30a0 0x7f85ec15a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f2800640 1 -- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85ec06cdf0 msgr2=0x7f85ec13c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f2800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85ec06cdf0 0x7f85ec13c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.264 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.259+0000 7f85f2800640 1 -- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85ec0b30a0 msgr2=0x7f85ec15a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85f2800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85ec0b30a0 0x7f85ec15a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85f2800640 1 -- 172.21.15.2:0/920978788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d40094d0 con 0x7f85ec144b80 2024-01-15T18:19:42.265 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85f2800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 0x7f85ec15d570 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c7b0 tx=0x7f85e400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:42.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4004b60 con 0x7f85ec144b80 2024-01-15T18:19:42.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e4004cc0 con 0x7f85ec144b80 2024-01-15T18:19:42.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec13c830 con 0x7f85ec144b80 2024-01-15T18:19:42.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e400aa90 con 0x7f85ec144b80 2024-01-15T18:19:42.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.260+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec15a810 con 0x7f85ec144b80 2024-01-15T18:19:42.266 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.262+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec13a140 con 0x7f85ec144b80 2024-01-15T18:19:42.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.263+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f85e400abf0 con 0x7f85ec144b80 2024-01-15T18:19:42.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.264+0000 7f85ea000640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f85c406c550 0x7f85c406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.267 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.264+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f85e4088870 con 0x7f85ec144b80 2024-01-15T18:19:42.268 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.264+0000 7f85f1e00640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f85c406c550 0x7f85c406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.269 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.266+0000 7f85f1e00640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f85c406c550 0x7f85c406ea10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00b810 tx=0x7f85dc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:42.271 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.268+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85ec13a140 con 0x7f85ec144b80 2024-01-15T18:19:42.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.474+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f85ec15a810 con 0x7f85ec144b80 2024-01-15T18:19:42.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.475+0000 7f85ea000640 1 -- 172.21.15.2:0/920978788 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v116) v1 ==== 80+0+0 (secure 0 0 0) 0x7f85ec15a810 con 0x7f85ec144b80 2024-01-15T18:19:42.478 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T18:19:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f85c406c550 msgr2=0x7f85c406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f85c406c550 0x7f85c406ea10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00b810 tx=0x7f85dc008040 comp rx=0 tx=0).stop 2024-01-15T18:19:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 msgr2=0x7f85ec15d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 0x7f85ec15d570 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f85e400c7b0 tx=0x7f85e400cc70 comp rx=0 tx=0).stop 2024-01-15T18:19:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 shutdown_connections 2024-01-15T18:19:42.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f85c406c550 0x7f85c406ea10 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f85ec144b80 0x7f85ec15d570 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f85ec0b30a0 0x7f85ec15a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 --2- 172.21.15.2:0/920978788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f85ec06cdf0 0x7f85ec13c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 >> 172.21.15.2:0/920978788 conn(0x7f85ec0b46d0 msgr2=0x7f85ec0b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.478+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 shutdown_connections 2024-01-15T18:19:42.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.479+0000 7f85f4800640 1 -- 172.21.15.2:0/920978788 wait complete. 2024-01-15T18:19:42.500 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:19:42.500 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:42.554 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:19:42.779 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.776+0000 7f789a600640 1 Processor -- start 2024-01-15T18:19:42.780 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.776+0000 7f789a600640 1 -- start start 2024-01-15T18:19:42.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.778+0000 7f789a600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f789415c0e0 0x7f78940cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.781 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.778+0000 7f789a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f789415cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.778+0000 7f789a600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f789415cb00 con 0x7f789415b7d0 2024-01-15T18:19:42.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.778+0000 7f789a600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78940ccec0 con 0x7f789415c0e0 2024-01-15T18:19:42.782 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.778+0000 7f789a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78940cd1a0 con 0x7f789406cb80 2024-01-15T18:19:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.779+0000 7f7893400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f789415c0e0 0x7f78940cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.779+0000 7f7893e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f789415cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.779+0000 7f788a800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f789415b7d0 0x7f789406c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:60924/0 (socket says 172.21.15.2:60924) 2024-01-15T18:19:42.783 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.779+0000 7f788a800640 1 -- 172.21.15.2:0/3735800896 learned_addr learned my addr 172.21.15.2:0/3735800896 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.780+0000 7f7893e00640 1 -- 172.21.15.2:0/3735800896 >> v2:172.21.15.137:3300/0 conn(0x7f789415c0e0 msgr2=0x7f78940cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.780+0000 7f7893e00640 1 --2- 172.21.15.2:0/3735800896 >> v2:172.21.15.137:3300/0 conn(0x7f789415c0e0 0x7f78940cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.780+0000 7f7893e00640 1 -- 172.21.15.2:0/3735800896 >> v1:172.21.15.137:6789/0 conn(0x7f789415b7d0 legacy=0x7f789406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.784 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.780+0000 7f7893e00640 1 -- 172.21.15.2:0/3735800896 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78940cd480 con 0x7f789406cb80 2024-01-15T18:19:42.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.780+0000 7f7892a00640 1 -- 172.21.15.2:0/3735800896 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502975705 0 0) 0x7f789415cb00 con 0x7f789415b7d0 2024-01-15T18:19:42.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.780+0000 7f7893400640 1 --2- 172.21.15.2:0/3735800896 >> v2:172.21.15.137:3300/0 conn(0x7f789415c0e0 0x7f78940cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:42.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.781+0000 7f7893e00640 1 --2- 172.21.15.2:0/3735800896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f789415cf90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f788c009d90 tx=0x7f788c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf8216f9ed5f63e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:42.785 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.782+0000 7f7892a00640 1 -- 172.21.15.2:0/3735800896 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c02cc50 con 0x7f789406cb80 2024-01-15T18:19:42.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.782+0000 7f7892a00640 1 -- 172.21.15.2:0/3735800896 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c02cdb0 con 0x7f789406cb80 2024-01-15T18:19:42.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.782+0000 7f7892a00640 1 -- 172.21.15.2:0/3735800896 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c034660 con 0x7f789406cb80 2024-01-15T18:19:42.786 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 -- 172.21.15.2:0/3735800896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 msgr2=0x7f789415cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 --2- 172.21.15.2:0/3735800896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f789415cf90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f788c009d90 tx=0x7f788c02bea0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 -- 172.21.15.2:0/3735800896 shutdown_connections 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 --2- 172.21.15.2:0/3735800896 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f789415cf90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 --2- 172.21.15.2:0/3735800896 >> v2:172.21.15.137:3300/0 conn(0x7f789415c0e0 0x7f78940cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 -- 172.21.15.2:0/3735800896 >> 172.21.15.2:0/3735800896 conn(0x7f789406da20 msgr2=0x7f789406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 -- 172.21.15.2:0/3735800896 shutdown_connections 2024-01-15T18:19:42.787 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.783+0000 7f789a600640 1 -- 172.21.15.2:0/3735800896 wait complete. 2024-01-15T18:19:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.785+0000 7f789a600640 1 Processor -- start 2024-01-15T18:19:42.788 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.785+0000 7f789a600640 1 -- start start 2024-01-15T18:19:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.786+0000 7f789a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f78940b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.790 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.786+0000 7f789a600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f789415b7d0 0x7f789415a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.786+0000 7f789a600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 0x7f7894071760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.786+0000 7f789a600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7894147ce0 con 0x7f789415c0e0 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.786+0000 7f789a600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78940722b0 con 0x7f789415b7d0 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.786+0000 7f789a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7894153680 con 0x7f789406cb80 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 0x7f7894071760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 0x7f7894071760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:46322/0 (socket says 172.21.15.2:46322) 2024-01-15T18:19:42.791 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 -- 172.21.15.2:0/311260331 learned_addr learned my addr 172.21.15.2:0/311260331 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893400640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f78940b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f788a800640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f789415b7d0 0x7f789415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 -- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 msgr2=0x7f78940b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f78940b1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 -- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f789415b7d0 msgr2=0x7f789415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f789415b7d0 0x7f789415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 -- 172.21.15.2:0/311260331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7884009400 con 0x7f789415c0e0 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f788a800640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f789415b7d0 0x7f789415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:42.792 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.787+0000 7f7893e00640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 0x7f7894071760 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f788c009d90 tx=0x7f788c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:42.793 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.788+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c040650 con 0x7f789415c0e0 2024-01-15T18:19:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.788+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c0407b0 con 0x7f789415c0e0 2024-01-15T18:19:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.788+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c040b60 con 0x7f789415c0e0 2024-01-15T18:19:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.788+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f788c009a70 con 0x7f789415c0e0 2024-01-15T18:19:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.788+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7894153c40 con 0x7f789415c0e0 2024-01-15T18:19:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.789+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78940ce700 con 0x7f789415c0e0 2024-01-15T18:19:42.794 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.790+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f788c040cc0 con 0x7f789415c0e0 2024-01-15T18:19:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.791+0000 7f788be00640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786c06c580 0x7f786c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.791+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f788c0ab8b0 con 0x7f789415c0e0 2024-01-15T18:19:42.795 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.792+0000 7f7893400640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786c06c580 0x7f786c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:42.796 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.793+0000 7f7893400640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786c06c580 0x7f786c06ea40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7884009bc0 tx=0x7f7884008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:42.798 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.795+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f788c077e50 con 0x7f789415c0e0 2024-01-15T18:19:42.996 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.992+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78940686f0 con 0x7f789415c0e0 2024-01-15T18:19:42.998 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.994+0000 7f788be00640 1 -- 172.21.15.2:0/311260331 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16780 (secure 0 0 0) 0x7f788c009450 con 0x7f789415c0e0 2024-01-15T18:19:42.998 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:42.999 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":116,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:15:19.802397+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786c06c580 msgr2=0x7f786c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786c06c580 0x7f786c06ea40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7884009bc0 tx=0x7f7884008040 comp rx=0 tx=0).stop 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 msgr2=0x7f7894071760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 0x7f7894071760 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f788c009d90 tx=0x7f788c004a40 comp rx=0 tx=0).stop 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 shutdown_connections 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f786c06c580 0x7f786c06ea40 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.001 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f789415c0e0 0x7f7894071760 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f789415b7d0 0x7f789415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 --2- 172.21.15.2:0/311260331 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f789406cb80 0x7f78940b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 >> 172.21.15.2:0/311260331 conn(0x7f789406da20 msgr2=0x7f789406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:43.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 shutdown_connections 2024-01-15T18:19:43.002 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:42.997+0000 7f789a600640 1 -- 172.21.15.2:0/311260331 wait complete. 2024-01-15T18:19:43.021 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:14:14.206729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:14:15.347816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:19:43.021 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:19:43.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.239+0000 7f635a200640 1 Processor -- start 2024-01-15T18:19:43.242 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.239+0000 7f635a200640 1 -- start start 2024-01-15T18:19:43.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.240+0000 7f635a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6354160980 0x7f63540cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.240+0000 7f635a200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f63540b4040 0x7f63540cfc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.240+0000 7f635a200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63540ce7b0 con 0x7f63540d01a0 2024-01-15T18:19:43.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.240+0000 7f635a200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63540cf010 con 0x7f63540b4040 2024-01-15T18:19:43.244 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.240+0000 7f635a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63540cea50 con 0x7f6354160980 2024-01-15T18:19:43.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.241+0000 7f6358e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6354160980 0x7f63540cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.241+0000 7f6359800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f63540d01a0 0x7f63540d0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:54632/0 (socket says 172.21.15.2:54632) 2024-01-15T18:19:43.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.241+0000 7f6359800640 1 -- 172.21.15.2:0/2945226615 learned_addr learned my addr 172.21.15.2:0/2945226615 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:43.245 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.241+0000 7f6353e00640 1 --2- 172.21.15.2:0/2945226615 >> v2:172.21.15.137:3300/0 conn(0x7f63540b4040 0x7f63540cfc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.242+0000 7f6358e00640 1 -- 172.21.15.2:0/2945226615 >> v2:172.21.15.137:3300/0 conn(0x7f63540b4040 msgr2=0x7f63540cfc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:43.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.242+0000 7f6358e00640 1 --2- 172.21.15.2:0/2945226615 >> v2:172.21.15.137:3300/0 conn(0x7f63540b4040 0x7f63540cfc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.242+0000 7f6358e00640 1 -- 172.21.15.2:0/2945226615 >> v1:172.21.15.137:6789/0 conn(0x7f63540d01a0 legacy=0x7f63540d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.242+0000 7f6358e00640 1 -- 172.21.15.2:0/2945226615 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63540ced30 con 0x7f6354160980 2024-01-15T18:19:43.246 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.242+0000 7f6353400640 1 -- 172.21.15.2:0/2945226615 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335104614 0 0) 0x7f63540ce7b0 con 0x7f63540d01a0 2024-01-15T18:19:43.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.244+0000 7f6358e00640 1 --2- 172.21.15.2:0/2945226615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6354160980 0x7f63540cf720 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63440097a0 tx=0x7f634402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d63fab9503384453 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:43.247 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.244+0000 7f6353400640 1 -- 172.21.15.2:0/2945226615 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344031830 con 0x7f6354160980 2024-01-15T18:19:43.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.244+0000 7f6353400640 1 -- 172.21.15.2:0/2945226615 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6344031990 con 0x7f6354160980 2024-01-15T18:19:43.248 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.244+0000 7f6353400640 1 -- 172.21.15.2:0/2945226615 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6344031d20 con 0x7f6354160980 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 -- 172.21.15.2:0/2945226615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6354160980 msgr2=0x7f63540cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 --2- 172.21.15.2:0/2945226615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6354160980 0x7f63540cf720 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63440097a0 tx=0x7f634402b8d0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 -- 172.21.15.2:0/2945226615 shutdown_connections 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 --2- 172.21.15.2:0/2945226615 >> v2:172.21.15.137:3300/0 conn(0x7f63540b4040 0x7f63540cfc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 --2- 172.21.15.2:0/2945226615 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6354160980 0x7f63540cf720 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 -- 172.21.15.2:0/2945226615 >> 172.21.15.2:0/2945226615 conn(0x7f63540d2ff0 msgr2=0x7f63540d33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 -- 172.21.15.2:0/2945226615 shutdown_connections 2024-01-15T18:19:43.249 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.245+0000 7f635a200640 1 -- 172.21.15.2:0/2945226615 wait complete. 2024-01-15T18:19:43.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.247+0000 7f635a200640 1 Processor -- start 2024-01-15T18:19:43.250 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.247+0000 7f635a200640 1 -- start start 2024-01-15T18:19:43.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.248+0000 7f635a200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63540b4040 0x7f6354153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.248+0000 7f635a200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63540d01a0 0x7f63541fb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.248+0000 7f635a200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 0x7f63540b7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.248+0000 7f635a200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63540b83f0 con 0x7f6354160980 2024-01-15T18:19:43.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.248+0000 7f635a200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541fa9b0 con 0x7f63540b4040 2024-01-15T18:19:43.252 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.248+0000 7f635a200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63540b8600 con 0x7f63540d01a0 2024-01-15T18:19:43.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 0x7f63540b7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 0x7f63540b7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41544/0 (socket says 172.21.15.2:41544) 2024-01-15T18:19:43.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 -- 172.21.15.2:0/3979827158 learned_addr learned my addr 172.21.15.2:0/3979827158 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:43.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6358e00640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63540b4040 0x7f6354153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.253 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6353e00640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63540d01a0 0x7f63541fb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 -- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63540d01a0 msgr2=0x7f63541fb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.254 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63540d01a0 0x7f63541fb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 -- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63540b4040 msgr2=0x7f6354153740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63540b4040 0x7f6354153740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.249+0000 7f6359800640 1 -- 172.21.15.2:0/3979827158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344009400 con 0x7f6354160980 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.250+0000 7f6359800640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 0x7f63540b7eb0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f634c00c700 tx=0x7f634c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.250+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00ce10 con 0x7f6354160980 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.250+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f634c00a640 con 0x7f6354160980 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.250+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f634c00aa70 con 0x7f6354160980 2024-01-15T18:19:43.255 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.250+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63540b88e0 con 0x7f6354160980 2024-01-15T18:19:43.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.250+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63541faf50 con 0x7f6354160980 2024-01-15T18:19:43.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.251+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63541fb140 con 0x7f6354160980 2024-01-15T18:19:43.256 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.253+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f634c00abd0 con 0x7f6354160980 2024-01-15T18:19:43.257 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.254+0000 7f6350c00640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f632c06c550 0x7f632c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.254+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f634c088050 con 0x7f6354160980 2024-01-15T18:19:43.258 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.255+0000 7f6358e00640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f632c06c550 0x7f632c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.259 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.256+0000 7f6358e00640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f632c06c550 0x7f632c06ea10 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6344036000 tx=0x7f634403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:43.263 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.260+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f634c0545f0 con 0x7f6354160980 2024-01-15T18:19:43.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.453+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6354074770 con 0x7f6354160980 2024-01-15T18:19:43.457 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.454+0000 7f6350c00640 1 -- 172.21.15.2:0/3979827158 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f634c018030 con 0x7f6354160980 2024-01-15T18:19:43.458 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:19:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f632c06c550 msgr2=0x7f632c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.460 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f632c06c550 0x7f632c06ea10 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6344036000 tx=0x7f634403c040 comp rx=0 tx=0).stop 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 msgr2=0x7f63540b7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 0x7f63540b7eb0 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7f634c00c700 tx=0x7f634c00cbc0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 shutdown_connections 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f632c06c550 0x7f632c06ea10 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6354160980 0x7f63540b7eb0 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f63540d01a0 0x7f63541fb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 --2- 172.21.15.2:0/3979827158 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f63540b4040 0x7f6354153740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.461 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 >> 172.21.15.2:0/3979827158 conn(0x7f63540d2ff0 msgr2=0x7f635415e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 shutdown_connections 2024-01-15T18:19:43.462 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.457+0000 7f635a200640 1 -- 172.21.15.2:0/3979827158 wait complete. 2024-01-15T18:19:43.478 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:19:43.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.693+0000 7fa820c00640 1 Processor -- start 2024-01-15T18:19:43.697 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.694+0000 7fa820c00640 1 -- start start 2024-01-15T18:19:43.698 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.695+0000 7fa820c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa81c0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.695+0000 7fa820c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c0d1250 con 0x7fa81c0b2210 2024-01-15T18:19:43.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.695+0000 7fa820c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa81c0ce8e0 con 0x7fa81c0d1870 2024-01-15T18:19:43.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.695+0000 7fa820c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0ce6d0 con 0x7fa81c0d1c50 2024-01-15T18:19:43.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa813e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fa81c0b2210 0x7fa81c0b26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46454/0 (socket says 172.21.15.2:46454) 2024-01-15T18:19:43.699 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa813e00640 1 -- 172.21.15.2:0/2406498354 learned_addr learned my addr 172.21.15.2:0/2406498354 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:43.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81be00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa81c0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.701 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 242578482 0 0) 0x7fa81c0d1250 con 0x7fa81c0b2210 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7fc003520 con 0x7fa81c0b2210 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 491956552 0 0) 0x7fa7fc003520 con 0x7fa81c0b2210 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 msgr2=0x7fa81c0d3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa81c0d3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 >> v1:172.21.15.137:6789/0 conn(0x7fa81c0d1870 legacy=0x7fa81c15d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c0cebc0 con 0x7fa81c0b2210 2024-01-15T18:19:43.702 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fa804001670 con 0x7fa81c0b2210 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.696+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 >> v1:172.21.15.2:6789/0 conn(0x7fa81c0b2210 legacy=0x7fa81c0b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81aa00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0039a0 0x7fa7fc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81aa00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81aa00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa7fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c0ce6d0 con 0x7fa7fc0047c0 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc00ba60 con 0x7fa7fc0039a0 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7fc00bc20 con 0x7fa81c0d1c50 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa813e00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.703 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81be00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa7fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa813e00640 1 -- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 msgr2=0x7fa7fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81b400640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0039a0 0x7fa7fc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa813e00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa7fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa813e00640 1 -- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0039a0 msgr2=0x7fa7fc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa813e00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0039a0 0x7fa7fc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa813e00640 1 -- 172.21.15.2:0/2406498354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc003520 con 0x7fa7fc0047c0 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81b400640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0039a0 0x7fa7fc003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.697+0000 7fa81be00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c0d1c50 0x7fa7fc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:43.704 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.698+0000 7fa813e00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0b2650 tx=0x7fa7fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48043448f7f9c892 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:43.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa804005ad0 con 0x7fa7fc0047c0 2024-01-15T18:19:43.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa804005c30 con 0x7fa7fc0047c0 2024-01-15T18:19:43.707 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa81aa00640 1 -- 172.21.15.2:0/2406498354 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa80402c830 con 0x7fa7fc0047c0 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa820c00640 1 -- 172.21.15.2:0/2406498354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7fc0047c0 msgr2=0x7fa7fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa820c00640 1 --2- 172.21.15.2:0/2406498354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa7fc0047c0 0x7fa7fc004c40 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0b2650 tx=0x7fa7fc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa81be00640 1 -- 172.21.15.2:0/2406498354 reap_dead start 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa820c00640 1 -- 172.21.15.2:0/2406498354 shutdown_connections 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa820c00640 1 -- 172.21.15.2:0/2406498354 >> 172.21.15.2:0/2406498354 conn(0x7fa81c0d30b0 msgr2=0x7fa81c0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa820c00640 1 -- 172.21.15.2:0/2406498354 shutdown_connections 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.699+0000 7fa820c00640 1 -- 172.21.15.2:0/2406498354 wait complete. 2024-01-15T18:19:43.708 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.701+0000 7fa820c00640 1 Processor -- start 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.701+0000 7fa820c00640 1 -- start start 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.702+0000 7fa820c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0047c0 0x7fa81c0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.702+0000 7fa820c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c1f6320 0x7fa81c160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.702+0000 7fa820c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 0x7fa81c1f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.702+0000 7fa820c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c15dbe0 con 0x7fa81c06b540 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.702+0000 7fa820c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1f6800 con 0x7fa7fc0047c0 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.702+0000 7fa820c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81c1f6ae0 con 0x7fa81c1f6320 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 0x7fa81c1f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.709 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 0x7fa81c1f2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41562/0 (socket says 172.21.15.2:41562) 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 -- 172.21.15.2:0/3441031580 learned_addr learned my addr 172.21.15.2:0/3441031580 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81b400640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0047c0 0x7fa81c0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 -- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c1f6320 msgr2=0x7fa81c160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c1f6320 0x7fa81c160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 -- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0047c0 msgr2=0x7fa81c0b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0047c0 0x7fa81c0b6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 -- 172.21.15.2:0/3441031580 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81c1f6dc0 con 0x7fa81c06b540 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.703+0000 7fa81be00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 0x7fa81c1f2170 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0b9160 tx=0x7fa81400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:43.710 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.704+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814017400 con 0x7fa81c06b540 2024-01-15T18:19:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.704+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa814011040 con 0x7fa81c06b540 2024-01-15T18:19:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.704+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8140177c0 con 0x7fa81c06b540 2024-01-15T18:19:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.704+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa81c1f70a0 con 0x7fa81c06b540 2024-01-15T18:19:43.711 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.704+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa81c06b9d0 con 0x7fa81c06b540 2024-01-15T18:19:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.705+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa81c074770 con 0x7fa81c06b540 2024-01-15T18:19:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.706+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fa8140203b0 con 0x7fa81c06b540 2024-01-15T18:19:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.707+0000 7fa813400640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7f406c580 0x7fa7f406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.707+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fa814088f70 con 0x7fa81c06b540 2024-01-15T18:19:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.708+0000 7fa81b400640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7f406c580 0x7fa7f406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:43.712 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.709+0000 7fa81b400640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7f406c580 0x7fa7f406ea40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0015b0 tx=0x7fa80c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:43.715 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.711+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8140554b0 con 0x7fa81c06b540 2024-01-15T18:19:43.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.901+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa81c0d2e90 con 0x7fa81c06b540 2024-01-15T18:19:43.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.902+0000 7fa813400640 1 -- 172.21.15.2:0/3441031580 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa814058ad0 con 0x7fa81c06b540 2024-01-15T18:19:43.905 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T18:19:43.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7f406c580 msgr2=0x7fa7f406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7f406c580 0x7fa7f406ea40 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa80c0015b0 tx=0x7fa80c009040 comp rx=0 tx=0).stop 2024-01-15T18:19:43.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 msgr2=0x7fa81c1f2170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:43.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 0x7fa81c1f2170 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7fa81c0b9160 tx=0x7fa81400f4a0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 shutdown_connections 2024-01-15T18:19:43.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fa7f406c580 0x7fa7f406ea40 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fa81c06b540 0x7fa81c1f2170 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fa81c1f6320 0x7fa81c160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 --2- 172.21.15.2:0/3441031580 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fa7fc0047c0 0x7fa81c0b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:43.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.904+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 >> 172.21.15.2:0/3441031580 conn(0x7fa81c0d30b0 msgr2=0x7fa81c0cc190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:43.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.905+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 shutdown_connections 2024-01-15T18:19:43.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:43.905+0000 7fa820c00640 1 -- 172.21.15.2:0/3441031580 wait complete. 2024-01-15T18:19:43.925 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:19:44.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.167+0000 7fc5bcc00640 1 Processor -- start 2024-01-15T18:19:44.170 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.167+0000 7fc5bcc00640 1 -- start start 2024-01-15T18:19:44.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.168+0000 7fc5bcc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b815c450 0x7fc5b806c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.169+0000 7fc5bcc00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80ce230 con 0x7fc5b806ce30 2024-01-15T18:19:44.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.169+0000 7fc5bcc00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b80ce510 con 0x7fc5b8144c00 2024-01-15T18:19:44.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.169+0000 7fc5bcc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b806d2c0 con 0x7fc5b815c450 2024-01-15T18:19:44.172 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.169+0000 7fc5b6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc5b806ce30 0x7fc5b8070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46456/0 (socket says 172.21.15.2:46456) 2024-01-15T18:19:44.173 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.169+0000 7fc5b6800640 1 -- 172.21.15.2:0/3515859063 learned_addr learned my addr 172.21.15.2:0/3515859063 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.169+0000 7fc5b5400640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b815c450 0x7fc5b806c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400868174 0 0) 0x7fc5b80ce230 con 0x7fc5b806ce30 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc598003520 con 0x7fc5b806ce30 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491481228 0 0) 0x7fc5b80ce510 con 0x7fc5b8144c00 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 --> v1:172.21.15.137:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b80ce230 con 0x7fc5b8144c00 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2588025773 0 0) 0x7fc598003520 con 0x7fc5b806ce30 2024-01-15T18:19:44.174 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b815c450 msgr2=0x7fc5b806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b815c450 0x7fc5b806c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 >> v1:172.21.15.137:6789/0 conn(0x7fc5b8144c00 legacy=0x7fc5b80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80ce7f0 con 0x7fc5b806ce30 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fc5b00019f0 con 0x7fc5b806ce30 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 >> v1:172.21.15.2:6789/0 conn(0x7fc5b806ce30 legacy=0x7fc5b8070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc598003b30 0x7fc598003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.170+0000 7fc5b4a00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc598004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b4a00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc598005180 0x7fc59800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.175 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b806d2c0 con 0x7fc5980047c0 2024-01-15T18:19:44.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59800bfa0 con 0x7fc598003b30 2024-01-15T18:19:44.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc59800c160 con 0x7fc598005180 2024-01-15T18:19:44.176 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b5400640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b815c450 0x7fc5b806c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b5400640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc598004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b5e00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc598003b30 0x7fc598003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.171+0000 7fc5b6800640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc598005180 0x7fc59800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.172+0000 7fc5b5400640 1 -- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc598005180 msgr2=0x7fc59800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.172+0000 7fc5b5400640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc598005180 0x7fc59800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.172+0000 7fc5b5400640 1 -- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc598003b30 msgr2=0x7fc598003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:44.178 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.172+0000 7fc5b5400640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc598003b30 0x7fc598003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.172+0000 7fc5b5400640 1 -- 172.21.15.2:0/3515859063 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc598003520 con 0x7fc5980047c0 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.172+0000 7fc5b5e00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc598003b30 0x7fc598003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.173+0000 7fc5b5400640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc598004c40 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80cd5a0 tx=0x7fc5980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c1b42f5132559f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.173+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0004030 con 0x7fc5980047c0 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.173+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a0004190 con 0x7fc5980047c0 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.173+0000 7fc5b4a00640 1 -- 172.21.15.2:0/3515859063 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0004520 con 0x7fc5980047c0 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.174+0000 7fc5bcc00640 1 -- 172.21.15.2:0/3515859063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 msgr2=0x7fc598004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.174+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/3515859063 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc598004c40 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80cd5a0 tx=0x7fc5980057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.179 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.174+0000 7fc5b6800640 1 -- 172.21.15.2:0/3515859063 reap_dead start 2024-01-15T18:19:44.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.174+0000 7fc5bcc00640 1 -- 172.21.15.2:0/3515859063 shutdown_connections 2024-01-15T18:19:44.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.174+0000 7fc5bcc00640 1 -- 172.21.15.2:0/3515859063 >> 172.21.15.2:0/3515859063 conn(0x7fc5b80b45f0 msgr2=0x7fc5b80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:44.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.175+0000 7fc5bcc00640 1 -- 172.21.15.2:0/3515859063 shutdown_connections 2024-01-15T18:19:44.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.175+0000 7fc5bcc00640 1 -- 172.21.15.2:0/3515859063 wait complete. 2024-01-15T18:19:44.180 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.177+0000 7fc5bcc00640 1 Processor -- start 2024-01-15T18:19:44.181 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.177+0000 7fc5bcc00640 1 -- start start 2024-01-15T18:19:44.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5bcc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc5b81ef0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5bcc00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b813b810 0x7fc5b81f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5bcc00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5b8155400 0x7fc5b815d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5bcc00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8155d50 con 0x7fc5980047c0 2024-01-15T18:19:44.182 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5bcc00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8151510 con 0x7fc5b8155400 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5bcc00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b8150be0 con 0x7fc5b813b810 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5b5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc5b81ef0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5b5e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc5b81ef0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41576/0 (socket says 172.21.15.2:41576) 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5b5e00640 1 -- 172.21.15.2:0/2245409019 learned_addr learned my addr 172.21.15.2:0/2245409019 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5b6800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5b8155400 0x7fc5b815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.178+0000 7fc5b5400640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b813b810 0x7fc5b81f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.179+0000 7fc5b6800640 1 -- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b813b810 msgr2=0x7fc5b81f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.183 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.179+0000 7fc5b6800640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b813b810 0x7fc5b81f1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.179+0000 7fc5b6800640 1 -- 172.21.15.2:0/2245409019 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 msgr2=0x7fc5b81ef0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.179+0000 7fc5b6800640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc5b81ef0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.184 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.179+0000 7fc5b6800640 1 -- 172.21.15.2:0/2245409019 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b8150ec0 con 0x7fc5b8155400 2024-01-15T18:19:44.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.180+0000 7fc5b6800640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5b8155400 0x7fc5b815d880 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80cf0c0 tx=0x7fc5b000a600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:44.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.180+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0007af0 con 0x7fc5b8155400 2024-01-15T18:19:44.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.180+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b000ae10 con 0x7fc5b8155400 2024-01-15T18:19:44.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.180+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5b00135f0 con 0x7fc5b8155400 2024-01-15T18:19:44.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.180+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b81511a0 con 0x7fc5b8155400 2024-01-15T18:19:44.185 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.180+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b8155880 con 0x7fc5b8155400 2024-01-15T18:19:44.186 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.182+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8150d20 con 0x7fc5b8155400 2024-01-15T18:19:44.187 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.184+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc5b0013750 con 0x7fc5b8155400 2024-01-15T18:19:44.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.186+0000 7fc5ae000640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc59006c5a0 0x7fc59006ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.186+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fc5b0088de0 con 0x7fc5b8155400 2024-01-15T18:19:44.191 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.187+0000 7fc5b5e00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc59006c5a0 0x7fc59006ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.193 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.190+0000 7fc5b5e00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc59006c5a0 0x7fc59006ea60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8001870 tx=0x7fc5a800e040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:44.198 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.194+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5b8150d20 con 0x7fc5b8155400 2024-01-15T18:19:44.504 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.501+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5b8150d20 con 0x7fc5b8155400 2024-01-15T18:19:44.506 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.503+0000 7fc5ae000640 1 -- 172.21.15.2:0/2245409019 <== mon.1 v2:172.21.15.137:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5b8150d20 con 0x7fc5b8155400 2024-01-15T18:19:44.506 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T18:19:44.510 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc59006c5a0 msgr2=0x7fc59006ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc59006c5a0 0x7fc59006ea60 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8001870 tx=0x7fc5a800e040 comp rx=0 tx=0).stop 2024-01-15T18:19:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5b8155400 msgr2=0x7fc5b815d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5b8155400 0x7fc5b815d880 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fc5b80cf0c0 tx=0x7fc5b000a600 comp rx=0 tx=0).stop 2024-01-15T18:19:44.511 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 shutdown_connections 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc59006c5a0 0x7fc59006ea60 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc5b8155400 0x7fc5b815d880 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc5b813b810 0x7fc5b81f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.507+0000 7fc5bcc00640 1 --2- 172.21.15.2:0/2245409019 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc5980047c0 0x7fc5b81ef0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.508+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 >> 172.21.15.2:0/2245409019 conn(0x7fc5b80b45f0 msgr2=0x7fc5b80714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.508+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 shutdown_connections 2024-01-15T18:19:44.512 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.508+0000 7fc5bcc00640 1 -- 172.21.15.2:0/2245409019 wait complete. 2024-01-15T18:19:44.544 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:19:44.833 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.830+0000 7f6792e00640 1 Processor -- start 2024-01-15T18:19:44.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.830+0000 7f6792e00640 1 -- start start 2024-01-15T18:19:44.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.833+0000 7f6792e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c06cfc0 0x7f678c0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.833+0000 7f6792e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f678c0cbed0 0x7f678c0b20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.833+0000 7f6792e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c143dc0 con 0x7f678c1439e0 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.833+0000 7f6792e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c15b620 con 0x7f678c0cbed0 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.833+0000 7f6792e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0cd9f0 con 0x7f678c06cfc0 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.834+0000 7f6790e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f678c0cbed0 0x7f678c0b20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.834+0000 7f6790e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f678c0cbed0 0x7f678c0b20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41580/0 (socket says 172.21.15.2:41580) 2024-01-15T18:19:44.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.834+0000 7f6790e00640 1 -- 172.21.15.2:0/1737830106 learned_addr learned my addr 172.21.15.2:0/1737830106 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.834+0000 7f6790e00640 1 -- 172.21.15.2:0/1737830106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c06cfc0 msgr2=0x7f678c0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.834+0000 7f6790e00640 1 --2- 172.21.15.2:0/1737830106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c06cfc0 0x7f678c0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.834+0000 7f6790e00640 1 -- 172.21.15.2:0/1737830106 >> v1:172.21.15.137:6789/0 conn(0x7f678c1439e0 legacy=0x7f678c06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.835+0000 7f6790e00640 1 -- 172.21.15.2:0/1737830106 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0cc570 con 0x7f678c0cbed0 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.836+0000 7f6790e00640 1 --2- 172.21.15.2:0/1737830106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c0b20d0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f6784009e50 tx=0x7f678402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844b2c8a3587d122 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.837+0000 7f678aa00640 1 -- 172.21.15.2:0/1737830106 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678402ca80 con 0x7f678c0cbed0 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.837+0000 7f678aa00640 1 -- 172.21.15.2:0/1737830106 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678402cbe0 con 0x7f678c0cbed0 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.837+0000 7f678aa00640 1 -- 172.21.15.2:0/1737830106 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67840343f0 con 0x7f678c0cbed0 2024-01-15T18:19:44.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 -- 172.21.15.2:0/1737830106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 msgr2=0x7f678c0b20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 --2- 172.21.15.2:0/1737830106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c0b20d0 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f6784009e50 tx=0x7f678402bee0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 -- 172.21.15.2:0/1737830106 shutdown_connections 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 --2- 172.21.15.2:0/1737830106 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c0b20d0 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 --2- 172.21.15.2:0/1737830106 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c06cfc0 0x7f678c0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 -- 172.21.15.2:0/1737830106 >> 172.21.15.2:0/1737830106 conn(0x7f678c0b46d0 msgr2=0x7f678c0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 -- 172.21.15.2:0/1737830106 shutdown_connections 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.838+0000 7f6792e00640 1 -- 172.21.15.2:0/1737830106 wait complete. 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.841+0000 7f6792e00640 1 Processor -- start 2024-01-15T18:19:44.892 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.841+0000 7f6792e00640 1 -- start start 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f6792e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c06cfc0 0x7f678c0718d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f6792e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f6792e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c1439e0 0x7f678c1523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f6792e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c134c60 con 0x7f678c0cbed0 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f6792e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15a3c0 con 0x7f678c06cfc0 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f6792e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0b07b0 con 0x7f678c1439e0 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f678b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.842+0000 7f678b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41582/0 (socket says 172.21.15.2:41582) 2024-01-15T18:19:44.893 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 -- 172.21.15.2:0/1354475519 learned_addr learned my addr 172.21.15.2:0/1354475519 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678be00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c06cfc0 0x7f678c0718d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f6790e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c1439e0 0x7f678c1523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 -- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c1439e0 msgr2=0x7f678c1523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c1439e0 0x7f678c1523f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 -- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c06cfc0 msgr2=0x7f678c0718d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c06cfc0 0x7f678c0718d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 -- 172.21.15.2:0/1354475519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6784009b30 con 0x7f678c0cbed0 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.843+0000 7f678b400640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c159a50 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f677400b4a0 tx=0x7f677400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:44.894 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.844+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774010610 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.844+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774010770 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.844+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774010b20 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.844+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c0b0a90 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.844+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c0b1050 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.845+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c0b1270 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.847+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f6774010c80 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.848+0000 7f6783e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f676406c550 0x7f676406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.849+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f6774088380 con 0x7f678c0cbed0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.853+0000 7f678be00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f676406c550 0x7f676406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:44.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.854+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6774054cf0 con 0x7f678c0cbed0 2024-01-15T18:19:44.896 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:44.863+0000 7f678be00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f676406c550 0x7f676406ea10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f678c0ae570 tx=0x7f677c009060 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:45.189 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.186+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f678c0686f0 con 0x7f678c0cbed0 2024-01-15T18:19:45.190 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.187+0000 7f6783e00640 1 -- 172.21.15.2:0/1354475519 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v116) v1 ==== 80+0+0 (secure 0 0 0) 0x7f677401e020 con 0x7f678c0cbed0 2024-01-15T18:19:45.190 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T18:19:45.194 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f676406c550 msgr2=0x7f676406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f676406c550 0x7f676406ea10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f678c0ae570 tx=0x7f677c009060 comp rx=0 tx=0).stop 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 msgr2=0x7f678c159a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c159a50 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f677400b4a0 tx=0x7f677400b960 comp rx=0 tx=0).stop 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 shutdown_connections 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f676406c550 0x7f676406ea10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f678c1439e0 0x7f678c1523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f678c0cbed0 0x7f678c159a50 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 --2- 172.21.15.2:0/1354475519 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f678c06cfc0 0x7f678c0718d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:45.195 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 >> 172.21.15.2:0/1354475519 conn(0x7f678c0b46d0 msgr2=0x7f678c06c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:45.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 shutdown_connections 2024-01-15T18:19:45.196 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:45.191+0000 7f6792e00640 1 -- 172.21.15.2:0/1354475519 wait complete. 2024-01-15T18:19:45.220 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi002.front.sepia.ceph.com... 2024-01-15T18:19:45.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:45.221 DEBUG:teuthology.orchestra.run.smithi002:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-15T18:19:45.310 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi002.stderr:ceph-fuse[76355]: fuse finished with error 0 and tester_r 0 2024-01-15T18:19:46.111 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:19:46.111 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:19:46.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:46.112 DEBUG:teuthology.orchestra.run.smithi002:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-15T18:19:46.164 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-01-15T18:19:46.164 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:46.215 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-15T18:19:46.215 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:19:46.216 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:46.272 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:19:46.273 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:46.319 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:19:46.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.537+0000 7fcd11800640 1 Processor -- start 2024-01-15T18:19:46.541 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.538+0000 7fcd11800640 1 -- start start 2024-01-15T18:19:46.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.539+0000 7fcd11800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcd0c15c0e0 0x7fcd0c0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.539+0000 7fcd11800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c15cb00 con 0x7fcd0c06cb80 2024-01-15T18:19:46.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.539+0000 7fcd11800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c0ccec0 con 0x7fcd0c15b7d0 2024-01-15T18:19:46.543 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.539+0000 7fcd11800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0c0cd1a0 con 0x7fcd0c15c0e0 2024-01-15T18:19:46.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.540+0000 7fcd0b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fcd0c06cb80 0x7fcd0c15cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46460/0 (socket says 172.21.15.2:46460) 2024-01-15T18:19:46.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.540+0000 7fcd0b400640 1 -- 172.21.15.2:0/1234746717 learned_addr learned my addr 172.21.15.2:0/1234746717 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:46.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.540+0000 7fcd0aa00640 1 --2- 172.21.15.2:0/1234746717 >> v2:172.21.15.137:3300/0 conn(0x7fcd0c15c0e0 0x7fcd0c0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.545 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.540+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502384607 0 0) 0x7fcd0c15cb00 con 0x7fcd0c06cb80 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.540+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccec003520 con 0x7fcd0c06cb80 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1617678398 0 0) 0x7fccec003520 con 0x7fcd0c06cb80 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 >> v2:172.21.15.137:3300/0 conn(0x7fcd0c15c0e0 msgr2=0x7fcd0c0cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 --2- 172.21.15.2:0/1234746717 >> v2:172.21.15.137:3300/0 conn(0x7fcd0c15c0e0 0x7fcd0c0cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 >> v1:172.21.15.137:6789/0 conn(0x7fcd0c15b7d0 legacy=0x7fcd0c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c0cd480 con 0x7fcd0c06cb80 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fcd04001670 con 0x7fcd0c06cb80 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 >> v1:172.21.15.2:6789/0 conn(0x7fcd0c06cb80 legacy=0x7fcd0c15cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.546 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccec003970 0x7fccec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec0047c0 0x7fccec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccec005180 0x7fccec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0cd1a0 con 0x7fccec005180 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec00c020 con 0x7fccec0047c0 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec00c1e0 con 0x7fccec003970 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.541+0000 7fcd0b400640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccec005180 0x7fccec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0a000640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec0047c0 0x7fccec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0aa00640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccec003970 0x7fccec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.547 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0b400640 1 -- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccec003970 msgr2=0x7fccec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:46.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0b400640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fccec003970 0x7fccec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0b400640 1 -- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec0047c0 msgr2=0x7fccec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:46.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0b400640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec0047c0 0x7fccec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.548 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.542+0000 7fcd0b400640 1 -- 172.21.15.2:0/1234746717 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec003520 con 0x7fccec005180 2024-01-15T18:19:46.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.543+0000 7fcd0b400640 1 -- 172.21.15.2:0/1234746717 reap_dead start 2024-01-15T18:19:46.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.543+0000 7fcd0b400640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccec005180 0x7fccec00b980 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c190 tx=0x7fccec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4669defe891dbc server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.543+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04005ad0 con 0x7fccec005180 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.543+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd04005c70 con 0x7fccec005180 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.543+0000 7fcd09600640 1 -- 172.21.15.2:0/1234746717 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0402c8a0 con 0x7fccec005180 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 -- 172.21.15.2:0/1234746717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccec005180 msgr2=0x7fccec00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccec005180 0x7fccec00b980 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c190 tx=0x7fccec0057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 -- 172.21.15.2:0/1234746717 shutdown_connections 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 --2- 172.21.15.2:0/1234746717 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fccec005180 0x7fccec00b980 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 -- 172.21.15.2:0/1234746717 >> 172.21.15.2:0/1234746717 conn(0x7fcd0c06da20 msgr2=0x7fcd0c06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:46.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 -- 172.21.15.2:0/1234746717 shutdown_connections 2024-01-15T18:19:46.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.544+0000 7fcd11800640 1 -- 172.21.15.2:0/1234746717 wait complete. 2024-01-15T18:19:46.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.546+0000 7fcd11800640 1 Processor -- start 2024-01-15T18:19:46.551 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.546+0000 7fcd11800640 1 -- start start 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd11800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec005180 0x7fcd0c1537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd11800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd0c071560 0x7fcd0c146c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd11800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 0x7fcd0c15a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd11800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c152710 con 0x7fcd0c14a390 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd11800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c1529f0 con 0x7fccec005180 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd11800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c152cd0 con 0x7fcd0c071560 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd0b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 0x7fcd0c15a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.553 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd0b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 0x7fcd0c15a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41608/0 (socket says 172.21.15.2:41608) 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.547+0000 7fcd0b400640 1 -- 172.21.15.2:0/2412122565 learned_addr learned my addr 172.21.15.2:0/2412122565 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0aa00640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec005180 0x7fcd0c1537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0a000640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd0c071560 0x7fcd0c146c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0b400640 1 -- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd0c071560 msgr2=0x7fcd0c146c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0b400640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd0c071560 0x7fcd0c146c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0b400640 1 -- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec005180 msgr2=0x7fcd0c1537b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0b400640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec005180 0x7fcd0c1537b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0b400640 1 -- 172.21.15.2:0/2412122565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c152fb0 con 0x7fcd0c14a390 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0a000640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd0c071560 0x7fcd0c146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd0b400640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 0x7fcd0c15a130 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c190 tx=0x7fcd0c15ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.548+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd040049e0 con 0x7fcd0c14a390 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.549+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd04004b40 con 0x7fcd0c14a390 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.549+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd04001640 con 0x7fcd0c14a390 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.549+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c153290 con 0x7fcd0c14a390 2024-01-15T18:19:46.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.549+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c14aa00 con 0x7fcd0c14a390 2024-01-15T18:19:46.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.550+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0c0ce3e0 con 0x7fcd0c14a390 2024-01-15T18:19:46.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.551+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fcd04032070 con 0x7fcd0c14a390 2024-01-15T18:19:46.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.552+0000 7fcd02a00640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcce406c550 0x7fcce406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:46.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.552+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fcd0403c070 con 0x7fcd0c14a390 2024-01-15T18:19:46.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.553+0000 7fcd0aa00640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcce406c550 0x7fcce406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:46.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.554+0000 7fcd0aa00640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcce406c550 0x7fcce406ea10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009fd0 tx=0x7fccfc00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:46.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.556+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd04075c60 con 0x7fcd0c14a390 2024-01-15T18:19:46.756 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.753+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd0c0686f0 con 0x7fcd0c14a390 2024-01-15T18:19:46.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.756+0000 7fcd02a00640 1 -- 172.21.15.2:0/2412122565 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16780 (secure 0 0 0) 0x7fcd04044020 con 0x7fcd0c14a390 2024-01-15T18:19:46.759 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:46.760 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":116,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:15:19.802397+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:19:46.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.758+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcce406c550 msgr2=0x7fcce406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.758+0000 7fcd11800640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcce406c550 0x7fcce406ea10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fccfc009fd0 tx=0x7fccfc00b040 comp rx=0 tx=0).stop 2024-01-15T18:19:46.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.758+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 msgr2=0x7fcd0c15a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:46.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.758+0000 7fcd11800640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 0x7fcd0c15a130 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400c190 tx=0x7fcd0c15ca70 comp rx=0 tx=0).stop 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 shutdown_connections 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcce406c550 0x7fcce406ea10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcd0c14a390 0x7fcd0c15a130 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcd0c071560 0x7fcd0c146c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 --2- 172.21.15.2:0/2412122565 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fccec005180 0x7fcd0c1537b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 >> 172.21.15.2:0/2412122565 conn(0x7fcd0c06da20 msgr2=0x7fcd0c0aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 shutdown_connections 2024-01-15T18:19:46.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:46.759+0000 7fcd11800640 1 -- 172.21.15.2:0/2412122565 wait complete. 2024-01-15T18:19:46.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:14:14.206729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:14:15.347816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:19:46.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:19:47.013 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.010+0000 7fe5d7800640 1 Processor -- start 2024-01-15T18:19:47.014 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.010+0000 7fe5d7800640 1 -- start start 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.012+0000 7fe5d7800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5d01439f0 0x7fe5d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.012+0000 7fe5d7800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d015dd70 con 0x7fe5d006c8f0 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.012+0000 7fe5d7800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5d00b38b0 con 0x7fe5d00b3420 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.012+0000 7fe5d7800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5d00711b0 con 0x7fe5d01439f0 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.012+0000 7fe5d4e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5d01439f0 0x7fe5d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.013+0000 7fe5d4e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fe5d01439f0 0x7fe5d00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41610/0 (socket says 172.21.15.2:41610) 2024-01-15T18:19:47.016 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.013+0000 7fe5d4e00640 1 -- 172.21.15.2:0/2894602564 learned_addr learned my addr 172.21.15.2:0/2894602564 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.013+0000 7fe5d4e00640 1 -- 172.21.15.2:0/2894602564 >> v1:172.21.15.137:6789/0 conn(0x7fe5d00b3420 legacy=0x7fe5d00cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.013+0000 7fe5d4e00640 1 -- 172.21.15.2:0/2894602564 >> v1:172.21.15.2:6789/0 conn(0x7fe5d006c8f0 legacy=0x7fe5d006ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.017 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.013+0000 7fe5d4e00640 1 -- 172.21.15.2:0/2894602564 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5d0143dd0 con 0x7fe5d01439f0 2024-01-15T18:19:47.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.014+0000 7fe5d4e00640 1 --2- 172.21.15.2:0/2894602564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d01439f0 0x7fe5d00702c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009870 tx=0x7fe5c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be34377cb37fd16d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.015+0000 7fe5cf400640 1 -- 172.21.15.2:0/2894602564 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0030860 con 0x7fe5d01439f0 2024-01-15T18:19:47.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.015+0000 7fe5cf400640 1 -- 172.21.15.2:0/2894602564 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c00309c0 con 0x7fe5d01439f0 2024-01-15T18:19:47.018 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.015+0000 7fe5cf400640 1 -- 172.21.15.2:0/2894602564 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c0030d50 con 0x7fe5d01439f0 2024-01-15T18:19:47.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.015+0000 7fe5d7800640 1 -- 172.21.15.2:0/2894602564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d01439f0 msgr2=0x7fe5d00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.015+0000 7fe5d7800640 1 --2- 172.21.15.2:0/2894602564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d01439f0 0x7fe5d00702c0 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0009870 tx=0x7fe5c002b9a0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.016+0000 7fe5d7800640 1 -- 172.21.15.2:0/2894602564 shutdown_connections 2024-01-15T18:19:47.019 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.016+0000 7fe5d7800640 1 --2- 172.21.15.2:0/2894602564 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d01439f0 0x7fe5d00702c0 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.016+0000 7fe5d7800640 1 -- 172.21.15.2:0/2894602564 >> 172.21.15.2:0/2894602564 conn(0x7fe5d00b45f0 msgr2=0x7fe5d00b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:47.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.016+0000 7fe5d7800640 1 -- 172.21.15.2:0/2894602564 shutdown_connections 2024-01-15T18:19:47.020 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.016+0000 7fe5d7800640 1 -- 172.21.15.2:0/2894602564 wait complete. 2024-01-15T18:19:47.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.018+0000 7fe5d7800640 1 Processor -- start 2024-01-15T18:19:47.021 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.018+0000 7fe5d7800640 1 -- start start 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.019+0000 7fe5d7800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d006c8f0 0x7fe5d013b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.019+0000 7fe5d7800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 0x7fe5d00ce2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.019+0000 7fe5d7800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5d01439f0 0x7fe5d00ce800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.019+0000 7fe5d7800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d013b610 con 0x7fe5d00b3420 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.019+0000 7fe5d7800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d0138880 con 0x7fe5d006c8f0 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.019+0000 7fe5d7800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5d013a030 con 0x7fe5d01439f0 2024-01-15T18:19:47.023 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 0x7fe5d00ce2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 0x7fe5d00ce2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41616/0 (socket says 172.21.15.2:41616) 2024-01-15T18:19:47.024 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 -- 172.21.15.2:0/1569753260 learned_addr learned my addr 172.21.15.2:0/1569753260 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:47.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5d4e00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d006c8f0 0x7fe5d013b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5d5800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5d01439f0 0x7fe5d00ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 -- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5d01439f0 msgr2=0x7fe5d00ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5d01439f0 0x7fe5d00ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.025 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 -- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d006c8f0 msgr2=0x7fe5d013b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d006c8f0 0x7fe5d013b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 -- 172.21.15.2:0/1569753260 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c00094d0 con 0x7fe5d00b3420 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5d5800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5d01439f0 0x7fe5d00ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.020+0000 7fe5cfe00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 0x7fe5d00ce2c0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800c430 tx=0x7fe5b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.021+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8011790 con 0x7fe5d00b3420 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.021+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b80118f0 con 0x7fe5d00b3420 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.021+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8011ca0 con 0x7fe5d00b3420 2024-01-15T18:19:47.026 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.021+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5d013a310 con 0x7fe5d00b3420 2024-01-15T18:19:47.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.021+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5d013a8d0 con 0x7fe5d00b3420 2024-01-15T18:19:47.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.022+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5d00afa60 con 0x7fe5d00b3420 2024-01-15T18:19:47.027 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.023+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe5b801c460 con 0x7fe5d00b3420 2024-01-15T18:19:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.025+0000 7fe5ccc00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5a806c550 0x7fe5a806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.025+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fe5b8088b40 con 0x7fe5d00b3420 2024-01-15T18:19:47.028 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.025+0000 7fe5d4e00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5a806c550 0x7fe5a806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.030 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.027+0000 7fe5d4e00640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5a806c550 0x7fe5a806ea10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0036000 tx=0x7fe5c003b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.033 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.030+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe5b8055080 con 0x7fe5d00b3420 2024-01-15T18:19:47.228 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.225+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5d00686f0 con 0x7fe5d00b3420 2024-01-15T18:19:47.229 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.226+0000 7fe5ccc00640 1 -- 172.21.15.2:0/1569753260 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5b80586a0 con 0x7fe5d00b3420 2024-01-15T18:19:47.230 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:19:47.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5a806c550 msgr2=0x7fe5a806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5a806c550 0x7fe5a806ea10 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe5c0036000 tx=0x7fe5c003b040 comp rx=0 tx=0).stop 2024-01-15T18:19:47.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 msgr2=0x7fe5d00ce2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.232 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 0x7fe5d00ce2c0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fe5b800c430 tx=0x7fe5b800c8f0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 shutdown_connections 2024-01-15T18:19:47.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5a806c550 0x7fe5a806ea10 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe5d01439f0 0x7fe5d00ce800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.233 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe5d00b3420 0x7fe5d00ce2c0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 --2- 172.21.15.2:0/1569753260 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe5d006c8f0 0x7fe5d013b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 >> 172.21.15.2:0/1569753260 conn(0x7fe5d00b45f0 msgr2=0x7fe5d00af250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:47.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 shutdown_connections 2024-01-15T18:19:47.234 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.229+0000 7fe5d7800640 1 -- 172.21.15.2:0/1569753260 wait complete. 2024-01-15T18:19:47.250 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:19:47.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.470+0000 7fb522400640 1 Processor -- start 2024-01-15T18:19:47.473 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.470+0000 7fb522400640 1 -- start start 2024-01-15T18:19:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb522400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fb51c15c450 0x7fb51c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb522400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb51c06ce30 0x7fb51c070b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.475 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb522400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb51c06d290 con 0x7fb51c144c00 2024-01-15T18:19:47.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb522400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c0ce230 con 0x7fb51c06ce30 2024-01-15T18:19:47.476 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb522400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb51c0ce510 con 0x7fb51c15c450 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb51b400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fb51c144c00 0x7fb51c0cd830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46480/0 (socket says 172.21.15.2:46480) 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.472+0000 7fb51b400640 1 -- 172.21.15.2:0/213642078 learned_addr learned my addr 172.21.15.2:0/213642078 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb51be00640 1 --2- 172.21.15.2:0/213642078 >> v2:172.21.15.2:3300/0 conn(0x7fb51c06ce30 0x7fb51c070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb513e00640 1 --2- 172.21.15.2:0/213642078 >> v2:172.21.15.137:3300/0 conn(0x7fb51c15c450 0x7fb51c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb51be00640 1 -- 172.21.15.2:0/213642078 >> v2:172.21.15.137:3300/0 conn(0x7fb51c15c450 msgr2=0x7fb51c06c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb51be00640 1 --2- 172.21.15.2:0/213642078 >> v2:172.21.15.137:3300/0 conn(0x7fb51c15c450 0x7fb51c06c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb51be00640 1 -- 172.21.15.2:0/213642078 >> v1:172.21.15.2:6789/0 conn(0x7fb51c144c00 legacy=0x7fb51c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.477 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb51aa00640 1 -- 172.21.15.2:0/213642078 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665041941 0 0) 0x7fb51c06d290 con 0x7fb51c144c00 2024-01-15T18:19:47.478 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.473+0000 7fb51be00640 1 -- 172.21.15.2:0/213642078 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c0ce7f0 con 0x7fb51c06ce30 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.474+0000 7fb51be00640 1 --2- 172.21.15.2:0/213642078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c070b10 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fb51c06f9f0 tx=0x7fb51402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58393c724a7b71b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.474+0000 7fb51aa00640 1 -- 172.21.15.2:0/213642078 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb51402cc70 con 0x7fb51c06ce30 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb51aa00640 1 -- 172.21.15.2:0/213642078 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51402cdd0 con 0x7fb51c06ce30 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb51aa00640 1 -- 172.21.15.2:0/213642078 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb514034660 con 0x7fb51c06ce30 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 -- 172.21.15.2:0/213642078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 msgr2=0x7fb51c070b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 --2- 172.21.15.2:0/213642078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c070b10 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7fb51c06f9f0 tx=0x7fb51402bec0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.479 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 -- 172.21.15.2:0/213642078 shutdown_connections 2024-01-15T18:19:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 --2- 172.21.15.2:0/213642078 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c070b10 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 --2- 172.21.15.2:0/213642078 >> v2:172.21.15.137:3300/0 conn(0x7fb51c15c450 0x7fb51c06c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 -- 172.21.15.2:0/213642078 >> 172.21.15.2:0/213642078 conn(0x7fb51c0b45f0 msgr2=0x7fb51c0b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 -- 172.21.15.2:0/213642078 shutdown_connections 2024-01-15T18:19:47.480 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.475+0000 7fb522400640 1 -- 172.21.15.2:0/213642078 wait complete. 2024-01-15T18:19:47.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.477+0000 7fb522400640 1 Processor -- start 2024-01-15T18:19:47.481 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.477+0000 7fb522400640 1 -- start start 2024-01-15T18:19:47.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.478+0000 7fb522400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c1efee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.478+0000 7fb522400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb51c144c00 0x7fb51c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.478+0000 7fb522400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb51c15c450 0x7fb51c1f0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.478+0000 7fb522400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c1f26a0 con 0x7fb51c06ce30 2024-01-15T18:19:47.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.478+0000 7fb522400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c1f1440 con 0x7fb51c144c00 2024-01-15T18:19:47.482 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.478+0000 7fb522400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51c1f1a90 con 0x7fb51c15c450 2024-01-15T18:19:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c1efee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c1efee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41626/0 (socket says 172.21.15.2:41626) 2024-01-15T18:19:47.483 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 -- 172.21.15.2:0/2917866001 learned_addr learned my addr 172.21.15.2:0/2917866001 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51be00640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb51c15c450 0x7fb51c1f0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 -- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb51c15c450 msgr2=0x7fb51c1f0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb51c15c450 0x7fb51c1f0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb513e00640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb51c144c00 0x7fb51c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 -- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb51c144c00 msgr2=0x7fb51c14f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb51c144c00 0x7fb51c14f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.484 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 -- 172.21.15.2:0/2917866001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb514009a70 con 0x7fb51c06ce30 2024-01-15T18:19:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.479+0000 7fb51b400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c1efee0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c4d0 tx=0x7fb50c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.480+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c0074e0 con 0x7fb51c06ce30 2024-01-15T18:19:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.480+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50c007660 con 0x7fb51c06ce30 2024-01-15T18:19:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.480+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50c004a20 con 0x7fb51c06ce30 2024-01-15T18:19:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.480+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb51c1f1d70 con 0x7fb51c06ce30 2024-01-15T18:19:47.485 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.480+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb51c1f2330 con 0x7fb51c06ce30 2024-01-15T18:19:47.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.481+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb51c0686f0 con 0x7fb51c06ce30 2024-01-15T18:19:47.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.482+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb50c004b80 con 0x7fb51c06ce30 2024-01-15T18:19:47.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.483+0000 7fb513400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb4f406c550 0x7fb4f406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.486 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.483+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fb50c087d80 con 0x7fb51c06ce30 2024-01-15T18:19:47.487 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.484+0000 7fb513e00640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb4f406c550 0x7fb4f406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.488 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.485+0000 7fb513e00640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb4f406c550 0x7fb4f406ea10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb504006fd0 tx=0x7fb504008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.491 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.488+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb50c0542c0 con 0x7fb51c06ce30 2024-01-15T18:19:47.682 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.679+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb51c140390 con 0x7fb51c06ce30 2024-01-15T18:19:47.683 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.680+0000 7fb513400640 1 -- 172.21.15.2:0/2917866001 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb50c0180a0 con 0x7fb51c06ce30 2024-01-15T18:19:47.683 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T18:19:47.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb4f406c550 msgr2=0x7fb4f406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb4f406c550 0x7fb4f406ea10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fb504006fd0 tx=0x7fb504008040 comp rx=0 tx=0).stop 2024-01-15T18:19:47.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 msgr2=0x7fb51c1efee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c1efee0 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fb50c00c4d0 tx=0x7fb50c00c990 comp rx=0 tx=0).stop 2024-01-15T18:19:47.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 shutdown_connections 2024-01-15T18:19:47.686 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb4f406c550 0x7fb4f406ea10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb51c15c450 0x7fb51c1f0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb51c144c00 0x7fb51c14f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 --2- 172.21.15.2:0/2917866001 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb51c06ce30 0x7fb51c1efee0 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.682+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 >> 172.21.15.2:0/2917866001 conn(0x7fb51c0b45f0 msgr2=0x7fb51c071420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:47.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.683+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 shutdown_connections 2024-01-15T18:19:47.687 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.683+0000 7fb522400640 1 -- 172.21.15.2:0/2917866001 wait complete. 2024-01-15T18:19:47.706 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:19:47.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.929+0000 7f7ee0c00640 1 Processor -- start 2024-01-15T18:19:47.933 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.929+0000 7f7ee0c00640 1 -- start start 2024-01-15T18:19:47.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7ee0c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7edc159f50 0x7f7edc0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7ee0c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc15a330 con 0x7f7edc0b32a0 2024-01-15T18:19:47.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7ee0c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7edc15dad0 con 0x7f7edc0b41b0 2024-01-15T18:19:47.934 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7ee0c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7edc15ddb0 con 0x7f7edc159f50 2024-01-15T18:19:47.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7edb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7edc159f50 0x7f7edc0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7ed3e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f7edc0b32a0 0x7f7edc0b3aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46494/0 (socket says 172.21.15.2:46494) 2024-01-15T18:19:47.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7ed3e00640 1 -- 172.21.15.2:0/4090127470 learned_addr learned my addr 172.21.15.2:0/4090127470 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:47.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.931+0000 7f7edb400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f7edc159f50 0x7f7edc0b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41642/0 (socket says 172.21.15.2:41642) 2024-01-15T18:19:47.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.932+0000 7f7edb400640 1 -- 172.21.15.2:0/4090127470 >> v1:172.21.15.137:6789/0 conn(0x7f7edc0b41b0 legacy=0x7f7edc06daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:47.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.932+0000 7f7edb400640 1 -- 172.21.15.2:0/4090127470 >> v1:172.21.15.2:6789/0 conn(0x7f7edc0b32a0 legacy=0x7f7edc0b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.932+0000 7f7edb400640 1 -- 172.21.15.2:0/4090127470 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc15e090 con 0x7f7edc159f50 2024-01-15T18:19:47.936 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.932+0000 7f7edaa00640 1 -- 172.21.15.2:0/4090127470 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398806617 0 0) 0x7f7edc15a330 con 0x7f7edc0b32a0 2024-01-15T18:19:47.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.933+0000 7f7edb400640 1 --2- 172.21.15.2:0/4090127470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc159f50 0x7f7edc0b2cf0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0097d0 tx=0x7f7ecc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472f964a6daae0a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.937 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.933+0000 7f7edaa00640 1 -- 172.21.15.2:0/4090127470 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc030610 con 0x7f7edc159f50 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.933+0000 7f7edaa00640 1 -- 172.21.15.2:0/4090127470 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc030770 con 0x7f7edc159f50 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.933+0000 7f7edaa00640 1 -- 172.21.15.2:0/4090127470 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc030b00 con 0x7f7edc159f50 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 -- 172.21.15.2:0/4090127470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc159f50 msgr2=0x7f7edc0b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/4090127470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc159f50 0x7f7edc0b2cf0 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc0097d0 tx=0x7f7ecc02b960 comp rx=0 tx=0).stop 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 -- 172.21.15.2:0/4090127470 shutdown_connections 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/4090127470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc159f50 0x7f7edc0b2cf0 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 -- 172.21.15.2:0/4090127470 >> 172.21.15.2:0/4090127470 conn(0x7f7edc06b5c0 msgr2=0x7f7edc06b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 -- 172.21.15.2:0/4090127470 shutdown_connections 2024-01-15T18:19:47.938 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.934+0000 7f7ee0c00640 1 -- 172.21.15.2:0/4090127470 wait complete. 2024-01-15T18:19:47.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.936+0000 7f7ee0c00640 1 Processor -- start 2024-01-15T18:19:47.939 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.936+0000 7f7ee0c00640 1 -- start start 2024-01-15T18:19:47.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.937+0000 7f7ee0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 0x7f7edc158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.937+0000 7f7ee0c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7edc0b41b0 0x7f7edc1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.937+0000 7f7ee0c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7edc159f50 0x7f7edc1f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.937+0000 7f7ee0c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc099a00 con 0x7f7edc0b32a0 2024-01-15T18:19:47.941 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.937+0000 7f7ee0c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc09bcb0 con 0x7f7edc159f50 2024-01-15T18:19:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.937+0000 7f7ee0c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7edc156f00 con 0x7f7edc0b41b0 2024-01-15T18:19:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 0x7f7edc158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 0x7f7edc158dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41650/0 (socket says 172.21.15.2:41650) 2024-01-15T18:19:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 -- 172.21.15.2:0/3743629919 learned_addr learned my addr 172.21.15.2:0/3743629919 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:47.942 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7ed3e00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7edc0b41b0 0x7f7edc1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 -- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7edc0b41b0 msgr2=0x7f7edc1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7edc0b41b0 0x7f7edc1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 -- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7edc159f50 msgr2=0x7f7edc1f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:47.943 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7edc159f50 0x7f7edc1f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.938+0000 7f7edb400640 1 -- 172.21.15.2:0/3743629919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc009400 con 0x7f7edc0b32a0 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.939+0000 7f7edb400640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 0x7f7edc158dc0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc02bf70 tx=0x7f7ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.939+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc03e070 con 0x7f7edc0b32a0 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.939+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ecc039a10 con 0x7f7edc0b32a0 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.939+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7edc099c50 con 0x7f7edc0b32a0 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.939+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ecc039da0 con 0x7f7edc0b32a0 2024-01-15T18:19:47.944 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.939+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7edc09b940 con 0x7f7edc0b32a0 2024-01-15T18:19:47.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.940+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7edc067970 con 0x7f7edc0b32a0 2024-01-15T18:19:47.945 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.941+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f7ecc04b400 con 0x7f7edc0b32a0 2024-01-15T18:19:47.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.942+0000 7f7ed3400640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7eb406c580 0x7f7eb406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:47.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.942+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f7ecc0ab070 con 0x7f7edc0b32a0 2024-01-15T18:19:47.946 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.943+0000 7f7ed3e00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7eb406c580 0x7f7eb406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:47.947 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.944+0000 7f7ed3e00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7eb406c580 0x7f7eb406ea40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7edc09d810 tx=0x7f7ec4007b90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:47.950 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:47.947+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ecc0775b0 con 0x7f7edc0b32a0 2024-01-15T18:19:48.144 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.141+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7edc074770 con 0x7f7edc0b32a0 2024-01-15T18:19:48.145 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.142+0000 7f7ed3400640 1 -- 172.21.15.2:0/3743629919 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7ecc034020 con 0x7f7edc0b32a0 2024-01-15T18:19:48.146 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T18:19:48.148 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7eb406c580 msgr2=0x7f7eb406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7eb406c580 0x7f7eb406ea40 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f7edc09d810 tx=0x7f7ec4007b90 comp rx=0 tx=0).stop 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 msgr2=0x7f7edc158dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 0x7f7edc158dc0 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc02bf70 tx=0x7f7ecc0046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 shutdown_connections 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f7eb406c580 0x7f7eb406ea40 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f7edc159f50 0x7f7edc1f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f7edc0b41b0 0x7f7edc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.149 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 --2- 172.21.15.2:0/3743629919 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f7edc0b32a0 0x7f7edc158dc0 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 >> 172.21.15.2:0/3743629919 conn(0x7f7edc06b5c0 msgr2=0x7f7edc0ae480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 shutdown_connections 2024-01-15T18:19:48.150 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.145+0000 7f7ee0c00640 1 -- 172.21.15.2:0/3743629919 wait complete. 2024-01-15T18:19:48.168 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:19:48.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.378+0000 7f2382800640 1 Processor -- start 2024-01-15T18:19:48.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.378+0000 7f2382800640 1 -- start start 2024-01-15T18:19:48.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f2382800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f237c0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f2382800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f237c06cfc0 0x7f237c15b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f2382800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237c0b35a0 con 0x7f237c0b30a0 2024-01-15T18:19:48.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f2382800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f237c0b4220 con 0x7f237c06cfc0 2024-01-15T18:19:48.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f2382800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c144f60 con 0x7f237c144b80 2024-01-15T18:19:48.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f237aa00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f237c0b30a0 0x7f237c06c8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46498/0 (socket says 172.21.15.2:46498) 2024-01-15T18:19:48.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.379+0000 7f237aa00640 1 -- 172.21.15.2:0/2253127589 learned_addr learned my addr 172.21.15.2:0/2253127589 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:48.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237b400640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f237c0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237be00640 1 --2- 172.21.15.2:0/2253127589 >> v2:172.21.15.137:3300/0 conn(0x7f237c06cfc0 0x7f237c15b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1463110682 0 0) 0x7f237c0b35a0 con 0x7f237c0b30a0 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f235c003520 con 0x7f237c0b30a0 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2188735802 0 0) 0x7f235c003520 con 0x7f237c0b30a0 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 msgr2=0x7f237c0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f237c0cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 >> v2:172.21.15.137:3300/0 conn(0x7f237c06cfc0 msgr2=0x7f237c15b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 --2- 172.21.15.2:0/2253127589 >> v2:172.21.15.137:3300/0 conn(0x7f237c06cfc0 0x7f237c15b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c0b3880 con 0x7f237c0b30a0 2024-01-15T18:19:48.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f2364001670 con 0x7f237c0b30a0 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 >> v1:172.21.15.2:6789/0 conn(0x7f237c0b30a0 legacy=0x7f237c06c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.380+0000 7f237a000640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f235c003970 0x7f235c003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237a000640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f235c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237a000640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c004d00 0x7f235c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c0b4220 con 0x7f235c003970 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c144f60 con 0x7f235c004d00 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f235c00b980 con 0x7f237c144b80 2024-01-15T18:19:48.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237b400640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f235c003970 0x7f235c003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237aa00640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f235c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237be00640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c004d00 0x7f235c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237b400640 1 -- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 msgr2=0x7f235c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237b400640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f235c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237b400640 1 -- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c004d00 msgr2=0x7f235c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237b400640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c004d00 0x7f235c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237b400640 1 -- 172.21.15.2:0/2253127589 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c003520 con 0x7f235c003970 2024-01-15T18:19:48.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237be00640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c004d00 0x7f235c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:48.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.381+0000 7f237aa00640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c144b80 0x7f235c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:48.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.383+0000 7f237b400640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f235c003970 0x7f235c003db0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f236c009ac0 tx=0x7f235c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4608c05f38f986e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:48.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.383+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c02c6b0 con 0x7f235c003970 2024-01-15T18:19:48.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.383+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236c02c810 con 0x7f235c003970 2024-01-15T18:19:48.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.383+0000 7f237a000640 1 -- 172.21.15.2:0/2253127589 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236c02cba0 con 0x7f235c003970 2024-01-15T18:19:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.383+0000 7f2382800640 1 -- 172.21.15.2:0/2253127589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f235c003970 msgr2=0x7f235c003db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.383+0000 7f2382800640 1 --2- 172.21.15.2:0/2253127589 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f235c003970 0x7f235c003db0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f236c009ac0 tx=0x7f235c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.384+0000 7f237be00640 1 -- 172.21.15.2:0/2253127589 reap_dead start 2024-01-15T18:19:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.384+0000 7f2382800640 1 -- 172.21.15.2:0/2253127589 shutdown_connections 2024-01-15T18:19:48.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.384+0000 7f2382800640 1 -- 172.21.15.2:0/2253127589 >> 172.21.15.2:0/2253127589 conn(0x7f237c0b46d0 msgr2=0x7f237c0b4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.384+0000 7f2382800640 1 -- 172.21.15.2:0/2253127589 shutdown_connections 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.384+0000 7f2382800640 1 -- 172.21.15.2:0/2253127589 wait complete. 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.386+0000 7f2382800640 1 Processor -- start 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.386+0000 7f2382800640 1 -- start start 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f2382800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c003970 0x7f237c139bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f2382800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 0x7f237c13c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f2382800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c13c5e0 0x7f237c15dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f2382800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c13d710 con 0x7f237c1f3a70 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f2382800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237c13d430 con 0x7f235c003970 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f2382800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237c13b5c0 con 0x7f237c13c5e0 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 0x7f237c13c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 0x7f237c13c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41672/0 (socket says 172.21.15.2:41672) 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 -- 172.21.15.2:0/546295988 learned_addr learned my addr 172.21.15.2:0/546295988 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:48.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237b400640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c003970 0x7f237c139bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237be00640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c13c5e0 0x7f237c15dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 -- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c13c5e0 msgr2=0x7f237c15dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c13c5e0 0x7f237c15dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 -- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c003970 msgr2=0x7f237c139bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c003970 0x7f237c139bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.387+0000 7f237aa00640 1 -- 172.21.15.2:0/546295988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c13b8a0 con 0x7f237c1f3a70 2024-01-15T18:19:48.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.388+0000 7f237aa00640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 0x7f237c13c0a0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f2364000f60 tx=0x7f236400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.388+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640074e0 con 0x7f237c1f3a70 2024-01-15T18:19:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.388+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364007640 con 0x7f237c1f3a70 2024-01-15T18:19:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.388+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237c13bb80 con 0x7f237c1f3a70 2024-01-15T18:19:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.388+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23640049e0 con 0x7f237c1f3a70 2024-01-15T18:19:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.388+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237c1f3ef0 con 0x7f237c1f3a70 2024-01-15T18:19:48.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.390+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f237c1f4170 con 0x7f237c1f3a70 2024-01-15T18:19:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.391+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2364004b40 con 0x7f237c1f3a70 2024-01-15T18:19:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.392+0000 7f2373400640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f235406c550 0x7f235406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.392+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f23640888a0 con 0x7f237c1f3a70 2024-01-15T18:19:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.392+0000 7f237b400640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f235406c550 0x7f235406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.393+0000 7f237b400640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f235406c550 0x7f235406ea10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f236c036000 tx=0x7f236c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:48.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.396+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f237c1f4170 con 0x7f237c1f3a70 2024-01-15T18:19:48.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.605+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f237c13b420 con 0x7f237c1f3a70 2024-01-15T18:19:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.606+0000 7f2373400640 1 -- 172.21.15.2:0/546295988 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v116) v1 ==== 80+0+0 (secure 0 0 0) 0x7f237c13b420 con 0x7f237c1f3a70 2024-01-15T18:19:48.610 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T18:19:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f235406c550 msgr2=0x7f235406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f235406c550 0x7f235406ea10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f236c036000 tx=0x7f236c030000 comp rx=0 tx=0).stop 2024-01-15T18:19:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 msgr2=0x7f237c13c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.613 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 0x7f237c13c0a0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f2364000f60 tx=0x7f236400dd40 comp rx=0 tx=0).stop 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 shutdown_connections 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f235406c550 0x7f235406ea10 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f237c13c5e0 0x7f237c15dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f237c1f3a70 0x7f237c13c0a0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 --2- 172.21.15.2:0/546295988 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f235c003970 0x7f237c139bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.609+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 >> 172.21.15.2:0/546295988 conn(0x7f237c0b46d0 msgr2=0x7f237c0ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.610+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 shutdown_connections 2024-01-15T18:19:48.614 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.610+0000 7f2382800640 1 -- 172.21.15.2:0/546295988 wait complete. 2024-01-15T18:19:48.632 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:19:48.632 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:48.689 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:19:48.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.914+0000 7f114a400640 1 Processor -- start 2024-01-15T18:19:48.918 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.914+0000 7f114a400640 1 -- start start 2024-01-15T18:19:48.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.915+0000 7f114a400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f1144159690 0x7f114415d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.915+0000 7f114a400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f11440ce5c0 0x7f114415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f114a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f11440b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.919 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f114a400640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114406cd30 con 0x7f11440ce5c0 2024-01-15T18:19:48.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f114a400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11440b3e20 con 0x7f1144159690 2024-01-15T18:19:48.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f114a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114406e2d0 con 0x7f114415dd00 2024-01-15T18:19:48.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f11440ce5c0 0x7f114415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f11440ce5c0 0x7f114415d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41682/0 (socket says 172.21.15.2:41682) 2024-01-15T18:19:48.920 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 -- 172.21.15.2:0/1357776352 learned_addr learned my addr 172.21.15.2:0/1357776352 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:48.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f1149a00640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f11440b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 -- 172.21.15.2:0/1357776352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 msgr2=0x7f11440b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:48.921 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f1149000640 1 --2- 172.21.15.2:0/1357776352 >> v2:172.21.15.137:3300/0 conn(0x7f1144159690 0x7f114415d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f11440b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 -- 172.21.15.2:0/1357776352 >> v2:172.21.15.137:3300/0 conn(0x7f1144159690 msgr2=0x7f114415d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 --2- 172.21.15.2:0/1357776352 >> v2:172.21.15.137:3300/0 conn(0x7f1144159690 0x7f114415d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f113bc00640 1 -- 172.21.15.2:0/1357776352 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1144070d60 con 0x7f11440ce5c0 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.916+0000 7f1149a00640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f11440b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.918+0000 7f113bc00640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f114415d7c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f1144067e00 tx=0x7f112c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5aa2d1a785ee16 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.918+0000 7f1143e00640 1 -- 172.21.15.2:0/1357776352 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c02ca70 con 0x7f11440ce5c0 2024-01-15T18:19:48.922 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.918+0000 7f1143e00640 1 -- 172.21.15.2:0/1357776352 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c02cbd0 con 0x7f11440ce5c0 2024-01-15T18:19:48.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.918+0000 7f1143e00640 1 -- 172.21.15.2:0/1357776352 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0343f0 con 0x7f11440ce5c0 2024-01-15T18:19:48.923 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 -- 172.21.15.2:0/1357776352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 msgr2=0x7f114415d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f114415d7c0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f1144067e00 tx=0x7f112c02bed0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 -- 172.21.15.2:0/1357776352 shutdown_connections 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f11440b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 --2- 172.21.15.2:0/1357776352 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f114415d7c0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 --2- 172.21.15.2:0/1357776352 >> v2:172.21.15.137:3300/0 conn(0x7f1144159690 0x7f114415d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 -- 172.21.15.2:0/1357776352 >> 172.21.15.2:0/1357776352 conn(0x7f11440b0840 msgr2=0x7f11440b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 -- 172.21.15.2:0/1357776352 shutdown_connections 2024-01-15T18:19:48.924 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.919+0000 7f114a400640 1 -- 172.21.15.2:0/1357776352 wait complete. 2024-01-15T18:19:48.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.921+0000 7f114a400640 1 Processor -- start 2024-01-15T18:19:48.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.921+0000 7f114a400640 1 -- start start 2024-01-15T18:19:48.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f114a400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f1144143f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f114a400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1144159690 0x7f11440b0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.927 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f114a400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f114414c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f114a400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1144148a80 con 0x7f11440ce5c0 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f114a400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1144147b90 con 0x7f1144159690 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f114a400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1144147e70 con 0x7f114415dd00 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f1149000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f1144143f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f1149000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f1144143f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41690/0 (socket says 172.21.15.2:41690) 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.922+0000 7f1149000640 1 -- 172.21.15.2:0/308658335 learned_addr learned my addr 172.21.15.2:0/308658335 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f113bc00640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1144159690 0x7f11440b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149a00640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f114414c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.928 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149000640 1 -- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 msgr2=0x7f114414c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149000640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f114414c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149000640 1 -- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1144159690 msgr2=0x7f11440b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149000640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1144159690 0x7f11440b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149000640 1 -- 172.21.15.2:0/308658335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c009b30 con 0x7f11440ce5c0 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f113bc00640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1144159690 0x7f11440b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.923+0000 7f1149000640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f1144143f80 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f113400c7f0 tx=0x7f113400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.924+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134004220 con 0x7f11440ce5c0 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.924+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1134004380 con 0x7f11440ce5c0 2024-01-15T18:19:48.929 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.924+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1134007860 con 0x7f11440ce5c0 2024-01-15T18:19:48.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.924+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1144148150 con 0x7f11440ce5c0 2024-01-15T18:19:48.930 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.924+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1144148710 con 0x7f11440ce5c0 2024-01-15T18:19:48.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.925+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1144074770 con 0x7f11440ce5c0 2024-01-15T18:19:48.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.926+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1134007bd0 con 0x7f11440ce5c0 2024-01-15T18:19:48.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.927+0000 7f1141600640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f111c06c550 0x7f111c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:48.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.928+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f1134088890 con 0x7f11440ce5c0 2024-01-15T18:19:48.931 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.928+0000 7f113bc00640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f111c06c550 0x7f111c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:48.932 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.929+0000 7f113bc00640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f111c06c550 0x7f111c06ea10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f112c035000 tx=0x7f112c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:48.935 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:48.932+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1134055030 con 0x7f11440ce5c0 2024-01-15T18:19:49.130 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.127+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11440b2220 con 0x7f11440ce5c0 2024-01-15T18:19:49.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.129+0000 7f1141600640 1 -- 172.21.15.2:0/308658335 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16780 (secure 0 0 0) 0x7f113401e030 con 0x7f11440ce5c0 2024-01-15T18:19:49.132 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:49.133 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":116,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:15:19.802397+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:19:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.131+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f111c06c550 msgr2=0x7f111c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.131+0000 7f114a400640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f111c06c550 0x7f111c06ea10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f112c035000 tx=0x7f112c030000 comp rx=0 tx=0).stop 2024-01-15T18:19:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.131+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 msgr2=0x7f1144143f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.131+0000 7f114a400640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f1144143f80 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f113400c7f0 tx=0x7f113400ccb0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.131+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 shutdown_connections 2024-01-15T18:19:49.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.131+0000 7f114a400640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f111c06c550 0x7f111c06ea10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.132+0000 7f114a400640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f114415dd00 0x7f114414c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.132+0000 7f114a400640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f1144159690 0x7f11440b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.132+0000 7f114a400640 1 --2- 172.21.15.2:0/308658335 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f11440ce5c0 0x7f1144143f80 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.132+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 >> 172.21.15.2:0/308658335 conn(0x7f11440b0840 msgr2=0x7f11440ac9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:49.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.132+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 shutdown_connections 2024-01-15T18:19:49.136 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.132+0000 7f114a400640 1 -- 172.21.15.2:0/308658335 wait complete. 2024-01-15T18:19:49.156 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-15T17:57:25.437842+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-15T18:14:14.206729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '113', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-15T18:14:15.347816+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '114', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-15T18:19:49.157 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-15T18:19:49.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.379+0000 7f9b08c00640 1 Processor -- start 2024-01-15T18:19:49.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.380+0000 7f9b08c00640 1 -- start start 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.381+0000 7f9b08c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 0x7f9b0409a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.381+0000 7f9b08c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9b040cc110 0x7f9b040cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.381+0000 7f9b08c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b0406b5c0 con 0x7f9b0415d530 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.381+0000 7f9b08c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b0406ece0 con 0x7f9b040cc110 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.381+0000 7f9b08c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040cbc80 con 0x7f9b040d19a0 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.382+0000 7f9b03400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9b0415d530 0x7f9b04099a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46508/0 (socket says 172.21.15.2:46508) 2024-01-15T18:19:49.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.382+0000 7f9b03400640 1 -- 172.21.15.2:0/3767133812 learned_addr learned my addr 172.21.15.2:0/3767133812 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:49.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.382+0000 7f9b03e00640 1 --2- 172.21.15.2:0/3767133812 >> v2:172.21.15.137:3300/0 conn(0x7f9b040cc110 0x7f9b040cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.382+0000 7f9b02a00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 0x7f9b0409a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.382+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2549501041 0 0) 0x7f9b0406b5c0 con 0x7f9b0415d530 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.382+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ae4003520 con 0x7f9b0415d530 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1615499958 0 0) 0x7f9ae4003520 con 0x7f9b0415d530 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 msgr2=0x7f9b0409a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 0x7f9b0409a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 >> v2:172.21.15.137:3300/0 conn(0x7f9b040cc110 msgr2=0x7f9b040cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 --2- 172.21.15.2:0/3767133812 >> v2:172.21.15.137:3300/0 conn(0x7f9b040cc110 0x7f9b040cc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b0406ea40 con 0x7f9b0415d530 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9af40019f0 con 0x7f9b0415d530 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 >> v1:172.21.15.2:6789/0 conn(0x7f9b0415d530 legacy=0x7f9b04099a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 0x7f9ae4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae4003ef0 0x7f9ae40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9ae4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b0406ece0 con 0x7f9ae4004d00 2024-01-15T18:19:49.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040cbc80 con 0x7f9ae4003ef0 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae400b980 con 0x7f9b040d19a0 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b03e00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9ae4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.383+0000 7f9b03400640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 0x7f9ae4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.384+0000 7f9b02a00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae4003ef0 0x7f9ae40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.384+0000 7f9b03e00640 1 -- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 msgr2=0x7f9ae4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.384+0000 7f9b03e00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b040d19a0 0x7f9ae4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.384+0000 7f9b03e00640 1 -- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae4003ef0 msgr2=0x7f9ae40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:49.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.384+0000 7f9b03e00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ae4003ef0 0x7f9ae40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.384+0000 7f9b03e00640 1 -- 172.21.15.2:0/3767133812 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae4003520 con 0x7f9ae4004d00 2024-01-15T18:19:49.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.385+0000 7f9b03e00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9ae4005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009ac0 tx=0x7f9ae40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec69f761903c2b52 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.385+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc02c6d0 con 0x7f9ae4004d00 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.385+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9afc02c830 con 0x7f9ae4004d00 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.385+0000 7f9b02000640 1 -- 172.21.15.2:0/3767133812 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9afc02cbc0 con 0x7f9ae4004d00 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b08c00640 1 -- 172.21.15.2:0/3767133812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 msgr2=0x7f9ae4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b08c00640 1 --2- 172.21.15.2:0/3767133812 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9ae4005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f9afc009ac0 tx=0x7f9ae40057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b03e00640 1 -- 172.21.15.2:0/3767133812 reap_dead start 2024-01-15T18:19:49.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b08c00640 1 -- 172.21.15.2:0/3767133812 shutdown_connections 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b08c00640 1 -- 172.21.15.2:0/3767133812 >> 172.21.15.2:0/3767133812 conn(0x7f9b040d0e40 msgr2=0x7f9b040d1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b08c00640 1 -- 172.21.15.2:0/3767133812 shutdown_connections 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.386+0000 7f9b08c00640 1 -- 172.21.15.2:0/3767133812 wait complete. 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.388+0000 7f9b08c00640 1 Processor -- start 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.388+0000 7f9b08c00640 1 -- start start 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b08c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9b040afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b08c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b041f2200 0x7f9b040c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b08c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b040ca110 0x7f9b040ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b08c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040b0160 con 0x7f9ae4004d00 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b08c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b04149d40 con 0x7f9b040ca110 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b08c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b040caad0 con 0x7f9b041f2200 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9b040afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9b040afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41704/0 (socket says 172.21.15.2:41704) 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 -- 172.21.15.2:0/687334145 learned_addr learned my addr 172.21.15.2:0/687334145 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b02a00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b041f2200 0x7f9b040c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 -- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b041f2200 msgr2=0x7f9b040c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b041f2200 0x7f9b040c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 -- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b040ca110 msgr2=0x7f9b040ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b040ca110 0x7f9b040ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.389+0000 7f9b03400640 1 -- 172.21.15.2:0/687334145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b040b0380 con 0x7f9ae4004d00 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.390+0000 7f9b03400640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9b040afc20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9af4002230 tx=0x7f9af400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.390+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af400ee90 con 0x7f9ae4004d00 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.390+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9af4013420 con 0x7f9ae4004d00 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.390+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af40137f0 con 0x7f9ae4004d00 2024-01-15T18:19:49.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.390+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b040c7fb0 con 0x7f9ae4004d00 2024-01-15T18:19:49.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.390+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b040c8570 con 0x7f9ae4004d00 2024-01-15T18:19:49.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.391+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b04074770 con 0x7f9ae4004d00 2024-01-15T18:19:49.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.392+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9af4013950 con 0x7f9ae4004d00 2024-01-15T18:19:49.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.393+0000 7f9afb400640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9adc06c550 0x7f9adc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.393+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f9af4089430 con 0x7f9ae4004d00 2024-01-15T18:19:49.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.394+0000 7f9b02a00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9adc06c550 0x7f9adc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.395+0000 7f9b02a00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9adc06c550 0x7f9adc06ea10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9b041f5de0 tx=0x7f9aec008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:49.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.398+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9af4055970 con 0x7f9ae4004d00 2024-01-15T18:19:49.592 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.589+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9b040c8730 con 0x7f9ae4004d00 2024-01-15T18:19:49.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.590+0000 7f9afb400640 1 -- 172.21.15.2:0/687334145 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9af4058f90 con 0x7f9ae4004d00 2024-01-15T18:19:49.594 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-01-15T18:19:49.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9adc06c550 msgr2=0x7f9adc06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9adc06c550 0x7f9adc06ea10 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9b041f5de0 tx=0x7f9aec008040 comp rx=0 tx=0).stop 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 msgr2=0x7f9b040afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9b040afc20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f9af4002230 tx=0x7f9af400e530 comp rx=0 tx=0).stop 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 shutdown_connections 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9adc06c550 0x7f9adc06ea10 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9b040ca110 0x7f9b040ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9b041f2200 0x7f9b040c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 --2- 172.21.15.2:0/687334145 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ae4004d00 0x7f9b040afc20 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 >> 172.21.15.2:0/687334145 conn(0x7f9b040d0e40 msgr2=0x7f9b040cd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:49.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 shutdown_connections 2024-01-15T18:19:49.598 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.593+0000 7f9b08c00640 1 -- 172.21.15.2:0/687334145 wait complete. 2024-01-15T18:19:49.615 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-15T18:19:49.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.835+0000 7fd3aec00640 1 Processor -- start 2024-01-15T18:19:49.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.836+0000 7fd3aec00640 1 -- start start 2024-01-15T18:19:49.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.837+0000 7fd3aec00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fd3a80702c0 0x7fd3a80b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.837+0000 7fd3aec00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd3a806c8f0 0x7fd3a806ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.837+0000 7fd3aec00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a815dd70 con 0x7fd3a81439f0 2024-01-15T18:19:49.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.837+0000 7fd3aec00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a806d230 con 0x7fd3a806c8f0 2024-01-15T18:19:49.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.837+0000 7fd3aec00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80711b0 con 0x7fd3a80702c0 2024-01-15T18:19:49.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.837+0000 7fd3acc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd3a806c8f0 0x7fd3a806ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd3acc00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fd3a806c8f0 0x7fd3a806ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41706/0 (socket says 172.21.15.2:41706) 2024-01-15T18:19:49.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd3acc00640 1 -- 172.21.15.2:0/4124105201 learned_addr learned my addr 172.21.15.2:0/4124105201 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:49.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd39f200640 1 --2- 172.21.15.2:0/4124105201 >> v2:172.21.15.137:3300/0 conn(0x7fd3a80702c0 0x7fd3a80b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd3acc00640 1 -- 172.21.15.2:0/4124105201 >> v2:172.21.15.137:3300/0 conn(0x7fd3a80702c0 msgr2=0x7fd3a80b3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:49.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd3acc00640 1 --2- 172.21.15.2:0/4124105201 >> v2:172.21.15.137:3300/0 conn(0x7fd3a80702c0 0x7fd3a80b3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd3acc00640 1 -- 172.21.15.2:0/4124105201 >> v1:172.21.15.137:6789/0 conn(0x7fd3a81439f0 legacy=0x7fd3a80cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:49.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd3acc00640 1 -- 172.21.15.2:0/4124105201 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a8143dd0 con 0x7fd3a806c8f0 2024-01-15T18:19:49.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.838+0000 7fd39f200640 1 --2- 172.21.15.2:0/4124105201 >> v2:172.21.15.137:3300/0 conn(0x7fd3a80702c0 0x7fd3a80b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:49.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.839+0000 7fd3acc00640 1 --2- 172.21.15.2:0/4124105201 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a806c8f0 0x7fd3a806ccf0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8072670 tx=0x7fd3a002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b09fc1a4cfe610 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:49.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.839+0000 7fd3a7400640 1 -- 172.21.15.2:0/4124105201 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a002ea90 con 0x7fd3a806c8f0 2024-01-15T18:19:49.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3a7400640 1 -- 172.21.15.2:0/4124105201 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a002ebf0 con 0x7fd3a806c8f0 2024-01-15T18:19:49.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3a7400640 1 -- 172.21.15.2:0/4124105201 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0009600 con 0x7fd3a806c8f0 2024-01-15T18:19:49.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 -- 172.21.15.2:0/4124105201 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a806c8f0 msgr2=0x7fd3a806ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 --2- 172.21.15.2:0/4124105201 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a806c8f0 0x7fd3a806ccf0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fd3a8072670 tx=0x7fd3a002e0f0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 -- 172.21.15.2:0/4124105201 shutdown_connections 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 --2- 172.21.15.2:0/4124105201 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a806c8f0 0x7fd3a806ccf0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 --2- 172.21.15.2:0/4124105201 >> v2:172.21.15.137:3300/0 conn(0x7fd3a80702c0 0x7fd3a80b3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 -- 172.21.15.2:0/4124105201 >> 172.21.15.2:0/4124105201 conn(0x7fd3a80b45f0 msgr2=0x7fd3a80b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 -- 172.21.15.2:0/4124105201 shutdown_connections 2024-01-15T18:19:49.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.840+0000 7fd3aec00640 1 -- 172.21.15.2:0/4124105201 wait complete. 2024-01-15T18:19:49.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.842+0000 7fd3aec00640 1 Processor -- start 2024-01-15T18:19:49.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.842+0000 7fd3aec00640 1 -- start start 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3aec00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3a806c8f0 0x7fd3a80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3aec00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3a80702c0 0x7fd3a815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3aec00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 0x7fd3a815c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3aec00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80ae370 con 0x7fd3a81439f0 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3aec00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a815d080 con 0x7fd3a806c8f0 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3aec00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a815c5c0 con 0x7fd3a80702c0 2024-01-15T18:19:49.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3acc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 0x7fd3a815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3acc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 0x7fd3a815c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41714/0 (socket says 172.21.15.2:41714) 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.843+0000 7fd3acc00640 1 -- 172.21.15.2:0/3442115457 learned_addr learned my addr 172.21.15.2:0/3442115457 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd39f200640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3a80702c0 0x7fd3a815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3acc00640 1 -- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3a80702c0 msgr2=0x7fd3a815bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3acc00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3a80702c0 0x7fd3a815bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3acc00640 1 -- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3a806c8f0 msgr2=0x7fd3a80ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3a7e00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3a806c8f0 0x7fd3a80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3acc00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3a806c8f0 0x7fd3a80ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:49.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3acc00640 1 -- 172.21.15.2:0/3442115457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a000bd40 con 0x7fd3a81439f0 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3a7e00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3a806c8f0 0x7fd3a80ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.844+0000 7fd3acc00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 0x7fd3a815c080 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000d240 tx=0x7fd3a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.845+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a002ea90 con 0x7fd3a81439f0 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.845+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a000b720 con 0x7fd3a81439f0 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.845+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0040880 con 0x7fd3a81439f0 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.845+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a815c7e0 con 0x7fd3a81439f0 2024-01-15T18:19:49.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.845+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a815cda0 con 0x7fd3a81439f0 2024-01-15T18:19:49.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.846+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a80686f0 con 0x7fd3a81439f0 2024-01-15T18:19:49.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.847+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd3a00409e0 con 0x7fd3a81439f0 2024-01-15T18:19:49.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.848+0000 7fd3a4c00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd38006c550 0x7fd38006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:49.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.848+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fd3a00adc50 con 0x7fd3a81439f0 2024-01-15T18:19:49.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.849+0000 7fd3a7e00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd38006c550 0x7fd38006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:49.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.850+0000 7fd3a7e00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd38006c550 0x7fd38006ea10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd39800a830 tx=0x7fd398008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:49.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:49.852+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3a007a490 con 0x7fd3a81439f0 2024-01-15T18:19:50.050 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.046+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd3a80b2db0 con 0x7fd3a81439f0 2024-01-15T18:19:50.051 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.047+0000 7fd3a4c00640 1 -- 172.21.15.2:0/3442115457 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v116) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd3a0035e80 con 0x7fd3a81439f0 2024-01-15T18:19:50.051 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 32 2024-01-15T18:19:50.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd38006c550 msgr2=0x7fd38006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd38006c550 0x7fd38006ea10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fd39800a830 tx=0x7fd398008040 comp rx=0 tx=0).stop 2024-01-15T18:19:50.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 msgr2=0x7fd3a815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 0x7fd3a815c080 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fd3a000d240 tx=0x7fd3a0004a40 comp rx=0 tx=0).stop 2024-01-15T18:19:50.054 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 shutdown_connections 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd38006c550 0x7fd38006ea10 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3a81439f0 0x7fd3a815c080 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3a80702c0 0x7fd3a815bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 --2- 172.21.15.2:0/3442115457 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3a806c8f0 0x7fd3a80ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 >> 172.21.15.2:0/3442115457 conn(0x7fd3a80b45f0 msgr2=0x7fd3a80aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 shutdown_connections 2024-01-15T18:19:50.055 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.050+0000 7fd3aec00640 1 -- 172.21.15.2:0/3442115457 wait complete. 2024-01-15T18:19:50.075 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-15T18:19:50.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.297+0000 7f24c7200640 1 Processor -- start 2024-01-15T18:19:50.300 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.297+0000 7f24c7200640 1 -- start start 2024-01-15T18:19:50.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.298+0000 7f24c7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.299+0000 7f24c7200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c80d1250 con 0x7f24c80d1c50 2024-01-15T18:19:50.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.299+0000 7f24c7200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24c80ce8e0 con 0x7f24c80b2210 2024-01-15T18:19:50.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.299+0000 7f24c7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80ce6d0 con 0x7f24c80d1870 2024-01-15T18:19:50.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.299+0000 7f24c6800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f24c80d1c50 0x7f24c80b26f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46524/0 (socket says 172.21.15.2:46524) 2024-01-15T18:19:50.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.299+0000 7f24c6800640 1 -- 172.21.15.2:0/229889401 learned_addr learned my addr 172.21.15.2:0/229889401 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:50.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.299+0000 7f24c5e00640 1 --2- 172.21.15.2:0/229889401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.300+0000 7f24c5400640 1 -- 172.21.15.2:0/229889401 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488065597 0 0) 0x7f24c80d1250 con 0x7f24c80d1c50 2024-01-15T18:19:50.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.300+0000 7f24c5e00640 1 -- 172.21.15.2:0/229889401 >> v1:172.21.15.137:6789/0 conn(0x7f24c80b2210 legacy=0x7f24c815d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.300+0000 7f24c5e00640 1 -- 172.21.15.2:0/229889401 >> v1:172.21.15.2:6789/0 conn(0x7f24c80d1c50 legacy=0x7f24c80b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.300+0000 7f24c5e00640 1 -- 172.21.15.2:0/229889401 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c80cebc0 con 0x7f24c80d1870 2024-01-15T18:19:50.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.301+0000 7f24c5e00640 1 --2- 172.21.15.2:0/229889401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80d3bd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f24b80097d0 tx=0x7f24b802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b4c6163d9eba6be server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.305+0000 7f24c5400640 1 -- 172.21.15.2:0/229889401 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b8031890 con 0x7f24c80d1870 2024-01-15T18:19:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.305+0000 7f24c5400640 1 -- 172.21.15.2:0/229889401 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24b80319f0 con 0x7f24c80d1870 2024-01-15T18:19:50.308 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.305+0000 7f24c5400640 1 -- 172.21.15.2:0/229889401 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24b8031d80 con 0x7f24c80d1870 2024-01-15T18:19:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.305+0000 7f24c7200640 1 -- 172.21.15.2:0/229889401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 msgr2=0x7f24c80d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.305+0000 7f24c7200640 1 --2- 172.21.15.2:0/229889401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80d3bd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f24b80097d0 tx=0x7f24b802b930 comp rx=0 tx=0).stop 2024-01-15T18:19:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.306+0000 7f24c7200640 1 -- 172.21.15.2:0/229889401 shutdown_connections 2024-01-15T18:19:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.306+0000 7f24c7200640 1 --2- 172.21.15.2:0/229889401 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80d3bd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.306+0000 7f24c7200640 1 -- 172.21.15.2:0/229889401 >> 172.21.15.2:0/229889401 conn(0x7f24c80d30b0 msgr2=0x7f24c80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:50.309 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.306+0000 7f24c7200640 1 -- 172.21.15.2:0/229889401 shutdown_connections 2024-01-15T18:19:50.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.306+0000 7f24c7200640 1 -- 172.21.15.2:0/229889401 wait complete. 2024-01-15T18:19:50.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.307+0000 7f24c7200640 1 Processor -- start 2024-01-15T18:19:50.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.307+0000 7f24c7200640 1 -- start start 2024-01-15T18:19:50.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.308+0000 7f24c7200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c80b2210 0x7f24c80c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.308+0000 7f24c7200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80c6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.308+0000 7f24c7200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 0x7f24c8157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.308+0000 7f24c7200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80c7120 con 0x7f24c80d1c50 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.308+0000 7f24c7200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80c7330 con 0x7f24c80b2210 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.308+0000 7f24c7200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24c80c7610 con 0x7f24c80d1870 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 0x7f24c8157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 0x7f24c8157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41730/0 (socket says 172.21.15.2:41730) 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 -- 172.21.15.2:0/860319649 learned_addr learned my addr 172.21.15.2:0/860319649 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24bfe00640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80c6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 -- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 msgr2=0x7f24c80c6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.314 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80c6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c5e00640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c80b2210 0x7f24c80c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 -- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c80b2210 msgr2=0x7f24c80c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c80b2210 0x7f24c80c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 -- 172.21.15.2:0/860319649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b8009400 con 0x7f24c80d1c50 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c5e00640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c80b2210 0x7f24c80c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.309+0000 7f24c6800640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 0x7f24c8157920 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c700 tx=0x7f24c000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.310+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000ce10 con 0x7f24c80d1c50 2024-01-15T18:19:50.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.310+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c000a640 con 0x7f24c80d1c50 2024-01-15T18:19:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.310+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c000a9f0 con 0x7f24c80d1c50 2024-01-15T18:19:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.310+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24c80c78f0 con 0x7f24c80d1c50 2024-01-15T18:19:50.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.310+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24c80c7eb0 con 0x7f24c80d1c50 2024-01-15T18:19:50.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.311+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24c80c80d0 con 0x7f24c80d1c50 2024-01-15T18:19:50.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.312+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f24c000ab50 con 0x7f24c80d1c50 2024-01-15T18:19:50.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.313+0000 7f24be000640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24a006c550 0x7f24a006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.313+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f24c0087fe0 con 0x7f24c80d1c50 2024-01-15T18:19:50.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.314+0000 7f24c5e00640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24a006c550 0x7f24a006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.315+0000 7f24c5e00640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24a006c550 0x7f24a006ea10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f24b8036000 tx=0x7f24b8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:50.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.317+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24c0054490 con 0x7f24c80d1c50 2024-01-15T18:19:50.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.511+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f24c8074770 con 0x7f24c80d1c50 2024-01-15T18:19:50.515 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.512+0000 7f24be000640 1 -- 172.21.15.2:0/860319649 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v116) v1 ==== 100+0+11 (secure 0 0 0) 0x7f24c0018030 con 0x7f24c80d1c50 2024-01-15T18:19:50.515 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 64 2024-01-15T18:19:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24a006c550 msgr2=0x7f24a006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24a006c550 0x7f24a006ea10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f24b8036000 tx=0x7f24b8008030 comp rx=0 tx=0).stop 2024-01-15T18:19:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 msgr2=0x7f24c8157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 0x7f24c8157920 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f24c000c700 tx=0x7f24c000cbc0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 shutdown_connections 2024-01-15T18:19:50.518 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f24a006c550 0x7f24a006ea10 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f24c80d1c50 0x7f24c8157920 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f24c80d1870 0x7f24c80c6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 --2- 172.21.15.2:0/860319649 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f24c80b2210 0x7f24c80c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 >> 172.21.15.2:0/860319649 conn(0x7f24c80d30b0 msgr2=0x7f24c815b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:50.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 shutdown_connections 2024-01-15T18:19:50.519 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.514+0000 7f24c7200640 1 -- 172.21.15.2:0/860319649 wait complete. 2024-01-15T18:19:50.537 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-15T18:19:50.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.755+0000 7f2b6ea00640 1 Processor -- start 2024-01-15T18:19:50.759 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.756+0000 7f2b6ea00640 1 -- start start 2024-01-15T18:19:50.760 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.757+0000 7f2b6ea00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f2b680b2340 0x7f2b680ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.757+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b6806c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.757+0000 7f2b6ea00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6806c950 con 0x7f2b6815ca80 2024-01-15T18:19:50.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.757+0000 7f2b6ea00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b680d0540 con 0x7f2b680b2340 2024-01-15T18:19:50.761 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.757+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680b2720 con 0x7f2b6806bfd0 2024-01-15T18:19:50.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.758+0000 7f2b6e000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2b6815ca80 0x7f2b6815cf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:46526/0 (socket says 172.21.15.2:46526) 2024-01-15T18:19:50.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.758+0000 7f2b6e000640 1 -- 172.21.15.2:0/3004995709 learned_addr learned my addr 172.21.15.2:0/3004995709 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:50.762 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.758+0000 7f2b6d600640 1 --2- 172.21.15.2:0/3004995709 >> v2:172.21.15.137:3300/0 conn(0x7f2b680b2340 0x7f2b680ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.758+0000 7f2b6cc00640 1 --2- 172.21.15.2:0/3004995709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b6806c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.758+0000 7f2b67e00640 1 -- 172.21.15.2:0/3004995709 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528295327 0 0) 0x7f2b6806c950 con 0x7f2b6815ca80 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.758+0000 7f2b67e00640 1 -- 172.21.15.2:0/3004995709 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b48003520 con 0x7f2b6815ca80 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.759+0000 7f2b67e00640 1 -- 172.21.15.2:0/3004995709 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 613639728 0 0) 0x7f2b48003520 con 0x7f2b6815ca80 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.759+0000 7f2b6cc00640 1 -- 172.21.15.2:0/3004995709 >> v2:172.21.15.137:3300/0 conn(0x7f2b680b2340 msgr2=0x7f2b680ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.759+0000 7f2b6cc00640 1 --2- 172.21.15.2:0/3004995709 >> v2:172.21.15.137:3300/0 conn(0x7f2b680b2340 0x7f2b680ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.759+0000 7f2b6cc00640 1 -- 172.21.15.2:0/3004995709 >> v1:172.21.15.2:6789/0 conn(0x7f2b6815ca80 legacy=0x7f2b6815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.759+0000 7f2b6cc00640 1 -- 172.21.15.2:0/3004995709 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b680ce700 con 0x7f2b6806bfd0 2024-01-15T18:19:50.763 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.759+0000 7f2b6d600640 1 --2- 172.21.15.2:0/3004995709 >> v2:172.21.15.137:3300/0 conn(0x7f2b680b2340 0x7f2b680ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-15T18:19:50.765 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.760+0000 7f2b6cc00640 1 --2- 172.21.15.2:0/3004995709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b6806c410 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b500097a0 tx=0x7f2b5002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7351dbd3b928c5d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b67e00640 1 -- 172.21.15.2:0/3004995709 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b600019f0 con 0x7f2b6806bfd0 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b67e00640 1 -- 172.21.15.2:0/3004995709 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5002e050 con 0x7f2b6806bfd0 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b67e00640 1 -- 172.21.15.2:0/3004995709 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50030730 con 0x7f2b6806bfd0 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b6ea00640 1 -- 172.21.15.2:0/3004995709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 msgr2=0x7f2b6806c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/3004995709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b6806c410 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2b500097a0 tx=0x7f2b5002b790 comp rx=0 tx=0).stop 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b6ea00640 1 -- 172.21.15.2:0/3004995709 shutdown_connections 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/3004995709 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b6806c410 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.766 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/3004995709 >> v2:172.21.15.137:3300/0 conn(0x7f2b680b2340 0x7f2b680ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.761+0000 7f2b6ea00640 1 -- 172.21.15.2:0/3004995709 >> 172.21.15.2:0/3004995709 conn(0x7f2b680d0e10 msgr2=0x7f2b680d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:50.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.762+0000 7f2b6ea00640 1 -- 172.21.15.2:0/3004995709 shutdown_connections 2024-01-15T18:19:50.767 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.762+0000 7f2b6ea00640 1 -- 172.21.15.2:0/3004995709 wait complete. 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.763+0000 7f2b6ea00640 1 Processor -- start 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.763+0000 7f2b6ea00640 1 -- start start 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.764+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b68155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.764+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 0x7f2b68151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.764+0000 7f2b6ea00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b6815ca80 0x7f2b681530c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.764+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6809eb10 con 0x7f2b680b2340 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.764+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68152140 con 0x7f2b6815ca80 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.764+0000 7f2b6ea00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6815d070 con 0x7f2b6806bfd0 2024-01-15T18:19:50.769 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 0x7f2b68151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 0x7f2b68151c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41742/0 (socket says 172.21.15.2:41742) 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 -- 172.21.15.2:0/2692554293 learned_addr learned my addr 172.21.15.2:0/2692554293 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6d600640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b68155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6e000640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b6815ca80 0x7f2b681530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 -- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 msgr2=0x7f2b68155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b68155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 -- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b6815ca80 msgr2=0x7f2b681530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.770 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b6815ca80 0x7f2b681530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6cc00640 1 -- 172.21.15.2:0/2692554293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b58009400 con 0x7f2b680b2340 2024-01-15T18:19:50.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.765+0000 7f2b6e000640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b6815ca80 0x7f2b681530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:50.771 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.766+0000 7f2b6cc00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 0x7f2b68151c00 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f2b500098c0 tx=0x7f2b500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:50.773 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.766+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5003c880 con 0x7f2b680b2340 2024-01-15T18:19:50.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.766+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5003c9e0 con 0x7f2b680b2340 2024-01-15T18:19:50.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.766+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5003cd90 con 0x7f2b680b2340 2024-01-15T18:19:50.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.766+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b50009400 con 0x7f2b680b2340 2024-01-15T18:19:50.774 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.766+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6809e550 con 0x7f2b680b2340 2024-01-15T18:19:50.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.767+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b6815ced0 con 0x7f2b680b2340 2024-01-15T18:19:50.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.768+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2b50039040 con 0x7f2b680b2340 2024-01-15T18:19:50.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.769+0000 7f2b65600640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b4006c550 0x7f2b4006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:50.775 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.769+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f2b500aa4e0 con 0x7f2b680b2340 2024-01-15T18:19:50.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.770+0000 7f2b6d600640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b4006c550 0x7f2b4006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:50.776 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.771+0000 7f2b6d600640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b4006c550 0x7f2b4006ea10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009c60 tx=0x7f2b58008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:50.777 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.774+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b6815ced0 con 0x7f2b680b2340 2024-01-15T18:19:50.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.980+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2b6815ced0 con 0x7f2b680b2340 2024-01-15T18:19:50.984 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.981+0000 7f2b65600640 1 -- 172.21.15.2:0/2692554293 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v116) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2b6815ced0 con 0x7f2b680b2340 2024-01-15T18:19:50.985 INFO:teuthology.orchestra.run.smithi002.stderr:listed 0 entries 2024-01-15T18:19:50.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b4006c550 msgr2=0x7f2b4006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.987 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b4006c550 0x7f2b4006ea10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009c60 tx=0x7f2b58008040 comp rx=0 tx=0).stop 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 msgr2=0x7f2b68151c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 0x7f2b68151c00 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f2b500098c0 tx=0x7f2b500046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 shutdown_connections 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2b4006c550 0x7f2b4006ea10 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2b6815ca80 0x7f2b681530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2b680b2340 0x7f2b68151c00 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 --2- 172.21.15.2:0/2692554293 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2b6806bfd0 0x7f2b68155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 >> 172.21.15.2:0/2692554293 conn(0x7f2b680d0e10 msgr2=0x7f2b680c6070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:50.988 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 shutdown_connections 2024-01-15T18:19:50.989 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:50.984+0000 7f2b6ea00640 1 -- 172.21.15.2:0/2692554293 wait complete. 2024-01-15T18:19:51.007 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi137.front.sepia.ceph.com... 2024-01-15T18:19:51.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-15T18:19:51.103 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi137.stderr:ceph-fuse[47446]: fuse finished with error 0 and tester_r 0 2024-01-15T18:19:51.107 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:19:51.139 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:19:51.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.140 DEBUG:teuthology.orchestra.run.smithi137:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-15T18:19:51.236 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-01-15T18:19:51.236 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-15T18:19:51.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053848,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249396k,nr_inodes=812349,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498796k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-15T18:19:51.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.288 DEBUG:teuthology.orchestra.run.smithi002:> ip netns list 2024-01-15T18:19:51.348 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-15T18:19:51.348 INFO:teuthology.orchestra.run.smithi002.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-15T18:19:51.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.349 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-15T18:19:51.454 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.454 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-15T18:19:51.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.541 DEBUG:teuthology.orchestra.run.smithi002:> sudo ip link delete ceph-brx 2024-01-15T18:19:51.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.674 DEBUG:teuthology.orchestra.run.smithi137:> ip netns list 2024-01-15T18:19:51.730 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-15T18:19:51.731 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-15T18:19:51.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-15T18:19:51.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-15T18:19:51.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-15T18:19:51.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo ip link delete ceph-brx 2024-01-15T18:19:52.055 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-15T18:19:52.067 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-15T18:19:52.067 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:19:52.301 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.298+0000 7f828ac00640 1 Processor -- start 2024-01-15T18:19:52.302 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.298+0000 7f828ac00640 1 -- start start 2024-01-15T18:19:52.303 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.300+0000 7f828ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 0x7f82840b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.300+0000 7f828ac00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f82840b2080 0x7f82840cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.300+0000 7f828ac00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82840ccbe0 con 0x7f8284143810 2024-01-15T18:19:52.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.300+0000 7f828ac00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82840b3520 con 0x7f82840b2080 2024-01-15T18:19:52.304 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.300+0000 7f828ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8284143bf0 con 0x7f828406e1e0 2024-01-15T18:19:52.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.301+0000 7f8288e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 0x7f82840b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.301+0000 7f8289800640 1 --1- >> v1:172.21.15.137:6789/0 conn(0x7f8284143810 0x7f828406dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.137:6789/0 says I am v1:172.21.15.2:54708/0 (socket says 172.21.15.2:54708) 2024-01-15T18:19:52.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.301+0000 7f8288e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 0x7f82840b1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:33056/0 (socket says 172.21.15.2:33056) 2024-01-15T18:19:52.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.301+0000 7f8289800640 1 -- 172.21.15.2:0/3866745852 learned_addr learned my addr 172.21.15.2:0/3866745852 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:52.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.301+0000 7f828a200640 1 --2- 172.21.15.2:0/3866745852 >> v2:172.21.15.137:3300/0 conn(0x7f82840b2080 0x7f82840cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.302+0000 7f8288e00640 1 -- 172.21.15.2:0/3866745852 >> v2:172.21.15.137:3300/0 conn(0x7f82840b2080 msgr2=0x7f82840cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:52.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.302+0000 7f8288e00640 1 --2- 172.21.15.2:0/3866745852 >> v2:172.21.15.137:3300/0 conn(0x7f82840b2080 0x7f82840cd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.302+0000 7f8288e00640 1 -- 172.21.15.2:0/3866745852 >> v1:172.21.15.137:6789/0 conn(0x7f8284143810 legacy=0x7f828406dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.302+0000 7f8288e00640 1 -- 172.21.15.2:0/3866745852 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82840b2ac0 con 0x7f828406e1e0 2024-01-15T18:19:52.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.304+0000 7f8288e00640 1 --2- 172.21.15.2:0/3866745852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 0x7f82840b1b40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f826c00ba40 tx=0x7f826c02dff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b16523140870b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:52.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.306+0000 7f8283e00640 1 -- 172.21.15.2:0/3866745852 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c02eda0 con 0x7f828406e1e0 2024-01-15T18:19:52.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.306+0000 7f8283e00640 1 -- 172.21.15.2:0/3866745852 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c036440 con 0x7f828406e1e0 2024-01-15T18:19:52.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.306+0000 7f8283e00640 1 -- 172.21.15.2:0/3866745852 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c0367d0 con 0x7f828406e1e0 2024-01-15T18:19:52.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.306+0000 7f828ac00640 1 -- 172.21.15.2:0/3866745852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 msgr2=0x7f82840b1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.306+0000 7f828ac00640 1 --2- 172.21.15.2:0/3866745852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 0x7f82840b1b40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f826c00ba40 tx=0x7f826c02dff0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.307+0000 7f828ac00640 1 -- 172.21.15.2:0/3866745852 shutdown_connections 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.307+0000 7f828ac00640 1 --2- 172.21.15.2:0/3866745852 >> v2:172.21.15.137:3300/0 conn(0x7f82840b2080 0x7f82840cd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.307+0000 7f828ac00640 1 --2- 172.21.15.2:0/3866745852 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f828406e1e0 0x7f82840b1b40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.307+0000 7f828ac00640 1 -- 172.21.15.2:0/3866745852 >> 172.21.15.2:0/3866745852 conn(0x7f82840b0840 msgr2=0x7f82840b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.307+0000 7f828ac00640 1 -- 172.21.15.2:0/3866745852 shutdown_connections 2024-01-15T18:19:52.312 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.307+0000 7f828ac00640 1 -- 172.21.15.2:0/3866745852 wait complete. 2024-01-15T18:19:52.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.309+0000 7f828ac00640 1 Processor -- start 2024-01-15T18:19:52.313 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.309+0000 7f828ac00640 1 -- start start 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.310+0000 7f828ac00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f828406e1e0 0x7f82841591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.310+0000 7f828ac00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 0x7f82840c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.310+0000 7f828ac00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8284143810 0x7f82841f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.310+0000 7f828ac00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284130af0 con 0x7f82840b2080 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.310+0000 7f828ac00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f828415a110 con 0x7f828406e1e0 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.310+0000 7f828ac00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82840c4ec0 con 0x7f8284143810 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 0x7f82840c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.315 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 0x7f82840c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41754/0 (socket says 172.21.15.2:41754) 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 -- 172.21.15.2:0/2271030563 learned_addr learned my addr 172.21.15.2:0/2271030563 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8289800640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f828406e1e0 0x7f82841591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f828a200640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8284143810 0x7f82841f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 -- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8284143810 msgr2=0x7f82841f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8284143810 0x7f82841f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 -- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f828406e1e0 msgr2=0x7f82841591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f828406e1e0 0x7f82841591b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.316 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.311+0000 7f8288e00640 1 -- 172.21.15.2:0/2271030563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c00b6a0 con 0x7f82840b2080 2024-01-15T18:19:52.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.312+0000 7f8288e00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 0x7f82840c4980 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f826c02e600 tx=0x7f826c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:52.317 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.312+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c040810 con 0x7f82840b2080 2024-01-15T18:19:52.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.312+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c040990 con 0x7f82840b2080 2024-01-15T18:19:52.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.312+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826c040d40 con 0x7f82840b2080 2024-01-15T18:19:52.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.312+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8284159870 con 0x7f82840b2080 2024-01-15T18:19:52.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.312+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8284159e30 con 0x7f82840b2080 2024-01-15T18:19:52.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.313+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8284130ca0 con 0x7f82840b2080 2024-01-15T18:19:52.318 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.314+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f826c03f3a0 con 0x7f82840b2080 2024-01-15T18:19:52.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.315+0000 7f8281600640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f825c06c550 0x7f825c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.315+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f826c0ad590 con 0x7f82840b2080 2024-01-15T18:19:52.319 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.316+0000 7f8289800640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f825c06c550 0x7f825c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.320 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.317+0000 7f8289800640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f825c06c550 0x7f825c06ea10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8274005880 tx=0x7f8274009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:52.323 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.320+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f826c079e50 con 0x7f82840b2080 2024-01-15T18:19:52.514 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.510+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82840cc870 con 0x7f825c06c550 2024-01-15T18:19:52.542 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.538+0000 7f8281600640 1 -- 172.21.15.2:0/2271030563 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288946 (secure 0 0 0) 0x7f8274001c50 con 0x7f825c06c550 2024-01-15T18:19:52.546 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:52.547 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:19:52.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f825c06c550 msgr2=0x7f825c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f825c06c550 0x7f825c06ea10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f8274005880 tx=0x7f8274009040 comp rx=0 tx=0).stop 2024-01-15T18:19:52.549 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 msgr2=0x7f82840c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 0x7f82840c4980 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f826c02e600 tx=0x7f826c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 shutdown_connections 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f825c06c550 0x7f825c06ea10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f8284143810 0x7f82841f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f82840b2080 0x7f82840c4980 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 --2- 172.21.15.2:0/2271030563 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f828406e1e0 0x7f82841591b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 >> 172.21.15.2:0/2271030563 conn(0x7f82840b0840 msgr2=0x7f8284141430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 shutdown_connections 2024-01-15T18:19:52.550 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.544+0000 7f828ac00640 1 -- 172.21.15.2:0/2271030563 wait complete. 2024-01-15T18:19:52.567 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2024-01-15T18:19:51.957916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369934362,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1746,"num_write_kb":5246057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3187,"ondisk_log_size":3187,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11070440,"kb_used_data":10499496,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":743904280,"statfs":{"total":773094113280,"available":761757982720,"internally_reserved":0,"allocated":10751483904,"data_stored":10744924764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":584568369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073776448,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":274,"num_write_kb":1048616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004677"},"pg_stats":[{"pgid":"17.3f","version":"116'53","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.408247+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:19:45.408247+0000","last_peered":"2024-01-15T18:19:45.408247+0000","last_clean":"2024-01-15T18:19:45.408247+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:19:45.408247+0000","last_undegraded":"2024-01-15T18:19:45.408247+0000","last_fullsized":"2024-01-15T18:19:45.408247+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.516826+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:19:45.516826+0000","last_peered":"2024-01-15T18:19:45.516826+0000","last_clean":"2024-01-15T18:19:45.516826+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:19:45.516826+0000","last_undegraded":"2024-01-15T18:19:45.516826+0000","last_fullsized":"2024-01-15T18:19:45.516826+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.675239+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:19:45.675239+0000","last_peered":"2024-01-15T18:19:45.675239+0000","last_clean":"2024-01-15T18:19:45.675239+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:19:45.675239+0000","last_undegraded":"2024-01-15T18:19:45.675239+0000","last_fullsized":"2024-01-15T18:19:45.675239+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'45","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.578232+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:19:45.578232+0000","last_peered":"2024-01-15T18:19:45.578232+0000","last_clean":"2024-01-15T18:19:45.578232+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:19:45.578232+0000","last_undegraded":"2024-01-15T18:19:45.578232+0000","last_fullsized":"2024-01-15T18:19:45.578232+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'27","reported_seq":49,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.370622+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:19:45.370622+0000","last_peered":"2024-01-15T18:19:45.370622+0000","last_clean":"2024-01-15T18:19:45.370622+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:19:45.370622+0000","last_undegraded":"2024-01-15T18:19:45.370622+0000","last_fullsized":"2024-01-15T18:19:45.370622+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'48","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410257+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:19:45.410257+0000","last_peered":"2024-01-15T18:19:45.410257+0000","last_clean":"2024-01-15T18:19:45.410257+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:19:45.410257+0000","last_undegraded":"2024-01-15T18:19:45.410257+0000","last_fullsized":"2024-01-15T18:19:45.410257+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'43","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.408637+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:19:45.408637+0000","last_peered":"2024-01-15T18:19:45.408637+0000","last_clean":"2024-01-15T18:19:45.408637+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:19:45.408637+0000","last_undegraded":"2024-01-15T18:19:45.408637+0000","last_fullsized":"2024-01-15T18:19:45.408637+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.398127+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:19:45.398127+0000","last_peered":"2024-01-15T18:19:45.398127+0000","last_clean":"2024-01-15T18:19:45.398127+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:19:45.398127+0000","last_undegraded":"2024-01-15T18:19:45.398127+0000","last_fullsized":"2024-01-15T18:19:45.398127+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.726565+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:19:45.726565+0000","last_peered":"2024-01-15T18:19:45.726565+0000","last_clean":"2024-01-15T18:19:45.726565+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:19:45.726565+0000","last_undegraded":"2024-01-15T18:19:45.726565+0000","last_fullsized":"2024-01-15T18:19:45.726565+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'47","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.827721+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:19:45.827721+0000","last_peered":"2024-01-15T18:19:45.827721+0000","last_clean":"2024-01-15T18:19:45.827721+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:19:45.827721+0000","last_undegraded":"2024-01-15T18:19:45.827721+0000","last_fullsized":"2024-01-15T18:19:45.827721+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'47","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.524665+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:19:45.524665+0000","last_peered":"2024-01-15T18:19:45.524665+0000","last_clean":"2024-01-15T18:19:45.524665+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:19:45.524665+0000","last_undegraded":"2024-01-15T18:19:45.524665+0000","last_fullsized":"2024-01-15T18:19:45.524665+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'48","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.634818+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:19:45.634818+0000","last_peered":"2024-01-15T18:19:45.634818+0000","last_clean":"2024-01-15T18:19:45.634818+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:19:45.634818+0000","last_undegraded":"2024-01-15T18:19:45.634818+0000","last_fullsized":"2024-01-15T18:19:45.634818+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'38","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.393963+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:19:45.393963+0000","last_peered":"2024-01-15T18:19:45.393963+0000","last_clean":"2024-01-15T18:19:45.393963+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:19:45.393963+0000","last_undegraded":"2024-01-15T18:19:45.393963+0000","last_fullsized":"2024-01-15T18:19:45.393963+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'51","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.775084+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:19:45.775084+0000","last_peered":"2024-01-15T18:19:45.775084+0000","last_clean":"2024-01-15T18:19:45.775084+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:19:45.775084+0000","last_undegraded":"2024-01-15T18:19:45.775084+0000","last_fullsized":"2024-01-15T18:19:45.775084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'34","reported_seq":53,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.681649+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:19:45.681649+0000","last_peered":"2024-01-15T18:19:45.681649+0000","last_clean":"2024-01-15T18:19:45.681649+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:19:45.681649+0000","last_undegraded":"2024-01-15T18:19:45.681649+0000","last_fullsized":"2024-01-15T18:19:45.681649+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'42","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.390578+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:19:45.390578+0000","last_peered":"2024-01-15T18:19:45.390578+0000","last_clean":"2024-01-15T18:19:45.390578+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:19:45.390578+0000","last_undegraded":"2024-01-15T18:19:45.390578+0000","last_fullsized":"2024-01-15T18:19:45.390578+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.458896+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:19:45.458896+0000","last_peered":"2024-01-15T18:19:45.458896+0000","last_clean":"2024-01-15T18:19:45.458896+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:19:45.458896+0000","last_undegraded":"2024-01-15T18:19:45.458896+0000","last_fullsized":"2024-01-15T18:19:45.458896+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'48","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.699878+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:19:45.699878+0000","last_peered":"2024-01-15T18:19:45.699878+0000","last_clean":"2024-01-15T18:19:45.699878+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:19:45.699878+0000","last_undegraded":"2024-01-15T18:19:45.699878+0000","last_fullsized":"2024-01-15T18:19:45.699878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.465695+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:19:45.465695+0000","last_peered":"2024-01-15T18:19:45.465695+0000","last_clean":"2024-01-15T18:19:45.465695+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:19:45.465695+0000","last_undegraded":"2024-01-15T18:19:45.465695+0000","last_fullsized":"2024-01-15T18:19:45.465695+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.700450+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:19:45.700450+0000","last_peered":"2024-01-15T18:19:45.700450+0000","last_clean":"2024-01-15T18:19:45.700450+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:19:45.700450+0000","last_undegraded":"2024-01-15T18:19:45.700450+0000","last_fullsized":"2024-01-15T18:19:45.700450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'56","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.427319+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:19:45.427319+0000","last_peered":"2024-01-15T18:19:45.427319+0000","last_clean":"2024-01-15T18:19:45.427319+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:19:45.427319+0000","last_undegraded":"2024-01-15T18:19:45.427319+0000","last_fullsized":"2024-01-15T18:19:45.427319+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.823341+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:19:45.823341+0000","last_peered":"2024-01-15T18:19:45.823341+0000","last_clean":"2024-01-15T18:19:45.823341+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:19:45.823341+0000","last_undegraded":"2024-01-15T18:19:45.823341+0000","last_fullsized":"2024-01-15T18:19:45.823341+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'39","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.605337+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:19:45.605337+0000","last_peered":"2024-01-15T18:19:45.605337+0000","last_clean":"2024-01-15T18:19:45.605337+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:19:45.605337+0000","last_undegraded":"2024-01-15T18:19:45.605337+0000","last_fullsized":"2024-01-15T18:19:45.605337+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'43","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.704152+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:19:45.704152+0000","last_peered":"2024-01-15T18:19:45.704152+0000","last_clean":"2024-01-15T18:19:45.704152+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:19:45.704152+0000","last_undegraded":"2024-01-15T18:19:45.704152+0000","last_fullsized":"2024-01-15T18:19:45.704152+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'38","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.411545+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:19:45.411545+0000","last_peered":"2024-01-15T18:19:45.411545+0000","last_clean":"2024-01-15T18:19:45.411545+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:19:45.411545+0000","last_undegraded":"2024-01-15T18:19:45.411545+0000","last_fullsized":"2024-01-15T18:19:45.411545+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'53","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.722921+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:19:45.722921+0000","last_peered":"2024-01-15T18:19:45.722921+0000","last_clean":"2024-01-15T18:19:45.722921+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:45.722921+0000","last_undegraded":"2024-01-15T18:19:45.722921+0000","last_fullsized":"2024-01-15T18:19:45.722921+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'37","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.383134+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:19:45.383134+0000","last_peered":"2024-01-15T18:19:45.383134+0000","last_clean":"2024-01-15T18:19:45.383134+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:19:45.383134+0000","last_undegraded":"2024-01-15T18:19:45.383134+0000","last_fullsized":"2024-01-15T18:19:45.383134+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'38","reported_seq":56,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.625653+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:19:45.625653+0000","last_peered":"2024-01-15T18:19:45.625653+0000","last_clean":"2024-01-15T18:19:45.625653+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:19:45.625653+0000","last_undegraded":"2024-01-15T18:19:45.625653+0000","last_fullsized":"2024-01-15T18:19:45.625653+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'46","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.705265+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:19:45.705265+0000","last_peered":"2024-01-15T18:19:45.705265+0000","last_clean":"2024-01-15T18:19:45.705265+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:19:45.705265+0000","last_undegraded":"2024-01-15T18:19:45.705265+0000","last_fullsized":"2024-01-15T18:19:45.705265+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'41","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.642104+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:19:45.642104+0000","last_peered":"2024-01-15T18:19:45.642104+0000","last_clean":"2024-01-15T18:19:45.642104+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:19:45.642104+0000","last_undegraded":"2024-01-15T18:19:45.642104+0000","last_fullsized":"2024-01-15T18:19:45.642104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'62","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.411970+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:19:45.411970+0000","last_peered":"2024-01-15T18:19:45.411970+0000","last_clean":"2024-01-15T18:19:45.411970+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:19:45.411970+0000","last_undegraded":"2024-01-15T18:19:45.411970+0000","last_fullsized":"2024-01-15T18:19:45.411970+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'54","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.560995+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:19:45.560995+0000","last_peered":"2024-01-15T18:19:45.560995+0000","last_clean":"2024-01-15T18:19:45.560995+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:19:45.560995+0000","last_undegraded":"2024-01-15T18:19:45.560995+0000","last_fullsized":"2024-01-15T18:19:45.560995+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'34","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.377219+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:19:45.377219+0000","last_peered":"2024-01-15T18:19:45.377219+0000","last_clean":"2024-01-15T18:19:45.377219+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:19:45.377219+0000","last_undegraded":"2024-01-15T18:19:45.377219+0000","last_fullsized":"2024-01-15T18:19:45.377219+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":20,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.725727+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:16:24.725727+0000","last_peered":"2024-01-15T18:16:24.725727+0000","last_clean":"2024-01-15T18:16:24.725727+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:16:24.725727+0000","last_undegraded":"2024-01-15T18:16:24.725727+0000","last_fullsized":"2024-01-15T18:16:24.725727+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.412953+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:19:45.412953+0000","last_peered":"2024-01-15T18:19:45.412953+0000","last_clean":"2024-01-15T18:19:45.412953+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:19:45.412953+0000","last_undegraded":"2024-01-15T18:19:45.412953+0000","last_fullsized":"2024-01-15T18:19:45.412953+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:14.988989+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:16:14.988989+0000","last_peered":"2024-01-15T18:16:14.988989+0000","last_clean":"2024-01-15T18:16:14.988989+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:16:14.988989+0000","last_undegraded":"2024-01-15T18:16:14.988989+0000","last_fullsized":"2024-01-15T18:16:14.988989+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'43","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.532353+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:19:45.532353+0000","last_peered":"2024-01-15T18:19:45.532353+0000","last_clean":"2024-01-15T18:19:45.532353+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:19:45.532353+0000","last_undegraded":"2024-01-15T18:19:45.532353+0000","last_fullsized":"2024-01-15T18:19:45.532353+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'3","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.687049+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:16:49.687049+0000","last_peered":"2024-01-15T18:16:49.687049+0000","last_clean":"2024-01-15T18:16:49.687049+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:16:49.687049+0000","last_undegraded":"2024-01-15T18:16:49.687049+0000","last_fullsized":"2024-01-15T18:16:49.687049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'56","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.396172+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:19:45.396172+0000","last_peered":"2024-01-15T18:19:45.396172+0000","last_clean":"2024-01-15T18:19:45.396172+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:19:45.396172+0000","last_undegraded":"2024-01-15T18:19:45.396172+0000","last_fullsized":"2024-01-15T18:19:45.396172+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.687023+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:16:49.687023+0000","last_peered":"2024-01-15T18:16:49.687023+0000","last_clean":"2024-01-15T18:16:49.687023+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:16:49.687023+0000","last_undegraded":"2024-01-15T18:16:49.687023+0000","last_fullsized":"2024-01-15T18:16:49.687023+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'39","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.786680+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:19:45.786680+0000","last_peered":"2024-01-15T18:19:45.786680+0000","last_clean":"2024-01-15T18:19:45.786680+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:19:45.786680+0000","last_undegraded":"2024-01-15T18:19:45.786680+0000","last_fullsized":"2024-01-15T18:19:45.786680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.731109+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:16:49.731109+0000","last_peered":"2024-01-15T18:16:49.731109+0000","last_clean":"2024-01-15T18:16:49.731109+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:16:49.731109+0000","last_undegraded":"2024-01-15T18:16:49.731109+0000","last_fullsized":"2024-01-15T18:16:49.731109+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'49","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.613573+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:19:45.613573+0000","last_peered":"2024-01-15T18:19:45.613573+0000","last_clean":"2024-01-15T18:19:45.613573+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:19:45.613573+0000","last_undegraded":"2024-01-15T18:19:45.613573+0000","last_fullsized":"2024-01-15T18:19:45.613573+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.704971+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:16:49.704971+0000","last_peered":"2024-01-15T18:16:49.704971+0000","last_clean":"2024-01-15T18:16:49.704971+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:16:49.704971+0000","last_undegraded":"2024-01-15T18:16:49.704971+0000","last_fullsized":"2024-01-15T18:16:49.704971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'53","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.610017+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:19:45.610017+0000","last_peered":"2024-01-15T18:19:45.610017+0000","last_clean":"2024-01-15T18:19:45.610017+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:19:45.610017+0000","last_undegraded":"2024-01-15T18:19:45.610017+0000","last_fullsized":"2024-01-15T18:19:45.610017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.818752+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:16:44.818752+0000","last_peered":"2024-01-15T18:16:44.818752+0000","last_clean":"2024-01-15T18:16:44.818752+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:16:44.818752+0000","last_undegraded":"2024-01-15T18:16:44.818752+0000","last_fullsized":"2024-01-15T18:16:44.818752+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'43","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407291+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:19:45.407291+0000","last_peered":"2024-01-15T18:19:45.407291+0000","last_clean":"2024-01-15T18:19:45.407291+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:19:45.407291+0000","last_undegraded":"2024-01-15T18:19:45.407291+0000","last_fullsized":"2024-01-15T18:19:45.407291+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.729197+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:16:44.729197+0000","last_peered":"2024-01-15T18:16:44.729197+0000","last_clean":"2024-01-15T18:16:44.729197+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:16:44.729197+0000","last_undegraded":"2024-01-15T18:16:44.729197+0000","last_fullsized":"2024-01-15T18:16:44.729197+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'42","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.391592+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:19:45.391592+0000","last_peered":"2024-01-15T18:19:45.391592+0000","last_clean":"2024-01-15T18:19:45.391592+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:19:45.391592+0000","last_undegraded":"2024-01-15T18:19:45.391592+0000","last_fullsized":"2024-01-15T18:19:45.391592+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.686133+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:16:44.686133+0000","last_peered":"2024-01-15T18:16:44.686133+0000","last_clean":"2024-01-15T18:16:44.686133+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:16:44.686133+0000","last_undegraded":"2024-01-15T18:16:44.686133+0000","last_fullsized":"2024-01-15T18:16:44.686133+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'40","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.640005+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:19:45.640005+0000","last_peered":"2024-01-15T18:19:45.640005+0000","last_clean":"2024-01-15T18:19:45.640005+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:19:45.640005+0000","last_undegraded":"2024-01-15T18:19:45.640005+0000","last_fullsized":"2024-01-15T18:19:45.640005+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.704016+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:16:44.704016+0000","last_peered":"2024-01-15T18:16:44.704016+0000","last_clean":"2024-01-15T18:16:44.704016+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:16:44.704016+0000","last_undegraded":"2024-01-15T18:16:44.704016+0000","last_fullsized":"2024-01-15T18:16:44.704016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'43","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741116+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:19:45.741116+0000","last_peered":"2024-01-15T18:19:45.741116+0000","last_clean":"2024-01-15T18:19:45.741116+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:19:45.741116+0000","last_undegraded":"2024-01-15T18:19:45.741116+0000","last_fullsized":"2024-01-15T18:19:45.741116+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'13","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:40.230861+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:17:40.230861+0000","last_peered":"2024-01-15T18:17:40.230861+0000","last_clean":"2024-01-15T18:17:40.230861+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:17:40.230861+0000","last_undegraded":"2024-01-15T18:17:40.230861+0000","last_fullsized":"2024-01-15T18:17:40.230861+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'43","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404696+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:19:45.404696+0000","last_peered":"2024-01-15T18:19:45.404696+0000","last_clean":"2024-01-15T18:19:45.404696+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:19:45.404696+0000","last_undegraded":"2024-01-15T18:19:45.404696+0000","last_fullsized":"2024-01-15T18:19:45.404696+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'186","reported_seq":202,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.420919+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:19:45.420919+0000","last_peered":"2024-01-15T18:19:45.420919+0000","last_clean":"2024-01-15T18:19:45.420919+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:19:45.420919+0000","last_undegraded":"2024-01-15T18:19:45.420919+0000","last_fullsized":"2024-01-15T18:19:45.420919+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":631739,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":185,"num_write_kb":705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'44","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410217+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:19:45.410217+0000","last_peered":"2024-01-15T18:19:45.410217+0000","last_clean":"2024-01-15T18:19:45.410217+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:19:45.410217+0000","last_undegraded":"2024-01-15T18:19:45.410217+0000","last_fullsized":"2024-01-15T18:19:45.410217+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.672899+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:16:39.672899+0000","last_peered":"2024-01-15T18:16:39.672899+0000","last_clean":"2024-01-15T18:16:39.672899+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:16:39.672899+0000","last_undegraded":"2024-01-15T18:16:39.672899+0000","last_fullsized":"2024-01-15T18:16:39.672899+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'52","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.597854+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:19:45.597854+0000","last_peered":"2024-01-15T18:19:45.597854+0000","last_clean":"2024-01-15T18:19:45.597854+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:19:45.597854+0000","last_undegraded":"2024-01-15T18:19:45.597854+0000","last_fullsized":"2024-01-15T18:19:45.597854+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.792106+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:16:39.792106+0000","last_peered":"2024-01-15T18:16:39.792106+0000","last_clean":"2024-01-15T18:16:39.792106+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:16:39.792106+0000","last_undegraded":"2024-01-15T18:16:39.792106+0000","last_fullsized":"2024-01-15T18:16:39.792106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'46","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559915+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:19:45.559915+0000","last_peered":"2024-01-15T18:19:45.559915+0000","last_clean":"2024-01-15T18:19:45.559915+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:19:45.559915+0000","last_undegraded":"2024-01-15T18:19:45.559915+0000","last_fullsized":"2024-01-15T18:19:45.559915+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":24,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.684911+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:16:39.684911+0000","last_peered":"2024-01-15T18:16:39.684911+0000","last_clean":"2024-01-15T18:16:39.684911+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:16:39.684911+0000","last_undegraded":"2024-01-15T18:16:39.684911+0000","last_fullsized":"2024-01-15T18:16:39.684911+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'24","reported_seq":43,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.533839+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:19:45.533839+0000","last_peered":"2024-01-15T18:19:45.533839+0000","last_clean":"2024-01-15T18:19:45.533839+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:19:45.533839+0000","last_undegraded":"2024-01-15T18:19:45.533839+0000","last_fullsized":"2024-01-15T18:19:45.533839+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.727345+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:16:34.727345+0000","last_peered":"2024-01-15T18:16:34.727345+0000","last_clean":"2024-01-15T18:16:34.727345+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:16:34.727345+0000","last_undegraded":"2024-01-15T18:16:34.727345+0000","last_fullsized":"2024-01-15T18:16:34.727345+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'36","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.633679+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:19:45.633679+0000","last_peered":"2024-01-15T18:19:45.633679+0000","last_clean":"2024-01-15T18:19:45.633679+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:19:45.633679+0000","last_undegraded":"2024-01-15T18:19:45.633679+0000","last_fullsized":"2024-01-15T18:19:45.633679+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.684130+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:16:34.684130+0000","last_peered":"2024-01-15T18:16:34.684130+0000","last_clean":"2024-01-15T18:16:34.684130+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:16:34.684130+0000","last_undegraded":"2024-01-15T18:16:34.684130+0000","last_fullsized":"2024-01-15T18:16:34.684130+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'44","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559602+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:19:45.559602+0000","last_peered":"2024-01-15T18:19:45.559602+0000","last_clean":"2024-01-15T18:19:45.559602+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:19:45.559602+0000","last_undegraded":"2024-01-15T18:19:45.559602+0000","last_fullsized":"2024-01-15T18:19:45.559602+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.791118+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:16:34.791118+0000","last_peered":"2024-01-15T18:16:34.791118+0000","last_clean":"2024-01-15T18:16:34.791118+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:16:34.791118+0000","last_undegraded":"2024-01-15T18:16:34.791118+0000","last_fullsized":"2024-01-15T18:16:34.791118+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'41","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.673339+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:19:45.673339+0000","last_peered":"2024-01-15T18:19:45.673339+0000","last_clean":"2024-01-15T18:19:45.673339+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:19:45.673339+0000","last_undegraded":"2024-01-15T18:19:45.673339+0000","last_fullsized":"2024-01-15T18:19:45.673339+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.727774+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:16:34.727774+0000","last_peered":"2024-01-15T18:16:34.727774+0000","last_clean":"2024-01-15T18:16:34.727774+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:16:34.727774+0000","last_undegraded":"2024-01-15T18:16:34.727774+0000","last_fullsized":"2024-01-15T18:16:34.727774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.866450+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:19:45.866450+0000","last_peered":"2024-01-15T18:19:45.866450+0000","last_clean":"2024-01-15T18:19:45.866450+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:19:45.866450+0000","last_undegraded":"2024-01-15T18:19:45.866450+0000","last_fullsized":"2024-01-15T18:19:45.866450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'3","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.671975+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:16:34.671975+0000","last_peered":"2024-01-15T18:16:34.671975+0000","last_clean":"2024-01-15T18:16:34.671975+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:16:34.671975+0000","last_undegraded":"2024-01-15T18:16:34.671975+0000","last_fullsized":"2024-01-15T18:16:34.671975+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'43","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.389233+0000","last_change":"2024-01-15T18:15:20.196078+0000","last_active":"2024-01-15T18:19:45.389233+0000","last_peered":"2024-01-15T18:19:45.389233+0000","last_clean":"2024-01-15T18:19:45.389233+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:19:45.389233+0000","last_undegraded":"2024-01-15T18:19:45.389233+0000","last_fullsized":"2024-01-15T18:19:45.389233+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:19.862117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.683005+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:16:29.683005+0000","last_peered":"2024-01-15T18:16:29.683005+0000","last_clean":"2024-01-15T18:16:29.683005+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:16:29.683005+0000","last_undegraded":"2024-01-15T18:16:29.683005+0000","last_fullsized":"2024-01-15T18:16:29.683005+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.587657+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:19:45.587657+0000","last_peered":"2024-01-15T18:19:45.587657+0000","last_clean":"2024-01-15T18:19:45.587657+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:19:45.587657+0000","last_undegraded":"2024-01-15T18:19:45.587657+0000","last_fullsized":"2024-01-15T18:19:45.587657+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.671013+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:16:29.671013+0000","last_peered":"2024-01-15T18:16:29.671013+0000","last_clean":"2024-01-15T18:16:29.671013+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:16:29.671013+0000","last_undegraded":"2024-01-15T18:16:29.671013+0000","last_fullsized":"2024-01-15T18:16:29.671013+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'42","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.765594+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:19:45.765594+0000","last_peered":"2024-01-15T18:19:45.765594+0000","last_clean":"2024-01-15T18:19:45.765594+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:19:45.765594+0000","last_undegraded":"2024-01-15T18:19:45.765594+0000","last_fullsized":"2024-01-15T18:19:45.765594+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.725995+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:16:29.725995+0000","last_peered":"2024-01-15T18:16:29.725995+0000","last_clean":"2024-01-15T18:16:29.725995+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:16:29.725995+0000","last_undegraded":"2024-01-15T18:16:29.725995+0000","last_fullsized":"2024-01-15T18:16:29.725995+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'36","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559183+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:19:45.559183+0000","last_peered":"2024-01-15T18:19:45.559183+0000","last_clean":"2024-01-15T18:19:45.559183+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:19:45.559183+0000","last_undegraded":"2024-01-15T18:19:45.559183+0000","last_fullsized":"2024-01-15T18:19:45.559183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.700861+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:16:29.700861+0000","last_peered":"2024-01-15T18:16:29.700861+0000","last_clean":"2024-01-15T18:16:29.700861+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:16:29.700861+0000","last_undegraded":"2024-01-15T18:16:29.700861+0000","last_fullsized":"2024-01-15T18:16:29.700861+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'41","reported_seq":81,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410772+0000","last_change":"2024-01-15T18:15:20.356060+0000","last_active":"2024-01-15T18:19:45.410772+0000","last_peered":"2024-01-15T18:19:45.410772+0000","last_clean":"2024-01-15T18:19:45.410772+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:19:45.410772+0000","last_undegraded":"2024-01-15T18:19:45.410772+0000","last_fullsized":"2024-01-15T18:19:45.410772+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:10.891015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'18","reported_seq":34,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:51.467225+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:18:51.467225+0000","last_peered":"2024-01-15T18:18:51.467225+0000","last_clean":"2024-01-15T18:18:51.467225+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:18:51.467225+0000","last_undegraded":"2024-01-15T18:18:51.467225+0000","last_fullsized":"2024-01-15T18:18:51.467225+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'46","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.536354+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:19:45.536354+0000","last_peered":"2024-01-15T18:19:45.536354+0000","last_clean":"2024-01-15T18:19:45.536354+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:45.536354+0000","last_undegraded":"2024-01-15T18:19:45.536354+0000","last_fullsized":"2024-01-15T18:19:45.536354+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.724998+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:16:24.724998+0000","last_peered":"2024-01-15T18:16:24.724998+0000","last_clean":"2024-01-15T18:16:24.724998+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:16:24.724998+0000","last_undegraded":"2024-01-15T18:16:24.724998+0000","last_fullsized":"2024-01-15T18:16:24.724998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'39","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404370+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:19:45.404370+0000","last_peered":"2024-01-15T18:19:45.404370+0000","last_clean":"2024-01-15T18:19:45.404370+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:19:45.404370+0000","last_undegraded":"2024-01-15T18:19:45.404370+0000","last_fullsized":"2024-01-15T18:19:45.404370+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.699700+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:16:24.699700+0000","last_peered":"2024-01-15T18:16:24.699700+0000","last_clean":"2024-01-15T18:16:24.699700+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:16:24.699700+0000","last_undegraded":"2024-01-15T18:16:24.699700+0000","last_fullsized":"2024-01-15T18:16:24.699700+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'44","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.695729+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:19:45.695729+0000","last_peered":"2024-01-15T18:19:45.695729+0000","last_clean":"2024-01-15T18:19:45.695729+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:19:45.695729+0000","last_undegraded":"2024-01-15T18:19:45.695729+0000","last_fullsized":"2024-01-15T18:19:45.695729+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.725579+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:16:24.725579+0000","last_peered":"2024-01-15T18:16:24.725579+0000","last_clean":"2024-01-15T18:16:24.725579+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:16:24.725579+0000","last_undegraded":"2024-01-15T18:16:24.725579+0000","last_fullsized":"2024-01-15T18:16:24.725579+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'56","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.774102+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:19:45.774102+0000","last_peered":"2024-01-15T18:19:45.774102+0000","last_clean":"2024-01-15T18:19:45.774102+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:19:45.774102+0000","last_undegraded":"2024-01-15T18:19:45.774102+0000","last_fullsized":"2024-01-15T18:19:45.774102+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.814331+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:16:24.814331+0000","last_peered":"2024-01-15T18:16:24.814331+0000","last_clean":"2024-01-15T18:16:24.814331+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:16:24.814331+0000","last_undegraded":"2024-01-15T18:16:24.814331+0000","last_fullsized":"2024-01-15T18:16:24.814331+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741049+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:19:45.741049+0000","last_peered":"2024-01-15T18:19:45.741049+0000","last_clean":"2024-01-15T18:19:45.741049+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:19:45.741049+0000","last_undegraded":"2024-01-15T18:19:45.741049+0000","last_fullsized":"2024-01-15T18:19:45.741049+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'11","reported_seq":37,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:40.222936+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:17:40.222936+0000","last_peered":"2024-01-15T18:17:40.222936+0000","last_clean":"2024-01-15T18:17:40.222936+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:17:40.222936+0000","last_undegraded":"2024-01-15T18:17:40.222936+0000","last_fullsized":"2024-01-15T18:17:40.222936+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'39","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.718813+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:19:45.718813+0000","last_peered":"2024-01-15T18:19:45.718813+0000","last_clean":"2024-01-15T18:19:45.718813+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:19:45.718813+0000","last_undegraded":"2024-01-15T18:19:45.718813+0000","last_fullsized":"2024-01-15T18:19:45.718813+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:19.723501+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:16:19.723501+0000","last_peered":"2024-01-15T18:16:19.723501+0000","last_clean":"2024-01-15T18:16:19.723501+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:16:19.723501+0000","last_undegraded":"2024-01-15T18:16:19.723501+0000","last_fullsized":"2024-01-15T18:16:19.723501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'31","reported_seq":54,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.579344+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:19:45.579344+0000","last_peered":"2024-01-15T18:19:45.579344+0000","last_clean":"2024-01-15T18:19:45.579344+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:19:45.579344+0000","last_undegraded":"2024-01-15T18:19:45.579344+0000","last_fullsized":"2024-01-15T18:19:45.579344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.728790+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:16:39.728790+0000","last_peered":"2024-01-15T18:16:39.728790+0000","last_clean":"2024-01-15T18:16:39.728790+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:16:39.728790+0000","last_undegraded":"2024-01-15T18:16:39.728790+0000","last_fullsized":"2024-01-15T18:16:39.728790+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":254,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:30.003748+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:17:30.003748+0000","last_peered":"2024-01-15T18:17:30.003748+0000","last_clean":"2024-01-15T18:17:30.003748+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:17:30.003748+0000","last_undegraded":"2024-01-15T18:17:30.003748+0000","last_fullsized":"2024-01-15T18:17:30.003748+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":633858,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":281,"num_write_kb":746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1335296,"data_stored":1267716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.296,"15min":1.325},"min":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.54000000000000004},"max":{"1min":1.782,"5min":2.1200000000000001,"15min":9.1379999999999999},"last":1.534},{"interface":"front","average":{"1min":1.268,"5min":1.385,"15min":1.528},"min":{"1min":0.88400000000000001,"5min":0.81999999999999995,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":1.905,"15min":27.32},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2210000000000001,"15min":1.4530000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.728,"5min":2.1000000000000001,"15min":29.167999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.24,"15min":1.3260000000000001},"min":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.52800000000000002},"max":{"1min":1.597,"5min":2.6179999999999999,"15min":8.5980000000000008},"last":1.1759999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3740000000000001,"15min":1.4179999999999999},"min":{"1min":0.94699999999999995,"5min":0.874,"15min":0.56000000000000005},"max":{"1min":1.7709999999999999,"5min":2.3900000000000001,"15min":12.709},"last":1.393},{"interface":"front","average":{"1min":1.141,"5min":1.256,"15min":1.488},"min":{"1min":0.79400000000000004,"5min":0.72499999999999998,"15min":0.56599999999999995},"max":{"1min":1.6539999999999999,"5min":1.8200000000000001,"15min":27.358000000000001},"last":0.629}]},{"osd":3,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.556},"min":{"1min":0.98599999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.643,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.4530000000000001,"15min":1.5189999999999999},"min":{"1min":1.01,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.7769999999999999,"5min":2.4940000000000002,"15min":16.088999999999999},"last":1.4850000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.381,"15min":1.5269999999999999},"min":{"1min":0.94899999999999995,"5min":0.64700000000000002,"15min":0.50700000000000001},"max":{"1min":1.681,"5min":6.0350000000000001,"15min":27.219000000000001},"last":1.23},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4339999999999999,"15min":1.641},"min":{"1min":0.90400000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7629999999999999,"5min":3.536,"15min":29.25},"last":1.2849999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4730000000000001,"15min":1.573},"min":{"1min":1.0209999999999999,"5min":0.68799999999999994,"15min":0.67000000000000004},"max":{"1min":1.8660000000000001,"5min":2.7120000000000002,"15min":11.834},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.5129999999999999,"15min":1.675},"min":{"1min":0.92900000000000005,"5min":0.71199999999999997,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.343}]},{"osd":6,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.635,"15min":1.8200000000000001},"min":{"1min":1.1850000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.929,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5409999999999999,"15min":1.8049999999999999},"min":{"1min":1.0289999999999999,"5min":0.63100000000000001,"15min":0.57299999999999995},"max":{"1min":1.841,"5min":2.754,"15min":29.106999999999999},"last":1.6719999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542415,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.2190000000000001,"15min":1.236},"min":{"1min":0.72299999999999998,"5min":0.68200000000000005,"15min":0.438},"max":{"1min":1.5409999999999999,"5min":2.77,"15min":8.8350000000000009},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1970000000000001,"15min":1.27},"min":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.59599999999999997},"max":{"1min":1.643,"5min":2.673,"15min":9.548},"last":1.0609999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2729999999999999,"15min":1.395},"min":{"1min":0.89300000000000002,"5min":0.70999999999999996,"15min":0.52100000000000002},"max":{"1min":1.5209999999999999,"5min":2.8700000000000001,"15min":13.269},"last":1.45},{"interface":"front","average":{"1min":1.262,"5min":1.3009999999999999,"15min":1.4119999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.57799999999999996},"max":{"1min":1.6990000000000001,"5min":3.1080000000000001,"15min":13.984999999999999},"last":0.92700000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.232,"15min":1.284},"min":{"1min":0.85299999999999998,"5min":0.79000000000000004,"15min":0.69799999999999995},"max":{"1min":1.7,"5min":1.9379999999999999,"15min":7.0030000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.373,"15min":1.425},"min":{"1min":0.81100000000000005,"5min":0.73899999999999999,"15min":0.49399999999999999},"max":{"1min":1.623,"5min":2.6909999999999998,"15min":8.9359999999999999},"last":1.2749999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.444,"15min":1.55},"min":{"1min":1.131,"5min":1.0089999999999999,"15min":0.68500000000000005},"max":{"1min":1.7629999999999999,"5min":3.339,"15min":16.855},"last":1.363},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.413,"15min":1.5209999999999999},"min":{"1min":1.111,"5min":1.046,"15min":0.628},"max":{"1min":1.657,"5min":2.0139999999999998,"15min":10.875},"last":1.1140000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.401,"15min":1.45},"min":{"1min":0.86699999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":1.645,"5min":3.0299999999999998,"15min":8.7539999999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.427,"15min":1.4970000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.819,"5min":5.125,"15min":10.118},"last":1.242}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3600000000000001,"15min":1.5209999999999999},"min":{"1min":0.92500000000000004,"5min":0.88,"15min":0.60299999999999998},"max":{"1min":1.8120000000000001,"5min":2.3210000000000002,"15min":16.309999999999999},"last":1.016},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5429999999999999,"15min":1.6200000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.66000000000000003},"max":{"1min":1.8029999999999999,"5min":3.0209999999999999,"15min":10.077},"last":1.3899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5289999999999999,"15min":1.6279999999999999},"min":{"1min":0.995,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":1.881,"5min":3.2450000000000001,"15min":10.302},"last":1.431},{"interface":"front","average":{"1min":1.504,"5min":1.663,"15min":1.6879999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":1.9059999999999999,"5min":7.6799999999999997,"15min":9.4079999999999995},"last":1.2989999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542417,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928972,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93442868,"statfs":{"total":96636764160,"available":95685496832,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2609999999999999,"15min":1.399},"min":{"1min":0.79300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.6180000000000001,"5min":4.1829999999999998,"15min":26.077999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3740000000000001,"15min":1.4299999999999999},"min":{"1min":1.099,"5min":0.77900000000000003,"15min":0.56399999999999995},"max":{"1min":1.6200000000000001,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.2789999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2549999999999999,"15min":1.4339999999999999},"min":{"1min":0.875,"5min":0.82499999999999996,"15min":0.53300000000000003},"max":{"1min":1.8,"5min":1.923,"15min":26},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.367,"15min":1.401},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.657,"5min":4.0869999999999997,"15min":12.619},"last":1.2050000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3460000000000001,"15min":1.5329999999999999},"min":{"1min":1.0700000000000001,"5min":0.69999999999999996,"15min":0.56699999999999995},"max":{"1min":1.712,"5min":2.6869999999999998,"15min":26.234999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.385,"5min":1.331,"15min":1.532},"min":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.629},"max":{"1min":1.766,"5min":1.8240000000000001,"15min":17.126999999999999},"last":1.105}]},{"osd":3,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3120000000000001,"15min":1.397},"min":{"1min":1.0429999999999999,"5min":0.82399999999999995,"15min":0.67200000000000004},"max":{"1min":1.792,"5min":2.375,"15min":12.722},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4830000000000001,"15min":1.6399999999999999},"min":{"1min":1.046,"5min":0.75900000000000001,"15min":0.67900000000000005},"max":{"1min":1.7989999999999999,"5min":3.238,"15min":26.134},"last":1.0449999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2769999999999999,"15min":1.4139999999999999},"min":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.442},"max":{"1min":1.77,"5min":2.5350000000000001,"15min":9.9979999999999993},"last":1.371},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.458,"15min":1.6930000000000001},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.498},"max":{"1min":1.875,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.4730000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3640000000000001,"15min":1.4339999999999999},"min":{"1min":1.0069999999999999,"5min":0.82299999999999995,"15min":0.45900000000000002},"max":{"1min":1.849,"5min":2.6640000000000001,"15min":12.769},"last":1.52},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5860000000000001,"15min":1.647},"min":{"1min":0.94799999999999995,"5min":0.79800000000000004,"15min":0.49199999999999999},"max":{"1min":1.909,"5min":11.851000000000001,"15min":26.186},"last":1.153}]},{"osd":7,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5960000000000001,"15min":1.798},"min":{"1min":1.1599999999999999,"5min":1.1359999999999999,"15min":0.65700000000000003},"max":{"1min":2.04,"5min":2.3700000000000001,"15min":13.706},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.54,"15min":1.679},"min":{"1min":1.038,"5min":1.0189999999999999,"15min":0.60099999999999998},"max":{"1min":2.0289999999999999,"5min":4.3700000000000001,"15min":21.728000000000002},"last":1.399}]}]},{"osd":4,"up_from":14,"seq":60129542416,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.343,"15min":1.27},"min":{"1min":0.92800000000000005,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.7490000000000001,"5min":13.288,"15min":15.044},"last":1.004},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.284,"15min":1.321},"min":{"1min":0.91600000000000004,"5min":0.71299999999999997,"15min":0.34699999999999998},"max":{"1min":1.5389999999999999,"5min":9.2409999999999997,"15min":11.196},"last":1.393}]},{"osd":1,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2569999999999999,"15min":1.3779999999999999},"min":{"1min":0.90400000000000003,"5min":0.78400000000000003,"15min":0.41499999999999998},"max":{"1min":1.655,"5min":3.5910000000000002,"15min":28.663},"last":1.341},{"interface":"front","average":{"1min":1.278,"5min":1.302,"15min":1.4370000000000001},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.6319999999999999,"5min":3.6739999999999999,"15min":15.096},"last":1.4590000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.333,"15min":1.4319999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.8170000000000002,"15min":16.023},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.381,"15min":1.5169999999999999},"min":{"1min":0.94899999999999995,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.7869999999999999,"5min":3.7930000000000001,"15min":18.858000000000001},"last":1.244}]},{"osd":3,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3029999999999999,"15min":1.393},"min":{"1min":0.80100000000000005,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.847,"5min":5.0599999999999996,"15min":11.449},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.4510000000000001,"15min":1.5720000000000001},"min":{"1min":0.995,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.7969999999999999,"5min":3.7080000000000002,"15min":18.027000000000001},"last":1.4970000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4390000000000001,"15min":1.6220000000000001},"min":{"1min":0.88900000000000001,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.8140000000000001,"5min":7.7439999999999998,"15min":54.938000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.339,"15min":1.482},"min":{"1min":0.96499999999999997,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":1.857,"5min":2.0960000000000001,"15min":13.031000000000001},"last":1.171}]},{"osd":6,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.4119999999999999,"15min":1.5509999999999999},"min":{"1min":1.1180000000000001,"5min":0.90300000000000002,"15min":0.495},"max":{"1min":1.8779999999999999,"5min":3.766,"15min":16.103999999999999},"last":1.429},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5920000000000001,"15min":1.6799999999999999},"min":{"1min":1.1559999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.909,"5min":8.1609999999999996,"15min":15.118},"last":1.5940000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.45,"15min":1.786},"min":{"1min":1.1919999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.0299999999999998,"5min":2.4689999999999999,"15min":23.390000000000001},"last":1.532},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5609999999999999,"15min":1.756},"min":{"1min":1.2270000000000001,"5min":1.0409999999999999,"15min":0.62},"max":{"1min":1.99,"5min":3.9020000000000001,"15min":16.763000000000002},"last":1.5620000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542417,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462020,"kb_used_data":1386492,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909820,"statfs":{"total":96636764160,"available":95139655680,"internally_reserved":0,"allocated":1419767808,"data_stored":1418939088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95199999999999996,"15min":1.1020000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":1.579,"5min":1.6599999999999999,"15min":36.408000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.101,"15min":1.093},"min":{"1min":0.69799999999999995,"5min":0.63900000000000001,"15min":0.377},"max":{"1min":1.6930000000000001,"5min":2.0230000000000001,"15min":5.0190000000000001},"last":1.171}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.99099999999999999,"15min":1.0640000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.34399999999999997},"max":{"1min":1.756,"5min":1.837,"15min":5.226},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.113,"5min":1.1899999999999999,"15min":1.22},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.669,"5min":2.0609999999999999,"15min":7.2169999999999996},"last":0.749}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2569999999999999,"15min":1.3500000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.637,"5min":2.0150000000000001,"15min":14.025},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.345,"15min":1.3400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.841,"5min":2.403,"15min":3.6240000000000001},"last":0.96199999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.4259999999999999},"min":{"1min":0.84799999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"max":{"1min":1.9350000000000001,"5min":3.1459999999999999,"15min":9.0069999999999997},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.448,"15min":1.6040000000000001},"min":{"1min":1.091,"5min":0.79100000000000004,"15min":0.69499999999999995},"max":{"1min":1.696,"5min":4.8570000000000002,"15min":15.769},"last":1.3360000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.468,"15min":1.579},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65000000000000002},"max":{"1min":1.7490000000000001,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.546},{"interface":"front","average":{"1min":1.468,"5min":1.5169999999999999,"15min":1.7809999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":2.1219999999999999,"5min":2.3959999999999999,"15min":38.832999999999998},"last":1.595}]},{"osd":6,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5469999999999999,"15min":1.7529999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.51400000000000001},"max":{"1min":1.804,"5min":7.1020000000000003,"15min":32.259999999999998},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.5740000000000001,"15min":1.7470000000000001},"min":{"1min":1.0389999999999999,"5min":0.85999999999999999,"15min":0.747},"max":{"1min":2.169,"5min":2.419,"15min":16.195},"last":1.448}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.6719999999999999,"15min":1.905},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":1.466},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.728,"15min":1.8009999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":1.976,"5min":16.734000000000002,"15min":17.622},"last":1.6439999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542416,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.012,"15min":1.0249999999999999},"min":{"1min":0.63500000000000001,"5min":0.44400000000000001,"15min":0.35199999999999998},"max":{"1min":1.5069999999999999,"5min":3.1589999999999998,"15min":10.256},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2150000000000001,"15min":1.1819999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.33700000000000002},"max":{"1min":1.8180000000000001,"5min":8.9719999999999995,"15min":10.319000000000001},"last":0.85799999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1,"15min":1.026},"min":{"1min":0.67900000000000005,"5min":0.48899999999999999,"15min":0.38600000000000001},"max":{"1min":1.573,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":1.157},{"interface":"front","average":{"1min":1.097,"5min":1.1890000000000001,"15min":1.2509999999999999},"min":{"1min":0.74099999999999999,"5min":0.59499999999999997,"15min":0.35299999999999998},"max":{"1min":1.5669999999999999,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.3149999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3240000000000001,"15min":1.4219999999999999},"min":{"1min":0.82899999999999996,"5min":0.54200000000000004,"15min":0.44400000000000001},"max":{"1min":1.5529999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.224,"15min":1.306},"min":{"1min":0.89000000000000001,"5min":0.56599999999999995,"15min":0.376},"max":{"1min":1.7689999999999999,"5min":2.7589999999999999,"15min":14.946},"last":1.177}]},{"osd":4,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.536,"15min":1.524},"min":{"1min":1.139,"5min":0.82099999999999995,"15min":0.66100000000000003},"max":{"1min":1.7769999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4990000000000001,"15min":1.5960000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.65300000000000002},"max":{"1min":1.768,"5min":2.867,"15min":10.279999999999999},"last":1.4059999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4770000000000001,"15min":1.5700000000000001},"min":{"1min":1,"5min":1,"15min":0.63500000000000001},"max":{"1min":1.6830000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.111},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.613,"15min":1.6200000000000001},"min":{"1min":1.0209999999999999,"5min":0.88200000000000001,"15min":0.71699999999999997},"max":{"1min":1.895,"5min":10.833,"15min":10.833},"last":1.5229999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.6180000000000001},"min":{"1min":1.0109999999999999,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.827,"5min":2.984,"15min":10.185},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.657,"15min":1.7250000000000001},"min":{"1min":1.107,"5min":1.1060000000000001,"15min":0.79700000000000004},"max":{"1min":1.843,"5min":10.917,"15min":22.382000000000001},"last":1.2030000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.6180000000000001,"15min":1.843},"min":{"1min":1.129,"5min":1.006,"15min":0.77700000000000002},"max":{"1min":1.897,"5min":3.052,"15min":19.986000000000001},"last":1.583},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6659999999999999,"15min":1.77},"min":{"1min":1.252,"5min":0.89300000000000002,"15min":0.877},"max":{"1min":1.931,"5min":3.069,"15min":11.893000000000001},"last":1.29}]}]},{"osd":1,"up_from":14,"seq":60129542416,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1137752,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93234088,"statfs":{"total":96636764160,"available":95471706112,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96599999999999997,"15min":1.1020000000000001},"min":{"1min":0.63300000000000001,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":1.7130000000000001,"5min":3.4510000000000001,"15min":19.218},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.232},"min":{"1min":0.65500000000000003,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.694,"5min":2.0939999999999999,"15min":19.823},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99099999999999999,"15min":1.1100000000000001},"min":{"1min":0.80000000000000004,"5min":0.54400000000000004,"15min":0.31900000000000001},"max":{"1min":1.776,"5min":2.5110000000000001,"15min":19.02},"last":1.603},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.258,"15min":1.401},"min":{"1min":0.85699999999999998,"5min":0.63300000000000001,"15min":0.41599999999999998},"max":{"1min":1.835,"5min":3.1400000000000001,"15min":29.251000000000001},"last":1.4279999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.4990000000000001},"min":{"1min":0.84899999999999998,"5min":0.58499999999999996,"15min":0.40899999999999997},"max":{"1min":1.6779999999999999,"5min":5.1459999999999999,"15min":27.021000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.194,"15min":1.391},"min":{"1min":0.77600000000000002,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.7669999999999999,"5min":3.0150000000000001,"15min":31.218},"last":1.5229999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.492,"15min":1.6890000000000001},"min":{"1min":1.1379999999999999,"5min":0.755,"15min":0.70599999999999996},"max":{"1min":1.8049999999999999,"5min":5.2169999999999996,"15min":27.059999999999999},"last":0.998},{"interface":"front","average":{"1min":1.409,"5min":1.4450000000000001,"15min":1.6359999999999999},"min":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.56499999999999995},"max":{"1min":1.9650000000000001,"5min":3.4159999999999999,"15min":31.433},"last":1.097}]},{"osd":5,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4730000000000001,"15min":1.645},"min":{"1min":1.149,"5min":0.82499999999999996,"15min":0.68500000000000005},"max":{"1min":1.8560000000000001,"5min":3.0950000000000002,"15min":20.012},"last":1.224},{"interface":"front","average":{"1min":1.339,"5min":1.431,"15min":1.52},"min":{"1min":0.97699999999999998,"5min":0.85799999999999998,"15min":0.69299999999999995},"max":{"1min":1.631,"5min":2.9409999999999998,"15min":6.758},"last":1.45}]},{"osd":6,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.639,"15min":1.794},"min":{"1min":1.28,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.9339999999999999,"5min":5.0629999999999997,"15min":29.530000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.575,"15min":1.802},"min":{"1min":1.2030000000000001,"5min":0.88400000000000001,"15min":0.61399999999999999},"max":{"1min":1.845,"5min":3.1960000000000002,"15min":31.186},"last":1.397}]},{"osd":7,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5169999999999999,"15min":1.8140000000000001},"min":{"1min":1.0860000000000001,"5min":0.95899999999999996,"15min":0.63},"max":{"1min":1.8460000000000001,"5min":3.3769999999999998,"15min":26.393999999999998},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.6240000000000001,"15min":1.8380000000000001},"min":{"1min":1.1659999999999999,"5min":0.89200000000000002,"15min":0.76100000000000001},"max":{"1min":1.913,"5min":3.5750000000000002,"15min":27.582000000000001},"last":1.7909999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1777940,"kb_used_data":1706572,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92593900,"statfs":{"total":96636764160,"available":94816153600,"internally_reserved":0,"allocated":1747529728,"data_stored":1746685443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97699999999999998,"15min":1.141},"min":{"1min":0.57199999999999995,"5min":0.42299999999999999,"15min":0.30599999999999999},"max":{"1min":1.5860000000000001,"5min":5.7560000000000002,"15min":20.437999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.236,"15min":1.278},"min":{"1min":0.77000000000000002,"5min":0.65300000000000002,"15min":0.32400000000000001},"max":{"1min":1.8660000000000001,"5min":6.3600000000000003,"15min":35.137999999999998},"last":1.052}]},{"osd":2,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.236,"15min":1.3879999999999999},"min":{"1min":0.66100000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.716,"5min":11.06,"15min":20.556000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2789999999999999,"15min":1.375},"min":{"1min":0.85099999999999998,"5min":0.60699999999999998,"15min":0.39000000000000001},"max":{"1min":1.8919999999999999,"5min":6.4939999999999998,"15min":35.183},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.238,"15min":1.1819999999999999},"min":{"1min":0.621,"5min":0.55000000000000004,"15min":0.42299999999999999},"max":{"1min":2.0230000000000001,"5min":10.885999999999999,"15min":10.885999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.266,"5min":1.3779999999999999,"15min":1.635},"min":{"1min":0.90000000000000002,"5min":0.73099999999999998,"15min":0.45100000000000001},"max":{"1min":1.621,"5min":9.3339999999999996,"15min":25.372},"last":0.71399999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.532,"15min":1.621},"min":{"1min":1.0509999999999999,"5min":1.014,"15min":0.67200000000000004},"max":{"1min":1.9710000000000001,"5min":9.9969999999999999,"15min":35.223999999999997},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.581,"15min":1.49},"min":{"1min":1.016,"5min":0.99399999999999999,"15min":0.58799999999999997},"max":{"1min":1.96,"5min":17.567,"15min":17.567},"last":1.5429999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6899999999999999,"15min":1.6499999999999999},"min":{"1min":1.099,"5min":0.92700000000000005,"15min":0.69799999999999995},"max":{"1min":1.9990000000000001,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.679,"15min":1.8300000000000001},"min":{"1min":1.0029999999999999,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.966,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.6419999999999999,"15min":1.923},"min":{"1min":1.1879999999999999,"5min":1.004,"15min":0.70599999999999996},"max":{"1min":1.875,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.244},{"interface":"front","average":{"1min":1.595,"5min":1.6619999999999999,"15min":1.8440000000000001},"min":{"1min":1.2130000000000001,"5min":0.90600000000000003,"15min":0.62},"max":{"1min":2.2149999999999999,"5min":6.2270000000000003,"15min":35.255000000000003},"last":1.512}]},{"osd":7,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.704,"15min":1.9319999999999999},"min":{"1min":1.153,"5min":1.089,"15min":0.56200000000000006},"max":{"1min":2.2919999999999998,"5min":9.9429999999999996,"15min":35.636000000000003},"last":1.591},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.6639999999999999,"15min":1.6919999999999999},"min":{"1min":1.1759999999999999,"5min":0.96499999999999997,"15min":0.64000000000000001},"max":{"1min":2.056,"5min":10.943,"15min":11.470000000000001},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":631851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":631809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:19:52.573 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:19:52.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.828+0000 7f618a600640 1 Processor -- start 2024-01-15T18:19:52.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.828+0000 7f618a600640 1 -- start start 2024-01-15T18:19:52.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.830+0000 7f618a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618413ce30 0x7f618415dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.830+0000 7f618a600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f618415e170 0x7f61840cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.830+0000 7f618a600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f61840cdfa0 0x7f618413f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.830+0000 7f618a600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f618415e5f0 con 0x7f618415e170 2024-01-15T18:19:52.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.830+0000 7f618a600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61840aeab0 con 0x7f61840cdfa0 2024-01-15T18:19:52.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.830+0000 7f618a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6184070380 con 0x7f618413ce30 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f618415e170 0x7f61840cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f618415e170 0x7f61840cda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41764/0 (socket says 172.21.15.2:41764) 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 -- 172.21.15.2:0/1493105236 learned_addr learned my addr 172.21.15.2:0/1493105236 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f6183400640 1 --2- 172.21.15.2:0/1493105236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618413ce30 0x7f618415dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f6183e00640 1 --2- 172.21.15.2:0/1493105236 >> v2:172.21.15.137:3300/0 conn(0x7f61840cdfa0 0x7f618413f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 -- 172.21.15.2:0/1493105236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618413ce30 msgr2=0x7f618415dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 --2- 172.21.15.2:0/1493105236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618413ce30 0x7f618415dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 -- 172.21.15.2:0/1493105236 >> v2:172.21.15.137:3300/0 conn(0x7f61840cdfa0 msgr2=0x7f618413f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:52.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 --2- 172.21.15.2:0/1493105236 >> v2:172.21.15.137:3300/0 conn(0x7f61840cdfa0 0x7f618413f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.831+0000 7f617a800640 1 -- 172.21.15.2:0/1493105236 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61840ce410 con 0x7f618415e170 2024-01-15T18:19:52.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.833+0000 7f617a800640 1 --2- 172.21.15.2:0/1493105236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618415e170 0x7f61840cda60 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6184143d80 tx=0x7f616c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f235bb278273777 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:52.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.833+0000 7f6182a00640 1 -- 172.21.15.2:0/1493105236 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c031870 con 0x7f618415e170 2024-01-15T18:19:52.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.833+0000 7f6182a00640 1 -- 172.21.15.2:0/1493105236 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c0319d0 con 0x7f618415e170 2024-01-15T18:19:52.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.833+0000 7f6182a00640 1 -- 172.21.15.2:0/1493105236 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c031d60 con 0x7f618415e170 2024-01-15T18:19:52.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 -- 172.21.15.2:0/1493105236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618415e170 msgr2=0x7f61840cda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 --2- 172.21.15.2:0/1493105236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618415e170 0x7f61840cda60 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f6184143d80 tx=0x7f616c02b910 comp rx=0 tx=0).stop 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 -- 172.21.15.2:0/1493105236 shutdown_connections 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 --2- 172.21.15.2:0/1493105236 >> v2:172.21.15.137:3300/0 conn(0x7f61840cdfa0 0x7f618413f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 --2- 172.21.15.2:0/1493105236 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618415e170 0x7f61840cda60 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 --2- 172.21.15.2:0/1493105236 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618413ce30 0x7f618415dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 -- 172.21.15.2:0/1493105236 >> 172.21.15.2:0/1493105236 conn(0x7f618406da20 msgr2=0x7f618406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 -- 172.21.15.2:0/1493105236 shutdown_connections 2024-01-15T18:19:52.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.834+0000 7f618a600640 1 -- 172.21.15.2:0/1493105236 wait complete. 2024-01-15T18:19:52.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.836+0000 7f618a600640 1 Processor -- start 2024-01-15T18:19:52.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.836+0000 7f618a600640 1 -- start start 2024-01-15T18:19:52.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f618a600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61840cdfa0 0x7f61841f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f618a600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 0x7f61841f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f618a600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618415e170 0x7f61840b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f618a600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61841f3e40 con 0x7f618413ce30 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f618a600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61841f0830 con 0x7f61840cdfa0 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f618a600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61840b1120 con 0x7f618415e170 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f617a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 0x7f61841f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f617a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 0x7f61841f02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:41774/0 (socket says 172.21.15.2:41774) 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.837+0000 7f617a800640 1 -- 172.21.15.2:0/3053061579 learned_addr learned my addr 172.21.15.2:0/3053061579 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617a800640 1 -- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618415e170 msgr2=0x7f61840b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617a800640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618415e170 0x7f61840b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617a800640 1 -- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61840cdfa0 msgr2=0x7f61841f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:52.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617a800640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61840cdfa0 0x7f61841f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:52.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617a800640 1 -- 172.21.15.2:0/3053061579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0094d0 con 0x7f618413ce30 2024-01-15T18:19:52.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617a800640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 0x7f61841f02f0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f616c02b950 tx=0x7f616c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:52.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c03bcc0 con 0x7f618413ce30 2024-01-15T18:19:52.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616c03be40 con 0x7f618413ce30 2024-01-15T18:19:52.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c045630 con 0x7f618413ce30 2024-01-15T18:19:52.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.838+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61840b13f0 con 0x7f618413ce30 2024-01-15T18:19:52.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.839+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61840b19b0 con 0x7f618413ce30 2024-01-15T18:19:52.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.840+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f618413e260 con 0x7f618413ce30 2024-01-15T18:19:52.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.841+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f616c04e450 con 0x7f618413ce30 2024-01-15T18:19:52.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.842+0000 7f617be00640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f615c06c550 0x7f615c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:52.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.842+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f616c02d040 con 0x7f618413ce30 2024-01-15T18:19:52.846 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.843+0000 7f6183400640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f615c06c550 0x7f615c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:52.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.844+0000 7f6183400640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f615c06c550 0x7f615c06ea10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f61740070b0 tx=0x7f6174007040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:52.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:52.847+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f618413e260 con 0x7f618413ce30 2024-01-15T18:19:53.046 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.043+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61840cec90 con 0x7f615c06c550 2024-01-15T18:19:53.074 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.071+0000 7f617be00640 1 -- 172.21.15.2:0/3053061579 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288946 (secure 0 0 0) 0x7f6174009bb0 con 0x7f615c06c550 2024-01-15T18:19:53.078 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:53.080 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:19:53.081 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f615c06c550 msgr2=0x7f615c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f615c06c550 0x7f615c06ea10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f61740070b0 tx=0x7f6174007040 comp rx=0 tx=0).stop 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 msgr2=0x7f61841f02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 0x7f61841f02f0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f616c02b950 tx=0x7f616c0046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 shutdown_connections 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f615c06c550 0x7f615c06ea10 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f618415e170 0x7f61840b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f618413ce30 0x7f61841f02f0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 --2- 172.21.15.2:0/3053061579 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61840cdfa0 0x7f61841f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 >> 172.21.15.2:0/3053061579 conn(0x7f618406da20 msgr2=0x7f618406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:53.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 shutdown_connections 2024-01-15T18:19:53.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.077+0000 7f618a600640 1 -- 172.21.15.2:0/3053061579 wait complete. 2024-01-15T18:19:53.099 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2024-01-15T18:19:51.957916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369934362,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1746,"num_write_kb":5246057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3187,"ondisk_log_size":3187,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11070440,"kb_used_data":10499496,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":743904280,"statfs":{"total":773094113280,"available":761757982720,"internally_reserved":0,"allocated":10751483904,"data_stored":10744924764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":584568369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073776448,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":274,"num_write_kb":1048616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004677"},"pg_stats":[{"pgid":"17.3f","version":"116'53","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.408247+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:19:45.408247+0000","last_peered":"2024-01-15T18:19:45.408247+0000","last_clean":"2024-01-15T18:19:45.408247+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:19:45.408247+0000","last_undegraded":"2024-01-15T18:19:45.408247+0000","last_fullsized":"2024-01-15T18:19:45.408247+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.516826+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:19:45.516826+0000","last_peered":"2024-01-15T18:19:45.516826+0000","last_clean":"2024-01-15T18:19:45.516826+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:19:45.516826+0000","last_undegraded":"2024-01-15T18:19:45.516826+0000","last_fullsized":"2024-01-15T18:19:45.516826+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.675239+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:19:45.675239+0000","last_peered":"2024-01-15T18:19:45.675239+0000","last_clean":"2024-01-15T18:19:45.675239+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:19:45.675239+0000","last_undegraded":"2024-01-15T18:19:45.675239+0000","last_fullsized":"2024-01-15T18:19:45.675239+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'45","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.578232+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:19:45.578232+0000","last_peered":"2024-01-15T18:19:45.578232+0000","last_clean":"2024-01-15T18:19:45.578232+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:19:45.578232+0000","last_undegraded":"2024-01-15T18:19:45.578232+0000","last_fullsized":"2024-01-15T18:19:45.578232+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'27","reported_seq":49,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.370622+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:19:45.370622+0000","last_peered":"2024-01-15T18:19:45.370622+0000","last_clean":"2024-01-15T18:19:45.370622+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:19:45.370622+0000","last_undegraded":"2024-01-15T18:19:45.370622+0000","last_fullsized":"2024-01-15T18:19:45.370622+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'48","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410257+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:19:45.410257+0000","last_peered":"2024-01-15T18:19:45.410257+0000","last_clean":"2024-01-15T18:19:45.410257+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:19:45.410257+0000","last_undegraded":"2024-01-15T18:19:45.410257+0000","last_fullsized":"2024-01-15T18:19:45.410257+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'43","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.408637+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:19:45.408637+0000","last_peered":"2024-01-15T18:19:45.408637+0000","last_clean":"2024-01-15T18:19:45.408637+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:19:45.408637+0000","last_undegraded":"2024-01-15T18:19:45.408637+0000","last_fullsized":"2024-01-15T18:19:45.408637+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.398127+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:19:45.398127+0000","last_peered":"2024-01-15T18:19:45.398127+0000","last_clean":"2024-01-15T18:19:45.398127+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:19:45.398127+0000","last_undegraded":"2024-01-15T18:19:45.398127+0000","last_fullsized":"2024-01-15T18:19:45.398127+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.726565+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:19:45.726565+0000","last_peered":"2024-01-15T18:19:45.726565+0000","last_clean":"2024-01-15T18:19:45.726565+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:19:45.726565+0000","last_undegraded":"2024-01-15T18:19:45.726565+0000","last_fullsized":"2024-01-15T18:19:45.726565+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'47","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.827721+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:19:45.827721+0000","last_peered":"2024-01-15T18:19:45.827721+0000","last_clean":"2024-01-15T18:19:45.827721+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:19:45.827721+0000","last_undegraded":"2024-01-15T18:19:45.827721+0000","last_fullsized":"2024-01-15T18:19:45.827721+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'47","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.524665+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:19:45.524665+0000","last_peered":"2024-01-15T18:19:45.524665+0000","last_clean":"2024-01-15T18:19:45.524665+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:19:45.524665+0000","last_undegraded":"2024-01-15T18:19:45.524665+0000","last_fullsized":"2024-01-15T18:19:45.524665+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'48","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.634818+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:19:45.634818+0000","last_peered":"2024-01-15T18:19:45.634818+0000","last_clean":"2024-01-15T18:19:45.634818+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:19:45.634818+0000","last_undegraded":"2024-01-15T18:19:45.634818+0000","last_fullsized":"2024-01-15T18:19:45.634818+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'38","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.393963+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:19:45.393963+0000","last_peered":"2024-01-15T18:19:45.393963+0000","last_clean":"2024-01-15T18:19:45.393963+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:19:45.393963+0000","last_undegraded":"2024-01-15T18:19:45.393963+0000","last_fullsized":"2024-01-15T18:19:45.393963+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'51","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.775084+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:19:45.775084+0000","last_peered":"2024-01-15T18:19:45.775084+0000","last_clean":"2024-01-15T18:19:45.775084+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:19:45.775084+0000","last_undegraded":"2024-01-15T18:19:45.775084+0000","last_fullsized":"2024-01-15T18:19:45.775084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'34","reported_seq":53,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.681649+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:19:45.681649+0000","last_peered":"2024-01-15T18:19:45.681649+0000","last_clean":"2024-01-15T18:19:45.681649+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:19:45.681649+0000","last_undegraded":"2024-01-15T18:19:45.681649+0000","last_fullsized":"2024-01-15T18:19:45.681649+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'42","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.390578+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:19:45.390578+0000","last_peered":"2024-01-15T18:19:45.390578+0000","last_clean":"2024-01-15T18:19:45.390578+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:19:45.390578+0000","last_undegraded":"2024-01-15T18:19:45.390578+0000","last_fullsized":"2024-01-15T18:19:45.390578+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.458896+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:19:45.458896+0000","last_peered":"2024-01-15T18:19:45.458896+0000","last_clean":"2024-01-15T18:19:45.458896+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:19:45.458896+0000","last_undegraded":"2024-01-15T18:19:45.458896+0000","last_fullsized":"2024-01-15T18:19:45.458896+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'48","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.699878+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:19:45.699878+0000","last_peered":"2024-01-15T18:19:45.699878+0000","last_clean":"2024-01-15T18:19:45.699878+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:19:45.699878+0000","last_undegraded":"2024-01-15T18:19:45.699878+0000","last_fullsized":"2024-01-15T18:19:45.699878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.465695+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:19:45.465695+0000","last_peered":"2024-01-15T18:19:45.465695+0000","last_clean":"2024-01-15T18:19:45.465695+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:19:45.465695+0000","last_undegraded":"2024-01-15T18:19:45.465695+0000","last_fullsized":"2024-01-15T18:19:45.465695+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.700450+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:19:45.700450+0000","last_peered":"2024-01-15T18:19:45.700450+0000","last_clean":"2024-01-15T18:19:45.700450+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:19:45.700450+0000","last_undegraded":"2024-01-15T18:19:45.700450+0000","last_fullsized":"2024-01-15T18:19:45.700450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'56","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.427319+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:19:45.427319+0000","last_peered":"2024-01-15T18:19:45.427319+0000","last_clean":"2024-01-15T18:19:45.427319+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:19:45.427319+0000","last_undegraded":"2024-01-15T18:19:45.427319+0000","last_fullsized":"2024-01-15T18:19:45.427319+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.823341+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:19:45.823341+0000","last_peered":"2024-01-15T18:19:45.823341+0000","last_clean":"2024-01-15T18:19:45.823341+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:19:45.823341+0000","last_undegraded":"2024-01-15T18:19:45.823341+0000","last_fullsized":"2024-01-15T18:19:45.823341+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'39","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.605337+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:19:45.605337+0000","last_peered":"2024-01-15T18:19:45.605337+0000","last_clean":"2024-01-15T18:19:45.605337+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:19:45.605337+0000","last_undegraded":"2024-01-15T18:19:45.605337+0000","last_fullsized":"2024-01-15T18:19:45.605337+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'43","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.704152+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:19:45.704152+0000","last_peered":"2024-01-15T18:19:45.704152+0000","last_clean":"2024-01-15T18:19:45.704152+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:19:45.704152+0000","last_undegraded":"2024-01-15T18:19:45.704152+0000","last_fullsized":"2024-01-15T18:19:45.704152+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'38","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.411545+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:19:45.411545+0000","last_peered":"2024-01-15T18:19:45.411545+0000","last_clean":"2024-01-15T18:19:45.411545+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:19:45.411545+0000","last_undegraded":"2024-01-15T18:19:45.411545+0000","last_fullsized":"2024-01-15T18:19:45.411545+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'53","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.722921+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:19:45.722921+0000","last_peered":"2024-01-15T18:19:45.722921+0000","last_clean":"2024-01-15T18:19:45.722921+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:45.722921+0000","last_undegraded":"2024-01-15T18:19:45.722921+0000","last_fullsized":"2024-01-15T18:19:45.722921+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'37","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.383134+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:19:45.383134+0000","last_peered":"2024-01-15T18:19:45.383134+0000","last_clean":"2024-01-15T18:19:45.383134+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:19:45.383134+0000","last_undegraded":"2024-01-15T18:19:45.383134+0000","last_fullsized":"2024-01-15T18:19:45.383134+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'38","reported_seq":56,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.625653+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:19:45.625653+0000","last_peered":"2024-01-15T18:19:45.625653+0000","last_clean":"2024-01-15T18:19:45.625653+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:19:45.625653+0000","last_undegraded":"2024-01-15T18:19:45.625653+0000","last_fullsized":"2024-01-15T18:19:45.625653+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'46","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.705265+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:19:45.705265+0000","last_peered":"2024-01-15T18:19:45.705265+0000","last_clean":"2024-01-15T18:19:45.705265+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:19:45.705265+0000","last_undegraded":"2024-01-15T18:19:45.705265+0000","last_fullsized":"2024-01-15T18:19:45.705265+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'41","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.642104+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:19:45.642104+0000","last_peered":"2024-01-15T18:19:45.642104+0000","last_clean":"2024-01-15T18:19:45.642104+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:19:45.642104+0000","last_undegraded":"2024-01-15T18:19:45.642104+0000","last_fullsized":"2024-01-15T18:19:45.642104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'62","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.411970+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:19:45.411970+0000","last_peered":"2024-01-15T18:19:45.411970+0000","last_clean":"2024-01-15T18:19:45.411970+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:19:45.411970+0000","last_undegraded":"2024-01-15T18:19:45.411970+0000","last_fullsized":"2024-01-15T18:19:45.411970+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'54","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.560995+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:19:45.560995+0000","last_peered":"2024-01-15T18:19:45.560995+0000","last_clean":"2024-01-15T18:19:45.560995+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:19:45.560995+0000","last_undegraded":"2024-01-15T18:19:45.560995+0000","last_fullsized":"2024-01-15T18:19:45.560995+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'34","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.377219+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:19:45.377219+0000","last_peered":"2024-01-15T18:19:45.377219+0000","last_clean":"2024-01-15T18:19:45.377219+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:19:45.377219+0000","last_undegraded":"2024-01-15T18:19:45.377219+0000","last_fullsized":"2024-01-15T18:19:45.377219+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":20,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.725727+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:16:24.725727+0000","last_peered":"2024-01-15T18:16:24.725727+0000","last_clean":"2024-01-15T18:16:24.725727+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:16:24.725727+0000","last_undegraded":"2024-01-15T18:16:24.725727+0000","last_fullsized":"2024-01-15T18:16:24.725727+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.412953+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:19:45.412953+0000","last_peered":"2024-01-15T18:19:45.412953+0000","last_clean":"2024-01-15T18:19:45.412953+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:19:45.412953+0000","last_undegraded":"2024-01-15T18:19:45.412953+0000","last_fullsized":"2024-01-15T18:19:45.412953+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:14.988989+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:16:14.988989+0000","last_peered":"2024-01-15T18:16:14.988989+0000","last_clean":"2024-01-15T18:16:14.988989+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:16:14.988989+0000","last_undegraded":"2024-01-15T18:16:14.988989+0000","last_fullsized":"2024-01-15T18:16:14.988989+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'43","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.532353+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:19:45.532353+0000","last_peered":"2024-01-15T18:19:45.532353+0000","last_clean":"2024-01-15T18:19:45.532353+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:19:45.532353+0000","last_undegraded":"2024-01-15T18:19:45.532353+0000","last_fullsized":"2024-01-15T18:19:45.532353+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'3","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.687049+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:16:49.687049+0000","last_peered":"2024-01-15T18:16:49.687049+0000","last_clean":"2024-01-15T18:16:49.687049+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:16:49.687049+0000","last_undegraded":"2024-01-15T18:16:49.687049+0000","last_fullsized":"2024-01-15T18:16:49.687049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'56","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.396172+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:19:45.396172+0000","last_peered":"2024-01-15T18:19:45.396172+0000","last_clean":"2024-01-15T18:19:45.396172+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:19:45.396172+0000","last_undegraded":"2024-01-15T18:19:45.396172+0000","last_fullsized":"2024-01-15T18:19:45.396172+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.687023+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:16:49.687023+0000","last_peered":"2024-01-15T18:16:49.687023+0000","last_clean":"2024-01-15T18:16:49.687023+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:16:49.687023+0000","last_undegraded":"2024-01-15T18:16:49.687023+0000","last_fullsized":"2024-01-15T18:16:49.687023+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'39","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.786680+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:19:45.786680+0000","last_peered":"2024-01-15T18:19:45.786680+0000","last_clean":"2024-01-15T18:19:45.786680+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:19:45.786680+0000","last_undegraded":"2024-01-15T18:19:45.786680+0000","last_fullsized":"2024-01-15T18:19:45.786680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.731109+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:16:49.731109+0000","last_peered":"2024-01-15T18:16:49.731109+0000","last_clean":"2024-01-15T18:16:49.731109+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:16:49.731109+0000","last_undegraded":"2024-01-15T18:16:49.731109+0000","last_fullsized":"2024-01-15T18:16:49.731109+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'49","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.613573+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:19:45.613573+0000","last_peered":"2024-01-15T18:19:45.613573+0000","last_clean":"2024-01-15T18:19:45.613573+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:19:45.613573+0000","last_undegraded":"2024-01-15T18:19:45.613573+0000","last_fullsized":"2024-01-15T18:19:45.613573+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.704971+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:16:49.704971+0000","last_peered":"2024-01-15T18:16:49.704971+0000","last_clean":"2024-01-15T18:16:49.704971+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:16:49.704971+0000","last_undegraded":"2024-01-15T18:16:49.704971+0000","last_fullsized":"2024-01-15T18:16:49.704971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'53","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.610017+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:19:45.610017+0000","last_peered":"2024-01-15T18:19:45.610017+0000","last_clean":"2024-01-15T18:19:45.610017+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:19:45.610017+0000","last_undegraded":"2024-01-15T18:19:45.610017+0000","last_fullsized":"2024-01-15T18:19:45.610017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.818752+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:16:44.818752+0000","last_peered":"2024-01-15T18:16:44.818752+0000","last_clean":"2024-01-15T18:16:44.818752+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:16:44.818752+0000","last_undegraded":"2024-01-15T18:16:44.818752+0000","last_fullsized":"2024-01-15T18:16:44.818752+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'43","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407291+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:19:45.407291+0000","last_peered":"2024-01-15T18:19:45.407291+0000","last_clean":"2024-01-15T18:19:45.407291+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:19:45.407291+0000","last_undegraded":"2024-01-15T18:19:45.407291+0000","last_fullsized":"2024-01-15T18:19:45.407291+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.729197+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:16:44.729197+0000","last_peered":"2024-01-15T18:16:44.729197+0000","last_clean":"2024-01-15T18:16:44.729197+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:16:44.729197+0000","last_undegraded":"2024-01-15T18:16:44.729197+0000","last_fullsized":"2024-01-15T18:16:44.729197+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'42","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.391592+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:19:45.391592+0000","last_peered":"2024-01-15T18:19:45.391592+0000","last_clean":"2024-01-15T18:19:45.391592+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:19:45.391592+0000","last_undegraded":"2024-01-15T18:19:45.391592+0000","last_fullsized":"2024-01-15T18:19:45.391592+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.686133+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:16:44.686133+0000","last_peered":"2024-01-15T18:16:44.686133+0000","last_clean":"2024-01-15T18:16:44.686133+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:16:44.686133+0000","last_undegraded":"2024-01-15T18:16:44.686133+0000","last_fullsized":"2024-01-15T18:16:44.686133+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'40","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.640005+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:19:45.640005+0000","last_peered":"2024-01-15T18:19:45.640005+0000","last_clean":"2024-01-15T18:19:45.640005+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:19:45.640005+0000","last_undegraded":"2024-01-15T18:19:45.640005+0000","last_fullsized":"2024-01-15T18:19:45.640005+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.704016+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:16:44.704016+0000","last_peered":"2024-01-15T18:16:44.704016+0000","last_clean":"2024-01-15T18:16:44.704016+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:16:44.704016+0000","last_undegraded":"2024-01-15T18:16:44.704016+0000","last_fullsized":"2024-01-15T18:16:44.704016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'43","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741116+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:19:45.741116+0000","last_peered":"2024-01-15T18:19:45.741116+0000","last_clean":"2024-01-15T18:19:45.741116+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:19:45.741116+0000","last_undegraded":"2024-01-15T18:19:45.741116+0000","last_fullsized":"2024-01-15T18:19:45.741116+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'13","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:40.230861+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:17:40.230861+0000","last_peered":"2024-01-15T18:17:40.230861+0000","last_clean":"2024-01-15T18:17:40.230861+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:17:40.230861+0000","last_undegraded":"2024-01-15T18:17:40.230861+0000","last_fullsized":"2024-01-15T18:17:40.230861+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'43","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404696+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:19:45.404696+0000","last_peered":"2024-01-15T18:19:45.404696+0000","last_clean":"2024-01-15T18:19:45.404696+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:19:45.404696+0000","last_undegraded":"2024-01-15T18:19:45.404696+0000","last_fullsized":"2024-01-15T18:19:45.404696+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'186","reported_seq":202,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.420919+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:19:45.420919+0000","last_peered":"2024-01-15T18:19:45.420919+0000","last_clean":"2024-01-15T18:19:45.420919+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:19:45.420919+0000","last_undegraded":"2024-01-15T18:19:45.420919+0000","last_fullsized":"2024-01-15T18:19:45.420919+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":631739,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":185,"num_write_kb":705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'44","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410217+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:19:45.410217+0000","last_peered":"2024-01-15T18:19:45.410217+0000","last_clean":"2024-01-15T18:19:45.410217+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:19:45.410217+0000","last_undegraded":"2024-01-15T18:19:45.410217+0000","last_fullsized":"2024-01-15T18:19:45.410217+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.672899+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:16:39.672899+0000","last_peered":"2024-01-15T18:16:39.672899+0000","last_clean":"2024-01-15T18:16:39.672899+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:16:39.672899+0000","last_undegraded":"2024-01-15T18:16:39.672899+0000","last_fullsized":"2024-01-15T18:16:39.672899+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'52","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.597854+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:19:45.597854+0000","last_peered":"2024-01-15T18:19:45.597854+0000","last_clean":"2024-01-15T18:19:45.597854+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:19:45.597854+0000","last_undegraded":"2024-01-15T18:19:45.597854+0000","last_fullsized":"2024-01-15T18:19:45.597854+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.792106+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:16:39.792106+0000","last_peered":"2024-01-15T18:16:39.792106+0000","last_clean":"2024-01-15T18:16:39.792106+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:16:39.792106+0000","last_undegraded":"2024-01-15T18:16:39.792106+0000","last_fullsized":"2024-01-15T18:16:39.792106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'46","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559915+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:19:45.559915+0000","last_peered":"2024-01-15T18:19:45.559915+0000","last_clean":"2024-01-15T18:19:45.559915+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:19:45.559915+0000","last_undegraded":"2024-01-15T18:19:45.559915+0000","last_fullsized":"2024-01-15T18:19:45.559915+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":24,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.684911+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:16:39.684911+0000","last_peered":"2024-01-15T18:16:39.684911+0000","last_clean":"2024-01-15T18:16:39.684911+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:16:39.684911+0000","last_undegraded":"2024-01-15T18:16:39.684911+0000","last_fullsized":"2024-01-15T18:16:39.684911+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'24","reported_seq":43,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.533839+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:19:45.533839+0000","last_peered":"2024-01-15T18:19:45.533839+0000","last_clean":"2024-01-15T18:19:45.533839+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:19:45.533839+0000","last_undegraded":"2024-01-15T18:19:45.533839+0000","last_fullsized":"2024-01-15T18:19:45.533839+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.727345+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:16:34.727345+0000","last_peered":"2024-01-15T18:16:34.727345+0000","last_clean":"2024-01-15T18:16:34.727345+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:16:34.727345+0000","last_undegraded":"2024-01-15T18:16:34.727345+0000","last_fullsized":"2024-01-15T18:16:34.727345+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'36","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.633679+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:19:45.633679+0000","last_peered":"2024-01-15T18:19:45.633679+0000","last_clean":"2024-01-15T18:19:45.633679+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:19:45.633679+0000","last_undegraded":"2024-01-15T18:19:45.633679+0000","last_fullsized":"2024-01-15T18:19:45.633679+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.684130+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:16:34.684130+0000","last_peered":"2024-01-15T18:16:34.684130+0000","last_clean":"2024-01-15T18:16:34.684130+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:16:34.684130+0000","last_undegraded":"2024-01-15T18:16:34.684130+0000","last_fullsized":"2024-01-15T18:16:34.684130+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'44","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559602+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:19:45.559602+0000","last_peered":"2024-01-15T18:19:45.559602+0000","last_clean":"2024-01-15T18:19:45.559602+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:19:45.559602+0000","last_undegraded":"2024-01-15T18:19:45.559602+0000","last_fullsized":"2024-01-15T18:19:45.559602+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.791118+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:16:34.791118+0000","last_peered":"2024-01-15T18:16:34.791118+0000","last_clean":"2024-01-15T18:16:34.791118+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:16:34.791118+0000","last_undegraded":"2024-01-15T18:16:34.791118+0000","last_fullsized":"2024-01-15T18:16:34.791118+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'41","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.673339+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:19:45.673339+0000","last_peered":"2024-01-15T18:19:45.673339+0000","last_clean":"2024-01-15T18:19:45.673339+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:19:45.673339+0000","last_undegraded":"2024-01-15T18:19:45.673339+0000","last_fullsized":"2024-01-15T18:19:45.673339+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.727774+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:16:34.727774+0000","last_peered":"2024-01-15T18:16:34.727774+0000","last_clean":"2024-01-15T18:16:34.727774+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:16:34.727774+0000","last_undegraded":"2024-01-15T18:16:34.727774+0000","last_fullsized":"2024-01-15T18:16:34.727774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.866450+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:19:45.866450+0000","last_peered":"2024-01-15T18:19:45.866450+0000","last_clean":"2024-01-15T18:19:45.866450+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:19:45.866450+0000","last_undegraded":"2024-01-15T18:19:45.866450+0000","last_fullsized":"2024-01-15T18:19:45.866450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'3","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.671975+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:16:34.671975+0000","last_peered":"2024-01-15T18:16:34.671975+0000","last_clean":"2024-01-15T18:16:34.671975+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:16:34.671975+0000","last_undegraded":"2024-01-15T18:16:34.671975+0000","last_fullsized":"2024-01-15T18:16:34.671975+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'43","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.389233+0000","last_change":"2024-01-15T18:15:20.196078+0000","last_active":"2024-01-15T18:19:45.389233+0000","last_peered":"2024-01-15T18:19:45.389233+0000","last_clean":"2024-01-15T18:19:45.389233+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:19:45.389233+0000","last_undegraded":"2024-01-15T18:19:45.389233+0000","last_fullsized":"2024-01-15T18:19:45.389233+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:19.862117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.683005+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:16:29.683005+0000","last_peered":"2024-01-15T18:16:29.683005+0000","last_clean":"2024-01-15T18:16:29.683005+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:16:29.683005+0000","last_undegraded":"2024-01-15T18:16:29.683005+0000","last_fullsized":"2024-01-15T18:16:29.683005+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.587657+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:19:45.587657+0000","last_peered":"2024-01-15T18:19:45.587657+0000","last_clean":"2024-01-15T18:19:45.587657+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:19:45.587657+0000","last_undegraded":"2024-01-15T18:19:45.587657+0000","last_fullsized":"2024-01-15T18:19:45.587657+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.671013+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:16:29.671013+0000","last_peered":"2024-01-15T18:16:29.671013+0000","last_clean":"2024-01-15T18:16:29.671013+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:16:29.671013+0000","last_undegraded":"2024-01-15T18:16:29.671013+0000","last_fullsized":"2024-01-15T18:16:29.671013+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'42","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.765594+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:19:45.765594+0000","last_peered":"2024-01-15T18:19:45.765594+0000","last_clean":"2024-01-15T18:19:45.765594+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:19:45.765594+0000","last_undegraded":"2024-01-15T18:19:45.765594+0000","last_fullsized":"2024-01-15T18:19:45.765594+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.725995+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:16:29.725995+0000","last_peered":"2024-01-15T18:16:29.725995+0000","last_clean":"2024-01-15T18:16:29.725995+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:16:29.725995+0000","last_undegraded":"2024-01-15T18:16:29.725995+0000","last_fullsized":"2024-01-15T18:16:29.725995+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'36","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559183+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:19:45.559183+0000","last_peered":"2024-01-15T18:19:45.559183+0000","last_clean":"2024-01-15T18:19:45.559183+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:19:45.559183+0000","last_undegraded":"2024-01-15T18:19:45.559183+0000","last_fullsized":"2024-01-15T18:19:45.559183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.700861+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:16:29.700861+0000","last_peered":"2024-01-15T18:16:29.700861+0000","last_clean":"2024-01-15T18:16:29.700861+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:16:29.700861+0000","last_undegraded":"2024-01-15T18:16:29.700861+0000","last_fullsized":"2024-01-15T18:16:29.700861+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'41","reported_seq":81,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410772+0000","last_change":"2024-01-15T18:15:20.356060+0000","last_active":"2024-01-15T18:19:45.410772+0000","last_peered":"2024-01-15T18:19:45.410772+0000","last_clean":"2024-01-15T18:19:45.410772+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:19:45.410772+0000","last_undegraded":"2024-01-15T18:19:45.410772+0000","last_fullsized":"2024-01-15T18:19:45.410772+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:10.891015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'18","reported_seq":34,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:51.467225+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:18:51.467225+0000","last_peered":"2024-01-15T18:18:51.467225+0000","last_clean":"2024-01-15T18:18:51.467225+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:18:51.467225+0000","last_undegraded":"2024-01-15T18:18:51.467225+0000","last_fullsized":"2024-01-15T18:18:51.467225+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'46","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.536354+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:19:45.536354+0000","last_peered":"2024-01-15T18:19:45.536354+0000","last_clean":"2024-01-15T18:19:45.536354+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:45.536354+0000","last_undegraded":"2024-01-15T18:19:45.536354+0000","last_fullsized":"2024-01-15T18:19:45.536354+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.724998+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:16:24.724998+0000","last_peered":"2024-01-15T18:16:24.724998+0000","last_clean":"2024-01-15T18:16:24.724998+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:16:24.724998+0000","last_undegraded":"2024-01-15T18:16:24.724998+0000","last_fullsized":"2024-01-15T18:16:24.724998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'39","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404370+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:19:45.404370+0000","last_peered":"2024-01-15T18:19:45.404370+0000","last_clean":"2024-01-15T18:19:45.404370+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:19:45.404370+0000","last_undegraded":"2024-01-15T18:19:45.404370+0000","last_fullsized":"2024-01-15T18:19:45.404370+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.699700+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:16:24.699700+0000","last_peered":"2024-01-15T18:16:24.699700+0000","last_clean":"2024-01-15T18:16:24.699700+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:16:24.699700+0000","last_undegraded":"2024-01-15T18:16:24.699700+0000","last_fullsized":"2024-01-15T18:16:24.699700+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'44","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.695729+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:19:45.695729+0000","last_peered":"2024-01-15T18:19:45.695729+0000","last_clean":"2024-01-15T18:19:45.695729+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:19:45.695729+0000","last_undegraded":"2024-01-15T18:19:45.695729+0000","last_fullsized":"2024-01-15T18:19:45.695729+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.725579+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:16:24.725579+0000","last_peered":"2024-01-15T18:16:24.725579+0000","last_clean":"2024-01-15T18:16:24.725579+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:16:24.725579+0000","last_undegraded":"2024-01-15T18:16:24.725579+0000","last_fullsized":"2024-01-15T18:16:24.725579+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'56","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.774102+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:19:45.774102+0000","last_peered":"2024-01-15T18:19:45.774102+0000","last_clean":"2024-01-15T18:19:45.774102+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:19:45.774102+0000","last_undegraded":"2024-01-15T18:19:45.774102+0000","last_fullsized":"2024-01-15T18:19:45.774102+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.814331+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:16:24.814331+0000","last_peered":"2024-01-15T18:16:24.814331+0000","last_clean":"2024-01-15T18:16:24.814331+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:16:24.814331+0000","last_undegraded":"2024-01-15T18:16:24.814331+0000","last_fullsized":"2024-01-15T18:16:24.814331+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741049+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:19:45.741049+0000","last_peered":"2024-01-15T18:19:45.741049+0000","last_clean":"2024-01-15T18:19:45.741049+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:19:45.741049+0000","last_undegraded":"2024-01-15T18:19:45.741049+0000","last_fullsized":"2024-01-15T18:19:45.741049+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'11","reported_seq":37,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:40.222936+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:17:40.222936+0000","last_peered":"2024-01-15T18:17:40.222936+0000","last_clean":"2024-01-15T18:17:40.222936+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:17:40.222936+0000","last_undegraded":"2024-01-15T18:17:40.222936+0000","last_fullsized":"2024-01-15T18:17:40.222936+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'39","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.718813+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:19:45.718813+0000","last_peered":"2024-01-15T18:19:45.718813+0000","last_clean":"2024-01-15T18:19:45.718813+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:19:45.718813+0000","last_undegraded":"2024-01-15T18:19:45.718813+0000","last_fullsized":"2024-01-15T18:19:45.718813+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:19.723501+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:16:19.723501+0000","last_peered":"2024-01-15T18:16:19.723501+0000","last_clean":"2024-01-15T18:16:19.723501+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:16:19.723501+0000","last_undegraded":"2024-01-15T18:16:19.723501+0000","last_fullsized":"2024-01-15T18:16:19.723501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'31","reported_seq":54,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.579344+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:19:45.579344+0000","last_peered":"2024-01-15T18:19:45.579344+0000","last_clean":"2024-01-15T18:19:45.579344+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:19:45.579344+0000","last_undegraded":"2024-01-15T18:19:45.579344+0000","last_fullsized":"2024-01-15T18:19:45.579344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.728790+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:16:39.728790+0000","last_peered":"2024-01-15T18:16:39.728790+0000","last_clean":"2024-01-15T18:16:39.728790+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:16:39.728790+0000","last_undegraded":"2024-01-15T18:16:39.728790+0000","last_fullsized":"2024-01-15T18:16:39.728790+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":254,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:30.003748+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:17:30.003748+0000","last_peered":"2024-01-15T18:17:30.003748+0000","last_clean":"2024-01-15T18:17:30.003748+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:17:30.003748+0000","last_undegraded":"2024-01-15T18:17:30.003748+0000","last_fullsized":"2024-01-15T18:17:30.003748+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":633858,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":281,"num_write_kb":746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1335296,"data_stored":1267716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.296,"15min":1.325},"min":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.54000000000000004},"max":{"1min":1.782,"5min":2.1200000000000001,"15min":9.1379999999999999},"last":1.534},{"interface":"front","average":{"1min":1.268,"5min":1.385,"15min":1.528},"min":{"1min":0.88400000000000001,"5min":0.81999999999999995,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":1.905,"15min":27.32},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2210000000000001,"15min":1.4530000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.728,"5min":2.1000000000000001,"15min":29.167999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.24,"15min":1.3260000000000001},"min":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.52800000000000002},"max":{"1min":1.597,"5min":2.6179999999999999,"15min":8.5980000000000008},"last":1.1759999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3740000000000001,"15min":1.4179999999999999},"min":{"1min":0.94699999999999995,"5min":0.874,"15min":0.56000000000000005},"max":{"1min":1.7709999999999999,"5min":2.3900000000000001,"15min":12.709},"last":1.393},{"interface":"front","average":{"1min":1.141,"5min":1.256,"15min":1.488},"min":{"1min":0.79400000000000004,"5min":0.72499999999999998,"15min":0.56599999999999995},"max":{"1min":1.6539999999999999,"5min":1.8200000000000001,"15min":27.358000000000001},"last":0.629}]},{"osd":3,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.556},"min":{"1min":0.98599999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.643,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.4530000000000001,"15min":1.5189999999999999},"min":{"1min":1.01,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.7769999999999999,"5min":2.4940000000000002,"15min":16.088999999999999},"last":1.4850000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.381,"15min":1.5269999999999999},"min":{"1min":0.94899999999999995,"5min":0.64700000000000002,"15min":0.50700000000000001},"max":{"1min":1.681,"5min":6.0350000000000001,"15min":27.219000000000001},"last":1.23},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4339999999999999,"15min":1.641},"min":{"1min":0.90400000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7629999999999999,"5min":3.536,"15min":29.25},"last":1.2849999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4730000000000001,"15min":1.573},"min":{"1min":1.0209999999999999,"5min":0.68799999999999994,"15min":0.67000000000000004},"max":{"1min":1.8660000000000001,"5min":2.7120000000000002,"15min":11.834},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.5129999999999999,"15min":1.675},"min":{"1min":0.92900000000000005,"5min":0.71199999999999997,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.343}]},{"osd":6,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.635,"15min":1.8200000000000001},"min":{"1min":1.1850000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.929,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5409999999999999,"15min":1.8049999999999999},"min":{"1min":1.0289999999999999,"5min":0.63100000000000001,"15min":0.57299999999999995},"max":{"1min":1.841,"5min":2.754,"15min":29.106999999999999},"last":1.6719999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542415,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.2190000000000001,"15min":1.236},"min":{"1min":0.72299999999999998,"5min":0.68200000000000005,"15min":0.438},"max":{"1min":1.5409999999999999,"5min":2.77,"15min":8.8350000000000009},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1970000000000001,"15min":1.27},"min":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.59599999999999997},"max":{"1min":1.643,"5min":2.673,"15min":9.548},"last":1.0609999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2729999999999999,"15min":1.395},"min":{"1min":0.89300000000000002,"5min":0.70999999999999996,"15min":0.52100000000000002},"max":{"1min":1.5209999999999999,"5min":2.8700000000000001,"15min":13.269},"last":1.45},{"interface":"front","average":{"1min":1.262,"5min":1.3009999999999999,"15min":1.4119999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.57799999999999996},"max":{"1min":1.6990000000000001,"5min":3.1080000000000001,"15min":13.984999999999999},"last":0.92700000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.232,"15min":1.284},"min":{"1min":0.85299999999999998,"5min":0.79000000000000004,"15min":0.69799999999999995},"max":{"1min":1.7,"5min":1.9379999999999999,"15min":7.0030000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.373,"15min":1.425},"min":{"1min":0.81100000000000005,"5min":0.73899999999999999,"15min":0.49399999999999999},"max":{"1min":1.623,"5min":2.6909999999999998,"15min":8.9359999999999999},"last":1.2749999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.444,"15min":1.55},"min":{"1min":1.131,"5min":1.0089999999999999,"15min":0.68500000000000005},"max":{"1min":1.7629999999999999,"5min":3.339,"15min":16.855},"last":1.363},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.413,"15min":1.5209999999999999},"min":{"1min":1.111,"5min":1.046,"15min":0.628},"max":{"1min":1.657,"5min":2.0139999999999998,"15min":10.875},"last":1.1140000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.401,"15min":1.45},"min":{"1min":0.86699999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":1.645,"5min":3.0299999999999998,"15min":8.7539999999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.427,"15min":1.4970000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.819,"5min":5.125,"15min":10.118},"last":1.242}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3600000000000001,"15min":1.5209999999999999},"min":{"1min":0.92500000000000004,"5min":0.88,"15min":0.60299999999999998},"max":{"1min":1.8120000000000001,"5min":2.3210000000000002,"15min":16.309999999999999},"last":1.016},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5429999999999999,"15min":1.6200000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.66000000000000003},"max":{"1min":1.8029999999999999,"5min":3.0209999999999999,"15min":10.077},"last":1.3899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5289999999999999,"15min":1.6279999999999999},"min":{"1min":0.995,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":1.881,"5min":3.2450000000000001,"15min":10.302},"last":1.431},{"interface":"front","average":{"1min":1.504,"5min":1.663,"15min":1.6879999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":1.9059999999999999,"5min":7.6799999999999997,"15min":9.4079999999999995},"last":1.2989999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542417,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928972,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93442868,"statfs":{"total":96636764160,"available":95685496832,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2609999999999999,"15min":1.399},"min":{"1min":0.79300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.6180000000000001,"5min":4.1829999999999998,"15min":26.077999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3740000000000001,"15min":1.4299999999999999},"min":{"1min":1.099,"5min":0.77900000000000003,"15min":0.56399999999999995},"max":{"1min":1.6200000000000001,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.2789999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2549999999999999,"15min":1.4339999999999999},"min":{"1min":0.875,"5min":0.82499999999999996,"15min":0.53300000000000003},"max":{"1min":1.8,"5min":1.923,"15min":26},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.367,"15min":1.401},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.657,"5min":4.0869999999999997,"15min":12.619},"last":1.2050000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3460000000000001,"15min":1.5329999999999999},"min":{"1min":1.0700000000000001,"5min":0.69999999999999996,"15min":0.56699999999999995},"max":{"1min":1.712,"5min":2.6869999999999998,"15min":26.234999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.385,"5min":1.331,"15min":1.532},"min":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.629},"max":{"1min":1.766,"5min":1.8240000000000001,"15min":17.126999999999999},"last":1.105}]},{"osd":3,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3120000000000001,"15min":1.397},"min":{"1min":1.0429999999999999,"5min":0.82399999999999995,"15min":0.67200000000000004},"max":{"1min":1.792,"5min":2.375,"15min":12.722},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4830000000000001,"15min":1.6399999999999999},"min":{"1min":1.046,"5min":0.75900000000000001,"15min":0.67900000000000005},"max":{"1min":1.7989999999999999,"5min":3.238,"15min":26.134},"last":1.0449999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2769999999999999,"15min":1.4139999999999999},"min":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.442},"max":{"1min":1.77,"5min":2.5350000000000001,"15min":9.9979999999999993},"last":1.371},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.458,"15min":1.6930000000000001},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.498},"max":{"1min":1.875,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.4730000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3640000000000001,"15min":1.4339999999999999},"min":{"1min":1.0069999999999999,"5min":0.82299999999999995,"15min":0.45900000000000002},"max":{"1min":1.849,"5min":2.6640000000000001,"15min":12.769},"last":1.52},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5860000000000001,"15min":1.647},"min":{"1min":0.94799999999999995,"5min":0.79800000000000004,"15min":0.49199999999999999},"max":{"1min":1.909,"5min":11.851000000000001,"15min":26.186},"last":1.153}]},{"osd":7,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5960000000000001,"15min":1.798},"min":{"1min":1.1599999999999999,"5min":1.1359999999999999,"15min":0.65700000000000003},"max":{"1min":2.04,"5min":2.3700000000000001,"15min":13.706},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.54,"15min":1.679},"min":{"1min":1.038,"5min":1.0189999999999999,"15min":0.60099999999999998},"max":{"1min":2.0289999999999999,"5min":4.3700000000000001,"15min":21.728000000000002},"last":1.399}]}]},{"osd":4,"up_from":14,"seq":60129542416,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.343,"15min":1.27},"min":{"1min":0.92800000000000005,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.7490000000000001,"5min":13.288,"15min":15.044},"last":1.004},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.284,"15min":1.321},"min":{"1min":0.91600000000000004,"5min":0.71299999999999997,"15min":0.34699999999999998},"max":{"1min":1.5389999999999999,"5min":9.2409999999999997,"15min":11.196},"last":1.393}]},{"osd":1,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2569999999999999,"15min":1.3779999999999999},"min":{"1min":0.90400000000000003,"5min":0.78400000000000003,"15min":0.41499999999999998},"max":{"1min":1.655,"5min":3.5910000000000002,"15min":28.663},"last":1.341},{"interface":"front","average":{"1min":1.278,"5min":1.302,"15min":1.4370000000000001},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.6319999999999999,"5min":3.6739999999999999,"15min":15.096},"last":1.4590000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.333,"15min":1.4319999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.8170000000000002,"15min":16.023},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.381,"15min":1.5169999999999999},"min":{"1min":0.94899999999999995,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.7869999999999999,"5min":3.7930000000000001,"15min":18.858000000000001},"last":1.244}]},{"osd":3,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3029999999999999,"15min":1.393},"min":{"1min":0.80100000000000005,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.847,"5min":5.0599999999999996,"15min":11.449},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.4510000000000001,"15min":1.5720000000000001},"min":{"1min":0.995,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.7969999999999999,"5min":3.7080000000000002,"15min":18.027000000000001},"last":1.4970000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4390000000000001,"15min":1.6220000000000001},"min":{"1min":0.88900000000000001,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.8140000000000001,"5min":7.7439999999999998,"15min":54.938000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.339,"15min":1.482},"min":{"1min":0.96499999999999997,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":1.857,"5min":2.0960000000000001,"15min":13.031000000000001},"last":1.171}]},{"osd":6,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.4119999999999999,"15min":1.5509999999999999},"min":{"1min":1.1180000000000001,"5min":0.90300000000000002,"15min":0.495},"max":{"1min":1.8779999999999999,"5min":3.766,"15min":16.103999999999999},"last":1.429},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5920000000000001,"15min":1.6799999999999999},"min":{"1min":1.1559999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.909,"5min":8.1609999999999996,"15min":15.118},"last":1.5940000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.45,"15min":1.786},"min":{"1min":1.1919999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.0299999999999998,"5min":2.4689999999999999,"15min":23.390000000000001},"last":1.532},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5609999999999999,"15min":1.756},"min":{"1min":1.2270000000000001,"5min":1.0409999999999999,"15min":0.62},"max":{"1min":1.99,"5min":3.9020000000000001,"15min":16.763000000000002},"last":1.5620000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542417,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462020,"kb_used_data":1386492,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909820,"statfs":{"total":96636764160,"available":95139655680,"internally_reserved":0,"allocated":1419767808,"data_stored":1418939088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95199999999999996,"15min":1.1020000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":1.579,"5min":1.6599999999999999,"15min":36.408000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.101,"15min":1.093},"min":{"1min":0.69799999999999995,"5min":0.63900000000000001,"15min":0.377},"max":{"1min":1.6930000000000001,"5min":2.0230000000000001,"15min":5.0190000000000001},"last":1.171}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.99099999999999999,"15min":1.0640000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.34399999999999997},"max":{"1min":1.756,"5min":1.837,"15min":5.226},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.113,"5min":1.1899999999999999,"15min":1.22},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.669,"5min":2.0609999999999999,"15min":7.2169999999999996},"last":0.749}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2569999999999999,"15min":1.3500000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.637,"5min":2.0150000000000001,"15min":14.025},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.345,"15min":1.3400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.841,"5min":2.403,"15min":3.6240000000000001},"last":0.96199999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.4259999999999999},"min":{"1min":0.84799999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"max":{"1min":1.9350000000000001,"5min":3.1459999999999999,"15min":9.0069999999999997},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.448,"15min":1.6040000000000001},"min":{"1min":1.091,"5min":0.79100000000000004,"15min":0.69499999999999995},"max":{"1min":1.696,"5min":4.8570000000000002,"15min":15.769},"last":1.3360000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.468,"15min":1.579},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65000000000000002},"max":{"1min":1.7490000000000001,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.546},{"interface":"front","average":{"1min":1.468,"5min":1.5169999999999999,"15min":1.7809999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":2.1219999999999999,"5min":2.3959999999999999,"15min":38.832999999999998},"last":1.595}]},{"osd":6,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5469999999999999,"15min":1.7529999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.51400000000000001},"max":{"1min":1.804,"5min":7.1020000000000003,"15min":32.259999999999998},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.5740000000000001,"15min":1.7470000000000001},"min":{"1min":1.0389999999999999,"5min":0.85999999999999999,"15min":0.747},"max":{"1min":2.169,"5min":2.419,"15min":16.195},"last":1.448}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.6719999999999999,"15min":1.905},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":1.466},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.728,"15min":1.8009999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":1.976,"5min":16.734000000000002,"15min":17.622},"last":1.6439999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542416,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.012,"15min":1.0249999999999999},"min":{"1min":0.63500000000000001,"5min":0.44400000000000001,"15min":0.35199999999999998},"max":{"1min":1.5069999999999999,"5min":3.1589999999999998,"15min":10.256},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2150000000000001,"15min":1.1819999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.33700000000000002},"max":{"1min":1.8180000000000001,"5min":8.9719999999999995,"15min":10.319000000000001},"last":0.85799999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1,"15min":1.026},"min":{"1min":0.67900000000000005,"5min":0.48899999999999999,"15min":0.38600000000000001},"max":{"1min":1.573,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":1.157},{"interface":"front","average":{"1min":1.097,"5min":1.1890000000000001,"15min":1.2509999999999999},"min":{"1min":0.74099999999999999,"5min":0.59499999999999997,"15min":0.35299999999999998},"max":{"1min":1.5669999999999999,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.3149999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3240000000000001,"15min":1.4219999999999999},"min":{"1min":0.82899999999999996,"5min":0.54200000000000004,"15min":0.44400000000000001},"max":{"1min":1.5529999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.224,"15min":1.306},"min":{"1min":0.89000000000000001,"5min":0.56599999999999995,"15min":0.376},"max":{"1min":1.7689999999999999,"5min":2.7589999999999999,"15min":14.946},"last":1.177}]},{"osd":4,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.536,"15min":1.524},"min":{"1min":1.139,"5min":0.82099999999999995,"15min":0.66100000000000003},"max":{"1min":1.7769999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4990000000000001,"15min":1.5960000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.65300000000000002},"max":{"1min":1.768,"5min":2.867,"15min":10.279999999999999},"last":1.4059999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4770000000000001,"15min":1.5700000000000001},"min":{"1min":1,"5min":1,"15min":0.63500000000000001},"max":{"1min":1.6830000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.111},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.613,"15min":1.6200000000000001},"min":{"1min":1.0209999999999999,"5min":0.88200000000000001,"15min":0.71699999999999997},"max":{"1min":1.895,"5min":10.833,"15min":10.833},"last":1.5229999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.6180000000000001},"min":{"1min":1.0109999999999999,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.827,"5min":2.984,"15min":10.185},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.657,"15min":1.7250000000000001},"min":{"1min":1.107,"5min":1.1060000000000001,"15min":0.79700000000000004},"max":{"1min":1.843,"5min":10.917,"15min":22.382000000000001},"last":1.2030000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.6180000000000001,"15min":1.843},"min":{"1min":1.129,"5min":1.006,"15min":0.77700000000000002},"max":{"1min":1.897,"5min":3.052,"15min":19.986000000000001},"last":1.583},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6659999999999999,"15min":1.77},"min":{"1min":1.252,"5min":0.89300000000000002,"15min":0.877},"max":{"1min":1.931,"5min":3.069,"15min":11.893000000000001},"last":1.29}]}]},{"osd":1,"up_from":14,"seq":60129542416,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1137752,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93234088,"statfs":{"total":96636764160,"available":95471706112,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96599999999999997,"15min":1.1020000000000001},"min":{"1min":0.63300000000000001,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":1.7130000000000001,"5min":3.4510000000000001,"15min":19.218},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.232},"min":{"1min":0.65500000000000003,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.694,"5min":2.0939999999999999,"15min":19.823},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99099999999999999,"15min":1.1100000000000001},"min":{"1min":0.80000000000000004,"5min":0.54400000000000004,"15min":0.31900000000000001},"max":{"1min":1.776,"5min":2.5110000000000001,"15min":19.02},"last":1.603},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.258,"15min":1.401},"min":{"1min":0.85699999999999998,"5min":0.63300000000000001,"15min":0.41599999999999998},"max":{"1min":1.835,"5min":3.1400000000000001,"15min":29.251000000000001},"last":1.4279999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.4990000000000001},"min":{"1min":0.84899999999999998,"5min":0.58499999999999996,"15min":0.40899999999999997},"max":{"1min":1.6779999999999999,"5min":5.1459999999999999,"15min":27.021000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.194,"15min":1.391},"min":{"1min":0.77600000000000002,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.7669999999999999,"5min":3.0150000000000001,"15min":31.218},"last":1.5229999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.492,"15min":1.6890000000000001},"min":{"1min":1.1379999999999999,"5min":0.755,"15min":0.70599999999999996},"max":{"1min":1.8049999999999999,"5min":5.2169999999999996,"15min":27.059999999999999},"last":0.998},{"interface":"front","average":{"1min":1.409,"5min":1.4450000000000001,"15min":1.6359999999999999},"min":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.56499999999999995},"max":{"1min":1.9650000000000001,"5min":3.4159999999999999,"15min":31.433},"last":1.097}]},{"osd":5,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4730000000000001,"15min":1.645},"min":{"1min":1.149,"5min":0.82499999999999996,"15min":0.68500000000000005},"max":{"1min":1.8560000000000001,"5min":3.0950000000000002,"15min":20.012},"last":1.224},{"interface":"front","average":{"1min":1.339,"5min":1.431,"15min":1.52},"min":{"1min":0.97699999999999998,"5min":0.85799999999999998,"15min":0.69299999999999995},"max":{"1min":1.631,"5min":2.9409999999999998,"15min":6.758},"last":1.45}]},{"osd":6,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.639,"15min":1.794},"min":{"1min":1.28,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.9339999999999999,"5min":5.0629999999999997,"15min":29.530000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.575,"15min":1.802},"min":{"1min":1.2030000000000001,"5min":0.88400000000000001,"15min":0.61399999999999999},"max":{"1min":1.845,"5min":3.1960000000000002,"15min":31.186},"last":1.397}]},{"osd":7,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5169999999999999,"15min":1.8140000000000001},"min":{"1min":1.0860000000000001,"5min":0.95899999999999996,"15min":0.63},"max":{"1min":1.8460000000000001,"5min":3.3769999999999998,"15min":26.393999999999998},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.6240000000000001,"15min":1.8380000000000001},"min":{"1min":1.1659999999999999,"5min":0.89200000000000002,"15min":0.76100000000000001},"max":{"1min":1.913,"5min":3.5750000000000002,"15min":27.582000000000001},"last":1.7909999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1777940,"kb_used_data":1706572,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92593900,"statfs":{"total":96636764160,"available":94816153600,"internally_reserved":0,"allocated":1747529728,"data_stored":1746685443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97699999999999998,"15min":1.141},"min":{"1min":0.57199999999999995,"5min":0.42299999999999999,"15min":0.30599999999999999},"max":{"1min":1.5860000000000001,"5min":5.7560000000000002,"15min":20.437999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.236,"15min":1.278},"min":{"1min":0.77000000000000002,"5min":0.65300000000000002,"15min":0.32400000000000001},"max":{"1min":1.8660000000000001,"5min":6.3600000000000003,"15min":35.137999999999998},"last":1.052}]},{"osd":2,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.236,"15min":1.3879999999999999},"min":{"1min":0.66100000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.716,"5min":11.06,"15min":20.556000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2789999999999999,"15min":1.375},"min":{"1min":0.85099999999999998,"5min":0.60699999999999998,"15min":0.39000000000000001},"max":{"1min":1.8919999999999999,"5min":6.4939999999999998,"15min":35.183},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.238,"15min":1.1819999999999999},"min":{"1min":0.621,"5min":0.55000000000000004,"15min":0.42299999999999999},"max":{"1min":2.0230000000000001,"5min":10.885999999999999,"15min":10.885999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.266,"5min":1.3779999999999999,"15min":1.635},"min":{"1min":0.90000000000000002,"5min":0.73099999999999998,"15min":0.45100000000000001},"max":{"1min":1.621,"5min":9.3339999999999996,"15min":25.372},"last":0.71399999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.532,"15min":1.621},"min":{"1min":1.0509999999999999,"5min":1.014,"15min":0.67200000000000004},"max":{"1min":1.9710000000000001,"5min":9.9969999999999999,"15min":35.223999999999997},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.581,"15min":1.49},"min":{"1min":1.016,"5min":0.99399999999999999,"15min":0.58799999999999997},"max":{"1min":1.96,"5min":17.567,"15min":17.567},"last":1.5429999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6899999999999999,"15min":1.6499999999999999},"min":{"1min":1.099,"5min":0.92700000000000005,"15min":0.69799999999999995},"max":{"1min":1.9990000000000001,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.679,"15min":1.8300000000000001},"min":{"1min":1.0029999999999999,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.966,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.6419999999999999,"15min":1.923},"min":{"1min":1.1879999999999999,"5min":1.004,"15min":0.70599999999999996},"max":{"1min":1.875,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.244},{"interface":"front","average":{"1min":1.595,"5min":1.6619999999999999,"15min":1.8440000000000001},"min":{"1min":1.2130000000000001,"5min":0.90600000000000003,"15min":0.62},"max":{"1min":2.2149999999999999,"5min":6.2270000000000003,"15min":35.255000000000003},"last":1.512}]},{"osd":7,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.704,"15min":1.9319999999999999},"min":{"1min":1.153,"5min":1.089,"15min":0.56200000000000006},"max":{"1min":2.2919999999999998,"5min":9.9429999999999996,"15min":35.636000000000003},"last":1.591},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.6639999999999999,"15min":1.6919999999999999},"min":{"1min":1.1759999999999999,"5min":0.96499999999999997,"15min":0.64000000000000001},"max":{"1min":2.056,"5min":10.943,"15min":11.470000000000001},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":631851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":631809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:19:53.104 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-15T18:19:53.104 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:19:53.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.360+0000 7f58c2000640 1 Processor -- start 2024-01-15T18:19:53.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.360+0000 7f58c2000640 1 -- start start 2024-01-15T18:19:53.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58c2000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f58bc159690 0x7f58bc15d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58c2000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc06cd30 con 0x7f58bc0ce5c0 2024-01-15T18:19:53.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58c2000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0b3e20 con 0x7f58bc06d1c0 2024-01-15T18:19:53.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58c2000640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58bc070d60 con 0x7f58bc159690 2024-01-15T18:19:53.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58c0c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f58bc159690 0x7f58bc15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58bbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f58bc0ce5c0 0x7f58bc15d7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55052/0 (socket says 172.21.15.2:55052) 2024-01-15T18:19:53.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.362+0000 7f58bbe00640 1 -- 172.21.15.2:0/686869841 learned_addr learned my addr 172.21.15.2:0/686869841 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:53.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.363+0000 7f58bb400640 1 -- 172.21.15.2:0/686869841 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140986417 0 0) 0x7f58bc06cd30 con 0x7f58bc0ce5c0 2024-01-15T18:19:53.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.363+0000 7f58bb400640 1 -- 172.21.15.2:0/686869841 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f589c003520 con 0x7f58bc0ce5c0 2024-01-15T18:19:53.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.363+0000 7f58c0c00640 1 -- 172.21.15.2:0/686869841 >> v1:172.21.15.137:6789/0 conn(0x7f58bc06d1c0 legacy=0x7f58bc06d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.363+0000 7f58c0c00640 1 -- 172.21.15.2:0/686869841 >> v1:172.21.15.2:6789/0 conn(0x7f58bc0ce5c0 legacy=0x7f58bc15d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.363+0000 7f58c0c00640 1 -- 172.21.15.2:0/686869841 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc071040 con 0x7f58bc159690 2024-01-15T18:19:53.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.364+0000 7f58c0c00640 1 --2- 172.21.15.2:0/686869841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc159690 0x7f58bc15d280 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0097a0 tx=0x7f58ac02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2baf9f7eff62c7d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:53.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.364+0000 7f58bb400640 1 -- 172.21.15.2:0/686869841 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac030610 con 0x7f58bc159690 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.364+0000 7f58bb400640 1 -- 172.21.15.2:0/686869841 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac030770 con 0x7f58bc159690 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.364+0000 7f58bb400640 1 -- 172.21.15.2:0/686869841 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac030b00 con 0x7f58bc159690 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 -- 172.21.15.2:0/686869841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc159690 msgr2=0x7f58bc15d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 --2- 172.21.15.2:0/686869841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc159690 0x7f58bc15d280 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f58ac0097a0 tx=0x7f58ac02b960 comp rx=0 tx=0).stop 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 -- 172.21.15.2:0/686869841 shutdown_connections 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 --2- 172.21.15.2:0/686869841 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc159690 0x7f58bc15d280 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 -- 172.21.15.2:0/686869841 >> 172.21.15.2:0/686869841 conn(0x7f58bc0b0840 msgr2=0x7f58bc0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:53.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 -- 172.21.15.2:0/686869841 shutdown_connections 2024-01-15T18:19:53.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.365+0000 7f58c2000640 1 -- 172.21.15.2:0/686869841 wait complete. 2024-01-15T18:19:53.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.367+0000 7f58c2000640 1 Processor -- start 2024-01-15T18:19:53.370 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.367+0000 7f58c2000640 1 -- start start 2024-01-15T18:19:53.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c2000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 0x7f58bc142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c2000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58bc0ce5c0 0x7f58bc0cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c2000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58bc159690 0x7f58bc145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c2000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1ee550 con 0x7f58bc06d1c0 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c2000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1ed8d0 con 0x7f58bc159690 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c2000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58bc1edbb0 con 0x7f58bc0ce5c0 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 0x7f58bc142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c0c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 0x7f58bc142a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51754/0 (socket says 172.21.15.2:51754) 2024-01-15T18:19:53.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.368+0000 7f58c0c00640 1 -- 172.21.15.2:0/2010864470 learned_addr learned my addr 172.21.15.2:0/2010864470 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c0c00640 1 -- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58bc0ce5c0 msgr2=0x7f58bc0cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c0c00640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58bc0ce5c0 0x7f58bc0cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c0c00640 1 -- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58bc159690 msgr2=0x7f58bc145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c0c00640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58bc159690 0x7f58bc145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c0c00640 1 -- 172.21.15.2:0/2010864470 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac009400 con 0x7f58bc06d1c0 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c0c00640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 0x7f58bc142a00 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f58ac02bf70 tx=0x7f58ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac03e070 con 0x7f58bc06d1c0 2024-01-15T18:19:53.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ac030e10 con 0x7f58bc06d1c0 2024-01-15T18:19:53.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac039c30 con 0x7f58bc06d1c0 2024-01-15T18:19:53.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.369+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc1ede90 con 0x7f58bc06d1c0 2024-01-15T18:19:53.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.370+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc142f40 con 0x7f58bc06d1c0 2024-01-15T18:19:53.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.371+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc1ee690 con 0x7f58bc06d1c0 2024-01-15T18:19:53.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.372+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f58ac04b400 con 0x7f58bc06d1c0 2024-01-15T18:19:53.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.373+0000 7f58b8c00640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f589406c550 0x7f589406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.373+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f58ac0aaeb0 con 0x7f58bc06d1c0 2024-01-15T18:19:53.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.374+0000 7f58bbe00640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f589406c550 0x7f589406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.375+0000 7f58bbe00640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f589406c550 0x7f589406ea10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f58bc1f3ce0 tx=0x7f58a4007930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:53.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.377+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58bc1ee690 con 0x7f58bc06d1c0 2024-01-15T18:19:53.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.570+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58bc159a70 con 0x7f589406c550 2024-01-15T18:19:53.601 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.598+0000 7f58b8c00640 1 -- 172.21.15.2:0/2010864470 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288946 (secure 0 0 0) 0x7f58a40058b0 con 0x7f589406c550 2024-01-15T18:19:53.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:53.606 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:19:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f589406c550 msgr2=0x7f589406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f589406c550 0x7f589406ea10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f58bc1f3ce0 tx=0x7f58a4007930 comp rx=0 tx=0).stop 2024-01-15T18:19:53.608 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 msgr2=0x7f58bc142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 0x7f58bc142a00 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f58ac02bf70 tx=0x7f58ac0046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 shutdown_connections 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f589406c550 0x7f589406ea10 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f58bc159690 0x7f58bc145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f58bc0ce5c0 0x7f58bc0cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 --2- 172.21.15.2:0/2010864470 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f58bc06d1c0 0x7f58bc142a00 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 >> 172.21.15.2:0/2010864470 conn(0x7f58bc0b0840 msgr2=0x7f58bc0b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 shutdown_connections 2024-01-15T18:19:53.609 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.603+0000 7f58c2000640 1 -- 172.21.15.2:0/2010864470 wait complete. 2024-01-15T18:19:53.625 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2024-01-15T18:19:51.957916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369934362,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1746,"num_write_kb":5246057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3187,"ondisk_log_size":3187,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11070440,"kb_used_data":10499496,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":743904280,"statfs":{"total":773094113280,"available":761757982720,"internally_reserved":0,"allocated":10751483904,"data_stored":10744924764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":584568369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":70,"apply_latency_ms":70,"commit_latency_ns":70000000,"apply_latency_ns":70000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1073776448,"num_objects":256,"num_object_clones":0,"num_object_copies":512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":256,"num_whiteouts":0,"num_read":256,"num_read_kb":1048576,"num_write":274,"num_write_kb":1048616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004677"},"pg_stats":[{"pgid":"17.3f","version":"116'53","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.408247+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:19:45.408247+0000","last_peered":"2024-01-15T18:19:45.408247+0000","last_clean":"2024-01-15T18:19:45.408247+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:19:45.408247+0000","last_undegraded":"2024-01-15T18:19:45.408247+0000","last_fullsized":"2024-01-15T18:19:45.408247+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:31:23.362167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.516826+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:19:45.516826+0000","last_peered":"2024-01-15T18:19:45.516826+0000","last_clean":"2024-01-15T18:19:45.516826+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:19:45.516826+0000","last_undegraded":"2024-01-15T18:19:45.516826+0000","last_fullsized":"2024-01-15T18:19:45.516826+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:14:55.855067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.675239+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:19:45.675239+0000","last_peered":"2024-01-15T18:19:45.675239+0000","last_clean":"2024-01-15T18:19:45.675239+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:19:45.675239+0000","last_undegraded":"2024-01-15T18:19:45.675239+0000","last_fullsized":"2024-01-15T18:19:45.675239+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'45","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.578232+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:19:45.578232+0000","last_peered":"2024-01-15T18:19:45.578232+0000","last_clean":"2024-01-15T18:19:45.578232+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:19:45.578232+0000","last_undegraded":"2024-01-15T18:19:45.578232+0000","last_fullsized":"2024-01-15T18:19:45.578232+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:50:48.313389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'27","reported_seq":49,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.370622+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:19:45.370622+0000","last_peered":"2024-01-15T18:19:45.370622+0000","last_clean":"2024-01-15T18:19:45.370622+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:19:45.370622+0000","last_undegraded":"2024-01-15T18:19:45.370622+0000","last_fullsized":"2024-01-15T18:19:45.370622+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:39.695392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'48","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410257+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:19:45.410257+0000","last_peered":"2024-01-15T18:19:45.410257+0000","last_clean":"2024-01-15T18:19:45.410257+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:19:45.410257+0000","last_undegraded":"2024-01-15T18:19:45.410257+0000","last_fullsized":"2024-01-15T18:19:45.410257+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:33:57.315186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'43","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.408637+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:19:45.408637+0000","last_peered":"2024-01-15T18:19:45.408637+0000","last_clean":"2024-01-15T18:19:45.408637+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:19:45.408637+0000","last_undegraded":"2024-01-15T18:19:45.408637+0000","last_fullsized":"2024-01-15T18:19:45.408637+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:41:56.525500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.398127+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:19:45.398127+0000","last_peered":"2024-01-15T18:19:45.398127+0000","last_clean":"2024-01-15T18:19:45.398127+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:19:45.398127+0000","last_undegraded":"2024-01-15T18:19:45.398127+0000","last_fullsized":"2024-01-15T18:19:45.398127+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.726565+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:19:45.726565+0000","last_peered":"2024-01-15T18:19:45.726565+0000","last_clean":"2024-01-15T18:19:45.726565+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:19:45.726565+0000","last_undegraded":"2024-01-15T18:19:45.726565+0000","last_fullsized":"2024-01-15T18:19:45.726565+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:29:02.127204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'47","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.827721+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:19:45.827721+0000","last_peered":"2024-01-15T18:19:45.827721+0000","last_clean":"2024-01-15T18:19:45.827721+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:19:45.827721+0000","last_undegraded":"2024-01-15T18:19:45.827721+0000","last_fullsized":"2024-01-15T18:19:45.827721+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:45:01.753450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'47","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.524665+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:19:45.524665+0000","last_peered":"2024-01-15T18:19:45.524665+0000","last_clean":"2024-01-15T18:19:45.524665+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:19:45.524665+0000","last_undegraded":"2024-01-15T18:19:45.524665+0000","last_fullsized":"2024-01-15T18:19:45.524665+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:17:21.476309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'48","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.634818+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:19:45.634818+0000","last_peered":"2024-01-15T18:19:45.634818+0000","last_clean":"2024-01-15T18:19:45.634818+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:19:45.634818+0000","last_undegraded":"2024-01-15T18:19:45.634818+0000","last_fullsized":"2024-01-15T18:19:45.634818+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:54:26.940050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'38","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.393963+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:19:45.393963+0000","last_peered":"2024-01-15T18:19:45.393963+0000","last_clean":"2024-01-15T18:19:45.393963+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:19:45.393963+0000","last_undegraded":"2024-01-15T18:19:45.393963+0000","last_fullsized":"2024-01-15T18:19:45.393963+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'51","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.775084+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:19:45.775084+0000","last_peered":"2024-01-15T18:19:45.775084+0000","last_clean":"2024-01-15T18:19:45.775084+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:19:45.775084+0000","last_undegraded":"2024-01-15T18:19:45.775084+0000","last_fullsized":"2024-01-15T18:19:45.775084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'34","reported_seq":53,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.681649+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:19:45.681649+0000","last_peered":"2024-01-15T18:19:45.681649+0000","last_clean":"2024-01-15T18:19:45.681649+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:19:45.681649+0000","last_undegraded":"2024-01-15T18:19:45.681649+0000","last_fullsized":"2024-01-15T18:19:45.681649+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:23:40.618824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'42","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.390578+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:19:45.390578+0000","last_peered":"2024-01-15T18:19:45.390578+0000","last_clean":"2024-01-15T18:19:45.390578+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:19:45.390578+0000","last_undegraded":"2024-01-15T18:19:45.390578+0000","last_fullsized":"2024-01-15T18:19:45.390578+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:21:45.606546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.458896+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:19:45.458896+0000","last_peered":"2024-01-15T18:19:45.458896+0000","last_clean":"2024-01-15T18:19:45.458896+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:19:45.458896+0000","last_undegraded":"2024-01-15T18:19:45.458896+0000","last_fullsized":"2024-01-15T18:19:45.458896+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:25:41.211159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'48","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.699878+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:19:45.699878+0000","last_peered":"2024-01-15T18:19:45.699878+0000","last_clean":"2024-01-15T18:19:45.699878+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:19:45.699878+0000","last_undegraded":"2024-01-15T18:19:45.699878+0000","last_fullsized":"2024-01-15T18:19:45.699878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.465695+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:19:45.465695+0000","last_peered":"2024-01-15T18:19:45.465695+0000","last_clean":"2024-01-15T18:19:45.465695+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:19:45.465695+0000","last_undegraded":"2024-01-15T18:19:45.465695+0000","last_fullsized":"2024-01-15T18:19:45.465695+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.700450+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:19:45.700450+0000","last_peered":"2024-01-15T18:19:45.700450+0000","last_clean":"2024-01-15T18:19:45.700450+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:19:45.700450+0000","last_undegraded":"2024-01-15T18:19:45.700450+0000","last_fullsized":"2024-01-15T18:19:45.700450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:54.104396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'56","reported_seq":84,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.427319+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:19:45.427319+0000","last_peered":"2024-01-15T18:19:45.427319+0000","last_clean":"2024-01-15T18:19:45.427319+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:19:45.427319+0000","last_undegraded":"2024-01-15T18:19:45.427319+0000","last_fullsized":"2024-01-15T18:19:45.427319+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:13:14.187755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.823341+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:19:45.823341+0000","last_peered":"2024-01-15T18:19:45.823341+0000","last_clean":"2024-01-15T18:19:45.823341+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:19:45.823341+0000","last_undegraded":"2024-01-15T18:19:45.823341+0000","last_fullsized":"2024-01-15T18:19:45.823341+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'39","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.605337+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:19:45.605337+0000","last_peered":"2024-01-15T18:19:45.605337+0000","last_clean":"2024-01-15T18:19:45.605337+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:19:45.605337+0000","last_undegraded":"2024-01-15T18:19:45.605337+0000","last_fullsized":"2024-01-15T18:19:45.605337+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:27:42.573183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'43","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.704152+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:19:45.704152+0000","last_peered":"2024-01-15T18:19:45.704152+0000","last_clean":"2024-01-15T18:19:45.704152+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:19:45.704152+0000","last_undegraded":"2024-01-15T18:19:45.704152+0000","last_fullsized":"2024-01-15T18:19:45.704152+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:29:49.217888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'38","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.411545+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:19:45.411545+0000","last_peered":"2024-01-15T18:19:45.411545+0000","last_clean":"2024-01-15T18:19:45.411545+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:19:45.411545+0000","last_undegraded":"2024-01-15T18:19:45.411545+0000","last_fullsized":"2024-01-15T18:19:45.411545+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:39:57.575686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'53","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.722921+0000","last_change":"2024-01-15T18:14:16.877022+0000","last_active":"2024-01-15T18:19:45.722921+0000","last_peered":"2024-01-15T18:19:45.722921+0000","last_clean":"2024-01-15T18:19:45.722921+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:45.722921+0000","last_undegraded":"2024-01-15T18:19:45.722921+0000","last_fullsized":"2024-01-15T18:19:45.722921+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:46:55.334726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'37","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.383134+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:19:45.383134+0000","last_peered":"2024-01-15T18:19:45.383134+0000","last_clean":"2024-01-15T18:19:45.383134+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:19:45.383134+0000","last_undegraded":"2024-01-15T18:19:45.383134+0000","last_fullsized":"2024-01-15T18:19:45.383134+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:06.761673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'38","reported_seq":56,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.625653+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:19:45.625653+0000","last_peered":"2024-01-15T18:19:45.625653+0000","last_clean":"2024-01-15T18:19:45.625653+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:19:45.625653+0000","last_undegraded":"2024-01-15T18:19:45.625653+0000","last_fullsized":"2024-01-15T18:19:45.625653+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'46","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.705265+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:19:45.705265+0000","last_peered":"2024-01-15T18:19:45.705265+0000","last_clean":"2024-01-15T18:19:45.705265+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:19:45.705265+0000","last_undegraded":"2024-01-15T18:19:45.705265+0000","last_fullsized":"2024-01-15T18:19:45.705265+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:38:43.567690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'41","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.642104+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:19:45.642104+0000","last_peered":"2024-01-15T18:19:45.642104+0000","last_clean":"2024-01-15T18:19:45.642104+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:19:45.642104+0000","last_undegraded":"2024-01-15T18:19:45.642104+0000","last_fullsized":"2024-01-15T18:19:45.642104+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:15:10.078594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'62","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.411970+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:19:45.411970+0000","last_peered":"2024-01-15T18:19:45.411970+0000","last_clean":"2024-01-15T18:19:45.411970+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:19:45.411970+0000","last_undegraded":"2024-01-15T18:19:45.411970+0000","last_fullsized":"2024-01-15T18:19:45.411970+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:43.564194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'54","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.560995+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:19:45.560995+0000","last_peered":"2024-01-15T18:19:45.560995+0000","last_clean":"2024-01-15T18:19:45.560995+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:19:45.560995+0000","last_undegraded":"2024-01-15T18:19:45.560995+0000","last_fullsized":"2024-01-15T18:19:45.560995+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:39.523314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'34","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.377219+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:19:45.377219+0000","last_peered":"2024-01-15T18:19:45.377219+0000","last_clean":"2024-01-15T18:19:45.377219+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:19:45.377219+0000","last_undegraded":"2024-01-15T18:19:45.377219+0000","last_fullsized":"2024-01-15T18:19:45.377219+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:27.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":20,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.725727+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:16:24.725727+0000","last_peered":"2024-01-15T18:16:24.725727+0000","last_clean":"2024-01-15T18:16:24.725727+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:16:24.725727+0000","last_undegraded":"2024-01-15T18:16:24.725727+0000","last_fullsized":"2024-01-15T18:16:24.725727+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:00:05.748126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.412953+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:19:45.412953+0000","last_peered":"2024-01-15T18:19:45.412953+0000","last_clean":"2024-01-15T18:19:45.412953+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:19:45.412953+0000","last_undegraded":"2024-01-15T18:19:45.412953+0000","last_fullsized":"2024-01-15T18:19:45.412953+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:14.988989+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:16:14.988989+0000","last_peered":"2024-01-15T18:16:14.988989+0000","last_clean":"2024-01-15T18:16:14.988989+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:16:14.988989+0000","last_undegraded":"2024-01-15T18:16:14.988989+0000","last_fullsized":"2024-01-15T18:16:14.988989+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'43","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.532353+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:19:45.532353+0000","last_peered":"2024-01-15T18:19:45.532353+0000","last_clean":"2024-01-15T18:19:45.532353+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:19:45.532353+0000","last_undegraded":"2024-01-15T18:19:45.532353+0000","last_fullsized":"2024-01-15T18:19:45.532353+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:34.316985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'3","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.687049+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:16:49.687049+0000","last_peered":"2024-01-15T18:16:49.687049+0000","last_clean":"2024-01-15T18:16:49.687049+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:16:49.687049+0000","last_undegraded":"2024-01-15T18:16:49.687049+0000","last_fullsized":"2024-01-15T18:16:49.687049+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:13.139067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'56","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.396172+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:19:45.396172+0000","last_peered":"2024-01-15T18:19:45.396172+0000","last_clean":"2024-01-15T18:19:45.396172+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:19:45.396172+0000","last_undegraded":"2024-01-15T18:19:45.396172+0000","last_fullsized":"2024-01-15T18:19:45.396172+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.687023+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:16:49.687023+0000","last_peered":"2024-01-15T18:16:49.687023+0000","last_clean":"2024-01-15T18:16:49.687023+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:16:49.687023+0000","last_undegraded":"2024-01-15T18:16:49.687023+0000","last_fullsized":"2024-01-15T18:16:49.687023+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:34:36.398718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'39","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.786680+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:19:45.786680+0000","last_peered":"2024-01-15T18:19:45.786680+0000","last_clean":"2024-01-15T18:19:45.786680+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:19:45.786680+0000","last_undegraded":"2024-01-15T18:19:45.786680+0000","last_fullsized":"2024-01-15T18:19:45.786680+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:31:40.500735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.731109+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:16:49.731109+0000","last_peered":"2024-01-15T18:16:49.731109+0000","last_clean":"2024-01-15T18:16:49.731109+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:16:49.731109+0000","last_undegraded":"2024-01-15T18:16:49.731109+0000","last_fullsized":"2024-01-15T18:16:49.731109+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:28:33.303438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'49","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.613573+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:19:45.613573+0000","last_peered":"2024-01-15T18:19:45.613573+0000","last_clean":"2024-01-15T18:19:45.613573+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:19:45.613573+0000","last_undegraded":"2024-01-15T18:19:45.613573+0000","last_fullsized":"2024-01-15T18:19:45.613573+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:30:42.076999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:49.704971+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:16:49.704971+0000","last_peered":"2024-01-15T18:16:49.704971+0000","last_clean":"2024-01-15T18:16:49.704971+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:16:49.704971+0000","last_undegraded":"2024-01-15T18:16:49.704971+0000","last_fullsized":"2024-01-15T18:16:49.704971+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:50.552071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'53","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.610017+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:19:45.610017+0000","last_peered":"2024-01-15T18:19:45.610017+0000","last_clean":"2024-01-15T18:19:45.610017+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:19:45.610017+0000","last_undegraded":"2024-01-15T18:19:45.610017+0000","last_fullsized":"2024-01-15T18:19:45.610017+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:04:04.569141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.818752+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:16:44.818752+0000","last_peered":"2024-01-15T18:16:44.818752+0000","last_clean":"2024-01-15T18:16:44.818752+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:16:44.818752+0000","last_undegraded":"2024-01-15T18:16:44.818752+0000","last_fullsized":"2024-01-15T18:16:44.818752+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:59:27.833018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'43","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407291+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:19:45.407291+0000","last_peered":"2024-01-15T18:19:45.407291+0000","last_clean":"2024-01-15T18:19:45.407291+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:19:45.407291+0000","last_undegraded":"2024-01-15T18:19:45.407291+0000","last_fullsized":"2024-01-15T18:19:45.407291+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:18:54.449125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.729197+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:16:44.729197+0000","last_peered":"2024-01-15T18:16:44.729197+0000","last_clean":"2024-01-15T18:16:44.729197+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:16:44.729197+0000","last_undegraded":"2024-01-15T18:16:44.729197+0000","last_fullsized":"2024-01-15T18:16:44.729197+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:20.253583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'42","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.391592+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:19:45.391592+0000","last_peered":"2024-01-15T18:19:45.391592+0000","last_clean":"2024-01-15T18:19:45.391592+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:19:45.391592+0000","last_undegraded":"2024-01-15T18:19:45.391592+0000","last_fullsized":"2024-01-15T18:19:45.391592+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:26.558597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.686133+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:16:44.686133+0000","last_peered":"2024-01-15T18:16:44.686133+0000","last_clean":"2024-01-15T18:16:44.686133+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:16:44.686133+0000","last_undegraded":"2024-01-15T18:16:44.686133+0000","last_fullsized":"2024-01-15T18:16:44.686133+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:12:18.237949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'40","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.640005+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:19:45.640005+0000","last_peered":"2024-01-15T18:19:45.640005+0000","last_clean":"2024-01-15T18:19:45.640005+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:19:45.640005+0000","last_undegraded":"2024-01-15T18:19:45.640005+0000","last_fullsized":"2024-01-15T18:19:45.640005+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:44.704016+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:16:44.704016+0000","last_peered":"2024-01-15T18:16:44.704016+0000","last_clean":"2024-01-15T18:16:44.704016+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:16:44.704016+0000","last_undegraded":"2024-01-15T18:16:44.704016+0000","last_fullsized":"2024-01-15T18:16:44.704016+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'43","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741116+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:19:45.741116+0000","last_peered":"2024-01-15T18:19:45.741116+0000","last_clean":"2024-01-15T18:19:45.741116+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:19:45.741116+0000","last_undegraded":"2024-01-15T18:19:45.741116+0000","last_fullsized":"2024-01-15T18:19:45.741116+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'13","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:40.230861+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:17:40.230861+0000","last_peered":"2024-01-15T18:17:40.230861+0000","last_clean":"2024-01-15T18:17:40.230861+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:17:40.230861+0000","last_undegraded":"2024-01-15T18:17:40.230861+0000","last_fullsized":"2024-01-15T18:17:40.230861+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:01:08.327524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'43","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404696+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:19:45.404696+0000","last_peered":"2024-01-15T18:19:45.404696+0000","last_clean":"2024-01-15T18:19:45.404696+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:19:45.404696+0000","last_undegraded":"2024-01-15T18:19:45.404696+0000","last_fullsized":"2024-01-15T18:19:45.404696+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:40.620750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'186","reported_seq":202,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.420919+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:19:45.420919+0000","last_peered":"2024-01-15T18:19:45.420919+0000","last_clean":"2024-01-15T18:19:45.420919+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:19:45.420919+0000","last_undegraded":"2024-01-15T18:19:45.420919+0000","last_fullsized":"2024-01-15T18:19:45.420919+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:35:13.261848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":631739,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":185,"num_write_kb":705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'44","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410217+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:19:45.410217+0000","last_peered":"2024-01-15T18:19:45.410217+0000","last_clean":"2024-01-15T18:19:45.410217+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:19:45.410217+0000","last_undegraded":"2024-01-15T18:19:45.410217+0000","last_fullsized":"2024-01-15T18:19:45.410217+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.672899+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:16:39.672899+0000","last_peered":"2024-01-15T18:16:39.672899+0000","last_clean":"2024-01-15T18:16:39.672899+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:16:39.672899+0000","last_undegraded":"2024-01-15T18:16:39.672899+0000","last_fullsized":"2024-01-15T18:16:39.672899+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:11:13.944366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'52","reported_seq":72,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.597854+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:19:45.597854+0000","last_peered":"2024-01-15T18:19:45.597854+0000","last_clean":"2024-01-15T18:19:45.597854+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:19:45.597854+0000","last_undegraded":"2024-01-15T18:19:45.597854+0000","last_fullsized":"2024-01-15T18:19:45.597854+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.792106+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:16:39.792106+0000","last_peered":"2024-01-15T18:16:39.792106+0000","last_clean":"2024-01-15T18:16:39.792106+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:16:39.792106+0000","last_undegraded":"2024-01-15T18:16:39.792106+0000","last_fullsized":"2024-01-15T18:16:39.792106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'46","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559915+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:19:45.559915+0000","last_peered":"2024-01-15T18:19:45.559915+0000","last_clean":"2024-01-15T18:19:45.559915+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:19:45.559915+0000","last_undegraded":"2024-01-15T18:19:45.559915+0000","last_fullsized":"2024-01-15T18:19:45.559915+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:36:57.496147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":24,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.684911+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:16:39.684911+0000","last_peered":"2024-01-15T18:16:39.684911+0000","last_clean":"2024-01-15T18:16:39.684911+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:16:39.684911+0000","last_undegraded":"2024-01-15T18:16:39.684911+0000","last_fullsized":"2024-01-15T18:16:39.684911+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:33:57.950697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'24","reported_seq":43,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.533839+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:19:45.533839+0000","last_peered":"2024-01-15T18:19:45.533839+0000","last_clean":"2024-01-15T18:19:45.533839+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:19:45.533839+0000","last_undegraded":"2024-01-15T18:19:45.533839+0000","last_fullsized":"2024-01-15T18:19:45.533839+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:00:31.047674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.727345+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:16:34.727345+0000","last_peered":"2024-01-15T18:16:34.727345+0000","last_clean":"2024-01-15T18:16:34.727345+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:16:34.727345+0000","last_undegraded":"2024-01-15T18:16:34.727345+0000","last_fullsized":"2024-01-15T18:16:34.727345+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:48:42.125667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'36","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.633679+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:19:45.633679+0000","last_peered":"2024-01-15T18:19:45.633679+0000","last_clean":"2024-01-15T18:19:45.633679+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:19:45.633679+0000","last_undegraded":"2024-01-15T18:19:45.633679+0000","last_fullsized":"2024-01-15T18:19:45.633679+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:25:06.662207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.684130+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:16:34.684130+0000","last_peered":"2024-01-15T18:16:34.684130+0000","last_clean":"2024-01-15T18:16:34.684130+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:16:34.684130+0000","last_undegraded":"2024-01-15T18:16:34.684130+0000","last_fullsized":"2024-01-15T18:16:34.684130+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:18:07.511362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'44","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559602+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:19:45.559602+0000","last_peered":"2024-01-15T18:19:45.559602+0000","last_clean":"2024-01-15T18:19:45.559602+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:19:45.559602+0000","last_undegraded":"2024-01-15T18:19:45.559602+0000","last_fullsized":"2024-01-15T18:19:45.559602+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.791118+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:16:34.791118+0000","last_peered":"2024-01-15T18:16:34.791118+0000","last_clean":"2024-01-15T18:16:34.791118+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:16:34.791118+0000","last_undegraded":"2024-01-15T18:16:34.791118+0000","last_fullsized":"2024-01-15T18:16:34.791118+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'41","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.673339+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:19:45.673339+0000","last_peered":"2024-01-15T18:19:45.673339+0000","last_clean":"2024-01-15T18:19:45.673339+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:19:45.673339+0000","last_undegraded":"2024-01-15T18:19:45.673339+0000","last_fullsized":"2024-01-15T18:19:45.673339+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.727774+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:16:34.727774+0000","last_peered":"2024-01-15T18:16:34.727774+0000","last_clean":"2024-01-15T18:16:34.727774+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:16:34.727774+0000","last_undegraded":"2024-01-15T18:16:34.727774+0000","last_fullsized":"2024-01-15T18:16:34.727774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:34:26.805060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.866450+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:19:45.866450+0000","last_peered":"2024-01-15T18:19:45.866450+0000","last_clean":"2024-01-15T18:19:45.866450+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:19:45.866450+0000","last_undegraded":"2024-01-15T18:19:45.866450+0000","last_fullsized":"2024-01-15T18:19:45.866450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'3","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.671975+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:16:34.671975+0000","last_peered":"2024-01-15T18:16:34.671975+0000","last_clean":"2024-01-15T18:16:34.671975+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:16:34.671975+0000","last_undegraded":"2024-01-15T18:16:34.671975+0000","last_fullsized":"2024-01-15T18:16:34.671975+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:01:41.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'43","reported_seq":74,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.389233+0000","last_change":"2024-01-15T18:15:20.196078+0000","last_active":"2024-01-15T18:19:45.389233+0000","last_peered":"2024-01-15T18:19:45.389233+0000","last_clean":"2024-01-15T18:19:45.389233+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:19:45.389233+0000","last_undegraded":"2024-01-15T18:19:45.389233+0000","last_fullsized":"2024-01-15T18:19:45.389233+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:19.862117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.683005+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:16:29.683005+0000","last_peered":"2024-01-15T18:16:29.683005+0000","last_clean":"2024-01-15T18:16:29.683005+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:16:29.683005+0000","last_undegraded":"2024-01-15T18:16:29.683005+0000","last_fullsized":"2024-01-15T18:16:29.683005+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:53:40.627022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.587657+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:19:45.587657+0000","last_peered":"2024-01-15T18:19:45.587657+0000","last_clean":"2024-01-15T18:19:45.587657+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:19:45.587657+0000","last_undegraded":"2024-01-15T18:19:45.587657+0000","last_fullsized":"2024-01-15T18:19:45.587657+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.671013+0000","last_change":"2024-01-15T18:14:15.869602+0000","last_active":"2024-01-15T18:16:29.671013+0000","last_peered":"2024-01-15T18:16:29.671013+0000","last_clean":"2024-01-15T18:16:29.671013+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:16:29.671013+0000","last_undegraded":"2024-01-15T18:16:29.671013+0000","last_fullsized":"2024-01-15T18:16:29.671013+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:48:53.365727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'42","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.765594+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:19:45.765594+0000","last_peered":"2024-01-15T18:19:45.765594+0000","last_clean":"2024-01-15T18:19:45.765594+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:19:45.765594+0000","last_undegraded":"2024-01-15T18:19:45.765594+0000","last_fullsized":"2024-01-15T18:19:45.765594+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.725995+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:16:29.725995+0000","last_peered":"2024-01-15T18:16:29.725995+0000","last_clean":"2024-01-15T18:16:29.725995+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:16:29.725995+0000","last_undegraded":"2024-01-15T18:16:29.725995+0000","last_fullsized":"2024-01-15T18:16:29.725995+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:59:03.677619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'36","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559183+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:19:45.559183+0000","last_peered":"2024-01-15T18:19:45.559183+0000","last_clean":"2024-01-15T18:19:45.559183+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:19:45.559183+0000","last_undegraded":"2024-01-15T18:19:45.559183+0000","last_fullsized":"2024-01-15T18:19:45.559183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:29.700861+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:16:29.700861+0000","last_peered":"2024-01-15T18:16:29.700861+0000","last_clean":"2024-01-15T18:16:29.700861+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:16:29.700861+0000","last_undegraded":"2024-01-15T18:16:29.700861+0000","last_fullsized":"2024-01-15T18:16:29.700861+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:30:58.175082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'41","reported_seq":81,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410772+0000","last_change":"2024-01-15T18:15:20.356060+0000","last_active":"2024-01-15T18:19:45.410772+0000","last_peered":"2024-01-15T18:19:45.410772+0000","last_clean":"2024-01-15T18:19:45.410772+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:19:45.410772+0000","last_undegraded":"2024-01-15T18:19:45.410772+0000","last_fullsized":"2024-01-15T18:19:45.410772+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:40:10.891015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'18","reported_seq":34,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:51.467225+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:18:51.467225+0000","last_peered":"2024-01-15T18:18:51.467225+0000","last_clean":"2024-01-15T18:18:51.467225+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:18:51.467225+0000","last_undegraded":"2024-01-15T18:18:51.467225+0000","last_fullsized":"2024-01-15T18:18:51.467225+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":38,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'46","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.536354+0000","last_change":"2024-01-15T18:14:16.883515+0000","last_active":"2024-01-15T18:19:45.536354+0000","last_peered":"2024-01-15T18:19:45.536354+0000","last_clean":"2024-01-15T18:19:45.536354+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:45.536354+0000","last_undegraded":"2024-01-15T18:19:45.536354+0000","last_fullsized":"2024-01-15T18:19:45.536354+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:26:05.836408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.724998+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:16:24.724998+0000","last_peered":"2024-01-15T18:16:24.724998+0000","last_clean":"2024-01-15T18:16:24.724998+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:16:24.724998+0000","last_undegraded":"2024-01-15T18:16:24.724998+0000","last_fullsized":"2024-01-15T18:16:24.724998+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:57:20.509922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'39","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404370+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:19:45.404370+0000","last_peered":"2024-01-15T18:19:45.404370+0000","last_clean":"2024-01-15T18:19:45.404370+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:19:45.404370+0000","last_undegraded":"2024-01-15T18:19:45.404370+0000","last_fullsized":"2024-01-15T18:19:45.404370+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.699700+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:16:24.699700+0000","last_peered":"2024-01-15T18:16:24.699700+0000","last_clean":"2024-01-15T18:16:24.699700+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:16:24.699700+0000","last_undegraded":"2024-01-15T18:16:24.699700+0000","last_fullsized":"2024-01-15T18:16:24.699700+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:00:41.388899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'44","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.695729+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:19:45.695729+0000","last_peered":"2024-01-15T18:19:45.695729+0000","last_clean":"2024-01-15T18:19:45.695729+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:19:45.695729+0000","last_undegraded":"2024-01-15T18:19:45.695729+0000","last_fullsized":"2024-01-15T18:19:45.695729+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:55.624199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.725579+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:16:24.725579+0000","last_peered":"2024-01-15T18:16:24.725579+0000","last_clean":"2024-01-15T18:16:24.725579+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:16:24.725579+0000","last_undegraded":"2024-01-15T18:16:24.725579+0000","last_fullsized":"2024-01-15T18:16:24.725579+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:52:12.317803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'56","reported_seq":79,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.774102+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:19:45.774102+0000","last_peered":"2024-01-15T18:19:45.774102+0000","last_clean":"2024-01-15T18:19:45.774102+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:19:45.774102+0000","last_undegraded":"2024-01-15T18:19:45.774102+0000","last_fullsized":"2024-01-15T18:19:45.774102+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:45:50.222496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:24.814331+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:16:24.814331+0000","last_peered":"2024-01-15T18:16:24.814331+0000","last_clean":"2024-01-15T18:16:24.814331+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:16:24.814331+0000","last_undegraded":"2024-01-15T18:16:24.814331+0000","last_fullsized":"2024-01-15T18:16:24.814331+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:02:52.869650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741049+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:19:45.741049+0000","last_peered":"2024-01-15T18:19:45.741049+0000","last_clean":"2024-01-15T18:19:45.741049+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:19:45.741049+0000","last_undegraded":"2024-01-15T18:19:45.741049+0000","last_fullsized":"2024-01-15T18:19:45.741049+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'11","reported_seq":37,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:40.222936+0000","last_change":"2024-01-15T18:14:15.875690+0000","last_active":"2024-01-15T18:17:40.222936+0000","last_peered":"2024-01-15T18:17:40.222936+0000","last_clean":"2024-01-15T18:17:40.222936+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:17:40.222936+0000","last_undegraded":"2024-01-15T18:17:40.222936+0000","last_fullsized":"2024-01-15T18:17:40.222936+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:55:36.483295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'39","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.718813+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:19:45.718813+0000","last_peered":"2024-01-15T18:19:45.718813+0000","last_clean":"2024-01-15T18:19:45.718813+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:19:45.718813+0000","last_undegraded":"2024-01-15T18:19:45.718813+0000","last_fullsized":"2024-01-15T18:19:45.718813+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:56:58.082932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:19.723501+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:16:19.723501+0000","last_peered":"2024-01-15T18:16:19.723501+0000","last_clean":"2024-01-15T18:16:19.723501+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:16:19.723501+0000","last_undegraded":"2024-01-15T18:16:19.723501+0000","last_fullsized":"2024-01-15T18:16:19.723501+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:55:09.132487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'31","reported_seq":54,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.579344+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:19:45.579344+0000","last_peered":"2024-01-15T18:19:45.579344+0000","last_clean":"2024-01-15T18:19:45.579344+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:19:45.579344+0000","last_undegraded":"2024-01-15T18:19:45.579344+0000","last_fullsized":"2024-01-15T18:19:45.579344+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:18.532202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.728790+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:16:39.728790+0000","last_peered":"2024-01-15T18:16:39.728790+0000","last_clean":"2024-01-15T18:16:39.728790+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:16:39.728790+0000","last_undegraded":"2024-01-15T18:16:39.728790+0000","last_fullsized":"2024-01-15T18:16:39.728790+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:56:42.925319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":254,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:30.003748+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:17:30.003748+0000","last_peered":"2024-01-15T18:17:30.003748+0000","last_clean":"2024-01-15T18:17:30.003748+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:17:30.003748+0000","last_undegraded":"2024-01-15T18:17:30.003748+0000","last_fullsized":"2024-01-15T18:17:30.003748+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2810,"ondisk_log_size":2810,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":633858,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":281,"num_write_kb":746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1335296,"data_stored":1267716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":260,"ondisk_log_size":260,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542415,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.296,"15min":1.325},"min":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.54000000000000004},"max":{"1min":1.782,"5min":2.1200000000000001,"15min":9.1379999999999999},"last":1.534},{"interface":"front","average":{"1min":1.268,"5min":1.385,"15min":1.528},"min":{"1min":0.88400000000000001,"5min":0.81999999999999995,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":1.905,"15min":27.32},"last":1.1100000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2210000000000001,"15min":1.4530000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.728,"5min":2.1000000000000001,"15min":29.167999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.21,"5min":1.24,"15min":1.3260000000000001},"min":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.52800000000000002},"max":{"1min":1.597,"5min":2.6179999999999999,"15min":8.5980000000000008},"last":1.1759999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3740000000000001,"15min":1.4179999999999999},"min":{"1min":0.94699999999999995,"5min":0.874,"15min":0.56000000000000005},"max":{"1min":1.7709999999999999,"5min":2.3900000000000001,"15min":12.709},"last":1.393},{"interface":"front","average":{"1min":1.141,"5min":1.256,"15min":1.488},"min":{"1min":0.79400000000000004,"5min":0.72499999999999998,"15min":0.56599999999999995},"max":{"1min":1.6539999999999999,"5min":1.8200000000000001,"15min":27.358000000000001},"last":0.629}]},{"osd":3,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.556},"min":{"1min":0.98599999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.643,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.379,"5min":1.4530000000000001,"15min":1.5189999999999999},"min":{"1min":1.01,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.7769999999999999,"5min":2.4940000000000002,"15min":16.088999999999999},"last":1.4850000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.381,"15min":1.5269999999999999},"min":{"1min":0.94899999999999995,"5min":0.64700000000000002,"15min":0.50700000000000001},"max":{"1min":1.681,"5min":6.0350000000000001,"15min":27.219000000000001},"last":1.23},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4339999999999999,"15min":1.641},"min":{"1min":0.90400000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7629999999999999,"5min":3.536,"15min":29.25},"last":1.2849999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4730000000000001,"15min":1.573},"min":{"1min":1.0209999999999999,"5min":0.68799999999999994,"15min":0.67000000000000004},"max":{"1min":1.8660000000000001,"5min":2.7120000000000002,"15min":11.834},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.5129999999999999,"15min":1.675},"min":{"1min":0.92900000000000005,"5min":0.71199999999999997,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.343}]},{"osd":6,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.635,"15min":1.8200000000000001},"min":{"1min":1.1850000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.929,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5409999999999999,"15min":1.8049999999999999},"min":{"1min":1.0289999999999999,"5min":0.63100000000000001,"15min":0.57299999999999995},"max":{"1min":1.841,"5min":2.754,"15min":29.106999999999999},"last":1.6719999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542415,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.2190000000000001,"15min":1.236},"min":{"1min":0.72299999999999998,"5min":0.68200000000000005,"15min":0.438},"max":{"1min":1.5409999999999999,"5min":2.77,"15min":8.8350000000000009},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1970000000000001,"15min":1.27},"min":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.59599999999999997},"max":{"1min":1.643,"5min":2.673,"15min":9.548},"last":1.0609999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2729999999999999,"15min":1.395},"min":{"1min":0.89300000000000002,"5min":0.70999999999999996,"15min":0.52100000000000002},"max":{"1min":1.5209999999999999,"5min":2.8700000000000001,"15min":13.269},"last":1.45},{"interface":"front","average":{"1min":1.262,"5min":1.3009999999999999,"15min":1.4119999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.57799999999999996},"max":{"1min":1.6990000000000001,"5min":3.1080000000000001,"15min":13.984999999999999},"last":0.92700000000000005}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.232,"15min":1.284},"min":{"1min":0.85299999999999998,"5min":0.79000000000000004,"15min":0.69799999999999995},"max":{"1min":1.7,"5min":1.9379999999999999,"15min":7.0030000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.321,"5min":1.373,"15min":1.425},"min":{"1min":0.81100000000000005,"5min":0.73899999999999999,"15min":0.49399999999999999},"max":{"1min":1.623,"5min":2.6909999999999998,"15min":8.9359999999999999},"last":1.2749999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.444,"15min":1.55},"min":{"1min":1.131,"5min":1.0089999999999999,"15min":0.68500000000000005},"max":{"1min":1.7629999999999999,"5min":3.339,"15min":16.855},"last":1.363},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.413,"15min":1.5209999999999999},"min":{"1min":1.111,"5min":1.046,"15min":0.628},"max":{"1min":1.657,"5min":2.0139999999999998,"15min":10.875},"last":1.1140000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.401,"15min":1.45},"min":{"1min":0.86699999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":1.645,"5min":3.0299999999999998,"15min":8.7539999999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.427,"15min":1.4970000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.819,"5min":5.125,"15min":10.118},"last":1.242}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3600000000000001,"15min":1.5209999999999999},"min":{"1min":0.92500000000000004,"5min":0.88,"15min":0.60299999999999998},"max":{"1min":1.8120000000000001,"5min":2.3210000000000002,"15min":16.309999999999999},"last":1.016},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5429999999999999,"15min":1.6200000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.66000000000000003},"max":{"1min":1.8029999999999999,"5min":3.0209999999999999,"15min":10.077},"last":1.3899999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5289999999999999,"15min":1.6279999999999999},"min":{"1min":0.995,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":1.881,"5min":3.2450000000000001,"15min":10.302},"last":1.431},{"interface":"front","average":{"1min":1.504,"5min":1.663,"15min":1.6879999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":1.9059999999999999,"5min":7.6799999999999997,"15min":9.4079999999999995},"last":1.2989999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542417,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928972,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93442868,"statfs":{"total":96636764160,"available":95685496832,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2609999999999999,"15min":1.399},"min":{"1min":0.79300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.6180000000000001,"5min":4.1829999999999998,"15min":26.077999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3740000000000001,"15min":1.4299999999999999},"min":{"1min":1.099,"5min":0.77900000000000003,"15min":0.56399999999999995},"max":{"1min":1.6200000000000001,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.2789999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2549999999999999,"15min":1.4339999999999999},"min":{"1min":0.875,"5min":0.82499999999999996,"15min":0.53300000000000003},"max":{"1min":1.8,"5min":1.923,"15min":26},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.367,"15min":1.401},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.657,"5min":4.0869999999999997,"15min":12.619},"last":1.2050000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3460000000000001,"15min":1.5329999999999999},"min":{"1min":1.0700000000000001,"5min":0.69999999999999996,"15min":0.56699999999999995},"max":{"1min":1.712,"5min":2.6869999999999998,"15min":26.234999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.385,"5min":1.331,"15min":1.532},"min":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.629},"max":{"1min":1.766,"5min":1.8240000000000001,"15min":17.126999999999999},"last":1.105}]},{"osd":3,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3120000000000001,"15min":1.397},"min":{"1min":1.0429999999999999,"5min":0.82399999999999995,"15min":0.67200000000000004},"max":{"1min":1.792,"5min":2.375,"15min":12.722},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4830000000000001,"15min":1.6399999999999999},"min":{"1min":1.046,"5min":0.75900000000000001,"15min":0.67900000000000005},"max":{"1min":1.7989999999999999,"5min":3.238,"15min":26.134},"last":1.0449999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2769999999999999,"15min":1.4139999999999999},"min":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.442},"max":{"1min":1.77,"5min":2.5350000000000001,"15min":9.9979999999999993},"last":1.371},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.458,"15min":1.6930000000000001},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.498},"max":{"1min":1.875,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.4730000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3640000000000001,"15min":1.4339999999999999},"min":{"1min":1.0069999999999999,"5min":0.82299999999999995,"15min":0.45900000000000002},"max":{"1min":1.849,"5min":2.6640000000000001,"15min":12.769},"last":1.52},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5860000000000001,"15min":1.647},"min":{"1min":0.94799999999999995,"5min":0.79800000000000004,"15min":0.49199999999999999},"max":{"1min":1.909,"5min":11.851000000000001,"15min":26.186},"last":1.153}]},{"osd":7,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5960000000000001,"15min":1.798},"min":{"1min":1.1599999999999999,"5min":1.1359999999999999,"15min":0.65700000000000003},"max":{"1min":2.04,"5min":2.3700000000000001,"15min":13.706},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.54,"15min":1.679},"min":{"1min":1.038,"5min":1.0189999999999999,"15min":0.60099999999999998},"max":{"1min":2.0289999999999999,"5min":4.3700000000000001,"15min":21.728000000000002},"last":1.399}]}]},{"osd":4,"up_from":14,"seq":60129542416,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.343,"15min":1.27},"min":{"1min":0.92800000000000005,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.7490000000000001,"5min":13.288,"15min":15.044},"last":1.004},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.284,"15min":1.321},"min":{"1min":0.91600000000000004,"5min":0.71299999999999997,"15min":0.34699999999999998},"max":{"1min":1.5389999999999999,"5min":9.2409999999999997,"15min":11.196},"last":1.393}]},{"osd":1,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2569999999999999,"15min":1.3779999999999999},"min":{"1min":0.90400000000000003,"5min":0.78400000000000003,"15min":0.41499999999999998},"max":{"1min":1.655,"5min":3.5910000000000002,"15min":28.663},"last":1.341},{"interface":"front","average":{"1min":1.278,"5min":1.302,"15min":1.4370000000000001},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.6319999999999999,"5min":3.6739999999999999,"15min":15.096},"last":1.4590000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.333,"15min":1.4319999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.8170000000000002,"15min":16.023},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.381,"15min":1.5169999999999999},"min":{"1min":0.94899999999999995,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.7869999999999999,"5min":3.7930000000000001,"15min":18.858000000000001},"last":1.244}]},{"osd":3,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3029999999999999,"15min":1.393},"min":{"1min":0.80100000000000005,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.847,"5min":5.0599999999999996,"15min":11.449},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.4510000000000001,"15min":1.5720000000000001},"min":{"1min":0.995,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.7969999999999999,"5min":3.7080000000000002,"15min":18.027000000000001},"last":1.4970000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4390000000000001,"15min":1.6220000000000001},"min":{"1min":0.88900000000000001,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.8140000000000001,"5min":7.7439999999999998,"15min":54.938000000000002},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.339,"15min":1.482},"min":{"1min":0.96499999999999997,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":1.857,"5min":2.0960000000000001,"15min":13.031000000000001},"last":1.171}]},{"osd":6,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.4119999999999999,"15min":1.5509999999999999},"min":{"1min":1.1180000000000001,"5min":0.90300000000000002,"15min":0.495},"max":{"1min":1.8779999999999999,"5min":3.766,"15min":16.103999999999999},"last":1.429},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5920000000000001,"15min":1.6799999999999999},"min":{"1min":1.1559999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.909,"5min":8.1609999999999996,"15min":15.118},"last":1.5940000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.45,"15min":1.786},"min":{"1min":1.1919999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.0299999999999998,"5min":2.4689999999999999,"15min":23.390000000000001},"last":1.532},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5609999999999999,"15min":1.756},"min":{"1min":1.2270000000000001,"5min":1.0409999999999999,"15min":0.62},"max":{"1min":1.99,"5min":3.9020000000000001,"15min":16.763000000000002},"last":1.5620000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542417,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462020,"kb_used_data":1386492,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909820,"statfs":{"total":96636764160,"available":95139655680,"internally_reserved":0,"allocated":1419767808,"data_stored":1418939088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95199999999999996,"15min":1.1020000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":1.579,"5min":1.6599999999999999,"15min":36.408000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.101,"15min":1.093},"min":{"1min":0.69799999999999995,"5min":0.63900000000000001,"15min":0.377},"max":{"1min":1.6930000000000001,"5min":2.0230000000000001,"15min":5.0190000000000001},"last":1.171}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.99099999999999999,"15min":1.0640000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.34399999999999997},"max":{"1min":1.756,"5min":1.837,"15min":5.226},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.113,"5min":1.1899999999999999,"15min":1.22},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.669,"5min":2.0609999999999999,"15min":7.2169999999999996},"last":0.749}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2569999999999999,"15min":1.3500000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.637,"5min":2.0150000000000001,"15min":14.025},"last":1.6850000000000001},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.345,"15min":1.3400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.841,"5min":2.403,"15min":3.6240000000000001},"last":0.96199999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.4259999999999999},"min":{"1min":0.84799999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"max":{"1min":1.9350000000000001,"5min":3.1459999999999999,"15min":9.0069999999999997},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.448,"15min":1.6040000000000001},"min":{"1min":1.091,"5min":0.79100000000000004,"15min":0.69499999999999995},"max":{"1min":1.696,"5min":4.8570000000000002,"15min":15.769},"last":1.3360000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.468,"15min":1.579},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65000000000000002},"max":{"1min":1.7490000000000001,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.546},{"interface":"front","average":{"1min":1.468,"5min":1.5169999999999999,"15min":1.7809999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":2.1219999999999999,"5min":2.3959999999999999,"15min":38.832999999999998},"last":1.595}]},{"osd":6,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5469999999999999,"15min":1.7529999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.51400000000000001},"max":{"1min":1.804,"5min":7.1020000000000003,"15min":32.259999999999998},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.5740000000000001,"15min":1.7470000000000001},"min":{"1min":1.0389999999999999,"5min":0.85999999999999999,"15min":0.747},"max":{"1min":2.169,"5min":2.419,"15min":16.195},"last":1.448}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.6719999999999999,"15min":1.905},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":1.466},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.728,"15min":1.8009999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":1.976,"5min":16.734000000000002,"15min":17.622},"last":1.6439999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542416,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.012,"15min":1.0249999999999999},"min":{"1min":0.63500000000000001,"5min":0.44400000000000001,"15min":0.35199999999999998},"max":{"1min":1.5069999999999999,"5min":3.1589999999999998,"15min":10.256},"last":0.71299999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2150000000000001,"15min":1.1819999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.33700000000000002},"max":{"1min":1.8180000000000001,"5min":8.9719999999999995,"15min":10.319000000000001},"last":0.85799999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1,"15min":1.026},"min":{"1min":0.67900000000000005,"5min":0.48899999999999999,"15min":0.38600000000000001},"max":{"1min":1.573,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":1.157},{"interface":"front","average":{"1min":1.097,"5min":1.1890000000000001,"15min":1.2509999999999999},"min":{"1min":0.74099999999999999,"5min":0.59499999999999997,"15min":0.35299999999999998},"max":{"1min":1.5669999999999999,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.3149999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3240000000000001,"15min":1.4219999999999999},"min":{"1min":0.82899999999999996,"5min":0.54200000000000004,"15min":0.44400000000000001},"max":{"1min":1.5529999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.224,"15min":1.306},"min":{"1min":0.89000000000000001,"5min":0.56599999999999995,"15min":0.376},"max":{"1min":1.7689999999999999,"5min":2.7589999999999999,"15min":14.946},"last":1.177}]},{"osd":4,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.536,"15min":1.524},"min":{"1min":1.139,"5min":0.82099999999999995,"15min":0.66100000000000003},"max":{"1min":1.7769999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4990000000000001,"15min":1.5960000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.65300000000000002},"max":{"1min":1.768,"5min":2.867,"15min":10.279999999999999},"last":1.4059999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4770000000000001,"15min":1.5700000000000001},"min":{"1min":1,"5min":1,"15min":0.63500000000000001},"max":{"1min":1.6830000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.111},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.613,"15min":1.6200000000000001},"min":{"1min":1.0209999999999999,"5min":0.88200000000000001,"15min":0.71699999999999997},"max":{"1min":1.895,"5min":10.833,"15min":10.833},"last":1.5229999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.6180000000000001},"min":{"1min":1.0109999999999999,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.827,"5min":2.984,"15min":10.185},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.657,"15min":1.7250000000000001},"min":{"1min":1.107,"5min":1.1060000000000001,"15min":0.79700000000000004},"max":{"1min":1.843,"5min":10.917,"15min":22.382000000000001},"last":1.2030000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.6180000000000001,"15min":1.843},"min":{"1min":1.129,"5min":1.006,"15min":0.77700000000000002},"max":{"1min":1.897,"5min":3.052,"15min":19.986000000000001},"last":1.583},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6659999999999999,"15min":1.77},"min":{"1min":1.252,"5min":0.89300000000000002,"15min":0.877},"max":{"1min":1.931,"5min":3.069,"15min":11.893000000000001},"last":1.29}]}]},{"osd":1,"up_from":14,"seq":60129542416,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1137752,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93234088,"statfs":{"total":96636764160,"available":95471706112,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96599999999999997,"15min":1.1020000000000001},"min":{"1min":0.63300000000000001,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":1.7130000000000001,"5min":3.4510000000000001,"15min":19.218},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.232},"min":{"1min":0.65500000000000003,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.694,"5min":2.0939999999999999,"15min":19.823},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99099999999999999,"15min":1.1100000000000001},"min":{"1min":0.80000000000000004,"5min":0.54400000000000004,"15min":0.31900000000000001},"max":{"1min":1.776,"5min":2.5110000000000001,"15min":19.02},"last":1.603},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.258,"15min":1.401},"min":{"1min":0.85699999999999998,"5min":0.63300000000000001,"15min":0.41599999999999998},"max":{"1min":1.835,"5min":3.1400000000000001,"15min":29.251000000000001},"last":1.4279999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.4990000000000001},"min":{"1min":0.84899999999999998,"5min":0.58499999999999996,"15min":0.40899999999999997},"max":{"1min":1.6779999999999999,"5min":5.1459999999999999,"15min":27.021000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.194,"15min":1.391},"min":{"1min":0.77600000000000002,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.7669999999999999,"5min":3.0150000000000001,"15min":31.218},"last":1.5229999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.492,"15min":1.6890000000000001},"min":{"1min":1.1379999999999999,"5min":0.755,"15min":0.70599999999999996},"max":{"1min":1.8049999999999999,"5min":5.2169999999999996,"15min":27.059999999999999},"last":0.998},{"interface":"front","average":{"1min":1.409,"5min":1.4450000000000001,"15min":1.6359999999999999},"min":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.56499999999999995},"max":{"1min":1.9650000000000001,"5min":3.4159999999999999,"15min":31.433},"last":1.097}]},{"osd":5,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4730000000000001,"15min":1.645},"min":{"1min":1.149,"5min":0.82499999999999996,"15min":0.68500000000000005},"max":{"1min":1.8560000000000001,"5min":3.0950000000000002,"15min":20.012},"last":1.224},{"interface":"front","average":{"1min":1.339,"5min":1.431,"15min":1.52},"min":{"1min":0.97699999999999998,"5min":0.85799999999999998,"15min":0.69299999999999995},"max":{"1min":1.631,"5min":2.9409999999999998,"15min":6.758},"last":1.45}]},{"osd":6,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.639,"15min":1.794},"min":{"1min":1.28,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.9339999999999999,"5min":5.0629999999999997,"15min":29.530000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.575,"15min":1.802},"min":{"1min":1.2030000000000001,"5min":0.88400000000000001,"15min":0.61399999999999999},"max":{"1min":1.845,"5min":3.1960000000000002,"15min":31.186},"last":1.397}]},{"osd":7,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5169999999999999,"15min":1.8140000000000001},"min":{"1min":1.0860000000000001,"5min":0.95899999999999996,"15min":0.63},"max":{"1min":1.8460000000000001,"5min":3.3769999999999998,"15min":26.393999999999998},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.6240000000000001,"15min":1.8380000000000001},"min":{"1min":1.1659999999999999,"5min":0.89200000000000002,"15min":0.76100000000000001},"max":{"1min":1.913,"5min":3.5750000000000002,"15min":27.582000000000001},"last":1.7909999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542415,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1777940,"kb_used_data":1706572,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92593900,"statfs":{"total":96636764160,"available":94816153600,"internally_reserved":0,"allocated":1747529728,"data_stored":1746685443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97699999999999998,"15min":1.141},"min":{"1min":0.57199999999999995,"5min":0.42299999999999999,"15min":0.30599999999999999},"max":{"1min":1.5860000000000001,"5min":5.7560000000000002,"15min":20.437999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.236,"15min":1.278},"min":{"1min":0.77000000000000002,"5min":0.65300000000000002,"15min":0.32400000000000001},"max":{"1min":1.8660000000000001,"5min":6.3600000000000003,"15min":35.137999999999998},"last":1.052}]},{"osd":2,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.236,"15min":1.3879999999999999},"min":{"1min":0.66100000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.716,"5min":11.06,"15min":20.556000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2789999999999999,"15min":1.375},"min":{"1min":0.85099999999999998,"5min":0.60699999999999998,"15min":0.39000000000000001},"max":{"1min":1.8919999999999999,"5min":6.4939999999999998,"15min":35.183},"last":0.60299999999999998}]},{"osd":3,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.238,"15min":1.1819999999999999},"min":{"1min":0.621,"5min":0.55000000000000004,"15min":0.42299999999999999},"max":{"1min":2.0230000000000001,"5min":10.885999999999999,"15min":10.885999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.266,"5min":1.3779999999999999,"15min":1.635},"min":{"1min":0.90000000000000002,"5min":0.73099999999999998,"15min":0.45100000000000001},"max":{"1min":1.621,"5min":9.3339999999999996,"15min":25.372},"last":0.71399999999999997}]},{"osd":4,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.532,"15min":1.621},"min":{"1min":1.0509999999999999,"5min":1.014,"15min":0.67200000000000004},"max":{"1min":1.9710000000000001,"5min":9.9969999999999999,"15min":35.223999999999997},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.581,"15min":1.49},"min":{"1min":1.016,"5min":0.99399999999999999,"15min":0.58799999999999997},"max":{"1min":1.96,"5min":17.567,"15min":17.567},"last":1.5429999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6899999999999999,"15min":1.6499999999999999},"min":{"1min":1.099,"5min":0.92700000000000005,"15min":0.69799999999999995},"max":{"1min":1.9990000000000001,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.679,"15min":1.8300000000000001},"min":{"1min":1.0029999999999999,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.966,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.3759999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.6419999999999999,"15min":1.923},"min":{"1min":1.1879999999999999,"5min":1.004,"15min":0.70599999999999996},"max":{"1min":1.875,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.244},{"interface":"front","average":{"1min":1.595,"5min":1.6619999999999999,"15min":1.8440000000000001},"min":{"1min":1.2130000000000001,"5min":0.90600000000000003,"15min":0.62},"max":{"1min":2.2149999999999999,"5min":6.2270000000000003,"15min":35.255000000000003},"last":1.512}]},{"osd":7,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.704,"15min":1.9319999999999999},"min":{"1min":1.153,"5min":1.089,"15min":0.56200000000000006},"max":{"1min":2.2919999999999998,"5min":9.9429999999999996,"15min":35.636000000000003},"last":1.591},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.6639999999999999,"15min":1.6919999999999999},"min":{"1min":1.1759999999999999,"5min":0.96499999999999997,"15min":0.64000000000000001},"max":{"1min":2.056,"5min":10.943,"15min":11.470000000000001},"last":1.264}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":651264,"data_stored":631851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":631809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:19:53.630 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-15T18:19:53.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.892+0000 7ff444a00640 1 Processor -- start 2024-01-15T18:19:53.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.892+0000 7ff444a00640 1 -- start start 2024-01-15T18:19:53.897 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.894+0000 7ff444a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff440164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.894+0000 7ff444a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400ccbe0 con 0x7ff440143810 2024-01-15T18:19:53.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.894+0000 7ff444a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff440143bf0 con 0x7ff44006e1e0 2024-01-15T18:19:53.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.894+0000 7ff444a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400b3520 con 0x7ff4400ce080 2024-01-15T18:19:53.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.894+0000 7ff43e800640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7ff440143810 0x7ff44006dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55058/0 (socket says 172.21.15.2:55058) 2024-01-15T18:19:53.898 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.894+0000 7ff43e800640 1 -- 172.21.15.2:0/1251478761 learned_addr learned my addr 172.21.15.2:0/1251478761 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43f200640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff440164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011435831 0 0) 0x7ff4400ccbe0 con 0x7ff440143810 2024-01-15T18:19:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff420003520 con 0x7ff440143810 2024-01-15T18:19:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1609283949 0 0) 0x7ff420003520 con 0x7ff440143810 2024-01-15T18:19:53.900 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 msgr2=0x7ff440164c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff440164c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 >> v1:172.21.15.137:6789/0 conn(0x7ff44006e1e0 legacy=0x7ff4400b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4400b2ac0 con 0x7ff440143810 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7ff430001670 con 0x7ff440143810 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 >> v1:172.21.15.2:6789/0 conn(0x7ff440143810 legacy=0x7ff44006dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4200039a0 0x7ff420003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200047c0 0x7ff420004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff4200051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.901 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4400b3520 con 0x7ff4200039a0 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42000bab0 con 0x7ff4200047c0 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.895+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42000de80 con 0x7ff4400ce080 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43e800640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4200039a0 0x7ff420003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43f200640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff4200051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43de00640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200047c0 0x7ff420004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43e800640 1 -- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 msgr2=0x7ff4200051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43e800640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff4200051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43e800640 1 -- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200047c0 msgr2=0x7ff420004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:53.902 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43e800640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200047c0 0x7ff420004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43e800640 1 -- 172.21.15.2:0/1251478761 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420003520 con 0x7ff4200039a0 2024-01-15T18:19:53.903 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.896+0000 7ff43f200640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff4400ce080 0x7ff4200051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:53.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.897+0000 7ff43e800640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4200039a0 0x7ff420003dc0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c240 tx=0x7ff4200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db158c5a5b1052e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:53.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff430005ad0 con 0x7ff4200039a0 2024-01-15T18:19:53.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff430005c50 con 0x7ff4200039a0 2024-01-15T18:19:53.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff43d400640 1 -- 172.21.15.2:0/1251478761 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43002c8b0 con 0x7ff4200039a0 2024-01-15T18:19:53.904 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff444a00640 1 -- 172.21.15.2:0/1251478761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4200039a0 msgr2=0x7ff420003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff444a00640 1 --2- 172.21.15.2:0/1251478761 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff4200039a0 0x7ff420003dc0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7ff43000c240 tx=0x7ff4200057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff43f200640 1 -- 172.21.15.2:0/1251478761 reap_dead start 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff444a00640 1 -- 172.21.15.2:0/1251478761 shutdown_connections 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff444a00640 1 -- 172.21.15.2:0/1251478761 >> 172.21.15.2:0/1251478761 conn(0x7ff4400b0840 msgr2=0x7ff4400b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff444a00640 1 -- 172.21.15.2:0/1251478761 shutdown_connections 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.898+0000 7ff444a00640 1 -- 172.21.15.2:0/1251478761 wait complete. 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.900+0000 7ff444a00640 1 Processor -- start 2024-01-15T18:19:53.905 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.900+0000 7ff444a00640 1 -- start start 2024-01-15T18:19:53.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff444a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200039a0 0x7ff4401ee9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff444a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff44013afb0 0x7ff44013c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.907 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff444a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 0x7ff44015c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff444a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff440138460 con 0x7ff44013d9c0 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff444a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff440138700 con 0x7ff4200039a0 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff444a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4401389e0 con 0x7ff44013afb0 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 0x7ff44015c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 0x7ff44015c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51770/0 (socket says 172.21.15.2:51770) 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 -- 172.21.15.2:0/3966770534 learned_addr learned my addr 172.21.15.2:0/3966770534 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43e800640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200039a0 0x7ff4401ee9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.908 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43de00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff44013afb0 0x7ff44013c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 -- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff44013afb0 msgr2=0x7ff44013c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff44013afb0 0x7ff44013c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 -- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200039a0 msgr2=0x7ff4401ee9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200039a0 0x7ff4401ee9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.902+0000 7ff43f200640 1 -- 172.21.15.2:0/3966770534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff440138cc0 con 0x7ff44013d9c0 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.903+0000 7ff43f200640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 0x7ff44015c470 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ff4400ae760 tx=0x7ff43800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.903+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438017400 con 0x7ff44013d9c0 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.903+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff438011040 con 0x7ff44013d9c0 2024-01-15T18:19:53.909 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.903+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438017780 con 0x7ff44013d9c0 2024-01-15T18:19:53.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.903+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44013cbe0 con 0x7ff44013d9c0 2024-01-15T18:19:53.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.903+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44013c890 con 0x7ff44013d9c0 2024-01-15T18:19:53.910 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.904+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff440138eb0 con 0x7ff44013d9c0 2024-01-15T18:19:53.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.905+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff4380203b0 con 0x7ff44013d9c0 2024-01-15T18:19:53.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.906+0000 7ff436a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff41806c550 0x7ff41806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:53.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.906+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7ff438088de0 con 0x7ff44013d9c0 2024-01-15T18:19:53.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.907+0000 7ff43e800640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff41806c550 0x7ff41806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:53.911 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.908+0000 7ff43e800640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff41806c550 0x7ff41806ea10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff430034000 tx=0x7ff430009030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:53.914 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:53.911+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff438055320 con 0x7ff44013d9c0 2024-01-15T18:19:54.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.107+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff440074770 con 0x7ff44013d9c0 2024-01-15T18:19:54.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.109+0000 7ff436a00640 1 -- 172.21.15.2:0/3966770534 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+16780 (secure 0 0 0) 0x7ff438058940 con 0x7ff44013d9c0 2024-01-15T18:19:54.113 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:19:54.113 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":116,"fsid":"c0e5773f-513d-405e-88b5-a3133821721f","created":"2024-01-15T17:57:12.310612+0000","modified":"2024-01-15T18:15:19.802397+0000","last_up_change":"2024-01-15T17:57:23.931822+0000","last_in_change":"2024-01-15T17:57:15.787748+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-15T17:57:25.437842+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-15T18:14:14.206729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"113","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-15T18:14:15.347816+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"114","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23efca61-d1a7-4ae6-8b77-93e0343256c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6809","nonce":2449281684}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6811","nonce":2449281684}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6815","nonce":2449281684}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":2449281684},{"type":"v1","addr":"172.21.15.2:6813","nonce":2449281684}]},"public_addr":"172.21.15.2:6809/2449281684","cluster_addr":"172.21.15.2:6811/2449281684","heartbeat_back_addr":"172.21.15.2:6815/2449281684","heartbeat_front_addr":"172.21.15.2:6813/2449281684","state":["exists","up"]},{"osd":1,"uuid":"97935595-7baf-4b8d-8f4b-09d72b71b15f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6801","nonce":3287240406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6803","nonce":3287240406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6807","nonce":3287240406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":3287240406},{"type":"v1","addr":"172.21.15.2:6805","nonce":3287240406}]},"public_addr":"172.21.15.2:6801/3287240406","cluster_addr":"172.21.15.2:6803/3287240406","heartbeat_back_addr":"172.21.15.2:6807/3287240406","heartbeat_front_addr":"172.21.15.2:6805/3287240406","state":["exists","up"]},{"osd":2,"uuid":"c591f415-915a-45d6-9a06-70ca6673ddb6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6818","nonce":3553325871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6822","nonce":3553325871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6827","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6828","nonce":3553325871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":3553325871},{"type":"v1","addr":"172.21.15.2:6826","nonce":3553325871}]},"public_addr":"172.21.15.2:6818/3553325871","cluster_addr":"172.21.15.2:6822/3553325871","heartbeat_back_addr":"172.21.15.2:6828/3553325871","heartbeat_front_addr":"172.21.15.2:6826/3553325871","state":["exists","up"]},{"osd":3,"uuid":"3699a393-59fb-49ae-9d69-a211393cd800","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6817","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6819","nonce":1263638529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6821","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6823","nonce":1263638529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6831","nonce":1263638529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6825","nonce":1263638529},{"type":"v1","addr":"172.21.15.2:6829","nonce":1263638529}]},"public_addr":"172.21.15.2:6819/1263638529","cluster_addr":"172.21.15.2:6823/1263638529","heartbeat_back_addr":"172.21.15.2:6831/1263638529","heartbeat_front_addr":"172.21.15.2:6829/1263638529","state":["exists","up"]},{"osd":4,"uuid":"764ae6bd-491a-47ba-b7c7-7530e7c38ad7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6825","nonce":3407951829}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6827","nonce":3407951829}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6831","nonce":3407951829}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3407951829},{"type":"v1","addr":"172.21.15.137:6829","nonce":3407951829}]},"public_addr":"172.21.15.137:6825/3407951829","cluster_addr":"172.21.15.137:6827/3407951829","heartbeat_back_addr":"172.21.15.137:6831/3407951829","heartbeat_front_addr":"172.21.15.137:6829/3407951829","state":["exists","up"]},{"osd":5,"uuid":"495645fe-5096-4216-bc23-705f01242354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6801","nonce":915165616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6803","nonce":915165616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6807","nonce":915165616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":915165616},{"type":"v1","addr":"172.21.15.137:6805","nonce":915165616}]},"public_addr":"172.21.15.137:6801/915165616","cluster_addr":"172.21.15.137:6803/915165616","heartbeat_back_addr":"172.21.15.137:6807/915165616","heartbeat_front_addr":"172.21.15.137:6805/915165616","state":["exists","up"]},{"osd":6,"uuid":"8d8a340d-f414-41b6-8003-0a7ec08d73a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6809","nonce":1848728157}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6811","nonce":1848728157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6815","nonce":1848728157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1848728157},{"type":"v1","addr":"172.21.15.137:6813","nonce":1848728157}]},"public_addr":"172.21.15.137:6809/1848728157","cluster_addr":"172.21.15.137:6811/1848728157","heartbeat_back_addr":"172.21.15.137:6815/1848728157","heartbeat_front_addr":"172.21.15.137:6813/1848728157","state":["exists","up"]},{"osd":7,"uuid":"a1687635-8b85-4352-940f-5f607a359beb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6817","nonce":463419779}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6819","nonce":463419779}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6823","nonce":463419779}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":463419779},{"type":"v1","addr":"172.21.15.137:6821","nonce":463419779}]},"public_addr":"172.21.15.137:6817/463419779","cluster_addr":"172.21.15.137:6819/463419779","heartbeat_back_addr":"172.21.15.137:6823/463419779","heartbeat_front_addr":"172.21.15.137:6821/463419779","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.088207+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.309220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.451352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.177740+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.378416+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.332539+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.328466+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-15T17:57:21.143717+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.b","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":2,"length":2}]},{"pool":7,"snaps":[{"begin":2,"length":2}]},{"pool":9,"snaps":[{"begin":2,"length":2}]},{"pool":13,"snaps":[{"begin":2,"length":2}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-15T18:19:54.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.111+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff41806c550 msgr2=0x7ff41806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:54.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.111+0000 7ff444a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff41806c550 0x7ff41806ea10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7ff430034000 tx=0x7ff430009030 comp rx=0 tx=0).stop 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 msgr2=0x7ff44015c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 0x7ff44015c470 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7ff4400ae760 tx=0x7ff43800f4a0 comp rx=0 tx=0).stop 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 shutdown_connections 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff41806c550 0x7ff41806ea10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff44013d9c0 0x7ff44015c470 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff44013afb0 0x7ff44013c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 --2- 172.21.15.2:0/3966770534 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff4200039a0 0x7ff4401ee9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 >> 172.21.15.2:0/3966770534 conn(0x7ff4400b0840 msgr2=0x7ff44006d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:54.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 shutdown_connections 2024-01-15T18:19:54.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:54.112+0000 7ff444a00640 1 -- 172.21.15.2:0/3966770534 wait complete. 2024-01-15T18:19:55.137 INFO:tasks.ceph:Scrubbing osd.0 2024-01-15T18:19:55.137 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:55.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.367+0000 7f882d600640 1 Processor -- start 2024-01-15T18:19:55.371 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.367+0000 7f882d600640 1 -- start start 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.369+0000 7f882d600640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f882815d270 0x7f88280b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.369+0000 7f882d600640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8828161890 0x7f8828160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.369+0000 7f882d600640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882806c5e0 con 0x7f88280b2b60 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.369+0000 7f882d600640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882806c8c0 con 0x7f8828161890 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.369+0000 7f882d600640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8828071c00 con 0x7f882815d270 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.369+0000 7f8827400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8828161890 0x7f8828160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f8828161890 0x7f8828160c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51774/0 (socket says 172.21.15.2:51774) 2024-01-15T18:19:55.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827400640 1 -- 172.21.15.2:0/3488063016 learned_addr learned my addr 172.21.15.2:0/3488063016 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:55.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827e00640 1 --2- 172.21.15.2:0/3488063016 >> v2:172.21.15.137:3300/0 conn(0x7f882815d270 0x7f88280b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827400640 1 -- 172.21.15.2:0/3488063016 >> v2:172.21.15.137:3300/0 conn(0x7f882815d270 msgr2=0x7f88280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827400640 1 --2- 172.21.15.2:0/3488063016 >> v2:172.21.15.137:3300/0 conn(0x7f882815d270 0x7f88280b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827400640 1 -- 172.21.15.2:0/3488063016 >> v1:172.21.15.137:6789/0 conn(0x7f88280b2b60 legacy=0x7f88280702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827400640 1 -- 172.21.15.2:0/3488063016 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8828071ee0 con 0x7f8828161890 2024-01-15T18:19:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.370+0000 7f8827e00640 1 --2- 172.21.15.2:0/3488063016 >> v2:172.21.15.137:3300/0 conn(0x7f882815d270 0x7f88280b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:55.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.371+0000 7f8827400640 1 --2- 172.21.15.2:0/3488063016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8828161890 0x7f8828160c60 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f882806bbf0 tx=0x7f881002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d47ffd5aebc90ad server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.372+0000 7f8826a00640 1 -- 172.21.15.2:0/3488063016 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810031750 con 0x7f8828161890 2024-01-15T18:19:55.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.372+0000 7f8826a00640 1 -- 172.21.15.2:0/3488063016 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88100318b0 con 0x7f8828161890 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.372+0000 7f8826a00640 1 -- 172.21.15.2:0/3488063016 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810031c40 con 0x7f8828161890 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.372+0000 7f882d600640 1 -- 172.21.15.2:0/3488063016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8828161890 msgr2=0x7f8828160c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.372+0000 7f882d600640 1 --2- 172.21.15.2:0/3488063016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8828161890 0x7f8828160c60 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f882806bbf0 tx=0x7f881002b7f0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.373+0000 7f882d600640 1 -- 172.21.15.2:0/3488063016 shutdown_connections 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.373+0000 7f882d600640 1 --2- 172.21.15.2:0/3488063016 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f8828161890 0x7f8828160c60 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.373+0000 7f882d600640 1 --2- 172.21.15.2:0/3488063016 >> v2:172.21.15.137:3300/0 conn(0x7f882815d270 0x7f88280b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.373+0000 7f882d600640 1 -- 172.21.15.2:0/3488063016 >> 172.21.15.2:0/3488063016 conn(0x7f88280b0840 msgr2=0x7f88280b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.373+0000 7f882d600640 1 -- 172.21.15.2:0/3488063016 shutdown_connections 2024-01-15T18:19:55.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.373+0000 7f882d600640 1 -- 172.21.15.2:0/3488063016 wait complete. 2024-01-15T18:19:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.375+0000 7f882d600640 1 Processor -- start 2024-01-15T18:19:55.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.375+0000 7f882d600640 1 -- start start 2024-01-15T18:19:55.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882d600640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88280b2b60 0x7f88280aeaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882d600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f882815d270 0x7f882815a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882d600640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8828161890 0x7f882815a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882d600640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882815ba30 con 0x7f882815d270 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882d600640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882815be50 con 0x7f8828161890 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882d600640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882815ab00 con 0x7f88280b2b60 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f8827400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f882815d270 0x7f882815a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f8827400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f882815d270 0x7f882815a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51782/0 (socket says 172.21.15.2:51782) 2024-01-15T18:19:55.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f8827400640 1 -- 172.21.15.2:0/3714289438 learned_addr learned my addr 172.21.15.2:0/3714289438 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.376+0000 7f882cc00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8828161890 0x7f882815a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827e00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88280b2b60 0x7f88280aeaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827400640 1 -- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88280b2b60 msgr2=0x7f88280aeaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827400640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88280b2b60 0x7f88280aeaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827400640 1 -- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8828161890 msgr2=0x7f882815a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827400640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f8828161890 0x7f882815a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827400640 1 -- 172.21.15.2:0/3714289438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88100094d0 con 0x7f882815d270 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827e00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88280b2b60 0x7f88280aeaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f8827400640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f882815d270 0x7f882815a080 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f8810000c00 tx=0x7f88100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.377+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810033c80 con 0x7f882815d270 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.378+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8810033de0 con 0x7f882815d270 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.378+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882815adb0 con 0x7f882815d270 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.378+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881003c620 con 0x7f882815d270 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.378+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882815b370 con 0x7f882815d270 2024-01-15T18:19:55.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.379+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88281611a0 con 0x7f882815d270 2024-01-15T18:19:55.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.380+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f881003a040 con 0x7f882815d270 2024-01-15T18:19:55.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.381+0000 7f881fe00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f880006c550 0x7f880006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.381+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f88100aa610 con 0x7f882815d270 2024-01-15T18:19:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.381+0000 7f881fe00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f88000749d0 0x7f8800076e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.381+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 --> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8800077540 con 0x7f88000749d0 2024-01-15T18:19:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.381+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88100aae20 con 0x7f882815d270 2024-01-15T18:19:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.382+0000 7f882cc00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f88000749d0 0x7f8800076e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.382+0000 7f8827e00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f880006c550 0x7f880006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.382+0000 7f882cc00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f88000749d0 0x7f8800076e70 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.383+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== osd.0 v2:172.21.15.2:6808/2449281684 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8800077540 con 0x7f88000749d0 2024-01-15T18:19:55.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.383+0000 7f8827e00640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f880006c550 0x7f880006ea10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8828070740 tx=0x7f8818008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.398+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 --> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f882815bb70 con 0x7f88000749d0 2024-01-15T18:19:55.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.398+0000 7f881fe00640 1 -- 172.21.15.2:0/3714289438 <== osd.0 v2:172.21.15.2:6808/2449281684 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f882815bb70 con 0x7f88000749d0 2024-01-15T18:19:55.402 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:55.402 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:55.402 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.398+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f88000749d0 msgr2=0x7f8800076e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.398+0000 7f882d600640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:6808/2449281684,v1:172.21.15.2:6809/2449281684] conn(0x7f88000749d0 0x7f8800076e70 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f880006c550 msgr2=0x7f880006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f880006c550 0x7f880006ea10 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f8828070740 tx=0x7f8818008040 comp rx=0 tx=0).stop 2024-01-15T18:19:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f882815d270 msgr2=0x7f882815a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 --2- 172.21.15.2:0/3714289438 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f882815d270 0x7f882815a080 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f8810000c00 tx=0x7f88100046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882cc00640 1 -- 172.21.15.2:0/3714289438 reap_dead start 2024-01-15T18:19:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 shutdown_connections 2024-01-15T18:19:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 >> 172.21.15.2:0/3714289438 conn(0x7f88280b0840 msgr2=0x7f88280cf5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 shutdown_connections 2024-01-15T18:19:55.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.399+0000 7f882d600640 1 -- 172.21.15.2:0/3714289438 wait complete. 2024-01-15T18:19:55.420 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-15T18:19:55.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.639+0000 7fd336c00640 1 Processor -- start 2024-01-15T18:19:55.642 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.639+0000 7fd336c00640 1 -- start start 2024-01-15T18:19:55.644 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd336c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd3300cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd336c00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33015cb00 con 0x7fd33006cb80 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd336c00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3300ccec0 con 0x7fd33015b7d0 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd336c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300cd1a0 con 0x7fd33015c0e0 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd334c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fd33006cb80 0x7fd33015cf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55070/0 (socket says 172.21.15.2:55070) 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd334c00640 1 -- 172.21.15.2:0/1702305673 learned_addr learned my addr 172.21.15.2:0/1702305673 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd32fe00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd3300cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.641+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133167498 0 0) 0x7fd33015cb00 con 0x7fd33006cb80 2024-01-15T18:19:55.645 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd310003520 con 0x7fd33006cb80 2024-01-15T18:19:55.647 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 223836713 0 0) 0x7fd310003520 con 0x7fd33006cb80 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 msgr2=0x7fd3300cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd3300cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 >> v1:172.21.15.137:6789/0 conn(0x7fd33015b7d0 legacy=0x7fd33006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300cd480 con 0x7fd33006cb80 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7fd3280019f0 con 0x7fd33006cb80 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 >> v1:172.21.15.2:6789/0 conn(0x7fd33006cb80 legacy=0x7fd33015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3100039a0 0x7fd310003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3100047c0 0x7fd310004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.648 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd310005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300cd1a0 con 0x7fd3100047c0 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31000ba30 con 0x7fd3100039a0 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31000bbf0 con 0x7fd33015c0e0 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.642+0000 7fd327e00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3100047c0 0x7fd310004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd32fe00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3100039a0 0x7fd310003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd334c00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd310005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd327e00640 1 -- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 msgr2=0x7fd310005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:55.649 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd327e00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd310005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd327e00640 1 -- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3100039a0 msgr2=0x7fd310003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd327e00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd3100039a0 0x7fd310003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd327e00640 1 -- 172.21.15.2:0/1702305673 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310003520 con 0x7fd3100047c0 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.643+0000 7fd334c00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd33015c0e0 0x7fd310005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.644+0000 7fd327e00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3100047c0 0x7fd310004c40 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fd33015bc10 tx=0x7fd3100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d85368003657c008 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.644+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3180052d0 con 0x7fd3100047c0 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.644+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31802e050 con 0x7fd3100047c0 2024-01-15T18:19:55.650 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.644+0000 7fd32f400640 1 -- 172.21.15.2:0/1702305673 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3180035f0 con 0x7fd3100047c0 2024-01-15T18:19:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd336c00640 1 -- 172.21.15.2:0/1702305673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3100047c0 msgr2=0x7fd310004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd336c00640 1 --2- 172.21.15.2:0/1702305673 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd3100047c0 0x7fd310004c40 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fd33015bc10 tx=0x7fd3100057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd334c00640 1 -- 172.21.15.2:0/1702305673 reap_dead start 2024-01-15T18:19:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd336c00640 1 -- 172.21.15.2:0/1702305673 shutdown_connections 2024-01-15T18:19:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd336c00640 1 -- 172.21.15.2:0/1702305673 >> 172.21.15.2:0/1702305673 conn(0x7fd33006da20 msgr2=0x7fd33006a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:55.653 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd336c00640 1 -- 172.21.15.2:0/1702305673 shutdown_connections 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.645+0000 7fd336c00640 1 -- 172.21.15.2:0/1702305673 wait complete. 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.647+0000 7fd336c00640 1 Processor -- start 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.647+0000 7fd336c00640 1 -- start start 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd336c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3100047c0 0x7fd33014bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd336c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 0x7fd330153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd336c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd33014c9d0 0x7fd3301541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd336c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330153210 con 0x7fd33014de90 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd336c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33014e320 con 0x7fd33014c9d0 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd336c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3301534b0 con 0x7fd3100047c0 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd327e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 0x7fd330153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.654 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd327e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 0x7fd330153cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51802/0 (socket says 172.21.15.2:51802) 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd327e00640 1 -- 172.21.15.2:0/4269124354 learned_addr learned my addr 172.21.15.2:0/4269124354 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd32fe00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3100047c0 0x7fd33014bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.648+0000 7fd327e00640 1 -- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3100047c0 msgr2=0x7fd33014bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd334c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd33014c9d0 0x7fd3301541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd327e00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3100047c0 0x7fd33014bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd327e00640 1 -- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd33014c9d0 msgr2=0x7fd3301541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd327e00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd33014c9d0 0x7fd3301541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd327e00640 1 -- 172.21.15.2:0/4269124354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330153790 con 0x7fd33014de90 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd327e00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 0x7fd330153cb0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fd31802bba0 tx=0x7fd318001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.655 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3180076c0 con 0x7fd33014de90 2024-01-15T18:19:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318007a30 con 0x7fd33014de90 2024-01-15T18:19:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318006be0 con 0x7fd33014de90 2024-01-15T18:19:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd33014e600 con 0x7fd33014de90 2024-01-15T18:19:55.656 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.649+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd33014c5d0 con 0x7fd33014de90 2024-01-15T18:19:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.650+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330153920 con 0x7fd33014de90 2024-01-15T18:19:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.652+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd318006d40 con 0x7fd33014de90 2024-01-15T18:19:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.653+0000 7fd32cc00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806c580 0x7fd30806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.653+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fd3180aa7c0 con 0x7fd33014de90 2024-01-15T18:19:55.657 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.654+0000 7fd32fe00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806c580 0x7fd30806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:55.658 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.655+0000 7fd32fe00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806c580 0x7fd30806ea40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd320009950 tx=0x7fd320008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:55.660 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.657+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd318076d00 con 0x7fd33014de90 2024-01-15T18:19:55.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.849+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd33015c750 con 0x7fd30806c580 2024-01-15T18:19:55.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.851+0000 7fd32cc00640 1 -- 172.21.15.2:0/4269124354 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd3200095c0 con 0x7fd30806c580 2024-01-15T18:19:55.854 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 0 to deep-scrub 2024-01-15T18:19:55.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.853+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806c580 msgr2=0x7fd30806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.853+0000 7fd336c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806c580 0x7fd30806ea40 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd320009950 tx=0x7fd320008040 comp rx=0 tx=0).stop 2024-01-15T18:19:55.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.853+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 msgr2=0x7fd330153cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:55.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.853+0000 7fd336c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 0x7fd330153cb0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fd31802bba0 tx=0x7fd318001120 comp rx=0 tx=0).stop 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 shutdown_connections 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fd30806c580 0x7fd30806ea40 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fd33014c9d0 0x7fd3301541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fd33014de90 0x7fd330153cb0 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 --2- 172.21.15.2:0/4269124354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fd3100047c0 0x7fd33014bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 >> 172.21.15.2:0/4269124354 conn(0x7fd33006da20 msgr2=0x7fd3300cb1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 shutdown_connections 2024-01-15T18:19:55.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:55.854+0000 7fd336c00640 1 -- 172.21.15.2:0/4269124354 wait complete. 2024-01-15T18:19:55.875 INFO:tasks.ceph:Scrubbing osd.1 2024-01-15T18:19:55.876 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:56.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.099+0000 7f408ae00640 1 Processor -- start 2024-01-15T18:19:56.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.100+0000 7f408ae00640 1 -- start start 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.101+0000 7f408ae00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4084144c00 0x7f40840cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.101+0000 7f408ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f408406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.101+0000 7f408ae00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40840ce230 con 0x7f408415c450 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.101+0000 7f408ae00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40840ce510 con 0x7f4084144c00 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.101+0000 7f408ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40840ce7f0 con 0x7f40840cdd70 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4084144c00 0x7f40840cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f4084144c00 0x7f40840cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51812/0 (socket says 172.21.15.2:51812) 2024-01-15T18:19:56.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 -- 172.21.15.2:0/3562416354 learned_addr learned my addr 172.21.15.2:0/3562416354 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:56.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083400640 1 --2- 172.21.15.2:0/3562416354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f408406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 -- 172.21.15.2:0/3562416354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 msgr2=0x7f408406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 --2- 172.21.15.2:0/3562416354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f408406c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 -- 172.21.15.2:0/3562416354 >> v1:172.21.15.137:6789/0 conn(0x7f408415c450 legacy=0x7f40840b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.102+0000 7f4083e00640 1 -- 172.21.15.2:0/3562416354 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40840705f0 con 0x7f4084144c00 2024-01-15T18:19:56.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.103+0000 7f4083e00640 1 --2- 172.21.15.2:0/3562416354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f40840cd830 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f4074009870 tx=0x7f407402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7eeabbfcce4b5f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f4082a00640 1 -- 172.21.15.2:0/3562416354 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4074031640 con 0x7f4084144c00 2024-01-15T18:19:56.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f4082a00640 1 -- 172.21.15.2:0/3562416354 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40740317a0 con 0x7f4084144c00 2024-01-15T18:19:56.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f4082a00640 1 -- 172.21.15.2:0/3562416354 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4074031b30 con 0x7f4084144c00 2024-01-15T18:19:56.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f408ae00640 1 -- 172.21.15.2:0/3562416354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 msgr2=0x7f40840cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f408ae00640 1 --2- 172.21.15.2:0/3562416354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f40840cd830 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f4074009870 tx=0x7f407402b8f0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f408ae00640 1 -- 172.21.15.2:0/3562416354 shutdown_connections 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.104+0000 7f408ae00640 1 --2- 172.21.15.2:0/3562416354 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f408406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.105+0000 7f408ae00640 1 --2- 172.21.15.2:0/3562416354 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f40840cd830 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.105+0000 7f408ae00640 1 -- 172.21.15.2:0/3562416354 >> 172.21.15.2:0/3562416354 conn(0x7f40840b45f0 msgr2=0x7f40840b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.105+0000 7f408ae00640 1 -- 172.21.15.2:0/3562416354 shutdown_connections 2024-01-15T18:19:56.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.105+0000 7f408ae00640 1 -- 172.21.15.2:0/3562416354 wait complete. 2024-01-15T18:19:56.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.106+0000 7f408ae00640 1 Processor -- start 2024-01-15T18:19:56.110 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.106+0000 7f408ae00640 1 -- start start 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.107+0000 7f408ae00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f4084153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.107+0000 7f408ae00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f4084131ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.107+0000 7f408ae00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f408415c450 0x7f4084137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.107+0000 7f408ae00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084154280 con 0x7f4084144c00 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.107+0000 7f408ae00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084131130 con 0x7f408415c450 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.107+0000 7f408ae00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4084131410 con 0x7f40840cdd70 2024-01-15T18:19:56.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f4084131ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f4084131ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51826/0 (socket says 172.21.15.2:51826) 2024-01-15T18:19:56.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 -- 172.21.15.2:0/1509985115 learned_addr learned my addr 172.21.15.2:0/1509985115 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:56.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f4084153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4088e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f408415c450 0x7f4084137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 -- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 msgr2=0x7f4084153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.113 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f40840cdd70 0x7f4084153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 -- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f408415c450 msgr2=0x7f4084137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f408415c450 0x7f4084137000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4083400640 1 -- 172.21.15.2:0/1509985115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40740094d0 con 0x7f4084144c00 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.108+0000 7f4088e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f408415c450 0x7f4084137000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.109+0000 7f4083400640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f4084131ef0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f406c00d800 tx=0x7f406c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.109+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c00b6c0 con 0x7f4084144c00 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.109+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c00b820 con 0x7f4084144c00 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.109+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c00bbb0 con 0x7f4084144c00 2024-01-15T18:19:56.114 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.109+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40841316f0 con 0x7f4084144c00 2024-01-15T18:19:56.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.109+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4084153f70 con 0x7f4084144c00 2024-01-15T18:19:56.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.110+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40841312a0 con 0x7f4084144c00 2024-01-15T18:19:56.115 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.111+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f406c00bd10 con 0x7f4084144c00 2024-01-15T18:19:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.112+0000 7f407be00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f405c06c580 0x7f405c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.112+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f406c089b10 con 0x7f4084144c00 2024-01-15T18:19:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.112+0000 7f407be00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f405c074a00 0x7f405c076ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.112+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 --> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f405c077570 con 0x7f405c074a00 2024-01-15T18:19:56.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.112+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f406c089e80 con 0x7f4084144c00 2024-01-15T18:19:56.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.113+0000 7f4083e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f405c06c580 0x7f405c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.113+0000 7f4088e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f405c074a00 0x7f405c076ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.113+0000 7f4088e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f405c074a00 0x7f405c076ea0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.114+0000 7f4083e00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f405c06c580 0x7f405c06ea40 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4074036000 tx=0x7f407403c040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.117 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.114+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== osd.1 v2:172.21.15.2:6800/3287240406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f405c077570 con 0x7f405c074a00 2024-01-15T18:19:56.132 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.129+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 --> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4084131580 con 0x7f405c074a00 2024-01-15T18:19:56.133 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.129+0000 7f407be00640 1 -- 172.21.15.2:0/1509985115 <== osd.1 v2:172.21.15.2:6800/3287240406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4084131580 con 0x7f405c074a00 2024-01-15T18:19:56.133 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:56.133 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:56.133 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:56.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f405c074a00 msgr2=0x7f405c076ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:6800/3287240406,v1:172.21.15.2:6801/3287240406] conn(0x7f405c074a00 0x7f405c076ea0 crc :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f405c06c580 msgr2=0x7f405c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.134 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f405c06c580 0x7f405c06ea40 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f4074036000 tx=0x7f407403c040 comp rx=0 tx=0).stop 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 msgr2=0x7f4084131ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 --2- 172.21.15.2:0/1509985115 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f4084144c00 0x7f4084131ef0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f406c00d800 tx=0x7f406c00dcc0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f4088e00640 1 -- 172.21.15.2:0/1509985115 reap_dead start 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 shutdown_connections 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.130+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 >> 172.21.15.2:0/1509985115 conn(0x7f40840b45f0 msgr2=0x7f408415b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.131+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 shutdown_connections 2024-01-15T18:19:56.135 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.131+0000 7f408ae00640 1 -- 172.21.15.2:0/1509985115 wait complete. 2024-01-15T18:19:56.155 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-15T18:19:56.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.370+0000 7f61e1e00640 1 Processor -- start 2024-01-15T18:19:56.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.371+0000 7f61e1e00640 1 -- start start 2024-01-15T18:19:56.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.372+0000 7f61e1e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f61dc15e5b0 0x7f61dc0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.372+0000 7f61e1e00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc0b2760 con 0x7f61dc0cf5c0 2024-01-15T18:19:56.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.372+0000 7f61e1e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc0d1bb0 con 0x7f61dc0cb790 2024-01-15T18:19:56.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.372+0000 7f61e1e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc0ae630 con 0x7f61dc15e5b0 2024-01-15T18:19:56.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.372+0000 7f61e1400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f61dc0cf5c0 0x7f61dc0aea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55076/0 (socket says 172.21.15.2:55076) 2024-01-15T18:19:56.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61e1400640 1 -- 172.21.15.2:0/378246284 learned_addr learned my addr 172.21.15.2:0/378246284 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:56.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61e0a00640 1 --2- 172.21.15.2:0/378246284 >> v2:172.21.15.137:3300/0 conn(0x7f61dc15e5b0 0x7f61dc0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379868745 0 0) 0x7f61dc0b2760 con 0x7f61dc0cf5c0 2024-01-15T18:19:56.378 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61bc003520 con 0x7f61dc0cf5c0 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2430326626 0 0) 0x7f61bc003520 con 0x7f61dc0cf5c0 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 >> v2:172.21.15.137:3300/0 conn(0x7f61dc15e5b0 msgr2=0x7f61dc0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 --2- 172.21.15.2:0/378246284 >> v2:172.21.15.137:3300/0 conn(0x7f61dc15e5b0 0x7f61dc0b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 >> v1:172.21.15.137:6789/0 conn(0x7f61dc0cb790 legacy=0x7f61dc0ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc0adc90 con 0x7f61dc0cf5c0 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.373+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f61d4001670 con 0x7f61dc0cf5c0 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 >> v1:172.21.15.2:6789/0 conn(0x7f61dc0cf5c0 legacy=0x7f61dc0aea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0039c0 0x7f61bc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61bc0047c0 0x7f61bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.379 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0052a0 0x7f61bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0ae630 con 0x7f61bc0047c0 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc00c1a0 con 0x7f61bc0052a0 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61bc00c360 con 0x7f61bc0039c0 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61dbe00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61bc0047c0 0x7f61bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61e0a00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0039c0 0x7f61bc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61dbe00640 1 -- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0039c0 msgr2=0x7f61bc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61e1400640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0052a0 0x7f61bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.374+0000 7f61dbe00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0039c0 0x7f61bc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.375+0000 7f61dbe00640 1 -- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0052a0 msgr2=0x7f61bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.380 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.375+0000 7f61dbe00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0052a0 0x7f61bc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.375+0000 7f61dbe00640 1 -- 172.21.15.2:0/378246284 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61bc003520 con 0x7f61bc0047c0 2024-01-15T18:19:56.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.375+0000 7f61e1400640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61bc0052a0 0x7f61bc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:56.381 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.375+0000 7f61e1400640 1 -- 172.21.15.2:0/378246284 reap_dead start 2024-01-15T18:19:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.376+0000 7f61dbe00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61bc0047c0 0x7f61bc004c40 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f61cc001e20 tx=0x7f61bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d984042d6e4fed42 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.376+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc003b70 con 0x7f61bc0047c0 2024-01-15T18:19:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.376+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc003cd0 con 0x7f61bc0047c0 2024-01-15T18:19:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.376+0000 7f61db400640 1 -- 172.21.15.2:0/378246284 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc005310 con 0x7f61bc0047c0 2024-01-15T18:19:56.382 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 -- 172.21.15.2:0/378246284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61bc0047c0 msgr2=0x7f61bc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61bc0047c0 0x7f61bc004c40 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f61cc001e20 tx=0x7f61bc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 -- 172.21.15.2:0/378246284 shutdown_connections 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 --2- 172.21.15.2:0/378246284 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61bc0047c0 0x7f61bc004c40 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 -- 172.21.15.2:0/378246284 >> 172.21.15.2:0/378246284 conn(0x7f61dc0d0e20 msgr2=0x7f61dc0d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 -- 172.21.15.2:0/378246284 shutdown_connections 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.377+0000 7f61e1e00640 1 -- 172.21.15.2:0/378246284 wait complete. 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.379+0000 7f61e1e00640 1 Processor -- start 2024-01-15T18:19:56.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.379+0000 7f61e1e00640 1 -- start start 2024-01-15T18:19:56.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61e1e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0047c0 0x7f61dc15d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61e1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 0x7f61dc0c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61e1e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61dc1f2bf0 0x7f61dc1f3070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61e1e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0ca6c0 con 0x7f61dc0c8f40 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61e1e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1f35b0 con 0x7f61dc1f2bf0 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61e1e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1f3a30 con 0x7f61bc0047c0 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 0x7f61dc0c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61dbe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 0x7f61dc0c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51838/0 (socket says 172.21.15.2:51838) 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.380+0000 7f61dbe00640 1 -- 172.21.15.2:0/1674357348 learned_addr learned my addr 172.21.15.2:0/1674357348 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61e0a00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0047c0 0x7f61dc15d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61dbe00640 1 -- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0047c0 msgr2=0x7f61dc15d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61e1400640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61dc1f2bf0 0x7f61dc1f3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61dbe00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0047c0 0x7f61dc15d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61dbe00640 1 -- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61dc1f2bf0 msgr2=0x7f61dc1f3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61dbe00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61dc1f2bf0 0x7f61dc1f3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61dbe00640 1 -- 172.21.15.2:0/1674357348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc0c9ab0 con 0x7f61dc0c8f40 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61dbe00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 0x7f61dc0c93c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f61cc000f80 tx=0x7f61cc0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.381+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc03e070 con 0x7f61dc0c8f40 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.382+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc02f420 con 0x7f61dc0c8f40 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.382+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc0c9d90 con 0x7f61dc0c8f40 2024-01-15T18:19:56.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.382+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc02f790 con 0x7f61dc0c8f40 2024-01-15T18:19:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.382+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc0ca350 con 0x7f61dc0c8f40 2024-01-15T18:19:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.383+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc0ca870 con 0x7f61dc0c8f40 2024-01-15T18:19:56.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.384+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f61cc02f8f0 con 0x7f61dc0c8f40 2024-01-15T18:19:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.385+0000 7f61d8c00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f61b406c580 0x7f61b406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.385+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f61cc0a9f60 con 0x7f61dc0c8f40 2024-01-15T18:19:56.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.386+0000 7f61e0a00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f61b406c580 0x7f61b406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.387+0000 7f61e0a00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f61b406c580 0x7f61b406ea40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f61c40099d0 tx=0x7f61c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.389+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61dc0ca870 con 0x7f61dc0c8f40 2024-01-15T18:19:56.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.585+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f61dc0ca6c0 con 0x7f61b406c580 2024-01-15T18:19:56.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.587+0000 7f61d8c00640 1 -- 172.21.15.2:0/1674357348 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f61dc0ca6c0 con 0x7f61b406c580 2024-01-15T18:19:56.590 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 1 to deep-scrub 2024-01-15T18:19:56.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f61b406c580 msgr2=0x7f61b406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f61b406c580 0x7f61b406ea40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f61c40099d0 tx=0x7f61c4008040 comp rx=0 tx=0).stop 2024-01-15T18:19:56.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 msgr2=0x7f61dc0c93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 0x7f61dc0c93c0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f61cc000f80 tx=0x7f61cc0014a0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 shutdown_connections 2024-01-15T18:19:56.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f61b406c580 0x7f61b406ea40 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f61dc1f2bf0 0x7f61dc1f3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f61dc0c8f40 0x7f61dc0c93c0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 --2- 172.21.15.2:0/1674357348 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f61bc0047c0 0x7f61dc15d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 >> 172.21.15.2:0/1674357348 conn(0x7f61dc0d0e20 msgr2=0x7f61dc0b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:56.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 shutdown_connections 2024-01-15T18:19:56.594 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.589+0000 7f61e1e00640 1 -- 172.21.15.2:0/1674357348 wait complete. 2024-01-15T18:19:56.613 INFO:tasks.ceph:Scrubbing osd.2 2024-01-15T18:19:56.613 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:56.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.829+0000 7ff9eca00640 1 Processor -- start 2024-01-15T18:19:56.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.829+0000 7ff9eca00640 1 -- start start 2024-01-15T18:19:56.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.831+0000 7ff9eca00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7ff9e8159ff0 0x7ff9e815db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.831+0000 7ff9eca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff9e80ae6d0 0x7ff9e815e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.831+0000 7ff9eca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e806cdc0 0x7ff9e80b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.831+0000 7ff9eca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e815eb20 con 0x7ff9e80ae6d0 2024-01-15T18:19:56.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.831+0000 7ff9eca00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9e815e610 con 0x7ff9e8159ff0 2024-01-15T18:19:56.835 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.831+0000 7ff9eca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8066390 con 0x7ff9e806cdc0 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff9e80ae6d0 0x7ff9e815e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7ff9e80ae6d0 0x7ff9e815e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51854/0 (socket says 172.21.15.2:51854) 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 -- 172.21.15.2:0/421158788 learned_addr learned my addr 172.21.15.2:0/421158788 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e7200640 1 --2- 172.21.15.2:0/421158788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e806cdc0 0x7ff9e80b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e6800640 1 --2- 172.21.15.2:0/421158788 >> v2:172.21.15.137:3300/0 conn(0x7ff9e8159ff0 0x7ff9e815db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 -- 172.21.15.2:0/421158788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e806cdc0 msgr2=0x7ff9e80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/421158788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e806cdc0 0x7ff9e80b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 -- 172.21.15.2:0/421158788 >> v2:172.21.15.137:3300/0 conn(0x7ff9e8159ff0 msgr2=0x7ff9e815db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:56.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/421158788 >> v2:172.21.15.137:3300/0 conn(0x7ff9e8159ff0 0x7ff9e815db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.832+0000 7ff9e5e00640 1 -- 172.21.15.2:0/421158788 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9e815e840 con 0x7ff9e80ae6d0 2024-01-15T18:19:56.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.833+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/421158788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e80ae6d0 0x7ff9e815e0d0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80cf190 tx=0x7ff9d002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d675c1563ffc92fa server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.834+0000 7ff9e5400640 1 -- 172.21.15.2:0/421158788 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0031740 con 0x7ff9e80ae6d0 2024-01-15T18:19:56.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.834+0000 7ff9e5400640 1 -- 172.21.15.2:0/421158788 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d00318a0 con 0x7ff9e80ae6d0 2024-01-15T18:19:56.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.834+0000 7ff9e5400640 1 -- 172.21.15.2:0/421158788 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d0031c30 con 0x7ff9e80ae6d0 2024-01-15T18:19:56.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.834+0000 7ff9eca00640 1 -- 172.21.15.2:0/421158788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e80ae6d0 msgr2=0x7ff9e815e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.838 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.834+0000 7ff9eca00640 1 --2- 172.21.15.2:0/421158788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e80ae6d0 0x7ff9e815e0d0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80cf190 tx=0x7ff9d002b7e0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 -- 172.21.15.2:0/421158788 shutdown_connections 2024-01-15T18:19:56.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 --2- 172.21.15.2:0/421158788 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e806cdc0 0x7ff9e80b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 --2- 172.21.15.2:0/421158788 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e80ae6d0 0x7ff9e815e0d0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 --2- 172.21.15.2:0/421158788 >> v2:172.21.15.137:3300/0 conn(0x7ff9e8159ff0 0x7ff9e815db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 -- 172.21.15.2:0/421158788 >> 172.21.15.2:0/421158788 conn(0x7ff9e80cbf70 msgr2=0x7ff9e80cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:56.839 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 -- 172.21.15.2:0/421158788 shutdown_connections 2024-01-15T18:19:56.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.835+0000 7ff9eca00640 1 -- 172.21.15.2:0/421158788 wait complete. 2024-01-15T18:19:56.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.837+0000 7ff9eca00640 1 Processor -- start 2024-01-15T18:19:56.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.837+0000 7ff9eca00640 1 -- start start 2024-01-15T18:19:56.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9eca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e806cdc0 0x7ff9e80b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9eca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e8159ff0 0x7ff9e80b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9eca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e8151fc0 0x7ff9e8152470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9eca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81516d0 con 0x7ff9e806cdc0 2024-01-15T18:19:56.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9eca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e8150ae0 con 0x7ff9e8159ff0 2024-01-15T18:19:56.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9eca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9e81529b0 con 0x7ff9e8151fc0 2024-01-15T18:19:56.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9e6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e806cdc0 0x7ff9e80b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9e6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e806cdc0 0x7ff9e80b2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51870/0 (socket says 172.21.15.2:51870) 2024-01-15T18:19:56.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.838+0000 7ff9e6800640 1 -- 172.21.15.2:0/4131722222 learned_addr learned my addr 172.21.15.2:0/4131722222 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:56.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e8159ff0 0x7ff9e80b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e7200640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e8151fc0 0x7ff9e8152470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e6800640 1 -- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e8151fc0 msgr2=0x7ff9e8152470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e6800640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7ff9e8151fc0 0x7ff9e8152470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e6800640 1 -- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e8159ff0 msgr2=0x7ff9e80b30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e6800640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e8159ff0 0x7ff9e80b30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e6800640 1 -- 172.21.15.2:0/4131722222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00094d0 con 0x7ff9e806cdc0 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7ff9e8159ff0 0x7ff9e80b30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9e6800640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e806cdc0 0x7ff9e80b2b60 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800d5f0 tx=0x7ff9d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d80125f0 con 0x7ff9e806cdc0 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9d8012750 con 0x7ff9e806cdc0 2024-01-15T18:19:56.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9d8012b00 con 0x7ff9e806cdc0 2024-01-15T18:19:56.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.839+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9e8150d30 con 0x7ff9e806cdc0 2024-01-15T18:19:56.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.840+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e81512f0 con 0x7ff9e806cdc0 2024-01-15T18:19:56.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.841+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff9e815a3d0 con 0x7ff9e806cdc0 2024-01-15T18:19:56.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.842+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff9d8012c60 con 0x7ff9e806cdc0 2024-01-15T18:19:56.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.843+0000 7ff9dea00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff9c006c580 0x7ff9c006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.843+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7ff9d8088f90 con 0x7ff9e806cdc0 2024-01-15T18:19:56.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.844+0000 7ff9dea00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7ff9c0074a00 0x7ff9c0076ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:56.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.844+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 --> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9c0077570 con 0x7ff9c0074a00 2024-01-15T18:19:56.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.844+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9d8089b20 con 0x7ff9e806cdc0 2024-01-15T18:19:56.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.844+0000 7ff9e7200640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7ff9c0074a00 0x7ff9c0076ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.844+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff9c006c580 0x7ff9c006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:56.848 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.845+0000 7ff9e7200640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7ff9c0074a00 0x7ff9c0076ea0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.846+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== osd.2 v2:172.21.15.2:6816/3553325871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff9c0077570 con 0x7ff9c0074a00 2024-01-15T18:19:56.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.846+0000 7ff9e5e00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff9c006c580 0x7ff9c006ea40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0035000 tx=0x7ff9d0008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:56.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.864+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 --> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff9e8151480 con 0x7ff9c0074a00 2024-01-15T18:19:56.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.864+0000 7ff9dea00640 1 -- 172.21.15.2:0/4131722222 <== osd.2 v2:172.21.15.2:6816/3553325871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff9e8151480 con 0x7ff9c0074a00 2024-01-15T18:19:56.868 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:56.868 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:56.868 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:56.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7ff9c0074a00 msgr2=0x7ff9c0076ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:6816/3553325871,v1:172.21.15.2:6818/3553325871] conn(0x7ff9c0074a00 0x7ff9c0076ea0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff9c006c580 msgr2=0x7ff9c006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7ff9c006c580 0x7ff9c006ea40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7ff9d0035000 tx=0x7ff9d0008030 comp rx=0 tx=0).stop 2024-01-15T18:19:56.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e806cdc0 msgr2=0x7ff9e80b2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 --2- 172.21.15.2:0/4131722222 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7ff9e806cdc0 0x7ff9e80b2b60 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7ff9d800d5f0 tx=0x7ff9d800dab0 comp rx=0 tx=0).stop 2024-01-15T18:19:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9e7200640 1 -- 172.21.15.2:0/4131722222 reap_dead start 2024-01-15T18:19:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 shutdown_connections 2024-01-15T18:19:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 >> 172.21.15.2:0/4131722222 conn(0x7ff9e80cbf70 msgr2=0x7ff9e80af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 shutdown_connections 2024-01-15T18:19:56.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:56.865+0000 7ff9eca00640 1 -- 172.21.15.2:0/4131722222 wait complete. 2024-01-15T18:19:56.887 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-15T18:19:57.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.090+0000 7f446e200640 1 Processor -- start 2024-01-15T18:19:57.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.090+0000 7f446e200640 1 -- start start 2024-01-15T18:19:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.092+0000 7f446e200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f44681469f0 0x7f4468071df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.092+0000 7f446e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f446806cb80 0x7f44680b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.092+0000 7f446e200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680ac870 con 0x7f4468146dd0 2024-01-15T18:19:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.092+0000 7f446e200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446815d970 con 0x7f44681469f0 2024-01-15T18:19:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.092+0000 7f446e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f446815dc30 con 0x7f446806cb80 2024-01-15T18:19:57.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.093+0000 7f445fe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f4468146dd0 0x7f446806c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55086/0 (socket says 172.21.15.2:55086) 2024-01-15T18:19:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.093+0000 7f445fe00640 1 -- 172.21.15.2:0/1852314444 learned_addr learned my addr 172.21.15.2:0/1852314444 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.093+0000 7f4467e00640 1 --2- 172.21.15.2:0/1852314444 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f446806cb80 0x7f44680b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.093+0000 7f4467400640 1 --2- 172.21.15.2:0/1852314444 >> v2:172.21.15.137:3300/0 conn(0x7f44681469f0 0x7f4468071df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.093+0000 7f4466a00640 1 -- 172.21.15.2:0/1852314444 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789804323 0 0) 0x7f44680ac870 con 0x7f4468146dd0 2024-01-15T18:19:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.093+0000 7f4466a00640 1 -- 172.21.15.2:0/1852314444 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4448003520 con 0x7f4468146dd0 2024-01-15T18:19:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.094+0000 7f4467e00640 1 -- 172.21.15.2:0/1852314444 >> v2:172.21.15.137:3300/0 conn(0x7f44681469f0 msgr2=0x7f4468071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.094+0000 7f4467e00640 1 --2- 172.21.15.2:0/1852314444 >> v2:172.21.15.137:3300/0 conn(0x7f44681469f0 0x7f4468071df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.094+0000 7f4467e00640 1 -- 172.21.15.2:0/1852314444 >> v1:172.21.15.2:6789/0 conn(0x7f4468146dd0 legacy=0x7f446806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.094+0000 7f4466a00640 1 -- 172.21.15.2:0/1852314444 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1062795091 0 0) 0x7f4448003520 con 0x7f4468146dd0 2024-01-15T18:19:57.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.094+0000 7f4467e00640 1 -- 172.21.15.2:0/1852314444 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446815df10 con 0x7f446806cb80 2024-01-15T18:19:57.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.095+0000 7f4467e00640 1 --2- 172.21.15.2:0/1852314444 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f446806cb80 0x7f44680b34a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4460009d90 tx=0x7f446002bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726ef82ddd1504ec server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.095+0000 7f4466a00640 1 -- 172.21.15.2:0/1852314444 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450001670 con 0x7f446806cb80 2024-01-15T18:19:57.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f4466a00640 1 -- 172.21.15.2:0/1852314444 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446002c8a0 con 0x7f446806cb80 2024-01-15T18:19:57.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f4466a00640 1 -- 172.21.15.2:0/1852314444 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446002cc10 con 0x7f446806cb80 2024-01-15T18:19:57.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 -- 172.21.15.2:0/1852314444 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f446806cb80 msgr2=0x7f44680b34a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 --2- 172.21.15.2:0/1852314444 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f446806cb80 0x7f44680b34a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4460009d90 tx=0x7f446002bd70 comp rx=0 tx=0).stop 2024-01-15T18:19:57.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 -- 172.21.15.2:0/1852314444 shutdown_connections 2024-01-15T18:19:57.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 --2- 172.21.15.2:0/1852314444 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f446806cb80 0x7f44680b34a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 --2- 172.21.15.2:0/1852314444 >> v2:172.21.15.137:3300/0 conn(0x7f44681469f0 0x7f4468071df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 -- 172.21.15.2:0/1852314444 >> 172.21.15.2:0/1852314444 conn(0x7f446806da20 msgr2=0x7f446806a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:57.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.096+0000 7f446e200640 1 -- 172.21.15.2:0/1852314444 shutdown_connections 2024-01-15T18:19:57.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.097+0000 7f446e200640 1 -- 172.21.15.2:0/1852314444 wait complete. 2024-01-15T18:19:57.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.098+0000 7f446e200640 1 Processor -- start 2024-01-15T18:19:57.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.098+0000 7f446e200640 1 -- start start 2024-01-15T18:19:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.099+0000 7f446e200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 0x7f4468132400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.099+0000 7f446e200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44681469f0 0x7f44680c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.099+0000 7f446e200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44680c6b60 0x7f44681430a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.099+0000 7f446e200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44681f2a60 con 0x7f446806cb80 2024-01-15T18:19:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.099+0000 7f446e200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680c6ff0 con 0x7f44681469f0 2024-01-15T18:19:57.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.099+0000 7f446e200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4468143e10 con 0x7f44680c6b60 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 0x7f4468132400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 0x7f4468132400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51876/0 (socket says 172.21.15.2:51876) 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 -- 172.21.15.2:0/3766521370 learned_addr learned my addr 172.21.15.2:0/3766521370 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467e00640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44680c6b60 0x7f44681430a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f445fe00640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44681469f0 0x7f44680c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 -- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44680c6b60 msgr2=0x7f44681430a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44680c6b60 0x7f44681430a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 -- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44681469f0 msgr2=0x7f44680c6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44681469f0 0x7f44680c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.105 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 -- 172.21.15.2:0/3766521370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4460009a70 con 0x7f446806cb80 2024-01-15T18:19:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.100+0000 7f4467400640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 0x7f4468132400 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f445800e9b0 tx=0x7f445800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.101+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4458009670 con 0x7f446806cb80 2024-01-15T18:19:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.101+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445800c9a0 con 0x7f446806cb80 2024-01-15T18:19:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.101+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445800cd50 con 0x7f446806cb80 2024-01-15T18:19:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.101+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44681440f0 con 0x7f446806cb80 2024-01-15T18:19:57.106 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.101+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44681446b0 con 0x7f446806cb80 2024-01-15T18:19:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.102+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4468144a20 con 0x7f446806cb80 2024-01-15T18:19:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.103+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f4458013470 con 0x7f446806cb80 2024-01-15T18:19:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.104+0000 7f445f400640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f444006c550 0x7f444006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.104+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f4458088a40 con 0x7f446806cb80 2024-01-15T18:19:57.108 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.105+0000 7f445fe00640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f444006c550 0x7f444006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.109 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.106+0000 7f445fe00640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f444006c550 0x7f444006ea10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f44500017a0 tx=0x7f4450009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.112 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.109+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4458055300 con 0x7f446806cb80 2024-01-15T18:19:57.305 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.302+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f44680ae630 con 0x7f444006c550 2024-01-15T18:19:57.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.304+0000 7f445f400640 1 -- 172.21.15.2:0/3766521370 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f44680ae630 con 0x7f444006c550 2024-01-15T18:19:57.307 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 2 to deep-scrub 2024-01-15T18:19:57.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f444006c550 msgr2=0x7f444006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f444006c550 0x7f444006ea10 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f44500017a0 tx=0x7f4450009040 comp rx=0 tx=0).stop 2024-01-15T18:19:57.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 msgr2=0x7f4468132400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 0x7f4468132400 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7f445800e9b0 tx=0x7f445800ee70 comp rx=0 tx=0).stop 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 shutdown_connections 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f444006c550 0x7f444006ea10 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f44680c6b60 0x7f44681430a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f44681469f0 0x7f44680c6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 --2- 172.21.15.2:0/3766521370 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f446806cb80 0x7f4468132400 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.306+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 >> 172.21.15.2:0/3766521370 conn(0x7f446806da20 msgr2=0x7f44680ace70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.307+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 shutdown_connections 2024-01-15T18:19:57.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.307+0000 7f446e200640 1 -- 172.21.15.2:0/3766521370 wait complete. 2024-01-15T18:19:57.329 INFO:tasks.ceph:Scrubbing osd.3 2024-01-15T18:19:57.329 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:57.554 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.551+0000 7fb376e00640 1 Processor -- start 2024-01-15T18:19:57.555 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.551+0000 7fb376e00640 1 -- start start 2024-01-15T18:19:57.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.553+0000 7fb376e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006c920 0x7fb3700cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.553+0000 7fb376e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb3700cd2c0 0x7fb37006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.553+0000 7fb376e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3700cd720 con 0x7fb37006d240 2024-01-15T18:19:57.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.553+0000 7fb376e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37006eb20 con 0x7fb3700cd2c0 2024-01-15T18:19:57.557 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.553+0000 7fb376e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700cd9f0 con 0x7fb37006c920 2024-01-15T18:19:57.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb3700cd2c0 0x7fb37006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fb3700cd2c0 0x7fb37006cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51884/0 (socket says 172.21.15.2:51884) 2024-01-15T18:19:57.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 -- 172.21.15.2:0/3479728937 learned_addr learned my addr 172.21.15.2:0/3479728937 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:57.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36fe00640 1 --2- 172.21.15.2:0/3479728937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006c920 0x7fb3700cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 -- 172.21.15.2:0/3479728937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006c920 msgr2=0x7fb3700cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:57.558 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 --2- 172.21.15.2:0/3479728937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006c920 0x7fb3700cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 -- 172.21.15.2:0/3479728937 >> v1:172.21.15.137:6789/0 conn(0x7fb37006d240 legacy=0x7fb3700ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:57.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.554+0000 7fb36f400640 1 -- 172.21.15.2:0/3479728937 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37006aec0 con 0x7fb3700cd2c0 2024-01-15T18:19:57.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.555+0000 7fb36f400640 1 --2- 172.21.15.2:0/3479728937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3700cd2c0 0x7fb37006cd00 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fb358009870 tx=0x7fb35802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e884f09abd905ccd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.559 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.556+0000 7fb36ea00640 1 -- 172.21.15.2:0/3479728937 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358031640 con 0x7fb3700cd2c0 2024-01-15T18:19:57.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.556+0000 7fb36ea00640 1 -- 172.21.15.2:0/3479728937 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3580317a0 con 0x7fb3700cd2c0 2024-01-15T18:19:57.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.556+0000 7fb36ea00640 1 -- 172.21.15.2:0/3479728937 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358031b30 con 0x7fb3700cd2c0 2024-01-15T18:19:57.560 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 -- 172.21.15.2:0/3479728937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3700cd2c0 msgr2=0x7fb37006cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 --2- 172.21.15.2:0/3479728937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3700cd2c0 0x7fb37006cd00 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fb358009870 tx=0x7fb35802b8f0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 -- 172.21.15.2:0/3479728937 shutdown_connections 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 --2- 172.21.15.2:0/3479728937 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb3700cd2c0 0x7fb37006cd00 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 --2- 172.21.15.2:0/3479728937 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006c920 0x7fb3700cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 -- 172.21.15.2:0/3479728937 >> 172.21.15.2:0/3479728937 conn(0x7fb3700cc170 msgr2=0x7fb3700cc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 -- 172.21.15.2:0/3479728937 shutdown_connections 2024-01-15T18:19:57.561 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.557+0000 7fb376e00640 1 -- 172.21.15.2:0/3479728937 wait complete. 2024-01-15T18:19:57.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.559+0000 7fb376e00640 1 Processor -- start 2024-01-15T18:19:57.562 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.559+0000 7fb376e00640 1 -- start start 2024-01-15T18:19:57.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb376e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb37006c920 0x7fb3700ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb376e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006d240 0x7fb3700b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb376e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3700cd2c0 0x7fb370143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb376e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700b56e0 con 0x7fb37006c920 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb376e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700ad980 con 0x7fb3700cd2c0 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb376e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3700adc60 con 0x7fb37006d240 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb36fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb37006c920 0x7fb3700ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb36fe00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb37006c920 0x7fb3700ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51898/0 (socket says 172.21.15.2:51898) 2024-01-15T18:19:57.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.560+0000 7fb36fe00640 1 -- 172.21.15.2:0/652895642 learned_addr learned my addr 172.21.15.2:0/652895642 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb36fe00640 1 -- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006d240 msgr2=0x7fb3700b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb36fe00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fb37006d240 0x7fb3700b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb36fe00640 1 -- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3700cd2c0 msgr2=0x7fb370143ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb36fe00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fb3700cd2c0 0x7fb370143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb36fe00640 1 -- 172.21.15.2:0/652895642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3580094d0 con 0x7fb37006c920 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb36fe00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb37006c920 0x7fb3700ad280 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb36000e870 tx=0x7fb36000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36000c940 con 0x7fb37006c920 2024-01-15T18:19:57.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36000caa0 con 0x7fb37006c920 2024-01-15T18:19:57.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3700adf40 con 0x7fb37006c920 2024-01-15T18:19:57.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36000ce50 con 0x7fb37006c920 2024-01-15T18:19:57.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.561+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb370144400 con 0x7fb37006c920 2024-01-15T18:19:57.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.562+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3700cba30 con 0x7fb37006c920 2024-01-15T18:19:57.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.564+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb360013470 con 0x7fb37006c920 2024-01-15T18:19:57.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.564+0000 7fb367e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb34806c5a0 0x7fb34806ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.565+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fb3600888b0 con 0x7fb37006c920 2024-01-15T18:19:57.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.565+0000 7fb367e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7fb348074aa0 0x7fb348076f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.565+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 --> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb348077610 con 0x7fb348074aa0 2024-01-15T18:19:57.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.565+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb360089420 con 0x7fb37006c920 2024-01-15T18:19:57.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.565+0000 7fb36f400640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb34806c5a0 0x7fb34806ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.565+0000 7fb374e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7fb348074aa0 0x7fb348076f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.566+0000 7fb374e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7fb348074aa0 0x7fb348076f40 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.566+0000 7fb36f400640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb34806c5a0 0x7fb34806ea60 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb358036000 tx=0x7fb358030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.567+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== osd.3 v2:172.21.15.2:6817/1263638529 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb348077610 con 0x7fb348074aa0 2024-01-15T18:19:57.584 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.581+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 --> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb37006b760 con 0x7fb348074aa0 2024-01-15T18:19:57.585 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb367e00640 1 -- 172.21.15.2:0/652895642 <== osd.3 v2:172.21.15.2:6817/1263638529 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb368001db0 con 0x7fb348074aa0 2024-01-15T18:19:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:57.585 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:57.585 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7fb348074aa0 msgr2=0x7fb348076f40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:6817/1263638529,v1:172.21.15.2:6819/1263638529] conn(0x7fb348074aa0 0x7fb348076f40 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb34806c5a0 msgr2=0x7fb34806ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.586 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fb34806c5a0 0x7fb34806ea60 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb358036000 tx=0x7fb358030000 comp rx=0 tx=0).stop 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb37006c920 msgr2=0x7fb3700ad280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 --2- 172.21.15.2:0/652895642 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fb37006c920 0x7fb3700ad280 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb36000e870 tx=0x7fb36000ed30 comp rx=0 tx=0).stop 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb374e00640 1 -- 172.21.15.2:0/652895642 reap_dead start 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 shutdown_connections 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 >> 172.21.15.2:0/652895642 conn(0x7fb3700cc170 msgr2=0x7fb37015b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 shutdown_connections 2024-01-15T18:19:57.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.582+0000 7fb376e00640 1 -- 172.21.15.2:0/652895642 wait complete. 2024-01-15T18:19:57.602 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-15T18:19:57.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.837+0000 7fe62c000640 1 Processor -- start 2024-01-15T18:19:57.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.837+0000 7fe62c000640 1 -- start start 2024-01-15T18:19:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe62c000640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fe62406cb80 0x7fe624166070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe62c000640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62415cb00 con 0x7fe62415b7d0 2024-01-15T18:19:57.842 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe62c000640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6240ccec0 con 0x7fe62415c0e0 2024-01-15T18:19:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe62c000640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6240cccd0 con 0x7fe62406cb80 2024-01-15T18:19:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe628c00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fe62415b7d0 0x7fe62406c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55094/0 (socket says 172.21.15.2:55094) 2024-01-15T18:19:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe628c00640 1 -- 172.21.15.2:0/1479903968 learned_addr learned my addr 172.21.15.2:0/1479903968 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.839+0000 7fe62a000640 1 --2- 172.21.15.2:0/1479903968 >> v2:172.21.15.137:3300/0 conn(0x7fe62406cb80 0x7fe624166070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.843 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.840+0000 7fe623e00640 1 -- 172.21.15.2:0/1479903968 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16433075 0 0) 0x7fe62415cb00 con 0x7fe62415b7d0 2024-01-15T18:19:57.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.840+0000 7fe623e00640 1 -- 172.21.15.2:0/1479903968 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe604003520 con 0x7fe62415b7d0 2024-01-15T18:19:57.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.840+0000 7fe62a000640 1 -- 172.21.15.2:0/1479903968 >> v1:172.21.15.137:6789/0 conn(0x7fe62415c0e0 legacy=0x7fe6240cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.840+0000 7fe62a000640 1 -- 172.21.15.2:0/1479903968 >> v1:172.21.15.2:6789/0 conn(0x7fe62415b7d0 legacy=0x7fe62406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.844 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.840+0000 7fe62a000640 1 -- 172.21.15.2:0/1479903968 --> v2:172.21.15.137:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6240cd1a0 con 0x7fe62406cb80 2024-01-15T18:19:57.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.842+0000 7fe62a000640 1 --2- 172.21.15.2:0/1479903968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62406cb80 0x7fe624166070 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009d90 tx=0x7fe61c02bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2254b8d7471419a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.842+0000 7fe623e00640 1 -- 172.21.15.2:0/1479903968 <== mon.1 v2:172.21.15.137:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c02ccf0 con 0x7fe62406cb80 2024-01-15T18:19:57.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.842+0000 7fe623e00640 1 -- 172.21.15.2:0/1479903968 <== mon.1 v2:172.21.15.137:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe61c02ce50 con 0x7fe62406cb80 2024-01-15T18:19:57.845 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.842+0000 7fe623e00640 1 -- 172.21.15.2:0/1479903968 <== mon.1 v2:172.21.15.137:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61c034660 con 0x7fe62406cb80 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 -- 172.21.15.2:0/1479903968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62406cb80 msgr2=0x7fe624166070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 --2- 172.21.15.2:0/1479903968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62406cb80 0x7fe624166070 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fe61c009d90 tx=0x7fe61c02bf40 comp rx=0 tx=0).stop 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 -- 172.21.15.2:0/1479903968 shutdown_connections 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 --2- 172.21.15.2:0/1479903968 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62406cb80 0x7fe624166070 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 -- 172.21.15.2:0/1479903968 >> 172.21.15.2:0/1479903968 conn(0x7fe62406da20 msgr2=0x7fe62406a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 -- 172.21.15.2:0/1479903968 shutdown_connections 2024-01-15T18:19:57.847 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.843+0000 7fe62c000640 1 -- 172.21.15.2:0/1479903968 wait complete. 2024-01-15T18:19:57.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.845+0000 7fe62c000640 1 Processor -- start 2024-01-15T18:19:57.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.846+0000 7fe62c000640 1 -- start start 2024-01-15T18:19:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62c000640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 0x7fe624154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62c000640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62415b7d0 0x7fe6241e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62c000640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe62415c0e0 0x7fe6241e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62c000640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe624150dd0 con 0x7fe62406cb80 2024-01-15T18:19:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62c000640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241e5670 con 0x7fe62415b7d0 2024-01-15T18:19:57.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62c000640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6241e5950 con 0x7fe62415c0e0 2024-01-15T18:19:57.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe629600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 0x7fe624154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe629600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 0x7fe624154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51900/0 (socket says 172.21.15.2:51900) 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe629600640 1 -- 172.21.15.2:0/101227009 learned_addr learned my addr 172.21.15.2:0/101227009 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe62a000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe62415c0e0 0x7fe6241e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.847+0000 7fe628c00640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62415b7d0 0x7fe6241e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.848+0000 7fe629600640 1 -- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe62415c0e0 msgr2=0x7fe6241e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.848+0000 7fe629600640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe62415c0e0 0x7fe6241e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.848+0000 7fe629600640 1 -- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62415b7d0 msgr2=0x7fe6241e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.848+0000 7fe629600640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62415b7d0 0x7fe6241e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:57.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.848+0000 7fe629600640 1 -- 172.21.15.2:0/101227009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c009a70 con 0x7fe62406cb80 2024-01-15T18:19:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.848+0000 7fe629600640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 0x7fe624154960 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fe61400c400 tx=0x7fe61400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.849+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6140052d0 con 0x7fe62406cb80 2024-01-15T18:19:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.849+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe614005430 con 0x7fe62406cb80 2024-01-15T18:19:57.853 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.849+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe614003830 con 0x7fe62406cb80 2024-01-15T18:19:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.849+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6241e5c30 con 0x7fe62406cb80 2024-01-15T18:19:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.849+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6241e61f0 con 0x7fe62406cb80 2024-01-15T18:19:57.854 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.850+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62415c7b0 con 0x7fe62406cb80 2024-01-15T18:19:57.855 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.852+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe614003990 con 0x7fe62406cb80 2024-01-15T18:19:57.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.853+0000 7fe621600640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5fc06c550 0x7fe5fc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:57.856 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.853+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fe614088060 con 0x7fe62406cb80 2024-01-15T18:19:57.857 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.854+0000 7fe628c00640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5fc06c550 0x7fe5fc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:57.858 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.855+0000 7fe628c00640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5fc06c550 0x7fe5fc06ea10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe6241513b0 tx=0x7fe60c007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:57.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:57.858+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe614054650 con 0x7fe62406cb80 2024-01-15T18:19:58.082 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.079+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6240686f0 con 0x7fe5fc06c550 2024-01-15T18:19:58.083 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.080+0000 7fe621600640 1 -- 172.21.15.2:0/101227009 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe6240686f0 con 0x7fe5fc06c550 2024-01-15T18:19:58.083 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 3 to deep-scrub 2024-01-15T18:19:58.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5fc06c550 msgr2=0x7fe5fc06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5fc06c550 0x7fe5fc06ea10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe6241513b0 tx=0x7fe60c007d30 comp rx=0 tx=0).stop 2024-01-15T18:19:58.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 msgr2=0x7fe624154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.086 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 0x7fe624154960 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fe61400c400 tx=0x7fe61400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 shutdown_connections 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fe5fc06c550 0x7fe5fc06ea10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fe62415c0e0 0x7fe6241e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fe62415b7d0 0x7fe6241e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 --2- 172.21.15.2:0/101227009 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fe62406cb80 0x7fe624154960 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 >> 172.21.15.2:0/101227009 conn(0x7fe62406da20 msgr2=0x7fe62415eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 shutdown_connections 2024-01-15T18:19:58.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.083+0000 7fe62c000640 1 -- 172.21.15.2:0/101227009 wait complete. 2024-01-15T18:19:58.105 INFO:tasks.ceph:Scrubbing osd.4 2024-01-15T18:19:58.105 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:58.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.332+0000 7f2f89a00640 1 Processor -- start 2024-01-15T18:19:58.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.332+0000 7f2f89a00640 1 -- start start 2024-01-15T18:19:58.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f89a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f8406d000 0x7f2f840b3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f89a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8415dd70 con 0x7f2f840702c0 2024-01-15T18:19:58.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f89a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f840711b0 con 0x7f2f841439f0 2024-01-15T18:19:58.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f89a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84143dd0 con 0x7f2f8406d000 2024-01-15T18:19:58.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f82000640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f2f840702c0 0x7f2f8406c8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55110/0 (socket says 172.21.15.2:55110) 2024-01-15T18:19:58.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f82000640 1 -- 172.21.15.2:0/1210137538 learned_addr learned my addr 172.21.15.2:0/1210137538 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:58.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.334+0000 7f2f83400640 1 --2- 172.21.15.2:0/1210137538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f8406d000 0x7f2f840b3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.335+0000 7f2f83400640 1 -- 172.21.15.2:0/1210137538 >> v1:172.21.15.137:6789/0 conn(0x7f2f841439f0 legacy=0x7f2f840cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.335+0000 7f2f83400640 1 -- 172.21.15.2:0/1210137538 >> v1:172.21.15.2:6789/0 conn(0x7f2f840702c0 legacy=0x7f2f8406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.335+0000 7f2f83400640 1 -- 172.21.15.2:0/1210137538 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f84070910 con 0x7f2f8406d000 2024-01-15T18:19:58.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.337+0000 7f2f83400640 1 --2- 172.21.15.2:0/1210137538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f8406d000 0x7f2f840b3420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0097d0 tx=0x7f2f7c02db40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c17a63270e20f539 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.340+0000 7f2f81600640 1 -- 172.21.15.2:0/1210137538 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c02f900 con 0x7f2f8406d000 2024-01-15T18:19:58.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.340+0000 7f2f81600640 1 -- 172.21.15.2:0/1210137538 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f7c02fa60 con 0x7f2f8406d000 2024-01-15T18:19:58.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.340+0000 7f2f81600640 1 -- 172.21.15.2:0/1210137538 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f7c02fdf0 con 0x7f2f8406d000 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.341+0000 7f2f89a00640 1 -- 172.21.15.2:0/1210137538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f8406d000 msgr2=0x7f2f840b3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.341+0000 7f2f89a00640 1 --2- 172.21.15.2:0/1210137538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f8406d000 0x7f2f840b3420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0097d0 tx=0x7f2f7c02db40 comp rx=0 tx=0).stop 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.341+0000 7f2f89a00640 1 -- 172.21.15.2:0/1210137538 shutdown_connections 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.341+0000 7f2f89a00640 1 --2- 172.21.15.2:0/1210137538 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f8406d000 0x7f2f840b3420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.341+0000 7f2f89a00640 1 -- 172.21.15.2:0/1210137538 >> 172.21.15.2:0/1210137538 conn(0x7f2f840b45f0 msgr2=0x7f2f840b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.341+0000 7f2f89a00640 1 -- 172.21.15.2:0/1210137538 shutdown_connections 2024-01-15T18:19:58.345 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.342+0000 7f2f89a00640 1 -- 172.21.15.2:0/1210137538 wait complete. 2024-01-15T18:19:58.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.344+0000 7f2f89a00640 1 Processor -- start 2024-01-15T18:19:58.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.344+0000 7f2f89a00640 1 -- start start 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.345+0000 7f2f89a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f8406d000 0x7f2f84159950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.345+0000 7f2f89a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f840702c0 0x7f2f8413a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.345+0000 7f2f89a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f841439f0 0x7f2f840ca610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.345+0000 7f2f89a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f841391c0 con 0x7f2f8406d000 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.345+0000 7f2f89a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84138540 con 0x7f2f841439f0 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.345+0000 7f2f89a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f84138820 con 0x7f2f840702c0 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f8406d000 0x7f2f84159950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f8406d000 0x7f2f84159950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51906/0 (socket says 172.21.15.2:51906) 2024-01-15T18:19:58.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 -- 172.21.15.2:0/3173819949 learned_addr learned my addr 172.21.15.2:0/3173819949 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:58.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82000640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f840702c0 0x7f2f8413a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f83400640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f841439f0 0x7f2f840ca610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 -- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f840702c0 msgr2=0x7f2f8413a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f2f840702c0 0x7f2f8413a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 -- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f841439f0 msgr2=0x7f2f840ca610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f2f841439f0 0x7f2f840ca610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.346+0000 7f2f82a00640 1 -- 172.21.15.2:0/3173819949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c009400 con 0x7f2f8406d000 2024-01-15T18:19:58.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.347+0000 7f2f82a00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f8406d000 0x7f2f84159950 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c400 tx=0x7f2f7400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.347+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f740052d0 con 0x7f2f8406d000 2024-01-15T18:19:58.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.347+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f74005430 con 0x7f2f8406d000 2024-01-15T18:19:58.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.347+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f74003830 con 0x7f2f8406d000 2024-01-15T18:19:58.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.347+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f84138b00 con 0x7f2f8406d000 2024-01-15T18:19:58.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.347+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f84159e90 con 0x7f2f8406d000 2024-01-15T18:19:58.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.348+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2f84070ed0 con 0x7f2f8406d000 2024-01-15T18:19:58.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.349+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2f74003990 con 0x7f2f8406d000 2024-01-15T18:19:58.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.350+0000 7f2f7aa00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f5c06c550 0x7f2f5c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.351+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f2f74088090 con 0x7f2f8406d000 2024-01-15T18:19:58.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.351+0000 7f2f7aa00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f2f5c0749d0 0x7f2f5c076e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.351+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 --> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2f5c077540 con 0x7f2f5c0749d0 2024-01-15T18:19:58.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.351+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f74088cd0 con 0x7f2f8406d000 2024-01-15T18:19:58.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.351+0000 7f2f82000640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f5c06c550 0x7f2f5c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.352+0000 7f2f83400640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f2f5c0749d0 0x7f2f5c076e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.352+0000 7f2f82000640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f5c06c550 0x7f2f5c06ea10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2f8413ee50 tx=0x7f2f6c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.356 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.353+0000 7f2f83400640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f2f5c0749d0 0x7f2f5c076e70 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.355+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== osd.4 v2:172.21.15.137:6824/3407951829 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2f5c077540 con 0x7f2f5c0749d0 2024-01-15T18:19:58.373 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.370+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 --> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2f84139430 con 0x7f2f5c0749d0 2024-01-15T18:19:58.374 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f7aa00640 1 -- 172.21.15.2:0/3173819949 <== osd.4 v2:172.21.15.137:6824/3407951829 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f2f84139430 con 0x7f2f5c0749d0 2024-01-15T18:19:58.374 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:58.375 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:58.375 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:58.375 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f2f5c0749d0 msgr2=0x7f2f5c076e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6824/3407951829,v1:172.21.15.137:6825/3407951829] conn(0x7f2f5c0749d0 0x7f2f5c076e70 crc :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f5c06c550 msgr2=0x7f2f5c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f2f5c06c550 0x7f2f5c06ea10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2f8413ee50 tx=0x7f2f6c009040 comp rx=0 tx=0).stop 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f8406d000 msgr2=0x7f2f84159950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 --2- 172.21.15.2:0/3173819949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f2f8406d000 0x7f2f84159950 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f2f7400c400 tx=0x7f2f7400c8c0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f83400640 1 -- 172.21.15.2:0/3173819949 reap_dead start 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 shutdown_connections 2024-01-15T18:19:58.376 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 >> 172.21.15.2:0/3173819949 conn(0x7f2f840b45f0 msgr2=0x7f2f8406c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:58.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 shutdown_connections 2024-01-15T18:19:58.377 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.372+0000 7f2f89a00640 1 -- 172.21.15.2:0/3173819949 wait complete. 2024-01-15T18:19:58.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-15T18:19:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.619+0000 7f30e2200640 1 Processor -- start 2024-01-15T18:19:58.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.619+0000 7f30e2200640 1 -- start start 2024-01-15T18:19:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.621+0000 7f30e2200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f30dc15c0e0 0x7f30dc0cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.621+0000 7f30e2200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f30dc15b7d0 0x7f30dc06c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.621+0000 7f30e2200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30dc15cb00 con 0x7f30dc15c4c0 2024-01-15T18:19:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.621+0000 7f30e2200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30dc0ccec0 con 0x7f30dc15c0e0 2024-01-15T18:19:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.621+0000 7f30e2200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30dc0cd1a0 con 0x7f30dc15b7d0 2024-01-15T18:19:58.625 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30db400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f30dc15c0e0 0x7f30dc0cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30dbe00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f30dc15c4c0 0x7f30dc15ece0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55126/0 (socket says 172.21.15.2:55126) 2024-01-15T18:19:58.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30dbe00640 1 -- 172.21.15.2:0/2622895702 learned_addr learned my addr 172.21.15.2:0/2622895702 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:58.626 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30d3e00640 1 --2- 172.21.15.2:0/2622895702 >> v2:172.21.15.137:3300/0 conn(0x7f30dc15b7d0 0x7f30dc06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30daa00640 1 -- 172.21.15.2:0/2622895702 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615162636 0 0) 0x7f30dc15cb00 con 0x7f30dc15c4c0 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30daa00640 1 -- 172.21.15.2:0/2622895702 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30bc003520 con 0x7f30dc15c4c0 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30db400640 1 -- 172.21.15.2:0/2622895702 >> v2:172.21.15.137:3300/0 conn(0x7f30dc15b7d0 msgr2=0x7f30dc06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30db400640 1 --2- 172.21.15.2:0/2622895702 >> v2:172.21.15.137:3300/0 conn(0x7f30dc15b7d0 0x7f30dc06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30db400640 1 -- 172.21.15.2:0/2622895702 >> v1:172.21.15.2:6789/0 conn(0x7f30dc15c4c0 legacy=0x7f30dc15ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30db400640 1 -- 172.21.15.2:0/2622895702 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30dc0cd480 con 0x7f30dc15c0e0 2024-01-15T18:19:58.627 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.622+0000 7f30d3e00640 1 --2- 172.21.15.2:0/2622895702 >> v2:172.21.15.137:3300/0 conn(0x7f30dc15b7d0 0x7f30dc06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.624+0000 7f30db400640 1 --2- 172.21.15.2:0/2622895702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15c0e0 0x7f30dc0cc5c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f30dc072830 tx=0x7f30cc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729a0aca740e7bd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.624+0000 7f30daa00640 1 -- 172.21.15.2:0/2622895702 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc031830 con 0x7f30dc15c0e0 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.624+0000 7f30daa00640 1 -- 172.21.15.2:0/2622895702 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc031990 con 0x7f30dc15c0e0 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.624+0000 7f30daa00640 1 -- 172.21.15.2:0/2622895702 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc031d00 con 0x7f30dc15c0e0 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 -- 172.21.15.2:0/2622895702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15c0e0 msgr2=0x7f30dc0cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 --2- 172.21.15.2:0/2622895702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15c0e0 0x7f30dc0cc5c0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f30dc072830 tx=0x7f30cc02b8d0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 -- 172.21.15.2:0/2622895702 shutdown_connections 2024-01-15T18:19:58.629 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 --2- 172.21.15.2:0/2622895702 >> v2:172.21.15.137:3300/0 conn(0x7f30dc15b7d0 0x7f30dc06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 --2- 172.21.15.2:0/2622895702 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15c0e0 0x7f30dc0cc5c0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 -- 172.21.15.2:0/2622895702 >> 172.21.15.2:0/2622895702 conn(0x7f30dc06da20 msgr2=0x7f30dc06a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:58.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 -- 172.21.15.2:0/2622895702 shutdown_connections 2024-01-15T18:19:58.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.625+0000 7f30e2200640 1 -- 172.21.15.2:0/2622895702 wait complete. 2024-01-15T18:19:58.630 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.627+0000 7f30e2200640 1 Processor -- start 2024-01-15T18:19:58.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.627+0000 7f30e2200640 1 -- start start 2024-01-15T18:19:58.631 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.628+0000 7f30e2200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 0x7f30dc13c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.628+0000 7f30e2200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30dc15c0e0 0x7f30dc14d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.628+0000 7f30e2200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30dc15c4c0 0x7f30dc15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.628+0000 7f30e2200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc13d540 con 0x7f30dc15b7d0 2024-01-15T18:19:58.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.628+0000 7f30e2200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc13a700 con 0x7f30dc15c4c0 2024-01-15T18:19:58.632 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.628+0000 7f30e2200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30dc13c930 con 0x7f30dc15c0e0 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 0x7f30dc13c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 0x7f30dc13c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51932/0 (socket says 172.21.15.2:51932) 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 -- 172.21.15.2:0/1560924030 learned_addr learned my addr 172.21.15.2:0/1560924030 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30dbe00640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30dc15c4c0 0x7f30dc15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30d3e00640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30dc15c0e0 0x7f30dc14d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 -- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30dc15c0e0 msgr2=0x7f30dc14d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30dc15c0e0 0x7f30dc14d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.634 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 -- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30dc15c4c0 msgr2=0x7f30dc15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30dc15c4c0 0x7f30dc15ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30db400640 1 -- 172.21.15.2:0/1560924030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30cc009400 con 0x7f30dc15b7d0 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.629+0000 7f30dbe00640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30dc15c4c0 0x7f30dc15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.630+0000 7f30db400640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 0x7f30dc13c010 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f30cc02b910 tx=0x7f30cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.630+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc03a830 con 0x7f30dc15b7d0 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.630+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30cc03a990 con 0x7f30dc15b7d0 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.630+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30cc03ad20 con 0x7f30dc15b7d0 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.630+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30dc13cc10 con 0x7f30dc15b7d0 2024-01-15T18:19:58.635 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.630+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30dc13d1d0 con 0x7f30dc15b7d0 2024-01-15T18:19:58.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.632+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30dc0686f0 con 0x7f30dc15b7d0 2024-01-15T18:19:58.636 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.632+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f30cc04d440 con 0x7f30dc15b7d0 2024-01-15T18:19:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.633+0000 7f30d3400640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30b406c550 0x7f30b406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.633+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f30cc0aac00 con 0x7f30dc15b7d0 2024-01-15T18:19:58.637 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.634+0000 7f30d3e00640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30b406c550 0x7f30b406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:58.638 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.635+0000 7f30d3e00640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30b406c550 0x7f30b406ea10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30dc0b43c0 tx=0x7f30c4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:58.641 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.638+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30cc077140 con 0x7f30dc15b7d0 2024-01-15T18:19:58.836 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.832+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f30dc13d680 con 0x7f30b406c550 2024-01-15T18:19:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.834+0000 7f30d3400640 1 -- 172.21.15.2:0/1560924030 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f30c400b6b0 con 0x7f30b406c550 2024-01-15T18:19:58.837 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 4 to deep-scrub 2024-01-15T18:19:58.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.836+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30b406c550 msgr2=0x7f30b406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.836+0000 7f30e2200640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30b406c550 0x7f30b406ea10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f30dc0b43c0 tx=0x7f30c4008040 comp rx=0 tx=0).stop 2024-01-15T18:19:58.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.836+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 msgr2=0x7f30dc13c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:58.840 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.836+0000 7f30e2200640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 0x7f30dc13c010 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f30cc02b910 tx=0x7f30cc0046c0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 shutdown_connections 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f30b406c550 0x7f30b406ea10 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f30dc15c4c0 0x7f30dc15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f30dc15c0e0 0x7f30dc14d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 --2- 172.21.15.2:0/1560924030 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f30dc15b7d0 0x7f30dc13c010 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 >> 172.21.15.2:0/1560924030 conn(0x7f30dc06da20 msgr2=0x7f30dc0713a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 shutdown_connections 2024-01-15T18:19:58.841 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:58.837+0000 7f30e2200640 1 -- 172.21.15.2:0/1560924030 wait complete. 2024-01-15T18:19:58.858 INFO:tasks.ceph:Scrubbing osd.5 2024-01-15T18:19:58.859 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:59.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.082+0000 7f627e800640 1 Processor -- start 2024-01-15T18:19:59.085 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.082+0000 7f627e800640 1 -- start start 2024-01-15T18:19:59.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f627e800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6278144c00 0x7f62780cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f627e800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62780ce230 con 0x7f627806d000 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f627e800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62780ce510 con 0x7f62780cdd70 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f627e800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62780ce7f0 con 0x7f6278144c00 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f6277400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6278144c00 0x7f62780cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f6277400640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f6278144c00 0x7f62780cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51936/0 (socket says 172.21.15.2:51936) 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f6277e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f627806d000 0x7f6278070b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55136/0 (socket says 172.21.15.2:55136) 2024-01-15T18:19:59.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.084+0000 7f6277400640 1 -- 172.21.15.2:0/3442874908 learned_addr learned my addr 172.21.15.2:0/3442874908 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:59.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.085+0000 7f6277400640 1 -- 172.21.15.2:0/3442874908 >> v1:172.21.15.137:6789/0 conn(0x7f62780cdd70 legacy=0x7f627806c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:59.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.085+0000 7f6277400640 1 -- 172.21.15.2:0/3442874908 >> v1:172.21.15.2:6789/0 conn(0x7f627806d000 legacy=0x7f6278070b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.085+0000 7f6277400640 1 -- 172.21.15.2:0/3442874908 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62780705f0 con 0x7f6278144c00 2024-01-15T18:19:59.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.086+0000 7f6277400640 1 --2- 172.21.15.2:0/3442874908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278144c00 0x7f62780cd830 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f626802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d5fc3585ea9efd server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.086+0000 7f6276a00640 1 -- 172.21.15.2:0/3442874908 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268030860 con 0x7f6278144c00 2024-01-15T18:19:59.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.086+0000 7f6276a00640 1 -- 172.21.15.2:0/3442874908 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62680309c0 con 0x7f6278144c00 2024-01-15T18:19:59.090 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.086+0000 7f6276a00640 1 -- 172.21.15.2:0/3442874908 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268030d50 con 0x7f6278144c00 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 -- 172.21.15.2:0/3442874908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278144c00 msgr2=0x7f62780cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 --2- 172.21.15.2:0/3442874908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278144c00 0x7f62780cd830 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f626802b9a0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 -- 172.21.15.2:0/3442874908 shutdown_connections 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 --2- 172.21.15.2:0/3442874908 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f6278144c00 0x7f62780cd830 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 -- 172.21.15.2:0/3442874908 >> 172.21.15.2:0/3442874908 conn(0x7f62780b45f0 msgr2=0x7f62780b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 -- 172.21.15.2:0/3442874908 shutdown_connections 2024-01-15T18:19:59.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.087+0000 7f627e800640 1 -- 172.21.15.2:0/3442874908 wait complete. 2024-01-15T18:19:59.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.089+0000 7f627e800640 1 Processor -- start 2024-01-15T18:19:59.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.089+0000 7f627e800640 1 -- start start 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f627e800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f627806d000 0x7f627812fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f627e800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62780cdd70 0x7f6278141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f627e800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278144c00 0x7f62781422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f627e800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6278142810 con 0x7f627806d000 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f627e800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f627813bbd0 con 0x7f62780cdd70 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f627e800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627812edc0 con 0x7f6278144c00 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f6277400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f627806d000 0x7f627812fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f6277400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f627806d000 0x7f627812fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51948/0 (socket says 172.21.15.2:51948) 2024-01-15T18:19:59.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.090+0000 7f6277400640 1 -- 172.21.15.2:0/2248093754 learned_addr learned my addr 172.21.15.2:0/2248093754 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:59.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f6277400640 1 -- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278144c00 msgr2=0x7f62781422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f6277400640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f6278144c00 0x7f62781422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f6277400640 1 -- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62780cdd70 msgr2=0x7f6278141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f6277400640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f62780cdd70 0x7f6278141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f6277400640 1 -- 172.21.15.2:0/2248093754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62680094d0 con 0x7f627806d000 2024-01-15T18:19:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f6277400640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f627806d000 0x7f627812fe60 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f6268004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268030810 con 0x7f627806d000 2024-01-15T18:19:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6268039d90 con 0x7f627806d000 2024-01-15T18:19:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f627812f0a0 con 0x7f627806d000 2024-01-15T18:19:59.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.091+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268043610 con 0x7f627806d000 2024-01-15T18:19:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.092+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f627812f660 con 0x7f627806d000 2024-01-15T18:19:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.093+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f62780686f0 con 0x7f627806d000 2024-01-15T18:19:59.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.094+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f626804c3b0 con 0x7f627806d000 2024-01-15T18:19:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.095+0000 7f626fe00640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c550 0x7f625006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.095+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f62680ab2f0 con 0x7f627806d000 2024-01-15T18:19:59.098 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.095+0000 7f626fe00640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7f62500749d0 0x7f6250076e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.095+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 --> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6250077540 con 0x7f62500749d0 2024-01-15T18:19:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.095+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62680ac020 con 0x7f627806d000 2024-01-15T18:19:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.096+0000 7f626e800640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c550 0x7f625006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.096+0000 7f6277e00640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7f62500749d0 0x7f6250076e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.097+0000 7f626e800640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c550 0x7f625006ea10 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f627813c390 tx=0x7f6260007d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.097+0000 7f6277e00640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7f62500749d0 0x7f6250076e70 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.100+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== osd.5 v2:172.21.15.137:6800/915165616 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6250077540 con 0x7f62500749d0 2024-01-15T18:19:59.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.114+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 --> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f627815c450 con 0x7f62500749d0 2024-01-15T18:19:59.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f626fe00640 1 -- 172.21.15.2:0/2248093754 <== osd.5 v2:172.21.15.137:6800/915165616 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f627815c450 con 0x7f62500749d0 2024-01-15T18:19:59.119 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:59.119 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:59.119 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7f62500749d0 msgr2=0x7f6250076e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6800/915165616,v1:172.21.15.137:6801/915165616] conn(0x7f62500749d0 0x7f6250076e70 crc :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c550 msgr2=0x7f625006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f625006c550 0x7f625006ea10 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f627813c390 tx=0x7f6260007d30 comp rx=0 tx=0).stop 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f627806d000 msgr2=0x7f627812fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 --2- 172.21.15.2:0/2248093754 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f627806d000 0x7f627812fe60 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f6268004730 comp rx=0 tx=0).stop 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f6277e00640 1 -- 172.21.15.2:0/2248093754 reap_dead start 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 shutdown_connections 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.116+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 >> 172.21.15.2:0/2248093754 conn(0x7f62780b45f0 msgr2=0x7f627815b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.117+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 shutdown_connections 2024-01-15T18:19:59.121 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.117+0000 7f627e800640 1 -- 172.21.15.2:0/2248093754 wait complete. 2024-01-15T18:19:59.136 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-15T18:19:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.350+0000 7f9ad4a00640 1 Processor -- start 2024-01-15T18:19:59.354 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.351+0000 7f9ad4a00640 1 -- start start 2024-01-15T18:19:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.352+0000 7f9ad4a00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f9ad015cf90 0x7f9ad015d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.352+0000 7f9ad4a00640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00d05e0 con 0x7f9ad00b2d90 2024-01-15T18:19:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.352+0000 7f9ad4a00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad00d08c0 con 0x7f9ad00b3200 2024-01-15T18:19:59.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.352+0000 7f9ad4a00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad00d13a0 con 0x7f9ad015cf90 2024-01-15T18:19:59.357 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.352+0000 7f9acea00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f9ad00b2d90 0x7f9ad00cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55142/0 (socket says 172.21.15.2:55142) 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.352+0000 7f9acea00640 1 -- 172.21.15.2:0/2568150207 learned_addr learned my addr 172.21.15.2:0/2568150207 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086179913 0 0) 0x7f9ad00d05e0 con 0x7f9ad00b2d90 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9acf400640 1 --2- 172.21.15.2:0/2568150207 >> v2:172.21.15.137:3300/0 conn(0x7f9ad015cf90 0x7f9ad015d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab0003520 con 0x7f9ad00b2d90 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1372467572 0 0) 0x7f9ab0003520 con 0x7f9ad00b2d90 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 >> v2:172.21.15.137:3300/0 conn(0x7f9ad015cf90 msgr2=0x7f9ad015d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 --2- 172.21.15.2:0/2568150207 >> v2:172.21.15.137:3300/0 conn(0x7f9ad015cf90 0x7f9ad015d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 >> v1:172.21.15.137:6789/0 conn(0x7f9ad00b3200 legacy=0x7f9ad00d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.358 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00d0ce0 con 0x7f9ad00b2d90 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f9ab8001670 con 0x7f9ad00b2d90 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 >> v1:172.21.15.2:6789/0 conn(0x7f9ad00b2d90 legacy=0x7f9ad00cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ab00039c0 0x7f9ab0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ab00047c0 0x7f9ab0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00051d0 0x7f9ab000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00d13a0 con 0x7f9ab00047c0 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab000c120 con 0x7f9ab00051d0 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.353+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab000c2e0 con 0x7f9ab00039c0 2024-01-15T18:19:59.359 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acea00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ab00047c0 0x7f9ab0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acfe00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00051d0 0x7f9ab000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acf400640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ab00039c0 0x7f9ab0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acea00640 1 -- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ab00039c0 msgr2=0x7f9ab0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acea00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ab00039c0 0x7f9ab0003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acea00640 1 -- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00051d0 msgr2=0x7f9ab000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acea00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00051d0 0x7f9ab000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acea00640 1 -- 172.21.15.2:0/2568150207 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab0003520 con 0x7f9ab00047c0 2024-01-15T18:19:59.360 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.354+0000 7f9acfe00640 1 -- 172.21.15.2:0/2568150207 reap_dead start 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.355+0000 7f9acea00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ab00047c0 0x7f9ab0004c40 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8007fb0 tx=0x7f9ab00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4282e67235ee4c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.355+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8005ad0 con 0x7f9ab00047c0 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.355+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab8005c50 con 0x7f9ab00047c0 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.355+0000 7f9ace000640 1 -- 172.21.15.2:0/2568150207 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802c7c0 con 0x7f9ab00047c0 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 -- 172.21.15.2:0/2568150207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ab00047c0 msgr2=0x7f9ab0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ab00047c0 0x7f9ab0004c40 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8007fb0 tx=0x7f9ab00057f0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 -- 172.21.15.2:0/2568150207 shutdown_connections 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/2568150207 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ab00047c0 0x7f9ab0004c40 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.362 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 -- 172.21.15.2:0/2568150207 >> 172.21.15.2:0/2568150207 conn(0x7f9ad008cad0 msgr2=0x7f9ad009c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 -- 172.21.15.2:0/2568150207 shutdown_connections 2024-01-15T18:19:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.356+0000 7f9ad4a00640 1 -- 172.21.15.2:0/2568150207 wait complete. 2024-01-15T18:19:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.358+0000 7f9ad4a00640 1 Processor -- start 2024-01-15T18:19:59.363 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.358+0000 7f9ad4a00640 1 -- start start 2024-01-15T18:19:59.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.359+0000 7f9ad4a00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00047c0 0x7f9ad0159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.364 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.359+0000 7f9ad4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 0x7f9ad00b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.359+0000 7f9ad4a00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ad00ca510 0x7f9ad00ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.359+0000 7f9ad4a00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00b2690 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.359+0000 7f9ad4a00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad0159990 con 0x7f9ab00047c0 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.359+0000 7f9ad4a00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad00cb2d0 con 0x7f9ad00ca510 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 0x7f9ad00b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 0x7f9ad00b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51968/0 (socket says 172.21.15.2:51968) 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 -- 172.21.15.2:0/3303286007 learned_addr learned my addr 172.21.15.2:0/3303286007 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:59.365 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 -- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ad00ca510 msgr2=0x7f9ad00ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ad00ca510 0x7f9ad00ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 -- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00047c0 msgr2=0x7f9ad0159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00047c0 0x7f9ad0159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 -- 172.21.15.2:0/3303286007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad00cb5b0 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.360+0000 7f9acea00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 0x7f9ad00b2150 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8007fb0 tx=0x7f9ab802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.361+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab803c040 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.361+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab802cc90 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.361+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8007070 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.361+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad00cb890 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.366 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.361+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad00cbe50 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.362+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad0074770 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.363+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9ab8032070 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.367 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.364+0000 7f9ac7400640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9aa806c580 0x7f9aa806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.364+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f9ab80aa110 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.368 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.365+0000 7f9acf400640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9aa806c580 0x7f9aa806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.369 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.366+0000 7f9acf400640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9aa806c580 0x7f9aa806ea40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0073010 tx=0x7f9ac0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.372 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.369+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ab8076770 con 0x7f9ad00b1cf0 2024-01-15T18:19:59.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.563+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ad00678e0 con 0x7f9aa806c580 2024-01-15T18:19:59.567 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.564+0000 7f9ac7400640 1 -- 172.21.15.2:0/3303286007 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9ad00678e0 con 0x7f9aa806c580 2024-01-15T18:19:59.567 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 5 to deep-scrub 2024-01-15T18:19:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9aa806c580 msgr2=0x7f9aa806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9aa806c580 0x7f9aa806ea40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f9ad0073010 tx=0x7f9ac0008040 comp rx=0 tx=0).stop 2024-01-15T18:19:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 msgr2=0x7f9ad00b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 0x7f9ad00b2150 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8007fb0 tx=0x7f9ab802e070 comp rx=0 tx=0).stop 2024-01-15T18:19:59.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 shutdown_connections 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f9aa806c580 0x7f9aa806ea40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f9ad00ca510 0x7f9ad00ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f9ad00b1cf0 0x7f9ad00b2150 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 --2- 172.21.15.2:0/3303286007 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f9ab00047c0 0x7f9ad0159450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.566+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 >> 172.21.15.2:0/3303286007 conn(0x7f9ad008cad0 msgr2=0x7f9ad00cfa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.567+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 shutdown_connections 2024-01-15T18:19:59.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.567+0000 7f9ad4a00640 1 -- 172.21.15.2:0/3303286007 wait complete. 2024-01-15T18:19:59.590 INFO:tasks.ceph:Scrubbing osd.6 2024-01-15T18:19:59.590 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:19:59.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.814+0000 7f0fbca00640 1 Processor -- start 2024-01-15T18:19:59.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.814+0000 7f0fbca00640 1 -- start start 2024-01-15T18:19:59.819 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fbca00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0fb4144c00 0x7f0fb40cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb406c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fbca00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb40ce230 con 0x7f0fb415c450 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fbca00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb40ce510 con 0x7f0fb4144c00 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fbca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40ce7f0 con 0x7f0fb40cdd70 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fba000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0fb4144c00 0x7f0fb40cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fba000640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0fb4144c00 0x7f0fb40cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51972/0 (socket says 172.21.15.2:51972) 2024-01-15T18:19:59.820 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fba000640 1 -- 172.21.15.2:0/141604246 learned_addr learned my addr 172.21.15.2:0/141604246 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:59.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.816+0000 7f0fb9600640 1 --2- 172.21.15.2:0/141604246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb406c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.817+0000 7f0fba000640 1 -- 172.21.15.2:0/141604246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 msgr2=0x7f0fb406c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:59.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.817+0000 7f0fba000640 1 --2- 172.21.15.2:0/141604246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb406c8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.821 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.817+0000 7f0fba000640 1 -- 172.21.15.2:0/141604246 >> v1:172.21.15.137:6789/0 conn(0x7f0fb415c450 legacy=0x7f0fb40b0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:19:59.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.817+0000 7f0fba000640 1 -- 172.21.15.2:0/141604246 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb40705f0 con 0x7f0fb4144c00 2024-01-15T18:19:59.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.818+0000 7f0fba000640 1 --2- 172.21.15.2:0/141604246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb40cd830 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009e50 tx=0x7f0fa402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69903f14d3fb6df3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.818+0000 7f0fb8c00640 1 -- 172.21.15.2:0/141604246 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa402ca70 con 0x7f0fb4144c00 2024-01-15T18:19:59.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.818+0000 7f0fb8c00640 1 -- 172.21.15.2:0/141604246 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa402cbd0 con 0x7f0fb4144c00 2024-01-15T18:19:59.822 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.818+0000 7f0fb8c00640 1 -- 172.21.15.2:0/141604246 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40343f0 con 0x7f0fb4144c00 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 -- 172.21.15.2:0/141604246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 msgr2=0x7f0fb40cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 --2- 172.21.15.2:0/141604246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb40cd830 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009e50 tx=0x7f0fa402bed0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 -- 172.21.15.2:0/141604246 shutdown_connections 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 --2- 172.21.15.2:0/141604246 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb406c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 --2- 172.21.15.2:0/141604246 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb40cd830 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 -- 172.21.15.2:0/141604246 >> 172.21.15.2:0/141604246 conn(0x7f0fb40b45f0 msgr2=0x7f0fb40b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 -- 172.21.15.2:0/141604246 shutdown_connections 2024-01-15T18:19:59.823 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.819+0000 7f0fbca00640 1 -- 172.21.15.2:0/141604246 wait complete. 2024-01-15T18:19:59.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.821+0000 7f0fbca00640 1 Processor -- start 2024-01-15T18:19:59.824 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.821+0000 7f0fbca00640 1 -- start start 2024-01-15T18:19:59.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.825 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb413d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fbca00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb415c450 0x7f0fb41405f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fbca00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb413d190 con 0x7f0fb4144c00 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fbca00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb413c8a0 con 0x7f0fb415c450 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fbca00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb413f150 con 0x7f0fb40cdd70 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fb9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb413d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fb9600640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb413d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51982/0 (socket says 172.21.15.2:51982) 2024-01-15T18:19:59.826 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.822+0000 7f0fb9600640 1 -- 172.21.15.2:0/311359657 learned_addr learned my addr 172.21.15.2:0/311359657 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:19:59.827 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fba000640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fbaa00640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb415c450 0x7f0fb41405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb9600640 1 -- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 msgr2=0x7f0fb4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb9600640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb9600640 1 -- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb415c450 msgr2=0x7f0fb41405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb9600640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0fb415c450 0x7f0fb41405f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb9600640 1 -- 172.21.15.2:0/311359657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa4009b30 con 0x7f0fb4144c00 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fba000640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0fb40cdd70 0x7f0fb4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb9600640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb413d620 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00c7f0 tx=0x7f0f9c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.828 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c004220 con 0x7f0fb4144c00 2024-01-15T18:19:59.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c0043a0 con 0x7f0fb4144c00 2024-01-15T18:19:59.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c007860 con 0x7f0fb4144c00 2024-01-15T18:19:59.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.823+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb413cb40 con 0x7f0fb4144c00 2024-01-15T18:19:59.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.824+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb40712c0 con 0x7f0fb4144c00 2024-01-15T18:19:59.829 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.825+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0fb406ce30 con 0x7f0fb4144c00 2024-01-15T18:19:59.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.826+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0f9c007bd0 con 0x7f0fb4144c00 2024-01-15T18:19:59.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.827+0000 7f0fb2000640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0f8c06c580 0x7f0f8c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.827+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f0f9c088830 con 0x7f0fb4144c00 2024-01-15T18:19:59.830 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.827+0000 7f0fb2000640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f0f8c074a00 0x7f0f8c076ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:19:59.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.827+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 --> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f8c077570 con 0x7f0f8c074a00 2024-01-15T18:19:59.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.827+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f9c089560 con 0x7f0fb4144c00 2024-01-15T18:19:59.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.827+0000 7f0fba000640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0f8c06c580 0x7f0f8c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.831 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.828+0000 7f0fbaa00640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f0f8c074a00 0x7f0f8c076ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:19:59.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.828+0000 7f0fba000640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0f8c06c580 0x7f0f8c06ea40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4035000 tx=0x7f0fa4030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.832 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.829+0000 7f0fbaa00640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f0f8c074a00 0x7f0f8c076ea0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:19:59.834 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.831+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== osd.6 v2:172.21.15.137:6808/1848728157 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0f8c077570 con 0x7f0f8c074a00 2024-01-15T18:19:59.849 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.846+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 --> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0fb40714b0 con 0x7f0f8c074a00 2024-01-15T18:19:59.850 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fb2000640 1 -- 172.21.15.2:0/311359657 <== osd.6 v2:172.21.15.137:6808/1848728157 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0fb40714b0 con 0x7f0f8c074a00 2024-01-15T18:19:59.850 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:19:59.850 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:19:59.850 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:19:59.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f0f8c074a00 msgr2=0x7f0f8c076ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fbca00640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6808/1848728157,v1:172.21.15.137:6809/1848728157] conn(0x7f0f8c074a00 0x7f0f8c076ea0 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0f8c06c580 msgr2=0x7f0f8c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fbca00640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0f8c06c580 0x7f0f8c06ea40 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4035000 tx=0x7f0fa4030000 comp rx=0 tx=0).stop 2024-01-15T18:19:59.851 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 msgr2=0x7f0fb413d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:19:59.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.847+0000 7f0fbca00640 1 --2- 172.21.15.2:0/311359657 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0fb4144c00 0x7f0fb413d620 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c00c7f0 tx=0x7f0f9c00ccb0 comp rx=0 tx=0).stop 2024-01-15T18:19:59.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.848+0000 7f0fbaa00640 1 -- 172.21.15.2:0/311359657 reap_dead start 2024-01-15T18:19:59.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.848+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 shutdown_connections 2024-01-15T18:19:59.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.848+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 >> 172.21.15.2:0/311359657 conn(0x7f0fb40b45f0 msgr2=0x7f0fb41421b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:19:59.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.848+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 shutdown_connections 2024-01-15T18:19:59.852 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:19:59.848+0000 7f0fbca00640 1 -- 172.21.15.2:0/311359657 wait complete. 2024-01-15T18:19:59.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-15T18:20:00.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.085+0000 7f0219e00640 1 Processor -- start 2024-01-15T18:20:00.089 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.085+0000 7f0219e00640 1 -- start start 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.087+0000 7f0219e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f021415a040 0x7f021415a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.087+0000 7f0219e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f02140b1b40 0x7f021406db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.087+0000 7f0219e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140b2ec0 con 0x7f021406e050 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.087+0000 7f0219e00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021406e4a0 con 0x7f021415a040 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.087+0000 7f0219e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02140b22b0 con 0x7f02140b1b40 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.087+0000 7f0218a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f021415a040 0x7f021415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0218a00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f021415a040 0x7f021415a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51984/0 (socket says 172.21.15.2:51984) 2024-01-15T18:20:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0218a00640 1 -- 172.21.15.2:0/3105086122 learned_addr learned my addr 172.21.15.2:0/3105086122 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0218a00640 1 -- 172.21.15.2:0/3105086122 >> v2:172.21.15.137:3300/0 conn(0x7f02140b1b40 msgr2=0x7f021406db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0213e00640 1 --2- 172.21.15.2:0/3105086122 >> v2:172.21.15.137:3300/0 conn(0x7f02140b1b40 0x7f021406db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0218a00640 1 --2- 172.21.15.2:0/3105086122 >> v2:172.21.15.137:3300/0 conn(0x7f02140b1b40 0x7f021406db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0218a00640 1 -- 172.21.15.2:0/3105086122 >> v1:172.21.15.137:6789/0 conn(0x7f021406e050 legacy=0x7f021415da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0218a00640 1 -- 172.21.15.2:0/3105086122 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140b2590 con 0x7f021415a040 2024-01-15T18:20:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.088+0000 7f0213e00640 1 --2- 172.21.15.2:0/3105086122 >> v2:172.21.15.137:3300/0 conn(0x7f02140b1b40 0x7f021406db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:20:00.093 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.090+0000 7f0218a00640 1 --2- 172.21.15.2:0/3105086122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021415a040 0x7f021415a420 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f02140cfcb0 tx=0x7f020402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38e64df04caaef4 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.090+0000 7f0213400640 1 -- 172.21.15.2:0/3105086122 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040317e0 con 0x7f021415a040 2024-01-15T18:20:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.090+0000 7f0213400640 1 -- 172.21.15.2:0/3105086122 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204031940 con 0x7f021415a040 2024-01-15T18:20:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.090+0000 7f0213400640 1 -- 172.21.15.2:0/3105086122 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204031cd0 con 0x7f021415a040 2024-01-15T18:20:00.094 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 -- 172.21.15.2:0/3105086122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021415a040 msgr2=0x7f021415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 --2- 172.21.15.2:0/3105086122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021415a040 0x7f021415a420 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f02140cfcb0 tx=0x7f020402b880 comp rx=0 tx=0).stop 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 -- 172.21.15.2:0/3105086122 shutdown_connections 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 --2- 172.21.15.2:0/3105086122 >> v2:172.21.15.137:3300/0 conn(0x7f02140b1b40 0x7f021406db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 --2- 172.21.15.2:0/3105086122 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021415a040 0x7f021415a420 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 -- 172.21.15.2:0/3105086122 >> 172.21.15.2:0/3105086122 conn(0x7f02140b0840 msgr2=0x7f02140b0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 -- 172.21.15.2:0/3105086122 shutdown_connections 2024-01-15T18:20:00.095 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.091+0000 7f0219e00640 1 -- 172.21.15.2:0/3105086122 wait complete. 2024-01-15T18:20:00.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.093+0000 7f0219e00640 1 Processor -- start 2024-01-15T18:20:00.096 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.093+0000 7f0219e00640 1 -- start start 2024-01-15T18:20:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0219e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 0x7f021414f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0219e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02140b1b40 0x7f02141f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0219e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021415a040 0x7f021414b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0219e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021414c240 con 0x7f021406e050 2024-01-15T18:20:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0219e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214166e80 con 0x7f02140b1b40 2024-01-15T18:20:00.097 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0219e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021414bd20 con 0x7f021415a040 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0218a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 0x7f021414f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0218a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 0x7f021414f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:51998/0 (socket says 172.21.15.2:51998) 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.094+0000 7f0218a00640 1 -- 172.21.15.2:0/741907732 learned_addr learned my addr 172.21.15.2:0/741907732 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0218a00640 1 -- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021415a040 msgr2=0x7f021414b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0218a00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021415a040 0x7f021414b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0218a00640 1 -- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02140b1b40 msgr2=0x7f02141f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.099 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0213e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02140b1b40 0x7f02141f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0218a00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02140b1b40 0x7f02141f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0218a00640 1 -- 172.21.15.2:0/741907732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02040094d0 con 0x7f021406e050 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0213e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02140b1b40 0x7f02141f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0218a00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 0x7f021414f570 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0204000c00 tx=0x7f02040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204032900 con 0x7f021406e050 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204032a60 con 0x7f021406e050 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204032e10 con 0x7f021406e050 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.095+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021414c590 con 0x7f021406e050 2024-01-15T18:20:00.100 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.096+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f021414cb10 con 0x7f021406e050 2024-01-15T18:20:00.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.097+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0214074770 con 0x7f021406e050 2024-01-15T18:20:00.101 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.098+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f020404e450 con 0x7f021406e050 2024-01-15T18:20:00.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.099+0000 7f0210c00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f01ec06c550 0x7f01ec06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.102 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.099+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f02040aab10 con 0x7f021406e050 2024-01-15T18:20:00.103 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.100+0000 7f0213e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f01ec06c550 0x7f01ec06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.104 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.101+0000 7f0213e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f01ec06c550 0x7f01ec06ea10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f021415d0d0 tx=0x7f01fc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.107 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.104+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0204077480 con 0x7f021406e050 2024-01-15T18:20:00.306 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.303+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f02140b1fa0 con 0x7f01ec06c550 2024-01-15T18:20:00.307 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.305+0000 7f0210c00640 1 -- 172.21.15.2:0/741907732 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f01fc00c040 con 0x7f01ec06c550 2024-01-15T18:20:00.307 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 6 to deep-scrub 2024-01-15T18:20:00.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.306+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f01ec06c550 msgr2=0x7f01ec06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.306+0000 7f0219e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f01ec06c550 0x7f01ec06ea10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f021415d0d0 tx=0x7f01fc008040 comp rx=0 tx=0).stop 2024-01-15T18:20:00.310 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.306+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 msgr2=0x7f021414f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.306+0000 7f0219e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 0x7f021414f570 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0204000c00 tx=0x7f02040046c0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 shutdown_connections 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f01ec06c550 0x7f01ec06ea10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f021415a040 0x7f021414b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f02140b1b40 0x7f02141f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 --2- 172.21.15.2:0/741907732 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f021406e050 0x7f021414f570 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 >> 172.21.15.2:0/741907732 conn(0x7f02140b0840 msgr2=0x7f021406d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 shutdown_connections 2024-01-15T18:20:00.311 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.307+0000 7f0219e00640 1 -- 172.21.15.2:0/741907732 wait complete. 2024-01-15T18:20:00.330 INFO:tasks.ceph:Scrubbing osd.7 2024-01-15T18:20:00.331 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-15T18:20:00.563 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.560+0000 7f88b2400640 1 Processor -- start 2024-01-15T18:20:00.564 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.560+0000 7f88b2400640 1 -- start start 2024-01-15T18:20:00.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b2400640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f88ac15d270 0x7f88ac0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.565 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 0x7f88ac160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b2400640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ac06c5e0 con 0x7f88ac0b2b60 2024-01-15T18:20:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b2400640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ac06c8c0 con 0x7f88ac15d270 2024-01-15T18:20:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac071c00 con 0x7f88ac161890 2024-01-15T18:20:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b1a00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f88ac0b2b60 0x7f88ac0702f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55156/0 (socket says 172.21.15.2:55156) 2024-01-15T18:20:00.566 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.562+0000 7f88b1a00640 1 -- 172.21.15.2:0/1972075890 learned_addr learned my addr 172.21.15.2:0/1972075890 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:20:00.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88a3c00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 0x7f88ac160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88b1000640 1 --2- 172.21.15.2:0/1972075890 >> v2:172.21.15.137:3300/0 conn(0x7f88ac15d270 0x7f88ac0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852167156 0 0) 0x7f88ac06c5e0 con 0x7f88ac0b2b60 2024-01-15T18:20:00.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888c003520 con 0x7f88ac0b2b60 2024-01-15T18:20:00.568 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494466327 0 0) 0x7f888c003520 con 0x7f88ac0b2b60 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 msgr2=0x7f88ac160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 0x7f88ac160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 >> v2:172.21.15.137:3300/0 conn(0x7f88ac15d270 msgr2=0x7f88ac0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 --2- 172.21.15.2:0/1972075890 >> v2:172.21.15.137:3300/0 conn(0x7f88ac15d270 0x7f88ac0b2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac071ee0 con 0x7f88ac0b2b60 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f88a4001670 con 0x7f88ac0b2b60 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 >> v1:172.21.15.2:6789/0 conn(0x7f88ac0b2b60 legacy=0x7f88ac0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.563+0000 7f88abe00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 0x7f888c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.569 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88abe00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f888c003f40 0x7f888c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88abe00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f888c004d00 0x7f888c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac06c8c0 con 0x7f888c004d00 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac071c00 con 0x7f888c003f40 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c00bab0 con 0x7f88ac161890 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1a00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f888c004d00 0x7f888c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1000640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 0x7f888c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88a3c00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f888c003f40 0x7f888c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1a00640 1 -- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 msgr2=0x7f888c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1a00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f88ac161890 0x7f888c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.570 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1a00640 1 -- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f888c003f40 msgr2=0x7f888c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1a00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f888c003f40 0x7f888c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.571 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.564+0000 7f88b1a00640 1 -- 172.21.15.2:0/1972075890 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c003520 con 0x7f888c004d00 2024-01-15T18:20:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.565+0000 7f88b1a00640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f888c004d00 0x7f888c005180 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f88ac06fca0 tx=0x7f888c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ee2757f2f9b240 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.566+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a4005ad0 con 0x7f888c004d00 2024-01-15T18:20:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.566+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a4005c30 con 0x7f888c004d00 2024-01-15T18:20:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.566+0000 7f88abe00640 1 -- 172.21.15.2:0/1972075890 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a402c7c0 con 0x7f888c004d00 2024-01-15T18:20:00.572 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.566+0000 7f88b2400640 1 -- 172.21.15.2:0/1972075890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f888c004d00 msgr2=0x7f888c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.566+0000 7f88b2400640 1 --2- 172.21.15.2:0/1972075890 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f888c004d00 0x7f888c005180 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f88ac06fca0 tx=0x7f888c0057f0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.566+0000 7f88b1a00640 1 -- 172.21.15.2:0/1972075890 reap_dead start 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.567+0000 7f88b2400640 1 -- 172.21.15.2:0/1972075890 shutdown_connections 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.567+0000 7f88b2400640 1 -- 172.21.15.2:0/1972075890 >> 172.21.15.2:0/1972075890 conn(0x7f88ac0b0840 msgr2=0x7f88ac0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.567+0000 7f88b2400640 1 -- 172.21.15.2:0/1972075890 shutdown_connections 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.567+0000 7f88b2400640 1 -- 172.21.15.2:0/1972075890 wait complete. 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.568+0000 7f88b2400640 1 Processor -- start 2024-01-15T18:20:00.573 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.569+0000 7f88b2400640 1 -- start start 2024-01-15T18:20:00.575 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.569+0000 7f88b2400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f888c004d00 0x7f88ac0ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.569+0000 7f88b2400640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac0af060 0x7f88ac1f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.569+0000 7f88b2400640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac1f5b70 0x7f88ac1f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.569+0000 7f88b2400640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac0aec40 con 0x7f88ac0af060 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.569+0000 7f88b2400640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac1f35d0 con 0x7f88ac1f5b70 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88b2400640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ac1f6540 con 0x7f888c004d00 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac0af060 0x7f88ac1f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac0af060 0x7f88ac1f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52006/0 (socket says 172.21.15.2:52006) 2024-01-15T18:20:00.576 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 -- 172.21.15.2:0/3747995186 learned_addr learned my addr 172.21.15.2:0/3747995186 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88b1a00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac1f5b70 0x7f88ac1f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 -- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f888c004d00 msgr2=0x7f88ac0ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88b1000640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f888c004d00 0x7f88ac0ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f888c004d00 0x7f88ac0ad8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 -- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac1f5b70 msgr2=0x7f88ac1f6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac1f5b70 0x7f88ac1f6000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88a3c00640 1 -- 172.21.15.2:0/3747995186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ac0adfc0 con 0x7f88ac0af060 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.570+0000 7f88b1a00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f88ac1f5b70 0x7f88ac1f6000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:20:00.577 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.571+0000 7f88a3c00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac0af060 0x7f88ac1f3090 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f8894011590 tx=0x7f8894011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.571+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894015870 con 0x7f88ac0af060 2024-01-15T18:20:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.571+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940159d0 con 0x7f88ac0af060 2024-01-15T18:20:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.571+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894015da0 con 0x7f88ac0af060 2024-01-15T18:20:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.571+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ac0ae2a0 con 0x7f88ac0af060 2024-01-15T18:20:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.571+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ac0ae860 con 0x7f88ac0af060 2024-01-15T18:20:00.578 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.572+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f88ac070a00 con 0x7f88ac0af060 2024-01-15T18:20:00.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.573+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f889401d400 con 0x7f88ac0af060 2024-01-15T18:20:00.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.574+0000 7f88a9600640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f888406c550 0x7f888406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.574+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f8894088fe0 con 0x7f88ac0af060 2024-01-15T18:20:00.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.574+0000 7f88a9600640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7f88840749d0 0x7f8884076e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.579 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.574+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 --> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8884077540 con 0x7f88840749d0 2024-01-15T18:20:00.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.574+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_get_version_reply(handle=1 version=116) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8894089760 con 0x7f88ac0af060 2024-01-15T18:20:00.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.575+0000 7f88b1000640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f888406c550 0x7f888406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.575+0000 7f88b1a00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7f88840749d0 0x7f8884076e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.576+0000 7f88b1a00640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7f88840749d0 0x7f8884076e70 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.576+0000 7f88b1000640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f888406c550 0x7f888406ea10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f889c007380 tx=0x7f889c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.582 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.578+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== osd.7 v2:172.21.15.137:6816/463419779 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8884077540 con 0x7f88840749d0 2024-01-15T18:20:00.596 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.593+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 --> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f88ac0aeeb0 con 0x7f88840749d0 2024-01-15T18:20:00.597 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88a9600640 1 -- 172.21.15.2:0/3747995186 <== osd.7 v2:172.21.15.137:6816/463419779 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f88ac0aeeb0 con 0x7f88840749d0 2024-01-15T18:20:00.598 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-01-15T18:20:00.598 INFO:teuthology.orchestra.run.smithi002.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-15T18:20:00.598 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7f88840749d0 msgr2=0x7f8884076e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6816/463419779,v1:172.21.15.137:6817/463419779] conn(0x7f88840749d0 0x7f8884076e70 crc :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f888406c550 msgr2=0x7f888406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f888406c550 0x7f888406ea10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f889c007380 tx=0x7f889c00b040 comp rx=0 tx=0).stop 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac0af060 msgr2=0x7f88ac1f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 --2- 172.21.15.2:0/3747995186 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f88ac0af060 0x7f88ac1f3090 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f8894011590 tx=0x7f8894011a50 comp rx=0 tx=0).stop 2024-01-15T18:20:00.599 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b1a00640 1 -- 172.21.15.2:0/3747995186 reap_dead start 2024-01-15T18:20:00.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 shutdown_connections 2024-01-15T18:20:00.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 >> 172.21.15.2:0/3747995186 conn(0x7f88ac0b0840 msgr2=0x7f88ac15f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:00.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 shutdown_connections 2024-01-15T18:20:00.600 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.595+0000 7f88b2400640 1 -- 172.21.15.2:0/3747995186 wait complete. 2024-01-15T18:20:00.616 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-15T18:20:00.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.860+0000 7fcde4e00640 1 Processor -- start 2024-01-15T18:20:00.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.861+0000 7fcde4e00640 1 -- start start 2024-01-15T18:20:00.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.862+0000 7fcde4e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 0x7fcde00d2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.862+0000 7fcde4e00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7fcde00b4cd0 0x7fcde00b5140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.862+0000 7fcde4e00640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde00d2780 con 0x7fcde015d5d0 2024-01-15T18:20:00.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.863+0000 7fcde4e00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde00d1980 con 0x7fcde00b4cd0 2024-01-15T18:20:00.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.863+0000 7fcde4e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00b5680 con 0x7fcde00b8ed0 2024-01-15T18:20:00.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.863+0000 7fcddd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 0x7fcde00d2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.863+0000 7fcddd400640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 0x7fcde00d2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.137:3301/0 says I am v2:172.21.15.2:45824/0 (socket says 172.21.15.2:45824) 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.863+0000 7fcddd400640 1 -- 172.21.15.2:0/1717685352 learned_addr learned my addr 172.21.15.2:0/1717685352 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.863+0000 7fcdde800640 1 --2- 172.21.15.2:0/1717685352 >> v2:172.21.15.137:3300/0 conn(0x7fcde00b4cd0 0x7fcde00b5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.864+0000 7fcddd400640 1 -- 172.21.15.2:0/1717685352 >> v2:172.21.15.137:3300/0 conn(0x7fcde00b4cd0 msgr2=0x7fcde00b5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.864+0000 7fcddca00640 1 -- 172.21.15.2:0/1717685352 <== mon.1 v1:172.21.15.137:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931775342 0 0) 0x7fcde00d2780 con 0x7fcde015d5d0 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.864+0000 7fcddd400640 1 --2- 172.21.15.2:0/1717685352 >> v2:172.21.15.137:3300/0 conn(0x7fcde00b4cd0 0x7fcde00b5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.864+0000 7fcddd400640 1 -- 172.21.15.2:0/1717685352 >> v1:172.21.15.137:6789/0 conn(0x7fcde015d5d0 legacy=0x7fcde00d1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.864+0000 7fcddd400640 1 -- 172.21.15.2:0/1717685352 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde00d1b70 con 0x7fcde00b8ed0 2024-01-15T18:20:00.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.866+0000 7fcddd400640 1 --2- 172.21.15.2:0/1717685352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 0x7fcde00d2c10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009870 tx=0x7fcdc802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f04eae23b9de4964 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.866+0000 7fcddca00640 1 -- 172.21.15.2:0/1717685352 <== mon.2 v2:172.21.15.137:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc802f830 con 0x7fcde00b8ed0 2024-01-15T18:20:00.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.866+0000 7fcddca00640 1 -- 172.21.15.2:0/1717685352 <== mon.2 v2:172.21.15.137:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc802f990 con 0x7fcde00b8ed0 2024-01-15T18:20:00.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.866+0000 7fcddca00640 1 -- 172.21.15.2:0/1717685352 <== mon.2 v2:172.21.15.137:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc802fd20 con 0x7fcde00b8ed0 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.867+0000 7fcde4e00640 1 -- 172.21.15.2:0/1717685352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 msgr2=0x7fcde00d2c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.867+0000 7fcde4e00640 1 --2- 172.21.15.2:0/1717685352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 0x7fcde00d2c10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009870 tx=0x7fcdc802ba10 comp rx=0 tx=0).stop 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.867+0000 7fcde4e00640 1 -- 172.21.15.2:0/1717685352 shutdown_connections 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.867+0000 7fcde4e00640 1 --2- 172.21.15.2:0/1717685352 >> v2:172.21.15.137:3300/0 conn(0x7fcde00b4cd0 0x7fcde00b5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.867+0000 7fcde4e00640 1 --2- 172.21.15.2:0/1717685352 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde00b8ed0 0x7fcde00d2c10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.867+0000 7fcde4e00640 1 -- 172.21.15.2:0/1717685352 >> 172.21.15.2:0/1717685352 conn(0x7fcde0072210 msgr2=0x7fcde00cfcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.868+0000 7fcde4e00640 1 -- 172.21.15.2:0/1717685352 shutdown_connections 2024-01-15T18:20:00.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.868+0000 7fcde4e00640 1 -- 172.21.15.2:0/1717685352 wait complete. 2024-01-15T18:20:00.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.870+0000 7fcde4e00640 1 Processor -- start 2024-01-15T18:20:00.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.870+0000 7fcde4e00640 1 -- start start 2024-01-15T18:20:00.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcde4e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 0x7fcde014c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcde4e00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcde00b8ed0 0x7fcde00cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcde4e00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde015d5d0 0x7fcde00ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcde4e00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde0149e70 con 0x7fcde00b4cd0 2024-01-15T18:20:00.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcde4e00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde00aca30 con 0x7fcde00b8ed0 2024-01-15T18:20:00.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcde4e00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde014ce50 con 0x7fcde015d5d0 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcddde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 0x7fcde014c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcddde00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 0x7fcde014c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52010/0 (socket says 172.21.15.2:52010) 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.871+0000 7fcddde00640 1 -- 172.21.15.2:0/631396982 learned_addr learned my addr 172.21.15.2:0/631396982 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcdde800640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde015d5d0 0x7fcde00ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddd400640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcde00b8ed0 0x7fcde00cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddde00640 1 -- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde015d5d0 msgr2=0x7fcde00ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddde00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde015d5d0 0x7fcde00ce1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddde00640 1 -- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcde00b8ed0 msgr2=0x7fcde00cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:00.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddde00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcde00b8ed0 0x7fcde00cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:00.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddde00640 1 -- 172.21.15.2:0/631396982 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc80094d0 con 0x7fcde00b4cd0 2024-01-15T18:20:00.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcddde00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 0x7fcde014c910 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000c440 tx=0x7fcdd000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.872+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0005480 con 0x7fcde00b4cd0 2024-01-15T18:20:00.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.873+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd00035f0 con 0x7fcde00b4cd0 2024-01-15T18:20:00.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.873+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde01f3240 con 0x7fcde00b4cd0 2024-01-15T18:20:00.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.873+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00039a0 con 0x7fcde00b4cd0 2024-01-15T18:20:00.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.873+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde01f3800 con 0x7fcde00b4cd0 2024-01-15T18:20:00.878 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.874+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde00686f0 con 0x7fcde00b4cd0 2024-01-15T18:20:00.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.875+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fcdd0003b00 con 0x7fcde00b4cd0 2024-01-15T18:20:00.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.876+0000 7fcdd6000640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdb806c550 0x7fcdb806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:00.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.876+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fcdd00880a0 con 0x7fcde00b4cd0 2024-01-15T18:20:00.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.877+0000 7fcddd400640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdb806c550 0x7fcdb806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:00.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.878+0000 7fcddd400640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdb806c550 0x7fcdb806ea10 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8036000 tx=0x7fcdc8008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:00.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:00.881+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcdd00545e0 con 0x7fcde00b4cd0 2024-01-15T18:20:01.087 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.084+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fcde00d2410 con 0x7fcdb806c550 2024-01-15T18:20:01.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.086+0000 7fcdd6000640 1 -- 172.21.15.2:0/631396982 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcdc802fb00 con 0x7fcdb806c550 2024-01-15T18:20:01.088 INFO:teuthology.orchestra.run.smithi002.stderr:instructed osd(s) 7 to deep-scrub 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdb806c550 msgr2=0x7fcdb806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdb806c550 0x7fcdb806ea10 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8036000 tx=0x7fcdc8008030 comp rx=0 tx=0).stop 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 msgr2=0x7fcde014c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 0x7fcde014c910 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7fcdd000c440 tx=0x7fcdd000c900 comp rx=0 tx=0).stop 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 shutdown_connections 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fcdb806c550 0x7fcdb806ea10 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fcde015d5d0 0x7fcde00ce1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.091 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fcde00b8ed0 0x7fcde00cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 --2- 172.21.15.2:0/631396982 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fcde00b4cd0 0x7fcde014c910 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 >> 172.21.15.2:0/631396982 conn(0x7fcde0072210 msgr2=0x7fcde00b61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:01.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 shutdown_connections 2024-01-15T18:20:01.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.087+0000 7fcde4e00640 1 -- 172.21.15.2:0/631396982 wait complete. 2024-01-15T18:20:01.109 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:20:01.334 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.332+0000 7f3cd3200640 1 Processor -- start 2024-01-15T18:20:01.335 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.332+0000 7f3cd3200640 1 -- start start 2024-01-15T18:20:01.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.333+0000 7f3cd3200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3cd4141880 0x7f3cd4070fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:01.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.333+0000 7f3cd3200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f3cd415ec60 0x7f3cd415f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:01.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.333+0000 7f3cd3200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd406dae0 con 0x7f3cd40cd490 2024-01-15T18:20:01.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.333+0000 7f3cd3200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd4071f20 con 0x7f3cd4141880 2024-01-15T18:20:01.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.333+0000 7f3cd3200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd406c6b0 con 0x7f3cd415ec60 2024-01-15T18:20:01.337 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.333+0000 7f3cd1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3cd4141880 0x7f3cd4070fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:01.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3cd1e00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f3cd4141880 0x7f3cd4070fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52014/0 (socket says 172.21.15.2:52014) 2024-01-15T18:20:01.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3cd1e00640 1 -- 172.21.15.2:0/3032158457 learned_addr learned my addr 172.21.15.2:0/3032158457 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:01.338 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3ccbe00640 1 --2- 172.21.15.2:0/3032158457 >> v2:172.21.15.137:3300/0 conn(0x7f3cd415ec60 0x7f3cd415f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3cd1e00640 1 -- 172.21.15.2:0/3032158457 >> v2:172.21.15.137:3300/0 conn(0x7f3cd415ec60 msgr2=0x7f3cd415f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3cd1e00640 1 --2- 172.21.15.2:0/3032158457 >> v2:172.21.15.137:3300/0 conn(0x7f3cd415ec60 0x7f3cd415f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3cd1e00640 1 -- 172.21.15.2:0/3032158457 >> v1:172.21.15.137:6789/0 conn(0x7f3cd40cd490 legacy=0x7f3cd40cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.334+0000 7f3cd1e00640 1 -- 172.21.15.2:0/3032158457 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd4071960 con 0x7f3cd4141880 2024-01-15T18:20:01.339 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.336+0000 7f3cd1e00640 1 --2- 172.21.15.2:0/3032158457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd4141880 0x7f3cd4070fc0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40722d0 tx=0x7f3cc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d736f0d956007b server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:01.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.337+0000 7f3cd1400640 1 -- 172.21.15.2:0/3032158457 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4031870 con 0x7f3cd4141880 2024-01-15T18:20:01.340 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.337+0000 7f3cd1400640 1 -- 172.21.15.2:0/3032158457 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc40319d0 con 0x7f3cd4141880 2024-01-15T18:20:01.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.337+0000 7f3cd1400640 1 -- 172.21.15.2:0/3032158457 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4031d60 con 0x7f3cd4141880 2024-01-15T18:20:01.341 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 -- 172.21.15.2:0/3032158457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd4141880 msgr2=0x7f3cd4070fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 --2- 172.21.15.2:0/3032158457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd4141880 0x7f3cd4070fc0 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40722d0 tx=0x7f3cc402b910 comp rx=0 tx=0).stop 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 -- 172.21.15.2:0/3032158457 shutdown_connections 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 --2- 172.21.15.2:0/3032158457 >> v2:172.21.15.137:3300/0 conn(0x7f3cd415ec60 0x7f3cd415f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 --2- 172.21.15.2:0/3032158457 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd4141880 0x7f3cd4070fc0 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 -- 172.21.15.2:0/3032158457 >> 172.21.15.2:0/3032158457 conn(0x7f3cd40b0840 msgr2=0x7f3cd40b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 -- 172.21.15.2:0/3032158457 shutdown_connections 2024-01-15T18:20:01.342 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.339+0000 7f3cd3200640 1 -- 172.21.15.2:0/3032158457 wait complete. 2024-01-15T18:20:01.343 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.341+0000 7f3cd3200640 1 Processor -- start 2024-01-15T18:20:01.344 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.342+0000 7f3cd3200640 1 -- start start 2024-01-15T18:20:01.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd3200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 0x7f3cd415d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:01.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd3200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd4141880 0x7f3cd41ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:01.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd3200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd415ec60 0x7f3cd413db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:01.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd3200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd413c200 con 0x7f3cd40cd490 2024-01-15T18:20:01.346 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd3200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd415c200 con 0x7f3cd415ec60 2024-01-15T18:20:01.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd3200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd413d020 con 0x7f3cd4141880 2024-01-15T18:20:01.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 0x7f3cd415d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:01.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.343+0000 7f3cd1e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 0x7f3cd415d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:52028/0 (socket says 172.21.15.2:52028) 2024-01-15T18:20:01.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 -- 172.21.15.2:0/1247825089 learned_addr learned my addr 172.21.15.2:0/1247825089 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:01.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3ccbe00640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd4141880 0x7f3cd41ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:01.347 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd2800640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd415ec60 0x7f3cd413db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 -- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd4141880 msgr2=0x7f3cd41ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd4141880 0x7f3cd41ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 -- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd415ec60 msgr2=0x7f3cd413db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd415ec60 0x7f3cd413db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 -- 172.21.15.2:0/1247825089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40094d0 con 0x7f3cd40cd490 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd1e00640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 0x7f3cd415d840 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3cc402b950 tx=0x7f3cc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:01.349 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc403ad30 con 0x7f3cd40cd490 2024-01-15T18:20:01.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc40443f0 con 0x7f3cd40cd490 2024-01-15T18:20:01.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd41eff30 con 0x7f3cd40cd490 2024-01-15T18:20:01.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc40447a0 con 0x7f3cd40cd490 2024-01-15T18:20:01.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.344+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd413c6f0 con 0x7f3cd40cd490 2024-01-15T18:20:01.350 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.345+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd4074770 con 0x7f3cd40cd490 2024-01-15T18:20:01.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.348+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3cc404d450 con 0x7f3cd40cd490 2024-01-15T18:20:01.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.348+0000 7f3cca000640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cac06c550 0x7f3cac06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:01.351 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.348+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f3cc402d040 con 0x7f3cd40cd490 2024-01-15T18:20:01.352 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.349+0000 7f3ccbe00640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cac06c550 0x7f3cac06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:01.353 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.351+0000 7f3ccbe00640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cac06c550 0x7f3cac06ea10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3cd413c1b0 tx=0x7f3cbc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:01.355 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.352+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cc4077630 con 0x7f3cd40cd490 2024-01-15T18:20:01.556 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.553+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cd40ac350 con 0x7f3cac06c550 2024-01-15T18:20:01.580 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.577+0000 7f3cca000640 1 -- 172.21.15.2:0/1247825089 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286402 (secure 0 0 0) 0x7f3cd40ac350 con 0x7f3cac06c550 2024-01-15T18:20:01.585 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:20:01.586 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:20:01.587 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cac06c550 msgr2=0x7f3cac06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cac06c550 0x7f3cac06ea10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f3cd413c1b0 tx=0x7f3cbc008040 comp rx=0 tx=0).stop 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 msgr2=0x7f3cd415d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 0x7f3cd415d840 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f3cc402b950 tx=0x7f3cc40046c0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 shutdown_connections 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f3cac06c550 0x7f3cac06ea10 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f3cd415ec60 0x7f3cd413db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f3cd4141880 0x7f3cd41ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.588 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 --2- 172.21.15.2:0/1247825089 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f3cd40cd490 0x7f3cd415d840 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:01.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.583+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 >> 172.21.15.2:0/1247825089 conn(0x7f3cd40b0840 msgr2=0x7f3cd40afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:01.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.585+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 shutdown_connections 2024-01-15T18:20:01.589 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:01.585+0000 7f3cd3200640 1 -- 172.21.15.2:0/1247825089 wait complete. 2024-01-15T18:20:01.608 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":787,"stamp":"2024-01-15T18:19:59.960764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369942616,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1751,"num_write_kb":5246067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3232,"ondisk_log_size":3232,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11070456,"kb_used_data":10499512,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":743904264,"statfs":{"total":773094113280,"available":761757966336,"internally_reserved":0,"allocated":10751500288,"data_stored":10744941272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":584568369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":641747685,"num_objects":153,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":144,"num_read_kb":589824,"num_write":166,"num_write_kb":626712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004319"},"pg_stats":[{"pgid":"17.3f","version":"116'53","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.473424+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:19:45.473424+0000","last_peered":"2024-01-15T18:19:45.473424+0000","last_clean":"2024-01-15T18:19:45.473424+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:19:45.473424+0000","last_undegraded":"2024-01-15T18:19:45.473424+0000","last_fullsized":"2024-01-15T18:19:45.473424+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'45","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.529480+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:19:45.529480+0000","last_peered":"2024-01-15T18:19:45.529480+0000","last_clean":"2024-01-15T18:19:45.529480+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:19:45.529480+0000","last_undegraded":"2024-01-15T18:19:45.529480+0000","last_fullsized":"2024-01-15T18:19:45.529480+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.675239+0000","last_change":"2024-01-15T18:14:16.889266+0000","last_active":"2024-01-15T18:19:45.675239+0000","last_peered":"2024-01-15T18:19:45.675239+0000","last_clean":"2024-01-15T18:19:45.675239+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:19:45.675239+0000","last_undegraded":"2024-01-15T18:19:45.675239+0000","last_fullsized":"2024-01-15T18:19:45.675239+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:19:33.351118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'45","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.623557+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:19:45.623557+0000","last_peered":"2024-01-15T18:19:45.623557+0000","last_clean":"2024-01-15T18:19:45.623557+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:19:45.623557+0000","last_undegraded":"2024-01-15T18:19:45.623557+0000","last_fullsized":"2024-01-15T18:19:45.623557+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'27","reported_seq":50,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.370847+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:19:45.370847+0000","last_peered":"2024-01-15T18:19:45.370847+0000","last_clean":"2024-01-15T18:19:45.370847+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:19:45.370847+0000","last_undegraded":"2024-01-15T18:19:45.370847+0000","last_fullsized":"2024-01-15T18:19:45.370847+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'48","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.658824+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:19:45.658824+0000","last_peered":"2024-01-15T18:19:45.658824+0000","last_clean":"2024-01-15T18:19:45.658824+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:19:45.658824+0000","last_undegraded":"2024-01-15T18:19:45.658824+0000","last_fullsized":"2024-01-15T18:19:45.658824+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'43","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.409266+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:19:45.409266+0000","last_peered":"2024-01-15T18:19:45.409266+0000","last_clean":"2024-01-15T18:19:45.409266+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:19:45.409266+0000","last_undegraded":"2024-01-15T18:19:45.409266+0000","last_fullsized":"2024-01-15T18:19:45.409266+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.398127+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:19:45.398127+0000","last_peered":"2024-01-15T18:19:45.398127+0000","last_clean":"2024-01-15T18:19:45.398127+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:19:45.398127+0000","last_undegraded":"2024-01-15T18:19:45.398127+0000","last_fullsized":"2024-01-15T18:19:45.398127+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:14.013299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'47","reported_seq":71,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.832988+0000","last_change":"2024-01-15T18:14:16.882681+0000","last_active":"2024-01-15T18:19:45.832988+0000","last_peered":"2024-01-15T18:19:45.832988+0000","last_clean":"2024-01-15T18:19:45.832988+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:19:45.832988+0000","last_undegraded":"2024-01-15T18:19:45.832988+0000","last_fullsized":"2024-01-15T18:19:45.832988+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'47","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.898878+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:19:45.898878+0000","last_peered":"2024-01-15T18:19:45.898878+0000","last_clean":"2024-01-15T18:19:45.898878+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:19:45.898878+0000","last_undegraded":"2024-01-15T18:19:45.898878+0000","last_fullsized":"2024-01-15T18:19:45.898878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'47","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.835474+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:19:45.835474+0000","last_peered":"2024-01-15T18:19:45.835474+0000","last_clean":"2024-01-15T18:19:45.835474+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:19:45.835474+0000","last_undegraded":"2024-01-15T18:19:45.835474+0000","last_fullsized":"2024-01-15T18:19:45.835474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'48","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.797358+0000","last_change":"2024-01-15T18:14:16.883343+0000","last_active":"2024-01-15T18:19:45.797358+0000","last_peered":"2024-01-15T18:19:45.797358+0000","last_clean":"2024-01-15T18:19:45.797358+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:19:45.797358+0000","last_undegraded":"2024-01-15T18:19:45.797358+0000","last_fullsized":"2024-01-15T18:19:45.797358+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'38","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.393963+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:19:45.393963+0000","last_peered":"2024-01-15T18:19:45.393963+0000","last_clean":"2024-01-15T18:19:45.393963+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:19:45.393963+0000","last_undegraded":"2024-01-15T18:19:45.393963+0000","last_fullsized":"2024-01-15T18:19:45.393963+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'51","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.775084+0000","last_change":"2024-01-15T18:14:16.885758+0000","last_active":"2024-01-15T18:19:45.775084+0000","last_peered":"2024-01-15T18:19:45.775084+0000","last_clean":"2024-01-15T18:19:45.775084+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:19:45.775084+0000","last_undegraded":"2024-01-15T18:19:45.775084+0000","last_fullsized":"2024-01-15T18:19:45.775084+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:38.196288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'34","reported_seq":54,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.805166+0000","last_change":"2024-01-15T18:14:16.881329+0000","last_active":"2024-01-15T18:19:45.805166+0000","last_peered":"2024-01-15T18:19:45.805166+0000","last_clean":"2024-01-15T18:19:45.805166+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:19:45.805166+0000","last_undegraded":"2024-01-15T18:19:45.805166+0000","last_fullsized":"2024-01-15T18:19:45.805166+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'42","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.486238+0000","last_change":"2024-01-15T18:14:16.887608+0000","last_active":"2024-01-15T18:19:45.486238+0000","last_peered":"2024-01-15T18:19:45.486238+0000","last_clean":"2024-01-15T18:19:45.486238+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:19:45.486238+0000","last_undegraded":"2024-01-15T18:19:45.486238+0000","last_fullsized":"2024-01-15T18:19:45.486238+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'44","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.750575+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:19:45.750575+0000","last_peered":"2024-01-15T18:19:45.750575+0000","last_clean":"2024-01-15T18:19:45.750575+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:19:45.750575+0000","last_undegraded":"2024-01-15T18:19:45.750575+0000","last_fullsized":"2024-01-15T18:19:45.750575+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'48","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.699878+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:19:45.699878+0000","last_peered":"2024-01-15T18:19:45.699878+0000","last_clean":"2024-01-15T18:19:45.699878+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:19:45.699878+0000","last_undegraded":"2024-01-15T18:19:45.699878+0000","last_fullsized":"2024-01-15T18:19:45.699878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.465695+0000","last_change":"2024-01-15T18:14:16.889765+0000","last_active":"2024-01-15T18:19:45.465695+0000","last_peered":"2024-01-15T18:19:45.465695+0000","last_clean":"2024-01-15T18:19:45.465695+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:19:45.465695+0000","last_undegraded":"2024-01-15T18:19:45.465695+0000","last_fullsized":"2024-01-15T18:19:45.465695+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:52:42.255935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'46","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.800014+0000","last_change":"2024-01-15T18:14:16.883837+0000","last_active":"2024-01-15T18:19:45.800014+0000","last_peered":"2024-01-15T18:19:45.800014+0000","last_clean":"2024-01-15T18:19:45.800014+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:19:45.800014+0000","last_undegraded":"2024-01-15T18:19:45.800014+0000","last_fullsized":"2024-01-15T18:19:45.800014+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'56","reported_seq":85,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.856705+0000","last_change":"2024-01-15T18:14:16.878893+0000","last_active":"2024-01-15T18:19:45.856705+0000","last_peered":"2024-01-15T18:19:45.856705+0000","last_clean":"2024-01-15T18:19:45.856705+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:19:45.856705+0000","last_undegraded":"2024-01-15T18:19:45.856705+0000","last_fullsized":"2024-01-15T18:19:45.856705+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'47","reported_seq":71,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.916236+0000","last_change":"2024-01-15T18:14:16.878483+0000","last_active":"2024-01-15T18:19:45.916236+0000","last_peered":"2024-01-15T18:19:45.916236+0000","last_clean":"2024-01-15T18:19:45.916236+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:19:45.916236+0000","last_undegraded":"2024-01-15T18:19:45.916236+0000","last_fullsized":"2024-01-15T18:19:45.916236+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'39","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.671802+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:19:45.671802+0000","last_peered":"2024-01-15T18:19:45.671802+0000","last_clean":"2024-01-15T18:19:45.671802+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:19:45.671802+0000","last_undegraded":"2024-01-15T18:19:45.671802+0000","last_fullsized":"2024-01-15T18:19:45.671802+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'43","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.858236+0000","last_change":"2024-01-15T18:14:16.884123+0000","last_active":"2024-01-15T18:19:45.858236+0000","last_peered":"2024-01-15T18:19:45.858236+0000","last_clean":"2024-01-15T18:19:45.858236+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:19:45.858236+0000","last_undegraded":"2024-01-15T18:19:45.858236+0000","last_fullsized":"2024-01-15T18:19:45.858236+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'38","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.820578+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:19:45.820578+0000","last_peered":"2024-01-15T18:19:45.820578+0000","last_clean":"2024-01-15T18:19:45.820578+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:19:45.820578+0000","last_undegraded":"2024-01-15T18:19:45.820578+0000","last_fullsized":"2024-01-15T18:19:45.820578+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'73","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.039273+0000","last_change":"2024-01-15T18:19:57.039365+0000","last_active":"2024-01-15T18:19:57.039273+0000","last_peered":"2024-01-15T18:19:57.039273+0000","last_clean":"2024-01-15T18:19:57.039273+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:57.039273+0000","last_undegraded":"2024-01-15T18:19:57.039273+0000","last_fullsized":"2024-01-15T18:19:57.039273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.039107+0000","objects_scrubbed":23,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.107907+0000","scrub_duration":0.24278212099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'37","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407476+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:19:45.407476+0000","last_peered":"2024-01-15T18:19:45.407476+0000","last_clean":"2024-01-15T18:19:45.407476+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:19:45.407476+0000","last_undegraded":"2024-01-15T18:19:45.407476+0000","last_fullsized":"2024-01-15T18:19:45.407476+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'38","reported_seq":57,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.858798+0000","last_change":"2024-01-15T18:14:16.879927+0000","last_active":"2024-01-15T18:19:45.858798+0000","last_peered":"2024-01-15T18:19:45.858798+0000","last_clean":"2024-01-15T18:19:45.858798+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:19:45.858798+0000","last_undegraded":"2024-01-15T18:19:45.858798+0000","last_fullsized":"2024-01-15T18:19:45.858798+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'46","reported_seq":71,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.786616+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:19:45.786616+0000","last_peered":"2024-01-15T18:19:45.786616+0000","last_clean":"2024-01-15T18:19:45.786616+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:19:45.786616+0000","last_undegraded":"2024-01-15T18:19:45.786616+0000","last_fullsized":"2024-01-15T18:19:45.786616+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'41","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.720474+0000","last_change":"2024-01-15T18:14:16.873552+0000","last_active":"2024-01-15T18:19:45.720474+0000","last_peered":"2024-01-15T18:19:45.720474+0000","last_clean":"2024-01-15T18:19:45.720474+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:19:45.720474+0000","last_undegraded":"2024-01-15T18:19:45.720474+0000","last_fullsized":"2024-01-15T18:19:45.720474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'62","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.751935+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:19:45.751935+0000","last_peered":"2024-01-15T18:19:45.751935+0000","last_clean":"2024-01-15T18:19:45.751935+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:19:45.751935+0000","last_undegraded":"2024-01-15T18:19:45.751935+0000","last_fullsized":"2024-01-15T18:19:45.751935+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'54","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.911588+0000","last_change":"2024-01-15T18:14:16.878159+0000","last_active":"2024-01-15T18:19:45.911588+0000","last_peered":"2024-01-15T18:19:45.911588+0000","last_clean":"2024-01-15T18:19:45.911588+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:19:45.911588+0000","last_undegraded":"2024-01-15T18:19:45.911588+0000","last_fullsized":"2024-01-15T18:19:45.911588+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'34","reported_seq":56,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.543728+0000","last_change":"2024-01-15T18:14:16.877134+0000","last_active":"2024-01-15T18:19:45.543728+0000","last_peered":"2024-01-15T18:19:45.543728+0000","last_clean":"2024-01-15T18:19:45.543728+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:19:45.543728+0000","last_undegraded":"2024-01-15T18:19:45.543728+0000","last_fullsized":"2024-01-15T18:19:45.543728+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"113'4","reported_seq":21,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:29.749642+0000","last_change":"2024-01-15T18:14:15.869271+0000","last_active":"2024-01-15T18:18:29.749642+0000","last_peered":"2024-01-15T18:18:29.749642+0000","last_clean":"2024-01-15T18:18:29.749642+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:18:29.749642+0000","last_undegraded":"2024-01-15T18:18:29.749642+0000","last_fullsized":"2024-01-15T18:18:29.749642+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'47","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.412953+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:19:45.412953+0000","last_peered":"2024-01-15T18:19:45.412953+0000","last_clean":"2024-01-15T18:19:45.412953+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:19:45.412953+0000","last_undegraded":"2024-01-15T18:19:45.412953+0000","last_fullsized":"2024-01-15T18:19:45.412953+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:14.988989+0000","last_change":"2024-01-15T18:14:15.875644+0000","last_active":"2024-01-15T18:16:14.988989+0000","last_peered":"2024-01-15T18:16:14.988989+0000","last_clean":"2024-01-15T18:16:14.988989+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:16:14.988989+0000","last_undegraded":"2024-01-15T18:16:14.988989+0000","last_fullsized":"2024-01-15T18:16:14.988989+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:38:38.894890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'43","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.866056+0000","last_change":"2024-01-15T18:14:16.879961+0000","last_active":"2024-01-15T18:19:45.866056+0000","last_peered":"2024-01-15T18:19:45.866056+0000","last_clean":"2024-01-15T18:19:45.866056+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:19:45.866056+0000","last_undegraded":"2024-01-15T18:19:45.866056+0000","last_fullsized":"2024-01-15T18:19:45.866056+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'3","reported_seq":20,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:19.713455+0000","last_change":"2024-01-15T18:14:15.880501+0000","last_active":"2024-01-15T18:19:19.713455+0000","last_peered":"2024-01-15T18:19:19.713455+0000","last_clean":"2024-01-15T18:19:19.713455+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:19:19.713455+0000","last_undegraded":"2024-01-15T18:19:19.713455+0000","last_fullsized":"2024-01-15T18:19:19.713455+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'56","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.396172+0000","last_change":"2024-01-15T18:14:16.889640+0000","last_active":"2024-01-15T18:19:45.396172+0000","last_peered":"2024-01-15T18:19:45.396172+0000","last_clean":"2024-01-15T18:19:45.396172+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:19:45.396172+0000","last_undegraded":"2024-01-15T18:19:45.396172+0000","last_fullsized":"2024-01-15T18:19:45.396172+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:11:37.451848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"113'2","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:19.713426+0000","last_change":"2024-01-15T18:14:15.880225+0000","last_active":"2024-01-15T18:19:19.713426+0000","last_peered":"2024-01-15T18:19:19.713426+0000","last_clean":"2024-01-15T18:19:19.713426+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:19:19.713426+0000","last_undegraded":"2024-01-15T18:19:19.713426+0000","last_fullsized":"2024-01-15T18:19:19.713426+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'39","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.825621+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:19:45.825621+0000","last_peered":"2024-01-15T18:19:45.825621+0000","last_clean":"2024-01-15T18:19:45.825621+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:19:45.825621+0000","last_undegraded":"2024-01-15T18:19:45.825621+0000","last_fullsized":"2024-01-15T18:19:45.825621+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:19.758774+0000","last_change":"2024-01-15T18:14:15.872408+0000","last_active":"2024-01-15T18:19:19.758774+0000","last_peered":"2024-01-15T18:19:19.758774+0000","last_clean":"2024-01-15T18:19:19.758774+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:19:19.758774+0000","last_undegraded":"2024-01-15T18:19:19.758774+0000","last_fullsized":"2024-01-15T18:19:19.758774+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'49","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.773242+0000","last_change":"2024-01-15T18:14:16.871608+0000","last_active":"2024-01-15T18:19:45.773242+0000","last_peered":"2024-01-15T18:19:45.773242+0000","last_clean":"2024-01-15T18:19:45.773242+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:19:45.773242+0000","last_undegraded":"2024-01-15T18:19:45.773242+0000","last_fullsized":"2024-01-15T18:19:45.773242+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:19.733663+0000","last_change":"2024-01-15T18:14:15.870870+0000","last_active":"2024-01-15T18:19:19.733663+0000","last_peered":"2024-01-15T18:19:19.733663+0000","last_clean":"2024-01-15T18:19:19.733663+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:19:19.733663+0000","last_undegraded":"2024-01-15T18:19:19.733663+0000","last_fullsized":"2024-01-15T18:19:19.733663+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'53","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.775311+0000","last_change":"2024-01-15T18:14:16.882969+0000","last_active":"2024-01-15T18:19:45.775311+0000","last_peered":"2024-01-15T18:19:45.775311+0000","last_clean":"2024-01-15T18:19:45.775311+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:19:45.775311+0000","last_undegraded":"2024-01-15T18:19:45.775311+0000","last_fullsized":"2024-01-15T18:19:45.775311+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:09.846612+0000","last_change":"2024-01-15T18:14:15.868283+0000","last_active":"2024-01-15T18:19:09.846612+0000","last_peered":"2024-01-15T18:19:09.846612+0000","last_clean":"2024-01-15T18:19:09.846612+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:19:09.846612+0000","last_undegraded":"2024-01-15T18:19:09.846612+0000","last_fullsized":"2024-01-15T18:19:09.846612+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'43","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.762269+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:19:45.762269+0000","last_peered":"2024-01-15T18:19:45.762269+0000","last_clean":"2024-01-15T18:19:45.762269+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:19:45.762269+0000","last_undegraded":"2024-01-15T18:19:45.762269+0000","last_fullsized":"2024-01-15T18:19:45.762269+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:09.762091+0000","last_change":"2024-01-15T18:14:15.863721+0000","last_active":"2024-01-15T18:19:09.762091+0000","last_peered":"2024-01-15T18:19:09.762091+0000","last_clean":"2024-01-15T18:19:09.762091+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:19:09.762091+0000","last_undegraded":"2024-01-15T18:19:09.762091+0000","last_fullsized":"2024-01-15T18:19:09.762091+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'42","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.391717+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:19:45.391717+0000","last_peered":"2024-01-15T18:19:45.391717+0000","last_clean":"2024-01-15T18:19:45.391717+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:19:45.391717+0000","last_undegraded":"2024-01-15T18:19:45.391717+0000","last_fullsized":"2024-01-15T18:19:45.391717+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:09.711829+0000","last_change":"2024-01-15T18:14:15.876246+0000","last_active":"2024-01-15T18:19:09.711829+0000","last_peered":"2024-01-15T18:19:09.711829+0000","last_clean":"2024-01-15T18:19:09.711829+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:19:09.711829+0000","last_undegraded":"2024-01-15T18:19:09.711829+0000","last_fullsized":"2024-01-15T18:19:09.711829+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'40","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.640005+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:19:45.640005+0000","last_peered":"2024-01-15T18:19:45.640005+0000","last_clean":"2024-01-15T18:19:45.640005+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:19:45.640005+0000","last_undegraded":"2024-01-15T18:19:45.640005+0000","last_fullsized":"2024-01-15T18:19:45.640005+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:15:27.595871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:09.731626+0000","last_change":"2024-01-15T18:14:15.877698+0000","last_active":"2024-01-15T18:19:09.731626+0000","last_peered":"2024-01-15T18:19:09.731626+0000","last_clean":"2024-01-15T18:19:09.731626+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:19:09.731626+0000","last_undegraded":"2024-01-15T18:19:09.731626+0000","last_fullsized":"2024-01-15T18:19:09.731626+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'43","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.887723+0000","last_change":"2024-01-15T18:14:16.872503+0000","last_active":"2024-01-15T18:19:45.887723+0000","last_peered":"2024-01-15T18:19:45.887723+0000","last_clean":"2024-01-15T18:19:45.887723+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:19:45.887723+0000","last_undegraded":"2024-01-15T18:19:45.887723+0000","last_fullsized":"2024-01-15T18:19:45.887723+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'13","reported_seq":30,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:09.762255+0000","last_change":"2024-01-15T18:14:15.871448+0000","last_active":"2024-01-15T18:19:09.762255+0000","last_peered":"2024-01-15T18:19:09.762255+0000","last_clean":"2024-01-15T18:19:09.762255+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:19:09.762255+0000","last_undegraded":"2024-01-15T18:19:09.762255+0000","last_fullsized":"2024-01-15T18:19:09.762255+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'43","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407675+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:19:45.407675+0000","last_peered":"2024-01-15T18:19:45.407675+0000","last_clean":"2024-01-15T18:19:45.407675+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:19:45.407675+0000","last_undegraded":"2024-01-15T18:19:45.407675+0000","last_fullsized":"2024-01-15T18:19:45.407675+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'189","reported_seq":206,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:55.852780+0000","last_change":"2024-01-15T18:14:15.870117+0000","last_active":"2024-01-15T18:19:55.852780+0000","last_peered":"2024-01-15T18:19:55.852780+0000","last_clean":"2024-01-15T18:19:55.852780+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:19:55.852780+0000","last_undegraded":"2024-01-15T18:19:55.852780+0000","last_fullsized":"2024-01-15T18:19:55.852780+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":639993,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":188,"num_write_kb":715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'44","reported_seq":66,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.909826+0000","last_change":"2024-01-15T18:14:16.868931+0000","last_active":"2024-01-15T18:19:45.909826+0000","last_peered":"2024-01-15T18:19:45.909826+0000","last_clean":"2024-01-15T18:19:45.909826+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:19:45.909826+0000","last_undegraded":"2024-01-15T18:19:45.909826+0000","last_fullsized":"2024-01-15T18:19:45.909826+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"113'2","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:59.698699+0000","last_change":"2024-01-15T18:14:15.873486+0000","last_active":"2024-01-15T18:18:59.698699+0000","last_peered":"2024-01-15T18:18:59.698699+0000","last_clean":"2024-01-15T18:18:59.698699+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:18:59.698699+0000","last_undegraded":"2024-01-15T18:18:59.698699+0000","last_fullsized":"2024-01-15T18:18:59.698699+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'52","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.719381+0000","last_change":"2024-01-15T18:14:16.881824+0000","last_active":"2024-01-15T18:19:45.719381+0000","last_peered":"2024-01-15T18:19:45.719381+0000","last_clean":"2024-01-15T18:19:45.719381+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:19:45.719381+0000","last_undegraded":"2024-01-15T18:19:45.719381+0000","last_fullsized":"2024-01-15T18:19:45.719381+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":16,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:39.792106+0000","last_change":"2024-01-15T18:14:15.878451+0000","last_active":"2024-01-15T18:16:39.792106+0000","last_peered":"2024-01-15T18:16:39.792106+0000","last_clean":"2024-01-15T18:16:39.792106+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:16:39.792106+0000","last_undegraded":"2024-01-15T18:16:39.792106+0000","last_fullsized":"2024-01-15T18:16:39.792106+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:43:17.518655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.658609+0000","last_change":"2024-01-15T18:14:16.870084+0000","last_active":"2024-01-15T18:19:45.658609+0000","last_peered":"2024-01-15T18:19:45.658609+0000","last_clean":"2024-01-15T18:19:45.658609+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:19:45.658609+0000","last_undegraded":"2024-01-15T18:19:45.658609+0000","last_fullsized":"2024-01-15T18:19:45.658609+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"113'8","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:59.709767+0000","last_change":"2024-01-15T18:14:15.875102+0000","last_active":"2024-01-15T18:18:59.709767+0000","last_peered":"2024-01-15T18:18:59.709767+0000","last_clean":"2024-01-15T18:18:59.709767+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:18:59.709767+0000","last_undegraded":"2024-01-15T18:18:59.709767+0000","last_fullsized":"2024-01-15T18:18:59.709767+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'24","reported_seq":44,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.819282+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:19:45.819282+0000","last_peered":"2024-01-15T18:19:45.819282+0000","last_clean":"2024-01-15T18:19:45.819282+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:19:45.819282+0000","last_undegraded":"2024-01-15T18:19:45.819282+0000","last_fullsized":"2024-01-15T18:19:45.819282+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"113'1","reported_seq":18,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:49.757642+0000","last_change":"2024-01-15T18:14:15.871711+0000","last_active":"2024-01-15T18:18:49.757642+0000","last_peered":"2024-01-15T18:18:49.757642+0000","last_clean":"2024-01-15T18:18:49.757642+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:18:49.757642+0000","last_undegraded":"2024-01-15T18:18:49.757642+0000","last_fullsized":"2024-01-15T18:18:49.757642+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'36","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.769576+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:19:45.769576+0000","last_peered":"2024-01-15T18:19:45.769576+0000","last_clean":"2024-01-15T18:19:45.769576+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:19:45.769576+0000","last_undegraded":"2024-01-15T18:19:45.769576+0000","last_fullsized":"2024-01-15T18:19:45.769576+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"113'2","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:49.708191+0000","last_change":"2024-01-15T18:14:15.872504+0000","last_active":"2024-01-15T18:18:49.708191+0000","last_peered":"2024-01-15T18:18:49.708191+0000","last_clean":"2024-01-15T18:18:49.708191+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:18:49.708191+0000","last_undegraded":"2024-01-15T18:18:49.708191+0000","last_fullsized":"2024-01-15T18:18:49.708191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'44","reported_seq":70,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559602+0000","last_change":"2024-01-15T18:14:16.895009+0000","last_active":"2024-01-15T18:19:45.559602+0000","last_peered":"2024-01-15T18:19:45.559602+0000","last_clean":"2024-01-15T18:19:45.559602+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:19:45.559602+0000","last_undegraded":"2024-01-15T18:19:45.559602+0000","last_fullsized":"2024-01-15T18:19:45.559602+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:00:54.572624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"113'1","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:16:34.791118+0000","last_change":"2024-01-15T18:14:15.874531+0000","last_active":"2024-01-15T18:16:34.791118+0000","last_peered":"2024-01-15T18:16:34.791118+0000","last_clean":"2024-01-15T18:16:34.791118+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:16:34.791118+0000","last_undegraded":"2024-01-15T18:16:34.791118+0000","last_fullsized":"2024-01-15T18:16:34.791118+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:04:37.182742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'41","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.673339+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:19:45.673339+0000","last_peered":"2024-01-15T18:19:45.673339+0000","last_clean":"2024-01-15T18:19:45.673339+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:19:45.673339+0000","last_undegraded":"2024-01-15T18:19:45.673339+0000","last_fullsized":"2024-01-15T18:19:45.673339+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:21.267130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:49.753253+0000","last_change":"2024-01-15T18:14:15.876158+0000","last_active":"2024-01-15T18:18:49.753253+0000","last_peered":"2024-01-15T18:18:49.753253+0000","last_clean":"2024-01-15T18:18:49.753253+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:18:49.753253+0000","last_undegraded":"2024-01-15T18:18:49.753253+0000","last_fullsized":"2024-01-15T18:18:49.753253+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.866450+0000","last_change":"2024-01-15T18:14:16.886434+0000","last_active":"2024-01-15T18:19:45.866450+0000","last_peered":"2024-01-15T18:19:45.866450+0000","last_clean":"2024-01-15T18:19:45.866450+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:19:45.866450+0000","last_undegraded":"2024-01-15T18:19:45.866450+0000","last_fullsized":"2024-01-15T18:19:45.866450+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'3","reported_seq":20,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:49.696756+0000","last_change":"2024-01-15T18:14:15.871712+0000","last_active":"2024-01-15T18:18:49.696756+0000","last_peered":"2024-01-15T18:18:49.696756+0000","last_clean":"2024-01-15T18:18:49.696756+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:18:49.696756+0000","last_undegraded":"2024-01-15T18:18:49.696756+0000","last_fullsized":"2024-01-15T18:18:49.696756+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'43","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:49.717729+0000","last_change":"2024-01-15T18:15:20.196078+0000","last_active":"2024-01-15T18:19:49.717729+0000","last_peered":"2024-01-15T18:19:49.717729+0000","last_clean":"2024-01-15T18:19:49.717729+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:19:49.717729+0000","last_undegraded":"2024-01-15T18:19:49.717729+0000","last_fullsized":"2024-01-15T18:19:49.717729+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:39.706294+0000","last_change":"2024-01-15T18:14:15.875131+0000","last_active":"2024-01-15T18:18:39.706294+0000","last_peered":"2024-01-15T18:18:39.706294+0000","last_clean":"2024-01-15T18:18:39.706294+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:18:39.706294+0000","last_undegraded":"2024-01-15T18:18:39.706294+0000","last_fullsized":"2024-01-15T18:18:39.706294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'44","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.587657+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:19:45.587657+0000","last_peered":"2024-01-15T18:19:45.587657+0000","last_clean":"2024-01-15T18:19:45.587657+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:19:45.587657+0000","last_undegraded":"2024-01-15T18:19:45.587657+0000","last_fullsized":"2024-01-15T18:19:45.587657+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:08:29.265440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.795122+0000","last_change":"2024-01-15T18:19:58.795153+0000","last_active":"2024-01-15T18:19:58.795122+0000","last_peered":"2024-01-15T18:19:58.795122+0000","last_clean":"2024-01-15T18:19:58.795122+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:19:58.795122+0000","last_undegraded":"2024-01-15T18:19:58.795122+0000","last_fullsized":"2024-01-15T18:19:58.795122+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.795017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.531385+0000","scrub_duration":0.0028610760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'42","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.813847+0000","last_change":"2024-01-15T18:14:16.888610+0000","last_active":"2024-01-15T18:19:45.813847+0000","last_peered":"2024-01-15T18:19:45.813847+0000","last_clean":"2024-01-15T18:19:45.813847+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:19:45.813847+0000","last_undegraded":"2024-01-15T18:19:45.813847+0000","last_fullsized":"2024-01-15T18:19:45.813847+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:39.754904+0000","last_change":"2024-01-15T18:14:15.876450+0000","last_active":"2024-01-15T18:18:39.754904+0000","last_peered":"2024-01-15T18:18:39.754904+0000","last_clean":"2024-01-15T18:18:39.754904+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:18:39.754904+0000","last_undegraded":"2024-01-15T18:18:39.754904+0000","last_fullsized":"2024-01-15T18:18:39.754904+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'36","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.559183+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:19:45.559183+0000","last_peered":"2024-01-15T18:19:45.559183+0000","last_clean":"2024-01-15T18:19:45.559183+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:19:45.559183+0000","last_undegraded":"2024-01-15T18:19:45.559183+0000","last_fullsized":"2024-01-15T18:19:45.559183+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:43:23.578855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"113'2","reported_seq":19,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:39.725756+0000","last_change":"2024-01-15T18:14:15.860126+0000","last_active":"2024-01-15T18:18:39.725756+0000","last_peered":"2024-01-15T18:18:39.725756+0000","last_clean":"2024-01-15T18:18:39.725756+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:18:39.725756+0000","last_undegraded":"2024-01-15T18:18:39.725756+0000","last_fullsized":"2024-01-15T18:18:39.725756+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'41","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410870+0000","last_change":"2024-01-15T18:15:20.356060+0000","last_active":"2024-01-15T18:19:45.410870+0000","last_peered":"2024-01-15T18:19:45.410870+0000","last_clean":"2024-01-15T18:19:45.410870+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:19:45.410870+0000","last_undegraded":"2024-01-15T18:19:45.410870+0000","last_fullsized":"2024-01-15T18:19:45.410870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'19","reported_seq":35,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:46.468834+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:19:46.468834+0000","last_peered":"2024-01-15T18:19:46.468834+0000","last_clean":"2024-01-15T18:19:46.468834+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:19:46.468834+0000","last_undegraded":"2024-01-15T18:19:46.468834+0000","last_fullsized":"2024-01-15T18:19:46.468834+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:50:12.999752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'66","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.918730+0000","last_change":"2024-01-15T18:19:57.918788+0000","last_active":"2024-01-15T18:19:57.918730+0000","last_peered":"2024-01-15T18:19:57.918730+0000","last_clean":"2024-01-15T18:19:57.918730+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:57.918730+0000","last_undegraded":"2024-01-15T18:19:57.918730+0000","last_fullsized":"2024-01-15T18:19:57.918730+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'66","last_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_deep_scrub":"116'66","last_deep_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.918639+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:38.022160+0000","scrub_duration":0.15638939399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:29.752926+0000","last_change":"2024-01-15T18:14:15.876417+0000","last_active":"2024-01-15T18:18:29.752926+0000","last_peered":"2024-01-15T18:18:29.752926+0000","last_clean":"2024-01-15T18:18:29.752926+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:18:29.752926+0000","last_undegraded":"2024-01-15T18:18:29.752926+0000","last_fullsized":"2024-01-15T18:18:29.752926+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'39","reported_seq":60,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404370+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:19:45.404370+0000","last_peered":"2024-01-15T18:19:45.404370+0000","last_clean":"2024-01-15T18:19:45.404370+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:19:45.404370+0000","last_undegraded":"2024-01-15T18:19:45.404370+0000","last_fullsized":"2024-01-15T18:19:45.404370+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:29.724024+0000","last_change":"2024-01-15T18:14:15.868558+0000","last_active":"2024-01-15T18:18:29.724024+0000","last_peered":"2024-01-15T18:18:29.724024+0000","last_clean":"2024-01-15T18:18:29.724024+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:18:29.724024+0000","last_undegraded":"2024-01-15T18:18:29.724024+0000","last_fullsized":"2024-01-15T18:18:29.724024+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'44","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.851713+0000","last_change":"2024-01-15T18:14:16.868082+0000","last_active":"2024-01-15T18:19:45.851713+0000","last_peered":"2024-01-15T18:19:45.851713+0000","last_clean":"2024-01-15T18:19:45.851713+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:19:45.851713+0000","last_undegraded":"2024-01-15T18:19:45.851713+0000","last_fullsized":"2024-01-15T18:19:45.851713+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:29.749471+0000","last_change":"2024-01-15T18:14:15.867924+0000","last_active":"2024-01-15T18:18:29.749471+0000","last_peered":"2024-01-15T18:18:29.749471+0000","last_clean":"2024-01-15T18:18:29.749471+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:18:29.749471+0000","last_undegraded":"2024-01-15T18:18:29.749471+0000","last_fullsized":"2024-01-15T18:18:29.749471+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'56","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.828635+0000","last_change":"2024-01-15T18:14:16.879855+0000","last_active":"2024-01-15T18:19:45.828635+0000","last_peered":"2024-01-15T18:19:45.828635+0000","last_clean":"2024-01-15T18:19:45.828635+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:19:45.828635+0000","last_undegraded":"2024-01-15T18:19:45.828635+0000","last_fullsized":"2024-01-15T18:19:45.828635+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:29.839333+0000","last_change":"2024-01-15T18:14:15.873250+0000","last_active":"2024-01-15T18:18:29.839333+0000","last_peered":"2024-01-15T18:18:29.839333+0000","last_clean":"2024-01-15T18:18:29.839333+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:18:29.839333+0000","last_undegraded":"2024-01-15T18:18:29.839333+0000","last_fullsized":"2024-01-15T18:18:29.839333+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'46","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.741049+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:19:45.741049+0000","last_peered":"2024-01-15T18:19:45.741049+0000","last_clean":"2024-01-15T18:19:45.741049+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:19:45.741049+0000","last_undegraded":"2024-01-15T18:19:45.741049+0000","last_fullsized":"2024-01-15T18:19:45.741049+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:05:32.449523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'12","reported_seq":47,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.182834+0000","last_change":"2024-01-15T18:19:58.182868+0000","last_active":"2024-01-15T18:19:58.182834+0000","last_peered":"2024-01-15T18:19:58.182834+0000","last_clean":"2024-01-15T18:19:58.182834+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:19:58.182834+0000","last_undegraded":"2024-01-15T18:19:58.182834+0000","last_fullsized":"2024-01-15T18:19:58.182834+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.182737+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:33:12.765302+0000","scrub_duration":0.0054820609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'39","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.849636+0000","last_change":"2024-01-15T18:14:16.883289+0000","last_active":"2024-01-15T18:19:45.849636+0000","last_peered":"2024-01-15T18:19:45.849636+0000","last_clean":"2024-01-15T18:19:45.849636+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:19:45.849636+0000","last_undegraded":"2024-01-15T18:19:45.849636+0000","last_fullsized":"2024-01-15T18:19:45.849636+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:19.750721+0000","last_change":"2024-01-15T18:14:15.876619+0000","last_active":"2024-01-15T18:18:19.750721+0000","last_peered":"2024-01-15T18:18:19.750721+0000","last_clean":"2024-01-15T18:18:19.750721+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:18:19.750721+0000","last_undegraded":"2024-01-15T18:18:19.750721+0000","last_fullsized":"2024-01-15T18:18:19.750721+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'31","reported_seq":55,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.872625+0000","last_change":"2024-01-15T18:14:16.885489+0000","last_active":"2024-01-15T18:19:45.872625+0000","last_peered":"2024-01-15T18:19:45.872625+0000","last_clean":"2024-01-15T18:19:45.872625+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:19:45.872625+0000","last_undegraded":"2024-01-15T18:19:45.872625+0000","last_fullsized":"2024-01-15T18:19:45.872625+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:18:59.754893+0000","last_change":"2024-01-15T18:14:15.873335+0000","last_active":"2024-01-15T18:18:59.754893+0000","last_peered":"2024-01-15T18:18:59.754893+0000","last_clean":"2024-01-15T18:18:59.754893+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:18:59.754893+0000","last_undegraded":"2024-01-15T18:18:59.754893+0000","last_fullsized":"2024-01-15T18:18:59.754893+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":254,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:17:30.003748+0000","last_change":"2024-01-15T17:57:27.050660+0000","last_active":"2024-01-15T18:17:30.003748+0000","last_peered":"2024-01-15T18:17:30.003748+0000","last_clean":"2024-01-15T18:17:30.003748+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:17:30.003748+0000","last_undegraded":"2024-01-15T18:17:30.003748+0000","last_fullsized":"2024-01-15T18:17:30.003748+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T17:57:25.939937+0000","last_clean_scrub_stamp":"2024-01-15T17:57:25.939937+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:04:03.805821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2850,"ondisk_log_size":2850,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":642112,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":286,"num_write_kb":756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1351680,"data_stored":1284224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542416,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.296,"15min":1.325},"min":{"1min":0.61599999999999999,"5min":0.61299999999999999,"15min":0.54000000000000004},"max":{"1min":1.782,"5min":2.1200000000000001,"15min":9.1379999999999999},"last":1.407},{"interface":"front","average":{"1min":1.268,"5min":1.385,"15min":1.528},"min":{"1min":0.88400000000000001,"5min":0.81999999999999995,"15min":0.69999999999999996},"max":{"1min":1.673,"5min":1.905,"15min":27.32},"last":1.3799999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2210000000000001,"15min":1.4530000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.728,"5min":2.1000000000000001,"15min":29.167999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.21,"5min":1.24,"15min":1.3260000000000001},"min":{"1min":0.77700000000000002,"5min":0.69299999999999995,"15min":0.52800000000000002},"max":{"1min":1.597,"5min":2.6179999999999999,"15min":8.5980000000000008},"last":1.889}]},{"osd":2,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.3740000000000001,"15min":1.4179999999999999},"min":{"1min":0.94699999999999995,"5min":0.874,"15min":0.56000000000000005},"max":{"1min":1.7709999999999999,"5min":2.3900000000000001,"15min":12.709},"last":1.655},{"interface":"front","average":{"1min":1.141,"5min":1.256,"15min":1.488},"min":{"1min":0.79400000000000004,"5min":0.72499999999999998,"15min":0.56599999999999995},"max":{"1min":1.6539999999999999,"5min":1.8200000000000001,"15min":27.358000000000001},"last":1.274}]},{"osd":3,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.3100000000000001,"15min":1.556},"min":{"1min":0.98599999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.643,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.244},{"interface":"front","average":{"1min":1.379,"5min":1.4530000000000001,"15min":1.5189999999999999},"min":{"1min":1.01,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.7769999999999999,"5min":2.4940000000000002,"15min":16.088999999999999},"last":1.734}]},{"osd":4,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.381,"15min":1.5269999999999999},"min":{"1min":0.94899999999999995,"5min":0.64700000000000002,"15min":0.50700000000000001},"max":{"1min":1.681,"5min":6.0350000000000001,"15min":27.219000000000001},"last":1.169},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.4339999999999999,"15min":1.641},"min":{"1min":0.90400000000000003,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.7629999999999999,"5min":3.536,"15min":29.25},"last":1.3420000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.4730000000000001,"15min":1.573},"min":{"1min":1.0209999999999999,"5min":0.68799999999999994,"15min":0.67000000000000004},"max":{"1min":1.8660000000000001,"5min":2.7120000000000002,"15min":11.834},"last":1.53},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.5129999999999999,"15min":1.675},"min":{"1min":0.92900000000000005,"5min":0.71199999999999997,"15min":0.60499999999999998},"max":{"1min":1.6100000000000001,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.6020000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.635,"15min":1.8200000000000001},"min":{"1min":1.1850000000000001,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":1.929,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.5409999999999999,"15min":1.8049999999999999},"min":{"1min":1.0289999999999999,"5min":0.63100000000000001,"15min":0.57299999999999995},"max":{"1min":1.841,"5min":2.754,"15min":29.106999999999999},"last":1.5700000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542417,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.2190000000000001,"15min":1.236},"min":{"1min":0.72299999999999998,"5min":0.68200000000000005,"15min":0.438},"max":{"1min":1.5409999999999999,"5min":2.77,"15min":8.8350000000000009},"last":0.752},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.1970000000000001,"15min":1.27},"min":{"1min":0.85299999999999998,"5min":0.81200000000000006,"15min":0.59599999999999997},"max":{"1min":1.643,"5min":2.673,"15min":9.548},"last":1.05}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.2729999999999999,"15min":1.395},"min":{"1min":0.89300000000000002,"5min":0.70999999999999996,"15min":0.52100000000000002},"max":{"1min":1.5209999999999999,"5min":2.8700000000000001,"15min":13.269},"last":0.875},{"interface":"front","average":{"1min":1.262,"5min":1.3009999999999999,"15min":1.4119999999999999},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.57799999999999996},"max":{"1min":1.6990000000000001,"5min":3.1080000000000001,"15min":13.984999999999999},"last":1.266}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.232,"15min":1.284},"min":{"1min":0.85299999999999998,"5min":0.79000000000000004,"15min":0.69799999999999995},"max":{"1min":1.7,"5min":1.9379999999999999,"15min":7.0030000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.321,"5min":1.373,"15min":1.425},"min":{"1min":0.81100000000000005,"5min":0.73899999999999999,"15min":0.49399999999999999},"max":{"1min":1.623,"5min":2.6909999999999998,"15min":8.9359999999999999},"last":0.96699999999999997}]},{"osd":3,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.444,"15min":1.55},"min":{"1min":1.131,"5min":1.0089999999999999,"15min":0.68500000000000005},"max":{"1min":1.7629999999999999,"5min":3.339,"15min":16.855},"last":1.204},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.413,"15min":1.5209999999999999},"min":{"1min":1.111,"5min":1.046,"15min":0.628},"max":{"1min":1.657,"5min":2.0139999999999998,"15min":10.875},"last":0.91400000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.401,"15min":1.45},"min":{"1min":0.86699999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":1.645,"5min":3.0299999999999998,"15min":8.7539999999999996},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.427,"15min":1.4970000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.819,"5min":5.125,"15min":10.118},"last":1.3600000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3600000000000001,"15min":1.5209999999999999},"min":{"1min":0.92500000000000004,"5min":0.88,"15min":0.60299999999999998},"max":{"1min":1.8120000000000001,"5min":2.3210000000000002,"15min":16.309999999999999},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5429999999999999,"15min":1.6200000000000001},"min":{"1min":0.88,"5min":0.88,"15min":0.66000000000000003},"max":{"1min":1.8029999999999999,"5min":3.0209999999999999,"15min":10.077},"last":1.425}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":1.5289999999999999,"15min":1.6279999999999999},"min":{"1min":0.995,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":1.881,"5min":3.2450000000000001,"15min":10.302},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.504,"5min":1.663,"15min":1.6879999999999999},"min":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":1.9059999999999999,"5min":7.6799999999999997,"15min":9.4079999999999995},"last":1.375}]}]},{"osd":5,"up_from":14,"seq":60129542419,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928972,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93442868,"statfs":{"total":96636764160,"available":95685496832,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2609999999999999,"15min":1.399},"min":{"1min":0.79300000000000004,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.6180000000000001,"5min":4.1829999999999998,"15min":26.077999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3740000000000001,"15min":1.4299999999999999},"min":{"1min":1.099,"5min":0.77900000000000003,"15min":0.56399999999999995},"max":{"1min":1.6200000000000001,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.125}]},{"osd":1,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2549999999999999,"15min":1.4339999999999999},"min":{"1min":0.875,"5min":0.82499999999999996,"15min":0.53300000000000003},"max":{"1min":1.8,"5min":1.923,"15min":26},"last":1.381},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.367,"15min":1.401},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.657,"5min":4.0869999999999997,"15min":12.619},"last":0.84899999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":1.3460000000000001,"15min":1.5329999999999999},"min":{"1min":1.0700000000000001,"5min":0.69999999999999996,"15min":0.56699999999999995},"max":{"1min":1.712,"5min":2.6869999999999998,"15min":26.234999999999999},"last":1.1639999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.331,"15min":1.532},"min":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.629},"max":{"1min":1.766,"5min":1.8240000000000001,"15min":17.126999999999999},"last":0.78700000000000003}]},{"osd":3,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3120000000000001,"15min":1.397},"min":{"1min":1.0429999999999999,"5min":0.82399999999999995,"15min":0.67200000000000004},"max":{"1min":1.792,"5min":2.375,"15min":12.722},"last":1.04},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4830000000000001,"15min":1.6399999999999999},"min":{"1min":1.046,"5min":0.75900000000000001,"15min":0.67900000000000005},"max":{"1min":1.7989999999999999,"5min":3.238,"15min":26.134},"last":1.494}]},{"osd":4,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2769999999999999,"15min":1.4139999999999999},"min":{"1min":0.86299999999999999,"5min":0.82099999999999995,"15min":0.442},"max":{"1min":1.77,"5min":2.5350000000000001,"15min":9.9979999999999993},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.458,"15min":1.6930000000000001},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.498},"max":{"1min":1.875,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.224}]},{"osd":6,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3640000000000001,"15min":1.4339999999999999},"min":{"1min":1.0069999999999999,"5min":0.82299999999999995,"15min":0.45900000000000002},"max":{"1min":1.849,"5min":2.6640000000000001,"15min":12.769},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5860000000000001,"15min":1.647},"min":{"1min":0.94799999999999995,"5min":0.79800000000000004,"15min":0.49199999999999999},"max":{"1min":1.909,"5min":11.851000000000001,"15min":26.186},"last":1.6060000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:03 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.5960000000000001,"15min":1.798},"min":{"1min":1.1599999999999999,"5min":1.1359999999999999,"15min":0.65700000000000003},"max":{"1min":2.04,"5min":2.3700000000000001,"15min":13.706},"last":1.012},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.54,"15min":1.679},"min":{"1min":1.038,"5min":1.0189999999999999,"15min":0.60099999999999998},"max":{"1min":2.0289999999999999,"5min":4.3700000000000001,"15min":21.728000000000002},"last":1.4379999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542418,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.343,"15min":1.27},"min":{"1min":0.92800000000000005,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.7490000000000001,"5min":13.288,"15min":15.044},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.284,"15min":1.321},"min":{"1min":0.91600000000000004,"5min":0.71299999999999997,"15min":0.34699999999999998},"max":{"1min":1.5389999999999999,"5min":9.2409999999999997,"15min":11.196},"last":1.091}]},{"osd":1,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.2569999999999999,"15min":1.3779999999999999},"min":{"1min":0.90400000000000003,"5min":0.78400000000000003,"15min":0.41499999999999998},"max":{"1min":1.655,"5min":3.5910000000000002,"15min":28.663},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.302,"15min":1.4370000000000001},"min":{"1min":0.70799999999999996,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.6319999999999999,"5min":3.6739999999999999,"15min":15.096},"last":1.5920000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.333,"15min":1.4319999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.657,"5min":3.8170000000000002,"15min":16.023},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.381,"15min":1.5169999999999999},"min":{"1min":0.94899999999999995,"5min":0.71299999999999997,"15min":0.52500000000000002},"max":{"1min":1.7869999999999999,"5min":3.7930000000000001,"15min":18.858000000000001},"last":1.7}]},{"osd":3,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3029999999999999,"15min":1.393},"min":{"1min":0.80100000000000005,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.847,"5min":5.0599999999999996,"15min":11.449},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.357,"5min":1.4510000000000001,"15min":1.5720000000000001},"min":{"1min":0.995,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.7969999999999999,"5min":3.7080000000000002,"15min":18.027000000000001},"last":1.7769999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.4390000000000001,"15min":1.6220000000000001},"min":{"1min":0.88900000000000001,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.8140000000000001,"5min":7.7439999999999998,"15min":54.938000000000002},"last":1.903},{"interface":"front","average":{"1min":1.4139999999999999,"5min":1.339,"15min":1.482},"min":{"1min":0.96499999999999997,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":1.857,"5min":2.0960000000000001,"15min":13.031000000000001},"last":1.196}]},{"osd":6,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.4119999999999999,"15min":1.5509999999999999},"min":{"1min":1.1180000000000001,"5min":0.90300000000000002,"15min":0.495},"max":{"1min":1.8779999999999999,"5min":3.766,"15min":16.103999999999999},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5920000000000001,"15min":1.6799999999999999},"min":{"1min":1.1559999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.909,"5min":8.1609999999999996,"15min":15.118},"last":1.819}]},{"osd":7,"last update":"Mon Jan 15 18:19:08 2024","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.45,"15min":1.786},"min":{"1min":1.1919999999999999,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.0299999999999998,"5min":2.4689999999999999,"15min":23.390000000000001},"last":1.48},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5609999999999999,"15min":1.756},"min":{"1min":1.2270000000000001,"5min":1.0409999999999999,"15min":0.62},"max":{"1min":1.99,"5min":3.9020000000000001,"15min":16.763000000000002},"last":1.8759999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542419,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462028,"kb_used_data":1386500,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909812,"statfs":{"total":96636764160,"available":95139647488,"internally_reserved":0,"allocated":1419776000,"data_stored":1418947342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.95199999999999996,"15min":1.1020000000000001},"min":{"1min":0.51700000000000002,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":1.579,"5min":1.6599999999999999,"15min":36.408000000000001},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.101,"15min":1.093},"min":{"1min":0.69799999999999995,"5min":0.63900000000000001,"15min":0.377},"max":{"1min":1.6930000000000001,"5min":2.0230000000000001,"15min":5.0190000000000001},"last":0.52700000000000002}]},{"osd":1,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.99099999999999999,"15min":1.0640000000000001},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.34399999999999997},"max":{"1min":1.756,"5min":1.837,"15min":5.226},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.113,"5min":1.1899999999999999,"15min":1.22},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.669,"5min":2.0609999999999999,"15min":7.2169999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.2569999999999999,"15min":1.3500000000000001},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.637,"5min":2.0150000000000001,"15min":14.025},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.345,"15min":1.3400000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.841,"5min":2.403,"15min":3.6240000000000001},"last":0.66200000000000003}]},{"osd":4,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3320000000000001,"15min":1.4259999999999999},"min":{"1min":0.84799999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"max":{"1min":1.9350000000000001,"5min":3.1459999999999999,"15min":9.0069999999999997},"last":0.71599999999999997},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.448,"15min":1.6040000000000001},"min":{"1min":1.091,"5min":0.79100000000000004,"15min":0.69499999999999995},"max":{"1min":1.696,"5min":4.8570000000000002,"15min":15.769},"last":0.83799999999999997}]},{"osd":5,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.468,"15min":1.579},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.65000000000000002},"max":{"1min":1.7490000000000001,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.121},{"interface":"front","average":{"1min":1.468,"5min":1.5169999999999999,"15min":1.7809999999999999},"min":{"1min":0.877,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":2.1219999999999999,"5min":2.3959999999999999,"15min":38.832999999999998},"last":0.88700000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.5469999999999999,"15min":1.7529999999999999},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.51400000000000001},"max":{"1min":1.804,"5min":7.1020000000000003,"15min":32.259999999999998},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.5740000000000001,"15min":1.7470000000000001},"min":{"1min":1.0389999999999999,"5min":0.85999999999999999,"15min":0.747},"max":{"1min":2.169,"5min":2.419,"15min":16.195},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon Jan 15 18:19:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.6719999999999999,"15min":1.905},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.728,"15min":1.8009999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":1.976,"5min":16.734000000000002,"15min":17.622},"last":0.93500000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542418,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.012,"15min":1.0249999999999999},"min":{"1min":0.63500000000000001,"5min":0.44400000000000001,"15min":0.35199999999999998},"max":{"1min":1.5069999999999999,"5min":3.1589999999999998,"15min":10.256},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.2150000000000001,"15min":1.1819999999999999},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.33700000000000002},"max":{"1min":1.8180000000000001,"5min":8.9719999999999995,"15min":10.319000000000001},"last":0.64000000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1,"15min":1.026},"min":{"1min":0.67900000000000005,"5min":0.48899999999999999,"15min":0.38600000000000001},"max":{"1min":1.573,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.097,"5min":1.1890000000000001,"15min":1.2509999999999999},"min":{"1min":0.74099999999999999,"5min":0.59499999999999997,"15min":0.35299999999999998},"max":{"1min":1.5669999999999999,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.0229999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3240000000000001,"15min":1.4219999999999999},"min":{"1min":0.82899999999999996,"5min":0.54200000000000004,"15min":0.44400000000000001},"max":{"1min":1.5529999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.224,"15min":1.306},"min":{"1min":0.89000000000000001,"5min":0.56599999999999995,"15min":0.376},"max":{"1min":1.7689999999999999,"5min":2.7589999999999999,"15min":14.946},"last":1.127}]},{"osd":4,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.536,"15min":1.524},"min":{"1min":1.139,"5min":0.82099999999999995,"15min":0.66100000000000003},"max":{"1min":1.7769999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.4990000000000001,"15min":1.5960000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.65300000000000002},"max":{"1min":1.768,"5min":2.867,"15min":10.279999999999999},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4770000000000001,"15min":1.5700000000000001},"min":{"1min":1,"5min":1,"15min":0.63500000000000001},"max":{"1min":1.6830000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.448},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.613,"15min":1.6200000000000001},"min":{"1min":1.0209999999999999,"5min":0.88200000000000001,"15min":0.71699999999999997},"max":{"1min":1.895,"5min":10.833,"15min":10.833},"last":1.2549999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.514,"15min":1.6180000000000001},"min":{"1min":1.0109999999999999,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.827,"5min":2.984,"15min":10.185},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.452,"5min":1.657,"15min":1.7250000000000001},"min":{"1min":1.107,"5min":1.1060000000000001,"15min":0.79700000000000004},"max":{"1min":1.843,"5min":10.917,"15min":22.382000000000001},"last":1.5369999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:19:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.6180000000000001,"15min":1.843},"min":{"1min":1.129,"5min":1.006,"15min":0.77700000000000002},"max":{"1min":1.897,"5min":3.052,"15min":19.986000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.6259999999999999,"5min":1.6659999999999999,"15min":1.77},"min":{"1min":1.252,"5min":0.89300000000000002,"15min":0.877},"max":{"1min":1.931,"5min":3.069,"15min":11.893000000000001},"last":1.5960000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542418,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1137752,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93234088,"statfs":{"total":96636764160,"available":95471706112,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.96599999999999997,"15min":1.1020000000000001},"min":{"1min":0.63300000000000001,"5min":0.36799999999999999,"15min":0.36199999999999999},"max":{"1min":1.7130000000000001,"5min":3.4510000000000001,"15min":19.218},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.071,"5min":1.0960000000000001,"15min":1.232},"min":{"1min":0.65500000000000003,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.694,"5min":2.0939999999999999,"15min":19.823},"last":0.73199999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.99099999999999999,"15min":1.1100000000000001},"min":{"1min":0.80000000000000004,"5min":0.54400000000000004,"15min":0.31900000000000001},"max":{"1min":1.776,"5min":2.5110000000000001,"15min":19.02},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.258,"15min":1.401},"min":{"1min":0.85699999999999998,"5min":0.63300000000000001,"15min":0.41599999999999998},"max":{"1min":1.835,"5min":3.1400000000000001,"15min":29.251000000000001},"last":1.038}]},{"osd":3,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.3779999999999999,"15min":1.4990000000000001},"min":{"1min":0.84899999999999998,"5min":0.58499999999999996,"15min":0.40899999999999997},"max":{"1min":1.6779999999999999,"5min":5.1459999999999999,"15min":27.021000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.194,"15min":1.391},"min":{"1min":0.77600000000000002,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.7669999999999999,"5min":3.0150000000000001,"15min":31.218},"last":1.093}]},{"osd":4,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":1.492,"15min":1.6890000000000001},"min":{"1min":1.1379999999999999,"5min":0.755,"15min":0.70599999999999996},"max":{"1min":1.8049999999999999,"5min":5.2169999999999996,"15min":27.059999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.4450000000000001,"15min":1.6359999999999999},"min":{"1min":0.94899999999999995,"5min":0.78300000000000003,"15min":0.56499999999999995},"max":{"1min":1.9650000000000001,"5min":3.4159999999999999,"15min":31.433},"last":1.4890000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4730000000000001,"15min":1.645},"min":{"1min":1.149,"5min":0.82499999999999996,"15min":0.68500000000000005},"max":{"1min":1.8560000000000001,"5min":3.0950000000000002,"15min":20.012},"last":1.367},{"interface":"front","average":{"1min":1.339,"5min":1.431,"15min":1.52},"min":{"1min":0.97699999999999998,"5min":0.85799999999999998,"15min":0.69299999999999995},"max":{"1min":1.631,"5min":2.9409999999999998,"15min":6.758},"last":1.3460000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.639,"15min":1.794},"min":{"1min":1.28,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.9339999999999999,"5min":5.0629999999999997,"15min":29.530000000000001},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.575,"15min":1.802},"min":{"1min":1.2030000000000001,"5min":0.88400000000000001,"15min":0.61399999999999999},"max":{"1min":1.845,"5min":3.1960000000000002,"15min":31.186},"last":1.4470000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5169999999999999,"15min":1.8140000000000001},"min":{"1min":1.0860000000000001,"5min":0.95899999999999996,"15min":0.63},"max":{"1min":1.8460000000000001,"5min":3.3769999999999998,"15min":26.393999999999998},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.55,"5min":1.6240000000000001,"15min":1.8380000000000001},"min":{"1min":1.1659999999999999,"5min":0.89200000000000002,"15min":0.76100000000000001},"max":{"1min":1.913,"5min":3.5750000000000002,"15min":27.582000000000001},"last":1.389}]}]},{"osd":0,"up_from":14,"seq":60129542417,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1777948,"kb_used_data":1706580,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92593892,"statfs":{"total":96636764160,"available":94816145408,"internally_reserved":0,"allocated":1747537920,"data_stored":1746693697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.97699999999999998,"15min":1.141},"min":{"1min":0.57199999999999995,"5min":0.42299999999999999,"15min":0.30599999999999999},"max":{"1min":1.5860000000000001,"5min":5.7560000000000002,"15min":20.437999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.161,"5min":1.236,"15min":1.278},"min":{"1min":0.77000000000000002,"5min":0.65300000000000002,"15min":0.32400000000000001},"max":{"1min":1.8660000000000001,"5min":6.3600000000000003,"15min":35.137999999999998},"last":0.82699999999999996}]},{"osd":2,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.236,"15min":1.3879999999999999},"min":{"1min":0.66100000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.716,"5min":11.06,"15min":20.556000000000001},"last":1.206},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.2789999999999999,"15min":1.375},"min":{"1min":0.85099999999999998,"5min":0.60699999999999998,"15min":0.39000000000000001},"max":{"1min":1.8919999999999999,"5min":6.4939999999999998,"15min":35.183},"last":1.093}]},{"osd":3,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.117,"5min":1.238,"15min":1.1819999999999999},"min":{"1min":0.621,"5min":0.55000000000000004,"15min":0.42299999999999999},"max":{"1min":2.0230000000000001,"5min":10.885999999999999,"15min":10.885999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.266,"5min":1.3779999999999999,"15min":1.635},"min":{"1min":0.90000000000000002,"5min":0.73099999999999998,"15min":0.45100000000000001},"max":{"1min":1.621,"5min":9.3339999999999996,"15min":25.372},"last":1.3180000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.532,"15min":1.621},"min":{"1min":1.0509999999999999,"5min":1.014,"15min":0.67200000000000004},"max":{"1min":1.9710000000000001,"5min":9.9969999999999999,"15min":35.223999999999997},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.581,"15min":1.49},"min":{"1min":1.016,"5min":0.99399999999999999,"15min":0.58799999999999997},"max":{"1min":1.96,"5min":17.567,"15min":17.567},"last":1.2829999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.6899999999999999,"15min":1.6499999999999999},"min":{"1min":1.099,"5min":0.92700000000000005,"15min":0.69799999999999995},"max":{"1min":1.9990000000000001,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.591,"5min":1.679,"15min":1.8300000000000001},"min":{"1min":1.0029999999999999,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.966,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.502}]},{"osd":6,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.6419999999999999,"15min":1.923},"min":{"1min":1.1879999999999999,"5min":1.004,"15min":0.70599999999999996},"max":{"1min":1.875,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.595,"5min":1.6619999999999999,"15min":1.8440000000000001},"min":{"1min":1.2130000000000001,"5min":0.90600000000000003,"15min":0.62},"max":{"1min":2.2149999999999999,"5min":6.2270000000000003,"15min":35.255000000000003},"last":1.3540000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.704,"15min":1.9319999999999999},"min":{"1min":1.153,"5min":1.089,"15min":0.56200000000000006},"max":{"1min":2.2919999999999998,"5min":9.9429999999999996,"15min":35.636000000000003},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.6639999999999999,"15min":1.6919999999999999},"min":{"1min":1.1759999999999999,"5min":0.96499999999999997,"15min":0.64000000000000001},"max":{"1min":2.056,"5min":10.943,"15min":11.470000000000001},"last":1.4419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":640105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":640063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:20:01.615 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.615 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.615 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.615 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.616 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.616 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.616 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.616 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.616 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.32 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.617 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.618 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.618 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.618 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.618 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.618 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.618 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.619 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.619 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.619 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.619 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.619 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.619 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.620 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.620 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.620 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.620 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.620 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.620 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.621 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.622 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.622 INFO:tasks.ceph:pgid 16.1a last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.622 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.622 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.622 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.622 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.623 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.623 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.623 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.623 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.623 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.623 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 16.13 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.624 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.625 INFO:tasks.ceph:pgid 16.10 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.625 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.625 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.625 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.625 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.625 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.626 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.626 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.626 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.626 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.626 INFO:tasks.ceph:pgid 16.b last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.626 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.627 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.627 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.627 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.627 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.627 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.627 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.628 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.628 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.628 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.628 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.628 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.628 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.629 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.629 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.629 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.629 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.629 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.629 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.630 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.630 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.630 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-15T17:57:25.939937+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=17, tm_min=57, tm_sec=25, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:01.630 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-15T18:20:05.073 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:05.081 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:21.631 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:20:21.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.859+0000 7f929b200640 1 Processor -- start 2024-01-15T18:20:21.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.859+0000 7f929b200640 1 -- start start 2024-01-15T18:20:21.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c15f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929b200640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f929c06cad0 0x7f929c06cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f929c0716d0 con 0x7f929c141880 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929b200640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929c071f20 con 0x7f929c06cad0 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c06c6b0 con 0x7f929c15ec60 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f929c06cad0 0x7f929c06cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f9299e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f929c141880 0x7f929c070fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:52180/0 (socket says 172.21.15.2:52180) 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f929a800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f929c06cad0 0x7f929c06cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55364/0 (socket says 172.21.15.2:55364) 2024-01-15T18:20:21.864 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.861+0000 7f9299e00640 1 -- 172.21.15.2:0/1742880801 learned_addr learned my addr 172.21.15.2:0/1742880801 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:20:21.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.862+0000 7f929a800640 1 -- 172.21.15.2:0/1742880801 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 msgr2=0x7f929c15f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:21.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.862+0000 7f929a800640 1 --2- 172.21.15.2:0/1742880801 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c15f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:21.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.862+0000 7f929a800640 1 -- 172.21.15.2:0/1742880801 >> v1:172.21.15.2:6789/0 conn(0x7f929c141880 legacy=0x7f929c070fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:21.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.862+0000 7f929a800640 1 -- 172.21.15.2:0/1742880801 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c071960 con 0x7f929c06cad0 2024-01-15T18:20:21.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.863+0000 7f929a800640 1 --2- 172.21.15.2:0/1742880801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c06cf60 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f9294009d90 tx=0x7f929402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f796bfe9b62ec3 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:21.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.863+0000 7f9299400640 1 -- 172.21.15.2:0/1742880801 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929402cbd0 con 0x7f929c06cad0 2024-01-15T18:20:21.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.863+0000 7f9299400640 1 -- 172.21.15.2:0/1742880801 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929402cd30 con 0x7f929c06cad0 2024-01-15T18:20:21.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.863+0000 7f9299400640 1 -- 172.21.15.2:0/1742880801 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929402cea0 con 0x7f929c06cad0 2024-01-15T18:20:21.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 -- 172.21.15.2:0/1742880801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 msgr2=0x7f929c06cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:21.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 --2- 172.21.15.2:0/1742880801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c06cf60 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f9294009d90 tx=0x7f929402be20 comp rx=0 tx=0).stop 2024-01-15T18:20:21.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 -- 172.21.15.2:0/1742880801 shutdown_connections 2024-01-15T18:20:21.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 --2- 172.21.15.2:0/1742880801 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c06cf60 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:21.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 --2- 172.21.15.2:0/1742880801 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c15f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:21.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 -- 172.21.15.2:0/1742880801 >> 172.21.15.2:0/1742880801 conn(0x7f929c0b0840 msgr2=0x7f929c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:21.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 -- 172.21.15.2:0/1742880801 shutdown_connections 2024-01-15T18:20:21.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.864+0000 7f929b200640 1 -- 172.21.15.2:0/1742880801 wait complete. 2024-01-15T18:20:21.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.866+0000 7f929b200640 1 Processor -- start 2024-01-15T18:20:21.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.866+0000 7f929b200640 1 -- start start 2024-01-15T18:20:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f929c141880 0x7f929c0ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c0cf2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0ce440 con 0x7f929c06cad0 2024-01-15T18:20:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c158c70 con 0x7f929c141880 2024-01-15T18:20:21.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c136ba0 con 0x7f929c15ec60 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c158730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:55378/0 (socket says 172.21.15.2:55378) 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 -- 172.21.15.2:0/906118249 learned_addr learned my addr 172.21.15.2:0/906118249 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9293e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f929c141880 0x7f929c0ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f929a800640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c0cf2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 -- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 msgr2=0x7f929c0cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c0cf2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 -- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f929c141880 msgr2=0x7f929c0ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:21.872 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.868+0000 7f9299e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f929c141880 0x7f929c0ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f9299e00640 1 -- 172.21.15.2:0/906118249 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9294009a70 con 0x7f929c06cad0 2024-01-15T18:20:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f9293e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f929c141880 0x7f929c0ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:20:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f929a800640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:20:21.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f9299e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c158730 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c480 tx=0x7f928c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c005ce0 con 0x7f929c06cad0 2024-01-15T18:20:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c005e40 con 0x7f929c06cad0 2024-01-15T18:20:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c0cd7c0 con 0x7f929c06cad0 2024-01-15T18:20:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c0072f0 con 0x7f929c06cad0 2024-01-15T18:20:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.869+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c0cdd80 con 0x7f929c06cad0 2024-01-15T18:20:21.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.871+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c0ce6b0 con 0x7f929c06cad0 2024-01-15T18:20:21.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.872+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f928c007510 con 0x7f929c06cad0 2024-01-15T18:20:21.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.873+0000 7f9292000640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f927406c550 0x7f927406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:21.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.873+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f928c087fb0 con 0x7f929c06cad0 2024-01-15T18:20:21.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.874+0000 7f9293e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f927406c550 0x7f927406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:21.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.875+0000 7f9293e00640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f927406c550 0x7f927406ea10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9284006fd0 tx=0x7f9284008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:21.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:21.880+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f928c054600 con 0x7f929c06cad0 2024-01-15T18:20:22.092 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.090+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f929c141e80 con 0x7f927406c550 2024-01-15T18:20:22.116 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.114+0000 7f9292000640 1 -- 172.21.15.2:0/906118249 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289105 (secure 0 0 0) 0x7f928400ae20 con 0x7f927406c550 2024-01-15T18:20:22.121 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:20:22.122 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:20:22.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f927406c550 msgr2=0x7f927406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:22.124 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f927406c550 0x7f927406ea10 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9284006fd0 tx=0x7f9284008040 comp rx=0 tx=0).stop 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 msgr2=0x7f929c158730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c158730 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c480 tx=0x7f928c00c940 comp rx=0 tx=0).stop 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 shutdown_connections 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f927406c550 0x7f927406ea10 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f929c15ec60 0x7f929c0cf2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f929c141880 0x7f929c0ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 --2- 172.21.15.2:0/906118249 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f929c06cad0 0x7f929c158730 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 >> 172.21.15.2:0/906118249 conn(0x7f929c0b0840 msgr2=0x7f929c0705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:22.125 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 shutdown_connections 2024-01-15T18:20:22.126 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:22.120+0000 7f929b200640 1 -- 172.21.15.2:0/906118249 wait complete. 2024-01-15T18:20:22.141 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2024-01-15T18:20:21.970247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369946886,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1753,"num_write_kb":5246073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3782,"ondisk_log_size":3782,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11074624,"kb_used_data":10499520,"kb_used_omap":12,"kb_used_meta":575027,"kb_avail":743900096,"statfs":{"total":773094113280,"available":761753698304,"internally_reserved":0,"allocated":10751508480,"data_stored":10744949812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":588828209},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005433"},"pg_stats":[{"pgid":"17.3f","version":"116'53","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.473424+0000","last_change":"2024-01-15T18:14:16.890645+0000","last_active":"2024-01-15T18:19:45.473424+0000","last_peered":"2024-01-15T18:19:45.473424+0000","last_clean":"2024-01-15T18:19:45.473424+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:19:45.473424+0000","last_undegraded":"2024-01-15T18:19:45.473424+0000","last_fullsized":"2024-01-15T18:19:45.473424+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'45","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.529480+0000","last_change":"2024-01-15T18:14:16.877013+0000","last_active":"2024-01-15T18:19:45.529480+0000","last_peered":"2024-01-15T18:19:45.529480+0000","last_clean":"2024-01-15T18:19:45.529480+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:19:45.529480+0000","last_undegraded":"2024-01-15T18:19:45.529480+0000","last_fullsized":"2024-01-15T18:19:45.529480+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'58","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.058139+0000","last_change":"2024-01-15T18:20:08.058201+0000","last_active":"2024-01-15T18:20:08.058139+0000","last_peered":"2024-01-15T18:20:08.058139+0000","last_clean":"2024-01-15T18:20:08.058139+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:20:08.058139+0000","last_undegraded":"2024-01-15T18:20:08.058139+0000","last_fullsized":"2024-01-15T18:20:08.058139+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:08.058029+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:08.058029+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.058029+0000","objects_scrubbed":19,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:17.015587+0000","scrub_duration":0.240566523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'45","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:55.853008+0000","last_change":"2024-01-15T18:14:16.891866+0000","last_active":"2024-01-15T18:19:55.853008+0000","last_peered":"2024-01-15T18:19:55.853008+0000","last_clean":"2024-01-15T18:19:55.853008+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:19:55.853008+0000","last_undegraded":"2024-01-15T18:19:55.853008+0000","last_fullsized":"2024-01-15T18:19:55.853008+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'27","reported_seq":50,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.370847+0000","last_change":"2024-01-15T18:14:16.884530+0000","last_active":"2024-01-15T18:19:45.370847+0000","last_peered":"2024-01-15T18:19:45.370847+0000","last_clean":"2024-01-15T18:19:45.370847+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:19:45.370847+0000","last_undegraded":"2024-01-15T18:19:45.370847+0000","last_fullsized":"2024-01-15T18:19:45.370847+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'48","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.658824+0000","last_change":"2024-01-15T18:14:16.883554+0000","last_active":"2024-01-15T18:19:45.658824+0000","last_peered":"2024-01-15T18:19:45.658824+0000","last_clean":"2024-01-15T18:19:45.658824+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:19:45.658824+0000","last_undegraded":"2024-01-15T18:19:45.658824+0000","last_fullsized":"2024-01-15T18:19:45.658824+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'43","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.409266+0000","last_change":"2024-01-15T18:14:16.875617+0000","last_active":"2024-01-15T18:19:45.409266+0000","last_peered":"2024-01-15T18:19:45.409266+0000","last_clean":"2024-01-15T18:19:45.409266+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:19:45.409266+0000","last_undegraded":"2024-01-15T18:19:45.409266+0000","last_fullsized":"2024-01-15T18:19:45.409266+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'44","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.399286+0000","last_change":"2024-01-15T18:14:16.888465+0000","last_active":"2024-01-15T18:19:45.399286+0000","last_peered":"2024-01-15T18:19:45.399286+0000","last_clean":"2024-01-15T18:19:45.399286+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:19:45.399286+0000","last_undegraded":"2024-01-15T18:19:45.399286+0000","last_fullsized":"2024-01-15T18:19:45.399286+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:13.940273+0000","last_change":"2024-01-15T18:20:13.940362+0000","last_active":"2024-01-15T18:20:13.940273+0000","last_peered":"2024-01-15T18:20:13.940273+0000","last_clean":"2024-01-15T18:20:13.940273+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:20:13.940273+0000","last_undegraded":"2024-01-15T18:20:13.940273+0000","last_fullsized":"2024-01-15T18:20:13.940273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:13.940094+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:13.940094+0000","last_clean_scrub_stamp":"2024-01-15T18:20:13.940094+0000","objects_scrubbed":21,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:46:44.099042+0000","scrub_duration":0.21865759100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'47","reported_seq":73,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.898878+0000","last_change":"2024-01-15T18:14:16.886960+0000","last_active":"2024-01-15T18:19:45.898878+0000","last_peered":"2024-01-15T18:19:45.898878+0000","last_clean":"2024-01-15T18:19:45.898878+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:19:45.898878+0000","last_undegraded":"2024-01-15T18:19:45.898878+0000","last_fullsized":"2024-01-15T18:19:45.898878+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'47","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.835474+0000","last_change":"2024-01-15T18:14:16.875308+0000","last_active":"2024-01-15T18:19:45.835474+0000","last_peered":"2024-01-15T18:19:45.835474+0000","last_clean":"2024-01-15T18:19:45.835474+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:19:45.835474+0000","last_undegraded":"2024-01-15T18:19:45.835474+0000","last_fullsized":"2024-01-15T18:19:45.835474+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'76","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.067580+0000","last_change":"2024-01-15T18:20:18.067609+0000","last_active":"2024-01-15T18:20:18.067580+0000","last_peered":"2024-01-15T18:20:18.067580+0000","last_clean":"2024-01-15T18:20:18.067580+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:20:18.067580+0000","last_undegraded":"2024-01-15T18:20:18.067580+0000","last_fullsized":"2024-01-15T18:20:18.067580+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'76","last_scrub_stamp":"2024-01-15T18:20:18.067485+0000","last_deep_scrub":"116'76","last_deep_scrub_stamp":"2024-01-15T18:20:18.067485+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.067485+0000","objects_scrubbed":28,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:51.591673+0000","scrub_duration":0.24627128300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'38","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.394749+0000","last_change":"2024-01-15T18:14:16.885512+0000","last_active":"2024-01-15T18:19:45.394749+0000","last_peered":"2024-01-15T18:19:45.394749+0000","last_clean":"2024-01-15T18:19:45.394749+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:19:45.394749+0000","last_undegraded":"2024-01-15T18:19:45.394749+0000","last_fullsized":"2024-01-15T18:19:45.394749+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'70","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.101682+0000","last_change":"2024-01-15T18:20:14.101766+0000","last_active":"2024-01-15T18:20:14.101682+0000","last_peered":"2024-01-15T18:20:14.101682+0000","last_clean":"2024-01-15T18:20:14.101682+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:20:14.101682+0000","last_undegraded":"2024-01-15T18:20:14.101682+0000","last_fullsized":"2024-01-15T18:20:14.101682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'70","last_scrub_stamp":"2024-01-15T18:20:14.101527+0000","last_deep_scrub":"116'70","last_deep_scrub_stamp":"2024-01-15T18:20:14.101527+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.101527+0000","objects_scrubbed":27,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:33:26.300645+0000","scrub_duration":0.32403122899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'41","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.927343+0000","last_change":"2024-01-15T18:20:09.927446+0000","last_active":"2024-01-15T18:20:09.927343+0000","last_peered":"2024-01-15T18:20:09.927343+0000","last_clean":"2024-01-15T18:20:09.927343+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:20:09.927343+0000","last_undegraded":"2024-01-15T18:20:09.927343+0000","last_fullsized":"2024-01-15T18:20:09.927343+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'41","last_scrub_stamp":"2024-01-15T18:20:09.927172+0000","last_deep_scrub":"116'41","last_deep_scrub_stamp":"2024-01-15T18:20:09.927172+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.927172+0000","objects_scrubbed":12,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:47.958926+0000","scrub_duration":0.12571378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'58","reported_seq":85,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.924633+0000","last_change":"2024-01-15T18:20:11.924771+0000","last_active":"2024-01-15T18:20:11.924633+0000","last_peered":"2024-01-15T18:20:11.924633+0000","last_clean":"2024-01-15T18:20:11.924633+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:20:11.924633+0000","last_undegraded":"2024-01-15T18:20:11.924633+0000","last_fullsized":"2024-01-15T18:20:11.924633+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:11.924423+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:11.924423+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.924423+0000","objects_scrubbed":18,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:06.081196+0000","scrub_duration":0.195491471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'44","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.750575+0000","last_change":"2024-01-15T18:14:16.880289+0000","last_active":"2024-01-15T18:19:45.750575+0000","last_peered":"2024-01-15T18:19:45.750575+0000","last_clean":"2024-01-15T18:19:45.750575+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:19:45.750575+0000","last_undegraded":"2024-01-15T18:19:45.750575+0000","last_fullsized":"2024-01-15T18:19:45.750575+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'48","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.801690+0000","last_change":"2024-01-15T18:14:16.877937+0000","last_active":"2024-01-15T18:19:45.801690+0000","last_peered":"2024-01-15T18:19:45.801690+0000","last_clean":"2024-01-15T18:19:45.801690+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:19:45.801690+0000","last_undegraded":"2024-01-15T18:19:45.801690+0000","last_fullsized":"2024-01-15T18:19:45.801690+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'61","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.024154+0000","last_change":"2024-01-15T18:20:09.024235+0000","last_active":"2024-01-15T18:20:09.024154+0000","last_peered":"2024-01-15T18:20:09.024154+0000","last_clean":"2024-01-15T18:20:09.024154+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:20:09.024154+0000","last_undegraded":"2024-01-15T18:20:09.024154+0000","last_fullsized":"2024-01-15T18:20:09.024154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'61","last_scrub_stamp":"2024-01-15T18:20:09.023969+0000","last_deep_scrub":"116'61","last_deep_scrub_stamp":"2024-01-15T18:20:09.023969+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.023969+0000","objects_scrubbed":21,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:22.924312+0000","scrub_duration":0.233465062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.356321+0000","last_change":"2024-01-15T18:20:14.356433+0000","last_active":"2024-01-15T18:20:14.356321+0000","last_peered":"2024-01-15T18:20:14.356321+0000","last_clean":"2024-01-15T18:20:14.356321+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:20:14.356321+0000","last_undegraded":"2024-01-15T18:20:14.356321+0000","last_fullsized":"2024-01-15T18:20:14.356321+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:14.356135+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:14.356135+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.356135+0000","objects_scrubbed":17,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:35:09.060881+0000","scrub_duration":0.21269460900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'76","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.437476+0000","last_change":"2024-01-15T18:20:09.437603+0000","last_active":"2024-01-15T18:20:09.437476+0000","last_peered":"2024-01-15T18:20:09.437476+0000","last_clean":"2024-01-15T18:20:09.437476+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:20:09.437476+0000","last_undegraded":"2024-01-15T18:20:09.437476+0000","last_fullsized":"2024-01-15T18:20:09.437476+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'76","last_scrub_stamp":"2024-01-15T18:20:09.437240+0000","last_deep_scrub":"116'76","last_deep_scrub_stamp":"2024-01-15T18:20:09.437240+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.437240+0000","objects_scrubbed":26,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:44:46.474036+0000","scrub_duration":0.288134063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'68","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:17.575494+0000","last_change":"2024-01-15T18:20:17.575599+0000","last_active":"2024-01-15T18:20:17.575494+0000","last_peered":"2024-01-15T18:20:17.575494+0000","last_clean":"2024-01-15T18:20:17.575494+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:20:17.575494+0000","last_undegraded":"2024-01-15T18:20:17.575494+0000","last_fullsized":"2024-01-15T18:20:17.575494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:17.575306+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:17.575306+0000","last_clean_scrub_stamp":"2024-01-15T18:20:17.575306+0000","objects_scrubbed":21,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:05.967956+0000","scrub_duration":0.230033495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'39","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.671802+0000","last_change":"2024-01-15T18:14:16.882438+0000","last_active":"2024-01-15T18:19:45.671802+0000","last_peered":"2024-01-15T18:19:45.671802+0000","last_clean":"2024-01-15T18:19:45.671802+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:19:45.671802+0000","last_undegraded":"2024-01-15T18:19:45.671802+0000","last_fullsized":"2024-01-15T18:19:45.671802+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'57","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.630610+0000","last_change":"2024-01-15T18:20:06.630687+0000","last_active":"2024-01-15T18:20:06.630610+0000","last_peered":"2024-01-15T18:20:06.630610+0000","last_clean":"2024-01-15T18:20:06.630610+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:20:06.630610+0000","last_undegraded":"2024-01-15T18:20:06.630610+0000","last_fullsized":"2024-01-15T18:20:06.630610+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:06.630489+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:06.630489+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.630489+0000","objects_scrubbed":18,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:43.143918+0000","scrub_duration":0.18958351200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'38","reported_seq":64,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.820578+0000","last_change":"2024-01-15T18:14:16.876107+0000","last_active":"2024-01-15T18:19:45.820578+0000","last_peered":"2024-01-15T18:19:45.820578+0000","last_clean":"2024-01-15T18:19:45.820578+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:19:45.820578+0000","last_undegraded":"2024-01-15T18:19:45.820578+0000","last_fullsized":"2024-01-15T18:19:45.820578+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'73","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.039273+0000","last_change":"2024-01-15T18:19:57.039365+0000","last_active":"2024-01-15T18:19:57.039273+0000","last_peered":"2024-01-15T18:19:57.039273+0000","last_clean":"2024-01-15T18:19:57.039273+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:57.039273+0000","last_undegraded":"2024-01-15T18:19:57.039273+0000","last_fullsized":"2024-01-15T18:19:57.039273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.039107+0000","objects_scrubbed":23,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.107907+0000","scrub_duration":0.24278212099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'37","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407476+0000","last_change":"2024-01-15T18:14:16.893052+0000","last_active":"2024-01-15T18:19:45.407476+0000","last_peered":"2024-01-15T18:19:45.407476+0000","last_clean":"2024-01-15T18:19:45.407476+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:19:45.407476+0000","last_undegraded":"2024-01-15T18:19:45.407476+0000","last_fullsized":"2024-01-15T18:19:45.407476+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'49","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.519889+0000","last_change":"2024-01-15T18:20:14.519960+0000","last_active":"2024-01-15T18:20:14.519889+0000","last_peered":"2024-01-15T18:20:14.519889+0000","last_clean":"2024-01-15T18:20:14.519889+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:20:14.519889+0000","last_undegraded":"2024-01-15T18:20:14.519889+0000","last_fullsized":"2024-01-15T18:20:14.519889+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:14.519687+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:14.519687+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.519687+0000","objects_scrubbed":17,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:50:17.600055+0000","scrub_duration":0.16789694699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'46","reported_seq":71,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.786616+0000","last_change":"2024-01-15T18:14:16.886552+0000","last_active":"2024-01-15T18:19:45.786616+0000","last_peered":"2024-01-15T18:19:45.786616+0000","last_clean":"2024-01-15T18:19:45.786616+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:19:45.786616+0000","last_undegraded":"2024-01-15T18:19:45.786616+0000","last_fullsized":"2024-01-15T18:19:45.786616+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'56","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:15.453872+0000","last_change":"2024-01-15T18:20:15.454009+0000","last_active":"2024-01-15T18:20:15.453872+0000","last_peered":"2024-01-15T18:20:15.453872+0000","last_clean":"2024-01-15T18:20:15.453872+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:20:15.453872+0000","last_undegraded":"2024-01-15T18:20:15.453872+0000","last_fullsized":"2024-01-15T18:20:15.453872+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:15.453578+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:15.453578+0000","last_clean_scrub_stamp":"2024-01-15T18:20:15.453578+0000","objects_scrubbed":16,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:16:00.845188+0000","scrub_duration":0.18720494500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'62","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.751935+0000","last_change":"2024-01-15T18:14:16.887705+0000","last_active":"2024-01-15T18:19:45.751935+0000","last_peered":"2024-01-15T18:19:45.751935+0000","last_clean":"2024-01-15T18:19:45.751935+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:19:45.751935+0000","last_undegraded":"2024-01-15T18:19:45.751935+0000","last_fullsized":"2024-01-15T18:19:45.751935+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'72","reported_seq":103,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.952179+0000","last_change":"2024-01-15T18:20:07.952237+0000","last_active":"2024-01-15T18:20:07.952179+0000","last_peered":"2024-01-15T18:20:07.952179+0000","last_clean":"2024-01-15T18:20:07.952179+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:20:07.952179+0000","last_undegraded":"2024-01-15T18:20:07.952179+0000","last_fullsized":"2024-01-15T18:20:07.952179+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:07.952059+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:07.952059+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.952059+0000","objects_scrubbed":22,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:04.767828+0000","scrub_duration":0.20312791999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'49","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.030986+0000","last_change":"2024-01-15T18:20:18.030986+0000","last_active":"2024-01-15T18:20:18.030986+0000","last_peered":"2024-01-15T18:20:18.030986+0000","last_clean":"2024-01-15T18:20:18.030986+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:20:18.030986+0000","last_undegraded":"2024-01-15T18:20:18.030986+0000","last_fullsized":"2024-01-15T18:20:18.030986+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:18.030841+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:18.030841+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.030841+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:05.232365+0000","scrub_duration":0.134479771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"116'6","reported_seq":31,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.264098+0000","last_change":"2024-01-15T18:20:11.264156+0000","last_active":"2024-01-15T18:20:11.264098+0000","last_peered":"2024-01-15T18:20:11.264098+0000","last_clean":"2024-01-15T18:20:11.264098+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:20:11.264098+0000","last_undegraded":"2024-01-15T18:20:11.264098+0000","last_fullsized":"2024-01-15T18:20:11.264098+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'6","last_scrub_stamp":"2024-01-15T18:20:11.263910+0000","last_deep_scrub":"116'6","last_deep_scrub_stamp":"2024-01-15T18:20:11.263910+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.263910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:06:05.388628+0000","scrub_duration":0.0087165769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'47","reported_seq":71,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.499228+0000","last_change":"2024-01-15T18:14:16.883437+0000","last_active":"2024-01-15T18:19:45.499228+0000","last_peered":"2024-01-15T18:19:45.499228+0000","last_clean":"2024-01-15T18:19:45.499228+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:19:45.499228+0000","last_undegraded":"2024-01-15T18:19:45.499228+0000","last_fullsized":"2024-01-15T18:19:45.499228+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.887618+0000","last_change":"2024-01-15T18:20:04.887618+0000","last_active":"2024-01-15T18:20:04.887618+0000","last_peered":"2024-01-15T18:20:04.887618+0000","last_clean":"2024-01-15T18:20:04.887618+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:20:04.887618+0000","last_undegraded":"2024-01-15T18:20:04.887618+0000","last_fullsized":"2024-01-15T18:20:04.887618+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:04.887335+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:04.887335+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.887335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:16:28.013839+0000","scrub_duration":0.003453177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'59","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.940971+0000","last_change":"2024-01-15T18:20:05.941055+0000","last_active":"2024-01-15T18:20:05.940971+0000","last_peered":"2024-01-15T18:20:05.940971+0000","last_clean":"2024-01-15T18:20:05.940971+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:20:05.940971+0000","last_undegraded":"2024-01-15T18:20:05.940971+0000","last_fullsized":"2024-01-15T18:20:05.940971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:05.940887+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:05.940887+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.940887+0000","objects_scrubbed":19,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:41:28.613623+0000","scrub_duration":0.17468556599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.370570+0000","last_change":"2024-01-15T18:20:08.370633+0000","last_active":"2024-01-15T18:20:08.370570+0000","last_peered":"2024-01-15T18:20:08.370570+0000","last_clean":"2024-01-15T18:20:08.370570+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:20:08.370570+0000","last_undegraded":"2024-01-15T18:20:08.370570+0000","last_fullsized":"2024-01-15T18:20:08.370570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:08.370363+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:08.370363+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.370363+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:27:33.863692+0000","scrub_duration":0.0069394490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'79","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.081362+0000","last_change":"2024-01-15T18:20:18.081448+0000","last_active":"2024-01-15T18:20:18.081362+0000","last_peered":"2024-01-15T18:20:18.081362+0000","last_clean":"2024-01-15T18:20:18.081362+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:20:18.081362+0000","last_undegraded":"2024-01-15T18:20:18.081362+0000","last_fullsized":"2024-01-15T18:20:18.081362+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'79","last_scrub_stamp":"2024-01-15T18:20:18.081158+0000","last_deep_scrub":"116'79","last_deep_scrub_stamp":"2024-01-15T18:20:18.081158+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.081158+0000","objects_scrubbed":23,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:45.499261+0000","scrub_duration":0.24476172700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.355457+0000","last_change":"2024-01-15T18:20:11.355522+0000","last_active":"2024-01-15T18:20:11.355457+0000","last_peered":"2024-01-15T18:20:11.355457+0000","last_clean":"2024-01-15T18:20:11.355457+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:20:11.355457+0000","last_undegraded":"2024-01-15T18:20:11.355457+0000","last_fullsized":"2024-01-15T18:20:11.355457+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:11.355263+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:11.355263+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.355263+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:18.416652+0000","scrub_duration":0.0069302460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'39","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.825621+0000","last_change":"2024-01-15T18:14:16.886897+0000","last_active":"2024-01-15T18:19:45.825621+0000","last_peered":"2024-01-15T18:19:45.825621+0000","last_clean":"2024-01-15T18:19:45.825621+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:19:45.825621+0000","last_undegraded":"2024-01-15T18:19:45.825621+0000","last_fullsized":"2024-01-15T18:19:45.825621+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.327294+0000","last_change":"2024-01-15T18:20:09.327345+0000","last_active":"2024-01-15T18:20:09.327294+0000","last_peered":"2024-01-15T18:20:09.327294+0000","last_clean":"2024-01-15T18:20:09.327294+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:20:09.327294+0000","last_undegraded":"2024-01-15T18:20:09.327294+0000","last_fullsized":"2024-01-15T18:20:09.327294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:09.327108+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:09.327108+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.327108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:36.860290+0000","scrub_duration":0.003411468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'73","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.265444+0000","last_change":"2024-01-15T18:20:19.265562+0000","last_active":"2024-01-15T18:20:19.265444+0000","last_peered":"2024-01-15T18:20:19.265444+0000","last_clean":"2024-01-15T18:20:19.265444+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:20:19.265444+0000","last_undegraded":"2024-01-15T18:20:19.265444+0000","last_fullsized":"2024-01-15T18:20:19.265444+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:20:19.265264+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:20:19.265264+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.265264+0000","objects_scrubbed":24,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:44.988416+0000","scrub_duration":0.25818551200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.459148+0000","last_change":"2024-01-15T18:20:04.459201+0000","last_active":"2024-01-15T18:20:04.459148+0000","last_peered":"2024-01-15T18:20:04.459148+0000","last_clean":"2024-01-15T18:20:04.459148+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:20:04.459148+0000","last_undegraded":"2024-01-15T18:20:04.459148+0000","last_fullsized":"2024-01-15T18:20:04.459148+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:04.458942+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:04.458942+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.458942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:42.315332+0000","scrub_duration":0.0039832870000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'71","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:00.901734+0000","last_change":"2024-01-15T18:20:00.901779+0000","last_active":"2024-01-15T18:20:00.901734+0000","last_peered":"2024-01-15T18:20:00.901734+0000","last_clean":"2024-01-15T18:20:00.901734+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:20:00.901734+0000","last_undegraded":"2024-01-15T18:20:00.901734+0000","last_fullsized":"2024-01-15T18:20:00.901734+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'71","last_scrub_stamp":"2024-01-15T18:20:00.901655+0000","last_deep_scrub":"116'71","last_deep_scrub_stamp":"2024-01-15T18:20:00.901655+0000","last_clean_scrub_stamp":"2024-01-15T18:20:00.901655+0000","objects_scrubbed":26,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:12.623865+0000","scrub_duration":0.173092522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.168864+0000","last_change":"2024-01-15T18:20:01.168915+0000","last_active":"2024-01-15T18:20:01.168864+0000","last_peered":"2024-01-15T18:20:01.168864+0000","last_clean":"2024-01-15T18:20:01.168864+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:20:01.168864+0000","last_undegraded":"2024-01-15T18:20:01.168864+0000","last_fullsized":"2024-01-15T18:20:01.168864+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:01.168637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:01.168637+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.168637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:58:03.821391+0000","scrub_duration":0.0034424299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'43","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.762269+0000","last_change":"2024-01-15T18:14:16.875332+0000","last_active":"2024-01-15T18:19:45.762269+0000","last_peered":"2024-01-15T18:19:45.762269+0000","last_clean":"2024-01-15T18:19:45.762269+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:19:45.762269+0000","last_undegraded":"2024-01-15T18:19:45.762269+0000","last_fullsized":"2024-01-15T18:19:45.762269+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.769090+0000","last_change":"2024-01-15T18:20:02.769178+0000","last_active":"2024-01-15T18:20:02.769090+0000","last_peered":"2024-01-15T18:20:02.769090+0000","last_clean":"2024-01-15T18:20:02.769090+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:20:02.769090+0000","last_undegraded":"2024-01-15T18:20:02.769090+0000","last_fullsized":"2024-01-15T18:20:02.769090+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.768819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.768819+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.768819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:01.570006+0000","scrub_duration":0.0044113540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'42","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.391717+0000","last_change":"2024-01-15T18:14:16.877861+0000","last_active":"2024-01-15T18:19:45.391717+0000","last_peered":"2024-01-15T18:19:45.391717+0000","last_clean":"2024-01-15T18:19:45.391717+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:19:45.391717+0000","last_undegraded":"2024-01-15T18:19:45.391717+0000","last_fullsized":"2024-01-15T18:19:45.391717+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.364660+0000","last_change":"2024-01-15T18:20:19.364715+0000","last_active":"2024-01-15T18:20:19.364660+0000","last_peered":"2024-01-15T18:20:19.364660+0000","last_clean":"2024-01-15T18:20:19.364660+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:20:19.364660+0000","last_undegraded":"2024-01-15T18:20:19.364660+0000","last_fullsized":"2024-01-15T18:20:19.364660+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'1","last_scrub_stamp":"2024-01-15T18:20:19.364443+0000","last_deep_scrub":"116'1","last_deep_scrub_stamp":"2024-01-15T18:20:19.364443+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.364443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:20:46.747691+0000","scrub_duration":0.003611015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'40","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.704963+0000","last_change":"2024-01-15T18:14:16.894412+0000","last_active":"2024-01-15T18:19:45.704963+0000","last_peered":"2024-01-15T18:19:45.704963+0000","last_clean":"2024-01-15T18:19:45.704963+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:19:45.704963+0000","last_undegraded":"2024-01-15T18:19:45.704963+0000","last_fullsized":"2024-01-15T18:19:45.704963+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.421790+0000","last_change":"2024-01-15T18:20:05.421839+0000","last_active":"2024-01-15T18:20:05.421790+0000","last_peered":"2024-01-15T18:20:05.421790+0000","last_clean":"2024-01-15T18:20:05.421790+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:20:05.421790+0000","last_undegraded":"2024-01-15T18:20:05.421790+0000","last_fullsized":"2024-01-15T18:20:05.421790+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:05.421607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:05.421607+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.421607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:33.730233+0000","scrub_duration":0.0039450379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'63","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:10.661978+0000","last_change":"2024-01-15T18:20:10.662064+0000","last_active":"2024-01-15T18:20:10.661978+0000","last_peered":"2024-01-15T18:20:10.661978+0000","last_clean":"2024-01-15T18:20:10.661978+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:20:10.661978+0000","last_undegraded":"2024-01-15T18:20:10.661978+0000","last_fullsized":"2024-01-15T18:20:10.661978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:10.661798+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:10.661798+0000","last_clean_scrub_stamp":"2024-01-15T18:20:10.661798+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:24.562142+0000","scrub_duration":0.234248187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'15","reported_seq":40,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.785377+0000","last_change":"2024-01-15T18:20:03.785437+0000","last_active":"2024-01-15T18:20:03.785377+0000","last_peered":"2024-01-15T18:20:03.785377+0000","last_clean":"2024-01-15T18:20:03.785377+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:20:03.785377+0000","last_undegraded":"2024-01-15T18:20:03.785377+0000","last_fullsized":"2024-01-15T18:20:03.785377+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'15","last_scrub_stamp":"2024-01-15T18:20:03.785184+0000","last_deep_scrub":"116'15","last_deep_scrub_stamp":"2024-01-15T18:20:03.785184+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.785184+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:03:53.983142+0000","scrub_duration":0.0088990630000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'43","reported_seq":67,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.407675+0000","last_change":"2024-01-15T18:14:16.883818+0000","last_active":"2024-01-15T18:19:45.407675+0000","last_peered":"2024-01-15T18:19:45.407675+0000","last_clean":"2024-01-15T18:19:45.407675+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:19:45.407675+0000","last_undegraded":"2024-01-15T18:19:45.407675+0000","last_fullsized":"2024-01-15T18:19:45.407675+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'193","reported_seq":218,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.790076+0000","last_change":"2024-01-15T18:20:01.790145+0000","last_active":"2024-01-15T18:20:01.790076+0000","last_peered":"2024-01-15T18:20:01.790076+0000","last_clean":"2024-01-15T18:20:01.790076+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:20:01.790076+0000","last_undegraded":"2024-01-15T18:20:01.790076+0000","last_fullsized":"2024-01-15T18:20:01.790076+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'193","last_scrub_stamp":"2024-01-15T18:20:01.789874+0000","last_deep_scrub":"116'193","last_deep_scrub_stamp":"2024-01-15T18:20:01.789874+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.789874+0000","objects_scrubbed":4,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:22.469663+0000","scrub_duration":0.01365885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644263,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":190,"num_write_kb":721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'57","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.643433+0000","last_change":"2024-01-15T18:20:08.643515+0000","last_active":"2024-01-15T18:20:08.643433+0000","last_peered":"2024-01-15T18:20:08.643433+0000","last_clean":"2024-01-15T18:20:08.643433+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:20:08.643433+0000","last_undegraded":"2024-01-15T18:20:08.643433+0000","last_fullsized":"2024-01-15T18:20:08.643433+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:08.643249+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:08.643249+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.643249+0000","objects_scrubbed":21,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:08.158988+0000","scrub_duration":0.22645187899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.780476+0000","last_change":"2024-01-15T18:20:03.780574+0000","last_active":"2024-01-15T18:20:03.780476+0000","last_peered":"2024-01-15T18:20:03.780476+0000","last_clean":"2024-01-15T18:20:03.780476+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:20:03.780476+0000","last_undegraded":"2024-01-15T18:20:03.780476+0000","last_fullsized":"2024-01-15T18:20:03.780476+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:03.780315+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:03.780315+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.780315+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:45.825709+0000","scrub_duration":0.0083488959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'71","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:15.624139+0000","last_change":"2024-01-15T18:20:15.624238+0000","last_active":"2024-01-15T18:20:15.624139+0000","last_peered":"2024-01-15T18:20:15.624139+0000","last_clean":"2024-01-15T18:20:15.624139+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:20:15.624139+0000","last_undegraded":"2024-01-15T18:20:15.624139+0000","last_fullsized":"2024-01-15T18:20:15.624139+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'71","last_scrub_stamp":"2024-01-15T18:20:15.623953+0000","last_deep_scrub":"116'71","last_deep_scrub_stamp":"2024-01-15T18:20:15.623953+0000","last_clean_scrub_stamp":"2024-01-15T18:20:15.623953+0000","objects_scrubbed":21,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:43.042056+0000","scrub_duration":0.25260574800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.908262+0000","last_change":"2024-01-15T18:20:04.908358+0000","last_active":"2024-01-15T18:20:04.908262+0000","last_peered":"2024-01-15T18:20:04.908262+0000","last_clean":"2024-01-15T18:20:04.908262+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:20:04.908262+0000","last_undegraded":"2024-01-15T18:20:04.908262+0000","last_fullsized":"2024-01-15T18:20:04.908262+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:04.908067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:04.908067+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.908067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:57.372868+0000","scrub_duration":0.0042479759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'68","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.679799+0000","last_change":"2024-01-15T18:20:09.679858+0000","last_active":"2024-01-15T18:20:09.679799+0000","last_peered":"2024-01-15T18:20:09.679799+0000","last_clean":"2024-01-15T18:20:09.679799+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:20:09.679799+0000","last_undegraded":"2024-01-15T18:20:09.679799+0000","last_fullsized":"2024-01-15T18:20:09.679799+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:09.679609+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:09.679609+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.679609+0000","objects_scrubbed":25,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:18.637167+0000","scrub_duration":0.26166856799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"116'12","reported_seq":37,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:00.222842+0000","last_change":"2024-01-15T18:20:00.222891+0000","last_active":"2024-01-15T18:20:00.222842+0000","last_peered":"2024-01-15T18:20:00.222842+0000","last_clean":"2024-01-15T18:20:00.222842+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:20:00.222842+0000","last_undegraded":"2024-01-15T18:20:00.222842+0000","last_fullsized":"2024-01-15T18:20:00.222842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:20:00.222745+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:20:00.222745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:00.222745+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:51.445709+0000","scrub_duration":0.010058510999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'24","reported_seq":44,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.819282+0000","last_change":"2024-01-15T18:14:16.893017+0000","last_active":"2024-01-15T18:19:45.819282+0000","last_peered":"2024-01-15T18:19:45.819282+0000","last_clean":"2024-01-15T18:19:45.819282+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:19:45.819282+0000","last_undegraded":"2024-01-15T18:19:45.819282+0000","last_fullsized":"2024-01-15T18:19:45.819282+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"116'2","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.745018+0000","last_change":"2024-01-15T18:20:06.745073+0000","last_active":"2024-01-15T18:20:06.745018+0000","last_peered":"2024-01-15T18:20:06.745018+0000","last_clean":"2024-01-15T18:20:06.745018+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:20:06.745018+0000","last_undegraded":"2024-01-15T18:20:06.745018+0000","last_fullsized":"2024-01-15T18:20:06.745018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-01-15T18:20:06.744839+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-01-15T18:20:06.744839+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.744839+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:58:35.444023+0000","scrub_duration":0.0075306430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'36","reported_seq":63,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.769576+0000","last_change":"2024-01-15T18:14:16.892347+0000","last_active":"2024-01-15T18:19:45.769576+0000","last_peered":"2024-01-15T18:19:45.769576+0000","last_clean":"2024-01-15T18:19:45.769576+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:19:45.769576+0000","last_undegraded":"2024-01-15T18:19:45.769576+0000","last_fullsized":"2024-01-15T18:19:45.769576+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.362044+0000","last_change":"2024-01-15T18:20:09.362155+0000","last_active":"2024-01-15T18:20:09.362044+0000","last_peered":"2024-01-15T18:20:09.362044+0000","last_clean":"2024-01-15T18:20:09.362044+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:20:09.362044+0000","last_undegraded":"2024-01-15T18:20:09.362044+0000","last_fullsized":"2024-01-15T18:20:09.362044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:09.361834+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:09.361834+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.361834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:42.293371+0000","scrub_duration":0.0070512709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'59","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.048473+0000","last_change":"2024-01-15T18:20:06.048581+0000","last_active":"2024-01-15T18:20:06.048473+0000","last_peered":"2024-01-15T18:20:06.048473+0000","last_clean":"2024-01-15T18:20:06.048473+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:20:06.048473+0000","last_undegraded":"2024-01-15T18:20:06.048473+0000","last_fullsized":"2024-01-15T18:20:06.048473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:06.048300+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:06.048300+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.048300+0000","objects_scrubbed":18,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:34.356925+0000","scrub_duration":0.18916992199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"116'2","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.904337+0000","last_change":"2024-01-15T18:20:07.904387+0000","last_active":"2024-01-15T18:20:07.904337+0000","last_peered":"2024-01-15T18:20:07.904337+0000","last_clean":"2024-01-15T18:20:07.904337+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:20:07.904337+0000","last_undegraded":"2024-01-15T18:20:07.904337+0000","last_fullsized":"2024-01-15T18:20:07.904337+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-01-15T18:20:07.904106+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-01-15T18:20:07.904106+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.904106+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:45.760496+0000","scrub_duration":0.0084473900000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'41","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.724682+0000","last_change":"2024-01-15T18:14:16.889436+0000","last_active":"2024-01-15T18:19:45.724682+0000","last_peered":"2024-01-15T18:19:45.724682+0000","last_clean":"2024-01-15T18:19:45.724682+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:19:45.724682+0000","last_undegraded":"2024-01-15T18:19:45.724682+0000","last_fullsized":"2024-01-15T18:19:45.724682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.399525+0000","last_change":"2024-01-15T18:20:06.399574+0000","last_active":"2024-01-15T18:20:06.399525+0000","last_peered":"2024-01-15T18:20:06.399525+0000","last_clean":"2024-01-15T18:20:06.399525+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:20:06.399525+0000","last_undegraded":"2024-01-15T18:20:06.399525+0000","last_fullsized":"2024-01-15T18:20:06.399525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:06.399262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:06.399262+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.399262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:44.431017+0000","scrub_duration":0.0039161960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'64","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.204795+0000","last_change":"2024-01-15T18:20:07.204980+0000","last_active":"2024-01-15T18:20:07.204795+0000","last_peered":"2024-01-15T18:20:07.204795+0000","last_clean":"2024-01-15T18:20:07.204795+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:20:07.204795+0000","last_undegraded":"2024-01-15T18:20:07.204795+0000","last_fullsized":"2024-01-15T18:20:07.204795+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:07.204595+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:07.204595+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.204595+0000","objects_scrubbed":30,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:19:38.823285+0000","scrub_duration":0.36587318699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.818977+0000","last_change":"2024-01-15T18:20:02.819041+0000","last_active":"2024-01-15T18:20:02.818977+0000","last_peered":"2024-01-15T18:20:02.818977+0000","last_clean":"2024-01-15T18:20:02.818977+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:20:02.818977+0000","last_undegraded":"2024-01-15T18:20:02.818977+0000","last_fullsized":"2024-01-15T18:20:02.818977+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:02.818798+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:02.818798+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.818798+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:08:10.712736+0000","scrub_duration":0.0072655630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'43","reported_seq":75,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:49.717729+0000","last_change":"2024-01-15T18:15:20.196078+0000","last_active":"2024-01-15T18:19:49.717729+0000","last_peered":"2024-01-15T18:19:49.717729+0000","last_clean":"2024-01-15T18:19:49.717729+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:19:49.717729+0000","last_undegraded":"2024-01-15T18:19:49.717729+0000","last_fullsized":"2024-01-15T18:19:49.717729+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.258303+0000","last_change":"2024-01-15T18:20:02.258363+0000","last_active":"2024-01-15T18:20:02.258303+0000","last_peered":"2024-01-15T18:20:02.258303+0000","last_clean":"2024-01-15T18:20:02.258303+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:20:02.258303+0000","last_undegraded":"2024-01-15T18:20:02.258303+0000","last_fullsized":"2024-01-15T18:20:02.258303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.258117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.258117+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.258117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:31.738131+0000","scrub_duration":0.003674961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'44","reported_seq":65,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.799867+0000","last_change":"2024-01-15T18:14:16.887800+0000","last_active":"2024-01-15T18:19:45.799867+0000","last_peered":"2024-01-15T18:19:45.799867+0000","last_clean":"2024-01-15T18:19:45.799867+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:19:45.799867+0000","last_undegraded":"2024-01-15T18:19:45.799867+0000","last_fullsized":"2024-01-15T18:19:45.799867+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.795122+0000","last_change":"2024-01-15T18:19:58.795153+0000","last_active":"2024-01-15T18:19:58.795122+0000","last_peered":"2024-01-15T18:19:58.795122+0000","last_clean":"2024-01-15T18:19:58.795122+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:19:58.795122+0000","last_undegraded":"2024-01-15T18:19:58.795122+0000","last_fullsized":"2024-01-15T18:19:58.795122+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.795017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.531385+0000","scrub_duration":0.0028610760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'59","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.536565+0000","last_change":"2024-01-15T18:20:18.536640+0000","last_active":"2024-01-15T18:20:18.536565+0000","last_peered":"2024-01-15T18:20:18.536565+0000","last_clean":"2024-01-15T18:20:18.536565+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:20:18.536565+0000","last_undegraded":"2024-01-15T18:20:18.536565+0000","last_fullsized":"2024-01-15T18:20:18.536565+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:18.536416+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:18.536416+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.536416+0000","objects_scrubbed":17,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:58.867375+0000","scrub_duration":0.181046237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.721429+0000","last_change":"2024-01-15T18:20:05.721465+0000","last_active":"2024-01-15T18:20:05.721429+0000","last_peered":"2024-01-15T18:20:05.721429+0000","last_clean":"2024-01-15T18:20:05.721429+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:20:05.721429+0000","last_undegraded":"2024-01-15T18:20:05.721429+0000","last_fullsized":"2024-01-15T18:20:05.721429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:05.721271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:05.721271+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.721271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:07.002373+0000","scrub_duration":0.003099302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'36","reported_seq":56,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.745588+0000","last_change":"2024-01-15T18:14:16.893384+0000","last_active":"2024-01-15T18:19:45.745588+0000","last_peered":"2024-01-15T18:19:45.745588+0000","last_clean":"2024-01-15T18:19:45.745588+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:19:45.745588+0000","last_undegraded":"2024-01-15T18:19:45.745588+0000","last_fullsized":"2024-01-15T18:19:45.745588+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.509693+0000","last_change":"2024-01-15T18:20:03.509754+0000","last_active":"2024-01-15T18:20:03.509693+0000","last_peered":"2024-01-15T18:20:03.509693+0000","last_clean":"2024-01-15T18:20:03.509693+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:20:03.509693+0000","last_undegraded":"2024-01-15T18:20:03.509693+0000","last_fullsized":"2024-01-15T18:20:03.509693+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:03.509469+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:03.509469+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.509469+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.140490+0000","scrub_duration":0.0069296399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'41","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410870+0000","last_change":"2024-01-15T18:15:20.356060+0000","last_active":"2024-01-15T18:19:45.410870+0000","last_peered":"2024-01-15T18:19:45.410870+0000","last_clean":"2024-01-15T18:19:45.410870+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:19:45.410870+0000","last_undegraded":"2024-01-15T18:19:45.410870+0000","last_fullsized":"2024-01-15T18:19:45.410870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'19","reported_seq":36,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:56.469074+0000","last_change":"2024-01-15T18:14:15.876092+0000","last_active":"2024-01-15T18:19:56.469074+0000","last_peered":"2024-01-15T18:19:56.469074+0000","last_clean":"2024-01-15T18:19:56.469074+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:19:56.469074+0000","last_undegraded":"2024-01-15T18:19:56.469074+0000","last_fullsized":"2024-01-15T18:19:56.469074+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:14.811812+0000","last_clean_scrub_stamp":"2024-01-15T18:14:14.811812+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'66","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.918730+0000","last_change":"2024-01-15T18:19:57.918788+0000","last_active":"2024-01-15T18:19:57.918730+0000","last_peered":"2024-01-15T18:19:57.918730+0000","last_clean":"2024-01-15T18:19:57.918730+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:57.918730+0000","last_undegraded":"2024-01-15T18:19:57.918730+0000","last_fullsized":"2024-01-15T18:19:57.918730+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'66","last_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_deep_scrub":"116'66","last_deep_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.918639+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:38.022160+0000","scrub_duration":0.15638939399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.767160+0000","last_change":"2024-01-15T18:20:04.767210+0000","last_active":"2024-01-15T18:20:04.767160+0000","last_peered":"2024-01-15T18:20:04.767160+0000","last_clean":"2024-01-15T18:20:04.767160+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:20:04.767160+0000","last_undegraded":"2024-01-15T18:20:04.767160+0000","last_fullsized":"2024-01-15T18:20:04.767160+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:04.766974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:04.766974+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.766974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:58.490311+0000","scrub_duration":0.0038318369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'39","reported_seq":61,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.404432+0000","last_change":"2024-01-15T18:14:16.892662+0000","last_active":"2024-01-15T18:19:45.404432+0000","last_peered":"2024-01-15T18:19:45.404432+0000","last_clean":"2024-01-15T18:19:45.404432+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:19:45.404432+0000","last_undegraded":"2024-01-15T18:19:45.404432+0000","last_fullsized":"2024-01-15T18:19:45.404432+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.514201+0000","last_change":"2024-01-15T18:20:01.514236+0000","last_active":"2024-01-15T18:20:01.514201+0000","last_peered":"2024-01-15T18:20:01.514201+0000","last_clean":"2024-01-15T18:20:01.514201+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:20:01.514201+0000","last_undegraded":"2024-01-15T18:20:01.514201+0000","last_fullsized":"2024-01-15T18:20:01.514201+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:01.514092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:01.514092+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.514092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:26.265860+0000","scrub_duration":0.0027639349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'60","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.337109+0000","last_change":"2024-01-15T18:20:07.337206+0000","last_active":"2024-01-15T18:20:07.337109+0000","last_peered":"2024-01-15T18:20:07.337109+0000","last_clean":"2024-01-15T18:20:07.337109+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:20:07.337109+0000","last_undegraded":"2024-01-15T18:20:07.337109+0000","last_fullsized":"2024-01-15T18:20:07.337109+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'60","last_scrub_stamp":"2024-01-15T18:20:07.336876+0000","last_deep_scrub":"116'60","last_deep_scrub_stamp":"2024-01-15T18:20:07.336876+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.336876+0000","objects_scrubbed":16,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:52.572228+0000","scrub_duration":0.19744029199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.373795+0000","last_change":"2024-01-15T18:20:08.373842+0000","last_active":"2024-01-15T18:20:08.373795+0000","last_peered":"2024-01-15T18:20:08.373795+0000","last_clean":"2024-01-15T18:20:08.373795+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:20:08.373795+0000","last_undegraded":"2024-01-15T18:20:08.373795+0000","last_fullsized":"2024-01-15T18:20:08.373795+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:08.373617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:08.373617+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.373617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:55.243354+0000","scrub_duration":0.0034628670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'80","reported_seq":113,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:12.520470+0000","last_change":"2024-01-15T18:20:12.520568+0000","last_active":"2024-01-15T18:20:12.520470+0000","last_peered":"2024-01-15T18:20:12.520470+0000","last_clean":"2024-01-15T18:20:12.520470+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:20:12.520470+0000","last_undegraded":"2024-01-15T18:20:12.520470+0000","last_fullsized":"2024-01-15T18:20:12.520470+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'80","last_scrub_stamp":"2024-01-15T18:20:12.520296+0000","last_deep_scrub":"116'80","last_deep_scrub_stamp":"2024-01-15T18:20:12.520296+0000","last_clean_scrub_stamp":"2024-01-15T18:20:12.520296+0000","objects_scrubbed":29,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:06.085847+0000","scrub_duration":0.32665812100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.144489+0000","last_change":"2024-01-15T18:20:03.144631+0000","last_active":"2024-01-15T18:20:03.144489+0000","last_peered":"2024-01-15T18:20:03.144489+0000","last_clean":"2024-01-15T18:20:03.144489+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:20:03.144489+0000","last_undegraded":"2024-01-15T18:20:03.144489+0000","last_fullsized":"2024-01-15T18:20:03.144489+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:03.144267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:03.144267+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.144267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:45.685511+0000","scrub_duration":0.0035700250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'46","reported_seq":71,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.086888+0000","last_change":"2024-01-15T18:14:16.888113+0000","last_active":"2024-01-15T18:20:01.086888+0000","last_peered":"2024-01-15T18:20:01.086888+0000","last_clean":"2024-01-15T18:20:01.086888+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:20:01.086888+0000","last_undegraded":"2024-01-15T18:20:01.086888+0000","last_fullsized":"2024-01-15T18:20:01.086888+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'12","reported_seq":47,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.182834+0000","last_change":"2024-01-15T18:19:58.182868+0000","last_active":"2024-01-15T18:19:58.182834+0000","last_peered":"2024-01-15T18:19:58.182834+0000","last_clean":"2024-01-15T18:19:58.182834+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:19:58.182834+0000","last_undegraded":"2024-01-15T18:19:58.182834+0000","last_fullsized":"2024-01-15T18:19:58.182834+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.182737+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:33:12.765302+0000","scrub_duration":0.0054820609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'59","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:12.977576+0000","last_change":"2024-01-15T18:20:12.977682+0000","last_active":"2024-01-15T18:20:12.977576+0000","last_peered":"2024-01-15T18:20:12.977576+0000","last_clean":"2024-01-15T18:20:12.977576+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:20:12.977576+0000","last_undegraded":"2024-01-15T18:20:12.977576+0000","last_fullsized":"2024-01-15T18:20:12.977576+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:12.977304+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:12.977304+0000","last_clean_scrub_stamp":"2024-01-15T18:20:12.977304+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:54.139872+0000","scrub_duration":0.23080210500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:10.753379+0000","last_change":"2024-01-15T18:20:10.753379+0000","last_active":"2024-01-15T18:20:10.753379+0000","last_peered":"2024-01-15T18:20:10.753379+0000","last_clean":"2024-01-15T18:20:10.753379+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:20:10.753379+0000","last_undegraded":"2024-01-15T18:20:10.753379+0000","last_fullsized":"2024-01-15T18:20:10.753379+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:10.753091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:10.753091+0000","last_clean_scrub_stamp":"2024-01-15T18:20:10.753091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:13.671420+0000","scrub_duration":0.0040498490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'44","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.536809+0000","last_change":"2024-01-15T18:20:11.536924+0000","last_active":"2024-01-15T18:20:11.536809+0000","last_peered":"2024-01-15T18:20:11.536809+0000","last_clean":"2024-01-15T18:20:11.536809+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:20:11.536809+0000","last_undegraded":"2024-01-15T18:20:11.536809+0000","last_fullsized":"2024-01-15T18:20:11.536809+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'44","last_scrub_stamp":"2024-01-15T18:20:11.536616+0000","last_deep_scrub":"116'44","last_deep_scrub_stamp":"2024-01-15T18:20:11.536616+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.536616+0000","objects_scrubbed":14,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:20.371492+0000","scrub_duration":0.152340903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.388151+0000","last_change":"2024-01-15T18:20:02.388199+0000","last_active":"2024-01-15T18:20:02.388151+0000","last_peered":"2024-01-15T18:20:02.388151+0000","last_clean":"2024-01-15T18:20:02.388151+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:20:02.388151+0000","last_undegraded":"2024-01-15T18:20:02.388151+0000","last_fullsized":"2024-01-15T18:20:02.388151+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.387965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.387965+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.387965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:11:59.203734+0000","scrub_duration":0.0038625069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"116'119","reported_seq":265,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.974811+0000","last_change":"2024-01-15T18:20:01.974882+0000","last_active":"2024-01-15T18:20:01.974811+0000","last_peered":"2024-01-15T18:20:01.974811+0000","last_clean":"2024-01-15T18:20:01.974811+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:20:01.974811+0000","last_undegraded":"2024-01-15T18:20:01.974811+0000","last_fullsized":"2024-01-15T18:20:01.974811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'119","last_scrub_stamp":"2024-01-15T18:20:01.974618+0000","last_deep_scrub":"116'119","last_deep_scrub_stamp":"2024-01-15T18:20:01.974618+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.974618+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:39.831008+0000","scrub_duration":0.053321348999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3377,"ondisk_log_size":3377,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":646382,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":288,"num_write_kb":762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359872,"data_stored":1292764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286,"ondisk_log_size":286,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542421,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3049999999999999,"15min":1.292},"min":{"1min":0.72599999999999998,"5min":0.61599999999999999,"15min":0.54000000000000004},"max":{"1min":1.5429999999999999,"5min":1.8140000000000001,"15min":4.0019999999999998},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.3819999999999999,"15min":1.5009999999999999},"min":{"1min":0.99399999999999999,"5min":0.88400000000000001,"15min":0.69999999999999996},"max":{"1min":1.6319999999999999,"5min":1.8300000000000001,"15min":27.32},"last":1.3520000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.24,"15min":1.4490000000000001},"min":{"1min":0.82199999999999995,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.8129999999999999,"5min":2.1000000000000001,"15min":29.167999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.258,"5min":1.2290000000000001,"15min":1.2929999999999999},"min":{"1min":0.80300000000000005,"5min":0.70599999999999996,"15min":0.52800000000000002},"max":{"1min":1.889,"5min":2.2349999999999999,"15min":4.2699999999999996},"last":1.1140000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3580000000000001,"15min":1.3480000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.56000000000000005},"max":{"1min":1.8759999999999999,"5min":2.194,"15min":4.1920000000000002},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.254,"15min":1.4399999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.56599999999999995},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":27.358000000000001},"last":1.016}]},{"osd":3,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3120000000000001,"15min":1.528},"min":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.6890000000000001,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.468},"min":{"1min":0.99099999999999999,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.734,"5min":2.2749999999999999,"15min":4.3879999999999999},"last":1.4199999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.371,"15min":1.49},"min":{"1min":0.87,"5min":0.87,"15min":0.50700000000000001},"max":{"1min":1.8089999999999999,"5min":6.0350000000000001,"15min":27.219000000000001},"last":0.87},{"interface":"front","average":{"1min":1.359,"5min":1.427,"15min":1.6020000000000001},"min":{"1min":0.89000000000000001,"5min":0.79900000000000004,"15min":0.57299999999999995},"max":{"1min":1.823,"5min":3.536,"15min":29.25},"last":1.3260000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.46,"15min":1.554},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.68799999999999994},"max":{"1min":1.8700000000000001,"5min":2.1509999999999998,"15min":11.834},"last":1.486},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5149999999999999,"15min":1.6459999999999999},"min":{"1min":1.169,"5min":0.92900000000000005,"15min":0.60499999999999998},"max":{"1min":2.145,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.169}]},{"osd":6,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.7729999999999999},"min":{"1min":1.29,"5min":1.1060000000000001,"15min":0.79200000000000004},"max":{"1min":2.4249999999999998,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.296},{"interface":"front","average":{"1min":1.577,"5min":1.5449999999999999,"15min":1.7490000000000001},"min":{"1min":1.234,"5min":1.0289999999999999,"15min":0.57299999999999995},"max":{"1min":2.085,"5min":2.754,"15min":29.106999999999999},"last":1.3899999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542421,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.1919999999999999,"15min":1.2370000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":8.8350000000000009},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1719999999999999,"15min":1.2669999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5089999999999999,"5min":1.78,"15min":9.548},"last":1.1299999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2390000000000001,"15min":1.379},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.53300000000000003},"max":{"1min":1.718,"5min":2.1419999999999999,"15min":13.269},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.2889999999999999,"15min":1.3580000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.57799999999999996},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":9.6389999999999993},"last":1.5069999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.234,"15min":1.278},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.69799999999999995},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":7.0030000000000001},"last":1.627},{"interface":"front","average":{"1min":1.343,"5min":1.357,"15min":1.425},"min":{"1min":0.84999999999999998,"5min":0.81100000000000005,"15min":0.49399999999999999},"max":{"1min":1.7390000000000001,"5min":1.889,"15min":8.9359999999999999},"last":1.3260000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4239999999999999,"15min":1.512},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.68500000000000005},"max":{"1min":1.708,"5min":1.946,"15min":16.855},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3919999999999999,"15min":1.476},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.628},"max":{"1min":1.976,"5min":1.976,"15min":8.0139999999999993},"last":1.286}]},{"osd":4,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":2.831,"5min":1.637,"15min":1.544},"min":{"1min":0.86099999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":33.161999999999999,"5min":33.161999999999999,"15min":33.161999999999999},"last":1.579},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.371,"15min":1.4990000000000001},"min":{"1min":0.995,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":10.118},"last":1.3879999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3129999999999999,"15min":1.5129999999999999},"min":{"1min":0.90700000000000003,"5min":0.88,"15min":0.67000000000000004},"max":{"1min":1.804,"5min":1.875,"15min":16.309999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.482,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.66000000000000003},"max":{"1min":1.8540000000000001,"5min":1.9079999999999999,"15min":10.077},"last":1.7130000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5029999999999999,"15min":1.6220000000000001},"min":{"1min":1.196,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":10.302},"last":1.673},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.526,"15min":1.6479999999999999},"min":{"1min":1.0680000000000001,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":2.0640000000000001,"5min":6.3029999999999999,"15min":8.0519999999999996},"last":1.4690000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542423,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":928972,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93442868,"statfs":{"total":96636764160,"available":95685496832,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2549999999999999,"15min":1.365},"min":{"1min":0.80400000000000005,"5min":0.70099999999999996,"15min":0.61099999999999999},"max":{"1min":1.502,"5min":4.1829999999999998,"15min":26.077999999999999},"last":1.115},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3759999999999999,"15min":1.4179999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.56399999999999995},"max":{"1min":1.6859999999999999,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.4490000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2529999999999999,"15min":1.401},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.53300000000000003},"max":{"1min":1.7230000000000001,"5min":1.923,"15min":26},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.375,"15min":1.359},"min":{"1min":0.84899999999999998,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.6699999999999999,"5min":4.0869999999999997,"15min":7.5309999999999997},"last":1.4179999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3320000000000001,"15min":1.498},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.56699999999999995},"max":{"1min":1.6120000000000001,"5min":1.774,"15min":26.234999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.222,"5min":1.349,"15min":1.5149999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.629},"max":{"1min":1.52,"5min":1.8240000000000001,"15min":17.126999999999999},"last":0.93600000000000005}]},{"osd":3,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.335,"15min":1.351},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.67200000000000004},"max":{"1min":1.595,"5min":2.375,"15min":7.6150000000000002},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.462,"5min":1.4850000000000001,"15min":1.6000000000000001},"min":{"1min":1.044,"5min":0.96499999999999997,"15min":0.67900000000000005},"max":{"1min":1.736,"5min":1.855,"15min":26.134},"last":1.28}]},{"osd":4,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.242,"15min":1.3799999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.442},"max":{"1min":1.5589999999999999,"5min":2.286,"15min":9.9979999999999993},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.4359999999999999,"15min":1.6459999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.498},"max":{"1min":1.732,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.333}]},{"osd":6,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.365,"15min":1.383},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.45900000000000002},"max":{"1min":1.833,"5min":2.6640000000000001,"15min":7.9240000000000004},"last":1.51},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4430000000000001,"15min":1.611},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.49199999999999999},"max":{"1min":1.71,"5min":2.0659999999999998,"15min":26.186},"last":1.3779999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.5760000000000001,"15min":1.7270000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.65700000000000003},"max":{"1min":1.8029999999999999,"5min":2.04,"15min":12.326000000000001},"last":1.542},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.538,"15min":1.556},"min":{"1min":0.755,"5min":0.755,"15min":0.60099999999999998},"max":{"1min":1.8109999999999999,"5min":4.3700000000000001,"15min":12.175000000000001},"last":1.0469999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542422,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.226,"15min":1.226},"min":{"1min":0.83699999999999997,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.502,"5min":1.8859999999999999,"15min":13.288},"last":1.073},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1990000000000001,"15min":1.262},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.34699999999999998},"max":{"1min":1.653,"5min":1.7030000000000001,"15min":9.2409999999999997},"last":1.0109999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2470000000000001,"15min":1.357},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":1.78,"15min":28.663},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2709999999999999,"15min":1.3899999999999999},"min":{"1min":0.92600000000000005,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.681,"5min":1.7789999999999999,"15min":13.614000000000001},"last":1.5289999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.3029999999999999,"15min":1.4179999999999999},"min":{"1min":0.83399999999999996,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.673,"5min":1.923,"15min":16.023},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.407,"5min":1.3819999999999999,"15min":1.4710000000000001},"min":{"1min":1.196,"5min":0.72199999999999998,"15min":0.52500000000000002},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":18.858000000000001},"last":1.655}]},{"osd":3,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2729999999999999,"15min":1.331},"min":{"1min":0.84699999999999998,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.6930000000000001,"5min":1.9690000000000001,"15min":5.0860000000000003},"last":1.881},{"interface":"front","average":{"1min":1.478,"5min":1.4470000000000001,"15min":1.5329999999999999},"min":{"1min":1.0600000000000001,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.792,"5min":2.1400000000000001,"15min":18.027000000000001},"last":1.476}]},{"osd":5,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4339999999999999,"15min":1.4099999999999999},"min":{"1min":0.83999999999999997,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.903,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.774},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.4279999999999999},"min":{"1min":0.94999999999999996,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":2.0049999999999999,"5min":2.0960000000000001,"15min":8.5660000000000007},"last":2.0059999999999998}]},{"osd":6,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.425,"15min":1.5409999999999999},"min":{"1min":1.1359999999999999,"5min":0.92900000000000005,"15min":0.495},"max":{"1min":1.865,"5min":2.0110000000000001,"15min":16.103999999999999},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5249999999999999,"15min":1.6240000000000001},"min":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.9350000000000001,"5min":1.966,"15min":12.936999999999999},"last":2.1739999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.655},"min":{"1min":1.2210000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.8540000000000001,"5min":2.4689999999999999,"15min":23.390000000000001},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.571,"15min":1.7050000000000001},"min":{"1min":1.25,"5min":1.0900000000000001,"15min":0.62},"max":{"1min":1.9690000000000001,"5min":2.1019999999999999,"15min":16.763000000000002},"last":1.724}]}]},{"osd":3,"up_from":14,"seq":60129542423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462032,"kb_used_data":1386504,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909808,"statfs":{"total":96636764160,"available":95139643392,"internally_reserved":0,"allocated":1419780096,"data_stored":1418951612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.0229999999999999,"15min":1.133},"min":{"1min":0.56599999999999995,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":9.0299999999999994,"5min":9.0299999999999994,"15min":36.408000000000001},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.095,"15min":1.0880000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":1.427,"5min":1.986,"15min":5.0190000000000001},"last":0.90900000000000003}]},{"osd":1,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.97099999999999997,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":1.2889999999999999,"5min":1.837,"15min":5.226},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1919999999999999,"15min":1.1910000000000001},"min":{"1min":0.48899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7869999999999999,"5min":2.0609999999999999,"15min":2.5790000000000002},"last":1.728}]},{"osd":2,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.248,"15min":1.3360000000000001},"min":{"1min":0.56599999999999995,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.6850000000000001,"5min":2.0150000000000001,"15min":14.025},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.343,"15min":1.337},"min":{"1min":0.58199999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.845,"5min":2.403,"15min":3.6240000000000001},"last":1.623}]},{"osd":4,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.403,"15min":1.448},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.59999999999999998},"max":{"1min":10.542,"5min":10.542,"15min":10.542},"last":1.371},{"interface":"front","average":{"1min":1.306,"5min":1.454,"15min":1.5920000000000001},"min":{"1min":0.83799999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.8100000000000001,"5min":4.8570000000000002,"15min":15.769},"last":1.2270000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.93100000000000005,"5min":0.82399999999999995,"15min":0.69299999999999995},"max":{"1min":2.1629999999999998,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.5389999999999999},{"interface":"front","average":{"1min":3.2229999999999999,"5min":1.8460000000000001,"15min":1.897},"min":{"1min":0.88700000000000001,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":2.4550000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.6120000000000001,"15min":1.7729999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.51400000000000001},"max":{"1min":10.59,"5min":10.59,"15min":32.259999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5820000000000001,"15min":1.6859999999999999},"min":{"1min":0.91000000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":6.8639999999999999},"last":1.4379999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.659,"15min":1.903},"min":{"1min":1.1970000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":2.383},{"interface":"front","average":{"1min":1.839,"5min":1.788,"15min":1.825},"min":{"1min":0.93500000000000005,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":10.625,"5min":16.734000000000002,"15min":17.622},"last":1.677}]}]},{"osd":2,"up_from":14,"seq":60129542422,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.99399999999999999,"15min":1.0089999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":1.4510000000000001,"5min":3.1589999999999998,"15min":10.256},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.998,"5min":1.1140000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.33700000000000002},"max":{"1min":1.653,"5min":2.1110000000000002,"15min":10.319000000000001},"last":1.0629999999999999}]},{"osd":1,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0029999999999999,"15min":1.0269999999999999},"min":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.38600000000000001},"max":{"1min":1.427,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.032,"5min":1.159,"15min":1.2290000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.35299999999999998},"max":{"1min":1.7,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.194}]},{"osd":3,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3,"15min":1.3879999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.44400000000000001},"max":{"1min":1.5349999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.214,"15min":1.29},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.376},"max":{"1min":1.5680000000000001,"5min":2.7589999999999999,"15min":14.946},"last":1.1259999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.5389999999999999,"15min":1.504},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.66100000000000003},"max":{"1min":1.8009999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.446,"15min":1.5700000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.65300000000000002},"max":{"1min":1.6830000000000001,"5min":2.867,"15min":10.279999999999999},"last":1.5980000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.45,"15min":1.556},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.63500000000000001},"max":{"1min":2.1000000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.466},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.599,"15min":1.5940000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.71699999999999997},"max":{"1min":1.8799999999999999,"5min":10.833,"15min":10.833},"last":1.2370000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.5009999999999999,"15min":1.5980000000000001},"min":{"1min":0.90400000000000003,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.8620000000000001,"5min":2.984,"15min":10.185},"last":1.369},{"interface":"front","average":{"1min":1.518,"5min":1.6399999999999999,"15min":1.7270000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.85899999999999999},"max":{"1min":2.194,"5min":10.917,"15min":22.382000000000001},"last":1.5640000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5880000000000001,"15min":1.8260000000000001},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":0.84299999999999997},"max":{"1min":1.8879999999999999,"5min":3.052,"15min":19.986000000000001},"last":1.498},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.651,"15min":1.7729999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.877},"max":{"1min":2.2799999999999998,"5min":3.069,"15min":11.893000000000001},"last":1.631}]}]},{"osd":1,"up_from":14,"seq":60129542422,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1137752,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93234088,"statfs":{"total":96636764160,"available":95471706112,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.95999999999999996,"15min":1.0960000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.36199999999999999},"max":{"1min":1.5669999999999999,"5min":3.4510000000000001,"15min":19.218},"last":1.3109999999999999},{"interface":"front","average":{"1min":0.998,"5min":1.0780000000000001,"15min":1.204},"min":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.825,"5min":1.825,"15min":19.823},"last":0.96799999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96899999999999997,"15min":1.0800000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.31900000000000001},"max":{"1min":1.603,"5min":1.776,"15min":19.02},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.21,"15min":1.385},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.41599999999999998},"max":{"1min":1.5,"5min":1.889,"15min":29.251000000000001},"last":1.5}]},{"osd":3,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3109999999999999,"15min":1.494},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.40899999999999997},"max":{"1min":1.5169999999999999,"5min":2.036,"15min":27.021000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.169,"15min":1.3700000000000001},"min":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.859,"5min":1.859,"15min":31.218},"last":1.252}]},{"osd":4,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4630000000000001,"15min":1.6850000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.70599999999999996},"max":{"1min":1.8360000000000001,"5min":1.9650000000000001,"15min":27.059999999999999},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.409,"15min":1.6100000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.56499999999999995},"max":{"1min":1.97,"5min":1.986,"15min":31.433},"last":1.4410000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4410000000000001,"15min":1.583},"min":{"1min":1.05,"5min":1.05,"15min":0.68500000000000005},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":20.012},"last":1.554},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.383,"15min":1.4730000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.69299999999999995},"max":{"1min":1.829,"5min":1.9410000000000001,"15min":3.9910000000000001},"last":1.109}]},{"osd":6,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.571,"15min":1.768},"min":{"1min":1.03,"5min":1.03,"15min":0.68300000000000005},"max":{"1min":1.9199999999999999,"5min":2.2189999999999999,"15min":29.530000000000001},"last":1.802},{"interface":"front","average":{"1min":1.524,"5min":1.548,"15min":1.7789999999999999},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.61399999999999999},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":31.186},"last":1.768}]},{"osd":7,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.492,"15min":1.7310000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.63},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":26.393999999999998},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.589,"15min":1.794},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.76100000000000001},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":27.582000000000001},"last":1.6379999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542421,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1782112,"kb_used_data":1706584,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92589728,"statfs":{"total":96636764160,"available":94811881472,"internally_reserved":0,"allocated":1747542016,"data_stored":1746697967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.96299999999999997,"15min":1.087},"min":{"1min":0.5,"5min":0.5,"15min":0.30599999999999999},"max":{"1min":1.385,"5min":5.7560000000000002,"15min":20.437999999999999},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.2130000000000001,"15min":1.139},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.32400000000000001},"max":{"1min":1.8089999999999999,"5min":6.3600000000000003,"15min":6.3600000000000003},"last":1.278}]},{"osd":2,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.2250000000000001,"15min":1.296},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.669,"5min":11.06,"15min":20.556000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.141,"5min":1.26,"15min":1.2490000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.39000000000000001},"max":{"1min":1.5800000000000001,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":1.5649999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.228,"15min":1.163},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.42999999999999999},"max":{"1min":1.9259999999999999,"5min":10.885999999999999,"15min":10.885999999999999},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3520000000000001,"15min":1.468},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.45100000000000001},"max":{"1min":1.538,"5min":9.3339999999999996,"15min":20.670000000000002},"last":1.1759999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.512,"15min":1.4810000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.1259999999999999,"5min":9.9969999999999999,"15min":9.9969999999999999},"last":1.7350000000000001},{"interface":"front","average":{"1min":1.29,"5min":1.5429999999999999,"15min":1.4319999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.58799999999999997},"max":{"1min":1.7709999999999999,"5min":17.567,"15min":17.567},"last":1.2250000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.6399999999999999,"15min":1.6339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.69799999999999995},"max":{"1min":2.0049999999999999,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.6319999999999999,"15min":1.7749999999999999},"min":{"1min":1.0780000000000001,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.7829999999999999,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.353}]},{"osd":6,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.6140000000000001,"15min":1.8220000000000001},"min":{"1min":1.244,"5min":1.1060000000000001,"15min":0.70599999999999996},"max":{"1min":1.7749999999999999,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.583,"5min":1.657,"15min":1.671},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.62},"max":{"1min":2.258,"5min":6.2270000000000003,"15min":8.3829999999999991},"last":1.865}]},{"osd":7,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.6990000000000001,"15min":1.7849999999999999},"min":{"1min":0.996,"5min":0.996,"15min":0.56200000000000006},"max":{"1min":2.363,"5min":9.9429999999999996,"15min":24.556999999999999},"last":1.9470000000000001},{"interface":"front","average":{"1min":1.484,"5min":1.6359999999999999,"15min":1.657},"min":{"1min":1.073,"5min":1.048,"15min":0.64000000000000001},"max":{"1min":2.129,"5min":10.943,"15min":11.470000000000001},"last":1.528}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":644375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":644333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:20:22.146 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.147 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.147 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.147 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.147 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.147 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.148 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.148 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.148 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.148 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.148 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.149 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.149 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.149 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.149 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.149 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.150 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.150 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.150 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.150 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.150 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.151 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.151 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.151 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.151 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.151 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.152 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.152 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.152 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.152 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.152 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-15T18:14:14.811812+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=14, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.152 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.153 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:22.153 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-15T18:20:35.220 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:35.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:20:42.154 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:20:42.383 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.380+0000 7fc2c6800640 1 Processor -- start 2024-01-15T18:20:42.384 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.380+0000 7fc2c6800640 1 -- start start 2024-01-15T18:20:42.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c6800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7fc2c00ce080 0x7fc2c0164c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:42.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c6800640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c00ccbe0 con 0x7fc2c0143810 2024-01-15T18:20:42.385 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c6800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c0143bf0 con 0x7fc2c006e1e0 2024-01-15T18:20:42.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c6800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c00b3520 con 0x7fc2c00ce080 2024-01-15T18:20:42.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c5400640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7fc2c0143810 0x7fc2c006dad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:33246/0 (socket says 172.21.15.2:33246) 2024-01-15T18:20:42.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c5400640 1 -- 172.21.15.2:0/797449727 learned_addr learned my addr 172.21.15.2:0/797449727 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:20:42.386 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.382+0000 7fc2c5e00640 1 --2- 172.21.15.2:0/797449727 >> v2:172.21.15.2:3300/0 conn(0x7fc2c00ce080 0x7fc2c0164c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:42.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.383+0000 7fc2bfe00640 1 -- 172.21.15.2:0/797449727 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814076253 0 0) 0x7fc2c00ccbe0 con 0x7fc2c0143810 2024-01-15T18:20:42.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.383+0000 7fc2bfe00640 1 -- 172.21.15.2:0/797449727 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a0005730 con 0x7fc2c0143810 2024-01-15T18:20:42.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.383+0000 7fc2c5e00640 1 -- 172.21.15.2:0/797449727 >> v1:172.21.15.137:6789/0 conn(0x7fc2c006e1e0 legacy=0x7fc2c00b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:20:42.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.383+0000 7fc2c5e00640 1 -- 172.21.15.2:0/797449727 >> v1:172.21.15.2:6789/0 conn(0x7fc2c0143810 legacy=0x7fc2c006dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:42.387 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.383+0000 7fc2c5e00640 1 -- 172.21.15.2:0/797449727 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c00b2ac0 con 0x7fc2c00ce080 2024-01-15T18:20:42.388 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.384+0000 7fc2c5e00640 1 --2- 172.21.15.2:0/797449727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c0164c00 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009d90 tx=0x7fc2b802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ca041d7883a259 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.384+0000 7fc2bfe00640 1 -- 172.21.15.2:0/797449727 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b802cc60 con 0x7fc2c00ce080 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.384+0000 7fc2bfe00640 1 -- 172.21.15.2:0/797449727 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b802cdc0 con 0x7fc2c00ce080 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.384+0000 7fc2bfe00640 1 -- 172.21.15.2:0/797449727 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8034660 con 0x7fc2c00ce080 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 -- 172.21.15.2:0/797449727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 msgr2=0x7fc2c0164c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 --2- 172.21.15.2:0/797449727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c0164c00 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009d90 tx=0x7fc2b802beb0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 -- 172.21.15.2:0/797449727 shutdown_connections 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 --2- 172.21.15.2:0/797449727 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c0164c00 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.389 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 -- 172.21.15.2:0/797449727 >> 172.21.15.2:0/797449727 conn(0x7fc2c00b0840 msgr2=0x7fc2c00b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 -- 172.21.15.2:0/797449727 shutdown_connections 2024-01-15T18:20:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.385+0000 7fc2c6800640 1 -- 172.21.15.2:0/797449727 wait complete. 2024-01-15T18:20:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.387+0000 7fc2c6800640 1 Processor -- start 2024-01-15T18:20:42.390 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.387+0000 7fc2c6800640 1 -- start start 2024-01-15T18:20:42.391 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c6800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c006e1e0 0x7fc2c013e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c6800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c014bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c6800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c0143810 0x7fc2c00c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c6800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c01e5800 con 0x7fc2c00ce080 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c6800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c013ed90 con 0x7fc2c006e1e0 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c6800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c01e5a10 con 0x7fc2c0143810 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c014bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c4a00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c014bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:50288/0 (socket says 172.21.15.2:50288) 2024-01-15T18:20:42.392 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.388+0000 7fc2c4a00640 1 -- 172.21.15.2:0/795737949 learned_addr learned my addr 172.21.15.2:0/795737949 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:20:42.393 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c5400640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c006e1e0 0x7fc2c013e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c4a00640 1 -- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c0143810 msgr2=0x7fc2c00c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c4a00640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c0143810 0x7fc2c00c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c4a00640 1 -- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c006e1e0 msgr2=0x7fc2c013e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c4a00640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c006e1e0 0x7fc2c013e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c4a00640 1 -- 172.21.15.2:0/795737949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b8009a70 con 0x7fc2c00ce080 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c4a00640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c014bf20 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c400 tx=0x7fc2a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a80052d0 con 0x7fc2c00ce080 2024-01-15T18:20:42.394 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a8005430 con 0x7fc2c00ce080 2024-01-15T18:20:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a8003830 con 0x7fc2c00ce080 2024-01-15T18:20:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.389+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c01e5cf0 con 0x7fc2c00ce080 2024-01-15T18:20:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.390+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c01e62b0 con 0x7fc2c00ce080 2024-01-15T18:20:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.391+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c00cbcf0 con 0x7fc2c00ce080 2024-01-15T18:20:42.395 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.392+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 43) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc2a8003990 con 0x7fc2c00ce080 2024-01-15T18:20:42.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.393+0000 7fc2bd600640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29806c5a0 0x7fc29806ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:20:42.396 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.393+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7fc2a80881f0 con 0x7fc2c00ce080 2024-01-15T18:20:42.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.394+0000 7fc2c5400640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29806c5a0 0x7fc29806ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:20:42.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.395+0000 7fc2c5400640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29806c5a0 0x7fc29806ea60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00011e0 tx=0x7fc2b0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:20:42.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.398+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc2a8054730 con 0x7fc2c00ce080 2024-01-15T18:20:42.593 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.589+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2c00cc870 con 0x7fc29806c5a0 2024-01-15T18:20:42.615 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.612+0000 7fc2bd600640 1 -- 172.21.15.2:0/795737949 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290798 (secure 0 0 0) 0x7fc2b0005890 con 0x7fc29806c5a0 2024-01-15T18:20:42.619 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:20:42.621 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:20:42.622 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29806c5a0 msgr2=0x7fc29806ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29806c5a0 0x7fc29806ea60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00011e0 tx=0x7fc2b0009040 comp rx=0 tx=0).stop 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 msgr2=0x7fc2c014bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c014bf20 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fc2a800c400 tx=0x7fc2a800c8c0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 shutdown_connections 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7fc29806c5a0 0x7fc29806ea60 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7fc2c0143810 0x7fc2c00c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.623 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7fc2c00ce080 0x7fc2c014bf20 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 --2- 172.21.15.2:0/795737949 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7fc2c006e1e0 0x7fc2c013e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:20:42.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.617+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 >> 172.21.15.2:0/795737949 conn(0x7fc2c00b0840 msgr2=0x7fc2c0142a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:20:42.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.618+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 shutdown_connections 2024-01-15T18:20:42.624 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:20:42.618+0000 7fc2c6800640 1 -- 172.21.15.2:0/795737949 wait complete. 2024-01-15T18:20:42.640 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":808,"stamp":"2024-01-15T18:20:41.979608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369946886,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1753,"num_write_kb":5246073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4373,"ondisk_log_size":4373,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11062144,"kb_used_data":10499520,"kb_used_omap":12,"kb_used_meta":562547,"kb_avail":743912576,"statfs":{"total":773094113280,"available":761766477824,"internally_reserved":0,"allocated":10751508480,"data_stored":10744949812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":576048689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005580"},"pg_stats":[{"pgid":"17.3f","version":"116'78","reported_seq":113,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:28.138656+0000","last_change":"2024-01-15T18:20:28.138656+0000","last_active":"2024-01-15T18:20:28.138656+0000","last_peered":"2024-01-15T18:20:28.138656+0000","last_clean":"2024-01-15T18:20:28.138656+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:20:28.138656+0000","last_undegraded":"2024-01-15T18:20:28.138656+0000","last_fullsized":"2024-01-15T18:20:28.138656+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'78","last_scrub_stamp":"2024-01-15T18:20:28.138374+0000","last_deep_scrub":"116'78","last_deep_scrub_stamp":"2024-01-15T18:20:28.138374+0000","last_clean_scrub_stamp":"2024-01-15T18:20:28.138374+0000","objects_scrubbed":25,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:47.961088+0000","scrub_duration":0.30264037199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'64","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:28.112068+0000","last_change":"2024-01-15T18:20:28.112068+0000","last_active":"2024-01-15T18:20:28.112068+0000","last_peered":"2024-01-15T18:20:28.112068+0000","last_clean":"2024-01-15T18:20:28.112068+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:20:28.112068+0000","last_undegraded":"2024-01-15T18:20:28.112068+0000","last_fullsized":"2024-01-15T18:20:28.112068+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:28.111850+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:28.111850+0000","last_clean_scrub_stamp":"2024-01-15T18:20:28.111850+0000","objects_scrubbed":19,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.785375+0000","scrub_duration":0.24068798999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'58","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.058201+0000","last_change":"2024-01-15T18:20:08.058201+0000","last_active":"2024-01-15T18:20:08.058201+0000","last_peered":"2024-01-15T18:20:08.058201+0000","last_clean":"2024-01-15T18:20:08.058201+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:20:08.058201+0000","last_undegraded":"2024-01-15T18:20:08.058201+0000","last_fullsized":"2024-01-15T18:20:08.058201+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:08.058029+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:08.058029+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.058029+0000","objects_scrubbed":19,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:17.015587+0000","scrub_duration":0.240566523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'68","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:21.129120+0000","last_change":"2024-01-15T18:20:21.129210+0000","last_active":"2024-01-15T18:20:21.129120+0000","last_peered":"2024-01-15T18:20:21.129120+0000","last_clean":"2024-01-15T18:20:21.129120+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:20:21.129120+0000","last_undegraded":"2024-01-15T18:20:21.129120+0000","last_fullsized":"2024-01-15T18:20:21.129120+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:21.128944+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:21.128944+0000","last_clean_scrub_stamp":"2024-01-15T18:20:21.128944+0000","objects_scrubbed":23,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:22.724305+0000","scrub_duration":0.25324432800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'37","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:27.424747+0000","last_change":"2024-01-15T18:20:27.424890+0000","last_active":"2024-01-15T18:20:27.424747+0000","last_peered":"2024-01-15T18:20:27.424747+0000","last_clean":"2024-01-15T18:20:27.424747+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:20:27.424747+0000","last_undegraded":"2024-01-15T18:20:27.424747+0000","last_fullsized":"2024-01-15T18:20:27.424747+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'37","last_scrub_stamp":"2024-01-15T18:20:27.424520+0000","last_deep_scrub":"116'37","last_deep_scrub_stamp":"2024-01-15T18:20:27.424520+0000","last_clean_scrub_stamp":"2024-01-15T18:20:27.424520+0000","objects_scrubbed":10,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:28:29.494758+0000","scrub_duration":0.109363945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'69","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:21.035113+0000","last_change":"2024-01-15T18:20:21.035167+0000","last_active":"2024-01-15T18:20:21.035113+0000","last_peered":"2024-01-15T18:20:21.035113+0000","last_clean":"2024-01-15T18:20:21.035113+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:20:21.035113+0000","last_undegraded":"2024-01-15T18:20:21.035113+0000","last_fullsized":"2024-01-15T18:20:21.035113+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'69","last_scrub_stamp":"2024-01-15T18:20:21.035022+0000","last_deep_scrub":"116'69","last_deep_scrub_stamp":"2024-01-15T18:20:21.035022+0000","last_clean_scrub_stamp":"2024-01-15T18:20:21.035022+0000","objects_scrubbed":21,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:06:15.159740+0000","scrub_duration":0.179535004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'60","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.440550+0000","last_change":"2024-01-15T18:20:23.440661+0000","last_active":"2024-01-15T18:20:23.440550+0000","last_peered":"2024-01-15T18:20:23.440550+0000","last_clean":"2024-01-15T18:20:23.440550+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:20:23.440550+0000","last_undegraded":"2024-01-15T18:20:23.440550+0000","last_fullsized":"2024-01-15T18:20:23.440550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'60","last_scrub_stamp":"2024-01-15T18:20:23.440397+0000","last_deep_scrub":"116'60","last_deep_scrub_stamp":"2024-01-15T18:20:23.440397+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.440397+0000","objects_scrubbed":17,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:11.569070+0000","scrub_duration":0.20668540199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:32.063388+0000","last_change":"2024-01-15T18:20:32.063563+0000","last_active":"2024-01-15T18:20:32.063388+0000","last_peered":"2024-01-15T18:20:32.063388+0000","last_clean":"2024-01-15T18:20:32.063388+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:20:32.063388+0000","last_undegraded":"2024-01-15T18:20:32.063388+0000","last_fullsized":"2024-01-15T18:20:32.063388+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:32.063198+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:32.063198+0000","last_clean_scrub_stamp":"2024-01-15T18:20:32.063198+0000","objects_scrubbed":19,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:35.428438+0000","scrub_duration":0.227368862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:13.940273+0000","last_change":"2024-01-15T18:20:13.940362+0000","last_active":"2024-01-15T18:20:13.940273+0000","last_peered":"2024-01-15T18:20:13.940273+0000","last_clean":"2024-01-15T18:20:13.940273+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:20:13.940273+0000","last_undegraded":"2024-01-15T18:20:13.940273+0000","last_fullsized":"2024-01-15T18:20:13.940273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:13.940094+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:13.940094+0000","last_clean_scrub_stamp":"2024-01-15T18:20:13.940094+0000","objects_scrubbed":21,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:46:44.099042+0000","scrub_duration":0.21865759100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'64","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:29.039071+0000","last_change":"2024-01-15T18:20:29.039119+0000","last_active":"2024-01-15T18:20:29.039071+0000","last_peered":"2024-01-15T18:20:29.039071+0000","last_clean":"2024-01-15T18:20:29.039071+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:20:29.039071+0000","last_undegraded":"2024-01-15T18:20:29.039071+0000","last_fullsized":"2024-01-15T18:20:29.039071+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:29.038992+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:29.038992+0000","last_clean_scrub_stamp":"2024-01-15T18:20:29.038992+0000","objects_scrubbed":17,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:50.890716+0000","scrub_duration":0.177717706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'75","reported_seq":112,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:26.268972+0000","last_change":"2024-01-15T18:20:26.269040+0000","last_active":"2024-01-15T18:20:26.268972+0000","last_peered":"2024-01-15T18:20:26.268972+0000","last_clean":"2024-01-15T18:20:26.268972+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:20:26.268972+0000","last_undegraded":"2024-01-15T18:20:26.268972+0000","last_fullsized":"2024-01-15T18:20:26.268972+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'75","last_scrub_stamp":"2024-01-15T18:20:26.268745+0000","last_deep_scrub":"116'75","last_deep_scrub_stamp":"2024-01-15T18:20:26.268745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:26.268745+0000","objects_scrubbed":28,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:41:07.930002+0000","scrub_duration":0.35549178100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'76","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.067580+0000","last_change":"2024-01-15T18:20:18.067609+0000","last_active":"2024-01-15T18:20:18.067580+0000","last_peered":"2024-01-15T18:20:18.067580+0000","last_clean":"2024-01-15T18:20:18.067580+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:20:18.067580+0000","last_undegraded":"2024-01-15T18:20:18.067580+0000","last_fullsized":"2024-01-15T18:20:18.067580+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'76","last_scrub_stamp":"2024-01-15T18:20:18.067485+0000","last_deep_scrub":"116'76","last_deep_scrub_stamp":"2024-01-15T18:20:18.067485+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.067485+0000","objects_scrubbed":28,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:51.591673+0000","scrub_duration":0.24627128300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'57","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:29.068517+0000","last_change":"2024-01-15T18:20:29.068609+0000","last_active":"2024-01-15T18:20:29.068517+0000","last_peered":"2024-01-15T18:20:29.068517+0000","last_clean":"2024-01-15T18:20:29.068517+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:20:29.068517+0000","last_undegraded":"2024-01-15T18:20:29.068517+0000","last_fullsized":"2024-01-15T18:20:29.068517+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:29.068261+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:29.068261+0000","last_clean_scrub_stamp":"2024-01-15T18:20:29.068261+0000","objects_scrubbed":19,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:36:32.030241+0000","scrub_duration":0.20564858599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'70","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.101682+0000","last_change":"2024-01-15T18:20:14.101766+0000","last_active":"2024-01-15T18:20:14.101682+0000","last_peered":"2024-01-15T18:20:14.101682+0000","last_clean":"2024-01-15T18:20:14.101682+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:20:14.101682+0000","last_undegraded":"2024-01-15T18:20:14.101682+0000","last_fullsized":"2024-01-15T18:20:14.101682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'70","last_scrub_stamp":"2024-01-15T18:20:14.101527+0000","last_deep_scrub":"116'70","last_deep_scrub_stamp":"2024-01-15T18:20:14.101527+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.101527+0000","objects_scrubbed":27,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:33:26.300645+0000","scrub_duration":0.32403122899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'41","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.927343+0000","last_change":"2024-01-15T18:20:09.927446+0000","last_active":"2024-01-15T18:20:09.927343+0000","last_peered":"2024-01-15T18:20:09.927343+0000","last_clean":"2024-01-15T18:20:09.927343+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:20:09.927343+0000","last_undegraded":"2024-01-15T18:20:09.927343+0000","last_fullsized":"2024-01-15T18:20:09.927343+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'41","last_scrub_stamp":"2024-01-15T18:20:09.927172+0000","last_deep_scrub":"116'41","last_deep_scrub_stamp":"2024-01-15T18:20:09.927172+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.927172+0000","objects_scrubbed":12,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:47.958926+0000","scrub_duration":0.12571378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'58","reported_seq":85,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.924633+0000","last_change":"2024-01-15T18:20:11.924771+0000","last_active":"2024-01-15T18:20:11.924633+0000","last_peered":"2024-01-15T18:20:11.924633+0000","last_clean":"2024-01-15T18:20:11.924633+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:20:11.924633+0000","last_undegraded":"2024-01-15T18:20:11.924633+0000","last_fullsized":"2024-01-15T18:20:11.924633+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:11.924423+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:11.924423+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.924423+0000","objects_scrubbed":18,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:06.081196+0000","scrub_duration":0.195491471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'66","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:24.225189+0000","last_change":"2024-01-15T18:20:24.225257+0000","last_active":"2024-01-15T18:20:24.225189+0000","last_peered":"2024-01-15T18:20:24.225189+0000","last_clean":"2024-01-15T18:20:24.225189+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:20:24.225189+0000","last_undegraded":"2024-01-15T18:20:24.225189+0000","last_fullsized":"2024-01-15T18:20:24.225189+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'66","last_scrub_stamp":"2024-01-15T18:20:24.225072+0000","last_deep_scrub":"116'66","last_deep_scrub_stamp":"2024-01-15T18:20:24.225072+0000","last_clean_scrub_stamp":"2024-01-15T18:20:24.225072+0000","objects_scrubbed":22,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:22.612626+0000","scrub_duration":0.26762065200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'69","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:25.149726+0000","last_change":"2024-01-15T18:20:25.149827+0000","last_active":"2024-01-15T18:20:25.149726+0000","last_peered":"2024-01-15T18:20:25.149726+0000","last_clean":"2024-01-15T18:20:25.149726+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:20:25.149726+0000","last_undegraded":"2024-01-15T18:20:25.149726+0000","last_fullsized":"2024-01-15T18:20:25.149726+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'69","last_scrub_stamp":"2024-01-15T18:20:25.149549+0000","last_deep_scrub":"116'69","last_deep_scrub_stamp":"2024-01-15T18:20:25.149549+0000","last_clean_scrub_stamp":"2024-01-15T18:20:25.149549+0000","objects_scrubbed":21,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:23.968047+0000","scrub_duration":0.27027720900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'61","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.024154+0000","last_change":"2024-01-15T18:20:09.024235+0000","last_active":"2024-01-15T18:20:09.024154+0000","last_peered":"2024-01-15T18:20:09.024154+0000","last_clean":"2024-01-15T18:20:09.024154+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:20:09.024154+0000","last_undegraded":"2024-01-15T18:20:09.024154+0000","last_fullsized":"2024-01-15T18:20:09.024154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'61","last_scrub_stamp":"2024-01-15T18:20:09.023969+0000","last_deep_scrub":"116'61","last_deep_scrub_stamp":"2024-01-15T18:20:09.023969+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.023969+0000","objects_scrubbed":21,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:22.924312+0000","scrub_duration":0.233465062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.356321+0000","last_change":"2024-01-15T18:20:14.356433+0000","last_active":"2024-01-15T18:20:14.356321+0000","last_peered":"2024-01-15T18:20:14.356321+0000","last_clean":"2024-01-15T18:20:14.356321+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:20:14.356321+0000","last_undegraded":"2024-01-15T18:20:14.356321+0000","last_fullsized":"2024-01-15T18:20:14.356321+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:14.356135+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:14.356135+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.356135+0000","objects_scrubbed":17,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:35:09.060881+0000","scrub_duration":0.21269460900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'76","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.437476+0000","last_change":"2024-01-15T18:20:09.437603+0000","last_active":"2024-01-15T18:20:09.437476+0000","last_peered":"2024-01-15T18:20:09.437476+0000","last_clean":"2024-01-15T18:20:09.437476+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:20:09.437476+0000","last_undegraded":"2024-01-15T18:20:09.437476+0000","last_fullsized":"2024-01-15T18:20:09.437476+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'76","last_scrub_stamp":"2024-01-15T18:20:09.437240+0000","last_deep_scrub":"116'76","last_deep_scrub_stamp":"2024-01-15T18:20:09.437240+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.437240+0000","objects_scrubbed":26,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:44:46.474036+0000","scrub_duration":0.288134063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'68","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:17.575494+0000","last_change":"2024-01-15T18:20:17.575599+0000","last_active":"2024-01-15T18:20:17.575494+0000","last_peered":"2024-01-15T18:20:17.575494+0000","last_clean":"2024-01-15T18:20:17.575494+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:20:17.575494+0000","last_undegraded":"2024-01-15T18:20:17.575494+0000","last_fullsized":"2024-01-15T18:20:17.575494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:17.575306+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:17.575306+0000","last_clean_scrub_stamp":"2024-01-15T18:20:17.575306+0000","objects_scrubbed":21,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:05.967956+0000","scrub_duration":0.230033495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'54","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:27.094581+0000","last_change":"2024-01-15T18:20:27.094637+0000","last_active":"2024-01-15T18:20:27.094581+0000","last_peered":"2024-01-15T18:20:27.094581+0000","last_clean":"2024-01-15T18:20:27.094581+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:20:27.094581+0000","last_undegraded":"2024-01-15T18:20:27.094581+0000","last_fullsized":"2024-01-15T18:20:27.094581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'54","last_scrub_stamp":"2024-01-15T18:20:27.094454+0000","last_deep_scrub":"116'54","last_deep_scrub_stamp":"2024-01-15T18:20:27.094454+0000","last_clean_scrub_stamp":"2024-01-15T18:20:27.094454+0000","objects_scrubbed":15,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:41:49.226716+0000","scrub_duration":0.196613649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'57","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.630610+0000","last_change":"2024-01-15T18:20:06.630687+0000","last_active":"2024-01-15T18:20:06.630610+0000","last_peered":"2024-01-15T18:20:06.630610+0000","last_clean":"2024-01-15T18:20:06.630610+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:20:06.630610+0000","last_undegraded":"2024-01-15T18:20:06.630610+0000","last_fullsized":"2024-01-15T18:20:06.630610+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:06.630489+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:06.630489+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.630489+0000","objects_scrubbed":18,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:43.143918+0000","scrub_duration":0.18958351200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'56","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:20.249800+0000","last_change":"2024-01-15T18:20:20.249893+0000","last_active":"2024-01-15T18:20:20.249800+0000","last_peered":"2024-01-15T18:20:20.249800+0000","last_clean":"2024-01-15T18:20:20.249800+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:20:20.249800+0000","last_undegraded":"2024-01-15T18:20:20.249800+0000","last_fullsized":"2024-01-15T18:20:20.249800+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:20.249662+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:20.249662+0000","last_clean_scrub_stamp":"2024-01-15T18:20:20.249662+0000","objects_scrubbed":18,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:13:08.460330+0000","scrub_duration":0.21997658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'73","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.039273+0000","last_change":"2024-01-15T18:19:57.039365+0000","last_active":"2024-01-15T18:19:57.039273+0000","last_peered":"2024-01-15T18:19:57.039273+0000","last_clean":"2024-01-15T18:19:57.039273+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:57.039273+0000","last_undegraded":"2024-01-15T18:19:57.039273+0000","last_fullsized":"2024-01-15T18:19:57.039273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.039107+0000","objects_scrubbed":23,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.107907+0000","scrub_duration":0.24278212099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'58","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:26.009755+0000","last_change":"2024-01-15T18:20:26.009801+0000","last_active":"2024-01-15T18:20:26.009755+0000","last_peered":"2024-01-15T18:20:26.009755+0000","last_clean":"2024-01-15T18:20:26.009755+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:20:26.009755+0000","last_undegraded":"2024-01-15T18:20:26.009755+0000","last_fullsized":"2024-01-15T18:20:26.009755+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:26.009677+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:26.009677+0000","last_clean_scrub_stamp":"2024-01-15T18:20:26.009677+0000","objects_scrubbed":21,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:02.258822+0000","scrub_duration":0.16302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'49","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.519889+0000","last_change":"2024-01-15T18:20:14.519960+0000","last_active":"2024-01-15T18:20:14.519889+0000","last_peered":"2024-01-15T18:20:14.519889+0000","last_clean":"2024-01-15T18:20:14.519889+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:20:14.519889+0000","last_undegraded":"2024-01-15T18:20:14.519889+0000","last_fullsized":"2024-01-15T18:20:14.519889+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:14.519687+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:14.519687+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.519687+0000","objects_scrubbed":17,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:50:17.600055+0000","scrub_duration":0.16789694699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'72","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.222197+0000","last_change":"2024-01-15T18:20:23.222197+0000","last_active":"2024-01-15T18:20:23.222197+0000","last_peered":"2024-01-15T18:20:23.222197+0000","last_clean":"2024-01-15T18:20:23.222197+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:20:23.222197+0000","last_undegraded":"2024-01-15T18:20:23.222197+0000","last_fullsized":"2024-01-15T18:20:23.222197+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:23.221964+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:23.221964+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.221964+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:41.425250+0000","scrub_duration":0.28073583600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'56","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:15.454009+0000","last_change":"2024-01-15T18:20:15.454009+0000","last_active":"2024-01-15T18:20:15.454009+0000","last_peered":"2024-01-15T18:20:15.454009+0000","last_clean":"2024-01-15T18:20:15.454009+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:20:15.454009+0000","last_undegraded":"2024-01-15T18:20:15.454009+0000","last_fullsized":"2024-01-15T18:20:15.454009+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:15.453578+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:15.453578+0000","last_clean_scrub_stamp":"2024-01-15T18:20:15.453578+0000","objects_scrubbed":16,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:16:00.845188+0000","scrub_duration":0.18720494500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'88","reported_seq":126,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:33.231110+0000","last_change":"2024-01-15T18:20:33.231110+0000","last_active":"2024-01-15T18:20:33.231110+0000","last_peered":"2024-01-15T18:20:33.231110+0000","last_clean":"2024-01-15T18:20:33.231110+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:20:33.231110+0000","last_undegraded":"2024-01-15T18:20:33.231110+0000","last_fullsized":"2024-01-15T18:20:33.231110+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'88","last_scrub_stamp":"2024-01-15T18:20:33.230776+0000","last_deep_scrub":"116'88","last_deep_scrub_stamp":"2024-01-15T18:20:33.230776+0000","last_clean_scrub_stamp":"2024-01-15T18:20:33.230776+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:10:37.372939+0000","scrub_duration":0.30655612300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'72","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.952237+0000","last_change":"2024-01-15T18:20:07.952237+0000","last_active":"2024-01-15T18:20:07.952237+0000","last_peered":"2024-01-15T18:20:07.952237+0000","last_clean":"2024-01-15T18:20:07.952237+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:20:07.952237+0000","last_undegraded":"2024-01-15T18:20:07.952237+0000","last_fullsized":"2024-01-15T18:20:07.952237+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:07.952059+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:07.952059+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.952059+0000","objects_scrubbed":22,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:04.767828+0000","scrub_duration":0.20312791999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'49","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.030986+0000","last_change":"2024-01-15T18:20:18.030986+0000","last_active":"2024-01-15T18:20:18.030986+0000","last_peered":"2024-01-15T18:20:18.030986+0000","last_clean":"2024-01-15T18:20:18.030986+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:20:18.030986+0000","last_undegraded":"2024-01-15T18:20:18.030986+0000","last_fullsized":"2024-01-15T18:20:18.030986+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:18.030841+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:18.030841+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.030841+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:05.232365+0000","scrub_duration":0.134479771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"116'6","reported_seq":32,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.264156+0000","last_change":"2024-01-15T18:20:11.264156+0000","last_active":"2024-01-15T18:20:11.264156+0000","last_peered":"2024-01-15T18:20:11.264156+0000","last_clean":"2024-01-15T18:20:11.264156+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:20:11.264156+0000","last_undegraded":"2024-01-15T18:20:11.264156+0000","last_fullsized":"2024-01-15T18:20:11.264156+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'6","last_scrub_stamp":"2024-01-15T18:20:11.263910+0000","last_deep_scrub":"116'6","last_deep_scrub_stamp":"2024-01-15T18:20:11.263910+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.263910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:06:05.388628+0000","scrub_duration":0.0087165769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'69","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:30.237548+0000","last_change":"2024-01-15T18:20:30.237548+0000","last_active":"2024-01-15T18:20:30.237548+0000","last_peered":"2024-01-15T18:20:30.237548+0000","last_clean":"2024-01-15T18:20:30.237548+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:20:30.237548+0000","last_undegraded":"2024-01-15T18:20:30.237548+0000","last_fullsized":"2024-01-15T18:20:30.237548+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'69","last_scrub_stamp":"2024-01-15T18:20:30.237218+0000","last_deep_scrub":"116'69","last_deep_scrub_stamp":"2024-01-15T18:20:30.237218+0000","last_clean_scrub_stamp":"2024-01-15T18:20:30.237218+0000","objects_scrubbed":22,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:33:42.436336+0000","scrub_duration":0.27258786400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.887618+0000","last_change":"2024-01-15T18:20:04.887618+0000","last_active":"2024-01-15T18:20:04.887618+0000","last_peered":"2024-01-15T18:20:04.887618+0000","last_clean":"2024-01-15T18:20:04.887618+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:20:04.887618+0000","last_undegraded":"2024-01-15T18:20:04.887618+0000","last_fullsized":"2024-01-15T18:20:04.887618+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:04.887335+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:04.887335+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.887335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:16:28.013839+0000","scrub_duration":0.003453177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'59","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.940971+0000","last_change":"2024-01-15T18:20:05.941055+0000","last_active":"2024-01-15T18:20:05.940971+0000","last_peered":"2024-01-15T18:20:05.940971+0000","last_clean":"2024-01-15T18:20:05.940971+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:20:05.940971+0000","last_undegraded":"2024-01-15T18:20:05.940971+0000","last_fullsized":"2024-01-15T18:20:05.940971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:05.940887+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:05.940887+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.940887+0000","objects_scrubbed":19,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:41:28.613623+0000","scrub_duration":0.17468556599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.370570+0000","last_change":"2024-01-15T18:20:08.370633+0000","last_active":"2024-01-15T18:20:08.370570+0000","last_peered":"2024-01-15T18:20:08.370570+0000","last_clean":"2024-01-15T18:20:08.370570+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:20:08.370570+0000","last_undegraded":"2024-01-15T18:20:08.370570+0000","last_fullsized":"2024-01-15T18:20:08.370570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:08.370363+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:08.370363+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.370363+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:27:33.863692+0000","scrub_duration":0.0069394490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'79","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.081362+0000","last_change":"2024-01-15T18:20:18.081448+0000","last_active":"2024-01-15T18:20:18.081362+0000","last_peered":"2024-01-15T18:20:18.081362+0000","last_clean":"2024-01-15T18:20:18.081362+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:20:18.081362+0000","last_undegraded":"2024-01-15T18:20:18.081362+0000","last_fullsized":"2024-01-15T18:20:18.081362+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'79","last_scrub_stamp":"2024-01-15T18:20:18.081158+0000","last_deep_scrub":"116'79","last_deep_scrub_stamp":"2024-01-15T18:20:18.081158+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.081158+0000","objects_scrubbed":23,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:45.499261+0000","scrub_duration":0.24476172700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.355457+0000","last_change":"2024-01-15T18:20:11.355522+0000","last_active":"2024-01-15T18:20:11.355457+0000","last_peered":"2024-01-15T18:20:11.355457+0000","last_clean":"2024-01-15T18:20:11.355457+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:20:11.355457+0000","last_undegraded":"2024-01-15T18:20:11.355457+0000","last_fullsized":"2024-01-15T18:20:11.355457+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:11.355263+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:11.355263+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.355263+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:18.416652+0000","scrub_duration":0.0069302460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'56","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:25.987945+0000","last_change":"2024-01-15T18:20:25.988008+0000","last_active":"2024-01-15T18:20:25.987945+0000","last_peered":"2024-01-15T18:20:25.987945+0000","last_clean":"2024-01-15T18:20:25.987945+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:20:25.987945+0000","last_undegraded":"2024-01-15T18:20:25.987945+0000","last_fullsized":"2024-01-15T18:20:25.987945+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:25.987848+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:25.987848+0000","last_clean_scrub_stamp":"2024-01-15T18:20:25.987848+0000","objects_scrubbed":17,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:55.244578+0000","scrub_duration":0.153668372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.327294+0000","last_change":"2024-01-15T18:20:09.327345+0000","last_active":"2024-01-15T18:20:09.327294+0000","last_peered":"2024-01-15T18:20:09.327294+0000","last_clean":"2024-01-15T18:20:09.327294+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:20:09.327294+0000","last_undegraded":"2024-01-15T18:20:09.327294+0000","last_fullsized":"2024-01-15T18:20:09.327294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:09.327108+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:09.327108+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.327108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:36.860290+0000","scrub_duration":0.003411468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'73","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.265444+0000","last_change":"2024-01-15T18:20:19.265562+0000","last_active":"2024-01-15T18:20:19.265444+0000","last_peered":"2024-01-15T18:20:19.265444+0000","last_clean":"2024-01-15T18:20:19.265444+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:20:19.265444+0000","last_undegraded":"2024-01-15T18:20:19.265444+0000","last_fullsized":"2024-01-15T18:20:19.265444+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:20:19.265264+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:20:19.265264+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.265264+0000","objects_scrubbed":24,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:44.988416+0000","scrub_duration":0.25818551200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.459148+0000","last_change":"2024-01-15T18:20:04.459201+0000","last_active":"2024-01-15T18:20:04.459148+0000","last_peered":"2024-01-15T18:20:04.459148+0000","last_clean":"2024-01-15T18:20:04.459148+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:20:04.459148+0000","last_undegraded":"2024-01-15T18:20:04.459148+0000","last_fullsized":"2024-01-15T18:20:04.459148+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:04.458942+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:04.458942+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.458942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:42.315332+0000","scrub_duration":0.0039832870000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'71","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:00.901734+0000","last_change":"2024-01-15T18:20:00.901779+0000","last_active":"2024-01-15T18:20:00.901734+0000","last_peered":"2024-01-15T18:20:00.901734+0000","last_clean":"2024-01-15T18:20:00.901734+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:20:00.901734+0000","last_undegraded":"2024-01-15T18:20:00.901734+0000","last_fullsized":"2024-01-15T18:20:00.901734+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'71","last_scrub_stamp":"2024-01-15T18:20:00.901655+0000","last_deep_scrub":"116'71","last_deep_scrub_stamp":"2024-01-15T18:20:00.901655+0000","last_clean_scrub_stamp":"2024-01-15T18:20:00.901655+0000","objects_scrubbed":26,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:12.623865+0000","scrub_duration":0.173092522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.168864+0000","last_change":"2024-01-15T18:20:01.168915+0000","last_active":"2024-01-15T18:20:01.168864+0000","last_peered":"2024-01-15T18:20:01.168864+0000","last_clean":"2024-01-15T18:20:01.168864+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:20:01.168864+0000","last_undegraded":"2024-01-15T18:20:01.168864+0000","last_fullsized":"2024-01-15T18:20:01.168864+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:01.168637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:01.168637+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.168637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:58:03.821391+0000","scrub_duration":0.0034424299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'59","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.355545+0000","last_change":"2024-01-15T18:20:34.355645+0000","last_active":"2024-01-15T18:20:34.355545+0000","last_peered":"2024-01-15T18:20:34.355545+0000","last_clean":"2024-01-15T18:20:34.355545+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:20:34.355545+0000","last_undegraded":"2024-01-15T18:20:34.355545+0000","last_fullsized":"2024-01-15T18:20:34.355545+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:34.355384+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:34.355384+0000","last_clean_scrub_stamp":"2024-01-15T18:20:34.355384+0000","objects_scrubbed":16,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:24.017915+0000","scrub_duration":0.200285095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.769090+0000","last_change":"2024-01-15T18:20:02.769178+0000","last_active":"2024-01-15T18:20:02.769090+0000","last_peered":"2024-01-15T18:20:02.769090+0000","last_clean":"2024-01-15T18:20:02.769090+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:20:02.769090+0000","last_undegraded":"2024-01-15T18:20:02.769090+0000","last_fullsized":"2024-01-15T18:20:02.769090+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.768819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.768819+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.768819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:01.570006+0000","scrub_duration":0.0044113540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'50","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:31.336324+0000","last_change":"2024-01-15T18:20:31.336395+0000","last_active":"2024-01-15T18:20:31.336324+0000","last_peered":"2024-01-15T18:20:31.336324+0000","last_clean":"2024-01-15T18:20:31.336324+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:20:31.336324+0000","last_undegraded":"2024-01-15T18:20:31.336324+0000","last_fullsized":"2024-01-15T18:20:31.336324+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'50","last_scrub_stamp":"2024-01-15T18:20:31.336136+0000","last_deep_scrub":"116'50","last_deep_scrub_stamp":"2024-01-15T18:20:31.336136+0000","last_clean_scrub_stamp":"2024-01-15T18:20:31.336136+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:24.182918+0000","scrub_duration":0.091319631999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.364660+0000","last_change":"2024-01-15T18:20:19.364715+0000","last_active":"2024-01-15T18:20:19.364660+0000","last_peered":"2024-01-15T18:20:19.364660+0000","last_clean":"2024-01-15T18:20:19.364660+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:20:19.364660+0000","last_undegraded":"2024-01-15T18:20:19.364660+0000","last_fullsized":"2024-01-15T18:20:19.364660+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'1","last_scrub_stamp":"2024-01-15T18:20:19.364443+0000","last_deep_scrub":"116'1","last_deep_scrub_stamp":"2024-01-15T18:20:19.364443+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.364443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:20:46.747691+0000","scrub_duration":0.003611015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'58","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:27.163881+0000","last_change":"2024-01-15T18:20:27.163950+0000","last_active":"2024-01-15T18:20:27.163881+0000","last_peered":"2024-01-15T18:20:27.163881+0000","last_clean":"2024-01-15T18:20:27.163881+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:20:27.163881+0000","last_undegraded":"2024-01-15T18:20:27.163881+0000","last_fullsized":"2024-01-15T18:20:27.163881+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:27.163745+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:27.163745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:27.163745+0000","objects_scrubbed":18,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:49.700312+0000","scrub_duration":0.20644792100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.421790+0000","last_change":"2024-01-15T18:20:05.421839+0000","last_active":"2024-01-15T18:20:05.421790+0000","last_peered":"2024-01-15T18:20:05.421790+0000","last_clean":"2024-01-15T18:20:05.421790+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:20:05.421790+0000","last_undegraded":"2024-01-15T18:20:05.421790+0000","last_fullsized":"2024-01-15T18:20:05.421790+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:05.421607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:05.421607+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.421607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:33.730233+0000","scrub_duration":0.0039450379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'63","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:10.661978+0000","last_change":"2024-01-15T18:20:10.662064+0000","last_active":"2024-01-15T18:20:10.661978+0000","last_peered":"2024-01-15T18:20:10.661978+0000","last_clean":"2024-01-15T18:20:10.661978+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:20:10.661978+0000","last_undegraded":"2024-01-15T18:20:10.661978+0000","last_fullsized":"2024-01-15T18:20:10.661978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:10.661798+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:10.661798+0000","last_clean_scrub_stamp":"2024-01-15T18:20:10.661798+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:24.562142+0000","scrub_duration":0.234248187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'15","reported_seq":40,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.785377+0000","last_change":"2024-01-15T18:20:03.785437+0000","last_active":"2024-01-15T18:20:03.785377+0000","last_peered":"2024-01-15T18:20:03.785377+0000","last_clean":"2024-01-15T18:20:03.785377+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:20:03.785377+0000","last_undegraded":"2024-01-15T18:20:03.785377+0000","last_fullsized":"2024-01-15T18:20:03.785377+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'15","last_scrub_stamp":"2024-01-15T18:20:03.785184+0000","last_deep_scrub":"116'15","last_deep_scrub_stamp":"2024-01-15T18:20:03.785184+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.785184+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:03:53.983142+0000","scrub_duration":0.0088990630000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:24.461850+0000","last_change":"2024-01-15T18:20:24.461923+0000","last_active":"2024-01-15T18:20:24.461850+0000","last_peered":"2024-01-15T18:20:24.461850+0000","last_clean":"2024-01-15T18:20:24.461850+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:20:24.461850+0000","last_undegraded":"2024-01-15T18:20:24.461850+0000","last_fullsized":"2024-01-15T18:20:24.461850+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:24.461726+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:24.461726+0000","last_clean_scrub_stamp":"2024-01-15T18:20:24.461726+0000","objects_scrubbed":20,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:54.635793+0000","scrub_duration":0.25293763800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'193","reported_seq":218,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.790076+0000","last_change":"2024-01-15T18:20:01.790145+0000","last_active":"2024-01-15T18:20:01.790076+0000","last_peered":"2024-01-15T18:20:01.790076+0000","last_clean":"2024-01-15T18:20:01.790076+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:20:01.790076+0000","last_undegraded":"2024-01-15T18:20:01.790076+0000","last_fullsized":"2024-01-15T18:20:01.790076+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'193","last_scrub_stamp":"2024-01-15T18:20:01.789874+0000","last_deep_scrub":"116'193","last_deep_scrub_stamp":"2024-01-15T18:20:01.789874+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.789874+0000","objects_scrubbed":4,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:22.469663+0000","scrub_duration":0.01365885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644263,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":190,"num_write_kb":721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'57","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.643433+0000","last_change":"2024-01-15T18:20:08.643515+0000","last_active":"2024-01-15T18:20:08.643433+0000","last_peered":"2024-01-15T18:20:08.643433+0000","last_clean":"2024-01-15T18:20:08.643433+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:20:08.643433+0000","last_undegraded":"2024-01-15T18:20:08.643433+0000","last_fullsized":"2024-01-15T18:20:08.643433+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:08.643249+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:08.643249+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.643249+0000","objects_scrubbed":21,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:08.158988+0000","scrub_duration":0.22645187899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.780476+0000","last_change":"2024-01-15T18:20:03.780574+0000","last_active":"2024-01-15T18:20:03.780476+0000","last_peered":"2024-01-15T18:20:03.780476+0000","last_clean":"2024-01-15T18:20:03.780476+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:20:03.780476+0000","last_undegraded":"2024-01-15T18:20:03.780476+0000","last_fullsized":"2024-01-15T18:20:03.780476+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:03.780315+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:03.780315+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.780315+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:45.825709+0000","scrub_duration":0.0083488959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'71","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:15.624139+0000","last_change":"2024-01-15T18:20:15.624238+0000","last_active":"2024-01-15T18:20:15.624139+0000","last_peered":"2024-01-15T18:20:15.624139+0000","last_clean":"2024-01-15T18:20:15.624139+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:20:15.624139+0000","last_undegraded":"2024-01-15T18:20:15.624139+0000","last_fullsized":"2024-01-15T18:20:15.624139+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'71","last_scrub_stamp":"2024-01-15T18:20:15.623953+0000","last_deep_scrub":"116'71","last_deep_scrub_stamp":"2024-01-15T18:20:15.623953+0000","last_clean_scrub_stamp":"2024-01-15T18:20:15.623953+0000","objects_scrubbed":21,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:43.042056+0000","scrub_duration":0.25260574800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.908262+0000","last_change":"2024-01-15T18:20:04.908358+0000","last_active":"2024-01-15T18:20:04.908262+0000","last_peered":"2024-01-15T18:20:04.908262+0000","last_clean":"2024-01-15T18:20:04.908262+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:20:04.908262+0000","last_undegraded":"2024-01-15T18:20:04.908262+0000","last_fullsized":"2024-01-15T18:20:04.908262+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:04.908067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:04.908067+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.908067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:57.372868+0000","scrub_duration":0.0042479759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'68","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.679799+0000","last_change":"2024-01-15T18:20:09.679858+0000","last_active":"2024-01-15T18:20:09.679799+0000","last_peered":"2024-01-15T18:20:09.679799+0000","last_clean":"2024-01-15T18:20:09.679799+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:20:09.679799+0000","last_undegraded":"2024-01-15T18:20:09.679799+0000","last_fullsized":"2024-01-15T18:20:09.679799+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:09.679609+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:09.679609+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.679609+0000","objects_scrubbed":25,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:18.637167+0000","scrub_duration":0.26166856799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"116'12","reported_seq":37,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:00.222842+0000","last_change":"2024-01-15T18:20:00.222891+0000","last_active":"2024-01-15T18:20:00.222842+0000","last_peered":"2024-01-15T18:20:00.222842+0000","last_clean":"2024-01-15T18:20:00.222842+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:20:00.222842+0000","last_undegraded":"2024-01-15T18:20:00.222842+0000","last_fullsized":"2024-01-15T18:20:00.222842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:20:00.222745+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:20:00.222745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:00.222745+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:51.445709+0000","scrub_duration":0.010058510999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'34","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:28.442016+0000","last_change":"2024-01-15T18:20:28.442084+0000","last_active":"2024-01-15T18:20:28.442016+0000","last_peered":"2024-01-15T18:20:28.442016+0000","last_clean":"2024-01-15T18:20:28.442016+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:20:28.442016+0000","last_undegraded":"2024-01-15T18:20:28.442016+0000","last_fullsized":"2024-01-15T18:20:28.442016+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'34","last_scrub_stamp":"2024-01-15T18:20:28.441883+0000","last_deep_scrub":"116'34","last_deep_scrub_stamp":"2024-01-15T18:20:28.441883+0000","last_clean_scrub_stamp":"2024-01-15T18:20:28.441883+0000","objects_scrubbed":10,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:23.267358+0000","scrub_duration":0.104048134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"116'2","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.745018+0000","last_change":"2024-01-15T18:20:06.745073+0000","last_active":"2024-01-15T18:20:06.745018+0000","last_peered":"2024-01-15T18:20:06.745018+0000","last_clean":"2024-01-15T18:20:06.745018+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:20:06.745018+0000","last_undegraded":"2024-01-15T18:20:06.745018+0000","last_fullsized":"2024-01-15T18:20:06.745018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-01-15T18:20:06.744839+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-01-15T18:20:06.744839+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.744839+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:58:35.444023+0000","scrub_duration":0.0075306430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'53","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:24.045496+0000","last_change":"2024-01-15T18:20:24.045646+0000","last_active":"2024-01-15T18:20:24.045496+0000","last_peered":"2024-01-15T18:20:24.045496+0000","last_clean":"2024-01-15T18:20:24.045496+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:20:24.045496+0000","last_undegraded":"2024-01-15T18:20:24.045496+0000","last_fullsized":"2024-01-15T18:20:24.045496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'53","last_scrub_stamp":"2024-01-15T18:20:24.045217+0000","last_deep_scrub":"116'53","last_deep_scrub_stamp":"2024-01-15T18:20:24.045217+0000","last_clean_scrub_stamp":"2024-01-15T18:20:24.045217+0000","objects_scrubbed":17,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:22.250266+0000","scrub_duration":0.17771706600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.362044+0000","last_change":"2024-01-15T18:20:09.362155+0000","last_active":"2024-01-15T18:20:09.362044+0000","last_peered":"2024-01-15T18:20:09.362044+0000","last_clean":"2024-01-15T18:20:09.362044+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:20:09.362044+0000","last_undegraded":"2024-01-15T18:20:09.362044+0000","last_fullsized":"2024-01-15T18:20:09.362044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:09.361834+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:09.361834+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.361834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:42.293371+0000","scrub_duration":0.0070512709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'59","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.048473+0000","last_change":"2024-01-15T18:20:06.048581+0000","last_active":"2024-01-15T18:20:06.048473+0000","last_peered":"2024-01-15T18:20:06.048473+0000","last_clean":"2024-01-15T18:20:06.048473+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:20:06.048473+0000","last_undegraded":"2024-01-15T18:20:06.048473+0000","last_fullsized":"2024-01-15T18:20:06.048473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:06.048300+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:06.048300+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.048300+0000","objects_scrubbed":18,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:34.356925+0000","scrub_duration":0.18916992199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"116'2","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.904337+0000","last_change":"2024-01-15T18:20:07.904387+0000","last_active":"2024-01-15T18:20:07.904337+0000","last_peered":"2024-01-15T18:20:07.904337+0000","last_clean":"2024-01-15T18:20:07.904337+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:20:07.904337+0000","last_undegraded":"2024-01-15T18:20:07.904337+0000","last_fullsized":"2024-01-15T18:20:07.904337+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-01-15T18:20:07.904106+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-01-15T18:20:07.904106+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.904106+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:45.760496+0000","scrub_duration":0.0084473900000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'63","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.038371+0000","last_change":"2024-01-15T18:20:23.038437+0000","last_active":"2024-01-15T18:20:23.038371+0000","last_peered":"2024-01-15T18:20:23.038371+0000","last_clean":"2024-01-15T18:20:23.038371+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:20:23.038371+0000","last_undegraded":"2024-01-15T18:20:23.038371+0000","last_fullsized":"2024-01-15T18:20:23.038371+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:23.038283+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:23.038283+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.038283+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:28:55.265617+0000","scrub_duration":0.25342030799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.399525+0000","last_change":"2024-01-15T18:20:06.399574+0000","last_active":"2024-01-15T18:20:06.399525+0000","last_peered":"2024-01-15T18:20:06.399525+0000","last_clean":"2024-01-15T18:20:06.399525+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:20:06.399525+0000","last_undegraded":"2024-01-15T18:20:06.399525+0000","last_fullsized":"2024-01-15T18:20:06.399525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:06.399262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:06.399262+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.399262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:44.431017+0000","scrub_duration":0.0039161960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'64","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.204795+0000","last_change":"2024-01-15T18:20:07.204980+0000","last_active":"2024-01-15T18:20:07.204795+0000","last_peered":"2024-01-15T18:20:07.204795+0000","last_clean":"2024-01-15T18:20:07.204795+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:20:07.204795+0000","last_undegraded":"2024-01-15T18:20:07.204795+0000","last_fullsized":"2024-01-15T18:20:07.204795+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:07.204595+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:07.204595+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.204595+0000","objects_scrubbed":30,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:19:38.823285+0000","scrub_duration":0.36587318699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.818977+0000","last_change":"2024-01-15T18:20:02.819041+0000","last_active":"2024-01-15T18:20:02.818977+0000","last_peered":"2024-01-15T18:20:02.818977+0000","last_clean":"2024-01-15T18:20:02.818977+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:20:02.818977+0000","last_undegraded":"2024-01-15T18:20:02.818977+0000","last_fullsized":"2024-01-15T18:20:02.818977+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:02.818798+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:02.818798+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.818798+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:08:10.712736+0000","scrub_duration":0.0072655630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'62","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:33.517403+0000","last_change":"2024-01-15T18:20:33.517489+0000","last_active":"2024-01-15T18:20:33.517403+0000","last_peered":"2024-01-15T18:20:33.517403+0000","last_clean":"2024-01-15T18:20:33.517403+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:20:33.517403+0000","last_undegraded":"2024-01-15T18:20:33.517403+0000","last_fullsized":"2024-01-15T18:20:33.517403+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'62","last_scrub_stamp":"2024-01-15T18:20:33.517228+0000","last_deep_scrub":"116'62","last_deep_scrub_stamp":"2024-01-15T18:20:33.517228+0000","last_clean_scrub_stamp":"2024-01-15T18:20:33.517228+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:49.734656+0000","scrub_duration":0.204264011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.258303+0000","last_change":"2024-01-15T18:20:02.258363+0000","last_active":"2024-01-15T18:20:02.258303+0000","last_peered":"2024-01-15T18:20:02.258303+0000","last_clean":"2024-01-15T18:20:02.258303+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:20:02.258303+0000","last_undegraded":"2024-01-15T18:20:02.258303+0000","last_fullsized":"2024-01-15T18:20:02.258303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.258117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.258117+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.258117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:31.738131+0000","scrub_duration":0.003674961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'61","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:31.013715+0000","last_change":"2024-01-15T18:20:31.013895+0000","last_active":"2024-01-15T18:20:31.013715+0000","last_peered":"2024-01-15T18:20:31.013715+0000","last_clean":"2024-01-15T18:20:31.013715+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:20:31.013715+0000","last_undegraded":"2024-01-15T18:20:31.013715+0000","last_fullsized":"2024-01-15T18:20:31.013715+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'61","last_scrub_stamp":"2024-01-15T18:20:31.013515+0000","last_deep_scrub":"116'61","last_deep_scrub_stamp":"2024-01-15T18:20:31.013515+0000","last_clean_scrub_stamp":"2024-01-15T18:20:31.013515+0000","objects_scrubbed":17,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:28:33.562019+0000","scrub_duration":0.19102684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.795122+0000","last_change":"2024-01-15T18:19:58.795153+0000","last_active":"2024-01-15T18:19:58.795122+0000","last_peered":"2024-01-15T18:19:58.795122+0000","last_clean":"2024-01-15T18:19:58.795122+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:19:58.795122+0000","last_undegraded":"2024-01-15T18:19:58.795122+0000","last_fullsized":"2024-01-15T18:19:58.795122+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.795017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.531385+0000","scrub_duration":0.0028610760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'59","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.536565+0000","last_change":"2024-01-15T18:20:18.536640+0000","last_active":"2024-01-15T18:20:18.536565+0000","last_peered":"2024-01-15T18:20:18.536565+0000","last_clean":"2024-01-15T18:20:18.536565+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:20:18.536565+0000","last_undegraded":"2024-01-15T18:20:18.536565+0000","last_fullsized":"2024-01-15T18:20:18.536565+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:18.536416+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:18.536416+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.536416+0000","objects_scrubbed":17,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:58.867375+0000","scrub_duration":0.181046237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.721429+0000","last_change":"2024-01-15T18:20:05.721465+0000","last_active":"2024-01-15T18:20:05.721429+0000","last_peered":"2024-01-15T18:20:05.721429+0000","last_clean":"2024-01-15T18:20:05.721429+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:20:05.721429+0000","last_undegraded":"2024-01-15T18:20:05.721429+0000","last_fullsized":"2024-01-15T18:20:05.721429+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:05.721271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:05.721271+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.721271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:07.002373+0000","scrub_duration":0.003099302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'49","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.022662+0000","last_change":"2024-01-15T18:20:23.022742+0000","last_active":"2024-01-15T18:20:23.022662+0000","last_peered":"2024-01-15T18:20:23.022662+0000","last_clean":"2024-01-15T18:20:23.022662+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:20:23.022662+0000","last_undegraded":"2024-01-15T18:20:23.022662+0000","last_fullsized":"2024-01-15T18:20:23.022662+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:23.022519+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:23.022519+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.022519+0000","objects_scrubbed":13,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:31.980076+0000","scrub_duration":0.16129085500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.509693+0000","last_change":"2024-01-15T18:20:03.509754+0000","last_active":"2024-01-15T18:20:03.509693+0000","last_peered":"2024-01-15T18:20:03.509693+0000","last_clean":"2024-01-15T18:20:03.509693+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:20:03.509693+0000","last_undegraded":"2024-01-15T18:20:03.509693+0000","last_fullsized":"2024-01-15T18:20:03.509693+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:03.509469+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:03.509469+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.509469+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.140490+0000","scrub_duration":0.0069296399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'41","reported_seq":82,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:45.410870+0000","last_change":"2024-01-15T18:15:20.356060+0000","last_active":"2024-01-15T18:19:45.410870+0000","last_peered":"2024-01-15T18:19:45.410870+0000","last_clean":"2024-01-15T18:19:45.410870+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:19:45.410870+0000","last_undegraded":"2024-01-15T18:19:45.410870+0000","last_fullsized":"2024-01-15T18:19:45.410870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:14:15.825359+0000","last_clean_scrub_stamp":"2024-01-15T18:14:15.825359+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'19","reported_seq":44,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.852686+0000","last_change":"2024-01-15T18:20:19.852751+0000","last_active":"2024-01-15T18:20:19.852686+0000","last_peered":"2024-01-15T18:20:19.852686+0000","last_clean":"2024-01-15T18:20:19.852686+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:20:19.852686+0000","last_undegraded":"2024-01-15T18:20:19.852686+0000","last_fullsized":"2024-01-15T18:20:19.852686+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'19","last_scrub_stamp":"2024-01-15T18:20:19.852492+0000","last_deep_scrub":"116'19","last_deep_scrub_stamp":"2024-01-15T18:20:19.852492+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.852492+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:31.239558+0000","scrub_duration":0.0039531749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'66","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.918788+0000","last_change":"2024-01-15T18:19:57.918788+0000","last_active":"2024-01-15T18:19:57.918788+0000","last_peered":"2024-01-15T18:19:57.918788+0000","last_clean":"2024-01-15T18:19:57.918788+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:57.918788+0000","last_undegraded":"2024-01-15T18:19:57.918788+0000","last_fullsized":"2024-01-15T18:19:57.918788+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'66","last_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_deep_scrub":"116'66","last_deep_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.918639+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:38.022160+0000","scrub_duration":0.15638939399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.767210+0000","last_change":"2024-01-15T18:20:04.767210+0000","last_active":"2024-01-15T18:20:04.767210+0000","last_peered":"2024-01-15T18:20:04.767210+0000","last_clean":"2024-01-15T18:20:04.767210+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:20:04.767210+0000","last_undegraded":"2024-01-15T18:20:04.767210+0000","last_fullsized":"2024-01-15T18:20:04.767210+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:04.766974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:04.766974+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.766974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:58.490311+0000","scrub_duration":0.0038318369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'58","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:22.058910+0000","last_change":"2024-01-15T18:20:22.058910+0000","last_active":"2024-01-15T18:20:22.058910+0000","last_peered":"2024-01-15T18:20:22.058910+0000","last_clean":"2024-01-15T18:20:22.058910+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:20:22.058910+0000","last_undegraded":"2024-01-15T18:20:22.058910+0000","last_fullsized":"2024-01-15T18:20:22.058910+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:22.058626+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:22.058626+0000","last_clean_scrub_stamp":"2024-01-15T18:20:22.058626+0000","objects_scrubbed":19,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:21.574364+0000","scrub_duration":0.19533523999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.514236+0000","last_change":"2024-01-15T18:20:01.514236+0000","last_active":"2024-01-15T18:20:01.514236+0000","last_peered":"2024-01-15T18:20:01.514236+0000","last_clean":"2024-01-15T18:20:01.514236+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:20:01.514236+0000","last_undegraded":"2024-01-15T18:20:01.514236+0000","last_fullsized":"2024-01-15T18:20:01.514236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:01.514092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:01.514092+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.514092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:26.265860+0000","scrub_duration":0.0027639349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'60","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.337206+0000","last_change":"2024-01-15T18:20:07.337206+0000","last_active":"2024-01-15T18:20:07.337206+0000","last_peered":"2024-01-15T18:20:07.337206+0000","last_clean":"2024-01-15T18:20:07.337206+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:20:07.337206+0000","last_undegraded":"2024-01-15T18:20:07.337206+0000","last_fullsized":"2024-01-15T18:20:07.337206+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'60","last_scrub_stamp":"2024-01-15T18:20:07.336876+0000","last_deep_scrub":"116'60","last_deep_scrub_stamp":"2024-01-15T18:20:07.336876+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.336876+0000","objects_scrubbed":16,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:52.572228+0000","scrub_duration":0.19744029199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.373842+0000","last_change":"2024-01-15T18:20:08.373842+0000","last_active":"2024-01-15T18:20:08.373842+0000","last_peered":"2024-01-15T18:20:08.373842+0000","last_clean":"2024-01-15T18:20:08.373842+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:20:08.373842+0000","last_undegraded":"2024-01-15T18:20:08.373842+0000","last_fullsized":"2024-01-15T18:20:08.373842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:08.373617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:08.373617+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.373617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:55.243354+0000","scrub_duration":0.0034628670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'80","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:12.520568+0000","last_change":"2024-01-15T18:20:12.520568+0000","last_active":"2024-01-15T18:20:12.520568+0000","last_peered":"2024-01-15T18:20:12.520568+0000","last_clean":"2024-01-15T18:20:12.520568+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:20:12.520568+0000","last_undegraded":"2024-01-15T18:20:12.520568+0000","last_fullsized":"2024-01-15T18:20:12.520568+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'80","last_scrub_stamp":"2024-01-15T18:20:12.520296+0000","last_deep_scrub":"116'80","last_deep_scrub_stamp":"2024-01-15T18:20:12.520296+0000","last_clean_scrub_stamp":"2024-01-15T18:20:12.520296+0000","objects_scrubbed":29,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:06.085847+0000","scrub_duration":0.32665812100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.144631+0000","last_change":"2024-01-15T18:20:03.144631+0000","last_active":"2024-01-15T18:20:03.144631+0000","last_peered":"2024-01-15T18:20:03.144631+0000","last_clean":"2024-01-15T18:20:03.144631+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:20:03.144631+0000","last_undegraded":"2024-01-15T18:20:03.144631+0000","last_fullsized":"2024-01-15T18:20:03.144631+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:03.144267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:03.144267+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.144267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:45.685511+0000","scrub_duration":0.0035700250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'72","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.219295+0000","last_change":"2024-01-15T18:20:34.219329+0000","last_active":"2024-01-15T18:20:34.219295+0000","last_peered":"2024-01-15T18:20:34.219295+0000","last_clean":"2024-01-15T18:20:34.219295+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:20:34.219295+0000","last_undegraded":"2024-01-15T18:20:34.219295+0000","last_fullsized":"2024-01-15T18:20:34.219295+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:34.219211+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:34.219211+0000","last_clean_scrub_stamp":"2024-01-15T18:20:34.219211+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:14.097860+0000","scrub_duration":0.31754085300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'12","reported_seq":48,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.182868+0000","last_change":"2024-01-15T18:19:58.182868+0000","last_active":"2024-01-15T18:19:58.182868+0000","last_peered":"2024-01-15T18:19:58.182868+0000","last_clean":"2024-01-15T18:19:58.182868+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:19:58.182868+0000","last_undegraded":"2024-01-15T18:19:58.182868+0000","last_fullsized":"2024-01-15T18:19:58.182868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.182737+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:33:12.765302+0000","scrub_duration":0.0054820609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'59","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:12.977682+0000","last_change":"2024-01-15T18:20:12.977682+0000","last_active":"2024-01-15T18:20:12.977682+0000","last_peered":"2024-01-15T18:20:12.977682+0000","last_clean":"2024-01-15T18:20:12.977682+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:20:12.977682+0000","last_undegraded":"2024-01-15T18:20:12.977682+0000","last_fullsized":"2024-01-15T18:20:12.977682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:12.977304+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:12.977304+0000","last_clean_scrub_stamp":"2024-01-15T18:20:12.977304+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:54.139872+0000","scrub_duration":0.23080210500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:10.753379+0000","last_change":"2024-01-15T18:20:10.753379+0000","last_active":"2024-01-15T18:20:10.753379+0000","last_peered":"2024-01-15T18:20:10.753379+0000","last_clean":"2024-01-15T18:20:10.753379+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:20:10.753379+0000","last_undegraded":"2024-01-15T18:20:10.753379+0000","last_fullsized":"2024-01-15T18:20:10.753379+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:10.753091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:10.753091+0000","last_clean_scrub_stamp":"2024-01-15T18:20:10.753091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:13.671420+0000","scrub_duration":0.0040498490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'44","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.536809+0000","last_change":"2024-01-15T18:20:11.536924+0000","last_active":"2024-01-15T18:20:11.536809+0000","last_peered":"2024-01-15T18:20:11.536809+0000","last_clean":"2024-01-15T18:20:11.536809+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:20:11.536809+0000","last_undegraded":"2024-01-15T18:20:11.536809+0000","last_fullsized":"2024-01-15T18:20:11.536809+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'44","last_scrub_stamp":"2024-01-15T18:20:11.536616+0000","last_deep_scrub":"116'44","last_deep_scrub_stamp":"2024-01-15T18:20:11.536616+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.536616+0000","objects_scrubbed":14,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:20.371492+0000","scrub_duration":0.152340903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.388151+0000","last_change":"2024-01-15T18:20:02.388199+0000","last_active":"2024-01-15T18:20:02.388151+0000","last_peered":"2024-01-15T18:20:02.388151+0000","last_clean":"2024-01-15T18:20:02.388151+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:20:02.388151+0000","last_undegraded":"2024-01-15T18:20:02.388151+0000","last_fullsized":"2024-01-15T18:20:02.388151+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.387965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.387965+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.387965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:11:59.203734+0000","scrub_duration":0.0038625069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"116'119","reported_seq":265,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.974811+0000","last_change":"2024-01-15T18:20:01.974882+0000","last_active":"2024-01-15T18:20:01.974811+0000","last_peered":"2024-01-15T18:20:01.974811+0000","last_clean":"2024-01-15T18:20:01.974811+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:20:01.974811+0000","last_undegraded":"2024-01-15T18:20:01.974811+0000","last_fullsized":"2024-01-15T18:20:01.974811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'119","last_scrub_stamp":"2024-01-15T18:20:01.974618+0000","last_deep_scrub":"116'119","last_deep_scrub_stamp":"2024-01-15T18:20:01.974618+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.974618+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:39.831008+0000","scrub_duration":0.053321348999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3968,"ondisk_log_size":3968,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":646382,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":288,"num_write_kb":762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359872,"data_stored":1292764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286,"ondisk_log_size":286,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3049999999999999,"15min":1.292},"min":{"1min":0.72599999999999998,"5min":0.61599999999999999,"15min":0.54000000000000004},"max":{"1min":1.5429999999999999,"5min":1.8140000000000001,"15min":4.0019999999999998},"last":1.337},{"interface":"front","average":{"1min":1.325,"5min":1.3819999999999999,"15min":1.5009999999999999},"min":{"1min":0.99399999999999999,"5min":0.88400000000000001,"15min":0.69999999999999996},"max":{"1min":1.6319999999999999,"5min":1.8300000000000001,"15min":27.32},"last":1.758}]},{"osd":1,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.24,"15min":1.4490000000000001},"min":{"1min":0.82199999999999995,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.8129999999999999,"5min":2.1000000000000001,"15min":29.167999999999999},"last":1.383},{"interface":"front","average":{"1min":1.258,"5min":1.2290000000000001,"15min":1.2929999999999999},"min":{"1min":0.80300000000000005,"5min":0.70599999999999996,"15min":0.52800000000000002},"max":{"1min":1.889,"5min":2.2349999999999999,"15min":4.2699999999999996},"last":1.089}]},{"osd":2,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3580000000000001,"15min":1.3480000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.56000000000000005},"max":{"1min":1.8759999999999999,"5min":2.194,"15min":4.1920000000000002},"last":1.46},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.254,"15min":1.4399999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.56599999999999995},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":27.358000000000001},"last":1.6060000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3120000000000001,"15min":1.528},"min":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.6890000000000001,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.282},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.468},"min":{"1min":0.99099999999999999,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.734,"5min":2.2749999999999999,"15min":4.3879999999999999},"last":1.573}]},{"osd":4,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.371,"15min":1.49},"min":{"1min":0.87,"5min":0.87,"15min":0.50700000000000001},"max":{"1min":1.8089999999999999,"5min":6.0350000000000001,"15min":27.219000000000001},"last":1.696},{"interface":"front","average":{"1min":1.359,"5min":1.427,"15min":1.6020000000000001},"min":{"1min":0.89000000000000001,"5min":0.79900000000000004,"15min":0.57299999999999995},"max":{"1min":1.823,"5min":3.536,"15min":29.25},"last":1.5369999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.46,"15min":1.554},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.68799999999999994},"max":{"1min":1.8700000000000001,"5min":2.1509999999999998,"15min":11.834},"last":1.238},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5149999999999999,"15min":1.6459999999999999},"min":{"1min":1.169,"5min":0.92900000000000005,"15min":0.60499999999999998},"max":{"1min":2.145,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.8120000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.7729999999999999},"min":{"1min":1.29,"5min":1.1060000000000001,"15min":0.79200000000000004},"max":{"1min":2.4249999999999998,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.861},{"interface":"front","average":{"1min":1.577,"5min":1.5449999999999999,"15min":1.7490000000000001},"min":{"1min":1.234,"5min":1.0289999999999999,"15min":0.57299999999999995},"max":{"1min":2.085,"5min":2.754,"15min":29.106999999999999},"last":1.1779999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542425,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.1919999999999999,"15min":1.2370000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":8.8350000000000009},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1719999999999999,"15min":1.2669999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5089999999999999,"5min":1.78,"15min":9.548},"last":1.514}]},{"osd":1,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2390000000000001,"15min":1.379},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.53300000000000003},"max":{"1min":1.718,"5min":2.1419999999999999,"15min":13.269},"last":1.333},{"interface":"front","average":{"1min":1.278,"5min":1.2889999999999999,"15min":1.3580000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.57799999999999996},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":9.6389999999999993},"last":1.129}]},{"osd":2,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.234,"15min":1.278},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.69799999999999995},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":7.0030000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.357,"15min":1.425},"min":{"1min":0.84999999999999998,"5min":0.81100000000000005,"15min":0.49399999999999999},"max":{"1min":1.7390000000000001,"5min":1.889,"15min":8.9359999999999999},"last":1.0860000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4239999999999999,"15min":1.512},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.68500000000000005},"max":{"1min":1.708,"5min":1.946,"15min":16.855},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.351,"5min":1.3919999999999999,"15min":1.476},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.628},"max":{"1min":1.976,"5min":1.976,"15min":8.0139999999999993},"last":1.0489999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":2.831,"5min":1.637,"15min":1.544},"min":{"1min":0.86099999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":33.161999999999999,"5min":33.161999999999999,"15min":33.161999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.371,"15min":1.4990000000000001},"min":{"1min":0.995,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":10.118},"last":1.379}]},{"osd":5,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3129999999999999,"15min":1.5129999999999999},"min":{"1min":0.90700000000000003,"5min":0.88,"15min":0.67000000000000004},"max":{"1min":1.804,"5min":1.875,"15min":16.309999999999999},"last":1.24},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.482,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.66000000000000003},"max":{"1min":1.8540000000000001,"5min":1.9079999999999999,"15min":10.077},"last":1.1919999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5029999999999999,"15min":1.6220000000000001},"min":{"1min":1.196,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":10.302},"last":1.464},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.526,"15min":1.6479999999999999},"min":{"1min":1.0680000000000001,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":2.0640000000000001,"5min":6.3029999999999999,"15min":8.0519999999999996},"last":1.5509999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542427,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912332,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93459508,"statfs":{"total":96636764160,"available":95702536192,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2549999999999999,"15min":1.365},"min":{"1min":0.80400000000000005,"5min":0.70099999999999996,"15min":0.61099999999999999},"max":{"1min":1.502,"5min":4.1829999999999998,"15min":26.077999999999999},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3759999999999999,"15min":1.4179999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.56399999999999995},"max":{"1min":1.6859999999999999,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.411}]},{"osd":1,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2529999999999999,"15min":1.401},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.53300000000000003},"max":{"1min":1.7230000000000001,"5min":1.923,"15min":26},"last":1.022},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.375,"15min":1.359},"min":{"1min":0.84899999999999998,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.6699999999999999,"5min":4.0869999999999997,"15min":7.5309999999999997},"last":1.496}]},{"osd":2,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3320000000000001,"15min":1.498},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.56699999999999995},"max":{"1min":1.6120000000000001,"5min":1.774,"15min":26.234999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.222,"5min":1.349,"15min":1.5149999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.629},"max":{"1min":1.52,"5min":1.8240000000000001,"15min":17.126999999999999},"last":1.5980000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.335,"15min":1.351},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.67200000000000004},"max":{"1min":1.595,"5min":2.375,"15min":7.6150000000000002},"last":1.6399999999999999},{"interface":"front","average":{"1min":1.462,"5min":1.4850000000000001,"15min":1.6000000000000001},"min":{"1min":1.044,"5min":0.96499999999999997,"15min":0.67900000000000005},"max":{"1min":1.736,"5min":1.855,"15min":26.134},"last":1.452}]},{"osd":4,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.242,"15min":1.3799999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.442},"max":{"1min":1.5589999999999999,"5min":2.286,"15min":9.9979999999999993},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.4359999999999999,"15min":1.6459999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.498},"max":{"1min":1.732,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.3089999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.365,"15min":1.383},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.45900000000000002},"max":{"1min":1.833,"5min":2.6640000000000001,"15min":7.9240000000000004},"last":1.371},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4430000000000001,"15min":1.611},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.49199999999999999},"max":{"1min":1.71,"5min":2.0659999999999998,"15min":26.186},"last":1.6779999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.5760000000000001,"15min":1.7270000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.65700000000000003},"max":{"1min":1.8029999999999999,"5min":2.04,"15min":12.326000000000001},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.538,"15min":1.556},"min":{"1min":0.755,"5min":0.755,"15min":0.60099999999999998},"max":{"1min":1.8109999999999999,"5min":4.3700000000000001,"15min":12.175000000000001},"last":1.7549999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542426,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.226,"15min":1.226},"min":{"1min":0.83699999999999997,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.502,"5min":1.8859999999999999,"15min":13.288},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1990000000000001,"15min":1.262},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.34699999999999998},"max":{"1min":1.653,"5min":1.7030000000000001,"15min":9.2409999999999997},"last":0.97199999999999998}]},{"osd":1,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2470000000000001,"15min":1.357},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":1.78,"15min":28.663},"last":1.641},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2709999999999999,"15min":1.3899999999999999},"min":{"1min":0.92600000000000005,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.681,"5min":1.7789999999999999,"15min":13.614000000000001},"last":0.88500000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.3029999999999999,"15min":1.4179999999999999},"min":{"1min":0.83399999999999996,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.673,"5min":1.923,"15min":16.023},"last":1.137},{"interface":"front","average":{"1min":1.407,"5min":1.3819999999999999,"15min":1.4710000000000001},"min":{"1min":1.196,"5min":0.72199999999999998,"15min":0.52500000000000002},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":18.858000000000001},"last":1.397}]},{"osd":3,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2729999999999999,"15min":1.331},"min":{"1min":0.84699999999999998,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.6930000000000001,"5min":1.9690000000000001,"15min":5.0860000000000003},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.4470000000000001,"15min":1.5329999999999999},"min":{"1min":1.0600000000000001,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.792,"5min":2.1400000000000001,"15min":18.027000000000001},"last":1.321}]},{"osd":5,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4339999999999999,"15min":1.4099999999999999},"min":{"1min":0.83999999999999997,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.903,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.4279999999999999},"min":{"1min":0.94999999999999996,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":2.0049999999999999,"5min":2.0960000000000001,"15min":8.5660000000000007},"last":1.603}]},{"osd":6,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.425,"15min":1.5409999999999999},"min":{"1min":1.1359999999999999,"5min":0.92900000000000005,"15min":0.495},"max":{"1min":1.865,"5min":2.0110000000000001,"15min":16.103999999999999},"last":1.859},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5249999999999999,"15min":1.6240000000000001},"min":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.9350000000000001,"5min":1.966,"15min":12.936999999999999},"last":1.5860000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.655},"min":{"1min":1.2210000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.8540000000000001,"5min":2.4689999999999999,"15min":23.390000000000001},"last":1.9119999999999999},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.571,"15min":1.7050000000000001},"min":{"1min":1.25,"5min":1.0900000000000001,"15min":0.62},"max":{"1min":1.9690000000000001,"5min":2.1019999999999999,"15min":16.763000000000002},"last":1.768}]}]},{"osd":3,"up_from":14,"seq":60129542427,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462032,"kb_used_data":1386504,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909808,"statfs":{"total":96636764160,"available":95139643392,"internally_reserved":0,"allocated":1419780096,"data_stored":1418951612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.0229999999999999,"15min":1.133},"min":{"1min":0.56599999999999995,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":9.0299999999999994,"5min":9.0299999999999994,"15min":36.408000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.095,"15min":1.0880000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":1.427,"5min":1.986,"15min":5.0190000000000001},"last":1.4610000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.97099999999999997,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":1.2889999999999999,"5min":1.837,"15min":5.226},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1919999999999999,"15min":1.1910000000000001},"min":{"1min":0.48899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7869999999999999,"5min":2.0609999999999999,"15min":2.5790000000000002},"last":1.1180000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.248,"15min":1.3360000000000001},"min":{"1min":0.56599999999999995,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.6850000000000001,"5min":2.0150000000000001,"15min":14.025},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.343,"15min":1.337},"min":{"1min":0.58199999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.845,"5min":2.403,"15min":3.6240000000000001},"last":1.375}]},{"osd":4,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.403,"15min":1.448},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.59999999999999998},"max":{"1min":10.542,"5min":10.542,"15min":10.542},"last":1.595},{"interface":"front","average":{"1min":1.306,"5min":1.454,"15min":1.5920000000000001},"min":{"1min":0.83799999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.8100000000000001,"5min":4.8570000000000002,"15min":15.769},"last":1.1899999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.93100000000000005,"5min":0.82399999999999995,"15min":0.69299999999999995},"max":{"1min":2.1629999999999998,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.5489999999999999},{"interface":"front","average":{"1min":3.2229999999999999,"5min":1.8460000000000001,"15min":1.897},"min":{"1min":0.88700000000000001,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":1.27}]},{"osd":6,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.6120000000000001,"15min":1.7729999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.51400000000000001},"max":{"1min":10.59,"5min":10.59,"15min":32.259999999999998},"last":1.677},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5820000000000001,"15min":1.6859999999999999},"min":{"1min":0.91000000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":6.8639999999999999},"last":1.427}]},{"osd":7,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.659,"15min":1.903},"min":{"1min":1.1970000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":1.496},{"interface":"front","average":{"1min":1.839,"5min":1.788,"15min":1.825},"min":{"1min":0.93500000000000005,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":10.625,"5min":16.734000000000002,"15min":17.622},"last":1.3300000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542426,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.99399999999999999,"15min":1.0089999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":1.4510000000000001,"5min":3.1589999999999998,"15min":10.256},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.998,"5min":1.1140000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.33700000000000002},"max":{"1min":1.653,"5min":2.1110000000000002,"15min":10.319000000000001},"last":1.25}]},{"osd":1,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0029999999999999,"15min":1.0269999999999999},"min":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.38600000000000001},"max":{"1min":1.427,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.032,"5min":1.159,"15min":1.2290000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.35299999999999998},"max":{"1min":1.7,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.6830000000000001}]},{"osd":3,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3,"15min":1.3879999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.44400000000000001},"max":{"1min":1.5349999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.214,"15min":1.29},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.376},"max":{"1min":1.5680000000000001,"5min":2.7589999999999999,"15min":14.946},"last":1.1930000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.5389999999999999,"15min":1.504},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.66100000000000003},"max":{"1min":1.8009999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.496},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.446,"15min":1.5700000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.65300000000000002},"max":{"1min":1.6830000000000001,"5min":2.867,"15min":10.279999999999999},"last":1.8260000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.45,"15min":1.556},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.63500000000000001},"max":{"1min":2.1000000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.405},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.599,"15min":1.5940000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.71699999999999997},"max":{"1min":1.8799999999999999,"5min":10.833,"15min":10.833},"last":1.5600000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.5009999999999999,"15min":1.5980000000000001},"min":{"1min":0.90400000000000003,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.8620000000000001,"5min":2.984,"15min":10.185},"last":1.9490000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.6399999999999999,"15min":1.7270000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.85899999999999999},"max":{"1min":2.194,"5min":10.917,"15min":22.382000000000001},"last":1.6379999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5880000000000001,"15min":1.8260000000000001},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":0.84299999999999997},"max":{"1min":1.8879999999999999,"5min":3.052,"15min":19.986000000000001},"last":2.0720000000000001},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.651,"15min":1.7729999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.877},"max":{"1min":2.2799999999999998,"5min":3.069,"15min":11.893000000000001},"last":1.746}]}]},{"osd":1,"up_from":14,"seq":60129542426,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141912,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93229928,"statfs":{"total":96636764160,"available":95467446272,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.95999999999999996,"15min":1.0960000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.36199999999999999},"max":{"1min":1.5669999999999999,"5min":3.4510000000000001,"15min":19.218},"last":0.749},{"interface":"front","average":{"1min":0.998,"5min":1.0780000000000001,"15min":1.204},"min":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.825,"5min":1.825,"15min":19.823},"last":0.85599999999999998}]},{"osd":2,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96899999999999997,"15min":1.0800000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.31900000000000001},"max":{"1min":1.603,"5min":1.776,"15min":19.02},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.21,"15min":1.385},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.41599999999999998},"max":{"1min":1.5,"5min":1.889,"15min":29.251000000000001},"last":1.258}]},{"osd":3,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3109999999999999,"15min":1.494},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.40899999999999997},"max":{"1min":1.5169999999999999,"5min":2.036,"15min":27.021000000000001},"last":1.123},{"interface":"front","average":{"1min":1.163,"5min":1.169,"15min":1.3700000000000001},"min":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.859,"5min":1.859,"15min":31.218},"last":0.996}]},{"osd":4,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4630000000000001,"15min":1.6850000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.70599999999999996},"max":{"1min":1.8360000000000001,"5min":1.9650000000000001,"15min":27.059999999999999},"last":1.367},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.409,"15min":1.6100000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.56499999999999995},"max":{"1min":1.97,"5min":1.986,"15min":31.433},"last":1.5549999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4410000000000001,"15min":1.583},"min":{"1min":1.05,"5min":1.05,"15min":0.68500000000000005},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":20.012},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.383,"15min":1.4730000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.69299999999999995},"max":{"1min":1.829,"5min":1.9410000000000001,"15min":3.9910000000000001},"last":1.512}]},{"osd":6,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.571,"15min":1.768},"min":{"1min":1.03,"5min":1.03,"15min":0.68300000000000005},"max":{"1min":1.9199999999999999,"5min":2.2189999999999999,"15min":29.530000000000001},"last":1.601},{"interface":"front","average":{"1min":1.524,"5min":1.548,"15min":1.7789999999999999},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.61399999999999999},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":31.186},"last":1.2370000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.492,"15min":1.7310000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.63},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":26.393999999999998},"last":1.321},{"interface":"front","average":{"1min":1.554,"5min":1.589,"15min":1.794},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.76100000000000001},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":27.582000000000001},"last":1.6599999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542425,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1782112,"kb_used_data":1706584,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92589728,"statfs":{"total":96636764160,"available":94811881472,"internally_reserved":0,"allocated":1747542016,"data_stored":1746697967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.96299999999999997,"15min":1.087},"min":{"1min":0.5,"5min":0.5,"15min":0.30599999999999999},"max":{"1min":1.385,"5min":5.7560000000000002,"15min":20.437999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.2130000000000001,"15min":1.139},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.32400000000000001},"max":{"1min":1.8089999999999999,"5min":6.3600000000000003,"15min":6.3600000000000003},"last":1.044}]},{"osd":2,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.2250000000000001,"15min":1.296},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.669,"5min":11.06,"15min":20.556000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.141,"5min":1.26,"15min":1.2490000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.39000000000000001},"max":{"1min":1.5800000000000001,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":0.93700000000000006}]},{"osd":3,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.228,"15min":1.163},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.42999999999999999},"max":{"1min":1.9259999999999999,"5min":10.885999999999999,"15min":10.885999999999999},"last":1.198},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3520000000000001,"15min":1.468},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.45100000000000001},"max":{"1min":1.538,"5min":9.3339999999999996,"15min":20.670000000000002},"last":1.0109999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.512,"15min":1.4810000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.1259999999999999,"5min":9.9969999999999999,"15min":9.9969999999999999},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.5429999999999999,"15min":1.4319999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.58799999999999997},"max":{"1min":1.7709999999999999,"5min":17.567,"15min":17.567},"last":1.4390000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.6399999999999999,"15min":1.6339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.69799999999999995},"max":{"1min":2.0049999999999999,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.623},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.6319999999999999,"15min":1.7749999999999999},"min":{"1min":1.0780000000000001,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.7829999999999999,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.341}]},{"osd":6,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.6140000000000001,"15min":1.8220000000000001},"min":{"1min":1.244,"5min":1.1060000000000001,"15min":0.70599999999999996},"max":{"1min":1.7749999999999999,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.508},{"interface":"front","average":{"1min":1.583,"5min":1.657,"15min":1.671},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.62},"max":{"1min":2.258,"5min":6.2270000000000003,"15min":8.3829999999999991},"last":1.3879999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.6990000000000001,"15min":1.7849999999999999},"min":{"1min":0.996,"5min":0.996,"15min":0.56200000000000006},"max":{"1min":2.363,"5min":9.9429999999999996,"15min":24.556999999999999},"last":1.27},{"interface":"front","average":{"1min":1.484,"5min":1.6359999999999999,"15min":1.657},"min":{"1min":1.073,"5min":1.048,"15min":0.64000000000000001},"max":{"1min":2.129,"5min":10.943,"15min":11.470000000000001},"last":1.5580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":644375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":644333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:20:42.646 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-15T18:14:15.825359+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=14, tm_sec=15, tm_wday=0, tm_yday=15, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=15, tm_hour=18, tm_min=19, tm_sec=54, tm_wday=0, tm_yday=15, tm_isdst=0) 2024-01-15T18:20:42.646 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-15T18:21:02.647 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-15T18:21:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.875+0000 7f0cdd800640 1 Processor -- start 2024-01-15T18:21:02.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.875+0000 7f0cdd800640 1 -- start start 2024-01-15T18:21:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.876+0000 7f0cdd800640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f0cd814af80 0x7f0cd806d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.876+0000 7f0cdd800640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0cd80700d0 0x7f0cd80704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:02.879 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.876+0000 7f0cdd800640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd8070a20 con 0x7f0cd806da60 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.876+0000 7f0cdd800640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd80b28f0 con 0x7f0cd80700d0 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.876+0000 7f0cdd800640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd80b1c70 con 0x7f0cd814af80 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0cd80700d0 0x7f0cd80704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f0cd80700d0 0x7f0cd80704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40556/0 (socket says 172.21.15.2:40556) 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 -- 172.21.15.2:0/1652119758 learned_addr learned my addr 172.21.15.2:0/1652119758 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cd7e00640 1 --2- 172.21.15.2:0/1652119758 >> v2:172.21.15.137:3300/0 conn(0x7f0cd814af80 0x7f0cd806d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 -- 172.21.15.2:0/1652119758 >> v2:172.21.15.137:3300/0 conn(0x7f0cd814af80 msgr2=0x7f0cd806d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:02.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 --2- 172.21.15.2:0/1652119758 >> v2:172.21.15.137:3300/0 conn(0x7f0cd814af80 0x7f0cd806d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:02.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 -- 172.21.15.2:0/1652119758 >> v1:172.21.15.137:6789/0 conn(0x7f0cd806da60 legacy=0x7f0cd8161c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:02.881 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.877+0000 7f0cdce00640 1 -- 172.21.15.2:0/1652119758 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80b1f50 con 0x7f0cd80700d0 2024-01-15T18:21:02.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.878+0000 7f0cdce00640 1 --2- 172.21.15.2:0/1652119758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd80700d0 0x7f0cd80704e0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80cfc80 tx=0x7f0cd002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04f0f4ccbb1433d server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:02.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cd7400640 1 -- 172.21.15.2:0/1652119758 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd002c9f0 con 0x7f0cd80700d0 2024-01-15T18:21:02.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cd7400640 1 -- 172.21.15.2:0/1652119758 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd002cb50 con 0x7f0cd80700d0 2024-01-15T18:21:02.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cd7400640 1 -- 172.21.15.2:0/1652119758 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00343f0 con 0x7f0cd80700d0 2024-01-15T18:21:02.882 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cdd800640 1 -- 172.21.15.2:0/1652119758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd80700d0 msgr2=0x7f0cd80704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cdd800640 1 --2- 172.21.15.2:0/1652119758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd80700d0 0x7f0cd80704e0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f0cd80cfc80 tx=0x7f0cd002be50 comp rx=0 tx=0).stop 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cdd800640 1 -- 172.21.15.2:0/1652119758 shutdown_connections 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.879+0000 7f0cdd800640 1 --2- 172.21.15.2:0/1652119758 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd80700d0 0x7f0cd80704e0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.880+0000 7f0cdd800640 1 --2- 172.21.15.2:0/1652119758 >> v2:172.21.15.137:3300/0 conn(0x7f0cd814af80 0x7f0cd806d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.880+0000 7f0cdd800640 1 -- 172.21.15.2:0/1652119758 >> 172.21.15.2:0/1652119758 conn(0x7f0cd80b0840 msgr2=0x7f0cd80b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.880+0000 7f0cdd800640 1 -- 172.21.15.2:0/1652119758 shutdown_connections 2024-01-15T18:21:02.883 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.880+0000 7f0cdd800640 1 -- 172.21.15.2:0/1652119758 wait complete. 2024-01-15T18:21:02.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.881+0000 7f0cdd800640 1 Processor -- start 2024-01-15T18:21:02.884 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.881+0000 7f0cdd800640 1 -- start start 2024-01-15T18:21:02.886 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.882+0000 7f0cdd800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 0x7f0cd8145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.882+0000 7f0cdd800640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd80700d0 0x7f0cd815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.882+0000 7f0cdd800640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd814af80 0x7f0cd80a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.882+0000 7f0cdd800640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80a9c20 con 0x7f0cd806da60 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.882+0000 7f0cdd800640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80a96c0 con 0x7f0cd80700d0 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.882+0000 7f0cdd800640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80a9e70 con 0x7f0cd814af80 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 0x7f0cd8145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 0x7f0cd8145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40562/0 (socket says 172.21.15.2:40562) 2024-01-15T18:21:02.887 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 -- 172.21.15.2:0/3204726576 learned_addr learned my addr 172.21.15.2:0/3204726576 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0ccfe00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd80700d0 0x7f0cd815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 -- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd814af80 msgr2=0x7f0cd80a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cdce00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd814af80 0x7f0cd80a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd814af80 0x7f0cd80a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 -- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd80700d0 msgr2=0x7f0cd815fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd80700d0 0x7f0cd815fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0cd7e00640 1 -- 172.21.15.2:0/3204726576 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0009b30 con 0x7f0cd806da60 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.883+0000 7f0ccfe00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd80700d0 0x7f0cd815fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:21:02.888 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.884+0000 7f0cd7e00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 0x7f0cd8145fd0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800d800 tx=0x7f0cc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:02.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.884+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc800b6c0 con 0x7f0cd806da60 2024-01-15T18:21:02.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.884+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc800b840 con 0x7f0cd806da60 2024-01-15T18:21:02.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.884+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd80aa150 con 0x7f0cd806da60 2024-01-15T18:21:02.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.884+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc800bbf0 con 0x7f0cd806da60 2024-01-15T18:21:02.889 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.884+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd80a93b0 con 0x7f0cd806da60 2024-01-15T18:21:02.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.885+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd80aa670 con 0x7f0cd806da60 2024-01-15T18:21:02.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.887+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 44) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f0cc800bd50 con 0x7f0cd806da60 2024-01-15T18:21:02.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.887+0000 7f0cd4c00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cb006c4c0 0x7f0cb006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:02.890 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.888+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f0cc8088da0 con 0x7f0cd806da60 2024-01-15T18:21:02.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.888+0000 7f0ccfe00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cb006c4c0 0x7f0cb006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:02.891 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.889+0000 7f0ccfe00640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cb006c4c0 0x7f0cb006e980 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0cd81fab00 tx=0x7f0cc0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:02.895 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:02.893+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cc80556c0 con 0x7f0cd806da60 2024-01-15T18:21:03.088 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.085+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 --> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0cd80a9fe0 con 0x7f0cb006c4c0 2024-01-15T18:21:03.111 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.109+0000 7f0cd4c00640 1 -- 172.21.15.2:0/3204726576 <== mgr.4106 v2:172.21.15.137:6832/2238446093 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290946 (secure 0 0 0) 0x7f0cd80a9fe0 con 0x7f0cb006c4c0 2024-01-15T18:21:03.116 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:21:03.117 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-01-15T18:21:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cb006c4c0 msgr2=0x7f0cb006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cb006c4c0 0x7f0cb006e980 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f0cd81fab00 tx=0x7f0cc0008040 comp rx=0 tx=0).stop 2024-01-15T18:21:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 msgr2=0x7f0cd8145fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 0x7f0cd8145fd0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f0cc800d800 tx=0x7f0cc800dcc0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 shutdown_connections 2024-01-15T18:21:03.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f0cb006c4c0 0x7f0cb006e980 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f0cd814af80 0x7f0cd80a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f0cd80700d0 0x7f0cd815fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 --2- 172.21.15.2:0/3204726576 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f0cd806da60 0x7f0cd8145fd0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 >> 172.21.15.2:0/3204726576 conn(0x7f0cd80b0840 msgr2=0x7f0cd81611e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.114+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 shutdown_connections 2024-01-15T18:21:03.120 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.115+0000 7f0cdd800640 1 -- 172.21.15.2:0/3204726576 wait complete. 2024-01-15T18:21:03.139 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":818,"stamp":"2024-01-15T18:21:01.987927+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5369947027,"num_objects":1311,"num_object_clones":1,"num_object_copies":2622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1311,"num_whiteouts":1,"num_read":651,"num_read_kb":2097275,"num_write":1755,"num_write_kb":5246075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4397,"ondisk_log_size":4397,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11062144,"kb_used_data":10499520,"kb_used_omap":12,"kb_used_meta":562547,"kb_avail":743912576,"statfs":{"total":773094113280,"available":761766477824,"internally_reserved":0,"allocated":10751508480,"data_stored":10744950094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":576048689},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004948"},"pg_stats":[{"pgid":"17.3f","version":"116'78","reported_seq":113,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:28.138656+0000","last_change":"2024-01-15T18:20:28.138656+0000","last_active":"2024-01-15T18:20:28.138656+0000","last_peered":"2024-01-15T18:20:28.138656+0000","last_clean":"2024-01-15T18:20:28.138656+0000","last_became_active":"2024-01-15T18:14:16.890189+0000","last_became_peered":"2024-01-15T18:14:16.890189+0000","last_unstale":"2024-01-15T18:20:28.138656+0000","last_undegraded":"2024-01-15T18:20:28.138656+0000","last_fullsized":"2024-01-15T18:20:28.138656+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'78","last_scrub_stamp":"2024-01-15T18:20:28.138374+0000","last_deep_scrub":"116'78","last_deep_scrub_stamp":"2024-01-15T18:20:28.138374+0000","last_clean_scrub_stamp":"2024-01-15T18:20:28.138374+0000","objects_scrubbed":25,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:38:47.961088+0000","scrub_duration":0.30264037199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"116'64","reported_seq":97,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:28.112068+0000","last_change":"2024-01-15T18:20:28.112068+0000","last_active":"2024-01-15T18:20:28.112068+0000","last_peered":"2024-01-15T18:20:28.112068+0000","last_clean":"2024-01-15T18:20:28.112068+0000","last_became_active":"2024-01-15T18:14:16.874408+0000","last_became_peered":"2024-01-15T18:14:16.874408+0000","last_unstale":"2024-01-15T18:20:28.112068+0000","last_undegraded":"2024-01-15T18:20:28.112068+0000","last_fullsized":"2024-01-15T18:20:28.112068+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:28.111850+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:28.111850+0000","last_clean_scrub_stamp":"2024-01-15T18:20:28.111850+0000","objects_scrubbed":19,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:50:32.785375+0000","scrub_duration":0.24068798999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"116'58","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.058201+0000","last_change":"2024-01-15T18:20:08.058201+0000","last_active":"2024-01-15T18:20:08.058201+0000","last_peered":"2024-01-15T18:20:08.058201+0000","last_clean":"2024-01-15T18:20:08.058201+0000","last_became_active":"2024-01-15T18:14:16.886752+0000","last_became_peered":"2024-01-15T18:14:16.886752+0000","last_unstale":"2024-01-15T18:20:08.058201+0000","last_undegraded":"2024-01-15T18:20:08.058201+0000","last_fullsized":"2024-01-15T18:20:08.058201+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:08.058029+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:08.058029+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.058029+0000","objects_scrubbed":19,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:17.015587+0000","scrub_duration":0.240566523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"116'68","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:21.129120+0000","last_change":"2024-01-15T18:20:21.129210+0000","last_active":"2024-01-15T18:20:21.129120+0000","last_peered":"2024-01-15T18:20:21.129120+0000","last_clean":"2024-01-15T18:20:21.129120+0000","last_became_active":"2024-01-15T18:14:16.883430+0000","last_became_peered":"2024-01-15T18:14:16.883430+0000","last_unstale":"2024-01-15T18:20:21.129120+0000","last_undegraded":"2024-01-15T18:20:21.129120+0000","last_fullsized":"2024-01-15T18:20:21.129120+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:21.128944+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:21.128944+0000","last_clean_scrub_stamp":"2024-01-15T18:20:21.128944+0000","objects_scrubbed":23,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:39:22.724305+0000","scrub_duration":0.25324432800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"116'37","reported_seq":68,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:27.424747+0000","last_change":"2024-01-15T18:20:27.424890+0000","last_active":"2024-01-15T18:20:27.424747+0000","last_peered":"2024-01-15T18:20:27.424747+0000","last_clean":"2024-01-15T18:20:27.424747+0000","last_became_active":"2024-01-15T18:14:16.884253+0000","last_became_peered":"2024-01-15T18:14:16.884253+0000","last_unstale":"2024-01-15T18:20:27.424747+0000","last_undegraded":"2024-01-15T18:20:27.424747+0000","last_fullsized":"2024-01-15T18:20:27.424747+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'37","last_scrub_stamp":"2024-01-15T18:20:27.424520+0000","last_deep_scrub":"116'37","last_deep_scrub_stamp":"2024-01-15T18:20:27.424520+0000","last_clean_scrub_stamp":"2024-01-15T18:20:27.424520+0000","objects_scrubbed":10,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:28:29.494758+0000","scrub_duration":0.109363945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"116'69","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:21.035113+0000","last_change":"2024-01-15T18:20:21.035167+0000","last_active":"2024-01-15T18:20:21.035113+0000","last_peered":"2024-01-15T18:20:21.035113+0000","last_clean":"2024-01-15T18:20:21.035113+0000","last_became_active":"2024-01-15T18:14:16.883264+0000","last_became_peered":"2024-01-15T18:14:16.883264+0000","last_unstale":"2024-01-15T18:20:21.035113+0000","last_undegraded":"2024-01-15T18:20:21.035113+0000","last_fullsized":"2024-01-15T18:20:21.035113+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'69","last_scrub_stamp":"2024-01-15T18:20:21.035022+0000","last_deep_scrub":"116'69","last_deep_scrub_stamp":"2024-01-15T18:20:21.035022+0000","last_clean_scrub_stamp":"2024-01-15T18:20:21.035022+0000","objects_scrubbed":21,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:06:15.159740+0000","scrub_duration":0.179535004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"116'60","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.440550+0000","last_change":"2024-01-15T18:20:23.440661+0000","last_active":"2024-01-15T18:20:23.440550+0000","last_peered":"2024-01-15T18:20:23.440550+0000","last_clean":"2024-01-15T18:20:23.440550+0000","last_became_active":"2024-01-15T18:14:16.874612+0000","last_became_peered":"2024-01-15T18:14:16.874612+0000","last_unstale":"2024-01-15T18:20:23.440550+0000","last_undegraded":"2024-01-15T18:20:23.440550+0000","last_fullsized":"2024-01-15T18:20:23.440550+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'60","last_scrub_stamp":"2024-01-15T18:20:23.440397+0000","last_deep_scrub":"116'60","last_deep_scrub_stamp":"2024-01-15T18:20:23.440397+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.440397+0000","objects_scrubbed":17,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:09:11.569070+0000","scrub_duration":0.20668540199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:32.063388+0000","last_change":"2024-01-15T18:20:32.063563+0000","last_active":"2024-01-15T18:20:32.063388+0000","last_peered":"2024-01-15T18:20:32.063388+0000","last_clean":"2024-01-15T18:20:32.063388+0000","last_became_active":"2024-01-15T18:14:16.888084+0000","last_became_peered":"2024-01-15T18:14:16.888084+0000","last_unstale":"2024-01-15T18:20:32.063388+0000","last_undegraded":"2024-01-15T18:20:32.063388+0000","last_fullsized":"2024-01-15T18:20:32.063388+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:32.063198+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:32.063198+0000","last_clean_scrub_stamp":"2024-01-15T18:20:32.063198+0000","objects_scrubbed":19,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:35.428438+0000","scrub_duration":0.227368862,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:13.940273+0000","last_change":"2024-01-15T18:20:13.940362+0000","last_active":"2024-01-15T18:20:13.940273+0000","last_peered":"2024-01-15T18:20:13.940273+0000","last_clean":"2024-01-15T18:20:13.940273+0000","last_became_active":"2024-01-15T18:14:16.878953+0000","last_became_peered":"2024-01-15T18:14:16.878953+0000","last_unstale":"2024-01-15T18:20:13.940273+0000","last_undegraded":"2024-01-15T18:20:13.940273+0000","last_fullsized":"2024-01-15T18:20:13.940273+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:13.940094+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:13.940094+0000","last_clean_scrub_stamp":"2024-01-15T18:20:13.940094+0000","objects_scrubbed":21,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:46:44.099042+0000","scrub_duration":0.21865759100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"116'64","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:29.039071+0000","last_change":"2024-01-15T18:20:29.039119+0000","last_active":"2024-01-15T18:20:29.039071+0000","last_peered":"2024-01-15T18:20:29.039071+0000","last_clean":"2024-01-15T18:20:29.039071+0000","last_became_active":"2024-01-15T18:14:16.886600+0000","last_became_peered":"2024-01-15T18:14:16.886600+0000","last_unstale":"2024-01-15T18:20:29.039071+0000","last_undegraded":"2024-01-15T18:20:29.039071+0000","last_fullsized":"2024-01-15T18:20:29.039071+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:29.038992+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:29.038992+0000","last_clean_scrub_stamp":"2024-01-15T18:20:29.038992+0000","objects_scrubbed":17,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:30:50.890716+0000","scrub_duration":0.177717706,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"116'75","reported_seq":112,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:26.268972+0000","last_change":"2024-01-15T18:20:26.269040+0000","last_active":"2024-01-15T18:20:26.268972+0000","last_peered":"2024-01-15T18:20:26.268972+0000","last_clean":"2024-01-15T18:20:26.268972+0000","last_became_active":"2024-01-15T18:14:16.874695+0000","last_became_peered":"2024-01-15T18:14:16.874695+0000","last_unstale":"2024-01-15T18:20:26.268972+0000","last_undegraded":"2024-01-15T18:20:26.268972+0000","last_fullsized":"2024-01-15T18:20:26.268972+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'75","last_scrub_stamp":"2024-01-15T18:20:26.268745+0000","last_deep_scrub":"116'75","last_deep_scrub_stamp":"2024-01-15T18:20:26.268745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:26.268745+0000","objects_scrubbed":28,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:41:07.930002+0000","scrub_duration":0.35549178100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":28,"num_write_kb":114688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"116'76","reported_seq":117,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.067580+0000","last_change":"2024-01-15T18:20:18.067609+0000","last_active":"2024-01-15T18:20:18.067580+0000","last_peered":"2024-01-15T18:20:18.067580+0000","last_clean":"2024-01-15T18:20:18.067580+0000","last_became_active":"2024-01-15T18:14:16.882631+0000","last_became_peered":"2024-01-15T18:14:16.882631+0000","last_unstale":"2024-01-15T18:20:18.067580+0000","last_undegraded":"2024-01-15T18:20:18.067580+0000","last_fullsized":"2024-01-15T18:20:18.067580+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'76","last_scrub_stamp":"2024-01-15T18:20:18.067485+0000","last_deep_scrub":"116'76","last_deep_scrub_stamp":"2024-01-15T18:20:18.067485+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.067485+0000","objects_scrubbed":28,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:12:51.591673+0000","scrub_duration":0.24627128300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117440512,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":30,"num_write_kb":114689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"116'57","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:29.068517+0000","last_change":"2024-01-15T18:20:29.068609+0000","last_active":"2024-01-15T18:20:29.068517+0000","last_peered":"2024-01-15T18:20:29.068517+0000","last_clean":"2024-01-15T18:20:29.068517+0000","last_became_active":"2024-01-15T18:14:16.885009+0000","last_became_peered":"2024-01-15T18:14:16.885009+0000","last_unstale":"2024-01-15T18:20:29.068517+0000","last_undegraded":"2024-01-15T18:20:29.068517+0000","last_fullsized":"2024-01-15T18:20:29.068517+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:29.068261+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:29.068261+0000","last_clean_scrub_stamp":"2024-01-15T18:20:29.068261+0000","objects_scrubbed":19,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:36:32.030241+0000","scrub_duration":0.20564858599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"116'70","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.101682+0000","last_change":"2024-01-15T18:20:14.101766+0000","last_active":"2024-01-15T18:20:14.101682+0000","last_peered":"2024-01-15T18:20:14.101682+0000","last_clean":"2024-01-15T18:20:14.101682+0000","last_became_active":"2024-01-15T18:14:16.885320+0000","last_became_peered":"2024-01-15T18:14:16.885320+0000","last_unstale":"2024-01-15T18:20:14.101682+0000","last_undegraded":"2024-01-15T18:20:14.101682+0000","last_fullsized":"2024-01-15T18:20:14.101682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'70","last_scrub_stamp":"2024-01-15T18:20:14.101527+0000","last_deep_scrub":"116'70","last_deep_scrub_stamp":"2024-01-15T18:20:14.101527+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.101527+0000","objects_scrubbed":27,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:33:26.300645+0000","scrub_duration":0.32403122899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"116'41","reported_seq":69,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.927343+0000","last_change":"2024-01-15T18:20:09.927446+0000","last_active":"2024-01-15T18:20:09.927343+0000","last_peered":"2024-01-15T18:20:09.927343+0000","last_clean":"2024-01-15T18:20:09.927343+0000","last_became_active":"2024-01-15T18:14:16.881039+0000","last_became_peered":"2024-01-15T18:14:16.881039+0000","last_unstale":"2024-01-15T18:20:09.927343+0000","last_undegraded":"2024-01-15T18:20:09.927343+0000","last_fullsized":"2024-01-15T18:20:09.927343+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'41","last_scrub_stamp":"2024-01-15T18:20:09.927172+0000","last_deep_scrub":"116'41","last_deep_scrub_stamp":"2024-01-15T18:20:09.927172+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.927172+0000","objects_scrubbed":12,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:47.958926+0000","scrub_duration":0.12571378,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"116'58","reported_seq":85,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.924633+0000","last_change":"2024-01-15T18:20:11.924771+0000","last_active":"2024-01-15T18:20:11.924633+0000","last_peered":"2024-01-15T18:20:11.924633+0000","last_clean":"2024-01-15T18:20:11.924633+0000","last_became_active":"2024-01-15T18:14:16.883786+0000","last_became_peered":"2024-01-15T18:14:16.883786+0000","last_unstale":"2024-01-15T18:20:11.924633+0000","last_undegraded":"2024-01-15T18:20:11.924633+0000","last_fullsized":"2024-01-15T18:20:11.924633+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:11.924423+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:11.924423+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.924423+0000","objects_scrubbed":18,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:53:06.081196+0000","scrub_duration":0.195491471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"116'66","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:24.225189+0000","last_change":"2024-01-15T18:20:24.225257+0000","last_active":"2024-01-15T18:20:24.225189+0000","last_peered":"2024-01-15T18:20:24.225189+0000","last_clean":"2024-01-15T18:20:24.225189+0000","last_became_active":"2024-01-15T18:14:16.876117+0000","last_became_peered":"2024-01-15T18:14:16.876117+0000","last_unstale":"2024-01-15T18:20:24.225189+0000","last_undegraded":"2024-01-15T18:20:24.225189+0000","last_fullsized":"2024-01-15T18:20:24.225189+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'66","last_scrub_stamp":"2024-01-15T18:20:24.225072+0000","last_deep_scrub":"116'66","last_deep_scrub_stamp":"2024-01-15T18:20:24.225072+0000","last_clean_scrub_stamp":"2024-01-15T18:20:24.225072+0000","objects_scrubbed":22,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:22.612626+0000","scrub_duration":0.26762065200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"116'69","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:25.149726+0000","last_change":"2024-01-15T18:20:25.149827+0000","last_active":"2024-01-15T18:20:25.149726+0000","last_peered":"2024-01-15T18:20:25.149726+0000","last_clean":"2024-01-15T18:20:25.149726+0000","last_became_active":"2024-01-15T18:14:16.877130+0000","last_became_peered":"2024-01-15T18:14:16.877130+0000","last_unstale":"2024-01-15T18:20:25.149726+0000","last_undegraded":"2024-01-15T18:20:25.149726+0000","last_fullsized":"2024-01-15T18:20:25.149726+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'69","last_scrub_stamp":"2024-01-15T18:20:25.149549+0000","last_deep_scrub":"116'69","last_deep_scrub_stamp":"2024-01-15T18:20:25.149549+0000","last_clean_scrub_stamp":"2024-01-15T18:20:25.149549+0000","objects_scrubbed":21,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:15:23.968047+0000","scrub_duration":0.27027720900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"116'61","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.024154+0000","last_change":"2024-01-15T18:20:09.024235+0000","last_active":"2024-01-15T18:20:09.024154+0000","last_peered":"2024-01-15T18:20:09.024154+0000","last_clean":"2024-01-15T18:20:09.024154+0000","last_became_active":"2024-01-15T18:14:16.888249+0000","last_became_peered":"2024-01-15T18:14:16.888249+0000","last_unstale":"2024-01-15T18:20:09.024154+0000","last_undegraded":"2024-01-15T18:20:09.024154+0000","last_fullsized":"2024-01-15T18:20:09.024154+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'61","last_scrub_stamp":"2024-01-15T18:20:09.023969+0000","last_deep_scrub":"116'61","last_deep_scrub_stamp":"2024-01-15T18:20:09.023969+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.023969+0000","objects_scrubbed":21,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:22.924312+0000","scrub_duration":0.233465062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.356321+0000","last_change":"2024-01-15T18:20:14.356433+0000","last_active":"2024-01-15T18:20:14.356321+0000","last_peered":"2024-01-15T18:20:14.356321+0000","last_clean":"2024-01-15T18:20:14.356321+0000","last_became_active":"2024-01-15T18:14:16.883435+0000","last_became_peered":"2024-01-15T18:14:16.883435+0000","last_unstale":"2024-01-15T18:20:14.356321+0000","last_undegraded":"2024-01-15T18:20:14.356321+0000","last_fullsized":"2024-01-15T18:20:14.356321+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:14.356135+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:14.356135+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.356135+0000","objects_scrubbed":17,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:35:09.060881+0000","scrub_duration":0.21269460900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"116'76","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.437476+0000","last_change":"2024-01-15T18:20:09.437603+0000","last_active":"2024-01-15T18:20:09.437476+0000","last_peered":"2024-01-15T18:20:09.437476+0000","last_clean":"2024-01-15T18:20:09.437476+0000","last_became_active":"2024-01-15T18:14:16.878208+0000","last_became_peered":"2024-01-15T18:14:16.878208+0000","last_unstale":"2024-01-15T18:20:09.437476+0000","last_undegraded":"2024-01-15T18:20:09.437476+0000","last_fullsized":"2024-01-15T18:20:09.437476+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'76","last_scrub_stamp":"2024-01-15T18:20:09.437240+0000","last_deep_scrub":"116'76","last_deep_scrub_stamp":"2024-01-15T18:20:09.437240+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.437240+0000","objects_scrubbed":26,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:44:46.474036+0000","scrub_duration":0.288134063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":28,"num_write_kb":106497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"116'68","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:17.575494+0000","last_change":"2024-01-15T18:20:17.575599+0000","last_active":"2024-01-15T18:20:17.575494+0000","last_peered":"2024-01-15T18:20:17.575494+0000","last_clean":"2024-01-15T18:20:17.575494+0000","last_became_active":"2024-01-15T18:14:16.878150+0000","last_became_peered":"2024-01-15T18:14:16.878150+0000","last_unstale":"2024-01-15T18:20:17.575494+0000","last_undegraded":"2024-01-15T18:20:17.575494+0000","last_fullsized":"2024-01-15T18:20:17.575494+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:17.575306+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:17.575306+0000","last_clean_scrub_stamp":"2024-01-15T18:20:17.575306+0000","objects_scrubbed":21,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:31:05.967956+0000","scrub_duration":0.230033495,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":24578,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"116'54","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:27.094581+0000","last_change":"2024-01-15T18:20:27.094637+0000","last_active":"2024-01-15T18:20:27.094581+0000","last_peered":"2024-01-15T18:20:27.094581+0000","last_clean":"2024-01-15T18:20:27.094581+0000","last_became_active":"2024-01-15T18:14:16.882157+0000","last_became_peered":"2024-01-15T18:14:16.882157+0000","last_unstale":"2024-01-15T18:20:27.094581+0000","last_undegraded":"2024-01-15T18:20:27.094581+0000","last_fullsized":"2024-01-15T18:20:27.094581+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'54","last_scrub_stamp":"2024-01-15T18:20:27.094454+0000","last_deep_scrub":"116'54","last_deep_scrub_stamp":"2024-01-15T18:20:27.094454+0000","last_clean_scrub_stamp":"2024-01-15T18:20:27.094454+0000","objects_scrubbed":15,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:41:49.226716+0000","scrub_duration":0.196613649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"116'57","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.630687+0000","last_change":"2024-01-15T18:20:06.630687+0000","last_active":"2024-01-15T18:20:06.630687+0000","last_peered":"2024-01-15T18:20:06.630687+0000","last_clean":"2024-01-15T18:20:06.630687+0000","last_became_active":"2024-01-15T18:14:16.883715+0000","last_became_peered":"2024-01-15T18:14:16.883715+0000","last_unstale":"2024-01-15T18:20:06.630687+0000","last_undegraded":"2024-01-15T18:20:06.630687+0000","last_fullsized":"2024-01-15T18:20:06.630687+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:06.630489+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:06.630489+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.630489+0000","objects_scrubbed":18,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:22:43.143918+0000","scrub_duration":0.18958351200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"116'56","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:20.249893+0000","last_change":"2024-01-15T18:20:20.249893+0000","last_active":"2024-01-15T18:20:20.249893+0000","last_peered":"2024-01-15T18:20:20.249893+0000","last_clean":"2024-01-15T18:20:20.249893+0000","last_became_active":"2024-01-15T18:14:16.875829+0000","last_became_peered":"2024-01-15T18:14:16.875829+0000","last_unstale":"2024-01-15T18:20:20.249893+0000","last_undegraded":"2024-01-15T18:20:20.249893+0000","last_fullsized":"2024-01-15T18:20:20.249893+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:20.249662+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:20.249662+0000","last_clean_scrub_stamp":"2024-01-15T18:20:20.249662+0000","objects_scrubbed":18,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:13:08.460330+0000","scrub_duration":0.21997658,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"116'73","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.039365+0000","last_change":"2024-01-15T18:19:57.039365+0000","last_active":"2024-01-15T18:19:57.039365+0000","last_peered":"2024-01-15T18:19:57.039365+0000","last_clean":"2024-01-15T18:19:57.039365+0000","last_became_active":"2024-01-15T18:14:16.876713+0000","last_became_peered":"2024-01-15T18:14:16.876713+0000","last_unstale":"2024-01-15T18:19:57.039365+0000","last_undegraded":"2024-01-15T18:19:57.039365+0000","last_fullsized":"2024-01-15T18:19:57.039365+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:19:57.039107+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.039107+0000","objects_scrubbed":23,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:55:44.107907+0000","scrub_duration":0.24278212099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"116'58","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:26.009801+0000","last_change":"2024-01-15T18:20:26.009801+0000","last_active":"2024-01-15T18:20:26.009801+0000","last_peered":"2024-01-15T18:20:26.009801+0000","last_clean":"2024-01-15T18:20:26.009801+0000","last_became_active":"2024-01-15T18:14:16.892688+0000","last_became_peered":"2024-01-15T18:14:16.892688+0000","last_unstale":"2024-01-15T18:20:26.009801+0000","last_undegraded":"2024-01-15T18:20:26.009801+0000","last_fullsized":"2024-01-15T18:20:26.009801+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:26.009677+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:26.009677+0000","last_clean_scrub_stamp":"2024-01-15T18:20:26.009677+0000","objects_scrubbed":21,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:21:02.258822+0000","scrub_duration":0.16302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886260,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":21,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"116'49","reported_seq":77,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:14.519960+0000","last_change":"2024-01-15T18:20:14.519960+0000","last_active":"2024-01-15T18:20:14.519960+0000","last_peered":"2024-01-15T18:20:14.519960+0000","last_clean":"2024-01-15T18:20:14.519960+0000","last_became_active":"2024-01-15T18:14:16.879616+0000","last_became_peered":"2024-01-15T18:14:16.879616+0000","last_unstale":"2024-01-15T18:20:14.519960+0000","last_undegraded":"2024-01-15T18:20:14.519960+0000","last_fullsized":"2024-01-15T18:20:14.519960+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:14.519687+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:14.519687+0000","last_clean_scrub_stamp":"2024-01-15T18:20:14.519687+0000","objects_scrubbed":17,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:50:17.600055+0000","scrub_duration":0.16789694699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"116'72","reported_seq":107,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.222197+0000","last_change":"2024-01-15T18:20:23.222197+0000","last_active":"2024-01-15T18:20:23.222197+0000","last_peered":"2024-01-15T18:20:23.222197+0000","last_clean":"2024-01-15T18:20:23.222197+0000","last_became_active":"2024-01-15T18:14:16.885862+0000","last_became_peered":"2024-01-15T18:14:16.885862+0000","last_unstale":"2024-01-15T18:20:23.222197+0000","last_undegraded":"2024-01-15T18:20:23.222197+0000","last_fullsized":"2024-01-15T18:20:23.222197+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:23.221964+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:23.221964+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.221964+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:37:41.425250+0000","scrub_duration":0.28073583600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"116'56","reported_seq":90,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:15.454009+0000","last_change":"2024-01-15T18:20:15.454009+0000","last_active":"2024-01-15T18:20:15.454009+0000","last_peered":"2024-01-15T18:20:15.454009+0000","last_clean":"2024-01-15T18:20:15.454009+0000","last_became_active":"2024-01-15T18:14:16.872692+0000","last_became_peered":"2024-01-15T18:14:16.872692+0000","last_unstale":"2024-01-15T18:20:15.454009+0000","last_undegraded":"2024-01-15T18:20:15.454009+0000","last_fullsized":"2024-01-15T18:20:15.454009+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:15.453578+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:15.453578+0000","last_clean_scrub_stamp":"2024-01-15T18:20:15.453578+0000","objects_scrubbed":16,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:16:00.845188+0000","scrub_duration":0.18720494500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":9,"num_read_kb":32769,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"116'88","reported_seq":126,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:33.231110+0000","last_change":"2024-01-15T18:20:33.231110+0000","last_active":"2024-01-15T18:20:33.231110+0000","last_peered":"2024-01-15T18:20:33.231110+0000","last_clean":"2024-01-15T18:20:33.231110+0000","last_became_active":"2024-01-15T18:14:16.887371+0000","last_became_peered":"2024-01-15T18:14:16.887371+0000","last_unstale":"2024-01-15T18:20:33.231110+0000","last_undegraded":"2024-01-15T18:20:33.231110+0000","last_fullsized":"2024-01-15T18:20:33.231110+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'88","last_scrub_stamp":"2024-01-15T18:20:33.230776+0000","last_deep_scrub":"116'88","last_deep_scrub_stamp":"2024-01-15T18:20:33.230776+0000","last_clean_scrub_stamp":"2024-01-15T18:20:33.230776+0000","objects_scrubbed":26,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:10:37.372939+0000","scrub_duration":0.30655612300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"116'72","reported_seq":104,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.952237+0000","last_change":"2024-01-15T18:20:07.952237+0000","last_active":"2024-01-15T18:20:07.952237+0000","last_peered":"2024-01-15T18:20:07.952237+0000","last_clean":"2024-01-15T18:20:07.952237+0000","last_became_active":"2024-01-15T18:14:16.877870+0000","last_became_peered":"2024-01-15T18:14:16.877870+0000","last_unstale":"2024-01-15T18:20:07.952237+0000","last_undegraded":"2024-01-15T18:20:07.952237+0000","last_fullsized":"2024-01-15T18:20:07.952237+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:07.952059+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:07.952059+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.952059+0000","objects_scrubbed":22,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:12:04.767828+0000","scrub_duration":0.20312791999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"116'49","reported_seq":80,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.030986+0000","last_change":"2024-01-15T18:20:18.030986+0000","last_active":"2024-01-15T18:20:18.030986+0000","last_peered":"2024-01-15T18:20:18.030986+0000","last_clean":"2024-01-15T18:20:18.030986+0000","last_became_active":"2024-01-15T18:14:16.876841+0000","last_became_peered":"2024-01-15T18:14:16.876841+0000","last_unstale":"2024-01-15T18:20:18.030986+0000","last_undegraded":"2024-01-15T18:20:18.030986+0000","last_fullsized":"2024-01-15T18:20:18.030986+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:18.030841+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:18.030841+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.030841+0000","objects_scrubbed":15,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:14:05.232365+0000","scrub_duration":0.134479771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"116'6","reported_seq":32,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.264156+0000","last_change":"2024-01-15T18:20:11.264156+0000","last_active":"2024-01-15T18:20:11.264156+0000","last_peered":"2024-01-15T18:20:11.264156+0000","last_clean":"2024-01-15T18:20:11.264156+0000","last_became_active":"2024-01-15T18:14:15.868981+0000","last_became_peered":"2024-01-15T18:14:15.868981+0000","last_unstale":"2024-01-15T18:20:11.264156+0000","last_undegraded":"2024-01-15T18:20:11.264156+0000","last_fullsized":"2024-01-15T18:20:11.264156+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'6","last_scrub_stamp":"2024-01-15T18:20:11.263910+0000","last_deep_scrub":"116'6","last_deep_scrub_stamp":"2024-01-15T18:20:11.263910+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.263910+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:06:05.388628+0000","scrub_duration":0.0087165769999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"116'69","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:30.237548+0000","last_change":"2024-01-15T18:20:30.237548+0000","last_active":"2024-01-15T18:20:30.237548+0000","last_peered":"2024-01-15T18:20:30.237548+0000","last_clean":"2024-01-15T18:20:30.237548+0000","last_became_active":"2024-01-15T18:14:16.883134+0000","last_became_peered":"2024-01-15T18:14:16.883134+0000","last_unstale":"2024-01-15T18:20:30.237548+0000","last_undegraded":"2024-01-15T18:20:30.237548+0000","last_fullsized":"2024-01-15T18:20:30.237548+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'69","last_scrub_stamp":"2024-01-15T18:20:30.237218+0000","last_deep_scrub":"116'69","last_deep_scrub_stamp":"2024-01-15T18:20:30.237218+0000","last_clean_scrub_stamp":"2024-01-15T18:20:30.237218+0000","objects_scrubbed":22,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:33:42.436336+0000","scrub_duration":0.27258786400000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"113'1","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.887618+0000","last_change":"2024-01-15T18:20:04.887618+0000","last_active":"2024-01-15T18:20:04.887618+0000","last_peered":"2024-01-15T18:20:04.887618+0000","last_clean":"2024-01-15T18:20:04.887618+0000","last_became_active":"2024-01-15T18:14:15.874630+0000","last_became_peered":"2024-01-15T18:14:15.874630+0000","last_unstale":"2024-01-15T18:20:04.887618+0000","last_undegraded":"2024-01-15T18:20:04.887618+0000","last_fullsized":"2024-01-15T18:20:04.887618+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:04.887335+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:04.887335+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.887335+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:16:28.013839+0000","scrub_duration":0.003453177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"116'59","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.940971+0000","last_change":"2024-01-15T18:20:05.941055+0000","last_active":"2024-01-15T18:20:05.940971+0000","last_peered":"2024-01-15T18:20:05.940971+0000","last_clean":"2024-01-15T18:20:05.940971+0000","last_became_active":"2024-01-15T18:14:16.879666+0000","last_became_peered":"2024-01-15T18:14:16.879666+0000","last_unstale":"2024-01-15T18:20:05.940971+0000","last_undegraded":"2024-01-15T18:20:05.940971+0000","last_fullsized":"2024-01-15T18:20:05.940971+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:05.940887+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:05.940887+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.940887+0000","objects_scrubbed":19,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:41:28.613623+0000","scrub_duration":0.17468556599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.370570+0000","last_change":"2024-01-15T18:20:08.370633+0000","last_active":"2024-01-15T18:20:08.370570+0000","last_peered":"2024-01-15T18:20:08.370570+0000","last_clean":"2024-01-15T18:20:08.370570+0000","last_became_active":"2024-01-15T18:14:15.879455+0000","last_became_peered":"2024-01-15T18:14:15.879455+0000","last_unstale":"2024-01-15T18:20:08.370570+0000","last_undegraded":"2024-01-15T18:20:08.370570+0000","last_fullsized":"2024-01-15T18:20:08.370570+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:08.370363+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:08.370363+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.370363+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:27:33.863692+0000","scrub_duration":0.0069394490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"116'79","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.081362+0000","last_change":"2024-01-15T18:20:18.081448+0000","last_active":"2024-01-15T18:20:18.081362+0000","last_peered":"2024-01-15T18:20:18.081362+0000","last_clean":"2024-01-15T18:20:18.081362+0000","last_became_active":"2024-01-15T18:14:16.889004+0000","last_became_peered":"2024-01-15T18:14:16.889004+0000","last_unstale":"2024-01-15T18:20:18.081362+0000","last_undegraded":"2024-01-15T18:20:18.081362+0000","last_fullsized":"2024-01-15T18:20:18.081362+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'79","last_scrub_stamp":"2024-01-15T18:20:18.081158+0000","last_deep_scrub":"116'79","last_deep_scrub_stamp":"2024-01-15T18:20:18.081158+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.081158+0000","objects_scrubbed":23,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:45.499261+0000","scrub_duration":0.24476172700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.355457+0000","last_change":"2024-01-15T18:20:11.355522+0000","last_active":"2024-01-15T18:20:11.355457+0000","last_peered":"2024-01-15T18:20:11.355457+0000","last_clean":"2024-01-15T18:20:11.355457+0000","last_became_active":"2024-01-15T18:14:15.879448+0000","last_became_peered":"2024-01-15T18:14:15.879448+0000","last_unstale":"2024-01-15T18:20:11.355457+0000","last_undegraded":"2024-01-15T18:20:11.355457+0000","last_fullsized":"2024-01-15T18:20:11.355457+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:11.355263+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:11.355263+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.355263+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:39:18.416652+0000","scrub_duration":0.0069302460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"116'56","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:25.987945+0000","last_change":"2024-01-15T18:20:25.988008+0000","last_active":"2024-01-15T18:20:25.987945+0000","last_peered":"2024-01-15T18:20:25.987945+0000","last_clean":"2024-01-15T18:20:25.987945+0000","last_became_active":"2024-01-15T18:14:16.885869+0000","last_became_peered":"2024-01-15T18:14:16.885869+0000","last_unstale":"2024-01-15T18:20:25.987945+0000","last_undegraded":"2024-01-15T18:20:25.987945+0000","last_fullsized":"2024-01-15T18:20:25.987945+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'56","last_scrub_stamp":"2024-01-15T18:20:25.987848+0000","last_deep_scrub":"116'56","last_deep_scrub_stamp":"2024-01-15T18:20:25.987848+0000","last_clean_scrub_stamp":"2024-01-15T18:20:25.987848+0000","objects_scrubbed":17,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:49:55.244578+0000","scrub_duration":0.153668372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"113'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.327294+0000","last_change":"2024-01-15T18:20:09.327345+0000","last_active":"2024-01-15T18:20:09.327294+0000","last_peered":"2024-01-15T18:20:09.327294+0000","last_clean":"2024-01-15T18:20:09.327294+0000","last_became_active":"2024-01-15T18:14:15.871043+0000","last_became_peered":"2024-01-15T18:14:15.871043+0000","last_unstale":"2024-01-15T18:20:09.327294+0000","last_undegraded":"2024-01-15T18:20:09.327294+0000","last_fullsized":"2024-01-15T18:20:09.327294+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:09.327108+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:09.327108+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.327108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:13:36.860290+0000","scrub_duration":0.003411468,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"116'73","reported_seq":110,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.265444+0000","last_change":"2024-01-15T18:20:19.265562+0000","last_active":"2024-01-15T18:20:19.265444+0000","last_peered":"2024-01-15T18:20:19.265444+0000","last_clean":"2024-01-15T18:20:19.265444+0000","last_became_active":"2024-01-15T18:14:16.871335+0000","last_became_peered":"2024-01-15T18:14:16.871335+0000","last_unstale":"2024-01-15T18:20:19.265444+0000","last_undegraded":"2024-01-15T18:20:19.265444+0000","last_fullsized":"2024-01-15T18:20:19.265444+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2024-01-15T18:20:19.265264+0000","last_deep_scrub":"116'73","last_deep_scrub_stamp":"2024-01-15T18:20:19.265264+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.265264+0000","objects_scrubbed":24,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:11:44.988416+0000","scrub_duration":0.25818551200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"113'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.459148+0000","last_change":"2024-01-15T18:20:04.459201+0000","last_active":"2024-01-15T18:20:04.459148+0000","last_peered":"2024-01-15T18:20:04.459148+0000","last_clean":"2024-01-15T18:20:04.459148+0000","last_became_active":"2024-01-15T18:14:15.870605+0000","last_became_peered":"2024-01-15T18:14:15.870605+0000","last_unstale":"2024-01-15T18:20:04.459148+0000","last_undegraded":"2024-01-15T18:20:04.459148+0000","last_fullsized":"2024-01-15T18:20:04.459148+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'1","last_scrub_stamp":"2024-01-15T18:20:04.458942+0000","last_deep_scrub":"113'1","last_deep_scrub_stamp":"2024-01-15T18:20:04.458942+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.458942+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:42.315332+0000","scrub_duration":0.0039832870000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"116'71","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:00.901734+0000","last_change":"2024-01-15T18:20:00.901779+0000","last_active":"2024-01-15T18:20:00.901734+0000","last_peered":"2024-01-15T18:20:00.901734+0000","last_clean":"2024-01-15T18:20:00.901734+0000","last_became_active":"2024-01-15T18:14:16.882645+0000","last_became_peered":"2024-01-15T18:14:16.882645+0000","last_unstale":"2024-01-15T18:20:00.901734+0000","last_undegraded":"2024-01-15T18:20:00.901734+0000","last_fullsized":"2024-01-15T18:20:00.901734+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'71","last_scrub_stamp":"2024-01-15T18:20:00.901655+0000","last_deep_scrub":"116'71","last_deep_scrub_stamp":"2024-01-15T18:20:00.901655+0000","last_clean_scrub_stamp":"2024-01-15T18:20:00.901655+0000","objects_scrubbed":26,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:02:12.623865+0000","scrub_duration":0.173092522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857792,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":26,"num_write_kb":102401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.168864+0000","last_change":"2024-01-15T18:20:01.168915+0000","last_active":"2024-01-15T18:20:01.168864+0000","last_peered":"2024-01-15T18:20:01.168864+0000","last_clean":"2024-01-15T18:20:01.168864+0000","last_became_active":"2024-01-15T18:14:15.866674+0000","last_became_peered":"2024-01-15T18:14:15.866674+0000","last_unstale":"2024-01-15T18:20:01.168864+0000","last_undegraded":"2024-01-15T18:20:01.168864+0000","last_fullsized":"2024-01-15T18:20:01.168864+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:01.168637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:01.168637+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.168637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:58:03.821391+0000","scrub_duration":0.0034424299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"116'59","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.355545+0000","last_change":"2024-01-15T18:20:34.355645+0000","last_active":"2024-01-15T18:20:34.355545+0000","last_peered":"2024-01-15T18:20:34.355545+0000","last_clean":"2024-01-15T18:20:34.355545+0000","last_became_active":"2024-01-15T18:14:16.874378+0000","last_became_peered":"2024-01-15T18:14:16.874378+0000","last_unstale":"2024-01-15T18:20:34.355545+0000","last_undegraded":"2024-01-15T18:20:34.355545+0000","last_fullsized":"2024-01-15T18:20:34.355545+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:34.355384+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:34.355384+0000","last_clean_scrub_stamp":"2024-01-15T18:20:34.355384+0000","objects_scrubbed":16,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:24.017915+0000","scrub_duration":0.200285095,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.769090+0000","last_change":"2024-01-15T18:20:02.769178+0000","last_active":"2024-01-15T18:20:02.769090+0000","last_peered":"2024-01-15T18:20:02.769090+0000","last_clean":"2024-01-15T18:20:02.769090+0000","last_became_active":"2024-01-15T18:14:15.863418+0000","last_became_peered":"2024-01-15T18:14:15.863418+0000","last_unstale":"2024-01-15T18:20:02.769090+0000","last_undegraded":"2024-01-15T18:20:02.769090+0000","last_fullsized":"2024-01-15T18:20:02.769090+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.768819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.768819+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.768819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:19:01.570006+0000","scrub_duration":0.0044113540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"116'50","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:31.336324+0000","last_change":"2024-01-15T18:20:31.336395+0000","last_active":"2024-01-15T18:20:31.336324+0000","last_peered":"2024-01-15T18:20:31.336324+0000","last_clean":"2024-01-15T18:20:31.336324+0000","last_became_active":"2024-01-15T18:14:16.877553+0000","last_became_peered":"2024-01-15T18:14:16.877553+0000","last_unstale":"2024-01-15T18:20:31.336324+0000","last_undegraded":"2024-01-15T18:20:31.336324+0000","last_fullsized":"2024-01-15T18:20:31.336324+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'50","last_scrub_stamp":"2024-01-15T18:20:31.336136+0000","last_deep_scrub":"116'50","last_deep_scrub_stamp":"2024-01-15T18:20:31.336136+0000","last_clean_scrub_stamp":"2024-01-15T18:20:31.336136+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:08:24.182918+0000","scrub_duration":0.091319631999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":8,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"116'1","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.364660+0000","last_change":"2024-01-15T18:20:19.364715+0000","last_active":"2024-01-15T18:20:19.364660+0000","last_peered":"2024-01-15T18:20:19.364660+0000","last_clean":"2024-01-15T18:20:19.364660+0000","last_became_active":"2024-01-15T18:14:15.875800+0000","last_became_peered":"2024-01-15T18:14:15.875800+0000","last_unstale":"2024-01-15T18:20:19.364660+0000","last_undegraded":"2024-01-15T18:20:19.364660+0000","last_fullsized":"2024-01-15T18:20:19.364660+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'1","last_scrub_stamp":"2024-01-15T18:20:19.364443+0000","last_deep_scrub":"116'1","last_deep_scrub_stamp":"2024-01-15T18:20:19.364443+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.364443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:20:46.747691+0000","scrub_duration":0.003611015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"116'58","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:27.163881+0000","last_change":"2024-01-15T18:20:27.163950+0000","last_active":"2024-01-15T18:20:27.163881+0000","last_peered":"2024-01-15T18:20:27.163881+0000","last_clean":"2024-01-15T18:20:27.163881+0000","last_became_active":"2024-01-15T18:14:16.893762+0000","last_became_peered":"2024-01-15T18:14:16.893762+0000","last_unstale":"2024-01-15T18:20:27.163881+0000","last_undegraded":"2024-01-15T18:20:27.163881+0000","last_fullsized":"2024-01-15T18:20:27.163881+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:27.163745+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:27.163745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:27.163745+0000","objects_scrubbed":18,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:53:49.700312+0000","scrub_duration":0.20644792100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.421790+0000","last_change":"2024-01-15T18:20:05.421839+0000","last_active":"2024-01-15T18:20:05.421790+0000","last_peered":"2024-01-15T18:20:05.421790+0000","last_clean":"2024-01-15T18:20:05.421790+0000","last_became_active":"2024-01-15T18:14:15.877363+0000","last_became_peered":"2024-01-15T18:14:15.877363+0000","last_unstale":"2024-01-15T18:20:05.421790+0000","last_undegraded":"2024-01-15T18:20:05.421790+0000","last_fullsized":"2024-01-15T18:20:05.421790+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:05.421607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:05.421607+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.421607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:33.730233+0000","scrub_duration":0.0039450379999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"116'63","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:10.661978+0000","last_change":"2024-01-15T18:20:10.662064+0000","last_active":"2024-01-15T18:20:10.661978+0000","last_peered":"2024-01-15T18:20:10.661978+0000","last_clean":"2024-01-15T18:20:10.661978+0000","last_became_active":"2024-01-15T18:14:16.867355+0000","last_became_peered":"2024-01-15T18:14:16.867355+0000","last_unstale":"2024-01-15T18:20:10.661978+0000","last_undegraded":"2024-01-15T18:20:10.661978+0000","last_fullsized":"2024-01-15T18:20:10.661978+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:10.661798+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:10.661798+0000","last_clean_scrub_stamp":"2024-01-15T18:20:10.661798+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:55:24.562142+0000","scrub_duration":0.234248187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"116'15","reported_seq":40,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.785377+0000","last_change":"2024-01-15T18:20:03.785437+0000","last_active":"2024-01-15T18:20:03.785377+0000","last_peered":"2024-01-15T18:20:03.785377+0000","last_clean":"2024-01-15T18:20:03.785377+0000","last_became_active":"2024-01-15T18:14:15.871132+0000","last_became_peered":"2024-01-15T18:14:15.871132+0000","last_unstale":"2024-01-15T18:20:03.785377+0000","last_undegraded":"2024-01-15T18:20:03.785377+0000","last_fullsized":"2024-01-15T18:20:03.785377+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'15","last_scrub_stamp":"2024-01-15T18:20:03.785184+0000","last_deep_scrub":"116'15","last_deep_scrub_stamp":"2024-01-15T18:20:03.785184+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.785184+0000","objects_scrubbed":2,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T06:03:53.983142+0000","scrub_duration":0.0088990630000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"116'63","reported_seq":95,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:24.461850+0000","last_change":"2024-01-15T18:20:24.461923+0000","last_active":"2024-01-15T18:20:24.461850+0000","last_peered":"2024-01-15T18:20:24.461850+0000","last_clean":"2024-01-15T18:20:24.461850+0000","last_became_active":"2024-01-15T18:14:16.883052+0000","last_became_peered":"2024-01-15T18:14:16.883052+0000","last_unstale":"2024-01-15T18:20:24.461850+0000","last_undegraded":"2024-01-15T18:20:24.461850+0000","last_fullsized":"2024-01-15T18:20:24.461850+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:24.461726+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:24.461726+0000","last_clean_scrub_stamp":"2024-01-15T18:20:24.461726+0000","objects_scrubbed":20,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:54.635793+0000","scrub_duration":0.25293763800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"116'195","reported_seq":220,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:48.865015+0000","last_change":"2024-01-15T18:20:01.790145+0000","last_active":"2024-01-15T18:20:48.865015+0000","last_peered":"2024-01-15T18:20:48.865015+0000","last_clean":"2024-01-15T18:20:48.865015+0000","last_became_active":"2024-01-15T18:14:15.866053+0000","last_became_peered":"2024-01-15T18:14:15.866053+0000","last_unstale":"2024-01-15T18:20:48.865015+0000","last_undegraded":"2024-01-15T18:20:48.865015+0000","last_fullsized":"2024-01-15T18:20:48.865015+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'193","last_scrub_stamp":"2024-01-15T18:20:01.789874+0000","last_deep_scrub":"116'193","last_deep_scrub_stamp":"2024-01-15T18:20:01.789874+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.789874+0000","objects_scrubbed":4,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:42:22.469663+0000","scrub_duration":0.01365885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":644404,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":192,"num_write_kb":723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"116'57","reported_seq":87,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.643433+0000","last_change":"2024-01-15T18:20:08.643515+0000","last_active":"2024-01-15T18:20:08.643433+0000","last_peered":"2024-01-15T18:20:08.643433+0000","last_clean":"2024-01-15T18:20:08.643433+0000","last_became_active":"2024-01-15T18:14:16.865660+0000","last_became_peered":"2024-01-15T18:14:16.865660+0000","last_unstale":"2024-01-15T18:20:08.643433+0000","last_undegraded":"2024-01-15T18:20:08.643433+0000","last_fullsized":"2024-01-15T18:20:08.643433+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'57","last_scrub_stamp":"2024-01-15T18:20:08.643249+0000","last_deep_scrub":"116'57","last_deep_scrub_stamp":"2024-01-15T18:20:08.643249+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.643249+0000","objects_scrubbed":21,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:08.158988+0000","scrub_duration":0.22645187899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.780476+0000","last_change":"2024-01-15T18:20:03.780574+0000","last_active":"2024-01-15T18:20:03.780476+0000","last_peered":"2024-01-15T18:20:03.780476+0000","last_clean":"2024-01-15T18:20:03.780476+0000","last_became_active":"2024-01-15T18:14:15.873011+0000","last_became_peered":"2024-01-15T18:14:15.873011+0000","last_unstale":"2024-01-15T18:20:03.780476+0000","last_undegraded":"2024-01-15T18:20:03.780476+0000","last_fullsized":"2024-01-15T18:20:03.780476+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:03.780315+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:03.780315+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.780315+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:57:45.825709+0000","scrub_duration":0.0083488959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"116'71","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:15.624139+0000","last_change":"2024-01-15T18:20:15.624238+0000","last_active":"2024-01-15T18:20:15.624139+0000","last_peered":"2024-01-15T18:20:15.624139+0000","last_clean":"2024-01-15T18:20:15.624139+0000","last_became_active":"2024-01-15T18:14:16.881461+0000","last_became_peered":"2024-01-15T18:14:16.881461+0000","last_unstale":"2024-01-15T18:20:15.624139+0000","last_undegraded":"2024-01-15T18:20:15.624139+0000","last_fullsized":"2024-01-15T18:20:15.624139+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'71","last_scrub_stamp":"2024-01-15T18:20:15.623953+0000","last_deep_scrub":"116'71","last_deep_scrub_stamp":"2024-01-15T18:20:15.623953+0000","last_clean_scrub_stamp":"2024-01-15T18:20:15.623953+0000","objects_scrubbed":21,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:00:43.042056+0000","scrub_duration":0.25260574800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.908262+0000","last_change":"2024-01-15T18:20:04.908358+0000","last_active":"2024-01-15T18:20:04.908262+0000","last_peered":"2024-01-15T18:20:04.908262+0000","last_clean":"2024-01-15T18:20:04.908262+0000","last_became_active":"2024-01-15T18:14:15.878086+0000","last_became_peered":"2024-01-15T18:14:15.878086+0000","last_unstale":"2024-01-15T18:20:04.908262+0000","last_undegraded":"2024-01-15T18:20:04.908262+0000","last_fullsized":"2024-01-15T18:20:04.908262+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:04.908067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:04.908067+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.908067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:21:57.372868+0000","scrub_duration":0.0042479759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"116'68","reported_seq":100,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.679799+0000","last_change":"2024-01-15T18:20:09.679858+0000","last_active":"2024-01-15T18:20:09.679799+0000","last_peered":"2024-01-15T18:20:09.679799+0000","last_clean":"2024-01-15T18:20:09.679799+0000","last_became_active":"2024-01-15T18:14:16.869259+0000","last_became_peered":"2024-01-15T18:14:16.869259+0000","last_unstale":"2024-01-15T18:20:09.679799+0000","last_undegraded":"2024-01-15T18:20:09.679799+0000","last_fullsized":"2024-01-15T18:20:09.679799+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'68","last_scrub_stamp":"2024-01-15T18:20:09.679609+0000","last_deep_scrub":"116'68","last_deep_scrub_stamp":"2024-01-15T18:20:09.679609+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.679609+0000","objects_scrubbed":25,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:18.637167+0000","scrub_duration":0.26166856799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":25,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"116'12","reported_seq":37,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:00.222842+0000","last_change":"2024-01-15T18:20:00.222891+0000","last_active":"2024-01-15T18:20:00.222842+0000","last_peered":"2024-01-15T18:20:00.222842+0000","last_clean":"2024-01-15T18:20:00.222842+0000","last_became_active":"2024-01-15T18:14:15.874618+0000","last_became_peered":"2024-01-15T18:14:15.874618+0000","last_unstale":"2024-01-15T18:20:00.222842+0000","last_undegraded":"2024-01-15T18:20:00.222842+0000","last_fullsized":"2024-01-15T18:20:00.222842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:20:00.222745+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:20:00.222745+0000","last_clean_scrub_stamp":"2024-01-15T18:20:00.222745+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:45:51.445709+0000","scrub_duration":0.010058510999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"116'34","reported_seq":62,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:28.442016+0000","last_change":"2024-01-15T18:20:28.442084+0000","last_active":"2024-01-15T18:20:28.442016+0000","last_peered":"2024-01-15T18:20:28.442016+0000","last_clean":"2024-01-15T18:20:28.442016+0000","last_became_active":"2024-01-15T18:14:16.892690+0000","last_became_peered":"2024-01-15T18:14:16.892690+0000","last_unstale":"2024-01-15T18:20:28.442016+0000","last_undegraded":"2024-01-15T18:20:28.442016+0000","last_fullsized":"2024-01-15T18:20:28.442016+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'34","last_scrub_stamp":"2024-01-15T18:20:28.441883+0000","last_deep_scrub":"116'34","last_deep_scrub_stamp":"2024-01-15T18:20:28.441883+0000","last_clean_scrub_stamp":"2024-01-15T18:20:28.441883+0000","objects_scrubbed":10,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:46:23.267358+0000","scrub_duration":0.104048134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":10,"num_object_clones":0,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":10,"num_write_kb":40960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"116'2","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.745018+0000","last_change":"2024-01-15T18:20:06.745073+0000","last_active":"2024-01-15T18:20:06.745018+0000","last_peered":"2024-01-15T18:20:06.745018+0000","last_clean":"2024-01-15T18:20:06.745018+0000","last_became_active":"2024-01-15T18:14:15.871353+0000","last_became_peered":"2024-01-15T18:14:15.871353+0000","last_unstale":"2024-01-15T18:20:06.745018+0000","last_undegraded":"2024-01-15T18:20:06.745018+0000","last_fullsized":"2024-01-15T18:20:06.745018+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-01-15T18:20:06.744839+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-01-15T18:20:06.744839+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.744839+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:58:35.444023+0000","scrub_duration":0.0075306430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"116'53","reported_seq":88,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:24.045496+0000","last_change":"2024-01-15T18:20:24.045646+0000","last_active":"2024-01-15T18:20:24.045496+0000","last_peered":"2024-01-15T18:20:24.045496+0000","last_clean":"2024-01-15T18:20:24.045496+0000","last_became_active":"2024-01-15T18:14:16.891593+0000","last_became_peered":"2024-01-15T18:14:16.891593+0000","last_unstale":"2024-01-15T18:20:24.045496+0000","last_undegraded":"2024-01-15T18:20:24.045496+0000","last_fullsized":"2024-01-15T18:20:24.045496+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'53","last_scrub_stamp":"2024-01-15T18:20:24.045217+0000","last_deep_scrub":"116'53","last_deep_scrub_stamp":"2024-01-15T18:20:24.045217+0000","last_clean_scrub_stamp":"2024-01-15T18:20:24.045217+0000","objects_scrubbed":17,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:43:22.250266+0000","scrub_duration":0.17771706600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"116'3","reported_seq":28,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:09.362044+0000","last_change":"2024-01-15T18:20:09.362155+0000","last_active":"2024-01-15T18:20:09.362044+0000","last_peered":"2024-01-15T18:20:09.362044+0000","last_clean":"2024-01-15T18:20:09.362044+0000","last_became_active":"2024-01-15T18:14:15.872174+0000","last_became_peered":"2024-01-15T18:14:15.872174+0000","last_unstale":"2024-01-15T18:20:09.362044+0000","last_undegraded":"2024-01-15T18:20:09.362044+0000","last_fullsized":"2024-01-15T18:20:09.362044+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:09.361834+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:09.361834+0000","last_clean_scrub_stamp":"2024-01-15T18:20:09.361834+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:20:42.293371+0000","scrub_duration":0.0070512709999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"116'59","reported_seq":94,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.048473+0000","last_change":"2024-01-15T18:20:06.048581+0000","last_active":"2024-01-15T18:20:06.048473+0000","last_peered":"2024-01-15T18:20:06.048473+0000","last_clean":"2024-01-15T18:20:06.048473+0000","last_became_active":"2024-01-15T18:14:16.894356+0000","last_became_peered":"2024-01-15T18:14:16.894356+0000","last_unstale":"2024-01-15T18:20:06.048473+0000","last_undegraded":"2024-01-15T18:20:06.048473+0000","last_fullsized":"2024-01-15T18:20:06.048473+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:06.048300+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:06.048300+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.048300+0000","objects_scrubbed":18,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:55:34.356925+0000","scrub_duration":0.18916992199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":22,"num_write_kb":73730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"116'2","reported_seq":27,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.904337+0000","last_change":"2024-01-15T18:20:07.904387+0000","last_active":"2024-01-15T18:20:07.904337+0000","last_peered":"2024-01-15T18:20:07.904337+0000","last_clean":"2024-01-15T18:20:07.904337+0000","last_became_active":"2024-01-15T18:14:15.874212+0000","last_became_peered":"2024-01-15T18:14:15.874212+0000","last_unstale":"2024-01-15T18:20:07.904337+0000","last_undegraded":"2024-01-15T18:20:07.904337+0000","last_fullsized":"2024-01-15T18:20:07.904337+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'2","last_scrub_stamp":"2024-01-15T18:20:07.904106+0000","last_deep_scrub":"116'2","last_deep_scrub_stamp":"2024-01-15T18:20:07.904106+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.904106+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:45.760496+0000","scrub_duration":0.0084473900000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"116'63","reported_seq":99,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.038371+0000","last_change":"2024-01-15T18:20:23.038437+0000","last_active":"2024-01-15T18:20:23.038371+0000","last_peered":"2024-01-15T18:20:23.038371+0000","last_clean":"2024-01-15T18:20:23.038371+0000","last_became_active":"2024-01-15T18:14:16.887967+0000","last_became_peered":"2024-01-15T18:14:16.887967+0000","last_unstale":"2024-01-15T18:20:23.038371+0000","last_undegraded":"2024-01-15T18:20:23.038371+0000","last_fullsized":"2024-01-15T18:20:23.038371+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:23.038283+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:23.038283+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.038283+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:28:55.265617+0000","scrub_duration":0.25342030799999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:06.399525+0000","last_change":"2024-01-15T18:20:06.399574+0000","last_active":"2024-01-15T18:20:06.399525+0000","last_peered":"2024-01-15T18:20:06.399525+0000","last_clean":"2024-01-15T18:20:06.399525+0000","last_became_active":"2024-01-15T18:14:15.875742+0000","last_became_peered":"2024-01-15T18:14:15.875742+0000","last_unstale":"2024-01-15T18:20:06.399525+0000","last_undegraded":"2024-01-15T18:20:06.399525+0000","last_fullsized":"2024-01-15T18:20:06.399525+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:06.399262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:06.399262+0000","last_clean_scrub_stamp":"2024-01-15T18:20:06.399262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:09:44.431017+0000","scrub_duration":0.0039161960000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"116'64","reported_seq":98,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.204980+0000","last_change":"2024-01-15T18:20:07.204980+0000","last_active":"2024-01-15T18:20:07.204980+0000","last_peered":"2024-01-15T18:20:07.204980+0000","last_clean":"2024-01-15T18:20:07.204980+0000","last_became_active":"2024-01-15T18:14:16.886102+0000","last_became_peered":"2024-01-15T18:14:16.886102+0000","last_unstale":"2024-01-15T18:20:07.204980+0000","last_undegraded":"2024-01-15T18:20:07.204980+0000","last_fullsized":"2024-01-15T18:20:07.204980+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'64","last_scrub_stamp":"2024-01-15T18:20:07.204595+0000","last_deep_scrub":"116'64","last_deep_scrub_stamp":"2024-01-15T18:20:07.204595+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.204595+0000","objects_scrubbed":30,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:19:38.823285+0000","scrub_duration":0.36587318699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125829120,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":30,"num_write_kb":122880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"116'4","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.818977+0000","last_change":"2024-01-15T18:20:02.819041+0000","last_active":"2024-01-15T18:20:02.818977+0000","last_peered":"2024-01-15T18:20:02.818977+0000","last_clean":"2024-01-15T18:20:02.818977+0000","last_became_active":"2024-01-15T18:14:15.871398+0000","last_became_peered":"2024-01-15T18:14:15.871398+0000","last_unstale":"2024-01-15T18:20:02.818977+0000","last_undegraded":"2024-01-15T18:20:02.818977+0000","last_fullsized":"2024-01-15T18:20:02.818977+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'4","last_scrub_stamp":"2024-01-15T18:20:02.818798+0000","last_deep_scrub":"116'4","last_deep_scrub_stamp":"2024-01-15T18:20:02.818798+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.818798+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:08:10.712736+0000","scrub_duration":0.0072655630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"116'62","reported_seq":102,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:33.517403+0000","last_change":"2024-01-15T18:20:33.517489+0000","last_active":"2024-01-15T18:20:33.517403+0000","last_peered":"2024-01-15T18:20:33.517403+0000","last_clean":"2024-01-15T18:20:33.517403+0000","last_became_active":"2024-01-15T18:15:19.822315+0000","last_became_peered":"2024-01-15T18:15:19.822315+0000","last_unstale":"2024-01-15T18:20:33.517403+0000","last_undegraded":"2024-01-15T18:20:33.517403+0000","last_fullsized":"2024-01-15T18:20:33.517403+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'62","last_scrub_stamp":"2024-01-15T18:20:33.517228+0000","last_deep_scrub":"116'62","last_deep_scrub_stamp":"2024-01-15T18:20:33.517228+0000","last_clean_scrub_stamp":"2024-01-15T18:20:33.517228+0000","objects_scrubbed":19,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:46:49.734656+0000","scrub_duration":0.204264011,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.258363+0000","last_change":"2024-01-15T18:20:02.258363+0000","last_active":"2024-01-15T18:20:02.258363+0000","last_peered":"2024-01-15T18:20:02.258363+0000","last_clean":"2024-01-15T18:20:02.258363+0000","last_became_active":"2024-01-15T18:14:15.874393+0000","last_became_peered":"2024-01-15T18:14:15.874393+0000","last_unstale":"2024-01-15T18:20:02.258363+0000","last_undegraded":"2024-01-15T18:20:02.258363+0000","last_fullsized":"2024-01-15T18:20:02.258363+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.258117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.258117+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.258117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:24:31.738131+0000","scrub_duration":0.003674961,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"116'61","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:31.013895+0000","last_change":"2024-01-15T18:20:31.013895+0000","last_active":"2024-01-15T18:20:31.013895+0000","last_peered":"2024-01-15T18:20:31.013895+0000","last_clean":"2024-01-15T18:20:31.013895+0000","last_became_active":"2024-01-15T18:14:16.887128+0000","last_became_peered":"2024-01-15T18:14:16.887128+0000","last_unstale":"2024-01-15T18:20:31.013895+0000","last_undegraded":"2024-01-15T18:20:31.013895+0000","last_fullsized":"2024-01-15T18:20:31.013895+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'61","last_scrub_stamp":"2024-01-15T18:20:31.013515+0000","last_deep_scrub":"116'61","last_deep_scrub_stamp":"2024-01-15T18:20:31.013515+0000","last_clean_scrub_stamp":"2024-01-15T18:20:31.013515+0000","objects_scrubbed":17,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:28:33.562019+0000","scrub_duration":0.19102684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.795153+0000","last_change":"2024-01-15T18:19:58.795153+0000","last_active":"2024-01-15T18:19:58.795153+0000","last_peered":"2024-01-15T18:19:58.795153+0000","last_clean":"2024-01-15T18:19:58.795153+0000","last_became_active":"2024-01-15T18:14:15.869306+0000","last_became_peered":"2024-01-15T18:14:15.869306+0000","last_unstale":"2024-01-15T18:19:58.795153+0000","last_undegraded":"2024-01-15T18:19:58.795153+0000","last_fullsized":"2024-01-15T18:19:58.795153+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:19:58.795017+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.795017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:44:11.531385+0000","scrub_duration":0.0028610760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"116'59","reported_seq":91,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:18.536640+0000","last_change":"2024-01-15T18:20:18.536640+0000","last_active":"2024-01-15T18:20:18.536640+0000","last_peered":"2024-01-15T18:20:18.536640+0000","last_clean":"2024-01-15T18:20:18.536640+0000","last_became_active":"2024-01-15T18:14:16.888237+0000","last_became_peered":"2024-01-15T18:14:16.888237+0000","last_unstale":"2024-01-15T18:20:18.536640+0000","last_undegraded":"2024-01-15T18:20:18.536640+0000","last_fullsized":"2024-01-15T18:20:18.536640+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:18.536416+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:18.536416+0000","last_clean_scrub_stamp":"2024-01-15T18:20:18.536416+0000","objects_scrubbed":17,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:20:58.867375+0000","scrub_duration":0.181046237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67109125,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:05.721465+0000","last_change":"2024-01-15T18:20:05.721465+0000","last_active":"2024-01-15T18:20:05.721465+0000","last_peered":"2024-01-15T18:20:05.721465+0000","last_clean":"2024-01-15T18:20:05.721465+0000","last_became_active":"2024-01-15T18:14:15.875840+0000","last_became_peered":"2024-01-15T18:14:15.875840+0000","last_unstale":"2024-01-15T18:20:05.721465+0000","last_undegraded":"2024-01-15T18:20:05.721465+0000","last_fullsized":"2024-01-15T18:20:05.721465+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:05.721271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:05.721271+0000","last_clean_scrub_stamp":"2024-01-15T18:20:05.721271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:18:07.002373+0000","scrub_duration":0.003099302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"116'49","reported_seq":78,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:23.022742+0000","last_change":"2024-01-15T18:20:23.022742+0000","last_active":"2024-01-15T18:20:23.022742+0000","last_peered":"2024-01-15T18:20:23.022742+0000","last_clean":"2024-01-15T18:20:23.022742+0000","last_became_active":"2024-01-15T18:14:16.893093+0000","last_became_peered":"2024-01-15T18:14:16.893093+0000","last_unstale":"2024-01-15T18:20:23.022742+0000","last_undegraded":"2024-01-15T18:20:23.022742+0000","last_fullsized":"2024-01-15T18:20:23.022742+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'49","last_scrub_stamp":"2024-01-15T18:20:23.022519+0000","last_deep_scrub":"116'49","last_deep_scrub_stamp":"2024-01-15T18:20:23.022519+0000","last_clean_scrub_stamp":"2024-01-15T18:20:23.022519+0000","objects_scrubbed":13,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:00:31.980076+0000","scrub_duration":0.16129085500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"116'3","reported_seq":29,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.509754+0000","last_change":"2024-01-15T18:20:03.509754+0000","last_active":"2024-01-15T18:20:03.509754+0000","last_peered":"2024-01-15T18:20:03.509754+0000","last_clean":"2024-01-15T18:20:03.509754+0000","last_became_active":"2024-01-15T18:14:15.858507+0000","last_became_peered":"2024-01-15T18:14:15.858507+0000","last_unstale":"2024-01-15T18:20:03.509754+0000","last_undegraded":"2024-01-15T18:20:03.509754+0000","last_fullsized":"2024-01-15T18:20:03.509754+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'3","last_scrub_stamp":"2024-01-15T18:20:03.509469+0000","last_deep_scrub":"116'3","last_deep_scrub_stamp":"2024-01-15T18:20:03.509469+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.509469+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:35:26.140490+0000","scrub_duration":0.0069296399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"116'63","reported_seq":112,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:40.356329+0000","last_change":"2024-01-15T18:20:40.356486+0000","last_active":"2024-01-15T18:20:40.356329+0000","last_peered":"2024-01-15T18:20:40.356329+0000","last_clean":"2024-01-15T18:20:40.356329+0000","last_became_active":"2024-01-15T18:15:19.822973+0000","last_became_peered":"2024-01-15T18:15:19.822973+0000","last_unstale":"2024-01-15T18:20:40.356329+0000","last_undegraded":"2024-01-15T18:20:40.356329+0000","last_fullsized":"2024-01-15T18:20:40.356329+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'63","last_scrub_stamp":"2024-01-15T18:20:40.356176+0000","last_deep_scrub":"116'63","last_deep_scrub_stamp":"2024-01-15T18:20:40.356176+0000","last_clean_scrub_stamp":"2024-01-15T18:20:40.356176+0000","objects_scrubbed":22,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:43:11.163077+0000","scrub_duration":0.26414565699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":58720256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"116'19","reported_seq":45,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:19.852751+0000","last_change":"2024-01-15T18:20:19.852751+0000","last_active":"2024-01-15T18:20:19.852751+0000","last_peered":"2024-01-15T18:20:19.852751+0000","last_clean":"2024-01-15T18:20:19.852751+0000","last_became_active":"2024-01-15T18:14:15.875688+0000","last_became_peered":"2024-01-15T18:14:15.875688+0000","last_unstale":"2024-01-15T18:20:19.852751+0000","last_undegraded":"2024-01-15T18:20:19.852751+0000","last_fullsized":"2024-01-15T18:20:19.852751+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'19","last_scrub_stamp":"2024-01-15T18:20:19.852492+0000","last_deep_scrub":"116'19","last_deep_scrub_stamp":"2024-01-15T18:20:19.852492+0000","last_clean_scrub_stamp":"2024-01-15T18:20:19.852492+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T19:50:31.239558+0000","scrub_duration":0.0039531749999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":40,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506,"num_omap_keys":10,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"116'66","reported_seq":105,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:57.918788+0000","last_change":"2024-01-15T18:19:57.918788+0000","last_active":"2024-01-15T18:19:57.918788+0000","last_peered":"2024-01-15T18:19:57.918788+0000","last_clean":"2024-01-15T18:19:57.918788+0000","last_became_active":"2024-01-15T18:14:16.882994+0000","last_became_peered":"2024-01-15T18:14:16.882994+0000","last_unstale":"2024-01-15T18:19:57.918788+0000","last_undegraded":"2024-01-15T18:19:57.918788+0000","last_fullsized":"2024-01-15T18:19:57.918788+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'66","last_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_deep_scrub":"116'66","last_deep_scrub_stamp":"2024-01-15T18:19:57.918639+0000","last_clean_scrub_stamp":"2024-01-15T18:19:57.918639+0000","objects_scrubbed":24,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:02:38.022160+0000","scrub_duration":0.15638939399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:04.767210+0000","last_change":"2024-01-15T18:20:04.767210+0000","last_active":"2024-01-15T18:20:04.767210+0000","last_peered":"2024-01-15T18:20:04.767210+0000","last_clean":"2024-01-15T18:20:04.767210+0000","last_became_active":"2024-01-15T18:14:15.876149+0000","last_became_peered":"2024-01-15T18:14:15.876149+0000","last_unstale":"2024-01-15T18:20:04.767210+0000","last_undegraded":"2024-01-15T18:20:04.767210+0000","last_fullsized":"2024-01-15T18:20:04.767210+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:04.766974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:04.766974+0000","last_clean_scrub_stamp":"2024-01-15T18:20:04.766974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T01:56:58.490311+0000","scrub_duration":0.0038318369999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"116'58","reported_seq":89,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:22.058910+0000","last_change":"2024-01-15T18:20:22.058910+0000","last_active":"2024-01-15T18:20:22.058910+0000","last_peered":"2024-01-15T18:20:22.058910+0000","last_clean":"2024-01-15T18:20:22.058910+0000","last_became_active":"2024-01-15T18:14:16.892327+0000","last_became_peered":"2024-01-15T18:14:16.892327+0000","last_unstale":"2024-01-15T18:20:22.058910+0000","last_undegraded":"2024-01-15T18:20:22.058910+0000","last_fullsized":"2024-01-15T18:20:22.058910+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'58","last_scrub_stamp":"2024-01-15T18:20:22.058626+0000","last_deep_scrub":"116'58","last_deep_scrub_stamp":"2024-01-15T18:20:22.058626+0000","last_clean_scrub_stamp":"2024-01-15T18:20:22.058626+0000","objects_scrubbed":19,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:39:21.574364+0000","scrub_duration":0.19533523999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303290,"num_objects":19,"num_object_clones":1,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.514236+0000","last_change":"2024-01-15T18:20:01.514236+0000","last_active":"2024-01-15T18:20:01.514236+0000","last_peered":"2024-01-15T18:20:01.514236+0000","last_clean":"2024-01-15T18:20:01.514236+0000","last_became_active":"2024-01-15T18:14:15.868219+0000","last_became_peered":"2024-01-15T18:14:15.868219+0000","last_unstale":"2024-01-15T18:20:01.514236+0000","last_undegraded":"2024-01-15T18:20:01.514236+0000","last_fullsized":"2024-01-15T18:20:01.514236+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:01.514092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:01.514092+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.514092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T18:31:26.265860+0000","scrub_duration":0.0027639349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"116'60","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:07.337206+0000","last_change":"2024-01-15T18:20:07.337206+0000","last_active":"2024-01-15T18:20:07.337206+0000","last_peered":"2024-01-15T18:20:07.337206+0000","last_clean":"2024-01-15T18:20:07.337206+0000","last_became_active":"2024-01-15T18:14:16.867671+0000","last_became_peered":"2024-01-15T18:14:16.867671+0000","last_unstale":"2024-01-15T18:20:07.337206+0000","last_undegraded":"2024-01-15T18:20:07.337206+0000","last_fullsized":"2024-01-15T18:20:07.337206+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'60","last_scrub_stamp":"2024-01-15T18:20:07.336876+0000","last_deep_scrub":"116'60","last_deep_scrub_stamp":"2024-01-15T18:20:07.336876+0000","last_clean_scrub_stamp":"2024-01-15T18:20:07.336876+0000","objects_scrubbed":16,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T21:24:52.572228+0000","scrub_duration":0.19744029199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:08.373842+0000","last_change":"2024-01-15T18:20:08.373842+0000","last_active":"2024-01-15T18:20:08.373842+0000","last_peered":"2024-01-15T18:20:08.373842+0000","last_clean":"2024-01-15T18:20:08.373842+0000","last_became_active":"2024-01-15T18:14:15.867089+0000","last_became_peered":"2024-01-15T18:14:15.867089+0000","last_unstale":"2024-01-15T18:20:08.373842+0000","last_undegraded":"2024-01-15T18:20:08.373842+0000","last_fullsized":"2024-01-15T18:20:08.373842+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:08.373617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:08.373617+0000","last_clean_scrub_stamp":"2024-01-15T18:20:08.373617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:51:55.243354+0000","scrub_duration":0.0034628670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"116'80","reported_seq":114,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:12.520568+0000","last_change":"2024-01-15T18:20:12.520568+0000","last_active":"2024-01-15T18:20:12.520568+0000","last_peered":"2024-01-15T18:20:12.520568+0000","last_clean":"2024-01-15T18:20:12.520568+0000","last_became_active":"2024-01-15T18:14:16.878580+0000","last_became_peered":"2024-01-15T18:14:16.878580+0000","last_unstale":"2024-01-15T18:20:12.520568+0000","last_undegraded":"2024-01-15T18:20:12.520568+0000","last_fullsized":"2024-01-15T18:20:12.520568+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'80","last_scrub_stamp":"2024-01-15T18:20:12.520296+0000","last_deep_scrub":"116'80","last_deep_scrub_stamp":"2024-01-15T18:20:12.520296+0000","last_clean_scrub_stamp":"2024-01-15T18:20:12.520296+0000","objects_scrubbed":29,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T22:59:06.085847+0000","scrub_duration":0.32665812100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":121634816,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":29,"num_write_kb":118784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:03.144631+0000","last_change":"2024-01-15T18:20:03.144631+0000","last_active":"2024-01-15T18:20:03.144631+0000","last_peered":"2024-01-15T18:20:03.144631+0000","last_clean":"2024-01-15T18:20:03.144631+0000","last_became_active":"2024-01-15T18:14:15.872919+0000","last_became_peered":"2024-01-15T18:14:15.872919+0000","last_unstale":"2024-01-15T18:20:03.144631+0000","last_undegraded":"2024-01-15T18:20:03.144631+0000","last_fullsized":"2024-01-15T18:20:03.144631+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:03.144267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:03.144267+0000","last_clean_scrub_stamp":"2024-01-15T18:20:03.144267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T23:28:45.685511+0000","scrub_duration":0.0035700250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"116'72","reported_seq":106,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:34.219295+0000","last_change":"2024-01-15T18:20:34.219329+0000","last_active":"2024-01-15T18:20:34.219295+0000","last_peered":"2024-01-15T18:20:34.219295+0000","last_clean":"2024-01-15T18:20:34.219295+0000","last_became_active":"2024-01-15T18:14:16.887485+0000","last_became_peered":"2024-01-15T18:14:16.887485+0000","last_unstale":"2024-01-15T18:20:34.219295+0000","last_undegraded":"2024-01-15T18:20:34.219295+0000","last_fullsized":"2024-01-15T18:20:34.219295+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'72","last_scrub_stamp":"2024-01-15T18:20:34.219211+0000","last_deep_scrub":"116'72","last_deep_scrub_stamp":"2024-01-15T18:20:34.219211+0000","last_clean_scrub_stamp":"2024-01-15T18:20:34.219211+0000","objects_scrubbed":26,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:49:14.097860+0000","scrub_duration":0.31754085300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"116'12","reported_seq":48,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:19:58.182868+0000","last_change":"2024-01-15T18:19:58.182868+0000","last_active":"2024-01-15T18:19:58.182868+0000","last_peered":"2024-01-15T18:19:58.182868+0000","last_clean":"2024-01-15T18:19:58.182868+0000","last_became_active":"2024-01-15T18:14:15.875397+0000","last_became_peered":"2024-01-15T18:14:15.875397+0000","last_unstale":"2024-01-15T18:19:58.182868+0000","last_undegraded":"2024-01-15T18:19:58.182868+0000","last_fullsized":"2024-01-15T18:19:58.182868+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'12","last_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_deep_scrub":"116'12","last_deep_scrub_stamp":"2024-01-15T18:19:58.182737+0000","last_clean_scrub_stamp":"2024-01-15T18:19:58.182737+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T00:33:12.765302+0000","scrub_duration":0.0054820609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1321,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"116'59","reported_seq":92,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:12.977682+0000","last_change":"2024-01-15T18:20:12.977682+0000","last_active":"2024-01-15T18:20:12.977682+0000","last_peered":"2024-01-15T18:20:12.977682+0000","last_clean":"2024-01-15T18:20:12.977682+0000","last_became_active":"2024-01-15T18:14:16.882189+0000","last_became_peered":"2024-01-15T18:14:16.882189+0000","last_unstale":"2024-01-15T18:20:12.977682+0000","last_undegraded":"2024-01-15T18:20:12.977682+0000","last_fullsized":"2024-01-15T18:20:12.977682+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'59","last_scrub_stamp":"2024-01-15T18:20:12.977304+0000","last_deep_scrub":"116'59","last_deep_scrub_stamp":"2024-01-15T18:20:12.977304+0000","last_clean_scrub_stamp":"2024-01-15T18:20:12.977304+0000","objects_scrubbed":23,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T03:05:54.139872+0000","scrub_duration":0.23080210500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":26,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:10.753379+0000","last_change":"2024-01-15T18:20:10.753379+0000","last_active":"2024-01-15T18:20:10.753379+0000","last_peered":"2024-01-15T18:20:10.753379+0000","last_clean":"2024-01-15T18:20:10.753379+0000","last_became_active":"2024-01-15T18:14:15.876181+0000","last_became_peered":"2024-01-15T18:14:15.876181+0000","last_unstale":"2024-01-15T18:20:10.753379+0000","last_undegraded":"2024-01-15T18:20:10.753379+0000","last_fullsized":"2024-01-15T18:20:10.753379+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:10.753091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:10.753091+0000","last_clean_scrub_stamp":"2024-01-15T18:20:10.753091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T02:28:13.671420+0000","scrub_duration":0.0040498490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"116'44","reported_seq":76,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:11.536809+0000","last_change":"2024-01-15T18:20:11.536924+0000","last_active":"2024-01-15T18:20:11.536809+0000","last_peered":"2024-01-15T18:20:11.536809+0000","last_clean":"2024-01-15T18:20:11.536809+0000","last_became_active":"2024-01-15T18:14:16.883461+0000","last_became_peered":"2024-01-15T18:14:16.883461+0000","last_unstale":"2024-01-15T18:20:11.536809+0000","last_undegraded":"2024-01-15T18:20:11.536809+0000","last_fullsized":"2024-01-15T18:20:11.536809+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":112,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'44","last_scrub_stamp":"2024-01-15T18:20:11.536616+0000","last_deep_scrub":"116'44","last_deep_scrub_stamp":"2024-01-15T18:20:11.536616+0000","last_clean_scrub_stamp":"2024-01-15T18:20:11.536616+0000","objects_scrubbed":14,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T04:15:20.371492+0000","scrub_duration":0.152340903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":25,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:02.388151+0000","last_change":"2024-01-15T18:20:02.388199+0000","last_active":"2024-01-15T18:20:02.388151+0000","last_peered":"2024-01-15T18:20:02.388151+0000","last_clean":"2024-01-15T18:20:02.388151+0000","last_became_active":"2024-01-15T18:14:15.872014+0000","last_became_peered":"2024-01-15T18:14:15.872014+0000","last_unstale":"2024-01-15T18:20:02.388151+0000","last_undegraded":"2024-01-15T18:20:02.388151+0000","last_fullsized":"2024-01-15T18:20:02.388151+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-15T18:20:02.387965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-15T18:20:02.387965+0000","last_clean_scrub_stamp":"2024-01-15T18:20:02.387965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T05:11:59.203734+0000","scrub_duration":0.0038625069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"116'119","reported_seq":265,"reported_epoch":116,"state":"active+clean","last_fresh":"2024-01-15T18:20:01.974811+0000","last_change":"2024-01-15T18:20:01.974882+0000","last_active":"2024-01-15T18:20:01.974811+0000","last_peered":"2024-01-15T18:20:01.974811+0000","last_clean":"2024-01-15T18:20:01.974811+0000","last_became_active":"2024-01-15T17:57:27.047645+0000","last_became_peered":"2024-01-15T17:57:27.047645+0000","last_unstale":"2024-01-15T18:20:01.974811+0000","last_undegraded":"2024-01-15T18:20:01.974811+0000","last_fullsized":"2024-01-15T18:20:01.974811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'119","last_scrub_stamp":"2024-01-15T18:20:01.974618+0000","last_deep_scrub":"116'119","last_deep_scrub_stamp":"2024-01-15T18:20:01.974618+0000","last_clean_scrub_stamp":"2024-01-15T18:20:01.974618+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-16T20:17:39.831008+0000","scrub_duration":0.053321348999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368710136,"num_objects":1286,"num_object_clones":1,"num_object_copies":2572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1286,"num_whiteouts":1,"num_read":525,"num_read_kb":2097165,"num_write":1306,"num_write_kb":5242895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":100663296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737459200,"data_stored":10737420272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3990,"ondisk_log_size":3990,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":646523,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":290,"num_write_kb":764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5506,"num_omap_keys":20,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1359872,"data_stored":1293046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":288,"ondisk_log_size":288,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1834576,"kb_used_data":1767368,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92537264,"statfs":{"total":96636764160,"available":94758158336,"internally_reserved":0,"allocated":1809784832,"data_stored":1808968013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3049999999999999,"15min":1.292},"min":{"1min":0.72599999999999998,"5min":0.61599999999999999,"15min":0.54000000000000004},"max":{"1min":1.5429999999999999,"5min":1.8140000000000001,"15min":4.0019999999999998},"last":1.675},{"interface":"front","average":{"1min":1.325,"5min":1.3819999999999999,"15min":1.5009999999999999},"min":{"1min":0.99399999999999999,"5min":0.88400000000000001,"15min":0.69999999999999996},"max":{"1min":1.6319999999999999,"5min":1.8300000000000001,"15min":27.32},"last":1.4650000000000001}]},{"osd":1,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.24,"15min":1.4490000000000001},"min":{"1min":0.82199999999999995,"5min":0.71399999999999997,"15min":0.53300000000000003},"max":{"1min":1.8129999999999999,"5min":2.1000000000000001,"15min":29.167999999999999},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.2290000000000001,"15min":1.2929999999999999},"min":{"1min":0.80300000000000005,"5min":0.70599999999999996,"15min":0.52800000000000002},"max":{"1min":1.889,"5min":2.2349999999999999,"15min":4.2699999999999996},"last":1.5009999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3580000000000001,"15min":1.3480000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.56000000000000005},"max":{"1min":1.8759999999999999,"5min":2.194,"15min":4.1920000000000002},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.254,"15min":1.4399999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.56599999999999995},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":27.358000000000001},"last":1.5569999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3120000000000001,"15min":1.528},"min":{"1min":1.0049999999999999,"5min":0.74399999999999999,"15min":0.61299999999999999},"max":{"1min":1.6890000000000001,"5min":2.1669999999999998,"15min":29.204999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":1.468},"min":{"1min":0.99099999999999999,"5min":0.82699999999999996,"15min":0.57799999999999996},"max":{"1min":1.734,"5min":2.2749999999999999,"15min":4.3879999999999999},"last":1.605}]},{"osd":4,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.371,"15min":1.49},"min":{"1min":0.87,"5min":0.87,"15min":0.50700000000000001},"max":{"1min":1.8089999999999999,"5min":6.0350000000000001,"15min":27.219000000000001},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.427,"15min":1.6020000000000001},"min":{"1min":0.89000000000000001,"5min":0.79900000000000004,"15min":0.57299999999999995},"max":{"1min":1.823,"5min":3.536,"15min":29.25},"last":1.294}]},{"osd":5,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.46,"15min":1.554},"min":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.68799999999999994},"max":{"1min":1.8700000000000001,"5min":2.1509999999999998,"15min":11.834},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5149999999999999,"15min":1.6459999999999999},"min":{"1min":1.169,"5min":0.92900000000000005,"15min":0.60499999999999998},"max":{"1min":2.145,"5min":5.0060000000000002,"15min":27.283000000000001},"last":1.6459999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":1.635,"15min":1.7729999999999999},"min":{"1min":1.29,"5min":1.1060000000000001,"15min":0.79200000000000004},"max":{"1min":2.4249999999999998,"5min":6.1289999999999996,"15min":27.393999999999998},"last":1.698},{"interface":"front","average":{"1min":1.577,"5min":1.5449999999999999,"15min":1.7490000000000001},"min":{"1min":1.234,"5min":1.0289999999999999,"15min":0.57299999999999995},"max":{"1min":2.085,"5min":2.754,"15min":29.106999999999999},"last":1.1819999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542429,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1260556,"kb_used_data":1193348,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":93111284,"statfs":{"total":96636764160,"available":95345954816,"internally_reserved":0,"allocated":1221988352,"data_stored":1221174661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.1919999999999999,"15min":1.2370000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.7490000000000001,"5min":1.7490000000000001,"15min":8.8350000000000009},"last":1.552},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.1719999999999999,"15min":1.2669999999999999},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.5089999999999999,"5min":1.78,"15min":9.548},"last":0.95299999999999996}]},{"osd":1,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2390000000000001,"15min":1.379},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.53300000000000003},"max":{"1min":1.718,"5min":2.1419999999999999,"15min":13.269},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.2889999999999999,"15min":1.3580000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.57799999999999996},"max":{"1min":1.7470000000000001,"5min":1.7470000000000001,"15min":9.6389999999999993},"last":1.0780000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.234,"15min":1.278},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.69799999999999995},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":7.0030000000000001},"last":1.429},{"interface":"front","average":{"1min":1.343,"5min":1.357,"15min":1.425},"min":{"1min":0.84999999999999998,"5min":0.81100000000000005,"15min":0.49399999999999999},"max":{"1min":1.7390000000000001,"5min":1.889,"15min":8.9359999999999999},"last":1.46}]},{"osd":3,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.4239999999999999,"15min":1.512},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.68500000000000005},"max":{"1min":1.708,"5min":1.946,"15min":16.855},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.351,"5min":1.3919999999999999,"15min":1.476},"min":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.628},"max":{"1min":1.976,"5min":1.976,"15min":8.0139999999999993},"last":1.3140000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":2.831,"5min":1.637,"15min":1.544},"min":{"1min":0.86099999999999999,"5min":0.77900000000000003,"15min":0.56299999999999994},"max":{"1min":33.161999999999999,"5min":33.161999999999999,"15min":33.161999999999999},"last":1.6140000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.371,"15min":1.4990000000000001},"min":{"1min":0.995,"5min":0.874,"15min":0.54900000000000004},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":10.118},"last":1.393}]},{"osd":5,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.3129999999999999,"15min":1.5129999999999999},"min":{"1min":0.90700000000000003,"5min":0.88,"15min":0.67000000000000004},"max":{"1min":1.804,"5min":1.875,"15min":16.309999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.482,"15min":1.619},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.66000000000000003},"max":{"1min":1.8540000000000001,"5min":1.9079999999999999,"15min":10.077},"last":1.7190000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:12 2024","interfaces":[{"interface":"back","average":{"1min":1.5840000000000001,"5min":1.5029999999999999,"15min":1.6220000000000001},"min":{"1min":1.196,"5min":0.995,"15min":0.72299999999999998},"max":{"1min":2.8500000000000001,"5min":2.8500000000000001,"15min":10.302},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.526,"15min":1.6479999999999999},"min":{"1min":1.0680000000000001,"5min":1.0449999999999999,"15min":0.58399999999999996},"max":{"1min":2.0640000000000001,"5min":6.3029999999999999,"15min":8.0519999999999996},"last":1.516}]}]},{"osd":5,"up_from":14,"seq":60129542431,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":912332,"kb_used_data":849284,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93459508,"statfs":{"total":96636764160,"available":95702536192,"internally_reserved":0,"allocated":869666816,"data_stored":868853537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.2549999999999999,"15min":1.365},"min":{"1min":0.80400000000000005,"5min":0.70099999999999996,"15min":0.61099999999999999},"max":{"1min":1.502,"5min":4.1829999999999998,"15min":26.077999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3759999999999999,"15min":1.4179999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.56399999999999995},"max":{"1min":1.6859999999999999,"5min":4.2279999999999998,"15min":9.8699999999999992},"last":1.546}]},{"osd":1,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.2529999999999999,"15min":1.401},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.53300000000000003},"max":{"1min":1.7230000000000001,"5min":1.923,"15min":26},"last":1.234},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.375,"15min":1.359},"min":{"1min":0.84899999999999998,"5min":0.82899999999999996,"15min":0.58999999999999997},"max":{"1min":1.6699999999999999,"5min":4.0869999999999997,"15min":7.5309999999999997},"last":1.288}]},{"osd":2,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3320000000000001,"15min":1.498},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.56699999999999995},"max":{"1min":1.6120000000000001,"5min":1.774,"15min":26.234999999999999},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.222,"5min":1.349,"15min":1.5149999999999999},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.629},"max":{"1min":1.52,"5min":1.8240000000000001,"15min":17.126999999999999},"last":1.49}]},{"osd":3,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.335,"15min":1.351},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.67200000000000004},"max":{"1min":1.595,"5min":2.375,"15min":7.6150000000000002},"last":1.522},{"interface":"front","average":{"1min":1.462,"5min":1.4850000000000001,"15min":1.6000000000000001},"min":{"1min":1.044,"5min":0.96499999999999997,"15min":0.67900000000000005},"max":{"1min":1.736,"5min":1.855,"15min":26.134},"last":1.663}]},{"osd":4,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.242,"15min":1.3799999999999999},"min":{"1min":0.621,"5min":0.621,"15min":0.442},"max":{"1min":1.5589999999999999,"5min":2.286,"15min":9.9979999999999993},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.4359999999999999,"15min":1.6459999999999999},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.498},"max":{"1min":1.732,"5min":2.1219999999999999,"15min":16.414000000000001},"last":1.365}]},{"osd":6,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.365,"15min":1.383},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.45900000000000002},"max":{"1min":1.833,"5min":2.6640000000000001,"15min":7.9240000000000004},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4430000000000001,"15min":1.611},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.49199999999999999},"max":{"1min":1.71,"5min":2.0659999999999998,"15min":26.186},"last":1.7470000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:06 2024","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.5760000000000001,"15min":1.7270000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.65700000000000003},"max":{"1min":1.8029999999999999,"5min":2.04,"15min":12.326000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.538,"15min":1.556},"min":{"1min":0.755,"5min":0.755,"15min":0.60099999999999998},"max":{"1min":1.8109999999999999,"5min":4.3700000000000001,"15min":12.175000000000001},"last":1.4730000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542430,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1711112,"kb_used_data":1643904,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":92660728,"statfs":{"total":96636764160,"available":94884585472,"internally_reserved":0,"allocated":1683357696,"data_stored":1682548170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.226,"15min":1.226},"min":{"1min":0.83699999999999997,"5min":0.68899999999999995,"15min":0.31900000000000001},"max":{"1min":1.502,"5min":1.8859999999999999,"15min":13.288},"last":1.101},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1990000000000001,"15min":1.262},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.34699999999999998},"max":{"1min":1.653,"5min":1.7030000000000001,"15min":9.2409999999999997},"last":1.3}]},{"osd":1,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2470000000000001,"15min":1.357},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.41499999999999998},"max":{"1min":1.7150000000000001,"5min":1.78,"15min":28.663},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.2709999999999999,"15min":1.3899999999999999},"min":{"1min":0.92600000000000005,"5min":0.66500000000000004,"15min":0.46200000000000002},"max":{"1min":1.681,"5min":1.7789999999999999,"15min":13.614000000000001},"last":1.0089999999999999}]},{"osd":2,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.3029999999999999,"15min":1.4179999999999999},"min":{"1min":0.83399999999999996,"5min":0.73899999999999999,"15min":0.48799999999999999},"max":{"1min":1.673,"5min":1.923,"15min":16.023},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.407,"5min":1.3819999999999999,"15min":1.4710000000000001},"min":{"1min":1.196,"5min":0.72199999999999998,"15min":0.52500000000000002},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":18.858000000000001},"last":1.246}]},{"osd":3,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2729999999999999,"15min":1.331},"min":{"1min":0.84699999999999998,"5min":0.59899999999999998,"15min":0.55600000000000005},"max":{"1min":1.6930000000000001,"5min":1.9690000000000001,"15min":5.0860000000000003},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.4470000000000001,"15min":1.5329999999999999},"min":{"1min":1.0600000000000001,"5min":0.995,"15min":0.54600000000000004},"max":{"1min":1.792,"5min":2.1400000000000001,"15min":18.027000000000001},"last":1.3660000000000001}]},{"osd":5,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4339999999999999,"15min":1.4099999999999999},"min":{"1min":0.83999999999999997,"5min":0.57399999999999995,"15min":0.41899999999999998},"max":{"1min":1.903,"5min":7.7439999999999998,"15min":7.7439999999999998},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.3700000000000001,"15min":1.4279999999999999},"min":{"1min":0.94999999999999996,"5min":0.51900000000000002,"15min":0.42399999999999999},"max":{"1min":2.0049999999999999,"5min":2.0960000000000001,"15min":8.5660000000000007},"last":1.5189999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.425,"15min":1.5409999999999999},"min":{"1min":1.1359999999999999,"5min":0.92900000000000005,"15min":0.495},"max":{"1min":1.865,"5min":2.0110000000000001,"15min":16.103999999999999},"last":1.159},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.5249999999999999,"15min":1.6240000000000001},"min":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.499},"max":{"1min":1.9350000000000001,"5min":1.966,"15min":12.936999999999999},"last":0.88100000000000001}]},{"osd":7,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.464,"15min":1.655},"min":{"1min":1.2210000000000001,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.8540000000000001,"5min":2.4689999999999999,"15min":23.390000000000001},"last":1.609},{"interface":"front","average":{"1min":1.6519999999999999,"5min":1.571,"15min":1.7050000000000001},"min":{"1min":1.25,"5min":1.0900000000000001,"15min":0.62},"max":{"1min":1.9690000000000001,"5min":2.1019999999999999,"15min":16.763000000000002},"last":1.2270000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542431,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1462032,"kb_used_data":1386504,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92909808,"statfs":{"total":96636764160,"available":95139643392,"internally_reserved":0,"allocated":1419780096,"data_stored":1418951753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":77330888},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.0229999999999999,"15min":1.133},"min":{"1min":0.56599999999999995,"5min":0.435,"15min":0.29799999999999999},"max":{"1min":9.0299999999999994,"5min":9.0299999999999994,"15min":36.408000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.095,"15min":1.0880000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.377},"max":{"1min":1.427,"5min":1.986,"15min":5.0190000000000001},"last":1.173}]},{"osd":1,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.97099999999999997,"15min":1.0429999999999999},"min":{"1min":0.46200000000000002,"5min":0.45900000000000002,"15min":0.41999999999999998},"max":{"1min":1.2889999999999999,"5min":1.837,"15min":5.226},"last":1.099},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.1919999999999999,"15min":1.1910000000000001},"min":{"1min":0.48899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7869999999999999,"5min":2.0609999999999999,"15min":2.5790000000000002},"last":1.3380000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.248,"15min":1.3360000000000001},"min":{"1min":0.56599999999999995,"5min":0.50900000000000001,"15min":0.44600000000000001},"max":{"1min":1.6850000000000001,"5min":2.0150000000000001,"15min":14.025},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.343,"15min":1.337},"min":{"1min":0.58199999999999996,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.845,"5min":2.403,"15min":3.6240000000000001},"last":1.5800000000000001}]},{"osd":4,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.403,"15min":1.448},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.59999999999999998},"max":{"1min":10.542,"5min":10.542,"15min":10.542},"last":1.7609999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.454,"15min":1.5920000000000001},"min":{"1min":0.83799999999999997,"5min":0.79100000000000004,"15min":0.748},"max":{"1min":1.8100000000000001,"5min":4.8570000000000002,"15min":15.769},"last":1.4279999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.93100000000000005,"5min":0.82399999999999995,"15min":0.69299999999999995},"max":{"1min":2.1629999999999998,"5min":2.2570000000000001,"15min":5.5220000000000002},"last":1.627},{"interface":"front","average":{"1min":3.2229999999999999,"5min":1.8460000000000001,"15min":1.897},"min":{"1min":0.88700000000000001,"5min":0.877,"15min":0.67900000000000005},"max":{"1min":44.354999999999997,"5min":44.354999999999997,"15min":44.354999999999997},"last":1.7130000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":1.6120000000000001,"15min":1.7729999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.51400000000000001},"max":{"1min":10.59,"5min":10.59,"15min":32.259999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5820000000000001,"15min":1.6859999999999999},"min":{"1min":0.91000000000000003,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.4260000000000002,"5min":2.4260000000000002,"15min":6.8639999999999999},"last":1.5229999999999999}]},{"osd":7,"last update":"Mon Jan 15 18:20:09 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":1.659,"15min":1.903},"min":{"1min":1.1970000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.2509999999999999,"5min":3.8149999999999999,"15min":24.146999999999998},"last":1.8149999999999999},{"interface":"front","average":{"1min":1.839,"5min":1.788,"15min":1.825},"min":{"1min":0.93500000000000005,"5min":0.89100000000000001,"15min":0.56100000000000005},"max":{"1min":10.625,"5min":16.734000000000002,"15min":17.622},"last":1.675}]}]},{"osd":2,"up_from":14,"seq":60129542430,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":957512,"kb_used_data":886144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93414328,"statfs":{"total":96636764160,"available":95656271872,"internally_reserved":0,"allocated":907411456,"data_stored":906602990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.99399999999999999,"15min":1.0089999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.371},"max":{"1min":1.4510000000000001,"5min":3.1589999999999998,"15min":10.256},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.998,"5min":1.1140000000000001,"15min":1.161},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.33700000000000002},"max":{"1min":1.653,"5min":2.1110000000000002,"15min":10.319000000000001},"last":1.306}]},{"osd":1,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.0029999999999999,"15min":1.0269999999999999},"min":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.38600000000000001},"max":{"1min":1.427,"5min":2.6059999999999999,"15min":4.4939999999999998},"last":1.036},{"interface":"front","average":{"1min":1.032,"5min":1.159,"15min":1.2290000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.35299999999999998},"max":{"1min":1.7,"5min":2.5070000000000001,"15min":7.5119999999999996},"last":1.5329999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":1.3,"15min":1.3879999999999999},"min":{"1min":0.622,"5min":0.622,"15min":0.44400000000000001},"max":{"1min":1.5349999999999999,"5min":2.1920000000000002,"15min":25.891999999999999},"last":1.734},{"interface":"front","average":{"1min":1.137,"5min":1.214,"15min":1.29},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.376},"max":{"1min":1.5680000000000001,"5min":2.7589999999999999,"15min":14.946},"last":1.202}]},{"osd":4,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.5389999999999999,"15min":1.504},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.66100000000000003},"max":{"1min":1.8009999999999999,"5min":15.281000000000001,"15min":15.281000000000001},"last":1.377},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.446,"15min":1.5700000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.65300000000000002},"max":{"1min":1.6830000000000001,"5min":2.867,"15min":10.279999999999999},"last":1.6299999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.45,"15min":1.556},"min":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.63500000000000001},"max":{"1min":2.1000000000000001,"5min":2.3740000000000001,"15min":11.071999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.599,"15min":1.5940000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.71699999999999997},"max":{"1min":1.8799999999999999,"5min":10.833,"15min":10.833},"last":1.4610000000000001}]},{"osd":6,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.5009999999999999,"15min":1.5980000000000001},"min":{"1min":0.90400000000000003,"5min":0.76500000000000001,"15min":0.64800000000000002},"max":{"1min":1.8620000000000001,"5min":2.984,"15min":10.185},"last":1.708},{"interface":"front","average":{"1min":1.518,"5min":1.6399999999999999,"15min":1.7270000000000001},"min":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.85899999999999999},"max":{"1min":2.194,"5min":10.917,"15min":22.382000000000001},"last":1.595}]},{"osd":7,"last update":"Mon Jan 15 18:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5880000000000001,"15min":1.8260000000000001},"min":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":0.84299999999999997},"max":{"1min":1.8879999999999999,"5min":3.052,"15min":19.986000000000001},"last":1.77},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.651,"15min":1.7729999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.877},"max":{"1min":2.2799999999999998,"5min":3.069,"15min":11.893000000000001},"last":1.6759999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542430,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1141912,"kb_used_data":1066384,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93229928,"statfs":{"total":96636764160,"available":95467446272,"internally_reserved":0,"allocated":1091977216,"data_stored":1091152862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.95999999999999996,"15min":1.0960000000000001},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.36199999999999999},"max":{"1min":1.5669999999999999,"5min":3.4510000000000001,"15min":19.218},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.998,"5min":1.0780000000000001,"15min":1.204},"min":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.38700000000000001},"max":{"1min":1.825,"5min":1.825,"15min":19.823},"last":0.84599999999999997}]},{"osd":2,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.96899999999999997,"15min":1.0800000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.31900000000000001},"max":{"1min":1.603,"5min":1.776,"15min":19.02},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.21,"15min":1.385},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.41599999999999998},"max":{"1min":1.5,"5min":1.889,"15min":29.251000000000001},"last":0.94799999999999995}]},{"osd":3,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":1.3109999999999999,"15min":1.494},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.40899999999999997},"max":{"1min":1.5169999999999999,"5min":2.036,"15min":27.021000000000001},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.169,"15min":1.3700000000000001},"min":{"1min":0.60799999999999998,"5min":0.55200000000000005,"15min":0.38800000000000001},"max":{"1min":1.859,"5min":1.859,"15min":31.218},"last":1.0389999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4630000000000001,"15min":1.6850000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.70599999999999996},"max":{"1min":1.8360000000000001,"5min":1.9650000000000001,"15min":27.059999999999999},"last":1.351},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.409,"15min":1.6100000000000001},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.56499999999999995},"max":{"1min":1.97,"5min":1.986,"15min":31.433},"last":1.3029999999999999}]},{"osd":5,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":1.4410000000000001,"15min":1.583},"min":{"1min":1.05,"5min":1.05,"15min":0.68500000000000005},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":20.012},"last":1.764},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.383,"15min":1.4730000000000001},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.69299999999999995},"max":{"1min":1.829,"5min":1.9410000000000001,"15min":3.9910000000000001},"last":1.6819999999999999}]},{"osd":6,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.571,"15min":1.768},"min":{"1min":1.03,"5min":1.03,"15min":0.68300000000000005},"max":{"1min":1.9199999999999999,"5min":2.2189999999999999,"15min":29.530000000000001},"last":1.526},{"interface":"front","average":{"1min":1.524,"5min":1.548,"15min":1.7789999999999999},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.61399999999999999},"max":{"1min":2.1960000000000002,"5min":2.1960000000000002,"15min":31.186},"last":1.627}]},{"osd":7,"last update":"Mon Jan 15 18:20:19 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.492,"15min":1.7310000000000001},"min":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.63},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":26.393999999999998},"last":1.8160000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.589,"15min":1.794},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.76100000000000001},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":27.582000000000001},"last":1.5700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542429,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1782112,"kb_used_data":1706584,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92589728,"statfs":{"total":96636764160,"available":94811881472,"internally_reserved":0,"allocated":1747542016,"data_stored":1746698108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.96299999999999997,"15min":1.087},"min":{"1min":0.5,"5min":0.5,"15min":0.30599999999999999},"max":{"1min":1.385,"5min":5.7560000000000002,"15min":20.437999999999999},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.2130000000000001,"15min":1.139},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.32400000000000001},"max":{"1min":1.8089999999999999,"5min":6.3600000000000003,"15min":6.3600000000000003},"last":1.1850000000000001}]},{"osd":2,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.2250000000000001,"15min":1.296},"min":{"1min":0.57199999999999995,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.669,"5min":11.06,"15min":20.556000000000001},"last":1.246},{"interface":"front","average":{"1min":1.141,"5min":1.26,"15min":1.2490000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.39000000000000001},"max":{"1min":1.5800000000000001,"5min":6.4939999999999998,"15min":6.4939999999999998},"last":1.3109999999999999}]},{"osd":3,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.228,"15min":1.163},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.42999999999999999},"max":{"1min":1.9259999999999999,"5min":10.885999999999999,"15min":10.885999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3520000000000001,"15min":1.468},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.45100000000000001},"max":{"1min":1.538,"5min":9.3339999999999996,"15min":20.670000000000002},"last":1.5129999999999999}]},{"osd":4,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.512,"15min":1.4810000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.1259999999999999,"5min":9.9969999999999999,"15min":9.9969999999999999},"last":1.571},{"interface":"front","average":{"1min":1.29,"5min":1.5429999999999999,"15min":1.4319999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.58799999999999997},"max":{"1min":1.7709999999999999,"5min":17.567,"15min":17.567},"last":1.619}]},{"osd":5,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.6399999999999999,"15min":1.6339999999999999},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.69799999999999995},"max":{"1min":2.0049999999999999,"5min":17.155999999999999,"15min":17.155999999999999},"last":1.46},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.6319999999999999,"15min":1.7749999999999999},"min":{"1min":1.0780000000000001,"5min":0.79700000000000004,"15min":0.76000000000000001},"max":{"1min":1.7829999999999999,"5min":9.3979999999999997,"15min":20.789999999999999},"last":1.821}]},{"osd":6,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.6140000000000001,"15min":1.8220000000000001},"min":{"1min":1.244,"5min":1.1060000000000001,"15min":0.70599999999999996},"max":{"1min":1.7749999999999999,"5min":5.8689999999999998,"15min":20.893000000000001},"last":1.708},{"interface":"front","average":{"1min":1.583,"5min":1.657,"15min":1.671},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.62},"max":{"1min":2.258,"5min":6.2270000000000003,"15min":8.3829999999999991},"last":1.865}]},{"osd":7,"last update":"Mon Jan 15 18:20:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.6990000000000001,"15min":1.7849999999999999},"min":{"1min":0.996,"5min":0.996,"15min":0.56200000000000006},"max":{"1min":2.363,"5min":9.9429999999999996,"15min":24.556999999999999},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.484,"5min":1.6359999999999999,"15min":1.657},"min":{"1min":1.073,"5min":1.048,"15min":0.64000000000000001},"max":{"1min":2.129,"5min":10.943,"15min":11.470000000000001},"last":1.76}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":663552,"data_stored":644516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":1321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":644474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1744846848,"data_stored":1744831219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1090527232,"data_stored":1090519412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":905969664,"data_stored":905969664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1417682944,"data_stored":1417675274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1681920000,"data_stored":1681916165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":868220928,"data_stored":868220928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1220546560,"data_stored":1220542586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1807745024,"data_stored":1807745024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-15T18:21:03.146 DEBUG:teuthology.orchestra.run.smithi002:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-15T18:21:03.397 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.395+0000 7f1431c00640 1 Processor -- start 2024-01-15T18:21:03.398 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.395+0000 7f1431c00640 1 -- start start 2024-01-15T18:21:03.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f1431c00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f142c15d270 0x7f142c0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.399 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f1431c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c161890 0x7f142c160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f1431c00640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f142c0b2b60 0x7f142c1611a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f1431c00640 1 -- --> v2:172.21.15.2:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c06c5e0 con 0x7f142c15d270 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f1431c00640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f142c06c8c0 con 0x7f142c0b2b60 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f1431c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c071c00 con 0x7f142c161890 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f142be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f142c15d270 0x7f142c0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f142be00640 1 --2- >> v2:172.21.15.2:3300/0 conn(0x7f142c15d270 0x7f142c0b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40388/0 (socket says 172.21.15.2:40388) 2024-01-15T18:21:03.400 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.397+0000 7f142be00640 1 -- 172.21.15.2:0/1409766541 learned_addr learned my addr 172.21.15.2:0/1409766541 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:21:03.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.398+0000 7f142be00640 1 -- 172.21.15.2:0/1409766541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c161890 msgr2=0x7f142c160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:03.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.398+0000 7f142be00640 1 --2- 172.21.15.2:0/1409766541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c161890 0x7f142c160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.398+0000 7f142be00640 1 -- 172.21.15.2:0/1409766541 >> v2:172.21.15.137:3300/0 conn(0x7f142c0b2b60 msgr2=0x7f142c1611a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:03.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.398+0000 7f142be00640 1 --2- 172.21.15.2:0/1409766541 >> v2:172.21.15.137:3300/0 conn(0x7f142c0b2b60 0x7f142c1611a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.398+0000 7f142be00640 1 -- 172.21.15.2:0/1409766541 --> v2:172.21.15.2:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142c071ee0 con 0x7f142c15d270 2024-01-15T18:21:03.401 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.399+0000 7f142be00640 1 --2- 172.21.15.2:0/1409766541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15d270 0x7f142c0b2620 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f142c06bb40 tx=0x7f141c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d36b59ed0d1d76 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.399+0000 7f142aa00640 1 -- 172.21.15.2:0/1409766541 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c031750 con 0x7f142c15d270 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.399+0000 7f142aa00640 1 -- 172.21.15.2:0/1409766541 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f141c0318b0 con 0x7f142c15d270 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.399+0000 7f142aa00640 1 -- 172.21.15.2:0/1409766541 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f141c031c40 con 0x7f142c15d270 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 -- 172.21.15.2:0/1409766541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15d270 msgr2=0x7f142c0b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 --2- 172.21.15.2:0/1409766541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15d270 0x7f142c0b2620 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f142c06bb40 tx=0x7f141c02b7f0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 -- 172.21.15.2:0/1409766541 shutdown_connections 2024-01-15T18:21:03.403 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 --2- 172.21.15.2:0/1409766541 >> v2:172.21.15.137:3300/0 conn(0x7f142c0b2b60 0x7f142c1611a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 --2- 172.21.15.2:0/1409766541 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c161890 0x7f142c160c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 --2- 172.21.15.2:0/1409766541 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c15d270 0x7f142c0b2620 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 -- 172.21.15.2:0/1409766541 >> 172.21.15.2:0/1409766541 conn(0x7f142c0b0840 msgr2=0x7f142c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 -- 172.21.15.2:0/1409766541 shutdown_connections 2024-01-15T18:21:03.404 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.400+0000 7f1431c00640 1 -- 172.21.15.2:0/1409766541 wait complete. 2024-01-15T18:21:03.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.402+0000 7f1431c00640 1 Processor -- start 2024-01-15T18:21:03.405 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.402+0000 7f1431c00640 1 -- start start 2024-01-15T18:21:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431c00640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0b2b60 0x7f142c15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431c00640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15d270 0x7f142c0adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431c00640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 0x7f142c0cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.407 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431c00640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c1f7520 con 0x7f142c161890 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431c00640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142c1f7800 con 0x7f142c0b2b60 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431c00640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f142c1f7ae0 con 0x7f142c15d270 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.403+0000 7f1431200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 0x7f142c0cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 0x7f142c0cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40396/0 (socket says 172.21.15.2:40396) 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 -- 172.21.15.2:0/1833109243 learned_addr learned my addr 172.21.15.2:0/1833109243 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f142be00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0b2b60 0x7f142c15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f142b400640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15d270 0x7f142c0adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.408 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 -- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15d270 msgr2=0x7f142c0adf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15d270 0x7f142c0adf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 -- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0b2b60 msgr2=0x7f142c15ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0b2b60 0x7f142c15ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f1431200640 1 -- 172.21.15.2:0/1833109243 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141c0094d0 con 0x7f142c161890 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.404+0000 7f142b400640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15d270 0x7f142c0adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.405+0000 7f1431200640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 0x7f142c0cade0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f142400b9e0 tx=0x7f142400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.405+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142400cb20 con 0x7f142c161890 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.405+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142400cca0 con 0x7f142c161890 2024-01-15T18:21:03.409 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.405+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1424014680 con 0x7f142c161890 2024-01-15T18:21:03.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.405+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c1f7dc0 con 0x7f142c161890 2024-01-15T18:21:03.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.405+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142c0ae4d0 con 0x7f142c161890 2024-01-15T18:21:03.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.406+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f142c074770 con 0x7f142c161890 2024-01-15T18:21:03.410 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.407+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 44) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f142401d3b0 con 0x7f142c161890 2024-01-15T18:21:03.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.409+0000 7f1423e00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406c510 0x7f140406e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.409+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f14240887b0 con 0x7f142c161890 2024-01-15T18:21:03.411 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.409+0000 7f142be00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406c510 0x7f140406e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.413 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.411+0000 7f142be00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406c510 0x7f140406e9d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f141c035000 tx=0x7f141c008030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:03.416 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.414+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1424055070 con 0x7f142c161890 2024-01-15T18:21:03.611 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.609+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f142c158d90 con 0x7f142c161890 2024-01-15T18:21:03.616 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.614+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v43)=0 v43) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1424019020 con 0x7f142c161890 2024-01-15T18:21:03.618 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.616+0000 7f1423e00640 1 -- 172.21.15.2:0/1833109243 <== mon.0 v2:172.21.15.2:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1424055070 con 0x7f142c161890 2024-01-15T18:21:03.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.617+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406c510 msgr2=0x7f140406e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.617+0000 7f1431c00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406c510 0x7f140406e9d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f141c035000 tx=0x7f141c008030 comp rx=0 tx=0).stop 2024-01-15T18:21:03.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.617+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 msgr2=0x7f142c0cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.620 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.617+0000 7f1431c00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 0x7f142c0cade0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f142400b9e0 tx=0x7f142400bea0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 shutdown_connections 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f140406c510 0x7f140406e9d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f142c161890 0x7f142c0cade0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f142c15d270 0x7f142c0adf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 --2- 172.21.15.2:0/1833109243 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f142c0b2b60 0x7f142c15ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 >> 172.21.15.2:0/1833109243 conn(0x7f142c0b0840 msgr2=0x7f142c15f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 shutdown_connections 2024-01-15T18:21:03.621 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.618+0000 7f1431c00640 1 -- 172.21.15.2:0/1833109243 wait complete. 2024-01-15T18:21:03.640 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-15T18:21:03.641 DEBUG:teuthology.orchestra.run.smithi002:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-15T18:21:03.660 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:21:03.658+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:03.660 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:21:03.658+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:03.660 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:21:03.658+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:03.661 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:21:03.658+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:03.661 INFO:tasks.ceph.mgr.x.smithi137.stderr:2024-01-15T18:21:03.658+0000 7fe212400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-15T18:21:03.860 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.858+0000 7f5c1b200640 1 Processor -- start 2024-01-15T18:21:03.861 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.858+0000 7f5c1b200640 1 -- start start 2024-01-15T18:21:03.862 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.860+0000 7f5c1b200640 1 --2- >> v2:172.21.15.137:3300/0 conn(0x7f5c1c143810 0x7f5c1c06dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.860+0000 7f5c1b200640 1 -- --> v1:172.21.15.2:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c06e470 con 0x7f5c1c06e010 2024-01-15T18:21:03.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.860+0000 7f5c1b200640 1 -- --> v1:172.21.15.137:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0ccbe0 con 0x7f5c1c0b2250 2024-01-15T18:21:03.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.860+0000 7f5c1b200640 1 -- --> v2:172.21.15.137:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c0b3520 con 0x7f5c1c143810 2024-01-15T18:21:03.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.860+0000 7f5c13e00640 1 --1- >> v1:172.21.15.2:6789/0 conn(0x7f5c1c06e010 0x7f5c1c0b1b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.2:6789/0 says I am v1:172.21.15.2:55310/0 (socket says 172.21.15.2:55310) 2024-01-15T18:21:03.863 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.860+0000 7f5c13e00640 1 -- 172.21.15.2:0/3806842796 learned_addr learned my addr 172.21.15.2:0/3806842796 (peer_addr_for_me v1:172.21.15.2:0/0) 2024-01-15T18:21:03.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19e00640 1 --2- 172.21.15.2:0/3806842796 >> v2:172.21.15.137:3300/0 conn(0x7f5c1c143810 0x7f5c1c06dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 <== mon.0 v1:172.21.15.2:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991060833 0 0) 0x7f5c04001110 con 0x7f5c1c06e010 2024-01-15T18:21:03.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 --> v1:172.21.15.2:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bfc003520 con 0x7f5c1c06e010 2024-01-15T18:21:03.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 <== mon.0 v1:172.21.15.2:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 645061535 0 0) 0x7f5bfc003520 con 0x7f5c1c06e010 2024-01-15T18:21:03.865 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 >> v2:172.21.15.137:3300/0 conn(0x7f5c1c143810 msgr2=0x7f5c1c06dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 --2- 172.21.15.2:0/3806842796 >> v2:172.21.15.137:3300/0 conn(0x7f5c1c143810 0x7f5c1c06dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 >> v1:172.21.15.137:6789/0 conn(0x7f5c1c0b2250 legacy=0x7f5c1c0cd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 --> v1:172.21.15.2:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c143bf0 con 0x7f5c1c06e010 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.861+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 <== mon.0 v1:172.21.15.2:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3412437658 0 0) 0x7f5c04001800 con 0x7f5c1c06e010 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 >> v1:172.21.15.2:6789/0 conn(0x7f5c1c06e010 legacy=0x7f5c1c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bfc0039c0 0x7f5bfc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0047c0 0x7f5bfc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bfc0051d0 0x7f5bfc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.866 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0b3520 con 0x7f5bfc0039c0 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc00c120 con 0x7f5bfc0047c0 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc00c2e0 con 0x7f5bfc0051d0 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19e00640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bfc0039c0 0x7f5bfc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c19e00640 1 -- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bfc0051d0 msgr2=0x7f5bfc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.862+0000 7f5c1a800640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bfc0051d0 0x7f5bfc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.863+0000 7f5c19e00640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5bfc0051d0 0x7f5bfc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.863+0000 7f5c19e00640 1 -- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0047c0 msgr2=0x7f5bfc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-15T18:21:03.867 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.863+0000 7f5c19e00640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0047c0 0x7f5bfc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.863+0000 7f5c19e00640 1 -- 172.21.15.2:0/3806842796 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc003520 con 0x7f5bfc0039c0 2024-01-15T18:21:03.868 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.863+0000 7f5c1a800640 1 -- 172.21.15.2:0/3806842796 reap_dead start 2024-01-15T18:21:03.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.864+0000 7f5c19e00640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bfc0039c0 0x7f5bfc003de0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009a90 tx=0x7f5bfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44cd1270d3ec41e server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:03.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.864+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c02c6b0 con 0x7f5bfc0039c0 2024-01-15T18:21:03.869 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.864+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5c0c02c810 con 0x7f5bfc0039c0 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.864+0000 7f5c19400640 1 -- 172.21.15.2:0/3806842796 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0c02cbc0 con 0x7f5bfc0039c0 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 -- 172.21.15.2:0/3806842796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bfc0039c0 msgr2=0x7f5bfc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bfc0039c0 0x7f5bfc003de0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c009a90 tx=0x7f5bfc0057f0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 -- 172.21.15.2:0/3806842796 shutdown_connections 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 --2- 172.21.15.2:0/3806842796 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5bfc0039c0 0x7f5bfc003de0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 -- 172.21.15.2:0/3806842796 >> 172.21.15.2:0/3806842796 conn(0x7f5c1c0b0840 msgr2=0x7f5c1c0b0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 -- 172.21.15.2:0/3806842796 shutdown_connections 2024-01-15T18:21:03.870 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.865+0000 7f5c1b200640 1 -- 172.21.15.2:0/3806842796 wait complete. 2024-01-15T18:21:03.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.867+0000 7f5c1b200640 1 Processor -- start 2024-01-15T18:21:03.871 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.867+0000 7f5c1b200640 1 -- start start 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1b200640 1 --2- >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0039c0 0x7f5c1c0c3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1b200640 1 --2- >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1c0c3ff0 0x7f5c1c0c4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1b200640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 0x7f5c1c1f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1b200640 1 -- --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c15cd70 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1b200640 1 -- --> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c1509b0 con 0x7f5bfc0039c0 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1b200640 1 -- --> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c0c4aa0 con 0x7f5c1c0c3ff0 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 0x7f5c1c1f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1a800640 1 --2- >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 0x7f5c1c1f67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.2:3300/0 says I am v2:172.21.15.2:40416/0 (socket says 172.21.15.2:40416) 2024-01-15T18:21:03.873 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c1a800640 1 -- 172.21.15.2:0/2326457766 learned_addr learned my addr 172.21.15.2:0/2326457766 (peer_addr_for_me v2:172.21.15.2:0/0) 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c13e00640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1c0c3ff0 0x7f5c1c0c4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.868+0000 7f5c19e00640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0039c0 0x7f5c1c0c3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1a800640 1 -- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1c0c3ff0 msgr2=0x7f5c1c0c4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1a800640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1c0c3ff0 0x7f5c1c0c4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1a800640 1 -- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0039c0 msgr2=0x7f5c1c0c3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1a800640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0039c0 0x7f5c1c0c3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1a800640 1 -- 172.21.15.2:0/2326457766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0c4d80 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.874 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c19e00640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0039c0 0x7f5c1c0c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1a800640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 0x7f5c1c1f67e0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c072a10 tx=0x7f5c1400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14003b70 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5c14003cd0 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c140052d0 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c0c5060 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.869+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c1f6d20 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.875 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.870+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c074770 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.871+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 4 ==== mgrmap(e 44) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f5c14005430 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.872+0000 7f5c12000640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bf406c4f0 0x7f5bf406e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-15T18:21:03.876 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.873+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5907+0+0 (secure 0 0 0) 0x7f5c14088550 con 0x7f5c1c1f63a0 2024-01-15T18:21:03.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.873+0000 7f5c19e00640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bf406c4f0 0x7f5bf406e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-15T18:21:03.877 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.874+0000 7f5c19e00640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bf406c4f0 0x7f5bf406e9b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c036000 tx=0x7f5c0c030000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-15T18:21:03.880 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:03.878+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c14054a10 con 0x7f5c1c1f63a0 2024-01-15T18:21:04.118 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.116+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 --> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c1c0cf9d0 con 0x7f5c1c1f63a0 2024-01-15T18:21:04.119 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.117+0000 7f5c12000640 1 -- 172.21.15.2:0/2326457766 <== mon.0 v2:172.21.15.2:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f5c14018020 con 0x7f5c1c1f63a0 2024-01-15T18:21:04.119 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-01-15T18:21:04.119 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6834/3995733153","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":3995733153},{"type":"v1","addr":"172.21.15.2:6834","nonce":3995733153}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":9039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.2:6835/3120477923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6833","nonce":3120477923},{"type":"v1","addr":"172.21.15.2:6835","nonce":3120477923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.137:6836/1748176015","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6834","nonce":1748176015},{"type":"v1","addr":"172.21.15.137:6836","nonce":1748176015}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-15T18:14:16.323114+0000","modified":"2024-01-15T18:14:19.369618+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9051},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9051":{"gid":9051,"name":"b","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.137:6837/2854972827","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6835","nonce":2854972827},{"type":"v1","addr":"172.21.15.137:6837","nonce":2854972827}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-15T18:21:04.120 INFO:teuthology.orchestra.run.smithi002.stderr:dumped fsmap epoch 75 2024-01-15T18:21:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.119+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bf406c4f0 msgr2=0x7f5bf406e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.119+0000 7f5c1b200640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bf406c4f0 0x7f5bf406e9b0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c036000 tx=0x7f5c0c030000 comp rx=0 tx=0).stop 2024-01-15T18:21:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.119+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 msgr2=0x7f5c1c1f67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-15T18:21:04.122 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.119+0000 7f5c1b200640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 0x7f5c1c1f67e0 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c072a10 tx=0x7f5c1400dc00 comp rx=0 tx=0).stop 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 shutdown_connections 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:6832/2238446093,v1:172.21.15.137:6833/2238446093] conn(0x7f5bf406c4f0 0x7f5bf406e9b0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] conn(0x7f5c1c1f63a0 0x7f5c1c1f67e0 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3301/0,v1:172.21.15.137:6790/0] conn(0x7f5c1c0c3ff0 0x7f5c1c0c4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 --2- 172.21.15.2:0/2326457766 >> [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] conn(0x7f5bfc0039c0 0x7f5c1c0c3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 >> 172.21.15.2:0/2326457766 conn(0x7f5c1c0b0840 msgr2=0x7f5c1c0700d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 shutdown_connections 2024-01-15T18:21:04.123 INFO:teuthology.orchestra.run.smithi002.stderr:2024-01-15T18:21:04.120+0000 7f5c1b200640 1 -- 172.21.15.2:0/2326457766 wait complete. 2024-01-15T18:21:04.141 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-15T18:21:04.141 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-15T18:21:04.141 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-15T18:21:04.141 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:04.159 INFO:tasks.ceph.mds.a:Stopped 2024-01-15T18:21:04.159 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-15T18:21:04.159 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:04.225 INFO:tasks.ceph.mds.c:Stopped 2024-01-15T18:21:04.225 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-15T18:21:04.226 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:04.251 INFO:tasks.ceph.mds.b:Stopped 2024-01-15T18:21:04.251 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-15T18:21:04.251 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:04.320 INFO:tasks.ceph.mds.d:Stopped 2024-01-15T18:21:04.320 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-15T18:21:04.321 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-15T18:21:04.321 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:05.361 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:05.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:08.216 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-15T18:21:10.423 INFO:tasks.ceph.osd.0:Stopped 2024-01-15T18:21:10.423 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-15T18:21:10.423 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:10.516 INFO:tasks.ceph.osd.1:Stopped 2024-01-15T18:21:10.517 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-15T18:21:10.517 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:14.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-15T18:21:14.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-15T18:21:16.618 INFO:tasks.ceph.osd.2:Stopped 2024-01-15T18:21:16.619 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-15T18:21:16.619 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:16.696 INFO:tasks.ceph.osd.3:Stopped 2024-01-15T18:21:16.697 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-15T18:21:16.697 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:20.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-01-15T18:21:20.031 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-15T18:21:22.798 INFO:tasks.ceph.osd.4:Stopped 2024-01-15T18:21:22.799 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-15T18:21:22.799 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:22.873 INFO:tasks.ceph.osd.5:Stopped 2024-01-15T18:21:22.874 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-15T18:21:22.874 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:22.946 INFO:tasks.ceph.osd.6:Stopped 2024-01-15T18:21:22.946 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-15T18:21:22.946 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:23.033 INFO:tasks.ceph.osd.7:Stopped 2024-01-15T18:21:23.034 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-15T18:21:23.034 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-15T18:21:23.034 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:25.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-15T18:21:25.435 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-01-15T18:21:29.136 INFO:tasks.ceph.mgr.y:Stopped 2024-01-15T18:21:29.136 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-15T18:21:29.136 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:30.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-15T18:21:30.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-01-15T18:21:35.238 INFO:tasks.ceph.mgr.x:Stopped 2024-01-15T18:21:35.238 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-15T18:21:35.238 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-15T18:21:35.239 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:35.293 INFO:tasks.ceph.mon.a:Stopped 2024-01-15T18:21:35.294 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-15T18:21:35.294 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:35.340 INFO:tasks.ceph.mon.b:Stopped 2024-01-15T18:21:35.340 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-15T18:21:35.340 INFO:teuthology.orchestra.run:waiting for 300 2024-01-15T18:21:35.376 INFO:tasks.ceph.mon.c:Stopped 2024-01-15T18:21:35.376 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-15T18:21:35.377 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-01-15T18:21:35.490 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:21:35.490 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-15T18:21:35.499 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:35.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:35.631 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:21:35.631 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-15T18:21:35.740 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-01-15T18:21:35.780 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:21:35.780 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-15T18:21:35.951 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:21:35.952 DEBUG:teuthology.orchestra.run.smithi002:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-15T18:21:36.174 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:21:36.174 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-15T18:21:36.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:21:36.395 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-15T18:21:36.554 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:21:36.555 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-15T18:21:36.720 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:21:36.721 DEBUG:teuthology.orchestra.run.smithi137:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-15T18:21:36.893 INFO:tasks.ceph:Archiving mon data... 2024-01-15T18:21:36.895 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/data/mon.a.tgz 2024-01-15T18:21:36.895 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-01-15T18:21:36.955 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.MYzDRnPhWw 2024-01-15T18:21:36.955 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MYzDRnPhWw 2024-01-15T18:21:37.269 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.MYzDRnPhWw 2024-01-15T18:21:37.433 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.MYzDRnPhWw is 1MB 2024-01-15T18:21:37.555 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.MYzDRnPhWw 2024-01-15T18:21:37.626 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/data/mon.b.tgz 2024-01-15T18:21:37.626 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-15T18:21:37.684 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.pplcX0WkaC 2024-01-15T18:21:37.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pplcX0WkaC 2024-01-15T18:21:37.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.pplcX0WkaC 2024-01-15T18:21:38.147 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.pplcX0WkaC is 1MB 2024-01-15T18:21:38.267 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.pplcX0WkaC 2024-01-15T18:21:38.324 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/data/mon.c.tgz 2024-01-15T18:21:38.325 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-01-15T18:21:38.376 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.zfLClXPABm 2024-01-15T18:21:38.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zfLClXPABm 2024-01-15T18:21:38.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.zfLClXPABm 2024-01-15T18:21:38.812 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.zfLClXPABm is 1MB 2024-01-15T18:21:38.931 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.zfLClXPABm 2024-01-15T18:21:38.986 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-15T18:21:38.987 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-15T18:21:38.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-15T18:21:39.151 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-15T18:21:39.210 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-15T18:21:39.210 INFO:tasks.ceph:Archiving crash dumps... 2024-01-15T18:21:39.212 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/remote/smithi002/crash 2024-01-15T18:21:39.212 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-15T18:21:39.316 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/remote/smithi137/crash 2024-01-15T18:21:39.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-15T18:21:39.413 INFO:tasks.ceph:Shutting down logrotate 2024-01-15T18:21:39.414 DEBUG:teuthology.orchestra.run.smithi002:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:39.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:39.546 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:39.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-15T18:21:39.743 INFO:tasks.ceph:Compressing logs... 2024-01-15T18:21:39.743 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-15T18:21:39.751 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-15T18:21:39.841 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-15T18:21:39.843 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36260.log 2024-01-15T18:21:39.843 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50942.log 2024-01-15T18:21:39.844 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36340.log 2024-01-15T18:21:39.844 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51075.log 2024-01-15T18:21:39.845 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36300.log 2024-01-15T18:21:39.845 INFO:teuthology.orchestra.run.smithi137.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.36260.log.gz 2024-01-15T18:21:39.846 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81102.log 2024-01-15T18:21:39.846 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50942.log.gz 2024-01-15T18:21:39.847 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47512.log 2024-01-15T18:21:39.847 INFO:teuthology.orchestra.run.smithi002.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51075.log.gz 2024-01-15T18:21:39.848 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36420.log 2024-01-15T18:21:39.848 INFO:teuthology.orchestra.run.smithi137.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36340.log.gz 2024-01-15T18:21:39.848 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36300.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.36300.log.gz 2024-01-15T18:21:39.848 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-15T18:21:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81102.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81102.log.gz 2024-01-15T18:21:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44430.log 2024-01-15T18:21:39.849 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75886.log 2024-01-15T18:21:39.850 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47512.log.gz 2024-01-15T18:21:39.851 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36420.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-15T18:21:39.851 INFO:teuthology.orchestra.run.smithi137.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36420.log.gz 2024-01-15T18:21:39.851 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47446.log 2024-01-15T18:21:39.851 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79396.log 2024-01-15T18:21:39.851 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44430.log.gz 2024-01-15T18:21:39.852 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-15T18:21:39.853 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75886.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.75886.log.gz --verbose 2024-01-15T18:21:39.853 INFO:teuthology.orchestra.run.smithi002.stderr: -- /var/log/ceph/ceph-client.admin.74115.log 2024-01-15T18:21:39.853 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71260.log 2024-01-15T18:21:39.853 INFO:teuthology.orchestra.run.smithi002.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.79396.log.gz 2024-01-15T18:21:39.854 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.47446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41928.log 2024-01-15T18:21:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49448.log 2024-01-15T18:21:39.855 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74115.log.gz 2024-01-15T18:21:39.855 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41971.log 2024-01-15T18:21:39.855 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71260.log.gz 2024-01-15T18:21:39.856 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45936.log 2024-01-15T18:21:39.857 INFO:teuthology.orchestra.run.smithi137.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-15T18:21:39.857 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70259.log 2024-01-15T18:21:39.858 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49448.log.gz 2024-01-15T18:21:39.858 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66127.log 2024-01-15T18:21:39.858 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41971.log.gz 2024-01-15T18:21:39.859 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.3.41928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46689.log 2024-01-15T18:21:39.859 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.45936.log: 91.5% -- replaced with /var/log/ceph/ceph-client.3.41928.log.gz 2024-01-15T18:21:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70259.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70259.log.gz 2024-01-15T18:21:39.859 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70133.log 2024-01-15T18:21:39.860 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42662.log 2024-01-15T18:21:39.861 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66127.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66127.log.gz 2024-01-15T18:21:39.861 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68195.log 2024-01-15T18:21:39.861 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65497.log 2024-01-15T18:21:39.862 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.46689.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-15T18:21:39.862 INFO:teuthology.orchestra.run.smithi137.stderr: 96.3% -- replaced with /var/log/ceph/ceph-client.1.47446.log.gz 2024-01-15T18:21:39.862 INFO:teuthology.orchestra.run.smithi137.stderr: 94.3% -- replaced with /var/log/ceph/ceph-client.1.45936.log.gz 2024-01-15T18:21:39.863 INFO:teuthology.orchestra.run.smithi002.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70133.log.gz 2024-01-15T18:21:39.863 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68195.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68195.log.gz 2024-01-15T18:21:39.863 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48358.log 2024-01-15T18:21:39.863 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65497.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-15T18:21:39.864 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.42662.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-15T18:21:39.864 INFO:teuthology.orchestra.run.smithi137.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.1.46689.log.gz 2024-01-15T18:21:39.865 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36380.log 2024-01-15T18:21:39.865 INFO:teuthology.orchestra.run.smithi137.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.1.42662.log.gz 2024-01-15T18:21:39.865 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65497.log.gz 2024-01-15T18:21:39.865 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48358.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48358.log.gz 2024-01-15T18:21:39.865 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50640.log 2024-01-15T18:21:39.866 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45006.log 2024-01-15T18:21:39.867 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66919.log 2024-01-15T18:21:39.867 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36460.log 2024-01-15T18:21:39.867 INFO:teuthology.orchestra.run.smithi137.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36380.log.gz 2024-01-15T18:21:39.868 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50640.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50640.log.gz 2024-01-15T18:21:39.868 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68237.log 2024-01-15T18:21:39.869 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70613.log 2024-01-15T18:21:39.869 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66919.log.gz 2024-01-15T18:21:39.870 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.45006.log: /var/log/ceph/ceph-client.admin.36460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41408.log 2024-01-15T18:21:39.870 INFO:teuthology.orchestra.run.smithi137.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.36460.log.gz 2024-01-15T18:21:39.870 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68237.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68237.log.gz 2024-01-15T18:21:39.870 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68647.log 2024-01-15T18:21:39.872 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-15T18:21:39.872 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65101.log 2024-01-15T18:21:39.872 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70613.log.gz 2024-01-15T18:21:39.873 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68647.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68647.log.gz 2024-01-15T18:21:39.873 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48948.log 2024-01-15T18:21:39.873 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.41408.log: 96.0% -- replaced with /var/log/ceph/ceph-client.1.45006.log.gz 2024-01-15T18:21:39.873 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44205.log 2024-01-15T18:21:39.874 INFO:teuthology.orchestra.run.smithi137.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.1.41408.log.gz 2024-01-15T18:21:39.874 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59503.log 2024-01-15T18:21:39.875 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65101.log.gz 2024-01-15T18:21:39.875 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48948.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48948.log.gz 2024-01-15T18:21:39.875 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2024-01-15T18:21:39.876 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36540.log 2024-01-15T18:21:39.876 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70939.log 2024-01-15T18:21:39.876 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59503.log.gz 2024-01-15T18:21:39.877 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.44205.log: /var/log/ceph/ceph-client.admin.36540.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.36540.log.gz 2024-01-15T18:21:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46265.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-01-15T18:21:39.877 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76806.log 2024-01-15T18:21:39.879 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51934.log 2024-01-15T18:21:39.880 INFO:teuthology.orchestra.run.smithi002.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70939.log.gz 2024-01-15T18:21:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76806.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.76806.log.gz 2024-01-15T18:21:39.880 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70697.log 2024-01-15T18:21:39.882 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36500.log 2024-01-15T18:21:39.882 INFO:teuthology.orchestra.run.smithi137.stderr: 95.0% -- replaced with /var/log/ceph/ceph-client.1.44205.log.gz 2024-01-15T18:21:39.882 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-15T18:21:39.882 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.admin.36500.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36500.log.gz 2024-01-15T18:21:39.882 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-15T18:21:39.883 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75294.log 2024-01-15T18:21:39.883 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7%/var/log/ceph/ceph-client.admin.70697.log: -- replaced with /var/log/ceph/ceph-client.admin.51934.log.gz 2024-01-15T18:21:39.883 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70697.log.gz 2024-01-15T18:21:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61152.log 2024-01-15T18:21:39.884 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63538.log 2024-01-15T18:21:39.885 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43390.log 2024-01-15T18:21:39.886 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75294.log.gz 2024-01-15T18:21:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54643.log 2024-01-15T18:21:39.886 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61152.log.gz 2024-01-15T18:21:39.886 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76848.log 2024-01-15T18:21:39.886 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63538.log.gz 2024-01-15T18:21:39.887 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81186.log 2024-01-15T18:21:39.887 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54643.log.gz 2024-01-15T18:21:39.888 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47800.log 2024-01-15T18:21:39.888 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.76848.log.gz 2024-01-15T18:21:39.889 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-client.1.43390.log: gzip -5 --verbose -- 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-15T18:21:39.889 INFO:teuthology.orchestra.run.smithi137.stderr: /var/log/ceph/ceph-mds.d.log 2024-01-15T18:21:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55149.log 2024-01-15T18:21:39.890 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81186.log.gz 2024-01-15T18:21:39.890 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48756.log 2024-01-15T18:21:39.890 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47800.log.gz 2024-01-15T18:21:39.891 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73787.log 2024-01-15T18:21:39.891 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55149.log.gz 2024-01-15T18:21:39.892 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-15T18:21:39.892 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74964.log 2024-01-15T18:21:39.892 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48756.log.gz 2024-01-15T18:21:39.893 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74785.log 2024-01-15T18:21:39.893 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73787.log.gz 2024-01-15T18:21:39.894 INFO:teuthology.orchestra.run.smithi137.stderr: 94.7%/var/log/ceph/ceph-mon.c.log: -- replaced with /var/log/ceph/ceph-client.1.43390.log.gz 2024-01-15T18:21:39.895 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48588.log 2024-01-15T18:21:39.895 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74964.log.gz 2024-01-15T18:21:39.895 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46323.log 2024-01-15T18:21:39.896 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.74785.log.gz 2024-01-15T18:21:39.896 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75466.log 2024-01-15T18:21:39.897 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48588.log.gz 2024-01-15T18:21:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46323.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46323.log.gz 2024-01-15T18:21:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81144.log 2024-01-15T18:21:39.898 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58911.log 2024-01-15T18:21:39.899 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.75466.log.gz 2024-01-15T18:21:39.899 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55434.log 2024-01-15T18:21:39.900 INFO:teuthology.orchestra.run.smithi002.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.81144.log.gz 2024-01-15T18:21:39.901 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53347.log 2024-01-15T18:21:39.901 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58911.log.gz 2024-01-15T18:21:39.902 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64207.log 2024-01-15T18:21:39.902 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55434.log.gz 2024-01-15T18:21:39.903 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41976.log 2024-01-15T18:21:39.903 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53347.log.gz 2024-01-15T18:21:39.904 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55029.log 2024-01-15T18:21:39.904 INFO:teuthology.orchestra.run.smithi002.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.64207.log.gz 2024-01-15T18:21:39.905 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71302.log 2024-01-15T18:21:39.905 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41976.log.gz 2024-01-15T18:21:39.906 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47884.log 2024-01-15T18:21:39.906 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55029.log.gz 2024-01-15T18:21:39.907 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72127.log 2024-01-15T18:21:39.908 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71302.log.gz 2024-01-15T18:21:39.909 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47884.log: gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.42724.log -- replaced with /var/log/ceph/ceph-client.admin.47884.log.gz 2024-01-15T18:21:39.909 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-01-15T18:21:39.910 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53389.log 2024-01-15T18:21:39.910 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72127.log.gz 2024-01-15T18:21:39.911 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81700.log 2024-01-15T18:21:39.911 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42724.log.gz 2024-01-15T18:21:39.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77236.log 2024-01-15T18:21:39.912 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53389.log.gz 2024-01-15T18:21:39.913 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44303.log 2024-01-15T18:21:39.913 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81700.log.gz 2024-01-15T18:21:39.914 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45018.log 2024-01-15T18:21:39.914 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.77236.log.gz 2024-01-15T18:21:39.915 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65623.log 2024-01-15T18:21:39.915 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44303.log.gz 2024-01-15T18:21:39.916 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45018.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45018.log.gz 2024-01-15T18:21:39.916 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71607.log 2024-01-15T18:21:39.917 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59870.log 2024-01-15T18:21:39.917 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65623.log.gz 2024-01-15T18:21:39.918 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64125.log 2024-01-15T18:21:39.918 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71607.log.gz 2024-01-15T18:21:39.919 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70301.log 2024-01-15T18:21:39.919 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59870.log.gz 2024-01-15T18:21:39.920 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50324.log 2024-01-15T18:21:39.920 INFO:teuthology.orchestra.run.smithi002.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.64125.log.gz 2024-01-15T18:21:39.921 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59377.log 2024-01-15T18:21:39.921 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70301.log.gz 2024-01-15T18:21:39.922 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74533.log 2024-01-15T18:21:39.923 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50324.log.gz 2024-01-15T18:21:39.924 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65959.log 2024-01-15T18:21:39.924 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59377.log.gz 2024-01-15T18:21:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74533.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74533.log.gz 2024-01-15T18:21:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55900.log 2024-01-15T18:21:39.925 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80682.log 2024-01-15T18:21:39.926 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65959.log.gz 2024-01-15T18:21:39.926 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46569.log 2024-01-15T18:21:39.927 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55900.log.gz 2024-01-15T18:21:39.928 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58471.log 2024-01-15T18:21:39.928 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.80682.log.gz 2024-01-15T18:21:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46569.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46569.log.gz 2024-01-15T18:21:39.929 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48486.log 2024-01-15T18:21:39.930 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74617.log 2024-01-15T18:21:39.930 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58471.log.gz 2024-01-15T18:21:39.931 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74236.log 2024-01-15T18:21:39.931 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48486.log.gz 2024-01-15T18:21:39.932 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62400.log 2024-01-15T18:21:39.933 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74617.log.gz 2024-01-15T18:21:39.933 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74236.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74236.log.gz 2024-01-15T18:21:39.934 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66505.log 2024-01-15T18:21:39.934 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55985.log 2024-01-15T18:21:39.935 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62400.log.gz 2024-01-15T18:21:39.936 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48052.log 2024-01-15T18:21:39.936 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66505.log.gz 2024-01-15T18:21:39.937 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55392.log 2024-01-15T18:21:39.937 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55985.log.gz 2024-01-15T18:21:39.938 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79918.log 2024-01-15T18:21:39.939 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48052.log.gz 2024-01-15T18:21:39.940 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46905.log 2024-01-15T18:21:39.940 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55392.log.gz 2024-01-15T18:21:39.941 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59041.log 2024-01-15T18:21:39.941 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.79918.log.gz 2024-01-15T18:21:39.941 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41933.log 2024-01-15T18:21:39.941 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46905.log.gz 2024-01-15T18:21:39.942 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80766.log 2024-01-15T18:21:39.942 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59041.log.gz 2024-01-15T18:21:39.943 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78640.log 2024-01-15T18:21:39.944 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41933.log.gz 2024-01-15T18:21:39.945 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80766.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.80766.log.gz 2024-01-15T18:21:39.945 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61512.log 2024-01-15T18:21:39.946 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61320.log 2024-01-15T18:21:39.946 INFO:teuthology.orchestra.run.smithi002.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78640.log.gz 2024-01-15T18:21:39.946 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77661.log 2024-01-15T18:21:39.947 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61512.log.gz 2024-01-15T18:21:39.948 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43486.log 2024-01-15T18:21:39.948 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61320.log.gz 2024-01-15T18:21:39.949 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77661.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.77661.log.gz 2024-01-15T18:21:39.949 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68323.log 2024-01-15T18:21:39.950 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67827.log 2024-01-15T18:21:39.950 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43486.log.gz 2024-01-15T18:21:39.951 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-01-15T18:21:39.951 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68323.log.gz 2024-01-15T18:21:39.952 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46079.log 2024-01-15T18:21:39.952 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67827.log.gz 2024-01-15T18:21:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45963.log 2024-01-15T18:21:39.954 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-01-15T18:21:39.954 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80086.log 2024-01-15T18:21:39.954 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46079.log.gz 2024-01-15T18:21:39.955 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78089.log 2024-01-15T18:21:39.955 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45963.log.gz 2024-01-15T18:21:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80086.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80086.log.gz 2024-01-15T18:21:39.956 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75125.log 2024-01-15T18:21:39.957 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72211.log 2024-01-15T18:21:39.958 INFO:teuthology.orchestra.run.smithi002.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.78089.log.gz 2024-01-15T18:21:39.959 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75125.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.75125.log.gz 2024-01-15T18:21:39.959 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52062.log 2024-01-15T18:21:39.959 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44056.log 2024-01-15T18:21:39.960 INFO:teuthology.orchestra.run.smithi002.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72211.log.gz 2024-01-15T18:21:39.961 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46695.log 2024-01-15T18:21:39.961 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52062.log.gz 2024-01-15T18:21:39.962 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63832.log 2024-01-15T18:21:39.962 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44056.log.gz 2024-01-15T18:21:39.963 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67422.log 2024-01-15T18:21:39.963 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46695.log.gz 2024-01-15T18:21:39.964 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63832.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63832.log.gz 2024-01-15T18:21:39.964 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65581.log 2024-01-15T18:21:39.965 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68521.log 2024-01-15T18:21:39.965 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67422.log.gz 2024-01-15T18:21:39.966 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65581.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65581.log.gz 2024-01-15T18:21:39.966 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63310.log 2024-01-15T18:21:39.968 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67973.log 2024-01-15T18:21:39.968 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68521.log.gz 2024-01-15T18:21:39.969 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63310.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63310.log.gz 2024-01-15T18:21:39.969 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77278.log 2024-01-15T18:21:39.970 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57559.log 2024-01-15T18:21:39.970 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67973.log.gz 2024-01-15T18:21:39.971 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77278.log: gzip -5 --verbose 86.9% -- -- replaced with /var/log/ceph/ceph-client.admin.77278.log.gz 2024-01-15T18:21:39.971 INFO:teuthology.orchestra.run.smithi002.stderr: /var/log/ceph/ceph-client.admin.54197.log 2024-01-15T18:21:39.972 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56521.log 2024-01-15T18:21:39.972 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57559.log.gz 2024-01-15T18:21:39.973 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62974.log 2024-01-15T18:21:39.973 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54197.log.gz 2024-01-15T18:21:39.974 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56521.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56521.log.gz 2024-01-15T18:21:39.974 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55350.log 2024-01-15T18:21:39.975 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66253.log 2024-01-15T18:21:39.975 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62974.log.gz 2024-01-15T18:21:39.976 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55350.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55350.log.gz 2024-01-15T18:21:39.976 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62721.log 2024-01-15T18:21:39.977 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58955.log 2024-01-15T18:21:39.978 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66253.log.gz 2024-01-15T18:21:39.978 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62721.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.62721.log.gz 2024-01-15T18:21:39.979 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-01-15T18:21:39.980 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45753.log 2024-01-15T18:21:39.980 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58955.log.gz 2024-01-15T18:21:39.981 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44934.log 2024-01-15T18:21:39.981 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-01-15T18:21:39.982 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73543.log 2024-01-15T18:21:39.982 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45753.log.gz 2024-01-15T18:21:39.983 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44934.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44934.log.gz 2024-01-15T18:21:39.983 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67380.log 2024-01-15T18:21:39.984 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81525.log 2024-01-15T18:21:39.984 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73543.log.gz 2024-01-15T18:21:39.985 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67059.log 2024-01-15T18:21:39.985 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67380.log.gz 2024-01-15T18:21:39.986 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71470.log 2024-01-15T18:21:39.986 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81525.log.gz 2024-01-15T18:21:39.988 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67059.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67059.log.gz 2024-01-15T18:21:39.988 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81658.log 2024-01-15T18:21:39.988 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54475.log 2024-01-15T18:21:39.989 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71470.log.gz 2024-01-15T18:21:39.989 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66860.log 2024-01-15T18:21:39.990 INFO:teuthology.orchestra.run.smithi002.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.81658.log.gz 2024-01-15T18:21:39.991 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75252.log 2024-01-15T18:21:39.991 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54475.log.gz 2024-01-15T18:21:39.992 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59251.log 2024-01-15T18:21:39.992 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66860.log.gz 2024-01-15T18:21:39.992 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78048.log 2024-01-15T18:21:39.993 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75252.log.gz 2024-01-15T18:21:39.994 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58868.log 2024-01-15T18:21:39.994 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59251.log.gz 2024-01-15T18:21:39.995 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78048.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.78048.log.gz 2024-01-15T18:21:39.995 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75760.log 2024-01-15T18:21:39.996 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68279.log 2024-01-15T18:21:39.996 INFO:teuthology.orchestra.run.smithi002.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58868.log.gz 2024-01-15T18:21:39.997 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58030.log 2024-01-15T18:21:39.997 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.75760.log.gz 2024-01-15T18:21:39.998 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56647.log 2024-01-15T18:21:39.998 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68279.log.gz 2024-01-15T18:21:39.999 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65539.log 2024-01-15T18:21:39.999 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58030.log.gz 2024-01-15T18:21:40.000 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53566.log 2024-01-15T18:21:40.001 INFO:teuthology.orchestra.run.smithi002.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.56647.log.gz 2024-01-15T18:21:40.001 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72295.log 2024-01-15T18:21:40.002 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65539.log.gz 2024-01-15T18:21:40.002 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80002.log 2024-01-15T18:21:40.002 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2024-01-15T18:21:40.003 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50069.log 2024-01-15T18:21:40.003 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72295.log.gz 2024-01-15T18:21:40.005 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45060.log 2024-01-15T18:21:40.005 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80002.log.gz 2024-01-15T18:21:40.006 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50069.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50069.log.gz 2024-01-15T18:21:40.006 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58114.log 2024-01-15T18:21:40.007 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56131.log 2024-01-15T18:21:40.007 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45060.log.gz 2024-01-15T18:21:40.008 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-01-15T18:21:40.008 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58114.log.gz 2024-01-15T18:21:40.009 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56131.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77825.log 2024-01-15T18:21:40.009 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56131.log.gz 2024-01-15T18:21:40.010 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41308.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-01-15T18:21:40.010 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69300.log 2024-01-15T18:21:40.011 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-01-15T18:21:40.011 INFO:teuthology.orchestra.run.smithi002.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77825.log.gz 2024-01-15T18:21:40.012 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80432.log 2024-01-15T18:21:40.012 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69300.log.gz 2024-01-15T18:21:40.013 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79355.log 2024-01-15T18:21:40.013 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-01-15T18:21:40.014 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47968.log 2024-01-15T18:21:40.014 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80432.log.gz 2024-01-15T18:21:40.015 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73745.log 2024-01-15T18:21:40.015 INFO:teuthology.orchestra.run.smithi002.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.79355.log.gz 2024-01-15T18:21:40.016 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45669.log 2024-01-15T18:21:40.017 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47968.log.gz 2024-01-15T18:21:40.018 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70529.log 2024-01-15T18:21:40.018 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73745.log.gz 2024-01-15T18:21:40.019 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59828.log 2024-01-15T18:21:40.019 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45669.log.gz 2024-01-15T18:21:40.019 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77702.log 2024-01-15T18:21:40.020 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70529.log.gz 2024-01-15T18:21:40.021 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59828.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59828.log.gz 2024-01-15T18:21:40.021 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47325.log 2024-01-15T18:21:40.022 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44514.log 2024-01-15T18:21:40.022 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77702.log.gz 2024-01-15T18:21:40.023 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75844.log 2024-01-15T18:21:40.023 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47325.log.gz 2024-01-15T18:21:40.024 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62535.log 2024-01-15T18:21:40.024 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44514.log.gz 2024-01-15T18:21:40.025 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75844.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75844.log.gzgzip 2024-01-15T18:21:40.025 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2024-01-15T18:21:40.026 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64975.log 2024-01-15T18:21:40.027 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62535.log.gz 2024-01-15T18:21:40.027 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48224.log 2024-01-15T18:21:40.027 INFO:teuthology.orchestra.run.smithi002.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2024-01-15T18:21:40.028 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49757.log 2024-01-15T18:21:40.028 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64975.log.gz 2024-01-15T18:21:40.031 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58723.log 2024-01-15T18:21:40.031 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48224.log.gz 2024-01-15T18:21:40.032 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75508.log 2024-01-15T18:21:40.033 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49757.log.gz 2024-01-15T18:21:40.033 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51161.log 2024-01-15T18:21:40.033 INFO:teuthology.orchestra.run.smithi002.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58723.log.gz 2024-01-15T18:21:40.033 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63874.log 2024-01-15T18:21:40.033 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75508.log.gz 2024-01-15T18:21:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51161.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51161.log.gz 2024-01-15T18:21:40.034 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63580.log 2024-01-15T18:21:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62847.log 2024-01-15T18:21:40.035 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63874.log.gz 2024-01-15T18:21:40.035 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44850.log 2024-01-15T18:21:40.036 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63580.log.gz 2024-01-15T18:21:40.037 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78458.log 2024-01-15T18:21:40.037 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62847.log.gz 2024-01-15T18:21:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44850.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44850.log.gz 2024-01-15T18:21:40.038 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60802.log 2024-01-15T18:21:40.039 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2024-01-15T18:21:40.040 INFO:teuthology.orchestra.run.smithi002.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.78458.log.gz 2024-01-15T18:21:40.040 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42178.log 2024-01-15T18:21:40.041 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60802.log.gz 2024-01-15T18:21:40.042 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56605.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2024-01-15T18:21:40.042 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62316.log 2024-01-15T18:21:40.043 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41298.log 2024-01-15T18:21:40.043 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42178.log.gz 2024-01-15T18:21:40.044 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81740.log 2024-01-15T18:21:40.044 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62316.log.gz 2024-01-15T18:21:40.045 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41298.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41298.log.gz 2024-01-15T18:21:40.045 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72379.log 2024-01-15T18:21:40.046 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78722.log 2024-01-15T18:21:40.046 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.81740.log.gz 2024-01-15T18:21:40.047 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52707.log 2024-01-15T18:21:40.047 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72379.log.gz 2024-01-15T18:21:40.048 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78722.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78722.log.gz 2024-01-15T18:21:40.048 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64000.log 2024-01-15T18:21:40.049 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.52707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75634.log 2024-01-15T18:21:40.050 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43892.log 2024-01-15T18:21:40.050 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64000.log.gz 2024-01-15T18:21:40.051 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75634.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75634.log.gz 2024-01-15T18:21:40.052 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73239.log 2024-01-15T18:21:40.052 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.2.43892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81270.log 2024-01-15T18:21:40.053 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73239.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73239.log.gz 2024-01-15T18:21:40.054 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.2.43892.log.gz 2024-01-15T18:21:40.054 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-01-15T18:21:40.055 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80348.log 2024-01-15T18:21:40.055 INFO:teuthology.orchestra.run.smithi002.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81270.log.gz 2024-01-15T18:21:40.056 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42346.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-01-15T18:21:40.056 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69384.log 2024-01-15T18:21:40.057 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76596.log 2024-01-15T18:21:40.057 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80348.log.gz 2024-01-15T18:21:40.058 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69384.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69384.log.gz 2024-01-15T18:21:40.059 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71143.log 2024-01-15T18:21:40.060 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59167.log 2024-01-15T18:21:40.060 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76596.log.gz 2024-01-15T18:21:40.061 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46449.log 2024-01-15T18:21:40.061 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71143.log.gz 2024-01-15T18:21:40.062 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59167.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59167.log.gz 2024-01-15T18:21:40.062 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46005.log 2024-01-15T18:21:40.063 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63142.log 2024-01-15T18:21:40.064 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46449.log.gz 2024-01-15T18:21:40.065 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68436.log 2024-01-15T18:21:40.065 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46005.log.gz 2024-01-15T18:21:40.066 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70798.log 2024-01-15T18:21:40.066 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63142.log.gz 2024-01-15T18:21:40.067 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47409.log 2024-01-15T18:21:40.067 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68436.log.gz 2024-01-15T18:21:40.068 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71428.log 2024-01-15T18:21:40.068 INFO:teuthology.orchestra.run.smithi002.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.70798.log.gz 2024-01-15T18:21:40.069 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68689.log 2024-01-15T18:21:40.070 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47409.log.gz 2024-01-15T18:21:40.071 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49406.log 2024-01-15T18:21:40.071 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.71428.log.gz 2024-01-15T18:21:40.072 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68689.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68689.log.gzgzip 2024-01-15T18:21:40.072 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-01-15T18:21:40.073 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49406.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49406.log.gz 2024-01-15T18:21:40.073 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42766.log 2024-01-15T18:21:40.074 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44556.log 2024-01-15T18:21:40.074 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-01-15T18:21:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42766.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42766.log.gz 2024-01-15T18:21:40.075 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41977.log 2024-01-15T18:21:40.077 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61236.log 2024-01-15T18:21:40.077 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44556.log.gz 2024-01-15T18:21:40.078 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63100.log 2024-01-15T18:21:40.078 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41977.log.gz 2024-01-15T18:21:40.079 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47241.log 2024-01-15T18:21:40.079 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61236.log.gz 2024-01-15T18:21:40.080 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58216.log 2024-01-15T18:21:40.081 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63100.log.gz 2024-01-15T18:21:40.081 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44472.log 2024-01-15T18:21:40.081 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47241.log.gz 2024-01-15T18:21:40.083 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80306.log 2024-01-15T18:21:40.083 INFO:teuthology.orchestra.run.smithi002.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58216.log.gz 2024-01-15T18:21:40.084 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70571.log 2024-01-15T18:21:40.084 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44472.log.gz 2024-01-15T18:21:40.085 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75970.log 2024-01-15T18:21:40.085 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.80306.log.gz 2024-01-15T18:21:40.086 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71853.log 2024-01-15T18:21:40.086 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70571.log.gz 2024-01-15T18:21:40.087 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68605.log 2024-01-15T18:21:40.087 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.75970.log.gz 2024-01-15T18:21:40.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43996.log 2024-01-15T18:21:40.088 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71853.log.gz 2024-01-15T18:21:40.089 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61970.log 2024-01-15T18:21:40.089 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68605.log.gz 2024-01-15T18:21:40.090 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.43996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64704.log 2024-01-15T18:21:40.091 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43996.log.gz 2024-01-15T18:21:40.092 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54240.log 2024-01-15T18:21:40.092 INFO:teuthology.orchestra.run.smithi002.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.61970.log.gz 2024-01-15T18:21:40.093 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78886.log 2024-01-15T18:21:40.093 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64704.log.gz 2024-01-15T18:21:40.094 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54240.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54240.log.gz 2024-01-15T18:21:40.094 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43330.log 2024-01-15T18:21:40.095 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45711.log 2024-01-15T18:21:40.095 INFO:teuthology.orchestra.run.smithi002.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.78886.log.gz 2024-01-15T18:21:40.096 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.43330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62619.log 2024-01-15T18:21:40.097 INFO:teuthology.orchestra.run.smithi002.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.43330.log.gz 2024-01-15T18:21:40.097 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2024-01-15T18:21:40.098 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45711.log.gz 2024-01-15T18:21:40.098 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62619.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62619.log.gz 2024-01-15T18:21:40.099 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65455.log 2024-01-15T18:21:40.100 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44259.log 2024-01-15T18:21:40.100 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2024-01-15T18:21:40.101 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65455.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65455.log.gz 2024-01-15T18:21:40.101 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70739.log 2024-01-15T18:21:40.102 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67338.log 2024-01-15T18:21:40.102 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44259.log.gz 2024-01-15T18:21:40.103 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70739.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70739.log.gz 2024-01-15T18:21:40.103 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57171.log 2024-01-15T18:21:40.105 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60335.log 2024-01-15T18:21:40.105 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.67338.log.gz 2024-01-15T18:21:40.106 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57171.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57171.log.gz 2024-01-15T18:21:40.106 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47756.log 2024-01-15T18:21:40.107 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61446.log 2024-01-15T18:21:40.107 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60335.log.gz 2024-01-15T18:21:40.108 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47756.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47756.log.gz 2024-01-15T18:21:40.108 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79734.log 2024-01-15T18:21:40.109 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64350.log 2024-01-15T18:21:40.110 INFO:teuthology.orchestra.run.smithi002.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.61446.log.gz 2024-01-15T18:21:40.110 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79734.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79734.log.gz 2024-01-15T18:21:40.110 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63226.log 2024-01-15T18:21:40.112 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74659.log 2024-01-15T18:21:40.112 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64350.log.gz 2024-01-15T18:21:40.113 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63226.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63226.log.gz 2024-01-15T18:21:40.113 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49842.log 2024-01-15T18:21:40.114 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66295.log 2024-01-15T18:21:40.114 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74659.log.gz 2024-01-15T18:21:40.115 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49842.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49842.log.gz 2024-01-15T18:21:40.115 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78763.log 2024-01-15T18:21:40.116 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66631.log 2024-01-15T18:21:40.116 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66295.log.gz 2024-01-15T18:21:40.117 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78763.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78763.log.gz 2024-01-15T18:21:40.117 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71895.log 2024-01-15T18:21:40.118 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74701.log 2024-01-15T18:21:40.118 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66631.log.gz 2024-01-15T18:21:40.119 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71895.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71895.log.gz 2024-01-15T18:21:40.120 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45549.log 2024-01-15T18:21:40.121 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51796.log 2024-01-15T18:21:40.121 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.74701.log.gz 2024-01-15T18:21:40.122 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45549.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45549.log.gz 2024-01-15T18:21:40.122 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-01-15T18:21:40.123 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56437.log 2024-01-15T18:21:40.123 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51796.log.gz 2024-01-15T18:21:40.124 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41641.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-01-15T18:21:40.124 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53608.log 2024-01-15T18:21:40.125 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78845.log 2024-01-15T18:21:40.125 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56437.log.gz 2024-01-15T18:21:40.126 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53608.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53608.log.gz 2024-01-15T18:21:40.127 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57946.log 2024-01-15T18:21:40.128 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79776.log 2024-01-15T18:21:40.128 INFO:teuthology.orchestra.run.smithi002.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78845.log.gz 2024-01-15T18:21:40.129 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54287.log 2024-01-15T18:21:40.129 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57946.log.gz 2024-01-15T18:21:40.130 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-01-15T18:21:40.130 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79776.log.gz 2024-01-15T18:21:40.131 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54287.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-15T18:21:40.131 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54287.log.gz 2024-01-15T18:21:40.132 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2024-01-15T18:21:40.132 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-01-15T18:21:40.133 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76012.log 2024-01-15T18:21:40.135 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2024-01-15T18:21:40.135 INFO:teuthology.orchestra.run.smithi002.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2024-01-15T18:21:40.136 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2024-01-15T18:21:40.137 INFO:teuthology.orchestra.run.smithi002.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.76012.log.gz 2024-01-15T18:21:40.138 INFO:teuthology.orchestra.run.smithi002.stderr: 89.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-15T18:21:40.138 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80558.log 2024-01-15T18:21:40.138 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2024-01-15T18:21:40.139 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66379.log 2024-01-15T18:21:40.139 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2024-01-15T18:21:40.140 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65413.log 2024-01-15T18:21:40.140 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.80558.log.gz 2024-01-15T18:21:40.141 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67464.log 2024-01-15T18:21:40.142 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66379.log.gz 2024-01-15T18:21:40.143 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-01-15T18:21:40.143 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65413.log.gz 2024-01-15T18:21:40.144 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79619.log 2024-01-15T18:21:40.144 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67464.log.gz 2024-01-15T18:21:40.145 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41086.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz 2024-01-15T18:21:40.145 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.81312.log 2024-01-15T18:21:40.146 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49901.log 2024-01-15T18:21:40.146 INFO:teuthology.orchestra.run.smithi002.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.79619.log.gz 2024-01-15T18:21:40.147 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-01-15T18:21:40.148 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81312.log.gz 2024-01-15T18:21:40.149 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45465.log 2024-01-15T18:21:40.149 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49901.log.gz 2024-01-15T18:21:40.150 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48180.log 2024-01-15T18:21:40.150 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-01-15T18:21:40.151 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48010.log 2024-01-15T18:21:40.151 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45465.log.gz 2024-01-15T18:21:40.152 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48180.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.48180.log.gz 2024-01-15T18:21:40.152 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42808.log 2024-01-15T18:21:40.153 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74869.log 2024-01-15T18:21:40.153 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48010.log.gz 2024-01-15T18:21:40.154 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47199.log 2024-01-15T18:21:40.155 INFO:teuthology.orchestra.run.smithi002.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42808.log.gz 2024-01-15T18:21:40.156 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50497.log 2024-01-15T18:21:40.156 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74869.log.gz 2024-01-15T18:21:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47199.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47199.log.gz 2024-01-15T18:21:40.157 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52230.log 2024-01-15T18:21:40.158 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71344.log 2024-01-15T18:21:40.158 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50497.log.gz 2024-01-15T18:21:40.160 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51287.log 2024-01-15T18:21:40.160 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52230.log.gz 2024-01-15T18:21:40.161 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48672.log 2024-01-15T18:21:40.161 INFO:teuthology.orchestra.run.smithi002.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.71344.log.gz 2024-01-15T18:21:40.162 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50780.log 2024-01-15T18:21:40.162 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51287.log.gz 2024-01-15T18:21:40.163 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2024-01-15T18:21:40.163 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48672.log.gz 2024-01-15T18:21:40.164 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64434.log 2024-01-15T18:21:40.164 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50780.log.gz 2024-01-15T18:21:40.165 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76536.log 2024-01-15T18:21:40.166 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2024-01-15T18:21:40.167 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51119.log 2024-01-15T18:21:40.167 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64434.log.gz 2024-01-15T18:21:40.168 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62012.log 2024-01-15T18:21:40.169 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76536.log.gz 2024-01-15T18:21:40.169 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73972.log 2024-01-15T18:21:40.169 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51119.log.gz 2024-01-15T18:21:40.171 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65017.log 2024-01-15T18:21:40.171 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62012.log.gz 2024-01-15T18:21:40.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51245.log 2024-01-15T18:21:40.172 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73972.log.gz 2024-01-15T18:21:40.173 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65017.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.65017.log.gz 2024-01-15T18:21:40.173 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72421.log 2024-01-15T18:21:40.174 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74458.log 2024-01-15T18:21:40.175 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51245.log.gz 2024-01-15T18:21:40.176 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72421.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72421.log.gz 2024-01-15T18:21:40.176 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55191.log 2024-01-15T18:21:40.177 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49364.log 2024-01-15T18:21:40.177 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74458.log.gz 2024-01-15T18:21:40.179 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79150.log 2024-01-15T18:21:40.179 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55191.log.gz 2024-01-15T18:21:40.180 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50900.log 2024-01-15T18:21:40.180 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49364.log.gz 2024-01-15T18:21:40.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79150.log: gzip -5 --verbose -- 85.3% /var/log/ceph/ceph-client.admin.63664.log 2024-01-15T18:21:40.181 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79150.log.gz 2024-01-15T18:21:40.182 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55560.log 2024-01-15T18:21:40.182 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50900.log.gz 2024-01-15T18:21:40.183 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63664.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.51546.log -- replaced with /var/log/ceph/ceph-client.admin.63664.log.gz 2024-01-15T18:21:40.184 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-01-15T18:21:40.185 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45627.log 2024-01-15T18:21:40.185 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55560.log.gz 2024-01-15T18:21:40.186 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72025.log 2024-01-15T18:21:40.186 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51546.log.gz 2024-01-15T18:21:40.187 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61404.log 2024-01-15T18:21:40.187 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45627.log.gz 2024-01-15T18:21:40.189 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72025.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.72025.log.gz 2024-01-15T18:21:40.189 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44892.log 2024-01-15T18:21:40.190 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49800.log 2024-01-15T18:21:40.190 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61404.log.gz 2024-01-15T18:21:40.191 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68563.log 2024-01-15T18:21:40.191 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44892.log.gz 2024-01-15T18:21:40.192 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.76355.log 2024-01-15T18:21:40.193 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49800.log.gz 2024-01-15T18:21:40.194 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67506.log 2024-01-15T18:21:40.194 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68563.log.gz 2024-01-15T18:21:40.195 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.76355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53975.log 2024-01-15T18:21:40.197 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79860.log 2024-01-15T18:21:40.197 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67506.log.gz 2024-01-15T18:21:40.198 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53975.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53975.log.gz 2024-01-15T18:21:40.198 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57213.log 2024-01-15T18:21:40.200 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53016.log 2024-01-15T18:21:40.200 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79860.log: /var/log/ceph/ceph-client.admin.57213.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.79860.log.gz 2024-01-15T18:21:40.201 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48402.log 2024-01-15T18:21:40.201 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57213.log.gz 2024-01-15T18:21:40.203 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41628.log 2024-01-15T18:21:40.203 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53016.log.gz 2024-01-15T18:21:40.204 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48402.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48402.log.gz 2024-01-15T18:21:40.205 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53305.log 2024-01-15T18:21:40.206 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72463.log 2024-01-15T18:21:40.207 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41628.log.gz 2024-01-15T18:21:40.207 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72337.log 2024-01-15T18:21:40.208 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53305.log.gz 2024-01-15T18:21:40.209 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75338.log 2024-01-15T18:21:40.209 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72463.log.gz 2024-01-15T18:21:40.210 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41631.log 2024-01-15T18:21:40.211 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75338.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.72337.log.gz 2024-01-15T18:21:40.212 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.75044.log 2024-01-15T18:21:40.212 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.75338.log.gz 2024-01-15T18:21:40.213 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41631.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41631.log.gz 2024-01-15T18:21:40.213 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77338.log 2024-01-15T18:21:40.214 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58259.log 2024-01-15T18:21:40.215 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75044.log.gz 2024-01-15T18:21:40.216 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62661.log 2024-01-15T18:21:40.216 INFO:teuthology.orchestra.run.smithi002.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77338.log.gz 2024-01-15T18:21:40.217 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58259.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.58259.log.gz 2024-01-15T18:21:40.217 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.50413.log 2024-01-15T18:21:40.218 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.69158.log 2024-01-15T18:21:40.219 INFO:teuthology.orchestra.run.smithi002.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.62661.log.gz 2024-01-15T18:21:40.220 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61362.log 2024-01-15T18:21:40.221 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50413.log.gz 2024-01-15T18:21:40.222 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.69158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45323.log 2024-01-15T18:21:40.223 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50455.log 2024-01-15T18:21:40.223 INFO:teuthology.orchestra.run.smithi002.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.61362.log.gz 2024-01-15T18:21:40.224 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50822.log 2024-01-15T18:21:40.225 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45323.log.gz 2024-01-15T18:21:40.226 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70403.log 2024-01-15T18:21:40.226 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50455.log.gz 2024-01-15T18:21:40.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56713.log 2024-01-15T18:21:40.228 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50822.log.gz 2024-01-15T18:21:40.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79232.log 2024-01-15T18:21:40.230 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.70403.log.gz 2024-01-15T18:21:40.230 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36322.log 2024-01-15T18:21:40.230 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56713.log.gz 2024-01-15T18:21:40.231 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81018.log 2024-01-15T18:21:40.232 INFO:teuthology.orchestra.run.smithi002.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.79232.log.gz 2024-01-15T18:21:40.233 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63016.log 2024-01-15T18:21:40.233 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36322.log.gz 2024-01-15T18:21:40.235 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44640.log 2024-01-15T18:21:40.235 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81018.log.gz 2024-01-15T18:21:40.236 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67221.log 2024-01-15T18:21:40.236 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63016.log.gz 2024-01-15T18:21:40.237 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50984.log 2024-01-15T18:21:40.238 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44640.log.gz 2024-01-15T18:21:40.239 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45795.log 2024-01-15T18:21:40.239 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67221.log.gz 2024-01-15T18:21:40.240 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77866.log 2024-01-15T18:21:40.240 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50984.log.gz 2024-01-15T18:21:40.242 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66211.log 2024-01-15T18:21:40.242 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45795.log.gz 2024-01-15T18:21:40.243 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61024.log 2024-01-15T18:21:40.243 INFO:teuthology.orchestra.run.smithi002.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.77866.log.gz 2024-01-15T18:21:40.244 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68145.log 2024-01-15T18:21:40.245 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66211.log.gz 2024-01-15T18:21:40.246 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61789.log 2024-01-15T18:21:40.246 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61024.log.gz 2024-01-15T18:21:40.247 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60886.log 2024-01-15T18:21:40.247 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68145.log.gz 2024-01-15T18:21:40.249 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.61789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78968.log 2024-01-15T18:21:40.250 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51712.log 2024-01-15T18:21:40.250 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60886.log.gz 2024-01-15T18:21:40.252 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78417.log 2024-01-15T18:21:40.252 INFO:teuthology.orchestra.run.smithi002.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78968.log.gz 2024-01-15T18:21:40.253 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51712.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51712.log.gz 2024-01-15T18:21:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46779.log 2024-01-15T18:21:40.254 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78417.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.78417.log.gz 2024-01-15T18:21:40.255 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59587.log 2024-01-15T18:21:40.256 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66421.log 2024-01-15T18:21:40.256 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46779.log.gz 2024-01-15T18:21:40.258 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50112.log 2024-01-15T18:21:40.258 INFO:teuthology.orchestra.run.smithi002.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.59587.log.gz 2024-01-15T18:21:40.259 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36706.log 2024-01-15T18:21:40.259 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66421.log.gz 2024-01-15T18:21:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50112.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50112.log.gz 2024-01-15T18:21:40.261 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67548.log 2024-01-15T18:21:40.262 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36706.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81439.log 2024-01-15T18:21:40.262 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36706.log.gz 2024-01-15T18:21:40.264 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74278.log 2024-01-15T18:21:40.264 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67548.log.gz 2024-01-15T18:21:40.265 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81439.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81439.log.gz 2024-01-15T18:21:40.265 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44388.log 2024-01-15T18:21:40.266 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74278.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.74278.log.gz 2024-01-15T18:21:40.267 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60377.log 2024-01-15T18:21:40.268 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73029.log 2024-01-15T18:21:40.268 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44388.log.gz 2024-01-15T18:21:40.270 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68101.log 2024-01-15T18:21:40.270 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60377.log.gz 2024-01-15T18:21:40.271 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40834.log 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73029.log.gz 2024-01-15T18:21:40.271 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-01-15T18:21:40.272 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68101.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68101.log.gz 2024-01-15T18:21:40.273 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78599.log 2024-01-15T18:21:40.274 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64746.log 2024-01-15T18:21:40.275 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40834.log.gz 2024-01-15T18:21:40.276 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51754.log 2024-01-15T18:21:40.276 INFO:teuthology.orchestra.run.smithi002.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78599.log.gz 2024-01-15T18:21:40.277 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76764.log 2024-01-15T18:21:40.277 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64746.log.gz 2024-01-15T18:21:40.278 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51754.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51754.log.gz 2024-01-15T18:21:40.279 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51884.log 2024-01-15T18:21:40.280 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76764.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76764.log.gz -5 2024-01-15T18:21:40.280 INFO:teuthology.orchestra.run.smithi002.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47157.log 2024-01-15T18:21:40.282 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42598.log 2024-01-15T18:21:40.282 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51884.log.gz 2024-01-15T18:21:40.283 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80264.log 2024-01-15T18:21:40.283 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47157.log.gz 2024-01-15T18:21:40.284 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42598.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42598.log.gz 2024-01-15T18:21:40.285 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55602.log/var/log/ceph/ceph-client.admin.80264.log: 2024-01-15T18:21:40.286 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80264.log.gz 2024-01-15T18:21:40.287 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54811.log 2024-01-15T18:21:40.288 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80850.log 2024-01-15T18:21:40.288 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55602.log.gz 2024-01-15T18:21:40.290 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54811.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54811.log.gz 2024-01-15T18:21:40.291 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80850.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80850.log.gz 2024-01-15T18:21:40.291 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59083.log 2024-01-15T18:21:40.293 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47714.log 2024-01-15T18:21:40.295 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59083.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-15T18:21:40.295 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59083.log.gz 2024-01-15T18:21:40.296 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47714.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47714.log.gz 2024-01-15T18:21:40.296 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40402.log 2024-01-15T18:21:40.298 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51329.log 2024-01-15T18:21:40.299 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40402.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40402.log.gz 2024-01-15T18:21:40.300 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60461.log 2024-01-15T18:21:40.301 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51329.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51329.log.gz 2024-01-15T18:21:40.302 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68057.log 2024-01-15T18:21:40.303 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60461.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60461.log.gz 2024-01-15T18:21:40.303 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77452.log 2024-01-15T18:21:40.305 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68057.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68057.log.gz 2024-01-15T18:21:40.306 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57862.log 2024-01-15T18:21:40.307 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77452.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77452.log.gz 2024-01-15T18:21:40.308 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60419.log 2024-01-15T18:21:40.309 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57862.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57862.log.gz 2024-01-15T18:21:40.310 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77784.log 2024-01-15T18:21:40.311 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60419.log.gz 2024-01-15T18:21:40.311 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71386.log 2024-01-15T18:21:40.312 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77784.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.77784.log.gz 2024-01-15T18:21:40.313 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58555.log 2024-01-15T18:21:40.315 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71386.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71386.log.gz 2024-01-15T18:21:40.316 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50371.log 2024-01-15T18:21:40.316 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58555.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58555.log.gz 2024-01-15T18:21:40.317 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62763.log 2024-01-15T18:21:40.319 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50371.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50371.log.gz 2024-01-15T18:21:40.320 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57778.log/var/log/ceph/ceph-client.admin.62763.log: 2024-01-15T18:21:40.320 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62763.log.gz 2024-01-15T18:21:40.321 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62054.log 2024-01-15T18:21:40.324 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57778.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57778.log.gz 2024-01-15T18:21:40.324 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70487.log 2024-01-15T18:21:40.324 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62054.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62054.log.gz 2024-01-15T18:21:40.325 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65227.log 2024-01-15T18:21:40.327 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70487.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70487.log.gz 2024-01-15T18:21:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58072.log 2024-01-15T18:21:40.328 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65227.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65227.log.gz 2024-01-15T18:21:40.329 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48714.log 2024-01-15T18:21:40.331 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58072.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58072.log.gz 2024-01-15T18:21:40.332 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48714.log: /var/log/ceph/ceph-client.admin.42682.log 2024-01-15T18:21:40.332 INFO:teuthology.orchestra.run.smithi002.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48714.log.gz 2024-01-15T18:21:40.333 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79273.log 2024-01-15T18:21:40.334 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/ceph-mds.d.log: 93.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-15T18:21:40.335 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42682.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42682.log.gz 2024-01-15T18:21:40.336 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.79273.log: --verbose -- /var/log/ceph/ceph-client.admin.66085.log 2024-01-15T18:21:40.336 INFO:teuthology.orchestra.run.smithi002.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79273.log.gz 2024-01-15T18:21:40.337 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61110.log 2024-01-15T18:21:40.339 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66085.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66085.log.gz 2024-01-15T18:21:40.340 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65059.log/var/log/ceph/ceph-client.admin.61110.log: 2024-01-15T18:21:40.340 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61110.log.gz 2024-01-15T18:21:40.341 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51669.log 2024-01-15T18:21:40.344 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65059.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65059.log.gz 2024-01-15T18:21:40.344 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51669.log: /var/log/ceph/ceph-client.admin.58301.log 2024-01-15T18:21:40.344 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51669.log.gz 2024-01-15T18:21:40.346 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81396.log 2024-01-15T18:21:40.347 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58301.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58301.log.gz 2024-01-15T18:21:40.348 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.81396.log: --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-01-15T18:21:40.349 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81396.log.gz 2024-01-15T18:21:40.350 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57255.log 2024-01-15T18:21:40.351 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41632.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-01-15T18:21:40.352 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48528.log 2024-01-15T18:21:40.353 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57255.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57255.log.gz 2024-01-15T18:21:40.354 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54769.log 2024-01-15T18:21:40.356 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48528.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48528.log.gz 2024-01-15T18:21:40.356 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54019.log 2024-01-15T18:21:40.357 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54769.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54769.log.gz 2024-01-15T18:21:40.358 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55819.log 2024-01-15T18:21:40.359 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54019.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54019.log.gz 2024-01-15T18:21:40.360 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72085.log 2024-01-15T18:21:40.360 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55819.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55819.log.gz 2024-01-15T18:21:40.361 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81228.log 2024-01-15T18:21:40.362 INFO:teuthology.orchestra.run.smithi137.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-15T18:21:40.363 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72085.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72085.log.gz 2024-01-15T18:21:40.364 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.81228.log: /var/log/ceph/ceph-client.admin.76494.log 2024-01-15T18:21:40.364 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81228.log.gz 2024-01-15T18:21:40.365 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41646.log 2024-01-15T18:21:40.367 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76494.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.76494.log.gz 2024-01-15T18:21:40.368 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56267.log 2024-01-15T18:21:40.368 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41646.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41646.log.gz 2024-01-15T18:21:40.369 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54517.log 2024-01-15T18:21:40.371 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56267.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56267.log.gz 2024-01-15T18:21:40.372 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.54517.log: /var/log/ceph/ceph-client.admin.63394.log 2024-01-15T18:21:40.372 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54517.log.gz 2024-01-15T18:21:40.373 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55476.log 2024-01-15T18:21:40.375 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63394.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63394.log.gz 2024-01-15T18:21:40.376 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67889.log 2024-01-15T18:21:40.376 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55476.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55476.log.gz 2024-01-15T18:21:40.377 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65833.log 2024-01-15T18:21:40.378 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67889.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67889.log.gz 2024-01-15T18:21:40.379 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.65833.log: /var/log/ceph/ceph-client.admin.80474.log 2024-01-15T18:21:40.380 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65833.log.gz 2024-01-15T18:21:40.381 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71937.log 2024-01-15T18:21:40.382 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80474.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80474.log.gz 2024-01-15T18:21:40.383 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77494.log/var/log/ceph/ceph-client.admin.71937.log: 2024-01-15T18:21:40.384 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71937.log.gz 2024-01-15T18:21:40.385 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57904.log 2024-01-15T18:21:40.387 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77494.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77494.log.gz 2024-01-15T18:21:40.388 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.57904.log: --verbose -- /var/log/ceph/ceph-client.admin.71218.log 2024-01-15T18:21:40.388 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57904.log.gz 2024-01-15T18:21:40.389 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45921.log 2024-01-15T18:21:40.390 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71218.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.71218.log.gz 2024-01-15T18:21:40.391 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57988.log 2024-01-15T18:21:40.392 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45921.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45921.log.gz 2024-01-15T18:21:40.393 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48444.log 2024-01-15T18:21:40.394 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57988.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57988.log.gz 2024-01-15T18:21:40.395 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49943.log 2024-01-15T18:21:40.396 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48444.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48444.log.gz 2024-01-15T18:21:40.397 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80180.log 2024-01-15T18:21:40.398 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49943.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.49943.log.gz 2024-01-15T18:21:40.399 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42556.log/var/log/ceph/ceph-client.admin.80180.log: 2024-01-15T18:21:40.400 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80180.log.gz 2024-01-15T18:21:40.401 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-01-15T18:21:40.402 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42556.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42556.log.gz 2024-01-15T18:21:40.403 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54685.log/var/log/ceph/ceph-client.admin.41969.log: 2024-01-15T18:21:40.404 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-01-15T18:21:40.405 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60930.log 2024-01-15T18:21:40.406 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54685.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54685.log.gz 2024-01-15T18:21:40.407 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50239.log 2024-01-15T18:21:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60930.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60930.log.gz 2024-01-15T18:21:40.408 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64308.log 2024-01-15T18:21:40.410 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50239.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50239.log.gz 2024-01-15T18:21:40.411 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.64308.log: --verbose -- /var/log/ceph/ceph-client.admin.61194.log 2024-01-15T18:21:40.411 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64308.log.gz 2024-01-15T18:21:40.412 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74743.log 2024-01-15T18:21:40.414 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61194.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61194.log.gz 2024-01-15T18:21:40.415 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63706.log 2024-01-15T18:21:40.415 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74743.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.74743.log.gz 2024-01-15T18:21:40.416 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50738.log 2024-01-15T18:21:40.418 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63706.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63706.log.gz 2024-01-15T18:21:40.419 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.50738.log: /var/log/ceph/ceph-client.admin.40120.log 2024-01-15T18:21:40.419 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50738.log.gz 2024-01-15T18:21:40.421 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42910.log 2024-01-15T18:21:40.421 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-01-15T18:21:40.422 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73661.log 2024-01-15T18:21:40.425 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42910.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42910.log.gz 2024-01-15T18:21:40.425 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36578.log 2024-01-15T18:21:40.425 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73661.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73661.log.gz 2024-01-15T18:21:40.427 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80934.log 2024-01-15T18:21:40.427 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.36578.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.36578.log.gz 2024-01-15T18:21:40.428 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74356.log 2024-01-15T18:21:40.429 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80934.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.80934.log.gz 2024-01-15T18:21:40.431 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74356.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.74356.log.gz 2024-01-15T18:21:40.431 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64620.log 2024-01-15T18:21:40.433 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71101.log 2024-01-15T18:21:40.435 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64620.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64620.log.gz 2024-01-15T18:21:40.435 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73501.log 2024-01-15T18:21:40.436 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71101.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71101.log.gz 2024-01-15T18:21:40.437 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76680.log 2024-01-15T18:21:40.438 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73501.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73501.log.gz 2024-01-15T18:21:40.439 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62889.log 2024-01-15T18:21:40.440 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76680.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76680.log.gz 2024-01-15T18:21:40.441 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78335.log 2024-01-15T18:21:40.442 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62889.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62889.log.gz 2024-01-15T18:21:40.443 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.78335.log: -- /var/log/ceph/ceph-client.admin.42514.log 2024-01-15T18:21:40.444 INFO:teuthology.orchestra.run.smithi002.stderr: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.78335.log.gz 2024-01-15T18:21:40.445 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68731.log 2024-01-15T18:21:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42514.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42514.log.gz 2024-01-15T18:21:40.447 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.68731.log: --verbose -- /var/log/ceph/ceph-client.admin.67590.log 2024-01-15T18:21:40.448 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68731.log.gz 2024-01-15T18:21:40.449 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68815.log 2024-01-15T18:21:40.450 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67590.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.67590.log.gz 2024-01-15T18:21:40.451 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.68815.log: /var/log/ceph/ceph-client.admin.60974.log 2024-01-15T18:21:40.452 INFO:teuthology.orchestra.run.smithi002.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.68815.log.gz 2024-01-15T18:21:40.453 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64042.log 2024-01-15T18:21:40.455 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60974.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60974.log.gz 2024-01-15T18:21:40.455 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75928.log 2024-01-15T18:21:40.456 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64042.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64042.log.gz 2024-01-15T18:21:40.457 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65917.log 2024-01-15T18:21:40.458 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75928.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75928.log.gz 2024-01-15T18:21:40.459 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44598.log/var/log/ceph/ceph-client.admin.65917.log: 2024-01-15T18:21:40.459 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65917.log.gz 2024-01-15T18:21:40.460 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44808.log 2024-01-15T18:21:40.462 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44598.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44598.log.gz 2024-01-15T18:21:40.463 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50581.log/var/log/ceph/ceph-client.admin.44808.log: 2024-01-15T18:21:40.464 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44808.log.gz 2024-01-15T18:21:40.464 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75550.log 2024-01-15T18:21:40.466 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50581.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50581.log.gz 2024-01-15T18:21:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73846.log 2024-01-15T18:21:40.467 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75550.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75550.log.gz 2024-01-15T18:21:40.469 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72253.log 2024-01-15T18:21:40.470 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73846.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73846.log.gz 2024-01-15T18:21:40.471 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5/var/log/ceph/ceph-client.admin.72253.log: --verbose -- /var/log/ceph/ceph-client.admin.40297.log 2024-01-15T18:21:40.471 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72253.log.gz 2024-01-15T18:21:40.472 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56563.log 2024-01-15T18:21:40.474 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40297.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40297.log.gz 2024-01-15T18:21:40.475 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79960.log 2024-01-15T18:21:40.475 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56563.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56563.log.gz 2024-01-15T18:21:40.476 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74827.log 2024-01-15T18:21:40.477 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79960.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.79960.log.gz 2024-01-15T18:21:40.479 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74827.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74827.log.gz 2024-01-15T18:21:40.479 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78681.log 2024-01-15T18:21:40.481 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46653.log 2024-01-15T18:21:40.482 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78681.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.78681.log.gz 2024-01-15T18:21:40.483 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74398.log 2024-01-15T18:21:40.485 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46653.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46653.log.gz 2024-01-15T18:21:40.485 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-15T18:21:40.485 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74398.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74398.log.gz 2024-01-15T18:21:40.486 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62096.log 2024-01-15T18:21:40.489 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.62096.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62096.log.gz 2024-01-15T18:21:40.490 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50197.log 2024-01-15T18:21:40.493 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58825.log 2024-01-15T18:21:40.494 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50197.log.gz 2024-01-15T18:21:40.495 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69923.log 2024-01-15T18:21:40.497 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58825.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58825.log.gz 2024-01-15T18:21:40.500 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-01-15T18:21:40.501 INFO:teuthology.orchestra.run.smithi002.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.69923.log.gz 2024-01-15T18:21:40.503 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55942.log 2024-01-15T18:21:40.505 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42304.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-01-15T18:21:40.506 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41978.log 2024-01-15T18:21:40.510 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55942.log: 87.5%/var/log/ceph/ceph-client.admin.41978.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41978.log.gz 2024-01-15T18:21:40.510 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56353.log 2024-01-15T18:21:40.510 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55942.log.gz 2024-01-15T18:21:40.512 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56353.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.56353.log.gz 2024-01-15T18:21:40.513 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66715.log 2024-01-15T18:21:40.516 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55308.log 2024-01-15T18:21:40.517 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66715.log.gz 2024-01-15T18:21:40.518 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30249.log 2024-01-15T18:21:40.521 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55308.log: 87.9%/var/log/ceph/ceph.tmp-client.admin.30249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52356.log 2024-01-15T18:21:40.521 INFO:teuthology.orchestra.run.smithi002.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30249.log.gz 2024-01-15T18:21:40.522 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55308.log.gz 2024-01-15T18:21:40.524 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58428.log 2024-01-15T18:21:40.528 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52356.log: /var/log/ceph/ceph-client.admin.58428.log: 91.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69965.log 2024-01-15T18:21:40.530 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52356.log.gz 2024-01-15T18:21:40.530 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58428.log.gz 2024-01-15T18:21:40.532 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59461.log 2024-01-15T18:21:40.535 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69965.log: /var/log/ceph/ceph-client.admin.59461.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69965.log.gz 2024-01-15T18:21:40.536 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59629.log 2024-01-15T18:21:40.537 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59461.log.gz 2024-01-15T18:21:40.539 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58385.log 2024-01-15T18:21:40.543 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58385.log: /var/log/ceph/ceph-client.admin.59629.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54329.log 2024-01-15T18:21:40.544 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59629.log.gz 2024-01-15T18:21:40.545 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58385.log.gz 2024-01-15T18:21:40.547 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2024-01-15T18:21:40.550 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50155.log: /var/log/ceph/ceph-client.admin.54329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79109.log 2024-01-15T18:21:40.552 INFO:teuthology.orchestra.run.smithi002.stderr: 86.9% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54329.log.gz 2024-01-15T18:21:40.553 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2024-01-15T18:21:40.554 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53890.log 2024-01-15T18:21:40.558 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53890.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53890.log.gz 2024-01-15T18:21:40.558 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70091.log 2024-01-15T18:21:40.562 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79109.log: /var/log/ceph/ceph-client.admin.70091.log: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65371.log 2024-01-15T18:21:40.562 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79109.log.gz 2024-01-15T18:21:40.564 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70091.log.gz 2024-01-15T18:21:40.567 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80222.log 2024-01-15T18:21:40.569 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80222.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65371.log.gz 2024-01-15T18:21:40.570 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71811.log 2024-01-15T18:21:40.571 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80222.log.gz 2024-01-15T18:21:40.573 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48274.log 2024-01-15T18:21:40.577 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48274.log: /var/log/ceph/ceph-client.admin.71811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69426.log 2024-01-15T18:21:40.579 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48274.log.gz 2024-01-15T18:21:40.579 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.71811.log.gz 2024-01-15T18:21:40.581 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62493.log 2024-01-15T18:21:40.584 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62493.log: /var/log/ceph/ceph-client.admin.69426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78130.log 2024-01-15T18:21:40.586 INFO:teuthology.orchestra.run.smithi002.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.69426.log.gz 2024-01-15T18:21:40.587 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62493.log.gz 2024-01-15T18:21:40.588 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80976.log 2024-01-15T18:21:40.592 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80976.log: /var/log/ceph/ceph-client.admin.78130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62932.log 2024-01-15T18:21:40.594 INFO:teuthology.orchestra.run.smithi002.stderr: 85.5% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.78130.log.gz 2024-01-15T18:21:40.594 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80976.log.gz 2024-01-15T18:21:40.596 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63496.log 2024-01-15T18:21:40.599 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63496.log: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77194.log 2024-01-15T18:21:40.599 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62932.log.gz 2024-01-15T18:21:40.600 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63496.log.gz 2024-01-15T18:21:40.602 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78253.log 2024-01-15T18:21:40.605 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78253.log: /var/log/ceph/ceph-client.admin.77194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78171.log 2024-01-15T18:21:40.607 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77194.log.gz 2024-01-15T18:21:40.607 INFO:teuthology.orchestra.run.smithi002.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.78253.log.gz 2024-01-15T18:21:40.608 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64849.log 2024-01-15T18:21:40.610 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58681.log 2024-01-15T18:21:40.611 INFO:teuthology.orchestra.run.smithi002.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.78171.log.gz 2024-01-15T18:21:40.612 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71059.log 2024-01-15T18:21:40.615 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64849.log: /var/log/ceph/ceph-client.admin.58681.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.64849.log.gz 2024-01-15T18:21:40.617 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58681.log.gz 2024-01-15T18:21:40.617 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77743.log 2024-01-15T18:21:40.617 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71059.log.gz 2024-01-15T18:21:40.619 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58639.log 2024-01-15T18:21:40.622 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77743.log: /var/log/ceph/ceph-client.admin.58639.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.77743.log.gz 2024-01-15T18:21:40.624 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.58639.log.gz 2024-01-15T18:21:40.624 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51627.log 2024-01-15T18:21:40.626 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64248.log 2024-01-15T18:21:40.628 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51627.log.gz 2024-01-15T18:21:40.629 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59948.log 2024-01-15T18:21:40.632 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64248.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64248.log.gz 2024-01-15T18:21:40.633 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59948.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59948.log.gz 2024-01-15T18:21:40.633 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67263.log 2024-01-15T18:21:40.635 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73930.log 2024-01-15T18:21:40.636 INFO:teuthology.orchestra.run.smithi002.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67263.log.gz 2024-01-15T18:21:40.638 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53794.log 2024-01-15T18:21:40.643 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73930.log: /var/log/ceph/ceph-client.admin.53794.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79191.log 2024-01-15T18:21:40.643 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.73930.log.gz 2024-01-15T18:21:40.644 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53794.log.gz 2024-01-15T18:21:40.644 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59419.log 2024-01-15T18:21:40.646 INFO:teuthology.orchestra.run.smithi137.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-15T18:21:40.649 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79191.log: /var/log/ceph/ceph-client.admin.59419.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79191.log.gz 2024-01-15T18:21:40.649 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63748.log 2024-01-15T18:21:40.650 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59419.log.gz 2024-01-15T18:21:40.652 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55071.log 2024-01-15T18:21:40.656 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63748.log: /var/log/ceph/ceph-client.admin.55071.log: 88.5%gzip -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.63748.log.gz /var/log/ceph/ceph-client.admin.44682.log 2024-01-15T18:21:40.657 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-01-15T18:21:40.658 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55071.log.gz 2024-01-15T18:21:40.659 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41642.log 2024-01-15T18:21:40.664 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44682.log: /var/log/ceph/ceph-client.admin.41642.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.44682.log.gz 2024-01-15T18:21:40.664 INFO:teuthology.orchestra.run.smithi002.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.60251.log 2024-01-15T18:21:40.665 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41642.log.gz 2024-01-15T18:21:40.667 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72169.log 2024-01-15T18:21:40.672 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60251.log: /var/log/ceph/ceph-client.admin.72169.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60125.log 2024-01-15T18:21:40.672 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60251.log.gz 2024-01-15T18:21:40.673 INFO:teuthology.orchestra.run.smithi002.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72169.log.gz 2024-01-15T18:21:40.674 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61928.log 2024-01-15T18:21:40.678 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60125.log: /var/log/ceph/ceph-client.admin.61928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52104.log 2024-01-15T18:21:40.679 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60125.log.gz 2024-01-15T18:21:40.682 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52104.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61928.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64578.log 2024-01-15T18:21:40.682 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-01-15T18:21:40.683 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52104.log.gz 2024-01-15T18:21:40.684 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76722.log 2024-01-15T18:21:40.686 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63958.log 2024-01-15T18:21:40.688 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7%/var/log/ceph/ceph-client.admin.76722.log: -- replaced with /var/log/ceph/ceph-client.admin.64578.log.gz 2024-01-15T18:21:40.689 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60293.log 2024-01-15T18:21:40.691 INFO:teuthology.orchestra.run.smithi002.stderr: 88.0%/var/log/ceph/ceph-client.admin.63958.log: -- replaced with /var/log/ceph/ceph-client.admin.76722.log.gz 2024-01-15T18:21:40.692 INFO:teuthology.orchestra.run.smithi002.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63958.log.gz 2024-01-15T18:21:40.693 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47367.log 2024-01-15T18:21:40.697 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60293.log: /var/log/ceph/ceph-client.admin.47367.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.60293.log.gz 2024-01-15T18:21:40.698 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77152.log 2024-01-15T18:21:40.698 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47367.log.gz 2024-01-15T18:21:40.699 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80808.log 2024-01-15T18:21:40.701 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56990.log 2024-01-15T18:21:40.703 INFO:teuthology.orchestra.run.smithi002.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77152.log.gz 2024-01-15T18:21:40.704 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77989.log 2024-01-15T18:21:40.707 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.56990.log: /var/log/ceph/ceph-client.admin.80808.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-01-15T18:21:40.708 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77989.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77989.log.gz 2024-01-15T18:21:40.710 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80808.log.gz 2024-01-15T18:21:40.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60167.log 2024-01-15T18:21:40.713 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-01-15T18:21:40.719 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56175.log 2024-01-15T18:21:40.719 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60167.log.gz 2024-01-15T18:21:40.723 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68773.log 2024-01-15T18:21:40.724 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56175.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56175.log.gz 2024-01-15T18:21:40.724 INFO:teuthology.orchestra.run.smithi137.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-15T18:21:40.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79578.log 2024-01-15T18:21:40.727 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68773.log.gz 2024-01-15T18:21:40.731 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62358.log 2024-01-15T18:21:40.732 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79578.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.79578.log.gz 2024-01-15T18:21:40.735 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62358.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62358.log.gz 2024-01-15T18:21:40.736 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69881.log 2024-01-15T18:21:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69881.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69881.log.gz 2024-01-15T18:21:40.742 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44724.log 2024-01-15T18:21:40.747 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73619.log 2024-01-15T18:21:40.748 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44724.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44724.log.gz 2024-01-15T18:21:40.753 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52146.log 2024-01-15T18:21:40.753 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.73619.log.gz 2024-01-15T18:21:40.754 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75382.log 2024-01-15T18:21:40.756 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52146.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52146.log.gz 2024-01-15T18:21:40.757 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75382.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75382.log.gz 2024-01-15T18:21:40.759 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54601.log 2024-01-15T18:21:40.760 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63184.log 2024-01-15T18:21:40.762 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54601.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54601.log.gz 2024-01-15T18:21:40.766 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63184.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63184.log.gz 2024-01-15T18:21:40.768 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53058.log 2024-01-15T18:21:40.770 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75167.log 2024-01-15T18:21:40.772 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53058.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53058.log.gz 2024-01-15T18:21:40.772 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46407.log 2024-01-15T18:21:40.774 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75167.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75167.log.gz 2024-01-15T18:21:40.774 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58156.log 2024-01-15T18:21:40.778 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40792.log 2024-01-15T18:21:40.780 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58156.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58156.log.gz 2024-01-15T18:21:40.780 INFO:teuthology.orchestra.run.smithi002.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46407.log.gz 2024-01-15T18:21:40.781 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44976.log 2024-01-15T18:21:40.782 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40792.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40792.log.gz 2024-01-15T18:21:40.788 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44976.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44976.log.gz 2024-01-15T18:21:40.789 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70897.log 2024-01-15T18:21:40.790 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79818.log 2024-01-15T18:21:40.792 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70897.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.70897.log.gz 2024-01-15T18:21:40.793 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65707.log 2024-01-15T18:21:40.795 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79818.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.79818.log.gz 2024-01-15T18:21:40.797 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76078.log 2024-01-15T18:21:40.799 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65707.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65707.log.gz 2024-01-15T18:21:40.801 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67727.log 2024-01-15T18:21:40.803 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76078.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.76078.log.gz 2024-01-15T18:21:40.805 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67101.log 2024-01-15T18:21:40.807 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67727.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.67727.log.gz 2024-01-15T18:21:40.810 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67101.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67101.log.gz 2024-01-15T18:21:40.812 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75210.log 2024-01-15T18:21:40.816 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77051.log 2024-01-15T18:21:40.818 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75210.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75210.log.gz 2024-01-15T18:21:40.818 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78212.log 2024-01-15T18:21:40.820 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77051.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.77051.log.gz 2024-01-15T18:21:40.822 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71769.log 2024-01-15T18:21:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78212.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.78212.log.gz 2024-01-15T18:21:40.824 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-15T18:21:40.829 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73071.log 2024-01-15T18:21:40.830 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mgr.y.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71769.log.gz 2024-01-15T18:21:40.834 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73071.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73071.log.gz 2024-01-15T18:21:40.836 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47926.log 2024-01-15T18:21:40.838 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65875.log 2024-01-15T18:21:40.840 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47926.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47926.log.gz 2024-01-15T18:21:40.841 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67931.log 2024-01-15T18:21:40.843 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65875.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65875.log.gz 2024-01-15T18:21:40.844 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63916.log 2024-01-15T18:21:40.846 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67931.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67931.log.gz 2024-01-15T18:21:40.847 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70217.log 2024-01-15T18:21:40.849 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63916.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63916.log.gz 2024-01-15T18:21:40.851 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40656.log 2024-01-15T18:21:40.853 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70217.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.70217.log.gz 2024-01-15T18:21:40.853 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66337.log 2024-01-15T18:21:40.855 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40656.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40656.log.gz 2024-01-15T18:21:40.859 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68015.log 2024-01-15T18:21:40.859 INFO:teuthology.orchestra.run.smithi002.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.66337.log.gz 2024-01-15T18:21:40.864 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45879.log 2024-01-15T18:21:40.865 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68015.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68015.log.gz 2024-01-15T18:21:40.869 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45879.log: 94.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-15T18:21:40.869 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45879.log.gz 2024-01-15T18:21:40.870 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58597.log 2024-01-15T18:21:40.873 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46989.log 2024-01-15T18:21:40.875 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58597.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.58597.log.gz 2024-01-15T18:21:40.876 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70175.log 2024-01-15T18:21:40.878 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46989.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46989.log.gz 2024-01-15T18:21:40.880 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53932.log 2024-01-15T18:21:40.882 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70175.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70175.log.gz 2024-01-15T18:21:40.885 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59293.log 2024-01-15T18:21:40.886 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53932.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53932.log.gz 2024-01-15T18:21:40.890 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59293.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59293.log.gz 2024-01-15T18:21:40.892 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55518.log 2024-01-15T18:21:40.896 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60717.log 2024-01-15T18:21:40.898 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55518.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55518.log.gz 2024-01-15T18:21:40.899 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59545.log 2024-01-15T18:21:40.900 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60717.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60717.log.gz 2024-01-15T18:21:40.902 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53431.log 2024-01-15T18:21:40.904 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59545.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.59545.log.gz 2024-01-15T18:21:40.904 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60556.log 2024-01-15T18:21:40.906 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53431.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53431.log.gz 2024-01-15T18:21:40.909 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44346.log 2024-01-15T18:21:40.910 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60556.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60556.log.gz 2024-01-15T18:21:40.912 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44346.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44346.log.gz 2024-01-15T18:21:40.913 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52974.log 2024-01-15T18:21:40.914 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53100.log 2024-01-15T18:21:40.917 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78927.log 2024-01-15T18:21:40.918 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53100.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53100.log.gz 2024-01-15T18:21:40.919 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49574.log 2024-01-15T18:21:40.920 INFO:teuthology.orchestra.run.smithi002.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52974.log.gz 2024-01-15T18:21:40.921 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78927.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.78927.log.gz 2024-01-15T18:21:40.924 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49574.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49574.log.gz 2024-01-15T18:21:40.925 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2024-01-15T18:21:40.926 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51371.log 2024-01-15T18:21:40.928 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48094.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2024-01-15T18:21:40.928 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53752.log 2024-01-15T18:21:40.932 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51371.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51371.log.gz 2024-01-15T18:21:40.932 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46205.log 2024-01-15T18:21:40.934 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53752.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53752.log.gz 2024-01-15T18:21:40.938 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74014.log 2024-01-15T18:21:40.938 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46205.log.gz 2024-01-15T18:21:40.940 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2024-01-15T18:21:40.942 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74014.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.74014.log.gz 2024-01-15T18:21:40.947 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70049.log 2024-01-15T18:21:40.947 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2024-01-15T18:21:40.949 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80044.log 2024-01-15T18:21:40.950 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70049.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70049.log.gz 2024-01-15T18:21:40.952 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64084.log 2024-01-15T18:21:40.954 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80044.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.80044.log.gz 2024-01-15T18:21:40.958 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81598.log 2024-01-15T18:21:40.958 INFO:teuthology.orchestra.run.smithi002.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.64084.log.gz 2024-01-15T18:21:40.960 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80516.log 2024-01-15T18:21:40.962 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81598.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.81598.log.gz 2024-01-15T18:21:40.966 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75592.log 2024-01-15T18:21:40.966 INFO:teuthology.orchestra.run.smithi002.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80516.log.gz 2024-01-15T18:21:40.968 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77948.log 2024-01-15T18:21:40.970 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75592.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.75592.log.gz 2024-01-15T18:21:40.974 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66001.log 2024-01-15T18:21:40.974 INFO:teuthology.orchestra.run.smithi002.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.77948.log.gz 2024-01-15T18:21:40.976 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69468.log 2024-01-15T18:21:40.979 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66001.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66001.log.gz 2024-01-15T18:21:40.982 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72987.log 2024-01-15T18:21:40.982 INFO:teuthology.orchestra.run.smithi002.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69468.log.gz 2024-01-15T18:21:40.984 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62577.log 2024-01-15T18:21:40.986 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72987.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.72987.log.gz 2024-01-15T18:21:40.988 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58513.log 2024-01-15T18:21:40.990 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62577.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62577.log.gz 2024-01-15T18:21:40.994 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71687.log 2024-01-15T18:21:40.994 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58513.log.gz 2024-01-15T18:21:40.996 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40960.log 2024-01-15T18:21:40.998 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71687.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.71687.log.gz 2024-01-15T18:21:41.002 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70343.log 2024-01-15T18:21:41.002 INFO:teuthology.orchestra.run.smithi002.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40960.log.gz 2024-01-15T18:21:41.004 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68392.log 2024-01-15T18:21:41.006 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70343.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70343.log.gz 2024-01-15T18:21:41.010 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68392.log.gz 2024-01-15T18:21:41.011 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55739.log 2024-01-15T18:21:41.013 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.49225.log 2024-01-15T18:21:41.016 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64891.log 2024-01-15T18:21:41.018 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.49225.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55739.log.gz 2024-01-15T18:21:41.021 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-01-15T18:21:41.023 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64891.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64891.log.gz 2024-01-15T18:21:41.027 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69687.log 2024-01-15T18:21:41.027 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-01-15T18:21:41.029 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40518.log 2024-01-15T18:21:41.031 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69687.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.69687.log.gz 2024-01-15T18:21:41.035 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77620.log 2024-01-15T18:21:41.035 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40518.log.gz 2024-01-15T18:21:41.037 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64476.log 2024-01-15T18:21:41.041 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46611.log 2024-01-15T18:21:41.043 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64476.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64476.log.gz 2024-01-15T18:21:41.043 INFO:teuthology.orchestra.run.smithi002.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.77620.log.gz 2024-01-15T18:21:41.045 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46611.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46611.log.gz 2024-01-15T18:21:41.047 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45423.log 2024-01-15T18:21:41.050 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46947.log 2024-01-15T18:21:41.052 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45423.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45423.log.gz 2024-01-15T18:21:41.054 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54415.log 2024-01-15T18:21:41.055 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46947.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46947.log.gz 2024-01-15T18:21:41.057 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59990.log 2024-01-15T18:21:41.060 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54415.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54415.log.gz 2024-01-15T18:21:41.061 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54372.log 2024-01-15T18:21:41.062 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59990.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59990.log.gz 2024-01-15T18:21:41.064 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45281.log 2024-01-15T18:21:41.065 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54372.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54372.log.gz 2024-01-15T18:21:41.069 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45281.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45281.log.gz 2024-01-15T18:21:41.070 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52020.log 2024-01-15T18:21:41.073 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57820.log 2024-01-15T18:21:41.075 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52020.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52020.log.gz 2024-01-15T18:21:41.077 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73323.log 2024-01-15T18:21:41.079 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57820.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.57820.log.gz 2024-01-15T18:21:41.082 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54987.log 2024-01-15T18:21:41.084 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73323.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73323.log.gz 2024-01-15T18:21:41.088 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54987.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54987.log.gz 2024-01-15T18:21:41.089 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-15T18:21:41.091 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2024-01-15T18:21:41.093 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44167.log 2024-01-15T18:21:41.095 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63790.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2024-01-15T18:21:41.097 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64933.log 2024-01-15T18:21:41.098 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44167.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44167.log.gz 2024-01-15T18:21:41.102 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66043.log 2024-01-15T18:21:41.102 INFO:teuthology.orchestra.run.smithi002.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64933.log.gz 2024-01-15T18:21:41.105 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48798.log 2024-01-15T18:21:41.107 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66043.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66043.log.gz 2024-01-15T18:21:41.110 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44766.log 2024-01-15T18:21:41.112 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48798.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48798.log.gz 2024-01-15T18:21:41.116 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44766.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44766.log.gz 2024-01-15T18:21:41.117 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69510.log 2024-01-15T18:21:41.119 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71981.log 2024-01-15T18:21:41.120 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69510.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69510.log.gz 2024-01-15T18:21:41.120 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78558.log 2024-01-15T18:21:41.122 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71981.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.71981.log.gz 2024-01-15T18:21:41.123 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2024-01-15T18:21:41.125 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78558.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78558.log.gz 2024-01-15T18:21:41.128 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56395.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2024-01-15T18:21:41.130 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41309.log 2024-01-15T18:21:41.131 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51976.log 2024-01-15T18:21:41.133 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41309.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41309.log.gz 2024-01-15T18:21:41.135 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44212.log 2024-01-15T18:21:41.136 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51976.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51976.log.gz 2024-01-15T18:21:41.138 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51840.log 2024-01-15T18:21:41.140 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.44212.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44212.log.gz 2024-01-15T18:21:41.143 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48882.log 2024-01-15T18:21:41.144 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51840.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.51840.log.gz 2024-01-15T18:21:41.146 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74194.log 2024-01-15T18:21:41.148 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48882.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.48882.log.gz 2024-01-15T18:21:41.153 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66673.log 2024-01-15T18:21:41.153 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74194.log.gz 2024-01-15T18:21:41.154 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2024-01-15T18:21:41.156 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66673.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66673.log.gz 2024-01-15T18:21:41.159 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76638.log 2024-01-15T18:21:41.161 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40347.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2024-01-15T18:21:41.166 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.76638.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76638.log.gz 2024-01-15T18:21:41.168 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64166.log 2024-01-15T18:21:41.171 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46163.log 2024-01-15T18:21:41.172 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64166.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.64166.log.gz 2024-01-15T18:21:41.175 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2024-01-15T18:21:41.177 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46163.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46163.log.gz 2024-01-15T18:21:41.179 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65749.log 2024-01-15T18:21:41.181 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40750.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gz 2024-01-15T18:21:41.182 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48316.log 2024-01-15T18:21:41.184 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65749.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.65749.log.gz 2024-01-15T18:21:41.188 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58997.log 2024-01-15T18:21:41.189 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48316.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48316.log.gz 2024-01-15T18:21:41.193 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41303.log 2024-01-15T18:21:41.193 INFO:teuthology.orchestra.run.smithi002.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.58997.log.gz 2024-01-15T18:21:41.195 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45239.log 2024-01-15T18:21:41.197 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41303.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41303.log.gz 2024-01-15T18:21:41.198 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79519.log 2024-01-15T18:21:41.204 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45239.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45239.log.gz 2024-01-15T18:21:41.205 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79519.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.79519.log.gz 2024-01-15T18:21:41.207 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66818.log 2024-01-15T18:21:41.211 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70655.log 2024-01-15T18:21:41.213 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66818.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66818.log.gz 2024-01-15T18:21:41.215 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62138.log 2024-01-15T18:21:41.216 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70655.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70655.log.gz 2024-01-15T18:21:41.217 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70981.log 2024-01-15T18:21:41.218 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62138.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62138.log.gz 2024-01-15T18:21:41.220 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72505.log 2024-01-15T18:21:41.222 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70981.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70981.log.gz 2024-01-15T18:21:41.225 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40458.log 2024-01-15T18:21:41.227 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72505.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.72505.log.gz 2024-01-15T18:21:41.229 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40458.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40458.log.gz 2024-01-15T18:21:41.231 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52890.log 2024-01-15T18:21:41.235 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70445.log 2024-01-15T18:21:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52890.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52890.log.gz 2024-01-15T18:21:41.237 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81354.log 2024-01-15T18:21:41.239 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70445.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.70445.log.gz 2024-01-15T18:21:41.240 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52932.log 2024-01-15T18:21:41.242 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81354.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81354.log.gz 2024-01-15T18:21:41.242 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45155.log 2024-01-15T18:21:41.244 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52932.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.52932.log.gz 2024-01-15T18:21:41.244 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53650.log 2024-01-15T18:21:41.246 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45155.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45155.log.gz 2024-01-15T18:21:41.249 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57643.log 2024-01-15T18:21:41.250 INFO:teuthology.orchestra.run.smithi002.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.53650.log.gz 2024-01-15T18:21:41.251 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56479.log 2024-01-15T18:21:41.256 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57643.log: /var/log/ceph/ceph-client.admin.56479.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56479.log.gz 2024-01-15T18:21:41.257 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61278.log 2024-01-15T18:21:41.260 INFO:teuthology.orchestra.run.smithi002.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57643.log.gz 2024-01-15T18:21:41.260 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65287.log 2024-01-15T18:21:41.262 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61278.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61278.log.gz 2024-01-15T18:21:41.263 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49490.log 2024-01-15T18:21:41.264 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65287.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65287.log.gz 2024-01-15T18:21:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49490.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49490.log.gz 2024-01-15T18:21:41.268 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-15T18:21:41.271 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2024-01-15T18:21:41.274 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42262.log 2024-01-15T18:21:41.276 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41307.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2024-01-15T18:21:41.281 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56225.log 2024-01-15T18:21:41.281 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42262.log.gz 2024-01-15T18:21:41.281 INFO:teuthology.orchestra.run.smithi002.stderr: 86.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-15T18:21:41.283 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-01-15T18:21:41.285 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56225.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56225.log.gz 2024-01-15T18:21:41.287 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42430.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2024-01-15T18:21:41.290 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46821.log 2024-01-15T18:21:41.293 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79009.log 2024-01-15T18:21:41.295 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46821.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46821.log.gz 2024-01-15T18:21:41.298 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77536.log 2024-01-15T18:21:41.300 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79009.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.79009.log.gz 2024-01-15T18:21:41.302 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59688.log 2024-01-15T18:21:41.304 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77536.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.77536.log.gz 2024-01-15T18:21:41.306 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-15T18:21:41.308 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59688.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59688.log.gz 2024-01-15T18:21:41.311 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75718.log 2024-01-15T18:21:41.314 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57129.log 2024-01-15T18:21:41.316 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75718.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75718.log.gz 2024-01-15T18:21:41.319 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57129.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57129.log.gz 2024-01-15T18:21:41.321 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53836.log 2024-01-15T18:21:41.322 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68479.log 2024-01-15T18:21:41.324 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53836.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.53836.log.gz 2024-01-15T18:21:41.327 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78804.log 2024-01-15T18:21:41.329 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68479.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68479.log.gz 2024-01-15T18:21:41.331 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68881.log 2024-01-15T18:21:41.333 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78804.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.78804.log.gz 2024-01-15T18:21:41.335 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49985.log 2024-01-15T18:21:41.337 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.68881.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68881.log.gz 2024-01-15T18:21:41.339 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-15T18:21:41.340 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49985.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49985.log.gz 2024-01-15T18:21:41.345 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60032.log 2024-01-15T18:21:41.348 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66757.log 2024-01-15T18:21:41.349 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64806.log 2024-01-15T18:21:41.351 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66757.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.66757.log.gz 2024-01-15T18:21:41.351 INFO:teuthology.orchestra.run.smithi002.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60032.log.gz 2024-01-15T18:21:41.353 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64806.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64806.log.gz 2024-01-15T18:21:41.357 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60844.log 2024-01-15T18:21:41.358 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74575.log 2024-01-15T18:21:41.360 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60844.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60844.log.gz 2024-01-15T18:21:41.360 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56027.log 2024-01-15T18:21:41.362 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74575.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74575.log.gz 2024-01-15T18:21:41.364 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73155.log 2024-01-15T18:21:41.366 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56027.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56027.log.gz 2024-01-15T18:21:41.366 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57297.log 2024-01-15T18:21:41.368 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73155.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73155.log.gz 2024-01-15T18:21:41.371 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57517.log 2024-01-15T18:21:41.372 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57297.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57297.log.gz 2024-01-15T18:21:41.375 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2024-01-15T18:21:41.376 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57517.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57517.log.gz 2024-01-15T18:21:41.377 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52314.log 2024-01-15T18:21:41.378 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2024-01-15T18:21:41.379 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63268.log 2024-01-15T18:21:41.381 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52314.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52314.log.gz 2024-01-15T18:21:41.383 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55644.log 2024-01-15T18:21:41.388 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63268.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63268.log.gz 2024-01-15T18:21:41.389 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55644.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55644.log.gz 2024-01-15T18:21:41.391 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59786.log 2024-01-15T18:21:41.393 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62805.log 2024-01-15T18:21:41.394 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59786.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59786.log.gz 2024-01-15T18:21:41.395 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48136.log 2024-01-15T18:21:41.397 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.62805.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.62805.log.gz 2024-01-15T18:21:41.398 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69342.log 2024-01-15T18:21:41.400 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48136.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48136.log.gz 2024-01-15T18:21:41.401 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65185.log 2024-01-15T18:21:41.403 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69342.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69342.log.gz 2024-01-15T18:21:41.405 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47115.log 2024-01-15T18:21:41.406 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65185.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65185.log.gz 2024-01-15T18:21:41.410 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47115.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47115.log.gz 2024-01-15T18:21:41.411 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66463.log 2024-01-15T18:21:41.412 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70007.log 2024-01-15T18:21:41.414 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66463.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66463.log.gz 2024-01-15T18:21:41.415 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54113.log 2024-01-15T18:21:41.416 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.70007.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70007.log.gz 2024-01-15T18:21:41.416 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78376.log 2024-01-15T18:21:41.420 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54113.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54113.log.gz 2024-01-15T18:21:41.422 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78376.log: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.78376.log.gz 2024-01-15T18:21:41.423 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67632.log 2024-01-15T18:21:41.426 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81782.log 2024-01-15T18:21:41.428 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67632.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67632.log.gz 2024-01-15T18:21:41.430 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73888.log 2024-01-15T18:21:41.432 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81782.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81782.log.gz 2024-01-15T18:21:41.432 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60636.log 2024-01-15T18:21:41.438 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73888.log: /var/log/ceph/ceph-client.admin.60636.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60636.log.gz 2024-01-15T18:21:41.438 INFO:teuthology.orchestra.run.smithi002.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73888.log.gz 2024-01-15T18:21:41.439 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73703.log 2024-01-15T18:21:41.442 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74056.log 2024-01-15T18:21:41.444 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73703.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.73703.log.gz 2024-01-15T18:21:41.445 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48630.log 2024-01-15T18:21:41.447 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.74056.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74056.log.gz 2024-01-15T18:21:41.448 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54559.log 2024-01-15T18:21:41.450 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48630.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.48630.log.gz 2024-01-15T18:21:41.452 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54559.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54559.log.gz 2024-01-15T18:21:41.453 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71512.log 2024-01-15T18:21:41.455 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46863.log 2024-01-15T18:21:41.456 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.71512.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71512.log.gz 2024-01-15T18:21:41.458 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73281.log 2024-01-15T18:21:41.460 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46863.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46863.log.gz 2024-01-15T18:21:41.461 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47073.log 2024-01-15T18:21:41.462 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73281.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73281.log.gz 2024-01-15T18:21:41.464 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42472.log 2024-01-15T18:21:41.466 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47073.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.47073.log.gz 2024-01-15T18:21:41.468 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42472.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42472.log.gz 2024-01-15T18:21:41.470 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57339.log 2024-01-15T18:21:41.472 INFO:teuthology.orchestra.run.smithi002.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.52707.log.gz 2024-01-15T18:21:41.472 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79437.log 2024-01-15T18:21:41.474 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57339.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57339.log.gz 2024-01-15T18:21:41.476 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53524.log 2024-01-15T18:21:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79437.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79437.log.gz 2024-01-15T18:21:41.478 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45381.log 2024-01-15T18:21:41.482 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48840.log 2024-01-15T18:21:41.484 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45381.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53524.log.gz 2024-01-15T18:21:41.484 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45381.log.gz 2024-01-15T18:21:41.486 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.48840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64518.log 2024-01-15T18:21:41.487 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48840.log.gz 2024-01-15T18:21:41.488 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64662.log 2024-01-15T18:21:41.491 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65665.log 2024-01-15T18:21:41.492 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64518.log.gz 2024-01-15T18:21:41.493 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64662.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64662.log.gz 2024-01-15T18:21:41.498 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65665.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65665.log.gz 2024-01-15T18:21:41.499 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60759.log 2024-01-15T18:21:41.502 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56087.log 2024-01-15T18:21:41.504 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60759.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60759.log.gz 2024-01-15T18:21:41.507 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75676.log 2024-01-15T18:21:41.509 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56087.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56087.log.gz 2024-01-15T18:21:41.511 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77578.log 2024-01-15T18:21:41.513 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75676.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75676.log.gz 2024-01-15T18:21:41.515 INFO:teuthology.orchestra.run.smithi002.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.69158.log.gz 2024-01-15T18:21:41.517 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77578.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.77578.log.gz 2024-01-15T18:21:41.518 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77907.log 2024-01-15T18:21:41.519 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42640.log 2024-01-15T18:21:41.521 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77907.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77907.log.gz 2024-01-15T18:21:41.522 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40560.log 2024-01-15T18:21:41.524 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42640.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42640.log.gz 2024-01-15T18:21:41.525 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2024-01-15T18:21:41.527 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40560.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40560.log.gz 2024-01-15T18:21:41.532 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45507.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2024-01-15T18:21:41.534 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-01-15T18:21:41.539 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47031.log 2024-01-15T18:21:41.540 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52422.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gz 2024-01-15T18:21:41.543 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-01-15T18:21:41.545 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47031.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47031.log.gz 2024-01-15T18:21:41.549 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73113.log 2024-01-15T18:21:41.550 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41637.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-01-15T18:21:41.553 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59209.log 2024-01-15T18:21:41.555 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73113.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73113.log.gz 2024-01-15T18:21:41.558 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53692.log 2024-01-15T18:21:41.560 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59209.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59209.log.gz 2024-01-15T18:21:41.563 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.53692.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53692.log.gz 2024-01-15T18:21:41.565 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80892.log 2024-01-15T18:21:41.567 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58343.log 2024-01-15T18:21:41.569 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80892.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80892.log.gz 2024-01-15T18:21:41.571 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69771.log 2024-01-15T18:21:41.573 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58343.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.58343.log.gz 2024-01-15T18:21:41.575 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75802.log 2024-01-15T18:21:41.576 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69771.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69771.log.gz 2024-01-15T18:21:41.579 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49532.log 2024-01-15T18:21:41.581 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75802.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.75802.log.gz 2024-01-15T18:21:41.583 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.49532.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49532.log.gz 2024-01-15T18:21:41.585 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54727.log 2024-01-15T18:21:41.586 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61066.log 2024-01-15T18:21:41.590 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45837.log 2024-01-15T18:21:41.591 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.61066.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61066.log.gz 2024-01-15T18:21:41.591 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67179.log 2024-01-15T18:21:41.592 INFO:teuthology.orchestra.run.smithi002.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54727.log.gz 2024-01-15T18:21:41.593 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73197.log 2024-01-15T18:21:41.595 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45837.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45837.log.gz 2024-01-15T18:21:41.597 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78499.log 2024-01-15T18:21:41.598 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.73197.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67179.log.gz 2024-01-15T18:21:41.598 INFO:teuthology.orchestra.run.smithi002.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73197.log.gz 2024-01-15T18:21:41.602 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66589.log 2024-01-15T18:21:41.604 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65791.log 2024-01-15T18:21:41.605 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66589.log: 87.5% 86.1% -- replaced with /var/log/ceph/ceph-client.admin.66589.log.gz 2024-01-15T18:21:41.605 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78499.log.gz 2024-01-15T18:21:41.606 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55233.log 2024-01-15T18:21:41.607 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65791.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65791.log.gz 2024-01-15T18:21:41.609 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77009.log 2024-01-15T18:21:41.610 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.55233.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55233.log.gz 2024-01-15T18:21:41.616 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77009.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81060.log 2024-01-15T18:21:41.616 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.77009.log.gz 2024-01-15T18:21:41.618 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75424.log 2024-01-15T18:21:41.619 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81060.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81060.log.gz 2024-01-15T18:21:41.620 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45197.log 2024-01-15T18:21:41.621 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.75424.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75424.log.gz 2024-01-15T18:21:41.624 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65143.log 2024-01-15T18:21:41.625 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.45197.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45197.log.gz 2024-01-15T18:21:41.625 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57736.log 2024-01-15T18:21:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65143.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65143.log.gz 2024-01-15T18:21:41.627 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64392.log 2024-01-15T18:21:41.630 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57736.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.57736.log.gz 2024-01-15T18:21:41.631 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42220.log 2024-01-15T18:21:41.633 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.64392.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64392.log.gz 2024-01-15T18:21:41.635 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46121.log 2024-01-15T18:21:41.637 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.42220.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42220.log.gz 2024-01-15T18:21:41.639 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46121.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46121.log.gz 2024-01-15T18:21:41.641 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66169.log 2024-01-15T18:21:41.644 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46365.log 2024-01-15T18:21:41.646 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66169.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.66169.log.gz 2024-01-15T18:21:41.646 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47634.log 2024-01-15T18:21:41.648 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46365.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46365.log.gz 2024-01-15T18:21:41.650 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63454.log 2024-01-15T18:21:41.652 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47634.log: 96.2% -- replaced with /var/log/ceph/ceph-client.0.76355.log.gz 2024-01-15T18:21:41.652 INFO:teuthology.orchestra.run.smithi002.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.47634.log.gz 2024-01-15T18:21:41.653 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63622.log 2024-01-15T18:21:41.654 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63454.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63454.log.gz 2024-01-15T18:21:41.657 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59335.log 2024-01-15T18:21:41.658 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63622.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63622.log.gz 2024-01-15T18:21:41.661 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66547.log 2024-01-15T18:21:41.662 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59335.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.59335.log.gz 2024-01-15T18:21:41.665 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.66547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51203.log 2024-01-15T18:21:41.667 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66547.log.gz 2024-01-15T18:21:41.670 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69729.log 2024-01-15T18:21:41.671 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51203.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51203.log.gz 2024-01-15T18:21:41.673 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50027.log 2024-01-15T18:21:41.674 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.69729.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69729.log.gz 2024-01-15T18:21:41.675 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54065.log 2024-01-15T18:21:41.677 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50027.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50027.log.gz 2024-01-15T18:21:41.682 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54065.log: 87.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72571.log 2024-01-15T18:21:41.684 INFO:teuthology.orchestra.run.smithi002.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54065.log.gz 2024-01-15T18:21:41.686 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59125.log 2024-01-15T18:21:41.687 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.72571.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.72571.log.gz 2024-01-15T18:21:41.689 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78294.log 2024-01-15T18:21:41.690 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.59125.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59125.log.gz 2024-01-15T18:21:41.692 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47554.log 2024-01-15T18:21:41.693 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.78294.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.78294.log.gz 2024-01-15T18:21:41.695 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54155.log 2024-01-15T18:21:41.696 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47554.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47554.log.gz 2024-01-15T18:21:41.698 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79692.log 2024-01-15T18:21:41.699 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.54155.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54155.log.gz 2024-01-15T18:21:41.702 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79692.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.79692.log.gz 2024-01-15T18:21:41.703 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79478.log 2024-01-15T18:21:41.706 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81483.log 2024-01-15T18:21:41.707 INFO:teuthology.orchestra.run.smithi002.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.79478.log.gz 2024-01-15T18:21:41.712 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80390.log 2024-01-15T18:21:41.713 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.81483.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81483.log.gz 2024-01-15T18:21:41.716 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80390.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80390.log.gz 2024-01-15T18:21:41.718 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52188.log 2024-01-15T18:21:41.720 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63058.log 2024-01-15T18:21:41.722 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52188.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52188.log.gz 2024-01-15T18:21:41.725 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58765.log 2024-01-15T18:21:41.727 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63058.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63058.log.gz 2024-01-15T18:21:41.729 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2024-01-15T18:21:41.730 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.58765.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58765.log.gz 2024-01-15T18:21:41.731 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47283.log 2024-01-15T18:21:41.733 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.40612.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2024-01-15T18:21:41.737 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56309.log 2024-01-15T18:21:41.738 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47283.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47283.log.gz 2024-01-15T18:21:41.742 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57601.log 2024-01-15T18:21:41.743 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.56309.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56309.log.gz 2024-01-15T18:21:41.747 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.57601.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57601.log.gz 2024-01-15T18:21:41.749 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-15T18:21:41.752 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60209.log 2024-01-15T18:21:41.755 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47842.log 2024-01-15T18:21:41.758 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.60209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52272.log 2024-01-15T18:21:41.760 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.47842.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47842.log.gz 2024-01-15T18:21:41.760 INFO:teuthology.orchestra.run.smithi002.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60209.log.gz 2024-01-15T18:21:41.764 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67017.log 2024-01-15T18:21:41.765 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.52272.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52272.log.gz 2024-01-15T18:21:41.768 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50539.log 2024-01-15T18:21:41.770 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.67017.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67017.log.gz 2024-01-15T18:21:41.772 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50539.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50539.log.gz 2024-01-15T18:21:41.774 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50281.log 2024-01-15T18:21:41.778 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80724.log 2024-01-15T18:21:41.780 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.50281.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50281.log.gz 2024-01-15T18:21:41.781 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77110.log 2024-01-15T18:21:41.785 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.80724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65329.log 2024-01-15T18:21:41.787 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.77110.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77110.log.gz 2024-01-15T18:21:41.787 INFO:teuthology.orchestra.run.smithi002.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.80724.log.gz 2024-01-15T18:21:41.788 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79314.log 2024-01-15T18:21:41.789 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.65329.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65329.log.gz 2024-01-15T18:21:41.792 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51466.log 2024-01-15T18:21:41.794 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79314.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.79314.log.gz 2024-01-15T18:21:41.798 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.51466.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51466.log.gz 2024-01-15T18:21:41.801 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41639.log 2024-01-15T18:21:41.804 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46737.log 2024-01-15T18:21:41.805 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.41639.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41639.log.gz 2024-01-15T18:21:41.807 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72848.log 2024-01-15T18:21:41.808 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.46737.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46737.log.gz 2024-01-15T18:21:41.809 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79068.log 2024-01-15T18:21:41.813 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.0.72848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63352.log 2024-01-15T18:21:41.815 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.79068.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.79068.log.gz 2024-01-15T18:21:41.818 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/ceph-client.admin.63352.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63352.log.gz 2024-01-15T18:21:41.840 INFO:teuthology.orchestra.run.smithi002.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-15T18:21:41.842 INFO:teuthology.orchestra.run.smithi002.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.61789.log.gz 2024-01-15T18:21:42.314 INFO:teuthology.orchestra.run.smithi002.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.56990.log.gz 2024-01-15T18:21:42.335 INFO:teuthology.orchestra.run.smithi137.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-15T18:21:42.663 INFO:teuthology.orchestra.run.smithi137.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-15T18:21:42.690 INFO:teuthology.orchestra.run.smithi002.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.49225.log.gz 2024-01-15T18:21:42.781 INFO:teuthology.orchestra.run.smithi137.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-15T18:21:43.069 INFO:teuthology.orchestra.run.smithi137.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-15T18:21:43.082 INFO:teuthology.orchestra.run.smithi137.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-15T18:21:43.087 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-01-15T18:21:43.087 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m3.285s 2024-01-15T18:21:43.087 INFO:teuthology.orchestra.run.smithi137.stderr:user 0m15.575s 2024-01-15T18:21:43.087 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m1.356s 2024-01-15T18:21:43.126 INFO:teuthology.orchestra.run.smithi002.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.72848.log.gz 2024-01-15T18:21:43.799 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-15T18:21:44.149 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-15T18:21:44.422 INFO:teuthology.orchestra.run.smithi002.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-15T18:21:44.428 INFO:teuthology.orchestra.run.smithi002.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-15T18:21:44.526 INFO:teuthology.orchestra.run.smithi002.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-15T18:21:44.727 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-15T18:21:44.732 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-01-15T18:21:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m4.932s 2024-01-15T18:21:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m27.088s 2024-01-15T18:21:44.733 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m4.285s 2024-01-15T18:21:44.733 INFO:tasks.ceph:Archiving logs... 2024-01-15T18:21:44.733 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/remote/smithi002/log 2024-01-15T18:21:44.734 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-15T18:21:48.068 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/remote/smithi137/log 2024-01-15T18:21:48.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-15T18:21:50.129 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-15T18:21:50.139 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-15T18:21:50.140 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-15T18:21:50.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-15T18:21:50.300 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-15T18:21:50.310 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-15T18:21:50.310 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T18:21:50.318 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-15T18:21:50.368 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-01-15T18:21:50.372 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-01-15T18:21:50.379 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T18:21:50.379 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-01-15T18:21:50.379 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +1450us[+1433us] +/- 82ms 2024-01-15T18:21:50.379 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1912us[-1929us] +/- 77ms 2024-01-15T18:21:50.380 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 28 +685us[ +669us] +/- 92ms 2024-01-15T18:21:50.380 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T18:21:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-15T18:21:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-01-15T18:21:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +1413us[+1413us] +/- 82ms 2024-01-15T18:21:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 -1944us[-1953us] +/- 77ms 2024-01-15T18:21:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 +661us[ +652us] +/- 92ms 2024-01-15T18:21:50.382 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-15T18:21:50.383 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-15T18:21:50.393 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-15T18:21:50.393 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-15T18:21:50.402 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-15T18:21:50.654 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-15T18:21:50.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T18:21:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:21:50.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-15T18:21:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:21:51.081 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-15T18:21:51.091 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-15T18:21:51.101 INFO:teuthology.task.internal:Duration was 1996.209726 seconds 2024-01-15T18:21:51.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-15T18:21:51.110 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-15T18:21:51.110 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-15T18:21:51.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-15T18:21:51.283 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T18:21:51.288 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-15T18:21:51.642 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-15T18:21:51.643 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-01-15T18:21:51.643 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-15T18:21:51.709 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-01-15T18:21:51.709 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-15T18:21:51.781 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-15T18:21:51.782 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-15T18:21:51.790 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-15T18:21:51.885 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-15T18:21:51.885 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-15T18:21:51.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-15T18:21:52.112 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-15T18:21:52.123 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-15T18:21:52.123 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-15T18:21:52.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-15T18:21:52.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-15T18:21:52.331 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:21:52.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:21:52.439 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-01-15T18:21:52.448 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-01-15T18:21:52.504 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:21:52.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:21:52.552 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-15T18:21:52.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-15T18:21:52.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-15T18:21:52.618 INFO:teuthology.task.internal:Transferring archived files... 2024-01-15T18:21:52.619 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/remote/smithi002 2024-01-15T18:21:52.619 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-15T18:21:52.732 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115/remote/smithi137 2024-01-15T18:21:52.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-15T18:21:52.838 INFO:teuthology.task.internal:Removing archive directory... 2024-01-15T18:21:52.838 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-15T18:21:52.846 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-15T18:21:52.900 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-15T18:21:52.911 INFO:teuthology.task.internal:Not uploading archives. 2024-01-15T18:21:52.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-15T18:21:52.922 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-15T18:21:52.922 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-15T18:21:52.929 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-15T18:21:52.981 INFO:teuthology.orchestra.run.smithi002.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 15 18:21 /home/ubuntu/cephtest 2024-01-15T18:21:52.988 INFO:teuthology.orchestra.run.smithi137.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 15 18:21 /home/ubuntu/cephtest 2024-01-15T18:21:52.991 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-15T18:21:53.001 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-15T18:21:53.022 INFO:teuthology.nuke:Checking targets against current locks 2024-01-15T18:21:53.046 DEBUG:teuthology.nuke:shortname: smithi002 2024-01-15T18:21:53.046 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T18:21:53.066 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:39:34.525941', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGVN2HzCjZzUS/rDqxDK4fI5PQ6Hqfv8NUsoE7qkLEBkoNhkeIfrx8AYn6LBKQs/L1bP/Xx3vpgnfhF63n6FjdE='} 2024-01-15T18:21:53.070 DEBUG:teuthology.nuke:shortname: smithi137 2024-01-15T18:21:53.070 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-15T18:21:53.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-15_17:39:27-fs:volumes-rishabh-clone-stats-testing-default-smithi/7517115', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-15 17:39:34.527854', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKbj9Ii0wnblDymIKws/pcdeuhJ0D9SbBrDDD61dzeofHosaESMxAJlvn1GsMltQv695/PYjgdKxaC35iKRBjj8='} 2024-01-15T18:21:53.119 INFO:teuthology.orchestra.console.smithi002:Power off 2024-01-15T18:21:53.120 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-15T18:21:53.144 DEBUG:teuthology.orchestra.console.smithi002:power off output: Chassis Power Control: Down/Off 2024-01-15T18:21:53.145 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:21:53.160 INFO:teuthology.orchestra.console.smithi137:Power off 2024-01-15T18:21:53.160 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-15T18:21:53.169 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-01-15T18:21:53.182 DEBUG:teuthology.orchestra.console.smithi137:power off output: Chassis Power Control: Down/Off 2024-01-15T18:21:53.182 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:21:53.204 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-01-15T18:21:57.172 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:21:57.205 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:21:57.282 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is on 2024-01-15T18:21:57.315 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is on 2024-01-15T18:22:01.284 DEBUG:teuthology.orchestra.console.smithi002:pexpect command: ipmitool -H smithi002.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:22:01.316 DEBUG:teuthology.orchestra.console.smithi137:pexpect command: ipmitool -H smithi137.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-15T18:22:01.394 DEBUG:teuthology.orchestra.console.smithi002:check power output: Chassis Power is off 2024-01-15T18:22:01.424 DEBUG:teuthology.orchestra.console.smithi137:check power output: Chassis Power is off 2024-01-15T18:22:01.495 INFO:teuthology.orchestra.console.smithi002:Power off completed 2024-01-15T18:22:01.525 INFO:teuthology.orchestra.console.smithi137:Power off completed 2024-01-15T18:22:01.630 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 1996.2097256183624 failure_reason: 'Test failure: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default mon.a-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 mon.b-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=351bee4afd7240b382703f065db83af7 status: fail success: false 2024-01-15T18:22:01.631 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-15T18:22:01.705 INFO:teuthology.run:FAIL